The Assessment Matrix - One Approach to Assessing Outcomes
|
Abstract Assessment strategies need to be manageable for the Classroom Teacher. All of the wonderful ideals surrounding Outcomes and Levels Assessment are meaningless if the teacher is overwhelmed by the processes involved. This article presents one approach that meets the five stated essentials of Outcomes Assessment while remaining manageable for the Classroom Teacher. |
Outcomes Assessment
The five stated essentials of Outcomes Assessment are that assessment should be:
Valid
Educative
Explicit
Fair
Comprehensive
From: http://www.curriculum.wa.edu.au/support/cd/CF/fwk03g.htm
In my opinion, the most important aspect is missing from the list above … Assessment must be “manageable”. If the burden of assessment becomes overwhelming for the classroom teacher, any attempt to achieve the five stated essentials will fail.
This article attempts to outline one approach to assessment that addresses all SIX essentials.
The Decisions
There are a number of decisions that need to be made before engaging in Outcomes, or “Levels” Assessment.
|
Decision 1 Are you aiming at discrete assessment “Tasks” that target a narrow range of Learning Areas/ Strands/ Outcomes, Or
|
As a secondary ‘specialist’ (ICT) teacher, my preference is for option “1”. I would find it unmanageable to use one Task to extract “Level” information about a wide range of Learning Areas. I sympathise with Primary School teachers who are expected to do exactly that. Obviously it can be done and it is being done in Secondary Schools in the form of Cross-Curricular Tasks. As outlined above, some “decisions” do need to be made. My first decision was to focus on discrete assessment Tasks that target one Learning Area and a narrow range of Strands.
The example cited in this article is used to assess student Levels in the Technology Strand of the Technology and Enterprise (T&E) Learning Area.
|
Decision 2 How many Levels will your assessment Task address? |
In an ideal world the Task should be sufficiently ‘open-ended’ to cater for a wide range of Levels. Realistically, however, it is best if you can fit all of your information on one sheet of standard A4 paper. This usually means restricting your Level range to four. In the typical secondary situation, this probably means Tasks that target from around Level 3 to Level 6. - You do need to be aware that some students may be outside the target range (above and below).
|
Decision 3 How many assessment Tasks will you use for the Topic/Subject/Course? How many times will you formally assess each Outcome? |
It should be noted that not all activities need to be assessed. It would be unmanageable to assess and record everything students do. Students must have the necessary skills required to demonstrate mastery of an Outcome. A considerable amount of class time will be spent on skills-development, particularly in Learning Areas such as T&E. I have found that two to three Assessment Tasks in a semester-long subject (2 hours per week for a semester) is manageable.
Developing the Matrix
Having made your Decisions and decided on a Task that addresses the Outcomes and allows Levels assessment, it is time to start putting together the actual documentation for the Task.
STEP 1 - Create a Table listing the Strand Statements targeted by the Task. In this example, only the “Technology” Strand is targeted. Most Tasks cover two, or three Strands.
STEP2 – In the right-hand column write a brief (simple) description of exactly what students must do to be credited with demonstrating mastery of that sub-strand statement.
This is the Key step. You must ‘translate’ the sub-Strand statements and phrase the requirements in terms of the Task. (In this example, students are required to create a CD ROM cover and an associated advertising Leaflet/Flier.)
The table becomes…
| TP 3 Uses an understanding of the relationship between aesthetics and social and environmental effects when generating and communicating designs, both graphically and through models; plans and achieves production, making safe and efficient use of resources; and evaluates whether criteria relating to functional and aesthetic criteria were met. | |
| TP 3.1 Investigating - Examines and identifies key design features, including aesthetic features, and environmental effects of products, systems, processes, services and environments. | List some of the design features you used on your CD ROM cover and Leaflet. (Text font, size, style, formatting, tables, borders, graphics, etc). Use appropriate referencing. |
| TP 3.2 Devising - Generates designs that take into account some social and environmental implications and communicates using a range of graphical representations, models and technical terms. | Submit your original Designs for your CD ROM Cover and Leaflet. Eg Hand drawn sketches. |
| TP 3.3 Producing - Plans and carries out the steps of production processes, making safe and efficient use of resources. | Submit your 'Time Plan' for this task indicating when you will be doing what. (Signed and Dated by your teacher) |
| TP 3.4 Evaluating - Assesses how well the ideas, products, systems, processes, services and environments used meet design requirements, including consideration of functional and aesthetic criteria. | Write a brief evaluation of the finished product assessing how well it meets the original design requirements. |
| TP 4 Considers the values and beliefs held by the developer and user when determining the appropriateness of technologies; applies this understanding when creating, communicating and justifying designs; and in planning, adapting and producing technologies includes social and environmental criteria when evaluating progress and results. | |
| TP 4.1 Investigating - Determines the appropriateness of technologies for communities and environments. | Putting leaflets in letter-boxes is one way of advertising a new product. Give another example of a method of advertising products like your CD ROM. Which one do you think is better? |
| TP 4.2 Devising - Creates and prepares design proposals that include options considered and reasons for choices made, and uses images to visualise ideas and work out how designs might be realised. | Your Designs should include reasons for your decisions and a sketch of the CD ROM cover and Leaflet highlighting the main features. |
| TP 4.3 Producing - Organises and implements production processes to own specifications, recognising hazards and adopting safe work practices. | Keep a diary of your activities and indicate how closely your actual day-to-day activities match your original Time Plan. |
| TP 4.4 Evaluating - Assesses the effectiveness of own designs, products, systems, processes, services and environments in relation to design requirements, including consideration of social and environmental criteria. | Your evaluation should include comments about the effectiveness of using Leaflets for advertising and other social and environmental issues such as: Impressions made on your customer, Paper wastage, etc. |
| TP 5 Understands the relationship between the needs of individuals, communities and environments and assesses the appropriateness of existing technologies; considers functional, aesthetic, social and environmental issues in design and production proposals; safely and effectively organises, implements and controls production so specified standards are achieved; and evaluates in an ongoing way and includes assessment of how results meet design requirements, including appropriate codes of conduct. | |
| TP 5.1 Investigating - Investigates and explains how the design, production and use of technologies are affected by the needs of communities and environments. | Describe three other methods of advertising new products such as your CD ROM. Outline the advantages and disadvantages of each. (This is best presented in a Table.) |
| TP 5.2 Devising - Creates and prepares design and production proposals that include an examination of a range of options; demonstrates consideration of functional, aesthetic, social and environmental issues; and communicates using graphics and technical languages associated with particular fields of technology. | Your Designs should be comprehensive and include a range of different approaches that could have been taken. |
| TP 5.3 Producing - Organises, implements and adjusts production processes based on detailed production plans, which are described using suitable technical languages and conventions and which achieve defined standards of quality and safety. | Keep your Time Plan updated so that it always represents what you intend doing. (Don't let it get out of date.) |
| TP 5.4 Evaluating - Assesses own products, systems, processes, services and environments according to specified design requirements, including ethical criteria and comparisons with similar technologies. | Your evaluation should include ethical considerations such as availability of word processors to all advertisers, copyright, word processing skills and general equity issues. |
| TP 6 Applies an understanding of the relationship between needs, availability of resources and existing circumstances to the development and production of technologies; justifies decisions relating to functional, aesthetic, social and environmental issues; effectively uses appropriate techniques, resources and equipment and undertakes tasks collaboratively; and writes evaluation reports based on recognised methods for assessing project outcomes. | |
| TP 6.1 Investigating - Analyses how needs, resources and circumstances affect the development and application of particular technologies. | Outline how advertising has changed over the past Century. Where appropriate, highlight any Technological, Environmental, or Social issues that motivated the change. |
| TP 6.2 Devising - Creates and prepares detailed design and production proposals that show how the ideas have been developed, justifying the functional, aesthetic, social and environmental choices made; and communicates using symbols, graphics and technical languages adapted to the needs of the audience. | Outline the reasons why your design changed. In particular, discuss the design decisions you made that impacted on your final design. |
| TP 6.3 Producing - Organises, implements and adjusts production processes involving efficient use of time and resources, collaborates with others on tasks carried out in specific sequences and achieves specified standards of quality and safety. | Work with a partner to create a CD ROM cover and advertising Leaflet for a mythical person, or group. Before you begin, develop a detailed Time Plan indicating the role of each person. In your final notes, outline exactly what each of you contributed. |
| TP 6.4 Evaluating - Prepares and presents evaluation reports using information from impact studies, product testing, market research and comparisons with similar work done by others. | Work with least two other people to research and develop a 'survey' aimed at recording the preferences of others in your class for each one of your three CD ROM labels. Present your findings to the rest of the class (verbally, or as a computer presentation). |
In the first trial of this approach, students were given the Table above as the assessment guide. Unfortunately many students found it difficult to interpret. The next stage was to re-arrange the information as a one-page “matrix” and give students the matrix, rather than the detailed Table. (The Table is available to students on the school Intranet.)
STEP 3 – Copy and paste the right-hand column into the Assessment Matrix as below …
Levels Matrix
Name: ________________________
Due Date: _____________________
Context: Information Processing Year 8 - CD ROM Cover and Leaflet
Technology Strand
| Sub-strand | Level 3 | Level 4 | Level 5 | Level 6 |
| Investigating Students investigate relevant issues and the values inherent in technological developments and applications of technology. They consider the implications for societies and environments when identifying opportunities and when meeting individual’s, group’s and communities’ needs. |
List some of the design features you used on your CD ROM cover and Leaflet. (Text font, size, style, formatting, tables, borders, graphics, etc).
Use appropriate referencing. |
Putting leaflets in letter-boxes is one way of advertising a new product. Give another example of a method of advertising products like your CD ROM. Which one do you think is better? | Describe three other methods of advertising new products such as your CD ROM. Outline the advantages and disadvantages of each. (This is best presented in a Table.) | Outline how advertising has changed over the past Century. Where appropriate, highlight any Technological, Environmental, or Social issues that motivated the change. |
| Devising Students generate ideas and prepare design and production proposals in response to the specifications developed from the design challenge. |
Submit your original Designs for your CD ROM Cover and Leaflet | Your Designs should include reasons for your decisions and a sketch of the CD ROM cover and Leaflet highlighting the main features. | Your Designs should be comprehensive and include a range of different approaches that could have been taken. | Outline the reasons why your designs changed. In particular, discuss the decisions you made that impacted on your design. |
| Producing
Students effectively manage production by adapting and adjusting processes in response to constraints and difficulties. |
Submit your 'Time Plan' for this task indicating when you will be doing what. | Your Time Plan should be Signed and Dated by your teacher before you begin working. | Keep a diary of your activities and indicate how closely your actual day-to-day activities match your original Time Plan. | Keep your Time Plan updated so that it always represents what you intend doing. (Don't let it get out of date.) Work with a partner to create a CD ROM cover and advertising Leaflet for a mythical person, or group. Before you begin, develop a detailed Time Plan indicating the role of each person. In your final notes, outline exactly what each of you contributed. |
| Evaluating
Students evaluate intentions, plans and actions with a view to modification and improvement. They develop and apply criteria to assess how well they have responded to the design challenge. |
Write a brief evaluation of the finished product assessing how well it meets the original design requirements. | Your evaluation should include comments about the effectiveness of using Leaflets for advertising and other social and environmental issues such as: Impressions made on your customer, Paper wastage, etc. | Your evaluation should include ethical considerations such as availability of word processors to all advertisers, copyright, word processing skills and general equity issues. | Work with least two other people to research and develop a 'survey' aimed at recording the preferences of others in your class for each one of your three CD ROM labels. Present your findings to the rest of the class (verbally, or as a computer presentation). |
Teacher Assessment
I have found it best to go through the actual Leveling process with the student beside me in a ‘show-me’ exercise. Students are encouraged to ‘assemble’ their documentation as per the Matrix sheet. Some students even use sub-Strand headings. As the assessor, I ask a series of questions …
“Show me your list of design features …”
“Show me your referencing …”
“Show me where you described one other method of advertising …”
“Show me where you described other methods of advertising …” and so on.
As the student shows they have addressed each descriptor a ‘tick’ is placed in the corresponding Matrix ‘box’. At some point the student is unable to show that they have addressed the indicator (or the way in which they have addressed it is not acceptable). That point indicates their ‘Level’.
There is obviously subjectivity and teacher-discretion involved. Regardless of other factors, we are not required to report sub-Strands. It is usual that the student receives ‘ticks’ in different levels across the Matrix. The teacher must then use their professional
judgment to assign an overall Level for the Strand.
Student Self Assessment
Students are encouraged to use the Matrix as a self-assessment (formative and summative) tool. In all cases where students carried out self-assessment, their assessment matched mine. The key to student self-assessment working so successfully is the care taken to write the descriptors in simple, precise, unambiguous language that student can understand.
Comments
You may have seen, or even used, similar assessment documents in the past.
The key feature of this approach is that the descriptors are written:
- in terms of the specific Task, and
- in simple, clear language students can understand.
Most of the documents I have seen simply copy and paste the Strand statements into a table similar to the Assessment Matrix described above. I know many teachers who find the Strand statements a little confusing. What hope have students of interpreting them? The approach described here relies on the teacher to ‘translate’ the Strand statements and rephrase them so that students know exactly what is expected.
The first few attempts at developing an Assessment Matrix took me around a week. I found it best to create the documents over a few days, rather than as a single sitting and to adopt a VERY critical approach. The aim is to keep it simple. Sometimes a single word makes all the difference.
Ideally, the Assessment Matrix should be ‘marked’ progressively during the activity, rather than at the end. It can be very time consuming to go through the entire Assessment Matrix with the whole class. I have found I can only complete four, or five in an hour (with everything else that has to happen during a lesson).
To “Matrix”, or “Rubric”?
The dictionary definitions of both terms are interesting. Neither is technically appropriate for assessment, but they are both catching on. The popularity of the movie “The Matrix” gives us a head start. My suggestion is to forget about “rubrics” and stick with the term: “matrix”.
What Next?
I now know of several people already using this approach. For a while I thought I was developing something new. I guess it goes to show that nothing is “new” and that we tend to waste a lot of time re-inventing the wheel.
Maybe we could all share our Tasks and Assessment Matrices along the lines of the “Curriculum Scenarios” posted on the
ecawa Web site.
(See: http://www.ecawa.asn.au/services/curriculum.htm
)
Maybe ecawa could get into the business of publishing this sort of thing … Pay teachers for their work. Make a profit. – It worked well for stawa.
For further information and templates see:
http://www.southwest.com.au/~jfuller/levels/documents2.htm
and
http://www.southwest.com.au/~jfuller/levels/spreadsheets1.htm
and
http://www.southwest.com.au/~jfuller/levels/spreadsheets1levels.htm
and
http://www.southwest.com.au/~jfuller/levels/spreadsheets1matrix.htm
Also see: Outcomes
and Assessment Simplified
Jim Fuller M. Ed.
Halls Head Community College
Internet: http://www.jfuller.org