Document 6522688

The pleasures and perils of
criterion referenced assessment:
Putting students at the centre of
learning in higher education
Merrilyn Goos
The University of Queensland
Here’s the plan
Student perspectives on assessment
Changing assessment policy and
practice: the UQ experience
Why criterion referenced assessment?
How can criterion referenced
assessment be effectively implemented?
Why assessment matters
Students can, with difficulty, escape from the
effects of poor teaching, they cannot
(by definition, if they want to graduate)
escape the effects of poor assessment.
(Boud, 1995)
1
Student-centred assessment
means …
specifying desired learning outcomes
specifying the assessment scheme with
criteria and standards
designing assessment tasks that have
authentic purposes and encourage a deep
approach to learning
providing students with timely feedback that
points out their strengths and weaknesses
and explains how to make improvements for
future learning
How do students experience
assessment?
Changing assessment policy and
practice: The UQ experience
Review of policy and practice in 1995/96:
no more 100% exams
must use a variety of modes of assessment
change from norm-referenced to criterionreferenced assessment
2
Planning for assessment
Transparency is essential - all information
about assessment requirements must be
provided in course specification
Assessment must match course objectives
Implementing assessment
Amount (Goldilocks principle)
Form
must be suitable to objectives
must use more than one form
Spread & timing
spread assessment across semester
rule of 2/3
Making judgments
about student performance
Use criteria and standards for each piece of
assessment
Aggregation of results for individual
assessment pieces requires judgment, must be
defensible, must be made explicit in course
specification
Final grade for course must reflect UQ grade
descriptors
3
UQ grade descriptors
1 Fail. Fails to demonstrate most or all of the basic requirements
of the course.
2 Fail. Demonstrates clear deficiencies in understanding and
applying fundamental concepts; communicates information or
ideas in ways that are frequently incomplete or confusing and
give little attention to the conventions of the discipline.
3 Fail. Demonstrates superficial or partial or faulty understanding
of the fundamental concepts of the field of study and limited
ability to apply these concepts; presents undeveloped or
inappropriate or unsupported arguments; communicates
information or ideas with lack of clarity and inconsistent
adherence to the conventions of the discipline.
UQ grade descriptors
4 Pass. Demonstrates adequate understanding and application of
the fundamental concepts of the field of study; develops routine
arguments or decisions and provides acceptable justification;
communicates information and ideas adequately in terms of the
conventions of the discipline.
5 Credit. Demonstrates substantial understanding of fundamental
concepts of the field of study and ability to apply these concepts
in a variety of contexts; develops or adapts convincing
arguments and provides coherent justification; communicates
information and ideas clearly and fluently in terms of the
conventions of the discipline.
UQ grade descriptors
6 Distinction. As for 5, with frequent evidence of
originality in defining and analysing issues or
problems and in creating solutions; uses a level, style
and means of communication appropriate to the
discipline and the audience.
7 High Distinction. As for 6, with consistent evidence
of substantial originality and insight in identifying,
generating and communicating competing
arguments, perspectives or problem solving
approaches; critically evaluates problems, their
solutions and implications.
4
Providing feedback
Progressive assessment
Feedback on every assessment piece
Student performance provides feedback
on teaching
Benefits of criterion referenced
assessment
Students are graded on the basis of the
quality of their work alone, not by reference
to how other students in the course perform.
Students have a clearer understanding of
what is required in assessment.
Teachers can more readily make and defend
judgments about students’ work.
Explicitness provides a language for
describing learning.
Criteria and standards
defined
Criteria are dimensions or properties or
characteristics of performance.
Example (essay task):
Comprehensiveness
Insight
Coherence
Communication
5
Criteria and standards
defined
Criteria are dimensions or properties or
characteristics of performance.
Standards are levels of attainment or quality
that represent performance benchmarks for
each criterion.
A criteria and standards
scheme
Criteria
(low)
Standards
(high)
Comprehensiveness
Insight
Coherence
Communication
Challenges
Coming to grips with the concept of a
standard
Working out how to set standards
Devising ways to communicate standards
to students and colleagues
Becoming proficient in the use of
standards
(Sadler, 2005)
6
Conceptualising standards
Experienced university teachers have
developed personal “standards” from:
accumulated discipline knowledge
recollection of having own work
graded as a student
previous personal experience in
grading student work
Ways of setting standards
Numerical cut offs
Tacit “in the head” knowledge
Exemplars
Verbal descriptions
A recommended process
for setting standards
Begin with a set of qualitative grading decisions (e.g.,
engage a group of teachers in discussing their
judgments when grading student work).
From these real judgments, identify the essential
characteristics (criteria) and levels of quality
(standards) in students’ work, selecting some works
as exemplars.
Develop an economical set of verbal descriptions and
exemplars embedded within the context of the
discipline that can be communicated to colleagues
and students.
7
Ways of
communicating standards
1. Decide on the number of standards
o
o
o
o
What is the best possible standard that can
be anticipated in this learning environment?
What is the least standard that will be
considered acceptable?
What standards lie between these two?
What standard can be anticipated as
unacceptable?
Ways of
communicating standards
2. Decide on how to label the standards
Grades
1-2
3-4
5
Letters
D
C
B
A
0-39%
40-59%
60-79%
>80%
% Bands
6-7
Marks
0-4
5-10
11-16
17-20
Labels
Fail
Competent
Advanced
Excellent
Ways of
communicating standards
3. Develop descriptions of each standard
for each criterion
o
o
o
o
Use brief, clear, specific language that is accessible
to students
Specify behaviour that can be demonstrated rather
than that which must be inferred
Avoid comparative terms (more, less)
Avoid terms that indicate quantity, frequency, or
amount (none, little, some; always, often,
sometimes, rarely)
8
A criteria and standards
scheme should …
be sufficiently specific with identifiable standards
be constructed so that each cell describes the
behaviour/activities of a student performing at that
standard
specify qualitative, rather than quantitative,
differences between standards for each criterion
have an internal hierarchy so that if headings are
removed, the scheme could be reassembled from the
evidence of the constituent statements alone
Criteria
Comprehensiveness
CC1
A
Demonstrates sophisticated
understanding of theoretical
perspectives on mathematics
education.
Engages thoughtfully
with a broad range of
these ideas to interpret
and critique past and/or
current professional
practices, and consider
possibilities for change.
B
C
Demonstrates clear
understanding of
theoretical perspectives
Applies a broad range of on mathematics
education, although
these ideas to interpret
and analyse professional some relevant material
may be omitted or its
practice.
significance overlooked.
Demonstrates thorough
understanding of theoretical
perspectives on mathematics
education.
D
E
Demonstrates some
understanding of theoretical
perspectives on mathematics
education, possibly marred by
omissions or misinterpretations.
Identifies some examples of
professional practice connected
with these ideas.
Demonstrates limited
understanding of theoretical
perspectives on mathematics
education.
Makes few connections between
these ideas and professional
practice.
Applies some of these ideas to
interpret relevant aspects of
professional practice.
Insight
CC2
Identifies significant
current issues and offers
insightful analysis and
critical evaluation of
these issues in the light
of theoretical
perspectives.
Identifies current issues and
offers critical analysis in the
light of theoretical perspectives.
Identifies some current issues
and explains their relevance in
terms of theoretical perspectives.
Identifies some issues but
relevance remains unclear.
Demonstrates limited ability to
identify current issues and
discuss relevance.
Coherence
CC3
Synthesises ideas and evidence
to produce cogent arguments and
conclusions.
Argument is thoughtfully
developed and elements are
sequenced and connected to
produce an integrated whole.
Produces reasoned arguments
and draws on evidence to
support conclusions.
Argument is logically developed
in terms of content and structure.
Produces well explained
arguments that connect ideas and
use evidence.
Ideas are presented in a sensible
sequence and move towards
conclusions.
Selection and sequencing of
ideas is not always logical or
connected.
Conclusions may not be well
supported by evidence or
arguments.
Difficult for reader to obtain
meaning from explanations.
Ideas are unconnected.
No conclusions drawn.
Communication
Presentation observes all
conventions of spelling,
punctuation, grammar.
Quoting and referencing are
technically correct, consistent,
and complete.
Writing communicates concisely
and effectively to intended
audience within the specified
word limit.
Presentation observes almost all
conventions of spelling,
punctuation, grammar.
Quoting and referencing are
complete, with minimal
technical errors or
inconsistencies.
Writing communicates clearly to
intended audience within
specified word limit.
Presentation observes most
conventions of spelling,
punctuation, grammar.
Quoting and referencing are
complete, although there may be
some technical errors or
inconsistencies.
Writing communicates
adequately to intended audience
within specified word limit.
Presentation is marred by some
errors in spelling, punctuation,
grammar.
There may be omissions or
inconsistencies in quoting and
referencing.
Format and structure make it
difficult for audience to follow
the writer’s intentions.
Presentation is compromised by
many errors in spelling,
punctuation, grammar.
Quoting and referencing are
incomplete or inconsistent.
Little attention given to format
and structure in communicating
to audience.
CC4
Criteria
Comprehensiveness
A
Demonstrates sophisticated
understanding of theoretical
perspectives on mathematics
education.
Engages thoughtfully with a broad
range of these ideas to interpret and
critique past and/or current
professional practices, and consider
possibilities for change.
Coherence
Synthesises ideas and evidence to
produce cogent arguments and
conclusions.
Argument is thoughtfully developed
and elements are sequenced and
connected to produce an integrated
whole.
C
Demonstrates clear understanding of
theoretical perspectives on
mathematics education, although
some relevant material may be
omitted or its significance
overlooked.
Applies some of these ideas to
interpret relevant aspects of
professional practice.
Produces well explained arguments
that connect ideas and use evidence.
Ideas are presented in a sensible
sequence and move towards
conclusions
9
Becoming proficient in
using standards
Develop in collaboration with colleagues in
your discipline.
Use in conjunction with exemplars of student
work.
Use social moderation processes to develop
shared understanding of standards amongst
teaching staff.
Use criteria and standards schemes to
provide feedback to students.
Criterion referenced
assessment: The pleasures
Can increase student motivation and
engagement in assessment.
Can encourage use of greater variety of
assessment tasks.
Criteria & standards can communicate to
students expectations regarding quality.
Grading with criteria & standards can
provide feedback to students about their
learning.
Development of criteria & standards can
stimulate academic conversations around
assessment.
Criterion referenced
assessment: The perils
Developing explicit, usable criteria &
standards takes time and effort.
Providing criteria & standards to students
does not guarantee they will understand
or use them!
Temptation to over-specify criteria &
standards risks trivialising and
fragmenting learning.
Aggregating results from individual
assessment tasks obscures overall pattern
of achievement on different criteria.
10
Are we there yet?
Assessment change is a long term process and people
are at different stages of the journey.
There are different but equally valuable ways to
implement good assessment practices that align with
university policies.
Assessment policies need regular monitoring and
review.
Teachers benefit from regular review of their
assessment practices by consulting with peers,
responding to student feedback, maintaining
currency with new policies.
The pleasures and perils of criterion
referenced assessment:
Putting students at the centre of
learning in higher education
Merrilyn Goos
The University of Queensland
11