Grading Rubrics

Grading Rubrics
Caroline R.H. Wiley and Darrell Sabers
Department of Educational Psychology
Assessment Coordinating Council - February 11, 2009
Tucson, AZ
1
Overview
Why use rubrics?
Psychometric considerations
Types of scoring systems
Rubric construction and examples
The Rubric for Rubrics (ETS, 2006)
Assessment Coordinating Council - February 11, 2009
2
Tucson, AZ
2
Why Use Rubrics?
Mechanism for judging the quality of student
work (Arter & Chappuis, 2006)
High-stakes nature of grades
Clarity
Advantageous for diagnosing learning or
summarizing learning (Assessment for learning vs.
Assessment of learning)
Reduce subjectivity as much as possible
Assessment Coordinating Council - February 11, 2009
3
Tucson, AZ
3
How Do I know If I Need a Rubric?
If you administer...
Essay tests
Papers (research, theoretical, etc.)
Constructed response items (paragraphs, a
few sentences, 1-2 words)
Performance assessments (speeches, debates,
presentations, skits, skills (car repair, medical
procedures, scientific procedures, etc.)
Assessment Coordinating Council - February 11, 2009
4
Tucson, AZ
4
Psychometric Considerations
Content validity
Learning objectives
Bluffing
Absence of bias
Halo effect
Consistency
With others
With yourself
Assessment Coordinating Council - February 11, 2009
5
Tucson, AZ
5
Types of Scoring Systems
Checklists
Rating scales
Analytic rubrics
Holistic rubrics
Assessment Coordinating Council - February 11, 2009
6
Tucson, AZ
6
Checklists
Gronlund & Waugh (2009)
Assessment Coordinating Council - February 11, 2009
7
Tucson, AZ
7
Checklists
McMillan (2007)
Assessment Coordinating Council - February 11, 2009
8
Tucson, AZ
8
Rating Scales
Reynolds, Livingtson, & Wilson (2006)
Assessment Coordinating Council - February 11, 2009
9
Tucson, AZ
9
Analytic Rubric
McMillan (2007)
Assessment Coordinating Council - February 11, 2009
10
Tucson, AZ
10
Analytic Rubric
McMillan (2007)
Assessment Coordinating Council - February 11, 2009
11
Tucson, AZ
11
Holistic Rubric
McMillan (2007)
Assessment Coordinating Council - February 11, 2009
12
Tucson, AZ
12
Generic Rubric Construction
Define learning objectives
Determine purpose
Procedures vs. final product
Determine assessment purpose
Assessment for learning vs. Assessment of learning
Determine best type to use
Depends on all of the above
If using analytic, determine components and weights
Determine number of levels to use
3 to 7
Determine criteria for the levels
Representative of desired learning objectives
Parallel
Eliminate double-penalties
Assessment Coordinating Council - February 11, 2009
13
Tucson, AZ
13
Example
Task: Using a thermometer
Preparation
10%
Administration
10%
Clean-up
5%
Creativity
40%
Excelling
(20-25 points)
Grasps non-bulb end
and wipes thermometer
downward.
Leaves thermometer in
mouth for 3 minutes,
removes thermometer
grasping non-bulb end,
and reads temperature
to the nearest tenth of
a degree.
Correctly records
temperature on patient’s
record and cleans
thermometer with an
alcohol wipe.
Is very creative in the
overall process of
measuring temperature.
Proficient
(15-19 points)
Either does not grasp
non-bulb end OR does
not wipe thermometer
downward.
Only does 2 of the 3
actions described in
the “Excelling”
category.
Records temperature on
patient’s record, but
does so sloppily OR
cleans thermometer with
non-alcohol wipes.
A little creative.
Student has own style
of taking temperature.
Assessment Coordinating Council - February 11, 2009
14
Below Proficient
(1-14 points)
Is not creative
in grasping non-bulb
end.
Does not leave
thermometer in mouth
for 3 minutes.
Doesn’t record temperature
on patient’s record AND
does not clean thermometer
at all.
Not creative at all.
No originality present.
Tucson, AZ
14
Stevens & Levi (2005)
Assessment Coordinating Council - February 11, 2009
15
Tucson, AZ
15
Stevens & Levi (2005)
Assessment Coordinating Council - February 11, 2009
16
Tucson, AZ
16
Watch Out For Weights
Using different number of
points in each component
leads to misleading scores
Checklist: 5/17 = 29%
Persuasiveness: 3/17 = 18%
Delivery: 3/17 = 18%
Sensitivity: 3/17 = 18%
Holistic rating: 3/17 = 18%
Double-penalty
Holistic rating?
McMillan (2007)
Assessment Coordinating Council - February 11, 2009
17
Tucson, AZ
17
The Rubric For Rubrics
ETS (2006)
Criterion 1: Coverage/Organization
1A: Covers the right content
1B: Criteria are well organized
1C: Number of levels fits targets and uses
Criterion 2: Clarity
2A: Levels defined well
2B: Levels parallel
Assessment Coordinating Council - February 11, 2009
18
Tucson, AZ
18
Using the Rubric for Rubrics:
GRE Analyze an Argument
GENERAL DESCRIPTIONS:
Score 6: A cogent, well-articulated critique of the
argument and conveys meaning skillfully.
Score 5: A generally thoughtful, well-developed critique
of the argument and conveys meaning clearly.
Score 4: A competent critique of the argument and
conveys meaning adequately.
Score 3: Some competence in its analysis of the issue
and in conveying meaning but is obviously flawed.
Score 2: Serious weaknesses in analytical writing.
Score 1: Fundamental deficiencies in analytical writing.
Assessment Coordinating Council - February 11, 2009
19
Tucson, AZ
19
Summary
Choose the appropriate scoring method based on the
purpose
Look for rubrics made elsewhere and adapt as necessary
Your criteria should reflect your learning objectives
And should be weighted appropriately
Criteria should be clearly defined and parallel across
levels
Check for double-penalty criteria
Number of levels should be reasonably distinguishable
Check that scores reflect knowledge appropriately
Develop the rubric as you write the task
Physically organize the rubric to be efficient and clear
Take it for a dry run
Assessment Coordinating Council - February 11, 2009
20
Tucson, AZ
20
References
Arter, J., & Chappuis, J. (2006). Creating & recognizing quality rubrics.
Educational Testing Service. Upper Saddle River, NJ: Pearson.
Gronlund, N. E., & Waugh, C. K. (2009). Assessment of student achievement.
(9th ed.). Upper Saddle River, NJ: Pearson.
McMillan, J. H. (2007). Classroom assessment: Principles and practice for effective
standards-based instruction (4th ed.). Boston: Allyn and Bacon
Reynolds, C. R., Livingston, R., & Wilson,V. (2006). Measurement and
assessment in education. Upper Saddle River, NJ: Prentice Hall.
Stevens, D. D., & Levi, J. A. (2005). Introduction to rubrics. Sterling,VA: Stylus
Publishing, LLC.
Assessment Coordinating Council - February 11, 2009
21
Tucson, AZ
21