Re-examining Kirkpatrick`s Four Levels

Transcription

502
Re-examining
Kirkpatrick’s Four Levels
Joe Pulichino, The eLearning Guild
December 7 & 8, 2006
Produced by
Designing & Developing
Online Assessments & Evaluations
Online Assessments and Evaluations
Re-examining the Usage and Value of
Kirkpatrick’s Four Levels of Training Evaluation
Joe Pulichino, Ed.D
Principal Consultant
Athena Learning Group
Historical and Critical Perspective
The history of Kirkpatrick’s four level
taxonomy, how it became a “model” for
training evaluation, and what critics have had
to say about it over the years.
2
Session 502 – Re-examining Kirkpatrick’s Four Levels, Joe Pulichino, The
eLearning Guild
Page 1
Current Practitioner Research
The history of Kirkpatrick’s four level taxonomy, how it became a
“model” for training evaluation, and what the critics have had to say
about it over the years.
A snapshot of the current usage of
Kirkpatrick’s four levels based on in-depth
research of The eLearning Guild’s 20,000+
practitioners.
3
Getting Value Out of Kirkpatrick
The history of Kirkpatrick’s four level taxonomy, how it became a
“model” for training evaluation, and what the critics have had to say
about it over the years.
A snapshot of the current usage of Kirkpatrick’s four levels based on
in-depth research of The eLearning Guild’s 20,000+ practitioners.
• How organizations get value out of using
Levels 3 and 4?
4
eLearning Guild
Page 2
Right Steps, Wrong Order
The history of Kirkpatrick’s four level taxonomy, how it
became a “model” for training evaluation, and what the
critics have had to say about it over the years.
A snapshot of the current usage of Kirkpatrick’s four
levels based on in-depth research of The eLearning
Guild’s 20,000+ practitioners.
How organizations get value out of using Levels 3 and 4?
Some suggestions, based on the
research, about getting value from
Kirkpatrick’s four levels by using them
in reverse order.
5
Poll #1: Rate your interest in each of today’s topics
Historical and Critical
Perspective
Current Practitioner
Research
Getting Value Out of
Kirkpatrick
Right Steps, Wrong
Order
6
eLearning Guild
Page 3
Who is this guy you’re listening to?
Former Director of Research, eLearning Guild
Consultant – Athena Learning
30 years in education
joe@AthenaLearn.com
7
About the Research
The data and charts included in this
presentation have been taken from
the recently published,
Usage and Value of Kirkpatrick’s
Four Levels of Training Evaluation
by Joe Pulichino.
You can obtain a copy of this report,
please visit the official Website of
The eLearning Guild:
www.elearningguild.com
8
eLearning Guild
Page 4
The Problem of Evaluation
“If you can’t measure it, you can’t manage it”
So, how best to evaluate the effectiveness of training
in a timely, cost-effective, and useful manner?
The Kirkpatrick Four Levels
•Industry standard
•Not fully implemented in most organizations
•Critical problems identified in the literature
•Are the Kirkpatrick four levels a viable solution?
9
Poll #2:
Rate your level of knowledge of Kirkpatrick’s four levels of
training evaluation? (Select only one)
•Highly knowledgeable
•Very Knowledgeable
•Fairly Knowledgeable
•Not Very Knowledgeable
•Not at All Knowledgeable
10
eLearning Guild
Page 5
Respondents’ Level of Knowledge
11
Kirkpatrick’s Quiz on the Four Levels
Q 1 - 4. What are the four levels?
Q 5. What do we call a test we administer before a training program.
Q 6. What do we call a test we administer after a training program
Q 7. What do we call a group that receives the training?
Q 8. What do we call a group that does not receive the training?
True or False:
Q9. You can borrow evaluation results from another organization or department if the
subject and leader are the same?
Q10.Those who plan, instruct and evaluate are better able to evaluate a program than
are the learners.
12
eLearning Guild
Page 6
Poll # 3
True or False
9. You can borrow evaluation results from another
organization or department if the subject and leader are
the same.
10. Those who plan, instruct and evaluate are better able
to evaluate a program than are the learners.
13
My Quiz Answers
1-4. What are the four levels?
5. What do we call a test we administer before a
training program.
6. What do we call a test we administer after a
training program?
7. What do we call a group that receives the
training?
8. What do we call a group that does not receive
the training?
True or False
9. You can borrow evaluation results from another
organization or department if the subject and
leader are the same.
10. Those who plan, instruct and evaluate are
better able to evaluate a program than are the
learners.
1. Reaction
2. Learning
3. Behavior
4. Results
5. pre-test
6. post-test
7. experiemental
group
8. control group
9. False
10. True

Did I pass?
14
eLearning Guild
Page 7
Dr. Kirkpatrick’s Evaluation
Joe:
You just barely passed.
The last answer is false and there is no hyphen in
pretest and posttest (according to my dissertation!).
Also, I deduct some points for misspelling experimental.
Don
15
My Response
“As far as Question 10 goes, I have a question for you. I
answered "false" because it seems that this statement is
not true in all cases, especially in regard to the four levels.
In terms of evaluating a program at Levels 3 or 4, aren't
professionals better than learners at evaluating a program?
How would a learner even begin to evaluate a program for
"results"? That's hard enough for professionals. How would
you answer this question? I used this quiz at my
presentation last week and this issue came up and it
provoked something of a debate.”
16
eLearning Guild
Page 8
The Kirkpatrick Shift
“Congratulations on passing - even if by a hair.
I like your answer to Number 10. For a joke, I have said the
correct answer is True. But then I ask people how they
voted. Practically all said False so I changed it. Next time I
won't change it.”
17
Corporate Training (1952)
18
eLearning Guild
Page 9
A Short History of Kirkpatrick’s Four Levels
ASTD
Standard
1976
First articles
on four-steps
in ASTD 1959
State of
the Art
1968
1959
Evidence
vs. Proof
1977
1978
Newstrom’s
Caution
Great
Ideas
Revisited
1989
Alliger &
Janak’s
Citation
Review
1994
1996
Holton’s
Critique
19
1998
2005
Bledsloe,
Anderson,
Dye
Kirkpatrick’s first articles on the four-steps
of training evaluation (1959)
Started with his doctoral studies
Context of the times: classroom, non-strategic, no evaluation
A successive four step process
• Reaction
• Learning
• Behavior
• Results
Purpose of evaluation
• Continue programs
• Improve programs
• Justify the training department
Limitations
• Correlations among levels
• Intervening variables
Intention
• Get started
• Develop the practice
20
eLearning Guild
Page 10
Kirkpatrick’s state of the art
of training evaluation (1968)
Determine and analyze current techniques
Four-step approach codified in ASTD Handbook (1967)
Nascent discipline
Reaction #1 at 78% of 110 businesses
Less than half using steps #2, 3, 4
Most of these efforts are superficial and subjective
No positive correlation between steps – only deals with frequency of
usage and description of techniques of each step
“future surveys and research will find that the ‘state of the art’ is
far advanced from where we find it today”

21
How did the four steps become
the four level evaluation model?
“In the November 1959 article, I used the term ‘four
steps.’ But someone, I don’t know who, referred to the
steps as ‘levels.’ The next thing I knew, articles and
books were referring to the four levels as the Kirkpatrick
model”. (1996)
“Trainers began to accept my four levels and passed the
word to others. Soon, it became the ‘Kirkpatrick model’.
The concepts, principles, and techniques were
communicated from one professional to another as
trainers began to apply one or more of the levels.” (2005)
22
eLearning Guild
Page 11
Poll # 4
Are the Kirkpatrick four levels a taxonomy or a model?
Does it make an difference?
23
Holton’s critique:
The Flawed Four-Level Evaluation Model (1996)
“Acknowledged by many practitioners as the standard in the field”
Importance of Holton’s critique
• Explains flaws of four levels, and
• Proposes concretely what is needed
Taxonomy versus Model
• Model (units, relationships, limits, systems states, deductions,
predictions)
• No model can be validated without measuring and accounting for
intervening variables.
Research supporting integrative evaluation models
Proposed model: influence on learning, performance, and results
outcomes
“No theoretical reason why a practitioner model
more sophisticated than the four level approach
but simpler to implement than the one described
here could not be developed.”
24
eLearning Guild
Page 12
Kirkpatrick’s response to Holton:
Great Ideas Revisited (1996)
“Holton says the model isn’t a model at all, but a
taxonomy, a classification”.
“Perhaps he is correct”.
“I don’t care whether it is a model or a taxonomy
as long as professionals find it useful in evaluating
programs”.
Simplicity – folks:
• “…aren’t much interested in a scholarly,
complex approach”.
Purpose:
• “Clarify the meaning of evaluation and offer
guidelines on how to get started and proceed”.
25
Poll #5
Rate the extent to which your organization uses each of
the Kirkpatrick four levels?
26
eLearning Guild
Page 13
Usage of Kirkpatrick’s Four Levels
27
Usage of Kirkpatrick’s Four Levels
28
eLearning Guild
Page 14
What drives the training budget?
29
What drives the training budget?
30
eLearning Guild
Page 15
Poll #6
What are some of the drivers
of Level 3 and 4 usage?
31
What are some of the drivers
of Level 3 and 4 usage?
Size of Training Budget
No
Competitive Pressures
Yes
Need for a Knowledgeable
& Skilled Workforce
Yes
32
eLearning Guild
Page 16
Why Organizations Use Kirkpatrick Level 3
33
Why Organizations Use Level 4
34
eLearning Guild
Page 17
When organizations use Level 3 and 4
evaluations, they have good reason to do so…
Why 3?
Why 4?
¨ Demo training impact
on performance – 80%
¨ Determine that results
were achieved – 80%
¨ Improve future
programs – 78%
¨ Demo training impact
on results – 76%
¨ Determine performance
changed – 74%
¨ Improve future
programs– 71%
¨ Justify Training – 44%
¨ Justify Training – 43%
35
Chat Answers
What do you think are the
biggest challenges to using
Levels 3 and 4?
36
eLearning Guild
Page 18
Why Organizations Do Not Use Level 3
37
We don’t use Level 3…
Why Not?
¨ Access to Data – 65%
¨ No Management
Support – 63%
¨ Too Time Consuming –
57%
38
eLearning Guild
Page 19
Challenges Implementing Level 3
39
Obstacles we had to overcome
using Level 3…
Obstacles We
Overcame:
Why Not?
¨ Time Required – 56%
¨ No Management
Support – 63%
¨ Making it a priority for
the training people 48%
57%
40
eLearning Guild
Page 20
Why Organizations Do Not Use Level 4
41
We don’t use Level 4…
Why Not?
65%
¨ No Management Support
– 59%
42
eLearning Guild
Page 21
Challenges Implementing Level 4
43
Obstacles we had to overcome
using Level 4…
Obstacles We
Overcame:
Why Not?
65%
¨ Time Required – 63%
¨ Expertise Required – 50%
¨ No Management Support
– 59%
44
eLearning Guild
Page 22
What about intervening variables?
What intervening variables do you
think should be measured as part of a
Level 3 or 4 evaluation?
45
Considering Intervening Variables
at Level 3
46
eLearning Guild
Page 23
Considering Intervening Variables
at Level 4
47
Organizations who use Level 3 and 4
evaluations obtain valuable data in terms
of measuring…
Effectiveness of Training Programs – 68%
Desired Change in Job Performance – 72%
Desired Business or Organizational Result – 74%
48
eLearning Guild
Page 24
How to get valuable data…
“Even though a training director may have done a
masterful job measuring trainees' reactions, that's no
assurance that any learning has taken place. Nor is that
an indication that participants' behavior will change
because of the training. And still farther away is any
indication of results that can attributed to the training.”
Levels 3 and 4 require “a more scientific approach and
the consideration of many factors” such as motivation to
improve, work environment, and opportunity to practice
the newly acquired knowledge or skills.
49
Level 3 Value of Data Obtained
50
eLearning Guild
Page 25
Level 4 Value of Data Obtained
51
The value of the data obtained
increases when…
Relationships between levels are measured
•Levels 2 & 3
•Levels 3 & 4
Intervening variables are considered
•Learner motivation to perform
•Opportunity to apply learning,
•Alignment of training with business results
•Organizational capability
52
eLearning Guild
Page 26
Data points worth considering…
• As the importance of competitive pressures and the
need for a knowledgeable and skilled workforce
increases, the usage of Levels 3 and 4 increases.
• The more that evaluators consider correlations
between the outcomes of Levels 1, 2, 3, and 4, the
more valuable the data obtained.
• The more that evaluators consider intervening
variables in Level 3 and 4 evaluations, the more
valuable the data obtained.
53
Suggestions
If you are doing Level 3 and 4 evaluations, make sure
your organization really cares about behavior and results.
Then, design with the end in mind. Think backwards from
results to behavior to learning to reaction.
Implement the evaluation process from beginning to end
using all four levels,
Measure relationships between outcomes at all four
levels and consider the effect of intervening variables.
54
eLearning Guild
Page 27
What happens when Level 4 evaluation
becomes the first step in the evaluation
process?
Intel case study
• Use the taxonomy to build a model
• Address issues left unrefined and undeveloped by Kirkpatrick
55
What happens when Level 4 evaluation
becomes the first step in the evaluation
process?
Intel case study
• Two steps: upfront analysis from Level 4 perspective,
and evaluation process beginning at Level 1.
•
Start with Level 4 at the systems level
•
Use Level 3 to identify barriers to transfer
•
Use Level 2 to evaluate learning and “transfer readiness”
•
Redefine Level 1 to include factors that influence outcomes on
other levels
• The most opportune time to impact cost and quality is prior to
program development and release
56
eLearning Guild
Page 28
Questions and Comments
57
Thank You!
Joe Pulichino
Athena Learning
joe@athenalearn.com
eLearning Guild
Page 29