Nebraska ASTD 4_9_14 - ATD

The “How-To” Measurement & Evaluation Series
Increased
productivity
through
skill building
Power Up Your Level
1 Evaluations and
Gain Surprisingly
Useful, Valued Data
Hawkeye Chapter ATD
September 16, 2015
Today’s Presentation
According to a 2009 ASTD research
study, 92% of organizations evaluate
at least some learning programs at
Level 1 of the Kirkpatrick/Phillips
five level evaluation model. Yet only
36% view the data they collect to be
valuable. If designed well and used
properly, Level 1 evaluations
produce valuable data regarding
whether or not a program is “on
track” to achieve key business
objectives.
In this session, you will:
• Analyze the common mistakes
made by many learning and
development professionals when
creating Level 1 evaluations.
• Learn 12 tips for overcoming these
mistakes to create Level 1 evaluations
that produce powerful, valued data.
• Discover how to create Level 1
evaluations that forecast participant
learning, intent to apply what was
learned back on the job and the
likely impact on business results.
About Your Presenter —
“I thought I knew everything I needed
about level 1 evaluations and then I
attended Ken Phillips’ workshop
sponsored by SEWI-ATD. I was
amazed at how much I learned in Ken’s
fast-paced participatory presentation.
Thanks to Ken, we can now create level
1s that provide us with usable data. The
value from this presentation more than
paid for my annual chapter
membership.”
–Mary Kay Wagner
KEN PHILLIPS, CPLP
Ken Phillips delivers all programs and workshops in his
signature style: professional, engaging, and approachable.
Ken is founder and CEO of Phillips Associates, a
consulting and publishing company with expertise in
performance management, measurement and evaluation of
learning and sales performance. He has more than 25 years
experience designing learning instruments and assessments
and has authored more than a dozen published learning
instruments. He regularly speaks to Association for Talent
Development (ATD) groups and university classes. Since 2008, he has spoken at
the ATD International Conference on topics related to measurement and
evaluation of learning.
Prior to pursuing a Ph.D. in the combined fields of organizational behavior and
educational administration at Northwestern University, Ken held management
positions with two colleges and two national corporations. In addition, he has
written articles that have appeared in T+D magazine, Training Today and
HR.com.
Currently Ken is serving his tenth year on the board of directors of the
Chicagoland Chapter of the Association for Talent Development (CCATD),
and was chapter president in 2008. He earned the Certified Professional in
Learning and Performance (CPLP) credential from national ATD in 2006 as a
pilot pioneer and was recertified in 2009 and again in 2012. In January 2009, he
also was appointed to serve a three-year term (2009-2011) as a member of the
ATD National Advisors for Chapters (NAC) committee.
Ken can be reached at (847) 231-6068 or ken@phillipsassociates.com.
Ken is a frequent presenter at ATD
chapter meetings and other industry
events. Here’s a preview of where
he’ll be speaking next.
23-24
Phillips Associates Open
Enrollment Workshop
Workshop: Mastering M & E:
Less Stress, More Success
Designing Level 1 – 4
Evaluations
8
Northwest Suburban SHRM
Chapter
Topic: State of the Art
Survey Design Techniques
24 Phillips Associates &
Chicagoland Chapter ATD
Workshop: 47 Tips and Best
Kept Secrets for Designing
Level 1 – 4 Evaluations
Copyright ©2015 by Phillips Associates
All rights reserved. Printed in the United States of America. Except as permitted under the United States Copyright Act of 1976, no part
of the publication may be reproduced or distributed in an form or by any means, or stored in a database or retrieval system, without the
prior written permission from Phillips Associates
Level 1 Reaction
Measuring and Evaluating Learning
SAMPLE LEVEL 1 EVALUATION
We are constantly striving to improve the effectiveness of our learning programs. Please take
a moment and give us your thoughtful feedback. Thank you!
Name: _____________________________________________________
Years with company: _________________________________________
Department: ________________________________________________
1. What were your expectations for the session?
1.
2.
3.
2. How successful were you in satisfying your expectations?
Very Successful
Generally
Successful
Limited Success
Not Successful
Expectation 1
Expectation 2
Expectation 3
3. What are your overall reactions to the program? Please indicate the extent to which you
agree or disagree with the following statements.
SA = Strongly Agree A = Agree D = Disagree SD = Strongly Disagree SA
A
D SD
a. Overall, I was satisfied with the program
b. I will use the content in my work or life
c. The workshop materials were well designed
d. The learning objectives were met
PHILLIPS ASSOCIATES
1
www.phillipsassociates.com
Level 1 Reaction
Measuring and Evaluating Learning
SAMPLE LEVEL 1 EVALUATION
4. What are your overall reactions to the facilitator? Please indicate the extent to which you
agree or disagree with the following statements.
SA = Strongly Agree A = Agree D = Disagree SD = Strongly Disagree SA
a. Overall, I was satisfied with the facilitator
A
D SD
b. Did the facilitator clearly describe the learning objectives?
c. Did the facilitator keep the session lively and interesting?
d. Did the facilitator communicate the subject matter well?
e. Did the facilitator maintain a friendly and supportive atmosphere?
5. What are your overall reactions to the facility? Please indicate the extent to which you
agree or disagree with the following statements.
SA = Strongly Agree A = Agree D = Disagree SD = Strongly Disagree SA
a. Overall, I was satisfied with the facility
A
D SD
b. The lighting was good
c. The room set-up and size were comfortable
d. The chairs were comfortable
e. The food provided was satisfactory
6. What parts of the session were most helpful to you personally?
7. What parts of the session did you find least helpful/meaningful?
8. What feedback do you have for the facilitator?
9. Would you recommend this program to other groups? Why or why not?
10. Additional comments:
PHILLIPS ASSOCIATES
2
www.phillipsassociates.com
Trainer-Centered vs. Learner-Centered Level 1 Evaluation Items *
Evaluation
Category
Program objectives
Trainer-centered
The program objectives were clearly
defined.
The program objectives were
covered by the instructor.
Course materials
The course materials were the
right level of complexity for my
background.
The course materials were well
organized.
Learner-centered
I understood the learning objectives.
I was able to relate each of the
program objectives to the learning
I received.
I was appropriately challenged by the
course material.
I found the course materials easy to
navigate.
The course materials aided my
The course materials complemented understanding of the program content.
the course content.
Content relevance
Overall, the program met my needs
and expectations.
The program was relevant to both me
and my job.
Facilitator
knowledge
The facilitator possessed a solid
understanding of the program
material.
My learning was enhanced by
the knowledge of the facilitator.
The facilitator shared personal
experiences related to the program
content.
Facilitator delivery
My learning was enhanced by
the personal experiences shared by
the facilitator.
The facilitator effectively delivered
the program material.
I felt engaged throughout the entire
program.
The facilitator did a good job of generating participant interaction.
It was easy for me to get and stay
actively involved during the session.
The facilitator used a variety of
instructional methods.
My learning was enhanced by the
variety of learning methods and
activities used during the program.
The pace of the program was good.
The duration of the session was
good.
The pace of the program aided my
understanding of the program content.
I was comfortable with the duration
of the session.
P H I L L I P S A S S O C I AT E S
3
www.phillipsassociates.com
Evaluation
Category
Program evaluation
Trainer-centered
Learner-centered
The knowledge test was a fair
assessment of the program content.
The knowledge test helped me to
focus my learning.
The role-plays and simulations were
realistic representations of actual
work situations.
I was given ample opportunity to
demonstrate my skills.
Breaks
The breaks were spaced at the right
times during the session.
I felt refreshed and re-energized after
each of the breaks.
Facility
The lighting was adequate.
I felt that the room atmosphere
enhanced my learning.
The temperature was comfortable.
I was pleased with the room set-up.
The coffee was hot.
I experienced minimal distractions
during the session.
* Adapted from Jim Kirkpatrick, “The New World Level 1 Reaction Sheets,” unpublished article.
P H I L L I P S A S S O C I AT E S
4
www.phillipsassociates.com
Identifying Scale Point Values
Specify a number between 0 and 100 that best indicates the value
(percentage) each word or phrase conveys.
Word or Phrase
Amount
(Percentage)
Always
___________
Never
___________
Neither agree nor disagree
___________
Somewhat ineffective
___________
Agree
___________
Very effective
___________
Strongly disagree
___________
Sometimes
___________
Very ineffective
___________
Almost always
___________
Somewhat effective
___________
Disagree
___________
Almost never
___________
Strongly agree
___________
Neither effective nor ineffective
___________
5
www.phillipsassociates.com
Level 1 Evaluation Job Aid
! Level 1 evaluation facts *
•
92% of organizations evaluate at least some learning programs at Level 1
•
Organizations that use Level 1s on average evaluate 78% of all programs
•
36% of organizations view the data collected as having high or very high
value
! Twelve tips for creating Level 1 evaluations that produce data with high
perceived value
•
Tips on Content
1. Only ask questions that lead to actionable data and focus on important
issues
2. Write learner-centered not trainer-centered evaluation items **
Example:
This: I found the room comfortable and conducive to learning.
Not this: The room lighting and temperature were conducive to learning.
3. Where appropriate, match up qualitative questions with quantitative
measures
Example:
In a word, how would you describe this session? ___________________
Using a number, how would you describe this session?
No Value
Great Value
1
2
3
4
5
6
7
4. Include at least one item asking participants how relevant the learning
event/material was to them and their job
Example:
How would you rate the overall relevance of this session to you and your job?
Not at all Relevant
Very Relevant
1
2
3
4
5
6
7
5. Include one item asking participants how likely they are to recommend the
program to friends and colleagues and use the net promoter score
methodology (percent of promoters – percent of detractors) to analyze
results ***
Example:
How likely are you to recommend this program to friends and colleagues?
Not at all Likely
Extremely Likely
1
2
3
4
5
6
7
6
www.phillipsassociates.com
6. Create predictive questions that forecast participant learning, intent to
apply what was learned back on-the-job and likely impact on business
result
Level 2 Learning Example:
How much did you know about developing individual performance objectives
before attending this session?
No Knowledge
Thorough Knowledge
1
2
3
4
5
6
7
How much do you know about developing individual performance objectives
after attending this session?
No Knowledge
Thorough Knowledge
1
2
3
4
5
6
7
Level 3 Behavior Example:
How likely are you to apply the skills and behaviors taught in this seminar backon-the-job?
Not at all Likely
Extremely Likely
1
2
3
4
5
6
7
What obstacles, if any, might keep you from applying what was learned?
_____________________________________________________
Level 4 Results Example:
How likely are any of the key business metrics tracked by your department to
improve as a result of you applying the knowledge and skills you learned in this
program?
Not at all Likely
Extremely Likely
1
2
3
4
5
6
7
How confident are you in your response to the previous question where 0 = no
confidence and 100 = extremely confident? ___________________________
•
Tips on Measurement
7. When collecting quantitative data using a Likert scale, create a response
scale with numbers at regularly spaced intervals and words only at each
end ****
Example:
Not at all
Completely
True
True
1
2
3
4
5
6
7
7
www.phillipsassociates.com
8. Use a response scale with an odd number of points (7, 9, & 11 point scales
are best) ****
9. Place small numbers at the left or low end of the scale and large numbers
at the right or high end of the scale
Example:
This:
Not at all True
Completely True
1
2
3
4
5
6
7
Not this: Completely True
7
6
5
4
3
2
Not at all True
1
10. Write items either as a continuum or as a statement
Example:
This: How effectively did the AV materials used during the session help reinforce
your understanding of the program material?
Not Effectively
Very Effectively
1
2
3
4
5
6
7
Or this: The AV materials used during the session helped reinforce my
understanding of the program material.
Not at all True
Completely True
1
2
3
4
5
6
7
Not this: Did the AV materials used during the session help reinforce your
understanding of the program material?
Strongly Disagree
Strongly Agree
1
2
3
4
5
6
7
8
www.phillipsassociates.com
•
Tip on Administration
11. Use Level 1 evaluations only to improve a learning program not to prove
something.
•
Tip on Format
12. Place questions regarding respondent demographics (e.g. name, title,
department, etc.) at end of evaluation form, make completion optional and
keep questions to a minimum.
* ASTD Research study: “The Value of Evaluation: Making Training Evaluations More Effective,” 2009.
** Jim Kirkpatrick, unpublished article, “The New World Level 1 Reaction Sheets,” 2008.
*** The Net Promoter Score Methodology was developed by Fred Reichheld and discussed in a Harvard
Business Review article titled: “The One Number You Need to Grow.”
**** Adapted from “Getting the Truth into Workplace Surveys, Palmer Morrel-Samuels, Harvard Business
Review, February, 2002, pp. 111-118.
9
www.phillipsassociates.com
This page intentionally left blank
Training Resources Available from Phillips Associates
Insightful Learning Products
Turnkey training resources for organizations and individuals
Phillips Associates offers an array of learning instruments, multi-rater assessment systems, handbooks (job aids),
manuals and games designed to develop specific skills in the areas of performance management and sales
performance. Some of our best-sellers include:
No Pain—All Gain! Performance
Coaching to Win: Coaching Meeting Inventory
Appraisal Meeting Inventory
Help managers conduct performance
appraisal meetings that improve
employee performance and are
viewed as valuable and worthwhile
using the Performance Appraisal
Meeting Inventory. The instrument
pinpoints a manager’s strengths and development needs
in six skill areas and outlines a simple six-step process
for conducting effective performance appraisal meetings.
The Coaching Meeting Inventory assesses a manager’s
strengths and development needs in eight skill areas and
outlines a logical step-by-step process for conducting
effective coaching discussions.
Getting On the Same Page: Objective Setting
Handbooks
Easy to read booklets (one for managers and one for
direct reports) filled with proven tips and techniques on
transforming the objective setting process from little
more than an annual ritual into a valuable, worthwhile
experience for everyone involved. Includes a set of
worksheets the reader can use to prepare a set of
objectives.
Acting the Part: Performance Management Role Plays
Ready-to-use role play scenarios designed to help
managers, supervisors and team leaders apply and
practice the skills needed for coaching effectively and
conducting productive performance appraisals.
Includes half-day workshop design and PowerPoint slides.
Superior Service Starts Here: Customer Service Skills
Inventory (CSSI)
CSSI consists of 32 statements divided into four keys of
providing superior customer service: Developing a
relationship of Interpersonal Trust, Listening, Problem
Solving and Professionalism. Based on how they answer
the statements, customer service reps gain valuable
insight into their greatest strengths and most pressing
development needs in delivering superior customer
service.
Taking Stock for Better Communication:
Communication Style Inventory (CSI)
CSI consists of 24 sets of paired statements describing
the four most commonly used communication styles:
Aggressive, Avoiding, Accommodating and Collaborative.
By choosing the statement that most nearly describes
how they would communicate in a particular situation,
respondents generate a profile of their use of the four
communication styles.
For descriptions of all Phillips Associates Learning Products, visit www.phillipsassociates.com
PERFORMANCE MANAGEMENT
PRODUCTS
• Performance Appraisal Meeting
Inventory
SALES PERFORMANCE PRODUCTS
MULTI-RATER ASSESSMENT PRODUCTS
• Face-to-Face Sales Skills Inventory
• Performance Management Profile
System
• Managing Your Own Performance
Appraisal Inventory
• Territory Management Style
Inventory
• Coaching Meeting Inventory
• Customer Service Skills Inventory
• Cross-Selling Skills Inventory
• Selling Skills Profile System
FOUNDATION SKILLS PRODUCTS
• Communication Style Inventory
• Keys to Effective Coaching
• Objective Setting Inventory
• Interpersonal Feedback Inventory
• Objective Setting Games
• Performance Management Role
Plays
Browse and buy online
from the PHILLIPS ASSOCIATES
Training Resources Online Catalog
www.phillipsassociates.com
• Objective Setting Handbooks
Save 10% on your entire order
• Performance Appraisal Handbooks
when you mention this code: HCATD0915
(offer expires 10/16/15)
Performance Management and Sales Performance Programs
In addition to our full offering of learning products,
Phillips Associates delivers interactive programs to help
employees achieve and maintain better on-the-job
performance. Our highly participative approach teaches
more than just concepts and practices—it builds lasting
skills. Some of our most popular programs include:
• Exceeding Expectations: Managing the Performance
Appraisal Process
• From Basics to Best in Class: Customer Service Skills
• Coaching to Win: Coaching Employee Performance
For a complete list of programs offered, please visit us at
www.phillipsassociates.com/seminars.
The “How-To” Measurement & Evaluation Series
Only Phillips Associates shows you how!
Presentations (75-90 minute learning programs with audience participation)
• Power Up Your Level 1 Evaluations and Gain Surprisingly Useful, Valued
Data
• Take Your Level 2s Up a Notch: Start Measuring Application, Not Just
Recall
• Capture Elusive Level 3 Data: The Secrets of Survey Design
• Business Results Made Visible: Design Proof Positive Level 4
Evaluations
• Now Put Your Data to Work and Use it to Make a Difference
2-day open enrollment workshop offered at various times throughout the year
• Mastering M & E Less Stress, More Success in Designing Level 1 – 4
Evaluations
For more information contact Ken Phillips at 847.231.6068 or ken@phillipsassociates.com
34137 N. Wooded Glen Drive | Grayslake, IL 60030
847.231.6068 | Info@phillipsassociates.com
www.phillipsassociates.com
I N T E R N AT I O N A L D I S T R I B U T O R S
CANADA
Training & Development Materials of
Canada
372 Moonstone road east, PO Box 38
Moonstone, ontario L0K 1N0
800.283.4964
UNITED KINGDOM
Management Learning Resources, Ltd.
PO Box 28, Carmarthen
Dyfed SA31 1DT
Wales, United Kingdom
01267-281661