Workplace Based Assessment Rediscovered

Transcription

Workplace Based Assessment Rediscovered
Workplace Based Assessment Rediscovered
Nancy Dudek MD MEd & Farhan Bhanji MD MSc
October 23, 2015
I do not have an affiliation (financial or otherwise)
with a pharmaceutical, medical device or
communications organization.
Je n’ai aucune affiliation (financière ou autre)
avec une entreprise pharmaceutique, un fabricant
d’appareils médicaux ou un cabinet de communication.
Author: Nancy Dudek & Farhan Bhanji
Date: October 23, 2015
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Objectives
• Upon completion of this session, participants will be
able to:
– Describe methods to overcome barriers to observing trainee
performance
– Transform verbal feedback into written comments
– Write comments that are specific, behavior-based and
supported by examples
– Use a new tool for narrative assessment
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
3
Writing effective workplace assessment reports: authenticity & impact | Nancy Dudek
Workplace Assessment
• Thought to be the best method of assessing professional competence
–
–
–
–
Ability to represent normal behaviours
Acknowledge competence
Identify trainees in need of remediation
For many aspects of medicine we do not have an alternate means of evaluation
• There are several tools for workplace assessment including:
–
–
–
–
–
Mini-clinical evaluation exercise
Direct observation of practical skill
Multi-source feedback
In-training evaluation
Daily encounter cards
• Tools usually consist of:
– List of items on a checklist or rating scale
– Written comments
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
4
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Workplace Assessment
• Formative and as well as summative roles
– Provide feedback (formative)
– Documented evidence that a trainee has met a set standard (summative)
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
5
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Workplace Assessment
• Competency Based Medical
Education (CBME) curricula
» Goal is to improve education
and performance of learners
» Requires direct observation
and workplace assessment
methodologies that reflect
trainee performance
accurately
» Quality workplace
assessments are a critical
component of medical trainee
assessment with CBME
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
6
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Issues with Workplace Assessment
• Main critique of these methods is a lack of reliability and validity
• Problems have been attributed to a weak instrument & a lack of
knowledge of how to use the instrument
– Note that in determining this, only the ratings portion (quantitative
part) of the assessments are examined
• Past work has focused on:
– Improving the various tools used for workplace assessment
– Training raters to more consistently assign the various quantitative
ratings given to the medical trainees being assessed
• Typically only minimal improvements noted
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
7
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Qualitative Assessment
• More recently:
– More emphasis on qualitative assessments
– Some suggesting that narrative descriptions replace numerical
ratings
– Research demonstrates that a large portion of the usefulness
of these methods is in the narrative (qualitative) part
• What is more objective?
– “My pain is 6 out of 10 doc”
– “My pain is constant, burning and interfering with my ability to
wear my prosthesis”
• What is more useful?
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
8
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Qualitative Assessment
• Misconception
– Subjective = Unreliable
– Objective = Reliable
• “Faculty need to recognize that numeric ratings are nothing
more than a process to synthesize and then represent a
composite judgment about a trainee.”
Holmboe et al., Academic Medicine, 2011
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
9
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Words and not Numbers
• Rich narrative assessments
of performance
– Enhance the formative
function
• Provides the information
required for guided
reflection
– Required for defensible
decisions in summative
assessments
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
10
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Words and not Numbers
• Rich narrative assessments of performance
– Can be compiled
• “reliability” obtained with adequate sampling
• Need data!!!!!
– Comments are minimal
– Comments lack quality
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
11
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Focus for Today
• Narrative aspect of workplace assessment
– Improving the quality of the comments
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
12
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Challenges
• What are the challenges to completing workplace
assessments in the residency training environment?
– Think of two challenges
– Discuss these challenges with the person next to you – Pair
– Share with the group
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
13
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Major Challenge
• Observation!!
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
14
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Observation Strategies
• Orient the trainee to being observed
• Two approaches
– Watch it all
– Watch bits and pieces
• Some aspect of history
• Repeat physical exam
• Provide the plan
• Introduce concept to patient
– “I’m a fly on the wall”
• Define what you need to watch
• Make a schedule to observe
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
15
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
What about the things you don’t observe?
• Examples:
» Interactions with other team members
• “I really don’t like being on call with Dr. X”
» Comments from patients and families
• “She is wonderful… keep her!”
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
16
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
What about the things you don’t observe?
• Thank them for telling you
• Ask them why
» Follow up questions when needed
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
17
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Give them Feedback
• Focus on behaviours – not
attitudes
– Talk about perception
• Be specific
• When possible discuss the
outcome
• Note their response to the
feedback
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
18
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Behaviours – Not Attitudes
• Example
– “Lazy” resident
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
19
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Lazy Resident
Behaviour
Outcome
Consistently late
Staff work late to accommodate
Does not follow up on tests
Missed critical issue
Does not answer pager
Called staff/resident on-call
Does not do assigned readings
Staff dissatisfaction & staff wastes
time on follow-up
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
20
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Documentation Strategies
• Narrative Assessment Tool
» Describe one aspect of
performance that was done well
• Was it “competent for
independent practice?
» Describe one aspect of
performance that could improve.
• What are your suggestions
for how the resident could
improve this?
» When was this assessment
reviewed with the resident?
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
21
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Improve Quality of Comments
• Transform your verbal feedback into written comments
• Comments should:
– Include the trainee’s response to feedback
– Have specific examples
– Provide recommendations for improving performance
– Be written in a supportive manner
– Be detailed enough for an independent reviewer to understand
the issues
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
22
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Practice – Video Clips
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
23
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
How would you make these comments better?
• Responds well to feedback
• Communication skills need work
• Read more
• Great case presentations
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
24
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Potential Options
•
Responds positively to feedback. Example - noted that you missed a quads lag by
not first checking passive ROM of the knee. Reviewed proper technique for quads
testing. On observation at a later point during the clinic you had altered your
physical exam appropriately
•
Tendency to use too much medical jargon when explaining issues to patients.
Example - In the patient with an abnormal lesion on the chest x-ray you said, “It
could be an infiltrate, a granuloma, a malignancy…”
•
Focus anatomy reading on the brachial plexus…. need to be able to draw the
plexus out so that neurological lesions can be mapped on the plexus
•
Case presentations in clinic are succinct and include all relevant info ex) patient
with depression and back pain… you were able to focus on the issues relevant to
the question asked by the referring doctor
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
25
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
A Final Exercise
• Think of two changes that you want to make to
increase the quantity or quality of your assessment
comments
• Develop a plan to make that happen
• Write it down
• Put January 23, 2016 in your calendar
– How well have you done?
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
26
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhani
Summary
• Need to watch
• Provide feedback
• Use a tool that provides the opportunity for narrative assessment
• Use your feedback to write your comments
• Comments need to be specific, behaviour-based and include
examples
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
27
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Questions
ndudek@toh.on.ca
The International Conference on Residency Education
fbhanji@royalcollege.ca
|
La Conférence internationale sur la formation des résidents
28
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
References
•
Albanese M. Rating educational quality: factors in the erosion of professional standards. Acad
Med. 1999 Jun;74(6):652-8.
•
Cohen GS, Blumberg P, Ryan NC, Sullivan PL. Do final grades reflect written qualitative
evaluations of student performance? Teach Learn Med. 1993;5(1):10-5.
•
Driessen E et al. The use of qualitative research criteria for portfolio assessment as an
alternative to reliability evaluation: a case study. Medical Education 2005;39:214-20.
•
Dudek NL, Marks MB & Regehr G. Failure to fail – the perspectives of clinical supervisors.
Academic Medicine 2005;80(10):S84-7.
•
Dudek N, Marks M, Lee C & Wood T. Assessing the quality of supervisors’ completed clinical
evaluation reports. Medical Education 2008;42:816-22.
•
Dudek N, Marks M, Dojeiji S. Completing a quality evaluation report – what clinical supervisors
need to know - a faculty development workshop. MedEdPORTAL; 2013. Available from:
www.mededportal.org/publication/9320
•
Dudek NL, Marks MB, Wood TJ, Dojeiji S, Bandiera G, Hatala R, Cooke L, Sadownik L. Quality
evaluation reports – can a faculty development program make a difference?. Medical Teacher
2012:34(11):e725-e731.
•
Dudek NL, Marks MB & Regehr G. Failure to fail – the perspectives of clinical supervisors.
Academic Medicine 2005;80(10):S84-7.
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
29
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
References
•
Dudek NL, Marks MB & Regehr G. Reluctance to fail poorly performing residents – explanations
and potential solutions. ACGME Bulletin April 2006;45-8.
•
Dudek NL, Dojeiji S. Twelve tips for completing quality in-training evaluation reports. Medical
Teacher – Accepted for publication May 2014.
•
Gray JD. Global rating scales in residency education. Acad Med. 1996 Jan;71(1 Suppl):S55-61.
•
Hatala R, Norman GR. In-training evaluation during an internal medicine clerkship. Acad Med.
1999 Oct;74(10 Suppl):S118-20.
•
Hauer KE, Homlboe ES, Kogan JR. Twelve tips for implementing tools for direct observation of
medical trainees’ clinical skills during patient encounters. Med Teach. 2001;33:27-33.
•
Holmboe ES, et al. Faculty development in assessment: the missing link in competency-based
medical education. Academic Medicine 2011;86(4):460-7.
•
Kassebaum DG, Eaglen RH. Shortcomings in the evaluation of students’ clinical skills and
behaviors in medical school. Acad Med. 1999 Jul;74(7):842-9.
•
Speer AJ, Solomon DJ, Ainsworth MA. An innovative evaluation method in an internal medicine
clerkship. Acad Med. 1996 Jan;71(1 Suppl):S76-8.
•
Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. JGIM.
1998;13:317-323.
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
30
Help us improve.
Aidons-nous à nous améliorer.
Your input matters.
Votre opinion compte.
• Download the ICRE App,
• Téléchargez l’application de la CIFR
• Visit the evaluation area
in the West Ballroom Foyer,
near Registration, or
• Visitez la zone d’évaluation, foyer
de la salle de bal Ouest, près du
comptoir d’inscription, ou
• Go to: http://www.royalcollege.ca/
icreevaluations to complete the
session evaluation.
• Visitez le http://www.collegeroyal
.ca/evaluationscifr afin de remplir
une évaluation de la séance.
You could be entered
to win 1 of 3
$100 gift cards.
Vous courrez la chance de gagner
l’un des trois chèques-cadeaux
d’une valeur de 100 $.
Reliability as a function of testing time
Testing
Time in
Hours MCQ1
CasePractice
Based
Video
InMini Assess- cognito
Oral Long
Short
Essay2 PMP1 Exam3 Case4 OSCE5 CEX6 ment7
SPs8
1
0.62
0.68
0.36
0.50
0.60
0.54
0.73
0.62
0.61
2
0.76
0.73
0.53
0.69
0.75
0.69
0.84
0.76
0.76
4
0.93
0.84
0.69
0.82
0.86
0.82
0.92
0.93
0.82
8
0.93
0.82
0.82
0.90
0.90
0.90
0.96
0.93
0.86
1Norcini
et al., 1985
2Stalenhoef-Halling et al., 1990
3Swanson, 1987
32
4Wass
et al., 2001
der Vleuten, 1988
6Norcini et al., 1999
5Van
7Ram
et al., 1999
2002
8Gorter,