Workplace Based Assessment Rediscovered

Transcription

Workplace Based Assessment Rediscovered
Nancy Dudek MD MEd & Farhan Bhanji MD MSc
October 23, 2015
I do not have an affiliation (financial or otherwise)
with a pharmaceutical, medical device or
communications organization.
Je n’ai aucune affiliation (financière ou autre)
avec une entreprise pharmaceutique, un fabricant
d’appareils médicaux ou un cabinet de communication.
Author: Nancy Dudek & Farhan Bhanji
Date: October 23, 2015
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhanji
Objectives
• Upon completion of this session, participants will be
able to:
– Describe methods to overcome barriers to observing trainee
performance
– Transform verbal feedback into written comments
– Write comments that are specific, behavior-based and
supported by examples
– Use a new tool for narrative assessment
The International Conference on Residency Education
|
La Conférence internationale sur la formation des résidents
3
Writing effective workplace assessment reports: authenticity & impact | Nancy Dudek
Workplace Assessment
• Thought to be the best method of assessing professional competence
–
–
–
–
Ability to represent normal behaviours
Acknowledge competence
Identify trainees in need of remediation
For many aspects of medicine we do not have an alternate means of evaluation
• There are several tools for workplace assessment including:
–
–
–
–
–
Mini-clinical evaluation exercise
Direct observation of practical skill
Multi-source feedback
In-training evaluation
Daily encounter cards
• Tools usually consist of:
– List of items on a checklist or rating scale
– Written comments
|
4
• Formative and as well as summative roles
– Provide feedback (formative)
– Documented evidence that a trainee has met a set standard (summative)
|
5
• Competency Based Medical
Education (CBME) curricula
» Goal is to improve education
and performance of learners
» Requires direct observation
and workplace assessment
methodologies that reflect
trainee performance
accurately
» Quality workplace
assessments are a critical
component of medical trainee
assessment with CBME
|
6
Issues with Workplace Assessment
• Main critique of these methods is a lack of reliability and validity
• Problems have been attributed to a weak instrument & a lack of
knowledge of how to use the instrument
– Note that in determining this, only the ratings portion (quantitative
part) of the assessments are examined
• Past work has focused on:
– Improving the various tools used for workplace assessment
– Training raters to more consistently assign the various quantitative
ratings given to the medical trainees being assessed
• Typically only minimal improvements noted
|
7
Qualitative Assessment
• More recently:
– More emphasis on qualitative assessments
– Some suggesting that narrative descriptions replace numerical
ratings
– Research demonstrates that a large portion of the usefulness
of these methods is in the narrative (qualitative) part
• What is more objective?
– “My pain is 6 out of 10 doc”
– “My pain is constant, burning and interfering with my ability to
wear my prosthesis”
• What is more useful?
|
8
Qualitative Assessment
• Misconception
– Subjective = Unreliable
– Objective = Reliable
• “Faculty need to recognize that numeric ratings are nothing
more than a process to synthesize and then represent a
composite judgment about a trainee.”
Holmboe et al., Academic Medicine, 2011
|
9
Words and not Numbers
• Rich narrative assessments
of performance
– Enhance the formative
function
• Provides the information
required for guided
reflection
– Required for defensible
decisions in summative
assessments
|
10
Words and not Numbers
• Rich narrative assessments of performance
– Can be compiled
• “reliability” obtained with adequate sampling
• Need data!!!!!
– Comments are minimal
– Comments lack quality
|
11
Focus for Today
• Narrative aspect of workplace assessment
– Improving the quality of the comments
|
12
Challenges
• What are the challenges to completing workplace
assessments in the residency training environment?
– Think of two challenges
– Discuss these challenges with the person next to you – Pair
– Share with the group
|
13
Major Challenge
• Observation!!
|
14
Observation Strategies
• Orient the trainee to being observed
• Two approaches
– Watch it all
– Watch bits and pieces
• Some aspect of history
• Repeat physical exam
• Provide the plan
• Introduce concept to patient
– “I’m a fly on the wall”
• Define what you need to watch
• Make a schedule to observe
|
15
What about the things you don’t observe?
• Examples:
» Interactions with other team members
• “I really don’t like being on call with Dr. X”
» Comments from patients and families
• “She is wonderful… keep her!”
|
16
What about the things you don’t observe?
• Thank them for telling you
• Ask them why
» Follow up questions when needed
|
17
Give them Feedback
• Focus on behaviours – not
attitudes
– Talk about perception
• Be specific
• When possible discuss the
outcome
• Note their response to the
feedback
|
18
Behaviours – Not Attitudes
• Example
– “Lazy” resident
|
19
Lazy Resident
Behaviour
Outcome
Consistently late
Staff work late to accommodate
Does not follow up on tests
Missed critical issue
Does not answer pager
Called staff/resident on-call
Does not do assigned readings
Staff dissatisfaction & staff wastes
time on follow-up
|
20
Documentation Strategies
• Narrative Assessment Tool
» Describe one aspect of
performance that was done well
• Was it “competent for
independent practice?
» Describe one aspect of
performance that could improve.
• What are your suggestions
for how the resident could
improve this?
» When was this assessment
reviewed with the resident?
|
21
Improve Quality of Comments
• Transform your verbal feedback into written comments
• Comments should:
– Include the trainee’s response to feedback
– Have specific examples
– Provide recommendations for improving performance
– Be written in a supportive manner
– Be detailed enough for an independent reviewer to understand
the issues
|
22
Practice – Video Clips
|
23
How would you make these comments better?
• Responds well to feedback
• Communication skills need work
• Read more
• Great case presentations
|
24
Potential Options
•
Responds positively to feedback. Example - noted that you missed a quads lag by
not first checking passive ROM of the knee. Reviewed proper technique for quads
testing. On observation at a later point during the clinic you had altered your
physical exam appropriately
•
Tendency to use too much medical jargon when explaining issues to patients.
Example - In the patient with an abnormal lesion on the chest x-ray you said, “It
could be an infiltrate, a granuloma, a malignancy…”
•
Focus anatomy reading on the brachial plexus…. need to be able to draw the
plexus out so that neurological lesions can be mapped on the plexus
•
Case presentations in clinic are succinct and include all relevant info ex) patient
with depression and back pain… you were able to focus on the issues relevant to
the question asked by the referring doctor
|
25
A Final Exercise
• Think of two changes that you want to make to
increase the quantity or quality of your assessment
comments
• Develop a plan to make that happen
• Write it down
• Put January 23, 2016 in your calendar
– How well have you done?
|
26
Workplace based assessment rediscovered | Nancy Dudek & Farhan Bhani
Summary
• Need to watch
• Provide feedback
• Use a tool that provides the opportunity for narrative assessment
• Use your feedback to write your comments
• Comments need to be specific, behaviour-based and include
examples
|
27
Questions
ndudek@toh.on.ca
fbhanji@royalcollege.ca
|
28
References
•
Albanese M. Rating educational quality: factors in the erosion of professional standards. Acad
Med. 1999 Jun;74(6):652-8.
•
Cohen GS, Blumberg P, Ryan NC, Sullivan PL. Do final grades reflect written qualitative
evaluations of student performance? Teach Learn Med. 1993;5(1):10-5.
•
Driessen E et al. The use of qualitative research criteria for portfolio assessment as an
alternative to reliability evaluation: a case study. Medical Education 2005;39:214-20.
•
Dudek NL, Marks MB & Regehr G. Failure to fail – the perspectives of clinical supervisors.
Academic Medicine 2005;80(10):S84-7.
•
Dudek N, Marks M, Lee C & Wood T. Assessing the quality of supervisors’ completed clinical
evaluation reports. Medical Education 2008;42:816-22.
•
Dudek N, Marks M, Dojeiji S. Completing a quality evaluation report – what clinical supervisors
need to know - a faculty development workshop. MedEdPORTAL; 2013. Available from:
www.mededportal.org/publication/9320
•
Dudek NL, Marks MB, Wood TJ, Dojeiji S, Bandiera G, Hatala R, Cooke L, Sadownik L. Quality
evaluation reports – can a faculty development program make a difference?. Medical Teacher
2012:34(11):e725-e731.
•
Dudek NL, Marks MB & Regehr G. Failure to fail – the perspectives of clinical supervisors.
Academic Medicine 2005;80(10):S84-7.
|
29
References
•
Dudek NL, Marks MB & Regehr G. Reluctance to fail poorly performing residents – explanations
and potential solutions. ACGME Bulletin April 2006;45-8.
•
Dudek NL, Dojeiji S. Twelve tips for completing quality in-training evaluation reports. Medical
Teacher – Accepted for publication May 2014.
•
Gray JD. Global rating scales in residency education. Acad Med. 1996 Jan;71(1 Suppl):S55-61.
•
Hatala R, Norman GR. In-training evaluation during an internal medicine clerkship. Acad Med.
1999 Oct;74(10 Suppl):S118-20.
•
Hauer KE, Homlboe ES, Kogan JR. Twelve tips for implementing tools for direct observation of
medical trainees’ clinical skills during patient encounters. Med Teach. 2001;33:27-33.
•
Holmboe ES, et al. Faculty development in assessment: the missing link in competency-based
medical education. Academic Medicine 2011;86(4):460-7.
•
Kassebaum DG, Eaglen RH. Shortcomings in the evaluation of students’ clinical skills and
behaviors in medical school. Acad Med. 1999 Jul;74(7):842-9.
•
Speer AJ, Solomon DJ, Ainsworth MA. An innovative evaluation method in an internal medicine
clerkship. Acad Med. 1996 Jan;71(1 Suppl):S76-8.
•
Turnbull J, Gray J, MacFadyen J. Improving in-training evaluation programs. JGIM.
1998;13:317-323.
|
30
Help us improve.
Aidons-nous à nous améliorer.
Your input matters.
Votre opinion compte.
• Download the ICRE App,
• Téléchargez l’application de la CIFR
• Visit the evaluation area
in the West Ballroom Foyer,
near Registration, or
• Visitez la zone d’évaluation, foyer
de la salle de bal Ouest, près du
comptoir d’inscription, ou
• Go to: http://www.royalcollege.ca/
icreevaluations to complete the
session evaluation.
• Visitez le http://www.collegeroyal
.ca/evaluationscifr afin de remplir
une évaluation de la séance.
You could be entered
to win 1 of 3
$100 gift cards.
Vous courrez la chance de gagner
l’un des trois chèques-cadeaux
d’une valeur de 100 $.
Reliability as a function of testing time
Testing
Time in
Hours MCQ1
CasePractice
Based
Video
InMini Assess- cognito
Oral Long
Short
Essay2 PMP1 Exam3 Case4 OSCE5 CEX6 ment7
SPs8
1
0.62
0.68
0.36
0.50
0.60
0.54
0.73
0.62
0.61
2
0.76
0.73
0.53
0.69
0.75
0.69
0.84
0.76
0.76
4
0.93
0.84
0.69
0.82
0.86
0.82
0.92
0.93
0.82
8
0.93
0.82
0.82
0.90
0.90
0.90
0.96
0.93
0.86
1Norcini
et al., 1985
2Stalenhoef-Halling et al., 1990
3Swanson, 1987
32
4Wass
et al., 2001
der Vleuten, 1988
6Norcini et al., 1999
5Van
7Ram
et al., 1999
2002
8Gorter,

Workplace Based Assessment Rediscovered

Transcription

Similar documents

Physical examination teaching for the 21st century:

your card! - Benefits Corporation Helps Chambers Grow Their

customer story

05_25_2016_Workplace_webinar_final

welcome to the active generation

Experience Guide

Pfeiffer Big Sur State Park - Volunteers for Outdoor California

The Chapel Hill Connection

l`actuel sur rue

Our Team - MUSC Health

Workplace Fun Ideas