Measurement What is the message?

Measurement
What is the message?
How Do We Know if a
Change is an Improvement?
“You can’t fatten a cow by weighing it”
- Palestinian Proverb
“If you can’t
• Improvement is
measure
NOT about it,
measurement
you can’t IMPROVE it”
• However…
Differences between research and improvement
Measurement for Research
Measurement for Learning and
Process Improvement
Purpose
To discover new knowledge
Tests
One large blind test
To bring new knowledge into daily
practice
Many sequential, observable tests
Biases
Control for as many biases as
Stabilize the biases from test to test
possible
Gather as much data as possible, just Gather just enough data to learn and
in case
complete another cycle
Data
Duration Can take long periods of time to
obtain results
Institute for Healthcare Improvement
© AQuA Academy
Small tests of significant changes
accelerate the rate of improvement
Types of Measures
•
Outcome Measures
– Where are we ultimately trying to go?
– Point to qualities that are valuable to stakeholders i.e. how is the
system performing? What is the result?
•
Process Measures
– Are we doing the right things to get there?
– Track that steps in the system are performing as planned.
– Help identify if changes are leading to improvement.
•
Balancing Measures
– ……………………………………………
© AQuA Academy
Balancing Unintended Consequences
“First, do no harm!”
Balancing Measures help us ‘keep an eye on’ other aspects
of the system as we focus on improving one part.
© AQuA Academy
How am I going to present my data?
• What will you be reporting?
• Who is your target audience?
• How often will you need to
communicate data?
• What is the purpose of your analysis?
© AQuA Academy
6
Number of Complaints Per Year
400
350
300
250
200
150
100
50
0
2010
2011
What conclusions can we draw from this?
Snapshot in time
70
60
50
40
30
20
10
0
Jan-10
May 2010
Jan-11
May 2011
Displaying Data
Run Charts
• Plot data in a timely order
• Can analyse by studying where values lie and
why they are distributed around the median
• Helps differentiate between Random and
Special Cause Variation
• Illustrates the impact of an improvement
intervention
© AQuA Academy
10
Number of Complaints per Quarter
140
No of complaints
120
100
80
60
40
20
0
Q1
Q2
Q3
2009
2010
Q4
Q1
Q2
Q3
2010
2011
Q4
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
70
Number of Complaints per Month
60
50
40
30
20
10
0
2010
2009
2011
2010
Audit: Year on year comparison
70
60
2010
2011
50
40
30
20
10
0
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Knowing What to Look For: Rules
• Runs: One or more data points which lie on
the same side of the line. Each time median is
crossed, a new line is identified.
• Trends: 5 or more points all going up or all
going down.
• Astronomical Data Points: Outlier that is
significantly higher or lower than median.
• Shift: 6 consecutive data points either above
or below the line.
Number of Complaints per Month
70
60
50
Astronomical
Data Points
Runs
40
Median
30
20
10
Shift
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
0
2009
2010
2010
2011
Does one way of presenting data
work for everyone?
Tips for Effective Measurement
• Plot data over time
• Seek usefulness, not perfection
• Use sampling
• Integrate measurement into the daily routine
(and is someone already collecting the data?)
• Use qualitative and quantitative data
• Accept Variation
© AQuA Academy
Benchmarking
•
•
•
•
•
•
•
•
Continuous search for better practices
Understanding practices behind performance gaps
Looking outside the organisation
Provides realistic & achievable targets
Challenges complacency
Supports continuous improvement
Visualise the improvement
Identify weak areas & what needs to be done to
improve
Any Questions?