Clinical decision rules: how to use them B Phillips Best practice

Downloaded from ep.bmj.com on August 1, 2012 - Published by group.bmj.com
Best practice
Clinical decision rules: how to use
them
B Phillips
Correspondence to
Dr Bob Phillips, Centre for
Reviews and Dissemination,
and Hull-York Medical School,
University of York YO10 5DD,
UK;
bob.phillips@doctors.org.uk
Accepted 16 February 2010
ABSTRACT
The first of this pair of papers outlined what a clinical
decision rule is and how one should be created. This
section examines how to use a rule, by checking
that it is likely to work (examining how it has been
validated), understanding what the various numbers
that tell us about “accuracy” mean, and considers
some practical aspects of how clinicians think and
work in order to make that information usable on the
front lines.
The fi rst key steps in managing a patient’s problem consist of accurately assessing the situation,
making a clear diagnosis, understanding the psychological and social impact of the illness and
negotiating an effective treatment plan. The fi rst
of these, the arrival at an accurate diagnosis, if
we are to undertake this using high-quality evidence, can be more difficult than we would like
to admit.
The fi rst of this pair of papers [whatever ref]
outlined what a clinical decision rule (CDR) is
and how one should be created. This section
examines how to use a rule, by checking that it
is likely to work (examining how it has been validated), understanding what the various numbers
that tell us about “accuracy” mean, and considers
some practical aspects of how clinicians think
and work in order to make that information
usable on the front lines.
VALIDATING A CDR
In order to even think about using a CDR, we
should be fairly sure that it will work as we want
it to. We would not use a drug that had just been
developed on cell lines in our little patients, and
so we should not do the equivalent with diagnostic or prognostic information (before you
start to argue about the relative dangers of toxic
chemicals and information, reflect on how a
false positive1 or false negative test result 2 has
influenced your own life or medical practice).
We need the CDR to have been validated—
and by “validated”, I mean shown to accurately
discriminate between the affl icted and the
untouched—and to give accurate estimates of
the proportion of patients with the affl iction in
each category. If we are using a “limping child”
rule in the emergency department to guide our
referrals to orthopaedics and early discharges,
we want those with none, one, two, three or
four predictors to have a meaningfully different risk of septic arthritis (ie, discriminate). We
88
also want the no-predictor group to stay with a
really low proportion, ideally zero, and the four
predictor children to have a high proportion with
septic arthritis, touching 95% or more, that is,
retain accuracy.
It is almost always the case that the rule does not
work quite as well in the validation set of patients.
This is not unexpected, as the rule is derived from
one particular group of patient’s data, and another
dataset will be similar but not identical. There are
a variety of modifications or recalibrations that
can be undertaken to fi ne tune a rule and the modification then used more successfully in ongoing
care. 3
What frequently happens is that the group that
has developed a CDR goes on to test that it really
works. This “temporal validation” provides some
evidence to suggest that the rule really does work
and often allows for optimisation of the rule to
adjust for the overenthusiastic data-driven estimates that it initially created. Various approaches,
from the complex to the straightforward, have
been used to do this successfully.4 5 The problem that remains is that the CDR might work in
SuperHospital-W, where Dr X can train her residents into knowing exactly what point #2 of the
CDR really means and what a positive or negative
looks like, but can it work in SuperHospital-Y,
where the physicians are restricted to using the
data from the journal article?4 For some CDR,
where the basis of the rule is quality-assured laboratory results and proven-to-be reproducible clinical signs or symptoms, this may be easy. For other
rules, this may not be so effective (as a quick aside,
it is often helpful to assess just how reproducible
measurements are. There is a host of issues around
this, and a very readable series of BMJ “Statistics
Notes” by Profs Bland and Altman published in
1996, 5–7 and their textbooks of practical medical
statistics8 9 are accessible reads).
This next level of validation, seeing if the
limping-child rule is rooted in only one hospital
or not, looks at the CDR use in different physical
locations but with similar clinical settings and has
been called geographical validation. Potentially,
you might want to see if it works across different
clinical settings (domain validation), for example
in the emergency department of secondary-care
hospitals and walk-in primary-care clinics,10 but
this may be irrelevant if you are working in the
same setting as rule development occurred in.
The fi nal step should be to demonstrate their
efficacy in routine practice with multi-site randomised controlled trials.11 The trial does not
seek to see if the rule differentiates folk, and
Arch Dis Child Educ Pract Ed 2010;95:88–92. doi:10.1136/adc.2009.174458
Downloaded from ep.bmj.com on August 1, 2012 - Published by group.bmj.com
Best practice
can be applied, but instead randomises between
those who have the rule used, and those who
do not, and captures in its results key patient- or
health-system-important outcomes: minimising length of hospital stay or invasive testing
or improved quality of life. This is hardly ever
undertaken but can be a very powerful way of
showing improved care with a diagnostic intervention of any sort.12
UNDERSTANDING ASSESSMENTS OF ACCURACY
Assuming you have found a rule that has been validated in some way and accepting it is unlikely to
have gone through all these steps, then the next
bit of information you will be wanting to understand is “how good is the rule anyway”?
The information that a rule presents us with
is often couched as a “result”: the rule is positive
or negative (eg, Ottowa ankle rules), or the result
is a value (eg, the neonatal outcome measure of
CRIB score). These results are then described in
the papers that examine them in terms of their
diagnostic accuracy, using phrases that induce
fear and occasionally nausea: sensitivity, specificity, likelihood ratios (LR) and predictive values.
Sensitivity and specificity
In its simplest form, a rule produces a positive or
negative result. The group under study—take an
emergency room setting for children presenting
with fever—can be thought of as coming from two
populations: those with the “disease”, for example
pneumonia, and those without it. The proposed
way to distinguish between these groups may be
an invented blood test result—the “lung injury
protein”, which goes up if you have a chest infection. The “test” is a “positive result” if the value is
greater than a defi ned “threshold” (See fig 1).
The threshold then has part of populations on
each side: the proportion of people with pneumonia who have a positive test is the sensitivity, and
the proportion of people without pneumonia who
have a negative test is the specificity.
Shifting where the threshold is drawn alters
these proportions. As fig 2 shows, pushing the line
for “positivity” upwards makes the test more specific (captures fewer people without the disease in
the defi nition) but becomes less sensitive (fails to
diagnose a greater number with the disease). The
reverse is true when the level for positive results
is reduced.
The situation has an added level of complexity
with three levels of test result (eg, low, medium,
high), but the core concept remains the same.
For clinical application, the important values
are not the proportion of people with a pneumonia who have a positive test (sensitivity) and proportion of people without pneumonia who have
a negative test (specificity) but instead, the proportion of people with a negative test result who
truly did not have pneumonia (negative predictive
value (NPV )) and the converse, the proportion of
people with a positive test who did have pneumonia (positive predictive value (PPV )).
So, why do researchers continue to emphasise
sensitivity and specificity?
You might be tempted to think, as with medieval priests chanting Latin, this is to keep the
information cloaked from the common doctor.
You would not be entirely incorrect. However,
there is a better reason. The clinically useful
NPV and PPV are highly dependent on what the
prevalence of the condition is in the population
(also called the “baseline risk”) unlike the sensitivity and specificity, which although they are
shop-floor useless, do not depend upon that value
(generally, but if you take a completely different
type of population, like using an adult thrombosis detection CDR in the NICU, for example, you
are never going to get the same results). Take the
example of testing our invented “lung injury protein” test. Figure 3 demonstrates how the positive
and negative predictive values vary under different prevalence’s of disease (in the central panel,
10% prevalence, it shows how the positive test
cases are made up of seven with pneumonia—
the red squares—and 23 without—the blue
squares. The negative test results are the three
pink “missed” cases of pneumonia and 67 who
really do not have pneumonia). The “correct”
value of a positive test ranges from 13% to 55%
depending on how prevalent pneumonia is in the
population under study.
The upshot of this is that the direct translation
of PPV and NPV from studies into clinical practice
can only be undertaken if the prevalence of the
condition is the same in your population as the
study. If it is not, you need a little bit of calculation to work out what the values will be in your
practice and if this would still be a viable rule.
ANOTHER APPROACH—LIKELIHOOD RATIOS
Figure 1 ”Lung injury protein” diagnostic test
example .1
Arch Dis Child Educ Pract Ed 2010;95:88–92. doi:10.1136/adc.2009.174458
An alternative approach which captures the idea
of “what a positive/negative test means” but does
not rely on population prevalence is the use of LR.
These values compare the proportion of patients
with the disease and without the disease in for
a given test result. In the LIP example where the
cut-off is “93” (fig 1, Sensitivity 72.0%, Specificity
74.2%) these values would be LR+ (LR for a
89
Downloaded from ep.bmj.com on August 1, 2012 - Published by group.bmj.com
Best practice
Figure 2 ”Lung injury protein” diagnostic test example 2: different cutoffs.
Figure 3 Effect of varying prevalence on the PPV and NPV of “lung injury protein” with a sensitivity of 72% and a
specificity of 74.8%. NPV, negative predictive value; PPV, positive predictive value.
positive
test)=72.0/(100–74.2)=74.2/25.8=2.99
and the LR− (LR for a negative test)=0.38. Such
values can be calculated for each level of a test
result and so are useful for multi-level as well
as dichotomous results. As these are ratios, it is
numbers that are further away from one that are
more useful in either making (above one) or ruling out (below one) a diagnosis. The authors fi nd
it easier to think about LR− as fractions instead
of decimals to understand it better (eg, not 0.01
but 1/100th).
The interpretation of these can be done arithmetically, or with a nomogram, or with a sort of
“rule of thumb”:
▶
Likelihood ratio of >10 or <0.1 (1/10th)
▶ -big push towards or away from the
diagnosis
Table 1 Accuracy measures for “LIP” values to detect
imaginary pneumonia for sensitivity and specificity and
likelihood ratios
90
Test cut-off Sensitivity Specificity LR+
LR−
73
93
109
0.11
0.38
0.79
98.8
72.0
22.7
15.9
74.8
97.7
1.17
2.99
9.87
▶
▶
▶
Likelihood ratio of >5 or <0.2 (1/5th)
▶ -smaller push towards or away from the
diagnosis
Likelihood ratio of >2 or <0.5 (0.5)
▶ -a nudge towards or away from the
diagnosis
Likelihood ratio of closer to 1 than 2 or 0.5
(0.5)
▶ -very little diagnostic usefulness
To compare the usefulness of these measures,
have a look at tables 1 and 2.
The tables show that with increasingly high
cut-offs, the value of a positive test in diagnosing pneumonia improves, but a negative becomes
less useful in ruling out the problem. It’s up to you
which you fi nd easiest to interpret.
THE PSYCHOLOGY OF DIAGNOSING
Understanding how we physicians use a CDR,
or any diagnostic information, requires a bit of
knowledge about how doctors think. A number of researchers from a range of backgrounds
have examined the diagnostic practices of medics. There is a wealth of research that shows we
do not generally “take a history and do a physical
Arch Dis Child Educ Pract Ed 2010;95:88–92. doi:10.1136/adc.2009.174458
Downloaded from ep.bmj.com on August 1, 2012 - Published by group.bmj.com
Best practice
examination” before diagnosing something13 14
and that we use a massive array of mental shortcuts (heuristics)14 15 which speed up our working
practice, yet can lead us into dangerously wrong
conclusions. In fitting CDR into the practice of
managing a clinical problem, it can be useful to
take one straightforward model as to how doctors
make a diagnosis.
The diagnostic process can be thought of in
three stages: initiation, refi nement and conclusion.16 The initiation stage is where a diagnosis, or
differential, begins to be considered. The refi nement is a working through of differentials, and
the conclusion is a point where a decision to act
has been made.
For most situations, initiation is the germ of a
diagnostic idea: the acknowledgement that the
feverish, crying baby presenting in the emergency
department might have a number of causes for
their illness, including bacterial meningitis. This
fi rst inkling requires further work: the refi nement
process. It is in this process that CDRs can help
guide clinicians and lead to a diagnostic conclusion. Other approaches include a formal Bayesian
analysis, pattern fitting or a stepwise rule-out of
significant serious diagnoses.
The conclusion part of the process may be an
actual diagnosis (“the child has pneumonia”), a
rule-out (“the baby doesn’t have bacterial meningitis”) or a decision not to come to a conclusion
(“we’re still not sure why your little one is poorly,
and need to undertake a few more tests’).
An alternative model of diagnosis, present in
situations such as seeing a toddler with trisomy
21, has a single eye-blink-fast step in it. You see
the child, and you have made the diagnosis of
Down’s syndrome before a thought process can
be recognised. This is a pretty universal paediatric example, but there is work to suggest that the
more expert a physician is in a particular area, the
more rapidly they come to a diagnosis and that
this in part is because of pattern recognition or fitting new cases to a mental “categorisation”.15 In
these settings, CDRs can struggle to get a (mental)
look-in to the process.
Yet, another set of theories (“dual process theory”) suggests that stepwise, rational approach
and the rapid fast-heuristic conclusion approach
can happen at the same time. Some clinicians
favour the “rationalising” approach, other the
“clinical intuition” approach, but both may be
occurring. If there is a confl ict, for example,
between what the CDR suggests and “intuition”
delivers, the person’s tendency may tip the balance of the thinking.17
CDRs should help our thinking by providing
good-quality guidance to avoid diagnostic errors
and minimise unnecessary tests. Commonly
diagnostic errors occur because of systems and
cognitive errors.18 19 Such errors can include a failure to syntheses diagnostic information correctly
and come to a premature diagnostic conclusion;
a lack of appreciation of the value of a sign or
symptom in making a diagnosis or exaggeration
Arch Dis Child Educ Pract Ed 2010;95:88–92. doi:10.1136/adc.2009.174458
of the accuracy of a test fi nding. Other reasons
for misdiagnosis would not be helped by the use
of CDRs, such as the true diagnosis being rare or
failure in the technical skill of the individual doctor in reading an x-ray or eliciting a physical sign.
USING CDRS IN PRACTISE
There remains, however, an almost emotional
difficulty in turning to a CDR, when instead, we
should be like House,20 Holmes21 or Thorndyke22
in making our diagnoses from skill and knowledge. This is despite the knowledge that in many
cases a CDR performs better than our in-built
doctoring. 23–26
It may be that we are un-nerved by now knowing that there is a 2% chance of septic arthritis,
rather than being safe in the ignorance that “children like that one” in “my clinical experience” do
not have an infected joint. Chance alone could
mean you send home ~150 children and not get
one returning to the orthopods despite that group
having a 2% chance of infection (it also raises
questions about what a diagnosis really is: “the
truth” or just a high enough probability of “the
truth”).
There may be other issues, such as an objection
to reduced professional autonomy, a perception
of impracticality or a belief that the rule does not
fit the population we are dealing with.27 It might
reflect that we are just the sorts of people who
prefer working with the heuristics of medical
experience that we have built over time.17 This
question, as to why we do not follow where the
best evidence should steer us, is a matter of ongoing debate and research. 28
On a positive note, the most effective uses
of CDRs seem to have been where the rule has
a clear clinical usefulness, it is championed by
well-respected local clinicians and it makes life
better for patients and clinicians.29 30 Lots of these
features can be picked up and put in place when
you are trying to promote the use of a CDR. You
will have decided, from an analysis based on the
content of these two articles, whether the rule is
technically good and likely to be true. You can
assess how much of an impact it should make on
your service, and you will be well placed to identify the key players who need to be “on side” in
order to get the rule used (Who will it benefit?
Let them know, and let them know how it will be
good for them! Who will it disadvantage—with
more work or nastier testing? Use case stories and
hard stats together to show how overall it makes
things better for the patients, who are the key
cause of concern for us all).
CONCLUSIONS
CDRs are potentially a way of making our diagnoses more accurate and/or quicker and less unpleasant to achieve. They have not always done this, 31
and their creation and testing is a potential minefield, as these articles have examined. The essential principles that should guide us in our clinical
91
Downloaded from ep.bmj.com on August 1, 2012 - Published by group.bmj.com
Best practice
work, when seeking to use a CDR, should remain
the same as in all other areas. We should seek to
use evidence that is valid, important and makes
a difference to patients. After all, that is why we
became paediatricians, is it not?
14.
Acknowledgements The author thanks to Professor Lesley
Stewart for her helpful comments on an earlier draft of this
manuscript.
17.
Funding This work was funded as part of an MRC Research
Training Fellowship.
Competing interests None.
Provenance and peer review Commissioned; externally peer
reviewed.
REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
92
15.
16.
18.
19.
20.
21.
Gigerenzer G. Reckoning with Risk: Learning to Live with
Uncertainty. London (UK): Penguin Press, 2003.
Wardle FJ, Collins W, Pernet AL, et al. Psychological impact
of screening for familial ovarian cancer. J Natl Cancer Inst
1993;85:653–7.
Janssen KJ, Moons KG, Kalkman CJ, et al. Updating methods
improved the performance of a clinical prediction model in new
patients. J Clin Epidemiol 2008;61:76–86.
Goodacre S, Sutton AJ, Sampson FC. Meta-analysis: The
value of clinical assessment in the diagnosis of deep venous
thrombosis. Ann Intern Med 2005;143:129–39.
Bland JM, Altman DG. Measurement error. BMJ 1996;313:744.
Bland JM, Altman DG. Measurement error and correlation
coefficients. BMJ 1996;313:41–2.
Bland JM, Altman DG. Measurement error proportional to the
mean. BMJ 1996;313:106.
Altman DG. Practical Statistics for Medical Research. London
(UK): Chapman & Hall, 1990.
Bland JM. An Introduction to Medical Statistics. Oxford (UK):
Oxford University Press, 2000.
Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating
and impact of clinical prediction rules: a review. J Clin Epidemiol
2008;61:1085–94.
Selby PJ, Powles RL, Easton D, et al. The prophylactic role of
intravenous and long-term oral acyclovir after allogeneic bone
marrow transplantation. Br J Cancer 1989;59:434–8.
Lee MT, Piomelli S, Granger S, et al.; STOP Study Investigators.
Stroke Prevention Trial in Sickle Cell Anemia (STOP): extended
follow-up and final results. Blood 2006;108:847–52.
Montgomery K. How doctors think: clinical judgment and the
practice of medicine: Clinical Judgement and the Practice of
Medicine. New York (NY): Oxford University Press, 2005.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
Norman G. Research in clinical reasoning: past history and
current trends. Med Educ 2005;39:418–27.
Elstein AS, Schwartz A, Schwarz A. Clinical problem solving
and diagnostic decision making: selective review of the
cognitive literature. BMJ 2002;324:729–32.
Heneghan C, Glasziou P, Thompson M, et al. Diagnostic
strategies used in primary care. BMJ 2009;338:b946.
Sladek RM, Phillips PA, Bond MJ. Implementation science: a
role for parallel dual processing models of reasoning? Implement
Sci 2006;1:12.
Graber ML, Franklin N, Gordon R. Diagnostic error in internal
medicine. Arch Intern Med 2005;165:1493–9.
Redelmeier DA. Improving patient care. The cognitive
psychology of missed diagnoses. Ann Intern Med
2005;142:115–20.
Heel-and-Toe F. House. M.D. http://www.fox.com/house//
index/htm#home (accessed 4 May 2010).
Doyle AC. The adventures of Sherlock Holmes. London:
Penguin Classics, 2007.
Freeman RA. John Thorndyke’s Cases related by Christopher
Jervis. E-published by Guttenberg at http://www.gutenberg.
org/etext/13882. Freeman R A. 2004.
Dawes RM, Faust D, Meehl PE. Clinical versus actuarial
judgment. Science 1989;243:1668–74.
Grove WM, Zald DH, Lebow BS, et al. Clinical versus
mechanical prediction: a meta-analysis. Psychol Assess
2000;12:19–30.
Quezada G, Sunderland T, Chan KW, et al. Medical and
non-medical barriers to outpatient treatment of fever and
neutropenia in children with cancer. Pediatr Blood Cancer
2007;48:273–7.
Lee TH, Pearson SD, Johnson PA, et al. Failure of information
as an intervention to modify clinical management. A timeseries trial in patients with acute chest pain. Ann Intern Med
1995;122:434–7.
Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians
follow clinical practice guidelines? A framework for
improvement. JAMA 1999;282:1458–65.
Richardson WS. We should overcome the barriers to evidencebased clinical diagnosis! J Clin Epidemiol 2007;60:217–27.
Bessen T, Clark R, Shakib S, et al. A multifaceted strategy for
implementation of the Ottawa ankle rules in two emergency
departments. BMJ 2009;339:b3056.
Gaddis GM, Greenwald P, Huckson S. Toward improved
implementation of evidence-based clinical algorithms:
clinical practice guidelines, clinical decision rules, and clinical
pathways. Acad Emerg Med 2007;14:1015–22.
Roukema J, Steyerberg EW, van der Lei J, et al. Randomized
trial of a clinical decision support system: impact on the
management of children with fever without apparent source. J
Am Med Inform Assoc 2008;15:107–13.
Arch Dis Child Educ Pract Ed 2010;95:88–92. doi:10.1136/adc.2009.174458
Downloaded from ep.bmj.com on August 1, 2012 - Published by group.bmj.com
Clinical decision rules: how to use them
B Phillips
Arch Dis Child Educ Pract Ed 2010 95: 88-92
doi: 10.1136/adc.2009.174458
Updated information and services can be found at:
http://ep.bmj.com/content/95/3/88.full.html
These include:
References
This article cites 24 articles, 10 of which can be accessed free at:
http://ep.bmj.com/content/95/3/88.full.html#ref-list-1
Email alerting
service
Receive free email alerts when new articles cite this article. Sign up in
the box at the top right corner of the online article.
Notes
To request permissions go to:
http://group.bmj.com/group/rights-licensing/permissions
To order reprints go to:
http://journals.bmj.com/cgi/reprintform
To subscribe to BMJ go to:
http://group.bmj.com/subscribe/