How to Use an Article Reporting a Multiple Treatment Comparison Meta-analysis

USERS’ GUIDES TO THE MEDICAL LITERATURE
How to Use an Article Reporting a Multiple
Treatment Comparison Meta-analysis
Edward J. Mills, PhD, MSc
John P. A. Ioannidis, MD, DSc
Kristian Thorlund, PhD, MSc
Holger J. Schu¨nemann, MD, PhD, MSc
Milo A. Puhan, MD, PhD
Gordon H. Guyatt, MD, MSc
CLINICAL SCENARIO
You are seeing a 45-year-old patient for
whom, 6 weeks previously, you prescribed paroxetine, a selective serotonin reuptake inhibitor (SSRI), for treatment of generalized anxiety disorder
(GAD). The patient reports reduced
anxiety, but also insomnia and a reduced interest in sex. You wonder if there
is another drug the patient might tolerate better while still maintaining treatment response. You retrieve a multiple
treatment comparison (MTC) metaanalysis that evaluates response and tolerability of all available GAD drugs.1 You
are not familiar with this type of study,
and you wonder if there are special issues to which you should attend in evaluating its methods and results.
MTC META-ANALYSES
Traditionally, a systematic review addresses the merits of one intervention
vs another (eg, placebo, or another active intervention). Data are combined
from all studies—often randomized
clinical trials (RCTs)—that meet eligibility criteria in what we will term a
pairwise meta-analysis. Compared with
single RCTs, meta-analysis improves
the power to detect differences and also
facilitates the examination of the extent to which there are important differences in treatment effects across
eligible RCTs—variability that is frequently called heterogeneity.2,3 Large,
1246
JAMA, September 26, 2012—Vol 308, No. 12
Multiple treatment comparison (MTC) meta-analysis uses both direct (headto-head) randomized clinical trial (RCT) evidence as well as indirect evidence from RCTs to compare the relative effectiveness of all included interventions. The methodological quality of MTCs may be difficult for clinicians
to interpret because the number of interventions evaluated may be large and
the methodological approaches may be complex. Clinicians and others evaluating an MTC should be aware of the potential biases that can affect the
interpretation of these analyses. Readers should consider whether the primary studies are sufficiently homogeneous to combine; whether the different interventions are sufficiently similar in their populations, study designs,
and outcomes; and whether the direct evidence is sufficiently similar to the
indirect evidence to consider combining. This article uses the existing Users’
Guides format to address study validity, interpretation of results, and application to a patient scenario.
www.jama.com
JAMA. 2012;308(12):1246-1253
unexplained heterogeneity may reduce a reader’s confidence in estimates of treatment effects.
A drawback of pairwise metaanalysis is that it evaluates the effects of
only 1 intervention vs 1 comparator
and does not permit inferences about the
relative effectiveness of several interventions unless all have been compared
directly in head-to-head trials. Yet, for
many medical conditions, there are numerous available interventions that
have—unfortunately—most frequently been compared with placebo and
seldom with one another.4,5 For example, despite 91 completed and ongoing RCTs addressing the effectiveness of
drugs for the treatment of rheumatoid arthritis, only 5 compare directly against
each other.4
Recently, another form of metaanalysis, the MTC meta-analysis (also
known as network meta-analysis because it involves creating a network of
treatments), has emerged.6,7 The MTC
approach provides estimates of effect
sizes for all possible pairwise comparisons whether or not they have been
compared head-to-head in RCTs.
FIGURE 1 displays examples of common networks of treatments. The eAppendix (available at http://www.jama
.com) provides a glossary of common
nomenclature found in MTCs.
When 2 interventions, A and B (eg,
paroxetine and lorazepam in FIGURE 2A),
Author Affiliations: Faculty of Health Sciences, University of Ottawa, Ottawa, Ontario, Canada (Dr
Mills); Department of Clinical Epidemiology and
Biostatistics, McMaster University, Hamilton,
Ontario, Canada (Drs Mills, Thorlund, Schu¨ nemann, and Guyatt); Stanford Prevention Research
Center, Departments of Medicine and Health
Research and Policy, Stanford University School of
Medicine, and Department of Statistics, Stanford
University School of Humanities and Sciences, Stanford, California (Dr Ioannidis); Department of Epidemiology, Johns Hopkins Bloomberg School of
Public Health, Baltimore, Maryland (Dr Puhan).
Corresponding Author: Edward J. Mills, PhD, MSc, Faculty of Health Sciences, University of Ottawa, Room
031, Thompson Hall, 35 University Private, Ottawa,
ON, K1N 7K4, Canada (edward.mills@uottawa.ca).
Users’ Guides to the Medical Literature Section Editor: Drummond Rennie, MD, Deputy Editor, JAMA.
©2012 American Medical Association. All rights reserved.
USERS’ GUIDES TO THE MEDICAL LITERATURE
Figure 1. Examples of Possible Network Geometry
Treatment or intervention node
Direct comparison in RCT
Star network
Single closed loop
B
A
G
Complex network
Connected network
A
B
A
B
C
G
A
C
C
F
D
B
D
D
C
E
E
F
The figure shows 4 network graphs. In each graph, lines show where direct comparisons exist from one or more trials. The star shows a network for which all interventions have a single mutual comparator. A single closed loop involves 3 interventions and can provide data to calculate both direct comparisons and indirect comparisons. A well-connected network in which all interventions have been compared against each other in multiple randomized controlled trials (RCTs). The complex
network has multiple loops and arms that may have sparse connections.
have not been compared directly, one
can still estimate their relative effect if
each has been compared directly against
another intervention, C (eg, placebo).
This is called an adjusted indirect comparison of A and B.8 Multiple treatment comparisons simultaneously include both direct and indirect evidence.
Indirect comparisons (and MTCs) make
the assumption that the relevant trials
are similar enough in essential features (eg, patient characteristics, definitions and measurements of outcomes, and risk of bias in the studies)9
to be combined.
There are 3 chief questions concerning the conduct of an MTC. First,
among trials available for pairwise comparisons, are the studies sufficiently homogeneous to combine for each intervention? Second, across the trials
involved in all interventions, are the
studies sufficiently similar, with the exception of the intervention (eg, in important features such as populations,
design, or outcomes), that they can be
compared? Third, where direct and indirect evidence exists, are the findings
sufficiently consistent that both direct
and indirect evidence can be relied
upon?
By including evidence from both direct and indirect comparisons, an MTC
may increase precision in estimates of
the relative effects of treatments and facilitate simultaneous comparisons, or
even ranking, of these treatments.7
However, because MTCs are methodologically sophisticated, they are often challenging to interpret.10
Herein, we will demystify the MTC
by using the 3-step validity-resultsapplicability approach of other Users’
Guides.11 BOX 1 includes all issues relevant to evaluating systematic reviews, with a discussion that highlights the issues most important in
MTCs.
ARE THE RESULTS
OF THE STUDY VALID?
Figure 2. A Simple Indirect Comparison and
Simple Closed Loop
Treatment or intervention node
Direct comparison in RCT
Indirect comparison
A Indirect comparison
Placebo
Paroxetine
Lorazepam
Did the Review Explicitly Address
a Sensible Clinical Question?
One can formulate questions of optimal patient management in terms of patients, interventions, comparators, and
outcomes.
Broader eligibility criteria may enhance generalizability of the results, but
may be misleading if participants are too
dissimilar and, as a consequence,
heterogeneity is large. Diversity of interventions may also be excessive if authors pool results from different doses,
or even different agents in the same
class (eg, all statins), based on the assumption that effects are similar. Readers should ask whether investigators
have been too broad in their inclusion
of different populations, of different
doses or different agents in the same
class, or of different outcomes, and
©2012 American Medical Association. All rights reserved.
B Closed loop
Nicotine replacement
therapy
Varenicline
Bupropion
A, In an indirect comparison, there is direct evidence from
paroxetine compared with placebo and direct evidence
of lorazepam compared with placebo. Therefore, the indirect comparison can be applied to determine the effect of paroxetine compared with lorazepam, even if no
direct head-to-head comparison exists for these 2 agents.
B, In the closed loop, there is direct evidence that compares nicotine replacement therapy with both varenicline
and also bupropion. There is also direct evidence comparing bupropion with varenicline. Therefore, enough information exists to evaluate whether the results are coherent between direct and indirect evidence.
JAMA, September 26, 2012—Vol 308, No. 12
1247
USERS’ GUIDES TO THE MEDICAL LITERATURE
Box 1. Critical Appraisal Guide to a Multiple Treatment Comparison
Meta-analysis
A. Are the results of the study valid?
Did the review explicitly address a sensible clinical question?
Was the search for relevant studies exhaustive?
Were there major biases in the primary studies?
B. What are the results?
What was the amount of evidence in the network?
Were the results similar from study to study?
Were the results consistent in direct and indirect comparisons?
What were the overall treatment effects and their uncertainty, and how did the
treatments rank?
Were the results robust to sensitivity assumptions and potential biases?
C. How can I apply the results to patient care?
Were all patient-important outcomes considered?
Were all potential treatment options considered?
Are any postulated subgroup effects credible?
What is the overall quality and what are limitations of the evidence?
Box 2. Using the Guide
Returning to our opening scenario, the
multiple treatment comparison we identified compared the efficacy and tolerability of available general anxiety disorder
(GAD) drugs, with a specific focus on
drugslicensedintheUnitedKingdom(because it was a British study) using RCT evidence.1 (P, GAD patients; I and C, available drugs; O, efficacy and tolerability.)
Patients in the included RCTs met similarly broad diagnostic criteria, including
being older and receiving inpatient and
outpatient care.29 The authors assumed all
doses of each drug were equivalent (no
good evidence exists for or against this assumption).Foroutcomes,theauthorsconsidered response (ⱖ50% reduction in
Hamilton Anxiety Scale), remission (final
score ⱕ7), and tolerability (not withdrawing because of adverse events). These outcomes are important to patients, and the
should look carefully at the resulting
heterogeneity.
When substantial clinical variability or statistical heterogeneity is present,
authors may conduct subgroup analyses or meta-regression to explain
heterogeneity. If such analyses are successful in explaining heterogeneity, the
1248
JAMA, September 26, 2012—Vol 308, No. 12
definitions of the outcomes were consistent across trials.
The search for published literature was
comprehensive, but the authors did not
search for unpublished data and did not
include RCTs of reboxetine, buspirone, or alprazolam, interventions for
which data were available. Because many
mental health RCTs are unpublished,20
the risk of publication bias is substantial. Eligibility assessment was done by
a single individual with a 10% random
selection of articles additionally reviewed by another reviewer. These same
reviewers independently extracted prespecified information on study characteristics and outcomes. The authors did
not perform a risk-of-bias assessment of
the included trials.
P indicates patients; I, interventions; C, comparators; and O, outcomes.
MTC may provide results that more optimally match the clinical settings and
the characteristics of the patient.12 For
example, in an MTC evaluating different statins for cardiovascular disease
protection, the authors used metaregression to address whether it was appropriate to combine results across
primary and secondary prevention
populations, different statins, and different doses of statins by examining
whether trials with these features exhibited different treatment effects.13
Including multiple control interventions (eg, placebo, no intervention, or
older standard of care) may enhance the
robustness and connectedness of the
treatment network. However, it is important to gauge and account for potential differences between control
groups. For example, due to potential
placebo effects, patients receiving placebo in a blinded RCT may have differing responses than patients receiving no intervention in a nonblinded
RCT. Thus, if 2 of 3 active treatments,
A and B, have been compared with placebo, and 2 of 3 active treatments, B and
C, have been compared with no intervention, the different choice of “control groups” may produce misleading
results. By examining whether certain
trials exhibit different treatment effects, meta-regression may address this
problem.
For example, in an MTC evaluating
the effectiveness of smoking cessation
therapies, the authors combined placebo-controlled groups with standardof-care control groups and then used
meta-regression to examine whether the
choice of control changed the effect
size.14 The authors found that trials
using placebo controls had smaller effect sizes than those using standard of
care, and this explained the identified
heterogeneity.
Was the Search for Relevant
Studies Exhaustive?
Some published MTCs have used the
search strategies from other systematic reviews as the basis for identifying potentially eligible trials. Readers
can be confident in such approaches
only if the authors have updated the
search to include recently published
trials.15
The eligible interventions can be unrestricted. Sometimes, however, the authors may choose to include only a specific set of interventions; eg, those
available in their country. Some indus-
©2012 American Medical Association. All rights reserved.
USERS’ GUIDES TO THE MEDICAL LITERATURE
try-initiated MTCs may choose to consider only a sponsored agent and its direct competitors.16 This may omit the
optimal agent for some situations and
tends to give a fragmented picture of the
evidence. It is typically best to include
all interventions.17 Data on clearly suboptimal or abandoned interventions
may still offer indirect evidence for
other comparisons.17
In an MTC of 12 treatments for major depression, the authors chose to exclude placebo-controlled RCTs and included only head-to-head active
treatment RCTs.18 However, publication bias in the antidepressant literature is well acknowledged,19,20 and by excluding placebo-controlled trials, the
analysis loses the opportunity to benefit from additional available evidence.21 Exclusion of eligible interventions, in this case placebos, may not just
decrease statistical power. Placebocontrolled trials may be different from
head-to-head comparison trials in their
conduct or in the degree of bias (eg, they
may have more or less publication bias
or selective outcome and analysis reporting). Thus, their exclusion may also affect
the point estimates of the effects of pairwise comparisons and may affect the relative ranking of regimens. Placing too
great an emphasis on the interpretation
of rankings rather than the relative comparisons may be misleading to clinicians and patients. When a network
meta-analysis of second-generation antidepressants was later conducted and did
include placebo-controlled trials, relying only on the relative differences between treatments using the same depression scale, the authors reached a different
interpretation from the earlier MTC.18,22,23
Finally, original trials often address
multiple outcomes. Selection of MTC
outcomes should not be data-driven but
should be based on importance for patients and consider both benefit and
harm outcomes.
Were There Major Biases
in the Primary Studies?
Trial-level limitations (eg, lack of concealment or blinding, or loss to followup) may bias results of RCTs.24 The
Box 3. Potential Reasons for Incoherence Between the Results
of Direct and Indirect Comparisons
Chance
Genuine diversity
Differences in enrolled participants (eg, entry criteria, clinical setting, disease
spectrum, baseline risk, selection based on prior response)
Differences in the interventions (eg, dose, duration of administration, prior administration [second-line treatment])
Differences in background treatment and management (eg, evolving treatment
and management in more recent years)
Differences in definition or measurement of outcomes
Bias in head-to-head (direct) comparisons
Optimism bias with unconcealed analysis
Publication bias
Selective reporting of outcomes and analyses
Inflated effect size in early stopped trials and in early evidence
Limitations in allocation concealment, blinding, loss to follow-up, analysis as
randomized
Bias in indirect comparisons
Each of the biasing issues above can affect the results of the direct comparisons
on which the indirect comparisons are based
Cochrane Collaboration provides detailed advice on dealing with triallevel biases in meta-analysis.25,26
Publication bias and selective outcome reporting bias 27,28 are major
threats to the results of pairwise metaanalyses and may affect the availability of information among MTCs. The
effect of these biases depends also on
whether specific interventions and comparisons are affected more than others
(BOX 2).
WHAT ARE THE RESULTS?
What Was the Amount of Evidence
in the Treatment Network?
One can gauge the amount of evidence in the treatment network from
the number of trials, total sample size,
and number of events for each treatment and comparison. Furthermore,
the extent to which the treatments are
connected in the network is an important determinant in the quality of the
evidence. Understanding the geometry of the network (nodes and links)
will permit the reader to examine the
larger picture and see what is com-
©2012 American Medical Association. All rights reserved.
pared with what.30 The authors should
present the structure of the network (as
in Figure 1 examples). Multiple treatment comparison analyses are feasible
when there is a consistent connection
in the network.
When different interventions have
only been compared with a single common comparator (eg, placebo), this is
a star network (Figure 1). A star network only allows for indirect comparison between active treatments, which
reduces confidence in effects, particularly if there are a limited number of
trials, patients, and events.31 When there
are data available using both direct and
indirect evidence of the same interventions, this is a closed loop. The presence of direct evidence increases confidence in the estimates of interest.
Often, a treatment network will include a mixture of exclusively indirect
links and closed loops. Most networks
have unbalanced shapes with many
trials of some comparisons but few or
none of others.30 In this situation, evidence may be of high quality for some
treatments and comparisons but of low
JAMA, September 26, 2012—Vol 308, No. 12
1249
USERS’ GUIDES TO THE MEDICAL LITERATURE
Box 4. Using the Guide
Returningtoourclinicalscenario,Figure3
displays the network of considered treatments. The authors excluded 13 relevant
RCTs because the interventions could not
be connected to any of the treatments in
the network. All treatments are informed
by placebo comparisons, some by more
trials than others, and some by direct comparison of active treatments . There are different degrees of evidence supporting the
comparative effectiveness of each of the
treatments.Forexample,thefluoxetineevidence comes only from a subgroup analysisofalargertrialthatcomparesitwithvenlafaxine and placebo. The evidence base
for fluoxetine is thus a very weak mix of
direct (1 trial) and indirect (1 fluoxetine
vs venlafaxine trial and 8 venlafaxine
vs placebo trials). In contrast, evidence for
pregabaline is based on a larger amount of
evidencefrombothdirectandindirectcomparisons. Five trials have compared pregabalinewithplacebo,2trialshavecompared
pregabalinewithlorazepam,and1trialhas
compared pregabaline with venlafaxine.
Here, the availability of 3 head-to-head
trials increases our confidence in effect
estimates.
Of 27 included RCTs, 23 reported
response to treatment, 23 reported withdrawals due to adverse effects, and 14
reported on remission, raising concern
about missing outcomes and selective
quality for others. Unfortunately, MTC
studies often fail to point out these differences in the evidence quality.
Were the Results Similar From
Study to Study?
Marked, unexplained differences in
treatment effects across trials in direct
comparisons of alternative agents, or of
single agents with no-treatment controls, lowers confidence in effect estimates. There are a number of statistical
measures available to guide assessment of whether the variability (heterogeneity) of results is high; eg, I2 and
other metrics.32 Authors should report the degree of heterogeneity in all
comparisons—the greater the heterogeneity, the less certain the results.
1250
JAMA, September 26, 2012—Vol 308, No. 12
reporting,inparticularforremission.Studiespredominantlyincludedpatientsreceiving treatment for between 4 and 12 weeks.
Because the authors did not report tests
for pairwise heterogeneity, we cannot tell
if the magnitude of effect was similar from
study to study in each of the direct comparisons or in each drug-placebo comparison. The authors did, however, conduct
severalsensitivityanalyses.Theauthorsalso
checked the coherence between direct and
indirectcomparisonsfromclosedloopsand
did not find significant incoherence.
They found that all drugs, except tigabine, exhibited significantly better response rates compared with placebo. Most
treatments offered about a 50% improvement in response rates over placebo. Regarding our patient’s current drug (paroxetine), no drug appeared significantly
better. When the authors calculated the
probability of which drug was best, fluoxetine ranked the highest (63%), whereas
paroxetine ranked third. Paroxetine had
a higher rate of drug discontinuation than
placebo (odds ratio, 2.51; 95% credible interval, 1.50-4.19), and ranked fourth for
tolerability. By comparison, fluoxetine
ranked first for response and third for tolerability. The results were not entirely robust in sensitivity analysis because the
best-ranked treatment for effectiveness
changed from fluoxetine to duloxetine.
Possible explanations of differences
in treatment effects can be examined
using subgroup analysis and metaregression. However, these analyses are
limited in the presence of small numbers of trials, and apparent subgroup effects often prove spurious, an issue to
which we will return in our discussion of applicability.33-35
Were the Results Consistent in
Direct and Indirect Comparisons?
Head-to-head direct comparisons of
treatments are generally more trustworthy than indirect comparisons.
However, head-to-head trials can also
yield misleading estimates; eg, when
conflicts of interest influence the choice
of comparators used or result in selec-
tive reporting. Therefore, indirect comparisons may on occasion provide more
trustworthy estimates.36
One can assess whether direct and indirect estimates yield similar effects
whenever there is a closed loop in the
network (as in Figure 2B). Statistical
methods exist for checking this type of
inconsistency, typically called a test for
incoherence.37,38
An evaluation examining 112 examples of direct vs indirect evidence
found that the results were statistically inconsistent 14% of the time.9 This
same evaluation found that comparisons with smaller number of trials and
measuring subjective outcomes had a
greater risk of incoherence. When incoherence is present, there are many explanations for the authors—and for
readers—to consider (BOX 3).
For example, a meta-analysis
examining the analgesic efficacy of
paracetamol (acetaminophen) plus
codeine in surgical pain displayed a
direct comparison indicating the intervention was more efficacious than
paracetamol alone (mean difference in
pain intensity change, 6.97; 95% CI,
3.56 to 10.37). The adjusted indirect
comparison did not show a significant
difference between paracetamol plus
codeine and paracetamol alone (−1.16,
95% CI; −6.95 to 4.64). 39 In this
example, the direct and indirect evidence was statistically significantly incoherent (P=.02). This indirect analysis
addresses a continuous subjective outcome (ie, pain intensity), and is thus
more likely to statistically demonstrate incoherence. The explanation for
incoherence may be that the direct trials
included patients who had lower pain
intensity at baseline, who had greater
representation in the direct trials, and
who may have been more responsive to
the addition of codeine.
Statistical testing is often not sufficient to document incoherence. Because MTCs are often imprecise, important differences may still exist in the
absence of a statistically significant difference. Authors should report on
whether they assessed incoherence and,
if they did, what they found.
©2012 American Medical Association. All rights reserved.
USERS’ GUIDES TO THE MEDICAL LITERATURE
What Were the Overall Treatment
Effects and Their Uncertainty,
and How Did Treatments Rank?
The treatment effects in an MTC are
typically displayed with common effect estimates along with 95% credible
intervals (CrIs). Credible intervals are
the Bayesian equivalent to the more
commonly understood, frequentist confidence intervals. When there are K interventions included in the treatment
network, there are K × (K −1)/2 possible pairwise comparisons. For example, if there are 7 interventions, then
there are 21 [7⫻(7 −1)/2] possible pairwise comparisons.
Besides presenting treatment effects, authors may also present the probability that each treatment is superior
to all other treatments, allowing ranking of treatments.40,41 Although this approach is appealing, it may also oversimplify an analysis because of fragility
in the rankings, because differences between the ranks may be too small to be
important, or because bias in the MTC
may importantly affect the rank order.
For example, an MTC examining directacting agents for hepatitis C found no
statistical difference for sustained virological response between teleprevir
and boceprevir (odds ratio, 1.42; 95%
CrI, 0.89-2.25); based on these results, the probability of being the best
favors teleprevir by far (93%) over boceprevir (7%).42 However, this 93%
probability provides a misleadingly
strong endorsement for teleprevir. One
may wish to know the probability that
teleprevir is better than boceprevir by
a clinically important margin, eg, at least
50% better, and this probability is only
about 40%. Therefore, probabilities of
being the best should be interpreted
with caution.
Were the Results Robust
to Sensitivity Assumptions
and Potential Biases?
Given the complexity of some MTC
meta-analyses, authors may assess the
robustness of their study findings by applying sensitivity analyses that show
how the results change if some criteria or assumptions change. Sensitivity
Figure 3. Treatment Network for the Drugs Considered in the Example Multiple Treatment
Comparison on Generalized Anxiety Disorder
Treatment or intervention node
n
Paroxetine
No. of RCTs in direct comparison
Lorazepam
Pregabalin
2
1
2
Fluoxetine
Sertraline
1
1
3
Duloxetine
Tiagabine
5
3
1
2
2
1
Escitalopram
Venlafaxine
5
4
8
Placebo
The lines between treatment nodes indicate the comparisons made throughout randomized clinical trials (RCTs).
The numbers on the lines indicate the number of RCTs informing a particular comparison. (The figure is based
on Baldwin et al.1)
analyses may include restricting the
analyses to trials with a low risk of bias
only or by examining different but related outcomes. The Cochrane Handbook for Systematic Reviews of Interventions 2 5 provides a discussion of
sensitivity analyses.
In an MTC on prevention of chronic
obstructive pulmonary disease (COPD)
exacerbations, the authors used the incidence rate as the primary outcome.
However, there is some debate on
whether incidence rates should be used
in COPD trials,43 so the authors conducted sensitivity analyses using the binary outcome of ever having an exacerbation. The results were sufficiently
similar to consider the analyses robust44 (BOX 4, FIGURE 3).
HOW CAN I APPLY THE RESULTS
TO PATIENT CARE?
Were All Patient-Important
Outcomes Considered?
Many MTCs report only 1 or a few selected outcomes of interest. For example, a recent MTC comparing the efficacy of antihypertensive treatments
only looked at heart failure and mortality,45 whereas an older MTC of an-
©2012 American Medical Association. All rights reserved.
tihypertensive treatments also considered coronary heart disease and stroke.46
Adverse events are infrequently assessed in meta-analyses and in MTCs,
reflecting poor reporting in primary
trials.47,48 Multiple treatment comparisons conducted in the context of health
technology assessment submissions and
evidence-based practice reports are
more likely to include multiple outcomes and assessments of harms than
the less lengthy, journal-based MTCs.
Were All Potential Treatment
Options Considered?
Multiple treatment comparisons may
place restrictions on what treatments
are examined. For example, for irritable bowel syndrome, an MTC may
focus on pharmacological agents,
neglecting RCTs of diet, peppermint
oil, and counseling. 49 Decisions to
focus on subclasses of drugs may also
be problematic. For example, in rheumatoid arthritis, biologic diseasemodifying antirheumatic drugs (biologics) are used for patients failing
conventional drugs. Five of the 9
available biologics are antitumor
necrosis factor agents (anti-TNFs).
JAMA, September 26, 2012—Vol 308, No. 12
1251
USERS’ GUIDES TO THE MEDICAL LITERATURE
Box 5. Using the Guide
The assessed outcomes (response, remission, and overall tolerability) are
important, but some additional outcomes would be useful to know (eg,
specific harms such as sexual dysfunction or insomnia). A concern is that
although 9 treatments were evaluated, only one randomized controlled trial evaluated the drug that
ranked the best (fluoxetine, comparing it to placebo and venlafaxine) and
that evidence was from a subgroup
analysis in the original trial where
fluoxetine was not better than its active comparator (venlafaxine). Only 3
drugs had been evaluated in direct
comparisons against at least 2 other
drugs. In the subgroup analysis examining only licensed dose drugs
available in the United Kingdom, the
results and rankings were different
than the primary analysis, thus lowering our confidence. In summary,
confidence in estimates is limited by
uncertainty regarding the risk of bias
in individual studies, the possibility of
publication bias, the small number of
patients studied and the correspondingly imprecise estimates, the lack of
information regarding heterogeneity
in individual comparisons, and a paucity of direct comparisons.
Are Any Postulated Subgroup
Effects Credible?
There are very few situations in which
investigations have convincingly established important differences in the relative effect of treatment according to patient characteristics.51 Criteria exist for
determining the credibility of subgroup
analyses.51 Multiple treatment comparisons allow a greater number of RCTs to
be evaluated and may offer more opportunities for subgroup analysis, but with
due skepticism and respect for credibility criteria.
In an MTC examining inhaled drugs
for COPD, the authors examined
whether a patient’s disease stage effected response according to severity of
airflow obstruction, measured by forced
expiratory volume in 1 second (FEV1).52
If FEV1 was 40% or less of predicted, then
long-acting anticholinergics, inhaled corticosteroids, and combination treatment reduced exacerbations significantly compared with long-acting
␤-agonists alone, but not if FEV1 was
greater than 40% predicted. This difference was significant for inhaled corticosteroids (P=.02 for interaction) and combination treatment (P=.01), but not for
long-acting anticholinergics (P=.46).
What Is the Overall Quality
of the Evidence,
and What Are the Limitations?
Box 6. Clinical Resolution
You conclude that low-quality evidence allows only weak inferences regarding the effectiveness of the treatment your patient is currently
receiving relative to other options.
You recognize, however, that other
drugs appear to have better tolerability profiles and will discuss with your
patient the possibility of switching to
one of these agents.
One recent MTC only considered
anti-TNFs and excluded other biologics.50 To the extent that the other biologic agents are equivalent or superior
to the anti-TNFs, their exclusion risks
giving clinicians the wrong impression of the best biologic agents.
1252
JAMA, September 26, 2012—Vol 308, No. 12
It is important to determine the quality
of the overall MTC so that a reader can
determine if the evidence provides
strong inferences. The following aspects are hallmarks of high-quality evidence in an MTC: individual studies are
at low risk of bias and publication bias
is unlikely; results are consistent in individual direct comparisons and individual comparisons with no-treatment
controls and are consistent between direct and indirect comparisons; sample
size is large and confidence intervals are
correspondingly narrow; and evidence
includes direct comparisons. If all of
these hallmarks are present and the differences in effect sizes are large, high
confidence in estimates may be warranted. However, in most cases, some
but not all of these hallmarks are present,
and confidence in key estimates are
likely to warrant only moderate or low
confidence.
As MTC methods become popular,
clinicians will find more than one metaanalysis addressing the same question
and—as in the case of the secondgeneration antidepressants that we have
already mentioned—may sometimes arrive at different conclusions.18,54 Differences in eligibility criteria and outcomes may explain these differences.
To illustrate, there are at least 15 published MTCs and health technology assessments on the comparative effectiveness of biologics for rheumatoid arthritis.
Across these published reports, one can
observe considerable discrepancies with
respect to the numbers and sets of included trials, assessment of heterogeneity and risk of bias, and inclusion/
exclusion/modification of outcomes.53
Concerns with each of these issues, as
well as discrepancies between MTCs,
leaves uncertainty regarding which of the
biologics has the greatest treatment effect (BOX 5 and BOX 6).
Author Contributions: Dr Mills had full access to all of
the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Financial Disclosures: All authors have completed and
submitted the ICMJE Form for Disclosure of Potential
Conflicts of Interest. Dr Mills reported consulting for
Merck & Co Inc, Pfizer Ltd, Novartis and Takeda on
MTC issues; receiving grant funding from the Canadian Institutes of Health Research (CIHR) Drug Safety
& Effectiveness Network (DSEN) NETMAN project to
develop methods and educational materials on MTCs;
and receiving salary support from the CIHR through
a Canada Research Chair. Dr Ioannidis reported receiving grant funding from the CIHR DSEN NETMAN
project to develop methods and educational materials on MTCs. Dr Thorlund reported consulting to
Merck & Co Inc, Pfizer Ltd, Novartis, Takeda, or
GlaxoSmithKline on MTC issues; receiving grant funding from the CIHR DSEN NETMAN project to develop methods and educational materials on MTCs;
and receiving salary support from the CIHR DSEN
NETMAN project. Dr Schu¨nemann is a coinvestigator on a grant from the CIHR DSEN NETMAN project
to develop methods and educational materials on
MTCs. Dr Guyatt reported consulting to BristolMyers Squibb and UpToDate Inc on MTC issues and
is a coinvestigator on a grant from the CIHR DSEN
NETMAN project to develop methods and educational materials on MTCs.
Funding/Support: The CIHR, through the DSEN
NETMAN project, provided support for this article.
Role of the Sponsor: The funding agency had no role
in the design or conduct of the study; in the collection, management, analysis, or interpretation of the
data; or in the preparation, review, or approval of the
manuscript. DSEN had no role in the design and conduct of the study; collection, management, analysis,
and interpretation of the data; and preparation, review, or approval of the manuscript.
©2012 American Medical Association. All rights reserved.
USERS’ GUIDES TO THE MEDICAL LITERATURE
Disclaimer: Several examples used to demonstrate concepts in this article originated from consulting projects.
Online-Only Material: The eAppendix is available at
www.jama.com.
Additional Contributions: We thank David Baldwin,
MBBS, DM, FRCPsych (University of Southampton),
for clarifications on his study.
REFERENCES
1. Baldwin D, Woods R, Lawson R, Taylor D. Efficacy of drug treatments for generalised anxiety disorder: systematic review and meta-analysis. BMJ. 2011;
342:d1199.
2. Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not always enough. Lancet. 1998;
351(9096):123-127.
3. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA,
Chalmers TC. Meta-analyses of randomized controlled trials. N Engl J Med. 1987;316(8):450-455.
4. Estellat C, Ravaud P. Lack of head-to-head trials
and fair control arms: randomized controlled trials of
biologic treatment for rheumatoid arthritis. Arch Intern Med. 2012;172(3):237-244.
5. Lathyris DN, Patsopoulos NA, Salanti G, Ioannidis
JP. Industry sponsorship and selection of comparators in randomized clinical trials. Eur J Clin Invest. 2010;
40(2):172-182.
6. Salanti G, Higgins JP, Ades AE, Ioannidis JP. Evaluation of networks of randomized trials. Stat Methods
Med Res. 2008;17(3):279-301.
7. Lu G, Ades AE. Combination of direct and indirect
evidence in mixed treatment comparisons. Stat Med.
2004;23(20):3105-3124.
8. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The
results of direct and indirect treatment comparisons
in meta-analysis of randomized controlled trials. J Clin
Epidemiol. 1997;50(6):683-691.
9. Song F, Xiong T, Parekh-Bhurke S, et al. Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study. BMJ.
2011;343:d4909.
10. Mills EJ, Bansback N, Ghement I, et al. Multiple
treatment comparison meta-analyses: a step forward
into complexity. Clin Epidemiol. 2011;3:193-202.
11. Guyatt G, Straus S, Meade MO, et al. Therapy (randomized trials). In: Guyatt B, Rennie D, Meade MO,
Cook DJ, eds. Users’ Guides to the Medical Literature:
A Manual for Evidence-Based Clinical Practice. 2nd ed.
New York, NY: McGraw-Hill Co; 2008:67-86.
12. Nixon RM, Bansback N, Brennan A. Using mixed
treatment comparisons and meta-regression to perform indirect comparisons to estimate the efficacy of
biologic treatments in rheumatoid arthritis. Stat Med.
2007;26(6):1237-1254.
13. Mills EJ, Wu P, Chong G, et al. Efficacy and safety
of statin treatment for cardiovascular disease: a network meta-analysis of 170,255 patients from 76 randomized trials. QJM. 2011;104(2):109-124.
14. Mills EJ, Wu P, Lockhart I, Thorlund K, Puhan M,
Ebbert JO. Comparisons of high dose and combination nicotine replacement therapy, varenicline and bupropion for smoking cessation: a systematic review and
multiple treatment meta-analysis [published online
ahead of print August 6, 2012]. Ann Med. 2012;44
(6):588-597.
15. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA
statement for reporting systematic reviews and metaanalyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med.
2009;151(4):W65-94.
16. Sutton A, Ades AE, Cooper N, Abrams K. Use of
indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics. 2008;
26(9):753-767.
17. Kyrgiou M, Salanti G, Pavlidis N, Paraskevaidis
E, Ioannidis JP. Survival benefits with diverse chemotherapy regimens for ovarian cancer: meta-analysis of
multiple treatments. J Natl Cancer Inst. 2006;98
(22):1655-1663.
18. Cipriani A, Furukawa TA, Salanti G, et al. Comparative efficacy and acceptability of 12 newgeneration antidepressants: a multiple-treatments
meta-analysis. Lancet. 2009;373(9665):746-758.
19. Turner EH, Matthews AM, Linardatos E, Tell RA,
Rosenthal R. Selective publication of antidepressant
trials and its influence on apparent efficacy. N Engl J
Med. 2008;358(3):252-260.
20. Ioannidis JP. Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized
trials? Philos Ethics Humanit Med. 2008;3:14.
21. Higgins JP, Whitehead A. Borrowing strength from
external trials in a meta-analysis. Stat Med. 1996;
15(24):2733-2749.
22. Ioannidis JP. Ranking antidepressants. Lancet.
2009;373(9677):1759-1760.
23. Gartlehner G, Hansen RA, Morgan LC, et al. Comparative benefits and harms of second-generation antidepressants for treating major depressive disorder:
an updated meta-analysis. Ann Intern Med. 2011;
155(11):772-785.
24. Devereaux PJ, Choi PT, El-Dika S, et al. An observational study found that authors of randomized
controlled trials frequently use concealment of randomization and blinding, despite the failure to report
these methods. J Clin Epidemiol. 2004;57(12):
1232-1236.
25. Sterne JAC, Egger M, Moher D. Addressing reporting biases. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Intervention. Version 5.1.0 (updated March 2011). The
Cochrane Collaboration; 2011.
26. Moher D, Pham B, Jones A, et al. Does quality of
reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet.
1998;352(9128):609-613.
27. Chan AW, Hro´bjartsson A, Haahr MT, Gøtzsche
PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;
291(20):2457-2465.
28. Young NS, Ioannidis JP, Al-Ubaydli O. Why current publication practices may distort science. PLoS
Med. 2008;5(10):e201.
29. Wittchen HU, Kessler RC, Beesdo K, Krause P,
Ho¨fler M, Hoyer J. Generalized anxiety and depression in primary care: prevalence, recognition, and
management. J Clin Psychiatry. 2002;63(Suppl 8):
24-34.
30. Salanti G, Kavvoura FK, Ioannidis JP. Exploring the
geometry of treatment networks. Ann Intern Med.
2008;148(7):544-553.
31. Mills EJ, Ghement I, O’Regan C, Thorlund K. Estimating the power of indirect comparisons: a simulation study. PLoS One. 2011;6(1):e16237.
32. Higgins JP, Thompson SG, Deeks JJ, Altman DG.
Measuring inconsistency in meta-analyses. BMJ. 2003;
327(7414):557-560.
33. Smith GD, Egger M. Going beyond the grand
mean: subgroup analysis in meta-analysis of randomised trials. In: Smith GD, Egger M, Altman DG,
eds. Systematic Reviews in Health Care: Metaanalysis in Context. 2nd ed. London, England: BMJ
Publishing Group; 2001:143-156.
34. Thompson SG, Higgins JP. How should metaregression analyses be undertaken and interpreted?
Stat Med. 2002;21(11):1559-1573.
35. Jansen J, Schmid C, Salanti G. When do indirect
and mixed treatment comparisons result in invalid
findings? a graphical explanation. Poster presented at:
19th Cochrane Colloquium; October 19-22, 2011; Madrid, Spain. P3B379.
36. Song F, Harvey I, Lilford R. Adjusted indirect comparison may be less biased than direct comparison for
evaluating new pharmaceutical interventions. J Clin
Epidemiol. 2008;61(5):455-463.
©2012 American Medical Association. All rights reserved.
37. Lu G, Ades A. Assessing evidence inconsistency
in mixed treatment comparisons. J Am Stat Assoc.
2006;101:447-459.
38. Dias S, Welton NJ, Caldwell DM, Ades AE. Checking consistency in mixed treatment comparison
meta-analysis. Stat Med. 2010;29(7-8):932-944.
39. Zhang WY, Li Wan Po A. Analgesic efficacy of
paracetamol and its combination with codeine and caffeine in surgical pain—a meta-analysis. J Clin Pharm
Ther. 1996;21(4):261-282.
40. Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for presenting results from
multiple-treatment meta-analysis: an overview and
tutorial. J Clin Epidemiol. 2011;64(2):163-171.
41. Golfinopoulos V, Salanti G, Pavlidis N, Ioannidis JP.
Survival and disease-progression benefits with treatment regimens for advanced colorectal cancer: a
meta-analysis. Lancet Oncol. 2007;8(10):898-911.
42. Diels J, Cure S, Gavart S. PIN7 the comparative efficacy of telaprevir versus boceprevir in treatmentnaive and treatment-experienced patients with genotype 1 chronic hepatitis. Value Health. 2011;14(7):
A266-A266.
43. Aaron SD, Fergusson D, Marks GB, et al; Canadian Thoracic Society/Canadian Respiratory Clinical Research Consortium. Counting, analysing and reporting
exacerbations of COPD in randomised controlled trials.
Thorax. 2008;63(2):122-128.
44. Mills EJ, Druyts E, Ghement I, Puhan MA. Pharmacotherapies for chronic obstructive pulmonary disease: a multiple treatment comparison meta-analysis.
Clin Epidemiol. 2011;3:107-129.
45. Sciarretta S, Palano F, Tocci G, Baldini R, Volpe M.
Antihypertensive treatment and development of heart
failure in hypertension: a Bayesian network metaanalysis of studies in patients with hypertension and high
cardiovascular risk. Arch Intern Med. 2011;171(5):
384-394.
46. Psaty BM, Lumley T, Furberg CD, et al. Health outcomes associated with various antihypertensive therapies used as first-line agents: a network meta-analysis.
JAMA. 2003;289(19):2534-2544.
47. Hernandez AV, Walker E, Ioannidis JP, Kattan MW.
Challenges in meta-analysis of randomized clinical
trials for rare harmful cardiovascular events: the
case of rosiglitazone. Am Heart J. 2008;156(1):2330.
48. Ioannidis JP, Evans SJ, Gøtzsche PC, et al; CONSORT
Group. Better reporting of harms in randomized trials:
an extension of the CONSORT statement. Ann Intern
Med. 2004;141(10):781-788.
49. Ford AC, Talley NJ, Spiegel BM, et al. Effect of fibre, antispasmodics, and peppermint oil in the treatment of irritable bowel syndrome: systematic review and
meta-analysis. BMJ. 2008;337:a2313.
50. Schmitz S, Adams R, Walsh CD, Barry M, FitzGerald
O. A mixed treatment comparison of the efficacy of antiTNF agents in rheumatoid arthritis for methotrexate nonresponders demonstrates differences between treatments: a Bayesian approach. Ann Rheum Dis. 2012;
71(2):225-230.
51. Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria to evaluate
the credibility of subgroup analyses. BMJ. 2010;
340:c117.
52. Puhan MA, Bachmann LM, Kleijnen J, Ter Riet G,
Kessels AG. Inhaled drugs to reduce exacerbations in
patients with chronic obstructive pulmonary disease: a
network meta-analysis. BMC Med. 2009;7:2.
53. Mills EJ, Thorlund K, Bansback N. Multiple treatment meta-analysis in rheumatoid arthritis can provide
weak inferences due to methodological approaches.
In: 2012 CADTH Symposium; April 15-17, 2012; Ottawa. OB1.
54. Gartlehner G, Morgan LC, Thieda P, et al. Drug Class
Review: Second Generation Antidepressants: Final Report Update 4. Portland, OR: Oregon Health & Science University; 2008.
JAMA, September 26, 2012—Vol 308, No. 12
1253