STATISTICS IN MEDICINE

Statist. Med. 2002; 21:26412652 (DOI: 10.1002/sim.1221)

A graphical method for exploring heterogeneity

in meta-analyses: application to a meta-analysis of 65 trials

Bertrand Baujat, C
edric Mah
e, Jean-Pierre Pignon; and Catherine Hill

Institut Gustave Roussy; D
epartement de Biostatistique et dEpid
emiologie; 39 rue Camille Desmoulins;
94805 Villejuif cedex; France

SUMMARY
Heterogeneity can be a major component of meta-analyses and by virtue of that fact warrants inves-
tigation. Classic analysis methods, such as meta-regression, are used to explore the sources of hetero-
geneity. However, it may be dicult to apply such a method in complex cases or in the absence of an
a priori hypothesis. This paper presents a graphical method to identify trials, groups of trials or groups
of patients that are sources of heterogeneity. The contribution of these trials to the overall result can
also be evaluated with this method. Each trial is represented by a dot on a 2D graph. The X-axis repre-
sents the contribution of the trial to the overall Cochran Q-test for heterogeneity. The Y-axis represents
the inuence of the trial, dened as the standardized squared dierence between the treatment eects
estimated with and without the trial. This approach has been applied to data from the Meta-Analysis
of Chemotherapy in Head and Neck Cancer (MACH-NC) comprising 10 850 patients in 65 randomized
trials. The graphical method allowed us to identify trials that contributed considerably to the overall
heterogeneity and had a strong inuence on the overall result. It also provided useful information for
the interpretation of heterogeneity in this meta-analysis. The proposed graphical method identies trials
that account for most of the heterogeneity without having to explore all possible sources of heterogene-
ity by subgroup analyses. This method can also be applied to identify types of patients that explain
heterogeneity in the treatment eect. Copyright ? 2002 John Wiley & Sons, Ltd.

KEY WORDS: meta-analysis; heterogeneity; graphical method; meta-regression; individual patient

data; head and neck cancer

1. INTRODUCTION

Meta-analysis is a method used to combine the results of dierent trials in order to ob-
tain a quantied synthesis. The size of individual clinical trials is often too small to detect

Correspondence to: Jean Pierre Pignon, Institut Gustave Roussy, D
epartement de Biostatistique et dEpid
emiologie,
39 rue Camille Desmoulins, 94805 Villejuif cedex, France
E-mail: jppignon@igr.fr

Contract=grant sponsor: Fondation pour la Recherche M
edicale

Contract=grant sponsor: Banque Nationale de Paris
Contract=grant sponsor: Smithkline and Beecham
Contract=grant sponsor: European Commission; contract=grant number: 961321
Contract=grant sponsor: Programme Hospitalier de Recherche Clinique; contract=grant number: IDF 95013

Received July 1999

Copyright ? 2002 John Wiley & Sons, Ltd. Accepted October 2001
2642 B. BAUJAT ET AL.

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
 GRAPHICAL METHOD FOR EXPLORING HETEROGENEITY IN META-ANALYSES 2643

treatment eects reliably, especially in oncology. Meta-analysis is a way to increase the power
of statistical analyses by pooling the results of all available trials. This can be done by using
summarized results of published trials or individual patient data obtained from all existing
trials (published or not). Whenever the results of the trials are heterogeneous, the interpre-
tation of the overall result may be dicult, especially if the dierences between trials are
not readily ascribable to variation in population characteristics. In this case, the sources of
heterogeneity should be investigated. The easiest way to explore heterogeneity consists of
studying interaction between treatment eect and groups of trials or patients through tests
for heterogeneity, but there is the risk of nding signicant results by chance with such ex-
ploratory analyses that involve repeated tests. Meta-regression techniques are also commonly
used. These methods, combined with the plot of the treatment eect by trial (Figure 1), solve
most of the diculties. However, both methods lack power, especially in complex cases.
In order to take into account the multiple sources of heterogeneity in an individual patient
data-based meta-analysis, we propose a graphical method for detecting the sources of hetero-
geneity and evaluating the contribution of these sources to the overall result. The descriptive
method we propose widens the scope of the investigations to encompass the consequences of
heterogeneity on the overall result and can be applied to both types of meta-analysis, that is,
those based on individual patient data or on published data. This method can be used alone
or in addition to meta-regression analysis.
In this paper, we rst recall the simple classic approaches used to detect heterogeneity.
Then, the graphical method is presented and illustrated by applying it to a large meta-
analysis [1]. Finally, the interest of this graphical method within the meta-analysis strategy is
discussed.

2. CLASSIC APPROACHES USED TO DETECT HETEROGENEITY

Ideally, the treatment eect should be homogeneous between trials, and the estimated treatment
eect of each trial
i should have a normal N(
; i2 ) distribution where
is the true treatment
eect and i2 is the sampling variance within the ith study (i =1; : : : ; k). This assumption does
not always reect reality. A test for homogeneity is widely used to verify this assumption
[2]. This test is based on the squared dierence between the estimated treatment eect in trial
i (
i ) and the overall estimated treatment eect (
)
weighted by the inverse of the estimated
variance of the treatment eect in trial i: 1= i .
2

(
i
)
2
Q= (1)
i2

Figure 1. Hazard ratio of death with loco-regional treatment plus chemotherapy (CT) versus loco-
regional treatment. The centre of each square represents the hazard ratio (HR) for individual trials and
the corresponding horizontal line its 95 per cent condence interval (CI). The area of the square is
proportional to the amount of information obtained from the trial. The broken line and the centre of
the black diamond represent the overall pooled HR and the extremities of the diamond represent its 95
per cent CI. The open diamonds represent the HR of the dierent categories representing the timing
of chemotherapy. Trials are ranked chronologically from the date the trial was started with the oldest
appearing rst. In the Risk Redn. (reduction), G means I .

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
2644 B. BAUJAT ET AL.

where
k

k

i
1

=

i=1  i=1 
i2 i2
Under the null hypothesis of homogeneity, this statistic follows a chi-square distribution with
k 1 degrees of freedom (d.f.). When the value of the Q-statistic is too high (that is, overall,
the trial eects are too far from the mean eect, taking into account the sampling variance),
the null hypothesis of homogeneity is rejected. This test for heterogeneity is considered to
have low power [3]. Hardy and Thompson [4] showed that the number of studies does not
have a substantial eect on this power, and that a reduction in the power of the Q-statistic
test for heterogeneity is observed particularly when the total information, namely the number
of patients in the meta-analysis is small, or when the variance of the treatment eect diers
considerably between trials, which is the case most of the time. Moreover, this statistic pro-
vides a test to detect heterogeneity in toto but does not identify which trials are the main
sources of heterogeneity. In order to identify this contribution to heterogeneity, Galbraith [5]
proposed a bivariate radial scatter plot of y =1=  i against x =
i =  i , with one point plotted
for each trial, where the points for trials with the same treatment eect will be on a same
line through the origin, and where the outlying trials will be easily visualized beyond the
approximated 95 per cent condence limits (Figure 2). Points with a large standard error
 i will fall near the origin while points with a small standard error  i will fall at a distance
from the origin.

3. GRAPHICAL METHOD

The purpose of the method is to present, on the same graph, the heterogeneity of the treatment
eects between subgroups of patients and the inuence each subgroup of patients exerts on the
overall result. Each subgroup of patients is represented by a dot on a 2D graph and the most
heterogeneous and inuential subgroup of patients will appear in the upper right corner of
the graph. To simplify matters, we will describe the method when each subgroup of patients
considered is a trial but it can also be applied to subgroups of patients (or even subgroups
of trials) dened according to their characteristics. The X-axis represents the contribution of
the trial to the Cochran Q-test for heterogeneity (1), that is, the squared dierence between
the treatment eect of the trial under consideration and the mean treatment eect, weighted
by the inverse of the estimated variance of the treatment eect in the trial:

(
i
) 2
x i = (2)
i2

Each x i statistic is approximately distributed as a 12 (even though we used estimates for the
overall treatment eect and for the variance) and a value of this statistic greater than the
chi-square quantile associated with type I error 0.05 (3.84) can therefore be considered as
outlying. However, since the purpose of the method is only to visualize the most heterogeneous
and inuential trials in relation to the others, no cut-o should be set a priori.
The Y-axis represents the inuence of the trial on the overall treatment eect, dened
as the square of the dierence between the treatment eect estimated with (
) and without

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
 GRAPHICAL METHOD FOR EXPLORING HETEROGENEITY IN META-ANALYSES 2645

Figure 2. Galbraiths radial plot of the odds ratio applied to data from the MACH-NC. Each trial is
plotted as a dot. Heterogeneous trials are visualized beyond the lines representing the approximated
95 per cent condence limits. Twelve (numbers 36; 38; 39; 45; 48; 52; 58; 70; 73; 91; 104; 108) out of 65
trials appear to be sources of heterogeneity. The numbers correspond to the trial reference number in the
MACH-NC [1] database (see Figure 1).

trial i (
i ), weighted by the inverse of the estimated variance of the treatment eect after
exclusion of the ith trial: 1=var(
i )=1=i
2
.

(
i
) 2
yi = (3)
 i
2

with



j
1

i =
j
=i 
j2 j
=i j2

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
2646 B. BAUJAT ET AL.

Exclusion of an outlying, inuential trial strongly modies the treatment eect
i , and
therefore increases the numerator of the statistic. Exclusion of a large trial would lead to
a larger increase in the overall variance than the exclusion of a small trial. Unlike the
X-axis in Galbraiths method, this axis takes into account data from the other trials in
order to plot the inuence of the trial under consideration: the information provided is
not only the weight of the trial but also the contribution of the trial to the overall
result.
A working example is provided in the Appendix, using the MantelHaenszelPeto
method [6].

4. APPLICATION

The Meta-Analysis of Chemotherapy in Head and Neck Cancers (MACH-NC) [1] included
10 850 patients randomized in 65 prospective trials comparing loco-regional treatment to the
same loco-regional treatment + chemotherapy. The size of the trials ranged from 27 to 680
patients. This meta-analysis was based on individual patient data. The overall hazard ratio of
death (HR) is 0.90 (95 per cent condence interval [0:85; 0:94]) demonstrating a signicant
benecial eect of chemotherapy on survival. Results are summarized in Figure 1. This graph
shows a larger benecial eect of chemotherapy when given concomitantly with radiother-
apy. The Cochran Q-test for heterogeneity shows signicant overall heterogeneity between
trials (Q =118, 64 degrees of freedom, p0:001). As this test is conservative, a signicant
value must be considered as strong evidence of heterogeneity. The Q-value can be used as a
reference value when we want to quantify overall heterogeneity. However, as this test lacks
power, non-signicance should not be considered as a proof of homogeneity. Figure 2 shows
the results of the MACH-NC using the Galbraith plot [5].

4.1. Trials
When applied to the trials, the graphical method shows that ve trials, located in the upper
right corner of the graph (numbers 45; 48; 73; 91; 104), contribute to 40 per cent of the statistic
of overall heterogeneity, and to 38 per cent of the sum of the inuences (Figure 3). Four
of these trials (80 per cent) are concomitant trials whereas concomitant trials represent only
40 per cent of the trials in the MACH-NC. As a sensitivity analysis, exclusion of the 811
patients from these ve trials would strongly reduce the heterogeneity (Q =71 with 59 d.f.,
p=0:14) which would become non-signicant. Moreover, there would be only a very slight
modication in the benet of chemotherapy (HR =0:91; 95 per cent CI [0:87; 0:96]). Trial 58
appears to be a source of heterogeneity but it is not very inuential because it is small. In
this trial, which included only 27 patients, all the patients in the control group died within a
year of follow-up, a very dismal result compared to the 32 per cent survival rate at 5 years
in the rest of the MACH-NC; this trial was conducted in the same centre as trial 48. The
hazard ratio of death with and without chemotherapy is 0.26 in this trial (95 per cent CI
[0:11; 0:63]), which is way below the overall 0.90. Although such is not the case here, the
graph could be more informative if a dierent colour or symbol was attributed to each trial
(for example according to the timing of chemotherapy).

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
 GRAPHICAL METHOD FOR EXPLORING HETEROGENEITY IN META-ANALYSES 2647

Figure 3. The graphical method applied to trials. Each trial is plotted as a number corresponding to
the trial reference number in the MACH-NC [1] database (see Figure 1). The X-axis represents the
contribution xi of the trial to the overall heterogeneity. The Y-axis represents the inuence yi of the
trial on the overall result. The most heterogeneous and inuential trials appear in the upper right area
of the graph. Y scale = yi 100.

4.2. Trial characteristics

The graphical method can also be applied to categories of trials grouped according to their
characteristics. Data concerning the trial characteristics collected in the MACH-NC included:
the timing of chemotherapy (adjuvant, that is, after loco-regional treatment, neoadjuvant, that
is, before loco-regional treatment, and concomitant, that is, given concomitantly or alternating
with radiotherapy); the type of chemotherapy grouped into four pre-established categories
(5-uorouracil (5FU) and platinum, multi-agent chemotherapy with platinum and without 5FU,
multi-agent chemotherapy without platinum, single-agent chemotherapy); the year of the rst
randomization grouped into three categories with roughly an equal number of trials in each
category (before 1980, between 1980 and 1985, after 1985); nally, the size of the trial
grouped into three categories with roughly an equal number of trials in each category (under
80, between 80 and 160, above 160 patients).
In Figure 4, the concomitant group (26 trials) appears to be a major source of inuential het-
erogeneity. The hazard ratio in this subgroup, if the four heterogeneous trials detected on Fig-
ure 3 are excluded, becomes 0.88 (95 per cent CI [0:81; 0:95]) instead of 0.81 (95 per cent CI
[0:76; 0:88]) before exclusion. This HR =0:88 is not very dierent from the overall HR =0:91
(95 per cent CI [0:87; 0:96]), which suggests that the four trials are responsible for a major
part of the heterogeneity of the results ascribed to the 26 concomitant chemotherapy trials.

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
2648 B. BAUJAT ET AL.

Figure 4. The graphical method applied to four trial characteristics with exclusive categories
in the MACH-NC database: date of start of the trials; size of the trials; timing and type
of chemotherapy. Each subgroup of trials is plotted as the abbreviated name of its cate-
gory. PolyCh + P = multi-agent chemotherapy including platinum; 5FU + P = 5-uorouracil + platinum;
S = size of the trial ranging from 80 to 160 patients; P = multi-agent chemotherapy without platinum;

Monochemotherapy, year of the trial before 1980.

The 13 trials involving multi-agent chemotherapy with platinum appear to be a source of

heterogeneity on the graph. Indeed, this subgroup of trials shows a non-signicantly increased
risk of death associated with chemotherapy compared to that of the other three chemotherapy
groups (HR =1:05, 95 per cent CI [0:94; 1:17]). However, the inuence of this subgroup on
the overall result is limited because the number of patients involved (16 per cent of the total
population) and the dierence in the hazard ratio (1.05 versus 0.90) are small.
Linear meta-regression analyses, regressing the eect of chemotherapy with the characteris-
tics of each trial, showed that three variables exerted a signicant eect chemotherapy ad-
ministered concomitantly with radiotherapy (yes=no) (p=0:04), the year of rst randomization
in the 19801985 period (yes=no) (p=0:03), the trial size in the three categories (p=0:04)
and a borderline eect of multi-agent chemotherapy with platinum (yes=no) (p=0:09). The
treatment benet was smaller in trials which started in the 19801985 period and decreased
with the size of the trial. Meta-regression measures heterogeneity between categories of trials
and ignores the inuence of the subgroups on the overall result.

4.3. Individual characteristics

As individual data are available in the MACH-NC, exploration of heterogeneity can be ex-
tended to individual covariates (Figure 5). The objective is no longer to explore interstudy
variation but rather to try to explain variations in the treatment eect between subgroups of

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
 GRAPHICAL METHOD FOR EXPLORING HETEROGENEITY IN META-ANALYSES 2649

Figure 5. The graphical method applied to four patient covariates in the MACH-NC database: sex; age;
tumour site, and stage. Each subgroup of patients is plotted as the abbreviated name of its category.

Oral cavity, oropharynx, hypopharynx, larynx, stage 1.

patients (distributed among the trials) according to their individual characteristics. The indi-
vidual covariates available were age in years grouped into three categories according to the
quantile (younger than 52, between 52 and 62, older than 62), sex, tumour stage grouped
into ve categories (oral cavity, oropharynx, larynx, hypopharynx, other sites), and tumour
stage grouped into four categories (1 to 4). The graph shows that no category of patients is
both heterogeneous and inuential; patients with stage 4 disease and under 52 years of age
appear to inuence the overall result while patients older than 62 and patients in the category
other sites appear to contribute to the heterogeneity. These ndings support the fact that, in
large meta-analyses like the MACH-NC, adjustment on individual covariates does not usually
modify the overall results and does not diminish overall heterogeneity.

5. DISCUSSION

In this paper, we have presented a graphical tool for meta-analysis, which can both detect
the sources of heterogeneity and evaluate their contribution to the overall result. The method
has been described in the context of a meta-analysis of trials but it can also be used for
meta-analyses of epidemiological studies. The statistics are easy to compute and no specic
software is required for the graph. This graphical tool can be used both for individual pa-
tient data-based meta-analyses and for literature-based meta-analysis. This tool can identify a
wide range of sources of heterogeneity: between trials; between groups of trials, and between
patient characteristics (for individual patient data-based meta-analysis). When an adjustment

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
 2650

Table A1. Teaching example of the method applied to a meta-analysis of six trials.
2
Trial Oi Ei var(Oi Ei ) = 1= i2 1=  i Trial treatment eect Overall treatment eect Graphical method
number after exclusion of the trial Measure of Measure of

Copyright ? 2002 John Wiley & Sons, Ltd.

heterogeneity inuence
O i Ei i=j (Oj Ej )

i =
i =
xi yi
var(Oi Ei ) i=j var(Oj Ej )

1 2 3 17 0.67 0.03 0.88 0.16

2 0.5 5 15 0.10 0.13 0.00 0.00
3 2 2 18 1:00 0.25 2.53 0.28
4 1:5 5 15 0:30 0.27 0.90 0.30
B. BAUJAT ET AL.

5 5 2 18 2.50 0:14 11.28 1.25

6 1:5 3 17 0:50 0.24 1.17 0.21
Total 2.5 20
= 0:125 Q = 16:76

Statist. Med. 2002; 21:26412652

 GRAPHICAL METHOD FOR EXPLORING HETEROGENEITY IN META-ANALYSES 2651

on individual covariates is required, the formula used for the graphical method can be eas-
ily generalized by using the adjusted regression coecient (and its standard deviation) to
estimate the treatment eect. Compared to classic graphical methods, this approach widens
the scope of investigations exploring the consequences of heterogeneity on the overall result.
Meta-regression is also a useful systematic method for investigating heterogeneity, and un-
like our graphical method, can model continuous covariates. Multi-dimensional analyses can
be performed with this method, but it mainly focuses on the detection of heterogeneity be-
tween groups of trials and does not provide information on the contribution of the subgroup
considered to the overall result.
Whatever the method used, the analysis of heterogeneity should often be a post hoc
exploratory analysis for a better understanding of the data. It is better to specify in advance
which subgroups will be considered, but it is not always possible. This analysis should never
be used to exclude trials from the analysis on the basis of their contribution to heterogeneity.
Exclusion of trials or of groups of trials should only be performed in sensitivity analyses
aimed at identifying trials responsible for the heterogeneity, and at assessing their inuence
on the overall result.
Since this graphical method aims at providing a descriptive approach to study heterogeneity,
the xed eect model appears to be the most appropriate. A random eect model, for instance
the Der Simonian and Laird method [7], can also be used but the graphical method becomes
less informative, since the plots a priori close in on the mean.
Another interesting approach for individual patient data-based meta-analyses would consist
of using the statistical methods developed for multivariate failure time data such as frailty
models. The failure times in one and the same trial can be considered to be correlated due
to an assumed common baseline hazard or unobserved characteristics and this correlation can
be modelled using a random eect model [810]. This approach would provide an interesting
alternative to the present models, by including heterogeneity in the model rather than trying
to isolate it.

APPENDIX

Table A1 shows a teaching example of the method applied to a meta-analysis of six trials.

ACKNOWLEDGEMENTS
We thank the Fondation pour la Recherche M
edicale, the Banque Nationale de Paris, and Smithkline
and Beecham for their nancial support. We also thank the MACH-NC Collaborative Group for the data
set and the institutions who supported the study: European Commission (Biomed 2 number 961321) and
Programme Hospitalier de Recherche Clinique (number IDF 95013). We are indebted to Luc Duchateau
for sharing software to compute the Galbraith graph and to Lorna Saint Ange for editing. Lastly, we
thank the referees for their very fruitful comments.

REFERENCES
1. Pignon JP, Bourhis J, Domenge C, Design
e L and the MACH-NC collaborative group. Chemotherapy added to
locoregional treatment for head and neck squamous cell carcinoma: result of three meta-analyses using updated
individual data. Lancet 2000; 355:949 955.
2. Cochran WG. The combination of estimates from dierent experiments. Biometrics 1954; 10:101 129.

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652
2652 B. BAUJAT ET AL.

3. Berlin JA, Laird NM, Sacks HS, Chalmers TC. A comparison of statistical methods for combining events rates
from clinical trials. Statistics in Medicine 1989; 8:154 176.
4. Hardy RJ, Thompson SG. Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine 1998;
17:841 856.
5. Galbraith RF. A note on graphical presentation of estimated odds ratios from several clinical trials. Statistics
in Medicine 1988; 7:889 894.
6. Early Breast Cancer Trialists Collaborative Group. Treatment of early breast cancer, Vol. 1. Worldwide evidence
19851990. Oxford University Press: Oxford, 1990.
7. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986; 7:177 188.
8. Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random eects. Statistics in Medicine
1996; 15:619 629.
9. Biggersta BJ, Tweedie RL. Incorporating variability in estimates of heterogeneity in the random eects model
in meta-analysis. Statistics in Medicine 1997; 16:753 768.
10. Sargent DJ, Zee BC, Milan C, Torri V, Francini G. Meta-analysis of individual-patient survival data using
random-eect models. In Meta-Analysis in Medicine and Health Policy, Stangl DK, Berry DA (eds). Marcel
Dekker Inc: New York, 2000; 255 275.

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:26412652