Introduction to Meta-Analysis

Data Analysis I

Ahmet H. Kirca, Ph.D.

Associate Professor
Michigan State University

Discussion

1
 In-Class Exercise

Overview of Meta-Analytic Data Analysis

Transformations, Adjustments and Outliers

The Inverse Variance Weight
The Mean Effect Size and Associated Statistics
Homogeneity Analysis
Fixed Effects Analysis of Heterogeneous Distributions
Fixed Effects Analog to the one-way ANOVA
Fixed Effects Regression Analysis
Random Effects Analysis of Heterogeneous
Distributions
Mean Random Effects ES and Associated Statistics
Random Effects Analog to the one-way ANOVA
Random Effects Regression Analysis

4
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

2
 Transformations
Some effect size types are not analyzed in their raw
form; correlation coefficient has a problematic
standard error formula.
Solution: Fishers Zr transformation.

Analyses performed on the Fishers Zr transformed

correlations.
1 r
ES Zr .5 ln
1 r
Finally results can be converted back into r with the
inverse Zr transformation.
e 2 ES Zr 1
r 2 ES Zr
e 1
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Adjustments

Hunter and Schmidt Artifact Adjustments

measurement unreliability (need reliability coefficient)
range restriction (need unrestricted standard deviation)
artificial dichotomization (correlation effect sizes only)
assumes a normal underlying distribution
Outliers
extreme effect sizes may have disproportionate influence on
analysis
either remove them from the analysis or adjust them to a
less extreme value
indicate what you have done in any written report

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

3
 Independent Set of Effect Sizes

Must be dealing with an independent set of effect sizes

before proceeding with the analysis.
One ES per study OR
One ES per subsample within a study

7
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

The Inverse Variance Weight

Studies generally vary in size.

An ES based on 100 subjects
is assumed to be a more
precise estimate of the
population ES than is an ES
1
based on 10 subjects.
se
n3
Therefore, larger studies
should carry more weight in
our analyses than smaller
studies.
Simple approach: weight each
ES by its sample size.
Better approach: weight by the
inverse variance.

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

4
 What is the Inverse Variance Weight?

The standard error (SE) is a direct index of ES precision.

SE is used to create confidence intervals.
The smaller the SE, the more precise the ES.
Hedges showed that the optimal weights for meta-
analysis are:

1
w
SE 2

9
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Inverse Variance Weight for the

Three Major League Effect Sizes

Standardized Mean Difference:

n1 n2 ES sm 1
se w
n1n2 2(n1 n2 ) se 2

Zr transformed Correlation Coefficient:

1
se w n3
n3

10
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

5
 Inverse Variance Weight for the
Three Major League Effect Sizes

Logged Odds-Ratio:

1 1 1 1 1
se w
a b c d se 2

Where a, b, c, and d are the cell frequencies of a 2 by

2 contingency table.

11
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Ready to Analyze!

We have an independent set of effect sizes (ES) that

have been transformed and/or adjusted, if needed.
For each effect size we have an inverse variance weight
(w).

12
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

6
 The Weighted Mean Effect Size

Start with the effect size

Study ES w (ES) and inverse
1 -0.33 11.91
2 0.32 28.57
variance weight (w) for 10
3 0.39 58.82 studies.
4 0.31 29.41

(w ES)
5 0.17 13.89
6 0.64 8.55
7 -0.33 9.80
ES
8
9
10
0.15
-0.02
0.00
10.75
83.33
14.93
w

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

The Weighted Mean Effect Size

Study ES w w*ES Start with the effect size

1 -0.33 11.91 -3.93
2 0.32 28.57 9.14
(ES) and inverse variance
3 0.39 58.82 22.94 weight (w) for 10 studies.
4 0.31 29.41 9.12
5 0.17 13.89 2.36 Next, multiply w by ES.
6 0.64 8.55 5.47
7 -0.33 9.80 -3.24 Repeat for all effect sizes.
8 0.15 10.75 1.61
9 -0.02 83.33 -1.67
10 0.00 14.93 0.00

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

7
 The Weighted Mean Effect Size

Study ES w w*ES Start with the effect size

1 -0.33 11.91 -3.93
2 0.32 28.57 9.14
(ES) and inverse variance
3 0.39 58.82 22.94 weight (w) for 10 studies.
4 0.31 29.41 9.12
5 0.17 13.89 2.36 Next, multiply w by ES.
6 0.64 8.55 5.47
7 -0.33 9.80 -3.24 Repeat for all effect sizes.
8 0.15 10.75 1.61
9 -0.02 83.33 -1.67
Sum the columns, w and
10 0.00 14.93 0.00 ES.
269.96 41.82
Divide the sum of (w*ES)
ES
( w ES ) 41.82 0.15 by the sum of (w).
w 269.96

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

The Weighted Mean Effect Size

Study ES w w*ES Start with the effect size

1 -0.33 11.91 -3.93
2 0.32 28.57 9.14
(ES) and inverse variance
3 0.39 58.82 22.94 weight (w) for 10 studies.
4 0.31 29.41 9.12
5 0.17 13.89 2.36 Next, multiply w by ES.
6 0.64 8.55 5.47
7 -0.33 9.80 -3.24 Repeat for all effect sizes.
8 0.15 10.75 1.61
9 -0.02 83.33 -1.67
Sum the columns, w and
10 0.00 14.93 0.00 ES.
269.96 41.82
Divide the sum of (w*ES)
ES
( w ES ) 41.82 0.15 by the sum of (w).
w 269.96

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

8
 The Standard Error of the Mean ES

Study ES w w*ES The standard error of

1 -0.33 11.91 -3.93
2 0.32 28.57 9.14
the mean is the square
3 0.39 58.82 22.94 root of 1 divided by the
4 0.31 29.41 9.12
5 0.17 13.89 2.36
sum of the weights.
6 0.64 8.55 5.47
7 -0.33 9.80 -3.24
8 0.15 10.75 1.61
9 -0.02 83.33 -1.67 1 1
seES 0.061
10 0.00 14.93
269.96
0.00
41.82 w 269.96

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Mean, Standard Error,

Z-test and Confidence Intervals

Mean ES
ES
( w ES ) 41.82 0.15
w 269.96
SE of the Mean ES
1 1
seES 0.061
w 269.96

Z-test for the Mean ES

ES 0.15
Z 2.46
seES 0.061
95% Confidence Interval
Lower ES 1.96( seES ) 0.15 1.96(.061) 0.03
Upper ES 1.96( seES ) 0.15 1.96(.061) 0.27

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

9
 Example

Homogeneity Analysis

Homogeneity analysis tests whether the assumption

that all of the effect sizes are estimating the same
population mean is a reasonable assumption.
If homogeneity is rejected, the distribution of effect
sizes is assumed to be heterogeneous.
Single mean ES not a good descriptor of the distribution
There are real between study differences, that is, studies
estimate different population mean effect sizes.
Two options:
model between study differences
fit a random effects model

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

10
 Q - The Homogeneity Statistic

Study ES w w*ES w*ES^2 Calculate a new

1 -0.33 11.91 -3.93 1.30
2 0.32 28.57 9.14 2.93
variable that is the ES
3 0.39 58.82 22.94 8.95 squared multiplied by
4 0.31 29.41 9.12 2.83
5 0.17 13.89 2.36 0.40
the weight.
6 0.64 8.55 5.47 3.50 Sum new variable.
7 -0.33 9.80 -3.24 1.07
8 0.15 10.75 1.61 0.24
9 -0.02 83.33 -1.67 0.03
10 0.00 14.93 0.00 0.00
269.96 41.82 21.24

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Calculating Q

We now have 3 sums:

w 269.96
( w ES ) 41.82
( w ES ) 21.24
2

Q is can be calculated using these 3 sums:

Q ( w ES 2 )
w ES 2

21.24
41.82 2
21.24 6.48 14.76
w 269.96

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

11
 Interpreting Q

Q is distributed as a Chi-Square
df = number of ESs - 1
Running example has 10 ESs, therefore, df = 9
Critical Value for a Chi-Square with df = 9 and p = .05
is:
16.92

Since our Calculated Q (14.76) is less than 16.92, we

fail to reject the null hypothesis of homogeneity.
Thus, the variability across effect sizes does not
exceed what would be expected based on sampling
error.

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Discussion

12
Assignment 3

13