Introduction to Meta-Analysis

How to Collect Data for a Meta-Analysis?

Ahmet H. Kirca, Ph.D.

Associate Professor
Michigan State University

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Data Collection = Coding (mostly)

So, What Are We Coding?

Effect Size (outcome variable)

Sample characteristics (e.g., industry, country,
size, age of the firm) (explanatory variables)
Study characteristics (e.g., authors, journal,
sample size, methods, procedures, designs)
(explanatory variables)

1
 The Effect Size

The effect size (ES) makes meta-analysis

possible
The ES encodes the selected research findings
on a numeric scale
There are many different types of ES measures,
each suited to different research situations
Each ES type may also have multiple methods
of computation

3
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Different Types of Effect Sizes

Standardized mean difference

Group contrast research
Treatment groups
Naturally occurring groups
Inherently continuous construct
Odds-ratio
Group contrast research
Treatment groups
Naturally occurring groups
Inherently dichotomous construct
Correlation coefficient
Association between variables research 4
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

2
 Different Types of Effect Sizes

Risk ratio
Group differences research (naturally occurring
groups)
Commonly used by epidemiologist and medical meta-
analyses
Inherently dichotomous construct
Easier to interpret than the odds-ratio

5
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Different Types of Effect Sizes

Proportion
Central tendency research
HIV/AIDS prevalence rates
Proportion of homeless persons found to be alcohol
abusers
Standardized gain score
Gain or change between two measurement points
on the same variable
Reading speed before and after a reading improvement
class

6
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

3
 The Standardized Mean Difference

X G1 X G 2 s12 n1 1 s22 n2 1
ES s pooled
s pooled n1 n2 2

Represents a standardized group contrast on an

inherently continuous measure
Uses the pooled standard deviation (some
situations use control group standard deviation)
Commonly called d or occasionally g

7
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

The Correlation Coefficient

ES r

Represents the strength of association between

two inherently continuous measures
Generally reported directly as r (the Pearson
product moment coefficient)

8
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

4
 The Odds-Ratio
The odds-ratio is based on a 2 by 2
contingency table, such as the one below
Frequencies
Success Failure ad
Treatment Group a b ES
bc
Control Group c d

The Odds-Ratio is the odds of success in the

treatment group relative to the odds of success in the
control group.

9
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

The Risk Ratio

The risk ratio is also based on data from a 2 by 2

contingency table, and is the ratio of the
probability of success (or failure) for each group
a / ( a b)
ES
c / (c d )

10
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

5
 Formulas for the Correlation Coefficient

Results typically reported directly as a

correlation
Any data for which you can calculate a
standardized mean difference effect size, you
can also calculate a correlation type effect size
See appendix B for formulas

11
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Meta-Analysis Stuff Website

http://mason.gmu.edu/~dwilsonb/ma.html

Check out:
The online effect size calculator
The old version of calculator (excel file)
Other macros

6
 13
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

14
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

7
 Coding Forms and Coding Manual

Paper Coding
See Appendix E from Lipsey and Wilson (2001)
See Multinationality Project Coding Manual and Coding Form
Coding Directly into a Computer Database

15
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Example

8
 Example Screen from a Computerized
Database for Direct Coding

Figure 5.11: Example FileMaker Pro Screen for Data Entry from the Challenge
Meta-Analysis

Database Structure Overheads 17

Coding Directly into a Computer Database

Advantages
Avoids additional step of transferring data from paper to
computer.
Easy access to data for data cleanup.
Data base can perform calculations during coding process
(e.g., calculation of effect sizes).
Faster coding.
Disadvantages
Can be time consuming to set up.
the bigger the meta-analysis the bigger the payoff
Requires higher level of computer skill.

Database Structure Overheads 18

9
 The Hierarchical Nature of Meta-Analytic Data

Meta-analytic data is inherently hierarchical

Multiple outcomes per study
Multiple measurement points per study
Multiple sub-samples per study
Results in multiple effect sizes per study
Any specific analysis can only include one effect size
per study (or one effect size per sub-sample within a
study)
Analyses almost always are of a subset of coded
effect sizes. Data structure needs to allow for the
selection and creation of those subset.

19
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage

Example

10
Recommendations

Prepare a Coding Manual that explains how studies identified in

the previous stage will be evaluated
Prepare a Coding Form for each study according to the Coding
Manual
Use multiple coders
Directly enter the data to an excel sheet
Do not include studies that use multivariate regression
coefficients unless you follow the appropriate procedures
Do not discard qualitative data vote counting can still be useful
Use visuals, tables, and numbers to describe the data collection
process in the paper

Discussion

11
Assignment 2

12