Application of Meta-Analysis

in Research

Anuraga Jayanegara
Faculty of Animal Science
Bogor Agricultural University
 Short CV:
• Anuraga Jayanegara
• Bojonegoro, 2 June 1983
• Married (27 July 2005; 22 years old; 1 wife & 5 children)
• 2003: BSc (Bogor Agric. Univ., Indonesia)
• 2008: MSc (Univ. Hohenheim, Germany)
• 2010: PgDip (Polytech. Univ. Catalunya, Spain)
• 2011: PhD (Swiss Fed. Inst. Tech. Zurich, Switzerland)

International teaching experience:

• ETH Zurich, Switzerland (2010-2011): Tropical Animal Nutrition
• Ghent University, Belgium (2013): Dairy Nutrition
• Mie University, Japan (2015-Present): Feed Science
• Hiroshima University, Japan (2018): Animal Nutrition
Department of Nutrition and Feed Technology
 Author:
• SCOPUS: articles 62, citations 578, h-index 13
• Google Scholar: citations 1230, h-index 19

Reviewer:
• 13 international journals, IF 0.4-3.4, Q1-Q4

Editor:
• Asian-Australasian Journal of Animal Sciences (SCOPUS Q1)
• Frontiers in Veterinary Science (SCOPUS Q1)
• Tropical Animal Science Journal (SCOPUS Q4)
• Journal of the Indonesian Tropical Animal Agriculture (SCOPUS Q4)

Department of Nutrition and Feed Technology

Background

• Tremendous increase in the number of

publications
• Increasing number of quantitative
measurements
• Quality of data varies from study to study
• Research stakeholders increasingly want more
quantitative knowledge and of better precision
• Forecasting and decision require quantitative
information

Department of Nutrition and Feed Technology

Department of Nutrition and Feed Technology
 • Research synthesis = review of primary research on
a given topic with a purpose of integrating the
findings (creating generalizations, conflict resolution)

• Systematic review = the type of research synthesis

on a precisely defined topic using systematic and
explicit methods to identify, select, critically appraise
and analyze relevant research

• Meta-analysis = statistical synthesis of the results of

separate studies (quantitative research synthesis)

Department of Nutrition and Feed Technology

Relationship between research synthesis,
systematic review and MA

Systematic Meta-
review analysis

Research synthesis

Department of Nutrition and Feed Technology

 What is wrong with narrative reviews?
• No strict criteria for selection of studies for review or
for judging study quality
– high degree of subjectivity
– low repeatability
• Low efficiency in handling a large number of studies
• Limited ability to deal with variation in study
outcomes
– the results of studies are often found to be
”inconsistent”, ”inconclusive” or ”conflicting”
– little help in conflict resolution and decision
making
Department of Nutrition and Feed Technology
 Advantages of meta-analysis

• More informative (”how much” instead of

”yes” or ”no”)
• More accurate (accounts for unequal error
variances among studies)

Department of Nutrition and Feed Technology

 Number of publications on MA in
ecological journals
140
Number of publications

120

100

80

60

40

20

0
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Year of publication

Department of Nutrition and Feed Technology

 Why to learn about meta-analysis?
• It provides more objective, informative and powerful way of
summarizing the results from individual studies as
compared to narrative/qualitative reviews
• It is getting more and more common in science
• Unlike narrative reviews which are usually invited
contributions, MA papers can be submitted to any journal
• It is good for your CV: review articles are in general cited
more often than primary research studies
• Some form of research synthesis should precede any
scientific work
• It helps to understand in what form data in primary studies
should be presented
• It changes the way you read and evaluate primary studies

Department of Nutrition and Feed Technology

 Procedure of meta-analysis

• Transformation of data or test statistics from individual

studies into a ’common currency’ (effect size)
• Combining effect sizes from individual studies into a
common estimate of the magnitude of the effect
• Estimating the significance of overall effect
• Estimating the statistical homogeneity of the overall
effect size
• Subdividing studies into groups and searching for
moderators

Department of Nutrition and Feed Technology

 Types of primary data

• Comparison of two groups (e.g. control and

experimental) in terms of continuous
response variables (means, sample sizes
and measures of variance – SD or SE)
• Relationship between two continuous
variables

Department of Nutrition and Feed Technology

 Effect size

• Hedges’ d

• Response ratio

• Original data (ANOVA-based MA)

Department of Nutrition and Feed Technology

Standardized mean difference
between means (Hedges’ d)

( Xe  Xc)
d J
s
Xe – mean of the experimental group
Xc – mean of the control group
s – pooled standard deviation
J – correction term that removes small-sample-size bias

Department of Nutrition and Feed Technology

 Pooled standard deviation:

( N e  1) s  ( N c  1) s 2 2
s e c
Ne  Nc  2

Correction term:
3
J  1
4( N e  N c  2)  1
Department of Nutrition and Feed Technology
Standardized difference between means

Department of Nutrition and Feed Technology

 The variance of Hedges’ d

Ne  Nc d 2
d  
Ne Nc 2( N e  N c )

Ne – sample size of the experimental group

Nc – sample size of the control group

Department of Nutrition and Feed Technology

 Hedges’ d
• Advantages:
– Works well for small sample sizes (N=5-10)
– Scale-free
• Problems:
– Interpretation of the magnitude of the effect
– Difference in d may reflect either differences in
the magnitude of the effect or in variance
among studies
– Some data needed for calculation of d (most
commonly SD or sample sizes) are often
missing
Department of Nutrition and Feed Technology
 Interpretation of magnitude of d

• Cohen’s benchmarks:
– |d | = 0.2 – small effects
– |d | = 0.5 – moderate effects
– |d | = 0.8 – large effects

Department of Nutrition and Feed Technology

 Response ratio

 Xe 
ln R  ln    ln( X e )  ln( X c )
 Xc 
Variance:

 ln R 
se  2

sc 2

2 2
Ne ( X e ) Nc ( X c )

Department of Nutrition and Feed Technology

 Advantages of response ratio

• Easily interpretable
• Results of primary studies are often
presented in the form of response rations
• Effect sizes are not affected by different
variance in control and experimental groups
• SD/SE are not needed for calculation of the
effect size (but needed to calculate variance)

Department of Nutrition and Feed Technology

 Pearson’s correlation coefficient (r)

• easy to interpret
– varies from –1 to +1
– Cohen’s ”rules-of-thumb”:
• | r | = 0.10 – small
• | r | = 0.25 – medium
• | r | = 0.40 - large
– coefficient of determination (r2)
– r2= % of variance explained

Department of Nutrition and Feed Technology

 • Could be converted into a d index:

2r
d
1 r 2

d
r
d 4 2

Department of Nutrition and Feed Technology

Combining effect sizes across studies
• In meta-analysis, effect sizes are usually
weighed by the inverse of the sampling
variance: w=1/v
• Weighing has two purposes: 1) it increases the
precision of the combined estimates and
increases the power of tests, and 2) it makes
certain statistics to have simpler sampling
distribution
• Weighing results in larger studies (large N)
contributing more to the overall magnitude of
the effect size than smaller studies
Department of Nutrition and Feed Technology
Combining effect sizes across studies

• The cumulative effect size: n

w E i i
E  i 1
n

w i 1
i
• The variance of E: 1
  2
E n

w
i 1
i

• The 95% confidence interval around E:

• 95% CI = E  1.96  
Department of Nutrition and Feed Technology
Identification of moderator variables
• Categorical moderators
– types of study organisms
– types of treatments
– types of experiments
– types of response variables
• Continuous moderators
– duration of the experiment
– intensity of treatment
– study location (latitude or altitude)
– year of publication
Department of Nutrition and Feed Technology
ANOVA-like approach for
categorical moderators

• Studies are subdivided into several groups, total

homogeneity then can be partitioned into within-
and between-group homogeneity: Qt = Qb + Qw
• The aim is to find the factor which produces
significant Qb and non-significant Qw
• If the variation within groups still persists, that
may indicate that other factors may exist. Then
the groups are subdivided further and so on.

Department of Nutrition and Feed Technology

 Continuous moderators

• weighted least square regression analysis

• cumulative meta-analysis

Department of Nutrition and Feed Technology

Basic Concept of Cumulative Meta-Analysis
Studies ordered
chronologically or
by covariates

Study 1

Study 2 Pool Studies 1 to 2 Cumulative M-A 1

Study 3 Pool Studies 1 to 3 Cumulative M-A 2

Study 4 Pool Studies 1 to 4 Cumulative M-A 3

Study n-1 Pool Studies 1 to n-1 Cumulative M-A n-2

Study n Pool Studies 1 to n Cumulative M-A n-1

Department of Nutrition and Feed Technology

Department of Nutrition and Feed Technology
Department of Nutrition and Feed Technology
Department of Nutrition and Feed Technology
 Stages of meta-analysis

1. Question formulation stage

2. Data collection stage
3. Data evaluation stage
4. Analysis and interpretation stage
5. Public presentation stage

Department of Nutrition and Feed Technology

 When is meta-analysis most useful?

• There is a moderate to large amount of

empirical work available
• The results are variable across studies
• The expected magnitude of the effect is
relatively weak
• The sample sizes of individual studies are
limited for some reason

Department of Nutrition and Feed Technology

 Question formulation

• Sufficient primary research on the topic must

exist

• Conceptual and operational definitions of

variables

Department of Nutrition and Feed Technology

 Data collection stage

• Previous reviews on the topic

• Lists of references in retrieved studies
• Reference databases
– keyword searches
– cited references searches
• Hand search
• Informal channels

Department of Nutrition and Feed Technology

 Examples of databases

• Scopus
• ISI current contents
• Google scholar
• EBSCO
• Science Direct

Department of Nutrition and Feed Technology

 Literature search and study selection
Keyword search returns (~5,000 papers)
into unfiltered reference library

Examine title
and abstract

Possibly relevant
Obviously irrelevant

Filtered reference library

~ 800-1,000 papers

Examine
full text

Relevant Irrelevant
Irrelevant
Accepted reference library reference
~ 30-80 papers library

Department of Nutrition and Feed Technology

 Retrieval of data from primary studies

• choosing a metric of effect size

• obtaining effect sizes or data needed for their
calculation from text, tables or graphs
– enlarging graphs
– scanning and digitizing graphs (ImageJ, DataThief)
• calculating effect sizes from raw data
• obtaining effect sizes from test statistics
• converting one measure of effect size into another
• converting other measures of variance into SD

Department of Nutrition and Feed Technology

Converting other measures of variance into SD

From SE:

SD  SE N
95% CI:

(CI u  CI l ) N
SD 
2 1.96
Department of Nutrition and Feed Technology
 Potential problems in meta-analysis

• Missing data
• Publication and dissemination bias
• Varying research quality
• Non-independence among comparisons

Department of Nutrition and Feed Technology

 ANOVA-based meta-analysis

Statistical models

Department of Nutrition and Feed Technology

Department of Nutrition and Feed Technology
Department of Nutrition and Feed Technology
Fixed or random effects?

Department of Nutrition and Feed Technology

Examples of meta-analyses

Department of Nutrition and Feed Technology

Department of Nutrition and Feed Technology
Department of Nutrition and Feed Technology
Department of Nutrition and Feed Technology
Department of Nutrition and Feed Technology
 Examples of my meta-analysis papers

1. Effect of condensed tannin on N digestion (2010)

2. Effect of dietary tannin on methane emission (2012)
3. Comparison between organic vs conventional milk quality
(2012)
4. Effect of saponin on methane emission (2014)
5. Nutrient intake and digestion among four domestic
ruminant species (2014)
6. Energy and protein requirement of Indonesian sheep (2017)
7. Nitrogen excretion of ruminants (2017)
8. Effect of 3-NOP on methane emission (2018)
9. Effect of tannin on silage quality (2019)

Department of Nutrition and Feed Technology

 Thank you very much
for your attention!

Department of Nutrition and Feed Technology