by

Vivien M. Talisayon, Ph.D

What is Research?
If there is no new knowledge
created, is it research?
If not systematic, is it research?
If not objective, is it research?

Research = systematic, objective
creation of knowledge


Why Do Research in Education?
To solve teaching-learning problems
To test innovations and trends
To determine effectiveness of programs
To modify or extend teaching-learning
theories


What Is Experimental Research in Education?
It is less precise, less systematic, and less
objective than scientific research. Why?

Subjects or data sources - people, not
things. No two persons are identical, even
so-called identical twins. Persons are not
things or animals/plants that are more
easily manipulated or controlled as in a
science laboratory. Human beings are
complicated!
In education, we do not test or
introduce innovations to harm
our students. NOT our calling as
teachers!
Students are ultimate
beneficiaries of educational
research.
True Experimental Research Design*
Experimental Group R 01 X 02
Control Group R 01 02

where R = randomly selected individual
subjects
01 = pretest, 02 = posttest
X = treatment
*Difficult to do with classes in school system
More Realistic/Common Design:
Quasi-Experimental Design
Experimental Group 01 X1 02
Comparison Group 01 X2 02
where 01 = pretest 02 = posttest
X1 = innovation tested
X2 = existing condition
Experimental and Comparison Groups
intact classes

t-test frequently used to determine
if mean of experimental group is
significantly (not due to chance
alone) higher than that of
comparison group
In this case, t-test is one-tailed, in
favor of experimental group, and
for two unrelated groups
(experimental and comparison
groups)
Effect of treatment that is significant (not
due to chance alone)
Size of effect of treatment in at least 95 out
of 100 cases (so-called 95% confidence
interval)
Size of a significant relationship between
two variables
Standardizes (quantitatively/statistically)
findings across studies for comparison,
synthesis and meta-analysis










where

.
Cohens d is difference of mean of Experimental Group 1
and mean of Comparison Group 2 divided by the pooled
standard deviation*, s. In symbols,


where n1 = size of Group1, s1 = standard deviation* of Group 1
n2 = size of Group 2, s2 = standard deviation* of Group 2




Standard deviation, s or SD =
measure of spread/distance of
numbers from the mean
















































































































































Big s/SD Small s/SD
13
ESRC RDI One Day Meta-analysis workshop (Marsh, OMara, Malmberg)
13


Activity 2: Seeing Patterns
Sample Size (N), Significance Level (p),Effect Size (d)
Study 1 Exp Cntr
N 10 10
M 105 100
SD 15 15
t 0.750
p 0.466
d 0.333
Study 2 Exp Cntr
N 50 50
M 105 100
SD 15 15
t 1.667
p 0.099
d 0.333
Study 3 Exp Cntr
N 100 100
M 105 100
SD 15 15
t 2.360
p 0.019
d 0.333
Study Mean 1 s1 Mean 2 s2 d
A 85 13 60 12
B 60 13 45 12
C 60 15 45 30
D 60 30 45 15
Average d
(2) Compute d of Teaching Strategy: Active Learning
(n1 = n2 in each study) in table below.

(1) Show that If n1 = n2, pooled s = square root [(s1 squared +
s2 squared)/2] and Cohens d = [square root [2] x (Mean 1
Mean 2)] / square root (s1 squared + s2 squared)

Study Mean 1 s1 Mean 2 s2 d
A 85 13 60 12 1.998
B 60 13 45 12 1.199
C 60 15 45 30 0.632
D 60 30 45 15 0.632
Average d 1.115
Teaching Strategy: Active Learning
(n1 = n2 in each study)
Google: effect size calculator
Examples: (1) Beckers
www.uccs.edu/~lbecker/
(2) Ellis,
www.polyu.edu.hk/mm/effectsizef
aqs/calculator/calculator.html
Use SPSS (Statistical Package
for the Social Sciences)


Cohens d levels (1988)
no (non-significant) effect, d=0:
below .20
small effect size .20 - .49
medium effect size .50 - .79
large effect size .80 and higher

Study Mean 1 s1 Mean 2 s2 d d level
A 85 13 60 12 1.998 large
B 60 13 45 12 1.199 large
C 60 15 45 30 0.632 medium
D 60 30 45 15 0.632 medium
Average d 1.115 large
Teaching Strategy: Active Learning
(n1 = n2 in each study)
pooled
G G
S
X X
g ES
2 1

= =
Hedges Effect Size, ES
C
G G
s
X X
ES
2 1

= A =
Glass Effect Size, (delta)


where t = t value, r =
correlation coefficient
Expressed in t and r:
df t
t
df t
t
r

=
|
|
.
|

\
|
+
=
2
2
2
Effect Size, r (correlation coefficient)
where t = t value
df = degrees of freedom
In Activity 2, n1 = n2 = n, df = n - 1

For t-test results, related or unrelated
groups, Effect Size, r:



t value for two unrelated groups
Study Mean
1
s1 Mean
2
s2 r*
A 85 13 60 12 0.707
B 60 13 45 12 0.514
C 60 15 45 30 0.302
D 60 30 45 15 0.302
Average 0.456
Teaching Strategy: Active Learning
(n1 = n2 in each study)
* Using Beckers sample calculator
Values of r from -1 to +1. Zero effect size =
non-significant r
Small effect size, significant r between 0
and .29
Medium effect size, significant r between
.30 and .69
Large effect size, significant r between
.70 and 1

Study Mean
1
s1 Mean
2
s2 r r level
A 85 13 60 12 0.707 large
B 60 13 45 12 0.514 medium
C 60 15 45 30 0.302 medium
D 60 30 45 15 0.302 medium
Average 0.456 medium
Teaching Strategy: Active Learning
(n1 = n2 in each study)
Study Mean
1
s1 Mean
2
s2 d d level r r level
A 85 13 60 12 1.998 large 0.707 large
B 60 13 45 12 1.199 large 0.514 medium
C 60 15 45 30 0.632 medium 0.302 medium
D 60 30 45 15 0.632 medium 0.302 medium
Average 1.115 large 0.456 medium
Teaching Strategy: Active Learning
(n1 = n2 in each study)
N
Z
N
r
2
2
2
= = =
_
|
Effect size, r, in using Chi-square value
& total sample size, N
Using Ellis effect size
calculator:
www.polyu.edu.hk/mm/
effectsizefaqs/calculato
r/calculator.html
Instructional strategies w/ medium to large
effect sizes (.59 to 1.61), are related to
increased student achievement (Appalachia
Educational Laboratory, 2005):
Metacognition (students thinking about their
thinking)
Active student engagement
Higher order thinking
Cooperative learning
Independent practice

Marzano (2003): 3 processes for
teaching metacognitive skills
(1) Provide students w/ specific
learning objectives before
each lesson (Effect size = .97)
(2) Give feedback on processes and
strategies students use
(Effect size = .74)
(3) Allot time for students to consider
how to approach a task and remind
them to activate specific thinking
behaviors (Effect size = .53)

Instructional Strategy Ave. Effect Size
Identifying similarities and differences 1.61
Summarizing and note-taking 1.00
Reinforcing effort and providing recognition 0.8
Homework and practice 0.77
Non-linguistic representations 0.75
Cooperative learning 0.73
Setting objectives and providing feedback 0.61
Generating and testing hypotheses 0.61
Questions, cues, and advance organizers 0.59
Marzano (2003)
EFFECT SIZE OF TEACHING
STRATEGIES
Hattie (2003, 2009): over 800 meta-analyses
on student achievement
Influence Effect
Size, d
Teacher Feedback 1.13
Student Prior Cognitive Ability 1.04
Instructional Quality 1.00
Direct Instruction 0.82
Remediation/Feedback 0.65
Student Disposition to Learn 0.61
Hattie (2003, 2009)

Influence Effect Size,
d
Class
Environment
0.56
Challenge of
Goals
0.52
Peer Tutoring 0.50
Mastery
Learning
0.50
Effect size studies are secondary, not
primary research, using and comparing
results of experimental studies.
Studies vary in research questions and
methodology sampling, instrumentation,
intervention conditions, data collection,
and data analysis
Effect size study is quantitative review of
literature where results of studies are
synthesized compared and contrasted.

Study on effect size of teaching strategies is
descriptive research (not experimental
research), using findings of experimental or
quasi-experimental researches.
Descriptive and experimental research: one is
not superior to the other. Good research:
original research questions and appropriate
methodology done with rigor
As literature review, effect size studies can
quickly help education stakeholders apply
synthesis to their work from vast, scattered,
global literature

However, consider limitations of
effect size study as secondary
research w/ many variations in
methodology of meta-analyzed or
synthesized studies
Challenge for researchers: derive
new frameworks from effect sizes
of studies.



Innovate
Boldly &
Shape
Tomorrow!

Compute
Happily

Validate
Faithfully

Review
Thoroughly
Analyze &
Conclude w/
Caution
Appalachia Educational Laboratory (2005).
Effective instructional strategies, Research
Digest, www.edvantia.org, accessed 5/20/13
Becker, L. Effect size calculator. University of
Colorado, Colorado Springs,
www.uccs.edu/~lbecker/, accessed 5/20/13
Cohen, J. (1988). Statistical power analysis
for the behavioral sciences (2
nd
ed.) Hillsdale,
NJ: Erlbaum.
Ellis, P. (2009), Effect size calculators,
www.polyu.edu.hk/mm/effectsizefaqs/calcula
tor/calculator.html, accessed 5/ 21/2013.
Glass, G, McGaw, B. and Smith, M (1981).
Meta-analysis in social science research.
Beverly Hills, CA: Sage.


Hattie, J. (2003). Teachers make a difference:
what is the research evidence. Australian
Council for Educational Research.
______ (2009). Visible learning: a synthesis of
over 800 meta-analyses related to
achievement. London: Routledge.
Huberty, C. J. (2002). A history of effect size
indices. Educational and Psychological
Measurement, 62, 227-240.
Marsh, H., OMara, A. & Malmberg, L. Meta-
analysis: effect size calculation, Department
of Education, University of Oxford,
www.education.ox.ac.uk/wordpress/wp-
content/.../1.2.-calculate_es.ppt , accessed
5/20/13
Marzano, R., Marzano, J., Pickering, D.
(2003). Classroom management that works:
research-based strategies for every teacher.
Virginia: ASCD Publications
McCartney, K., & Rosenthal, R. (2000). Effect
size, practical importance, and social policy
for children. Child Development, 71, 173-
180.
Olejnik, S., & Algina, J. (2000). Measures of
effect size for comparative studies:
Applications, interpretations, and
limitations. Contemporary Educational
Psychology, 25, 241-286.


STAY HAPPY!