Beruflich Dokumente
Kultur Dokumente
Richard P. PHELPS
2012, Richard P PHELPS International Test Commission, 8th Conference, Amsterdam, July, 2012 1
Meta-analysis
A method for summarizing a large research literature, with a single, comparable measure.
lacking sufficient time and money, hundreds of other studies will not be reviewed
2012, Richard P PHELPS International Test Commission, 8th Conference, Amsterdam, July, 2012 3
1. Included only those studies that found an effect from testing on student achievement or on teacher instruction
2012, Richard P PHELPS International Test Commission, 8th Conference, Amsterdam, July, 2012 4
2. when:
a test is newly introduced, or newly removed quantity of testing is increased or reduced test stakes are introduced or increased, or removed or reduced
2012, Richard P PHELPS International Test Commission, 8th Conference, Amsterdam, July, 2012 5
Methodology type
Quantitative Surveys and public opinion polls (US & Canada) Qualitative
TOTAL
2012, Richard P PHELPS
669
International Test Commission, 8th Conference, Amsterdam, July, 2012
1698
7
d between 0.25 & 0.50 weak effect d between 0.50 et 0.75 medium effect d more than 0.75 strong effect
10
Quantitative studies
(population coverage 7 million persons)
11
Bare bones effect size adjusted for measurement error d +0.71 a stronger effect
d +0.88
a strong effect
12
Treatment Group is made aware of performance, and control group is not receives targeted instruction (e.g., remediation) is tested with higher stakes than control group is tested more frequently than control group
13
14
International
Number of Mean Studies Effect Size 5 1.02 99 0.93 45 0.81 11 0.64 160
15
Pre-post Experiment, Quasi-experiment Multivariate Experiment, posttest only Pre-post (with shadow test) Total
16
Number of Mean Studies Effect Size 9 1.60 118 33 160 0.91 0.57
17
Number Mean of Studies Effect Size 115 0.95 6 39 160 0.72 0.71
18
19
Percent
20
Number and percent of survey items, by test stakes and target group
% 62 23 4 11
% 46 33 14 7
21
100
80
60
40
20
1960
1965
1970
1975
1980 Year
1985
1990
1995
2000
2005
22
23
24
Methodology Case study Experiment or pre-post study Interviews (individual or group) Journal Review of official records, documents, reports
% 43 7 27 1 12
Research review
Survey TOTAL
8
22 281
3
8 100
25
Percent of studies 84 10 2 3
2 4
Negative
TOTAL
2012, Richard P PHELPS
3
244
1
100
1
100
26
Achievement is improved
Yes
%
95
Mixed results
No TOTAL
1
10 211
<1
5 100
% 96 4 100
27
medium 67 8 1 3 1 80
low 42 6 1 1 1 51
medium 27 5 1 33
low 38 7 1 5 51
unknown 6
Total 204 24 5 8 3
244
28
Generally positive
High-stakes tests High level of study rigor Student attitudes toward test positive Teacher attitudes toward test positive Student achievement improved Instruction improved Large-scale testing
93
71 46 60 55 95 92 86
95
42 48 71 80 95 100 68
29
30
With a dismissive research literature review, a researcher assures all that no other researcher has studied the same topic
31
Firstness claims
With a firstness claim, a researcher insists that he or she is the first to ever study a topic
32
Richard P. PHELPS
2012, Richard P PHELPS International Test Commission, 8th Conference, Amsterdam, July, 2012 34