Beruflich Dokumente
Kultur Dokumente
impacts
the toolbox
By
Jonas Mnsson* & Marie Gartell**
* Linnaeus University & Swedish
National Audit Office,
** Swedish National Audit Office
Summary of principles of
policy evaluation
Evaluation questions
1. Have goals been achieved?
2. Is goal fulfilment due to the project under
examination?
3. Is the value of the observed impacts larger than the
cost of achieving them?
4. (Is there something within the activity that have
influenced the results?)
Evaluation type
1. Monitoring
2. Impact evaluation
3. Efficiency evaluation (Cost-benefit analysis, CBA)
4. (Process evaluation)
Impact evaluation
Evaluation question: Has the goals been
achieved and are they an effect of the
project?
Principles
Activity
Result
A
A
Meas
Impact = Result Aure
Activit
Resul Result B
yB
tB
By comparing the results of activity A with the results of
activity B its possible to talk about the impact of A,
relative to B.
Case 1
Result A = impact +
selection + O Result B =
O
Result A Result B =
impact + selection + O
O=
impact + selection
How can we handle
Identification
Assignment of treated and untreated
should be exogenous
P(T) = P(UT)
The goal with methods within the
201604-14
Evaluation designs
Non
experimental
Goal
Experimental
Random
design
Matchin
g
Chara
cterist
ic
match
ing
CEM
PSM
Synth
Learning objective
The objective with the toolbox lecture is
Presentation
Method
Question to be answered
(some times illustration)
Papers that can be used as reference
Used in SNAO
Papers are convenience sampled and
Experimental
Designs
supposed to be measured
If done correct low costs
Disadvantages - problems
Often small scale experiment, i.e. low
external validity
If something goes wrong, e.g. systematic
Papers
Bennmarker, Helge & Grnqvist, Erik &
Natural experiment
A
Natural experiments
Principles:
The shock into a system creates a random
is a shock?
Practical
Papers
Mnsson & Delander (2007), Forensic evaluation:
Field experiment
Used to investigate decision making
e.g. discrimination
The respondent react to a fake
situation
Can keep a lot of factors constant and
1
8
201604-14
Field experiment
Advantages
High internal validity
Disadvantages
Ethnical issues is it O.K. to fool someone?
Is it O.K. that the parliaments audit
201604-14
Papers
Laboratory experiment
Used to investigate behaviours
experiments)
Can
languages/dialects
2
1
201604-14
Laboratory experiment
Advantages
High internal validity
Disadvantages
Low external validity
Costly
In earlier studies (Kahnemann nobel prize
2
2
Papers
Ahmed, A. (2010) What is in a surname? The r
ole of ethnicity in
economic decision making. Applied Economics,
2010, 42(21),
2715-2723.
Arai, M, Gartell, M, Rdin, M och G zcan
Quasiexperimental
design
Quasi-experimental design
Common:
Impact
Selection
Simple
Regression techniques
Matching techniques
Combinations of matching and regressions
Objective with all methods is to eliminate
Regression techniques
Controlling for
heterogeneity
Treat
ed
Befor
Untrea
ted
C
e
After
B
D
B-A = effect for treated, D-C=
effect UT
C-A = Initial difference =
selection effect
Impact = (B-A)-(D-C)=(B-D) +
(C-A)
Difference in Difference
Di
D
1
2
R
1
0
3
A
4
B
B
2
Regression techniques
In practice
D1 = 1 if Treated, D1=0 if Untreated
D2 = 1 if After, D2 = 0 if Before
D3=D1 x D2 = Difference in
Difference estimate
Y 1 D1 2 D2 3 D3 i
X
Advant
Handle large samples = external validity
ages
Intuitive
Disadvantages
Does
Papers
Avdic, D och M Gartell (2015) "The study
Regression discontinuity
9
8
7
6
T = 1 om S
>
T=c 0 om S
<c
5
4
3
2
1
0
3
2
201604-14
Regression discontinuity
Identification is that the reform has a
3
3
Advant
Handle a lot of information = high external
ages
validity
Is assumed to produce high internal validity
Disadvantages
Fuzzy
Papers
Fredriksson, P., Oosterbeek, H. and
Matching
approaches
individuals/firms/organisations that in
the respect of selection is as close as
possible to a treated unit.
Logic: If we can find untreated units that
Early studies
For each individual that are treated, find an individual
with the same important characteristics with in the
target group and use these statistical twins and control
group.
Example:
Match on gender (2), education (3), immigrant
background (2), age (6 intervals)
=2*3*2*6=72 different comparisons.
The curse of dimensionality
Propensity score
Rosenbaum & Rubin (1983), partly also
Result:
Advantages
1. Gives the evaluator a possibility to
replicate a RCT
2. Research (Dehijia and Wahba, 1998;
1999) has shown the propensity
score
matching
method
can
replicate the results of a RCT (high
internal validity)
3. Since PSM dont have any limitations
on data (more than accessibility) large
samples can be evaluated (high
external validity)
Disadvantages
Evaluators
Heterogeneity
Problem
If there are factors that affect selection
and outcome that are
not observable in the data and no good
proxy variable exist
=> biased estimates and standard errors
4
3
201604-14
Papers
Caliendo, M. and Kopeinig, S. (2005). Some
Audit reports
Riksrevisionen (2008) Std till start av
nringsverksamhet, Ett
framgngsrikt program, RiR 2008:24,
Riksrevisionen.
2010:05, Riksrevisionen
Mnsson & Lundin (2016) When outcome
4
6
201604-14
Advant
Free of functional form assumptions
ages
(non-parametric)
Manage heterogeneity e.g. gender,
education etc..
Disadvant
age:
Data
intensive
No
solution
might
Papers
Blackwell,
Mnsson
4
8
04-14
Audit
reports
Riksrevisionen 2014:05
Deltidsarbetslshet
Combinations between
matching and
regressions
Paper
Widerstedt, B. & Mnsson, J. (2015), Can business co
control group
Compare over time by e.g. DiD
Exempel: Falkenhall et al. (2016),
Example
Reduction in the VAT for Swedish
restaurants
Construct a synthetic restaurant industry
by using data from other industries
The method is completely data driven
and do not relay on any subjective
choice
Before
SCG
After
Advantage
Sometimes its the only option to close in
Papers
Abadie, A., Diamond, A. & Hainmuller, J.
implementation = natural
experiment) => compare outcome
Selection?
Can we observe (in data) characteristics
most times)
Is there a common goal and the selection is
quite homogeneous
Yes =Props score
No => matching on characteristics
constructed?
No-> Efficiency not possible
Yes -> Efficiency evaluation (Value for Money in
an impact context)
Summary
To identify impacts a relevant control
6
0
group is needed
Identification, identification,
identification.
Randomising is the most secure way
however sometimes it is neither possible
or suitable
Look for variation in the
implementation natural
experiment
Quasi experimental methods is
ange
presentationens
titel under
"infogaintense
2016almost
always
data
sidhuvud/sidfot"
04-14
) P V OJ
M A D ( W ITH
aw
OJ
)PY
COPY
M A D ( W I TH
IAl
( OPV
M
D[
IT H
A ti"'i'lr -
't
(J
5martDraw
MAOf
I Al
'WI TH
A TR
Quasi
F!'
0
rimental
Dl
OF
W I TH A U l A l
5
1
5martDraw
1;...-.rimental
OF
OJ
5martDraw
aw
)
Ul
COPY
IHDf
5
1
O PV 0
OF
Synthetic
>P v o r
aw
>PY OF
aw
=
diff-in diff
conditional difference
in difference
)p
V OF
aw
MAD I W I I H
H I Al
COPV
OF
5martDraw
Ot
5
1