Experiments and Quasi-Experiments

(SW Chapter 11)

Why study experiments?
Ideal randomized controlled experiments provide a
benchmark for assessing observational studies.
Actual experiments are rare ($$$ but influential.
!xperiments can solve the threats to internal validity
of observational studies" but they have their o#n
threats to internal validity.
$$%$
&hinking about experiments helps us to understand
'uasi%experiments" or (natural experiments") in #hich
there some variation is (as if) randomly assigned.
$$%*
Terminology: experiments and quasi-experiments
An experiment is designed and implemented
consciously by human researchers. An experiment
entails conscious use of a treatment and control group
#ith random assignment (e.g. clinical trials of a drug
A quasi-experiment or natural experiment has a
source of randomization that is (as if) randomly
assigned" but this variation #as not part of a conscious
randomized treatment and control design.
Program evaluation is the field of statistics aimed at
evaluating the effect of a program or policy" for
example" an ad campaign to cut smoking.
$$%+
Dierent types o experiments: three examples
,linical drug trial- does a proposed drug lo#er
cholesterol?
oY . cholesterol level
oX . treatment or control group (or dose of drug
/ob training program (/ob &raining 0artnership Act
oY . has a 1ob" or not (or Y . #age income
oX . #ent through experimental program" or not
,lass size effect (&ennessee class size experiment
oY . test score (2tanford Achievement &est
$$%3
oX . class size treatment group (regular" regular 4
aide" small
$$%5
!ur treatment o experiments: "rie outline
Why (precisely do ideal randomized controlled
experiments provide estimates of causal effects?
What are the main threats to the validity (internal and
external of actual experiments 6 that is" experiments
actually conducted #ith human sub1ects?
7la#s in actual experiments can result in X and u being
correlated (threats to internal validity.
2ome of these threats can be addressed using the
regression estimation methods #e have used so far-
multiple regression" panel data" I8 regression.
$$%9
#deali$ed Experiments and Causal Ee%ts
(SW Se%tion 11&1)
An ideal randomized controlled experiment randomly
assigns sub1ects to treatment and control groups.
:ore generally" the treatment level X is randomly
assigned-
Y
i
.
;
4
$
X
i
4 u
i
If X is randomly assigned (for example by computer
then u and X are independently distributed and E(u
i
<X
i

. ;" so =>2 yields an unbiased estimator of
$
.
&he causal effect is the population value of
$
in an
ideal randomized controlled experiment
$$%?
Estimation o %ausal ee%ts in an ideal randomi$ed
%ontrolled experiment
@andom assignment of X implies that E(u
i
<X
i
. ;.
&hus the =>2 estimator
$
A
is unbiased.
When the treatment is binary"
$
A
is 1ust the difference
in mean outcome (Y in the treatment vs. control
group (
treated
Y
6
control
Y
.
&his differences in means is sometimes called the
differences estimator&
$$%B
'otential 'ro"lems (ith Experiments in 'ra%ti%e
(SW Se%tion 11&))
Threats to #nternal *alidity
$. Failure to randomize (or imperfect randomization
for example" openings in 1ob treatment program are
filled on first%come" first%serve basisC latecomers are
controls
result is correlation bet#een X and u
$$%D
Threats to internal +alidity, %td&
*. Failure to follow treatment protocol (or (partial
compliance-
some controls get the treatment
some (treated) get controls
(errors%in%variables) bias- corr(X"u ;
Attrition (some sub1ects drop out
suppose the controls #ho get 1obs move out of to#nC
then corr(X"u ;
$$%$;
Threats to internal +alidity, %td&
.& Experimental effects
experimenter bias (conscious or subconscious-
treatment X is associated #ith (extra effort) or
(extra care") so corr(X"u ;
sub1ect behavior might be affected by being in an
experiment" so corr(X"u ; (Ea#thorne effect
/ust as in regression analysis #ith observational data"
threats to the internal validity of regression #ith
experimental data implies that corr(X"u ; so =>2 (the
differences estimator is biased.
/eorge Elton 0ayo and the 1a(thorne Experiment
$$%$$
2ub1ects in the Ea#thorne plant experiments" $D*3 6 $D+*
$$%$*
Threats to External *alidity
$. Fonrepresentative sample
*. Fonrepresentative (treatment) (that is" program or
policy
+. General e'uilibrium effects (effect of a program can
depend on its scaleC admissions counseling
3. &reatment v. eligibility effects (#hich is it you #ant
to measure- effect on those #ho take the program" or
the effect on those are eligible
$$%$+
2egression Estimators o Causal Ee%ts 3sing
Experimental Data
(SW Se%tion 11&.)
7ocus on the case that X is binary (treatmentHcontrol.
=ften you observe sub1ect characteristics" W
$i
"I"W
ri
.
!xtensions of the differences estimator-
ocan improve efficiency (reduce standard errors
ocan eliminate bias that arises #hen-
treatment and control groups differ
there is (conditional randomization)
there is partial compliance
$$%$3
&hese extensions involve methods #e have already
seen 6 multiple regression" panel data" I8 regression
$$%$5
Estimators o the Treatment Ee%t
1
using
Experimental Data (X 4 1 i treated, 5 i %ontrol)
Dep&
+"le
#nd&
+"le(s)
method
differences Y X =>2
differences%in%
differences
Y .
Y
after

6 Y
before
X =>2 ad1usts for initial
differences bet#een
treatment and control
groups
differences #ith
addJl regressors
Y X"W
$
"
I"W
n
=>2 controls for
additional sub1ect
characteristics W
$$%$9
Estimators (ith experimental data, %td&
Dep&
+"le
#nd&
+"le(s)
method
differences%in%
differences #ith
addJl regressors
Y .
Y
after

6 Y
before
X"W
$
"
I"W
n
=>2 ad1usts for group
differences 4 controls
for sub1ect charJs W
Instrumental
variables
Y X &2>2 Z . initial random
assignmentC
eliminates bias from
partial compliance
&2>2 #ith Z . initial random assignment also can be
applied to the differences%in%differences estimator and
the estimators #ith additional regressors (WJs
The dieren%es-in-dieren%es estimator
$$%$?
2uppose the treatment and control groups differ
systematicallyC maybe the control group is healthier
(#ealthierC better educatedC etc.
&hen X is correlated #ith u" and the differences
estimator is biased.
&he differences%in%differences estimator ad1usts for pre%
experimental differences by subtracting off each
sub1ectJs pre%experimental value of Y
o
before
i
Y . value of Y for sub1ect i before the expt
o
after
i
Y . value of Y for sub1ect i after the expt
o Y
i
.
after
i
Y
6
before
i
Y

. change over course of expt
$$%$B
$
A
diffs in diffs

. (
" treat after
Y
6
" treat before
Y
6 (
" control after
Y
6
" control before
Y

$$%$D
The dieren%es-in-dieren%es estimator, %td.
($ (Kifferences) formulation-
Y
i
.
;
4
$
X
i
4 u
i
#here
Y
i
.
after
i
Y
6
before
i
Y
X
i
. $ if treated" . ; other#ise

$
A

is the diffs%in%diffs estimator

$$%*;
The dieren%es-in-dieren%es estimator, %td.
(* !'uivalent (panel data) version-
Y
it
.
;
4
$
X
it
4
*
D
it
4
+
G
it
4 v
it
" i . $"I"n
#here
t . $ (before experiment" * (after experiment
D
it
. ; for t . $" . $ for t . *
G
it
. ; for control group" . $ for treatment group
X
it
. $ if treated" . ; other#ise
. D
it
G
it
. interaction effect of being in treatment
group in the second period

$
A

is the diffs%in%diffs estimator

$$%*$
#n%luding additional su"6e%t %hara%teristi%s (W7s)
&ypically you observe additional sub1ect characteristics"
W
$i
"I"W
ri
Kifferences estimator #ith addJl regressors-
Y
i
.
;
4
$
X
i
4
*
W
$i
4 I 4
r4$
W
ri
4 u
i
Kifferences%in%differences estimator #ith WJs-
Y
i
.
;
4
$
X
i
4
*
W
$i
4 I 4
r4$
W
ri
4 u
i
$$%**
#here Y
i
.
after
i
Y
6
before
i
Y
.
$$%*+
Why in%lude additional su"6e%t %hara%teristi%s (W7s)8
$. Efficiency- more precise estimator of
$
(smaller
standard errors
*. Check for randomization. If X is randomly assigned"
then the =>2 estimators #ith and #ithout the WJs
should be similar 6 if they arenJt" this suggests that X
#asnJt randomly designed (a problem #ith the expt.
Note- &o check directly for randomization"
regress X on the WJs and do a %test.
+. !d"ust for conditional randomization (#e$ll return to
this later%
$$%*3
Estimation (hen there is partial %omplian%e
,onsider diffs%in%diffs estimator" X . actual treatment
Y
i
.
;
4
$
X
i
4 u
i
2uppose there is partial compliance- some of the
treated donJt take the drugC some of the controls go to
1ob training any#ay
&hen X is correlated #ith u" and =>2 is biased
2uppose initial assignment" Z" is random
&hen ($ corr(Z"X ; and (* corr(Z"u . ;
&hus
$
can be estimated by &2>2" #ith instrumental
variable Z . initial assignment
&his can be extended to WJs (included exog. variables
$$%*5
Experimental Estimates o the Ee%t o
2edu%tion: The Tennessee Class Si$e Experiment
(SW Se%tion 11&9)
0ro1ect 2&A@ (2tudent%&eacher Achievement @atio
3%year study" $$* million
Lpon entering the school system" a student #as
randomly assigned to one of three groups-
oregular class (** 6 *5 students
oregular class 4 aide
osmall class ($+ 6 $? students
regular class students re%randomized after first year to
regular or regular4aide
Y . 2tanford Achievement &est scores
$$%*9
De+iations rom experimental design
0artial compliance-
o$;M of students s#itched treatment groups because
of (incompatibility) and (behavior problems) 6 ho#
much of this #as because of parental pressure?
oFe#comers- incomplete receipt of treatment for
those #ho move into district after grade $
Attrition
ostudents move out of district
ostudents leave for privateHreligious schools
$$%*?
2egression analysis
&he (differences) regression model-
Y
i
.
;
4
$
&mallClass
i
4
*
'eg!ide
i
4 u
i
#here
&mallClass
i
. $ if in a small class
'eg!ide
i
. $ if in regular class #ith aide
Additional regressors (WJs
oteacher experience
ofree lunch eligibility
ogender" race
$$%*B
Kifferences estimates (no WJs
$$%*D
$$%+;
1o( "ig are these estimated ee%ts8
0ut on same basis by dividing by std. dev. of Y
Lnits are no# standard deviations of test scores
$$%+$
Eo# do these estimates compare to those from the
,alifornia" :ass. observational studies? (,h. 3 6 ?
$$%+*
$$%++
Summary: The Tennessee Class Si$e Experiment
@emaining threats to internal validity
partial complianceHincomplete treatment
ocan use &2>2 #ith Z . initial assignment
o&urns out" &2>2 and =>2 estimates are similar
(Nrueger ($DDD" so this bias seems not to be large
:ain findings-
&he effects are small 'uantitatively (same size as
gender difference
!ffect is sustained but not cumulative or increasing
biggest effect at the youngest grades
$$%+3
What is the Dieren%e :et(een a Control *aria"le
and the *aria"le o #nterest8
(SW ;pp& 11&.)
!xample- (free lunch eligible) in the 2&A@ regressions
,oefficient is large" negative" statistically significant
0olicy interpretation- (aking students ineligible for a
free school lunch #ill im)rove their test scores*
Is this really an estimate of a causal effect?
Is the =>2 estimator of its coefficient unbiased?
,an it be that the coefficient on (free lunch eligible)
is biased but the coefficient on &mallClass is not?
$$%+5
$$%+9
E+am)le- (free lunch eligible") ctd.
,oefficient on (free lunch eligible) is large" negative"
statistically significant
0olicy interpretation- (aking students ineligible for a
free school lunch #ill im)rove their test scores*
Why (precisely can #e interpret the coefficient on
&mallClass as an unbiased estimate of a causal effect"
but not the coefficient on (free lunch eligible)?
&his is not an isolated exampleO
o=ther (control variables) #e have used- gender"
race" district income" state fixed effects" time fixed
effects" city (or state population"I
What is a (control variable) any#ay?
$$%+?
Simplest %ase: one X, one %ontrol +aria"le W
Y
i
.
;
4
$
X
i
4
*
W
i
4 u
i
<or example,
W . free lunch eligible (binary
X . small classHlarge class (binary
2uppose random assignment of X depends on W
ofor example" 9;M of free%lunch eligibles get small
class" 3;M of ineligibles get small class
onote- this #asn$t the actual &,!' randomization
)rocedure 6 this is a hypothetical example
$$%+B
7urther suppose W is correlated #ith u
$$%+D
Y
i
.
;
4
$
X
i
4
*
W
i
4 u
i
&u))ose-
&he control variable W is correlated #ith u
Given W . ; (ineligible" X is randomly assigned
Given W . $ (eligible" X is randomly assigned.
,hen-
Given the value of W" X is randomly assignedC
&hat is" controlling for W" X is randomly assignedC
&hus" controlling for W" X is uncorrelated #ith u
:oreover" E(u<X"W doesnJt depend on X
&hat is" #e have conditional mean independence:
$$%3;
E(u<X"W . E(u<W
$$%3$
#mpli%ations o %onditional mean independen%e
Y
i
.
;
4
$
X
i
4
*
W
i
4 u
i
2uppose E(u<W is linear in W (not restrictive . could
add /uadratics etc*- then"
E(u<X"W . E(u<W .
;
4
$
W
i
(P
so
E(Y
i
<X
i
"W
i
. E(
;
4
$
X
i
4
*
W
i
4 u
i
<X
i
"W
i

.
;
4
$
X
i
4
*
W
i
4 E(u
i
<X
i
"W
i

.
;
4
$
X
i
4
*
W
i
4
;
4
$
W
i
by (P
. (
;
4
;
4
$
X
i
4 (
$
4
*
W
i
$$%3*
#mpli%ations o %onditional mean independen%e:
&he conditional mean of Y given X and W is
E(Y
i
<X
i
"W
i
. (
;
4
;
4
$
X
i
4 (
$
4
*
W
i
&he effect of a change in X under conditional mean
independence is the desired causal effect-
E(Y
i
<X
i
. +4+"W
i
6 E(Y
i
<X
i
. +"W
i
.
$
+
or

$
.
( < " ( < "
i i i i i i
E Y X + + W E Y X + W
+
+

If X is binary (treatmentHcontrol" this becomes-

$
.
( < $" ( < ;"
i i i i i i
E Y X W E Y X W
+

#hich is the desired treatment effect.

$$%3+
#mpli%ations o %onditional mean independen%e, %td&
Y
i
.
;
4
$
X
i
4
*
W
i
4 u
i
,onditional mean independence says-
E(u<X"W . E(u<W
#hich" #ith linearity" implies-
E(Y
i
<X
i
"W
i
. (
;
4
;
4
$
X
i
4 (
$
4
*
W
i
&hen-
&he =>2 estimator
$
A
is unbiased.

*
A

is not consistent and not meaningful

&he usual inference methods (standard errors"
hypothesis tests" etc. apply to
$
A
.
$$%33
So, (hat is a %ontrol +aria"le8
A control variable W is a variable that results in X
satisfying the conditional mean independence condition-
E(u<X"W . E(u<W
Lpon including a control variable in the regression" X
ceases to be correlated #ith the error term.
&he control variable itself can be (in general #ill be
correlated #ith the error term.
&he coefficient on X has a causal interpretation.
&he coefficient on W does not have a causal
interpretation.
$$%35
Example: Ee%t o tea%her experien%e on test s%ores
:ore on the design of 0ro1ect 2&A@-
&eachers didnJt change school because of the expt.
Within their normal school" teachers #ere randomly
assigned to smallHregularHreg4aide classrooms.
What is the effect of X . years of teacher education?
,he design im)lies conditional mean inde)endence-
W . school binary indicator
Given W (school" X is randomly assigned
&hat is" E(u<X"W . E(u<W
$$%39
W is plausibly correlated #ith u (nonzero school fixed
effects- some schools are betterHricherHetc than others
$$%3?
$$%3B
Example: tea%her experien%e, %td&
Without school fixed effects (*" the estimated effect of
an additional year of experience is $.3? (&E . .$?
0Controlling for the school) (+" the estimated effect of
an additional year of experience is .?3 (&E . .$?
Kirection of bias makes sense-
oless experienced teachers at #orse schools
oyears of experience picks up this school effect
=>2 estimator of coefficient on years of experience is
biased up #ithout school effectsC #ith school effects"
=>2 yields unbiased estimator of causal effect
2chool effect coefficients donJt have a causal
interpretation (effect of student changing schools
$$%3D
Quasi-Experiments
(SW Se%tion 11&=)
A quasi-experiment or natural experiment has a source
of randomization that is (as if) randomly assigned" but
this variation #as not part of a conscious randomized
treatment and control design.
&#o cases-
(a &reatment (X is (as if) randomly assigned (=>2
(b A variable (Z that influences treatment (X is
(as if) randomly assigned (I8
$$%5;
T(o types o quasi-experiments
(a &reatment (X is (as if) randomly assigned (perhaps
conditional on some control variables W
E+- !ffect of marginal tax rates on labor supply
oX . marginal tax rate (rate changes in one state"
not anotherC state is (as if) randomly assigned
(b A variable (Z that influences treatment (X is
(as if) randomly assigned (I8
!ffect on survival of cardiac catheterization
X . cardiac catheterizationC
Z . differential distance to ,, hospital
$$%5$
E%onometri% methods
(a &reatment (X is (as if) randomly assigned (=>2
Kiffs%in%diffs estimator using panel data methods-
Y
it
.
;
4
$
X
it
4
*
D
it
4
+
G
it
4 u
it
" i . $"I"n
#here
t . $ (before experiment" * (after experiment
D
it
. ; for t . $" . $ for t . *
G
it
. ; for control group" . $ for treatment group
X
it
. $ if treated" . ; other#ise
. D
it
G
it
. interaction effect of being in treatment
group in the second period

$
A

is the diffs%in%diffs estimatorI

$$%5*
The panel data dis-in-dis estimator simpliies to
the >%hanges- dis-in-dis estimator (hen T 4 )
Y
it
.
;
4
$
X
it
4
*
D
it
4
+
G
it
4 u
it
" i . $"I"n (P
7or t . $- D
i$
. ; and X
i$
. ; (nobody treated" so
Y
i$
.
;
4
+
G
i$
4 u
i$
7or t . *- D
i*
. $ and X
i*
. $ if treated" . ; if not" so
Y
i*
.
;
4
$
X
i*
4
*
4
+
G
i*
4 u
i*
so
Y
i
. Y
i*
6Y
i$
. (
;
4
$
X
i*
4
*
4
+
G
i*
4u
i*
6 (
;
4
+
G
i$
4u
i$

.
$
X
i
4
*
4 (u
i$
6 u
i*
(since G
i$
. G
i*

or
$$%5+
Y
i
.
*
4
$
X
i
4 v
i
" #here v
i
. u
i$
6 u
i*
(PP
$$%53
Dieren%es-in-dieren%es (ith %ontrol +aria"les
Y
it
.
;
4
$
X
it
4
*
D
it
4
+
G
it
4
3
W
$it
4 I 4
+4r
W
rit
4 u
it
"
X
it
. $ if the treatment is received" . ; other#ise
. G
it
D
it
(. $ for treatment group in second period
If the treatment (X is (as if) randomly assigned"
given W" then u is conditionally mean indep. of X-
E(u<X"D"G"W . E(u<D"G"W
=>2 is a consistent estimator of
$
" the causal effect
of a change in X
In general" the =>2 estimators of the other
coefficients do not have a causal interpretation.
$$%55
(b A variable (Z that influences treatment (X is
(as if) randomly assigned (I8
Y
it
.
;
4
$
X
it
4
*
D
it
4
+
G
it
4
3
W
$it
4 I 4
+4r
W
rit
4 u
it
"

X
it
. $ if the treatment is received" . ; other#ise
. G
it
D
it
(. $ for treatment group in second period
Z
it
. variable that influences treatment but is
uncorrelated #ith u
it
(given WJs
&2>2-
X . endogenous regressor
D"G"W
$
"I"W
r
. included exogenous variables
Z . instrumental variable
$$%59
'otential Threats to Quasi-Experiments
(SW Se%tion 11&?)
&he threats to the internal +alidity of a 'uasi%
experiment are the same as for a true experiment" #ith
one addition.
3. Failure to randomize (imperfect randomization
Is the (as if) randomization really random" so that X
(or Z is uncorrelated #ith u?
=& Failure to follow treatment protocol attrition
9. Experimental effects (not applicable
?. !nstrument invalidit" (relevance 4 exogeneity
(:aybe healthier patients do live closer to ,, hospitals
6they might have better access to care in general
$$%5?
&he threats to the external +alidity of a 'uasi%
experiment are the same as for an observational study.
5. Fonrepresentative sample
9. Fonrepresentative (treatment) (that is" program or
policy
E+am)le- ,ardiac catheterization
&he ,, study has better external validity than
controlled clinical trials because the ,, study uses
observational data based on real%#orld
implementation of cardiac catheterization.
Eo#ever that study used data from the early D;Js 6 do its
findings apply to ,, usage today?
$$%5B
Experimental and Quasi-Experiments Estimates in
1eterogeneous 'opulations
(SW Se%tion 11&@)
We have discussed (the) treatment effect
Qut the treatment effect could vary across individuals-
o!ffect of 1ob training program probably depends on
education" years of education" etc.
o!ffect of a cholesterol%lo#ering drug could depend
other health factors (smoking" age" diabetes"I
If this variation depends on observed variables" then
this is a 1ob for interaction variablesO
Qut #hat if the source of variation is unobserved?
$$%5D
1eterogeneity o %ausal ee%ts
When the causal effect (treatment effect varies among
individuals" the population is said to be #eterogeneous.
When there are heterogeneous causal effects that are not
linked to an observed variable-
What do #e #ant to estimate?
o=ften" the average causal effect in the population
oQut there are other choices" for example the average
causal effect for those #ho participate (effect of
treatment on the treated
What do #e actually estimate?
ousing =>2? using &2>2?
$$%9;
'opulation regression model (ith heterogeneous
%ausal ee%ts-
Y
i
.
;
4
$i
X
i
4 u
i
" i . $"I"n

$i
is the causal effect (treatment effect for the i
th

individual in the sample
7or example" in the /&0A experiment"
$i
could be zero
if person i already has good 1ob search skills
What do #e #ant to estimate?
oeffect of the program on a randomly selected person
(the (average causal effect) 6 our main focus
oeffect on those most (least? benefited
oeffect on those #ho choose to go into the program?
$$%9$
The ;+erage Causal Ee%t
Y
i
.
;
4
$i
X
i
4 u
i
" i . $"I"n
&he average causal effect (or average treatment effect
is the mean value of
$i
in the population.
We can think of
$
as a random variable- it has a
distribution in the population" and dra#ing a different
person yields a different value of
$
(1ust like X and Y
7or example" for person R+3 the treatment effect is not
random 6 it is her true treatment effect 6 but before she
$$%9*
is selected at random from the population" her value of

$
can be thought of as randomly distributed.
The a+erage %ausal ee%t, %td&
Y
i
.
;
4
$i
X
i
4 u
i
" i . $"I"n
&he average causal effect is E(
$
.
What does =>2 estimate-
(a When the conditional mean of u given X is zero?
(b Lnder the stronger assumption that X is randomly
assigned (as in a randomized experiment?
1n this case2 34& is a consistent estimator of
the average causal effect.
$$%9+
!AS (ith 1eterogeneous Causal Ee%ts
Y
i
.
;
4
$i
X
i
4 u
i
" i . $"I"n

(a 2uppose E(u
i
<X
i
. ; so cov(u
i
"X
i
. ;.
If X is binary (treatedHuntreated"
$
A
.
treated
Y
6
control
Y

estimates the causal effect among those #ho receive
the treatment.
Why? 7or those treated"
treated
Y
reflects the effect of
the treatment on them. Qut #e donJt kno# ho# the
untreated #ould have responded had they been
treatedO
$$%93
T#e mat#- suppose X is binary and E(u
i
<X
i
. ;.
&hen
$
A
.
treated
Y
6
control
Y
7or the treated-
E(Y
i
<X
i
.$ .
;
4 E(
$i
X
i
<X
i
.$ 4 E(u
i
<X
i
.$
.
;
4 E(
$i
<X
i
.$
7or the controls-
E(Y
i
<X
i
.; .
;
4 E(
$i
X
i
<X
i
.; 4 E(u
i
<X
i
.;
.
;

&hus-
$
A

)

E(Y
i
<X
i
.$ 6 E(Y
i
<X
i
.; . E(
$i
<X
i
.$
. average effect of the treatment on the treated
$$%95
!AS (ith heterogeneous treatment ee%ts: general X
(ith E(u
i
BX
i
) 4 5
$
A
.
*
XY
X
s
s

)

*
XY
X

.
; $
cov( "
var(
i i i i
i
X u X
X
+ +
.
; $
cov( " cov( " cov( "
var(
i i i i i i
i
X X X u X
X
+ +
.
$
cov( "
var(
i i i
i
X X
X

(because cov(u
i
"X
i
. ;
If X is binary" this simplifies to the (effect of
treatment on the treated)
Without heterogeneity"
$i
.
$
and
$
A

)

$

In general" the treatment effects of individuals #ith
large values of X are given the most #eight
$$%99
(b Fo# make a stronger assumption- that X is randomly
assigned (experiment or 'uasi%experiment. &hen
#hat does =>2 actually estimate?
I X
i
is randomly assigned" it is distributed
independently of
$i
" so there is no difference
bet#een the population of controls and the
population in the treatment group
&hus the effect of treatment on the treated . the
average treatment effect in the population.
$$%9?
&he math-
$
A

)

$
cov( "
var(
i i i
i
X X
X

.
$
$
cov( "
<
var(
i i i
i
i
X X
E E
X

1
' ;
1
]


.
$
cov( "
var(
i i
i
i
X X
E
X

1
1
]
.
$
var(
var(
i
i
i
X
E
X

1
1
]
. E(
$i

Summary
If X
i
and
$i
are independent (X
i
is randomly
assigned" =>2 estimates the average treatment effect.
If X
i
is not randomly assigned but E(u
i
<X
i
. ;" =>2
estimates the effect of treatment on the treated.
$$%9B
Without heterogeneity2 the effect of treatment on the
treated and the average treatment effect are the same
#* 2egression (ith 1eterogeneous Causal Ee%ts
2uppose the treatment effect is heterogeneous and the
effect of the instrument on X is heterogeneous-
Y
i
.
;
4
$i
X
i
4 u
i
(e'uation of interest
X
i
.
;
4
$i
Z
i
4 v
i
(first stage of &2>2
In general" &2>2 estimates the causal effect for those
#hose value of X (probability of treatment is most
influenced by the instrument.
$$%9D
#* (ith heterogeneous %ausal ee%ts, %td&
Y
i
.
;
4
$i
X
i
4 u
i
(e'uation of interest
X
i
.
;
4
$i
Z
i
4 v
i
(first stage of &2>2
Intuition-
2uppose
$i
Js #ere kno#n. If for some people
$i
.
;" then their predicted value of X
i
#ouldnJt depend
on Z" so the I8 estimator #ould ignore them.
&he I8 estimator puts most of the #eight on
individuals for #hom Z has a large influence on X.
&2>2 measures the treatment effect for those #hose
probability of treatment is most influenced by X.
$$%?;
,he mathI
Y
i
.
;
4
$i
X
i
4 u
i
(e'uation of interest
X
i
.
;
4
$i
Z
i
4 v
i
(first stage of &2>2
&o simplify things" suppose-

$i
and
$i
are distributed independently of (u
i
"v
i
"Z
i

E(u
i
<Z
i
. ; and E(v
i
<Z
i
. ;
E(
$i
;
&hen
$
A
,&4&

)

$ $
$
(
(
i i
i
E
E

(derived in 2W App. $$.3

&2>2 estimates the causal effect for those individuals
for #hom Z is most influential (those #ith large
$i
.
$$%?$
When there are heterogeneous %ausal ee%ts, (hat
TSAS estimates depends on the %hoi%e o instrumentsC
With different instruments" &2>2 estimates different
#eighted averagesOOO
2uppose you have t#o instruments" Z
$
and Z
*
.
oIn general these instruments #ill be influential for
different members of the population.
oLsing Z
$
" &2>2 #ill estimate the treatment effect for
those people #hose probability of treatment (X is
most influenced by Z
$
o&he treatment effect for those most influenced by Z
$

might differ from the treatment effect for those most
influenced by Z
*
$$%?*
When does TSAS estimate the a+erage %ausal ee%t8
Y
i
.
;
4
$i
X
i
4 u
i
(e'uation of interest
X
i
.
;
4
$i
Z
i
4 v
i
(first stage of &2>2
$
A
,&4&

)

$ $
$
(
(
i i
i
E
E

&2>2 estimates the average causal effect (that is"

$
A
,&4&

)

E(
$i
if-
oIf
$i
and
$i
are independent
oIf
$i
.
$
(no heterogeneity in e'uation of interest
oIf
$i
.
$
(no heterogeneity in first stage e'uation
Qut in general
$
A
,&4&
does not estimate E(
$i
O
$$%?+
Example- ,ardiac catheterization
Y
i
. survival time (days for A:I patients
X
i
. received cardiac catheterization (or not
Z
i
. differential distance to ,, hospital
!'uation of interest-
&urvivalDays
i
.
;
4
$i
CardCath
i
4 u
i
7irst stage (linear )robability model-
CardCath
i
.
;
4
$i
Distance
i
4 v
i
7or #hom does distance have the great effect on the
probability of treatment?
7or those patients" #hat is their causal effect
$i
?
$$%?3
!'uation of interest-
&urvivalDays
i
.
;
4
$i
CardCath
i
4 u
i
7irst stage (linear )robability model-
CardCath
i
.
;
4
$i
Distance
i
4 v
i
&2>2 estimates the causal effect for those #hose
value of X
i
is most heavily influenced by Z
i
&2>2 estimates the causal effect for those for #hom
distance most influences the probability of treatment
What is their causal effect? ((We might as #ell go to
the ,, hospital" its not too much farther)
&his is one explanation of #hy the &2>2 estimate is
smaller than the clinical trial =>2 estimate.
$$%?5
1eterogeneous Causal Ee%ts: Summary
Eeterogeneous causal effects means that the causal (or
treatment effect varies across individuals.
When these differences depend on observable variables"
heterogeneous causal effects can be estimated using
interactions (nothing ne# here.
When these differences are unobserved (
$i
the average
causal (or treatment effect is the average value in the
population" E(
$i
.
When causal effects are heterogeneous" =>2 and &2>2
estimateI.
$$%?9
!AS (ith 1eterogeneous Causal Ee%ts
X is: 2elation "et(een X
i
and
u
i
:
Then !AS estimates:
binary E(u
i
<X
i
. ; effect of treatment on the
treated- E(
$i
<X
i
.$
X randomly assigned (so
X
i
and u
i
are independent
average causal effect E(
$i

general E(u
i
<X
i
. ;
#eighted average of
$i
"
placing most #eight on
those #ith large <X
i
6
X
<
X randomly assigned
average causal effect E(
$i

Wit#out #eterogeneit"$
$i
.
$
and
$
A

)

$
in all these
cases.
$$%??
TSAS (ith 1eterogeneous Causal Ee%ts
&2>2 estimates the causal effect for those individuals
for #hom Z is most influential (those #ith large
$i
.
What &2>2 estimates depends on the choice of ZOO
In ,, example" these #ere the individuals for #hom
the decision to drive to a ,, lab #as heavily
influenced by the extra distance (those patients for
#hom the !:& #as other#ise (on the fence)
&hus &2>2 also estimates a causal effect- the average
effect of treatment on those most influenced by the
instrument
oIn general" this is neither the average causal effect
nor the effect of treatment on the treated
$$%?B
Summary: Experiments and Quasi-Experiments
(SW Se%tion 11&D)
Experiments:
Average causal effects are defined as expected values
of ideal randomized controlled experiments
Actual experiments have threats to internal validity
&hese threats to internal validity can be addressed (in
part by-
opanel methods (differences%in%differences
omultiple regression
oI8 (using initial assignment as an instrument
$$%?D
Summary, %td&
Quasi-experiments:
Suasi%experiments have an (as%if) randomly assigned
source of variation.
&his as%if random variation can generate-
oX
i
#hich satisfies E(u
i
<X
i
. ; (so estimation
proceeds using =>2C or
oinstrumental variable(s #hich satisfy E(u
i
<Z
i
. ;
(so estimation proceeds using &2>2
Suasi%experiments also have threats to internal vaidity
$$%B;
Summary, %td&
T(o additional su"tle issues:
What is a control variable?
oA variable W for #hich X and u are uncorrelated"
given the value of W (conditional mean
independence- E(u
i
<X
i
"W
i
. E(u
i
<W
i

oE+am)le- 2&A@ T effect of teacher experience

#ithin their school" teachers #ere randomly
assigned to regularHreg4aideHsmall class
=>2 provides an unbiased estimator of the causal
effect" but only after controlling for school
effects.
$$%B$
Summary, %td&
What do =>2 and &2>2 estimate #hen there is
unobserved heterogeneity of causal effects?
In general" #eighted averages of causal effects-
oIf X is randomly assigned" then =>2 estimates the
average causal effect.
oIf X
i
is not randomly assigned but E(u
i
<X
i
. ;" =>2
estimates the average effect of treatment on the
treated.
oIf E(u
i
<Z
i
. ;" &2>2 estimates the average effect of
treatment on those most influenced by Z
i
.
$$%B*