Type I & Type II Errors,

Power of a Statistical Test, &

Effect Size
four of the most confusing topics in
introductory statistics packaged in a
way that will hopefully make them
clear
Nicole Radziwill, James Madison University
@nicoleradziwill / radziwnm@jmu.edu
Feel free to use & share with citation!

Type I & Type II Errors

When you perform a statistical test, youre only taking ONE
SAMPLE from a population and there are tons of different
samples you could potentially be collecting. You have to think
about your sample in the context of ALL the potential samples
you could have collected fortunately this is made easy
thanks to the sampling distributions of proportions and means.
Type I Error, also called , is the likelihood
that you incorrectly reject the null hypothesis
Type II Error, also called , is the likelihood
that you fail to detect the effect
that your alternative
hypothesis is trying
to uncover

The Pregnancy Test Example

H0: You are not pregnant.
HA: You ARE pregnant.
There are four things that can happen when you take the pregnancy
test:
The test can be ACCURATE, and say that:
Reality
1) youre pregnant when you actually are, OR
H False
H0 True
H00 False
2) youre not pregnant when you actually ARENT
(Really
Or it can be INACCURATE, and say that:
POS
3) youre pregnant when youre NOT
(Reject
What the
H0)
(a FALSE ALARM), orPregnanc
NEG
4) youre not
y
(Fail to
pregnant when
Test Said
Reject
you actually ARE.
H0)
(a FAILURE TO DETECT the effect)

(Really
(Really NOT
Are
Pregnant)
pregnant)

(Really ARE
NOT
pregnant)
pregnant)

The Pregnancy Test Example

H0: You are not pregnant.
HA: You ARE pregnant.
There are four things that can happen when you take the pregnancy
test:
The test can be ACCURATE, and say that:
Reality
1) youre pregnant when you actually are, OR
H False
H0 True
H00 False
2) youre not pregnant when you actually ARENT
(Really
Or it can be INACCURATE, and say that:
POS
3) youre pregnant when youre NOT
(Reject
What the
H0)
(a FALSE ALARM), orPregnanc
NEG
4) youre not
y
(Fail to
pregnant when
Test Said
Reject
you actually ARE.
H0)
(a FAILURE TO DETECT the effect)

(Really
(Really NOT
Are
Pregnant)
pregnant)

= probability
of
this test raising
a
FALSE ALARM

(Really ARE
NOT
pregnant)
pregnant)

= probability
of
this test
FAILING TO
DETECT your
pregnancy!

The Pregnancy Test Example

H0: You are not pregnant.
HA: You ARE pregnant.
If Type II Error is the probability that a statistical test performed
on this particular sample FAILS TO DETECT THE EFFECT
(pregnancy), then what is the probability that a test using this
sample will SUCCESSFULLY DETECT THE EFFECT?
Reality
P(successfully detecting the effect) =
1 P(NOT detecting the effect) = 1 -

POS

What the
This (1 - ) is called
Pregnanc
The POWER OF
y
Test Said
THE TEST!

(Reject
H0)

NEG

(Fail to
Reject
H0)

H0 True
(Really
(Really NOT
Are
Pregnant)
pregnant)

= probability
of
this test raising
a
FALSE ALARM

H False
H00 False
(Really
(Really ARE
NOT
pregnant)
pregnant)

= probability
of
this test
FAILING TO
DETECT your
pregnancy!

Consequences of Pregnancy
Test Errors
Type I Error The test raised a false alarm and got you
either very worried or excited. You may have already
run to the store to buy baby supplies, incurring
unnecessary costs. You may have spent days or weeks
panicking until you realized that the test was faulty.
Type II Error - The test failed to detect your
pregnancy and you didnt stop drinking or smoking,
therefore potentially harming a life. You didnt seek
prenatal medical attention.

In a case like this, the test designers need to think about

how to minimize BOTH Type I and Type II Errors. There
are psychological and cost ramifications if either kind
happens.

Getting Picked Up By the

Cops
The scenario: A cop pulls you over
and says he smells pot in your car. He
is trying to figure out whether or not to take you to jail. The cop needs
to do a little hypothesis test in his head to figure out whether to bring
you in.
H0: You are not guilty.
HA: You ARE guilty. You have been smoking pot in your car.
There are four things that can happen here:
Reality
The cop can be ACCURATE, and say that:
H0 True
H0 False
1) youre guilty when you actually are, OR
(Really NOT
(Really ARE
Guilty;pot
it was a
2) youre not guilty and you havent been
smoking
Guilty)
Im
SKUNK)

Taking
Or he can be INACCURATE, and say You
that:to
Jail
3) youre guilty when youre NOT

What the
(a FALSE ARREST), or
Cop
4) youre not
Decided
guilty when
you actually ARE.
(a FAILURE TO DETECT the effect
and a GET OUT OF JAIL FREE card)

(Reject
Im
H0)

Letting
You Go
(Fail to
Reject H0)

Getting Picked Up By the

Cops
The scenario: A cop pulls you over
and says he smells pot in your car. He
is trying to figure out whether or not to take you to jail. The cop needs
to do a little hypothesis test in his head to figure out whether to bring
you in.
H0: You are not guilty.
HA: You ARE guilty. You have been smoking pot in your car.
There are four things that can happen here:
Reality
The cop can be ACCURATE, and say that:
H0 True
H0 False
1) youre guilty when you actually are, OR
(Really NOT
(Really ARE
Guilty;pot
it was a
2) youre not guilty and you havent been
smoking
Guilty)
Im
SKUNK)

Taking
Or he can be INACCURATE, and say You
that:to
Jail
3) youre guilty when youre NOT

What the
(a FALSE ARREST), or
Cop
4) youre not
Decided
guilty when
you actually ARE.
(a FAILURE TO DETECT the effect
and a GET OUT OF JAIL FREE card)

(Reject
Im
H0)

Letting
You Go
(Fail to
Reject H0)

= probability
of
this test
leading to
FALSE ARREST

= probability
of
this cop
FAILING TO
DETECT your
pot smoking!

Consequences of Cop Errors

Type I Error The test raised a false alarm and got
you sent to jail even though you didnt deserve it. It
cost you time and maybe even cost you money
bailing yourself out, or fighting court fees!!
Type II Error - The cop let you go when he had
reason to send you to jail! Probably good for you,
bad for the cop (who may be trying to make his
arrest quota for the month). Possibly bad for society,
but other factors would have to be considered.
Probably best to focus on keeping the Type I Error as
low as possible in these cases its more
problematic to have a lot of false alarms than to let a
few pot smokers off free here and there.

Getting Picked Up By the

Cops
The scenario: A cop pulls you
over and #2
suspects that you have just
committed a murder. He is trying to figure out whether or not to take
you to jail. The cop needs to do a little hypothesis test in his head to
figure out whether to bring you in.
H0: You are not guilty.
HA: You ARE guilty. You are hiding evidence and a body in your trunk.
There are four things that can happen here:
Reality
The cop can be ACCURATE, and say that:
H0 True
H0 False
1) youre guilty when you actually are, OR
(Really NOT
(Really ARE
Guilty; you just
2) youre not guilty - you havent just killed
someone
Guilty)
Im
look sketchy)

Taking
Or he can be INACCURATE, and say You
that:to
Jail
3) youre guilty when youre NOT

What the
(a FALSE ARREST), or
Cop
4) youre not
Decided
guilty when
you actually ARE.
(a FAILURE TO DETECT the murder
and eluding the law)

(Reject
Im
H0)

Letting
You Go
(Fail to
Reject H0)

Getting Picked Up By the

Cops
The scenario: A cop pulls you
over and #2
suspects that you have just
committed a murder. He is trying to figure out whether or not to take
you to jail. The cop needs to do a little hypothesis test in his head to
figure out whether to bring you in.
H0: You are not guilty.
HA: You ARE guilty. You are hiding evidence and a body in your trunk.
There are four things that can happen here:
Reality
The cop can be ACCURATE, and say that:
H0 True
H0 False
1) youre guilty when you actually are, OR
(Really NOT
(Really ARE
Guilty; you just
2) youre not guilty you havent just killed
someone
Guilty)
Im
look sketchy)

Taking
Or he can be INACCURATE, and say You
that:to
Jail
3) youre guilty when youre NOT

What the
(a FALSE ARREST), or
Cop
4) youre not
Decided
guilty when
you actually ARE.
(a FAILURE TO DETECT the murder
and eluding the law)

(Reject
Im
H0)

Letting
You Go
(Fail to
Reject H0)

= probability
of
this test
leading to
FALSE ARREST

= probability
of
this cop
FAILING TO
DETECT the
murder!

Consequences of Cop Errors

(#2)
Type I Error The test raised a false alarm and got you
sent to jail and into a HUGE legal mess even though you
didnt deserve it. It costs you time and will probably
cost you tons of money bailing yourself out, court
fees, lawyers a trial!!
Type II Error - The cop let you go when you had a body
in the trunk! Probably good for you, bad for the cop, and
VERY BAD for society.
Id want to keep the Type II Error as low as possible in this
case, and risk some false alarms to AVOID letting killers
go. Would you?? (Notice that keeping the Type II Error
low ALSO keeps the power of the test pretty high.)

Power of the Test

Power = 1 -
Its the probability of successfully detecting an effect
The power of the test INCREASES as the effect size
increases
So girls, if you are just ONE day late, the effect size is
small it will be difficult for that pregnancy test to give
you an accurate result. Youll have a high Type II Error at
this time.
If you are a week late, the effect size is bigger the Type
II Error will be lower, because the pregnancy is easier to
detect.
If you are a month late, the effect size is HUGE and the
Type II Error will be even smaller.

Interrelationships
P0 is the proportion that
youre assuming is true
often a standard or
known value
P* is the TRUE population
proportion
The bigger the difference
between P0 and P*, the
bigger the EFFECT SIZE
(the bottom curve shifts
to the right as effect size
gets bigger)
If the proportion you
measured from your
sample is BIGGER than
the real proportion P*,
youll incorrectly reject
the null and incur a Type
I Error.

If the proportion you measured from your

sample is LESS than the real
proportion P*, but STILL GREATER than
the values in the top curve that are
way out to the left all on their own,
Youll fail to reject the null and incur a