ANOVA

A Scenario for ANOVA

 A scenario for ANOVA
• You want to compare UN scores of 3 schools (or
classes, etc).
– School A
– School B
– School C
• You can conduct a t-test for each pair of schools:
– School A versus School B
– School B versus School C
– School C versus School A
 You will need more than one t-test for multiple
comparison of pairs of means.
 Several t-tests for the multiple
comparison of pairs of means

• (1) Multiple t-tests will inflate the overall Type I error

rate. By performing several t-tests for pairs of means,
we capitalize on chance when we perform repeated
tests.
• (2) These t-tests are not independent! The multiple
tests of contrasts are not independent.

 ANOVA
 ANOVA
• ANOVA avoids this type of Type I error inflation by
conducting a single test whether the entire set of
sample means suggests that the samples were drawn
from the same general population.
• ANOAVA is used to determine the probability that
differences in means (across several groups) are due
solely to sampling error.

i.e., standard p-value

Like in t-tests
error
 ANOVA What about in the
previous NAPLAN
example?
• More flexible than t-test
– More than two groups
– More than one independent variable (i.e., Factor)
– Levels in factors can have many levels as desired.
– Remember: ANOVA still takes one dependent
variable!
 ANOVA
• Independent variables • Dependent variable
(IV): One or more (DV): A continuous
categorical variables variable
(e.g., gender, social (e.g., test scores,
class, school type, etc.).
treatment, education
program)

ANOVA (One-way, Two-way, Factorial)

 The Purpose of ANOVA
• To determine whether the means are significantly Same as
t-tests
different from each other.
• To determine whether the means of the dependent Same as
variable are significantly different from each other. t-tests
• To determine whether the means of the dependent
variable for an independent variable are significantly
different from each other. Same as
t-tests

• To determine whether the means of the dependent

variable for each level of an independent variable are
significantly different from each other. Same as
t-tests
 Hypothesis Testing in ANOVA

• Null hypothesis: There is no difference in group

means
• Alternative hypothesis: There is difference in
group means

Ho: μ1 = μ2 = μ3 = … = μk
HA: μ1 ≠ μ2 ≠ μ3 ≠ … ≠ μk
 ANOVA - post hoc tests
• Examination of the group means enables you to
assess the relative standing of each group on the
dependent measure.
• The F statistic test assesses the null hypothesis of
equal means, and it does not address the question
of which means are different.
• For example, in a three‐group situation, all three
groups may differ significantly, or two may be
equal but differ from the third.
• To assess these differences, the researcher can
employ either planned comparisons or post hoc
tests.
 The Purpose of ANOVA
• ANOVA can tell you whether:
– Groups scores differ from one another.
– Additional analysis can be done to find out:
• One group has a higher mean score than all the other
groups.
• One group has a higher mean score than another group.
 ANOVA: F-statistic
• F-statistic is the ratio of MSB to MSW.
• It is a measure of:
– how much variance is attributable to the different
groups versus the variance expected from
random sampling.
• The ratio of MSB to MSW is similar in concept
to the t statistic, but in this case gives us a
value for the F statistic.
 Essence of ANOVA
• An ANOVA is an examination of means (i.e., mean differences) based on
means themselves and variations from the means.

• Concept: If everyone did have the same score, then the overall mean
would have zero variance. When not everyone scores exactly the same,
then, it is possible to express each person’s score as a deviation from the
grand mean.

• ANOVA is based on a measure of variation, called sum of squares (SS).

• Definition of The Sum of Squares (SS)

– The Sum of Squares (SS) refers to the sum of the squares of the deviations
from a mean for an entire group.
– The Sum of Squares (SS) refers to the sum of the squared deviations about a
mean.
– The Sum of Squares (SS) refers to the sum of the squared deviations about a
mean, which is the primary component that is used to compute variance.
 Essence of ANOVA
• Definition of The Sum of Squares (SS)
– The Sum of Squares (SS) refers to the sum of the
squares of the deviations from a mean for an entire
group.
Student1 4 -1 1
Student2 3 -2 4
Student3 6 1 1
Student4 4 -1 1
Student5 7 2 4
Student6 3 -2 4
Student7 7 2 4
Student8 6 1 1
Student9 2 -3 9
Student10 8 3 9

5 0 38
Mean Deviation SS
 Essence of ANOVA
• The ANOVA seeks to determine how much total
variation can be explained by the independent
variable (or group identity), by dividing the
total variance (i.e., sum of squares, SST) into
two parts:
a) the variance attributable to the variables in the
study (between sums of squares, SSB)
b) the variance attributable to variables not included
in the study (within sums of squares, SSW)
 Essence of ANOVA
• The elements of an ANOVA
– Differences between a group mean and the grand
mean ( SSB )
– Differences between an individual’s score and his or
her group means (SSW)
– Differences between an individual’s score and the
grand mean (SST)

• ANOVA tests whether the amount of variance

explained by the independent variable (SSB) is a
significant proportion relative to the variance that
has not been explained (SSW).
 Essence of ANOVA

Experimental Group Control Group

Test score of Test score of

Experimental What you want to see Experimental
Group Group
 Essence of ANOVA
Within-group Within-group
variation in the Error Error variation in the
scores scores

Experimental Group Control Group

Test score of
Test score of
Experimental What you want to see Control Group
Group
Essence of ANOVA

Without the membership of the groups!

 Essence of ANOVA

Experimental Group Control Group

Test score of Test score of

Experimental Experimental
Group Group
 Within-groups Variation
• Within-groups variation: The extent to which
individuals within a group differ from each other
within their own group.
• It is expressed by the sum of squared deviations of
their individual scores from their own group mean.

• The group differences are kept constant

– The mean of these within-groups variations gives an
idea of the differences between the individuals, without
this having anything to do with the group identity.
 Between-groups Variation
• Between-groups variation: The extent to which
groups differ from each other.
• It is expressed by the sum of squared deviations
of the group means from the grand mean. (You
need to calculate the group means, first, of
course, to calculate the deviation of the group
means from the grand mean.)

• A between-groups variation gives an idea of the

differences between the groups, without this
having anything to do with the individuals. If this
variation is multiplied by the group size, we
obtain the between-groups variation.
 Between-groups Variation

Student1 4 -1 1 Control
Student2 3 -2 4 Control
Student3
Student4
6
4
1
-1
1
1
Control
Control
4.8 (4.8) - (5) = -0.2 0.04
Student5 7 2 4 Control
Student6 3 -2 4 Experiment
Student7 7 2 4 Experiment
Student8
Student9
6
2
1
-3
1
9
Experiment
Experiment
5.2 (5.2) - (5) = 0.2 0.04
Student10 8 3 9 Experiment

5 0 38 5 0.08
Mean Deviation SS

SSB
0.08 x 5 = 0.4 (between sums of
squares)
 Within-groups Variation

Student1 4 Control -0.8 0.64

Student2 3 Control -1.8 3.24
Student3 6 Control 4.8 1.2 1.44
Student4 4 Control -0.8 0.64
Student5 7 Control 2.2 4.84
Student6 3 Experiment -2.2 4.84
Student7 7 Experiment 1.8 3.24
Student8 6 Experiment 5.2 0.8 0.64
Student9 2 Experiment -3.2 10.24
Student10 8 Experiment 2.8 7.84

5
Mean
0 38
Deviation SS
37.6
SSW
(Within sums of
squares)
 SST = SSB + SSW

38 = 0.4 + 37.6

SST SSB SSW

(Total sums of (between sums of (Within sums of
squares) squares) squares)
 Total Variation
• Total variation is the sum of squared
deviations of individual scores from the grand
mean.
• This gives an idea of the differences that have
to do with both individuality as well as the
group identity.
• Total variation = the between-groups variation
+ the within-groups variation
 Total Variation

Student1 4 -1 1 Control
Student2 3 -2 4 Control
Student3 6 1 1 Control
Student4 4 -1 1 Control
Student5 7 2 4 Control
Student6 3 -2 4 Experiment
Student7 7 2 4 Experiment
Student8 6 1 1 Experiment
Student9 2 -3 9 Experiment
Student10 8 3 9 Experiment

5 0 38
Mean Deviation SS

SST
 Variation & Variance

• Variation divided by N (technically, degrees of

freedom)  Variance
 Degree of Freedom
• The between-groups variation indicates that part
of the differences that is exclusively due to the
groups. In dividing the between-groups variation
by the number of degrees of freedom, we obtain
the between-groups variance.
• The within-groups variation indicates that part of
the differences that is exclusively due to the
individuals. In dividing the within-groups
variation by the number of degrees of freedom,
we obtain the within groups variance.
 Degree of Freedom
• Degree of freedom for the between-groups variance:
• dB = number of groups – 1
• dB = number of groups – Number of Grand mean
• dB = (k – 1) degrees of freedom when the number of groups is k.
• If you have 3 groups,
– dB = 3-1  3 groups and the grand mean is fixed to 1.
• Degree of freedom for the within-groups variance:
• dW = k(n -1) = kn – k = N - k
• dW = number of groups multiplied by (number of individuals
within the group - 1).
• If you have 3 groups, and there are 8 individuals within the
group,
– dW = 3(8 - 1) = 21, because for each of the three groups there are
eight individuals, for which the group mean is fixed.
• F-tests are based on (k – 1) and (N – k) degrees of freedom.
 ANOVA: F-statistic
• ANOVA F-test is based on:
F-statistic, F-ratio, F-score, F-value…
Variation = Sum of Squares

SSB / dfB

SSW / dfW
Variation = Sum of Squares
 ANOVA

• ANOVA = ANalysis Of VAriance

• Variances (estimated) of the dependent variable of
the groups are compared.
• Estimates of variances are divided into:
– Between-groups estimate of variance
– Within-groups estimate of variance
• The total variance is partitioned into the within-
groups variation and the between-groups variation.
– TOTAL = WITHIN + BETWEEN
 Between‐groups of variance
• Between‐groups estimate of variance (MSB:
mean square between groups): The estimate of
variance is the variability of the group means on
the dependent variable. It is based on deviations
of group means from the overall grand mean of
all scores.
– Under the null hypothesis of no group difference (i.e.,
μ1 = μ2 = μ3 = … = μk), this variance estimate reflects
any group effects that exist; that is, differences in
group means increase the expected value of MSB.
– Note that any number of groups can be
accommodated.
 Within‐groups of variance
• Within‐groups estimate of variance (MSW: mean
square within groups): The estimate of the
average respondent variability on the dependent
variable within the groups. It is based on
deviations of individual scores from their
respective group means.

– MSW is comparable to the standard error between

two means calculated in the t test as it represents
variability within groups. The value MSW is sometimes
referred to as the error variance.
ANOVA

Grand mean
 ANOVA

Within groups mean Within groups mean

 ANOVA

Between‐groups of variance
 ANOVA

Within‐groups of variance
 ANOVA

• F-statistic in ANOVA: The differences between

groups are compared with the differences
between individuals within their own groups.
 ANOVA: F-statistic
• The ratio of MSB to MSW is a measure of how
much variance is attributable to the different
groups versus the variance expected from
random sampling.
 ANOVA: Interpreting F-statistic
• F-test is based on F-statistic, F-ratio, F-score, F-
value…
Differences between
groups

Differences between
individuals within each group

• F = 1 : the groups differ as much as the

individuals already differ from each other. The
groups have no effect on the dependent variable.
• F > 1 : the variance of groups rises above the
variance of individuals within each group.
 ANOVA: F-statistic
• Differences between the groups increase MSB
large values of the F statistic  lead to
rejection of the null hypothesis of no
difference in means across groups.

– If the analysis has several different

factors/treatments (independent variables), then
estimates of MSB are calculated for each
factor/treatment and F statistics are calculated for
each treatment. This approach allows for the
separate assessment of each treatment.
 ANOVA: Interpreting F-statistic
• Hypothesis testing for F-test follows a process
similar to the t test.
• To determine whether the F statistic is
sufficiently large to support rejection of the
null hypothesis.
F-distribution
 ANOVA: F-tests
Step 1. Calculate/obtain the F statistic.
Step 2. If the p-value attached to the calculated
F statistic exceeds the significance level
(acceptable level, error rate, often 0.05 level) ,
you can reject the null hypothesis that the
means across all groups are equal.
 F-tests
• Decomposition of the total variation of the dependent variable into the between- and
within-variations.
1

• Divide the between- and within-variations by the numbers of degrees of freedom to

obtain the between- and within-variances.
2

• Calculate F-statistics (the ratio of the between- and within-variance) in order to

compare the group differences with the differences among individuals within each
3 group.

• Obtain a p-value associated with the observed F-value

4

• Compare the p-value (based on the calculated F-value) to a significance level (or a
critical value of F).
5
 Results of F-tests
• The probability of finding an F-value of xxxx
(or an even more extreme value) under Ho is
xxxx.
• If this probability (p-value) is very low (e.g.,
less than a alpha level = 0.05), then we can
reject the null hypothesis.
• If this probability (p-value) is not very small
(e.g., greater than a alpha level = 0.05), then
we cannot reject the null hypothesis.
 The probability of
observing an F-ratio of
1.638 or more given the
null hypothesis is true is
0.203.
p-values

In the SPSS computer output, we do not find a critical F* value, but rather
the expression of the associated p-value (“Sig”). This is the empirical
significance level, which represents a probability of finding an F-ratio (or
an even more extreme value) under Ho.
 Results of F-tests
• You set a significance level, 0.05  The probability
of 0.05.
• When the p-value is less than 0.05 (i.e.,
significance level of 0.05), you reject the null
hypothesis and concludes that the groups scores
differ.
• When the p-value is greater than 0.05 (i.e.,
significance level of 0.05), you cannot reject the
null hypothesis and concludes that the groups
scores do not differ.
 ANOVA &
Sample size
 ANOVA & Sample size
• Sample size requirements from increasing either
the number of levels or the number of factors
• For each group, a sample size should be at least
about 20 or so observations.
• Thus, increasing the number of levels in any
factor requires an increase in sample size. Also,
analyzing multiple factors can create a situation
of large sample size requirements rather quickly.
• When two or more factors are included in the
analysis, the number of groups formed is the
product of the number of levels, not their sum.
 ANOVA & Sample size
School 1 School 2
25 25 50
Gender School 1 School 2
Male
Female
25
25
25
25
100
SES School 1 School 2
Low
Medium
25
25
25
25
150
High 25 25

By adding one factor with two levels and

three levels
 Sample size in Experiments
• Researchers should be careful when
determining both the number of levels for a
factor as well as the number of factors to be
included, especially in controlled experimental
settings where the necessary sample size per
cell is much more difficult to achieve!
Assumption in ANOVA: Independence

• Critical assumption of ANOVA (or MANOVA)

requires that the dependent measures for
each respondent be totally uncorrelated with
the responses from other respondents in the
sample.
• A lack of independence severely affects the
statistical validity of the analysis unless
corrective action is taken.
 A Final Note on the General F-test
• This F-test only confirms the existence of
significant differences among the group means.
• It does not follow necessarily that every subset of
two means shows a significant difference.
• The latter is investigated by means of separate
tests (multiple comparison tests), in which the
testing is more conservative (i.e., smaller than the
usual alpha).
ANOVA: F-tests
using
Critical Values
 ANOVA: F-tests
• As with the t test, a researcher can use certain
F values as general guidelines when sample
sizes are relatively large.

• These values are just the tcrit value squared.

Critical Values in t-statistic & F-statistic
 ANOVA: F-tests
Step 1. Determine the critical value for the F statistic
(Fcrit) by referring to the F distribution with (k – 1) and (N
– k) degrees of freedom for a specified significance level
(where N = N1 + .. . + Nk and k = number of groups).
Step 2. Calculate the F statistic (or use computer
software).
Step 3. If the value of the calculated F statistic exceeds
the critical value of F (Fcrit), conclude that the means
across all groups are not all equal. (Again, the computer
programs provide the F value and the associated
significance level, so you can directly assess whether it
meets an acceptable level.)
Data name: “Simple Data for ANOVA”
 One-way ANOVA

Conduct one-way ANOVA on Test1 test scores by

Students’ Birth month