See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/313966026

Generalizability Theory

Chapter · January 2015

DOI: 10.1002/9781118625392.wbecp352

CITATIONS READS
0 809

4 authors, including:

Richard J. Shavelson
Stanford University
260 PUBLICATIONS   17,911 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Positive Learning in the Age of Information (PLATO) View project

G Theory View project

All content following this page was uploaded by Richard J. Shavelson on 28 December 2017.

The user has requested enhancement of the downloaded file.

1322 GENERALIZABILITY THEORY
Caspi, A., & Moffitt, T. E. (2006).
Gene-environment interactions in psychiatry:
Joining forces with neuroscience. Nature Reviews
Neuroscience, 7, 583–590.
Caspi, A., Moffitt, T. E., Cannon, M., McClay, J.,
Murray, R., Harrington, H., … Craig, I. W.
(2005). Moderation of the effect of
adolescent-onset cannabis use on adult psychosis
by a functional polymorphism in the
catechol-O-methyltransferase gene: Longitudinal
evidence of a gene X environment interaction.
Biological Psychiatry, 57, 1117–1127.
Caspi, A., Sugden, K., Moffitt, T. E., Taylor, A.,
Craig, I. W., Harrington, H., … Poulton, R.
(2003). Influence of life stress on depression:
Moderation by a polymorphism in the 5-HTT
gene. Science, 301(5631), 386–389.
Cicchetti, D., & Rogosch, F. A. (2012). Gene ×
environment interaction and resilience: Effects of
child maltreatment and serotonin, corticotropin
releasing hormone, dopamine, and oxytocin
genes. Development and Psychopathology, 24,
411–427.
Karg, K., Burmeister, M., Shedden, K., & Sen, S.
(2011). The serotonin transporter promoter
variant (5-HTTLPR), stress, and depression
meta-analysis revisited: Evidence of genetic
moderation. Archives of General Psychiatry,
68(5), 444–454.
Kim-Cohen, J., Caspi, A., Taylor, A., Williams, B.,
Newcombe, R., Craig, I. W., & Moffitt, T. E.
(2006). MAOA, maltreatment, and
gene-environment interaction predicting
children’s mental health: New evidence and a
meta-analysis. Molecular Psychiatry, 11,
903–913.
Kim-Cohen, J., & Turkewitz, R. (2012). Resilience
and measured gene-environment interactions.
Development and Psychopathology, 24,
1297–1306.
Mehta, D., & Binder, E. B. (2012). Gene ×
environment vulnerability factors for PTSD: The
HPA-axis. Neuropharmacology, 62, 654–662.
Meyer-Lindenberg, A., & Weinberger, D. R. (2006).
Intermediate phenotypes and genetic
mechanisms of psychiatric disorders. Nature
Reviews Neuroscience, 7, 818–827.
Pelayo-Teran, J. M., Suarez-Pinilla, P., Chadi, N., &
Crespo-Facorro, B. (2012). Gene-environment
interactions underlying the effect of cannabis in
first episode psychosis. Current Pharmaceutical
Design, 18, 5024–5035.
Risch, N., Herrell, R., Lehner, T., Liang, K., Eaves,
L., Hoh, J., … Merikangas, K. (2009). Interaction
between the serotonin transporter gene
(5-HTTLPR), stressful life events, and risk of
depression: A meta-analysis. Journal of the
American Medical Association, 301(23),
2462–2471.
van Winkel, R., Stefanis, N. C., & Myin-Germeys, I.
(2008). Psychosocial stress and psychosis: A
review of the neurobiological mechanisms and
the evidence for gene-stress interaction.
Schizophrenia Bulletin, 43, 1095–1105.
Wong, M. Y., Day, N. E., Luan, J. A., Chan, K. P., &
Wareham, N. J. (2003). The detection of
gene-environment interaction for continuous
traits: Should we deal with measurement error by
bigger studies or better measurement?
International Journal of Epidemiology, 32(1),
51–57.
Further Reading
Caspi, A., Hariri, A. R., Holmes, A., Uher, R., &
Moffitt, T. E. (2010). Genetic sensitivity to the
environment: The case of the serotonin
transporter gene and its implications for studying
complex diseases and traits. American Journal of
Psychiatry, 167, 509–527.
Monroe, S., & Reid, M. (2008). Gene-environment
interactions in depression research: Genetic
polymorphisms and life-stress polyprocedures.
Psychological Science, 19(10), 947–956.
Generalizability Theory
Min Li,1 Richard J. Shavelson,2,3
Yue Yin,4 and Ed Wiley2
1 University of Washington, Seattle, U.S.A., 2 SK Partners,
U.S.A., 3 Stanford University, U.S.A., and 4 University of
Illinois, Chicago, U.S.A.
Generalizability (G) theory is a psychome-
tric theory based on a statistical sampling
approach that partitions scores into their
underlying multiple sources of variation. G
theory was initially introduced by Cronbach
and his colleagues (Cronbach, Rajaratnam, &
Gleser, 1963) as an extension of Cronbach’s
classic paper on internal-consistency alpha

reliability (Cronbach’s alpha; Cronbach, 1951)
and further developed by others (Brennan,
2000, 2001; Cronbach, Gleser, Nanda, &
Rajaratnam, 1972; Shavelson & Webb, 1981,
1991; Webb, Shavelson, & Haertel, 2006). It
addresses the consistency or dependability
of score interpretations by recognizing and
estimating the magnitude of measurement
error from multiple sources.
This entry describes the theoretical foun-
dation and applications of G theory for an
audience of clinical psychologists, not psy-
chometricians. An example is introduced to
discuss the basic concepts of G theory, an
overview of G theory, and its applications.
One-facet and multifacet designs are then
described, within which all the basic con-
cepts introduced will be presented along with
examples. The entry concludes with a summary
of important topics around the use of G theory
and calls attention to additional resources.
Conceptual Underpinnings of G
Theory
G Theory as a Sampling Approach
When using a psychological or behavioral
instrument to measure phenomena of inter-
est (e.g., bullying, anxiety, racial identity,
belonging, or learning disability), clinical psy-
chologists typically focus on obtaining a stable,
consistent estimate of a patient’s level on that
scale, for diagnosis of severity and treatment.
The core of G theory lies in the notion that
an instrument can likely include only a small
sample of items or tasks from the much larger
universe of items or tasks that could have been
included instead. Our ultimate goal in admin-
istering a test is to gain enough information
from this limited sample to generalize across
the larger universe. Our success in generalizing
from limited sample to broad universe depends
on the nature and magnitude of error encoun-
tered due to our ability to sample only a small
part of the targeted universe. G theory provides
an approach for pinpointing magnitudes of
measurement error that make the generaliza-
tion from sample to universe hazardous, and
GENERALIZABILITY THEORY 1323
it provides an estimate of the consistency of
such generalization by examining the extent to
which scores vary due to person and to various
sources of measurement error.
Four sets of interrelated concepts are criti-
cal for understanding G theory: (a) universe
defined by facet(s), (b) partitioning of measure-
ment error, (c) relative and absolute decisions,
and (d) generalizability study and decision
study. In what follows, these concepts are intro-
duced by situating them in a study of measures
of disaster survivors’ psychological impair-
ment (Gleser, Green, & Winget, 1978). Twenty
adult survivors from a dam disaster were inter-
viewed independently by two interviewers.
Each interview was then independently scored
by three raters for the extent of long-term
psychiatric impairment on three subdimen-
sions (i.e., anxiety, depression–isolation, and
hostility–belligerence) and one overall scale,
based on the Psychiatric Evaluation Form with
a scale from 1 (none) to 6 (extreme).
Universe Defined by Facet(s)
G theory addresses the fact that scores assigned
to persons vary not only due to person
differences in the personality, behaviors, symp-
toms, abilities, or skills measured, but also
due to various sources of measurement error
associated with “facets” of the measurement
(e.g., interviewers and scorers in the example).
The variance attributable to person differences
is assumed to be “wanted” or “expected” score
variation—a person’s universe scores. A uni-
verse score is defined as an expected value of
all the observations on a person across the
universe of possible options; it is equivalent to
a true score in classical test theory. Universe
or universe of generalization is then defined by
specifying admissible facets of a measurement
without changing the construct of interest, and
levels for each facet that can be considered
exchangeable or parallel. Facet refers to each
of the characteristics (factors) used to generate
or define the universe of observations, such
as test item, survey form, rater, and occasion.
Each facet has a set of varying yet comparable
levels or categories, called levels or conditions.
1324 GENERALIZABILITY THEORY
A particular measurement procedure is then
a combination of specific facet conditions
sampled from such a universe.
In the survivor study (Gleser et al., 1978),
three raters scored the interviews of 20 sur-
vivors conducted by two interviewers. Making
use of two facets, interviewer and rater, this
study is a two-facet design of survivor (s),
interviewer (i), and rater (r), denoted s × i × r.
The universe of generalization is constituted
by an infinite number of interviewers and by
all the raters who can quantify the interviews.
A universe score for a particular survivor is
then the expected value of all the admissible
observations of this person from the universe
of generalization. The particular measurement
sampled from the universe contains two levels
at the interviewer facet and three levels at the
rater facet. The interviewers and the raters
defined in the universe are not simply any
random interviewers or raters. Instead, the
universe of interviewers should be regarded
as comparable to the two study interviewers
concerning, for example, the type and amount
of training received, and the interview prompts
used. The raters should be comparable to the
three raters sampled in the study in terms
of, for example, scoring experience, type and
amount of training received, and familiarity
with the measured construct.
Partitioning of Measurement Error
G theory offers a framework to conceptualize
and estimate multiple sources of score variance.
One critical step in applying G theory is to par-
tition the variance of observed scores into the
variance due to person and the measurement
error due to the main and interaction effects of
multiple sources (facets).
In the survivor study, the survivor is the
object of measurement, and measurement
error may arise from inconsistencies due to
the two facets (interviewer and rater). More
specifically, the measurement error can be
attributed to sampling errors with inter-
viewers, raters, their interactions, and other
unspecified sources. For instance, measure-
ment error associated with interviewers might
represent the fact that, over all 20 interviews,
one interviewer tended to elicit informa-
tion indicating systematically higher levels of
impairment than did the other interviewer.
G theory, supported by statistical procedures,
allows simultaneous estimation of the variance
due to each source, which is called a variance
component.
Relative Decisions and Absolute
Decisions
In G theory, the reliability coefficient is defined
as the proportion of score variance attributable
to person with respect to the total variance
due to both person and various sources of
measurement error. G theory recognizes that
decision makers using a measurement, such as
psychologists, clinicians, researchers, parents,
policy makers, and managers, may want to
make two types of decisions with the scores:
relative and absolute. Corresponding to these
two types of decisions, different G coefficients
can be computed.
Relative decisions are associated with
norm-referenced interpretations of scores.
The decisions concern the consistency of
scores in ranking or sorting individuals
according to differences in their personal-
ity, behaviors, knowledge, and/or skills. For
example, relative decision is involved when
we examine whether different items measur-
ing ability of matching familiar figures rank
patients consistently for impulsivity level, or
whether scores from different interviewers
judged by different raters sort survivors con-
sistently for their psychiatric impairment. In
contrast, absolute decisions are associated with
criterion-referenced or domain-referenced
interpretations of scores. They concern index-
ing the exact or absolute level of individuals’
construct for a given domain regardless of
the performance of others, such as driving
tests in everyday life. The measurement deci-
sions are made based on the absolute level
of scores from varying items regardless of
the relative ranking among the measured
persons.

Generalizability Studies and Decision
Studies
G theory distinguishes between generalizability
(G) studies and decision (D) studies. The pur-
pose of a G study is to estimate components of
score variance associated with various sources.
A D study takes these estimated variance com-
ponents to evaluate and optimize among alter-
natives for subsequent measurement.
Let us turn again to the survivor study.
Variance components can be estimated from
the G study. They indicate the magnitude
of estimated variance associated with the
measurement procedure using only one
interviewer and one rater, which is similar
to a single-measure intraclass correlation
coefficient. These magnitudes-of-variance
components then form the basis for a series of
D studies to make decisions for future measure-
ment procedures as the sampling of some facets
is changed. For instance, one might be inter-
ested in reducing the number of interviewers
but increasing the number of raters to maintain
the reliability of the measurement of psychic
injury (n′i = 1 and n′r = 5 instead of ni = 2 and
nr = 3). Variance components and reliability
coefficients can be calculated to determine
whether this hypothesized procedure meets
the reliability requirement and identify the
minimum number of interviewers and the
minimum number of raters needed to obtain
dependable scores.
Designs Used in G and D Studies
There are a variety of designs that decision
makers can select from when conducting their
G and D studies. These designs are specified
with respect to three important properties
of facets. They are pertinent to theoriza-
tion of sampling the universe and therefore
generalization of score interpretations.
First, designs can be crossed or nested when
facets are either crossed or nested with the
object of measurement (person, in most cases)
or other facets. Crossed refers to the designs
in which all the sampled levels of any facet
are present for all persons and for all levels of
any other facet(s). In contrast, nested refers to
GENERALIZABILITY THEORY 1325
the fact that only some of the sampled levels
of a given facet are present for each level of
the other facet(s). The survivor study is a fully
crossed s × i × r design as this measurement
procedure has all the persons interviewed by
the two interviewers and then scored by four
raters. But suppose that, for practical reasons,
each participant is interviewed by different sets
of two interviewers using the same interview
protocol, but the interview responses are ana-
lyzed by four raters. In this case, interviewer
is nested within survivor (which is denoted as
i : s, where “:” is read as “nested with”) because
not all of its sampled levels in the interviewer
facet are represented with scores for all the
survivors. This particular design, denoted as
(i : s) × r, considers the following five sources of
score variation: s; r; i : s; sr; and ir : s, e. Some-
times, person as the object of measurement
can be nested within a facet. For example, a
researcher may examine the use of an obser-
vation protocol on individuals’ interaction
with peers in the group therapy setting where
clients are nested within a group observed by
two raters on five occasions. This design can
be represented as (Person : Group) × Rater ×
Occasion.
Second, designs can be random or mixed
models. Random models have only random
facets, whereas mixed models have both ran-
dom and fixed facets. G theory is essentially
a random effects theory. Typically, a random
facet is generated if the levels of the studied
measurement are randomly sampled from all
the levels of a facet. A facet is also considered as
random when the intended universe of gener-
alization for this facet is infinitely large and the
levels included in the universe are exchange-
able to the selected levels even though these
levels have not been randomly sampled, for
example, the facets of interviewers and raters.
In contrast, a decision maker may consider
a facet as fixed when (a) the universe of gen-
eralization is rather limited or finite, so that
all the levels of the facet are included in the
studied measurement; or (b) the studied levels
involved in the measurement are deliberately
selected based on theoretical reasons, and the
1326 GENERALIZABILITY THEORY
researcher is not interested in generalizing to
any other levels. Suppose that a school district
collaborates with a counseling psychologist
to introduce and examine the reliability of
a self-efficacy measure that involves judg-
ments made by raters with students’ responses.
The district uses three well-trained special-
ists as raters. In this example, rater is likely
treated as a fixed facet because the district
does not plan to expand the pool of raters
beyond the set of three individuals who have
been trained by the psychologist. However,
if the district anticipated recruiting more
staff members as raters in the future, then
rater would have to be treated as a random
facet.
A third characteristic of designs deals with
hidden or implicit facets. Hidden facets stem
from the fact that (a) only one particular
level of the facet is represented in a G study
design with a clearly articulated goal of making
the measurement procedure manageable and
controlling other facets that may contribute
to the measurement error; or (b) the facets
are confounded with other facets or objects of
measurement but are not recognized. In both
scenarios, the hidden facets are not explicitly
planned, recognized, or represented in the
designs. An example of the first scenario is
that when persons respond to the impulsivity
test, the items are administered in English,
in a paper-and-pencil mode, presented in a
particular layout and color of paper, and on
a particular occasion. The facets of language,
mode, layout, paper color, and occasion thus
are hidden because only one level of each is
represented in the design, although they are not
explicitly specified. It is impossible to estimate
the amount of measurement error attributable
to hidden facets (Cronbach, Linn, Brennan,
& Haertel, 1997; Shavelson, Ruiz-Primo, &
Wiley, 1999). The estimated coefficients from
such data do not take into account the pos-
sibility that persons’ scores might vary on
other levels of any of those hidden facets and,
consequently, overestimate generalizability
when researchers want to generalize over
languages, modes, layouts, paper color, or
occasions.
In the second scenario, in which a hidden
facet is confounded with another source of
variance (e.g., the object of measurement),
the levels of this hidden facet vary as the
levels of the other source of variance vary.
For example, the facet of item order is often
confounded with the item facet in a typical
p × i design, as items appear in different orders
(or the respondents may choose to complete
the measurement in different orders). As
another example, when interview questions
elicit persons’ personal accounts to describe
their perceived stress, the selected account
or anecdote becomes a hidden facet. It is
confounded with the object of measurement
(person) because individuals may recall very
different anecdotes, which makes it impossible
to disentangle the main effect of person and
the anecdote effect. Again, the estimated coef-
ficients from the data of this design could lead
to an overestimation of generalizability when
a decision maker wants to generalize over
anecdotes, because the design fails to consider
the possibility that persons’ stress scores might
vary on other levels of the anecdote selection
facet.
Differences in designs lead to variation in
procedures used to decompose and estimate
the variance components, and subsequently
calculate the measurement error and reli-
ability coefficients. The next two sections
focus on describing designs that consist of
one or multiple facets that are crossed or
nested.
One-Facet Design
G theory operationalizes a measurement as a
procedure that can result in a sample from a
universe of admissible observations. Studies of
one-facet design involve only one facet, and the
universe is defined by a single random facet. If
a psychologist plans to generalize from clients’
responses on one occasion to a much larger set
of occasions, occasion is a facet. The universe
would be defined by all admissible occasions
(e.g., every morning). If a manager wants to

generalize from employees’ performance rated
by two raters to a much larger set of raters,
rater is a facet. The universe would be defined
by all admissible raters (e.g., all the raters who
have the expertise to rate the employees).
Using the impulsivity test as an example,
decision makers are typically indifferent to
the particular test score on one set of items.
Instead, they would be quite willing to sub-
stitute another set of items to measure the
construct that they are interested in. In other
words, the decision makers are more interested
in an examinee’s general responses to any items
from the universe of generalization rather than
that examinee’s score on any particular set of
items.
A one-facet p × i design has four sources
of variability. The first source is due to the
systematic differences among persons, the
object of measurement. It reflects individuals’
personality, attitudes, behaviors, performance,
knowledge, skills, and so on. The second
source of variability is due to the difference in
test items. Some items are easy, some difficult,
and some in between for all the individuals.
The third source of variability is due to the
interaction between individuals and items.
For example, an item testing individuals’ skills
of addition in the context of money would
be easier for a student who has received an
allowance from parents than one who has
not. The fourth source of variability, which
is usually confounded with the third source,
may be due to randomness (e.g., lucky guesses
or careless mistakes), other systematic but
unidentified or unknown sources of variability
(e.g., individuals taking the test on different
occasions or in different classroom settings),
or both.
Once the magnitude of each source of score
variation is estimated, the measurements
already made in the G study can be evaluated.
In addition, D studies can be conducted to
redesign measurements in order to reach a
desirable reliability (e.g., lengthening the test
and hiring more raters). For a one-faceted
universe, two designs could be used in G and
D studies to estimate variance components: the
GENERALIZABILITY THEORY 1327
one-facet crossed design and one-facet nested
design.
One-Facet Crossed Design
In the one-facet crossed design, each person is
given the same random sample of items. The
design is denoted as p × i, where p refers to
person (or examinee), i refers to item, and ×
means “crossed with.” The statistical model
used to estimate variance components in the
one-facet crossed design is described next.
Statistical model. The observed score for
any person on any item is denoted as Xpi . The
universe score, denoted as μp , is defined as
a person’s average score over the entire item
universe, Ei (Xpi ), where the symbol E is an
expectation operator and the subscript i desig-
nates that the expectation is taken over items.
Similarly, μi is the population mean for item
i, defined as Ep (Xpi ), the expected value of Xpi
over persons. μ is the grand mean over both
the population and universe, Ep Ei (Xpi ).
An observed score for one person on one
item (Xpi ) can be decomposed as follows:
Xpi = μ[grand mean]
+ (μp − μ) [person effect]
+ (μi − μ) [item effect]
+ (Xpi − μp − μi + μ)[person × item
interaction confounded with residual]
(1)
Each effect other than the grand mean in
Equation 1 has a distribution with a mean
of zero and its own variance σ2 , called the
variance component. For example, the vari-
ance component for person effect is defined as
σ2p = Ep (μp − μ). 2 That is, it is the average of
the squared deviations of the persons’ universe
scores from the grand mean. The variance
component for items can be defined similarly
as σ2i . The final effect, the p × i interaction
confounded with residual, also has a mean
of zero and a variance denoted as σ2pi,e . This
effect includes the p × i interaction effect, which
reflects the fact that not all people find the same
1328 GENERALIZABILITY THEORY

items easy or difficult, confounded with unsys-
tematic variability and unspecified variability
from the hidden facets not explicitly included
or controlled in the one-facet G study because
with only one observation per cell in the per-
son × item data matrix, the interaction cannot
be disentangled from “within-cell” variability.
In sum, the variance of the collection of
observed scores, Xpi ; overall persons; and items
in the universe is expressed as the sum of the
three variance components:
σ2X = σ2p + σ2i + σ2pi,e (2)
pi
That is, the variance of item scores can be par-
titioned into independent sources of variation
due to differences between persons as objects of
measurement, σ2p , and two sources of measure-
ment error: variation due to item, σ2i , and vari-
ation due to the p × i interaction confounded
with the residual, σ2pi,e .
In the one-facet crossed design, the gen-
eralizability coefficient (ρ2 ), calculated as
σ2p ∕(σ2p + σ2Rel. ), pertains to relative decisions.
Specifically, the variance component, σ2pi,e ,
contributes to relative measurement error
because the interaction between person and
item influences the relative standing of indi-
viduals. In contrast, the variance component
due to item effect, σ2i , does not contribute
to relative measurement error because the
variance in item difficulty, which influences
everybody, does not influence the ranking
of individuals. The dependability coefficient
(Φ) is calculated as σ2p ∕(σ2p + σ2Abs. ) for abso-
lute decisions. Here, all variance components
except the object of measurement contribute to
the absolute measurement error, that is, σ2i and
σ2pi,e in this one-facet crossed design. Usually,
the generalizability coefficient is larger than or
equal to the dependability coefficient.
Numerical example for the G study. Let’s use
a specific example to illustrate how to estimate
variance components for the one-facet crossed
design. Suppose that eight items were given
to 10 individuals, and the data are displayed
in Table 1. Items are regarded as random
samples from the infinite universe of all items,
and persons are randomly sampled from the
population of individuals.
Analysis of variance (ANOVA) can be used
to estimate the variance components. Several
statistical packages are available for conducting
G and/or D studies based on the ANOVA
approach: (a) GENOVA and urGENOVA
(along with mGENOVA), authored by Bren-
nan as a family of programs; (b) the Variance
Component in SPSS; and (c) Proc Varcomp
in SAS. The GENOVA suite of computer pro-
grams can run G and D studies for a variety of
designs when facets have different properties
and missing values may be present. In contrast,

Table 1 Item scores on eight items of 10 examinees (modified from Brennan, 2001, p. 28).

Item Person
Person 1 2 3 4 5 6 7 8 Mean

1 1 0 1 0 0 0 0 0 0.250
2 1 1 1 0 0 1 0 0 0.500
3 1 1 1 1 1 0 0 0 0.625
4 1 1 0 1 1 0 0 1 0.625
5 1 1 1 1 1 0 1 0 0.750
6 1 1 1 0 1 1 1 0 0.750
7 1 1 1 1 1 1 1 0 0.875
8 1 1 1 1 0 1 1 1 0.875
9 1 1 1 1 1 1 1 1 1.000
10 1 1 1 1 1 1 1 1 1.000
Item mean 1 0.9 0.9 0.7 0.7 0.6 0.6 0.4 0.725

Table 2 ANOVA estimates of variance components for the one-facet crossed design data in Table 1.
Source of Sum of
variation squares
Person (p) 3.950
Items (i) 2.750
Interaction with residual (pi,e) 9.250
the SPSS or SAS programs can only produce
results for G studies; users then have to use
the variance components estimated by the two
programs to calculate parameters for D studies.
Two-way ANOVA, with score as the
dependent variable, and person and item
as independent variables, is conducted to
calculate mean squares (MSs) for persons,
items, and residual. Meanwhile, the MSs can be
expressed as the expected mean square (EMS),
weighted sums of variance components, as
follows (for derivations of EMS equations, see
Kirk, 1982):
EMSp = σ2pi,e + ni σ2p
EMSi = σ2pi,e + np σ2i
EMSpi,e = σ2pi,e
These equations can be used to estimate each
variance component, after the EMS is replaced
with corresponding observed MSs and the
exact value of σ2 is replaced with estimated
σ2 , ̂
σ2 . The equations can be solved in order to
estimate each variance component as follows:
̂
σ2pi,e = MSpi,e
̂
σ2p = (MSp − ̂
σ2pi,e )∕ni = (MSp − MSpi,e )∕ni
̂
σ2i = (MSi − ̂
σ2pi,e )∕np = (MSi − MSpi,e )∕np
Table 2 presents the ANOVA result and
the variance components estimated by using
the formulas in this subsection. Based on the
estimated variance components, ρ2 for rela-
tive decision is calculated as σ2p ∕(σ2p + σ2pi,e )=
0.0365/(0.0365 + 0.1470) = 0.199. Φ for abso-
lute decision is calculated as σ2p /(σ2p +σ2i +σ2pi,e )
= 0.0365 / (0.0365 + 0.0246 + 0.1470) = 0.175.
GENERALIZABILITY THEORY 1329
df Mean Variance component Estimated variance
squares calculation component
9 0.439 (MSp − MSpi, e )/ni 0.0365
7 0.393 (MSi − MSpi, e )/np 0.0246
63 0.147 MSpi, e 0.1470
D studies. It is important to notice that the
variance components estimated in Table 2
reflect the variance of a one-item test; there-
fore, the generalizability coefficients calculated
are associated with scores from the one-item
test. It is not surprising that the coefficients
are rather low. If multiple items (n′i ) are
given to the examinees, the magnitude of
variance components and coefficients will
be improved. D studies can be conducted
to estimate how the magnitude of variance
components and the corresponding general-
izability coefficients vary by the number of
items.
From the central limit theorem in statis-
tics, it is well known that the variance
of mean scores is the variance of indi-
vidual scores divided by the sample size
of scores from which the mean is calcu-
lated. Therefore, the variance component
associated with n′i items, when the score
interpretation is based on the mean of
scores from n′i items, can be estimated as
follows:
σ2i′ = σ2i ∕n′i
σ2pi,e′ = σ2pi,e ∕n′i
In the example discussed earlier, if 10 items
rather than one item are used, both σ2i and
σ2pi,e will become one tenth of the original
ones; that is, σ2i′ = 0.00246 and σ2pi,e′ = 0.0147.
Correspondingly,
ρ2 = σ2p ∕(σ2p + σ2pi,e ∕n′i )
= 0.0365∕(0.0365 + 0.147∕10)
= 0.713
1330 GENERALIZABILITY THEORY
Φ = σ2p ∕(σ2p + σ2i ∕n′i + σ2pi,e ∕n′i )
= 0.0365∕(0.0365 + 0.0246∕10
+ 0.147∕10) = 0.680
Compared with the coefficients associated
with one item, the error variances are reduced
and the coefficients are higher. This is anal-
ogous to the Spearman–Brown formula in
classical theory. That is, the decision maker
can reduce error variance by increasing the
number of items and then using the aver-
age (or sum) over the items as the person’s
score. Similarly, by setting up the desirable
generalizability coefficient or dependability
coefficient, the decision maker can estimate
and determine the value of n′i (the needed
length of the test) to maintain the techni-
cal requirements of the measurement. For
example, if a teacher wants ρ2 to reach 0.80
in the example discussed, she can solve the
following equation to estimate the number
of items required: ρ2 = σ2p ∕(σ2p + σ2pi,e ) =
0.0365∕(0.0365 + 0.147∕n′i ) = 0.80. Therefore,
n′i = 16.1, so at least 17 items are needed for ρ2
to be 0.80.
One-Facet Nested Design
In a one-facet nested design, each person
is given a different set of random items.
For instance, in the example of item as
the nested facet with 10 students, the eight
items given to each person are all dif-
ferent. That is, items 1 to 8 are given to
person 1, items 9 to 16 are given to per-
son 2, and so on until items 73 to 80 are
given to person 10. The design is denoted as
i : p.
Table 3 ANOVA estimates of variance for data in Table 1 (assuming that the data are from a one-facet
nested design instead of a one-facet crossed design).
Source of Sum of df Mean
variation squares squares
Person (p) 3.950 9 0.439
i:p, e 12.000 70 0.171
Statistical model. In the nested design,
because each person is given a different set or
group of items, effects attributable solely to
items are indistinguishable from interaction
and other random effects. Suppose that the data
in Table 1 are collected from a nested design;
the column means in Table 3 then do not exist,
because persons’ scores in each column are on
different items, and it is meaningless to take an
average for each item across persons. There-
fore, the item effect does not exist in the linear
model of nested design. When the one-facet
design is nested, an observed score for one
person on one item (Xpi ) can be decomposed
as follows:
Xpi = μ [grand mean]
+ (μp − μ) [person effect]
+ (Xpi − μp ) [nested effect confounded
with residual] (3)
Correspondingly, the variance of the collec-
tion of observed scores, Xpi , for overall persons
and items in the universe is expressed as the
sum of the two variance components:
σ2X = σ2p + σ2i∶p,e (4)
pi
The ANOVA procedure can then be used to
calculate the MS and then estimate the variance
components.
Numerical example. To use space efficiently,
this section continues to use the data in Table 1,
but now imagine that different persons take dif-
ferent samples of items. That is, let us assume
that Items 1 to 8 are different sets for different
persons. Table 3 presents the ANOVA results
and the estimated variance components.
Variance component Estimated variance
calculation component
(MSp − MSresidual )/ni 0.0335
MSresidual 0.1710

Once variance components are estimated, ρ2
and Φ coefficients can be calculated. Unlike
a one-facet crossed design, errors for relative
decision and absolute decision are the same
because error variance associated with item
main effect cannot be distinguished from
other sources of errors. Accordingly, ρ2 = Φ =
0.0335 / (0.0335 + 0.1710) = 0.164.
D studies. Similar to D studies for a one-facet
crossed design, when the number of items
increases, the error variance decreases and
the coefficients of ρ2 and Φ increase. D
studies allow decision makers to estimate
how the magnitude of coefficients changes
with the number of items and determine the
number of items required to reach certain
coefficients.
Because the variance of the person effect (σ2p )
is the variance of universe score that will not
change, the D study only needs to calculate the
variance component for the nested effect con-
founded with the residual when using n′i items.
The formula is σ2i∶p,e′ = σ2i∶p,e ∕n′i .
Suppose that 10 items are given to examinees.
Coefficients for relative decision and absolute
decision can be calculated as ρ2 = Φ = 0.0335
/ (0.0335 + 0.1710 / 10) = 0.662. Similar to the
solution in one-facet crossed design, to reach a
coefficient of 0.80, the following equation needs
to be solved as ρ2 = Φ = 0.0335 ∕ (0.0335 +
0.1710∕n′i ) = 0.80. n′i is found to be 20.4; that
is, at least 21 items are needed to reach coeffi-
cients of 0.80.
Multifacet Designs
This section uses designs with two facets as
multifacet examples to describe the crossed
and nested models used in G and D studies.
Because the difference between crossed and
nested designs and the statistical models have
been described already in a previous section
of this entry, this section focuses on the inter-
pretation of variance components involving
multiple facets without getting into detail about
the technical aspects and describing extensively
each of the two different designs due to space
constraints. Instead, a G study example of a
crossed design will be introduced, and then
GENERALIZABILITY THEORY 1331
both crossed design and nested design used in
D studies will be described. In the descriptions,
the basic concepts for G theory will be reviewed
to articulate and stress the essence of applying
G theory in approaching the measurement
issues.
A multifacet universe is defined by more
than one facet, which may be crossed or nested
with the object of measurement (person) and
other facet(s), and may be random or fixed. If
a psychologist plans to generalize from clients’
responses from a set of questionnaire items on
one occasion to a larger set of items on a much
larger set of occasions, item and occasion are
the two facets involved. The universe would
be defined by the infinite number of items
on all admissible occasions. If a researcher
wants to generalize from toddlers’ interaction
with their peers rated by two raters on eight
indicators to a much larger set of raters on a
much larger set of indicators, then both rater
and indicator are facets that should be con-
sidered. The universe would be defined by all
admissible raters (e.g., all the raters who have
the expertise to rate the children) on all admis-
sible indicators of the construct of interest.
As described in these examples, studies that
involve two or more facets are called multifacet
designs.
Using the item and rater facets as a con-
crete example, there may potentially be a
variety of designs that the psychologist can
consider for her study. Table 4 lists some
examples. Depending on the structure of facets
(e.g., how facets are crossed or nested), the
variability of observed scores is partitioned
into different main effects and interaction
effects, except the person effect as the object
of measurement (for most cases). The math-
ematical calculations for these effects differ
as well, but this is not the focus of this entry
(for additional explanation, see Brennan,
2001).
Among all the possible designs that a
researcher can select, the fully crossed design
(p × i × r) is most advantageous for the G study,
because other nested designs, or a crossed
design with different sampling of items and
1332 GENERALIZABILITY THEORY
Table 4 Examples of two-facet designs.
Variability due to object
of measurement
Design (Person effect)
p×i×r p i, r
p × (i : r) p r, i:r
(i : p) × r p r, i:p
i : (p × r) p r
(i × r) : p p i:p, r:p
raters, can be examined in the D studies based
on the estimated variance components in the
G study. However, the utility of the nested
designs in the D studies can be rather limited.
For example, the design of either i : (p × r) or
(i × r) : p can be considered in the D stud-
ies when a G study employs the design of
p × i × r, but not vice versa, because an effect
confounded with other effects in the G study
cannot be decomposed subsequently in D stud-
ies. Therefore, if resources allow, researchers
should consider using the fully crossed design
in the G studies as this design offers more flex-
ibility for the designs in D studies. Given the
constraints imposed by resources, researchers
then need to specify the needed resources
in terms of items constructed, raters to be
trained, and other logistics (e.g., recruiting
raters or item developers). Assume that the
measurement procedure to be studied implies
10 items and two raters. For example, the
design of p × (i : r) indicates that five items will
be scored by Rater 1 and another five items
by Rater 2, whereas the design of (i × p) : r
indicates that each person will take a mea-
surement with a different set of 10 items,
but all the responses will be scored by two
raters. Therefore, the selection of the study
design is largely determined by the intended
generalizations in the D studies, as well as by
the resources available to conduct the data
collection.
Similar to the one-facet design, the gen-
eralizability and dependability coefficients
Variability due to measurement error
Main effects Interaction effects
pi, pr, ir, pir, e
pr, pi:r, e
pr, ir:p, e
pr, i:pr, e
ir:p, e
can be calculated using the relative mea-
surement error and absolute measurement
error, respectively. The D studies involve a
decision-making process that considers the
pattern of variance components associated
with multiple facets, evaluates the options
available by manipulating the properties of
facets, and decides the optimum measurement
procedure that elegantly balances the technical
requirements and the logistic cost. In what
follows, two designs, p × i × r for the multifacet
crossed design and p × (i : r) as an example of a
multifacet nested design, are described to illus-
trate the reasoning for partitioning the score
variability, estimating the generalizability and
dependability coefficients, and conducting D
studies.
A G Study Example Using the Multifacet
Crossed Design
In the multifacet crossed design, each person is
given the same random sample of items scored
by the same sample of raters. The design is
denoted as p × i × r, where p refers to person
(or examinee) as object of measurement and i
and r refer to item and rater facets respectively.
The statistical model used to estimate variance
components in this crossed design is described
below.
Statistical model The observed score for
any person on any item given by any rater is
denoted as Xpir . This observed score in the
two-facet crossed design can be decomposed

as follows:
Xpi = μ [grand mean]
+ (μp − μ) [person effect]
+ (μi − μ) [item effect]
+ (μr − μ) [rater effect]
+ (μpi − μp − μi + μ)
[person × item effect]
+ (μpr − μp − μr + μ)
[person × rater effect]
+ (μir − μi − μr + μ) [item × rater effect]
+ (Xpir − μpi − μpr − μir
+ μp + μi + μr − μ)
[person × item × rater effect confounded
person × item × residual] (5)
Given the main effects and interaction effects
in Equation 5, the variance of observed scores
can be decomposed as the sum of seven vari-
ance components, including person variance as
the object of measurement and measurement
errors accounted by different sources:
σ2X = σ2p + σ2i + σ2r + σ2pi + σ2pr + σ2pir,e . (6)
pir
That is, the variance of item scores can be
partitioned into independent sources of varia-
tion due to differences between persons as the
object of measurement, σ2p , and six sources of
measurement error: variation due to two main
effects, σ2i and σ2r ; variation due to two-way
interactions, σ2pi , σ2pr , and σ2ir ; and σ2pir,e , the
p × i × r three-way interaction confounded
with the unsystematic or unspecified residual.
For relative decisions, all variance compo-
nents that influence the relative standing of
individuals contribute to relative measurement
error; these include σ2pi , σ2pr , and σ2pir,e . For
absolute decisions, all variance components
except the object of measurement contribute
to the absolute measurement error; that is, all
the variance components except the person
GENERALIZABILITY THEORY 1333
variance in this p × i × r design. A generaliz-
ability coefficient and dependability coefficient
are then calculated based on these two types of
measurement error.
Numerical example for the G study. Let’s use
a specific example to illustrate how to esti-
mate variance components for the multifacet
crossed design. Suppose that 10 open-ended
items to measure interpersonal skills were
given to 50 individuals and responses were
scored by three raters using a 1–4 scale. Due
to space limitations, the full data set is not
included. Table 5 displays the data layout.
Because all the individuals are measured by all
the levels of facets (i.e., 10 items and three
raters), item and rater are crossed facets.
The observations are regarded as randomly
sampled from the universe of generalization
as scores on all the admissible items rated
by admissible raters.
Based on the MS results of ANOVA, each
variance component can be estimated (Table 6;
estimating variance components based on the
ANOVA method is described in the previous
section, and for additional explanations, see
Kirk, 1982). The relative magnitudes of the
estimated variance components, except for
person variance, provide information about
potential sources of measurement error that
influence this psychological measure of inter-
personal skills. The person variance due to
differences in universe scores (σ2p = 0.1884)
is relatively large compared to other compo-
nents, accounting for 28% of total variation
even when only one item and one rater are
sampled from the universe. The other larger
components are two interaction effects, σ2pir,e
and σ2pi , accounting for 38% and 32% of score
variation, respectively. Specifically, σ2pir,e as
the largest variance component reflects that
some individuals’ scores can be relatively low
or high depending on certain items graded by
certain raters; in other words, persons’ scores
of interpersonal skills vary across items and
raters. The large σ2pi suggests that the relative
standings of individuals’ performance vary
across items. The σ2i only accounts for 2% of
Table 5 Item scores for crossed Person × Item × Rater G study.

Item
1 2 3 … 10
Person Rater Rater Rater Rater Rater Rater Rater Rater Rater Rater Rater Rater Rater Rater Rater
1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

1 4 4 4 1 1 2 2 2 3 3 4 4
2 2 3 3 2 2 2 1 2 1 2 2 2
3 4 4 4 1 1 1 1 2 2 3 3 4
…
N = 50 3 4 3 1 2 1 2 2 1 4 4 4

Table 6 Estimated variance components in the p × i × r example.
Source of variability Symbol of variance
component
Person (p) σ2p
Item (i) σ2i
Rater (r) σ2r
p× i σ2pi
p× r σ2pr
i× r σ2ir
p × i × r, e σ2pir,e
the score variance, which is a rather negligible
source of measurement error. This shows that,
despite the fact that some items more likely
lead to slightly higher scores for all individuals
than other items, this unevenness in item
difficulty contributes little to the overall score
variation when other variations are taken into
consideration. Finally, other components (i.e.,
σ2r , σ2pr , and σ2ir ), all of which are associated
with the rater facet, appear not to contribute
much to the score variability. This provides
evidence that the training of raters and the
use of scoring systems minimize the potential
measurement error from the rater main effect
and other related interaction effects.
After pinpointing relative and absolute mea-
surement errors, the generalizability coefficient
(ρ2 ) and dependability coefficient (Φ) are to be
calculated when only one item and one rater are
sampled from the universe of admissible obser-
vations as follows:
ρ2 = σ2p ∕(σ2p + σ2Rel. )
= σ2p ∕(σ2p + σ2pi + σ2pr + σ2pir,e )
= 0.1884∕(0.1884 + 0.2132 + 0.0028
+ 0.2589) = 0.284
Φ = σ2p ∕(σ2p + σ2Abs. ) = σ2p ∕(σ2p + σ2i + σ2r
+ σ2pi + σ2pr + σ2ir + σ2pir,e )
= 0.1884∕(0.1884 + 0.0111 + 0 + 0.2132
+ 0.0028 + 0.0014 + 0.2589) = 0.279
GENERALIZABILITY THEORY 1335
Estimated variance Total variability (%)
component
0.1884 28
0.0111 02
0.0000 00
0.2132 32
0.0028 00
0.0014 00
0.2589 38
Examples of D Studies Using the Crossed
and Nested Designs
Based on the estimated variance components in
Table 6, more items and raters are apparently
needed in order to increase the generalizabil-
ity or dependability of score inferences. Assume
that the D studies still use the crossed design
with both facets as random. Table 7 reports the
estimated variance components, measurement
errors, and generalizability and dependability
coefficients when the number of items is 5, 10,
or 15 and the number of raters is 2, 3, or 4. As
more items and more raters are included in a
measurement procedure, smaller measurement
error and higher coefficients result.
Alternatively, the results from D studies can
be presented in figures. For instance, Figure 1
shows the dependability coefficients for given
numbers of items (n′i ranging from 2 to 14)
and raters (n′r ranging from 2 to 4). Table 7 and
Figure 1 suggest that the benefit of increasing
items in order to reduce the amount of error
variance starts to diminish at approximately
nine items, and the benefit of having more
raters reaches a plateau at three raters.
The results of D studies also make it evi-
dent that the development, specification, and
evaluation of new measurement procedures
can be handled systematically by the G theory
approach. For example, in order to maintain
a dependability coefficient at 0.80, a measure-
ment procedure of using eight items and two
raters or seven items with three raters can
be considered. Depending on the financial
and human resources, a decision maker can
Table 7 Estimated variance components (EVCs) for p × i × r random effects design based on results of the G study in Table 6.

Source of EVC EVC in D studies

variability
in G n′i = 5 10 15 5 10 15 5 10 15
study n′r = 2 2 2 3 3 3 4 4 4

Person (p) 0.1884 σ2p 0.188 0.188 0.188 0.188 0.188 0.188 0.188 0.188 0.188
Item (i) 0.0111 σ2i /n′i 0.002 0.001 0.001 0.002 0.001 0.001 0.002 0.001 0.001
Rater (r) 0.0000 σ2r /n′r 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
p×i 0.2132 σ2pi /n′i 0.043 0.021 0.014 0.043 0.021 0.014 0.043 0.021 0.014
p×r 0.0028 σ2pr /n′r 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001
2
i×r 0.0014 σir /(n′i n′r ) 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
p × i × r, e 0.2589 σ2pir,e /(n′i n′r ) 0.026 0.013 0.009 0.017 0.009 0.006 0.013 0.006 0.004
Relative measurement error (σ2Rel. ) 0.070 0.035 0.024 0.061 0.031 0.021 0.057 0.028 0.019
Absolute measurement error (σ2Abs. ) 0.072 0.036 0.025 0.063 0.032 0.022 0.059 0.030 0.020
Generalizability coefficient (ρ2 ) 0.729 0.843 0.887 0.755 0.858 0.900 0.767 0.870 0.908
Dependability coefficient (Φ) 0.723 0.839 0.883 0.749 0.855 0.895 0.761 0.866 0.904

0.90
0.85
Dependability coefficient (φ)
0.80
0.75
0.70
0.65
0.60
0.55
2 4
Figure 1 Dependability coefficients for given number of items and raters.
compare these measurement procedures and
select the more efficient one with adequate
psychometric requirements.
As explained earlier, a decision maker may
choose the nested design to plan the possible
measurement procedures in the D studies
given the logistic considerations. In this case,
each person does not have scores from all the
levels of facets. There are many variations of
nested designs, depending on the relationship
between the object of measurement and the
facets, for example p × (i : r) or i : r : p. Each
requires a different procedure to compile
test items and arrange the scoring. In what
follows, these two nested designs are used
to illustrate the conceptual understanding for
conducting D studies and interpreting variance
components.
Again, 10 items and three raters are the
measurement conditions sampled from the
universe. In the p × (i : r) design, each indi-
vidual would be administered the same set of
30 items, among which 10 items are scored by
Rater 1, another 10 items are scored by Rater 2,
GENERALIZABILITY THEORY 1337
n=2
n=3
n=4
6 8 10 12 14
Number of items
and the last set of 10 items by Rater 3. For the
two random facets, item is a nested facet and
rater is a crossed facet. In the i : r : p design, both
facets are nested facets. This design requires
each individual to be tested by a different set
of 30 items and scored by a different group of
three raters while each rater only graded 10
items. Based on results of the crossed G study
from Table 6, Table 8 presents the estimated
variance components of a series of D studies
using the two nested designs. Similar to the D
studies for one-facet designs, the error variance
can be reduced and the coefficients of ρ2 and
Φ are increased when more levels of item and
rater facets are sampled from the universe.
Depending on the magnitudes of variance
components reported in the G studies, the
patterns of estimated variance components for
different designs can differ dramatically or can
be similar, as shown in the two examples in
Table 8. Considering that facets can be fixed
or random in D studies, decision makers can
propose even more variations of complicated
Table 8 Estimated variance components (EVCs) for p × (i : r) and i : r : p random effects designs based on results of the p × i × rG study in Table 6.

Source of EVC EVC in D studies

variability using 1
rater and n′i = 5 10 15 5 10 15 5 10 15
1 item n′r = 2 2 2 3 3 3 4 4 4

p × ( i : r ) Design
Person (p) 0.188 σ2p 0.188 0.188 0.188 0.188 0.188 0.188 0.188 0.188 0.188
Rater (r) 0.000 σ2r ∕n′r 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
Item (i:r) 0.012 σ2i ∶ r ∕(n′i × n′r ) 0.001 0.001 0.000 0.001 0.000 0.000 0.001 0.000 0.000
p×r 0.003 σ2pr ∕n′r 0.002 0.002 0.002 0.001 0.001 0.001 0.001 0.001 0.001
pi : r, e 0.472 σ2pi ∶ r,e ∕(n′i × n′r ) 0.047 0.024 0.016 0.031 0.016 0.010 0.024 0.012 0.008
Generalizability coefficient (ρ2 ) 0.793 0.879 0.913 0.855 0.917 0.945 0.883 0.935 0.954
Dependability coefficient (Φ) 0.790 0.874 0.913 0.851 0.917 0.945 0.879 0.935 0.954
i : r : p Design
Person (p) 0.188 σ2p 0.188 0.188 0.188 0.188 0.188 0.188 0.188 0.188 0.188
r:p 0.003 σ2r ∶ p ∕n′r 0.002 0.002 0.002 0.001 0.001 0.001 0.001 0.001 0.001
i : r : p, e 0.484 σ2i ∶ r ∶ p,e ∕(n′i × n′r ) 0.048 0.024 0.016 0.032 0.016 0.011 0.024 0.012 0.008
Generalizability coefficient (ρ2 ) 0.790 0.879 0.913 0.851 0.917 0.940 0.883 0.935 0.954
Dependability coefficient (Φ) 0.790 0.879 0.913 0.851 0.917 0.940 0.883 0.935 0.954

designs to evaluate and make decisions about
future measurement procedures (for details on
different designs and estimation of the variance
components, see Brennan, 2001).
Additional Concepts and Issues
The previous three sections are intended to
provide a brief review on the basic concepts
of G theory and its application in tackling
measurement issues. This section focuses
on three topics that are relevant to the use
of G theory in investigating psychological
measurements. Discussion of more advanced
topics and concepts, such as multivariate
G theory and estimation of variance com-
ponents for fixed facets, can be found in
Brennan (2001) and Shavelson and Webb
(1991).
Use of G Theory in Validity Studies
Although G theory was originally introduced
by Cronbach and his colleagues (Cronbach
et al., 1972) as an extension of reliability the-
ory and has been frequently implemented to
address reliability issues by researchers in the
field of psychological and educational mea-
surement, it can also be applied to validity
studies (Brennan, 2001; Kane, 1982). G the-
ory conceptualizes the variance of observed
scores as a collection of variability due to
multiple effects, including the main effect due
to the object of measurement as it reflects
the systematic difference among individu-
als in the construct of interest. The main
effects of facets are specified, as well as their
interaction effects. Facets studied in a G or
D study can be more pertinent to drawing
appropriate inferences or interpretations
on scores than to the precision (reliability
and consistency) of score interpretations.
As Kane (1982) points out, a construct can
be operationally defined as a dispositional
attribute in terms of universe scores and facets
as measurement conditions that should not
produce any construct-irrelevant variance.
He argues that this invariance law allows
the multifacet sampling approach to offer a
GENERALIZABILITY THEORY 1339
straightforward interpretation of construct
validity.
For example, with respect to measuring
learning disability, the facet of task domain
or measurement method (such as drawing
a design, conceptual grouping, or numerical
memory) can be of interest for researchers.
Then, a Person × (Item : Domain) nested
design can be used to examine whether
the test scores from the sampled domains
measure individuals consistently or not.
Analysis showing that variance components
related to the domain facet, both main effect
and interaction effects, do not contribute
to the overall score variation thus can be
considered evidence for construct validity,
analogous to convergent validity. To con-
sider another example, researchers who
are interested in evaluating the accuracy
of self-report scores can conceptualize the
self-reporting method as one level in compari-
son to another level, observation by therapists,
a method that has been empirically vali-
dated. Then, the variance components related
to the method facet can verify whether the
self-report scores from patients provide accu-
rate interpretations of the intended construct
or not.
In summary, when a decision maker defines
the construct to be measured and specifies the
facets and the universe, to some degree she
also defines which facets are relevant to the
validity claims of score interpretations, which
are primarily related to reliability claims,
and which may be pertinent to both. This
knowledge should then guide the design, anal-
ysis, and interpretation of G studies and D
studies.
Estimation of Variance Components
The isolation and estimation of variance
components in G studies can be carried out
using various methods. This entry describes
a commonly used one, the ANOVA method.
Other methods include likelihood-based
methods such as maximum likelihood
and restricted maximum likelihood (e.g.,
Searle, 1987; Searle, Casella, & McCulloch,
1340 GENERALIZABILITY THEORY
1992), resampling methods such as boot-
strap and jackknife methods (e.g., Brennan,
Harris, & Hanson, 1987; Wiley, 2000), and
Bayesian analysis (e.g., Brennan, 2001).
Generally, the latter two groups of meth-
ods are less sensitive to the assumptions of
score distribution, especially for dichotomous
scores.
When conducting G studies with the
ANOVA method, researchers may encounter
negative values for the estimated variance
components, mostly in multifacet designs.
Conceptually, variance should not be negative;
therefore, the negative values often signal
problems due to sample error or model mis-
specification (Shavelson & Webb, 1981); in
other words, the levels for certain facet(s) are
not comparable. Two strategies are often used
to address negative estimates if the values
are small: (a) Cronbach’s algorithm, which
substitutes zero for the negative estimate,
and uses zero wherever needed to calculate
other variance components (Cronbach et al.,
1972); and (b) Brennan’s algorithm, which
uses negative values in the calculation of all
the variance components, and then sets all the
negative values to zero for the estimated vari-
ance components at the end (Brennan, 2001).
Brennan’s is the preferred approach. That said,
statistical methods other than ANOVA, such
as Bayesian analysis and maximum likelihood,
can be used to avoid the problem of negative
estimates.
Standard Error Estimates
One critical concept in interpreting the results
from G and D studies is the estimated standard
error. Like the standard error of measurement
in classical reliability theory, the standard
error estimates based on relative or absolute
measurement errors can inform researchers
about the confidence interval for individuals’
universe scores. Results of G and D studies
can be interpreted meaningfully through the
comparison between the interval range and
the score scale. Furthermore, the standard
error of the variance components should be
taken into account when the variance com-
ponents are examined. Large standard errors
indicate the instability of the estimates of
variance components; that is, the variance
components are not robust due to the small
sample size of subjects. In particular, when
the absolute value of a negative estimate is
relatively larger than its standard error, that
variance component then should not be treated
as zero. Finally, G theory can be applied to esti-
mate the individual-level standard error by
acknowledging, as Brennan and his colleagues
(Brennan & Lee, 1999; Lee, Brennan, & Kolen,
2000) point out, that the standard error of
measurement of raw scores or scale scores
is conditional to the level of the measured
construct; for example, some individuals are
more inconsistent in responding to items
than others when they frequently guess the
responses. This adds complexity in estimat-
ing and interpreting the standard error of
measurement.
SEE ALSO: Construct Validity; Reliability
References
Brennan, R. L. (2000). (Mis)conceptions about
generalizability theory. Educational
Measurement: Issues and Practice, 19(1), 5–10.
Brennan, R. L. (2001). Generalizability theory. New
York: Springer Verlag.
Brennan, R. L., Harris, D. J., & Hanson, B. A.
(1987). The bootstrap and other procedures for
examining the variability of estimated variance
components in testing contexts (ACT Research
Report Series 87–7). Iowa City, IA: American
College Testing Program.
Brennan, R. L., & Lee, W. -C. (1999). Conditional
scale-score standard errors of measurement
under binomial and compound binomial
assumptions. Educational and Psychological
Measurement, 59, 5–24.
Cronbach, L. J. (1951). Coefficient alpha and the
internal structure of tests. Psychometrika, 16(3),
297–334.
Cronbach, L. J., Gleser, G. C., Nanda, H., &
Rajaratnam, N. (1972). The dependability of
behavioral measurements. New York: Wiley.
Cronbach, L. J., Linn, R. L., Brennan, R. L., &
Haertel, E. H. (1997). Generalizability analysis

for performance assessments of student
achievement or school effectiveness. Educational
and Psychological Measurement, 57, 373–399.
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C.
(1963). Theory of generalizability: A
liberalization of reliability theory. British Journal
of Statistical Psychology, 16, 137–163.
Gleser, G. C., Green, B. L., & Winget, C. N. (1978).
Quantifying interview data on psychic
impairment of disaster survivors. Journal of
Nervous and Mental Disease, 166(3), 209–216.
Kane, M. T. (1982). A sampling model of validity.
Applied Psychological Measurement, 6, 125–160.
Kirk, R. E. (1982). Experimental design: Procedures
for the behavioral sciences (2nd ed. ). Pacific
Grove, CA: Brooks/Cole.
Lee, W. -C., Brennan, R. L., & Kolen, M. J. (2000).
Estimators of conditional scale-score standard
errors of measurement: A simulation study.
Journal of Educational Measurement, 37, 1–28.
Searle, S. R. (1987). Linear models for unbalanced
data. New York: Wiley.
Searle, S. R., Casella G., & McCulloch, C. M. (1992).
Variance components. New York: Wiley.
Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W.
(1999). Note on sources of sampling variability in
science performance assessments. Journal of
Educational Measurement, 36, 61–71.
Shavelson, R. J., & Webb, N. M. (1981).
Generalizability theory: 1973–1980. British
Journal of Mathematical and Statistical
Psychology, 34, 133–166.
Shavelson, R. J., & Webb, N. M. (1991).
Generalizability theory: A primer. Newbury Park,
CA: Sage.
Webb, N. M., Shavelson, R. J., & Haertel, E. H.
(2006). Reliability coefficients and
generalizability theory. In C. R. Rao & S.
Sinharay (Eds. ), Handbook of statistics (Vol. 2,
pp. 81–124). Amsterdam: Elsevier.
Wiley, E. W. (2000). Bootstrap strategies for variance
component estimation: Theoretical and empirical
results. Unpublished doctoral dissertation,
Stanford University, CA.
Further Reading
Hoyt, W. T., & Melby, J. N. (1999). Dependability of
measurement in counseling psychology: An
introduction to generalizability theory.
Counseling Psychologist, 27(3), 325–352.
View publication stats
GENERALIZED ANXIETY DISORDER 1341
Generalized Anxiety
Disorder
Sandra J. Llera1 and Michelle G.
Newman2
1 Towson University, U.S.A. and 2 Pennsylvania State
University, U.S.A.
Generalized anxiety disorder (GAD) is char-
acterized as a disorder of excessive worry and
anxiety that is not targeted toward any one
type of situation and is therefore generalized.
Given that the constructs of worry and anxiety
underlie many of the psychological disorders,
GAD has been considered the basic anxiety
disorder. In early nosological descriptions,
there was much overlap in symptomatology
between GAD and other disorders; in fact,
GAD has only recently been defined as an
independent disorder in its own right. Since
then, researchers have worked to further elu-
cidate the nature, etiology, and mechanisms
of GAD.
Definition
According to the current edition of the DSM
(DSM-5; American Psychiatric Association
2013), GAD is defined by its central crite-
rion of excessive, chronic, and uncontrollable
worry over a number of different topic areas.
For instance, people with GAD may worry
about their relationships with others, finances,
school or work performance, health, and even
daily chores. This pattern is distinct from the
worry that characterizes other disorders. Those
with panic disorder may worry about having
another panic attack, those with an eating
disorder may worry about their weight, and
individuals with a specific phobia (e.g., dogs
or heights) may worry about encountering
their feared object or situation. In contrast,
those with GAD worry about a wide variety of
topics, and the target of their worries may vary
throughout the day.
Worry has been defined as a sequence of
perseverative thoughts and images associ-
ated with negative affect and focused on