Measurement Validity: A Shared Standard for Qualitative and Quantitative Research

Author(s): Robert Adcock and David Collier

Source: The American Political Science Review, Vol. 95, No. 3 (Sep., 2001), pp. 529-546
Published by: American Political Science Association
Stable URL: http://www.jstor.org/stable/3118231
Accessed: 04/12/2010 18:02

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/action/showPublisher?publisherCode=apsa.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Political Science Association is collaborating with JSTOR to digitize, preserve and extend access to
The American Political Science Review.

http://www.jstor.org
 Aeica Political
American Science Review
PoitclSineRveVo.9,N.3Spmbr20 Vol. 95, No. 3 September 2001

Measurement AShared
Validity: Standard
forQualitative
andQuantitative
Research
ROBERT ADCOCK and DAVID COLLIER University of California,Berkeley
Scholarsroutinelymake claims thatpresuppose the validityof the observationsand measurementsthat
operationalize their concepts. Yet, despite recent advances in political science methods, surprisingly
little attention has been devoted to measurementvalidity.Weaddressthis gap by exploringfour themes.
First, we seek to establish a shared framework that allows quantitative and qualitative scholars to assess
more effectively,and communicate about, issues of valid measurement.Second, we underscorethe need to
draw a clear distinction between measurement issues and disputes about concepts. Third, we discuss the
contextual specificity of measurement claims, exploring a variety of measurement strategies that seek to
combine generalityand validityby devotinggreaterattention to context.Fourth, we address the proliferation
of termsfor alternativemeasurementvalidationprocedures and offeran account of the three main types of
validation most relevant to political scientists.

R esearchers routinely make complex choices
about linking concepts to observations, that is,
about connecting ideas with facts. These choices
raise the basic question of measurement validity: Do
the observations meaningfully capture the ideas con-
tained in the concepts? We will explore the meaning of
this question as well as procedures for answering it. In
the process we seek to formulate a methodological
standard that can be applied in both qualitative and
quantitative research.
Measurement validity is specifically concerned with
whether operationalization and the scoring of cases
adequately reflect the concept the researcher seeks to
measure. This is one aspect of the broader set of
analytic tasks that King, Keohane, and Verba (1994,
chap. 2) call "descriptive inference," which also encom-
passes, for example, inferences from samples to popu-
lations. Measurement validity is distinct from the va-
lidity of "causal inference" (chap. 3), which Cook and
Campbell (1979) further differentiate into internal and
external validity.1 Although measurement validity is
interconnected with causal inference, it stands as an
important methodological topic in its own right.
New attention to measurement validity is overdue in
political science. While there has been an ongoing
concern with applying various tools of measurement
validation (Berry et al. 1998; Bollen 1993; Elkins 2000;
Hill, Hanna, and Shafqat 1997; Schrodt and Gerner
Robert Adcock (adcockr@uclink4.berkeley.edu) is a Ph.D candi-
date, Department of Political Science, and David Collier
(dcollier@socrates.berkeley.edu) is Professorof Political Science,
Universityof California,Berkeley,CA 94720-1950.
Amongthe manycolleagueswho haveprovidedhelpfulcomments
on this article,we especiallythank ChristopherAchen, Kenneth
Bollen,HenryBrady,EdwardCarmines,RubetteCowan,PaulDosh,
ZacharyElkins,JohnGerring,KennethGreene,ErnstHaas,Edward
Haertel, Peter Houtzager,Diana Kapiszewski,Gary King, Marcus
Kurtz,James Mahoney,SebastianMazzuca,Doug McAdam,Ger-
ardo Munck, Charles Ragin, Sally Roever, Eric Schickler,Jason
Seawright,JeffSluyter,RichardSnyder,RuthStanley,LauraStoker,
and three anonymousreviewers.The usual caveats apply.Robert
Adcock'sworkon this projectwas supportedby a NationalScience
FoundationGraduateFellowship.
1 These involve,respectively,the validityof causalinferencesabout
the casesbeingstudied,and the generalizability of causalinferences
to a broaderset of cases (Cook and Campbell1979,50-9, 70-80).
1994), no major statement on this topic has appeared
since Zeller and Carmines (1980) and Bollen (1989).
Although King, Keohane, and Verba (1994, 25, 152-5)
cover many topics with remarkable thoroughness, they
devote only brief attention to measurement validity.
New thinking about measurement, such as the idea of
measurement as theory testing (Jacoby 1991, 1999), has
not been framed in terms of validity.
Four important problems in political science re-
search can be addressed through renewed attention to
measurement validity. The first is the challenge of
establishing shared standards for quantitative and qual-
itative scholars, a topic that has been widely discussed
(King, Keohane, and Verba 1994; see also Brady and
Collier 2001; George and Bennett n.d.). We believe the
skepticism with which qualitative and quantitative re-
searchers sometimes view each other's measurement
tools does not arise from irreconcilable methodological
differences. Indeed, substantial progress can be made
in formulating shared standards for assessing measure-
ment validity. The literature on this topic has focused
almost entirely on quantitative research, however,
rather than on integrating the two traditions. We
propose a framework that yields standards for mea-
surement validation and we illustrate how these apply
to both approaches. Many of our quantitative and
qualitative examples are drawn from recent compara-
tive work on democracy, a literature in which both
groups of researchers have addressed similar issues.
This literature provides an opportunity to identify
parallel concerns about validity as well as differences in
specific practices.
A second problem concerns the relation between
measurement validity and disputes about the meaning
of concepts. The clarification and refinement of con-
cepts is a fundamental task in political science, and
carefully developed concepts are, in turn, a major
prerequisite for meaningful discussions of measure-
ment validity. Yet, we argue that disputes about con-
cepts involve different issues from disputes about mea-
surement validity. Our framework seeks to make this
distinction clear, and we illustrate both types of dis-
putes.
A third problem concerns the contextual specificity

529
Measurement Validity: A Shared Standard for Qualitative and Quantitative Research
of measurement validity-an issue that arises when a
measure that is valid in one context is invalid in
another. We explore several responses to this problem
that seek a middle ground between a universalizing
tendency, which is inattentive to contextual differences,
and a particularizingapproach, which is skeptical about
the feasibility of constructing measures that transcend
specific contexts. The responses we explore seek to
incorporate sensitivity to context as a strategy for
establishing equivalence across diverse settings.
A fourth problem concerns the frequently confusing
language used to discuss alternative procedures for
measurement validation. These procedures have often
been framed in terms of different "types of validity,"
among which content, criterion, convergent, and con-
struct validity are the best known. Numerous other
labels for alternative types have also been coined, and
we have found 37 different adjectives that have been
attached to the noun "validity" by scholars wrestling
with issues of conceptualization and measurement.2
The situation sometimes becomes further confused,
given contrasting views on the interrelations among
different types of validation. For example, in recent
validation studies in political science, one valuable
analysis (Hill, Hanna, and Shafqat 1997) treats "con-
vergent" validation as providing evidence for "con-
struct" validation, whereas another (Berry et al. 1998)
treats these as distinct types. In the psychometrics
tradition (i.e., in the literature on psychological and
educational testing) such problems have spurred a
theoretically productive reconceptualization. This liter-
ature has emphasized that the various procedures for
assessing measurement validity must be seen, not as
establishing multiple independent types of validity, but
rather as providing different types of evidencefor valid-
ity. In light of this reconceptualization, we differentiate
between "validity"and "validation."We use validity to
refer only to the overall idea of measurement validity,
and we discuss alternative procedures for assessing
validity as different "types of validation." In the final
part of this article we offer an overview of three main
types of validation, seeking to emphasize how proce-
dures associated with each can be applied by both
quantitative and qualitative researchers.
In the first section of this article we introduce a
framework for discussing conceptualization, measure-
ment, and validity. We then situate questions of validity
in relation to broader concerns about the meaning of
concepts. Next, we address contextual specificity and
equivalence, followed by a review of the evolving
discussion of types of validation. Finally, we focus on
three specific types of validation that merit central
2 We have found the following adjectives attached to validity in
discussions of conceptualization and measurement: a priori, appar-
ent, assumption, common-sense, conceptual, concurrent, congruent,
consensual, consequential, construct, content, convergent, criterion-
related, curricular, definitional, differential, discriminant, empirical,
face, factorial, incremental, instrumental, intrinsic, linguistic, logical,
nomological, postdictive, practical, pragmatic, predictive, rational,
response, sampling, status, substantive, theoretical, and trait. A
parallel proliferation of adjectives, in relation to the concept of
democracy, is discussed in Collier and Levitsky 1997.
530
September 2001
attention in political science: content, convergent/dis-
criminant, and nomological/construct validation.
OVERVIEWOF MEASUREMENT VALIDITY
Measurement validity should be understood in relation
to issues that arise in moving between concepts and
observations.
Levels and Tasks
We depict the relationship between concepts and ob-
servations in terms of four levels, as shown in Figure 1.
At the broadest level is the background concept, which
encompasses the constellation of potentially diverse
meanings associated with a given concept. Next is the
systematized concept, the specific formulation of a
concept adopted by a particular researcher or group of
researchers. It is usually formulated in terms of an
explicit definition. At the third level are indicators,
which are also routinely called measures. This level
includes any systematic scoring procedure, ranging
from simple measures to complex aggregated indexes.
It encompasses not only quantitative indicators but
also the classification procedures employed in qualita-
tive research. At the fourth level are scores for cases,
which include both numerical scores and the results of
qualitative classification.
Downward and upward movement in Figure 1 can be
understood as a series of research tasks. On the
left-hand side, conceptualization is the movement from
the background concept to the systematized concept.
Operationalization moves from the systematized con-
cept to indicators, and the scoring of cases applies
indicators to produce scores. Moving up on the right-
hand side, indicators may be refined in light of scores,
and systematized concepts may be fine-tuned in light of
knowledge about scores and indicators. Insights de-
rived from these levels may lead to revisiting the
background concept, which may include assessing al-
ternative formulations of the theory in which a partic-
ular systematized concept is embedded. Finally, to
define a key overarching term, "measurement" involves
the interaction among levels 2 to 4.
Defining Measurement Validity
Valid measurement is achieved when scores (including
the results of qualitative classification) meaningfully
capture the ideas contained in the corresponding con-
cept. This definition parallels that of Bollen (1989,
184), who treats validity as "concerned with whether a
variable measures what it is supposed to measure."
King, Keohane, and Verba (1994, 25) give essentially
the same definition.
If the idea of measurement validity is to do serious
methodological work, however, its focus must be fur-
ther specified, as emphasized by Bollen (1989, 197).
Our specification involves both ends of the connection
between concepts and scores shown in Figure 1. At the
concept end, our basic point (explored in detail below)
is that measurement validation should focus on the
 Ameica
AmericanPoiia Science
Political cec Review
eiwVl 5 No.
Vol. 95, o 3

FIGURE1. Conceptualizationand Measurement:Levels and Tasks

I

Level 1. Backgl round Concept

The broadconstellatiion of meanings and
understandingsassociated witha given concept.

Task: ConceF itualization Task: Revisiting Background

Formulatinga systelmatizedconcept through Concept. Exploringbroaderissues concerning
reasoning about the backgroundconcept, in the backgroundconcept in lightof insightsabout
lightof the goals of Iresearch. scores, indicators,and the systematized concept.

Level 2. Systematized Concept

A specific formulationof a concept used by a
given scholar or groupof scholars;

J
Task: Operationalization
commonlyinvolves an explicitdefinition.
2.
Task: Modifying Systematized
Developing,on the basis of a systema- Concept. Fine-tuningthe systematized
tized concept, one or more indicators or
concept, possibly extensively revisingit, in
for scoring/classifyingcases. lightof insightsabout scores and indicators.

C
*0 Level 3. Indiicators
E
//
I Also referredto as "measures"and "opera-
tinnalinztinns " In nlalitativ ,e research,these
II,I %,.11..1i,.A,?I

are the operaltionaldefinittions employed in

y

U) /I - cllassifyingcaases.

aa Task: Scoring Cases

2
0 Task: Refining Indicators
E Applyingthese indicatorsto produce Modifyingindicators,or potentiallycreating
scores for the cases being analyzed. new indicators,in lightof observed scores.

Level 4. Scores for Cases

The scores for cases generated by a particular
indicator.These includebot:hnumericalscores
and the resultsof qualital
tiveclassification
L I

relation between observationsand the systematized
concept;any potential disputesabout the background
concept should be set aside as an important but
separate issue. With regardto scores, an obvious but
crucialpoint must be stressed:Scores are never exam-
ined in isolation;rather,they are interpretedand given
meaningin relationto the systematizedconcept.
In sum, measurementis validwhen the scores (level
4 in Figure 1), derivedfrom a given indicator(level 3),
can meaningfullybe interpretedin termsof the system-
atized concept (level 2) that the indicator seeks to
operationalize.It would be cumbersometo refer re-
peatedly to all these elements, but the appropriate
focus of measurementvalidationis on the conjunction
of these components.
Measurement Error, Reliability, and Validity
Validityis often discussedin connectionwith measure-
ment errorand reliability.Measurementerrormay be
systematic-in whichcase it is calledbias-or random.
Random error,which occurs when repeated applica-
tions of a given measurementprocedureyield incon-
sistent results, is conventionallylabeled a problemof
reliability.Methodologistsoffer two accounts of the
relationbetween reliabilityand validity.(1) Validityis
sometimes understood as exclusivelyinvolving bias,
that is errorthat takes a consistentdirectionor form.
From this perspective,validityinvolvessystematicer-
ror, whereas reliabilityinvolves random error (Car-
mines and Zeller 1979, 14-5; see also Babbie 2001,

531
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch
II
144-5). Therefore,unreliablescores may still be cor-
rect "on average"and in this sense valid. (2) Alterna-
tively,some scholarshesitate to view scores as valid if
they contain large amounts of random error. They
believe validityrequiresthe absence of both types of
error.Therefore,theyviewreliabilityas a necessarybut
not sufficientconditionof measurementvalidity(Kirk
and Miller 1986, 20; Shively1998, 45).
Ourgoal is not to adjudicatebetweenthese accounts
but to state them clearlyand to specifyour own focus,
namely,the systematicerrorthat ariseswhen the links
among systematizedconcepts, indicators,and scores
are poorlydeveloped.This involvesvalidityin the first
sense stated above. Of course, the randomerror that
routinelyarisesin scoringcases is also important,but it
is not our primaryconcern.
A finalpoint shouldbe emphasized.Becauseerroris
a pervasivethreat to measurement,it is essential to
view the interpretationsof scoresin relationto system-
atized concepts as falsifiable claims (Messick 1989,
13-4). Scholarsshould treat these claimsjust as they
would any casualhypothesis,that is, as tentativestate-
ments that require supportingevidence. Validity as-
sessmentis the searchfor this evidence.
AND CHOICES
MEASUREMENTVALIDITY
ABOUTCONCEPTS
A growing body of work considers the systematic
analysisof conceptsan importantcomponentof polit-
ical science methodology.3How shouldwe understand
the relation between issues of measurementvalidity
and broader choices about concepts, which are a
centralfocus of this literature?
Conceptual Choices: Forming the
Systematized Concept
We view systematizedconcepts as the point of depar-
ture for assessingmeasurementvalidity.How do schol-
ars formsuchconcepts?Becausebackgroundconcepts
routinelyincludea varietyof meanings,the formation
of systematized concepts often involves choosing
among them. The number of feasible options varies
greatly.At one extremeare conceptssuch as triangle,
which are routinelyunderstoodin terms of a single
conceptualsystematization;at the other extreme are
"contestedconcepts"(Gallie 1956),suchas democracy.
A carefulexaminationof diversemeaningshelpsclarify
the options,but ultimatelychoices must be made.
These choices are deeply interwinedwith issues of
theory, as emphasizedin Kaplan's(1964, 53) paradox
of conceptualization:"Properconcepts are needed to
formulatea good theory,but we need a good theoryto
arrive at the proper concepts.... The paradox is
resolvedby a processof approximation: the better our
3
Examples of earlier work in this tradition are Sartori 1970, 1984
and Sartori, Riggs, and Teune 1975. More recent studies include
Collier and Levitsky 1997; Collier and Mahon 1993; Gerring 1997,
1999, 2001; Gould 1999; Kurtz 2000; Levitsky 1998; Schaffer 1998.
Important work in political theory includes Bevir 1999; Freeden
1996; Gallie 1956; Pitkin 1967, 1987.
532
II
September 2001
I
concepts,the better the theorywe can formulatewith
them, and in turn,the betterthe conceptsavailablefor
the next, improvedtheory."Various examplesof this
intertwiningare exploredin recent analysesof impor-
tant concepts, such as Laitin's (2000) treatment of
languagecommunityand Kurtz's(2000) discussionof
peasant.Fearon and Laitin's(2000) analysisof ethnic
conflict,in whichthey begin with their hypothesisand
ask what operationalizationis needed to capturethe
conceptions of ethnic group and ethnic conflict en-
tailed in this hypothesis,furtherillustratesthe interac-
tion of theoryand concepts.
In dealingwith the choices that arise in establishing
the systematizedconcept,researchersmustavoidthree
commontraps.First,they should not misconstruethe
flexibilityinherentin these choices as suggestingthat
everythingis up for grabs. This is rarely,if ever, the
case. In any field of inquiry,scholarscommonlyasso-
ciate a matrix of potential meaningswith the back-
groundconcept. This matrixlimits the range of plau-
sible options, and the researcherwho straysoutside it
runsthe riskof being dismissedor misunderstood.We
do not mean to implythat the backgroundconcept is
entirelyfixed.It evolvesover time, as new understand-
ingsare developedandold ones are revisedor fall from
use. At a giventime, however,the backgroundconcept
usuallyprovidesa relativelystablematrix.It is essential
to recognizethat a real choice is being made, but it is
no less essential to recognize that this is a limited
choice.
Second, scholarsshould avoidclaimingtoo much in
defendingtheirchoice of a givensystematizedconcept.
It is not productive to treat other options as self-
evidentlyruled out by the backgroundconcept. For
example,in the controversyover whether democracy
versusnondemocracyshouldbe treatedas a dichotomy
or in termsof gradations,there is too muchrelianceon
claims that the backgroundconcept of democracy
inherentlyrulesout one approachor the other (Collier
and Adcock 1999, 546-50). It is more productiveto
recognize that scholarsroutinelyemphasizedifferent
aspectsof a backgroundconceptin developingsystem-
atized concepts,each of which is potentiallyplausible.
Rather than make sweeping claims about what the
backgroundconcept "really"means, scholars should
present specific arguments,linked to the goals and
context of their research,that justify their particular
choices.
A thirdproblemoccurswhen scholarsstop short of
providinga fleshed-outaccount of their systematized
concepts.This requiresnot just a one-sentencedefini-
tion, but a broaderspecificationof the meaning and
entailmentsof the systematizedconcept. Within the
psychometricsliterature,Shepard(1993, 417) summa-
rizes what is required: "both an internal model of
interrelateddimensionsor subdomains"of the system-
atized concept, and "an externalmodel depictingits
relationshipto other [concepts]."An exampleis Bol-
len's (1990, 9-12; see also Bollen 1980) treatmentof
political democracy,which distinguishesthe two di-
mensionsof "politicalrights"and "politicalliberties,"
clarifiesthese by contrastingthem with the dimensions
 Amerian
American Political Poitica
Sciece
Science Reiew
Review
developedby Dahl, and exploresthe relationbetween
them. Bollen further specifies political democracy
through contrastswith the concepts of stability and
social or economic democracy.In the language of
Sartori(1984,51-4), this involvesclarifyingthe seman-
tic field.
One consequenceof this effortto providea fleshed-
out account may be the recognitionthat the concept
needs to be disaggregated.Whatbegins as a consider-
ation of the internaldimensionsor componentsof a
single concept may become a discussionof multiple
concepts. In democratictheory an importantexample
is the discussionof majorityrule and minorityrights,
which are variouslytreated as componentsof a single
overall concept of democracy,as dimensions to be
analyzedseparately,or as the basisfor formingdistinct
subtypes of democracy (Dahl 1956; Lijphart 1984;
Schmitterand Karl1992).Thiskindof refinementmay
resultfrom new conceptualand theoreticalarguments
or fromempiricalfindingsof the sort that are the focus
of the convergent/discriminant validationprocedures
discussedbelow.
Measurement Validity and the Systematized
Versus Background Concept
We stated earlierthat the systematizedconcept,rather
than the backgroundconcept, should be the focus in
measurementvalidation.Consideran example.A re-
searchermay ask:"Is it appropriatethat Mexico,prior
to the year 2000 (when the previouslydominantparty
handed over power after losing the presidentialelec-
tion), be assigneda score of 5 out of 10 on an indicator
of democracy?Does this score really capture how
'democratic'Mexico was compared to other coun-
tries?"Sucha questionremainsunderspecifieduntilwe
knowwhether "democratic"refers to a particularsys-
tematized concept of democracy,or whether this re-
searcher is concerned more broadly with the back-
groundconcept of democracy.Scholarswho question
Mexico's score should distinguishtwo issues: (1) a
concern about measurement-whether the indicator
employed producesscores that can be interpretedas
adequatelycapturingthe systematizedconceptused in
a given study and (2) a conceptualconcern-whether
the systematizedconcept employed in creating the
indicatoris appropriatevis-a-visthe backgroundcon-
cept of democracy.
We believevalidationshouldfocus on the firstissue,
whereas the second is outside the realm of measure-
ment validity.This distinctionseems especiallyappro-
priate in view of the large numberof contested con-
cepts in political science. The more complex and
contestedthe backgroundconcept,the more important
it is to distinguishissues of measurementfrom funda-
mental conceptualdisputes.To pose the question of
validitywe need a specificconceptualreferentagainst
which to assess the adequacyof a given measure. A
systematizedconcept provides that referent. By con-
trast,if analystsseek to establishmeasurementvalidity
in relation to a backgroundconcept with multiple
Vl.
Vol. 95
95, No.
No. 3
competingmeanings,they may find a differentanswer
to the validityquestionfor each meaning.
By restrictingthe focusof measurementvalidationto
the systematized concept, we do not suggest that
political scientists should ignore basic conceptual is-
sues. Rather,argumentsaboutthe backgroundconcept
and those about validitycan be addressedadequately
only when each is engaged on its own terms, rather
than conflated into one overly broad issue. Consider
Schumpeter's(1947, chap.21) proceduraldefinitionof
democracy.This definitionexplicitlyrulesout elements
of the backgroundconcept, such as the concernwith
substantivepolicy outcomes, that had been central to
what he calls the classical theory of democracy.Al-
though Schumpeter'sconceptualizationhas been very
influentialin politicalscience, some scholars(Harding
andPetras1988;Mouffe1992)havecalledfor a revised
conceptionthat encompassesother concerns,such as
social and economicoutcomes.This importantdebate
exemplifiesthe kind of conceptualdisputethat should
be placed outside the realm of measurementvalidity.
Recognizingthat a givenconceptualchoice does not
involve an issue of measurementvalidity should not
precludeconsideredargumentsabout this choice. An
exampleis the argumentthat minimaldefinitionscan
facilitatecausalassessment(Alvarezet al. 1996,4; Karl
1990, 1-2; Linz 1975, 181-2; Sartori 1975, 34). For
instance,in the debateabouta proceduraldefinitionof
democracy,a pragmaticargumentcan be made that if
analystswish to studythe casualrelationshipbetween
democracyand socioeconomicequality,then the latter
must be excluded from the systematizationof the
former.The point is thatsuchargumentscan effectively
justify certain conceptual choices, but they involve
issues that are differentfromthe concernsof measure-
ment validation.
Fine-Tuning the Systematized Concept with
Friendly Amendments
We definemeasurementvalidityas concernedwith the
relationamongscores,indicators,and the systematized
concept,butwe do not ruleout the introductionof new
conceptual ideas during the validation process. Key
here is the back-and-forth,iterativenatureof research
emphasizedin Figure 1. Preliminaryempiricalwork
may help in the initialformulationof concepts.Later,
even after conceptualizationappears complete, the
applicationof a proposed indicatormay produce un-
expected observationsthat lead scholars to modify
their systematizedconcepts. These "friendlyamend-
ments" occur when a scholar, out of a concern with
validity,engagesin furtherconceptualworkto suggest
refinementsor make explicit earlier implicit assump-
tions. These amendmentsare friendlybecause they do
not fundamentallychallenge a systematizedconcept
but instead push analyststo capturemore adequately
the ideas containedin it.
A friendly amendmentis illustratedby the emer-
gence of the "expandedproceduralminimum"defini-
tion of democracy(Collierand Levitsky1997, 442-4).
Scholars noted that, despite free or relatively free
533
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch
elections, some civilian governmentsin Central and
South America to varying degrees lacked effective
power to govern.A basic concernwas the persistence
of "reserveddomains"of militarypower over which
elected governmentshad little authority(Valenzuela
1992,70). Becauseproceduraldefinitionsof democracy
did not explicitlyaddress this issue, measures based
upon them could result in a high democracyscore for
these countries,but it appearedinvalidto view them as
democratic.Some scholars therefore amended their
systematizedconcept of democracyto add the differ-
entiatingattributethat the elected governmentmustto
a reasonabledegreehave the powerto rule (Karl1990,
2; Loveman1994, 108-13; Valenzuela 1992, 70). De-
bate persistsover the scoringof specificcases (Rabkin
1992, 165), but this innovation is widely accepted
among scholarsin the proceduraltradition(Hunting-
ton 1991, 10; Mainwaring,Brinks, and Perez-Linan
2001;Markoff1996, 102-4). As a resultof this friendly
amendment,analystsdid a betterjob of capturing,for
these new cases, the underlyingidea of procedural
minimumdemocracy.
SPECIFICITY,AND
CONTEXTUAL
VALIDITY,
EQUIVALENCE
Contextualspecificityis a fundamentalconcern that
ariseswhen differencesin contextpotentiallythreaten
the validityof measurement.This is a centraltopic in
psychometrics,the field that has produced the most
innovative work on validity theory. This literature
emphasizesthat the same score on an indicatormay
have differentmeanings in different contexts (Moss
1992, 236-8; see also Messick 1989, 15). Hence, the
validationof an interpretationof scores generatedin
one context does not imply that the same interpreta-
tion is validfor scoresgeneratedin anothercontext.In
political science, this concern with context can arise
when scholarsare makingcomparisonsacrossdifferent
world regionsor distincthistoricalperiods.It can also
arise in comparisonswithina national(or other) unit,
giventhat differentsubunits,regions,or subgroupsmay
constitute very different political, social, or cultural
contexts.
The potential difficultythat context poses for valid
measurement, and the related task of establishing
measurementequivalenceacrossdiverseunits,deserve
more attention in political science. In a period when
the quest for generalityis a powerfulimpulse in the
social sciences,scholarssuch as Elster (1999, chap. 1)
have strongly challenged the plausibilityof seeking
general,law-likeexplanationsof politicalphenomena.
A parallelconstrainton the generalityof findingsmay
be imposed by the contextualspecificityof measure-
ment validity.We are not arguingthat the quest for
generalitybe abandoned.Rather, we believe greater
sensitivityto context may help scholarsdevelop mea-
sures that can be validly applied across diverse con-
texts. This goal requires concerted attention to the
issue of equivalence.
534
September
I 2001
Contextual Specificity in Political Research
Contextualspecificityaffects many areas of political
science. It has long been a problemin cross-national
survey research (Sicinski 1970; Verba 1971; Verba,
Nie, and Kim 1978, 32-40; Verba et al. 1987,Appen-
dix). An exampleconcerningfeaturesof nationalcon-
text is Cain and Ferejohn's(1981) discussionof how
the differingstructureof partysystemsin the United
States and Great Britainshouldbe taken into account
when comparingpartyidentification.Contextis also a
concernfor surveyresearchersworkingwithina single
nation,who wrestlewith the dilemmaof "inter-person-
ally incomparableresponses"(Brady1985).For exam-
ple, scholarsdebate whether a given surveyitem has
the same meaning for different population sub-
groups-which could be defined, for example,by re-
gion, gender, class, or race. One specific concern is
whether populationsubgroupsdiffersystematicallyin
their "response style" (also called "response sets").
Some groups may be more disposed to give extreme
answers,and others may tend toward moderate an-
swers(Greenleaf1992).Bachmanand O'Malley(1984)
show that responsestyle varies consistentlywith race.
They argue that apparently important differences
acrossracialgroupsmayin partreflectonly a different
mannerof answeringquestions.Contextualspecificity
also can be a problemin surveycomparisonsover time,
as Baumgartnerand Walker (1990) point out in dis-
cussinggroupmembershipin the United States.
The issue of contextual specificityof course also
arises in macro-level research in internationaland
comparativestudies (Bollen, Entwisle,and Anderson
1993, 345). Examples from the field of comparative
politicsare discussedbelow. In internationalrelations,
attention to context, and particularlya concern with
"historicizingthe concept of structure,"is central to
"constructivism"(Ruggie 1998, 875). Constructivists
argue that modern internationalrelations rest upon
"constitutiverules" that differ fundamentallyfrom
those of both medievalChristendomand the classical
Greek world (p. 873). Although they recognize that
sovereigntyis an organizingprincipleapplicableacross
diversesettings,the constructivistsemphasizethat the
"meaningand behavioralimplicationsof this principle
vary from one historicalcontext to another"(Reus-
Smit 1997, 567). On the other side of this debate,
neorealistssuch as Fischer(1993, 493) offer a general
warning:If pushedto an extreme,the "claimto context
dependency"threatensto "makeimpossiblethe collec-
tive pursuit of empiricalknowledge."He also offers
specifichistoricalsupportfor the basic neorealistposi-
tion that the behaviorof actorsin internationalpolitics
follows consistent patterns. Fischer (1992, 463, 465)
concludes that "the structurallogic of action under
anarchyhas the characterof an objectivelaw,"whichis
grounded in "an unchangingessence of human na-
ture."
The recurringtension in social research between
particularizing and universalizingtendenciesreflectsin
part contrastingdegrees of concern with contextual
specificity.The approachesto establishingequivalence
AmericanPolitical
American PoliticalScience
Science Review
Review
discussedbelowpointto the optionof a middleground.
These approachesrecognizethatcontextualdifferences
are important,but they seek to combine this insight
with the quest for generalknowledge.
The lessons for political science are clear. Any
empiricalassessmentof measurementvalidityis neces-
sarilybased on a particularset of cases, and validity
claimsshouldbe made, at least initially,with reference
to this specific set. To the extent that the set is
heterogeneous in ways that may affect measurement
validity,it is essentialto (1) assess the implicationsfor
establishingequivalenceacross these diverse contexts
and, if necessary,(2) adoptcontext-sensitivemeasures.
Extension to additionalcases requiressimilarproce-
dures.
Establishing Equivalence: Context-Specific
Domains of Observation
One important means of establishing equivalence
across diverse contexts is careful reasoning, in the
initial stages of operationalization,about the specific
domainsto whicha systematizedconceptapplies.Well
before thinking about particularscoring procedures,
scholars may need to make context-sensitivechoices
regardingthe partsof the broaderpolity,economy,or
society to which they will applytheir concept.Equiva-
lent observationsmay require,in differentcontexts,a
focus on what at a concrete level might be seen as
distincttypes of phenomena.
Some time ago, Verba (1967) called attentionto the
importanceof context-specificdomainsof observation.
In comparative research on political opposition in
stable democracies,a standardfocus is on political
parties and legislativepolitics, but Verba (pp. 122-3)
notes that this may overlookan analyticallyequivalent
form of oppositionthat crystallizes,in some countries,
in the domainof interestgrouppolitics.Skocpol(1992,
6) makes a parallelargumentin questioningthe claim
that the United Stateswas a "welfarelaggard"because
social provisionwas not launchedon a largescale until
the New Deal. This claim is based on the absence of
standardwelfare programsof the kind that emerged
earlierin Europe but fails to recognizethe distinctive
forms of social provisionin the United States, such as
veterans'benefits and support for mothers and chil-
dren. Skocpol arguesthat the welfare laggardcharac-
terization resulted from looking in the wrong place,
that is, in the wrong domainof policy.
Locke and Thelen (1995, 1998) have extended this
approachin their discussionof "contextualizedcom-
parison."They argue that scholarswho studynational
responsesto externalpressurefor economicdecentral-
ization and "flexibilization"routinely focus on the
points at which conflict emerges over this economic
transformation.Yet, these "stickingpoints" may be
located in differentpartsof the economicand political
system.With regardto laborpoliticsin differentcoun-
tries, such conflictsmay arise over wage equity,hours
of employment,workforce reduction, or shop-floor
reorganization.These differentdomainsof labor rela-
tions must be examinedin order to gatheranalytically
Vol. 95,
Vol. No. 3
95, No.
equivalentobservationsthat adequatelytap the con-
cept of stickingpoint. Scholarswho look only at wage
conflicts run the risk of omitting, for some national
contexts,domainsof conflictthat are highlyrelevantto
the concept they seek to measure.
By allowingthe empiricaldomainto whicha system-
atized conceptis appliedto varyacrossthe unitsbeing
compared,analystsmay take a productivestep toward
establishingequivalenceamong diversecontexts.This
practice must be carefullyjustified, but under some
circumstancesit can makean importantcontributionto
valid measurement.
Establishing Equivalence: Context-Specific
Indicators and Adjusted Common
Indicators
Two other ways of establishingequivalence involve
carefulwork at the level of indicators.We will discuss
context-specificindicators,4and what we call adjusted
commonindicators.In this second approach,the same
indicator is applied to all cases but is weighted to
compensatefor contextualdifferences.
An exampleof context-specificindicatorsis found in
Nie, Powell, and Prewitt's (1969, 377) five-country
study of political participation.For all the countries,
they analyzefour relativelystandardattributesof par-
ticipation.Regardinga fifth attribute-membershipin
a political party-they observe that in four of the
countriespartymembershiphas a roughlyequivalent
meaning, but in the United States it has a different
form and meaning.The authorsconcludethat involve-
ment in U.S. electoralcampaignsreflectsan equivalent
formof politicalparticipation.Nie, Powell,and Prewitt
thus focus on a context-specificdomainof observation
(the procedurejust discussedabove) by shiftingtheir
attention,for the U.S. context,frompartymembership
to campaignparticipation.They then take the further
step of incorporatingwithin their overall index of
political participationcontext-specificindicatorsthat
for each case generatea score for what they see as the
appropriatedomain.Specifically,the overallindex for
the United States includes a measure of campaign
participationratherthan partymembership.
A differentexampleof context-specificindicatorsis
found in comparative-historical research,in the effort
to establish a meaningfulthreshold for the onset of
democracyin the nineteenthand early twentiethcen-
tury, as opposed to the late twentieth century.This
effort in turn lays a foundation for the comparative
analysisof transitionsto democracy.One problemin
establishingequivalenceacross these two eras lies in
the fact that the plausibleagendaof "full"democrati-
zation has changed dramaticallyover time. "Full"by
the standardsof an earlierperiodis incompleteby later
standards.For example,by the late twentiethcentury,
universalsuffrageand the protectionof civil rightsfor
the entire nationalpopulationhad come to be consid-
ered essential featuresof democracy,but in the nine-
4 This approachwas originallyused by Przeworski
andTeune (1970,
chap.6), who employedthe label "system-specific
indicator."
535
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch
teenth centurythey were not (Huntington1991,7, 16).
Yet, if the more recent standardis applied to the
earlierperiod,cases are eliminatedthathavelong been
considered classic examples of nascent democratiza-
tion in Europe. One solution is to compare regimes
with respect to a systematizedconcept of full democ-
ratization that is operationalized according to the
normsof the respectiveperiods(Collier 1999,chap. 1;
Russett 1993, 15; see also Johnson1999, 118). Thus, a
differentscoringprocedure-a context-specificindica-
tor-is employedin each period in order to produce
scoresthat are comparablewith respectto this system-
atized concept.5
Adjusted common indicators are another way to
establishequivalence.An exampleis found in Moene
and Wallerstein's(2000) quantitativestudy of social
policy in advancedindustrialsocieties, which focuses
specificallyon publicsocialexpendituresfor individuals
outside the laborforce. One componentof their mea-
sure is public spending on health care. The authors
argue that in the United States such health care
expenditureslargelytargetthose who are not members
of the labor force. By contrast, in other countries
health expendituresare allocated without respect to
employmentstatus. Because U.S. policy is distinctive,
the authorsmultiplyhealth care expendituresin the
other countriesby a coefficientthat lowerstheir scores
on thisvariable.Theirscoresare therebymaderoughly
equivalent-as part of a measure of public expendi-
tures on individualsoutside the labor force-to the
U.S. score.A paralleleffortto establishequivalencein
the analysis of economic indicators is provided by
Zeitsch,Lawrence,and Salernian(1994, 169),who use
an adjustmenttechnique in estimating total factor
productivityto take accountof the differentoperating
environments,and hence the differentcontext, of the
industriesthey compare.Expressingindicatorsin per-
capitatermsis also an exampleof adjustingindicators
in light of context. Overall, this practice is used to
addressboth very specificproblemsof equivalence,as
with the Moene and Wallersteinexample,as well as
more familiarconcerns,such as standardizingby pop-
ulation.
Context-specificindicators and adjusted common
indicatorsare not always a step forward,and some
scholarshaveself-consciouslyavoidedthem.The use of
such indicatorsshould match the analyticgoal of the
researcher.For example,manywho study democrati-
zation in the late twentiethcenturydeliberatelyadopt
a minimumdefinitionof democracyin order to con-
centrateon a limitedset of formalprocedures.Theydo
this out of a convictionthat these formal procedures
are important,even though they may have different
meaningsin particularsettings.Even a scholarsuch as
O'Donnell (1993, 1355),who has devoted great atten-
tion to contextualizingthe meaning of democracy,
insists on the importanceof also retaininga minimal
definition of "political democracy"that focuses on
5 A well-known
example of applying different standards for democ-
racy in making comparisons across international regions is Lipset
(1959, 73-4).
536
September 2001
basic formalprocedures.Thus,for certainpurposes,it
can be analyticallyproductive to adopt a standard
definitionthatignoresnuancesof contextandapplythe
same indicatorto all cases.
In conclusion,we note that althoughPrzeworskiand
Teune's (1970) and Verba'sargumentsabout equiva-
lence are well known, issues of contextualspecificity
and equivalencehave not receivedadequateattention
in political science. We have identifiedthree tools-
context-specificdomains of observation,context-spe-
cific indicators,and adjustedcommon indicators-for
addressingthese issues, and we encouragetheir wider
use. We also advocate greater attention to justifying
their use. Claimsabout the appropriatenessof contex-
tual adjustmentsshould not simplybe asserted;their
validity needs to be carefully defended. Later, we
explorethree types of validationthat may be fruitfully
applied in assessing proposals for context-sensitive
measurement.In particular,content validation,which
focuses on whether operationalizationcaptures the
ideas containedin the systematizedconcept,is central
to determiningwhether and how measurementneeds
to be adjustedin particularcontexts.
ALTERNATIVEPERSPECTIVESON TYPES
OF VALIDATION
Discussionsof measurementvalidityare confounded
by the proliferationof differenttypesof validation,and
by an even greaternumberof labels for them. In this
section we reviewthe emergenceof a unifiedconcep-
tion of measurementvalidityin the field of psychomet-
rics, propose revisionsin the terminologyfor talking
about validity,and examinethe importanttreatments
of validationin political analysisofferedby Carmines
and Zeller, and by Bollen.
Evolving Understandings of Validity
In the psychometrictradition,currentthinkingabout
measurementvaliditydevelopedin two phases. In the
firstphase,scholarswrote about"typesof validity"in a
waythat often led researchersto treateach type as if it
independentlyestablisheda distinctformof validity.In
discussingthis literaturewe follow its terminologyby
referringto types of "validity."As noted above, in the
rest of this articlewe refer insteadto types of "valida-
tion."
The firstpivotaldevelopmentin the emergenceof a
unified approach occurred in the 1950s and 1960s,
when a threefold typology of content, criterion,and
constructvaliditywas officiallyestablishedin reaction
to the confusiongeneratedby the earlierproliferation
of types.6Other labels continued to appear in other
disciplines,but this typologybecame an orthodoxyin
6 The second of these is often called criterion-related validity.
Regarding these official standards, see American Psychological As-
sociation 1954, 1966; Angoff 1988, 25; Messick 1989, 16-7; Shultz,
Riggs, and Kottke 1998, 267-9. The 1954 standards initially pre-
sented four types of validity, which became the threefold typology in
1966 when "predictive" and "concurrent"validity were combined as
"criterion-related" validity.
AmericanPolitical
American PoliticalScience
Science Review
Review
psychology.A recurringmetaphorin that field charac-
terized the three types as "somethingof a holy trinity
representingthree differentroads to psychometricsal-
vation"(Guion 1980, 386). These types may be briefly
defined as follows.
* Content validity assesses the degree to which an
indicatorrepresentsthe universeof content entailed
in the systematizedconcept being measured.
* Criterionvalidityassesses whether the scores pro-
ducedby an indicatorare empiricallyassociatedwith
scores for other variables,called criterionvariables,
which are considered direct measures of the phe-
nomenon of concern.
* Constructvalidityhas had a range of meanings.One
centralfocus has been on assessingwhethera given
indicatoris empiricallyassociatedwith other indica-
tors in a way that conformsto theoreticalexpecta-
tions about their interrelationship.
These labels remain very influentialand are still the
centerpiecein some discussionsof measurementvalid-
ity, as in the latest edition of Babbie's (2001, 143-4)
widely used methodstextbookfor undergraduates.
The second phase grewout of increasingdissatisfac-
tion with the "trinity"and led to a "unitarian"ap-
proach (Shultz, Riggs, and Kottke 1998, 269-71). A
basic problem identified by Guion (1980, 386) and
others was that the threefold typologywas too often
taken to mean that any one type was sufficientto
establishvalidity(Angoff 1988, 25). Scholarsincreas-
ingly argued that the differenttypes should be sub-
sumedundera singleconcept.Hence, to continuewith
the priormetaphor,the earliertrinitycame to be seen
"in a monotheistic mode as the three aspects of a
unitarypsychometricdivinity"(p. 25).
Much of the second phase involveda reconceptual-
ization of constructvalidityand its relationto content
and criterionvalidity.A centralargumentwas that the
latter two may each be necessaryto establishvalidity,
but neitheris sufficient.They shouldbe understoodas
part of a larger process of validationthat integrates
"multiplesources of evidence"and requiresthe com-
binationof "logicalargumentand empiricalevidence"
(Shepard 1993, 406). Alongside this development,a
reconceptualization of constructvalidityled to "amore
comprehensive and theory-basedview that subsumed
other more limited perspectives"(Shultz, Riggs, and
Kottke 1998, 270). This broader understandingof
constructvalidityas the overarchinggoal of a single,
integratedprocessof measurementvalidationis widely
endorsed by psychometricians.Moss (1995, 6) states
"thereis a close to universalconsensusamongvalidity
theorists"that "content-and criterion-relatedevidence
of validityare simply two of many types of evidence
that supportconstructvalidity."
Thus, in the psychometricliterature(e.g., Messick
1980, 1015), the term "constructvalidity"has become
essentiallya synonymfor what we call measurement
validity.We have adoptedmeasurementvalidityas the
name for the overall topic of this article, in part
because in politicalscience the label constructvalidity
commonlyrefers to specificproceduresratherthan to
Vol.
Vol. 95, No. 3
the general idea of valid measurement.These specific
proceduresgenerallydo not encompasscontentvalida-
tion and have in common the practice of assessing
measurementvalidityby takingas a point of reference
establishedconceptualand/ortheoreticalrelationships.
We findit helpfulto groupthese proceduresinto two
typesaccordingto the kindof theoreticalor conceptual
relationship that serves as the point of reference.
Specifically,these types are based on the heuristic
distinctionbetweendescriptionand explanation.7First,
some procedures rely on "descriptive"expectations
concerningwhethergiven attributesare understoodas
facets of the same phenomenon.This is the focus of
what we label "convergent/discriminant validation."
Second,other proceduresrely on relativelywell-estab-
lished "explanatory"causal relations as a baseline
against which measurementvalidity is assessed. In
labelingthis second group of procedureswe drawon
Campbell's(1960, 547) helpful term, "nomological"
validation, which evokes the idea of assessment in
relation to well-establishedcausal hypothesesor law-
like relationships.This second type is often called
construct validity in political research (Berry et al.
1998;Elkins2000).8Out of deferenceto this usage, in
the headings and summarystatementsbelow we will
refer to nomological/construct validation.
Types of Validation in Political Analysis
A baseline for the revised discussion of validation
presentedbelow is providedin workby Carminesand
Zeller, and by Bollen. Carminesand Zeller (1979, 26;
Zeller and Carmines1980, 78-80) argue that content
validationand criterionvalidationare of limitedutility
in fields such as political science. While recognizing
that contentvalidationis importantin psychologyand
education,they arguethat evaluatingit "hasprovedto
be exceedinglydifficultwith respectto measuresof the
more abstractphenomenathat tend to characterizethe
social sciences"(Carminesand Zeller 1979, 22). For
criterionvalidation,these authors emphasize that in
many social sciences, few "criterion"variables are
available that can serve as "real" measures of the
phenomenaunderinvestigation,againstwhichscholars
can evaluatealternativemeasures(pp. 19-20). Hence,
for manypurposesit is simplynot a relevantprocedure.
Although Carmines and Zeller call for the use of
multiple sources of evidence, their emphasis on the
limitationsof the first two types of validation leads
them to give a predominant role to nomological/
constructvalidation.
In relation to Carminesand Zeller, Bollen (1989,
185-6, 190-4) addsconvergent/discriminant validation
7 Descriptionand explanationare of courseintertwined,but we find
this distinctioninvaluablefor exploringcontrastsamongvalidation
procedures.While these proceduresdo not alwaysfit in sharply
boundedcategories,manydo indeed focus on either descriptiveor
explanatoryrelationsand hence are productivelydifferentiatedby
our typology.
8 See also the mainexamplesof constructvalidationpresentedin the
majorstatementsby CarminesandZeller 1979,23, andBollen 1989,
189-90.
537
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch
to theirthree typesand emphasizescontentvalidation,
whichhe sees as both viableand fundamental.He also
raises general concerns about correlation-basedap-
proachesto convergentand nomological/construct val-
idation,and he offersan alternativeapproachbased on
structuralequationmodelingwith latentvariables(pp.
192-206). Bollen sharesthe concernof Carminesand
Zeller that, for most social research,"true"measures
do not exist againstwhich criterionvalidationcan be
carriedout, so he likewise sees this as a less relevant
type (p. 188).
These valuable contributionscan be extended in
severalrespects.First,with referenceto Carminesand
Zeller's critique of content validation,we recognize
that this procedure is harder to use if concepts are
abstractand complex.Moreover,it often does not lend
itself to the kind of systematic,quantitativeanalysis
routinely applied in some other kinds of validation.
Yet, like Bollen (1989, 185-6, 194),we are convincedit
is possible to lay a secure foundation for content
validationthat will make it a viable,and indeed essen-
tial, procedure. Our discussion of this task below
derives from our distinctionbetween the background
and the systematizedconcept.
Second, we share the convictionof Carminesand
Zeller that nomological/construct validationis impor-
tant, yet given our emphasison content and conver-
gent/discriminantvalidation,we do not privilegeit to
the degree they do. Our discussionwill seek to clarify
some aspectsof how this procedureactuallyworksand
will addressthe skepticalreactionof manyscholarsto
it.
Third,we have a twofoldresponseto the critiqueof
criterionvalidationas irrelevantto mostformsof social
research.On the one hand,in some domainscriterion
validationis important,and this must be recognized.
For example, the literature on response validity in
surveyresearchseeks to evaluateindividualresponses
to questions, such as whether a person voted in a
particularelection,by comparingthemto officialvoting
records (Anderson and Silver 1986; Clausen 1968;
Katoshand Traugott1980). Similarly,in panel studies
it is possibleto evaluatethe adequacyof "recall"(i.e.,
whetherrespondentsremembertheirown earlieropin-
ions, dispositions,and behavior)throughcomparison
with responses in earlier studies (Niemi, Katz, and
Newman 1980). On the other hand, this is not one of
the most generallyapplicabletypes of validation,and
we favor treatingit as one subtypewithinthe broader
categoryof convergentvalidation.As discussedbelow,
convergentvalidationcomparesa given indicatorwith
one or more other indicatorsof the concept-in which
the analyst may or may not have a higher level of
confidence.Even if these otherindicatorsare as fallible
as the indicatorbeing evaluated,the comparisonpro-
vides greaterleveragethan does lookingonly at one of
them in isolation.To the extentthat a well-established,
direct measure of the phenomenon under study is
available,convergentvalidationis essentiallythe same
as criterionvalidation.
Finally,in contrastboth to Carminesand Zeller and
to Bollen, we will discussthe applicationof the differ-
538
September 2001
ent types of validationin qualitativeas well as quanti-
tativeresearch,usingexamplesdrawnfromboth tradi-
tions. Furthermore,we will employcrucialdistinctions
introducedabove,includingthe differentiationof levels
presentedin Figure 1, as well as the contrastbetween
specific proceduresfor validation,as opposed to the
overallidea of measurementvalidity.
THREE TYPES OF MEASUREMENT
VALIDATION:QUALITATIVEAND
QUANTITATIVEEXAMPLES
We now discuss various procedures,both qualitative
and quantitative,for assessingmeasurementvalidity.
We organizeour presentationin terms of a threefold
typology:content,convergent/discriminant, and nomo-
logical/construct validation. The goal is to explicate
each of these types by posing a basic questionthat, in
all three cases, can be addressedby both qualitative
and quantitativescholars.Two caveatsshouldbe intro-
duced. First, while we discuss correlation-basedap-
proaches to validity assessment, this article is not
intendedto providea detailedor exhaustiveaccountof
relevantstatisticaltests. Second,we recognizethat no
rigid boundariesexist among alternativeprocedures,
given that one occasionallyshades off into another.
Our typology is a heuristic device that shows how
validationprocedurescan be groupedin termsof basic
questions,and therebyhelps bringinto focus parallels
and contrastsin the approachesto validationadopted
by qualitativeand quantitativeresearchers.
Content Validation
Basic Question. In the frameworkof Figure 1, does a
given indicator (level 3) adequatelycapture the full
content of the systematizedconcept (level 2)? This
"adequacyof content"is assessedthroughtwo further
questions. First, are key elements omitted from the
indicator? Second, are inappropriateelements in-
cludedin the indicator?9An examinationof the scores
(level 4) of specific cases may help answer these
questionsabout the fit between levels 2 and 3.
Discussion. In contrastto the other typesconsidered,
contentvalidationis distinctivein its focus on concep-
tual issues, specifically,on what we have just called
adequacyof content. Indeed, it developedhistorically
as a correctiveto formsof validationthatfocusedsolely
on the statisticalanalysisof scores, and in so doing
overlookedimportantthreatsto measurementvalidity
(Sireci 1998, 83-7).
Because contentvalidationinvolvesconceptualrea-
soning, it is imperativeto maintainthe distinctionwe
made between issues of validationand questionscon-
cerningthe backgroundconcept. If content validation
is to be useful, then there must be some ground of
conceptual agreement about the phenomena being
investigated(Bollen 1989, 186; Cronbachand Meehl
9 Some readers may think of these questions as raising issues of "face
validity."We have found so many different definitions of face validity
that we prefer not to use this label.
American Political Science Review
Scec Rve
IPoiia
AeiaI
1955, 282). Withoutit, a well-focusedvalidationques-
tion may rapidlybecome entangled in a broaderdis-
pute over the concept. Such agreementcan be pro-
vided if the systematizedconcept is taken as given, so
attention can be focused on whether a particular
indicatoradequatelycapturesits content.
Examples of Content Validation. Within the psycho-
metrictradition(Angoff1988,27-8; Shultz,Riggs,and
Kottke 1998, 267-8), content validationis understood
as focusing on the relationshipbetween the indicator
(level 3) and the systematizedconcept (level 2), with-
out reference to the scores of specificcases (level 4).
We will first present examplesfrom political science
that adopt this focus. We will then turnto a somewhat
different, "case-oriented"procedure (Ragin 1987,
chap. 3), identifiedwith qualitativeresearch,in which
the examinationof scores for specific cases plays a
centralrole in contentvalidation.
Two examplesfrom political researchillustrate,re-
spectively,the problemsof omission of key elements
from the indicatorand inclusionof inappropriateele-
ments.Paxton's(2000) articleon democracyfocuseson
the firstproblem.Her analysisis particularlysalientfor
scholarsin the qualitativetradition,given its focus on
choices about the dichotomousclassificationof cases.
Paxtoncontraststhe systematizedconcepts of democ-
racy offered by several prominent scholars-Bollen,
Gurr,Huntington,Lipset,Muller,and Rueschemeyer,
Stephens,and Stephens-with the actualcontentof the
indicatorsthey propose. She takes their systematized
concepts as given, which establishescommonconcep-
tual ground. She observesthat these scholarsinclude
universalsuffragein whatis in effecttheirsystematized
concept of democracy,but the indicatorsthey employ
in operationalizingthe concept consider only male
suffrage.Paxton thus focuses on the problemthat an
importantcomponent of the systematizedconcept is
omitted from the indicator.
The debateon Vanhanen's(1979, 1990)quantitative
indicatorof democracyillustratesthe alternativeprob-
lem that the indicatorincorporateselements that cor-
respond to a concept other than the systematized
concept of concern. Vanhanen seeks to capture the
idea of politicalcompetitionthat is part of his system-
atizedconceptof democracyby including,as a compo-
nent of his scale,the percentageof votes won by parties
other than the largestparty.Bollen (1990, 13, 15) and
Coppedge (1997, 6) both question this measure of
democracy, arguing that it incorporates elements
drawn from a distinct concept, the structureof the
partysystem.
Case-Oriented Content Validation. Researchers en-
gaged in the qualitativeclassificationof cases routinely
carryout a somewhatdifferentprocedurefor content
validation,based on the relation between conceptual
meaningand choices aboutscoringparticularcases. In
the vocabularyof Sartori(1970, 1040-6), this concerns
the relation between the "intension"(meaning) and
"extension"(set of positivecases) of the concept. For
Sartori,an essentialaspectof conceptformationis the
procedureof adjustingthis relationbetween cases and
Vol. 95,
Vol. 95, No. 3
concept.In the frameworkof Figure 1, this procedure
involvesrevisingthe indicator(i.e., the scoringproce-
dure) in order to sort cases in a way that better fits
conceptual expectations, and potentially fine-tuning
the systematizedconcept to better fit the cases. Ragin
(1994, 98) terms this process of mutual adjustment
"double fitting." This procedure avoids conceptual
stretching(Collierand Mahon1993;Sartori1970),that
is, a mismatchbetweena systematizedconceptand the
scoringof cases, which is clearlyan issue of validity.
An example of case-orientedcontent validation is
found in O'Donnell's(1996) discussionof democratic
consolidation.Some scholarssuggestthatone indicator
of consolidationis the capacityof a democraticregime
to withstandsevere crises. O'Donnell argues that by
this standard, some Latin American democracies
would be consideredmore consolidatedthan those in
southernEurope.He findsthis an implausibleclassifi-
cation because the standardleads to a "reductioad
absurdum"(p. 43). This exampleshows how attention
to specific cases can spur recognitionof dilemmasin
the adequacyof contentandcanbe a productivetool in
content validation.
In sum,for case-orientedcontentvalidation,upward
movementin Figure 1 is especiallyimportant.It can
lead to both refiningthe indicatorin lightof scoresand
fine-tuningthe systematizedconcept. In addition,al-
though the systematizedconcept being measured is
usually relativelystable, this form of validationmay
lead to friendlyamendmentsthat modify the system-
atized concept by drawingideas from the background
concept. To put this another way, in this form of
validationboth an "inductive"componentand concep-
tual innovationare especiallyimportant.
Limitations of Content Validation. Content validation
makes an importantcontributionto the assessmentof
measurementvalidity,but alone it is incomplete,for
two reasons.First,althougha necessarycondition,the
findingsof content validationare not a sufficientcon-
dition for establishingvalidity(Shepard 1993, 414-5;
Sireci 1998, 112). The key point is that an indicator
with valid content may still produce scores with low
overall measurementvalidity,because furtherthreats
to validitycan be introducedin the coding of cases. A
second reason concerns the trade-offbetween parsi-
mony and completenessthat arisesbecause indicators
routinelyfail to capturethe full content of a system-
atized concept. Capturingthis content may require a
complexindicatorthat is hardto use and adds greatly
to the time and cost of completingthe research.It is a
matterof judgmentfor scholarsto decide when efforts
to further improvethe adequacyof content may be-
come counterproductive.
It is usefulto complementthe conceptualcriticismof
indicatorsby examiningwhether particularmodifica-
tions in an indicatormakea differencein the scoringof
cases. To the extent that such modificationshave little
influence on scores, their contributionto improving
validity is more modest. An example in which their
contributionis shown to be substantialis providedby
Paxton(2000). She developsan alternativeindicatorof
539
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch September 2001

democracythat takes female suffrage into account, atheoretically.Scholarsshouldhavespecificconceptual

comparesthe scores it produceswith those produced reasonsfor expectingconvergenceif it is to constitute
by the indicatorsshe originallycriticized,and shows evidencefor validity.Let us suppose a proposedindi-
that her revisedindicatoryields substantiallydifferent cator is meant to capturea facet of democracyover-
findings.Her content validationargumentstands on looked by existingmeasures;then too high a correla-
conceptualgroundsalone, but her informationabout tion is in fact negativeevidenceregardingvalidity,for
scoring demonstratesthe substantiveimportanceof it suggeststhat nothingnew is being captured.
her concerns.This comparisonof indicatorsin a sense An exampleof discriminantvalidationis providedby
introducesus to convergent/discriminantvalidation,to Bollen's (1980, 373-4) analysisof voter turnout.As in
whichwe now turn. the studiesjust noted, differentmeasuresof democracy
are compared,but in this instancethe goal is to find
Convergent/Discriminant Validation empiricalsupportfor divergence.Bollen claims,based
on contentvalidation,thatvoter turnoutis an indicator
Basic Question. Are the scores (level 4) producedby of a
alternativeindicators(level 3) of a given systematized concept distinctfrom the systematizedconcept of
political democracy.The low correlationof voter turn-
concept (level 2) empiricallyassociatedand thus con- out with other
proposedindicatorsof democracypro-
vergent? Furthermore,do these indicators have a vides discriminantevidencefor this claim.Bollen con-
weaker associationwith indicatorsof a second, differ- cludes that turnoutis best understoodas an indicator
ent systematizedconcept, thus discriminatingthis sec- of
ond groupof indicatorsfrom the first?Strongerasso- ized politicalparticipation,which shouldbe conceptual-
as distinctfrom politicaldemocracy.
ciationsconstituteevidence that supportsinterpreting
indicatorsas measuringthe same systematizedcon- data Although qualitativeresearchersroutinelylack the
necessaryfor the kind of statisticalanalysisper-
cept-thus providingconvergentvalidation;whereas formed by Bollen, convergent/discriminant validation
weaker associationssupportthe claim that they mea- is no means irrelevantfor them. often assess
by
sure different concepts-thus providingdiscriminant whetherthe scores for alternativeindicators They
validation.The specialcase of convergentvalidationin converge
or diverge.Paxton,in the examplediscussedabove, in
whichone indicatoris takenas a standardof reference, effect uses discriminantvalidationwhen she
and is used to evaluate one or more alternativeindi- alternative compares
indicatorsof democracyin order
qualitative
cators, is called criterionvalidation,as discussedabove.
to show that recommendationsderived from her as-
Discussion. Carefully defined systematizedconcepts, sessmentof content validationmake a differenceem-
and the availabilityof two or more alternativeindica- pirically.This comparison,based on the assessmentof
tors of these concepts, are the starting point for scores, "discriminates"among alternativeindicators.
convergent/discriminantvalidation. They lay the Convergent/discriminant validation is also employed
groundworkfor argumentsthat particularindicators when qualitativeresearchersuse a multimethodap-
measurethe same or differentconcepts,which in turn proachinvolving"triangulation" amongmultipleindi-
create expectationsabout how the indicatorsmay be catorsbasedon differentkindsof datasources(Brewer
empiricallyassociated. To the extent that empirical and Hunter 1989;Campbelland Fiske 1959;Webb et
findingsmatch these "descriptive"expectations,they al. 1966). Orum,Faegin, and Sjoberg(1991, 19) spe-
providesupportfor validity. cificallyargue that one of the great strengthsof the
Empirical associations are crucial to convergent/ case study tradition is its use of triangulationfor
discriminantvalidation,but they are often simplythe enhancingvalidity.In general,the basic ideas of con-
point of departure for an iterative process. What vergent/discriminant validationare at work in qualita-
initially appears to be negative evidence can spur tive researchwhenever scholars compare alternative
refinementsthat ultimatelyenhance validity.That is, indicators.
the failureto find expectedconvergencemay encour- Concernsabout
Convergent/DiscriminantValidation. A
age a returnto the conceptualand logical analysisof first concernhere is that scholarsmight think that in
indicators,whichmaylead to theirmodification.Alter- validation empiricalfindings
convergent/discriminant
natively, researchers may conclude that divergence always dictate conceptual choices. This frames the
suggeststhe indicatorsmeasuredifferentsystematized issue too narrowly.For example,Bollen (1993, 1208-9,
concepts and may reevaluate the conceptualization 1217) analyzesfour indicatorsthat he takes as compo-
that led them to expect convergence.This process nents of the of political liberties and four
illustratesthe intertwiningof convergentand discrimi- indicatorsthatconcept
he understandsas aspectsof democratic
nant validation. rule. An examinationof Bollen's covariancematrix
Examples of Convergent/Discriminant Validation. reveals that these do not emerge as two separate
Scholars who develop measures of democracy fre- empiricaldimensions.Convergent/discriminant valida-
quently use convergent validation.Thus, analystswho tion, mechanicallyapplied,might lead to a decision to
create a new indicatorcommonlyreportits correlation eliminatethis conceptualdistinction.Bollen does not
with previouslyestablished indicators (Bollen 1980, take that approach.He combinesthe two clustersof
380-2; CoppedgeandReincke1990,61;Mainwaringet indicatorsinto an overall empiricalmeasure, but he
al. 2001, 52; Przeworskiet al. 1996, 52). This is a also maintainsthe conceptual distinction.Given the
valuable procedure, but it should not be employed conceptualcongruencebetweenthe two sets of indica-

540
American Political Science Review
I Ie IR I
tors and the concepts of political liberties and demo-
craticrule, the standardof contentvalidationis clearly
met, and Bollen continues to use these overarching
labels.
Another concern arises over the interpretationof
low correlationsamongindicators.Analystswho lack a
"true"measure againstwhich to assess validitymust
base convergentvalidationon a set of indicators,none
of which may be a very good measureof the system-
atized concept. The result may be low correlations
among indicators,even though they have sharedvari-
ance that measuresthe concept.One possiblesolution
is to focus on this shared variance,even though the
overall correlationsare low. Standardstatisticaltech-
niques may be used to tap this sharedvariance.
The opposite problemalso is a concern:the limita-
tions of inferring validity from a high correlation
among indicators.Such a correlationmay reflect fac-
tors other than valid measurement.For example,two
indicators may be strongly correlated because they
both measure some other concept;or they may mea-
sure differentconcepts,one of whichcauses the other.
A plausibleresponse is to think through,and seek to
rule out, alternativereasonsfor the high correlation.10
Althoughframingthese concernsin the languageof
high and low correlationsappearsto orient the discus-
sion toward quantitativeresearchers,qualitativere-
searchersface parallelissues. Specifically,these issues
arise when qualitativeresearchersanalyzethe sorting
of cases producedby alternativeclassificationproce-
dures that representdifferentways of operationalizing
either a given concept (i.e., convergentvalidation)or
two or more conceptsthat are presumedto be distinct
(i.e., discriminantvalidation).Giventhatthese scholars
are probablyworkingwith a smallN, they maybe able
to drawon their knowledgeof cases to assess alterna-
tive explanationsfor convergences and divergences
among the sortingof cases yielded by differentclassi-
fication procedures.In this way, they can make valu-
able inferences about validity.Quantitativeresearch-
ers, by contrast, have other tools for making these
inferences,to whichwe now turn.
ConvergentValidation and Structural Equation Models
with Latent Variables. In quantitative research, an
importantmeans of respondingto the limitationsof
simple correlational procedures for convergent/dis-
criminantvalidationis offered by structuralequation
models with latent variables(also called LISREL-type
models). Some treatments of such models, to the
extentthat they discussmeasurementerror,focus their
attentionon randomerror,that is, on reliability(Hay-
duk 1987, e.g., 118-24; 1996).11However,Bollen has
made systematicerror, which is the concern of the
10On the appropriatesize of the correlation,see Bollenand Lennox
1991,305-7.
1 To take a political science application,Green and Palmquist's
(1990) studyalso reflectsthis focus on randomerror.By contrast,
Green (1991) goes fartherby consideringboth randomand system-
aticerror.Likethe workby Bollendiscussedbelow,these articlesare
an impressive demonstrationof how LISREL-typemodels can
incorporatea concernwith measurementerror into conventional
statistical analysis, and how this can in turn lead to a major
Vol. 95, No. 3
present article, a central focus in his major method-
ological statementon this approach(1989, 190-206).
He demonstrates,for example,its distinctivecontribu-
tion for a scholarconcernedwith convergent/discrimi-
nantvalidationwho is dealingwith a data set with high
correlationsamongalternativeindicators.In this case,
structuralequationswith latent variablescan be used
to estimatethe degreeto whichthese high correlations
derivefrom sharedsystematicbias, ratherthan reflect
the valid measurementof an underlyingconcept.12
This approachis illustratedby Bollen (1993) and
Bollen and Paxton's(1998, 2000) evaluationof eight
indicatorsof democracytaken from data sets devel-
oped by Banks, Gastil, and Sussman.13For each indi-
cator,Bollen and Paxtonestimatethe percentof total
variancethat validlymeasuresdemocracy,as opposed
to reflectingsystematicand randomerror.The sources
of systematic error are then explored. Bollen and
Paxtonconclude,for example,that Gastil'sindicators
have "conservative" bias, givinghigherscores to coun-
tries that are Catholic, that have traditionalmonar-
chies, and that are not Marxist-Leninist(Bollen 1993,
1221; Bollen and Paxton 2000, 73). This line of re-
search is an outstandingexampleof the sophisticated
use of convergent/discriminant validation to identify
potentialproblemsof politicalbias.
In discussingBollen's treatmentand applicationof
structuralequationmodelswe would like to note both
similarities, and a key contrast, in relation to the
practice of qualitative researchers.Bollen certainly
sharesthe concernwith carefulattentionto concepts,
and with knowledgeof cases, that we have emphasized
above, and that is characteristicof case-orientedcon-
tent validationas practicedby qualitativeresearchers.
He insiststhat complexquantitativetechniquescannot
replace careful conceptualand theoreticalreasoning;
rather they presuppose it. Furthermore,"structural
equationmodels are not very helpful if you have little
idea aboutthe subjectmatter"(Bollen 1989,vi; see also
194). Qualitativeresearchers,carryingout a case-by-
case assessmentof the scores on differentindicators,
could of course reach some of the same conclusions
about validityand politicalbias reachedby Bollen. A
structuralequation approach,however, does offer a
reevaluationof substantivefindings-in this case concerningparty
identification(Greene 1991,67-71).
12Two pointsaboutstructuralequationmodelswith latentvariables
shouldbe underscored.First,as noted below,these modelscan also
be usedin nomological/construct validation,andhenceshouldnot be
associatedexclusivelywithconvergent/discriminant validation,which
is the applicationdiscussedhere. Second,we have emphasizedthat
convergent/discriminant validationfocuseson "descriptive"
relations
among concepts and their components.Withinthis framework,it
merits emphasisthat the indicatorsthat measure a given latent
variable (i.e., concept) in these models are conventionallyinter-
pretedas "effects"of thislatentvariable(Bollen1989,65;Bollenand
Lennox1991,305-6). These effects,however,do not involvecausal
interactionsamongdistinctphenomena.Suchinteractions,whichin
structuralequationmodels involvecausalrelationsamongdifferent
latentvariables,are the centerpieceof the conventionalunderstand-
ing of "explanation."By contrast, the links between one latent
variableandits indicatorsare productivelyunderstoodas involvinga
"descriptive" relationship.
13See, for example,Banks1979;Gastil 1988;Sussman1982.
541
MeasurementValidity:IA Shared
I Standardfor Qualitativeand QuantitativeResearch
I I September 2001

fundamentallydifferentprocedurethat allowsscholars tion. When other approachesyield positive evidence,

to assess carefullythe magnitudeand sourcesof mea- however, then nomologicalvalidation is valuable in
surement error for large numbers of cases and to teasing out potentiallyimportantdifferencesthat may
summarize this assessment systematicallyand con- not be detected by other types of validation.Specifi-
cisely. cally, alternativeindicatorsof a systematizedconcept
may be stronglycorrelatedand yet performvery dif-
Nomological/Construct Validation ferentlywhen employedin causal assessment.Bollen
Basic Question. In a domain of researchin which a (1980, 383-4) shows this, for example, in his assess-
ment of whether regime stabilityshould be a compo-
given causalhypothesisis reasonablywell established, nent of measuresof democracy.
we ask: Is this hypothesisagain confirmedwhen the
cases are scored (level 4) with the proposedindicator Examplesof Nomological/Construct Validation. Lijp-
(level 3) for a systematizedconcept(level 2) that is one hart's(1996) analysisof democracyand conflictman-
of the variables in the hypothesis?Confirmationis agement in India provides a qualitativeexample of
treatedas evidencefor validity. nomologicalvalidation,which he uses to justify his
classificationof India as a consociationaldemocracy.
Discussion. We shouldfirstreiteratethat becausethe
term "constructvalidity"has become synonymousin Lijphartfirst draws on his systematizedconcept of
consociationalismto identify descriptivecriteria for
the psychometricliteraturewith the broadernotion of
measurementvalidity, to reduce confusion we use classifyingany given case as consociational.He then
uses nomologicalvalidationto furtherjustifyhis scor-
Campbell'sterm "nomological"validationfor proce- ing of India (pp. 262-4). Lijphartidentifiesa series of
dures that address this basic question. Yet, given causalfactorsthat he
common usage in political science, in headings and to arguesare routinelyunderstood
produce consociational regimes, and he observes
summarystatementswe call this nomological/construct that these factorsare presentin India.Hence, classify-
validation.We also propose an acronymthat vividly
ing India as consociationalis consistentwith an estab-
capturesthe underlyingidea: AHEM validation;that lished causalrelationship,whichreinforcesthe plausi-
is, "Assumethe Hypothesis,Evaluatethe Measure."
bilityof his descriptiveconclusionthatIndiais a case of
Nomologicalvalidationassessesthe performanceof consociationalism.
indicatorsin relationto causal hypothesesin order to Another qualitativeexampleof nomologicalvalida-
gain leverage in evaluating measurement validity. tion is found in a classic study in the tradition of
Whereas convergent validation focuses on multiple
indicatorsof the same systematizedconcept, and dis- comparative-historical analysis,PerryAnderson'sLin-
theAbsolutist State.14 Anderson(1974, 413-5)
criminantvalidationfocuses on indicatorsof different eages of
is concernedwith whetherit is appropriateto classify
concepts that stand in a "descriptive"relation to one as "feudalism"the politicaland economicsystemthat
another,nomologicalvalidationfocuses on indicators
of differentconceptsthat are understoodto standin an emergedin Japanbeginningroughlyin the fourteenth
century,whichwould place Japanin the same analytic
explanatory,"causal"relation with one another. Al- categoryas Europeanfeudalism.His argumentis partly
though these contrasts are not sharplypresented in descriptive,in that he asserts that "the fundamental
most definitions of nomologicalvalidation, they are resemblance betweenthe two historicalconfigurations
essential in identifyingthis type as distinctfrom con- as a whole [is] unmistakable"(p. 414). He validateshis
vergent/discriminant validation. In practice the con- classification
trastbetween descriptionand explanationdependson by observing that Japan's subsequent
development, like that of postfeudalEurope,followed
the researcher'stheoreticalframework,but the distinc- an economic that his theory explainsas the
tion is fundamentalto the contemporarypractice of trajectory
historicallegacyof a feudal state. "Thebasic parallel-
politicalscience. ism of the two great experiencesof feudalism,at the
The underlyingidea of nomologicalvalidationis that
scores which can validly be claimed to measure a opposite ends of Eurasia,was ultimatelyto receive its
most arrestingconfirmationof all, in the posterior
systematizedconceptshouldfit well-establishedexpec- destinyof each zone" (p. 414). Thus,he uses evidence
tationsderivedfromcausalhypothesesthat involvethis
concerning an expected explanatoryrelationshipto
concept.The firststep is to take as given a reasonably increaseconfidencein his descriptivecharacterization
well-established causal hypothesis, one variable in of Japan as feudal. Anderson,like Lijphart,thus fol-
which corresponds to the systematized concept of lows the two-step procedureof makinga descriptive
concern.The scholarthen examinesthe associationof claim about one or two cases, and then offeringevi-
the proposed indicatorwith indicatorsof the other dence for the
validityof this claimby observingthat it
concepts in the causal hypothesis.If the assessment is consistentwith an explanatoryclaimin whichhe has
produces an association that the causal hypothesis confidence.
leads us to expect, then this is positive evidence for A quantitativeexampleof nomologicalvalidationis
validity. found in Elkins'sevaluationof the proposalthat de-
Nomologicalvalidationprovidesadditionalleverage mocracyversus nondemocracyshould be treated as a
in assessing measurementvalidity. If other types of rather than in terms of gradations.One
validationraise concernsabout the validityof a given dichotomy,
indicator and the scores it produces, then analysts
probablydo not need to employ nomologicalvalida- 14SebastianMazzucasuggestedthis example.

542
 Ameicn
Pliica
American Political Siece evew
Science Review ol 95
Vol. 95, N.
No. 3

potential defense of a dichotomousmeasure is based which the hypothesiseither is or is not reconfirmed,

on convergentvalidation.Thus,Alvarezand colleagues using the proposed indicator. Rather, nomological
(1996, 21) show that their dichotomous measure is validationmayfocus,as it does in Elkins(2000;see also
stronglyassociatedwith graded measures of democ- Hill, Hanna, and Shafqat 1997), on comparingtwo
racy.Elkins (2000, 294-6) goes on to applynomolog- differentindicatorsof the same systematizedconcept,
ical validation,exploringwhether,notwithstandingthis and on askingwhichbetter fits causal expectations.A
association, the choice of a dichotomous measure tentative hypothesis may not provide an adequate
makesa differencefor causalassessment.He compares standardfor rejectingclaims of measurementvalidity
tests of the democraticpeace hypothesisusing both outright,but it may serve as a point of reference for
dichotomousand graded measures.Accordingto the comparing the performance of two indicators and
hypothesis, democracies are in general as conflict therebygainingevidencerelevantto choosingbetween
prone as nondemocraciesbut do not fightone another. them.
The key finding from the standpointof nomological Anotherresponseto the concernthat causalhypoth-
validationis that this claim is stronglysupportedusing eses may be too tentative a groundfor measurement
a graded measure, whereas there is no statistically validation is to recognize that neither measurement
significantsupport using the dichotomous measure. claimsnor causalclaimsare inherentlymore epistemo-
These findings give nomological evidence for the logicallysecure.Both typesof claimsshouldbe seen as
greater validity of the graded measure, because they falsifiablehypotheses.To take a causal hypothesisas
better fit the overall expectations of the accepted given for the sake of measurementvalidationis not to
causal hypothesis.Elkins'sapproachis certainlymore say that the hypothesisis set in stone. It maybe subject
complex than the two-step procedure followed by to criticalassessmentat a later point. Campbell(1977/
Lijphartand Anderson,but the basic idea is the same. 1988,477) expressesthispointmetaphorically: "Weare
like sailorswho must repaira rottingship at sea. We
Skepticismabout Nomological Validation. Many schol- trust the great bulk of the timberswhile we replace a
ars are skeptical about nomologicalvalidation. One
concern is the potentialproblemof circularity.If one particularlyweak plank. Each of the timberswe now
trust we may in turn replace. The proportionof the
assumesthe hypothesisin orderto validatethe indica- we are replacingto thosewe treatas soundmust
planks
tor, then the indicatorcannot be used to evaluatethe alwaysbe small."
same hypothesis.Hence, it is importantto specifythat
any subsequenthypothesis-testingshould involve hy-
potheses different from those used in nomological CONCLUSION
validation. In conclusion,we returnto the four underlyingissues
A second concern is that, in additionto taking the that frameour discussion.First,we have offereda new
hypothesisas given, nomologicalvalidationalso pre- account of different types of validation. We have
supposes the valid measurementof the other system- viewed these types in the frameworkof a unified
atized concept involved in the hypothesis. Bollen conceptionof measurementvalidity.None of the
spe-
(1989, 188-90) notes that problems in the measure- cific types of validation alone establishes validity;
ment of the second indicator can undermine this rather, each provides one kind of evidence to be
approachto assessingvalidity,especiallywhen scholars integratedinto an overallprocessof assessment.Con-
rely on simple correlationalprocedures. Obviously, tent validationmakesthe indispensablecontributionof
researchersneed evidence about the validity of the assessing what we call the adequacy of content of
second indicator.Structuralequation models with la- indicators.Convergent/discriminant validation-taking
tent variablesoffera quantitativeapproachto address- as a baseline descriptiveunderstandingsof the rela-
ing such difficultiesbecause, in additionto evaluating tionship among concepts, and of their relation to
the hypothesis,these models can be specifiedso as to indicators-focuses on sharedand nonsharedvariance
provide an estimate of the validity of the second among indicatorsthat the scholar is evaluating.This
indicator. In small-N, qualitative analysis, the re- approachuses empiricalevidence to supplementand
searcherhas the resourceof detailed case knowledge temper content validation.Nomological/construct val-
to help evaluate this second indicator. Thus, both idation-taking as a baseline an established causal
qualitativeand quantitativeresearchershave a means hypothesis-adds a further tool that can tease out
for making inferences about whether this important additional facets of measurement validity not ad-
presuppositionof nomological validation is indeed dressedby convergent/discriminant validation.
met. We are convinced that it is useful to carefully
A third problemis that, in many domainsin which differentiatethese types. It helps to overcome the
politicalscientistswork,there may not be a sufficiently confusion derivingfrom the proliferationof distinct
well-establishedhypothesisto make this a viable ap- types of validation,and also of terms for these types.
proach to validation. In such domains, it may be Furthermore,in relationto methodssuch as structural
plausible to assume the measure and evaluate the equationmodels with latent variables-which provide
hypothesis,but not the otherwayaround.Nomological sophisticatedtools for simultaneouslyevaluatingboth
validationthereforesimplymaynot be viable.Yet, it is measurementvalidity and explanatoryhypotheses-
helpful to recognizethat nomologicalvalidationneed the delineationof typesservesas a usefulreminderthat
not be restrictedto a dichotomousunderstandingin validationis a multifacetedprocess. Even with these

543
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch
models, this processmust also incorporatethe careful
use of contentvalidation,as Bollen emphasizes.
Second,we have encouragedscholarsto distinguish
between issues of measurementvalidityand broader
conceptualdisputes.Buildingon the contrastbetween
the backgroundconcept and the systematizedconcept
(Figure 1), we have exploredhow validityissues and
conceptualissues can be separated.We believe that
this separationis essential if scholars are to give a
consistentfocus to the idea of measurementvalidity,
and particularlyto the practiceof content validation.
Third, we examined alternative procedures for
adaptingoperationalizationto specific contexts:con-
text-specificdomains of observation,context-specific
indicators, and adjusted common indicators. These
proceduresmake it easier to take a middle position
between universalizingand particularizingtendencies.
Yet, we also emphasize that the decision to pursue
context-specificapproachesshouldbe carefullyconsid-
ered and justified.
Fourth, we have presented an understandingof
measurementvalidationthat can plausiblybe applied
in both quantitativeand qualitativeresearch.Although
most discussionsof validation focus on quantitative
research,we have formulatedeach type in terms of
basic questions intended to clarify the relevance to
both quantitativeandqualitativeanalysis.We havealso
given examples of how these questions can be ad-
dressedby scholarsfromwithinboth traditions.These
examplesalso illustrate,however,that while they may
be addressingthe same questions, quantitativeand
qualitative scholars often employ different tools in
findinganswers.
Withinthis framework,qualitativeand quantitative
researcherscan learn from these differences.Qualita-
tive researcherscould benefit from self-consciously
applyingthe validationproceduresthat to some degree
they may alreadybe employingimplicitlyand, in par-
ticular, from developing and comparing alternative
indicatorsof a givensystematizedconcept.Theyshould
also recognize that nomological validation can be
importantin qualitativeresearch,as illustratedby the
Lijphartand Anderson examplesabove. Quantitative
researchers,in turn, could benefit from more fre-
quently supplementingother tools for validation by
employinga case-orientedapproach,using the close
examinationof specific cases to identify threats to
measurementvalidity.
REFERENCES
Alvarez,Michael,Jose Antonio Cheibub,FernandoLimongi,and
AdamPrzeworski.1996."Classifying
PoliticalRegimes."Studiesin
ComparativeInternationalDevelopment 31 (Summer): 3-36.
AmericanPsychologicalAssociation.1954. "TechnicalRecommen-
dationsfor PsychologicalTests and DiagnosticTechniques."Psy-
chological Bulletin 51 (2, Part 2): 201-38.
AmericanPsychologicalAssociation.1966.Standards
for Educational
and Psychological Tests and Manuals. Washington, DC: American
PsychologicalAssociation.
and
Anderson,BarbaraA., andBrianD. Silver.1986."Measurement
Mismeasurementof the Validity of the Self-ReportedVote."
American Journal of Political Science 30 (November): 771-85.
544
September 2001
I
Anderson, Perry. 1974. Lineages of the Absolutist State. London:
Verso.
Angoff,WilliamH. 1988."Validity:An EvolvingConcept."In Test
Validity,ed. HowardWainerand HenryI. Braun.Hillsdale,NJ:
LawrenceErlbaum.Pp. 19-32.
Babbie, Earl R. 2001. The Practice of Social Research, 9th ed.
Belmont,CA: Wadsworth.
Bachman,JeraldG., and PatrickM. O'Malley.1984. "Yea-Saying,
Nay-Saying,and Going to Extremes:Black-WhiteDifferencesin
ResponseStyles."PublicOpinionQuarterly
48 (Summer):491-509.
Banks, Arthur S. 1979. Cross-National Time-Series Data Archive
User'sManual. Binghamton:Center for Social Analysis, State
Universityof New York at Binghamton.
Baumgartner,FrankR., and Jack L. Walker.1990. "Responseto
Smith's'Trendsin VoluntaryGroupMembership:Commentson
Baumgartnerand Walker':MeasurementValidityand the Conti-
nuity of Research in Survey Research."AmericanJournalof
Political Science 34 (August): 662-70.
Berry, William D., Evan J. Ringquist,Richard C. Footing, and
Russell L. Hanson. 1998. "MeasuringCitizen and Government
Ideologyin the AmericanStates, 1960-93."AmericanJournalof
Political Science 42 (January): 327-48.
Bevir, Mark. 1999. The Logic of the History of Ideas. Cambridge:
CambridgeUniversityPress.
Bollen,KennethA. 1980."Issuesin the ComparativeMeasurement
of Political Democracy." American Sociological Review 45 (June):
370-90.
Bollen, Kenneth A. 1989. StructuralEquations with Latent Variables.
New York:Wiley.
Bollen, Kenneth A. 1990. "PoliticalDemocracy:Conceptualand
Measurement Traps." Studies in ComparativeInternationalDevel-
opment25 (Spring):7-24.
Bollen,KennethA. 1993."LiberalDemocracy:ValidityandMethod
Factors in Cross-National Measures."American Journal of Political
Science37 (November):1207-30.
Bollen, KennethA., BarbaraEntwisle,and Arthur S. Anderson.
1993."MacrocomparativeResearchMethods."AnnualReviewof
Sociology 19: 321-51.
Bollen, Kenneth A., and Richard Lennox. 1991. "Conventional
Wisdomon Measurement:A StructuralEquationPerspective."
Psychological Bulletin 110 (September): 305-14.
Bollen, Kenneth A., and Pamela Paxton. 1998. "Detection and
Determinantsof Biasin SubjectiveMeasures."
AmericanSociolog-
ical Review63 (June):465-78.
Bollen,KennethA., andPamelaPaxton.2000."SubjectiveMeasures
of Liberal Democracy." ComparativePolitical Studies 33 (Febru-
ary):58-86.
Brady,Henry.1985."ThePerilsof SurveyResearch:Inter-Personally
Incomparable 11 (3-4): 269-91.
Responses."PoliticalMethodology
Brady,Henry,andDavidCollier,eds. 2001.RethinkingSocialInquiry:
Diverse Tools, Shared Standards. Berkeley: Berkeley Public Policy
Press, Universityof California,and Boulder, CO: Roman &
Littlefield.
Research:A
Brewer,John, and Albert Hunter. 1989. Multimethod
Synthesis of Styles. Newbury Park, CA: Sage.
Cain,BruceE., andJohnFerejohn.1981."PartyIdentificationin the
United States and Great Britain." ComparativePolitical Studies 14
(April):31-47.
Campbell,DonaldT. 1960."Recommendations for APA Test Stan-
dards Regarding Construct,Trait, or DiscriminantValidity."
American Psychologist 15 (August): 546-53.
Campbell,DonaldT. 1977/1988."DescriptiveEpistemology:Psycho-
logical,Sociological,and Evolutionary." and Episte-
Methodology
mology for Social Science: Selected Papers. Chicago: University of
ChicagoPress.
Campbell,DonaldT., and DonaldW. Fiske. 1959."Convergentand
Matrix."
DiscriminantValidationby the Multitrait-Multimethod
Psychological Bulletin 56 (March): 81-105.
Carmines,EdwardG., and RichardA. Zeller. 1979.Reliabilityand
ValidityAssessment. Beverly Hills, CA: Sage.
Clausen, Aage. 1968. "Response Validity:Vote Report."Public
Opinion Quarterly41 (Winter): 588-606.
in the
of a Concept:'Corporatism'
Collier,David.1995."Trajectory
Studyof LatinAmericanPolitics."In LatinAmericain Compara-
tive Perspective:Issues and Methods, ed. Peter H. Smith. Boulder,
CO:Westview.Pp. 135-62.
American PoliticalScience
American Political Science Review
Review
Collier,David,and RobertAdcock.1999."Democracyand Dichot-
omies: A PragmaticApproachto Choices about Concepts."An-
nualReviewof PoliticalScience2: 537-65.
Collier,David,and StevenLevitsky.1997."Democracywith Adjec-
tives: ConceptualInnovationin ComparativeResearch."World
Politics49 (April):430-51.
Collier,David,andJamesE. Mahon,Jr. 1993."Conceptual'Stretch-
ing' Revisited:Adapting Categoriesin ComparativeAnalysis."
AmericanPoliticalScienceReview87 (December):845-55.
Collier, Ruth Berins. 1999. Paths TowardDemocracy.Cambridge:
CambridgeUniversityPress.
Cook,ThomasD., andDonaldT. Campbell.1979.Quasi-Experimen-
tation.Boston:HoughtonMifflin.
Coppedge,Michael.1997."Howa LargeN CouldComplementthe
Small in DemocratizationResearch."Paper presented at the
annual meeting of the AmericanPolitical Science Association,
Washington,DC.
Coppedge,Michael,and WolfgangH. Reinicke.1990. "Measuring
Polyarchy."Studiesin Comparative International
Development25
(Spring):51-72.
Cronbach,Lee J., and Paul E. Meehl. 1955. "ConstructValidityin
PsychologicalTests."Psychological Bulletin52 (July):281-302.
Dahl, Robert A. 1956.A Prefaceto DemocraticTheory.Chicago:
Universityof ChicagoPress.
Elkins,Zachary.2000. "Gradationsof Democracy:EmpiricalTests
of AlternativeConceptualizations." AmericanJournalof Political
Science44 (April):293-300.
Elster, Jon. 1999.Alchemiesof the Mind. New York: Cambridge
UniversityPress.
Fearon,James,and DavidD. Laitin.2000."Ordinary Languageand
ExternalValidity:SpecifyingConceptsin the Studyof Ethnicity."
Paperpresentedat the annualmeetingof the AmericanPolitical
ScienceAssociation,Washington,DC.
Fischer,Markus.1992."FeudalEurope,800-1300:CommunalDis-
course and ConflictualPolitics."InternationalOrganization46
(Spring):427-66.
Fischer,Markus.1993. "On Context,Facts, and Norms."Interna-
tionalOrganization 47 (Summer):493-500.
Freeden,Michael.1996.IdeologiesandPoliticalTheory: A Conceptual
Approach.Oxford:OxfordUniversityPress.
Gallie,W. B. 1956."EssentiallyContestedConcepts."Proceedings of
theAristotelianSociety51: 167-98.
Gastil,RaymondD. 1988.Freedomin the World:PoliticalRightsand
CivilLiberties,1987-1988.New York:FreedomHouse.
George,AlexanderL., and AndrewBennett.N.d. CaseStudiesand
TheoryDevelopment.Cambridge,MA: MIT Press.Forthcoming.
Gerring,John. 1997. "Ideology:A DefinitionalAnalysis."Political
ResearchQuarterly 50 (December):957-94.
Gerring,John. 1999. "WhatMakes a Concept Good? A Criterial
Frameworkfor UnderstandingConceptFormationin the Social
Sciences."Polity31 (Spring):357-93.
Gerring,John. 2001. PracticalKnowledge: A CriterialApproachto
Social Science Methodology.New York: CambridgeUniversity
Press.
Gould, Andrew C. 1999. "ConflictingImperativesand Concept
Formation."Reviewof Politics61 (Summer):439-63.
Green,DonaldPhilip.1991."TheEffectsof MeasurementErroron
Two-StageLeast-SquaresEstimates."In PoliticalAnalysis,vol. 2,
ed. JamesA. Stimson.Ann Arbor:Universityof MichiganPress.
Pp: 57-74.
Green,DonaldPhilip,andBradleyL. Palmquist.1990."OfArtifacts
and PartisanInstability."
AmericanJournalof PoliticalScience34
(August):872-902.
Greenleaf, Eric A. 1992. "MeasuringExtreme Response Style."
PublicOpinionQuarterly 56 (Autumn):382-51.
Guion, Robert M. 1980. "On TrinitarianDoctrines of Validity."
ProfessionalPsychology11 (June):385-98.
Harding,Timothy,and JamesPetras.1988. "Introduction: Democ-
ratizationand the ClassStruggle."LatinAmericanPerspectives 15
(Summer):3-17.
Hayduk,LeslieA. 1987.Structural EquationModelingwithLISREL.
Baltimore,MD: JohnsHopkinsUniversityPress.
Hayduk,Leslie A. 1996. LISREL:Issues,Debates,and Strategies.
Baltimore,MD: JohnsHopkinsUniversityPress.
Hill, Kim Q., Stephen Hanna, and Sahar Shafqat. 1997. "The
Vol. 95, No.
Vol. No. 3
Liberal-Conservative Ideologyof U.S. Senators:A New Measure."
AmericanJournalof PoliticalScience41 (October):1395-413.
Huntington,SamuelP. 1991.TheThirdWave:Democratization in the
TwentiethCentury.Norman:Universityof OklahomaPress.
Jacoby,William G. 1991. Data Theoryand DimensionalAnalysis.
NewburyPark,CA: Sage.
Jacoby,William G. 1999. "Levels of Measurementand Political
Research:An OptimisticView."AmericanJournalof Political
Science43 (January):271-301.
Johnson,Ollie A. 1999."PluralistAuthoritarianism in Comparative
Perspective:White Supremacy,Male Supremacy,and Regime
Classification."NationalPoliticalScienceReview7: 116-36.
Kaplan,Abraham.1964. The Conductof Inquiry.San Francisco:
Chandler.
Karl, Terry Lynn. 1990. "Dilemmasof Democratizationin Latin
America."Comparative Politics22 (October):1-21.
Katosh, John P., and Michael W. Traugott. 1980. "The Conse-
quencesof ValidatedandSelf-ReportedVotingMeasures."Public
OpinionQuarterly 45 (Winter):519-35.
King,Gary,RobertO. Keohane,and SidneyVerba.1994.Designing
SocialInquiry:ScientificInferencein QualitativeResearch.Prince-
ton, NJ: PrincetonUniversityPress.
Kirk,Jerome,and MarcL. Miller. 1986.Reliabilityand Validityin
Qualitative Research.BeverlyHills, CA: Sage.
Kurtz,MarcusJ. 2000. "Understanding PeasantRevolution:From
Conceptto Theoryand Case."Theoryand Society29 (February):
93-124.
Laitin,DavidD. 2000."WhatIs a LanguageCommunity?" American
Journalof PoliticalScience44 (January):142-55.
Levitsky,Steven. 1998. "Institutionalizationand Peronism:The
Concept, the Case and the Case for Unpackingthe Concept."
PartyPolitics4 (1): 77-92.
Lijphart,Arend. 1984. Democracies:Patternsof Majoritarianand
ConsensusGovernment in Twenty-One Countries.New Haven,CT:
Yale UniversityPress.
Lijphart,Arend.1996."ThePuzzleof IndianDemocracy:A Conso-
ciational Interpretation." AmericanPolitical Science Review90
(June):258-68.
Linz, Juan J. 1975. "Totalitarianand AuthoritarianRegimes."In
Handbookof PoliticalScience,vol. 3, ed. Fred I. Greensteinand
Nelson W. Polsby.Reading,MA:Addison-Wesley.Pp. 175-411.
Lipset,SeymourM. 1959. "SomeSocial Requisitesof Democracy:
EconomicDevelopmentand PoliticalLegitimacy." AmericanPo-
liticalScienceReview53 (March):69-105.
Locke, Richard,and KathleenThelen. 1995."Applesand Oranges
Revisited:ContextualizedComparisonsandthe Studyof Compar-
ativeLaborPolitics."Politicsand Society23 (September):337-67.
Locke, Richard,and KathleenThelen. 1998."Problemsof Equiva-
lence in ComparativePolitics: Apples and Oranges, Again."
Newsletter of the APSAOrganizedSectionin ComparativePolitics
9 (Winter):9-12.
Loveman, Brian. 1994. "'Protected Democracies' and Military
Guardianship: PoliticalTransitionsin LatinAmerica,1979-1993."
Journalof Interamerican Studiesand WorldAffairs36 (Summer):
105-89.
Mainwaring,Scott, Daniel Brinks,and Anibal Perez-Lifinn.2001.
"ClassifyingPolitical Regimes in Latin America, 1945-1999."
Studiesin Comparative International Development36 (1): 37-64.
Markoff,John.1996.Wavesof Democracy.ThousandOaks,CA:Pine
Forge.
Messick,Samuel.1980."TestValidityandthe Ethicsof Assessment."
AmericanPsychologist35 (November):1012-27.
Messick,Samuel.1989."Validity."In EducationalMeasurement, ed.
RobertL. Linn.New York:Macmillan.Pp. 13-103.
Moene,KarlOve,andMichaelWallerstein.2000."Inequality,Social
Insuranceand Redistribution."WorkingPaper No. 144, Juan
MarchInstitute,Madrid,Spain.
Moss, PamelaA. 1992."ShiftingConceptionsof Validityin Educa-
tional Measurement:Implicationsfor PerformanceAssessment."
Reviewof EducationalResearch62 (Fall):229-58.
Moss,PamelaA. 1995."ThemesandVariationsin ValidityTheory."
EducationalMeasurement: IssuesandPractice14 (Summer):5-13.
Mouffe,Chantal.1992.Dimensionsof RadicalDemocracy.London:
Verso.
Nie, NormanH., G. BinghamPowell,Jr.,andKennethPrewitt.1969.
545
MeasurementValidity:A SharedStandardfor Qualitativeand QuantitativeResearch
"SocialStructure,and PoliticalParticipation:
DevelopmentalRe-
lationships, Part I." American Political Science Review 63 (June):
361-78.
Niemi, Richard,RichardKatz,and David Newman.1980."Recon-
The Failureof the PartyIdentification
structingPastPartisanship:
Recall Questions." American Journal of Political Science 24 (No-
vember): 633-51.
O'Donnell,Guillermo.1993. "On the State, Democratizationand
SomeConceptualProblems."WorldDevelopment 27 (8): 1355-69.
O'Donnell,Guillermo.1996."Illusionsabout Consolidation."Jour-
nal of Democracy 7 (April): 34-51.
Orum, Anthony M., Joe R. Feagin, and Gideon Sjoberg. 1991.
The Natureof the Case Study."In A Casefor the
"Introduction:
Case Study,ed. Joe R. Feagin,AnthonyM. Orum,and Gideon
Sjoberg.ChapelHill:Universityof NorthCarolinaPress.Pp. 1-26.
Paxton,Pamela.2000."Womenin the Measurementof Democracy:
Problems of Operationalization." Studies in ComparativeIntera-
tional Development 35 (Fall): 92-111.
Pitkin, Hanna Fenichel. 1967. The Concept of Representation.Berke-
ley: Universityof CaliforniaPress.
Pitkin,HannaFenichel.1987."RethinkingReification."Theoryand
Society16 (March):263-93.
Przeworski,Adam, Michael Alvarez,Jose Antonio Cheibub,and
FernandoLimongi.1996. "WhatMakes DemocraciesEndure?"
Journal of Democracy 7 (January): 39-55.
Przeworski,Adam, and HenryTeune. 1970. Logic of Comparative
Social Inquiry. New York: John Wiley.
Rabkin, Rhoda. 1992. "The Aylwin Governmentand 'Tutelary'
Democracy:A Concept in Searchof a Case?"Journalof Inter-
american Studies and WorldAffairs 34 (Winter): 119-94.
Ragin,CharlesC. 1987.TheComparative
Method.Berkeley:Univer-
sity of CaliforniaPress.
Ragin, Charles C. 1994. Constructing Social Research. Thousand
Oaks,CA: Pine Forge.
Reus-Smit,Christian.1997."TheConstitutionalStructureof Inter-
national Society and the Nature of FundamentalInstitutions."
International Organization51 (Autumn): 555-89.
Ruggie, John G. 1998. "WhatMakes the World Hang Together?
Neo-Utilitarianismand the Social ConstructivistChallenge."In-
ternational Organization52 (Autumn): 855-85.
Russett, Bruce. 1993. Graspingthe Democratic Peace. Princeton, NJ:
PrincetonUniversityPress.
Sartori,Giovanni. 1970. "ConceptMisformationin Comparative
Research." American Political Science Review 64 (December):
1033-53.
Sartori,Giovanni.1975."TheTowerof Babel."In Towerof Babel,
ed. GiovanniSartori,Fred W. Riggs,and HenryTeune. Interna-
tional StudiesAssociation,Universityof Pittsburgh.Pp. 7-37.
Sartori, Giovanni, ed. 1984. Social Science Concepts: A Systematic
Analysis.BeverlyHills, CA: Sage.
Sartori,Giovanni,FredW. Riggs,and HenryTeune. 1975.Towerof
Babel: On the Definition and Analysis of Concepts in the Social
Sciences.InternationalStudies Association,Universityof Pitts-
burgh.
Schaffer, Frederic Charles. 1998. Democracy in Translation: Under-
standing Politics in an Unfamiliar Culture. Ithaca, NY: Cornell
UniversityPress.
546
September 2001
Schmitter,PhilippeC., and TerryLynnKarl. 1992. "The Types of
DemocracyEmergingin SouthernandEasternEuropeand South
and Central America." In Bound to Change: ConsolidatingDemoc-
racy in East Central Europe, ed. Peter M. E. Volten. New York:
Institutefor EastWestStudies.Pp. 42-68.
Schrodt,PhilipA., and DeborahJ. Gerner.1994."ValidityAssess-
ment of a Machine-CodedEvent Data Set for the MiddleEast,
1982-92." American Journal of Political Science 38 (August):
825-54.
Schumpeter, Joseph. 1947. Capitalism, Socialism and Democracy.
New York:Harper.
Shepard,Lorrie.1993."Evaluating
TestValidity."Reviewof Research
in Education 19: 405-50.
Shively, W. Phillips. 1998. The Craft of Political Research. Upper
SaddleRiver,NJ: PrenticeHall.
Shultz,KennethS., MattL. Riggs,and JanetL. Kottke.1998."The
Need for an Evolving Concept of Validity in Industrialand
PersonnelPsychology:Psychometric,
Legal,andEmergingIssues."
CurrentPsychology 17 (Winter): 265-86.
Sicinski,Andrzej.1970. "'Don'tKnow'Answersin Cross-National
Surveys." Public Opinion Quarterly34 (Spring): 126-9.
Sireci,StephenG. 1998."TheConstructof ContentValidity."Social
Indicators Research 45 (November): 83-117.
Skocpol, Theda. 1992. Protecting Soldiers and Mothers. Cambridge,
MA: HarvardUniversityPress.
Sussman,LeonardR. 1982."TheContinuingStrugglefor Freedom
of Information."
In Freedomin the World,ed. RaymondD. Gastil.
Westport,CT: Greenwood.Pp. 101-19.
Valenzuela,J. Samuel. 1992. "DemocraticConsolidationin Post-
TransitionalSettings:Notion, Process, and FacilitatingCondi-
tions." In Issues in Democratic Consolidation, ed. Scott Mainwar-
ing, Guillermo O'Donnell, and J. Samuel Valenzuela. Notre
Dame, IN: Universityof Notre Dame Press.Pp. 57-104.
Vanhanen, Tatu. 1979. Power and the Means to Power. Ann Arbor,
MI: UniversityMicrofilmsInternational.
Vanhanen, Tatu. 1990. The Process of Democratization. New York:
CraneRussak.
Verba,Sidney.1967."SomeDilemmasin ComparativeResearch."
WorldPolitics 20 (October): 111-27.
Verba,Sidney.1971."Cross-National
SurveyResearch:The Problem
of Credibility." In Comparative Methods in Sociology, ed. Ivan
Vallier.Berkeley:Universityof CaliforniaPress.Pp. 309-56.
Verba,Sidney,StevenKelman,GaryR. Orren,IchiroMiyake,Joji
Watanuki,IkuoKabashima,andG. DonaldFerree,Jr. 1987.Elites
and the Idea of Equality. Cambridge, MA: Harvard University
Press.
Verba, Sidney,NormanNie, and Jae-On Kim. 1978.Participation
and PoliticalEquality.Cambridge:CambridgeUniversityPress.
Webb,Eugene J., Donald T. Campbell,RichardD. Schwartz,and
Lee Sechrest. 1966. UnobtrusiveMeasures:NonreactiveResearch in
the Social Sciences. Chicago: Rand McNally.
Zeitsch,John,Denis Lawrence,andJohnSalernian.1994."Compar-
ing Like with Like in ProductivityStudies:Apples, Orangesand
EconomicRecord70 (June):162-70.
Electricity."
Zeller,RichardA., and EdwardG. Carmines.1980.Measurement in
the Social Sciences: The Link between Theoryand Data. Cambridge:
CambridgeUniversityPress.