This article was downloaded by: [Universita di Padova]

On: 23 October 2013, At: 06:36

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,
UK

Scientific Studies of Reading

Publication details, including instructions for
authors and subscription information:
http://www.tandfonline.com/loi/hssr20

Measuring Reading
Comprehension
Jack M. Fletcher
Published online: 19 Nov 2009.

To cite this article: Jack M. Fletcher (2006) Measuring Reading Comprehension,

Scientific Studies of Reading, 10:3, 323-330, DOI: 10.1207/s1532799xssr1003_7
To link to this article: http://dx.doi.org/10.1207/s1532799xssr1003_7

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the
information (the Content) contained in the publications on our platform.
However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness,
or suitability for any purpose of the Content. Any opinions and views
expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the
Content should not be relied upon and should be independently verified with
primary sources of information. Taylor and Francis shall not be liable for any
losses, actions, claims, proceedings, demands, costs, expenses, damages,
and other liabilities whatsoever or howsoever caused arising directly or
indirectly in connection with, in relation to or arising out of the use of the
Content.
This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan,
sub-licensing, systematic supply, or distribution in any form to anyone is
expressly forbidden. Terms & Conditions of access and use can be found at
http://www.tandfonline.com/page/terms-and-conditions

SCIENTIFIC STUDIES OF READING, 10(3), 323330

Copyright 2006, Lawrence Erlbaum Associates, Inc.

Measuring Reading Comprehension

Downloaded by [Universita di Padova] at 06:36 23 October 2013

Jack M. Fletcher
Department of Psychology
University of Houston

The five articles in this special issue are a blend of experimental and correlational
approaches that exemplify advances in contemporary approaches to assessment of
reading comprehension. They illustrate how inferences about reading comprehension are determined in part by the material presented for comprehending and the format that is used for assessing comprehension of the material. In the future, the approaches to measuring reading comprehension in these articles could be further
integrated by perspectives that cut across particular approaches to measurement and
begin to utilize multimethod modeling of the latent variables that underlie different
tests of reading comprehension.

How is reading comprehension best measured? The five articles in this special issue each address a set of problems that affect how reading comprehension is measured, utilizing an interesting blend of experimental and correlational methodologies. The clear consensus across these articles is that the measurement issues are
complicated, reflecting the complex, multidimensional nature of reading comprehension. Historically, education and psychology researchers have identified multiple approaches to measurement, with different decades characterized by an emphasis on different approaches to assessment. As Pearson and Hamm (2005)
summarized, early research identified that reading comprehension involved multiple components that would appear depending on the formats used to present the
material to be read and the manner in which the person was asked to indicate their
understanding of the material that was read. Despite this historical emphasis, many
modern approaches to the assessment of reading comprehension are one dimensional, with little variation in the material the person reads and relatively narrow
response formats that do not vary within the test. Thus, some tests rely almost exCorrespondence should be sent to Jack M. Fletcher, Department of Psychology, University of
Houston, 2151 West Holcombe Boulevard, 222 TMC Annex, Houston, TX 772045053. E-mail:
JackFletcher@uh.edu

Downloaded by [Universita di Padova] at 06:36 23 October 2013

324

FLETCHER

clusively on multiple choice, others on fill-in-the-blank (cloze), and others on retells. The drive for high reliability, especially on high-stakes assessments, often
leads to significant restrictions of both the type of material that must be read and
the response formats. Yet, as the article by Cutting and Scarborough (2006/this issue) clearly demonstrates, the inferences that are made about how well an individual person comprehends written material vary depending on how it is assessed.
Inferring how well a person comprehends is the real problem in measuring
reading (and language) comprehension (RAND Reading Study Group, 2002). In
contrast to word recognition, where the behavior of interest is fairly overt and different methods give similar outcomes at the latent variable level (accounting for
method variance and error of measurement), the assessment of reading comprehension is difficult because it is not an overt process that can be directly observed.
Rather, only the products of the process of comprehending are observed, and an inference is made about the nature of the processes and the quality of the comprehension. As the article by Francis et al. (2006/this issue) demonstrates, any single,
one-dimensional attempt to assess reading comprehension is inherently imperfect.
From the psychometric view of Francis et al., differences across methods used to
measure reading comprehension can be interpreted as degrees of imperfection in
how well different indicators identify one or more latent variables that make up
reading comprehension. In this commentary I examine the five articles from this
admittedly psychometric perspective.
This perspective does help place the five articles in this issue in perspective. Each
explicitly focuses on issues involving measurement, specifically, what is read and
how comprehension of what is read is measured. Three articles are essentially experimental in their origins and address both of these determinants of reading comprehension in different ways. Millis, Magliano, and Todaro (2006/this issue) focused on
the comprehension of expository text and asked for responses that represented
think-alouds. These responses were then analyzed at the level of discourse using latent semantic analysis (LSA). The validity of this approach was evaluated by correlating the products of LSA with a passage taken from NelsonDenny reading comprehension test (Brown, Fischo, & Hanna, 1993). Questions were developed to
represent literal (text-based) and inferential understanding of the passages. They
found good convergence between the LSA methods of assessing reading comprehension and the indices that are obtained in the NelsonDenny.
Although the difficulty level of the passages was controlled in Millis et al.s
(2006/this issue) article, Deane, Sheehan, Sabatini, Futagi, and Kostin (2006/this
issue) explicitly demonstrated that variations in the lexical and discourse elements
of different passages accounted for variations in how comprehension was tested at
different grade levels. They measured text-based characteristics of lexiled Grade 3
and 6 expository and narrative books. Based in part on LSA, they identified a variety of text characteristics that were factor analyzed to identify different dimensions
of text variability. Comparisons of the factor structure of Grade 3 and 6 texts

Downloaded by [Universita di Padova] at 06:36 23 October 2013

MEASURING READING COMPREHENSION

325

showed differences in a variety of indices of complexity that accounted for differences in readability and undoubtedly would explain differences in levels of comprehension depending on the level of reading ability.
The third experimental article (Rayner, Chace, Ashby, & Slattery, 2006/this issue) examined the role of eye movements as indicators of comprehension processes. In a series of studies, the authors monitored eye movements and showed
that variations in eye movements reflect the difficulty of the passage as well as difficulties dealing with inconsistencies in the text. When the text is more difficult,
comprehension is inconsistent, and there is a much higher probability of a regressive eye movement. The nature of regressive eye movements when reading text is
an indicator of the operation of different reading comprehension processes. Thus,
the presence of long regression usually indicates that the reader has found a part of
the text in which his or her understanding is not adequate, leading him or her to reread earlier sections of the passage.
The two correlational studies explicitly show that the inferences about reading
comprehension depend on how reading comprehension is measured. Cutting and
Scarborough (2006/this issue) addressed this issue explicitly by examining the
contributions of different measures of word decoding and oral language ability to
three different reading comprehension tests: the Wechsler Individual Achievement
Test (WIAT; Wechsler, 1992), the GatesMacGinitie (MacGinitie, Maria, &
Dryer, 2000), and the Gray Oral Reading TestThird Edition (Wiederholt &
Bryant, 1992). These three tests vary in terms of the material the person reads and
the format used to indicate his or her understanding. The WIAT presents short passages of 2 to 3 sentences that are read silently. The participant then answers
open-ended questions about each passage. One question is literal, and the other is
inferential, and a short-answer approach is used. In contrast, the GatesMacGinitie
involves passages up to 15 sentences that are also read silently. The response format is multiple choice. Finally, the Gray Oral Reading TestThird Edition involves passages of about 6 to 7 sentences that are read aloud. The response format
involves multiple-choice questions. In contrast to the GatesMacGinitie and
WIAT, the participant must respond with the passage out of view. Not surprisingly,
Cutting and Scarborough found that the pattern of correlations of these different
assessments to different measures of decoding and oral language skills varies and
is not constant.
Francis et al. (2006/this issue) explored similar issues but used a latent variable
approach to test specific models about the relation of decoding, language ability,
and comprehension. They also found that the relation of decoding and language
abilities depends on how comprehension is assessed. Two standardized tests of
reading comprehension were used: Passage Comprehension from the WoodcockJohnsonIII (Woodcock, McGrew, & Mather, 2001) and the Diagnostic Assessment of Reading Comprehension (DARC). The Passage Comprehension
subtest is a widely used measure that involves reading sentences or short passages,

Downloaded by [Universita di Padova] at 06:36 23 October 2013

326

FLETCHER

with a cloze procedure as a response format. The DARC is an experimental test

that involves reading passages of three sentences, but it is specifically designed to
control for the level of decoding that is required. The passages have been carefully
constructed to manipulate components that involve text memory and inferencing
as well as knowledge access and integration. They also measure comprehension
using linguistic discourse methods. Not surprisingly, the decoding and phonological awareness measures are more strongly related to the WoodcockJohnsonIII
Passage Comprehension test, and the contribution to these types of measures are
not strongly related to the DARC.
These five articles are all noteworthy for the different components of reading
comprehension that are assessed. In the remainder of this commentary, I highlight
three issues that seem to be particularly important for the measurement of reading
comprehension: nature of the text, how reading comprehension is assessed, and individual differences.

NATURE OF THE TEXT

All five of these articles demonstrate that the material the participant is asked to
read is a major determinant of the inference that is made about the quality of comprehension. Clearly, different inferences will be made depending not only on the
difficulty of the text but also on the semantic, syntactic, and related characteristics
of the text. This is most clearly demonstrated in Deane et al.s (2006/this issue) article but is also inherent in the eye movement study by Rayner et al. (2006/this issue) and the explicit effort to minimize the role of certain text characteristics in
Francis et al. (2006/this issue). Deane et al.s findings are particularly interesting,
showing that variations in text characteristics are related to standard ways of assessing the readability of the passages. This research is consistent with other studies of text characteristics, most notably by Hiebert (2002) and Foorman, Francis,
Davidson, Harm, and Griffin (2005). Foorman et al. developed a computer program for identifying lexical, semantic, and syntactic features of text in six different
commercial basal reading programs. They found considerable variability not only
in the composition of the text but also in the frequency of specific words taught in
the program. Such studies illustrate the importance of understanding text variability as a determinant of the inferences made about reading comprehension. Certainly, having participants read text that is relatively restricted in composition limits the inferences that can be made. Examinations of test characteristics should be
linked with methods derived from discourse analysis, which forms the basis for the
approach of Deane et al. to analyzing text characteristics. Millis et al. (2006/this issue) nicely demonstrate the value of LSA not only for understanding the nature of
the passage but also for linking comprehension of the passage to the text.

MEASURING READING COMPREHENSION

327

Downloaded by [Universita di Padova] at 06:36 23 October 2013

RESPONSE FORMATS
All of the articles in this series demonstrate the importance of considering the format by which reading comprehension is assessed. This is most explicitly addressed
by Cutting and Scarborough (2006/this issue), who show differential discriminant
validity for a set of predictive variables as they predicted reading comprehension
outcomes. Similarly, the latent variable analyses of Francis et al. (2006/this issue)
also show differential forms of discriminant validity. The assessments of reading
comprehension in these two studies included standardized tests but expanded to
experimental and discourse-level procedures. Francis et al. and Rayner et al.
(2006/this issue) utilized a method that is independent of the participants actual
response to assess reading comprehension, showing that regressive eye movements are good indicators of difficulties in comprehension and that these difficulties are related to the complexity of the text. The use of LSA for analyzing
think-alouds in Millis et al.s (2006/this issue) article seems particularly promising. It would be interesting to know more about how think-alouds are correlated
with performance on standardized tests of reading comprehension like those used
by Cutting and Scarborough. The use of the passages in the Nelson Denny is interesting, but the response format was somewhat modified. Across studies, the
finding of positive correlations across different measures of reading comprehension that varied in the format by which comprehension was measured was interesting. Clearly, there are one or more latent variables that represent the different components of reading comprehension that are imperfectly indexed by individual
measures, but these indices are correlated. Method bias, measurement error, and
the operation of general factors must be addressed in understanding these correlations, but these components can be modeled.

INDIVIDUAL DIFFERENCES
Virtually all of the articles mentioned the need for diagnostic tests of reading comprehension. A diagnostic test of reading comprehension would provide information on the strengths and weaknesses of individual people and their ability to comprehend reading material. Such assessments may suggest interventions at the
classroom or individual level that could be applied to enhance reading comprehension. Much is known about teaching reading comprehension, so knowing more
about how to link specific forms of instruction with the needs of individual readers
would be helpful.
Interestingly, the samples in each of these studies tended to be relatively restricted in terms of the range of individual differences, with the possible exceptions
of the samples examined by Cutting and Scarborough (2006/this issue) and Francis et al. (2006/this issue). The expansion of these methods into more diverse popu-

Downloaded by [Universita di Padova] at 06:36 23 October 2013

328

FLETCHER

lations would be interesting, but larger samples would be needed. In the end, it is
likely that what one would conclude about diagnostic tests of reading comprehension would depend on how reading comprehension is assessed. The notion that individual differences can be understood with a single procedure is probably questionable. Adequate assessments of reading comprehension will have to rely on
multiple procedures that manipulate both the material that is read and the response
format. It is likely that the assessment will need to be expanded beyond just what
might happen in a psychoeducational assessment or a group-based high-stakes assessment and take into account comprehension as it actually occurs in the classroom. Far too much may have been made of the idea that reading comprehension is
an interaction among the reader and the text and occurs in a situated model, but
these are nonetheless relevant considerations when it comes to understanding individual differences. It may be useful to incorporate observational techniques that involve systematic querying of teachers about the quality of a particular students
reading comprehension.
This need highlights the importance of beginning to integrate research across
studies of the sort observed in this special issue and to begin to approach reading
comprehension from a multimethod perspective in which a full range of underlying latent variables can be assessed and analytic methods like those used in Francis
et al. (2006/this issue) are applied to understand the relations of the constructs at
the level of the latent variable and not simply at the level of the observed variables.

CONCLUSIONS
If the goal is to develop diagnostic tests, which was mentioned by the authors of each
of the five articles, then approaches to the assessment of reading comprehension
need to incorporate multiple indicators to enhance the precision in which the underlying latent variables are measured. Otherwise, the results may have what the construct validity world has termed a mono-operation bias (Cook & Campbell, 1979).
Although some argue that issues in assessing reading comprehension are so complex
that no psychometric approach can ever be adequate, the real issue is attempting to
model the complexity through experimental and correlational techniques. We need
multimethod research that moves beyond the unidimensional approaches that characterize most contemporary approaches to measurement and looks at relations at the
latent variable level. Cook and Campbell identified construct underrepresentation,
in which a single variable does not adequately index the underlying constructs, as a
major factor limiting inferences about complex human behaviors. They noted that
the fundamental problem in construct validity research was that the operations
which are meant to represent a particular cause or effect construct can be construed in
terms of more than one construct (p. 59) or is underidentified because of mono-operation bias. Research that isolates specific factors in reading comprehension is im-

MEASURING READING COMPREHENSION

329

Downloaded by [Universita di Padova] at 06:36 23 October 2013

portant. What is also needed is research that integrates across methods and specifically attempts to identify different constructs that are specific to reading
comprehension and their relation to other constructs that make up reading and other
cognitive skills. Such approaches will move beyond mono-operation bias and lead to
assessments that capture the richness of reading comprehension. Tests based on such
analyses not only will permit better inferences about reading comprehension but
also will be diagnostic, because variability within the test will be apparent for some
readers. This variability may be tied to differential instruction, which is the ultimate
purpose of assessing reading comprehension.

ACKNOWLEDGMENTS
This research was supported in part by a grant from the National Institute of Child
Health and Human Development, HD052117, Texas Center for Learning
Disabilities. I gratefully acknowledge contributions of Rita Taylor to manuscript
preparation.

REFERENCES
Brown, J. I., Fischo, V. V., & Hanna, G. S. (1993). Nelson-Denny Reading Test. Chicago: Riverside.
Cook, T. D., & Campbell, D. A. (1979). Quasi-experimental design: Design and analysis issues. Chicago: Rand McNally.
Cutting, L. E., & Scarborough, H. S. (2006). Prediction of reading comprehension: Relative contributions of word recognition, language proficiency, and other cognitive skills can depend on how comprehension is measured. Scientific Studies of Reading, 10, 277299.
Deane, P., Sheehan, K. M., Sabatini, J., Futagi, Y., & Kostin, I. (2006). Differences in text structure and
its implications for assessment of struggling readers. Scientific Studies of Reading, 10, 257275.
Foorman, B. R., Francis, D. J., Davidson, K. C., Harm, M. W., & Griffin, J. (2004). Variability in text
features in six grade 1 basal reading programs. Scientific Studies of Reading, 8, 167197.
Francis, D. J., Snow, C. E., August, D., Carlson, C. D., Miller, J., & Iglesias, A. (2006). Measures of
reading comprehension: A latent variable analysis of the diagnostic assessment of reading comprehension. Scientific Studies of Reading, 10, 301322.
Hiebert, E. H. (2002). Standards, assessments, and text difficulty. In A. E. Farstrup & S. J. Samuels
(Eds.), What research has to say about reading instruction (pp. 337391). Newark, DE: International
Reading Association.
Millis, K., Magliano, J., & Todaro, S. (2006). Measuring discourse-level processes with verbal protocols and latent semantic analysis. Scientific Studies of Reading, 10, 225240.
Pearson, P. D., & Hamm, D. N. (2005). The assessment of reading comprehension: A review of practicesPast, present, and future. In S. G. Paris & S. A. Stahl (Eds.), Childrens reading comprehension and assessment (pp. 1369). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
RAND Reading Study Group. (2002). Reading for understanding. Washington, DC: RAND Education.
Rayner, K., Chace, K. H., Slattery, T. J., & Ashby, J. (2006). Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading, 10, 241255.

330

FLETCHER

Downloaded by [Universita di Padova] at 06:36 23 October 2013

Wechsler, D. L. (1992). Wechsler Individual Achievement Test. San Antonio, TX: Psychological Corporation.
Wiederholt, L., & Bryant, B. (1992). Gray Oral Reading Test3. Austin, TX: PRO-ED.
Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). WoodcockJohnson III Tests of Achievement.
Itasca, IL: Riverside.