Which Comprehension? Artifacts in the Measurement of Reading Comprehension Author(s): Janet Ross Kendall, Jana M.

Mason and William Hunter Source: The Journal of Educational Research, Vol. 73, No. 4 (Mar. - Apr., 1980), pp. 233-236 Published by: Taylor & Francis, Ltd. Stable URL: http://www.jstor.org/stable/27539756 . Accessed: 25/01/2014 23:35
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Taylor & Francis, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to The Journal of Educational Research.

http://www.jstor.org

This content downloaded from 148.206.53.9 on Sat, 25 Jan 2014 23:35:31 PM All use subject to JSTOR Terms and Conditions

Which

Comprehension?

Artifacts of Reading

in

the Measurement
JANETROSS KENDALL
Simon Fraser University

Comprehension

JANA M. MASON
University of Illinois, Urbana-Champaign

WILLIAM HUNTER
Mount St. Vincent University

ABSTRACT
prehension typically choice, scores employed

Differences
were obtained and to measure

in fifth graders' reading com

using four different tasks comprehension (multiple and four read different to effects and readability for pas passage. It or

suits based on one set of materials to others?committing the language-as-fixed-effect fallacy. Moreover, Gibson
and Levin (6) state, "... almost any measure raises ques

recall, maze) cloze, were that ing passages equated according Data formulas. revealed analyses significant and an interaction between task sage, task, concluded that the choice of

was

sion and passage testing procedure, not does allow practice, generalization of reading tional definitions comprehension. suggest serious limitations research and of most testing. comprehension

a particular whether to

comprehen in research other These

tions of validity, reliability, and generalizability to any other material or task" (p. 409). Some comprehension tasks rely on memory (recall), others rely on search (maze) or on organization of information (cloze). Thus tasks actually tap quite different strate comprehension
gies and processing mechanisms. Passages, too, can be con

opera results reading

contemporary

structed in quite different ways, for example, descriptive stories and myths. These differences are described more fully in Pennock (12). Further, because passages contain more or less familiar content to particular individuals, comprehension can be affected by the topic (14). There fore, it can be true for any investigation of reading com prehension that different conclusions might have been drawn if some other sets of tasks or passages had been supplied. The primary purpose of this study was to examine
some variations in reading comprehension scores that re

tasks have been used to measure readers' com for Severalprehension. For many years the multiple-choice mat has been incorporated in standardized achievement tests and in basal reading materials. Alternatives to the mul tiple choice test include recall, in which, after people read a passage, they are urged to describe everything they can remember about it [e.g., Bransford & Johnson (3)]. An other, cloze, is usually constructed by deleting every fifth or seventh word (2) and scored by counting as correct only the exact words that were deleted. Somewhat simi lar to the cloze ismaze in which every fifth or seventh word can be replaced by three choices, one of which is the correct word (7).
Often practitioners and researchers presume that writ

sult from different

binations.

task and passage measurement

com

Method The subjects were 164 fifth graders from four elemen tary schools located in a large eastern Canadian city. In order to assure comparability, participating schools were chosen only if the average of the raw scores for fifth Achievement Test (MAT) graders on theMetropolitan
Comprehension subtest corresponded to grade equiva

ten materials (passages) within a given study are similar or that scores from different measures of comprehension would rank individuals in the same order. That is, they
assume an equivalence of passage type or an equivalence

lents between
Multiple

5.2 and 5.7.

recall, cloze, and maze tasks were

choice,

of comprehension task type. These assumptions of equivalence are not universally accepted, however. For example, Clark (4) has argued that researchers must be cautious in generalizing from re

constructed for four reading passages from the McCall Crabbs Standard Test Lessons inReading (11). This set of reading materials was selected because it has been the criterion for readability formulas more often "than any other single criterion" (9:53), because the passages were

This content downloaded from 148.206.53.9 on Sat, 25 Jan 2014 23:35:31 PM All use subject to JSTOR Terms and Conditions

234
short and complete, companied
questions. Passages

OF EDUCATIONAL THE JOURNAL RESEARCH

"Whales" and "Paper," the two expository passages, were the more difficult. These observations are con firmed by the analysis of variance (Table 2). The inter action between tasks and passages, which was also signi ficant, shows that the difficulty of the "Whales" and "Paper" passages seemed to depend on the type of measurement task employed. "Paper" was the more diffi cult to comprehend with the cloze task, whereas "Whales'1 was the more difficult with the recall and maze tasks. A second set of analyses was carried out to measure the effects of reading ability on comprehension tasks and passages. Students were divided into three groups on the basis of their Canadian Tests of Basic Skills Comprehen sion subtest scores. The analyses showed expected main effects of reading ability, passages, and tasks; the inter actions between ability and passages and between ability and tasks were not significant. (See Tables 3 and 4.) Discussion tasks The finding that the maze and multiple-choice were consistently easier than recall and cloze is not sur prising (1, 2,13). Children generally find comprehension tasks easier when selecting from given answers (maze and than when remembering an answer (re multiple-choice)

and because each passage was ac to twelve normed, multiple-choice by eight

were chosen that show a grade-level

range of 5.2 to 5.6 when 70 percent of comprehension the questions are answered correctly. The readability (grade five, according to the Fry formula), passage length (127 to 142 words), and location of main idea (first sen tence) of the passages were equated, but the structure of the passages was deliberately varied. One passage, "Wood pecker," iswritten in a fairy tale form. "Ants" is descrip tive, though it also has a story form. The other two are expository: "Paper" contains a list-like explanation of the steps inmaking paper; "Whales" uses a causal structure to describe why people tag whales. The passages and tasks were organized into a 4 X 4 Latin Square design with students randomly assigned to one of four groups. Each student read the four pas sages, each one under a different comprehension condi tion. So that instructions and practice for each compre hension task could be given to the students as a group, the conditions were given in this order: (a) cloze, with every fifth word replaced by a blank; (b) McCall-Crabb multiple-choice comprehension questions; (c) free recall, with students asked to write down everything they could remember about the passage; and (d) maze, with every fifth word replaced by three alternatives. The students worked at their own pace with no time limits; the only constraint was that for the multiple choice and recall tasks they were not allowed to look back at the text. The number of correct responses was the dependent measure for three of the tasks. On the re call task, the number of idea units was measured technique based on work by Johnson (8). Results Inspection
that the maze

Table

2.-Analysis

of Variance,

Passage

and Task

Effects

Source
Between-subjects

SS

MS df

using a

Groups
Subjects Within-subjects within groups

.07
7.35

3
160

.02
.046

.435

of proportions
and multiple-choice

correct (Table 1) shows

tasks were consis

Passages

.79

.263

17.08*

Tasks
Passages Error X tasks

38.65
.38 7.4

3
.063 6 480

12.88

835.68*
4.11*

tently easier than recall and cloze, and that the fairy tale passage, "Woodpecker," was consistently easier than the others for all task types. "Ants," which is nearly a story form, was always second or third in difficulty, and

.0154

*p <

.001

Table 1.?Average Condition

Proportions

Correct

for Each

Table Task/Passage

3.-Analysis

of Variance,

Task

and Reading

Ability

Effects

Source
Tasks Average across tasks Between-subjects Cloze Choice .856 .951 .835 .816 Recall .276 .378 .334 .374 Maze .862 .938 .935 .906 .615 .705 .654 .631

SS

MS df

Ability
Error Within-subjects Tasks Tasks Error X ability

3.97
6.71

2
163

1.99
.04

48.23*

"Whales" "Woodpecker" "Ants" "Paper" across Average passages

.465 .554 .510 .426

38.10 .22 8.92 .04 489

3 6

12.70

695.92* 2.02

.02

.489

.864

.341

.910 *p < ,001

This content downloaded from 148.206.53.9 on Sat, 25 Jan 2014 23:35:31 PM All use subject to JSTOR Terms and Conditions

KENDALL-MASON-HUNTER

235

Table

4.-Analysis

of Variance,

Passage

and Reading

Ability

Effects

cations

of

this

conclusion

are warranted.

First,

researchers

Source
Between-subjects

SS

df

MS

Ability
Error Within-subjects Passages Passages Error X ability

3.97
6.71

2
163

1.99
.04

48.23*

should be cautious about presenting research that purports to label different passages as equal; critical analysis of materials is necessary as well as a willingness of researchers to make their materials readily available to others. Second, further research is needed to determine what other fac
tors, such as text cohesiveness, text structure, or topic

.75 .25 46.30

.25 3 .04 489 6 .09

2.66** .44

familiarity, affect passage difficulty and interact with measurement of reading comprehension. Third, teachers a variety of tasks and passage types to consider ought when instructing students in reading comprehension
since any one these type results of measure suggest may a need be too restrictive. varied com Fourth, for more

*p < **p <

.001 .05

prehension measures for testing and evaluating children. It is apparent that the difficulty found here in obtaining
comparable tasks and passages makes comprehension

call) or constructing an answer (cloze). Such differences in difficulty should be considered when planning a com
prehension test.

test interpretation

risky.

Also
ance

to be considered
significantly

is the type of passage. Perform

among the four passages, even

NOTE
are available from the first author Materials the Faculty of Education, Simon Fraser University, B.C., Canada. on request Burnaby, at

varied

though they were carefully selected and were controlled for readability. This suggests that seemingly comparable passages are very difficult to equate. As a result, until we
better understand what factors in text make passages more

or less difficult, practitioners and researchers must be test results cautious when interpreting comprehension on different based reading passages. The combinations of the various tasks and passages also produced systematically different results. Thus the problems associated with comparing studies using differ ent operational definitions may be even greater than pre viously suspected. While the fairy tale "Woodpecker" was
always the easiest passage, "Paper" was the hardest on two

REFERENCES
J. R. "Comparable Cloze and Multiple Choice Test Scores." Journal 10 (1967): Comprehension of Reading 291-99 J. R. "The Cloze Readability 2. Bormuth, Elemen Procedure." 1. Bormuth,

taryEnglish 45 (1968): 429-36.

3. Bransford, J. B., and Johnson, M. K. "Contextual Prerequi sites for Understanding: of Comprehen Some Investigations sion and Recall." Journal and Verbal of Verbal Learning Behavior 11 (1972): 717-26. 4. Clark, H. H. "The Language-as-Fixed-Effect A Fallacy: of Language in Psychological Statistics Research." Critique Journal 335-59. of Verbal Learning and Verbal Behavior 12 (1973): Read

of the tasks, and "Whales" was hardest on the other two. It is conceivable that the two expository passages had differential effects depending on their cohesiveness.
"Paper" had well-sequenced information whereas

"Whales" presented the conclusions first and the motiva tion information last. These kinds of deviations have been shown to effect recall (15) and reconstruction of stories (10). Thus the task and passage interaction provides further evidence of heretofore unsuspected differences. The main effect of reading ability in the analyses of ability with tasks and passages was not at all surprising. It showed that the better readers obtained higher scores on the comprehension tasks and passages than did the poorer readers, a finding teachers would certainly expect.
Conclusion

5. Davis, F. B. "Research in Comprehension in 4 (1968): 499-545. ing Research Quarterly 6. Gibson, E. J., and Levin, H. The Psychology Mass.: M.I.f. Cambridge, Press, 1975. 7. Guthrie, J. T. "Reading and Comprehension in Good and Poor Readers." Journal sponses

Reading." of Reading.

Re Syntactic of Educational

Psychology 65 (1973): 294-99.

8. of Prose as a Function R. "Recall of the Structural Johnson, or the Linguistic Units." Journal and Ver of Verbal Learning bal Behavior 9 (1970): 12-20. Iowa: Klare, G. R. The Measurement Ames, of Readability. Iowa State University 1963. Press, J. Story Structure and E.; Mason, McClure, J.; and Barnitz, on Children's to Sequence Tech. Age Effects Stories, Ability 122. Urbana: of Illinois, Center for the University Rep. No. 1979. Study of Reading, May L. M. Standard Test Lessons in McCall, W. A., and Crabbs, New York: Teachers Uni Reading. Press, Columbia College 1961. versity, ed. Reading at Four Linguistic Pennock, C, Comprehension Levels. Newark, Delaware: International Associa Reading tion, 1979. E. F., and Culhane, J. W. "Comparable Cloze and Rankin, Choice Comprehension Test Scores." Journal Multiple of 13 (1969): 193-98. Reading M. S.; Joag-Dev, C; and Anderson, R. C. "A Steffensen,

9. 10.

11.

question
used swer, of

The interaction between passage and task raises the of which, if any, of the operational definitions
here is actually course, "reading is that each comprehension." of them is, and, The at the an same

12.

13.

time, none of them is. That is, as Davis (5) has argued,
comprehension is not a unitary construct. Several impli 14.

This content downloaded from 148.206.53.9 on Sat, 25 Jan 2014 23:35:31 PM All use subject to JSTOR Terms and Conditions

236
Cross-Cultural 15. Stein, N., Perspective on Reading

THE JOURNAL OF EDUCATIONAL RESEARCH

Comprehension." of Organization and Instructional Urbana: Set on Story Memory, Tech. Rep. No. of Illinois, Center for the Study 68. of Read

Reading Research Quarterly 15 (1979): 10-29.

and Nezworski, T. The Effects

University 1978. ing, January

^WESTERN EDUCATION
A Quarterly
Editor: "... tive Raymond

EUROPEAN
Journal of Translations
E. Wanner, U. S. Office of Education studies

to the scholarly contribution an important ? Journal education." Library Contributing Editors

of compara

Suzanne Brickman, lege,

Audebert,

City

University Internationale University Spain.

W. Paris; William National, Col Max Eckstein, of Pennsylvania; Queens University J. C. van Eyndhoven, of New York; Sebastiaan University f?r Institut of Amsterdam; Christoph F?hr, Deutsches Institut P?dagogique Marcello Luketti, Frankfurt; Forschung, P?dagogische of Jose M. Diaz de Rabago, of Rome; Santiago, University

Forthcoming

Issues

EDUCATIONAL REFORM AND THE WORLD OF WORK

Educational Economic in Sweden Educational in France Educational Quarterly. Institutions: Individuals: Policy First $105.00 $30.00
M. E. Sharpe, Inc., Publisher 901 N. Broadway, White Plains,

Planning Change,

and Secondary Educational Needs,

School

Reform

in Italy School Reform

and Secondary

Needs,

Economic

Change

and Secondary

School

Reform

and Secondary issue: Spring

School 1969.

Reform

in England

?s

N.Y.

10603

This content downloaded from 148.206.53.9 on Sat, 25 Jan 2014 23:35:31 PM All use subject to JSTOR Terms and Conditions