Beruflich Dokumente
Kultur Dokumente
20(3) 223-2351999
RICHARD E. MAYER
JILL L. QUlLlCl
ROXANA MORENO
University of California, Santa Barbara
ABSTRACT
An after-school computer club was developed in which language-minority
children learned to master a series of educational computer games through
reading instructions, interacting with peers, and interacting with adult men-
tors. Did twenty-five elementary school children who regularly participated
in an after-school computer club during an academic year (treatment group)
learn generalizable problem-solving skills as compared to twenty-five non-
participating peers who were matched for grade level, gender, and English
language proficiency (comparison group)? Based on a dynamic assessment
given at the end of the academic year, treatment students were more success-
ful than comparison students in learning how to play a new educational
computer game that was presented as a paper-and-pencil mathematics puzzle
learning task. This study shows how an informal educational environment
can foster generalizable problem-solving skills that transfer to learning in a
school environment.
The goal of this study was to examine the cognitive consequences of participating
in an after-school computer club. In light of strong claims concerning the benefits
of learning to use computers and the increasing role of informal educational
environments (such as out-of-school computer clubs), it is worthwhile to carefully
investigate what children learn from attending an after-school computer club [ 11.
To examine this issue, an after-school computer club was developed at a local
Boys and Girls Club in which elementary school children learned to master a
*This research was supported by a grant from the Andrew W. Mellon Foundation.
223
Q 1999, Baywood Publishing Co., Inc.
doi: 10.2190/V1UG-2W6F-4RYH-5R9T
http://baywood.com
224 / MAYER, QUlLlCl AND MORENO
written responses from the Wizard. Children who gain extensive experience may
achieve a special status as Young Assistants to the Wizard.
What is learned in an after-school computer club? One possible answer is that
children learn content-specific skills, including the specific mathematical and
language arts material that is covered in the educational computer games that the
children learn to use. For example, Blanton, Moorman, Hayes, and Warner found
that children who learned to use educational software for mathematics and lan-
guage arts in an after-school computer club showed greater improvement in their
knowledge of mathematics and language arts than did matched comparison stu-
dents [ 5 ] . A second possible answer is that children learn computer-specific skills,
including computer literacy skills such as how to load a file or how to quit a
program. For example, Schustack, Strauss, and Worden found that the amount of
participation in an after-school computer club was related to children’s computer
literacy as measured by an extensive test of computer skills involved in the club
[4]. In addition to learning content-specific and computer-specific skills, a third
possible answer is that children learn generalizable cognitive skills, including
learning strategies such as learning how to learn to solve new problems and
puzzles. This third kind of learning outcome is the focus of the current study; in
particular, we are interested in the degree to which children learn generalizable
cognitive skills in an informal educational environment. This is an important issue
because the search for problem-solving transfer has a long and somewhat disap-
pointing history in educational research [6,7].
In spite of the historical difficulty in documenting educational experiences that
promote problem solving transfer, learning to use educational software is some-
times suggested as a way of achieving this elusive goal. Yet, in contrast to claims
for the value of educational computing in promoting transferable cognitive skills,
there has not been overwhelming supporting evidence [8, 91. The current study
provides a direct test of the cognitive transfer hypothesis, namely that exposure to
certain kinds of educational computing environments can transfer to improved
cognitive processing in different situations. Thus, the current study contributes
to an emerging research based on the generalizable cognitive consequences of
exposure to educational computing experiences.
To assess the degree to which children learn generalizable skills in an edu-
cational computing environment, we developed a dynamic assessment in which
children learned to solve a series of mathematical puzzles that they had never seen
before. The mathematical learning task was a paper-and-pencil version of a
mathematics computer game called Puzzle Tanks which none of the children had
ever used [lo]. Children worked individually with an experimenter by reading
instructions and attempting to solve a series of three puzzles (with immediate
feedback after each “move” they made). If children learn generalizable cognitive
skills in the computer club, we predict the treatment group will produce fewer
errors than the comparison group in learning to solve the novel mathematical
puzzles. A secondary prediction-focusing on the quality of performance-is that
226 I MAYER, QUlLlCl AND MORENO
the treatment group will be more likely to invent sophisticated solution methods
than the comparison group.
METHOD
Participantsand Design
The participants were fifty children who attended an elementary school in
southern California. Ninety-two percent of the children in the study spoke Spanish
as their first language and were designated as “limited English proficient” by
the school. Using a matching paradigm, half the children served in the treatment
group and half were in the control group. For each child in the treatment group
there was a matched child in the comparison group, with matching based on level
of English proficiency, grade level, and gender.
Materials
We constructed a hand-simulated version of the mathematics education pro-
gram, Puzzle Tanks [lo], using a 8-1/2 x 11 inch sheets of paper consisting of a
two-page instructional booklet, problem 1, problem 2, and problem 3.
The instructional booklet is reproduced in Appendix A. At the top of the page,
students are introduced to the puzzle tank situation consisting of four tanks-an
overhead tank containing an unlimited amount of juice, a left tank with a capacity
of seven gallons, a right tank with a capacity of three gallons, and a truck tank to
be filled to exactly ten gallons-and eight valves controlling the movement of
juice through pipes from one tank to another-the arrow from the overhead tank to
the left tank, the arrow from the overhead tank to the right tank, the arrow from the
left tank to the right tank, the arrow from the right tank to the left tank, the a m w
from the left tank to the overhead tank, the arrow from the left tank to the truck
tank, the arrow from the right tank to the truck tank, and the arrow from the right
tank to the overhead tank.
Then, four pairs of graphics are presented. On the first left graphic, the arrow
from the overhead tank to the left tank is circled and the student is asked to write
numbers in the left tank, right tank, and truck tank using a black pencil. On the first
right graphic, after the student writes the numbers, the experimenter circles the
same a m w and writes in the correct values (7 for the left tank, 0 for the right tank,
and 0 for the truck tank) using a red pencil. On the second left graphic, the arrow
from the left tank to the right tank is circled and the student is asked to write
numbers in the left tank, right tank, and truck tank using a black pencil. On the
second right graphic, after the student writes the numbers, the experimenter circles
the same arrow and writes in the correct values (4 for the left tank, 3 for the right
tank, and 0 for the truck tank) using a red pencil. On the third left graphic, the
arrow from the right tank to the truck tank is circled, and the student is asked to
WHAT IS LEARNED? I 227
w
SIZE: 6 SIZE: 5
GOAL: 11
rl
y'
1- " GOAL:
n
11
L-4
w
SIZE: 5
SIZE:
L2JE
GOAL: 11
Figure 1. A puzzle tank problem sheet. [Note: For the graphic on the left,
the student circles an arrow and writes numbers in each tank using a black pencil;
for the graphic on the right, the experimenter circles the same arrow and
writes the correct numbers in each tank using a red pencil. This procedure
is repeated until the problem is solved or eighteen pairs of graphics
have been used, whichever comes first.]
WHAT IS LEARNED? I 229
Procedure
At the beginning of the school year, several students from each participating
class in an elementary school were invited to join an after-school computer club at
a local Boys and Girls Club. The students were selected based on teacher recom-
mendation to represent the diversity of children at the school, with a special
focus on language-minority children. ’henty-five students (i.e., treatment group)
accepted the invitation and attended the computer club at least ten times during
the academic year. Club activities emphasized learning to use a wide variety of
educational software in a social environment that included working with peers
and working with college-student mentors as described in Mayer [l].
The comparison students came from the same classrooms and had the same
general characteristics as the treatment students but had never attended the com-
puter club. To identify matched comparison students, the experimenters obtained
the latest English language proficiency scores for all students in the school. The
scores ranged in five steps from LEP-1 (lowest level of limited English profi-
ciency) to LEP-5 (highest level of limited English proficiency); in addition,
students could be classified as English only (monolingual speakers of English) or
fluent English proficient (bilingual speakers of English). Each treatment student
was matched with a comparison student who had the same English language
proficiency classification, was in the same grade level, and was the same gender.
In some cases, the matching was slightly imperfect (e.g., LEP-2 matched with
LEP-3) but these minor variations did not favor one group over the other.
During the last three weeks of the academic year each treatment student and
each comparison student was tested individually at hisher school during the
school day. The student and experimenter sat together at a picnic table in a quiet
outdoors area. The experimenter explained that she was studying how children
learn to play a computer game and briefly described the game. First, the experi-
menter handed out a clipboard containing the two-page instructional booklet for
Puzzle Tanks (as shown in Appendix A) and read aloud the instructions at the top
of the first page. Next, the student was shown an example graphic in which the left
tank had a seven gallon capacity, the right tank had a three gallon capacity, and the
goal was to get exactly ten gallons in the truck tank. The student was told that the
pipe from the overhead tank to left tank was opened-indicated by circling the
corresponding arrow-and the student was asked to write numbers using hisher
pencil to show how much juice will be in each tank. Then, in the corresponding
graphic the experimenter wrote the correct answers and stated that circling the
arrow from the overhead tank to the left tank means that “I will fill the first tank to
the top.” This process was repeated for three additional graphics as shown in the
bottom of the first page and top of the second page in Appendix A. For each of the
four example graphics, the student was asked to write in numbers and received
immediate feedback and explanation from the experimenter. Finally, the
experimenter read aloud the instructions on the bottom of the second page. The
230 I MAYER, OUlLlCl AND MORENO
instructions were similar to those that came with the Puzzle Tanks game and were
intended to familiarize the student with the procedure that was used on each of
the three learning problems.
The experimenter removed the instmctional booklet and handed the student a
clipboard containing a sheet for the first problem. Beginning with the top graphic
on the left, the student was asked use a black pencil to circle an arrow-indicating
the opening of valve-and to write a number in the left tank, right tank, and truck
tank-representing how many gallons of juice would be in each after the valve
was opened. Then, the experimenter indicated the correct answers on the cor-
responding graphic to the right using a red pencil. On the next row of graphics, the
student was again asked to circle the next arrow and write the numbers using the
graphic to the left, and the experimenter again indicated the correct answers in red
in the corresponding graphic to the right. This process was repeated until the
problem was solved or the student used eighteen graphics. Then, the next problem
was presented, and the same procedure was followed. When one sheet was filled
(consisting of 3 “moves” by the student), another sheet was placed on the clip-
board. After the student completed the third problem, he or she was thanked and
given a small gift. The experimenter kept a record of the ordering of completed
sheets for each student. The Puzzle Tanks game was not used at the computer club
nor was it used in the children’s elementary school; none of the children in the
study expressed prior familiarity with the game.
RESULTS
The primary issue addressed in this study concerns whether students who had
participated in a computer club in which they learned to use a variety of educa-
tional programs were better able to learn a new piece of educational software that
was presented as a paper-and pencil mathematics task than were matched non-
participators. The children’s ability to transfer their learning to a new task, i.e., a
Puzzle Tanks game, was examined in two ways: by analyzing the frequency of
errors and the quality of solution methods.
First, the mean error rate on the three learning problems is an indication of the
children’s difficulty in learning to follow the instructions for the Puzzle Tanks
game. The error rate provides an indication of the children’s ability to use feed-
back to help them learn how to solve mathematical puzzles that they had never
seen before. In learning to play the puzzle tanks games over the course of three
learning problems, the treatment group failed to solve 24 percent (SD = 28) of
the problems within eighteen trials whereas the comparison group failed on 43
percent (SD = 35) of the problems. The moderate error rates indicate that the
Puzzle Tanks task was an appropriate diagnostic instrument for our participant
pool, with neither group displaying either a ceiling or floor effect. A paired t-test
revealed that the difference in error rates is statistically significant, t(24) = 2.347,
WHAT IS LEARNED? / 231
p c .03. The effect size is .53. This finding provides evidence that the treatment
group had less difficulty in learning how to solve novel mathematical puzzles-
i.e., to play the Puzzle Tanks game-than did the comparison group.
Second, we performed a more fine grained analysis of the solution strategies
that children used in learning to play the Puzzle Tanks game. On the last two
learning problems students could use a more sophisticated solution method-
which we call a subtraction strategy-r a less sophisticated method-which we
call an addition strategy. Based on students’ patterns of moves, we tallied the
number of times students used either of these two strategies. In learning to play the
Puzzle Tanks game over the course of the final two learning problems, children
in the treatment group used the subtraction strategy on 42 percent (SD= 42) of
the problems whereas the comparison group used the subtraction strategy on
20 percent (SD= 32) of the problems. A paired t-test revealed that this difference
< .07.The effect size is .68. In contrast,
is marginally significant, t(24) = 1 . 9 0 1 , ~
children in the treatment group used the addition strategy on 20 percent (SD= 24)
of the problems whereas the comparison group used it on 24 percent (SD= 28) of
the problems, reflecting a nonsignificant difference, r(24) < 1, p > .20. Overall,
our analysis of strategies suggests that the superior performance of the treatment
group in learning to solve puzzle tank problems is reflected in their invention of
more sophisticated (rather than less sophisticated) strategies as compared to the
comparison group.
DISCUSSION
This study provides promising evidence that students can learn generalizable
cognitive skills in an informal educational environment such as an after-school
computer club. In addition to learning content-specific skills (e.g., the specific
target math skills in the software they used) and computer-specific skills (e.g.,
computer literacy skills required to use the software), students who attended an
after-school computer club also learned how to learn-that is, they developed
generalizable skills that helped them learn how to play a novel computer game. In
a sense, learning to use educational software improved children’s literacy in that
computer club participants learned how to understand instructions and feedback
for educational games and puzzles. Strategies for learning how to learn could be
transferred from an informal educational environment to learning a new academic
task in a school environment-solving mathematics puzzles presented in paper-
and-pencil format. Although not directly tested in this study, our results are
consistent with the idea that informal learning environments may be particularly
helpful for language minority students because they are freed from the constraints
imposed by formal educational settings.
This study of the cognitive consequences of participation in an after-school
computer club provides an advance over our previous study [2] in two important
232 / MAYER, QUlLlCl AND MORENO
ways. First, in our previous study the dependent measure was a static paper-and-
pencil test of students’ comprehension of arithmetic word problems; in the current
study the dependent measure was dynamic assessment of students’ learning of a
new mathematics game. Thus, the current dependent measures provide a new way
of assessing the degree of problem-solving transfer from an educational comput-
ing environment to a more school-based learning environment. Unlike the static
test used in our previous study, the Puzzle Tanks test allowed us to examine
students’ ability to learn as well as the quality of their problem solving strategies.
Second, in our previous study the students were participating in the first year of
the after-school computer club, whereas the current study involved a new set of
students who participated in the third year of operation of the after-school com-
puter club. Thus, by using new students who participated in a somewhat refined
version of the club, we are able to demonstrate that our previous results were not
a one-time fluke.
We used a paper-and-pencil version of the Puzzle Tanks administered as a
one-on-one interview at the child’s school rather than a computer version admin-
istered at the club for two primary reasons. First, a one-on-one interview format
provided the kind of high quality in-depth data we needed when working with
elementary school children. Second, we wanted to see whether experience in an
informal computer-based learning environment would transfer to a more school-
like learning environment.
As in most field studies, it is not possible to pinpoint which aspects of the
after-school computer club were most important in promoting problem-solving
transfer. Clearly, participating in the computer club resulted in better problem-
solving transfer than not participating; but it is not possible to determine how
much of the improvement is attributable to students’ experience with learning
to use educational computing software, to working one-on-one with peers and
mentors, or to other aspects of the computer club. Further research is needed
to pinpoint the relative contributions of the social (e.g., role of mentors and
peers) and cognitive (e.g., role of computing software) aspects of the Fifth Dimen-
sion program.
ACKNOWLEDGMENTS
APPENDIX A
L
-SIZE: I" in the example. Above rhc d r is
an unlimited supply of Wonder Juice. YOU can
circle an m w undcr rhc supply lo fill Up 3
rank. Thcrc is llro a tanker m c k u rhc bottom.
Your job IS lo gel a ccnain amount of juice into SIZE: 1
WZE:
rhc truck. h rhc example Ihc goal is 10 gallons.
marked u "GOAL: 10". By circling m o w s you
can cmpiy the contents o i a iank inio the tNck.
you c3n cmpiy the contcnts of a t u k back in10
he unlimitcd supply ai ihc lop. or you can pour
Ihc conicnts of onc tank into the oihcr tank.
Today. you w i l l solve 1 few problcms. E x h lime
you circlc one of the eight 3 m w s . p l c s c wnlc COAL: I0
numbers ,howm;: how many gallons of juicc
you think rlrc now in tmk 1.iank 2. md the tmck.
Then. I w i l l w n t c down the correct numbers.
If you circle this i r r o w . how many gallons If you circle thts arrow. I will f i l l lhc first t3nk
W Y I I I be In c3ch rink and the truck! Write lo Ihc lop:
u
ihc numbcrs usins your pencil.
( X I 0
I f you circlc t h u 3rruw. huw many galloar I f yau circlc this amow. I wtll pour thc cmtcnts
will bc ~n each imk md thc iNCk? Write or thc first tank into the sccond tank until ihc
the number3 using yaur pcncal. rcccrnd tank is full.
L7J
GOAL: 10
234 I MAYER, QUlLlCl AND MORENO
APPENDIX A (Cont'd.)
If you circle this m o w . how many gallons If you circle this u m w . I will cmpty lhc
w i l l be in c l c h t m k md thc truck? Wntc contents of the wcond t a n k into Ihc tanker
the numbers using your pencil. truck
SIZE: 744-J
-RL: 10 y +-EL: 10
If you circle this arrow. how many gallons If you circlc tbis m o w . I w i l l cmpty the contents
wall be In each tank and tbc truck? Writc of the first tank back into the unlimited supply:
SIZE: wu u
thc numbcrs using your pencil.
d7
SIZE: 3
SIZE:
dyLL7 SIZE: 3
E x h iimc you circlc an a m w . I will wrilc numbers in tmk I. tank 2. and thc truck to show bow
much juicc is in cach anc. Tblhcn. yau should circle ihc neal arrow. Try to solve lhc pmblcm
using I. fcw slcps Y parsiblc.
REMEMBER: It is imponant 10 try to solvc each pmblcm using Y few stcps JS possible.
WHAT IS LEARNED? / 235
REFERENCES
1. R. E. Mayer, Out-Of-School Learning: The Case of an After-School Computer Club,
Journal of Educational Computing Research, 16, pp. 333-336.1997.
2. R. E. Mayer, J. Quilici, R. Moreno, R. Duran, S.Woodbridge, R. Simon, D. Sanchez,
and A. Lavezzo. Cognitive Consequences of Participation in a Fifth Dimension After-
School Computer Club. Journal of Educational Computing Research, 16, pp. 353-369,
1997.
3. A. Nicolopolou and M. Cole, The Fifth Dimension, Its Playworld, and Its Institutional
Contexts: The Generation and Transmission of Shared Knowledge in a Culture of
Collaborative Learning, in Contexts for Learning: Sociocultural Dynamics in
Children's Development, E. A. Foreman, N. Minnick, and C. A. Stone (eds.), Oxford
University Press, New York, pp. 283-314, 1993.
4. M. W. Schustack, R. Strauss, and P. E. Worden, Learning about Technology in
a Non-Instructional Environment, Journal of Educational Computing Research, 16,
pp. 337-352, 1997.
5. W. E. Blanton, G. B. Moorman, B. A. Hayes, and M. L. Warner, Effects of Partici-
pation in the Fifth Dimension in Far Transfer, Journal of Educational Computing
Research, 16, pp. 371-396, 1997.
6. R. E. Mayer and M. C. Wittrock, Problem Solving and Transfer, in Handbook of
Educational Psychology, D. Berliner and R. Calfee (eds.), Macmillan, New York,
pp. 45-61, 1996.
7. A. McKeough, J. Lupart, and A. Marini (4s.). Teaching for Transfer, Erlbaum.
Mahwah, New Jersey, 1995.
8. Cognition and Technology Group at Vanderbilt, Looking at Technology in Context:
A Framework for Understanding Technology and Education Research, In Handbook
of Educational Psychology, D. C. Berliner and R. C. Calfee (eds.), Macmillan, New
York, pp. 807-840, 1996.
9. R. E. Mayer (ed.), Teaching and Learning Computer Programming, Erlbaum,
Hillsdale, New Jersey, 1988.
10. Sunburst, Puzzle Tanks [computer program], Author, Pleasantville, New York, 1996.