Research methods in language learning

The experimental method

quencies; a n d c o r r e l a t i o n , w h i c h enables us to e x a m i n e the degree o f associ-

TABLE 2-9 SUMMARY OF NEW Z E A L A N D BOOK FLOOD EXPERIMENT
a t i o n between scores. I n the n e x t section, w e l o o k at types o f experiments.
Rationale T o examine possible correlations between extensive
reading/wide availability of books and reading
Types of experiments habits and skills
Experimental subjects M a o r i children in two schools
E a r l i e r i n the chapter, I p o i n t e d o u t that i t was n o t a l w a y s feasible to carry Control subjects None
o u t a true e x p e r i m e n t - that circumstances such as the i m p o s s i b i l i t y o f r a n - Independent variable Wide access to books
d o m l y assigning subjects t o e x p e r i m e n t a l and c o n t r o l groups sometimes d i c -
Def)endent variables Scores on reading tests
tated that a quasi- o r pre-experiment rather t h a n a true e x p e r i m e n t be c o n -
Qualitative information on reading habits
d u c t e d . (Of course, i t is also possible that a quasi- o r pre-experiment is carried
Outcome Significant improvement i n reading scores
o u t because o f ignorance o n matters o f research design o n the part o f the
researchers.) In this section, I shall describe the differences between these three
types o f e x p e r i m e n t s . T h e different designs are illustrated by C o h é n a n d M a n -
i o n (1985), w h o describe a n d c r i t i q u e three experiments reported i n the l i t - TABLE 2.IO CONTRASTING PRE-EXPERIMENTS, QUASI-EXPERIMENTS TRIIF
EXPERIMENTS
erature: the N e w Z e a l a n d B o o k F l o o d E x p e r i m e n t , the B r a d f o r d B o o k F l o o d
E x p e r i m e n t , a n d the U n d e r s t a n d i n g of E l e c t r i c a l C i r c u i t s E x p e r i m e n t . (I shall Type Characteristics
o n l y describe the first o f these studies. F o r a n a c c o u n t a n d c r i t i q u e o f the other
Pre-experiment M a y have pre- and posttreatment tests, but lacks a control
t w o studies, see C o h é n a n d M a n i o n 1985.)
group
T h e N e w Z e a l a n d B o o k F l o o d E x p e r i m e n t was established t o e x a m i n e the
Quasi-experiment Has both pre- and posttests and experimental and control
effects o f extensive r e a d i n g a n d the w i d e a v a i l a b i l i t y o f b o o k s o n the reading
groups, but no random assignment of subjects
habits a n d skills o f p r i m a r y school c h i l d r e n . T h e t w o schools w h i c h t o o k part
True experiment Has both pre- and posttests, experimental and control
i n the s t u d y c o n t a i n e d large numbers of M a o r i ( N e w Z e a l a n d aboriginal) c h i l -
d r e n w h o were believed t o have l i m i t e d access t o b o o k s . T h e c h i l d r e n were groups, and random assignment of subjects
g i v e n sets o f p r e - e x p e r i m e n t a l a t t i t u d i n a l , vocabulary a n d r e a d i n g tests. T h e y
w e r e t h e n ' f l o o d e d ' w i t h books selected b y teachers a n d librarians. D a t a
i n c l u d e d scores o n posttests administered s i x m o n t h s after the i n i t i a t i o n o f t h e a d d i t i o n o f o n e o r m o r e c o n t r o l groups. G i v e n t h e fact that i t w a s n o t
the project, as w e l l as qualitative data o n the r e a d i n g habits o f five c h i l d r e n feasible t o assign subjects at r a n d o m t o e x p e r i m e n t a l a n d c o n t r o l schools, i t
f r o m each g r o u p . I n f o r m a l reading inventories were also used t o o b t a i n Infor- w o u l d be i m p o r t a n t f o r the researchers t o m a t c h the schools i n terms o f size,
m a t i o n o n the r e a d i n g behaviours o f the subjects. Dependent variables b a c k g r o u n d , o r g a n i s a t i o n , teaching m e t h o d s , a n d b a c k g r o u n d abilities o f
i n c l u d e d the q u a n t i t y o f reading u n d e r t a k e n a n d the skills, interests, a n d atti- learners. T h e research design w o u l d also be strengthened b y p r o v i d i n g sub-
tudes o f the subjects. Subjects s h o w e d significant i m p r o v e m e n t s o n a l l o f these jects w i t h c a r e f u l l y selected pre- a n d posttreatment tests. F i n a l l y , i t w o u l d be
measures. T h e e x p e r i m e n t is summarised i n T a b l e 2.9. h i g h l y desirable t o collect qualitative d a t a o n r e a d i n g habits, classroom teach-
T h i s s t u d y is a n e x a m p l e o f a one-group pretest posttest design. (In fact. i n g procedures, a n d so o n , i n b o t h e x p e r i m e n t a l a n d c o n t r o l classrooms. Such
C o h é n a n d M a n i o n c a l i i t a pre-experiment.) Such studies have w e a k internal q u a l i t a t i v e i n f o r m a t i o n is often c r u c i a l for the i n t e r p r e t a t i o n o f q u a n t i t a t i v e
v a l i d i t y because i t is almost impossible to state w i t h a n y confidence that the d a t a , as Spada (1990) has s h o w n . T h e major differences between pre-experi-
dependent variables, that is, postintervention i m p r o v e m e n t s i n r e a d i n g , were ments, quasi-experiments, a n d true experiments are set o u t ¡n T a b l e 2 . 1 0 .
due t o the independent variable, that is, the b o o k ' f l o o d ' . M a n y other reasons
for i m p r o v e m e n t s c o u l d be advanced, i n c l u d i n g the effect o f t a k i n g part i n an
e d u c a t i o n a l i n n o v a t i o n , the fact that the c h i l d r e n m a t u r e d d u r i n g the course The psychometric study: an example
o f the e x p e r i m e n t , the effect o f other variables w h i c h have n o t been c o n -
t r o l l e d , a n d so o n . I n this section, I s h a l l illustrate t h e characteristics o f p s y c h o m e t r i c research
T h e i n t e r n a l v a l i d i t y o f this study w o u l d have been greatly i m p r o v e d by b y t a k i n g y o u i n some detail t h r o u g h a research ref)ort based o n s u c h a s t u d y .

40

Research methods in language learning
T h e s t u d y is a n i n v e s t i g a t i o n b y C h a u d r o n a n d R i c h a r d s (1986) i n t o the effect
o f discourse m a r k e r s o n t h e c o m p r e h e n s i o n o f lectures b y n o n - n a t i v e speak-
i n g u n i v e r s i t y students. I have chosen this s t u d y because i t exemplifies some
o f the c e n t r a l p o i n t s w h i c h have been discussed i n the chapter.
RESEARCH AREA
T h e general research área investigated b y C h a u d r o n a n d R i c h a r d s w a s t h e
c o m p r e h e n s i o n o f u n i v e r s i t y lectures b y n o n - E n g l i s h s p e a k i n g b a c k g r o u n d
students.
<»
AIM
T h e a i m o f t h e s t u d y w a s t o investígate the effects o f discourse signáis a n d
m a r k e r s o n t h e c o m p r e h e n s i o n o f lectures by second langauge learners.
QUESTIONS/HYPOTHESES
T w o research questions w e r e posed by the researchers:
1. W h a t is the effect o n L 2 learners' c o m p r e h e n s i o n o f lectures o f the use o f
discourse m a r k e r s w h i c h i n d i c a r e the o v e r a l l o r g a n i s a t i o n o f lectures -
that is, m a c r o m a r k e r s , w h i c h signal the m a c r o structure o f a lecture
through h i g h l i g h t i n g t h e major i n f o r m a t i o n i n the lecture a n d the
s e q u e n c i n g o r i m p o r t a n c e o f that i n f o r m a t i o n ?
2. W h a t is the effect o n L 2 learners' c o m p r e h e n s i o n o f lectures o f the use o f
m i c r o m a r k e r s , w h i c h indícate l i n k s between sentences w i t h i n t h e lecture
o r f u n c t i o n as fillers?
These t w o questions w e r e f o r m u l a t e d as three research hypotheses:
Hypothesis 1: L 2 learners w o u l d c o m p r e h e n d a lecture better w h e n m i c r o
markers were added than w h e n n o markers were added.
Hypothesis 2: L 2 learners w o u l d c o m p r e h e n d the lecture w i t h m a c r o m a r k e r s
better t h a n the lecture w i t h o n l y m i c r o m a r k e r s .
Hypothesis 3: L 2 learners w o u l d c o m p r e h e n d best the lecture w i t h b o t h m i c r o
and macro markers.
In other w o r d s , the researchers were suggesting (1) that the c o m p r e h e n s i o n
o f lectures w o u l d be enhanced i f the lecturer used different types o f signáis;
(2) t h a t lectures i n w h i c h b o t h the overall structure a n d o r g a n i s a t i o n , as w e l l
as between-sentence l i n k s , are e x p l i c i t l y signalled w i l l be m o r e c o m p r e h e n -
sible t h a n lectures i n w h i c h there are n o signáis, o r i n w h i c h o n l y one t y p e o f
s i g n a l is present, a n d (3) t h a t s i g n a l l i n g o v e r a l l s t r u c t u r e a n d o r g a n i s a t i o n w i l l
result i n greater c o m p r e h e n s i o n t h a n s i g n a l l i n g relationships between sen-
tences o n l y .
42
The experimental methoc
JUSTIFICATION ( W H Y T H E Q U E S T I O N IS WORTH
INVESTIGATING)
T h e researchers justified the research o n the g r o u n d s that, w i t h greater n u m
bers o f n o n - n a t i v e speakers o f E n g l i s h e n t e r i n g universities a r o u n d the w o r l c
i n w h i c h E n g l i s h is t h e médium o f i n s t r u c t i o n , there is a need f o r researcf
i n t o t h e processes u n d e r l y i n g p e r f o r m a n c e i n a second o r f o r e i g n language
w i t h i n a u n i v e r s i t y e n v i r o n m e n t . S u c h research c a n p r o v i d e guidance f o i
teacher e d u c a t i o n , i n s t r u c t i o n a l materials, a n d c u r r i c u l u m development.
In t h e i r l i t e r a t u r e r e v i e w , t h e researchers contrast b o t t o m - u p a n d t o p - d o w n
c o m p r e h e n s i o n strategies. B o t t o m - u p strategies begin w i t h the processing of
i n d i v i d u a l s o u n d s , t h e n w o r d s , then clauses, a n d t h e n w h o l e messages. T o p -
d o w n strategies, o n the o t h e r h a n d , begin w i t h o v e r a l l k n o w l e d g e o f the sub-
ject m a t t e r a n d s t r u c t u r e o f t h e text t o be c o m p r e h e n d e d . F o r m a n y years
there has been a n o n g o i n g debate about the relative merits o f t o p - d o w n a n d
b o t t o m - u p approaches t o language processing. W h i l e the debate has dealt
p r i n c i p a l l y w i t h r e a d i n g c o m p r e h e n s i o n , i t has also been i m p o r t a n t i n the lit-
erature o n l i s t e n i n g (see, f o r e x a m p l e , A n d e r s o n a n d L y n c h 1988; R o s t 1990).
T h e debate, a n d the research i n t o c o m p r e h e n s i o n processes o n w h i c h i t rests,
is i m p o r t a n t f o r p e d a g o g y , because the different approaches have clear i m p l i -
c a t i o n s f o r the design o f t e a c h i n g strategies a n d classroom materials. T h e bot-
t o m - u p a p p r o a c h suggests that w e begin by t e a c h i n g the i n d i v i d u a l elements
w h i c h c o n s t i t u t e language (that is, phonemes a n d graphemes), t h e n w o r d s
a n d phrases, t h e n sentences, a n d finally w h o l e texts. T h e t o p - d o w n a p p r o a c h ,
o n t h e o t h e r h a n d , suggests that w e begin w i t h w h o l e texts a n d encourage
learners t o use t h e i r k n o w l e d g e o f t e x t structure a n d t h e o v e r a l l purpose o f
the s p e a k e r / w r i t e r t o o r i e n t t h e m t o the text. I n recent years, i t has been sug-
gested t h a t b o t h b o t t o m - u p a n d t o p - d o w n strategies are i m p o r t a n t , a n d that
the g o o d reader/listener is able t o e m p l o y b o t h a p p r o p r i a t e l y .
C h a u d r o n a n d R i c h a r d s argüe that b o t h t o p - d o w n a n d b o t t o m - u p p r o -
cessing are i m p o r t a n t i n u n d e r s t a n d i n g lectures, a n d they cite studies w h i c h
suggest t h a t b o t h m a c r o m a r k e r s a n d m i c r o m a r k e r s c a n facilitate this under-
s t a n d i n g . M a c r o m a r k e r s c a n help learners w i t h the o v e r a l l o r g a n i s a t i o n o f a
l e c t u r e , as w e l l as assist i n c o m p r e h e n d i n g the f u n c t i o n a l i n t e n t i o n o f the
speaker i n r e l a t i o n t o subsidiary parts o f the lecture. M i c r o m a r k e r s , w h i c h
serve as b r e a t h i n g spaces a n d w h i c h p r o v i d e learners w i t h o p p o r t u n i t i e s t o
e m p l o y b o t t o m - u p processing strategies, s h o u l d also facilitate t h e c o m p r e -
h e n s i o n o f lectures.
SUBJECTS
T h e r e w e r e t w o g r o u p s o f subjects i n the s t u d y : a p r e - u n i v e r s i t y g r o u p c o n -
s i s t i n g o f 7 1 E S L ( E n g l i s h as a second language) students enroUed i n a u n i -
43
 The experimental method
Research methods in language learning
" A n d so, what we've come to by now was that by 1803, the original thirteen
versity language p r o g r a m , a n d a u n i v e r s i t y g r o u p , c o n s i s t i n g o f 81 E S L stu- colonies had doubled in size . . . "
dents e n r o l l e d i n u n i v e r s i t y programs. T h e researchers h a d placement test a n d
standardised test scores o n the subjects.
PROCEDURE

MATERIALS T h e f o u r different versions o f the lecture w e r e assigned at r a n d o m to the sub-

jects, w h o were p r o v i d e d w i t h a set of i n s t r u c t i o n s o n h o w to d o cloze, múl-
F o u r versions o f a lecture o n A m e r i c a n h i s t o r y w e r e used i n the s t u d y . T h e
t i p l e c h o i c e , a n d true/false questions. T h e y were also g i v e n t w o short practice
baseline versión was a ' n a t u r a l ' versión presented to E S L students ('natural'
l i s t e n i n g passages.
i n t h a t it was n o t altered by the researchers). A m a c r o versión c o n t a i n e d
m a r k e r s o f discourse o r g a n i s a t i o n such as the f o l l o w i n g : Dependent variable: Scores o n cloze, múltiple choice, and true/false
questions.
w h a t I ' m g o i n g to talk about t o d a y . . .
let's g o b a c k to the b e g i n n i n g . . . Independent variable: M i c r o a n d m a c r o m a r k e r s i n academic lectures.
this b r o u g h t about n e w problems . . .

T h e m i c r o versión c o n t a i n e d ' f i l l e r ' phrases a n d m a r k e r s o f intersentential TYPE OF DATA

r e l a t i o n s s u c h as the f o l l o w i n g : Subjects' scores o n the cloze, múltiple c h o i c e , a n d true/false tests. Subjects'
p l a c e m e n t scores.
temporal links: t h e n , a n d , n o w , after this, at t h a t t i m e
causal links: because, so (Before p r o c e e d i n g , y o u m i g h t l i k e to pause a n d t h i n k about the statistical
contrastive relationships: but, actually tools the experimenters m i g h t use to analyse t h e i r d a t a . A r e the researchers
relative emphasis: y o u see, unbelievably, o f course interested i n c o m p a r i n g means, i n c o m p a r i n g frequencies, o r i n investigating
framing/segmentation: w e l l , O K , a l l right? t h e e x t e n t t o w h i c h t w o or m o r e events correlate?)

T h e f i n a l versión c o m b i n e d versions t w o a n d three, a n d c o n t a i n e d b o t h m a c r o

a n d m i c r o m a r k e r s . T h e f o l l o w i n g extracts, t a k e n f r o m the s t u d y , s h o w TYPE OF ANALYSIS

e x a m p l e s o f a l l f o u r versions. T h e d a t a w e r e analysed u s i n g the f o l l o w i n g statistical procedures: Pearson

p r o d u c t - m o m e n t correlations w e r e c a l c u l a t e d between the l i s t e n i n g c o m p r e -
Baseline versión:
h e n s i o n a n d the v a r i o u s placement test scores. (As w e saw earlier, c o r r e l a t i o n
" T h e United States carne into existence officially i n 1783 after eight years of war is used to test the strength o f association between sets o f scores - i n this
instance, the strength o f association between l i s t e n i n g c o m p r e h e n s i o n a n d
" B y 1803, the original thirteen colonies had doubled i n size . . . " p l a c e m e n t scores. In essence, the researchers w e r e a s k i n g , 'Does a student w h o
scored well on the placement test also score well on the listening
Micro versión:
comprehension?')
" W e l l , the United States carne into existence ofi6cially in 1783 after eight years of
Possible differences i n the m e a n scores o n the different versions o f the lec-
war... ture w e r e tested by means o f a n analysis o f covariance ( A N C O V A ) . (This per-
f o r m s a s i m i l a r a n a l y t i c a l task as analysis o f variance, but takes i n t o a c c o u n t
" A n d so, by 1803, the original thirteen colonies had doubled in size . . . "
differences i n the learners' p r o f i c i e n c y levéis.)
Macro versión:
" T o begin w i t h , the United States carne into existence of&cially in 1783 after eight CONCLUSIONS

years of war . . . " T h e researchers f o u n d that m a c r o m a r k e r s (that is, the higher-order discourse
" W h a t we've come to by now was that by 1803, the original thirteen colonies had m a r k e r s w h i c h indícate the overall o r g a n i s a t i o n of a text) were m o r e c o n -
doubled in size . . . " d u c i v e t o successful tecali of the lecture t h a n m i c r o markers (that is, the
Micro-tnacro versión: l o w e r - o r d e r m a r k e r s o f segmentation a n d intersentential markers). T h e y
" W e l l , to begin w i t h , the United States carne into existence ofBcially i n 1783 after p o i n t o u t that this is i n líne w i t h the t o p - d o w n theory of c o m p r e h e n s i o n .
eight years of war . . . "
45
Research methods in language learning
w h i c h suggests that e x p l i c i t l y signalled g u i d a n c e o n the o r g a n i s a t i o n of major
ideas i n the discourse w i l l help listeners c o n s t r u c t appropriate schematic m o d -
els o f the lecture.
A n o t h e r f i n d i n g was that the m i c r o m a r k e r s d i d n o t facilitate c o m p r e h e n -
s i o n . T h e y a c c o u n t for this by suggesting that these m a r k e r s probably d o n o t
a d d e n o u g h content t o m a k e the subsequent i n f o r m a t i o n m o r e c o m p r e h e n -
sible, that the m a r k e r s m a y a c t u a l l y m a k e the lecture less comprehensible by
d i s t r a c t i n g the listeners, a n d that the baseline versión m a y already have been
s l o w e n o u g h t o a l l o w learners t o o b t a i n the m á x i m u m comprehensible i n p u t '
w i t h o u t the a i d o f further pauses a n d b u i l t - i n r e d u n d a n c y .
A n o t h e r s u r p r i s i n g f i n d i n g w a s that the c o m b i n e d m i c r o - m a c r o versión
w a s a c t u a l l y inferior t o the m a c r o versión. T h e researchers attempt t o
a c c o u n t for this f i n d i n g by suggesting that the i n s e r t i o n o f the m i c r o m a r k e r s
increased the l i s t e n i n g l o a d o n the subjects w i t h o u t a d d i n g i n f o r m a t i o n
w h i c h m i g h t have facilitated their o v e r a l l c o m p r e h e n s i o n , a n d that this
d e t r a c t e d f r o m the effect o f the m a c r o m a r k e r s alone.
CRITIQUE
T h i s s t u d y is a c a r e f u l l y conceived a n d c o n d u c t e d investigation i n t o t h e lis-
t e n i n g c o m p r e h e n s i o n o f N E S B u n i v e r s i t y a n d pre-university students w h i c h
illustrates some i m p o r t a n t characteristics o f the e x p e r i m e n t a l m e t h o d . T h e
researchers derive a n u m b e r o f questions/hypotheses f r o m their r e v i e w o f the
e m p i r i c a l a n d theoretical literature o n the subject at h a n d . T h e y suggest rela-
t i o n s h i p s between variables, a n d test these o u t t h r o u g h the exposure o f r a n -
d o m l y assigned subjects t o c o n t r o l a n d e x p e r i m e n t a l groups. T h e data y i e l d e d
by the e x p e r i m e n t are t h e n subjected t o statistical analysis, w h i c h enables the
researchers t o assess the p r o b a b i l i t y that the independent variable d i d have a n
effect o n the dependent variable.
T h e study is p a r t i c u l a r l y interesting t o students o f research methods,
because the researchers d o n o t attempt t o gloss over or m i n i m i z e the p r a c t i c a l
dif&culties w h i c h emerged i n the course o f the s t u d y , ñ o r the interpretive
problems associated w i t h the unexpected outcomes o f the s t u d y . T h e fact that
p l a c e m e n t a n d standardised test data were used, rather t h a n a l i s t e n i n g pre-
test, means that w e s h o u l d probably v i e w this s t u d y as a q u a s i - e x p e r i m e n t
rather t h a n a true e x p e r i m e n t . In their paper, the researchers a c k n o w l e d g e
the c o m p l i c a t i o n s that the use o f these test data occasioned - f o r e x a m p l e ,
some o f the subjects w e r e tested several m o n t h s after the others, w h i c h m a d e
t h e i r pretest scores h i g h l y suspect, a n d these subjects were subsequently
e x c l u d e d f r o m the research. Ideally, the researchers s h o u l d have c o n s t r u c t e d
a test t o operationalise the construct ' l i s t e n i n g c o m p r e h e n s i o n ' a n d used this
test, o r versioñs'of it, for b o t h pre- a n d posttesting purposes.
I have already discussed the i m p o r t a n c e o f constructs i n research a n d the
need f o r researchers t o operationalise t h e i r constructs. A s w e have seen, c o n -
The experimental method
structs are unobservable qualities w h i c h are assumed t o u n d e r l i e observable
b e h a v i o u r . O p e r a t i o n a l i s i n g a c o n s t r u c t means d e f i n i n g i t i n terms o f observ-
able b e h a v i o u r . In the case o f C h a u d r o n a n d R i c h a r d s , the k e y c o n s t r u c t is
' l i s t e n i n g c o m p r e h e n s i o n ' , a n d the researchers operationalise the construct i n
terms o f the a b i l i t y t o answer true/false, múltiple choice, a n d cloze questions
about the l i s t e n i n g text. W h i l e the use o f such tests has been criticised as i n a d -
equate as a measure o f o v e r a l l c o m p r e h e n s i o n (see, f o r e x a m p l e , A s l a n i a n
1985), this type o f e x p e r i m e n t a l research m u s t necessarily rest o n such mea-
sures. In defence o f C h a u d r o n a n d R i c h a r d s , i t s h o u l d be p o i n t e d o u t t h a t
t h e y attempted t o strengthen the i n t e r n a l v a l i d i t y o f the research b y u s i n g
n o t one b u t three measures o f c o m p r e h e n s i o n . T h e i n t e r n a l v a l i d i t y m a y have
been strengthened f u r t h e r by s u p p l e m e n t i n g the q u a n t i t a t i v e data w i t h q u a l -
i t a t i v e d a t a , such as f o l l o w - u p i n t e r v i e w data o f the type e m p l o y e d b y
Aslanian.
T h e o t h e r n o t e w o r t h y aspect o f the s t u d y is the w a y i n w h i c h the research-
ers dealt w i t h the u n e x p e c t e d finding t h a t the c o m b i n e d m i c r o - m a c r o versión
w a s i n f e r i o r t o the m a c r o versión. T h e researchers considered a n d dismissed
the n o t i o n that the groups h e a r i n g these passages w e r e less p r o f i c i e n t , a n d also
the possibility that the q u a l i t y o f the r e c o r d i n g for that p a r t i c u l a r versión was
i n f e r i o r . T h e o n l y e x p l a n a t i o n they c o u l d c o m e u p w i t h was that a d d i n g the
m i c r o m a r k e r s increased the listeners' a t t e n t i o n requirements w i t h o u t a d d i n g
s e m a n t i c i n f o r m a t i o n , a n d that this detracted f r o m the effect o f the m a c r o
m a r k e r s . W h i l e this e x p l a n a t i o n m a y n o t be p a r t i c u l a r l y c o n v i n c i n g , i t is t h e
researchers' best guess at the probable cause. Research, even w h e n carried o u t
i n c o n t r o l l e d settings, o f t e n results i n some o u t c o m e s w h i c h are c o u n t e r - i n t u -
i t i v e i n the l i g h t o f other o u t c o m e s a n d therefore difficult t o a c c o u n t f o r . T h i s
is o n e such finding. It underlines the fact that e x p e r i m e n t a l researchers, n o
less t h a n those engaged i n n a t u r a l i s t i c , interpretive i n q u i r y , have t o interpret
a n d a c c o u n t for their d a t a . W i t h h i n d s i g h t , i t m i g h t have been wise t o c o n -
d u c t a p i l o t s t u d y . S u c h a s t u d y c o u l d have revealed some o f the problems
w h i c h emerged (such as problems w i t h t h e m i c r o versión), a n d thereby ena-
bled t h e researchers t o strengthen their major i n v e s t i g a t i o n .
Conclusión
T h e f o r m a l e x p e r i m e n t a n d its variants, the pre-experiment a n d quasi-exper-
i m e n t , are i m p o r t a n t research tools i n language s t u d y , a n d they have a d d e d
s i g n i f i c a n t l y t o o u r k n o w l e d g e o f language l e a r n i n g , teaching, a n d use.
E x p e r i m e n t s are designed t o collect d a t a i n such a w a y that threats t o the
r e l i a b i l i t y a n d v a l i d i t y o f the research are m i n i m i s e d . E x p e r i m e n t a l research-
ers are p a r t i c u l a r l y c o n c e r n e d w i t h the issue o f e x t e r n a l v a l i d i t y , a n d the for-
m a l e x p e r i m e n t is specifically designed t o enable the researcher to extrapólate
the o u t c o m e s o f the research f r o m the sample t o the broader p o p u l a t i o n . In
47
134 Second Language Research Methods
References
Bailey, K . M . 1981. 'Competitiveness and anxiety in adult second language
learning: Looking at and through the diary studies' in H . W . Seliger, and
M . H . Long (eds.): Classroom Oriented Research in Second Language
Acquisition. Rowley, Mass.: Newbury House.
Brown, R. 1973. A First Language. Cambridge, Mass.: M I T Press.
Cohén, A . and Hosenfeld, C , 1981. 'Some uses of mentalistic data in
second language research.' Language Learning 31/2:285-314.
Ericsson, K . A . and Simón, H . A . 1984. Protocol Analysis: Verbal Reports
as Data. Cambridge, Mass.: M I T Press.
Creen, J . L . and Wallatt C . (eds.) 1981. Ethnography and Language in
Educational Settings. N o r w o o d , N . J . : Ablex.
Jacob, E . 1987. 'Qualitative research traditions: A review.' Review of
Educational Research 57/1:1-50.
K a m i l , M . L . , Langer, J. A . , and Shanahan, T . 1985. Understanding
Reading and Writing Research. Boston: AUyn and Bacon.
Long, M . H . 1983. 'Inside the "black b o x " : Methodological ¡ssues in
classroom research on language learning' in H . W . Seliger and M . Long
(eds.) Classroom Oriented Research in Second Language Acquisition.
Rowley, Mass.: Newbury House.
Purcell, E. T . and Suter, R . W . 1980. 'Predictors of pronunciation accuracy:
A reexamination.' Language Learning 30/2: 271—287.
Scherer, A . C . and Wertheimer, M . 1964. A Psycholinguistic Experiment
in Foreign Language Teaching. N e w Y o r k : M c G r a w - H i l l .
Schinke-Llano, L . A . 1983. 'Foreigner talk in content classrooms' in H . W .
Seliger and M . Long: op. cit.
Seliger, H . W . 1989. 'Semantic transfer constraints on the production of
English passive by Hebrew-English bilinguals' in H . Dechert and M .
Raupach (eds.): Transfer in Language Production. N o r w o o d , N . J . :
Ablex.
Shulman, L . S. 1981. 'Disciplines of inquiry in education: an overview.'
Educational Researcher ]\xnd]n\y: 5—12.
Smith, J . K . and Heshusius, L . 1986. 'Closing down the conversation: The
end of the quantitative-qualitative debate among educational inquirers.'
Educational Researcher 15/1: 4—12.
Tarone, E. 1982. 'Systematicity and attention in interlanguage.' Language
Learning 32: 6 9 - 8 4 .
White, L . 1985. 'The acquisition of parameterized grammars: subjacency
in second language acquisition.' Second Language Research 1/1:1—17.
7 Research design: experimental
research
Science, after all, is fundamentally about process; learning why and
how things happen is the soul of our discipline. Y o u can't abandon
the search for cause in favor of a dry documentation of pattern. Y o u
must take risks of uncertainty in order to probé the deeper
questions, rather than stopping with sterile security.
(Stephen Jay Gould: 'Darwinism defined: The difference betvveen fact and
theory.' Discover. January 1987.)
Introduction
In C h a p t e r 6 we discussed qualitative a n d descriptive research
procedures i n second language a c q u i s i t i o n . In this chapter, we w i l l
discuss e x p e r i m e n t a l research. T h i s o r d e r reflects the current
interest a m o n g second language researchers i n descriptive investig-
a t i o n , a n d also reflects w h a t m a y be considered a ' n a t u r a l o r d e r ' o r
developmental sequence. T h a t is, qualitative and descriptive research,
being hypothesis-generating, often c u l m i n a t e i n testing those
hypotheses experimentally. T h e idea of a conceptual, developmental
sequence i n the research process was i m p l i e d i n o u r discussions i n
Chapters 2 and 3.
A s we n o t e d i n C h a p t e r 3 , research questions a n d hypotheses
may emerge f r o m different sources, such as o b s e r v a t i o n . A theory,
w h i c h is a set o f interrelated hypotheses, can then be d r a w n u p o n
for research w h i c h tests these hypotheses i n a c o n t r o l l e d context
such as an e x p e r i m e n t .
In this chapter, w e are concerned w i t h h o w the basic elements o f
an e x p e r i m e n t c a n be arranged i n o r d e r to give results w h i c h are
b o t h i n t e r n a l l y a n d externally v a l i d . T h e v a r i o u s aspects of v a l i d i t y
were discussed extensively i n C h a p t e r 5 . In this chapter, w e w i l l
discuss o n l y those designs that relate to experimental research. In
C h a p t e r 6, w e discussed n o n - e x p e r i m e n t a l approaches to research
(qualitative and descriptive research) and compared these approaches
to e x p e r i m e n t a l research i n relation to the research parameters
introduced in Chapter 2.
E x p e r i m e n t a l research is carefully constructed so that variables
can be c o n t r o l l e d a n d m a n i p u l a t e d . In terms o f the research
136 Second Language Research Methods Research design: experimental 137

parameters discussed i n C h a p t e r 2 , e x p e r i m e n t a l research is

2 T h e treatment
analytic a n d deductive. F i g u r e 7.1 s h o w s where experimental
research w o u l d be p l a c e d relative to the research parameters. T h i s refers to a n y t h i n g done to g r o u p s i n o r d e r to measure its effect.
T h e treatment is n o t a r a n d o m experience w h i c h the groups might
have, b u t a controlled a n d intentional experience, such as exposure
Experimental to a language teaching m e t h o d specially constructed for the
research experiment, o r materials presented u n d e r c o n t r o l l e d circumstances,
design say, i n a language l a b o r a t o r y . T r e a t m e n t s are the independent
1 Synthetic/hplistic < X > Analytic/ variable i n the research.
constituent
2 Heuristic 3 Measurement o r observation
^ > Deductive
3 Level of control and
manipulation: L o w < M e a s u r e m e n t o r o b s e r v a t i o n refers to h o w the effects o f the
X > High treatment w i l l be evaluated o r o b s e r v e d . T h e effects o f a treatment
4 Level of explicitness
m i g h t be evaluated by means o f a language test, a judgment, o r a
of data collection: L o w <•
H ^ High c o m m u n i c a t i v e task; o r p h y s i c a l measurements such as response
t i m e c a n be measured electronically o r spectrograms c a n be taken.
Figure 7.1 Experimental research and the research parameters F o l l o w i n g conventions established by C a m p b e l l a n d Stanley
(1963), the c o m p o n e n t s o f e x p e r i m e n t a l research designs w i l l be
The major components of experimental research s y m b o l i z e d as f o l l o w s :

A l l e x p e r i m e n t a l approaches i n v o l v e the c o n t r o l o r m a n i p u l a t i o n o f X = a n e x p e r i m e n t a l treatment such as a teaching m e t h o d , o r

the three basic c o m p o n e n t s o f the e x p e r i m e n t : the population,
treatment, a n d the measurement ofthe treatment. T h i s chapter w i l l
o n l y be concerned w i t h those aspects o f these c o m p o n e n t s relating
to the design o f the research, a n d n o t to questions o f statistical
analysis. T h e latter are dealt w i t h i n C h a p t e r 9.
1 T h e type and number of groups
E x p e r i m e n t a l research is c o n c e r n e d w i t h s t u d y i n g the effects o f
specified a n d c o n t r o l l e d treatments given to subjects u s u a l l y f o r m e d
i n t o g r o u p s . G r o u p s c a n be f o r m e d especially f o r the purposes o f
the experiment o r they c a n be ' n a t u r a l ' i n the sense that they
already exist p r i o r to the research. N a t u r a l g r o u p s often c o r r e s p o n d
to ' n o m i n a l ' data categories such as those discussed i n C h a p t e r 5 ,
f o r e x a m p l e , males, Spanish speakers, subjects w h o have learned a
language f o r at least five years, o r the class i n r o o m 4 0 5 . T h e
researcher has to decide w h e t h e r to use p r e - e x i s t i n g groups, as they
o c c u r i n nature, o r t o construct t h e m f o r the experiment. T h i s
difference between n a t u r a l g r o u p s a n d those f o r m e d specifically f o r
an e x p e r i m e n t is one o f the features that distinguishes true
e x p e r i m e n t a l designs f r o m q u a s i - e x p e r i m e n t a l designs.
the exposure to specially constructed materials, a n d so o n .
O = o b s e r v a t i o n o r measurement o f the effects o f the treatment.
R = r a n d o m i z a t i o n , o r the r a n d o m assignment o f subjects t o
g r o u p s i n o r d e r to c o n t r o l f o r extraneous variables.
E x p e r i m e n t a l research design consists p r i m a r i l y o f v a r i o u s w a y s o f
o r g a n i z i n g a treatment ( X ) a n d a n o b s e r v a t i o n o r measurement
( O ) . W e w i l l e x a m i n e a representative sample o f possible research
designs a n d discuss their advantages a n d disadvantages.
E x p e r i m e n t a l research designs m a y be g r o u p e d i n t o several
categories, a l l o f w h i c h have a n u m b e r o f v a r i a t i o n s , depending o n
the c o n d i t i o n s under w h i c h the research is being c o n d u c t e d . T h e
discussion hete w i l l be l i m i t e d t o single group designs, i n w h i c h
research m i g h t be c o n d u c t e d w i t h o n l y one g r o u p ; control group
designs, i n w h i c h one g r o u p receives a treatment w h i l e the other,
representíng the same p o p u l a t i o n as the e x p e r i m e n t a l subjects, does
n o t receive a treatment; factorial designs, w h i c h a l l o w f o r the
investigation o f a n u m b e r o f independent variables at the same
t i m e ; a n d quasi-experimental designs, i n w h i c h e x p e r i m e n t a l
research is c o n d u c t e d i n situations w h i c h c a n n o t be completely
controlled or manipulated.
138 Second Language Research Methods
A l l o f the designs discussed i n this chapter have numerous
v a r i a t i o n s that d e p e n d o n the specific c o n d i t i o n s under w h i c h the
research is c o n d u c t e d . It is i m p o r t a n t , therefore, for the reader to
u n d e r s t a n d the general principies o f e x p e r i m e n t a l design i n order to
be able to construct satisfactory v a r i a t i o n s of these designs dictated
by different c o n d i t i o n s , research questions, a n d so o n .
Single group designs
1 One-shot design: X O (Pilot study design)
T h i s is e x p e r i m e n t a l research design at its most basic. In this design,
a single treatment is given to a single g r o u p or i n d i v i d u a l . T h e
g r o u p o r i n d i v i d u a l is then observed, tested, or measured.
T h i s design m i g h t also be referred to as the ' p i l o t s t u d y ' design
because it does not c o n t r o l f o r any o f the extraneous variables
discussed i n C h a p t e r 5 . T h e design is p r o b l e m a t i c a n d , for m a n y
researchers, it is never seriously considered as a viable design for
e x p e r i m e n t a l research. T h e m a i n p r o b l e m w i t h this design is that
there is n o w a y o f k n o w i n g the characteristics o f the g r o u p or
i n d i v i d u a l before the treatment o r experience. H o w e v e r , the design
is useful as a means o f p i n p o i n t i n g w h a t to a v o i d i n experimental
research. It can also be used f o r p r e l i m i n a r y testing of instruments
o r ' r u n t h r o u g h s ' o f e x p e r i m e n t a l procedures.
Example
A second language teacher employs a new method for teaching the
communicative use of the target language. After three weeks of using this
new method, a test is administered to the class. The class performs well on
the test.
It is clear that there are serious p r o b l e m s w i t h this design, and
that claims made f o r findings u s i n g it must be made w i t h great
c a u t i o n . T h e discussion o n extraneous variables i n C h a p t e r 5
indicates that this design does not c o n t r o l f o r m a n y factors.
T h e r e are instances i n w h i c h the 'one-shot design' is a l l that is
available to the researcher because it is not possible to set u p
e x p e r i m e n t a l c o n d i t i o n s a n d c o n t r o l f o r extraneous variables. It
m i g h t be that there are n o t e n o u g h subjects for r a n d o m i z a t i o n i n t o
groups, o r it m a y be i m p o s s i b l e to find o u t w h a t the pre-treatment
c o n d i t i o n s were. A n e x a m p l e o f such a situation w o u l d be the study
of language a c q u i s i t i o n by transient p o p u l a t i o n s whóse past
educational h i s t o r y is u n o b t a i n a b l e . In such cases, the researcher is
líníited to this type o f design.
Because o f its l i m i t a t i o n s , it is i m p o r t a n t to q u a l i f y the findings o f
Research design: experimental 139
'one-shot' research i n terms o f the factors w h i c h affect internal and
external v a l i d i t y . T h i s research design is more appropriate to p i l o t
studies i n w h i c h the researcher wishes to try o u t instruments or
treatments before r u n n i n g a full-scale experiment.
2 O n e group pre-test + post-test: O I X 02
T h i s design attempts to use the subjects as their o w n controls and to
elimínate the need f o r a c o n t r o l g r o u p design. T h i s design is
sometimes referred to as a 'repeated measures' design because
subjects are observed o r measured t w i c e o n the dependent variable.
T h e design is efficient because it c o n t r o l s a n u m b e r of extraneous
variables w h i c h can affect the h o m o g e n e i t y of subjects w h e n more
t h a n one g r o u p is i n v o l v e d . T o some degree, the design also
controls for a t t r i t i o n o r loss o f subjects. Since the same group is
used for b o t h pre-test a n d post-test, it does not need to be matched
to another g r o u p .
U n f o r t u n a t e l y , there are a n u m b e r o f disadvantages to this design
that the researcher must be a w a r e of. O n e o f the p r i m a r y problems
is that there is n o certainty that the possible differences that appear
i n 0 2 are the result o f treatment X ; they m i g h t simply be changes
that w o u l d have taken place a n y w a y . (See the discussion o f history,
a t t r i t i o n , a n d m a t u r a t i o n i n C h a p t e r 5.) F o r example, i n classroom
research where the independent v a r i a b l e m i g h t be a set of materials
o r a language teaching m e t h o d , there is n o w a y o f k n o w i n g whether
changes between the before a n d after treatment measures ( O I and
0 2 ) were the result o f i n c i d e n t a l exposure to language material o r
to some other experience that the subjects m a y have h a d . In the case
o f second language learners, i n c i d e n t a l exposure to the second
language outside the c l a s s r o o m m a y affect performance. In foreign
language contexts, this m a y be a less i m p o r t a n t factor because
i n c i d e n t a l exposure can m o r e easily be c o n t t o l l e d .
A n o t h e r possible disadvantage o f this design is that the pre-test,
O I , m a y sensitize the subjects to specific aspects o f the treatment,
X , a n d thus c o n f o u n d w h a t is measured by the post-test, 0 2 .
Example
In a study on the effects of second language instruction, the researcher
wishes to establish that the group has no knowledge of a grammatical
structure which will be taught in the lesson (X). In order to establish this,
two tests are constructed which specifically test for the target structure in a
discrete point test. One test will be used as the pre-test and the other will
become the post-test.
140 Second Language Research Methods
In this case, the pre-test m a y highhght o r d r a w attention to the
target structure. Because the g r o u p has been sensitized to this
structure by the pre-test, it m a y learn differently f r o m a group which
has n o t been sensitized i n this w a y . T h e pre-test hete acts not o n l y
to estabhsh a basehne n o r m for k n o w l e d g e of the language being
m e a s u r e d , b u t also acts as a p r e - i n s t r u c t i o n phase for the treatment
X . T h i s p r o b l e m m a y be m i t i g a t e d to some extent by u s i n g more
indirect measures to establish baseline norms of k n o w l e d g e before
the treatment. F o r e x a m p l e , a pre-treatment s p e a k i n g o r w r i t i n g
sample might reveal levéis o f knowledge, without d r a w i n g attention
to the language f o r m s w h i c h w i l l be taught d u r i n g the treatment
stage.
3 T i m e - s a m p l i n g designs: O I , 0 2 , 0 3 , O n . . . X , O n -I-1,
O n -H 2. . .
These are also referred to as 'time-series' designs because a n u m b e r
o f samples o r observations are taken over a p e r i o d o f time. T h e y
can be d i s t i n g u i s h e d f r o m n o n - e x p e r i m e n t a l l o n g i t u d i n a l research
because they have a c o n t r o l l e d treatment, X , inserted after a
n u m b e r o f observations o r measurements. T h e use o f this type of
design is a n o t h e r w a y o f o v e r c o m i n g the problems inherent i n the
previous design.
T h e procedure o f t a k i n g a number of measurements o r observations
o f the subject p o p u l a t i o n before a n d after the treatment a l l o w s us
to ascribe any changes i n the subjects' performance to the treatment
w i t h greater assurance. It a l l o w s the researcher to develop a n o r m
for the p o p u l a t i o n over t i m e a n d thus discount history as a possible
factor. A f t e r a series o f observations, a regular pattern o f change
s h o u l d emerge, thus r e v e a l i n g the a m o u n t of change that w o u l d be
expected as a result o f t i m e , i n c i d e n t a l exposure to other language
sources, a n d m a t u r a t i o n . A sudden o r noticeable difference
between the o b s e r v a t i o n (On) i m m e d i a t e l y preceding the treatment
and the o b s e r v a t i o n f o l l o w i n g treatment ( O n + 1) can then be said
w i t h m o r e confidence to result f r o m X . O t h e r variants o f this design
include a time-series i n w h i c h several treatments are p a i r e d w i t h
several observations o v e r time.
In this type o f design, the researcher does not have access to a
c o n t r o l g r o u p , a n d m u s t be content w i t h results o b t a i n e d f r o m one
g r o u p . Since this design also operares under real-life c o n d i t i o n s a n d
n o t necessarily w i t h groups specially constituted for the research, it
Research design: experimental 141
may also be c o n s i d e r e d one o f the ' q u a s i - e x p e r i m e n t a l ' designs
discussed b e l o w .
A s w e have n o t e d a n u m b e r o f times, history is one of the greatest
threats to i n t e r n a l v a l i d i t y . In the case o f second language
a c q u i s i t i o n , it is p o s s i b l e that certain changes i n language a b i l i t y
c o u l d be the result o f n a t u r a l d e v e l o p m e n t a l processes unrelated to
a specific treatment.
Example
A study wishes to investígate the effectiveness of teaching students the use
of relative clauses i n English. For the purposes of the research, a 'treatment'
is constructed consisting of a lesson which gives the formal features of
relative clause formation in English along with practice in using relative
clauses in speech and writing. The treatment is preceded by several in-class
writing assignraents ( O I , 0 2 , 0 3 ) spaced over several weeks. For each
writing assignment, the number of relative clauses in learners' writing is
tallied and categorized. After the treatment, several similar writing
assignments are given ( O n + 1 , O n + 2, O n H-3) and the number of
relative clauses are counted and categorized.
In s u m m a r y , c o l l e c t i n g d a t a over a n extended p e r i o d before a n d
after the treatment a l l o w s us to establish a n o r m a l pattern o f
p e r f o r m a n c e f o r the language o r a c q u i s i t i o n behavior under
investigation. T h i s a l l o w s the researcher to exelude the possible
i n t e r a c d o n o f i n c i d e n t a l exposure to language material outside the
c l a s s r o o m , o r any n a t u r a l d e v e l o p m e n t a l change w h i c h m a y take
place regardless o f i n s t r u c t i o n .
Designs using control groups
C o n d u c t i n g a n e x p e r i m e n t means that at least one independent
variable is m a n i p u l a t e d a n d its effect measured by some dependent
variable w h i l e other factors are c o n t r o l l e d i n various w a y s . In the
single g r o u p designs, the e x p e r i m e n t a l g r o u p acts as its o w n
c o n t r o l ; the c o m p a r i s o n is between its performance w i t h o u t
treatment a n d its p e r f o r m a n c e w i t h treatment. In the designs
discussed b e l o w , the c o m p a r i s o n w i l l be for the treatment effect
between t w o o r m o r e g r o u p s . I m p l i c i t i n the use of c o n t r o l groups is
the i m p o r t a n t a s s u m p r i o n that the c o n t r o l g r o u p represents the
same p o p u l a t i o n as the e x p e r i m e n t a l g r o u p : it is as if w e are
c o m p a r i n g the same individuáis w i t h a n d w i t h o u t treatment. F o r
this reason, the m u l d - g r o u p designs are concerned w i t h measures to
standardize the g r o u p s being c o m p a r e d , so that claims for
 142 Second Language Research Methods
difference i n the p e r f o r m a n c e o n the dependent variable (O) have
b o t h i n t e r n a l a n d external v a l i d i t y .
1 Static group o r 'pre-experimental' design
X 01
OI
T h i s w o u l d seem to be a n ' e c o n o m i c a l ' design because it allows
US to u t i l i z e g r o u p s w h i c h already exist, such as classes f o u n d i n a
s c h o o l . In this design, a treatment is administered to one g r o u p a n d
its p e r f o r m a n c e is c o m p a r e d w i t h another, seemingly equivalent
g r o u p w h i c h has received n o treatment. T h e p r o b l e m w i t h this
design is that there is n o w a y o f k n o w i n g whether the groups are, i n
fact, equivalent before ' X ' . T h a t is, differences i n p e r f o r m a n c e o n
the dependent v a r i a b l e (O) m a y be due to intrinsic g r o u p
differences such as the fitst language b a c k g r o u n d , sex, exposure to
the second language, the time o f day d u r i n g w h i c h i n s t r u c t i o n takes
place, the level o f m o t i v a t i o n o f the groups, a n d the effects o f
different teachers o n g r o u p response.
T h i s design m a y also be considered ' q u a s i - e x p e r i m e n t a l ' (see
below) a n d is m o s t a p p e a l i n g f o r second language experiments
c o n d u c t e d i n s c h o o l e n v i r o n m e n t s because it requires the least
a m o u n t o f d i s r u p t i o n o f s c h o o l routines. It does n o t require the
reassignment o f subjects to groups different f r o m those i n w h i c h
they are already f o u n d a n d it requires n o rearrangements of
schedules o r reassignment o f teachers. These apparent advantages,
h o w e v e r , c a n also be sources o f i n v a l i d i t y i f variables are not
c o n t r o l l e d . F o r e x a m p l e , i f the same instructor teaches b o t h g r o u p s ,
teacher variables m a y be better c o n t r o l l e d ; o r i f the groups
c o m p a r e d receive their i n s t r u c t i o n at about the same time o f d a y ,
fatigue m a y be d i s c o u n t e d as h a v i n g a variable effect.
O n e w a y to a v o i d some o f these problems w h e n u s i n g a static
design is f o r the experimenter to m a t c h subjects i n the t w o groups
for v a r i o u s characteristics such as placement test scores, sex, first
language, a n d teacher rankings to make the groups more comparable.
M a t c h i n g is a w a y o f increasing the c o m p a r a b i l i t y o f the groups
w h e n subjects c a n n o t be r a n d o m l y assigned. T h u s , the v a r i a d o n o n
the static design w i l l l o o k l i k e the pre-test designs discussed b e l o w ,
but w i t h o u t r a n d o m i z a t i o n . T h i s w i l l n o t solve a l l o f the p r o b l e m s
discussed above but it w i l l give some indication o f the comparability
o f the stadc g r o u p s .
Research design: experimental 143
Group A : , . X OI
Group B: '"^^'^^^"^ = O T
A variation o n the c o n t r o l group matching design
A v a r i a t i o n o n the c o n t r o l g r o u p m a t c h i n g design can be f o u n d i n
the study c o n d u c t e d b y L i g h t b o w n and L i b b e n (1984), i n w h i c h the
use o f cognates b y f r a n c o p h o n e E S L learners w a s c o m p a r e d w i t h a
c o n t r o l g r o u p o f native speakers. In this study, the treatment
consisted o f b o t h c o n t r o l a n d e x p e r i m e n t a l subjects v i e w i n g a film
and then t a k i n g a m o d i f i e d cloze test i n w h i c h target concepts were
left b l a n k . T h e test w a s then f o l l o w e d b y an acceptability task i n
w h i c h the same text w a s presented w i t h the blanks filled i n , some
w i t h acceptable cognates a n d others w i t h unacceptable cognates.
The acceptability p e r f o r m a n c e o f the E S L learners w a s c o m p a r e d
w i t h that o f the native anglophones ( N S ) . In this case, the cloze
acted as a f o r m o f pre-test t o establish that the E S L learners d i d
have the concepts w i t h w h i c h to judge acceptability later. T h e
design, therefore m i g h t be described as f o l l o w s :
^ E S L cloze ( O I ) acceptability ( 0 2 ) ^ ^
film ( X ) ^ ^ comparison
^ N S cloze ( O I ) acceptability ( 0 2 ) ^ ^
2 Pre-test/post-test w i t h r a n d o m i z a t i o n o f groups
.Group A ' : O I X 02
G r o u p A —» R
• Group A": O I — 02
R a n d o m i z a t i o n is a p r o c e d u r e w i t h w h i c h to reduce the a m o u n t
of systematic error that m i g h t result f r o m biases i n the assignment
of subjects t o g r o u p s . It also provides better c o n t r o l o f variables
that c o u l d affect i n t e r n a l v a l i d i t y . B y r a n d o m i z i n g , w e are c l a i m i n g
that any effects o f extraneous variables occur b y chance a n d that
chance is e q u a l l y d i s t r i b u t e d between b o t h groups.
A n i m p o r t a n t advantage o f r a n d o m i z a t i o n is that it provides the
o p t i o n o f n o t u s i n g a pre-test as a m e t h o d o f m a k i n g groups m o r e
c o m p a r a b l e . If subjects are assigned to groups r a n d o m l y , it is safe
to assume that the effect o f m a n y subject variables is spread evenly
t h r o u g h o u t the t w o g r o u p s . T h i s is m o r e likely to be the case w h e n
 144 Second Language Research Methods
the size o f the g r o u p s i n the experiment is large. T h e larger the
n u m b e r o f subjects i n each g r o u p , the more l i k e l y it is that subject
variables w i l l negate each other t h r o u g h the r a n d o m i z a t i o n process.
(See also 'Size o f subject p o p u l a t i o n ' i n C h a p t e r 5.)
T h e f o l l o w i n g e x a m p l e demonstrares another possible threat to
i n t e r n a l v a l i d i t y w h i c h can be solved by r a n d o m i z e d assignment of
subjects.
Example
Volunteers are solicited for a study on the effects of computer-assisted
instruction on second language achievement. From the population of
volunteers, subjects are assigned to the computer-assisted group first and to
the control group receiving no computer-assisted instruction second.
(Reversing the order of assigning volunteers will not change the effect.)
E v e n t h o u g h the assigning o f subjects to the treatment o r the
c o n t r o l g r o u p w o u l d seem to be r a n d o m , it is l i k e l y that those
v o l u n t e e r i n g first are different f r o m those w h o m a y have been
cajoled i n t o p a r t i c i p a t i n g i n the study. T h e order o f assignment m a y
be a reflection o f m o t i v a t i o n at some level a n d is also l i k e l y to affect
the results o f the study.
Sometimes the p r o b l e m o f h o w to assign volunteers c a n n o t be
overeóme. In a study reported by E u b a n k (1987) o n the a c q u i s i t i o n
o f n e g a t i o n i n G e r m á n as a foreign language, o n l y six subjects
v o l u n t e e r e d f r o m a t o t a l o f a p p r o x i m a t e l y three h u n d r e d students
registered f o r a b e g i n n i n g Germán course, even t h o u g h a l l o f the
p o t e n t i a l subjects h a d been offered m o n e y for p a r t i c i p a t i o n . G i v e n
such s m a l l n u m b e r s a n d the l i k e l i h o o d that these six stalwarts were
h i g h l y m o t i v a t e d , the results of such a study must be v i e w e d w i t h
caution.
W i t h a sufficient n u m b e r of subjects, the p r o b l e m o f biases i n
levéis o f m o t i v a t i o n o r other factors can be c o n t r o l l e d , a n d internal
v a l i d i t y heightened, by assigning subjects to groups by u s i n g a
r a n d o m i z a t i o n (R) p r o c e d u r e . T h i s can be done t h r o u g h the use of
a table o f r a n d o m numbers f o u n d i n the back o f statistics b o o k s , by
d r a w i n g ñames o u t o f a hat, o r by any other means w h i c h w i l l
guarantee that subjects are not assigned o n the basis o f some
conscious o r u n k n o w n bias.
In the pre-test/post-test w i t h r a n d o m i z a t i o n design, it is assumed
that a l l subjects are members of some g r o u p A , w h i c h is i n t u r n a
representative sample o f the greater p o p u l a t i o n o f second language
l e a r n e r s . " T h a t is, G r o u p A , f r o m w h i c h the e x p e r i m e n t a l a n d
control groups w i l l be d r a w n , is itself representative. The population
Research design: experimental 145
of G r o u p A is fed t h r o u g h a r a n d o m i z a t i o n process (R), a n d
members are r a n d o m l y assigned to either G r o u p A ' o r G r o u p A " .
O n c e the t w o g r o u p s are f o r m e d , the pre-test, O I , is administered
to b o t h .
T h e r e are v a r i a t i o n s o n Design 2. F o r e x a m p l e :
<
^Ol
R' ^
—OI
shows that r a n d o m i z a t i o n can elimínate the p r o b l e m o f pre-testing.
T h i s design, w i t h o u t pre-testing, is useful w h e n assumptions can be
made a b o u t the baseline k n o w l e d g e o f the p o p u l a t i o n f r o m w h i c h
the g r o u p s are f o r m e d .
F o r e x a m p l e , i f w e w i s h to test the effect of l a b o r a t o r y t r a i n i n g
on the a b i l i t y to p r o n o u n c e second language sounds, w e m a y w i s h
to start w i t h a p o p u l a t i o n defined as beginners and assign t h e m
r a n d o m l y to e x p e r i m e n t a l o r c o n t r o l groups. In this w a y we c a n
a v o i d g i v i n g t h e m a pre-test o f p h o n o l o g i c a l a b i l i t y , w h i c h m i g h t
sensitize t h e m to the treatment itself.
Factorial designs
F a c t o r i a l designs are s i m i l a r to the true e x p e r i m e n t a l designs
discussed above a n d i n c l u d e all of the elements f o u n d i n those
designs, such as g r o u p i n g by r a n d o m i z a t i o n , pre- a n d post-testing,
and treatments. T h e difference is that the effects o f several
independent variables m a y be tested at the same time.
In s i m p l e e x p e r i m e n t a l designs, the a d d i t i o n a l variables are
referred to as extraneous variables. (See C h a p t e r 5.) These variables
must be c o n t r o l l e d i f a n effect is to be attributed to a specific
independent v a r i a b l e . H o w e v e r , i n factorial designs, m o r e t h a n one
variable m a y be treated as an independent v a r i a b l e .
Example
In a study to test for the effect of language laboratory training on
pronunciation, it is also decided to measure for the effect of such training
on learners of different proficiency levéis, as well as the effect of different
kinds of language laboratory practice.
In this e x a m p l e , w e are h y p o t h e s i z i n g that the effect o f practice
versus n o practice i n the language l a b o r a t o r y c o u l d be a f u n c t i o n o f
both the p r o f i c i e n c y level of the learner a n d the type of practice
a d m i n i s t e r e d i n the l a b o r a t o r y .