QUANTITATIVE RESEARCH

IN TESL
AND APPLIED LINGUISTICS
(PART 1)

Sugeng Hariyanto

Program Pascasarjana
Universitas Islam Malang
2010

Goals of the Course

What is your first reaction when you here the word "research"?
Excited, happy, sad, or frightened?
Some people are frightened with the word and show some allergic
symptoms. This short introduction will help you understand the basic
concepts of quantitative research in TESL and Applied Linguistics.
At the end of the semester, you should be able to:
1. Mention three main paradigms (basic principles) in research
2. Choose appropriate quantitative research designs for given research
topics.
3. choose the appropriate sampling procedure for given topics
4. design the measurement instruments for a given research topic,
5. find a research topic
6. do a literature review, and
7. write a quantitative research proposal in TESL or Applied Linguistic
field of study.
In addition, you will also learn some basic terminology used in the research
field.
Oops, almost forget. The tile of this course is

QUANTITATIVE RESEARCH

Chapter 1

Definition of Research1

In order to plan and carry out research, it is necessary for you

to know what we mean by research, as well as in the
specialized fields of language teaching and language
acquisition.

Research is an ORGANIZED
and SYSTEMATIC way of
FINDING ANSWERS to
QUESTIONS.
SYSTEMATIC because there is a definite set of procedures and steps which you will
follow. There are certain things in the research process which are always done in order to
get the most accurate results.
ORGANIZED in that there is a structure or method in going about doing research. It is a
planned procedure, not a spontaneous one. It is focused and limited to a specific scope.
FINDING ANSWERS is the end of all research. Whether it is the answer to a hypothesis
or even a simple question, research is successful when we find answers. Sometimes the
answer is no, but it is still an answer.
QUESTIONS are central to research. If there is no question, then the answer is of no use.
Research is focused on relevant, useful, and important questions. Without a question,
research has no focus, drive, or purpose.

http://linguistics.byu.edu/faculty/henrichsenl/researchmethods/RM_1_01.html

Chapter 2

Kinds and Paradigms of Research

Research can be seen from different sides.
First, research can be seen from how we will use the finding. Seen from this
perspective, research can be classified into three kinds2:
BASIC RESEARCH is concerned with knowledge for the sake of theory. Its design is
not controlled by the practical usefulness of the findings.
APPLIED RESEARCH is concerned with showing how the findings can be applied or
summarized into some type of teaching methodology.
PRACTICAL RESEARCH goes one step further and applies the findings of research to
a specific "practical" teaching situation.
Practical research may be based on theory that came from previously done basic research.
Or, theory may be generated by the combination of results from various practical research
projects. The

Research is like a plant

that grows and grows
and grows and grows...
When it is grown
it throws off seeds of all types
(basic, applied, and practical),
which in turn sprout and
create more research projects.
The process continues with
all of the new research 'plants'
throwing off seeds, creating
additional, related research projects
of various types.
Soon there is a body of basic, applied, and practical
research projects related to similar topics.
And the process goes on and on...
2

http://linguistics.byu.edu/faculty/henrichsenl/researchmethods/RM_1_04.html

Second, from how you conduct the research, research can be classified as: quantitative,
qualitative research, and action research. To understand this better, we will see the basic
principles behind them.
Research paradigm
Classifying research paradigms
Guba and Lincoln (1994) state that the basic beliefs that define a particular research
paradigm may be summarised by the responses given to three fundamental questions:
1. The ontological question i.e. what is the form and nature of reality
2. The epistemological question i.e. what is the basic belief about knowledge (i.e. what
can be known)
3. The methodological question i.e. how can the researcher go about finding out
whatever s/he believes can be known.
In this short summary, we will only see the differences of in terms of methodological
perspective.

Questions for analysing

paradigms
Role of
researcher

Methodological questions

Role of values

Methods

Type of studies

Positivism
Objective, independent
from the subject
Investigator often
controls the
investigated

Science is value-free
Values have no place
in research must
eliminate all bias

Empirical
Structured and
replicable observation
Quantification /
measurement
Experimental directly
manipulate variables
and observe
Survey studies
Verification of
hypotheses
Statistical analysis
Quantitative descriptive
studies

Research paradigms
Interpretivism
Co-creator of meaning
Brings own subjective
experience to the
research
Tries to develop an
understanding of the
whole and a deep
understanding of how
each part relates and is
connected to the whole

Values are an integral part

of social life no values
are wrong, only different

Unstructured observation
Open interviewing
Discourse analysis

Try to capture insider

knowledge

Field research, conducted

in natural settings in order
to collect substantial
situational information

Critical Theory
Adopts role of facilitator
encouraging the
participation and
involvement of the
subjects who become
partners in the research
process

Facts can never be isolated

from values
Values of the researcher
influence the research
Participatory action
research
Dialogical methods which
encourage dialogue
between researcher and
researched

To put it simply, see the following graph:

Positivism

Interpretivism

Critical
theory

Quantitative
research

Qualitative
research

Action
research

From now on, we will focus our discussion on QUANTITATIVE

RESEARCH. Lets now see the most simple way to differentiate
quantitative and qualitative research.
Table 1 : Comparison of features of Quantitative and Qualitative approaches to
research3

Quantitative

Qualitative

Both are systematic in their approach

Objective

Subjective

Deductive

Inductive

Generalisable

Not generalisable

Numbers

Words

Systematic = follow certain procedure

Objective = does not involve personal opinion and no bias; not involve the
researcher in the situation of the research
Subjective = involve personal opinion/judgment or subjective element, may
involve the researcher in the situation of the research
Numbers = rely on NUMBERS to show evidence and come to the conclusion
Words = rely on WORDS to show evidence and come to the conclusion
A quantitative study, measuring how long a student practice certain English
pronunciation a day, can be purely objective.
A qualitative research may want to discover how students feel about their regular
English pronunciation practice; the researcher has to come into contact with the
3

http://www.fortunecity.com/greenfield/grizzly/432/rra2.htm

students and make judgments from the students answers.

Quantitative research is inclined to be deductive.
In other words it tests theory.
This is in contrast to most qualitative research which tends to be inductive.
In other words it generates theory.
Using the pronunciation exercise above, the quantitative approach might test the
hypothesis that "regular practice improves English pronunciation.
A qualitative approach which explores the feelings of the students who do the
practice might generate the theory "whether daily pronunciation practice increase
the psychological burden and/or how the psychological burden can be handled."
Quantitative designs of research tend to produce results that can be generalized.
Using our pronunciation practice, at least the results of the quantitative study
tend to hold true for the department under consideration (if the sampling
procedure is appropriate).
However, qualitative studies tend to produce results that cannot be generalized
because there is no ideal sampling procedure.
Therefore, objectivity, deductiveness, generalizability and numbers are
features often associated with quantitative research.
Reference:
Guba EG and Lincoln YS (1994) Competing paradigms in qualitative research. Ch 6
In: Denzin and Lincoln (1994) Handbook of Qualitative Research. Sage
Publishers. USA.

Chapter 3

Types of Quantitative Research Design

Research design simply refers to the plan for carrying out the research. Quantitative
research designs can be classified into two main designs: experimental designs and nonexperimental or descriptive designs.

1.
2.
3.
4.

Experimental research
Ex-post facto research
Correlational research
Survey research

Descriptive
research

EXPERIMENTAL RESEARCH DESIGNS

Experimental designs refer to the plan of research that involves a treatment.
Experimental designs can further classified as pre-experimental, true experimental, or
quasi-experimental. They are distinguished based on the random selection of subjects,
the degree of control applied by the researcher and whether randomization is done or
not. Random selection of subjects refers to the random sampling in picking up subjects
from the population. Degree of control refers to the degree of the researcher control
over other variables that are not investigated but may influence the result of experiment.
Randomization refers to the random procedure in assigning subjects into the control and
experimental groups. See the table below:
Design
Pre-experimental
True experimental
Quasi-experimental

Control of other Randomization

variables
NO
NO
YES
YES
YES
NO

If the control over other influential (extraneous) variable is not done and randomization
is not done, the design is called pre-experimental. If the control over extraneous
variables and randomization are done, it is true-experimental. If the control over the
extraneous variables is done, but randomization is NOT done, it is called quasiexperimental.
Research design should have internal validity and external validity. A research design
has an internal validity if with the design the treatment can bring about changes in the
dependent variable. Some phenomena that threat the internal validity are: selection,
history, maturation, pre-testing, instrumentation, regression, and the interaction effect
among these factors. External validity refers to the generalizability of the result to the
other groups/settings, population or operational definitions.

If you choose to experimental research, one of the most important task to do is choose a
design that gives you the best combination of internal and external validity. The design
should be practical enough so you can actually do the research.
No particular type fits all situations. Certain situation often determines what is practical
or possible. Therefore, sometimes we may be forced to use a design which is not the
most ideal, but practical.
Within the realm of experimental research, there are three major types of design:

TRUE-EXPERIMENTAL
QUASI-EXPERIMENTAL
PRE-EXPERIMENTAL

Read on to know further.

TRUE-EXPERIMENTAL DESIGNS
TRUE-EXPERIMENTAL DESIGNS must employ the following:

Treatment (manipulation)
Random selection of subjects
Use of control groups
Random assignments to control and
experimental groups

The true-experimental designs can be classified as

randomized subjects posttest-only control group
design, randomized matched subjects posttest-only
control group design, randomized subjects pretestposttest control group design, Solomon three-group
design, Solomon four-group design, and simple
factorial designs.

To discuss about experimental

designs, we need to use some
terms and symbols.
X = independent variable or the
treatment.
E = experimental group (group that
receives the treatment)
C = control group (group that DOES
NOT receive the treatment)
S = subjects
R = randomization/random
assignment
Mr = matching of subjects

They can be illustrated as follows.

Randomized subjects, posttest-only control group design is illustrated as follows:
(R)
(R)

E
C

Y2
Y2

Randomized matched subjects, posttest-only control group

design is illustrated as follows:
(Mr)

E
C

To discuss about the

designs, we should
know several sampling
procedures.

Y2
Y2

Randomized subjects, prestest-posttest control group design is illustrated as follows:

(R)
(R)

E
C

Y1
Y1

Y2
Y2

Solomon three-group design is illustrated as follows:

(R)
(R)
(R)

E
C1
C2

Y1
Y1

Y2
Y2
Y2

Solomon four-group design is illustrated as follows:

(R)
(R)
(R)
(R)

E
C1
C2
C3

Y1
Y1

Y2
Y2
Y2
Y2

10

Simple factorial design is illustrated as follows:

Level
Level 1
Level 2

X1
Group 1
Group 2

X2
Group 3
Group 4

Advantages of the true-experimental design include:

Greater internal validity

Causal claims can be investigated

Disadvantages:

Less external validity (not like real world conditions)

Not very practical

11

QUASI-EXPERIMENTAL RESEARCH DESIGNS

QUASI-EXPERIMENTAL DESIGNS usually have some kind of control and
experimental group, but these groups probably weren't randomly selected.
The Quasi-experimental designs can be classified as non-randomized control group
pretest-posttest design, counter balanced design, one group time series, and control
group time series designs. See the illustration below.
Non-randomized control group, pretest-posttest design is illustrated as follows:
E
C

Y1
Y1

Y2
Y2

Counter balanced design is illustrated as follows:

1
2
3
4

X1
Group A
Group C
Group B
Group D

X2
B
A
D
C

X3
C
D
A
B

X4
D
B
C
A

One-group time series design is illustrated as follows:

Y1

Y2

Y3

Y4

Y5

Y6

Y7

Y8

From the illustration, it is known that measurement is done periodically (four times)
before and (four times) after the treatment. The result of the periodic measurements (8
times) may be similar to either one of these patterns.

Figure 1. Illustration of possible Outcome Patterns in a Time Design

12

Pattern A shows that the treatment X has an effect. Pattern B shows temporary effect of
the treatment. Pattern C shows no effect and pattern D shows no effect of the treatment
and the strong influence of extraneous variables.
Weakness: effect of history
Control group time series design is illustrated below:
E
C

Y1
Y1

Y2
Y2

Y3
Y3

Y4
Y4

Y5 Y6
Y5 Y6

Y7
Y7

Y8
Y8

Some advantages of the quasi-experimental design include:

Greater external validity (more like real world conditions)
Much more feasible given time and logistical constraints
Disadvantages:
Not as many variables controlled (less causal claims)

13

PRE-EXPERIMENTAL RESEARCH DESIGNS

PRE-EXPERIMENTAL DESIGNS do not have several criteria of true-experimental
research. They may not employ random selection in most cases and may just employ a
single group. This group receives the "treatment," there is no control group. Pilot
studies, one-shot case studies, and most research using only one group, fall into this
category.
The pre-experimental designs can be classified as one shot study, one-group pretestposttest design and static group comparison design. They can be illustrated as follows.
Type I: The One Shot Study
P
>
U
>

>

P = The general population

U = An unrandomized group
X = The intervention (the independent variable)
O = The post-intervention observation (the dependent variable)
The single shot case study is the simplest research design. In this design, some treatment
is administered to an unrandomized group arbitrarily selected from the general
population. Then the results are observed and recorded.
One-group pretest-posttest design is illustrated below:
Y1

Y2

If the sampling is considered, the illustration is as follows.

P

>

>

Y1

>

>

Y2

P = The general population

U = An unrandomized group
O1 = The pre-intervention observation
X = The intervention (the independent variable)
O2 = The the post-intervention observation (the dependent variable)
The single group pre-post design is similar to the single shot study but adds an
additional observation before the intervention to provide a control for comparison. In
this design, an unrandomized group is arbitrarily selected from the general population,
observed, and the data recorded. An intervention is administered and then the results are
observed and recorded.

14

Static group comparison design is like the above design but it has a control group to
provide a control for comparison. In this design, two unrandomized groups are
arbitrarily selected from the general population. One is designated as the experimental
group and the other as the control group. A treatment is administered to the
experimental group and then both groups are observed and recorded.
Static group comparison design is illustrated below:
E
C

Y2
Y2

If the sampling is considered, the illustration is as follows.

>
U1
>
X
>
O1
P
>
U2
>
O2
P = The general population
U1 = An unrandomized experimental group
U2 = An unrandomized control group
X = The intervention (the independent variable)
O1 = The post-intervention observation of the experimental group (the dependent
variable)
O2 = The post-intervention observation of the control group
The advantages are:

Very practical
Set the stage for further research

Disadvantages:

Lower validity

Reference:
Ary, Donald, et al. 2002. Introduction to Research in Education. Belmont: Wadswroth
Group
Henrichsen, Lynn. 1997. Taming the Research Beast: Research Methods in TESL and
Language Acquisition.
http://linguistics.byu.edu/faculty/henrichsenl/researchmethods/RN_0
_01.html

15

Chapter 4

Non Experimental Designs/Descriptive

Designs
Non experimental designs or sometimes called descriptive designs can be classified
into: ex post facto, correlational and survey designs.
Ex post facto research designs
Ex post facto research design or also called causal-comparative research design can be
used to test hypothesis concerning the relationship between two variables. Basically,
this is similar to experimental research. The difference is that treatment is not applied
because the phenomenon observed has occurred in the past and it is not possible to
apply treatment to the subject due to ethical reasons or other reasons. This design can
also be used to simply see the relationship between two variables.
An important thing to do is to make sure that the variable that happens before the other
variable is the independent variable.
The designs are
Design 1:
Independent variable
Dependent variable
Subjects are already known to be different > Test hypothesis concerning the possible
on the INDEPENDENT variable(s)
DEPENDENT variable
In this design, the researcher tries to see the consequence of the difference of
independent variable on the dependent variable. He starts by identifying the differences
in the independent variable, then collect the related data on the dependent variable and
text the hypothesis.
Example:
You know that some of your students got English lessons in elementary school (in grade
4 to 6) and others did not. So, you can identify this is the independent variable (previous
English lesson experience). Now you can collect the data on the development of their
English score from grade 7 to grade 9 as the dependent variable (English achievement
development).
Design 2:
Dependent variable
Independent variable
Subjects are already known to be different > Test hypothesis concerning the possible
on the DEPENDENT variable(s)
INDEPENDENT variable

16

In design two, the researcher starts with the fact that the subjects have differences
concerning dependent variable. Then, he collects data on the possible causes of the
difference (independent variable). Finally he tests the hypothesis.
Example:
You know that your students have different ability in pronunciation. Measure them as
the dependent variable. Then, look for the possible cause, for example, Arabic learning
experience as the independent variable (previous experience in learning a foreign
language). And you can relate the two variables.
Correlational Research Design (with two variables)
The design is very simple. However, please be careful. Correlational research design
can be used to prove the existence of causal relationship.

Variable 1
Collect data on Variable 1

Variable 2
Testing hypothesis
Collect data on Variable 2 Test hypothesis

Reference:
Ary, Donald, et al. 2002. Introduction to Research in Education. Belmont: Wadswroth
Group

17

Chapter 5

Survey Research Design

Survey is a research technique of gathering data by asking questions to a group of
individual (respondents).
Types of survey
Based on the focus and scope, survey can be identified as either census or sample
survey. Census covers the whole population of interest. A sample survey only study a
portion (sample) of this population of interest.
Based on the time dimension, surveys can be classified as longitudinal survey and crosssectional survey. A longitudinal survey is done several times at different points of time
on the same group of respondents. The time of measurement can be every year or every
two years, or other interval to see the changes over time. This type can be divided into
two: panel survey, trend study, cohort study. In panel survey, the same subjects are
measured over time. In trend study, the measurement are not done on the same subjects
over time, but on the sample taken randomly from the same group in different time. The
members of the group may be different. Because the sample was taken randomly, they
are considered equal. For example, survey of the SMU LAB students on smoking and
friendship is done every two years, with different sample because the previous students
must have graduated.
In cohort studies, the sample is taken from a group with the same membership. For
example, a survey is done on the topic of the value of independent work. The subjects
are the person experiencing modular learning experience. The data are collected when
the students where in grade 2, then two years after that, and every two other years up to
five times of data collection. The real sample can be different, but they must be graduate
of the same school.
Cross-sectional survey
This study asks questions to cross-section sample of the population at a single point in
time. For example, the research can ask Grade 7, Grade 8, and Grade 9 students about
their opinion of the importance of English at the same time rather than asking student of
Grade 7 this year and ask them again next year and next two year (longitudinal).
Data-gathering techniques include: personal interview, telephone interview, mailed
question, direct questionnaire, and internet questionnaire.
Assignment: Decide the design for your research idea. Of course, there is only one
design that you will use for your research.
Reference:
Ary, Donald, et al. 2002. Introduction to Research in Education. Belmont: Wadswroth
Group.

18