Sie sind auf Seite 1von 24

..

,-
.. �

,-

\ '.,.'
L...- ...

SCHOOL-BASED ORAL ENGLISH TEST: THE

BACKWASH EFFECT

NOR AZMI B. MOST AFA

NORMAH BT. OTHMAN

No rnb or Ps r otahan �\::._�t(CY1'5'>


Tr.rikh Pernlf'har __ . :'_l \.. _c'\ __

Ht1 '9'" .. ::':-' ,'I:\.. �C _

Bllangan N2:"I-.d _.�,_ _

KOD PENYELIDlKAN UPSI: 01-03-22-05

2007
ACKNOWLEDGEMENT

We would like to thank: the following people. Without their assistance, the completion of

this study would not have been possible.

To all the Principals, teachers and students from SMK Trolak, SMK Seri Kandi

and SM Methodist, we extend great appreciation for all their support and cooperation

extended to us in completing this study.

Most importantly, we would like to extend our greatest love, gratitude and

families for their support and patience throughout the completion of


appreciation to our

this study.

Dr. Nor Azmi b. Mostafa

Dr. Normah bt. Othman

1·"
Ur�N[n�;il F..:. -t
.......
.

,
ABSTRACT

This qualitative study investigated the backwash effect of the Oral English Test (OET)

conducted at school-based level in selected secondary schools in the Batang Padang

district ofPerak. More specifically, the study looked at how English as a Second

Language (ESL) teachers prepare students for the School-based Oral English Test in

school, how ESL teachers test Oral English in a school-based context, and examined the

backwash effects of the Oral English Test conducted at school level on classroom

instructions and student performance. A total of five teachers and five students from

three secondary schools in the district of Batang Padang were interviewed in the study.

Data analysis results produced several findings which indicated that the School-based

Oral English Test produced beneficial backwash on the ESL teachers' classroom

instructions and on the performance of the students in the test. Results also indicated

beneficial backwash effecting the students tested in relation to the content of the test. In

addition, the results also indicated that the conduct of the School-based Oral English Test

did produce beneficial backwash on the students in the form of excellent performance in

the oral test.

11
CONTENTS

ACKNOWLEDGEMENT i

ABSTRACT ii

LIST OF CONTENTS '" ,


iv

LIST OF TABLES '"


vi

1.0 INTRODUCTION 1

2.0 PURPOSE OF THE STUDY 1

3.0 OBJECTIVES OF THE STUDY 1

4.0 STATEMENT OF THE PROBLEM 2

5.0 SIGNIFICANCE OF THE STUDY 2

6.0 THEORETICAL FRAMEWORK 2

6.1 The Backwash


Effect , '"
'"
2

7.0 RESEARCH QUESTIONS 10

8.0 RESEARCH DESIGN ,


11

9.0 SAMPLE '" '"


,
11

10.0 INSTRUMENTATION 11

11.0 DATA COLLECTION 13

]2.0 DATA ANALYSIS 14

]3.0 RESULTS 15

13.1 Students' Academic Performance 15

13.2 Results from the Interviews of the


Students 15

III
13.3 Results from the Interviews of the
Teachers 20

14.0 CONCLUSIONS 32

14.1 Research Question 1: What is the extent of the backwash effects


produced by the School-based Oral English Test on the classroom
instructions by ESL teachers in preparing students for the
test? 32

14.2 Research Question 2: What is the extent of the backwash effects


produced by the content of the School-based Oral English Test on the
academic performance of students on the test? 34

14.3 Research Question 3: What is the extent of the backwash effects


produced by the conduct of the School-based Oral English Test on the
academic performance of students on the test? 30

15.0 RECOMMENDATIONS 37

16.0 REFERENCES 38

APPENDICES

A. LETTEROF
APPROVAL 39

B. INTERVIEW GUIDELINE 40

C. SCHOOL-BASED ORAL ENGLISH TEST GUIDELINES


AND ASSESSMENT FORMS 42

D. TRANSCRIPTIONS OF THE INTERVIEWS , 61

lV
LIST OF TABLES

Table Pages

12.1 Students' Grades on the School-based Oral English Test '"


14

v
LIST OF FIGURES

Figure Pages

6.1 Backwash: Positive and Negative '"


4

6.2 Backwash: Overt and Covert 7

VI
1.0 INTRODUCTION

According to Bachman (2000), the single most important consideration in both the

development of language tests and the interpretation of their results is the purpose or

purposes the particular tests are intended to serve. He points out that the major uses of

language tests are as sources of information for making decisions within the context of

educational programs, and as indicators of abilities or attributes that are of interest in

research on language, language acquisition, and language teaching.

In educational settings the major uses oftest scores are related to evaluation, or

making decisions about people or programs. In the context of research, the interpretation

of test results is of both theoretical and applied interest, whereby such interpretation can

assist in our understanding of the nature oflanguage proficiency, and may have

implications for language learning and language teaching.

2.0 PURPOSE OF THE STUDY

The purpose of this research is to investigate the backwash effect of the Oral English Test

conducted at school-based level in selected secondary schools in the district of Batang

Padang in Perak.

3.0 OBJECTIVES OF THE STUDY

The objectives of this research are:

(i). to investigate how English as a Second Language (ESL) teachers prepare students for

the School-based Oral English Test in school;

1
(ii). to investigate how ESL teachers test Oral English in a school-based context,

(iii).to examine the backwash effect of the School-Based Oral English Test on classroom

instruction and student performance

4.0 STATEMENT OF THE PROBLEM

The study investigates the problems faced by students with respect to the effects of

backwash on their performance on the school-based oral test, and the extent to which the

effects can be either positive and negative. This is to enable educators to ensure that tests

be treated as part of learning experiences for all involved.

5.0 SIGNIFICANCE OF THE STUDY

The Oral English for the English Sijil Pelajaran Malaysia (SPM) Examination was once

standardized throughout the country. It was handled by the Malaysian Board of

Examination. Since 2003 the Oral English Test for the English SPM Examination was

conducted at school-based level.

When the Oral English Test is conducted at school-based level, different schools

may have different contents of the test. Therefore, the standards of the tests may vary

from one school to another. However, the standards of the students' performance at the

national level are expected to be more or less the same because they are expected to

perform at their best level, especially when they reach the tertiary education. The

findings of this study may ensure that the student are within the expectations of tertiary

education.

2
6.0 THEORETICAL FRAMEWORK

6.1 The Backwash Effect

Backwash is usually described as the effect of testing on teaching and learning practices

(Messick, 1996). In literature, it is also referred to as "washback", "test impact", "test

feedback", "curriculum alignment", and also as "measurement-driven instruction". It is

not only determined by test writers, but also on the ways in which teachers and learners

interpret test requirements, and work with them in the classroom. Backwash is also

described as the effect of testing on the teaching and learning processes. According to

Shohamy, Donits-Schrnidt, and Ferman (1996), backwash is the connections between

testing and learning.

The washback or backwash effect has been generally defined as the effect of

assessment on teaching and learning. This effect can be either positive or negative

(Figure 6.1). Teachers and students tend to think of the negative effects oftesting such as

'test-driven' curricula and only studying and learning what they need to know for the test

(Coombe and Hubley, 2004). Positive washbacklbackwash assumes that testing and

Curriculum design are both based on clear course outcomes which are known to both

students and teachers/testers. If students perceive that tests are markers oftheir progress

towards achieving these outcomes, they have a sense of accomplishment. Thus, tests

must be part oflearning experiences for all involved (Coombe and Hubley, 2004).

According to Bachman (1990), positive backwash occurs when the assessment used

reflects the skills and content taught in the classroom. However, in many cases and

particularly in high stakes testing, the curriculum is driven by the assessment, leading to

3
negative backwash.

Figure 6.1: Backwash: Positive and Negative

BACKWASH
EFFECTS

POSITIVE NEGATIVE

While Gates (1995) considers backwash as the influence of testing on teaching

and learning, Messick (1996) also referred to backwash as the extent to which the

introduction and use of a test influences language teachers and learners to do

things they would not otherwise necessarily do. In this respect, teaching to pass the test

is necessary in the classrooms when the aim is to prepare students for a particular

examination.

According to Brown (1999), backwash can be analyzed into aspects of a

curriculum that negative backwash can affect and ways that positive backwash can be

fostered. Backwash, whether it is positive or negative, can be a potential advantage or

threat to language teaching curriculum because, through backwash, a test can steer a

curriculum in one direction or another, in terms of teaching, course content, course

characteristics, and class time, either with or against the better judgment of the

administrators, teachers, students, parents, etc. (Brown, 1999).

I
From the point of view of testing, thinking about backwash can help us to think

about test validity. Backwash becomes negative backwash when there is a mismatch

between the construct definition and the test, or between the content, (e.g. the

material/abilities being taught) and the test. Given that the definition of validity is the

degree to which a test is measuring what it claims to measure, any such mismatch

between the construct or content that a test is designed to measure and the test, would be

a threat to the test's validity (Brown, 1999).

Watanabe (2004) (cited in Cheng, et al., 2004) characterizes the backwash

effect according to the dimensions of specificity, intensity, length, intentionality and

value, along with what aspects of an examination can influence teachers and learners, and

factors influencing backwash. He further states that qualitative methodology may better

characterize backwash than quantitative methods and then outlines a general method for

investigating the phenomenon.

Prodromou (1993) (cited in Bedford, 2004) points out in his overview of the

backwash effect that the issue has not been fully explored in second language research in

spite of playing such dominant role in classrooms. He believes that many assumptions

about backwash are simplistic and untested with little observation to back them up. This

is confirmed by Alderson and Wall (1991) (cited in Bedford, 2004) who have conducted

extensive research into the area. They argue that backwash is a far more complex issue

than simply the effect of testing and teaching. They believe that there is no consistent

relationship between tests and their impact. Rather, specific areas such as teaching

content and methodology, teacher competence, assessment methods and resources

5
available need to be investigated as well as the extent of the impact and whether or not it

is positive or negative. Their findings also include the potential effect of backwash on

the whole education system.

There is evidence to suggest that tests have backwash effects on teaching and

learning -

the Alderson and Wall WashbackfBackwash Hypothesis (Alderson and Wall,

1993) (cited in Cheng, 1997). The extensive use of test scores for various educational

and societal purposes in society nowadays has made the effect of backwash a distinct

educational phenomenon. Cheng (1997) conducted a study on the backwash effect of the

Hong Kong Certificate of Education Examination (HKCEE) in English in Hong Kong

secondary schools by employing various methodological techniques such as

questionnaires, interview and classroom observations in sampled schools in Hong Kong.

Findings indicate that backwash effect works quickly and efficiently in bringing about

changes in teaching materials, which is due largely to the adaptable and commercial

nature of Hong Kong society, and slowly and reluctantly and with difficulties in the

methods teachers employ.

Backwash is also described in the literature as overt or covert (Figure 6.2).

According to Prodomou (1995) (cited in Bedford, 2004», overt backwash is

usually negative and seen in the explicit use of examination papers or examples from

textbooks that emphasize the skills used in exams.

6
Figure 6.2 Backwash: Overt and Covert

BACKWASH
EFFECTS

OVERT COVERT
(Explicit) (Implicit)

As a result, reading and writing are given more emphasis than speaking and

listening. He sees the implicit consequences of covert backwash as more of an

underlying, unconscious process stemming from assumptions about how students learn.

He likens it to teaching a textbook as if it were a testbook. However, he offers a solution

for transforming negative backwash into positive by shifting to a learner centered

approach with an emphasis on the language process rather than being too preoccupied

with the end-product.

McNamara (2000) (cited in Bedford, 2004) raises the possibility of the type of

assessment being an important factor. He sees performance assessments that require

integrated content and skills as having more positive backwash than discrete item testing

which often stifles communicative teaching approaches. McKay (2000) (cited in Bedford,

2004) discusses the effects of the format of standards that consist of a checklist of

learning outcomes listed vertically. She believes that this encourages teachers to focus on

individual descriptors rather than engage students in an integrated cyclical learning

approach.

7
Although commonly assumed to have a negative effect on the classroom, some

researchers feel that backwash should be viewed positively, as a means of altering

classroom behavior. McKay (2000) mentioned a study conducted by Rea-Dickens and

Gardner (2000) which found that ESL standards were central to legitimizing the English

as an Additional Language work in schools in England. In addition, Brindley (1998)

(cited in Bedford, 2004) comments on students receiving diagnostic feedback on the

success of their learning because of explicit performance criteria. There is a direct link

between attainment targets, course objectives and learning activities. Therefore

assessment is closely integrated with instruction.

A study by Priest (1996) attempted to find evidence of testing backwash in the

reading classroom. A survey of English teachers' attitudes and views on testing in

Brunei Darussalam was conducted. Findings indicate that the teachers believe that there

is a backwash effect from examinations, and that there appear to be a number of

differences between teachers' views depending on gender and age. The study further

indicate that there is evidence that backwash from examination does exist, and that the

introduction of a new examination in Brunei Darussalam presents an opportunity for

future researchers to study changes in the backwash effect both before and after the

introduction of the new examination.

Andrews (2004) (cited in Cheng, et al., 2004) looks at the relationship between

curriculum and backwash. He outlines the negative influences of tests and how tests are

used to promote innovation in curricula, especially language tests. He illustrates these

through Sri Langka's "0" level test revisions, Hong Kong's Certificate of Education

I
Examination (English Language), university entrance examinations in Japan, TOEFL,

and general education in England, Trinidad and Uganda. He concludes with suggestions

to be thoughtfully considered when introducing or revising high-stakes tests.

Stecher, Chun and Barron (2004) (cited in Cheng, et aI., 2004) report in their

study of the effects of assessment-driven testing (W ASL) on writing teaching in

Washington State University USA The testing is administered in grades 4, 7 and 10 in

certain subject areas for each grade. They focuses on grades 4 and 7 because those were

the only grades tested at the time of the study. As a result of analyzing statewide surveys

to both principals and teachers, the researchers found that although the approach to

writing, a process approach, changed little before and after the tests were instituted,

curriculum (writing conventions, emphasis on audience, purpose, styles and formats) and
1

nstructional methods (greatest emphasis on W ASL rubrics for student feedback) did

change. The study concludes that the W ASL influenced instruction; however, it was

difficult to determine the importance of the standards in shaping the instruction.

Hayes and Read (2004)(cited in Cheng, et al., 2004) investigate the academic

module of the IELTS in New Zealand. In the first phase, they used questionnaires and 23

follow-up interviews with teachers preparing students. In the second phase, a classroom

study compared two courses, one with an English for Academic Purpose (EAP)

orientation while the other a more test-focused. Phase One of the study showed that most

IELTS preparation courses are in private, commercial language schools. From Phase

Two of the study, they conclude that a commercial, test-focused course may have more

negative backwash effects, while an EAP course addresses a wider range of needs

9
directly related to academic study and promoting language development.

Watanabe (2004)(cited in Cheng, et al., 2004) reports on the effect of Japanese

university entrance examinations on secondary school instruction. Classroom

observations were done in high schools from which students had been accepted by

prestigious universities regularly and, not coincidently, offered special exam preparatory

classes. His conclusions from the study were that teachers' beliefs and educational

background were important for beneficial backwash to take place.

Lastly, a study was conducted by Luxia Qi (2004)(cited in Cheng, et at, 2004) on

China's National Matriculation English Test (NMET). The study was done by

interviewing participants, 8 test constructors, 3 English inspectors, and 10 senior

secondary school teachers from both rural and urban schools. The conclusions are that

the NMET has had little intended positive backwash because linguistic knowledge and

teaching only skills tested on the NMET are still emphasized in the classroom.

7.0 RESEARCH QUESTIONS

The study is conducted to answer the following research questions:

(i). What is the extent of the backwash effects produced by the School-based Oral

English Test on the classroom instructions by ESL teachers in preparing students for

the test?

(ii). What is the extent of the backwash effects produced by the content of the School­

based Oral English Test on the academic performance of students on the test?

(iii). What is the extent of the backwash effects produced by the conduct of the School-

10
based Oral English Test on the academic performance of students on the test?

8.0 RESEARCH DESIGN

This is a qualitative study utilizing the descriptive analysis technique. The methods used

to conduct this study are:

(i). Examining documents and records of the Oral English Test results conducted at

school-based level.

(ii). Interviewing ESL teachers and students to find out about the backwash effect.

9.0 SAMPLE

A total of five teachers and five students from three schools in the district of Batang

Padang in the state ofPerak were selected as samples in this study. The selection process

was conducted through the technique of cluster sampling. The schools involved were as

follow:

(i). SMK Trolak -

2 teachers

(ii). SMK Seri Kandi -

2 teachers and 2 students

(iii).SM Methodist, Tanjong Malim, Perak. -

1 teacher and 3 students

10.0 INSTRUMENTATION

The instrument used to collect the data for this study is the interview question set (Refer

to Appendix B). The items for the interview question set were formulated by the

researchers themselves based on the needs of the study. Both researchers are experienced

English language lecturers with more than 18 years teaching experience in secondary

11
schools, teacher training college, and the university.

Students' performance were assessed based on a manual which contained

the framework and guidelines for the implementation of the school-based SPM Oral

English Test (O.E.T) provided by the Examination Board, Ministry of Education. It was

designed to assist teachers in conducting the test and to provide student candidates with

essential information in preparing for the test. It also outlined the aims and objectives of

testing SPM Oral and the criteria for assessment. It was hoped that preparation toward

taking the test would help do develop students' oral ability in line with the learning

objectives as stated in the English Language Syllabus for Malaysian Secondary Schools.

The O.E.T also aimed to assess students' ability to use the English language in everyday

life.

The test assesses the students' ability to:

I. converse on the topic effectively;

11. speak fluently;

iii. speak coherently;

IV. give appropriate responses;

v .

speak using the language appropriately within context;

VI. speak using correct and acceptable pronunciation;

vii. speak using correct grammar,

viii. speak using a wide range of appropriate vocabulary.

Four models of Oral Assessment had been developed. Examples illustrating each

of these four models were included in the manual. School candidates were assessed

12
using Models 1 and 2 in Form Four and Models 3 or 4 in Form Five. However, they

served only as guidelines and student candidates must prepare their own materials for

assessment purposes. Model I focused on individual presentation. Several suggested

speaking activities included presenting information from non-linear forms, giving

descriptions, narrating, presenting reports, and giving opinions. Model 2 focused on

interaction between the candidate and the teacher assessor. Several suggested speaking

activities included asking for and giving instructions, talking and sharing information,

giving descriptions, and interviewing.

Model 3 focused on pair work. Several suggested activities included talking and

sharing information, solving problems, making enquiries and ordering goods and services,

making and responding to complaints, interviewing, asking and giving instructions, and

role-playing. Model 4 focused on group work. Several suggested speaking activities

included talking and sharing information, solving problems, giving opinions, and role­

playing (Kementerian Pendidikan Malaysia, 2002) ..

11.0 DATA COLLECTION

Approval for the research was obtained from the Educational Planning and Research

Division (EPRD) and the Perak state education department.

The interviews were conducted in the middle of 2006 at the three selected schools

in the district of Batang Padang, Perak. One teacher was interviewed in SMK Trolak,

two teachers and two students were interviewed in SMK Seri Kandi, and two teachers

and three students were interviewed in SM Methodist. All the interviews were conducted

13
by one of the researchers at the respective schools. The interview sessions were

recorded on tape for reference and analysis purposes.

12.0 DATA ANALYSIS

The recorded interview sessions were converted into transcripts and analyzed

descriptively to provide the results for the study (Refer to Appendix D).

For the purpose of analysis for the study, the five teachers were labeled

TEACHER A and TEACHER B for the teachers from SMK Trolak, TEACHER C and

TEACHER D for the teachers from SMK Sri Kandi, and TEACHER E for the teacher

from SM Methodist. The five students were labeled STUDENT A and STUDENT B for

the students from SMK Sri Kandi, and STUDENT C, STUDENT D and STUDENT E for

those from SM Methodist.

The students' performance on the school-based oral test were measured based on

marks awarded in 5 bands:

Excellent 9-10

Good 6-8

Satisfactory 4-5

Weak 2-3

Very weak 1

be referred to in Appendix C.
A detailed profile of the OET assessment constructs can

14
13.0 RESULTS

This section presents the breakdown of the grades obtained by the students on the

school-based oral English test at the end of the year and results of the analysis of the

transcripts gathered through the interviews of teachers and students in the study. The

results are divided two sections. These include results from the interviews of the students

and results from the interviews of the teachers.

13.1 Students' Academic Performance

The following are the grades by the students on the test conducted by the teachers in the

respective schools:

Table 13.1: Students' Grades on the School-based Oral English Test

Student MarkslBand
A 9 (Excellent)
B 9 (Excellent)
C 9 (Excellent)
0 9 (Excellent)
E 9 (Excellent)

• Students are given marks based on 5 bands from 1 (very weak) to a maximum of

10 (excellent) for the test

13.2 Results from the Interviews of the Students

The following are the analysis of the results from the interviews of the students.

Student A

(i). Conduct of the Test

She sat for the test which was conducted by her English teacher in the school.

15
Altogether, she was tested four times with a duration of five minutes each. She felt that

"
she could improve her English by sitting for the test. Exp. ...
Yes, I can improve myself. I

call speak English." Even though she was tested by her own teacher from the school,

she did not mind being tested by teachers from other schools if there was need to do so.

However, if given a choice she would prefer to be tested by her own English teachers
" "

from the school. Exp. ... Own teacher ...

(ii). Content of the test

She would like to improve her speaking and writing skills more compared to the other
u

language skills as she felt that they were more important. Exp. ...

Speakingcan,

writing also can." She agreed that speaking skill is very important for her to have when

"

she leave school after her studies. Exp. ... When we are out we can speak to other

"

people.

Student B

Student B was interviewed together with Student A in the same session.

(i). Conduct of the test

She also sat for the test which was conducted by her English teacher in the school. She

was also tested four times with a duration of about five minutes each. Even though

she was tested by her own teacher from the school, she did not mind being tested by

teachers from other schools if there was need to do so.

In contrast with Student A, if given a choice she would prefer to be tested by

English teachers from other schools. This was because she felt that she would learn

16
"

something different. Exp. ... Teacher (from) other school ... Because we can know

"

(something) different.

(ii). Content of the Test

She would also like to improve her writing and speaking skills more compared to the

other language skills as she felt that they were more important. She also agreed that

speaking skill is very important for her to have when she leave school after her studies.

Student C

(i). Conduct of the Test

The oral test was conducted by the English teacher in the school. She was satisfied with

"

the test as she was told by her teacher that she did very well in the test. Exp. ...

My
"

teacher said I did ok.

(ii). Preparation for the Test

She mentioned that her English teacher gave instructions about the test in the classroom

"

during teaching periods before the students sat for the test. Exp. ...
Yes, she told us how
"

to, what we should do when we want to perform in our oral English test.

However, her teacher did not give any exercises for her class to do in preparation

for the test. The students had to practice on their own to prepare for their test. Exp.
"

...
Yes, but ifwe want to practice we do it 011 our own." For the test, she was given

the topic one week before the test to prepare. She hoped that for the future session, more

17