7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

A critical review of the IELTS writing test

Hacer Hande Uysal

ELT J (2010) 64 (3): 314-320. DOI: https://doi.org/10.1093/elt/ccp026

Published: 17 April 2009

Abstract
Administered at local centres in 120 countries throughout the world, IELTS
(International English Language Testing System) is one of the most widely used
large-scale ESL tests that also o ers a direct writing test component. Because of its
popularity and its use for making critical decisions about test takers, it is crucial to
draw attention to some issues regarding the assessment procedures of IELTS.
Therefore, the present paper aims to provide a descriptive and critical review of the
IELTS writing test by focusing particularly on various reliability issues such as single
marking of papers, readability of prompts, comparability of writing topics, and
validity issues such as the de nition of the international writing construct, without
considering variations among rhetorical conventions and genres around the world.
Consequential validity-impact issues will also be discussed and suggestions will be
given for the use of IELTS around the world and for future research to improve the
test.

Issue Section: Point and counterpoint

Skip to Main Content

Introduction
https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 1/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

Large-scale ESL tests such as Cambridge certi cate exams, IELTS, and TOEFL (the
Test of English as a Foreign Language) are widely used around the world, and they
play an important and critical role in many people's lives as they are often used for
making critical decisions about test takers such as admission to universities.
Therefore, it is necessary to address the assessment procedures of such large-scale
tests on a regular basis to make sure that they meet professional standards and to
contribute to their further development. However, although there have been several
publications evaluating these tests in general, these publications often do not o er
detailed information speci cally about the writing component of these tests.
Scholars, on the other hand, acknowledge that writing is a very complex and di cult
skill both to be learnt and assessed, and that it is central to academic success
especially at university level. For this reason, the present article aims to focus only on
the assessment of writing, particularly in the IELTS test, because as well as being one
of the most popular ESL tests throughout the world it is unique among other tests in
terms of its claims to assess English as an international language, indicating a
recognition of the expanding status of English. After a brief summary of the IELTS
test in terms of its purpose, content, and scoring procedures, the article aims to
discuss several reliability and validity issues about the IELTS writing test to be
considered both by language testing researchers and test users around the world.

The IELTS writing test

General background information

The IELTS writing test is a direct test of writing in which tasks are communicative
and contextualized for a speci ed audience, purpose, and genre, re ecting recent
developments in writing research. There is no choice of topics; however, IELTS states
that it continuously pre-tests the topics to ensure comparability and equality. IELTS
has both academic and general training modules consisting of two tasks per module.
In the academic writing module, for Task 1, candidates write a report of around 150
words based on a table or diagram, and for Task 2, they write a short essay or general
report of around 250 words in response to an argument or a problem. In the general
Skip to Main module,
training Content in Task 1, candidates write a letter responding to a given problem,
and in Task 2, they write an essay in response to a given argument or problem. Both
academic and general training writing modules take 60 minutes. The academic
https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 2/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

writing component serves the purpose of deciding university admission of

international students, whereas general writing serves the purposes of completing
secondary education, undertaking work experience or training, or meeting
immigration requirements in an English speaking country.

Trained and certi ed IELTS examiners assess each writing task independently giving
more weight to Task 2 in marking than Task 1. After testing, writing scores along
with other scores from each module of the test are averaged and rounded to produce
an overall band score. However, how these descriptors are turned into band scores is
kept con dential. There is no pass/fail cut scores in IELTS. Detailed performance
descriptors have been developed which describe written performance at the nine
IELTS bands and results are reported as whole and half bands. IELTS provides a
guidance table for users on acceptable levels of language performance for di erent
programmes to make academic or training decisions; however, IELTS advises test
users to agree on their own acceptable band scores based on their experience and
local needs.

Reliability issues
Hamp-Lyons (1990) de nes the sources of error that reduce the reliability in a
writing assessment as the writer, task, and raters, as well as the scoring procedure.
IELTS has initiated some research e orts to minimize such errors, including the
scoring procedure, and to prove that acceptable reliability rates are achieved.

In terms of raters, IELTS states that reliability is assured through training and
certi cation of raters every two years. Writing is single marked locally and rater
reliability is estimated by subjecting a selected sample of returned scripts to a second
marking by a team of IELTS senior examiners. Shaw (2004: 5) reported that the
inter-rater correlation was approximately 0.77 for the revised scale and g-
coe cients were 0.840.93 for the operational single-rater condition. Blackhurst
(2004) also found that the paired examinersenior examiner rating from the sample
IELTS writing test data produced an average correlation of 0.91. However, despite the
reported high reliability measures, in such a high-stakes international test, single
marking is not adequate. It is widely accepted in writing assessment that multiple
Skip to Main Content
judgements lead to a nal score that is closer to a true score than any single
judgement (Hamp-Lyons 1990). Therefore, multiple raters should rate the IELTS
writing tests independently and inter- and intra-rater reliability estimates should be
https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 3/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

constantly calculated to decide about the reliability and consistency of the writing
scores.

IELTS also claims that the use of analytic scales contributes to higher reliability as
impressionistic rating and norm referencing are discouraged, and greater
discrimination across bands is achieved. However, Mickan (2003) addressed the
problem of inconsistency in ratings in IELTS exams and found that it was very
di cult to identify speci c lexicogrammatical features that distinguish di erent
levels of performance. He also discovered that despite the use of analytic scales,
raters tended to respond to texts as a whole rather than to individual components.
Falvey and Shaw (2006), on the other hand, found that raters tended to adhere to the
assessment scale step by step, beginning with task achievement then moving on to
the next criterion. Given the controversial ndings about rater behaviour while using
the scales, more precise information about the scale and about how raters determine
scores from analytical categories should be documented in more detail to con rm
IELTS claims about the analytic scales.

IELTS pre-tests the tasks to ensure they conform to the test requirements in terms of
content and level of di culty. O'Loughlin and Wigglesworth (2003) investigated task
di culty in Task 1 in IELTS academic writing and found di erences among tasks in
terms of the language used. It was found that the simpler tasks with less information
elicited higher performance and more complex language from responders in all
pro ciency groups. Mickan, Slater, and Gibson (2000), on the other hand, examined
the readability of test prompts in terms of discourse and pragmatic features and the
test-taking behaviours of test takers in the writing test and found that the purpose
and lexicogrammatical structures in the prompts in uenced the task comprehension
and writing performance.

IELTS also states that topics or contexts of language use, which might introduce a
bias against any group of candidates of a particular background, are avoided.
However, many scholars highlight that controlling the topic variable is not an easy
task as it is highly challenging to determine a common knowledge base that can be
accessed by all students from culturally diverse backgrounds and who might have
varied reading experiences of the topic or content area (Kroll and Reid 1994). Given
the
Skip toimportance
Main Content of the topic variable on writing performance and the di culty of
controlling it in such an international context, continuous research on topic
comparability and appropriateness should be carried out by IELTS.

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 4/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

The research conducted by IELTS has been helpful in understanding some variables
that might a ect the reliability and accordingly the validity of the scores. As
indicated by research, di erent factors interfere with the consistency of the writing
test to varying degrees. Therefore, more research is necessary especially in the areas
of raters, scale, task, test taker behaviour, and topic comparability to diagnose and
minimize sources of error in testing writing. Shaw (2007) suggests the use of
electronic script management (ESM) data in further research to understand various
facets and interactions among facets, which may have a systematic in uence on
scores.

Validity issues
IELTS makes use of both expert judgements by academic sta from the target
domain and empirical approaches to match the test tasks with the target domain
tasks and to achieve high construct representativeness and relevance. Moore and
Morton (1999), for example, compared IELTS writing task items with 155
assignments given in two Australian universities. They found that IELTS Task 1 was
representative of the target language use (TLU) content, while IELTS Task 2, which
requires students to agree or disagree with a proposition, did not match exactly with
any of the academic genres in the TLU domain as the university writing corpus was
based on external sources as opposed to IELTS Task 2, which was based on prior
knowledge as a source of information. IELTS Task 2 had a greater similarity to non-
academic public forms of discourse such as a letter to the editor; however, IELTS
Task 2 could also be considered close to the genre essay, which was the most
common of the university tasks (60 per cent). In terms of rhetorical functions, the
most common function in the university corpus was evaluation, parallel to IELTS
Task 2. As a conclusion, it was suggested that an integrated reading-writing task
should be included in the test to increase authenticity. Nevertheless, IELTS claims
are based on the investigation of TLU tasks from only a limited contextBritish and
Australian universitiesthus, representativeness and relevance of the construct and
meaningfulness of interpretations in other domains are seriously questionable.

In terms of the constructs and criteria for writing ability, general language construct
in IELTS is de ned both in terms of language ability based on various applied
Skip to Main Content
linguistics and language testing models and in terms of how these constructs are
operationalized within a task-based approach. Task 1 scripts in both general and

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 5/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

academic writing are assessed according to task ful lment, coherence, cohesion,
lexical resource, and grammatical range and accuracy criteria. Task 2 scripts are
assessed on task response (making arguments), lexical resource, and grammatical
range and accuracy criteria. However, according to Shaw (2004), the use of the same
criteria for both general and academic writing modules is problematic, and this
application was not adequately supported by scienti c evidence. In addition, with the
new criteria that have been in use since 2005, the previous broad category
communicative quality has been replaced by coherence and cohesion, causing
rigidity and too much emphasis on paragraphing (Falvey and Shaw 2006). Therefore,
it seems as if traditional rules of form rather than meaning and intelligibility have
recently gained weight in construct de nitions of IELTS.

IELTS also claims that it is an international English test. At present, its claim is
grounded on the following issues (Taylor 2002).

1 Re ecting social and regional language variations in test input in terms of

content and linguistic features, such as including various accents.

2 Incorporating an international team (UK, Australia, and New Zealand) which is

familiar with the features of di erent varieties in the test development process.

3 Including NNS as well as NS raters as examiners of oral and written tests.

However, the English varieties that are considered in IELTS include only the varieties
of the inner circle. Except for the inclusion of NNS raters in the scoring procedure,
the attempts of IELTS to be considered as an international test of English are very
limited and narrow in scope. As an international English language test, IELTS
acknowledges the need to account for language variation within the model of
linguistic or communicative competence (Taylor 2002); however, its construct
de nition is not any di erent from other language tests. If IELTS claims that it
assesses international English, it should include international language features in its
construct de nition and provide evidence to support that IELTS can actually measure
English as an international language.

In addition, Taylor (2002) suggests that besides micro-level linguistic variations,

macro-level discourse variations may occur across cultures. Therefore, besides
Skip to Main Content
addressing the linguistic varieties of English around the worldWorld Englishes
the IELTS writing test should also consider the variations among rhetorical

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 6/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

conventions and genres around the worldworld rhetoricswhile de ning the

writing construct especially related to the criteria on coherence, cohesion, and logical
argument. Published literature presents evidence that genre is not universal, but
culture speci c; and people in di erent parts of the world di er in terms of their
argument styles and logical reasoning, use of indirectness devices, organizational
patterns, the degree of responsibility given to the readers, and rhetorical norms and
perceptions of good writing. In particular, the ability to write an argumentative
essay, which is used in the IELTS writing test, is found to demonstrate unique
national rhetorical styles across cultures; IELTS corpus database should be used to
nd common features of argumentative writing that are used by all international test
takers to describe the international argumentative writing construct (Taylor 2004).
This is especially important as Cambridge ESOL (the UK partner in IELTS) plans to
develop a common scale for L2 writing ability in the near future.

It is also important for IELTS to consider these cultural di erences in rater training
and scoring. Purves and Hawisher (1990), based on their study on an expert rater
group, suggest that culture-speci c text models also exist in readers heads and they
form the basis for the acceptance and appropriateness of written texts and a ect the
rating of student writing. For example, di erences between NS and NNS raters were
found in terms of their evaluation regarding topics, cultural rhetorical patterns, and
sentence-level errors (Kobayashi and Rinnert 1996). Therefore, it is also crucial to
investigate both NS and NNS raters rating behaviours with relation to test-taker
pro le.

In terms of consequences, the impact of IELTS on the content and nature of

classroom activity in IELTS classes, materials, and the attitudes of test users and test
takers has been investigated. However, these are not enough. IELTS should also
consider the impact of its writing test in terms of the chosen standards or criteria on
the international communities in a broader context. Considering IELTS claims to be
an international test, the judging of written texts from students of various cultural
backgrounds according to one writing standard (based on Western writing norms)
may not be fair. Taylor (2002) states that people who are responsible for language
assessment should consider how language variation a ects the validity, reliability,
and impact of the tests and should provide a clear rationale for why they include or
Skip to Main Content
exclude more than one linguistic variety and where they get their norms from.

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 7/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

As for the washback e ects of IELTS, at present, it is believed in the academic world
that international students and scholars must learn Western academic writing so that
they can function in the Anglo-American context. This view, in a way, imposes
Western academic conventions on all the international community, showing no
acceptance for other varieties. According to Kachru (1997), however, this may result
in the replacement of each and every rich creative national style in the world with the
Western way of writing. This view is re ected in most other tests of English as well.
However, because IELTS claims to be an international test of English, it should
promote rhetorical pluralism and raise awareness of cultural di erences in rhetorical
conventions rather than promoting a single Western norm of writing as pointed out
by Kachru (1997). Therefore, considering the high washback power of IELTS,
communicative aspects of writing rather than strict rhetorical conventions should be
emphasized in the IELTS writing test.

Conclusion

To sum up, IELTS is committed to improving the test further and has been carrying
out continuous research to test its reliability and validity. However, some issues such
as the fairness of using a single prescriptive criterion on international test takers
coming from various rhetorical and argumentative traditions and the necessity of
de ning the writing construct with respect to the claims of IELTS to be an
international test of English, have not been adequately included in these research
e orts. In addition, some areas of research on the reliability of test scores highlight
serious issues that need further consideration. Therefore, the future research agenda
for IELTS should include the following issues.

In terms of reliability:

the comparability and appropriateness of prompts and tasks for all test takers
should be continuously investigated

multiple raters should be included in the rating process and inter- and intra-rater
reliability measures should be constantly calculated
Skip to Main Content
more research is needed regarding scales and how scores are rounded to a nal
score

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 8/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

rater behaviour while using the scales should be investigated.

IELTS has rich data sources such as ESM in hand; however, so far this source has not
been fully exploited to understand interactions among the above-mentioned factors
with relation to test taker and rater pro le.

In terms of improving the validation e orts with regard to the IELTS writing module:

future research should be performed to explore whether the characteristics of the

IELTS test tasks and the TLU tasks match, not only in the domain of the UK and
Australia, but also in other domains

cultural di erences in writing should be considered both in the construct

de nitions and rater training e orts

research in respect of determining the construct of international English ability

and international English writing ability should also be conducted by using the
already existing corpus of IELTS, and consequences of the assessment practices
and criteria in terms of their impact on power relationships in the world context
should also be taken into consideration

test users should also have responsibility for undertaking their own research to
ensure that the test is appropriate for their own institutional or contextual needs.

References

Blackhurst A.IELTS test performance data 2003,Research Notes,2004,vol.18(pg.18-20)

Falvey P, Shaw SD.IELTS writing: revising assessment criteria and scales (phase 5),Research
Notes,2006,vol.23(pg.7-12)

Hamp-Lyons L.Kroll B.Second language writing assessment issues,Second Language Writing:

Research Insights for the Classroom,1990New YorkCambridge University Press

Kachru Y.Smith LE, Forman ML.Culture and argumentative writing in world Englishes,Literary
StudiesEast
Skip and West: World Englishes 2000 Selected Essays,1997Honolulu, HIUniversity of
to Main Content
Hawai'i Press

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 9/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

Kobayashi H, Rinnert C.Factors a ecting composition evaluation in an EFL context: cultural

rhetorical pattern and readers background,Language Learning,1996,vol.46/3(pg.397-437)
Google Scholar CrossRef

Kroll B, Reid J.Guidelines for designing writing prompts: clarifications, caveats, and
cautions,Journal of Second Language Writing,1994,vol.3/3(pg.231-55)
Google Scholar CrossRef

Mickan P, Slater S, Gibson C.A Study of Response Validity of the IELTS Writing Module,IELTS
Research Reports Vol. 3, Paper 2,2000Canberra, AustraliaIDP IELTS Australia

Mickan P.What is Your Score? An Investigation into Language Descriptors from Rating Written
Performance,IELTS Research Reports Vol. 5, Paper 3,2003Canberra, AustraliaIDP IELTS Australia

Moore T, Morton J.Authenticity in the IELTS Academic Module. Writing Test: A Comparative
Study of Task 2 Items and University Assignments,IELTS Research Reports Vol. 2, Paper
4,1999Canberra, AustraliaIDP IELTS Australia

O'Loughlin K, Wigglesworth G.Task Design in IELTS Academic Writing Task 1: The E ect of
Article Navigation
Quantity and Manner of Presentation of Information on Candidate Writing.,IELTS Research
Reports Vol. 4, Paper 3,2003Canberra, AustraliaIDP IELTS Australia

Purves A, Hawisher G.Beach R, Hynds S.Writers, judges, and text models,Developing Discourse
Practices in Adolescence and Adulthood. Advances in Discourse Processes. Vol. 39,1990Norwood,
NJAblex Publishing

Shaw SD.IELTS writing: revising assessment criteria and scales (phase 3),Research
Notes,2004,vol.16(pg.3-7)

Shaw SD.Modelling facets of the assessment of writing within an ESM environment,Research

Notes,2007,vol.27(pg.14-9)

Taylor L.Assessing learner's English: but whose/which English(es)?,Research

Notes,2002,vol.10(pg.18-20)

Taylor L.Second language writing assessment: Cambridge ESOL's ongoing research

Skip to Main Content
agenda,Research Notes,2004,vol.16(pg.2-3)

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 10/97
7/4/2017 critical review of the IELTS writing test | ELT Journal | Oxford Academic

This article forms one part of a Point/Counterpoint section. It will appear together with its
opposing Counterpoint view in the printed journal.

A version of this Paper was presented at the 5th International ELT Research Conference,
Canakkale, Turkey in May 2008.

Author notes

Final revised version received November 2008

The Author 2009. Published by Oxford University Press; all rights reserved.

View Metrics

Email alerts
New issue alert
Advance article alerts
Skip to Main Content
Article activity alert

https://academic.oup.com/eltj/article/doi/10.1093/elt/ccp026/433418/A-critical-review-of-the-IELTS-writing-test 11/97