Wolf, D. P. 1989. Portfolio assessment: Sampling student work. Educational Leadership, 46, pp. 35-39.

Creating Placement Tests by Joel Murray

Many educators around the world find themselves in growing schools or programs with more and more students needing placement into appropriate classes. Many schools use tests that are intended for other purposes--e.g., the Secondary Level English Proficiency (SLEP) Test, the Test of English as a Foreign Language (TOEFL), or the Michigan. However, these can be inaccurate for placing students in classes. Some schools have no placement procedures, and students are assigned to classes haphazardly. Many ESL/EFL teachers face the task of creating placement tests despite their lack of experience with testing beyond the tests they give in their own classrooms. Many educators don't know how to begin or proceed in creating placement tests that yield results that are reliable (i.e., that are consistent from one administration of the test to another), valid (i.e., that measure what you want them to measure and not something else) and accurate (i.e., that place students into the appropriate levels with little or no error). There are several important steps to creating good placement tests. Assemble an Assessment Team The first step in creating a placement test is assembling an assessment team composed of parties interested in the placement process. The concerns of individual stakeholders should be addressed to help define the purpose of the test and make decisions about it. The assessment team should include administrators responsible for students and the curriculum, coordinators responsible for implementing the curriculum, and teachers representing a cross-section of classes. The team should also include one element that is often missing in the design of placement tests: the test-takers themselves. As Bradshaw (1990, 27) notes, "there seems to be no reason why some

degree of collection of test-takers' and test-users' reactions cannot be included as part of the design of any new test." Focus on the Test-Takers Once the assessment team has been assembled, they should describe or characterize the test-takers who will be taking the placement test. Characteristics taken into account should be those relevant to the testtakers' identity and cultural background: age, gender, first language, country of origin, place of residence, languages already learned, current stage of learning, reasons for taking the test or for entering the school, personal and professional interests, and amount of background knowledge. Describing the test-takers can help with the choice of appropriate test material and test techniques. Define the Test Objectives After the test-takers have been characterized, the objectives for the placement test should be outlined using the following questions: What is the aim of the test? What should be tested? Why? How should it be tested? These questions can be answered by looking at the curriculum, the students, the type of classes, and so on. Hughes (1989, 48) has stated that this stage is essential "to make oneself perfectly clear about what it is one wants to know and for what purpose." Choose a Test Type The objectives having been defined, the team should decide the type of placement test to be used and what it should contain. First, should the test be direct or indirect or a combination of the two? Direct testing requires the test-taker to perform the skill to be measured (e.g., having test-takers write something in order to see how well they write) while indirect testing measures something dependent upon an underlying skill (e.g., having testtakers answer multiple-choice comprehension questions to see how well they have understood something to which they have listened). Second, should the test be discrete point or integrative? Discrete point testing involves testing one thing at a time, item by item, (e.g., a multiplechoice grammar test on the present perfect) while integrative testing involves testing the combination of many elements in the completion of a task (e.g., a test involving the writing of a grammatically accurate, wellplanned, unified, cohesive paragraph). The distinction between discrete point and integrative testing is, according to Hughes (1989, 17), "not unrelated to that between indirect and direct testing," with discrete point tests being almost always indirect, and integrative tests direct. Finally, should the test be norm- or criterion-referenced? A normreferenced test is one in which the amount of knowledge or material known

by each test-taker is compared with that known by other test-takers, with the aim of spreading the test-takers out along a continuum of general abilities or proficiencies so that differences among them are reflected in the scores (Brown 1995), while a criterion-based test is one in which the assessment of the amount of knowledge or material known by each testtaker is compared with a level of achievement or a set of criteria. Which type of test should one choose? Unfortunately, the answer is not always straightforward and depends upon what is to be included in the test. First, for placement tests involving speaking and writing, direct testing is best because these two productive skills provide something that can be directly observed and measuredutterances and written material. For placement tests involving listening and reading, indirect testing should be used as these two receptive skills yield something that can neither be directly observed nor measured but can only be inferredcomprehension of an utterance or a reading passage. Second, because of the relationship between direct/indirect testing and discrete/integrative testing, for tests involving grammar or listening, discrete point testing should be used. The rationale for this recommendation is that the underlying ability--grammatical structure or listening comprehension--is difficult to test efficiently by other means. The ability is thus best assessed by breaking it down into its individual elements, such as knowledge of the past progressive or the understanding of the gist of a lecture, and by assessing enough of these elements to indicate the test-taker's underlying ability. For speaking or writing, integrative testing is best because the successful completion of the speaking or writing task--making a comprehensible utterance or writing a sentence-does not lend itself well to being broken down and the sum of a number of individual elements may not represent the whole of the ability. For tests involving reading, a combination of discrete point and integrative testing is best, as the underlying ability--understanding a written passage--is difficult to assess accurately by using exclusively one or the other form of measurement. This ability is thus best evaluated both by separating it into components (by, for example, having multiple-choice questions that focus on reference words, vocabulary in context, and the like) and by using a combination of elements (by, for example, requiring the test-taker to write a summary or a paraphrase of the passage). Finally, the placement test should be a norm-referenced test, for normreferenced tests, as stated earlier, compare what is known by each testtaker with that known by other test-takers. Because, according to theory, most scores should be close to the average with fewer scores out near the periphery (think of the bell curve here), norm-referenced tests will allow you to define distinct levels of performance and to make distinctions between individual performances (McNamara 2000, 63)--exactly what a

placement test should do. In order for a placement test to be normreferenced, the test has to be normed; that is, it must be trialed with a representative population of test-takers in order to obtain the scores representative of each level of placement at an institution or in an educational situation. Once that is done, the results can be used to set cutoff points (the lower and upper end of ranges of scores) that will allow students to be placed at different levels. Choose Test Content Concerning what the placement test should contain, it seems logical that the test should reflect the curriculum of the school, yet in practice this has not always been the case. As Brown (1989, 66) remarked, "we decided to develop a placement battery that would be related in content to the curriculum of our institute--a proposal that struck us as strangely novel." Nonetheless, the answers to a number of questions can help determine what should be included in a placement test: * How many sections should the test contain? * How long should the sections be? * How should the sections be differentiated? * How many items should there be in each section? * What is the target situation for the test and how could it be simulated? * What text types (written and/or spoken) should be chosen? * What language skills should be tested? * What aspects of the language should be tested? * What tasks should be required? * What test methods should be used? * What is the curriculum of the classes into which the placement test is placing test- takers? Create the Test Once the test type and content have been chosen, the test itself can be created. The test questions should be carefully crafted: the questions should be as easy as possible for the test-takers to understand and process. Unnecessarily complex questions only increase the cognitive load on the test-takers and do nothing to contribute to measuring what you seek to measure. In addition, tests should be relatively short--not too long for the test-takers to answer nor too long for the scorers to mark. Develop Scoring Guides After the test is created, scoring guides should be developed. These guides are intended for the markers of the placement test to help them assess testtakers' performance reliably and consistently. Of course, scoring guides are not necessary for multiple-choice questions, which simply require an

answer key. Rather, scoring guides are applicable to subjective or openended questions--for example, those requiring short essay answers or oral interaction. Scoring guides can be either holistic or analytical, with holistic meaning that one score is assigned to an answer (e.g., a mark out of 10), analytic meaning that separate scores for a number of different aspects are assigned to an answer (e.g., a mark out of 10 for grammar, 10 for content, and 10 for structure, for a total of 30 marks). Each type of scoring guide has a number of advantages and disadvantages. One advantage of holistic scoring is that it can be quick for the markers. Two disadvantages are the fact that the scoring scale must be well conceived and that there should be more than one scorer in order to ensure reliability. There are at least two advantages of analytical scoring: markers have to consider certain aspects of the test-taker's performance that they might otherwise overlook and the results can be used for diagnostic purposes. One disadvantage is that analytical marking is timeconsuming. The type of scoring guide that should be used depends on each testing situation. If time is not at a premium, analytical scoring is the better choice. Test the Test Having created the scoring guide, the placement test itself should be tested. In other words, before the test is implemented, it should be trialed with a set of students who are representative of the "real" test-takers. The trial will probably reveal problems with the test that could not be identified during its creation. For example, too many items on the test may be too difficult or too easy, open-ended test items may confuse test-takers, writing tasks may result in less response than expected because of poor or somehow insufficient wording, or multiple-choice items may be ambiguous or open to disagreement. The results of the trial should be analyzed. For multiple-choice test items, you should calculate the facility value (the level of difficulty of each test item) and the discrimination index (the ability of each item to discriminate between test-takers who did well and those who did not). For advice on how to do so, see Alderson et al. (1995, 80-86), Hughes (1989, 161162) or Harrison (1983, 127-133). For open-ended or subjective items, you should decide whether the items resulted in what you intended and whether the scoring guide, if applicable, is serviceable. Analyzing the results should yield important information about the test items and the test itself. For instance, if certain test items are too easy, they should perhaps be discarded or moved toward the beginning of the test. If others are too difficult, perhaps they should be re-worded or moved toward the end of the test.

Train the Testers Once the results of the trial have been analyzed and the test has been altered accordingly, the people who are going to administer and score the placement test should be trained. The test administrators, who may be support people but not educators, should receive instruction in delivering the test consistently and correctly with little or no variation from administration to administration. Alderson, Clapham, and Wall (1995, 115) maintain that it is important that [test] administrators understand the nature of the test they will be conducting, the importance of their own role and the possible consequences for candidates if the administration is not carried out correctly. Scorers should be provided with scoring guides and opportunities to calibrate their assessments of subjective test items. Weir (1990, 86) believes that this step is important, noting that considerable attention shouldbe paid to the development of relevant and adequate scoring criteria and examiners must be trained and standardized in the use of these. In conclusion, creating a placement test does not have to be an intimidating or overwhelming task for those instructors who have little or no experience in testing. By following the suggestions listed here, you should be able to create a placement test that is reliable, valid and accurate.
References Alderson, J.C., C. Clapham and D. Wall. 1995. Language test construction and evaluation. Cambridge: Cambridge University Press. Bradshaw, J. 1990. Test-takers reactions to a placement test. Language Testing 7(1):13-30. Brown, J. D. 1989. Improving ESL placement tests using two perspectives. TESOL Quarterly 23(1): 65-83. Brown, J. D. 1995. The elements of language curriculum: A systematic approach to program development. New York: Heinle & Heinle. Harrison, A. 1983. A language testing handbook. London: Macmillan Press. Hughes, A. 1989. Testing for language teachers. Cambridge: Cambridge University Press. McNamara, T. 2000. Language testing. Oxford: Oxford University Press. -----------------------------------