Evaluation 10/08/06 Evaluation Parameters for ESL Listening and Speaking Classes

This discussion is broken into two sections. This first section discusses the actual parameters and grade production process as well as expected users of the evaluation. The second part is focused on a rationale for this evaluation procedure since it is not the standard and accepted procedure used in commercial testing systems (such as the TOEFL or IELTS). Parameters The evaluation is directed only towards the students and their parents (in primary or secondary schools where the parents are more active stakeholders) but most especially it considers the student as primary stakeholder. The rationale for this approach to evaluation is based on the role of the listening and speaking class in most curriculum settings as being less important to the overall evaluation of student success. In many EFL contexts listening and speaking is subsumed under a generalist approach to language teaching that includes reading and writing as well as listening and speaking in the same class period. At NIS, listening and speaking is not considered a part of the core curriculum that determines class rank and GPA averages for the transcript. This makes it less important pedagogically for the students to be compared to any outside standard. This does not mean that there are no standards, or that the students could not be compared using outside evaluation methods, only that for the purposes of this class in this situation there is no need for this kind of evaluation, especially when the evaluation process can be used to further pedagogical concerns. The pedagogical considerations are these: 1. all evaluation is presentation based: in other words, they are graded solely on class presentations 2. they cannot fail a presentation if they attempt it: only by failing to do the preparation or to give a presentation can a student get a failing mark 3. they cannot achieve a 100% grade: 100% is the grade that a perfect native speaker could get in the presentation situation. 4. Students must listen to each other during the presentations and try to recognize strong and weak presentations themselves, reviewing the grading sheet after the presentations allows students to compare their ideas to the evaluators opinion. 5. The students cannot read a presentation, to achieve success in the class they must learn to speak extemporaneously, freely and clearly. 6. Grading for each presentation is comparative: this means that the student with the highest grade is the student with the highest point count for this presentation with this class on this day. The evaluation is done based on the following criteria (see the attached grade sheet as well):

Fluency: how smoothly the student speaks, whether there are inappropriate

Evaluation

pauses or breaks word stress: for advanced/senior students word pronunciation (based on stress) is crucial and forms a key element of the class curriculum sentence focus: the use of emphasis, elision, intonation and control of the microtools inherent in the English discourse system is also a key part of the advanced/senior curriculum confidence: comfort in speaking, voice control and other affective aspects verb grammar: for beginner/ junior students grammar mistakes are a key focus of the class noun grammar: for beginner/ junior students syntax: because of the preponderance of Asian students, syntax (sentential word order) is a key concept for beginner/ junior students. Vocabulary: for all students the appropriate use of vocabulary in speaking is essential Time: the relationship between the length of presentation, and therefor the opportunity to make mistakes, and the grade is something that helps the students focus on speaking clearly and with purpose.

Clearly some of the criteria are more focused on the two different levels. Problems with grammar and syntax in a senior/ advanced student indicates a problem with placement and can indicate that the student will have continuing problems in the program unless they are placed in a more appropriate level. Late in the year as beginner/ junior students begin to master the grammar and syntax the affective and discourse elements become more important. For this reason all elements are considered in both levels. One of the key differences in this grading system is my rejection of the rubric system. While I find that my system of grading and a rubric system generally match in final grade results (in the past I have asked other teachers to grade with me and then compared their grades, done with various rubric systems, and mine and found almost no differences), the system I use provides the student with a more graphically useful representation of their work. They can tell at a glance whether their mistakes came at the beginning or the end of their presentation, sometimes they can even pinpoint the exact word they made a mistake on. The system also requires the teacher/ evaluator to focus entirely on the quality of the language, rather than on the content of the presentation. Since language is the focus of the program (content exists to provide a medium for the language instruction in this case) it is less important that the student has perfect content. The evaluation process for the teacher can be exhausting because it requires continual and focused listening to the presentation language (not the content). This is the drawback of the system as noted by other teachers who have tried the method. I feel that the advantages of the graphical nature of the grade sheet overcomes this drawback, but I also appreciate how hard it can be. The evaluator follows a simple repetitive process of listening in 15 second waves across the grade sheet. While listening for any mistakes in any of the categories which will give a - or minus sign in the appropriate section the evaluator tries to listen for a second or two for success in each of the 8 categories in turn. At the end of each wave a tic mark can go in the time box

Evaluation

to mark 15 seconds of time passing. If the student doesn't have noticeably positive or good/ appropriate language in the section being looked for then the evaluator moves on to the next item. For example: most students can begin with a check in fluency if they begin at all. The evaluator then listens for word stress: a correctly pronounced multi-syllable word will get a check mark (as opposed to a minus sign) for the presenter, no multisyllable words will get no marks, a mispronunciation for a junior at the beginning of the year will get no mark while a mispronunciation at any time (in the wave) will get a minus sign for a senior/ advanced student. These two sections have taken 2-3 seconds and the evaluator moves on to sentence focus and repeats the process with each subsequent category. At the end of the wave a tic mark goes in the time section and a quick check of the clock completes the first wave. The presentation happens in an unusual environment as well. Instead of a single student standing in front of the class while the rest of the class and the teacher sit in class formation staring at them, presentations are given in a relaxed and uncontrolled environment where students can mill around both the teacher and the presenter. Students can watch the grading process and see the relationship between mistake and marks, as well as between good language and good marks. As well, students can have their friends with them while they present, sometimes holding them or holding hands while they present. All this gives the presentations a different, less stressful appearance to an outside observer although the students feel the stress and sometimes act out because of it. I have used this method with classes of as many as 65 students and found that the students perform more easily and comfortably than when they have to stand and perform. The evaluation process in this way is considered as an integral part of the pedagogical process, not a test that is apart from the learning process, but rather a natural part of the learning. The actual grading is simple and short and completed quickly so that the students can look at the grade sheet before the end of the class period and get instant recall of their mistakes and good points. The checks and minuses are added and subtracted. Every 15 seconds is worth another 1 point. So a presentation that had 12 checks, 4 minuses and lasted for 30 seconds would be: 12-4+2=10. This, in itself however means nothing, it is only in comparison to all the scores from that presentation that the score gets a percentage rate. When this is finished the sheet is passed around so each student can see their own grades, as well as others. The students are constantly reminded during this phase that:

everyone has good and bad days, don't feel bad or good about one grade look at what you did well: build on your strengths don't try to fix everything at once: think of one thing you can improve and work on that there will be more presentations, soon!

It should be clear from the above that affective factors are of primary concern to the evaluation process for these classes. These affective factors are focused on the pedagogical process so that the evaluation is itself part of the process, not in a

Evaluation backwash relationship but rather as a direct and intentional pedagogical tool.

Evaluation Rationale It should be immediately clear from the discussion above that there is no evaluation of listening ability in these classes. The reasons for this are based, first and foremost, in the approach that views language acquisition as being of greater value than language learning. This primary dichotomy informs all the following discussion. Acquisition assumes that listening is the first skill that anyone has to master in the language acquisition process. Reading, writing and speaking are built on this primary skill. Therefore, any evaluation of reading, writing or speaking is automatically an evaluation of listening ability. Direct evaluation of listening is, however, extremely problematic. Perhaps with the use of modern MRI brain imaging technology it would be possible to see that listening information was being processed properly, but any other evaluation requires a second layer of evaluation based in one or more of the other skills. For example, listening to a recorded piece and answering multiple choice questions about the piece requires not just listening but reading skills. This kind of evaluation can easily be seen as a test of speed reading if the recorded material is at natural speed. All the standard tests of listening ability suffer from similar difficulties: there is no way to evaluate pure listening ability. Acquisition also changes the focus of the class from learning content to acquiring the language that holds the content. Listening tests tend to focus on the ability to recognize important content information (which can even include grammatical information), which tends to focus the students on keyword recognition as opposed to function words and discourse and pragmatics tools which are of great importance to acquisition. For example, while Asian students can recognize the important content words in a lecture or reading, they often fail to see or recognize important grammar and discourse markers at the end of words which is at the root of their problems with grammar and syntax. Affective elements of language also gain greater importance in acquisition based language teaching. For this reason a non-rubric system is used to provide an immediate, graphic evaluative system that allows the student to see their strengths and weaknesses. For affective reasons the students are focused on their strengths in postevaluation discussions rather than discussing weaknesses. They are encouraged to find single solvable instances of weakness to improve. It is practically impossible to fail the evaluation so that student knows that their work on the presentation will be recognized no matter what the outcome. A rubric system doesn't allow the student to see what mistakes they made, or to

Evaluation see where they had good work that the evaluator recognized. The system I use allows both the teacher and the student to have greater appreciation for and use of the evaluation to improve their language. Finally, this approach eschews the use of any tests or exams. As described previously the presentation format used allows the students to mill and mingle around the evaluator and the presenter. This reduces the feeling of pressure on the presenter and allows them to view the presentation as a part of the class rather than as a separate piece of examination. Further, the presentation schedule is fairly intense, especially for the senior/ advanced class which can sometimes have two evaluated presentations in a single class period. For the junior/ beginner class the schedule is less intense, but the process of preparing and practicing the presentation is a large part of the class.