Assessing Learners Progress: Quantitative Research in EFL

Communicative Competence is the goal of Communication Skills courses at ____________ University College. These classes focus on improving Prep Year English students ability to communicate in the four skills areas: Speaking (primarily), Writing (secondarily), with residual Listening, Reading, and Grammar improvement expectations.

Communication Skills II students, taught this past semester, were required to do four major presentations -- increasing in complexity -- to expand their ability to convey personal information, future goals, persuasive speech (requiring research and paraphrasing to call an audience to action on a relevant social issue); and a total skillsconsolidation newscast presentation. The newscast requires students to work in groups: gathering, and writing for production, relevant news stories in a news production format -- including oral and visual presentations related to each story. Each of these presentations requires research (reading for comprehension) and comprehension of basic English grammar, appropriate vocabulary, plus topical knowledge and the ability to craft English as a Foreign Language into presentations which allow students to make themselves understood by an audience of native and non-native English listeners.

Raina Foster Bichara designed the Communication Skills syllabi to build students self-confidence and expand their competency in language usage. Graded rubrics for each presentation generally show improvements in oral presentation skills for all students who stay enrolled, on-task and complete required assignments.

After teaching Communication Skills I and II for two semesters here, I desired to add a quantitative research assessment, in addition to the existing qualitative assessments, of students oral presentations . My goal was to try and measure whether improvement of oral communicative competence can translate into improvement in basic comprehension and grammar skills on written tests. I selected tests designed by Kenneth Beare at About.com designated for beginner-level adult students to first, assess students starting points (Basic Grammar and Comprehension). At the end of the semester, these tests were re-administered to measure students ending points in Mr. Beares Basic Grammar and Comprehension assessments. This paper presents results from three Communication Skills II classes: Sections 202, 203 and 204.

The first two days of classes, students were administered writing assessments in the form of writing prompts to be compared with end-year Journal submissions: measuring beginning- and ending-level skills in spelling, grammar, and punctuation. Those results are not included in todays presentation. Todays results are from the written tests administered to Communication Skills II students: Kenneth Beares 50

Basic English Questions Quiz Comprehension and Beginner English Grammar: 5[8] Question Quiz 1. At the beginning of Semester 2, average class scores for Comprehension -out of a possible 100 -- were as follows: Sec. 202, 54.76, Sec. 203, 80.22 points and Sec. 204 86.33 points. Section 202s scores of a rounded 55 points represented the average level of JUC students, based upon my previous teaching experience for Grammar, Writing, Listening and Speaking as well as Communication Skills. However,

since it was the beginning of Semester 2, I rated them as lower-level beginners; as my expectations for each class was 65 median after successful completion of Semester 1. Sections 203 and 204s scores of rounded 80 and 86 points placed them as high-level beginners, still. The hope was to increase these written test scores: applying a Vocabulary, Structure, and Opportunity (Peace, 2008) strategy developed and practiced when teaching Conversation skills to high school students in Korea in weekly classes and seasonal three-week Winter and Summer Intensive-Study English Camps.

Average results for initial oral assessments were middle-beginner: sentence constructions with subject-verb agreement disorders, misplaced or missing adjectives, need for vocabulary expansion for specific purposes. The approach was initially roleplay, requiring students to visually read for comprehension, then read aloud for understanding, adding inflections to enhance their being understood by listeners; finally an impromptu re-writing of the dialogue scenes, requiring students to re-present content in a personalized fashion. These activities are ones that I chose to engage class participants in understanding, and being understood, in their foreign language: using chunks of language to learn context for new vocabulary as well as social context for vocabulary applications. Role-plays, I believe, are extremely important for the second language learner to become comfortable speaking in the foreign language, learn inflection and rhythm by patterning in initial group repetitions, and to finally display effective communication skills by making the audience understand original dialogue in targeted topics.

Again, the end-goal of Mrs. Bicharas Communication Skills I and II syllabi is increased oral competency. Communicative competency is measured as the ability of the L2 English learner to be understood in her second language.

According to Canale and Swain (1980), communicative competence can be defined in three components: 1. grammatical competence: words and rules 2. sociolinguistic competence: appropriateness 3. strategic competence: appropriate use of communication strategies Students continued learning via the Vocabulary, Structure, and Opportunity strategy completing presentations for Future Goals, How to . . . [do a task], Persuasive Speech, and Newscast -- Special Report. New Vocabulary for How to . . . was introduced using a YouTube cooking show video presented by Julia Child (The Omelette Show). Video viewing was augmented by script-reading and gap-fill activities for same, designed to tune listening skills, comprehension and spelling skills; while presenting language patterns for explaining how to do something in English. Students in all three classes, represented in this report, averaged 17 out of 20 possible points in qualitative assessments. According to the peoples encyclopedia, wikipedia, Communicative competence is a term in linguistics which refers to a language user's grammatical knowledge of syntax, morphology, phonology and the like, as well as social knowledge about how and when to use utterances appropriately.

. . . A more recent survey of communicative competence by Bachman (1990) divides it into the broad headings of "organizational competence," which includes both grammatical and discourse (or textual) competence, and "pragmatic competence," which includes both sociolinguistic and "illocutionary" competence. Strategic Competence is associated with the interlocutors' ability in using communication strategies (Faerch & Kasper, 1983; Lin, 2009).

Again, the quantitative research project was designed to determine whether increased communicative competency would have an effect on basic grammar and comprehension through standardized testing. Results were mixed and inconclusive due to averaging factors. In at least two classes, two- to three students who had very high beginning scores -- between 90 and 100 -- dropped 10-30 points on final assessments in grammar and comprehension. As oral presentations, such as How to . . . indicated communicative competence improvements among all students, one can question what was going on the last day of class when student re-took tests covering grammar and comprehension material their original high scores showed theyd previously mastered. Overall, however, there was improvement in FINAL assessments.

Section 202s initial Comprehension average (January) was 54.76. Final average (May) was 59.07. Their Grammar average in January was 46.04. Final class average 50.88. That indicates an average increase of 14.85 in Comprehension scores and 12.86 in Grammar scores among those who improved. There was a decrease (influenced greatly by the downward leap of previously high-scoring students) where two and four students went down, respectively, in Composition and Grammar;

averaging 21-point loss and 8.23 loss, again, respectively. Increases in composition test scores ranged from 2- to 24 points; while grammar increases ranged between 3.13to 25.63 points among Section 202 students. Section 203s initial Comprehension average was 80.22; and Grammar average was 65.69. Final averages were 81.31 and 76.92, respectively, in Comprehension and Grammar. There was an average 10 point increase in Comprehension scores and an average 13.55 point increase in Grammar scores among those students whose marks went up. Decrease in scores ranged from 2- to 22 points in Composition, with an average of 12 for the six who went down; and four students went down in grammar, ranging from 1.68- to 17 points. Recorded increases in test scores for comprehension ranged from 3- to 22 points; grammar increases ranged from 3.44- to 34.38 points. Again, among those whose scores dropped were people whod previously scored extremely high. There was no opportunity for cheating during initial assessments in January; so, one can only imagine what went on the last day of class, in May, when these tests were re-administered. Section 204s initial averages were 86.33 and 75.87 for Composition and Grammar, respectively. There was an average 6.8 increase in Comprehension

scores and an average 7.68 increase in grammar scores among those who improved. Two who scored very high in initial tests dropped 14- and 32 points in comprehension and 13- and 34.4 in grammar: skewing class averages to 83.52 final score for comprehension and 77.14 final score for grammar. Of this number, approximately four scores remained the same in comprehension, and 3 remained the same in grammar:

students scored neither better nor worse. Among students who improved, scores were recorded as 2- to 14 points in composition, and 3- to 10 points in grammar for Section 204.

Results are mixed and inconclusive; except that in areas of improvement, credit can be given to the overall ___________ University College English program, as students were simultaneously studying all four language skills while completing Communication Skills II.

Where students dropped points, one might argue a lack of attentiveness on the last day of class, or decreased attention to detail influenced by overconfidence in speaking abilities. Theyd just completed an oral examination before the written test was administered. Required to speak about the whole of the semesters work, using correct grammar, vocabulary and social context; students honestly earned an average 8.75 out of 10 possible points. Im offering to you copies of these assessments, with the hope that you will find

them useful at the beginning, middle or end of regular course assignments. I used these assessments when teaching the four core skills in Thailand -- at beginning, middle and end of each semester -- to provide external assessments of students abilities. This universitys Timing Schedule makes it difficult to incorporate Kenneth Beares tests into Grammar, Listening and Speaking classes; but, Ive found it quite useful in Writing, as initial assessment of students capacity. These external assessments are learning tools, as well, providing students with an idea of what they are expected to know by the end of the year; hopefully, when concepts are further learned during the

term, theyll remember and readily absorb what they know is required of a Prep Year student. I find it helpful to know where students are beginning, and how they are ending; as Communication Skills assessments measure oral performance, but ultimately these EFL Prep Year students will be judged on grammar and comprehension, as well. Thank you.

References
Bachman, L. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press. ISBN 0-19-437003-8 Canale, M. and Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics 1, 1-47. Peace, Shelia A. (2008). Vocabulary, Structure and Opportunity: Teacher Research into Task- Based Teaching/Multiple Intelligences Theory Methodology for Second Language Acquisition. http://www.scribd.com/doc/9825940/Teacher-Research-Project-Copyright-December-29-2008 Wikipedia. (2012) Communicative Competence. http://en.wikipedia.org/wiki/Communicative_competence.