7/23/2011

PerformancebasedAssessment
Rajamangala UniversityofTechnologyLanna 24July2011

BordinChinda,PhD EnglishDivision,DepartmentofWesternLanguages ChiangMaiUniversity

Performanceassessment
drivingtest Olympicdiving anartistsportfolio asurgeons1sttriplebypassoperation useofalanguage(e.g.,English)

Performanceassessment
Wecannotjudgehowwellapersoncanuse thelanguageunlesswemakethemperform thelanguage Wecannotjudgehowwelltheyperformifwe cannotspecifythetask

7/23/2011

Performanceassessment
atestinwhichtheabilityofcandidatesto performparticulartasksisassessed Tasks
designedtomeasurelearnersproductive languageskillsthroughperformances allowlearnerstoexhibitthekindsoflanguage skillsthatmayberequiredinarealworldcontext

Performanceassessment
Tasksforthesamepurposeandpopulation shouldhavethesamespecificationatasktype
specification

Everynewtaskshouldbeunique Everytaskwiththesamespecificationshould performthesameforthesamepeople,fora parallelgroupofpeopleonadifferent occasion,andregardlessofwhodoesthe scoring

Performanceassessment
Eachnewtaskneedstobevalidatedbefore beingusedinalivesituation Eachperformanceassessment needstohave anongoingqualityassuranceprocess

7/23/2011

Performanceassessment
Characteristics thereisaperformancebythecandidate theperformanceisjudgedusinganagreed setofcriteria thereisadegreeofauthenticity ofthe assessmenttasks

Performanceassessment
Alanguagetestissaidtobeauthentic when itmirrorsasexactlyaspossiblethecontent andskillsundertest,e.g. conversationtesttotestforconversation skills anessaytesttotestforwritingessayskills atestofmedicalEnglishfornonEnglish speakingdoctorswishingtoworkinEnglish medium

Performanceassessment
RATER RATING(SCORE) SCALE(Ratingcriteria/ratingrubrics) PERFORMANCE INSTRUMENT CANDIDATE

7/23/2011

Performanceassessment therater needstousearating scaleinratingaperformanceto arriveatascoreforthat performance

Performanceassessment
teachersinclassrooms(raters),arerequiredto makecomplicatedjudgements morethantherightwrongdecisionsinMC wherethestudentsresponsescanbemarked aseithercorrectorincorrect

Performanceassessment
WritingGoalsandObjectives Beforeaperformanceassessmentiswritten, theteachershouldclearlyidentifygoalsand objectivesoftheactivity. Goals arebroadstatementsofexpected studentoutcomes. Objectives dividethegoalsintoobservable behaviours.

7/23/2011

Performanceassessment
WritingGoalsandObjectives Questions: WhatdoIhopetolearnaboutmystudents' knowledgeorskills? Whatcontent,skillsandknowledgeshouldthe activitybedesignedtoassess? WhatevidencedoIneedtoevaluatethe appropriateskillsandknowledge?

Performanceassessment
Recommendationsforwritinggoalsandobjectives: Thestatementofgoalsandaccompanying objectivesshouldprovideaclearfocusforboth instructionandassessment. Bothgoalsandobjectivesshouldreflect knowledgeandinformationthatisworthwhilefor studentstolearn. Therelationshipbetweenagivengoalandthe objectivesthatdescribethatgoalshouldbe apparent.

Performanceassessment
Recommendations(Cont.): Alloftheimportantaspectsofthegivengoal shouldbereflectedthroughtheobjectives. Objectivesshoulddescribemeasurablestudent outcomes. Goalsandobjectivesshouldbeusedtoguidethe selectionofanappropriateassessmentactivity.

7/23/2011

Performanceassessment
Recommendationsfordevelopingperformance assessments: Theselectedperformanceshouldreflectavalued activity. Thecompletionofperformanceassessments shouldprovideavaluablelearningexperience. Thestatementofgoalsandobjectivesshouldbe clearlyalignedwiththemeasurableoutcomesof theperformanceactivity.

Performanceassessment
Recommendations(Cont.):
Thetaskshouldnotexamineextraneousor unintendedvariables. Performanceassessmentsshouldbefairandfree frombias.

Performanceassessment
Recommendationsforadministeringperformance assessments: Bothwrittenandoralexplanationsoftasksshouldbe clearandconciseandpresentedinlanguagethatthe studentsunderstand. Scoringrubricsshouldbediscussedwiththe studentsbeforetheycompletetheassessment activity.

7/23/2011

Assessmentcriteria
Aratingscale(orrubrics inAm.Engl.)isa scaleforthedescriptionoflanguage proficiencyconsistingofaseriesof constructedlevelsagainstwhichalanguage learnersperformanceisjudged.

Assessmentcriteria
Thelevelsorbandsarecommonly characterisedintermsofwhatsubjectscando withthelanguageandtheirmasteryof linguisticfeatures.

Assessmentcriteria
Ratingscalesalsorepresentthemostconcrete statementoftheconstructbeingmeasuredor descriptors. Thedescriptorsdescribethelevelof performancerequiredofcandidatesateach pointonaproficiencyscale.

7/23/2011

Assessmentcriteria
Recommendationsfordevelopingscoringrubrics: Thecriteriasetforthwithinascoringrubric shouldbeclearlyalignedwiththerequirements ofthetaskandthestatedgoalsandobjectives. Thecriteriasetforthinscoringrubricsshouldbe expressedintermsofobservablebehavioursor productcharacteristics. Scoringrubricsshouldbewritteninspecificand clearlanguagethatthestudentsunderstand.

Assessmentcriteria
Recommendations(Cont.):
Thenumberofpointsthatareusedinthescoring rubricshouldmakesense. Theseparationbetweenscorelevelsshouldbeclear. Thestatementofthecriteriashouldbefairandfree frombias.

Analyticvs. holisticratingscales
Withananalyticscale,ratersareaskedto judgeseveralcomponentsofaperformance separatelyastraits,criteria,ordimensionsof performance. Thesecomponentsaredividedsothatthey canbejudgedseparatelyratherthangivinga singlescorefortheentireperformance.

7/23/2011

Analyticvs. holisticratingscales
Analyticscalesareusedwhen planninginstructiontoshowrelativestrengths andweaknessesofaperformance teachingstudentsthenatureofaquality performance givingdetailedfeedback, knowinghowtopreciselydescribequalityis moreimportantthanspeed

Analyticvs. holisticratingscales
withaholisticscale,raters areaskedtogivea judgementonacandidatesperformanceasa whole,orinotherwords,asinglescoreforan entireperformancebasedonanoverall impressionofacandidateswork

Analyticvs. holisticratingscales
Holisticscalesareusedwhen speedofscoringismoreimportantthan knowingpreciselyhowtodescribequality, theperformancesaresimple whenaquicksnapshotofoverallachievement istheobjective

7/23/2011

Stepsindesigning analyticratingscales
Step1:Reexaminethelearningobjectivesto beaddressedbythetask.Thisactivityallows youtomatchyourscoringguidewithyour objectivesandactualinstruction.

Stepsindesigning analyticratingscales
Step2:Identifyspecificobservableattributes thatyouwanttosee(aswellasthoseyou dontwanttosee)yourstudentsdemonstrate intheirproduct,process,orperformance. Specifythecharacteristics,skills,or behavioursthatyouwillbelookingfor,aswell ascommonmistakesyoudonotwanttosee.

Stepsindesigning analyticratingscales
Step3:Brainstormcharacteristicsthat describeeachattribute.Identifywaysto describeaboveaverage,average,andbelow averageperformanceforeachobservable attributeidentifiedinStep2.

10

7/23/2011

Stepsindesigning analyticratingscales
Step4:Writethoroughnarrativedescriptions forexcellentworkandpoorworkforeach individualattribute. Describethehighestandlowestlevelsof performanceusingthedescriptorsforeach attributeseparately.

Stepsindesigning analyticratingscales
Step5:Completetherubricbydescribing otherlevelsonthecontinuumthatranges fromexcellenttopoorworkforthecollective attributes. Writedescriptionsforallintermediatelevels ofperformance.

Stepsindesigning analyticratingscales
Step6:Collectsamplesofstudentworkthat exemplifyeachlevel. Thesewillhelpyouscoreinthefutureby servingasbenchmarks.

11

7/23/2011

Stepsindesigning analyticratingscales
Step7:Revisethecriteria,asnecessary.Be preparedtoreflectontheeffectivenessofthe criteriaandreviseitpriortoitsnext implementation.

Ratingscales
Beginning 1 Criteria1 Description Developing 2 Accomplished 3 Exemplary 4

Score

reflecting beginning levelof performance

Description reflecting movement toward masterylevel of performance

Description reflecting achievement ofmastery levelof performance

Description reflecting highestlevel of performance

Criteria2 Criteria3

Ratingscales
Eachcriterionstatementshouldbemodifiedto describeeachleveloftheperformances attribute(s). Thechoiceofwordsthatdescribethechanging valuesoftheattributeusuallydependson verbalqualifiers.

12

7/23/2011

Ratingscales
Threeverbalqualifierscalesarecommonly used: amount(e.g.,few,some,all) frequency(e.g.,rarely,sometimes,often),and intensity(e.g.little,some,very).

Ratingscales
Meaningfuldescriptionsofperformances
criteria levels(standards) keepthefocusonthelanguage

Reliability
Areliable assessmentisonethatelicitssimilar performancesallthetime:
fromdifferentpeoplewiththesamebehaviour ondifferentoccasions onsametasktypebutdifferenttasks whendifferentpeoplerate

13

7/23/2011

Validity
Avalid assessmentstartsbyasking
whatdowewantlearnerstoknow? whatdowewantthemtobeabletodo? howwellshouldtheybeabletodoit? whatdoesitlooklike?

Avalid assessmenttellstheanswerstothese questionstoeveryoneinvolved

Sample1
Ratingcriteriaforan essay
4 Entriesarecreativelywritten.Proceduresand resultsareclearlyexplained.Reportiswell organized. 3 Entriescontainmostoftherequiredelements andareclearlywritten. 2 Entriesareincomplete.Theremaybesome spellingorgrammarerrors. 1 Writingismessyandentriescontainspelling errors.Pagesareoutoforderormissing.

Problem(consistency)
Theattributes/categoriesarenotexplicit Atlevel1,focusareonaccuracyofspelling andorganization Atlevel2,organizationisdropped(accuracy remains) Atlevel3,clarityisadded Atlevel4,vaguerequirementofcreativityis added

14

7/23/2011

Suggestions
Requiredelements 4 3 2 1 Organization Clarity

Suggestions
4 3 2 1 Requiredelements All oftherequiredelementsarepresent Most oftherequiredelementsarepresent Some oftherequiredelementsarepresent Few oftherequiredelementsarepresent

Suggestions
4 3 2 1 Organization Thereportis extremely organized. Thereportis mainly organized. Thereportis moderately organized. Thereportis slightly organized.

15

7/23/2011

Suggestions
4 3 2 1 Clarity Thereportis extremely clear. Thereportis mainly clear. Thereportis moderately clear. Thereportis slightly clear.

Task
AssessmentCriteriaforGroupInteraction
Pronunciation& Delivery 6 5 4 3 2 1 0 Communication Strategies Vocabulary& Language Patterns Ideas& Organization

References
Brown,J.D.(2005).Testinginlanguageprogram:acomprehensiveguidetoEnglishlanguageassessment. NewYork:McGrawHill. Davies,A.,Brown,A.,Elder,C.,Hill,K.,Lumley,T.andMcNamara,T.(1999).Dictionaryoflanguagetesting. Cambridge:CambridgeUniversityPress. Davison,C.(2007).Viewsfromthechalkface:EnglishlanguageschoolbasedassessmentinHongKong. LanguageAssessmentQuarterly,4(1),3768. Fulcher,G.andDavidson,F.(2007).Languagetestingandassessment:anadvancedresourcebook.Oxon: Routledge. HampLyons,L.(2007).Theimpactoftestingpracticesonteaching:Ideologiesandalternative.InJ. Cummins&C.Davison(Eds.),InternationalhandbookofEnglishlanguageteaching(Vol.PartI,pp.487 504).NewYork:Springer. Hughes,A.(2003).Testingforlanguageteachers.Cambridge:CambridgeUniversityPress. Mertler,CraigA.(2001).Designingscoringrubricsforyourclassroom.PracticalAssessment,Research& Evaluation,7(25).RetrievedMay5,2011fromhttp://PAREonline.net/getvn.asp?v=7&n=25. McNamara,T.(2000).Languagetesting.Oxford:OxfordUniversityPress. Moskal,BarbaraM.(2003).Recommendationsfordevelopingclassroomperformanceassessmentsand scoringrubrics.PracticalAssessment,Research&Evaluation,8(14).RetrievedJuly23,2011from http://PAREonline.net/getvn.asp?v=8&n=14.

16