Beruflich Dokumente
Kultur Dokumente
BillCerbin
UWLaCrosse,CenterforAdvancingTeaching&Learning
Prioritizethesubjectmatter
Whatsubjectmattertopics,ideas,conceptsare
1. Essential
2. Important
3. Worthbeingfamiliarwith
Useitemsthatmeasurethecourse/unitlearning
objectives.Whatarethemajorobjectivesoftheunitor
coursethatshouldbemeasuredbythetest?
CreateaTestPlan(TableofSpecifications)that
classifieseachitemaccordingtowhattopicorconcept
ittestsANDwhatobjectiveitaddresses.Thetablecan
helpyouwriteatestthathascontentvaliditythereisamatchbetweenwhatwastaughtandwhatis
tested.Thetablehelpsinsurethatyou
1. emphasizethesamecontentyouemphasizedindaytodayinstruction(e.g.,moreitemsabout
topicXandfewerabouttopicYbecauseyouconsiderXtobemoreimportantandyouspent
moretimeonX)
2. aligntestitemswithlearningobjectives(e.g.,importanttopicsmightincludeitemsthattest
interpretation,application,prediction,andunimportanttopicsmightbetestedonlywithsimpler
recognitionitems)
3. donotoverlookorunderemphasizeanareaofcontent
TestPlan(TableofSpecifications)Format
SubjectMatter
concepts,topics,ideas
TopicA
TopicB
TopicC
TopicD
Knowledge(recall)
LearningObjectives
Analyze
Apply
Interpret
Total
Etc.
Writematchingpairsofitemsaftereachclass.Whiletheclassperiodisstillfreshwrite23itemsthat
focusonthemostimportantmaterialfromtheclass.Ideally,writematchingpairsofitems,i.e.,two
itemsthattestthesamething.Useoneforformativeevaluation(e.g.,noncreditquizorreview)andthe
otheroneforsummativeevaluation,i.e.,thetest).
Thequestionstem
1. Therearetwoacceptableformsofstemsa)onethatposesacomplete,directquestionandb)
onethatisastatementthatneedstobecompleted.
Directquestion:Whatcolorresultsfromthemixtureofequalpartsofyellowandblue
paint?
Completion:TheactualelectionoftheU.S.presidenttoofficeisdoneby
2. Putallrelevantinformationinthestemandmakeitsufficientlyspecifictostandonitsown
withoutqualification.
3. Alsotrytomakethestemshortandsharplyfocused.
4. Forquestionsthataskforrecallofinformationdonotintroduceinformationintothestemthat
hasnotbeenpartofthematerialcoveredinclass.
5. Avoidnegativewordingbecauseitcanconfusestudents.
Bad:WhichofthefollowingisnotacharacteristicofBrutalism?
Better:WhichofthefollowingbestdistinguishesBrutalismfromotherarchitectural
movement?
Correct/Bestanswer
1. Writethebestanswerfirst,thenresponsealternatives
2. Besurethereisonlyonebestanswer
Responsealternatives(AKAresponseoptionsordistractors)
1. Shouldbebriefandgrammaticallyparallel.Makeresponsealternativegrammaticallyconsistent
withthestem.Anexampleofagrammaticallyinconsistentitem:
ThefunctionsoftheFederalReservearetoprovidethenationwithanelasticmoneysupplyandto
A. helpstabilizetheeconomy
B. correctionofnationalincomestatistics
C. correctionoftaxlaws
D. helplevypropertytaxes
2. Makeresponsesroughlyequalinlength.FYIInstructorstendtomakethebestanswerlonger.
3. Eliminaterepetitiveinformationfromresponsealternatives
Bad:Between1950and1965
A. Interestratesincreased
B. Interestratesdecreased
C. Interestratesfluctuatedgreatly
D. Interestratesdidnotchange
Better:Whatwasthetrendininterestratesbetween1950
and1965?
A. Increasedonly
B. Decreasedonly
C. Increased,thendecreased
D. Remainedunchanged
4. Takespecialcareoravoidusingnoexceptionwordssuchasnever,all,none,andalways.
Thesewordstendtogivecluesaboutthecorrectanswer.Fewstatementsareabsoluteor
universallytrue.
5. Avoidusingalloftheaboveasanalternative.Ifthestudentcanidentifyonealternativethatis
nottruetheniteliminatesthatalternativeplusalloftheabove,makingiteasiertoguess.
6. Varythepositionofthebestanswer.InstructorstendtoputthebestanswerintheBorC
position.
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009
7. Varythenumberofresponseoptionsasappropriate.
A. Itisacceptablepracticetovarythenumberofresponsealternativesonatest.
B. Thereisnosinglebestnumberofresponsealternatives.Researchindicatesthat3
alternativesisjustaseffectiveasfour.
C. Ifappropriate,varythenumberofresponsealternativeswithinthesametest.
8. Avoidusingalloftheaboveornoneoftheabovejusttofillspace
Analternativetotheallofformatistoaskstudentstofindallcorrectanswerstoanitem.
Forexample,Acceptablepracticesforresponsealternativesonmultiplechoicetestsinclude
A.Thereisnosinglebestnumberorresponsealternativesformultiplechoiceitems
B.Avoidusingalloftheaboveandnoneoftheaboveasresponsealternatives
C.Responsealternativesshouldbeplausibleanswers
D.Responsealternativesforatestitemshouldberoughlyequalinlength
Caveat:Scantroncanthandlemultiplecorrectresponses
9. Ifyouhavedifficultycreatingenoughplausibleincorrectanswersforanitem,putitonthetestas
afillinitemandstudentswillproducearangeofincorrectresponsesthatyoucanusenexttime.
Overalltestconstruction
1. TestInstructions.
Instructstudentstoselectthebestanswerandacknowledgethatsometimesthe
responseoptionshaveelementsofaccuracy,butthereisabestanswerforeachitem.
Usingthecorrectanswerinvitesargumentsfromstudentsthattheiranswersare
correctaswell.
Statewhethertherearerewardsorpenaltiesforguessing.
Typesofadvantages:mightgetadditionalcredit;instructorallowsstudentstojustify
theirguessesandgivespartialcredit.
Typeofpenalty:Instructordeductsadditionalpointsforincorrectanswers.
Considerusing:Idontknowasaresponsealternative.
2. Includeafeweasyitemsatthebeginningofthetest.Youcanhelptoreducetestanxietyby
includingafeweasyitemstostartthetest.
3. Grouptestitemsrelatedtothesametopics.Itemsrelatedtoaspecifictopicshouldbegrouped
togetheronthetest.Thisallowsstudentstothinkabouteachtopicandsectionofthematerial
ratherthanjumpfromtopictotopic.Considerusinglabelsorheadingstoindicatetopicsand
thengrouprelateditemsundertheheadings.
4. Testlength.Ifyouwanttotestwhatstudentsknow,asopposedtohowfasttheycanreadand
respond,besuretogivesufficienttimetocompletethetest.Otherwise,youpenalizepoor
readers,slowreaders,deliberateandreflectiveresponding.Studentscancomplete12multiple
choiceitemsperminute.
5. Timelimits.Studentswithcertainformsoflearningdisabilitiesneedlongertocompleteatest.
StudentswhosefirstlanguageisnotEnglishwillalsoneedlonger.Whileyoucanlikely
accommodatesuchstudentsindividually,considercarefullywhateducationalgoalwouldlead
youtoimposeatimelimitonatestatall.Anotherclassfollowingintoyourroomisnotagood
3
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009
educationalreasontorefusetoallowastudenttofinishatest.Numerousshorttestsorquizzes
workbetterformoststudentsthanbig,highstakestests.
6. Testtakingaccommodations:StudentswithdisabilitiesmustregisterwithDisabilityResource
Services(DRS)toqualifyforaccommodations.DRSwillletyouknowwhatkindsof
accommodationsastudentneedsfortesttaking.Inadditiontomoretime,anothercommon
needisforaseparate,quietroom.Hallwaysarepoorchoicesbecauseofdistractions.
Improvingitemsusingfeedbackanditemanalysis
1. Feedbackfromfellowinstructors
2. FeedbackfromstudentsReviewtestresultswithstudents;opportunitytogivefeedbackto
studentsandgetfeedbackfromthemabouttestitems
3. ItemAnalysis.IfyouhavetestsscoredbyITyoucanrequestanitemanalysisthatprovidesitem
difficultyanditemdiscrimination.IThasahandoutthathelpsyouinterprettheanalysis.Askfor
RevisedTestScoringDisplayandItemAnalysis.
A. itemdifficultyPercentageofstudentswhocorrectlyansweritem.
Desirabledifficultylevelsfordifferenttypesofitems
QuestionType
5responseoptions
4responseoptions
3responseoptions
True/False
ItemDifficulty
%correct
60%
62%
66%
75%
Reviewitemsattheextremesveryeasy,>90%correct,ordifficult,<20%correct.
B. itemdiscriminationRelationshipbetweenhowwellstudentsdoonitemandhowwell
theydooverallonthetest.Highdiscriminationisgooditmeansthatstudentswhodo
wellonthetesttendtogettheitemcorrectandthosewhodopoorlyonthetesttendto
getitincorrect
HowWellItem
Discriminates
Verygood
Good
Fairlygood
Poor
Discrimination
Value
>.40
.30.39
.2029
<.20
Highdiscrimination(>.40):Studentswithhightestscoresrespondedcorrectlyand
studentswithlowscoresrespondedincorrectly.
Verylowdiscrimination(<.19):Studentswithhightestscoresdopoorlyontheitem,and
lowscoringstudentsmaydobetter.
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009
ExamplesofDifferentTypesofMultipleChoiceItems
Multiplechoicequestionsaskstudentstodiscriminateamongdifferentplausibleoptionsandselectthe
bestanswer.Typically,theseitemsarewrittenatabasiclevelandstudentscananswerthequestion
basedonrecognitionorrecallmemory.Thereisnothingwrongwithitemsthattestforbasicknowledge
orfamiliarity.But,whatifyouwanttotestmorecomplexobjectivessuchasunderstanding,problem
solving,oranalyticalreasoning?
Totestcomplexlearningobjectivesyoucaneither1)usedifferentformsofitemssuchasessay,short
answer,performancetasks)or2)developmultiplechoiceitemsthataddresstheobjectives.
Multiplechoicequestionsthatmeasurecomplexlearningobjectives.Itischallengingtowritemultiple
choicequestionsthattestcomplexlearning.Typicallytheseitemspresentthestudentwithascenario,
passage,orgraphicrepresentationandthenposeseveralquestionsrelatedtothematerial.Thesecan
beusedtoassesscomplexthinkingabilitiessuchasprediction,interpretation,application,orevaluation.
Herearesomealternativetypesofmultiplechoiceitems(Davis,2009,pp.391393).
Predictionquestion.Inthestemofthequestionpresentaproblemorsituationandaskstudentsto
predictanoutcome.
Identifyaprincipleortheoryquestion.Inthestemofthequestionpresentanexampleofsome
phenomenonandaskstudentstoidentifyaprincipleortheorythatitillustrates.
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009
Assertionreasonquestions(ARQ).Thestemofthesequestionsincludestwostatementsanassertion
andareasonlinkedbybecause(Davis,2009,p.393)
Youaretheteacherquestion.Thestemofthequestionasksthestudenttoassumetheroleofteacher
andthenevaluateshortwrittenpassages(i.e.,flawedshorttestanswers).
Multiplechoice+shortwrittenjustification.Ifyouwanttoknowthereasoningbehindtheiranswersask
studentstowriteashortjustificationofwhytheyselectedtheirspecificchoice.
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
6
WritingBetterObjectiveTests
November25,2009
ConcepTests.Harvardphysicist,EricMazur,usesConcepTeststoevaluatestudentsconceptual
understanding.Typically,thestemoftheitemincludesadiagram,pictureordescriptionofasituation
thatillustratesaphysicsconcept,andasksthestudenttopredictanoutcome.Belowaresomeexamples
fromMazur,E.(1997).Peerinstruction:Ausersmanual.PrenticeHall,UpperSaddleRiver,NJ.
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009
HandoutReferences
Slavin,R.E.(2000).Educationalpsychology:Theoryandpractice,6thedition.AllynandBacon:Boston,
MA.
Davis,B.G.(2009).Toolsforteaching,2ndedition.JosseyBassPublisher:SanFrancisco.
Stiggins,R.J.(1994).StudentCenteredClassroomAssessment.MacmillanPublisher:NY.
Wiggins,G.(1998).Understandingbydesign.AssociationfoSupervisionandCurriculumDevelopment:
Alexandria,VA.
AvailablefromIT,RevisedTestScoringDisplayandItemAnalysis.
OnlineResourcerelatedtowritingobjectivetests
http://testing.byu.edu/info/handbooks/betteritems.pdf
StevenJ.Burton,etal.HowtoPrepareBetterMultipleChoiceTestItems:GuidelinesforUniversity
Faculty,BrighamYoungUniversityTestingServicesandTheDepartmentofInstructionalScience,1991.
Allyouneedinoneplace.Explainstheadvantagesandlimitationsofmultiplechoicetests,howto
decidewhetheryoushouldusesuchtestsornot,howtowritequestionsthatmeasuremorecomplex
typesoflearningthanrecalloffacts,andprovidesavarietyofformatsformultiplechoicequestions.
Evenhasachecklistattheend.
CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009