WritingBetterObjectiveTests

BillCerbin
UWLaCrosse,CenterforAdvancingTeaching&Learning

Prioritizethesubjectmatter
Whatsubjectmattertopics,ideas,conceptsare
1. Essential
2. Important
3. Worthbeingfamiliarwith

Useitemsthatmeasurethecourse/unitlearning
objectives.Whatarethemajorobjectivesoftheunitor
coursethatshouldbemeasuredbythetest?

CreateaTestPlan(TableofSpecifications)that
classifieseachitemaccordingtowhattopicorconcept
ittestsANDwhatobjectiveitaddresses.Thetablecan
helpyouwriteatestthathascontentvaliditythereisamatchbetweenwhatwastaughtandwhatis
tested.Thetablehelpsinsurethatyou
1. emphasizethesamecontentyouemphasizedindaytodayinstruction(e.g.,moreitemsabout
topicXandfewerabouttopicYbecauseyouconsiderXtobemoreimportantandyouspent
moretimeonX)
2. aligntestitemswithlearningobjectives(e.g.,importanttopicsmightincludeitemsthattest
interpretation,application,prediction,andunimportanttopicsmightbetestedonlywithsimpler
recognitionitems)
3. donotoverlookorunderemphasizeanareaofcontent

TestPlan(TableofSpecifications)Format
SubjectMatter
concepts,topics,ideas
TopicA
TopicB
TopicC
TopicD

Knowledge(recall)

LearningObjectives
Analyze
Apply

Interpret

Total

Etc.

Writematchingpairsofitemsaftereachclass.Whiletheclassperiodisstillfreshwrite23itemsthat
focusonthemostimportantmaterialfromtheclass.Ideally,writematchingpairsofitems,i.e.,two
itemsthattestthesamething.Useoneforformativeevaluation(e.g.,noncreditquizorreview)andthe
otheroneforsummativeevaluation,i.e.,thetest).

Thequestionstem
1. Therearetwoacceptableformsofstemsa)onethatposesacomplete,directquestionandb)
onethatisastatementthatneedstobecompleted.
Directquestion:Whatcolorresultsfromthemixtureofequalpartsofyellowandblue
paint?
Completion:TheactualelectionoftheU.S.presidenttoofficeisdoneby

2. Putallrelevantinformationinthestemandmakeitsufficientlyspecifictostandonitsown
withoutqualification.
3. Alsotrytomakethestemshortandsharplyfocused.
4. Forquestionsthataskforrecallofinformationdonotintroduceinformationintothestemthat
hasnotbeenpartofthematerialcoveredinclass.
5. Avoidnegativewordingbecauseitcanconfusestudents.
Bad:WhichofthefollowingisnotacharacteristicofBrutalism?
Better:WhichofthefollowingbestdistinguishesBrutalismfromotherarchitectural
movement?

Correct/Bestanswer
1. Writethebestanswerfirst,thenresponsealternatives
2. Besurethereisonlyonebestanswer

Responsealternatives(AKAresponseoptionsordistractors)
1. Shouldbebriefandgrammaticallyparallel.Makeresponsealternativegrammaticallyconsistent
withthestem.Anexampleofagrammaticallyinconsistentitem:
ThefunctionsoftheFederalReservearetoprovidethenationwithanelasticmoneysupplyandto
A. helpstabilizetheeconomy
B. correctionofnationalincomestatistics
C. correctionoftaxlaws
D. helplevypropertytaxes

2. Makeresponsesroughlyequalinlength.FYIInstructorstendtomakethebestanswerlonger.

3. Eliminaterepetitiveinformationfromresponsealternatives

Bad:Between1950and1965
A. Interestratesincreased
B. Interestratesdecreased
C. Interestratesfluctuatedgreatly
D. Interestratesdidnotchange

Better:Whatwasthetrendininterestratesbetween1950
and1965?
A. Increasedonly
B. Decreasedonly
C. Increased,thendecreased
D. Remainedunchanged

4. Takespecialcareoravoidusingnoexceptionwordssuchasnever,all,none,andalways.
Thesewordstendtogivecluesaboutthecorrectanswer.Fewstatementsareabsoluteor
universallytrue.

5. Avoidusingalloftheaboveasanalternative.Ifthestudentcanidentifyonealternativethatis
nottruetheniteliminatesthatalternativeplusalloftheabove,makingiteasiertoguess.

6. Varythepositionofthebestanswer.InstructorstendtoputthebestanswerintheBorC
position.

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009

7. Varythenumberofresponseoptionsasappropriate.
A. Itisacceptablepracticetovarythenumberofresponsealternativesonatest.
B. Thereisnosinglebestnumberofresponsealternatives.Researchindicatesthat3
alternativesisjustaseffectiveasfour.
C. Ifappropriate,varythenumberofresponsealternativeswithinthesametest.

8. Avoidusingalloftheaboveornoneoftheabovejusttofillspace
Analternativetotheallofformatistoaskstudentstofindallcorrectanswerstoanitem.
Forexample,Acceptablepracticesforresponsealternativesonmultiplechoicetestsinclude
A.Thereisnosinglebestnumberorresponsealternativesformultiplechoiceitems
B.Avoidusingalloftheaboveandnoneoftheaboveasresponsealternatives
C.Responsealternativesshouldbeplausibleanswers
D.Responsealternativesforatestitemshouldberoughlyequalinlength
Caveat:Scantroncanthandlemultiplecorrectresponses

9. Ifyouhavedifficultycreatingenoughplausibleincorrectanswersforanitem,putitonthetestas
afillinitemandstudentswillproducearangeofincorrectresponsesthatyoucanusenexttime.

Overalltestconstruction
1. TestInstructions.
Instructstudentstoselectthebestanswerandacknowledgethatsometimesthe
responseoptionshaveelementsofaccuracy,butthereisabestanswerforeachitem.
Usingthecorrectanswerinvitesargumentsfromstudentsthattheiranswersare
correctaswell.
Statewhethertherearerewardsorpenaltiesforguessing.
Typesofadvantages:mightgetadditionalcredit;instructorallowsstudentstojustify
theirguessesandgivespartialcredit.
Typeofpenalty:Instructordeductsadditionalpointsforincorrectanswers.
Considerusing:Idontknowasaresponsealternative.
2. Includeafeweasyitemsatthebeginningofthetest.Youcanhelptoreducetestanxietyby
includingafeweasyitemstostartthetest.
3. Grouptestitemsrelatedtothesametopics.Itemsrelatedtoaspecifictopicshouldbegrouped
togetheronthetest.Thisallowsstudentstothinkabouteachtopicandsectionofthematerial
ratherthanjumpfromtopictotopic.Considerusinglabelsorheadingstoindicatetopicsand
thengrouprelateditemsundertheheadings.
4. Testlength.Ifyouwanttotestwhatstudentsknow,asopposedtohowfasttheycanreadand
respond,besuretogivesufficienttimetocompletethetest.Otherwise,youpenalizepoor
readers,slowreaders,deliberateandreflectiveresponding.Studentscancomplete12multiple
choiceitemsperminute.
5. Timelimits.Studentswithcertainformsoflearningdisabilitiesneedlongertocompleteatest.
StudentswhosefirstlanguageisnotEnglishwillalsoneedlonger.Whileyoucanlikely
accommodatesuchstudentsindividually,considercarefullywhateducationalgoalwouldlead
youtoimposeatimelimitonatestatall.Anotherclassfollowingintoyourroomisnotagood
3

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009

educationalreasontorefusetoallowastudenttofinishatest.Numerousshorttestsorquizzes
workbetterformoststudentsthanbig,highstakestests.
6. Testtakingaccommodations:StudentswithdisabilitiesmustregisterwithDisabilityResource
Services(DRS)toqualifyforaccommodations.DRSwillletyouknowwhatkindsof
accommodationsastudentneedsfortesttaking.Inadditiontomoretime,anothercommon
needisforaseparate,quietroom.Hallwaysarepoorchoicesbecauseofdistractions.

Improvingitemsusingfeedbackanditemanalysis
1. Feedbackfromfellowinstructors
2. FeedbackfromstudentsReviewtestresultswithstudents;opportunitytogivefeedbackto
studentsandgetfeedbackfromthemabouttestitems
3. ItemAnalysis.IfyouhavetestsscoredbyITyoucanrequestanitemanalysisthatprovidesitem
difficultyanditemdiscrimination.IThasahandoutthathelpsyouinterprettheanalysis.Askfor
RevisedTestScoringDisplayandItemAnalysis.
A. itemdifficultyPercentageofstudentswhocorrectlyansweritem.
Desirabledifficultylevelsfordifferenttypesofitems

QuestionType
5responseoptions
4responseoptions
3responseoptions
True/False

ItemDifficulty
%correct
60%
62%
66%
75%

Reviewitemsattheextremesveryeasy,>90%correct,ordifficult,<20%correct.

B. itemdiscriminationRelationshipbetweenhowwellstudentsdoonitemandhowwell
theydooverallonthetest.Highdiscriminationisgooditmeansthatstudentswhodo
wellonthetesttendtogettheitemcorrectandthosewhodopoorlyonthetesttendto
getitincorrect
HowWellItem
Discriminates
Verygood
Good
Fairlygood
Poor

Discrimination
Value
>.40
.30.39
.2029
<.20

Highdiscrimination(>.40):Studentswithhightestscoresrespondedcorrectlyand
studentswithlowscoresrespondedincorrectly.

Verylowdiscrimination(<.19):Studentswithhightestscoresdopoorlyontheitem,and
lowscoringstudentsmaydobetter.

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009

ExamplesofDifferentTypesofMultipleChoiceItems

Multiplechoicequestionsaskstudentstodiscriminateamongdifferentplausibleoptionsandselectthe
bestanswer.Typically,theseitemsarewrittenatabasiclevelandstudentscananswerthequestion
basedonrecognitionorrecallmemory.Thereisnothingwrongwithitemsthattestforbasicknowledge
orfamiliarity.But,whatifyouwanttotestmorecomplexobjectivessuchasunderstanding,problem
solving,oranalyticalreasoning?

Totestcomplexlearningobjectivesyoucaneither1)usedifferentformsofitemssuchasessay,short
answer,performancetasks)or2)developmultiplechoiceitemsthataddresstheobjectives.

Multiplechoicequestionsthatmeasurecomplexlearningobjectives.Itischallengingtowritemultiple
choicequestionsthattestcomplexlearning.Typicallytheseitemspresentthestudentwithascenario,
passage,orgraphicrepresentationandthenposeseveralquestionsrelatedtothematerial.Thesecan
beusedtoassesscomplexthinkingabilitiessuchasprediction,interpretation,application,orevaluation.
Herearesomealternativetypesofmultiplechoiceitems(Davis,2009,pp.391393).

Predictionquestion.Inthestemofthequestionpresentaproblemorsituationandaskstudentsto
predictanoutcome.

Identifyaprincipleortheoryquestion.Inthestemofthequestionpresentanexampleofsome
phenomenonandaskstudentstoidentifyaprincipleortheorythatitillustrates.

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009


Assertionreasonquestions(ARQ).Thestemofthesequestionsincludestwostatementsanassertion
andareasonlinkedbybecause(Davis,2009,p.393)

Youaretheteacherquestion.Thestemofthequestionasksthestudenttoassumetheroleofteacher
andthenevaluateshortwrittenpassages(i.e.,flawedshorttestanswers).

Multiplechoice+shortwrittenjustification.Ifyouwanttoknowthereasoningbehindtheiranswersask
studentstowriteashortjustificationofwhytheyselectedtheirspecificchoice.

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
6
WritingBetterObjectiveTests
November25,2009

ConcepTests.Harvardphysicist,EricMazur,usesConcepTeststoevaluatestudentsconceptual
understanding.Typically,thestemoftheitemincludesadiagram,pictureordescriptionofasituation
thatillustratesaphysicsconcept,andasksthestudenttopredictanoutcome.Belowaresomeexamples
fromMazur,E.(1997).Peerinstruction:Ausersmanual.PrenticeHall,UpperSaddleRiver,NJ.

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009

HandoutReferences

Slavin,R.E.(2000).Educationalpsychology:Theoryandpractice,6thedition.AllynandBacon:Boston,
MA.

Davis,B.G.(2009).Toolsforteaching,2ndedition.JosseyBassPublisher:SanFrancisco.

Stiggins,R.J.(1994).StudentCenteredClassroomAssessment.MacmillanPublisher:NY.

Wiggins,G.(1998).Understandingbydesign.AssociationfoSupervisionandCurriculumDevelopment:
Alexandria,VA.

AvailablefromIT,RevisedTestScoringDisplayandItemAnalysis.

OnlineResourcerelatedtowritingobjectivetests

http://testing.byu.edu/info/handbooks/betteritems.pdf
StevenJ.Burton,etal.HowtoPrepareBetterMultipleChoiceTestItems:GuidelinesforUniversity
Faculty,BrighamYoungUniversityTestingServicesandTheDepartmentofInstructionalScience,1991.
Allyouneedinoneplace.Explainstheadvantagesandlimitationsofmultiplechoicetests,howto
decidewhetheryoushouldusesuchtestsornot,howtowritequestionsthatmeasuremorecomplex
typesoflearningthanrecalloffacts,andprovidesavarietyofformatsformultiplechoicequestions.
Evenhasachecklistattheend.

CATLColloquiumHandoutAvailableonlineathttp://catl.typepad.com/
WritingBetterObjectiveTests
November25,2009