7/27/2015

InterpretingItemAnalysis

InterpretingItemAnalysis

Let'slookatwhatwehaveandseewhatwecansee
90%ofitemanalysisisjustcommonsense...
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

PotentialMiskey
IdentifyingAmbiguousItems
EqualDistributiontoallalternatives.
Alternativesarenotworking
Distractertooatractive.
Questionnotdiscriminating.
Negativediscrimination.
TooEasy.
Omit.
&11.RelationshipbetweenDindexandDifficulty(p).
ItemAnalysisofComputerPrintouts

.
1. Whatdoweseelookingatthisfirstone?[PotentialMiskey]

UpperLowDifferenceDTotalDifficulty
1.*A143.25.17
B13
C105
D33
O<meansomitornoanswer
#1,morehighgroupstudentschoseCthanA,eventhoughAissupposedlythecorrect
answer
morelowgroupstudentschoseAthanhighgroupsogotnegativediscrimination
only.16%ofclassgotitright
mostlikelyyoujustwrotethewronganswerkeydown
>thisisaneasyandverycommonmistakeforyoutomake
betteryoufindoutnowbeforeyouhandbackthenwhenkidscomplain
ORWORSE,theydon'tcomplain,andteachthemselvesthatyourmiskeyasthe
"correct"answer
socheckitoutandrescorethatquestiononallthepapersbeforehandingthemback
Makesit105Difference=5D=.34Total=15difficulty=.50
>niceitem
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

1/10

7/27/2015

InterpretingItemAnalysis

OR:
youcheckandfindthatyoudidn'tmiskeyit>thatistheansweryouthought
twopossibilities:
1. onepossibilityisthatyoumadeslipofthetongueandtaughtthemthewrong
answer
anythingyousayinclasscanbetakendownandusedagainstyouonan
examination....
2. morelikelymeanseven"good"studentsarebeingtrickedbyacommon
misconception>
You'renotsupposedtohavetrickquestions,somaywanttodumpit
>givethosewhogotitrighttheirpoint,buttotalrestofthemarksoutof
24insteadof25
Ifscoresarehigh,oryouwanttomakeapoint,mightletitstand,andthenteachtoit>
sometimesiftheygetcaught,willhelpthemtorememberbetterinfuture
suchas:
veryfinedistinctions
crucialstepswhichareoftenoverlooked
REVISEitfornexttimetoweaken"B"
alternativesarenotsupposedtodrawmorethanthekeyedanswer
almostalwaysanitemflaw,ratherthanusefuldistinction
2. Whatcanweseewith#2:[Canidentifyambiguousitems]

UpperLowDifferenceDTotalDifficulty
2.A65
B12
*C752.1312.40
D13
O
#2,aboutequalnumbersoftopstudentswentforAandD.
Suggeststheycouldn'ttellwhichwascorrect
either,studentsdidn'tknowthismaterial(inwhichcaseyoucanreteachit)
ortheitemwasdefective>
lookattheirfavoritealternativeagain,andseeifyoucanfindanyreasontheycouldbe
choosingit
oftenitemsthatlookperfectlystraightforwardtoadultsareambiguoustostudents
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

2/10

7/27/2015

InterpretingItemAnalysis

FavoriteExamplesofambiguousitems.
ifyouNOWrealizethatDwasadefensibleanswer,rescorebeforeyouhanditbackto
giveeveryonecreditforeitherAorDavoidsarguingwithyouinclass
ifit'sclearlyawronganswer,thenyounowknowwhicherrormostofyourstudentsare
makingtogetwronganswer
usefuldiagnosticinformationontheirlearning,yourteaching
3. Equallytoallalternatives

UpperLowDifferenceDTotalDifficulty
3.A43
B34
*C541.069.30
D34
O
item#3,studentsrespondaboutequallytoallalternatives
usuallymeanstheyareguessing
Threepossibilities:
1. maybematerialyoudidn'tactuallygettoyet
youdesignedtestinadvance(becauseI'veconvincedyoutoplanahead)but
didn'tactuallygeteverythingcoveredbeforeholidays....
oritemonacommonexamthatyoudidn'tstressinyourclass
2. itemsobadlywrittenstudentshavenoideawhatyou'reasking
3. itemsodifficultstudentsjustcompletelybaffled
reviewtheitem:
ifbadlywritten(byotherteacher)oronmaterialyourclasshasn'ttaken,tossit
out,rescoretheexamoutoflowertotal
BUTgivecredittothosethatgotit,toatotalof100%
ifseemswellwritten,buttoohard,thenyouknowto(re)teachthismaterialfor
restofclass....
maybethe3whogotitaretopthreestudents,
toughbutvaliditem:
OK,ifitemtestsvalidobjective
wanttoprovideoccasionalchallengingquestionfortopstudents
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

3/10

7/27/2015

InterpretingItemAnalysis

butmakesureyouhaven'tdefined"top3students"as"thoseabletofigure
outwhattheheckI'mtalkingabout"

4. Alternativesaren'tworking

UpperLowDifferenceDTotalDifficulty
4.A15
*B1477.4721.77
C02
D00
O
example#4>noonefellforD>soitisnotaplausiblealternative
questionisfineforthisadministration,butreviseitemfornexttime
tossalternativeD,replaceitwithsomethingmorerealistic
eachdistracterhastoattractatleast5%ofthestudents
classof30,shouldgetatleasttwostudents
ormightacceptoneifyoupositivelycan'tthinkofanotherfourthalternative
otherwise,donotreusetheitem
iftwoalternativesdon'tdrawanystudents>mightconsiderredoingastrue/false
5. Distractertooattractive

UpperLowDifferenceDTotalDifficulty
5.A710
B12
C11
*D523.207.23
O

sample#5>toomanygoingforA
>noONEdistractershouldgetmorethankey
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

4/10

7/27/2015

InterpretingItemAnalysis

>noonedistractershouldpullmorethanabouthalfofstudents
doesn'tleaveenoughforcorrectanswerandfivepercentforeachalternative
keepforthistime
weakenitfornexttime
6. Questionnotdiscriminating

UpperLowDifferenceDTotalDifficulty
6.*A770.0014.47
B32
C21
D35
O
sample#6:lowgroupgetsitasoftenashighgroup
onnormreferencedtests,pointistorankstudentsfrombesttoworst
soindividualtestitemsshouldhavegoodstudentsgetquestionright,poorstudentsget
itwrong
testoveralldecideswhoisagoodorpoorstudentonthisparticulartopic
thosewhodowellhavemoreinformation,skillsthanthosewhodolesswell
soifonaparticularquestionthosewithmoreskillsandknowledgedoNOTdo
better,somethingmaybewrongwiththequestion
questionmaybeVALID,butofftopic
E.G.:restoftestteststhinkingskill,butthisisamemorizationquestion,skilled
andunskilledequallyaslikelytorecalltheanswer
shouldhavehomogeneoustest>don'thaveamathiteminwithsocialstudies
ifwantedtogetreallyfancy,shoulddoseparateitemanalysisforeachcellof
yourblueprint...aslongasyouhadsixitemspercell
questionisVALID,ontopic,butnotRELIABLE
addressesthespecifiedobjective,butisn'tausefulmeasureofindividual
differences
askingGrade10sCapitalofCanadaisontopic,butsincetheywillallgetitright,
won'tshowindividualdifferencesgiveyoulowD
7. NegativeDiscrimination
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

5/10

7/27/2015

InterpretingItemAnalysis

UpperLowDifferenceDTotalDifficulty
7.*A7103.2017.57
B33
C21
D31
O
D(discrimination)indexisjustuppergroupminuslowergroup
variesfrom+1.0to1.0
ifalltopgotitright,alllowergotitwrong=100%=+1
ifmoreofthebottomgroupgetitrightthanthetopgroup,yougetanegativeDindex
ifyouhaveanegativeD,meansthatstudentswithlessskillsandknowledgeoverall,
aregettingitrightmoreoftenthanthosewhothetestsaysarebetteroverall
inotherwords,thebetteryouare,themorelikelyyouaretogetitwrong
WHATCOULDACCOUNTFORTHAT?
Twopossibilities:
usuallymeansanambiguousquestion
thatisconfusinggoodstudents,butweakstudentstooweaktoseetheproblem
lookatquestionagain,lookatalternativesgoodstudentsaregoingfor,toseeif
you'vemissedsomething
OR:
oritmightbeofftopic
>somethingweakerstudentsarebetterat(likerotememorization)thangood
students
>notpartofsamesetofskillsasrestoftest>suggestsdesignflawwithtableof
specificationsperhaps
((ifyouendupwithawholebunchofDindicesonthesametest,mustmeanyouactually
havetwodifferentdistinctskills,becausebydefinition,thelowgroupisthehighgroupon
thatbunchofquestions
>enduptreatingthemastwoseparatetests))
ifyouhavealargeenoughsample(liketheprovincialexams)thenwetosstheitem
andeitherdon'tcountitorgiveeveryonecreditforit
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

6/10

7/27/2015

InterpretingItemAnalysis

withsampleof100studentsorless,couldjustberandomchance,sobasicallyignoreit
intermsofTHISadministration
kidswroteit,givethemmarktheygot
furthermore,ifyoukeepdroppingquestions,mayfindthatyou'restartingtodevelop
seriousholesinyourblueprintcoverageproblemforsampling
butyouwanttotrackstuffthisFORNEXTTIME
ifit'snegativeonadministrationafteradministration,consistently,likelynotrandom
chance,it'sscrewingupinsomeway
wanttobuildyourfuturetestsoutofthoseitemswithhighpositiveDindices
thehighertheaverageDindicesonthetest,themoreRELIABLEthetestasa
wholewillbe
reviseitemstoincreaseD
>ifgoodstudentsareselectingoneparticularwrongalternative,makeitless
attractive
>orincreaseprobabilityoftheirselectingrightanswerbymakingitmore
attractive
mayhavetoincludesomeitemswithnegativeDsifthosearetheonlyitemsyouhave
forthatspecification,andit'sanimportantspecification
whatthismeansisthattherearesomeskills/knowledgeinthisunitwhichare
unrelatedtorestoftheskills/knowledge
>butmaystillbeimportant
e.g.,statisticspartofthiscoursemaybeterribleonthosestudentswhoarethebest
itemwriters,sincewritingtendstobeassociatedwiththeoppositehemisphereinthe
brainthanmath,right...butstillimportantobjectiveinthiscourse
maylowerreliabilityoftest,butincreasescontentvalidity
8. TooEasy

UpperLowDifferenceDTotalDifficulty
8.A01
*B14131.0627.90
C01
D11
O
tooeasyortoodifficultwon'tdiscriminatewelleither
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

7/10

7/27/2015

InterpretingItemAnalysis

difficulty(p)(forproportion)variesfrom+1.0(everybodygotitright)to0(nobody)
REMEMBER:THEHIGHERTHEDIFFICULTYINDEX,THEEASIERTHEQUESTION
iftheitemisNOTmiskeyedorsomeotherglaringproblem,it'stoolatetochangeafter
administered>everybodygotitright,OK,givethemthemark
TOODIFFICULT=30to35%(usedtoberuleinBranch,nownot...)
iftheitemistoodifficult,don'tdropit,justbecauseeverybodymissedit>youmust
havethoughtitwasanimportantobjectiveoritwouldn'thavebeenonthere
andunlessliterallyEVERYONEmissedit,whatdoyoudowiththestudentswhogotit
right?
givethembonusmarks?
cheatthemofamarktheygot?
furthermore,ifyoudroptoomanyquestions,losecontentvalidity(specs)
>iftwoorthreegotitrightmayjustberandomchance,
sowhyshouldtheygetabonusmark
however,DONOTREUSEquestionswithtoohighorlowdifficulty(p)valuesinfuture
ifdifficultyisover85%,you'rewastingspaceonlimiteditemtest
askingGrade10stheCapitalofCanadaisprobablywasteoftheirtimeandyours>
unlessthisisaparticularlyvitalobjective
sameappliestoitemswhicharetoodifficult>nouseaskingGrade3stosolve
quadraticequation
butyoumaywanttorevisequestiontomakeiteasierorharderratherthanjusttossit
outcold
ORSOMEEXCEPTIONSHERE:
Youmayhaveconsciouslydecidedtodevelopa"Mastery"styletests
>willoftenhaveveryeasyquestions&expecteveryonetogeteverything
tryingtoidentifyonlythosewhoarenotreadytogoon
>inwhichcase,don'tuseanyquestionwhichDOESNOThaveadifficultylevel
below85%orwhatever
Oryoumaywantatesttoidentifythetoppeopleinclass,thereachforthetopteam,and
designawholetestofreallytoughquestions
>havelowdifficultyvalues(i.e.,veryhard)
sodependsabitonwhatyouintendtodowiththetestinquestion
thisiswhatmakesthedifficultyindex(proportion)sohandy
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

8/10

7/27/2015

InterpretingItemAnalysis

1. youcreateabankofitemsovertheyears
>usingitemanalysisyougetbetterquestionsallthetime,untilyouhavea
wholebunchthatworkgreat
>canthentailormakeatestforyourclass
youwanttocreateaneasiertestthisyear,youpickquestionswithhigher
difficulty(p)values
youwanttomakeachallengingtestforyourgiftedkids,chooseitemswithlow
difficulty(p)values
>formostapplicationswillwanttosetdifficultylevelsothatitgivesyou
averagemarks,nicebellcurve
governmentuses62.5>fouritemmultiplechoice,middleofbellcurve,
2. starttestswithaneasyquestionortwotogivestudentsarunningstart
3. makesurethatthedifficultylevelsarespreadoutoverexaminationblueprint
notallhardgeographyquestions,easyhistory
unfairtokidswhoarebetteratgeography,worseathistory
turnsclassoffgeographyiftheyequateitwithtoughquestions
>REMEMBERherethatdifficultyisdifferentthancomplexity,Bloom
socanhavedifficultrecallknowledgequestion,easysynthesis
synthesisandevaluationitemswilltendtobeharderthanrecallquestionssoif
findhigherlevelsaremoredifficult,OK,buttrytobalancecellsasmuchas
possible
certainlycontentcellsshouldbetheroughlythesame
9. OMIT

UpperLowDifferenceDTotalDifficulty
9.A21
B34
*C734.2610.33
D11
O24
Ifnearendofthetest
1. >theydidn'tfinditbecauseitwasonthenextpage
http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

9/10

7/27/2015

InterpretingItemAnalysis

formatproblem
OR
2. >yourtestistoolong,6ofthem(20%)didn'tgettoit
OR,ifmiddleofthetest:
3. >totallybaffledthembecause:
waytoodifficultfortheseguys
orbecausealso2fromhighgrouptoo:ambiguouswording
10. &
11. RELATIONSHIPBETWEENDINDEXANDDIFFICULTY(p)

UpperLowDifferenceDTotalDifficulty
10.A05
*B150151.015.50
C05
D05
O

11.A32
*B8710.615.50
C23
D23
O
10isaperfectitem>eachdistractergetsatleast5
discriminationindexis+1.0
(ACTUALLYPERFECTITEMWOULDHAVEDIFFICULTYOF65%TOALLOWFOR
GUESSING)
highdiscriminationDindicesrequireoptimallevelsofdifficulty
butoptimallevelsofdifficultydonotassurehighlevelsofD
11hassamedifficultylevel,differentD
onfouritemmultiplechoice,studentdoingtotallybychancewillget25%

http://www.uleth.ca/edu/runte/tests/iteman/interp/interp.html

10/10