Topi

c: CommonEl
ement
sinSet
ti
ngPer
for
manceSt
andar
ds&Rev
iew
ofAr
ti
cle

Cour
se: St
andar
dSet
ti
ngI
nAssessment

Di
vi
sionofEducat
ion
Uni
ver
sit
yofEducat
ionTownShi
pLHR

Tabl
eofCont
ent

CommonEl
ement
sinSet
ti
ngPer
for
manceSt
andar
ds

1)Sel
ect
ingandTr
aini
ngSt
andar
d-Set
ti
ngPar
ti
cipant
s
 2)Pr
ovi
dingFeedbackt
oPar
ti
cipant
s

a.Nor
mat
iveFeedback

b.Real
i
tyFeedback

c.I
mpactFeedback

3)Pr
ofessi
onal
Gui
del
i
nesf
orSt
andar
dSet
ti
ng
4)Ev
aluat
ingSt
andar
dSet
ti
ng

5)Ref
erence

Ar
ti
cler
evi
ew

Ti
tl
e:Compar
isonofTwoSt
andar
d-set
ti
ngMet
hodsf
orAdv
ancedCar
diacLi
fe
Suppor
tTr
aini
ng.

Sel
ect
ingandTr
aini
ngSt
andar
d-Set
ti
ngPar
ti
cipant
s
Par
ti
cipant
sint
hest
andar
d-set
ti
ngpr
ocessar
ecr
it
ical
tot
hesuccessoft
heSt
ruggl
e
andar
easour
ceofv
ari
abi
l
ityofst
andar
d-set
ti
ngr
esul
ts.TheSt
andar
dsf
orEducat
ional
andPsychol
ogi
calTest
ing(
AERA/
APA/NCME,
1999)pr
ovi
desgui
danceon
r
epr
esent
ati
on,
sel
ect
ion,
andt
rai
ningofpar
ti
cipant
s(cal
l
ed“
judges”i
ntheSt
andar
ds)
.
Regar
dingt
henumberandchar
act
eri
sti
csofpar
ti
cipant
s,heSt
t andar
dsi
ndi
cat
ethat
“
asuf
fi
cient
lyl
argeandr
epr
esent
ati
vegr
oupofj
udgesshoul
dbei
nvol
vedt
opr
ovi
de
r
easonabl
eassur
ancet
hatr
esul
tswoul
dnotv
arygr
eat
lyi
fthepr
ocesswer
erepeat
ed”
(
p.54)
.Fur
ther
,theSt
andar
dsr
ecommendst
hat“
thequal
i
ficat
ionsofanyj
udges
i
nvol
vedi
nst
andar
dset
ti
ngandt
hepr
ocessbywhi
cht
heyar
esel
ect
ed”shoul
dbef
ull
y
descr
ibedandi
ncl
udedaspar
toft
hedocument
ati
onoft
hest
andar
d-set
ti
ngpr
ocess.
Af
terpar
ti
cipant
sar
esel
ect
ed,
per
hapst
hef
ir
stexposur
etheywi
l
lhav
etot
hest
andar
d-
set
ti
ngpr
ocesswi
l
lbei
nfor
mat
ional
mat
eri
alspr
epar
edbyt
heent
it
yresponsi
blef
ort
he
t
est
ingpr
ogr
am.Thesemat
eri
alsar
eof
tendel
i
ver
edt
opar
ti
cipant
sinadv
anceoft
he
act
ual
standar
dset
ti
ng,
all
owi
ngsuf
fi
cientt
imef
orpar
ti
cipant
stor
eadt
hemat
eri
als,
f
ami
l
iar
izet
hemsel
veswi
thaschedul
eoragenda,
begi
nunder
standi
ngt
hepur
poseof
t
hest
andar
dset
ti
ng,
andcommuni
cat
ewi
tht
heappr
opr
iat
eper
sonsr
egar
dingany
quest
ionsorconcer
ns.Theadv
ancemat
eri
alsmayi
ncl
udecopi
esofanycont
ent
st
andar
ds,
jobanal
yses,
testspeci
fi
cat
ions,
sampl
etesti
temsort
asks,
hist
ori
cal
cont
extoft
hest
andar
dset
ti
ng,
stat
ementofpur
posef
orset
ti
ngst
andar
ds,
and
l
ogi
sti
cal
mat
eri
alssuchasi
nfor
mat
ionr
elat
edt
oanyhonor
ari
um,
trav
elar
rangement
s,
andsoon.
Af
terr
evi
ewi
ngadv
ancemat
eri
als,
thenextaspectoft
het
rai
ningpar
ti
cipant
sencount
er
wi
l
lli
kel
ybeatt
heact
ual
standar
d-set
ti
ngmeet
ing.
Af
ir
stkeyaspectoft
hatmeet
ing
Or
alwel
comeandi
ntr
oduct
iont
othepur
poseofst
andar
dset
ti
ng.
Whatt
het
askpar
ti
cipant
swi
l
lbeaskedt
oper
for
miscl
ear
lyci
rcumscr
ibed(
e.g.
,
t
hatpar
ti
cipant
sar
eal
soi
nfor
medaboutwhatt
hepur
poseandt enot
asksar ,if
appr
opr
iat
e).
Thenextaspectoft
rai
ningpar
ti
cipant
sist
orev
iewanddi
scusst
hem t
oensur
e
t
hatpar
ti
cipant
shav
east
ronggr
aspoft
hatessent
ial
aspectoft
hepr
ocess.
Ast
hef
inal
componentoft
rai
ning,
atl
eastonej
unct
urei
nthet
rai
ningpor
ti
onof
st
andar
dset
ti
ng,
par
ti
cipant
sar
eusual
l
yadmi
nist
eredasur
vey
,sel
f-
assessment
,
ev
aluat
ion,
ordat
acol
l
ect
ioni
nst
rument
.
Thepur
poseoft
hedat
acol
l
ect
ioni
stogat
heri
nfor
mat
iononsev
eral
aspect
sof
 t
het
rai
ningt
hatcansuppor
tthev
ali
dit
yoft
hest
andar
d-set
ti
ngpr
ocess.
Forexampl
e,ani
nst
rumentmi
ghtquest
ionpar
ti
cipant
sabout

Pr
ovi
dingFeedbackt
oPar
ti
cipant
s
Anot
hercommonal
i
tycut
ti
ngacr
ossnear
lyal
lst
andar
d-set
ti
ngpr
ocedur
esi
sthe
pr
ovi
si
onoff
eedbackt
opar
ti
cipant
sfol
l
owi
ngt
heexpr
essi
onoft
hei
rjudgment
sabout
i
tems,
tasks,
exami
nees,
andsoon,
dependi
ngont
hepar
ti
cul
arst
andar
d-set
ti
ng
met
hodused.Thef
eedbacki
sof
tenpr
ovi
dedi
ndi
vi
dual
l
yandconf
ident
ial
l
ytoeach
par
ti
cipant
;iti
sal
socommonf
orf
eedbacksummar
ies(
e.g.
,fr
equencydi
str
ibut
ions,
means,
etc.
)tobepr
ovi
dedwi
tht
hei
dent
it
iesofi
ndi
vi
dual
par
ti
cipant
sremov
edt
o
assur
ethei
ranony
mit
y.
Manyst
andar
d-set
ti
ngappr
oachescompr
iser
oundsofj
udgment
s.I
neachr
ound,
par
ti
cipant
smaybepr
ovi
dedv
ari
ouski
ndsofi
nfor
mat
ion,
incl
udi
ngasummar
yoft
hei
r
ownj
udgment
s,asummar
yoft
hei
rint
ernal
consi
stency
,ani
ndi
cat
ionofhowt
hei
r
j
udgment
scompar
etot
hej
udgment
sofot
herpar
ti
cipant
s,ani
ndi
cat
ionofv
ari
abi
l
ityi
n
par
ti
cipant
s’r
ati
ngs,
andt
hel
i
kel
yimpactoft
hei
ndi
vi
dual
and/
orgr
oupj
udgmentoft
he
exami
neepopul
ati
on.
Theki
ndsoff
eedbacki
nfor
mat
ionpr
ovi
dedt
opar
ti
cipant
scanbegr
oupedi
ntot
hree
cat
egor
iesaccor
dingt
othet
ypeofi
nfor
mat
ioncommuni
cat
ed.
Thecat
egor
iesi
ncl
ude
 Nor
mat
ivei
nfor
mat
ion
 Real
it
yinf
ormat
ion
 I
mpacti
nfor
mat
ion
Nor
mat
ivei
nfor
mat
ion,
whi
chpr
ovi
despar
ti
cipant
swi
thdat
aonhowt
hei
rrat
ings
compar
ewi
thot
herpar
ti
cipant
s’j
udgment
s;r
eal
i
tyi
nfor
mat
ion,
whi
chpr
ovi
des
par
ti
cipant
swi
thdat
aont
heact
ual
per
for
manceofexami
neesoni
tems,
tasks,
andso
andi
on; mpacti
nfor
mat
ion(
somet
imesr
efer
r oasconsequencei
edt nfor
mat
ion)
,whi
ch
pr
ovi
despar
ti
ci
pant
swi
thdat
aont
heef
fectoft
hei
rjudgment
sont
heexami
neegr
oup.
Tabl
epr
ovi
desal
i
sti
ngofexampl
esofeacht
ypeoff
eedback.
Pr
ofessi
onalGui
del
inesf
orSt
andar
dSet
ti
ng
I
naddi
ti
ont
ofundament
alcl
ari
tyaboutt
hepur
posef
orset
ti
ngst
andar
ds,
acr
it
ical
step
i
npl
anni
ngf
orast
andar
d-set
ti
ngact
ivi
tyi
stobecomef
ami
l
iarwi
thpr
ofessi
onal
l
y
r
ecommendedqual
i
tycont
rol
stepst
hatshoul
dbebui
l
tint
othepr
ocess.
AsLi
nn(
2006)hasdescr
ibed,
theSt
andar
dspr
ovi
desgui
danceonawi
der
angeoft
est
dev
elopmentandadmi
nist
rat
ionr
elat
edact
ivi
ti
es.Howev
er,
itgi
vesspeci
fi
cat
tent
iont
o
t
hepr
ocessofst
andar
dset
ti
ng.
Tabl
epr
ovi
desanabr
idgedcol
l
ect
ionoft
hesest
atement
s(number
edsequent
ial
l
y,wi
th
eachi
ndi
vi
dual
stat
emental
socal
l
eda“
Standar
d”)
.
Ev
aluat
ingSt
andar
dSet
ti
ng
I
tisani
mpor
tantaspectofv
ali
dat
iont
hatanyst
andar
d-set
ti
ngpr
ocess,
gat
her
ev
idencebear
ingont
hemanneri
nwhi
cht
hepar
ti
cul
arst
andar
dset
ti
ngmet
hodwas
desi
gnedandi
mpl
ement
ed,
andt
heext
entt
owhi
chpar
ti
cipant
sint
hepr
ocess,
we'
re
abl
etounder
stand,
appl
y,andhav
econf
idencei
ntheev
ent
ual
per
for
mancest
andar
ds
t
heyr
ecommend.Ev
aluat
ionofst
andar
dset
ti
ngi
samul
ti
facet
edendeav
orwi
thmany
pot
ent
ial
sour
cesofev
aluat
ioni
nfor
mat
ion(
seeHambl
eton&Pi
toni
ak,
2006)
.
Ev
aluat
ionofst
andar
dset
ti
ngcanbet
houghtofasbegi
nni
ngwi
thacr
it
ical
appr
aisal
of
t
hedegr
eeofal
i
gnmentbet
weent
hest
andar
d-set
ti
ngmet
hodsel
ect
edandt
hepur
pose
anddesi
gnoft
het
est
,thegoal
soft
hest
andar
d-set
ti
ngagency
,andt
hechar
act
eri
sti
cs
oft
hest
andar
dset
ter
s.
Ev
aluat
ioncont
inueswi
thacl
oseexami
nat
ionoft
heappl
i
cat
ionoft
hest
andar
d-set
ti
ng
pr
ocedur
e:Towhatext
entdi
ditadher
efai
thf
ull
ytot
hepubl
i
shedpr
inci
plesoft
he
pr
ocedur
e?Di
ditdev
iat
einunexpect
ed,
undocument
edway
s?I
fther
ear
edev
iat
ions,
ar
etheyr
easonabl
eadapt
ati
ons,
speci
fi
edandappr
ovedi
nadv
ance,
andconsi
stent
wi
tht
heov
eral
lgoal
soft
heact
ivi
ty?
.
Thet
abl
epr
ovi
desi
nst
ruct
ionaboutel
ement
sofev
aluat
ion.
Ref
erence
Gr
egor
yJ.
Cizek,
(2007)
,St
andar
dAGui
det
oEst
abl
i
shi
ngandEv
aluat
ingPer
for
mance
St
andar
dsonTest
s,SagePubl
i
cat
ions,
London.
 Ar
ti
cl
eRev
iew

I
ntr
oduct
ion
Topi
c
Compar
isonofTwoSt
andar
d-set
ti
ngMet
hodsf
orAdv
ancedCar
diacLi
feSuppor
t
Tr
aini
ng
Jour
nal&Dat
e
Academi
cMedi
ci
ne:
 
Vol
ume80-I
ssue10,
Oct
ober2005.
Aut
hor
Way
ne,
DianeB.
;Fudal
a,Moni
caJ.
;But
ter
,John;
Siddal
l
,Vi
vaJ.
;Fei
ngl
ass,
Joe;
Wade,
Leonar
dD.
;McGaghi
e,Wi
l
li
am C
Thear
ti
cleconsi
stedonmedi
cal
Educat
ionandt
wogener
almet
hodshav
ebeenused
bymedi
cal
educat
orst
oder
ivemi
nimum passi
ngscor
esont
est
sandcl
i
nical
ev
aluat
ions:
(
a)Testori
tem basedmet
hods(
e.g.
,Angof
f),
(
b)Compr
omi
seorgr
oup-
basedmet
hods(
e.g.
,
Hof
stee)
Goal
soft
hest
udy
Thi
sst
udyi
sdesi
gnedt
oaddr
esst
hreedi
ff
erentgoal
s.
Fi
rst
,toest
abl
i
shMPSsf
ort
hesi
xACLScheckl
i
stpr
otocol
susi
ngt
wodi
ff
erent
st
andar
dset
ti
ngmet
hods:
Second,
toassesst
hedegr
eeofexper
tagr
eementwhenusi
ngt
het
womet
hods
andev
aluat
ethest
abi
l
ityofexper
ts’
standar
d-set
ti
ngj
udgment
sov
ert
enweeks.
Thi
rd,
tocompar
etheabsol
utev
aluesoft
heAngof
fandHof
steemi
nimum
passi
ngACLSscor
esandt
hei
rpossi
bleconsequencesf
orexami
neecer
ti
fi
cat
ion
usi
nghi
stor
ical
dat
a.
Met
hodol
ogy
I
nthi
sst
udy
,12pr
ofessi
onal
swer
esel
ect
edasasampl
e.I
nwhi
chEl
evenexper
t
j
udgeswer
eboar
d-cer
ti
fi
edphy
sici
ansf
rom speci
alt
ies:
anest
hesi
ology(
2),
car
diol
ogy
(
1),
emer
gencymedi
ci
neandpedi
atr
ics(
1),
gener
ali
nter
nal
medi
cine(
1),
pul
monar
y
andcr
it
ical
car
e(3)
,andsur
gical
cri
ti
cal
car
e(3)and12thexper
tjudgewasan
adv
ancedt
raumanur
sewi
th20y
ear
sofI
CUexper
ience.Anexper
tpanel
prov
ided
 i
tem-
based(
Angof
f)andgr
oup-
based(
Hof
stee)j
udgment
saboutsi
xACLS
per
for
mancecheckl
i
stsont
wooccasi
onssepar
atedbyt
enweeks.AndI
nter
-r
ater
r
eli
abi
l
iti
esandt
estr
etestr
eli
abi
l
ity(
stabi
l
ity
)oft
hej
udgment
swer
ecal
cul
ated.
Passi
ngst
andar
dsar
ecompar
edt
ohi
stor
ical
ACLSper
for
mancedat
afr
om t
wo
gr
oupsofACLS-
tr
ainedi
nter
nal
medi
ciner
esi
dent
s.

ForDat
acol
l
ect
ionusi
ngt
heAngof
fmet
hodi
nvol
vedexper
tjudgedi
scussi
onand
def
ini
ti
onof“
bor
der
li
ne”r
esi
dent
s,agr
oupt
hathasa50/
50chanceofpassi
ngt
he
ski
l
lev
aluat
ions.Eachj
udgeest
imat
edt
hepr
opor
ti
onofbor
der
li
ner
esi
dent
swho
woul
drespondcor
rect
lyt
oeachcheckl
i
sti
tem f
oreachoft
hesi
xACLSski
l
ls.The
ACLSski
l
lscheckl
i
stsi
nlengt
hfr
om 16i
temst
o31i
tems(
vent
ri
cul
arf
ibr
il
lat
ion)
.

ForHof
stee,
eachpanel
i
str
ecor
dedj
udgment
sinr
esponset
ofourquest
ionsabout
eachoft
hesi
xACLSski
l
ls.Whati
sthe:

(
a)mi
nimum accept
abl
epassi
ngscor
e?
(
b)maxi
mum accept
abl
epassi
ngscor
e?
(
c)mi
nimum accept
abl
efai
l
urer
ate?
(
d)maxi
mum accept
abl
efai
l
urer
ate
Dat
aAnal
ysi
s

Dat
aanal
ysi
swasdonei
nfourst
eps.

 Fi
rst
,MPSwascal
cul
atedf
oreachoft
hesi
xACLSski
l
lsf
orbot
htheAngof
fand
Hof
steemet
hods.

 Second,
thei
nter
-r
aterr
eli
abi
l
ityoft
heAngof
fcheckl
i
sti
tem j
udgment
sandt
he
Hof
steej
udgment
swer
eassessedusi
ngt
hei
ntr
a-cl
asscor
rel
ati
oncoef
fi
cient
.

 Thi
rd,
thest
abi
l
ityoft
heMPSsov
ert
imewasassessedusi
ngt
est
-r
etest
cor
rel
ati
ons.

 Four
th,
theAngof
fandHof
steeMPSsf
oreachACLSski
l
lwer
ecompar
edusi
ng
pr
etestper
for
mance.

Concl
usi
on

I
nthi
sresear
ch,
rigor
ousandsy
stemat
icst
andar
d-set
ti
ngmet
hodscanbeusedt
o
der
ivemi
nimum passi
ngscor
esf
ort
hecl
i
nical
ski
l
lsr
epr
esent
edi
nACLSpr
ocedur
es.
Sy
stemat
icst
andar
dset
ti
ngf
orACLSpr
ocedur
esi
sanecessar
yst
ept
owar
dthe
cr
eat
ionofmast
eryl
ear
ningeducat
ional
progr
ams.TheHof
steemet
hodconsi
stent
ly
pr
oducedmor
est
ri
ngentpassi
ngscor
est
hant
heAngof
fmet
hod.Pr
esetpassi
ng
scor
escanbeusedi
nmast
eryl
ear
ningpr
ogr
amswher
elear
nereducat
ional
out
comesar
eequi
val
entwhi
l
ethet
imeneededt
oreacht
heout
comesv
ari
esamong
i
ndi
vi
dual
s