Beruflich Dokumente
Kultur Dokumente
MODELS
VIEWPOINTS ON
EDUCATIONAL
AND HUMAN
SERVICES
EVALUATION
SECOND EDITION
edited by:
Daniel L. Stufflebeam
George F. Madaus
Thomas Kellaghan
Kluwer Academic Publishers
\a
Et"*-"i. services <http://wwlwkap nl>
f\
T'tt llehlkhet one$ seouflB on thi' book Jot roune Be dnd bttlk pnfthases'
Fot Iurthet infonation, eflait to <'flichael.witians@nhap'con>'
'end
::-: -1 :: .._ r.\acnlivcly in the
: :: i..t :.!. i.,r crantplc, ttred
.. ' - j\r,,\\ lro\\'extcrr_
::. 5. OUTCOME EVALUATION
sllrll outlir)c i1 ltulrlber oflssues r'.risccL bv ourco tc cr':tlullion. ,rrd corrsrrlcr hoN it
lrrs euong rrldition:rl lpploachcs. l\,lost oi our
jllusrrativc nuterilLl Nill ,,rrrre liorrr
rlic flelcl of cchtc.ttiotr. nllerc otttcotlle cvalrLltillrl pnrtr:lblv has h]d lr\ grtiltcil
inrprct. Llut such er.llu:rtioll is b! n" "ttt"t '
orrtrrrerl to educltlon
progrrnr dulirrg its liti'. et it conlplclion. ol lxtel itl lirlrc to rsscss loll1l tu rrl1 eft((t\'
N'lost iicqucrrrlr,. thc tircus is oll olltcolllcs ll rhc aonlplclloll of I progr':tlll'
Fiith. lt is r)or usuJi itt oLltcolrlc c\'llLriliclrl to \cck t() dcscribc or spccily rvhet
is:rcrtL:rll,v hlppenitr* |t e Pro{rrrrt. though rhc kinc] of infol rnetilrn obt:rinerl rvill
obviolrsl-r'. irr gcuer'.rl telrlls ra lclsl. hc cholcll lo l-cllect progr:lrtl ;lctivilics- ln ll:irl-v
crrcumsluces ln wlrich outconlc c\,ilLltrioD ts uscd- I descriptioll of PloLr.lrll Jrtiv
itrcs {,ould bc verv tliflicult, if rt 111 possibk'. Thir is becluse nlalv proqr'IllN arc
extrcmell conrplcr rncl crll onlv bc corlsiderccl froqranls ir) the bro:rdest sr:rlse of
rhe rvor cl (:.g.. elcnrelrt.llv ccltlcation) Sttch proelenrs lle pe raps lllolc ic'ur!lclv
dcscribctl es complc\es of progrlrrlls, uhich lrc irllplcnrcrttcd in a vlrictv oi rvr,vs'
nrd hrl rvhich lhL' lcrlll \vslr'lll lniqht bc nlore afpropli:lte
Sirth. $'hile ()Lrtc()rrlc cvlluiltiolls lllirv e\chc$' dcscriptiolls of ptogr'rnl 'r'rr'ltitt,
--
:. OL,r.orr. t:lilLritLoi 99
Fililrl-ri gr-o$th i' ourco'rc' c'elrL.rtio' rcflects iucrcrsi'g Lr\c or':rsscsl'e'r irs a
policv rool. Irr rhe fic1cl of crlrrcrrion. rhis rnvolrcs a shiti liorn ahL- usc ol- lsles\
rrenr l^ti)r':rtron lir' locllizeri rrst.nctior:rl clccisio' .raki'g to cenrrllizcd high
strkcs poliq, niatins rrd lccorLnrrbility nroli.orirls (Mldaus & Raczt,k, 1996).
Sirth. rhL: enrn'th ot oLrtt.oure cvelulriou orrcs ntLrch to it r-eorglrrization of the
fribiia \ervi.c irr severll cou'ores. tcs'lring lrr the usc of relrtl'elv :tLltonoltloL^
service ptovitlers (e.g., Nitrio itl Hc:rlth iicrr,icc fiusts l rl qt.;tnt-ntrirtrel cd schools
ir llritlrrr). With rlccentr:Lliz.ttion of pr.og.rr' :lLlthorit\: .1ld the conscquenr loss .f
dilcct contlol ovcr thc irDplcntcltrrjorr of progrtits. the ncc(l llosc fbr ncu. coD
tractull alranqenrenl\ \\'ith rclvicc prolidcrs :rnd fbr. regulidon :rnd cornpliencc
nrolriror-ing. "Qrrilitr"' and "st:rndtrcls" itre thc' thcilre rcrnts. .lrrci c|illuittroD .rrrrltLc
rttcnts erc dcsiqned to chcck tltilt organizetiols :rre dcllvcring flc\iLrle. cost cflc.trve
ser|iccs to cirizcrr users (l)o1li.t. l9cl3).
Frnrllr: I sit.ation i. *fiich ero*th i. ilen*n,.l ibr p'blic ser'iccs rrcl socirl
progrrrr lirnding (c.g., eclLrcltion, he:rlrh c:r1e. s6r:ill sc,cprirr,) rs qrog,irrq nrorc rapid1.,
thi'rr-sou'ces a;lr) be fbu.d lirr crplnsior L'.rr1s to rhe |ced lbr grc:Ltcr.etiicicrc,,,,
rvhich in tlnli c:rlls fbr sclecttriry l'tlecidirre \ ttt pt.ogr:ltns:tre to be co.till.ecl
iud rrh;r! lc\\' acrivities arc- to bc hrrncheci (l3ll1ock. 1999; l)ur.atr, Monrier. &
Snrith, 1995r lrollin. 1993).
a::i:::_ -:i!. ot (Madaus & I{ellaghan, 1992). In 1999. 33 of the Unitcd Stares had or shorrll' rvill
)ssessnrcnt :ls a
! :a: - ::.r:l the LtsL- of assess h:rve high sukes (c.g., high school graduarion. encling social prornotion) irltached to
:. -::t:: :o .enfalized high
0-
thcir tests. uhile 1.1 stlrcs Iirk rnoder':rtc'srakcs (e.g., a special diploma) to thcir
: l::r:,:i -\ R.iczek. 1996). assessnlerrt syslelns. Sanclions nl:ly iln'olr'e linancial considerations tbr districts.
i:: : .:'orgtniz;ttion of thc schools. or te:rchers. Sornctimes, howcver, the mere publicatioll of ourconrc inlor
: -,: : ::irii\-L.lv tltto orllous rutioli is considcred a sl)ction. There rvould appear lo be nvo principles uriclcrly-
:, ::i i::::a-nlJiItained schools ing the rrse ofsanctions- First. indivicluals and institlrtiorrs that are subject lo \]neliurr!
:::. : : ::ti aonsequcnt loss of \ril1 trke ilctiorr to obuirl rc$'irrcls:rnd:rvoicl purlishnlent. Sccond. lf informlrron on
r ::: ::-.-i .trosr' for netv con- or.rtcomes is trtought into the public clomain, principles of conpetitior. .vjll conre
::: .:--: r.::on .tnd contplilnce ]1rto oper:rtion. arrd. xs in lhe commercial rvor1c1. thosc that do rvell r,vill thrivc, thosc
: :::r , ::r.i .-r.rlurtion arranqe th:rt do poorly rvill rvith c-r' arval'.
::: . :::: :::sible. cost ellecrivc ORIGINS OF OUTCOME EVAIUATION
:::'::I: \er\'iccs itnd sociel The r.rtioralc firr, and pr-actice of. outcome evaluation owe x debt to el least sr{
: -::: : :roq_ing tlore rapiclly
of assessnlent in educrtior-t, school effec
soLlrccs: traditional cvalu;ltion. rratlitions
.: :.: Ljr gr!.Ltcr ctttaterLL\'. tivencss lnd cducarion productron function reselrch, dre perfornrance manageDlent
:r. i:r::: io bc continued
.tac'
nlovenlent, :rccounrrbiliry concerns. and technical devcloprrlcnts.
i:. -.,: l)urtn. Monnict., & Traditional Evaluation
A consider:rtion of the outconles of progrrnls is an intcglal fiertLre of many tradi
tioial approaches to evdullion, .rr1d, Lrp !o lhe 1970s. educational cvelurtiolls focused
: : :-i rre elaltution. C)nc is prim:rrilv on rssc'ssinq frogr:lnl outconles- The crtrphasis on outconles i5 1tro5t
: -. ::: rrociltecl u'ith org:r obliotlr in objectives-oricrltcd cvalu:rtion approaches. T,vlcr (1949). lor exarlple,
: ::- ::':-.:tous in lacking such focusecl or cducation:r1 objcctivcs ancl thcit rnclsurement ir the contcxr of cur
_---: t:.-:t,.ol. \\'L-rc
to sPeciii ricuiun evalu]tion. Orher approaches in thc !leriaD nrdition also accorded prorrri-
.- ' a,rrtltl. :lnL1 \\oul(1 ncocc' to rhe specjtcatlon of oblccti,.'cs and.Judgmeits of lhc c'xlcnl ro rvhich the,v
:-- :: - -.:t:L.ut (scc Schntidr. coulcl be saicl to ha\-e been achieved on the basis of progranl oLllcolne data (c.g.,
: --_ ::::r i,i orLlcorre cvaht, ProvLrs, 1971). Horvcvcr, these approaches diflerecl frorn rnarrv current outcome evrl-
-- , : :r:rrCr itrtpaCt tvhen uation ctlolts in linking prograln objectives lo lhe goals or objectivcs of individu:r1
: .. : ,::ithitrg otrtcomcs of sahooh or teachers rerhcr than to starcrvide curricululrr llrnrervorks, rvhile outconres
:: :-- -:: ::t:.\.trious \Lalcs hlve \\'erc not uscd for l-righ stakcs dccisions or t'br accountability purposes.
':. :: r: j:-_tt-_ -e.rrni g st:rnderds
Traditions of Assessment in Education
:- :...::: .,. hielcrnent related
:-: .:::i..lsileAssessrlent Feu' pcople would clisalirec u'irh thc vielr'' thrt lhc oLltconles of edtrc;ition ale
':: I i. ntc'iisures the pcr important. Hoq'cver, eqreelrtettt u'ould not bt' :rs rvidesprcacl on the rel-rlive irrrp,rr-
:-i: ::::,lltg srandlrds con taricc of outcorllcs, since inciividuals ciiffer irr therr Perceptiolls of the froniil(llar
:- _: ::: ::quiertrenh of the thar should bc giren to lhc varief\' of goals or oLrjeclives thal h:rve bccn posirecl for
::.:-.:: -_. ..: Eilucation. schoolnlg. Literacy anrl rturl)eracv skills lrc usuellv :lccorded pxrriculir lltti'orrancr.
1998.
artcl thc'use of irriornration on outconles to nrake decisions abour the eflictir'clless
of schools rnd teachcrs, besed on stuclents lcqr-lisilions of thcsc skills, rexches b:lck
:::- L. tu Lru fltc Lou-
.a-::: -: :iii. Policvnrakers arc: into the llsr centurl Pcrhaps thc best-kno$'n cxanrple\ of this lpproach ere
': - :: :'.r -_,,rrriculult, instruc pavnrent by resulls schernes t'hich were irttroclucecl irlto Br'ltish schools in 1862 to
l:a :::r::::!rIi. pilrcDts, or lhC help inrprove studcnts'llterxcy :rnd nlxthen tical skills ancl teachcr clltcierrcl', rvlile
:: -:::r.:: :.::oat.1!CCl rvirh thr:nt
:1t the sr rc tinre \aving nlouc\'. I1l thcsc schenres, thc allocillion of iirnds to schools
102 ll Qne\tjoir\./\,1ethor1s Ori.nr.d Errlurtior MoJelr
::. -i ..rr:rl e\atliDations of schoolirrg. Rellecting lhe inpur-oulptlt rrotlcl, indiclrors ttscd b,v lhe Natiorial
:.: -::: iti students $'as Ccnter tor iclucation Sl:rtlsti:s of thc U.S. l)epartmellt of Education trorv iliclude
conrcxt and outcotlc cht:t (Steln, 1986) At rhe international levcl' OECI)
(1997)
in desclibirrg lhc ednciltiorr systetns of menlber colrntries hls' duling lhc 1990s'
.:-: .:::lill\ 1n the Unitcd alld econornic contcxt of cdu-
-,,:,- : il-Lies (1897) r'.'ork used inclicetJrs to clescrjbe tllc dernogt:rpbic. soci;rl.
in edtrcation, the learning enrirorl
: - :: :::::::.tlt on outcollcs, .ttior). dDancisl and huntln resources investcd
:, . :: : .-r: $hich related rcr Il1ent eild lhc organiz:ltion of schools, and stuclcnt :rchievcnlenl
::r: : : :: :icquerltly used to The Perforrnance Managernent Movement
.: a:.:. : :hcir- l'eluc in this ttlakers is rlol nc\\' in
r::-.rr:...ri thcv coulLl also Sensitivit,v to the nceds of plogr:rItr nlanagers and decision
cu"l,r,rti,rr,. Stulilebeam (1983), fol ex:rnrP1e. corlsidercd tliat the decision that h:rd
:: :: r'. :r'.rrtirlq'etlucatiotlel the key conccln ofthe cval
to be llrde. rathcr thlll progrlnl otrjectivcs, should be
uator. Clrrrent intcrest il thc use of cv:rluation frndirlgs for nl:rnagenient decisions
; : :_:-: -r\ad iltcreesinglY as rvhich hes its roots
a ::::.'ir:rg nh.rt is rvrorlg in has a rather difl:rent origitl. hou'evcl: perfornullce lnanagemetrl'
in *re 1930s but sre*' in populariq in thc late 198ils and in d.rc 1990s alongside
r . ::i.r:.: h.rr rccc'ivecl. rcflect rl:ul
--': : . irr rhe purPose for rnore cstablished evalr'latiorl :rppro:rches Whilc rhe gcner-al aims of performance
agcmeit "lo base;uclenrenrs of lhe eflectiveness of progrlttl eflblts on nlore appro-
::: ::. ,-: .1,.r:lin u'hich have
: :, :::-l ._!.n.liaiolts lo the out- p'ri"t. trust!!'orthY inlorllation, alld to jlllPrcve thcse etTorts" (Rlalock' 1999' p
",,d
:::: : : :ir:r.rooll of ihe e\tclll itS; d,, no, cliller fiom the ritrts of ntanv r-nole lraditional evaluation approachcs'
...u..prs,.,r,d"rlvi,te pcrfurnlance rlrenagclllen! difli'r frorrl such epprolchcs
in a
-.: :: :olrir tlr.ikers is to be
nulnber of lvlvs.
-: : i: r.:rl1fa!\ of Educetion Whi]c trlclitiollal evairrltion qle\\' out ofsocial sciencc resealch, adoPdng
ils L]asic
:: ::: :::..r \ct oi 199'1, Thrs
conccpts and techniques, perfornrancc Irlanagenlent has ils roots in :i-
bureaucrltic
..1 ;. 1, ,L.r111i ibrrn part of I ones relat-
cnvir;1n1enl. It is trased orl plalloirlg rnd rrrlln:lgernent ideas' particularly
_,:: : -i:_:i. iahools, and corn
,:-: ::: :l:.::rge lhen lhe rcsul$ ing to tlulliry lssLlL-Allce' cusl()lllcr satisf:ction. and conlinnous illrpr-ovcnlent ll
deter-
:-: ::: . r'.)+1. Ihls leeisllti(t) iniolves defirting perfotnlancc in ter s oftesults, setting pcrfornance tlrgets'
rnining thc.*t"rti to tnhi,,lt resr.ll$ lre acirievecl using perfotllrancc indic;rtors' and
_ :r:.:::: rrst is still orl hold
basir'rgl ..r.r,.,rc. alkrcalion decisions on perfbrmatlce inlorntation ltr 'Lrnl is to
::-: : :.: ..-siiiation in desigtt- limited nurnbcr of oulcorlle nlcasures
pror,ide r'.ipicl ancl continuous fccdback on a
th"t aa" p".a.iu"d to be of interest to policl'nrakcrs achninistretors stakcholdcrs'
polirici.rns, arld cuslolllcr\. arttl ro be ofvalue itr n king decisiolrs
::.::..: Research (Blalock' 19t)9;
I)avics, 1999). The tnertlger, not the "scierttlfic" polic-v analyst' is the charisrnatic:
: :: : -:- ::rrrn prorluctiorl func and thc achicr-enlcrlr of per-
figure; eiicienc,v lncl ccononrY are lhe ltrain conccrnsl
--:.. :.-:-:i i itrndardized lests, fuirur"rt." t..g.t, is thc sign of "rdminisuarive heafth' (l'}ollitt' 1993)'
: ,: i \n i.pul-output reP-
.---. :__i .i'. !'d: studenl acllievc lt rves ll rlis col)!e\l lhet rllaniger]lent illfor-lrration svsterrrs (M[S) grerv irr the
1980s, desigrrecl to spccifv llre slructtucs ancl procedutcs governing
the collection'
:::. :,-:r,-. rilcnti6cd ls fallilY The dcveloprrlert was'
analysis, pre-se ritarioll , errd use ofinft)rllrldon in organizatjons-
: :.r. -::::;a:L'ri\dcs (e.g.. cltrrent thc- growrh atrd irlcrcrsirlg tonr
:rr leasr in part. I responsc to lhc ,leed lo ilonilor
-. ::::: rr.raher mtlo) (sec decisions eborrt rcsoulce :rllocation ()utcome
arld to justi$
plexrr-r, of systenls
infbr-
r-:i : : r,. rlLr.lttotl collcct data
.""1,r.tio,t fits reldily into thls picture b,v providing relarivelv sirnplc statlstical
lration rLroul e or aclivir,v on :r tirllely basis' Whilc more or less
s,vslenl, proqrarll'
-::- ,:-:.i:trt Jahievencnt The it is llot the
ra:r: :::l l lluativc' relcvnrce) conrplc\ ellalYscs lll:ty lccoDlP:rnv this lllortnation in sorne cvaluadons'
prittrar-v purpose of oulconte evaluation to provide thelll'
: : :-:. r --:lLrt concePtualization
l0l JI ' 1'r ,,. \l 'l'. I I | . \,1 .
Acountability
In reccnr veers. ;rccotLntabilitv has acluevecl incr-eesing pronrinencc ir qovernment
icLnil)islratiols in nuny countrics. Melsures ro cortrol ho\r stikeholders djschargc
their obligations her-e bccn clcvised ls I rrrecLrnisrrr for cle:rling u,ith jssuc.s u,hich
rrisc lron I nrurbcr of phenorrcrr:t: incrcrsing dcnrand lor scr.vtccs couplecl rvirh
clinrinishing lesources; :r nruhiplic:rriol of rclbllr str:ltcgic\; \\'eak :ldtnlrtr,!r.ttl\c
inslrurroitsi lrrrl corrrpctinq velucs tncl clemands in plur-alist cukurcs. Thcsc rrrcl-
Jrrr-es, \hich hlve Lreen :rppliecl to n renqe ofpublic scrvices. nright sccltl r re:lson
:rblc \\'lt to bring order to corttplc\ irnd poorly ulclcrstoorl cnvironnlents. It i\
cnvislgctl rirrt intirrnr:rtion brsed on dre mc:rlures \vould leacl to rhe urc of eclnin-
istretive controls ovcr the Lrsc of inpurs to cr)slLrc lhat spcaificcl procedures :Lre conr
plicti rvith. it rright rlso sinrplv irtvoh.e the iclenrification ofproducrs rhlt rnect
13ut
.r speciliecl \trnderd rnLl prodrrcts thrr clo not. It ls rc-glrdc,:l:ls a relarivelv sinrplc
lrcl straightloru..trcl t.tsk to use detr fi-om en olltconle ev:rllr:ltior lo phac drc onus
for chrnqe rnd rdjustnrent or) thc person or jrstilrltiol) itlcltiiied as being lccourt-
rble, rnd to ph.L- orrL'ls rrll\t in the opcrition ot a competiti\.e nr:rrkct and the thre:rr
or pro]]risc ofsarrctions ro briDg abour rhe desired efli'cr. In thls siru:rtion, the otrus
is r)ot on:r nlrn:lqer to idertlfv riesir:rblc aspccts of irlrflcrr)entution ol conclltjolls
thet ncL'.1 to lrc chlrgcrl. He or shc doc! not have to trv ro undclsrirnd or t'xphrn
nh_v sourc inclividu:rls or instirrrtlons xle "cfli'crivc" lnd sonc irrc not. A1l that is
rleccs\i1rv is ro iclcntili thc cllcctivc ercl tlrc rrolcflectivt. arrcl to heve st:rrisrical dlte
to sllpport the judqnlent.
Despite problcnrs rssoci.rtccl \\'ith oLlt(:o1llc c\'.thriltjor considcrcd be1ou,', eccount
:lbilin i\srlcs loorn llrgc in considcrations ofschool refolm toda\i For L'xiljlplc. llrc
Erirrcltlorlrl Improvemcnt Acr edopted ill Tenncssce in 1991 ctcatcd the need to
specili rhe rncxns b,l' whiclr te:lchcrs, schools. arrcl scllool s,vstcms could bc hcld
,rcaourrt.rLrlc lbr rttccting oLrjcctilcs ser firr'leDtressee's educrtion st.stcrlts. Siticc tLrc
tircrrs r':rr on ploduct mthcr thlrr on proc_css. ill orltaonrcs-bascd ilssessrnent s,vstenl
r.,'ls cstabllshccl llld hls been enrbccltlecl in the Tennessee Velue Addcd Asscss lcrit
S,vstenr (lVAAS) rvhich lornrs :rn intesrrl p;lt of lcgisl;rtion (Santlers & Horn,
l9e1).
Technical Developments
Thc arrjl;rbllltr. of rchrirclv kxr, cost tccIroloqlcs uidr nrassivc col)puhrLli !.Lp.L
bilities Lus gleirtlv xirlcd the devcloprrrcnt. not onl) of lar'{lc sc:lle tcslinq progr:llls
lo obtijrr olrtaornc dilti, bLrt itlso of nr,1 rlcfrrft rll,)rnr.Lt]L'n \\'\tcfr\ rn gcncril end
logisticrl pl:rrrning. Outcorrrc evelurdor rs grcarlr'liciliratcd by thc- caprcrr t', rr,rrc
vust :uu()Llnts of d:Lta, () link clrre collccrccl :ll diflctcrt porrts ir) tiulc. .rrr,:1 to carrv
oLrt \{)phi\lic:1lcLl sr:ltisticrl :trl.rlvrc\.
for thc usc of lcsoulces/iJlputs ro proviclers, c:ln tre found in:r nide r:rnqe ofcoun
:: :: : : -_a lll qovernnleDt tries. This is a charrgc liorrr a situation rn which, Lrp to recenth', nroniloling arcl
.: :,::ir,rlJcrs discharge evarlu:lrion svstenls \l'erc nlore c()ncelned \\'ith re\oLrlces end inrplcncrit:ttiou tl-un
: ::-::: ...:rh issues rvhich $ith assc-ssir)SI rc'sults. Lr nrrn\' aountlies, aspacts of pcrforrrrarrce nersures rre nolv
: : : .::.ti:i corLplctl r,r'irh undeN'ritten b,r' leglshtior.
.:::r:,'atk .rdntinistrative Li lhc Unitccl Stetes. rhe (iovcrnurenr Pcrlirrrrarcc and l{csulrs Act (GI'RA) of
---.:.. -:::.irc'r. Thcsc tnca 1993 lrs implemcritcal ir) C)ctober 1t)97 ls a responsc to lcports oi *,astc end inef
:.-i:. :: :::tt scenl a teason frclcrcv in governrnent spenrline- Tb 1cslole public confidcnce in governnrerrr, all
r::.: . .:r.. irornlcllts. Il is fidelrl lgcrrcic-s rr'ould be heltl rccorlrrtilblc for rchit'ving progrlnr rcsults. scrvicc
:.::-r:.':lie Lrse oi adIrtill- queliry cus.onlel s:rdsfirctlorr, and ibr pr<)ricliDg Congress uith srrlllcient infbrrl:r
::--: r -- :rlracdules tre conl- llorr to inrprovc decision makinq. l)olbrnr.incc'rrc',rsulenrent rvorLlcl bc rctltLircd ard
,::: :: - -.rodLlcts rh:lt nreet the lesrrlting clltl rvorrlrl bc rladc'public.A renqc ,rf public:rtiorls proviclrng.r r:rr,,
::::: -i .r rcl.rtivclv siniplc nllc fbr. end descIiption ol pcrlirlnrrn.r ]lrcr\LncIrL-ft ( rrrf.urD:j l,.r r'csulrs"). as
:,:-::r :_. :o Pl:lce lhc onus rvell as expcricncc ir lts lnc- iras been preparecl b1' the U.S. (lenerel Accounring
:j:::.::..-i r< bcirrg ac,::orLnt ()ilice and orher:rqencies (http:/ /."vu ,,rr cc-r.rst]lr.gqytfatll!+talgprdi$.blut).
.:: . :: ::Let irnd the thrcitt Major ch;rrgr's heve occurred ir government rqer)cies fb1lo$'xrq the legislation.
.: , :::. .irulrion. the onus Fol eramplc. thc Urrircd Srrtcs Agencv firr lnternationel Developnrenr (USAII)) has
: : :::.::::.:tii{rt) ol co ditions developeci fil its F.rncieci plojects e 'rcsults fi.inrcrvort" rlhlch irrvolvcs spccihc:rtion
=. : -:rJ:rrrlnd or expiain
-i
of golls, objcctir'cs, irdicarors r'itlr periodic tarqet!, intelrrrediate results, :lnd long
, :: . :r.- r)or. AII that is ternl nct r-e,iu1$ (r'eprc'scrtting the effect of rhe interventioD) (Tbftb1oD Wciss,
: :::r:r-1\'c ir.ldstical data Bcrtrand. & Tc-rrcll. 199t)).
Ev:rluetion rcrivjtv outside tlie Unitcd Stltes is not well doclunented. However,
. : .::::r.1 L-elot. eccount- lL sccnrs relson:rble to sat' that the c\tL-nt or rengc of cv:rlr.rrtion rctivirics lbund in
: : ::: -:-:.. For elentplc', rhe thc Urrirctl Statcs is nor fbuntl clscrvherc, despitc r rccclt srugc oicvrhrrtron.rttiv
. -::.:rL.d rhc recd to it\", or nt lelst r recoqrrltiol) of its need. in menv countries. ln Sp:rin. lirr er:rnrple.
. ,::r:rs cou1c1 bc held govcrnnrenr hes rcsponcled to leeisliuiorr rccllliring r'vxlurdon lollo\\'in!a govcr rrl]rcnt
:: :. .. itl'nr\. Sincc the xction ir1 contraatirq sclvlccs. creatinq condition\ for conlpctltion, rnd rrisinlj the
_i :. :.:-:
.liic\srtreDl svste]]r issue of accountebilin- lhe responsc rcllccls l plefi'rcnce lirr eveluetion lpprolcht's
,,-: ::. l.lJcd Asscssnrent thrt lre cornpatiblc lith rhc production of nlan:lgenrelrt conlol indic:ltors and irre
:,. .: :: \.rtdels & I{oln, useflrl in inforninq decisrol nrking in rhc policv process. For e\rn1ple, the (l:lrihr
Hcalth Services Admini\trirtive ()t'Icc rrrorritors poprihdons scrvccl. c_ost, lnd oLrtputs
(e.g., nrrrrbcr of visirs pL'r' inhabitrnt per dr\'. lrLrrbcr' .lrt1 cost of prc'scriptions)
(J3all:rrt. 19t)8).
:a :: .. .:' aolllpuh[g Cl1pi1 LIsc of cvnlultiolr (throueh usnallv of .r rathcr olcl ftshionecl v:rrierv*) hxs rlso
=:-.-.-.-: tciting Plograllls
grorvn ir othcr courtrics clurinr: the 1t)8{)s lnd 1991)s. Lr I)cnnr;rrk, rredi-
r:rpi,:11,v
. ::i':t)\ irt {eticr]l and tionll cnrpiricrl nlerhodobgies (usuelll, survevs) to pr-ovide Lhta lbr politicll and
:r: i:rp:lciw o storc ori1anizilliorrl (lcvclofnrclt. .ontrol. lroritoring, rnd moLler niz:rti()n xrc lavored
: :irc'. .1nd lo clrrv (Hrnsson. lc)c)7). Lr Frlncc, -'rvitlespr-crd inirtLr:ltron with prrblic polic,v evrlu:rlio!r"
rs a nlc:lns of nrorielnizinq public service hrs bcen rcportcd (l)urarr. Morrrricr. &
Srrrith. 1995. p. .15). ln ltell: dem;urds to ploduce :rn evlrlulrtion firrnelork for- r'eceut
refi)rnrs iri lrerlth scrviccs (.d::iatLla li:ztt:iont' dtlla sa rd puhli,:a) havc resulted in ren
slons belrl'eell :rn rpprorch tocused orr nlilnaqL'rnerlt rDd one mor-e olienlecl to ctlcc-
ir :.'trirrq shnderds dveness ard quality rs\es\n1ent. Norrvll. also sccllrs to be showing signs of increlsine
:.::..:r oi responsibilitv enthusix\m ftrr evllultion. thouqh i\sues have nor vet cievelopecl with the shiirp ess
-
106 Il. Q!c5tioni/N1cth.d\ ().i.rrcd Frrlurnon \1o.1c1.
of fbcus observable in Anglo Saxor cor-rntries (see Nervs frorn the (lommuniry -Lrrrr/
rdliot, 19()8, 1,373 379).ln drc Russien Federation, thc requirement of e unilbrnr
curriculurn in schools is being rcplaccd by grcltcr ilutononrv fbr rcgional :ruthor'i-
tics and schools in conjunction rvlth ourcome based curriorh (Bakker, 1t)99).!(hile
the evaluation rmbitions of mrnv counblcs sL'crrr less rh:rn nlodest, rexlizetiolr is
being hanrpereci b_v lack oi data, expertise, instrurrrcnts. .iud drc iuG:lstrr.rclLuc
required f-or large-scalc data collection :rnd :rnelysis.This point has been nlrde regard
ine rhe developnrerlt of ev:rlu:rtion ir thc l)coplc'.s llepublic ofChina, where eval
urtion \\'as u kno\\1r Lrp to rht'carLv 1980s, but is ro\\'scc| to bc inrportant in the
conte\t of n:ltion:11 developrnelt and economic qro*'th. Mariy steps :rre beirrg Lrkerl
ro inrprovL'rhe colultrys evalu:rtion capacitv (Hong & Risr, 1997).
'We turn riorv to dcscr iprions of spccific olltconle evalu:ltion eflbrts in education
at srrte level (U.S.), n:rtional 1evel. ancl iutcrnltiolJ lcvcl.
ol district
:i-.-i - _:::1r: (-omnuniry, -tlal- thc U.S. Congress lnd irnplelnented by trained 6eld staff' usually 'school
educarional achlevelnents at
_j.-. ::.:-:::_:lcnr oi a urrifornt personnel. Thi survcy is designed to nrelsure students
g."d., and reports the percentage ofstudenls scorinll in the thrce
.i::: :: . :,\! regional authori_ lp.cifi.d
"ges "rr.l
a'or*ao""..i"t p.r{o_.rt ."tetoties: "bmic"' "prohcrcnr"' and "advanced" lt also
,--:--:,-: r Brkkcr. 1999).While "r'ta"
of subpopulations defned by demographic cb'rr't'leristics
--: :: ,-_. :tr.rJc.t. realiz:ltion is "r.rrriu.r ".hi""'.ntents C)\'er the Ye'ars' cletails of thc adrnrnistration
:::-:..::. :::i thc iDfrastructure ty ,p..tft. background
"rJNaEi' It".,c ch"rtgcd; for"*ptti"ttt
era''ple, in the tiequency of assessmenl :rnd itr t6c
l-:. : . ::_: :r:. l.een nrade regarcl- "f everv second yeat on
i-::-:-:-_ .r: Clhina, rvhcre eval grade lcvel targelerl. At present, assessnlents arc conducted
_.. :::: :!r 'oe rtrportant in the s'ar,rpl., of rt.til"rtt, irt glacles 4, 8' and 12 Eleven
instructional areas have been
and rvriting' math
..--- 1.1-:-. i.rpi:lle being tirken assessed periodically. Most recent repol$ have focused on rcaditig
have beetr teported by state'
,f -\ :- -r9:1. ernatics and science, histor.v, geography, and civics Data
':.!-::::.'r. edbrts in education gendcr, ethniciq'. type of comnrunitl" and region'
' Nr,iort"l are now fcature of other educaliolr systcms lhrough-
.... '
"rr"rrrtt"rt* in industridized "tln-v (e g Australia' Canada' Finland'
out lhe \&'orld, nor only countrics '
irr..", t."t"nA, thc Nerherlancls, Norn'a-v, Nerv Zelland' Unitcd Kingdom)
Sr'veclcn'
A vlrien of rpprorchcs. dcp.-ttdittg on lhc oLllLolllc to Lrc essc'ssccl hrr bccn rrscrl
it or-rtconrc e\lxllt:ttion. 1n erllulrions lD thc. ficlcl of cducillloll. .tssrr.trleut. ,rf
\tLlLlcnt ilchiL,\,c tent lls!rlll! iltvol\c rhe rd tiDish.rrioD oicesls,jl c:t:rnrirrrtlons. fhc
pcrlbrm:rrrccs of in.livrdLtll rrudcnll rllrv tllr'n bc' .rggregrt.'d to thc leu:l of thc
tcrclrcr. school. clist lct. strte, or c\'en r).itioll to ello\'-jud*nlcrlts to bc Itr:rrlc lltout
irahicrcrnclrt rt thc LlcsicLl lcvcl.
-hrclgnrertrs rtnv bc
ntrdr orl thc Lrrsrr oirr .rdjnsteLl r.'strlri. Irl flrltish lelque r:rblcs.
r1,.. perc"ur"gcs of strr,lcnrs iD schoolr l$,er.lcd \'ert tg gradcs on p11l)lr,
( \.rLllil1.L-
riors ("pcribrnr:urcc trblcs') hrve bccn publishe':l since 1992- l;r the Urrirccl Stltcs'
nlost stlle .lacottnlirbilin \\\lcrlls lll tlrc plrst corllparcti lcllools:rllrl:cbool tLstlicts
on the br\is oi unrrd-juste,-l oLrtconrc nrr.lsLrrcs ((iuskcv & Ki1'cr, I99ll) Sintillrl,r" irr
irrtcrnrtiorrei corrll.rl:ltl\'c strtclies. coutltlics ele rrllkccl oll lh' blsis of nnerljlLsted
InBucnccs other llt.lrl litc Progr,llll Ltcirlq rllonitoled llic distirlcrion is rcrclill illtrs
trltccl jrt rhe clse oi stuclcrrt tcllicveltlcr)r\. \\ltlch rre gencrrllr'rccogtlizcr:l rs l-cllect
rn!l .l vrlicn oirrtllucrlccs. irlclu.lillL gerlctic r'tldonrlrcnt. llchielcrllcnt ur rL iLrirrq
,ci,,r,,1.,t,,,1 thc rupfort rtrlal .lislst:lnac lhit sludcrrts rc'ci\'c:lt llolllc 'rrld in thc corrr
tunin. rll of $frrclt nt,tl bc i clcPerlrlcnt ofschool rnd tcrchcr inlltrcrlccs (Serldcrr
& Hor.D. | ()9,1I \Xlel.(cr ct rl.. L()tl-l). lfsruclerrts tiiicr 6orrr school to scl"rl irr rheir'
lerr,ls of :rchicvc|tctrt rtltcn entcr.i g .r st ltool. tceslrtcs ()1 :lb\olLrtc level'i of stlrdcnr
.rchieYcnrent .ll a littcr (l]ta lt)J-r rlot.ldcqtl.ltcl-v Ici'lect 't school\ suc'c\r lll
I r'r\illq
stuclenrs fiont thcil irlitr:rl cno.\ ler,cls. Hortcvcr. it seerrrs r.'rsorl.rble l() sr\' !h.ll
schools enrl irrsritrrrions tlrorrld be h!-l(l .rcroLlltt.tblc crtrl-r' ftrI tltirrgs thlt thct'cen bc
5. C)utcoDle Ev.rlurlon 109
:t :::-::-',t:tL aducatioll systenls expected to influcncc, not lbr dre characteristics srudents bring rvith thern rvhen
drey cornc to school (Wooclhousc & GoldsteiD, 1996).
j--:::-,:: -_: .ahie\-c1lIenl, science In line rvith this thinking, several attellpts have been lrladc to develoP slatistical
:-:r:,.: :: .t foreisn language. nlethodolo€;ies that \I'-ilJ pernrir an assessnleDt of lhe confibutions of schools lo
--:j:::i :: i'ju..1rion, f, d prepri- student developmerlt in situirlions in rvhich the noararldon assiqnnlent of students
.: :r]-.: r::'-- icscribed and conr- is assun-icd.These ncthodologies are bxsed on lwo concepts. OIlc rel:ltes lo "norllllrl"
=- :::::-::J Jnd inpienlented acadelnic proqlession, rvhlch is thc avcrage ProEircssion that students nlake fronl a
:::::. r, ', ::ien of correlates of given sterting poilt over a particular Pcriod in thc schooi systen (dcscribed as
i=::: _::,trmnin' to leirrll, tl1e' "expectcd" proeress).The orher is related to thc extent to s'hich individual students
-_._:::::::!. lnd resources in tl1e or groups of students (e.g., iD a cJass or school) cxceed or f:rll bclor"' lhat lver:lgc
progrcss in the spccified time period.The diflerence is rcllarded as representin€i the
value which a particr ar class ot scbool has "addcd" lo students'pro€lress.
Shtistical proccclures are usuxlly basecl ort nruftiple-reqrcssion analysis and involve
::r: :. :: li\c\Sed. haS been USed conlparinll actual studcnt outcorrcs widl expectations or Predictions determined
: : :: :iu;rrion. asscsgneDts of curpiricallv on the basis of relevant inputs (attenclutcc, gcnder, ethniciry earlicr
:.':r :: :;!t! or crltlinations. The aclievcnrenr). The rnosr sophisticated of these aPProaches use longitudind student
: .::r::riiJ to the level of the data. in r,vhich individual sltldctts'eailicr achievencnt scores xre In:ttched lvilh their
': .::=::i.ltti to bc trradc about latcr achieverncnr scores. ln the ltnDesscc Value Addcd Assessnrent Systeir, for
exairplc, estinuted student gain scores are aggrcgaled to the levels of leacher, school,
-::..- :' In Brirish lcague t:rbles, rnd system and are conlpaled rvith national Dornr laains, which each school is
:,--:--:
-:rla\ on public exanrina expectcd to achicvc. Schools u'ith scores icss than ttvo slandrrd deviations below
::,-. -'1. ln rhe Urrilcd States, the noml lllust show posirivc ptogrcss tll risk intcrvention by thc Srate (Sanders &
::-:: .-:coli and school disrricts Horn, 199'1).
-:r.. ,.. K:rcr. 1990). Sirnilarll'. in Problcms associatcd with the use of valuc rdded nteasulcs include inadequate
--i:r _ tnc basis of unad.iusrcd co\''eraqe of dre lchicvcmcnls of schools, which rna,v vlry by curliculuil area,
lladc leve1, rnd teacher; itrcomplete data lbr students arising 6onl absenrecisru L,r
:::: . . --::tion is concerned pri student turnover ratc; regrcssion to lhe nlean in st:rtistical analyses; problenN wilh
::-::-:-: ,)i the educational expe- rcliability of nreasures \-he1l lhe nunibcr of studeols in a school is sorall; and hor'v
::::: :). :ti\\e\_er. concern about to fictor in lhe contextuxl eI1;ct on lchievcmenl crealed b1' rhe abilit,r' level of
j-::_ .,r'. ri e\: urriorr rcsults arc stlrdentsin l school or class (Sandcrs & Horl, 199'1; School Curriculum atrd
,r :r-: :: .l jirirrsuishing betrveen Asscssnlent Authoritl 1994; Tymnx, 1995; Webster et al., 1994; Woodhouse &
-:-_ :--:. :lrrr :ue directly attrlb- Goldstcin, 1996).
:::-.: ::: .:.i.irion to nc! ilnpirct,
: :. --ir.:rrlion is reedily illus ISSUES IN OUTCOME EVALUATION
:: :::::.,. recognized as feflec!- Despite its popularin lhe use of outcolllc evaluatiotl gives rise to a seri-_s of issrlcs
1i_:-::.:. : _:tre\'erteJtl on enlerinq Frrst, since outcorrte evalLration rests prinarily on assl-llllpliolls related to planntng,
::-:: . : .!: :tlrnt!- and in the COln- incentives, accountability, ancl consutnerisrn, it is not likelv to lead to greltcr Lrnder
- -: :.:-_::r inlluences (Sanders standilg of rvhat goes on in progratls, or to an identilication of the fA.tors that
i=::::::: ,-_lt..ol to school in their allict outcomes (e.g., the relative contributions of tcachers, schools, and a v:rricry of
i *-:. :: :::rrlurc levels ofstudent other influences, rvithin a prograrn or oulslde it). However, many lvould regard
:-. - ''alrri iLrccels to lno\-firg progrcss in uridersrlnding "horv" rnd "wh)r" programs havc arl inlpact r\ ilnporllnl
:: ir:::ri :..1!on3ble to say that lor rell iorprovenen!. Second. and relatcd lo lhe first Pornt, is lhe issue of idcnri
: r:.. :r::ntngi thar thev can be flcation :utd specilicltiorr of thc responsibiliry of provitlcrs ind clietlts, particulariy
110 II Q!.rtions/Nl.rhods OricDrcd tlrlurtion Modch
in sltuations in rvhich rolcs rrtay be enrbiguous errr:1 rot clc-ar1v seprnted. Ho$: does
olre establish thal r plrticuhr ourconrc wAs- eve[ in p]rt, rl]leDable ft) rhe inllucncc
of :r persoD to t'horn rcsponsibilitv for it nr:r,v hrve been :rssigncd? For cratnple,
rvhilc' it ls rc'asonablc to ilsslLine thet a school end Lclchcrs bcar solre responsibiliry
for studerlt lchjeverJrerrt. do Do! stLldcrits :tnd p:rrerrts llso Lre:rr responsibiliry? If
thls js so, horv shoul,-1 responsibiliW benveen the perties bc apportloncd? Ald
should the alpportiollllrclrt be the sanrc lor all srudcnrs. in all circuntsLrrces, at l1l
:rge levels?
Thir-d. peribrnrance indicetor's ntay be used. recorcle,:l, lnd interpreted in vervinq
\vays, thus llivillg risc to prcblems of corlparabilitl. For exanrplc, a colc sct of lDcil-
sures clevelopecl tr1, e Fetler:rl ltllcr:tgerlcv'I:rsk Force lo Inonilor rnxrkel Progl:lnrs
in rhc Unitcd States rvas dcsigneci to lornr the basis ofst:rte levcl rn:rira!!c1lrc11t infor-
r tion lvstenr supporling perfor|ill1rce ll1onitolillq. Ilowever. since no stxre oper
ales a full_v irrrcgrated dita svslem serving nrultiple prograrrs, and slnce choice of
peforrDancc trrcasrrrcs djiler Aonr olc progrtnt to :lDother. clata rre nor directlv
comp:rr:rble (l3la1ock, 1999).
Folrrth, since nr ry outconrc evaluarions locus on a lintitc'd rirngc of L,utLuDlr\.
rhc daca thar art obtlincd llr:ty not acicquatclv rcflcct svslclr or progran goals and
objecrives.The lcnlpt:laion. of r:ourse. is to fucus on what is e.rsy to llrersurej but
lhls rlay be to the derrlnren! of imporunt obJecrives. Perrin (1c)9ll) rcrninds us rher
"rtnnv lctivities i| tLre public policy rcllur. by tbcir very llaturc, arc conplcx lDd
rnrangible rnd cannot be reducccl to :l nlrmericrl figure . . . What is nreasurecl, or
evc'n nreasueble. oftcn bears little resenrblance to lvhat i\ relevlrt" (pp. 373 373).
Houever, focusinla oll a linlitecl set of outconcs is likel-v ro rtrcal that othcr or-rt-
comes will be neglectccl in prograni illipler[elt:]tiol.
Fllrir, rvhen outconrc' c'vilLladon i\ :l\sociated with high srakes, nrc'c!ing thc
recluirerlrents of lrie:rsurinq ancl reporting tnay bccone rnorc ilnporirnt thin $hat
a program lr'?s desiqned to achicve. relulting in goal clisplacenent. ln educarion. lor
cvrnplc. rvhcrr .rsscssllrcl-it r'esults bcconrc drc goal ofinsrlncrion, thc tr'uc prrrpose
of the instructioul proces\ mxv bc subvcrtecl as goals arc lcoriented to neet or
erceed "stendards." Further, eflbrts to improve perforr[aice on the mealure clo not
ncccssirrll) rcsult in il]UrovcrleDt in the ilreas that proqrams \l'ere designed to
rclieve. When rlcctinlt slandarcls beconrcs thc basis fbr budgetary cleclsjols. thcrc
is fhc furrher coDsccllrence rhal ploqrarns thilt ilec! starrclerds, ratlicr th:rn progralD
goels. ma,v bc co|tintLc'd. rvhiLc pKrqranrs that nreet goals, but Dot standards, rnay be
discontinued.
Slxdr, rvhc'n ev:lluarions are bascd on plecletcrllilcLl objecrivcs or srantlards, it is
unlikcl,v that unirrendell or uranricip:rted consequcr.cs rvlll bc clcrcctcd. Scvenrh,
the interpretatjon of drta itr outcome evalultions nliry not adequately acknowledge
diversities in the environrlent in rvhich progranrs u,'ere implenrerlred. lt nre-v rvell
be that:r proeram is "successftrl" in one coDtext, bu! no! irr anothcr. FinaiJy, drc cost
ofoutcol[e eva]uadorl rnav clivert funds fiom other tleeds, a not uninportant con
sideration at a time of r-esource corstrai)ts (Baftisdch er al-, 1999: Ulalock. 11)99:
I)avics. 1999; N.rrricllo, 199t1; Perrin, 1t)98).
5 ()uti!ne Evrlu?tio. 111
OF EVALUATION
OUTCOME EVALUATION AND OTHER FORMS
:: : ::: i::lr.ltell. HOrv dOes ht amotrg traditional
::-- ::--::: :al. ro thc influence ln conclusion, rve tray ask:'Wherc does oulcome er'aluation
tiom three not
: ::: :::r--Tred? For exenple, ;;o;;-*t to'proslr:ulr evaluarion? The question nray be addresed
-exci.rsive
:::.::- ..:: rone lcsponsibility nt,.,**it,u points of vierv: the context in *'hich an evrluation is
"irli."iy process rnd decision
:= -. ::-,.r rcsponsibility? If ;;;;i..i .tr, its ,ticti,odologl; ancl its relarionship ro the poiicy
:r---:, ::' lFPorliolled? And nakirg.
:::: : .- :trclrntstanccs, lt all Context
as it hes recelltly developed'
:::. :::-i :::cerpreted in varYing As far as context is concerned, oLrtcone ev'lultion'
Iays' frtting rnolc conlfortablv
t:: :::,::-:i:- r core set ofntea- ,liftt., frott, traditionai approlches in a nunrber of
eval'atior approrch. First it
: :: : i:_r:or rll.rrket progranrs ."iri ir, u*""g.r'"1 rnre;enrs thar rvith any program in theorY f,t
knorviedge that
:::-:= -.. c] nlanlgenlent infor- ,.na, a t parr of :, b.trcattcratic routine' providing
"
**, i, ."l.u,trt, ,,t pohc,v Second, it frecluently involves accounlabiliry consid
i: ._. ::- sincc no slale oper
.-.
^"y rcporting of per{ormance
::::::-:_...rnrl since choice of .'*tinn,. ."l"ti,tg lo tha scrulinizetion of programs and
ev:rlr-retion is in tbe contcrt ofvery
indicators. Thircl, the nost conlnlon use ofsuch
=-:::::. Jat.r are not directllt broacl and compiex Prosrans (represeoted in'
for example' all rhc etTorts nrade by
yeers) r:rther lhitn nlore discrete and
: r :::_:::i r;1nqc of oLrtcomes, a school or school svstelll ovet e number of
as ntost conlnoniy
::,:-:::: trr Prograrn goals and ,,,l,r." al.".ly speciftecl prograns Fourth, outcolue eval[ation'
or expcrimental pro
:- ,:.a:: -i aui\ !o nleasure, but or"a,ir.,l, r.i",., ,,, ,.rtt jorrtg p,".,tae ralher lilan !o illnovativc
,o -,aa"..*, i"t'"r or econonric problcms' Thus' it is
.'ot 'ormally
: --i:rrt:: 1r)98) renrinds us that ;;"r*';:";; progr'ln evaltr'LtiL'n i\' nol
isso.iat.d..,itlt triai runs ofnerv progrrnrs, as rraditional
-- i::. : :i-rrc. ,lrc corllPiex and inplelrlenta
:,--::. \\_hrt is nreasured. or ,."u cor.bi.ecl *,ith qu,,litaiive approaches to itssess progfarr
".r"*1,
tion and irnPact
:
-: :, ::l;'r ani' (pp. 373 373).
:i: .-_ neln th:tt orher out- MethodologY
a
The tnethodologies of outconle cvaluatiorl bavc some
afliliry rvrth (lrl\ (Lqbfl\)
:::-. -:::-, lr:ekes. rlreeting the on Popperian logical positivism'
::r: :: ::: irllportant than what npp.o".lt",, r'hich rvere largely based
".r"i.r",i,:,,r-t
ernploying qutntitallve ,,,"",ot"'' chains' ancl xsPlralions tolvards general-
d"tloJti*
. ::.: .::::::nt. ln educalton, for
izlrion.while outconre inciicators in themselves rvill ot
provide valid causal knor'vl-
:: :,-.:-_::r.n, rhe true purposc evidelced in cfforts to identi$'
.'aga inr.r"* in causaliry associatecl with thcil use is
::=0. - : ::arriil)led to nleet ()r utrderlying thc usc of addcd value
:--::.. : r:r rhe ncasure do not coirelares of;rchievc.rc.r and in the assrtt.ptio.s
'-: :.:::i;rr \\'ere designcd to techniques.
point ro an afnniry with tradi
. ::: :-::iien decisiorrs, there Whjie thesc aspecls of oulconle evalua[on ll1ay
there are also indicatrotts that
: ._,.:-il::r. nther thrrr Prograln tional viervs of e.raluat""i end indced of research'
is perceived as :r genre that is distincl trolll tradidonal evalua
i: -i -: r10r standards, mly be .r.rr.ont"
-- "u,,luatiort conclusion seelrs warranted t'hen onc
tion (see Blalock, 1999; Pollitt, 1993)'This and ioter-
consiiers that outcolne lllorlitoring in n:rtional
i'"ptt'""t"d aisessncnts
:: -,.',_:]\_ei or standards, il is x bti'lg ittlrttot"tl by gover nmenr' :rncl international
:r-:, .r, aa detected. Sevenlh, ."",p-r*e studies)
"",r"."i rf,. ,",,t" titt.tt *rote trrdrrronrl apprcaches lo
::. _ . :;.'qu;lelv acknorvledge ";;;;t;, "', ""d "ti.-ptnd'tttll ''f'
cialuation (sec, e-g, European Ciornmission' 1997)
ii-:: r::r:-i'ntenred, It Dray rvcll
: ::: :: ::r!rr:hcr. Finally, the cost Policy and Decision Making
::::::. : :ror Lrrlinrportant con rhe usc of orttcone evalua
'::i- :: :.. . 1999: Blalock, 1999; At this stage, tl-rcre is iittle docurlentlrtion arailable on
tiotrs in a poiic-v coI1!c\t The extellt to which
inforttlation dcrivcd liom such evei-
112 Il O,,.rfor\/N1.rh.J\ ().i.nri i,rtrlrrin(,, M.dcl\
u:rtions enters thc policl lren:r rvill no doubt cliller iionr countrv ao countr\',
depending on a coLrntrv's traditiors oi llovcrrnncrlL end of poJicr' antl decision
i killlj, JS wcll as on rhe relationships t'hich luvc .lrc:rdy bccn cslrblishcd Lrctw!-cn
polic\uukers. decisiorr rnrkers, and ev:llLr:rtors. lnsot:rr as dre methociologr' of
outcorne evxlLrrtions seenrs r:losc to rhat illyolved in errrpiricll qLr llrrl-rhrc
epproaches. rvith thclr rrlion:rl view of the policl proccss. onc might cxpcct
outcorne infbrmarion to bc consiclcrccl !-xoqenoLrs lo lhe proces\, pror-iciing "oblec:
tive", "ncutral".:rnd epolitical l:rlorrrrltror to bL- use.l instr unrenrell,v in policl'anrl
decision making. Irr thls vicu: as in earlv evaluiltiolr cflbrts, thc c'v.rluitor hrs r role
ro plu rrr resoh'rng polic-v issues. but not ;rs a plaver in the actu:rl policv ptocess
(Rrdre1li & I)crtc, 1996). This conclusion is reinforcctl \vhcn \ic consiclcr thL'
nurnber of outcor[e evarlurtion pro]L'cls iD lvhich rhere often is no iclentif_rab1e "evrl
u:rtor." Indccd, thc lcrm e\-aluillion often does not haYc a proDrincnt placc in dis
courses on lhe activides of rvhet rvc' alc c.r11ing outconle ernlualioir.
This shorrid not sLrlprise rLs. grven the lirrrited rrr-rnbcr of goals of rrrlbr r,r.rrror
r
productiorr thlt arc consldclcd relevent to oLrtc(xlle evahr:ition. Of the sir go:rls iclen
tified b,v Bl.r1ock (1999) that nrorc convcntion:rl nlethods of evxjlretiorl \tlrt to
nrccr. outcollrc evalu:lti()n is ljkcl\.to addr-ess orrlv onc: ,:lctcrrninlng iia progrrnr\
outcornes for clicnts (and pcrhaps its ner inlprct) rle codsi\rent rvith dr-sircd out-
conres :lncl lo improve there outconrcs. ()ulcome evaluatioD is not likel,v to provi(1e
infornratiol on Blalockls t'ivc other gorls: $hethcr or not a prognlr).s irrtcncntrons
are as intended; r'hc'thcr r progr:rn is beiug delivered ro the iitended trrllct pop-
ulation; \'uhcther r prollr:rm is bcine inrplcnrcnred as intcndecl; identificrtion of the
rDljor influc|ccs shepine e progrrllis oulco res: or rhc ipproprielcncss, ulilit\', and
\ocietal vrlue ofpolicics o \rhich x progrxn is brsecl.
Thc u':Lf in which outcome cvaluirlior irlbrnration is predicteci to \\''()l k in sonle
svstems suqllesls lhat thc cilbrt to eccdrlrnoclitte dre inlor nrlliorr in policv r,r.ill bc
slight. lf, iirr example. the prinrc plrrlosc of provicling outcclrrle infolrrr.rtiun L,1r
school pcforrnancc ls ro ltrach to it re\\'alds or pllnishnrc'nls lbr school districrs.
sch()ols. or terc|els, lhcrr tber-e rvould seem ro be little nc.e,:l to rellec! on, ol lry
to understand. horv schools t'ilnctiorl, or $'h:rt it i\ etrout proqrrns that ficilit:rtcs
studclrt !jro\\th. Perheps, the LlLlcstiors r:riscd bv these issues rre too demrnding and
ch:rllenging for a brrsv adnrluistrator. lhe e:rsier corusc is ro irnport lnarket modeh
and lcavc it ro conlpetition and consurner choice to bririg lboLrt dcsitctl tcfbrnr.
Horvevcr. ls long as this:lpproach is fblloucrl, rnlnv questions thit li:ive tr:]didon-
allv occupiecl evitlultols \\ill rern:lin Lrnanswclccl: doL's r ploqranr contlibute to
inprovcnrcnr. is it ecluitable. rvhet are the unintendccl colscqrierraes. and et t'h:rt
cost i! chlllgc achle\,-ed?