:+2&281765($/,7<"

3$57,&,3$725<021,725,1*$1'(9$/8$7,21

$/,7(5$785(5(9,(:

,'6:25.,1*3$3(5

0DULVRO(VWUHOOD
DQG-RKQ*DYHQWD

6800$5<
:LWK PRUH DQG PRUH RSSRUWXQLWLHV WR SURPRWH EURDGEDVHG SDUWLFLSDWLRQ LQ GHYHORSPHQW WKHUH LV LQFUHDVHG
UHFRJQLWLRQWKDWPRQLWRULQJDQGHYDOXDWLRQVKRXOGDOVREHSDUWLFLSDWRU\7KLVSDSHUSUHVHQWVDOLWHUDWXUHUHYLHZ
RI H[SHULHQFHV LQ SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ 30 ( IURP DURXQG WKH ZRUOG XVHG LQ GLIIHULQJ
FRQWH[WV DQG LQYROYLQJ DOO NLQGV RI VWDNHKROGHUV ² 1*2V GRQRUV UHVHDUFK LQVWLWXWLRQV JRYHUQPHQW SHRSOH
V
RUJDQLVDWLRQV DQG FRPPXQLWLHV ,W LQWURGXFHV WKH NH\ SULQFLSOHV RI 30 ( LWV DSSOLFDWLRQV IRU GLIIHULQJ
SXUSRVHVDQGDQXPEHURIWRROVDQGPHWKRGVXVHGLQFOXGLQJSDUWLFLSDWRU\OHDUQLQJPHWKRGRORJLHVDV ZHOO DV
PRUH FRQYHQWLRQDO DSSURDFKHV )LQDOO\ LWUDLVHV NH\ LVVXHV DQGEURDG FKDOOHQJHV HPHUJLQJIURP WKH OLWHUDWXUH
DQGRXWOLQHVSRWHQWLDODUHDVIRUIXWXUHUHVHDUFK


$&.12:/('*0(176
7KLVSDSHUZDVRULJLQDOO\SUHSDUHGIRUWKH,QWHUQDWLRQDO:RUNVKRSRQ3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ
KHOG DW WKH ,QWHUQDWLRQDO ,QVWLWXWH IRU 5XUDO 5HFRQVWUXFWLRQ ,,55 LQ WKH 3KLOLSSLQHV LQ 1RYHPEHU 
)XQGLQJ IRU WKH OLWHUDWXUH UHYLHZ ZDV UHFHLYHG IURP WKH &62 DQG 3DUWLFLSDWLRQ 3URJUDPPH 8QLWHG 1DWLRQV
'HYHORSPHQW3URJUDPPHDQGWKH,QVWLWXWHRI'HYHORSPHQW6WXGLHV,'67KHZRUNVKRSZDVFRVSRQVRUHGE\
,,55 ,'6 ,QWHUQDWLRQDO ,QVWLWXWH IRU (QYLURQPHQW DQG 'HYHORSPHQW ,,(' 2[IDP +RQJ .RQJ 6LNLOL]D
,QWHUQDWLRQDO 8VHU
V 3HUVSHFWLYHV :LWK $JULFXOWXUDO 5HVHDUFK DQG 'HYHORSPHQW 83:$5' DQG WKH
3KLOLSSLQHV3DUWLFLSDWRU\5XUDO$SSUDLVDO35$1HWZRUN
$WWKHZRUNVKRSDGGLWLRQDOFDVHVDQGH[SHULHQFHVZHUHSUHVHQWHGZKLFKIXUWKHUHQULFKHGXQGHUVWDQGLQJRI
SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ 30 ( 7KHVH GLVFXVVLRQV DQG FDVH VWXGLHV PD\ EH IRXQG LQ WKH
:RUNVKRS 3URFHHGLQJV DYDLODEOH IURP ,,55 DV ZHOO DV LQ 3/$ 1RWHV  3DUWLFLSDWRU\ 0RQLWRULQJ DQG
(YDOXDWLRQDYDLODEOHIURP,,('LQ/RQGRQ$IRUWKFRPLQJERRNRQZLOOIXUWKHUH[SORUHWKHPDMRULVVXHVDQG
FKDOOHQJHVLQWKHILHOGGUDZLQJIURPOHVVRQVDQGWKHUDQJHRIH[SHULHQFHVWKDWHPHUJHGGXULQJWKHZRUNVKRS


,1752'8&7,21
,Q UHFHQW \HDUV
SDUWLFLSDWLRQ
 KDV EHFRPH D FULWLFDO FRQFHSW LQ GHYHORSPHQW ,QWHUQDWLRQDOO\ GRQRUV
JRYHUQPHQWV DQG 1*2V DUH LQVLVWLQJ XSRQ SDUWLFLSDWRU\ DSSURDFKHV LQ DVVHVVLQJ QHHGV DQG LQ LPSOHPHQWLQJ
SURJUDPPHV ,Q KLV UHFHQW ERRN :KRVH 5HDOLW\ &RXQWV" 5REHUW &KDPEHUV  GHVFULEHV WKH QHZ
DSSURDFKZKLFKVWDUWVZLWKSHRSOH
VNQRZOHGJHDVWKHEDVLVIRUSODQQLQJDQGFKDQJH
:LWK LQFUHDVHG HPSKDVLV RQ WKH LPSRUWDQFH RI SDUWLFLSDWLRQ LQ GHYHORSPHQW WKHUH LV DOVR D JURZLQJ
UHFRJQLWLRQ WKDW PRQLWRULQJ DQG HYDOXDWLRQ RI GHYHORSPHQW DQG RWKHU FRPPXQLW\EDVHG LQLWLDWLYHV VKRXOG EH
SDUWLFLSDWRU\ $V LQVWLWXWLRQV EHFRPH PRUH LQFOXVLYH LQ WKH
IURQWHQG
 RI SURMHFW GHYHORSPHQW ² WKDW LV LQ
SURPRWLQJSDUWLFLSDWLRQLQDSSUDLVDODQGLPSOHPHQWDWLRQ²WKHQTXHVWLRQVRI
ZKRPHDVXUHV
UHVXOWV DQG
ZKR
GHILQHV
 VXFFHVV EHFRPH FULWLFDO
:KR FRXQWV UHDOLW\"
 PD\ SURYH DV VLJQLILFDQW D TXHVWLRQ DV
ZKRVH UHDOLW\
FRXQWV"

3DUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQ30 (LVSDUWRIDZLGHUKLVWRULFDOSURFHVVZKLFKKDVHPHUJHG
RYHUWKHODVW\HDUVRIXVLQJSDUWLFLSDWRU\UHVHDUFKLQGHYHORSPHQW30 (GUDZVIURPYDULRXVSDUWLFLSDWRU\
UHVHDUFK WUDGLWLRQV LQFOXGLQJ SDUWLFLSDWRU\ DFWLRQ UHVHDUFK 3$5 VSHDUKHDGHG E\ WKH ZRUN RI 3DROR )UHLUH
)DOV%RUGDDQGRWKHUVSDUWLFLSDWRU\OHDUQLQJDQGDFWLRQLQFOXGLQJ5DSLG5XUDO$SSUDLVDO55$
DQGODWHU3DUWLFLSDWRU\5XUDO$SSUDLVDO35$GUDZLQJRQWKHZRUNRI5REHUW&KDPEHUVDQGPDQ\RWKHUV
DQG IDUPLQJ V\VWHPV UHVHDUFK )65 RU IDUPLQJ SDUWLFLSDWRU\ UHVHDUFK )35 GHYHORSHG E\ $PDQRU 
)DUULQJWRQDQG0DUWLQDQGRWKHUV
:KLOH LQWHUQDWLRQDOO\ ZH KDYH VHHQ D JUHDWHU LQWHUHVW LQ SDUWLFLSDWRU\ DSSURDFKHV WR UHVHDUFK DQG
GHYHORSPHQWZHKDYHDOVRVHHQDQLQFUHDVHGFRQFHUQZLWKPRQLWRULQJDQGHYDOXDWLRQE\GRQRUVJRYHUQPHQWV
1*2V DQG RWKHUV 7KLV LV DIIHFWHG E\ VHYHUDO IDFWRUV WKH WUHQG LQ PDQ\ PDQDJHPHQW FLUFOHV WRZDUGV

SHUIRUPDQFHEDVHGDFFRXQWDELOLW\
DQG
PDQDJHPHQWE\UHVXOWV
JURZLQJVFDUFLW\RIIXQGVOHDGLQJWRDGHPDQG
IRUGHPRQVWUDWHGVXFFHVVDPRYHWRZDUGVGHFHQWUDOLVDWLRQDQGGHYROXWLRQSURYLGLQJDQHHGIRUQHZIRUPVRI
RYHUVLJKWDQGWKHJURZLQJFDSDFLW\RI1*2VDQGFRPPXQLW\EDVHGRUJDQLVDWLRQVDVDFWRUVLQWKHGHYHORSPHQW
SURFHVV6LQFHWKHVFRQFHSWVRISDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQKDYHHQWHUHG WKH SROLF\PDNLQJ
GRPDLQ RI ODUJHU GRQRU DJHQFLHV DQG GHYHORSPHQW RUJDQLVDWLRQV PRVW QRWDEO\ WKH )RRG DQG $JULFXOWXUH
2UJDQLVDWLRQ )$2 WKH 8QLWHG 6WDWHV $JHQF\ IRU ,QWHUQDWLRQDO 'HYHORSPHQW 86$,' WKH 'DQLVK
,QWHUQDWLRQDO'HYHORSPHQW$JHQF\'$1,'$WKH8.'HSDUWPHQWIRU,QWHUQDWLRQDO'HYHORSPHQW>IRUPHUO\
WKH 2'$@ '),' WKH 6ZHGLVK ,QWHUQDWLRQDO 'HYHORSPHQW $XWKRULW\ 6,'$ WKH 1RUZHJLDQ $JHQF\ IRU
,QWHUQDWLRQDO 'HYHORSPHQW 125$' DQG WKH :RUOG %DQN +RZHV   VHH DOVR 5XGTYLVW DQG
:RRGIRUG%HUJHU
:KLOHWKHUHPD\EHJURZLQJLQWHUHVWLQ30 (WKHFRQFHSWLWVHOILVQRWQHZ'RFXPHQWHGH[SHULHQFHVRI
30 (LQWKHGHYHORSPHQWILHOGEHJDQWRVXUIDFHDVHDUO\DVWKHV+RZHVGLVFXVVHVVRPHRIWKHVH
HDUO\H[SHULHQFHVLQ30 (FLWLQJZRUNLQWKLVILHOGVSRQVRUHGE\2[IDPLQWKHV)HXHUVWHLQ
VZRUN
ZLWKUXUDOZRPHQLQ+RQGXUDVWRGHYHORSSULPDU\KHDOWKFDUHV\VWHPVDVZHOODVSDUWLFLSDWRU\IRUPVRI0 (
HVWDEOLVKHGLQ)UDQFRSKRQHFRXQWULHVXQGHUWKHQDPH
DQLPDWLRQUXUDOH
3UDWWDQG%R\GHQ:KLOHPDQ\
RUJDQLVDWLRQV FRQWLQXHG WR FDUU\ RXW PRUH FRQYHQWLRQDO IRUPV RI 0 ( VRPH 1*2V VXFK DV :RUOG
1HLJKERXUV8VHUV
3HUVSHFWLYHV:LWK$JULFXOWXUDO5HVHDUFKDQG'HYHORSPHQW83:$5'DQGWKH$JHQF\IRU
&RRSHUDWLRQ DQG 5HVHDUFK LQ 'HYHORSPHQW $&25' KDYH ORQJ EHHQ H[SHULPHQWLQJ ZLWK PRUH LQQRYDWLYH


SDUWLFLSDWRU\DSSURDFKHVVHH$UPRQLDDQG&DPSLODQ&DPSRVDQG&RXSDO5XJK6RPHRIWKH
HDUOLHVWSXEOLFDWLRQVRQSDUWLFLSDWRU\DSSURDFKHVWRPRQLWRULQJDQGHYDOXDWLRQLQFOXGHZRUNE\WKH6RFLHW\IRU
3DUWLFLSDWRU\ 5HVHDUFK LQ $VLD 35,$  3DUWLFLSDWRU\ 5HVHDUFK DQG (YDOXDWLRQ ([SHULPHQWV LQ
5HVHDUFK DV D 3URFHVV RI /LEHUDWLRQ DQG E\ 0DULH7KHUHVH )HXHUVWHLQ  3DUWQHUV LQ (YDOXDWLRQ
(YDOXDWLQJ'HYHORSPHQWDQG&RPPXQLW\3URJUDPPHVZLWK3DUWLFLSDQWV
$VLQWHUHVWKDVJURZQLQSDUWLFLSDWRU\DSSURDFKHVWRPRQLWRULQJDQGHYDOXDWLRQVRKDVWKHUDQJHRIWHUPV
XVHG WR GHVFULEH LW 7KHUH LV JUHDW YDULDWLRQ LQ WKH ZD\ RUJDQLVDWLRQV ILHOG SUDFWLWLRQHUV UHVHDUFKHUV HWF
XQGHUVWDQG WKH PHDQLQJ DQG SUDFWLFH RI SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ 7KH OLWHUDWXUH UHYLHZ IRXQG
WKDWWKHUHLVQRVLQJOHFRKHUHQWFRQFHSWXDOGHILQLWLRQRI30 (UDWKHUWKHUHLVZLGHVFRSHIRULQWHUSUHWDWLRQDV
30 ( PD\ PHDQ GLIIHUHQW WKLQJV WR GLIIHUHQW SHRSOH $EERW DQG *XLMW  &DPSLODQ  35,$ 
6RPHRIWKHWHUPVXVHGWRGHVFULEHGLIIHUHQW30 (DSSURDFKHVDUH

• 3DUWLFLSDWRU\HYDOXDWLRQ3(
• 3DUWLFLSDWRU\PRQLWRULQJ30
• 3DUWLFLSDWRU\DVVHVVPHQWPRQLWRULQJDQGHYDOXDWLRQ3$0(
• 3DUWLFLSDWRU\LPSDFWPRQLWRULQJ3,0
• 3URFHVVPRQLWRULQJ3UR0
• 6HOIHYDOXDWLRQ6(
• $XWRHYDOXDWLRQ
• 6WDNHKROGHUEDVHGHYDOXDWLRQ6WDNHKROGHUDVVHVVPHQW
• &RPPXQLW\PRQLWRULQJ&LWL]HQPRQLWRULQJ&0

7KRXJKWKHUHKDVEHHQJURZWKLQWKHXVHRI30 (DSSURDFKHVLQDYDULHW\RIVHWWLQJVDQGFRQWH[WVWKHUHKDV
EHHQOLWWOHZRUNWKDWKDVVWXGLHGRUGRFXPHQWHGWKHSURFHVVDQGLPSDFWRI30 (LWVHOI7KLVSDSHUDWWHPSWVWR
PDNH D FRQWULEXWLRQ WRZDUGV WKLV HQG E\ UHYLHZLQJ WKH DYDLODEOH OLWHUDWXUH RQ 30 ( IRFXVLQJ RQ KRZ DQG
ZKHUHLWLVEHLQJXVHG&KDSWHULWVXQGHUO\LQJFRQFHSWV&KDSWHUDQGLVVXHVDQGFKDOOHQJHVIRUWKHILHOG
&KDSWHU  ,Q DGGLWLRQ ZH KDYH FRPSLOHG DQ DQQRWDWHG OLVW RI PDQXDOV DQG UHVRXUFHV RQ WKH
WRROV
 DQG
PHWKRGVLQ30 (VHH$SSHQGL[$GGLWLRQDOLQIRUPDWLRQDERXWWKHVHUHVRXUFHVPD\EHIRXQGLQWKH35$
DEVWUDFWVZKLFKDUHDYDLODEOHIURPWKH,QVWLWXWHRI'HYHORSPHQW6WXGLHV,'6
7KLV OLWHUDWXUH UHYLHZ LV GUDZQ IURP D QXPEHU RI VRXUFHV LQFOXGLQJ MRXUQDO DUWLFOHV XQSXEOLVKHG
JUH\

OLWHUDWXUHPDQXDOVDQGJXLGHVSROLF\GRFXPHQWVDFDGHPLFSDSHUVDQQRWDWHGELEOLRJUDSKLHVDQGRWKHURYHUYLHZ
SDSHUV RQ 30 ( HVSHFLDOO\ WKRVH E\ $EERW DQG *XLMW  RQ PRQLWRULQJ HQYLURQPHQWDO FKDQJH E\ WKH
&HQWHURI&RQFHUQ&21&(51DQGE\$UPRQLDDQG&DPSLODQZKRZHUHVXSSRUWHGE\,'6WR
GRDUHYLHZRI$VLDQH[SHULHQFHV2WKHUUHJLRQDODQGFRXQWU\RYHUYLHZVZHUHFRQVLGHUHGLQFOXGLQJDVWXG\RI
30 ( H[SHULHQFHV LQ %ROLYLD $OFRFHU /L]iUUDJD 'HOJDGLOOR HW DO  DQG D 30 ( ZRUNVKRS UHSRUW RI
$IULFDQH[SHULHQFHV3$0)25.:HKDYHUHOLHGKHDYLO\RQPDWHULDOVIRXQGLQWKH35$UHDGLQJURRPDW
WKH,QVWLWXWHRI'HYHORSPHQW6WXGLHVWKH,'6OLEUDU\WKH,,('5HVRXUFH&HQWUHPDWHULDOVFROOHFWHGE\-RKQ
*DYHQWD WKURXJK KLV HDUOLHU ZRUN ZLWK WKH 1DWLRQDO /HDUQLQJ ,QLWLDWLYH LQ WKH 8QLWHG 6WDWHV DQG RXU RZQ


SHUVRQDO ILOHV DQG FROOHFWLRQV ,Q DGGLWLRQ ZH KDYH EHQHILWHG IURP FRQYHUVDWLRQV ZLWK D QXPEHU RI 30 (
SUDFWLWLRQHUVDQGFROOHDJXHVDW,'6DQGHOVHZKHUH
$IWHUFRPSLOLQJDQGUHYLHZLQJDVPXFKRIWKHDYDLODEOHOLWHUDWXUHDVSRVVLEOHZHVHOHFWHGFDVHVWXGLHVIRU
FORVHUDQDO\VLVVHH$SSHQGL[,QVHOHFWLQJWKHVHFDVHVZHVRXJKWWRDFKLHYHDUHSUHVHQWDWLRQRIWKHGLIIHUHQW
VHFWRUVUHJLRQVSXUSRVHVWRROVDQGPHWKRGVDFWRUVDQGSDUWLFLSDQWVLQZKLFK30 (LVEHLQJXVHG7KHFDVHV
SURYLGH LQVLJKWV LQWR WKH ULFK GLYHUVLW\ RI 30 ( H[SHULHQFHV WKDW H[LVWV JOREDOO\ DV ZHOO DV LQWR LVVXHV DQG
FKDOOHQJHVIRUWKHILHOG
7KLVSDSHULVZKDWWKHWLWOHLPSOLHVDOLWHUDWXUHUHYLHZ:HGRQRWVHHNWRPDNHDSDUWLFXODUDUJXPHQWDERXW
30 (RWKHUWKDQWRVHOHFWDVHULHVRILVVXHVIRUIXWXUHZRUNERWKFRQFHSWXDOO\DQGLQSUDFWLFH:HKDYHEHHQ
FRQVWUDLQHG E\ WLPH DQG GLVFRYHUHG YHU\ TXLFNO\ WKDW WKH KXJH DPRXQW DQG GLYHUVLW\ RI WKH SXEOLVKHG DQG
XQSXEOLVKHG OLWHUDWXUH ZRXOG PDNH D FRPSUHKHQVLYH
VWDWH RI WKH DUW
 UHYLHZ LPSRVVLEOH :H KDYH DOVR EHHQ
KHDYLO\FRQVWUDLQHGE\DIRFXVRQOLWHUDWXUHDYDLODEOHLQ(QJOLVKWKRXJKZHKDYHVRXJKWPDWHULDOVLQ6SDQLVKDQG
)UHQFK DV ZHOO 5HJDUGOHVV RI LWV OLPLWDWLRQV ZH KRSH WKDW WKLV RYHUYLHZ ZLOO DOVR FRQWULEXWH WR WKH IXUWKHU
VWUHQJWKHQLQJRIFRQFHSWVDQGSUDFWLFHVLQWKHSDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQILHOG

7+(385326(62)3$57,&,3$725<021,725,1*$1'(9$/8$7,21
,QFUHDVLQJO\DURXQGWKHZRUOGSDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQLVEHLQJXVHGIRUGLIIHULQJSXUSRVHVDQG
LQ GLIIHULQJ VHFWRUV )DUPHUV LQ ,QGLD %UD]LO 9LHWQDP DQG 0H[LFR DUH EHFRPLQJ PRUH HIIHFWLYH SODQQHUV DQG
GHFLVLRQ PDNHUV FKRRVLQJ DQG OHDUQLQJ IURP DOWHUQDWLYH SURGXFWLRQ VWUDWHJLHV :RPHQ LQ ,QGLD DUH EHFRPLQJ
LQYROYHG LQ KHDOWK SODQQLQJ DQG LQ HVWDEOLVKLQJ DQG PDQDJLQJ WKHLU RZQ VDYLQJV DQG FUHGLW SURJUDPPHV
&RPPXQLW\ OHDGHUV LQ WKH 8QLWHG 6WDWHV DUH GHYHORSLQJ WKHLU RZQ YLVLRQ RI FKDQJH DQG DFWLYHO\ VHHNLQJ WR
LQIOXHQFHDQGUHIRUPJRYHUQPHQWSROLF\)XQGLQJDQGVXSSRUWLQJLQVWLWXWLRQVEDVHGLQ%DQJODGHVK DQG LQ WKH
8. DUH H[SHULPHQWLQJ ZLWK SDUWLFLSDWRU\ DSSURDFKHV WR VWUHQJWKHQ DQG LPSURYH WKHLU RZQ SHUIRUPDQFH LQ
SURYLGLQJGHYHORSPHQWDVVLVWDQFH
,QWKLVVHFWLRQZHH[SORUHWKHUDQJHRIZD\VLQZKLFKSDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQLVEHLQJXVHG
LQDGHYHORSPHQWFRQWH[W$SSHQGL[SUHVHQWVDVXPPDU\RIFDVHH[DPSOHVZKLFKZHVHOHFWHGLQRXUUHYLHZ
RIWKHOLWHUDWXUHWRLOOXVWUDWHWKHW\SHVRISURMHFWVDQGSURJUDPPHVXVLQJSDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQ
LQ GLIIHULQJ VHFWRUV DQG UHJLRQV $V WKH PDWUL[ VKRZV WKH PDMRULW\ RI GRFXPHQWHG H[SHULHQFHV RI 30 (
VXUYH\HGRFFXULQWKHDJULFXOWXUDOHQYLURQPHQWDODQGUXUDOGHYHORSPHQWVHFWRUV$OWKRXJKFRQVFLRXVDWWHPSWV
ZHUHPDGHWRDFKLHYHDEDODQFHGUHSUHVHQWDWLRQRIFDVHVWXGLHVIRUWKHSXUSRVHRIWKLVRYHUYLHZZHIRXQGWKDW
GRFXPHQWHGH[SHULHQFHRI30 (LQWKHKHDOWKDQGHGXFDWLRQVHFWRUZDVOHVVUHDGLO\DYDLODEOH
*LYHQWKHYDULHW\RIDSSURDFKHVWR30 (LWLVGLIILFXOWWRGHYHORSFDWHJRULHVZKLFKGLVWLQJXLVKEHWZHHQLWV
W\SHVDQGXVHVWKRXJKVHYHUDOVWXGLHVKDYHWULHGWRGRVRVHH$EERWDQG*XLMW$UPRQLDDQG&DPSLODQ
&21&(51,QWKHLUVXUYH\RIH[SHULHQFHVLQ$VLD$UPRQLDDQG&DPSLODQQRWHGWKDWWKHUH
H[LVWVDJUHDWGLYHUVLW\LQFRQFHSWVPHWKRGVDQGDSSOLFDWLRQVDGRSWHG7KH\VXUYH\FDVHVGUDZQIURPGLIIHUHQW
VHFWRUVRIWKH
GHYHORSPHQW
ILHOGLQFOXGLQJDJULFXOWXUHSXEOLFVHUYLFHJRYHUQPHQWKHDOWKHQWHUSULVHOLYHOLKRRG
HQYLURQPHQWRUUHVRXUFHFRQVHUYDWLRQDQGLQWHJUDWHGFRPPXQLW\GHYHORSPHQW+RZHYHUWKHDXWKRUVSRLQWRXW
WKDWWKHPRVWFRPPRQH[SHULHQFHVRI30 (SHUWDLQHGWRDJULFXOWXUDOUHVHDUFKDQGFRPPXQLW\GHYHORSPHQW,Q
DGGLWLRQUHSRUWLQJDQGGRFXPHQWDWLRQRI30 (SURFHVVHVDQGILQGLQJVZHUHXVXDOO\XQGHUWDNHQE\LQGLYLGXDOV


DQGLQVWLWXWLRQVIURPWKHQRQJRYHUQPHQWRUSULYDWHVHFWRUUDWKHUWKDQIURPJRYHUQPHQWVHFWRUV1*2VDQG
FRPPXQLW\RUSHRSOH
VRUJDQLVDWLRQV32VZHUHDOVRUHJDUGHGDVNH\DFWRUVLQLQWURGXFLQJDQGVXVWDLQLQJWKH
30 (SURFHVV2QHRWKHUREVHUYDWLRQZDVWKDW30 (JHQHUDOO\ ZDV FDUULHG RXW LQ D SURMHFW FRQWH[W DQG LQ
VXSSRUWRIHLWKHUUHVHDUFKRXWUHDFKRUWUDLQLQJJRDOV
7KH6RXWK²6RXWK6KDULQJ)RUXPZKLFKEURXJKWWRJHWKHU30 (H[SHULHQFHVLQ$IULFDIRXQGWKDW30 (
PD\EHXVHGIRUWKHPDQDJHPHQWRISURMHFWVVHOIDVVHVVPHQWSURMHFWUHYLHZVRUJDQLVDWLRQDODVVHVVPHQWDQG
FDSDFLW\ EXLOGLQJ 30 ( LV JHQHUDOO\ FDUULHG RXW E\ GLIIHUHQW VWDNHKROGHUV LQFOXGLQJ FRPPXQLW\EDVHG
RUJDQLVDWLRQV1*2VJRYHUQPHQWVDQGGRQRUVFRQVXOWDQWVDQGUHVHDUFKHUV3$0)25.
6LPLODUO\LQWKHLUVXUYH\RI%ROLYLD$OFRFHU/L]iUUDJD'HOJDGLOORHWDOILQGWKDWDGLYHUVHUDQJHRI
DFWRUV EHFRPH LQYROYHG LQ WKHVH SDUWLFLSDWRU\ SURFHVVHV QDPHO\ LQGLJHQRXV JURXSV VRFLDO RUJDQLVDWLRQV RU
SHRSOH
V RUJDQLVDWLRQV OHDGHUV GLULJHQWHV DQG ORFDO DQG FHQWUDO JRYHUQPHQW RIILFLDOV 3ULQFLSDO VHFWRUV
UHSUHVHQWHGLQWKHVXUYH\LQFOXGHDJULFXOWXUHDQGOLYHVWRFNDJURSHFXDULDIRUHVWU\QDWXUDOUHVRXUFHFRQVHUYDWLRQ
PLFURHQWHUSULVH EDVLF LQIUDVWUXFWXUH SULPDU\ KHDOWK FDUH HGXFDWLRQ DV ZHOO DV LQWHUVHFWRUDO DQG SROLWLFDO
GHYHORSPHQWLHFDSDFLW\EXLOGLQJDQGSROLWLFDORUJDQLVLQJ
,QDQRWKHUUHYLHZRIPRQLWRULQJDQGHYDOXDWLRQLQWKHHQYLURQPHQWDOILHOG$EERWDQG*XLMWPDNH
VLPLODUREVHUYDWLRQVZLWKUHJDUGWRWKHVHWWLQJVDQGFLUFXPVWDQFHVXQGHUZKLFK30 (LVSUDFWLVHG+RZHYHU
WKH\ SRLQW RXW WKDW WKH PDMRULW\ RI GHYHORSPHQW OLWHUDWXUH RQ 30 ( VXUYH\HG GHVFULEHV SDUWLFLSDWRU\
DSSURDFKHV ZLWKLQ D SURMHFWUHODWHG FRQWH[W UHIOHFWLQJ H[WHUQDOO\ OHG DSSURDFKHV 7KH DXWKRUV QRWH WKDW IHZ
H[DPSOHVGHVFULEHFRPPXQLW\OHGDSSURDFKHVWR0 (HVSHFLDOO\WKRVHZKLFKDGGUHVVFRQWLQXRXVPRQLWRULQJ
RIWKHZLGHUQDWXUDOHQYLURQPHQWDQGWUDFNLQJRIORFDOFKDQJHV
2XURZQVXUYH\RI30 (OLWHUDWXUHDOVRGHPRQVWUDWHVWKDW30 (WKLQNLQJDQGSUDFWLFHLVZLGHVSUHDGDQG
H[WUHPHO\GLYHUVH5DWKHUWKDQLGHQWLI\LQJDQGFODVVLI\LQJWKHYDULRXV
W\SHV
RI30 (ZHKDYHWULHGWRJURXS
WKHUDQJHRIH[SHULHQFHVLQWHUPVRIWKHSXUSRVHVIRUZKLFK30 (LVEHLQJXVHGDQGLQZKDWNLQGVRISURMHFWV
DQGVHWWLQJV%DVHGRQRXUVXUYH\RIFDVHVDQGNH\OLWHUDWXUHILYHJHQHUDOIXQFWLRQVRI30 (PD\EHLGHQWLILHG

• ,PSDFWDVVHVVPHQW
• 3URMHFWPDQDJHPHQWDQGSODQQLQJ
• 2UJDQLVDWLRQDOVWUHQJWKHQLQJRULQVWLWXWLRQDOOHDUQLQJ
• 8QGHUVWDQGLQJDQGQHJRWLDWLQJVWDNHKROGHUSHUVSHFWLYHV
• 3XEOLFDFFRXQWDELOLW\

2IWKHVHWKHVXUYH\RIOLWHUDWXUHVHH$SSHQGL[ILQGVWKDW30 (LVSULPDULO\XVHGLQGHYHORSPHQWIRUWKH

ILUVW WZR DUHDV RI LPSDFW DVVHVVPHQW DQG SURMHFW PDQDJHPHQW +RZHYHU WKH FDVH VWXGLHV DOVR GHPRQVWUDWH
LQFUHDVHG XVH RI 30 ( IRU QHZHU SXUSRVHV LQFOXGLQJ RUJDQLVDWLRQDO VWUHQJWKHQLQJ XQGHUVWDQGLQJ DQG
QHJRWLDWLRQ RI VWDNHKROGHUV
 SHUFHSWLRQV DQG LQFUHDVHG SXEOLF DFFRXQWDELOLW\ ,Q SUDFWLFH RI FRXUVH WKHVH
IXQFWLRQV RI 30 ( DUH LQWHUGHSHQGHQW DQG RIWHQ RYHUODS 8OWLPDWHO\ WKH XVHV DQG SXUSRVH RI 30 ( ZLOO
GHSHQGRQWKHSDUWLFXODUREMHFWLYHVDQGLQIRUPDWLRQQHHGVRIWKHSURMHFWRUSURJUDPPHLQLWLDWLYH 83:$5'



30 (IRU,PSDFW$VVHVVPHQW
2QHFRPPRQIXQFWLRQRI30 (LVWRHYDOXDWHWKH LPSDFW RI D JLYHQ SURJUDPPH DQG WKH FKDQJHV WKDW KDYH
RFFXUUHG DV D UHVXOW RI SURJUDPPH LQLWLDWLYHV 35,$   7KH HPSKDVLV LV RQ WKH FRPSDULVRQ EHWZHHQ
SURJUDPPHREMHFWLYHVDQGDFWXDODFKLHYHPHQW$VVHVVLQJSURMHFWLPSDFWVFDQKHOSGLVWLQJXLVKZKHWKHURUQRWD
SURMHFWLQWHUYHQWLRQVDUHLQIDFWDFKLHYLQJWKHLULGHQWLILHGREMHFWLYHVZKHWKHURUQRWESURJUDPPHREMHFWLYHV
UHPDLQUHOHYDQWRYHUWLPHFWKHEHVWDFWLRQVWUDWHJLHVKDYHEHHQSXUVXHG
&21&(51  UHIHUV WR WKLV DSSURDFK DV SDUWLFLSDWRU\ LPSDFW PRQLWRULQJ 3,0 ZKLFK PD\ EH
FKDUDFWHULVHGE\WKHZD\GLIIHUHQWDFWRUVRUVWDNHKROGHUVDWYDULRXVOHYHOV²JUDVVURRWVRUJDQLVDWLRQV1*2VDQG
IXQGLQJDJHQFLHV²ZRUNWRJHWKHULQRUGHUWRDVVHVVDQGUHIOHFWRQSURMHFWLPSDFWV+RZHYHUWKLVIXQFWLRQRI
30 ( KLJKOLJKWV WKH DFWLYH LQYROYHPHQW RI WKH ORFDO SRSXODWLRQ HVSHFLDOO\ SHRSOH OLNHO\ WR EHQHILW DQG JDLQ
IURPWKHSURJUDPPHVRUWKRVHZLWKGLUHFWSURJUDPPHLQYROYHPHQWHJILHOGOHYHOVWDIIVHH*RKO*HUPDQQ
DQG6FKZDU]35,$
6LPLODUO\ -DFNVRQ  GHILQHVWKH FRQFHSW RI SDUWLFLSDWRU\ LPSDFW DVVHVVPHQW3,$
DV D SURFHVV RI
HYDOXDWLRQRIWKHLPSDFWVRIGHYHORSPHQWLQWHUYHQWLRQVZKLFKLVFDUULHGRXWXQGHUWKHIXOORUMRLQWFRQWURORI
ORFDO FRPPXQLWLHV LQ SDUWQHUVKLS ZLWK SURIHVVLRQDO SUDFWLWLRQHUV    ,Q 3,$ FRPPXQLW\ UHSUHVHQWDWLYHV
SDUWLFLSDWHLQWKHGHILQLWLRQRILPSDFWLQGLFDWRUVWKHFROOHFWLRQRIGDWDWKHDQDO\VLVRIGDWDWKHFRPPXQLFDWLRQ
RI DVVHVVPHQW ILQGLQJV DQG HVSHFLDOO\ LQ SRVWDVVHVVPHQW DFWLRQV GHVLJQHG WR LPSURYH WKH LPSDFW RI
GHYHORSPHQWLQWHUYHQWLRQVLQWKHORFDOLW\

6HYHUDOH[SHULHQFHVRQ30 (PD\EHGUDZQXSRQWRLOOXVWUDWHKRZLWPD\IXQFWLRQDVDQLPSDFWHYDOXDWLRQ
WRRO ,Q 1RUWKHUQ *KDQD LPSDFW HYDOXDWLRQ ZDV XVHG WR WUDFN SURJUHVV DQG FKDQJH DV D UHVXOW RI SURMHFW
LQWHUYHQWLRQVRYHUWLPHVHH%R[
7KRPSVRQ DQG 3UHWW\  DOVRGHVFULEH D SDUWLFLSDWRU\ LPSDFWVWXG\ RI D VRLO DQG ZDWHU FRQVHUYDWLRQ
SURJUDPPH VXSSRUWHG E\ WKH JRYHUQPHQW RI .HQ\D ,Q WKLV SDUWLFXODU H[SHULHQFH JRYHUQPHQW ILHOG ZRUNHUV
FROODERUDWHGZLWKORFDOSHRSOHOLYLQJRQRUQHDUSURJUDPPHVLWHVWRDVVHVVWKHOLQNDJHVEHWZHHQWKHSURFHVVRI
SURMHFW LPSOHPHQWDWLRQ DQG WKH VXEVHTXHQW LPSDFWV LQ WHUPV RI VRLO DQG ZDWHU FRQVHUYDWLRQ DGRSWLRQ E\ WKH
FRPPXQLWLHV7KHVWXG\IRXQGWKDWWKHLPSOHPHQWDWLRQUDWHVIRUVRLODQGZDWHUFRQVHUYDWLRQZHUHVXEVWDQWLDOO\
JUHDWHU ZKHQ SDUWLFLSDWLRQ LQ SODQQLQJ DQG LPSOHPHQWDWLRQ ZDV LQWHUDFWLYH DQG LQWHUGLVFLSOLQDU\ EHWZHHQ
JRYHUQPHQWH[WHQVLRQLVWVDQGORFDOSHRSOHDQGZHUHOHVVVRZKHQSDUWLFLSDWLRQZDVVLPSO\FRQVXOWDWLYH

30 (IRU3URMHFW3ODQQLQJDQG0DQDJHPHQW
:KLOHRQHSXUSRVHRI30 (PD\EHWRDVVHVVWKHLPSDFWRIDSURMHFWRUSURJUDPPHRYHUWLPHDQRWKHUPD\EH
WR JDLQ LQ D WLPHO\ DQG HIIHFWLYH ZD\ LQIRUPDWLRQ ZKLFK FDQ EH XVHG IRU LPSURYLQJ SURMHFW SODQQLQJ DQG
LPSOHPHQWDWLRQVHH%URZQ$VDSURMHFWPDQDJHPHQWWRRO30 (LVXVHGE\GLIIHUHQWVWDNHKROGHUVWR
DQDO\VH DQG UHIOHFW V\VWHPDWLFDOO\ RQ WKHLU H[SHULHQFH DQG WR SODQ IRU IXWXUH JRDOV DQG DFWLYLWLHV 83:$5'

&DPSRVDQG&RXSDODUJXHWKDWRQHRIWKHPDLQIXQFWLRQVRISDUWLFLSDWRU\HYDOXDWLRQLVWRSURYLGH
VWDNHKROGHUVDQG SURJUDPPH PDQDJHUV ZLWK LQIRUPDWLRQ WR DVVHVV ZKHWKHU  SURMHFW REMHFWLYHVKDYH EHHQ PHW
DQGKRZUHVRXUFHVKDYHEHHQXVHGLQRUGHUWRKHOSLPSURYHSURJUDPPHLPSOHPHQWDWLRQDQGPDNH


 Box 2.1 D eve loping an ev aluat ion f r ame wo rk fo r t rac king p roje ct im pact s ov e r ti me
A participatory impact assessment approach was used to evaluate the Northern Region Rural Integrated Programme (NORRIP) in
Northern Ghana. It assessed the impact of local capacity building on access to improved water supply and sanitation services by
using an evaluation framework known as the Village Development Capacity Index (VDCI). The VDCI framework was developed by
an external evaluation team commissioned by NORRIP after intensive consultations with different project stakeholders, including
village representatives. This index attempts to rank communities in the project area in terms of their performance on poverty
indicators and on village development capacity indicators.
Data was collected through interviews with households and key informants, focus group discussions, and field observation.
Rural residents provided feedback on collected data. The major stakeholders – who include village representatives, implementing
agencies, the donor and other allied agencies – participated in the process of allocating scores (ranging from one to five) on each of
these indicators to the villages under study. The scoring process facilitated dialogue and consensus building among the various
stakeholders, with village representatives providing feedback on the findings of the VDCI scoring exercise. Scores on the VDCI for
each village can then be used by monitoring teams to track changes and impacts over time to assess progress.
Source: Gariba 1995; Jackson 1995

FULWLFDOGHFLVLRQVDERXWSURMHFWIXQGLQJVHHDOVR35,$²7KLVLVLOOXVWUDWHGLQ*XMDUDW,QGLDZKHUH

30 (ZDVXVHGWRDLGYLOODJHFRPPXQLWLHV LQGHFLVLRQPDNLQJ IRU LPSURYHG SODQQLQJ DQG PDQDJHPHQW RI D
ZDWHUVKHGSURJUDPPHVHH%R[

Box 2.2 PM & E as an int ernal is ed p roc ess fo r p ro ject plan ning and m anag ement
The Aga Khan Rural Support Programme (AKRSP), a non-governmental organisation promoting natural resource management,
works collaboratively with village communities in rural Gujarat in developing a watershed management programme. The programme
integrates a participatory planning and monitoring component, which includes the following broad sequence of steps:

• Village extension volunteers engage farmers in discussions to identify their priority concerns, which become key variables for
information gathering.
• Farmers are encouraged to make 'ground maps' of their fields which serve as base line and impact study maps, showing
before/after conditions of soil and water conservation treatments.
• These 'before' and 'after' maps are collected, followed by intensive discussions among farmers regarding the findings and
inconsistencies in data collected.
• Presentation of findings leads on to a discussion of alternative technologies. For instance, experiments may be carried out to
compare different types of soil and water conservation techniques.
• Generation of technology and production strategies, which are then adapted to village, circumstances.

This approach encourages farmers to become more reflective and conscious about different strategies for improving productivity
and managing scarce resources. PM&E becomes an internalised process for local communities who use this approach in order to
make decisions regarding production, investment, and technology choice. In this context, participatory monitoring is linked to project
planning and management, leading to decision-making and identifying alternative action strategies.
Source: Shah, Hardwaj, and Ambastha 1993


 2WKHU FDVHV KDYH DOVR XVHG 30 ( IRU WKLV SXUSRVH )RU H[DPSOH LQ WKH 6LDYRQJD $JULFXOWXUDO
'HYHORSPHQW 3URMHFW 6$'3 LQ =DPELD 30 ( LV GHVLJQHG PDLQO\ IRU WKH SXUSRVH RI LPSURYLQJ SURMHFW
PDQDJHPHQWFRPSDULQJSODQQHGDQGDFWXDODFKLHYHPHQWVLQRUGHUWRVXJJHVWLPSURYHPHQWVIRUIXWXUHSODQQLQJ
DQGLPSOHPHQWDWLRQ1DJHOHWDO,QDGGLWLRQWKHLQIRUPDWLRQJHQHUDWHGWKURXJKWKH0,/6V\VWHPRIWKH
0RXQWDLQ5XUDO'HYHORSPHQW3URJUDPPH05'3SURJUDPPHLQ9LHWQDPVHH%R[LVDOVRXVHGJHQHUDOO\
IRUSURMHFWPDQDJHPHQWDQGSODQQLQJIXWXUHDFWLYLWLHV

30 (IRU2UJDQLVDWLRQDO6WUHQJWKHQLQJDQG,QVWLWXWLRQDO/HDUQLQJ
$QRWKHUPDMRUIXQFWLRQRI30 (LVWRFUHDWHDOHDUQLQJSURFHVVWRVWUHQJWKHQRUJDQLVDWLRQDODQGLQVWLWXWLRQDO
OHDUQLQJ,QWKLVFRQWH[WVHOIHYDOXDWLRQDVRQHDSSURDFKWR30 (LVXQGHUWDNHQIRUSHRSOHWRHYDOXDWHWKHYHU\
REMHFWLYHV RI WKH SURMHFW WKHPVHOYHV DQG WR DVVHVV WKHLU RZQ RUJDQLVDWLRQDO FDSDFLWLHV
ZHUH REMHFWLYHV WRR
OLPLWHGRURYHUO\DPELWLRXV"'LGWKH\UHIOHFWWKHIHOWQHHGVRUUHDOQHHGVRIPHPEHUVRIWKHFRPPXQLW\"
HWF
5XJK
2QHRIWKHPDLQREMHFWLYHVRIVHOIHYDOXDWLRQLVWRHQKDQFHWKHVXVWDLQDELOLW\UHSOLFDELOLW\DQGHIIHFWLYHQHVV
RIGHYHORSPHQWHIIRUWVWKURXJKWKHVWUHQJWKHQLQJRISHRSOH
VRUJDQLVDWLRQDOFDSDFLWLHV,WDLPVWRHQDEOHSHRSOH
WR NHHS WUDFN RI WKHLU SURJUHVV E\ LGHQWLI\LQJ DQG VROYLQJ SUREOHPV WKHPVHOYHV DQG E\ EXLOGLQJ RQ DQG
H[SDQGLQJDUHDVRIDFWLYLW\ZKHUHVXFFHVVLVUHFRJQLVHG&21&(51
7KH SDUWLFLSDWRU\ LPSDFW VWXG\ RI D VRLO DQG ZDWHU FRQVHUYDWLRQ SURJUDPPH LQ .HQ\D LOOXVWUDWHV KRZ
30 (FDQFRQWULEXWHWRRUJDQLVDWLRQDOOHDUQLQJ:KLOHWKHLPSDFWVWXG\ZDVXVHIXOLQUHYHDOLQJORFDOIDUPHUV

SHUVSHFWLYHV DQG FULWHULD IRU DVVHVVLQJ SURMHFW LPSDFWV WKH ILQGLQJV DUH DOVR EHLQJ XVHG WR KHOS LPSURYH WKH
0LQLVWU\RI$JULFXOWXUH
VRZQRSHUDWLQJSURFHGXUHVLQWHUPVRILQYROYLQJORFDOIDUPHUVPRUHDFWLYHO\LQSURMHFW
SODQQLQJ LPSOHPHQWDWLRQ PRQLWRULQJ DQG DQDO\VLV 7KRPSVRQ DQG 3UHWW\  ,Q DQRWKHU H[DPSOH LQ WKH
3DUWLFLSDWLYH,QWHJUDWHG'HYHORSPHQWRI:DWHUVKHGV3,'2:SURMHFWVHH%R[VHOIHYDOXDWLRQVHUYHGWR
VWUHQJWKHQSDUWQHUVKLSVEHWZHHQGLIIHUHQWVWDNHKROGHUVDVSDUWLFLSDQWVLQFUHDVHGWKHLUXQGHUVWDQGLQJRISURMHFW
SURFHVVHVDQGRXWFRPHVDQGFODULILHGWKHLUUROHVZLWKLQWKHSURMHFW6RPPHUGHVFULEHVWKHSURFHVVDV
DQ
LQWHULQVWLWXWLRQDOOHDUQLQJDSSURDFK
ZKHUHE\MRLQWUHIOHFWLRQHQFRXUDJHVSURMHFWSDUWQHUVWRFDUU\RXW D PRUH
GLUHFWDQGLQGHSWKDQDO\VLVRISURMHFWDFKLHYHPHQWVRIWKHLURZQSHUIRUPDQFHZLWKLQLWDQGWRGUDZVLJQLILFDQW
OHVVRQVIURPWKHPIRUIXWXUHSODQQLQJ
VHHDOVR+XPEHUW'UR]
,Q DQRWKHU FDVH WKH &RRSHUDWLYH IRU $PHULFDQ 5HOLHI (YHU\ZKHUH &$5( =DPELD HVWDEOLVKHG D
SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ SURFHVV ZKLFK DLPV DW EXLOGLQJ RUJDQLVDWLRQDO FDSDFLWLHV DW ERWK WKH
SURMHFWDQGWKHFRPPXQLW\OHYHOVHH%R[


 Bo x 2.3 B u ild ing a le a rning o rgan isat ion
Since 1994, CARE Zambia has promoted the concept of a ’learning organisation’ in order to encourage more innovative and
adaptive learning, especially among CARE staff members. CARE pursues seven major strategies, which are considered
benchmarks or ’building blocks’ for a learning organisation:

• thriving on change
• encouraging experimentation
• communicating success and failure
• facilitating learning from the surrounding environment (i.e. from communities and beneficiaries)
• facilitating learning from staff through training and group learning
• rewarding learning
• promoting a sense of caring

CARE promotes various initiatives as part of the process of becoming a learning organisation, which have important implications for
project monitoring and evaluation. Staff receive training in using participatory approaches and methods, and work together with
communities and beneficiaries in learning how to plan and monitor project activities, analyse results and use information to modify
activities.
In the Livingstone Food Security Project (LFSP), CARE Zambia staff established and trained community based teams –
known as village management committees (VMCs) – to design, implement and monitor community-driven development activities. In
the LFSP project, local participants identify different levels of wealth and different categories for ranking wealth in each village. The
information provides staff and VMCs a basis for tracking changes experienced by individual households in each wealth or well-being
category over time. Hence, capacity building and institutional learning not only takes place at the project level among field staff, but
also at the community level. This is important for monitoring the impact of project activities and determining actions to overcome
challenges.
Source: Ward 1997

30 (IRU8QGHUVWDQGLQJDQG1HJRWLDWLQJ6WDNHKROGHU3HUVSHFWLYHV
30 (PD\DOVREHXVHGDVDSURFHVVZKLFKDOORZVGLIIHUHQWVWDNHKROGHUVWRDUWLFXODWHDQGSUHVHQWWKHLUQHHGV
LQWHUHVWVDQGH[SHFWDWLRQV7KLVSURFHVVFDQEHVKDSHGLQZD\VWKDWHQDEOHSHRSOHWRXQGHUVWDQG
WKHYLHZVDQG
YDOXHVWKH\VKDUHDQGZRUNWKURXJKWKHLUGLIIHUHQFHVZLWKRWKHUVGHYHORSORQJHUWHUPVWUDWHJLHVWDNHFDUHIXOO\
UHVHDUFKHGDQGSODQQHGDFWLRQVZKLFKILWVWKHLUFRQWH[WVSULRULWLHVDQGVW\OHVRIRSHUDWLQJ
3DUDFKLQLDQG0RWW

5DMDNXWW\HODERUDWHVRQWKHFRQFHSWRIVWDNHKROGHUEDVHGHYDOXDWLRQ6%(ZKLFKKHGHVFULEHVDVDQ
RQJRLQJGLDORJXHDQGQHJRWLDWLRQDPRQJVWDNHKROGHUV+RZHYHUKHDOVRSRLQWVWRWKHGLIILFXOWLHVRIUHVROYLQJ
FRPSHWLQJDQGFRQIOLFWLQJVWDNHKROGHUSHUFHSWLRQVHVSHFLDOO\ZKHQSDUWLFXODUVWDNHKROGHUJURXSVDUHSRZHUOHVV
YLVjYLV RWKHUV 1HYHUWKHOHVV WKH YDOXH RI WKLV DSSURDFK LV LQ OHDUQLQJ DERXW SHRSOHV
 FRQFHUQV DQG HQDEOLQJ
SULPDU\VWDNHKROGHUV²WKHEHQHILFLDULHV²WRVSHDNIRUWKHPVHOYHV
,Q WKHLU RYHUYLHZ $EERW DQG *XLMW  FRQFHQWUDWH RQ FRPPXQLW\EDVHG PRQLWRULQJ RI WKH ZLGHU
HQYLURQPHQW ZKLFK GHVFULEHV KRZ FRPPXQLWLHV WUDFN ORFDO HQYLURQPHQWDO FKDQJHV 7KH\ IRFXV RQ ORFDO


SHUFHSWLRQV RI DQG H[SHULHQFHV LQ QRWLQJ UHFRUGLQJ DQG PRQLWRULQJ FKDQJH EXW ZLWK HPSKDVLV RQ EXLOGLQJ
SDUWQHUVKLSVEHWZHHQPXOWLSOHVWDNHKROGHUV
2QHFDVHH[DPSOHLQ.HQ\DGHVFULEHVDSDUWLFLSDWRU\HYDOXDWLRQH[SHULHQFHWKDWVSHFLILFDOO\ORRNHGDWKRZ
GLIIHUHQWVWDNHKROGHUVSHUFHLYHGWKHLPSDFWVRIDKHDOWKSURMHFWDQGWKHLURZQH[SHULHQFHVDVSURMHFWSDUWLFLSDQWV
VHH%R[
2WKHUFDVHVWXGLHVVKRZKRZ30 (LVXVHGWRQHJRWLDWH EHWZHHQ GLIIHUHQW VWDNHKROGHU SHUFHSWLRQV DQG
LQWHUHVWV HVSHFLDOO\ ZLWK UHVSHFW WR LGHQWLI\LQJ HYDOXDWLRQ REMHFWLYHV DQG LQGLFDWRUV )RU H[DPSOH $&25' LQ
0DOLDWWHPSWVWRHVWDEOLVKSURMHFWREMHFWLYHVFRQVLVWHQWZLWKFULWHULDVHOHFWHGE\ERWKWKHORFDOFRPPXQLW\DQG
WKHDJHQF\ZLWK WKH DJHQF\ HYDOXDWLQJ SURMHFW DFKLHYHPHQWV DJDLQVW FRPPXQLW\GHWHUPLQHG LQGLFDWRUV 5RFKH
,Q0H[LFRVHH%R[GLIIHUHQWVWDNHKROGHUVLGHQWLI\WKHLURZQHYDOXDWLRQFULWHULDDQGLQGLFDWRUVZKLFK
DUHODWHUJURXSHGWRJHWKHUXQGHUDJHQHUDOHYDOXDWLRQIUDPHZRUN%ODXHUWDQG4XLQWDQDU

30 (IRU3XEOLF$FFRXQWDELOLW\DQG3ROLF\
7UDGLWLRQDOO\ PRQLWRULQJ DQG HYDOXDWLRQ KDYH EHHQ XVHG E\ GRQRU DQG JRYHUQPHQW DJHQFLHV WR KROG
EHQHILFLDULHV DQG SURJUDPPH UHFLSLHQWV DFFRXQWDEOH WR DJUHHG JRDOV DQG SHUIRUPDQFH WDUJHWV ,Q WKLV
FRQYHQWLRQDOFRQWH[WRI0 (VXSSRUWLQJRURXWVLGHDJHQFLHVXVXDOO\GHILQHZKDWLVPRQLWRUHGDQGHYDOXDWHG
DQGKRZPRQLWRULQJDQGHYDOXDWLRQLVWREHFRQGXFWHG0XFKRISDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQJUHZ

Box 2.4 Understanding project impacts and processes: what stakeholder perspectives revealed in Kenya
Participatory evaluation of the Makueni Community-Based Nutrition Project (MCBNP) was undertaken in order to investigate
stakeholder perceptions of the project. An external evaluation team facilitated the process, which looked at project outcomes, project
benefits, potential negative impacts on diverse groups, and how various groups felt about the project and their suggestions for
improvement. The key participants of the participatory evaluation exercise included donor representatives, government officers,
project field staff, nutrition-related community organisations and their members, as well as non-project community members.
The participatory evaluation exercise demonstrated that focusing mainly on project impacts only revealed a partial view of a
project that seemed so successful: a conventional nutrition rehabilitation centre was transformed into a community-based nutrition
project, using participatory approaches and methods to plan and implement programme activities. However, looking more closely at
different stakeholders’ perspectives showed that there were several areas for further improvement related to the process of project
implementation, namely:

• increased equity in terms of who benefited from project activities;

• maximising existing resources and capacity building for project sustainability;
• more participatory decision-making procedures;
• meeting unmet expectations of different stakeholders;
• clarifying confusion over the roles and responsibilities of different stakeholders in project management.

The participatory evaluation exercise not only helped highlight significant and problematic issues based on stakeholders’
perspectives, but it also allowed different participants to reflect, compare, and learn from each other’s experiences and perceptions.
In turn, people are better able to assess the process of project impacts and suggest what they would do differently to modify the
current process.
Source: Cornwall, Gashigi, Kabutha, and Sellers 1997; Cornwall 1997


LQUHVSRQVHWRVXFKWRS²GRZQDQGRXWVLGHDSSURDFKHVWRDVVHVVPHQWLQVLVWLQJWKDWVWDNHKROGHUVRUEHQHILFLDULHV
EHLQYROYHGLQWKHSURFHVVRI0 (
,QFUHDVLQJO\ DV D PRUH GLYHUVH EDVH RI SDUWLFLSDQWV OHDUQ WKH VNLOOV RI HYDOXDWLRQ DQG PRQLWRULQJ
FRQYHQWLRQDO DSSURDFKHV WR 0 ( DUH EHLQJ FKDOOHQJHG 30 ( LV UHJDUGHG QRW RQO\ DV D PHDQV RI KROGLQJ
SURMHFWEHQHILFLDULHVDQGSURJUDPPHUHFLSLHQWVDFFRXQWDEOHEXWDOVRDVDZD\IRUSURMHFWSDUWLFLSDQWVDQGORFDO
FLWL]HQVWKHPVHOYHVWRPRQLWRUDQGHYDOXDWHWKHSHUIRUPDQFHRIGRQRUDQGJRYHUQPHQWDOLQVWLWXWLRQV
0DUVGHQDQG2DNOH\IRULQVWDQFHSRLQWRXWWKDWWKHUHQHHGVWREHD
IXQGDPHQWDOUHDOLJQPHQWRI
WKHUHODWLRQVKLSEHWZHHQGRQRUDJHQFLHVDQGEHQHILFLDULHV
7KH\SURSRVHEXLOGLQJ SDUWQHUVKLSV EHWZHHQ WKHVH
PDMRUVWDNHKROGHUVZKLFKZRXOGDOORZUHFLSURFDOHYDOXDWLRQVWRWDNH SODFH VRWKRVH GRQRUV WKHPVHOYHV DUH
VXEMHFWWRVRPHIRUPRIDFFRXQWDELOLW\,QWKLVFRQWH[WDFFRXQWDELOLW\EHFRPHVDWZRZD\H[FKDQJHUHODWLRQVKLS
EHWZHHQWKRVHZKRSURYLGHILQDQFLDOUHVRXUFHVDQGWKRVHZKROHJLWLPLVHWKHGLVEXUVHPHQWRIWKRVHUHVRXUFHV
7KLVUHDOLJQPHQWZRXOGDOORZEHQHILFLDU\JURXSVWREHWWHUSUHVHQWDQGDUWLFXODWHWKHLU RZQ QHHGV FULWHULD DQG
H[SHFWDWLRQV WKXV SURYLGLQJ WKHP ZLWK ZLGHU RSSRUWXQLWLHV WR QHJRWLDWH WKHLU REMHFWLYHV ZLWK GRQRUV DQG
VXSSRUWLQJLQVWLWXWLRQV
,Q WKH 8QLWHG 6WDWHV FLWL]HQ PRQLWRULQJ KDV EHHQ RQH DSSURDFK WKURXJK ZKLFK ORFDO FLWL]HQV KROG
JRYHUQPHQWVDFFRXQWDEOHDQGDVVHVVWKHH[WHQWWRZKLFKSXEOLFSURJUDPPHVPHHWWKHQHHGVRIWKHFRPPXQLW\
3DUDFKLQLDQG0RWWVXUYH\DQXPEHURIWKHVHH[SHULHQFHVWKDWIRFXVRQFRPPXQLW\EDVHGPRQLWRULQJIRU
LQFUHDVHGSXEOLFDFFRXQWDELOLW\7KHLURYHUYLHZORRNVDWFRPPXQLW\PRQLWRULQJDQGHYDOXDWLRQRIJRYHUQPHQW
DQGSULYDWHVHFWRULQVWLWXWLRQVSROLFLHVSURJUDPPHVDQGSUDFWLFHV(PSKDVLVLVSODFHGRQEXLOGLQJUHVHDUFKDQG
PRQLWRULQJVNLOOVDQGFDSDFLWLHVIRUORFDOSHRSOHWRKROGKLJKHUOHYHOLQVWLWXWLRQVDQGSROLF\PDNHUVDFFRXQWDEOH
IRUHIIHFWLYHO\PHHWLQJDQGUHVSRQGLQJWRWKHLUQHHGV7KHVH H[SHULHQFHV GHPRQVWUDWH KRZ ORFDO FRPPXQLWLHV
WKHPVHOYHV JDWKHU DQG DQDO\VH GDWD LQ RUGHU WR GHWHUPLQH WKHLU RZQ SROLF\ SULRULWLHV DQG GHYHORS FRPPRQ
VWUDWHJLHVIRUDFWLRQDQGFKDQJH)RULQVWDQFHLQWKHVWKH1DWLRQDO&LWL]HQV0RQLWRULQJ3URMHFWWUDLQHG
XUEDQQHLJKERXUKRRGVLQWKH8QLWHG6WDWHVWRPRQLWRUKRZFLW\JRYHUQPHQWVVSHQWILQDQFLDOUHVRXUFHVDOORFDWHG
VSHFLILFDOO\ IRU FRPPXQLW\ GHYHORSPHQW .RW]  0RUH UHFHQWO\ WKH (=(& /HDUQLQJ ,QLWLDWLYH ZRUNHG
ZLWK JUDVVURRWV
FLWL]HQ OHDUQLQJ WHDPV
 WR PRQLWRU WKH UXUDO (PSRZHUPHQW =RQH(QWHUSULVH &RPPXQLW\
SURJUDPPHDQDWLRQDOFRPPXQLW\UHYLWDOLVDWLRQSURJUDPPHRIWKH&OLQWRQDGPLQLVWUDWLRQVHH%R[
7KHVHHIIRUWVWRVWUHQJWKHQORFDOFDSDFLWLHVIRUHQVXULQJSXEOLFDFFRXQWDELOLW\LQWXUQEHFRPHDPHDQVIRU
SHRSOH WR LQIOXHQFH SROLF\PDNLQJ GHFLVLRQV ,Q WKH 8QLWHG 6WDWHV +HDOWK &DUH )RU $OO +&)$ LV DQ
RUJDQLVDWLRQWKDWVXSSRUWVFRPPXQLW\EDVHGUHVHDUFKDQGOHDUQLQJWRVWUHQJWKHQORFDODGYRFDF\RIKHDOWKFDUH
UHIRUPDQGSROLF\VHH%R[(OVHZKHUH35,$LQWKHSDVWWZRGHFDGHVKDVEHHQZRUNLQJZLWKJUDVVURRWVDQG
QRQJRYHUQPHQWDO RUJDQLVDWLRQV LQ ,QGLD DQG 6RXWK $VLD WR SURPRWH SDUWLFLSDWRU\ HYDOXDWLRQ 0RUH UHFHQWO\
35,$ KDV DSSOLHG WKH 30 ( DSSURDFK WR DVVHVV QDWLRQDOO\ LPSOHPHQWHG SURJUDPPHV WR LQIRUP IXWXUH
JRYHUQPHQW SROLF\ $FKDU\D .XPDU 6DW\DPXUWL HW DO  2QH SDUWLFXODU FDVH UHIHUV WR D SDUWLFLSDWRU\
HYDOXDWLRQRID\HDUJRYHUQPHQWKHDOWKSURMHFWLQ,QGLDZKLFKLQYROYHGEHQHILFLDU\RUJDQLVDWLRQVLQWKH
HYDOXDWLRQSURFHVVLELG


 Bo x 2.5 B u ild ing s tronger a nd h ealth ie r communi tie s
Among health care reform circles, HCFA has built an exemplary reputation as an organisation which transforms consumers –
especially of low- and middle-income groups – into highly motivated and informed activists who become a powerful and credible
voice in tackling health care issues and policies. Through various HCFA programmes, local people gain capacities and skills in
'thinking through what health means to them, how the present health care system works and affects their lives, how it might be
changed, and what issues they should tackle collectively in order to improve the quality and responsiveness of health care' (p. 28).
Successful HCFA programmes include, among others:

• Health Care Helpline – a telephone network service which assists people to access care and mobilises those callers with the
same concerns to explore common strategies for action. The helpline also allows HCFA to stay on top of issues and investigate
new areas for research and advocacy.
• Community needs assessment/Financial analysis – which helps communities to understand their needs and assets, and
analyses the financial resources of institutions, such as hospitals and health centres. It helps people relate their priorities and
needs to the financial assets and conditions of health care institutions, and subsequently takes action on the basis of such
analyses, e.g. in terms of allocating health care resources.
• Distribution of training and other educational materials – helps consumers understand their legal rights, how to redress
grievances, how to improve or defend programmes through the legislative process, etc.
Source: Parachini and Mott 1997

,QWHUQDWLRQDOO\ FLYLO VRFLHW\EDVHG PRQLWRULQJ DQG HYDOXDWLRQ RI PXOWLODWHUDO GHYHORSPHQW EDQNV 0'%V DQG
OHQGLQJSROLFLHVKDYHSUROLIHUDWHGRYHUWKHSDVWGHFDGHZLWKFLYLOVRFLHW\EDVHGLQVWLWXWLRQVJHQHUDWLQJUHOLDEOH
ILHOGEDVHG LQIRUPDWLRQ WR DVVHVV WKH HIIHFWLYHQHVV RI SURMHFW OHQGLQJ )R[  $ UHFHQW LQLWLDWLYH LV RQH
XQGHUWDNHQ E\ WKH 1*2 :RUNLQJ*URXS RI WKH :RUOG %DQNWR PRQLWRU WKH LPSOHPHQWDWLRQ RI :RUOG %DQN
SDUWLFLSDWLRQSROLFLHV.H\SURMHFWVLQ$VLD/DWLQ$PHULFDDQG$IULFDKDYH EHHQ FKRVHQIRU PRQLWRULQJ ZLWK
ORFDO1*2VWUDLQHGWRVKRZKRZSDUWLFLSDWLRQDFWXDOO\RFFXUVLQSURJUDPPHLPSOHPHQWDWLRQDWJURXQGOHYHO
7KHVH ILQGLQJV ZLOO WKHQ EH SUHVHQWHG EDFN WR WKH 1*2 ZRUNLQJ JURXS DQG WR :RUOG %DQN OHDGHUV IRU
GLVFXVVLRQ,'5DQG35,$
$OVRDVSDUWRIDJURZLQJWUHQGZRUOGZLGHWRPRYHWRZDUGVGHFHQWUDOLVDWLRQDQGORFDOJRYHUQDQFHQDWLRQ
VWDWHJRYHUQPHQWVDQGODUJHSXEOLFLQVWLWXWLRQVDUHJLYLQJJUHDWHULPSRUWDQFHWRORFDOSDUWLFLSDWLRQ)RUH[DPSOH
LQ%ROLYLDVWDWHOHJDOUHIRUPVQRZDOORZORFDOFRPPXQLWLHVHVSHFLDOO\LQ UXUDO UHJLRQV WR SOD\ D ODUJHU UROH LQ
GHILQLQJORFDOGHYHORSPHQWDQGWKHDOORFDWLRQRIVWDWHUHVRXUFHV$OFRFHU/L]iUUDJD'HOJDGLOORHWDO²
/DZVRQSRSXODUSDUWLFLSDWLRQQRZPDNHLWFRPSXOVRU\IRUPXQLFLSDOJRYHUQPHQWVWRIRUPXODWHDQGFDUU\RXW
WKHLU RZQ GHYHORSPHQW SODQV 6LJQLILFDQWO\ OHJDO UHIRUPV IXUWKHU VWLSXODWH WKDW ORFDOO\ HOHFWHG UHSUHVHQWDWLYHV
NQRZQDV&RPLWpVGH9LJLODQFLD
YLJLODQFHFRPPLWWHHV
LPSOHPHQWDQGPRQLWRUGHYHORSPHQWSODQVJHQHUDWHGDW
WKHPXQLFLSDOOHYHO$OFRFHU/L]iUUDJD'HOJDGLOORHWDO
$VHYLGHQFHGLQWKHVXUYH\RIOLWHUDWXUH30 (LVDSSOLHGLQDZLGHYDULHW\RIFRQWH[WVDQGVHWWLQJV DQG
XVHGIRUYDULRXVSXUSRVHV'HWHUPLQLQJKRZ30 (LVXVHGDQGIRUZKDWSXUSRVHVZLOOGHSHQGHVVHQWLDOO\RQ
WKHREMHFWLYHVRIWKHPRQLWRULQJDQGHYDOXDWLRQSURFHVVLWVHOI8OWLPDWHO\WKHFKRLFHRIREMHFWLYHFRPHVEDFNWR


WKHTXHVWLRQRIZKRLVSDUWLFLSDWLQJDQGZKRLVQHJRWLDWLQJLQWKHSURFHVV7KRVHTXHVWLRQVWDNHXVWRWKHFRUH
YDOXHVDQGSULQFLSOHVRI30 (LWVHOI

  35,1&,3/(6 $1' 3+,/2623+< 2) 3$57,&,3$725< 021,725,1* $1'

(9$/8$7,21
2XU OLWHUDWXUH UHYLHZ XQFRYHUHG PDQ\ GLIIHUHQW WHUPV IRU SDUWLFLSDWRU\ DSSURDFKHV WR PRQLWRULQJ DQG
HYDOXDWLRQDQGDZHDOWKRIH[SHULHQFHVGHYHORSHGLQSDUWLFXODUFDVHV$OWKRXJKWKHUHLVDZLGHYDULDWLRQLQWKH
SUDFWLFHRI30 (SHUKDSVPRUHFRPPRQDOLW\LVIRXQGLQLWVSULQFLSOHV7KLVVHFWLRQEHJLQVE\UHYLHZLQJNH\
GLIIHUHQFHV EHWZHHQ SDUWLFLSDWRU\ HYDOXDWLRQ DQG RWKHU PRUH WUDGLWLRQDO DSSURDFKHV 7KHQ ZH WXUQ WR D
GLVFXVVLRQ RI IRXU EURDG
SULQFLSOHV
 RI SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ LGHQWLILHG LQ WKH VXUYH\ RI
OLWHUDWXUHSDUWLFLSDWLRQOHDUQLQJQHJRWLDWLRQDQGIOH[LELOLW\

(PHUJLQJ$SSURDFKHVWR(YDOXDWLRQ
,QWKHODVWGHFDGHSDUWLFXODUO\LQWHUHVWKDVEHHQJURZLQJLQSDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQWRFRXQWHU
PRUHWUDGLWLRQDO
WRS²GRZQ
DSSURDFKHVWRHYDOXDWLRQ:KLOHWKHUHDUHPDQ\YDULDWLRQVZLWKLQZKDWZHZLOOFDOO

FRQYHQWLRQDO
DSSURDFKHVWRPRQLWRULQJDQGHYDOXDWLRQWKH\FDQEHEURDGO\FKDUDFWHULVHGDVIROORZV

• IRFXVHGRQPHDVXUHPHQW
• DQRULHQWDWHGWRWKHQHHGVRISURJUDPPHIXQGHUVDQGSROLF\PDNHUVUDWKHUWKDQSDUWLFLSDQWVRUORFDOSHRSOH
• VWULYLQJIRUREMHFWLYLW\DQGGLVWDQFHEHWZHHQHYDOXDWRUDQGSDUWLFLSDQWV
• FRQGXFWHGIRUWKHSXUSRVHRIPDNLQJMXGJHPHQWVUDWKHUWKDQHPSRZHUPHQW

$GYRFDWHV RI SDUWLFLSDWRU\ DSSURDFKHV SRLQW RXW WKDW FRQYHQWLRQDO DSSURDFKHV GUDZ IURP WKH JHQHUDOO\
:HVWHUQWUDGLWLRQRIVFLHQWLILFLQYHVWLJDWLRQ0F$UWKXU7DQGRQ7KH\DWWHPSWWRSURGXFH
LQIRUPDWLRQ WKDW LV QHFHVVDULO\
REMHFWLYH

YDOXHIUHH
 DQG
TXDQWLILDEOH
 ² WHUPV WKDW DUH DOO EHLQJ FKDOOHQJHG
+LVWRULFDOO\ WKHVH WUDGLWLRQDO DSSURDFKHV HPSKDVLVH DFKLHYLQJ SURJUDPPH HIIHFWLYHQHVV DQG SUDFWLFDO XWLOLW\
*UHHQH  7KH\ UHPDLQ H[WHUQDOO\ RULHQWHG JHDUHG WRZDUGV HQKDQFLQJ FRVW HIILFLHQF\ DQG DFFRXQWDELOLW\
DQG XVXDOO\ UHTXLUH TXDQWLWDWLYH PHWKRGV IRU RYHUDOO SURJUDPPH DVVHVVPHQW LELG 6LQFH RXWVLGHUV DUH XVXDOO\
FRQWUDFWHGWRFRQGXFWWKHHYDOXDWLRQIRUWKHVDNHRIHQKDQFLQJ
REMHFWLYLW\
SDUWLFLSDQWVZKRPD\EHDIIHFWHGE\
WKHILQGLQJVRIDQHYDOXDWLRQKDYHOLWWOHRUQRLQSXWLQWKHSURFHVVHLWKHULQGHWHUPLQLQJTXHVWLRQVDVNHGDQG
W\SHVRILQIRUPDWLRQREWDLQHGRUUHIOHFWLQJRQDQGXVLQJHYDOXDWLRQILQGLQJV5XELQ
,Q WKHLU ERRN )RXUWK *HQHUDWLRQ (YDOXDWLRQ *XED DQG /LQFROQ  DUJXH WKDW SDVW WUDGLWLRQV RU

JHQHUDWLRQV
 RI HYDOXDWLRQ W\SLFDOO\ DGRSWHG DVFLHQWLILF WHFKQLFDO DQG PRUHPDQDJHULDO DSSURDFK LQRUGHU WR
REWDLQ
REMHFWLYH
 GDWD IRU HYDOXDWLRQ SXUSRVHV 7KH\ FKDUDFWHULVH WKH ILUVW WKUHH HYDOXDWLRQ WUDGLWLRQV DV
PHDVXUHPHQWRULHQWHG GHVFULSWLRQRULHQWHG DQG MXGJHPHQWRULHQWHG VHH %R[  )RXUWK JHQHUDWLRQ
HYDOXDWLRQLQWKHLUDUJXPHQWLVFKDUDFWHULVHGE\QHJRWLDWLRQEHWZHHQYDULRXVVWDNHKROGHUVSDUWLFLSDWLRQLQHYHU\
VWDJH RI WKH HYDOXDWLRQ SURFHVV DQG D IRFXV RQ DFWLRQ ,Q WKH LQWHUQDWLRQDO GHYHORSPHQW FRPPXQLW\ DV
HOVHZKHUHWKHUHKDVEHHQJURZLQJGLVVDWLVIDFWLRQZLWKFRQYHQWLRQDOPRQLWRULQJDQGHYDOXDWLRQDSSURDFKHV7KH
DUJXPHQWV DJDLQVW EOXHSULQW WRS²GRZQ DSSURDFKHV WR GHYHORSPHQW HVSHFLDOO\ ZLWK UHVSHFW WR 0 ( DUH


GLVFXVVHGZLGHO\LQWKHOLWHUDWXUHVHHIRUH[DPSOH)HXHUVWHLQ*UHHQH*XEDDQG/LQFROQ35,$
  5XELQ  83:$5'  7KH IROORZLQJ VXPPDULVHV WKH PDMRU FULWLFLVPV RI FRQYHQWLRQDO
DSSURDFKHVWR0 (PDGHLQWKHOLWHUDWXUH

• 7KH\KDYHSURYHQFRVWO\DQGLQHIIHFWLYHLQWHUPVRIPHDVXULQJDQGDVVHVVLQJSURMHFWDFKLHYHPHQWV
• 7KH\KDYHIDLOHGWRLQYROYHDFWLYHO\SURMHFWEHQHILFLDULHVDQGRWKHUVZKRPD\EHGLUHFWO\DIIHFWHGE\0 (
• 3URMHFWHYDOXDWLRQKDVEHFRPHDQLQFUHDVLQJO\VSHFLDOLVHGILHOGDQGDFWLYLW\FRQGXFWHGDQGFRQWUROOHGPRVWO\
E\RXWVLGHUVDQGUHPRYHGIURPWKHRQJRLQJSODQQLQJDQGLPSOHPHQWDWLRQRIGHYHORSPHQWLQLWLDWLYHV
• 7KH\ VHUYH SULPDULO\ DV D WRRO WR FRQWURO DQG PDQDJH SURJUDPPHV DQG UHVRXUFHV DOLHQDWLQJ LQWHQGHG
EHQHILFLDULHVDQGRWKHUV LQYROYHG LQ SURJUDPPH SODQQLQJ DQG LPSOHPHQWDWLRQ IURP WDNLQJ SDUW LQ SURMHFW
DSSUDLVDO
• (PSKDVLV RQ TXDQWLWDWLYH PHDVXUHV WHQGV WR LJQRUH TXDOLWDWLYH LQIRUPDWLRQ ZKLFK KHOSV SURYLGH D IXOOHU
XQGHUVWDQGLQJRISURMHFWRXWFRPHVSURFHVVHVDQGFKDQJHV

Box 3.1 ’F ou r gen e ratio n s’ of p ro ject ev alu ation (Gub a and Lin coln)
First generation evaluation emerged in the 1900s and may be characterised as measurement-oriented. It is associated with the
tradition of educational research and scientific management in business and industry. Tests were commonly used to measure the
progress of students in schools or to determine the most productive methods to make working environments more efficient and
effective. The role of the evaluator was generally technical – to provide and apply tools or instruments for measurement.
Second generation evaluation focused more on description and led to programme evaluations. This tradition of evaluations
emphasised the achievement of objectives and the analysis of programme strengths and weaknesses, which were utilised to guide
refinements and revisions. The role of the evaluator became essentially that of describer, although earlier technical functions were
also retained.
Third generation evaluation included judgement as an integral part of evaluation. Judgement required that the programme
objectives themselves be taken as problematic; hence, goals and not simple performance were subject to evaluation. The
development of standards against which the judgement can be made assumed prominence. Consequently, evaluators also
assumed the role as judges and helped 'clients' (decision makers for whom the evaluations were ultimately geared) determine
standards for judgement.
Fourth generation evaluation refers to the most recent evolution in evaluation practice. Its key emphasis is on evaluation as a
process of negotiation, incorporating various stakeholders more centrally into the evaluation process. Fourth generation evaluation
takes into account stakeholders' consensual and competing claims, concerns, and issues. It recognises that peoples' diverse
perspectives and interests are shaped in a major way by their particular value systems, which in turn are influenced by their specific
physical, psychological, social and cultural contexts. Through negotiation, fourth generation evaluation helps identify a course of
action for stakeholders. The evaluator plays a role primarily as a facilitator or 'orchestrator' in the negotiation process with
stakeholders, who participate in the design, implementation and interpretation of the evaluation as full partners.
Adapted from Campos and Coupal (1996) and Guba and Lincoln (1989)


,QUHVSRQVHWRWKHSUREOHPVWKDWKDYHEHFRPHDVVRFLDWHGZLWKFRQYHQWLRQDODSSURDFKHVWR0 (QHZZD\VRI
WKLQNLQJDERXWDQGSUDFWLVLQJPRQLWRULQJDQGHYDOXDWLRQKDYHJUDGXDOO\HYROYHG7KHVHLQQRYDWLYHDSSURDFKHV
DLPWRPDNHPRQLWRULQJDQGHYDOXDWLRQPRUHUHVSRQVLYHDQGDSSURSULDWHWRSHRSOH
VQHHGVDQGUHDOOLIHFRQWH[WV
7KH\ UHIOHFW JURZLQJ H[SHULHQFH ZLWK JUDVVURRWVEDVHG H[SHULHQFHV DQG
ERWWRP XS
 VWUDWHJLHV WKDW HPSOR\
SDUWLFLSDWRU\PHWKRGRORJLHV7KHVHSDUWLFLSDWRU\ DSSURDFKHV WKHUHIRUH UHFRJQLVH WKH LPSRUWDQFH RI LQFOXGLQJ
EHQHILFLDULHVDQGRWKHUSURMHFWSDUWLFLSDQWVLQWKHPRQLWRULQJDQGHYDOXDWLRQSURFHVVLWVHOI(PSKDVLVLVVKLIWHG

DZD\IURPH[WHUQDOO\FRQWUROOHGGDWDVHHNLQJSURJUDPPHVWRZDUGVWKHUHFRJQLWLRQRIORFDOO\UHOHYDQWSURFHVVHV
IRUJDWKHULQJDQDO\VLQJDQGXVLQJLQIRUPDWLRQ
$EERWDQG*XLMW7KLVIRXUWKJHQHUDWLRQRIHYDOXDWLRQ
KLJKOLJKWV WKH LPSRUWDQFH RI FRQWH[W ² VRFLDO SROLWLFDO DQG YDOXH V\VWHPV ² LQ WKH HYDOXDWLRQ SURFHVV ,Q
VXPPDU\ WKH PDLQ DUJXPHQWV JHQHUDOO\ PDGH IRU SDUWLFLSDWRU\ DSSURDFKHV WR PRQLWRULQJ DQG HYDOXDWLRQ
GUDZLQJRQ$EERWDQG*XLMW&21&(51)HXHUVWHLQ5XJK6RPPHUDUH

• (QKDQFHG SDUWLFLSDWLRQ HVSHFLDOO\ RI EHQHILFLDULHV LQ 0 ( KHOSV LPSURYH XQGHUVWDQGLQJ RI WKH
GHYHORSPHQWSURFHVVLWVHOI
• ,QFUHDVHGDXWKHQWLFLW\RI0 (ILQGLQJVWKDWDUHORFDOO\UHOHYDQW
• ,PSURYHPHQWRIWKHVXVWDLQDELOLW\RISURMHFWDFWLYLWLHVE\LGHQWLI\LQJVWUHQJWKVDQGZHDNQHVVHVIRUEHWWHU
SURMHFWPDQDJHPHQWDQGGHFLVLRQPDNLQJ
• ,QFUHDVLQJORFDOOHYHOFDSDFLW\LQ0 (ZKLFKLQWXUQFRQWULEXWHVWRVHOIUHOLDQFHLQRYHUDOOSURMHFWLPSOHPHQWDWLRQ
• 6KDULQJRIH[SHULHQFHWKURXJKV\VWHPDWLFGRFXPHQWDWLRQDQGDQDO\VLVEDVHGRQEURDGEDVHGSDUWLFLSDWLRQ
• 6WUHQJWKHQHGDFFRXQWDELOLW\WRGRQRUV
• 0RUHHIILFLHQWDOORFDWLRQRIUHVRXUFHV

1DUD\DQ3DUNHU  VXPPDULVHV KRZ VKH VHHV WKH GLIIHUHQFHV EHWZHHQ FRQYHQWLRQDO DQG SDUWLFLSDWRU\
DSSURDFKHVWRHYDOXDWLRQVHH%R[

Box 3.2 D iffe rence s bet we en c onvent iona l an d pa rt icip ato ry eva luat ion

Conventional Participatory
Who External experts Community members, project staff, facilitator
What Predetermined indicators of success, People identify their own indicators of success, which may
principally cost and production outputs include production outputs
How Focus on ’scientific objectivity’; distancing of Self-evaluation; simple methods adapted to local culture;
evaluators from other participants; uniform, open, immediate sharing of results through local involvement
complex procedures; delayed, limited in evaluation processes
access to results
When Usually upon completion of project/programme; More frequent, small-scale evaluations
sometimes also mid-term
Why Accountability, usually summative, To empower local people to
to determine if funding continues initiate, control and take corrective action
Source: Narayan-Parker 1993: 12


 ,QSUDFWLFHDVWKHFDVHVUHYLHZHGKHUHVKRZWKHGLIIHUHQFHVEHWZHHQWKHVHWZRDSSURDFKHVWRHYDOXDWLRQ
DUHQRWDVFOHDUDVLQWKHRU\3DUWLFLSDWRU\DSSURDFKHVRIWHQHQJDJHRXWVLGHH[SHUWVEXWLQGLIIHUHQWUROHVDQG
UHODWLRQVKLSV 6RPHWLPHV WKH\ XVH SUHGHWHUPLQHG LQGLFDWRUV RI VXFFHVV 4XDOLWDWLYH UHVHDUFK PHWKRGV PD\ EH
XVHG LQ ERWK DSSURDFKHV 7KH WLPLQJ RI SDUWLFLSDWRU\ HYDOXDWLRQ LV QRW DOZD\V RQJRLQJ $QG WKH TXHVWLRQ RI
ZKHWKHU SDUWLFLSDWRU\ DSSURDFKHV FDQ PHHW WKH SXUSRVHV RI DFFRXQWDELOLW\ LQFOXGLQJ DFFRXQWDELOLW\ WR ZKRP
DQGIRUZKDWLVDQLVVXHZHZLOOUHWXUQWRLQWKHILQDOVHFWLRQRIWKLVUHSRUW

7KH3ULQFLSOHRI3DUWLFLSDWLRQ
3HUKDSV ZKDW PRVW GLVWLQJXLVKHV 30 ( IURP PRUH FRQYHQWLRQDO
WUDGLWLRQDO
 DSSURDFKHV WR PRQLWRULQJ DQG
HYDOXDWLRQLVLWVHPSKDVLVRQSDUWLFLSDWLRQ%DVHGRQWKHUHYLHZRIOLWHUDWXUHRQHPDMRU
FRUH
IHDWXUHRI30 (
LGHQWLILHG LV LWV UHFRJQLWLRQ RI ZKR DFWXDOO\ SDUWLFLSDWHV 30 ( DFNQRZOHGJHV WKDW WKHUH DUH PXOWLSOH
VWDNHKROGHUVZKRDUHRUVKRXOGEHSDUWLFLSDQWVLQPRQLWRULQJDQGHYDOXDWLRQ6WDNHKROGHUVDUHWKRVHZKRGLUHFWO\
RULQGLUHFWO\EHFRPHLQYROYHGLQGHFLGLQJZKDWDSURMHFWRUSURJUDPPHVKRXOGDFKLHYHDQGKRZLWVKRXOGEH
DFKLHYHG 7KH\ PD\ LQFOXGH EHQHILFLDULHV SURMHFW RU SURJUDPPH VWDII DQG PDQDJHPHQW DW ORFDO UHJLRQDO
QDWLRQDORULQWHUQDWLRQDOOHYHOVUHVHDUFKHUVJRYHUQPHQWDJHQFLHVDQG GRQRUV )HXHUVWHLQ  GHVFULEHV WKH
HVVHQWLDOIHDWXUHRI30 (DVD
UHDOSDUWQHUVKLSLQGHYHORSPHQW
ZKHUHE\SHRSOHDUHLQYROYHGLQGHFLGLQJZKHQ
DQGKRZWRPRQLWRUDQGHYDOXDWHDQDO\VHFRPPXQLFDWHDQGXVHLQIRUPDWLRQ
VHHDOVR*RKO6FRWW9LOOLHUV
D 7KH 6RXWK²6RXWK 6KDULQJ )RUXP KRVWHG E\ 3$0)25. LQ $XJXVW  HVWDEOLVKHG D FRPPRQ
GHILQLWLRQ RI 30 ( SUHPLVHG RQ WKH LGHD WKDW VWDNHKROGHU SDUWLFLSDWLRQ LV LQWHUDFWLYH DQG FROODERUDWLYH
3$0)25.
'HVSLWH WKH FRPPRQ HPSKDVLV RQ FROODERUDWLRQ DQG FRQVXOWDWLRQ PDQ\ SRLQW RXW WKDW GLIIHUHQW SHRSOH
JLYHYHU\GLIIHUHQWPHDQLQJVWRWKHFRQFHSWRI
SDUWLFLSDWLRQ
,QSUDFWLFHWKHUHLVFRQVLGHUDEOHYDULDWLRQLQWKH
OHYHODQGGHJUHHRISDUWLFLSDWLRQEHLQJVRXJKWDPRQJWKHYDULRXVVWDNHKROGHUV6HH81'37DEOH7KH
SXUSRVH PHWKRGV DQG WKH UROH RI WKH
HYDOXDWRU
 DQG WKH LPSDFW RI WKH HYDOXDWLRQ ZLOO YDU\ FRQVLGHUDEO\
GHSHQGLQJRQWKHW\SHRIHYDOXDWLRQDQGWKHOHYHORISDUWLFLSDWLRQ
7KHUHDUHWZRPDLQZD\VWRFKDUDFWHULVHSDUWLFLSDWLRQLQPRQLWRULQJDQGHYDOXDWLRQE\ZKRPLWLVLQLWLDWHG
DQG FRQGXFWHG DQG ZKRVH SHUVSHFWLYHV DUH SDUWLFXODUO\ HPSKDVLVHG 7KH ILUVW GLVWLQJXLVKHV EHWZHHQ
PRQLWRULQJDQGHYDOXDWLRQWKDWLVH[WHUQDOO\OHGLQWHUQDOO\OHGRUMRLQWO\OHG7KHVHFRQGGLVWLQJXLVKHVEHWZHHQ
ZKLFKVWDNHKROGHUVDUHHPSKDVLVHG²DOOPDMRUVWDNHKROGHUVEHQHILFLDULHVRUPDUJLQDOLVHGJURXSV
6HYHUDONH\UHIHUHQFHVLGHQWLI\GLIIHUHQW
PRGHV
RIFRQGXFWLQJ30 (LQWHUPVRIZKRWKHPDMRUDFWRUVDUH
LQLQLWLDWLQJDQGIDFLOLWDWLQJWKH0 (SURFHVVVHH&21&(51)HXHUVWHLQ *RVOLQJ DQG (GZDUGV
5XJK6FRWW9LOOLHUVD83:$5':DGVZRUWK7KHUHLVXVXDOO\DFRQWLQXXPDFURVV
WKHVHGLIIHUHQW
PRGHV
RI30 (GHSHQGLQJRQWKHGHJUHHDQGQDWXUHRISDUWLFLSDWLRQE\YDULRXVVWDNHKROGHU
JURXSVLQ0 (&DPSLODQ7KHVH
PRGHV
PD\EHFDWHJRULVHGDVIROORZV


 7DEO H / H YHOV  RI HQ G²XV HUSDU WLFLSD WLR QLQH YDO XDW L RQ

'LPHQVLRQV
/RZ 0HGLXP +LJK
RI
HYDOXDWLRQ
/HYHOVRI
SDUWLFLSDWLRQ

(YDOXDWLRQ &RPPLVVLRQHGRUREOLJDWRU\ ([WHUQDOHYDOXDWRULQYLWHVHQG (YDOXDWLRQLQZKLFKHQGXVHUV

LQLWLDWRU HYDOXDWLRQW\SLFDOO\SDUWRI XVHUVWRDVVLVWLQRQHRUPRUH FROODERUDWHZLWKH[WHUQDOIDFLOLWDWRU
SURJUDPPHGHYHORSPHQW0HHWV
LQVWLWXWLRQDOQHHGV(YDOXDWLRQ
GRQHWRRQRUDERXWSHRSOH
3XUSRVH -XVWLI\RUFRQWLQXHIXQGLQJ
(QVXUHDFFRXQWDELOLW\
VXVWDLQDELOLW\E\OLQNLQJHQGXVHUV
/HYHOVRIIXQGLQJRUVXVWDLQHG
VXSSRUW
4XHVWLRQ² $JHQF\KHDGVDGPLQLVWUDWRUV
HYDOXDWLRQWDVNV
*DLQLQVLJKWVLQWRGHYHORSPHQW
DFWLYLW\IURPHQGXVHUV
SHUVSHFWLYH
6KLIWIRFXVIURPLQVWLWXWLRQDO
FRQFHUQVWRHQGXVHUQHHGVDQG
LQWHUHVWV
(QGXVHUVZLWKH[WHUQDOHYDOXDWRU
RUDPRQJWKHPVHOYHVWRDVVHVV
UHYLHZDQGFULWLFDOO\UHIOHFWRQ
VWUDWHJLHVIRUPXODWHGIRUWKHP
3URPRWHVHOIVXIILFLHQF\DQG
WRHYDOXDWLRQSODQQLQJF\FOH
'HYHORSUHOHYDQWHIIHFWLYH
SURJUDPPHGHFLVLRQPDNLQJEDVHG
RQHQGXVHUYLHZVRSLQLRQVDQG
UHFRPPHQGDWLRQV,QFUHDVH
RZQHUVKLSLQDQGUHVSRQVLELOLW\IRU
VXFFHVVIDLOXUHRIGHYHORSPHQW
LQWHUYHQWLRQV
(QGXVHUVH[WHUQDOIDFLOLWDWRU
PDNHUV" RXWVLGHFOLHQWHOHSHUVRQVDQG DWYDULRXVVWDJHVRIHYDOXDWLRQ SHUVRQVPRVWDIIHFWHGE\
GLVWDQFHVIURPHYDOXDWLRQVLWH JHQHUDOO\GHWHUPLQHGE\WKH GHYHORSPHQWLQWHUYHQWLRQ
HYDOXDWRU

0HWKRGV (VWDEOLVKHGUHVHDUFKGHVLJQV 4XDOLWDWLYHPHWKRGVIDYRXUHGEXW 5HOLHVRQKLJKO\LQWHUDFWLYH

VWDWLVWLFDODQDO\VHVUHOLDQFHRQ DOVRLQFOXGHTXDQWLWDWLYHPHWKRGV TXDOLWDWLYHPHWKRGVEXWGRHVQRW
YDULRXVTXDQWLWDWLYHPHWKRGV 9DOXHVDSURFHVVIRFXVVHGRQ GLVUHJDUGTXDQWLWDWLYHWRROV
7KH
3URGXFWILQGLQJVRULHQWDWHG RSHQHQGHGLQTXLULHV8VHV SURFHVVLVWKHSURGXFW

PDWKHPDWLFDOLQQDWXUH PHWKRGVWKDWJLYHYRLFHWR ,QYHQWLYHQHVVDQGFUHDWLYLW\

'RPLQDWHGE\PDWKZKL]NLGV YRLFHOHVVQHVV HQFRXUDJHGWRDGDSWWKHPHWKRGV
WRWKHFRQWH[WEHLQJHYDOXDWHG

(YDOXDWRU
V (YDOXDWRUWDNHVOHDGLQGHVLJQLQJ (YDOXDWRUZRUNVFROODERUDWLYHO\DW (YDOXDWRUEHFRPHVPRUHRID
YHUVXV HYDOXDWLRQ)RUPXODWHV YDULRXVVWDJHVZLWKHQGXVHUV,V IDFLOLWDWRU)DFLOLWDWRUDFWVDV
)DFLOLWDWRU
V TXHVWLRQVVXUYH\IRUPVZLWKQR SDUWQHULQHYDOXDWLRQDQGLPSDUWV FDWDO\VWFRQILGDQWHDQG
5ROH LQSXWIURPWKRVHHYDOXDWHG HYDOXDWLRQVNLOOV6KDUHVOHDGZLWK FROODERUDWRU7DNHVOHDGIURPHQG
6WHHUVE\VHWWLQJGHVLJQ HQGXVHUV XVHUV+DVIHZLIDQ\SUH
$VVXPHVREMHFWLYHQHXWUDO GHWHUPLQHGTXHVWLRQV
GLVWDQWVWDQFH

,PSDFW 5HSRUWVSXEOLFDWLRQVFLUFXODWHG 6KDUHGGDWDJDWKHULQJEXWOLPLWHG (QGXVHUPRUHFDSDEOHRI

2XWFRPH LQKRXVH)LQGLQJVUDUHO\
FLUFXODWHGDPRQJHQGXVHUV
)LQGLQJVORRSLQWRSODQQLQJ
VWDJHZLWKOLWWOHLQSXWIURPHQG
XVHUV
SDUWLFLSDWLRQLQGDWDDQDO\VLV(QG
XVHUYLHZVORRSLQWRSODQQLQJ
VWDJH,QFUHDVHGXQGHUVWDQGLQJRI
HQGXVHUH[SHULHQFHV
PHDQLQJIXOGHFLVLRQPDNLQJEDVHG
RQHIIHFWLYHLQYROYHPHQWLQ
HYDOXDWLRQ)LQGLQJVEHFRPH
SURSHUW\RIHQGXVHUVRU
FRPPXQLW\3DUWLFLSDWLRQLQ
DQDO\VLVLVFULWLFDO

6RXUFH81'3:KR$UHWKH4XHVWLRQ0DNHUV"$3DUWLFLSDWRU\(YDOXDWLRQ2IILFHRI(YDOXDWLRQDQG6WUDWHJLF3ODQQLQJS


([WHUQDOO\OHG30 (7KHVHHIIRUWVDUHJHQHUDOO\RUJDQLVHGDQGLQLWLDWHGH[WHUQDOO\DQGFRQGXFWHGPDLQO\E\
LQGLYLGXDOVRUJURXSVFRQVLGHUHGDVKDYLQJQRGLUHFWLQYROYHPHQWRUQRGLUHFWSHUVRQDORULQVWLWXWLRQDOLQWHUHVWLQ
WKH RXWFRPH RI WKH SURMHFW RU LQLWLDWLYH $V RXWVLGHUV WKH\ DUH FRPPLVVLRQHG XVXDOO\ E\ SURMHFW GRQRUV DQG
VXSSRUWLQJ DJHQFLHV WR SUHVHQW D PRUH
REMHFWLYH
 DQG
XQELDVHG
 SRLQW RI YLHZ DQG DFKLHYH D PRUH EDODQFHG
IDFLOLWDWLRQ RI RSLQLRQVEHWZHHQ DOO WKH VWDNHKROGHUV &DPSLODQ  5XELQ  6FRWW 9LOOLHUV ,Q WKLV
FRQWH[WWKHRXWVLGHHYDOXDWRUVHUYHVSULPDULO\DVDIDFLOLWDWRUWRSURYLGHLQVLJKWVIURPKLVKHUH[SHULHQFHLQWR
WKH30 (SURFHVVDQGWRKHOSVWDNHKROGHUVGHWHUPLQHDQGFRQGXFWWKHLURZQ0 (DFWLYLWLHV5XJK
,Q RQH H[DPSOH DQ H[WHUQDO HYDOXDWLRQ WHDP ZDV FRPPLVVLRQHG E\ WKH GRQRU DJHQF\ WR FRQGXFW D
SDUWLFLSDWRU\ LPSDFW DVVHVVPHQW RI SRYHUW\ DOOHYLDWLRQ SURJUDPPHV LQ *KDQD ZKLFK SURYLGHG LQIRUPDWLRQ
UHJDUGLQJ SURMHFW LPSDFWV RQ ORFDO OLYHOLKRRGV DQG FDSDFLW\ EXLOGLQJ 7KURXJK H[WHQVLYH FRQVXOWDWLRQV ZLWK
YLOODJH UHSUHVHQWDWLYHV DQG SURMHFW VWDNHKROGHUV WKH WHDP GHVLJQHG D 9LOODJH 'HYHORSPHQW &DSDFLW\ ,QGH[ LQ
ZKLFK WR UDQN SURJUHVV DQG VWDWXV RI GLIIHUHQW YLOODJHV VHH %R[  ,Q DQRWKHU FDVH &RUQZDOO HW DO 
GHVFULEH WKH H[SHULHQFH RI DQ H[WHUQDOO\ FRPPLVVLRQHG HYDOXDWLRQ WHDP ZKLFK DWWHPSWHG WR LGHQWLI\ WKH
SHUFHSWLRQVRIWKHYDULRXVSURMHFWVWDNHKROGHUVUHJDUGLQJWKHLPSDFWDQGSURFHVVRIDKHDOWKUHODWHGSURMHFWVHH
%R[

,QWHUQDOO\OHG30 (7KHVHHIIRUWVDUHFDUULHGRXWPDLQO\E\WKRVHGLUHFWO\LQYROYHGLQSURMHFWRUSURJUDPPH
SODQQLQJ DQG LPSOHPHQWDWLRQ 7KH\ LQFOXGH ORFDO SHRSOH DQG ILHOGEDVHG VWDII ZKR DUH FRQVLGHUHG
LQVLGHUV

&DPSLODQ   VHH DOVR 'DYLV&DVH  6RPH ZRXOG FRQVLGHU WKLV WR EH DQ LGHDO IRUP RI 30 (
ZKHUHE\FRPPXQLW\PHPEHUVWKHPVHOYHVLQLWLDWH0 (DQGSOD\DPDMRUUROHLQLWVLPSOHPHQWDWLRQ,QWHUQDOO\
OHG0 (DUHSHUFHLYHGDVFRQWULEXWLQJWRORFDOFDSDFLW\EXLOGLQJDQGRUJDQLVDWLRQDOVWUHQJWKHQLQJ5XELQ
5XJK$VORFDOSHRSOHJDLQJUHDWHUFRQWURORYHUWKH0 (SURFHVVLQWHUQDOO\LQLWLDWHG0 (DUHDOVR
FRQVLGHUHGPRUHOLNHO\WREHVXVWDLQHGDQGEHFRPHDQLQWHJUDOSDUWRIFRPPXQLW\DFWLYLWLHVLELG
6HYHUDO RI WKH FDVHV UHYLHZHG LQYROYH LQWHUQDOO\GULYHQ 30 ( LQFOXGLQJ $&25'
V DQG  WKH $JD .KDQ
5XUDO6XSSRUW3URJUDPPH
V$.563H[SHULHQFHVLQZKLFKILHOGEDVHGVWDIIDQGIDUPHUV WKHPVHOYHV GHVLJQHG
DQGFDUULHGRXWWKH0 (SURFHVVVHH%R[DQGFRPPXQLW\EDVHGLQLWLDWLYHVLQWKH8QLWHG6WDWHVVXFKDV
WKRVHXQGHUWDNHQE\WKH+&)$DQGWKH(=(&/HDUQLQJ,QLWLDWLYHVHH%R[HVDQG2WKHUH[SHULHQFHV
RI FRPPXQLW\EDVHG PRQLWRULQJ DUH GRFXPHQWHG E\ 5HGG %DUQD 8JDQGD ZKHUHE\ ORFDO YLOODJH FRXQFLOV DUH
GHYHORSLQJ LQGLFDWRUV WR PRQLWRU FKDQJHV RFFXUULQJ LQ WKHLU FRPPXQLWLHV LH WKH SUHVHQFH RI SLW ODWULQHV
LPPXQLVHGFKLOGUHQGHIRUHVWDWLRQDQGUHIRUHVWDWLRQSRYHUW\DOOHYLDWLRQLQLWLDWLYHVDQGWKHLULPSDFWVDGRSWLRQRI
IDPLO\SODQQLQJHWF*XLMW

-RLQW 30 (  7KHVH FRPELQH DSSURDFKHV RI LQWHUQDO DQG H[WHUQDO 0 ( DQG DLP WR DVVHVV SURMHFW RU
FRPPXQLW\EDVHG HIIRUWV IURP WKH YLHZSRLQWV RI ERWK LQVLGHUV DQG RXWVLGHUV 7KH XQGHUO\LQJ REMHFWLYH LV WR
DFKLHYHDPRUHKROLVWLFSHUVSHFWLYHDQGLQYROYHDPRUHGLYHUVHVHWRIVWDNHKROGHUV&DPSLODQVHHDOVR
5XELQ([DPSOHV LQFOXGH 3,'2: LQ,QGLDVHH %R[  DQGWKH &&'% LQ %DQJODGHVK ZKR XQGHUWDNH
VHOIHYDOXDWLRQ LQ ZKLFK H[WHUQDOO\FRPPLVVLRQHG DGYLVRUVIDFLOLWDWRUV DV ZHOO WKH PDMRU SURMHFW VWDNHKROGHUV
GRQRUVSURMHFWRIILFHUVDQGILHOGEDVHGVWDIISDUWLFLSDWHLQ0 (SODQQLQJDQGLPSOHPHQWDWLRQ
,QWKHLUUHYLHZRIOLWHUDWXUH$UPRQLDDQG&DPSLODQREVHUYHWKDWPDQ\RIWKHFDVHVVXUYH\HGDUH
FDUULHGRXWMRLQWO\E\WHDPPHPEHUVIURPZLWKLQDQGRXWVLGHWKHSURMHFWRULQVWLWXWLRQDOWKRXJKWKH\SRLQWRXW


WKDWWKHUHLVDFRQWLQXXPEHWZHHQLQWHUQDOO\DQGH[WHUQDOO\OHGIRUPVRI30 (,QPDQ\FDVHVWKHH[WHUQDO
HYDOXDWRU  VHUYHV DV D IDFLOLWDWRUWUDLQHU IRU LQWHUQDO FDSDFLW\ EXLOGLQJ DQG VXVWDLQDELOLW\ RI WKH 0 ( SURFHVV
+RZHYHU WKH DXWKRUV DOVR LGHQWLI\ D WUHQG WRZDUGV PRUH LQWHUQDOO\ OHG 0 ( IRU WKH IROORZLQJ UHDVRQV 
LQFUHDVHGDWWHQWLRQWRWKHSURFHVVRIOHDUQLQJDVDQHVVHQWLDOFRPSRQHQWRIDQ\GHYHORSPHQWDFWLYLW\DQGWKH
QHHGIRUJUHDWHUFRVWHIILFLHQF\([WHUQDOHYDOXDWLRQVZHUHXVHGZKHQWKHUHZDVDQHHGIRUJUHDWHU
REMHFWLYLW\
LQ
UHSRUWLQJDQGZKHQLQIRUPDWLRQZDVQHHGHGDVLQSXWVLQWRSURMHFWPDQDJHPHQWDQGSROLF\7KH\REVHUYHWKDWLQ
WKHVHFRQWH[WVWKHSDUWLFLSDWLRQRIPRVWVWDNHKROGHUJURXSVZDVOLPLWHGPDLQO\DVVRXUFHVRILQIRUPDWLRQXVLQJ
FRQVXOWDWLYHDQGSDUWLFLSDWRU\PHWKRGV
3DUWLFLSDWLRQLQ0 (PD\YDU\QRWRQO\LQWHUPVRIWKHOHYHODQGGHJUHHRISDUWLFLSDWLRQEXWDOVRRIWKH
HPSKDVLVSODFHGRQSDUWLFXODUJURXSVRIVWDNHKROGHUV&21&(51GLVFXVVHVWKHGLIIHUHQWW\SHVRISDUWLFLSDWLRQ
LQ 0 ( DQG GLVWLQJXLVKHV EHWZHHQ WKRVH WKDW LQYROYH DOO VWDNHKROGHUV DQG WKRVH WKDW IRFXV RQ WKH ORFDO
FRPPXQLW\ RU LQGLYLGXDO LQYROYHPHQW   %DVHG RQ WKH VXUYH\ RI OLWHUDWXUH 30 ( HIIRUWV PD\ EH
GLVWLQJXLVKHGWRWKHH[WHQWWKDWWKH\KLJKOLJKWWKHLQYROYHPHQWRI

• 0DMRUVWDNHKROGHUVLQFOXGLQJWKRVHZKRPD\EHGLUHFWO\DQGLQGLUHFWO\DIIHFWHGE\RULQYROYHGLQWKHSURMHFW
RUSURJUDPPHLQWHUYHQWLRQV
• %HQHILFLDULHVRISURMHFWRUSURJUDPPHLQWHUYHQWLRQV
• 0DUJLQDOLVHGJURXSVLQFOXGLQJZRPHQWKHYHU\SRRUFKLOGUHQSHRSOHZLWKGLVDELOLWLHV

:KLOH LQ WKHRU\ WKHVH VKRXOG EH LQFOXVLYH FDWHJRULHV ² PDMRU VWDNHKROGHUV IRU H[DPSOH VKRXOG LQFOXGH
EHQHILFLDULHVZKRVKRXOGLQWXUQLQFOXGHPDUJLQDOLVHGJURXSV²LQSUDFWLFHWKLVLVQRWQHFHVVDULO\VR3URMHFWVWKDW
IRFXV RQ PDMRU VWDNHKROGHUV PD\ KDYH RQO\ D VPDOO UROH IRU GLUHFW EHQHILFLDULHV 3URMHFWV WKDW IRFXV RQ
EHQHILFLDULHVPD\LQSUDFWLFHSD\OLWWOHDWWHQWLRQWRPDUJLQDOLVHGJURXSV$JDLQWKHOLWHUDWXUHVKRZVZLGHYDULDWLRQ
LQWKHDSSOLFDWLRQRI30 (DQGLQWKHHPSKDVLVJLYHQWRWKHLQYROYHPHQWRIGLIIHUHQWJURXSVRIVWDNHKROGHUV
6RPHZULWHUVFKRRVHWRHPSKDVLVHWKHSDUWLFLSDWLRQRIDOOWKHPDMRUVWDNHKROGHUV)RULQVWDQFH$EERW
DQG *XLMW GHVFULEH 30 DV
PRQLWRULQJ DSSURDFKHV WKDW GHYHORS SDUWQHUVKLSV RI PXOWLSOH VWDNHKROGHUV IRU
HIILFLHQW HIIHFWLYH DQG VRFLDOO\ LQFOXVLYH PRQLWRULQJ
   ,Q DGGLWLRQ 1DUD\DQ3DUNHU GHILQHV 3( DV D
FROODERUDWLYHSUREOHPVROYLQJSURFHVVWKDWLQYROYHVDOOOHYHOVRIXVHUV²ORFDOFRPPXQLW\PHPEHUVJRYHUQPHQW
DQGFRPPXQLW\RIILFLDOVSURMHFWDQGSURJUDPPHVWDII²LQVKDUHGGHFLVLRQPDNLQJ VHH DOVR *RVOLQJ
DQG(GZDUGV5XELQ5XJK%\LQYROYLQJDOOWKH UHOHYDQW SURMHFWVWDNHKROGHUV WKHRXWFRPHV
DQGILQGLQJVRIPRQLWRULQJDQGHYDOXDWLRQDUHH[SHFWHGWRFDWHUWRLQIRUPDWLRQQHHGVRIDYDULHW\RIXVHUJURXSV
ZLWKLQDQGRXWVLGHWKHSURMHFW&DPSLODQ+RZGLIIHUHQW VWDNHKROGHUVEHFRPH DFWLYH SDUWLFLSDQWV LQ
30 (LVLOOXVWUDWHGLQWKHQXPEHURIFDVHVWXGLHVVXUYH\HGLQWKHOLWHUDWXUHLQFOXGLQJWKH3,'2:3URMHFWLQ
.DUQDWDND,QGLDVHH%R[
2WKHU DSSURDFKHV WR 30 ( KRZHYHU KLJKOLJKW WKH LQYROYHPHQW RI EHQHILFLDULHV HVSHFLDOO\ WKRVH
VWDNHKROGHUV ZKR DUH GLUHFWO\ LQYROYHG DQG DIIHFWHG E\ SURJUDPPH LQWHUYHQWLRQV )RU LQVWDQFH 35,$ 
KROGVWKHYLHZWKDWLQSDUWLFLSDWRU\ HYDOXDWLRQ SHRSOH ZKR DUH GLUHFWO\ LQYROYHG LQ GHYHORSPHQW SURJUDPPHV
HLWKHUDVSODQQHUVLPSOHPHQWHUVRUEHQHILFLDULHVVKRXOGWDNHSDUWLQDQGFRQWUROWKHHYDOXDWLRQSURFHVV,QWKLV


 Box 3.3 St ak ehold e rs of the PI DO W p ro ject be come t h ei r own ev aluato rs
The Participative Integrated Development of Watersheds (PIDOW) Project seeks to promote sustainable land use of semi-arid
zones in Karnataka, India. During the mid-term review of the PIDOW project, project partners agreed to conduct a joint participatory
self-evaluation exercise. The major project stakeholders included:

• local rural communities, represented by Watershed Organisations (Sanghas)

• the Karnataka state government, represented by the Dryland Development Board (DLDB)
• the voluntary agency MYRADA, and
• the funding agency, the Swiss Development Corporation (SDC).

The self-evaluation process was carried out by teams whose members comprised the various project partners (government and
NGO field staff with SDC project advisers). A series of village workshops were held which involved extensive discussions with
Sangha members. The self-evaluation process provided an opportunity for all major stakeholders to jointly assess project
achievements and constraints, but it was also a means for project partners to exchange their views regarding the current operation
of PIDOW. The process also allowed project partners to look into their own performance and clarify each partner’s responsibilities
within PIDOW. The findings of the SE-experience later served as a basis for making adjustments in the overall project design and
structure and for planning the project’s next implementation phase.
Source: Humbert-Droz 1992; Sommer 1993.

ZD\3(
LVDQDWWHPSWDWUHGHILQLQJDQGUHDIILUPLQJGHYHORSPHQWDVD
ERWWRPXS

SHRSOHFHQWUHG
DQG
SHRSOH
FRQWUROOHG
SURFHVVDQGQRWDWHFKQRFUDWLFWRSGRZQLQWHUYHQWLRQ
35,$
,WVKRXOGEHQRWHGKRZHYHUWKDWDPRQJWKHFDVHVWXGLHVVXUYH\HGQRGRFXPHQWHGH[SHULHQFHRI30 (
ZDV REVHUYHG RFFXUULQJ VROHO\ DW WKH YLOODJH RU FRPPXQLW\ OHYHO $OWKRXJK WKLV ILQGLQJ GRHV QRW QHFHVVDULO\
UHIOHFW D JHQHUDO FRQGLWLRQ RI 30 ( SUDFWLFH HVSHFLDOO\ VLQFH PRVW H[SHULHQFHV DW WKH JUDVVURRWV DUH UDUHO\
GRFXPHQWHGLWGRHVKRZHYHULQGLFDWHWRVRPHH[WHQWWKDWWKHSUDFWLFHRI30 (WDNHVSODFHDFURVVLQVWLWXWLRQDO
OHYHOV 7KHVH FDVH VWXGLHV LOOXVWUDWH KRZ 30 ( IDFLOLWDWHV LQWHULQVWLWXWLRQDO OLQNDJHV DQG FROODERUDWLRQ DPRQJ
YDULRXV VWDNHKROGHUV $OO RI WKH FDVHV VXUYH\HG LQYROYHG SURMHFWOHYHO VWDII DQG EHQHILFLDULHV LQ FRQGXFWLQJ
30 ( ZKLOHIHZHU FDVHV LQYROYHG VWDNHKROGHUV ORFDWHG DW DOO LQVWLWXWLRQDO OHYHOV LQFOXGLQJ GRQRUV DQG SROLF\
PDNHUVVHH$SSHQGL[
'DYLV&DVH  DUJXHV WKDW 3DUWLFLSDWRU\ $VVHVVPHQW 0RQLWRULQJ DQG (YDOXDWLRQ 3$0( VKRXOG EH
EDVHGSULPDULO\RQWKHLQIRUPDWLRQQHHGVRIWKHLQVLGHUVUHVSRQGLQJWRZKDWWKH\QHHGWRNQRZ6KHGHILQHV
LQVLGHUV DV WKRVH ZKR LGHQWLI\ ZLWK DQG EHORQJ WR D FRPPXQLW\ RU ZLWK VXVWDLQHG UHODWLRQVKLSV ZLWK WKH
FRPPXQLW\7KURXJK3$0(LQVLGHUVVHWREMHFWLYHVDQGDFWLYLWLHVDQGPRQLWRUDQGHYDOXDWHZKHWKHUWKHVHKDYH
EHHQDFKLHYHGDQGZKHWKHUWKHVHFRQWLQXHWRKROGUHOHYDQFHWRLQVLGHUV
QHHGV7KHUROHRIRXWVLGHUVLVPDLQO\WR
HQFRXUDJHDQGKHOSLQVLGHUVLGHQWLI\WKHLUQHHGVVHWREMHFWLYHVPRQLWRUDQGHYDOXDWHWKHDFWLYLWLHV
7KH 9LHWQDP 6ZHGHQ 0RXQWDLQ 5XUDO 'HYHORSPHQW 3URJUDPPH 05'3 VHUYHV DV DQ H[DPSOH RI D

SHRSOHFHQWUHG
SURFHVVZKHUHE\ PRQLWRULQJ DQG HYDOXDWLRQ LV EDVHGSULPDULO\ RQ WKH LQIRUPDWLRQ QHHGV DQG
FRQFHUQVLGHQWLILHGE\IDUPHUEHQHILFLDULHVWKHPVHOYHVVHH%R[


 Bo x 3.4 A 'b ottom–up , top –down ' st rateg y of M & E that beg ins with fa rm ers
The Vietnam Sweden Mountain Rural Development Programme (MRDP) is an integrated rural development programme that seeks
to regenerate green productive uplands in Northern Vietnam, while promoting sustainable and secure local livelihoods. The
underlying principle of the MRDP is that ’people are willing and capable of taking responsibility for their own development’. Based on
this underlying premise, the MRDP established the Management Information and Learning System (MILS) which is designed to
facilitate the exchange of information and experience between participating organisations at different management levels (farmer,
village, commune, district, province and ministry).
However, what is unique about MILS is the central role local farmers occupy in project planning, monitoring, review, and
documentation of all programme activities included in the MRDP. The process of planning, monitoring, and evaluating programme
activities begins at the farmer and village level. Information and experience generated at the local community level is then
communicated up to the next level, which allows for a continuous ’bottom-up, top-down’ dialogue between different management
levels. MILS helps provide useful information and understanding between management staff at all levels which allows for more
effective project planning and implementation that is responsive to local needs and experiences.
Source: Vietnam Sweden Mountain Rural Development Programme, Management Information and Learning System, Province Level Training
Document No.1

2WKHUDSSURDFKHVWR30 (HVSHFLDOO\VWUHVVWKHLPSRUWDQFHRILQYROYLQJUHODWLYHO\PDUJLQDOLVHGJURXSVVXFK
DVWKHYHU\SRRUZRPHQDQGFKLOGUHQSHRSOHZLWKGLVDELOLWLHVDPRQJRWKHUV$FFRUGLQJWRWKHVHSHUVSHFWLYHV
SDUWLFLSDWRU\ HYDOXDWLRQ LV DERXW LQYROYLQJ WKH OHDVW SRZHUIXO YLVLEOH DQG DVVHUWLYH DFWRUV LQ HYDOXDWLQJ
GHYHORSPHQW HIIRUWV &DPSRV DQG &RXSDO   VHH DOVR *RVOLQJ DQG (GZDUGV  ² ,Q WKHLU
RYHUYLHZ&21&(51GLVFXVVHV30 (LQUHODWLRQWRJHQGHULVVXHVDQGVXUYH\VWKHOLWHUDWXUHSHUWDLQLQJ
WRZRPHQDQGHTXLW\:DOWHUVHWDODOVRSUHVHQWJXLGHOLQHVIRUGHYHORSLQJJHQGHUFRQVFLRXVPRQLWRULQJ
DQGHYDOXDWLRQDUJXLQJWKDWSURMHFWVDQGSURJUDPPHVPXVWEHDVVHVVHGLQWHUPVRIWKHLUHIIHFWVRQWKHSRZHU
EDODQFHEHWZHHQPHQDQGZRPHQ
.DU HW DO D E GHVFULEHV DQ H[SHULHQFH LQ SDUWLFLSDWRU\ LPSDFW PRQLWRULQJ ZKLFK HVSHFLDOO\
LQYROYHG WKH XUEDQ SRRU LQFOXGLQJ ZRPHQ LQ DVVHVVLQJ WKH LPSDFWV RI DQ XUEDQ SURMHFW RQ WKHLU RZQ OLYLQJ
HQYLURQPHQWV DQG VRFLDO FRQGLWLRQV 7KH XUEDQ SURMHFW KDG VRXJKW WR LPSURYH LQIUDVWUXFWXUH SULPDU\ KHDOWK
FDUHDQGFRPPXQLW\GHYHORSPHQW%R[

7KH3ULQFLSOHRI/HDUQLQJ
7KHFRQFHSWRIOHDUQLQJLVWKHVHFRQGPDMRUXQGHUO\LQJSULQFLSOHRI30 (7KHNH\HPSKDVLVLVRQ
SUDFWLFDO
RU

DFWLRQRULHQWHG
 OHDUQLQJ )RU LQVWDQFH 35,$  FKDUDFWHULVHV SDUWLFLSDWRU\ HYDOXDWLRQ DV D SURFHVV RI
LQGLYLGXDODQGFROOHFWLYHOHDUQLQJGHVFULELQJLWDVDQ
HGXFDWLRQDOH[SHULHQFH
IRUWKRVHYDULRXVSDUWLHVLQYROYHGLQ
DGHYHORSPHQWSURJUDPPH3HRSOHEHFRPHPRUHDZDUHDQGFRQVFLRXVRIWKHLUVWUHQJWKVDQGZHDNQHVVHVWKHLU
ZLGHUVRFLDOUHDOLWLHVDQGWKHLUYLVLRQVDQGSHUVSHFWLYHVRIGHYHORSPHQWRXWFRPHV,WLVWKLVOHDUQLQJSURFHVVWKDW
FUHDWHVFRQGLWLRQVFRQGXFLYHWRFKDQJHDQGDFWLRQ


 Box 3.5 I d entif ying loc al resid ents ’ o wn p e rcept ions and c rite r ia in a ss essi ng th e Cal cutta S lum
Improvem ent P roj ect (C SI P)
One of the main objectives of the participatory impact assessment (PIA) was to understand local residents’ perceptions and criteria
for assessing the impact of CSIP interventions, in order to suggest ways for improving project management. It was felt that the
existing means to evaluate the impact of the CSIP, which relied primarily on conventional surveys, only generated quantitative data
which did not provide enough information on how the project truly made an impact on people’s lives. Therefore, PIA was used to
identify local residents’ views regarding project achievements and impact on their livelihoods and status.
For instance, with respect to family planning and contraceptive use, one indicator was the percentage of eligible couples
using a family planning method (i.e. condoms, oral pills, etc.). Through PRA and other participatory methods, it was found that the
percentage of couples ranged from 46% to 60% in different slums. However, the PIA process also revealed different preference
criteria for different family planning methods among men and women. For example, positive perceptions of the condom included
easy availability, greater comfort for the woman, and lack of need for medical consultation, while negative perceptions included lack
of control by the woman over its use and the impossibility of it being kept confidential from the husband. The importance of including
both men’s and women’s views was highlighted.
Source: Kar et al. 1997a.

7KH SURFHVV RI OHDUQLQJ LQ 30 ( LV IXUWKHU SHUFHLYHG DV D PHDQV IRU ORFDO FDSDFLW\ EXLOGLQJ 3DUWLFLSDQWV
LQYROYHG LQ 30 ( JDLQ VNLOOV ZKLFK VWUHQJWKHQ ORFDO FDSDFLWLHV IRU SODQQLQJ SUREOHP VROYLQJ DQG GHFLVLRQ
PDNLQJ :DGVZRUWK  83:$5'  3DUWLFLSDQWV REWDLQ JUHDWHU XQGHUVWDQGLQJ RI WKH YDULRXV IDFWRUV
LQWHUQDODQGH[WHUQDOWKDWDIIHFWWKHFRQGLWLRQVDQGG\QDPLFVRIWKHLUSURMHFWWKHEDVLVIRUWKHLUVXFFHVVHVDQG
IDLOXUHV DQG WKH SRWHQWLDO VROXWLRQV RU DOWHUQDWLYH DFWLRQV &DPSRV DQG &RXSDO  7KH H[SHULHQFH RI WKH
(=(& /HDUQLQJ ,QLWLDWLYH DOVR VHUYHV WR GHPRQVWUDWH KRZ FRPPXQLW\EDVHG 0 ( VWUHQJWKHQV ORFDO
FDSDELOLWLHVVHH%R[7KHOHDUQLQJSURFHVVLQ30 (LVSUHPLVHGXSRQWKHSULQFLSOHWKDWLWVXSSRUWVH[LVWLQJ
FRPPXQLW\VNLOOVDQGUHVRXUFHV)HXHUVWHLQ[LDUJXHVWKDWSDUWLFLSDWRU\HYDOXDWLRQEXLOGVRQZKDWSHRSOH
DOUHDG\NQRZDQGGRXVLQJDQGGHYHORSLQJSHRSOH
VFXUUHQWDELOLWLHVDQGVNLOOVWRPRQLWRUDQGHYDOXDWHWKHLURZQ
SURJUHVVVHHDOVR&DPSRVDQG&RXSDO'DYLV&DVH)RULQVWDQFHLQ*XMDUDW,QGLDWKH$.563KDV
ZRUNHG ZLWK IDUPHUV LQ SURPRWLQJ VRLO DQG ZDWHU FRQVHUYDWLRQ 7KURXJK SDUWLFLSDWRU\ PHWKRGV IDUPHUV
WKHPVHOYHVLGHQWLI\DJULFXOWXUDOSUREOHPVDQGOHDUQKRZWRPRQLWRUDQGHYDOXDWHFRQVHUYDWLRQPHDVXUHVWHVWHG
E\ PDNLQJ
JURXQG PDSV
 RI WKHLU ILHOGV XVLQJ ORFDO PDWHULDOV LH VWRQHV VDQG HWF )DUPHUV WKHQ WRJHWKHU
FRPSDUHDQGGLVFXVVILQGLQJVEDVHGRQWKHLUPDSVZKLFKDOORZWKHPWRPDNHSURGXFWLRQGHFLVLRQVDQGJHQHUDWH
WHFKQRORJLHVWKDWDUHDGDSWHGWRYLOODJHFLUFXPVWDQFHVVHH%R[
7KH FRQFHSW RI 30 ( DV DQ H[SHULHQWLDO OHDUQLQJ F\FOH VHUYHV WR HPSKDVLVH WKH SRLQW WKDW LQ 30 (
SDUWLFLSDQWVWRJHWKHUOHDUQIURPH[SHULHQFHDQGJDLQWKHDELOLWLHVWRHYDOXDWHWKHLURZQQHHGVDQDO\VHWKHLURZQ
SULRULWLHVDQGREMHFWLYHVDQGXQGHUWDNHDFWLRQRULHQWHGSODQQLQJ$QHVVHQWLDOIHDWXUHRIWKLVOHDUQLQJF\FOHLVIRU
VWDNHKROGHUJURXSVWRUHIOHFWFRQWLQXRXVO\RQWKH LPSDFW RI WKHLU HYDOXDWLRQ DQG ZKHUH WKH SURFHVV LV OHDGLQJ
WKHPOHDUQLQJIURPWKHLURZQVXFFHVVHVDQGPLVWDNHV3IKRO(YDOXDWLRQWKHQEHFRPHVSDUWRIDQRQJRLQJ
SURFHVVRIFRPPXQLW\OHDUQLQJDQGFDSDFLW\EXLOGLQJ)RUH[DPSOHLQWKHFDVHRIWKH05'3LQ9LHWQDPVHH
%R[30 (LVXQGHUWDNHQRQDPRQWKO\TXDUWHUO\DQGDQQXDOEDVLVE\GLIIHUHQWVWDNHKROGHUVDWGLIIHUHQW
PDQDJHPHQWOHYHOVZKLFKDOORZVIRUFRQVWDQWIHHGEDFNUHYLHZDQGDGMXVWPHQWVWRLPSOHPHQWDWLRQSODQV


 Box 3.6 Le arning as cap acit y bui lding
The EZ/EC Learning Initiative is intended to monitor and measure the impact of the Empowerment Zone/Enterprise Community
programme, a major federal programme under the Clinton administration in the United States. The EZ/EC programme provides
financial and institutional support for rural communities to develop their own community development initiatives.
The EZ/EC Learning Initiative was designed to provide the means through which community representatives themselves learn
how to monitor and measure the progress of their EZ/EC programme as it develops, and suggest midcourse adjustments to the
agencies running the EZ/EC programme. Learning team members gain new skills and enhance their capacities in monitoring
change indicators and analysing their implications, in facilitation of meetings, and in public speaking. Local skills development has
led to increased self-confidence, which in turn has enhanced the abilities of people to develop their own strategic goals for change
and pursue leadership roles in effecting change within their communities. Learning team members have successfully analysed and
presented their findings and recommendations concerning programme implementation to EZ/EC administrators and government
officials.
Source: Gaventa, Morrissey, et al. 1997; Parachini and Mott 1997

+RZHYHUOHDUQLQJPD\DOVRWDNHSODFHDWDQRUJDQLVDWLRQDODQGLQVWLWXWLRQDOFRQWH[WDVGHPRQVWUDWHGE\WKHFDVHRI
&$5(=DPELDVHH%R[$VSDUWRILWVSURFHVVRIEHFRPLQJD
OHDUQLQJRUJDQLVDWLRQ
&$5(=DPELDSURPRWHV
YDULRXV VWUDWHJLHV DQG LQLWLDWLYHV WR GHYHORS KLJKHU OHYHOV RI NQRZOHGJH DQG VNLOOV HVSHFLDOO\ RI &$5( VWDII
PHPEHUV)RXUOHYHOVRIOHDUQLQJDUHLGHQWLILHG+DPHODQG3UDKDOHGFLWHGLQ:DUG

• /HYHO/HDUQLQJIDFWVNQRZOHGJHSURFHVVHVDQGSURFHGXUHV
• /HYHO/HDUQLQJQHZMREVNLOOVWKDWDUHWUDQVIHUDEOHWRRWKHUVLWXDWLRQV
• /HYHO/HDUQLQJWRDGDSWDQGWRGHULYHOHVVRQVIURPVXFFHVVDQGIDLOXUH
• /HYHO/HDUQLQJWREHLQQRYDWLYHDQGFUHDWLYHGHVLJQLQJWKHIXWXUHUDWKHUWKDQPHUHO\DGDSWLQJWRLW

0RYLQJWRKLJKHUOHYHOVRIOHDUQLQJZLWKLQDQRUJDQLVDWLRQ
UHTXLUHVFUHDWLYLW\LQQRYDWLRQDGDSWLYHWKLQNLQJDQG
DGHHSHUXQGHUVWDQGLQJRISDUWLFLSDWLRQDVDQRQJRLQJSURFHVVUDWKHUWKDQDVDQLQGLYLGXDODFWLYLW\
:DUG

7KH3ULQFLSOHRI1HJRWLDWLRQ
7KH VXUYH\ RI OLWHUDWXUH DOVR VXJJHVWV WKDW30 ( LV LQFUHDVLQJO\ SHUFHLYHG DV D VRFLDO SURFHVV IRU QHJRWLDWLQJ
EHWZHHQ SHRSOH
V GLIIHUHQW QHHGV H[SHFWDWLRQV DQG ZRUOGYLHZV )RU LQVWDQFH 'H 5DHGW  GHVFULEHV
30 ( DV D
VRFLDO HQWHUSULVH RI SXEOLF GLVFRXUVH DQG LQWHUDFWLRQ
 HQJDJLQJ YDULRXV VWDNHKROGHUV ZLWK GLYHUVH
LQWHUHVWV LQ QHJRWLDWLQJ DQG DFKLHYLQJ WKHLU REMHFWLYHV $FFRUGLQJ WR *XED DQG /LQFROQ  
IRXUWK
JHQHUDWLRQHYDOXDWLRQ
HVVHQWLDOO\LVDSURFHVVRIQHJRWLDWLRQEDVHGRQWKHSUHPLVHWKDWGLIIHUHQWVWDNHKROGHUVZLOO
KDYHGLIIHUHQWFODLPVFRQFHUQVDQGLVVXHVZKLFKLQWXUQDUHVKDSHGE\WKHFRQWH[WVSHFLILFVRFLDOYDOXHVRIWKH
GLIIHUHQWVWDNHKROGHUVVHHDOVR0DUVGHQDQG2DNOH\(YHQZKHQWKHUHLVDQHPSKDVLVRQRQHSDUWLFXODU
JURXS RI VWDNHKROGHUV ² EHQHILFLDULHV IRU H[DPSOH ² WKH DSSURDFK UHFRJQLVHV WKH FRPSOH[ VHW RI LQWHU
UHODWLRQVKLSVLQYROYHG


 :KHQPXOWLSOHVWDNHKROGHUVDUHLQYROYHGLQWKHPRQLWRULQJDQGHYDOXDWLRQSURFHVVQHJRWLDWLRQLVSHUFHLYHG
DV FRQWULEXWLQJ WRZDUGV WKH EXLOGLQJ RI WUXVW DQG FKDQJLQJ SHUFHSWLRQV EHKDYLRXUV DQG DWWLWXGHV DPRQJ
VWDNHKROGHUV ZKLFK DIIHFW WKH ZD\ WKH\ FRQWULEXWH WR WKH SURMHFW 6FRWW9LOOLHUV ZULWHV WKDW
WKH SXUSRVH RI
ZRUNLQJWRJHWKHURQFRQVXOWDWLYH0 (LVWRLPSURYHDOOSDUWLHV
XQGHUVWDQGLQJRIWKHLURZQDQG HDFK RWKHU
V
LQWHUHVWVSHUFHSWLRQVDQGUROHVLQWKHSURMHFWLQRUGHUWKDWWKHSURMHFWRXWFRPHFDQEHFRQWLQXRXVO\LPSURYHG
IRUDOOSDUWLHV
,QWKHFDVHRIWKH3,'2:SURMHFWVHH%R[WKHVHOIHYDOXDWLRQH[HUFLVHDOORZHG
SURMHFWVWDNHKROGHUVWRLQFUHDVHWKHLUXQGHUVWDQGLQJRIWKHLURZQUROHVDQGUHVSRQVLELOLWLHVZKLFKLQWXUQKHOSHG
VWUHQJWKHQFROODERUDWLRQDQGLQVWLOOWUXVWDPRQJSURMHFWSDUWQHUV+XPEHUW'UR]6RPPHU
0DQ\ZULWHUVDOVRDFNQRZOHGJHWKDWWKHQHJRWLDWLRQSURFHVVRI30 (LVDKLJKO\SROLWLFDOH[HUFLVHZKLFK
QHFHVVDULO\DGGUHVVHVLVVXHVRIHTXLW\SRZHUDQGVRFLDOWUDQVIRUPDWLRQ)RULQVWDQFH6FRWW9LOOLHUVHPSKDVLVHV
WKDWWKHSURFHVVLQYROYHVPRUHEDODQFHGUHSUHVHQWDWLRQDPRQJWKHYDULRXVVWDNHKROGHUV
&RQVXOWDWLYHVNLOOVDLP
WRHQKDQFHHTXLW\UHFRJQLVLQJWKHLPSRUWDQFHRIWKHRSLQLRQVRIQRUPDOO\VLOHQWPHPEHUVRIWKHSURMHFWDVZHOO
DVWKHLPSRUWDQWRSLQLRQVRIWKRVHZKROHDGWKHSURMHFW

,Q DGGLWLRQ *XED DQG /LQFROQ  DUJXH WKDW WKH SURFHVV RI QHJRWLDWLRQ ZLOO HLWKHU
HQIUDQFKLVH RU
GLVHQIUDQFKLVH
VWDNHKROGHUJURXSVLQYDULRXVZD\VLHWKURXJKWKHVHOHFWLYHLQYROYHPHQWRIWKHVHVWDNHKROGHUVLQWKH
GHVLJQ LPSOHPHQWDWLRQ UHSRUWLQJ DQG XVH RI HYDOXDWLRQ +HQFH QHJRWLDWLRQ FDQ EHFRPH DQ HPSRZHULQJ RU
GLVHPSRZHULQJSURFHVV(PSRZHUPHQWLVGHILQHGLQWHUPVRIWKHGHJUHHWRZKLFK
IXOOSDUWLFLSDWLYH
LQYROYHPHQWLQ
HYHU\DVSHFWRIGHVLJQLPSOHPHQWDWLRQLQWHUSUHWDWLRQDQGUHVXOWLQJDFWLRQLVDFKLHYHG
&RXVLQVDQG:KLWPRUHIXUWKHUHODERUDWHRQWKHLVVXHRISRZHUDQGUDLVHNH\TXHVWLRQVIRFXVLQJRQ
ZKRFUHDWHVDQGZKRFRQWUROVWKHSURGXFWLRQRINQRZOHGJHDQGLQIRUPDWLRQLQPRQLWRULQJDQGHYDOXDWLRQ7KH\
DUJXH WKDW WKH PDLQ DLP RI
WUDQVIRUPDWLYH
 SDUWLFLSDWRU\ HYDOXDWLRQ LV WR HPSRZHU SHRSOH WKURXJK DQ
HGXFDWLRQDOOHDUQLQJSURFHVVE\ZKLFKYDULRXVVRFLDOJURXSVSURGXFHNQRZOHGJHDERXWWKHLUUHDOLW\FODULI\DQG
DUWLFXODWH WKHLU QRUPV DQG YDOXHV DQG UHDFK FRQVHQVXV DERXW IXUWKHU DFWLRQ 
FRQVFLHQWLVDWLRQ
 &RXVLQV DQG
:KLWPRUH7KHHYDOXDWLRQSURFHVVLVWKHUHIRUHXVHGWRWUDQVIRUPSRZHUUHODWLRQVDQGDPHOLRUDWHVRFLDO
LQHTXLWLHVE\SURPRWLQJVRFLDODFWLRQDQGFKDQJH
7KHSROLWLFVRIQHJRWLDWLRQEHFRPHHYLGHQWSDUWLFXODUO\LQWKHFRQWH[WRIGHYHORSLQJLQGLFDWRUVDQGFULWHULD
IRU PRQLWRULQJ DQG HYDOXDWLRQ HVSHFLDOO\ LQ GHWHUPLQLQJ ZKRVH SHUVSHFWLYHV DUH UHSUHVHQWHG LQ VHOHFWLQJ
LQGLFDWRUV7KHVHFDVHVGHPRQVWUDWHWKHXQGHUO\LQJSROLWLFDOG\QDPLFVLQKHUHQWLQVWDNHKROGHUUHODWLRQVKLSVDQG
LQWHUDFWLRQ)RUH[DPSOHWKH&KULVWLDQ&RPPLVVLRQIRU'HYHORSPHQWLQ%DQJODGHVK&&'%H[SHULPHQWHGZLWK
DSDUWLFLSDWRU\PRQLWRULQJIUDPHZRUNZKLFKDWWHPSWHGWRLQFOXGHWKHLQWHUHVWVDQGFRQFHUQVRIDOOVWDNHKROGHUV
LQ WKH RUJDQLVDWLRQDO KLHUDUFK\ E\ DVNLQJ WKHP WR LGHQWLI\ VLJQLILFDQW FKDQJHV DV D UHVXOW RI WKHLU ZRUN 7KH
SURFHVVUHYHDOHGWKDWDOPRVWDOOFKDQJHVUHSRUWHGE\WKHVWDNHKROGHUVZHUHSRVLWLYH)LHOGOHYHOVWDIIH[SODLQHG
WKHUDWLRQDOHLQWHUPVRISHUFHLYHGULVNVWRWKHLUMREVHFXULW\LIWKH\HQJDJHGLQPRUHFULWLFDOUHSRUWLQJWRVHQLRU
OHYHOVWDIIRQWKHRWKHUKDQGVHQLRUVWDIILQGLFDWHGWKHLUFRQFHUQDERXWGRQRUDQGJRYHUQLQJERDUGUHVSRQVHVWR
QHJDWLYHFKDQJHV'DYLHV$VLPLODUH[SHULHQFHZDVIRXQGLQ0H[LFRZKHUHIDUPHUEHQHILFLDULHVLQLWLDOO\
UHPDLQHGUHWLFHQW
RIWHQFDXWLRXVDQGSROLWH
LQDUWLFXODWLQJWKHLUFULWHULDIRUHYDOXDWLQJDJULFXOWXUDOSURJUDPPHV
LQWKHSUHVHQFHRIH[WHQVLRQLVWVDQGGRQRUUHSUHVHQWDWLYHV8VHRISDUWLFLSDWRU\PHWKRGVODWHUHQDEOHGIDUPHUVWR
H[SUHVV WKHLU YLHZV PRUH GLUHFWO\ DQG PRUH FRQVWUXFWLYHO\ ZKLFK KHOSHG H[SODLQ WR H[WHQVLRQLVWV IDOOLQJ
SDUWLFLSDWLRQE\YLOODJHUVLQILHOGDQGHYDOXDWLRQZRUN%ODXHUWDQG4XLQWDQDUVHH%R[


 Bo x 3 . 7 Est a b li sh i n g co m m o n o bj e cti ves bas ed on di ff er ent stak ehol d e r i nte r est s
A project between two partner organisations – CETAMEX-N, an NGO based in southern Mexico, and World Neighbours, an
international development agency based in the US – was undertaken to design an evaluation framework that would feed into
ongoing farming and research development work, as well as current issues facing local farmer-extensionists associated with
CETAMEX-N and village participants. The framework sought was to allow different actors in the programme – including farming
families, communities, extensionists, NGO funders, etc.– to evaluate, separately and jointly, project activities for socio-political,
environmental and economic processes and impacts. Specifically, focus was given to changes in roles and attitudes in the
CETAMEX-N institution itself, impact of extension work, and on the participation by different stakeholders in the programme's work.
Identifying participants' evaluation criteria was regarded as a preliminary step to establish indicators for the evaluation
framework. However, it was recognised that it would be impossible to evaluate a project by indicators responding to all these criteria.
One alternative to collate and combine stakeholder criteria was to group together 'loosely' external and internal criteria according to
the six broad categories developed originally by the Inter-American Foundation's 'Cone' framework. Adopting this approach still
allowed different stakeholders' views to be represented.
Source: Blauert and Quintanar 1997

7KH3ULQFLSOHRI)OH[LELOLW\
)OH[LELOLW\ DQG H[SHULPHQWDWLRQ DUH UHJDUGHG DV LQWHJUDO DVSHFWV RI 30 ( 0DQ\ FRQFXU WKDW WKHUH LV QR
EOXHSULQW RU SUHVFULEHG VHW RI DSSURDFKHV WR FDUU\LQJ RXW 30 ( UDWKHU WKH\ YLHZ WKH SURFHVV 30 ( DV
FRQWLQXDOO\ HYROYLQJ DQG DGDSWLQJ DFFRUGLQJ WR SURMHFWVSHFLILF FLUFXPVWDQFHV DQG QHHGV 2QH DVSHFW RI
LQWHJUDWLQJIOH[LELOLW\LQWKHGHVLJQDQGSUDFWLFHRI30 (LVLQPDNLQJWKHSURFHVVLWVHOIUHVSRQVLYHDQGUHOHYDQW
WRWKHQHHGVRIVWDNHKROGHUV6SHFLILFDOO\30 (VKRXOGEHFRQWH[WXDOZKLFKWDNHVLQWRDFFRXQWORFDOFRQGLWLRQV
VRFLRFXOWXUDOHFRQRPLFSROLWLFDOLQVWLWXWLRQDOFRQWH[WVVHH0DUVGHQDQG2DNOH\83:$5'7KLV
DFFRXQWVIRUWKHYHU\ZLGHUDQJHLQWKHSUDFWLFHRISDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQDVUHYHDOHGLQWKH
FDVHVWXGLHV
3IRKO LOOXVWUDWHV WKLV HYROYLQJ DQG DGDSWLYH SURFHVVLQ WKHIRUP RI DQ
HYDOXDWLRQ FKDNUD
 GUDZLQJ
IURP WKH V\PERO RI D FKDNUD RU ZKHHO 7KH FKDNUD UHSUHVHQWV F\FOLFDO UHSHWLWLRQ RI UHDVVHVVPHQW DQG VHOI
HYDOXDWLRQ LQ RUGHU WR DOORZ IRU DGMXVWPHQWV LQ WKH HYDOXDWLRQ SURFHVV LWVHOI +RZHYHU -REHV SHUVRQDO
FRPPXQLFDWLRQ VXJJHVWV WKDW D F\FOLFDO YLHZ PD\ QRW DGHTXDWHO\ FDSWXUH WKH FKDQJLQJ IRUP RI 30 ( DQG
ZRXOGUDWKHUGHVFULEHWKHDFWXDOSURFHVVRI30 (DV
DYHU\VORZPXOWLOD\HUHGEDFNZDUGDQGIRUZDUGVWRS
VWDUWH[SHULHQFH
7KHGHJUHHWRZKLFK30 (LQFRUSRUDWHVIOH[LELOLW\LQLWVGHVLJQDQGLPSOHPHQWDWLRQKDVEHHQ
GRFXPHQWHG LQ VHYHUDO SURMHFWUHODWHG H[SHULHQFHV )RU H[DPSOH LQ KHU ZRUN ZLWK KLJKODQG DJULFXOWXUDO
FRPPXQLWLHVLQ3HUXRQWHFKQRORJ\ GHYHORSPHQW DQG YDOLGDWLRQ )HUQDQGH]  OHDUQHG WKDW DVVHVVLQJ DQG
HYDOXDWLQJSURGXFWLRQSUREOHPVZLWKIDUPHUVUHTXLUHGDKLJKOHYHORIFUHDWLYLW\DQGIOH[LELOLW\RQWKHSDUWRIWKH
UHVHDUFKHUVLQYROYHG5HVHDUFKHUVKDGWRFRQWLQXDOO\DGDSWUHVHDUFKPHWKRGVRIDVVHVVPHQWWRWKHORFDOUHDOLWLHV
RIIDUPHUV²WDNLQJLQWRDFFRXQWHFRORJLFDOHFRQRPLFVRFLDODQGRUJDQLVDWLRQDOIDFWRUV
$QRWKHUH[DPSOHLVWKH0DQDJHPHQW,QIRUPDWLRQDQG/HDUQLQJ6\VWHP0,/6RIWKH05'3SURJUDPPH
LQ9LHWQDP6ZHGHQ0RXQWDLQ'HYHORSPHQW3URJUDPPH05'3VHH%R[ZKLFKXVHVDIOH[LEOHSODQQLQJ
GHVLJQWKDWHQDEOHVIDUPLQJFRPPXQLWLHVFRQWLQXDOO\WRUHDVVHVVDQGDGMXVWDFWLYLWLHVEDVHGRQWKHLUQHHGVDQG


GHPDQGV $OVR WKH (=(& /HDUQLQJ ,QLWLDWLYH ZKLFK LV GHVLJQHGWR PRQLWRU DQG PHDVXUH WKH LPSDFW RI WKH
(=(& 3URJUDPPH VHH %R[  DGRSWV DQ HYDOXDWLRQ IUDPHZRUN EDVHG RQ D FROODERUDWLYH SURFHVV WKDW LV
SXUSRVHIXOO\GHVLJQHGWRUHYLHZDQGLPSURYHSURJUDPPHLPSOHPHQWDWLRQRQDUHODWLYHO\FXUUHQWEDVLV
2QHH[SHULHQFHLQ,QGLDLOOXVWUDWHVKRZDIOH[LEOH30 (V\VWHPFDQHIIHFWLYHO\PRQLWRUFKDQJHVDQGLPSDFWVDV
ZHOODVUHVSRQGWRQHHGVDQGFDSDFLWLHVRIGLIIHUHQWVWDNHKROGHUVVHH%R[
:KLOH WKHVH SULQFLSOHV DUH LPSRUWDQW SXWWLQJ WKHP LQWR SUDFWLFH UHSUHVHQWV HQRUPRXV FKDOOHQJHV 7KH
IROORZLQJVHFWLRQZLOOIRFXVRQWRROVDQGDSSURDFKHVXVHGWR30 (DFWLYLWLHV

75$16/$7,1*3$57,&,3$725<021,725,1*$1'(9$/8$7,21,172
35$&7,&(
$VWKHYDOXHRI30 (LVPRUHEURDGO\UHFRJQLVHGWKHUHLVJURZLQJQHHGWRV\VWHPDWLVHLWVSUDFWLFHLQRUGHU
WKDW WKH SURFHVV LWVHOI PD\ EH VKDUHG UHSOLFDWHG DQG LPSURYHG :KDW DUH WKH NH\ VWHSV RU VWDJHV LQ D 30 (
SURFHVV" :KDW WRROV DQG WHFKQLTXHV VKRXOG EH XVHG" :KR VKRXOG EH LQYROYHG DQG KRZ" 7R H[SORUH WKHVH
TXHVWLRQVWKLVVHFWLRQGUDZVXSRQDVXUYH\RIOLWHUDWXUHDQGDQXPEHURIPDQXDOVDQG JXLGHERRNV ZKLFK DUH
QRZ DYDLODEOH IRU 30 ( VHH $SSHQGL[  $OWKRXJK WKHUH LV ZLGH YDULDWLRQ LQ DFWXDO 30 ( SUDFWLFH VRPH
FRPPRQJXLGHOLQHVDQGPHWKRGVDUHHPHUJLQJZKLFKKHOSGHILQHWKHSURFHVVIRUHVWDEOLVKLQJDQGFRQGXFWLQJ
30 (7KLVVHFWLRQSURYLGHVDEULHIRYHUYLHZRI30 (PHWKRGRORJ\IROORZHGE\DGHVFULSWLRQRIVRPHRI
WKHWRROVDQGWHFKQLTXHVFRPPRQO\XVHG

Bo x 3.8 A fl exib le app roac h to group-ba sed sav ings and c redit
A prototype internal learning system was used in India for operating group-based savings and credit programmes with poor women.
The tasks of ’internal learning’ are decentralised to all levels within the savings and credit programme – at the level of individual
members, self-help groups, cluster or federation of groups, area organisers, programme staff, etc. – so that all members are actively
involved in the process of planning, data collection, analysis, and documentation. One of the distinguishing features of the system is
its use of pictorial diaries, known as 'learning diaries', in which information is shared, aggregated and forwarded at every level of the
programme. Women use the diaries to record and reflect upon changes in their lives as shown in the diaries. These learning diaries
are organised into chapters, which address different areas relating to personal, social, economic, and institutional aspects of
women's lives.
Although the learning diaries suggest and organise what could be monitored, the system is designed in a way that allows for
considerable flexibility in choosing the structure and content of the diaries. The diaries may be tailored differently and used
purposefully to answer specific monitoring and evaluation questions. Different levels of users can select and modify the various
chapters or dimensions of change to respond to their own unique experiences, capacities, and needs. As members and groups
become more skilled, other chapters can be added later on. This degree of flexibility enables users at all levels to carry out
evaluation experiments and conduct more in-depth sub-analyses, which still allow for general inferences to be made across the
various programme levels.
Source: Naponen, n.d.


:KDWDUHWKH6WHSVRU6WDJHVRI30 ("
'UDZLQJ IURP &21&(51
V  RYHUYLHZ WKHUH DUH IRXU PDMRU VWHSV RU VWDJHV RI DSSO\LQJ 30 ( LQ
SUDFWLFH

• 3ODQQLQJRUHVWDEOLVKLQJWKHIUDPHZRUNIRUD30 (SURFHVVLQFOXGLQJLGHQWLILFDWLRQRIREMHFWLYHVDQG
LQGLFDWRUV
• *DWKHULQJGDWD
• 'DWDDQDO\VLV
• 'RFXPHQWDWLRQUHSRUWLQJDQGVKDULQJRILQIRUPDWLRQ

7KHVH PDLQ VWDJHV RI 30 ( DUH ZHOOLOOXVWUDWHG LQ WKH H[SHULHQFH RI WKH /HDUQLQJ ,QLWLDWLYH ZKHUH OHDUQLQJ
WHDPV KDYH GHYHORSHG DQG DSSOLHG WKH
/HDUQLQJ :KHHO
 IRU FLWL]HQ PRQLWRULQJ VHH 7DEOH  7HDP PHPEHUV
ZRUN ZLWK D UHJLRQDO UHVHDUFKHU ZKR KHOSV IDFLOLWDWH WKH SURFHVV DQG SURYLGHV UHVHDUFK VXSSRUW DQG WUDLQLQJ
(DFKSKDVHLVEURNHQGRZQLQWRGLVWLQFWDFWLYLWLHVWKDWDUHXQGHUWDNHQLQDVHULHVRIZRUNVHVVLRQV

3ODQQLQJD30 (SURFHVVHVWDEOLVKLQJREMHFWLYHVDQGLQGLFDWRUV
7KH SODQQLQJ VWDJH LV JHQHUDOO\ FRQVLGHUHG WKH PRVW FULWLFDO WR WKH VXFFHVV DQG HIIHFWLYHQHVV RI WKH 30 (
SURFHVV EHFDXVH LW UHTXLUHV D OHQJWK\ SURFHVV RI QHJRWLDWLRQ FRQWHVWDWLRQ DQG FROODERUDWLYH GHFLVLRQ PDNLQJ
DPRQJYDULRXVVWDNHKROGHUV,WLVDWWKLVVWDJHWKDWGLIIHUHQWVWDNHKROGHUJURXSVFRPHWRJHWKHUDQGYRLFHWKHLU
FRQFHUQVQHHGVDQGH[SHFWDWLRQV
$WWKLVVWDJHHVWDEOLVKLQJWKHREMHFWLYHVRIWKH30 (SURMHFWEHFRPHVFULWLFDO7KHUHOHYDQWVWDNHKROGHU
JURXSVLQLWLDOO\QHHGWREHLGHQWLILHGDQGVHOHFWHGDQGVHYHUDOPDMRUTXHVWLRQVWKHQQHHGWREHFODULILHG7KHVH
NH\ TXHVWLRQV LQFOXGH ZKDW LQIRUPDWLRQ IRU ZKRP DQG KRZ LW VKDOO EH LQFRUSRUDWHG LQWR SURMHFW RU
SURJUDPPH LPSOHPHQWDWLRQ VHH 6FRWW9LOOLHUV D 7KLV SURFHVV VKRXOG HQWDLO LQWHQVLYH QHJRWLDWLRQV
EHWZHHQVWDNHKROGHUVUHJDUGLQJWKHQHHGVDQGREMHFWLYHVRIWKHGLIIHUHQWSDUWLHV6FRWW9LOOLHUVDVHHDOVR
$EERWDQG*XLMW&DPSRVDQG&RXSDO83:$5'
,GHQWLI\LQJ REMHFWLYHV DQG PRQLWRULQJ LQGLFDWRUV FDQ EH WKH PRVW GLIILFXOW SDUW RI SODQQLQJ D 30 (
SURFHVV6FRWW9LOOLHUVDSRLQWVRXWWKDWLQJHQHUDOPDQ\SHRSOHGRQRWJLYHPXFKWKRXJKWWRZK\WKH\
DUHSDUWLFLSDWLQJLQ0 (DQGVLPSO\DFFHSWWKDWDSURMHFWKDVEHHQGHVLJQHGIRUWKHPE\VRPHRQHHOVHDVLWZDV
LQ PRVW FDVHV WUDGLWLRQDOO\ 7KHUHIRUH D FRQVXOWDWLRQ SURFHVV DPRQJ WKH YDULRXV VWDNHKROGHUV WR GHWHUPLQH
REMHFWLYHVDQGLQGLFDWRUVLVUHJDUGHGE\PDQ\DVFULWLFDOWRVXFFHVVIXOO\FDUU\LQJRXWDQGVXVWDLQLQJ30 (
$OWKRXJK PXFK DWWHQWLRQ LV RIWHQ IRFXVHG RQ LQGLFDWRU GHYHORSPHQW $EERW DQG *XLMW  VWUHVV WKH
LPSRUWDQFHRIILUVWHVWDEOLVKLQJREMHFWLYHVDVDNH\VWHSEHIRUHLQGLFDWRUVDUHGHILQHGVHHDOVR6FRWW9LOOLHUV,Q
RUGHUWRGHWHUPLQHWKHREMHFWLYHVRI30 (RQHPXVWNQRZZKRWKHDFWRUVDUHLQWKHSURFHVV DQG ZKR WKH
HQGXVHUVZLOOEHZK\WKHSURMHFWLVWREHFDUULHGRXWDQGKRZWKHUHVXOWVDQGSURFHVVDUHWREHXVHG(QGXVHUV
PD\ EH DW PXOWLSOH OHYHOV LQFOXGLQJ FRPPXQLW\ PHPEHUV SURMHFW ORFDO VWDII GRQRUV GHYHORSPHQW DJHQFLHV
UHVHDUFKQDWLRQDOOHYHOSROLF\PDNHUV*2VDQG1*2VRUHYHQLQGLUHFWEHQHILFLDULHVLQDFRPPXQLW\RUWKH
ZLGHUJHQHUDOSXEOLF:KDWLVWREHPRQLWRUHGDQGHYDOXDWHGDQGKRZWKHSURFHVVLVWREHXVHGZLOOHVVHQWLDOO\


GHSHQGRQZKRQHHGVWKHUHVXOWVDQGLQIRUPDWLRQRI30 (DQGZKRLVLQYROYHGLQGHWHUPLQLQJWKHSURFHVV
VHH%R[
+DYLQJ DJUHHG XSRQ REMHFWLYHV RI 30 ( LQGLFDWRUV FDQ WKHQ EH HVWDEOLVKHG 7KHUH DUH PDQ\ YDULHG
GHILQLWLRQVIRULQGLFDWRUVVHH)HXHUVWHLQ*RVOLQJDQG(GZDUGV+DUWFLWHGLQ$EERWDQG*XLMW
(VVHQWLDOO\LQGLFDWRUVDUHGLVWLQJXLVKHGIURPREMHFWLYHVLQWKDWLQGLFDWRUVDUHVDLGWREHUHODWLYHO\WDQJLEOH
DQG VSHFLILF ZKLOH REMHFWLYHV PD\ UHSUHVHQW WKH EURDGHU RYHUDOO SHUVSHFWLYHV RI WKH SURMHFW RU LQLWLDWLYH
UHIOHFWLQJLWVYDOXHVYLVLRQVDQGORQJWHUPJRDOV35,$
6HYHUDO FULWHULD IRU VHOHFWLQJ LQGLFDWRUV VXJJHVW WKDW LQGLFDWRUV EH YDOLG FRVWHIIHFWLYH UHOHYDQW VHQVLWLYH
YHULILDEOHVLPSOHPHDQLQJIXOWRDOOLQYROYHGDPRQJRWKHUVVHH$EERWDQG*XLMW&21&(51$
UHODWLYHO\ VLPSOHU VHW RI FULWHULD IRU GHWHUPLQLQJ LQGLFDWRUV LV EDVHG RQ WKH DFURQ\P
60$57
 VSHFLILF
PHDVXUDEOHDFWLRQRULHQWHGUHDOLVWLFDQGWLPHIUDPHG$EERWDQG*XLMWVHHDOVR6FRWW9LOOLHUVD
6FRWW9LOOLHUVDDUJXHVWKDWLQGLFDWRUVVHOGRPQHHGWREHH[WUHPHO\SUHFLVHDQGWKDWWKH\VKRXOGEHHDV\WR
FROOHFWDQGEHXVHIXOWRWKHSURMHFWGHFLVLRQPDNLQJSURFHVVVHHDOVR3IRKO+RZHYHURWKHUVDUJXH
WKDWGHILQLQJLQGLFDWRUVPD\QRWEHDVVWUDLJKWIRUZDUGVLQFHWKH\QHHGWREH
VXJJHVWHGDGDSWHGQHJRWLDWHGDQG
DSSURYHG
E\DOOWKHUHOHYDQWVWDNHKROGHUV$EERWDQG*XLMWVHHDOVR5XELQ7KHVHOHFWLRQRI
GLIIHUHQWNLQGVRILQGLFDWRUVZLOOXOWLPDWHO\GHSHQGRQZKDWLVEHLQJDVVHVVHGZKRWKHHQGXVHUVDUHDQGKRZ
LQIRUPDWLRQ ZLOO EH XVHG DV DOUHDG\ GHPRQVWUDWHG E\ $VVHVVRULD H 6HUYLFRV D 3URMHWRV HP $JULFXOWXUD
$OWHUQDWLYD
V$637$H[SHULHQFHLQ%UD]LOVHH%R[VHHDOVR0F$UWKXU²
)RU LQVWDQFH RQH SDUWLFXODU FDVH VKRZHG KRZ LQGLFDWRUV FDQ UHIOHFW GLIIHULQJ FRQFHUQV RI VWDNHKROGHUV
ZLWKLQORFDOFRPPXQLWLHVSDUWLFXODUO\GLIIHUHQFHVEHWZHHQLQWHQGHGEHQHILFLDULHV7KHFDVHGHVFULEHVDQ1*2
V
H[SHULHQFHLQ0DOLDQGKLJKOLJKWVKRZGLIIHUHQWLQGLFDWRUVZHUH GHILQHG DFFRUGLQJWR DJH JHQGHU RFFXSDWLRQ
DQGZHDOWKVWDWXVRISDUWLFLSDQWVVHH%R[

Bo x 4.1 F o r whom i s i nfor m at i on i ntend ed?

The AS-PTA has worked in partnership with farmers and local rural organisations in promoting economically and ecologically
sustainable forms of agricultural development. One of the most significant lessons identified in monitoring and evaluating the impact
of their work has been knowing the end use of the data collected.
Initially, the focus of data collections was on the direct measurement of biophysical properties. For example, some of the
indicators identified for measuring ’soil and water conservation’ were degree of soil retention, moisture retention, and organic matter
content. However, these were found to be impossible to measure given the limited human and financial constraints of an
understaffed and under-fundedd NGO, and especially given that these indicators were not considered a priority by farmers.
After discussing for whom and for what purpose the information was intended, it became apparent that such precise scientific
data were unnecessary. The information was intended for farmer-to-farmer extensions and donor reporting, both of which did not
require scientifically valid data. Instead, indicators were chosen that gave a rough estimation based on farmers’ own observations;
consequently, rather than obtaining exact organic matter content, indicators reflected the frequency with which farmers noticed a
significant change as a result of project interventions.
Source: Guijt and Sidersky 1996



 Box 4.2 R ecogn isin g lo cal diff e renc es in in dic ato r de vel opment: AC O R D’s ex pe ri ence in Ma li
ACORD began working with agro-pastoralists in Mali on a project that aimed to regenerate a riverine fodder crop (Panicum bourgou)
along the Niger river. In the process of establishing indicators for crop regeneration, ACORD learned that differences existed within
households and between social groups in terms of how success is measured.
It had been assumed by most people – particularly by field extensionists and external aid agencies working in the area – that
the primary reason that groups were interested in this regeneration project was to ensure adequate fodder for their animals during
the dry season. This was true for many individuals, especially men whose criteria for success would be their ability to offer ACORD
staff a calabash of milk. However, discussion with women revealed that they would judge success by asking the children if they had
drunk more kundou – a sweet drink also made from this grass. Discussions with beneficiaries found this criterion for success to be a
key indicator that would allow rapid appraisal of several objectives of the project: if the kundou was available to children, it would
then indicate that there had been enough to satisfy the needs of animals.
ACORD further learned that differences in criteria not only revealed different priorities between men and women, but also
between women who owned livestock and those who did not, between female-headed and male-headed households, and between
pastoral groups, depending on their levels of sendentarisation and social origins of the household.
Source: Roche 1993: 29–30.

*DWKHULQJGDWD
2QFHLQIRUPDWLRQQHHGVDQGREMHFWLYHVDUHLGHQWLILHGGHWHUPLQLQJKRZWRFROOHFWDQGJDWKHULQIRUPDWLRQLVWKH
QH[W PDMRUVWHS LQWKH 30 ( SURFHVV $ ZLGH YDULHW\ RI WRROV RU WHFKQLTXHV PD\ EH XVHG IRU GDWDJDWKHULQJ
SXUSRVHV 0DQ\ HPSKDVLVH WKDW WKH FKRLFH RI WRROV ZLOO YHU\ PXFK GHSHQG RQ FRQWH[W DQG SURMHFWVSHFLILF
FULWHULD 6HYHUDO NH\ TXHVWLRQV WKDW QHHG WR EH DGGUHVVHG DW WKLV VWDJH LQFOXGH :KHUH FDQ WKH LQIRUPDWLRQ EH
IRXQG" :KLFK WRROV VKRXOG EH XVHG" :KR ZLOO JDWKHU WKH LQIRUPDWLRQ DQG ZKHQ" $ EULHI VXPPDU\ RI WKH
GLIIHUHQWNLQGVRIWRROVDYDLODEOHIRU30 (DQGWKHJHQHUDOJXLGHOLQHVIRUFKRRVLQJDPRQJWKHVHWHFKQLTXHVZLOO
EHGLVFXVVHGODWHULQWKLVVHFWLRQ

'DWDDQDO\VLV
7KLVQH[WVWDJHRI30 (LQYROYHVSURFHVVLQJRUDQDO\VLQJGDWDWKDWKDVEHHQPRQLWRUHGDQGFROOHFWHGDOWKRXJK
LGHDOO\DQDO\VLVVKRXOGWDNHSODFHWKURXJKRXWWKH30 ( F\FOH VHH *RVOLQJ DQG (GZDUGV7UDGLWLRQDOO\
GDWDDQDO\VLVLVRIWHQWDNHQRYHUE\RXWVLGHUVRUE\VWDNHKROGHUVORFDWHGDWKLJKHULQVWLWXWLRQDOOHYHOV7KHLGHDRI
30 (LVWRLQYROYHDWDOOOHYHOVDOOHQGXVHUVDQGVWDNHKROGHUV²LQFOXGLQJEHQHILFLDULHVLQGDWDDQDO\VLV
$WWKLVVWDJHRI30 (VWDNHKROGHUVJURXSVHQJDJHLQFULWLFDOUHIOHFWLRQDQGWKLQNLQJDERXWWKHSUREOHPV
DQGFRQVWUDLQWVWKHVXFFHVVHVDQGRXWFRPHVRIWKHLUHIIRUWVDQGDFWLYLWLHVZKLFKWKH\KDYHXQGHUWDNHQ6FRWW
9LOOLHUVDVXJJHVWVVRPHRIWKHIROORZLQJDUHDVRIGLVFXVVLRQLQDQDO\VLQJLQIRUPDWLRQ

• $QDO\VLQJWKHUHOHYDQFHRISURMHFWRUSURJUDPPHREMHFWLYHVWRWKHQHHGVDQGLQWHUHVWVRIWKHVWDNHKROGHUV
• /HDUQLQJWKHLPSDFWRIWKHDFWLYLWLHVXQGHUWDNHQ
• 5HYLHZLQJWKHSURFHVVE\ZKLFKWKHSURMHFWRSHUDWHGDQGKRZGHFLVLRQVZHUHPDGH


,WLVDUJXHGWKDWSDUWRIHVWDEOLVKLQJ30 (LQFOXGHVGHILQLQJWKHVFRSHRIGDWDDQDO\VLVVSHFLILFDOO\ZKDWLVWREH
DQDO\VHG KRZ DQG E\ ZKRP VHH *RVOLQJ DQG (GZDUGV  3IRKO  +RZ GDWD ZLOO EH DQDO\VHG ZLOO
GHSHQG SULPDULO\ RQ WKH WRROV RU WHFKQLTXHV FKRVHQ IRU JDWKHULQJ GDWD DQG RQ WKH LQIRUPDWLRQ UHTXLUHG E\
SDUWLFLSDQWVIRUDPRUHGHWDLOHGGLVFXVVLRQVHH3IRKO:DGVZRUWK

'RFXPHQWDWLRQUHSRUWLQJDQGVKDULQJ
7KH ILQDO VWDJH RI 30 ( LQFOXGHV GRFXPHQWDWLRQ UHSRUWLQJ DQG VKDULQJ RI LQIRUPDWLRQ JHQHUDWHG IURP WKH
30 (SURFHVV3UHVXPDEO\UHFRUGVRIFROOHFWHGLQIRUPDWLRQDUHNHSWIURPWKHYHU\VWDUWRIWKHSURMHFWLQRUGHU
WR SUHVHQW D
KROLVWLF
 SLFWXUH RYHU WKH SURMHFW OLIHWLPH 6HYHUDO NH\ UHIHUHQFHV DQG PDQXDOV GLVFXVV LQ JUHDWHU
GHWDLOJXLGHOLQHVRQKRZWRGRFXPHQWUHSRUWDQGGLVVHPLQDWH0 (LQIRUPDWLRQVHHIRUH[DPSOH)HXHUVWHLQ
 5XJK  6FRWW9LOOLHUV D 6WHSKHQV  6RPH RI WKH PDLQ FULWHULD IRU GRFXPHQWDWLRQ DQG
UHSRUWLQJLQFOXGHFODULW\VLPSOLFLW\EUHYLW\XVHRIYLVXDOVWLPHOLQHVVIDPLOLDULW\DQGDFFHVVLELOLW\
$WWKLVVWDJHVHYHUDOLPSRUWDQWLVVXHVQHHGWREHFRQVLGHUHGVSHFLILFDOO\SHUWDLQLQJWRWKHRZQHUVKLSDQG
XVHRILQIRUPDWLRQDQGILQGLQJV5XELQDUJXHVWKDWFOHDUUXOHVVKRXOGEHHVWDEOLVKHGRQKRZLQIRUPDWLRQZLOOEH
XVHGDQGGLVVHPLQDWHGHVSHFLDOO\ZKHQVHYHUDOSDUWLHVZLWKGLIIHULQJQHHGVDQGLQWHUHVWVDUHLQYROYHG²
)RUH[DPSOHLQPRVWFDVHVLQIRUPDWLRQLVGLVVHPLQDWHGLQDZULWWHQIRUPDWRUPRUHIRUPDOODQJXDJHZKLFK
PD\ QRW EH DSSURSULDWH IRU UHDFKLQJ ORFDO DXGLHQFHV 7KHUHIRUH D NH\ HPSKDVLV LV SODFHG RQ FRPPXQLFDWLQJ
UHVXOWVDQGILQGLQJVRI30 (LQGLIIHUHQWZD\VUHVSRQGLQJWRHQGXVHUQHHGV)RUPVRIFRPPXQLFDWLRQPD\
LQFOXGHPRUHLQIRUPDOVW\OHVRIUHSRUWLQJQDPHO\DGRSWLQJWKHORFDOODQJXDJHRUXVLQJPRUHYLVXDOWHFKQLTXHV
&21&(51
,Q WKH FDVH RI WKH 05'3 SURJUDPPH LQ 9LHWQDP YLOODJHUV FRPPXQLFDWH WKHLU ILQGLQJV WR KLJKHU
PDQDJHPHQW OHYHOV E\ SUHVHQWLQJ YLVXDOEDVHG LQIRUPDWLRQ LQ WKH IRUP RI JUDSKV WDEOHV PDSV DQG GLDJUDPV
JHQHUDWHGIURPWKHLU35$DFWLYLWLHV6KDKHWDODOVRGHVFULEHKRZIDUPHUVXVHJURXQGPDSVRIWKHLUILHOGV
WRSUHVHQWUHVXOWVRIIDUPH[SHULPHQWVDQGWRVWLPXODWHJURXSDQDO\VLVDQGGLVFXVVLRQ
:KLOHLWLVLPSRUWDQWWRXQGHUVWDQGWKHEDVLF IUDPHZRUN DQG IHDWXUHV RI D 30 ( SURFHVV RQH ZLOO DOVR
QHHGWRNQRZKRZWRIDFLOLWDWHWKLVSURFHVV7KHIROORZLQJJGLVFXVVLRQWXUQVWRWKHPHWKRGV²WKH
WRROV
DQG
WHFKQLTXHV²FRPPRQO\XVHGIRUSXWWLQJ30 (LQWRSUDFWLFH


7RROV
DQG7HFKQLTXHVIRU30 (
7KHUHDUHQRZDZLGHYDULHW\RIWRROVDQGWHFKQLTXHVIRU30 (GLVFXVVHGLQWKHOLWHUDWXUHDQGGHVFULEHGLQD
QXPEHU RI NH\ PDQXDOV DQG JXLGHERRNV RQ 30 ( VHH $SSHQGL[  :KLOH WKHVH WRROV IRU 30 ( KDYH
JHQHUDOO\ EHHQ DSSOLHG IRU GDWDFROOHFWLRQ SXUSRVHV WKHVH WHFKQLTXHV DUH DOVR XVHG IRU SODQQLQJ DQDO\VLV
GRFXPHQWDWLRQDQGUHSRUWLQJVHH)HXHUVWHLQ'DYLV&DVH*RVOLQJDQG(GZDUGV3IRKO
7KH\PD\EHFDWHJRULVHGDVIROORZV

• 35$DQG35$UHODWHGWRROV
• $XGLRYLVXDOWRROV
• 4XDQWLWDWLYHWRROV
• 7RROVGHULYHGIURPWKH
DQWKURSRORJLFDO
WUDGLWLRQ


35$DQG35$UHODWHGWHFKQLTXHV
3DUWLFLSDWRU\DVVHVVPHQW35$DOVRNQRZQDV
SDUWLFLSDWRU\UXUDODSSUDLVDO
DQG
SDUWLFLSDWRU\UDSLGDVVHVVPHQW

FRPSULVHVDUDQJHRIYLVXDOLVDWLRQLQWHUYLHZLQJDQGJURXSZRUNPHWKRGVVHH%R[7KHVHWHFKQLTXHVKDYH
SURYHQYDOXDEOHLQHQDEOLQJSHRSOHWRH[SUHVVWKHLUYLHZVDQGVKDUHLQIRUPDWLRQLQXQFRYHULQJWKHLUUHDOLWLHVDQG
SULRULWLHV DQG LQ VWLPXODWLQJ GLVFXVVLRQ DQG DQDO\VLV  ,'6 3ROLF\ %ULHILQJ  %DVHG RQ RXU VXUYH\ RI
OLWHUDWXUHDQGFDVHVWXGLHV35$PHWKRGVDUHFRPPRQO\DSSOLHGIRUPRQLWRULQJDQGHYDOXDWLRQVHHDOVR)RUG
)RUGHWDO0XNKHUMHHQG2VXJDDQG0XWD\LVD:HOERXUQ

Bo x 4.3 Met hods u sed in p a rtic ipato r y ru ra l app r ais al ( P R A)

Visualised analysis
• Participatory mapping and modelling
• Aerial photograph analyses
• Transect and group walks
• Seasonal calendars
• Daily and activity profiles
• Historical profiles and trend analyses
• Timelines and chronologies
• Matrix scoring and preference ranking
• Venn and network diagramming
• Flow diagrams on systems and impact
• Pie diagrams

Interviewing and sampling methods

• Semi-structured interviewing
• Direct observation
• Focus group
• Key informants
• Ethno-histories
• Futures possible
• Well-being and wealth ranking
• Social maps

Group and team dynamics method

• Team contracts
• Team review sessions
• Interview guides
• Rapid report writing
• Work sharing (taking part in local activities)
• Villager and shared presentations
• Process notes and diaries
Source: Cornwall, Guijt, and Welbourn 1993: 22


'LDJUDPPLQJ
2QHRIWKHNH\VWUHQJWKVRI35$LVLWVHPSKDVLVRQYLVXDOLVDWLRQWHFKQLTXHVVXFKDVGLDJUDPPLQJ&RUQZDOO
*XLMW:HOERXUQ 9LVXDOLVDWLRQ WHFKQLTXHV SURYLGH RSSRUWXQLWLHVIRU FUHDWLYH UHIOHFWLRQ EHFDXVH WKH\
HQDEOHSHRSOHWR
UHSUHVHQWWKHLURZQLGHDVLQDIRUPWKDWWKH\FDQGLVFXVVPRGLI\DQGH[WHQG
LELG$V35$
XVXDOO\WDNHVSODFHLQJURXSVWKLVHQFRXUDJHVZLGHUSDUWLFLSDWLRQIURPSHRSOHDQGDOORZVIRUFURVVFKHFNLQJRI
LQIRUPDWLRQJHQHUDWHG
,Q,QGLDIRUH[DPSOHIDUPHUVGUHZ
JURXQGPDSV
RIWKHLUILHOGVLQRUGHUWRREVHUYHFKDQJHVRYHUWLPHDVD
UHVXOWRIDGRSWLQJVRLODQGZDWHUFRQVHUYDWLRQPHDVXUHV6KDKHWDO7KHVHJURXQGPDSVDOORZHGIDUPHUV
WR FRPSDUH
EHIRUHDQGDIWHU
 FRQGLWLRQV RQ WKHLU ILHOGV DQG VXEVHTXHQWO\ GHYHORS PRUH HIIHFWLYH SURGXFWLRQ
VWUDWHJLHV $OVR LQ 9LHWQDP WKH 05'3 SURJUDPPH HPSOR\HG D GLYHUVH UDQJH RI 35$ WHFKQLTXHV LQFOXGLQJ
PDSSLQJ DQG GLDJUDPPLQJ IRU PRQLWRULQJ DQG HYDOXDWLQJ YLOODJH DFWLYLWLHV 05'3  )LQDOO\ VNHWFKLQJ
9HQQ LQVWLWXWLRQDO GLDJUDPPLQJ DQG RWKHU 35$ PHWKRGV ZHUH XVHG LQ .HQ\D WR DVVHVV DQG VWUHQJWKHQ
RUJDQLVDWLRQDO FDSDFLWLHV RI D JRYHUQPHQW DJHQF\ LQ SURPRWLQJ VRLO DQG ZDWHU FRQVHUYDWLRQ 7KRPSVRQ DQG
3UHWW\
$QRWKHU IRUP RI GLDJUDPPLQJ LV WKH XVH RI SLFWXUHV LQ MRXUQDOV DQG GLDULHV :KLOH ZULWWHQ MRXUQDOV LH
UHFRUGNHHSLQJUHSRUWVHWFDUHFRPPRQSLFWXUHMRXUQDOVDUHDXQLTXHLQQRYDWLRQIRUUHFRUGLQJLQIRUPDWLRQ
2QH H[SHULHQFH LQ ,QGLD GHVFULEHV KRZ ZRPHQ XVH SLFWRULDO GLDULHV WR NHHS WUDFN RI WKHLU JURXS VDYLQJV DQG
FUHGLWSHUIRUPDQFHVHH%R[:RPHQLGHQWLI\WKHLUQHHGVDQGDVVHVVWKHLUVRFLRHFRQRPLFVWDWXVE\PDUNLQJ
FKRLFHVXVLQJIDPLOLDUSLFWXUHVDQGV\PEROV'LDULHVDUHGHVLJQHGWREHVLPSOHXVHUIULHQGO\DQGYLVXDOEDVHG
ZKLFKILWWKHQHHGVRISRRUZRPHQZKRODFNOLWHUDF\DQGQXPHUDF\VNLOOV

$XGLRYLVXDOWHFKQLTXHV
,Q DGGLWLRQ WR 35$ WKHUH DUH RWKHU WRROV WKDW VSHFLILFDOO\ XVH DXGLRYLVXDO WHFKQLTXHV IRU PRQLWRULQJ DQG
HYDOXDWLRQ 7KHVH DUH GLVFXVVHG LQ JUHDWHU GHWDLO LQ WKH OLWHUDWXUH VHH &RUQZDOO *XLMW DQG :HOERXUQ 
)HXHUVWHLQ*RVOLQJDQG(GZDUGV.UDPVMRDQG:RRG6RPHRIWKHPDUHOLVWHGKHUH

• 8VHRIYLGHRV
• 6WRU\WHOOLQJ
• 3RSXODUWKHDWUH
• 6RQJV
• 3KRWRYRLFH

6HYHUDO FDVH VWXGLHV GHVFULEH WKH DSSOLFDWLRQ RI DXGLRYLVXDO WHFKQLTXHV IRU 30 ( )RU H[DPSOH LQ WKH
SDUWLFLSDWRU\ LPSDFW HYDOXDWLRQ RI WKH 1HSDO 5HVRXUFH 0DQDJHPHQW 3URMHFW GUDPD SHUIRUPDQFHV DQG VRQJV
ZHUHXVHGWRKHOSLGHQWLI\YLOODJHUV
SHUFHSWLRQVRIFKDQJHVLQWKHLUFRPPXQLWLHVDVDUHVXOWRIWKHSURMHFW7KHVH
WRROV DOVR KHOSHG YLOODJHUV GHILQH ZKDW VKRXOG EH HYDOXDWHG DQG ZKDW WKH\ SULRULWLVH DV WKH REMHFWLYHV RI WKH
SURMHFW:LJEROGXV
$PRUHUHFHQWLQQRYDWLRQLVWKHXVHRI
SKRWRYRLFH
IRU30 (DVGHVFULEHGE\:DQJ<DQ/LQJDQG0LQJ
/LQJ  ,Q WKLV SURFHVV ORFDO SHRSOH WKHPVHOYHV SURGXFH YLVXDO LPDJHV WKURXJK WKH XVH RI DQ LQVWDPDWLF


FDPHUD 7KH SKRWRJUDSKV WKHQ VHUYH DV D FDWDO\VW WR GHSLFW UHIOHFW RQ DQG GLVFXVV VRFLDO FRQGLWLRQV DIIHFWLQJ
SHRSOH
VOLYHV,QUXUDO&KLQDYLOODJHZRPHQXVHGSKRWRYRLFHDVDWRROWRWDONDERXWWKHLUVRFLDOVWDWXVDQGOLYLQJ
FRQGLWLRQVDQGWRVKRZWKHH[WHQWWRZKLFKGHYHORSPHQWSURJUDPPHVEHQHILWHGWKHLUFRPPXQLW\

4XDQWLWDWLYHWHFKQLTXHV
:KLOHTXDQWLWDWLYHPHWKRGVDUHWUDGLWLRQDOO\DVVRFLDWHGZLWKPRUHFRQYHQWLRQDOQRQSDUWLFLSDWRU\DSSURDFKHVWR
0 (PRUHUHFHQWDWWHPSWVKDYHWULHGWRPDNHWKHVHPHWKRGVPRUHSDUWLFLSDWRU\DQGDFFHVVLEOHWRORFDOSHRSOH
6RPH RI WKHVH TXDQWLWDWLYH WHFKQLTXHV ZKLFK KDYH EHHQ DGDSWHG WR ILW D SDUWLFLSDWRU\ SURFHVV LQFOXGH 
FRPPXQLW\VXUYH\VDQGPHWKRGVXVHGIRUHFRORJLFDODVVHVVPHQW

&RPPXQLW\VXUYH\V
)HXHUVWHLQ GHVFULEHV KRZVXUYH\V FRXOG EH GHYHORSHG DQG FDUULHG RXW ZLWK FRPPXQLW\ LQYROYHPHQW LQ
RUGHUWRILWORFDOQHHGVDQGFRQWH[WV,QFRPPXQLW\VXUYH\VORFDOPHPEHUVVKRXOGEHLQYROYHGLQGHFLGLQJZKDW
LQIRUPDWLRQ WR REWDLQ KRZ DQG ZKHQ WR REWDLQ LQIRUPDWLRQ DQG IRU ZKDW SXUSRVH 6RPH H[DPSOHV RI
FRPPXQLW\EDVHG VXUYH\ IRUPV DUH GHVFULEHG E\ 5XJK  7KHVH LQFOXGH WKH
+HDOWK +DSSHQLQJV VXUYH\
IRUPZKLFKFDQEHXVHGE\SUHOLWHUDWHKHDOWKYROXQWHHUVWRNHHSUHFRUGVRIKHDOWKUHODWHGLQIRUPDWLRQLQWKHLU
FRPPXQLWLHV $QRWKHU H[DPSOH LV WKH
IDUPHU
V RZQ UHFRUG
 IRUP ZKLFK KHOSV IDUPHUV WHVW QHZ LGHDV DQG
FRQGXFWRQIDUPH[SHULPHQWV
,QWKH6LDYRQJD$JULFXOWXUDO'HYHORSPHQW3URMHFWLQ=DPELDIDUPHUVWKHPVHOYHVGHVLJQWKHLURZQUHFRUG
VKHHWVIRUPRQLWRULQJ DQG FRPSDULQJ GLIIHUHQWIDUPLQJ SUDFWLFHV7KHVH UHFRUGV VHUYH WRIDFLOLWDWH GLVFXVVLRQV
EHWZHHQILHOGH[WHQVLRQLVWVDQGIDUPHUVDQGKHOSWRPDNHIDUPHUV
QHHGVDQGLQWHUHVWVPRUHWUDQVSDUHQW1DJHO
HWDO

(FRORJLFDODVVHVVPHQWV
(FRORJLFDO DVVHVVPHQWV KDYH DOVREHHQ DGDSWHG WR LQWHJUDWH D PRUH SDUWLFLSDWRU\ DSSURDFK ,Q WKHLU RYHUYLHZ
$EERWDQG*XLMWGHVFULEHKRZORFDOSHRSOHDUHOHDUQLQJKRZWRXVHVWDQGDUGHFRORJLFDOPHWKRGVWRPDNH
DQLQYHQWRU\RIDQGPRQLWRUDQLPDODQGSODQWSRSXODWLRQV$OWKRXJKWKHDXWKRUVUHFRJQLVHWKDWLQVRPHFDVHV
ORFDO SHRSOH UHPDLQ H[FOXGHG IURP WKH UHVHDUFK GHVLJQ DQG GDWD DQDO\VLV GXH ODUJHO\ WR WKH VWDWLVWLFDO
UHTXLUHPHQWVRIWKHHFRORJLFDODVVHVVPHQWPHWKRGVXVHGDWWHPSWVWRHQVXUHWKDWSDUWLFLSDQWVXQGHUVWDQGHDFK
VWDJHRIWKHUHVHDUFKSURFHVVKHOSEXLOGORFDOFDSDFLWLHVWRFROOHFWGDWDDQGDQDO\VHHQYLURQPHQWDOFKDQJH
$QRWKHUSURPLVLQJDSSURDFKLVELRUHVRXUFHIORZGLDJUDPV%XLOGLQJRQSDUWLFLSDWRU\PDSSLQJDQGDJUR
HFRV\VWHP DQDO\VLV ELRUHVRXUFH IORZ GLDJUDPV KHOS IDUPHUV DVVHVV FXUUHQW IDUP FRQGLWLRQV DQG SODQ KRZ WR
PRQLWRUH[SHULPHQWDOFKDQJHVLQWKHLUIDUPLQJV\VWHPV83:$5',QRQHVWXG\LQWKH3KLOLSSLQHV
IDUPHUVGHVLJQHGFRQFHSWXDOPRGHOVRIWKHLURZQIDUPLQJV\VWHPVZKLFKWKH\XVHGWRUHFRUGUHVRXUFHIORZV
IDUP LQSXWV DQG RXWSXWV /LJKWIRRW 'DOVJDDUG DQG %LPEDR  )DUPHUV XVHG WKHVH PRGHOV WR KHOS WKHP
PRQLWRUFKDQJHVRQWKHLUIDUPVDQGLPSURYHUHVRXUFHPDQDJHPHQWDQGH[SHULPHQWDWLRQVNLOOVLELG


7RROVIURPWKH
DQWKURSRORJLFDO
WUDGLWLRQ
2WKHU 30 ( WHFKQLTXHV KDYH DOVR HYROYHG IURP WKH WUDGLWLRQ RI DQWKURSRORJ\ 7KHVH LQFOXGH SDUWLFLSDQW
REVHUYDWLRQDQGWKHXVHRIRUDOWHVWLPRQLHV

3DUWLFLSDQWREVHUYDWLRQ
LVDWHUPFRPPRQO\ XVHG E\ DQWKURSRORJLVWV WR GHVFULEH REVHUYDWLRQ WHFKQLTXHV
ZKHQVWXG\LQJWKHOLIHVW\OHDQGEHKDYLRXUVRIORFDOFRPPXQLWLHVLQGLIIHUHQWFXOWXUDOFRQWH[WVZKLOHWDNLQJSDUW
LQ GD\WRGD\ ORFDO DFWLYLWLHV ,Q WKH FRQWH[W RI 30 ( SDUWLFLSDQW REVHUYDWLRQ VKRXOG WDNH SODFH ZLWK WKH
NQRZOHGJH RI WKH SHRSOH EHLQJ REVHUYHG DQG PD\ HYHQ EH XQGHUWDNHQ DV D JURXS DFWLYLW\ ZLWK ORFDO SHRSOH
WKHPVHOYHVIXQFWLRQLQJDVSDUWLFLSDQWREVHUYHUV)HXHUVWHLQ
2UDO WHVWLPRQLHV FDQ DOORZ SHRSOH WR DUWLFXODWH WKHLU RZQ SHUVSHFWLYHV DQG SUHVHQW WKHLU RZQ DFFRXQWV
DERXW WKH KLVWRU\ RI D SODFH RU D SDUWLFXODU HYHQW RU DERXW WKHLU RZQ OLYHV 3HUVRQDO WHVWLPRQLHV KHOS EXLOG D
SLFWXUH RI ZKDW KDV KDSSHQHG RYHU WLPH RU LOOXPLQDWH SUREOHPV GLIIHULQJ SHUVSHFWLYHV DQG LQWHUHVWV IRU
GLVFXVVLRQ*RVOLQJ DQG (GZDUGV  VHH DOVR 5XJK  6OLP DQG 7KRPSVRQ  )RU H[DPSOH LQ WKH
1HSDO5HVRXUFH0DQDJHPHQW3URMHFWRUDOKLVWRULHVRIYLOODJHUVSUHVHQWHGORFDODFFRXQWVRIWKHSURMHFW
VLPSDFW
ZKLFKUHIOHFWHGYDULHGH[SHULHQFHVDQGFKDQJHVWKDWRFFXUUHGLQGLIIHUHQWFRPPXQLWLHVDVDUHVXOWRIWKHSURMHFW
:LJEROGXV

&KRRVLQJ30 (7RROVDQG7HFKQLTXHV
*LYHQ WKH UDQJH RI WRROV FRQVLGHUHG SRWHQWLDOO\ XVHIXO IRU 30 ( VRPH FULWHULD VKRXOG EH NHSW LQ PLQG IRU
GHFLGLQJ ZKLFK WHFKQLTXHV ZRXOG EH PRVW DSSURSULDWH *XLGHOLQHV IRU FKRRVLQJ WKH PRVW DSSURSULDWH DQG
UHOHYDQWWHFKQLTXHVIRU30 (DUHSURYLGHGLQWKHOLWHUDWXUHVHH&21&(51)HXHUVWHLQ*RVOLQJ
DQG (GZDUGV  1DUD\DQ3DUNHU  5XJK  6FRWW9LOOLHUV D ,Q JHQHUDO WRROV DQG WHFKQLTXHV
VKRXOG

• &RPSOHPHQWWKHDSSURDFKDQGSKLORVRSK\RIWKHSURMHFW
• %HSHUFHLYHGE\FRPPXQLW\SDUWLFLSDQWVDVDZD\WRKHOSWKHPDGGUHVVWKHLUTXHVWLRQVDQGSUREOHPVQRW
VLPSO\DVLQIRUPDWLRQDERXWWKHPJDWKHUHGE\RUIRURXWVLGHUV
• ,QYROYHHQGXVHUVLQERWKGDWDJDWKHULQJDQGLQDQDO\VLQJGDWD
• 0DWFKWKHVNLOOVDQGDSWLWXGHVRISDUWLFLSDQWV
• $GDSWWRILWSHRSOHV
GD\WRGD\DFWLYLWLHVDQGQRUPDOUHVSRQVLELOLWLHV
• 3URYLGHWLPHO\LQIRUPDWLRQQHHGHGIRUGHFLVLRQPDNLQJ
• 3URGXFHUHVXOWVZKLFKDUHUHOLDEOHDQGHYHQLIQRWTXDQWLWDWLYHFUHGLEOHHQRXJKWRFRQYLQFHRWKHUV
• %HFRQVLVWHQWLQFRPSOH[LW\DQGFRVWWRPDWFKWKHOHYHORIHYDOXDWLRQFDOOHGIRUHJVLPSOHURXWLQHYHUVXV
PRUHFRPSUHKHQVLYHPDMRUHYDOXDWLRQV
• 5HLQIRUFHFRPPXQLW\VROLGDULW\FRRSHUDWLRQDQGLQYROYHPHQW
• %HJHQGHUVHQVLWLYHZLWKVSHFLDOHIIRUWVWRLQFOXGHZRPHQ
• 2QO\REWDLQWKHLQIRUPDWLRQQHHGHG
DGDSWHGIURP5XJKDQG1DUD\DQ3DUNHU²


'HSHQGLQJRQWKHPHWKRGVFKRVHQHDFKWRROZLOOJHQHUDWHSDUWLFXODUNLQGVRILQIRUPDWLRQDQGUHTXLUHGLIIHUHQW
UHVRXUFHV DQG VNLOOV  3IRKO   5XJK   +HQFH PDQ\ XVXDOO\ UHFRJQLVH WKH LPSRUWDQFH RI
FRPELQLQJ GLIIHUHQW PHWKRGV GHSHQGLQJ RQ WKH LQIRUPDWLRQ UHTXLUHG WKH SXUSRVH RI WKH H[HUFLVH DQG WKH
DYDLODELOLW\ RI UHVRXUFHV $EERW DQG *XLMW  &21&(51  *RVOLQJ DQG (GZDUGV  3IRKO 
6FRWW9LOOLHUV D &RPELQLQJ WHFKQLTXHV LV FRQVLGHUHG XVHIXO IRU GDWD
WULDQJXODWLRQ
 ZKLFK DOORZV
LQIRUPDWLRQWREHFRPSDUHGFRQILUPHGUHIXWHGRUVXEVWDQWLDWHGLELG
2IFRXUVHWKH30 (SURFHVVGRHVQRWHQGZLWKWKHLQIRUPDWLRQJDWKHULQJSURFHVV(TXDOO\LPSRUWDQWDUH
WKHVWHSVRIDQDO\VLQJDQGDVVHVVLQJWKHILQGLQJVUHSRUWLQJDQGXVLQJWKHILQGLQJVIRUDFWLRQLPSURYHPHQWDQG
FKDQJH OHDUQLQJ DQG UHIOHFWLQJ RQ QH[W VWHSV HYDOXDWLQJ WKH SURFHVV UHFRJQLVLQJ DQG FHOHEUDWLQJ WKH
DFFRPSOLVKPHQWVPDGHVHHIRULQVWDQFHWKH/HDUQLQJ&\FOHGHYHORSHGE\WKH&RPPXQLW\3DUWQHUVKLS&HQWHU
WKH8QLYHUVLW\RI7HQQHVVHHS7KURXJKRXWHDFKVWDJHRI30 (GHFLGLQJZKRVKRXOGEHLQYROYHG
DQG RQ ZKDW WHUPV LV SHUKDSV WKH PRVW FULWLFDO DFWLYLW\ IRU VXVWDLQLQJ WKH 30 ( SURFHVV 7KH QH[W VHFWLRQ
H[SORUHVVRPHRIWKHVHSUREOHPDWLFLVVXHVDQGSRLQWVWRFKDOOHQJHVIRULPSURYLQJWKHSUDFWLFHRI30 (

&+$//(1*(6$1',668(6)25)857+(5',6&866,21$1')8785(
5(6($5&+
&OHDUO\ WKHUH DUH QR VLQJOH IRUPXOD RU IL[HG JXLGHOLQHV WKDW H[SODLQ
ZKDW 30 ( LV
 DQG
KRZ WR GR 30 (

1HYHUWKHOHVVPXFKRIWKHOLWHUDWXUHDQGFDVHVWXGLHVUHYLHZHGKHUHGHPRQVWUDWHDFRQFHUWHGHIIRUWWRGHYHORS
DQGFRQWULEXWHWRZDUGVDPRUHFRKHUHQWERG\RINQRZOHGJHDERXWWKHSUDFWLFHRI30 (,QWKLVFRQWH[WLGHDV
DQGDFWXDOH[SHULHQFHVRI30 (DUHJUDGXDOO\HPHUJLQJZKLFKSURYLGHZLGHVFRSHIRUPRUHFUHDWLYHWKLQNLQJ
DQGOHDUQLQJ+RZHYHUWKLVSURFHVVRIH[SHULPHQWDOOHDUQLQJLVUDUHO\DWURXEOHIUHHXQGHUWDNLQJ&21&(51
  DQG D QXPEHU RI FRPPRQ FRQFHUQV DQG SUHVVLQJ LVVXHV DUH DOUHDG\ EHLQJ UDLVHG ZLWK UHVSHFW WR
LPSURYLQJWKHSUDFWLFHRI0 (
7KLV VHFWLRQ KLJKOLJKWV VHYHUDO RI WKHVH EURDG FKDOOHQJHV DQG LVVXHV IURP WKH OLWHUDWXUH DQG FDVH VWXGLHV
UHYLHZHG :H WKXV KRSH WR VWLPXODWH IXUWKHU GLVFXVVLRQ DQG SRLQW RXW SRWHQWLDO DUHDV IRU IXWXUH UHVHDUFK DQG
DFWLRQ7KHLVVXHVWREHH[SORUHGLQFOXGH

• 8QGHUVWDQGLQJ30 (DVDVRFLDOSURFHVVLQFOXGLQJWKHUROHRISRZHUFRQIOLFWDQGFRQIOLFWUHVROXWLRQ
DVZHOODVWUDQVODWLQJSDUWLFLSDWLRQLQWRSUDFWLFH
• 0HWKRGRORJLFDOLVVXHVLQFOXGLQJGHYHORSLQJQHZVWDQGDUGVRI
ULJRXU
FKRRVLQJDSSURSULDWHLQGLFDWRUV
DQGWKHUROHRIWKHIDFLOLWDWRU
• ,QVWLWXWLRQDOLVDWLRQRI30 (LQFOXGLQJLVVXHVRIUHVRXUFHVDQGVFDOLQJXS
• 7KHQHHGIRUPRUHGRFXPHQWDWLRQRI30 (H[SHULHQFHVLQFOXGLQJWKHQHHGIRUSDUWLFLSDWRU\
PRQLWRULQJRI30 (LGHQWLI\LQJHQDEOLQJIDFWRUVDQGGRFXPHQWLQJRXWFRPHV

8QGHUVWDQGLQJ30 ($VD6RFLDO3URFHVV
7KH FRQFHSW RI 30 ( DV D VRFLDOO\ QHJRWLDWHG SURFHVV ² QRW VLPSO\ DV D PHWKRG RU VHW RI WHFKQLTXHV ² LV
JHQHUDOO\UHFRJQLVHGLQWKHOLWHUDWXUH,QWKLVFRQWH[W30 (LVYLHZHGDVDSURFHVVZKHUHE\GLIIHUHQWJURXSVRI


VWDNHKROGHUV DUWLFXODWH DQG SUHVHQW WKHLU RZQ YLHZV QHHGV DQG H[SHFWDWLRQV 7KLV SURFHVV LV JHQHUDOO\
FKDUDFWHULVHGDVDFROODERUDWLYHGHFLVLRQPDNLQJHQGHDYRXUZKLFKWDNHVLQWRDFFRXQWWKHPXOWLSOHREMHFWLYHVDQG
LQWHUHVWVRIVWDNHKROGHUVEXWZLWKWKHRYHUDOODLPRIUHDFKLQJJURXSFRQVHQVXV

7KHUROHRISRZHU
,Q VSLWH RI WKLV JHQHUDO UHFRJQLWLRQ RI 30 ( DV D VRFLDO SURFHVV RI QHJRWLDWLRQ WKHUH LV D QHHG IRU D PRUH
H[SOLFLW DQG RSHQ GLVFXVVLRQ DERXW WKH FRPSOH[ VRFLDO G\QDPLFV DQG SRZHU UHODWLRQVKLSV WKDW XQGHUOLH DQG
FRQVWLWXWH WKH DFWXDO SUDFWLFH RI 30 ( 0DUVGHQ DQG 2DNOH\   SRLQW RXW WKDW
%HKLQG VXFK D
QHJRWLDWLRQDUHTXHVWLRQVDERXWRZQHUVKLSDQGFRQWURORINQRZOHGJHDQGWKHXVHV WR ZKLFK LW PLJKW EH SXW

5XELQ   IXUWKHU REVHUYHV WKDW WKHUH LV

D QHHG IRU JUHDWHU FODULW\ EHWZHHQ DOO WKH SDUWLHV LQYROYHG
UHJDUGLQJKRZE\ZKRPDQGDJDLQVWZKDWFULWHULD>GHYHORSPHQW@ZRUNLVJRLQJWREHPHDVXUHG
+HQFHDNH\
TXHVWLRQWKDWLVUDLVHGSHUWDLQVWRLVVXHVRISRZHUZKRFRQWUROVDQGZKRLQIOXHQFHVWKH0 (SURFHVV"
$GGUHVVLQJWKHLVVXHRISRZHULQ30 (LPSOLHVUHFRJQLVLQJXQHTXDOVRFLDOUHODWLRQVKLSVDQGSRVLWLRQV²
HLWKHUEHWZHHQGLIIHUHQWDFWRUVVWDNHKROGHUVLQFOXGLQJWKHIDFLOLWDWRURUDWGLIIHUHQWLQVWLWXWLRQDOOHYHOVYLOODJH
SURJUDPPHRUSROLF\OHYHOV²WKDWXQGHUOLHWKH30 (SURFHVVLWVHOI,QWKHLURYHUYLHZ$UPRQLDDQG&DPSLODQ
SRLQWRXWWKDWLQYROYLQJDJUHDWHUQXPEHURIDFWRUVLQ30 (PD\KDYHLPSOLFDWLRQVIRUFKDQJLQJRU
UHLQIRUFLQJ SRZHU UHODWLRQV DPRQJ WKHP HVSHFLDOO\ ZLWK UHVSHFW WR WKH LQIOXHQFH ZKLFK GRQRUV DQG
LPSOHPHQWLQJDJHQFLHVWUDGLWLRQDOO\KROGRYHUWKH0 (SURFHVV7KHFULWLFDOLVVXHLVWRGHWHUPLQHWRZKDWH[WHQW
WKHVH WUDGLWLRQDOO\ PRUH LQIOXHQWLDO VWDNHKROGHU JURXSV ZRXOG ZLOOLQJO\ VKDUH WKLV SULYLOHJHG VWDWXV ZLWK RWKHU
SDUWLFLSDQWVVSHFLILFDOO\ZLWKORFDOSHRSOH
5XELQ   PDNHV D VLPLODU SRLQW LQ TXHVWLRQLQJ ZKR FKRRVHV WKH FULWHULD IRU GHWHUPLQLQJ

VXFFHVV
6KHQRWHVWKDWDJUHDWGHDORIRSHQQHVVDQGWUDQVSDUHQF\LVUHTXLUHGEHWZHHQIXQGHUVDQGWKHIXQGHG
LQQHJRWLDWLQJZKLFKFULWHULDZLOOEHXVHGIRUGHWHUPLQLQJFKDQJHDQGSURJUHVV0RUHH[SORUDWLRQWKHUHIRUHLV
QHHGHGRIWKHPHDQVE\ZKLFKSDUWLFLSDQWVDWWKHJUDVVURRWVYLOODJHOHYHOZRXOGEHDEOHWRVWDWHZKDW
PHDVXUHV
WKH\ZLOOXVHWRMXGJHWKHSURJUHVVRISURMHFWVGHVLJQHGWREHQHILWWKHP
5XELQ²
+RZHYHU$EERW DQG *XLMW  SRLQWRXW WKHQHHG WR UHFRJQLVH QRW RQO\ GLIIHUHQFHV LQ LQIOXHQFH DQG
VWDWXV EHWZHHQ VWDNHKROGHUV
 JURXSV EXW DOVR ZLWKLQ VWDNHKROGHU JURXSV WKHPVHOYHV $V KLJKOLJKWHG LQ
$&25'
VH[SHULHQFHLQ0DOLLQGLFDWRUVIRUPRQLWRULQJSURMHFWLPSDFWVGLIIHUHGDPRQJFRPPXQLW\PHPEHUV
DFFRUGLQJWRJHQGHULQGLYLGXDODQGKRXVHKROGVWDWXV5RFKH$OVRZKLOHYLOODJHUVLQ9LHWQDPDVVXPHD
SDUDPRXQWUROHLQGHVLJQLQJSODQQLQJDQGLPSOHPHQWLQJ0 (DFWLYLWLHVWKH\DOVRDFNQRZOHGJHWKDWWKHSURFHVV
KDVWHQGHGWRH[FOXGHPDUJLQDOLVHGJURXSVLHSRRUHUKRXVHKROGVDQGDUHWKHUHIRUHWDNLQJWKLVLQWRDFFRXQWLQ
SODQQLQJIXWXUHDFWLYLWLHV6FRWW9LOOLHUVDSHUVFRPP,QDGGLWLRQ.DUHWDOESRLQWRXWLQWKH
&DOFXWWDFDVHWKDWWKHSUREOHPRIGRPLQDWLRQE\FHUWDLQVWDNHKROGHUJURXSV²HVSHFLDOO\E\WKHPRUHHGXFDWHG
DQGSROLWLFDOO\FRQQHFWHGSDUWLFLSDQWV²KDGWREHUHVROYHGE\
SOHDGLQJZLWKSHRSOHWROHDYHWKH>3,$@H[HUFLVH
RQWKHJURXQGVRIQHHGLQJWRGLVFXVVXUJHQWDQGSULYDWHLVVXHVZLWKRWKHUV



7KHUROHRIFRQIOLFWDQGFRQIOLFWUHVROXWLRQ
,WVKRXOGSHUKDSVEHVHOIHYLGHQWWKDWDVQHZYRLFHVDQGQHZPHDVXUHVRIVXFFHVVHQWHUDQHYDOXDWLRQSURFHVV
FRQIOLFWPLJKWDOVRHPHUJHRYHU
ZKRVHUHDOLW\FRXQWV
&KDPEHUV$VFRQIOLFWHPHUJHVWKHUHLVDQHHGWR
EHWWHUXQGHUVWDQGLWVEDVLVDVZHOODVSURFHVVHVIRUFRQIOLFWQHJRWLDWLRQ
7KH HPSKDVLV RQ SDUWLFLSDWLYH FRQVXOWDWLRQ DQG FROODERUDWLRQ EHWZHHQ VWDNHKROGHU JURXSV LQ 30 (
OLWHUDWXUH OHDGV XV WR EHOLHYH WKDW JURXS FRQVHQVXV LV UHDGLO\ DFKLHYHG LQ WKH QHJRWLDWLRQ SURFHVV RI 30 (
+RZHYHUWKHUHDOSRWHQWLDOIRUFRQIOLFWDQGPHFKDQLVPVIRUUHVROYLQJFRQIOLFWLVVHOGRPH[SOLFLWO\GLVFXVVHGLQ
WKHOLWHUDWXUHDQGFDVHVWXGLHVUHYLHZHG)RULQVWDQFHLQUHIHUHQFHWRWKH6$'3DJULFXOWXUDOH[WHQVLRQSURMHFWLQ
=DPELD1DJHOHWDOLQFOXGHFRQIOLFWDQDO\VLVDVSDUWRIWKH30 (SURFHVV
WRLGHQWLI\SRWHQWLDODUHDVRI
FRQIOLFWDWDOOOHYHOV
\HWQRFRQFUHWHSURSRVDOVRUSURFHGXUHVDUHSURYLGHGRQKRZWRVROYHRUDOOHYLDWHWKHVH
SUREOHPV%DVHGRQWKHLUVXUYH\RIH[SHULHQFHVLQ%ROLYLD$OFRFHU/L]iUUDJD'HOJDGLOORHWDOREVHUYHWKDWRQO\
LQ VRPH FDVHV LV WKHUH H[SOLFLW UHFRJQLWLRQ RI FRQIOLFW ZKLFK VRPH ZRXOG DUJXH UHIOHFWV WKH
PRUH SROLWLFDO

GLPHQVLRQ RI SDUWLFLSDWLRQ 7KH DXWKRUV HPSKDVLVH WKH QHHG WR DUULYH DW D GHHSHU XQGHUVWDQGLQJ RI FRQIOLFW
HVSHFLDOO\ZLWKLQWKHFRQWH[WRIGHFHQWUDOLVDWLRQDQGOHJDOUHIRUPVWKDWVXSSRUWSDUWLFLSDWLRQ$OFRFHU/L]iUUDJD
'HOJDGLOORHWDO²
$QH[FHSWLRQLVWKHFDVHVWXG\LQ%UD]LOZKLFKGHVFULEHSRWHQWLDOFRQIOLFWVEHWZHHQVWDNHKROGHUJURXSV²
WKH 1*2 $637$ UXUDO ZRUNHU
V WUDGH XQLRQ UHSUHVHQWDWLYHV ORFDO IDUPHUV ² DQG WKH PHDQV IRU WKHLU
UHVROXWLRQ ,Q WKLV FDVH GLIIHUHQFHV LQ SULRULWLHV DQG QHHGV E\ WKH GLIIHUHQW VWDNHKROGHUV PDGH LW GLIILFXOW WR
GHFLGHKRZEHVWWRPRQLWRULQGLFDWRUV:KLOHVWDNHKROGHUVHYHQWXDOO\DJUHHGWKURXJKMRLQWGLVFXVVLRQVWRXVHOHVV
TXDQWLWDWLYHPHDVXUHVRQHZD\RIUHVROYLQJWKLVFRQIOLFWZDVIRUWKH1*2VWDIIWRUHFRJQLVHWKDWWKH\VKRXOG
WDNH RQ WKH UHVSRQVLELOLW\ RI PRQLWRULQJ WKHLU RZQ LQGLFDWRUV LI WKH\ ZDQWHG WR REWDLQ PRUH VFLHQWLILF DQG
ULJRURXVGDWDWRIXOILOWKHLURZQLQIRUPDWLRQQHHGVVHH*XLMWDQG6LGHUVN\

7UDQVODWLQJSDUWLFLSDWLRQLQWRSUDFWLFH
:KLOH RQH RI WKH FRUH SULQFLSOHV RI 30 ( LV LWV HPSKDVLV RQ SDUWLFLSDWLRQ WKHUH LV VWLOO JUHDW DPELJXLW\ LQ
WUDQVODWLQJ WKH FRQFHSW DQG REMHFWLYHV RI SDUWLFLSDWLRQ LQWRSUDFWLFH 6HYHUDO RYHUYLHZV RQ 30 ( PDNH WKHVH
REVHUYDWLRQV)RUH[DPSOH$UPRQLDDQG&DPSLODQSRLQWRXWWKDWWKHLQWHUSUHWDWLRQRI
SDUWLFLSDWLRQ

YDULHV ZLGHO\ DPRQJ WKH FDVH VWXGLHV RI $VLDQ H[SHULHQFHV UHYLHZHG DQG WKDW WKHUH LV QR DFFHSWHG PLQLPXP
VWDQGDUG ZKHQ 0 ( PD\ TXDOLI\ DV SDUWLFLSDWRU\ $OVR LQ WKHLU VXUYH\ RI GRQRU DJHQF\ GRFXPHQWV DQG
H[SHULHQFHVZLWKSDUWLFLSDWRU\HYDOXDWLRQV5XGTYLVWDQG:RRGIRUG%HUJHUDUJXHWKDWSDUWLFLSDWLRQLV
RIWHQGHILQHGYHU\JHQHUDOO\RUWDNHQWRPHDQDUDQJHRIVWDNHKROGHUUROHV,QJHQHUDOWKHUHLVDUHFRJQLVHGQHHG
WRFODULI\DQGUHILQHWKHFRQFHSWRI
SDUWLFLSDWLRQ
²VSHFLILFDOO\ZLWKUHVSHFWWRWKHUROHVRIGLIIHUHQWVWDNHKROGHUV
ZLWKLQ DQG RXWVLGH FRPPXQLWLHV DQG WKH OHYHOV DQG GHJUHHV RI SDUWLFLSDWLRQ DW GLIIHUHQW VWDJHV RI 30 (
WKURXJKRXW WKH SURMHFW F\FOH $EERW DQG *XLMW  $UPRQLD DQG &DPSLODQ  5XGTYLVW DQG :RRGIRUG
%HUJHUVHHDOVR%URZQ0DUVGHQDQG2DNOH\83:$5'
+HQFH WKH H[WHQW WR ZKLFK SDUWLFLSDWLRQ LQ 0 ( LV DFKLHYHG LQ SUDFWLFH UHPDLQV D FRQWHVWHG LVVXH )RU
LQVWDQFH0DUVGHQ2DNOH\DQG3UDWWSRLQWRXWWKDWWKHUHLVFXUUHQWO\OLWWOHGRFXPHQWHGH[SHULHQFHRI
SURJUDPPHV DQG SURMHFWV WKDW GHPRQVWUDWH D WUXO\ SDUWLFLSDWRU\ DSSURDFK WR HYDOXDWLRQ 3DUDFKLQL DQG 0RWW
IXUWKHUQRWHWKDWEXLOGLQJSDUWQHUVKLSVDPRQJSURMHFWVWDNHKROGHUVLVVWLOOYHU\PXFKDQH[FHSWLRQUDWKHUWKDQ


WKHUXOHLQFRPPXQLW\EXLOGLQJLQLWLDWLYHV²5XGTYLVWDQG:RRGIRUG%HUJHU
VDVVHVVPHQW
RI GRQRU H[SHULHQFHV ZLWK SDUWLFLSDWRU\ HYDOXDWLRQ UHYHDOHG WKDW RQO\ D IHZ FDVHV DFWLYHO\ LQYROYHG SULPDU\
VWDNHKROGHUV ² WKH EHQHILFLDULHV ² LQ VKDSLQJ GHYHORSPHQW GHFLVLRQV LH LQ GHWHUPLQLQJ HYDOXDWLRQ REMHFWLYHV
DQDO\VLVDQGLQWHUSUHWDWLRQ
6HYHUDO RYHUYLHZV RI 30 ( VXJJHVW WKDW SDUWLFLSDWLRQ PD\ YDU\ DW GLIIHUHQW VWDJHV RI WKH 30 ( DQG
SURMHFWF\FOHVZLWKOHVVLQYROYHPHQWE\DOOVWDNHKROGHUVSDUWLFXODUO\ORFDOSDUWLFLSDQWVDWHDUO\VWDJHVGHVLJQ
DQG SODQQLQJ DV ZHOO DV DW ODWHU VWDJHV DQDO\VLV DQG GLVVHPLQDWLRQ RI 0 ( )RU LQVWDQFH $UPRQLD DQG
&DPSLODQSRLQWRXWWKDWLQJHQHUDOORFDOSDUWLFLSDWLRQGHFUHDVHVDVWKHSURMHFWSURJUHVVHVIURPWKH
SODQQLQJVWDJHVWRLPSOHPHQWDWLRQDQGWRPRQLWRULQJDQGHYDOXDWLRQ$EERWDQG*XLMWDOVRIRXQG
WKLV WR EH HYLGHQW LQ WKHLURYHUYLHZ RISDUWLFLSDWRU\ PRQLWRULQJ RI WKH HQYLURQPHQW :KLOH DOO FDVHV UHYLHZHG
ZHUHKLJKO\SDUWLFLSDWRU\GXULQJWKHGDWDFROOHFWLRQVWDJHIHZLQYROYHGDOOVWDNHKROGHUVLQHDUO\DQGODWHUVWDJHV
+RZHYHUWKHDXWKRUVREVHUYHWKDWWKRVHFDVHVZKLFKGLGLQFOXGHDOOVWDNHKROGHUVWKURXJKRXWWKH30 (
SURFHVV SURGXFHG PRUH UHOHYDQW DQG XVHIXO LQIRUPDWLRQ IRU VWDNHKROGHUV WKDQ WKRVH DSSURDFKHV ZKLFK RQO\
LQFOXGHGORFDOSHRSOHDVGDWDJDWKHUHUV
6HYHUDO FDVH VWXGLHV VKRZ WKH GLIILFXOW\ RI HQVXULQJ ORFDO SDUWLFLSDWLRQ HVSHFLDOO\ LQ XVLQJ DQG DQDO\VLQJ
LQIRUPDWLRQ )RU H[DPSOH &$5( =DPELD
V PRVW VLJQLILFDQW FKDOOHQJH LQ LPSOHPHQWLQJ D SDUWLFLSDWRU\
PRQLWRULQJV\VWHPLVWRRYHUFRPHWHQGHQFLHVRIH[WUDFWLQJLQIRUPDWLRQDQGWRHPSRZHUSURMHFWEHQHILFLDULHVE\
JHQXLQHO\LQYROYLQJWKHPLQWKHDQDO\VLVDQGXVHRIFROOHFWHGLQIRUPDWLRQ:DUG6LPLODUO\LQUHIHUHQFH
WRWKH&DOFXWWD6OXP,PSURYHPHQW3URMHFW&6,3.DUHWDOE²SRLQWRXWWKDWRQHPDMRUSUREOHP
RIWKH3,$SURFHVVZDVLWV
ODFNRIV\VWHPDWLFLQYROYHPHQWRIWKHSDUWLFLSDQWVLQWKHDQDO\VLVRIWKHGDWD
ZKLFK
ZDV FDUULHG RXW E\ RXWVLGHUV +HQFH WKHUH ZDV D FRQFHUQ WKDW
WKH SURFHVV ZDV H[WUDFWLYH UDWKHU WKDQ
HPSRZHULQJ
 SDUWLFXODUO\ IRU WKH VOXP GZHOOHUV 7KH DXWKRUV UHFRPPHQG WKDW IXWXUH 3,$V VKRXOG VHHN WR
IDFLOLWDWHDQDO\VLVRIILQGLQJVZLWKFRPPXQLW\JURXSVDQGWKDWORFDOSDUWLFLSDWLRQEHLQWHJUDWHGLQSURMHFWGHVLJQ
LPSOHPHQWDWLRQ PRQLWRULQJ DQG HYDOXDWLRQ 6XFK D SURFHVV ZRXOG FUHDWH D JUHDWHU VHQVH RI ORFDO RZQHUVKLS
ZKLFKLQWXUQZRXOGHQKDQFHSURMHFWVXVWDLQDELOLW\DQGHIIHFWLYHQHVV
2Q WKH RWKHU KDQG RWKHU FDVH VWXGLHV GHPRQVWUDWH KRZ WR RSHUDWLRQDOLVH WKH LGHD RI SDUWLFLSDWLRQ )RU
H[DPSOH *XLMW DQG 6LGHUVN\  GHVFULEH KRZ WKH SDUWLFLSDWRU\ PRQLWRULQJ LQ %UD]LO LQYROYHG LGHQWLI\LQJ
WKUHHOHYHOVRIIDUPHUSDUWLFLSDWLRQDUUDQJHGDFFRUGLQJWRLQFUHDVLQJJURXSVL]HZKLFKKHOSFODULI\WKHVSHFLILF
UROHV DQG DFWLYLW\ DUHDV IRU YDULRXV ORFDO IDUPLQJ JURXSV DQG DGGUHVV WKHLU GLIIHUHQW PRQLWRULQJ QHHGV DQG
LQWHUHVWV2WKHUFDVHVWXGLHV²LQFOXGLQJWKH(=(&/HDUQLQJ,QLWLDWLYHWKH+HDOWK&DUHIRU$OO+&)$DQG
WKH05'3SURJUDPPHLQ9LHWQDP²IXUWKHULOOXVWUDWHKRZLQWHQGHGEHQHILFLDULHVEHFRPHDFWLYHO\LQYROYHGDWDOO
VWDJHVRIWKH30 (F\FOHLQFOXGLQJSODQQLQJDQDO\VLVDQGGRFXPHQWDWLRQ

0HWKRGRORJLFDO,VVXHV
:KLOHLWLVLPSRUWDQWWRUHFRJQLVHWKHVRFLDODQGSROLWLFDOLVVXHVHPEHGGHGZLWKLQWKHSUDFWLFHRISDUWLFLSDWRU\
PRQLWRULQJ DQG HYDOXDWLRQ WKHUH DUH DOVR PHWKRGRORJLFDO LVVXHV WKDW QHHG IXUWKHU GLVFXVVLRQ DQG
HODERUDWLRQUHILQHPHQW


'HYHORSLQJQHZVWDQGDUGVRIULJRXU
6RPH RIWKH OLWHUDWXUH VXJJHVWV WKDW WR XVH D SDUWLFLSDWRU\ HYDOXDWLRQ DSSURDFK UDWKHUWKDQ D PRUH WUDGLWLRQDO
DSSURDFK LV WR DFFHSW WUDGHRIIV EHWZHHQ VFLHQWLILF ULJRXU RQ WKH RQH KDQG DQG ORFDOO\ PHDQLQJIXO DQG
SDUWLFLSDWRU\DSSURDFKHVRQWKHRWKHU,WLVRIWHQDVVXPHGWKDWTXDQWLWDWLYHGDWDUHSUHVHQWVDFHUWDLQGHJUHHRI
REMHFWLYLW\DQGUHSOLFDELOLW\WKDWFDQEHJHQHUDOLVHGDQGVFDOHGXSZKLOHTXDOLWDWLYHLQIRUPDWLRQUHIOHFWVDPRUH
VXEMHFWLYHLQWHUSUHWDWLRQWKDWLVFRQWH[WVSHFLILFDQGORFDOO\UHOHYDQW
+RZHYHU YLHZLQJ WUDGHRIIV LQ WHUPV RI DFKLHYLQJ SDUWLFLSDWLRQ DQG ULJRXU KDV VHULRXV OLPLWDWLRQV )LUVW
TXHVWLRQVPD\EHUDLVHGDERXWWKHREMHFWLYLW\HYHQRIFRQYHQWLRQDOVFLHQWLILFDSSURDFKHV6LPLODUO\ZHFDQQRW
PDNH DVVXPSWLRQV WKDW TXDQWLWDWLYH PHDVXUHV ZLOO QRW EH XVHG LQ SDUWLFLSDWRU\ DSSURDFKHV QRU WKDW WKH\ DUH
QHFHVVDULO\PRUHREMHFWLYHWKDQTXDOLWDWLYHPHDVXUHV7KHUHLVWKHUHIRUHDQHHGWRUHH[DPLQHZKHWKHURUQRW
WKHUHDUHLQKHUHQWWUDGHRIIVLQFRPELQLQJ
WUDGLWLRQDO
DQG
SDUWLFLSDWRU\
DSSURDFKHVDQGWRLGHQWLI\WKHLPSOLFLW
DVVXPSWLRQVVXUURXQGLQJWKHTXDQWLWDWLYHDQGTXDOLWDWLYHGHEDWHKRZWKH\DUHUHVROYHGRUDYRLGHGDQGXQGHU
ZKDWFRQGLWLRQVDQGFLUFXPVWDQFHV5DWKHUWKDQSRVHWKHLVVXHDVHLWKHUVFLHQWLILFULJRXURUSDUWLFLSDWLRQPRUH
LPSRUWDQWLVWRGHYHORSQHZVWDQGDUGVRIYDOLGLW\IRU30 (
+RZHYHULQWKHLURYHUYLHZ$EERWDQG*XLMWREVHUYHWKDWWKHUHLVD
FRQVWDQWWXJRIZDUEHWZHHQ

EHLQJSDUWLFLSDWRU\
DQG
GRLQJULJRURXVPRQLWRULQJ
DVFRQYHQWLRQDOO\SHUFHLYHG
$VPRQLWRULQJPRYHVDZD\
IURPDVFLHQWLVWGRPLQDWHGDSSURDFKWRZDUGVJUHDWHUFRPPXQLW\LQYROYHPHQW
ZKDWLVORVWDQGZKDWLVJDLQHG

RIWHQEHFRPHVDNH\GLOHPPDGXULQJWKH30 (SURFHVV*LYHQWKDWWKHUHLVYHU\OLWWOHLQIRUPDWLRQDYDLODEOHLQ
WKH OLWHUDWXUH ZKLFK H[SORUHV WKHVH NH\ FRQFHUQV WKH DXWKRUV UDLVH WKH TXHVWLRQ ZKHWKHU LQ IDFW WUDGHRIIV
QHFHVVDULO\ H[LVW EHWZHHQ ULJRXU RQ WKH RQH KDQG DQG SDUWLFLSDWLRQ RQ WKH RWKHU RU ZKHWKHU WKH\ FDQ EH
FRPELQHG$EERWDQG*XLMW,QWKHFDVHRI&(7$0(;1LQ0H[LFRIRULQVWDQFHWKHIOH[LEOHXVHRIWKH
*UDVVURRWV'HYHORSPHQW)UDPHZRUNGHYHORSHGE\WKH,QWHU$PHULFDQ)RXQGDWLRQLVRQHDWWHPSWWRHVWDEOLVK
D
GDWDEDVHWKDWDOORZVTXDQWLWDWLYHSUHVHQWDWLRQRIUHVXOWVRQTXDOLWDWLYHLQGLFDWRUV
%ODXHUWDQG4XLQWDQDU

0DUVGHQ DQG 2DNOH\  DUJXH WKDW WKH GHFLVLRQ WR XVH HLWKHU TXDQWLWDWLYH RU TXDOLWDWLYH GDWD LV QRW D
SDUDPRXQWLVVXHEXWUDWKHUGHWHUPLQLQJWKHSXUSRVHDQGWKHLQWHUSUHWDWLRQRILQIRUPDWLRQLVFRQVLGHUHGPRUH
LPSRUWDQW6LPLODUO\$EERWDQG*XLMWDUJXHWKDWLQSUDFWLVHWKHEDODQFHEHWZHHQVHHNLQJVFLHQWLILFULJRXU
DQGFRPPXQLW\SDUWLFLSDWLRQZLOOGHSHQGJUHDWO\RQWKHREMHFWLYHVRIWKHPRQLWRULQJSURFHVVLWVHOI)RUH[DPSOH
PRUHFRQYHQWLRQDOVFLHQWLILFDSSURDFKHVPD\EHXVHGLIFRQVLVWHQF\DQGSURRIRILPSDFWLVUHTXLUHGZKLOHPRUH
IOH[LEOHTXDOLWDWLYHPHWKRGVPD\EHXVHGZKHUHOHDUQLQJLVHPSKDVLVHGLELGVHHDOVR83:$5'
2WKHUV KRZHYHU YLHZ WKH FRPELQDWLRQ RI GLIIHUHQW DSSURDFKHV LQ 30 ( SRWHQWLDOO\ D PXFK PRUH
SUREOHPDWLFSURFHVVDVLWDWWHPSWVWRLQFRUSRUDWHWKHGLYHUVHH[SHFWDWLRQVRIPXOWLSOHVWDNHKROGHUVDQGEDODQFH
GLIIHULQJREMHFWLYHV6HYHUDOFDVHH[SHULHQFHVFDQKHOSLOOXVWUDWHDQGGHILQHVRPHRIWKHVHSUREOHPDWLFLVVXHV)RU
LQVWDQFH&DPSLODQKLJKOLJKWVWKHSRWHQWLDOWHQVLRQEHWZHHQJDWKHULQJTXDQWLWDWLYHGDWDDQGVHFXULQJORFDO
SDUWLFLSDWLRQEDVHGRQ83:$5'
VH[SHULHQFHVZRUNLQJ ZLWK IDUPHUV LQ WKH 3KLOLSSLQHV,Q WKLV FDVHIDUPHUV
IRXQGWKHV\VWHPDWLFUHFRUGLQJRIVRLOORVVPHDVXUHPHQWVWRPHHWUHVHDUFKGDWDUHTXLUHPHQWWRRFXPEHUVRPH
DQGWKHUHIRUHKDUGO\ SDUWLFLSDWHG LQGDWD FROOHFWLRQ 6XEVHTXHQWO\ UHVHDUFKHUV IURP 83:$5' KDG WRPRGLI\
WKHLULQLWLDOTXDQWLWDWLYHDSSURDFKWRZDUGVDPRUHRSHQHQGHGDSSUDLVDORIFKDQJHVLQVRLOFRQGLWLRQVLQRUGHUWR
WDNHLQWRDFFRXQWORFDOFDSDFLWLHVDQGLQYROYHORFDOIDUPHUVLQFROOHFWLQJDQGDQDO\VLQJLQIRUPDWLRQ


&KRRVLQJDQGGHYHORSLQJLQGLFDWRUV
0XFKRIWKHOLWHUDWXUHDOVRKLJKOLJKWVWKHLPSRUWDQFHRILGHQWLI\LQJLQGLFDWRUVDVDQLQWHJUDOSDUWRIWKH30 (
F\FOH +RZHYHU WKH SURFHGXUHV DQG SULQFLSOHV IRU GHYHORSLQJ 0 ( LQGLFDWRUV DUH QRW DOZD\V FOHDU RU
VWUDLJKWIRUZDUGLQSUDFWLFH6HYHUDOPDMRUSRLQWVPD\EHUDLVHGZKLFKSHUWDLQHVSHFLDOO\WRWKHGHFLVLRQPDNLQJ
SURFHVVRILQGLFDWRUGHYHORSPHQW
7KHILUVWPDMRULVVXHUHODWHVWRZKRLV LQYROYHG LQ VHWWLQJ PRQLWRULQJ DQG HYDOXDWLRQ LQGLFDWRUV ,Q RWKHU
ZRUGVZKRVHWVWKHFULWHULDIRUGHYHORSLQJLQGLFDWRUV":KRLVLQYROYHGDQGZKRKROGVLQIOXHQFHLQGHFLGLQJ
ZKDW W\SHV RI LQIRUPDWLRQ ZLOO EH FROOHFWHG KRZ LQIRUPDWLRQ ZLOO EH XVHG IRU ZKRP DQG IRU ZKDW SXUSRVH
7KHVH TXHVWLRQV ZLOO DJDLQ QHHG WR DGGUHVV XQGHUO\LQJ VRFLDO DQG SROLWLFDO SURFHVVHV LQ QHJRWLDWLQJ LQGLFDWRUV
DPRQJWKHYDULRXVVWDNHKROGHUV
$VWKHFDVHZLWKIDUPHUVLQ%UD]LOGHPRQVWUDWHVNQRZLQJZKRWKHHQGXVHUVRILQIRUPDWLRQZRXOGEH²LQ
WKLV FDVH ORFDO IDUPHUV DQG GRQRUV ² ZDV D FULWLFDO IDFWRU LQ GHFLGLQJ ZKDW DQG KRZ WR PRQLWRU DQG HYDOXDWH
*XLMWDQG6LGHUVN\$OVRWKHH[SHULHQFHRI3,$LQ*KDQDKLJKOLJKWVKRZGLYHUJHQWLQWHUHVWVRIGLIIHUHQW
VWDNHKROGHUVPDNHLWGLIILFXOWWRLGHQWLI\ZKDWWRHYDOXDWH,Q*KDQDLQRUGHUWRVDWLVI\ERWKGRQRUVDQGSURMHFW
LPSOHPHQWHUV DW WKH YLOODJH OHYHO FRQWHQGLQJ REMHFWLYHV RI EXLOGLQJ YLOODJHOHYHO FDSDFLW\  DQG RI HQVXULQJ WKH
LPPHGLDWHGHOLYHU\RILPSURYHGZDWHUVXSSO\ZHUHHYHQWXDOO\FRPELQHGLQWRRQHREMHFWLYHDVVHVVLQJWKHLPSDFW
RIFDSDFLW\EXLOGLQJRQDFFHVVWRLPSURYHGZDWHUVXSSO\DQGVDQLWDWLRQVHUYLFHV²
2QHRWKHUVLJQLILFDQWLVVXHLVWKHQHHGWRILQGQHZNLQGVDQGW\SHVRILQGLFDWRUVWRDVVHVVQRQWUDGLWLRQDO
DSSURDFKHV LQ GHYHORSPHQW HVSHFLDOO\ WKRVH SHUWDLQLQJ WR SDUWLFLSDWLRQ HPSRZHUPHQW DQG VRFLDO SURFHVVHV
6HYHUDOLQQRYDWLYHDSSURDFKHVWRLQGLFDWRUGHYHORSPHQWDUHLOOXVWUDWHGLQWKHFDVHVWXGLHV)RULQVWDQFHLQWKH
&(7$0(;1SURMHFWLQ0H[LFR%ODXHUWDQG4XLQWDQDUGHVFULEHVDQ0 (IUDPHZRUNWKDWDOORZVIRU
V\VWHPDWLF GDWD UHFRUGLQJ EXW EDVHG RQ WKH IOH[LEOH GHYHORSPHQW RI LQGLFDWRUV ZKLFK FRPELQH WKH GLIIHUHQW
FULWHULD RI WKH YDULRXV VWDNHKROGHUV VHH %R[  $OVR WKH H[SHULPHQWDO DSSURDFK DGRSWHG E\ WKH &&'% LQ
%DQJODGHVKGHPRQVWUDWHVKRZPRQLWRULQJRIFKDQJHQHHGQRWEHEDVHGRQLQGLFDWRUVSHUVHEXWUDWKHURQKRZ
WKHGLIIHUHQWVWDNHKROGHUVDWYDULRXVOHYHOVSHUFHLYHWKRVHFKDQJHV'DYLHV7KHFDVHRIWKHFRPPXQLW\
EDVHG QXWULWLRQ SURJUDPPH LQ .HQ\D IXUWKHU VKRZV KRZ SDUWLFLSDWRU\ HYDOXDWLRQ FDQ HOLFLW GLIIHUHQW
VWDNHKROGHUV
 FULWHULD IRU HYDOXDWLQJ SURJUDPPH LPSDFW EDVHG RQ WKHLU SHUFHSWLRQV RI WKH SURFHVV RI SURMHFW
LPSOHPHQWDWLRQ WKHLU RZQ H[SHFWDWLRQV DQG H[SHULHQFHV DV ZHOO DV WKHLU WKRXJKWV DQG LGHDV DERXW IXWXUH
FKDQJHVDQGDFWLRQ
$WKLUGNH\DUHDLQTXHVWLRQLVNQRZLQJWKHH[WHQWWRZKLFKEDVHOLQHLQIRUPDWLRQLVUHTXLUHGIRULQGLFDWRU
GHYHORSPHQW %DVHOLQH LQIRUPDWLRQ PD\ EH XVHG WR SURYLGH D EDVLV IRU LGHQWLI\LQJ SULRULW\ QHHGV DQG IRU
DVVHVVLQJ KRZ D SURJUDPPH LV FKDQJLQJ DQG SURJUHVVLQJ )HXHUVWHLQ  ² ,Q PRVW FDVHV EDVHOLQH
GDWDLVRIWHQPLVVLQJRULQFRPSOHWHPDNLQJDWWHPSWVWRZDUGVFRPSDULVRQVRYHUWLPHYHU\GLIILFXOWDQGUHOLDQW
RQKLVWRULFDOUHFRQVWUXFWLRQ2QWKHRWKHUKDQGSURWUDFWHGEDVHOLQHVWXGLHVPD\EHWRRFXPEHUVRPHDQGWKXV
UHVXOWLQ
FRPPXQLW\IDWLJXH
LQWHUPVRIGDWDJDWKHULQJXQOHVVDFFRPSDQLHGE\RWKHUUHOHYDQWSURMHFWDFWLYLWLHV
&21&(51
2EWDLQLQJEDVHOLQHLQIRUPDWLRQZDVDSHUWLQHQWLVVXHLQVHYHUDORIWKHFDVHVWXGLHVVXUYH\HG)RULQVWDQFHLQ
WKHFDVHRI3,'2:
VVHOIHYDOXDWLRQ6RPPHUQRWHVWKDWRQHPDMRUOLPLWDWLRQRIWKHSURFHVVZDVQRW
KDYLQJ HQRXJK DYDLODEOH LQIRUPDWLRQ DERXW WKH KLVWRULFDO EDFNJURXQG DQG FRQWH[W SULRU WR SURMHFW


LPSOHPHQWDWLRQ ZKLFK PDGH LW GLIILFXOW WR UHODWH WKH FXUUHQW VLWXDWLRQ WR SURMHFW LQWHUYHQWLRQV 6LPLODUO\
:LJEROGXV HW DO   SRLQW RXW WKH GLIILFXOWLHV RI HYDOXDWLQJ WKH LPSDFWV RI WKH 1HSDO 5HVRXUFH
0DQDJHPHQW 3URMHFW GXH WR WKH ODFN RI GRFXPHQWDWLRQ DW WKH EHJLQQLQJ RI 1503
V LQYROYHPHQW LQ ORFDO
FRPPXQLWLHV ,Q UHIHUHQFH WR WKH 3,$ H[SHULHQFH LQ *KDQD *DULED   DUJXHV WKDW WKH FKDOOHQJH LV WR
VLPSOLI\WKHSDUDPHWHUVRIEDVHOLQHGDWDFROOHFWLRQLQZD\VWKDWFDQSURYLGHDFRQVLVWHQWPHFKDQLVPIRUWUDFNLQJ
SHUIRUPDQFHRYHUWLPH6KHQRWHVWKDWPRUHRIWHQEDVHOLQHVWXGLHVWHQGWREHFRPSOH[DQGGLIIXVHSURYLGLQJ
GHWDLOHGLQIRUPDWLRQWKDWLVOHVVXVHIXORYHUWLPHDVDEDVHOLQHIRUDVVHVVLQJIXWXUHLPSDFWV

7KHUROHRIWKHIDFLOLWDWRU
,QJHQHUDOWKHOLWHUDWXUHVXUYH\UHYHDOHGWKDWD
JRRG
IDFLOLWDWRUVHUYHVPDLQO\DVDFDWDO\VWRUVWLPXODWRUEULQJLQJ
WRJHWKHUYDULHGH[SHULHQFHVDQGSHUFHSWLRQVRIWKHYDULRXVSDUWLFLSDQWV0RUHRYHUIDFLOLWDWRUVVKRXOGEHDEOHWR
EHIOH[LEOHDVNJRRGTXHVWLRQVDWWKHULJKWWLPHOLVWHQZHOOEXLOGJURXSUDSSRUWDQGWUXVWDPRQJSDUWLFLSDQWV
HQFRXUDJH SDUWLFLSDWLRQ DQG VKDULQJ RI LGHDV DQG IRFXV RQ LPSRUWDQW LVVXHV IRU JURXS DQDO\VLV DQG DFWLRQ
SODQQLQJVHH&DPSRVDQG&RXSDO)HXHUVWHLQ6FRWW9LOOLHUVD
+RZHYHU DV WKH VXUYH\ RI FDVH VWXGLHV VKRZV WKH GRFXPHQWDWLRQ RI DFWXDO 30 ( H[SHULHQFHV UDUHO\
HODERUDWHV RQRUUHFRJQLVHV WKHUROH RIIDFLOLWDWRUV WKHQHFHVVDU\VNLOOV LQ VKDSLQJ WKH RXWFRPH RI WKH 30 (
SURFHVV RU KRZ WKRVH VNLOOV DUH GHYHORSHG )XUWKHUPRUH WKHUH LV PXFK OHVV FRQVLGHUDWLRQ JLYHQ WR SDUWLFXODU
VNLOOVWKDWSHUWDLQWRQHJRWLDWLRQDQGFRQIOLFWUHVROXWLRQ)LQDOO\WKHUHLVQHHGWRDVVHVVWKHSRWHQWLDOXVHIXOQHVVRI
EXLOGLQJ DQ HYDOXDWLRQ WHDP UDWKHU WKDQ UHO\LQJ RQ LQGLYLGXDO IDFLOLWDWRUV DV D PHDQV RI WDNLQJ DGYDQWDJH RI
PXOWLSOHVNLOOVDQGH[SHULHQFHVLQHIIHFWLYHO\FDUU\LQJRXWWKH30 (SURFHVV,QWKLVUHJDUG.DUHWDOD
HPSKDVLVH WKH LPSRUWDQFH RI EXLOGLQJ DQ HYDOXDWLRQ WHDP LQ FRQGXFWLQJ 3,$ LQ *KDQD :LWK UHVSHFW WR WKH
+RUL]RQV 3URMHFW 0HUULILHOG /DQFDVWHU DQG .LUNKDP  DOVR UHFRJQLVH WKH YDOXH RI WHDP EXLOGLQJ EXW
GLVFXVVVRPHRIWKHDGYDQWDJHVDQGGLVDGYDQWDJHVRIGLIIHUHQWHYDOXDWRUVZRUNLQJWRJHWKHU

7KH,QVWLWXWLRQDOLVDWLRQRI30 (
$V30 (LQYROYHVDZLGHUEDVHRISDUWLFLSDQWVSDUWQHUVKLSVKDYHHYROYHGQRWRQO\DPRQJVWLQGLYLGXDODFWRUV
EXW DOVR DPRQJVW VWDNHKROGHU JURXSV DW GLIIHUHQW LQVWLWXWLRQDO OHYHOV 7KH VXUYH\ RI OLWHUDWXUH DQG FDVH VWXGLHV
VKRZVWKDWLQWHULQVWLWXWLRQDOOLQNDJHVEHWZHHQGLIIHUHQWVWDNHKROGHUVLQFDUU\LQJRXW0 (DUHEHFRPLQJPXFK
PRUHHYLGHQW
,QVSLWHRIWKHVHPRXQWLQJHIIRUWVWKHUHLVVWLOODQHHGWRDGGUHVVLVVXHVSHUWDLQLQJWRWKHLQVWLWXWLRQDOLVDWLRQ
DQG VXVWDLQDELOLW\ RI 30 ( SUDFWLFH $UPRQLD DQG &DPSLODQ  IRXQG IHZ FDVHV WKDW DFWXDOO\ GRFXPHQW
KRZWKHLQVWLWXWLRQDOLVDWLRQRI30 (LVFDUULHGRXWRYHUWKHORQJWHUP$NH\TXHVWLRQWKDWUHPDLQVLVZKHWKHU
30 (FDQEHVXVWDLQHGEH\RQGWKHSURMHFWRUSURJUDPPHOLIH F\FOH IXOO\ WDNHQ RYHU E\ ORFDO SHRSOH DQGRU
EXLOWLQWRWKHVWDQGDUGRSHUDWLQJSURFHGXUHVRIIRUPDOLQVWLWXWLRQVLELG83:$5'UDLVHVDVLPLODUSRLQW
FRQFHUQLQJ WKH XQFHUWDLQ IDWH RI 30 ( EH\RQG WKH OLIH RI WKH SURMHFW DQG VWUHVVHV WKH QHHG WR SD\ JUHDWHU
DWWHQWLRQ WR KRZ 30 ( LV LQVWLWXWLRQDOLVHG DV SDUW RI WKH QRUPV DQG VWDQGDUGV RI DJHQFLHV DQG ORFDO
FRPPXQLWLHV,QWKHLUVXUYH\RI%ROLYLD$OFRFHU/L]iUUDJD'HOJDGLOORHWDOIXUWKHUSRLQWRXWWKDWZKLOHPDQ\
UHFRJQLVH WKH LPSRUWDQFH RI LQVWLWXWLRQDOLVLQJ SDUWLFLSDWRU\ DSSURDFKHV WKHUH LV QR DJUHHPHQW UHJDUGLQJ ZKDW


V\VWHPDWLVLQJ30 (DFWXDOO\HQWDLOVKRZLWPLJKWEHDFKLHYHGE\ZKRPDQGIRUZKRPLWVKRXOGEHXQGHUWDNHQ

7KH FDVH VWXGLHV KLJKOLJKW VRPH RI WKHVH SUREOHPDWLF LVVXHV )RU H[DPSOH 7KRPSVRQ DQG 3UHWW\ 
GHVFULEH KRZ WKH SUDFWLFH RI SDUWLFLSDWRU\ LPSDFW DVVHVVPHQW LV QRZ UHJDUGHG E\ JRYHUQPHQW LQVWLWXWLRQV LQ
.HQ\D DV D IXQGDPHQWDO SDUW RI VRLO DQG ZDWHU FRQVHUYDWLRQ 6 :& LQLWLDWLYHV 6LJQLILFDQW FKDQJHV LQ WKH
RSHUDWLRQDO IHDWXUHV RI JRYHUQPHQW DJHQFLHV KDYH RFFXUUHG LQFOXGLQJ ZLGHVSUHDG VXSSRUW DW DOO OHYHOV ZLWKLQ
.HQ\D
V6 :&EUDQFKIRUWKHXVHRIQHZSDUWLFLSDWRU\PHWKRGVLQSODQQLQJDQGLPSOHPHQWDWLRQLQYROYLQJORFDO
IDUPHUVDQGIRUFROODERUDWLRQEHWZHHQGLIIHUHQWJRYHUQPHQWGHSDUWPHQWV7KRPSVRQDQG3UHWW\²
$OVR LQ UHIHUHQFH WR FLWL]HQEDVHG PRQLWRULQJ LQLWLDWLYHV LQ WKH 86 VXFK DV WKH +&)$ DQG (=(& /HDUQLQJ
,QLWLDWLYH3DUDFKLQLDQG0RWWGHVFULEHKRZJUDVVURRWVEDVHG0 (DUHEXLOGLQJORFDOFDSDFLWLHVVRSHRSOH
LQWKHLURZQULJKWPD\EHFRPHHIIHFWLYHSXEOLFSROLF\DQDO\VWVDQG
RXWVLGHDGYRFDWHVIRUFKDQJH
,QWKHFDVHRI
&&'%
VH[SHULPHQWDODSSURDFKWRSDUWLFLSDWRU\PRQLWRULQJ'DYLHVREVHUYHVWKDWZKLOHWKHPRQLWRULQJ
SURFHVV LWVHOI KDV GHPRQVWUDWHG WKH FDSDFLW\ WR HYROYH LQ WHUPV RI FKDQJHV LQ LWV SURFHGXUHV DQG SHUFHLYHG
IXQFWLRQVDVLGHQWLILHGE\VWDNHKROGHUVLWVIXWXUHUHPDLQVXQFHUWDLQGXHWRGLIIHUHQFHVLQRSLQLRQDPRQJVHQLRU
OHYHO&&'%VWDIIUHJDUGLQJWKHYDOXHRIWKHSURFHVV

5HVRXUFHUHTXLUHPHQWVRI30 (
5HODWHG WR LVVXHV RI LQVWLWXWLRQDOLVDWLRQ DQG VXVWDLQDELOLW\ LV WKH QHHG WR EHWWHU DVVHVV WKH
FRVWV
 RI 30 ( LQ
WHUPVRIUHVRXUFHUHTXLUHPHQWV7KHDYDLODELOLW\ RI UHVRXUFHV LV D FULWLFDOIDFWRU WKDW DIIHFWV WKH ZD\ 30 ( LV
FRQGXFWHG:KLOHWKHUHVRXUFHVUHTXLUHGIRUFDUU\LQJRXW30 (DUHGLVFXVVHGZLGHO\LQWKHOLWHUDWXUHVHHIRU
H[DPSOH &DPSRV DQG &RXSDO  )HXHUVWHLQ  *RVOLQJ DQG (GZDUGV  5XELQ  IHZ FDVHV
DFWXDOO\GRFXPHQWWKHDPRXQWRIUHVRXUFHVQHHGHGRYHUWLPHDQGZKHQDQGKRZWKHVHUHVRXUFHUHTXLUHPHQWV
DUHVSHFLILHG,GHDOO\UHVRXUFHUHTXLUHPHQWVVKRXOGEHDJUHHGXSRQIURPWKHRXWVHWDQGLQWHJUDWHGLQWRSURMHFW
SODQQLQJ DQG VFKHGXOLQJ VHH &DPSRV DQG &RXSDO  )HXHUVWHLQ  5HVRXUFH UHTXLUHPHQWV LQFOXGH WKH
IROORZLQJ  ILQDQFLDO UHVRXUFHV WLPH DYDLODELOLW\ DQG HVSHFLDOO\ KXPDQ UHVRXUFHV LQ WHUPV RI OHYHO RI
FRPPLWPHQWHIIRUWDQGFDSDFLWLHV
:LWKUHVSHFWWRWLPHDYDLODELOLW\DQGFRQVWUDLQWVPDQ\KDYHDUJXHGWKDWDGRSWLQJSDUWLFLSDWRU\DSSURDFKHV
WR0 (WKDWLQYROYHPRUHVWDNHKROGHUVZLOOJHQHUDOO\UHTXLUHVXEVWDQWLDOWLPHFRPPLWPHQWVVHH&DPSRVDQG
&RXSDO  )HXHUVWHLQ  ,Q WKH FDVH RI 3,'2:
V VHOIHYDOXDWLRQ H[HUFLVH ZKLFK WRRN RYHU  PRQWKV
+XPEHUW'UR]  KLJKOLJKWV KRZ D VHOIHYDOXDWLRQ H[HUFLVH LV PXFK PRUH WLPHFRQVXPLQJ HVSHFLDOO\ LQ
WHUPVRISURMHFWVWDIIGD\VDVFRPSDUHGWRWKHH[WHUQDOHYDOXDWLRQSUHYLRXVO\FDUULHGRXWIRU3,'2:+RZHYHU
KHQRWHVWKDWWKHFRQVLGHUDEOHWLPHLQYHVWPHQWVKRXOGEHZHLJKHGDJDLQVWWKHTXDOLW\RIWKH6(RXWSXWVDQGWKH
JUHDWHUOLNHOLKRRGWKDWILQGLQJVDQGUHFRPPHQGDWLRQVFDQEHRSHUDWLRQDOLVHG,QDGGLWLRQKHHVWLPDWHVWKDWWLPH
UHTXLUHPHQWVPD\EHUHGXFHGFRQVLGHUDEO\DVWKHPHWKRGRORJ\LVEHWWHUXQGHUVWRRGDQGPRUHIXOO\LQWHJUDWHG
LQWRSURMHFWHYDOXDWLRQRYHUWLPH
)LQDOO\ KXPDQ UHVRXUFHV RI 30 ( DOVR QHHG FRQVLGHUDWLRQ HVSHFLDOO\ DV WKH SDUWLFLSDWRU\ SURFHVV ZLOO
UHTXLUH JUHDWHU GHJUHHV RI FRRUGLQDWLRQ DGPLQLVWUDWLYH HIIRUWV DQG ORQJWHUP FRPPLWPHQW RQ WKH SDUW RI
VWDNHKROGHUVDWDOOLQVWLWXWLRQDOOHYHOV83:$5'+XPDQUHVRXUFHVLQFOXGHDFTXLULQJWKHQHFHVVDU\VNLOOV
IRU FDUU\LQJ RXW DQG VXVWDLQLQJ 30 ( ,Q WKLV UHJDUG WUDLQLQJ IRU 30 ( KDV UHFHQWO\ UHFHLYHG FRQVLGHUDEOH


DWWHQWLRQLQWKHOLWHUDWXUH+RZHYHULQWKHLURYHUYLHZ&21&(51QRWHVWKDWWKHUHVHHPVWREHYHU\OLWWOH
DYDLODEOH GRFXPHQWDWLRQ RQ WUDLQLQJ LQ 30 ( DOWKRXJK VRPH H[SHULHQFHV LQ WKLV DUHD LQFOXGH WUDLQLQJ
ZRUNVKRSVFRQGXFWHGE\VHYHUDO1*2VQDPHO\$&25'$FWLRQ$LG2;)$0DQG&21&(51LWVHOI5HFHQW
H[SHULHQFHV LQ WUDLQLQJ IRU 30 ( LQFOXGH ZRUNVKRSV RUJDQLVHG IRU WKH 05'3 SURJUDPPH LQ 9LHWQDP VHH
6FRWW9LOOLHUVEIRUWKHWUDLQLQJPDQXDOXVHG

7KHVFDOLQJXSRI30 (
$VSDUWLFLSDWRU\DSSURDFKHVWR0 (DUHLQFUHDVLQJO\EDVHGRQLQWHULQVWLWXWLRQDOOLQNDJHVWKHUHLVQHHGIRUPRUH
RSHQGLVFXVVLRQDERXWWKHVFDOLQJXSRI30 (DWKLJKHULQVWLWXWLRQDOOHYHOVHVSHFLDOO\DWWKHOHYHORIQDWLRQDO
DQGLQWHUQDWLRQDOSROLF\7KHUHDUHWZRLVVXHVZLWKUHJDUGWRVFDOLQJXS30 (WKHVFDOLQJXSRIPLFUROHYHO
LQIRUPDWLRQ JHQHUDWHG DW WKH YLOODJH DQG SURMHFW OHYHOV DQG  WKH LPSOLFDWLRQV RI LQWHJUDWLQJ SDUWLFLSDWRU\
DSSURDFKHV VSHFLILFDOO\ LQ WHUPV RI UHTXLULQJ V\VWHPDWLF DWWLWXGLQDO DQG SURFHGXUDO FKDQJHV DW DOO KLJKHU
LQVWLWXWLRQDOOHYHOVRISODQQLQJLPSOHPHQWDWLRQDQGHYDOXDWLRQSUDFWLFH
0F$UWKXUUDLVHVWKHTXHVWLRQRIKRZPLFUROHYHOGDWDSURGXFHGDWWKHYLOODJHDQGSURMHFWOHYHO
FDQEHVWEHDQDO\VHGDQGXVHGIRUGHYHORSLQJQDWLRQDODQGPDFUROHYHOPDQDJHPHQWVWUDWHJLHVDQGSROLFLHV+H
DUJXHV WKDW WKH XVHIXOQHVV DQG FRVWHIIHFWLYHQHVV RI 30 ( PD\ EH RSHQ WR TXHVWLRQ XQOHVV VLWHVSHFLILF
UHVHDUFKDQGLQQRYDWLRQVFDQSURYLGHWKHEDVHVIRUGHYHORSLQJSODQQLQJDWKLJKHULQVWLWXWLRQDOOHYHOV,QDFDVH
VWXG\ RI FRPPXQLW\EDVHG UHVRXUFH PDQDJHPHQW SUDFWLFHV LQ .HQ\D )RUG HW DO  LQGLFDWH WKDW WKHUH DUH
SURPLVLQJ SURVSHFWV RI WDNLQJ SDUWLFLSDWRU\ ILQGLQJV DW ORFDO OHYHOV DQG XVLQJ WKHP LQ UHJLRQDO LQIRUPDWLRQ
V\VWHPV /RFDO ILQGLQJV LQ WXUQ FUHDWH SRVVLELOLWLHV IRU RUJDQLVLQJ GDWD WKDW FDQ LQIRUP FHQWUDO
JRYHUQPHQWGRQRUSROLF\DQGGHFLVLRQPDNHUVDERXWZK\UHVRXUFHXVHUVSHUIRUPLQSDUWLFXODUZD\VRU DGRSW
VSHFLILFSUDFWLFHV7KH05'3SURJUDPPHLQ9LHWQDPDOVRSUHVHQWVRQHVXFFHVVIXOH[SHULHQFHLQXVLQJPLFUR
OHYHOLQIRUPDWLRQIRUDQQXDOSODQQLQJDWGLVWULFWDQGSURYLQFLDOOHYHOV
,Q %ROLYLD WKH 3'&5 5XUDO &RPPXQLW\ 'HYHORSPHQW 3URMHFW RU 3UR\HFWR GH 'HVDUUROOR GH &RPXQLGDGHV
5XUDOHV XQGHU WKH 9LFH0LQLVWU\ RI 3RSXODU 3DUWLFLSDWLRQ 9LFHPLQLVWHULR GH 3DUWLFLSDFLyQ 3RSXODU \ )RUWDOHFLPLHQWR
0XQLFLSDO HVWDEOLVKHG DQG LPSOHPHQWHG D V\VWHP RI PRQLWRULQJ DQG HYDOXDWLQJ SURJUDPPH RSHUDWLRQV  ZKLFK
PD\EHDSSOLHGDWQDWLRQDOSURYLQFLDOPXQLFLSDODQGORFDOFRPPXQLW\OHYHOV$OFRFHU/L]iUUDJD'HOJDGLOORHWDO

$QRWKHU VLJQLILFDQW LVVXH RI VFDOLQJ XS SHUWDLQV WR FKDQJHV LQ RUJDQLVDWLRQDO SURFHGXUHV DQG
FXOWXUH

DWWLWXGHVDWKLJKHULQVWLWXWLRQDOOHYHOVZLWKUHVSHFWWR0 (7KLVLQYROYHVDVVHVVLQJWRZKDWH[WHQWKLJKHUOHYHO

LQVWLWXWLRQV²GRQRUVDQGJRYHUQPHQWDJHQFLHV²DJUHHWRDZLGHUVKDULQJRISRZHUDQGFRQWUROHVSHFLDOO\RYHU
IXQGLQJUHVRXUFHVDQGGHFLVLRQPDNLQJ
$V GHPRQVWUDWHG LQ D VXUYH\ RI GRQRU SROLF\ GRFXPHQWV WKHUH VHHPV WR EH FRQFHUWHG LQWHUHVW DQG
UHFHSWLYLW\RQWKHSDUWRIGRQRUVDQGIXQGLQJDJHQFLHVWRLQWHJUDWHSDUWLFLSDWRU\WHFKQLTXHVDQGWKHFRQFHSWRI
SDUWLFLSDWLRQ LQ HYDOXDWLRQ SUDFWLFH VHH 5XGTYLVW DQG :RRGIRUG%HUJHU  6HYHUDO FDVH VWXGLHV ² VXFK DV
WKRVHZKLFKGHVFULEH3,'2:
VH[SHULHQFHZLWKVHOIHYDOXDWLRQ&&'%
VSDUWLFLSDWRU\PRQLWRULQJDSSURDFKDQG
WKH 05'3
V HYDOXDWLRQ IUDPHZRUN ² IXUWKHU LOOXVWUDWH KRZ SDUWLFLSDWRU\ GHFLVLRQ PDNLQJ LV RFFXUULQJ DW DOO
PDQDJHPHQWOHYHOV


 +RZHYHUPDQ\DUJXHWKDWUHDOVWXPEOLQJEORFNVVWLOOUHPDLQZLWKUHJDUGWRV\VWHPDWLVLQJ30 (SUDFWLFH
LQWR SROLF\ )RU LQVWDQFH $UPRQLD DQG &DPSLODQ  QRWH LQ WKHLU VXUYH\ WKDW SHRSOH IRXQG LW GLIILFXOW WR
FDUU\ RXW SDUWLFLSDWRU\ HYDOXDWLRQV ZLWKLQ EXUHDXFUDWLF RUJDQLVDWLRQV EHFDXVH RI LQVWLWXWLRQDO UHVLVWDQFH WR WKH
VKDULQJ RI GHFLVLRQPDNLQJ SRZHU HJ EHWZHHQ VXSSRUWLQJ RUJDQLVDWLRQV DQG SURMHFW LPSOHPHQWHUV
EHQHILFLDULHV $OFRFHU /L]iUUDJD 'HOJDGLOOR HW DO DOVR IRXQG LQ WKHLU VXUYH\ RI %ROLYLD WKDW GHVSLWH VWDWH OHJDO
UHIRUPV SURPRWLQJ SDUWLFLSDWLRQ LQ ORFDO JRYHUQPHQW WKHUH VWLOO UHPDLQ ZLWKLQ RUJDQLVDWLRQV D JHQHUDO ODFN RI
DZDUHQHVVDQGLQDSSURSULDWHXVHRISDUWLFLSDWRU\PHWKRGVHVSHFLDOO\ZLWKUHJDUGWRPRQLWRULQJDQGHYDOXDWLRQ
  0DUVGHQ DQG 2DNOH\  IXUWKHU FRPPHQW WKDW IXQGLQJ FRQVWUDLQWV DQG OLQNDJHV LQ 30 ( PD\
DFWXDOO\GLVFRXUDJHUHDOPXWXDOSDUWQHUVKLSVIURPGHYHORSLQJEHWZHHQVWDNHKROGHUVDWGLIIHUHQWOHYHOVWKHUHIRUH
FUHDWLQJ SUREOHPV IRU VFDOLQJ XS WKH 30 ( SURFHVV 5XGTYLVW DQG :RRGIRUG%HUJHU   DUJXH WKDW
SDUWLFLSDWRU\HYDOXDWLRQVWLOOUHPDLQVYHU\PXFKDQDSSURDFKVHSDUDWHIURPWKHPDLQVWUHDPRIHYDOXDWLRQZKHUH
FRQYHQWLRQDO TXDQWLWDWLYH PHWKRGV DUH VWLOO YDOXHG E\ GRQRU DQG JRYHUQPHQW DJHQFLHV LQ WHUPV RI JHQHUDWLQJ

REMHFWLYH
DQG
SUHFLVH
GDWD

7KH1HHGIRUIXUWKHU'RFXPHQWDWLRQRI30 (
:KLOHWKHOLWHUDWXUHRQ30 (JHQHUDOO\UHFRJQLVHVWKHYDOXHDQGLPSRUWDQFHRIGRFXPHQWLQJH[SHULHQFHVWKH
V\VWHPDWLFGRFXPHQWDWLRQRI30 (LVUDUHO\XQGHUWDNHQLQDFWXDOH[SHULHQFH%DVHGIURPWKHFDVHVWXGLHVZH
VXUYH\HG JUHDWHU HPSKDVLV DSSHDUV WR EH RQ WKH GRFXPHQWDWLRQ RI WKH ILQGLQJV DQG UHVXOWV RI SDUWLFLSDWRU\
HYDOXDWLRQ UDWKHU WKDQ WKH SURFHVV RI FDUU\LQJ RXW 30 ( LWVHOI :KLOH WKH GRFXPHQWDWLRQ RI 30 (
SURFHVVHVLVUDUHO\KLJKOLJKWHGLQWKHOLWHUDWXUHLWVHHPVHYLGHQWWKDWWKHVHSURFHVVHV²LQRWKHUZRUGVWKHYHU\
QDWXUH RI KRZ 30 ( LV DFWXDOO\ FRQGXFWHG ² YHU\ PXFK LQIOXHQFHV DQG VKDSHV WKH RXWFRPHV RI ZKDW LV
OHDUQHGDQGZKDWLQIRUPDWLRQLVREWDLQHGIURP30 (SUDFWLFH
$UPRQLD DQG &DPSLODQ  DOVR SRLQW RXW WKLV JDS LQ WKHLU VXUYH\ RI $VLDQ FDVHV 7KH\ IRXQG OLWWOH
HYLGHQFHRIGHOLEHUDWHHIIRUWVWRGRFXPHQW30 (H[SHULHQFHV7KH\REVHUYHGWKDWSURFHVVHVDUHQRWH[SOLFLWO\
GLVFXVVHGDQGDUHEDUHO\UHIOHFWHGLQJHQHUDOSURMHFWUHSRUWVZKHUHWKH\DUHPHQWLRQHGLQSDVVLQJRUIRXQGLQ
XQFRQVROLGDWHGILHOGQRWHVDV
UDZGDWD
VHHDOVR5XGTYLVWDQG:RRGIRUG%HUJHU0RVWFDVHVWXGLHV
GHVFULEHWKHVWHSVDQGDSSURDFKHVXVHGIRUFDUU\LQJRXW30 (EXWZLWKRXWHODERUDWLQJRQKRZDQGXQGHUZKDW
FRQGLWLRQVDQGFRQWH[WVYDULRXVVWDNHKROGHUVFRPHWRJHWKHUWRPDNHFULWLFDOGHFLVLRQVDERXWWKHSURFHVVQRUGR
PRVWFDVHVHODERUDWHRQKRZVWDNHKROGHUSHUFHSWLRQVDWWLWXGHVDQGEHKDYLRXUVDIIHFWWKHZD\WKH\FRQWULEXWHWR
30 ( DFWLYLWLHV 6LPLODUO\ ZKLOH WKHUH DUH PXOWLSOH UHDVRQV JLYHQ WR DIILUP WKH EHQHILWV RI 30 ( ZH KDYH
IRXQGUHODWLYHO\IHZVWXGLHVZKLFKGRFXPHQWLWVLPSDFWDQGRXWFRPHVRYHUWLPH
$VUDLVHGGXULQJWKH6RXWK²6RXWK6KDULQJ)RUXPWKHIRUPRIGRFXPHQWDWLRQVKRXOGDGGUHVVWKHQHHGVRI
GLIIHUHQWVWDNHKROGHUVEDVHGRQWKHIROORZLQJNH\TXHVWLRQV

• ZKRQHHGVWKHLQIRUPDWLRQ"
• ZKDWLVWKHPRVWDSSURSULDWHPRGHRIGRFXPHQWLQJGDWD"
• ZKRVHDQDO\VLV"


'HSHQGLQJ RQ ZKR QHHGV WKH LQIRUPDWLRQ WKH UHVXOWV RI 30 ( VKRXOG EH GLVVHPLQDWHG LQ D IRUPDW DQG
ODQJXDJHWKDWLVDSSURSULDWHWRWKHXVHU
VH[LVWLQJFDSDFLWLHVUHVRXUFHVDQGQHHGV'RFXPHQWDWLRQDOVRUHIOHFWV
JDLQLQJ RZQHUVKLS RI LQIRUPDWLRQ ,Q WKLV UHJDUG DQDO\VLV RI GRFXPHQWHG GDWD VKRXOG EH XQGHUWDNHQ E\
VWDNHKROGHUVWKHPVHOYHVDFFRUGLQJWRWKHLULQWHUHVWVDQGQHHGV3$0)25.

7KHQHHGIRUSDUWLFLSDWRU\PRQLWRULQJRI30 (
:KLOH30 (RIIHUVPDQ\SRWHQWLDOEHQHILWVLQWHUPVRISURMHFWRUSURJUDPPHVXFFHVVLILWLVFDUULHGRXWSRRUO\
RULQDSSURSULDWHO\WLPHDQGUHVRXUFHVPD\JRWRZDVWHDQGSUREOHPVPD\YHU\ZHOOJRXQQRWLFHGVXEVHTXHQWO\
KLQGHULQJSURMHFWSHUIRUPDQFHDQGFRPPXQLW\EXLOGLQJ,QRUGHUWRJXDUGDJDLQVWWKHVHLQKHUHQWGDQJHUVWKHUH
DOVRVHHPVWREHEURDGDJUHHPHQWLQWKHOLWHUDWXUHUHJDUGLQJWKHQHHGIRUV\VWHPDWLFDQGSDUWLFLSDWRU\SURFHGXUHV
WRPRQLWRUDQGHYDOXDWHWKH30 (SURFHVVLWVHOIVHH&21&(51)HXHUVWHLQ5XELQ6FRWW
9LOOLHUVD
30 (RI30 (
LVUHJDUGHGDVDPHDQVWRFRQWLQXRXVO\XSGDWHDQGLPSURYHWKHSURFHVVLWVHOI
RUDVDPHFKDQLVPZKHUHE\SDUWLFLSDQWVFDQJDXJHZKHWKHUWKH\DUHREWDLQLQJWKHLQIRUPDWLRQWKH\QHHG WKH
WHFKQLTXHVXVHGDUHDSSURSULDWHDQGWKHSURFHVVDVDZKROHLVRSHUDWLQJDVSODQQHG$GDSWLQJ30 (DFFRUGLQJ
WR FKDQJLQJ FLUFXPVWDQFHV PD\ DOVR SUHYHQW WKH SURFHVV IURP EHFRPLQJ D VWDWLF V\VWHP ZKLFK GLVFRXUDJHV
LQQRYDWLRQFUHDWLYLW\DQGFULWLFDOUHIOHFWLRQ+RZHYHULQSUDFWLFHWKHVH
VHOIUHYLHZ
 IHHGEDFN PHFKDQLVPV DUH
UDUHO\XQGHUWDNHQRUQRWFOHDUO\VSHFLILHGLQWHUPVRIDFWXDORSHUDWLRQDOSURFHGXUHVDQGDUHXVXDOO\RQO\FDUULHG
RXWE\FRXQWU\RULQWHUQDWLRQDOKHDGRIILFHVVHH&21&(51
6HYHUDORIWKHFDVHVWXGLHVUHYLHZHGKHUHVHUYHWRLOOXVWUDWHKRZ
30 (RI30 (
PD\EHFRQGXFWHGDQG
ZKDWLWPLJKWFRQWULEXWHWRWKHRYHUDOO30 (SURFHVV$EERWDQG*XLMWGHVFULEHKRZRQHLPSRUWDQW
REMHFWLYHRIPRQLWRULQJWKHPRQLWRULQJSURFHVVLQ%UD]LOZDVWRDVVHVVWKHTXDOLW\RIWKHZRUNFDUULHGRXWWKXV
IDUE\LQYROYLQJWKHYDULRXVVWDNHKROGHUV,QWKLVFDVHUHSUHVHQWDWLYHVIURPWKH1*2DQG5XUDO:RUNHUV
7UDGH
8QLRQDVZHOODVWKHIDFLOLWDWRUVHOHFWHGVHYHUDOFULWHULDWRDVVHVVWKHPHWKRGVDQGLQGLFDWRUVXVHG5HIHUULQJWR
WKHVDPHFDVHVWXG\LQ%UD]LO*XLMWDQG6LGHUVN\IXUWKHUSRLQWRXWWKDWDOORZLQJIRUFRQWLQXDOFKDQJHDQG
LPSURYHPHQWWRWKH30 (V\VWHPXOWLPDWHO\UHVWVRQEXLOGLQJORFDOFDSDFLWLHVVRWKDWIDUPHUVNQRZZKHQDQG
KRZWRDGDSWWKHFXUUHQWV\VWHP,QWKHFDVHRI3,'2:6RPPHUHPSKDVLVHVWKDWRQHLPSRUWDQWIHDWXUH
RIWKHVHOIHYDOXDWLRQH[HUFLVHZDVWRDOORZIRUFRUUHFWLRQDQGPRGLILFDWLRQPLGZD\WKURXJKWKH6(SURFHVVLQ
RUGHUWRSUHYHQWWKHSURFHVVIURPEHFRPLQJPHFKDQLFDODQGWRXVHSUHOLPLQDU\ILQGLQJVIRUDFKLHYLQJPRUHLQ
GHSWKDQDO\VLV

,GHQWLI\LQJWKHHQDEOLQJDQGFDSDFLW\EXLOGLQJIDFWRUV
,IZHDUHFRQFHUQHGZLWKVSUHDGLQJWKHSUDFWLFHRI30 (ZHDOVRQHHGWRXQGHUVWDQGDJUHDWGHDOPRUHDERXW
WKH HQDEOLQJ IDFWRUV LH WKH NLQGV RI FRQGLWLRQV DQG FRQWH[WV ZKLFK HQDEOH RU PDNH SRVVLEOH D VXFFHVVIXO
XQGHUWDNLQJ RI WKH 30 ( SURFHVV &XUUHQWO\ WKHUH LV QR FRQVHQVXV LGHQWLI\LQJ XQGHU ZKDW FLUFXPVWDQFHV
30 ( LV DSSURSULDWH DQG HIIHFWLYH &RXVLQV DQG :KLWPRUH   $V 0F$UWKXU   VWDWHV

(YDOXDWLRQV QHHG WR LQGLFDWH QRW RQO\ ZKDW KDV ZRUNHG DQG ZK\ EXW DOVR XQGHU ZKDW VRFLDO HFRQRPLF DQG
HQYLURQPHQWDOFRQGLWLRQVFDQDSDUWLFXODUWHFKQRORJ\RULQQRYDWLRQEHUHSOLFDWHG



 'XULQJ WKH 6RXWK²6RXWK 6KDULQJ )RUXP RI $IULFDQ H[SHULHQFHV VHYHUDO IDFWRUV ZHUH LGHQWLILHG DV
LPSRUWDQWIRUDVXVWDLQDEOHDQGHIIHFWLYH30 (V\VWHPQDPHO\

• FRPPXQLW\LQYROYHPHQWDQGRZQHUVKLS
• DFFRXQWDELOLW\DQGHPSRZHUPHQW
• DFWLYHOREE\LQJDQGDGYRFDF\IRU30 (

+RZHYHU SDUWLFLSDQWV DOVR LGHQWLILHG VHYHUDO NH\ FRQVWUDLQWV WR 30 ( SUDFWLFH ZKLFK KLJKOLJKW SUREOHPDWLF
LVVXHVUHODWHGWRPHWKRGRORJ\UHVRXUFHUHTXLUHPHQWVLQFOXGLQJFDSDFLWLHVDQGWLPHDYDLODELOLW\DVZHOODVVRFLR
HFRQRPLFDQGSROLWLFDOFRQGLWLRQV3$0)25.
%DVHGRQRXUUHYLHZRIOLWHUDWXUHWKHUHVWLOOUHPDLQGLIIHUHQFHVLQRSLQLRQUHJDUGLQJWKHSDUWLFXODUFRQWH[WV
LQ ZKLFK 30 ( PD\ EH DSSURSULDWH DQG LQDSSURSULDWH 6RPH DUJXH WKDW30 ( PD\ QRW EH DV HIIHFWLYH DQG
XVHIXOLQSURMHFWVWKDWGRQRWDOUHDG\LQFRUSRUDWHDSDUWLFLSDWRU\DSSURDFKLQLWVGHVLJQDQGLPSOHPHQWDWLRQ)RU
LQVWDQFH5XGTYLVWDQG:RRGIRUG%HUJHUDUJXHWKDW
WKHLQWURGXFWLRQRISDUWLFLSDWRU\WHFKQLTXHVLQ
SURMHFWVWKDWKDYHQRWIROORZHGDSDUWLFLSDWRU\DSSURDFKLQSODQQLQJPD\KDYHWRILQGZD\VLHUHVFKHGXOLQJ
SUHVFULEHGWLPHIUDPHVDQGUHRULHQWLQJGHFLVLRQPDNLQJSURFHGXUHVWRGHDOZLWKFHUWDLQLQKHUHQWFRQWUDGLFWLRQV
DQG SUDFWLFDO GLIILFXOWLHV
 $OFRFHU /L]iUUDJD 'HOJDGLOOR HW DO DOVR IRXQG WKDW LQ %ROLYLD
VWUXFWXUDO
 OLPLWV WR
SDUWLFLSDWLRQ DUH YDULHG LQFOXGLQJ VRFLRFXOWXUDO HFRQRPLF JHRJUDSKLFDO HQYLURQPHQWDO SROLWLFDO DQG OHJDO
IDFWRUVLHOLPLWHGKXPDQDQGHFRQRPLFUHVRXUFHVDQGFDSDFLWLHVDSDWK\IRUPDOEXUHDXFUDFLHVHWF²
2WKHUVWDNHWKHYLHZWKDW30 (PD\VWLOOEHFDUULHGRXWHYHQLISURMHFWVZHUHQRWLQLWLDOO\GHVLJQHGLQD
SDUWLFLSDWRU\ PDQQHU 7KH GHJUHH RI LQFRUSRUDWLQJ SDUWLFLSDWLRQ LQ 0 ( ZLOO GHSHQG RQ YDULRXV IDFWRUV
LQFOXGLQJGHJUHHRIZLOOLQJQHVVDQGFRPPLWPHQWRIDOOVWDNHKROGHUVDYDLODELOLW\RIWLPHDQGUHVRXUFHVH[WHUQDO
FRQVWUDLQWV DQG WKH DYDLODELOLW\ RI EDVHOLQH GDWD &DPSRV DQG &RXSDO  $V WKH &6,3 SURMHFW LQ &DOFXWWD
GHPRQVWUDWHV D SDUWLFLSDWRU\ LPSDFW DVVHVVPHQW FRXOG VWLOO EH FRQGXFWHG HIIHFWLYHO\ HYHQ LI WKH SURMHFW ZHUH
LQLWLDOO\GHVLJQHGWRS²GRZQZLWKRXWSULRUFRQVXOWDWLRQZLWKORFDOSHRSOH.DUHWDOE$OWKRXJKLQWKLV
FDVHWKHUHZHUHFHUWDLQOLPLWVWRORFDOSDUWLFLSDWLRQLHORFDOHOLWHJURXSVGRPLQDWLQJLQIRUPDWLRQJDWKHULQJDQG
OLPLWHG ORFDO LQYROYHPHQW LQ GDWD DQDO\VLV WKH 3,$ SURFHVV QHYHUWKHOHVV GLG UHVXOW LQ ORFDO SHRSOH EHFRPLQJ
PRUHDZDUHRIFRPPXQLW\FRQGLWLRQVDQGLQVRPHLQVWDQFHVSHRSOHWKHPVHOYHVWDNLQJDFWLRQWRLPSURYHWKHLU
VXUURXQGLQJVLELG

&RQFOXVLRQ
2YHU WKH ODVW WZR GHFDGHV SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ KDV EHHQ XVHG LQ KXQGUHGV RI SURMHFWV LQ
GLIIHULQJFRQWH[WVDQGSURJUDPPHVDFURVVWKHZRUOG,QWKLVSDSHUZHKDYHSURYLGHGDUHYLHZRIVRPHRIWKHVH
H[SHULHQFHV GUDZLQJ IURP WKHP GLVFXVVLRQV RI WKH SXUSRVHV DQG SULQFLSOHV WKDW XQGHUSLQ WKH SUDFWLFH RI
SDUWLFLSDWRU\ PRQLWRULQJ DQG HYDOXDWLRQ )LQDOO\ LQ WKLV ODVW VHFWLRQ ZH KDYH LGHQWLILHG IRXU EURDG WKHPHV ²
XQGHUVWDQGLQJ 30 ( DV D VRFLDO SURFHVV PHWKRGRORJLFDO LVVXHV WKH LQVWLWXWLRQDOLVDWLRQ DQG VFDOLQJ XS RI
30 (DQGWKHQHHGIRUPRUHGRFXPHQWDWLRQ²ZKLFKSRVHFKDOOHQJHVIRUIXWXUHUHVHDUFKDQGLPSOHPHQWDWLRQ
7KHVHDUHODUJHLVVXHVZKLFKDOVRIDFHEURDGHUDVSHFWVRISDUWLFLSDWRU\UHVHDUFKDVZHOO7KHLVVXHVFDQQRWEH
DGGUHVVHGHDVLO\EXWDVDQGZKHQWKH\DUHZHZLOOGHHSHQRXUNQRZOHGJHDQGRXUSUDFWLFHLQWKHILHOG


127(6
7KHVHLQFOXGH3DWWD6FRWW9LOOLHUV(VWKHU0HEUDKWX.DWMD-REHV-DPHV%ODFNEXUQ$QGUHD&RUQZDOODQG
-XWWD%ODXHUW7KDQNVDOVRWR-XOLHW0HUULILHOGIRUUHDGLQJDQGFRPPHQWLQJRQHDUOLHUGUDIWVDQGWR6LQQHW:HEHU
DQG-HQQ\(GZDUGVIRUZRUGSURFHVVLQJDVVLVWDQFH
$SDSHURQ
3DUWLFLSDWRU\(YDOXDWLRQ²LWVUROHDQGFRQWULEXWLRQVLQLPSURYLQJSXEOLFVHUYLFHV
²LV
IRUWKFRPLQJ
'HYHORSHGE\WKH/HDUQLQJ,QLWLDWLYH&RPPXQLW\3DUWQHUVKLS&HQWHU7KH8QLYHUVLW\RI7HQQHVVHH
.QR[YLOOH6LPLODU
ZKHHOV
KDYHEHHQGHYHORSHGE\:RRGKLOODQG5RELQVDQG1HZ(FRQRPLFV
)RXQGDWLRQ
2WKHUYHUVLRQVRIWKLVDFURQ\PLQFOXGH6LPSOH0HDVXUDEOH$WWUDFWLYH5HOLDEOH7LPHO\²RU6LPSOH
0HDVXUDEOH$FFXUDWH5HOLDEOHDQG7LPHERXQG
0RUHGHWDLOHGGLVFXVVLRQVDERXW35$LWVKLVWRU\DQGSUDFWLFHDUHDYDLODEOHLQWKHOLWHUDWXUHVHHIRUH[DPSOH
&KDPEHUVDE&KDPEHUVDQG*XLMW&RUQZDOODQG-HZNHV


$S SHQ G L[ &DV H([ DPS OH RI 3DU W LF LSD W RU\ 0 RQ LW RU LQJ DQ G( YDOX DW LR Q

)HDWXUHV &RXQWU\ 6HFWRU 3ULPDU\ .H\ 7RROV0HWKRGV8VHG

&DVH6WXG\ 5HJLRQ )XQFWLRQV3XUSRVHV $FWRUV3DUWLFLSDQWV

6HOIHYDOXDWLRQLQ .DUQDWDND,QGLD ,QWHJUDWHGZDWHUVKHG 2UJDQLVDWLRQDO 6'&GRQRU ²VHPLVWUXFWXUHGJURXSGLVFXVVLRQV

3,'2:SURMHFW GHYHORSPHQW ,QVWLWXWLRQDOOHDUQLQJ *RYHUQPHQWDJHQFLHV ²YLOODJHZRUNVKRSV
1*20<5$'$ ²ILHOGWUDQVHFWV
/RFDOZDWHUVKHG ²KLVWRULFDOSURILOHV
RUJDQLVDWLRQV ²UDQNLQJ
²PDSSLQJ

3URMHFWR3DUDLED 1RUWKHDVW%UD]LO $JULFXOWXUH ,PSDFWDVVHVVPHQW $637$SURMHFWVWDII ²ELRSK\VLFDOVFLHQWLILFPHDVXUHPHQWV

$637$1*2 3URMHFWPDQDJHPHQWDQG 5XUDO:RUNHUV
7UDGH ²PDSSLQJ
SODQQLQJ 8QLRQUHSUHVHQWDWLYHV ²GLDJUDPPLQJ
/RFDOIDUPHUV ²FULWLFDOLQFLGHQWDQDO\VLVWRDVVHVV
LQVWLWXWLRQDOUHODWLRQVKLSV
²
REMHFWLYHWUHH
IRULGHQWLI\LQJSULRULWLHV

9LHWQDP6ZHGHQ 1RUWKHUQ ,QWHJUDWHGUXUDO 3URJUDPPH 1DWLRQDOSURYLQFLDO ²KRXVHKROGLQWHUYLHZLQJ

0RXQWDLQ5XUDO 9LHWQDP GHYHORSPHQW PDQDJHPHQWDQG DQGGLVWULFWJRYHUQPHQW ²ILHOGVXUYH\V
'HYHORSPHQW SODQQLQJ DXWKRULWLHV ²LQWHUHVWJURXSGLVFXVVLRQV
3URJUDPPH05'3 9LOODJHPDQDJHPHQW ²YLOODJHPHHWLQJV
JURXSFRPPXQH ²SKRWRGRFXPHQWDWLRQ
DXWKRULWLHV ²GLDJUDPPLQJ
/RFDOIDUPHUV ²ZHDOWKUDQNLQJ
²KRXVHKROGOLYHOLKRRGDQDO\VLV
²ODQGXVHWUDQVHFWVHWF

1HSDO5HVRXUFH 5DPHFKDSDQG )RUHVWFRQVHUYDWLRQ ,PSDFWDVVHVVPHQW 1503SURMHFWWHDP ²FDVHKLVWRULHV

0DQDJHPHQW3URMHFW 'KDGLQJGLVWULFWV DQGPDQDJHPHQW 6WDNHKROGHUSHUVSHFWLYHV 9LOODJHUV ²OLIHKLVWRULHV
1503 1HSDO HQYLURQPHQW ²PDSSLQJ
²GLDJUDPPLQJ
²GUDPD
²VRQJV


$SSHQGL[&DVH([DPSOHRI3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ

)HDWXUHV &RXQWU\ 6HFWRU 3ULPDU\ .H\ 7RROV0HWKRGV8VHG

&DVH6WXG\ 5HJLRQ )XQFWLRQV3XUSRVHV $FWRUV3DUWLFLSDQWV

)DPLO\/LIH .HQ\D +HDOWK1XWULWLRQ 6WDNHKROGHUSHUVSHFWLYHV ([WHUQDOHYDOXDWLRQWHDP ²WLPHOLQHVWUHQGV

7UDLQLQJ3URJUDPPH &KLOGFDUH 3URYLQFLDODQGGLYLVLRQDO ²YHQQGLDJUDPV
)/73 JRYHUQPHQWDXWKRULWLHV ²NH\LQIRUPDQWLQWHUYLHZV
3URMHFWPJPW6WDII ²IRFXVJURXSGLVFXVVLRQV
&RPPXQLW\RUJDQLVDWLRQV ²REVHUYDWLRQ
DQGYLOODJHUV ²UDQNLQJ
²FDUGVRUWLQJH[HUFLVHV
&(7$0(; 2D[DFDVRXWKHUQ $JULFXOWXUH 2UJDQLVDWLRQDO ([WHUQDOHYDOXDWLRQWHDP ²RUJDQLVDWLRQDOHWKQRJUDSKLHVIRFXV
1RFKL[WOiQ 0H[LFR ,QVWLWXWLRQDOOHDUQLQJ &(7$0(;DQG:RUOG RQJURXSVZPRUHFOHDUO\GHILQHG
&(7$0(;1 3URJUDPPH0JPWDQG 1HLJKERUV1*2 ERXQGDULHVDVRSSRVHGWRVWXGLHVRI
SODQQLQJ IXQGHUV ZKROHVRFLHWLHV
3URMHFWIDUPHU ²RUDOKLVWRULHV
H[WHQVLRQLVWV ²GLDJUDPPLQJLQFOXGLQJYHQQ
/RFDOFRPPXQLWLHVDQG GLDJUDPV
IDUPLQJIDPLOLHV ²VHDVRQDOFDOHQGDUV
²WUHQGDQDO\VLV
²PDWUL[UDQNLQJ
²PDSSLQJ

&KULVWLDQ %DQJODGHVK :RPHQ 2UJDQLVDWLRQDO &&'%GRQRUV ²PRQWKO\UHSRUWLQJDQG

&RPPLVVLRQIRU *URXSEDVHGVDYLQJV ,QVWLWXWLRQDOOHDUQLQJ +HDGRIILFHVWDII PHHWLQJVIHHGEDFNVHVVLRQV
'HYHORSPHQWLQ DQGFUHGLWPDLQO\ 3URMHFWRIILFHVHQLRUDQG
%DQJODGHVK&&'% GLUHFWHGWRZDUGV MXQLRUVWDII
ZRPHQ /RFDOEHQHILFLDU\
UHSUHVHQWDWLYHV


$SSHQGL[&DVH([DPSOHRI3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ

)HDWXUHV &RXQWU\ 6HFWRU 3ULPDU\ .H\ 7RROV0HWKRGV8VHG

&DVH6WXG\ 5HJLRQ )XQFWLRQV3XUSRVHV $FWRUV3DUWLFLSDQWV

&DWFKPHQW .HQ\D (QYLURQPHQW 2UJDQLVDWLRQDO *RY

WPLQLVWU\RIILFHUV ²SXEOLFPHHWLQJV
DSSURDFKWRVRLODQG $JULFXOWXUH ,QVWLWXWLRQDOOHDUQLQJ 3URJUDPPHVWDII ²SHUVRQDOGLDULHVSURFHVVQRWHV
ZDWHUFRQVHUYDWLRQ 3URJUDPPH0JPW /RFDOIDUPHUV ²IRFXVJURXSGLVFXVVLRQV
6 :&RIWKH SODQQLQJ ²NH\LQIRUPDQWLQWHUYLHZV
0LQLVWU\RI ²RUDOKLVWRULHV
$JULFXOWXUH ²WUHQGDQDO\VLV
²WUDQVHFWZDONV
²ZHDOWKUDQNLQJ
²PDSSLQJ
²GLDJUDPPLQJHWF

-RLQWHYDOXDWLRQWHDP ²WLPHOLQHV
&DOFXWWD6OXP ,QGLD 8UEDQ&RPPXQLW\ ,PSDFW
LQFOXGLQJFRQVXOWDQWVDQG ²VRFLDOPDSSLQJ
,PSURYHPHQW3URMHFW GHYHORSPHQW DVVHVVPHQW3URFHVV
SURMHFWVWDII ²VHDVRQDOFDOHQGDU
&6,3 HYDOXDWLRQ
/RFDOUHVLGHQWVE\VRFLDO ²YHQQGLDJUDPV
JURXSLQJZRPHQ\RXWK ²IORZGLDJUDPV
SROLWLFDOOHDGHUVHOGHUV ²JURXSGLVFXVVLRQV
HWF
+HDOWK&DUHIRU 86$ +HDOWK 3XEOLFDFFRXQWDELOLW\ 3URJUDPPHVWDII ²WHOHSKRQHQHWZRUNLQIRUPDWLRQ
$OO+&)$ /RFDOFLWL]HQV KHOSOLQH
²RUJDQLVHJURXSVLQWRVLQJOHLVVXH
FDPSDLJQV
²WUDLQLQJLQUHVHDUFKDQGILQDQFLDO
DQDO\VLV
²GLVWULEXWLRQRIWUDLQLQJHGXFDWLRQDO
PDWHULDOV


$SSHQGL[&DVH([DPSOHRI3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ

)HDWXUHV &RXQWU\ 6HFWRU 3ULPDU\ .H\ 7RROV0HWKRGV8VHG

&DVH6WXG\ 5HJLRQ )XQFWLRQV3XUSRVHV $FWRUV3DUWLFLSDQWV

&RPPXQLW\ 5XUDOUHJLRQVRI &RPPXQLW\ 3XEOLFDFFRXQWDELOLW\ /HDUQLQJWHDPPHPEHUV ²LQWHUYLHZV

3DUWQHUVKLS&HQWUH WKH86$ GHYHORSPHQW &RRUGLQDWRUV ²IRFXVJURXSGLVFXVVLRQV
/HDUQLQJ,QLWLDWLYH 5HJLRQDOUHVHDUFKHUV ²SXEOLFGRFXPHQWDWLRQUHYLHZ
DVVLJQHGWROHDUQLQJ ²FRQWHQWDQGPHGLDDQDO\VLV
WHDPV ²TXHVWLRQQDLUHV
/RFDOOHDGHUVDQG ²PDSSLQJ
FRPPXQLWLHV ²SHUVRQDOREVHUYDWLRQ
JRYHUQPHQWDODXWKRULWLHV ²ZLQGVKLHOGVXUYH\V
DWORFDODQGQDWLRQDOOHYHOV ²RUDOKLVWRULHV
²HGXFDWLRQDOWHVWVFRUHVHWF

1RUWKHUQ *KDQD 5XUDOGHYHORSPHQW ,PSDFWDVVHVVPHQW ([WHUQDOHYDOXDWLRQWHDP ²KRXVHKROGTXDQWLWDWLYHVXUYH\V

5HJLRQDO5XUDO ZDWHUVXSSO\DQG &,'$GRQRU ²IRFXVJURXSGLVFXVVLRQVYLOODJH
,QWHJUDWHG VDQLWDWLRQ *RYHUQPHQWOLQH PHHWLQJVZRUNVKRSV
3URJUDPPH LPSURYHPHQW DJHQFLHV ²9LOODJH'HYHORSPHQW&DSDFLW\,QGH[
1255,3 1255,3VWDII WRRO9'&,
9LOODJHOHDGHUVDQG
UHVLGHQWV
,PSDFWDVVHVVPHQW
3URWRW\SH ,QGLD :RPHQ 3URJUDPPHVWDIIRI ²JURXSGLVFXVVLRQVDQGPRQWKO\
3URJUDPPH
,QWHUQDO/HDUQLQJ *URXSEDVHGVDYLQJV LPSOHPHQWLQJ1*2V PHHWLQJV
PDQDJHPHQWDQG
6\VWHPIRU DQGFUHGLW 6(:$35$'$1 ²SLFWRULDOGLDULHV
SODQQLQJ
/LYHOLKRRGDQG SURJUDPPHVIRUSRRU )::%
0LFUR&UHGLW ZRPHQ :RPHQ
VJURXSVDQG
3URJUDPPHV LQGLYLGXDOPHPEHUV
)DUPHU 3KLOLSSLQHV $JULFXOWXUH1DWXUDO ,PSDFWDVVHVVPHQW 2XWVLGHUHVHDUFKHUV ²ELRUHVRXUFHIORZGLDJUDPV
3DUWLFLSDWRU\ UHVRXUFHPDQDJHPHQW 3URJUDPPH /RFDOIDUPHUV ²IDUPHUJURXSV
3URFHGXUHV² PDQDJHPHQWDQG ²WLPHVHULHVGDWDDQDO\VLV
)DUPLQJ6\VWHPV SODQQLQJ
5HVHDUFK


$SSHQGL[&DVH([DPSOHRI3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ

)HDWXUHV &RXQWU\ 6HFWRU 3ULPDU\ .H\ 7RROV0HWKRGV8VHG

&DVH6WXG\ 5HJLRQ )XQFWLRQV3XUSRVHV $FWRUV3DUWLFLSDQWV

$&25'
V 0DOL $JULFXOWXUH1DWXUDO 2UJDQLVDWLRQDO $&25'DQ1*2 ²YLOODJHPHHWLQJVDQGZRUNVKRSV
H[SHULHQFHZLWKDXWR UHVRXUFHPDQDJHPHQW ,QVWLWXWLRQDOOHDUQLQJ SURMHFWVWDII ²LQGLYLGXDOJURXSGDWDDQGDFWLYLW\
HYDOXDWLRQ ,PSDFWDVVHVVPHQW /RFDODJURSDVWRUDOLVWV ILOHV
DQGRWKHUFRPPXQLW\ ²DQLPDWLRQWHFKQLTXHV
PHPEHUV
$WWKHHQGRIWKH\HDU
SURJUDPPHZLOODOVR
LQYROYHH[WHUQDO
HYDOXDWRUV6WDWHORFDO
GHYHORSPHQWFRPPLWWHHV
9LOODJHOHYHOVRLO
*XMDUDW,QGLD $JULFXOWXUH1DWXUDO 3URJUDPPH $.563SURJUDPPHVWDII ²PDSSLQJ
DQGZDWHU
UHVRXUFHPDQDJHPHQW PDQDJHPHQWDQG /RFDOO\HOHFWHGYLOODJH ²IDUPHUJURXSGLVFXVVLRQV
FRQVHUYDWLRQ
SODQQLQJ H[WHQVLRQYROXQWHHUV
SURJUDPPH
/RFDOIDUPHUV
VXSSRUWHGE\WKH$JD
.KDQ5XUDO6XSSRUW
3URJUDPPH
$.563

+RUL]RQ3URMHFW 0DLQH86$ $GXOWHGXFDWLRQ 3URJUDPPH +RUL]RQ3URMHFW ²OHDUQHUVIRFXVJURXSVSKRQHLQ

PDQDJHPHQWDQG IDFLOLWDWRUV
FDWDO\VWV
DQG SHUVRQRQOLQHFKDWV
SODQQLQJ LQWHUDJHQF\SURMHFWVWDII ²LQWHUYLHZV
DWVWDWHUHJLRQDODQG ²VXUYH\VSKRQHLQSHUVRQ
ORFDOOHYHOV ²MRXUQDOV
/HDUQHUVLQORFDO$GXOW
(GXFDWLRQSURJUDPPHV
)LHOGH[WHQVLRQLVWV ²IDUPHUV
RZQUHFRUGV
6LDYRQJD =DPELD $JULFXOWXUH 3URJUDPPH
9LOODJHH[WHQVLRQJURXSV ²UHFRUGLQJVKHHWVDQGERRNOHWV
$JULFXOWXUDO PDQDJHPHQWDQG
/RFDOIDUPHUV ²H[WHQVLRQDJHQWV
ZRUNGLDULHV
'HYHORSPHQW3URMHFW SODQQLQJ
²H[WHQVLRQSUHIHUHQFHUDQNLQJ
6$'3


$SSHQGL[&DVH([DPSOHRI3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ

)HDWXUHV &RXQWU\ 6HFWRU 3ULPDU\ .H\$FWRUV 7RROV0HWKRGV8VHG

&DVH6WXG\ 5HJLRQ )XQFWLRQV3XUSRVHV
3DUWLFLSDQWV
3URMHFWIRUWKH 3HUX $JULFXOWXUH 3URJUDPPH ,QWHUGLFLSOLQDU\WHDPRI ²JURXSGLVFXVVLRQVEZIDUPHUVDQG
9DOLGDWLRQRI PDQDJHPHQWDQG SURMHFWUHVHDUFKHUV UHVHDUFKWHDP
7HFKQRORJ\IRU SODQQLQJ DJULFXOWXUDOLVWVDQGVRFLDO ²XVHRIJUDSKVSRVWHUVIRUUHFRUGLQJ
+LJKODQG VFLHQWLVWVORFDOIDUPLQJ GRFXPHQWDWLRQDQGSUHVHQWDWLRQRI
&RPPXQLWLHV FRPPXQLWLHVDQG ILQGLQJV
KRXVHKROGV ²SUHOLPLQDU\FRPPXQLW\PHHWLQJVRQ
SURMHFWRULHQWDWLRQ
²SDUWLFLSDQWREVHUYDWLRQGHVFULSWLYH
UHFRUGLQJE\UHVHDUFKWHDP
0RQLWRULQJ
*OREDO 0XOWLVHFWRUDO 3XEOLFDFFRXQWDELOLW\ :RUOG%DQN ²ZRUNVKRSVRQSDUWLFLSDWRU\
SDUWLFLSDWLRQLQ
*RYHUQPHQWV PRQLWRULQJ
:RUOG%DQN
1*2:RUNLQJ*URXS ²UHJLRQDOPHHWLQJV
SURJUDPPHV
IURP$IULFD$VLD/DWLQ ²JOREDOSDUWLFLSDWLRQFRQIHUHQFH
$PHULFD ²GDWDVKRXOGUHIOHFWTXDOLWDWLYHDQG
&RPPXQLWLHV TXDQWLWDWLYHDVSHFWVRISDUWLFLSDWLRQ


$33(1',;

0$18$/6$1'722/.,76)253$57,&,3$725<021,725,1*$1'
(9$/8$7,21
$Q$QQRWDWHG%LEOLRJUDSK\

$&25'0DOL$XWR(YDOXDWLRQ3URMHFW/RQGRQ$&25'
&RQWDFWLQIRUPDWLRQ
$&25'RU6266DKHO
$YDLODEOHIRUSKRWRFRS\LQJIURP35$UHDGLQJURRP,'6

6266DKHO7ROSXGGOH6WUHHW/RQGRQ12;7
7KLVGRFXPHQWGHVFULEHVWKHDXWRHYDOXDWLRQSURJUDPPHLQWKH*DRDQG7RPERXFWRXUHJLRQVZKHUH
$&25'0DOLLVZRUNLQJWRVXSSRUWLQIRUPDOYLOODJHJURXSVDQGDVVRFLDWLRQV$PHWKRGRORJLFDODSSURDFK
LVRXWOLQHGWKDWLQYROYHVFRPPXQLWLHVIURPSURMHFWGHVLJQWRHYDOXDWLRQ7KHSDSHUGHVFULEHVWKHVWDJHVRI
WKHDXWRHYDOXDWLRQSURFHVVLQZKLFKERWK$&25'VWDIIDQGFRPPXQLWLHVVHWFULWHULDDQGLQGLFDWRUVDVZHOO
DVWDNHSDUWLQWKHHYDOXDWLRQ

$XEHO-XGL3DUWLFLSDWRU\3URJUDP(YDOXDWLRQ$0DQXDOIRU,QYROYLQJ3URJUDP6WDNHKROGHUVLQ
WKH(YDOXDWLRQ3URFHVV6HQHJDO&DWKROLF5HOLHI6HUYLFHV
&RQWDFWLQIRUPDWLRQ
$YDLODEOHWKURXJK3$&71HZ<RUN
%DVHGRQH[SHULHQFHLQ/DWLQ$PHULFD$VLDDQG$IULFDWKHDXWKRUGHWDLOVWKHYDULRXVVWHSVLQYROYHGLQ
SDUWLFLSDWRU\HYDOXDWLRQDQGGUDZVH[DPSOHVIURPILHOGH[SHULHQFH

'DYLV&DVH'
$UF\&RPPXQLW\)RUHVWU\3DUWLFLSDWRU\$VVHVVPHQW0RQLWRULQJDQG(YDOXDWLRQ
5RPH)$2
&RQWDFWLQIRUPDWLRQ
$YDLODEOHDWWKH35$UHDGLQJURRP,'6IRUSKRWRFRS\LQJ
7KLVYROXPHSURYLGHVDJRRGRYHUYLHZDQGGLVFXVVLRQRIWKHFRQFHSWXDODQGPHWKRGRORJLFDOLVVXHVUHODWHG
WRSDUWLFLSDWRU\DVVHVVPHQWPRQLWRULQJDQGHYDOXDWLRQ3$0(DSSURDFK6L[PHWKRGVDUHXVHGZKLFKIDOO
XQGHU3$0(
FRPPXQLW\VLWHVHOHFWLRQ

FRPPXQLW\SUREOHPDQDO\VLV

SDUWLFLSDWRU\EDVHOLQHV


SDUWLFLSDWRU\PRQLWRULQJDQGRQJRLQJHYDOXDWLRQ
DQG
LQIRUPDWLRQDQDO\VLV
DQG
FRPPXQLFDWLRQRI
UHVXOWV
(DFKRIWKHPHWKRGVDUHGLVFXVVHGDWOHQJWKZKLFKLQFOXGHVWKHLUGHVFULSWLRQSXUSRVHJXLGHOLQHV
IRUXVHDQGLPSOHPHQWDWLRQ7KHWRROVRULQVWUXPHQWVDWRWDORIXVHGE\WKH3$0(DSSURDFKDUHWKHQ
GLVFXVVHGLQGHWDLOVSHFLI\LQJWKHLUREMHFWLYHVUHVRXUFHVUHTXLUHGWKHLUSRWHQWLDODSSOLFDWLRQVWKHLUEHQHILWV
DQGDGYDQWDJHVDQGWKHLUOLPLWDWLRQV$QDQQRWDWHGELEOLRJUDSK\SUHVHQWVDVXUYH\RIWKHOLWHUDWXUHUHODWHG
WR3$0(2WKHUDSSOLFDWLRQVIRU3$0(DUHLGHQWLILHGLQRWKHUVHFWRUVVXFKDVKHDOWKFDUHILVKHULHVDQG
DJULFXOWXUH


'DYLV&DVH'DQG3*URYH7KH&RPPXQLW\
V7RRO%R[7KH,GHD0HWKRGVDQG7RROVIRU
3DUWLFLSDWRU\$VVHVVPHQW0RQLWRULQJDQG(YDOXDWLRQLQ&RPPXQLW\)RUHVWU\5RPH)$2
&RQWDFWLQIRUPDWLRQ
)RRGDQG$JULFXOWXUH2UJDQL]DWLRQ 25 81,38%
9LDGHOOH7HUPHGL&DUDFDOOD )$VVHPEO\'ULYH
5RPH,WDO\ /DQKDP0DU\ODQG
7HO 7HO
)D[
7KHILHOGPDQXDOLVGLYLGHGLQWRWKUHHVHFWLRQV6HFWLRQLQWURGXFHVWKHLGHDRISDUWLFLSDWRU\DVVHVVPHQW
PRQLWRULQJDQGHYDOXDWLRQ3$0(DQGSURYLGHVWZRZD\FRPPXQLFDWLRQH[HUFLVHVIRUILHOGVWDII6HFWLRQ
GLVFXVVHVKRZWRGHWHUPLQHLQIRUPDWLRQQHHGVDQGKRZWRDQDO\VHDQGSUHVHQWLQIRUPDWLRQ6HFWLRQ
GHVFULEHVWKHWRROVIRULQIRUPDWLRQJDWKHULQJDQGSURYLGHVJXLGHOLQHVIRUWKHVHOHFWLRQRIWRROV7KHUHDUH
WRROVOLVWHGLQWKLVODWWHUVHFWLRQLQFOXGLQJVHPLVWUXFWXUHGLQWHUYLHZVUDQNLQJPDSSLQJGUDZLQJHWF

)HXHUVWHLQ0DULH7KHUHVH3DUWQHUVLQ(YDOXDWLRQ(YDOXDWLQJ'HYHORSPHQWDQG&RPPXQLW\
3URJUDPPHVZLWK3DUWLFLSDQWV+RQJ.RQJ0DF0LOODQ(GXFDWLRQ/WG
&RQWDFWLQIRUPDWLRQ
7HDFKLQJ$LGVDW/RZ&RVW
32%R[
6W$OEDQV+HUWIRUGVKLUH$/$;
8.
7KLVERRNSURYLGHVDFRPSUHKHQVLYHGLVFXVVLRQRIWKHFRQFHSWXDODQGPHWKRGRORJLFDOLVVXHVUHODWHGWR
30 (ZLWKIUHTXHQWUHIHUHQFHVWRUHDOOLIHILHOGH[SHULHQFHV7KHILUVWFKDSWHUORRNVDWWKHPDLQTXHVWLRQV
WREHDVNHGDVDSUHOLPLQDU\VWHSLQSDUWLFLSDWRU\HYDOXDWLRQ&KDSWHUGHWDLOVWKHSODQQLQJDQGRUJDQLVLQJ
RIHYDOXDWLRQDQGSUHVHQWVGLIIHUHQWW\SHVRIHYDOXDWLRQPHWKRGVRUWRROVWKDWPD\EHHPSOR\HGGHSHQGLQJ
RQHYDOXDWLRQREMHFWLYHV7KHWKLUGFKDSWHUFRQVLGHUVWKHXVHRIH[LVWLQJUHVRXUFHVDQGNQRZOHGJHIRU
HYDOXDWLRQSXUSRVHV&KDSWHUORRNVDWYDULRXVNLQGVRIHYDOXDWLRQWRROVVXFKDVVXUYH\VLQWHUYLHZV
TXHVWLRQQDLUHVGUDZLQJVPDSSLQJYLGHRVHWF,WSURYLGHVVLPSOHYLVXDOLQVWUXFWLRQVRQKRZWRDSSO\
GLIIHUHQWHYDOXDWLRQPHWKRGV7KHILQDOFKDSWHUVFKDSWHUVDQGH[DPLQHWKHZD\VLQZKLFKHYDOXDWLRQ
UHVXOWVFDQEHDQDO\VHGDQGUHSRUWHGDQGVXEVHTXHQWO\FRQWULEXWHWRVWUHQJWKHQLQJDQGPDNLQJ
SURJUDPPHVPRUHHIIHFWLYH


*RVOLQJ/RXLVDDQG0LNH(GZDUGV7RRONLWV$3UDFWLFDO*XLGHWR$VVHVVPHQW0RQLWRULQJ
5HYLHZDQG(YDOXDWLRQ/RQGRQ6DYHWKH&KLOGUHQ
&RQWDFWLQIRUPDWLRQ
6DYHWKH&KLOGUHQ
0DU\%DWFKHORU+RXVH
*URYH/DQH
/RQGRQ6(5'
8.
'LVFXVVHVWKHEURDGXQGHUO\LQJSULQFLSOHVRIDVVHVVPHQWPRQLWRULQJUHYLHZDQGHYDOXDWLRQ'HVFULEHV
GLIIHUHQWDSSURDFKHVWRDVVHVVPHQWPRQLWRULQJUHYLHZDQGHYDOXDWLRQLQFOXGLQJSDUWLFLSDWRU\QRQ
SDUWLFLSDWRU\DQGMRLQWDSSURDFKHVLQWHUPVRIWKHLUPHULWVDQGZHDNQHVVHV7KHUDQJHRIWRROVSUHVHQWHG
LQFOXGH35$VXUYH\VORJLFDOIUDPHZRUNDQDO\VLVFRVWHIIHFWLYHQHVVDQDO\VLV6WUHQJWKV:HDNQHVVHV
2SSRUWXQLWLHVDQG&RQVWUDLQWV6:2&DQDO\VLVDPRQJRWKHUV,QFOXGHVVSHFLILFTXHVWLRQVWRFRQVLGHU
ZKHQSODQQLQJDVVHVVPHQWPRQLWRULQJUHYLHZDQGHYDOXDWLRQ

*XLMW,3DUWLFLSDWRU\0RQLWRULQJDQG,PSDFW$VVHVVPHQWRI6XVWDLQDEOH$JULFXOWXUH,QLWLDWLYHV
$Q,QWURGXFWLRQWRWKH.H\(OHPHQWV'LVFXVVLRQ3DSHU1R6XVWDLQDEOH$JULFXOWXUHDQG5XUDO
/LYHOLKRRGV3URJUDPPH,,('8.,,('
&RQWDFW,QIRUPDWLRQ
,QWHUQDWLRQDO,QVWLWXWHIRU(QYLURQPHQWDQG'HYHORSPHQW,,('
(QGVOHLJK6WUHHW
/RQGRQ:&+''
8.
7HO
)D[
7KLVGRFXPHQWLVDSUDFWLFDOLQWURGXFWLRQRQWKHPHWKRGRORJ\RIHVWDEOLVKLQJDSDUWLFLSDWRU\PRQLWRULQJ
DQGHYDOXDWLRQSURFHVVIRUVXVWDLQDEOHDJULFXOWXUHLQLWLDWLYHV,WZDVLQLWLDOO\ZULWWHQWRJXLGHWKHILUVWVWDJH
RIDQDFWLRQUHVHDUFKSURFHVVRQPRQLWRULQJDQGLPSDFWDVVHVVPHQWZLWKVPDOOVFDOHSURGXFHUVUXUDO
ZRUNHUVXQLRQVDQGQRQJRYHUQPHQWDORUJDQLVDWLRQVHQJDJHGLQVXVWDLQDEOHDJULFXOWXUHLQ%UD]LO7KH
SDSHULQWURGXFHVNH\FRQFHSWVDQGRXWOLQHVNH\VWHSVLQGHYHORSLQJDPRQLWRULQJV\VWHP,WGLVFXVVHVWKH
FRPSOH[LW\RIGHYHORSLQJLQGLFDWRUVDQGWKHUDQJHRISRVVLEOHPHWKRGVZLWKH[DPSOHVIURPDJULFXOWXUH
7KLVSDSHUZLOOEHRILQWHUHVWWRWKRVHZRUNLQJLQQDWXUDOUHVRXUFHPDQDJHPHQWEXWDOVRWRRWKHUVLQWHUHVWHG
JHQHUDOO\LQSDUWLFLSDWRU\DFWLRQUHVHDUFKPRQLWRULQJDQGHYDOXDWLRQ


+RSH$QQHDQG6DOO\7LPPHOUHYLVHGHGLWLRQ7UDLQLQJIRU7UDQVIRUPDWLRQ$+DQGERRNIRU
&RPPXQLW\:RUNHUV9ROXPHV²*ZHUX=LPEDEZH0DPER3UHVV
&RQWDFWLQIRUPDWLRQ
'DYLG3KLOLS3XEOLVKHUV
32%R[
&ODUHPRQW
6RXWK$IULFD
RU
,73XEOLFDWLRQV
²6RXWKDPSWRQ5RDG
/RQGRQ:&%++
8.
$OVRDYDLODEOHLQ6SDQLVKDQG)UHQFK
IRUWKH6SDQLVKHGLWLRQFRQWDFW6HPLOOD=RQD$SGR²,0RQWVHUUDW&LXGDGGH*XDWHPDOD
*XDWHPDOD)D[
IRUWKH)UHQFKHGLWLRQFRQWDFW7KH5RPDQ&DWKROLF'LRFHVHRI3RUW/XLVFR&DULWDV
,QWHUQDWLRQDOLV'
(VWDLQJ6KXW3RUW/XLV0DXULWLXV)D[
$OOODQJXDJHHGLWLRQVFDQEHSXUFKDVHGIURP*UDLOYLOOH%RRNVWRUH/RYHODQG2+86$)D[

9ROXPHGHDOVPDLQO\ZLWKKHOSLQJORFDOFRPPXQLWLHVLGHQWLI\DQGUHIOHFWXSRQLVVXHVZKLFKWURXEOHWKHP
PRVWDQGRQZKLFKWKH\ZRXOGOLNHWRVWULYHIRUFKDQJH9ROXPHDGGUHVVHVJURXSOHDGHUVKLSVNLOOVDQG
JURXSG\QDPLFVLQFOXGLQJGHFLVLRQPDNLQJSODQQLQJIRUDFWLRQDQGHYDOXDWLRQ9ROXPHPRYHVIURPWKH
ORFDOOHYHOWRWKHZLGHUFRQWH[W,WSURYLGHVWRROVIRUDQDO\VLQJKRZIRUFHVRSHUDWLQJDWQDWLRQDODQG
LQWHUQDWLRQDOOHYHOVDIIHFWORFDOHIIRUWVWRZDUGVGHYHORSPHQWDQGFKDQJHDVZHOODVWRROVIRURUJDQLVDWLRQDO
VWUHQJWKHQLQJDQGIRUSODQQLQJZRUNVKRSV

1DUD\DQ3DUNHU'3DUWLFLSDWRU\(YDOXDWLRQ7RROVIRU0DQDJLQJ&KDQJHLQ:DWHUDQG
6DQLWDWLRQ:RUOG%DQN7HFKQLFDO3DSHU:DVKLQJWRQ'&:RUOG%DQN
&RQWDFWLQIRUPDWLRQ
81'3352::(663URJUDP0DQDJHU
(DVWWK6WUHHW5RRP))
1HZ<RUN1HZ<RUN
7HO
)D[
7KLVERRNLVSDUWRIWKH352::(66VHULHVRQZDWHUVXSSO\DQGVDQLWDWLRQSURMHFWV352::(66VWDQGV
IRU
3URPRWLRQRIWKH5ROHRI:RPHQLQ:DWHUDQG(QYLURQPHQWDO6DQLWDWLRQ6HUYLFHV
,WHODERUDWHVRQ
WKHFRQFHSWRISDUWLFLSDWRU\HYDOXDWLRQDQGWKHJXLGHOLQHVIRUXVLQJSDUWLFLSDWRU\PHWKRGVLQHYDOXDWLRQ
HVSHFLDOO\ZLWKUHVSHFWWRWKHGHYHORSPHQWRI0 (LQGLFDWRUV'HWDLOHGH[DPSOHVRIH[SHULHQFHVDUH


 SURYLGHGZLWKLQWKHFRQWH[WRIZDWHUDQGVDQLWDWLRQSURJUDPPHV
>QRWDVJRRGLQWHUPVRISUHVHQWLQJUDQJHRIWRROVIRU30 (XVHEXWHODERUDWHVRQLQGLFDWRUGHYHORSPHQW@

3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ$+DQGERRNIRU7UDLQLQJ)LHOG:RUNHUV
%DQJNRN5HJLRQDO2IILFHIRU$VLDDQGWKH3DFLILF5$3$)$2
&RQWDFWLQIRUPDWLRQ
$OH[DQGHU6WHSKHQV
5HJLRQDO6RFLRORJLVWDQG:RPHQLQ'HYHORSPHQW2IILFHU
)$25HJLRQDO2IILFHIRU$VLDDQGWKH3DFLILF
3KUD$WLW5RDG
%DQJNRN7KDLODQG
7KLVPDQXDOIRFXVHVVSHFLILFDOO\RQYLOODJHOHYHOSDUWLFLSDWRU\PRQLWRULQJDQGHYDOXDWLRQ7KHPDQXDO
SURYLGHVDVDPSOHRIYDULRXVWRROVWKDWHPSOR\VLPSOHQXPHULFDQGYLVXDOWHFKQLTXHVLQGRFXPHQWLQJ
SURMHFWRXWFRPHVDVZHOODVSURFHVVHVLHDWWLWXGHVYDOXHVH[WHQWRISDUWLFLSDWLRQ7KHGLIIHUHQW30 (
WRROVGLVFXVVHGJLYHOHVVHPSKDVLVRQKRZWRHVWDEOLVKDQ0 (IUDPHZRUNDQGKRZWRDQDO\VHJHQHUDWHG
LQIRUPDWLRQIRUSDUWLFLSDWRU\OHDUQLQJDQGDFWLRQ

3IRKO-DFRE3DUWLFLSDWRU\(YDOXDWLRQ$8VHU
V*XLGH1HZ<RUN3ULYDWH$JHQFLHV
&ROODERUDWLQJ7RJHWKHU3$&7
&RQWDFWLQIRUPDWLRQ
$WWQ'DQLHO6DQWR3LHWUR
8QLWHG1DWLRQV3OD]D
1HZ<RUN1<
7HO
7HOH[
$QRXWFRPHRIDZRUNVKRSVSRQVRUHGE\86$,'6UL/DQNDDQGFRQGXFWHGE\3$&7D86392WKLV
PDQXDOLVGHVLJQHGWRSURYLGHEURDGJXLGHOLQHVIRUHVWDEOLVKLQJDSDUWLFLSDWRU\HYDOXDWLRQSURFHVVWKDW
PHHWVWKHQHHGVRIGLIIHUHQWVWDNHKROGHUVGRQRUVSURMHFWVWDIIEHQHILFLDULHVHWF,WORRNVDWWKHNH\
FRQFHSWVGHILQLQJSDUWLFLSDWRU\HYDOXDWLRQDQGHODERUDWHVRQDVWHSE\VWHSDSSURDFKIRUFDUU\LQJRXW
SDUWLFLSDWRU\HYDOXDWLRQ7KHPHWKRGVGLVFXVVHGLQFOXGHWKHGHVLJQRIDQHYDOXDWLRQIUDPHZRUNVDPSOH
LQGLFDWRUVHYDOXDWLRQWRROVGDWDDQDO\VLVDQGUHSRUWLQJ7KHPDQXDOSURYLGHVDILQDOVHFWLRQRQWUDLQLQJDQG
VDPSOHPRQLWRULQJDQGHYDOXDWLRQH[HUFLVHV


3UHWW\-XOHV1,UHQH*XLMW,DQ6FRRQHVDQG-RKQ7KRPSVRQ$7UDLQHU
V*XLGHIRU
3DUWLFLSDWRU\/HDUQLQJDQG$FWLRQ/RQGRQ,,('6XVWDLQDEOH$JULFXOWXUH3URJUDPPH
&RQWDFWLQIRUPDWLRQ
6XVWDLQDEOH$JULFXOWXUH3URJUDPPH
,,('
(QGVOHLJK6WUHHW
/RQGRQ:&+''
8.
$OWKRXJKSULPDULO\DWUDLQHU
VJXLGHLWLQFOXGHVDUDQJHRIJDPHVDQGH[HUFLVHVZKLFKPD\EHXVHGLQ
FDUU\LQJRXW30 (7KHVHWRROVDUHJHDUHGIRUHQKDQFLQJJURXSG\QDPLFVLPSURYLQJOLVWHQLQJDQG
REVHUYDWLRQLPSURYLQJDQDO\VLVDQGUHIOHFWLRQFRQGXFWLQJHYDOXDWLRQIRUVHOIJURXSDVVHVVPHQWDQGIRU
DFWLRQSODQQLQJ

3URFHVV0RQLWRULQJ3UR0:RUN'RFXPHQWIRUSURMHFWVWDII(VFJERUQ*HUPDQ\'HXWVFKH

*HVHOOVFKDIWIU7HFKQLVFKH=XVDPPHQDUEHLW*7=*PE+
&RQWDFWLQIRUPDWLRQ
3LORW3URMHFW1DWXUDO5HVRXUFH0DQDJHPHQWE\6HOI+HOS3URPRWLRQ1$506
:DFKVEOHLFKH
'%RQQ
)HGHUDO5HSXEOLFRI*HUPDQ\
7HO
)D[
'HVFULEHVWKHPHWKRGRORJ\RISURFHVVPRQLWRULQJLQFOXGLQJGHWDLOHGGLVFXVVLRQVRIYDULRXVWHFKQLTXHVWKDW
PD\EHDSSOLHG)RXUEDVLFIXQFWLRQVRI3UR0DUHRXWOLQHGSURFHVVVHOHFWLRQREVHUYDWLRQUHIOHFWLRQDQG
DFWLRQ3UR0LVSHUFHLYHGDVDPRQLWRULQJDQGHYDOXDWLRQSURFHVVWKDWWDNHVSODFHDWGLIIHUHQWOHYHOV
LQYROYLQJGLIIHUHQWDFWRUV3UR0PD\RFFXUDWRUEHWZHHQWKHOHYHORILQGLYLGXDODFWLYLW\UHODWLRQVRI
FRRSHUDWLRQZLWKLQDQRUJDQLVDWLRQFRRSHUDWLRQZLWKRWKHUDFWRUVDQGLQVWLWXWLRQVDQGWKHZLGHUVRFLR
HFRQRPLFDQGSROLWLFDOFRQWH[W7RROVIRU3UR0HPSKDVLVHOHDUQLQJDQGUHIOHFWLRQWKURXJKV\VWHPDWLF
DQDO\VLVFRQIOLFWQHJRWLDWLRQDQGJURXSFROODERUDWLRQDVZHOODVSODQQLQJIRUDFWLRQ

5XJK-LP6HOI(YDOXDWLRQ,GHDVIRU3DUWLFLSDWRU\(YDOXDWLRQRI5XUDO&RPPXQLW\'HYHORSPHQW
3URMHFWV2NODKRPD&LW\:RUOG1HLJKERUV
&RQWDFWLQIRUPDWLRQ
:RUOG1HLJKERUV,QF
1:6WUHHW
2NODKRPD&LW\2.
7KHPDQXDOGHVFULEHVWKHFRQFHSWXDOWKHRUHWLFDOPHWKRGRORJLFDOLQVWLWXWLRQDOLVDWLRQLVVXHVUHODWHGWR
SURMHFWHYDOXDWLRQ7KHSUHPLVHRIWKHPDQXDOLVWKDWHYDOXDWLRQLVXQGHUWDNHQIRUGHFLVLRQPDNLQJ
SXUSRVHVZLWKDIRFXVRQEHQHILFLDULHVDQGORFDOVWDIIDVGHFLVLRQPDNHUVDQGHYDOXDWRUVLQRUGHUWRLQFUHDVH


 FRPPLWPHQWWRWKHHQWLUHHYDOXDWLRQSURFHVV7KHPDQXDOGLVWLQJXLVKHVEHWZHHQGLIIHUHQWOHYHOVRI
HYDOXDWLRQDQGWKHGLIIHUHQWWLPHIUDPHVIRUWKHYDULRXVOHYHOVRIHYDOXDWLRQ7KHILQDOVHFWLRQVGLVFXVVWKH
EDVLVRUFRQVLGHUDWLRQVIRUGHFLGLQJZKLFKWHFKQLTXHVWRDSSO\DQGWRFRPPXQLFDWHILQGLQJV7KHDSSHQGL[
SUHVHQWVSDUWLFLSDWRU\HYDOXDWLRQFDVHVWXGLHVDQGVHYHUDOHYDOXDWLRQWRROV

6HOHQHU'DQLHOZLWK&KULVWRSKHU3XUG\DQG*DEULHOD=DSDWDQG'RFXPHQWLQJ(YDOXDWLQJDQG
/HDUQLQJIURPRXU'HYHORSPHQW3URMHFWV$3DUWLFLSDWRU\6\VWHPDWL]DWLRQ:RUNERRN(FXDGRU
,,55
&RQWDFWLQIRUPDWLRQ
,QWHUQDWLRQDO,QVWLWXWHIRU5XUDO5HFRQVWUXFWLRQ
5HJLRQDO2IILFHIRU/DWLQ$PHULFD
&DVLOOD
4XLWR(&8$'25
7HO)D[
HPDLOGDQLHO#LLUUHF[HF
7KLVZRUNERRNGHVFULEHV
V\VWHPDWLVDWLRQ
DVDPHWKRGRILQVWLWXWLQJDFRQWLQXRXVSURFHVVRISDUWLFLSDWRU\
UHIOHFWLRQDQGHYDOXDWLRQ,WHODERUDWHVRQWKHFRQFHSWRIV\VWHPDWLVDWLRQDVZHOODVWKHPHWKRGRORJLFDO
SURFHGXUHVWRIROORZLQSODQQLQJRIV\VWHPDWLVDWLRQ7KHILQDOVHFWLRQLQFOXGHVDQXPEHURI
WRROV
WKDWPD\
EHXVHGIRUFRQGXFWLQJWKHV\VWHPDWLVDWLRQSURFHVV$OWKRXJKWKHWRROVXVHGIRUREWDLQLQJDQGUHFRUGLQJ
LQIRUPDWLRQWHQGWRHPSKDVLVHWKHXVHRIZULWWHQH[HUFLVHVZRUNVKHHWVDQGWDEOHVUDWKHUWKDQPRUHRSHQ
HQGHGDQGYLVXDOWHFKQLTXHVWKHTXHVWLRQVDQGLVVXHVUDLVHGZLWKUHVSHFWWRPHWKRGRORJLFDOSURFHGXUHVDUH
LQVWUXFWLYH

:DGVZRUWK<RODQG(YHU\GD\(YDOXDWLRQRQWKH5XQ0HOERXUQH$FWLRQ5HVHDUFK,VVXHV
$VVRFLDWLRQ
&RQWDFWLQIRUPDWLRQ
$FWLRQ5HVHDUFK,VVXHV$VVRFLDWLRQ
WK)ORRU5RVV+RXVH
)LQGHUV/DQH
0HOERXUQH9LFWRULD$XVWUDOLD
7KLVERRNLVDXVHIXOJXLGHIRUGHYHORSLQJDQGFRQGXFWLQJDSURMHFWHYDOXDWLRQIUDPHZRUN&RQFHSWXDO
LVVXHVUHODWHGWRHYDOXDWLRQDUHLQLWLDOO\GLVFXVVHG7KHSURFHVVRI
GRLQJHYDOXDWLRQ
LVWKHQH[SORUHGZLWK
UHIHUHQFHWRGHYHORSLQJD
FXOWXUHRIHYDOXDWLRQ
WKURXJKGDLO\ZHHNO\PRQWKO\DQGDQQXDO
LQIRUPDOIRUPDOHYDOXDWLRQSURFHGXUHV7KHVHSURFHGXUHVPD\HLWKHUEHLQIRUPDODQGRUIRUPDOLQGLYLGXDO
DQGRUFROODERUDWLYHHIIRUWV)LQDOFKDSWHUVSURYLGHDQHYDOXDWLRQ
WRROER[
RIPRGHOVDQGWHFKQLTXHVDV
ZHOODVUHIHUHQFHVIRUIXUWKHUXVHIXOUHDGLQJ


:KRDUHWKH4XHVWLRQPDNHUV"$3DUWLFLSDWRU\(YDOXDWLRQ+DQGERRN1HZ<RUN2(63
&RQWDFWLQIRUPDWLRQ
2IILFHRI(YDOXDWLRQDQG6WUDWHJLF3ODQQLQJ
8QLWHG1DWLRQV'HYHORSPHQW3URJUDPPH
2QH8QLWHG1DWLRQV3OD]D1HZ<RUN1<
7KLVKDQGERRNRQSDUWLFLSDWRU\DSSURDFKHVWRSURMHFWHYDOXDWLRQLVZULWWHQVSHFLILFDOO\IRU81'3VWDII,W
GLVFXVVHVFRQFHSWXDODVZHOODVSUDFWLFDOFRQVLGHUDWLRQVIRUDSSO\LQJSDUWLFLSDWRU\HYDOXDWLRQLQFOXGLQJWKH
WRROVDQGPHWKRGVXVHGLQSDUWLFLSDWRU\HYDOXDWLRQLHIRFXVJURXSGLVFXVVLRQVORJIUDPHDQDO\VLV
PDSSLQJ35$WHFKQLTXHVDPRQJRWKHUV7KHILQDOVHFWLRQSUHVHQWVDFDVHVWXG\RID81'3H[SHULHQFH
DQGKLJKOLJKWVWKHFKDOOHQJHVREVWDFOHVDQGLQVLJKWV

:RRGKLOO-LPDQG5RELQV/LVD3DUWLFLSDWRU\(YDOXDWLRQIRU/DQGFDUHDQG&DWFKPHQW*URXSV$
*XLGHIRU)DFLOLWDWRUV$XVWUDOLD*UHHQLQJ$XVWUDOLD
&RQWDFWLQIRUPDWLRQ
*UHHQLQJ$XVWUDOLD/LPLWHG
32%R[
<DUUDOXPOD$&7
(PDLOJHQHUDO#JUHHQLQJDXVWUDOLDRUJDX
$QH[FHOOHQWUHFHQWIDFLOLWDWRU
VJXLGHZLWKPXFKEURDGHUDSSOLFDWLRQWKDQODQGFDUH*RRGRQWKHVWHSVRI
PRQLWRULQJDQGDQH[FHOOHQWWRROER[IRUSDUWLFLSDWRU\HYDOXDWLRQ


5()(5(1&(6
$EERW -RDQQH DQG ,UHQH *XLMW 
&KDQJLQJ 9LHZV RQ &KDQJH $ :RUNLQJ 3DSHU RQ 3DUWLFLSDWRU\
0RQLWRULQJ RI WKH (QYLURQPHQW
 :RUNLQJ 3DSHU ,QWHUQDWLRQDO ,QVWLWXWH IRU (QYLURQPHQW DQG
'HYHORSPHQW,,('
$FKDU\D%<.XPDU96DW\DPXUWLDQG57DQGRQ5HIOHFWLRQVRQ3DUWLFLSDWRU\(YDOXDWLRQ²WKH
3ULYDWH9ROXQWDU\2UJDQLVDWLRQIRU+HDOWK,,392+([SHULHQFH3DSHUSUHSDUHGIRUWKH

,QWHUQDWLRQDO:RUNVKRSRQ3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ([SHULHQFHDQG/HVVRQV
&DYLWH
3KLOLSSLQHV1RYHPEHU²
$OFRFHU -RHO 3LODU /L]iUUDJD -KRQQ\ 'HOJDGLOOR HW DO  6RQGHR 6REUH 8WLOL]DFLyQ GH 0pWRGRV
3DUWLFLSDWLYRV GH 0RQLWRUHR \ (YDOXDFLyQ HQ %ROLYLD $ 6XUYH\ RI WKH 3UDFWLFH RI 3DUWLFLSDWRU\
0RQLWRULQJDQG(YDOXDWLRQ0HWKRGVLQ%ROLYLD3DSHUSUHSDUHGIRU WKH
,QWHUQDWLRQDO :RUNVKRS RQ
3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ([SHULHQFHDQG/HVVRQV
&DYLWH3KLOLSSLQHV1RYHPEHU²
$PDQRU .RMR 
$QDO\WLFDO $EVWUDFWV RQ )DUPHU 3DUWLFLSDWRU\ 5HVHDUFK
 $JULFXOWXUDO $GPLQLVWUDWLRQ
8QLW2FFDVLRQDO3DSHU/RQGRQ2YHUVHDV'HYHORSPHQW,QVWLWXWH
$UFKHU 'DYLG DQG 6DUD &RWWLQJKDP  5()/(&7 0RWKHU 0DQXDO 5HJHQHUDWHG )UHLUHDQ /LWHUDF\
WKURXJK(PSRZHULQJ&RPPXQLW\7HFKQLTXHV/RQGRQ$FWLRQ$LG
$UPRQLD 5LFDUGR & DQG 'LQGR 0 &DPSLODQ 
3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ 7KH $VLDQ
([SHULHQFH
 5HJLRQDO RYHUYLHZ SDSHU SUHSDUHG IRU WKH ,QWHUQDWLRQDO :RUNVKRS RQ 3DUWLFLSDWRU\
0RQLWRULQJDQG(YDOXDWLRQ&DYLWH3KLOLSSLQHV²1RYHPEHU/RV%DQRV/DJXQD8VHUV
3HUVSHFWLYHV
ZLWK$JULFXOWXUDO5HVHDUFKDQG'HYHORSPHQW83:$5'
%ODXHUW-XWWDDQG(GXDUGR4XLQWDQDU
6HHNLQJ/RFDO,QGLFDWRUV3DUWLFLSDWRU\6WDNHKROGHU(YDOXDWLRQRI
)DUPHUWR)DUPHU3URMHFWV
3DSHUSUHSDUHGIRUWKH::)ZRUNVKRSRQ
8VHRI$SSUDLVDO0HWKRGRORJLHV
IRU&RQVHUYDWLRQDQG'HYHORSPHQW3URMHFWV
2D[DFD0H[LFR²-XQH
%URZQ'DYLG
,QIRUPDWLRQ6\VWHPVIRU1*2V
5XUDO([WHQVLRQ%XOOHWLQ$SULO²
&DPSLODQ 'LQGR 
0DNLQJ 3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ 30 ( :RUN 7KLUWHHQ 9LJQHWWHV
IURPWKH)LHOG
SS²LQ6HOIDVVHVVPHQW3DUWLFLSDWRU\'LPHQVLRQVRI3URMHFW0RQLWRULQJDQG
(YDOXDWLRQ/RV%DQRV3KLOLSSLQHV83:$5'
&DPSRV-HQQLHDQG)UDQFRLV3UDOLQH&RXSDO3DUWLFLSDWRU\(YDOXDWLRQ3UHSDUHGIRUWKH81'3GUDIW
&KDPEHUV5REHUW:KRVH5HDOLW\&RXQWV"3XWWLQJWKH/DVW)LUVW/RQGRQ,QWHUPHGLDWH7HFKQRORJ\
3XEOLFDWLRQV
BBBBBD
3DUWLFLSDWRU\5XUDO$SSUDLVDO35$$QDO\VLVRI([SHULHQFH
:RUOG'HYHORSPHQW²

BBBBBE
3DUWLFLSDWRU\5XUDO$SSUDLVDO35$&KDOOHQJHV3RWHQWLDODQG3DUDGLJP
:RUOG'HYHORSPHQW
²
&KDPEHUV5REHUWDQG,UHQH*XLMW
35$²)LYH<HDUV/DWHU:KHUH$UH:H1RZ"
)RUHVWV7UHHVDQG
3HRSOH1HZVOHWWHU
&RPPXQLW\ 3DUWQHUVKLS &HQWHU 8QLYHUVLW\ RI 7HQQHVVHH  5XUDO (PSRZHUPHQW =RQHV(QWHUSULVH
&RPPXQLWLHV /HVVRQV IURP WKH /HDUQLQJ ,QLWLDWLYH )LQGLQJV DQG 5HFRPPHQGDWLRQV RI WKH


 &RPPXQLW\ 3DUWQHUVKLS &HQWHU (=(& /HDUQLQJ ,QLWLDWLYHV 9ROXPH , .QR[YLOOH 7HQQHVVHH
8QLYHUVLW\RI7HQQHVVHH
&21&(51:RUOGZLGH
5HYLHZRI3DUWLFLSDWRU\0RQLWRULQJDQG(YDOXDWLRQ

&RUQZDOO$QGUHD-RKQ*DVKLJL&KDULW\.DEXWKDDQG7LOO\6HOOHUVD
$5HSRUWRIWKH3DUWLFLSDWRU\3URFHVV
(YDOXDWLRQ RI WKH )DPLO\ /LIH 7UDLQLQJ 3URJUDPPH )/73 D .HQ\D *RYHUQPHQW 3URJUDPPH ZLWK
6XSSRUWIURP'DQLGD²-DQXDU\

&RUQZDOO$QGUHD1RYHPEHUSHUVRQDOFRPPXQLFDWLRQ
&RUQZDOO$QGUHD,UHQH*XLMWDQG$OLFH:HOERXUQ
$FNQRZOHGJLQJ3URFHVV&KDOOHQJHVIRU$JULFXOWXUDO
5HVHDUFKDQG([WHQVLRQ0HWKRGRORJ\
,'6'LVFXVVLRQ3DSHU%ULJKWRQ,QVWLWXWHRI'HYHORSPHQW
6WXGLHV
&RUQZDOO$QGUHDDQG5DFKHO-HZNHV
:KDWLV3DUWLFLSDWRU\5HVHDUFK"
6RFLDO6FLHQFH0HGLFLQH
²
&RXVLQV-%UDGOH\DQG(OL]DEHWK :KLWPRUH 
)UDPLQJ3DUWLFLSDWRU\ (YDOXDWLRQ
 3DSHUSUHVHQWHG DW WKH
DQQXDOPHHWLQJRIWKH&DQDGLDQ(YDOXDWLRQ6RFLHW\2WWDZD0D\
'DYLHV 5 - 
$Q (YROXWLRQDU\ $SSURDFK WR 2UJDQLVDWLRQDO /HDUQLQJ $Q ([SHULPHQW E\ DQ 1*2 LQ
%DQJODGHVK
LQ'DYLG0RVVH-RKQ)DUULQJWRQDQG$ODQ5HZHGV'HYHORSPHQWDV3URFHVV
&RQFHSWVDQG0HWKRGVIRU:RUNLQJZLWK&RPSOH[LW\/RQGRQ5RXWOHGJH
'DYLV&DVH '
$UF\   &RPPXQLW\ )RUHVWU\ 3DUWLFLSDWRU\ $VVHVVPHQW 0RQLWRULQJ DQG
(YDOXDWLRQ5RPH)$2
BBBBB  7KH &RPPXQLW\
V 7RROER[ 7KH ,GHD 0HWKRGV DQG 7RROV IRU 3DUWLFLSDWRU\ $VVHVVPHQW
0RQLWRULQJDQG(YDOXDWLRQLQ&RPPXQLW\)RUHVWU\&RPPXQLW\)RUHVWU\)LHOG0DQXDO5RPH)$2
GH5DHGW&DURO
3URMHFW0RQLWRULQJDQG(YDOXDWLRQ'RHVD3DUWLFLSDWRU\$SSURDFK0DNHD'LIIHUHQFH"

SS²LQ6HOIDVVHVVPHQW3DUWLFLSDWRU\'LPHQVLRQVRI3URMHFW0RQLWRULQJDQG(YDOXDWLRQ/RV
%DQRV3KLOLSSLQHV83:$5'
)DUULQJWRQ -RKQ DQG $GULHQQH 0DUWLQ 
)DUPHU 3DUWLFLSDWLRQ LQ $JULFXOWXUDO 5HVHDUFK $ 5HYLHZ RI
&RQFHSWV DQG 3UDFWLFHV
 $JULFXOWXUDO $GPLQLVWUDWLRQ 8QLW 2FFDVLRQDO 3DSHU  /RQGRQ 2YHUVHDV
'HYHORSPHQW,QVWLWXWH
)HUQDQGH] 0DULD ( 
3DUWLFLSDWRU\ 5HVHDUFK ZLWK &RPPXQLW\EDVHG )DUPHUV
 SS ² LQ %HUWXV
+DYHUNRUW -RKDQ YDQ GHU .DPS DQG $QQ :DWHUV%D\HU HGV -RLQLQJ )DUPHUV
 ([SHULPHQWV
([SHULHQFHV LQ 3DUWLFLSDWRU\ 7HFKQRORJ\ 'HYHORSPHQW /RQGRQ ,QWHUPHGLDWH 7HFKQRORJ\
3XEOLFDWLRQV
)HXHUVWHLQ 0DULH7KHUHVH  3DUWQHUV LQ (YDOXDWLRQ (YDOXDWLQJ 'HYHORSPHQW DQG &RPPXQLW\
3URJUDPPHVZLWK3DUWLFLSDQWV/RQGRQ0DFPLOODQ(GXFDWLRQ/WG
)RUG 5LFKDUG  35$ IRU 0RQLWRULQJ DQG (YDOXDWLRQ $ 9LOODJH /RJERRN IURP $PERGLUDILD
0DGDJDVFDU:RUFHVWHU0$&ODUN8QLYHUVLW\DQG$))-.0
)RUG 5LFKDUG DQG )UDQFLV /HOR
(YDOXDWLQJ 3DUWLFLSDWRU\ 5XUDO $SSUDLVDO /LVWHQLQJ WR 9LOODJH /HDGHUV LQ
.DNX\XQL /RFDWLRQ
 3DSHUV LQ ,QWHUQDWLRQDO 'HYHORSPHQW DQG 6RFLDO &KDQJH &ODUN 8QLYHUVLW\
3DSHUUHSULQWIURP)RUHVWV7UHHVDQG3HRSOH1HZVOHWWHU-XQH:RUFHVWHU0$ &ODUN
8QLYHUVLW\


)RUG 5LFKDUG %DUEDUD 7KRPDV6OD\WHU )UDQFLV /HOR HW DO 
&RQVHUYLQJ 5HVRXUFHV DQG ,QFUHDVLQJ
3URGXFWLRQ8VLQJ3DUWLFLSDWRU\7RROVWR0RQLWRUDQG(YDOXDWH&RPPXQLW\%DVHG5HVRXUFH0DQDJHPHQW
3UDFWLFHV
&HQWHUIRU&RPPXQLW\%DVHG'HYHORSPHQW&ODUN8QLYHUVLW\
*DULED6XOOH\
3DUWLFLSDWRU\,PSDFW$VVHVVPHQWDVD7RROIRU&KDQJH/HVVRQV)URP5HFHQW([SHULHQFH
LQ 3RYHUW\ $OOHYLDWLRQ 3URMHFWV LQ $IULFD
 3DSHU SUHVHQWHG DW WKH 3DQHO 6HVVLRQ RQ 3DUWLFLSDWRU\ ,PSDFW
$VVHVVPHQWDWWKH,QWHUQDWLRQDO(YDOXDWLRQ&RQIHUHQFH9DQFRXYHU&DQDGD1RYHPEHU²
*RKO (EHUKDUG 'RUVL *HUPDQQ DQG %XUNKDUG 6FKZDU]  3DUWLFLSDWRU\ ,PSDFW 0RQLWRULQJ )RXU
9ROXPHV  *URXS%DVHG ,PSDFW 0RQLWRULQJ  1*2%DVHG ,PSDFW 0RQLWRULQJ 
$SSOLFDWLRQ ([DPSOHV  7KH &RQFHSW RI 3DUWLFLSDWRU\ ,PSDFW 0RQLWRULQJ %UDXFVFKZHLJ
*HUPDQ\*$7(*7=
*RVOLQJ/RXLVDDQG0LNH(GZDUGV7RRONLWV$3UDFWLFDO*XLGHWR$VVHVVPHQW0RQLWRULQJ5HYLHZ
DQG(YDOXDWLRQ/RQGRQ6DYHWKH&KLOGUHQ
*UHHQH -HQQLIHU 
4XDOLWDWLYH 3URJUDP (YDOXDWLRQ 3UDFWLFH DQG 3URPLVH
 SS ² LQ 1RUPDQ .
'HQ]LQDQG<YRQQD6/LQFROQHGV+DQGERRNRI4XDOLWDWLYH5HVHDUFK7KRXVDQG2DNV&DOLIRUQLD
6DJH3XEOLFDWLRQV,QF
*7=
3URFHVV0RQLWRULQJ3UR0
:RUN'RFXPHQWIRU3URMHFW6WDII%RQQ*7=*$7(
*XED(JRQ*DQG<YRQQH6/LQFROQ)RXUWK*HQHUDWLRQ(YDOXDWLRQ/RQGRQDQG&DOLIRUQLD6DJH
3XEOLFDWLRQV
*XLMW,UHQH3DUW0RYLQJ6ORZO\DQG5HDFKLQJ)DU,QVWLWXWLRQDOLVLQJ3DUWLFLSDWRU\3ODQQLQJIRU
&KLOG&HQWUHG &RPPXQLW\ 'HYHORSPHQW $Q ,QWHULP $QDO\VLV IRU 5HGG %DUQD²(DVW $IULFD
/RQGRQ 5HGG %DUQD²(DVW $IULFD527' 8JDQGD DQG ,QWHUQDWLRQDO ,QVWLWXWH IRU (QYLURQPHQW DQG
'HYHORSPHQW,,('
²²²,PSDFWVDQG,QVWLWXWLRQV3DUWQHUVDQG3ULQFLSOHV7KLUG5HYLHZRIWKH'HYHORSPHQWDQG8VH
RI3DUWLFLSDWRU\5XUDO$SSUDLVDODQG3ODQQLQJE\5HGG%DUQD8JDQGD/RQGRQ,,('DQG.DPSDOD
5HGG%DUQD8JDQGD
²²² 
3DUWLFLSDWRU\ 0RQLWRULQJ DQG ,PSDFW $VVHVVPHQW RI 6XVWDLQDEOH $JULFXOWXUH ,QLWLDWLYHV $Q
,QWURGXFWLRQ WR WKH .H\ (OHPHQWV
 'LVFXVVLRQ 3DSHU  /RQGRQ 6XVWDLQDEOH $JULFXOWXUH DQG 5XUDO
/LYHOLKRRGV3URJUDPPH,,('
*XLMW,UHQHDQG3DEOR6LGHUVN\
$JUHHLQJRQ,QGLFDWRUV
LQ,/(,$1HZVOHWWHU'HFHPEHU²
+RZHV0LFN
/LQNLQJ3DUDGLJPVDQG3UDFWLFH.H\,VVXHVLQWKH$SSUDLVDO0RQLWRULQJDQG(YDOXDWLRQRI
%ULWLVK1*23URMHFWV
LQ-RXUQDORI,QWHUQDWLRQDO'HYHORSPHQW²
+XPEHUW'UR] %ODLVH 
3,'2: 6HOI(YDOXDWLRQ 0DLQ )HDWXUHV DQG ([SHULHQFHV
 6ZLVV 'HYHORSPHQW
&RRSHUDWLRQ%DQJDORUH
,QVWLWXWH IRU 'HYHORSPHQW 5HVHDUFK ,'5 DQG 6RFLHW\ IRU 3DUWLFLSDWRU\ 5HVHDUFK LQ $VLD 35,$
7UDLQLQJ
:RUNVKRSRQ0RQLWRULQJ3DUWLFLSDWLRQLQ:RUOG%DQN3URJUDPPHV0DUFK
PLPHR
-DFNVRQ(GZDUG72FWREHU
3DUWLFLSDWRU\,PSDFW $VVHVVPHQW IRU3RYHUW\ $OOHYLDWLRQ 2SSRUWXQLWLHV
IRU &RPPXQLWLHV DQG 'HYHORSPHQW $JHQFLHV
 3DSHU SUHVHQWHG DW WKH 3DQHO 6HVVLRQ RQ 3DUWLFLSDWRU\
,PSDFW $VVHVVPHQW DW WKH ,QWHUQDWLRQDO (YDOXDWLRQ &RQIHUHQFH 9DQFRXYHU 1RYHPEHU ²  DQG


 SUHSDUHGIRULQFOXVLRQLQDERRNRQHYDOXDWLRQLQGHYHORSLQJFRXQWULHVWREHSXEOLVKHGE\WKH,QWHUQDWLRQDO
'HYHORSPHQW5HVHDUFK&HQWUHRI&DQDGDDQGWKH,QWHUQDWLRQDO6HUYLFHIRU1DWLRQDO$JULFXOWXUDO5HVHDUFK
.DU .DPDO ZLWK 61 'DWWD 6 *RVZDPL HW DO D
3DUWLFLSDWRU\ ,PSDFW $VVHVVPHQW RI &DOFXWWD 6OXP
,PSURYHPHQW3URMHFW6OXP'ZHOOHUV
3HUFHSWLRQVDQG$QDO\VLVRQ2YHUDOO6OXP(QYLURQPHQW/LYHOLKRRG
6WDWXV RI :RPHQ DQG &KLOGUHQ DQG WKH )XWXUH RI &RPPXQLW\ ,QYROYHPHQW 7RZDUGV D %HWWHU /LYLQJ

)HEUXDU\²0D\&DOFXWWD
BBBBBE 3DUWLFLSDWRU\ ,PSDFW $VVHVVPHQW &DOFXWWD 6OXP ,PSURYHPHQW 3URMHFW 0DLQ )LQGLQJV
5HSRUW9ROXPH&DURO\Q6OHLJKWKROPHHG1HZ'HOKL&DOFXWWD0HWURSROLWDQ'HYHORSPHQW$XWKRULW\
ZLWKWKH8UEDQ3RYHUW\2IILFH'HSDUWPHQWIRU,QWHUQDWLRQDO'HYHORSPHQW'),'
.RW]1LFN
&LWL]HQVDV([SHUWV
:RUNLQJ3DSHU
.UDPVMR %RVVH DQG *HRIIUH\ ' :RRG  %UHDNLQJ WKH &KDLQV &ROOHFWLYH $FWLRQ IRU 6RFLDO -XVWLFH
$PRQJWKH5XUDO3RRURI%DQJODGHVK/RQGRQ,QWHUPHGLDWH7HFKQRORJ\3XEOLFDWLRQV
/LJKWIRRW&OLYH3HWHU'DOVJDDUGDQG0DU\$QQ%LPEDR
)DUPHU3DUWLFLSDWRU\3URFHGXUHVIRU0DQDJLQJ
DQG0RQLWRULQJ6XVWDLQDEOH)DUPLQJ6\VWHPV
LQ-RXUQDORI$VLDQ)DUP6\VWHPV$VVRFLDWLRQ²
0DUVGHQ 'DYLG DQG 3HWHU 2DNOH\ HGV 
(YDOXDWLQJ 6RFLDO 'HYHORSPHQW 3URMHFWV
 'HYHORSPHQW
*XLGHOLQHV2[IRUG2[IDP
0DUVGHQ 'DYLG 3HWHU 2DNOH\ DQG %ULDQ 3UDWW  0HDVXULQJ WKH 3URFHVV *XLGHOLQHV IRU (YDOXDWLQJ
6RFLDO'HYHORSPHQW8.,175$&
0F$UWKXU +DUROG - -U 
3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ 3DVVLQJ )DG RU WKH /RJLFDO 6WHS LQ
5HVHDUFK DQG 'HYHORSPHQW 0HWKRGRORJ\"
 SS ² LQ
,QWHUIDFLQJ 30 ( ZLWK WKH 5HVHDUFK DQG
'HYHORSPHQW 3URFHVV $Q ,QWURGXFWLRQ
 LQ 6HOIDVVHVVPHQW 3DUWLFLSDWRU\ 'LPHQVLRQV RI 3URMHFW
0RQLWRULQJDQG(YDOXDWLRQ/RV%DQRV3KLOLSSLQHV83:$5'
0HUULILHOG -XOLHW $OGHQ /DQFDVWHU DQG .ULVWL .LUNKDP 
+RUL]RQ
V 3DUWLFLSDWRU\ (YDOXDWLRQ 3URFHVV

+RUL]RQ3URMHFW)LQDO5HSRUW0DLQH86$
05'3 9LHWQDP 6ZHGHQ 0RXQWDLQ 5XUDO 'HYHORSPHQW 3URJUDPPH ² $SULO  0RXQWDLQ
,QIRUPDWLRQ /HDUQLQJ 6\VWHP 3URYLQFH /HYHO 7UDLQLQJ 'RFXPHQW 9LHWQDP 3URJUDPPH %RDUG
2IILFH
BBBBB 1RYHPEHU  9LOODJH (YDOXDWLRQ RI WKH )DUP /HYHO )RUHVWU\ 3URMHFW 9LHWQDP 7KH 3HRSOHV
&RPPLWWHHRI7X\HQ4XDQJ3URYLQFHDQGWKH$JULFXOWXUHDQG)RUHVW'HSDUWPHQW
0XUSK\-RVHWWH
*RRG(QRXJK6RRQ(QRXJK$8VHU2ULHQWHG$SSURDFKWR0RQLWRULQJDQG(YDOXDWLRQ
LQ([WHQVLRQ$JHQFLHV
LQ5XUDO([WHQVLRQ%XOOHWLQ$SULO²
1DJHO 8ZH -HQV  'HYHORSLQJ D 3DUWLFLSDWRU\ ([WHQVLRQ $SSURDFK $ 'HVLJQ IRU WKH 6LDYRQJD
'LVWULFW =DPELD %HUOLQ 7HFKQLFDO 8QLYHUVLW\ &HQWHU IRU $GYDQFHG 7UDLQLQJ LQ $JULFXOWXUDO
'HYHORSPHQW
1DSRQHQ +HO]L QG
3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ²$ 3URWRW\SH ,QWHUQDO /HDUQLQJ 6\VWHP IRU
/LYHOLKRRGDQG0LFUR&UHGLW3URJUDPV
XQSXEOLVKHGSDSHU
1DUD\DQ3DUNHU ' 
3DUWLFLSDWRU\ (YDOXDWLRQ 7RROV IRU 0DQDJLQJ &KDQJH LQ :DWHU DQG 6DQLWDWLRQ

:RUOG%DQN7HFKQLFDO3DSHU:DVKLQJWRQ'&:RUOG%DQN
1HZ(FRQRPLFV)RXQGDWLRQ
&RPPXQLWLHV&RXQW3XWWLQJ/RFDO,QGLFDWRUVRQWKH0DS
/RQGRQ1()


2VXJD %HQ DQG 'DYLG 0XWD\LVD 0D\ 
8VH RI 35$ LQ 3URJUDPPH 5HYLHZV DQG (YDOXDWLRQV .H\
6WUHQJWKV:HDNQHVVHVDQG/HVVRQV

2YHUVHDV'HYHORSPHQW$GPLQLVWUDWLRQ2'$3URMHFW&\FOH0DQDJHPHQW6\VWHPV

3DUDFKLQL /DUU\ ZLWK $QGUHZ 0RWW -XO\ 
6WUHQJWKHQLQJ &RPPXQLW\ 9RLFHV LQ 3ROLF\ 5HIRUP
&RPPXQLW\%DVHG0RQLWRULQJ/HDUQLQJDQG$FWLRQ6WUDWHJLHVIRUDQ(UDRI'HYHORSPHQWDQG&KDQJH
$
6SHFLDO5HSRUWIRUWKH$QQLH(&DVH\)RXQGDWLRQGUDIW
3DUWLFLSDWRU\ 0HWKRGRORJLHV )RUXP RI .HQ\D 3$0)25. $XJXVW  5HSRUW RI :RUNVKRS RQ 8VLQJ
3DUWLFLSDWRU\0HWKRGRORJLHVIRU0RQLWRULQJDQG(YDOXDWLRQ1DLUREL.HQ\D6RXWK6RXWK6KDULQJ)RUXP
3IRKO -DFRE  3DUWLFLSDWRU\ (YDOXDWLRQ $ 8VHU
V *XLGH 1HZ <RUN 3ULYDWH $JHQFLHV &ROODERUDWLQJ
7RJHWKHU3$&7
3UDWW%DQG-%R\GHQ7KH)LHOG'LUHFWRU
V+DQGERRN2[IRUG2[IRUG8QLYHUVLW\3UHVV
3URFHVV 0RQLWRULQJ 3UR0 :RUN 'RFXPHQW IRU 3URMHFW 6WDII  (VFJERUQ *HUPDQ\ 'HXWVFKH
*HVHOOVFKDIWIXU7HFKQLVFKH=XVDPPHQDUEHLW*7=*PE+
5DMDNXWW\ 6 
3HRSOH
V 3DUWLFLSDWLRQ LQ 0RQLWRULQJ DQG (YDOXDWLRQ RI 5XUDO 'HYHORSPHQW 3URJUDPV
&RQFHSWVDQG$SSURDFKHV
-RXUQDORI5XUDO'HYHORSPHQW,QGLD²
5DKPDQ0XKDPPHG$QLVXU
4XDOLWDWLYH'LPHQVLRQVRI6RFLDO'HYHORSPHQW(YDOXDWLRQ
SS²LQ
'DYLG 0DUVGHQ DQG 3HWHU 2DNOH\ HGV
(YDOXDWLQJ 6RFLDO 'HYHORSPHQW 3URMHFWV
 'HYHORSPHQW
*XLGHOLQHV2[IRUG2[IDP3DSHUSUHVHQWHGDWWKH6ZDQVHD&RQIHUHQFH²6HSWHPEHU
5HFNHUV8WH3DUWLFLSDWRU\3URMHFW(YDOXDWLRQ$OORZLQJ/RFDO3HRSOHWR+DYHWKHLU6D\$Q1*2
*XLGHIRU&RPPXQLW\'ULYHQ3URMHFW(YDOXDWLRQ%DVHGRQDFDVHVWXG\DPRQJWKH$ULDORI.HQ\D
V
DULG UDQJHODQGV (QYLURQPHQWDO /LDLVRQ &HQWUH ,QWHUQDWLRQDO (/&, DQG 8QLWHG 1DWLRQV (QYLURQPHQW
3URJUDPPH81(3
5RFKH&KULV0DOL$XWRHYDOXDWLRQ$Q1*2([SHULHQFHZLWK&RPPXQLW\EDVHG(YDOXDWLRQ
LQ5XUDO
([WHQVLRQ%XOOHWLQ$SULO²
5XGTYLVW$QGHUVDQG3UXGHQFH:RRGIRUG%HUJHU)HEUXDU\
(YDOXDWLRQDQG3DUWLFLSDWLRQ6RPH/HVVRQV

$UHSRUWSUHSDUHGIRUWKH'$&([SHUW*URXSRQ$LG(YDOXDWLRQ'HSWIRU(YDOXDWLRQDQG,QWHUQDO$XGLW
6ZHGLVK,QWHUQDWLRQDO'HYHORSPHQW&RRSHUDWLRQ$JHQF\
5XELQ)UDQFHV$%DVLF*XLGHWR(YDOXDWLRQIRU'HYHORSPHQW:RUNHUV2[IRUG2[IDP
5XJK-LP
&DQ3DUWLFLSDWRU\(YDOXDWLRQ0HHWWKH1HHGVRI$OO6WDNHKROGHUV"$&DVH6WXG\(YDOXDWLQJ
WKH:RUOG1HLJKERUV:HVW$IULFD3URJUDP
'LVFXVVLRQ3DSHU:RUOG1HLJKERUV
BBBBBB   6HOI(YDOXDWLRQ ,GHDV IRU 3DUWLFLSDWRU\ (YDOXDWLRQ RI 5XUDO &RPPXQLW\
'HYHORSPHQW3URMHFWV2NODKRPD&LW\2NODKRPD:RUOG1HLJKERUV,QF
6FRWW9LOOLHUV3DWWDD
&RQVXOWDWLYH6NLOOVIRU3URMHFW0RQLWRULQJDQG(YDOXDWLRQ
8QSXEOLVKHGSDSHU
BBBBB E
&RQVXOWDWLYH 6NLOOV IRU 3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ &RXUVH 3URJUDPPH
 &RXUVH
3URJUDPPH3URMHFW2IILFHUDQG&RXQWHUSDUW:RUNVKRS3DUW
BBBBB$XJXVW3HUVRQDOFRPPXQLFDWLRQ
6KDK$PLWD
3URFHVV'RFXPHQWDWLRQ5HVHDUFK
LQ3/$1RWHV)HEUXDU\²


6KDK 3DUPHVK *LULVK +DUGZDM DQG 5DQMLW $PEDVWKD 
*XMDUDW ,QGLD 3DUWLFLSDWRU\ 0RQLWRULQJ +RZ
)DUPHUV ([WHQVLRQ 9ROXQWHHUV DQG 1*2 6WDII :RUN 7RJHWKHU LQ 9LOODJHOHYHO 6RLO DQG :DWHU
&RQVHUYDWLRQ3URJUDPPH
LQ5XUDO([WHQVLRQ%XOOHWLQ$SULO²
6OLP +XJR DQG 3DXO 7KRPSVRQ  /LVWHQLQJ )RU D &KDQJH 2UDO 7HVWLPRQ\ DQG 'HYHORSPHQW
/RQGRQ3DQRV3XEOLFDWLRQV/WG
6RFLHW\ IRU 3DUWLFLSDWRU\ 5HVHDUFK LQ $VLD 35,$  3DUWLFLSDWRU\ (YDOXDWLRQ ,VVXHV DQG &RQFHUQV
1HZ'HOKL35,$
BBBB3DUWLFLSDWRU\5HVHDUFKDQG(YDOXDWLRQ([SHULPHQWVLQ5HVHDUFKDVD3URFHVVRI/LEHUDWLRQ
1HZ'HOKL35,$
6RPPHU 0DUWLQ 
:KRVH 9DOXHV 0DWWHU" ([SHULHQFHV DQG /HVVRQV IURP D 6HOI(YDOXDWLRQ LQ 3,'2:
3URMHFW.DUQDWDND,QGLD0D\²1RYHPEHU
6ZLVV'HYHORSPHQW&RRSHUDWLRQ%DQJDORUH)LHOG2IILFH
6WHSKHQV $OH[DQGHU   3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ $ +DQGERRN IRU 7UDLQLQJ
)LHOG:RUNHUV%DQJNRN5HJLRQDO2IILFHIRU$VLDDQGWKH3DFLILF5$3$)$2
6ULQLYDVDQ 9LML 
7KH 0HWKRGRORJ\ RI 3DUWLFLSDWRU\ (YDOXDWLRQ
 SS ² LQ : )HUQDQGHV DQG 5
7DQGRQ HGV 3DUWLFLSDWRU\ 5HVHDUFK DQG (YDOXDWLRQ ([SHULPHQWV LQ 5HVHDUFK DV D 3URFHVV RI
/LEHUDWLRQ1HZ'HOKL,QGLDQ6RFLDO,QVWLWXWH
7DQGRQ 5DMHVK 
3DUWLFLSDWRU\ (YDOXDWLRQ DQG 5HVHDUFK 0DLQ &RQFHSWV DQG ,VVXHV
 SS ² LQ :
)HUQDQGHVDQG57DQGRQHGV3DUWLFLSDWRU\5HVHDUFKDQG(YDOXDWLRQ([SHULPHQWVLQ5HVHDUFKDV
D3URFHVVRI/LEHUDWLRQ1HZ'HOKL,QGLDQ6RFLDO,QVWLWXWH
7KRPSVRQ -RKQ DQG -XOHV 1 3UHWW\ 
6XVWDLQDEOH ,QGLFDWRUV DQG 6RLO &RQVHUYDWLRQ $ 3DUWLFLSDWRU\
,PSDFW6WXG\DQG6HOIHYDOXDWLRQRIWKH&DWFKPHQW$SSURDFKRIWKH0LQLVWU\RI$JULFXOWXUH.HQ\D
3DSHU
WREHVXEPLWWHGWRWKH-RXUQDORI6RLODQG:DWHU&RQVHUYDWLRQ-XQHSS
7RSVRH-HQVHQ%HQWH'HFHPEHU
3RSXODU3DUWLFLSDWLRQ0RQLWRULQJDQG(YDOXDWLRQLQ,QWHJUDWHG 5XUDO
'HYHORSPHQW 7KH &DVH RI 3'5, LQ *XLQHD%LVVDX
 3RSXODU 3DUWLFLSDWLRQ 3URJUDPPH :RUNLQJ
3DSHU'HYHORSPHQW6WXGLHV8QLW'HSDUWPHQWRI6RFLDO$QWKURSRORJ\8QLYHUVLW\RI6WRFNKROP
83:$5'
,QWHUIDFLQJ30 (ZLWKWKH5HVHDUFKDQG'HYHORSPHQW3URFHVV$Q,QWURGXFWLRQ
SS²
LQ 6HOIDVVHVVPHQW 3DUWLFLSDWRU\ 'LPHQVLRQV RI 3URMHFW 0RQLWRULQJ DQG (YDOXDWLRQ /RV %DQRV
3KLOLSSLQHV83:$5'
81'3  :RUNVKRS RQ *RDOV DQG ,QGLFDWRUV IRU 0RQLWRULQJ DQG (YDOXDWLRQ IRU :DWHU DQG
6DQLWDWLRQ1HZ<RUN81'3:RUOG%DQN:DWHUDQG6DQLWDWLRQ3URJUDPPH
81'3  :KR $UH WKH 4XHVWLRQ0DNHUV" $ 3DUWLFLSDWRU\ (YDOXDWLRQ +DQGERRN 2IILFH RI
(YDOXDWLRQDQG6WUDWHJLF3ODQQLQJ
8SKRII1RUPDQ
$)LHOG0HWKRGRORJ\IRU3DUWLFLSDWRU\6HOI(YDOXDWLRQRI333*URXSDQG,QWHUJURXS
$VVRFLDWLRQDQG3HUIRUPDQFH
3DSHUSUHSDUHGIRUWKH3HRSOH
V3DUWLFLSDWLRQ3URJUDPPHRIWKH81)RRG
DQG$JULFXOWXUH2UJDQL]DWLRQ&RUQHOO8QLYHUVLW\
8SKRII1RUPDQ
$)LHOG0HWKRGRORJ\IRU3DUWLFLSDWRU\6HOI(YDOXDWLRQ
LQ&RPPXQLW\'HYHORSPHQW
-RXUQDO²
BBBBB 
3DUWLFLSDWRU\ (YDOXDWLRQ RI )DUPHU 2UJDQL]DWLRQV
 &DSDFLW\ IRU 'HYHORSPHQW 7DVNV
 LQ
$JULFXOWXUDO$GPLQLVWUDWLRQDQG([WHQVLRQ²


BBBBB  $ )LHOG 0HWKRGRORJ\ IRU 3DUWLFLSDWRU\ 6HOI(YDOXDWLRQ RI 333 *URXS DQG ,QWHU*URXS
$VVRFLDWLRQ 3HUIRUPDQFH 3UHSDUHG IRU WKH 3HRSOH
V 3DUWLFLSDWLRQ 3URJUDPPH RI WKH 81 )RRG DQG
$JULFXOWXUH2UJDQL]DWLRQ,WKDFD&RUQHOO8QLYHUVLW\
:DGVZRUWK<RODQG(YHU\GD\(YDOXDWLRQRQWKH5XQ0HOERXUQH$FWLRQ5HVHDUFK,VVXHV$VVRFLDWLRQ
:DOWHUV +HWWLH $QQHW +HUPDQV 0DUJUHHW YDQ GHU +HO  0RQLWRULQJ DQG (YDOXDWLRQ IURP D *HQGHU
3HUVSHFWLYH$*XLGHOLQH7KH+DJXH6191HWKHUODQGV'HYHORSPHQW2UJDQLVDWLRQ
:DUG 3HQH  *HWWLQJ WKH 5LJKW (QG RI WKH 6WLFN 3DUWLFLSDWRU\ 0RQLWRULQJ DQG (YDOXDWLRQ LQ DQ
2UJDQLVDWLRQDO&RQWH[W3DSHUSUHSDUHGIRUWKH
,QWHUQDWLRQDO:RUNVKRSRQ3DUWLFLSDWRU\0RQLWRULQJDQG
(YDOXDWLRQ([SHULHQFHDQG/HVVRQV
&DYLWH3KLOLSSLQHV1RYHPEHU²
:DQJ&DUROLQH<XDQ<DQ/LQJDQG)HQG0LQJ/LQJ
3KRWRYRLFHDVD7RROIRU3DUWLFLSDWRU\(YDOXDWLRQ
7KH&RPPXQLW\
V9LHZRI3URFHVVDQG,PSDFW
LQ-RXUQDORI&RQWHPSRUDU\+HDOWK²
:HOERXUQ$OLFH
1RWHVIRU&KULVWLDQ$LG6WDIIRQ35$7HFKQLTXHVRI3RWHQWLDO8VHIRUWKH-)6%XUNLQD
)DVR(YDOXDWLRQ([HUFLVH-XQH
PLPHR
:LJEROGXV6HHUS0DDLNH$:LJEROGXV3KDXGD6*XUXQJ3UDKODG%%KDQGDUL&KDQGUD.$U\DODQG/D[PL
' .KDGND  3DUWLFLSDWRU\ (YDOXDWLRQ LQ 3UDFWLFH 1503
V ([SHULHQFH LQ &RPPXQLW\
&DSDFLW\ 'HYHORSPHQW LQ 1HSDO .DWKPDQGX 1HSDO 5HVRXUFH 0DQDJHPHQW 3URMHFW 1503 DQG
8QLWHG0LVVLRQWR1HSDO801
:RRGKLOO -LP DQG /LVD 5RELQV  3DUWLFLSDWRU\ (YDOXDWLRQ IRU /DQGFDUH DQG &DWFKPHQW *URXSV $
*XLGHIRU)DFLOLWDWRUV$XVWUDOLD*UHHQLQJ$XVWUDOLD