Ghid Oecd PT Prvenirea Confl

GUIDANCE ON EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES
Working draft for application period
A joint project of the DAC Network on Conflict, Peace and Development Co-operation and the DAC Network on Development Evaluation
GUIDANCE ON EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES

WORKING DRAFT FOR APPLICATION PERIOD
DEVELOPMENT ASSISTANCE COMMITTEE
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT
The OECD is a unique forum where the governments of 30 democracies work together to address the economic, social and environmental challenges of globalisation. The OECD is also at the forefront of efforts to understand and to help governments respond to new developments and concerns, such as corporate governance, the information economy and the challenges of an ageing population. The Organisation provides a setting where governments can compare policy experiences, seek answers to common problems, identify good practice and work to co-ordinate domestic and international policies. The OECD member countries are: Australia, Austria, Belgium, Canada, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The Commission of the European Communities takes part in the work of the OECD. OECD Publishing disseminates widely the results of the Organisation's statistics gathering and research on economic, social and environmental issues, as well as the conventions, guidelines and standards agreed by its members.
Published in French under the title: Guide pour lvaluation des activits de prvention des conflits et de construction de la paix Document de travail pour la priode dapplication
OECD (2008) No reproduction, copy, transmission or translation of this publication may be made without written permission. Applications should be sent to OECD Publishing: rights@oecd.org or by fax (+33-1) 45 24 13 91. Permission to photocopy a portion of this work should be addressed to the Centre Franais dexploitation du droit de Copie, 20 rue des GrandsAugustins, 75006 Paris, France (contact@cfcopies.com).
EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES OECD 2008
FOREWORD
Thisworkingdraftdevelopsguidanceonconductingeffectiveevaluationsofconflictpreventionand peacebuildingwork.Thecurrentworkingdraftwillbeusedforaoneyearapplicationphasethrough 2008.ItistheresultofanongoingcollaborativeprojectbytheOECDDACNetworksonDevelopment EvaluationandonConflict,PeaceandDevelopmentCooperation(CPDC).ThetwoNetworksbeganthis collaborationin2005,respondingtotheneedexpressedbyCPDCmembersforgreaterclarityregarding techniquesandissuesofevaluationintheirfield.Anassessmentofpastconflictandpeaceevaluations andastudyofcurrentpracticeswereundertakenin2006andidentifiedaneedforfurtherguidance.In 2007aresearchpiece,EncouragingEffectiveEvaluationofConflictPreventionandPeacebuilding Activities:TowardsDACGuidance,wascompletedbyCDACollaborativeLearningProjectsand subsequentlypublishedasaninputtothedevelopmentofthisguidance. ThecurrentworkingdraftreflectsvaluablecontributionsfrommembersofbothDACNetworks. EspeciallyimportanthavebeenthecontributionsofAsbjrnEidhammerandCristinaHoyos,thelead membersfromtheEvaluationNetworkandtheCPDCNetwork,respectively.IntheDACSecretariat, LisaWilliams(CPDCNetwork)andHansLundgren(EvaluationNetwork)ledthetwoyearprocessto developthedraftguidancealongwithNathalieBienvenu,MarkDownes,AnnaHellstrm,SebastianLing, AlexandraTrzeciakDuval,AsbjrnWeeandMeganKennedy;MsKennedyfinalisedthisdocument.The researchanddraftingprocesshasalsobenefitedfromtheinputsofexpertsinmembercountriesand beyond:BeateBull;ClareHarkinandJuliaCompton;advisorsinaCriticalReviewPanel,including MaryB.AndersonandhercolleaguesatCDACollaborativeLearningProjects,DianaChigasand PeterWoodrow;ThaniaPaffenholz;TonyVaux;RobertPicciotto;RobertMuggah;aswellasother membersfrombothNetworksandexpertsinthefieldsofevaluationandpeacebuildingtoonumerousto listhere. Giventhecomplexityofworkinthisfieldandtheneedtoaddressdifferentaudiences,evaluators andpeacebuildingpractitionersalike,thisworkingdrafthasextensiveannexescontainingspecific informationtocomplimenttheshortermaintext.Themaintextisdividedintoageneralintroduction,an outlineofkeyplanningandprogrammingsteps,andadescriptionoftheevaluationprocessitself. Individualreadersmaychoosetofocusonparticularsections,accordingtotheirinterestandneeds. Thefuturedevelopmentofwebbasedandothermultimediaformatsfortheguidanceisalso envisioned.Thisworkingdraftwillbeappliedtofieldevaluationsover2008andwillberevisedfor submissiontotheDACbasedonexperiencegainedduringtheapplicationperiod.Itisintendedto contributetothelargerongoingprocessesofpromotingrigorousandcoordinatedevaluationof developmentassistance,whileimprovingdonorpoliciesandpracticesforengaginginconflictaffected areas. TheDACNetworksonDevelopmentEvaluation andonConflict,PeaceandDevelopmentCooperation

Background:Keydonorcommitmentsonevaluationandpeacebuilding
DACEvaluationQualityStandards(fortestphaseapplication)(2007) https://www.oecd.org/dataoecd/30/62/36596604.pdf TheParisDeclarationonAidEffectiveness(2005) http://www.oecd.org/document/18/0,2340,en_2649_3236398_35401554_1_1_1_1,00.html DACGuidelinesonHelpingPreventViolentConflict(2001)[includingDACGuidelinesonConflict, PeaceandDevelopmentCooperation(1998)] http://www.oecd.org/dataoecd/15/54/1886146.pdf OECDDACGuidelinesonSecuritySystemReformandGovernance(2005) http://www.oecd.org/dataoecd/8/39/31785288.pdf DACPrinciplesforGoodInternationalEngagementinFragileStatesandSituations(2007) https://www.oecd.org/dataoecd/61/45/38368714.pdf DevelopmentAssistanceManualDACPrinciplesforEffectiveAid(1992) http://www.oecd.org/dataoecd/31/12/2755284.pdf WholeofGovernmentApproachestoFragileStates(2006) https://www.oecd.org/dataoecd/15/24/37826256.pdf
EVALUATINGCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIESOECD2008
TABLEOFCONTENTS
FOREWORD ..................................................................................................................................................... . 3 PREFACE .......................................................................................................................................................... . 8 Whatisthepurposeofthisguidance?........................................................................................................ 8 Whowillbenefitfromtheguidanceandhow?........................................................................................... 8 Howtousetheguidance............................................................................................................................. 9 Emerginglessonsfromtheanalyticalworkunderpinningthisguidance.................................................. 0 1 INTRODUCTIONTHECONFLICTPREVENTIONANDPEACEBUILDINGCONTEXT........................................... 1 1 Theneedforstrategicpolicydevelopment............................................................................................... 2 1 Evaluatingconflictpreventionandpeacebuilding.................................................................................... 3 1 Keyterms................................................................................................................................................... 4 1 Understandingpeace............................................................................................................................. 5 1 Peacebuilding......................................................................................................................................... 5 1 Conflictprevention................................................................................................................................. 6 1 Conflictprevention,peacebuildingandconflictsensitivity:Whentousethisguidance?.................... 6 1 OVERVIEWOFKEYSTEPSINPLANNINGANDEVALUATINGCONFLICTPREVENTIONANDPEACEBUILDING WORK............................................................................................................................................................. 9 1 1.Introduction........................................................................................................................................... 9 1 2.Somebasicprinciples............................................................................................................................. 9 1 Conflictsensitivity.................................................................................................................................. 0 2 Genderawareness ................................................................................................................................. 1 . 2 Protectionandethicalresponsibilities................................................................................................... 2 2 Otherconsiderations.............................................................................................................................. 2 2 3.Outlineofthemainelements................................................................................................................ 3 2 3.1Preconditions:Programmingtoimproveworkandstrengthenevaluation........................................ 4 2 3.2.Planningandpreparingtheevaluation............................................................................................... 6 2 3.2.1Definethepurposeanduseoftheevaluation ............................................................................. 6 . 2 3.2.2Decidethescopeoftheevaluation .............................................................................................. 7 . 2 3.2.3Doorobtainaconflictanalysis..................................................................................................... 8 2 3.2.4Outlinekeyevaluationquestions.................................................................................................. 9 2 3.2.5Taketimingandlogisticalissuesintoconsideration..................................................................... 0 3 3.2.6Coordinatewithotheractors....................................................................................................... 1 3 3.2.7Considerconductingajointevaluation ........................................................................................ 2 . 3 3.2.8Selectevaluationcriteria............................................................................................................... 3 3 3.2.9Deviseevaluationmanagement.................................................................................................... 3 3 3.2.10DevelopTermsofReference....................................................................................................... 3 3 3.2.11Selecttheevaluationteam......................................................................................................... 4 3 3.2.12Contracting.................................................................................................................................. 4 3
3.3Conductingtheevaluation................................................................................................................... 5 3 3.3.1Identifytheimplementationlogicandtheoryofchange............................................................. 5 3 3.3.2Dealwithmissingbaselinesandothergaps................................................................................. 6 3 3.3.3Gatherdata................................................................................................................................... 6 3 3.3.4Examinetheeffortusingvariouscriteria...................................................................................... 9 3 3.3.5Lookatthebigpicture.................................................................................................................. 5 4 3.4Concludingandlearningfromtheevaluation..................................................................................... 7 4 3.4.1Drawconclusionsandmakerecommendations........................................................................... 7 4 3.4.2Ensurequality................................................................................................................................ 7 4 3.4.3Conductreporting......................................................................................................................... 8 4 3.4.4Disseminate,feedbackandengageinalearningprocess............................................................. 8 4 CONCLUSION................................................................................................................................................. 0 5 BIBLIOGRAPHY............................................................................................................................................... 1 5 ANNEX1SELECTEDLISTOFKEYCONCEPTSANDTERMINOLOGY................................................................ 6 5 ANNEX2USEFULWEBSITESANDRESOURCES.............................................................................................. 0 6 ANNEX3PLANNINGANDMONITORINGPROGRAMMES,PROJECTSANDPOLICIESINCONFLICT PREVENTIONANDPEACEBUILDINGCONTEXTS............................................................................................ 2 6 ANNEX4CONFLICTANALYSIS....................................................................................................................... 8 6 ANNEX5CONFLICTSENSITIVITY ................................................................................................................... 5 . 7 ANNEX6UNDERSTANDINGANDEVALUATINGTHEORIESOFCHANGE....................................................... 7 7 ANNEX7EVALUATIONAPPROACHES ........................................................................................................... 5 . 8 ANNEX8SAMPLETERMSOFREFERENCE(TOR)........................................................................................... 0 9 ANNEX9FRAMEWORKFORFEEDBACK........................................................................................................ 3 9 TABLES Table1.Hierarchyofevaluationscopes................................................................................................ 7 2 Table2.Keyquestionsforconflictandpeaceanalysis.......................................................................... 9 2 Table3.Evaluationvs.Monitoring......................................................................................................... 5 6 Table4.Asampleofimpactindicatorsandmethods............................................................................ 7 6 Table5.Summaryofselectedconflictanalysistools............................................................................. 1 7 Table6.Commontheoriesofchange.................................................................................................... 2 8 FIGURES Figure1.Conflictpreventionandpeacebuildingwork........................................................................... 8 1 Figure2.SampleResultsChainforSecuritySystemReform.................................................................. 4 6 BOXES Box1.AssessingtheImpactofDevelopmentCooperationinConflictZones..................................... 9 1 Box2.Conflict(in)sensitiveevaluation................................................................................................ 1 2 Box3.Confrontingpsychosocialtrauma............................................................................................ 2 2
Box4.Integratinganalysis,evaluationandredesign........................................................................... 5 2 Box5.Whatareevaluationsusedfor?................................................................................................ 7 2 Box6.WorkingtogetherinMauritania ................................................................................................ 2 . 3 Box7.Jointmultidonorevaluation:Learningtogetheraboutpeace.................................................. 3 3 Box8.Composinganevaluationteam:Questionstoconsider............................................................ 4 3 Box9.Makingtheoriesofchangeexplicitforevaluation..................................................................... 6 3 Box10.Quotebox:Datainpostconflictsituations............................................................................... 7 3 Box11.Quotebox:Howsecurityaffectsaccessanddata..................................................................... 9 3 Box12.Typesofsuccessandfailure....................................................................................................... 1 4 Box13.Efficiencyandspendingonsecurity........................................................................................... 3 4 Box14.Coverageexample...................................................................................................................... 5 4 Box15.DoNoHarm:AnExamplefromTajikistan.................................................................................. 6 7
PREFACE
Whatisthepurposeofthisguidance?
Asgrowingsharesofaidresources,timeandenergyarebeingdevotedtoconflictpreventionand peacebuildingprojects,programmes,andpolicystrategies,moreevidencedemonstratingthe effectivenessoftheseendeavoursisessential.Thereisanincreasedinterestamongdonorsand practitioners,aswellaspeopleaffectedbyviolentconflict,tolearnmoreaboutwhatdoesanddoesnot work,andwhy.Thisquesttoimproveourunderstandingofwhatcontributespositivelytopeaceis motivatedbythedesiretodevelopmorecoherent,coordinatedandeffectiveinterventionsatalllevels. Theprimarygoalofthisguidanceistoprovidedirectiontothoseundertakingevaluationsofconflict preventionandpeacebuildingprojects,programmes,andpolicies(hereafterreferredtoasactivities).It aimstoassistpolicymakersandpractitionersworkingintheconflictpreventionandpeacebuildingfield tobetterunderstandtheroleandutilityofevaluation,andatthesametimetohelpthoseworkinginthe fieldofevaluationbetterunderstandthesensitivitiesthatapplyinthisfield.Withthatdualobjectivein mind,thisguidancewillofferadviceonthoseaspectsofevaluatingconflictpreventionandpeacebuilding activitiesthatdifferfromevaluationofhumanitariananddevelopmentinterventions.
Whowillbenefitfromtheguidanceandhow?
Differenttargetaudienceswillbenefitindifferentwaysfromthistext.Theprimaryaudience includespolicystaff;fieldanddeskofficersfromdonordevelopmentagencies,especiallythose responsibleforconflictpreventionandpeacebuildingpolicystrategiesoractivitieswhomaybeinvolved incommissioningorsupportingevaluations;andevaluationmanagerswithindonoragencies. Implementersandprogrammemanagers,includingnongovernmentalandinternationalorganisations (NGOs,IOs),UnitedNationsorganisationsandotherdevelopmentagencies,willalsobenefitfrom enhancedunderstandingoftheuseandvalueofevaluationanditsimplicationsforprogramming. Evaluationconsultantsworkinginconflictenvironmentsorhiredforarelevantconflictpreventionor peacebuildingevaluationwillgainaclearerviewofwhatcommissionersexpectfromtheirwork. Specifically,thisguidancesupportsevaluatorsandthosecommissioningevaluationsby: Providinggreaterclarityonkeyemergingconceptsinthisfieldandtipsfordealingwith commonproblems. Suggestingtechniquesfortheuseofconflictanalysestobetterassesswhetheractivitiesina particularconflictarerelevantandprevent'doingharm'. Furnishingprinciplesforethicalevaluationinconflictenvironments. Demonstratingtheimportanceofassessingassumptionsabouthowpeacecanbeachieved (theoriesofchange). SpecifyinghowDACCriteriaforevaluatingdevelopmentassistancecanbeadaptedtothisfield. ProvidingadviceondraftingTermsofReferenceandpickingeffectiveteams.
Theguidancedocumentalsohelpsconflictpreventionandpeacebuildingpractitionersandpolicy makersby: Promotingtheuseofevaluationtoimprovelearningandaccountability,andsuggestingwaysin whichevaluationcanprovidelessonsaboutoperationaldesign(beyondwhatislearnedthrough auditandmonitoring). 1 Encouragingfurthercriticalreflectionaboutwhatactuallycontributestopeaceandwhatdoes not. Helpingtorefinetheoriesaboutthecausesanddynamicsofconflict(andthelinksbetween them),whichwillinturnleadtomorerelevantinterventions,ultimatelyenhancing effectiveness.
Howtousetheguidance
Thisguidanceisnotmeanttoserveasanallencompassingorconstrainingmanualforconflict preventionandpeacebuildingevaluations.Rather,itshouldcontributetothoughtfulapproachesby highlightingandclarifyingspecificchallengesforevaluatinginthisfield.Itoutlinesthekeystepstotake andmainpointstoconsiderduringanevaluationprocess.Thisinformationshouldbeappliedintelligently andadaptedcarefullytospecificcontexts.Thereportbuildsonexistingliteratureandexperiencesandis thusnotexhaustive;aBibliographyprovidesfurthergeneralresourcesforthereader.Giventhediversity oftheintendedaudience,somesectionsmaybemoreorlessrelevantforindividualreaders;andthetext belowaimstoclarifythespecifictargetsofeachsection,includingtheannexes. TheIntroductionhighlightsthedevelopmentofandchallengestoevaluationintheconflict preventionandpeacebuildingfield,andisthereforekeyforallreaders.Italsooutlineskeyconcepts, whichwillbeofparticularrelevancetothosewithlimitedexperienceintheconflictandpeacedomains. Themainsectionwalksthereaderthroughthekeyelementsoftheevaluationprocess.Thisoutline willbeusefulforreaderswithlimitedevaluationbackgroundandcanhelpprovideafootpathforthinking through,planning,supportingandperformingtheevaluationitself.Italsoprovidesseasonedevaluators withfurtherideasforworkinthisparticularfield.Thesectionbeginswithadescriptionofupstream measurestohelpprogrammeplannersandpolicymakerscreateeffective,assessableintervention strategiesandactivities. Annexesarereferredtothroughoutthetext.Annex1includesalistofkeyterms.Thefinalannexisa feedbackframeworkthatcanbeusedtocommentonthisworkingdraftorprovideinsightfromthetest applicationphase(2007/2008).
Areviewofmorethan75evaluationsintheconflictpreventionandpeacebuildingfieldpointedtoanoveremphasisonfinancial managementissuesandalackoflessonlearning(FAFO,2006).
Emerginglessonsfromtheanalyticalworkunderpinningthisguidance
Thejointprocessofdevelopingthisguidancehasbeguntorevealsomeimportantlessonsfordonoragencies andothersworkingintheconflictpreventionandpeacebuildingfield.Thefollowinglistofemerginglessons willberevisedandupdatedoncethisworkingdrafthasbeenfieldtested. 1) Donorsshouldpromotethesystematicuseofevaluationforallconflictpreventionand peacebuildingwork,andrequireimplementingpartners,suchasNGOs,toconductevaluations. Evaluationcansupportlearningandaccountabilityasprofessionalsinthisareaofdevelopmentco operationstrivetoimprovepracticeandresults.Suchlearningiskeytobecomingmoreeffectiveat buildingpeace. 2) Aclearneedforabetterstrategicpolicyframeworkforconflictpreventionandpeacebuilding workhasbeendemonstrated.Thereisaneedtoevaluateatthestrategiclevelandtolookatthe interconnectionsbetweenstrategies,policies,programmesandprojects.Policiesandoperationsin thissensitivefieldneedtobemoreeffectivelylinkedagoalwhichcouldbeachievedinpartby workingwithpractitionersandpolicymakerstoupdatetheexistingDACGuidelinesonHelping PreventViolentConflict(includingthe1998GuidelinesonConflict,PeaceandDevelopmentCo operation),inwhichdonorsrecognisedthatworkontheseissuesisacentralpartofdevelopment, extendingbeyondhumanitarianassistancealone. 3) Evaluationsshouldbefacilitatedthroughbetterprogrammedesign,evenintheplanningstages when,forinstance,objectivesshouldbeclearlyarticulatedtofacilitatefutureassessmentofresults. Thereisageneralneedforfurtherdevelopmentintermsofplanning,funding,managementand implementationofactivitiesthattrytopreventconflictorbuildpeace.Inthisfieldingeneral,there isaneedtobuildtailoredtoolsforlearningandaccountabilitytocontributetothe professionalizationofinterventions,includingtheidentificationofbestpractices. 4) Coherentandcoordinatedinterventionandpolicystrategiesareneededtomakeprogress towardspeace.Donorscannotrelysolelyonaidandmustlookatotherpolicyinstrumentsand theirimpactsonconflictandthechancesforpeace.Strategicengagementatvariouslevelsand acrossgovernmentsisessential. 5) Conceptsanddefinitionsofpeacebuildingandconflictpreventionrequireclarification.Evaluators shouldworkwithstaff,policymakers,managersandstakeholderstodetermineandassessthe conceptsofpeacetheiractivityisoperatingon. 6) Theresultsofconflictanalysisneedtobetranslatedintoaction,usedtoinfluencethe programmingandevaluationprocessesandlinkedtootherformsofanalysis,suchasgovernance assessments,poweranddriversofchangeanalysis,aswellasearlywarningindicators.(Note:As fieldapplicationsareconductedandaslearningandpracticesevolve,thislistmayberefined.) 7) Theuseofmixedmethodapproachestoevaluationsisrecommendedduethecomplexityand multifacetednatureofinterventionsinthisfield. 8) Jointevaluationsallowformoreharmonisedapproachesthatdemonstratehoweffortsofdifferent donorsaddup.Involvingcountrypartnersisalsoimportantforunderstandinghowchangeoccurs andisakeyelementofsupportingtheParisDeclaration.
10
INTRODUCTION THECONFLICTPREVENTIONANDPEACEBUILDINGCONTEXT
Mostcontemporaryarmedconflictstakeplacewithinstates,andthemajorityoftheirvictimsare civilians.Notonlyisthehumancostofarmedconflictdevastating;itsimpactsonpolitical,socialand economicdevelopmentareprofound.Whenviolentconflictbreaksout,developmentisderailed.The benefitsofdevelopmentassistancecanbereversedbyviolentconflict,whichisnotonlyan accompanimentofpovertybutoneofitsmaincauses.Thereisalsoanemergingunderstandingthat developmentassistanceandotherdonorpolicies(whennotwelldesigned,implementedandco ordinated)canincreasetensionsorrestraincapacitiesforpeace. Whilescholarsindiverseacademicdisciplineshavelongbeenconcernedwithissuesofwarand peace,conflictanalysisandpeaceresearchonlyemergedasadistinctinterdisciplinaryacademicfieldin the1960s.Inthelastdecade,approachestounderstandingandrespondingtoboththeimmediateand structuralcausesofviolentconflicthaveevolvedsignificantly.Workinfragilesituations 2 andconflict affectedcountrieshasbecomeanintegralpartofthedevelopmentchallenge,andisnowseenasa prerequisiteofsustainabledevelopment. Policyinstrumentshavebeendevelopedtohelpdonorsandpartnercountriesdesignandimplement strategiesthataddressmultiplethreatsandinsecurities,andtocontributetoconflictpreventionand peacebuilding.Innovativeoperationalandanalyticaltoolshavebeenproposed,thoughtheyarestill underused.Inparticular,conflictanalysishasyettoexertamajorinfluenceonplanninganddesign. Recognisingthatmuchremainstobedonetoimprovethepeacebuildingactivities,donorswith thehelpofpartnercountriescommittedthemselvesinthe2005ParisDeclarationonAidEffectiveness toachievingmoresynchronisedandeffectivemonitoringandevaluationapproaches,especiallyinconflict areas. 3 Newdonorstrategies,suchaswholeofgovernmentapproachesandanemergingemphasison policycoherencereflectagrowinginterestinandcommitmenttofacingthesechallenges.Still,working coherentlyacrossgovernmentsandorganisationsonthecomplexinterfaceofdevelopment,diplomacy, defence,tradeandfinanceremainschallenging.
Analystsanddonorsstillholddifferentnotionsofwhatfragilitymeans.IntheevolutionoftheconceptamongOECDmembers, fragilestateswereonceequatedwithdifficultpartners.Amorenuancedapproachhassinceemergedbutdifferentagencies haveadopteddifferentconcepts.Asusedhere,"fragilesituationsreferstonational,regionalandlocalterritorieswherethe state(theexecutive,thelegislativeandthejudiciary,includingcentralandlocalauthorities)lacksthecapacityand/orpolitical willandlegitimacytosupportequitabledevelopment.Thesesituationstendtobecharacterisedbypoorgovernance,tobeprone toviolentconflict,andtoshowlimitedprogresstowardstheMillenniumDevelopmentGoals.Anaggregateofgovernanceand securitycriteria,orofcapacity,accountabilityandlegitimacycriteria,isusuallyusedtomeasurefragility. 3 TheParisDeclarationonAidEffectivenessdemonstratesdonorcommitmenttoadapttodifferingcountryenvironmentsand togiveincreasedattentiontofragilestatesandconflictaffectedcountries.Seefulltextat: http://www.oecd.org/dataoecd/11/41/34428351.pdf.
11
Theneedforstrategicpolicydevelopment
Donorsandothersworkingtohelppreventviolentconflictandsupportpeacehaverecognisedthe needtodevelopmorestrategic,coherentandcoordinatedpoliciesandprogrammesinthisfield.Inthe late1990s,aslessonsemergedfromamajorjointevaluationofemergencyassistancetoRwanda,andin theaftermathofadevastatinglyviolentdecade,pressureonandfromdonorstofindbetterwaysof preventingviolenceincreased. 4 Ongoingchallengestoworkonconflictandpeaceincludethefactthatworkinandonconflict involvesactivitiesthatdifferfromtraditionaldevelopmentoperations,andtakeplaceinhighlypoliticised environments.Comparedtothenumberofhumanitariananddevelopmentactivities,thereisarelatively smallsampleofconflictpreventionandpeacebuildingactivitiesperseandthereforeofevaluations fromwhichtodrawexperienceorguidance. 5 Inaddition,numerousvariablesaffectconflictandpeace dynamicsandoutcomes.Thus,evenwhereusefulconclusionsorlessonscanbegleanedfroman experienceinoneconflictenvironment,theymaynotbereadilyapplicabletootherconflictcontexts, evenwithinthesameregion.Lessonlearninganddiscernmentofgoodpracticeisthereforedifficultand itisstillnotclearwhetherdonorsareusingtherightobjectivesandeffectivestrategiestoachieve them. Further,thesocalledfishbowleffectofahighlypoliticisedandoftenmediadenseenvironment meansthatthereissometimesgreatpublicattentionon,andcorrespondinglyhighstakesfor,evaluators. Whenhumansufferingishighanddonorcontributionslargeandvisible,thedesiretoseepositiveresults canplaceadditionalpressuresonevaluatorsandmanagersinthefield.Atthesametimethereisa tendencyforuneven,shorttermorunsystematicfundingandengagement,whichmayfollowthe headlinesofmajorviolentconflictsandresultinthesocalledforgottencrisesanddonororphans. Suchdynamicsdonotlendthemselvestothoughtfullearningfromandimprovinguponpastexperiences. Theresultsofavailableresearchstudies 6 andoperationalevaluations 7 haveidentifieda considerablegapbetweenthepolicyintentofdonorcountriesanddefactooutcomesontheground.A numberofthesestudieshavehighlightedthefactthatconflictpreventionandpeacebuildingpolicies, programmesandprojectslackcoherencewitheachother,aswellaswithanoverallcountrystrategy. 8 Clearly,thereisagrowingneedforthoughtfulexaminationofdonorpracticesintheseareas. Distinctiveapproachesarerequiredtoeffectivelydeliverconflictpreventionandpeacebuildingaid especiallyintermsofownership,harmonisation,alignmentandresults.Asevaluationisoftenan importantlearningtool,itishopedthatbetterevaluationwillhelpaddresssomeoftheseconcerns. Giventhegapsbetweendonorintentionsandoutcomesinthefield,coordinationproblems,andnewly
TheRwandaevaluationwasundertakenby19OECDmemberbilateraldonoragencies,theEuropeanUnionandthe DevelopmentAssistanceCommittee(DAC)SecretariatoftheOECD;ninemultilateralagenciesandUNunits;thetwocomponents oftheInternationalRedCrossandRedCrescentMovements;andfiveinternationalNGOorganisations(Eriksson,J.etal.,1996). FormoreontheimpactsoftheevaluationseeBortonandEriksson,2004. 5 FAFO,2006. 6 SeeDobbinsetal.,2005;Paris,2004;Collieretal.,2003;andStedman,CousensandRothchild,2002. 7 Amongothers:Cutillo,2006;Dahrendorf,2003;Donini,2002;Porter,2002;Stockton,2002;Sommers,2000;Reindorpand Wiles,2001;Duffield,LautzeandJones,1998;andErikssonetal.,1996. 8 Forinstance,theUtsteinStudy,thatanalyzed336peacebuildingprojectssupportedorimplementedby,Germany,the Netherlands,theUnitedKingdomandNorwaydeterminedthattherewasastrategicdeficitbetweenthestrategicpolicylevel andthefieldprogrammes,andthatthisgappresentedasignificantobstacletosustainablepeacebuilding(Smith,2003).
12
emergingaidinstruments(especiallyinthesecuritysector),donorsshouldcontinueconsideringhowbest toadoptmorecoordinatedandwholeofgovernmentapproachestoevaluationitself.Forinstance, whenplanningevaluationstrategiesorcalendarsitisimportanttoplannotonlytocoverindividual peacebuildingactivitiesbuttolookatoverallcontributionstopeacebothinandacrossconflictareas. Thisworkonevaluationthereforerepresentsacontributiontotheongoingprocessofimproved donorpoliciesandmoreeffectiveinterventionsinconflictaffectedareasandsituationsoffragility.
Evaluatingconflictpreventionandpeacebuilding
Evaluationofferssystematicandobjectiveassessmentsoftherelevance,effectiveness,impact, sustainabilityandefficiencyofinterventions.Ithelpstoascertainthequalityofpoliciesandprogrammes, toenhancetheperformanceofparticipants,toidentifygoodpracticesandtodefineappropriate standardsforfutureoperations. 9 Proponentsoftransparent,evidencebasedpolicymakingand programmedesignsuggestthatevaluationisneededtotracktherelevanteffectsofconflictprevention andpeacebuilding,informthedesignofmorestrategicapproaches,andenhanceaccountability.Amore strategicapproachcanbeencouragedbyevaluationsthatlinkprogramme,policyandprojectlevels acrossgovernmentsandamongdonors.Asnewinstrumentsforimprovedaideffectivenessemergeinthe conflictfield(andindevelopmentassistanceoverall),toolsandapproachesforevaluationareevolvingas well. 10 Yet,aidpractitionersandprogrammershavesometimesresistedevaluation.Theyfrequentlyassert thatevaluationtakestime,consumesscarceskillsandresources,makesfutileattemptstoquantifythe unquantifiable,putsforwardunrealisticrecommendationsordivertsmanagementandstaffattention awayfromothervitalandurgenttasks. 11 Manypractitionersfeelthatestablishedevaluationapproaches areinadequateforassessingthenuancedworkofconflictpreventionandpeacebuilding(suchas measuringchangingattitudesorculturalshifts). 12 Aselsewhere,thisresistancetomonitoringand evaluationmustbeovercome,notleastbecauseevaluationisbecominganearlyuniversalobligationof manyfunders(includingbothpublicandprivatedonors).Furthermore,policymakersandpractitioners havebeguntoappreciatetheneedforbetterevaluationinordertolearnfromexperience, professionalisepeaceworkandimprovepolicymaking,coordinationandprogramming.Itishopedby manythatbetterlearningfromevaluationwillincreaseeffectiveness. Giventhisgrowinginterestinandsupportforevaluationandthelargeinvestmentofresourcesin conflictpreventionwork,itisperhapssurprisingthatthereisstillcomparativelylittleevidence demonstratingtheeffectivenessofconflictpreventionandpeacebuildingwork. 13 Partoftheexplanation
OECD,2002. Seeforinstancerecentworkonjointandmultidonorevaluations,countryprogrammeevaluations,impactevaluationand generalbudgetsupportevaluations:www.oecd.org/dac/evaluationnetwork. 11 AndersonandOlson,2003. 12 ChurchandShouldice,2003. 13 AccordingtotheOECDDAC,OfficialDevelopmentAssistance(ODA)forconflictpreventionandpeacebuildingmorethan doubledduringtheperiod200005,asmeasuredbyreportingonthesixconflictcodes(securitysystemmanagementand reform;civilianpeacebuilding,conflictpreventionandresolution;postconflictpeacebuilding;reintegrationandSALWcontrol; landmineclearance;andchildsoldiers).Overall,ODAflowsincreasedfromaboutUSD650millionin2000tomorethan USD1.6billionin2005.However,thesefiguresareincomplete,astheydonotincludelargenonODAexpensesforsecurity relatedspendinginareassuchasdemilitarisation,thetrainingofmilitaryinnonmilitarymatterssuchashumanrights,orthe extensionofODAinrelationtopeacekeepingactivities.
10
13
forthelackofsystematicevaluationcanbeexplainedbytheperceptionthatthisfieldposespecial challengestoevaluation,including: Theconflictcontext,whichinvolveshighriskstosecurityandhumanlife,inacomplex,rapidly changingenvironment. Thisisanemergingandfastevolvingarea,anditspolicyframeworkisstilldeveloping. Relativelylimitedtheoreticalfoundations,includinglackofagreeduponorprovedstrategiesof howtoeffectivelyworktowardspeace.Strategiesputforthorprogrammessuggestedareoften contested,anditcanbedifficultforthoseworkinginthisfieldtobackuptheiractionswith soundempiricalevidence. Thelackofpreconditionsandinputsforeffectiveevaluationsincludingnobaselines,little monitoring,andmissing,unreliableorcontradictorydata,aswellastheoftenineffective articulationofobjectivesandtheories,whichmakesprogrammeslesseasilyevaluable. 14 Thelackofclearlystatedandtestabletheoriesofchange(theimplicitorexplicitunderstandings ofhowonehopesthatwhatoneisdoingwillcontributetopeace). Differencesinterminologies,planningculturesandapproachesbetweenvariousactorsworking inconflictareas(defence,development,humanitarian,trade,diplomacy,etc.). Thedifficultyofunderstandingimpactsandassigningattribution. 15
Fortheseandotherreasons,evaluationofconflictpreventionandpeacebuildingeffortshasonly recentlybeencarriedoutsystematically.Thefieldneedstobuilditsownlearningandaccountabilitytools tocontributetotheprofessionalisationofinterventions,includingtheidentificationofbestpractices. Thisguidanceworkstofacilitateanimportantstepinthatdirection.
Keyterms
Giventhecontinuedevolutioninthisfield,workingdefinitionsofkeytermsandconceptshaveyetto besettled.Overlyflexibleandholisticdefinitionsleadtofragmentationofeffortsandlackofselectivityin preventioninterventions.Alackofconsensusonvocabularycancreateconfusionandformsabarrierto harmonisedorcoordinatedapproaches.Reachingagreementonterminologywouldhelpharmonise policieswithinthedevelopmentcommunity.Tocontributetothisongoingprocess,alistoftermsand conceptsisincludedinAnnex1.Thefollowingexplainskeyconceptsastheyareusedinthisguidance.
14
Evaluability:theextenttowhichanactivitycanbeevaluatedinareliableandcrediblefashion,i.e.areitsobjectivesadequately definedanditsresultsverifiable?(OECD,2002). 15 Attributionistheascriptionofacausallinkbetweenobserved(orexpected)changesandaspecificintervention.While attributionposesaprobleminallsocialsciences,influidconflictcontextsattributingtheimpactofanyparticularpolicyorsingle interventiononthecomplexarrayofactorsanddynamicscanbeevenmoredifficult.Forexample,actorsworkingfromother directionsbeyondthescopeoftheevaluation(militaryinterventionsortradepolicyforexample)mayactuallyberesponsiblefor changesthatareattributedtoconflictpreventionorpeacebuildingactivities.
14
Understandingpeace Afrequentcomplaintheardamongevaluatorsinthisfieldisthelackofclarityandconsensus regardingtheactualgoalsofpeacebuildingwork.Whatdoesittangiblymeantocontributetopeace? Avarietyofdefinitionsorunderstandingsofpeaceareatplayinconflictpreventionandpeacebuilding contents.Forexample,JohanGaltungmadethedistinctionbetweennegative(absenceofwar)and positivepeace(societywithoutphysicalnorstructuralviolence).Anemergingcommonunderstandingin thefieldisthatthepathtosustainedpeaceleadsthroughconflicttransformationandsocialchange 16 and finallyasetofbuildingblocksneedstobeinplacewhenpositivepeaceisapproached. 17 Thereisstill debateaboutwhatpeaceisandhowitcanbeachieved;suchdebateshaveprovedtobeanobstaclefor evaluationsinpeacebuilding.Achievementsinconflictpreventionandpeacebuildingcannotbeevaluated withoutaclearvisionofwhatkindofpeaceshouldbebuiltmakingitallthemoreimportantforthose planning,implementingorevaluatingpeaceworktobeclearaboutwhatmeaningsorimplicitdefinitions areinuse.Makingimplicitunderstandingsofpeaceexplicitwillbeusefultobothevaluatorandplanner andwillhelpcoordination. Peacebuilding Peacebuildinghasbecomeanoverarchingtermforanentirerangeofactionsdesignedto contributetobuildingacultureofpeace.Thetermpeacebuildingbecamepartofthepolicyvocabulary throughtheUnitedNationsAgendaforPeace:PreventiveDiplomacy,PeaceMakingandPeacekeepingof 1992,andhasevolvedconsiderablyamongpractitioners,policymakersandthegeneralpublicoverthe pastdecade. 18 Themid1990switnessedarapidincreaseinpeacebuildingactivitiesbyavarietyofactors, rangingfrominternationalandregionalorganisations(theUnitedNations,theEuropeanUnion,the AfricanUnion)toacademicinstitutions,foundations,civilsocietygroups,socialmovements,business groups,andthemedia. Peacebuildinghasoftenbeendescribedinthepostconflictcontext(thoughthetermisusedby somebeforeandduringconflict)asactiontoidentifyandsupportmeasuresandstructuresthatwill strengthenandsolidifypeaceinordertoavoidarelapseintoconflict. 19 Forthisguidance,peacebuilding coversabroadrangeofmeasuresimplementedinthecontextofemerging,currentorpostconflict situationsandwhichareexplicitlyguidedandmotivatedbyaprimarycommitmenttothepreventionof violentconflictandthepromotionofalastingandsustainablepeace. 20
16 17
Lederach,2002. PaffenholzandReychler,2007. 18 TheconceptwassubsequentlyelaboratedinSecurityResolution1325onwomen,peaceandsecurityin2000.InFebruary 2001,aSecurityCouncilPresidentialStatementrecognisedthatpeacemaking,peacekeepingandpeacebuildingareclosely interrelated.In2004,theUnitedNationsHighLevelPanelonThreats,ChallengesandChangeclarifiedthatpeacebuildingshould focusonstatebuilding,usuallybutnotexclusivelyinpostconflictcountries.TheUNhasestablishedthePeacebuilding Commission,whichnowcoordinatesconflictpreventionandpeacebuildingefforts. 19 UN,1992. 20 DACIssuesBrief,2005.
15
Conflictprevention Adecadeago,conflictpreventionreferredonlytoactionsundertakenintheshorttermtoreduce manifesttensionsandtopreventtheoutbreakorrecurrenceofviolentconflict. 21 Itnowincludeslong termengagementaswellasshorttermresponses. 22 Itaddressesthebuiltincapacitiesofsocietiesto dealwithconflictinginterestswithoutresorttoviolence. 23 Italsoextendstothemanagementofdisputes withdestabilisingpotentials.Suchworkhelpsdelegitimisethebeliefthatviolenceisaninevitableor acceptablewayofresolvingdisputes,makingnonviolentalternativesknownandmoreattractive, addressingstructuralandimmediatecausesandreducingvulnerabilitytotriggers. Thegoalisnottopreventallconflict.Someconflictisnatural,inevitableandoftenapositivepartof developmentandotherchangeprocesses.Instead,theemphasisisonpreventingharmfulviolent responsestotheinevitablediverginginterestsorclashingobjectivesextantinallsocieties. Conflictprevention,peacebuildingandconflictsensitivity:Whentousethisguidance? Comparativeanalysisofactivitiesacrosscontextsandtimesshowsthatapolicyorapproachthat waslabelledconflictpreventionorpeacebuildinginsomeplacesisnotnecessarilydescribedassuch elsewhere. 24 Theconfusionoverdefinitionsofconflictpreventionandpeacebuildingleadsmanyto assume(falsely)thatbybeingconflictsensitivetheyareipsofactoalsodoingpeacebuildingwork.It alsoleadsmanypeopleworkinginconflicttoassumethatadvancesincriticalstructuralareaswill contributeautomaticallytothereductionofconflictandthepromotionofpeace.Forinstance,many donorfundedprogrammesandpoliciesareundertakenontheassumptionthatprogresstowards liberalisation,economicgrowth,prosperity,humanrightsanddemocracycontributetopeace.Evidence showsthatthisisnotalwaysthecasewhilesomeofthoseeffortsdocontributetopeace,othershave negativeornegligibleeffectsonviolentconflict.Developmentcooperationshouldthereforedeliberately workinandonconflictratherthansimplyattemptingtogetaroundconflict. 25 Giventhisconfusion,thefollowingdefinitionofconflictpreventionandpeacebuildingwas developedtoserveasthebasisforthisguidance: Conflictpreventionandpeacebuildingactivitiesareprojects,programmes,policies,strategies orotherinterventionsthatadoptgoalsandobjectivesaimedatpreventingconflictorbuilding peace; they are usually (but not always) focused on a particular conflict zone an area threatenedby,inthemidstof,orrecoveringfromseriousintergroupviolence. 26
21
SeeOECDDevelopmentMinistersStatementonConflict,PeaceandDevelopmentCooperation(May1997)inOECDDAC, 1998and2001. 22 Somepolicymakersandacademicsdistinguishbetweenoperationalandstructuralpreventionorbetweenearlyandlate prevention.Forthepurposesofthisguidance,conflictpreventioncomprisesallofthesecategories.SeeforexampleMenkhaus inPicciottoandWeaving,eds.2006. 23 SeeOECDMinistersStatementonHelpingPreventViolentConflictinOECDDAC,1998and2001. 24 OECDDACandCDA,2007. 25 OECDDAC,1998and2001. 26 Thereisnointernationallyagreeddefinitionofthetermviolence.Thetermintergroupviolenceisusedheretodistinguish theintentionalandillegitimateuseofarmedforce,includingbothinterandintrastateconflicts,fromothertypesofviolence suchascriminalactivityandinterpersonalviolence.Whilesomeconflictpreventionandpeacebuildingactivitiesmayaddress suchformsofviolence,thisguidancedealsprimarilywithintergroupviolence(war).
16
Usinganinterventionsgoalsandobjectivesasthedeterminingcriteriahelpstoclarifythedifference betweenconflictsensitiveprogrammingandconflictpreventionandpeacebuildingactivities.Thefocus ofthisguidanceisonpoliciesandactivitiesworkingonconflictmeaningtheyareintentionallytryingto impactconflictandpeaceprospects,notonconflictsensitiveevaluationsperse(thoughsomepoliciesor projectsworkinginconflictmayalsobenefitfromthisguidanceandsomeadvicewillbefurnishedon conflictsensitivity).Alleffortsundertakeninconflictareasshouldbeconflictsensitive.Interventions intendedtopreventconflictandbuildpeacemustalsobeaccountablefortheireffectivenessinimpacting onthespecificfactorsthatdriveandshapeconflictandthecontributionstheymaketopeace. 27 Still,agoalsbaseddefinitioncanbedifficulttooperationalise.Inordertoprovideamorepractical definition,fourkeycategoriesofconflictpreventionandpeacebuildingactionandstrategywereoutlined whiledevelopingthisguidance. 28Thefirstcategorycoversinterventionsthatsupportthepromotionofa cultureofjustice,truthandreconciliation,whichcanbecriticalinpostconflictregionsinordertoheal thewoundsofconflictandreconnectsociety.Second,capacitybuildingandpromotionofgood governancearecriticaltohumansecurity,especiallywherestatesareunableorunwillingtodeploy peacefulmeanstoresolveconflictorsustainablyandindependentlyfacilitateprovisionofkeybasic services.Third,conflictpreventionandpeacebuildingpoliciesandactionsoftenworktocreateincentives forsystemsthatpromotethepeacefulresolutionofconflict.Supportingreformofsecurityandjustice institutionsincludingthejudiciary,penal,policing,parliaments,defenceandmilitaryactorsiscritical andshouldbeseenasalongtermprojecttoachievedemocraticgovernanceoverallsecurityinstitutions andforceswhiledevelopingawiderjusticeandsecuritysystemthatupholdstheruleoflawandrespect forthedignityofpoorpeople.Finally,socioeconomicdevelopmentandthepoliciestosupportitalso matter,before,afterandevenduringhostilities.Addressingstructuralviolenceandinequalityisessential toreducingtensionsandenhancingasocietyscapacitytopreventviolenceandisthusoftenafocusof conflictpreventionwork. Figure1outlinesexamplesofthesetypesofwork,towhichthisguidancecouldbeapplied,andis meanttopromotethinkingabouttheinterrelatedareasofinterventionthatarerequiredtopromote sustainablepeace.Thisisnotanexhaustivelist.Thelistisindicativeandshouldnotbeinterpretedasa limitationonthetypesofinterventionsthatcouldcontributetopeacenordoalllistedinterventions necessarilyalwayscontributetopeace.
27
Foramoredetaileddiscussiononthedifferencesbetweenconflictsensitivedevelopmentandexplicitconflictpreventionand peacebuildingactivities,seetheOECDDACandCDA,2007. 28 AjointworkshopoftheCDACollaborativeLearningProjectsandmembersoftheDACNetworkonDevelopmentEvaluation andtheDACNetworkonConflict,PeaceandDevelopmentCooperationwasheldinOsloin2006.
17
Figure1.
Conflictpreventionandpeacebuildingwork
Socio economic development

Balancedphysical reconstruction Soundandequitable economic management Equitableand balancedpoverty reduction Genderequality Equitableaccessto services Repatriationand reintegrationofthe displaced Sustainableuseof andequitableaccess tonaturalresources Socialinclusion projects
Good governance
Civilsociety development Freedomsof expression, association,etc. Mediadevelopment Powersharing Participatory processes,access Democratisationand electoralprocesses Transparencyand accountability Anticorruption Humanrights protection Ruleoflaw
Reform ofjusticeand security institutions

Securitysystem reform(police, military,intelligence) Smallarmsandlight weaponsreduction Nonviolent accompaniment Communitypolicing Peacekeeping Nonviolentobservers Disarmament, demobilisationand reintegrationof combatants Demining
Culture ofjustice, truthand reconciliation

Dialogueamong conflictinggroups Enhancingnonviolent disputeresolution systems Prejudicereductionor diversitytraining Traumahealing Capacitybuildingand traininginconflict resolution Peaceeducation Transitionaljustice processes Warcrimestrials
Transformingattitudes,structures,relationshipsandbehaviors
Source:InspiredbytheJointUtsteinStudyofPeaceBuilding,Utsteinpalette(Smith,2004a,pp.2728)andmodifiedduringaworkshopofthe DACNetworksinOsloin2006.Bottomtextfrom:InternationalAlert(2007bp.6)
18
OVERVIEWOFKEYSTEPS INPLANNINGANDEVALUATING CONFLICTPREVENTIONANDPEACEBUILDINGWORK
1.Introduction
Eachofthechallengesmentionedintheintroductionmakeschoosingaspecificapproachto evaluatingconflictpreventionandpeacebuildingactivitiesandpoliciesparticularlydifficult.Established evaluationapproachesmaybemoreorlessusefulbefore,duringandafterwidespreadviolentconflict. Manyworkinginthefield,bothevaluatorsandprogrammers,feelfrustratedbystandardevaluation approaches,claimingthattheyfailtocapturemanyofthesignificantdimensionsinvolvedinworkingin situationsofconflictandfragility. 29 Questionsofmeasurement,timescale,data,complexityand attributionhaverepeatedlybeenhighlightedasparticularweaknessesofestablishedmethodswhen theseareappliedtoconflictpreventionandpeacebuildingevaluations. 30 Thissectioncontributestoresolvingtheseconcernsbyoutlininggeneralprinciplesfortheevaluation process,includingageneralapproachtoplanningandconductinganevaluation.Inaddition,alistof commonevaluationmethods,withhighlightsofconflictspecificstrengthsandweaknesses,areoutlined inAnnex7.Thefollowingprinciplesshouldbetakenintoaccountduringplanning,includedintheTerms ofReference(TOR)andcarriedthroughouttheevaluationprocess.Whenappliedcarefully,these principlescanenhancethecredibility,useandrigouroftheevaluationprocessesandproducts.
Box1.
AssessingtheImpactofDevelopmentCooperationinConflictZones
[PossibleexamplefromNorthEastAfghanistantobeaddedbyGermany]
Source:
2.Somebasicprinciples
Thereisnotonecorrectorblueprintapproachforundertakingconflictpreventionandpeacebuilding evaluations.Evaluationisatoolboxandthegoldenruleistoapplytherighttoolfortherightquestion.It followsthereforethatonemustfirstconsidertheusesoftheevaluationanditspurpose.Whatdowe needtoknowforaccountabilityorlearning?Whatinformationcouldhelpimproveprogrammedesignor influencepolicymaking?Oncetheuseandpurposeoftheevaluationhavebeendetermined,andconflict analysiscarriedout,thechoiceoftheapproachwillbemorestraightforward.
29 30
ChurchandShouldice,2003. See,amongothers:FAFO,2006;OECDDACandCDA,2007;ChurchandRogers,2006;USAIDandManagementSystems International,2006.
19
Giventhecomplexnatureofsuchinterventionsitisoftennecessarytocombinedifferent approachesinordertoanswertheevaluationquestions.Today,themostcommonlyusedmethodin developmentevaluationisamixedmethodresultsbasedapproach,usingbothqualitativeand quantitativeinformation.Otherapproachesmayalsobeuseful,dependingonthecontext.Mostsuch evaluationsprovideusefulinformationonresultsatoutputandoutcomelevels,andonprocesses.Single methodevaluationsarenotconsideredadequateforconflictpreventionandpeacebuildinganalysis. 31 Theadvantagesanddisadvantagesofaparticularapproach,orcombinationofapproaches,shouldbe consideredinlightoftheevaluationpurpose,context,budgetandtimeavailable. Conflictsensitivity Sometimespolicies,projectsandprogrammesworkinginoronconflictdoharmoftenwithout intendingto. 32 Doingharminaconflictsituationmeanshavingimpacts(intendedornot,director indirect)thataggravategrievances,increasetensionorvulnerabilities,orperpetuateconflictinsome way.Thenotionofconflictsensitivityisintendedtomitigatesuchharmbyencouragingsystematically takingintoaccountboththepositiveandnegativeimpactofinterventionsontheconflictcontextsin whichtheyareundertaken,andviceversa. 33 Thisissuefirstemergedandenteredpolicydiscussionsin thecontextofhumanitarianinterventions,andsoonspreadthroughoutthedevelopmentfield.Conflict sensitivityisnowakeypillarofdevelopmentpolicyandinterventionstrategies. 34 Ithasnowbecomeclearthateffortstoaddressconflictissuesdirectlyorpreventviolencecanalso doharmbyfailingtoaccountfortheinadvertentimpactsoftheirhiring,targeting,timing,andother decisionsontheconflict. 35 Inotherwords,justbecausetheyareconflictpreventionandpeacebuilding efforts,thatdoesnotmeantheyareexemptfrombeingconflictsensitive.Itisalsoevidentthat evaluationstakingplacebefore,duringorafteraviolentconflictshouldbesensitivetoconflict. Theevaluationprocessitselfshouldbeconflictsensitiveandanevaluationoftheevaluators;aself revieworanintroductorystatementtotheevaluationreportmayexplainwhatmeasureswereorwere nottakentoensuretheconflictsensitivityoftheevaluationitself.Inaseparatestep,theevaluatorswill alsoassesswhetherornottheevaluationtargethasbeenconflictsensitive.Foralistoftoolsusedin creatingandevaluatingconflictsensitiveevaluations,policiesandprogrammes,seeAnnex5.
31 32
OECDDACandCDA,2007. Anderson,1999a. 33 InternationalAlert,2007a. 34 Forinstance,theOECDhasproducedguidelineswhichassertthatinternationalassistancemust,ataminimum,avoidnegative effectsonconflict,and,wherepossible,makeapositivecontributiontoconflictpreventionandpeacebuilding.DACGuidelineson HelpingPreventViolentConflict(2001and1998):www.oecd.org/dac/conflict/preventionguidelines. 35 Anderson,1999bandUvin,1999.
20
Box2.
Conflict(in)sensitiveevaluation
[EXAMPLENEEDED:Exampleofsensitiveorinsensitiveevaluation?IllustrationofproblematicHawthorneffects?Staff, translatorsorothersbeingthreatened/harmedafterparticipatinginevaluation?Other]
Genderawareness Fieldexperiencesandextensiveresearchshowthatthewaywomenandmenexperience,engagein andareaffectedbyviolentconflictdiffersaccordingtotheirgenderedidentities. 36 Conflictitselfcan oftenplayamajorroleinformingasocietysunderstandingofandresponsestogenderroles(whatit meanstobeamanorwomanandwhatisexpectedofandtoleratedfromeach),andviceversa. Additionally,violentconflictisnearlyalwaysaccompaniedbyasurgeinviolencetowardswomen. 37 Specificworkhasdevelopedintheareaofgender,peaceandsecurityinrecognitionofthis. 38 Further resourcesongenderandconflictcanbefoundinAnnex2. Aclearandcriticalunderstandingofgenderwithintheparticularconflictcontextistherefore extremelyimportantforbothevaluatorsandprogrammers.Overlysimplisticviewsofmenandwomenin conflictareoftenbasedonmisleadingstereotypes(manasaggressor,womanasvictim)whichare neitheraccuratenorusefulandcanleadtopoorlydesignedormistargetedinterventionsthatdonot contributetoreducingviolencenorconsolidatingpeace,andthatcanevenhaveharmfuleffects. Thoseplanninganevaluationwillneedtodeterminehowthisunderstandingwillbetakeninto accountwhenconductingtheevaluation,andwhethergenderoughttobeincludedasanevaluation themeorfocus.Theevaluationteamshouldbothtakethisissueintoaccountintheirownworkand consideritduringtheirevaluationoftheactivityinquestion.
36 37
See,amongothers:InternationalAlert,2001;USAID,2007a;UnitedNationsDevelopmentProgramme,2002;Sida,2003. USAID,2007a,p.11. 38 SeeforexampletheInternationalAlertwebsite:www.conflictsensitivity.org.
21
Box3.
Confrontingpsychosocialtrauma
Psychosocialtraumamayaffectmuchlargernumbersofpeoplethanisoftenevidenttoanoutsider,particularly oneunfamiliarwiththelocallanguageanduntrainedinthediagnosisofposttraumaticstressdisorders.People beinginterviewedduringorafteramajorviolentconflictmayhaveexperiencedviolencefirsthand;theymay havebeenforciblydisplaced,hadrelativesandfriendskilled,orperhapsseentheirpersonal,socialorcultural identitiesshattered.Chronicinsecurityandwidespreadgenderbasedviolence,includingthesystematicuseof rapeandotherformsoftorture,compoundtrauma.Widespreadtraumawillnodoubtimpactinteractions betweenlocalpeopleandevaluatorteamsandshouldbehandledwithgreatcare.Thevalueanduseof informationcollectedfromlocalswillhavetobeweighedagainstthepotentiallyharmfuleffectsofexplaining traumaticexperiencestoevaluators.Exposedtosuchextremeexperiences,andperhapshavingthemselves experiencedorwitnessedviolence,itisnotunheardofformembersofevaluationteamstoalsoexperiencemild formsoftraumaticstressdisorders.Thepracticeofofferingcounsellingorothersupporttoreturningevaluators shouldnotberuledout.
Source:AdaptedfromtheDACGuidanceonEvaluatingHumanitarianAidinComplexEmergencies(OECD,1999).
Protectionandethicalresponsibilities Evaluationmanagersneedtobeawareofthefactthatconductingevaluationsinconflictzonesmay putevaluationteamsandstakeholdersatrisk.Protectionconcernshavetobeincludedinevaluation design,budgetingandmanagementespeciallywhenwidespreadviolenceisimminentorongoing. Evaluatorsorteamsshouldbecloselylinkedtoincountryofficesandthesecuritymanagementsystem. Inaddition,evaluatorsshouldkeepinmindthatthewaytheyact,includingboththeexplicitandimplicit messagestheytransmit,mayaffectthedegreeofrisk.Inthiscontextitisespeciallyimportanttoconsider thesafetyofinterpretersandotherlocalstaff,partnersandbeneficiaries,whomevaluatorsmay inadvertentlyexposetogreaterrisksthantheythemselvesface.Ethicalissuesthatmayariseduringthe evaluation,particularlyastheyrelatetotheapproachchosen,shouldbemadeclearthroughtheconflict analysis,beaddressedattheoutsetoftheprocessandincludedinthetermsofreference.Thisispartof doingaconflictsensitiveevaluation. Otherconsiderations Fundamentalprinciplesofestablishedevaluationpracticeshouldbeapplied(seeforexamplethe DACEvaluationQualityStandards).Independenceofanevaluationteam,participation,transparencyand inclusivenessareparticularlyimportantevaluationprincipleswhenworkinginthisfield.Local stakeholdersshouldbeinvolvedinplanningandconductingtheevaluationasmuchaspossible,inorder toensureatransparentprocessthatwillbevaluedandacceptedbythosedirectlyaffected.However, particularcaremustbetakenwhendecidingwhomtoinvolveandhowinthecontextofviolentconflict. Annex7discussesusingparticipatoryapproachesinconflictzones.Transparency,abasicevaluation principle,canbeakeydimensionofconflictsensitivework:atransparentprocessinbothprogramming andevaluatingcanhelpreducesuspicionandtension,thoughthedegreeofopennessmaydependupon securityconcerns.
22
3.Outlineofthemainelements
Buildingontheprinciplesoutlinedabove,thissectionbeginswithkeyupstreammeasuresor preconditionsforevaluationforpolicymakers,programmemanagersandplanners(3.1).These suggestionswillhelpstaffandpolicymakerspreparefor,supportandlearnfromevaluation.Next,three sections(3.23.4)overviewplanning,conductingandreportingevaluations.Thesesectionswillbeofuse tothosecommissioning,planning,orconductingevaluationsaswellasthosepolicyandprogrammestaff preparingtobeinvolvedinorlearnfromanevaluation.Thesesectionscoversomekeystepstoconsider whenevaluatingconflictpreventionandpeacebuildingactivities,includingdimensionsofestablished evaluationpracticeasadaptedtothespecificchallengesinthisfield. Pleasenotethattheelementsbelowarenotnecessarilysequentialandmaybeordereddifferently, ormaybemoreorlessimportant,dependingontheindividualevaluationpurposeandcontext.Theroles ofindividualsinthevariousstepsarespecifiedwhererelevant.
23
3.1Preconditions:Programmingtoimproveworkandstrengthenevaluation
Summary:Keystepsforpolicymakers, programmeplannersandmanagers
Establishrelevant,clearandmeasureableobjectives Stateanexplicittheoryofchangeandprogrammelogic Completeandmonitoraconflictanalysis Developandmonitorrelevantindicators Focusonstrategyandpolicycoherence Conductsystematic,rigorousevaluation
Thisisnotaguideonhowtoplan,fund,manageorimplementpoliciesorprojectsthattryto preventconflictorbuildpeace.However,asoutlinedabove,thereisageneralneedforfurther developmentineachoftheseareas.Betterinterventionstrategiesandpolicycoherenceareneededto makeprogresstowardspeace.Donorscannotrelysolelyonaidbutmustlookatotherpolicyinstruments andtheirimpactsonconflictandthechancesforpeace.Strategicengagementatvariouslevelsand acrossandbetweengovernmentsisessential. 39 Systematic,independentandrigorousevaluationshould beleveragedtohelpimprovepracticesandpoliciesinthisfieldaswell.Consistent,highquality evaluationatboththeproject/programmelevelandthestrategicpolicylevelwillcontributetoimproving effectiveness. Inturn,programmemanagersandpolicymakerscanbettersupportandlearnfromevaluationsby helping to systematically establish certain preconditions for evaluation. Ideally, a number of conditions should be in place before an evaluation process begins. The most essential elements are: baselines (including a conflict analysis) and future scenarios; clear and measurable objectives; a testable programme logic and theory of change; and monitoring tools, including performance information and indicators (in order to measure achievements on the way). Conflict prevention and peacebuilding activitiesoftenlacksomeorallofthesepreconditionsforavarietyofreasons,especiallywhentheyare performedduringandafteropenarmedconflict,(oftenduetotheoftenlimitedtimeforplanning).Thisis similartoevaluationsofcomplexemergencies.40 Tohelpaddresstheabovementionedissues,someprogrammeplanning,monitoringand managementelementsarecoveredinAnnex3.Programmeplanners,policymakers,implementingstaff andmanagers,andevaluatorscanworktogethertostrategiseabouthowbesttoconfronttheseissues before,duringandaftertheevaluationprocess.
39 40
Formoreonpolicycoherenceandinternationaldevelopmentassistance,visit:www.oecd.org/development/policycoherence. SeeDACGuidanceforEvaluatingHumanitarianAssistanceinComplexEmergencies: http://www.oecd.org/dataoecd/9/50/2667294.pdf.
24
Box4.
Integratinganalysis,evaluationandredesign
DFIDusedafullstrategicconflictanalysisstudyasthebasisforaconflictpreventionandpeacebuilding evaluationofitsentireprogrammeinNepal.ThisConflictsensitiveProgrammeReviewthenfedintoa revisedcountrystrategy.Amongthechangesthatcameaboutwasagreateremphasisontransparency, becauseithadbeenshownthatthiscouldreducetensionslocallyandpreventMaoistinterference.The reviewalsohighlightedtheneedforanactiveequalopportunitiespolicytoensurethatallsocialgroups wererepresentedamongDFIDstaff.Conflictanalysis,strategyandevaluationwereintegrated.
Source:VauxforOECDandCDA,2007.
25
3.2.Planningandpreparingtheevaluation
Summary:Keyelementsofplanninganevaluation forthosecommissioningorpreparingevaluations
Definethepurposeanduseoftheevaluation Decidethescopeoftheevaluation Outlinekeyevaluationquestions Doorobtainaconflictanalysis Taketimingandlogisticalissuesintoconsideration Coordinatewithotheractors Considerconductingajointevaluation Selectevaluationcriteria Deviseevaluationmanagement DevelopTermsofReference Selecttheevaluationteam Contracting 3.2.1Definethepurposeanduseoftheevaluation Beginbyasking:Whatisthisevaluationmeanttoascertain?Definingthepurposeandobjectivesof anevaluationisthemostimportantplanningstep.Ifthepurposeisnotclear,theevaluationwillnotbe clear.Anevaluationcanhaveanumberofdifferentpurposes(sometimessimultaneously),suchas: Learningandimprovement:Systematiseknowledgeofresultsandperformance,whichcanhelp improvethisorsimilaractivities.Evaluationscarriedoutwhileapolicyisstillbeingappliedora programmeimplementedareusefulforimprovement.(Thistypeofpurposeismostrelevantto immediatestakeholdersandthoseplanningorimplementingsimilarprojects.) Accountability,controlordocumentation:Findoutwhetheranactivityhasbeenperformedas intendedand/orwhethertheexpectedresultshavebeenachieved.(Accountabilityoriented evaluationsaremostlyusefulfordonorsandthewiderpublic.) Determiningtheusesoftheevaluationiscloselylinkedtodecidingitspurpose.Whoistoreceivethe findings,whoistheaudienceandwhatwilltheydowiththeresults?Willtheevaluationbeusedfor programmeorpolicyredesign?Renewedfundingdecisions?Forthedesignoffuturesimilaractivities?To learnaboutprocesses?Totesttheoriesofchange? Theusefulnessoftheevaluationresultsisanimportantprincipleforallevaluationsespeciallyin thisfield,where,asmentionedpreviously,thereissomeresistancetoand/orunfamiliaritywithrigorous evaluation.Evaluationswillbeinhigherdemandifthestakeholdersinvolvedfindtheresultsof evaluationsusefulfortheirownwork.
26
Box5.
Whatareevaluationsusedfor?
Thefollowingexamples,fromdonoragencyevaluationsofconflictpreventionandpeacebuildingwork, highlightdifferentwaysevaluationcanbeused:
Accountabilityanddocumentation:Theaimofthereviewwastoassesswhethersupporthasledto increasedstabilityinthePalestinianterritories. Controlandlearning:"ThisreportwaspreparedtoascertainwhetherAsianDevelopmentBankpolicy conditionshadbeenmetandwhethertheyledtoachievementoftheTajikistanPostConflict InfrastructureProgramsstatedobjectivesorpurpose." Learning:Thisreviewwascommissionedinordertoprovideabetterunderstandingofthequalityand effectivenessofUKassistancetoSecurityandJusticeSectorReforminAfrica. Improvement:Sidahascommissionedanassessmentoflessonslearnedfromsupporttoconflict managementandpeacebuildingtoserveasaninputfordevisingnewstrategies

Source:ReportscanbefoundontheDevelopmentEvaluationResourceCentre(DEReC)website: www.oecd.org/dac/evaluationnetwork/derec.
3.2.2Decidethescopeoftheevaluation Thescopeshouldbeclearlydefinedbyspecifyingtheissuescovered,fundsspent,thetimeperiod, typesofinterventions,geographicalcoverageandtargetgroupsaswellasotherelementsofthepolicyor interventiontobeaddressedintheevaluation.Itisimportanttodelimittheevaluationscopeaccording topurpose,resourcesandtime.Alsostatetheevaluationquestionsforwhichanswerswillnotbesought. Questionstobeaskedare:Howfaralongtheresultschain(inputs outputs outcomes impacts) willtherevieworevaluationgo?(Notethatinthefieldsofconflictpreventionandpeacebuildingthelinks tothebigpicturecanbeespeciallyimportantseeSection3.3.5.)Willitlookforimmediateandlong termimpactsoninstitutions,onsocietyoronbroadconflictdynamics,and,ifso,how?
Table1.
Hierarchyofevaluationscopes
Typeofevaluation Systemwide Definition
Evaluationoftheresponsebythewhole systemtoaparticulararmedconflictor outbreakofviolence Evaluationofapartofthesystem(such asathematicorsectorstudy)
Example
JointEvaluationofEmergencyAssistance toRwanda(1996)
Partialsystem
Evaluationofcooperationinterventions andFrenchNGO'sincrisisandfragile institutionalcontexts(ongoing2007)
27
Singleagencyresponse
Evaluationoftheoverallresponsetoa particulararmedconflict(orseriesof violentevents)byaparticularfunding, channelling,orimplementingagency Evaluationofasingleproject, programmeorpolicyundertakenbya singleagency
Peaceandconflictimpactassessmentof theSwissAngolaProgramme(2002)
Singleagency, singleproject
TheMidtermReviewofthePalestinian FinnishEducationProgramme(ongoing 2007)
Source:AdaptedfromOECD,1999.ExampletitlesdrawnfromDEReC.
3.2.3Doorobtainaconflictanalysis Bydefinition,oneoftheprimarychallengestoevaluatingconflictpreventionandpeacebuilding workisthatboththeinterventioninquestionandtheevaluationitselftakeplacewithinaconflict context.Whethertheevaluation(aswellastheactivityitisassessing)istakingplaceduringthetense periodsbeforeanoutbreakofviolence(wherepreventionisthepriority),duringopenconflictor immediatelyfollowing,orinareasthathaveexperiencedmajorviolenceinthepastandareenteringa periodofrecoveryandlongtermpreventiontheconflictcontextwillhavemajorimplications.Insome cases,aninsecureenvironmentwillhaveprofoundimplicationsfortheachievementsandmodalitiesof anactivityorevaluation. Oneofthebestwaystofacethechallengeofworkinginthiscontextisbyusingsomeformof conflictanalysis.Aconflictanalysisidentifiesthekeyfactorsrelatingtoconflictandthelinkagesbetween them,pointingtosourcesanddynamicsofconflictaswellaspeace.Preferablythisincludesabaseline analysisperformedduringtheplanningstageoftheintervention,aswellasupdatesandconflict monitoring(overtime).Forcomparison,acurrentorupdatedanalysisatthetimeoftheevaluationis alsoneeded.Athoroughanduptodateunderstandingoftheconflictisthefirststepforaconflict sensitiveevaluationprocess.Also,inordertoassessrelevance,theevaluationteamwillneedtoexamine thetargetsdesignandimpactsinrelationtoaconflictanalysis. Evaluatorswillalwaysneedtohavesomesortofconflictanalysis,thoughtheymaynotnecessarily needtoperformonethemselves.Forinstance,theevaluationcouldbebasedonanalysisprovidedbya donoragency,theevaluationtargetitself,athirdpartyoraparticipatoryprocesswithstakeholders;or,it couldbeanassessmentcommissionedspecificallyfortheevaluation.Anactivitymayhaveincludeda conflictorriskanalysisinitsplanningprocesses.Ifsuchabaselineexists,evaluatorswillneedtoconsider whetherthiswasaccurate,whetheritwastranslatedintorelevantstrategiesandobjectives,whetherit wasadaptedtotheconflictovertime,andanyfurtherconstraintsthathavebeencreatedbytheconflict situation.Wastheanalysiskeptuptodateanddidtheprogrammeadaptappropriately? Asimpleandpracticalwayofdevelopingananalysisistoconductaworkshopwithkeystakeholders. Asitisnotalwayspossibletogetallcompetingperspectivesfromthedifferentpartiesatthesametime, itmaybenecessarytointerviewotherpeople(includingrepresentativesnotnecessarilydirectlyinvolved intheconflict)separatelytogainawiderunderstandingoftheconflict.Bewarethatitwilllikelybe difficulttogainconsensusonthenatureoftheconflictcontendinggroupswillnotagree. Therearemanydifferentmodelsandframeworksforconflictanalysisusedbydevelopmentco operationagenciesandothersengagedinworkinginandonconflict.Thosecommissioningevaluations needtogiveconflictanalysiscarefulthoughtandmakeexplicitthebasisforanalysisintheirTermsof
28
Reference.Theanalysismethodselectedshouldbewelladaptedtothecontext,theevaluationscopeand availableresources.Itfollowsthatthosefundingevaluationsshouldensurethatresourcesfortheconflict analysisprovidedareproportionaltothetaskenvisaged.KeyquestionstoincludearelistedinTable2 andabriefsummaryofdifferentconflictanalysismodelsortechniqueshasbeenoutlinedinAnnex4.

Table2.
Keyquestionsforconflictandpeaceanalysis
Whatisthepolitical,economic,andsocioculturalcontext? Whatareemergentpolitical,economicandsocialissues? Whatconflictprone/affectedareascanbesituatedwithinthecontext?Whatarethe geographicdimensions? Isthereahistoryofconflict? Whatarethestructuralcausesofconflict? Whatissuescanbeconsideredasproximateordynamiccausesofconflict? Whattriggerscouldcontributetotheoutbreak/furtherescalationofconflict? Whatstrategiesfordealingwithconflictcontributetoviolence? Whatnewfactorscontributetoprolongingconflictdynamics? Whatfactorscancontributetopeace?Whatfactorsarebringingpeoplestogether? Whoarethemainactors(peoplewhoperpetuateormitigatetheconflict)? Whataretheirinterests,goals,positions,capacitiesandrelationships? Whatcapacitiesforpeacecanbeidentified?Whocanmakeadifference? Whatactorscanbeidentifiedasspoilers(thosewhobenefitfromongoingviolenceorwho resistmovementtowardspeace)?Why?Aretheyinadvertentorintentionalspoilers? Whatarecurrentconflicttrends?Negativereinforcingcycles? Whatarewindowsofopportunity? Whatscenarioscanbedevelopedfromtheanalysisoftheconflictprofile,causesandactors? Howmightdifferentscenariosplayoutgivenlikelyfuturedevelopments(shortandlongrun)?
Profile
Conflictcauses andpotentials forpeace
Actors
Dynamicsand futuretrends
Source:AdaptedfromInternationalAlert,2007aandPaffenholzandReychler,2007.
3.2.4Outlinekeyevaluationquestions Theevaluationmanagementshoulddevelopdetailedquestions(linesofinquiry)thatwillbe answered.Thesewilllargelybedeterminedbythetypeofintervention,thestageofimplementationand whattheevaluationhopestoachieve.Forevaluationsofworkinandonconflict,focusingonpeace relatedqueries(asoutlinedhere)setsastandardofhighperformanceintermsofcontributiontopeace beyondthecriticallyimportantstandardofconflictsensitivity.Somepossibleevaluationfocifordifferent typesofinterventionsfollow. Explicitpeacebuildingefforts,whichhaveincorporatedspecificgoalsthatdeliberatelyseektoexert apositiveeffectonconflictinacontextwherethereisongoingorrecentlyhaltedviolence,couldbe
29
basedontheprimaryinquiry:Isthiseffortmakingarelevantcontributiontodurablepeace,by deliberatelyandeffectivelyaddressingkeydrivingfactors 41 ofconflictamongcrucialconflict actors/constituencies? Explicitconflictpreventioneffortsthathaveadoptedconflictrelatedgoalsandobjectivesina contextinwhichthereareindications,throughearlywarningsystemsorothermechanismsofalert,that violenceislikelyintheshortorlongtermmightfocusonthequestion:Isthiseffortmakinga demonstrablecontributiontopreventingviolence,eitherbyinterveningswiftlytoavertescalating violenceorbyaddressinglongtermstructuraldriversofconflict? Developmentinterventionsinconflictpronecontextsareusuallyprimarilyaimedatdevelopment objectivessuchashealth,education,andinfrastructure,butmayalsohavethepotentialformakinga positivecontributiontopeace.Whenevaluatingsuchaneffort'sactualcontributiontopeace(beyond assessingconflictsensitivity)evaluatorsmightlookat:hasanassessmentofpeaceconflictdynamics beenundertaken(andmaintained)andhasthisinfluenceddevelopmentprogrammingchoices?Does thiseffort(orcouldit)engageonkeysocialtensionsthathavebeenidentifiedasdrivingfactorsofpast, currentorpotentialconflict? Moreinformationandfieldtestsareneededtodeterminewhetherornotitwouldbeusefulto evaluatehumanitarianinterventionsinthisway.Anevaluationofhumanitarianworkwouldlikelyfocus onconflictsensitivity,ratherthanachievementofspecificpeacebuildingobjectivesperse. 42 Key questionsmightinclude:Doestheinterventionavoidcreatingtensionswithinthecrisisaffected community;betweendisplacedpeopleandhostcommunities;betweenagenciesoverthetypeand quantityofassistance,etc.FormoreonevaluatinghumanitarianassistanceseeOECDDAC(1999), "GuidanceforEvaluatingHumanitarianAssistanceinComplexEmergencies". 3.2.5Taketimingandlogisticalissuesintoconsideration Thereareusuallystandardtimeframesforconductingevaluations.Schedulesandevaluationplans areoftendecidedwellinadvance.However,thetimingforevaluatingconflictpreventionand peacebuildinginterventionsshouldbedeterminednotonlybythephaseofthepolicy,programmeor projectcycle,butalsoinrelationtocurrentconflictrealitiesthisispartoftheethicalresponsibilitiesof thoseplanningandconductingevaluations.Timingoftheevaluationshouldbeappropriateforcurrent dynamicsoftheconflictitself,andusefulforinformingpolicydiscussionand/orprogrammeadjustments (accordingtoobjectives).Thosecommissioningevaluationsmayhavetoadjusttheirexpectationsgiven conflictrelatedrestraints.TheTermsofReference(TOR)shouldbeclearaboutrealistictimeframes.To identifytherighttimeandgoodentrypointsforanevaluation,considerthefollowingquestions:
41
Keydrivingforcesarethemostimportantforcesdrivingtheevolutionoftheconflict.Theyaretheelementswithoutwhichthe conflictwouldbesignificantlydifferent. 42 Humanitarianinterventionsinconflictsituationsusuallydonotfocusonpeacebuildingasacoreobjective(andoftencannot duetotheoverridingconcerntomaintaintheneutralityandimpartialityofhumanitarianaction).Theymay,undercertain circumstances,stillmakeacontributiontopeace,forexamplebycreatingneutralspaceinwhichdialoguecanoccur. Humanitarianinterventionsmust,however,alwaysbesensitivetotheprevailingpeaceconflictdynamicsandseektoensurethat theydonoharm.Forexample,externalshocks,whetherhumanmadeornatural,arelikelytohaveupsetsocialstructuresand heightenedtensionsoveraccesstoscarceresources(food,shelter,medicalsuppliesetc).Insuchsituations,humanitariansmust becarefultoavoidexacerbatingunderlyingsourcesoftensionorcreatingnewsourcesofconflictwithincommunities.Therefore, whenevaluatinghumanitarianaction,evaluatorsmight,inadditiontoassessingconflictsensitivity,alsoexaminethepeaceand conflictimpactsofinterventions.
30
Whatishappeningintheconflict?Atwhatstageistheconflictcycle?Watchoutespeciallyfor potentialconflicttriggers(elections,controversialcelebrations,etc.). Wouldanevaluationatthismomentbedisruptivetothepolicy,projectorprogrammeitself? Wouldanevaluationprovokepoliticalreactionthatcouldunderminetheintervention,bycalling attentiontotheinterventionorbyinadvertentlyfeedingpoliticalforcesinoppositiontoapeace process? Wouldanevaluationputinterventionstakeholdersatpersonalorpoliticalrisk? Hastheeffortbeeninplacelongenoughtoprovideusefulexperienceandlearning?Isthe assessmentofoutputs,outcomesandimpactsbasedonarealistictimescale? Howlonghasitbeensinceanypreviousevaluationorreviewwasperformed? Arethereanylogisticalissuesthatmustbetakenintoconsideration(securityrestrictions, weatherpatterns,majornationalholidays,accesstotransport,etc.)?
3.2.6Coordinatewithotheractors Toworktowardsacoordinated,wholeofgovernmentapproach,thosecommissioningan evaluation,particularlydonorgovernments,shouldconsidertherolesofandrelationshipswithother actorsinthefield.Strategiseaboutinvolvinghumanitarian,development,military,securityandotherkey actorsinadditiontothoseimmediatelyinvolvedwiththeprogrammeorpolicyinquestion.Thisis importantforpromotingharmonisationandconsistencyofdonorwork.Theevaluationplannerswillalso needtodeterminewhetherornotparticipationfromthepartnercountryisappropriateandusefulinthis conflictcontext. Increasingly,developmentagencies,humanitarianorganisationsandsecurityforcesareworking togetherinconflictpreventionandpeacebuildingenvironments.Thecurrentemphasisamongmany donorsandinternationalorganisationsonwholeofgovernmentapproachescanoftenmeanthata greatvarietyofactorsfromdiversebackgroundswillbeinvolvedinanyoneevaluation.Thesearchfor greatercoordinationraisesdelicateissuesintherelationshipamongdiplomacy,defence,development andhumanitarianestablishments. Conflictpreventionandpeacebuildingactors(e.g.development,diplomacyanddefence professionals,humanitarianworkers,etc.)aremotivatedbydifferentpolicyincentives,speakdifferent languages,donotoperateunderthesameorganisationalumbrellaandmustworktogetherwhile respectingoneanother'smandates.Theyoftenworkbasedondifferentorevencontradictorytheoriesof change(see3.3.1andAnnex6).Ifhumanitarianordevelopmentagentsrefusetobecomeinvolvedin situationswheredefencemayhavethelead,thoseagenciesforgotheopportunitytorelievecivilians fromthecostsofwar.Yetjoiningthewholeofgovernmenteffortinthemidstofaconflictmayrisk abrogatingtheprinciplesofimpartialityandneutralityonwhichtheircredibilitydepends.Somehave claimedthatmaintaininganacceptabledegreeofindependencefortheevaluationinhighprofilearmed
31
conflictsettingsismoredifficultthanusual,especiallywhenthereisrelianceonthemilitaryfor security. 43 Coordinatingsuchadiversegroupwillrequirespecialconsiderationofevaluationteamsize,pre departurecoordination,andaddressingdifferencesininterventionmethodsandtheoriesofchange. Evaluatorsshouldbeskilledandknowledgeableaboutthesevariousactorsandtheirinterplayinpre, postandopenviolentconflicts.Workingtogetherforevaluation,ifhandledcarefully,canproduce positiveresultsandincreaselearning.Still,evaluatorsmustbecriticalandrigorousinexaminingtheroles andeffectivenessofallactorsinvolvedbothintheirownworkandthatoftheevaluationtarget.

Box6.
WorkingtogetherinMauritania
In2006aninteragencyteamcomprisedofmembersfromtheUnitedStatesStateDepartment,USAIDsAfrica Bureau,USAIDsOfficeofConflictManagementandMitigation,theDepartmentofDefencesEuropeanCommand andSpecialOperationsCommandEuropeparticipatedtogetherinaTransSaharaCounterterrorismPartnership AssessmentinMauritania.Lessonslearnedoninteragencycoordinationforsuchjointexercisesincludedtheneed for:standardisedpredeploymentorientationandbriefingsforpersonnelengagedinthefield;harmonised reportingdocuments;aninteragencypilotactivity;andappropriatestaffinglevelstoimplementamultiagency initiative.
Source:USAID,2007b.
3.2.7Considerconductingajointevaluation Jointevaluations(acrossseveraldonorsandprogrammesorinvolvingthepartnercountry)shouldbe promotedbecausetheycancontributetomoreharmonisedapproachesandcangenerateadditional learningabouthowavarietyofactivitiesaddupinaparticularconflict.Thiscanhelpaddress(thoughit doesnotentirelyeliminate)theattributionchallenges,andprovidesinsightintotheemergingroleof crossdonorandcrossministry/departmentcoordination.Asinthedevelopmentandhumanitarian fields,jointapproachestoevaluationsofconflictpreventionandpeacebuildingareevolvingandshould befurtherencouragedanddevelopedinstepwithjointassistancestrategiesandothernewaid instruments. Somejointorcumulativeevaluationslookmorebroadlyatmanyorevenallinterventionsina particular conflict zone to assess the combined impacts from those multiple efforts. Steps should be takentomanagethepoliticsofmultipleactorsthatcanconfuseandweakenevaluation,andtoensure thatthelearningissufficienttojustifytheextracostsofcoordination.Finally,whenjointevaluationsare performed,itiscriticaltoincludelocalpeoplesothattheexternalagenda(i.e.donoragenda)doesnot distortinternalconcerns(includingevaluationagendas).Anumberofpracticalsuggestionsarecontained in the DAC Evaluation Networks Guidance on Managing Joint Evaluations and Effective Practices in ConductingaJointMultidonorEvaluation.44
43
44
USAID,2006. OECDDAC,2006bandOECDDAC,2000.
32
Box7.
Jointmultidonorevaluation:Learningtogetheraboutpeace
InSeptember1999,theJapaneseMinistryofForeignAffairs,JapaneseInternationalCooperationAgencyand theCanadianInternationalDevelopmentAgencyheldtheCanadaJapanSymposiumonPeacebuildingfor DevelopmentinTokyo,withthecooperationofNGOsandresearchinstitutes.Theyagreedtoajointreview ofpeacebuildingprojectsbypublicandprivatesectorsinGuatemalaandCambodia.TheCanadaJapanJoint PeacebuildingLearningProjectbringsanewanduniquedimensiontoevaluation,onethatprovidesgreater understandingofwhatisbeinglearnedfromtheeffortsofJapaneseandCanadianNGOsandstrengthens theirmutualcapacitiestocontributetopeacebuildingactivities.
Source:Jones,George,andStanley,2001.
3.2.8Selectevaluationcriteria Determinethemostimportantcriteriaformeetingtheevaluationsobjectives.Whenplanningand conductinganevaluationandpreparingtheTermsofReference,considerationshouldbegiventothefive OECDDACCriteriaforEvaluatingDevelopmentAssistance,namelyrelevance,effectiveness,efficiency, sustainabilityandimpact. 45 Inthecourseoftheparticipatoryprocessthatledtothedevelopmentofthis guidance,itwassuggestedthattheoriginalfiveshouldbesupplementedbyadditionalcriteria(suchas theonesusedintheDACGuidanceonHumanitarianAssistanceinComplexEmergencies),including: coherence,coverage,linkagesandvalues(meansandends).Itishopedthattheapplicationphasewill resultinfeedbackontheusefulnessoftheseadditionalcriteria.Theuseofeachcriteriainevaluation (includingkeyquestionsforeach)isdescribedfurtherinSection3.3.4. 3.2.9Deviseevaluationmanagement Determineproceduresandresponsibilitiesformanagementoftheevaluation,i.e.whoisresponsible forwhat(headquarters,fieldoffice,evaluators,partners,etc.).Stateclearlythelevelofindependenceof theteam,andstatewhohastheleadcoordinationfunction.Shouldtherebeareferencegroup? 46 Ifitis ajointevaluation,shouldtherebeasteeringgrouporamanagementgroup,orboth?Howwillrelations withthepartnergovernment(s)bemanaged,andbywhom?Timeplans,deadlinesandfundingshouldbe clearandrealisticandbesufficientlyflexibletoadapttoarapidlychangingcontext. 3.2.10DevelopTermsofReference Theparticularitiesofevaluatinginthisfield,aswellasthevariousplanningstepsdetailedhere, shouldbereflectedintheTermsofReference(TOR).TORsoutlinewhatisexpectedoftheevaluatoror evaluationteam.Thisdocumentwillhelpguidetheevaluationprocessandisagreedtobythose commissioning,planning,managingandconductingtheevaluation.TORsshouldspecifytowhomexactly reportswillbesentandwhetherornotafinalreportwillbepublished.Annex8hasasampleTOR.
45 46
ForalistoftheDACCriteriasee:www.oecd.org/dac/evaluationnetwork
Areferencegroupisgenerallyanadvisorycommitteethathelpsprovideguidanceandservesasanintermediarybetween managementandtheevaluators.Itmayalsoprovideindependentoversightoftheevaluation.Thegroupismadeupofavariety ofstakeholdersandexperts.
33
3.2.11Selecttheevaluationteam Anevaluationteammadeupofmemberswithcomplementaryskillsandviewsisrecommended. Peoplewhoarethemselvesknowledgeableaboutconflictandpeacearecriticaltothequalityofconflict preventionandpeacebuildingevaluations.However,itisequallyimportanttohaveknowledgeable evaluationexpertsintheteam.Amixofinsidersandoutsidersfromtheconflictandtheintervention contextsandgoodgenderbalancearealsotobeconsidered.Particularattentionneedstobegiventothe perceptionofbiasoftheteam.Box8outlineskeyquestionstoguidechoosingtheevaluationteam.
Box8.
Composinganevaluationteam:Questionstoconsider

Giventhepurposesoftheevaluationandthemainlinesofinquiry,whataretheneededattributes(skills, experience,relationships)ofanevaluatororevaluationteam? Inparticular,doestheteamhavedemonstratedknowledgeoftheconflictpreventionandpeacebuilding fieldandpractice;abilitytoperformconflictanalysis;knowledgeofevaluationapproachesandmethods; experienceinthecountry/regioninvolved;relevantlanguageskills? Whatistheworkingstyleofprospectiveevaluators?Doeshe/she(dothey): a)demonstrateskillsandcomfortworkinginpotentiallydangerousandpoliticallysensitivesituationsina calm,nonthreateningmanner;b)employinterpersonalapproachesthataretransparent,trustingand evokingtrust;andc)exhibitskillsformanagingconflictsandtension? Howwilltheteambeviewedbyconflictactors?Mightcertainindividualcharacteristicsbasedon (perceived)religion,skincolour,gender,nationalityandlanguage,forinstanceexposetheteamto additionalrisksoraccusationsofbiasinthisparticularconflictcontext?
3.2.12Contracting Evaluationcontractsneedtofollowtherulesandguidelinesofthecommissioningagency.Oncethe outlineplanoftheevaluationisclear,andtheTORcomplete,theagencyordepartmentwilleither conducttheevaluationinhouseorcontractouttheevaluation.Thismaydependonthesizeand complexityofthecontractaswellastheavailableexpertiseandstaffresources.Whentheevaluationis contractedout,proposalswillbesubmittedbyconsultants,researchgroupsorresearchprofessionals. Themanagerswillthenselectfromvarioussubmittedbids.
34
3.3Conductingtheevaluation Summary:Keyprocesselementsforevaluators
Identifytheimplementationlogicandtheoryofchange Dealwithmissingbaselinesandothergaps Gatherdata Examinetheeffortusingvariouscriteria Lookatthebigpicture
3.3.1Identifytheimplementationlogicandtheoryofchange Implementationlogicorprogrammelogicaretermsusedtodescribewhyanactivityorpolicyis doingwhatitisdoing.Atheoryofchangeisatermcloselyrelatedtoimplementationlogicthatisoften usedinthisfieldtodescribethelinksbetweeninputs,theimplementationstrategyandtheintended outputsandoutcomes.Itdescribestheassumedorhopedcausalrelationshipbetweentheactivityor policyandits(intended)effectsonlargerpeacemakinggoals.Atheoryofchangecanalsobedescribedas asetofbeliefsabouthowandwhyaninitiativewillworktochangetheconflict. 47 Suchtheoriescantake thesimpleformat:WebelievethatbydoingX(action)successfully,wewillproduceY(movement towardspeace),orbemuchmorecomplex.Accurateandclearlystatedtheoriesofchangearenecessary foreffectiveprogrammingandshouldthereforebethesubjectofevaluation.SeeAnnex6foradetailed outlineoftheoriesofchangeandhowtheycanbeevaluated. Animportantrelatedtermusedinvariousfieldsisprogrammetheory:anexplicittheoryormodelof howaprogrammeaimstoproducetheintendedoutputs,outcomesandimpacts.Aprogrammetheory oftencombinesatheoryofchangeandanimplementationmodel. 48 Wellfoundedtheoriesofchange areattheheartofeffectiveworkinallfields. Inthepeacebuildingandconflictpreventionfielditisespeciallyimportantforevaluatorstoidentify andassesstheoriesofchange,becausethesetheoriesaretoooftenimplicit,unexaminedanduntested. Forexample,inKosovotheinternationalcommunityoperatedforseveralyearsundertheassumption (theoryofchange)thatpeacecouldbeachievedbyimprovingrelationsbetweenthetwomainconflicting parties.Basedonthistheory,itfundedmanyprogrammespromotingdialogue,exchanges,youth interactions,womensgroups,andsoforthallaimedatcrosscommunalrelationshipbuilding.However, astudyfoundthatworkwithineachseparatecommunitytocreatemoreresponsibleleadershiphada muchgreatereffectonpeacethanthebicommunalwork. 49 Someinterventionswillalreadyhaveaclearlystatedvisionofwhattheyhopetoachieve,asthiswill havebeendoneaspartofdesignandplanning.Otherswilleitherbelessexplicitordeliberatelyavoidany suchstatements(oftenforpoliticalordiplomaticreasons).Inmany(perhapsmost)cases,thesetheories
47 48
Weiss,1995;ChurchandRogers,2006,p.11. Animplementationmodeldescribeshowstaffandresourcesareusedtodeliverservices(BambergerandMabry,2006,p.191). 49 CDACollaborativeLearningProjects,incooperationwithCAREInternational,2006.
35
areunconsciousandunstated.Theyareembeddedintheskillsandapproachesthatconflictprevention andpeacebuildingpractitionersandpolicymakershavelearned,thecapacitiesoftheirorganisations, personalattachmentstofavouritemethodologies,andtheindividualperspectivesvariousdecision makersbringtothepeacebuildingprocess.Ideasaboutwhatwillcontributetopeaceatbothmicroand macrolevelsmayalsobeinfluencedbydonorobjectivesandinternationalpoliticaldynamics. Wherethetheoryisnotclearlystateditispossibleforanevaluatortoelicitordiscernthelogic behindtheactivityaspartoftheevaluationprocess,especiallyindiscussionswiththeimplementation team.Ineithercase,theevaluationwillunpackandmapouttheinputs,outputsanddesiredfuture outcomesandtheexpectedconnectionbetweentheseinordertoevaluatewhetherthestrategy beingusedislogicalandeffective.
Box9.
Makingtheoriesofchangeexplicitforevaluation
Anevaluationshouldexplicitlystatetheunderlyingassumptionsortheoryofchangeofthetargetpolicyor programme,asinthefollowingexcerptfromareportonNetherlandsassistanceforthePalestinianTerritories: Theevaluationtakesasapremisethatbalancedsocioeconomicdevelopmentandafunctioningcivilsociety inthePalestinianterritoriesisfavourabletothepeaceprocess. However,clearlystatingtheexpectedlinkbetweenprogrammeactions/strategiesandpeaceoutcomesisonlya firststep.Thecausalrelationshipsandassumptionsunderlyingthetheoryofchangeshouldalsobeexamined. Forinstance,inthisexample,examinerswouldwanttoverifywhetherornotandhowbalancedsocioeconomic developmentandcivilsocietycontributedtothepeaceprocessasassumed.
Source:MakkenandWijmenga,1999.
3.3.2Dealwithmissingbaselinesandothergaps Asmentionedintheintroduction,evaluatorsandevaluationmanagerswillsometimesencounter deficienciesinpolicies,strategiesandinterventions,suchas:unclearorunstatedobjectives,an unarticulatedtheoryofchangeorprogrammelogic,missingindicators,nomonitoringdata,orno baselineinformation.Todealwiththesegaps,considerwaysto(re)constructorcompensateformissing baselinesandotherdataduringtheevaluationprocess(bearinginmindthatthiscanneverfullyreplace solidplanning).Tipsonhowtocompensateorworkaroundsuchgapswithoutendangeringthequalityof theevaluationcanbefoundinanumberofotherpublications,e.g.theWorldBank'sguideonevaluating undertimeanddataconstraints. 50 3.3.3Gatherdata Akeyphaseofevaluatinginvolvescollectingdataandobtainingthestoryoftheprogrammeor policy,includingexplanationsbythepeopleinvolvedregardingwhyitunfoldedthewayitdid.Most
50
WorldBank,2005;Bambergeretal.,2004.FurtheradviceonqualitativedatameasuresisprovidedintheDACGuidancefor EvaluatingHumanitarianAssistanceinComplexEmergencies.
36
often,evaluatorsgatherinformationfroma)programmedocumentsandreports,b)monitoringdata(if available),andc)fieldvisitsand/orinterviewswithprogrammestaff,partnerorganisations,local/national officials,participants,andarangeofappropriatepublics.
Box10.
Quotebox:Datainpostconflictsituations
Postconflictsituationsteemwithalltypesofinformation:rumours,conjectures,halftruths,firsthand,second handandthirdhandinformation,misinformationandsometimes,too,therightinformation,attherighttimeto therightpeople.Interestsandagendasofallkinds,concernsoversurvivalandrecovery,dailypressuresandhopes forabetterfutureaffectthedisseminationofinformationinmanydifferentways.Addingtotheconfusion,few internationalstaffarelikelytospeakthelocallanguagesandradioandtelevisionservicesmaybepatchyornon existent.
Source:MinistryofForeignAffairsofJapanandUNDP,2002.
Duringandaftermajorviolenceandinsituationsofhightension,significantdataproblemsoften limitevaluatorswork(todifferentdegrees).Thissectionofferssolutionstosomeofthedataproblems. Others,however,donothaveclearsolutionsitisimportanttobeawareofthepotentialdifficulties theymaycauseandtoworkovertimeonstrategiesforimprovingdatareliability.Somekeydata challengesinconflictcontextsarelistedbelow: Accesstoreliabledataisscarceandmisinformationisrife. Lackofbaselineandmonitoringdata,duetoinconsistentengagement,destructionofrecords, orincompetent/nonfunctioningcollectionmechanisms. Thereareoftenpoliticalobstacleslimitingevaluatorsaccessacrossconflictinggroups.When thereisapoliticalmotivationforlimitinginformation,programmestaff,policymakersand/or localauthoritiesmayevenblockaccesstocertainregionsorgroups. Securityconcernsorphysicalbarriers(suchasroadblocksormilitarycheckpoints)maylimitdata collection. Oftenarapidturnoverofstaffandahigherthanusualproportionofinexperiencedstaff (especiallyinhotconflicts)compoundthelackofbaselineandmonitoringdata. Emphasisonrapidinterventionorworkingduringwindowsofopportunitycanhinder establishmentofbaselinesandcollectionofmonitoringdata.
Inaconflictzoneorjustafteraconflict,where/whenmistrustisrifeandmostintervention stakeholdersalsohaveastakeintheconflict,thereliabilityofdataandinformationprovided(inaddition
37
toitssimpleavailability)isoftenparticularlyproblematic. 51 Variousactorsmayhavediverseoreven contradictoryinterpretationsofaninterventionsimpacts(positive/negative)orrelevance,basedupon theirownpositionwithintheconflict. Toensurereliability,ifpossible,usemultiplesourcesortypesofinformationandsoundquantitative andqualitativedata.Thedataandinformationusedshouldbetriangulatedwherepossible.Asmuchas possible,informationsourcesshouldbetransparentandreliable.Verifythedatasvaliditybefore analysingthem.Thisisoftendonewithkeystakeholdersandinterviewobjects.Bycombiningmultiple datasourcesandvalidationapproaches,evaluatorsseektoovercomethebiasthatcomesfromsingle informantsorsingleobservers. Gatheringdatatounderstandimpactscanbecomequitecomplexinaconflictpreventionand peacebuildingevaluation.Avarietyofopinionsandperspectivesareatplay;eachmayhavesomevalidity forindividuals,butmaycontradictotherinterpretations.Toaddressthiscomplexity,methodsshouldbe rigorousaboutincludingthefullrangeofpointsofview,evenwhenthoseviewsconflictwithone another.Inparticular,evaluationsshould,wheneverpossible,includeperspectivesbothfromwithinand outsidethecapital,aswellastheperspectivesofthevarietyofgroupsandsubgroups(actorsandtheir constituencies)involvedintheconflict,eveniftheinterventionsworkonlywithoneside.Toavoid increasingtensionbetweengroups,decisionsabouthowtoinvolvevariousgroupsshouldbebasedona clearunderstandingofstakeholderrolesandinterests(stakeholderanalysis).Somesampling methodologies(randomsamplingforinstance)mayfailtobefullyinclusive.Whereaccessorsecurity concernsimpingeondatagathering,othermethodsincludingconsultingwithproxiesshouldbe employedtoensuretheinclusionofperspectivesfromallsidesofaconflict. Manyinterventionsinthisfieldworktopreventconflictbycreatingchangeinpeople'sattitudes, thoughtprocessesandrelationships.Suchworkoftenfocusesmoreonsupportingprocessesratherthan concretequantifiableoutputsandoutcomes.Someestablishedtypesofquantitativedatamaynot adequatelyaddresstheseissues.Itisoftenimportantthereforetocollectattitudinaldata,orconduct interviews,workshopsorfocusgroupdiscussionswithstakeholderstocollectsupplementaryqualitative data.Measuringintangiblechangesinareassuchasperceptionshasbeenafrustratingandelusivetask formanystakeholdersinthisfield.Suchconcernsshouldbeincorporatedintoplanning,budgeting,data collectionandselectionoftheevaluationapproach.
51
Reliabilityreferstotheconsistencyordependabilityofdataandevaluationjudgements,withreferencetothequalityofthe instruments,proceduresandanalysesusedtocollectandanalysedata.Evaluationinformationisreliablewhenrepeated observationsundersimilarconditionsproducesimilarresults.Reliabilitycontributestocredibilitythatcanbeadditionally enhancedthroughatransparentevaluationprocess(OECDDAC,1991).
38
Box11.
Quotebox:Howsecurityaffectsaccessanddata
Experienceshowsthatensuringthesecurityofevaluationteamsisamajorproblem.Oftenwe areunabletorecruitthemostqualifiedevaluatorstovisit[conflictaffected]countries.Evenwhen wefieldtheteam,evaluatorsmaybeunabletovisitthesiteoreventalktothelocalpeople. Undertheseconditions,manyevaluationsarewritteninthesafeconfinesoftheofficesofhost governmentsorofthemajorfundingagencies[Thishasserious]implicationsforstrategiesof datacollection.
Source:Kumar,2007.
Finally,datacollectioninparticularcanexertnegativeorpositiveeffectsontheconflict;therefore, specialattentiontoconflictsensitivity,basedonacurrentconflictanalysis,isespeciallyimportantduring thisphase.Participatorydatacollectionmayprovevaluablebutshouldbeusedwithcaution,becauseof potentialpartisanshipandthedynamicsofpoweranddominanceinconflictsettings.(Formoreon participatoryapproachesseeAnnex7.) The lack of quality data, including missing baseline and monitoring information, remains a serious challengeforevaluationsbecausesuchgapslimitpossibilitiesforbefore/aftercomparisonsandmakesit moredifficulttounderstandcontributionstochangeovertime.Betterplanningearlyon(asdescribedin Section3.1)shouldhelptoimprovethissituation.Still,furtherexplorationandrefinementofstrategies fordealingwithdataproblemsisneeded. 3.3.4Examinetheeffortusingvariouscriteria Applythecriteriachosenintheplanningstagestoexplorethepeaceeffort.Thisanalysisoften formsthemaincontentofanevaluationreport.Otherevaluativelensesmightalsobeappliedtoa particularintervention,forinstancecriteriaconcerninggender,rightsbasedapproaches,andcultural sensitivityagain,thiswilldependonthescope. Thesectionbelowoutlineseachcriterionandillustratestheadaptationofthecriteriatotheconflict preventionandpeacebuildingfield.52 Theevaluationcriteriaareinterlinkedandinterdependent.Each criterionshedslightontheinterventionbeingevaluatedfromaslightlydifferentperspective,todevelop ascomprehensiveapictureaspossibleoftheintervention.Assuchtheyareintendedtooverlap,butnot toduplicate.Whenreadtogethertheyshouldassisttheevaluationteamindevelopingaclear,holistic understandingoftheactivityorpolicybeingevaluated. Relevance Therelevancecriterionisusedtoassesstheextenttowhichtheobjectivesandactivitiesofthe intervention(s)respondtotheneedsofthepeacebuildingprocess.Thepeacebuildingrelevancelinksthe analysisoftheconflictsituationandthepeacebuildingprocesswiththeinterventionsobjectives,and thusseekstofindoutwhetheraninterventionisontherighttracktocontributetopeacebuilding.
52
ThisadaptionoftheDACCriteriaismostlydrawnfromPaffenholz&Reychler(2007)withadditionsfromCDA(2007)and Anderson(2003).ItalsodrawsontheOECDDAC(1991)andtheOECDDACGlossary(OECD,2002).
39
Relevanceoftheinterventionmightchangeovertimewithchangingcircumstances.Inordertoascertain therelevanceofaninterventionforpeacebuilding,itisalsonecessarytoassesswhatotheractorsare presentandwhethertheinterventiontiesinwithoverallstrategiesandpolicyframeworks. Assessingtheinterventionsconflictanalysisisakeypartofevaluatingrelevanceinthisfield.Ifa conflictanalysishasalreadybeencarriedoutbystaff,managersorothersinvolvedinthedesignand implementation,theaccuracyanduseofthisanalysisshouldbeassessed.Assessingwhetherornotthe explanationisproving(wasproved)accuratewillbeimportantnotonlyasanaspectofthisparticular evaluationbutbecauseitcontributestolearningandtherefinementoftheoriesaboutwhyviolence occursandwhatarethemostimportantdeterminantsofconflictdynamics.Ifnoprocessofsystematic analysishastakenplacebefore,theevaluatorteammaydiscusswithstaffandstakeholdersto understandwhatunderlying(unarticulated)conflictunderstandingisguidingtheirwork,orfacilitatea moreformalexercisetodevelopaconflictanalysiswithstakeholderbuyin. Questionsaboutrelevancemightinclude: Istheconflictpreventionandpeacebuildinginterventionbasedonanaccurate(anduptodate) analysisoftheconflict?Isitworkingontherightissuesinthiscontextatthistime? Doesitthereforeaddressrelevantcausesofconflicts,keydynamicsanddrivingfactors,orkey drivingconstituenciesoftheconflict? Arethestatedgoalsandobjectivesrelevanttoissuescentraltotheconflict?Doactivitiesand strategiesfitobjectives? Hastheeffortrespondedflexiblytochangingcircumstancesovertime?Hastheconflictanalysis beenrevisitedorupdatedtoguideactioninchangingcircumstance? Whatistherelevanceoftheinterventionasperceivedbybeneficiariesandexternalobservers?
Effectiveness Effectivenessisusedtoevaluatewhetheraninterventionhasreacheditsintendedobjectives,with respecttoitsimmediatepeacebuildingenvironment,inatimelyfashion.Thekeytoevaluating effectivenessandthusthelinkagebetweenoutputs,outcomesandimpactsisfindingouttowhatdegree theenvisagedobjectiveshavebeenfulfilled,andnotingchangesthattheinterventionhasinitiatedorto whichithascontributed.Assessmentshouldcoverboththedesiredchangestheprojectaimedto achieve,aswellasunintendedpositiveandnegativechanges.Somequestionstoaskmightinclude: Hastheinterventionachieveditsstated(orimplicit)purpose,orcanitreasonablybeexpected todosoonthebasisofitsoutputs? Is(orwill)theeffortachieveprogresswithinareasonabletimeframe?Isitpossibleto acceleratetheprocess?Shouldtheeffortbesloweddownforanyreason? Doestheeffortpromptpeopleincreasinglytoresistviolenceandprovocationstoviolence? Dothestakeholdersaffectedhaveasignificantimpactontheconflict?(Aretheright/keypeople ormanypeoplebeingaddressed?)Weregenderandrelevanthorizontalinequalities(ethnic, religious,geographical,etc.)takenintoconsideration?
40
Doestheeffortresultinanincreaseinpeoplessecurityandintheirsenseofsecurity? Doestheeffortimprovenonviolentformsofconflictresolutionorpowermanagement? Doestheeffortresultinrealimprovementinrelationsamonggroupsinconflict? Whatmajorfactorsarecontributingtoachievementornonachievementofobjectives?
The implementation or programme logic and theory of change should be tested. Is this the right thingtobedoingatthistimeinthiscontext?Isitbasedonareasonabletheoryandlogic,andarethese proving (or have they proved) to be true by data collected on outputs, outcomes and impacts? Remember that a programme or policy may do good or do well and still not change the underlying dynamicsorkeydrivingfactorsoftheconflict.Evaluatorsshouldtrytodeterminewhethersucharesultis causedbyfaultydesignlogic,aflawedtheoryofchangeorother factors. Furtherstrategiesandreallife examplescanbefoundinAnnex6.
Box12.
Typesofsuccessandfailure
Whenevaluatingitisimportanttodistinguishbetweenandanalysevarioustypesoffailure.Inthis fieldithelpstodistinguishafailureofthetheoryofchangefromafailureinimplementation.Doingso willhelpincreaseknowledgeaboutwhatisworkinginconflictpreventionandpeacebuilding,andwhy. Theoryfailureindicatesthefailureofaconflictpreventionorpeacebuildingactivityduetoaflawed causalrelationship,i.e.thatunderlyingassumptionsabouthowtomakechangeinthiscontextarefalse. Afaultytheoryofchangecouldbebasedonaninaccurateconflictanalysis,oritcouldreflectmisdirected priorities,unintendedimpactsormismatchedobjectives.Implementationfailurereferstoaproblemwith theexecutionoftheactivityitself(inputs/outputs,staffcapability,timing,location,orbudget)or managementsystems.Suchproblemscouldincludesuddenchangesintheconflictthatdisruptor reverseprogress,despiteanotherwisewelldesignedactivity.
Impact Thecriterionofimpactreferstopositiveandnegative,primaryandsecondarylongtermeffects producedbyanintervention,directlyorindirectly,intendedorunintended.Intheconflictpreventionand peacebuildingcontextthecriterionisusedtoidentifyandevaluatetheeffectsapolicyorprogrammehas hadonthepeacebuildingandconflictenvironment.Impactscanberelativelyimmediateorlongerterm. Itisnotnecessarytoholdaconflictpreventionandpeacebuildinginterventiontoanultimatestandardof achievingpeace.Rather,theevaluationshouldidentifytheeffectsoftheinterventiononthekey drivingfactorsandactorsoftheconflict. Evaluatorsshouldassessresultsatdifferentlevels:project,programme,strategyandpolicy,aswell asfromthelocalleveltotheregionalandnationallevel.Analysetheinformationgatheredandassessthe effortsresults,followingthewellacceptedresultschain(inputs outputs outcomes impacts). Thoughthereareoftenlongtimescalesforseeingimpactsinbuildingorsustainingpeace,notallimpacts arelongtermandsomewillbeimmediate.Evaluatorsalwaysneedtolookatimpacts. Changesinbehaviourandattitude,inadditiontobeingdifficulttomeasureandsubjecttosetbacks, oftentakealongtime.Withthisinmind,ifevaluatorsdeterminethatitisnotreasonabletoexpect
41
significantconflictimpactsyet,theyshouldfocusonoutcomesandtestthetheoryofchangeand programmelogictopredictwhetherthecurrentstrategiesarelikely(overthelongrun)tomakea contributiontopeace.Whenviolentconflictisstillraging,evaluatorsmayhavetofocusmorenarrowly onoutputindicatorsforquickmeasuresofvisibleresultsandtangibleshorttimespanchangesincurrent conditions,ratherthantryingtoevaluatewideroutcomesorimpacts. 53 Keyquestionsmightinclude: Howhasthesituationchangedovertime,andwhatisthecontributionoftheinterventionto thosechanges? Whichchangesinattitudes,behaviours,relationshipsorpractices(ofhowmany people/classifiedaccordingtohorizontaldivisions)canbeascertained? Hastheinterventionledtopolicychanges?Bywhom?Howdotheserelatetotheconflict? Arethereanysecondarynegativeeffects? Forindividualprojects/programmes/policies:Whataretheprimaryandsecondary,directand indirect,positiveandnegative,intendedandunintended,immediateandlongterm,shortterm andlastingeffectsoftheeffort?Doesitimpactsignificantlyonkeyconflictorpeacefactors? Forcountrystrategies,policies,multiprogrammeorjointevaluations:Whatarethecombined andcumulativeeffects,primaryandsecondary,directandindirect,positiveandnegative, intendedandunintended,immediateandlongterm,shorttermandlasting,ofthemultiple efforts?Howdotheserelate,innontrivialways,totheconflictorpeaceprocessanditskey elements?
Understandingallimpacts,positiveandnegative,shouldinvolvelookingatconflictsensitivity. Evaluatorsmayneedtobeparticularlyattentivetoexaminingissuessuchas:thetargetagencysown waysofworking,includinghiringofstaff,selectionofbeneficiaries,selectionofpartners,programme design,etc.;relationswithlocalauthorities,includingmilitaryactors(protectingmandatesand humanitarianspace);unequaldevelopment(takingformerandcurrenttrendsintoconsideration);and socialhierarchies(instaffselectionandpromotion,inrelationshipswithlocalcommunitiesandlocal authorities,inrelationshipswithpartnerorganisations,etc.). 54 Sustainability Sustainabilityisdefinedasthecontinuationofbenefitsfromadevelopmentinterventionaftermajor assistancehasbeencompleted.Itincludestheprobabilityofcontinuedlongtermbenefitsandresilience toriskovertimeandincludesfinancial,institutional,humanresource,managementandotherelements. Asinotherfields,sustainabilityalsoincludesownershipofpeaceanddevelopmentprocesses. Experienceandpeaceresearchdemonstratethatpeacebuildingprocessesarelongtermandthusneed longtermengagementthatcanweathersetbacks. 55 Inconflictregionsthismustincludeaddressing spoilerswhohaveaninterestinsustainingtheconflict. Whichstepshavebeentakenorareplannedtocreatelongtermprocesses,structuresand institutionsforpeacebuilding?
53 54
Somepractitionersfeelthatoutputindicatorsarerelativelyeasiertotrackandmaybelesssusceptibletoattributionfailures. Anderson,1999a. 55 SeeamongothersOECDDACandCDA,2007.
42
Willnewinstitutionsdesignedtoaddressconflictssurvive?Aretheybeingused? Willhardwonimprovementsinintergrouprelationshipspersistinthefaceofchallenges? Willthepartiestoanegotiatedagreementhonourandimplementit?Areeffectivemechanisms andincentivesinplacetofacilitateimplementation? Hasameaningfulhandingoverorexitstrategybeendevelopedwithlocalpartnersoractors thatenablethesepartnerstobuildorcontinuetheirownpeacebuildinginitiatives? Havethosewhobenefitfromongoingviolenceorinstabilityorwhoresistmovementtowards peace(spoilers)beenaddressedadequately? Doestheeffortresultinthecreationorreformofpoliticalinstitutionsormechanismsthatdeal meaningfullywithgrievancesorinjustices? Doestheeffortcontributetomomentumforpeacebyencouragingparticipantsand communitiestodevelopindependentinitiatives?
Box13.
Efficiencyandspendingonsecurity
ArecentjointevaluationofhumanitarianaidandreconstructionassistancedeliveredinAfghanistan between2001and2005statesthattherewasanapproximately20%securityoverheadoverall.The evaluationpointedoutthattheseunexpected(orunderestimated)costsmadetheAfghanistan programmesconsiderablymoreexpensivethansimilarprogrammesinothercountries.
Source:Chr.MichelsenInstitute,2005.
Efficiency Thiscriterionisusedtoassesshoweconomicallyresources/inputs(funds,expertise,time,etc.)are convertedtoresults.Inaconflictcontext,costsassociatedwithpreventionworkwilloftenbecompared withtheestimatedcostsofwaroranoutbreakofviolentconflict.Yet,asavertedconflictsareinvisible, counterfactualanalysiscomparingcostsofpreventioncanbedifficult.Inadditiontocomparingthecost ofsupportingpeacewiththecostsofwar,evaluationshouldlookatprioritiesisthisparticularwayof workingagainstviolencethemostefficientoption? Whenlookingatdevelopmentorhumanitarianinterventionsinconflictareas,evaluatorsshould determineefficiencyasrelatedtootheroptionsforsupportingpeaceinthis(orasimilar)conflictcontext. Comparingthetargetwithinterventionsinareasnotexperiencingseriousconflictisnotparticularly useful,asthecostsandrisksinvolvedareusuallynotcomparable. Whenlookingatefficiency,consider: Doestheinterventiondeliveritsoutputandoutcomesinanefficientmanner(resultsagainst costs)? Howdoesthisparticularprogrammeorpolicyapproachcompareincoststootheroptionsfor achievingthesamegoals?
43
Howefficientisthegeneralmanagementoftheintervention(steering,management, organisationalandgovernancestructuresandprocedures)? Howwellareresourcesusedtoachieveresults?
Coherence(andcoordination) Intheconflictpreventionandpeacebuildingcontexts,apolicy,programmeorprojectcannotbe assessedinisolation.Whatmayseemappropriatefromthepointofviewofoneactivitymaynotbe appropriatefromthepointofviewofthesystemasawhole.Itisimportanttoconsiderthedegreeto whichtheinterventionisconsistentwithoralignedtothelargerpolicycontexts(nationaland international)withinwhichitistakingplace;thedegreetowhichitformspartofandisconnectedtoa conflictstrategyoroverallcountryframework;andthedegreetowhichitiscoordinatedwithother policies,programmesorprojectswithinitsconflictenvironment,thematicclusterorregion. Itisalsoimportant,however,thatclosecoordinationwithotheractors/sectorsnotbeassumedto automaticallycontributetoachievingresults.Atleasttwopotentialnegativeeffectshavebeenidentified, namelyunduepressureonlocalactorsasaresultofacoordinatedblockapproachamonginternational actors,andinappropriateinfluencesontheneutrality,impartialityandindependenceofhumanitarian actors.Evaluationsshouldnotonlyassesscoherenceandcoordination,butalsothevaluethatwas generated(ifany)bythetimeandresourcesinvestedincoordinationandcoherence,andwhat(ifany) unintendedconsequencessucheffortsmayhavestimulated.Askquestionssuchas: Wascoordinationfactoredintoinputs(wasitbudgetedfor)andoutputs(isitexplicitlylistedas anoutput,andisitpartoftheexpectedevaluationreporting)? Hasacoherentapproachbeeninstitutionalised?Isharmonisationacrossactorsgrowing? Howmuchtimeandwhatresourceswerespentoncoordination?Wasitefficient(cost/benefit andappropriateness)? Didcoordinatedworkresultinimprovedcoherenceinpolicy? Howweregenderandanyrelevantconflictspecificinequalitiestakenintoconsiderationwhen decisionsweretakenaboutwithwhomandhowtocoordinate(especiallyinthecontextofco ordinationwithlocalactors)? Whatwerethemainconstraintsandchallengesforcoherence?Howwasgoodcoordination achieved,andisitreplicableinothersituations?
Thefollowingoptionalcriteriaareprovidedforuseduringthetestingapplicationphase. Linkages Intheconflictandpeacefieldthiscriterioncoverstheconnectionsbetweenactivitiesandpoliciesat differentlevelsandacrosssectors.Thereisevidencethatevaluatinglinkagesamongindividual/personal
44
changesandsociopoliticalchangesarecriticalforcumulativepeacebuildingeffectiveness. 56 This criterionmightalsobeusedtoexaminethelinksbetweenpeacekeepingorpeacebuildinginterventions andlongertermdevelopmentprocesses. Coverage Thecoveragecriterionmaybeusedforassessingpolicylevelconflictpreventionandpeacebuilding effortsandmayalsoapplytoprogrammeefforts.Itcanbeassessedwithintheconflictinquestionfor examplebylookingatthecoverageofatargetpopulationorgeographicalarea.Arecertainregions excludedorincludedandhowdoesthisimpacttheabilitytoreachallthoseinneed?Howdoescoverage, orthelackofit,affectcurrentconflictdynamics?Itmayalsobeappliedatamoregloballeveltolookat howmuchattentionisbeingpaidtovariousconflicts.Anevaluatormightask:dodonorpolicies effectivelycoverall(potential)conflicts?Howdocontributionstooneparticularconflictregionorcountry asopposedtoanotherrelatetoneed?Aretherehiddenorforgottenconflictsthatreceivelittleor nointernationalattention?
Box14.
Coverageexample
[Exampleneededofcoverageissues,challenges,affectsoflackofcoverage,evaluatingcoverage,etc.?]

Consistencywithvalues Interviewswithanumberofpractitionersinthisfieldhighlightedthepossibleneedordesireforan additionalcriterionthatwouldassesstheconsistencyofvaluesinmeansandends. 57 Programmers, managersandevaluatorshavebeenfrustratedbypoliciesandfieldprogrammesthatclaimtosupport peace,yetworkinwaysthatincreasetension,arebiasedordisrespectfulofdifferences,promote patriarchyanddivision,oraregenerallyinsensitive.Suchactionsandattitudesareseenascontradicting theaimofpeacemakingwork.Thiscriterioncouldalsobeusedtoassessconflictsensitivity. 3.3.5Lookatthebigpicture Thesuccessofindividualpeacebuildingorconflictpreventionprogrammesoftendependsuponthe politicalanddiplomaticpressurethattheinternationalcommunityhasorhasnotexertedonthe involvedgovernment(s)andwarringparties.Conflictpreventionandpeacebuildingevaluationsfocused onlyattheprojectandprogrammeorsectorleveloftenfailtoidentifyimportanteffectsorconstraintsat theleveloftheoverallsystem.Evaluationteamsshouldthereforeexaminetherelationshipbetween interventionsandthepoliticalordiplomaticpressurethattheinternationalcommunity,particularly majordonorsandneighbouringcountries,haveexertedorfailedtoexert.Thisoftenmeansthata
56 57
AndersonandOlson,2003. OECDDACandCDA,2007.
45
broaderperspectivewillbeneededforconflictandpeacerelatedevaluations.Inadditiontoassessing theimplementation,managementandoutcomesofactivities,evaluatorsshouldaskwhethertheactivity has,orisconnectedto,alargerconflictpreventionandpeacebuildingstrategy. Tensionsarisingamongvariouspolicyobjectivesandinstrumentsusedbyanyoneagencyor governmentsignalthatpolicycoherenceisneeded.Disappointmentwithadhocconflictengagementor narrowstrategiesexclusivelyreliantonaidhasfurtherencouragedasurgeinresearchandstrategising aroundpolicycoherence.Policycoherencehasemergedasakeyrequirementofdevelopment effectiveness:allpolicyvectorsshouldpullinthesamedirection. 58 Evaluatorswilllikelyneedtoexamineotherdonorpoliciesbeyondtheofficialdevelopment assistanceorparticularpeaceactivityinquestion.Whenexaminingdonorpoliciesorcountrystrategiesin conflict,itisimportanttoconsiderthepossibleusefulnessofotherpolicyinstrumentsavailabletoa donororpartnergovernment.Fundingforaparticularconflictpreventionprojectorpeacebuilding initiativecanbeovershadowed,contradicted,orontheotherhandsupportedandsustained,byother actionsofthesamegovernment(s). Dependingonscope,evaluatorslookingatsingleprojectsorprogrammesshoulddeterminewhat policiesorstrategies(countrystrategies,broaderpolicystatements,diplomaticmeasures,specific approachestoconflictpreventionandpeacebuilding)applytotheinterventionbeingevaluated.Dothe activitiessquarewiththerelevantpolicies?Iftheprojectisjudgedsuccessfulyetdivergesfromoverall policiesorstrategies,whatdoesthatsuggestregardingthepoliciesthemselves?Iftheinterventions complywiththepolicyyetappeartofail,whataretheimplicationsforthepolicy? Whileitwouldnotberealisticforeverysingleevaluationtocovertheentireprogrammeandpolicy arena,theselinksmustbetakenintoconsiderationwhenplanningindividualevaluationsandwhen makingoverallevaluationplansforexamplewhenagenciesplan,scheduleandbudgetforfuture evaluations.Whenmappingoutlongertermevaluationplans,agenciesshouldseriouslyconsider undertakingevaluationsthatcaptureandassessthesemacrostrategicissues,inadditiontostandard projectorprogrammelevelevaluations.
58
Manning,2003.
46
3.4Concludingandlearningfromtheevaluation Summary:Concludingtheevaluationprocessand learningfromevaluation forevaluators,managers,policymakersandprogrammestaff

Drawconclusionsandmakerecommendations Ensurequality Conductreporting Disseminate,feedbackandengageinalearningprocess 3.4.1Drawconclusionsandmakerecommendations Theultimategoalofanyevaluationistoanswerthekeyquestionsaskedandpresenttheresultsina usefulway.Analysisofthecollecteddatashouldleadtologicalconclusions,findingsand recommendationsthatwillhelpimprovetheevaluatedactivityorfutureactivities.Findings,conclusions, recommendationsandlessonslearnedshouldberigorous,relevant,targetedtotheintendedusersofthe evaluationsandactionable(meaningthosewhoreceivetheinformationareabletoactuponit).This shouldbepresentedclearly,showingaclearlineofevidencetosupporttheconclusions,orprovidingan explanationifthisisnotpossible.Logicalandevidencebasedconclusionswillhelpimprovecomparability acrossevaluationsandbetweendiverseactorsintheconflictpreventionandpeacebuildingfield.Over time,suchconclusionswillcontributetobetterstrategiesforinterventioninconflict,forexampleby demonstratingtheaccuracy(orlackthereof)ofconflictanalyses,showingwhetherornottheoriesof changework,oridentifyingappropriateindicators. Dependingonthetypeofevaluation,conclusionsandrecommendationsmaybedevelopedina participatoryformat.Forexample,initialfindingscouldbepresentedtoagroupofkeystakeholderswho wouldthenworkwithevaluatorstodrawusefulconclusions.Ontheotherhand,ifthefocusofthe evaluationisonaccountability,theevaluatorsarelikelytotakeamoreindependentapproach. Evaluatorsmaydiscovermajordifferencesofopinionregardingnotonlywhathappened,butthe valueofoutcomesandimpacts,particularlybecauseindividualandgroupunderstandingsarehighly determinedbytheconflictandtheirownrolesinit.Whatisviewedasasuccessfulinterventionbysome groupsmayhaveharmedanothergrouporincreasedtension.Asanevaluatoritisimportanttoinclude andbalanceinsider/outsiderperspectives,andmaintainasobjectiveapositionaspossible. 3.4.2Ensurequality Sinceevaluationrecommendationsoftenhavepolicyandfundingimplications,itisimportantthat theyaresubjecttoreliablequalityassuranceprocesses.Theevaluationteamshouldexplicitlyaddress qualityassurance(indeedthismayberequiredbythosecommissioningtheevaluation).Thiswillensure thatthevettingofreportsgoesthroughseveralqualityassurancesteps,includingtheteamsinternal ratingaswellasthecommissioningagencyssystem.ThedraftDACEvaluationQualityStandardsisa goodguideforreviewingevaluationproducts(suchasthefinalreport)andprocesses.
47
3.4.3Conductreporting Reportingtakesplacethroughouttheevaluationprocess(includingduringstepslistedinSections3.2 and3.3).Thereareoftenthreemainreportingsteps: 59 Inceptionreport:Afterconductingaconflictanalysisandgatheringinitialinformation,the evaluatorswilldraftaninceptionreportdescribinghowtheteamintendstoconductthe evaluationandanswerthequestionssetoutintheTOR.Itpresentsrisksandchallenges,the methodologytobeused,datacollectiontools,indicatorsifrelevant,operationalisationofthe mainquestions,casestudies(ifnotselectedpriortocommissioningtheteam),thestructureof thereportandaworkplanfortheremainingwork.Stakeholdersusuallycommentonthe inceptionreports,oftenaspartofareferencegroup. Draftreport:Adraftoftheevaluationreportisoftencirculatedwidelyforcomments,andisa chanceforstakeholderstocommentontheevaluation.Sufficienttimeforcommentingshould becalculatedintotheoveralltimeframe. Finalreport:Thoughanevaluationmayresultinmanydifferentoutputs,awrittenreportis almostalwayscompleted.Reportsandpresentationswillneedtobetranslatedintolocally relevantlanguage(s)tofacilitatesharingwithallstakeholders.Thefinalreportissentto stakeholders.Targetgroupsfordisseminationshouldbeagreedoninthebeginningofthe process.ManyorganisationsnowusetheInternetasanalternatemeansofpublishingthe report,eitherinpartorinitsentirety.Inanycase,becarefultoprotectconfidentiality(as needed)andthesafetyofthosewhocontributedtotheevaluation.
3.4.4Disseminate,feedbackandengageinalearningprocess Implementtheplans(determinedinthepreparationstage)forfollowupanddisseminationof lessonslearnedthroughappropriatemeansofcommunicatingtheresultstothetargetgroup(s).The commissioningagencyshouldcommunicatethemainfindingsandconcreterecommendationsofthe evaluationtomanagers/decisionmakers.Thisshouldbedonetogetherwithanassessmentofthequality oftheevaluation.Theevaluationteammayplayspecificrolesinthisprocess,butfollowupisnormally theresponsibilityofthepersonorunitthatcommissionedtheevaluation.Guidanceoneffectivefeedback practicescanbefoundintheDACEvaluationNetworksEvaluationFeedbackforEffectiveLearningand Accountability. 60 Sharingtheoutcomesofanevaluationcanbedifficultgivenpractitionersoftenstrongattachment totheirstrategiesandcommitmenttotheirwork.Receptivitycanbeenhancedbyemphasisingthe learningaspectsofevaluationthismayrequiretactandfacilitationskills. Evaluationshouldfeedbackintotheupstreammeasuresofplanningandprogrammedesign (outlinedinSection3.3.1andAnnex3)andhelptoaddresssomeofthechallengesinthisfieldby providingmoreevidenceonthevalidity(orlack)ofvarioustheoriesandprovidingdataforcomparison
59
Someagenciesmayalsorequireanoutbriefbeforetheteamdepartsfromafieldvisit(betweentheinceptionanddraft reports),topromoteaccountability. 60 OECD,2001.
48
andreference.Evaluationsperformedwhileapolicyorprogrammeisstillinplacecanbeusedtoadjust orredesignit.Havingcompletedtheevaluationandlearningprocess,stakeholders,decisionmakers, managersandstaffshouldbebetterabletounderstandandimproveoutcomesandimpacts,making morelastingcontributionstopeace.
49
CONCLUSION
[Contentsummarisingkeypoints/lessonstobeaddedfollowingapplicationphase]
50
BIBLIOGRAPHY
AfricanUnion(2006),PolicyFrameworkforPostConflictReconstructionandDevelopment(PCRD),African Union,AddisAbaba. Anderson,M.B.(1999a),DoNoHarm:HowAidCanSupportPeaceorWar,LynneRienner,London. Anderson,M.B.(1999b),TheInfluenceofAidinSituationsofViolentConflict:ASynthesisand CommentaryontheLessonsLearnedfromCaseStudiesontheLimitsandScopefortheUseof DevelopmentAssistance. Anderson,M.B.andL.Olson(2003),ConfrontingWar:CriticalLessonsforPeacePractitioners, ReflectingonPeacePracticeProject,CollaborativeForDevelopmentAction,Inc.,Cambridge,MA. Aoi,C.,R.ThakurandC.H.deConing,eds.(2007),UnintendedConsequencesofPeacekeeping,United NationsUniversityPress,NewYork. Bamberger,Michaeletal.(2004),RealWorldEvaluationinaNutshell,abriefsummaryoftheapproach. FullarticleinAmericanJournalofEvaluation,Vol.25,No.1,Spring,underthetitleShoestring Evaluation. Bamberger,M.,J.RughandL.Mabry(2006),RealWorldEvaluationWorkingUnderBudget,Time,Data, andPoliticalConstraints,SagePublications,Inc.,ThousandOaks. BarthEide,E.,A.T.Kaspersen,R.KentandK.vonHippel(2005),ReportonIntegratedMissions:Practical PerspectivesandRecommendations,NorwegianInstituteofInternationalAffairs(NUPI),Oslo. Beck,T.(2006),EvaluatingHumanitarianAction:UsingtheOECDDACCriteriaAnALNAPGuidefor HumanitarianAgencies,OverseasDevelopmentInstitute(ODI),London. Borton,J.andJ.Eriksson(2004),LessonsfromRwandaLessonsforToday.AssessmentoftheImpact andInfluenceofJointEvaluationofEmergencyAssistancetoRwanda,DanishMinistryofForeign Affairs,DanishStateInformationCentre,December.http://danida.netboghandel.dk/ CDACollaborativeLearningProjects(2000),OptionsforAidinConflict:LessonsfromFieldExperience, CollaborativeforDevelopmentActionCDACollaborativeLearningProject,Cambridge,MA. CDACollaborativeLearningProjects,incooperationwithCAREInternational(2006),HasPeacebuilding MadeaDifferenceinKosovo?,TheCollaborativeforDevelopmentActionInc.,Cambridge,MA. Chesterman,S.(2002),TiptoeingThroughAfghanistan:TheFutureofUNStateBuilding,International PeaceAcademy,NewYork. Chr.MichelsenInstitute,CopenhagenDevelopmentConsultingandtheGermanAssociationof DevelopmentConsultants(2005),JointEvaluation:HumanitarianandReconstructionAssistance toAfghanistan,200105fromDenmark,Ireland,theNetherlands,SwedenandtheUnited Kingdom,Chr.MichelsenInstitute,October. Church,C.andRogers,M.(2006),DesigningforResults:IntegratingMonitoringandEvaluationin ConflictTransformationPrograms,SearchforCommonGround,Washington,DC. Church,C.andJ.Shouldice(2003),TheEvaluationofConflictResolutionInterventions,PartII:Emerging PractiseandTheory,INCORE,London.
51
Collier,P.etal.(2003),BreakingtheConflictTrap:CivilWarsandDevelopmentPolicy,WorldBankand OxfordUniversityPress,Washington,DC. CSIS(2002),PostConflictReconstruction:TaskFramework,CenterforStrategicandInternationalStudies (CSIS)andtheAssociationoftheUnitedStatesArmy(AUSA),Washington,DC. Cutillo,A.(2006),InternationalAssistancetoCountriesEmergingfromConflict:AReviewofFifteenYears ofInterventionsandtheFutureofPeacebuilding,InternationalPeaceAcademy(IPA),NewYork. Dahrendorf,N.(2003),AReviewofPeaceOperations:ACaseforChange,KingsCollege,London. deConing,C.H.(2007),CivilMilitaryCoordinationandUNPeacebuildingOperationsinH.Langholtz, B.KondochandA.Wells(eds.),InternationalPeacekeeping:TheYearbookofInternationalPeace Operations,Volume11,KoninklijkeBrillN.V.,Brussels. DevelopmentEvaluationResourceCentre(DEReC),OECDDACNetworkonDevelopmentEvaluation, www.oecd.org/dac/evaluationnetwork/derec Dobbins,J.E.etal.(2005),TheUNsRoleinNationBuilding:FromCongotoIraq,RandCorporation, Washington,DC. Donini,A.(2002),ThePoliciesofMercy:UNCoordinationinAfghanistan,MozambiqueandRwanda, OccasionalPaper#22,ThomasJ.WatsonJr.InstituteforInternationalStudies,BrownUniversity, Providence. Doyle,M.W.andN.Sambanis(2000),InternationalPeacebuilding:ATheoreticalandQuantitative Analysis,WorldBank,Washington,DC. Duffield,M.,S.LautzeandB.Jones(1998),StrategicHumanitarianCoordinationintheGreatLakesRegion 19961997,UnitedNationsOfficefortheCoordinationofHumanitarianAffairs(OCHA),NewYork. Eriksson,J.etal.(1996),TheInternationalResponsetoConflictandGenocide:LessonsfromtheRwanda ExperienceSynthesisReport,JointEvaluationofEmergencyAssistancetoRwanda,DANIDA, Copenhagen. FAFOInstituteforAppliedInternationalStudies(2006),EvaluatingConflictPreventionandPeacebuilding Activities,reportpreparedfortheOECDDACNetworkonDevelopmentEvaluationandthe NetworkonConflict,PeaceandDevelopmentCooperation.(unpublished) IASC(2004),CivilMilitaryRelationsinComplexEmergencies,UNInterAgencyStandingCommittee(IASC), UnitedNations,NewYork,June. InternationalAlert(2001),GenderandPeaceSupportOperations:Opportunities&Challengesto ImprovePractice,GenderandPeacebuildingProgram,InternationalAlert,London. http://www.internationalalert.org/pdfs/pso_briefing_2001.PDF InternationalAlert(2007a),ConflictsensitiveApproachestoDevelopment,HumanitarianAssistanceand PeaceBuilding:ToolsforPeaceandConflictImpactAssessment,Chapter2,InternationalAlert, London.www.conflictsensitivity.org. InternationalAlert(2007b),ActingonCommitments:HowEUStrategiesandProgrammingCanBetter PreventViolentConflict,InternationalAlert,Saferworld,andtheEuropeanPeacebuildingLiaison Office. Jones,R.,Dr.P.GeorgeandC.Stanley(2001),CanadaJapanJointPeacebuildingLearningProject GuatemalaMissionReport,postedonDEREC,www.oecd.org/dac/evaluationnetwork.derecand: http://www.jica.go.jp/english/evaluation/program/thematic/pdf/2001_7.pdf.
52
Kumar,K.(2007),Personalcommunication,Washington,DC. Lederach,J.P.(2002),BuildingPeace:SustainableReconciliationinDividedSocieties,USIPPress, Washington,DC. Licklider,R.(1995),TheConsequencesofNegotiatedSettlementsinCivilWars194593,American PoliticalScienceReview,Vol.89,No.3. Lund,M.(2004),PreventingViolentConflicts:ConflictSensitiveDevelopmentinthe21stCentury, CommissionedbytheConflictPreventionandReconstructionUnit,SocialDevelopment Department,WorldBank,ManagementSystemsInternational,Inc.,Washington,DC. Makken,F.andP.Wijmenga(1999),ReviewoftheNetherlandsDevelopmentProgrammeforthe PalestinianTerritories,19931998,IOB,theNetherlands,November. Manning,R.(2003),EffortsandPoliciesoftheMembersoftheDevelopmentAssistanceCommittee, DevelopmentCooperationDirectorate,OECD,Paris. Menkhaus,K.(2006),ConflictPreventionandHumanSecurity:IssuesandChallengesinR.Picciottoand R.Weaving(eds),SecurityandDevelopment:InvestinginPeaceandProsperity,Routledge,New York. Minear,L.andU.Chellia(1992),UNCoordinationoftheInternationalHumanitarianResponsetotheGulf Crisis,ThomasJ.WatsonInstituteforInternationalStudies,Providence. MinistryofForeignAffairsofJapanandUNDP(2002),PostconflictAssistanceoftheGovernmentof JapanthroughUNDPinKosovoandEastTimor,UNDPEvaluationOffice,NewYork. NEPAD(2005),AfricanPostConflictReconstructionPolicyFramework,NEPAD,Midrand. NORAD(2006),ProceedingsoftheWorkshoponDevelopingaGuidanceforEvaluatingConflict PreventionandPeacebuildingActivities,30November1DecemberOslo,Norway. OECDDAC(1999),GuidanceforEvaluatingHumanitarianAssistanceinComplexEmergencies,DAC NetworkonDevelopmentEvaluationOECD,Paris. OECDDAC(2001),EvaluationFeedbackforEffectiveLearningandAccountability,DACNetworkon DevelopmentEvaluation,OECD,Paris. OECDDAC(2002),GlossaryofKeyTermsinEvaluationandResultsBasedManagement,DACNetwork onDevelopmentEvaluation,OECD,Paris. OECDDAC(1991),DACPrinciplesfortheEvaluationofDevelopmentAssistance,OECD,Paris. OECDDAC(2000),EffectivePracticesinConductingaJointMultidonorEvaluation,DACNetworkon DevelopmentEvaluation,OECD,Paris. OECDDAC(1998and2001),DACGuidelinesonHelpingPreventViolentConflict,OECD,Paris. OECDDAC(2006a),DACEvaluationQualityStandardsDraftforTestApplicationPhase,DACNetwork onDevelopmentEvaluation,OECD,Paris. OECDDAC(2006b),GuidanceonManagingJointEvaluations,DACNetworkonDevelopment Evaluation,OECD,Paris. OECDDAC(2007),"ManualonSecuritySystemReform:SupportingSecurityandJustice",DACNetwork onConflict,PeaceandDevelopmentCooperation,OECD,Paris.
53
OECDDACandCDACollaborativeLearningProjects(2007),EncouragingEffectiveEvaluationofConflict PreventionandPeacebuildingActivities:TowardsDACGuidance,ConsultativeReportprepared fortheOECDDACNetworkonDevelopmentEvaluationandtheDACNetworkonConflict,Peace andDevelopmentCooperation,OECD,Paris. Paffenholz,T.andL.Reychler(2007),AidforPeace:AGuidetoPlanningandEvaluationforConflictZones, Nomos,BadenBaden. Paris,R.(2004),AtWarsEnd:BuildingPeaceAfterCivilConflict,CambridgeUniversityPress,Cambridge. Picciotto,R.andR.Weaving,eds.(2006),Securityanddevelopment:investinginpeaceandprosperity, Routledge,London. Porter,T.(2002),AnExternalReviewoftheCAP,UnitedNationsOfficefortheCoordinationof HumanitarianAffairs(OCHA),NewYork. Pugh,M.andN.Cooper(2004),WarEconomiesinaRegionalContext,LynneRienner,London. Reindorp,N.andP.Wiles(2001),HumanitarianCoordination:LessonsfromRecentFieldExperience,Study commissionedbytheOfficefortheCoordinationofHumanitarianAffairs(OCHA),Overseas DevelopmentInstitute(ODI),London. SIDA(2003),ReflectiononExperiencesofEvaluatingGenderEquality,SidaStudiesinEvaluation03/01, DepartmentforEvaluationandInternalAudit,Stockholm. SIDA(2004),ConflictSensitiveDevelopmentCooperation:HowtoConductaConflictAnalysis,Swedish DevelopmentAgency(SIDA),Stockholm. Sommers,M.(2000),TheDynamicsofCoordination,ThomasJ.WatsonJr.InstituteofInternational Affairs,OccasionalPaper#40,BrownUniversity,Providence. Smith,D.(2004),TowardsaStrategicFrameworkforPeacebuilding:GettingTheirActTogether,Overview ReportoftheJointUtsteinStudyonPeacebuilding,EvaluationReport1/2004,MinistryofForeign Affairs,Oslo. Stedman,S.J.(2005),APeacebuildingCommissionasAnInstrumentofConflictPreventionin A.Melbourn(ed.),Development,SecurityandConflictPrevention:SecurityasaMillenniumGoal, AnnaLindhProgrammeonConflictPrevention,MadariagaEuropeanFoundation,Stockholm. Stedman,S.J.,E.CousensandD.Rothchild,eds.(2002),EndingCivilWars:TheImplementationofPeace Agreements,LynneRienner,Boulder. Stockton,N.(2002),StrategicCoordinationinAfghanistan,AfghanistanResearchandEvaluationUnit (AREU),Kabul. Strand,A.(2003),WhosHelpingWho?NGOCoordinationofHumanitarianAssistance,Doctoral Dissertation,DepartmentofPolitics,UniversityofYork,York. Taylor,L.(2001),GoodMonitoringandEvaluationPractice,www.parcinfo.org. Tschirgi,N.(2006),TheSecurityDevelopmentNexus:ResearchFindingsandPolicyImplications, InternationalPeaceAcademy(IPA),NewYork. UN(1992),AnAgendaforPeace:PreventiveDiplomacy,PeacemakingandPeacekeeping,UnitedNations, NewYork.
54
UN(2000a),WethePeoples:TheRoleoftheUnitedNationsinthe21stCentury,ReportoftheSecretary Generaltothe54thSessionoftheGeneralAssembly,A/54/2000,UnitedNations,NewYork, 27March. UN(2000b),ReportofthePanelonUnitedNationsPeaceOperations,ReportoftheSecretaryGeneralto the55thSessionoftheGeneralAssembly,A/55/502,UnitedNations,NewYork,20October. UN(2004a),ReportoftheUNDG/ECHAWorkingGrouponTransitionIssues,UNDevelopmentGroup (UNDG),NewYork. UN(2004b),AMoreSecureWorld:OurSharedResponsibility,ReportoftheSecretaryGeneralsHigh LevelPanelonThreats,ChallengesandChange,UnitedNations,NewYork. UN(2005),InLargerFreedom:TowardsSecurity,DevelopmentandHumanRightsforAll,Reportofthe SecretaryGeneraloftheUnitedNations,UnitedNations,NewYork. UN(2006),DeliveringAsOne,ReportoftheSecretaryGeneralsHighLevelPanelonSystemwide Coherence,UnitedNations,NewYork. UnitedNationsDevelopmentProgramme(2002),GenderApproachesinConflictandPostconflict Situations,http://www.undp.org/women/docs/gendermanualfinalBCPR.pdf. USAIDandManagementSystemsInternational(2006),Postconflictmonitoringandevaluation,USAID, Washington,DC. USAID(2007a),Women&ConflictAnIntroductoryGuideforProgramming,UnitedStatesAgencyfor InternationalDevelopment(USAID),Washington,DC. USAID(2007b),Website:highlights,http://www.usaid.gov/our_work/crosscutting_programs/conflict. Uvin,P.(1999),IncentivesandDisincentivesforInfluencingConflictSituations,DACJournal,Vol.2, No.3,OECDDAC,Paris.http://www.oecd.org/dataoecd/31/24/1843044.pdf. Uvin,P.(2002),TheDevelopment/PeacebuildingNexus:ATypologyandHistoryofChangingParadigms, JournalofPeacebuilding&Development,Vol.1,No.1. VanBrabant,K.(2001),Understanding,PromotingandEvaluatingCoordination:AnOutlineFramework inD.S.GordonandF.H.Toase,AspectsofPeacekeeping,FrankCass,London. Walker,T.(2007),PersonalCommunication,Washington,DC,August. Wood,B.(2003),DevelopmentDimensionsofConflictPreventionandPeacebuilding,UNDevelopment Programme(UNDP),NewYork. WorldBank(2005a),TowardsaConflictSensitivePovertyReductionStrategy:Lessonsfroma RetrospectiveAnalysis,WorldBank,Washington,DC. WorldBank(2005b),ConductingQualityImpactEvaluationsunderBudget,TimeandDataConstraints, IndependentEvaluationGroup,WorldBank,Washington,DC.
55
ANNEX1 SELECTEDLISTOFKEYCONCEPTSANDTERMINOLOGY
activityactionstakenorworkperformedthroughwhichinputs,suchasfunds,technicalassistanceand othertypesofresourcesaremobilisedtoproducespecificoutputs(OECDDAC,2002:15). attributiontheascriptionofacausallinkbetweenobserved(orexpectedtobeobserved)changesand aspecificintervention(OECDDAC,2002:17). baselinestudyananalysisdescribingthesituationpriortoadevelopmentintervention,againstwhich progresscanbeassessedorcomparisonsmade(OECDDAC,2002:18). conflictanalysisasystematicstudyofthepolitical,economic,social,historicalandculturalfactorsthat directlyinfluencetheshape,dynamicsanddirectionofexistingorpotentialconflicts.Itincludesan analysisofconflictcausesanddynamicsaswellasassessmentsoftheprofiles,motivations,objectives andresourcesofconflictprotagonists(CDA:2007:2132andConflictSensitiveApproachesto Development,HumanitarianAssistanceandPeacebuilding,ToolsforPeaceandConflictImpact Assessment,2004:Chapter2). conflictmappingarepresentationofthemainaspectsofaconflictanalysis,illustratingrelationships betweenactors,causes,causalrelationships,etc. conflictpreventionactionsundertakentoreducetensionsandtopreventtheoutbreakorrecurrenceof violentconflict.Beyondshorttermactions,itincludesthenotionoflongtermengagement.Itconsistsof operationalprevention,i.e.immediatemeasuresapplicableinthefaceofcrisis),andstructural prevention,i.e.measurestoensurethatcrisesdonotariseinthefirstplace,or,iftheydo,thattheydo notrecur((OECDDAC,2001:86andUnitedNations,ReportoftheSecretaryGeneral,Preventionof ArmedConflict,2001:para.8). conflictsensitivitysystematicallytakingintoaccountboththepositiveandnegativeimpactsof interventions,intermsofconflictorpeacedynamics,onthecontextsinwhichtheyareundertaken,and, conversely,theimplicationsofthesecontextsforthedesignandimplementationofinterventions (ConflictSensitiveApproachestoDevelopment,HumanitarianAssistanceandPeacebuilding,Toolsfor PeaceandConflictImpactAssessment,2004:Introduction). counterfactualthesituationorconditionwhichhypotheticallymayprevailforindividuals, organisations,orgroupsweretherenointervention.(Forexample:thewarthatwouldhaveoccurred hadapeacebuildinginterventionnottakenplace) countryprogrammeoneormoredonorsoragencysportfolioofinterventionsandtheassistance strategybehindthem,inapartner(recipient)country. drivingfactorsofconflictthetrends,currents,causesorfundamentalinfluencesthataffectaconflict andhelpdetermineitscharacteristics,directionandultimateoutcome.
56
evaluabilityextenttowhichanactivityorprogrammecanbeevaluatedinareliableandcredible fashion.Evaluabilityassessmentscallfortheearlyreviewofaproposedactivityorprogrammeinorderto ascertainwhetheritsobjectivesareadequatelydefinedanditsresultsverifiable(OECDDAC,2002:21). evaluationevaluationreferstotheprocessofdeterminingmerit,worthorvalueofanactivity,policyor program.Itconsistsinthesystematicandobjectiveassessmentofanongoingorcompletedproject, programmeorpolicy,itsdesign,implementationandresults.Theaimistodeterminetherelevanceand fulfilmentofobjectives,developmentefficiency,effectiveness,impactandsustainability.Anevaluation shouldprovideinformationthatiscredibleanduseful,enablingtheincorporationoflessonslearnedinto thedecisionmakingprocessofbothrecipientsanddonors.(MichaelScriven,EvaluationThesaurus, FourthEdition,Sage,1991andOECDDAC,2002:21). exanteevaluationevaluationperformedbeforeimplementationofanintervention(DACGlossary). expostevaluationevaluationofaninterventionafterithasbeencompleted(DACGlossary). formativeevaluationevaluationintendedtoimproveperformance,mostoftenconductedduringthe implementationphaseofprojectsorprogrammes. fragilestate/fragilesituationnational,regionalandlocalterritorieswherethestate(theexecutive, thelegislativeandthejudiciary,includingcentralandlocalauthorities)lacksthecapacityand/orpolitical willandlegitimacytosupportequitabledevelopment.Thesesituationstendtobecharacterizedbypoor governance,tobepronetoviolentconflict,andtoshowlimitedprogresstowardstheMillenniumGoals. Anaggregateofgovernanceandsecuritycriteria,orofcapacity,accountabilityandlegitimacycriteriaare usuallyusedasmeasuresoffragility. goalthehigherorderobjectivetowhichadevelopmentinterventionisintendedtocontribute(OECD DAC,2002:25). impactspositiveornegative,primaryandsecondaryeffectsproducedbyanintervention,directlyor indirectly,intendedorunintended(OECDDAC,2002:24).Resultsthatliebeyondimmediateoutcomesor sphereofaninterventionandinfluencetheintensity,shapeorlikelihoodofaconflict. indicatorquantitativeorqualitativefactororvariablethatprovidesasimpleandreliablemeansto measureachievement,toreflectthechangesconnectedtoanintervention,ortohelpassessthe performanceofadevelopmentactor(OECDDAC,2002:25). inputsthefinancial,human,andmaterialresourcesusedforthedevelopmentintervention(OECDDAC, 2002:25). interventionageneraltermtorefertothesubjectoftheevaluationandmayrefertoanactivity, project,programme,strategy,policy,topic,sector,operationalarea,institutionalperformanceetc. Examplesarepolicyadvice,projects,programs(DACEvaluationQualityStandards). jointevaluationanevaluationtowhichdifferentdonoragenciesand/orpartnersparticipate(OECD DAC,2002:26). logicalframework(Logframe)managementtoolusedtoimprovethedesignofinterventions,most oftenattheprojectlevel.Itinvolvesidentifyingstrategicelements(inputs,outputs,outcomes,impact)
57
andtheircausalrelationships,indicators,andtheassumptionsorrisksthatmayinfluencesuccessand failure.Itthusfacilitatesplanning,executionandevaluationofdevelopmentinterventions. monitoringacontinuingfunctionthatusessystematiccollectionofdataonspecifiedindicatorsto providemanagementandthemainstakeholdersofaninterventionwithinformationregardingtheuseof allocatedfunds,theextentofprogress,thelikelyachievementofobjectivesandtheobstaclesthatstand inthewayofimprovedperformance(OECDDAC,2002:278). objective(projectorprogrammeobjective)theintendedphysical,financial,institutional,social, environmental,orotherresultstowhichaprojectorprogrammeisexpectedtocontribute(DAC Glossary). outcomethelikelyorachievedshorttermandmediumtermeffectsofaninterventionsoutputs. (OECDDAC,2002:28) outputstheproducts,capitalgoodsandserviceswhichresultfromaconflictpreventionand peacebuildingintervention(OECDDAC,2002:28). participatoryevaluationevaluationmethodinwhichrepresentativesofagenciesandstakeholders (includingbeneficiaries)worktogetherindesigning,carryingoutandinterpretinganevaluation(OECD DAC,2002:28). peaceanalysisanassessmentofthepeacebuildingenvironment,includingexistingpeaceefforts, actors,deescalatingfactors(reducearmedconflictortensions),andconnectors.(Paffenholzand Reychler,2007). peacebuildingactionsandpoliciesaimedatpreventingtheoutbreak,therecurrenceorcontinuation ofarmedconflict,encompassingawiderangeofpolitical,developmental,humanitarianandhuman rightsprogramsandmechanisms,includingshortandlongtermactionstailoredtoaddressthe particularneedsofsocietiesslidingintoconflictoremergingfromit(UNSecurityCouncilPresidential Statement,S/PRST/2001/5,4278thmeeting,February2001).Includeslongtermsupportto,and establishmentof,viablepoliticalandsocioeconomicandculturalinstitutionscapableofaddressingthe proximateandrootcausesofconflicts,aswellasotherinitiativesaimedatcreatingthenecessary conditionsforsustainedpeaceandstability(OECDDAC,2001:86). peacebuildingrelevanceassessmenttheaimistoassesswhethertheoveralldirectionofaplannedor ongoinginterventioncorrespondstothecountrys(conflictareas)peacebuildingneedsasmappedoutin theconflict/peaceanalysis.APBRelevanceAssessmentcomparestheobjectivesandmainactivitiesof theplannedorexistinginterventionwiththeidentifiedpeacebuildingneeds,examineshowandtowhat extenttheyareconsistentwithneeds,helpsavoidduplicatingotheractorspastandpresentactivities (PaffenholzandReychler,2007). policycoherencethesystematicpromotionofmutuallyreinforcingpolicyactionsacrossgovernment departmentsandagenciescreatingsynergiestowardsachievingtheagreedobjectives(OECDDAC,2003:2). Therearefourdimensionsofcoherence:a)consistencybetweenendsandmeansofapolicy;b)consistencyof policiesandactivitiesacrossgovernmentdepartments;c)consistencyofpoliciesandactivitiespursuedby differentactors;andd)alignmentofpolicies,activitiesandprocessesbetweenexternalactorsandconflict affectedorconflictpronecountries(Picciotto,2006:3).
58
qualityassurance(panel)qualityassuranceencompassesanyactivitythatisconcernedwithassessing andimprovingthemeritortheworthofadevelopmentinterventionoritscompliancewithgiven standards(DACGlossary). resultsbasedmanagement(RBM)aManagementstrategyfocusingonperformanceandachievement ofoutputs,outcomesandimpacts. resultstheoutput,outcomeorimpact(intendedorunintended,positiveand/ornegative)ofa developmentintervention. riskassessments/riskanalysisananalysisoranassessmentoffactors(calledassumptionsinthe logframe)affectorarelikelytoaffectthesuccessfulachievementofaninterventionsobjectives.A detailedexaminationofthepotentialunwantedandnegativeconsequencestohumanlife,health, property,ortheenvironmentposerbyanintervention;asystematicprocesstoprovideinformation regardingsuchundesirableconsequences;theprocessofquantificationoftheprobabilitiesandexpected impactsfor(DACGlossary). stakeholdersagencies,organisations,groupsorindividualswhohaveadirectorindirectinterestinthe interventionoritsevaluation(DACGlossary). steeringgroupajointgroupofmembersfromtheDACNetworkonDevelopmentEvaluationand NetworkonConflict,PeaceandDevelopmentCooperationwhohelpedguidethedevelopmentofthis guidance.Membersinclude:Norway(LeadforEvaluationNetwork),Sweden,Switzerland(leadforCPDC) andtheUnitedKingdom. summativeevaluationastudyconductedattheendofanintervention(oraphaseofthatintervention) todeterminetheextenttowhichanticipatedoutcomeswereproduced.Summativeevaluationis intendedtoprovideinformationabouttheworthoftheprogramme(DACGlossary). termsofreference(TOR)awrittendocumentpresentingthepurposeandscopeoftheevaluation,the methodstobeused,thestandardagainstwhichperformanceistobeassessedoranalysesaretobe conducted,theresourceandtimeallocated,andreportingrequirements.Twootherexpressions sometimesusedwiththesamemeaningarescopeofworkandevaluationmandate(DACGlossary). theoryofchangetheassumptionsthatlinkaprogramsinputsandactivitiestotheattainmentof desiredends.Asetofbeliefsabouthowandwhyaninitiativewillworktochangetheconflict.Itincludes bothimplementationtheoryandprogrammetheory(Weiss,1995;Church&Rogers,2006:11). theorybasedevaluationanevaluationthattrackstheanticipatedsequenceoflinkagesfrominputsand activitiestooutcomesandimpacts(Weiss,1995). triangulationtheuseofmultipletheories,methodsand/ordatasourcestoverifyandsubstantiatean assessment.Itisusedtoovercomethebiasesthatcomefromunitarydisciplines,singleobservers,self interestedinformantsandpartialmethods(OECDDAC,2002:37;Weiss,1995). validitytheextenttowhichthedatacollectionstrategiesandinstrumentsmeasurewhattheypurport tomeasure(DACGlossary).
59
ANNEX2 USEFULWEBSITESANDRESOURCES
Asmentionedpreviouslythereisnosingleblueprintfordoinganevaluationandmuchremainstobe learnedabouthowbesttoconfrontsituationsoffragilityandconflict.Manyresourcesalreadyexistthe appropriateonesshouldbedrawnuponbasedonthespecificneedsofthereaderandtheparticular evaluation.Thisannexcontainssomekeyresourcesasapointofdeparture.[Listtobeedited/addedto duringapplicationphase] Evaluationguidanceandtools OECDDACPrinciplesfortheEvaluationofDevelopmentAssistance OECDDACGuidanceforEvaluatingHumanitarianAssistanceinComplexEmergencies ALNAPEvaluatingHumanitarianAction:UsingtheOECDDACCriteria DFIDGuidelinesforMonitoringandEvaluatingInformationandCommunicationforDevelopment (ICD)Programmes UNDPHandbookonMonitoring&EvaluatingforResults,UnitedNationsDevelopment Programme(NotethatsomeofthehandbookslanguageandguidelinesarespecifictoUNDPand maynotbeapplicabletononUNDPevaluations.) OECDDACGlossaryofKeyTermsinEvaluationandResultsBasedManagement Approachestoevaluation PerformanceMonitoring&EvaluationTips:ConductingAParticipatoryEvaluation,U.S.Agency forInternationalDevelopment(USAID)(1996)http://www.dec.org/pdf_docs/pnabs539.pdf DavidFetterman(2000)FoundationsofEmpowermentEvaluation,ThousandOaks,CA:SAGE MichaelQuinnPatton(2002)UtilisationFocusedEvaluation(UFE)Checklist, http://www.wmich.edu/evalctr/checklists/ufe.pdf Rothman,J.(1997)ActionEvaluationandConflictResolutionTraining:Theory,MethodandCase Study,InternationalNegotiation,vol.2:45170. Evaluatingpeacebuilding AMeasureofPeace:PeaceandConflictImpactAssessmentofDevelopmentProjectsInConflict Zones,KennethBush(IDRCsPeacebuildingandReconstructionProgrammeInitiativePBRPI) http://web.idrc.ca/uploads/userS/10757546941Working_Paper1.doc PeaceandConflictImpactAssessment(PCIA),HandbookVersion2.2September2005,aconflict preventionandpostconflictreconstruction(CPR)networkresource. http://cpr.web.cern.ch/cpr/library/Tools/PCIA_HandbookEn_v2.2.pdf Conflictpreventionandpeacebuilding DACGuidelinesHelpingPreventViolentConflict www.oecd.org/dac/conflict/preventionguidelines OECDDACPrinciplesforGoodInternationalEngagementinFragileStatesandSituations http://www.oecd.org/dataoecd/61/45/38368714.pdf WholeofGovernmentApproachestoFragileStates http://www.oecd.org/dataoecd/15/24/37826256.pdf
60
JereondeZeevwandKrishnaKumar(eds.)(2006)PromotingDemocracyinPostconflictSocieties, LynneRiennerPublishers TheRealityofAid2006FocusonConflict,SecurityandDevelopmentCooperation Conflictsensitiveprogrammemanagement.SDCJanuary2006 http://162.23.39.120/dezaweb/ressources/resource_en_24650.pdf FEWER,InternationalAlert,Saferworld,CECORE,AfricaPeaceForum,CHA,"ConflictSensitive ApproachestoDevelopment,HumanitarianAssistanceandPeacebuildingAResourcePack" WorldBank,2002:Theconflictanalysisframework:identifyingconflictrelatedobstaclesto development. CPRNetwork.CompendiumofOperationalFrameworksforPeacebuildingatwww.cprnet.net UnitedNationsDepartmentforEconomicandSocialAffairs(UNDESA).ConflictAnalysisandEarly ResponseTrainingManual NetherlandsInstituteofInternationalRelationsClingendael.ConflictPrognosis:AConflictand PolicyAssessmentFramework AssociationoftheU.S.ArmyandCenterforStrategicandInternationalStudies.PostConflict ReconstructionTaskList CanadianInternationalDevelopmentAgency.ProgrammingforResultsinPeacebuilding ObjectivesTreeandPerformanceIndicators StiftungWissenschaftundPolitik.ConflictImpactAssessment:APracticalToolforPrioritising DevelopmentAssistanceinUnstableSituations,CPNGuide DepartmentforInternationalDevelopment.U.K.StrategicConflictAssessment:GuidanceNotes EuropeanCommission.ChecklistforRootCausesofConflict
Understandingviolenceandconflictdynamics Horizontalinequality Stewart,F,(Forthcoming,2008)HorizontalinequalityandconflictUnderstandinggroupconflictin multiethnicsocieties,PalgraveMacmillan. Naturalresourcecompetition/Greenwars/Environmentalconflict Diehl,P.F.andGleditsch,N.P.,(eds.)(2001),EnvironmentalConflict,WestviewPress:Boulderand Oxford. HomerDixon,T.F.(1999),Environment,ScarcityandViolence.PrincetonUniversity. R.Matthew,M.HalleandJ.Switzer,(eds.2002),Conservingthepeace:resources,livelihoodsand security,IUCN/IISD. Genderandconflict DACGenderTipsheetonEvaluation,https://www.oecd.org/dataoecd/2/13/1896352.pdf "GenderandDoNoHarm:Dosanddon'ts"KompetenzzentrumFriedensfrderungCenterfor Peacebuilding http://www.swisspeace.ch/typo3/fileadmin/user_upload/pdf/KOFF/genderdonoharm.pdf GendersensitiveprogrammedesignandplanninginconflictaffectedsituationsResearchReport, JudyElBushra,AshaElKaribandAngelaHadjipateras,ACORD,January2002.
61
ANNEX3 PLANNINGANDMONITORINGPROGRAMMES,PROJECTSANDPOLICIES INCONFLICTPREVENTIONANDPEACEBUILDINGCONTEXTS
Evaluationanditsrequirementsmustbeanintegralpartofaidprogrammingfromthestart. 61 In ordertopromoteandsupportgoodevaluationofconflictpreventionandpeacebuildingprogrammes, practitionersinthisfieldandevaluatorscanworktogethertobridgethegapbetweenplanning, programmedesignandevaluation.Thisgapisoftencharacterisedbybroadorinconsistentobjectives thatmaynotbewelllinkedtoanoverarchingstrategyorpolicy;alackofconflictanalysis;sketchyneeds assessmentsoftencoupledwiththeproblemoflittlebaselinedata;poorlyarticulatedtheoriesofchange andrelatedprogrammeobjectives;andweakornonexistentmonitoringsystems.Insteadofwaitingfor anevaluationtouncoverthis,programmemanagerscanstrivetoimproveplanningandprogramming, thereforeleavingscarceevaluationresourcestofocusonthemoreimportantquestionsofrelevance, effectiveness,efficiency,sustainability,andcoherence. Byfocusingonthechallenginginterconnectionsbetweenplanning,monitoringandevaluation,this annexsupportsconflictpreventionandpeacebuildingpractitionerstoimprovestandardsofprogramming andconsequentlyenhanceevaluation.
Planning
Planninginconflictpreventionandpeacebuildingisaboutidentifyingthemostrelevant contribution(s)thatdonors,practitionersandtheirorganisationscanmaketoprovidesupportinthisfield toaspecificcountry.Thisinvolvesdesigningtheinterventionorprogrammeinsuchawaythatitcan reachitsobjectiveseffectively. Conflictanalysis Oneofthefirststepsinplanningforconflictpreventionandpeacebuildingprogrammesisto conductaconflictanalysis.Thiswilldevelopanunderstandingofthecontextinwhichanyintervention maytakeplace.Anunderstandingoftheconflictthenbecomesthebasisonwhichallotheractivityis planned.Conflictanalysisisoftenusedtomakeamapofwhatastrategicresponsetoconflictwouldlook likeandshouldbeintegratedintoprogrammedesign.Astrategicresponsemapcanalsobeusedasan inputtootherplanningprocesses. Asasimplerule,conflictanalysisshouldbeundertakenwheneverstrategicplansareformedor revised.Incountrieswherethereisnoactualconflictbutincreasingtensionorahistoryofconflict,there maystillbeacaseforconflictanalysis.Thiscouldbecombinedwithotherformsofriskmapping,covering eventssuchasnaturaldisasters,asawayofidentifyingtensionsandrisks.
61
"Clearidentificationoftheobjectiveswhichanaidactivityistoachieveisanessentialprerequisiteforobjective evaluation."(OECDDAC,1992).
62
Whereconflictanalysishasbeenintegratedintostrategyandprogrammedesigntheprogrammecan beevaluatedagainstitsstatedaimsandobjectives.Unfortunately,suchanalysisisoftenlackingamongst conflictpreventionandpeacebuildingprogrammes,makingevaluationmorechallenging.Aidagencies shouldthereforeintegrateconflictintotheirplanningcyclesandoverallstrategiesratherthantreatit separately.Evenwherethisisdoneevaluatorsmayneedtoupdateortesttheanalysis,butthisis relativelyeasycomparedwithdoinganewconflictanalysis. Goalsandobjectives Thenextstepinprogrammedevelopment,followingaconflictanalysisandidentificationofkey pointsofintervention,istoarticulatetheprogrammegoalsandobjectives.Clarityandspecificityinthe goalsandobjectiveswillfacilitatesubsequentmonitoringandevaluation.Assumingthatconflictanalysis pointstoappropriateobjectives,itisimportanttothendeterminehowthesearelinkedtospecific interventionsandultimatelyoutcomes.Developingatheoryofchange,acleararticulationofthe assumptionsunderpinningthecausalpathwaysofanintervention,isonewaytoidentifywhetherthese linkagesareappropriate.(SeeAnnex6formoreinformationontheoriesofchange.)Ensuringthat theoriesofchangeareexplicitandadoptedascentraltotheplanningprocesswillfacilitateboth evaluationandmonitoring. Intheconflictpreventionfield,programmeorpolicygoalsaretoooftenverygeneralorvague, formulatedsomethinglikethis: TheprogrammewillcontributetopeacebuildingincountryXthroughpeacejournalism. Suchagoalisdifficulttoevaluate,becauseitisgeneralandlacksaclearlystatedassumptionofcausal relationships.Itisdifficulttoplanasuccessfulinterventionaroundsuchavaguegoalandtheworkwill thereforebelessfocusedandeffective.Wheregoalshavenotbeenmadeclearbymanagersorpolicy makers,theevaluatorwillhavetounpackorreconstructtheintendedorimplicitgoalsasbestshe/he can.Policymakers,programmestaffandmanagerscanworktoimproveandclarifytheirgoalsand objectives.Reformulated,thegoalmightreadasfollows: Theprogrammewillcontributetothereductionofintergrouptensionbyraisingtheawarenessof journalistsregardingtheimpactsofreportingonconflict.Itwillimprovetheirskillsregardinghowto handleissuesofstereotypesconstructively. Thisgoalismoremeasurableandobservable.Inthisgoal,theexpectedresultisclear(reduced tension)andthemeasurableorobservableoutcomesarealsostated(increasedawarenessandskill).A programmewithgoalsstatedthiswaywillbeeasiertoevaluate.
Performancemanagement:frameworksandmonitoring
Performancemanagementcanbeimprovedthroughtheuseofspecifictools,includinglogical frameworks,resultsbasedmanagementtechniques,andcoherentmonitoringprocesses. Frameworks Logicalframeworks(logframe)areinstrumentsdesignedtofacilitatethedesign,implementation andmonitoringofdiscretepoliciesandprojects.Alogframecanhelparticulateobjectivesandgoals.It canidentifytherelationshipsbetweeninputs,outputs,outcomesandimpacts.Itcanalsoensurethat
63
indicatorsaretightandrobust,whilesignallingpotentialrisksandwaystomitigatethem.Thereisarisk thatlogframesareusedaschecklists,butaslongastheyaretreatedasprogrammingtools,theyoffer certainadvantagesoverotherprogrammemanagementtechniques.Theycanforexamplehelp systematisethoughtsaroundassumptionsandhelpmaketherationalefortheprogrammetransparent. Resultsbasedmanagement(RBM)isanapproachtomanagementthatintegratesstrategies, stakeholders,resourcesandmeasurementsmetricstoimprovedecisionmaking,transparency,and accountability.RBMessentiallyrequiresthatplannersandimplementerscarefullyarticulateperformance andoutcomeindicators.Theapproachisfrequentlylinkedtotheuseoflogicalframeworks.Insome cases,aresultschain,togetherwithalogframe,maybeadoptedtoallowforlongitudinalmeasurement. Thiscanbedepictedasaflowchart(seeFigure3.1)andillustratesthecausalsequenceforan interventionthatbeginswithinputsandactivities,movingthroughprocesses,outputs,outcomesand impacts,thatmaybeexpectedtooccuroveraperiodoftime.Figure3.1depictsasampleforworkin securitysystemreform,afastgrowingareaofconflictpreventionandpeacebuildingactivities.
Figure2.
Sampleresultschainforsecuritysystemreform
Inputs Measurethe resourcesa programmeuses: Personnel Equipment Funds Processes/Activities Outputs Measurestheactivitiesa Measurestheresultof programmeundertakes: theactivities: Policetrained Publiceducation Trainingprogrammes Paralegalsupport established Institutionaldialogue Courtuser committees established Outcomes Measuresthe consequencesofthe programme: Improvedpolice performance Availablelegal advice Reducedprison overcrowding Impacts Theultimate achievementforthe widercommunity: Moreeffectiveand responsivesecurity systemthatdelivers justiceandsafetyto localpeople
Source:adaptedfortheOECDDACManualonSecuritySystemReform:SupportingSecurityandJusticefrom,TheMonitoringandEvaluation Framework,EvaluationOffice,UNDP,2002.
Monitoring Monitoringisattheheartofgoodperformancemanagement.Monitoringprocessesrevealhowa programmeisprogressingandcannotonlyencouragetherecalibrationofprogrammesasconflict contextsshift,theycanalsomakeanevaluationeasiertoconductandmoreindepth.Measuringthe performanceofconflictpreventionandpeacebuildingactivitiesiscentraltoimprovingpracticeandto donoraccountability.Unlessinterventionsarehavingademonstrableinfluenceonclearlydefined indicatorsofconflictpreventionandpeace,theymaybeperceivedaslackingcredibilityandultimately legitimacyamongststakeholders.Monitoringshould,therefore,bedevelopedasacentralpillarofany programmaticorprojectlevelinvestment.Monitoringinvolvestraining,informationcollectionand management,analysisandcommunication. Itisimportanttodifferentiatebetweenmonitoring,whichisanongoingsubjectiveinternalprocess ofcollectinginformationinordertoprovidefeedbackonanactivitysprogress,andevaluation,whichis typicallyanexternalandindependentrevieworassessmentofanongoingorcompletedactivity.See Table3.1foranoutlineofthedifferences.
64

Table3.
Evaluationvs.Monitoring
Monitoring
ContinuousorPeriodic Programmeobjectivestakenasgiven Predefinedindicatorsofprogressassumedtobe appropriate Tracksprogressagainstasmallnumberofpredefined indicators Focusonintendedresults Quantitativeandqualitativemethods Dataroutinelycollected Doesnotanswercausalquestions Usuallyaninternalmanagementfunction
Source:SIDA,(2007)www.sida.se.
Evaluation
Episodic,adhoc Programmeobjectivesassessedinrelationtohigher levelgoalsortothedevelopmentproblemtobesolved. Validityandrelevanceofpredefinedindicatorsopento question Dealswithawiderangeofissues Identifiesbothunintendedandintendedresults Qualitativeandquantitativemethods Multiplesourcesofdata Providesanswerstocausalquestions Oftendonebyexternalevaluatorsandofteninitiatedby externalagents
Thereareahostofchallengestodevelopingeffectivemonitoringmethodsincrisis,conflictand postconflictcontexts.First,thereareintrinsicdifficultiesinestablishingcausalityincomplexanddynamic opensystems,i.e.whetheroutcomeYcanbeattributedtoinvestmentX.Somehavedescribedthisasthe attributiongapandobservedhowahostoffactorscaninfluenceoutcomesandimpactsofspecific interventionsinconflictcontexts.Second,thereisfrequentlyashortageofreliablesurveillanceand surveydataand/orlocalcapacitytoundertakerobustlongitudinalassessments.Third,thereisoftenlittle internationalordomesticwilltoinvestinadditionalproceduresforestablishingcausalrelationshipsin contextswhereneedandthedynamicsbetweendonors/recipientsdriveinterventions.Assuch,thereis frequentlyareluctanceinsomecasesevenresistanceamongpolicymakersandpractitionersto demonstratequantifiableimpacts.62 Donorsandpractitionerscanmitigatetheseconstraintsbyrecognisingthatmanyriskfactorsmay affectprogramme/projectperformanceaseffectivenesscanbemeasuredifotherfactorsareconsidered. Itisequallycrucialthatdonorsintroduceincentivesformonitoring.Donorsshouldaimtosupport
62
Indicatorscanalsobemisusedandshouldbehandledwithcare,asseenintheexplosionofgovernanceindicatorswhereeven themostcarefullyconstructedoftheselacktransparencyandcomparabilityovertime,sufferfromselectionbias,andarenot wellsuitedtohelpdevelopingcountriesimprovethequalityoflocalgovernance.See:Arndt,C.andC.Oman(2006),Usesand AbusesofGovernanceIndicators,OECD.
65
stakeholderownershipofmonitoringprocessesthroughadequatefinancing,localcapacitydevelopment andtheintroductionofchecksandbalancestoensurethatdataqualityandanalysisisofahighstandard. Goodmonitoringshouldbebaseduponthebaselinedatacollectedduringthedesignphase,whichis thencontinuouslyupdated.Itmakesmostsensetomonitoratoutcomelevels,asthisallowsforassessing theinterventionsrelationshiptoconflictand/orpeace.Forexample,apeacejournalismtrainingproject mightreporttheamountoftrainingcoursesthathavetakenplaceandthenassumethatthistrainingwill leadtochangesinreporting(assumedimpact).Goodmonitoringmethodsshouldratherassessthe outcomesofthetraining,e.g.havethejournaliststrainedactuallychangedtheirreportingstyle?Dothey forexampleusefewerstereotypesabouttheconflictingactors,etc.?Datacollectionforsuchmonitoring (suchasassessmentofarticles,radioorTVnews)wouldhavetobepartoftheprojectdesignfromthe verybeginning. Thereisnouniversalmonitoringtemplateorsetofgenericindicatorsthatappliestoallconflict preventionandpeacebuildingcontexts.Rather,formshouldfollowfunctionandbetailoredtolocal dynamics.Theminimumparametersofcreatingamonitoringprocessarestraightforward.Forexample, themonitoringprocessrequiresthearticulationofrealisticindicatorsofperformancethatrationally followfromwellarticulatedprogramme/projectobjectives.Theseindicatorscanbeetic,derived externally(e.g.numberofconsultationsbetweenprimarystakeholders;reductionsinhomicideand incidentsofarmedviolence;increasedaccesstojusticeandpolicinginstitutions;increasedparticipation insocialnetworksandassociations),and/oremid,determinedinternally(e.g.qualityofconsultations betweenprimarystakeholders;changedperceptionsofsecurityinareaswheretheprojectwas undertaken;improvedconfidenceinspecificinstitutions/services).Inidealcases,indicatorsare developedthroughparticipatoryprocesses,includingfocusgroupsandconsultativemeetings,withlocal stakeholdersandbeneficiaries.Theprocessofindicatordevelopmentiscentraltothelegitimacyof monitoringandevaluationprocesses.Regardlessofhowtheyaredetermined,indicatorsmustbeSMART (specific,measurable,attainable,relevantandtimebound)63 . Usingmultiplemonitoringmethodsmaybeespeciallyusefulinconflictzoneswherecertainsources ofdataarenotavailableorhavequestionableaccuracy.Monitoringstrategiesshouldbegroundedin localcapacitiesandtheconflictcontext.Forinstance,theinstrumentsintroducedtomonitorspecific interventionsinacomparativelydatarichcountrysuchasSouthAfricamaybeverydifferentthanthose proposedinAfghanistanorSomalia.Monitoringcanpassivelydrawfromexistingnationalandmunicipal surveillancesystems(e.g.publichealthstatistics,judicial/policerecords,mortuarydata,education curriculasurveillance),fromperiodiccrosssectionhouseholdsurveysand/orfromspecificstudies undertakenbyacademicandpractitionerorganisations.Monitoringcanalsoinvolvemoreproactive methodsrangingfromrobustlongitudinalsurveillanceofseculartrendsthroughspeciallydesigned surveillanceandsurveysormedia/incidentmonitoringmechanisms,toregularsitevisits;smallscale purposivepanelsurveys;participantobservationandparticipatoryassessmentsundertakenwithlocal partnersandprojectbeneficiaries.Akeycriterionforeffectivemonitoringisthatitisachievableand linkedtoadequatetrainingandinvestment.
63
DACEvaluationQualityStandards(OECDDAC,2006).
66
Table4.
Asampleofimpactindicatorsandmethods
IndicatorType Changesin intentionalmortality Changesinintentional mortalityand morbidity Changesinhuman rights violations MinimumDataNeeds Fatalinjuryprofile. Fatalandnonfatalinjury profiles. Objectiveindicatorsof victimisation,kidnapping, disappearance,detention, displacement,andsexual andgenderbasedviolence. Qualitativeindicatorsof insecurityandlocallydefined indicatorsofvictimisation, forexampleinsecurity providers. Qualitativeandquantitative indicatorsofsocioeconomic variables,including associationmembership, networksandlocal exchanges. Quantitativeand representativeindicatorsof socioeconomicimpacts, armedviolenceand victimisation. DataSourceandMethods Morgues,Parishes,andINGOs.Monthlyreporting fromprimarysources. ReferralHospitals,Clinicsandhealthposts.Sentinel surveillanceandadministeredweekly/monthly. Humanrights/gendermonitoringmechanisms, internationalandnationalhumanrightsreports,police registrationdata.Archivalreviewadministeredona weeklyormonthlybasis. Smallrepresentativesamples(women,men,youth, gangmembers,communityleaders,etc)andweeklyor monthlyfocusgroupsadministeredbytrained practitionersorcommunityinterlocutors. Smallcluster/panelsurveysofadefinedpopulation group(e.g.beneficiaries,hostcommunities,control group)incatchmentareas.Cohortpanelsurveysona quarterlybasis. Reliability/ Validity Deductive/ Low Deductive/ Medium Deductive/ Medium
Changesinrealand perceived security
Inductive/ Low
Changesinsocial capitalformation
Inductive/ Medium
Changesinhousehold andcommunity victimisation
Largescalesimplerandom/clustersurvey(confidence intervalneedstobedecided)toassessimpacts. Epidemiologicalteamcouldbedevelopedtosupport theprocessandcarryoutpre/postsurveysinurban andruralcentres.
Inductive/ High
Insummary,arangeofconditionsforreliableandcomprehensivemonitoringwillcontributeto successfulevaluatingintheconflictpreventionandpeacebuildingcontext.Whenaconflictprevention andpeacebuildinginterventionhasbeenongoingforsometime,anevaluationshouldbeplannedto triangulatethemonitoringfindingsandtoaskbroaderrelevanceandstrategyquestions.
67
ANNEX4 CONFLICTANALYSIS
Introduction InSection3,thisguidancesuggeststheuseofconflictanalysisinevaluatingconflictpreventionand peacebuildingprogrammesandpolicies.Conflictanalysishelpstoidentifywhatisneededtoaddressthe conflictandtounderstandthecontextinwhichaninterventionwillbeimplemented.Assuch,many practitionerswillalreadybefamiliarwiththeuseofthistoolinthedesignofprojectsandprogrammes. Themethodologyisequallyrelevantforevaluationteams,whoneedtounderstandthecausesandthe contextofconflictandpeaceinordertoevaluatetherelevance,effectivenessandimpactsofthe programmeorpolicy.Thisannexfurtherexplainstheroleofconflictanalysisinthecontextofevaluation. Avarietyofconflictanalysisframeworksareavailabletopractitionersandevaluators.While differentinapproachandcoverage,mostoftheseframeworksbringtheuserthroughthesamestepsof identifying:theprimarycausesofconflictandpeaceatvariouslevels;keystakeholders(actorsand groups)whoareaffectedbyorinfluenceontheevolutionoftheconflict;thecontextinwhichconflict andpeacebuildingistakingplace(political,economic,socialetc) 64 ;andanassessmentofthedynamicsof theconflict,howitmightevolveintothefutureandwhatopportunitiesexistforinterruptingescalation. Thereareanumberofdifferentapproachesandtoolsavailabletoanalysingtheseelements,which arenotmutuallyexclusive;theyareoftenusedincombinationwitheachother.Thechoiceofapproach willdependonthepurposeofevaluation,thelevelatwhichtheprogrammeorpolicyisbeing implemented,andthekindsofimpactssoughtoranticipated. Conductingorreviewingaconflictanalysisforanevaluation Evaluationteamsareprimarilyconcernedwithconflictanalysisfromtwoperspectives.First,in assessingrelevanceitwillbeimportanttounderstandwhetherandhowaprogrammeimplementation orpolicydevelopmentgroupdevelopedtheirunderstandingoftheconflictandcontext.Inotherwords, whatwasthebasisfortheirdeterminationofprioritiesatthepolicylevelorprogrammedirections. Second,inordertoassesstheimpactsofpoliciesorprogrammes,theevaluationteamneedsto understandtheconflictthatprogrammesandpoliciesareattemptingtoinfluenceorchange.An evaluationteamthusneedstounderstandthedifferentapproachesto,andtoolsfor,conflictanalysisto beabletoreviewtheadequacyoftheanalysisperformedorconductitsownanalysisifonedoesnot exist. Checklistforreviewingaconflictanalysis Ifaconflictanalysishasbeendoneaspartoftheprogrammedesign,theevaluationteamwillneed toreviewtheanalysisandassessitsqualityandappropriateness(relevance)attheoutsetofthe programmeandhowitwasadapted(ornot)overtime.Theymayfind,forexample,thattheoriginal
64
See"Conflictsensitiveapproachestodevelopment,humanitarianassistanceandpeacebuilding:Aresourcepack," availableathttp://conflictsensitivity.org.
68
analysiswascorrectandobjectivesandstrategiesa'goodfit'atthattime,butchangesintheconflict haverenderedtheoriginalapproachirrelevantorinappropriate.Theevaluationteamwillalsoneeda currentanalysisagainstwhichtoassesstheoriginalanalysisforwantedimpact.Inthisprocess,the evaluationteamshouldposethefollowingquestions: 1. Giventheresourcesandcapacitiesoftheagencyororganisationbeingevaluated,wasthe appropriateconflictanalysistoolchosentoguidethedesignandimplementationofthe programme(s)orpolicy(ies)?Didthetoolgenerateadequateinformationtodeterminetherelevance oftheinterventiontotheneedsofthepeacebuildingprocess;totheeffectivenessoftheprogramme designsandimplementation;andtoassesstheappropriatenessofthetheoryofchange? Istheconflictanalysisuptodate?Doesitcapturetheevolutionoftheconflictfromthetimethe programmeorpolicywasinitiated?(Ifnot,theevaluationteammayneedtoupdatetheanalysis.) Wastheprocessofconflictanalysisappropriateandeffective? a. Wastheanalysisconductedbyskilledpeoplewithanunderstandingofconflictandofthecontext andrelatedhistory? b. Didtheanalysisgatherinformationfromawiderangeofsources?Diditincludeperspectives fromallthemainstakeholdersintheconflict? c. Wastheanalysisconductedinaconflictsensitivemanner?Forexample,diditavoidexacerbating divisionsbythewayquestionswereasked?Iftheanalysiswasconductedbyconvening stakeholderworkshops,didthefacilitatorspossess,orlack,sufficientskillstoengageconflicting partiesinaproductivediscussion?Didtheanalysisprocessputresearchers(andlocalpartners)at riskbysendingthemtoinsecureareas?Diditputintervieweesatriskbyexposingthemto retaliation? 4. Wastheanalysisdoneattheappropriatelevel?Forexample,ifaprogrammeistobeinitiatedatthe provinciallevel,wasanationalanalysissupplementedbyananalysisofconflictdynamicswithinthe province? Weretheconclusionsreasonable?Werecriticalelementsmissingfromtheanalysis?Towhatdegree wastheanalysisshapedbytheexpertiseoftheagencyortheirgeneralbeliefsabouthowtobring aboutpositivechange? Wastheanalysislinkedtostrategy?
2.
3.
5.
6.
Checklistforchoosingtheappropriatetoolforconflictanalysis65 Iftheproject,programmeorpolicybeingevaluateddidnotconductaconflictanalysisinthedesign phase,oriftheanalysisisnotuptodate,theevaluationteammayneedtoundertakeoneitself.The levelofeffortandresourcesdedicatedtotheanalysisshouldbescaledtothescopeoftheprogrammeor policy.Theanalysiscouldrangefrombuildingintobeneficiaryandstakeholderinterviewsafewquestions abouttheconflict,toconductingadeskstudyandlimitedsetofadditionalinterviewsinthefield,to undertakinganextensiveprocessusingoneofthecomprehensivetoolsdevelopedbybilateraland
65
InternationalAlertwww.conflictsensitivity.org
69
multilateraldonors(seeTable4.1).Evaluatorsmightconsiderafewquestionsindecidingwhattoolor combinationoftoolstouse: 1.Purpose Doesthetoolprovidesufficientinformationoncauses,actors,dynamicsandthecontextof conflictandpeacetoassesstherelevanceoftheprogrammeorpolicytotheneedsofthe peacebuildingprocess? Doesthetoolprovideinformationontheappropriateissueareas,attheappropriateleveland depth,tohelpevaluatetheeffectivenessandimpactsoftheprogrammeorpolicy? 2.Assumptions Dotheevaluatorssharetheassumptionsaboutconflictunderlyingthetool?Isthetools understandingorassumptionaboutthenatureofconflictappropriateforthespecificcontextin whichtheprogrammeorpolicyisbeingimplemented? Doesthisperspectivecorrespondtothemandateandvaluesoftheorganisationbeingevaluated? 3.Methodologyandresourceimplications Doesthetoolsproposedmethodologymatchthepurposeoftheanalysis? Doesthetoolsproposedmethodologyagreewiththewaysofworkingoftheevaluationteam? Doestheevaluationteamhavethecapacity(skills,expertise,access,etc.)tousethetoolwell? Howlongdoesittaketoproduceareliableconflictanalysis? Whataretheresourceimplicationsoftheselectedtool(stafftime,travel,seminarcosts, facilities,datamanagement)? Istheevaluationteamabletoallocateorsecuretherequiredresources? Arangeofresourcesforconflictanalysis: Belowisanoutlineofanumberofconflictanalysistoolsdevelopedandusedbydonors;governmentand multilateralagencies;andNGOs.Itisnotanexhaustivelist,butisintendedtoprovidearepresentative sampleofthevarietyoftoolsdevelopedtofacilitatedesignofprogrammeswithdifferentgoalsvisvis conflictpreventionandpeacebuilding,aswellasofarangeofapproachestoanalysis.Foreachtool,the table summarises the main field of activity for which the tool is relevant: Development (DEV), HA (humanitarianassistance),PB(peacebuilding),FP(foreignpolicy),aswellasthepurpose,potentialusers, assumptions, methodology and resource requirements. The far right column describes how/when the toolisbeingappliedtoevaluation.
70
Table5.
Summaryofselectedconflictanalysistools Purpose Potential users Assumptions Methodologyand effort Evaluation application

Comprehensive; coversbothconflict sensitivity, preventionand peacebuilding Goodfornational level;maynotbe sufficientfor sectoralmicrolevel projects Shortlifespan
1.StrategicConflictAssessment(SCA)DFID DEV http://www.dfid.gov.uk/Pubs/files/conflictassessmentguidance.pdf Assesseffectsof DFIDand Combinepolitical Combinationofdesk partnerbilateral andeconomic studyandfield conflicton andmultilateral dimensions; consultations programme,of agencies'desk greed/grievance; Assessmentteam programmeon officers structuresand (5people). conflict,and actors Consultationmeetings identify incountry opportunitiesto 6weeksto2months, contributeto withminimum2weeks peacebuilding forfieldresearch
2.ConflictAnalysisFramework(CAF)WorldBank DEV http://siteresources.worldbank.org/INTCPR/2145741112883508044/20657757/CAFApril2005.pdf Countrystrategic Multilateral Povertyand Checklistofrisks Primarilyusefulfor conflictinterlinked. followedbyanalysisof conflictsensitivity, planningtoensure organisations' deskstaffand Development specificvariables withemphasison thatPoverty assistancecanhelp FullCAFanalysis weakeningcausesof ReductionStrategy planners countriesbecome resourceintensive conflict,actors Papersdonot moreresilientto butcanbesimplified pursuingconflict exacerbateconflict violentconflict andopportunities forviolence 3.ConflictAssessmentFrameworkUSAID DEV http://rmportal.net/tools/conflictassessmentandmanagementtools/higherlevel_conflictassmt/view Countryand Donordesk Pullstogetherbest Combinationofdesk Relevanttoboth officers, researchoncauses, study,incountry conflictsensitivity, programme implementing levelandnatureof visits,workshopsand preventionand strategicplanning partners, conflicttoidentify interviews. peacebuilding toidentifyand windowsof Includessignificant Qualitymayvary prioritisecausesof missionstaff, embassystaff, opportunity stafftimeTakes dependingon conflictbasedon other about2months robustnessof understandingof government methodologyused impact officials togatherdata 4.ConflictrelatedDevelopmentAnalysisUNDP DEVwww.undp.org Conflictrelated Development Conflictcausedby Datacollectionand agencystaffand combinationof analysisfollowedby programme donorsworking security,political, workshoporexpert planningand inconflictprone economicandsocial studytoanalyse reviewaimedto andaffected causesandactor currentresponsesand understand situations interests suggestwaysforward linkagesbetween Developmentcan Effortdependson developmentand causeviolence methodfordata conflict,increasing collection positiveimpactof development efforts. Development focusedandlinked toprogramming Usefulatcountry orsectorlevel,less atmicrolevel Qualityofanalysis dependsonrigorof datacollection
71
5.ManualforConflictAnalysisSIDADEV/HA http://www.sida.se/sida/jsp Country/ Development Conflictsdrivenby Deskstudy, Focusondifferent agencystaff, structuralinstability, consultationsand levelsof programme/ projectplanningto implementing struggleforpower workshoptoconsider programming. partners andinfluence,and programme Relevantbothfor improve mutualfearand implications conflictsensitivity effectivenessof insecurity Localownershipof andplanningat development analysisimportant countryandsector cooperationand 612weeks,pending levels humanitarian scopeofdeskstudy Nomethodology assistanceinareas affectedbyviolent conflict 6.AidforPeacePaffenholz/ReychlerDEV/HA Paffenholz,T.andReychler,L.(2007). Assesspeaceand Development Examinesboth Deskstudy/surveyof Addressesboth conflictandpeace otherinterventions; conflictsensitivity conflictrelevance, andforeign fieldmissionwith35 andpeaceand risksandeffectsof ministryofficials factors Frameworkfor daytrainingand conflict developmentand analysisof workshop programming humanitarian peacebuilding Potentiallytime Providesspecific projectsor deficienciesand consumingandcostly, guidanceon programmes needs,conflictrisks pendingtimefor integratingpeace andeffectsof baselinestudyand andconflictlens interventionon mappingandnumber intoevaluation conflict offieldvisitsand workshops 7.DoNoHarm/Localcapacitiesforpeaceproject CDA CollaborativeLearningProjectsDEV/HA Anderson,M.(1999)andwww.cdainc.com Improvedesign Donor,NGO Focusondividers Workshop, Usefulforconflict (international andconnectorsin integrationinto and sensitivity;not conflict,followedby standardprocedures implementationof andlocal)staff sufficientforconflict analysishow Effortlimitedto conflictsensitive preventionand programme workshop projects/ peacebuilding componentsaffect programmes Rapidresults them throughimpact Potentiallygood assessmentand formicroconflict lessonslearnedfor analysis projectredesign 8.Workingwithconflict:skillsandstrategiesforaction Respondingtoconflict PB RespondingtoConflict,1046BristolRoad,SellyOak,Birmingham,B296LJ,UK.www.respond.org Conflictanalysis Localand Analysisnot Collectionoftoolsfor Individualand objectiveaimedto participatoryconflict focusingonconflict international flexibletoolswith NGOstaff,field improve analysis,toimprove transformationto wideapplicability andHQ understandingof understandingof improve Mainlyforproject conflictrealityon conflictanddifferent programme andlocallevels,not ongoingbasis perspectives planning, forstrategic Emphasises Effortdependingon implementation planning perceptionsof format(workshop, andimpact partiesand meetingsetc.) monitoring structuralfactors Requiressignificant facilitationandconflict resolutionskills
72
9.MakingSenseofTurbulentContextsAnalysistoolsforhumanitarianactors WorldVisionDEV/HA ContactWorldVisionInternational,PeacebuildingandReconciliation,www.wvi.org Aimstoimprove NGOemergency Focusonchronic Collectionoftoolsto Focusesonmacro abilitytoanalyse response, politicalinstability, analyseactors, level;howconflict dynamicsof development notjustviolent symptomsand willaffect conflict politicaleconomyof programmein conflictstoimpact andadvocacy staff Seesconflictas conflict,generate future programmeand cyclicalwithperiods futurescenarios,and Flexibleand projectplanning ofpeacefollowedby analysestrategicand adaptabletospecific andadvocacyin conflict operational contexts emergency implications Canbeusedfor situations Effortpendingon analysisofclusters scopeofdata ofcountries collectionand workshop 10.Conflictanalysisandresponsedefinition FEWER PB http://fewerinternational.org/images/lib/160_6.pdf(forWANEPtrainingmoduleonframework) http://www.reliefweb.int/library/documents/studman2.pdf Earlywarning, Diplomats, Focusonconflict Ongoing Comprehensive, countrystrategic donordesk dynamicsand participatoryanalysis mainlycountry planning officers,NGOs identificationof bylocalcivilsociety level,butcanbe overalltrends andgovernment adaptedtolocal organisations communalconflicts Effortmodestfor Qualitylargely deskstudy;morefor dependingnatureof trainingorworkshops participation 11.SystemsAnalysisDEV/PB ForfurtherinformationcontactCDACollaborativeLearningProjects,www.cdainc.com orBerghofResearchCenter,www.berghofcenter.org Strategicplanning Donorand Viewsconflictasa Identifiesfactorsfor Usefulformacro foreignministry collectionofparts conflictandpeace, andmesoanalysis forprogrammes staff;NGOstaff thatinteract actorsandtriggers; aimedtodetermine andprojectsto togetherand throughmapping relevanceand identifydriving influenceeachother interactionsoffactors impactforstrategic factorsofconflict Conflictdynamics 13dayseffort, planning andpeaceand affectedto usuallyinaworkshop Usefulforjoint interrelations stakeholders setting,toproduce analysis/evaluation amongthem understandingsof systemsmap; Notsufficientto conflict additionaltimeto guideproject gatherinformation implementationor aboutcausesand activitydesign actors 12.ECChecklistforrootcausesofconflictEuropeanCommission DEV http://ec.europa.eu/external_relations/cfsp/cpcm/cp/list.htm EnsureEUpolicies Multi and Focuson Checklistbasedon Mainlyfor contributeto bilateraldonor presence/absenceof existingknowledge; monitoringand conflictprevention deskofficers, elementsofliberal externalresearch earlywarning throughearly diplomatic democraticstate+ capacity Proposesa warningand actors identitygoup Limitedeffortas generalisedlistof awarenessraising relationsand mainlydeskbased structuralroot inEUdecision social/regional causes;doesnottie forumsof inequalities factorstoconflictin
73
problemsof countrieswithhigh assessedriskof conflict 13.ConflictandPolicyAssessmentFramework(CPAF) ClingendaelInstitute DEV/F http://www.clingendael.nl/publications/2000/20000602_cru_paper_vandegoor.pdf Aimstolinkearly Donorand Focusonindicators Externalresearch embassystaff ofinternalconflict andanalysistotrack warningtopolicy involvedwith andstatefailure indicatorsandidentify planningand foreignpolicy UsesFundfor problemareasand implementation anddevelop Peacesmeasures aspectsforresponse mentissues forsustainable fordiscussionin securityasgoal workshopsetting Effortdependson sizeofworkshops,and involvementof externalconsultants
particular countries/regions
Notprogamme specific,butfocuses onbroadpolicyor programme development Facilitatesclarity ondevelopments andtrends,not causes
Purpose

Potential users
Assumptions
Methodologyand effort
Evaluation application
74
ANNEX5 CONFLICTSENSITIVITY
Theprinciplesofconflictsensitivity,adoptedbytheOECDin2001,assertthatinternational assistancemust,ataminimum,avoidnegativeeffectsonconflict'donoharm'and,wherepossible, makeapositivecontributiontoconflictpreventionandpeacebuilding.Box5.1illustratesanexampleof theunintendedharmthatcanbecausedwheninterventionsarepoorlyplannedorimplementedin relationtotheconflict.OneofthemorewidelyusedtoolsforconflictsensitivityistheDoNoHarm Framework(seeAnderson,1997below),whichisparticularlyusefulindrawingattentiontothe unintendedconsequencesofaidplanningandpractice.Althoughitwasoriginallydevelopedfor humanitarianaiditisalsoregularlyappliedtodevelopmentandpeacebuildinginterventions. Beingconflictsensitiveandevaluatingtheconflictsensitivityofthetarget,aretwoimportant dimensionsofevaluatingconflictpreventionandpeacebuildingwork.Thoseplanningandimplementing policiesorprogrammesshouldconsiderwaystomaketheirworkconflictsensitive.Aclearandcritical assessmentofanactivityorpolicy'simpactswillcoverbothintendedandunintendedconsequencesand willthereforeprovideinsightsonthesensitivityofthetarget.Evaluatorscanhelpassesswhetherornot thisstandardhasbeenachievedaswellasprovideinsightsonhowtomakeinterventionsmoreconflict sensitive.AsmentionedinSections3.2and3.3,thoseinvolvedincommissioningandconducting evaluationitselfmustalsobeconsciousofpossibleimpactsontheconflict(andviceversa)andshould thereforeconsiderdonoharmandotherconflictsensitivityprincipleswhenplanningandconductingan evaluation.Thefollowinglistofresources,thoughbynomeansexhaustive,providesthereaderwith someinitialtoolsforunderstandingthebasicprinciplesofconflictsensitiveengagement. ResourcesforConflictSensitivity:[moreresourcestobeadded] DACGuidelinesHelpingPreventViolentConflict www.oecd.org/dac/conflict/preventionguidelines DoNoHarm:HowAidCanSupportPeaceorWarbyMaryB.Anderson(1997),LynneRienner Publishers,Boulder,London, InternationalAlertConflictSensitivityhttp://conflictsensitivity.org/ "ConflictSensitiveProgrammeManagementintheInternationalCooperation."SDC(2005) http://www.sdc.admin.ch/ressources/resource_en_24650.pdf "PeaceandConflictSensitivityinInternationalCooperation.AnIntroductoryOverview" ThaniaPaffenholzInternationalePolitikundGasellschaft,2005 InternationalFederationofRedCrossandRedCrescentSocieties2003:Aid:supportingor underminingrecovery?LessonsfromtheBetterProgrammingInitiative. TowardaConflictSensitivePovertyReductionStrategyWorldBank,2005
75
Box15.
DoNoHarm:AnExamplefromTajikistan
AttheendofthecivilwarinTajikistan,oneinternationalNGOundertookmassivehousing reconstructioninasouthernprovince.Theintentoftheeffortwas:i)toencouragepeoplewhohad beendisplacedduringthefightingtoreturntotheregion;andii)tosupportreconciliationbetween thetwogroupswhohadfoughtbygettingthemtoworktogetherinrebuildingthedestroyed villages.Priorityforreconstructionwenttothevillagesthathadsufferedthemostdamage.In these,theNGOworkedwithlocalpeopletodecidewhichhouseswouldberebuiltandtoorganize workcrewstodotheconstruction.Theyagreedthatanyonefromthevillagewhowantedajob wouldbehiredinthesecrews. Afewmonthslater,theyhadsuccessfullysponsoredthereconstructionofalmost60percent ofthedamagedhousingintheregion.However,onedayalocalmancameintotheNGOcompound withaKalashnikovandthreatenedthestaff,saying,Whyareyoufavouringthatgroupthatwe defeatedinthewar?Ifyoudon'tstartbuildingsomehousesformyclan,Iwillkillyou.TheNGO staffmemberswereastounded.Theyhadmeanttobecompletelyinclusiveandtoensurethat everyonewhosufferedintheconflictreceivedequalattention.Whattheyhadnotknown,untilthis moment,wasthata)duringtheconflict,thegreatestdamagewasdoneinvillagesoccupiedbyonly one(ratherthanboth)ofthelocal,warringgroups.Byfocusingtheirassistanceontheareasof greatestdamage,andbyhiringpeopletoworkontheconstructionwhocamefromthosevillages, theyhadinadvertentlyprovidedalmostalloftheirassistancetoonesideoftheconflict(andthe losingsideatthat).Theirprojectdesignhadunintentionallyreinforcedexistingintergroup divisionsbyfocusingonvillagesthatweremonoethnicandprovidingalltheirsupporttothese groups. Withaprojectredesign,theNGOwasabletosupplybuildingmaterialsandsupporttomulti ethnicvillages,todamagedhomesoftheotherethnicity,andtocommunitybuildingsthatboth groupssharedsuchasschools,clinicsandmosques.
Source:Anderson,1997;andCDACollaborativeforDevelopmentProjects,2000.
76
ANNEX6 UNDERSTANDINGANDEVALUATINGTHEORIESOFCHANGE
Whataretheoriesofchange? Aidworkinrelationtoconflictandpeaceisoftenbasedonapproachesandtacticsthatarerooted inimplicittheoriesofchange. 66 Inmanycasessuchtheoriesaresubconsciousandunstated.Theyare embeddedintheskillsandapproachesofindividualpractitionersandpeacebuildingorganisations,their capacitiesandtechnologies,attachmentstofavouritemethodologies,andtheperspectivestheybring tothepeacebuildingprocess. Inthecaseofanantibiasprogrammeforjournalists,thequestionwouldbehowtheplanned workshops,consciousnessraising,andskilldevelopmentwillactuallychangeconflictreporting.The programmecouldtrackthelanguageusedinreportingbeforetheireffortandafter,andmightalso surveypublicattitudes.Atthesametime,theprogrammecouldseewhethertheiractivitieswere achievingtheexpectedresultsorifunexpectedobstaclesappeared.Forinstance,itmightturnoutthat individualjournalistshaveverylittleinfluenceovertheuseofinflammatorylanguage.Instead,itmightbe discoveredthateditorsandownersdeterminetheuseofcolourful:languagetoboostsales,suggesting thatthetheoryaboutinducingchangeinreportingbytrainingjournalistswasflawed. Onetaskrelatedtothisistoidentifythesourcesofthetheories.Aretheya)experiencebased(i.e. fromtheprogrammedesignersownpersonalandprofessionalexperience,orbasedontheexperienceof thestakeholdersandbeneficiariesconsultedduringtheprogrammedesignperiod)orb)researchbased? Evaluationcancontributetoimprovedesignandimplementationofongoingprogrammes,andcan uncoverwhetherlackofsuccess,orsuccess,isduetoprogrammedesignandprogrammetheory,or programmeimplementation. Ausefulfirststepinenhancingstrategiesinconflictpreventionandpeacebuildingprogrammingand evaluationistobecomemoreexplicitaboutunderlyingassumptionsabouthowchangecomesabout thatis,theoriesofhowtoachievepeace.Conflictpreventionandpeacebuildingactivitiesarecarriedout basedonspecificideasandgoalsforwhattheyhopetoachieve.Suchdecisionsarebasedonanumberof factorsincludingassumptionsabouthowtobringaboutpeaceandtheoriesabouthowtobringabout change.Peacepractitionersselectmethods,approachesandtacticsthatarerootedinarangeof theoriesofhowpeacecanbeachievedinaspecificcontext.Itisimportanttodisclosethesetheories ofchange,bothtotestthetheoriesagainsttherealitiesoftheconflictandtoprovidethebasisfor evaluatingprogresstowardsrelatedobjectives.
66
Aprogrammetheoryisanexplicittheoryormodelofhowaprogrammeisintendedtoproducetheintendedoutputs, outcomesandimpacts,andthefactorsaffectingordeterminingitssuccess.Aprogrammetheoryoftencombineatheoryof changeandanimplementationmodel.Atheoryofchangemodeldescribesthelinkagebetweenprojectinputs,the implementationstrategyandtheintendedoutputsandoutcomes.Animplementationmodeldescribeshowstaffandresources areusedtodeliverprogrammeservicestotargetpopulation.Source:RealWorldEvaluation,byBamberger,Rugh,Mabry,p.191, 2006.
77
Theoriesofpeacebuildingincludethosepresentedattheendofthisannex,thoughasystematic inquiryintoongoingandpastconflictpreventionandpeacebuildingworkwouldlikelyrevealother theoriesunderlyingpeacebuildingprogrammes. 67 Notethatthesetheoriesarenotmutuallyexclusivea singleprogrammecanbebasedontwoormoreofthem. Sometheoriesfocusonwhoneedstochange:whichindividualsandgroupsinsocietyorwhich relationshipsneedtochange.Othertheoriesconcentrateonwhatneedstochange:aninstitution,a policy,asocialnorm.Stillothertheoriesaretieddirectlytoaparticularmethodologyorapproach:how thechangecanorshouldhappen. Evaluatingconflictpreventionandpeacebuildingtheoriesofchange Theimpacts,effectiveness,relevance,efficiencyandsustainabilityofaconflictpreventionand peacebuildingactivityresttoalargeextentontheaccuracyofitsunderlyingtheoryofchange.Afalseor incompletetheorymaybeakeyexplanatoryfactorforaprogramme,projectorpolicysfailure.In contrast,goodtheories(basedonanuptodate,thoroughconflictanalysis)contributetoeffective conflictpreventionandpeacebuildingactionandsuccessfulinterventions.Analysisofthetheoryof changeisthereforeakeyaspectofanyconflictpreventionandpeacebuildingevaluation.Thepertinent theoryshouldbereviewedintheevaluationreportandbecoveredintheevaluationsfindings, conclusionsandlessonslearned.Suchanalysiswillhelpcontributetoamorerefinedunderstandingof howtobringaboutchangeforpeace. Whenconductinganevaluation,theevaluatororevaluationteamshouldascertainthetheoriesof changeofthepeacebuildinginterventioninquestion.Whiletheyareoftenvariationsonthegeneric theoriespresentedinthetableattheendofthisannex,forthepurposeofevaluation,thetheories shouldbereframedusingtheinterventionsparticulartermsandinrelationtothespecificcontext. Attimes,thetheoriesinoperationareobvious,evenifunstated,inprogrammeproposalsandother documents.Moreoften,thetheoriesneedtobeuncoveredthroughinterviewswithimplementingstaff andotherstakeholdersorcanbeconfirmedbythosediscussions.Theevaluationprocessmayalso revealthatdifferentstaffmembersareproceedingondifferentassumptions(theories)abouthowtheir effortswillpromotechangetowardspeace.Thus,theevaluationprocessitselfcanbeusefulforhelping toclarifythisimportantdimensionofinterventionstrategy. Thetworeallifeexampleswhichfollowillustratethesepoints. Example1:EvaluatinggrassrootsconflictpreventioninLiberia Inthewakeofthe14yearcivilwarinLiberia,alargeinternationalNGOreceiveddonorfundingto developCommunityPeaceCouncils(CPCs),acommunitybasedmechanismforresolvingarangeof disputes,withanexplicitlyinterethnicapproach.TheCPCsweredesignedtopromotegreater democraticparticipationthroughleadershipdevelopment.Theevaluationteamfirstidentifiedunderlying theoriesofchangeandprogrammeassumptions(derivedmainlyfromdiscussionswithlocaland internationalstaffmembers):
67
AninitiallistofthesetheorieswasderivedfromreviewingtheReflectingonPeacePracticecasesstudies;see [http://www.cdainc.com/cdawww/project_profile.php?pid=RPP&pname=Reflecting%20on%20Peace%20Practice]forafulllistof thecasesstudies.
78
TheoriesofchangefortheCommunityPeaceCouncils: Theory#1:Byestablishinganewcommunitylevelmechanismforhandlingarangeofdispute types,wewillcontributetokeepingthepeaceandavoidingincidentsthathavethepotentialfor escalatingintoseriousviolence. Theory#2:Bycreatinginclusivestructuresforcommunityproblemsolving,wecanimprove communication,respect,andproductiveinteractionsamongsubgroupsinthecommunity,and improvetheaccessofdisenfranchisedgroupstodecisionmaking. Theory#3:Bycreatinganewleadershipgroupinfusedwithdemocraticconceptsandprovided withcriticalskills,wecanfostermoreeffectiveandresponsiveleadership. Theevaluationteamthendiscussedwhetherandhowthesetheoriesofchangewereappropriatefor thesituationinLiberia,andhowtheywereplayingoutintheprogramme.Tobegin,theteamconducted anupdatedconflictanalysis,basedoninterviewsandfocusgroupswithawiderangeofpeopleinthe communitiesthemselves.Theteamthenexaminedwhethertheprogrammewashavingtheeffects envisionedinthetheoryofchange.Forexample,theteamexaminedwhatkindsofconflictstheCPCs handled,andwhetherthoseconflictshadthepotentialforescalatingandincitingwidespreadviolence.If theydid,thentheCPCswoulddirectlycontributetostoppingakeyfactorinviolentconflict.If,however, thoseconflictswereunconnectedtothedrivingfactorsoftheconflictorthelocallevelconflicthandling mechanismswerenotabletoaddressthetypesofconflictmostlikelytoescalate,thentheCPCswould makelittleornocontributiontoPeaceWritLarge. TheevaluationteamfoundthattheCPCswere,forthemostpart,nothandlingthemostseriousand volatiledisputes,whichconcernedlandissues.Theteamthenexploredwhetherthiswasduetoafailure inprogrammeimplementation,or,alternatively,atheoryofchangethatwasincompleteorinaccurate. Themainconclusionwasthat,whiletheCPCsweresetupandtrainedwell,ascommunitieswere repopulatedandtraditionalleadershippatternswerereestablished,theCPCsweremostlyexcludedfrom handlinglandissues.Atthesametime,thehope(andtheory)regardingalternativeleadershipmodels provedunfounded,astraditionalleadersgainedcontrolovertheCPCsorusedthemtoaddressissues theypreferredthatsomeoneelsedealwith.Theevaluationrecommendedthattheagencyworkto expandthemandateandcapabilityoftheCPCsforhandlinglanddisputes,byconnectingthemtoland commissionsandotheremerginggovernmentstructures.ItshouldalsobesaidthattheCPCsdid representausefuldevelopmentaladvance,eveniftheywereunabletofulfil,ascompletelyashoped,a contributiontoPeaceWritLarge. Example2:TheimpactofinternationalpeacebuildingpoliciesandprogramminginKosovo CDACollaborativeLearningProjectsperformedanextensivestudyregardingthereasonsforthe recurrenceofinterethnicviolenceinKosovointhespringof2004,andtherelationshipofthatviolence topoliciesandprogrammesundertakenbytheinternationalcommunity.Amongotherthings,thestudy identifiedthetheoriesofchangeunderlyingthevariousapproachestoimprovingethnicrelations.Asis oftenthecase,theseunderlyingtheorieswerestronglyinfluencedbythepoliciesand(unspoken) assumptionsoftheinternationalcommunity.Themultipleaidanddevelopmentprogrammeswere directlylinkedtoimplementationofinternationallyestablished"StandardsforKosovo"andwidelyheld beliefsregardingrefugeereturns,interethnicrelations,andafuturemultiethnicstateasthebasisfor peacebuilding.
79
TheKosovoexampleconcernsmanyagenciesandmultipleprogrammes.Thestudyidentifiedmajor programmingapproaches,andassociatedtheoriesofchange,someofwhicharelistedhere,andthen examinedtheeffectivenessofeach,andtheirrelationship(ifany)topreventingviolence. A.Interethnicandinterreligiousdialogue InKosovo,thebulkofwhatagenciesandcommunitymembersidentifiedaspeacebuildingwas labelleddialogue.Dialogueencompassedawiderangeofactivities:fromsocialcontacttostructured conversationsaboutidentityandpromotionofmutualunderstanding,toproblemsolvingrelatedto concreteissues,tonegotiationandmediationofagreementsonlanduseintheMunicipalWorking GroupsonReturn.ThemostfrequenttheoriesofchangefordialogueeffortsinKosovowere: Theory#1:ByinvolvingKosovarSerbsandAlbaniansinmutualdiscussions,wecandevelopthe conditionsforthesafe,successfulandpeacefulreturnofIDPstotheirhomes.This,inturn,will promotereintegration,stabilisationoftheenvironmentandwillreverseoneofthenegative consequencesoftheconflict. Theory#2:Ifweengagecommunitymembersinparticipatoryapproachestodecisionmaking andimplementationofdevelopmentactivities,wecanstrengthencommunityrelationships. Theory#3:Ifwepromotecooperationacrossethniclinesregardingnonpoliticalissuesof commoninterest(HIV/AIDS,druguse,businessandentrepreneurship,womensrights, infrastructure,etc.),wecanbuildstrongerinterethnictiesandunderstanding. B.Trainingandpeaceeducation Traininginconflictresolution,humanrights,nonviolentcommunicationandrelatedtopicswas doneinmanycommunities,and,withdialogue,wasoneofthemostpopularapproachesto peacebuildingprogramming.Youthcamps,peacecamps,archaeologicalcamps,artcampsandmany otherswerewidespread,asweremultiethnicprogrammesoftechnicaltrainingincomputers,project management,marketing,andothertechnicalorprofessionaltopics.Toalesserextent,schoolbased peaceeducationprogrammesweredeveloped,includinghumanrightseducationandtolerance educationforchildren. Theory#1:Ifweprovidepeoplewithbetterskillsforconflictresolution,thiswillincreasethe abilityofcommunitiestosettledisputesnonviolentlyandreducethelikelihoodofviolence. Theory#2:Ifpeopletalkandplaytogethertheywillbuildrelationshipsandbreakdown stereotypes. C.Multiethnicprojectsandinstitutions Alongwithdialogueandtraining,joint(interethnic)projectsandinstitutionscomprisedasignificant proportionofthepeacebuildingprogramminginthecommunitiesthatwereincludedintheKosovo study.Someoftheprojectsweretheoutcomeoforfollowuptodialogue,aimingtotakethe communicationandrelationshipbuildingbeyondmeretalk. Theory#1:Ifwedevelopactivitiesthatprovideeconomicbenefitstobothethniccommunities (economicinterdependence),peoplewillhaveincentivestoresisteffortstoinciteviolence.
80
Theory#2:Ifweprovideopportunitiesforpeopletoworktogetheronpracticalissuesacross ethniclines,itwillhelpbreakdownmistrustandnegativestereotypes,aswellasdevelophabits ofcooperation. Theory#3:Ifpeoplehavejobsandeconomicstability,theywillbelesshostiletotheother ethnicgroup. D.Democraticgovernanceandcapacitybuilding Manyinternationaldonors,agenciesandNGOshaveimplementedpeacebuildingactivitiesdesigned tostrengthenmunicipalgovernmentinstitutionstosupportintegrationofminorities,better communicationanddialogue,andsustainablereturns. Theory#1:Ifwecanimproveadministrationandservicedeliveryandestablishnon discriminatorypolicies,thiswillreduceinterethnictensionsanddemonstratetheviabilityofa multiethnicKosovo. Manyprogrammesandpoliciesintegratedseveralapproachesandtheoriesofchange.For example,aprogrammetofacilitatereturnsofKosovoSerbminoritiesincludedseveralactivitiesand approachesreflectingacombinationofdifferenttheories: dialoguebetweenthehostcommunityandreturneeswasfacilitatedontheassumptionthat dialoguewouldallayfearsandreestablishrelationshipsthatwouldallowreturneestoreturnto theirhomesinpeace(theoryA#1); multiethniccommitteestodecidecommunityprioritiesfordevelopmentaid(theoryA#2); provisionofequipmentandseedstoamultiethnicagriculturalcooperative(theoriesC#1,2).
Oncethetheorieshadbeenidentified,theycouldbeassessedinrelationtothedrivingfactorsof conflictandthefactorscontributingtotheabsenceofviolenceinsomeplacesinMarch2004.TheKosovo studyidentifiedpatternsofinterethnicviolenceandidentifyingfactorsthatcontributedtothe preventionofinterethnicviolencethroughextensiveinterviewsincommunities(includingsomethat experiencedviolenceinMarch2004andsomethatdidnot).Theteamthenexaminedtheprogramming approachesandtheirrelationship(ifany)tothefactorsthathelpedcommunitiesavoidviolence. Thestudyfoundthatthefailureofpeacebuildingprogrammingtoachievedesiredimpactswasdue inparttofaultytheoriesofchange,andinparttoproblemsinprogrammedesignandimplementation. Designproblemsincludedfailuresintheparticipantselectionprocesses,fragmentationof programming,insufficientfollowupandlimitedresourcesforsoftaspectsofprogramming.Intermsof implementationstrategy,returneeswerenotcentralactorswithrespecttoviolence,althoughtheywere importantvictimsoftheconflict.Thechannellingofaidtoreturneesandcommunitiesacceptingreturns, itturnedout,promptedresentment,increasinginterethnicdivisionsratherthanimprovingrelations betweengroups. Inpart,thetheoryofchangeonwhichtheprogrammingwasbasedwasfaulty.Withrespectto interethnicdialoguebetweenhostcommunitiesandreturnees,thestudyfoundthatwhiledialogue activitiesopenedspaceforinterethnicinteractionthatmightotherwisenothavehappened,andhad
81
somepowerfulpersonaleffectsandledtosomecooperativeactivitiesacrossethniclines,theyneither strengthenedcommunityrelationshipsnorledtocollectiveoppositiontoviolence. Theassumptionthatthechangesinattituderesultingfromdialoguewouldleadtochangesin politicalattitudesandactions,ortrickleouttoinfluenceothersinthecommunityortrickleupto influencekeydecisionmakers,provedtobewrong.InbothKosovoAlbanianandKosovoSerb communities,implicitintracommunitypressures,orrulesofthegame,restrictedtheboundariesof permissibleinteractiontogenerallynonvisiblebusinessinteractionsandmademaintenanceand expansionofinterethniclinkagesdifficult. Thesetwoexamplesillustratejustsomeofthecommontheoriesofchangeunderlyingpoliciesand projectsworkingforpeace.Othersarelisted,alongwithexamplemethodsforeachinTable6.1.
Table6.
Commontheoriesofchange
Theoryofchange
Individualchange:Ifwetransformtheconsciousness, attitudes,behavioursandskillsofmanyindividuals,we willcreateacriticalmassofpeoplewhowilladvocate peaceeffectively.
Examplesofmethods
Individualchangethroughtraining,personal transformationorconsciousnessraisingworkshopsor processes;dialoguesandencountergroups;trauma healing.
Healthyrelationshipsandconnections:Strong Processesofintergroupdialogue;networking; relationshipsareanecessaryingredientfor relationshipbuildingprocesses;jointeffortsandpractical peacebuilding.Ifwecanbreakdownisolation, programmesonsubstantiveproblems. polarisation,division,prejudiceandstereotypes between/amonggroups,wewillenableprogressonkey issues. Withdrawaloftheresourcesforwar:Warsrequirevast amountsofmaterial(weapons,supplies,transport,etc.) andhumancapital.Ifwecaninterruptthesupplyof peopleandgoodstothewarmakingsystem,itwill collapseandpeacewillbecomepossible. Reductionofviolence:Ifwereducethelevelsof violenceperpetratedbycombatantsand/ortheir representatives,wewillincreasethechancesof bringingsecurityandpeace. Campaignsaimedatcuttingofffunds/nationalbudgets forwar;conscientiousobjectionand/orresistanceto militaryservice;internationalarmscontrol;arms(and other)embargoesandboycotts.
Ceasefires;creationofzonesofpeace; withdrawal/retreatfromdirectengagement; introductionofpeacekeepingforces/interposition; observationmissions;accompanimentefforts;promotion ofnonviolentmethodsforachieving political/social/economicends;reformofsecuritysector institutions(military,police,justicesystem/courts, prisons). Longtermcampaignsforsocialandstructuralchange; truthandreconciliationprocesses;changesinsocial institutions,laws,regulations,andeconomicsystems.
Socialjustice:Ifweaddresstheunderlyingissuesof injustice,oppression/exploitation,threatstoidentity andsecurity,andpeoplessenseofinjury/victimisation, itwillreducethedriversofconflictandopenupspace forpeace.
82
Theoryofchange
Goodgovernance:Peaceissecuredbyestablishing stable/reliablesocialinstitutionsthatguarantee democracy,equity,justice,andfairallocationof resources.
Examplesofmethods
Newconstitutionalandgovernance arrangements/entities;powersharingstructures; developmentofhumanrights,ruleoflaw,anti corruption;establishmentofdemocratic/equitable economicstructures;economicdevelopment; democratisation;electionsandelectionmonitoring; increasedparticipationandaccesstodecisionmaking. Raisethecostsandreducethebenefitsforpoliticalelites ofcontinuingwarandincreasetheincentivesforpeace; engageactiveandinfluentialconstituenciesinfavourof peace;withdrawinternationalsupport/fundingfor warringparties. Mobilisegrassrootsgroupstoeitheropposewarorto advocatepositiveaction;useofthemedia;nonviolent directactioncampaigns;education/mobilisationeffort; organisingadvocacygroups;dramatic/publiceventsto raiseconsciousness. Officialnegotiationsamongrepresentativesofkey parties;track1andtrack2dialoguesamong influentialpersons;civilsocietydialoguesinsupportof negotiations.
Politicalelites:Ifwechangethepoliticalcalculusand perceptionofinterestsofkeypolitical(andother) leaders,theywilltakethenecessarystepstobring peace.
Grassrootsmobilisation:Whenthepeoplelead,the leaderswillfollow.Ifwemobiliseenoughopposition towar,politicalleaderswillbeforcedtobringpeace.
Peaceagreements/accords:Someformofpolitical settlementisaprerequisitetopeacewemustsupport anegotiationprocessamongkeypartiestotheconflict andviolence.
Economicaction:Peoplemakepersonaldecisions,and Useofgovernmentorfinancialinstitutionstochange decisionmakersmakepolicydecisionsbasedona supplyanddemanddynamics;controlincentiveand systemofrewards/incentivesand rewardsystems;boycottsandembargoes. punishment/sanctionsthatareessentiallyeconomicin nature.Ifwecanchangetheeconomiesassociatedwith warmaking,wecanbringpeace. Publicattitudes:Warandviolencearepartlymotivated byprejudice,misperceptions,andintoleranceof difference.Wecanpromotepeacebyusingthemedia (televisionandradio)tochangepublicattitudesand buildgreatertoleranceinsociety. Transitionaljustice:Societiesthathaveexperienced deeptraumaandsocialdislocationneedaprocessfor handlinggrievances,identifyingwhathappened,and holdingperpetratorsaccountable.Addressingthese issueswillenablepeopletomoveontoreconstructa peacefulandprosperoussociety. Communityreintegration:Ifweenabledisplacedpeople (IDPs/refugees)toreturntotheirhomesandlivein relativeharmonywiththeirneighbours,wewill contributetosecurityandeconomicrecovery. TVandradioprogrammesthatpromotetolerance; modellingtolerantbehaviour;symbolicactsof solidarity/unity;dialogueamonggroupsinconflict,with subsequentpublicity.
Truthandreconciliationcommissions;criminal prosecutionsandwarcrimestribunals;reparations; communityreconciliationprocesses;traditionalritesand ceremonies;institutionalreforms.
Negotiationandproblemsolvingtoenablereturns; intergroupdialogue;excombatantcommunity engagement;processesforhandlinglandclaims;trauma healing.
83
Theoryofchange
Cultureofpeace:Ifwetransformculturalandsocietal norms,valuesandbehaviourstorejectviolence, supportdialogueandnegotiation,andaddressthe fundamentalcausesoftheconflict,wecandevelopthe longtermconditionsforpeace.
Examplesofmethods
Peaceeducation;povertyeradication;reductionofsocial inequalities;promotionofhumanrights;ensuringgender equality;fosteringdemocraticparticipation;advancing tolerance;enhancingthefreeflowof information/knowledge;reducingtheproductionofand trafficinarms.
84
ANNEX7 EVALUATIONAPPROACHESFOR CONFLICTPREVENTIONANDPEACEBUILDINGEVALUATIONS
Thefollowingsectionlaysoutsomeoftheprincipleevaluationapproachescommonlyusedin developmentevaluation,specifyinghowandwhytheymay(ormaynot)beusefulinconflictcontexts. Specificevaluationsmaydrawonelementsfromvariousapproachesdependingonthepurposeofthe evaluation.AfurtheroverviewofprosandconsofvariousapproachesisalsocontainedinChurch& Rogers(2006). Itishopedthatfurtherdetailedinformationaboutthebesttypesofevaluationmethodsinthis fieldwillemergeduringtheapplicationofthisguidanceandcontributetoprovidingmorespecific guidanceonthemostusefulevaluationapproachesinvariousconflictsituations. Resultsbasedevaluationapproach Theobjectiveistoassesswhetheractivitieshaveachievedtheintendedresults(outputs,outcomes orimpacts).Havegovernmentsandorganisationstrulydeliveredwhattheypromisedtostakeholders? Wereresultsachieved?Howwhereresultsachieved,orwhyweretheynotachieved?Thisapproach respondstogrowingdemandonthepartofdonorsandthepublicatlargeforresults,andisareactionto themoreprocessorientedevaluations.Tomeasuretheresultsofaninterventionabeforeandafter comparisonisusuallyused,withthehelpofbaselinestudiesoraninitialconflictanalysis,andindicators. Whenbaselinestudiesandresultchains(i.e.theoriesofchange)areweakornonexistent,evaluators havetoconstructthemaspartoftheevaluationprocess.Mostofthepreconditionsforresultsbased evaluationsareoftenlackinginconflictpreventionandpeacebuildingprojects,policiesorprogrammes. Abenefitofaresultsbasedevaluationprocessespeciallywhenitisaparticipatoryoneisthat stakeholdersofanactivitygettounderstandtheusefulnessofbaselinestudies,conflictanalysisand monitoringbenchmarkssuchasindicatorsandcanbeinspiredtomakegreaterplaceforitinfuture activities.Aweaknessofmanyresultsbasedevaluationsisthatbyusingabeforeandaftercomparison, theyoftendonotcapturewhathappenedintheimplementationprocessandinsidetheproject(the BlackBoxapproach)aweaknesswhichcanbeparticularlydetrimentalincomplexconflictsettings wherehowaprojectorpolicyisimplementedmightbejustasimportantaswhatitproduced.Sometimes anadditionalfocusonthiscanshedlightonwhyorwhynotintendedresultswereachieved. Participatoryevaluationapproach Theparticipationofdifferentstakeholdersofaninterventioninaconflictpreventionand peacebuildingevaluationcanbebeneficialintermsofbecomingasharedlearningexperienceforthe participants.Participationcanalsobevitalbecauseitcancontributetoincreasedtransparency,aswellas tobringingoutdifferentperspectivesandvoices.Inthisway,conflictpreventionandpeacebuilding evaluationcanalsomakeacontributiontopeacebuildinginitself.Therearedifferentdegreesof participationfromhigh(interventionstakeholdersincludedintoevaluationteam)tolowparticipation (stakeholdersinvolvedthroughbriefings,debriefingsandotherformsofinformationsharing).However, itmustagainbestressedthatpotentialforbiasedanddistortedfindingscouldarisefromparticipatory
85
evaluationmethodologieswhenlocalpeopleare,themselves,involvedinaconflict.Oneshouldtherefore notassumethatparticipatoryevaluationsarealwaysbest.Evaluationplannersshouldexplorewhether andhowparticipatorymethodologiesaffectthequalityandresultsoftheconflictpreventionand peacebuildingevaluationinagivencontext. Thistypeofevaluationapproachputsemphasisontheinvolvementofprimarystakeholdersofthe intervention(interventionstaff,partnersordonors)intotheevaluationteam.Participatoryapproaches ensureownershipandseektoenhanceuseandrelevanceoftheevaluationresults.Forconflict preventionandpeacebuildingevaluationsparticipatoryevaluationapproachescanbeveryusefulasthey contributetoanumberofevaluationprinciplesthatareimportantinhighlysensitivepolitical environmentssuchastransparency,credibility,andinclusivenessofviewsorownership.Moreover,they contributetolearningandcapacitybuildingaboutevaluationandplanningandthustothe professionalizationoftheconflictpreventionandpeacebuildingfield. However,asdescribedinSection3,thedegreeofparticipationandtheactorsinvolvedisadelicate issueandneedstobedecidedforeachevaluationcontext.Problemswithparticipatoryapproachesin conflictsituationscancomeforexamplefromtheinvolvementofpartnergovernmentsortheiragencies. Whilemostagreethatpartnergovernmentsorlocalgovernmentauthoritiesshouldbeinvolvedin evaluationsingeneral,manyadviseagainstthegovernmenttakingaleadroleorfullpartnershipin conflictpreventionandpeacebuildingevaluations.Theconcernsarethatifthegovernmentispartofthe conflict,thenpartnershipwithgovernmentinevaluationcanbiastheevaluationandhavenegative effectsontheconflictitself.Thesameistrueforactivitiesinvolvingconflictparties.Heretheissueof participation,degreeandlevelneedstobecarefullydecideduponasitmightconflictwithother evaluationprinciples. Withregardtotheuseofparticipatoryevaluation,donorsandevaluatorsshouldconsiderwhenand howtoinvolvelocalpeoplefromgovernmenttobeneficiariesandothersinthedesignoftheevaluation, basedon: Thedegreeofpoliticisationandpolarisationinthesituation Thedegreetowhichashareddefinitionoftheproblemexists.Isitpossibletodefineasetof indicators,criteriaordomainsforchangethatissharedbyallmajorgroupsofstakeholders? Thepowerrelationsamongthevariouspartiesandtheirrelationtotheprogramme.Whatisthe dangerthatoneperspectiveorgroupwilldominatetheprocessofdefiningindicatorsor evaluatingsuccess?Forexample:considerpowerdynamicswhenchoosingthelanguage evaluationworkshopswillbeheldinwillthechoiceoflanguageputonegroupatan advantage? Isitpossibletoaccessallrelevantstakeholderpointsofviewatallcriticalstagesofthe participatoryprocess?Areallsidesabletoparticipateifnotequally,atleastinasubstantialand meaningfulwayintheevaluation?Forexample,ifparticipationintheevaluationinvolvestravel, whohasaccesstomeansoftransport,dovisarequirementsaffectdifferentgroupsdifferently, willmilitaryclosuresorothersecuritythreatspreventcertaingroupsfromjoiningin?
86
Theorybasedevaluationapproach Anyevaluationthatidentifiesimplicitorexplicitassumption,hypothesesortheoriescanbe categorisedastheorybasedevaluation.Agoodtheorybasedapproachforevaluationgoesbeyond simplydrawingaprogrammelogicmodelbutinvestigatesthecasuallinkagesbetweendifferentvariables inordertofindoutwhethertheunderlyingassumptionsortheoriesofchangearecorrect.Animportant questionfortheprogrammedesigneriswhethertheassumptionsunderpinningaprogrammetheoryare basedonevidence,experience,orbeliefs. 68 Theorybasedevaluationtechniquesareusefultoassess conflictpreventionandpeacebuildingtheoriesofchangebecausetheseinterventionsoftenlackaclear baselinetocompareormeasureagainst. Tothereforeunpack(i.e.makeexplicit)thetheoryuse(ed)andassumptionstheyarebasedon,can contributetofosteringunderstandingaboutwhy,orwhynot,thingsworkandisthereforeveryusefulfor learningandaccountabilitypurposes.Forexample,questionssuchasthesecanbeanswered:whatisand howistheprogrammelogiccorrelatedwiththeconflictanalysis,whatisandhowdoesthetheoryof changecorrelatewiththefindingsfromtheconflictanalysis.)Theapproachallowsustotestwhetherthe interventionwasineffectivebecauseofpoorimplementationorflawedtheories.Forinterventionswith noexplicittheoriesofchange,moreresourcesmustbeallocatedtotheevaluationteamandprojectstaff providedupfronttouncoverassumptionsandtheirsources. Processevaluation Processevaluationisanevaluation(orpartofanevaluation)thatfocusesontheprocessof implementation,i.e.thewayinwhichtheinterventionswork,ratherthanconcentratingonthe achievementornonachievementofobjectives.Processevaluationshouldbeanintegralpartofconflict preventionandpeacebuildingevaluationasprocessesareoftenjustasimportantasoutcomes.For example,peacenegotiationsfocusmainlyontheprocessofbringingtheconflictpartiestogetherand shouldbeseenasanimportantelementofthepeaceprocess.Incasethereisnotanimmediatepeace accordasanoutcome,buttheverynegotiationroundhascontributedtoaprocesscommitmentofthe involvedparties,itcanbeseenasasuccess. Moreover,manyconflictpreventionandpeacebuildingactivitiesareindirectlytryingtoachievetheir conflictpreventionandpeacebuildingobjectives.Thismeansthatsomeprocesselementsarethemselves importantobjectivesandshouldthereforebeassessed.Forexample,adversarygroupsarebrought togetherduetoatechnicalreasonlikeIsraeliandPalestinianfilmmakersthatarejointlyorganisinga filmfestival.Theobjectiveistheattitudechangeamongtheparticipantsrelatedtotheprocess,rather thantheeffectivenessofthefilmfestival.Ifthefestivaltookplace,butwasplannedexclusivelybyonly Israelifilmmakersitwouldnothaveachievedthesamesuccesstowardspeacebuilding. Actionevaluation Actionevaluationisamethodwhereevaluators/facilitatorsworkwiththeinterventionteam,its partnersanddonorsinaniterativeprocessthroughoutthelifeoftheactivitytodefinegoals,examine assumptionsandvaluesunderlyingthegoals,anddefineandimplementmethodsfortestingwhetherthe goalsarebeingmet.Thisisthusanintegratedplanning,monitoringandevaluationapproach.Action
68
Bamberger,RughandMabry,2006.
87
evaluationisusefulforlongtermprojectsoperatinginhighlydynamicconflictcontexts,andwhichneed tobenimbleiftheyaregoingtocreatechange.Intermsofresultmeasurement,thisapproachisbetter usedtogetherwithotherapproachesthathaveamoreclassicfocusondatacollectionandevaluation judgement. 69 Selfevaluation Sometimesitwillbeusefultoincludeselfevaluation.Aprocessofselfexaminationcanbeusefulfor theimplementationstaff(attheprojectorprogrammelevel,oratthelevelofdonoror"wholeof government"strategylevel).Interventioneffortsoftenusesuchaprocessasasimplemeansofmid coursereview.Itcanalsobeincorporatedasastepinalargerevaluation/reviewexerciseorpriortoa largerevaluationandcouldbeusedasabasisfordevelopingtermsofreference. Goalfreeevaluation Anevaluationinwhichtheevaluationteamdeliberatelyavoidslearningwhatthegoalsofan interventionare,orwere,soastoavoidbeingoverlyfocussedonintendedoutcomes.Therationale behindthisapproachisthatunintendedresultsofinterventionsareasimportantasintendedones.The resultsidentifiedbytheevaluatorasaresultoftheprogrammearethencomparedtotheneedsofthe affectedpopulationtodetermineiftheprogrammewaseffective. 70 Goalfreeevaluationissometimescalledneedsbasedevaluationbecauseneedsassessmentisone oftheimportanttoolsusedtodeterminetheeffectstobeinvestigated.Thiscanprovehelpfulto evaluationinfastchangingcontextssuchasconflictsituationsandpeaceprocesses.Thismaybe especiallytruewhentheformerobjectivesofanactivityarenotrelevantanymore,butwherethe activityneverthelessmighthaveresultedinvalidcontributionstothepeaceprocess.Itisalsomaintained thatgoalfreeevaluationminimisesbiasintheevaluationprocessbecauseitisnotbasedonthe programmelogicoftheprogrammeteam.Goalfreeevaluationisgenerallymorecostlythanusingagoal basedapproach,astheevaluationteammustconsultabroadersetofissuesandawiderrangeof stakeholders. 71 Outcomeevaluationapproach Thisapproachcanbepartofanevaluationoranentireassessmentthatfocusesonoutcomesofan intervention,i.e.thechangesthathavebeenachievedbetweenoutputsandimpactsbythe interventions,intendedorunintended.Theoutcomeevaluationreferredtohereworksbackwardsfrom theoutcome.Theyinvolvemakingjudgmentsabouttheinterrelationshipbetweeninputsandoutputson theonehandandoutcomesontheotherbutdonotstartbyanalysingprojects.Thisapproachis suggestedintheUNDPguidelinesonoutcomeevaluation. 72 Aperceivedadvantagewiththisapproachforconflictpreventionandpeacebuildingevaluationis thatitstartswithwhatnormallycomeslastinotherevaluations:withanalysingchangesintheoutcome.
69 70
ChurchandRogers,2006 ChurchandRogers,2006:116117. 71 ChurchandRogers,2006.

72
UNDP,2002,www.undp.org
88
Forinstance:iftheintendedoutcomewasthatjournalistsshouldchangetheirstyleofreporting,andifit wasfoundthattherehasbeennochangeinreporting,thenonecouldworkbackwardsandfindthe reasonsforthis,andconsiderhowtoimprovethechancesofachievingthischangeinreporting.Ifthe evaluatedinterventionhasfailedtocontributeandotherfactorshavebeenmoreimportant,thisisan importantlesson.
89
ANNEX8 SAMPLETERMSOFREFERENCE(TOR)
Thefollowingtermsofreferenceexampleisprovidedtogivereadersanideaofthetypeofinformation toincludeinaconflictpreventionandpeacebuildingTORitisindicativeandshouldnotbetakenasa formmodel.InarealTOR,theinformationmightbeprovidedinadifferentsequenceandmoredetail wouldbegivenwhereneeded.FurthertipsondraftingTORscanbefoundinthe"GuidanceforEvaluating HumanitarianAssistanceinComplexEmergencies,"(OECDDAC1999)and"Effectivepracticesin conductingajointmultidonorevaluation,"(OECDDAC2000).
TermsofReference: EvaluationoftheAgency's"PeaceJournalism"programme inconflictareaX(20002003)

Definethepurposeanduseoftheevaluation.Isthepurposelearningorcontrol?Willtheevaluationbe usedtodecideonfuturefunding?Toinformfuturesupport?Toprovideinputtonewstrategy? The purpose of this evaluation is to determine whether or not the peace journalism programme wasimplementedaccordingtoagencyregulations(control/accountability)andwhatcontribution thepeacejournalismtraininghasmadetoconflictpreventionincountryx(learning)andifpeace journalism makes a significant contribution towards peace (testing the theory of change). The evaluationwillbepublishedandprovidedtoprogrammemanagersandcountryfieldstaff. Descriptionoftheevaluationobjectandscope.Whatarethespecificobjectivesoftheevaluation?Isit todocumentachievements?Assesssomeoralloftheactivity'sobjectives?Willitlookatimplementation strategiesandprocesses?Willtheevaluationhaveaparticipatoryfocus?Willitlookatunderlying assumptionsfortheprogramme/theprogrammestheoryofchange?WhichDACevaluationcriteriawill beused(impact,relevance,sustainability,efficiencyandeffectiveness)? Theevaluationwillexaminetheentirepeacejournalismprogrammefrom20002003,including outputs,outcomesandimpactsonpeaceandconflictdynamics.Specificallyitwillassesswhether ornotpeacejournalismisaneffectiveandefficientcontributiontopeacebuilding.ThefiveDAC evaluationcriteriawillbeusedtoassesstheprogramme,aswellasanexaminationofcoverageof theconflictedaffectedpopulation. Describetherationaleoftheevaluation:Whythisevaluationatthispointoftime?Describethe longevity,amountoffunding,andriskstiedtotheintervention.Arethereanyspecificeventsthathave triggeredtheevaluation(unveilingofcorruption,resultsthatruncountertointentionsinthe intervention,newresearchcomingout)? TheconflictsituationisworseningincountryXandthepublicinthedonorcountryisdemandingto knowhowouragencyhasbeeninvolvedinrecentchanges.Also,theagencyisconsideringfunding similarjournalismprogrammesinotherregionsandwouldliketoknowifthisisaneffective strategytopursue.
90
Describethescope,timeframe,objectivesandnatureoftheactivityyouwanttoevaluate.Specifying issuestobecovered,budgetandfundsspent,thetimeperiodtobeevaluated,typesofactivities, geographicalcoverage,targetgroups,aswellasotherelementsoftheconflictpreventionandpeace buildinginterventionaddressed,suchascontextualissues. Thepeacejournalismprogrammeinvolvedthetrainingof50journalistsfrom8municipaldistricts andfourworkshopsforinteriorministrystaff.Thetrainingstookplaceoverthecourseoftwodays andwererunbyagencystaffandlocalorganisationpartnersThetotalfundsdispersedwere 500,000.Theprogrammewasmeanttocontributetopeacebyreducingbiasinreportingand makejournalistsmoreawayofthesourcesanddynamicsofconflictinrelationtotheirwork (theoryofchange)EachtraininginvolvedactivitiesleadbycountrystaffofagencyThe workshopswereheldParticipantsincluded57%women,andwere30%ofthedominantreligious group(70%fromminorityreligiousgroup),40%fromminorityethnicgroupAand60%fromB Theprogrammehasnotbeenreviewed.Countryandprogrammestaffprovidedtwiceyearlyself assessmentsshowingoutputsandachievementofbasicoutcomeobjectivesincludingnumberof journaliststrainedEvaluationsofworkshopsandtrainingswerecompletedbyparticipants, Whilestaffhasfeltthiswasasuccessfulprogrammeoverall,recentescalationsinviolencehave raisedconcernsaboutimpact.Manyparticipantshavechangedtheirviewsoftheprogrammein lightofthechangingsituationontheground. Providedirectionsintermsofapproachestobeused.Whatwillbeinvolvedintheevaluation,how shouldtheevaluationbeconducted,etc.Whatwillbethelevelofstakeholderinvolvementinevaluation process? Theevaluatorswillundertakeathoroughconflictanalysisandthendraftandinceptionreport.The evaluation will include a desk review of the programme selfevaluations and participant evaluations, as well as spending and country reports from the agency and other donors in the region.TheevaluationteamwillvisitcountryXforaparticipatoryworkshopwithprogrammestaff and embassy staff, as well as to interview programme participants. Evaluators should use a standard theory based approach to assess whether or not peace journalism is an effective, efficient,sustainable,relevantandimpactfulprogrammechoiceinthiscontext. Logisticalandsafetyconcerns:addressethicalbehaviourinconflictenvironmentsandprovideguidance onsafetyandlogistics. Duetosafetyconcerns,theevaluationteamwillvisitregions1and2,butwillnotconduct interviewinregion3.Forregion3,evaluatorswillinsteadmeetwithproxiesinthedonorcountry andthecapitalofcountryXinstead.ThevisitshouldtakeplaceduringMarchAsneeded,security unitswillbeprovidedtoenterregion2. Principles:Whatstandardsandprinciplesaretobefollowed.Refertheteamtoanyrelevantpolicy documentsoragencyagreements. Theevaluationshouldfollowouragency's"Principlesforengagementinconflictsituations"and adheretothedraftDACEvaluationQualityStandards.Theteamisalsoexpectedtoadheretoour Agency'sGuidelinesonGenderSensitiveDevelopmentAssistance.Thefinalreportwillbereviewed accordingtoDACQualityStandardsbeforebeingaccepted.
91
Managementarrangements,qualitycontrolandreporting.Whowillbeinchargeofeachtaskand oversight?Whowilltheevaluationteamreportto?Isthereaneedtoestablishasteeringmechanismfor theevaluation?Whowillberesponsibleforensuringinformationsharingamongteammembers?Who willbeinvolvedindrawingandassessingconclusions?Whatreportswillbegenerated?Willtheybe publicorconfidential?Willtheybepublishedorplacedontheinternet?Willthereportsandconclusions bechecked?Whatqualitycontrolsystemswillbeused? TheteamwillreportdirectlytotheevaluationdepartmentcountryprogrammemanagerMrs.X andwillalsoworkwithasmallreferencegroupincludingX,Y,Zwhowillreviewandcommenton theinceptionreportTheteamwillcompleteafieldreportwhichwillbepresentedina participatoryworkshoptocountrystaffbeforecompletingthefieldmission.Thefinalreportwillbe reviewedbybeforebeingacceptedforpublication. Requirementsoftheteam(includingcomposition).Whoshoulddotheevaluationandwhat characteristicsdotheyneedtohave?Whatisthesizeofthetime?Whattimecommitmentisinvolved? Whattypesofindividualsareneededforthisparticularevaluationinthisparticularcontext? Theteamshouldincludeexpertsinethnicconflictandlandbaseddisputes,withexperienceinthis region.Theteamshouldbebalancedintermsofgenderandshouldalsoincludeexperienced evaluators.Theleadmembershouldhaveconflictevaluationexperienceinthis(oranother) conflictregion.AtleasttwomembersshouldbefluentinlanguageAandlanguageBandall membersshouldbecomfortableworkingunderdifficultcircumstancesandshouldhavegood communicationskillsandnonaggressiveattitudes Budgetandschedule.Howwilltheevaluationbefunded?Havetherebeenarrangementsmadefor securitycostsorotheradditionalcostsassociatedwithworkinginaconflictenvironment?Arefunds availableforconflictanalysis?(Bidsmayalsobeacceptedandthencomparedtoestablishthe appropriatefundingneeded.)Whenwilltheevaluationbeconducted? ThefinaldraftreportshouldbecompletedinAugust.Thebudgetfortheconflictanalysisanddesk studyisX,andforthefieldvisit(includingsecuritydetailifneeded)X,000
92
ANNEX9 FRAMEWORKFORFEEDBACK
Inordertocontributetoimprovingthisdocumentandbuildingaknowledgebaseoflessonsin evaluatingconflictpreventionandpeacebuildingwork,theSecretariatsoftheDACNetworkson DevelopmentEvaluationandConflict,PeaceandDevelopmentCooperationwouldappreciativereader inputparticularlyduringapplicationsofthisworkingdraft.
Pleasedetachandsendresponsesviaemailto:dacevaluation.contact@oecd.org Pleasedetachandsendresponsesviaemailto:dacevaluation.contact@oecd.org orbypost:

DCD/EvaluationNetworkSecretariat 2,rueAndrPascal 75775ParisCedex16 FRANCE
Thesurveycanalsobecompletedonlineat:www.oecd.org/dac/evaluationnetwork
1.Pleaseprovidethefollowinginformationonyourselfandyourorganisation: Name,position:________________________________________ Contactemail:__________________________________________ Organisation,ministryoragency:__________________________ Division/department:___________________________________ Country:_______________________________________________ 2.Overallhowwouldyouratetheguidanceintermsof:
usefulness

Excellent
Verygood
OK Unsatisfactory Satisfactory
Poor
Notsure
clarity

relevancetoyour work completeness
accesability("user friendly") qualityof information
93
3.Howhaveyouusedtheworkingguidancedocument(duringtheapplicationphase)? Pleasedescribethedetails:howitwasapplied,when,where,withwhom,inwhatcontext,etc.

4.Didyouexperienceanydifficultiesorchallengeswhenusingtheguidance? YesNoNotsure Pleasedescribeanyproblems(suchasareasnotsufficientlycoveredintheguidance,misunderstandings ofthetext,areasyouneedmoreinformationon,etc.).
5.Ifthereareanyspecificexamplesorinsights("lessonslearned")fromyourapplication(s)thatmight beinterestingorusefultoincludeinthefinalguidance(particularlyintheemptyexampleboxesinthe text),pleasedescribetheseindetailonaseparatesheet.
94
6.Pleaseprovidespecificfeedbackonthequalityandusefulnessofcontentsectionsoftheguidance usingthematrixbelow.Clarifyingcommentscanbeaddedatthebottomoronaseparatesheet. Excellent/ Incomplete/ Notuseful/ Usefuland Vital Unclear/Needs Shouldbe Complete information moreinfo* removed Introduction:TheConflictPreventionand PeacebuildingContext Theneedforstrategicpolicydevelopment Evaluatingconflictpreventionand peacebuilding Keyterms

Understandingpeace
Peacebuilding
Conflictprevention
Conflictprevention,peacebuildingand conflictsensitivity:Whentousethis guidance? Overviewofkeystepsinplanningand evaluatingconflictpreventionand peacebuildingwork 1.Introductiontosection 2.Somebasicprinciples
Conflictsensitivity
Genderawareness
Protectionandethicalresponsibilities
Otherconsiderations
3.1Preconditions:Programmingtoimprove workandstrengthenevaluation 3.2.Planningandpreparingtheevaluation 3.2.1Definethepurposeanduseofthe evaluation 3.2.2Decidethescopeoftheevaluation 3.2.3Doorobtainaconflictanalysis 3.2.4Outlinekeyevaluationquestions 3.2.5Taketimingandlogisticalissuesinto consideration 3.2.6Coordinatewithotheractors 3.2.7Considerconductingajointevaluation
95
3.2.8Selectevaluationcriteria 3.2.9Deviseevaluationmanagement 3.2.10DevelopTermsofReference 3.2.11Selecttheevaluationteam 3.2.12Contracting 3.3Conductingtheevaluation 3.3.1Identifytheimplementationlogicand theoryofchange 3.3.2Dealwithmissingbaselinesandother gaps 3.3.3Gatherdata 3.3.4Examinetheeffortusingvariouscriteria 3.3.5Lookatthebigpicture 3.4Concludingandlearningfromthe evaluation 3.4.1Drawconclusionsandmake recommendations 3.4.2Ensurequality 3.4.3Conductreporting 3.4.4Disseminate,feedbackandengageina learningprocess ANNEX1SELECTEDLISTOFKEYCONCEPTS ANDTERMINOLOGY ANNEX2USEFULWEBSITESAND RESOURCES ANNEX3PLANNINGANDMONITORING ANNEX4CONFLICTANALYSIS ANNEX5CONFLICTSENSITIVITY ANNEX6UNDERSTANDINGAND EVALUATINGTHEORIESOFCHANGE ANNEX7EVALUATIONAPPROACHES
ANNEX8SAMPLETERMSOFREFERENCE (TOR) *Ifyoucheckedincomplete/unclearpleaseexplaininmoredetailwhatismissingorwhatinformation youwouldliketosee.Also,feelfreetorefertosectionnumberstoprovidemorespecificcommentson contentsectionsortheoverallstructureofguidance.Continueonaseparatesheetifneeded:
96
GUIDANCE ON EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES Working draft for application period
When violent conflict breaks out, development is derailed and the human, societal and financial costs are high. Efforts to prevent and reduce violence conflict have intensified over recent years. With increasing shares of aid resources, time and energy being dedicated to conflict prevention and peacebuilding interventions, there is a growing interest amongst the donor community to learn what works, what does not work and why. In response to this challenge, two Networks of the OECD-DAC, working on conflict and on evaluation, initiated a process to develop guidance on evaluating conflict prevention and peacebuilding activities (policies, programmes and projects). Improved evaluation practice will enable systematic learning, which will in turn enhance the effectiveness of donor investments. It will assist experts and implementing organisations in improving the quality of conflict prevention and peacebuilding work and thereby contribute to preventing violent conflicts and securing the pre-conditions for sustainable development. The present working draft document will be used for one year and then revised and finalised towards the end of 2008. The Guidance on Evaluating Conflict Prevention and Peacebuilding Activities covers the key dimensions of evaluating in this field. It begins by outlining some of the challenges to evaluation in this area, and highlights the learning and accountability benefits of systematic rigorous evaluation of peacebuilding and conflict prevention activities. The main section of the guidance then walks the reader through the key elements of the evaluation process: programme planning, policy design and the establishment of evaluation preconditions; the design and set-up of an evaluation; collecting and analysing data; conducting the evaluation; and finally, learning from the evaluation process. The document also highlights areas where further learning is needed and outlines some of the key lessons that have emerged from the cooperative process of developing this guidance. This guidance is designed to fit the intersecting needs of practitioners in conflict prevention and peacebuilding, who may have limited familiarity with evaluation practices, and evaluators, who may have limited experience with evaluating conflict prevention and peacebuilding work. It will be of interest to donor policy staff and desk officers, responsible for conflict prevention and peacebuilding programmes, policies, and projects, both in headquarters and in the field; and for evaluation managers and consultants. The audience also includes non-governmental organisations (NGOs), international organisations, United Nations organisations and other development agencies working in conflict affected regions. In addition, this guidance will be relevant to practitioners in partner countries, partner governments and academics. See also: Encouraging effective evaluation of conflict prevention and peacebuilding activities: Towards DAC Guidance (OECD DAC 2007) http://www.oecd.org/dataoecd/52/3/39660852.pdf
www.oecd.org/dac/evaluationnetwork

Ghid Oecd PT Prvenirea Confl

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Ghid Oecd PT Prvenirea Confl

Hochgeladen von

Copyright:

Verfügbare Formate

GUIDANCE ON EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES

Working draft for application period

GUIDANCE ON EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES

DEVELOPMENT ASSISTANCE COMMITTEE

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT

EVALUATING CONFLICT PREVENTION AND PEACEBUILDING ACTIVITIES OECD 2008

Socio economic development

Reform ofjusticeand security institutions

Culture ofjustice, truthand reconciliation

OVERVIEWOFKEYSTEPS INPLANNINGANDEVALUATING CONFLICTPREVENTIONANDPEACEBUILDINGWORK

ChurchandShouldice,2003. See,amongothers:FAFO,2006;OECDDACandCDA,2007;ChurchandRogers,2006;USAIDandManagementSystems International,2006.

See,amongothers:InternationalAlert,2001;USAID,2007a;UnitedNationsDevelopmentProgramme,2002;Sida,2003. USAID,2007a,p.11. 38 SeeforexampletheInternationalAlertwebsite:www.conflictsensitivity.org.

Formoreonpolicycoherenceandinternationaldevelopmentassistance,visit:www.oecd.org/development/policycoherence. SeeDACGuidanceforEvaluatingHumanitarianAssistanceinComplexEmergencies: http://www.oecd.org/dataoecd/9/50/2667294.pdf.

Evaluationofcooperationinterventions andFrenchNGO'sincrisisandfragile institutionalcontexts(ongoing2007)

Evaluationoftheoverallresponsetoa particulararmedconflict(orseriesof violentevents)byaparticularfunding, channelling,orimplementingagency Evaluationofasingleproject, programmeorpolicyundertakenbya singleagency

TheMidtermReviewofthePalestinian FinnishEducationProgramme(ongoing 2007)

Conflictcauses andpotentials forpeace

Areferencegroupisgenerallyanadvisorycommitteethathelpsprovideguidanceandservesasanintermediarybetween managementandtheevaluators.Itmayalsoprovideindependentoversightoftheevaluation.Thegroupismadeupofavariety ofstakeholdersandexperts.

Weiss,1995;ChurchandRogers,2006,p.11. Animplementationmodeldescribeshowstaffandresourcesareusedtodeliverservices(BambergerandMabry,2006,p.191). 49 CDACollaborativeLearningProjects,incooperationwithCAREInternational,2006.

often,evaluatorsgatherinformationfroma)programmedocumentsandreports,b)monitoringdata(if available),andc)fieldvisitsand/orinterviewswithprogrammestaff,partnerorganisations,local/national officials,participants,andarangeofappropriatepublics.

Doestheeffortresultinanincreaseinpeoplessecurityandintheirsenseofsecurity? Doestheeffortimprovenonviolentformsofconflictresolutionorpowermanagement? Doestheeffortresultinrealimprovementinrelationsamonggroupsinconflict? Whatmajorfactorsarecontributingtoachievementornonachievementofobjectives?

Somepractitionersfeelthatoutputindicatorsarerelativelyeasiertotrackandmaybelesssusceptibletoattributionfailures. Anderson,1999a. 55 SeeamongothersOECDDACandCDA,2007.

Howefficientisthegeneralmanagementoftheintervention(steering,management, organisationalandgovernancestructuresandprocedures)? Howwellareresourcesusedtoachieveresults?

Thefollowingoptionalcriteriaareprovidedforuseduringthetestingapplicationphase. Linkages Intheconflictandpeacefieldthiscriterioncoverstheconnectionsbetweenactivitiesandpoliciesat differentlevelsandacrosssectors.Thereisevidencethatevaluatinglinkagesamongindividual/personal

3.4Concludingandlearningfromtheevaluation Summary:Concludingtheevaluationprocessand learningfromevaluation forevaluators,managers,policymakersandprogrammestaff

Someagenciesmayalsorequireanoutbriefbeforetheteamdepartsfromafieldvisit(betweentheinceptionanddraft reports),topromoteaccountability. 60 OECD,2001.

andreference.Evaluationsperformedwhileapolicyorprogrammeisstillinplacecanbeusedtoadjust orredesignit.Havingcompletedtheevaluationandlearningprocess,stakeholders,decisionmakers, managersandstaffshouldbebetterabletounderstandandimproveoutcomesandimpacts,making morelastingcontributionstopeace.

ANNEX3 PLANNINGANDMONITORINGPROGRAMMES,PROJECTSANDPOLICIES INCONFLICTPREVENTIONANDPEACEBUILDINGCONTEXTS

Changesinrealand perceived security

Changesinhousehold andcommunity victimisation

Largescalesimplerandom/clustersurvey(confidence intervalneedstobedecided)toassessimpacts. Epidemiologicalteamcouldbedevelopedtosupport theprocessandcarryoutpre/postsurveysinurban andruralcentres.

Summaryofselectedconflictanalysistools Purpose Potential users Assumptions Methodologyand effort Evaluation application

Notprogamme specific,butfocuses onbroadpolicyor programme development Facilitatesclarity ondevelopments andtrends,not causes

AninitiallistofthesetheorieswasderivedfromreviewingtheReflectingonPeacePracticecasesstudies;see [http://www.cdainc.com/cdawww/project_profile.php?pid=RPP&pname=Reflecting%20on%20Peace%20Practice]forafulllistof thecasesstudies.

Socialjustice:Ifweaddresstheunderlyingissuesof injustice,oppression/exploitation,threatstoidentity andsecurity,andpeoplessenseofinjury/victimisation, itwillreducethedriversofconflictandopenupspace forpeace.

Politicalelites:Ifwechangethepoliticalcalculusand perceptionofinterestsofkeypolitical(andother) leaders,theywilltakethenecessarystepstobring peace.

Grassrootsmobilisation:Whenthepeoplelead,the leaderswillfollow.Ifwemobiliseenoughopposition towar,politicalleaderswillbeforcedtobringpeace.

Peaceagreements/accords:Someformofpolitical settlementisaprerequisitetopeacewemustsupport anegotiationprocessamongkeypartiestotheconflict andviolence.

Truthandreconciliationcommissions;criminal prosecutionsandwarcrimestribunals;reparations; communityreconciliationprocesses;traditionalritesand ceremonies;institutionalreforms.

Negotiationandproblemsolvingtoenablereturns; intergroupdialogue;excombatantcommunity engagement;processesforhandlinglandclaims;trauma healing.

ANNEX7 EVALUATIONAPPROACHESFOR CONFLICTPREVENTIONANDPEACEBUILDINGEVALUATIONS

ChurchandRogers,2006 ChurchandRogers,2006:116117. 71 ChurchandRogers,2006.

TermsofReference: EvaluationoftheAgency's"PeaceJournalism"programme inconflictareaX(20002003)

Pleasedetachandsendresponsesviaemailto:dacevaluation.contact@oecd.org Pleasedetachandsendresponsesviaemailto:dacevaluation.contact@oecd.org orbypost:

relevancetoyour work completeness

accesability("user friendly") qualityof information

4.Didyouexperienceanydifficultiesorchallengeswhenusingtheguidance? YesNoNotsure Pleasedescribeanyproblems(suchasareasnotsufficientlycoveredintheguidance,misunderstandings ofthetext,areasyouneedmoreinformationon,etc.).

5.Ifthereareanyspecificexamplesorinsights("lessonslearned")fromyourapplication(s)thatmight beinterestingorusefultoincludeinthefinalguidance(particularlyintheemptyexampleboxesinthe text),pleasedescribetheseindetailonaseparatesheet.

Conflictprevention,peacebuildingand conflictsensitivity:Whentousethis guidance? Overviewofkeystepsinplanningand evaluatingconflictpreventionand peacebuildingwork 1.Introductiontosection 2.Somebasicprinciples

ANNEX8SAMPLETERMSOFREFERENCE (TOR) *Ifyoucheckedincomplete/unclearpleaseexplaininmoredetailwhatismissingorwhatinformation youwouldliketosee.Also,feelfreetorefertosectionnumberstoprovidemorespecificcommentson contentsectionsortheoverallstructureofguidance.Continueonaseparatesheetifneeded:

Das könnte Ihnen auch gefallen