~~~.. ~~~~~~~~~~~~~~~.~.

-.--. ... .. .

,....

WORLD

BANK

TECHNICAL

PAPER

NO.

334

Work in progress
for

\WrP33L0 9

public discussion

Performance Monitoring Indicators Handbook

AGRICULTURE
4

3g1, i',

AIUSTMUENT I EDUCATION ENVIRONMENT

FMANCIAI SECTOR - -x.t - : @HOUSING AND UR_AN DEVELOPMENT _.USTRY AND MINING
}

_ _

~~~~~~~~~~~~~OIll AND GAS

POPULAnON, HEALTH. AND NUrRMON POVERTYREDUCTION

i:
?

DEVELOPMENT _ SECTOR -1 ',Wiiun MANAGEMENT TECHNICAL ASSISTANCE TELECOMMUNICATIONS TRANSPORT - AND WASTEWATER

RobertolIosse Leigh Ellen Sontheimer

RECENT WORLD BANK TECHNICAL PAPERS No. 256 No. 258 No. 259 No. 260 No. 261 No. 262 No. 263 No. 264 No. 265 No. 266 No. 267 No. 268 No. 269 No. 270 No. 271 No. 272 No. 273 No. 274 No. 275 No. 276 No. 277 No. 278 No. 279 No. 280 No. 281 No. 282 No. 283 No. 284 No. 285 No. 286 No. 287 No. 288 No. 289 No. 290 No. 291 No. 292 No. 293 No. 294 of Smallholder Irrigationin SouthAsia Campbell, Designand Operation Countries De Geyndt, Managingthe Qualityof HealthCarein Developing of a Conference Proceedings Chaudry, Reid, and Malik, editors, Civil ServiceReformin LatinAmericaand the Caribbean: and Improvements Humphrey, PaymentSystems:Principles,Practice, Needsin the Asia Region for Childrenwith SpecialEducational Lynch, Provision Lee and Bobadilla, HealthStatisticsforthe Americas of WaterResources Strategy Le Moigne, Subramanian, Xie,and Giltner, editors, A Guideto theFormulation GrowingMediaforTreeSeedlingNurseries Miller and Jones, Organicand Compost-Based of the Women's The Participatory ProjectPlanningApproach Viswanath, BuildingPartnershipsfor PovertyReduction: (WEMTOP) OutreachProgram EnterpriseManagementTraining Hill and Bender, Developing the Regulatory EnvironmentforCompetitive AgriculturalMarkets Valdes and Schaeffer,Surveillance ofAgriculturalPricesand Trade: A Handbookfor the DominicanRepublic and Trade: A Handbookfor Colombia Valdes and Schaeffer,SurveillanceofAgriculturalPrices Scheierling, Overcoming AgriculturalPollutionof Water:TheChallenge of IntegratingAgriculturaland Environmental Policies in the EuropeanUnion Banerjee,Rehabilitation ofDegradedForestsin Asia Are FindingAlternativesto Abatement: A Study of How Enterprises Development and Pollution Ahmed, Technological Chlorofluorocarbons Greaney and Kellaghan, Equity Issuesin PublicExaminations in Developing Countries Grimshaw and Helfer, editors, VetiverGrassforSoil and WaterConservation, LandRehabilitation, and Embankment A Collection of Papers and NewslettersCompiledby the VetiverNetwork Stabilization: Govindaraj, Murray, and Chellaraj, HealthExpenditures in LatinAmerica Heggie, Management and FinancingofRoads:An Agendafor Reform Africa for Auditors:TheirPotentialforSub-Saharan Johnson, Quality ReviewSchemes Convery, Applying Environmental Economics in Africa Countries Management: Considerationsfor Developing and Karunaratne, Air Quality Wijetilleke Anderson and Ahzned, The CaseforSolarEnergyInvestments Proceedings ofa WorldBankConference Reformin Latin Americaand the Caribbean: Rowat, Malik, and Dakolias, Judicial Issuesin Forestry: Selected CaseStudiesin Asia and Economic Shen and Contreras-Hermosilla, Environmental (OCP) ControlProgram Kim and Benton, Cost-BenefitAnalysisof the Onchocerciasis A New ZealandPerspective Jacobsen, Scobie and Duncan, Statutory Interventionin AgriculturalMarketing: A of Agricultural PriceandTradePolicies: with Roldos and Chiara, Surveillance and Schaeffer in collaboration Valdes Handbookfor Uruguay Brehm and Castro, TheMarketfor WaterRights in Chile:MajorIssues Countries Developing Tavoulareas and Charpentier, CleanCoalTechnologiesfor the Next Decade Prospectsfor Gillham, Bell,Arin, Matthews, Rumeur, and Hearn, CottonProduction and Learningin AfricanEnterprises Capabilities Biggs,Shaw, and Srivastiva, Technological the World'sLakesand Reservoirs Dinar, Seidl, Olem, Jorden, Duda, and Johnson, Restoringand Protecting Weijenberg,Dagg, Kampen Kalunda, Mailu, Ketema, Navarro, and Abdi Noor, StrengtheningNationalAgricultual Action Research Systems in Easternand CentralAfrica:A Frameworkfor ofAgriculturalPriceand Trade Vald6s and Schaeffer in collaboration with Errazuriz and Francisco, Surveillance A Handbookfor Chile Policies: and Progress Transferin Mexico:Process Management Gorriz, Subramanian, and Simas, Irrigation Preker and Feachem, MarketMechanismsand the HealthSectorin Centraland EasternEurope of AgriculturalPrice Valdesand Schaeffer in collaboration with Sturzenegger and Bebczuk, Surveillance and TradePolicies: A Handbookfor Argentina (List continues on the inside back cover)

WORLD BANK TECHNICAL

PAPER NO. 334

Performance Monitoring Indicators Handbook

RobertoMosse Leigh Ellen Sontheimer

The WorldBank D. C. Washington,

Copyright 1996 The International Bank for Reconstruction and Development/THE WORLDBANK 1818H Street, N.W. Washington, D.C. 20433,U.S.A. All rights reserved Manufactured in the United States of America First printing September 1996 Technical Papers are published to communicate the results of the Bank's work to the development community with the least possible delay. The typescript of this paper therefore has not been prepared in accordance with the procedures appropriate to formal printed texts, and the World Bank accepts no responsibility for errors. Some sources cited in this paper may be informal documents that are not readily available. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations, or to members of its Board of Executive Directors or the countries they represent. The World Bank does not guarantee the accuracy of the data included in this publication and accepts no responsibility whatsoever for any consequence of their use. The boundaries, colors, denominations, and other information shown on any map in this volume do not imply on the part of the World Bank Group any judgment on the legal status of any territory or the endorsement or acceptance of such boundaries. The material in this publication is copyrighted. Requests for permission to reproduce portions of it should be sent to the Office of the Publisher at the address shown in the copyright notice above. The World Bank encourages dissemination of its work and will normally give permission promptly and, when the reproduction is for noncommercial purposes, without asking a fee. Permission to copy portions for classroom use is granted through the Copyright Clearance Center, Inc., Suite 910, 222 Rosewood Drive, Danvers, Massachusetts 01923, U.S.A. The complete backlist of publications from the World Bank is shown in the annual Index of Publications, which contains an alphabetical title list (with full ordering information) and indexes of subjects, authors, and countries and regions. The latest edition is available free of charge from the Distribution Unit, Office of the Publisher, The World Bank, 1818 H Street, N.W., Washington, D.C. 20433, U.S.A., or from Publications, The World Bank, 66, avenue d'Iena, 75116 Paris, France. ISSN: 0253-7494 Both authors work in the Bank's Operations Policy Group in the Operations Policy Department. Robert Mosse is operations adviser; Leigh Ellen Sontheimer is a consultant. Library of Congress Cataloging-in-Publication Data Mosse, Roberto, 1944Performance monitoring indicators handbook / Roberto Mosse and Leigh Ellen Sontheimer. p. cm. - (World Bank technical paper ; no. 334) Includes bibliographical references. ISBN0-8213-3731-9 1. World Bank. 2. Economic development projects-Evaluation. 1. Sontheimer, Leigh Ellen, 1964- . II. World Bank. III. Title. IV. Series. HG3881.5.W57M685 1996 338.9'0068'6-dc2O 96-27346 CIP

CONTENTS
FOREWORD V

ACKNOWLEDGMENTS

VI

PERFORMANCE

MONITORING

INDICATORS

Whatare performance monitoring indicators? 1 Why are performance monitoring indicators important? 1 Portfolio Management Task Forcefindings 1 OEDstudies 2 Next Steps Action Plan 2 Whatare theadvantages of performance monitoring indicators? 3 Howare performance monitoring indicators developed? 4 Thelogicalframework 4 Antecedents to the logical framework 5 Importance of clarifyingassumptions 7 Hierarchyof objectives and the link to performcnce indicators 8 Organizing the hierarchy of objectives 8 Types of performance indicators 11 Results indicators 11 Riskindicators 14 Efficacyindicators 14 Waysof measuring performance 15 Directmeasures 15 Indirectmeasures 15 Earlypointers:intermediate and leading indicators 15 Quantitative and qualitativemeasures 16 Measurement scope 17 Specialstudies 17 General principles for selecting indicators 18 Problems with definingindicators 18 Problems with measuring impact 19 Indicatorbenchmarks and international comparators 19 Howdo performance monitoring indicators affect theBank's work? 19 Economic and sector work 19 Project cycle 20 Project identification,preparation,and preappraisal 20

iii

Appraisal andnegotiations 21 Implementation andsupervision 21 Evaluation 21 Performance indicators andproject management 21 Field-level implementers 22 Implementation unitmanagers 22 Borrower officials andBankstaff 23 Some related issues 23 Good practice in monitoring andevaluation 23 Datacollection and management 24 Institutional arrangements andcapacity building 25 Feedback frommonitoring andevaluation efforts andinterpreting indicators 25 Relation toloancovenants 26 Working at cross purposes 26 Last word 27 Notes 27
SECTORAL NOTES ON INDICATORS

28

Whattopics do thenotes cover? 28 Howwerethenotes developed? 29 Howare indicators selected fromthenotes? 29 Where arethenotes available? 29 Howto learnmore 29
EXAMPLES OF INDICATORS

3 1

Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9

Honduras Basic Education Project (SAR 13791-HO, March 8, 1995) 32 Indonesia Rural Electrification Project (SAR 12920-IND, February 3, 1995) 33 Indonesia: Second Agricultural Research Management Project (SAR 13933-IND, April21, 1995) 35 Lithuania Siauliai Environment Project (SAR 14981 -LT, November 9, 1995) 37 PeruRural Roads Rehabilitation andMaintenance Project (SAR 14939-PE, November 6, 1995) 38 ThePhilippines Women's Health andSafe Motherhood Project (SAR 13566-PH, January 27, 1995) 40 Venezuela Agricultural Extension Project (SAR 13591 -VE, March7, 1995) 42 Chad: Structural Adjustment Credit (President's Report P-6785-CD) 43 Morocco: Financial Markets Development Loan (President's Report P-6633-MOR) 45

iv

FOREWORD As part of ongoingeffortsto improvethe qualityand impactof itswork, the World Bankis placingnew emphasis on theuseof performance monitoringindicators.These indicators, which are basedon a logical framework of projectobjectives and endmeansrelationships, help generate morethoughtful, logicallyconstructed project designs.And because theyserveas benchmarks againstwhich to measure project progress toward development objectives, theyresultin moremeaningful projectmonitoring and evaluation. Over the posttwo yearsBankstaffhavedevelopednoteson suggested performance monitoringindicatorsfor eachof the mainsectors in whichthe Bankis active.Thesenotesoffer a framework for useby taskmanagers, borrowers,and project implementation unitsin analyzingthe relationship betweenobjectives and monitorable outcomes and impacts.Theyalsooffer a menuof possible indicators. Thishandbook, which introduces and supplements the sectornotes,is divided into of indicators were developed; threesections. Thefirstsectionexplainswhy menus providesthe background on the logical framework and typologyof indicators; describes how indicatorsare developedand applied in projectdesign,supervision, useof indiand evaluation;and discusses importantissues relatedto the meaningful cators.Thesecondsection describes the sectornoteson indicatorsand their useand explainshow to get copies.Thethird sectionprovidesexamples of performance indicatorsdeveloped for Bank-financed projectsand showshow the indicatorswere objectives. developed on thebasisof each project's development

Myrna Alexander Director, OperationsPolicy May 1996

ACKNOWLE

DGME

NTS

This handbook was edited by PatriciaRogers of the World Bank's OperationsPolicy Department and Meto de Coquereaumont and PaulHoltzof AmericanWriting Corporation. Laurel Morais and Christian Perez(American Writing Corporation) laid out the text. Theauthorsare gratefulto Gerald Britan(U.S.Agencyfor International Development),MosesThompson (Teams Technologies, Inc.),UlrichThumm (Operations Evaluation Department, World Bank),and DeborahWetzel (Development Economics, World Bank)for their valuablecontributions and comments. Theauthorsalsothankthe followingBanktaskmanagers for preparingobjectives and indicatorsmatrixes: Maria Dalupan,HennieDeboeck, DelyGapasin,JoseLuis Irigoyen,Rama Lakshminarayanan, DavidNielson,Arun Sanghvi,Anna Sant'Anna, StanleyScheyer, Sari Soderstrom, and Christopher Walker.

vi

PERFORMANCE INDICATORS

MONITORING

Whatareperformance monitoring indicators?

Performance indicatorsare measures of projectimpacts,outcomes, outputs, and inputsthat are monitoredduring projectimplementation to assess progress toward projectobjectives. Theyare also usedlater to evaluate a project'ssuccess. Indicators organize information in a way that clarifiesthe relationships between a project's impacts,outcomes, outputs, and inputsand help to identifyproblems along the way that can impedethe achievement of projectobjectives.

Whyareperformance monitoring indicators important?

In recent yearsseveralimportantstudies-mostnotablythe Portfolio Management Task ForceReportand reviewsby the OperationsEvaluation Department (OED)havefound that the monitoring and evaluationof Bank-financed projectsdid not focus adequatelyon the factorsthat are criticalfor positive development impact.Boththe taskforce and OEDconcludedthat performance indicatorsshouldbe integratedwith the monitoring and evaluationprocedures usedby the Bankand itsborrowers. ThePortfolioManagement TaskForceReport(alsoknownas the Wapenhans Report) 1 analyzedthe factorsthat affectthe development impactof Bankoperations. As part of this analysisthe taskforce focusedon how evaluationmethodologies, includingthe calculationof economicrate of returnand the projectrating methodology, were used during projectappraisaland supervision to enhancethe qualityof Bank-supported projects.Thetaskforcefound that projectratingswere not providing implementation units,borrowers,and Banktaskmanagers with adequatefeedbackabout progress toward development impactfor severalreasons: * Toomuchemphasis was placed on the mechanics (physical and financial)of project implementation. * Therisksand factorsthat mostinfluenceprojectoutcomes werepoorly identified. * Objectivecriteria,transparency, and-since the methodology dependedon the judgment of individuals-consistency across unitswere lacking. * Ratings tendedto be overlyoptimistic. Porifollo Management Task Farcefindings

Without appropriate feedback, none of theparties concerned with project outcomes could make appropriate, informed decisions about whether andhowto adjust project design or implementation arrangements to better achieve a project's intended objectives. A related problem isthattheobjectives themselves areoften notwellthought outor clearly articulated, clouding thedevelopment of appropriate performance monitoringindicators andmaking monitoring andevaluation evenmore difficult. Further, thetaskforcepointed outthattheBank's traditional method of appraisal and evaluation of development impact-thecalculation of economic rateof return or netpresent value-cannot be applied to all projects andthatforsome projects, a meaningful cost-benefit analysis is notfeasible. Even for projects for whichnetpresent value or economic rateof return is calculated, Bankpractice is to calculate it onlya fewtimes-during appraisal, duringa midterm review, uponrestructuring if necessary, andat thetimeof theImplementation Completion Report. Neither thecalculations northecriticalvariables thataffect them aremonitored explicitly throughout implementation. Moreover, thereportpointed out,in manyinstances costs andbenefitscouldbe moreeasily identified andmeasured withperformance indicators. The taskforce concluded thattheBank's project rating methodology andsupervision reporting system should be adapted to include theuseof project performance monitoring indicators derived froma project's development objectives andimplementation plan. OED studies TwoOED studies reviewing theBank's record on themonitoring andevaluation of 2 projects reinrorced thefindings of thetaskforcereport. In fiscal1994 a study assessing twenty years of Bankprojects foundthatBank guidelines anddirectives on monitoringandevaluation hadnotbeen followed adequately, either duringproject appraisal (when monitoring andevaluation are planned) orduringimplementation. Butthestudy alsofoundevidence thatthesituation waschanging. A follow-up study of monitoring andevaluation plans in a sample of fiscal1995 projects suggests thatthequalityof such planning hasimproved. Theimprovement is evident in therisinguseof performance monitoring indicators-the share of projects with at leastsome indicators rosefrom72 percent in fiscal1993 to 77 percent in fiscal 1995.Nevertheless, theexpanding useof indicators hasnotbeenmatched by arrangements for datacollection or monitoring andevaluation capacity-building efforts in either theBank or borrowing countries. Relatively fewprojects (14 percent of thesample reviewed by OEDin fiscal1995)achieve overall goodpractice in comprehensive design or useof monitoring andevaluation. And performance monitoring indicators, although morewidelyused, areweakin structure and usually do notfollow a logicalframework or a typology, andthere is notalways follow-through on datacollection. NextSteps ActionPlan Inresponse to these concerns, Bank management hasmade efforts to foster theuseof performance indicators. In theNextSteps ActionPlan, which wasdesigned to implement therecommendations of theWapenhans taskforce, management calledfor incorporating performance monitoring indicators in theproject ratingsystem used for project monitoring (through Form 590 andtheAnnual Report on Portfolio Performance), to

bettermonitorprogress towarda project's development objectives. Management also recognized that the Bankneededto developsector-specific indicators to help borrowmorenarrowlyand to derivelogicalmeaersand Bankstaffdefineprojectobjectives achievement of project and impactin orderto measure sures of projectoutcomes objectives. TheBankalso needed to supportthe useof indicatorsin the revised project rating system. Therefore the actionplan calledfor the sectordepartments withinthe to developsetsof sector-specific indicatorsthatare mostreleCentral Vice Presidencies in the vantto projectdesignand monitoring-the sectornoteson indicatorsdiscussed secondpart of thishandbook. Staffwouldthenbe requiredto ensure that key sectorspecificprojectimpactindicators were identifiedin projectappraisaldocumentation and that progress was monitored againstthesebenchmarks.

indicators? of performance monitoring What are theadvantages

Performance monitoring involves periodicallymeasuring a project's progress toward objectives and givingfeedbackon the results to decisionexplicitshortand long-term (box 1). who canusethe information in variouswaysto improveperformance makers indicators Box1. Uses of performance
a developmentproject to a sales plan, incorpoSTPATEGIc PtANNING. For any program or activity, From

intothe designforces greaterconsideration of the criticalassumpratingperformance measurement help and causal paths.' Thus performance indicators tionsthat underlie thatprogram's relationships
clarify the objectivesand logic of the program.
PERFORMANCE ACCOUNTING. Performanceindicators can

helpinformresource allocation decisions if they

activities and thereby promote the most efficient use are usedto direct resourcesto the most successful

of resources.
DURING PROGRAM IMPLEMENTATION. Measuring progressagainst indicators FORECASTING AND EARLY WARNING

thatcanbe usedfor planning, identifying performance, providing feedback maypointtowardfuture areasneeding improvement, and suggesting whatcan bedone.
RESULTS. Good performanceindicators measurewhat a program has achievedrelaMEASURING PROGRAM

thustheypromote accountability. not justwhatit hascompleted; tiveto itsobjectives,

Performance indicators can be usedto demonstrateprogram MARKETING AND PJBUC RELAnONS. PROGRAM

thevalueof a datacan beusedto communicate Performance anexternal audience. results to satisfy officials andthe public. to elected programor project or whichto measure otherprojects data against indicators can generate Performance BENCHMARKING. identifying good fromsuccess, programs by learning a wayto improve programs. They also provide
performers,and leaming from their experienceto improve the performanceof others. satisfacPerformanceindicatorscan be usedto measurecustomer(beneficiary) QUALITY MANAGEMENT. whether and how the program is improving their lives. tion, and thereby assess Lessons Learned Stewort, andJamieAdrian,'PerformanceMeosurement: JerryVanSant, Leslie 1. Thomas J. Cook, 23181:1303-15 (19951. WorldDevelopment for Development Management,'

of performance benefits significant themost anditsclients of theBank Forthepurposes andmonitorsupervision project planning), (strategic design in project accrue indicators manageandquality results (measuring evaluation andproject results), ing (forecasting asproject egforts become evident information anddatacollection ment). The needed serve astools system, indicators monitoring areformulated. Ina performance objectives at indicators of performance arethevalues Baselines theflow ofchange. for measuring at theend. arethevalues targets period; of theplanning thebeginning andfromtheirdirectderivacome fromtheirmeasurability of indicators Thebenefits risk,andbenefieconomic, in sector, which aregrounded objectives, tionfromproject withquantified inputs andactivities linka project's specifically Indicators ciaryanalysis. or theborrower indicators, selecting Before and impact. outputs of expected measures of measures which must consider unitandtheBank implementation project arebeing objectives proposed andhowa project's whether will tellthem performance of the definition to a moreprecise contributing andwhowill benefit-thereby achieved dataareavailthenecessary must alsoaskwhether andtheBank Borrowers objectives. outcomes. to theindicator should do in response whatusers ableanddecide it isobjecbecause compelling is more andevaluation monitoring With indicators, indicators Furthermore, or puredescription. judgments on personal tive,notbased If designed implementation. in viewthroughout goalsclearly ultimate holda project's scope of and needs information thespecific meet indicators correctly, and used staff,implementation withimplementation-field concerned of all theparties authority unitandits implementation The Bank management. and taskmanager, unit,borrower, to-indicatorsof to respond position in a better interested-and are most subunits in interested most are borrowers andits TheBank andoutputs. riskfactors, inputs, helpall theindicators Thus impact. anddevelopment outcome, of output, indicators them. to concern of greatest on theareas focus parties indicatorsdeveloped? monitoring Howare performance
3 projects. of individual objectives ontheunique be based must indicators Performance logical on an underlying alsobe based should indicators Butanysetof performance respective their and components with project objectives project thatlinks framework is framework stages.The implementation at different andoutputs activities, inputs, its at achieving aimed should be a project under since anyaction objective-driven, to mixof outputs thebest objectives, development Giventheproject's objectives. A derived. are outputs these will yield that andcomponents objectives these achieve in box2. is shown of thepointofviewof thelogicalframework schematic general and objectives a project's from indicators performance howto derive Understanding framework. of thelogical of theconcept some understanding requires components

andan analytic projects for conceptualizing isa methodology Thelogicalframework understandclearly and project a complex communicate to the power that has frmework tool power toolwhose planning of paper.It isa participatory ably on a singlesheet beneficiaries intended views of range of the full on howwellit incorporates depends The Iogicl

Box 2. Point ofviewofthe logicl framework

DEvELoWaq OSECUnE Describes the proiect's realoutcome-theimpacthattheproject's outputs will hae on the beneficiary, institution, or system in terms of changed behavior or improved perwformrc. Thedevelopment objective defines theprojectssuccess.

PROJECT OUuws

Define whattheprojectcan6e helddirectlyaccountable for producing-theprojeds deliverables thegoodsand services it will produce. Typically, outputs are indepwdnt, synergistic, andintegrated.

Clusters of ac6ivifies that rdefine howv theproducts and services Wfill 6e deli-YOO (hchnical ossistance, physical infrastructure, and the lilke).
Source:TeamTechnologies, Inc.

and otherswho have a stakein the projectdesign. It is bestusedto help project designers and stakeholders:
e

*
*

* *

Setproperobjectives Defineindicatorsof success Identifykey activityclusters (projectcomponents) Define critical assumptions on which the projectis based Identifymeansof verifyingprojectaccomplishments Defineresources requiredfor implementation.

As an up-frontplannerthe logical framework can be usedto helpdesigntoolsfor projectimplementation and evaluation.Knowingits advantages and limitationshelps in assessing thevalue of the logical frameworkmethodology at variouspointsin the projectcycle (box 3). Antecedents to the logical framework Thelogical framework can improvethe identification, preparation,and performance appraisal process by clarifying a project'sdesignand makingit transparent to the borrower,the lender,and beneficiaries and otherstakeholders. Theframework assumes that projectsare instruments of changeand that they are selected from

Ibx3. Advantages and lihitations ofusing the logkal framework method

ADVANTAGES

* Ensures thatfundamental queshons areasked andweaknesses are analyzed in orderto provide decisionmakers with better and morerelevant information. * Guidessystematic andlogicalanalysis of the interrelated elements thatconstitute a well-designed project. * Improves planning 6y highlighting linkages technology, andeffects on theenvironment.

LMTA11ONS * May giveriseto rigidityin project administra-

tionwhenobjectives andextemal factors specified at the outset are overemphasized. Thiscon be avoided by regularproject reviews at whichthe keyelements canbe reevaluated and adjusted. . As a general analytictool,is policyneutral on questions of income distribution, employment opportunities, access to resources, local participation, costandfeasibility of strategies and between project elements andextemcal factors.

* Facilitates common understanding andbetter . Isonly oneof several toolsto be usedduring communication between decisionmakers, managers, project preparation, implementation, and andotherparties involved in the project. evaluafion. It doesnot replace beneficiary analysis; fimeplanning; economic, financial, and * Usedalongvith systematic monitoring, ensures cost-benefit analysis; environmental impact continuity of approach whenoriginalprojectstaff assessment; or similartools. are replaced. * Mayfacilitate communication between govemments anddonoragencies onceit has beenadoptedby moreinstitutions. . Makesit easierto undertake sectoral studies and comparative studies in generol, if used widely.

amongalternatives as the mostcost-effective way of achievinga desiredoutcome. It bringstogetheras its antecedents severalprojectmanagement perspectives: * Results-oriented management. Projects beginwith a setof objectives ratherthan components, and success is measured by the degreeto which development objectives havebeenmet.Management is heldaccountable for results. * Basicscientific method.Projects are experiments undertaken in an uncertain world. Fromthispoint of view a projectis a structured process for learningabout what produces results. Thisperspective assumes that projects are learningsystems. * Systems analysis.A project, like a system, is notdefineduntilwe havedefinedthe largersystem of whichit is a part.Projects takeplacenot in a vacuum but in a relationship with an external environmentof organizations, institutions, and other projects.

* Contractlaw. Everycontract(project)hasthesamebasicfeatures: . A setof deliverables (outputs) . Circumstances thatconstitute forcemajeure(assumptions) . A meetingof the minds about what thedeliverables will produce (development objective). * Cause and effect. Thecoreconcept underlying thelogicalframework is cause and effect.Thebetterthecause and effectlinkages between objectives, the bettertheprojectdesign.Bydefinition, eachprojecthasthisif-then or cause and effectlogic embedded in it. If theprojectproduces certainoutcomes undercertainconditions,

thenit canbe expected to achieve certain other outcomes. Forexample, if theproject in place, andassuming seed andputs a credit system supplies farmers withimproved there isadequate rainfall, thenproduction will increase. The logical framework forces project planners to make this cause andeffect logicexplicit, butit does notguarantee a gooddesign. The validity of thecause andeffect logicdepends onthequality and experience of thedesign team. Importance of clarifying assumptions Assumptions are riskandenabling factors-external conditions thatare outside the directcontrol of theproject (figure1).Achieving objectives candepend on whether assumptions holdtrue.Assumptions aremade aboutthedegree of uncertainty (degree of risk) between different levels of objectives. Thelowertheuncertainty that certain assumptions will holdtrue,thestronger theproject design. Failing assumptions canderaila project asoftenaspoorly executed outputs. necandeffect relationships are thecoreconcept of goodproject design, If cause essary and sufficient conditions are thecorollary. The cause and effect relationships between levels in theproject's hierarchy of objectives (inputs to outputs, outputs to outconditions for achievand impact, impact to relevance) describe thenecessary comes seepages12-13). This is the objectives (fordefinition of terms ing development internal logicof theproject. conditions at eachlevelfor achieving thenext Alsoimportant arethesufficient sufficient condicanbe met). The higherlevel(conditions in whichthenextobjective external factors must also be made expliet Figure 1. Assumptions/risks obout
Program jectve caegory/Couny Assislonce Slrabgy Assumptions Risks Risks

Development objective

Assumptions/ Risks

Otput

tAssumptions/

Components IF I |

Assumplios/\

~Ri spks

>AND

the (shown later in box 4), highlighting representation of the logical framework Note: Thisis a simplified (blankherelwould normallycontain of assumptions and risks. Thetwo centrolcolumns importance measures for verifyingeachindicator. indicators and monitoring and supervision performance Source: Team Technologies, Inc.

This is the are theassumptions. of objectives in thehierarchy levels tionsbetween plusthe conditions) (necessary hierarchy Theobjectives external logicof theproject. ideaof theproclearer together giveonea much conditions) (sufficient assumptions represents. graphically iswhatthelogicalframework This ject's design. indicators and thelinkto performance of objectives Hierarchy (outsetof terms have cometo agreeona common of thelogicalframework Users objecdevelopment goals, achievements, outputs, indicators, targets, results, comes, to or classifying objectives. areessentially waysof referring tives). These terms in a prolevels of objectives assumes thatthereareseveral logicalframework The of levthereisno logicallimitto thenumber Though of objectives). ject(ahierarchy The effectively. to manage morethanfourlevels project teams find it difficult els,most indicators predicated typically specifies four.The WorldBank uses logicalframework activities, inputs for project of objectives: threelevels thatspecifies on a framework FortheWorldBank's and impacts. andoutcomes activities, of project outputs while to project management, strictly andoutputs correspond inputs purposes, objectives. to theproject's development andimpacts correspond outcomes theBankandits reflects theagreement between Assistance Strategy TheCountry overall development goals andtheanticipated meansborrowers on a country's assistance-for theBank(andother andsector work,technical projects, economic are relevant them. Projects must showhowtheirobjectives to helpachieve financiers) development goals. of overall country to therealization withinthelogibe designed should indicators a setof performance Forany project theproject's begins with of performance indicators The development cal framework. and of activities andtheiroutputs theassociated hierarchy objectives andreflects and the activities conducted component. The for each project outcomes intended of toward theachievement are inputs of objectives at lowerlevels results achieved 5 or country level. sectoral, program, objectives, at theinstitutional, higher-level project objecultimate on theproject's thushinges at eachlevel of indicators Thedefinition disof preventable to reduce theincidence Iforexample, tive,whichcanbe modest ambitious in immunizations) or more by increasing ease withina givenpopulation scale(toreduce childmortality). of objectives thehierarchy Organizing the objectives, of project thehierarchy schematically presents Thelogicalframework themeans of eachobjective, theachievement for measuring indicators performance to critical factors) (risk andenabling andtheassumptions eachindicator, of verifying for a framework logical a sample level.Box4 illustrates thenextobjective achieving in Uganda. project andchilddevelopment nutrition informafor performance theneed objectives-and among project relationships The logicand program theoverall depicting bygraphically tion-can bealsobe clarified withtheover2).Thetree begins tree(figure in an objectives expectations performance of theCoun(consistent withtheobjectives of a project objectives development arching are objectives whichthese through outputs thelower-level lists Strategy), try Assistance

Box4. Logkai fromework fortheUganda Nutrition and Child Developmen Prsct

NARRA

suMRY I IMPACT * Improved health, nurition, andcogniivestatus of young children

PEWORMANC NDICATORS

MONTOING AND RSUbSIO4N

ASSmwPT

AmD mS6

* Reduced prevalence of protein energy malnutrition-measured by underweight preschool childrenfrom25%to 13% by 2003

* Noainal anthropomeiric survey

(impact to relevance) * Oiler national programs in heath (Immunizoaion, primaryhealhcore), food production, primaryeducaionare sustained; synergy with then programs will achieve higherprojedimpoct

Reduced micronuirient deficiency * Demogrophic andhealth problems (IDM prevalence reduced survey by 50%in endemic areas, anemia in children reduced to 33%, wormlood reduced to 50%,andxeophthalmia reduced io 2%by 2003) * Netenrollment in grode 1 * Schoa enrollment records increased to 80%by 2003 OUTCOMES * Improved childcare, nutrition, andhealth services * Improved inoome generaion copocity forwomen (outcome to impoct) * UNICEF maintains interest in partnership withprojed . Governmen signs childbill into w * Partnership withother donors
sustained

. Number of children reached by services

* Number oFwomen reached

. Annual projec report

* Annualprojedreport

by services

OUTPUTS * Established community-based * Number of children andwomen childcare program who participated in program * Number of health day outreach effortslounched * Number of ECD daycorecenters
established

* Projedc monitoring survey,

(output tooutcome) * Decentralizoaion policyimplemented

household surveys * Project monitoring reports * Projedc monitoring reports

* Established micronutrient program

* Number oFvitamin A capsules,

. Projecmonitaring reports
* Survey somple of swtsold

* Distridgovernments continue support

iron tablets distributed . Percentge of saltiodized

* Established women's income component

* Number of women who received * Projedmonitaring reports

trainingin skillsfor livelihood/ childcare enterprise INPUTS * btal $42 million,including operations and maintenance and monitoring and evaluation (octivity to output) * Sufficient interest fromcommunities

ACTIVITIES * Community-6ased growth monitoring programs * Heoah-day outreoch programs * Matching grants Fr oommunity ECD center * Massmedia campaigns * Gronts for women's livelihood capocity building * Vitamin A capsules, irontablet distributed * Iodizing of salt a Deworming compoign

. Adiviies will 6everified through regular projedmonitoring

* Mothers applynewskillsand

knowledge

TaskManager: Marib Garcia.

proFgm troe for ahypothetkal chiM health Figure 2.Partial oblectlves

OBJECTIVE Reduce childmortality
down ansvers Reoding the questions why and -whoat do we how? needto do to achiev this result?

IMPACTS Reduced incidenceof preventable

Improved nutrition

Improved family health practices

Expanded availoble care

disease

OUTCOMES Increased polio immunizations

useof Increased other Increased immunizations oral rehydration

OUTPUTS Expanded school immunizations

I=

Increased availability of vaccines

Improved public awareness

=
Improve training Expand facilities

Reading across answers ese? the eshonwhoat =at dditional things do weneed todo to achieve the next

=aobjecthve?

ACTIVITIES Incrase supplies

the upanswers Reading question 'sawhato? INPUTS Provide funds Enforce regulations Provide vaccines
-what is the

significonce af our accomphdshments?

in bold are expanded in thefigure to showthe Notethat thisis a partial objectives tree.Only the elements the projectobjective. variousactivities and the intended outcomes, outputs, and inputs neededto achieve Thesometreatment can be applied to all elements in the figureto complete the objectives tree. Agencies:Issues Source: AdoptedfromGeraldM. Britan,'Measuring Program Performance for Federal Office). D.C.: U.S.General Accounting Indicators'(Washington, and Opfionsfor Performance

required to undertake projectacfivities. Using achieved,and shows the specificinputs goals helps verifythe logic of projectdesign treeto represent a project's an objectives and confirmthatthe right indicators weredefinedto measure performance. that are necessary contree shouldincludeall the lower-evelresults An objectives Thatincludes externalfactorsand objectives. ditionsfor achievinghigher-level that do not actuallyfall assumptions about conditionsin the wider projectenvironment factors(seepages7 external risk and sensitivity of the project-the underthe control throughout the projectalong with mustbe monitored and 14).6 These aspects to ensure impacts,outcomes, outputs,and inputs,and testedduring implementation remainvalid. that the assumptions treeshownin figure 2, eachof the lower-level In theexample of the partialobjectives shownin the figurebut not only on the factorsand assumptions results is dependent of the projectshown could be the responsibility alsoon others that are not shown.These Butthey of spacelimitations. herebecause in the diagram,with the detailssuppressed or of the government, projector projects of another couldalso be the responsibility the project. and external riskfactorsunderlying makingthema partof the assumptions

10

Forexample, "expanded school immunizations" depends, asshown, on "increasing supplies," "improving training" and"expanding facilities"all, according to thefigure, responsibilities of theproject. It alsodepends on "increased availability ofvaccines" and"improved publicawareness," which could be partof theproject (though thefigure doesnotshow that) or partof another project andtherefore partof theproject's assumptions andexternal riskfactors. A toolto conceptualize, design, andappraise projects, suchasan objectives tree or similar analytical tool-and theperformance indicators derived fromit-should not 7 It should be considered unchangeable. be continually reevaluated duringeach phase of project preparation, implementation, andevaluation. If theresults analyzed duringproject implementation pointto a need to change themeans of achieving projectobjectives, themixof inputs (activities to be undertaken) anddefinition of outputs should be adjusted andnewindicators derived to measure thenewlydefined targets. Thegeneral steps thatgo intoformulating project objectives andconstructing the logicalstructure of a project are important aspects of theidentification andpreparationstages of theBank's project cycle(box5).8 There aredifferent waysto measure performance for anygiven variable(objective, impact, outcome, output, input). A system of indicators anchored in thelogicalframeworkapproach is modeled in figure3. Whilefigure3 shows a comprehensive system of indicators in orderto provide a complete frame of reference, theelements of the system thatare notcovered in detailby thishandbook areshown in dottedlines. These elements areincluded in OED's evaluations of project performance. Results indicators Results indicators measure project results relative to project objectives. Results are measured at thelevel determined by a project's objectives. Remember thatfollowing thelogicalframework approach, theyshould bedefined starting withtheimpact and outcome indicators (andworkingbackwards to theinputindicators).
INPUT INDICATORS. Input indicators measure thequantity (and sometimes thequality) of resources provided forproject activities. Depending on theproject, these caninclude:

Types ofperformance indicators

* Funding-counterpart funds, Bankloanfunds, cofinancing, grants * Guarantees * Human resources-number of person-years for members of theimplementation unit, consultants, andtechnical advisers * Training * Equipment, materials, andsupplies, or recurrent costs of these items-forexample, textbooks, syringes, vaccines, classroom facilities.
OUTPUT INDICATORS. Outputindicatorsmeasure the quantity(andsometimes the qual-

ity)of thegoods or services created or provided through theuseof inputs. Depending on theproject, these caninclude such elements as:

11

Box 5. Processes withinthe logical framework

STEPI PROBLEM AND BENEFICIARY ANALYSIS

This analysis aHtempts to answer several questions: What is the problem at stake, and who is involved in it?What are the needs, expectations, intentions, andmotivations of thevariousstakeholders, disaggregoted 6y gender and otherrelevant socialcriteria? Doestheproblemrequire extemaldevelopment assistance, or couldit be resolved some otherway2Thisstepis normally part of economic and sector workor of the project identification and preparation exercise whenthereis noprior economic and sector work, and it contributes to theformulation of the Country Assistance Strategy.
STEP 2 OBJECTIVES ANALYSIS: CONSIDERATION OF ALTERNATIVES AND PERFORMANCE EXPECTATIONS Thisstepidentifies needs or problems and transforms themintosolutions-specific, quantified objectives. Eachobjective maybebrokendowninto subobjectives and outputs to be achieved at different stages of the project. There arealmost alwaysalternative waysof meeting a projectsobjectives, and themethod chosen will affect the configuration of outputs. Thus consideration of alternatives and selection of themostappropriate is part of thisstep.Consideration oFaltematives entails economic, riskand sensitivity, cost-effectiveness, environmental and social analyses, andbeneficiary consultation. Based on theobjectives analysis, themainproject elements-comprising thewholelogicalchain, including sector and project objectives, planned outputs, necessary activities, and theirinputs-can be graphically represented in thelogicalframework or an objectives tree (see figures1 and 2). Each element-inputs, activities., outputs, outcomes, and impacts, and theriskand enabling factors thataffectthern-is measuroble. Thisis whereindicators come in. Indicators thatdefine and measure eachof these elements are identified duringtheobjectives analysis andprojectplanning stage. For this reason it is essential to formulate andexpress project objectives andoutputs in a waythatallowsmeasurement of bothshort-term and longer-term results. Definition of anoutput,outcome, or impactobjectiveshould include: * Target group(for whom) * Quantity or level(howmuch) * Quality(in terms of design standards and environmental or otherimpact) * Time (whenit should be accomplished) * Location (whereit should happen) STEP 3 FINALIZATION OF PROJECT DESIGN AND INDICATORS

In thisstepplanners carefully examine the project to ensure thatall itselements are logicallyrelated. Planners alsoassess theintegrity of indicators and realism of targets at thisstage,takingintoaccount all project assumptions and baseline data, andfinalize their plans.

* * * *

Clientsvaccinated(by a healthproject) Farmers visited(anextension project) Miles of roads built (a highwayproject) Electricity generationand transmission facilitiesinstalled(a rural electrification project) * Pollution controlmeasures installedor incentives or regulations enforced(a pollution controlor air or waterquality improvement project).
OUTCOME AND IMPACT INDICATORS. Outcome and impact indicators measurethe quan-

tity and qualityof the results achievedthroughthe provisionof projectgoodsand services.Dependingon the kind of project,thesecan include:

12

igure 3. Asystem ofindicators

RISK INDICATORS DIRECT INDICATORS
Relevance -- - --------Outcomesand impacts Outputs Inputs

EFFICACY INDICATORS

Riskor enabling factors Riskor enabling factors Riskor enabling factors

Sustainability Effectiveness Efficiency

------ Explicit development of these kinds of indicators is notrequired forWorldBank projects.

* * * *

Reduced incidence of disease(throughvaccinations) Improvedfarmingpractices(throughextension visits) Increased vehicleuseor trafficcounts(throughroad construction or improvement) Increased rural supplyand consumption of electricity(through expansionof electricity network) * Reduced mortalityor lower healthcosts(throughimproved family healthpractices or improvednutrition,or cleanerair and water)
RELEVANCE INDICATORS.Some projectshaveintendedimpacts on higher-order objec-

tivesthat are not capturedby directoutcomeindicatorssuchas the onesdescribed above.For instance,someprojects havenationalor sectoralobjectives, and for them impactmustbe measured at thoselevels.Projects mayalso haveunintended-often negative-impacts.These too can be capturedby evaluationstudies. Relevance indicatorsmeasure trendsin the wider policy problems that projectimpactsare expected to influence. If appropriateto the project,theseindicators shouldbe used.Depending on the project,thesemayinclude: * Improved nationalhealthas measured by healthindicators(through improved healthcare, healthsystem performance) * Increased farmprofitsand reduced food costs(through improved farmingpractices) * Reduced transportation costsand expandedeconomic development (throughroad construction or improvement) * Improved economicgrowthand enhanced consumer well-being(throughexpanded electrification, pollutioncontrols,and othernew technology). A system of indicatorsdrawn fromthe logical frameworkmeasures the relevance of projectresults as well as outcomes and impacts(seefigure 3). However, such effects can be very difficultto attributeto individualprojectresults. For the Bank's monitoringpurposes, therefore,measurement of results stopswith project impactrelative to projectobjectives. Thatsaid, as mentioned above, relevance indicatorsare appropriatefor objectives that are sectoralor national in scope.

13

Riskindicators Riskindicatorsmeasure the status of the exogenous factorsidentifiedas critical throughthe riskand sensitivity analysis(riskand enablingfactors)performedas part of a project'seconomicanalysis. These are the factorsthat are determined to be the mostlikely to havea directinfluence on the outcomeof variousaspects of the project (for example,economic pricesfor power or competitive salariesfor projectstaff)-the assumptions that are madeabout conditions externalto the project.A project'sobjectivescan only be achievedif the logical means-end relationship of the project elements is secureand the externalriskfactorsare favorable. Efficacyindicators Efficacyindicatorsshow how well the resultsat one level of project implementation havebeen translatedinto resultsat the nextlevel: the efficiencyof inputs, effectivenessof project outputs,and sustainabilityof project impact. They measurea project'sefficacy in achieving its objectives,ratherthan its results. The logical frameworkapproach sometimes usesthesemeasures in addition to direct indicators of results. For the Bank'smonitoringpurposes, the measurement of results usuallystopswith impact: Bankstaffare not expected to measure sustainability, effectiveness, or efficiencyas part of projectsupervision or Implementation Completion Reports but ratherto stickto "direct results" indicators.However, efficacyindicatorscan be relevant indicatorsof results if efficiency(ofan institution, for example) is a projectobjective.Moreover,OEDuses thesemeasures in its evaluations of projectperformance. It is for thesereasons that thesedescriptions of indicatorsare includedhere. inputs neededper unit of output produced-for example,physicalinputs,dollars, or labor required per unit of output. Accountabilityindicators(whichare the central focusof muchproject and financial auditing) can be considereda subsetof efficiency indicators.They measure the extentto which resources are available for and appropriately applied to the activitiesfor which they were targeted.
EFFECTIVENESS INDICATORS. Effectiveness indicatorsusuallyrepresent the ratio of EFFICIENCY INDICATORS. Efficiencyindicatorsusuallyrepresent the ratio of

outputs(or the resources usedto producethe outputs)per unit of project outcome or impact, or the degreeto which outputsaffect outcomes and impacts.For example: * Numberof vaccinations administered (or their cost)per unitdeclinein morbidity rate (illness prevented) or per unitdeclinein mortalityrate * Numberof farmers visitedper measured changein farm practices(number of farmers adoptingnew practices), or number of farmers adoptingnew practicesper unit increasein agriculturalproductivity * Miles of read built per unit increase in vehicleusage,or new road usageper unit decrease in trafficcongestion.

14

SUSTAINABILITY INDICATORS.Sustainability indicators represent thepersistence of pro-

jectbenefits overtime,particularly afterproject funding ends. They couldinclude, for example: * * * * Disease incidence trends afterexternal funding for a vaccination project ends Persistence of changed farming practices afterextension visits are completed Maintenance anduseof roads afterhighway construction ends Persistence of institutions (programs, organizations, relationships, andsoon)createdto deliver projectbenefits. Ways of measuring performance

Information on performance asmeasured by impact, outcome, output, input,efficacy, andriskindicators canbe expressed andgathered in different ways. Thechoice of indicator andmeans of collection depend on dataavailability, timeconstraints, and cost-benefit considerations aswellastherelationships between thevariables. Direct measures Direct measures correspond precisely to results at anyperformance level. Forinstance, quantities of goods delivered orcounts of clients served aredirectmeasures of output, instances of change in beneficiary behavior area directmeasure of project outcome, anda decrease in infant mortality isa directmeasure of project impact. A given variablecouldpossibly serve asan indicator of results at anyof various levels (input, output,outcome, or impact) depending on theproject objectives. Indirect measures Indirect measures correspond less precisely thandirectmeasures to theperformance sought. They are often used where directmeasures aretoodifficult, inconvenient, or costly to obtain.Indirect measures arebased on a known relationship between the performance variable andthemeasure chosen to express it-for example, using lowerfarmgate prices asan indirect indicator of increased agricultural productivity, using declining freightor taxitariffsasan indirect measure of decreasing trafficcongestion, or using reduced numbers of consumer complaints asan indirect indicator of improved customer processing. Early pointers: intermediate andleading indicators At times information on likelyproject results is needed before finalperformance data are available. At othertimes it isimportant to gaugewhether a project is on track eventhough finalresults havenotyetbeen achieved. Inbothcases intermediate or leading indicators canprovidean earlyassessment of performance (figure 4). Intermediate indicators measure intermediate results or intervening steps toward project objectives. They usually measure changes associated with theultimate impact sought butfor whichinformation canbe obtained earlier. Forexample, fertilizer purchases couldbe used asa preliminary indicator of changed farming practices, or increased nutritional knowledge asan indicator of improved eating practices. Two points must be keptin mind,however. First, intermediate indicators oftenrepresent

15

Figure 4. Early pointers

Development objective Sustainable timberproduction

-4

CL~

~~C

0 I

25

years

I Lifeof project Actual impact

Source: Team Technologies, Inc.

preliminarylinksin a causalchain, so their usefulness dependson the validity of the hypothesis that linksthosemeasures to final performance results. Second,intermediate indicatorssometimes represent results from initial or selected programsites,so their usefulness dependson the extentto which theyprove to be representative. In addition,it is sometimes possible to identifyleadingindicators (or indexes of indicators)that areclearlylinkedwith longer-term results. While similarto intermediate indicatorsin concept, leadingindicators generally havemultiple applications and a statistically valid recordof reliability(for example, theU.S.indexof leadingeconomic indicators). Quantitative and qualitativemeasures FormanyBank-financed projects,indicators of impacts, outcomes, outputs, and inputs are easilyquantified, that is, measured by definednumerical values.These are typically the basisfor calculations of economicrateof returnor net present valueduring appraisal.Duringimplementation the monitoring and evaluation system of the project or entitybeingfinancedcan providethesedata by recording,for example,the number of students matriculated and the rate of repetition, the number of farmers visitedby extension workers,or the quantities of fuelconsumed, electricity sold,or technical capacitylost.Dataon kilometers of road maintained, tollscollected, volumeof traffic, and so on can easilybe collected.In fact, thereis a dangerof collectingtoo much information. In someinstances thecostof collectingdata on projectoutcomes is highfor example,a new system mayhaveto be developed to measure educational attainment.It is important to determine how precise and timelyinformation needsto be, and who will collectit and at what cost,beforeinformation-gathering effortsare initiated.

16

A project'soutcomes and impactsmay not manifest themselves as directly measurablenumericinformation,however.These projects' effectsmay be felt more in termsof the attitudesof beneficiaries.For example,do parentsnow have more input into their children'seducation?Is teachermorale higher?A purely narrative descriptionof theseeffectsmaybe insufficientto measure results,however.It may be necessary to deviseways of measuring outcomes and impactsin quantitative terms,convertingqualitativedescriptionsinto quantitativeinformation.Thisconversion can be achievedusing surveytechniques suchas beneficiaryassessments, rapid rural appraisals,or focusgroup interviewswith structured questions. The informationgained throughthesetechniques can be usedto calculatenominal measures,rank orderingsof categories,and frequencycounts.Ratioand interval scales can also be used.Thepotential for quantifying informationon projectresultsshould not be underestimated. Measurement scope Measurement scoperefersto the useof samplepopulations.Performance indicators sometimes measureresults directly for an entire target population(of individuals or organizations)throughadministrativerecords,observations, or census surveys.Often, however,the scopeof measurement is limitedto a sampleof targetsor sites.Thisapproach raisesan additional technicalissue:how reliably can overall project performancebe statisticallyinferred on the basisof this sample? Sometimes performanceis measuredin only one project setting,or in a very few, as casestudiesinsteadof statisticalsampling.While case studiescan provide useful informationon how projects work (or why theydo not work as expected)and how theycan be improved,care mustbe taken (evenmore than in the useof samples)not to assurne that resultsfrom one site necessarily representproject performanceoverall. Accordingly,suchcase studiesare usuallyconductedin the context of speciol studies(seebelow) ratherthan as a replacement for broader performancedata.
Special studies

Sometimes a project's routine monitoringand evaluationdata do not provide sufficient information.If an unexpectedproblemarises,additional in-depthanalysis through specialstudiescan guide the way toward solving it---and avoiding it in the future.Specialstudiesare formativeevaluationsof the fundamentals of problems and their origins, and in that way differ from monitoring indicators,which are part of an early warning system.For instance,project managersmight need to learn moreabout the causal linksamong project outputs,outcomes, and impacts,especially when indicatorsrevealthat the broader purposes of a project are not being achievedeventhough its planned outputsare being delivered.In this case something clearly is wrong: the project logic may be faulty,assumptions about risk and enabling factorsmay be invalid, or somenecessary input may be lacking. Special studiesoften provide importantfeedbackfor project redesignand higher-level policy debates.

17

General principlesfor selecting indicators

Threegeneralprinciplesshouldguide the selection of performance indicators:indicators mustbe meaningful and relevant,a reliable system for collectingthe requisite data mustbe developedin a timelyfashion,and the borrower'sinstitutional capacity for usinga monitoringand evaluationsystem-and itswillingness to do so-must be 9 takeninto account.These generalprinciplesimply the followingconsiderations: * Relevance. Theindicatorsselected mustbe relevantto the basicsectoral developmentobjectives of the projectand, if possible,to overallcountryobjectives. * Selectivity. Theindicatorschosen for monitoringpurposes shouldbe few and meaningful. It is recommended that the Bankmonitorno morethana dozen indicators, at leasthalf of themimpactindicatorsthat explicitlymeasure projectimpact againsteach major development objective. * Practicality of indicators,borrowerownership,and data collection.If performance indicatorsare to meaningfully reflecta project'sobjectives, theyshouldbe selected jointly by the borrowerand the Bankduring participatoryprojectpreparation,and the data they measure shouldbe usefulto both projectand country.Thedata requiredto compilethe key indicatorsmustbe easilyavailable; if collectingthe data will requirea specialeffort,this needshouldbe identifiedearly in the project cycleand includedin the projectdesign.Datacollectioneffortscan thenbe programmeddirectlyinto projectimplementation or contracted to institutions that do surveywork at the sectoral, regional,or nationallevel. * Intermediate and leading indicators.In the absence of moredefiniteimpactindicators, early pointersof development impactmaybe usedduring projectimplementation to indicateprogresstowardachievingprojectobjectives. In manycases outcomeindicatorstogether with indicatorsof riskfactorscan serveas suitable intermediate or leading indicatorsof impact. Quantitative and qualitativeindicators. Tothe extentpossible, performance indicatorsshouldallow for quantitative measurement of development impact.For some projectobjectives (for instance, capacitybuilding)it may be necessary to develop qualitativeindicatorsto measure success, which shouldstill allow credible and dispassionate monitoring.(Fordetailson converting qualitativeindicatorsto quantitative measures of the qualitative aspectsbeing investigated, seepages 16--17.) Problems with defining indicators A previous sectiondescribed the differenttypesof indicatorsusedto measure levelsof performance. However, it is sometimes difficultto distinguish between a project'soutputsand outcomes, for example,or betweenoutcomes and impacts. When defining indicators,it is importantto thinkof the typologyof indicatorsas a continuum mirroring the logical means-end relationship of the project:inputsto variousactivities, which yield outputs, all of whichcontribute to outcomes and impacts.A particularmeasure's logical relationto the project'sstrategic objectivewill definethetype of indicatorthat it represents. Consideran agriculturalextension project.Twopossible indicatorsare the number of farmerstrainedand the number of farmers adoptingthe recommended techniques.

18

Thesecond indicator is obviously an outcome, butwhataboutthefirst? Training isan output of theproject, butit isalsoan inputintoimproving agricultural practices. This example demonstrates theneed to thinkof indicators in terms of a continuum, with inputs leading through activities to certain outputs thatleadto desired outcomes and impacts. It isnotalways easyto distinguish between outputs andoutcomes or impacts, whereas inputs are usually straightforward. Consider a project thathopes to improve girl'semployment prospects by improving theiraccess to formalandvocationaleducation. An output indicator wouldbe thenumber of girlsgraduating from secondary andvocational schools, an outcome indicator wouldbe thenumber of girls employed fromproject schools, andan impact indicator wouldbe theirearnings relativeto earlierlevels or relative to average cohort earnings. Problems withmeasuring impact Impact indicators are themost difficult to measure andcollect, mainly because of lags between project implementation andimpact or,putanother way,between thetimeof impact andthetimeit isfeasible to collect datarelating to impact. Butthemonitoring of project impact duringimplementation is oneof themainmotivations for using performance monitoring indicators. Using leading indicators andintermediate indicators asproxies for impact is a way to tackle themeasurement problem. Beneficiary assessments, rapidruralappraisals, andfocus groupinterviews are useful waysof collecting qualitative impact data.Before andafterhousehold andcommunity surveys are excellent toolsfor collecting comprehensive impact data,butthetimeandcost required make them impractical for regular project monitoring. They are moreoften used duringproject identification, intermediate checkpoints (such asmidterm reviews), implementation completion, andex post evaluation. Whenselecting indicators duringproject preparation andappraisal, theborrower (withtheBank's assistance, asneeded) should usebaseline dataandcomparative datafromotherprograms to settargets for theindicators it will monitor-thatis,the minimum values thatit expects to aimfor.Some of theindicator menus issued by the Bank's Central VicePresidencies provide comparator values asan indication of what is highandlow fora givenvariable. Bank staffshould consult therelevant sector departments of theCentral VicePresidencies on theselection of indicators for a projectandshould seek theiradvice on targets. Howdo performance monitoring indicators affect theBank's work? Indicators playan integral rolein theBank's work,fromeconomic andsector work, through theentire project cycle, to ex post evaluation. Indicators of sector performance canbe bothderived fromandused in theanalysis doneaspartof economic andsector workandused laterto inform project development, project impact monitoring, andsector monitoring. Besides providing a frameworkfor evaluating sector performance, sector-specific performance indicators can Economic and sector work Indicator benchmarks andinternational comparators

19

helppolicymakers, taskmanagers, andBankstaffrationalize a country assistance and lending strategy, select project strategies, andcreate a framework for monitoring project activities (seebelow). Sectoral indicators, obtained fromsector workor from generally available sector data,areespecially useful duringproject identification and appraisal to clarifyproject objectives andlaterto assess impact.

Project cycle A schematic representation of howperformance monitoring indicators fit intotheprojectcycleisshown in figure5. Project identification, preparation, andpreappraisal Project planning starts by defining objectives thatreflect a project's desired development impact; these objectives maybehierarchically arranged butthey have to be defined in terms of themeasurable targets onwhichperformance indicators will be based. The proposed objectives guidetheselection of a first-round setof indicators duringproject identification anddesign. As partof theproblem, beneficiary, andobjectives analyses, several questions should be discussed withtheborrower andotherstakeholders: Whatarewe tryingto achieve? Howdo we measure whatwe are achieving? Whattypes of indicators or measures will we need to develop? Whattarget values should we use? Theprocess of selecting performance indicators helps borrowers andBankstaff define project objectives moreclearly, setmeasurable goals,andensure thata project's activities leadlogically to therealization of itsobjectives. During project preparationit is important to develop a clearideaof thebaseline values of impact indicators andtheplanned target values for thedevelopment impact of theproject, as wellasthemeans for measuring progress (thelogical chainof indicators). This activity helps theborrower andtheBank assess therealism of project objecmethods for measuring indicators. tives anddetermine dataavailability andcollection (The nextsection discusses issues associated withdefinition of objectives anddata Concentrating on howto quantify andmeasure impact at thisstage measurement.) andconsistency of project comhelps theborrower andtheBankimprove thedesign ponents to achieve thedesired results ascost-effectively aspossible. Thiseffortmay Figure 5. Performance indicators and the project cycle
PROJECT IDENTIFIEATION, PREPARATION, PREAPPRAISAL

Perform baseline studies Analyze sectoral doat Idenlify risk and enabling factors

PREPARATION/
APPRAISSAL

IMPLEMENTATION/ SUPERVISION/
SUPERVISION OOMPLETION

COMPLETION
ONWARDS

Identify risk, input, output, outcome, and impact indicalors

Use indicators lomonitor implemenitlion

Evaluate output and outcome indicators

Evoluate outcome and impact indicators of project success

20

also point to the needto developmonitoring capacity beforeimplementing the-project

or to buildmonitoring andevaluation capacity development intotheproject. Appraisal andnegotiations Asproject planning isfinalized, project objectives aretranslated intospecific project components. Indicators for measuring progress toward eachobjective andfor monitoringtheprovision of project inputs andthestate of riskandenabling factors identified in theeconomic andrisksensitivity analyses are alsofinalized duringappraisal. The implementation plandeveloped by theborrower andappraised bytheBank must include theperformance monitoring indicators-inputs, outputs, outcome, and impact indicators-for each aspect of project implementation. During appraisal theborrower andtheBank agree on no more thanabout a half-dozen of themost important input andoutput indicators andabout thesame number of outcome andimpact indicators-keyperformance indicators thattheBank will monitor. These indicators-as agreed performance benchmarks, notcovenanted targets-are setoutin theproject appraisal documentation andin theBank's legalagreement withtheentity responsible for project implementation (see BankProcedure 10.00,Annex B andrelated Operational Memorandums). Incases where target values arecritical to theattainment of project objectives, theborrower andtheBank mayalso wantto make them a covenant of theloanagreement (see page26). Implementation andsupervision During project implementation theborrower andproject implementation unitmonitor theindicators developed duringproject design, following theschedule of theimplementation plan.During regular supervision missions andthemidterm review theBank monitors thekeyindicators defined andagreedwiththeborrower at appraisal. These indicators formthebasis for theBank's measures of implementation performance (IP ratings) andimpact (development objective, or DO ratings) on Form 590. If theindicators originally developed fortheproject become inappropriate because external conditions change or project design isrestructured, theBank andborrower develop
new indicatorsto reflectthe changedcircumstances. Evaluation

Afterimplementation iscompleted, theachievement of project objectives isassessed using measurable indicators-the outcome andimpact indicators developed during projectdesign-that compare theproject's actual impact withitsintended impact. The useof indicators removes subjectivity fromevaluations, givingthem a more objective basis. Thelogicof project design governs thelogicof monitoring: theimplementation unit monitors inputs andoutputs in greater detailthandoestheborrowing government or theBank. Implementation managers areprimarily concerned with thetactics of programimplementation, whileborrowers andBankstaff are more concerned withthe strategic implications of project implementation andalternatives for realizing project objectives. Performance indicators and project management

21

Tomeet thedifferent information needs of thevarious stakeholders, performance indicators usually need to be hierarchically structured. This structure can beaccomplished through nested networks of partially overlapping performance information systems in whichlower-level performance indicators are summarized, or selectively skimmed, asa basis for higher-level information. Thefollowing sections describe the different performance indicators andinformation needs of thevarious actors responsiblefor project implementation. (This information is summarized in box 6.)
Field-level implementers

Implementation managers are primarily concerned with themethods used to implement a particular project or program component. Withintheirrelatively narrow areas of responsibility, theyneedto knowthatnecessary inputs (human, financial, andphysical resources) are available andthatplanned outputs (goods andservices) are being produced. They alsoneed to knowthatresources are beingused legitimately, that financial accountability isensured, andthatinputs arebeingefficiently transformed intooutputs. Implementation managers in thefieldalsoneedat leastsome information on program outcomes andimpacts; thatis, theyneed to knowtheextent to whichprogramoutputs are achieving theirintended results sothattheycanfine-tune project implementation andmotivate performance. Implementation unitmanagers of an implementation unit(orof whatever organizational unitisresponsiThe managers bleforoverall project management) are usually responsible for implementing an entire Box 6. Project management and performance information needs
IMPLEMENTERS IN THEFIELD NEED

* Input indicators
a e

Output indicators Risk indicators

* [Efficiency indicators]
* Some outcome andimpact indicators
THEIMPLEMENTATION UNITNEEDS

including site-comparative indicators asappropriate * Summary input and output indicators, asappropriate * Outcome indicators, including site-comparative indicators * [Effectiveness indicators]
indicators * Risk
a

Impact indicators

THEBORROWER AND THEBANKNEED

* Summary input indicators * Summary output indicators

Risk indicators impact, [and relevance] indicators * Key outcome,
a e

[Sustainability indicators]

monitoring orproject supervision. Note: Indicators inbrackets are not a required part ofBank

22

program or a majorprojectcomponent. Thisresponsibility involves a rangeof activities, objective. Because somestrategic oftenacross a number of sites,aimedat achieving theyshould for projectimplementation, havea primaryresponsibility these managers play a supporting role in definingstrategic objectives, choosing amongthemor articulatpolicies fromwhichtheseobjectives deriveduringprojectpreparation. ingthe underlying Implementation unitmanagers are concerned with managing their programbetter, selecting and fine-tuning projectactivities to improveoutcomes, and enhancing project impact.Thus they needsummary information on projectinputs and outputs,particularly comparative information across sites,so thattheycan identifyemergingproblems and directmanagerial attention and resources to solvingthem.More important,theyneed comparative information on projectoutcomes. Implementation managers also needat leastsomeinformation on projectimpactto validatethe significance of their project strategy. Borrower officialsand Bankstaff Representatives of the borrowerand Bankstaffare primarilyconcerned with strategic management. Theyhaveparticipatedin articulating the policyassumptions thatunderlie Moreover, the projectand the strategic objectives the projectis intendedto achieve. theymust continue to pushthese objectives in thecontext of a dynamicexternalenvironand outputs to be surethat theyneedsummary information on projectinputs ment.Thus any emerging implementation or outcome problems are beingidentifiedand addressed at appropriateprojectmanagement levelsand to be able to answerquestions aboutthe project.Tomanagestrategically, however, borrowers and Bankstaffmostneedcomparative information on projectoutcomes and impacts and riskfactors.When impactis can alsofeeddirectly thisinformation measured in relationto thecostof projectinputs, PortInaddition, in the context of the Country intoa performance-based budgetsystem. folio Performance Review or Country Assistance Strategy update,the borrowerand to whichprojects are expected need information on the nationaltrends Bankmanagers to contribute.

Some related issues

Several factorsthatare part of good practicein monitoring and evaluation are integral data collection and mandata, in particular to the effective useof projectperformance arrangements for managing information, and the useof feedagement, the institutional In addition, users needto be awareof the backfrom monitoring and evaluation. pitfallsin the definition loancovenants and indicators and potential difference between structure indicators to work, a management For performance monitoring of indicators. indicators are a tool:on thatvalueresults mustbe in place.Performance and incentives theyinspireaction. their owntheycan do nothing,but in the properenvironment It is criticalthat managers-Bank,borrower,and projectmanagers-regard perforand strategic mission as an integralpart of their institution's mancemeasurement 10 plan. Oftentheydo not; instead,theyseeit as an adjunctto the plan, in the same Good practice in and monitoring evaluation

23

afterthemore to be satisfied asa requirement way thattheymayseeevaluation and monitoring isdone."lTobe effective implementation workof project important have projectplanners design. Once duringproject must be addressed evaluation theirnext andselected logicalindicators, thestrategic objectives carefully defined instituandmanagement, for datacollection be therequirements should consideration building, andtheuseof feedback fromthedata. tionalarrangements andcapacity andmanagement Datacollection of of dataaboutit. Indicators thelogicalsource determines of a variable Thenature primarily fromproject are derived activities outputs of project inputs andof some mayrequire that outputs, outcomes, andimpacts however, indicators of some records; thatuseparticipaincluding those studies, using surveys or special data becollected it is almost always better to piggyback regular surtorymethods. Whereit is possible, surveys (such astheLiving ontoexisting nationally or internationally supported veys facility. Special studto create a data collection Study) than Standards Measurement entity. or subcontracted to a private by theproject unitdirectly iesmaybe managed Anyproposal to collect dataforan indicator requires a discussion of: * * * * Objectives of thestudy or survey Sources of data of collection method and proposed Choices Likely reliability of thedata.

(such as and impact indicators particularly outcome of some indicators, Collection may rates, educational achievement, or cropproduction) morbidity and mortality systems. Before census or survey andqualityof national depend on theexistence thatthe andtheBankmust confirm such project indicators, theborrower choosing dataarestatistiandthattheavailable are in placeandreliable necessary systems complexity of statisbytheproject. The orareacovered callyvalidfor thepopulation to it isbetter mean thatin manycases causality of attributing ticsandtheproblems to thanto attempt response asproxymeasures andbeneficiary useservice delivery impact directly. measure agenkeptbytheparticipating fromrecords indicators arederived Manyoutput monitoring of project for purposes Forthisreason, fieldsites. cies,oftenat project should examine project planners selection), (including indicator andevaluation design to itscapacity to assess procedures andreporting record-keeping theimplementer's generate data. to thesource close andused becollected are thatdatashould points The essential a notto create It is important andreliable. be cost-effective andthatdatacollection a bureauHaving such structure. withina project bureaucracy measurement separate theriskthat and presents cost-effective isnotusually for dataproduction cratichome responsiwiththose thedatamayhavelittlecontact for producing responsible those thenumwhatis behind do notfullyunderstand thedatausers blefor using it. Thus for the havelittleappreciation andthedatacollectors to use, berstheyare given

24

face, for which performance issues that stakeholders and seniorproject managers measurement shouldbe fully integrated Justas performance data would be useful. with projectdesign,information generated shouldbe integrated with the project's management structure. As with any otherprojectcomponent, the benefitsof usingperformancedata mustat leastequalthe costof collectingthem.A projectcan provide the bestinformation at the leastcostby usingexistingdata, samplingtechniques, rapid appraisal methods, and othercreativecollectionmethods. A few otherconsiderations about data collectionshouldguidethe designof indicators. Aboveall, thedata shouldmeasure results, not just processes. Theperformance measured by thedata shouldfocuson what the projectis accomplishing, especially in termsof its impacton people.Thepointis not only to knowwhat projectsare doing, but to knowwhethertheyare doing any good. Selectivity is desirable.Performance analysisshouldbe limitedto the few areas that are directlyrelevantto the project'sstrategic objectives, as definedby the borrowerand Bank. Finally,performance measurement systems shouldusedata that are not construed as threatening by thosewho are reportingit. Simplytelling projectmanagerswhetherborrowers,Bankstaff,or thoseworkingfor them-to reportdata on their projectswithoutactivelyinvolvingthemin the performance measurement process, without explaininghow and by whom thedata are going to be used,and withoutassuring is likely to be viewedas themthat the data will not be usedto judge thempersonally is such a compliance mode of performance measurement threatening. Relyingon likelyto backfire,not only minimizingmanagerownershipbut also likely producing bureaucratic resistance and possibly data corruption. Institutional arrangements and capacitybuilding for the institutional arrangements Datacollectionarrangements haveimplications between the implementation unit,the borrower, and the Bank.Therationalefor indicanot justto the Bank. torsdemands thatthe indicatordata be of valueto the borrower, A project'soverallmonitoring and evaluation designshouldbuild on the reporting unitand the borrower,while further arrangements alreadyusedby theimplementation needs,designdata collecdevelopingthe technical skillstheyneedto plan information in a formatthat is tion, execute studies and surveys, analyzedata, and reportresults If necessary, the Bankcan financetechnical relevantto projectmanagement. monitoring and evaluaassistance and equipment requiredto designand implement tion systems and to meettrainingneeds. Feedback frommonitoringand evaluationeffortsand interpretingindicators will be Care shouldbe takento timedata collectionactivitiesso that information availablewhen it is needed,for exampleat the timeof the Bank'sportfolioor standards are specified as for which operatingperformance midtermreviews. Projects processes call for local capacityto plan and an objective,or for which decentralized of how indicamanagework programs and budgets,will need specialconsideration tor findingsare usedto inform decisions.

25

Ideally,impactis evaluthrough comparison. are bestinterpreted Project indicators to providean ideaof what the data with projectoutcomes, baseline ated by comparing suchanalysis is indicators wouldbe if the projecthad not beencarriedout. Butbecause can useotherkindsof comparisons: of indicators not practical,analysis sometimes timeseries (moreillustrative beforeand afteranalysis, relative to targets, actualresults or comparisons of controlgroups. than simplebeforeand aftercomparisons), outputsor are commonin the valuesthat are usedto measure or fluctuations Trends or controlgroupsshouldbe and impacts.If thisis the case,timeseries outcomes and controlgroupshelpverify used.Timeseriesdata capturetrendsor fluctuations, For example,the controlgroup chosenis critical for comparattributionof causality. or incomeamong a project'sbenattainment in averageeducational ing the increase in a nonproject area: thecontrolgroup mustbe identical eficiarieswith the increase otherthan the presence of the project.Finally,any to the projectsamplein all respects betweenexogenous factors interpretation of indicatorsmustconsiderthe relationship of risk the monitoring reason why Thisis one and the indicatorstheymay influence. factorsis so important.

Relation to loan

notactions.Thus standard Indicators are factsabout projectimplementation results, performance targets as indicative is to consider covenants practiceby the Bankand borrowers are usedprimarily is thatthe agreedtargets and not legallybinding.Theunderstanding objectives. and in realizingdevelopment in projectimplementation to gaugeprogress or targetsisconsidered whereattainment of certainindicators However, in instances objectives (forexample,in the caseof certain to the attainment of a project's essential by a project), theseindicators entitysupported of a profit-making financialindicators notonly as performance in the legaldocuments shouldbe incorporated (target values) (seepage 21). butalso as a loancovenant benchmarks because the borrowerand the Bank objectives lack well-defined Projects sometimes are or shouldbe. Forexample,it is somecannotagreeon what a project'spurposes or to truly resources objectiveis to transfer timesunclearwhethera project'sultimate objectives mustbe agreedon and strategic alleviatepoverty.A project'spurposes action guide for management indicatorsprovidethe clearest early,and performance programs when theylogicallyreflectclear programpriorities.In practice,however, While it is difficultto portray multipleand evenconflictingobjectives. mayencompass it may treeor logical framework, within a singleobjectives purposes multiplestrategic thesemultiplepurtreesreflecting to defineparallel objectives sometimes be possible that resultin in makingdecisions data could thenbe useful Theperformance poses. can be prioritized if suchobjectives especially objectives, among competing tradeoffs 12 or weighted. of accountabilbring a measure Anotherdifficultyis thatclearlydefinedobjectives project ambiguous In somecases ity that may makesomeofficialsuncomfortable. indiperformance may,in fact, be politicallydesirable.Delineatinguseful objectives of willingnessto comcatorsmay provedifficultif thisis the case;and in the absence itselffromthe project. the Bankshoulddissociate mit to clearly definedobjectives,

Workingat cross purposes

26

theyprovide areonly In theendperformance monitoring indicators andthefeedback economic and analysis, (economic andfinancial analysis asgoodastheunderlying and theproject design, supporting assessment) work,social andenvironmental sector to project formuThelogicalframework approach thedatato beassessed overtime. notthefull extent of project for project design andevaluation, lationis onlya structure design or evaluation. Noneof thetoolsdescribed in thishandbook can replace andriskandsensitivity analysis or sound economic, financial, social,environmental, theanalysis, thelogicalframecomprehensive monitoring andevaluation. Together, for continuous analysis anda holistic work,andtheindicators forma system approach to project design, monitoring, andevaluation. Notes
1. Effective Implementation: Key to Development Impact, September 1992. 2. An Overview of Monitoring and Evaluation in the World Bank, OED Report 13247, June 30, 1994, and Monitoring and Evaluation Plans in ProjectsApproved in Fiscal Year 1995, OED draft report, October 6, 1995. 3. The logical framework approach and indicator typology described in this section draw extensively from material prepared by R. Moses Thompson,Team TechnologiesInc.; Gerald M. Britan, 'Measuring Program Performancefor Federal Agencies: Issuesand Options for PerformanceIndicators,' prepared for the U.S. General Accounting Office, 1991; and 'Guidelines for Result-Based Planning, Management and Monitoring," preliminary version, from the Swedish International DevelopmentAgency, January 1995. 4. Several formal techniques for collaborative project planning and managementare variations on the for example. overall logical framework concept described here: Logical frameworks, ZOPP,and TeamUP, Appendix 1 of the World Bonk Porticipation Sourcebook (February 1996) describes ZOPP(objectivesand other methods and tools that support participatory development, oriented project planning), TeamUP, and provides additional references. 5. The relationship among project objectives may also involve aggregation across spoce, with resultsin different project locations added to yield regional objectives and achievements. Hierarchically linked and spatially aggregated objectives often exist in tandem. For instance, a notional child health project whose overall objective is to reduce child mortality may have a subobjective of reducing the incidence of preventable disease by educating and inoculating a certain number of individuals nationwide-that is, the sumof the individuals reached by individual clinics throughout the notion. 6. See also Handbook on EconomicAnalysis of InvestmentOperations (Washington, D.C.:World Bank, 1996), Chapter 10, "Risk and SensitivityAnalysis." 7. Various formal logical framework techniques use tools suchas problem and objective trees, situation analyses, and project planning matrixes to graphiand SWOT (strengths-weaknesses-opportunities-threats) cally representthe problem, beneficiary, and objectives analysis. 8. Even if a borrower is not explicitly aware of or does not follow the logical framework approach in designing a project, the Bank can follow the approach to appraise the project. 9. This discussion draws heavily on a report of the EastAsia Working Group on Project Performance Monitoring Indicators prepared for consideration by the EastAsia Regional Management Group. The working group included staff from the EastAsia Region, the Asia Technical Department, Human ResourcesServices, OED, and Operations Policy. 10. For additional discussion, see Monitoring and Evaluotion Plansin Staff Appraisol ReportsIssued in Fiscal Year 1995: A Follow-up to OED's Report 'An Overview of Monitoring and Evaluation in the World Bank,' OED Report 15222, December29, 1995. 11. ThomasJ. Cook, Jerry VanSant, LeslieStewart, and Jamie Adrian, "Performance Measurement: Learned for DevelopmentManagement," World Development23(8): 1303-15 (1995). Lessons 12. Briton, p. 20.

Last word

27

SECTORAL

NOTES

ON

INDICATORS

Tohelp Bankstaffand borrowersselectand useperformance indicators,the Bankis developingsectoralnotesthat discuss the useof indicatorsin relationto the major objectives or categories of problems normallyaddressed in Bank-financed projects. Eachnoteincludes menusof possibleindicators.

Whattopics dothenotes cover?

Eighteen sectoralnoteswill eventually be prepared.Theywill covereachsectorin which the Bankworksand areasof emphasis that crosseconomicand socialsectors, suchas environmental concerns, povertyreduction, publicsectormanagement, and technical assistance. Of the sectorperformance indicatornotesplanned,seventeen havebeen issued: Agriculture Economic adjustment Education Environment Financialsector Housing Industryand mining Oil and gas Population, health,and nutrition Poverty reduction Power Private sectordevelopment Technical assistance Telecommunications Transport Urbandevelopment Water and wastewater

Theonly notethat remains to be issuedis publicsectormanagement (expected in June 1996). All of the notesexceptone are in theirfirstedition, that is, theywill be revised as the Bankand its clientsgain experience with theuseof indicators.(The educationnote is in its second edition and is considered to be an exampleof best practice.)Further work on thesenoteswill: Standardize themto makethemmoreconsistent with the methodological framework described earlier * Incorporate morebestpractices and lessons learned * Incorporate international comparator data at the global and regional levelsto facilitate benchmarking amongcountries, provinces, regions,and so on, to appreciate what is high or low in a given instance.

28

Howwere thenotes developed?

As part of the Next StepsAction Plan,the CentralVice Presidencies, sectordepartments, and Development Economics Vice Presidency were askedto developsectorspecificperformance monitoringindicators. TheOperationsPolicyDepartment (OPR) of coordinated the effortsto developthe indicatornotes,facilitatingthe exchange ideasamong sectordepartments and reviewingthe product.After several departments had produceddrafts,OPRformeda qualityreviewgroupcomprising seniorstafffrom the Regions, the CentralVice Presidencies, and OED.Thegroup in approach. reviewedall of the notesand developeda framework for consistency Fromtheseinitial efforts,the Bankdevelopedthe framework for performance indicatorsthat hasbeen applied in mostof thesenotes.Mostof the sectornotes:
* Followa typologyof indicatorsbasedon a logical frameworkapproachto project

design (indicators of projectinputs,outputs,outcomes, impacts,riskand enabling factors,efficiency, effectiveness, and relevance); and Provide an executive summary, a menuof recommended key indicators,and real of indicatorsusedin Bank-financed projects. examples

Howare indicators selected fromthenotes?

materialsto specificsectors and to overarchingconcerns Thenotesare reference adjustment, and environmental issues. suchas povertyreduction,macroeconomic Theyare meantto guide taskmanagersand borrowersin applying performance monitoringindicators.They are not, however,intendedto replacethe judgmentor or borrowers.The notesprovide menusof indicators knowledgeof task managers and must that are neitherexhaustive nor mandatory-indicators are project-specific be customized by the borrowerand taskmanagerto project,sector,and country circumstances. Sinceevery projecthas its own uniqueobjectives,task managers and borrowersmustdevelop indicatorsthat correspondto theseobjectives,and not care shouldbe restrictthemselves to the menus provided in the notes.Conversely, taken not to "order the entire menu;"rather,it shouldbe treatedas an indicativelist from which to choosethe mostappropriate selections.

Whereare thenotes available?

to all SeniorOperaby theoriginatingCentralVice Presidencies Thenotesare issued and sectordivisionchiefs.Staffmembers directors, projectadvisers, tionsAdvisers, thattheir department copiesof the notespertainingto the sectors shouldhavereceived covers or havebeen notifiedthat the notesare available.Additionalcopiesare availThenoteswill also be availablein electronic able fromthe originatingdepartments. the relevanttaskmanagerlisted Network(contact form through the Bank'sEnterprise taskmonagers for Thedepartments, availability). below to confirma note'selectronic and persons to contactfor a copy of the noteare the notes,

29

Sector Agriculture Economic adjustment Education* Environment Financial sector Housing ond urban development Industry andmining Oil and gas Population, health, andnutrition Poverty reduction Power Private sector development Public sector management Technical assistance Telecommunications Transport Waterand wastewater
*

Department AGR DEC HDD ENV FSD TWVU IEN IEN HDD PSP IEN PSD PSP OPR IEN TWU TWU

Task manager Cornelis de Haan Deborah Wetzel Sverrir Sigurdsson John Dixon DavidScott/ Monika Queisser Patricia Annez Felix Remy VVilliam Porter

Contactpersonand extension JoyceSaboya Deborah Wetzel Joe-Shin Yang John Dixon HediaArbi Laura O'Connor Elisa Torre Kyran O'Sullivan 38959 31698 81418 38594 34663 37009 80323 32722 33639 35705 32722 38 1 31 37493 84015 34515 85784 33749

Tom Merrick Vivion Octran Soniyo Carvalho SoniyaCarvalho Jean-Pierre Charpenlier Kyran O'Sullivan SyedMahmood Shirley Wallace MichaelL.O Steven.s MichaelStevens NimrodRaphoeli NimrodRaphaeli Rogati Koyani Rogati Koyani ColinGannon Colin Gannon Guillermo Yepes Rose Poole

Second editionnoteissued.

Howto learnmore
Besides developing theirsecond-edition notes, theCentral VicePresidencies are disseminating thefirstedition notes and advising Regional staff on theuseof performance monitoring indicators. TheCentral VicePresidencies willcontinue to sponsor workshops on theirindicator notes and, whenrequested, advise project staff on all aspects of theuseof performance monitoring indicators. Inaddition, the Bank's Learning and Leadership Center is providing training on performance indicators as partof itsregular courses on project preparation, appraisal, and supervision.

30

EXAMPLES

OF

INDICATORS

Thefollowingexamples of key performance indicators were developed for several Bank-financed projects.These projectsrepresent currentbestpracticein the useof performance monitoring indicators;mostwere highlightedin OED'sfiscal 1995 follow-upreviewof monitoring and evaluation(Monitoringand Evaluation Plansin StaffAppraisalReports Issued in FiscalYear95, Report15222). Foreach project,a matrixpresents the project'sobjectives, with indicatorsof inputs,outputs, risk factors, outcomes, and impacts. Thematrixes do not list all the detailedindicatorslistedin the borrower'sProjectImplementation Plan;theyonly showthe key indicatorsthat the Bankwill monitoras the basisfor projectsupervision and evaluation. These matrixes weredevisedby the project'staskmanagers with OPR's guidance. Theinformation setout in thesematrixes was alsogiven in the StaffAppraisalReport for eachof the projects, althoughnot in thisformat.In the futurethe mostimportant projectoutcomeand impactindicators,suchas thosepresented in thesematrixes, shouldbe listedin everyprojectappraisaldocument and monitoredusingForm590.

31

Table 1. Summary ofobjectives and key performance indicators, Honduras Basic Education Project (Staff Appraisal Report 13791 -HO, March 8, 1995)
INPUTS (Resources provided for project activities)
*

OBJECIIVES Improve quality of lecirning andstudent performance in the primary education cycle

OUTPUTS (Goods andservices produced by theproject)

RISKS ANDCRITICAL ASSUMPTIONS (Theoutcome is dependent on... )

*

OUTCOMES AND IMPACTS (of project activities)

IDAcredit($30 million)

* Government funds

($9.8 million) * German lKfW)funds ($13 3million)

Strengthening the * 10-20%increase in institutional capacity students' testscores of theMinistry of Education * I 0%reduction in repetition maytake longer to anddropoutrates achieve thananticipated * 5-10% increase in future
earnings of primary school graduates from poor families

Inadequateprovision Fundswill be uisedto~ of counterpartfunds * Trainprimary and *30,000 trained teachers, by the government preprimaryteachers, principals,and supervisors principals,and supervisors * Supply textbooks., didactic materials,and ~ibrory books 4.3 million new textbooks and 20,000 library books, 8,500 poor rural students with didactic materialseach year - 60% of indigenous primary students receive bilingual education - 8 roundsof math aind Spani-sh te5stapplied to 20% of primary students

*Encouragebilingual education

* Financeexternal evaluationsof student academic achievements

* Improveschoolsinipoor * 290 expanded or ruralareas and appoint rehabilitatedand furnished additionailteachers rural schools; 1, 200 new rural preschoolprograms

Strengthen capacity of the Ministry of Educationto deliver basic education services

*IDA, bilateral, and * Reorganize Ministry of governmentfundswill Education finance technical * Decentralize servicesto assistance, equipment, 18 departments office improvements5, staff * Municipalities maintain training, monitoringand schools evaluationefforts, and teacher performance incentives

*30% reductionin administrativecosts * 40% reductionin centralstaff * Budgetdeficitsavoided * Lowerteacher absenteeism * Greater efficiencyin the use of public resources for basic educaitionservices,as

mneasured by loweroverhead
costs (by X%) Taskmanager.Anna Sant'Anna.

32

Table 2. Summary ofobjectives and key performoance Indicotors, Indonesia Rural Electrification Prolect (Staff Appraisal Report 12920-IND, February 3, 1995)
INPUTS (Resources provided for project activities) OUTPUTS (Goods and services produced by theprolect) RISKS AND CRITICAL ASSUMPTIONS (The outcome is dependent on... None.WellwithinPLN's implementation and financial capacity customers

OBJECTIVES Expand coverage and supply of electricity to rural areas

OUTCOMES ANDIMPACTS iof project octivities)

* Supply electricity to 7,000 additional rural villages covering 2. 1 million newcustomers * Meetelectricity consumption targets of 35-45 kilowott hours a month, depending on region * Increase penetration of televisions, radios, andother appliances * Increase rural household electrification from 32%to more than 40%by 1998

* IBRD loan($398 miliion) Expand electricity * Government/PLN funds network ($442.8million) * 28,000 kmcof MV lines * Ministry of Cooperotives* 35,000 kmc of LV lines ($0.5 million) * 1.3 million poles

Funds will be used to procure equipment, works, consultants, and training andtechnical assistance for capacity building Establish incentives for private sector andlocal cooperatives to provide a larger share of rural electricity distribution and renewable energy generation for rural power supply

* 833 MVAdistribution transformers

* Publish small power

* PLN maybeslowto * Strengthen theenabling purchase tariff andstondard enter intopowerpurchase environment forprivate sector powerpurchase contract for contracts andcooperatives to generate, saleof electricity by private * Delayed implementationdistribute, andsellwholesale generators to PLN of thepolicy andregulotoryandretail-level powerfrom * Esiablish bulksupply framework to oversee retail renewable energy sources. tariff for saleof bulkelectricity pricing andservice aspects * Increase megawaits and of private distributors megawatt hours of non-PLN generation provided by renewable energy from 0 to 75 megawatts and 300 megawatt hours a yearby 1998 e Increase thenumber of cooperatives engaged in electricity distribution who buy bulkpowerfrom PLN * Reduce power generation requirements from high-cost diesel plants to 75 megawatts and200 megawatt hours by 1998

Table continues on thenext page.

33

Table 2 (continued)
INPUT OUTPUT (Resources provided (Goods andservices for project activitiesl produced by theprojectl RISKS AND CRITICAL ASSUMPTIONS (The outcome is dependent on...

OBJECTIVES

OUTCOME/IMPACT (of project activities)

Advance theefforts initiated under theFirst Rural Electrification Project to placetherural electrification program onanefficient andsustainable footing by * Enhancing theefficiency of rural electricity delivery by increasing and strengthening local participation
* Training program not program to enable village-level keeping pacewithtargets technicians to undertake selected distribution and customer service functions Itroining target numbers to be determined) * Expand PLNraining * Increase from 2,000 to

4,000 thenumber of village cooperatives engaged in distribution-related operations andmaintenance andcustomer service functions, increasing coverage from7 million to 12 million rural customers * Enhance PLN's ability to extend itscustomer base without proportionately increasing itsorganizational size a Improve staff productivity index from already high250 customers peremployee
* Target 36 million kilowatts peryearincrease inelectricity consumption by rurol businesses * Create thepotential for 15,000 newjobsinrural businesses andincreased human capital developmeni and participation benefits to village cooperatives

Maximizing theeconomic benefits of rural electricity by promoting itsproductive uses

* Convert 30,000 small rural businesses to electricity forproduction purposes

* Weekmanagement by PIN

Initiate pilotprograms to testnewdesigns and construction management methods withthepotenlial forfurther lowering the costs of rural electricity extension andintensifioation

Complete fourpilotprojects: * Single Wire Earth Return systems * Reinforced concrete pole * Low-cost substations * Contract administration company

* PLN delays in timely * Generate dataon completion of pilotprojects adaptability of lechnical options * Improper implementation andconstruction management limits valueof operational techniques andsavings potentiol experience inrural electricity delivery * 10-25% reduction incosts, depending on program * 15-20%reduction incosts of rural electricity delivery in next stage of government's rural electricity program

Task manager: ArunSanghvi.

34

Table 3. Summary ofobjedtives and key performance Indicators, Indonesia: Second Agrickutural Research Management Proled (SAR 13933-IND, April 21,1995)
INPUTS (Resources provided For project activities) OUTPUTS (Goods andservices produced by theproject) RISKS AND CRITICAL ASSUMPTIONS (Theoutcome is OUTCOMES AND IMPACTS dependent on .) (ofproject activities)

OBJECTIVES Establish ond operate eightagricultural technology assessment institutes in 12 provinces to serve as regional centers forfarming systems research andtechnology transfer

* IBRD loan($63 Eight agricultural technology * Sustainability of local * 25%increase inadoption of million) assessment institutes withnew counterpart funding locahon-specific techinologies and * Government counterpart andrehabilitated facilities * Relocation of MRD best practices tested by funds J$38.8 million) andequipment offering. staff to regional locations agricultural institutes * Dotabases onAEZ * Strengthening of * Decentralize agricultural Funds will finance civil characteristics and extension andsupport research anddevelopment works, equipment, books, regional information services at locallevels system, focused onlocalneeds, training, research, and * Support services for farmers in placewithin fiveyears technical assistance andextension workers * 30%increase innumber of on-farm triols, demonstrations, Theeightinstitutes willcreate andtechnology tronsfer five yearresearch and activities involving agricultural development master plans institutes' clients andreceive staff training * Train 75%of agricultural andtechnical assistance institutes' staff in research and under this component extension methods, technical fields, andfarming systems * Increase arlention to social andgender issues inresearch anddevelopment planning, project design and implementation, and monitoring andevaluation a Ensure that o more responsive anddecentralized research and development system isin place within fiveyears e 10% increase in productivity of majorfarming systems * Improve dissemination of information andnewtechnology to farmers andusers * Introduce standard * Increased commitment* Adoptslandard research research anddevelopment of localgovernments to anddevelopment management management procedures support decentralized procedures for planning, priority tested by theagricultural research and seiting, monitoring andevaluainstitutes development tion,finance, andadministrolion * Develop databases on * 20%increose in use of AEZcharacteristics in nine databases byMRD, agricultural additional agricultural institutes, localgovernments, and institutes universities for national andlocal * Setupinformation and planning andpolicymaking accounting systems in * Implement research and 17 agricultural institutes development master plans and * Create fiveyear research annual plans andprojects in anddevelopment master plans 17 agricultural institutes for nine odditional * Increase technology transfer agricultural institutes support forfarmers, extension * Establish three pilotsoil workers, andclients by the labsforfarmers agricultural institutes andtheir * Provide staff training and colfborators technical assistance on research * Increose AARD staff l(bout anddevelopment management 70%in agricultural institutes for nine additional agricultural andproject implementation institutes, theMRD Secretariat, units) trained in research and andreseorch institutes development tools andpractices

Reform research and development management at theregional levelto ensure relevance of research anddevelopment to users andto improve research quality andeffectrveness

Table continues onthe nextpage.

35

T . 3 (citued)
INPUTS
OBJECTIVES (Resources provided forproject octvities)

OUTPUTS
(Goods andservices produced by the project)

RISKS ANDCRMCAL ASSUM?P1ONS

(The outcome is dependen on... OUTCOMES AND MPACTS (of project activities) * Manageinsitutsandstaff beher * Increase gavernment allocation foragricultural research anddevelopment (to 1 %of agricukural GDP)

Fund research inpriority areas at national reseorch institutes tosupport regional resech anddevelopment programs

* 25%increase in reseorch * L.inted government outputs from projects on courhrpart funds for pnority areas (livestck, priority areas and hsheries horticuiture, tree disciplines craops) andpriority disciplines fbioiechnolpg, agribusiness, morketing, sociobogy) * Rehobilitate speciolized loboratories inselected national research insritutes * Train scientisb ondprovide tachnicol support in priority areas anddiscipines

* 5%increase in governmrent allocation foragricultural research anddevelopment; 10% increase infunds for priority areos and disciplines * 25% increase innumber of technologies generated by national research institutes for on-farm testing anddemonstration by agncultural institutes * Rehabilitate specialized loboratories in national research institutes to support priority areas anddisciplines * 10% increase in number of trained scientists (posigroduate) in notional research institutes vwvrking on priority areas anddisciplines * Generate high-quolity research in nraional research insfHtutes and agriculural institutes * Secure increased and sustained funding forpriority research areas anddisciplines

Improve access to eenmaly generated technobgies and strengthen collaboration withAsia-Pocific and international centers, the private sector; and universihes

* Colloborating inslituions'* Improve technologies and

counterpart funds

newinformation from joinl projects under theUniversity Grant Program * Inroduce nevw echnologies, monagement tools, and stondard methods from joirt projects vwith Asia-Pacific and international centers, NARS, universities, andtheprivte sectr * Trin staf andjointpopers from theScientific Echange Program

* Iaeased language * 30%increase in number of capacity of AARD staff colobortiveprojects and * Icreased commiment activihes vith intemational and of ensemnl groups to Asio-Pocific research oenters colaborate andfund localuniversities, andtheprivote joird octivilies sector * increase exposure of Indonesian scientists ond monagers to international and regional forums and20%increase in number of scientific papers in referred journals * 25% increase in scientists, managers, andagriculturol institute staff trained outside Indonesia * Increase exposure of Indonesian scientists and managers to globalresearch system * Foster collaboration among NARS andbetween NARS and IARCs

Task manager: DefyGaposin.

36

Table 4. Summary ofobectives ii key perfonrmce hWktos,

Uthuonla Slaulal Envlunnt Proedt (Stuff Appraisal Reort 1498l-IT, Nvovmbw 9, 1995)
INPUTS (Resources provided for project activities) OUTPUTS (Goods andservices produced by theproject) RISKS ANDCRITICAl ASSUMPTIONS [The outcome is dependent on .. I

OBJECTIVES

OUTCOMES ANDMPACTS (ofproject activitiesl

Reduce pollutant * IBRD loan($6.2million) * Rehabilitate sewer network * Problems with securing * Increase treated wasteloads from theSiouliai * Bilateral gronts ($8.54 * Rehabilitate wastewater locol funding water from 40,000cubic area into theUpper million) treatment plant meters a dayto 50,000
Lielupe River Basin
* Government funds 1$7.6millioni M Municipality funds ($0.4 million)

Funds will be used to procure equipment, works, consultants, andtechnical assistance itrainingl

* Construct newwastewater treatment plant * Implement pollution control measures at pig farms * Implement pollution control measures for agricultural runoff

cubicmeters a day * Reduce nitrogen, phosphorus, and other pollution at thetrectment plants'outlets, at the mouth of theLielupe River, and at thewastewater treatment plant,fromX tons a yearto Y tonso year * Reduce pollution levels at selected points downstream fromagricultural pilotsites and pig farms 9 Lower (byXX}health care
costs

* Increase (by X%) tourism revenues * Increose international political goodwill Improve quality, reliability, andcost efficiency of water supply and wastewater services in Siauliai
* * * *

Rehabilitate equipment Provide newequipment Restructure waterutility Troin people

* Abilityto adjust tariffs * Revenue collection

difficulties * Political difficulties withorganizational restructuring (staff reduction)

* Improve drinking water quality (lowerironcontent andsofter potablewater) * Ensure fewerbreaks and trouble callsonthe water supply anddistribution system and thewastewater collection andconveyance system * Provide an adequate operating ratio(less than 85%) andodequate working ratio(less than70%X for thewaterutility * Ensure regular and

Improve localand regional environmental quality monitoring and enforcement system in theUpperLielupe River Basin

* Provide monitoring and

* Potential coordination laboratory equipment difficulties between * Provide otherequipment concerned parties * Train people * Develop management plans to reduce industrial pollution andsludge * Develop emergency management plan

accurale monitoring of water quality * Ensure regular enforcement visits at pollution sources

Task manager: SariSoderstrom.

37

Table 5. Summary ofobledives and key performance indicators, PonRural Roads Rehabilitation and Maintenacne Project (Staff Appraisal Report 14939-PE, November 6, 1995) RISKS AND CRITICAL INPUTS OUTPUTS ASSUMPTIONS (Resources provided (Goods and services (The outcome is forproject activities) produced bythe project) dependent on ...
Inter-American Development Bankloan ($90 million) * Government funds ($70.2 million)
'

OBJECTIVES
ond increase reliability of vehicular access in rural areasto expand markets forogricultural andnonfarm products by:

OUTCOMES AND IMPACTS (ofproject octivities(

program areaslinkedby a reliable and affordable publictransportation system * 30%lowerfreight toriffs for transport of agricultural output * 30%lowerpossenger tariffs for transportin rural areas * 30%increase in commercial traffic(buses, pickups, and trucks) within three years of completion of roadimprovements * Alleviate ruralpoverty in the 12 poorest departments (X% change in poverty measure)
* Reduce dustpollution and

Reduce tronsport costs * IBRD loan ($90million) * Bring 5,000 km ofrural * Favorable macroeconomic * 80% ofcommunities in
roadsup to adequate standards for traffic * Bring 2,500 kmof connecting roadsup to goodcondition * Maintain7,500 km * Integrating inaccessible Funds will be usedto of roadsroutinely according zones withregional procure technical to guidelines developed economic centers assistance, equipment, under the project and works conditions and terms of trade * Favoroble natural conditions in project area * Security situation doesnot prevent execution of works * Local construction industry develops capacity to undertake largevolume of works

* Improving transport conditions in rural villages

* Improve 220 kmof streets * Communities willingto

in at least 300 villages * 20%of iniegrated rood projects include a nonmotorized transport component

participote andcommit resources * Successful coordination withotherprograms to enhance access to intermediate means of transport

increased traffic and pedestriansafety in 200 villoges * Improve trafficconditions and 25 villages develop capacity to manage the unclassified neKworks of trocks * Raise livingstandards for .5 millionruralresidents through increased Iby Y%) infrastructure access
* Generate 35,000 oneyearequivalent nonskilled seosonal jobs * Generate morethan 4,000 nonskilled permanent jobs

Increase employment through rehabilitation andmaintenance of ruralroads

* Contract outto local

* Continued government communities andcontractors support for poverty at least$250 million eradication policyand invested in works theproject

Table continues on the next page.

38

Table 5(confloed)
INPUTS OUTPUTS (Resources provided (Goods and services for project activities) produced by the projectl RISKS AND CRlTICAL ASSUMPTIONS (The outcome is dependent on... I

OBJECTIVES

OUTCOMES AND IMPACTS (of project activities)

Increase institutional copocity at thelocallevel andincrease capacity for small andmedium-size enterprises to manage and carry outsustainable maintenance andupgrading of roods

Provide technical assistance for: * Government commitment to * Establish rural roadsunit * Planning andmanogement empowering municipalities within MTCto planand of rural roads within MTC, * Municipalities' porticipation monitor rurol roodinvesinment PERT-PCR, andlcol in thetechnical assistance andtransporafion performance governments program * Revise fundional and * Developing microenterprises* System of intergavernmental jurisdictional classification andsmall andmedium-size iransfers andlocalrevenue of roads andinventory enterprises for roadmaintenance sources in placeasa resuh of all public roods * Developing thelocal of thedecentralization agenda * Streamline localgovernment construction industry. procurement procedures in Studies on: accordance withtheaction * Local RoadAdministration plandeveloped under the Practices Local Roods Administration * Rural Roads Funding Practices study * Implement mechanism to secure funds for rural maintenance agreed between central and localgovernments in accordance withthe actionplandeveloped under the Rural Roads Funding study * Encourage more than 300 local contradors and consultants to register with PERT-PCR andparticipote in projecactivities * Ensure that100 microenterprises and small andmediumsizeparticipate in routine maintenance * Ensure that60 lof 117) provincial municipalities adoptinstitutional programs designed under theproject

Task manager: Jose Luis Irigoyen.

39

Table 6. Summry ofobjedives and keyperformace Indacators, 27,1995) 13566-PH, January Appraisal Report Prolet (Staff Hotho and Saf Motherhood The Philippires Womn's
INPUTS iResources provided for project activities)
*

OBJECTIVES health status Improve women of oflow-income

OUTPUTS services iGoods and bytheproject) produced

RISKS ANDCRITICAL ASSUMPTIONS (The outcome is on... j dependent

ANDIMPACTS OUTCOMES (ofproject activities) proportion of * Increase receiving prenatal and women
family postnatal care, and RTI/STD and planning, cervical cancer management from X% to Y% * Lower from 60%to X% theshare of lowerincome notattended by a women trained physician to Y% the * Increase from X% share of pregnant and lactating women completing micronutrient supplementation regimes dropin share of * 25% contracting puerperal wvomen (among thetotal infections number of diagnosed pregnancies) dropinshare of low * 10% birthweight newborns * Reduce matemal mortality ratefrom208 per 100,000 toX per 100,000 livebirths by 2005 * Lower fertility ratefrom 4.3%to X% by 2005

IBRD loan ($18million)

loan ($54million) * ABB ageby * KfW cofinancing reprodudive increasing substantially ($14.24million) ($1 3 theiruse of effective and * ECcofinancing million) sustainable health care * AIDAB cofinancing services. Specific objectives: ($10.55 million) development * LGU counterpart funds ($26.66 million)

* Ensure that90%of to Y% the * LGUs have themanagerial * Increase from X% qualityrange,Micronutrients, food * Improving with capocity to women referred share of women withaccess to andtechnical andaccess to women's fortifioation, immunizations, ore obstetric complications theproject implement motemal andprenatal kits, drugs and delivery health services RTI/STD monaged correctly supplies, health facilities care,family planning, inwomen * 25%increase cash flowto cancer oore, and * Adequate technical andcervicol renovation, acfivities referred andappropriately support project andmedical counseling assistance, decide to managed for RTls in theten * Ensure that X% of the * MostLGUs andlaboratory equipment to: population resides within Y km porticipate intheproject provinces increase inwomen * Sufficient andeffective * 20% of a health facility * Strengthen maternal for and managed coordination by thevarious detected thatX% of women core * Ensure andprenatal cervical cancer inthe 15 donors isdeveloped and carethrough a whosought facilities and services provinces weresatisfied maintained * Strengthen family heolth facility withtheservices services plOanning * Implement RTI/STD prevention management program * Design andimplement cervical cancer screening program

continues on thenextpage. Table

40

Thbl 6 (continu.d)
INPUTS (Resources provided for project activities) OUTPUTS (Goods and services produced by the project) RISKS AND CRITICAL ASSUMPTIONS (The outcome is dependent on... )

OBJECTIVES

OUTCOMES AND IMPACTS (of project activities)

* Strengthening LGU Technical assistance, * Increase to X% onthe-job * Skills leamed in training * Ensure thatless fhan25%of capocity to manage health workshops andtraining, competency levels of staff programs by midwives health focilities report stock-outs services andDOHcapacitybroadcast time, office trained inclinical, diagnostic, andbarangay health of essential dwugs to provide technical, finon- andother equipment, counseling, interpersonal, workers areused * Reduce cost of logistics to cial,andlogistical support travel, salaries honoraria,communication, historr effectively ascommunity less than8% of value of incremental operating taking, and referral outreoch tools commodities costs, supplies, computerpractices * Midwives andbarangay hardware andsoftware, * Increase to X% the health workers arenot warehouse construction timeliness andaccuracy tooaverburdened with andrenovation, vehicles,of procurement anddistribution additional patients to logistics, andmaterial of drugs, reagents, andother deliver quality services production publication, medical supplies * Local contractors and anddistribution contracts consulting firms have the to: technical expertise to conduct * Design andimplement studies in-service training program forpublichealth care workers * Establish andoperate project monagement office * Strengthen notionol publichealth andlogistics system * Increasing local * Designed andimple- * Form Xcommunity andNGO involvement in mented household-evelpartnerships toempower women's health programs community outreach women andtheir communities programs to improve their health * Designed andimple- * Provincelevel LGUs conduct mented integrated infor- Xtraining programs mation, education, and * Fund X proposals communication strategy submiffed by NGO * Designed andimple- partnerships at community mented community level partnerships program * Improving knowledge bosefor health policy formulation
* Consultants, studies, * Ensure thatat least 80%of

those surveyed areawareof onepreventive intervention for RTls, twosymptoms of RTIs, and onecomplication dueto RTI. * Ensure that at least 90%of women surveyed know about antenatal Tr injections andiron folate supplements

monitoring, and evaluation

* Complete X studies on thecosteffectiveness of service delivery options * Improve health policy framework leading to increased cost-effectiveness of various options to deliver health services to women * Establish protocols for better detection anddiagnosis of RTIs andeorly cervical cancer

Task managers: Stanley Scheyer, Maria Dalupon, andRama Lakshminarayanan.

41

Tab7. Sommnry ofobletWvs _I key perfomance idiators, Venezuela Aglcultural Extension Proec (Staff Appraisal Report 13591-VE, March 7,1995)
INPUTS (Resources provided for project activities) OUTPUTS (Goods and services produced by theproject) RISKS AND CRITICAL ASSUMPTIONS (The outcome is dependent on ...)

OBJECTIVES Helppoor farmers raise theirproductivity and incomes

OUTCOMES AND IMPACTS (of project activitiesl

* Farmers learn about and adoptnewtechnologies (measure: Xbeneficiaries adopt at least onetechnology recommended by the extension servicel * Extensionists learn new technologies fromuniversities andspecialists (measure: X training events andfielddays eachyear, withYextensionists participating) * Applied research conducted to address farmer's expressed needs (measure: Xfieldtrials conducted by specialists together withextensionistsl * Higher incomes for90,000 beneficiaries (measures: Xh 0 of participating farmers have yield increases overprevious period, average yieldincrease Y/ of participating farmers increase their netincome over previous period) * Economic growth in rural communities (measure: average netincome change over pevious period of participating famers) * Less environmental domage (soildegradation, erosion, runoff) caused by poorfarmers (measure: Xpotson participating farms with environmental improvement overprevious period

* IBRD loon1$39million)* Establish extension offices * Availability of counterpart * Government funds ineachstate andmunicipality funding ($21 million) (18 states and 180 * Sufficient administrative * State government funds municipalities) coordination in Helpsmall famiers ($11 milionl * Provide 50 training programs establishing a new improve theenvironmental* Municipal governmentforextensionists eachyear nationwide program sustainability of their funds ($6 millionl * Provide fourextension visits agricultural activities * Farmers funds to eachform each year, ($2million) covering 90,000 farmers * Fund at least twm annual visits Funds will beused bysubject motter specialists to trainagricultural to every municipality's farms extensionists, establish andextension offices newextension offices, * Sponsor onefielddayeach andprovide extension yearin every state visits to farmers * Sponsor twofieldtrials each yeorin every municipality * Establish Farmers associations inevery municipality * Coordinate montlymeetings of extensionists andsubject matter specialists ineachstae

Task manager: David Nielson.

42

Table S. Summary ofobledives and key perfonmance indiktors, Chad: Strudural Adjustment Credit (President's Report P-6785-CD) INPUTS OUTPUTS (Resources provided IGoods and services forproject activities)produced bythe project) RISKS AND CRITICAL ASSUMPTIONS iThe outcome is dependent on ... )

OBJECTIVES Resbore publik sector capacity and improve pubik secor finances Civil service reform

OUTCOMES AND IMPACTS (ofproject activities)

* $30million

* Complete acivil service * Norenewal inpolitical unrest * Reduce civil service vmge * Capocity-building census * Sufficient institutionol capocity bill from 5.8% ofGDP in project ($9.5 million) * Abolish automoaic wage * New administration follows 1995 to5.1 %in 1998 increases through onprogram * Prepare anorganizational chort for civil service * Introduce single taxpayer code * Reform special enterprise regimes inline with regional economic agreements * Renegotiate customs treatment forprivate firms and public enterprises
e

Public finance rehabiliiation

* Increase revenue from 9% ofGDP in 1995 to 11.5% in1998

international trade taxes e Establish exoneration quotas forimports bydiplomats Increase development and poverty alleviation impact of public spending Demobilization Public expenditure management
* Reduce army size, with assistance for resettlement * Abolish off-budget operations * Eliminate arrears to domestic suppliers * Reach agreement with * Cut army from 47,000 to

Centralize collection of all

21,000 soldiers

Development ofpublic expenditure program

for 1996 andcapital for 1996-98

IDA on1996 budget and program

1996-98 public investment

In1996 budget: * Limit wages and salaries

to5.7% of GDP * Limit goods and services to 2.6% of GDP

expenditure program

* Lmit overall current expenditures toless than 13.5%ofGDP * Limit capitol expenditure toless than 14% ofGDP Table continues onthe next page.

43

Taleb 8 (cont0Ined)
INPUTS [Resources provided for project activities) OUTPUTS IGoods and services produced by theproject)
* Increase resources for

OBJECTIVES Improved resource allocation to social sectors

RISKS AND CRITICAL ASSUMPTIONS (The outcome is dependent on... )

OUTCOMES AND IMPACTS (of project activitiesj In 1996budget: * 26% increase inallocation to educotion, heolth, ond women andsocial affairs over 1995 level (20% real increase( * 43%increase in road maintenance, from CFAF 2.8 billion in 1995to CFAF 4.0 billion in 1996

primary health, primary education,welfare of women and children, androadmaintenance

Create favomble nvironmient for prisvte seor gwth Strengthened judicial andregulatory environment
* Simplify procedures fr licensing businesses * Amend Articles 116 and 142of laborcode - Simplify investment code * Amend thebusiness code, including compony and bankruptcy laws * Ratify OHADA * Sellpublic shares of BTCD andBDT * Sell Meridien's share inBMBT * Reduce public sector membership on theboard of CNPS * Prepore action planto settle government's debtto CNPS andCNRT * Liquidate SONACOT, BICIT Coisse Sucre, andFIP * Prepore Hotel duChari,BIEP, INT, andONHPV forprivatization * OfferSONAPA, SOTEC, BDT, BTCD, andBMBT for privotization * Initiate divesiture strategy for STEE ONPT, TIT, COTONCHAD, andSONASUT * Launch COTONCHAD
* Increase producer pricefor firstgrade cotton from CFAF 120 perkilogram to CFAF 140 perkilogram * 10% annual increase in cotton exports between 1995 and 1998

* Increase pivateinvestment

from1.4% of GDPin 1995to 4.5%in 1998 (excluding impactof Dobaoil)

Financial system restructuring

State enterprise divesiture

Cotton sector reform

human resource development strategy * Improve producer pricing * Adopt twopartcotton producer pricing mechanism

Task managers: Amadou CisseandEmmanuel Akpa.

44

Table 9. Summary ofobledives and key performace Indiators, Morocco: Finandal Markets Development Loam (President's Report P-6633-MOR)
INPUTS OUTPUTS (Resources provided (Goods andservices for projedactivities) produced by theproject) RISKS ANDCRITICAL ASSUMPTIONS (The outcome is dependent on... )

OBJECTIVES Treasury financing

OUTCOMES AND LMPACTS lof project activities)

Enoble Treasury to finance * IBRD loon($250 itself at market terms million)

* Eliminate mandatory Treasury * Nonagricultural GDP grows* Increase ratioof Treasury

bondto sight deposits ratio

from 2. in 1995 to 3.2% bondauctions to total Treasury in 1996, to 3.6%in 1997 domes6c debtfrom 25%in 1995 to 30%in 1996, to 35%in 1997
* Ratio of debtoutstanding to GDPfalls from 221%in 1995 to 200% in 1996, to 182% in 1997 * Current account deficit/GDP

Increase liquidity of * Technical assistance* Adoptlw on negotiable public securities and * Debt management-securities strengthen secondary $50,000 (France; * Issue Central Bank circulars markets for Treasury bonds on Treasury bondtenders Eliminate remaining taxbias Strengthen public debt management system
* Accounting* Eliminate fiscal incentives

$75,000 (Froncel

onTreasury bonds
* Complete training program

ratiofalls from3.5%in 1995 to 2.9%in 1996,to 2.8%in 1997

* Budget deficit/GDP ratiofalls from 3.5%in 1995 to 3.0%in 1996, to 2.5%in 1997 * Fixed capital Formation/GDP ratioincreases from 21.2% in 1995t 22.0 in 1996, to 22.8 in 1997 * No deterioration inbanks' Finoncial condition

Indirect monetory conto Implement indirect monetary control policy * Central Bank sets bose rates * Issue Central Bank circular estoblishing terms andconditions of its interventions inmonetary market * Calculate reserve requirement ona dailybasis
* Eliminate ceiling on lending rates * Limit ratioof Central Bank

advance rateto interbonk ratemaximum spread to 2% through 1997

Liberalize interest rates Capital market developrmnt Diversify financing sources forprivate firms, strengthen accounting fromework, anddeepen capital markets

* Technical assistance * Issue decrees on stack for a central depositoryexchange anddelivery and * Implement accounting poyments systemframework $1 million (Francel * Present draftlow oncorporations to House of Representatives * Approve rules governing national council foraccounting * Present drah law on preporation, publication, and cerfification of consolidoted financial accounts * Eliminote government guarantees ondomestic bond issues by public enterprises * Develop occounting rules forinsurance companies

* Increase corporate bond issues from 0 in 1995to DH 2 billion in 1996, to DH4 billion in 1997 * Incrrease ratioof stock market capitalization to GDP from17% in 1995to 19% in 1996,to 23%in 1997

Table continues on thenextpoge.

45

Te 9(Cethied)
INPUTS (Resources provided for project activities)
* Technical assistance for

OBJECTIVES Increase institutional savings

OUTPUTS (Goods and services produced by the project) * Compete study of and achon planon potential role of institutional investors
* Define trading fees

RISKS AND CRmCAL ASSUMPTIONS (The outcome is dependent on... )

OUTCOMES AND IMPACTS (of projectactivities) Same as above (increase corporate bondissues, stock market capitalization)

contractual savings$600,000 (Francel

Encourage investors to * Technical assistance trodesecurities onrfloor onstock exchange andincrease transparencyoperations, upgrading, of stock exchange andtroding (CIDA) operations Strengthen securities commission andstock exchange Bknmng syslen Accelerate prvatization program Establish foreign exchange market

* Implement action planto strengthen securifies commission andstock exchange

* Sell state's shares inBCP,

BNDE, andCIH
* Announce creation of forex

market * Issue circular establishing forex market andmodalities of operafions * Issue accounting standards forforeign currency operations by banks * Eliminate initialminimum IDH 100,000)required to opennonresident Moroccan accounts in foreign currencies * Increase foreign currency limits of exporters of goods and srvicesfrom 5%and 10% to 20%
* Develop offsite andon-site

Strengthen bonk supervision

supervision * Submit regular reports prepored by independent external auditors oertified by theCentral Bank

Task manager: Emmonuel Forestier.

46

Distributors Oi ~ DiShibUtors of W orld Bank v". n Publications

Prices and terms wry Prircs ndcredit redt cutersvry. fToam country to country.
Consult your local distributor beforeplacing an order
ALBANIA AdnmLid. Plerl Rexheip Slr. Pal 98 StLI Ap1 4 Team Ta (42) 2741922172 Fax (42) 27419 ARGENTWA Oticina delLie trternrtorat AvCordoalSi7 Il2DBurmnAkes TtL (I) S1543156 Fax (1)8154-4

F.wCHNA Chra Frralciat6Eronox

P&aarroc SA 35 StnamStr.

L
VlaDu

CosrruTocnau Sansor SPA D Carasbr 1/1

Navesan hronrnabnCenter Book Depnmarnte

SINGAPORE,TAIWAK tYANMAR BRUN

PuthAngHoLsce 8~~~~~~~~S DaFoSiDonsgJA Beyaqo

10682 AEhenri Tel. (1)%4-IM2 Fx: (1)364-8254

Ctle PosWeS52 50125 Frrene 645-415 Tel:(55)

F(55)641U257 FaGKNdAL

PO. Bo er.ad N48D020dso6 Tel (22)57-3300 Fax (22}68-1901

Pa12tMeEtsahaP W"Asia Pacific Pie.Lid 41 KA"rPdds Road#04-03

Van Diarrms E r WerM Ch. de Laa 41 CH18D7Ebn Tel(021) 943267 Fax. (021) 94338S

Fac (56)10-401-7345 Tel (56) 10-333-8257

COLOMBIA kdocanwe Ltea. Apetado Aersa 34270 Tel (1)205-2790 Fax (1) 285-2798 COTE DW1OIRE CantecEdism atdeDdilom Kte s (CEODA) 04B.P.541 Abeae 04 Platu Tel:225-24t610 Fee 225-25-0567

KONG, MtACAG HONG Asia 25 Ltd

Seles &C4culabon Depeatrnt Seabrd House. unil1101-02 Wrdne Streea Cenate 22-28 HorgKong Tel:952253D-1409 Farx8522526-1107 URL: ht0pJAr.saie@seasia2DDD.oo.hk HUNGARY Founat tor t4aet Ecorny D)ardbsal ( 17-19 H-ll BII7 iBudvpt Tel: 36 12042951 or 3612042948 Fu: 3601 2042953 tIA AbedPutus Ltd. 751Mouril Rad Madas -50002 Tat:(44)852-393 Fax (44) 852-649 tNDONESSA p1 ksdeLiniLed 20 Jalan BomabudLr PO. Boa 181 Jeana 10320 Tel:(21)390-4240 Fax: (21)421-4289 tRAN Kra K Putibs PO.Boo19575-511 Talrs Te: (21) 258-3723 Fax98(21)258-3723 KOW Sae Ca.Puths Khaled EslasLb Ave6tnSbeel Kuarheh Deelmo ND. 8 Te*M Td: 871719or8716104 Fax 8682479 E-rca.tralatseagneOdocdhr tELAUND Gvae Suppiess Agency Oftig an tSoM 4-S Haro4r Road DrtAn 2 Tel:(1)461-3111 Fax (1)475-2670 RAIEL Yoresol LIa e Lbi P.O. Box 6056 Tel Aviv 61560 Tet:(3) 5281397 Fax (3)5285-307 FO.Y.Ineren POBox 13056 TelAvri61130 Te()9 5461423 Far (3)541442 Pattie Aedspib ES tdres birrr enSie P0.B 19562 Jerral Tel- 21271219

JAMAICA

PAKISTAN

349316 Swngapore G aok Whee Biiding Tel. (65) 741-5166

Fax (65)742-9356 a-mae. arhgaleOas riaors cOn REPUBLIC SLOVAK SovalrG.TG. Lid pnttWea 4 POBox 152 852996bsdeva S Tel:(7) 39472 Faxa (7639485 SOUTH AFRCA, BOTSWAANA For me eas. Oxtord Unsereny Press SnAham Atrnc PO Box1l41 Cape Tossn W TexI (21)45-7266 Fa- (21)45-7265 Forebtc n orders. ntnMeralitDa SArdcnpbon Serve P.O. Bax 41095 Crall" Jarrebg 2024 Tet.(11)88D 844 Fax:(11) 880-6248 SPAIN Musni-Prrrea Lienos. SA CaslaloS37 28DD1 Maidd Te: (1) 431-3399 Fax (1)575-3998 ht9pJA BalasiArPesa Matrd-Praen Bardona CoaredeCerl, 391 0O9 BarO Tat:(3) 4-30DM Fax (3(487-7659 SRILANKA THE MALDES LakeHorse Bookshp PO.Box 244 155, Se 9 rirtkA Gaedinw Maretr Cobnibo 2 Tat.(1)32105 Fax (1)432104 SWEDEN Fdza Cua m Service Regaking l2 S-10647 Shddcir let )69D9089D Fax () 214777 Wuweeal-WituirAS P.O.Box 1306 S-171 25 Soae Tet: (8) 70F-7-50 Fax (8)27-071 SW6YZENLAO L4x iiePayol Serniickub trSw9 Ce4-brte 30 1002 LauttxW iat 028441-3229 Fax (021(3414251

TANZANIA Ordod Ureausit Press

mottia Stre PO Box 5298 Dar es Saealra Tel:(51) 292D Fax(51)46922 THAIAND Carel Batooks L3b6tn 306Silom Road Bwtkok Tel.(2)235-540 Fea (2)237-8321 JAIL TRtNIDAD & TOBAGO, Syl ah Saedis UInd tl9WadsSbreel Curepe Tbi, West lAn Tel W6-68-5654 Fax: 80966625654 UGANDA GLAso Ltd MaCrar Bi*dg PO Box9897 PM 16/4 Jqa Fd. Krrab TeV/Fax: (41) 254763 UN1TED KV3 WM Maods Ltd PO.Box 3 AMorl Hwalne GU34 2PG England Tel (1420)86840 Fax (1420)8 88M ZAMBIA Uarsty Bookshop GrCt EaSRood CUT PO. Box 32379 Luab-k Td (1)213221 EaL 482 ZIMBAt1E LA.rrnWTFZimabwe 0.)aW Road. Asbnar Troule PO.Bo.STl25 Sorenon HarMu Tel:(4) 6216617 Fax: (4) 621670

Ian Rardte PLbtshs Lid 206Old Hope Road Kinslon6 Tel: 809-927-2095 Fu 809-977-0243 JAPAN Easterrn Bok Servce Hrsgo3-CSane, Burkyo-ku l3 Tokyo Tel (03)3918-1966 Fax (03) 3018-0664 UFL hItpiMwbekko&.or.p(/-svI-ebs KEYA Atrica Beak Samoc (EA) Ud. CUrs House, Wna o Sueet PO Box 45245 Natob Tel: (2)23641 Fax: (2)330272 KOREA, REUBLIC Of Deo TdhivgCo. Ltd. P.O. Br 34 YaDida Seiu Tel: (2) 735-1631/4 Fa (2)784-0315 MALAYSIA Unrery at Maya Coopereli Rookiop,Ulred PO.Box1127 JaenP PbuamBr 59700 Kuaea Luirpur Tel:(3)756-50D Fax (3)755-24 MEXCO WFOTEC Apauloo Postal 22-660 14060 Tiper, MetxioD.F. TdW: (5)60-011 Fax:(5( 624-2822 NETHERLANDS De Lnetboomt nOr-Pudbtkalad, P.O. Box 22 7480 AEH-ksbaerge Te1: (53)574-604 Far (53)572-9296 NEW ZEALAND EBSCO N12 Ld. Prva5b MelBag89914 New el Aucland Tet:(9)524-8119 Fax 9)524-8C67 NIGERIA Uriiseady Pre LUmed TlseCwOwBuildingJetldro Priate Mod Bg 5095 budan Tat:(22)41-13506 Fasx (22)41-2856

MArza Book Agasey 65,Shrlerah-ea e-Azemn Box No 729 P.O. Laxare 540D Tel (42) 7353661 Fax (42)7565263 Ontord Urw ty Press Bangoa lor"n Sharae Faisat PCBox 13633 Karaehi-75350 Tel (21) 446307 Fax:(21)454-7640 E0-mad: oapO p kIeccmcootpk PERU Eddonit DesamrcoSA Apest 3824 Lman I Td4 (14) 2830 Fax:(14)28626 PItIJltES hlerninla BODIsotare Calmer oc SLAG 720 iyland10 Caxmorerhn Tower 2 H.V deda Cos"a corner Vaero SI. Matab, Mlts Mus Tat: (2)817-9676 Fax (2)817-1741 POLAND kieandiobrm Pddeg Serse Ul.PoleAs311/37 00-577 Werzrwe Tel.(2)62S-6089 Fax (2)621-7255 PORTUGAL LAmed Poulpt RoL DoCasino 70-74 120 LUsbo Tel:(1)347-4982 Fax:(1)347-024 ROMAIA Coewd DeLbmrr Bucuresi SAk Str.LcaO no. 26, ndra3 Baucdrst Ted: (1)6139645 Fax(1)3124000 RUSSIAN FEDERATION dweabe ovs Mr, go, Krovh* PereubL Moscw 101031 Tel:(95)9178749 Fax (95)917 62 59 SALD ARABIA, QATAR Jari Book SUore PO. Box 3196 Riyadi11471 Tat:(1)477-3140 Faxx(1)477-2940

CYPRUS Canrl or Appbed Farch Cyprus Carge 6,DiogesesStt Engo P.O. Boo 2DD AUSTRALIA, RJ, AMPUA NEW GLKNEA, Naesi Tl: (2)44-1730 SOLOMON SLAND, VAt41ATU, AND WESTERN SAMOA Fax (2)46-2051 DA Ilotrukalon Srces 6486WtalhoRoad CZECH RE BLtC lcartn 3132 NaIselotrsmaori CantWr Vire POd*i, K rska 5 Td. (61)39210 77T7 CS- 11357Prague I Fax (61)3921O 7788 TeL(2)2422-9433 URLtlinpAtwaradret.corLeu Fax (2) 2422-1484 URLtipJAwMwMW.r2 AUSTRIA Garld ansd Co DENMARK Graben 31 SwnkAram A-lOll Wb1 RoWF rerInr4 th 11 Td (1(51247-31-0 OK-1970FradbrnrergC Fax (1)512-47-31-29 Tel:(3l-351942 Fax (31(357822 JIGLADEBH Moo AstatO Deoeepmenwl EGYPT ARAB REPUBLIC OF AsAna SDiely(MIDAS) AtAhrrneu DOibm Agencry Hoa S.RoPd 16 AlGae Sas DlwOs t FVArea Cairo Dhake1209 Tel-(2) 57-0603 T. (2)326427 Fax:(2(576-633 Fua (2(81118I The bddt East Oter 41,StartSIre BELOWI JaantDe Lmb Cairo x A. duoir202 l2) Tel:(3973 10968Bet_ Faxr(2)393-9732 Tit 2)5385169 Fax (2) 531141 FAD Akeerwan Keka;;ppa IlAZL PO. Box120 Putae cnlariciYatkfrr mis FSN40101 HeSsnli It. (0)12141 WrL Rem Patb Gre,, 209 Fax:p) 121-4441 01409 SaoPW, SP. URLttpkoolinelztatln bal Tt (11)2z94 Fac III) 258-60 FRANCE UFLht.IpJawoLbk WortBan Preddte 66, ensu lire CAIIADA 75116 Paw Rard Pawt"igC. it Tet (1)40-JI14-56V7 1294 AJRom Fed Fax (1)40-88-30-56 Onms0ChnsKRB3W8 Tdt61-74143 GERMANY F&ac 613-741-5439 UNO-Vs URL, htMii1wabxrLC/-eufi Poppledler Ai 55 53115 Brm Tt MO 212940 Fax (228)217492

RECENT WORLD BANK TECHNICAL PAPERS (continued) No. 295 No. 296 No. 297 No. 298 No. 299 Marketsin Centraland EasternEurope Pohl, Jedrzejczak, and Anderson, CreatingCapital A Global Review BiomassGasifiersforHeatand Power: Stassen, Small-Scale Bulatao, Key IndicatorsforFamilyPlanningProjects Africa:FromAnalysis toAction in Sub-Saharan Odaga and Heneveld, Girls and Schools in Tropical Forestry Relatedto Participatory Tamale, Jones, and Pswarayi-Riddihough, Technologies Countries and Subtropical A Reviewof WorldBankExperiences Climates: RainfedAgriculturein Mediterranean No. 300 Oram and de Haan, Technologiesfor Africa Region,July 1987 to April 1995 Techinical Department, of Publications: No. 301 Mohan, editor, Bibliography in the UpperLeraba ImpactAssessmentof Settlementand Developmenit No. 302 Baldry,Calamari, and Yameogo, Environmental Basin in Count:WorldBank ProjectDesignsand the Qualityof PrimaryEducation No. 303 Heneveld and Craig, Schools Africa Sub-Saharan World Applicationsin RuralAreasof the Developing No. 304 Foley,Photovoltaic Africa Angloplhone and TrainingofAccountantsin Sub-Saharan No. 305 Johnson, Education The Roleof Internaland ExternalIncentives Perfornance: No. 306 Muir and Saba, ImprovingState Enterprise Research Participatory No. 307 Narayan, Toward Policyin Krakow:Can EconomicIncentivesReally No. 308 Adarnson and others, EnergyUse,Air Pollution,and Environmental Help? Supply and DemandBalances No. 309 The World Bank/FOA/UNIDO/Industry Fertilizer Working Group, Worldand Regional and Potash,1993/94-1999/2000 for Nitrogen,Phosphate, Control of the Onchocerciasis Settlementand Development No. 310 Elder and Cooley,editors, Sustainable ofa MinisterialMeeting Area:Proceedings Programme 1989-1993 No. 311 Webster,Riopelleand Chidzero, WorldBankLendingfor SmallEnterprises Instruments of Policies and Overview World Bank: An at the No. 312 Benoit, ProjectFinance TheEmergingRoleof the PrivateSector No. 313 Kapur, Airport Infrastructure: A Handbookfor of AgriculturalPriceand TradePolicies: with Ramos, Surveillance collaboration Schaeffer in Valdes and No. 314 Ecuador Making the Most of GlobalTrade and NationalTradeFacilitation: Technology No. 316 Schware and Kirnberley,Information Guide to Best Practice and NationalTradeFacilitation: No. 317 Schware and Kirnberley,InformationTechnology NationalAgriculturalResearch Toure, Strengtlening Abdi Noor, and Ayling, Dahniya, Ouayogode, No. 318 Taylor,Boukambou, Action Systems in the Humid and Sub-humid Zonesof West and CentralAfrica:A Frameworkfor No. 320 Srivastava, Lambert and Vietmeyer,MedicinalPlants:An ExpandingRolein Development and Development Conservation and Agriculture:Implicationsfor No. 321 Srivastava, Smith, and Forno, Biodiversity of Non-TimberForestResources and Management No. 322 Peters, TheEcology Economies in Transitionial of PublicEnterprises Governance No. 323 Pannier, editor, Corporate Programs Household Electrification Photovoltaic No. 324 Cabraal, Cosgrove-Davies, and Schaeffer,Best Practicesfor withiPowerGeneration Experience No. 325 Bacon, Besant-Jones,and Heidarian, EstimatingConstructionCostsand Schedules: Countries Projects in Developing of Health, Africa:The Convergence No. 326 Colletta, Balachander, Liang, The Conditionof YoungChildrenin Sub-Saharan Nutrition, and EarlyEducation A Handbookfor of AgriculturalPriceand TradePolicies: No. 327 Valdes and Schaeffer in collaboration with Martin, Surveillance Paraguay and AbsolutePoverty in Asiaand LatinAmerica No. 328 De Geyndt, SocialDevelopment Department, AfricaRegion,July 1987to April 1996 Technical of Publications: No. 329 Mohan, editor, Bibliography An EmpiricalAnalysis Eunmpe: andEastern No. 332 Pohl, Djankov,and Anderson, RestructuringLargeIndustrialFirmsin Central A Case of HealthInterventions: and the Cost-Effectiveness No. 333 Jha, Ranson, and Bobadilla,Measuringthe Burdenof Disease Study in Guinea

i~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

II

i%P~~~~~~

,i!I

-~~~~~~a

:~~~~~~~~~~~~~~~~~~u.

I~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ill

''ll,