United States Agency for International Development Bureau for Policy, Planning, and Learning

Evaluation of Policies, Reforms, and Other Major Activities

REVISED PPL EVALUATION DESIGN FOCUSING ON THE PROGRAM CYCLE

January 9, 2013

Extract on Evaluability Assessment

3 Identifying the Evaluation Focus
PPLs theory of change shows the wide range of products and services it provides to the Agency and USG partners. This section discusses the evaluation teams proposal to focus evaluation efforts on the Program Cycle and its core components: Agency policy and strategies; the CDCS, Project Design and implementation; evaluation and monitoring, all linked by a process of continuing learning and adapting, and better resource allocation decision, all for the single purpose of producing development results.

3.1 Evaluability assessment and identifying an evaluation design organizing principle

From June 2010 to June 2012, PPL undertook a range of important policies, reforms, and activities, and of these, identified 12 for inclusion (see Box 1) in the evaluation SOW to be considered by the evaluation team for their evaluability. PPL was interested in moving beyond its monitoring of and reporting on the various outputs of USAID Forward and the 20112015 Policy Framework to assess the effectiveness, utility, and extent to which [these] 12 policies, reforms, and activities were influencing USAID planning, programming, operations, and budgetary decision making, especially in USAID field Missions.
Box 1: The 12 Policies, Reforms and Activities Identified in the Evaluation SOW 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Drafting and release of Policy Framework Revival of country strategy planning/CDCS Revival of Project Design process Engagement of broader Donor Community Donor engagement toolkit Development and implementation of USAIDs Evaluation Policy Launch of Washington-based GeoCenter for geospatial mapping Launch of Grand Challenges initiative Establishment of a formal policy development process Completion of Agency-level policies and strategies Institutionalization of adaptive learning throughout USAID Ongoing provision of technical support on evaluation, Project Design, and strategic planning

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC

A principal task of the evaluation design phase was the development and application of a methodology to assess the evaluability of each of these 12 reforms and activities listed in the SOW (plus any others the evaluation team deemed relevant). The evaluation team assessed these reforms in terms of how well each reform met three evaluability criteria: 1) objectives and desired results clearly articulated; 2) feasibility of measuring results at this time; and, 3) evaluation findings likely to be used to make changes at this time. Some PPL staff noted that some of the 12 policies, reforms and activities listed in the evaluation SOW did not represent the most important work of some PPL offices; rather, they represented discrete initiatives that while useful, were not always well integrated into the broader efforts of the concerned PPL Office and the changes in behavior and ways of doing Agency business that were at the core of its change agenda. In addition, the significant diversity of these 12 activities and the many stakeholders targeted by them did not provide an organizing principal for a coherent evaluation design effort. For some of these reforms, policies and activities, either inadequate time had passed or the complexity of measurement was too great given time and resource constraints. Annex B provides a summary of the findings the team produced with respect to the evaluability of each of these 12 reforms.

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC

Annex B: Evaluability of 12 PPL Policies, Reforms, and Other Major Activities

PPL Reform or Activity
*in order as listed in SOW

Are objectives and desired results clearly articulated? Objectives: Provide staff and partners worldwide with a clear sense of core development priorities that translate Presidential guidance on global development into more detailed operational principles Encourage and inspire Missions and partners to focus relentlessly on achieving and measuring development impact Desired results: Enhance coherence, focus, and achievement of development results across USAID Objectives: Reinstitute the development discipline of strategic planning into the development cycle Bring rigor and evidence-based decision making to the development of country strategies with a focus on results Desired results: Maximize the impact of development cooperation in line with the Quadrennial Diplomacy and Development Review (QDDR) and the Presidential Policy Directive on Global Development (PPD-6) Objectives: Identify and realize when and how best to achieve development results in the most effective manner, through application of rigor and best evidence, broadening the range of implementing options, incorporating learning, using peer review, and including collaboration and mutual accountability

Are results measurable at this time?

1. Drafting and release of Policy Framework

Too early to measure impact on development results within the Agency or its impact on partners development programs Possible to measure the utility of this document in shaping Country Development Cooperation Strategy (CDCS), evaluations, and strategic partnerships in the field and at the Bureau Possible to measure knowledge of the Policy Framework among partners, and whether they have made use of it in their own priority setting Too early to measure the impact of CDCS on improved development results Possible to assess, for the currently completed CDCS, how well Missions incorporate existing evidence, follow CDCS guidance, set priorities, incorporate PPL policies and strategies, and achieve a quality product Possible to assess perceptions of CDCSs contribution to improved program planning in Missions Too early to measure impact of Project Design process Possible to measure compliance with Project Design process and guidance through review of Project Designs completed to date Perceived usefulness of the design process

Will evaluation findings be useful to make changes at this time? Level of knowledge, understanding, and use of the Policy Framework within the Agency and outside can inform improvements in communication strategies

2. Revival of country strategy planning/CDCS

3. Revival of Project Design process

As the CDCS has been highlighted as a principal reform in the return of USAID to a culture of disciplined development, it is important for it to be included in the evaluation. Even if there is no intent to revise CDCS guidance further, findings can inform needs for future technical support (see 12. below) The results would be preliminary here, given the recent release of this guidance, but could well inform improvements in the guidance and in the support processes needed (see 12. below) 3

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC

PPL Reform or Activity

*in order as listed in SOW

Are objectives and desired results clearly articulated? Desired results: Increase the effectiveness of development interventions and maximize the impact of limited resources Objectives: Promote the Agencys international policy leadership role among other USG agencies, development partners and donors, and host governments Improve program quality at Mission level through better engagement, coordination with and participation in joint development partner efforts Desired results: Focus the international development effectiveness agenda squarely on achieving results, committing to higher standards of accountability and transparency, increasing the impact of USG cooperation and advancing global prosperity and security

Are results measurable at this time?

Will evaluation findings be useful to make changes at this time?

4. Engagement of broader Donor Community (including Busan)

and guidance at Mission level could be measured for those who have undergone this process Too early and possibly difficult to systematically evaluate the impact of USAID broader donor engagement on other donors actions Possible to evaluate the level of success in changing USAIDs image as an international development policy / thought leader within the broader donor community, including perceptions of its role and effectiveness in a number of targeted international forums (e.g., Busan) Possible to evaluate at the field level how Missions have incorporated broader donor engagement agreements into processes and programming at the country level Too early to evaluate impact of toolkit on development results or Aid Effectiveness Possible to measure whether Missions and Bureaus are aware of the toolkit, have accessed it (used it), find it valuable in helping them to better engage with donors in their given contexts, and in decreasing the administrative burden of partnerships start-up, and whether they have taken new steps to increase donor engagement Too early to evaluate full implementation of Evaluation Policy and its results, as no countries have completed the new Program Cycle, although there may be some cases of learning from evaluations conducted post-

Not clear what actions will be taken at this stage, based on perceptions of USAID leadership in international and interAgency policy Level of Mission integration of Office of Donor Engagement into programming could inform support needed to improve or enhance how well Missions are aligned with USG agreements at the international level Level of current awareness and use could inform improvements in the tool itself and communicate and support strategies with Missions and Bureaus

5. Development of donor engagement toolkit

Objectives: Increase knowledge throughout USAID on types of donor partnerships possible Assist Bureaus and Missions to be strategic and take advantage of such partnerships Desired results: Missions and Bureaus create and manage donor partnerships to undertake joint activities as a means to improve coordination, leverage resources, and improve Aid Effectiveness Objective: Re-introduce culture and practice of rigorous evaluation into the Agency Conduct 250 quality evaluations by 2013

6. Development and implementation of USAIDs Evaluation

Evaluation of policy development process itself probably of limited value

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC

PPL Reform or Activity

*in order as listed in SOW

Are objectives and desired results clearly articulated? All Missions and Bureaus design/commission impact and performance evaluations Desired results: Agency is learning from its own programming experience and from the evidence base Development programming shifts due to results from evaluations Objectives: Expand and institutionalize use of geospatial tools and analysis to support open and transparent data efforts, evidence-based development (planning and monitoring), and strengthen inter-Agency cooperation Desired results: Better coordinated and targeted development projects, more effective and efficient use of resources (by USAID and in collaboration with other partners) Objectives: Leverage innovation, science, and technology to create transformational, scalable and sustainable changes with potential to improve efficiency and decrease cost of development Desired results: USAID incorporating creative approaches to programming; resources leveraged from other donors for transformational, scalable solutions; higher USAID profile that leads to global action; transformational innovations implemented at scale; improved development results

Are results measurable at this time?

Policy

policy Possible to measure current level of evaluation culture, application to date of Evaluation Policy within Missions and Bureaus, and perceived capacity to follow evaluation policy Too early to evaluate full impact of GeoCenter on USAID programming or AID transparency (not enough use to date) Possible to measure current level of knowledge about and use of the GeoCenter capacity by Missions and operating units

Will evaluation findings be useful to make changes at this time? Level of current understanding of, commitment to, and capacity for evaluation can inform a way forward

7. Launch of Washingtonbased GeoCenter for geospatial mapping

Level of current understanding and applications can inform a way forward

8. Launch of Grand Challenges initiative

Too early to measure the effects of Grand Challenges on changing the development landscape, as 2011 grants will not have had time to be completed, disseminated, and thus, scaled up by others to any large degree Possible to measure such aspects as bringing in new actors (both in applications and in recipients to date); how the Grand Challenges for Development are perceived among key technical players in those fields and among Missions and operational units in USAID; how participating in the Grand Challenges has had short-term effects on (successful and unsuccessful) applicants, and how other contributing donors feel about their investments to date Possible to review the PTT process and newer

While gathering information about perceptions to date could be of value to improving the program, it is too early to see the effects of these grants, and it may be more warranted to design a longer-term evaluation strategy

9. Establishment

Objectives: Identify areas where Agency-wide

Comparing the two

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC

PPL Reform or Activity

*in order as listed in SOW

Are objectives and desired results clearly articulated? policies or strategies are warranted Steer creation of new or revised policies or strategies that will guide all parts of the Agency Desired results: Policies built on field experience and broader evidence base Ownership and champions for policies developed within the Agency

Are results measurable at this time?

of a formal policy development process via Policy Task Teams (PTTs)

process (re: resilience policy development) to examine what has been lost and gained in the new process. This would require clear articulation of priority qualities of a policy development process to evaluate: Efficiency? Sustainability? Inclusiveness? Ownership? Understandability by those who will use it? Too early (and possibly difficult) to examine impact of development process on uptake of policies themselves since each policy has its own parameters Probably too early to assess impact of policies on those outside the Agency in any systematic way, although could assess knowledge of and reactions to USAID policies Possible to review Program Cycle products from early stages (CDCS, Project Design) for application of policy guidance in priority setting and intervention selection Possible to evaluate how aware Missions and regional Bureaus are of these policies, how they made use of them, and what difficulties they found in using them Cannot evaluate institutionalization yet, as learning cycle was only recently introduced and integrated into Program Cycle Possible to measure current level of learning culture, application to date of learning agenda in selected Missions and Bureaus, and successes and issues related to application

Will evaluation findings be useful to make changes at this time? processes could inform how the new process could be adjusted to be more effective

10. Completion of Agency-level policies and strategies

Objectives: Provide Missions and Bureaus with cutting edge, evidence-based policy and strategy guidance to help fill major development/sector gaps with credible policies Provide Missions and Bureaus with standardized and evidence-based guidance on each element of the Program Cycle Desired results: Application of policy guidance within an evidence-based, sustainable and relevant Program Cycle based on sound policy and strategy guidance Other USG agencies and donors build their policies based on USAIDs leadership Objectives: All USAID Missions, Bureaus and Offices develop a learning agenda and use results of learning to improve programming Desired results: Programs aligned/adapted to dynamic context; increased accountability; strengthened technical knowledge base; increased relevance and sustainability of programs for greater effectiveness

As PPL has a list of at least six new policies, findings about incorporation of policy guidance could inform improvements in communication of and support for policy application

11. Institutionalization of adaptive learning throughout USAID

Level of current understanding, commitment, and constraints to learning agenda can inform a way forward, although there is limited experience to date

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC

PPL Reform or Activity

*in order as listed in SOW

Are objectives and desired results clearly articulated? Objectives: Build capacity in Missions and Regional and Pillar Bureaus to follow Program Cycle guidance Desired results: 80 Missions adhering to the Program Cycle to effectively develop, implement, and improve development programming

Are results measurable at this time?

12. Ongoing provision of technical support on evaluation, Project Design, and strategic planning

Too early to evaluate full Program Cycle at this stage, as no Mission has completed it Possible to measure current access and use of technical support (including tools such as ProgramNet), perceived capacity to implement Program Cycle components, effective implementation to date (through document review) and satisfaction with technical support

Will evaluation findings be useful to make changes at this time? Effectiveness and scalability of current technical support is an important question for expanding the Program Cycle application within all 80 Missions Information on what has worked well, what can be built on and possible strategies (retail to wholesale approach) can inform a way forward

USAID PPL Evaluation Design | January 9, 2013 | EnCompass LLC