Challenges of integrating evaluation into operations

Dinuk Jayasuriya, Development Policy Centre, ANU

Current state of play

Rigorous Evaluations FY11
World Bank: Over 300 IFC: 23 quasi-experimental and experimental studies

No dedicated unit; M&E Officer initiative

Presentation
Focus on RCTs Draw on my experiences at IFC/WB when discussing the challenges of integrating RCTs into operations

Challenges for RCT

Buy-in for the evaluation (eg: Tonga)
This is required from all stakeholders (donors/partner government/consultants/project team)

Continual support from senior management for the completion of a RCT

Overcoming the challenge

incentives for all parties involved.

Challenges for RCT

Buy-in for the evaluation (eg: Tonga)
This is required from all stakeholders (donors/partner government/consultants/project team)

Continual support from senior management for the completion of a RCT

Overcoming the challenge

Ensure adequate incentives for all parties involved

Challenges for RCT

Understanding why and how we are doing the evaluation (eg: Sri Lanka)
Avoid risk of contamination

Ensure sufficient financial resources.

Challenges for RCT

Understanding why and how we are doing the evaluation (eg: Sri Lanka)
Avoid risk of contamination

Overcoming the challenge

Have sufficient human resources including an independent in-country Field Coordinator

Challenges for RCT

Ensure sufficient financial resources (eg: Tonga)
WB evaluation budgeted 800K over two years My budget began at 100K and ended at 20K

Challenges for RCT

Ensure sufficient financial resources (eg: Tonga)
WB evaluation budgeted 800K over two years My budget began at 100K and ended at 20K

Overcoming the challenge

Ensure budget is sufficiently large, approved and

Challenges for RCT

Ensure sufficient human resources
Principal investigator

Evaluation technical partners

Advisors for evaluation design (internal and external) Econometrician

Support staff
IE coordinator Field coordinator Research assistant

Challenges for RCT

Ensure sufficient human resources (cont)
Local support staff
Advisor Manager

Data Collection
Data collection team Survey design leader

Partner coordination

Challenges for RCT

Ensure sufficient human resources Overcoming the challenge
Decide up-front the cost-benefit of having a smaller or larger team and the associated risks

Challenges for RCT

Requirements for a RCT unavailable
Statistical power (Tonga) Spillover effects (Kiribati)

Challenges for RCT

Requirements for a RCT unavailable
Statistical power (Tonga) Spillover effects (Kiribati)

Overcoming the challenge

Expert opinion Baseline survey data analysis

Challenges for RCT

Delays occur in interventions (eg: PNG).

Challenges for RCT

Delays occur in interventions (eg:PNG)

Overcoming the challenge

Certain level of access to senior management to suggest alterations to the evaluation/program plans given any delays

Brief project management team that delays may

Challenges for RCT

Evaluation is requested just prior to program implementation (eg: PNG)

Challenges for RCT

Evaluation is requested just prior to program implementation (eg: PNG)

Overcoming the challenge

People should be aware of the time taken to undertake evaluations Sufficient time is available to design the evaluation

Challenges for RCT

Ethical Concerns (eg: Tonga)

Challenges for RCT

Ethical Concerns (eg: Tonga)

Overcoming the challenge

Consider the impact of no intervention given the local context Consider if evaluation is justified given possible concerns Explain benefits of evaluation

Challenges for RCT

Managing expectations
Not all RCTs are administered successful Focus on short-term outcomes Average effect Results might be negative One RCT is not enough to demonstrate intervention success

Attrition

Challenges for RCT

Pacific specific challenges
Sample size/sufficient statistical power Administering evaluation across multiple islands
Pacific time Cost Coordination

Other Evaluations
Challenges of ongoing qualitative evaluations Challenges of ex-post evaluations

FY11 Annual Report

IFC Advisory Services expenditure approximately 200 million USD 30 per cent access to finance acts increased by 7.46 billion

FY11 Annual Report

IFC Advisory Services expenditure approximately 200 million USD 30 per cent access to finance Some Impacts
16.9 million people benefited from microloans 600,000 clients benefited from SME projects

Integrating evaluations
IFC and rigorous evaluations . still a long way to go.