Prepared for CARE Nepal

Projects

Handbook
for
Participatory Evaluation
(For Internal Project Evaluation)

Prepared by:
Bishnu Pokhrel
April 2001
 TABLE OF CONTENTS

ABBREVIATION........................................................................................................................................................5

ACKNOWLEDGEMENTS ........................................................................................................................................6

1. INTRODUCTION AND BACKGROUND............................................................................................................7

2. EVALUATION MODEL AND CONCEPT OF INTERNAL EVALUATIONS...............................................7

2.1 AN EVALUATION MODEL ....................................................................................................................................7
2.1 PROCESS OF INTERNAL EVALUATION ..................................................................................................................8
3. EVALUATION WORKS........................................................................................................................................9
3.1 DECISION TO EVALUATE......................................................................................................................................9
3.2 PREPARING THE TERMS OF REFERENCE (TOR) ..................................................................................................10
3.3 SELECTION OF EVALUATION TEAM ...................................................................................................................10
3.4 PREPARING EVALUATION PLAN .........................................................................................................................11
3.5 CONDUCTING THE EVALUATION ........................................................................................................................11
3.5.1 Steps to be followed: .................................................................................................................................11
3.5.2 Writing the report......................................................................................................................................12
4. RESPONSIBILITY OF PROJECT STAFF AND COMMUNITY TEAM MEMBERS ................................13
4.1 STAFF SELF EVALUATION ..................................................................................................................................13
4.1.1 Facilitator/Team Leader:..........................................................................................................................13
4.2 COMMUNITY EVALUATION ................................................................................................................................14
4.2.1 Project Staff (Observer and Co-facilitator): .............................................................................................14
4.2.2 Facilitator from the communities ..............................................................................................................14
5. QUALITY REQUIREMENTS.............................................................................................................................14

6. DATA COLLECTION METHODS.....................................................................................................................14

6.1 PROJECT STAFF SELF EVALUATION ...................................................................................................................14
6.2 COMMUNITY EVALUATION ................................................................................................................................15
7. USE OF EVALUATIONS.....................................................................................................................................16

8. DISSEMINATION OF LESSONS LEARNED...................................................................................................16

9. FEEDBACK OF LESSONS LEARNED .............................................................................................................17

REFERENCES ..........................................................................................................................................................18

ANNEXES ..................................................................................................................................................................19

ANNEX 1. INTERNAL EVALUATION MODEL .................................................................................................19

ANNEX 2. PROCESS OF INTERNAL EVALUATIONS .....................................................................................20

ANNEX 2.1 USE OF DIFFERENT PRA TOOLS .............................................................................................................20
ANNEX 2.2 DIAMOND RANKING PROCESS ................................................................................................................21
ANNEX 2.3: SECONDARY DATA REVIEW .................................................................................................................22

3
ANNEX 3. IMPORTANCE –SUCCESS MATRIX AND FLOW DIAGRAM ....................................................23

ANNEX 4. FLOW DIAGRAM................................................................................................................................26

ANNEX 5. DIAMOND RANKING TO FIND OUT STRENGTHS AND WEAKNESSES................................28

5.1 DIAGRAM OF DIAMOND RANKING .....................................................................................................................29
ANNEX 5.1 PROJECT APPROACHES AND PROCESSES EVALUATION ..........................................................................30
ANNEX 5. 2 EVALUATION OF PROJECT SUPPORT SYSTEM .......................................................................................32
ANNEX 6. SOCIAL AND RESOURCE MAP .......................................................................................................34

ANNEX 7. RESOURCE LOCATION AND PIE DIAGRAMMING ....................................................................37

ANNEX 8. TIMELINE..............................................................................................................................................41

4
Abbreviation

AHW Animal Health Worker

ANR Agriculture and Natural Resource Management

BTRT Begnas Tal Rupa Tal

CBO Community Based Organization

FCHV Female Community Health Volunteer

IGA Income Generating Activities

LF Leader Farmer

MG Mother Group

NFE Non Formal Education

NGO Non Governmental Organization

PRA Participatory Rural Appraisal

SFCP Small Farmers Community Project

TBA Traditional Birth Attendent

UC User Committee

VDC Village Development Committee

WG Women Group

5
ACKNOWLEDGEMENTS
I wish to express my sincere gratitude and profound appreciation to CARE Nepal project staff
who have kindly and generously extended their help in the preparation of this Participatory
Evaluation Handbook (an Internal Evaluation Handbook for CARE Nepal supported projects).

I am deeply indebted to the communities who helped to test the various PRA tools.

I am grateful to Ms Shova Gurung, Mr. Padma N. Poudel, Mr. Gopal Shrestha and Om Ghimire
for their support during field-test. I am also grateful to Ms. Marcy Vigoda, Dr. Balaram Thapa
and Mr. Govinda Rajbhandari and other Senior Management Team members. Similarly, I am
grateful to Specialists and Training Officers for their support.

Bishnu Pokhrel

6
1. Introduction and Background

Internal Evaluation is relatively a new system in CARE Nepal. For the first time, community
representatives were invited in the final evaluations of two projects viz. Begnas Tal Rupa Tal
Watershed Management Project (BTRT), Kaski District and Small Farmers Community Project
(SFCP), Rapti Zone. These were the first events whereby the community representatives got
opportunities to observe and evaluate their projects.

In fiscal year 1993/94, CARE Nepal introduced internal evaluation in Remote Area Basic Needs
Projects of Bajura and Solukhumbu Districts. Then after Mahottari Natural Resource
Management Project, Mustang Natural Resource Management Project, Remote Area Basic
Needs Projects (Gorkha) and Upper Andhi Khola Watershed Management Project (Syangja)
conducted internal evaluations.

There are two types of internal evaluations–Self-Evaluation and Community Evaluation. The
project self evaluation is a process in which the project staff and /or others involved in a project
analyze and/or measure the achievements and limitations over a period of time. The community
evaluation involves the evaluation of a project, process or activities by community members who
are involved in the project. It may involve some facilitation by project staff, but this should be
minimal1.

In the past, internal evaluations were conducted separately and reports were prepared in different
ways. In some cases, the recommendations of staff and community evaluations were different
and therefore it was found difficult to implement them. Efforts weren’t made to compare the
recommendations of internal evaluations for the easy implementation. The evaluation findings
were not shared with communities.

The new thoughts for internal evaluations are different than in the past. Both the evaluations are
conducted separately as before but there must be a single report having common conclusions and
recommendations so these could be implemented easily for better programming. In case of
community evaluation, no external evaluators (except project staff) are involved. The
beneficiaries themselves evaluate their programs.

2. Evaluation Model and Concept of Internal Evaluations

2.1 An Evaluation Model

Internal evaluations are based on three-dimensional evaluation model as given in Annex 1. The
overall concept of evaluations is to assess development factors against each evaluation
components. The evaluation factors, which are applicable to all CARE Nepal projects, are as
follows:

1
This definition is taken from allnepal communication #871, which was written by Ms. Marcy Vigoda.
7
 Immediate outputs or achievements of the project
Local capacity building
Local human resource development
Gender aspects in development
Use of appropriate technologies
Financial and economic aspects
Other factors: Depend upon project's goals (impact and effect goals)

Some other development factors, which could be project specific, are as follows:
Environmental protection
Institutional coordination
Natural resource management, etc.

The internal evaluations must focus on the following components of Output, Effect and Impact
goals2:

i. Efficiency (achievements of results): How the results stand in relation to the effort
expanded. How economically inputs are converted into outputs. Whether the same
results could have been achieved in another, better way.

ii. Effectiveness (achievements of objectives): The extent to which the purpose (this is
called effect goal in CARE) has been achieved; whether this can be expected to happen
on the basis of the outputs of the project.

iii. Impact (other effects of the project): The changes and effects positive and negative,
planned and unforeseen of the project, seen in relation to target groups and others who
are affected.

iv. Relevance (the direction and usefulness of the project): The degree to which the project
can still be justified in relation to local and national development priorities.

v. Sustainability (the long-term viability of the project): An assessment of the extent to

which the positive effects of the project will still continue after external assistance has
been concluded.

A development program is sustainable when it is able to deliver an appropriate level of benefits

for an extended period of time after major financial, managerial, and technical assistance from an
external donor terminated (Guidelines for Evaluation, Danida, 1994).

2.1 Process of Internal Evaluation

The two types of internal evaluations are very much integral parts of each other and hence cannot
be separated. These are conducted separately and may have different findings, but the
2
The definitions of evaluation components are taken from Handbook for evaluators, Royal Ministry of Foreign
Affairs, Norway, November 1993
8
conclusions and recommendations should be compared and presented together in one section.
The advantage of preparing single report is that the communities and project will have better
understanding. This process also helps to provide feedback to the external evaluators and project
management team. The process of internal evaluation is given in Figure 1.

Figure 1. Process of Staff Self and Community Evaluations

Self Evaluation- by project Community Evaluations-

staff and Counterpart by beneficiaries and
Representatives participating communities

Evaluation findings Evaluation findings

Conclusions

Recommendations

Dissemination of lessons
learned

3. Evaluation Works

3.1 Decision to Evaluate

Some of the reasons for internal evaluations are as follows:

• There is a need of periodic evaluation, which should be done by beneficiaries and project
staff to determine and document impacts (both positive and negative).
• The projects are planned with specific phases and therefore it is necessary to internal
evaluations just before the mid term or final evaluations. This may also provide feedback to
external evaluations.
• The plan of future components depends on experience from preceding-phases.
• To assess whether some of the project activities should be changed, terminated or continued.
• To identify the problems, their causes and solutions.
• This is a process of community empowerment and local capacity building.
• There is a need for building staffs’ capacity to do internal evaluation.

9
3.2 Preparing the Terms of Reference (ToR)

It is recommended to prepare a single ToR for internal evaluation, which shall satisfy the
requirements of staff and communities. The following elements are included in a standard ToR.

i. Project background: project context and rationale.

ii. Purpose (objectives) of evaluation: reasons related to project performance and intended
use of results.
iii. Scope and methodologies: scope of evaluation, type of analysis, methods to be used and
degree of details.
iv. Issues to be covered: issues related to efficiency, effectiveness, impacts and
sustainability. Key questions related to these factors are also written.
v. Evaluation team: number of team members and their roles, required qualification and
experiences.
vi. Timetable: approximate date and allocation of time for different tasks.
vii. Field work: areas to be covered and institutions or groups or individuals to be consulted
during and at the end of fieldwork.
viii. Reporting: size of report and deadlines for draft version and final report.
ix. Reference materials: the recommended documents. paper, publications, books, reports,
etc. are included.

These elements are included in a standard ToR. Since project staff and beneficiaries conduct the
internal evaluations a short ToR (2-3 page long) covering relevant elements as mentioned above
is recommended. This can enable the team to carry out the evaluations smoothly. Depending
upon the needs project management can make decision on contents.

3.3 Selection of Evaluation Team

Since staff self evaluation is done by the project staff, it is recommended to ask all project staff
to participate the evaluation. Regarding the community evaluation, the experience of evaluation
team members is vital to the success. Other personal qualities like the ability to communicate,
analytical skill and teamwork capabilities are also important. Therefore, in case of community
evaluation, literacy is not an important criterion. While selecting the evaluation team for
community members the following factors are considered.

i. Experience in rural development: a Team Leader or a Team comprising two to three

members should lead the evaluations. S/he should have experience in rural development.

The ideal persons for leading the community evaluations would be Community Based
Organization (CBO) member, other social workers, Village Development Committee
(VDC) members and Local NGO representatives. They would be from the evaluating
community or outside the community. It depends upon the project management’s
decision.
10
 Project Coordinator, Project Manager, Sector Heads, Training Officer or Specialists can
lead the evaluation. This also depends upon project decision.

ii. Professional expertise: The team members should have professional expertise on the
areas planned for the evaluations. For example, a traditional birth attendant (TBA) is not
a right choice to ask for evaluating community forestry program.

iii. Cross-disciplinarity: The team members should consist of persons with the different
professional background. In community evaluation, Leader Farmers, Animal Health
Workers, Mother Group (MG)/ Women Group (WG) representatives, Caretakers of rural
infrastructure, TBAs, Female Community Health Volunteer (FCHVs), Village
Development Committee (VDC) representatives, etc. are involved. This also depends
upon the nature of a project to be evaluated.

iv. Gender balance: Since the evaluation factors also include gender aspects in development,
team members also require having gender analysis skill. The team should comprise of
both women and men.

3.4 Preparing Evaluation Plan

The evaluation plan provides a clear description of the evaluation. The methods should also be
clearly written. The evaluation Plan should describe what the evaluation team is supposed to do,
as well as when, where, how and why. Separate evaluation plans can be prepared for staff and
community evaluations.

Some of the checklists are given below:

i. Review of available documents and relevant literature: This includes baseline survey
reports and maps/diagrams, project information management system reports, project
implementation reports, etc.

ii. Evaluation questions: discussions of main problem areas.

iii. Methods: methodological approaches.

iv. Workplan: timetable and work division among the team members.

v. Documentation: The documentation process should also be planned. This includes note
taking during evaluation work, writing, drawings, maps and diagram, etc.

3.5 Conducting the Evaluation

3.5.1 Steps to be followed:

The following steps are recommended for conducting the evaluations:

11
i. Orientation to the team members: All evaluation team members should be oriented about
the evaluation objectives, methodologies and responsibilities. In case of staff evaluation,
an orientation in the beginning is necessary.

ii. Initial talks: This includes discussion on ToR, area selection including sample size and
other logistical arrangements.

iii. Data collection and fieldwork: This includes review of documents, interviews, field
studies, field visits, group discussion, etc. Ideally it is not relevant to conduct fieldwork
for staff self evaluation.

In case of staff self evaluation, field-based staff are key informants and therefore it may
or may not be necessary to do field observations. Project progress report review is
necessary for assessing the achievements.

Participating communities by using visual diagrams and maps does community

evaluation. Interviews and group discussions are combined with other Participatory Rural
Appraisal (PRA) tools such as resource map and social map to make it more participatory
and visual.

iv. Data analysis: The team leaders are responsible for data analysis. For the community
evaluations, the community members may or may not have analytical skills and therefore
project staff have to facilitate to analyze information and draw conclusions and
recommendations. In case of staff self evaluation, Project Manager and Sector Heads are
responsible for data analysis.

v. Final stage: Finalization of evaluation report and presentation of conclusions and

recommendations is the final stage. Project Manager, selected Sector Heads or field staff
and selected community representative are responsible for finalization of the report.

vi. Debriefing: After finalizing international evaluation report, a debriefing among project
staff and selected community members is needed to organize. Field-based staff can
debrief evaluation findings and recommendations to community members.

3.5.2 Writing the report

Two separate brief reports are recommended to prepare. Project staff self-evaluation report can
be written either in Nepali or in English. But the community evaluation reports are written in
Nepali. Then after both reports should be combined to draw common conclusions and
recommendations. In the past, most projects had written reports separately and therefore it was
difficult to draw conclusions and to implement them for better programming as well.

The following elements are recommended to include in evaluation report:

12
i. Executive summary
Summary of conclusions and recommendations
ii. Introduction
Methodology and approach
Scope and limitation

vii. Project description

Background for the evaluation
Project areas
Project components
iii. Evaluation findings:
Efficiency of the project: progress and resource use in the project; achievements of
results; results in relation to resource use (if possible)
Effectiveness of the project: success of project achieving its goals/objectives; factors
affecting the achievement of these objectives
Impacts: Foreseen and unforeseen impacts (at individual level and community level);
Factors and processes, which explain project impacts.
Sustainability: Factors affecting sustainability- economic, institutional, technological,
socio-cultural, environmental factors
iv. Conclusions and recommendations
v. Lessons learned
vi. Annexes : Some of the annexes are as follows:
ToR
Map of project area and area evaluated.
Team Members- community and staff members participated
Evaluation schedule
Maps and diagrams
Other relevant documents

It is recommended that the main document of combined report should not exceed 20 pages.
Additional information should be kept in annexes.

4. Responsibility of Project Staff and Community Team Members

4.1 Staff Self Evaluation

4.1.1 Facilitator/Team Leader:

Prepare ToR.
Design evaluation plan
Orient project staff who are participating the evaluation.
Select and orient community evaluation facilitators
Assist to organize evaluation
Finalize evaluation report
Debrief the findings to staff and other relevant stakeholders

13
4.2 Community Evaluation

4.2.1 Project Staff (Observer and Co-facilitator):

Prepare ToR (It could be a combine ToR with Staff Self Evaluation).
Design evaluation plan.
Assist to select and orient community evaluation facilitators
Assist to design evaluation plan and organize evaluation
Assist to write evaluation report

4.2.2 Facilitator from the communities

Participate in community evaluation orientation
Select participants in consultation with communities
Facilitate or assist to facilitate community evaluation
Assist to site staff to design evaluation plan.
Assist to site staff to write evaluation report.
Assist to site staff for debriefing.

After final report, the evaluation team leaders are responsible for debriefing the outcomes to
project office, local level counterpart and communities.

5. Quality Requirements

It is recommended to lower both the level of ambition and level of precision. Obtaining reliable
information may be difficult and time/resource consuming.

In general, the information obtained and conclusions drawn from the evaluation works should be
reliable and valid. The descriptions and conclusions should be valid for more than specific
situation i.e. within the project area. It should also be impartial.

6. Data Collection Methods

PRA tools are recommended for internal evaluations. Different PRA tools and their relation for
internal evaluations have been put in logical order in Annexes. It is recommended to use
appropriate tools depending upon the specific situation and nature of information to be gathered.
The following tools are recommended for internal evaluations.

6.1 Project Staff Self Evaluation

The following methods are recommended for staff self evaluation:

i. Success-Importance Matrix: To find out successful and failure activities and core
activities of the project, Success-Importance Matrix is used. Refer Annex 3 for method of
using the matrix.

14
ii. Flow Diagram: To find out positive and negative impacts of core activities of a project,
the Flow Diagram is drawn. For detail refer Annex 4.

iii. Diamond Ranking: It is also a PRA tool, which is used to find out strengths and
weaknesses of a project or an intervention. The possible areas for analysis by using
Diamond Ranking in internal evaluation are program core activities; process and
approaches of project and project support system (administrative and finance) See Annex
5 for the detail.

iv. Analysis of Gender Aspects: Gender aspect is a cross cutting issue and therefore this is
included in all methods.

v. Review of Secondary Data (achievements against the planned targets): Evaluators should
also analyze the achievement against project goals. This can be presented in logframe
format. By reviewing project database and Project Implementation Report, the
achievements can be reported. Refer Annex 2.3.

6.2 Community Evaluation

The following methods are recommended for community evaluation:

i. Social and Resource Mapping: Social and Resource maps are useful tools of internal
evaluations. The following areas can be covered by using these maps:
• Location of various groups in the community.
• Location and coverage of human resources such as Animal health Workers (AHW),
Leader Farmer (LF), Maintenance Workers, Female Community Health Volunteers
(FCHV), etc.
• Various development activities implemented with the support of project. Comparison
of before and after project can also be done.
• The changes in communities can be identified.
• Coverage of the project can be seen.

The details are found in Annex 6.

ii. Importance - Success Matrix. See section 6.1.

iii. Diamond Ranking. See section 6.1.

iv. Resource Location: Resources allocated for core activities is evaluated by using this tool.
Based on resource allocation, sustainability of project activities can be assessed. See
Annex 7 for details.

Pie diagrams are drawn after Resource Location exercises. Trend analysis of
contribution on the core activities by CARE, community, VDC, counterpart and other

15
 agencies can be found. The trend analysis (past, present and future), gives the picture of
sustainability of project activities. See Annex 7 for details.

v. Timelines: (only for the core activities): This tool is used to assess impacts of the project
and communities’ capacity at the local level to manage development activities. See
Annex 8 for details.

vi. Analysis of gender aspects: Since it is a cross cutting issue, it has to be incorporated in all
exercises of internal evaluations. It is expected that the evaluation team members will
cover the following areas:
Gender related needs
− Practical needs: access to basic commodities such as water; access to basic social
services; access to IGAs, etc.
− Strategic needs: Reducing inequalities in the division of labor between sexes;
reducing the burden of work connected to home and child-care; access to credit,
etc.
• Gender role analysis:
− Community management: Work in maintaining societal functions such as
water supply, health services, education, etc.
− Reproductive roles
− Productive roles

7. Use of Evaluations

Major uses of internal evaluation are as follows:

Evaluation findings have wider use. If the internal evaluations are conducted just before
external evaluations (mid-term or final), the outcomes can influence the external evaluation.

Project management team can use the findings and recommendations for improving program
qualities.

This is a process for empowering the local communities and building the local capacity in
monitoring and evaluation. It is also a process of building skills and increasing knowledge of
project staff.

8. Dissemination of Lessons Learned

The lessons learned could be used in the following areas:

Annual reports and Project Implementation Reports preparation.

Annual Implementation Plan and Annual Operating Plan development.
Seminars and workshops.
Counterpart reporting.
Project briefing papers.
16
 Feedback for external evaluations.
Revision of project approaches and strategies.

9. Feedback of Lessons Learned

The lessons learned can be used as feedback for:

New projects or re-designed projects
Improving program qualities
Improving management support
Changing the strategies and guidelines
Improving the service delivery system.
Feedback for communities
Others

17
References

1. Evaluation of Development Assistance, Handbook for Evaluators and Managers, Royal

Ministry of Foreign Affairs, Norway, November 1993

2. Guidelines for Evaluation, DANIDA, February 1994

3. Ideas for Project self-evaluation and community Evaluation, Marcy Vigoda, June 1996

4. Community Evaluation Report, Buffer Zone Development Project, Bardia, June 1999

5. Various PRA notes.

18
 Input
Annexes

Process

Efficiency
Output
Sustainability

Effect
Relevance
Annex 1. Internal Evaluation Model

Effectiveness
Impact
Impact

Achievements of the project

Local capacity building

Local human resource development

Gender aspects in development

Use of appropriate technologies

Financial and economic aspects

Development Factors

Note: This model is adapted from Hand Book for Evaluators, Royal Ministry of Foreign Affairs, Norway, 1993
Environmental protection

Institutional coordination
Annex 2. Process of Internal Evaluations

Annex 2.1 Use of different PRA tools

Social and Resource map Importance – Success Core activities Flow Diagram
Matrix

Coverage by Most important activities Positive impacts,

Groups, local HR and
activities
Changes- baseline vs.
current
Major problems and their
causes and solution
Least important activities negative impacts,
their causes and
solution
Most successful activities
Least successful activities

Timelines Resource location of Pie Diagram or Bar

core activities Chart

Communities capacity to Sustainability of core Trend analysis of

continue the core activities activities contributions/
subsidy

Note: It is not always necessary to combine all the tools together. Evaluators decide the right tools for internal evaluation.

20
Annex 2.2 Diamond ranking process

Processes and
approaches of the Strengths
project
Causes & solutions
Weaknesses

Core activities Strengths

Causes & solutions

Diamond Ranking Weaknesses

Project support system

(administrative and Strengths
finance)
Causes & solutions
Weaknesses

21
Annex 2.3: Secondary Data Review

Project logical framework progress report

Project Achievements

Synthesis of Annual Project Information

(API)

Synthesis of Project Information

Management System (PIMS)

22
Annex 3. Importance –Success Matrix and Flow Diagram

The objectives of drawing Importance –Success matrix is to find out successful and important
activities of the project, which will enable to develop flow diagram for evaluating impacts
(positive and negative).

The following steps are recommended:

A. Finding important and successful activities and thereby core activities

i. Divide the participants into Community Organization group (including NFE and IGA),
ANR, Infrastructure and Health. The minor activities are not recommended to include in
the evaluation.
ii. Ask the participants to list the activities implemented. The facilitator should have a
checklist of activities. The participants can use symbols to indicate the activities.
iii. Ask the participants to categorize the activities in order of importance. Then again ask
them to put into three categories- Most Important, Average Important and Least
Important.
iv. Ask the participants to categorize the activities in order of success (as most, average and
least successful). Then again ask them to put into three categories- Highly Successful,
Average Successful and Least Successful.
v. Summarize the activities in the following matrix

Importance - Success Matrix

Importance

High Average Low

(Medium)

I IV VII
Most
Successful

(Medium)
Average

II V VIII

III VI IX
Least
vi. After completing the matrix, start to identify core activities. The core activities should be
from cell I, II and III. Activities fall in cell IV and V can also be considered as core
activities. Identify the core activities and share with the participants.
vii. Make conclusion based on the matrix result.

B. Finding of positive and negative impacts of the core activities:

viii. Divide the participants into different groups. Let them to choose the core activities (at
least two activities per group).
ix. Brief the task. Ask the participants to list positive and negative changes/impacts of the
selected activities. The impacts should be presented in a flow diagram as given in Annex
4. Give specific examples for each change.
x. After finding the positive and negative impacts by the small groups, share it in plenary.
xi. Again, ask the small groups to go back and do the analysis on negative impacts. The
causes and solutions should be identified for the negative impacts.
xii. Conclude and present.

24
An Example of Importance - Success Matrix

Importance

High Average Low

(Medium)
• Pit latrine
• Slab culvert
Most

• Boring
• Kitchen garden
• Dalit Group Formation
• Plantation • Non formal
• Nursery establishment education
Demonstration Cross visit
Successful

• •
(Medium)
Average

• Women Group formation

• User Committee formation
• Leader Farmers Selection
• Hand pump
• Goat raising
• Improved Cook stove
• Forest Management Plan
Least

preparation
• Agroforestry farm
management training
Source: Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June,
1999

Before filling out the matrix, the participants had set criteria for Important Activities and
Successful Activities. Based on the set criteria, the participants discussed the reasons for high
importance, Most Successful, and so on. The final outcomes were also included in the report.

Core Activities of the User Committee:

1. Pit latrine 9. Leader Farmers Selection
2. Slab culvert 10. Hand pump
3. Boring 11. Goat raising
4. Kitchen garden 12. Non formal education
5. Plantation 13. Improved Cook stove
6. Nursery establishment 14. Agroforestry farm management
7. Demonstration training
8. Group formation 15. Management training
Source: Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June,
1999

In this example, the evaluation concluded that Cross Visit program is not the core activity.

25
Annex 4. Flow Diagram
DRINKING WATER Safe Personal hygiene
SUPPLY SYSTEM Piped Water available drinking improved
ESTABLISHED water
available
Less water for crop Time
land irrigation (One Poor drainage (5 saved
lps decreased) tapstand posts have) (average
30min)
Water User Committee (Out Water for Water for kitchen
of 11 EC members, 3 are cattle (from 8 garden (from 7
women and 2 are Dalits) tapstands) tapstands)

Crop production Causes of disease

deceased (villagers reported)
Livestock Availability of
Conflict raised (Out of health green vegetables
15 taps, 2 closed due to Maintenance fund raised improved (villagers reported)
conflict)

Workload of
children increased
Instability of Maintained & repaired Production
system the system regularly of meat &
(one paid caretaker milk
exist) increased

Other community Water system Use of surplus time

development works sustainable Less food available in productive works
affected (farmers perception) (villagers reported)

Income increased

HEALTH
STATUS:
improved/
not improved

Negative impacts Positive impacts

An example of Flow Drawing

Irrigation
Boring

Water available for Due to insufficiency Drainage

farm land irrigation of irrigation water, problem raised
conflict raised during off
during cropping season
season

Production
increased

Conflicts and
other problems
raised
Income increased Family income
increased
Source: Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone
Development Project, 14-15 June, 1999

27
Annex 5. Diamond ranking to find out strengths and weaknesses

Diamond ranking is a tool, which can be used to identify strengths and weaknesses of a
project. Three areas of the project can be analyzed. They are:

1. Project core activities.

2. Project processes and approaches.
3. Project support system (administration and finance).

After finding strengths and weaknesses, further analysis can be done. The analysis of
results gives the efficiency and effectiveness of the project.

The following steps are recommended;

i. Brief the task to the participants.

ii. Divide the participants into three groups- project core activities, process and
approach and project support system.
iii. In the small group, ask the participants to list strengths of the given topic. Let
them to rank the strengths in order.
iv. Again ask the participants to list the weaknesses of the given topic. Let them to
rank the weaknesses in order.
v. Put the strengths and weaknesses in Diamond Matrix.
vi. Share outcomes in plenary: Find out the reasons of weaknesses and their
solutions, and supporting factors for strengths.
vii. Conclude and present.

28
5.1 Diagram of Diamond Ranking

Strengths
II II

III III III

IV IV IV IV

4 4 4 4

Weaknesses
3 3 3

2 2

Shape of Diamond Ranking is given in the figure above. Ideally it would be nice if the
strengths and weaknesses can be presented in a diamond shape. But it is not necessary
that the project should always have equal number of strengths and weaknesses.
Therefore some of the boxes can be kept blank. For example a project may have ten
strengths and five weaknesses. In such case five boxes in weaknesses are blank.

29
Annex 5.1 Project Approaches and Processes Evaluation
Some of the areas that can be analyzed by using Diamond Ranking are given below.

1. Beneficiaries participation in project

− Planning- Infrastructure, Agriculture, Forestry, NFE, IGAs, Group formation,
Training, others (specify)
− Project implementation- Infrastructure, Agriculture, Forestry, NFE, IGAs, Group
formation, Training, others (specify)
− Project monitoring (participatory monitoring)- Infrastructure, Agriculture,
Forestry, NFE, IGAs, Group formation, Training, others (specify)

2. Maintenance of facilities
− Infrastructure (drinking water system, irrigation system, others)
− Agriculture (kitchen garden, agroforestry farm, orchards, etc)
− Forestry (Community plantation, private plantation, community forestry, etc)
− IGAs

3. Human Resource Development

− Agriculture sector (Leader Farmers, Animal Health workers)
− Infrastructure (Caretakers)
− Forestry sector (Nursery Manager, Heralu, etc)
− NFE sector (Facilitators, Supervisors)
− IGA (Masons, Carpenters, etc)

4. Institutional coordination
4.a
CARE UC/CBOs

Counter
part

4.b Institutional coordination

CARE VDC

Counter
part

30
4.c Institutional coordination-

CARE DDC

Counter
part

4.d Institutional coordination-

CARE Other
agencies

Counter
part

5. Local Institutional capacity building

− User Committee/User Groups
− Women Group
− Disadvantage Support Group/ Dalit Groups
− Other Groups (specify)
6. Evaluate sectoral activities

7. Implementation modalities
− Partnership with NGOs, CBOs and Government Counterpart.
− Direct implementation

8. Counterpart staff deputation to the project and its positive impact

31
Annex 5. 2 Evaluation of Project Support System

Under the project support system the following areas can be covered.

i. Administrative support
ii. Financial support
iii. Logistic support

The facilitators should identify the areas that have to be analyzed. Since the evaluation
depends upon the nature of project, it cannot be generalized. Administrative and
financial support systems can be lumped together for analysis. This depends upon
participants’ choice.

32
An Example of Diamond Ranking

Construction
materials are
available on time

Awareness program Availability of

and infrastructure irrigation water
projects are through tube well

Strengths
implemented. boring and drinking
water system

For plantation and Technical support Good quality seeds

agroforestry, the available in time distributed for
saplings are available increasing crop
on time. production.

People Financial and technical Clean drinking Goat raining program

participated in support for tube well water available is effective for
latrine and DWS boring provided from hand pump income generation
construction

Private pipes have Projects has not led the Delay in project
not been provided forest protection works planning and
to Dalits decision making

Conflicts raised in
Weaknesses

the communities due

to insufficiency of
water

Programs have not Project does not

been not launched for work for river
poor as Dalits training works

Failure in plantation
protection

Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone
Development Project, 14-15 June, 1999

33
Annex 6. Social and Resource Map

Social and Resource maps can be used for internal evaluations. Both of the maps can be
combined together. This is an easy way of involving illiterate people in the participatory
community evaluation process. The following steps are recommended for evaluation:

i. Choose appropriate place for drawing the maps. Cleaned ground without grass is
the best place for drawing maps. Since the whole process may last for about three
hours it is recommended to choose comfortable place.

ii. Ask participants to select the persons who want to draw social and resource map.

iii. Let the participants draw the maps on ground by using locally available materials.

iv. Check whether all the visible things such as village road, forest, physical
infrastructure, school, VDC building, stream, etc. are included in the map or not.

v. Ensure that the following things, if applicable are also located in the maps

− Location of Community Based Organizations (CBOs) and NGOs in the

community
− Available human resources such as Animal Health Worker (AHW), Leader
Farmer (LF), Caretakers, Female Community Health Volunteers (FCHV), etc
− Coverage of core activities and mapping the coverage of activities
− Baseline vs. current activities.

vi. Ask the participants to list major problems such as deforestation, landslide, etc. and
locate in the map, if possible.

vii. Do analysis of the problems- (problems-causes-consequences) and find possible

solutions

viii. Conclude and copy the map in book.

34
An Example of Use of Social and Resource Map

After preparing the Social and Resource Map, the participants listed the major activities
of their village. Then, before the project and after the project has been compared.

The outcomes of Social and Resource Map preparation is given below:

SN Activity Before Project After Project
1 Hand pipe 21 25
2 Kitchen Garden (improved) 26
3 Treadle pump 3
4 Boring for irrigation 3
5 Improved Cook Stove 5
6 Latrine 38
7 Slab culvert 1
8 Nursery 1
9 Non formal education 5 6
10 Leader Farmer 4
11 Household covered by Leader 43
Farmers
12 Demonstration 7
13 Goat Raising (IGA) 15HHs
14 Community plantation 4 ha
15 Private plantation 15 Kattha by 9 HHs
16 Agroforestry farm 2 nos
17 Study visit 6 persons
18 Women Group formation 1
19 User Committee formation 1
20 Forest Management Plan 1
preparation and discussion
21 Orientation workshop for User 1 event for 11 members
Committee
22 Management training 3 members
23 Leader Farmers training 4
24 ICS training 1
25 Goat Raising training 14
26 Nursery Naike training 1
27 NFE facilitator training 1
28 NFE Supervisor training 1
29 Fruit tree pruning training 3
30 Dalit Sub Committee formation 11
31 Provision of Caretaker (Heralu) 1
for plantation
32 Gender awareness training 11
Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

35
Problems, their causes and the solutions

Based on the Social and Resource Map, the participants were asked to find out the major
problems, their causes and the solutions. An example is given below:

SN Activity Problems Causes Solutions

1 Hand The hand pump does Unavailability of Management of
pump not work good quality washer good quality
in the village materials
Breaking of hand Lack of maintenance Provision of
pump parts training maintenance training
Low discharge of the Unavailability of Provision of
pump experienced persons experienced persons
for pump installation for pump installation

2 Plantation Damage by the Open grazing Control of open

domestic animals system exist grazing system.
Low survival rate No fencing Provision of barbed
wire or live fence
Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15
June, 1999

36
Annex 7. Resource Location and Pie Diagramming

Resource location analysis is a PRA tool, which is used to find out the resources used for
an activity. The resources can be divided into three types as follows:
− Community resources
− CARE/the Project resources
− Outside resources

The following steps are recommended to find out various resources allocated for an
activity:
i. Choose the activities to be analyzed and share with the participants.
ii. Ask the communities to list the resources required for each activity.
iii. Distribute the resources in the matrix given in figure below.

Symbols in visual Symbols in visual form Symbols in visual form

form
ooo ooo o
ooo

Community resources CARE/the Project Other agencies'

resources resources, specify
Name Symbol Name Symbol Name Symbol
Locally Technical  Money $
available support 
materials
Money $

iv. To know the contribution percentage of different agencies give Soya bean or
maize seeds or any types of locally available materials to the participants and ask
them to distribute to different segments as per the actual situation.
v. Take note simultaneously.

37
vi. Do analysis for each activity by asking the following questions:

− Has the activity created dependency on the project? What was the past trend?
− If the project has stopped to support the resources, would it be continued?
− What should be the activity phase-out strategy?

vii. Conclusion: Draw conclusion for each activity in the following areas:
− Appropriateness of use of project resources
− Sustainability of the activity after the project
− Use/ mobilization of local resources

viii. Repeat the above processes (step i to vii) for other activities.

Drawing the Pie Diagram:

To analyze the contribution status of core activities by different stakeholders, Pie

Diagram or bar chart are used. Take the information from resource location exercise, and
do the analysis as follows:

i. Put the information in the following table.

SN Core activities Contribution by (percentage or other appropriate unit)

Project Communities VDCs Counterpart Other
agencies
1
2
3
4
5
6
7
8
9
10

ii. Draw the Pie Diagram as follows:

• To draw Pie for row values, it is straightforward.
• To draw the Pie for column values, take five activities (maximum) at a time, calculate
average percentage and draw Pie.

38
 Pie row
Pie column

iii. Analyze the findings in the following patterns: For this pie diagram should be related
with Importance-Success matrix, which gives core activities.

Subsidy/contribution Important Successful

High Most Highly
Low Least Least

The possible combinations are:

I. High subsidy, most important and highly successful
II. High subsidy, most important and least successful
III. High subsidy, least important and highly successful
IV. High subsidy, least important and least successful
V. Low subsidy, most important and highly successful
VI. Low subsidy, most important and least successful
VII. Low subsidy, least important and highly successful
VIII. Low subsidy, least important and least successful

ii. Conclusions:

Type I: Relatively better than the other categories. These are the areas of
strengths.
Type II: Find out the reasons of least successful activities. Does the project want to
move it to type I?
Type III: Need to rethink and further analysis.
Type IV: Drop it.
Type V: The best areas. Continue.
Type VI: Find out the reasons of least successful activities and gradually improve
the program.
Type VII: Need to rethink
Type VIII: Why do we want to continue this activity? Drop it.

Note: This exercise can also be done during staff self evaluation by using secondary data
from the project file.

39
Example of Resource Location and Pie Diagramming

Activities Community Project Other Agencies

Boring for • Unskilled labor • Pipe • Not involve
Irrigation contribution • Rent for pumping
• Meals for skilled set
labor • Skilled labor cost
• All other imported
materials needed
for the tube well
boring
Non Formal • Room for class • Copy and pencil
education • Copy and pencil • Kerosene
• Lantern
• Blackboard
• Tin Trunk
• Training
• Cash for facilitator
and supervisor

Kitchen garden • Labor work • Seed

• Land • Technical training
• Fertilizer
• Tools
• Others
Plantation • Transportation • Saplings • Delegation
• Tools • Technical support of authority
• Labor work • Management
• Fencing plan
(temporary)
• Others

SN Core activities Contribution (percentage)

Project Communit VDCs Counterpart Other
ies agencies
1 Boring for 87% 13% 0 0 0
Irrigation
2 Non Formal 90% 10% 0 0 0
education
3 Kitchen garden 5% 95% 0 0 0
4 Plantation 25% 70% 0 5%

Source: Community Evaluation Report, Asneri User Committee, Buffer Zone Development Project, 14-15 June, 1999

40
Annex 8. Timeline

Timeline can be used to assess the inputs of the project and communities' capacity
building at the local level to manage development activities.

The following steps are recommended:

i. Choose an activity or process or approach to be analyzed.

ii. Choose appropriate variables.
iii. Check the knowledge of participants about the chosen topic. Such as if the topic is
"People's participation in development works", then the participants should know
the past trend, present status and expected trend in future.
iv. Prepare appropriate place for doing the exercise and choose appropriate materials
such as maize seeds.
v. Draw boxes for each variable. An example is given below:

People's participation in development works

Project Started

People
Participation oo oooo ooooo ooooo
Men
oo o ooo ooooo ooooo
oo ooo ooooo ooooooo
Women

10 yrs ago 5 yrs ago 2 yrs ago now

ooooo oooooo oooooo

Men ooooo oooooo oooooo

Villagers
Women ooooo oooooo
would like ooooo
ooooo oooooo
oo
to participate
now 2 yrs in future 5 yrs in future

End of Project

i. Draw conclusion and make recommendations.

41
An Example of Timeline

Time Line of Pit latrine construction

5 years 2 years At 2 years 5 years

ago ago present after after
Number of Household s Dalits 0 0 2 10 28
Others 2 9 36 50 60
Use of Latrine Dalits 0 0 2 10 28
Others 2 9 36 45 60

5 years 2 years At 2 years 5 years

ago ago present after after
Open grazing Dalits 30 25 10 5 0
Others 53 28 15 10 0

Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

42