The Vietnam Sweden Poverty Alleviation Framework Programme

ANNEX 6 Management Information System and Monitoring and Evaluation Framework (MIS/M&E)
Background, Issues and Process

Prepared by: Lars Rylander

Planning Facilitation Team

Hanoi
June 2003

Table of Contents

1. INTRODUCTION ..................................................................................................................3 2. MONITORING AND EVALUATION .....................................................................................3 2.1 Definitions ........................................................................................................................................ 3 2.2 Setting up the M&E......................................................................................................................... 4 2.3 Indicators ......................................................................................................................................... 5 3. MANAGEMENT INFORMATION SYSTEM .........................................................................7 3.1 Reporting.......................................................................................................................................... 7 Monitoring risk.................................................................................................................................... 7 3.2 Database and data processing ........................................................................................................ 8 3.3 Management system ........................................................................................................................ 8 3.4 Capacity building............................................................................................................................. 8 4. EXPERIENCES IN SIMILAR PROGRAMMES ....................................................................9 5. IMPLICATIONS FOR THE PAP MIS .................................................................................10 5.1 Findings from consultations ......................................................................................................... 10 5.2 System design ................................................................................................................................. 11 5.3 Reporting........................................................................................................................................ 12 5.4 Structure of management ............................................................................................................. 12 5.5 Capacity building........................................................................................................................... 13 6. CONCLUSIONS FOR THE INCEPTION PERIOD.............................................................13

Attachments: Draft Quarterly Report national level Draft Quarterly Report provincial level Monitoring format overall progress -draft Monitoring format provincial level - draft

1. Introduction
Monitoring and Evaluation (M&E) and Management Information Systems (MIS) are concepts that often are used interchangeably to describe how information on the progress of a project is used for follow-up and management. In this section the term M&E is used to define the way that progress information is made available to find out if the programme is implemented as planned and is achieving its objectives. The MIS describes the how the subsequent database is organised, updated and applied for management of the project. This attachment deals first with the general definitions and methods for setting up of monitoring and evaluation, and then describes how the database should be structured and managed. It comments briefly on MIS used in similar programmes in Vietnam, and concludes with specifications that must be observed when the MIS for Chia Se defined and taken into use.

2. Monitoring and Evaluation

2.1 Definitions
Monitoring is the process of continuously collecting information about the progress of the project. Collecting the information is a normal part of day-to-day work. The purpose of doing that is to help decide whether activities are being implemented as planned and that the outputs related to these activities are reached. The information is then used to make decisions about improving the management and implementation of the project. Evaluation is a process to measure the impact and effectiveness of a project, in order to provide lessons learned. This is done by determining the achievement of the project objectives. Project evaluations are separately scheduled activities performed at specific intervals (for example in the middle or at the end of a project). M&E are closely related follow up activities, but that there are also some differences, as shown in the table below:
Monitoring When is it done? What information is collected? With what purpose? Continuous Directly available information about implementation of activities To check that activities are being implemented as planned and outputs are achieved Project staff and stakeholders as part of their day to day work Evaluation At fixed points More detailed information; may be harder to get To see whether the objectives are being reached Internal, or external team with specialist knowledge, assisted by project staff and stakeholders To judge the impact on the target population, adjust objectives; decide about the future of the project.

Who does it?

How is the result used?

To improve quality of implementation and adjust planning. As input to evaluation.

It is very important that the M&E is based on a participatory methodology. The key issues for participation are outlined below:

4 Participation Negotiation All stakeholders participate in selection of indicators and data analysis. Stakeholders negotiate what will be monitored and evaluated, how and when data will be collected and analysed, what the data mean, and how it will be shared and action taken. Participation and negotiation lead to collective learning, ownership and investment in key findings by those most able to use the results for corrective action. As learning improves and may provoke changes in implementation or the strategic approach, which calls for a flexible approach to management and monitoring.

Learning

Flexibility

2.2 Setting up the M&E

The table below is a schematic overview of basic steps in setting up the M&E system and of the M&E actions that should be taken at the various stages in project implementation1:
Stage Monitoring Decide what information is needed (indicators); Decide how often the information should be collected; Decide how it will be collected and its quality verified, and any implications for budget or planning; Decide who should be responsible for collecting it; Evaluation Setting the objectives for the evaluation; Decide what information is needed; Consider how it can be collected; Decide questions for a survey; Decide geographic area; Decide sample size; Determine no. of people and amount of time needed; Study logical framework; Draw up evaluation team TOR

Planning

Preparation

Design and test any data collection records; Prepare forms for data presentation; Train people who is responsible for monitoring; Produce a manual for monitoring with functional areas and accountability.

Design data collection tools (survey forms, interview guidelines,); Decide how to select the sample; Pretest and revise data collection tools; Train evaluation staff; Make arrangements for evaluation in villages;

Data Collection

Collect the agreed upon information on a routine basis; Monitor the functioning of the system;

Collect and study existing information (reports, proposals,); Select households / sites/ ; Conduct the interviews / observations /; Start data processing.

Analysis and Check

Compare collected data with agreed indicators, and note differences; Identify any other issues; Look for cause of any problems, and identify options for action.

Tabulate data; Interpret the results and draw conclusions; Decide on recommendations; Check analysis with key informants.

The diagramme is based on the SEILA M&E manual, IFAD,1999

Reporting of Results

Document data and findings; Provide feedback to project management and stakeholders.

Summarize data in tables and graphs; Write the final report.

Use of Results

Use results to improve management and implementation of project.

Define priorities; Plan the implementation of priority actions / changes /

2.3 Indicators
Indicators are realistic and measurable criteria of project progress. They must be defined before the project starts, and they allow stakeholders to monitor or evaluate whether the project does what it said it would do. In project planning, indicators form the link between theory and practice. Indicators are either quantitative, qualitative or time-based. Preferably an indicator should express all three aspects, (the expected Quantity, Quality and Time), or simply QQT. . Quantitative Indicators can be expressed as a number. For example, the number of people attending a training, the number of wells constructed, the average rice harvest per hectare, the cost of fertilizer. Qualitative Indicators indicate the quality of something, and they cannot normally be expressed as a number. For example, Womens participation in decision making in the village planning meeting, or improved working relations among staff. Because qualitative indicators are hard to measure directly, it may often be necessary to measure something else instead. For example, instead of measuring improved participation directly, one can look at the number of meetings organised by the village, how many people attended, if there was a women majority in the meeting, what decisions were made, and who made them. This kind of information then gives an idea of the increase in participation in decision-making. (The indicators we measure instead of the qualitative information are then called proxy indicators or indirect indicators). Time-based indicators state when a result is planned to be achieved and aims at revealing if there are any delays in project implementation, which is a signal that there are problems in the implementation of certain activities. Generally speaking, a good indicator is: Relevant, which means it measures an important part of an objective or output; Objective. If two people measure the same indicator using the same tool, they should get the same result. The indicator should be based on fact, rather than feelings or impressions (another way to say this is to say that it should be Measurable); Available. Indicators should be based on data that is readily available, or on data that can be collected with reasonable extra effort as part of the implementation of the (sub-)project. Realistic. It should not be too difficult or too expensive to collect the information (related to the next one in the list); Specific, which means that the measured changes should be attributable to the project, and they should be expressed in precise terms;

6 The Chia Se has been developed in close consultation with stakeholders and analyzed from problems and main constraints to a sustainable livelihood using a logframe format to address the logical relations between activities and outputs leading to poverty alleviation. In the logic of the logframe, inputs are used to implement activities, activities produce specific outputs, the outputs contribute to the project objective, and the project objective contributes to the programme objective. This is shown schematically in the drawing below.

Indicators Program for ObjectivesObjective Indicators are mainly used for evaluation.

Project Indicators Objective Indicators for Outputs are Indicators Outputs mainly used for monitoring. No separate indicators are Activities Inputs set for activities or inputs.

Monitoring tells whether activities are being implemented as planned, and whether the envisaged outputs in terms of establishing LPMD, LDF, effective structures etc are being produced as a result of the activities. This means that the monitoring system should concentrate on the indicators set for outputs. Evaluation tells whether the outputs result in poverty alleviation, and so concentrates more on indicators set at the objective level (although monitoring information is often also input for evaluation as well).
Stage Monitoring
(measures planned outputs against actual outputs)

Type of Indicator Focus on outputs Mostly quantitative, and often available as part of daily work Focus on objectives. Often qualitative, and needs special effort to collect

Sources of data Progress reports Field visits Observation Reports, plans Surveys Observation

Evaluation
(measures effect, or impact)

The table shows that it is important to make a distinction between impact indicators and progress indicators. Impact indicators are used to measure overall impact and effectiveness of the program during occasional evaluations (normally once per year). These indicators will help the management committees to ensure that the project is leading to poverty alleviation, as defined by the impact indicators. Progress indicators are used for the monitoring of implementation, and reported monthly or quarterly in the progress reports. In addition, they serve as input for project evaluations. Progress indicators are defined for outputs only; they are used to ensure that the defined outputs are being

7 reached. Although progress reports include information on activities, no separate indicators are defined for those.

3. Management Information System

The way the M&E data is used to manage a project is usually attributed to the Management Information System (MIS), and comprises the database and its updating in the form of format for reporting and the practical use of it, i.e. the project management.

3.1 Reporting
Implementation of activities are normally reported on a recurrent basis (as they happen), whereas consolidated analysis of the pace of project implementation should be done based on quarterly progress reports to the relevant management levels. The annual report summarises achievements in relation to budgets and outputs on an annual bases. It should be complemented with regular annual evaluations of the objectives and/or special evaluations of conditions in the project that needs a closer look. The most fundamental requirement of the reports from the MIS is that is should provide management structures with timely information that the project is performing according to plans and budgets. Having said that, it is of course equally important that the MIS provides information when (parts of) the project is not performing as envisaged as a means to address and review weaknesses in the project implementation framework or in the project strategy. It is important to understand that the MIS (and the logframe it is built on) is not a blueprint for project steering. On the contrary, the MIS must be regarded as an instrument for continuous learning that will provide feed back throughout the whole project period. It should be used for regular updating of both progress and impact indicators as knowledge increases about their accuracy, and the relations between outputs and objectives. It may turn out that these relations are not clear in reality, which then will call for a review in the overall strategy. The MIS expresses the intricate relationship between budgets, activities, outputs and the various progress and impact indicators, and it is highly unlikely, in fact impossible, that the assumptions made at the start of the project as regards these relations would remain the same during implementation. All decisions that will subsequently be taken on changes and adaptations of the logframe is, in fact, the expression of learning. They should be recorded by the management committees so that the learning process can be backtracked and documented. In this respect the information provided to the MIS must clearly indicate the demonstrated performance at the output and objectives level, not intentions or estimations. This will allow the management committee to assess the effectiveness of the project strategy. The information fed back from the MIS and the analysis made of it may provoke the management committee to decide on reallocations that will help increase the overall impact of the project. Reallocations implies that resources are reduced in implementing activities that prove to be less effective (unless external factors impede more negatively than expected see risk below), and likewise that resources are increased areas where demonstrated performance is better than expected and where there is room for expansion or replication of activities or components. It is crucial that this type of flexibility is maintained, particularly in the Chia Se, which is experimental in its strategy and must be totally open for learning at all levels.

Monitoring risk
It is not unusual that evaluation reports find that there is a gap between project outputs and objectives, implying that projects can be assessed to deliver services and goods according to plan, without

8 apparently having an impact at objectives level. For instance, regular evaluations may prove that the poverty situation is not improved even though most of the activities are implemented and most outputs are achieved. In such cases there is either a flaw in the logframe structure between what has been defined as outputs and objectives, or (and more likely) an underestimation of the external factors that may influence the project, negatively or positively. The logframe analysis includes an assessment of external factors and important assumptions, and suggests the way in which they need to be monitored during implementation.

3.2 Database and data processing

At the lowest level of project activities, the information can often be summarised, consolidated and analysed based on the data sheets, regular reports and other written information collected during a certain period. However, when activities take place in several locations and at different organisational levels (for instance local provincial national) it is not possible to maintain a manual information system. In order to provide the necessary overview of the situation in different aspects of the project, data must be structured in ways that allow for quick and accurate data processing. This implies that standardisation of data reporting must be done to allow for computerised processing. To allow for effective project management, the database must mirror the logframe structure, and comprise all activities with budgets, and all outputs with their respective indicators. When being timely updated, the database gives a good picture of the implementation status of the project and all its activities, not only at each management level, but in the Chia Se as a whole. The objectives are often defined in qualitative terms and compares the situation when the project started with the future desired situation, envisaged at the completion of the project. Such data is most often collected through regular or special evaluations that are additional to the computerised database. In this respect it is important to note that proper baseline data must established before the start of the project, as well as the was for updating. In the Chia Se there are a number of impact indicators that cannot be used for management unless a baseline is drawn up. Most of them need be set up through a survey of the actual situation, and may also for require action-based research as a special evaluation activity over the project period to ensure that the impact is reasonably well captured and assessed.

3.3 Management system

Reporting should be done as close as possible to actual implementation. Moreover, it must of course be done in a fashion that ensures that data is not manipulated with or biased in any sense, but is objectively reported (to avoid the situation of garbage in, garbage out). The best insurance for this is when a participatory approach is used, and when it becomes in the interest of stakeholders to have proper and accurate information, so that also data collection becomes a demand-driven function. Review meetings shall take place at each relevant level of management, normally in the form of steering or management committees, which in its composition will reflect involved stakeholders, including representatives of the target groups or beneficiaries. When designing the reporting and management information system, as much as possible coordination should be sought with existing databases. Likewise, alignment with existing administrative systems and routines should be done when feasible in order to avoid duplication of efforts or overlap of work.

3.4 Capacity building

In parallel with the designing of the MIS, the training needs and incentives analysis must done. It shall include at least persons involved in the following functions.

9 data collection, analysis, decision-making, data processing.

4. Experiences in similar programmes

The RIDEF programme had a rather elaborate MIS, managed from district level. However, the staff training was not sufficient and data quality was sometimes low. Moreover, there were no baselines laid for impact indicators and the evaluations were conducted without being clearly or logically connected back to the poverty situation at the initiation of the programme. Based on the RIDEF experience and similar programmes implemented by UNCDF, the work on a generic MIS has been done. It is developed for implementation, including integrated budget follow-up, and is basically a tool for monitoring. However, it is multi-level system and built on the two main components also included in the PAP (Local Development Fund and Capacity-building). Moreover it is a modular system so it can be customised for any project with the same orientation. It has been tested in six projects, and is being implemented as MIS in altogether 18 projects worldwide. If possible and suitable, the system should be assessed for use in the PAP, which would save both time and financial resources for software development. The MRDP and its component for Commune and Village Development Budgets was also very ambitious in trying to track the impact of the implementation of the local development funds at commune and village level. Also in the case of MRDP it soon turned out that the information to be used was too complicated to collect and make constructive use of. Forms were used at village level to state for instance the meeting attendance broken down into gender and minorities, but were for various reasons not presented in a timely fashion and, hence, not very useful as monitoring instrument. One explanation was that the monitoring system was not enough participatory, and that data collectors felt they just collected and forwarded information to central authorities. Some more recent MIS have been developed by the Danida-supported ASPS (Agriculture Sector Programme Support) and by ADB in the Forestry Support Programme (FSP). Both systems are quite comprehensive and covers all components and levels in project implementation (from village to central programme management level). In FSP progress information in the various components at village level is recorded on an continuous basis, being sent to communal project management structures for consolidation and (quarterly) transfer (in hard copy forms) to district, province and central levels. Data is processed at provincial level in the ProMIS database, with progress reports sent back to communes, districts and provinces for management. The ASPS system is organised in a similar way, but with data consolidated per component at programme management level. The system focuses on the progress of work plan, disbursements and outputs, not the higher logframe levels. The ASPS system also gives information on gender and minority group involvement, and moreover classifies activities in categories of overspending and target fulfilment on an aggregate basis. The ASPS system is constructed in a way that each activity has one indicator (done or not done). Moreover it is based on the logic that when all activities pertaining to one output are implemented, then the output is logically considered to be reached. Hence, the system does not include external factors, i.e. that other, non-project factors exist and may influence the actual possibilities to reach the planned output. The existence of risk in this sense is thus eliminated in the MIS for ASAP (but not in real project implementation).

10

5. Implications for the MIS for Chia Se

The purpose of the MIS for the Vietnam Sweden Poverty Alleviation Programme is to report timely and accurately - on results and achievements in relation to the plan and strategy expressed in the Programme Document and the different Project Document. Again, the MIS provides the type of information needed for learning and management. For this reason, the MIS2 has to be derived from to the LFA (logframe) of the programme as a whole and for each of the projects included in the programme. This also implies that the full structure of the MIS cannot be derived until the final LFA matrices and related work plans with detailed activity plans and budgets are finalised and approved, which will be part of the Inception Report3. The finalisation of the initial set of LFA matrices, including work programmes, will be done in a participatory process and mainly be based on a further planning of the activities. This process will also result in agreements on which progress indicators that should be used. During this period there must also be dialogue on the impact indicators to be used. The discussions and testing of these will feed into the MIS, during development but also later in the process. In addition to these general guidelines for MIS development and purpose, the following considerations should guide the design of the MIS.

5.1 Findings from consultations

The field visits confirmed certain points of departure that shall be respected in the MIS development work. There is little if any horizontal coordination and exchange of information and experiences between various programmes aiming at the same objective poverty alleviation in the selected provinces. This refers both to the national programmes, such as the 135, and donor programmes. Although DPI is generally represented in steering committees for donor-funded programmes for poverty alleviation, DOLISA - being accountable for the 135 programme - is normally not included in these management committees. Representatives of DOLISA stated that they knew little or nothing about donor funded PA programmes, and there was no vehicle for information sharing on the poverty situation in the province. Most MIS in operation in the provinces include project implementation information (monitoring) but do not rely on reviewed and updated hard data for assessments of the changes in poverty. Hence it is difficult to assess how effective the different programmes are in this respect. This refers to particularly to 135, which use its poverty definition to target families that are eligible for government allowances. The poverty situation is updated at irregular intervals, and it does not seem that DOLISA uses this database for evaluations of the impact of the programme or other government interventions to reduce poverty. It is only a selection tool. Few, if any, MIS are based on a participatory approach and there are no incentives to provide information. Information quality is often poor. Donor funded programmes have established their own databases on the poverty situation and the impact of their interventions, and these databases are customised to meet their specific needs, and rarely shared. Donor funded programmes are not aligned with the national framework for project implementation and reporting. This creates unnecessary information gaps and duplication of administrative resources.

2 3

In the following text the term MIS will be used for the compound concepts MIS and M&E This is the case even if the UNCDF generic MIS can be acquired.

11 The computer capacity at provincial government level is weak, but gradually improving through systems development and training. Financial management is done at district level, but accounts are reviewed and reconciled at provincial level.

5.2 System design

In developing the MIS some basic structural requirements should be acknowledged: The budget must be activity based and each activity must be classified as to status of implementation (done, on-going, not done); Report forms used for monitoring and evaluation should be clear (single-purpose) and brief, and well tested before application. Each output should have at least one progress indicator (if possible broken down into annual targets), and each objective should have at least one impact indicator. In each case, the indicators must clearly state the level of achievement independently of the rate of implementation of related activities or outputs. This is important, since it means the risk element is included in the MIS. Outputs which are not fully accomplished according to the progress indicator, despite that all activities have been implemented, are of particular importance in this respect. They say that either something was wrong in the elaboration of logframe for this part of the project, or that an important assumption of risk has developed into a more serious factor of influence than expected. The same goes for objectives that are not affected in the expected way despite that all outputs have been reached. In either case these gaps will provide room for a review of the project and its strategy; The risk analyses in the Programme Document and the respective Project Document shall be taken into account when the final LFA structure is developed, and be parameters in the monitoring and evaluation system. The way the objectives are defined makes it necessary to define a baseline through a special study on the situation. For instance in the project documents regarding Ha Giang, Quang Tri and Yen Bai baseline information (and evaluation formats) must be established for the following indicators: Improved sustainable livelihoods, Access and control over resources, Equity, and Empowerment. Likewise, they way the outputs are defined they need to be specified with the use of progress indicators Policy development, Management structure, Local Planing and Management Development, and Local Development Fund.

12

As regards the LDF, the criteria for use of the funds will serve as one type of baseline to be referred to in LDF evaluations. Recorded high fulfilment of criteria is one factor in assessments of possible increases for future resource allocations toLDFs. As regards decentralisation, which is not formulated as an outright objective but as a project strategy, a special research project should be considered focusing on the policy formulation function in the provinces. In a decentralised regime, the policy issues concerning livelihood issues in the province should increasingly be formulated in a bottom up perspective, based on the actual constraints for poverty alleviation. This implies that a baseline needs to be established also in this respect. It would make sense to include this function, and other special studies that may be justified, in the mandate for the Programme Advisory Group.

5.3 Reporting
The following specifications should be followed when the MIS is being developed: Data should be collected and reported on standardised and unified forms. Data should be collected at source and be transferred to and reviewed at the relevant management level (Village Meeting, CPMU, DPMU and PSC); One copy of form should be kept at collection level, and one copy sent to nearest higher level for data aggregation and report compilation - duplication of data collection must be avoided; Computerisation of data should be done at level where there is a suitable IT environment (presently not lower than provincial level, but capacity should be developed at district level to computerise data for better management);

5.4 Structure of management

The way the MIS is to be applied is explained in Diagram 8 in the respective project documents for Quang Tri and Ha Giang, and in Diagram 6.3 in the Programme Document. Quarterly Progress Reports with project monitoring information will be discussed at Village Meetings. All progress reports will also be sent to Secretariat of the Provincial Steering Committee for compilation and computerisation into the MIS database (data will also be forwarded to the Secretariat of the Joint Working Group at national level). The provincial secretariat produces reports that are needed for all management structures: the Provincial Quarterly Report for the Provincial Steering Committee comprising all district reports; the District Quarterly Report for DPMU comprising all commune reports; the Communal Quarterly Report for CPMU comprising all village reports.

Eventually, when activities at village level are becoming more diversified and complex to follow in a manual system, also the village progress reports will be sent for computerisation. At national level, the Quarterly Progress Reports from each project provincial and national are compiled into the PAP Quarterly Progress Report to be discussed by the Joint Working Group.

13 Each management level has its management agenda, which becomes increasingly more integrated the higher the management level is. Each level has its responsibility and is accountable for project implementation. Overall programme management is the responsibility of the Joint Working Group. Over the programme period the conditions for computerised data processing at lower levels shall be improved.

5.5 Capacity building

The capacity building component of the Chia Se will include capacity building at all management levels as regards project management in order to make best use of the MIS in actual management. It will also include the IT component. For further information, see Attachment 6, section C, Scope for capacity-building in the Chia Se .

6. Conclusions for the inception period

During the Inception period a number of activities need to be taken to have the MIS in operation when programme implementation starts. After finalising the LFA for the whole programme the following step needs to be taken:

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8

Set up the MIS to mirror the LFA. This step includes both system specifications and software development. Define baselines for all outputs and objectives. Define progress and impact indicators Define and test all information cards or sheets per activity Develop plan for computerisation and computer resources and organisation Develop Job descriptions for all persons involved in MIS activities Develop a Manual for Programme Management Develop a Manual for Decentralised Project Management Initiate capacity building

A special action and time plan needs to be set up for each project (please refer to the relevant project documents for Quang Tri, Yen Bai, Ha Giang and the national components)