Introduction to Health Monitoring

and Evaluation

By
Abraha W.Michael, BSc N; BA Mgt; MSc in HM&E

Nov., 2009
Mekelle, Tigray, Ethiopia

1
 Presentation out Line

„ Introduction
„ Basic concepts of management
„ Supervision and supervisor: Principles, Qualities, and
Types
„ Basic concepts of services and programmes
„ Basic concepts of Monitoring and Evaluation

2
 Objectives
„ At the end of the training each participant
should be able to:
1. Discuss about service organisations.
2. State the qualities required by a supervisor
3. Define what monitoring is.
4. Develop a Logic Model on common health service
programmes
5. Identify the participants in the monitoring and
evaluation process and become familiar with their roles
6. List purposes of programme evaluation

3
 Introduction

„ A service organisation is a social system composed of

independent and interrelated subsystems.
„ It comes into being when there are persons:
„ able to communicate with each other,
„ who are willing to contribute an action and to
accomplish a common purpose.
Burton and Thakur, 2006; and Kumar and Mittal, 2003

„ It exists to combine human abilities to achieve stated goals.

„ Humans make the service system adapt to and shape the

external environment by focusing on some activities that
create something of value for individual clients or group of
clients.
Kumar and Mittal, 2003

4
What Resources are used by an organisation?
› Multiple resources are needed by an operating
organisation.
› These resources can be represented by 8M’s.
Ramasamy (2003).
Basic Resources: Fundamental Functions: Stated Objectives:
The 8M’s End Result
The process of Management

Management
Men & women Planning Actuating
Motivation
Methods Organising Coordinating
Materials
Machine
Money
Market Illustration adapted from Terry and Franklin, 1999

› These resources will remain as resources in the

absence of proper management.
5
Definition
„ Management of an organization primarily should focus on
the people who compose the organisation because they are
part and parcel of the organization and key for the
achievement of its objectives. (Mookerjee,1996)

„ What is management?
„ Management is the art of getting things done by a group of
people with the effective utilisation of available resources.
„ How many persons are needed to form a management? A
minimum of two persons are essential to form a
management.
„ A single person cannot be treated as a managing body
running any organisation.
Ramasamy, 2003

6
Cont’d
„ Motivation:
„ Motivation is a frequently used but poorly
understood term. (George and Jones ,1999)

„ However, it is considered a central force behind

the performance of employees in an organization.
(Michael, 1998)

„ Motivationis a set of processes concerned with

the force that energises behaviour and directs it
toward attaining some goal. (Ramasamy (2003),
Mookerjee (1996) and Hodgetts (1984))

„ Motivation is subject to an antecedent conditions

and consequent behaviour just like perception
and learning. (Michael,1998)
7
…..Cont’d

„ Motivation is one of the most important

aspects of actuating.
„ Motivation energises people’s behaviour

„ Behaviour activates action and

„ Action leads to actuation.

Michael (1998)

8
Cont’d

„ Work motivation is defined as the psychological

forces within a person that determine the direction of
the person’s level of persistence in the face of
obstacles toward attaining some goal.

„ Main denominators of the above definition are:

„ Direction of behavior,

„ Level of effort, and

„ Level of persistence of an individual’s attempt toward the

achievement of goals.
George and Jones (1999), Michael (1998), and Hodgetts (1984)

9
Cont’d
„ Changes on social, political, economic and other factors
may have a significant effect on the people and may
even change the orders of priority which they fix for
themselves.
„ But it is quite doubtful whether the people change
basically. (Ramasamy, 2003)
„ Also, it may be easy to identify how a person can be
motivated, but it is often difficult to conclude why an
individual is motivated.
„ Consequently, any systematic analysis of motivation
must be concerned with both how and why people act
as they do.
Hodgetts (1984)
10
 Supervision
„ Supervision is defined as guiding and directing efforts
of employees and other resources to accomplish stated
work outputs. (Terry and Franklin, 1999)
„ “Supervisor” refers to a person who is responsible for
overseeing the performance of employees at work.
„ Synonyms of Supervisor: Forman, Departmental
head, Charge head, Departmental in charge, overseer,
Sectional head, Head clerk, Charge man, Chief clerk,
Head assistant, Inspector, Superintendent, or Section
officer, etc (Ramasamy (2003)

11
 Principles of supervision
1. Effective supervision ensures the achievement of the
objectives of an organisation.
2. Effective supervision is the function of the supervisor.
3. The definition of the roles of the supervisor and
acceptance of the organisation are responsible for the
efficiency of the supervisor.
4. The supervisor should analyse his group and decide which
course of action helps to achieve the objectives of the
organisation.
5. The success of any supervisor depends upon the effective
fulfilling of group needs.
6. Group’s survival and its progress towards its objectives
are the parameters to judge the efficiency of the
supervision. 12
 Functions of Supervisor
„ A supervisor’s work is directly concerned with the
workers’ activities which are responsible for achieving
the objectives of an organisation.
„ Hence, the supervisor is regarded as a key person in
management.
„ Responsibilities of a supervisor are classified under:
„ Responsibilities towards workers or subordinates

„ Responsibilities towards management

„ Responsibilities towards his own functions

„ Responsibilities towards his colleagues

13
 Qualities of supervisor
„ A supervisor should have:
1. Technical knowledge and skills necessary to
his/her post.
2. Knowledge of the organisation (principles, degree
of delegation of authority, number of supervisees,
functions required specialisation, etc)
3. Ability to talk well (the art of talking)
4. Administrative ability (The art of getting co-
operation or co-ordination from workers)
5. Ability to listen
6. Honesty

14
Cont’d

7. Ability to memorise
8. Understand and respect the feelings of others
9. Ability for orderly thinking (Know the orderly
execution of work)
10. Complete information
11. Ability to judge the people (The ability of fair
assessment of workers)
12. Physical appearance (Should be commanding than
his subordinates)
13. Patience

15
Cont’d

14. Self-motivated

15. Should not express his frustration regarding his

job, salary and working conditions in the presence
of his subordinates

16. Should have self-confidence and nourish the self-

confidence to his subordinates

16
 Types of supervision
„ Autocratic or authoritarian supervision: The
supervisor have full power and full responsibility for
group action. He expects only obedience.

„ Free-rein or independent supervision: The

supervisor does not interfere with his subordinate’s
performance. He wants to exercise minimum control.

„ Democratic supervision: The supervisor takes a

decision only after receiving ideas and suggestions from
his subordinates.

17
 Kinds of supervisors
„ Bureaucratic-Regulative: (The supervisor follows
rules and regulations)
„ Autocratic-Directive: (One way communication, strict
control over subordinates, superior on the technical
side of the job)
„ Idiocratic-Manipulative: Supervisor has close contact
with the top management and subordinates. He cares
about his security and advancement. He is of easy going
type. Subordinates have selfish mentality.
„ Democratic-Integrative: The supervisor wants to get
recognition, advancement and security for both his
subordinates and himself.

18
 Supervisory techniques

„ Supervisory technique is a method or procedure to be

adopted in the work of supervision. e.g.:
„ Consultative or democratic technique

„ Authoritarian or dictatorial technique

„ Non-interfering or free-rein technique

„ A supervisory technique determines the type of

supervision and it is applied on the basis of attitude of
the subordinates.

19
 Monitoring and Evaluation
„ Every time a management decision is made, some type
of rational evaluation should have been made. (Grobe,
1978; 62;1)

„ That is, monitoring and evaluation are fundamental

elements of programme management.
Kayrooz and Trevitt, 2006

„ Evaluation has a broad scope. One can evaluate any

thing-including evaluations.
„ But the “Big Six” P’s that can be evaluated are:
Programmes, Policies, Products, Personnel,
Performance and Proposals. (Scriven,1993)

20
 Defining a programme

9 Be specific
Make it comprehensible, clear

9 Be precise
Mention things precisely

9 Develop Models
Make use of models when making a presentation
Give a broad perspective

21
 What is a Programme ?
„ Programmes is a part of a complex system of services
which operate to serve a particular population in need.
(Kumar and Mittal, 2003 and Swanson and Eisenberg, 1996, (19): 488)

„ Programme is a complex of people, organisation,

management, and resources that collectively make up a
continuing endeavour to reach some particular
educational, social or commercial goal. (Fitzpatrick, Sanders
and Worthen, 2004)

„ Programme is a series of activities supported by a group of

resources aiming to achieve specific outcomes among
particular individuals, groups, and communities.
„ Programme is a social system that evolves and transforms
itself in space and time.
22
 Programme Space

Time

Context
Form

23
 Programme:
structure – process – finality
„ The program is a structure operating processes to achieve
finalities.

„ Structure is the organisation of relations amongst actors and

between actors, knowledge (s) and resources.

„ Process structure is a socio-technical network

(actors/knowledge/resources) that uses and produces
knowledge and actions

„ Process is a group of actions taken by actors to obtain

finalities

„ A process is a group of actions linked by their finalities

24
Cont’d

„ The programme is a project of social transformation

operated by state institutions to resolve given problems

Finality is responding to a situation deemed

problematic

25
 Programme as a system of action

Time

Process

S-T Network
Programming
System
Finalities
Context
Form
26
Programming system and evaluation system
in programme space

Time

Evaluation
Programming
System
System

Context
Form

27
 Relation between programming system
and evaluation system
Programmingsystem
Programming system Evaluation system
Evaluation system

Socio technical
Socio technical network
network
STN programming
Actors – knowledge -
- evaluator OR knowledge
resources and evaluation tools
Process
Process
Planning Definition of the project
Implementation Production of knowledge
Sustainability Utilization of knowledge

Finality
Finality
Reduction of problems Improvement of the
program 28
Alternative names of programme components

Program Components
CDC (1999) Donabedian Evidence Based,
(1990) Strategic Planning
(WB, 2005)
Input Structure Input/Resources
Activities Processes Processes/
Activities

Output Intermediate Short term outcome

results
Outcome Final results Medium and longer
Impact term outcomes
29
 Monitoring
„ Monitoring is a continuous function that uses the
systematic collection of data on specified indicators to
provide management and the main stakeholders of an
ongoing development intervention with indications of
the extent of progress and achievement of objectives
and progress in the use of allocated funds. (Kusek and Rist,
2004)

„ Monitoring is:
„ Continuous

„ Systematic

30
Cont’d

„ Reporting is an integral part of monitoring.

Monitoring information is:
„ Compiled in standard and ad hoc reports;
„ Shared with implementing partners, donors and
beneficiaries;
„ Used to draw conclusions in evaluations.

„ What are the focus areas of monitoring?

31
Cont’d
„ Monitoring is frequently directed at learning whether a
programme is:
1. Serving the right people

2. Delivering the right king of services or

3. Spending its money in the right ways

(Rossi and Freeman, 2004)

„ Monitoring starts with a standard in view. E.g.., It knows

which categories of people are supposed to be served,
what kinds of activities should be carried out and how
money should be spend.
„ These standards give monitoring process a focus and
measures what the programme does against these
yardsticks. (Weiss, 1998)
32
 Elements of Implementation Monitoring
(traditionally used for projects)
„ Description of the problem or situation before the
intervention
„ Benchmarks for activities and immediate outputs
„ Data collection on inputs, activities, and immediate outputs
„ Systematic reporting on provision of inputs
„ Systematic reporting on production of outputs
„ Directly linked to a discrete intervention (or series of
interventions)
„ Designed to provide information on administrative,
implementation, and management issues as opposed to
broader development effectiveness issues.
Kusek and Rist (2004)
33
 Elements of Results Monitoring
(used for a range of interventions and strategies)
„ Baseline data to describe the problem or situation before
the intervention
„ Indicators for outcomes
„ Data collection on outputs and how and whether they
contribute toward achievement of outcomes
„ More focus on perceptions of change among stakeholders
„ Systemic reporting with more qualitative and quantitative
information on the progress toward outcomes
„ Done in conjunction with strategic partners
„ Captures information on success or failure of partnership
strategy in achieving desired outcomes.
Kusek and Rist (2004)
34
 The advantages of conducting monitoring

„ The cost of routine monitoring is significantly

lower than the costs of an external evaluation
„ Creates increased opportunities for staff to
develop their monitoring and evaluation skills.
„ The use of monitoring supports the continuous
assessment of the environment.
„ Monitoring facilitates a more efficient and useful
evaluation process.
Kusek and Rist (2004)

35
 The disadvantages of using only a
monitoring approach
„ Using staff as monitors takes them away from their
core functions. This could result in extensive hidden
costs.
„ Monitoring processes are often seen as less objective,
and therefore less credible, than external evaluations;
„ Internal staff do not usually have requisite levels of
technical skills in monitoring techniques. This could
result in significant errors in monitoring design and
implementation.
Kusek and Rist (2004)

36
Exercise
„ Imagine that you are climbing to Emba Alage.
„ Checking along the way to make sure you are
following the correct path is
like______________

„ Checkingwhether or not you reached the top of

Emba Alage is like________________

37
 Logic Model (LM)

„ What is a LM?
„ A LM is a commonly-used tool to clarify and depict
a program within an organization. It serves as a
foundation for programme planning and evaluation.

„ A LM is one tool for illustrating an underlying

programme theory.

„ A LM illustrates the linkages between programme

components and outcomes.

„ A LM is the basis for a convincing story of a

programme's expected performance.
38
 The benefits of using the LM tool
„ Builds a common understanding of the programme and
expectations for resources, customers reached and results,
thus, is good for sharing ideas, identifying assumptions,
team building, and communication.
„ Helpful for programme design or improvement,
identifying projects that are critical to goal attainment,
redundant, or have inconsistent or implausible linkages
among programme elements,
„ Communicates the place of a programme in the
organisation or problem hierarchy, particularly if there are
shared logic charts at various management levels and
„ Points to a balanced set of key performance measurement
points and evaluation issues, thus improves data
collection and usefulness, and meets requirements.
39
 Purposes of LM
„ Logic model is a versatile tool that can support many
management activities, such as:
1. Programme Planning: The logic model is a valuable
tool for programme planning and development.
2. Programme Management: Logic model creates linkage
between resources, activities, and outcomes. A
logic model can be the basis for developing a more
detailed management plan.
3. Communication: A well-built logic model is a
powerful communications tool.

40
Cont’d

4. Consensus-Building: A LM builds common

understanding and promotes buy-in among both
internal and external stakeholders about what a
programme is, how it works, and what it is trying
to achieve.

5. Fundraising: A sound LM demonstrates to funding

agencies that you have purposefully identified what
your programme will do, what it hopes to achieve,
and what resources you will need to accomplish
your work.

41
Elements of a LM

42
 Definition of common elements of a LM
„ Inputs are the resources used in a programme. They
include financial, human or material resources, e.g.,
„ Technical personnel for testing
„ HIV test kits

„ Activities include all those action steps necessary to

produce the desired effects programme outputs. e.g.,
„ Training of human resources for counselling and
testing
„ Referring people testing positive to treatment
services
„ Selecting of HIV positive patients for
opportunistic infection
43
Cont’d
„ Outputs are the immediate consequences of the
gathering of inputs through programme activities. e.g..,
„ Number of appointments provided
„ Number of counselling done
„ Number of qualifications carried out
„ Number of users who received pre testing counselling
„ Number of HIV tests carried out

„ Results are effects upon the target population. The

results include several types of effects, and may focus on
awareness, attitudes, behaviour, etc. e.g.,:
„ Increased of condom use
„ Improved quality of HIV/AIDS services
„ Reduced risk behaviours
44
Cont’d

„ Impacts are related to long-term cumulative effects

of a programme. They are rarely attributed to a
single programme or intervention. e.g.,:
„ Reduced HIV transmission rate

„ Reduced AIDS occurrence

„ Reduced HIV vertical transmission rate

„ Reduced HIV/AIDS mortality/mortality

45
 STD/AIDS Programmes’ LM

Diagnosis and Inputs Activities Outputs Outcomes Impacts

Planning Long-term
(Resources) (Interventions (Immediate (Intermediate
Accumulated
and Services) Effects) Effects)
Effect
Examples:
Situation Analysis HR Trainings n. of trained people Adopted professional
Analysis of responses Financing Educational Actions n. of preservatives practices HIV Inc/Prev
Stakeholder needs Materials Treatments distributed. Avoided risk Social norms
Resources Analysis Facilities Interventions n. of distributed kits Behavior STD Inc/Prev
Collaboration Plans Supplies n. of assisted users Service use AIDS Morb/Mort
n. of tests performed Clinical results Economic Impac
INFORMATION Quality of life

Data for Program Population-related Data:

Development Program Data health, social-behaviors

Besides monitoring data, some programmes carry out process

and result evaluations.
46
Exercise
„ Consider one of the health centres in your district
provides VCT, PMTCT, ART, ANC, etc services.

„ Using one of the above listed programmes construct a

LM.

47
 LM for VCT Programme
Problem Identification: The HIV infection rate continues to grow, highlighting the importance of people – both HIV positive and negative – to
be aware of their immunologic status so to develop personal risk reduction strategies or to improve their health status by accessing care,
support and treatment services. People who do not know they are HIV negative may not be motivated to remain HIV negative, while those
who are HIV positive may not use prevention interventions to reduce HIV transmission to their children and others, or to use available
services.

OUTPUTS IMPACTS
INPUTS ACTIVITIES OUTCOMES

Funding Training of testing n. of professionals

and counseling qualified in VCT Improvement of VCT
teams and local quality
managers Decreased HIV
n. of users who
received pre testing transmission rate
Staff Increase in the
counseling,
VCT trainers Service delivery: accessibility and
n. of users tested,
post testing n. of users who utilization of VCT
counseling to all received the result services
Reduction of HIV
users and the post testing Reduced risky prevalence
Protocols, counseling behaviors
Guidelines, for Users (HIV+ or -)
VCT Availability of PMCT with incorporated
services for HIV risk reduction
positive pregnant n. of HIV positive strategies and
pregnant women treatment strategies Improvement of Decreased
women
served by the health parameters of morbidity and
HIV test kits mortality related to
PMTCT HIV positive people
Increase of the HIV
Referring HIV
utilization of
positive users to
n. of HIV positive attention, prevention
treatment and
users referred to the and treatment
Referral counseling services
treatment and services for HIV+
system for counseling services and HIV- people,
prevention,
care, support
and treatment

48
 Example of Theoretical Evaluation Model for CTA
Logic behind the intervention: Not knowing the immunologic status impedes that HIV negative people stay motivated to
continue HIV negative; and that the ones who are HIV positive look for attention services, treatment and social support.
Counseling and testing are fundamental for the awareness of the immunologic status. The availability of the testing reduces
HIV transmission.

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

Planned
10000 users who
Availability of pre received pre testing
Testing and 90% of the
and post testing counseling
Counseling sexually active Reduction of HIV
counseling to all 9000 users
Team men from the occurrence
users 9000 tested users, developing risk
community with
reduction
consistent use of
6000 users that strategies and/or
preservative in all
received the result adherence to the
occasional
and post testing treatment
relations
counseling

Executed
7000 users received
Availability of pre 60% of the
Testing and pre testing
and post testing 3500 users sexually active
Counseling counseling, Maintenance of HIV
counseling to users developing risk men from the
Team 5000 users tested, occurrence
reduction community with
strategies and/or consistent use of
4000 users who
adherence to the preservative in all
received the result
treatment occasional
and the post test
relations
counseling

49
 Stakeholders
„ Partners can be persons or organizations having a deal
in an evaluation and the knowledge gained from it.
(CDC: Framework for Programme Evaluation in Public Health:
September 17, 1999; 48 (No. RR-11)

„ There are many ways to classify stakeholders. e.g.,

„ Internal and external

„ Those who are extensively collaborative and

extensively non-collaborative. (Brandon, 1998)
„ Uninvolved stakeholders can be uncooperative and
critical in subsequent debates about evaluation results.
(Shadish, 1994; 15(3): 347-358)

50
Cartography of actors’ interests towards evaluation
(Patton, 1997)

Stakeholders,the
Stakeholders, theprogram
programand
andthe
theevaluation
evaluation

Role in the Role in the

Identification
program evaluation
stakes

high
ofstakes
Importanceof
Importance

Moderate

low

51
 Evaluation
„ “To evaluate consists fundamentally in making a judgment
of value regarding an intervention, a service or regarding
any one of their components, purposing to help in
decision making”
Contandriopoulos et al. (1997)

„ Health service programme evaluation is a systematic way to

improve and account for public health actions by
involving procedures that are useful, feasible, ethical, and
accurate. (CDC: Framework for Programme Evaluation in Public
Health: September 17, 1999; 48 (No. RR-11)

„ Programme evaluation is the systematic collection, analysis,

and reporting of information about a programme to assist in
decision-making.
52
Cont’d

„ Evaluation is the systematic and objective assessment of an

on-going or completed operation, programme or policy, its design,
implementation and results. The aim is to determine the
relevance and fulfilment of objectives, as well as
efficiency, effectiveness, Impact (overall goal) and
sustainability.
Kusek and Rist (2004)

„ “Evaluation is not to prove, it is to improve.” Kellogg

Foundation (1998)

53
 Purposes of Evaluation
„ Rossi and Freeman(2004) identified four purposes of
evaluation. They are:
„ Programme improvement
„ Accountability
„ Knowledge generation and
„ Hidden agenda

ƒ Mark, Henry and Julnes, (2000) also identified four

primary purposes for which evaluation findings can be
employed. These are:
ƒ Assessment of merit and worth
ƒ Programme and organisational improvement
ƒ Oversight and compliance
ƒ Knowledge development
54
Cont’d
1. Assessment of merit and worth: Refers to
judgement at individual or societal level of the value
of a programme.
• It helps (legislators, government) to decide
whether to renew, to stop or expand the
programme.
2. Programme and organisational improvement:
Involves efforts to produce and use information to
modify and enhance programme operations.
• Programme improvement evaluation typically
provides information on current operations,
outputs and outcomes.
• In this case usually timely feedback is provided to
programme staff and those in charge of making
the recommended adjustments. 55
Cont’d

3. Oversight and compliance: Indicates the extent to

which a programme meets specified expectations
such as the directives of statutes, or other mandates,
including requirements to reach specified levels of
performance.
• Indicate whether a program is meeting formal
expectations.

4. Knowledge development: Refers to efforts to

discover and test general theories and propositions
about social processes and mechanisms as they occur
in the context of social policies and programmes.

56
 Tool
What do we know about the programme we are
about to evaluate?

Components What do we know?

Context What problems does this programme intend to solve? Which
populations is the programme aimed at? (destined to) In
which communities is it implemented?

Structure Who are the actors in the programme? What is their

knowledge? What are their resources? Their
interdisciplinary relationships? Intersectoral ?

Process Which actions have been planned? Which actions have been
accomplished and how are they related to each other?
Does the programme benefit from recurring resources? Is
there an established routine to its actions?

Finality What results does the programme aim to achieve? How

many can be observed?

Time How many years has the programme been in operation?

Has it undergone any changes through time?

Reflexivity Is there an
programme?
established mechanism to revise the

57
 Evaluating what?

What do we want to learn about the

programme and why?

1st evaluation process

Defining the project

58
Evaluating what? Doesthe
Does
programme
the
programme
answer
answer
theneeds?
the needs?
Isititthe
Is theright
right
Programme?
Programme? Whatresults
What results
doesititgive?
does give?

Haveyou
Have youtaken
taken
thenecessary
the necessary
stepsto
steps to
succeed?
succeed?

Arethe
the Canyou
Can youprove
prove
Are
interventions
interventions ititworks?
works?
fit?
fit?
YOUR
PROJECT

Source: Denis Allard,

SPPASP-SCFP, 2001
59
 Evaluation questions

The main areas in which an evaluation focuses are given below

Relevance
Relevance
Implementation
Implementation Time

Structure
Structure Process
Process
Process
S-T Network Programming
System
Finalities Results
Results
Context
Form
Impacts
Impacts
60
 Questions about relevance

Are we intervening
in the right
manner?
Time
Are we
intervening on
the right
problem? Process Are we intervening
for the right
S-T Network
Programming population?
System
Finalities
Context
Form
61
 Questions about implementation

Is the programme adjusted What

to the context it is trying to transformations did
be part of? Does ongoing the programme
strategy apply in this Time
undergo in different
context? contexts and
through time?

Process

S-T Network
Programming
System
Finalities
Context
Form
62
 Questions about structure

Are we succeeding in
mobilising the
necessary resources
and actors to carry out Time
the programme? Are
the right actors
mobilised?
Process

S-T Network Programming

System
Finalities
Context
Form

63
 Questions about processes
Have we taken the
necessary steps
Has the expected to succeed?
degree of Time
collaboration been
reached?
How innovative
Process
is the
programme?
S-T Network Programming
System
Finalities
Context
Form
64
 Questions about results

What results have Time

been achieved? Have the expected
results been
achieved?
Process

S-T Network Programming

System
Finalities
Context
Form
65
 Questions about impact
Does the programme
really answer the
Can we prove the
needs?
programme makes a
Time difference?

Does the programme help

reduce the problem
for which it was
implemented? Process
Are the expected
S-T Network Programming results, impacts,
System achieved?

Finalities
Context
Form
66
 Evaluation Approaches
„ Formative Evaluation:
„ Is performed during the entire planning process and
programme execution.
„ Purposes learning and the construction of
alternatives for programme improvement.
„ Answers questions such as:

„ How can the intervention be modified to achieve

its outputs and outcomes?
„ Are there better solutions compared to those
proposed by the programme?
„ How do the components of this programme
relate amongst themselves?
67
Cont’d

„ Summative Evaluation: usually performed to

provide judgment to managers or users about a
programme’s pertinence and merit.
„ Answers questions such as:
„ Is the program effective?

„ Should the programme be continued?

„ Normative Evaluation is a compliance assessment

„ Answers questions such as:
„ Does the programme comply with prescribed
norms?

68
 Relation between Formative and Summative
Evaluation in a Programme

Formative Evaluation
Relative Emphasis

Summative Evaluation

Programme Life

69
 Relationship among Evaluation, Planning,
Implementation and Effects

Planning Implementation Effects

- Monitoring of Inputs and - Outcome Monitoring

- Availability Studies outputs -Outcome Evaluation
- Situation Diagnosis - Process Evaluation - Impact Monitoring
-Analysis of Implementation - Impact Evaluation

70
 Evaluation way of Thinking
Effects upon the Targeted
Population
Achieved
Is there a difference?
Performance
Is that difference acceptable? Level
How can that difference be explained?

Planned

Intervention

Executed

Implementation
Level

Planned
71
 What do we monitor or evaluate?
„ Input/Output Monitoring: Refers to follow up of information
about inputs or resources and about outputs resulting from
the programme activities.
„ It answers questions such as:
„ Which services were provided?
„ Which resources were used?

„ What is the number of people who received

assistance?
„ Examples:
„ Input: Follow up of the number of contraceptive
pills acquired every month by the programme.
„ Output: Follow up of the number of women who
received pills from the service every month. 72
 Process/implementation Evaluation
„ Process evaluation is an evaluation employed to
ascertaining how well a programme is operating.

„ It doesn’t represent a single distinct evaluation

procedure.

„ The focus of the evaluation is on the programme itself:

its operations, activities, functions, performance,
component parts, resources and so forth.

„ It is usually approached as equivalent to an

implementation analysis.

73
Cont’d
„ Why process evaluations?
„ The main reasons for conducting process evaluations
are:
„ Accountability
„ Programme development and improvement
„ To help others set up similar services or networks
„ Process evaluation supplements programme process
monitoring.
„ Programme process monitoring is a systematic and
continual documentation of key aspects of programme
performance that assesses whether the programme is
operating as intended or according to some appropriate
standard.
74
Cont’d
„ Process evaluation answers questions such as:
„ Was the intervention implemented according to
what was expected?
„ Are the actions wherever they are supposed to be,
reaching the population which it was
programmed to reach?
„ Do the users have access to the intervention?
Which barriers make that access difficult or
unfeasible?
„ Example: Was the programme implemented as
planned? That is, the qualification was adequate, inputs
were available when needed, etc.
75
 Outcome Monitoring
„ Outcome monitoring is the continual measurement of
intended of intended outcomes of the programme,
usually of the social conditions it intended to improve.

„ Is the follow up of information related to the

programme’s expected outcome.

„ Usually is related to a period of time, such as, short

and medium-term.

76
Cont’d

„ Outcome monitoring answers questions such as:

„ Did the expected effect in the target population
happen?

„ Did the increase in the number of condoms

distributed result in an increase of the use of
condoms by users?

„ Example: What is the percentage of males that reported

condom use in sexual relations with occasional
partners?

77
 Outcome Evaluation
„ Contemplates or encompasses the explanations about
the reasons why the programme activities achieved or
not its outcomes referring to the target population.
Gives emphasis to causal relations between
intervention and effect.
Answers question such as:
„ Does the intervention explain the expected
outcomes?
„ Example: Was the Project responsible for the
increase of condom use? How?

78
 Impact Monitoring

„ Impact monitoring usually relates to the follow up

of epidemiologic trends of a disease (surveillance).

„ It answers questions such as:

„ What effects do all interventions have upon
AIDS prevalence ?

„ Example: Systematic follow up of AIDS prevalence

(AIDS epidemiologic surveillance).

79
 Impact Evaluation

„ Analysis of the relations between epidemiologic trends

of a disease and the control programmes and other
associated factors.

„ Answers questions such as:

„ How much of the aids prevalence variation is
due to the programme?

„ Example: How much of the reduction of AIDS

prevalence was due to the Programme?
80
 Programme Evaluation Matrix

c o n t ext Judgment Matrix

indicators standards merit

Description Matrix

ts
pu
planned executed

in
congruency

s e
iti
tiv
ac
contingencies

st
uc
od
pr

Description Value
81
Stake 1973