HOW TO MEASURE A RESULT

CONFIRM THAT THE RESULT SHOULD BE MEASURED

STEP 1: WHAT DO WE WANT TO KNOW?
What is the information we would like to know in the future? How
often do we want to know it?
For example: Our goal is to help 10.000 women increase their income.
What we would like to know is a) if this is the case, b) by how much their
income has increased, c) how sustainable this increase in income really is
after our support ends.
STEP 2: SHOULD IT BE MEASURED? WHY?
Ensure that the measurement will be used; if yes, clarify for what
it will be used; estimate how much the information is worth (e.g.
by thinking about how much you would pay to get that
information).
For example: Data in changes in household income and sustainability will
help us to adjust the project and tell us if we are overall successful or not.
And: Since we absolutely need to know if what we do works, we would be
ready to pay up to 10% of the project funds for this information.
PREPARATION FOR MEASURING A RESULT
STEP 3: DE/REFINE WHAT RESULT YOU WANT TO MEASURE
Check if the result is really formulated on right level (e.g. that it
is not an activity or an output that can be delivered with nearly
100% certainty); check if its clear who is supposed to change
behaviour/performance; check if the result statement is timebound;
For example: An increase in income is clearly not an output, since it is
beyond our control. And: we expect that 10,000 women from low-income
households in 2 provinces increase their income. We expect that to
happen within 3 years.
STEP 4: DECOMPOSE WHAT SHOULD BE MEASURED
Decompose any uncertain variable into constituent parts to
identify directly observable things that are easier to measure.
For example: Income from non-formal economic activities of 10,000
women before taxes.

STEP 5: CHECK SECONDARY SOURCES

Have others already measured it (or parts of it)?
For example: The Bureau of Statistics collects this data, but only every 5
years not frequent enough for our purpose.
OK, LETS MEASURE IT
STEP 6: WHAT LEVEL OF ACCURACY DO WE WANT? WHAT SHOULD THE
ERROR BE?
Measure just enough (optimal ignorance); keep the
information value in mind (see step 2).
For example: We would like to know the changes in income with a degree
of uncertainty of ca. 15% (meaning + or 15%).
STEP 7: DETERMINE MEASUREMENT INSTRUMENTS
Can the change be observed? If observing it in total is not
feasibly, can you observe a sample of it?
For example: Yes, increased income can be observed by frequently visiting
the 10,000 women. However, that might not be feasible. That leaves us
with the option of measuring a small, randomly selected sample of these
women.
Does it leave a trail? Think like a forensic investigator; if there is
no direct trail, does it lead to consequences which leave a trail
(think of a proxy measurement)?
For example: Increased income will leave a trail. Expenditure might raise
with increasing income; local tax payments might increase; living
conditions might raise; saving rates might increase. However, we conclude
that none of these measurements provide us with the sufficiently accurate
measurement.
If not, can it be tagged? Tagging involves adding some sort of
tracker so that it does leave a trail and/or can be observed.
For example: We can provide 5% randomly selected women with an
inexpensive mobile phone, a SIM card and some credit after they
complete our training. We can then call them or send them an SMS with 3
simple questions on a regular basis to collect the information we want.
If not, can it be forced to occur (e.g. through an experiment)?
If not, the result is probably not sufficiently well defined. Repeat
steps 1 to 7.