How can evaluations of Cohesion policy be

improved? Lessons from peer reviews

Terry Ward
Applica and Evaluation Helpdesk

7th International Evaluation, Audit and Monitoring

Conference

Budapest, 19-20 October 2017

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Helpdesk activities

 Peer reviews of evaluations – one of tasks of Evaluation Helpdesk set up

to support DG Regio and DG Empl on evaluations of Cohesion policy
programmes

 Central objective of peer reviews – to improve evaluations and to make

programmes more effective as a result

 Other tasks of Helpdesk have same end:

o to review all evaluations undertaken on Cohesion policy programmes
o to assess evaluation plans and suggest ways to improve
o to provide support to Member States on specific aspects of evaluation
o to organise a summer school for MAs on evaluation each year

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Peer reviews

 Peer reviews of evaluations – continuous activity carried out for DG Regio

under different projects since 2011 (for DG Empl since 2015)

 Selected impact evaluations subjected to critical appraisal by a panel of

evaluation experts

 Aim: to identify strengths and weaknesses and suggest how to improve –

so as to provide practical guidance on components of good quality
evaluations and examples to follow

 Initial focus on methodology – on methods used - CIE, TBE, CBA,

econometric modelling, case studies, etc. – and whether applied
appropriately and correctly

 From 2011 to 2016, 43 evaluations reviewed (mostly ERDF), selected as

appearing to be of good quality from cursory inspection

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Main points to emerge

 Quality of evaluations not high in most cases – of 43 selected as being

among best, 5-6 assessed as being good, for many, findings not reliable
 In majority of cases, techniques applied competently but often important
steps in analysis missing:
o policy measure being evaluated not explained clearly enough
o literature review not undertaken
o theory of change not described
o means by which measure supposed to achieve outcomes not set out
o triangulation of methods not applied
o reliability and suitability of data not verified
o sensitivity or robustness checks of results not made
o policy conclusions not linked to findings or overly general
o some policy implications missed or not reported

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Points on evaluation process

 Peer reviews also revealed deficiencies in evaluation process itself:

o terms of references poorly framed
o evaluation questions too many, too vague, too ambitious
o budget available insufficient given requirements
o time-scale unrealistic
o methods stipulated not linked to evaluation questions
o supervision of evaluations inadequate or choice of evaluators
inappropriate – if not, deficiencies would have been picked up or
would not have occurred
 May reflect lack of competence of Managing Authorities concerned to
judge evaluations
 Impact evaluations demanding - require reasonably high level of
understanding to assess suitability of methods applied and validity of
results produced

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Points from assessment of evaluation plans

 Review of evaluation plans undertaken in 2016 and 2017 confirmed short-

comings
 Proposed design of planned evaluations and data requirements poorly
described in many cases:
o evaluation questions inadequately defined or too numerous for single
evaluation if aim to obtain meaningful findings to improve policy
o evaluation methods proposed not linked to questions and in some
cases inappropriate to answer questions - methods seem to be
proposed without consideration of what they can deliver
o data needed for evaluations identified only in general terms –
insufficient effort to relate data needed to those available, to identify
gaps in availability and how they might be filled
o applies in particular to CIEs and data for control groups – need to be
identified well in advance to ensure available and fit for purpose

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Lessons learned
 Evaluations hard to judge from final report alone – need to consider whole
process
 Evaluations unlikely to improve unless ToRs and evaluation questions get
better and methods stipulated are appropriate
 And unless commissioning bodies competent to judge what they are fed

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Lessons learned
 Evaluations hard to judge from final report alone – need to consider whole
process
 Evaluations unlikely to improve unless ToRs and evaluation questions get
better and methods stipulated are appropriate
 And unless commissioning bodies competent to judge what they are fed
Response
 For past year, peer reviews on evaluation dossiers (2 on ERDF+CF and 1
on ESF) – on Call for tender and ToRs, criteria for selecting evaluators,
budget, inception and interim reports as well as final report

 At each review meeting, completed evaluation on 2007-2013 period and

ongoing or planned evaluation on current 2014-2020 period assessed

 Also with participation of commissioning body – MA

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Results of new format

 Reviews have proved more meaningful– not only in identifying deficiencies

in evaluations but underlying reasons, enabling more useful suggestions
for improvement to be made

 Participation of MAs has helped – able to explain reasons for courses of

action (for ToRs, choice of evaluators, etc.) as well as main goals of
evaluation and how far findings and policy recommendations taken up

 MAs have heard comments on evaluations first hand, been directly

involved in discussions on particular issues, learned about problems with
what has been done and how they could have been avoided or overcome

 Also gained better understanding of what constitutes a good evaluation, of

how to improve in future in drawing up ToRs and evaluation questions,
selecting evaluators and following evaluations as carried out

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Points to emerge

 Impact evaluations invariably carried out at wrong time – too early for
impact of measures to become apparent

 This because of conflicting objectives - want to know effect of

intervention but also want findings to feed into policy-making process, to
influence design of intervention or how implemented

 Should affect how evaluations are undertaken – particularly what is

measured - implies focus on mechanisms and intermediate outcomes
rather than final objectives

 TBE clearly relevant in this – if can observe interventions are having

predicted intermediate effect then can assume they are on course to have
end-effect expected

 More generally, need to identify mechanisms by which measures will have

effect expected - applies to CIEs as much as to TBEs

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Points to emerge (continued)

 Control groups difficult to identify in CIEs - e.g. in case of measures to

assist long-term unemployed into work, how to identify those with similar
characteristics but not received support of any kind and to compile data
on them?

 General implication – data requirements of evaluations need to be

considered well before evaluation takes place – preferably when policy
first formulated

 Ideally evaluation requirements need to be built into policy design – how

policy is to be evaluated and what data will be used important steps in
policy design process

 Makes possible to adopt innovative approaches - such as through pilot

schemes and quasi-experimental evaluations

ISMERI EUROPA
 How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Key condition for improvement

 For significant improvements to be made, not only MAs but political

masters have to believe important to carry out evaluations – to know
what works, why and in what context to make policies more effective

 If evaluation findings not thought important to improve policies and

results do not feed into policy-making process, if undertaken only because
of Regulations, interest in improving them inevitably limited

 Improvements in evaluation therefore have to start at top, with policy-

makers who in many cases need to change mind-set

 Need to regard evaluations not as inconvenient condition for getting EU

funding but as vital to making policies more effective

 Until then, MAs constrained in what they can do and have little incentive
to make improvements

ISMERI EUROPA
How can evaluations of Cohesion policy be improved? Lessons from peer reviews

Thank you
for
your attention

ISMERI EUROPA