Sie sind auf Seite 1von 13

DIGITAL OFFPRINT

Zeitschrift für
Evaluation,
volume 19,
issue 1/2020
ISSN 1619-5515

© Waxmann
Verlag GmbH,
2020

Zeitschrift
1/2020

für Evaluation
• Theorie, Methoden und Praxis der Evaluation
Multiperspektivische Einzelfall-Evaluation zur Wirkung von organisationalen
Schreibfortbildungen
Zeitschrift für Evaluation

Determinanten der Studiendauer – individuelle oder institutionelle Faktoren?

1/2020
Sekundärdatenanalyse einer bundesweiten Absolvent(inn)enbefragung
Nutzung von Routinedaten bei der Evaluation gesundheitsbezogener
Maßnahmen. Ein Beispiel für Datentriangulation von projektbezogenen
Evaluationsdaten und Routinedaten
Gesundheitsförderung durch Supervision – zur Wirksamkeit von
Einzelsupervision als Unterstützungsmaßnahme für Lehrkräfte
Die Institutionalisierung der Evaluation in Europa

• Information & Service


Praxisberichte:
Entwicklung von strategischen Analyseinstrumenten – ein Beispiel aus der
deutschen Entwicklungszusammenarbeit
Young and Emerging Evaluators Team Experience in a Real Evaluation Process
Projektevaluation im Bundesprogramm Biologische Vielfalt: Erkenntnisse
aus der DeGEval-Jahrestagung 2019 und weiteren Aktivitäten zum Thema Evaluation
Literaturüberblick und Rezensionen:
Bergmüller et al.; Eberli; von Werthern; Hense et al.

• DeGEval …Info

19. Jg. · Heft 1 · April 2020

Steinfurter Str. 555 Fon +49 2 51 – 2 65 04-0 info@waxmann.com Further information


48159 Münster Fax +49 2 51 – 2 65 04-26 www.waxmann.com on ZfEval here.
Germany
Praxisberichte

Zeitschrift für Evaluation


19. Jahrgang 2020, Heft 1, S. 149-160
https://doi.org/10.31244/zfe.2020.01.08
© 2020 Waxmann

Young and Emerging Evaluators Team


Experience in a Real Evaluation Process
Carlos Alberto Montero Corrales,1, 2 Laura Brenes-Alfaro1, 3

Abstract: The objective of this paper is to share with other young evaluators the challenges, pos-
sibilities and recommendations generated from a real evaluation process. An overview of the expe-
rience generated by a Young and Emerging Evaluators Team (YEET) of the Costa Rica chapter of
EvalYouth in the execution of the “Evaluation of selected services in the local governments of Bar-
va, Santa Bárbara and Aserrí, Costa Rica” (2018) serves as basis for the systematization process pre-
sented here. The document shows the background that motivated stakeholders of the National Evalu-
ation System of Costa Rica to support a real process evaluation carried out by young evaluators. This
evaluation process had two objectives: 1. to evaluate the quality of some services offered by the local
governments of Barva, Santa Bárbara and Aserrí in Costa Rica; and 2. to compare the factors that af-
fect the quality of provision of services in the three local governments studied. The implementation
of this real process evaluation revealed challenges in areas such as the object of evaluation, method-
ologies, data analysis, time management and the overall use of the evaluation. Finally, the article con-
cludes with lessons learned and key insights for both young and emerging, and established evaluators.

Keywords: Young and Emerging Evaluators, Evaluation, Municipal Services, Costa Rica

1. Introduction

To share with other young evaluators the challenges, possibilities and recommen-
dations of a real process evaluation, the objective of this article is to present an
overview of the experience generated by a Young and Emerging Evaluators Team
(YEET) of the chapter of EvalYouth Costa Rica, related to the implementation of
the “Evaluation of selected services in the municipalities of Barva, Santa Bárbara
and Aserrí, Costa Rica” (2018).

1 Co-first authors; both contributed equally to the work presented. Beyond the authors, Vilma Lean-
dro-Zúñiga (Psychology School, University of Costa Rica) was the third YEET member involved
in the evaluation, while Andrea Meneses-Rojas (DEval – FOCELAC) provided coordination and
technical support.
2 Public Administration School and Public Health School, University of Costa Rica
3 Environmental Pollution Research Center (CICA), University of Costa Rica

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
150 Zeitschrift für Evaluation, Jg. 19, Heft 1, 2020 – Information & Service, S. 149-160

This paper is divided into five sections: The first part is related to the origins of the
evaluation and the reasons for a YEE team to lead the process (background); the
next section contains the basic characteristics of the evaluation process and design.
The third part briefly discusses the main results of the evaluation focusing on the
services assessed in the local governments of Aserrí, Barva and Santa Bárbara. The
fourth section analyses the main challenges faced by the young evaluation team,
such as those related to the creation of the evaluation design, the execution process,
the fieldwork, and the diffusion of results. Finally, the paper closes with key lessons
that may contribute to the development of evaluation capacities in young evaluators.

2. Background

The German Institute for Development Evaluation (DEval) within project FOCEV-
AL (Evaluation Capacity Development, by its acronym in Spanish) in Costa Rica
introduced the idea to strengthen the evaluation capacities of young evaluators inter-
ested in developing their evaluation skills in Costa Rica and in the region.
This initiative was complemented by FOCEVAL’s objectives to strengthen the
role of evaluation in political decision-making processes and in control of policy in
Costa Rica and selected Latin American countries. The priority of the project was to
consolidate successful activities and processes in Costa Rica, such as helping young
and upcoming evaluators to develop practical experience and to foster participation
in evaluations by civil society (DEval n.d.).
In addition to this, starting in 2008, the CEval (Centrum für Evaluation/Center
for Evaluation) of Saarland University (UdS, Germany) began to collaborate with
the Research and Training Center in Public Administration (CICAP, by its acro-
nym in Spanish) of the University of Costa Rica (UCR) for its Master’s Program in
Evaluation of Social Development Projects and Programs (MEPPD). This university
collaboration is funded by the German Academic Exchange Service (DAAD) (DE-
val 2017: 2).
As a result, from the start of the cooperation between CEval, DEval, FOCEV-
AL, and MEPPD in Costa Rica, the actors agreed to support the inclusion of young
and emerging evaluators in a real evaluation process. CICAP and CEval, the univer-
sity cooperation arm, offered guidance and scientific support in this context.
As a result, the evaluation process had a strong capacity building character, in
terms of technical and methodological capacities to design and execute evaluation
processes. The YEET was advised by Prof. Dr. Reinhard Stockmann from CEval,
specifically in the phases of evaluation design and in the use of evaluation results.
In addition, a member of FOCEVAL’s technical team coordinated and held the re-
sponsibility to manage, support and guide YEET at a technical level for the imple-
mentation of the evaluation process (DEval 2017: 2).
Finally, CICAP coordinated with the local governments to find those willing to
benefit from the evaluation processes, resulting in the selection of Barva, Aserrí and
Santa Bárbara. CICAP also offered support to conduct 999 telephone interviews.

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
C. A. Montero Corrales, L. Brenes-Alfaro: Young and Emerging Evaluators Team 151

The three members of YEET were chosen from an open call through social media
and via email (from different organizations, including MEPPD, RedEvalCR4 and
EvalYouth Costa Rica), resulting in the formation of the team which worked togeth-
er from March through to October 2018. The evaluators fulfilled the YEE profile at
least in one of the EvalYouth requirements: younger than 35 years old (2 members),
evaluators (2), individuals who had less than five years of professional experience
(3), or development professionals who had technical knowledge on evaluation and
were willing to become professional evaluators (1). Including the team coordinator,
the academic background of the team was in Public Administration, Sociology and
Psychology.
The real evaluation carried out by YEET was in line with one of EvalYouth or-
ganization’s overarching goals, which intended “to promote Young and Emerging
Evaluators (YEE), including young women, to become competent, experienced and
well-networked professionals who contribute to evaluation capacity at national, re-
gional and international levels” (EvalPartners 2019).
The evaluation process began with different meetings between the organizations
that integrated the management team (MT): DEval, CEval, CICAP, and MEPPD;
while the Ministry of Planning and Economic Policy (MIDEPLAN, by its acronym
in Spanish) and EvalYouth Costa Rica became the support team (ST). The manage-
ment team should be “the technical and methodological guidance of the Evalua-
tion Team (ET) through direct support in the development of the evaluation design,
work plan and instruments” (DEval 2017: 3); and the ST should offer technical sup-
port activities related to the evaluation and dissemination of results (DEval 2017:
3).

3. Main characteristics of the subject of evaluation and the


evaluation design

Costa Rica is a Central American country divided into 81 local governments. Each
is governed by three political internal stakeholders: the Municipal Council (made up
of aldermen, who have voice and vote, and trustees who have voice); the mayor and
the vice mayor, who organize the resources to offer the municipal public services to
the citizens.
According to the administrative legal division of the territory of Costa Rica,
each cantón is integrated by districts that are smaller territorial units, and each dis-
trict is governed by District Councils, composed of one trustee and other voluntary
representatives.

4 Evaluation and Monitoring Network of Costa Rica (RedEvalCR, by its acronym in Spanish), is a
voluntary organization for professional evaluation in Costa Rica which was established in 2018
with the mission to promote dialogue, participation and exchange between people interested in
evaluation and monitoring, through a collaborative network, for the strengthening of evaluation in
Costa Rica.

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
152 Zeitschrift für Evaluation, Jg. 19, Heft 1, 2020 – Information & Service, S. 149-160

In 2010, the General Law on the Transfer of Competencies of the Executive Pow-
er to the Local Governments (Asamblea Legislativa de la República de Costa Rica
2010), intended to devolve the implementation of public services through local gov-
ernments. Yet, local governments hardly developed the necessary institutional ca-
pacities to do this in an effective, efficient, and relevant way with the necessary cit-
izen participation that should characterize the public administration according with
the Municipal Code (Law 7793, 1998).
As a result, an evaluation of the process and of the satisfaction of municipal of-
ficers and citizens became necessary to learn about and understand the quality of
the provision of some of these selected services; and to compare several factors that
can have an effect on the management of the services provided by the three local
governments under study.
The evaluated services for comparison were: solid waste management, road
maintenance, road cleaning, cemeteries, and citizens’ attention platforms. Also, the
YEE team evaluated an additional service in each municipality chosen by the mayor
or the Municipal Council: in Barva, the aqueduct office; in Santa Bárbara, the par-
ticipatory budget; and in Aserrí, the collections management process.
The following evaluation questions and criteria addressed the objectives:
– How do the services offered by the local governments meet the needs of the
population of each canton? (Relevance)
– How do the stakeholders value the selected services in the local governments?
(Efficacy)
– To what extent do the resources affect the management of the services provided
by the local governments? (Efficiency)
– What are the strengths and opportunities for improving the provision of selected
services in selected local governments? (Learning)
– What are the lessons learned from the comparison of the provision of services
provided in the three local governments under study? (Civil Participation)
Triangulation of information was crucial to answer these questions. The following
methods were applied for the collection of data:
– telephone survey (convenience sample of 999 citizens, 333 in each local gov-
ernment),
– self-applied questionnaires (census of 33 members at the administrative and op-
erational municipal staff directly related to the provision of the services),
– guided interviews (25 interviews with the mayors, vice mayors and the persons
in charge of the single services),
– guided telephone interviews (10 interviews with trustees and suppliers),
– focus groups (6 focus groups with civil society groups, aldermen and trustees),
– bibliographic review of laws and specific regulations by each local government.
The data were analyzed in three steps. First, the YEET built analysis categories
based on the evaluation framework, specifically based on the evaluation questions,
and also took into account the emerging categories according to the criteria of rel-
evance, efficacy, efficiency, learning and citizen participation. The quantitative data
were analyzed applying the statistics software package SPSS to compute correla-

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
C. A. Montero Corrales, L. Brenes-Alfaro: Young and Emerging Evaluators Team 153

tions. The information was then analyzed considering the analytical categories5; it
was contextualized by each local government’s situational background, taking into
account institutional contexts, operational capacity and socio-geographical charac-
teristics. Finally, the team compared the results between the three local governments
to understand the factors that influence the quality of the selected services.

4. Main results of the evaluation

The evaluation process generated multiple results with regard to each local govern-
ment and each service. The following section discusses the main findings for each
service and compares local governments by criteria.

a) Solid waste management


In the three municipalities ordinary waste collection is considered positive and is
highly valued, as it is a service that is considered basic, very necessary, and of-
fered frequently. Two municipalities (Barva and Santa Bárbara) outsource waste col-
lection which makes this service more efficient and effective. However, there is a
shortcoming of information and communication with regard to specific aspects of
the service, especially regarding non-traditional collection.
In addition, citizen participation mechanisms are underdeveloped; only Barva
works closely together with Development Associations or civil society groups.

b) Road maintenance service


The main findings related to the relevance and efficacy of road maintenance are: 1.
In Aserrí and Barva, mostly community-based organizations know about the service,
demand and execution of road maintenance services; 2. the three local governments
hold a valid Five-Year Road Plan in which needs and priorities of the local govern-
ment are identified; and 3. the maintenance and building of the cantonal road net-
work depend economically on the timeliness and amount of the central government
budget transfer.
Regarding efficiency and civil participation, the main findings were: 1. The pro-
fessional conformation of the Technical Department of Road Management (UTGV

5 The analytical categories by criteria were: 1. Relevance: program theory evaluation, level of
knowledge services, level of knowledge responsibilities, rural access, urban access, older peo-
ple access, disabled people access, women access, sufficiency of services, coverage, context ad-
aptation, cantonal context; 2. Efficacy: citizens’ satisfaction level, quality of service, municipal
management (control, supervision, evaluation, monitoring), human rights perspective status, gen-
der perspective status, departmental coordination, coordination between civil council and mayor,
interagency coordination, municipal mayor’s office coordination, regulatory framework and lo-
cal government planning; 3. Efficacy: allocated resources, vulnerable group allocated resources,
exclusion/inclusion practices, department coordination; 4. Citizen participation: strategies, notion
of citizen participation, citizen participation of the vulnerable groups, citizen participation at the
phase of planning the service, strategies knowledge, transparency and corruption; and finally, the
criteria of Knowledge: strength, challenges, general recommendations, vulnerable groups recom-
mendations, gender and human rights recommendations.

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
154 Zeitschrift für Evaluation, Jg. 19, Heft 1, 2020 – Information & Service, S. 149-160

by its acronym in Spanish) in Aserrí and Barva is made up of people with a range
of professional profiles: road engineer, social worker, administration officer, road
technician, driver, and operator; 2. in 2018 UTGV of Barva executed 106% of the
budget, the UTGV of Aserrí executed between 80% and 85% and Santa Barbara lo-
cal government executed 7% of the ordinary and extraordinary budget assigned by
law. With regard to the citizen participation criteria, from the total number of peo-
ple surveyed (999 respondents), 73% said they did not know if there were spaces in
the community to express opinions, needs, and demands, while 23% said they knew
and 4% did not respond. Regarding participation in any activity organized by the lo-
cal government, 71% said they had never participated in any activity organized by
the municipality, while 27% had done so.

c) Road cleaning
With regard to the relevance and efficacy of this service, the main findings were the
following: 1. In the three municipalities, there are well-established procedures and
work routes that expedite and allow the work organization and planning; 2. only a
small percentage of the surveyed citizens of the three municipalities gave a general
assessment of the service because it is a service that is only provided in the first dis-
trict, however, the overall rating of the service was positive; and 3. caring for peo-
ple with disabilities is indirect, through compliance with Law 7600 in infrastructure
works carried out for this service.

d) Cemeteries
The main findings related to the service of cemeteries are 1. there are established
procedures that expedite and allow an agile performance and prompt service in all
three municipalities, and in Aserrí and Barva there is an office and exclusive staff
for this service; 2. the assessment of the citizenship was positive; and 3. however,
the multiplicity of functions according to the number of field workers affects the ef-
ficiency.

e) Citizens attention platform


The main findings in these services, by the criteria of relevance and efficacy, were:
1. In the three local governments, the behavior of attention to the citizens is per-
ceived as the main strength of the service, by the different stakeholders consulted;
2. in the three municipalities, priority is given to the vulnerable population; 3. the
long waiting time is a finding that must be attended in the three cantons.

f) The sixth service: chosen by each local government


Aqueduct office (Barva): The stakeholders value the service positively. People have
water at all times, and the quality is good. However, late payment by citizens is an
important obstacle to the modernization of infrastructure necessary to provide a bet-
ter service.
Participatory budget (Santa Bárbara): The participatory budget is an unsatisfied
need in the canton since the service is currently not offered even when demanded
by the citizens.

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
C. A. Montero Corrales, L. Brenes-Alfaro: Young and Emerging Evaluators Team 155

Collection management process (Aserrí): The collection management service pres-


ents a series of factors that limit its relevance; the people don’t know about this ser-
vice and it is not efficient because it fails to comply with current inputs, as well as
the expected and planned results.

5. Challenges of the YEE team

As Billings (2017: 25) indicates: “There is no substitute for experience in the pro-
gram evaluation domain. As students advancing to the firsthand world of evaluation,
we are forewarned about the challenges that we will face as evaluators in real-life
situations. We will need to recognize the disconnection between theory and practice
and the differences between theory-dependent research and field-dependent evalua-
tion”.
Evidently, as a YEET, the challenges were part of the evaluation and represent-
ed an important part of the learning as real-life situations presented themselves. The
challenges regarding this evaluation were the following:

a) The evaluand
To get acquainted with the local government services in short time: The YEET con-
sisted of four persons, three of which did not have experience in local governments
until this evaluation. For this reason, the YEET faced the challenge of developing
a sound understanding of the organizational structure, the process, the products and
the main stakeholders involved in each service by each local government, conduct-
ing interviews with those in charge of the services, all the members never had eval-
uated a local government. For this reason, the YEET had built the theory of change
by each service.

b) Methodological
– Not enough findings after data triangulation analysis: Some interviewees did
provide only little information, others a lot. In some cases, the YEET was not
able to balance for triangulation and perform a better analysis of the informa-
tion from all stakeholders.
– Cost-benefit analysis according to the resources of the evaluation: The main
challenge in this topic was to calculate the time in hours of the application, sys-
tematization, and analysis for each data collection method and the groups of re-
spondents in order to find out the methodological scope according to the avail-
able resources. Once this was done, the YEET had to identify the main benefits
of the findings generated by each method with regard to the evaluation ques-
tions. Finally, with this cost-benefit analysis, the YEET took decisions like not
applying data collection instruments to stakeholders that were not key or prima-
ry stakeholders of the evaluation, and, for example, change the survey method
applied from face-to-face interviews to telephone interviews, always trying to
make sure that the most significant findings would be obtained.

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
156 Zeitschrift für Evaluation, Jg. 19, Heft 1, 2020 – Information & Service, S. 149-160

– Between the ‘gold standard’ and the ‘real scenery’ for drawing the sample: The
main challenge was to think in alternatives to draw the sample of the three lo-
cal government population to the application of the telephone survey accord-
ing to the evaluation budget. With the support of Prof. Dr. R. Stockmann, the
YEET created two ways to resolve the challenge: One alternative was to have a
representative sample, applying surveys house by house at 1% of the total pop-
ulation of the three cantons which meant 1300 surveys applied by students or
by a company contracted to that task; and the second alternative was to have a
non-representative sample, applying the number of telephone interviews (999)
that the evaluation process could afford with 2.000 euros, or not to do surveys
and just apply a qualitative approach in the evaluation process. The representa-
tive sample alternative was the ‘gold standard’ in the methodology of the pro-
cess evaluation but was the most expensive one with regard to the evaluation
budget. Nevertheless, the second alternative – even being a non-representative
sample – was the ‘real scenery’ of the evaluation, forcing the YEET to decide
‘real scenery’ in accordance with the budget of the evaluation.

c) Data analysis
– Incomplete and outdated databases of the citizens in each local government:
The main challenge was to realize that the databases of citizen contact data,
provided by the local governments, did not include the information for all
adults (18 years old or above) in the canton, and the existing contact data (e.g.
telephone number) were outdated. This problem was overcome by exploring
the contact data in the databases, calling a sample of 100 citizens, cleaning the
databases and getting updated information from other departments in the local
government that were not evaluated. The quality of the databases directly influ-
enced the statistical representativeness of the results obtained by the 999 tele-
phone interviews.

d) Use of the evaluation


– Development of the commitment to use the recommendations: Although local
government stakeholders were involved in the entire evaluation process, the co-
ordinators and the mayors did not develop a commitment to implement the rec-
ommendations made. The challenge was that the YEET had to acquire more
skills to implement instruments for developing the necessary commitment to
improve the quality of services through evidence-based decision making in lo-
cal governments.
– Communication of the results: It was important to communicate the findings
and recommendations of each service to the person in charge of it, the mem-
bers of the local organizations and other stakeholders from each community.
The challenge for the YEET was that the information, findings, and results have
to be brief, concise and easy to understand, which is not common in reports di-
rected to other audiences or in sociological or psychological research.
Another challenge regarding the communication of the results was to incorporate
all stakeholders into the final stage of the evaluation: The transfer of the results and

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
C. A. Montero Corrales, L. Brenes-Alfaro: Young and Emerging Evaluators Team 157

recommendations, so the use of the evaluation could be wider. In that stage of the
process, time was very limited, according to the time defined to carry out the eval-
uation and the pre-established days for each evaluator. Therefore, the YEET could
not hold workshops with the stakeholders who supported the process except the fi-
nal spaces provided in the meetings of the Municipal Councils of each canton (open
to the public).

e) Time management
– The time established for this evaluation was limited, as well as that of each
member of the YEET, in the sense of other commitments and responsibilities
(i.e. other jobs): In particular, time shortage was a complicated matter to solve.
The ET had 46 days each for this evaluation; however, because of changes not
foreseen in the process, the available information, slow response by local gov-
ernments and other factors delayed some of the activities. Therefore, the con-
tract had to be extended.
This is a particularity for YEET. Not every person, who is beginning in evaluation,
has their total time available for the evaluations as they have other full- or part-time
jobs.

6. Lessons learned

Generally speaking, the purpose of lessons learned is “to bring together any insights
gained during a project that can be usefully applied on future projects” (Project
Smart 2019). In this sense, the evaluation “has an important role to play as a central
knowledge provider and can help inform new thinking with knowledge from past
experiences (...). Capturing and using lessons derived from evaluations can provide
a range of benefits within a program or project” (Spilsbury et al. 2007: 5).
Regarding the work done by a YEET, “a lesson learned is knowledge or un-
derstanding gained by experience. The experience may be positive, as in a success-
ful test or mission, or negative, as in a mishap or failure” (Secchi 1999, cited after
Weber/Aha/Becerra-Fernández 2001: 18). In this case, as a YEET, the new knowl-
edge is tightly related to the learning criteria in a new field of knowledge. The ET
identified the following lessons learned from the process:
– To explore the databases of citizens’ contact data before including it in the
methodology: It means that the evaluability analysis, specifically the analyses
of the quality of data and information available is very important to reduce the
risk of data being offered that have no value for the evaluation.
– To have a management team for learning and solving questions: The manage-
ment team, composed of many professionals with experience in the field, al-
lowed the YEET to increase their skills and learning. However, this was only
partially fulfilled because some institutions did not participate actively during
the entire process.

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
158 Zeitschrift für Evaluation, Jg. 19, Heft 1, 2020 – Information & Service, S. 149-160

– Having knowledge in advance about the evaluand’s context allows designing


and executing an evaluation according to the specific field, in this case, local
government management. This way, to know the laws, organizational structure
and performance, local governments’ sociological context and the process of the
services, would have helped to build a results chain for clarifying the situational
background of the evaluant (cf. Leandro 2018).
– The importance of developing trust, good communication and commitment in
the YEET: Regarding the internal relations between the members of the evalu-
ation team, it is always necessary to have a good and open communication, as
well as commitment, trust and shared responsibilities, which was the case of
this YEET.
– To clarify the know-how of consultancy in an evaluation process: The evalua-
tion process allowed the team to have more clarity on how evaluation consul-
tancies work as well as what the current state of evaluation around Costa Rica’s
public entities was, and to know more about the activities of FOCEVAL, CI-
CAP, CEval, and Mideplan in the evaluation field. During and after this experi-
ence, the subject of YEE was approached from the RedEvalCR which generated
more interest and additional information.
– Skills acquired in a real evaluation process: The main areas of learning were
the different ways of designing and structuring an evaluation, closely related
to the observations and feedback from the management team. Also, an exten-
sive learning of methodological aspects and evaluation implementation was ob-
tained: its difficulties, the processes, and the variables to consider while the
evaluation is underway. Therefore, the YEET members strengthened their abil-
ities in the methodological area: the complete evaluation process, as a whole,
with its design, implementation, etc.
– To share the experience obtained in real process evaluations through networks,
forums, and articles to communicate the challenges, lessons learned and recom-
mendations, to strengthen the evaluation capacities of young people interested
in developing as evaluators in Costa Rica and the region.
– To consider how to reach the stakeholders so they will use the results of the
evaluation: Mideplan indicates that “the elements of improvement must be for-
mulated under a constructive approach which promotes corrective actions rath-
er than resistance” (2017a: 6); and the evaluation should be considered as “a
mean that generates opportunities for improvement for management, it deepens
knowledge and learning about the intervention and identifies new spaces for ac-
tion” (2017b: 69). As a YEET, an important lesson is how to approach the dif-
ferent stakeholders so they consider the process important, necessary and un-
derstandable to the point of using the results and recommendations, and not as
a threat.
The final stage of the evaluation, the presentation or return of results, must be as
important as any other stage of the evaluation, and has to consider the character-
istics, position, capacities, and context of each group of stakeholders. Considering
that, the recommendations must be practical and feasible, as should be “an action

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
C. A. Montero Corrales, L. Brenes-Alfaro: Young and Emerging Evaluators Team 159

proposal, based on a conclusion, that guides decision-making (...), that adds signif-
icant value and is presented using accessible language” (Mideplan 2017a: 13-14).

7. Conclusions

Following the comprehensive and detailed evaluation exercise, the YEET concluded
that it is necessary to carry out actions to promote the inclusion of YEE in different
areas related to the field of program evaluation. Processes like this are very valu-
able and generate the knowledge and learning that the YEE need and are looking
for, according to the national and international context. The creation of YEE teams,
as well as the presence of YEE on teams with senior evaluators, must be considered
in evaluation processes.
The young and emerging evaluators need to learn how to manage a real process
evaluation, how to overcome the most common methodological challenges and how
to improve the usefulness of the evaluation depending on their organizational con-
text. Concerning the evaluand of the present study, it is essential to strengthen the
evaluation capacities to develop more processes like this, to transform the quality of
the local governments’ services and to improve public management.
All in all, this evaluation was an enriching experience and a feedback space
provided with important tools that will also allow taking on new evaluation expe-
riences both in Costa Rica and abroad.

8. References

Asamblea Legislativa de la República de Costa Rica (2010): Law N° 8801. Ley General de Transfe-
rencia de Competencias del Poder Ejecutivo a las Municipalidades. San José, Costa Rica.
Billings, Meghan (2017): The World of Evaluation: Challenges Faced by Student Evaluators. In: Jour-
nal of Multidisciplinary Evaluation, 13 (28), pp. 24-28.
DEval (2017): Evaluation of Municipal Services in Costa Rica. Activity Requested: “Team of Young
and Emerging Evaluators for the Evaluation of Municipal Services in Costa Rica” (TdR). San
José, Costa Rica: DEval – FOCEVAL.
DEval (n.d.): PROJECT: Evaluation Capacity Development in Selected Countries in Latin America –
FOCEVAL. Retrieved from: https://www.deval.org/en/projekt-foceval.html [October 26, 2019].
EvalPartners (2019): EvalYouth Initiative: Why EvalYouth? Retrieved from: https://www.evalpartners.
org/evalyouth#why [November 17, 2019].
Leandro, Vilma (2018): Lecciones Aprendidas y Retos en la Implementación de Evaluaciones o de
TFIAS: Intercambio EJES Ecuador/Costa Rica. Semana de la Evaluación en América Latina y
el Caribe 2018.
Mideplan. (2017a): Guía de Informes de Evaluación. Orientaciones para su Elaboración: Estructura y
Contenido. San José, Costa Rica: MIDEPLAN.
Mideplan. (2017b): Manual de Evaluación para Intervenciones Públicas: Gestión de Evaluaciones en
el Sistema Nacional de Planificación. San José, Costa Rica: MIDEPLAN.
Project Smart. (2019): Lessons Learned: A Cost-Effective Project Management Tool. Retrieved from:
http://www.projectsmart.co.uk/lessons-learned.html [November 24, 2019].

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro
160 Zeitschrift für Evaluation, Jg. 19, Heft 1, 2020 – Information & Service, S. 149-160

Spilsbury, Mike/Perch, Catrina/Norgbey, Seg/Rauniyar, Ganesh/Battaglino, Cristina (2007): Lessons


Learned from Evaluation. A Platform for Sharing Knowledge (Special Study Paper Number 2).
Nairobi, Kenya: United Nations Environment Programme (UNEP).
Weber, Rosina/Aha, David W./Becerra-Fernández, Irma (2001): Intelligent Lessons Learned Systems.
In: International Journal of Expert Systems Research & Applications, 20 (1), pp. 17-34. https://
doi.org/10.1016/S0957-4174(00)00046-4

© Waxmann Verlag GmbH | Digital offprint for


Carlos Alberto Montero Corrales & Laura Brenes-Alfaro