Krystal Smith, Advanced Research MSW Graduate Student Laurie Moses, Advanced Research MSW Graduate Student Suzanne Martin, Advanced Research MSW Graduate Student Erica Smith, Advanced Research MSW Graduate Student Brittney Asch, Advanced Research MSW Graduate Student Marcie Fisher-Borne, PhD, MSW, MPH, Assistant Professor, Department of Social Work
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page ii
TABLE OF CONTENTS TABLE OF CONTENTS ............................................................................................................................................. ii LIST OF TABLES .......................................................................................................................................................... iv ACKNOWLEDGEMENTS .......................................................................................................................................... 1 EXECUTIVE SUMMARY ........................................................................................................................................... 2 PROJECT BACKGROUND ........................................................................................................................................ 2 LITERATURE REVIEW ............................................................................................................................................. 5 Overview of Hunger ....................................................................................................................................................... 5 Defining Hunger and Malnutrition .............................................................................................................................. 5 Monitoring and Evaluation: Organizational Capacity and Current Trends ......................................................... 6 Building Quality Partnerships: Understanding Success ........................................................................................... 7 EVALUATION JUSTIFICATION AND METHODS ....................................................................................... 8 Constituent Groups Involved: Internal Staff and Board Stakeholders and Lead Partners ................................. 8 Methods ............................................................................................................................................................................ 8 KEY FINDINGS SHN STAFF AND BOARD ................................................................................................ 13 Defining Hunger ........................................................................................................................................................... 13 Defining Monitoring and Evaluation ......................................................................................................................... 13 Perceptions of M&E in SHN: Current Status and Future Needs ......................................................................... 14 The Value of Monitoring and Evaluation for SHN Stakeholders ........................................................................ 14 Monitoring and Evaluation: Barriers and Challenges .............................................................................................. 15 The Future of M&E ..................................................................................................................................................... 17 KEY FINDINGS PARTNER ORGANIZATIONS ........................................................................................ 19 Partner Characteristics ............................................................................................................................................. 19 Partners Perceptions of SHN ...................................................................................................................................... 20 Partner Monitoring and Evaluation Capacity ........................................................................................................... 23 DISCUSSION AND SUMMARY ............................................................................................................................. 25 RECOMMENDATIONS ........................................................................................................................................... 28 REFERENCES .............................................................................................................................................................. 31 Appendix A .................................................................................................................................................................... 32 Appendix B .................................................................................................................................................................... 33 SHN Monitoring and Evaluation Stakeholder Report, 2014 Page iii
Appendix C .................................................................................................................................................................... 35 Appendix D .................................................................................................................................................................... 45 Appendix E .................................................................................................................................................................... 49 Appendix F ..................................................................................................................................................................... 57 Appendix G .................................................................................................................................................................... 61 Appendix H .................................................................................................................................................................... 63 Appendix I...................................................................................................................................................................... 63
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page iv
LIST OF TABLES TABLE OF TABLES Table 1: Hunger Related Terminology..12 Table 2: Key Terms and Definitions.13 Table 3: Summary of Quantitative Data Sources...16 Table 4: Summary of Qualitative Data Sources.....18 Table 5: Lead Partners Total Food Received27 Table 6: Survey Responses on Types of SHN Support Received...28
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 1
ACKNOWLEDGEMENTS We wish to thank the following people for their assistance and support with this evaluation: Thank you to the Stop Hunger Now (SHN) partners who participated in the interviews. These individuals work tirelessly with minimal resources and staff. They gave us their valuable time for the interviews and truly understood the importance of participation. Without their input our team could not create this report which is intended to assist SHN with increasing their capacity for program monitoring and evaluation. To the SHN staff, leadership team, and board members we interviewed - thank you for allowing us to speak with you about your perspectives as SHN stakeholders. Your dedication to the mission of Stop Hunger Now, to end hunger in our lifetime, is exemplified by the work you do at every level. Your contribution to our research has provided a more holistic understanding of SHNs organization and the passionate people who work for and with SHN. Our team is especially grateful for Nina DaSilva Batista and Matt Anlyan, SHN Aid Distribution Managers, who worked with us to provide a thorough understanding of SHN as an organization, SHNs lead partners, and an overview of their experiences working with various partners over the years. Your collaboration and feedback has been paramount to our work. We also give a very special thank you to our professor Dr. Marcie Fisher-Borne, Assistant Professor in the Department of Social Work at North Carolina State University. From the onset of our Advanced Research course your enthusiasm for teaching and research has been contagious! By trusting the process we have created, what we hope to be, a meaningful report with relevant and useful information. Upon completion of this course, we will take away a once-in-a-lifetime experience, for which we are eternally grateful. With your guidance we have developed our own real- world experience working with the international aid organization, Stop Hunger Now, which will serve as a catalyst for our continued work as professionals. We are more competent, confident, and committed to becoming leaders who advocate for social justice on every front. We would like to thank the rest of our Advanced Research Student Colleagues. Though this has been a different experience for each of us, we have gained so much by sharing this journey together. It is said there is strength in numbers and our group is no exception. We have achieved what we sought and look to the future knowing there ARE great people out there doing amazing work - we know this because we know each of you will be out there. Last but not least, thank you to our Department of Social Work at North Carolina State University. This international section is a first-time experience for the Department and we appreciate the opportunity to be a part of it! We believe it is courses like this that will demonstrate and highlight the amazing talent within our Masters of Social Work Program. Thank you for your willingness to share in our journey and your support for our education.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 2
EXECUTIVE SUMMARY Project Background and Literature Stop Hunger Now partners with organizations around the world to provide food and other resources to over a thousand facilities and encourages transformational development in the communities served. This report, built out of the NCSU Collaborative of 13 Social Work graduate students and Professor Dr. Fisher-Borne, seeks to bring the perspectives of key internal (staff and board) and external (lead and on the ground partners) into dialogue to understand how Stop Hunger Now can be more effective in partner relationships and building the monitoring and evaluation (M&E) programs stakeholders need and want in order to be successful. Evaluation Justification and Methods Evaluating and understanding the needs of Stop Hunger Now stakeholders are vital to the success and growth of the organization. This evaluation addressed the following questions: 1. What are the perspectives of Stop Hunger Now staff and board members on the nature of Stop Hunger Nows efforts to fight hunger and what role does monitoring and evaluation play in this work? 2. How do partners describe the impact of Stop Hunger Now meals? 3. How do partners define partnership success? 4. How do partners explain their capacity for M&E 5. How would partners like to work with Stop Hunger Now to further their M&E? Key stakeholder interviews were conducted with 11 members of the Stop Hunger Now staff, executive leadership, and board in order to understand the current perceptions, needs, and vision for monitoring and evaluation. Thirteen lead partners were selected to be interviewed about their experiences as partners with Stop Hunger Now and the impact of Stop Hunger Now meals in their communities. In addition, 21 of 26 lead partners completed survey regarding the quality of partnership with Stop Hunger Now, the technology available to them, and how they would like to move forward with monitoring and evaluation. These data were combined with 45 SHN Impact reports submitted in 2012. Evaluation Findings The stakeholders interviewed chose varying definitions for how Stop Hunger Now defines hunger and monitoring and evaluation, with some respondents focused on the impact of food at the individual or facility level, and others focusing on Stop Hunger Now's effectiveness and accountability as an organization. The majority of stakeholders felt that monitoring and evaluation within the organization was just getting started. Many believed M&E would require additional SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 3
resources and that the current processes in place were not sufficient to capture the work of Stop Hunger Now. Most stakeholders saw M&E as critical to their day-to-day work that would demonstrate to donors a "return on investment." A lack of internal M&E capacity and investment along with challenges with partner capacity to conduct M&E were all cited as issues. Internal stakeholders requested a clear timeline for the process of developing monitoring and evaluation systems that were in line with organizational priorities. The lead partners interviewed described how partnering with Stop Hunger Now impacted their work in substantial ways. The communication and passion displayed by Stop Hunger Now distribution staff was cited by lead partners as the biggest strength. Partners requested additional training on partnership expectations as well as resources and strategies to conduct M&E more effectively. Partners expressed the difficult realities of tracking food to individual facilities as they were often understaffed and unequipped with record-keeping tools such as computers and Internet. This, along with difficulty transporting the food to various on-the-ground partners due to high costs, were some of the major barriers expressed by partners interviewed. Discussion and Summary A review of the findings revealed a critical need for clarity and cohesion around the organizational definition of hunger and the overarching strategies Stop Hunger Now should prioritize to address the issue of hunger. There was not unity amongst staff and board around these overarching strategies. With increased clarity on how Stop Hunger Now is moving towards its mission, Stop Hunger Now will be able to develop a detailed, sustainable M&E program that can further inform organizational development and demonstrate the worth of investing in Stop Hunger Now. The findings also indicate that Stop Hunger Now has an incredible opportunity to build and support the M&E capacity of their 26 lead partners as a strategy to improve data tracking and overall impact. Recommendations 1) Engage in a strategic planning process that would lead to concrete goals and priorities around impact outlining priorities for the next 1-3 years. This process should involve: a. Clarifying key definitions related to mission such as hunger b. Integrating best practices from current global international development work c. Defining and operationalizing transformational development and how efforts labeled as such might impact Stop Hunger Now priorities d. Defining and clarifying partner priorities 2) Determine the role and priorities of organizational monitoring and evaluation that are designed to build partner capacity for M&E. 3) Continue to build collaborative university partnerships around M&E to maximize resources, impact students and universities, and objectively assess impact. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 4
PROJECT BACKGROUND The North Carolina State University Department of Social Work partnered with Stop Hunter Now (SHN) over the 2013-2014 academic year to build monitoring and evaluation (M&E) capacity. In 2011, Stop Hunger Now partnered with UNC Chapel Hill to produce monitoring and evaluation reports for Uganda and Nicaragua and to create forms to assist with future monitoring and evaluation (M&E) efforts. The SHN NCSU partnership began through Dr. Marcie Fisher-Bornes application for a $15,000 Extension, Engagement, & Economic Development Seed Grant to fund the collaborative and assist with travel costs for students to conduct site visits. Through this partnership, 14 advanced graduate research students in Social Work undertook five key projects for SHN: 1) The creation of a training manual for SHN staff and volunteers participating in monitoring and evaluation trips 2) The creation of logic models that communicate core SHN organizational elements (see Appendix I in report for the organization logic model) 3) A country site visit and report for Haiti 4) A country site visit and report for the Dominican Republic 5) A stakeholder report collecting data from SHN executive staff, board members, and SHN lead partners Report Focus This specific report focuses on the perspectives of key internal and external stakeholders (Board, leadership Staff, and lead partners) on addressing global hunger, qualities of successful partnerships, and the role of monitoring and evaluation in the efforts to end hunger. Stop Hunger Now works with partners around the globe to impact the issue of hunger. As such, understanding what makes a successful partnership is vital. This report sought to: Identify and describe current challenges and strengths within the organization that may impact the development of the M&E program Gain understanding of the opinions of stakeholders on what steps to take moving forward with M&E strategies Collect valuable process information from internal and external stakeholders to be used to improve SHN monitoring and evaluation in the future Gain perspective and assess SHN partners overall experiences with SHN and SHN meals
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 5
LITERATURE REVIEW Overview of Hunger There are currently 842 million people suffering from chronic hunger (1 in 8 people globally), according to the Food and Agriculture Organization (FAO) (2013a). Undernutrition contributes to 3.1 million deaths of children under age five each year, almost half of the total worldwide deaths (Black et al., 2013). The majority of those suffering from chronic hunger and malnutrition live in developing countries. Hunger affects the body and the mind of individuals in several ways. An individual who experiences hunger for long periods of time demonstrates stunting, poorer immune system, such as staying sick longer or developing malaria, and an increase in emotional and psychological difficulties, such as depression (Dutta, Anaele, & Jones, 2013; Kirkpatrick, McIntyre, & Potestto, 2010). Defining Hunger and Malnutrition Hunger is not a condition which is easily confined to one simple definition. Accordingly, the Food and Agriculture Organization (2013a) presents a set of terms, which differentiate the varying manifestations of hunger (see Table 1). Hunger can be endured as a short-term condition or on a chronic basis and may range in levels of severity from mild to life-threatening. Table 1: Hunger Related Terminology from FAO (2013a) Term Definition Hunger A state, lasting for at least one year, of inability to acquire enough food, defined as a level of food intake insufficient to meet dietary energy requirements. Food Insecurity A situation that exists when people lack severe access to sufficient amounts of safe and nutritious food for normal growth and development and an active and healthy life. It may be caused by the unavailability of food, insufficient purchasing power, inappropriate distribution or inadequate use of food at the household level. Food insecurity may be chronic, seasonal or transitory. Malnutrition An abnormal physiological condition caused by inadequate, unbalanced or excessive consumption of macronutrient and /or micronutrients. Malnutrition includes under-nutrition and over-nutrition as well as micronutrient deficiencies. Undernourishment The outcome of undernourishment, and /or poor absorption and/or poor biological use of nutrients consumed as a response to repeated infectious disease. It includes being underweight for ones age, too short for ones age (stunted), dangerously thin for ones height (wasted) and deficient in vitamins and minerals (micronutrient malnutrition) SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 6
Defining Monitoring and Evaluation In order to justify the importance of monitoring and evaluation in fighting global hunger, we first must define key terms. Table 2 provides the definitions for terms used throughout this report. Table 2: Key Terms and Definitions Key Term Definition Monitoring The systematic and routine collection of information from projects or programs to improve activities. It also creates accountability to stakeholders. Evaluation To assess a completed project or program (or a phase of an ongoing project or program) to see whether it has or had the desired impact (Sport and Development, 2013). Capacity Refers to having the resources, skills, and knowledge to achieve desired performance or effectiveness (Connolly, 2007). Basic Training A term that consists of helping partners determines their long-term goals, their short term goals, and ways to move towards accomplishing those goals. On-the-Ground Partner Refers to any organization that has and is distributing Stop Hunger Now meals Monitoring and Evaluation: Organizational Capacity and Current Trends From funders to charity-rating groups such as Charity-Navigator, increasingly organizations are being asked to show mission-related results and how resources are being utilized to achieve expected outcomes. In an era of deeper organizational accountability to funders, monitoring and evaluation plays a critical role in demonstrating that organizations are serious about assessing impact. For example, by 2016, Charity Navigator will include an NGO rating system based on results. See Appendix A for the five core elements that will be asssessed under the new guidelines. Monitoring and Evaluation is a crucial piece of organizational capacity and is highly beneficial for improving services and increasing impact (Carman 2011; Fine, Thayer, & Coghlan, 2000; Forti & Yazbak, 2012). Forti and Yazbak (2012) looked in-depth at nonprofits that are most effective in evaluating the services they provide, such as Youth Villages and ROCA. The leaders of these organizations argued it is possible to do performance measurement with few resources, if you identify one or two things to examine in a more methodical way. They highlighted the importance of building communities of practice amongst measurement staff and to equip all staff members l to encorporate measurement in their work from frontline practitioners to the CEO (Forti & Yazbak, 2012, p. 17). Organizations that demonstrate impact attract resources to scale (Forti & Yazbak, SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 7
2012, p. 2) and the benefits outweigh the costs: both to individual constituents and to the organizations overall health and capacity. USAID published a report on current trends in evaluation policy and practice and the overarching goal of enhancing aid effectiveness. The report argued that evaluation is only valuable when findings are applied and performance is improved (Hageboeck, 2009). Current M&E trends also include the centrality of stakeholder involvement in developing evaluation questions, gathering and analyzing data, and formulating action recommendations (Hageboeck, 2009). Participatory evaluation is a central concept that is in line with reorientation of the UN to treat beneficiaries as equal partners and consultants in order to sustain intended project benefits. Funding demands have heightened for food relief efforts that are credible and demonstrate the strength of evidence through rigorous evaluation methods (Hageboeck, 2009). Building Quality Partnerships: Understanding Success Creating partnerships that are successful and understanding the communities needs are vital to create real change around the complex issue of global hunger. Working with partners in the community can be challenging, particularly for international organizations. These partnerships need to recognize and be able to integrate the assets that the community organizations have to offer they know best what their community needs to be successful (Connors & Prelip, 2000). Creating a strong partnership is important, but maintaining the partnership is difficult and takes additional work. Creating clear boundaries and a framework in the beginning of the partnership will prevent confusion and allow for clear communication (Rein & Stott, 2008). With good communication comes trust that both partners will have the resources they need and will be able to rely on each other (Connors & Prelip, 2000). Partnerships must also be able to review their successes and failures to create a stronger partnership that can be sustainable (Heady, 2000). Knowing the community needs leads to creating a solution that will solve the issue of hunger and create an environment that prevents it from reoccurring. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 8
EVALUATION JUSTIFICATION AND METHODS This mixed methods evaluation is a component of a larger North Carolina State University (NCSU) Monitoring and Evaluation Collaborative, led by Dr. Marcie Fisher-Borne and graduate student consultants in the Masters of Social Work programs Advanced Research and Evaluation Sequence. All research activities were approved by the North Carolina State University Institutional Review Board in December of 2013. This evaluation project had two primary constituent groups and separate evaluation goals for each. Constituent Groups Involved: Internal Staff and Board Stakeholders and Lead Partners Evaluation Goals: Internal Staff and Board Stakeholders Understand the perspective of key staff and board members regarding hunger, monitoring and evaluation, how M&E impacts their work, and what they need from M&E Identify and describe current challenges and strengths within the organization that may impact the development of the M&E program Gain understanding of the opinions of stakeholders on what steps to take moving forward Collect valuable process information to be used to improve SHN monitoring and evaluation in the future Evaluation Goals: SHN Partners Gain perspective and assess SHN partners experiences with SHN and SHN meals Identify and describe partner perceptions of program achievements, implementation facilitators and barriers Collect valuable process information to be used to improve SHN monitoring and evaluation in the future See Appendix B for the detailed evaluation timeline. The section that follows describes the studys data collection method, data sources, and data analysis strategies. Methods Data sources for the evaluation included both quantitative and qualitative data collected from SHN lead partners and SHN key staff and board members. The section that follows describes each type of data collection method, data sources, and data analysis strategies. Quantitative Data Collection Quantitative data included surveys and reports submitted by lead partner organizations. The Stop Hunger Now Partner Survey consisted of 34 questions and was administered online through Qualtrics SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 9
survey software. All 26 lead partners were asked to complete the survey, and 22 lead partners (85%) had completed surveys at the time of analysis. The survey addressed partner characteristics, perceptions of the relationship with Stop Hunger Now, lead partners relationships with their on-the- ground partners, and their overall capacity for monitoring and evaluation. See Appendix C for the partner survey instrument. Lead partners had the option of including their name at the end of the survey or leave the survey anonymous. All participants who completed the survey were entered into a raffle for a $50 Visa gift card. Additional quantitative data came from the 2012 Impact Reports. All partners are expected to complete annual impact reports where they report on meal distribution to individual facilities, data collected on the impact of meals on beneficiaries, and any recommendations or challenges they would like to address with Stop Hunger Now. In Fall 2012, all reports (n=45) were compiled in an Excel document and a summary report was completed by the SHN Social Work Graduate Intern. Impact Reports were submitted by lead partners on a country basis, so partners receiving SHN shipments in multiple countries must submit reports for each country. See Appendix D for the Impact Report form and Appendix E for the 2012 Impact Report Summary. See Table 3 for a summary of quantitative data sources. Table 3. Summary of Quantitative Data Sources Qualitative Data Collection The two principal qualitative sources for this evaluation were 13 interviews with lead and on the ground partners and 11 interviews with key internal stakeholders from Stop Hunger Now. The NCSU research team drafted a semi-structured Partnership Success Interview Guide (Appendix F) that was reviewed by the Principal Investigator and SHN Aid Distribution Staff. Lead partners who exemplified partner success were selected by the Aid Distribution staff and Social Work Intern and invited to be interviewed. Of all partners invited, only two declined the opportunity to be interviewed, and two more partners were selected to be interviewed instead. Thirteen graduate social work students from North Carolina State University conducted these interviews. Each student interviewed one SHN partner who was identified by SHN staff for an interview. To ensure consistency, training was conducted with those conducting the interviews. Partners were asked a set of 15 open-ended questions to help structure the interview process and make sure partners answer the same questions. These questions were created to develop an understanding of the partners Data Sources Total Number Total Lead Partners 26 Lead Partner Surveys Completed 22 Document Review 2012 Impact Reports 45 2012 Impact Report Summary 1 SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 10
relationship to SHN and the impact of their partnership between SHN as well as to identify key components of successful partnerships. Questions focused on the partnership with SHN, experiences partners had with SHN food distribution and meals, and communication between partners. Lastly, questions focused on the impact SHN meals had on the partners with their direct beneficiaries and in the community as a whole. Handwritten notes were used to record participants answers and interviews were recorded to confirm accuracy of data and allow interviewers to review information they might have missed (Harvey, 2011). Interviews took from 30 to 60 minutes to complete. The CitiHope International interview was conducted by the Research Team when we traveled to the Dominican Republic in February of 2014. The Internal Stakeholder Interview guide was created by the SHN Aid Distribution Staff and two consultants: Doug Johnson and Dr. Marcie Fisher-Borne, the Principal Investigator for the NCSU research team. This guide asked 10 open-ended questions regarding stakeholders understanding of hunger and monitoring and evaluation, the role of monitoring and evaluation in their work with Stop Hunger Now, their perceptions of the current status of monitoring and evaluation and how they would like it to be in the future (see Appendix G). These interviews were conducted by the Aid Distribution staff and consultant Doug Johnson in person and by phone. Aid distribution staff took notes on respondents answers to each question. Members of the NCSU research reviewed notes, interview recordings and transcribed each interview. For a complete summary of qualitative sources, see Table 4. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 11
Table 4. Summary of Qualitative Sources Data Sources Total Number Total Lead Partner Interviews 13 Lead SHN Partner Organization Alliance for Children Everywhere Zambia Children of Vietnam Vietnam Convoy of Hope Haiti, El Salvador, Kenya, Philippians
Ganta United Methodist Mission Liberia Guatemala Village Partners Guatemala Haiti Outreach Ministries Haiti Hearts and Hands for Haiti Haiti Kidz Konnect 4 Jesus Belize Lesea Global Feed the Hungry Uganda ORPHANetwork Nicaragua Reach Now International* (Missions Touch Ministries - intermediate partner- interviewed) Zambia, Haiti, Honduras, Nicaragua, Somalia
SHN of Southern Africa (affiliate) South Africa St. Andres Episcopal School (intermediate partner) Haiti Stakeholder Interviews 11 Internal SHN Stakeholder Interviews Executive Staff 5 Non-Executive Staff 4 Board Members 2 Total Interviews 24
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 12
Quantitative Analysis The online survey was analyzed using Qualtrics software. Responses were incorporated into the findings of the partner success interviews. Impact reports were organized by section and partner in an excel document then summarized to highlight level of M&E being conducted by partners, challenges and barriers partners face, innovative uses of meals, bags, and boxes, unique transformational development initiatives, and other information that can be used by SHN to improve services. The 2012 Impact Report Summary findings were also incorporated into the Partner Success and Survey findings. Qualitative Analysis For all stakeholder interviews, data related to each evaluation question were extracted from the interviews (see Appendix F and G for interview guides) and sorted into a codebook matrix that summarized major themes by data source using Atlas.ti software (for partner success interviews) and Google Drive documents and folders (for Stakeholder Interviews). Analysis of data content across sources identified instances in which multiple respondents identified common experiences and ideas. This process allowed for comparison and contrasting of themes related to common program components and objectives. Data were analyzed according to content analysis procedures and transcripts were read and coded independently to identify initial themes. This coding was checked by the research team for consistency and accuracy. Any quotations presented reflect the general themes and opinions of a majority of the participants. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 13
KEY FINDINGS SHN STAFF AND BOARD This section highlights key findings from interviews with 11 SHN leadership staff and board members in January of 2014 and addresses findings in light of the studys evaluation questions. Defining Hunger The first question asked of all stakeholders was to identify the definition of hunger that most closely aligned with the work of SHN. Definitions of undernourishment, food insecurity, and chronic hunger developed by FAO from Table 1 on page 5 of this report were shared with all stakeholders and board members interviewed. Five stakeholders (45%) chose the definition of food insecurity reasoning that its about having enough access to food, it spells out the causes of hunger, and hunger is really defined by people who are truly dying of starvation. These quotes illustrate that even though respondents chose the same answer, they had varying ways of conceptualizing hunger. Three stakeholders (27%) identified chronic hunger as the organizations leading definition of hunger. Their explanations included, were really trying to stop starvation and I view hunger as involving a persistent basis. The remaining three respondents created their own definitions of hunger. One connected hunger directly to school feeding. Another explained, Hunger is a lack of food and we deal with that by packaging food. Finally, two respondents noted that SHN deals with hunger that is chronic as opposed to simply starvation, and that hunger is the lack of sustainable sources of food, that are both safe and high quality. Defining Monitoring and Evaluation Respondents were also asked to broadly define monitoring and evaluation (M&E). This question elicited a wide variety of responses with some major themes. Of all respondents, 45% said that monitoring and evaluation were two separate things with separate meanings; the majority defined M&E as one concept. Six (54%) connected evaluation to impact and five (45%) connected monitoring to process. Some chose to discuss the purpose of M&E for SHN. Three individuals (27%) said that the purpose of monitoring and evaluation is to inform the organizations actions and decision-making. Two respondents said that M&E helps to show whether the organization is moving towards its mission. Three respondents (18%) framed M&E in terms of the impact SHN has on individuals eating the meals regularly (2 specifically talking about children). Three stakeholders (27%) framed monitoring in terms of the partners use of food and the expectations set for partners. Three stakeholders framed monitoring in terms of SHNs effectiveness in managing the resources they are entrusted with (i.e. meals, money etc.).
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 14
Perceptions of M&E in SHN: Current Status and Future Needs To be honest, Im glad we are having this conversation, because I dont know a ton about it. --Respondent Staff and board members shared varying perceptions of SHNs current monitoring and evaluation process. They also shared different views on how the organization currently gathered and understood data. Two respondents openly stated that they did not know what was happening with M&E. One respondent shared concerns that frontline staff was less aware of what was happening and unclear why the organization was focusing efforts on M&E. In particular, some concerns were focused on the ability to provide tangible data on impact. Another stakeholder stated that more collaboration along with communication was needed around the data and its impact: The story is lost and it becomes about how many meals you packaged. We forget that we are here to end hunger. The majority of stakeholders talked about M&E just getting started or being fairly surface level. Three stakeholders cited a lack of resources and financial support for M&E as a challenge. Some stakeholders were concerned with how SHN could move forward because we shouldnt measure things we cant prove or because there was not a clear understanding across the organization in regards to the value of M&E. Another stakeholder said that SHN lacked a specific process, data points, [or a] distribution/selection process. Two stakeholders were concerned about a lack of a scientific approach. On the other hand, one stakeholder believed that SHN was already collecting a fair amount of data and SHN needed to work to show what we are collecting. Another felt we are really doing a good job with current resources. Some stakeholders also felt that SHN was at a critical point because of how quickly SHN had grown: Weve set ourselves up as a leader in this field and now we need to back that up. Finally, some stakeholders shared concerns regarding the processes that were currently in place. Specifically, there were concerns that impact forms were sent out and not sent back, that meals needed to be counted and verified (a higher standards of accountability with partners), and that the SHNs distribution model made M&E very complex and challenging. Multiple stakeholders saw partners limitations with data collection and analysis as another key barrier in moving forward with M&E. The Value of Monitoring and Evaluation for SHN Stakeholders Everyone agrees its a good idea, but if people dont put an effort into it then its useless. . . If we dont get this addressed in 3 - 5 years we wont be here because other people will and they will do it [evaluation] better. --Respondent All stakeholders were asked to identify how monitoring and evaluation affected their work and what information they (and their team) needed in order to be successful in meeting their work goals. Three respondents said that M&E was a crucial part of their day-to-day job. One noted, I will be using this data every day, every week. More and more people are interested in what is the impact of our work. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 15
The majority of respondents (6 of 11) discussed how M&E provided crucial information for donor support. Four respondents specifically said that M&E validated their work in a way that was critical to eliciting contributions. For example, one respondent noted: Organization donors like to see more evidence/fact-based, scientific based results from what they are investing in and return on their investment. Corporations, foundations, universities, any of those types of donors will demand that we have the above kinds of results to show them. Another stakeholder shared, for large donations, its probably the number one key barrier that we face to get large dollars, is to lay down results. Its a key piece of what we need to move forward. In fact, nine of the eleven respondents identified communication with potential donors and supports as the primary way M&E affected their work. While increasing donor dollars was a key reason cited to conduct M&E, stakeholders also discussed the importance of achieving the organizational mission: To me it comes down to the mission. How are we achieving our mission? What impact is it having toward ending world hunger? Three of the eleven stakeholders discussed connection to mission as part of how they will use M&E in their job. One respondent noted: I see it as proactively capturing data that indicates impact and that cannot only inform potential donors, but also really inform operations. . . There are all kinds of reasons to have solid data. Three additional respondents affirmed the importance of integrating what is learned through monitoring and evaluation to improve future actions taken by the organization. Monitoring and Evaluation: Barriers and Challenges The major barriers and challenges with M&E identified by stakeholders included: 1. Internal infrastructure and M&E capacity 2. Internal communication and unity to incorporate M&E 3. Ability to assess impact in complex environments and with partner capacity (i.e. in developing countries) Internal Infrastructure and M&E Capacity Things are going to fall apart if we dont have the systems, structures, and processes in place to demonstrate whats going on. --Respondent Two respondents discussed their belief that SHN did not have the infrastructure in place to do the M&E work desired, including information storage, information analysis, and clear processes. Four cited insufficient resources (particularly in terms of staff and money) for M&E as a major barrier: Lack of control and lack of resources. Would love to have people on the ground in each country, but dont have resources . . . Would like to visit every partner in the field every year, but we dont have the capacity. Four stakeholders directly addressed their dissatisfaction with impact reports as the current strategy to collect data from partners. On a logistical level, one stakeholder felt the problem with M&E was the forms we get do not tally the number of meals we send out. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 16
Another said, Im bothered by length of time needed to get reports back. We want to get 100% reporting compliance, or close, but were not there yet. Others felt that the impact reports were imperfect tools-not received regularly, not objective, and often, incomplete. As one stakeholder noted, on sample impact reports, a lot [that] we asked for we didnt get. Are they too tough to answer? Why are they not being answered? Regarding stories, many interviewees said that anecdotal data was all SHN currently collected. Even so, over half the interviewees reported that the stories SHN does have are not sufficient for their needs and they would like more stories to share with external stakeholders and donors. One interviewee, when asked about challenges and barriers, shared, Number one, theres not a lot of stories to tell. It would be awesome toshare a handful of stories. That would go a long way with some of these partners. One stakeholder expressed the gap between talking about individuals who eat SHN meals and the mission of SHN this way: In terms of measuring success you know when you start talking about weight of the student it is not getting to how are you going to change the condition [of hunger]. Developing and Communicating a Unified Vision Its going to be critical we are using consistent talking points and data that is validated. Everyone [is sharing] based on their own experience. If called into question, can we demonstrate data? --Respondent Clear communication, was a consistent challenge named by stakeholders. One respondent discussed how their program staff was confused as to why resources were going into M&E when program staff were rarely asked for hard data on the impact of SHN. One stakeholder said, Weve got to figure out a way that if we are making a significant change in any of the verbiage or numbers or anything that were communicating it effectively across the organization. A few stakeholders expressed that the current M&E processes had not been shared with them and therefore they did not know what is happening. One said, Staff dont understand what other staff are doing which leads to a fall out. There were a few stakeholders concerned about consistent buy in across the organization for M&E. One said [Some] wonder or question the investment, which is going to be substantial, in the M&E when they don't see or hear the need for it, as much as someone on development or corporateRarely at an event do you get grilled about How can you prove your food works?
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 17
Understanding Partner Capacity in the Context of Developing Countries. We distribute aid well, logistically, but have no control over day-to-day. With working internationally, it creates risk that things wont go as planned. Local capacity in country and cultural understanding about M&E are limitations as well. Not a culture of M&E/reporting in developing countries. They also dont have resources, so its hard to collect information from the field. --Respondent A few stakeholders brought up concerns regarding the complex environments where SHN meals and aid are distributed. Five of the eleven stakeholders interviewed brought questions such as: How much can we claim as a result of our food? How are we going to determine what the impact of our meal is? Even if we can get their outcomes, what part does our product play in that outcome? We can report on program but how much claim can we put on it being the result of our product?...We can get a lot of success understanding their program. Its a doubt that we can take it further than this. Stakeholders also expressed concerns about the capacity of SHNs partners to collect data and report data back to SHN. SHN currently has 26 lead partners who are distributing meals to over 1,000 individual facilities around the world. Many of these partners are multinational aid organizations that are U.S. based and distribute SHN meals and aid to smaller organizations in specific countries. In total, four stakeholders discussed the limitations of SHNs partners as a major barrier and challenge to monitoring and evaluation. The Future of M&E Stakeholders were asked the best strategies to move forward with M&E. Specifically, staff and board were asked what types of partners and what types of programs should be prioritized in gathering data and how to conduct more in-depth evaluation of the impact of SHN meals. Gathering Data. There was a range of responses related to how (and with what programs) to gather data. Three respondents said that the organization should no longer focus on crisis/disaster intervention; two felt that crisis intervention was particularly important. Two respondents felt SHN should focus on the [partners] we trust most and whoever we can. Two respondents felt it was important that data be collected for the end users, those who eat the meals. Transformational development, which was equated first with school feeding then with vocational, medical, maternal and child health, was another priority cited by a number of internal stakeholders. Two respondents discussed the necessity of having key performance indicators or criteria for each type of program SHN supports. The majority of stakeholders felt that the most important thing in determining partners was their capacity to collect meaningful data. Specifically, respondents discussed partners ability to prove their success through the data that they give you and that the data should be reliable and consistent. Other respondent simply said SHN should focus data collection with: whoever returns our forms. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 18
Focus of M&E Efforts. Some respondents shared specific ideas as to where SHN should focus M&E efforts. One recommendation was to complete more measuring during early stages of partnership with a new organization in order to see the impact SHN had on their organization and constituents. One stakeholder felt it was better to focus on smaller organizations that do specific focused programs, another felt it was better to work with larger organizations who could already have reporting in place. Two respondents felt it was important to focus on U.S. based partners who were easier to contact. One stakeholder wanted to prioritize partners who have a great opportunity to break the cycle of poverty. Other strategies mentioned by stakeholders were the importance of doing research on organizational standards for M&E. This could include doing surveys and interviews to see what donors or foundations were looking to assess impact. Additional ideas included collaborating with universities and engaging corporate partners with M&E and conducting more M&E trips instead of vision trips. Communication with Staff and Partners. One recommendation related to communication with staff regarding M&E involved having more time on staff calls devoted to sharing results and partner stories. Stakeholders requested more collaboration and communication around the data and how SHN was reaching its mission. Stakeholders requested more anecdotal stories to give a better understanding of the educational piece of SHNs work, as there were funders who would give to education but might not give to hunger. Stakeholders also requested sharing a partner of the month at staff meetings, spending more time on staff calls talking about stories or results, and having a better system for stories as well as sharing partner photos. Others asked for a clearer timeline for the M&E process and coordination across the organization on the flow and formatting of the information shared. The Importance of M&E for Funding. Stakeholders believed that SHN needed to demonstrate that of amongst all the organizations competing for funding, SHN was worth giving to because of the work that was done and the way resources were used to fight hunger. One respondent noted that people will pay more for the ability to truly make a difference. Some respondents requested information on M&E standards and benchmarks related to nonprofit spending and a clearer understanding on what foundations look for in whom they fund. Most respondents, when discussing the needs of volunteers, suggested that volunteers wanted and needed to know where the meals were going coupled with some success stories of individuals and/or facilities. There was a general sentiment that M&E was important not only for grants and large donors, but also individual volunteers as well. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 19
KEY FINDINGS PARTNER ORGANIZATIONS This section highlights findings from interviews conducted with 13 of the 26 current lead partners with Stop Hunger Now (50% of total SHN lead partners), online surveys completed by 22 lead partners (85% of total SHN lead partners), and 45 partners responses to the 2012 Impact Reports. Together, these findings contain feedback from all of Stop Hunger Nows lead partners. Partner Characteristics Lead partner organizations serve anywhere from one to 130 countries and one to over 170 facilities. Table 5 outlines the six lead partner organizations that received the most SHN meals in 2012. Of these six, partners interviewed are highlighted in light gray. On average, partners had worked with Stop Hunger Now for four years, and the partners interviewed provide SHN meals to over 456 facilities, totaling 59% of the SHN meals provided to partners. The interviewed partners complete demographic information can be viewed in Appendix H. Of the partners who completed the survey, 50% reported that Stop Hunger meals constituted 80- 100% of food used in their programs and 71% reported that SHN meals constituted 40-100% of food used in programs. The majority of survey respondents ran the on-the-ground programs directly. The highest ranked priority areas of focus for partners were education (52%), followed by spiritual development (50%) and health (44%). Fifty percent of partners said that their requests for food increased during the last year, and 54% expected their demand for food to increase in 2014. Of partners interviewed, 50% said that their long-term goal was based on self-sustainability so that outside help was no longer needed. Other long term goals included healthier children (12.5%), education (12.5%), and helping children and the communities reach their full potential (12.5%).
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 20
Table 5. Lead Partners Total SHN Food Received 2014 Schedule Organization Country Number of Containers % of total SHN Meals Given Convoy of Hope Haiti, El Salvador, Kenya, Philippines 18 15.4% Reach Now ** Zambia, Haiti, Honduras, Nicaragua, Somalia 18 15.4% Salesian Mission** Haiti/Timor/SW/SL/Sudan 16 13.7% CitiHope International** Dominican Republic 9 7.7% Lesea Global Feed the Hungry** 23 Countries 9 7.7% CRS Burkina Faso 8 6.8% Gray highlighting indicate partners interviewed for this report. ** Indicates partners who completed the online survey. Partners Perceptions of SHN You are the most responsive, supportive, and reliable partner we have. Contact and communication with SHN are super excellent and efficient. --Survey Respondents Partners selected three different ways of defining Stop Hunger Now: a food donor organization (50%), an international development organization (25%) and a faith based relief and development organization (25%). Overwhelmingly, partners expressed satisfaction with the partnership with Stop Hunger Now. When asked in the online survey to rate the quality of partnership with SHN from 1 (poor) to 5 (excellent), 79% partners selected the rating of five and 21% selected four. Almost all partners agreed or strongly agreed that SHN understood the communities they work in, made major contributions to the sector, and were well aligned with their organizational values. When considering what SHN was doing well within their partnerships, 25% of partners interviewed listed the positive reputation of SHN as a strong benefit for their organization. Communication (12.5%), site visits (12.5%), and additional services supported by SHN (12.5%) were also listed as strong elements of the SHN partnership. Twelve (57%) took the opportunity to write compliments to Stop Hunger Now in the final comment box. Of the partners interviewed, 37.5% stated that they were in constant communication with SHN and found this to be a major success of the partnership. All partners stated that they were happy with the current level of communication with SHN and would not change communication in any way. One of the greatest benefits that partners discussed in their interviews was support from SHN that extended beyond the meals. Table 6 illustrated the percentage of partners who received support in addition to meals. In terms of SHNs capacity to grow and change, 82% of partners agreed that SHN learned from its mistakes and made improvements. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 21
Table 6. Survey Responses on Types of SHN Support Received 6. What type of support have you received from Stop Hunger Now in the past? (Check all that apply) # Answer Response % 1 Food/Meals 23 100% 2 Water Filters 5 22% 3 Medical Supplies 11 48% 4 Financial Support 6 26% 5 Other In-Kind donations 9 39% 6 Other (please specify) 6 26%
Partner Experiences with SHN Partners Experiences with Food Distribution Partners shared their experiences with distributing SHN meals, including information on the tracking and delivery of food shipments. Of the partners interviewed, 100% said that they received advance notice of shipments, ranging from three weeks to two months of notice, for when a shipment would be delivered in country. The biggest barriers discussed was 1) the ability of lead partners to track SHN meals as the food was distributed to the smaller, on the ground partners that were more in rural areas (36%), and 2) the cost of distribution (20%). Other barriers identified included: difficulties with customs (13%), trouble transporting the food (8%), or having storage space for the food (4%). Preparing SHN Meals All of the partners interviewed stated that meals were prepared using multiple bags of food instead of preparing one bag at a time. Of the lead partners interviewed, 20% suggested that SHN create larger meal bags that served more than six servings. Many partners shared on the Impact Reports how they incorporated SHN meals to create new recipes. ORPHANetwork of Nicaragua discussed how they separated out soy to create fried cakes, while Salesian Missions used soy to make muffins. At Tarahumara Ministries in Mexico, facilities added beans and more water to create a soup out of the meals. In addition to creative recipes, partners also reported adding to the meals as well. Of the partners who responded to the survey, 60% reported they added another source of protein, such as chicken, or vegetables to add additional nutrients to the meal. When asked to rate the quality of SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 22
SHN meals on a scale from 1 (poor) to 5 (excellent), 85% rated the meals as excellent. The majority of partners interviewed gave this rating because of the nutrients in the SHN food. In the Impact Reports, partners were given the opportunity to comment on taste and acceptability of meals as well as recommend changes. Two partners requested alternatives to soy, five requested better taste, five cited a need for cooking fuel, and two discussed storage issues. When asked about vitamin packet use, five were putting in vitamin packets too soon, one was not putting them in at all, and one partner was using one packet per two bags of Stop Hunger Now rice. After preparing the food, a number of partners reported reusing the materials the meals arrived in. For example, some of the partners interviewed, and some partners who completed the Impact Report, reported using the boxes for carpentry purposes, such as repairing roofs on houses. Some of additional purposes included using shipping boxes in arts and crafts, and using both boxes and bags for storage of materials such as books. According to Kidz Konnect 4 Jesus, The boxes are valued as a Rubbermaid container. In the Impact Report, Salesian Missions stated that the recycled bags are sold by the children to make a profit for their family. Only 17% of partners reported that they were not sure if their partners were using the boxes or bags, and 9% reported that they were not using the bags. The Impact of SHN Meals SHN is a major blessing to the children and people of Belize. It is making a MAJOR impact across a variety of fronts. We could not even begin to do what we do without the assistance from SHN. --Respondent Partners discussed the impact that SHN meals had on their organization and their constituents. In the online survey, 57% of partners reported that SHN meals constituted approximately 81-100% of the food they used in their programs. Interestingly, in the partner interviews, 83% stated that they received additional food from other international partners, such as from Feed My Starving Children and Milk and Medicine. A significant portion of partners interviewed (43%) stated that SHN meals aided their agencys ability to expand programming. When asked in the lead partner interviews how SHN meals benefitted the agency, Ganta United Methodist Mission stated, By receiving the pending food support, the students will be guaranteed at least one good meal a day, which will encourage attendance and attention in class. Another partner said on the survey: SHN is a major blessing to the children and people. It is making a MAJOR impact across a variety of fronts. We could not even begin to do what we do without the assistance from SHN. These statements were a way to highlight the importance of meals within schools. Some partners mentioned the cost effectiveness of meals. Overwhelmingly, the majority of partners cited healthy children as the primary impact of SHN meals. In the Impact Reports, partners who were tracking health indicators reported children gained weight and most SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 23
partners indicated there were significant health improvements in their constituents in 2012 after receiving SHN meals. Overall, partners interviewed felt that the support received from SHN created long-term change in the communities they served. Of the partners interviewed, 20% said that this change was through the partners being able to effectively provide education. Other changes mentioned included allowing parents to support their children (16%), moving toward self-sustainability (12%), and helping people served reach their potential (12%). Some partners did provide suggestions for ways that SHN could be more efficient in meal support with on the ground partners. These changes included making bags of food larger for programs that serve multiple meals (20% of partners interviewed recommended this) and shipping vitamin packets separately (10% of partners interviewed) for countries that could produce and purchase rice. Partner Monitoring and Evaluation Capacity Current M&E Practices of Partners In the online survey, 96% of partners said they collected data or required reporting to monitor work. Of those collecting data, 65% used a standard reporting form consistently, and 26% received reports from on-the ground partners on a monthly basis while 26% can receive them at any time. Partners use of tracking tools for the impact of SHN meals was also observed in the lead partners interview and the impact report. To keep record of how SHN meals were benefitting their agency and people served, 19% of lead partners interviewed kept records of weight, height, or attendance while 14% kept teacher reports. These results were also demonstrated in partners responses to the Impact Report, nine lead partners tracked health indicators and five partners tracked academic indicators such as graduation rates, attendance, and academic performance. Some partners recorded the benefits of SHN meals by taking pictures of children (9%) before and after they began the feeding program. Many partners described the visible impact of the meals on their constituents in the impact reports. Partners also reported in the interviews that they tracked the impact of SHN meals through reports on Body Mass Index (9%), categorizing children into malnourishment groups (5%), and self- reports/stories (5%). Partner Technology Capacity According to the partner survey, 71% of on the ground partners had access to internet and over 83% were comfortable using the internet. However, only 17% of lead partners said their on the ground partners always had access to internet and only 21% said all their on the ground partners had the computers and software they needed to do their work. About 60% of lead partners were in communication with their on the ground partners daily or weekly and 66% of on the ground partners checked email daily. Cell phones were also used for communication: 48% of lead partners reported communicating with on the ground partners via cell phone on a weekly basis. These responses suggest that email is most likely the best method for data collection for on the ground SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 24
partners and lead partners and Stop Hunger Now needs to ensure that the M&E strategies they install are in line with this method of communication. The major technological challenges (in order of prevalence) were access to computers, access to internet, electricity, and the costs associated with technology. Need for Increased Training and Collaboration with Stop Hunger Now Partners expressed interest in receiving training from SHN on the expectations of the partnership. The majority of partners interviewed (67%) stated that they had not received any formal training from SHN since the partnership began. Partners requested specific training on the expectations of SHN to clarify the goals of the partnership. Another request made by partners interviewed involved training on food nutrition and the ways partners can improve their services. Partners also suggested that their agencies increase collaboration with SHN to promote each others work. In the survey, only 37% thought that it was likely or very likely that SHN would make changes to their work as a result of their input in the survey. This indicates that partners may have some doubts about the usage of data they provide to SHN. Nevertheless, partners indicated significant interest in collaborating with M&E. Partners were asked to select two items they would most like SHN to do to improve M&E in the future. 42% of partners wanted SHN to undertake more monitoring with partners by providing staff and support to collect needed data. 38% of partners wanted SHN to help partners monitor in ways that are useful for them and the people they work with. 29% of partners wanted SHN to provide more training and assistance for partners to monitor and report on work. 21% of partners wanted SHN to focus more attention on long term changes. 17% of lead partners wanted SHN to draw more on partners expertise in developing ways to monitor progress. Most importantly, 92% of partners reported that they did not want Stop Hunger Now to communicate directly with their on the ground partners for data collecting and monitoring. Reasons for this response included creating extra work and confusion for on the ground partners, changing the context of relationships, language barriers, lack of knowledge of on the ground partners and desire to collect data together. One respondent said We are very familiar with our partners and understand their capacity. We also have our own goals for data collection and do not want to give conflicting messages. In interviews, partners were asked to consider how to further strengthen partnerships and many respondents indicated they would like more M&E resources to track how the meals affect their community and the individuals. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 25
DISCUSSION AND SUMMARY This qualitative evaluation sought to (1) understand key SHN staff and board members perspectives on hunger and evaluation; (2) understand the process of tracking SHN meal distribution and how food is creating an impact for partners; and, (3) determine the strengths and areas of growth for SHN partnerships to be used to improve SHN in the future. Below, we highlight key areas that demonstrate strengths such as communication between partners, as well as the actions needed to ensure the program is moving towards its broader goals of impacting hunger. 1. Defining Hunger: Need for Clarity and Cohesion With internal stakeholder interviews (SHN lead staff and board members), the first interview question elicited an important finding: there was a lack of clarity and cohesion around the organizational definition of hunger. Respondents chose different definitions from a sample list and provided varying explanations for their choice. According to the stakeholders answers, the following things were not clear: Whether SHN is focused on alleviating hunger in the lives of individuals or in communities Whether the focus is on caloric intake or micronutrient deficiencies The degree of hunger SHN aims to address Unpacking the very broad mission of ending hunger is a critical first step to setting outcomes to assess impact. Depending on how hunger is defined, the ultimate outcomes will be different; therefore, monitoring and evaluation structures will be different. 2. SHN Mission: Need for Measurable Outcomes Many stakeholders explicitly discussed the importance of demonstrating progress toward the mission. Having a mission of ending hunger in our lifetime makes it challenging to show concrete progress toward achieving the mission, as it is very broad. Furthermore, the resources of SHN are limited. Some stakeholders explicitly addressed this concern in their interviews. The findings show that there may be insufficient strategic planning, or communication of strategic plans related to impact and the SHN mission of ending hunger. There was also a stated need for increased awareness of the best practices and recent research on development work. Most partners surveyed reported that the root causes of hunger in their communities were unemployment and lack of agriculture. If Stop Hunger Now is working to end hunger, a focus on the root causes of hunger in the communities served is critical. Stakeholders did not address the component of their mission around creating a movement by engaging volunteers, even though the majority of Stop Hunger Now staff efforts are spent on meal packaging events. Once clear, measurable outcomes and organizational priorities of SHN are cohesively defined, SHN staff can work towards a clear understanding of M&E and make decisions based on their organizational priorities. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 26
3. Monitoring and Evaluation: Need for Clarity and Cohesion No clear or consistent definition or purpose of monitoring and evaluation emerged from the findings. Stakeholders seemed to be divided on: How internal monitoring and evaluation should be done How monitoring and evaluation would be done with SHNs partners The primary purpose of M&E: Donor support, program improvement, or measure movement toward mission Furthermore, internal stakeholder interviews discussed impact and M&E methods that cannot be reconciled with current Stop Hunger Now activities. Most stakeholders talked about impact on individuals or facilities, but at this point, SHN meal distribution is done through 26 lead partners. Meals are given in shipments to each of these partners, who then determine how meals are distributed to individual facilities within particular countries. There are a few partners who directly manage a few facilities where all the meals go, but most are distributing boxes of meals to many different organizations in many different regions of countries. It may be more feasible for Stop Hunger Now to work with their 26 lead partners than the over 1,000 on the ground facilities in 28 countries throughout the world. Furthermore, partner responses in the survey and interviews suggest they would like to be part of the M&E process and would prefer Stop Hunger Now to work directly with them versus their on the ground facilities. 4. Partnership Success and Relationships with Lead Partners Partners were never explicitly asked to provide a definition of partnership success but were asked about their perspectives on key characteristics of partnership: communication with SHN, training from SHN, and how they understand the needs of the facilities where they distribute SHN meals. There were several benefits identified during the interviews. Partners felt that the reputation of SHN alone had been a benefit to their agency by strengthening their relationship with communities served. Partners also indicated that a benefit to their agency was the constant and reliable communication and the passion displayed by SHN staff. Partners knew Nina and Matt by name and felt that they could always count on them. Some partners discussed additional benefits beyond receiving meals, which included receiving medical supplies, vitamins, and water filters. Partners did express some areas of growth for the partnership. Of the lead partners interviewed, 71% said they never received training from SHN and expressed a need for training on the specific expectations of the partnership. This is significant in light of stakeholders concerns about partners capacity to conduct M&E. Perhaps if clearer expectations were communicated to partners, they would be able to conduct the M&E tasks desired by SHN. Partners were interested in receiving training on food nutrition, ways they can improve their services, and how to do M&E in general. Many partners also expressed that they would like more M&E resources in order to learn how the meals affect the people and communities they serve. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 27
It is also important to note that all partners prepared meals using multiple bags and the majority were using several boxes of SHN bags at each meal preparation. Because of this, 20% requested larger meal packages as these would be beneficial to the facilities meal preparation. 5. Increased Internal Resources in order to Collaborate with Partners on Monitoring and Evaluation It is evident in the interviews and the survey results that lead partners are looking to Stop Hunger Now for assistance with developing their monitoring and evaluation capacity. Stop Hunger Now has already significantly impacted their work by providing food and other donations, but could further expand their impact if they began to focus on empowering their partners and building their capacity to do M&E. In order to do so, a significant internal investment in M&E will be necessary. The only question on the survey that elicited equal amounts of negative and positive responses was the likelihood that Stop Hunger Now would use this survey to make changes to their work. This finding is also supported by the responses of internal stakeholders, some of which had very little knowledge of current M&E efforts or of Stop Hunger Nows partners. Because of the diversity of Stop Hunger Now partners, Stop Hunger Now will not be able to create a single method of data collection that works across the board and will need dedicated staff resources in order to analyze the data that does exist, publishing findings, and make organizational changes based on the findings. There is a strong case to be made for the value of empowering partners to do strategic planning, monitoring and evaluation in order to reach their goals of self- sustainability and reducing poverty in the areas they work, but if Stop Hunger Now is not doing these things internally, it will be challenging to train partners to engage meaningfully in M&E efforts.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 28
RECOMMENDATIONS 1. Recommendation: Engage in a Strategic Planning Process Developing Concrete Goals and Priorities around Impact In order to monitor and evaluate the work being done by Stop Hunger Now, concrete goals and priorities must be in place. We have identified four key steps in this process. A. Clarify Key Definitions Related to Mission The most important definition to address is the definition of hunger. There are many ways of framing hunger and SHNs framing will greatly impact how they go about their work. B. Integrate Best Practices from Global International Development and Hunger Relief Work Stop Hunger Now work must be in line with the most recent data and information available on international development in order for them to continue to be competitive as a nonprofit worth investing in and, more importantly, in order for them to move toward their mission of ending hunger. For example, a valuable reference SHN could use is the WFP Strategic Plan (2014-2017) in order to determine strategic objectives, goals associated with each objective and main tools used to accomplish goals (such as school feeding, advocacy, maternal and child health programs, community based management of acute malnutrition etc.). C. Unpack, Define and Operationalize Transformational Development Transformational development is a term created by Stop Hunger Now to describe the kind of work their partners are doing that goes beyond passing out meals. An important thing to consider is whether all of a certain type of organization (such as schools) are automatically labeled transformational development, or whether the phrase goes beyond the type of facility to the work that they are doing that creates long-term sustainability. D. Define and Clarify Partner Priorities Once organizational priorities have been developed, a more systematic process for partner alignment with priorities can be created and implemented as well. Some key questions to consider: Will SHN focus on the first 1000 days of a childs life? To what extent is school feeding a priority? Will SHN focus on areas of most severe food insecurity and/or individuals who have highest rates of malnourishment? To what extent will meal distribution be at the discretion of the lead partner? What will be the policies in place for when partners fail to match organizational priorities?
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 29
2. Recommendation: Determine the Role and Priorities of SHN Monitoring and Evaluation Monitoring and Evaluation efforts should be driven by a detailed, measurable plan that is embedded in the larger organizational strategic plan of Stop Hunger Now. This plan should outline the specific goals, objectives, and activities for monitoring and evaluation over the next 1-3 years. Key considerations for the M&E plan should include: internal unity and buy-in, adequate resources, alignment with current U.S. and international work, and clear communication with all staff and board members. Internal stakeholders cited the lack of investment in M&E efforts (i.e. lack of funding) as a major issue with current M&E practices. The majority also discussed how crucial M&E is to their role and to the overall organization. The M&E plan should span all aspects of the organization, including domestic and international programs and activities. As such, it may be most ideal for M&E to exist outside of any one department within the organization. International M&E Lead partners are key to understanding impact. Stop Hunger Now should invest M&E resources to better understand the impact of meals through the lead 26 partner organizations. Focusing impact on the lead partner level would allow SHN to focus its M&E efforts in a way that is feasible and respects the wishes of lead partners. Lead partners requested that they would most like SHN to help them monitor in ways that are useful and provide the support needed to collect data. Critically, lead partners (92% of those surveyed) have stated that they would prefer SHN to work directly with them (versus with on the ground partners) to build M&E efforts in order to avoid confusion and to collect data together. Furthermore, SHN partners were willing and eager to provide necessary information and to collaborate with SHN in monitoring and evaluation efforts. The 85% survey response rate is significantly higher than a average online response rate (33%), and speaks to the quality of relationship already in existence between SHN and their partners. Strengthening lead partner relationships does not preclude the opportunity for SHN to collect and share meaningful stories of individuals that partners serve. Partners communicated a strong willingness to have SHN engage with on the ground partners but requested that M&E efforts be forged in collaboration with them. Domestic M&E In order to understand impact on volunteers, Stop Hunger Now needs to develop measures and monitoring processes for tracking volunteers and organizations engagement in fighting hunger after their connection to Stop Hunger Now. If one of the goals of the organization is to create a movement then incorporating evaluation of the domestic program into the wider M&E efforts is critical.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 30
Using Existing Data and Resources Stop Hunger Now currently has a significant amount of data that are yet to be analyzed. In the partner survey, partners were least confident about the likelihood of SHN using the survey information to improve their operations and organization (only 37% thought they were likely or very likely to use the survey data. Internal data that require further analysis include data from the impact reports, data from the Guatemala/El Salvador and Belize SHN trips, and compilation and dissemination of data from the various reports that have been created by UNC, NCSU, and SHN staff. It is not enough for data to be compiled into a variety of reports, there needs to be a system in place for this information to impact the operations and organization of SHN in order to meet the standards outlined by Charity Navigator in Appendix A as well as address the needs and expectations for M&E expressed in the key stakeholder interviews. Externally, the vast majority of lead partners have indicated a willingness to share their data and the high response rates on the survey and interviews indicate that partners are accessible and looking to SHN to inform their own organizational M&E practices. 3. Recommendation: Building Collaborative Efforts for M&E External M&E partners allow for increased resources, expertise, and stronger credibility with M&E efforts to support SHNs efforts. For example, the partnership with NCSU has allowed SHN to have the equivalent of 2 years of full time work in seven months and two country-level reports completed in a period of four months time. By continuing collaborative efforts with universities, such as NCSU, SHN has an increased capacity to deepen partner relationships as more individuals are working to interview partners, take M&E trips, write reports that can be sent to partners, and discover and highlight the incredible work that partners and individual facilities are doing around the world. University partnerships also create opportunities to bring in the most recent and innovative research on international development and hunger. Additionally, external evaluation provides a critical outsider perspective and is an industry standard (Hageboeck, 2009). Considering the significant workload required for M&E, university partnerships can be key partners in expanding SHN capacity in strategic planning and M&E. Beyond building M&E, partnering with graduate-level students gives SHN the opportunity to have a significant impact on volunteers that can live long beyond the initial volunteer experience. For the NCSU Collaborative team, after a year of in-depth partnership with SHN, there will be more opportunities to apply our knowledge and training, build cross-cultural relationships, and build a future movement to alleviate hunger. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 31
REFERENCES Connors, K. & Prelip, M. (2000). Principle 3: The partnership builds upon identified strengths and assets, but also addresses areas that need improvement. Partnership Perspectives, 1(2), 19-26. Dutta, M.J., Anaele, A., & Jones, C. (2013). Voices of hunger: Addressing health disparities through the culture-centered approach. Journal of Communication, 63, 159-180. Engel, R.J., & Schutt, R.K. (2012). The practice of research in social work (3 nd ed.). Thousand Oaks, CA: Sage. Food and Agriculture Organization. (2013a). Basic definitions. In Hunger portal. Retrieved from http://www.fao.org/hunger/en/ Food and Agriculture Organization. (2013b). FAO strategy for partnerships with civil society organizations. Retrieved from http://www.fao.org/docrep/meeting/027/mf999e.pdf Food and Agriculture Organization. (2013c). The state of food insecurity in the world. Retrieved from http://www.fao.org/hunger/en/ Hageboeck, Molly (2009). Trends in international development evaluation theory, policy, and practices. United States Agency for International Development. Retrieved from: http://www.alnap.org/resource/8441.aspx Harvey, W.S. (2013). Strategies for conducting elite interviews. Qualitative Research, 11(4), 431-441. Heady, H.R. (2000). Principle 9: Partnerships take time to develop and evolve over time. Partnership Perspectives, 1(2), 71-78. Kirkpatrick, S.I., McIntyre, L., & Potestto, M. (2010). Child hunger and long-term adverse consequences for health. Archives of Pediatrics and Adolescent Medicine, 164(8), 754- 762. Myers, N. & Kent, J. (2001). Food and hunger in Sub-Saharan Africa. The Environmentalists, 21, 41-69. OBoyle, E. & OBoyle, M. (2012). Global poverty, hunger, death, and disease. International Journal of Social Economics, 39(1/2), 4-17. Rein, M. & Stott, L. (2009). Working together: Critical perspectives on six cross-sector partnerships in Southern Africa. Journal of Business Ethics, 90, 79-98. Smith, L.C. (1999). Can FAOs measure of chronic undernourishment be strengthened? Food Policy, 23(5), 425-445. Stop Hunger Now. (2013). Mission and History. Retrieved from http://www.stophungernow.org/mission-history. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 32
Appendix A Charity Navigator - 5 Core Elements Element One: Alignment of Mission, Solicitations and Resources We determine if the charitys funding solicitation materials (on a donations page, public financial reports, or for the first year the charity is reviewed only - materials sent to CN) are in accordance with how it allocates its resources as per the program expenses break out on the most recent Form 990. All subsequent rating elements are evaluated for each individual program area until at least 60% of overall program expenses are represented. Element Two: Results Logic and Measures o Is the organizations causal logic plausible? Is the organizations statement on how their activities lead to pre-defined outputs and outcomes likely under normal circumstances? (Outputs are the immediate results of the activities for example, how many people were fed or sheltered, or how many roads built. Outcomes are the medium- and long term changes. They might be improvements in nutritional status or health indicators.) o Is there an indication of how much of the action is required to produce the pre-defined outputs and outcomes? Has the charity given thought to the required amount and time that is needed to produce the pre- defined outputs and outcomes? o Is the causal logic based on reasonable evidence? Does the report include reference to evidence that these actions have in fact been demonstrated to produce the intended results? References could be evidence from other organizations or from past activity by the charity itself. o Are there specified measures (indicators) to be collected and a plan to do so? Does the charity state what success looks like and how it will collect the data to know if it has been achieved? Element Three: Validators For the purposes of Results Reporting assessment, we recognize the work of other organizations which, through their processes, standards or analysis, examine or assess the charities we evaluate. Charity Navigator realizes that it is not the only organization that evaluates charities; and that the evaluation processes of many other organizations have merit. Our intent therefore is to give charities credit for making public on their own websites or other materials the results of evaluations or analyses from these other validator organizations. o Charities indicate which organizations they are members in good standing of, or show evidence of their participation in the organization on their website, or in materials emailed to Charity Navigator. Element Four: Constituent Voice This element assesses whether and how well a charity collects and publishes feedback from it primary constituents. (Primary constituents are the people who are meant to be the direct recipients of benefits created by the organizations actions.) SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 33
o Does the charity publish feedback data from its primary constituents? Is there a rigorous method of data collection using the same questions and capturing the answers consistently? o Does this data indicate if it is representative of all primary constituents? The feedback data must in some way indicate why the data set could be seen to be representative of the whole group. o Does this data include an explanation of why the organization believes the feedback is frank and honest? Is there an explanation in the feedback data that explains why the feedback is frank and honest, for example, is it collected anonymously, and/or using multiple methods etc. o Is this data presented in a way that shows changes over time going back at least one year? Does the data show how answers to the same questions to the same constituent group change over time? o Does this data include questions that speak to the organizations effectiveness? Do any of the questions address how well the organizations activities meet the need of the respondents? Do any questions ask if the organizations activities lead to improvements or positive changes in their lives? o Does the organization report back to its primary constituents what it heard from them? Have the findings from the data collection process been reported back to those who provided the data and others like them? Element Five: Published Evaluation Reports This element assesses whether a charity is publishing regular evaluation reports that cover the results of a majority of their programs. o Does the charity publish evaluation reports that cover the results of its programs at least every five years? o Are those reports based on recognized techniques to understand their results? o Does the charity explain what, if anything, it is changing as a result of the findings in the evaluation report?
Appendix B Gantt Chart & Detailed Timeline SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 34
Starting Month Goal Activity
Completion Date Aug-13 Research background on topic Advanced standing Social Work students will begin researching background on definition of hunger, Stop Hunger Now, and partnership success. Nov-13 Nov-13 Submit IRB proposal for evaluation Principal Investigator will submit IRB proposal. Nov-13 Jan-14 Advance Standing Social Work graduate students will begin interviewing. Thirteen randomly selected partners will be interviewed by thirteen advance standing social work graduate students. Interviews will be approximately minutes to complete. Feb-14 Stakeholder interviewees selected and contacted Stake holders were identified for the interviews and contacted to gain consent to interview them Interview Guide Developed The interview guide was developed to standardize the questions for all interviewees. Feb-14 Interviews conducted Stake holders were identified for the interviews and contacted to gain consent to interview them Mar-14 Interviews transcribed Graduate students will transcribe interviews the interviews they conducted with partners Mar-14 Data analysis Data cleaning and analysis will be completed. Qualitative data will be coded from interviews and Impact Report. Apr-14 Apr-14 Completion of Evaluation The evaluation will be completed and submitted for review. Feedback will be provided when appropriate. Apr-14
8/27/2013 10/16/2013 12/5/2013 1/24/2014 3/15/2014 5/4/2014 Background Research IRB Proposal Create Interview Questionnaires Conduct Interviews Transcribe Interviews Code & Analyze Data Full Evaluation Draft Finalize Full Evaluation Present Report SHN SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 35
Appendix C Stop Hunger Now Partner Survey Q37 Stop Hunger Now Partner Survey Stop Hunger Now is working to understand how to continue to build strong partnerships to end hunger in our lifetime. Your input is valuable to us. We are asking our lead partners to complete this short anonymous survey to provide us critical feedback. By completing this survey by Friday, March 14 at 5 pm, you will be entered in a drawing for a $50 visa gift card! You may include your name in the last question or email kasmit13@ncsu.edu to notify us that you have taken the survey and chose to remain anonymous. Q1 How many countries do you serve? Q2 Which countries do you serve? (If more than 3 countries, please list top 3 countries served and additional regions where you work.) Q3 How long have you been a Stop Hunger Now partner? Q4 Stop Hunger Now meals constitute approximately what percent of the food used in your programs? Less than 20% (1) 20%-40% (2) 41%-60% (3) 61%-80% (4) 81%-100% (5) This varies too much to average (6) Q5 Age Range of the people you feed (percentages should add up to 100%): ______ 0-3 yrs old (1) ______ 4-10 yrs old (2) ______ 11-20 yrs old (3) ______ 21-50 yrs old (4) ______ 51+ years old (5) Q6 What type of support have you received from Stop Hunger Now in the past? (Check all that apply) Food/Meals (1) Water Filters (2) Medical Supplies (3) Financial Support (4) Other In-Kind donations (5) Other (please specify) (6) ____________________ SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 36
Q7 What is your relationship with the organizations on-the-ground responsible for meal distribution/preparation? My organization is the one on the ground running programs (1) We run them directly (2) We run them indirectly (3) We fund all of their work (4) We fund some of their work (5) We provide in kind aid only (6) Other (please specify) (7) ____________________ Q8 Please rank the priority areas of focus for your organization where 1 is the highest priority. ______ Hunger/Food Security (1) ______ Refugees and Protection (2) ______ Disaster Resilience and Response (3) ______ Health (4) ______ Education (5) ______ Child development (Non-school based programs for children and youth) (6) ______ Spiritual Development (7) ______ Job Training (8) ______ Other (please specify) (9)
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 37
Q9 Thinking about the beginning of your partnership with Stop Hunger Now - please select the extent to which you agree or disagree with the following statements.
Strongly Agree (1) Agree (2) Neither Agree nor Disagree (3) Disagree (4) Strongly Disagree (5) The time that passed from starting discussions to receiving support from SHN was reasonable. (1)
The amount of food from SHN was well matched to our needs. (2)
SHN asked for more information during the agreement process than other NGOs/funders. (3)
During the agreement process, we did not feel pressured by SHN to change our priorities. (4)
Q10 Do you currently collect data/require reporting from your partners or programs to monitor your work? SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 38
Yes (1) No (2) Answer If Do you currently collect data/require reporting from your partners to monitor your work? Yes Is Selected Q11 What types of data do you collect from your partners or programs? Answer If Do you currently collect data/require reporting from your partners to monitor your work? Yes Is Selected Q12 Do you use standard reporting forms? (By "standard" we mean using the same form consistently to collect information) Yes (1) No (2) Answer If Do you currently collect data/require reporting from your partners to monitor your work? Yes Is Selected Q13 How often do your on-the-ground partners or staff provide reports to your organization? Bi monthly (1) Monthly (2) Every 6 months (3) Yearly (4) Anytime/no set schedule (5) Other (6) ____________________ Q14 Please select two items that you would MOST like SHN to do to improve monitoring and reporting in the future Accept reports in different formats (1) Provide more training and assistance for us to monitor and report on our work (2) Respond and discuss our reports with us (3) Focus more attention on long term changes (4) Allow more time to complete reports (5) Help us monitor in ways that are useful for us and the people we work with (6) Draw more on our expertise in developing ways to monitor progress (7) Undertake more monitoring with us by providing staff and support to collect needed data (8) Involve us in deciding how to monitor and report (9) Simplify monitoring and reporting (10) Visit us more often (11) Other (12) ____________________ SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 39
Q15 If you are an organization that has multiple on-the-ground partners: Would you prefer that Stop Hunger Now correspond directly with on the ground partners for data collection and monitoring? This question is not applicable to my organization (1) Yes, and this is why: (2) ____________________ No, and this is why: (3) ____________________ Q16 What are the main uses of technology in your organization? ("We mainly use technology for _________") Data sharing (1) Email (2) Mandatory reporting (3) Social-Media, networking etc. (4) Fundraising and Gaining Donor Support (5) Sharing pictures or video (6) Pay-pal sharing of funds (7) Other (8) ____________________ Q17 How stable is your on-the-ground partner's or staff's Internet connection? They always have access to Internet (1) Somewhat stable (Access to internet at least half of the time). (2) Not very stable-goes out frequently (3) This varies from partner to partner (Please elaborate in text box) (4) Our on the ground partners have no Internet connection (5) Must go off-site for internet access. If so, where (i.e. internet cafe) (6) ____________________ Unknown (7) Q18 How comfortable are your partners or on-the-ground staff in using the Internet? Very comfortable (1) Somewhat comfortable (2) Not comfortable (3) Unknown (4) Other (5) ____________________ Q19 How frequently do you communicate with your on-the-ground partners by email? Daily (1) 2-3 Times a Week (2) Once a Week (3) 2-3 Times a Month (4) SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 40
Once a Month (5) Several Times a Year (6) Never (7) Other (8) ____________________ Q20 How often do your on-the-ground partners check email on average? A few times a day (1) Daily (2) 2-3 Times a Week (3) Once a Week (4) 2-3 Times a Month (5) Once a Month (6) Varies significantly from partner to partner (please elaborate if possible) (7) ____________________ Unknown (8) Q21 How frequently do you communicate with your on-the-ground partners by cellular/mobile phone? Daily (1) 2-3 Times a Week (2) Once a Week (3) 2-3 Times a Month (4) Once a Month (5) Several Times a Year (6) Never (7) Other (8) ____________________ Q22 What is the main cellular/mobile provider used by staff and partners in each country? You can list up to 5 countries. (For example Haiti - Voila) Country 1, Provider (1) Country 2, Provider (2) Country 3, Provider (3) Country 4, Provider (4) Country 5, Provider (5) Q23 Do you incur additional costs when sending files/video via cellular phone that would inhibit your ability to do so? Yes (1) Maybe (please elaborate) (2) ____________________ SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 41
No (3) Not applicable (4) Q24 How many of your on-the-ground partners or staff have the computers and software they need to adequately do their work? All (1) Most (2) Some (3) None (4) Q25 Does your agency have other technology challenges? Please list challenges. Q26 Please select the extent to which you agree or disagree with the following statements.
Strongly agree (1) Agree (2) Neither Agree nor Disagree (3) Disagree (4) Strongly Disagree (5) Don't Know (6) Our organizational values and SHN's organizational values are well aligned. (1)
We can comfortably discuss our SHN partnership in the communities where we serve. (2)
SHN understands the communities we work in. (3)
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 42
SHN has made a major contribution to the sector(s) that we work in. (4)
SHN learns from its mistakes and makes improvements to how it works. (5)
Food shipments and Support (including funding) are sent when SHN says they will be. (6)
SHN understands our working environment and cultural context. (7)
SHN has explained what it expects from us to maintain the partnership. (8)
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 43
We understand SHN's plans and strategies. (9)
Q27 If you had to choose one label to define Stop Hunger Now, what would it be? (There is no one right answer) A crisis response organization (1) A food donor organization (2) An international development organization (3) A faith based relief and development organization (4) A church based relief and development organization (5) Other definition: (6) ____________________ I don't know/cannot define (7) Q28 How likely do you think it is that SHN will make changes to their work as a result of your input in this survey? Very Unlikely (1) Unlikely (2) Somewhat Likely (3) Likely (4) Very Likely (5) Other: (6) ____________________ Q29 How do you rate the quality of the partnership with SHN? 1 (poor) (1) 2 (2) 3 (3) 4 (4) 5 (excellent) (5) Q30 During the last year, did requests for food assistance increase, decrease or stay the same? Increase (1) Decrease (2) Stay the same (3) Q31 Do you expect the demand for food assistance to increase, decrease, or stay the same during 2014? SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 44
Increase (1) Decrease (2) Stay the same (3) Q32 What do you think are the main causes of hunger in the communities/countries you serve? Q33 Is there anything else that you would like to say? This can be feedback, suggestions, questions, additional information about your work or anything else you would like to share with us. Q34 Almost done! We greatly appreciate your participation in this survey. We would like to provide you with the opportunity to let us know who you are and what organization you represent, as this information is very helpful to us in understanding the results of this survey and what it means for our organization. HOWEVER, if you prefer for this to remain anonymous please leave the box below BLANK. Otherwise, please include your name. If you choose to remain anonymous but would still like to be entered in the drawing for a $50 gift card, please email kasmit13@ncsu.edu to notify us that you have completed the survey. Q35 This is the end! Once you submit this question, your survey responses will be recorded and you will not be able to change any answers. If you would like to make any edits feel free to go back and review any questions. We greatly appreciate your time and effort in completing this survey.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 45
Appendix D SHN Impact Report
In order to improve our programs and provide information to our donors, we require meal distribution partners to provide feedback on the effect of the donated meals on their programs. Thank you for your valued partnership! General Information Organization Name Program Country Name of person completing report E-mail address Date(s) Cargo Arrived
Please include photographs of the donated cargo being used or distributed when returning this report Distribution Information Please list the institutions/projects that received the cargo, the number of people served, the amount of the cargo used in each institution (boxes or percent of total) and the project category or type: school, orphanage, general distribution/feeding center, child development program, vocational training, medical program, elder care, crche/daycare, or shelter (attach additional sheet if necessary): Institution/Project Name City or Town Number of People Served Project Category/Type Amount of Cargo Distributed
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 46
Please describe how the meals were distributed to the final recipients. Include: how they were transported, how recipients were chosen, whether meals were served or sent home
If meals were sent home, how were recipients trained to prepare meals? If meals cooked and served, what training do those serving receive?
How did the institutions listed above prepare the meals? Are ingredients separated and cooked independently? Is the entire bag cooked at once? What is added to the meals?
How are the flavoring/vitamin packets used? Are all of the packets used? Are they given to everyone or only certain people? Are they used each time the rest of the ingredients are prepared?
Were any of the meals found to be sold, lost, damaged or otherwise used in ways not intended?
What other food items were used in these programs? Were they purchased or donated?
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 47
Were the meal boxes reused in your programs? If yes, how were they used?
Results of Feeding Program What indicators does your organization use to determine your programs progress?
Please describe any changes that you have seen in the following indicators as a result of the feeding program(s) Program Indicators Changes Measurement Period Average Weight Gain Average Height Increase Graduation Rates Enrollment Increases Overall Standard of Living
Please list any other changes or results of your program(s) related to the Stop Hunger Now meals
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 48
Do you have other needs that affect the use of the food (for example: cookware, water, heating fuel, distribution staff etc.) or the progress of your program in general?
Pictures, Testimonials and Success Stories So that we can personally thank our donors, please share two brief personal stories or testimonies of how this cargo benefited or changed the lives of people in your programs. 1.
2.
Please let us know what your experience has been with the taste and acceptability of the meals, including suggestions for changes:
Thank you for completing this report. Please either e-mail completed report to Nina Batista at nbatista@stophungernow.org, fax to 919-839-8971 or mail to Stop Hunger Now.
Signature: ______ Date:
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 49
Appendix E SHN Impact Report Summary Total 2012 Reports: 45 Partners measuring the indicators on impact forms (potential target organizations for piloting M&E initiative): Country School Academic Indicators Haiti St. Andrews School Grades, graduation rates, national test scores, Haiti Youth Build Haiti Enrollment, attendance, graduation rates Haiti Free The Kids Haiti Graduation Rates and Test Scores Liberia Reach Now Academic Performance/Attendance records Zambia Reach Now Attendance/test scores/nat. exams Kenya Reach Now Attendance Honduras Salesian Missions Graduation Rates, Enrollment Rates
Country Partner Health Indicators Kenya Mission of Mercy Height and weight data is collected quarterly and numbers are given that show increases Turkana Cluster, Kenya Mission of Mercy Measurable data regarding height and weight increase: 2kg and 10 cm a month on average South Sudan Mission of Mercy Quarterly weight and height gain average: 1 kg and 7 to 12 cm, measured at beginning and end of quarter. Philippines RASFI Tracking the overall health of the children; weighing; visits; interview (did not see these stats in report). SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 50
Haiti Free the Kids Haiti Measure height/weight gain, graduation rates/test scores, but this info is not provided in impact report Haiti Haiti Outreach Ministries Measuring weight: average gain 12.4 lbs, high capacity for data collection, teachers and students testify to educational value of SHN meals Haiti Robert Ford Orphanage Indicators: typical child gains 2 to 4lbs a yr, 1/2 in, 40 children a yr move to secondary school Belize Kidz Konnect 4 Jesus Medical team that comes every 90 days for assessment, indicate health and behavior improvements, no data provided. Nicaragua ORPHANetwork Measure BMI instead of height/weight, keep track of number of kids coming to feeding programs, saw malnutrition decrease from 26% to 19% Honduras Salesian Missions Average Weight Gain 3-4 lbs. each month (30 days). Average Height gain: 5-6 cm each 6 months. Cambodia Reach Now Average Weight Gain 3-4 lbs. each month (30 days)
Positive Program Impact Innovative Uses of Meals: Organization Idea ORPHANetwork, Nicaragua Separate out soy and fry into cakes SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 51
Use of Ziplock Bags Organization Location How they were used Salesian Missions Honduras Recycled and sold by children Youthbuild Haiti Haiti Children take bags home for personal use.
RASFI, Philippines Mix rice with coconut milk/milk for porridge Add chicken or egg Sweeten with chocolate, milk or coconut milk Salesian Missions, Sierra Leone Making cookies and muffins with soy.
The rice, carrots and in some cases chives, or something resembling chives, are separated from the rest of the ingredients and cooked separately. This combination of rice can be served in many ways and combined with many other ingredients, such as fish or chicken in gravy. Sometimes only fish and seasoning are added.
The rest of the ingredients are sometimes ground up by hand, and then cooked with fish, and sliced potatoes or sliced cassava. Salesian Missions, Peru Separate soy and cook it in soups Tarahumara Ministries, Mexico Add additional vegetables, beans and water to make soup. SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 52
Use of Boxes Organization Location How they were used Salesian Missions Sierra Leone Sold in the market. Some families use as hung ceilings suspended from the roof of the house to reduce heat or laid on the floor to reduce cold of the floor for sleeping. ORPHANetwork Nicaragua Roofing and walls, props for skits for youth Reach Now Cambodia Cambodia Used for storage and extras are burned for fuel. Reach Now Liberia Liberia Sent home with students and staff. Robert Ford Orphanage Haiti Lining inner walls of local homes. Hearts and Hands for Haiti Haiti Recycling paper and rice straw, making briquettes for cooking Many partners Various Storage-in schools, given to community members etc.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 53
Partners reporting non-traditional outreach or programs Partner/Location Program/Services Salesian Missions, S. Sudan. Vocational training center/leper colony/street boys center Salesian Missions, Honduras Alcoholics Anonymous Mission of Mercy, Kenya Home visits by Social Workers Reach Now, Kenya A committee was formed to select the most needy beneficiaries for the food Reach Now, Zambia Personal relationships with partners to determine meal distribution to especially vulnerable children Reach Now, Cambodia Engaging widows in the community in caring for orphans
Agricultural Program Free the Kids, Haiti Agricultural Program Youthbuild Haiti Youth Training Center Case Studies/Community Development Initiatives Mission of Mercy, Malindi, Kenya Parent meetings and involving parents in opportunities for project involvement. Parents bringing firewood, help cooking on program days, etc. Salesian Missions, Sierra Leone The Mama Margaret program is intended for children between the ages of 4 and 8, 80% of them belong to families in which only the mother is present, some of them work in the market selling anything to survive, the children do not go to school, and sometimes work all day among adults. Through the program we approach the mothers and offer them a plate of free food for boys and girls, and in exchange we ask them to allow us to have 3 hours a day of informal class with the children. We take them to the school where a group of young volunteers attends to them, they have an hour of study, half an hour of rest (the children sleep for 30 minutes), then educational games and finally, lunch. St. Andres, Haiti National examinations indicate that the St. Andres students do exceptionally well. One of our 9th grade students was recognized by the President of Haiti this past summer. Our graduates are also doi ng very well and will be agents of transformation in Haiti in the future. *St. Andres mentioned they were at maximum capacity-would increase meals increase their capacity? SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 54
Youthbuild Haiti, Haiti In all our Training Centers we have a group of 150 Youths between 14 to 24 Years of age in trainin g for different fields like Carpentry, Plumbing, Electricity, Tiling, Masonry, Electronic Tech. etc. Each group stays in the program for a period of six months. Quote about SHN meals: This was a great help to our feeding program, which helps keep the students in school. The meal was sent by ground transportation and was served at the Training Centers. Many (students) after the six months find a job, some decide to push their knowledge further and return to regular school, and some start a micro enterprise. The program works and the meal is really an important incentive to keep the youth coming back to the training program.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 55
Partner Issues/Complaints Issue Partner Location Soy Salesian Missions Peru Orphan network Nicaragua Taste of Food Mission of Mercy Kenya Reach Now Liberia RASFI Philippines IGLEMEN Nicaragua ORPHANetwork Nicaragua Salesian Missions Peru Putting Vitamin Packet in at the end Free the Kids Haiti Salesian Mission Peru Tarahumara Ministries Mexico RASFI Philippines St Andrews Episcopal School Haiti Needing Cooking Fuel Salesian Missions Honduras Kids Konnect 4 Jesus Belize Youthbuild Haiti Haiti Robert Ford Orphanage Haiti Reach Now Liberia Storage Issues IGLEMEN Nicaruagua ORPHANetwork Nicaragua Not Putting in Vitamin Packets Christian Alliance for Children Zambia Salesian Missions Peru (one packet per two bags)
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 56
Overall Recommendations 1. Focus on Partners who have an active and innovative transformational feeding model-contact them to see about expansion. Make strategic investments with partners doing the most transformational development work. 2. Consider Monitoring and Expanding Capacity Building Grants Small grants ($500-10,000) that partners can apply for in order to carry out community development initiatives (i.e. local farming, vocational training, micro enterprise.) Teams doing meal-packaging events could fund these grants. 3. Using Impact Data to Grow SHNs Outreach Consider using data from impact report to share a partner spotlight at meal packaging events; focus on highlighting partners instead of just individual students for marketing and testimonials. 4. Develop a plan to get impact data from Convoy of Hope and Feed the Hungry. 5. Help Partners Shift from Impact Report forms to Annual Report Form. Create an incentive system and/or a rating system for partners regarding their data collection and reporting. For example, a reward for partners who turn in their impact reports for the deadline, who successfully implement CHAP forms etc. 6. Conduct an online focus group with leaders of various partner organizations to discuss M&E priorities, capacity, etc. from their perspective (engage them and have them be leaders in their own M&E processes). 7. Provide partners with data collection mechanisms as is necessary (i.e. scales, height charts, MU AC bands, and/or forms for data collection). Example: Adding a MU AC band to each box of Stop Hunger Now Meals sent out in 2014 if there is evidence that this is the best way for partners to collect nutritional information.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 57
Appendix F Partnership Success Interview Guide Information to be gathered from leader(s) within partner organization who oversees all facilities. GENERAL INFORMATION INTERVIEWER INSTRUCTIONS Familiarize yourself with these questions and your partners organization prior to the interview. For data accuracy, ask the questions exactly as written and ask for more specific information depending on the interviewees answers.
Beginning Script (please read before beginning phone interview):
Thank you for participating in this interview today. The purpose of this interview is to gain a better understanding of how SHN food distribution is being implemented and to find out what has been working well, and what may need improvement in the program. Your organization has been identified as a strong SHN partner and we would like to ask you questions related to your partnership as well as your experience supporting your local on-the-ground partners (Unless this IS an on-the-ground partner).
This interview is entirely voluntary and will last between 30-60 minutes. We will share the findings from this interview in a summary form with SHN. When we do, we will not use individual organizational names and we will make every effort to ensure that individual comments cannot be attributed to any single person. If you are willing to participate, we would also like to audiotape this interview. Please be as honest as possible as your feedback is very helpful. This in no way impacts the meals/shipments your organization receives. [If you are given verbal permission to proceed, you can begin the interview.] Interview Completed by:
Name :
Title/Role:
Location:
Length of partnership with SHN:
Type of organization: (E.g. school, orphan, etc.)
Appendix I SHN Logic Model SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 58
1. TO BEGIN THE INTERVIEW WE WILL FOCUS ON HOW THE PARTNERSHIPS TRACK FOOD SHIPMENTS (in your country of destination).
a. How many organizations or facilities do you provide food for? How do you track food shipments?
b. Does your agency know in advance when you will receive food from SHN? (Example: what is the window or timeframe)
c. What is your biggest barrier to being able to distribute food?
E. THESE NEXT FEW QUESTIONS FOCUS ON HOW PARTNERS DESCRIBE THE IMPACT OF SHN MEALS.
a. How has receiving SHN meals benefited your agency?
b. How has receiving SHN meals benefited the individuals you serve and/or the individuals your on-the-ground partners serve?
c. What indicators does your organization use to determine your programs success? (Example: do you track/keep records of the impact of meals through stories or through other measurements such as the height/weight of children?)
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 59 F. THE FOLLOWING QUESTIONS ASK ABOUT WHAT IS HAPPENING TO FOOD AND FOOD PACKAGES IN THE COUNTRY OF DESTINATION.
a. Do you know what the in-country staff does with any of the boxes/bags after using the food? For example, do any re-use the boxes or bags? (example: do they know that the bag is re-sealable)
b. Do you know if in-country staff cook or serve meals using single bags or do they cook large meals using multiple bags? If so, which method do they primarily use?
G. ONE OF THE REASONS WE ARE DOING THIS INTERVIEWS IS TO BETTER UNDERSTAND PARTNERSHIP SUCCESS AND HOW TO HELP SHN BE SUCCESSFUL.
a. Do you have other partners or donors that provide food? If so, what is one thing that your other partners are doing that would be beneficial for Stop Hunger Now to do?
b. What is one thing that SHN is doing within their partnership that you would find beneficial if all other partners were doing?
c. What training or information have you been provided regarding the SHN guidelines for partnerships? What other training / info would you like to have?
d. How could SHN strengthen the partner relationship with your agency? What are some of the things that would be helpful?
e. Are you happy with the amount of communication you have from Stop Hunger Now? Would you like more or less communication?
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 60 f. What is the main, long-term goal of your work? If answer is spiritual in nature, drill down to get a more tangible answer
g. Does the support you receive from Stop Hunger Now create (or help you create) long-term change in the community(ies) you serve, or short-term change? If long-term change, how? If not, what could SHN do to create long-term change?
h. Is there a way SHN could be more efficient in supporting your work and long-term change? If so, what way(s)?
i. What, if any, benefits does your organization see from using Stop Hunger Now meals versus buying all food locally, in-country?
j. How would you rate the quality of Stop Hunger Now meals on a scale from 1-5, where 5 is the highest quality and 1 is the lowest? Please explain why you chose that rating. What can SHN do to improve the quality? Circle Rating: 1 2 3 4 5
Explanation/Suggestion(s):
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 61 Appendix G Stakeholder: Date: Interviewer:
SHN Staff Stakeholder Interviews As part of our monitoring & evaluation work this year, we are talking to a number of stakeholders, including executive staff and board members about their vision of monitoring and evaluation work within Stop Hunger Now. This will inform how we move forward in developing a monitoring & evaluation procedure. The interview should take 30- 60 minutes.
1) In thinking about our mission, to end hunger in our lifetime we are trying to understand the definitions of hunger we are working with (show the list on separate page)How do you view our organizations definition of hunger? Can choose one of the sample definitions or define in own words
2) How would you define Monitoring and Evaluation (M&E) work broadly?
3) What is your perception of how Monitoring and Evaluation works within Stop Hunger Now at this point in time?
4) How does M&E work impact you and your work/role at Stop Hunger Now?
5) What issues/problems/pain points do you currently have regarding distribution of aid, monitoring and evaluation and demonstration of SHNs impact? a) Are you getting everything you need currently? If not, what do you need?
6) From your knowledge of the Strategic Impact project (SIP), do you think were headed in the right direction to give you the tools and information you need to be successful in your position? Please explain why or why not.
7) In your opinion, what types of partners should Stop Hunger Now be working with to gather data? Why? (Example: large, small, focused, diverse, on-the-ground, US-based, level of need)
8) Where should Stop Hunger Now be working to gather data? Why? (Through school, orphanage, medical, child/youth development, general feeding, crisis response, locally run)
9) What, in your opinion, can our team do to make monitoring and evaluation a more strategic part of Stop Hunger Now?
10) For your department, what M&E / impact information do you and your team need to be successful in your job duties and goals over the long-term?
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 62 Sample Definitions of Hunger and Related Terms Definition 1. The status of people whose food intake does not include enough calories (energy) to meet minimum physiological needs. 2. A situation that exists when people lack secure access to sufficient amounts of safe and nutritious food for normal growth and development and an active and healthy life. It may be caused by the unavailability of food, insufficient purchasing power, inappropriate distribution or inadequate use of food at the household level. 3. A state, lasting for at least one year, of inability to acquire enough food, defined as a level of food intake insufficient to meet dietary energy requirements. For the purposes of this report, hunger was defined as being synonymous with chronic undernourishment.
SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 63 Appendix H Interviewed Partners Detailed Information
SHN Partner Agency Agency Location SHN Partner Since Number of Facilities Receiving SHN Meals Types of Programs Percent of Total SHN Meals Given Convoy of Hope Haiti, El Salvador, Kenya, Philippines Honduras, Panama 2008 29 Schools; Child Development; General Crisis 15% Missions Touch Ministries (Intermediate partner of Reach Now) Zambia 2007 45 +/- Schools; Prison; Other 2% ORPHANetwork Nicaragua 2009 40 +/- Orphanage 6% Haiti Outreach Ministries Haiti 2007 3 Schools; Churches 5% Children of Vietnam Vietnam 2008 86 School; Medical; eldercare; general; shelter; vocational; Creche 3% Alliance for Children Everywhere Zambia 2009 13 Schools; orphanage, general 1.5% Kidz Konnect 4 Jesus Belize 2012 30 Schools 1.5% Ganta United Methodist Mission Liberia 2012 3 Medical; Schools 0.2% Guatemala Village Partners Guatemala 2010 3 Schools; Orphanage 0.8% Hearts and Hands for Haiti Haiti 2010 17 Schools; Orphanage; Churches; Elder- Care 1.5% Stop Hunger Now Southern Africa (affiliate) South Africa 2011 173 Preschools; General 0.8% St. Andres Episcopal School (Intermediate partner) Haiti 2011 1 School 0.8% Appendix I SHN Logic Model SHN Monitoring and Evaluation Stakeholder Report, 2014 Page 64