Evaluation of the perceived effectiveness and impact of a clean drinking water-

focused non-profit organization

Nick Kelley








ESSM 675
Drs. Steven Whisenant and Vicky Chen
May 07, 2013
Evaluation of clean water non-profit organization 2


Abstract
This paper aims to evaluate the effectiveness and humanitarian impacts of the efforts put
forth by the non-profit organization charity: water. Charity: waters mission is to provide clean
and safe drinking water to people in developing nations through implementation of water
infrastructure projects such as sand filtration systems, groundwater wells, and rainwater
harvesting catchments. Evaluation was performed through use of a survey distributed to charity:
water employees and partners. Survey questions focused on five key areas: Project Selection,
Data Collection, Health Impacts, Capacity Building, and Awareness. Results were quantified and
analyzed to determine the direction of charity: waters efforts. Survey responses indicated that
charity: waters overall efforts are perceived as quite favorable with particular regard to Project
Selection and Health Impacts. Areas requiring improvement include Data Collection, Awareness,
and Capacity Building.

Introduction
Established in 2006, charity: waters mission is to bring clean and safe drinking water to
people in developing nations
4
. They work with local partners in developing communities to
choose water technologies appropriate for that area. Such projects include systems to capture and
store water from natural springs, biosand filters to clean contaminated water, hand-dug and
drilled wells to access groundwater, catchments to harvest rainwater, and rehabilitating broken or
abandoned projects
3
. The local partners work at the village-level where the community elects
members to form a Water Committee. This committee is then trained on how to manage and
maintain the local project and the whole community contributes to the construction of the
project.
Evaluation of clean water non-profit organization 3


Financial support for the projects comes from fundraising efforts and donations provided
by the general public. Charity: water follows a 100% model in which all public donations are
used to fund clean water projects. Organizational costs are covered by private donors,
foundations, corporate sponsors, and an annual gala
5
. Charity: water strives to maintain a high
level of financial transparency and accountability. In 2012, the charity evaluator Charity
Navigator gave charity: water an overall score of 67.32 out of 70, including a 66.22 financial
performance subscore and a perfect 70 accountability and transparency subscore, placing charity:
water as one of their highest-ranked charities
2
. The financial structure of charity:water does
appear successful, as new staff members have been added to their website directory over the past
few years, indicating growth.
While charity: water provides a shining example of financial management, they do not
possess the same level of impression for display of data and research for their completed
projects. In 2012, the non-profit GiveWell evaluated charity: water, focusing on charity: waters
program activities rather than its financials. GiveWell concluded their evaluation without making
a recommendation due to uncertainty regarding charity: waters humanitarian impacts and
relative effectiveness of their partner selection process
8
. GiveWell conducted an interviews with
charity: water in which charity: water admitted to having a lack of data regarding both the status
of their completed projects as well as the humanitarian impacts of their projects, stating that in
some cases their projects or the communities they work in are too new to yield useful data
6,7
.
With a plethora of water-centric NGOs and non-profits providing support and
implementing projects in developing communities around the world, its difficult to determine
which ones are providing significant benefit and which ones are merely following the current
trends. While charity:waters efforts are commendable, they are not guaranteed to succeed or
Evaluation of clean water non-profit organization 4


remain viable in the long-term. Adequate data collection and analysis should be performed to
reveal the status of their efforts. Knowing what works and what doesnt will help charity:water
focus their efforts to provide the most benefit and ensure that their projects remain intact and
useful for the local communities. Otherwise, some or many of these efforts may be in vain with
communities reverting back to their initial conditions when the projects do not live up to the
expectations or fail prematurely without support.

Goal and Objectives
The primary goal of this study is to assess the effectiveness and humanitarian impacts of
charity: waters completed projects as observed by charity: water employees and partners. Due to
the lack of data regarding their completed projects, the results of this analysis may provide some
useful data regarding areas of improvement and future efforts by the organization or those
looking to do similar work. Also, having charity: water members evaluate their own organization
may provide them some insight, allowing for reflection of their efforts and priorities.
As a charity organization, charity: water relies on the resources and goodwill of others to
support their work. As such, charity: water has the responsibility of ensuring that their efforts are
effective and providing noticeable and long-lasting benefits to the communities they serve.
Otherwise, the resources and support that charity: water receives might be better spent
elsewhere.

Evaluation of clean water non-profit organization 5


Literature Review
Increasing the availability of safe drinking water has been estimated to prevent up to 94%
of diarrheal disease cases
1
. Many organizations contribute to the effort of providing clean
drinking water and proper hygiene facilities, but implementing such projects and maintaining
them in the long-term are two different endeavors, the latter of which many organizations fall
short on.
Charity: water makes the case of implementing, for the most part, simple technologies
that can be installed and maintained by local community members. But even hand-pumps, which
are considered among the simpler of water infrastructure technology, may see significant rates of
non-functionality usually due to misuse and vandalism
10
. Average estimates indicate that over a
third of all hand-pumps in rural sub-Saharan Africa are non-functional at any time
12
, and one
study found two rural communities in South Africa with local hand-pumps non-functional more
than half the time
13
. Typically, infrastructure failures are the result of insufficient maintenance
and operation and there is no guarantee that someone will either be around or take the time to fix
it. When non-operational periods do occur, many people revert to obtaining their water from
nearby surface water sources which are often unfit for consumption
13
.
Implementing clean water technologies is the easy part. Establishing a local management
system to maintain those technologies and ensure their continued functionality is where many
NGOs and similar organizations fail. Many agencies focus on water infrastructure, but put little
emphasis on building management capacity
11
. While the current trend of community
management is widely applied, levels of project sustainability remain unacceptably low
9
.
Agencies often act as facilitators in the creation of a local water committee which is then
responsible for all water supply management issues in the community. However, such
Evaluation of clean water non-profit organization 6


management systems often breakdown due to a number of factors, including disinterest in
continued support from community members, often due to a lack of long-term incentives; no
mechanism or strategy to replace committee members; the water committee loses the trust and
respect of the community; decline of community members willingness to pay maintenance fees;
and not having enough capital to replace major items when they breakdown.
Most of these issues do not occur until one to three years after project completion
9
, at
which time many agencies involved in such projects have left under the illusion that the short
period of success and enthusiasm immediately following project completion will continue.

Research Method
The chosen evaluation method was a survey (see Appendix A) distributed to all charity:
water employees, and to as many organization partners and volunteers as was possible.
Distribution was focused toward people directly involved in charity: waters operations and
project efforts. The survey was organized into five blocks or categories of evaluation: Project
Selection, Data Collection, Health Impacts, Capacity Building, and Awareness.
The survey categories focused on the areas considered most pertinent for evaluating the
perceived effectiveness and humanitarian impact of charity: waters projects:
Project Selection assessed the selection, completion, and maintenance of projects.
Data Collection assessed the perceived amount of data collected and required during
different project phases.
Health Impacts assessed observed community health before and after project
implementation.
Evaluation of clean water non-profit organization 7


Capacity Building assessed the participation and involvement of community members
and local institutions in project development and implementation, and the efforts of
charity: water to build local capacity.
Awareness assessed the awareness and sharing of project information to other
communities, institutions, and organizations.
Each block contained three to five questions with each question using a 5-point Likert scale
format. Five answer choices were provided for each question with each choice linked to a
quantitative value on the scale of +1, +0.5, 0, -0.5, -1. Survey results were quantified and
analyzed using the HEAR methodology and appropriate EMYCIN formulas:

I
A
+ I
B
(I
A
* I
B
) I
A
> 0 I
B
> 0
I
A
+ I
B
+ (I
A
* I
B
) I
A
< 0 I
B
< 0
I
A
+ I
B
/ 1 min[ |I
A
|, |I
B
| ] Otherwise

A range coefficient, or weight factor, was calculated for each question to determine the
range of response values while taking into consideration the number of non-responses. Range-
adjusted values were then calculated for all responses for each question, and these values were
then used to calculate a combined weight, or index value, for each question. The index values of
all questions in a block were then combined in a similar fashion to provide a block index value.
This block index value can then be assessed qualitatively by comparing to the value scale used
for the survey answer choices.
Results and Discussion
Evaluation of clean water non-profit organization 8


As illustrated in Table 1, survey responses appear to indicate a generally positive
perception of charity: waters activities by its members. However, these values must be
considered in the context of the questions asked in each block.

Table 1. Survey blocks and overall index values.


Prior to survey distribution, predictions were made as to how each survey category would
fare. Project Selection and Awareness were predicted to receive favorable results due to charity:
waters selection process using partners stationed at potential project sites where local conditions
can be easily evaluated. Spreading awareness and sharing information is also a priority for
charity: waters mission. From the survey results, Project Selection received a very favorable
0.803, indicating that respondents view charity: waters project efforts quite satisfactorily.
However, Awareness did not fare as well, garnering a -0.081. While awareness may be a priority
for the organization, clearly respondents felt that much improvement could be made in that area.
This may be due to several factors. Part of it may be charity: waters efforts at spreading
awareness and information sharing, efforts which may leave considerable room for
improvement. There may also be difficulty spreading awareness to local communities and
governments, either due to missing or inadequate communication channels, physical distance, or
mismanagement and disorganization.
Health Impacts and Capacity Building were predicted to receive neutral to favorable
results depending on respondents observations and limited analysis of community conditions.
Project
Selection
Data
Collection
Health
Impacts
Capacity
Building
Awareness Overall
Index Value 0.803 0.113 0.566 -0.538 -0.081 0.822
Evaluation of clean water non-profit organization 9


Health Impacts yielded an index of 0.566 which fits in the predicted outcome range. The
indications of how this block was received may be misleading, however, as two of the three
questions asked about the observed status of a project community at least six months after
implementation. There were no questions relating specifically to long-term status of a
community, such as 2 or 3 years after implementation when the probability of project failures or
community disinterest increases. The responses also come from people who are most likely
either unqualified, or have not spent enough time with a project community to accurately assess
the health and vitality of the community.
Capacity Building yielded a less favorable index of -0.538, indicating that involvement in
projects is not very high. Looking closer at the individual questions asked in this block, it can be
seen that a small perceived improvement in project community capacity does take place. Two
questions asked about a communitys capacity before and after project implementation, yielding
index values of -0.233 and -0.191, respectively
A
. Similar to the Health Impacts block, these
questions do not specify a time restraint for how long after implementation observations should
be considered. Capacity takes time to develop so a more thorough investigation into project
communities would be necessary to see if charity: waters efforts have effected any significant
changes in local capacity.
Finally, Data Collection was predicted to receive neutral to unfavorable results due to an
admitted lack of data collection by the leaders of charity: water. Data Collection had an index of
0.113, close to being neutral indicating that most respondents did not know enough about the
data collected or required for project implementation and monitoring. This result fits the
prediction and is an area that charity: water themselves have said they need to improve.

A
see Appendix B for raw survey responses, range-adjusted, and question index values.
Evaluation of clean water non-profit organization 10


Overall, the survey, and thus charity: waters efforts and operations, were considered
quite favorable by respondents with an overall index value of 0.822. Since the survey covered a
broad range of topics, a closer look reveals significant areas of improvement. Given that the
survey was also distributed to people of varying occupations and roles within the charity: water
network, there are likely data gaps in the survey results. Coupled with this is potential bias
included in the wording of questions and responses provided. More focused efforts to evaluate
different areas of charity: waters operations would undoubtedly provide more useful results.

Conclusions
The goal of this study was to evaluate charity: waters effectiveness and humanitarian
impacts of their clean water projects. The evaluation was performed through use of a survey
distributed to charity: water employees and partners. The results of the survey indicate that the
survey respondents generally favor charity: water overall, and particularly when it comes to the
selection and health impacts of their projects. Data collection, and awareness and information
spreading are both areas that could see improvement while capacity building requires significant
improvement.
While the survey used in this study is not comprehensive or final by any means, it is a
good starting point for future evaluation efforts. If charity: water identifies and acknowledges
their weak points, of which they have already accomplished to some extent, and makes efforts to
strengthen those points then hopefully charity: waters continued and future projects will succeed
where others have failed.
References
Evaluation of clean water non-profit organization 11


1. Bartram J, Gordon B. The global challenge of water quality and health. Water Pract.
Technol. 2008;3(4).
2. Charity Navigator [Internet]. Glen Rock (NJ); c2012. Charity: water; 2012 [cited 2013 Oct
25]; [about 7 screens]. Available from:
http://www.charitynavigator.org/index.cfm?bay=search.summary&orgid=12548#.UqKfXuK
ArSj
3. Charity: water [Internet]. New York City (NY); c2013. Dollars to projects; 2013 [cited 2013
Oct 25]; [about 8 screens]. Available from: http://www.charitywater.org/d2p/
4. Charity: water [Internet]. New York City (NY); c2013. Our mission; 2013 [cited 2013 Oct
25]; [about 2 screens]. Available from: http://www.charitywater.org/about/mission.php
5. Charity: water [Internet]. New York City (NY); c2013. The 100% model; 2013 [cited 2013
Oct 25]; [about 3 screens]. Available from: http://www.charitywater.org/100percent/
6. GiveWell [Phone transcript]. c2012. Charity: water / GiveWell / Good Ventures meeting;
2012 [cited 2013 Oct 25]; [about 6 pages]. Available from:
http://www.givewell.org/files/DWDA%202009/charitywater/charity%20water%20Notes,%2
0061812.doc
7. GiveWell [Phone transcript]. c2012. GiveWells notes from a meeting between GiveWell
and charity: water, July 17
th
, 2012; 2012 [cited 2013 Oct 25]; [about 3 pages]. Available
from:
http://www.givewell.org/files/DWDA%202009/charitywater/CHARITY%20WATER%20-
%20Edited%20Conversation%20Notes%202.pdf
Evaluation of clean water non-profit organization 12


8. GiveWell [Internet]. San Fransisco (CA); c2012. charity: water; 2012 [cited 2013 Oct 25];
[about 5 screens]. Available from: http://www.givewell.org/international/charities/charity-
water
9. Harvey PA, Reed RA. Community-managed water supplies in Africa: sustainable or
dispensable. Community Dev J. 2006;42(3):365-378.
10. Hoko Z, Hertle J. An evaluation of the sustainability of a rural water rehabilitation project in
Zimbabwe. Phys Chem Earth A/B/C. 2006;31(15):699-706.
11. Lammerink MP. Community managed rural water supply: experiences from participatory
action research in Kenya, Cameroon, Nepal, Pakistan, Guatemala and Colombia. Community
Dev J. 1998;33(4):342-352.
12. Majuru B, Jagals P, Hunter PR. Assessing rural small community water supply in Limpopo,
South Africa: water service benchmarks and reliability. Sci Total Environ. 2012;435-
436:479-486.
13. Majuru B, Mokoena MM, Jagals P, Hunter PR. Health impact of small-community water
supply reliability. Int J Hyg Envir Heal. 2011;214:162-166.

Evaluation of clean water non-profit organization 13


Appendix A - Survey

Project Selection
1. Is the project evaluation and selection process satisfactory?
(A) Very (B) Moderately (C) Somewhat (D) Not at all (E) Unsure
2. How often are completed projects as successful as hoped or predicted?
(A) Very often (B) Often (C) Occasionally (D) Rarely (E) Unsure
3. How well are projects maintained after the first year of completion?
(A) Very well (B) Well (C) Poorly (D) Very poorly (E) Unsure
4. How often are broken or abandoned projects rehabilitated?
(A) Very often (B) Often (C) Occasionally (D) Rarely (E) Unsure

Data Collection
1. Is the amount of data collected before project implementation adequate to continue the
project?
(A) Very (B) Moderately (C) Somewhat (D) Not at all (E) Unsure
2. Is the amount of project data collected after completion satisfactory for monitoring efforts?
(A) Very (B) Moderately (C) Somewhat (D) Not at all (E) Unsure
3. What amount of data is required to maintain a completed project?
(A) A lot (B) Moderate amount (C) Some (D) None (E) Unsure

Evaluation of clean water non-profit organization 14


Health Impacts
1. What is the observed health of the community before a project?
(A) High (B) Good (C) Poor (D) Very poor (E) Unsure
2. What is the observed health of the community at least six months after completion?
(A) High (B) Good (C) Poor (D) Very poor (E) Unsure
3. What is the observed morale of the community at least six months after completion?
(A) Much better (B) Better (C) Worse (D) Much worse (E) Same

Capacity Building
1. How involved are local community members in project implementation?
(A) Very (B) Moderately (C) Somewhat (D) Not at all (E) Unsure
2. How involved are local governments/authorities in project implementation?
(A) Very (B) Moderately (C) Somewhat (D) Not at all (E) Unsure
3. What is the capacity of the community for implementing the project beforehand?
(A) Very high (B) High (C) Low (D) Very low (E) Unsure
4. What is the capacity of the community for maintaining the project after completion?
(A) Very high (B) High (C) Low (D) Very low (E) Unsure
5. How involved is charity: water in building capacity beyond water projects?
(A) Very (B) Moderately (C) Somewhat (D) Not at all (E) Unsure

Evaluation of clean water non-profit organization 15


Awareness
1. How well is project awareness spread to other communities outside the target community?
(A) Very well (B) Well (C) Poorly (D) Very poorly (E) Unsure
2. How well are local governments/authorities made aware of the projects?
(A) Very well (B) Well (C) Poorly (D) Very poorly (E) Unsure
3. How much improvement could be made with information sharing efforts?
(A) None (B) Some (C) Moderate Amount (D) A lot (E) Unsure

Evaluation of clean water non-profit organization 16


Appendix B Raw Survey Responses, Range Adjusted, and Question Index Values


Table B-1. Raw responses for the Project Selection survey block.



Table B-2. Range adjusted and question index values for the Project Selection survey block.



Table B-3. Raw responses for the Data Collection survey block.



Table B-4. Range adjusted and question index values for the Data Collection survey block.



Project
Selection
Total
Responses
Non-
Responses
Very Often / Well Unsure
Occasionally /
Somewhat
Rarely /
Poorly
1 0.5 0 -0.5 -1
Q1 30 4 14 10 5 1 0
Q2 33 1 15 4 10 2 2
Q3 26 8 7 11 5 2 1
Q4 34 0 5 12 6 6 5
Project
Selection
Very Often / Well Unsure
Occasionally /
Somewhat
Rarely /
Poorly
Question
Index
1 0.5 0 -0.5 -1
Q1 0.4044 0.1444 0.0000 -0.0144 0.0000 0.4830
Q2 0.4408 0.0588 0.0000 -0.0294 -0.0588 0.4238
Q3 0.1864 0.1464 0.0000 -0.0266 -0.0266 0.2670
Q4 0.1471 0.1765 0.0000 -0.0882 -0.1471 0.0968
Data
Collection
Total
Responses
Non-
Responses
Very Moderately Unsure Somewhat Not at all
1 0.5 0 -0.5 -1
Q1 27 7 6 2 7 7 5
Q2 28 6 9 5 5 5 4
Q3 19 15 2 5 3 7 2
Data
Collection
Very Moderately Unsure Somewhat Not at all
Question
Index
1 0.5 0 -0.5 -1
Q1 0.1646 0.0274 0.0000 -0.0960 -0.1372 -0.0400
Q2 0.2526 0.0702 0.0000 -0.0702 -0.1122 0.1580
Q3 0.0222 0.0277 0.0000 -0.0388 -0.0222 -0.0114
Evaluation of clean water non-profit organization 17


Table B-5. Raw responses for the Health Impacts survey block.



Table B-6. Range adjusted and question index values for the Health Impacts survey block.



Table B-7. Raw responses for the Capacity Building survey block.



Table B-8. Range adjusted and question index values for the Capacity Building survey block.



Health
Impacts
Total
Responses
Non-
Responses
High / Much
Better
Good /
Better
Unsure /
Same
Poor / Worse
Very Poor /
Much Worse
1 0.5 0 -0.5 -1
Q1 27 7 2 5 8 6 6
Q2 27 7 6 11 7 2 1
Q3 29 5 15 11 1 1 1
Health
Impacts
High / Much
Better
Good /
Better
Unsure /
Same
Poor / Worse
Very Poor /
Much Worse
Question
Index
1 0.5 0 -0.5 -1
Q1 0.0549 0.0686 0.0000 -0.0823 -0.1646 -0.1291
Q2 0.1646 0.1509 0.0000 -0.0274 -0.0274 0.2501
Q3 0.4281 0.1570 0.0000 -0.0143 -0.0285 0.4965
Capacity
Building
Total
Responses
Non-
Responses
Very
Moderately
/ High
Unsure
Somewhat /
Low
Not at all /
Very Low
1 0.5 0 -0.5 -1
Q1 32 2 4 11 4 10 3
Q2 30 4 1 10 5 5 9
Q3 26 8 1 2 5 14 4
Q4 27 7 4 4 3 7 9
Q5 25 9 4 1 10 1 9
Capacity
Building
Very
Moderately
/ High
Unsure
Somewhat /
Low
Not at all /
Very Low
Question
Index
1 0.5 0 -0.5 -1
Q1 0.1172 0.1611 0.0000 -0.1465 -0.0879 0.0487
Q2 0.0289 0.1444 0.0000 -0.0722 -0.2600 -0.1737
Q3 0.0266 0.0266 0.0000 -0.1864 -0.1065 -0.2327
Q4 0.1097 0.0549 0.0000 -0.0960 -0.2469 -0.1909
Q5 0.1024 0.0128 0.0000 -0.0128 -0.2304 -0.1426
Evaluation of clean water non-profit organization 18


Table B-9. Raw responses for the Awareness survey block.



Table B-10. Range adjusted and question index values for the Awareness survey block.





Awareness
Total
Responses
Non-
Responses
Very Well /
None
Well / Some Unsure
Poorly /
Moderate
Very Poorly /
A lot
1 0.5 0 -0.5 -1
Q1 25 9 10 5 2 4 4
Q2 33 1 8 5 9 3 8
Q3 34 0 1 4 1 26 2
Awareness
Very Well /
None
Well / Some Unsure
Poorly /
Moderate
Very Poorly /
A lot
Question
Index
1 0.5 0 -0.5 -1
Q1 0.2560 0.0640 0.0000 -0.0512 -0.1024 0.1823
Q2 0.2351 0.0735 0.0000 -0.0441 -0.2351 0.0307
Q3 0.0294 0.0588 0.0000 -0.3824 -0.0588 -0.3636