You are on page 1of 22

MERCADEO SOCIAL

Capítulo 15

Developing a Plan for Monitoring and Evaluation

KOTLER, Philip y Nancy LEE (2008) Social Marketing: Influencing Behaviors for Good. Tercera
edición. SAGE Publications, pp. 323-340.

Lectura perteneciente al:

Tema 3: Segunda etapa del Mercadeo Social: El


Diseño estratégico

Comentario sobre la lectura


un buen plan de evaluación que permita medir los resultados planteados en
el Plan de Mercadeo.

Maestría en Gerencia Social

La presente selección de textos ha sido preparada de manera exclusiva para los participantes
del curso Mercadeo Social, que forma parte de la Maestría en Gerencia Social. Queda
prohibida su difusión y reproducción por cualquier medio o procedimiento, total o parcial,
fuera del marco del presente curso.

Pontificia Universidad Católica del Perú


Developing a Plan for Monitoring and Evaluation

Marketing is a learning game. You make a decision. You watch the results. you learn from
the result. Then you make better decisions.

-Phillip Kotler

Now you've reached a step you may not be eager for developing a plan for monitoring
and evaluation. If this is true for you, your experiences and conversions may be similar to
those people making the following common laments:

• My administrators and additional funders think it’s nice I can report on how many
PSAs we ran and how many hits we got to our web site, but I can see it in their eyes. It is not
enough. They want to know how many more people got an HIV/AIDS test as a result of our
efforts. And actually, that’s not even enough. What they really want to know is how many
positives did we find and how much did it cost us to find each one.
• You think that is hard. In my line of work, they want to know if the fish are any healthier

• Most of the evaluation strategies I have looked at could cost as much as the small budget
I have far this campaign. I honestly can’t justify it. And yet, everyone seems to want it.

• Quite frankly, my concern is with the results. What if it's bad news-that we didn’t reach
our goal? They like the plan, are going to fully fund it, and trust chat we know what we're
doing. Bad news could dampen any further work like chis.

ln this chapter's opening highlight, you'll read how one program director addresses
these challenges, developing an evaluation approach far a your suicide prevention program
that she can stand behind and others respect.

... 323
PART V: MANAGING SOCIAL MARKETING PROGRAMS
Youth Suicide Prevention (2007)

SUE EASTGARD, MSW, DIRECTOR

Youth Suicide Prevention Program of Washington


Background
On the weekend of Martin Luther Kíng's holiday, Trevor Simpson drove north to the Tulalip
Indian Reservation from his home in Edmonds, Washington. He went alone, he went
without others knowing where he was going or why. In the quiet of the night, Trevor hung
himself with the battery cable of his car.

Trevor was a popular student at Edmonds- Woodway High School. He was intelligent, earning
a 3.9 grade point average. He was an athlete, playing halfback for the school’s foot ball team
and coaching his younger brother's soccer team. He was adored by his parents and
extended family. So why did he take his own life?
In 2003 there were 4,232 youth (10-24 years of age) completed suicides in the United
States. Many, many more young people made suicide attempts that resulted in emergency
room visits and hospitalizations. Boys and young men are much more likely to end their
lives by suicide, while girls and young women make more attempts. Among ethnic groups,
Native American young people have the highest rates of suicide.

Depression is a significant component of most suicidal deaths. Depression is a chemical


imbalance you can be born with it or you can develop it as a result of situational crises.
Depression in children and adolescents looks different than it does in adults; irritability,
anxiety, hypersensitivity to criticism, and physical complaints are common symptoms of
depression for young people. Sadly, many depressed teens are not diagnosed or do not
receive adequate treatment.

About 80% of the time, depressed teens exhibit clues or warning signs about their thoughts
of suicide. Trevor asked his friend Monica the day before he died, “lf you were going to kill
yourself, how would you do it?" In hindsight we can probably appreciate that Trevor was
trying to talk about suicide, but unfortunately Monica did not know that talking about
suicide was a warning sign Trevor also gave his favorite baseball cap to his friend Jason,
indicating that he was going to get another one. Jason admired Trevor's hat and was pleased
to accept it as a sign of friendship. Had Jason known that giving away prized possessions
was another sign, he would have reacted differently. In addition to these "clues," we need
to also be watchful for a preoccupation with death, a hopeless mood, increased drug and
alcohol use, and a change in normal activities. Prior suicidal behavior is also an important
factor, as past behavior influences present and future behavior.
Chapter 15: Developing a Plan for Monitoring and Evaluation 325

Target Audiences

The Youth Suicide Prevention Program (YSPP) is committed to reducing suicidal


behaviors in Washington State by raising public awareness, by providing training, and by
supporting communities to cake action. Educational presentations to identify the warning
signs and helpful intervention strategies are delivered to parents, teachers, coaches, and
mentors. In middle and high schools across the state. student prevention teams facilitate
!earning far their peers regarding depression and how to help a friend who may be at
risk of suicide. Formal training on suicide assessment and intervention is held for
professional caregivers. YSPP also works with coalitions to plan and implement local
suicide prevention efforts.

Desired Behaviors

Three behaviors are promoted to target audiences concerned that a youth is suicidal.
The first way to be helpful to suicidal teens is to show that you care: ask how they are
doing, listen to their feelings, and avoid giving advice or demanding that they "get over it."
lt is important that we attempt to understand their frustrations, their worries, their
problems. As adults we usually have a capacity for greater perspective thank goodness!
Suicidal teens typically do not want to die; they warm to find a solution to their problems.

The second intervention strategy is to ask the question. lf you suspect young people may be
at risk of suicide, it is important to ask directly and calmly, “Are you thinking about
harming yourself about dying?" This question will not plant the idea of suicide in their brain;
it will actually give permission if the thoughts are present to talk about them. Giving
permission to talk about suicide can relieve pressure; not talking about suicide can leave
adolescents feeling hurt and more alone.
lf the answer to the question about suicide is yes or you are concerned that it is, then it is
time to get help, the third intervention strategy. Help might mean calling a hotline or talking
to a school counselor, coach, or favorite teacher. lt is important not to leave suicidal youths
alone.
Results
The mission of YSPP is to reduce the incidence of suicidal behaviors by building public
awareness, offering training, and supporting communities to tackle the problem locally.
To build awareness we support peer-to-peer programming in middle and high schools
across the state. While students do not always know whether their efforts have made a
difference in the statistics related to suicidal behavior, they know that isolation and bullying
are two behaviors that can enhance the risk of depression and suicide. As one student-
participant said, "I have no idea if we saved any lives, but no one sits by themselves at lunch
anymore.'' The peer-to-peer campaign messages might include "Be a lifesaver" or "Take your
friend by the hand before their hand takes their life."
In addition to student education, YSPP talks with caregivers parents, scout leaders,
pastors, educators, and soccer coaches- about adolescent depression and suicide. We
reach out to the public through our Web site as well as through brochures, posters,
newspaper articles, and 1V and radio interviews. We work to address the stigma that gets
in the way of seeking help in a time of crisis.

YSPP is providing training to frontline gatekeepers. We believe that learning how to intervene
in a suicidal situation is as important as learning CPR; suicide intervention can be seen as
psychological first aid. Our training workshops develop competencies in four areas: attitudes,
knowledge, intervention, and resources. We adopt a highly interactive, applied learning
approach ensuring that the participant comes away with skills and not just extensive notes
written on a tablet but never looked at again.
Finally, we work to strengthen community safety nets for our kids. If every community
built a stronger support system for youth, perhaps fewer young people would fall
through the cracks. As one community advocate said, “we want to change the incredible
sadness into an energy that would be usable and positive and actually do something."
YSPP works to promote cooperation and colaboration between mental and physical
health resources and families. We advocate for the early identification of children's
mental health issues and encourage families to seek help, rather than avoid it. We
support the accreditation and incorporation of local crisis hotlines into the national
Lifeline-
1-800-273-TALK. We provide technical assistance and consultation for local groups that
want to address the problems of depression and hopelessness in their community.
Results

Every suicide prevention program needs to include evaluation activities that help to
determine the efficacy of the services. Given the complexity of the issue (and the cost
of doing research), it may be impractical, even impossible, to determine whether a
program is responsible for a decrease or increase in mortality and morbidity rates. We
can, however answer questions related to changes in attitudes, skills, and knowledge.
Research shows, for example, that more exposure to suicide prevention training creates a
greater likelihood that students will use the three critical skills of “show you care ask about
suicide, and get help." As a result, we attempt to answer the following questions in order to
evaluate our activities.

• Did training increase the likelihood that a teacher will be able to accurately assess and
intervene with suicidal youth?

• Were there more referrals to the school counselor as a result of a curriculum focused on
enhancing help seeking behaviors?

• Did parents who have attended an educational presentation feel more comfortable asking
young people directly about suicide?
• Can more youth identify the warning signs for suicide after the introduction of a
curriculum than before?
Knowledge and skills gained by training participants at the close of a 2-day workshop
persisted after 3 months and in some instances even persisted at 9 and 12 months. Over
35% of training participants reported contact with one or more suicidal youth and were
referring youth to a wide range of resources. As a result of training, participants reported
more comfort in helping a suicidal person, along with more confidence that they would
try to help and a greater feeling of competence when actually helping.

DEVELOPING A PLAN FOR MONITORING AND EVALUATION

We recommend that you take time to develop a plan for monitoring and evaluating
your social marketing effort before creating your budget in Step 9 and implementation plan
in Step 10. You will want your final budget to include funding for this critical activity and your
implementation plan to include action items to ensure it happens.
This chapter will guide you in determining these funding needs and identifying related
activities. lt is intended to help by outlining components of a monitoring and evaluation plan,
posed in the form of questions you'll want to answer in a sequential way, staring with
the toughest one, of course.
• Why are you conducting this measurement?
• What will you measure?
• How will you conduce these measurements?
• When will these measurements be taken?
• How much will it cost?

One distinction is important to clarify up front: the difference between the term
monitoring and the term evaluation.
Monitoring refers to measurements that are conducted sometime after you launch your
social marketing effort but before it is completed. It is purpose is to help you determine
whether you need to make midcourse corrections that will ensure that you reach your
ultimate marketing goals. Remember the VERB Summer Scorecard campaign mentioned
in Chapter 2 encouraging tweens to monitor their physical activity and try new ways to be
phystcally active throughout the summer? Program managers might be eager to know in
July how the tweens are doing toward the goal to get 24 squares stamped by the end
of the summer. Are the partners, such as public pools, honoring the special promised deals?
Are parents being supportive of the effort? Methods to conduct monitoring in this
particular case might simply involve talking with a few kids, partners, and parents on the
phone and then making recommendations for any increased or new activities to support
the kids during the rest of the summer.
Evaluation, on the other hand, is a measurement and final report on what happened,
answering the bottom-line question: Did you reach your goals for changes in behaviors,
knowledge, and attitudes? \'what percentage of tweens completed all 24 squares, and
how does this compare to the plan's goal? Additional questions are also likely to be
addressed in the evaluation. Were activities implemented on time and on budget? Were
there any unintended consequences that will need to be addressed now or in future similar
projects? Which program elements worked well to support outcomes? Which ones
didn't? Was there anything missing? What will you do differently next time, if there is a
next time?
WHY ARE YOU CONDUCTING THIS MEASUREMENT?

Your purpose for this measurement often shapes what you measure, how you measure and
when Consider the differing implications for your plan for each of the following potential
reasons for your effort:

• To fulfill a grant requirement


• To do better the next time we conduct this same campaign
• To (hopefully) get continued, even increased funding
• To help us decide how to prioritize and allocate our resources going forward
• To alert us to midcourse corrections we need to make in order to achieve goals

For a Grant Requirement: Sometimes the nature of the monitoring and/or evaluation will be
predetermined by specifications in a grant. Consider an example where a city receives a grant
from a state department of transportation (DOT) to increase the use of pedestrian flags in
the city’s eight crosswalks in a downtown corridor. Assume the DOT is hoping that this city's
campaign strategies are successful and that these strategies can then be shared by DOT with
other cities in the state. The campaign's evaluation plan will certainly include measuring
levels of flag usage before and after the campaign. And the funder will need to be assured
that the data were collected using a systematic, reliable, and verifiable methodology, one
that could be replicated in other cities as well.

To Do Better Next Time: What if, instead, you are sincerely interested in measuring what
happened so that you can improve results in your next similar effort. Perhaps it is a pilot
and )'OU want to evaluate the campaign elements to decide what worked well and should
be repeated, what could be improved, and what elements should be "dropped" next time
around. Imagine a countywide effort to reduce smoking around children in cars and that a
pilot the first year is to help determine what elements of the campaign should be used when
rolled out m Year 2 countywide. The pilot includes a packet of materials sent home from
school with children in the elementary schools and contains a secondhand tobacco smoke
Information card, a plug to replace the cigarette lighter, a smoke free pledge card, and an
air freshener with the campaign's slogan, "Please Smoke Outside." Follow-up surveys with
parents will then measure changes in levels of smoking around their child in the car as well
as parents' ratings on which of the materials in the packet they noticed, used, and said were
influential. Imagine further that the results indicated that some of the parents thought the
air freshener would reduce the harmful effects of the smoke, so they did not change their
habits. This finding, of course, would then lead the county to eliminate this U.SO item when
rolled out countywide.

To Get Support for Continued Funding: Often the purpose of an evaluation is to persuade
funders to reinvest in the project to sustain it into the future. As would be imagined, key to
this will be co identify criteria the funders will use to make their decisions

to then create an evaluation plan that includes measures that will provide chis
information. Consider the Road Crew case in Wisconsin mentioned in Chapter 9, the one
describing a service using limousines and ocher luxury vehicles to pick people at their home,
business, or hotel, take them to the bars of their choice and home at the end of the evening
all for about S15-$20 an evening. A key statistic funders for chis program were interested m
was a cost benefit analysis, and the program's evaluation methodology provided just that. It
showed an estimated $15,300 cost per crash avoided compared with a savings to all those
impacted by a crash of S610, 000

To Help Determine Resource Allocation: Management may also, or instead, want or use an
evaluation effort to help decide how resources should be allocated in the future. In King
County, Washington, for example, the Department of Natural Resources and Parks wanted
an evaluation survey co help decide which of some 30 community outreach efforts should
receive more funding and which, perhaps, should be pulled back This objective led to a plan
to measure household behaviors that each of these 30 programs sought to influence (e.g.,
leave grass clippings on the lawn). Those programs with the greatest potential marker
opportunity for growth were then considered first for increased support, with market
opportunity determined by the percentage of house- holds doing the behavior sometimes,
but not on a regular basis (the in action stage of change) or not doing the behavior at all
but considering doing it (the contemplation stage of change).

To Decide if Course Corrections Are Needed. This purpose will lead to a monitoring effort,
measuring sometime after an effort launches, but before completion, to deter- mine
whether goals are likely to be met based on how the market is responding. Think back to the
inconvenient Truth case on global warming. Given some of the pushback from critics of global
warming, it would be interesting to know if the issue should be reframed as "climate
protection" or "climate change."
WHAT WILL YOU MEASURE?

What you will measure to achieve your evaluation purpose is likely to fall into one or
more of three categories: outputs, outcomes, and impacts. As you will read, required
efforts and rigor vary significantly by category.
Output/Process Measures

The easiest and most straightforward measures are those describing your campaign’s
outputs, sometimes referred to as process measures. This measure focuses on quantifying
your marketing activities as much as possible. They are distinct from outcome measures,
those focusing on target audience response to these activities. Many are available in your
records and databases.'

• Numbers of Materials Distributed: This measure refers to numbers to numbers of mailings,


brochures, flyers, key chains, bookmarks, booklets, and coupons handed out. Note this does
not indicate whether they were read or used only that they were distributed.

• Reach and Frequency: Reach refers to the number of different people or households
exposed to a particular image or message during a specified period. Frequency is the number
of times within this time frame, on average, that the target audience was exposed to the
communication.

• Media Coverage: Measures of media and public relations efforts, also referred to as
earned media, may include reporting on numbers of column inches in newspapers and
magazines, minutes on television and radio news and special programs, and people in the
audience attending a planned speaker's events. Efforts are often made to determine and
report what this coverage would have cost if it had been paid for.

• Total impressions/Cost per Impression: This measurement combines information from


several categories, such as reach and frequency, media exposure, and material
dissemination. Typically these numbers are combined to create an estimate of the total
number of people in the target market who were exposed to campaign elements. Taking
this to the next level of rigor, to achieve a cost per impression, total campaign costs
associated with this exposure can then be divided by the estimated number of people
exposed to the campaign. For example, consider statewide campaign targeting mothers
to increase children's fruit and vegetable consumption; the campaign may have collected
exposure information from media buys (e.g., parenting magazines) and any additional
efforts (e.g., messages on grocery bags). Let’s assume they were able to estimate that
100,000 mothers were exposed to these campaign efforts and that the associated costs
were $10,000. Their cost per impression would be $10. These statistics can then be used
over time to compare the cost efficiency of varying strategies. Suppose, for example, that in
a subsequent campaign, efforts reached 200,000 mothers after funds were redirected to
sending messages from child care and preschools, thus reducing the cost per impression
to SS.
• Implementation of Program Elements: An audit of major activities planned and
implemented (or not) may shed light on campaign outputs and outcomes. Did you do
everything you planned to do? Did you complete activities on time and on budget? This audit
can help address the tendency many of us have to expect campaign goals to be achieved,
even though we did not implement all planned activities or spend originally allocated funds
in planned time frames.

Outcome Measures

Measuring outcomes are a little more rigorous, as you are now assessing customer response
to your outputs, most likely involving some type of primary research surveys. Ideally, these
measures were determined by the goals you established in Step 4, the specific measurable
results you want your program to achieve one or more of the following types.
• Changes in Behavior. This may be measured and stated in terms of a change in percentage
(e.g., adult binge drinking decreased from 1796 to 6%), a percentage increase or decrease
(e.g., seatbelt usage increased by 20%), and / or a change in numbers (e.g., 40,000 new
households signed up for food waste recycling bins, increasing the total number of
households participating from 60,000 to 100,000).

• Changes in Behavior Intent: This measure might be appropriate for campaigns with
minimal exposure or when campaigns have been running for only short periods of time. It
may be the most appropriate measure for campaigns targeting those in the
precontemplation stage, when the social marketer's goal is to move them to contemplation
and then (eventually) to the action stage.

• Changes in Knowledge: This may include changes in awareness of important (acts (e.g.,
five drinks at one sitting is considered binge drinking), information (e.g., an estimated
75,000 people are on waiting lists for organ transplants), and recommendations (e.g., ear
five or more servings of vegetables and fruit daily for better health).
• Changes in Belief: Typical Indicators include attitude indicators (e.g., my vote doesn't
count), opinions (e.g., native planes are not attractive), and values (e.g., tanning is worth
the risk).

• Responses to Campaign Elements: Here you may be counting calls to an 800 number
(e.g., for a booklet on natural gardening), redemption of coupons (e.g., far a bike helmet),
mail or Internet orders or requests for more information (e.g., far a free consultation on
home earthquake preparedness), purchase of tangible objects that were promoted (e.g.,
number of new low- flow toilets sold or energy saving light- bulbs compared with numbers
in prior year), and services provided (e.g., number of blood pressure checks given at a mall
event), or participation rates (e.g., number of coupons far home emergency kits redeemed).

• Campaign Awareness: Though not necessarily an indicator of impact or success, measures


of awareness of campaign elements provide some feedback on the extent to which the
campaign was noticed and recalled. Measurements might include levels of unaided
awareness. (e.g., what you have seen or heard lately in the news about legal limits for
blood alcohol levels while driving); aided awareness (e.g., what have you seen or heard
lately in the news about our state's new 0.08% legal limit); or proven awareness (e.g., where
did you read or hear about this change in the law).

• Customer Satisfaction Levels: Customer satisfaction levels associated with service


components of the campaign provide important feedback for analyzing results and far
planning future efforts (e.g., ratings on levels of satisfaction with counseling at the Women,
Infants, and Children [WIC] program clinics).

• Partnerships and Contributions Created: Levels of participation and contributions from


outside sources are significant and represent positive responses to your campaign, even
though they may not be a reflection of the impact on your target audience behaviors. These
may include numbers of and hours spent by volunteers, partners, and coalition members
participating in the campaign, as well as amounts of cash and in-kind contributions received
from foundations, media, and businesses.

• Policy Changes: A legitimate campaign goal may focus on causing an important change in
polices or infrastructures that will encourage and/or support behavior change. In the
interest of oral heath for children, for example, efforts have paid off in some communities
persuading grocery stores to remove candy and gum from checkout lanes.

Impact Measures

This measure is the most rigorous, costly, and controversial of all measurement types. In
this category, you are attempting to measure the impact that changes in behavior you
achieved (e.g., more homeowners using natural fertilizers) have had on the social issue
your plan is addressing (e.g., water quality). It would, indeed, be great or be able to report
on the following types of impact measures, in addition to outputs and outcomes.

• Lives saved (e.g., from reducing drinking and driving)


• Diseases prevented (e.g., from increased physical activity)
• Injuries avoided (e.g., from safer workplace practices)
• Water quality improved (e.g., from taking prescription drugs back to pharmacies)
• Water supply increased (e.g., from increased purchases of low flow toilets)
• Air quality improved (e.g., from fewer leaf blowers being used in a community)
• Landfill reduced (e g., from composting food waste)
• Wildlife and habitats protected (e.g., from decreases in littering)
• Animal cruelty reduced (e.g., from Increases in spaying and neutering)
• Crimes prevented (e.g., from increases in use of motion sensors for outdoor lighting)

The reality is that not only is this rigorous and costly to determine, it may in fact be
inappropriate and inaccurate to try to connect your campaign activities with these
impacts, even though they were designed with them in mind.

Several key points can assuage you and others. First, you need or trust, or assume, that the
behavior that was chosen for your campaign is one that can have an impact on the issue (e.g.,
that folic acid can help prevent some birth defects). Second, you may need to wait longer to
measure, as there may be a lag between adopting the behavior and seeing the impact (e.g.,
increased physical activity or lower blood pressure levels). finally, your methodology for
measurement may need to be quite rigorous, controlling for variables that may also be
contributing to the social issue (e.g., there may not be an improvement in water quality in a
lake if during your campaign period a new manufacturer in the area started polluting the
same waters). You will need to be diligent and forthright about whether you believe you can
even determine and claim this victory

How Monitoring and Evaluation Relates to Logic Models


The terms output, Outcome, and impact are also used as labels in program evaluation logic
models. You may be familiar with ones with labels similar to those used in Table 15.1. In
these models, there are usually the additional categories of inputs and Activities, ones
included with Outputs as part of your Process Evaluation. As also noted in the table,
Outcomes are more near term and Impacts long term. In reality, there would be
multiple mentions m each column and arrows showing relationships between inputs
and activities, outputs and short-term outcomes, and short-term out· comes and long-term
impacts.

HOW WILL YOU MEASURE?

Our third step in developing an evaluation and monitoring plan is to identify methodologies
and techniques that will be used to actually measure indicators established in our first step.
Chapter 5 outlined typical research methodologies available to you, a few
Table 1;.1 Hypothetical Excerpts From a Logic Model for a Suicide Prevention Program
Evaluation

PROCESS EVALUATION IMPACT Evaluation


Inputs activities outputs Short terms- Long terms impacts
outcomes
Federal, Identification Youth and families increased Reduced suicided
state, of greatest are informed youth and among teens
and local suicide risks about risk factors family
funding factors for for and resource awareness and
middle availability at knowledge
school youth community forums suicide
warning sings
of which are most typical for evaluation and monitoring measures. In general, audience
surveys will be the primary technique used in measuring outcomes, given the focus the actual
influence you have had on your target audience in terms of behavior, knowledge, and beliefs.
Records, contact reports, anecdotal comments, and project progress reports will be the
primary sources of data for measuring outputs. impact measures, on the other hand, may
require more scientific or technical surveys.
Quantitative Surveys are needed when reliable data are key to evaluation (e.g., percentage
increase in levels of physical activity) and are most commonly conducted using telephone
surveys, self-administered questionnaires, and/or in -person inter. views. These may be
proprietary or shared cost-studies in which several organizations have questions for similar
populations. They may even rely on established surveys such as the Behavioral Risk Factor
Surveillance System (BRFSS), presented in Chapter 7
Qualitative Surveys should be considered when evaluation requirements are less stringent
or more subjective in nature and include methodologies such as focus groups, informal
interviews, and capturing anecdotal comments. Focus groups might be appropriate for
exploring with child care providers which components of the immunization tracking kits were
most and least useful and why. This information might then refocus efforts for the next kit
reprint. Informal interviews might be used to undemand why potential consumers walked
away from the low flow toilet display, even after reading accompanying materials and
hearing testimonials from volunteers. Anecdotal comments regarding a television campaign
might be captured on phone calls to a sexual assault resource line.

Observation Research is often more reliable than self-reported data and, when possible,
the most appropriate technique and can be used for evaluating behaviors such as wearing
a life vest, washing hands before returning to work, or topping off gas tanks. It may also
provide more insight for assessing skill levels and barriers than self-reported data (e.g.,
sorting garbage and placing it in proper containers or observing a WIC client finding her way
around a farmers' market for the first time).
Scientific or Technical Surveys may be the only sure methodology to assess the impact of your
efforts If you are charged with reporting back on the difference your efforts made in
reducing diseases, saving lives. improving water quality, and the like, you will need help
designing and conducting reliable scientific surveys that not only are able to measure
changes in these indicators but can also link these changes or your social marketing
campaign.'
Control Groups used in combination with quantitative and scientific or technical surveys will
further ensure that results can be closely tied to your campaign and program efforts. A drug
and alcohol prevention campaign might be implemented in high schools in one community
but not in another similar community. Extra precautions can even be taken to ensure the
similarity of the control groups by conducting surveys prior to the selection of the groups
and then factoring in any important differences. Results on reported drug use in the control
group of high schools are then compared with those in the other (similar) communities.

Records and Databases will be very useful for several indicators, particularity (hose
measuring responses to campaign elements and dissemination of campaign materials. This
may involve keeping accurate track of number of visits to a Web site and length of time spent,
numbers of calls (e.g., ca a tobacco quitline), numbers of requests (e.g., for child care
references), numbers of visits (e.g., to a teen clinic), numbers of people served (e.g., at car
sear inspections), or numbers of items collected (e.g., at a needle exchange)- This effort
may also involve working with suppliers and partners to provide similar information from
their records and databases, such as numbers of coupons redeemed (e.g., far trigger
locks), tangible objects sold (e.g., compost tumblers featured in the campaign), or requests
received (e.g., organ donation applications processed).

WHEN WILL YOU MEASURE?

Earlier, we distinguished between evaluation and monitoring, referring to final assessments


of efforts as evaluation and ongoing measurements as monitoring. Timing, then, for
measurement efforts is likely to happen as follows:

1. Prior to campaign launch, sometimes referred to as precampaign or baseline measures


2. During campaign implementation, thought of as tracking and monitoring surveys, and
may be one time only or over a period of years (i.e., longitudinal surveys)

3. Post campaign activities, referring to measurements taking place when all campaign
elements are completed. Providing data on short term outcomes and long-term impact.

Baseline measures are critical when campaigns have specific goals for change and future
campaign efforts and funders will rely on these measures for campaign assessment. These
are then compared with post campaign results, providing a pre- anti post evaluation
measure. Monitoring efforts during campaigns are often conducted to provide input for
changes midstream and to track changes over time. Post campaign (final) assessments are
the most typical points in time for evaluation, especially when resources and tight time
frames prohibit additional efforts. A few programs will use all points in time for evaluation,
most common when significant key constituent groups or funders require solid evidence of
campaign outcomes.
HOW MUCH WILL IT COST?
Costs for recommended monitoring and evaluation activities will vary from minimal costs
for those that simply involve checking records and databases or gathering anecdotal
comments, or moderate costs for those involving citizen surveys or observation research,
to potentially significant costs for those needing scientific or technical surveys. Ideally,
decisions to fund these activities will be based on the value they "'ill contribute to your
program. If it will assist you in getting support and continued funding for your program, it
may be a wise investment. If it helps you refine and improve your effort going forward, there
is likely to be a payback in terms of return on your investment once a methodology is
determined based on your research purpose; you can then assess these potential costs
versus potential benefits.

ETHICAL CONSIDERATIONS IN EVALUATION PIANNING


Ethical considerations for monitoring and evaluation are similar to those discussed regarding
research and have the most focus on the respondents that are surveyed for the evaluation.
One additional issue worthy of mention is the extent ta which you should (or can) measure
and report on unintended outcomes (consequences) as well, both positive and negative.
For example, many program managers are now reporting concerns with their success in
encouraging recycling. Although volumes of materials are being, recycled that might
otherwise have been put in landfills, managers believe they have significantly increased the
use of recyclable materials. Anecdotal comments such as these confirm their fears: "I don't
worry about printing extra copies anymore because l am using recycled paper and I'll put
any copies not used in the recycling bin" and "l don’t worry about buying small bottles of
water to carry around because I can recycle them" As a result, environmentalist in some
communities are now beginning to direct more of their efforts to the other two legs of their
"three- legged stool'': "Reduce use" and "Reuse."
CHAPTER SUMARY

Key components of an evaluation and monitoring plan are determined by answers to the
following questions:
• Why are you conducting this measurement?
• What will you measure?
• How will you conduct these measurements?
• When will these measurements be taken?
• How much will it cost?

Reasons why you are measuring will guide your research plan, as methodologies will vary
according to your reason for measurement. Is it to fulfill a grant requirement? To do better
the next time you conduct this same campaign, to (hopefully) get continued, even increased
funding? To help you decide how to prioritize and allocate your resources going forward? Or
to alert you to midcourse corrections you need to make in order or achieve goals?

What you will measure to achieve your evaluation purpose is likely to fall into one or more
of three categories: outputs, outcomes, impacts Output measures report on campaign
activities, outcomes on target audience responses, and impacts on improvements in social
conditions, as a result of targeted behavior changes.
Optional techniques far measurement include surveys that are quantitative, qualitative,
observational, or scientific/technical in nature, as well as ones that use control groups and
rely on records and databases.
You will also determine in this plan timing far evaluations, considering opportunities to
measure prior to the campaign launch, during campaign implementation, and post
campaign.
Finally, you determine costs for your proposed efforts, ones it was suggested should
be weighed in light of potential benefits.
IMPROVING SERVICE DELIVERY AL THE INTERNAL REVENUE SERVICE (2007)

PETE WEBB
Pacific Consulting Group

Purpose

While the mandate of the internal Revenue Service (IRS) is to collect tax revenues
through the administration and enforcement of tax laws, the lRS's mission statement is
"Provide America’s taxpayers top quality service by helping them understand and meet
their tax responsibilities and by applying the tax law with integrity and fairness to all."
Commissioner Mark Everson has said that the lRS's "working equation” for fulfilling its
mission is "Service Plus Enforcement Equals Compliance." The provision of high quality
service to taxpayers is a critical element in maximizing the number of citizens who voluntarily
comply with desired behaviors to complete tax returns correctly and pay the right amount
of taxes owed on time. Failure to do so creates a societal problem in that tax short- falls
either increase the burden on law abiding citizens to make up the difference or chat
important government programs that rely on tax revenues are underfunded or eliminated
altogether.

The lRS provides services to taxpayers through a variety of distribution channels. The
most heavily used includes a nationwide network of about 450 walk-in tax assistance
offices (7 million visits annually), a toll-free 800 number (61 million calls annually), and a
Web site (143 million contacts m 2005). Services most sought by taxpayers include getting
forms and publications, answering tax law questions, getting refund information, tax
return_ preparation assistance, and getting information about a notice received from the
IRS. Despite heavy use of all three-service delivery channels, many taxpayers familiar with a
particular channel are unaware of how another might better serve their needs. In addition,
faced with severe budget constraints, the IRS must improve the efficiency of its services by
cutting back on those that cost the most without reducing the overall level of service to
taxpayers.

Methodology
To help achieve this goal, research was undertaken to determine the specific attributes of
service most important to taxpayers in order to help develop marketing strategies for
migrating taxpayers from more expensive to less expensive service delivery channels.
(Costs per contact in 2005 were $28.73 for tax assistance centers, $19 46 for tell free
assisted, S 71 for toll-free automated, and $.13 for the Web site.) The audience for this
research was all individual taxpayers, numbering approximately 130 million. In addition,
Congress has mandated that particular attention be paid to disadvantaged groups including
low income, handicapped, senior citizen, and limited English proficiency populations to
ensure that strategies developed for the tax-paying population as a whole do not adversely
affect any of these groups.
The survey research method employed was choice based conjoint analysis. This
method is growing rapidly in popularity for realistic portrayal of decisions consume make
among competing alternatives in the marketplace. It presents survey respondents with a
number of hypothetical scenarios, each showing several possible choices of service delivery
channels with different levels of service on several attributes tax payers consider important
in obtaining the service they need. Respondents evaluate each alternative relative to the
others and select the one they would prefer if these were the actual choices they faced
in the marketplace. For example, one scenario might look like this:

Each respondent is asked to evaluate 8-10 of these scenarios, each showing a different
combination of attribute levels for each of the channel choices. While there are literally
thousands of possible combinations of attribute levels, only a small fraction is required
in order to determine exactly what is driving each respondent's choices and what those
choices would be for any possible combination. Output from the conjoint analysis consists
of a numeric rating of the relative importance of each service attribute and the overall
preferences for each channel for any combination of attribute levels. This can be expressed
at the individual respondent level for the respondent group as a whole, or for any particular
subgroup of respondents of interest.
We determined which particular service attributes to include in the analysis through a series
of focus groups with taxpayers from a wide spectrum of geographic and socioeconomic
groups. To ensure that the conjoint cask was not too difficult for respondents, an initial list
of 12-15 attributes was pared to the 4 noted in the example above (research has
shown that more than 4 or 5 attributes in the conjoint task leads to unrealistic simplification
on the part of respondents in order to complete the choice task). For each of these
attributes, three levels were presented across the scenarios respondents evaluated. The
ranges for each attribute were chosen to reflect the lowest Levels taxpayers would consider
acceptable, the highest levels the IRS could realistically hope to achieve, and a level
somewhere in between.
The survey was administered online using respondents recruited from a panel maintained
by a leading market research firm. Either respondents completed the survey online or, for
those without computers, using an interactive television set top box.
Findings

Results of this research proved very favorable to the IRS m terms of its objective of shifting
taxpayers from high cost to low-cost service delivery channels. In particular, the Web
site is preferred over either office visits or assisted telephone calls far most information
seeking interactions. In addition, the telephone is preferred to office visits for most
transactional Interactions.
To achieve the desired migration requires a range of marketing strategies now being
developed and implemented by the IRS The main difference between the results of the
conjoint analysis and the behavior currently observed m the marketplace is that the con-
joint exercise reflects full awareness on the pan of respondents about what each channel
can provide on the attributes taxpayers consider important.
In actuality, awareness of this information is severely limited. For example, many taxpayers
who prefer to go to an lRS office for service have never used the Web or even the toll-
free telephone line and therefore have no idea that they could get the same service they
do in person in many instances much quicker Thus, relative to preferences, the Web
site is currently severely underutilized relative to all other channels and the toll-free
line is underutilized relative to office visits.
ln addition to raising awareness of the capabilities of the Web and the toll-free telephone
line (and especially the automated toll-free service), the IRS is considering the use of
a range of marketing techniques for achieving their service objectives covering all of the
4p’s.
For example.
• Product: Content quality enhancements for the Web site: improved first- contact
resolution for the toll-free line

• Price: Charging a fee (or even limiting access) to higher income taxpayers for interactions
with staff at IRS offices

• Place: Reducing wait times for reaching toll-free representatives (especially during the
January April tax season); closing some IRS offices

• Promotion: Promoting the toll-free line and the Web site at IRS offices; promoting the
Web site while on hold for a toll free phone representative
15.2 Service Delivery Options and Levels of Service

IRS Tax IRS Office Toll Free Web Site Toll Free
Assistance Line
Method Automated Line Assisted
Access Time To reach office To find the right To find the right To speak to rep
and speak to menu choice section
rep 3 minutes 15 minutes 5 minutes
60 minutes

Servicing Once you see To listen to and To read and To get an


Time rep understand understand answer to
5 minutes answer answer question
10 minutes
3 minutes 1 minute
Hours of Regular 24 hours, 7 days 24 hours, 7 days Regular
Availability business hours, business hours
evenings and plus evenings
weekends
Percentage 85% 95% 75% 95%
First contact
Resolution