You are on page 1of 313

Public Policy Analysis

Introduction to the Public Policy


Process

Policy Analysis in the PolicyMaking Process


Policy analysis is the activity of creating

knowledge of and in the policy-making process.


In creating knowledge of policy-making processes
policy analysts investigate the causes,
consequences, and performance of public policies
and programs.
Such knowledge remains incomplete, however,
unless it is made available to policymakers and the
public they are obligated to serve.

The Process of Policy Inquiry


This course provides a methodology for policy

analysis.
Methodology is a system of standards, rules and
procedures for creating, critically assessing, and
communicating policy-relevant knowledge.

Problem solving is a key element of the

methodology of policy analysis


Policy analysis is also a methodology for
formulating problems as a part of a search for
solutions.

The Methodology of Policy


Analysis
The methodology of policy analysis draws

from and integrates elements of multiple


disciplines
Political science
Sociology
Psychology
Economics
Philosophy

The Methodology of Policy


Analysis (contd.)
Policy analysis is partly descriptive, seeking

knowledge about causes and consequences


of public policies.
But, policy analysis is also normative,
creating and critiquing knowledge claims
about the value of public policies for past,
present,and future generations.

The Methodology of Policy


Analysis (contd.)
Policy-relevant knowledge involves choosing

among competing values such as health, wealth,


security, peace, justice, equality, and freedom.
To choose among competing values is not a technical
judgment, but a moral judgment. Thus, policy analysis
is a form of applied ethics.

Policy analysis also seeks to create knowledge that

improves the efficiency of choices among


alternative policies.

The Methodology of Policy


Analysis (contd.)
The methodology of policy analysis aims at

creating, critically assessing, and


communicating policy-relevant knowledge.

The Methodology of Policy


Analysis (contd.)
Knowledge in this context refers to

plausibly true beliefs, as distinguished from


beliefs that are certainly true, or even true
with a particular statistical probability
Statistical evidence performs only a
supplementary or reinforcing role.

The Methodology of Policy


Analysis (contd.)
In short, policy analysis has developed a

core of moderately coherent underlying


theories, a variety of methods that enjoy
reasonably broad assent among
practitioners, a tradition of criticism
directed at political, ideological, and ethical
issues raised by policy analysis, and
systematic as well as anecdotal evidence of
having improved the capacities of clients to
solve problems.

The Methodology of Policy


Analysis (contd.)
A key feature of research and analysis on

social problems over the past 40 or more


years is the growing recognition of
complexity.
The methodological core of policy analysis can
be broadly characterized as a form of critical
multiplism.

The Methodology of Policy


Analysis (contd.)
For critical multiplism, inductive

plausibility, not certainty, is the defining


characteristic of knowledge and a major
standard of success in policy inquiry.
The analyst does not enumerate supporting or
confirming cases but identifies, evaluates, and
eliminates competing explanations.

The Methodology of Policy


Analysis (contd.)
The other major standard is policy

relevance.
Knowledge that assists in formulating and
solving problems, as these problems are
experienced by policymakers and citizens on
whom the policies have an impact, including
the disenfranchised.

The Methodology of Policy


Analysis (contd.)
General guidelines for policy inquiry under

critical multiplism.

Multiple operationalism.
The use of multiple measures of policy constructs.

Multimethod research.
The use of multiple methods to observe policy processes and
outcomes.

Multiple analytic synthesis.


The synthesis and critical assessment of available analyses of
similar policies and programs.

Multivariate analysis.
The inclusion of multiple variables in policy models.

The Methodology of Policy


Analysis (contd.)
General guidelines for policy inquiry under

critical multiplism (contd.)

Multiple stakeholder analysis.


The investigation of interpretive frameworks and perspectives
of multiple policy stakeholders.

Multiple perspective analysis.


The incorporation into policy analysis of multiple perspectives
ethical, political, organizational, economic, social, cultural,
psychological, and technical.

Multimedia communications.
The use by policy analysts of multiple communications media
is essential for ensuring that knowledge is policy relevant.

The Methodology of Policy


Analysis (contd.)
Given typical time and financial constraints,

it is usually impossible to observe all of


these guidelines.
In addition, depending on the stage of the
policy process being analyzed, several
guidelines may not be relevant.
Multiple methods makes analysts less likely
to commit errors resulting from limited
perspectives.

Policy-Relevant Information
The methodology of policy analysis designed to

answer five questions:


What is the nature of the problem?
Policy problems.

What present and past policies have been established to


address the problem, and what are their outcomes?
Policy outcomes.

How valuable are these outcomes in solving the


problem?
Policy performance.

Policy-Relevant Information
The methodology of policy analysis

designed to answer five questions:


What policy alternatives are available to
address the problem, and what are their likely
future outcomes?
Policy futures.

What alternatives should be acted on to solve


the problems?
Policy actions.

Five types of policy relevant


information.
POLICY
PERFORMANCE

POLICY
OUTCOMES

POLICY
PROBLEMS

POLICY
ACTIONS

POLICY
FUTURES

Policy-Analytic Procedures
Methodology of policy analysis incorporates

standards, rules, and procedures.

The standards and rules govern the selection of


procedures and the interpretation of results.

The five procedures common to all human

problem solving:

Definition
Prediction
Prescription
Description
Evaluation

Policy-Analytic Procedures
Five procedures as applied to policy analysis:
Problem structuring (definition)
Yields information about the conditions giving rise to a policy
problem.

Forecasting (prediction)
Supplies information about future consequences of acting on
policy alternatives, including doing nothing.

Recommendation (prescription)
Provides information about the relative value or worth of these
future consequences in solving or alleviating the problem.

Policy-Analytic Procedures
Monitoring (description)
Yields information about the present or past
consequences of acting on policy alternatives.

Evaluation (evaluation)
Provides information about the value or worth of
these consequences in solving or alleviating the
problem.

Policy-Analytic Procedures
The five procedures serve as the means of

organizing particular methods and techniques of


analysis.
Methods are general procedures for producing and
transforming policy relevant information.
Each of the methods is supported by several techniques.

The complete framework for problem-centered

policy analysis combines policy-relevant


information transformed by policy analytic
procedures.

Five policy-analytic procedures

Forecasting

Evaluation
Problem
Structuring
Problem
Structuring

Problem
Structuring

Problem
Structuring
Monitoring

Recommendation

Problem-centered policy analysis


POLICY
PERFORMANCE
Forecasting

Evaluation

Problem
Structuring

POLICY
OUTCOMES

POLICY
PROBLEMS

Problem
Structuring

Problem
Structuring
POLICY
FUTURES

Problem
Structuring
Monitoring

Recommendation
POLICY
ACTIONS

Phases of the policy-making


process
PHASE

CHARACTERISTICS

ILLUSTRATION

AGENDA SETTING

Elected and appointed officials place


problems on the public agenda. Many
problems are not acted on at all, while
others are addressed only after long
delays.

A state legislator and her cosponsor


prepare a bill that goes to the Health and
Welfare Committee for study and
approval. The bill stays in committee and
is not voted on.

POLICY
FORMULATION

Officials formulate alternative policies to


deal with a problem. Alternative policies
assume the form of executive orders, court
decisions, and legislative acts.

A state court considers the use of


standardized achievement tests such as the
SAT on grounds that the tests are biased
against women and minorities.

POLICY ADOPTION

A policy alternative is adopted with the


support of a legislative majority,
consensus among agency directors, or a
court decision.

In Roe v. Wade Supreme Court justices


reach a majority decision that women have
the right to terminate pregnancies through
abortion.

POLICY
IMPLEMENTATION

An adopted policy is carried out by


administrative units which mobilize
financial and human resources to comply
with the policy.

The city treasurer hires additional staff to


ensure compliance with a new law which
imposes taxes on hospitals that no longer
have tax-exempt status.

POLICY
ASSESSMENT

Auditing and accounting units in


government determine whether executive
agencies, legislatures, and courts are in
compliance with statutory requirements of
a policy and achieving its objectives.

The General Accounting Office monitors


social welfare programs such as
Temporary Assistance to Needy Families
(TANF) to determine the scope of welfare
fraud.

Policy analytic procedures and


phases of policy-making
Problem
Structuring

Forecasting

Recommendation

AGENDA SETTING

POLICY
FORMULATION

POLICY ADOPTION

Monitoring

POLICY
IMPLEMENTATION

Evaluation

POLICY
ASSESSMENT

Problem Structuring
Problem structuring can supply policy-relevant

knowledge that challenges the assumptions


underlying the definition of problems reaching the
policy-making process through agenda-setting.
Problem structuring can assist in discovering
hidden assumptions, diagnosing causes, mapping
possible objectives, synthesizing conflicting
views, and designing new policy options.

Forecasting
Forecasting can provide policy-relevant knowledge

about future states of affairs which are likely to occur


as a consequence of adopting alternatives, including
doing nothing, that are under consideration at the
phase of policy formulation.
Forecasting can examine plausible, potential, and
normatively valued futures, estimate the consequences
of existing and proposed policies, specify probable
future constraints on the achievement of objectives,
and estimate the political feasibility (support and
opposition) of different options.

Recommendation
Recommendation yields policy-relevant

knowledge about the benefits and costs of


alternatives the future consequences of which have
been estimated through forecasting, thus aiding
policymakers in the policy adoption phase.
Recommendation helps estimate levels of risk and
uncertainty, identify externalities and spillovers,
specify criteria for making choices, and assign
administrative responsibility for implementing
policies.

Monitoring
Monitoring provides policy-relevant knowledge

about the consequences of previously adopted


policies, thus assisting policy-makers in the policy
implementation phase.
Monitoring helps assess the degree of compliance,
discover unintended consequences of policies and
programs, identify implementational obstacles and
constraints, and locate sources of responsibility for
departures from policies.

Evaluation
Evaluation yields policy-relevant knowledge

about discrepancies between expected and actual


policy performance, thus assisting policymakers in
the policy assessment phase of the policy-making
process. Evaluation not only results in
conclusions about the extent to which problems
have been alleviated; it also may contribute to the
clarification and critique of values driving a
policy, aid in the adjustment or reformulation of
policies, and establish a basis for restructuring
problems.

Public Policy Analysis


The Functions of Policy Argument

Introduction
Policy argumentation is central to policy

analysis and the policy-making process.


The analysis and evaluation of policy
argumentation are central to the process of
critical thinking.

The Structure of Policy Arguments


A policy argument is the product of

argumentation, which is the process.


In real-life policy settings, arguments are complex
and prone to misunderstanding.
To minimize misunderstanding, we use the
structural model of argument developed by
Stephen Toulmin, which is designed to investigate
structures and processes of practical reasoning.

The Structure of Policy Arguments


The conclusions of practical arguments are

always uncertain, as are the reasons and


evidence that lead to these conclusions.

The Structure of Policy Arguments


Types of knowledge claims.
A knowledge claim is the conclusion of a
policy argument.
Three types of knowledge claims:
Designative: questions of fact. What are the
observed outcomes of a policy and why did they
occur?
Evaluative: questions of value. Was the policy
worthwhile?
Advocative: questions of right action. Which policy
should be adopted?

The Structure of Policy Arguments


Types of knowledge claims (contd.).
Policy arguments contain six elements:
Information (I), Claim (C), Warrant (W), Backing (B), Rebuttal (R),
Qualifier (Q).

The first four of these elements are present in every policy


argument.
The claim C is the conclusion or output of an argument,
which is supported by policy-relevant information I, which
is the beginning or input of the argument.
The warrant W is the justification, or reason, for concluding
C from I.
The qualifier Q indicates that C has a given truth or
plausibility.

The Structure of Policy Arguments


Example:
The Senator supports the privatization of the
federal highway system, which have significant
gains in efficiency and a reduction in taxes.
Considering that the privatization of public
services has been successful in other areas, this
is definitely a no brainer. Besides this is the
same conclusion as a panel of experts on
privatization.

The Structure of Policy Arguments


The underdetermination of conclusions by

information.
Policy-relevant information does not fully determine
the conclusions of policy arguments. Information does
not speak for itself.
Identical information can and often does lead to
different conclusions, which we call policy claims to
emphasize the fallible and indeterminate character of
arguments.

The Structure of Policy Arguments


Example: Policy-relevant information from the Coleman

Report Black students attending primarily black schools


had lower achievement test scores than black students
attending primarily white schools.
Designative claim and qualifier: Since schools in large
urban areas are primarily black, the hopes of blacks for
higher education achievement [simply] cannot be realized.
Evaluative claim and qualifier: The Coleman Report is
[clearly] a racist document based on ethnically biased
achievement tests.
Advocative claim and qualifier: [There is no question] that
a national policy of compulsory school busing ought to be
adopted to achieve integrated schools.

The Structure of Policy Arguments


Warrants and rebuttals.
Although each of the claims about the Coleman
report begins with the same information, very
different conclusions are drawn.
Differences are due not to the information, but
to the role of the warrants in justifying
(plausibility or implausibility) the claims on the
basis of the information supplied.

Modes of Policy Argumentation


Modes of policy argumentation are the characteristic

routes followed by information as it is transformed into


policy claims.
The several different modes of argument involve reasoning
from authority, method, generalization, classification,
intuition, cause, sign, motivation, analogy, parallel case,
and ethics.
Each of the eleven modes of argument has a different type
of warrant, and multiple modes can be found in any policy
argument.
The warrants are the reasons offered by the proponent or
opponent of a policy to justify a claim, or inference, based
on the information supplied.

Modes of Policy Argumentation


Authority
Reasoning from authority is based on warrants
having to do with the achieved or ascribed
statuses of producers of policy-relevant
information. For example, experts, insiders,
scientists, specialists, gurus, power brokers.
Footnotes and references are disguised
authoritative arguments.

Argumentation from Authority

Modes of Policy Argumentation


Method
Reasoning from method is based on warrants
about the approved status of methods or
techniques used to produce information. The
focus is on the achieved or ascribed status or
"power" of procedures, rather than persons.
Examples include approved statistical,
econometric, qualitative, ethnographic, and
hermeneutic methods and techniques.

Argumentation from Method

Modes of Policy Argumentation


Generalization
Reasoning from generalization is based on similarities
between samples and populations from which samples
are selected. Although samples can be random,
generalizations can also be based on qualitative
comparisons. The assumption is that what is true of
members of a sample will also be true of members of
the population not included in the sample. For example,
random samples of n 30 are taken to be representative
of the (unobserved and often unobservable) population
of elements from which the sample is drawn.

Argumentation from Generalization

Modes of Policy Argumentation


Classification
Reasoning from classification has to do with
membership in a defined class. The reasoning is
that what is true of the class of persons or
events described in the warrant is also true of
individuals or groups which are members of the
class described in the information. An example
is the untenable ideological argument that
because a country has a socialist economy it
must be undemocratic, because all socialist
systems are undemocratic.

Argumentation from Classification

Modes of Policy Argumentation


Cause
Reasoning from cause is about the activity of
generative powers ("causes") and their consequences
("effects"). For example, a claim may be made based on
general propositions, or laws, of economics that state
invariant relations between cause and effect. Other
causal claims are based on observing conditions that
must be satisfied to infer that a policy has a specified
effect. Most argumentation in the social and natural
sciences is based on reasoning from cause.

Argumentation from Cause

Argumentation from Cause

Modes of Policy Argumentation


Sign
Reasoning from sign is based on signs, or indicators,
and their referents. The presence of a sign indicates the
presence of an event or condition, because the sign and
what it refers to occur together. Examples are indicators
of institutional performance such as "organizational
report cards" and "benchmarks," or indicators of
economic performance such as "leading economic
indicators." Signs are not causes, because causality
must satisfy additional requirements not expected of
signs.

Argumentation from Sign

Modes of Policy Argumentation


Motivation
Reasoning from motivation is based on the
motivating power of goals, values, and inten
tions in shaping individual and collective
behavior. For example, a claim that citizens will
support the strict enforcement of pollution
standards might be based on reasoning that,
since citizens are motivated by the desire to
achieve the goal of clean air and water, they
will act to offer their support.

Argumentation from Motivation

Modes of Policy Argumentation


Intuition
Reasoning from intuition is based on the
conscious or preconscious cognitive, emotional,
or spiritual states of producers of policyrelevant information. For example, the
awareness that an advisor has some special
insight, feeling, or "tacit knowledge" may serve
as a reason to accept his judgment.

Argumentation from Intuition

Modes of Policy Argumentation


Analogy-Metaphor
Reasoning from analogies and metaphors is based on
similarities between relations found in a given case and
relations characteristic of a metaphor, analogy, or
allegory. For example, the claim that government
should "quarantine" a country by interdicting illegal
drugswith the illegal drugs seen as an "infectious
disease"is based on reasoning that, since quarantine
has been effective in cases of infectious diseases,
interdiction will be effective in the case of illegal drugs.

Argumentation from Analogy

Modes of Policy Argument


Parallel Case
Reasoning from parallel case is based on
similarities among two or more cases of policy
making. For example, a reason that a local
government should strictly enforce pollution
standards is that a parallel policy was
successfully implemented in a similar local
government elsewhere.

Argumentation from Parallel Case

Modes of Policy Argument


Ethics
Reasoning from ethics is based on the rightness or
wrongness, goodness or badness, of policies or their
consequences. For example, policy claims are
frequently based on moral principles stating the
conditions of a "just" or "good" society, or on ethical
norms prohibiting lying in public life. Moral principles
and ethical norms go beyond the values and norms of
particular individuals or groups. In public policy, many
arguments about economic benefits and costs involve
unstated or implicit moral and ethical reasoning.

Ethical Argumentation

Systems of Argumentation
Completeness. Elements of an argument should

comprise a genuine whole that encompasses all


appropriate considerations. For example, the
plausibility of arguments about the effects of a
policy depends on whether such arguments
encompass a full range of plausible rival
explanations similar in form and content to classes
of rival hypotheses (threats to validity) developed
in the tradition of quasi-experimentation.

Systems of Argumentation
Consonance. Elements of an argument

should be internally consistent and


compatible. For example, ethical arguments
concerning the justice or fairness of a policy
are plausible to the degree that they
incorporate a system of internally and
externally consistent ethical hypotheses.

Systems of Argumentation
Cohesiveness. Elements of an argument

should be operationally connected. For


example, the plausibility of an ethical
argument depends on whether responses to
several levels of descriptive and valuative
questions levels ranging from verification
and validation to vindication are
operationally linked.

Systems of Argumentation
Functional regularity. Elements of an argument

should conform to an expected pattern. For


example, statistical arguments that offer estimates
of parameters of unobserved (and often
unobservable) populations are plausible to the
degree that patterns in the sample and the
population from which it is drawn are functionally
regular or uniform, not irregular, based on sample
data and background knowledge.

Class Example: Excerpt from Bush


Speech on Iraq
It is clear that we need to change our strategy in Iraq. So

my national security team, military commanders, and


diplomats conducted a comprehensive review. We
consulted Members of Congress from both parties, allies
abroad, and distinguished outside experts. We benefited
from the thoughtful recommendations of the Iraq Study
Group a bipartisan panel led by former Secretary of State
James Baker and former Congressman Lee Hamilton. In
our discussions, we all agreed that there is no magic
formula for success in Iraq. And one message came
through loud and clear: Failure in Iraq would be a disaster
for the United States.

Class Example: Excerpt from Bush


Speech on Iraq
The consequences of failure are clear: Radical Islamic

extremists would grow in strength and gain new recruits.


They would be in a better position to topple moderate
governments, create chaos in the region, and use oil
revenues to fund their ambitions. Iran would be
emboldened in its pursuit of nuclear weapons. Our enemies
would have a safe haven from which to plan and launch
attacks on the American people. On September the 11th,
2001, we saw what a refuge for extremists on the other
side of the world could bring to the streets of our own
cities. For the safety of our people, America must succeed
in Iraq.

Class Example: Excerpt from Bush


Speech on Iraq
Many are concerned that the Iraqis are becoming too

dependent on the United States and therefore, our policy


should focus on protecting Iraq's borders and hunting
down al Qaeda. Their solution is to scale back America's
efforts in Baghdad or announce the phased withdrawal of
our combat forces. We carefully considered these
proposals. And we concluded that to step back now would
force a collapse of the Iraqi government, tear that country
apart, and result in mass killings on an unimaginable scale.
Such a scenario would result in our troops being forced to
stay in Iraq even longer, and confront an enemy that is
even more lethal.

Public Policy Analysis


Problem Structuring

Nature of Policy Problems


Problem structuring, which is a continuously

recurring phase of policy inquiry in which analysts


search among competing problem formulations of
different stakeholders, is no doubt the most
important activity performed by policy analysts.
It is so important because policy analysts seem to
fail more often because they solve the wrong
problem than because they get the wrong solution
to the right problem.

Nature of Policy Problems


Beyond problem solving.
Policy analysis is a dynamic, multilevel process
in which methods of problem structuring take
priority over methods of problem solving (see
figure).

Priority of Problem Structuring


in Policy Analysis
Problem
Sensing

PROBLEM
SITUATION

Problem
Structuring

Problem
Dissolving

POLICY
PROBLEM

NO

Problem
Unsolving

RIGHT
PROBLEM?

YES
Problem
Solving

POLICY
SOLUTION

YES

POLICY
SOLUTION?

Problem
Resolving

Nature of Policy Problems


Beyond problem solving.
Distinctions among problem-related processes.
Problem sensing versus problem structuring.
Problem structuring versus problem solving.
Problem resolving versus problem unsolving and problem
dissolving.

Characteristics of problems.
Interdependence of policy problems (policy messes).
Analytic versus holistic approaches.

Subjectivity of policy problems.


Artificiality of policy problems.
Dynamics of policy problems.

Nature of Policy Problems


Characteristics of problems (contd.)
In short, systems of problems (messes) cannot be
decomposed into independent subsets without running
the risk of producing the right solution to the wrong
problem.
The whole is greater (qualitatively different) than the
sum of its parts.
A recognition of the interdependence, subjectivity,
artificiality, and dynamics of policy problems alerts us
to the possible unanticipated consequences that may
follow from policies based on the right solution to the
wrong problem.

Nature of Policy Problems


Problems versus issues.
If policy problems are really systems of
problems, then policy issues are equally
complex.
Policy issues reflect not only conflict over
courses of actions, but over definitions of the
problem.

Nature of Policy Problems


Major Issues

Secondary Issues

Functional Issues

Minor Issues

Nature of Policy Problems


Problems versus issues.
Major issues.
Those encountered at highest levels of government
within and between federal, state, and local
jurisdictions.
Involve questions of agency mission.

Secondary issues.
Located at the level of agency programs at the
federal, state, and local levels.
The setting of program priorities and the definition
of target groups and beneficiaries.

Nature of Policy Problems


Problems versus issues.
Functional issues.
Located at both the program and project levels.
Involve questions of budget, finance, and
procurement.

Minor issues.
Located at the level of specific projects.
Involve questions of personnel, staffing, employee
benefits, vacation times, working hours, and
standard operating procedures.

Nature of Policy Problems


Problems versus issues.
Strategic policies are policies where the
consequences are relatively irreversible.
Operational polices are policies where the
consequences are relatively reversible.

Nature of Policy Problems


Three classes of policy problems.
Well-structured.
Prototype: completely computerized decision
problems.

Moderately structured.
Prototype: prisoners dilemma.

Ill-structured.
Prototype: most important problems.

Nature of Policy Problems


STRUCTURE OF PROBLEM
Moderately
Structured

ELEMENT

Well structured

Ill Structured

Decision maker(s)

One or few

One or few

Many

Alternatives

Limited

Limited

Unlimited

Utilities (values)

Consensus

Consensus

Conflict

Outcomes

Certainty or risk

Uncertainty

Unknown

Probabilities

Calculable

Incalculable

Incalculable

Problem Structuring in Policy


Analysis
The requirements for solving ill-structured

problems demand that the analyst take an


active role in defining the problem.

Problem Structuring in Policy


Analysis
Creativity in problem structuring.
The product of the analysis is sufficiently novel that
most people could not or would not have arrived at the
same solution;
The process of analysis is sufficiently unconventional
that it involves the modification or rejection of previous
accepted ideas;
The process of analysis requires sufficiently high
motivation and persistence that analysis takes place
with high intensity or over long periods of time;

Problem Structuring in Policy


Analysis
Creativity in problem structuring (contd.).
The product of analysis is regarded as valuable
by analysts, policymakers, and other
stakeholders, since it provides an appropriate
solution to the problem; And.
The problem initially posed is so ambiguous,
value, and ill defined that part of the task is to
formulate the problem itself.

Problem Structuring in Policy


Analysis
Phases of problem structuring.
Problem search.
Problem definition.
Problem specification.
Problem sensing.

Problem Structuring in Policy


Analysis
METAPROBLEM
Problem
Definition

Problem
Search

PROBLEM
SITUATION

SUBSTANTIVE
PROBLEM

Problem
Sensing

Problem
Specification
FORMAL PROBLEM

Problem Structuring in Policy


Analysis
Errors of the third type (EIII).
How well do the substantive and formal problems
correspond to the original problem situation?
If most problem situations are messes, then models should
reflect complexity.

Types of errors.
Type I rejecting the null hypothesis when it is true.
Type II accepting the null hypothesis when it is false.
Type III solving the wrong problem.

Types of Policy Models


Policy models are simplified representations

of selected aspects of a problem situation


constructed for specific purposes.
By definition, they are artificial constructs.

Descriptive models.
The purpose of descriptive models is to explain
and/or predict the causes and consequences of
policy choices.
Used to monitor the outcomes of policy actions and
to forecast performance.

Types of Policy Models


Normative models.
Explain and predict, but also to provide rules
and recommendations for optimizing some
utility or value.
Example: compound interest.

S n (1 r ) S 0
n

Forms of Policy Models


Verbal models.
Expressed in everyday language.
The equivalent of substantive problems.
Limitation: the reasons for recommendations and
predictions may be hidden.
Symbolic models.
Use mathematical symbols to describe relationships
among key variables believed to characterize a
problem. The premises must be made explicit.

Y a bX

Forms of Policy Models


Table 1. Mean education and mean income
Birmingham, Alabama neighborhoods
50000

Adjusted income 2000

40000

30000

20000

10000

Rsq = 0.6369

0.0

.1

.2

.3

.4

.5

.6

.7

.8

Proportion with high school or higher

.9

1.0

Forms of Policy Models


Procedural models.
Represent dynamic relationships among
variables believed to characterize a policy
problem.
Example:
Decision tree.

Methods of Problem Structuring


Boundary analysis (used to estimate boundaries of

metaproblem).

Saturation sampling.
Elicitation of problem representations.
Boundary estimation.

Classificational analysis.
Uses logical division and logical classification.
Criteria.

Substantive relevance.
Exhaustiveness.
Disjointness.
Consistency.
Hierarchical distinctiveness.

Methods of Problem Structuring


Hierarchy analysis.
Possible clauses, plausible causes, and
actionable causes.
Uses same rules as classificational analysis.
Synectics.
Personal analogies.
Direct analogies.
Symbolic analogies.
Fantasy analogies.

Methods of Problem Structuring


Brainstorming.
Groups should be composed of knowledgeable
subjects.
Idea generation and idea evaluation should be kept
separate.
Atmosphere should be open and permissive.
Idea-evaluating should only begin after idea-generating
has ceased.
At the end of idea-evaluating, ideas should be
prioritized and incorporated into a proposal that
contains a conceptualization of the problem and its
potential solutions.

Methods of Problem Structuring


Multiple perspective analysis.
Technical perspective.
Organizational perspective.
Personal perspective.
Assumptional analysis.
Stakeholder identification.
Assumption surfacing.
Assumption challenging.
Assumption pooling.
Assumption synthesis.

Methods of Problem Structuring


Argumentation mapping.
Assessing probability and plausibility of policy
argument warrants.

Policy Restraint and Barriers to


Change
A rationale for stability.
Deliberation and public participation are at least as
important as rapid and efficient policymaking.
Rapid policy change can happen with the right
combination of political factors, but during normal
periods, minorities can block policy change and the
system preserves their right to do so.
Policy does not change rapidly because most of the
public does not support such change except under rare
circumstances.

Policy Restraint and Barriers to


Change
Fragmentation.
Fragmentation is a double-edge sword.
On the one hand, it requires multiple serial
majorities to promote change.
On the other hand, it offers multiple points of access
to the policy process.

Dimensions of fragmentation.
Separation of powers.
Division of powers (federalism).

Policy Restraint and Barriers to


Change
Table 2.2. The Balance of Power
Function

Congress

President

Courts

Legislative

Make laws

Recommend laws; veto


laws; make regulations
that have the force of
law.

Review laws to
determine legislative
intent, new
interpretations

Executive

Override vetoes;
legislative vetoes of
regulations

Enforce and implement


laws.

Review executive acts;


restrain executive
actions.

Judicial

Impeach judges and the


president, call witnesses
to hearings, set judicial
jurisdiction.

Pardon criminals,
nominate judges

Interpret laws.

Note: The primary function of each branch is indicated in the boxes with the diagonal lines.

Public Policy Analysis


Forecasting Policy Futures

Forecasting in Policy Analysis


Forecasting is a procedure for producing factual

information about future states of society on the


basis of prior information about policy problems.
Forms of forecasting.
Projection.
A forecast based on the extrapolation of current and historical
trends into the future.

Prediction.
A forecast based on explicit theoretical assumptions.

Conjecture.
A forecast based on informed or expert judgments about future
states of society.

Forecasting in Policy Analysis


Aims of forecasting.
Forecasts provide information about future
changes in policies and their consequences.
Forecasting permits greater control through
understanding past policies and their
consequences, implying that the future is
determined by the past.
Forecasting also enables us to shape the future
in an active manner, irrespective of what has
happened in the past.

Forecasting in Policy Analysis


Limitations of forecasting.
Forecast accuracy.
Recent simple forecasting models have had huge errors in
recent years (as much as 50%) in econometric forecasting.

Comparative yield.
Both simple and complex theoretical models have been no
more accurate than simple extrapolative models and informed
expert judgment.

Context.
Institutional (nonprofit more accurate than business and
government).
Temporal (long-term less accurate than short-term).
Historical (modern complexity reduces accuracy).

Assumption drag.

Forecasting in Policy Analysis


Types of futures.
Potential futures.
Future societal states that may occur.

Plausible futures.
Future states that, on the basis of assumptions about causation
in nature and society, are believed to be likely if policymakers
do not intervene to redirect the course of events.

Normative futures.
Potential and plausible futures which are consistent with an
analysts conception of future needs, values, and opportunities.
The specification of normative futures narrows the range of
potential and plausible futures, thus linking forecasts to
specific goals and objectives.

Forecasting in Policy Analysis


Table 1. Contrast between goals and objectives
CHARACTERISTIC

GOALS

OBJECTIVES

Broadly stated (. . . To
upgrade the quality of
health care)

Concrete (. . . Increase
the number of physicians
by 10 percent)

Definition of terms

Formal ( . . . The quality


of health care refers to
the accessibility of
medical services)

Operational ( . . . The
quality of health care
refers to the number of
physicians per 100,000
persons . . .)

Time period

Unspecified ( . . . In the
future)

Specified ( . . . In the
period 2006-2016)

Nonquantitative
(adequate health care
insurance)

Frequently quantitative
( . . . The number persons
covered per 1,000
persons)

Broadly defined ( . . .
People in need of care)

Specifically defined ( . . .
Families with incomes
below $19,000)

Specification of purposes

Measurement procedure

Treatment of target groups

Forecasting in Policy Analysis


Sources of goals, objectives, and alternatives.
Authority.
Insight.
Method.
Scientific theories.
Motivation.
Parallel case.
Analogy.
Ethical systems.

Approaches to Forecasting
Approaches to forecasting.
Decide what to forecast, or determine the object
of the forecast.
Decide how to make the forecast, or select one
or more bases for the forecast.
Choose techniques that are most appropriate for
the object and base selected.

Approaches to Forecasting
Objects.
The object of a forecast is the point of reference
of a projection, prediction, or conjecture.
Objects of forecasting.

Consequences of existing policies.


Consequences of new policies.
Contents of new policies.
Behavior of policy stakeholders.

Approaches to Forecasting
Bases.
The basis of a forecast is the set of assumptions
or data used to establish the plausibility of
estimates of consequences of existing or new
policies, the content of new policies, or the
behavior of policy stakeholders.

Approaches to Forecasting
Bases (contd.).
Bases for forecasts.
Trend extrapolation (projection).
The extension into the future of trends observed in the
past.
Based on inductive logic.
Theoretical assumptions (prediction).
Systematically structured and empirically testable sets of
laws or propositions that make predictions about the
occurrence of one event based on another.
Based on deductive logic.
Informed judgments (conjecture).
Knowledge based on experience and insight, rather than
inductive or deductive reasoning.
Based on retroductive logic.

Methods and Techniques of


Forecasting
APPROACH

BASIS

APPROPRIATE
TECHNIQUES

PRODUCT

Extrapolative
forecasting

Trend extrapolation

Classical time-series analysis


Linear trend estimation
Exponential weighting
Data transformation
Catastrophe methodology

Projections

Theoretical forecasting

Theory

Theory mapping
Causal modeling
Regression analysis
Point and interval estimation
Correlational analysis

Predictions

Judgmental forecasting

Informed judgment

Conventional Delphi
Policy Delphi
Cross-impact analysis
Feasibility assessment

Conjectures

Methods and Techniques of


Forecasting
Classical time-series analysis.
Time series are made up four components.

Secular trend.
Seasonal variation.
Cyclical fluctuations.
Irregular movements.

Linear trend estimation.


Nonlinear time series.

Oscillations.
Cycles.
Growth curves.
Decline curves.
Catastrophes.

Time Series Example

Methods and Techniques of


Forecasting
Linear forecasting example.
Major disaster data set, 1953-2007.
Disaster Declarations = 9.851 + .803 * years; R2 = .592.
Where 1953 is set to zero.

1953 1971: Disaster Declarations = 13.516 + 0.340 * years;


R2 = .103.
1972 - 1988: Disaster Declarations = 71.419 - 1.551 * years;
R2 = .497.
1989 - 2007: Disaster Declarations = -9.368 + 1.281* years; R 2
= .329.
Projection for 2008 (55) using whole data set: 54.0.
Projection for 2008 (55) using post-1988 set: 61.1.

Methods and Techniques of


Forecasting
Theoretical forecasting.
Theory mapping.
Types of causal arguments.
Convergent arguments.
Divergent arguments.
Serial arguments.
Cyclic arguments.
Procedures for uncovering the structure of an argument.
Separate and number each assumption.
Underline the words that indicate claims (therefore,
thus, hence).
When specific words are omitted, but implied, supply the
appropriate logical indicators in brackets.
Arrange number assumptions and claims in an arrow
diagram that illustrates the causal argument or theory.

Methods and Techniques of


Forecasting
Example for theory mapping from Daniels

and Clark-Daniels.
http://www.csub.edu/~rdaniels/Pages%20from
%20DanielsClarkDanielsIJMED.pdf
.

Methods and Techniques of


Forecasting
Causal modeling and regression analysis.
The creation of theoretical models on the basis
of previous research and logic.
The testing of these models with causal analysis
methods such as path analysis and structural
equation modeling.
Example:
http://www.csub.edu/~rdaniels/Model%20of%20Pre
sidential%20Disaster%20Decision-Making.xls
.

Methods and Techniques of


Forecasting
Judgmental forecasting.
Delphi technique.
Principles.

Selective anonymity.
Iteration.
Informed multiple advocacy.
Polarized statistical response.
Structured conflict.
Computer conferencing.

Methods and Techniques of


Forecasting
Judgmental forecasting (contd.).
Delphi technique (contd.).
Steps.
Issue specification.
Selection of advocates.
Questionnaire design.
Forecasting items (probability of occurrence).
Issue items (ranking of importance).
Goal items (desirability and feasibility).
Option items (alternatives).
Analysis of first-round results.
Development of subsequent questionnaires.
Organization of group meetings.
Preparation of final report.

Methods and Techniques of


Forecasting
Delphi article.
http://www.csub.edu/~rdaniels/ch3b1.pdf.

The Public Policy-Making


Process
Official and Unofficial Actors and
Their Roles in Public Policy

Introduction
Political science traditions.
Institutionalism focus on texts of constitutions, laws,
and other written statements of policies and the
relationships between formal government institutions.
Behaviorism focus on political motivations of
individuals, acting singly and in groups, often through
polling, game theory, and statistical techniques.
Neo-institutionalism focus on organizations and
systems in which individuals interact and achieve
political and policy goals through explicit or implicit
rules and operating procedures.

Introduction
Main categories of actors in the policy

process.
Official actors statutory or constitutional
responsibilities.
Legislative, executive, and judiciary.

Unofficial actors participation with no


explicit legal authority.
Interest groups, media.

Legislatures
First listed branch in the federal and most

state constitutions.
Source of considerable research.

Primary function is lawmaking. Number of

bills and resolutions gives some idea of how


busy legislatures are.

Legislatures

Jurisdiction

House or
Assembly
Bills

House or
Assembly
Concurrent
Resolution
s

House or
Assembly
Joint
Resolution
s

Senate
Bills

Senate
Concurrent
Resolution
s

Senate
Joint
Resolution
s

U.S.
Congress
(20012002)

5,767

521

125

3,181

160

53

California
Legislature
(20012002)

3.283

264

66

2,277

114

56

Legislatures
Burden eased by staff.
Bills sifted by committee structure at both

the federal and state level.


Committee chairs wield significant power.
Most bills fail to move past their first
committee hurdles because they are largely
symbolic gestures.

Legislatures
Other critical functions performed by legislators

that affect public policy.

Casework activities to help constituents with


government agencies or to gain a privilege or benefit.
Supports reelection.

Oversight Monitor the implementation of public


policy.
Government Accountability Office www.gao.gov. Studies
public programs and makes recommendations to improve
efficiency, effectiveness, and accountability.
Public hearings.
Help understand issues.
Reveal shortcomings in current policies.
Make political capital.

Legislative Organization

Legislative Organization
California process.
http://www.leginfo.ca.gov/bil2lawx.html.

Legislative Organization
Most of the critical work on public policy is done

in committees, which review legislation, propose


and vote on amendments, and, in the end, decide
whether a bill will die at the committee level or be
elevated for consideration by the full body.
One of the most critical elements of legislative
organization is the organization on party lines.

Legislature Critiques of Public


Policy Process
Many people argue that legislatures are out

of touch with the people.


To understand why legislatures work as
they do, you need to understand two
elements of the legislature: the nature of the
members of the body and the organization
and nature of the branch itself.

Legislature Critiques of Public


Policy Process
The primary goal of the typical legislator is

reelection. Casework allows legislators to please


voters.
Home style and hill style.

Legislatures are decentralized institutions,

especially Congress.
Committees and subcommittees.
Decentralization and centralization of party leadership.
Issue networks and policy subsystems.

Legislatures Implications for


Policy Making
Decentralization and casework focus makes

complex and change-oriented legislation


difficult to pass.

The Executive Branch


For the sake of discussion, the executive branch

can be considered in two parts: the administration,


staff, and appointees; and the bureaucracy.
Advantages of an elected executive in the policy
process.

Veto power.
Unitary branch of government.
Media and public attention.
Informational advantage over the legislature.

The Executive Branch


Elected executive limitations.
Power to persuade.
The size of the Executive Office of the
President.
Elected executives focus on agenda-setting.

Administrative Agencies and


Bureaucrats
Characteristics of bureaucracy.
Fixed and official jurisdictional areas.
Hierarchical organization.
Written documentation.
Expert training of staff.
Career, full-time occupation.
Standard operating procedures.
Key complaints about bureaucracy.
Size.
Red tape.

Administrative Agencies and


Bureaucrats
What Do Government Agencies Do?
Government agencies provide services that are
uneconomical for the private sector (public
goods free-rider problem).
Public goods are indivisible and nonexclusive.

Complaints tend to focus on speed, efficiency,


and effectiveness of public service delivery.

Administrative Agencies and


Bureaucrats
Bureaucracy and the problem of accountability.
The key problem is the question of accountability.
Most public employees are appointed on merit, not
accountability to elected officials.
Early thinking focused on separation of politics and
administration.
Modern thinking: Agency decisions are political and in
the realm of administrative discretion.
Problem: no single, agreed-upon definition of the
public interest.
Administrative discretion: ability to make decisions
with minimal interference.

The Courts
The ability to interpret legislative and

executive actions: judicial review.


Courts are the weakest because their
authority rests on the legitimacy of the law
and their ability to argue their case.
Legislatures and executives initiate public
policy, while courts react to the practical
effects of such policies.

Unofficial Actors and Their Roles in


Public Policy
Individual citizens.
Low political participation.
Voting.
Other forms of participation: campaigning,
contacting, etc.

Despite this, citizens can be mobilized:


Recall election in California.

Generally speaking, individuals want the most


services for ourselves while paying the least
taxes for those services.

Unofficial Actors and Their Roles in


Public Policy
Interest groups.
Interest groups have been part of the political
scene since the founding.
Madison and the dangers of faction.

Since the 1960s the number of groups has


greatly expanded.
Transportation, mass communication, expansion of
government.

Few legal barriers to the creation of groups.

Unofficial Actors and Their Roles in


Public Policy
Interest groups.
The power of interest groups varies.
Knowledge, money, information.
Group size, peak associations.
Intensity, direct economic interest, ideological
commitment.
Social movements (combinations of interest
groups).

Unofficial Actors and Their Roles in


Public Policy
Types of interest groups.
Institutional versus membership groups.
Economic (private) versus public interest versus
ideological groups.
Benefits, free-rider problems.

Activities of interest groups.


Lobbying.
Campaign contributions.
Access (well-off).
Mass mobilization, protest, and litigation.
Riots and protest marches.

Unofficial Actors and Their Roles in


Public Policy
Political parties.
Functions.

Voting cues.
Transmission of political preferences.
Creation of packages of policy ideas.
Organization of the legislative branch.

Think tanks and other research organizations.


Brookings, Cato, Urban Institute, Rand, American
Enterprise Institute.
Ideological, scholarly, and methodological distinctions.

Unofficial Actors and Their Roles in


Public Policy
Communications media.
The news media are important actors in the policy
process.
Newspapers National versus regional versus local.
TV is the central news medium. Older population, networks;
younger population, cable news.
Entertainment programming can be equally important.
Movies, t.v., videogames.

Medias primary function in policy process is agendasetting. Media coverage correlates with institutional
attention.

Unofficial Actors and Their Roles in


Public Policy
Communications media.
News media are not just passive actors.

Interest try to arouse media focus.


Time and space constraints require discretion.
Profit-driven businesses.
Competitive biases of news gathering: dramatic and
narrative qualities of the story.

Unofficial Actors and Their Roles in


Public Policy
Subgovernments, issue networks, and domains.
Policy domain is the substantive area of policy over
which participants in policy-making compete and
compromise.
The political culture and legal environment influence
the domains.
Policy community inside the domain consists of the
actors actively involved in policy making in that
domain.
Iron triangles one way of organizing the policy community.
Issue networks may be more accurate description.

Unofficial Actors and Their Roles in


Public Policy
Subgovernments, issue networks, and domains.
Prying open policy networks (major corporate interests
usually dominate).
But, policy change is possible by prying open a
domain.

Focusing events.
Social movements and mobilization.
Exploiting the decentralization of American government.
Going public.

Public Policy Analysis


Recommending Policy Actions.

Recommendation in Policy
Analysis
The policy-analytic procedure of

recommendation enables analysts to


produce information about the likelihood
that future courses of action will result in
consequences that are valuable to some
individual, group, or society as a whole.

Recommendation in Policy
Analysis
The procedure of recommendation involves

the transformation of information about


policy futures into information about policy
actions that will result in valued outcomes.
Policy recommendations are normative
(advocative) rather than empirical
(descriptive) or evaluative.

Recommendation in Policy
Analysis
Characteristics of advocative claims.
Actionable: Advocative claims focus on actions that
make be taken to resolve a policy problem.
Prospective: Advocative claims occur prior to the time
that actions are taken.
Value laden: Advocative claims require both that
actions have the predicted consequences, but also that
those consequences have value for society.
Ethically complex: Advocative claims can be intrinsic
(valued as ends in themselves) or extrinsic (valued
because they will produce some other value).

Recommendation in Policy
Analysis
Simple model of choice.
Advocative claims are only possible when the
analyst is confronted by a situation of choice
between two or more alternatives.
Simple model.
The definition of a problem requiring action.
The comparison of consequences of two or more
alternatives to resolve the problem.
The recommendation of the alternative that will
result in a preferred outcome.

Recommendation in Policy
Analysis
Simple model of choice.
A1

O1

A2

O2

O1

>

O2
A1

Conditions for choice.


Single decision maker.
Certainty.
Immediacy of consequences.

Recommendation in Policy
Analysis
Complex model of choice.
Conditions.
Multiple stakeholders.
Uncertainty about outcomes.
The passage of time between actions and
consequences.

Results.
Intransitivity of choice.

Recommendation in Policy
Analysis
Forms of rationality.
Given the conditions of complex choice, there
are multiple forms of rationality.

Technical rationality.
Economic rationality.
Legal rationality.
Social rationality.
Substantive rationality.

Recommendation in Policy
Analysis
Rational-comprehensive theory.
An individual or collective decision-maker must
identify a policy problem on which there is a consensus
among all relevant stakeholders.
An individual or collective decision-maker must define
and consistently rank all goals and objectives whose
attainment would represent a resolution of the problem.
An individual or collective decision-maker must
identify all policy alternatives that may contribute to
the attainment of each goal and objective.

Recommendation in Policy
Analysis
Rational-comprehensive theory.
An individual or collective decision-maker must
forecast all consequences that will result from the
selection of each alternative.
An individual or collective decision-maker must
compare each alternative in terms of its consequences
for the attainment of each goal and objective.
An individual or collective decision-maker must choose
that alternative that maximizes the attainment of
objectives.

Recommendation in Policy
Analysis
Disjointed incremental theory.
Consider only those objectives that differ incrementally
from the status quo.
Limit the number of consequences forecast for each
alternative.
Make mutual adjustments in goals and objectives, on
the one hand, and alternatives on the other.
Continuously reformulate problems and hence goals,
objectives, and alternatives in the course of acquiring
new information.

Public Policy Analysis


Monitoring Policy Outcomes

Introduction
The consequences of policy actions are

never fully known in advance and, for this


reason, it is essential to monitor policy
actions after they have occurred.
In fact, policy recommendations may be
viewed as hypotheses about the relationship
between policy actions and policy
outcomes: if action A is taken at time t1,
outcome O will result at time t2.

Monitoring in Policy Analysis


Monitoring is the policy-analytic procedure used

to produce information about the causes and


consequences of public policies.
Monitoring, since it permits analysts to describe
relationships between policy-program operations
and their outcomes, is the primary source of
knowledge about policy implementation.
Monitoring is primarily concerned about with
establishing factual premises about public policy.

Monitoring in Policy Analysis


Monitoring performs at least four major

functions:

Compliance: monitoring helps determine whether


the actions of program administrators, staff, and
other stakeholders are in compliance with standards
and procedures imposed by legislatures, regulatory
agencies, and professional bodies.
Auditing: monitoring helps determine whether
resources and services intended for certain target
groups and beneficiaries have actually reached
them.

Monitoring in Policy Analysis


Monitoring performs at least four major

functions:

Accounting: monitoring produces information


that is helpful in accounting for social and
economic changes that follow the
implementation of broad sets of public policies
and programs over time.
Explanation: monitoring also yields
information that helps to explain why the
outcomes of public polices and programs differ.

Monitoring in Policy Analysis


Sources of information.
Information must be:
Relevant.
Macronegative versus micropositive.

Reliable.
Observations precise and dependable.

Valid.
Information about policy outcomes should measure what
we think it is measuring.

Monitoring in Policy Analysis


Sources of information.
Information on policy outcomes.
Historical Statistics of the United States

Current Opinion

Statistical Abstract of the United States

United States Census of Population by States

County and City Data Book

Congressional District Data Book

Census Use Study

National Opinion Research Center General Social


Survey

Social and Economic Characteristics of Students

Survey Research Center National Election Studies

Educational Attainment in the United States

National Clearinghouse for Mental Health


Information

Current Population Reports

National Clearinghouse for Drug Abuse Information

The Social and Economic Status of the Black


Population in the United States

National Criminal Justice Reference Service

Female Family Heads

Child Abuse and Neglect Clearinghouse Project

Monthly Labor Review

National Clearinghouse on Revenue Sharing

Handbook of Labor Statistics

Social Indicators

Congressional Quarterly

State Economic and Social Indicators

Law Digest

Monitoring in Policy Analysis


Sources of information.
When information is not available from
existing sources, monitoring may be carried out
by some combination of questionnaires,
interviewing, field observation, and the use of
agency records.

Monitoring in Policy Analysis


Types of policy outcomes.
Consequences:
Policy outputs the goods, services, or resources received by
target groups and beneficiaries.
Policy impacts actual changes in behavior and attitudes that
result from policy outputs.

Populations:
Target groups individuals, communities, or organizations on
whom policies and programs are designed to have an effect.
Beneficiaries groups for whom the effects of policies are
beneficial or valuable.

Monitoring in Policy Analysis


Types of policy actions. Policy actions have

two major purposes.

Regulation actions designed to ensure


compliance with certain standards or
procedures.
Allocation actions that require inputs of
money, time, personnel, and equipment.
Regulative and allocative actions may have
consequences that are distributive or
redistributive.

Monitoring in Policy Analysis


Types of policy actions.
Policy inputs the resources time, money,
personnel, equipment, and supplies used to
produce outputs and impacts.
Policy processes the administrative,
organizational, and political activities and
attitudes that shape the transformation of policy
inputs into policy outputs and impacts.

Monitoring in Policy Analysis


POLICY ACTIONS

POLICY OUTCOMES

ISSUE AREA
Inputs

Processes

Outputs

Impacts

Criminal Justice.

Dollar
expenditures for
salaries,
equipment,
maintenance.

Illegal arrests as
percentages of
total arrests.

Criminals arrested
per 100,000
known crimes.

Criminals
convicted per
100,000 known
crimes.

Municipal
Services.

Dollar
expenditures for
sanitation workers
and equipment.

Morale among
workers.

Total residences
served.

Cleanliness of
streets.

Social welfare.

Number of social
workers.

Rapport with
welfare recipients.

Welfare cases per


social worker.

Standard of living
of dependent
children.

Monitoring in Policy Analysis


Definitions and indicators.
Variables versus constants.
Definitions.
Constitutive definitions gives meaning to words used to
describe variables by using synonymous words.
Provide no concrete rules or guidelines for actually
monitoring changes.
Operational definitions gives meaning to a variable by
specifying the operations necessary to experience and measure
it.
Indicators directly observable characteristics.

Approaches to Monitoring
APPROACH

TYPES OF
CONTROL

TYPE OF
INFORMATION
REQUIRED

Social systems accounting

Quantitative

Available and/or new


information

Social experimentation

Direct manipulations and


quantitative

New information

Social auditing

Quantitative and/or
qualitative

New information

Research and practice


synthesis

Quantitative and/or
qualitative

Available information

Approaches to Monitoring
MANIPULABLE ACTIONS
POLICY
INPUTS

POLICY
PROCESSES

CONTROLLED OUTCOMES
POLICY
OUTPUTS

POLICY
IMPACTS

In1

P1

O1

Im1

In2
.
.
.
Ini

P2
.
.
.
Pj

O2
.
.
.
Om

Im2
.
.
.
Imn

PC1

E1

SES 1

PC2
.
.
.
PCp

E2
.
.
.
Eq

SES 2
.
.
.
SES r

PRECONDITIONS

UNFORESEEN
EVENTS

UNMANIPULABLE CAUSES

SIDE-EFFECTS
AND
SPILLOVERS
UNCONTROLLED EFFECTS

Social Systems Accounting


An approach and set of methods that permit

analysts to monitor changes in objective


and subjective social conditions over time.

Social Systems Accounting


The major analytic element of social

systems accounting is the social indicator.


Statistics that measure social conditions and
changes therein over time for various segments
of a population. By social conditions, we mean
both the external (social and physical) and the
internal (subjective and perceptional) contexts
of human existence in a given society.

Social systems accounting


Representative social indicators.
AREA

INDICATOR

Health and illness

Persons in state mental hospitals

Public safety

Persons afraid to walk alone at night

Education

High school graduates aged 25 and older

Employment

Labor force participation by women

Income

Percent of population below the poverty line

Housing

Households living in substandard units

Leisure and recreation

Average annual paid vacation in manufacturing

Population

Actual and projected population

Government and politics

Quality of public administration

Social values and attitudes

Overall life satisfaction and alienation

Social mobility

Change from fathers occupation

Physical environment

Air pollution index

Science and technology

Scientific discoveries

Social Systems Accounting


Must assume that the reasons for indicator change

are related to policy actions. This is not always


the case.
Advantages.

Identify areas of insufficient information.


When indicators provide reliable information, possible
to modify policies.

Social Systems Accounting


Limitations.
Choice of indicators influenced by values of analysts.
Social indicators frequently do not cover manipulable
policy actions.
Most social indicators are based available data about
objective (rather than subjective) social conditions.
Social indicators provide little information about how
inputs are transformed into outcomes.

Social Systems Accounting

Social Systems Accounting

Social Systems Accounting

Coefficientsa

Breaks in
disaster patterns
1950 - 1971

Model
1

1972 - 1988

1989 - 2006

(Constant)
Number of years
(Constant)
Number of years
(Constant)
Number of years

Unstandardized
Coefficients
B
Std. Error
13.516
2.561
.340
.243
71.419
11.060
-1.551
.403
-6.745
22.133
1.218
.494

a. Dependent Variable: Total Disaster Declarations

Standardized
Coefficients
Beta
.322
-.705
.525

t
5.277
1.400
6.458
-3.849
-.305
2.465

Sig.
.000
.179
.000
.002
.764
.025

Social Systems Accounting

Social Experimentation
Use of social indicators often leads to

random innovation rather than systematic


change.
Social experimentation is the process of
systematically manipulating policy actions
in way that permits more or less precise
answers to questions about the sources of
change in policy outcomes.

Social Experimentation
Characteristics and procedures.
Direct control of experimental treatment.
Comparison (control) groups.
Random assignment.
Goal: internal validity the capacity of

experiments and quasi-experiments to make


valid causal inferences about the effects of
actions on outcomes.

Social Experimentation
Threats to internal validity.
History.
Maturation.
Instability.
Instrumentation and testing.
Mortality.
Selection.
Regression artifacts.

Social Experimentation
Social experimentation is weakest in the

area of external validity or generalizability.


Neglects policy processes.

Seattle-Denver Income Maintenance


Experiment
A social experiment is a field test of one or

more social programs or, to use the


phraseology of the natural sciences, a test of
one or more "treatments."
A social experiment is a field test in the
sense that families or individuals are
actually enrolled in a pilot social program
offering some type of special benefit or
service.

Seattle-Denver Income Maintenance


Experiment
It is "experimental" in the sense that

families or individuals are enrolled in each


of the tested programs on the basis of a
random assignment process, for example,
the flip of a coin.

Seattle-Denver Income Maintenance


Experiment
To draw conclusions about the effects of the

treatment, it is necessary to collect


information about the people who are
enrolled in each experimental program and
about those who receive no special
treatment (called the control group), and
then to compare them on the basis of the
collected information.

Seattle-Denver Income Maintenance


Experiment
The primary focus of SIME/DIME the

effect of income maintenance on work


effort or labor supply was a good subject
for experimentation .
An increase in unearned income or a
decrease in the effective wage rate were
hypothesized to lead to decreases in labor
supply.

Seattle-Denver Income Maintenance


Experiment
In the case of SIME/DIME, the experiment

was in fact designed to test the effect of two


different kinds of social programs on
participant work effort.
The two policies were a variety of cash
transfer or negative income tax programs
and several combinations of job counseling
and education or training subsidy programs.

Seattle-Denver Income Maintenance


Experiment
Any observed effects of the experiment was

interpreted as the differential effects of the


experimental treatment compared to
existing government programs.
Thus any observed experimental-control
differences in outcomes must be interpreted
as estimates of the effect of replacing the
early 1970s status quo with the
experimental programs.

Seattle-Denver Income Maintenance


Experiment
Table 4.
Labor Supply Response of Husbands:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
3-year sample
-1.6
-7.3
-7.3
-0.5
-0.2
5-year sample
-5.9 -12.2 -13.2 -13.6 -12.3
Total Sample
-3.1
-9
-9.3

Seattle-Denver Income Maintenance


Experiment
Table 5.
Labor Supply Response of Wives:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
-4
-16.5 -15.2 -2
+13.4
3-year sample
-15.1 -26.5 -21.6 -27.1 -24
5-year sample
-8.1 -20.1 -17.4

Total Sample

Seattle-Denver Income Maintenance


Experiment
Table 6.
Labor Supply Response of Female Heads:
A. Overall NIT response
(Percentage difference in annual hours worked)
Year
Sample Group
1
2
3
4
5
-5.5 -14.1 -21.6 -8.9
-7.7
3-year sample
-7.9 -15
-21.2 -28.3 -31.8
5-year sample
-6.3 -14.3 -21.4

Total Sample

Social Auditing
One of the limitations of social systems

accounting and social experimentation is


that both approaches neglect or
oversimplify policy processes.

Social Auditing
Social auditing explicitly monitors relations

among inputs, processes, outputs, and


impacts in an effort to trace policy inputs
from disbursement to final recipient.
Social auditing helps to determine whether
ineffective outcomes are the result of
inadequate inputs or processes that divert
resources or services from intended target
groups or beneficiaries.

Social Auditing
Processes monitored of two main types:
Resource diversion (from target or beneficiary
groups).
Resource transformation (changes in meaning
of policy actions from administrator to
recipient).

Social Audit Steps


The three phases of a social audit
Phase 1: Design and data collection
Clarify the strategic focus
Design instruments, pilot test
Collect information from households and key
informants in a panel of representative
communities

Social Audit Steps


Phase 2: Evidence-based dialogue and

analysis
Link household data with information from
public services
Analyze findings in a way that points to action
Take findings back to the communities for their
views about how to improve the situation
Bring community members into discussion of
evidence with service providers/planners.

Social Audit Steps


Phase 3: Socialization of evidence for

public accountability
Work-shopping
Communication strategy
Evidence-based training of planners and service
providers
Media training
Partnerships with civil society

Research and practice synthesis.


An approach to monitoring that involves the

systematic compilation, comparison, and


assessment of the results of past efforts to
implement public policies.
Two primary sources of available information.
Case studies of policy formulation and implementation.
Research reports that address relations among policy
actions and outcomes.

Research and practice synthesis.


Two methods.
Case survey method a set of procedures used
to identify and analyze factors that account for
variations in the adoption and implementation
of policies.
Requires case coding scheme, a set of categories
that capture key aspects of policy inputs, processes,
outputs, and impacts.

Research and practice synthesis


Two methods.
Research survey method a set of procedures
used to compare and appraise results of past
research on policy actions and outcomes.
Yields empirical generalizations about sources of
variation in policy outcomes, summary assessments
of the confidence researchers have in these
generalizations, and policy alternatives or action
guidelines that are implied in these generalizations.
Requires the construction of a format for extracting
information about research reports.

Research and practice synthesis


Advantages.
Comparatively efficient way to compile and
appraise an increasingly large body of cases
and research reports on policy implementation.
The case survey method is one among several
ways to uncover different dimensions of policy
processes that affect policy outcomes.
The case survey method is a good way to
examine both objective and subjective
conditions.

Research and practice synthesis


Limitations.
All related to reliability and validity of
information.

Research and Practice Synthesis:


Example
David H. Greenberg; Charles

Michalopoulos; Philip K. Robins.


Do Experimental and Nonexperimental
Evaluations Give Different Answers
about the Effectiveness of GovernmentFunded Training Programs?

Research and Practice Synthesis:


Example
Abstract
This paper uses meta-analysis to investigate

whether random assignment (or experimental)


evaluations of voluntary government-funded
training programs for the disadvantaged have
produced different conclusions than
nonexperimental evaluations.
Information includes several hundred estimates
from 31 evaluations of 15 programs that operated
between 1964 and 1998.

Research and Practice Synthesis:


Example
The results suggest that experimental and

nonexperimental evaluations yield similar


conclusions about the effectiveness of
training programs, but that estimates of
average effects for youth and possibly men
might have been larger in experimental
studies.

Program Evaluation

Source
Basic Guide to Program Evaluation
Written by
Carter McNamara, MBA, PhD, Authenticity Co
nsulting, LLC
. Copyright 1997-2006.
Adapted from the
Field Guide to Nonprofit Program Design, Mark
eting and Evaluation
.

A Brief Introduction ...


Note that the concept of program evaluation can

include a wide variety of methods to evaluate


many aspects of programs in nonprofit or forprofit organizations.
However, personnel do not have to be experts in
these topics to carry out a useful program
evaluation. The "20-80" rule applies here, that
20% of effort generates 80% of the needed results.
It's better to do what might turn out to be an
average effort at evaluation than to do no
evaluation at all.

Some Myths about Program


Evaluation
Many people believe evaluation is a useless

activity that generates lots of boring data


with useless conclusions.
More recently (especially as a result of
Michael Patton's development of utilizationfocused evaluation), evaluation has focused
on utility, relevance and practicality at least
as much as scientific validity.

Some Myths about Program


Evaluation
Many people believe that evaluation is about

proving the success or failure of a program.


This myth assumes that success is implementing
the perfect program and never having to hear from
employees, customers or clients again -- the
program will now run itself perfectly.
This doesn't happen in real life.
Success is remaining open to continuing feedback
and adjusting the program accordingly.
Evaluation gives you this continuing feedback.

Some Myths about Program


Evaluation
Many believe that evaluation is a highly unique and

complex process that occurs at a certain time in a certain


way, and almost always includes the use of outside
experts.
Many people believe they must completely understand
terms such as validity and reliability.
They don't have to.
They do have to consider what information they need to
make current decisions about program issues or needs.
And they have to be willing to commit to understanding
what is really going on.

So What is Program Evaluation?


First, we'll consider "what is a program?"
Typically, organizations work from their mission to

identify several overall goals which must be reached to


accomplish their mission.
In public agencies and nonprofits, each of these goals often
becomes a program.
Public and nonprofit programs are organized methods to
provide certain related services to constituents, e.g.,
clients, customers, patients, etc.
Programs must be evaluated to decide if the programs are
indeed useful to constituents.

So What is Program Evaluation?


Program evaluation is carefully collecting

information about a program or some aspect of a


program in order to make necessary decisions
about the program.
Don't worry about what type of evaluation you
need or are doing -- worry about what you need to
know to make the program decisions you need to
make, and worry about how you can accurately
collect and understand that information.

Where Program Evaluation is


Helpful
Understand, verify or increase the impact of

products or services on customers or clients.


Improve delivery mechanisms to be more
efficient and less costly.
Verify that you're doing what you think
you're doing.

Other Reasons to Conduct


Evaluations
Facilitate management's really thinking

about what their program is all about,


including its goals, how it meets it goals
and how it will know if it has met its goals
or not.
Produce data or verify results that can be
used for public relations and promoting
services in the community.

Other Reasons to Conduct


Evaluations
Produce valid comparisons between

programs to decide which should be


retained, e.g., in the face of pending budget
cuts.
Fully examine and describe effective
programs for duplication elsewhere.

Basic Ingredients: Organization


and Program(s)
You Need An Organization:
This may seem too obvious to discuss, but before an
organization embarks on evaluating a program, it
should have well established means to conduct itself as
an organization.
You Need Program(s):
To effectively conduct program evaluation, you should
first have programs. That is, you need a strong
impression of what your customers or clients actually
need.
Next, you need some effective methods to meet each of
those goals. These methods are usually in the form of
programs.

Programs
Inputs are the various resources needed to

run the program, e.g., money, facilities,


customers, clients, program staff, etc.
The process is how the program is carried
out.

Programs
The outputs are the units of service, e.g., number

of customers serviced, number of clients


counseled, children cared for, artistic pieces
produced, or members in the association.
Outcomes are the impacts on the customers or on
clients receiving services, e.g., increased mental
health, safe and secure development, richer artistic
appreciation and perspectives in life, increased
effectiveness among members, etc.

Planning Your Program


Evaluation
Depends on what information you need to

make your decisions and on your resources.


Your program evaluation plans depend on what
information you need to collect in order to make
major decisions.
But the more focused you are about what you want to
examine by the evaluation, the more efficient you can
be in your evaluation, the shorter the time it will take
you and ultimately the less it will cost you.

Key Considerations
For what purposes is the evaluation being

done, i.e., what do you want to be able to


decide as a result of the evaluation?
Who are the audiences for the information
from the evaluation, e.g., customers,
bankers, funders, board, management, staff,
customers, clients, citizens, etc.

Key Considerations
What kinds of information are needed to make the

decision you need to make and/or enlighten your


intended audiences, e.g., information to really
understand the process of the product or program
(its inputs, activities and outputs), the customers
or clients who experience the product or program,
strengths and weaknesses of the product or
program, benefits to customers or clients
(outcomes), how the product or program failed
and why, etc.

Key Considerations
From what sources should the information be

collected, e.g., employees, customers, clients,


groups of customers or clients and employees
together, program documentation, etc.
How can that information be collected in a
reasonable fashion, e.g., questionnaires,
interviews, examining documentation, observing
customers or employees, conducting focus groups
among customers or employees, etc.

Key Considerations
When is the information needed (so, by

when must it be collected)?


What resources are available to collect the
information?

Goals-Based Evaluation
1. How were the program goals (and objectives, is

applicable) established? Was the process


effective?
2. What is the status of the program's progress
toward achieving the goals?
3. Will the goals be achieved according to the
timelines specified in the program
implementation or operations plan? If not, then
why?
4. Do personnel have adequate resources (money,
equipment, facilities, training, etc.) to achieve the
goals?

Goals-Based Evaluation
5. How should priorities be changed to put more focus on

achieving the goals? (Depending on the context, this


question might be viewed as a program management
decision, more than an evaluation question.)
6. How should timelines be changed (be careful about
making these changes - know why efforts are behind
schedule before timelines are changed)?
7. How should goals be changed (be careful about making
these changes - know why efforts are not achieving the
goals before changing the goals)? Should any goals be
added or removed? Why?
8. How should goals be established in the future?

Process-Based Evaluations
1. On what basis do employees and/or the

customers decide that products or services


are needed?
2. What is required of employees in order to
deliver the product or services?

Process-Based Evaluations
3. How are employees trained about how to

deliver the product or services?


4. How do customers or clients come into
the program?
5. What is required of customers or client?
6. How do employees select which products
or services will be provided to the customer
or client?

Process-Based Evaluations
7. What is the general process that

customers or clients go through with the


product or program?
8. What do customers or clients consider to
be strengths of the program?
9. What do staff consider to be strengths of
the product or program?
10. What typical complaints are heard from
employees and/or customers?

Process-Based Evaluations
11. What do employees and/or customers

recommend to improve the product or


program?
12. On what basis do employees and/or the
customer decide that the product or services
are no longer needed?

Outcome-Based Evaluations
Identify the major outcomes that you want to

examine or verify for the program under


evaluation.
Choose the outcomes that you want to examine,
prioritize the outcomes and, if your time and
resources are limited, pick the top two to four
most important outcomes to examine for now.
For each outcome, specify what observable
measures, or indicators, will suggest that you're
achieving that key outcome with your clients.

Outcome-Based Evaluations
Specify a "target" goal of clients, i.e., with

what number or percent of clients you


commit to achieving specific outcomes?
Identify what information is needed to show
these indicators.
Decide how can that information be
efficiently and realistically gathered.
Analyze and report the findings.

Methods to Collection Information


Questionnaires, surveys, checklists.
Interviews.
Documentation review.
Observation.
Focus groups.
Case studies.

Methods to Collection Information

Methods to Collection Information

Methods to Collection Information

Ethics: Informed Consent from


Program Participants
Note that if you plan to include in your evaluation,

the focus and reporting on personal information


about customers or clients participating in the
evaluation, then you should first gain their consent
to do so.
They should understand what you're doing with
them in the evaluation and how any information
associated with them will be reported.

Ethics: Informed Consent from


Program Participants
You should clearly convey terms of

confidentiality regarding access to


evaluation results.
They should have the right to participate or
not.
Have participants review and sign an
informed consent form.

Selecting Which Methods to Use


The overall goal in selecting evaluation method(s) is to get

the most useful information to key decision makers in the


most cost-effective and realistic fashion.
Note that, ideally, the evaluator uses a combination of
methods, for example, a questionnaire to quickly collect a
great deal of information from a lot of people, and then
interviews to get more in-depth information from certain
respondents to the questionnaires.
Perhaps case studies could then be used for more in-depth
analysis of unique and notable cases, e.g., those who
benefited or not from the program, those who quit the
program, etc.

Selecting Which Methods to Use


1. What information is needed to make current
2.

3.
4.
5.

decisions about a product or program?


Of this information, how much can be collected
and analyzed in a low-cost and practical manner,
e.g., using questionnaires, surveys and checklists?
How accurate will the information be (reference
the above table for disadvantages of methods)?
Will the methods get all of the needed
information?
What additional methods should and could be used
if additional information is needed?

Selecting Which Methods to Use


6. Will the information appear as credible to

decision makers, e.g., to funders or top


management?
7. Will the nature of the audience conform to the
methods, e.g., will they fill out questionnaires
carefully, engage in interviews or focus groups,
let you examine their documentations, etc.?
8. Who can administer the methods now or is
training required?
9. How can the information be analyzed?

Four Levels of Evaluation:


There are four levels of evaluation information

that can be gathered from clients, including


getting their:

1. reactions and feelings (feelings are often poor


indicators that your service made lasting impact)
2. learning (enhanced attitudes, perceptions or
knowledge)
3. changes in skills (applied the learning to enhance
behaviors)
4. effectiveness (improved performance because of
enhanced behaviors)

Four Levels of Evaluation:


Usually, the farther your evaluation

information gets down the list, the more


useful is your evaluation. Unfortunately, it
is quite difficult to reliably get information
about effectiveness. Still, information
about learning and skills is quite useful.

Analyzing and Interpreting


Information
Always start with your evaluation goals:
When analyzing data (whether from
questionnaires, interviews, focus groups, or
whatever), always start from review of your
evaluation goals, i.e., the reason you undertook
the evaluation in the first place. This will help
you organize your data and focus your analysis.

Analyzing and Interpreting


Information
Basic analysis of "quantitative" information (for

information other than commentary, e.g., ratings,


rankings, yes's, no's, etc.):
1. Make copies of your data and store the master copy
away. Use the copy for making edits, cutting and
pasting, etc.
2. Tabulate the information, i.e., add up the number of
ratings, rankings, yes's, no's for each question.
3. For ratings and rankings, consider computing a mean,
or average, for each question.
4. Consider conveying the range of answers, e.g., 20
people ranked "1", 30 ranked "2", and 20 people
ranked "3".

Analyzing and Interpreting


Information
Basic analysis of "qualitative" information

(respondents' verbal answers in interviews, focus


groups, or written commentary on
questionnaires):
1. Read through all the data.
2. Organize comments into similar categories.
3. Label the categories or themes, e.g., concerns,
suggestions, etc.
4. Attempt to identify patterns, or associations and causal
relationships in the themes.
5. Keep all commentary for several years after
completion in case needed for future reference.

Interpreting Information:
1. Attempt to put the information in perspective.
a)
b)
c)
d)
e)
f)

Compare results to what you expected;


Management or program staff
Common standards for your services
Original program goals (program evaluation)
Indications of accomplishing outcomes (outcomes evaluation)
Description of the program's experiences, strengths, weaknesses,
etc. (process evaluation).

2. Consider recommendations to help program staff improve

the program, conclusions about program operations or


meeting goals, etc.
3. Record conclusions and recommendations in a report
document, and associate interpretations to justify your
conclusions or recommendations.

Reporting Evaluation Results


1. The level and scope of content depends on

to whom the report is intended,


2. Be sure employees have a chance to
carefully review and discuss the report.
Translate recommendations to action plans,
including who is going to do what about the
program and by when.

Reporting Evaluation Results


3. Bankers or funders will likely require a report that

includes an executive summary (this is a summary of


conclusions and recommendations, not a listing of what
sections of information are in the report -- that's a table of
contents); description of the organization and the program
under evaluation; explanation of the evaluation goals,
methods, and analysis procedures; listing of conclusions
and recommendations; and any relevant attachments,
4. Be sure to record the evaluation plans and activities in an
evaluation plan which can be referenced when a similar
program evaluation is needed in the future.

Pitfalls to Avoid
1. Don't balk at evaluation because it seems far too

"scientific." It's not. Usually the first 20% of effort


will generate the first 80% of the plan, and this is
far better than nothing.
2. There is no "perfect" evaluation design. Don't
worry about the plan being perfect. It's far more
important to do something, than to wait until
every last detail has been tested.

Pitfalls to Avoid
3. Work hard to include some interviews in your evaluation

methods. Questionnaires don't capture "the story," and the


story is usually the most powerful depiction of the
benefits of your services.
4. Don't interview just the successes. You'll learn a great
deal about the program by understanding its failures,
dropouts, etc.
5. Don't throw away evaluation results once a report has
been generated. Results don't take up much room, and
they can provide precious information later when trying
to understand changes in the program.

Research and Practice Synthesis:


Example
The results also suggest that variation

among nonexprimental estimates of


program effects is similar to variation
among experimental estimates for men and
youth, but not for women (for whom it
seems to be larger), although small sample
sizes make the estimated differences
somewhat imprecise for all three groups.

Research and Practice Synthesis:


Example
The policy implications of the findings are

discussed.
2006 by the Association for Public Policy
Analysis and Management

Techniques for Monitoring


TABULAR
DISPLAYS

INDEX
NUMBERS

INTERRUPTED
TIMESERIES
ANALYSIS

CONTROLSERIES
ANALYSIS

REGRESSION DISCONTINUITY
ANALYSIS

APPROACH

GRAPHIC
DISPLAYS

Social systems
accounting

Social auditing

Social
experimentation

Research and
practice
synthesis

Recommendation in Policy
Analysis
Disjointed incremental theory.
Analyze and evaluate alternatives in a sequence of
steps, such that choices are continuously amended over
time, rather than made at a single point in time.
Continuously remedy existing social problems, rather
than solve problems completely at one point in time.
Share responsibilities for analysis and evaluation with
many groups in society, so that the process of making
choices is fragmented or disjointed.

Recommendation in Policy
Analysis
Arrows impossibility theorem.
It is impossible for democratic decision makers
in a democratic society to meet conditions of
the rational comprehensive model.
Individual choices cannot be aggregated through
majority voting procedures to create a collective
decision that will produce a single best solution for
all parties.

The Voters Paradox


COMMITTEE MEMBER
Brown (criterion: risk)

PREFERENCE
A (solar) preferred to B (coal)
B (coal) preferred to C (nuclear)
A (solar preferred to C (nuclear)

Jones (criterion: feasibility)

B preferred to C
C preferred to A
B preferred to A

Smith (criterion: efficiency)

C preferred to A
A preferred to B
C preferred to B

Majority (intransitive and cyclic)

A preferred to B
B preferred to C
C preferred to A

Recommendation in Policy
Analysis
Arrows impossibility theorem.
Reasonable conditions for democratic decision
procedures.

Nonrestriction of choices.
Nonperversity of collective choice.
Independence of irrelevant alternatives.
Citizens sovereignty.
Nondictatorship.

Recommendation in Policy
Analysis
Bounded rationality.
Decision makers engage in satisficing behavior
(identify courses of action that are good
enough.).
Consider the most evident alternatives that produce
a reasonable increase in benefits.

Rationality as constrained maximization.


Rational choice within the boundaries of
constraint.

Recommendation in Policy
Analysis
Criteria for policy recommendation.
Effectiveness does a given alternative result in the
achievement of a valued outcome (technical
rationality).
Efficiency the amount of effort needed to produce a
given level of effectiveness (economic rationality).
Adequacy the extent to which any given level of
effectiveness satisfies the needs, values, or
opportunities that gave rise to the problem.
Fixed costs and variable effectiveness (type I, maximize
effectiveness).
Fixed effectiveness and variable costs (type II, minimize
costs).
Variable costs and variable effectiveness (type III, efficiency).
Fixed costs and fixed effectiveness (type IV, do nothing).

Recommendation in Policy
Analysis
Criteria for policy recommendation.
Equity the distribution of effects and effort among
different groups in society (legal and social rationality).

Maximize individual welfare.


Protect minimum welfare.
Maximize net welfare.
Maximize redistributive welfare.

Responsiveness satisfies the needs, preferences, or


values of particular groups.
Appropriateness the value or worth of a programs
objectives and the tenability of assumptions underlying
these objectives.

Approaches to Recommendation
Public versus private choice.
Nature of public policy processes.
Numerous stakeholders with conflicting values.

Collective nature of public policy goals.


Multiple conflicting criteria for choice.

Nature of public goods.


Specific, collective, and quasi-collective goods.

Supply and demand (market mechanisms).


Discuss.

Approaches to Recommendation
Public choice.
Problems with supply demand models of
public policy.

Multiple legitimate stakeholders.


Collective and quasi-collective goods.
Limited comparability of income measures.
Public responsibility for social costs and benefits.

Approaches to Recommendation
Benefit-cost analysis.
Characteristics.
Measure all costs and benefits to society of a program
including intangibles.
Traditional benefit-cost analysis emphasizes economic
rationality: net benefits are greater than zero and higher than
alternative uses.
Traditional benefit-cost analysis uses the private marketplace
as the point of departure in recommending programs.
Social benefit-cost analysis also measures redistributional
benefits.

Approaches to Recommendation
Types of costs and benefits.
Inside versus outside costs and benefits.
Tangible versus intangible costs and benefits.
Direct versus indirect costs and benefits.
Net efficiency versus redistributional benefits.

Approaches to Recommendation
Tasks in benefit-cost analysis.
Problem structuring.
Specification of objectives.
Identification of alternative solutions.
Information search, analysis, and interpretation.
Identification of target groups and beneficiaries.
Estimation of costs and benefits.
Discounting of costs and benefits.
Estimation of risk and uncertainty.
Choice of decision criterion.
Recommendation.

Benefit-cost and Costeffectiveness Analysis


Benefit-cost analysis is an applied branch of

economics that attempts to assess service


programs by determining whether the total societal
welfare has increased (in aggregate more people
have been made better off) because of the project
or program.
Steps.
Determine the benefits of a proposed or existing
program and place a dollar value on those benefits.
Calculate the total costs of the program.
Compare the benefits and costs.

Benefit-cost and Costeffectiveness Analysis


Simple steps pose real challenge, especially

estimating intangible benefits.


Procedure still useful in uncovering
assumptions and estimating value of
intangibles.

Benefit-cost and Costeffectiveness Analysis


Cost-effectiveness analysis.
The major costing alternative to benefit-cost analysis.
Relates the cost of a given alternative to specific measures of
program objectives.
For example, dollar per life saved on various highway
safety programs.

Often the first step in a benefit-cost analysis.


Especially useful if analyst cannot quantify benefits,
but has fairly specific program objectives.
Key problem: situation where there are multiple
benefits. Results often very subjective.
2nd key problem: does not produce a bottom line
number.

Benefit-cost and Costeffectiveness Analysis


A private sector analogy.
Benefit-cost analysis similar to financial analysis in
private sector.
Should the firm have done the project at all, i.E., Is the project
producing a satisfactory rate of return?
Public version: is the program a success, i.E., Has it improved
social welfare?
What other options are there for the use of the firms
resources?
Public version: should the program be continued when
weighed against alternative uses for the governments funds.

Benefit-cost and Costeffectiveness Analysis


A private sector analogy.
Benefits and costs do not occur simultaneously
in either private or public sector.
R&D costs, marketing, capital investment, training.
Government not priced, thus benefits are more
broadly defined.

Alternative uses of resources (opportunity


costs).

Benefit-cost and Costeffectiveness Analysis


Benefit-cost illustration.
Costs
Year

R&D

Capital

O&M

Total

Benefits

$1500

$2000

$0

$3500

$0

500

2000

2000

4500

3500

2000

2500

4500

5500

2000

3000

5000

6500

2000

3500

5500

8500

$10000

$11000

$23000

$24000

Totals

Year

$2000

Benefit-Costs

Present Value, Benefits-Costs (10%)

($3500)

($3500)

(1000)

(909)

1000

826

1500

1127

3000

2049

$1000

($407)

Totals

Benefit-cost and Costeffectiveness Analysis


Formula for net present value.
y2
y2
y3
y3
yx
yx
B

C
B

C
B

C
NPV B y1 C y1

...
2
1 r
(1 r )
(1 r ) x 1

Formula sensitive to choice of rate of return.


Could also use return on investment.
Discount rate that would make the present value zero.

Benefit-cost and Costeffectiveness Analysis


Continuing or not continuing the project.
Costs
Year
6

R&D

Capital

$500

O&M

Total

Benefits

$400

$4500

$5400

$9000

400

5000

5400

8500

400

6000

6400

8500

400

7500

7900

8000

10

400

8000

8400

7500

$2000

$31000

$33500

$41500

Totals

Year

$500

Benefit-Costs

Present Value, Benefits-Costs (10%)

$3600

3600

3100

2818

2100

1736

100

75

(900)

(615)

$8000

$7614

10
Totals

Benefit-cost and Costeffectiveness Analysis


Total versus marginal benefits and costs.
When considering the overall profitability of a project,
an agency will consider the total costs involved in
getting the project started through its operations cycle.
But at any point in time, when an agency is continuing
or discontinuing a project or program, it only considers
the marginal costs or benefits.
Definitions.
Marginal cost: incremental cost of producing one more unit of
output.
Marginal benefit: incremental benefit of that one unit of
output.

Public sector usually does not do it on the basis of a


single unit, but what are the benefits being generated
now versus the costs now?

Benefit-cost and Costeffectiveness Analysis


Public-private sector differences and

similarities.
Similarities alternative uses for funds, onetime costs, recurring costs, land, labor, capital.
Differences.
Distributional considerations.
Spillovers.

Framework for Analysis


Benefit

Indicator

Measure

Dollar value

Assumptions

Indicator

Measure

Dollar value

Assumptions

Real
Direct
Tangible
Intangible
Indirect
Tangible
Intangible

Cost
Real
Direct
Tangible
Intangible

Framework for Analysis


Real benefits and costs versus transfers.
Real: net gains and losses to society.
Transfers: merely alter the distribution of
resources in society.

Framework for Analysis


Direct and indirect benefits and costs.
Direct costs and benefits are closely related to
the primary objectives of the project.
Direct costs personnel, facilities, equipment,
material, project administration.

Indirect costs and benefits are byproducts,


multipliers, spillovers, or investment effects.
Indirect costs are unintended costs that result from
government action.
Indirect benefits might include benefits of space
exploration.

Framework for Analysis


Tangible and intangible benefits and costs.
Tangible benefits and costs can be converted
readily into dollar figures.
Intangible benefits and costs are those things
that cannot be directly assigned an explicit
price.
Determining the geographic scope of

analysis.

Spillover effects may determine true


geographic jurisdiction.

Framework for Analysis


(Agricultural Dam Example)
Real Benefits

Nature of Benefit/Cost

Direct
Tangible

Increased farm output


New supply of water

Intangible

Maintaining family farms

Indirect
Tangible

Reduced soil erosion

Intangible

Preservation of rural society

Real Costs
Direct
Tangible

Construction material, labor, operations and maintenance, direct program supervision by agency

Intangible

Loss of recreational value of land or river

Indirect
Tangible

Administrative overhead of government


Diversion of water and its effects
Increased salinity

Intangible

Loss of wilderness area

Transfers

Relative improvement of profit for farm implement industry

Measuring Benefits
Evaluation problem difficult for

government because of multiple benefits


and intangibles.

Measuring Benefits
Sources of data.
Existing records and statistics kept by agency.
Feedback from clients.
Ratings by trained observers.
Experience of other governments or private or
nonprofit corporations.
Special data gathering.
Whenever possible analyst should use

market value or willingness to pay.

Measuring Benefits
Valuing benefits.
Cost savings.
Time saved.
Lives saved.
Increased productivity or wages.
Recreational benefits.
Land values.
Alternatives to market prices.
See handout.

Measuring Costs
Cost categories.
One-time, fixed, or up-front costs.
Ongoing investment costs.
Recurring costs.
Compliance costs.
Mitigation measures.

Measuring Costs
Valuing indirect costs.
Flat overhead figure.
Does the project actually costs increased administrative
burden?

Costs to the private sector.


Valuing the use of capital.
Valuing the damage effects of government programs.
Other cost issues.
Sunk costs.
Interest costs.

Analysis of Benefits and Costs


Framework.
Retrospective.
Snapshot.
Prospective.

Analysis of Benefits and Costs


Importance of using present value.
Choice of an appropriate discount rate.
Private sector rate.
Low social discount rate.
Long-term treasury bill rates.

Adjustment for inflation.

Analysis of Benefits and Costs


Presenting the results.
Net present value (B-C).
Benefit cost ratio (B/C).
Return on investment (discount rate to reduce
present value to zero).
Appropriate perspective.
Costs and benefits vary across stakeholders.
May conduct analyses from several
perspectives.

Analysis of Benefits and Costs


Sensitivity analysis.
Use spreadsheet analysis to vary the assumptions.
Intangibles.
Relate intangibles to dollar results: if there are net
costs, do the intangible benefits overcome the deficit?
Particular problems.
Equity concerns (weighting of values).
Multiple causation and co-production problems.

The Public Policy-Making


Process
How to Deliver Oral Testimony
Based on a Written Statement

Introduction
Goal:
To speak authoritatively and to answer
questions responsively in public deliberation.
Objective:
Skill of writing speakable text, skill of speaking
easily from written text, and readiness to
answer anticipated and unanticipated questions.

Introduction
Scope:
Pinpointed topic pertinent to hearings purpose and the
witnesss role.

Product:
Two expected communication products:
Short oral summary.
Full written statement.

Strategy:
Confident and useful public testimony resulting from
advance preparation.

The Public Policy-Making


Process
How to Inform Policy Makers in a
Briefing Memo or Opinion Statement

Introduction
Goal:
Recognition of meaningful information in a mass of details
and representation highlighting the significance of
information for a user.

Objective:
Skills of distilling, listening, recording, observing,
evaluating sources, relating details to context, interpreting
details accurately in context, and selecting details according
to relevance; capability of stating informed opinion that is
aware of and responsive to other opinions.