Citizen Report Cards A Presentation on Methodology

Participation and Civic Engagement Group, Social Development Department, The World Bank

A SIMPLE REPORT CARD

NAME: ABCD SUBJECT MAX. MARKS
English Mathematics Chemistry Physics Biology Total 100 100 100 100 100 500

CLASS: IX-B PERFORMANCE

50 95 80 84 70 379

Rank in Class: 7 (out of 35)

USE OF REPORT CARDS

Simple evaluation tool Provides summative feedback on performance Comparison on performance across different (though often related) areas Structured for simple communication Starting point for reflection & corrective action Scope for comparison with peers

THE CITIZEN REPORT CARD

A powerful instrument of collective public voice at all levels of government and all stages of democracies A survey that solicits and aggregates user perceptions on quality of public services (not an opinion poll) Exacts public accountability creates competition between monopolies! Provides signals to agencies and pressures for change Requires a skilled combination of 4 things: 1) understanding of socio-political context 2) technical competence 3) media and advocacy campaign 4) institutionalizing

CONTENT OF REPORT CARDS

Feedback from actual users of services regarding
Availability of service Access to the service Reliability of the service Quality of the service Satisfaction with service Responsiveness of service provider Hidden costs - corruption & support Willingness to pay Quality of life

systems

REPORT CARDS PROVIDE CONCLUSIONS ON

Citizen satisfaction with quality of each service Comparison of service providers on reliability and user satisfaction Responsiveness and quality of problem solving by agencies Estimate of hidden costs incurred by citizens

IN STRATEGIC TERMS
WEAK PROCESSES REFORM POOR SERVICE REPORT CARDS DISATISFACTION ADMIN. DEMAND IMPROVEMENT MOBILISE

Eg. SATISFACTION WITH SERVICES

(Survey in 1999 with 1036 households) Agency No. of users % satisfied %dissatisfied Power Company 1027 47 14 Water Board 871 42 18 Food rations 793 52 10 Telephone Co. 424 67 07 Driving License 174 32 27 Urban Authority 92 16 38 Police 65 34 26

Eg. RANKING OF SERVICES BY CITIZEN SATISFACTION

(Survey in 1999) Agency Rank Power Company 3 Water Board 4 Food rations 2 Telephone Co. 1 Driving License 6 Urban Authority 7 Police 5

Eg. AGENCY RESPONSIVENESS

(Survey with households who had a problem) Agency No. of visits % satisfied with staff Power Company 3 51 Water Board 4 54 Food rations 3 60 Telephone Co. 3 28 Driving License 2 31 Urban Authority 8 15 Police 2 33

Stages Involved in a Citizen Report Card Initiative:

I. II. III. IV. V. VI. Identifying Scope, Actors and Purpose Choosing an Approach Designing Questionnaires Sampling Execution of Survey Data Analysis

VII. Dissemination of Findings VIII. Institutionalization

Key Points in Each Stage:

I. Identifying Scope, Actors and Purpose
What do you want to know? About whom? How will you use the information? Who will pay the bill? Mix of quantitative and qualitative techniques Approach should reflect type of data to be collected Focus group interactions with users and providers to decide priorities Frame valid and effective questions use lead in, qualifying, warm-up and demographic questions Avoid bias in instructions, wording, order, etc. Pre-test piloting will reveal unexpected responses!

II. Choosing an Approach

III. Designing Questionnaires

Key Points in Each Stage:

IV. Sampling
Stratification geographic, usage based Method: Quota Sampling, use of booster interviews Choose Respondents Household head, women, children

V. Execution of Survey Training of survey personnel/enumerators Pre-test enumerators run Pilot Survey Group Training & Manuals Random spot monitoring VI. Data Analysis Consolidate data Data Analysis averages, data ranges, frequencies, etc Data presentation tables, charts and figures

Key Points in Each Stage:

VII. Dissemination of Findings Constructively critical or reform oriented Wide media coverage radio, TV, newspapers Orientation for journalists Interface between users and providers Internet VIII. Institutionalization Should not be one-off exercise Performance based resource allocation between districts/sectors/states Performance based service contracts for facilities Information from Report Card feeds Planning/ Policy Making/Election campaigns

Diverse Applications of the Report Card

Urban service studies 7 Indian cities, Kampala Report Card by Uganda Management Institute Provincial & national studies on service delivery India & the Philippines, part of NSDS in Uganda Sector studies - public hospitals, Bangalore Program evaluation rural food security, Tamil Nadu, India, rural water and sanitation, Maharashtra, India Governance reform projects - Bangladesh, the Phillipines, Peru, Sri Lanka, Ukraine & Vietnam

Case Example: Millennium Report Card

Objective:
To provide independent assessment of key public services based on peoples feedback, and observation in 24 states across India Build baseline database and benchmarks to track progress

Focus:
5 basic public services drinking water, health, PDS, education and road transport 4 areas for measuring effectiveness Access, Usage, Reliability/Quality, and Satisfaction

Case Example: Millennium Report Card

Sampling:
24 states, 115 districts, 36,542 households of which were rural 2,304 villages profiled & 12,256 public facilities observed Stratified by rural-urban, village size & town class, and socio-cultural regions (for rural sample) Reliability of findings at 90% confidence at lowest cut Analysis also done by rural/urban, socio-cultural regions, village class, caste & income-capability index.

Case Example: Millennium Report Card

Some Key Findings:
(All India)

1. Health Access: Access to Medical Facility within 3km: 73% Quality/Reliability: Presence of doctors at public health facility: 70% Satisfaction: Full satisfaction with behavior of doctors: 15%

Case Example: Millennium Report Card

Some Key Findings:
(All India)

2. Education
Access: Access to Educational Facility within 1km: 66% Satisfaction/Quality: Full satisfaction with quality of physical infrastructure in primary schools: 10% Full satisfaction with the behavior of government primary school teachers: 16%

Case Example: Millennium Report Card

Some Key Findings (contd.):

3. Water:
Access: Access to protected public drinking water source within 100 m: 55 % Quality/Reliability: No breakdown of public drinking water source: 76% Satisfaction: Full satisfaction with adequacy & quality of drinking water: 22%
Source: The State of Indias Public Services: Benchmarks for the New Millennium, presentation by Public Affairs Center, April 2002.

Case Example: Millennium Report Card

Source: The State of India's Public Services: Benchmark for the New Millennium, presentation by Dr. Samuel Paul, April 25, 2002.

Case Example: Millennium Report Card

Measuring Quality of Governance and Rural Infrastructure of States

State

Rural State Infrastructure Ranking

1 2 3 4 5 6 7 8 Maharashtra Gujarat Himachal Sikkim Tamil Nadu Karnataka West Bengal Haryana

Governance Parameters Ranking

1 2 3 4 5 6 7 8

Kerala Maharashtra Tamil Nadu Karnataka Gujarat Punjab Haryana M.P.

Stages of Impact by CRCs

Awareness & acceptance of user feedback in govt. & Civil society Consultation with communities by agencies / govt. in local planning Internal monitoring systems of government start recording user feedback Independent assessment institutionalized into performance mgmt. Community feedback integrated into local governance

Key Enabling Conditions

Leadership & policymakers use hard data Networking & discussion in civil society Practice of dialogue between service agencies and communities Culture of comparison and debate Poor performance triggers political action by political leadership Agenda has longer term time frame Local institutions with integrity and competence for survey & advocacy

Issues & Limitations

Role of expectations - variations in response to service standards Limitations in comparability across services Capability & cost considerations Large sample required for heterogeneous population and lesser used services Effort & time to stimulate action by service agencies & civil society Lack of predictability in how different players respond

Thank you!

Questions?

I. Identifying the Scope, Actors & Purpose

1. What do you want to know?
What are the issues or problems What does the community have to say about these issues Can this information add value to existing studies and current action Do you want to focus on a single utility or server provider Do you want to gather comparative information from a wide range of utilities What is the purpose of the study Will it reshape current programs Will it be used to consider developing new programs Who will see the results of the study

4. How can the data be obtained?

What methods How to identify the specific population to be measured What specific tests, measures and questionnaire items are needed to arrive at the desired information Will you use random sampling and/or focus group What level of skill do staff and fieldworkers require Do you have enough resources to finance say a 12 week-12 person project Are there resources for contingencies

5. How will you collect the data?

-

2. About whom?
-

3. How will you use the info.?

-

6. Who will pay the bill?

-

II. Choosing an Approach

Approach should reflect the kind of information you want to gather Report card methodology is rooted in quantitative research methods but enhanced by qualitative findings from interviews and observations Report card methodology uses and mixes two primary research techniques: focus groups and questionnaires
Helps reduce bias Reveals errors in measurement Verifies and cross-checks data Increases response rates

Set general parameters related to sampling units, sample size, and the method of interviewing. Each decision has cost implications

Choosing an Approach (contd.): Qualitative and Quantitative Research Methods

Qualitative Research Is a method of information gathering that conveys feeling or insights. It is based on a small sample (max. 30 people). It mainly uses observation and unstructured interviews. It is used to: Generate hypotheses Clarify issues prior to undertaking quantitative studies Assess citizen perceptions of public utility or service provider Examine emotional responses of citizens to interaction with public service agency and agents Quantitative Research Is based on statistical principles. It uses sampling methods, questionnaires and computer-based data processing to answer questions of how much, who, where and when. It tends to be more expensive and time-consuming , but provides a certain degree of reliability. It is used to: Establish the level of citizen satisfaction or dissatisfaction with public service providers Rank order the agencies to the level of public satisfaction or dissatisfaction ratings Purpose options and reform with the weight of quantitative backing

III. Design of Questionnaires

Focus group interactions with users and providers they decide priorities Frame valid and effective questions use lead in, qualifying, warm-up and demographic questions Checklist:
Specify and rank order the information objective of the survey from most important to least important Enumerate the kinds of information needed from respondents that relate to each information objective Rank the items in each topical group in the order of their importance to the study For each item in each group, answer: why, who, how, and what Place the most interesting item at the beginning of the questionnaire

Avoid bias in instructions, wording, order, etc. Pre-test piloting will reveal unexpected responses!

Examples for Designing Questionnaire

INVESTIGATOR INFORMATION: Name & date of survey LEAD IN QUESTION - Good morning. We are from abc college and collecting data about the work of the new health facility. FILTER QUESTIONS Has anybody in your family used the facility?(Yes/No) DEMOGRAPHICS How old are the members of your family (GRID) AWARENESS: Do you know that you can get free medicines? (Y/N) AVAILABILITY: Does your village have the health facility? (Y/N) EASE OF ACCESS: How far away is the facility from your home? USAGE: Which facility do you normally use for major illness? Dist. Hospital/Private doctor/Village health facility

Examples for Designing Questionnaire

RELIABILITY: Was the doctor present when you went for treatment most recently? Yes/No QUALITY: How long did you have to wait to meet the doctor? . BEHAVIOUR: What do you feel about behaviour of the doctor in treating you? Fully satisfied, somewhat satisfied, not satisfied SATISFACTION: What do you feel about quality of treatment? Fully satisfied, somewhat satisfied, not satisfied REASONS: Why are you dissatisfied? COSTS: How much did you pay for treatment? .. HIDDEN COSTS: Did you pay any extra money or bribe for getting medicines from the health facility? Yes/No SUGGESTIONS

IV. Sampling
Define Population and decide Sample Size No fixed rule, but budget matters Stratification geographic, usage based Method: Simple Multi-stage random sampling or Quota Sampling use of boosters Choose unit of analysis household/individual Choose respondents Household head, women, children

Sampling - example
State sample of 1500 divided into rural and urban based on proportions of population Districts ranked on development index; 3 districts each selected from above & below median clusters Villages in selected districts stratified by size a, b & c; Sample of 10 villages divided as per frequency, and selected randomly from each category Sample of 25 households per village Social map of village to divide village sample in proportion of communities & spread over the area. Right hand rule from starting point with 5th house Reliability at 90% confidence at district level

VIII. Institutionalization
Should not be one-off exercise Implementing agency CSO (India), Public Sector (UK), Oversight agency (US) Report Card becomes basis for performance based resource allocation between districts/sectors/states Performance based service contracts for facilities Information from Report Card feeds Planning/ Policy Making/Election campaigns Setting up of Government Task Force and Citizens Forums

Applications of the Report Card

Case Example
Bangalore Report Card Karnataka Report Card Millennium Report Card (India) Filipino Report Card

Level/Area of Focus
Urban Metropolitan Services State Level National Level

Application
Comparison of services provided by municipality Comparison across 5 sectors in State Comparison of performance of different States for 5 basic services Evaluation of performance of pro-poor services

National Level public Services

Case Example 2: Karnataka Report Card

Aim of Study:
Provide independent assessment of key public services based on peoples feedback, and observation Build baseline database and benchmarks to track progress Create public awareness and mobilize responsiveness

Focus:
5 basic public services drinking water, health, PDS, education and road transport 4 areas for measuring effectiveness Access, Usage, Reliability/Quality, and Satisfaction

Case Example 2: Karnataka Report Card

Sampling:
State divided into 4 regions 2 districts selected from each region, 1919 households covered Sample had urban and rural mix

Findings:
Large expenditure on public services not a guarantee for easy access, quality or efficiency. State has good scores on water, education and transport State weak in area of health care, and PDS lack of access and transparency

Case Example 2: Karnataka Report Card

Sample of Indicators used to assess performance with Drinking Water
Public Use Use Protected Unit Public Private Source Source Source within 100m
Dist 1 Dist 2 Dist 3 Dist 4

Breakdown of public taps within 3 months

Breakdown of public handpumps within 3 months

Households involved in maintenance

Willingness to pay more

Access/Usage

Quality/Reliability

Case Example 2: Karnataka Report Card

Sample of Findings related to performance of Drinking Water: Willingness to pay around 57% of sampled households willing to pay for better services Satisfaction 38% of sampled households using protected sources were satisfied Reasons for dissatisfaction: (a) Quality of water not good (83%), (b) Supply was inadequate (36%), (c) Supply was irregular (27%)
Source: The State of Karnatakas Public Services: Benchmarks for the New Millennium, Public Affairs Center, Bangalore, 2002.

Case Example: Filipino Report Card

Context: Assesses the performance of selected government services based on client feedback Focus on the Lingap Para sa Mahihirap (Care for the Poor) Program Conducted by Social Weather Station and World Bank Country Team

Case Example: Filipino Report Card

Summary Findings: Non-poor use health facilities more than the poor Primary health facilities are frequently bypassed High prices of medicines are a burden Children drop out of school for health and economic reasons Class size, textbooks and facilities are rated poorly Drop in client satisfaction with private schools 2 out of 5 Filipinos do not get water from formal sources Access to housing programs is very limited

Case Example: Filipino Report Card

Case Example: Filipino Report Card

Case Example: Filipino Report Card

Case Example: Filipino Report Card Institutionalization:

Department of Budget and Management (DBM) has
started program to develop performance-based indicators based on Report Card DBM has agreed to contract out the report card to independent CSOs An advisory panel comprising CSO representatives, key government oversight agencies, service providers and the private sector will be convened to guide the CRC exercise