Survey methodology

Cristina Giudici
University La Sapienza
Roma
References
Groves R.M. et al. Survey methodology, Wiley-
interscience, 2004
Canada national statistical agency:
http://www.statcan.gc.ca/
Eurostat methodological publications
European Values Study (EVS)
A survey usually originates when an individual or institution is confronted
with an information need and the existing data are insufficient
A survey is a systematic method of gathering information from (a sample
of) entities for the purpose of constructing quantitative descriptors of the
attributes of the larger population of wich the entities are members
INTRODUCTION :
What is survey methodology ?
What is a survey ?
Survey methodology is the study of survey methods
How survey work to produce statistics
Respondent
answers to
quenstions
Inference
Characteristics of a
respondent
Statistical
computing
Characteristics of
the sample
Characteristics of
the population
Inference
Chapter I

The life cycle of a survey
A survey from a process perspective
Define research objectives
Choose mode of
collection
Construct and
pretest a
questionnaire
Choose
sampling frame
Design and
select sample
Recruit and
measure sample
Code and edit data
Make postsurvey adjustments
Perform analysis
The life cycle of a survey from a
design perspective
Construct
Measurement
Response
Edited
Response
Survey statistics
Target Population
Sampling Frame
Sample
Respondent
Postsurvey Adjustments
The Measurement dimension describes
what data are to be collected about the
observational units in the sample
The Representational dimention
concerns what population are
described by the survey
What
is the
survey
about?
Who
is the
survey
about?
The measurement dimension
Constructs are the elements of information that are
sought by the researcher :
How many incidents of crimes with victims there were in the last year;
The consumption of beer in the last month;
The degree of knowledge of mathematics of childrens
Measurements are ways to gather information about
constructs :
Questions posed to a respondent (During the last 6 month, did you call the police
to report something that happened to you that you thought was a crime?)
NB: the critical task for maesurement is to design questions that produce answers
reflecting perfectly the construct we are trying to measure.
Response could be produced in a variety of means
But in general the nature of the response is determined by the
nature of the measurement
Editing of data may examine the full distribution of
answers and look for atypical patterns of responses
Edited responses are the data from wich inference is made about the values of
the construct for an individual respondent
Construct
Measurement
Response
Edited
Response
The representational dimension
The target population is the set of unit to be studied
The adult population living in households in 2009;
The frame population if the set of target population members
that has chance to be selected into the survey sample :
In a simple case it is a list of all units in the target population, but sometimes it is a set
of units imperfectly linked to population members.
i.e. a list of telephone numbers when the target population is the adult population
The sample is the group from wich measurement will be
sought. In many case it is a very small fraction of the the sampling frame
Postsurvey adjustments consist on weighting up the
underrepresented groups in order to improve the survey
estimate
Because of mismatches of the sampling frame and the target population
(coverage problems) statistics based on the respondents can differ from
caracteristics of the target population. Examination of non response patterns may
suggest an underrepresentation of some groupes relative to the sampling frame
Target Population
Sample
Respondent
Postsurvey
Adjustments
Sampling Frame
Respondents are the elements successfully measures.
Non respondents is the complement
The life cycle of a survey from a
quality perspective
Construct
Measurement
Response
Edited
Response
Survey statistics
Target Population
Sampling Frame
Sample
Respondent
Postsurvey Adjustments
Validity
Measurement
error
Processing
error
Coverage
error
Sampling
error
Nonresponse
error
Adjustments
error
Coverage of a target
population by a frame
Frame population
Target population
Undercoverage
Elements in the
target
population
missing from the
frame
i.e.:non telephone
household, using a
telephone frame to
cover the full
household population
Covered population
Ineligible units
Ineligible units
Elements in the
frame that are no
member of the
target population
i.e.:business telephone
numbers, using a
telephone frame to
cover the full household
population
Undercoverage
Evaluating survey questions:
Are the answers good measures of the
intended construct?
Exemple of methods that can be used
to evaluate draft survey questions
Expert reviews

The substantive expert review the wording, the
order and the structure of questions, the
response alternatives etc.
A small number of target population participate in a
systematic discussion about the survey topic. The
researcher learn about the nomenclature of the concept,
the common perspective taken by the target population
on key issues etc
Focus groups
Questionnaire pretest
Researcher test how questions are read
and answered. A behaviour coding is
often used
Evaluating survey questions:
Exemple of behavior codes for interviewer
and respondent behaviors
Interview Questioning Behavior (choose one)
1. Reads questions exacty as worded
2. Reads questions with minor changes
3. Reads questions so that meaning is altering

Respondent Behaviors (check as many as apply)
1. Interrupt question reading
2. Asks for clarification of question
3. Gives adequate answer
4. Gives answer qualifies about accuracy
5. Gives answer inadequate for questions
6. Answers dont know
7. Refuses to answer

Chapter II

Methods of data collection
Traditional
data collection methods
Mailing paper questionnaires to respondents,
who fill them out and mail them back
Having interviewers call to respondents on the
telephone and ask them the question in a
telephone interview
Sending the interviewers to the respondents
home or office to administer the questions in
face-to-face (FTF) interviews

Alternatives methods of data collection
Face to face
Telephone
Mail
CATI
computer assisted
telephone interviewing
CAPI computer assisted personal interviewing
TDE
Touchtone
data entry
OCR/ICR
Optical/intelligent
caracter recognition
FAX

Disk by Mail E-mail Web
Computerised Self
Administered
Questionnaires
IVR
Interactive
voice response
SAQ
Self administered
questionnaire

Walkman
Text
CASI

Audio
CASI

Video
CASI

Alternatives methods of data
collection (a)
OCR/ICR
Optical/intelligent
caracter recognition
FAX

Disk by Mail E-mail Web
Mail
Alternatives methods of data
collection (b)
CATI
computer assisted
telephone interviewing
TDE
Touchtone
data entry
IVR
Interactive
voice response
Telephone
Face to face CAPI computer assisted personal interviewing
SAQ
Self administered
questionnaire

Walkman
Text
CASI

Audio
CASI

Video
CASI

Alternatives methods of data
collection (c)
Chapter III

DESIGNING
A
QUESTIONNAIRE
QUESTIONS AND ANSWERS
IN SURVEYS

A questionaire
is a standardised set of questions administered to the
respondents in a survey
Respondents are required to interpret a preestablished set
of questions and to supply the information these questions
seek.
The cognitive processes in
answering questions
Comprehension
of the question
Retrieval of
information
Judgment and
estimation
Reporting an
answer
A simple model of the survey
response process
Responses could be biased by
acquiescence (the tendency to
agree)
social desirability (the tendency to
present oneself in a favourable light by
underreporting undesirable attributes
and overreporting desirable one)
BUT
Respondents often take shortcuts to get
through the interview more quickly
OR
they have motives that override their
desire to provide accurate information
The cognitive processes in
answering questions (2)
The satisficing model
(Krosnik and Alwin, 1987)
Some respondents try to satisfy (to take a low road answering more
superficially)
whereas others try to optimise (to take an high road by careful
answering questions)
Satisficing respondents do not seek to understand the question completely, just
well enough to provide a reasonable answer
Problems in answering
survey questions
Failure to encode the information sought
Misinterpretation of the questions
Forgetting and other memory problems
Estimation strategies
Problems in formatting answer
More or less deliberate misreporting
Failure to follow instruction
FORMATTING THE ANSWER
Survey items can take a variety of formats;
the most common are:
1) Open-ended qustions that call for
numerical answers
2) Closed questions with ordered response
scales
3) Closed questions with categorial
response options

1 - Open-ended qustions that call
for numerical answers
Now, thinking about your physical health,
which includes physical illness and
injury, for how many days during the past
30 was your physical health not good?
Note that:
Open-ended items yield more exact information than closed items
2 - Closed questions with ordered
response scales

Would you say that in general your health is:
1 Excellent
2 Very good
3 Good
4 Fair
5 Poor
The interviewer is instructed to
please read the answer
categories, but not the number
attached to them!
Note that :
with some type of rating respondents seem to shy away from the
negative end of the scale
When the scale points have numerical labels, the label can affect the
answer (e.g. if respondents are asked to rate their success in life)
3 - Closed questions with categorial
response options
Are you:
1 Married
2 Divorced
3 Widowed
4 Separated
5 Never married
6 A member of an unmarried couple
Note that :
The respondent may not wait to hear or read all the option; they may
select the firs reasonable answer they consider (primacy effect)
The opposite coul happen: the last option the interviewer read may be
the first one that respondent think about (recency effect)

GUIDELINES FOR WRITING GOOD
QUESTIONS (Sudman and Bradburn)
Non sensitive questions about behavior
The key problem with many questions about behavios is that
respondents may forget some or all of the relevant information,
or that their answer may reflect inaccurate estimate
In order to reduce memory problems it is essential to play
attention to the wording of the question and to provide
memory help
Attitude questions
Attitude questions are a very commen class of survey
questions. The most frequent problems deals with the
wording of questions, the question order and the format of
response scales
Non sensitive questions about behavior
Play attention to the wording
With closed questions, include all reasonable
possibilities as explicit response options
Are you:
Married
Divorced
Widowed
Separated
Never married
Are you:
Married
Single

Non sensitive questions about behavior
Play attention to the wording
Make the question as specific as possible
(about who it covers, what time period, which behaviours)
In a tipical
week, how
often do you
read a
newspaper?
Over the last month,
that is .. how
often do you read a
newspaper in a
tipical week?
Non sensitive questions about behavior
Play attention to the wording
Use words that virtually all respondents will understand
Have you ever
had a
miocardial
infarction?
Have you ever
had a heart
attack?
Non sensitive questions about behavior
Provide memory help
Uses aided recall
(or ask separate questions about subcategories)
Please look cerefully at the
following list of volountary
organisations: which, if any,
do you belong to?

A Religious organisations
B Cultural organisations
C Political groups
D Other
To which
volountary
organisation do
you belong to?

Attitude questions
Play attention to the wording
Clearly specify the attitude object of interest

Do you think the
Government is spending
too litte, about the right
amount, or too much on
education?

Do you think the
Government is spending
too litte, about the right
amount, or too much on
higher education?

Attitude questions*
play attention to the wording
Measure the strength of the attitute
using a response scale, a separate item
or multiple items that can be combined into a scale

*note that an attitude have generally a direction (agree or disagree)
and an intensity (strongly disagree.strongly agree)

Do you agree or disagree with the following statement?
Government is spending too little on education
1 Agree strongly
2 Agree
3 Neither agree nor disagree
4 Disagree
5 Disagree strongly

Attitude questions
reduce impact of question order



When asking general and specific questions about a
topic, ask the general question first
(otherwise, the answer to the general question is likely to be affected by the
number and content of specific questions)

When asking questions about about multiple items,
start with the least popular

(the unpopular questions are likely to seem even less appealing
when they follow more popular questions)
When asking general and specific questions about
a topic, ask the general question first
Please tell me whether or not you think it should be
possible for a pregnant woman to obtain a legal
abortion if: the woman wants it for any reason?
1. Yes
2. No
3. Don't know
Please tell me whether or not you think it should be possible
for a pregnant woman to obtain a legal abortion if: there is
a strong chance of a serious defect in the baby?
1. Yes
2. No
3. Don't know
USA GSS



Use closed questions for measuring aptitudes;
(open answers are difficult to code)

Use five-to seven-point response scales
and label every scale point
(verbal label ensure that interpret the scale in the same way)
Attitude questions
play attention to the response scale
Now is up to you!