Next week
Please bring with you a questionnaire that you
Questionnaire Design
Questionnaires
Questionnaire's Impact on Data
Accuracy
It must communicate to the respondent what
the researcher is asking
It must communicate to the researcher what
the respondent has said
are hoping to run in the future, or one that
you ran in the past please bring 4-5 copies of
it, as we will make several groups and discuss
it in groups
Please also bring laptops if you have them
Questionnaire
A set of questions designed to generate the
data necessary for accomplishing a research
project's objectives
Sources of Distortion
A sloppy questionnaire can lead to a great
deal of distortion
The entry of an intermediary, or the
interviewer, into the communication channel
can also distort the questionnaire
1
 Questionnaire Design Elements The Questionnaire Design Process

Does each question have the most appropriate form (e.g.,

structured versus nonstructured)?

Is each question relevant and properly worded to obtain

meaningful, valid responses?

Is the sequencing of the questions likely to introduce any

bias?

Are the layout and appearance of the questionnaire

conducive to accurate and easy data collection?

10 | 7

Question Form Nonstructured Questions

Nonstructured questions Consider the following illustrative questions
Open-ended How old are you?___________________
What do you like most about owning your own
Structured questions home? _____________
Fixed-response Will you please describe your thoughts about a
person who shoplifts items from a grocery store to
keep from going hungry?
_________________________________________
_________________________________________
____

10 | 9

Structured Questions:
Structured Questions
Dichotomous Questions
In which of the following categories does your
age fall? Do you smoke cigarettes?
___Yes ___No
_______ Less than 18
_______ 18 to 30
_______ 31 to 45 Have you ever watched CNN News?
_______ 46 to 60 ___Yes ___No
_______ Over 60

10 | 11 10 | 12

2
 Multiple-Category Questions Multiple-Category Questions (Contd)
In your opinion, which product category is the most suited for What do you like about Bank of Americas
making purchases on the Internet?
online banking service?
____Automobiles ____24 hour service
____Books ____Transaction privacy
____Electronic Items (TVs, video cameras, VCR and
DVD players, etc) ____No need to travel do it from home
____Videos, CDs ____Paying bills online no need to mail bills
____Computer-related items
____Travel (airlines, car rental, hotel reservations, etc.) ____Other ____(please specify)

Response Category Sequence Response Category Content

Approximately how many long-distance Response choices should be
telephone calls do you make per week? Collectively exhaustive
______ 0 to 1 call taken together
______ 2 to 3 calls Mutually exclusive
______ 4 to 5 calls not overlap
______ 6 to 7 calls
______ More than 7 calls
The response categories in this question
follow a natural sequence

Response Category Question Relevance and Wording

The response choices are collectively exhaustive but not
mutually exclusive
Can the research
On the average, how many cans of cola do you drink per week
objective(s) be No
Yes
____0 to 3 cans fulfilled without
asking this
____3 to 6 cans question?
____More than 6 cans
Changing the 3 in the second category to a 4 will make the
response choices mutually exclusive The question may still
be a candidate for
question should be
revision or deletion
deleted
and must be further
examined

3
 Writing a Questionnaire for Will the Respondent
International Markets Answer the Question?
Caution
Use of the same word can have different Questions about personal financial matters or
meanings in different cultures sexual behavior are examples of sensitive
Word Equivalence questions
Think about how different people can think of Words used in the questions should not be
things in different way ambiguous to the respondents
In Chile, the word authorities does not mean the
national government, but the local police

Double-Barreled Questions Avoiding Double-Barreled Questions

Suppose the following question is to be used in conducting a survey of the This revision is usually accomplished by breaking one question
general public into several questions
Do you feel firms today are concerned about their employees and
customers?
_____ Yes _____ No _____ No opinion Thus our illustrative question can be reworded as two
separate questions
A no response can be interpreted three different ways Do you feel firms today are concerned about their employees?
The respondent feels firms are concerned about neither employees ___ Yes ___ No ___ No opinion
nor customers
Do you feel firms today are concerned about their customers?
The respondent feels firms are concerned about employees but not
___ Yes ___ No ___ No opinion
customers
The respondent feels firms are not concerned about employees
although they are concerned about customers

Leading Questions Avoiding Leading Questions

Don't you think offshore drilling for oil is The two leading questions we saw earlier can
environmentally unsound? be modified for neutrality as follows
___Yes ___ No ___ No opinion What is your feeling about the environmental
impact of offshore drilling for oil?
Do you think the quality of products on the ___ Offshore drilling is environmentally sound
market today is as high as it used to be ten ___ Offshore drilling is environmentally unsound
years ago? ___ No opinion
___ Yes ___ No ___ No opinion

4
 Avoiding Leading Questions (Contd) Unbalanced Questions
How do you feel about the quality of products on the market How important is price to you in buying a new
today compared with the quality of products on the market
ten years ago? car?

___Product quality is better now than ten years ago. ___More important than any other factor
___Product quality is worse now than ten years ago.
___Product quality is the same now as ten years ago.
___Extremely important
___ No opinion. ___Important
___Somewhat important
___Unimportant

Avoiding Questions with Implicit

Balanced Questions
Assumptions
How important is price to you in buying a new
car? Are you favorable, indifferent, or unfavorable toward a 10%
increase in city taxes?
___Favorable ___Indifferent ___ Unfavorable
___Very important
___Relatively important The above question can be improved as
Are you favorable, indifferent, or unfavorable toward a 10%
___Neither important nor unimportant increase in city taxes to repair potholes in the city's streets?
___Relatively unimportant ___Favorable ___Indifferent ___Unfavorable
___Very unimportant

Avoiding Questions with Implicit

Assumptions (Contd)

Do you drink coffee while watching football, baseball, and so

on?
____ Yes ____ No Sequencing of Questions
The above question can be reworded as
During which of the following activities do you drink
coffee?(Check as many as apply.)
___ Attending ballgames
___Attending sporting events other than ballgames
___Watching ballgames on TV
___Watching sporting events other than ballgames on TV

5
 Arrangement of Related
Skip Patterns
Questions
Cluster questions that focus on the same topic Proper sequencing of questionnaire items is
essential to avoid complicated skip patterns
Grouping questions into meaningful clusters can increase
respondents' ease in answering the questions and reduce the
that may confuse interviewers and
chance of response errors respondents

Question Sequence Needing

Improved Question Sequence
Improvement
11. Have you always lived in this prefecture?
11. Do you own or rent your current place of residence? Yes (Go to Question 12)
Own Rent No (Go to Question 11a)
12. How long have you lived in this prefecture? 11a. In which prefecture did you live immediately before moving into this
prefecture?
Less than one year
12. How long have you lived in this prefecture?
One year to less than 5 years Less than one year
5 years or more One year to less than 5 years
13. If you have lived outside this prefecture, in which prefecture did you live 5 years or more
immediately before moving here? 13. Do you own or rent the place where you live?
14. If you have lived in this prefecture 5 years or more and if you currently Own (Go to Question 15)
rent your residence, do you intend to buy a home within the next two years? Rent (Go to Question 14)
Yes No Dont know 14. Do you intend to buy a home within the next two years?
15. How long have you lived in your current place of residence? Yes No Dont know
15. How long have you lived in your current place of residence?
Less than one year
Less than one year
One year to less than 5 years One year to less than 5 years
5 years or more 5 years or more

Jargon
Jargon only has meaning for a small number of people.
Normal people in the street might have a difficult time
Choice of Words and understanding it Cohesion?
Questionnaire Design What do you think
Chemistry?
about the state of IT?
cohesion in your
Social
neighborhood?
Slides and ideas adapted from a presentation by science?

Dyah Fatma, 10th July 2013 Linguistic?

6
 Translating terms
One word in one language can be translated
into different words in another language.
cognition
Reliability and Validity
Slides and ideas adapted from a presentation by
Bulga and Tiffany Chua, 10th July 2013
Anecdotalism should be avoided
Make every effort to falsify our initial hunches
about relations between phenomena in our
data.
Only if the relationship cannot be refuted in
any way, we are in a position to talk about
objective knowledge.
Meaning of a word
Meaning of a word Does your word
represent what you really want to ask your
respondents?
Afraid = ?
Do you
Worried
feel afraid Scared
when you Nervous
walk alone
at night?
Anecdotalism
Research reports which appear to tell
entertaining stories or anecdotes, but fail to
convince the reader of their scientific
credibility (this is similar to journalism)
From the field notes, researcher selects a few
exemplary instances and not others. Typicality
or generality of the instances are questioned.
Reliability
There are three types of reliability referred to
in quantitative research, which relate to:
(1) the degree to which a measurement, given
repeatedly, remains the same
(2) the stability of a measurement over time;
(3) the similarity of measurements within a
given time period . (Kirk & Miller, 1986)
7
 Validity
Whether the means of measurement are accurate and whether
they are actually measuring what they are intended to measure
(Golafshani, 2003).
Quantitative researchers actively cause or affect the interplay
between construct and data in order to validate their
investigation, usually by the application of a test or other process.
In this sense, the involvement of the researchers in the research
process would greatly reduce the validity of a test (Wainer &
Braun, 1998). However, please note later the point on hiring
people!!
Valid and reliable, please!
Giving reliability to audio and video
data
Provide the real words of the subjects (access
to raw data)
Transcribe seemingly trivial but often crucial
pauses and overlaps in the recordings:
Mmm, yes, pauses etc.
Conversation analysis (CA) transcription
conventions are emerging: symbols and
concepts.
Reliability Vs. Validity
Reliable: Same observation goes to the
same category every time
A thermometer that shows the same
reading of 82 Celsius each time it is plunged
into boiling water.
Valid: Findings that cannot be proven
false. Very close to the truth.
A thermometer that gives differing readings
around 100 Celsius in a boiling water.
Can social studies be reliable?
Society is not like chemical elements. Its
properties are always changing. You cannot
measure 2 times and get the same results.
If society does not have any stable properties,
why do a research about it? The result will be
already false by the time the research is done.
Improving validity: Triangulation
Comparing different kinds of data
(quantitative and qualitative), and different
methods (observation and interviews) to see
whether they corroborate each other.
E.g. field strategy that combines document
analysis, respondent and informant
interviewing, direct participation, observation,
and introspection.
8
 Improving validity: Respondent
validation
Take the findings back to the subjects and
have them verify them.
But, subjects do not have the privileged
Quantitative vs. Qualitative
status as commentators on their own actions.
So such feedback cannot be taken as direct Slides and ideas adapted from a presentation by
validation or refutation of the observer`s Bulga, Chihiro and Tiffany Chua, 10th July 2013
inferences.
Rather such so-called `validation` do generate
further data and analysis

Quantitative VS. Qualitative Quantitative vs. Qualitative

Quantitative: Quantitative research
In survey research one is satisfied by achieving Pros
significant correlation. If nearly all your data support Effective method for gathering
your hypothesis, the job is done. information about selected
characteristics of
1. Members of a group
Qualitative: 2. The group as a whole
Qualitative research
Pros
Effective method for drawing
distinctions of kind rather than
amount and complexities of the
lives of individuals, groups, and
institutions

Comprehensive data treatment: Cons Cons

Small datasets inspected thoroughly. Not effective method for Not effective method for
A precise model that comprehensively describes the describing the patterning of generalizations
phenomenon, instead of a simple correlational statement qualities and the complexities Trustworthiness
of historical, cultural, and social Objectivity
about antecedent and consequent conditions. developments

Lying
Reliability of answers is a problem
Other stuff that I have learnt from People who want to show off = Lying
Especially if something sensitive. Ex.) personal income
the 2013 course Bias, especially when samples are not random or under
incentives.
Slides and ideas from various students who took the
Keep your format clear and as simple as possible
course in 2013
messy and complicated questionnaires decreases the
respondents motivation. This can lead to less reliable
results.

54

9
 Adapting questionnaires from other
people
Using questionnaires from other people
-Best if you have never made a questionnaire before.
-Already pre-tested.
-No problem as long as you reference the source.
Improve your own questionnaires!
-Second questionnaire should be better than first
-However you want to build it in a way that you can still
use results of first questionnaire.
55
Pre-testing, Computer
Questionnaires
Creating Online Questionnaires
Creating an online questionnaire is simplified
by a growing number of websites that allow
researchers to design and launch their survey
from one site
Google documents allows you to easily create
such questionnaires is an example of one such
site
Various other points
-Questionnaires should be based on theoretical
underpinnings i.e. informed by literature review
-Sample size is very important to ensure statistical
significance of the data
-Appropriate sampling technique is crucial to avoid biases
e.g. purposive, random, snowball sampling
-Hiring people to administer questionnaires is not good idea
as some sometimes answer the questionnaires by
themselves which is against research ethics
-Pre-testing of questionnaires is important to ensure validity
56
Pretesting
Pretesting is administering a questionnaire to
a limited number of potential respondents
and other individuals capable of pointing out
design flaws
Why pre-test?
Provide you with additional ideas to improve your
questionnaire
Determine how long it takes to finish it (avoid
questionnaires longer than 20-30m)
Improve the design
Pick up on mistakes highlighted in this presentation!
Example on how to create a
google questionnaire
Who knows how to do this???
Shall I explain now how to do one??
10