Quantitative surveys: a short guide

A survey is a systematic quantitative method that aims to collect information from a

sample of the population such that the results are representative of that population
within a certain degree of error. Surveys are primarily used to collect information
(through the use of a structured and standardized questionnaire) on the perceptions,
opinions, knowledge, attitudes and behaviour of a sample using primarily closed-ended
questions. Surveys are not be used to explore opinions in-depth.

Instructions: The steps

1. Clarify the purpose of the survey

Figure out what you are trying to learn and have a clear image in your mind of the
research objectives. This will help you determine whom you will survey, formulate the
questionnaire and decide on the kind of statistical analysis you need to perform. If your
goals are unclear, the results will probably be unclear.

2a. Select your sample

Usually, the population will be too large for you to attempt to survey all its members. A
small, but carefully chosen sample can be used to represent the population. Sampling
is used to cut costs and effort while still obtaining information from a representative
sample of the target population. It is essential that the number of individuals
participating in the survey be large enough to produce results that are reliable and valid
and represent the target population with the minimum statistical error. If unsure, there
are some websites that help you calculate your sample size and your error (e.g.,
http://www.raosoft.com/samplesize.html, http://www.surveysystem.com/sscalc.htm,
http://www.researchinfo.com/docs/calculators/samplesize.cfm,
http://www.macorr.com/ss_calculator.htm, etc)
The main questions in selecting your sampling design are (apart from your budget of
course):
- Which are the demographics?
- What is the size of your target population?
- Are you interested in any sub-groups of the population? It is sometimes useful to
establish quotas to ensure that your sample accurately reflects relevant sub-groups
in your target population.
- How many will be included?

2b. Sampling methods

Sampling methods can be either probability (each member of the population has an
equal probability of being surveyed) or non probability (members are selected from the
population in some non-random way). The advantage of probability sampling is that

1
http://mechanisms.energychange.info
sampling error can be calculated. In non probability sampling, the degree to which the
sample differs from the population remains unknown.

Probability methods
Random sampling: the purest form of probability sampling. Each member of the
population has an equal and known chance of being selected. When there are very
large populations, it is often difficult or impossible to identify every member of the
population, so the pool of available subjects becomes biased.
Systematic sampling: After the sample size has been calculated, every Nth record is
selected from a list of population members. As long as the list does not contain any
hidden order, this sampling method is as good as the random sampling method.
Stratified sampling: The population is divided into groups of individuals that are similar
in some respect. After the population has been divided into the strata, a random
selection of a proportion of individuals from each stratum is made.
Cluster sampling: used if the target population is spread over a large geographic area.
The survey area (such as a territory) is divided into clusters. A random sample of these
clusters is drawn and individuals within the cluster participate in the survey.

Non probability methods

Convenience sampling: you simply take what is “convenient”. In this type of sampling
you cannot measure the degree of confidence because the group selected may not be
representative of the entire population. However, sometimes representativeness is not
as important as ensuring that you have specific individuals selected. This method is
often used during preliminary research efforts to get a grosso modo estimate of the
results, without incurring the cost or time needed to select a random sample.
Judgment sampling: you select the sample based on judgment. This is an extension of
convenience sampling. For example, a researcher may decide to draw the entire
sample from one representative area, even though the population includes all cities.
When using this method, you must be confident that the chosen sample is truly
representative of the entire population.
Quota sampling: it is the non probability equivalent of stratified sampling. Like stratified
sampling, you first identify the strata and their proportions as they are represented in
the population. Then convenience or judgment sampling is used to select the required
number of subjects from each stratum. This differs from stratified sampling, where the
stratums are filled by random sampling.
Snowball sampling is a special non probability method used when the desired sample
characteristic is very rare. It may be extremely difficult to locate respondents in these
situations. Snowball sampling relies on referrals from initial subjects to generate
additional subjects.

2
http://mechanisms.energychange.info
3. Decide on your survey method

The methodology will affect response rate, the kind of questions that can be asked and
how they are asked. When choosing the most appropriate method you must consider
the best way to communicate with potential respondents, skills of staff / interviewers,
the availability of internal resources and time available. The key is to choose a method
that will:
- give you the highest response rate with your particular target group
- be the most convenient for them
- fit your time line
- fit your budget
- fit your staff and resources

The primary methods for obtaining survey research are the following:

a) Personal / face-to-face Interviews

Advantages
- Usually gives a higher response rate
- Preferable for addressing complex issues where some explanation may be needed
or when or visual materials are shown
- Reduces non-response to individual questionnaire items
- The ability to find the target population. For example, you can find people who have
attended an energy-change initiative much more easily outside the location in
which it is taking place than by calling phone numbers at random.
- Longer interviews are more easily tolerated.
Disadvantages
- Social desirability bias
- More time consuming and expensive

b) Telephone interviews
Advantages
- It is possible to achieve high response rates
- The amount of non-response to questionnaire items can be minimized
- Quick results
- Less costly than face to face interviews
- You can dial random telephone numbers when you do not have the actual
telephone numbers of potential respondents
Disadvantages
- Sometimes difficult to reach a selected resident of a household
- Long or complex questions should be avoided
- Can be confused with telemarketing sales, which drops the response rate
- No working person is home during the day. This limits calling time to a "window" of
about 18:00-21:00.
- You cannot show visual material for assessment

3
http://mechanisms.energychange.info
c) Mail interviews (self completed)
Advantages
- Social desirability bias is minimized
- Cost effective
- The questionnaire can include pictures
- They are not intrusive
Disadvantages
- You need to ask simple, easy to answer questions, as it is self completed
- Some questions may not be complete on returned questionnaires
- Too much time is needed until all questionnaires are filled and sent back to you
- You can not be sure of who exactly has filled in the questionnaire
- You cannot reach populations of lower educational and literacy levels, or
immigrants
- Low response rate. One way of improving response rates is to mail a letter telling
your sample to watch for a questionnaire in the next week or two. Another is to
follow up a questionnaire mailing after a couple of weeks with a card asking people
to return the questionnaire. Another way is to use an incentive. Another possibility is
to include the people who return completed surveys in a drawing for a prize. A third
is to offer a copy of the top line results to those who complete the questionnaire.

d) Email Surveys (self completed)

Advantages
- Fast results
- Economical
- You can attach pictures and sound files
- The novelty element of an email survey often stimulates higher response levels
than ordinary mail surveys.
Disadvantages
- Email surveys do not reflect the population as a whole in cases with low internet
usage penetration, or practical exclusion of older target groups
- You cannot use email surveys to generalize findings to the whole population.
People who have email are different from those who do not, even when matched on
demographic characteristics, such as age and gender.
- You need to ask simple, easy to answer questions, as it is self completed
- You must possess (or purchase) a list of email addresses.
- Some people will respond several times or pass questionnaires along to friends to
answer
- People can easily quit in the middle of a questionnaire. They are not as likely to
complete a long email questionnaire as they would be if talking with a good
interviewer.
- Many people dislike unsolicited email.

e) Internet (Web Page) interviews (self completed)

4
http://mechanisms.energychange.info
Advantages
- Fast results
- Economical
- You can show pictures, video and play sound
- The novelty element often stimulates higher response levels
- You can use complex question skipping logic and randomizations
- Some people will give more honest answers to questions about sensitive topics
- On average, people give longer answers to open-ended questions on internet
questionnaires than they do on other kinds of self-administered surveys.
Disadvantages
- Internet surveys do not reflect the population as a whole in cases with low internet
usage penetration, or practical exclusion of older target groups
- You need to ask simple, easy to answer questions, as it is self completed
- People can easily quit in the middle of a questionnaire. They are not as likely to
complete a long questionnaire on the web as they would be if talking with a good
interviewer.
- Depending on your software, there is often no control over people responding
multiple times to bias the results.

4. Design your questionnaire

- The first step in writing a questionnaire is to determine if there is an existing

questionnaire (published literature or other organizations similar to your own) that
can be used to collect the information you want, even if you need to adjust it. If
there are no existing questionnaires available, then it will be necessary to design a
new instrument.
- The first rule is to design the questionnaire to fit the medium (i.e., telephone,
internet, etc)
- Start with an introduction or welcome message.
- Ideally, the early questions in a survey should be easy and pleasant to answer.
These kinds of questions encourage people to continue the survey. In telephone or
personal interviews they help build rapport with the interviewer. Grouping together
questions on the same topic also makes the questionnaire easier to answer.
- Avoid the temptation to ask questions because it would be "interesting to know". As
a general rule, avoid long questionnaires.
- Include other experts and relevant decision-makers in the questionnaire design
process.
- Use simple, comprehensive and to-the-point language. The questions must be
clearly understood by the respondent.
- Allow a “Don't Know”, “Other” or “None” response to questions. In most cases,
these are wasted answers as far as the results are concerned, but are necessary
alternatives to avoid frustrated respondents.
- Leave your demographic questions (age, gender, income, education, etc.) until the
end of the questionnaire.

5
http://mechanisms.energychange.info
Qualities of a Good Question
- Avoids emotionally charged or leading words and phrases that point towards a
certain answer.
- Asks for an answer on only one dimension. Do not put two questions into one. For
example if you ask “To what extent did the X initiative influence you to unplug your
TV when not in use and change your energy saving settings in your PC?”, the
respondent will not know to which of the two to answer. A good question asks for
only one "bit" of information.
- Can accommodate all possible answers so that you do not confuse and frustrate
the respondent. Add all the necessary response categories and allow multiple
responses.
- If you want only one answer from each person, ensure that the options are mutually
exclusive. A good question leaves no ambiguity in the mind of the respondent.
There should be only one appropriate choice for the respondent to make.
- Follows comfortably from the previous question. Transitions between questions
should be smooth.

Question Types
a) Open ended vs. closed questions
- In closed questions, response categories are provided and interviewers /
respondents simply have to circle or choose an option
- Open-ended questions are asked without specific response alternatives;
respondents need to write / say their own answer. They are good for depth and
unbiased opinions
- Open-ended questions are best used when having multiple response options may
be too leading and result in biased answers
b) Multiple-choice
- In such questions, respondents choose one or multiple response answers
- Make sure you include all the relevant alternatives as answer choices.
c) Scales (e.g., rating, ranking or agreement)
- Scales is an issue on which researchers differ; whether to use a scale with an odd
or even number of points. Some like to force people to give an answer that is
clearly positive or negative. Others feel it is important to offer a neutral, middle
option.
- Ranking example: Of the following 3 services, which one would you feel is most
important? Which one is second-most important? And which one is the third-most
important?
- Agreement example: On a scale of 1 to 5 where 5 is strongly agree and 1 is
strongly disagree how would you rate your agreement or disagreement with the
following statement…

If you need help designing the questionnaire, you can download a free questionnaire
wizard (e.g., http://www.dobney.com/Technology/QWizard.zip).

6
http://mechanisms.energychange.info
5. Pre-test the questionnaire – conduct pilot interviews

The last step in questionnaire design is to test it with a small number of interviews
before conducting your main survey. Ideally, you should test it on the same kinds of
people you will include in the main study. If that is not feasible, have a few people,
other than the question writer, try the questionnaire. Piloting can reveal unanticipated
problems with questionnaire length, wording, comprehension, question sequence,
instructions to skip questions, inadequate response alternatives in closed questions,
etc. It can help you see if the interviewees understand your questions and give useful
answers. After the pilots, you can revise the questionnaire, accordingly, if needed.

6. Conduct the interviews (for face-to-face and telephone)

A good interviewer follows a few simple guidelines which ensure detailed, accurate,
and unbiased data. Some tips follow:
- Read the questions as written, do not change the wording
- Do not suggest responses (with verbal or non verbal cues); allow the respondent to
answer without input from you.
- If respondents get off topic in open ended questions, you should remind them of the
question. Sometimes repeating the question will help. If the respondent has
difficulty thinking of a response, resist the temptation to “help”.
- Record answers neatly and thoroughly
- Occasionally, a respondent will want to terminate before completion. If this is the
case, you should make sure the interview gets done. This can be accomplished by
telling the respondent how important his/her opinions are and working together to
get through the interview quickly. If it is impossible to complete the interview,
arrange a time to call back at the respondent’s convenience to complete the
interview.
- For telephone interviews
o Do not simply hang up when someone refuses to participate. Politely ask
the respondent why they do not wish to take part. If they are busy you can
arrange a more convenient time to call back. If they are concerned about
the nature of the study, you can answer their questions. If they are nervous
about offering their opinions, you can reassure them that all information is
strictly confidential and there are no right and wrong answers.
o Try calling back 3 to 5 times before assuming you cannot reach the person
at that number.
o You should record on the sample list what happens every time you attempt
to do an interview with a person (refusal, absence, wrong number, ineligible
respondent, scheduled call-back, busy line, etc).
- For face to face interviews
o The most appropriate location to conduct a face to face interview is a place
where your sample frequents and is comfortable for them to participate. If

7
http://mechanisms.energychange.info
 you are randomly selecting respondents, it is important to utilise more than
one location in order to ensure a better representation.
o While at the location, select every Nth passer by to ensure randomisation

7. Process and analyse the data

Unless you have been entering responses directly into a computer programme,
processing the data involves preparing and translating the data for analysis. It involves
taking the completed questionnaires and putting them into a format that can be
summarised and interpreted.
Coding: separate answers to open ended questions into groups that can share a code
list. When no new responses are found, rewrite codes and assign a number to each
code.
Data entry and tabulation: data entry and tabulation can be easily done with computer
software packages. Write the corresponding codes beside each question on each
questionnaire. Then you can enter the data in a programme such as SPSS or Excel.
Most statistical applications have data entry capabilities.
Data analysis: once the data has been entered into your statistical package, the
analyses required to answer your research questions can be performed. For most
surveys simple descriptive statistics (frequencies, means, medians, ranges, etc) may
be all that is needed to be able to interpret your results. More complex analyses may
be required when comparisons are needed between subgroups or for measurements
taken at different times. Confidence intervals, t-tests (to compare results for continuous
data), Chi-square (to compare results for categorical data), Factor analysis, Cluster
analysis, and Correspondence analysis are some of the most common analysis
performed.

8
http://mechanisms.energychange.info
 Checklist for quantitative surveys

Remember to do the following:

 Clarify the research aims
 Decide whom you need to survey / target group
 Decide on the sample size
 Decide on your sampling method
 Choose the appropriate survey methodology
 Formulate the questionnaire
 Run some pilots
 Revise the questionnaire, if needed, according to pilot’s output
 Follow all necessary steps according to your selected methodology and conduct the
interviews
 Code your data
 Perform data entry
 Prepare tabulation of your data
 Analyse data based on research needs

9
http://mechanisms.energychange.info