Chapter 1 - Introduction to Surveys

1.0 Introduction
What is a survey? A survey is any activity that collects information in an organised and methodical
manner about characteristics of interest from some or all units of a population using well-defined
concepts, methods and procedures, and compiles such information into a useful summary form. A
survey usually begins with the need for information where no data or insufficient data exist.
Sometimes this need arises from within the statistical agency itself, and sometimes it results from a
request from an external client, which could be another government agency or department, or a private
organisation. Typically, the statistical agency or the client wishes to study the characteristics of a
population, build a database for analytical purposes or test a hypothesis.

A survey can be thought to consist of several interconnected steps which include: defining the objectives,
selecting a survey frame, determining the sample design, designing the questionnaire, collecting and
processing the data, analysing and disseminating the data and documenting the survey.

The life of a survey can be broken down into several phases. The first is the planning phase, which is
followed by the design and development phase, and then the implementation phase. Finally, the entire
survey process is reviewed and evaluated.

The purpose of this chapter is to provide an overview of the activities involved in conducting a statistical
survey, with the details provided in the following chapters and appendices. To help illustrate the teaching
points of this manual, the reader is encouraged to read the Case Study manual which takes the reader
through the planning, design and implementation of a fictitious statistical survey.

1.1 Steps of a Survey

It may appear that conducting a survey is a simple procedure of asking questions and then compiling the
answers to produce statistics. However, a survey must be carried out step by step, following precise
procedures and formulas, if the results are to yield accurate and meaningful information. In order to
understand the entire process it is necessary to understand the individual tasks and how they are
interconnected and related.

The steps of a survey are:

- formulation of the Statement of Objectives;
- selection of a survey frame;
- determination of the sample design;
- questionnaire design;
- data collection;
- data capture and coding;
- editing and imputation;
- estimation;
- data analysis;
- data dissemination;
- documentation.

A brief description of each step follows.

1.1.1 Formulation of the Statement of Objectives

One of the most important tasks in a survey is to formulate the Statement of Objectives. This establishes
not only the surveys broad information needs, but the operational definitions to be used, the specific
topics to be addressed and the analysis plan. This step of the survey determines what is to be included in
the survey and what is to be excluded; what the client needs to know versus what would be nice to know.

How to formulate objectives and determine survey content is explained in Chapter 2 - Formulation of
the Statement of Objectives.

1.1.2 Selection of a Survey Frame

The survey frame provides the means of identifying and contacting the units of the survey population.
The frame is in the form of a list, for example:
- a physical list such as a data file, computer printout or a telephone book;
- a conceptual list, for example a list of all vehicles that enter the parking lot of a shopping centre
between 9:00 a.m. and 8:00 p.m. on any given day;
- a geographic list in which the units on the list correspond to geographical areas and the units within
the geographical areas are households, farms, businesses, etc.

Usually, the statistical agency has the choice of using an existing frame, supplementing an existing frame
or creating a new one. The frame chosen determines the definition of the survey population and can affect
the methods of data collection, sample selection and estimation, as well as the cost of the survey and the
quality of its outputs. Survey frames are presented in Chapter 3 - Introduction to Survey Design.

1.1.3 Determination of the Sample Design

There are two kinds of surveys: sample surveys and census surveys. In a sample survey, data are
collected for only a fraction (typically a very small fraction) of units of the population while in a census
survey, data are collected for all units in the population. Two types of sampling exist: non-probability
sampling and probability sampling. Non-probability sampling provides a fast, easy and inexpensive way
of selecting units from the population but uses a subjective method of selection. In order to make
inferences about the population from a non-probability sample, the data analyst must assume that the
sample is representative of the population. This is often a risky assumption given the subjective method of
selection. Probability sampling is more complex, takes longer and is usually more costly than non-
probability sampling. However, because units from the population are randomly selected and each units
probability of selection can be calculated, reliable estimates can be produced along with estimates of the
sampling error and inferences can be made about the population. Since non-probability sampling is
usually inappropriate for a statistical agency, this manual focuses on probability sampling.

There are many different ways to select a probability sample. The sample design chosen depends on such
factors as: the survey frame, how variable the population units are and how costly it is to survey the
population. The sample design in part determines the size of the sample, which impacts directly on survey
costs, the time required to complete the survey, the number of interviewers required and other important
operational considerations. There is no magical solution and no perfect recipe for determining sample
size. Rather, it is a process of trying to fulfil as many requirements as possible one of the most
important being the quality of the estimates as well as operational constraints.
 3

The relative strengths and weaknesses of sample surveys and censuses are covered in Chapter 3 -
Introduction to Survey Design. Non-probability and probability sample designs are presented in
Chapter 6 - Sample Designs. Guidelines for determining the required sample size are covered in
Chapter 8 - Sample Size Determination and Allocation.

1.1.4 Questionnaire Design

A questionnaire (or form) is a group or sequence of questions designed to obtain information on a

subject from a respondent. Questionnaires play a central role in the data collection process since they
have a major impact on data quality and influence the image that the statistical agency projects to the
public. Questionnaires can either be in paper or computerised format.

Problems faced during questionnaire design include: deciding what questions to ask, how to best word
them and how to arrange the questions to yield the information required. The goal is to obtain information in
such a way that survey respondents understand the questions and can provide the correct answers easily in a
form that is suitable for subsequent processing and analysis of the data. While there are well- established
principles for questionnaire design, crafting a good questionnaire remains an art requiring ingenuity,
experience and testing. If the data requirements are not properly transformed into a structured data
collection instrument of high quality, a good sample can yield bad results.

Questionnaire design is covered in Chapter 5 - Questionnaire Design.

1.1.5 Data Collection

Data collection is the process of gathering the required information for each selected unit in the
survey. The basic methods of data collection are self-enumeration, where the respondent completes the
questionnaire without the assistance of an interviewer, and interviewer-assisted (either through personal
or telephone interviews). Other methods of data collection include direct observation, electronic data
reporting and the use of administrative data.

Data collection can be paper-based or computer-assisted. With paper-based methods, answers are
recorded on printed questionnaires. With computer-assisted methods, the questionnaire appears on the
screen of the computer and the answers are entered directly into the computer. One benefit of computer-
assisted methods is that data capture the transformation of responses into a machine-readable format
occurs during collection, thereby eliminating a post-collection processing activity. Another benefit is
that invalid or inconsistent data can be identified more easily than with a paper questionnaire.

Methods of data collection are covered in Chapter 4 - Data Collection Methods. The use of
administrative data is discussed in Appendix A - Administrative Data. Data collection activities,
including such interviewer activities as listing, tracing, and methods of organising data collection are
covered in Chapter 9 - Data Collection Operations.

1.1.6 Data Capture and Coding

After the data are collected, they are coded and, if a computer-assisted collection method was not used,
captured. Coding is the process of assigning a numerical value to responses to facilitate data capture
and processing in general. Some questions have coded response categories on the questionnaire, others
 4

are coded after collection during a manual or automated process. Data capture and coding are expensive
and time-consuming activities that are critical to data quality since any errors introduced can affect the
final survey results. Therefore, emphasis should be placed on error prevention in the early stages. Two
methods of monitoring and controlling errors are quality assurance and quality control. The purpose of
quality assurance is to anticipate problems and prevent them while the purpose of quality control is to
ensure that the number of errors that occur are within acceptable limits.

Data capture and coding are covered in Chapter 10 - Processing. Quality issues are covered in
Appendix B - Quality Control and Quality Assurance.

1.1.7 Editing and Imputation

Editing is the application of checks to identify missing, invalid or inconsistent entries that point to data
records that are potentially in error. The purpose of editing is to better understand the survey processes
and the survey data in order to ensure that the final survey data are complete, consistent and valid. Edits
can range from simple manual checks performed by interviewers in the field to complex verifications
performed by a computer program. The amount of editing performed is a trade-off between getting every
record perfect and spending a reasonable amount of resources (time and money) achieving this goal.

While some edit failures are resolved through follow-up with the respondent or a manual review of the
questionnaire, it is nearly impossible to correct all errors in this manner, so imputation is often used to
handle the remaining cases. Imputation is a process used to determine and assign replacement values to
resolve problems of missing, invalid or inconsistent data.

Although imputation can improve the quality of the final data, care should be taken in order to choose an
appropriate imputation methodology. Some methods of imputation do not preserve the relationships
between variables or can actually distort underlying relationships in the data. The suitability of the
method chosen depends on the type of survey, its objectives and the nature of the error.

Editing and imputation are covered in Chapter 10 - Processing.

1.1.8 Estimation

Once the data have been collected, captured, coded, edited and imputed, the next step is estimation.
Estimation is the means by which the statistical agency obtains values for the population of interest so
that it can draw conclusions about that population based on information gathered from only a sample
of the population. An estimate may be a total, mean, ratio, percentage, etc.

For a sample survey, the basis of estimation is the units weight, which indicates the average number of
population units it represents. A population total can be estimated, for example, by summing the weighted
values of the sampled units. The initial design weight is determined by the sample design. Sometimes,
adjustments are made to this weight, for example, to compensate for units that do not respond to the
survey (i.e., total nonresponse) or to take into account auxiliary information. Nonresponse adjustments
may also be applied to data from a census survey.

Sampling error occurs in sample surveys since only a portion of the population is enumerated and the
sampled units do not have exactly the same characteristics as all of the population units that they
represent. An estimate of the magnitude of the sampling error for each estimate should always be
 5

provided to indicate to users the quality of the data.

Estimation of summary statistics is covered in Chapter 7 - Estimation. Estimating sampling error is

discussed in Chapter 7 - Estimation and Chapter 11 - Analysis of Survey Data.

1.1.9 Data Analysis

Data analysis involves summarising the data and interpreting their meaning in a way that provides clear
answers to questions that initiated the survey. Data analysis should relate the survey results to the
questions and issues identified by the Statement of Objectives. It is one of the most crucial steps of a
survey since the quality of the analysis can substantially affect the usefulness of the whole survey.

Data analysis may be restricted to the survey data alone or it may compare the surveys estimates with
results obtained from other surveys or data sources. Often, it consists of examining tables, charts and
various summary measures, such as frequency distributions and averages to summarise the data.
Statistical inference may be used in order to verify hypotheses or study the relationships between
characteristics, for instance, using regression, analysis of variance or chi-square tests.

Data analysis is covered in Chapter 11 - Data Analysis.

1.1.10 Data Dissemination

Data dissemination is the release of the survey data to users through various media, for example,
through a press release, a television or radio interview, a telephone or facsimile response to a special
request, a paper publication, a microfiche, electronic media including the Internet or a microdata file on a
CD, etc..

Delivery and presentation of the final results is very important. It should be easy for the users to find,
understand, use and interpret the survey results correctly. Results from the survey should be summarised
and the strengths and weaknesses of the data indicated, with important details highlighted through a
written report that includes tables and charts.

Before disseminating data, a data quality evaluation should be performed in order to help assess and
interpret the survey results and the quality of the survey and to inform users so that they can judge for
themselves the usefulness of the data. It may also provide valuable input to improve the survey (if
repeated) or other surveys. This evaluation, and its accompanying report, should include a description of
the survey methodology along with measures and sources of error.

As part of the dissemination process, many statistical agencies are required by law to protect the
confidentiality of respondents information. Disclosure control refers to those measures taken to protect
disseminated data so that the confidentiality of respondents is not violated. It involves, for instance,
identifying and eliminating (or modifying) table cells that risk revealing information about an individual.
Usually, some data have to be suppressed or modified. Before choosing a disclosure control method,
various methods should be compared with respect to their impact on the survey results and an individuals
risk of disclosure.

Data dissemination is covered in Chapter 12 - Data Dissemination.

 6

1.1.11 Documentation

Documentation provides a record of the survey and should encompass every survey step and every survey
phase. It may record different aspects of the survey and be aimed at different groups, such as
management, technical staff, designers of other surveys and users. For example, a report on data quality
provides users a context for informed use of the data. A survey report that includes not only what
decisions were made, but also why they were made provides management and technical staff with useful
information for future development and implementation of similar surveys. During implementation,
documentation of procedures for staff helps to ensure effective implementation.

How to organise a report and guidelines for writing are covered in Chapter 12 - Data Dissemination.

1.2 Life Cycle of a Survey

The survey steps presented above are not necessarily sequential: some are conducted in parallel, others
for example, editing are repeated at different times throughout the survey process. Every step must first
be planned, then designed and developed, implemented and ultimately evaluated. The phases in the life of a
survey are described below.

1.2.1 Survey Planning

The first phase of the survey process is planning. But before any planning can take place, a management
and planning structure must be selected and implemented. One commonly used structure is the project or
survey team approach, whereby an interdisciplinary survey team is given responsibility for the planning,
design, implementation, and evaluation of the survey and of its planned products. The interdisciplinary
team is composed of members having different technical skills; for example, a statistician, a computer
programmer, an expert in the field of study, a data collection expert, etc.

Survey planning should be conducted in stages of increasing detail and exactitude. At the preliminary, or
survey proposal stage, only the most general notion of the data requirements of the client may be known.
Once a survey proposal has been formulated, it is important to determine whether a new survey is
necessary, keeping in mind options, costs and priorities of the client and the statistical agency. Sometimes
much or all of the information desired can be obtained from administrative files of governments,
institutions and agencies. Alternatively, it may be possible to add questions to an existing surveys
questionnaire or, it may be possible to redesign an existing survey.

If it is decided that alternative data sources cannot meet the information needs, the team proceeds to
formulate the Statement of Objectives and to develop some appreciation of frame options, the general
sample size, precision requirements, data collection options, schedule and cost. A decision about the
feasibility of the survey is usually made at this point.

After the objectives of the survey are clear, each team member prepares the component plans associated
with his or her responsibility within the team. During this stage, planning becomes a more complex
matter. The advantages and disadvantages of alternative methodologies should be examined and
compared in terms of: coverage, mode of data collection, frequency, geographical detail, response burden,
quality, cost, resources required and timeliness.
 7

In the later stages of the survey process, plans are revised, elaborated and refined, and more detailed
aspects are examined. Each and every activity and operation needs some form of plan for design,
development and implementation. Planning continues throughout the entire survey process with
modifications being made as required.

The details of planning are covered in Chapter 13 - Survey Planning and Management.

1.2.2 Design and Development

Having established a broad methodological framework, it is possible to carry out detailed work on the
various steps of a survey in, what is referred to as, the design and development phase. The overall
objective of this phase is to find the set of methods and procedures that achieve an appropriate balance of
quality objectives and resource constraints.

It is during this phase that any required pretests or pilot surveys are carried out to assess, for example, the
adequacy of the questionnaire, suitability of the survey frame, operational procedures, etc. All field
materials (e.g., interviewer training and instruction manuals, sample control documents) are prepared for
the data collection stage. Software programs for computer administered questionnaires are developed, or
adapted, and tested. Sample selection and estimation procedures are finalised in the form of
specifications. Specifications for coding, data capture, editing and imputation are all prepared to set the
stage for data processing.

To be effective, procedures should be designed to control and measure the quality at each step of the
survey (using quality assurance and quality control procedures) and to assess the quality of the final
statistical products.

1.2.3 Implementation

Having ensured that all systems are in place, the survey can now be launched. This is the implementation
phase. All survey control forms and manuals are printed, along with the questionnaire (if paper
questionnaires are used). Interviewers are trained, the sample is selected and information is collected, all in
a manner established during the development phase. Following these activities, data processing begins.
Processing activities include data capture, coding, editing and imputation. The result is a well-structured
and complete data set from which it is possible to produce required tabulations and to analyse survey
results. These results are then checked for confidentiality and disseminated. At every step, data quality
should be measured and monitored using methods designed and developed in the previous phase.

1.2.4 Survey Evaluation

Survey evaluation is an ongoing process throughout the survey. Every step of the survey should be
evaluated in terms of its efficiency, effectiveness and cost, particularly in the case of repeated surveys, so
that improvements in their design and implementation can be made over time. This involves assessments
of the methods used, as well as evaluations of operational effectiveness and cost performance. These
evaluations serve as a test of the suitability of the technical practices. They also serve to improve and
guide implementation of specific concepts or components of methodology and operations, within and
across surveys. They support the activities and provide measures and assessments of the quality
limitations of the program data. As well, each survey step is evaluated to provide insight into
 8

shortcomings or problems in other steps of the survey. For example, editing and imputation can provide
information on problems with the questionnaire.

Evaluations of previous surveys or pilot surveys are important when planning a new statistical activity:
they can help formulate realistic survey objectives, provide an idea of the expected quality of the data and
essential information for survey design and data processing.

1.3 Summary
What is a survey? A survey is any activity that collects information in an organised and methodical
manner. It is usually motivated by the need to study the characteristics of a population, build a database
for analytical purposes or test a hypothesis.

What are the steps of a survey? A survey is a much more complex procedure than simply asking questions
and compiling answers to produce statistics. Numerous steps must be carried out following precise
methods and procedures, if the results are to yield accurate information. These steps include formulating
the survey objectives, determining the sample design, designing the questionnaire, performing data
collection, processing and tabulating data and disseminating results.

How are the steps implemented? Execution of a survey can be described as a life cycle with four phases.
The first phase is planning during which the survey objectives, methodology, budget and schedule of
activities are established. The next phase is the design and development of the survey steps. The third
phase is the implementation of the survey steps. During implementation, quality is measured and
monitored in order to ensure that the processes are working as planned. Finally, the survey steps are
reviewed and evaluated.

Bibliography

Cochran, W.G. 1977. Sampling Techniques. John Wiley and Sons, New York.

Des Raj. 1972. The Design of Sample Surveys. McGraw-Hill Series in Probability and Statistics, New
York.

Moser C.A. and G. Kalton. 1971. Survey Methods in Social Investigation. Heinemann Educational Books
Limited, London.

Srndal, C.E., B. Swensson and J. Wretman. 1992. Model Assisted Survey Sampling. Springer-Verlag,
New York.

Satin, A. and W. Shastry. 1993. Survey Sampling: A Non-Mathematical Guide Second Edition.
Statistics Canada. 12-602E.

Statistics Canada. 1987. Quality Guidelines. Second Edition.

Statistics Canada. 1998. Statistics Canada Quality Guidelines. Third Edition. 12-539-X1E.