Preparing a Formal Survey - Guidelines

1 2 What are formal surveys? .....................................................................................2 Steps in a formal survey .......................................................................................2 2.1 2.2 2.3 3 4 Prepare ..........................................................................................................2 Implement ......................................................................................................2 Analyse ..........................................................................................................3

Defining the objectives of the survey ....................................................................3 Hypotheses ...........................................................................................................3 4.1 4.2 4.3 4.4 The unit of analysis........................................................................................4 The variables and indicators..........................................................................4 Values............................................................................................................5 The nature of the relationship ........................................................................5

Choosing a sampling technique............................................................................6 5.1 5.2 5.3 5.4 Simple random sampling ...............................................................................6 Multi-stage sampling......................................................................................6 Stratified sampling .........................................................................................6 Grid sampling.................................................................................................7

6 7 8

Drawing up a sampling frame ...............................................................................7 Defining the sample size.......................................................................................7 Designing the questionnaire .................................................................................8 8.1 8.2 Formulating the questions .............................................................................8 Processing the data. ......................................................................................9

9 10

Precautions in interviewing .................................................................................10 Acknowledgements .........................................................................................10

ICRA Learning Materials - Preparing a Formal Survey Guidelines - 1/10

1 What are formal surveys?

Quantitative or formal surveys are methods based on: Some form of random sampling of units (e.g. farms) Standardised questionnaires measuring the same variable characteristics (e.g. acreage, nitrogen use per ha, number of milking cattle, net farm income, etc.) of all units in the sample. This combination of random sampling of units and standardised measurements of variable characteristics (called variables) of these units allows statistical extrapolation of the results to the population from which the sample was drawn. Quantitative methods also allow the statistical test of hypotheses about relations between variables measured for all units in the sample. These methods are called quantitative, because they allow extrapolation, not because they collect quantitative information. Quantitative surveys collect mainly information that either numerical (e.g. number of sheep) or information that can be translated into numbers (e.g. gender of the household head: male = 0, female = 1).

2 Steps in a formal survey

An important difference between quantitative surveys and informal methods is the rigidity of the procedure. A quantitative survey is carried out in a number of linear steps:

2.1

Prepare
Define objectives Formulate hypotheses (final) Define unit of analysis Define variables Identify indicators & values (operationalise) Choose sampling procedure Make sample frame and draw sample Construct and pretest questionnaire Elaborate coding procedure Prepare tables, charts to be used for analysis Train interviewers Inform the population and relevant authorities

2.2

Implement
Collect and check data Encode the data Put data into matrix

ICRA Learning Materials - Preparing a Formal Survey Guidelines - 2/10

2.3

Analyse
Analyse and extrapolate frequency distribution, average, mean, mode, standard deviation, etc. Carry out a statistical test of hypotheses (to establish the relationships between variables)

Only during the preparation phase is there some room for iteration. Once the implementation has begun, there is very little one can do to readjust the process. Any major readjustment during the implementation would jeopardise the rigorously standardised procedure of measurement on which the whole concept of quantitative surveys is based.

3 Defining the objectives of the survey

Before you carry out a formal survey, you should have carefully defined the objectives and the information that you need, and have determined that a formal survey is the right method to gain that information. Formal surveys are mainly used when statistical inference is necessary to verify a hypothesis, and when quantitative comparisons are necessary. In ARD, quantitative surveys are exclusively used when we want to numerically extrapolate the results to a larger population. This is the case: When we want to estimate with a certain degree of accuracy e.g. how many farms in an area fall in each of a number of categories (e.g. 0-5 ha, >5-10 ha, >10 ha; dairy vs. fruits vs. pigs; female or male headed; etc.). When we want to obtain a statistical test of hypotheses about the relationships between different variables in a population (e.g. female headed farms use less nitrogen per ha than male headed farms). In all other situations - for gaining an insight into stakeholder perceptions, or to generate hypotheses, for example - more informal methods may more appropriate. In these circumstances, it makes no sense to subject oneself to the rigour of formal surveys, which seriously limits the flexibility of researchers to iterate and adjust the process in the light of new information gained. If done at all, quantitative surveys need to be done well. A badly done formal survey has neither the flexibility of informal approaches, nor the rigour to convince critics that the extrapolations are valid. If done well, formal surveys are time consuming. In a rapid appraisal, when time is short, one should seriously consider whether the results obtained justify the time and effort needed for a quantitative survey. This learning resource focuses on the use of quantitative surveys to verify hypotheses developed previously from analysis of secondary information or primary information gathered with informal methods.

4 Hypotheses
A hypothesis, as the term is used here, is a proposition about a relationship between two or more variables. Within the ARD procedure, such hypotheses usually concern:
ICRA Learning Materials - Preparing a Formal Survey Guidelines - 3/10

The numerical importance of different categories of farms in a typology, and/or The relationships between the distinguishing characteristics of these categories, (i.e. between the variables defining the typology). The ultimate objective of such a verification process is to check the findings made earlier in the ARD process with informal methods, which have usually been obtained at a limited number of farms or localities. It may be necessary to verify that these earlier findings are representative of the larger population for which the study aims to develop research recommendations. Four aspects need to be well defined in a formal survey used to verify (or falsify) a hypothesis. These are:

4.1

The unit of analysis

A unit of analysis the unit to be sampled - can be anything. In an ARD study it can be a village or other geographical, administrative or social community, a household, a person, a farm, a field, a crop, a herd, an animal, etc. As we are mainly concerned here with farm typologies, the unit is more likely to be the farm, the farmer or the household. We call it unit of analysis because we consider it to be the object of research. In a quantitative survey the information needed on this unit may well have to come from another entity. If the farm is the unit of analysis, one probably has to obtain information about the farm from the farmer, his wife (wives), and possibly even others, such as labourers on the farm, etc. The survey questionnaire is then addressed at these people. But that does not mean that they then become the unit of analysis. This is still the farm. Defining the unit of analysis can be quite tedious. It may be worth asking other people in your group (from other countries and cultures) to define a farm or a household and you will see that it will not be easy to define these units clearly for statistical purposes.

4.2

The variables and indicators

These are characteristics of the unit of analysis that can take on different values in which we are interested. Returning to farm typologies, these variables can be a combination of: Structural variables (farm acreage, capital, labour, irrigation, etc); Functional variables (various aspects of the operation of the farm, landuse, off-farm activities); Variables regarding the farm environment (agroecology, market access, settlement pattern); Problems that characterise and distinguish these farm categories; Aspects of the farm that influence the feasibility of specific solutions for these problems. When we have managed to identify the variables, we will often find that these have been defined at an abstract, conceptual level. They are constructs of the mind, but cannot be observed as such in the real world. To make them measurable we have
ICRA Learning Materials - Preparing a Formal Survey Guidelines - 4/10

to relate them to characteristics of the farm or household that we can observe. These measurable proxies for the variable are called indicators. For example, if you want to test the hypothesis that soil quality maintenance is poorer on large than on small farms, the first variable can e.g. be indicated by the average annual amount of money spent on this maintenance. This amount, of course, is not the maintenance itself, but it is an indicator. This one indicator may not be precise enough and you may need to combine it with other indicators: some farmers may compensate for low cash expenditure by spending more labour time on soil quality maintenance, for example. Sometimes one variable is therefore measured by a combination of various indicators, combined in what is called a scale. Finding indicators that are proxies for variables is called operationalisation. This is a complex step that needs a lot of attention. Suppose you want to investigate if farmers who own their land maintain it better than farmers who rent their land or are sharecroppers. Owning or renting may seem simple variables, but what if the ownership appears to be collective (e.g., a group of brothers), or if they rent from their parents (so that they will later inherit the land)?

4.3

Values

Indicators can take on different measurable values for different units at the same time or for the same unit at different times. These values are not always quantitative or numerical. Some may be nominal (e.g. male/female). Others only allow you to put the measured units in a certain order (e.g. from less to more), without you being able to say exactly how much less/more each farm scores compared to the others. Some indicators are actually numerical (e.g. average annual amount of money spent on soil quality maintenance per ha of cultivated land). Defining indicators and the values that these indicators can take on is equivalent to defining the operations by which a variable can be measured. In social science this is called an operational definition of the variable (to be distinguished from the conceptual definition of the variable itself). And the process of finding indicators and values is called operationalisation.

4.4

The nature of the relationship

The hypothesis concerns a relationship between two or more variables. The set of hypotheses to be tested may concern various dependent variables, each caused by one or more independent variables. The letter X is often used to indicate the causal variable, while Y indicates the effect variable. A relationship may be positive (Y increases if X increases) or negative (Y decreases with increasing value of X). The hypothesis is usually tested by plotting the sampled units of analysis against the values of the indicators, and determining the likelihood of a statistically significant relationship. This is similar to the agronomists field trial where the plots are the units of analysis, the variables are fertiliser level (X) and yield (Y), and a t-test or an analysis of variance forms the statistical test. Usually the assumption is that a statistically significant relationship shows cause and effect, where one or more independent or causal variables have an effect on a dependent variable. The value of the latter is supposed to be a function of the value of the former, and therefore depends on it. But such a relationship has to be tested for logical consistency. For example, X must have preceded Y in time if X is to be a
ICRA Learning Materials - Preparing a Formal Survey Guidelines - 5/10

cause of Y. Also, the possibility that changes in Y are caused by another variable that is also strongly correlated with X should be considered.

5 Choosing a sampling technique

There are two types of sampling: Probability sampling. In this approach, each individual or sampled unit has a known but not necessarily equal probability of being selected (necessary to guard against biases, in particular those related to the accessibility of sites or persons). This type of sampling is essential if statistical inference on quantitative data is to be made from the results of the survey. Probability sampling depends on the availability of an up-todate, comprehensive sample frame that lists all the individuals or units in the population that can be sampled. Non-probability sampling. This type of sampling does not permit statistical inference, but can be very useful for selecting individuals for informal surveys, or for looking at different opinions, for getting an idea of the diversity in an area, etc. In this unit, we concentrate on random probability sampling and especially simple random sampling, multi-stage sampling, stratified sampling and grid sampling.

5.1

Simple random sampling

The main principle of this method is that each individual or sampling unit should have an equal chance of being sampled. The sampling units are therefore selected at random using a table of random numbers from a statistics book, from a random number generator on a pocket calculator, or from names in a hat.

5.2

Multi-stage sampling

If the target population exists at different levels: district village farms, for example, the principle of successive random sampling, going from the most general level to the most precise level is called multi-stage sampling. The advantages are that a) the cost of the sampling widely dispersed units is reduced and b) a sampling frame is constructed more easily, and c) it avoids the use of a very large sampling frame. In this method a sample of communities, for example, is first defined by random selection from a list (e.g. drawn up from a map); within each sampled community, a sample of households or farms is then randomly selected from a list in each selected community; and so on. In a three-stage sample the sampled strata might be districts, communities and households.

5.3

Stratified sampling

In stratified sampling, the population is divided into relatively uniform subpopulations according to some characteristic of interest. A separate sample is then taken from each stratum or subpopulation. This is more efficient than a random sample where there is more uniformity within strata than between strata. Stratified sampling should only be used if each unit or individual can be identified as belonging to one particular stratum. For example, if different groups of farmers can be defined through using an
ICRA Learning Materials - Preparing a Formal Survey Guidelines - 6/10

unambiguous typology. The advantage of these methods is that the precision of estimations is increased; the disadvantage is that prior knowledge of the population is needed.

5.4

Grid sampling

Where there is no sampling frame, or list of farms, an alternative method is to locate farms geographically. There are two main ways of doing this. The first is to take points from a map through a random sample of coordinates or the superimposition of a grid over a map (or aerial photograph). The fields, farms or units closest to these points are then selected for sampling. However, if a detailed map is used and the points located within farms, there will be a bias towards larger farms (or farms which are dispersed over several plots). Selection of the nearest household is therefore a better technique. If houses are not near roads, the time and effort used in this sampling method should not be underestimated.

6 Drawing up a sampling frame

Random samples are drawn from a sampling frame, which is defined as: the structure or list containing all the units of analysis (of the target population) from which the sample could be drawn. It can be very difficult to obtain a truly comprehensive and up-to-date sampling frame that fits perfectly the demands of the study. Creating a sampling frame oneself is sometimes very costly (in case of a large area or a large population). Some possible sources of sampling frames are given below:
Sources for sampling frames Village lists or electoral registers. Local bureaux of statistics. Census maps and cadastral surveys. Advantages, disadvantages Usually complete. May not be complete or up-to-date. Do they exist? Are they complete, recent? Difficult to find the owners. Is risk of bias as only the larger elements maybe recognised. Can be difficult or costly to obtain. Lists may not be comprehensive and may be biased towards participating farmers (the progressive farmers?). Can be biased lists, but may also facilitate access to marginal individuals

Aerial photographs.

Extension workers. Project personnel. Lists from specific sectors: health personnel, head teachers, and religious leaders.

7 Defining the sample size

After having clearly defined the target population and after having defined the units of analysis, the next step is to define the size and type of sample. Sampling techniques enable us to obtain sub-populations of the parent population containing the characteristics we want to study. While sampling should, in principle be determined according to the precision required (i.e. it should allow the measurement of the characteristics of the population with a defined degree of confidence or error), in
ICRA Learning Materials - Preparing a Formal Survey Guidelines - 7/10

practice the definition of a sample frame and size is usually a compromise between precision, coverage and cost of the overall survey. Increasing the size of the sample reduces the sampling error (i.e. the likelihood that the sample does not represent accurately the overall population). However, this may imply reducing the detail or number of characteristics measured, or supervision of the sampling. In principle, the sample size should be chosen to keep sampling error lower than some predetermined value appropriate for the type of conclusions desired, based on the variability of the variables to be measured. (The sample ratio, the number of units of the sample with respect to the total number of units of the parent population, has no influence on the statistical representation). In practice however, there are usually a number of variables being measured and the variance of these variables is not usually known in advance. A rule of thumb is therefore to use a sample size of 30-50 farmers, or sampling units, which usually represents fairly well the variation in a sample of farms within a particular target group. If more characteristics are being studied, or if a greater precision is required (e.g. for a baseline figure of yield, against which future yields might be compared) it may be necessary to increase this number.

8 Designing the questionnaire

A questionnaire needs to be designed to allow comparison of answers from different respondents. The questionnaire should therefore be standardised. Features of questionnaire design are the layout, formulating the questions and testing and adapting the survey.

8.1

Formulating the questions

Different sorts of questions can be included in a questionnaire: Open questions are those to which any answer can be accepted: e.g. how is the management of the irrigation scheme organized? These questions cannot be used for quantitative analysis, but are used to explore issues. If your questionnaire has mainly this type of question, you should reconsider if a formal survey is the best method to obtain the information you need. Closed questions are those to which a definite answer is required: e.g. How many rice varieties did you plant on your farm this year. These sorts of questions can be used to test or verify hypotheses. Pre-coded questions are closed questions, for which the range of possible answers has been predetermined. This makes it easier to process data later on, but unless designed carefully this type of question may make a respondent choose between several unsuitable answers; e.g.. o do you use the rice variety introduced by the extension agency yes no

In this case, the farmer may not know which variety is promoted by the extension agency, or may not grow rice.
ICRA Learning Materials - Preparing a Formal Survey Guidelines - 8/10

When formulating questions, remember: Clarity: All notions and concepts should be clearly defined, and ensure that all members of the team and enumerators (if used) have the same understanding of the concept. Simplicity: Put questions put in a direct form in a vocabulary that is easy to understand. Use local terminology (measures of area, weight). Precision: Be precise with regard to the selected units, the degree of precision (do you want to obtain an exact value or an approximate value?) in the formulation of questions. Neutrality: do not ask leading questions - that is questions that suggest or hint that a particular answer is the correct one. Sensitivity: consider if your respondents are likely to feel offended or answer honestly. Start with the general and easy questions (for example details of the family). Leave the difficult and sensitive questions (such as questions about income flows, assets) to the end. In case of a questionnaire, data are generally processed in tables. In such cases, the layout of the questionnaire can be organised in such a way that ulterior processing of data is facilitated.

8.2

Processing the data.

The processing of information (management of data) should already be considered when elaborating the questionnaire or checklist. The different operations (mode of handling, type of tests) are chosen on the basis of an examination of the relevance of the data with respect to the relation objectives / expected results. The operations of data processing include checking the data, correcting errors and describing data failures. The sooner you start this process, the more chance you have to identify errors and omissions. You should consider: Layout. (Are all units of measurement standardised? How will you convert qualitative data into numerical/nominal data?) Pre-coding. (Which coding classes will you use? Do all possible answers fit within these classes?) Storing/registering data. (What software will you use? Who will enter the data? How long will this take?) Processing and analysis (How will you present the data? What tables/ graphs, etc? Who will do these analyses?). It is difficult to make major changes to a questionnaire once the survey is underway. Part of the work will be lost if all the respondents have not answered the questions in the way you expected, or when the answers cannot be processed. Therefore, it is a good idea to pre-test the questionnaire with a few initial interviews, in order to process these, verify and adjust the questionnaire (add or take out questions, clarify any questions that have not been understood by the respondents, change units of measurement, etc). The ease of data processing and analysis should also be tested at this stage: while there is still time to adjust the questionnaire.

ICRA Learning Materials - Preparing a Formal Survey Guidelines - 9/10

9 Precautions in interviewing
As with all interviews (both formal and informal), it is important for the interviewer to: Respect the local customs and uses, and working times when the timing of interviews is being arranged. Avoid lengthy questionnaires that take more than about 40 minutes to fill in. Include only those questions that are necessary for your objectives. Try to fit in avoid showing fancy clothes or cars where possible. Know at least some expressions (e.g. simple greetings) of the local language. Introduce yourself. Explain to the respondent the purpose of the survey, why or how he/she has been selected for interview. Clarify what is expected from the respondent and how the interview will be conducted, (e.g. so that the respondent is free to express his/her opinions), and what use will be made of the interview results. Confirm the importance and value of the answers that are given. Go through the appropriate local authorities. Try not to be too formal even where a questionnaire is being applied, the interview should more resemble a conversation than just a set of questions. Focus on what his counterpart says rather than just follow the prepared list of questions; even when using a closed questionnaire, you should leave some space for answers to the respondent. Seek the logic behind the answers, rather than merely record what is said. Be careful to explain or rephrase questions that are not clear or not understood (e.g. using unfamiliar units of measurement).

10 Acknowledgements
This learning resource was prepared for ICRA (www.icra-edu.org) by Jon Daane and Richard Hawkins, using information from Development Oriented Research in Agriculture: An ICRA Textbook by Hal Mettrick; published by ICRA, 1993. It forms part of the ICRA learning resources, written for use in capacity building in Agricultural Research for Development. You are welcome to use these materials for non-profit purposes; we only ask you to send us a short email at Secretariat.ICRA@wur.nl, or leave a comment on our webpage telling us how you have used them and how you think they could be improved Thank you!

ICRA Learning Materials - Preparing a Formal Survey Guidelines - 10/10