Sie sind auf Seite 1von 5

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/11173521

The Importance of Pilot Studies

Article  in  Nursing standard: official newspaper of the Royal College of Nursing · June 2002
DOI: 10.7748/ns2002.06.16.40.33.c3214 · Source: PubMed

CITATIONS READS

357 41,970

2 authors, including:

Edwin Roland Van Teijlingen


Bournemouth University
454 PUBLICATIONS   7,679 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Exploring hormone use among male to female transgender (MTF) populations in Nepal View project

A mixed-methods study of social workers working with people using un-prescribed Anabolic Androgenic steroids View project

All content following this page was uploaded by Edwin Roland Van Teijlingen on 02 June 2014.

The user has requested enhancement of the downloaded file.


Sociology at Surrey University of Surrey

social
research
• The term ‘pilot studies’ refers to mini
UPDATE
• Pilot studies are a crucial element of a • Pilot studies fulfil a range of important
versions of a full-scale study (also called good study design. Conducting a pilot functions and can provide valuable
‘feasibility’ studies), as well as the spe- study does not guarantee success in the insights for other researchers. There is
cific pre-testing of a particular research main study, but it does increase the like- a need for more discussion amongst re-
instrument such as a questionnaire or in- lihood. searchers of both the process and out-
terview schedule. comes of pilot studies.

The importance of pilot studies

Edwin R. van Teijlingen and Vanora Hundley


The term pilot study is used in two different tionnaire survey. Next the questionnaire, e.g. Dr Edwin van Teijlingen is a
ways in social science research. It can refer the wording and the order of the questions, medical sociologist and Senior
to so-called feasibility studies which are or the range of answers on multiple-choice Lecturer in Public Health at the
“small scale version[s], or trial run[s], done questions, might be piloted. A final pilot Department of Public Health,
in preparation for the major study” (Polit et could be conducted to test the research proc- University of Aberdeen. His re-
al., 2001: 467). However, a pilot study can ess, e.g. the different ways of distributing and
search interests include the or-
also be the pre-testing or ‘trying out’ of a collecting the questionnaires. For example,
particular research instrument (Baker 1994: a recent study exploring nurses’ and mid- ganisation of maternity care at
182-3). One of the advantages of conduct- wives’ attitudes to research followed this pat- home and abroad, substance
ing a pilot study is that it might give advance tern. In this study focus groups were used misuse, and psychosocial as-
warning about where the main research to identify key issues from which a question- pects of genetics.
project could fail, where research protocols naire could be developed, and this was then Dr Vanora Hundley is a midwife
may not be followed, or whether proposed piloted prior to the study proper (Hundley and Lecturer at the Centre for
methods or instruments are inappropriate or et al. 2000). On a much larger scale, the Advanced Studies in Nursing,
too complicated. In the words of De Vaus largest (decennial) survey in the UK, the University of Aberdeen. Her re-
(1993: 54) “Do not take the risk. Pilot test Census (of 29th April 2001), tested methodo-
search interests include mid-
first.” These are important reasons for un- logical and other changes to the 1991 Cen-
dertaking a pilot study, but there are ad- sus questionnaire on over 100,000 house- wifery care, consumer satisfac-
ditional reasons, for example convincing holds in 1997. This 1997 Census Test “pro- tion and preferences, and re-
funding bodies that your research proposal vided essential information on public reac- search utilisation.
for the main study is worth funding. Thus tion to new questions and form style as well
pilot studies are conducted for a range of as assessing the success of collection and
different reasons (see Table 1). processing methods” (Office for National
Pilot studies can be based on quantitative Statistics, General Register Office for Scot-
and/or qualitative methods and large-scale land, Northern Ireland Statistical & Research
studies might employ a number of pilot stud- Agency 1999: 15).
ies before the main survey is conducted. Pilot studies may also try to identify poten-
Thus researchers may start with “qualitative tial practical problems in following the re-
data collection and analysis on a relatively search procedure. For example, in a recent
ISSUE unexplored topic, using the results to design Scottish study of maternity care the pilot
a subsequent quantitative phase of the study” phase demonstrated that the proposed

35 (Tashakkori & Teddlie 1998: 47). The first


phase of a pilot might involve using in-depth
interviews or focus groups to establish the
means of distributing the questionnaires
would not be adhered to (van Teijlingen et
al. 2001). Without consulting the research
issues to be addressed in a large-scale ques- team, the person responsible for distribut-
social research UPDATE

ing the questionnaires from the hospital come obvious until the larger scale study is
records department decided that it was bet- conducted.
ter to distribute them through the commu- A further concern is that of contamination.
nity midwives. This was despite the fact that This may arise in two ways:
the hospital itself had suggested the records 1. where data from the pilot study are in-
department as a means of distribution. Other cluded in the main results;
problems such as poor recording and re- Table 1. Reasons for conducting pilot
2. where pilot participants are included in
sponse rates can also be identified and pre- studies
the main study, but new data are col-
cautionary procedures or safety nets can be • Developing and testing adequacy of
lected from these people.
devised. research instruments
Social scientists engaged in predominantly
The steps used to pilot a questionnaire on a • Assessing the feasibility of a (full-
quantitative research are likely to argue that:
small group of volunteers, who are as simi- scale) study/survey
“an essential feature of a pilot study is that
lar as possible to the target population, are
the data are not used to test a hypothesis or • Designing a research protocol
listed in Table 2.
included with data from the actual study • Assessing whether the research pro-
Pilot studies can also uncover local politics when the results are reported” (Peat et al.
or problems that may affect the research tocol is realistic and workable
2002: 57). The obvious concern is that if
process. In the study described above, the • Establishing whether the sampling
there were problems with the research tool
managers of maternity services had different frame and technique are effective
and modifications had to be made in the light
perceptions of what the forthcoming of the findings from the pilot study, data • Assessing the likely success of pro-
changes in the Data Protection Act (1998) could be flawed or inaccurate. However, posed recruitment approaches
allowed them to do about the involvement where an established and validated tool is • Identifying logistical problems which
of their clients in research. In the above men- being used and the pilot study is determin- might occur using proposed methods
tioned recent Scottish study of maternity care ing other methodological aspects such as • Estimating variability in outcomes to
one head of midwifery voiced ethical con- recruitment rates, it could be argued that help determining sample size
cerns about the use of follow-up or reminder such data may be of value. • Collecting preliminary data
letters due to a previous local incident, where
A common problem isthe inclusion of pilot • Determining what resources (fi-
parents of an ill baby had been sent a ques-
study participants in the site(s) of the main nance, staff) are needed for a planned
tionnaire which was felt to be inappropriate,
study. Here the concern is that such partici- study
and as a result of changes to the UK Data
pants have already been exposed to an in- • Assessing the proposed data analy-
Protection Act. Consequently reminders
tervention and, therefore, may respond dif- sis techniques to uncover potential
were sent out via the head of midwifery in
ferently from those who have not previously problems
case there were any problems with the new-
experienced it. This may be positive, for ex-
born baby. • Developing a research question and
ample the participants may become more
Problems of pilot studies research plan
adept at using a new tool or procedure. How-
It should be recognised pilot studies may also ever it may also be negative with participants • Training a researcher in as many el-
have a number of limitations. These include showing a decline in following a protocol ements of the research process as pos-
the possibility of making inaccurate predic- because it is no longer novel. Indeed both sible
tions or assumptions on the basis of pilot changes in behaviour have long been recog- • Convincing funding bodies that the
data; problems arising from contamination; nised and a ‘run in’ period, where an inter- research team is competent and knowl-
and problems related to funding. These is- vention is introduced prior to a study, is of- edgeable
sues are now discussed in turn. ten used for these reasons. The concern • Convincing funding bodies that the
Completing a pilot study successfully is not about including participants from the pilot main study is feasible and worth fund-
a guarantee of the success of the full-scale study in the main study arises because only ing
survey. Although pilot study findings may those involved in the pilot, and not the whole • Convincing other stakeholders that
offer some indication of the likely size of the group, will have had the experience. In some the main study is worth supporting
response rate in the main survey, they can- cases however it is simply not possible to
not guarantee this because they do not have exclude these pilot-study participants be-
a statistical foundation and are nearly always cause to do so would result in too small a
based on small numbers. Furthermore, sample in the main study. This problem
other problems or headaches may not be- arises in particular where the samples are
clusters, for example schools, prisons or
hospitals. In such cases one can conduct a
social research UPDATE

sensitivity analysis (or sub-group analysis) Why are pilot studies not Table 2. Pilot study procedures to im-
to assess to what extent the process of
piloting influences the size of the interven-
reported? prove the internal validity of a question-
Publication bias may occur because of a ten- naire
tion effect.
dency for journals to accept only papers that • administer the questionnaire to pilot
Contamination is less of a concern in quali- have statistically significant results and not subjects in exactly the same way as it will
tative research, where researchers often use to report non-significant effects (Mahoney be administered in the main study
some or all of their pilot data as part of the 1977; Chann 1982; Dickersin, 1990). A recent • ask the subjects for feedback to iden-
main study. Qualitative data collection and study exploring research on passive smok- tify ambiguities and difficult questions
analysis is often progressive, in that a sec- ing found a difference of two years in the • record the time taken to complete
ond or subsequent interview in a series median time to publication between findings the questionnaire and decide whether it
should be ‘better’ than the previous one as from significant and non-significant studies is reasonable
the interviewer may have gained insights (Misakian & Bero 1998). It follows that pa-
from previous interviews which are used to • discard all unnecessary, difficult or
pers reporting methodological issues, such
improve interview schedules and specific ambiguous questions
as those identified during the pilot phase of
questions. Some have therefore argued that • assess whether each question gives
a study, will also be less attractive to pub-
in qualitative approaches separate pilot stud- an adequate range of responses
lishers.
ies are not necessary (e.g. Holloway • establish that replies can be inter-
1997:121). For example, a qualitative inter- Selective publication of research results has
been recognised as a problem. It may lead preted in terms of the information that
viewer conducting 15 focus group inter- is required
views will listen to the recordings or read to an overestimation of the effectiveness of
through the transcripts of the first three or interventions, exposing patients to useless • check that all questions are answered
four in order to improve the questions, the or harmful treatments, while overestimation • re-word or re-scale any questions that
way of introducing the issues into the group of adverse effects may mean that patients are are not answered as expected
interview or even to add new topics. Thus, denied effective forms of care (Oxman et al. • shorten, revise and, if possible, pilot
although there is no specific pilot study, 1994). In the past editors have recognised again
analysis of the earlier focus groups may help the dangers of publication bias with respect
(Source: Table 3.23 in Peat et al. 2002: 123)
improve the later ones. However, Frankland to clinical trials and have offered ‘an amnesty
and Bloor (1999: 154) argue that piloting for unpublished trials’ in an attempt to over-
provides the qualitative researcher with a come these problems (Smith & Roberts,
“clear definition of the focus of the study” 1997). However, it is equally important to
which in turn helps the researcher to con- ensure that lessons learned with respect to
centrate data collection on a narrow spec- the research method are shared, otherwise
trum of projected analytical topics. Piloting patients may be subjected to poorly devel- social research UPDATE is
of qualitative approaches can also be car- oped tools or money may be wasted because distributed without charge on
methods of recruitment failed. A consistent
ried out if “the researcher lacks confidence
selection bias favouring reports of primary
request to social researchers
or is a novice, particularly when using the in the United Kingdom by the
interview technique” (Holloway 1997: 121). research over papers on research methods,
Problems may also arise where a pilot study
theoretical thinking, or secondary analysis, Department of Sociology at
can lead to many of researchers re-inventing the University of Surrey as part
requires a significant investment of re- the wheel without having had the opportu-
sources, making it difficult for the study
nity to learn from other people’s experience. of its commitment to support-
team to call a halt to the research after an ing social research training
unsuccessful pilot study. Researchers might Conclusions
be tempted to make considerable changes It has been said that pilot studies are likely
and development.
in the main study, rather than deciding that to be “underdiscussed, underused and Contributions to social re-
the proposed study is not possible with the underreported” (Prescott and Soeken, 1989 search UPDATE that review
available resources, time, population, etc. In p60). Full reports of pilot studies are rare in
contrast, funding bodies may be reluctant the research literature (Lindquist, 1991;
current issues in social re-
to fund a further study if the pilot has been Muoio et al, 1995, van Teijlingen et al. 2001). search and methodology in
substantial as they may view the research as When reported, they often only justify the about 2,500 words are wel-
no longer original, especially if results from research methods or particular research tool come. All UPDATE articles
the pilot study are published. used. Too often research papers only refer
to one element of the pilot study, for exam- are peer-reviewed.
social research UPDATE

ple, to the ‘pre-testing’ or ‘pilot testing’ of a Crosswaite, C. and Curtice, L. (1994), Dissemi- Teijlingen van, E., Rennie, A.M., Hundley, V.,
questionnaire (De Vaus, 1993). Such papers nating research results-the challenge of Graham, W. (2001), The importance of
simply state: “the questionnaire was tested bridging the gap between health research conducting and reporting pilot studies: the
for validity and reliability.” When pilot stud- and health action. Health Promotion example of the Scottish Births Survey,
International 9: 289. Journal of Advanced Nursing 34: 289-295
ies are mentioned in more detail in academic
De Vaus, D.A. (1993), Surveys in Social
papers and reports, researchers regularly
Research (3rd edn.), London: UCL Press.
comment that they “had learned from the
Frankland, J. and Bloor, M. (1999), Some issues
pilot study” and made the necessary changes, arising in the systematic analysis of focus
without offering the reader details about group material, In: Barbour, R. and
what exactly was learnt. Some of these proc- Kitzinger, J. (eds) Developing Focus Group
esses and outcomes from both successful Research: Politics, Theory & Practice,
and failed pilot studies might be very useful London: Sage
to others embarking on projects using simi- Holloway, I. (1997). Basic Concepts for
lar methods and instruments. This is par- Qualitative Research, Oxford: Blackwell
ticularly important because pilot studies can Science.
be “time-consuming, frustrating, and fraught Hundley, V, Milne, J, Leighton-Beck, L. et al.
with unanticipated problems, but it is better (2000). Raising research awareness among
to … deal with them before investing a great midwives and nurses: does it work? Journal
of Advanced Nursing 31 (1): 78-88.
deal of time, money, and effort in the full
Lindquist, R. (1991). Don’t forget the pilot
study” (Mason and Zuercher, 1998). It has
work! Heart Lung 20: 91-92.
also been argued that the current research
Mason, D.J. and Zuercher, S.L. (1995). Pilot
climate demands accountability from re- studies in clinical nursing research. Journal
searchers, which means that there is a need of the New York State Nursing Association
to ensure the best possible use of research 26: 11.
results (Crosswaite and Curtice 1994). We Muoio, R., Wolcott, L., and Seigel, H. (1995). A
would like to go one step further and argue win-win situation: The pilot program.
that researchers have an ethical obligation Journal of Continuing Education in
to make the best use of their research expe- Nursing. 26: 230-233.
rience by reporting issues arising from all Office for National Statistics, General Register
parts of a study, including the pilot phase. Office for Scotland, Northern Ireland
Statistical & Research Agency. (1999), 2001
Well-designed and well-conducted pilot stud-
Census Information Paper, Government
ies can inform us about the best research Statistical Services: London.
process and occasionally about likely out- Peat, J., Mellis, C., Williams, K. and Xuan W.
comes. Therefore investigators should be (2002), Health Science Research: A
encouraged to report their pilot studies, and Handbook of Quantitative Methods,
in particular to report in more detail the ac- London: Sage.
tual improvements made to the study design Polit, D.F., Beck, C.T. and Hungler, B.P. (2001),
and the research process. Essentials of Nursing Research: Methods, social research UPDATE
Appraisal and Utilization. 5th Ed., (ISSN: 1360-7898)
References Philadelphia: Lippincott Williams & Wilkins
Baker, T.L. (1994), Doing Social research (2nd
Prescott, P.A. and Soeken, K.L. (1989), The is published by
Edn.), New York: McGraw-Hill Inc. potential uses of pilot work. Nursing Department of Sociology
Bowling, A. (1997), Research Methods in Research 38: 60-62.
Health: Investigating Health & Health Rosenberg, K.M. and Daly, H.B. (1993),
University of Surrey
Services, Buckingham: Open University Foundations of Behavioural Research: A Guildford GU2 5XH
Press. p 232. Basic Question Approach, Fort Worth/
Burns, N. and Grove, S.K. (1999), Understand-
United Kingdom.
London: Harcourt Brace College Publishers.
ing Nursing Research (2 nd edn.), Philadel- p 182-184. Tel: 01483 300800
phia: W.B. Saunders Company. p 40. Tashakkori, A and Teddlie, C. (1998), Mixed Fax: 01483 689551
Crombie, I.K. and Davies, H.T.O. (1997), Methodology: Combining Qualitative & Edited by Nigel Gilbert
Research in Health Care: Design, Conduct Quantitative Approaches, Sage
and Interpretations of Health Services (e-mail: gng@soc.surrey.ac.uk)
Research, London: Wiley.
Winter 2001 © University of Surrey

View publication stats