METHODS AND TOOLS IN PUBLIC HEALTH

A Handbook for Teachers, Researchers and Health Professionals

Title DELPHI ANALYSIS
Module: 1.5.3 ECTS (suggested): 0.25
Author(s), degrees, Neda Milevska-Kostova, MSc, MCPPM
institution(s) Centre for Regional Policy Research and Cooperation  “Studiorum”,  
Skopje, R. Macedonia
William N. Dunn, PhD, Professor
Graduate School of Public and International Affairs (GSPIA), University
of Pittsburgh, PA, USA
Address for Neda Milevska-Kostova
correspondence Centre for Regional Policy Research  and  Cooperation  “Studiorum”
Nikola Parapunov BB, kompleks Makoteks 1 kat
Skopje, R. Macedonia
e-mail: nmilevska@studiorum.org.mk
Keywords Delphi technique, Delphi study
Learning objectives After completing this module students and public health professionals
should be able to:
 distinguish projections and predictions;
 contrast potential, plausible, and normative futures;
 evaluate judgmental forecasting methods;
 use policy Delphi analysis to make point and interval forecasts
 create a Delphi forecast for an issue.
Abstract Policy Delphi analysis (PDA) is one of the most important and widely used
methods of using expert judgment to make forecasts. The module gives
overview to the three forms of forecasting used for Delphi analysis, and
explores the techniques used in these types of forecasts. Based on the described
assessment of the strengths and limitations of PDA, the module also describes
the outline of the use of a 2-round Delphi exercise in class. The purpose is to
forecast the probable outcomes in the pre-defined health policy case.
Teaching methods An introductory lecture gives the students insight into characteristics of
Delphi technique. The theoretical knowledge is illustrated with a pre-
developed case study. Students are required to read some recommended
readings before the introductory lectures. After the introductory lectures,
they are given the assignment to prepare a Delphi forecast design, based on
the theoretical background, which they discuss with other students,
especially the designing and planning phase and expected outcomes of the
PDA. In continuation, they need to create a virtual PDA, using published
materials on the selected health issue and from own knowledge of the
problem and present their findings to other students in the class.
Specific  work  under  teacher  supervision/individual  students’  work  proportion:  
recommendations 40%/60% ;
for teachers  facilities: a lecture room, a computer room;
 equipment: LCD projection, whiteboard (flipchart), access to the Internet
and bibliographic data-bases;
 training materials: recommended readings;
 target audience: master degree students according to the Bologna process.
Assessment of Multiple choice questionnaire (MCQ) and case problem presentations.
students

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 423
DELPHI ANALYSIS
Neda Milevska-Kostova, William N. Dunn

THEORETICAL BACKGROUND
History and objectives of Delphi Method
A relatively old but still fairly often used method for objective and reliable exploration of
new ideas through a consultation of independent experts in a creative way of forecasting to
furnish arguments for decision making - is a short compiled definition of the Delphi
method, which was developed in the 1940s by the RAND Corporation, as a tool that
evolved in the process of experimentation in the technology forecasting studies. In 1944,
General Arnold asked Theodor von Karman to prepare a forecast of future technological
capabilities that might be of interest to the military (1). Later, in 1959 Helmer and fellow
RAND researcher Rescher published a paper on "The Epistemology of the Inexact
Sciences," which provided the philosophical base for forecasting (2). The paper argued
that in fields in which the instruments are not yet developed to the point of scientific laws,
the testimony of experts is permissible and should be acceptable. Thus the Delphi method
was developed, recognizing human judgement as legitimate and useful input in generating
forecasts - solving the problem of how to use human judgement and, specifically, how to
combine the testimony of a number of experts into a single useful statement (3).

Objectives of the Delphi method

The Delphi method is a judgemental forecasting technique for obtaining,
exchanging and developing informed opinion through a consensus about the most
probable future by iteration.

Characteristics of the Delphi method

The Delphi method is an exercise in-group communication among a panel of
geographically dispersed experts (4). The technique allows experts to systematically deal
with a complex problem or task. In its essence, the Delphi technique is rather
straightforward, with set rules and principles of its performance; it comprises of sets/series
of questionnaires sent to the individual and independent experts, which have been pre-
selected for the task. According to Fowles (2) anonymity, controlled feedback, and
statistical response characterize Delphi. The group interaction in Delphi is anonymous, in
the sense that comments, forecasts are not linked to their generator but are presented to the
group in such a way as to suppress any identification (3).

Principles of the Delphi Method

Anonymity
Anonymity is the first and by far the most important principle of the Delphi. This
approach allows for equality among the participating experts, avoiding the positions of
authority or dominance of one's opinion or judgement. The panel of experts is selected
based on their knowledge or opinion on the issue.

Delphi Analysis
424 METHODS AND TOOLS IN PUBLIC HEALTH
 The questionnaires are designed is such a way to provoke and develop
individual responses to the problems posed and to enable the experts to refine their
views within the progress of the group work towards the goal of the task. The main
point behind the Delphi method is to overcome the disadvantages of conventional
committee action (3).

Iteration
Another important principle of the Delphi is the possibility of iteration - repetitiveness of
the process as long as the experts, and especially the group coordinator feels that there is a
need for refinement of the views and statements. Of course, this does not imply that all
experts necessarily have to agree on certain point (see the principle of stakeholders
disagreement below), but reaching a point at which all participants feel comfortable with
the outcome is necessary precondition for considering the Delphi analysis to have
undergone is a successful manner.
The rounds of exchanging judgements can be repeated as required, but most often
the process is completed after 2 or 3 cycles; with either the expert consensus (see below),
which is common for the traditional Delphi approach, or with stakeholder disagreement
(see below), which is a newer principle often related to the policy Delphi approach.

Controlled feedback
The process is coordinated by a person called facilitator, who needs to have both
analytical and managerial skills, as he/she is responsible for sending out
questionnaires with instructions, collecting them back and summarizing views, as
well as preparing a new set of questions that would serve as further distiller of the
ideas and opinions, if and when consensus is not reached during the first or
subsequent rounds. His/her analytical skills are needed for the process of synthesis
of results and working towards building a group consensus.

Statistical group response

In order to represent the full range of opinions and not only the ones reached by
consensual agreement of the panel of experts, the summary of individual responses
are presented in the form of measures of central tendency (usually the median),
dispersion (the interquartile range) and frequency distributions (histograms and
frequency polygons) (4,5).

Expert consensus
Traditionally the Delphi method has aimed at a consensus of the most probable
future by iteration; the number of cycles to reach the consensus was irrelevant, as
long as there is obvious progress in moving towards the anticipated consensus.
However, the weakness of this aspect is that not always the consensus can be
reached, regardless of the number of repetitions, leading to necessary change in the
selected panel of experts, which in turn shows certain imperfection of the method,
and weakens the interest of the remaining expert members, as a result of the
appearing sense of time being wasted. Thus, in late 1960s alternative Delphi
approaches were introduced, explained further below.

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 425
 Stakeholders’  disagreement
The Policy Delphi (6) launched by Murray Turoff instead is a decision support method
aiming at structuring and discussing the diverse views of the preferred future; the Policy
Delphi, seeks to generate the strongest possible opposing views on the potential
resolutions of a major policy issue. In the author's view, a policy issue is one for which
there are no experts, only informed advocates and referees (6).
Policy Delphi begins by using snowball sampling to maximize (rather than
minimize) differences among multiple perspectives, using these differences to inform
the development of consensus and predictive accuracy in policy forecasting (5, 7, 8,
9). In the face of the policy issue, the expert becomes an advocate for effectiveness
or efficiency and must compete with the advocates for concerned interest groups
within the society or organization involved with the issue. The Policy Delphi also
rests on the premise that the decision maker is not interested in having a group
generate his decision; but rather, have an informed group present all the options and
supporting evidence for his consideration (9).

Other types of Delphi

Besides the traditional and the policy Delphi, several other modified types have
been proposed.
The Argument Delphi (10) developed by Osmo Kuusi focuses rather on the
ongoing discussion and finding relevant arguments than on the output of the debate
itself. The process is based on four level classification of statements, which are
usually at least in part mutually exclusive; experts at first make a very simple
evaluation on whether they approve or dissaprove the statement/topic, after what
they develop arguments pro et contra for acceptance or rejection of given statement.
The Disaggregative Policy Delphi (11) developed by Petri Tapio uses cluster
analysis as a systematic tool to construct various scenarios of the future in the latest
Delphi round. The respondent's view on the probable and the preferable future are
dealt with as separate cases.

Forecasting used in health policy

There are several types of forecasting techniques used in the health policy
development; they often employ judgemental or statistical methods of forecasting.
The statistical methods (extrapolation, multivariate forecasting and econometric
forecasting) are used when there is sufficient statistical data; however, when there
are insufficient data sources or there is low emphasis on the accuracy of the
forecast, experts use judgemental methods, such as the unstructured (unaided) or
structured, of which Delphi is one example.
The Delphi technique is used often in the health sector when there is
insufficient or unreliable data to conduct a statistical forecast. Projections
developed by Delphi panels are believed to be more accurate than forecasts b ased
on unaided judgment. There is limited direct evidence of the accuracy of forecasts
using the Delphi method (12).

Delphi Analysis
426 METHODS AND TOOLS IN PUBLIC HEALTH
 Steps of Delphi
Conventional (traditional) Delphi
A Delphi method, as defined by its creators (2) has the following 10 steps:
1. Formation of a Delphi team to undertake and to monitor the project.
2. Selection of one or more panels to participate in the exercise. Customarily,
the participants are experts in the investigation area.
3. Development of the first round Delphi questionnaire.
4. Testing the questionnaire for proper wording (e.g., ambiguities, vagueness).
5. Transmission of the first questionnaires to the panelists.
6. Analysis of the first round responses.
7. Preparation of the second round questionnaires (and possible testing).
8. Transmission of the second round questionnaires to the panelists.
9. Analysis of the second round responses. (Steps 7 to 9 are reiterated as long
as desired or necessary to achieve stability in the results.)
10. Preparation of a report by the analysis team to present the conclusions of the
exercise.

However, both the creators and other authors (13) argue that the most important
issue in this process is the understanding of the aim of the Delphi exercise by all
participants. Otherwise the panelists may answer inappropriately or become
frustrated and lose interest (3).
On the other hand, the Policy Delphi can be conducted in a number of
different ways, depending on the context and the skill and ingenuity of the persons
using the technique. Since Policy Delphi is a major research undertaking, it
involves a large number of technical questions, sampling, questionnaire design,
reliability and vailidity, and data analysis and interpretation (5). The steps of the
Policy Delphi, which are somewhat modified from the conventional approach, are:
1. issue specification,
2. selection of advocates,
3. questionnaire design,
4. analysis of first-round results,
5. development of subsequent questionnaires,
6. organization of group meetings,
7. preparation of final report.

As the Policy Delphi is more often used for health policy development, its steps are
elaborated in the Table 1.

Table 1. Steps of Policy Delphi (6).

Step Description Example*
Issue Analyst decides upon the specific To develop a range of possible
specification issues for which the Delphi will national drug-abuse policy
be conducted. options

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 427
Table 1. Cont.
Step Description
Selection of Selection of key stakeholders in
advocates the issue area, including experts
(preferably with opposing or
conflicting positions on the issue)
Questionnaire Development of questionnaire
design based on the selected issue;
depending on the level of
knowledge of the analyst and
selected experts, the questionnaire
is either done completely by the
analyst (using scales for
measurement, see example below)
or the analyst develops an open-
end questions set, that are further
distilled by the experts through
providing their opinions.
Analysis of first- Analyst attempts to determine the
round results initial positions on the forecasts,
issues, goals and opinions.
Use of summary measures due to
expected conflicting assessments
of various advocates (avoiding
presentation of central tendency
only)
Development of The results of prior rounds are
sub-sequent used for development of
questionnaires subsequent ones; include
summary of arguments for the
most conflicting judgements.
Thye analyst should count on
decreased rate of response in the
subsequent rounds, expecially
those involved on voluntary basis.
Organization of Brinings advocates (stakeholders)
group meetings in face-to-face discussion of
reasons, assumptions and
arguments of various positions;
useful for immediate feedback.
428 METHODS AND TOOLS IN PUBLIC HEALTH
Example*
A list of about 100 experts was
set, with invitations initially sent
to 45 persons, for an expected
positive response from 25 persons
(in fact, positive responses
obtained from 38 experts).
The first questionnaire was
pretested, and the entire
forecasting section deleted when
it was determined that the time to
complete the questionnaire was
decreased considerably by
deleting this section. The
questionnaire consisted of 4
sections: development of
objectives, transition matrix,
policy issue statements and
additional items.
24 of 35 respondents (69%)
returned the filled-in
questionnaire; respondents were
asked substantive-issue questions,
self-rating of own expertise in
drug abuse, expectations from the
study, etc.
Experts listed 78 policy issues
that were distilled down to 55; the
187 key indicators were culled to
153.
The 2nd questionnaire developed
after round One was completed; it
included only two sections:
National drug-abuse policy
objectives and Policy issue
statements.
There were four issues that
exhibited marked differences
between policy experts and
nonexperts in the importance of
issues.
-
Delphi Analysis
Table 1. Cont.
Step Description Example*
Preparation of Analyst is responsible for drafting Final report covering the main
final report the final report, based on the conclusions of the questionnaire
questionnaire results and face-to- analysis, as the group meting was
face discussions; the final report not held in this example.
reveals a review of various issues
and options available, including a
complete description of all
conflicting positions.

Figure 1. Policy Delphi steps and process

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 429
 Design of questionnaire
First questionnaire
As explained earlier, the Delphi usually consists of several rounds; of all, only the
first questionnaire can be prepared in advance and all other are derived based on the
synthesized results of previous rounds.
In the first round, the Delphi process traditionally begins with an open-ended
questionnaire. The open-ended questionnaire serves as the cornerstone of soliciting
specific information about a content area from the Delphi subjects (14). For
example, in the first questionnaire, participants might be asked to provide their
judgment (15) on a most probable period by which a drug abuse policy will give
visible   effects.   After   receiving   subjects’   responses,   investigators   convert   the  
collected information into a well-structured questionnaire. This first-round
questionnaire is used as the survey instrument for the second round of data
collection. It should be noted that it is both an acceptable and a common
modification of the Delphi process format to use a structured questionnaire in
Round 1 that is based upon an extensive review of the literature (16). Kerlinger (17)
noted that the use of a modified Delphi process is appropriate if basic information
concerning the target issue is available and usable.

Table 2. Types of items and scales used in Policy Delphi Questionnaire (5)
Type of item Item Scale
Forecast According to a projection of researchers at [1] Certainly reliable
the National Public Health Institute, over [2] Reliable
20% of young people age 15-24 are
smoking marijuana, and this percentage [3] Risky
will be doubled in the coming 10 years. [4] Unreliable
How certain are you that this projection is [0] No judgement
reliable?
Issue Personal use of marijuana should/should [1] Very important
not be legalized. [2] Important
How important is this issue relative to [3] Slightly important
others?
[4] Unimportant
[0] No judgement
Goal One goal of National drug abuse policy is [1] Very desirable
to increase the awareness of difference [2] Desirable
between drug use (responsible) and drug
abuse (irresponsible). [3] Undesirable
How desirable is this objective? [4] Very undesirable
[0] No judgement
Options Drug abuse education is reported to [1] Definitely feasible
contribute towards reduction of potential [2] Feasible
users.
[3] Possibly feasible
How feasible ins this policy option?
[4] Definitely unfeasible
[0] No judgement

Delphi Analysis
430 METHODS AND TOOLS IN PUBLIC HEALTH
 Results analysis and providing feedback
Analysis of the results
When questionnaires are collected, the analyst or the facilitator is the one that
summarizes the responses, synthetizes various positions and puts forward the
arguments already pointed out by the advocates. Each sub-sequent round is
structured and prepared based on the results of the previous one; however, the
results are not presented to the panel of advocates/experts at each round in aform of
report, but rather in a form of a new questionnaire (see above Design of
questionnaire).
Data analysis can involve both qualitative and quantitative data; usually
qualitative data is dealt with if conventional Delphi studies, which use open-ended
questions   to   solicit   subjects’   opinions,   are   conducted   in   the   initial   iteration.  
Subsequent iterations are to identify and hopefully achieve the desired level of
consensus among panelists.
The statistics used in Delphi studies are most commonly measures of
central tendency (means, median, and mode) and level of dispersion (standard
deviation and inter-quartile range) in order to present information concerning the
collective judgments of respondents (18). Generally, the uses of median and mode
are favored (16).

Providing feedback
As mentioned above, the results analysis from each round serves as feed into the
next   round   of   challenging   experts’/advocates’  positions   and   opinions  on   the   issue;;  
once the issue is exhausted, on there is seem no further argumentative development,
the facilitator/analyst approaches to writing the final report. The final report is
aimed primarily at informing of all opposing positions and alternatives presented by
the advocates/experts, supported by arguments as presented by the panel. Yet, as
this may become a long list of different standpoints that might confuse the policy-
or decision-maker, other presentations are used, taking into consideration the
possibility to present all opinions, including the outlayers. Some are mentioned
below:
 difference graph,
 histogram or bar chart,
 box-and-whisker plot,
 lists of measures of central tendency and dispersion, etc.

Figure 2. Providing feedback - Difference graph (5).

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 431
 2030

18

2020

2010

2000
N= 18

Figure 3. Providing feedback – box-whisker plot.

On-line tools for Delphi analysis

Available (free) software
The website of Forecasting Principles is offering on-line usable application free of
charge for trying up the basic features of the Delphi method. Following the link
http://armstrong.wharton.upenn.edu/delphi2/ you can register as administrator, start
your own survey, in which the application will help you:
 select experts,
 develop questions and scales,
 obtain responses from the experts,
 summarize a report after each round.

This tool, developed by J. Scott   Armstrong   (19)   is   a   very   useful   starter’s   kit   for  
application of Delphi method into your own research.

Risks and disadvantages of Delphi analysis

Risks associated with Delphi
While offering a large possibilities for forecasting by use of experts opinion
(Delphi) or advocates positions (Policy Delphi), this method has its own risks,
associated mainly with the inseparable personal subjectivism and the partial or
complete ignorance of the advocates in the issue. As the outcome of a Delphi

Delphi Analysis
432 METHODS AND TOOLS IN PUBLIC HEALTH
sequence is nothing but opinion, and the results of the sequence are only as valid as
the opinions of the experts who made up the panel (20), if not carefully selected, the
panel of experts, or advocates for that matter, can lead the discussion into
predicition of a highly unlikely future outcome; in some cases this can further
strengthen the confidence of the part of the experts or advocates which do not
possess great knowledge or informed opinion in the issue.

Disadvantages of Delphi
As with other research methods, Delphi has been reported to have its own
disadvantages; most of them associated with improper selection or invalid use of
the method for a given type of outcome desired. It is up to the researcher/analyst to
determine if this method will produce valid and usable results.
The challenges that the researcher should think of prior to selection of this
method as their prefered tool, is related to the internal consistency and reliability of
judgements among experts, which - if low or lacking - may lead to low
reproduceability of forecasts based on the results elicited; sensitivity of results to
ambiguity and respondent reactivity in the questionnaires used for data collection;
difficulty in assessing the degree of expertise held by participating experts (21).
Another problem identified by research into the implementation and
application of Delphi surveys has been the tendency for experts to over-simplify
particular issues, and treat them as isolated events. This is particularly the case in
forecasting, where experts tend to think in terms of linear sequential events, rather
than applying a holistic view that involves complex chains and associations. Again,
it is up to the researcher/analyst to extract this aspect from the answers of
experts/advocates through the sub-sequent rounds of questionning. Other techniques
for multi-dimensional   analysis   are   also   available,   such   as   ‘cross   impact   matrix  
forecasting’   intended   to   compare   a   range   of   ‘possible   futures’   against   each   other;;  
but those are discussed elsewhere (4,22,23).

Advantages of Delphi
Although the approach was originally developed to capture expertise in uncertain
and emergent domains, it tends to be used in evaluation when significant expertise
exists on the subject, for example in the case of programmes that are not innovative.
The method is recommended when the questions posed are simple (a programme
with few objectives, of a technical nature) and for the purpose of establishing a
quantitative estimation of the potential impacts of an isolated intervention (e.g.
increase in taxes or in the price of health services). It is also recommended in an ex
ante evaluation context if the evaluation concerns public intervention of a technical
nature. But, it may also be used to specify relations of causes and potential effects
in the case of innovative interventions. It is particularly useful when a very large
territory is being dealt with since there are no experts' travel expenses, only
communication costs.
It has been found to be particularly useful in programmes related to public
health issues (such as, policies for drug use reduction and prevention of HIV/AIDS)
and education (1,4). In general, the Delphi method is useful to explore and unpack
specific, single-dimension issues. There is less support for its use in complex,
multi-dimensional modelling. In these cases, the evidence does suggest that data

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 433
gathered by Delphi surveys is a useful input, when supported by data gathered from
other sources, to complex scenario-building (24).
Nevertheless, according to some authors, still the context plays great role in
deciding whether and when to use the Delphi method.

CASE STUDY: AIDS VACCINATION POLICY: A SCENARIO

ANALYSIS USING THE DELPHI METHOD
Background
The Delphi method was used to explore and identify the potential implications
associated with the introduction of a first AIDS vaccination in Switzerland (25,26).
Thirty participants with an interest in the field contributed anonymously to the
study. The study focused on an existing scenario which modelled the characteristics
of a first preventive, partially effective, vaccination against AIDS.

The process
The Delphi consultation was carried out in three stages. In the first round, the
participants were asked to:
 list the objectives to be achieved in the first five years;
 evaluate the acceptability and feasibility of proposals concerning the
development of a public health strategy and the AIDS vaccination;
 estimate the potential use of the vaccination by different groups of users.

The outcome
The used Delphi method in the consultation process produced two main outcomes:
firstly, a set of strategies and recommendations for the development of a framework
of AIDS prevention campaigns and, secondly, an institutional framework for the
setting up of a future AIDS vaccination strategy.

Follow-up
In parallel, in 2003, clinical trials of a new vaccine against HIV, started in
Switzerland and the United Kingdom. EuroVacc, the foundation organising the
trials, has tested two vaccines: DNA-C, developed by Professor Hans Wolf of the
University of Regensburg, Germany, and its booster, NYVAC, developed by the
French pharmaceutical company Aventis. About 160 healthy volunteers - half in
London and half in the Swiss city of Lausanne, where EuroVacc is based – have
been subjected to the test of the vaccine for safety.
In 2005 the combined vaccine consisting of DNA vaccine and NYVAC booster
was tested in Switzerland, the Netherlands, the United Kingdom, Spain, Italy,
Germany, and Sweden in hundreds of people seen as being at high risk of HIV
infection, including gay men, drug users, and commercial sex workers. Volunteers’  
rate of infection was monitored and compared with the rate of infection in similar
groups of people who have not been given the vaccine.

Delphi Analysis
434 METHODS AND TOOLS IN PUBLIC HEALTH
EXERCISE
Task 1
Before   class,   based   on   the   Table   2   above,   write   as   many   as   you   can   “forecast,”  
“issue,”   “goal,”   and   “options”   questions   related   to the Case study above. These
questions will be collected and used for the in-class Delphi exercise.

Task 2
After the in-class discussion of the collected questions, assume the role given by the
teacher and fill out the sample questionnaire that will be distributed by the teacher.
One of the students will be assigned the role of analyst who will chair the group
meeting in the next class. Role-play exercise in the class.

Task 3
Based on the role-play held in the class, prepare a final report (assuming the role of
analyst). The teacher will consider this paper as an assessment for the module.

REFERENCES
1. Cornish E. The study of the future: an introduction to understanding the art and
science  for  understanding  and  shaping  tomorrow’s  world.  Washington,  DC: World
Future Society; 1977.
2. Fowles J. Handbook of futures research. Greenwood Press: Connecticut; 1978.
3. Illinois Institute of Technology. The Delphi method [homepage on the Internet].
Available from URL: http://www.iit.edu/~it/delphi.html. Accessed: April 6, 2009.
4. Adler M, Ziglio E. Gazing into the oracle. Bristol, PA: Jessica Kingsley Publishers;
1996.
5. Dunn WN. Public policy analysis. New Jersey: Pearson Prentice Hall; 2004.
6. Turoff M. The design of a policy Delphi. Technol Forecast Soc Change. 1970;2:2.
7. Mahmoud E. Accuracy in forecasting: a survey. J Forecast. 1984;3:139-59.
8. Rescher N. Predicting the future. Albany, NY: State University of New York Press;
1998.
9. Turoff M. The policy Delphi. In: Linstone H, Turoff M, Helmer O. editors. The
Delphi method: techniques and applications. New York: Adisson Wesley; 2002.
p.80-98.
10. Kuusi O. Expertise in the future use of generic technologies. Research Reports 59.
Helsinki: Government Institute for Economic Research Finland; 1999.
11. Tapio P. Disaggregative policy Delphi: using cluster analysis as a tool for systematic
scenario formation. Technol Forecast Soc Change. 2002;70:83-101.
12. Forcastingprinciples.com [homepage on the Internet] Methods for health forecasting,
forecasting principles; c1997-2007. Available from URL:
http://www.forecastingprinciples.com/Health/methods.html. Accessed: April 6, 2009.
13. Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program planning:
a guide to nominal group and Delphi processes. Glenview, IL: Scott&Foresman; 1975.
14. Custer RL, Scarcella JA, Stewart, BR. The modified Delphi technique: a rotational
modification. J Vocat Techn Educat. 1999; 15:1-10.

Delphi Analysis
METHODS AND TOOLS IN PUBLIC HEALTH 435
 15. Gordon TJ. The Delphi method. Future research methodology. Washington, D.C.:
AC/UNU Millenium Project; 1994. Available from URL:
www.gerenciamento.ufba.br/Downloads/delphi%20(1).pdf. Accessed: April 6, 2009.
16. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess
Res Evaluat. 2007;12:1-8. Available from URL:
http://www.pareonline.net/getvn.asp?v=12&n=10. Accessed: April 6, 2009.
17. Kerlinger FN. Foundations of behavioral research. New York: Holt, Rinehart, and
Winston, Inc.; 1973.
18. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique.
J Advanc Nurs. 2000;32:1008-15.
19. Imerial College London [homepage on the Internet].The development of the Delphi
software was funded in part by International Institute of Forecasters. Available from
URL: http://www3.imperial.ac.uk. Accessed: April 6, 2009.
20. Martino J. The Precision of Delphi estimates. Technol Forecast Soc Change.
1970;1:293-9.
21. Makridakis S, Wheelwright SC. Forecasting methods for management. 5 th ed.
Chichester: Wiley; 1989.
22. Gordon TJ, Hayward H. Initial experiments with the cross-impact matrix method of
forecasting. Futures. 1968;1:100-16.
23. Gatewood RD, Gatewood EJ. The use of expert data in human resource planning:
guidelines from strategic forecasting. Human Resource Planning. 1983;5:83-94.
24. European Commission. Evaluating socio economic development. Sourcebook 2:
methods & techniques, Delphi method. Luxembourg: European Commission; 2005.
p.110-9.
25. Zuber P. Introducing a first AIDS vaccination in Switzerland: a Delphi policy
analysis.  Lausanne:  Institut  de  médecine  sociale  et  preventive;;  1994.
26. Fleck F. Trials of AIDS vaccine to start in Switzerland and United Kingdom. Br Med J
2003; 326:952.

RECOMMENDED READINGS
1. Dunn WN. Public policy analysis. New Jersey, NY Pearson Prentice Hall; 2004
p.180-98.
2. Jillson IA. The National drug abuste policy Delphi: Progress report and findings to
date. In: Linstone HA, Turoff M, Helmer O. editors. The Delphi method: techniques
and applications. New York: Addison-Wesley; 2002. p.124-59
3. Turoff M. The Policy Delphi. In: Linstone H, Turoff M, Helmer O. editors. The
Delphi method: techniques and applications. New York: Addison-Wesley; 2002 p.80-
98.
4. Zuber P. Introducing a first AIDS vaccination in Switzerland: a Delphi policy
analysis.  Lausanne:  Institut  de  médecine  sociale  et  preventive;;  1994.

Delphi Analysis
436 METHODS AND TOOLS IN PUBLIC HEALTH