Multi-criteria Evaluation Revisited

Wendy Proctor1 and Ejaz Qureshi2

Policy and Economic Research Unit, CSIRO Land and Water
Corresponding author: email wendy.proctor@csiro.au fax+61 262465560

ABSTRACT

Multi-criteria Evaluation (MCE) is increasingly being used in the assessment of

natural resource management options which involve complex ecological,
economic and social outcomes and interactions. This paper draws on the results of
research experiences and reviews MCE applications to identify methodological
issues with conducting MCE and future research possibilities. Some of the key
issues that are covered include: the problems of choosing and defining the
decision makers; interacting with groups of decision makers; MCE applications to
different cultures; validation of results and sensitivity analysis and the need for
consistency in the meaning of terms used in Multi-criteria approaches.

Introduction
This paper gives a review of Multi-criteria Evaluation (MCE) when applied to environmental
issues from two practitioners with a long history of practical applications of this technique in
Australia. It draws on the results of this research to give implications for future research needs
for MCE. It includes research experiences of the authors and a review of the field of
applications in this area and looks at methodological issues that need to be addressed and
overcome and suggests future research possibilities. Some of the key issues that are covered
include: the problems of choosing and defining the decision makers that should be involved in
the analysis, interacting with groups of decision makers with multiple weights and how to
address this within the MCE framework, the suitability of MCE applications to different
cultures, the need for validation of results and sensitivity analysis and the need for much
greater consistency of the meaning of terms used in Multi-criteria approaches.

Who decides?
One of the most asked questions when presenting the finding of a Multi-criteria Evaluation is
that of how did you go about choosing the stakeholders involved in the analysis? A
stakeholder can be defined as a person or group who can affect or is affected by the outcomes
of the decision at hand. It is generally considered that for the decision process to be a
responsible and fair one, that process requires consideration of the effects on all stakeholders.
In analyses involved with natural resource management, the questions at hand often concern
public goods or resources such as decisions involving public forests, wetlands or oceans. It is
generally considered appropriate that the public should be the decision maker in these cases.
In recent years, increasing attention has been given to incorporating public participation into
public policy formulation, in particular natural resource policy, in the literature (Bass 1995;
Buchy et al. 2000; Buchy and Hoverman 2000; Race and Buchy 1998; Cassels and Valentine
1988; Fagence 1977; Slocum 1995). The advantages of allowing public involvement in
natural resource policy-making have been well documented and such participation often
strives for wider community understanding and therefore sanctioning of the policies con-
cerned (Hoverman 1997). In this way it is hoped that decisions are more likely to command
assent and therefore lead to the desired outcomes if they have been formulated with public
support.

167
 Proctor and Qureshi

While much has been written on the importance of taking account of the interests of the
public through the various stakeholder groups, there is far less guidance in the MCE literature
as to how to identify appropriate stakeholder groups and their representatives, in a natural
resource management context. In particular two questions need to be answered: how should
relevant stakeholder groups for management of a particular resource bundle be identified; and
how should individuals be selected to represent the interests and objectives of the selected
stakeholder groups. The first of these questions involves selecting the most appropriate
groups or individuals for which objectives and preference weights are to be derived. The
second involves issues of sample size and sample selection methods.

If the number of stakeholder groups must be restricted for pragmatic purposes, a number of
questions arise:
what type of process should be used for selecting stakeholder groups?
should groups be chosen on the basis of some measure of size, importance, influence,
amount of technical knowledge, extent of potential effect or vulnerability to decisions?
are the relevant groups relatively obvious to the researcher, so that the choices are simple
to make?
do the relevant groups automatically identify themselves, e.g. because they are highly
vocal or because some form of community consultation process is invariably used?
should greater weight be given to the objectives of some stakeholder groups than others?
how do the relevant stakeholder groups vary with scale of the analysis (district, regional,
national)?
should government agencies be included as stakeholders, or should these be viewed as
representatives of other stakeholders?
should researchers (including those carrying out the MCE) be included as a stakeholder
group?

One way of choosing stakeholders is for them to be selected based on a demographic

stratification? of the population that will be affected by the decision. The choice of stake-
holders can be made using a random sample or a stratified random sample of this relevant
population. This selection process can be based on a telephone survey in which both
demographic and attitudinal data is collected. Such data includes gender, age, place of
residence, ranking of the environment in relation to other social issues, occupation, income,
income source and level of education. However, this is an extremely expensive process and it
is very rare to see such a selection procedure carried out in Multi-criteria Evaluation.

Another technique is that of Stakeholder Analysis. Stakeholder analysis may be thought of as

an expansion of the traditional narrow view of the firm, where only individuals or groups
who supplied resources or bought products were viewed as important stakeholders
(Grimble and Chan, 1995, p. 114). Although stakeholder analysis as such is new in the field
of multicriteria analysis, other disciplines notably anthropology and sociology have used
this method for many years, and an extensive literature exists about qualitative approaches to
this type of research (e.g. see Patton, 1990).

Often less formal and more ad hoc methods are used to select stakeholders. For example, in
one study it was suggested that stakeholders be initially identified through reputation, focus
groups or demographic analysis (Grimble and Chan (1995). However, often most groups will
be self-selecting or at least the choice will be readily apparent. Community groups frequently
seek opportunities to put forward their views. Staff of government resource management
agencies typically are those charged with the responsibility to provide advice to decision
makers in government. Researchers in government agencies and academia who select
resource management projects (and perhaps obtain grants to work on them) - in effect
nominate themselves. Frequently, the selection process will not take place in one shot but

168
 Proctor and Qureshi

rather will be iterative, where discussions with pre-identified stakeholders reveal other,
previously unknown, stakeholders.

At the same time, the government is the representative of the community in a broad sense, so
including government and community groups could be considered duplication. This stance is
reinforced if we take the view that government is the honest and unbiased servant of the
community, and should not have any agenda of its own to promote.

A difficulty with having government as a stakeholder concerns the balance of power between
stakeholders. As noted by Yu (1999, p. 54-55) in the context of the Chinese coal industry,
[t]he government represents the most formal and powerful stakeholder of the industry As
a stakeholder, their interests and requirements supercede all others Literature suggests
that, normally, these two roles (as owner and as administrator) are in conflict and have a
decisive influence on the performance of public organisations.

It is perhaps a nave view that government is the unbiased servant of the people. There is a
substantial literature on agency goals. The professional progress of officers in government
departments will depend on the size and influence of their departments; government
employees can have a vested interest in policy decisions, in terms of status, salaries,
opportunities for interesting work and indeed their continued employment. It is also apparent
that a high level of client capture is often present among government departments, and that
considerable departmental revenue can be at stake from resource management decisions.

Table 1 gives a broad overview of the various methods that have been used to choose
stakeholders and shows the variety of different techniques undertaken.

Table 1 Multi -criteria Analysis studies and types of preference weights or stakeholders

Author Application area Stakeholders

Janssen and Ranking of 118 agricultural land No stakeholders reported
Herwijnen (1991) regions according to their suitability for
combined land use and change in land
use in Holland.
Bodini and Giavelli Investigation of compatibility between Eight viewpoints were constructed
(1992) conservation and development on including seven on the basis of the
Salina Island, Aeolian Archipelago, professions (productive, service sectors, job
Italy. seekers, housewives, students and retired
persons) and one neutral viewpoint used for
reference purpose
Janssen et al (1990) Application to locate two sites out of Determined priorities by experts of the
nine for newly proposed nuclear power government advisory board on Physical
plants in the Netherlands. Planning and ordered the criteria from most
to least important.
Villa et al (1996) Evaluation of the vegetation No stakeholders reported
component of suburban park in
northern Italy.
Rosato and Stellin Evaluation of management schemes for No stakeholder group was identified.
(1993) rural land. Instead, they used hypothetical weights for
agricultural production and environmental
quality and conservation.
Balteiro and Forest planning problems in Spain. Derived the preferential weights from six
Romero (1998) academic members (expert committee)
from the staff of the Forestry School at the
Technical University of Madrid.

169
 Proctor and Qureshi

Howard and Nelson Application to area-based harvest Used two sets of criteria weights. First, with
(1993) scheduling and forest land allocation. equal weights and second with unequal
weights which were chosen arbitrarily.
Qureshi (1999) To determine optimal riparian Obtained preferences weights of five
revegetation options in a catchment in different stakeholder groups towards
north Queensland, Australia. riparian revegetation options by carrying
out a survey through questionnaires.
Proctor (2001) Evaluation of scenarios of Used stakeholders identified in official
convervation versus logging in a government process (Regional Forest
forested area of NSW Agreement process)
Proctor and Evaluation of future recreation and Used natural resource managers involved in
Drechsler (in press) tourism management options in developing a future plan for the region
highland Victoria
Van Huylenbroeck Analysis of conflicts between rural Involved three different interest groups in
and Coppens (1995) development scenarios in the Gordon the evaluation. The group of local planners
District, Scotland. and decision makers of the district was
asked to compare each pair of criteria on a
nine-point scale. On the basis of these
scores, they derived ranking of the criteria.
Then they used these weights to determine
the ranking of alternatives.
Janssen and Evaluation of land-reallotment plans in Obtained ordinal weights by interviews of
Rietveld (1985) the Netherlands. different research teams involved in the
project. Aggregated these unweighted
average weights to obtain aggregate
weights. These weights were used in the
analysis to obtain ranking of the
alternatives.

The point that we wish to make is that whichever method is used, it should be explained in
detail and validated for the particular decision context at hand. If detailed survey process or
Stakeholder Analyses are not undertaken then one way of doing this is to ask those
stakeholder already involved to consider if all of the stakeholder that should be invited to
partake have been included.

The problem of how to choose the stakeholders leads into the next issue when considering
multiple decision makers.

Multiple Weights
It seems that many MCE techniques have been developed with the intent that only one
decision maker would be involved (perhaps the analyst herself) and, although research is now
growing (see for example Emond and Mason 2002, dAngelo et. al 1998, Mateos et. al 2003,
Belton and Stewart 2001) there has been a distinct lack of guidance in the Multi-criteria
Evaluation literature as to how to carry out interactive group analyses when multiple stake-
holder weightings are required. Often MCE does not address the difficult task of how to
analyse or aggregate multiple weights and most methods provide some sort of average over
the various weights and therefore important information concerning the extent of different
priorities is lost in the process. Although the technique does facilitate a rigorous and objective
method of incorporating public participation into a decision-making process some parts of this
process require skills that are not addressed in the MCE literature. For example, in carrying
out the first part of an analysis to elicit criteria, indicators and preferences, a large amount of
time can be spent in interacting with groups individually or and as part of regular meetings.
Conducting such a facilitating role in issues that are highly contentious in nature requires
certain skills which are not addressed in most text books on Multi-criteria Evaluation. The

170
 Proctor and Qureshi

Analytic Hierarchy Process appears to be the only method (albeit briefly) which addresses the
problem of conducting interactive group decision-making. To be effective in this part of the
process, it is necessary for the analyst to borrow from the literature on negotiation, facilitation
and meeting techniques. Related to this point is the importance of discussing and explaining
the different criteria of assessment used. This is of utmost priority when asking the decision-
maker to complete a weightings questionnaire of the importance of the different criteria and
sub-criteria. This aspect becomes evident when it is discovered that some time can be spent
following up and clarifying individual respondents questions about the exact meaning of the
various criteria and sub-criteria. There exists great scope for incorporating some other
techniques, for example, the deliberative approach of the Citizens Jury into the Multi-criteria
Evaluation (see for example, Coote and Lenaghan 1997 and Proctor 2001). Deliberative
Multi-criteria Evaluation (Proctor and Drechsler, in press; Proctor et. al, in press), has been
developed in response to the problem of dealing with multiple stakeholders and weightings
and has been successfully applied in various complex decision making processes with up to
twenty one stakeholders taking part.

Cross cultural implementation

Another aspect of concern of the MCE technique is whether or not it can provide a decision-
making aid when the decision-makers are not from Western cultures that this technique has
been largely developed for. In a previous study using the Analytic Hierarchy Process (Proctor
2001) Indigenous Australian representatives in the process could not be encouraged to take
part in the pairwise comparison questionnaire (the technique developed for the AHP where
only two criteria are compared at a time rather than trying to compare and weight over the
whole of the criteria set). Even though this may have been due to the late arrival of
Indigenous stakeholders into the official government-led process, it may also indicate that the
particularly structured approach of the AHP does not appeal to the decision-making processes
of Indigenous peoples and this may be true of other MCE weighting methods. It was
impossible to draw conclusions on this because of the presence of other serious problems that
have been identified with regards to the inclusion of Indigenous peoples in the official
government process that this MCE had linked into (i.e. the MCE was not part of the official
process but used the stakeholder group that was part of the official process). These problems
were identified as:
a lack of clarity as to the role of the Indigenous representatives in the RFA process
the centralised, technical and highly political nature of the process, and
barriers to participating in the process due to, for example, lack of experience and
differing communication styles (Engel 2000).

A study on appropriate Multi-criteria Decision Support tools for aiding Indigenous decision-
makers was carried out by Venn and Harrison (2001). They found that non-participatory
practices utilising decision support tools were best for the Wik people (the Indigenous people
of the Aurukun Shire, Cape York Peninsula, Queensland) in the case study that they carried
out. In particular, they reported that the evaluation of criteria via a scale of intensity of
preferences may be meaningless due to:
poor literacy
inadequacy of short term participatory processes
the fact that such processes require knowledge of the internal governance structure of the
tribe that may not be obtainable or obvious (Venn and Harrison 2001, p. 1099).

Their study, which was to help the Wik people decide between different forest options on
their land had parallels and problems with the inclusion of the Indigenous people in the RFA
process (see for example Engels conclusion, this page and observations made on the Eden
RFA process, p. 125) and they noted that:

171
 Proctor and Qureshi

Wik people have expressed a preference not to be involved in multi-stakeholder

workshops. In the past, Wik people and their representatives have had unfavourable
dealings with some government organisations that would be obvious inclusions in
multi-stakeholder consultations. Wik people feel they will be disadvantaged in such a
forum where other stakeholders have more detailed knowledge and have greater
capacity to articulate their particular preferences and concerns about forest manage-
ment options than Wik people (Venn and Harrison 2001, p. 1100).

They concluded that one decision support tool, Goal Programming, was an appropriate aid to
support the decisions of the Wik people but did not give recommendations for what type of
participatory process should be used when incorporating the preferences of Indigenous
peoples in a Multi-criteria Evaluation, opting instead to recommend against using any
participatory process.

Validation of results and sensitivity analysis

Verification refers to building the model correctly, i.e. substantiating that a model properly
implements its specifications (OKeefe et al., 1991). Verification ensures that the model has
been developed in a formally correct manner in accordance with a specified methodology
(Geissman and Schultz, 1991). In the case of an algebraic model implemented by computer
program, verification establishes that the program has been written correctly for the computer
and that it behaves as intended.

Validation refers to building the correct model, i.e. establishing that a model achieves an
acceptable level of accuracy in its predictions (OKeefe et al., 1991). Tests of data validity,
conceptual validity and operational validity may be conducted to determine whether the
structure of the model is appropriate for its intended purpose (Harrison, 1987). In practice,
most attempts at model validation appear to check for close agreement between outputs of the
model and those of the real system, i.e. the validation process has largely been confined to
checking operational validity, or the ability of a model to mimic the real system. For these
comparisons to be valid, model and real system outputs need to be have been generated for
the same management policies and under the same sets of uncertain environmental values.

Sensitivity analysis examines the extent of variation in predicted performance when

parameters are systematically varied over some range of interest, either individually or in
combination. This form of testing may be viewed as checking the stability of a model which
has been validated as far as is practicable. Sensitivity analysis provides further confidence in a
model, and indicates priority areas for refinement if further versions of a model are to be
developed.

Sensitivity analysis is carried out for several reasons. The first is because of the nature of the
MCE process, which inherently contains varying levels of uncertainty because of the
qualitative and subjective choice of various parameters. MCE has been criticised for being
and inexact procedure on these grounds making it very important to test how robust the
results are. A second and very important reason is that the sensitivity analysis of the MCE
procedure enables the data and the decision making problem to be explored at greater depths.
This provides greater insights into the nature of the decision problem, unravel its complexities
and may even provide recommendations for future analyses. A sensitivity analysis of the
results may be carried out in order to take into account the uncertainty in estimating some of
the figures involved. A decision-maker may be unsure of a particular weighting and may
provide a range of weightings which can be analysed. Similarly, the impacts of the various
options under different criteria may fall within a statistically estimated range that can be
incorporated into the analysis. Sensitivity analysis can also consider the effects of different
techniques used in the weighting procedure, for example.

172
 Proctor and Qureshi

It is very important that the sensitivity of outcomes can be tested for different values of the
most crucial and contentious criteria and impacts. For example, in a group decision-making
situation, if it were found that there was a great disparity in preferences for a certain criterion
then it may be enlightening to find out how the overall results change with the changes in
preference levels for this criterion. If the results are not greatly affected, then the criterion can
take less importance in the overall process and the decision-makers can concentrate on other
criteria and trade-offs. If the results are extremely sensitive to this criterion, then closer
scrutiny should be given to it by confirming values and measurements.

The sensitivity analysis is given a dominant role in a technique incorporating multiple

decision-makers and risk analysis called Multi-criteria Mapping (Stirling and Mayer 1999).
Mapping refers to that part of the analysis where the results are expressed in terms of
various, systematically applied, sensitivities with prescriptive conclusions being drawn
only conditionally, by reference to the clearly-defined perspectives taken by different
participants (Stirling and Mayer 1999, p. 69). Another example of sensitivity analyses
accounting for risk and uncertainty involved with the data uses Monte Carlo simulation to
estimate probability distributions for the underlying data sets so that the estimated means and
variances of the data can be incorporated to assess outcomes (Van Delft and Nijkamp 1977).

As with validation of models in other areas, MCE modellers have paid less attention to the
credibility of models than to model development. Validation is not generally regarded as a
critical component in the development of well-structured models such as mathematical
programming models since the model structure is well recognised, and reasonably well
accepted. However, MCE models go beyond mathematical programming, and hence have a
greater testing requirement.

Table 2 summarises testing procedures employed in recent MCE studies in natural resource
and environmental management. Except for Braat (1992), none of these studies carried out the
three components of model testing (verification, validation and sensitivity analysis). Jones et
al. (1990) tested their multi-attribute model by obtaining the views of model users through
workshops. Sensitivity analysis was reported by Bodini and Giavelli (1992), Hokkanen and
Salminen (1994), Abu-Taleb and Mareschal (1995), Janssen (1996) and van den Bergh et al.
(1996). Others did not report any steps to establish the credibility of their models.

Table 2: Evaluation procedures reported in studies of natural resource and environmental

management

Topic, author and country Model testing procedures adopted

Sustainable Environmental Management,
Netherlands (Nijkamp, 1989).
Multi-attribute Value Model for the Study of Energy
Policy, U.K. (Jones et al., 1990).
Compatibility of conservation and development,
Salina Island, Aeolian Archipelago, Italy (Bodini and
Giavelli, 1992.
Sustainable Multiple Use of Forest Ecosystems,
Netherlands (Braat, 1992).
Solid Waste Management System, Finland
(Hokkanen and Salminen, 1994).
Environmental Assessment of Electric Transmission
Project, Quebec, Canada (Rousseau and Martel,
1994).
No evaluation steps reported.
Validation by obtaining comments from users of the
model
Sensitivity analysis to test the results but no
validation.
Verification, validation and sensitivity analysis used
to evaluate the model.
Sensitivity of alternatives evaluated by changing
weights and threshold values. Technical reliability
evaluated using expert questionnaire.
No evaluation process reported.

173
 Proctor and Qureshi

Energy Planning, Greece (Mavrotas et al., 1994). No evaluation procedures reported.

Water Resource Planning, South Africa (Stewart and
Scott, 1995).
Water Resource Planning, Middle East (Abu-Taleb
and Mareschal, 1995).
Coastal Management, Greece (Moriki et al., 1995).
Sustainable Use of Forests, Australia (Janssen, 1996).
Multicriteria Evaluation for a Greek Island Region
(ven den Bergh et al., 1996).
Land and Water Management Plan, NSW, Australia
(Assim et al., 1998).
Proctor (2001)
Proctor and Drechsler (in press)
Questionnaire administered to all interest group
representatives to establish credibility of model.
Sensitivity analysis to compare alternative options
under different weighting schemes and changes in
constraint levels.
No evaluation steps reported.
Sensitivity analysis carried out by changing criteria
scores (25%)
Sensitivity of results regarding uncertainty of
weights and scores investigated.
No evaluation steps reported.
Extensive sensitivity analysis and evaluation by
questionnaire to stakeholders
Sensitivity analysis and evaluation by questionnaire
to stakeholders

Consistency of Terms
The literature covering methods or aids in the process of decision-making is extensive and, as
mentioned previously, covers many different types of applications in many different disci-
plines. This may be one reason why there appears to be little consensus as to the correct
naming of the particular class of decision tools used. For example, the terms Multi-criteria
Evaluation, Multi-criteria Analysis and Multi-attribute Decision-making are terms which refer
to the same methodology (see for example: Resource Assessment Commission 1992d; Yoon
and Hwang 1995). Similarly, the term Multiple Objective Decision Methods has been used to
refer to the overall class of decision aids in some studies (for example, Rietveld 1980) whilst
Multi-objective Decision-making has most often been used to refer to those techniques which
involve continuous alternatives and incorporate mathematical optimisation methods (Kazana
1999, Yoon and Hwang 1995). The term Multi-criteria Decision-making is most often used to
refer to the whole class of techniques which attempt to aid the decision-maker or decision-
makers in choosing between conflicting options or alternatives, which may have conflicting
and multiple objectives and criteria for evaluation.

In this paper, Multi-criteria Evaluation refers to the class of decision aiding techniques which:
decide between a finite number of options,
use a set of criteria to judge the options, and
use a method by which the options are ranked, based on how well they satisfy each of the
criteria.

The terminology used in many studies on MCE is also often conflicting and confusing. In this
study, the term option is used and is synonymous with terms such as alternative, plan,
policy, action and candidate used in other studies. Options are those outcomes, policy
alternatives or resource use scenarios that are to be decided between. Ultimately the chosen
option will be that one that best satisfies some overall objective. The term criteria refer to
those aspects of each alternative that need to be assessed as to how well each of them meets
the desired objective. In some studies though it is synonymous with the terms goals and
attributes found in other studies (Yoon and Hwang 1995). Indicators refer to the lowest
level of sub-criteria which are measurable.

174
 Proctor and Qureshi

Figure 1 describes taxonomy of various Multi-criteria Decision-making (MCDM) methods.

The whole group is divided into two sub-groups. The first group classed as Multi-objective
Decision-making (MODM), refers to those techniques which design the options using
continuous data and mathematical optimising or linear programming methods. The second
group, classed as Multi-criteria Evaluation (MCE) or Multi-attribute Decision-making,
includes techniques that facilitate a decision between discrete pre-defined options. The grey
shaded boxes describe the attributes of a group of techniques which contain the particular
MCE technique applied in the case study of this research.

All Multi-criteria Decision-making problems attempt to find as close to an ideal or optimal

solution as possible by choosing amongst conflicting objectives. In MODM problems, the
solution involved is an objective one that maximises an objective function subject to a number
of constraints defined mathematically by the criteria. In MCE problems, this solution is
subjective on the part of the decision-maker. In decision theory terms this solution is known
as a non-dominated solution. In economics it is referred to as a Pareto-optimal solution as
follows:
A feasible solution in MCDM is nondominated if there exists no other feasible
solution that will yield an improvement in one objective/attribute without causing a
degradation in at least one other objective/attribute (Hwang and Yoon 1981, p. 23).

Under the MCE group of techniques a further distinction can be made based on the nature of
the criteria. Qualitative, ordinally measured criteria are used in methods such as Regime,
Permutation, Evamix, the Expected Value Method and the Lexicographic Method. Quantit-
ative, cardinally measured criteria are used both in techniques which have monetary values as
the basis of their decision rule, as with Cost Benefit Analysis and also in those which use
some other value or decision rule. In essence, the important point is for practitioners to be
consistent and clear on the meaning of the terms rather than consistent with the terms
themselves. MCE is often refered to as MCA or MCDA etc. (depending on the particular
preference of the author) but should not for example, be used in studies which refer to
optimisation over continuous alternatives.

175
 Proctor and Qureshi

Figure 4.1 A taxonomy of Multi-criteria Decision-making methods (Source: RAC 1992b;

Kazana 1999; Yoon and Hwang 1995)

Multi-Criteria
Decision-Making

Nature of alternatives

Discrete Continuous
-alternatives are -alternatives generated
pre-defined by mathematical models

Multi-Criteria Multi-Objective Decision

Analysis/Multi-Attribute Making/Multi-Objective
Decision-Making Linear Programming

Geometric definition
Nature of criteria/attributes Interactive Methods
of 'best' solution

MINSUM MINMAX INTERVAL

Quantitative Qualitative
(cardinal) (ordinal)

Trade-off
Estimation

Basis for
decision rule
Goal
Surrogate
Programming
Worth Trade-off

Monetary Other
Implicit
values values
Trade-offs

Cost Benefit Weighted Summation

Analysis Lexicographic Weighted Goal
Weighted Product
Methods Programming
Median Ranking Method
Utility theory
Ideal Point
TOPSIS
Analytic Hierarchy Process Source: RAC 1992, Kazana 1999, Yoon and
Concordance Analysis (ELECTRE) Hwang 1995.

176
 Proctor and Qureshi

Conclusions
In this paper we have attempted to raise a number of issues related to MCE which we feel
require greater acknowledgement, consideration and research. We have not provided any
conclusive answers but have raised what we believe are opportunities for significantly
improving the application of MCE. In particular, these include: the need for greater
recognition of the importance of choosing stakeholders and validation of the method by which
this is done, more research into dealing with multiple weights and the means by which
multiple stakeholders are interacted with, much more research into the adequacy of MCE
techniques to be applied in different cultural contexts, greater verification and validation of
models used and sensitivity analysis of results and finally, greater consistency of terms used
in the MCE decision making literature.

References
Abu-Taleb, M.F. and Mareschal, B., (1995), Theory and methodology, water resources planning in the
Middle East: application of the PROMETHEE V multicriteria method, European Journal of Operational
Research 81(1), 500-511.
Belton, V. and Stewart, T. J. (2001) Multiple Criteria Decision Analysis: An Integrated Approach.
Kluwer. Berlin.
Bass, S. (1995) Participation in Policy Processes, Forestry and Land Use Program, International
Institute for Environmental Development, London.
Bodini, A. and Giavelli, G (1992) Compatibility of Conservation and Development in Salina Island,
Environmental Management, 16(5), pp. 633-656.
Braat, L. C., (1992). Sustainable Multiple use of Forest Ecoystems. An Economic-Ecological Analysis for
Forest Management in the Netherlands, PhD Thesis, De Vrije Universiteit te Masterda, Central
Huisdrukkerij VU.
Buchy, M., Ross, H. and Proctor, W. (2000) Enhancing the Information Base of Participatory
Approaches to Australian Natural Resource Management, Commissioned Report to the Land and Water
Resources Research and Development Corporation, Canberra.
Buchy, M. and Hoverman, S. (2000) Understanding Public Participation Planning: a review, Forest
Policy and Economics,Vol. 1, pp.15-25.
Cassels, D. and Valentine, P. 1988, From Conflict to Consensus: Towards a Framework for Community
Control of the Public Forests and Wildlands, Australian Forestry, Vo. 51, pp.47-56.
Coote, A. and Lenaghan, J. 1997, Citizens Juries: Theory and Practice. Institute for Public Policy
Research, London.
dAngelo, A, Eskandari A, Szidarovszky F. (1998) Social choice procedures in water resource
management, Journal of Environmental Management 52, pp. 203210.
Emond, E J, Mason, D W, (2002) A New Rank Correlation Coefficient with Application to the
Consensus Ranking Problem. Journal Of Multi-Criteria Decision Analysis 11, pp. 1728.
Engel, A. 2000, Dilemmas of Participation: A Case Drawn From the Southern Regional Forest
Agreement (RFA) Process.
Fagence, M. 1977, Citizen Participation in Planning, Pergamon Press, Oxford.
Geissman, J.R. and Schultz, R.D. (1991), Verification and Validation of Expert Systems, In Gupta, U.
(Ed.), Validating and Verifying Knowledge-based Systems pp. 12-19, IEEE Computer Society press,
Washington.
Grimble, R. and Chan, M. (1995), Stakeholder Analysis for Natural Resource Management in
Developing Countries: Some Practical Guidelines for Making Management More Particpatory and
Effective, Natural Resources Forum, 19(2), pp. 113-124.
Harrison, S. (1987), Validation of Models: Methods, Applications and Limitations, in Computer
Assisted Management of Agricultural Production Systems, RMIT Press.
Hokkanen, J. and Salminen, P., (1994), The choice of a solid waste management system by using the
ELECTRE III decision aid method, in Paruccini, M. (Ed.) Applying Multiple Criteria Aid for Decision to
Environmental Management, Kluwer Academic Publishers, Boston, pp. 11-155.
Hoverman, S. 1997, Environmentalism and Social Change: Public Participation in Australian Forest
Management, Unpublished PHD thesis, University of Hawaii.
Howard, A.F and Nelson, J. (1993), Area-based Harvest Scheduling and Allocation of Forest land using
Methods for Multiple-criteria Decision Making, Canadian Journal of Forest Research, 23(2) 151-158.

177
 Proctor and Qureshi

Hwang, C. and Yoon, K. 1981, Multiple Attribute Decision Making, in Beckman, M. and Kunzi, H.
(eds) Lecture Notes in Economics and Mathematical Systems, Springer-Verlag, Berlin.
Janssen, R. (1996), Multicriteria Evaluation for Sustainable Use of Australian Forests. In Van den
Bergh, J. (Ed.), Ecological Economics and Sustainable Development: Theory, Methods and Applications,
pp. 231-242, Edward Elgar, Cheltenham.
Janssen, R. and Rietveld, P. (1985), Multicriteria Evaluation of Land-Reallotment Plans: A Case Study,
Environment and Planning A, 17(12), pp.1653-1668.
Janssen, R. and Van Herwijnen (1991), Graphical Decision Support Applied to Decisions changing the
Use of Agricultural Land, in Korhonon, P, Lewandowski, A. and Wallenius, J. (Eds.), Multiple Criteria
Decision Making, Springer, Berlin, pp. 293-302.
Janssen, R., Nijkamp, P. and Rietveld, P. (1990), Qualitative Multicriteria Methods in the Netherlands,
in Bana e Costa, C. (Ed.) Readings in Multiple Criteria Decision Aid, Springer Verlag, Berlin, pp. 383-
405.
Jones, M., Hope, C. and Hughes, R., (1990) A Multi-attribute Value Model for the Study of UK Energy
Policy, Journal of the Operations Research Society, 41(10), pp. 919-929.
Kazana, V. (1999), A Critical Overview of Multiple Criteria Decision Making Methods for natural
Resource Planning and Management with Reference to Practice in Sisak, L., Jobstl, H. and Merlo, M.
(Eds.) Proceedings of the Symposium From Theory to Practice: Gaps and Solutions in Managerial
Economics and Accounting in Forestry, Prague May 13-15, 1999.
Mateos, A, Jimenez A, Rios-Insua S (2003) Modelling Individual and Global Comparisons for Multi-
Attribute, Preferences, Journal Of Multi-Criteria Decision Analysis, 12, pp. 177190.
Mavrotas, G., Diakoulaki, D. and Assimacopoulos, D., (1994), Energy Planning and Trade-offs between
Environmental and Economic Criteria, in Paruccini, M. (Ed.) Applying Multiple Criteria Aid for
Decision to Environmental Management, Kluwer Academic Publishers, Boston, pp. 187-198.
Moriki, A., Coccossis, H. and Karydis, M., (1995), Multicriteria Evaluation in Coastal Management,
Journal of Coastal Research, 12(1), pp. 171-178.
Nijkamp, P., (1989), Multi-criteria Analysis: A Decision Support System for Sustainable Environmental
Management, in Archibugi, F. and Nijkamp, P. (Eds.) Economy and Ecology towards sustainable
Development, Kluwer Academic Publishers, London, pp. 203-259.
OKeefe, R. M., Osman, B. and Smith, E.P., (1991), Validating Expert Systems Performance, In Gupta,
U. (Ed.) Validating and Verfying Knowledge Based Systems, IEEE Computer Society Press, Washington,
pp. 2-11.
Paton, M. (Ed.) (1990), Qualitative Evaluation and Research Methods, 2nd Edition, Sage Publications,
Newbury Park.
Proctor, W. (2001), Multi-criteria Analysis and Enviromental Decision-making: A Case Study of
Australias Forests, Unpublished PhD Thesis, Centre for Resource and Environmental Studies,
Australian National University.
Proctor, W. and Drechsler, M. (in press), Deliberative Multi-criteria Evaluation, Environment and
Planning C- Government and Policy, Pion, UK.
Proctor, W., McQuade, C. and Dekker, A. (in press), Managing Environmental and Health Risks from a
Lead and Zinc Smelter: An Application of Deliberative Multi-criteria Evaluation in Herath, G. and Prato,
T. (Eds) Using multi-criteria decision analysis in natural resource management, Kluwer, London.
Qureshi, M. (1999), Development and Implementation of a Decision Support Process for Sustainable
Catchment Management, Unpublished PhD Thesis, University of Queensland.
Race, D. and Buchy, M. 1998, A Role for Community Participation in Australian Forest Management?,
Rural Society, Vol. 9, No. 2, pp. 405-419.
Resource Assessment Commission 1992d, Multi-criteria Analysis as a Resource Assessment Tool, RAC
Research Paper No. 6, March, AGPS, Canberra.
Rietveld, P. 1980, Multiple Objective Decision Methods and Regional Planning in Andersson, A. and
Isard, W. (Eds) Studies in Regional Science and Urban Economics, Volume 7, North-Holland Publishing
Company, Amsterdam.
Rosato, P. and Stellin, G. (1993), A Multi-Criteria approach to Territorial Management: The Case of the
Caorle and Bibione Lagoon Nature Park, Agricultural Systems, 41(1), pp. 399-417.
Rousseau, A. and Martel, J. (1994), Environmental Assessment of an Electic Transmission Line Project:
a MCDA Method, in Paruccini, M. (Ed.) Applying Multiple Criteria Aid for Decision to Environmental
Management, Kluwer Academic Publishers, Boston, pp. 163-186.
Saaty, T. 1980, The Analytic Hierarchy Process. McGraw-Hill, New York.
Slocum, R. (Ed.) (1995), Power, Process and Participation: Tools for Change, Intermediate Technology
Publications, London.

178
 Proctor and Qureshi

Stewart, T. and Scott, L., (1995), A Scenario Based Framework for Multicriteria Decision Analysis in
Water Resources Planning, Water Resources Research, 31(11), pp. 2835-2843.
Stirling, A. and Mayer, S. 1999, Rethinking Risk: A Pilot Multi-Criteria Mapping of a Genetically
Modified Crop in Agricultural Systems in the UK, Science and Technology Policy Research Unit,
University of Sussex.
Van Delft, A. and Nijkamp, P. 1977, Multi-criteria analysis and regional decision-making, in Nijkamp,
P. (Ed.) Studies in Applied Regional Science, Vol. 8, Martinus Nijhoff Social Sciences Division, Leiden.
Van den Bergh, J., Janssen, R., Van Herwijnen, M. and Nijkamp, P. (1996), Integrated Dynamic and
Spatial Modelling and Multicriteria Evaluation for a Greek Island Region, In Van den Bergh, J. (Ed.),
Ecological Economics and Sustainable Development: Theory, Methods and Applications, ch. 11, Edward
Elgar, Cheltenham.
Van Huylenbroaeck, G. and Coppens, A. (1995), Multicriteria Analysis of the Conflicts Between Rural
Development Scenarios in the Gordon District, Journal of Environmental Management and Planning,
38(3), pp. 393-407.
Villa, F., Ceroni, M. and Mazza, A., (1996), A GIS Based Method for Multi-Objective Evaluation of
Park Vegetation, Landscape and Urban Planning, 35(4), pp. 203-212.
Venn, T. J. and Harrison, S.R. 2001, Appropriate Multi-Criteria Decision Support tools for aiding
Indigenous decision-makers in Aurukun Shire, Cape York Peninsula, in Ghassemi, F., McAleer, M. and
Scoccimarro, M. (Eds.), MODSIM 2001, Modelling and Simulation Society of Australia and New
Zealand Inc. Proceedings, ANU, 10-13 December, Vol. 3: Socioeconomic Systems.
Yoon, K. and Hwang, C. 1995, Multiple Attribute Decision Making: An Introduction, in Lewis-Beck,
M. (Ed.), Quantitative Applications in the Social Sciences, Series/Number 07-104, Sage Publications,
Thousand Oaks.
Yu, X. (1999), Stakeholder Interests and the Performance of the Chinese State Coal Sector,
Unpublished PhD Thesis, University of Queensland.

179