LECTURE #13*

EVALUATION: THE MULTI-CRITERIA DECISION PROBLEM

13.1 Introduction

The 3rd essential element of EA is the evaluation of alternatives for decision making We
need to determine we should reject or accept alternative that produce the least and acceptable
environmental impacts. It is essential to explain the logic to say “no” and the choice among
alternatives to say “yes.” Some people may consider the EA a planning instead of a “get an
approval” process. In fact, the decision to reject and approve is implicit in every EA. There is
no method requirement the act and the key issue is “the method” of evaluation. The proponent
can choose the best for his/her proposal. Although there is a class of problems in planning and
decision theory, the challenge we have in EA is a multi-objective (e.g. aquatic, water quality,
soil, vegetation, wildlife) and multi-criteria (natural environment, human health, social resources,
economy) problem. Why is it difficult to evaluate alternatives? Up to this element of EA, we
have dis-aggregrated our problem into a number of impact analyses. It is the time to put them all
back together to synthesize the overall impact and evaluate whether we can decide the fate of this
project. In do so, we recognize that value to each impact be assigned for the overall impact
evaluation. It means EA practitioners are forced to judge relative value of impacts and make
trade off. The difficulty is “what is worth more?” Is 100 m of stream run of trout habitat worth
more than 2 ha of deer habitat?

13.2 Challenge in Evaluation

Evaluation is the element in the EA process whether “trade-offs” are (inevitable) made.
It is important to explain explicitly and exactly how the trade-offs are made. Remember, we are
moving from disaggregation to aggregation in the EA process as illustrated in the following
diagram.

*This lecture was developed using materials supplied by Prof. Ron Pushchat, School of
Occupational and Public Health, Ryerson University.

1
 Impacts

Proposed Integrated
Action for View of Decision
All alternatives Impacts

Disaggregation Aggregation

Phase 1: Analysis Phase 2: Synthesis

Examining (e.g. visual inspection) the impacts does not always reveal the correct alternative.
For example, the Go-transit routing project. There were 7 route alternatives and 63
environmental impact criteria (e.g. air, water, physical, social, cultural, land use, etc.). In total,
environmental impact matrix had 441 cells where impact levels and weights had to be assigned.
The client (Go-transit) thought they could visually assess the matrix and determine which one
was best. Although human mind is a good microprocessor, it is not good enough to process
many cells simultaneously. In fact, visual examination is often biased as human tendency to
select good aspects and ignore bad aspects.

2
 Alternatives
Resource 0 to 7 km 7 to 13 km
A B M B C M
Hydrology

Vegetation

Fish

Wildlife

Sociology
(include
safety)
Land use
(include
recreation)
Visual

Noise

Note: The larger the circle the more adverse the impact. Alternative is the preferred route.
Alternative schemes Description
0 km – 7 km
- A Parallel to railway
- B Parallel to existing highway – wide median
- C Parallel to existing highway – barrier median
7 km – 13 km
- C Moving the railway – new east bound lands on old railway line
- B Parallel to existing highway on north side – minimum median
- M Parallel to existing highway – barrier median

Which is the preferred route?

The common response to evaluation problem is too many alternatives and too many values for
each criterion. We need an evaluation method that (1) systematically counts up the “goods” and

3
“bads” for each alternative and (2) indicates what is valued more than something else. It is the
relative value that is hard to determine without some knowledge of society’s value to nature and
human environment. Whose values should we use and how do we discover those values? This
is the most unsatisfactory and challenging component of EA because this is also the hardest part
to do it fairly. It is hard for us as a society to reach agreement among many people for all the
criteria. See K. Arrow’s Phd thesis (1952) “Social choice and Individual Values.” He won the
Noble Prize in Economics in 1972.

The multi-criterion decision problem in EAs is irresolvable because

(a) Multiple objectives (e.g. maintain water quality, provide adequate capacity, etc.) are
binary choices (i.e. one or other, can’t have both); and
(b) Multiple criteria (e.g. dissolved oxygen > 5 mg/L, hectares of Class 1 agricultural farm
land) are many and in different units of measurement (e.g. mg/L, no. of jobs).
Every evaluation method has some means of evaluating “apples” and “oranges”, some ranking of
importance or significance for each criterion, and some input of values.

Since we need to weigh each criterion, how we determine the weight? Should we ask the
proponent or public? Many argue that evaluation should be part (a) of a public choice process;
(b) open to public; (c) transparent to public; and (d) responsive to public. This is understandable
because
(a) The choice is not made to benefit a private interest. It is to protect environmental
quality as stated in the purpose of EA Act and reflect public values.
(b) The choice is part of democratic process. It should be able to be influenced by those
affected, be “seen” to be fair, and virtually free of bias (recognizing proponent writes
the EA document). There are evidence of attempt to remove bias in EA as all
alternatives must be considered.

How can we make the EA decision a public one? Should we get every stakeholder or the
proponent to vote for the favourite alternatives? To attempt to do it fairly, we should get public
to value the environmental features and let the proponent use them to evaluate and suggest the
favourable choice. In fact, one measure of evaluation success may be degree to which public
values are recognized.

13.3 Evaluation Method

Each EA evaluation uses a method (process) for arriving at decision even though the EA
Act does not require that a specific method be used. Every evaluation method allows judgements
about significance of impacts be made using a set of criteria. Unfortunately, some or many EAs
appear to be an evaluation of one preferred alternative even though there are always alternatives
for each proposed project. In the pre-EA past, alternative methods were evaluated by proponent
judgement and usually based on proponent interests (e.g. level of service, cost and return, and
schedule). EA has forced the proponent to show how the decision is made. It opens decision
making to be scrutinized and changes the nature of decision making. The choice of evaluation

4
methods is important. Without explicit method, the proponent is free to decide.

Every EA uses an evaluation method which could be explicit or implicit. There is no

single or certain method that can be universally applied. None of the methods are completely
satisfactory. There are four fundamental problems in evaluation methods:
(a) Weighting
There is no specific method for specifying weights. Weights are always intuitive and
judgemental and should be determined in a public process. However, it is a major source
of frustration for those who want a simple single “aggregated value.” Generally, it is
assumed that (i) weights are independent of (i) one another (i.e. a big impact in aquatic
species oe snot mean a big impact in water quality); and (ii) the raw criteria scores (i.e. a
big raw score does not increase the size of the weight and make it more important).
Additionally, it is assumed that weights are determined “A Prior” to prevent altering
weights to change the outcome of evaluation.
(b) Measurement units
Many criteria have incommensurable units (e.g. mixture of quantitative and quality units)
and cannot be measured together. In order to add the scores up for all criteria, we need to
transform incommensurable units into a common scale. The common scale can be in
dollars in the cost/benefit analysis or abstract units.
(c) Aggregation
For matrix aggregation, we can either do visual inspection/interpretation or add up scores
for each criterion to produce a “grand index” for each alternative. The “grand index”
enables us to compare alternatives and determine the best alternative. However, some
practitioners suggest that “grand index” aggregate be avoided because it conceals
important information about individual impacts and qualitative data cannot be aggregated
with quantitative data.
(d) Decision criterion
The key outcome from evaluation is the decision to reject or accept the alternative. In
order to do this, we have to set the decision criterion. If you are deciding in an EA the
best alternative in terms of environment, you have already decided all alternatives are
minimally acceptable. Many EAs try to avoid this question in evaluation and send all
alternatives and the evaluation outcomes to the decision makers such as the
Environmental Advisory Board or the Cabinet.

13.4 Significant Impacts

Most EIA legislations around the world emphasize that projects be rejected if the impacts
to natural and/or human environment is significant. Unfortunately, the term “significant” is
never defined at any point in these EA legislations. So, how can we reject a project with
significant environmental impacts without defining what is “significant” supposed to mean?
How can we define “significant”? Dictionary definition of “a significant impact” is meaningful,
important, or notable impact. In EA literature, “a significant impact” is defined as outside
acceptable limits. In fact, the definition of “significant impact” is highly subjective. It depends
upon the knowledge or judgement of natural system of the assessor based on accumulated

5
knowledge and experience. In the end, “significant” is a “collective judgement” of assessors,
stakeholders, and affected public and is an expression of values.

Some guides to judging significant impacts are:

(a) Probability of occurrence
If an impact is highly likely, it is generally more significant. Although an impact may
have low probability, it may have a high impact (e.g. release of carcinogenic/toxic
compounds in Grand River is not insignificant).

(b) Magnitude of impact

Quantitative/qualitative estimate of size of extent of impact can be used to indicate
significant impact (e.g. 50% reduction in available habitat). However, impacts that are
large in magnitude are not necessarily significant. For example, dredging for a new
highway bridge may diminish fish population by 90%. However, fish population may be
re-established in one season after dredging and no permanent environment loss.

(c) Duration
How long will the impact last? Days, months, years, decades.

(d) Reversibility
Can the impact be reversed by human action or naturally rehabilitated?

(e) Environmental policy

What is the effect on local, national or international laws, treaties, etc? (e.g. Kyoto
Accord).

Although none of the above criteria are necessarily more important than the others, they are aids
to assessors in making decision.

13.5 Characteristics of a Good Evaluation Method

Although it appears that no specific method is universally applicable for all projects,
there are characteristics of a good evaluation method. They are:

(a) Objective Evaluation

A good evaluation method should be objective and independent of bias or will of the EA
proponent. EA is already biased (appearance of bias) since it was done by the proponent.
Thus, it is important that data are collected fairly and from unbiased sources.

(b) Systematic Evaluation

Clear procedures should be followed to synthesize data for evaluation. Same results
should be replicated by anyone using the same data and method.

(c) Rational Evaluation

6
 The sequence of evaluation should be logical. For instance, weights should be decided
before using them in evaluation. Fullest appearance of being fair should be emphasized
in the evaluation.

(d) Ease of use and operation

Many evaluation methods use considerable resources and give little indication of impacts.
Thus, the better methods should be easy to use and operate. Although the Leopold
Matrix Method (LMM) is a poor simple method, it has been used for many projects. The
LMM will be explained in the next lecture.