You are on page 1of 12

European Journal of Operational Research 197 (2009) 10631074

Contents lists available at ScienceDirect

European Journal of Operational Research

journal homepage:

Sustainable energy futures: Methodological challenges in combining scenarios

and participatory multi-criteria analysis q
Katharina Kowalski a,*, Sigrid Stagl a, Reinhard Madlener b, Ines Omann c

SPRU Science and Technology Policy Research, Environment and Energy Programme, Freeman Centre, University of Sussex, Brighton BN1 9QE, UK
Institute for Future Energy Consumer Needs and Behavior (FCN), Faculty of Business and Economics/E. ON Energy Research Center, RWTH Aachen University,
Templergraben 55, 52056 Aachen, Germany
SERI Sustainable Europe Research Institute, Garnisongasse 7/27, 1090 Vienna, Austria

a r t i c l e

i n f o

Article history:
Received 15 January 2007
Accepted 5 December 2007
Available online 13 March 2008
Participatory multi-criteria analysis
Energy scenarios
Renewable energy
Sustainable development

a b s t r a c t
This paper analyses the combined use of scenario building and participatory multi-criteria analysis
(PMCA) in the context of renewable energy from a methodological point of view. Scenarios have been
applied increasingly in decision-making about long-term consequences by projecting different possible
pathways into the future. Scenario analysis accounts for a higher degree of complexity inherent in systems than the study of individual projects or technologies. MCA is a widely used appraisal method, which
assesses options on the basis of a multi-dimensional criteria framework and calculates rankings of
options. In our study, five renewable energy scenarios for Austria for 2020 were appraised against 17 sustainability criteria. A similar process was undertaken on the local level, where four renewable energy scenarios were developed and evaluated against 15 criteria. On both levels, the scenario development
consisted of two stages: first an exploratory stage with stakeholder engagement and second a modelling
stage with forecasting-type scenarios. Thus, the scenarios consist of a narrative part (storyline) and a
modeled quantitative part. The preferences of national and local energy stakeholders were included in
the form of criteria weights derived from interviews and participatory group processes, respectively.
Especially in the case of renewable energy promotion in Austria, the paper systematically analyses the
potentials and limitations of the methodology (1) for capturing the complexity of decision-making about
the long-term consequences of changes in socio-economic and biophysical systems and (2) for appraising
energy futures. The paper concludes that assessing scenarios with PMCA is resource intense, but this
methodology captures successfully the context of technology deployment and allows decision-making
based on a robust and democratic process, which addresses uncertainties, acknowledges multiple legitimate perspectives and encourages social learning.
2008 Elsevier B.V. All rights reserved.

1. Introduction
When aiming at sustainable development new challenges for
strategic decision-making arise from the multi-dimensionality of
the sustainability goal, from the complexity of socio-economic
and biophysical systems, and from the long-term nature of the
problems faced. The current trend of rising fossil fuel prices and
observed climate change, and other adverse environmental and
societal impacts of energy use, make the exploration for more sus-

An earlier version of this paper was presented at the 19th Mini EURO Conference
on Operational Research Models and Methods in the Energy Sector (ORMMES06),
Coimbra, Portugal, 68 September 2006.
* Corresponding author. Tel.: +43 1 51522 1316; fax: +43 1 51522 7325.
E-mail addresses: (K. Kowalski),
(S. Stagl), (R. Madlener),
(I. Omann).

0377-2217/$ - see front matter 2008 Elsevier B.V. All rights reserved.

tainable ways to use energy more important than ever. Decisionmaking for sustainable energy futures requires methods that allow
for the complexities of socio-economic and biophysical systems
and that address uncertainties of long-term consequences. For this
purpose, we need novel combinations of methods, which, first,
support the exploration of complexities and uncertainties (opening up phase; Stirling, 2005) and, second, organise the information
and aid decision-making (closing down phase; Stirling, 2005).
Multi-criteria analysis (MCA) has become increasingly popular in
the context of environmental decision-making about energy issues.
The nature of the conflicts about sustainability arising in energy
supply and energy use are shared with wider environmental issues; they are conflicts among heterogeneous socio-economic
interests as well as conflicts between socio-economic interests
and the capacity and functioning of natural systems.
The benefits of combining participatory methods with analytical tools (such as MCA) are widely acknowledged (for a review


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

see Stagl, 2007). Recent methodological innovations, such as social

multi-criteria analysis (Munda, 2004), highlight explicitly social
and technical incommensurability1 and thus acknowledge the operational challenges of evaluating complex (reflexive) systems. The
methodology adopted in our study builds on complex systems theory and recent developments in decision analysis about the use of
MCA in public policy contexts.
In this paper, we elaborate on the merits and challenges of combining participatory MCA (PMCA) with scenario building for analysing and aiding decision-making in a public policy context,
specifically in an energy policy context.
Scenarios are particularly well suited for exploring complexities
and uncertainties (opening up phase; Stirling, 2005). However,
most applications of PMCA evaluate options in the form of measures, individual projects, technologies, or single actions. In the
few studies that do assess energy scenarios (see three studies in
the literature review in Section 2.2), the scenarios are combinations of instruments without integrating the different dimensions
of sustainability. The MCA of scenarios integrating qualitative
and quantitative data is not trivial in the sense that both the whole
picture, presented by a coherent storyline, as well as the different
parts, the dimensions of sustainable development, have to be addressed within the evaluation. The analysis entails a translation
of the qualitative scenario parameters, from the scenario storyline
into quantitative data and an assessment by use of both qualitative
and quantitative indicators. The step of applying a multi-criteria
algorithm reduces complexity by aggregating the multi-dimensional information given in the assessment matrix into a ranking
of scenarios.
In the case studies reported here, scenario analysis and PMCA
were combined for developing and appraising renewable energy
scenarios on multiple levels. For Austria as a whole and two local
communities in the southeast of Austria, a participatory process
with experts and stakeholders was undertaken to develop renewable energy scenarios for 2020 and to evaluate them along the sustainability dimensions. Particularly, several renewable energy
scenarios for Austria were appraised against sustainability criteria.
A similar process on the local level produced renewable energy
scenarios that were appraised against locally relevant sustainability criteria. The resulting ranking of the MCA helps to inform the
discussion and decision process on sustainable energy futures
ongoing in Austria.
This paper is organised as follows. Section 2 introduces the
methods of scenario-building and MCA, and discusses the combination of these methods. Section 3 provides a concise description
of the multi-level case study of Austrian renewable energy options
on the national and local level. Section 4 reports on the combined
application of scenario building and PMCA in the case study context and discusses the challenges arising from this methodology
and its benefits. Section 5 concludes.
2. Methods for strategic decision-making on sustainable energy
2.1. Scenario building
Over the last two decades, the application of scenarios as a
judgemental forecasting tool has gained much popularity for strategic analysis and longer-term planning (Bunn and Salo, 1993). In
scientific assessments, scenarios are usually based on an internally
consistent and reproducible set of assumptions or theories about

Social incommensurability arises from multiple legitimate perspectives of a

socio-ecological system. Technical incommensurability, in contrast, arises from the
different assessments of complex systems by use of different types of models.

the key relationships and driving forces of change, which are derived from the understanding of both history and the current situation. Often scenarios are formulated with the help of formal
numerical or analytical models. Scenarios are representations of
alternative futures that help to explore the space of possible futures. Each scenario is one alternative representation of how the
future might unfold. Using scenarios assists in the understanding
of possible future developments of complex systems. Hence scenarios, as a collection of futures, are intended to establish the
boundaries of our uncertainty and the limits to plausible futures
(Wilson, 1975).
The construction of scenarios for exploring alternative future
developments under a set of assumed conditions has a long tradition in strategic decision-making (WBCSD, 1998; Shell, 2005) and
especially in decision-making in an energy context (e.g., Ito et al.,
1997; Haldi, 2000; Nakicenovic, 2000; Oniszk-Poplawska et al.,
2003; IEA, 2001; BMUNR, 2004; Ghanadan and Koombey, 2005).
Environmental and energy scenarios have been developed for different temporal and spatial scales, from global and (inter-)national
long-term scenarios (IPCC, 2004; Nakicenovic et al., 1998a; Raskin
et al., 1998; Eames, 2002) to local mid-term scenarios (Georgopoulou et al., 1997). Among the non-energy scenarios, climate change
and land-use change scenarios are most prominent (IPCC, 2004;
Nakicenovic, 1998b, 2000). Scenario building has become one of
the main methods used for addressing the complexity and uncertainty inherent in long-term challenges, such as sustainable
The literature distinguishes different types of scenarios (EEA,
2004; IPCC, 2004; Berkhout et al., 2002), which include2: (1) extrapolatory approaches (forecasting), assuming that the future is essentially defined as a continuum of the past; (2) normative scenarios,
which are orientated towards certain milestones and where the actions are enumerated assuming that the future can be created; and
(3) exploratory scenarios. Exploratory scenarios differ from the former two insofar as they do not claim to predict the future but
endeavour to describe a possibility space. Implicit in forecasting is
a rather mechanistic view of systems, implying the general supposition that simplified quantitative models can portray the future. In
exploratory scenarios, the future is a social construction about which
legitimately diverse opinions exist. Typically, they include a narrative element a storyline and some quantitative indicators
(Berkhout et al., 2002). Further distinguishing factors among the
range of scenario-building techniques are, first, the degree of stakeholder or public participation in the scenario-building process and,
second, whether scenarios consist of a descriptive, narrative part
and/or a quantitative, modeled part.
In an ideal world, thorough data collection and good modelling
would enable us to understand systems well and to determine possible future system states. However, some physical and socio-economic systems are poorly understood, and uncertainty is high, so
that appraisal tools can only be used as heuristics and are best
communicated with images and stories. By linking qualitative narratives about the future and quantitative elements, scenarios facilitate our understanding of how systems work and evolve.
Therefore, scenarios are useful tools for scientific assessments,
for learning about complex systems, and for informing policy-makers (Jefferson, 1983; Davis, 1999).
Scenarios are artefacts of the complex reality of environmental
and socio-economic systems that bring to the fore the relevant aspects of potential futures. These artefacts are effective means for
developing, better understanding and communicating pathways
This is not an exclusive list of types of scenarios, other typologies exist. For
instance Bunn and Salo (1993), referring to Ducot and Lubben (1980), distinguish
between exploratory and anticipatory scenarios on the one hand, and descriptive vs.
normative scenarios as well as trend vs. peripheral scenarios on the other hand.


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

Table 1
Summary of the literature review on MCA applied to energy issues


Lootsma et al.

Long-term electricity
supply strategies

et al. (1997)
Hobbs and Horn
Mirasgedis and
Beccali et al.
et al. (1998)
Diakoulaki et al.
Goumas et al.

Regional energy
Demand-side planning

Electre III

Electricity generation

Displaced Ideal

Renewable energy
diffusion strategies
Renewable energy
sources promotion
Energy analysis and
policy making
Planning and assessment

Electre III, fuzzy sets

Promethee II

Afgan and
et al. (2002)

New and renewable

energy technologies
(power plants)
Policy instruments for
promoting wind energy

Beccali et al.
and Polatidis
Renn (2003)

Renewable energy
Renewable energy

Greening and
Kablan (2004)

Coordinated energy and

environmental policies

Nigim et al.
Cavallaro (2005)
Cavallaro and
Ciraolo (2005)
Gamboa and
Munda (2007)

Energy scenarios

Energy conservation
Pre-feasibility ranking of
alternative renewable
energy sources
Four different renewable
energy plants
Wind energy plants
Locating wind turbines

MCA method

Utadis (preference
disaggregation procedure)
Promethee II

Information deficiency method

Conceptual framework for

multi-criteria assessment of
energy policy instruments
Electre III
Promethee II

Value tree method, multiattribute-utility analysis (MAU)


Special features

Real world

National Energy Outlook

1987 for the Netherlands

3 baseload electricity supply strategies

(nuclear, coal, gas) combined with 3
economic growth scenarios
Focus on strategies, sensitivity analysis
Supplementary interviews, use of followup questionnaire
Comparative analysis of power
generation systems


Renewable energy options

of an island (Crete)
Natural gas demand in a
Canadian province
Ranking of systems based
on external environmental
Regional scale (Sardegna
Island, Italy)


Inclusion of fuzzy logic, comparative

study (Electre vs. fuzzy sets)
Sensitivity analysis done


13 EU member countries
and the US
Geothermal energy
projects (rural community
in Northern Greece)
10 different technologies

Focus on energy intensities of countries


Stochastic analysis


Information deficiency method


Policy design

3 stakeholder groups


Regional scale (Sardegna

Island, Italy)
Scenarios for geothermal
resource use in Greece
Waste energy utilisation



2 stakeholder groups (church

representatives, engineers)
Review paper




Prioritisation of policy instruments



Waterloo region, Southern

Ontario (Canada)

Local (community) level


Promethee I and II

Sensitivity analysis


Naiade, fuzzy sets methodology

Sicily Island (Messina

Salina Island (Italy)

Naiade, fuzzy sets methodology

Catalonia (Spain)

Analysis of coalition formation of


toward the scenarios. The result is a multi-dimensional representation of futures, which is demanding to compare and evaluate without further systematic and computational tools.
2.2. MCA
MCA is widely applied for decision aid in the energy management and energy policy contexts, including a few assessments of
the environmental performance of energy systems (e.g., Stewart
and Horowitz, 1991; Mirasgedis and Diakoulaki, 1997). Most applications on energy issues focus on technical planning (e.g., as described in Hobbs and Meier, 2000) and typically do not include
stakeholders in a systematic and participatory way. However, like
in other research areas, a trend towards increased involvement of
stakeholders can be observed in energy research. Examples are
Haralambopoulos and Polatidis (2003), Greening and Bernow
(2004), Madlener and Stagl (2005) and Stagl (2006). In the following, we provide a review of the literature on MCA applied to energy
issues. The articles covered are both theoretical/conceptual and
empirical. They include studies with and without public and stakeholder engagement. Table 1 summarises the key literature.


2.3. Combining methodologies: Participatory MCA and scenario

MCA is a form of integrated sustainability assessment (ISA). ISA
is an operational evaluation and decision support approach that is
suitable for addressing complex problems featuring high uncertainty, conflicting objectives, different forms of data and information, multiple interests and perspectives, and the accounting for
complex and evolving biophysical and socio-economic systems.3
ISA aims to support the development of (long-term) cross-sectoral
policies that specifically aim at sustainable development. We argue
here that the combination of scenarios and participatory MCA is a
potent ISA methodology.

The EU project MATISSE (6th Framework Programme,
developed the following definition: ISA is a cyclical, participatory process of scoping,
envisioning, experimenting, and learning through which a shared interpretation of
sustainability for a specific context is developed and applied in an integrated manner
in order to explore solutions to persistent problems of unsustainable development
(Weaver and Rotmans, 2006, p. 12).


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

To integrate multiple interests and perspectives and conflicting

objectives the ARTEMIS research team developed jointly with
stakeholders and external experts a concise set of scenarios. They
provide alternative representations of uncertain future states (narrative storylines, qualitative descriptions), as well as detailed modelling of different technology mixes, also taking into account
natural/technical potentials and political aims.
MCA complements scenario development by offering an algorithm to systematically aggregate multi-dimensional information
and, therefore, reducing the scenario (information) complexity in
a transparent way (closing down phase). The scenarios can be assessed according to the sustainability dimensions with different
forms of data and information. Conflicting objectives are transparently structured and accounted for. Decision-makers can be effectively supported. MCA, used in a participatory manner, provides a
method that fulfils the requirements of ISA at least theoretically
(Omann, 2004).

society strategies for a more sustainable energy system (supply

and demand); a contribution to current debates on different approaches for stakeholder participation regarding future energy options; and guidance and new insights for the structuring of
participatory decision-making processes in other problem settings.
Since this paper focuses on methodological questions when
combining scenario analysis and PMCA, the following discussion
of the case studies focuses on the scenario-building process and
the multi-criteria analysis. The following sections describe the case
study design and the scenario-development process with particular attention to methodological issues. More technical details,
including specific data, have been reported elsewhere (e.g., Madlener et al., 2007; Bohunovsky et al., 2007a; Bohunovsky et al.,
2007b; For a detailed account of the
preference formation and recording process and a discussion of
the role of weights in MCA see Omann (2006).
3.2. Introduction to the ARTEMIS case studies

3. Sustainable energy futures for Austria

3.1. Introduction of the ARTEMIS project
For the transition to sustainable energy systems, the substitution of modern renewable for conventional non-renewable (fossil,
nuclear) energy technologies plays a paramount role. In the renewable energy sector new technological developments, which are expected by many experts to lead to a shift towards a more
decentralised use of energy technologies, call for more public and
stakeholder engagement. The work presented here is part of the
ARTEMIS project, which focused on the role of renewable energy
technologies (RETs) as part of a sustainable energy system. In this
3-year project, funded by the Austrian Science Foundation, an
interdisciplinary research team of six researchers worked on two
parallel case studies in cooperation with energy stakeholders and
experts. Workshops and personal interviews were the main elements of the transdisciplinary research design. Invited stakeholders were identified in a stakeholder analysis based on two
dimensions: (1) high influence of the institution on a change of
the energy system and (2) strong effect on the institution from a
change in the energy system. In the national case study, participating stakeholders attended two workshops and they were interviewed. On the local level, stakeholders attended three
workshops and a final meeting where results were presented;
moreover, they had some self-organised meetings. The stakeholders were not compensated for their time and effort.
The ARTEMIS project aimed at enhancing and applying a participatory MCA tool for the integrated assessment of alternative
renewable energy scenarios for Austria as a whole, and also for
two local communities situated in the federal province of Styria.4
In essence, the analysis involved the following five main steps: (1)
developing a limited number of scenarios for sustainable (renewable) energy futures; (2) producing a detailed list of criteria for the
assessment of the social, economic, environmental and technical impacts of RETs; (3) assessing impacts with life-cycle-analyses; (4) eliciting individual stakeholders preferences and group preferences;
and (5) applying a multi-criteria aggregation method (here: PROMETHEE II) to obtain rankings of the scenarios considered.
The main outcomes of the ARTEMIS project are the development
of an impact matrix and an MCA procedure for the exploration of different energy scenarios by decision-makers at two spatial levels; a
contribution to the development of government, business and civil

Styria, located in the southeast of Austria, is one out of nine Austrian federal
provinces (Lnder).

The national case study involved representatives from different

interest groups and institutions as well as experts from the energy
field (for the list of participating stakeholders in the national case
study see Table C.1 in Appendix C). The study builds on existing energy scenarios for Austria (Kratena and Schleicher, 2001; Haas
et al., 2001; Haberl et al., 2002; Haberl et al., 2002). The project
team started with quality checks for comprehensiveness, consistency, and coherence of these scenarios (see also Bunn and Salo,
1993). For example, for the trend extrapolations until 2020 total
available potentials of renewable energy sources were checked.
The ARTEMIS team developed jointly with Austrian energy stakeholders five renewable energy scenarios (see Appendix A for short
scenario descriptions; for the respective additional contributions of
renewables relative to the base year 2002 see Fig. A.1). For the purpose of scenario development and discussion, and for the deliberation of criteria and weights, two workshops (held in Vienna in
May 2005 and November 2005, respectively) and a total of 25
interviews with Austrian stakeholders and experts were undertaken. New insights for the ongoing national discussion process
in Austria are offered by a systematic illustration of the scenario
impacts in all dimensions of sustainable development (Appendix
B, Table B.1), the description of the social preferences (for mean
criteria weighting of national stakeholders in Appendix D), and
the final ranking of energy scenarios based on this information.
The local case study was carried out for, and in collaboration
with, the two Styrian communities Raabau and Ldersdorf. Four
energy scenarios with the focus on different RETs as well as on energy efficiency were developed and evaluated according to their
expected impacts on sustainable development (Appendix A,
Fig. A.2, and Appendix B, Table B.2). An additional aim of the local
case study besides the general project aims was to deliver decision
support for the communities concerning the direction of their future use of RETs. This case was undertaken in cooperation with
the Local Energy Agency (LEA) of this region and served also as a
(science-guided) preparatory phase for participation in the socalled e5 Programme5 from 2006 onwards (Bohunovsky et al.,
2006, 2007b). The research team worked together with LEA, local
decision-makers, local experts, and the general public, in organising
participatory workshops, information events and in conducting
interviews (for the local stakeholder list see Appendix C, Table
C.2). The four workshops focused on scenario development; criteria
definition; social preferences and weights for the criteria (for group

e5 is a programme for qualification and labelling of communities that want to
contribute to a more sustainable development by making intensive use of renewable
energy sources and by increasing energy efficiency (

K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

criteria weighting of local stakeholders see Appendix D), and interpretation of the MCA results, sensitivity analyses, and the transition
phase to the e5 process. Compared to the national case study, the local case study was not only more adapted to local circumstances and
constraints, but was also more project-oriented.
3.3. Scenario building in ARTEMIS
The ARTEMIS scenarios were developed in a two stage process (first exploration, then forecasting). Initially, exploratory
scenarios were visualising possible paths within a space of natural potentials, political options, and societal interests. At this
stage the role of stakeholders was to discuss and inform about
the key alternative renewable energy pathways for Austria as a
whole and for the two Styrian communities. The scenario themes
plus their key RETs and the scenario parameters were agreed in
this phase. The scenarios were not only useful for investigating
key features of possible developments, but they also served as
a communication tool for discussing drivers of energy system
and conflicting objectives in future renewable energy resources
and their interaction, for addressing uncertainty and for stimulating consideration of alternatives to a business-as-usual baseline. During the second stage of scenario development, the
research team transformed the descriptive scenarios into forecasting scenarios consisting of a qualitative storyline and a modeled quantitative representation.
The role of the stakeholders was to critically reflect on the modelling assumptions and to discuss the quantitative interpretation of
the scenario themes. However, it was not possible to capture every
aspect of the storyline in the models. The differentiation between
small-scale (decentralised) and large-scale (centralised) energy
systems is a specific thematic scenario focus in both case studies,
which was integrated in the modelling. The social, economic, environmental and technical impacts of small-scale technologies were
accounted for in a differentiated way to large-scale technology impacts. An example for the transformation from the qualitative storyline into the modeled scenarios is described in Section 3.3.2.
Modelling allows answering questions such as What is the natural potential of certain renewable resources?, How much energy
is produced by which renewable source in 2020?, How many
houses are passive houses or have new insulations? etc.
3.3.1. National case study
The national scenarios focus on Austrian electricity and heat
production from renewable energy sources until 2020 (see Appendix A and esp. Fig. A.1; for a more detailed description of the national scenarios see Kowalski et al., 2006; Madlener et al., 2007).
The scenarios were designed to address specific questions, represented by the scenario parameters: the technological mix and
capacity in 2020, decentralised versus centralised energy systems,
necessary institutional changes, and the time frame for the investment decisions to be taken. External influencing factors such as the
oil price and EU policy were not addressed explicitly, since concrete political strategies of implementation were not the focus of
the study. In ARTEMIS, scenarios are descriptions of a possible situation of the renewable energy systems in 2020 with details of
technological capacity. Within this time scale it is assumed that
there is some leeway for changes in the feasibility and market situation of RETs, but not for the development of new technologies. In
contrast to the local case study the modelling of the national scenarios was a top-down process based on natural and technical
potentials of RETs.
In Workshop I, the stakeholders discussed a range of 16 exploratory scenarios based on a systematic combination of scenario
parameters suggested by the research team. The stakeholders
adapted the scenario parameters and selected the most relevant


scenarios for detailed elaboration and modelling towards forecasting scenarios. For example, the differentiation between heat and
electricity scenarios, as originally proposed by the research team,
was replaced by combined heat and electricity scenarios. The process of quantitative modelling was based on the identification of
key technologies through stakeholders and the use of existing
(moderate and ambitious) energy demand growth rates from the
literature (Haas et al., 2001; Kratena and Schleicher, 2001). Further
assumptions were drawn on how the narrative parameters configure the technology mix, e.g., decentralised versus centralised energy systems. In a second feedback loop, these assumptions were
confirmed and adapted with stakeholders participating in Workshop II. The final national scenarios were taken up by an on going
Austrian research project on participatory modelling of energy scenarios (Renewable Energy in Austria: Modelling Possible Development Scenarios until 2020, see
3.3.2. Local case study
The local scenarios describe possible developments of electricity and heat supply and expected changes in energy demand in
Ldersdorf and Raabau until 2020 (for a detailed description of
the local scenarios, see Bohunovsky et al., 2006, 2007b). They represent four possible futures of the energy system of the area. The
spectrum of possible scenarios ranges from a concentration on
electricity production by biogas plants and the revitalisation of
an already existing hydropower plant, to the supply of heat by
community-owned wood chips district heating plants and decentralised and privately-owned small-scale biomass heating systems.
The scenarios also include a range of energy efficiency measures
(e.g., improving thermal insulation of building shells, constructing
passive solar houses, introducing other energy-saving measures)
and thus implicitly consider reducing future energy demand.
The design elements and themes of the four local energy scenarios were first identified by the research team, based on existing
data, literature and policy reviews, then discussed with local energy experts, decision makers and the public, and readjusted during the exploratory scenario stage.
The main elements in the storylines were: (1) focus on heat or
electricity from renewables (all scenarios include some supply for
heat and power from renewables); (2) resulting RETs; (3) degree of
(de-)centralisation of energy supply; and (4) role of energy-saving
measures by the municipality and households.
The quantitative modelling of the scenarios (i.e., how much energy is used from what sources and what kind of plants, etc.) is
based on the 2001 building and housing census and the communal
energy balance (Energie Gesamt Rechnung; Zelle and Schechtner,
2004). The stock of houses/flats, their heating systems and the
existing electricity mix of the Styrian energy utility company STEWAG was the starting point for the modelling in the local case
study. From that the number of new houses for 2020, the number
of passive houses and the number of better insulated houses were
extrapolated. Based on the scenario storylines the key RETs and
their scales were identified (GWh, PJ) for the existing houses and
the new ones in 2020. The energy demand in 2020 (in two scenarios it is reduced compared to a business-as-usual situation, due to
stronger efforts for energy efficiency measures) was estimated for
that reason. From this basis, it was possible with the help of several
data sets (GEMIS/ko-Institut, 2005; Neubarth and Kaltschmitt,
2000) and qualitative assessments to capture the impacts of the
scenarios on the evaluation criteria. In Appendix A an overview
of the RETs in the scenarios is given. More information on the impacts can be found at the project webpage ( or can be requested from the authors.
The resulting scenarios are described by a narrative storyline,
the total energy demand, the share of renewable energy in total energy demand, the description of the RETs used for heat production


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

as well as used for electricity production (number of plants per

technology and amount of electrical energy and heat produced).
3.4. Multi-criteria methodology used in ARTEMIS
Multi-criteria methods were found to be useful for their ability
to address problems that are characterised by conflicting assessments, and to deal with incommensurable values. Roy and Brans
et al. (1986) developed MCA algorithms that are based on weaker
assumptions than previously known utility-based multi-criteria
algorithms (e.g., no additive utility function necessary) and that require less information from decision-makers (preference intensities, substitution rates). Examples of this type of MCA include
ELECTRE I, IS, II, III, IV, TRI (where ELECTRE stands for Elimination
et Choix Traduisant la Ralit) and PROMETHEE I, II (Preference
Ranking Organization Method for Enrichment Evaluations).
The aim of MCA is not so much to identify an optimal solution
but rather to facilitate the identification of compromise solutions
in a transparent and fair way.
In the ARTEMIS case studies, PROMETHEE was used to evaluate local and national renewable energy scenarios. The set of
evaluation criteria, which were derived from systems theory
and from the Helmholtz approach (integrative sustainability concept; cf. Coenen and Grunwald, 2003; Kopfmller et al., 2001),
were discussed with the stakeholders (the impact matrices can
be found in Appendix B, Tables B.1 and B.2). The impacts were
determined by drawing on existing databases, such as GEMIS
(ko-Institut, 2005), as well as own modelling and expert assessments. On the local and the national level the GEMIS database
was used for environmental impacts, and expert and citizen interviews (citizens were involved only on the local level) for appraising the social, economic, and technical impacts. The weights on
the criteria were derived by using the SIMOS method, which calculates weights for criteria from a ranking of criteria according to
their importance by the stakeholders (Figueira and Roy, 2002). In
a group discussion process called silent negotiation (Pictet and
Bollinger, 2005) the local stakeholders were able to agree on a
joint (or group) ranking of criteria (see Appendix D). Additionally,
individual rankings were performed to derive more legitimate
data (the preferences of the individual participants coincided
strongly with the group preferences; for 2 out of 11 participants
the individual weights were clearly different from the group
weights). In contrast, on the national level only individual rankings of sustainability criteria were collected in personal interviews, due to the fact that the availability of stakeholders for
workshops was more limited (see Appendix D, Tables D.1 and
D.2, for mean stakeholder weighting of the sustainability criteria).
On the national level 16 MCA rankings of the scenarios were
calculated based on the specific sets of weights, derived from
the stakeholder interviews. These rankings were not aggregated

to a group weight, but instead treated separately in a comparative way.

As can be seen from Fig. 1a, essentially there are three clusters
of scenarios, which are also robust in the sensitivity analyses. Scenario E Large impact in small-scale use and C Investment into the
future are ranked first, performing best with respect to the sustainability criteria accounted for. Scenario B Extension of competitive advantage is in a middle position. Scenario A Fast and known
and scenario D Extensive use of biomass, two scenarios with
rather centralised production systems, were ranked last. The bad
performance in the environmental indicators is the main reason
for the low rank of scenario D and in the social indicators for scenario A. The increase of the renewable energy capacity within the
total heat and electricity supply by 2020 varies among the scenarios between plus 50% and 102%, and is highest for scenario D
Extensive use of biomass. The result of the assessment is in contrast to the current public discourse on renewable energy in Austria, which is focused on an increased use of biomass in future
energy systems.
The result of the MCA ranking on the local level is very stable
and robust especially concerning the top rank, which is held by
scenario 4 (see Fig. 1b), focusing on the reduction of energy demand and the support of small-scale, privately-owned renewable
energy conversion plants. Sensitivity analyses (changing different
weights, using all individual weighting schemes, changing the
preference functions thresholds) do not displace scenario 4 from
the first rank. Scenarios 1 and 3 (both including large power plants)
are usually ranked last because their environmental and social performances are rather poor. However, if costs or regional economic
development get extremely high weights (more than 25%), then
they rank first or second.
The share of renewables for total heat and power supply varies
between 64% and 81% and is highest in scenario 3.
4. Discussion
This section discusses the contributions and challenges of the
combined use of scenario building and participatory MCA encountered in the present study.
4.1. Contributions of the research project
Motivated by the new challenges for strategic decision-making (multi-dimensionality of the sustainability goal, complexity
of socio-economic and biophysical systems, and long-term nature of problems), the ARTEMIS project demonstrated that the
combined methodology of scenario development and participatory MCA responds successfully to these challenges. This section
discusses the representation of socio-economic and biophysical

Fig. 1. Scenario ranking at the national level (based on the most common individual rankings derived from individual weights) and the local level (based on the ranking
derived from group weights).

K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

systems in the assessment, process rationale of the decision, and

4.1.1. Representation of socio-economic and biophysical systems
The ARTEMIS project combined scenario development and participatory MCA such that the complexities of the relevant socioeconomic and biophysical systems and uncertainties of long-term
impacts could be explored. The information was structured according to key sustainability dimensions; the multi-dimensional data
were then aggregated in order to aid decision-making.
While sustainable development is generally characterised by
three dimensions environmental, economic and social the social dimension of sustainability is more often than not ignored because of a lack of data. In the ARTEMIS project, we included social
sustainability on equal footing in the analysis and collected relevant data through expert interviews.
The rich representation of possible futures with a limited number of scenarios, and the transparent presentation of the data in the
multi-criteria framework, enabled stakeholders to explore the options in a comprehensive but at the same time structured and focused way. This process of opening up (exploration of scenarios,
discussion of criteria, review of impact data) was followed by a
process of closing down (calculation of the rankings, reviewing,
and learning), which aids decision-making.
4.1.2. Decision process
The participatory decision-making process started with a
comprehensive stakeholder analysis, aimed at representing the
main societal interests in the workshops. The design of the
workshops, which were led by professional facilitators, ensured
that all participating stakeholders had opportunities to speak
and that minority views were also heard. In order to cover all
dimensions of the theoretically informed criteria frame, the
assessment drew on a range of different types of knowledge
expert and layperson knowledge; quantitative and qualitative
data. Once the rankings for different participants (individual
weights) and for the group as a whole (group weights) were calculated, the participants were given not only the opportunity to
review their assessments in comparison to those of other stakeholders, but they were also encouraged to explore different outcomes with different weightings, data etc. In this respect we
agree with the European Working Group on Multicriteria Decision Aiding ( that this learning process is essential for robust decision-making. The learning of
participants was monitored in repeated surveys for both case
studies. We observed different types of learning cognitive
learning, learning from others and learning about the decisionprocess methods (for more details see Garmendia and Stagl,
4.1.3. Outcomes
The ARTEMIS project delivered rankings of alternative future
energy scenarios on the national and local level for Austria. Scenario development, assessment and exploration of results were
conducted in a participatory way, leading to more legitimate
and robust results than merely technical analyses. In order to
facilitate the implementation of the results in the policy arena,
we made the ease of implementation (e.g., need for forming
and establishing new institutions) an explicit dimension in
building the scenarios. However, we need to highlight that the
outcomes of the project are possible starting points for decisions
in the policy arena but not decisions per se. The final part of the
workshops was also dedicated to identifying options for transitions to a more sustainable energy future. On the local level,
the project results had immediate impact. The communities
decided to become a member of the E5 programme and started


work on a plan for their energy future based on the recommendation resulting from the ARTEMIS case study. On the national
level, the ARTEMIS case study contributed to the ongoing energy
discourse mainly by generating an assessment of energy scenarios that is based on life cycle analysis.
4.2. Challenges
One of the main challenges faced by researchers dealing with
scenario analysis is to convey the complexities behind the scenarios first into narrative scenario descriptions of adequate length and
style and, second, into a combined quantitative and qualitative
assessment tool. Moreover, it requires an organised and systematic
development procedure that achieves credible scenarios in terms
of comprehensiveness, consistency and coherence (Bunn and Salo,
Another challenge is the stakeholder participation. Renn (2003),
for example, reports that in their study basic assessment criteria
for energy systems were developed in eight meetings, but that
not all stakeholder groups were able to participate in the entire
assessment and weighting procedure. Ultimately, only two stakeholder groups representatives of engineers and power plant
operators and of the churches participated in the entire participatory process (i.e., establishment of criteria, assessment and evaluation by means of criteria, weighting and sensitivity analysis),
thus invalidating the results as the representative outcome of a
process involving all relevant stakeholders. In our case studies,
we also struggled with non-participation and dropouts of stakeholders; in most cases it was not clear at all why participation fluctuated as it did (see also Kowalski et al., 2006; Bohunovsky et al.,
Finally, the choice of MCA algorithm that fits the research design of the project is challenging. Applying an MCA method, such
as AHP or MACBETH, is advantageous when opinions, references,
etc., which are difficult to quantify, form the basis of the data
that decision-makers have to judge. By contrast, the subjective
scaling of concrete quantitative numbers needed for pairwise
comparison of elements in the same hierarchy may lead to
losses in accuracy with such a method (Nigim et al., 2004). In
scenario assessments, one usually deals with a mix of indicators,
some of which are more prone to subjective assessment than
others. Other methods rely on expert opinion in some intangible
aspects, which implies just another subjective judgement of certain criteria. Overall, these pros and cons call for a fine-tuned
mix of analytical tools and participatory methods, which has
been sought in ARTEMIS.

5. Conclusions
MCA is a tool that was found to be useful for aiding decisionmaking in the context of sustainable development. It helps to
overcome some of the problems of monetary valuation, accounts
for the multiple dimensions and long-term nature of sustainable
development, and supports transparent and robust decisionmaking processes. If framed as a participatory, social learning
exercise, MCA can also address the dynamic aspect of decisionmaking about evolving complex systems. A collective understanding of the need to combine analytical and participatory
methods was developed over the last few years in decision analysis, ecological economics, sustainability science, and science &
technology studies. This paper argues that the combination of
MCA and scenario analysis is a promising, but in terms of design and implementation a challenging methodological option.
The participatory process is essential for addressing problems,
such as future energy systems, arising from the complexity of


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

socio-economic and biophysical systems featuring high uncertainty, conflicting objectives, different forms of data and information, multiple interests and perspectives. However, the
conditions under which public and stakeholder engagement is
most effective, and quality criteria for the implementation of
participatory exercises, are still under development (Blackstock
et al., 2007). Within the current institutional framework of representative democracies there is also considerable tension between public and stakeholder participation and legislatively
delegated authority. By introducing public participation into
the procedures of administrative agencies, which are themselves
responsible to democratically elected officials, public lines of
decision-making are sometimes crossed. While the largest potential of MCA for decision-making on sustainable development lies
in implementing MCA in combination with participatory techniques, this is also where key methodological and institutional
questions still need to be answered.
In the context of scenarios it is generally easier for stakeholders to deal and work with stories than with purely quantitative
information. On the one hand, the scenarios enable stakeholders
to be visionary and to build possible6 futures containing their
ideas, experiences, and perspectives. On the other hand, scenarios
can be an important communications tool with other stakeholders
and with the research team. For the appraisal, however, it is necessary to structure and quantify the narrative scenario information. This step entails defining the key parameters of the
scenarios (e.g., total energy demand in GWh, reduction in CO2
emissions in tons, share of renewables in %) and developing an
assessment matrix. The assessment matrix shows the possible impacts in a transparent manner and is open to scrutiny from all
parties involved in the appraisal process. Exploring the assessment
matrix also allows building up preferences about criteria. In the final stage of the appraisal process, the information from the
assessment matrix and criteria weights is used to calculate a ranking of the scenarios. In the methodology presented in this paper,
stakeholders are given the opportunity to explore the role of the
assessment matrix and weights further by making reasoned
changes, if they wish.
Assessing scenarios instead of single technology options in the
form of renewable energy pathways with an MCA, and applying
a participatory MCA instead of simpler assessment tools or just
descriptive assessments, is more resource-intensive. However, we
argue in this paper that the results capture the context of technology deployment better and build on a more robust and democratically based decision process, which addresses uncertainties,
acknowledges multiple legitimate perspectives and encourages social learning.

The authors would like to acknowledge financial support received from the Austrian Science Foundation for the ARTEMIS project (FWF Grant No. P16734-G04; June 2003 - May 2006; Moreover, we thank Lisa Bohunovsky,
Martin Bruckner and Eneko Garmendia for their contributions to
the ARTEMIS project and comments on the paper, as well as the
numerous experts and stakeholders who gave valuable input to
the project in workshops and interviews. Finally, helpful comments received from several participants of the 19th Mini EURO
Conference on Operational Research Models and Methods in the

Note: scenario analysis encourages stakeholders to explore possible futures, not
only futures that are currently perceived as probable.

Energy Sector (ORMMES06), Coimbra, Portugal, 68 September

2006, are gratefully acknowledged.

Appendix A. Scenario descriptions

A.1. Short description of the five national ARTEMIS energy scenarios
Scenario A Fast and known. This scenario focuses primarily on
technologies that are in intensive use already today and promise
rapid capacity expansion. Biomass and wind power plant capacities are increased markedly, as is the contribution of sewage gas
plants. The use of biomass resources is done in such a way that
the domestic potentials of biomass residue are rather sufficient.
Key technologies: solar thermal, biomass (heat and CHP only), wind
power, sewage gas, as well as low energy buildings (new
Scenario B Extension of competitive advantage. The main strategy in this scenario is to push those technologies most with which
positive experience regarding technology exporting have been
gathered in the past. Biomass district heating, biogas and solar
thermal plants are particularly favoured. An important goal is to
strengthen technological leadership. Key technologies: individual
biomass systems, community biomass CHP systems, solar thermal,
geothermal, small hydropower, wind power, and passive house
Scenario C Investments into the future. Scenario C aims at a
long-term investment strategy. The focus is put on the decentralised generation of electricity and the promotion of capital-intensive but at the same time promising technologies (e.g.,
photovoltaics, PV). The growing electricity demand can only be
met partially in this scenario by the slowly increasing contribution of renewables, but trend-setting structural investments
into the building stock are being made (e.g., equipping public
buildings with PV systems). Key technologies: PV (primarily on
rooftops and facades), multi-functional energy centres, biogas
feed-in, geothermal energy, energy-efficient energy supply systems, passive buildings (new constructions), and renovation of
the old building stock.
Scenario D Extensive use of biomass. In Scenario D the main
focus is on renewable heat generation from biomass resources;
those technologies are pushed most with which good experience
has been made in the past and where sufficient domestic resources are available. Central biomass and biogas plants (with
or without CHP) are being built. Additionally, larger wind farms
are promoted. Key technologies: biomass combustion and gasification (esp. CHP), where the biomass used also stems from dedicated energy plantations and from imports, biogas (esp. CHP),
and solar thermal.
Scenario E Large impact in small-scale use. Scenario E implies a
concentration mainly on technologies for local energy supply from
renewables (individual and collective systems). Heat pumps and
heating and CHP plants for the use of solid and gaseous biomass
are promoted most, as are solar thermal and PV systems. Key technologies: biomass (individual and communal CHP plants with district heating), biogas (single systems, CHP), heat pumps, wind
power, solar thermal, PV systems, and passive house components
(see Fig. A.1).

A.2. Short description of the four local ARTEMIS energy scenarios

Scenario 1 Power production based on RETs. The communities
support mainly the power production from RETs, as a reply to
the increasing electricity demand. Mainly centralised plants
(biogas, PV systems, and hydropower) are being installed.


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074






















Geothermal, Electr.

Biogas CHP, total

Heat pump

Biomass CHP, total

Sewage gas CHP

PV (only buildings + facade)

Geothermal, heat production

Biogas CHP (heat sales)

Biomass electr., total

Wind energy, total

Sewage gas CHP

Solar thermal, total

Small hydropower (<10 MW)

Biomass combustion, heat prod.

Biomass CHP, total

(a) Heat energy

(a) Electricity

Fig. A.1. Summary plots of national-level energy scenarios AE considered in the participatory process (additional contribution of renewables relative to the base year 2002).













Small Hydropower



(a) Electricity

Solar thermal
District heating (Biomass based)
Heat pump

Wood logs
Wood chips

(a) Heat energy

Fig. A.2. Summary plots of local-level energy scenarios 14 considered in the participatory process (additional contribution of renewables relative to the base year 2002).

Scenario 2 Renewable energy from small, private plants. In this

scenario the focus is put on supporting small, private systems (solar thermal, pellet heating, wood chips, small scale photovoltaics).
In the majority of the new houses heating systems based on RETs
are being installed.
Scenario 3 Reduction of energy demand and renewable energy from
large plants. In this scenario a number of measures are undertaken

to reduce energy demand. The communities support initiatives of

cooperatives and groups to install heating systems based on RETs.
Scenario 4 Reduction of energy demand and heat from small
plants. Energy is the main issue within the development plans of
the communities. Energy-saving and efficiency-increasing measures are strongly supported and small, private plants using RETs
for heat are installed (see Fig. A.2).


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

Appendix B. Impact matrices

Table B.1
Impact matrix of the national ARTEMIS case study

Climate change properties

CO2 equivalents
Air quality
SO2 equivalents
Particulate matter
Rational use of resources
Cumulated energy input
Cumulated material input
Water quality
Constant & variable costs
Regional self-determinacy
Social cohesion
Diversity of technologies
Effect on public spending
Import independency
Quality of landscape
Social justice
Technological advantage
Ecological justice
Security of supply


Scenario A

Scenario B

Scenario C

Scenario D

Scenario E

























E3 /TJ

Rather low
Rather low
Rather low
Rather low

Rather low
Rather low
Rather low

Rather high
Rather high
Rather high

Rather low

Rather high
Rather high

Table B.2
Impact matrix of the local ARTEMIS case study

Climate change properties

CO2 equivalents
Air quality
SO2 equivalents
Stratospheric ozone
Rational use of resources
Cumulated energy input
Cumulated material input
Sealed land
Influence on habitats
Influence of water habitats
Influence of soil habitats
Social justice
Regional social cohesion
Regional economic development
Import independency
Quality of landscape
Security of supply


Scenario 1

Scenario 1

Scenario 3

Scenario 4






















K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

Table D.1 (continued)

Appendix C. List of stakeholders

Table C.1
List of stakeholders participating in the national case study (W, workshop; I,
Governmental bodies

Form of

Federal Ministry of Agriculture, Forestry, Environment and Water

Federal Ministry of Transport, Innovation and Technology
Austrian Chamber of Agriculture
Austrian Chamber of Labour
Austrian Association for Consumer Information
Federation for Austrian Industry

W, I
W, I
W, I
W, I

Private firms
Austrian Business Council for Sustainable Development
Austrian Forest Corporation


Power distributors
Oekostrom AG (green electricity distributor)
Austrian Power Distribution GmbH
Wien Energie GmbH

W, I

Eurosolar Austria
Austrian Wind Power Association
Austrian Small Hydro Power Association
WWF Austria
Global 2000 Austria
Working Group for Renewable Energy, Vienna, Lower Austria


Research Institutes
Centre for Social Innovation

W, I


Technological advantage
Regional self-determinacy
Social cohesion
Ecological justice
Quality of landscape

Mean weight

Standard deviation



Table D.2
Group weighting of sustainability criteria in the local case study

Regional economic development

Climate change impact
Air quality
Optimal use of resources
Security of supply
Diversity of technologies and resources
Import independency
Quality of landscape
Regional social cohesion
Influence on the habitat
Social justice

Group rank

Group weights




Table C.2
List of stakeholder participating in the local case study
Local energy experts
Operator of a wood chip facility
Electrician responsible for street lights
Architect (specialised on wood construction and energy efficiency)
Retailer of heating systems
Student (environmental systems science programme)
Owner of an existing small water power plant
Regional and national energy experts
Biogas expert
Landesenergieverein, expert on grants and subsidies
Solar thermal heat and photovoltaics expert
e5 energy team
Mayors and deputy mayors of two municipalities
3 citizens
8 citizens

Appendix D. Criteria weights

Table D.1
Mean weights of sustainability criteria in the national case study (the mean of 16
individual stakeholder weightings)

Climate change properties

Security of supply
Rational use of resources
Social justice
Water quality
Import independency
Air quality
Diversity of technologies
Effect on public spending

Mean weight

Standard deviation



Afgan, N.H., Carvalho, M.G., 2002. Multi-criteria assessment of new and renewable
energy power plants. Energy 27, 739755.
Beccali, M., Cellura, M., Ardente, D., 1998. Decision making in energy planning: The
ELECTRE multicriteria decision analysis approach compared to a fuzzy-sets
methodology. Energy Conversion Management 39 (1618), 18691881.
Beccali, M., Cellura, M., Mistretta, M., 2003. Decision-making in energy planning.
Application of the Electre method at regional level for the diffusion of
renewable energy technology. Renewable Energy 28, 20632087.
Berkhout, F., Hertin, J., et al., 2002. Socio-economic futures in climate change impact
assessment: Using scenarios as learning machines. Global Environmental
Change 12 (2), 8395.
Blackstock et al., 2007. Developing and applying a framework to evaluate
participatory research for sustainability. Ecological Economics 60 (4), 726742.
BMUNR, 2004. kologisch optimierter Ausbau der Nutzung erneuerbarer Energien
in Deutschland, Forschungsvorhaben im Auftrag des Bundesministeriums fr
Umwelt, Naturschutz und Reaktorsicherheit, FKZ 901 41 803, Stuttgart,
Heidelberg, Wuppertal.
Bohunovsky, L., Bruckner, M., Omann, I., Madlener, R., Stagl, S., 2006. Integrierte
Nachhaltigkeitsbewertung von Energieszenarien, Eine lokale Fallstudie im
Projekt ARTEMIS. Paper presented at the 9th Symposium Energieinnovation,
1517 February 2006, Graz, Austria.
Bohunovsky, L., Madlener, R., Omann, I., Bruckner, M., Stagl, S., 2007a. In:
kologisch Wirtschaften. Lokale Energiesysteme der Zukunft. Integrierte
Nachhaltigkeitsbewertung von lokalen Energieszenarien, 2/2007, pp. 4750.
Bohunovsky, L., Bruckner, M., Omann, I., 2007b. Partizipative Entwicklung von
Schwerpunkten und Handlungsfeldern im Einsatz von Technologien zur
Nutzung von Erneuerbaren Energien der e5-Gemeinde Raabau-Ldersdorf,
2007. SERI Studies 6, Available from: <>.
Brans, J.-P., Mareschal, B., Vincke, P., 1986. How to select and how to rank projects
The PROMETHEE method. European Journal of Operational Research 24 (2),
Bunn, D.W., Salo, A.A., 1993. Forecasting with scenarios. European Journal of
Operational Research 68, 291303.
Cavallaro, F., 2005. An integrated multi-criteria system to assess sustainable energy
options: An application of the PROMETHEE method. Fondazione Eni Enrico
Mattei, Nota di Lavaro, University of Molise, Campobasso.
Cavallaro, F., Ciraolo, L., 2005. A multicriteria approach to evaluate wind energy
plants on an Italian island. Energy Policy 33, 235244.
Coenen, R., Grunwald, W., 2003. Nachhaltigkeitsprobleme in Deutschland. Analyse
und Lsungsstrategien. Sigma, Berlin.
Davis, G.R., 1999. Foreseeing a refracted future. Scenario & Strategy Planning 1 (1),
Diakoulaki, D. et al., 1999. The use of a preference disaggregation method in energy
analysis and policy making. Energy The International Journal 24, 157166.
Ducot, C., Lubben, G.J., 1980. A typology for scenarios. Futures 12 (1), 5157.
Eames, M., 2002. The development and use of the UK environmental future
scenarios: Perspectives from cultural theory. Greener Management
International 37, 5370.


K. Kowalski et al. / European Journal of Operational Research 197 (2009) 10631074

EEA, 2004. Cloudy Crystal Balls An assessment of recent European and global
scenario studies and models. Experts corner report, Prospects and scenarios No.
Enzensberger, N. et al., 2002. Policy instruments fostering wind energy projects A
multi-perspective evaluation approach. Energy Policy 30, 793801.
Figueira, J., Roy, B., 2002. Determining the weights of criteria in the ELECTRE type
methods with a revised Simos procedure. European Journal of Operational
Research 139, 317326.
Gamboa, G., Munda, G., 2007. The problem of windfarm location: A social multicriteria evaluation framework. Energy Policy 35, 15641583.
Garmendia, E., Stagl, S., 2006. Social learning in public participation for
sustainability: Measuring the quality of participatory approaches.
Participatory Approaches in Science & Technology (PATH) Conference, 47
June 2006, Edinburgh, Scotland.
Georgopoulou, E., Lalas, D., Papagiannakis, L., 1997. A Multicriteria Decision Aid
approach for energy planning problems: The case of renewable energy option.
European Journal of Operational Research 103, 3854.
Georgopoulou, E., Sarafidis, Y., Diakoulaki, D., 1998. Design and implementation of a
group DSS for sustaining renewable energies exploitation. European Journal of
Operational Research 109, 483500.
Ghanadan, R., Koombey, J.B., 2005. Using energy scenarios to explore alternative
energy pathways in California. Energy Policy 33, 11171142.
Goumas, M.G., Lygerou, V.A., Papayannakis, L.E., 1999. Computational methods for
planning and evaluation geothermal energy projects. Energy Policy 27, 147
Greening, L.A., Bernow, S., 2004. Design of coordinated energy environmental
policies: use of multi-criteria decision-making. Energy Policy 32, 721735.
Haas, R., Berger, M., Kranzl, L., 2001. Strategien zur weiteren Forcierung
erneuerbarer Energietrger in sterreich unter besonderer Bercksichtigung
des EU-Weissbuches fr erneuerbare Energien und der Campaign for Take-off,
Studie der TU Wien im Auftrag des BMWA und BMLFUW, Juli 2001.
Haberl, H., Krausmann, F., Erb, K., Schulz, N., Adensam, H., 2002. Biomasseeinsatz
und Landnutzung sterreich 19952020, Social Ecology Working Paper 65, im
Auftrag des BMBWK.
Haldi, P.A., 2000. Multicriteria/Multi-stakeholder comparative assessment of
electricity generation scenarios in the sustainability context: A Swiss Case
Study. Massachusetts Institute of Technology. University of Tokyo, Swiss
Federal Institute of Technology, Lausanne, Alliance for Global Sustainability,
pp. 126.
Haralambopoulos, D.A., Polatidis, H., 2003. Renewable energy projects: structuring
a multi-criteria group decision-making framework. Renewable Energy 28, 961
Hobbs, B.F., Horn, G.T.F., 1997. Building public confidence in energy planning: A
multimethod MCDM approach to demand-side planning at BC gas. Energy
Policy 25 (3), 357375.
Hobbs, B.F., Meier, P., 2000. Energy Decisions and the Environment. A Guide to the
Use of Multicriteria Methods. Kluwer Academic Publishers, Boston.
IEA (International Energy Agency), 2001. Energy to 2050, Scenarios for a Sustainable
Future, funded by IEA/OECD, Paris.
IPCC (Intergovernmental Panel on Climate Change), 2004. In: Nakicenovic, N.,
Swart, R. (Eds.). Special Report on Emission Scenarios, funded by World
Meteorological Organization (WMO) and the United Nations Environment
Programme (UNEP).
Ito, K., Uchiyama, Y., et al., 1997. Study on GHG Control Scenarios by Life Cycle
Analysis World Energy Outlook until 2100. Energy Conversion and
Management 38, 607614.
Jefferson, M., 1983. Economic uncertainty and business decision-making. In:
Wiseman, J. (Ed.), Beyond Positive Economics? Macmillan Press, London, pp.
Kablan, M.M., 2004. Decision support for energy conservation promotion: An
analytic hierarchy process approach. Energy Policy 32, 11511158.
Kopfmller, J., Brandl, V., Jrissen, J., Paetau, M., Banse, G., Coenen, R., et al., 2001.
Nachhaltige Entwicklung integrativ betrachtet. Sigma, Konstitutive Elemente,
Regeln, Indikatoren, Berlin.
Kowalski, K., Madlener, R., Stagl, S., 2006. Neue Wege der integrierten Bewertung
von nationalen Energieszenarien fr sterreich, Paper presented at the 9th
Symposium Energieinnovation, 1517 February 2006, Graz, Austria.
Kratena, K., Schleicher, S., 2001. Energieszenarien bis 2020, Studie des
sterreichischen Instituts fr Wirtschaftsforschung (WIFO) im Auftrag des
BMWA und BMLFUW, sterreichisches Institut fr Wirtschaftsforschung, Wien,
April 2001.

Lootsma, F.A., Boonekamp, P.G.M., Cooke, R.M., van Oostvoorn, F., 1990. Choice of a
long-term strategy for the national electricity supply via scenario analysis and
multi-criteria analysis. European Journal of Operational Research 48,
Madlener, R., Stagl, S., 2005. Sustainability-guided promotion of renewable
electricity generation. Ecological Economics 53, 147167.
Madlener, R., Kowalski, K., Stagl, S., 2007. New ways for an integrated valuation of
national energy scenarios: The case of renewable energy use in Austria. Energy
Policy 35(12), 60606074.
Mirasgedis, S., Diakoulaki, D., 1997. Multicriteria analysis vs. externalities
assessment for the comparative evaluation of electricity generation systems.
European Journal of Operational Research 102, 364379.
Munda, G., 2004. Social multi-criteria evaluation: methodological foundations and
operational consequences. European Journal of Operational Research 158 (3),
Nakicenovic, N., 1998b. Emissions scenarios database and review of scenarios.
Mitigation and Adaptation Strategies for Global Change 5, 95120.
Nakicenovic, N., 2000. Greenhouse gas emissions scenarios. Technological
Forecasting and Social Change 65 (2), 149166.
Nakicenovic, N., Grbler, A., McDonald, A. (Eds.), 1998a. Global Energy Perspectives.
Cambridge University Press, Cambridge.
Neubarth, J., Kaltschmitt, M. (Eds.), 2000. Erneuerbare Energien in sterreich.
Systemtechnik. Potenziale, Wirtschaftlichkeit, Umweltaspekte. Springer, New
Nigim, K., Munier, N., Green, J., 2004. Pre-feasibility MCDM tools to aid communities
in prioritizing local viable renewable energy sources. Renewable Energy 29,
Omann, I., 2004. Multi-criteria decision aid as an approach for sustainable
development analysis and implementation, Ph.D. Thesis, University of Graz.
Omann, I., Kowalski, K., Madlener, R., Stagl, S., 2006. The influence of social
preferences on the multi-criteria evaluation of energy scenarios. PATH
conference paper, Edinburgh, June 2006.
Oniszk-Poplawska, A., Rogulska, M., et al., 2003. Renewable-energy developments
in Poland to 2020. Applied Energy 76 (13), 101110.
ko-Institut, 2005. Global Emission Model for Integrated Systems (GEMIS) Version
4.3, Institut fr angewandte kologie e.V., Darmstadt/Freiburg i.Br. Available
from: <>.
Pictet, J., Bollinger, D., 2005. The silent negotiation or how to elicit collective
information for group MCDA without excessive discussion. Journal of
Multicriteria Decision Analysis 13, 199211.
Raskin, P., Gallopin, G., Gutman, P., Hammond, A., Swart, R., 1998. Bending the
curve: Toward global sustainability. A report of the Global Scenario Group,
PoleStar Series Report 8. Stockholm Environment Institute, Stockholm.
Renn, O., 2003. Social assessment of waste energy utilization scenarios. Energy
The International Journal 28, 13451357.
Shell, 2005. Shell Global Scenarios to 2025. The future business environment:
trends, trade-offs and choices, June 2005.
Stagl, S., 2006. Multicriteria evaluation and public participation: The case of UK
energy policy. Land Use Policy 23 (1), 5362.
Stagl, S., 2007. Emerging methods for sustainability valuation and appraisal SDRN
rapid research and evidence review. Sustainable Development Research
Network, London. Available from: <>.
Stewart, W.R., Horowitz, E.R., 1991. Environmental factor weighting at the federal
energy regulatory commission. Socio-Economic Planning Sciences 25 (2), 123
Stirling, A., 2005. Opening up or closing down: analysis, participation and power in
the social appraisal of technology. Japan Journal for Science Technology and
Society 14, 6383.
World Business Council for Sustainable Development (WBCSD) 1998. Exploring
Sustainable Development.
Weaver, P.M., Rotmans, J., 2006. Integrated Sustainability Assessment. What? Why?
How? Matisse Working Paper No. 1.
Wilson, I., 2004. From Scenario Thinking to Strategic Action, citing Wilson I, 1975,
Presentation to the American Association for the Advancement of Science, 2000.
action.asp>), May 2004.
Zelle, K., Schechtner, O., 2004. Energie Gesamt Rechnung. Endenergieverbrauch
2000, Ldersdorfund Raabau. Verbrauchsstrukturen aus zusammengefassten
Daten von statistikbasierten EGR-Modellrechnungen fr alle Gemeinden des
Bundeslandes Steiermark, ADIP-Graz.