Expert Systems with Applications 36 (2009) 52725283

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier.com/locate/eswa

An integrated decision support system dealing with qualitative and

quantitative objectives for enterprise software selection
Ceyda Gngr S en a,*, Hayri Baral a, Seluk Sen b, Hseyin Baslgil a
a
b

Department of Industrial Engineering, Yildiz Technical University, Yildiz, Istanbul 34349, Turkey
Production Department, Audio Electronics, Umraniye, Istanbul 81260, Turkey

a r t i c l e

i n f o

Keywords:
Enterprise software selection
Qualitative and quantitative objectives
Fuzzy multi-criteria decision making
procedure
Multi-objective programming
ERP

a b s t r a c t
Previous methods for enterprise software selection generally take into account the attributes that are
restricted to some nancial factors, such as costs and benets. However, the literature lacks studies on
considering the evaluation of both functional and non-functional suitability of software alternatives versus various requirements. This study presents a new decision support system for combining these two
kinds of evaluation to select suitable enterprise software. A hierarchical objective structure that contains
both qualitative and quantitative objectives is proposed to evaluate software products systematically.
This approach uses a heuristic algorithm, a fuzzy multi-criteria decision making procedure and a multiobjective programming model to make nal selection decision. All the phases of presented method are
applied in an electronic companys ERP software selection project to validate it with a real application.
The satisfactory results are obtained during this project. The company can select the right software to
t its business processes instead of adapting its business processes to t the software.
2008 Elsevier Ltd. All rights reserved.

1. Introduction
Severe market competition has dramatically transformed the
business environment with the result that companies need to reduce total costs, maximize return on investment, shorten lead
times, and be more responsive to customer demands. Highly dynamic markets call for effective enterprise software systems to enhance competitive advantage (Wei, Chien, & Wang, 2005).
Organizations have several options for acquiring business software
applications. Acquisition through development includes development by an internal IT group or custom development by third parties. Alternatively, organizations may acquire software through the
purchase of pre-developed congurable systems from software
vendors. Over time this approach has become the dominant means
of software acquisition, accounting for approximately 70% of corporate business software expenditures (Holland & Light, 1999).
Due to the growth in specialized software companies, coupled with
diverse skill requirements, and rapidly changing technology, organizations are increasingly purchasing enterprise software packages
instead of custom developing their own software applications
(Sherer, 1993). Enterprise software packages are pre-written by a
vendor to provide a set of standard functions usable by a wide variety of companies, regardless of size or industry. Commercial off the

* Corresponding author. Tel.: +90 212 3832873; fax: +90 212 2585928.
E-mail address: cgungor@yildiz.edu.tr (C.G. S
en).
0957-4174/$ - see front matter 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.eswa.2008.06.070

Shelf (COTS) is other term that refers to enterprise software such as

accounting, e-commerce, human resources (HR), customer relationship management (CRM), supply chain management (SCM),
and enterprise resource planning (ERP) systems.
Purchasing the appropriate enterprise software requires a comprehensive selection process from a nite number of alternatives
that contains multiple objectives with conicts, and involves usage
of data that can be quantitative like cost and qualitative like linguistic variables. In this process, a systematic and repeatable selection methodology is a crucial need for minimizing the uncertainty
and risk. In literature, the majority of methods for selecting enterprise software aim to reduce the potentially large number of comparisons needed to evaluate many applications against many
requirements, either through some process of eliminating potential solutions (Kontio et al., 1995; Lawlis, Mark, Thomas, & Courtheyn, 2001; Tran & Liu, 1997) or by letting available
functionality shape requirements (Maiden & Ncube, 1999). Other
perspectives on software selection focus on the criteria that organizations consider in selecting commercial software (Wei et al.,
2005), the various decision-making techniques (Kontio et al.,
1995; Lai, Wong, & Cheung, 2002; Wei et al., 2005), and managing
the risks inherent in the selection process (Sherer, 1993). However,
these methods can not contain sufcient detailed attributes and
generally take into account the attributes that are restricted to
some nancial factors, such as costs and benets. Furthermore,
many of them involve only the consideration of non-functional
criteria which are not easy to quantify, but do not offer a

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

comprehensive process for including system requirements into the

decision-making process.
Therefore, the literature lacks studies on combining the evaluation of functional and non-functional suitability of software alternatives. This study aims to provide a new decision support
system for integrating these two kinds of evaluation for selecting
suitable enterprise software. In the proposed method, a specic
objective hierarchy that contains both qualitative and quantitative
objectives is presented for enterprise software selection problem.
The systematic procedure can help to obtain functional and nonfunctional suitability of software alternatives, versus various criteria. These evaluations are combined in a multi-objective mathematical programming model to make nal selection decision.
The rest of the paper is organized as follows: in Section 2, the
steps and details of the proposed decision support system integrating heuristic algorithm, fuzzy multi-criteria decision making meth-

5273

od and multi-objective programming model is introduced for

enterprise software selection. In Section 3, an actual application
of the proposed method is presented. Finally, Section 4 concludes
the study.
2. An integrated enterprise software selection method
Our integrated enterprise software selection method (ESSM)
comprises two dimensional analyses for selecting most suitable
enterprise software. The rst dimension of the ESSM involves qualitative evaluations that are performed to obtain functional and
non-functional suitability for software alternatives on the basis
of system requirements and non-functional criteria through
assigning subjective ratings. The second dimension of the ESSM includes quantitative evaluations of project factors, cost and implementation time. Fig. 1 represents the objective hierarchy of the

Fig. 1. The objective hierarchy of the proposed ESSM.

Fig. 2. Proposed enterprise software selection procedure.

5274

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

ESSM and the criteria used for the evaluation process. To clearly
present the proposed ESSM, a stepwise procedure is described.
Fig. 2 shows the proposed enterprise software selection procedure.
2.1. Form a project team and conduct BPR
Software selection process affects several functions in the organization; therefore, such a decision should be made according to
the consensus of a cross-functional team of decision makers with
various points of views and who represent different services of
the company. Hence, the rst step of our approach is to form a
team that consists of all related managers and supervisors of a
company, such as a general manager, production manager, quality
manager, IT manager and R&D manager. In essence, an enterprise
software project involves not only installing a new information
technology system to replace the legacy system, but also reshaping
the business processes to overcome the challenges of a dynamic
market (Wei & Wang, 2004). The software package requires not
only integration with existing systems, but also some customization to the idiosyncratic business processes of the adopting organization. The project team can develop the functional requirements
of enterprise software during the business process reengineering
(BPR), and then incorporate these characteristics appropriately into
the decision model. Therefore, it is necessary to undertake to BPR
to rationalize and standardize the workows of all business processes in advance.
2.2. Identify the criteria for enterprise software selection
A requirements analysis of a software system is often considered to be one of the most crucial steps in the software acquisition
process (Liu, 1998), in which statements describing the functions
and characteristics of the forthcoming software system should be
developed and agreed upon. In order to achieve high user-satisfaction and high feasibility, the software requirements should be carefully specied and analyzed. The software requirements are
divided into two separate categories: functional and non-functional
(Karlsson, 1997). In this step, project team decomposes both functional and non-functional requirements into a hierarchical criteria
set.
The functional requirements are the core of the statement,
describing the functions of the software system that are expected
by the stakeholders. Functional requirements typically describe
the relationships between all valid (and invalid) inputs to the software system and the corresponding outputs of the software system. Our framework relies on user-dened functional
requirements instead of a short-list of evaluator-generated criteria
to determine product suitability. The cross-functional team can
easily gather the basic functional requirements during the BPR
analysis. These requirements may be represented in a list of process and business related functions, scenarios or use-cases, and
should include essential user requirements and standards. In some
organizations, comprehensive market research is called for, indepth interviews with users are carried out, and current versions
of software systems are analyzed. Such basic requirements are often both vague and non-veriable. Hence, we propose to expand
these primary requirements into secondary and tertiary, more detailed requirements.
In many software acquisition methods functional requirements
are resolved, but non-functional requirements (NFRs) are more or
less deliberately put aside (Karlsson, 1997). Representing and
modelling NFRs such as usability or reliability may be less straightforward, since such requirements are often inherently problematic.
Traditionally, features of a system that are not covered by its
functional description have been called non-functional requirements
(NFRs) (Bosch & Molin, 1999; Buschmann, Meunier, Rohnet,

Sommerland, & Stal, 1996; Karlsson, 1997). These requirements

are notorious for being difcult to elicit, express, quantify and test.
Even though, a vaster literature describes different sets of nonfunctional criteria (Erol & Ferrell, 2003; Illa et al., 2000; Kunda &
Brooks, 1999; Ochs, Pfahl, Chrobok, & Nothhelfer-Kolb, 2000; Standards 9126 (1991); Wei & Wang, 2004; Wei et al., 2005), these criteria should be reviewed and selected according to system
requirements and priorities by the team members.
In the third Early Aspects: Aspect-Oriented Requirements Engineering and Architecture Design Workshop, which was held in
Lancaster, UK, on March 21, 2004, participants proposed a number
of research directions for the eld (Tekinerdogan, Moreira, Araujo,
& Clements, 2004). One of the directions was to identify or
catalogue common non-functional requirements. In this workshop,
Brito and Moreira (2004) present a requirements engineering model that proposes the use of catalogues to help with the identication and specication of non-functional requirements. According
to these directions, we conduct an extensive literature review,
and determine the 56 NFRs as important for consideration during
the software selection process.
We incorporate the requirements engineering researchers view
of NFRs as selection criteria in our approach (Glinz, 2005; Mylopoulos, Chung, & Nixon, 1992). After organizing the factors addressed
in prior studies, the NFRs are classied into three main criteria, ci
(i = 1, 2, 3), such as quality characteristics, technology factors,
and socio-economic factors. On the second level, the main criteria
are divided into sub-criteria, cij (j = 1, 2, . . . ,n(i)), where n(i) denotes
the number of sub-criteria of the main criteria ci. On the third level,
some sub-criteria are divided into other sub-criteria, cijk (k = 1,
2, . . . ,n(j)), where n(j) denotes the number of sub-criteria of cij.
Fig. 3 depicts the criteria hierarchy, which consists of three levels.
The identied criteria are used in the proposed approach as a nonfunctional criteria template. If new criteria emerge to satisfy
changing business needs, then they can be included in the proposed template.
2.3. Collect information concerning software vendors and systems
A wide range of information concerning enterprise software
vendors and systems should be obtained from professional magazines, exhibitions, yearbooks, the Internet, and the other sources.
Instead of using a short-list of candidate products, we propose to
identify every product that possibly addresses the requirements.
To narrow the list into the serious candidates, evaluators compose
a questionnaire involving all system requirements and submit it to
vendors of potential products. Furthermore, evaluators in cooperation with users can create appropriate scenarios and investigate
product performance in these scenarios.
After examining the inputs from vendors, the project team can
eliminate the clearly unqualied vendors and thereby reduce the
number of alternatives. The nal identied vendors are requested
to provide information in response to the specic questions included in the checklist and demonstrate their real systems. The
decision makers then assess the information and demonstration
to evaluate the functional and non-functional suitability of software alternatives.
2.4. Conduct qualitative evaluations
In this dimension of the proposed method, qualitative evaluations are conducted to obtain functional and non-functional suitability of software alternatives, versus various criteria. The
weights given to functional and non-functional selection criteria
are often qualitatively described or imprecisely measured. It is easier for a decision maker to describe the importance of a criterion,
by using common language (Lin, Hsu, & Sheen, 2007).

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

5275

Fig. 3. Non-functional criteria hierarchy.

For example, non-functional criteria like risk factors, efciency

and the ability of a vendor may not be precise. Moreover, the fulf-

ilment of non-functional criteria is relative. Assigning absolute

numbers to these criteria tends to be problematic, since there is

5276

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

no clear distinction between the numbers. For example, how to

separate priority 4 from priority 5 is not obvious or repeatable.
The weights of these criteria and evaluation ratings under these
criteria are frequently assessed in linguistic terms like high,
poor, among others rather than crisp numbers. Hence, in this section we present a multi-criteria decision-making method, based on
fuzzy set theory and a random experiment in order to convert
qualitative data into numerical numbers.
Fuzzy sets theory was introduced by Zadeh (1965) to deal with
problems in which phenomena are imprecise and vague. All the
information about a fuzzy set is described by its membership function. Fuzzy numbers are represented by membership functions
used to handle imprecise information, such as close to 5 or very
important. There are various types of fuzzy numbers, each of
which may be more suitable than others for analyzing a given
ambiguous structure; the present approach uses triangular fuzzy
numbers. The use of triangular tness functions is fairly common
in the literature (Ayag & zdemir, 2006; Detyniecki & Yager,
2001) because triangular fuzzy numbers are among the few fuzzy
number forms that are easy to manage from a computational point
~ l; u has the folof view. A triangular fuzzy number denoted as M
lowing triangular type membership function:

8
>
< x  l=m  l; l 6 x 6 m;
lM~ x u  x=u  m; m 6 x 6 u;
>
:
otherwise:
0;

In this study, the importance weight of each functional and nonfunctional criteria, and the rating values of software alternatives versus non-functional criteria are considered as linguistic terms, LVk
(k 1; 2; . . . ; K), such as very strong, strong, medium, weak and none
(LV1 = very strong-VS, LV2 = strong-S, LV3 = medium-M, LV4 = weakW, LV5 = none-N). These linguistic terms can be expressed via triangular fuzzy numbers as shown in Table 1. The membership functions
of the ve linguistic values are shown in Fig. 4.
Using the fuzzy representation to quantify a human response
requires a defuzzication process. Defuzzication is dened as
the mapping of a fuzzy set F to elements of the universe considered
signicant with respect to F (Yager & Filev, 1994). There are a lot of
well known techniques in transforming a fuzzy set to a crisp deterministic value (Roychowdhury & Pedrycz, 2001). In this paper, a
defuzzication operator, namely, RAGE (RAndom GEneration),
which performs a random experiment to select the element is
used. The reason for using this procedure is that it is easy for the
decision makers to use and calculate. The RAGE method treats
the membership function as a probability distribution and uses a
random experiment to generate a crisp deterministic value within
the domain of the linguistic response.
Consider a fuzzy decision D which is a fuzzy subset of {d1, . . . ,dk}
with a membership function lD : fd1 ; . . . ; dk g ! 0; 1: An operator
R
Defuzz : Ffd1 ; . . . ; dk g ! fd1 ; . . . ; dk g is called the randomized
defuzzication operator for the fuzzy decision D if
R

Defuzz D dj ;

where dj is an outcome of the random experiment.

Table 1
Fuzzy sets and membership functions
Fuzzy set

Membership function

Domain

Triangular (l, m, u)

Very strong
Strong

l(x) = (x  7.5)/(10.0  7.5)

l(x) = (x  5)/(7.5  5.0)
l(x) = (10.0  x)/(10.0  7.5)
l(x) = (x  2.5)/(5.0  2.5)
l(x) = (7.5  x)/(7.5  5.0)
l(x) = (x  0)/(2.5  0)
l(x) = (5.0  x)/(5.0  2.5)
l(x) = (2.5  x)/(2.5  0)

7.5 6 x 6 10.0
5.0 6 x 6 7.5
7.5 6 x 6 10.0
2.5 6 x 6 5.0
5.0 6 x 6 7.5
0 6 x 6 2.5
2.5 6 x 6 5.0
0 6 x 6 2.5

7.5, 10.0, 10.0

5.0, 7.5, 10.0

Medium
Weak
None

2.5, 5.0, 7.5

0.0, 2.5, 5.0

Fig. 4. Membership functions for linguistic variables.

The RAGE defuzzication procedure is as follows:

1. Transform the fuzzy set into the probability distribution P by
using the Eq. (3):

lD di
; i 1; . . . ; I:
i1 lD di

Pdi pi PI

2. Divide the unit interval into n intervals, one for each element di
of the fuzzy decision D. Denote these intervals:

Ri ai ; bi  i 1; . . . ; n:

Dene these Ri as:

a1 0

b1 P1 ;

ai bi1

bi ai Pi :

3. Perform a random experiment generating a random number (r)

from the uniform distribution on the interval [0, 1].
4. If r 2 Ri then use dj di :
The application of this process requires software support to carry out all calculations. Therefore, an application interface was
developed in Microsoft Excel environment with the Visual Basic
language. To automate randomized experiment, a Visual Basic
macro was also developed. In this way, qualitative data in the form
of linguistic variables can easily transform into numerical values.
2.4.1. Perform functional suitability analysis
The aim of functional suitability analysis is to evaluate the software alternatives by the system requirements determined in Section 2.2 and to obtain functional suitability score for each
product. This evaluation is a qualitative process that produces data
on how well each alternative meets the identied requirements.
Now suppose M decision makers provide their opinion on the
importance of each primary (wi, i = 1, 2, . . . ,I) and secondary functional requirement (wij, j = 1, 2, . . . ,n(i)) using one of the linguistic
variables provided. These weights are calculated by using following equations:

wi

M
X

Defuzz LV m
8i;
k

m1

wij

M
X

Defuzz LV m
8i and j;
k

m1
R

0.0, 0.0, 2.5

where Defuzz LV m
k is the outcome of the random experiment from
the appropriate membership function for linguistic variable LVk that

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

was chosen by DMm (m = 1, 2, . . . ,M). The normalized weights of

requirements, Nwi and Nwij, are calculated by using following
equations:

wi
Nwi PI

wij
Nwij Pni
i1 wij

i1 wi

On the other hand, team members assign numerical values (0, 1, 2

or 3) to each tertiary system requirement indicating the expected
performance value of the resulting software system (eijk, k = 1,
2, . . . ,n(j)) by reaching consensus. Requirements receive a 3 for full
coverage, 2 for partial coverage, 1 for inadequate coverage and
0 for no coverage. During the vendor demonstrations, the software alternatives are evaluated according to the actual performance. Numerical values to each tertiary requirement indicating
the breadth of coverage for the s. candidate product (psijk, s = 1,

5277

2, . . . ,S) are assigned by using the same scale. The reason for using
a numerical scale instead of fuzzy set theory is that the fulllment
for functional requirements is not relative. It is easy for decision
makers to reach consensus on the breadth of coverage of a functional requirement like the system must be able to print out invoices for an alternative.
Thereafter, the heuristic algorithm is performed as depicted in
Fig. 5. This algorithm allows comparing psijks to eijks and determining the functional suitability score of s. alternative (fsijk) for each
tertiary system requirement. By applying this algorithm, functional
suitability score for each secondary (fsij) and primary (fsi) system
requirement, and the overall functional suitability score of s. alternative are also obtained. This algorithm produces the scores varying in the range of [0, 1].
Beginning of this step, an acceptable level of functional coverage, f (f e [0, 1]), which indicates the degree of optimism of the
decision makers, has to be determined. A larger f represents a lesser degree of optimism. Software alternatives with the functional

Fig. 5. Heuristic algorithm for obtaining the functional suitability scores.

5278

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

suitability score that is higher than or equal to f are selected for the
detailed evaluation. In other words, meeting at least f percent of
system requirements is enough to purchase an enterprise software
for this organization. On the other hand, if no product meets an
acceptable level of functional coverage, f, as determined by the
decision makers, building a custom-designed application is the
only way to meet specic system requirements. In this case, the required capabilities must rely on custom development rather than
purchasing a packaged product. Consequently, this step of our proposed method supports the organizations build or buy decision.
2.4.2. Perform non-functional suitability analysis
In non-functional suitability analysis, team members rate nonfunctional criteria determined in Section 2.2 to indicate how well
each product performs. The importance weights of non-functional
criteria are similarly computed like the importance weight of functional requirements. It is dened w0i as the weight of main nonfunctional criteria ci (i = 1, 2, 3); w0ij the weight of sub-criteria cij
(j = 1, 2, . . . , n(i)), and w0ijk the weight of sub-criteria cijk (k = 1,
2, . . . ,n(j)). The terms differ from those used to describe the importance weight of the functional requirements; however, the computation strategy is identical. Finally the normalized weights of
criteria, Nw0i , Nw0ij , Nw0ijk , are calculated by using following
equations:

w0
Nw0i P3 i

10

0
i1 wi
0
wij
Nw0ij Pni ;
0
i1 wij
0
wijk
Nw0ijk Pnj
0
i1 wijk

11
12

Thereafter, the product ratings under the lowest level criteria on the
hierarchical structure are computed by using the same linguistic
variables and defuzzication procedure. Dene Rsijk (s = 1, 2, . . . ,S)
as rating of s. product under sub-criteria cijk (i = 1, 3; j = 1,
2, . . . ,n(i); k = 1, 2, . . . ,n(j)). It is computed by following equation
for quality characteristics and socio-economic factors:

PM

R
m
m1 Defuzz LV k

Rsijk

i 1; 3 and 8s;

13

where the terms are the same as previously dened.

For the second level sub-criteria of technological factors, it is
dened Rsij (s=1,2, . . . ,S) as rating of s. product under sub-criteria
cij (i = 2; j = 1,2, . . . , n(i)). The same procedure is repeated and Rsij
s are calculated.
Let nfsij denote the non-functional suitability score of s. product
for sub-criteria cij:

nfsij

nj
X

Rsijk  Nw0ijk

i 1; 3 8j and s;

14

k1

nfsi is the non-functional suitability score of s. product for main criteria ci and is calculated by using following equations:

nfsi

ni
X

nfsij  Nw0ij

i 1; 3 8s;

15

Rsij  Nw0ij

i 2 8s:

16

j1

nfsi

ni
X
j1

Finally, calculate the overall non-functional suitability score (nfs) for

s. product by the Eq. (17):

nfs

3
X
i1

nfsi  Nw0i

8s:

17

This formula produces overall rating in the familiar 010 range.

Non-functional suitability scores are then divided by 10 to normalize them onto the range [0, 1].
2.5. Conduct quantitative evaluations
Much research has dened project factors as attributes involved
in project management like total cost of ownership and time of
implementation (Chen & Gorla, 1998; Teltumbde, 2000; Wei &
Wang, 2004). For different decision makers the values of these
quantitative data are the same. Hence, in many software selection
methods, these attributes are considered as quantitative attributes
that can be numerically evaluated.
Enterprise software products have a differential cost association- some are considered more expensive than others. The cost
here relates, however, to the total cost. According to Chen and
Gorla (1998) the cost of project is an estimate of the total cost of
the project, which is based on estimated completion time and
the resources used. Teltumbde (2000) denes total costs as cost
of license, training, implementation, maintenance, customization
and hardware requirements. The cost data on enterprise software
projects are difcult to obtain. Kontio et al. (1995) discuss two
main approaches for cost estimation: use of cost models or work
breakdown structure analysis. Cost models, in theory, may provide
a way to obtain unbiased estimates, but their problem is that traditional cost models are not very applicable for enterprise software
cost estimation, and it is difcult to capture all relevant factors into
a single cost model. The work breakdown structure analysis may
be, for many organizations, the feasible method for enterprise software cost estimation: the development and integration tasks for
software are listed and decomposed, effort for each task estimated
and total effort summed up. The disadvantage of this approach is
that it can be very sensitive to bias or the experience of the personnel. Our proposed method does not propose a method or model for
enterprise software cost estimation. Whatever approach is used in
our software selection method, we extend the nancial enterprise
software evaluation by allowing the consideration of other attributes like quality factors (e.g., reliability, maintainability, portability, efciency, etc.) and business or strategic issues that may
inuence the decision. Theoretically, these factors are not included
in the cost estimation model.
In this step, total cost of ownership (TCOs) and implementation
time (ts) for s. software alternative are numerically measured by
team members. The values of these quantitative attributes are collected from the data by the enterprise software vendor provided or
the data that negotiated with the vendor.
After qualitative and quantitative evaluations, the data has
now been transformed into a format capable of being used to
parameterize a multi-objective model that will assist the decision
maker. Specically, the quantitative data like cost or implementation time of software alternatives that have been collected are
combined with the qualitative data that has been quantied in
the form of the suitability scores. These data can be used as input
data for the multi objective model as described in the following
section.
2.6. Build and solve the multi-objective mathematical programming
model
The nal functional element of the proposed methodology is the
multi-objective model. Two objectives are considered that we assume conict, namely, maximizing suitability of software alternatives, and minimizing project factors. As shown in objective
hierarchy (Fig. 1), there are four sub-objectives of the model,
namely, maximizing functional suitability, maximizing nonfunctional suitability, minimizing total cost of ownership and

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

minimizing total implementation time. The formulation of the 01

goal programming model is as follows:

Min: z P l

!


ki dsi l 1 or 2

Pl

ki dpi l 1 or 2

s:t:

S
X

fs xs ds1 1

s1

Functional suitability related objective;

S
X

nfs xs ds2 1

s1

Non-functional suitability related objective;

S
X

TCOs xs dp1  dp1 TCO

s1

Total cost of ownership  related objective;

S
X

t s xs dp2  dp2 T 

s1

Implementation time  related objective;

S
X
s1

dsi ;

xs 1 Choice constraint;

dpi ; dpi P 0 i 1; 2;

xs 0 or 1 s 1; 2 . . . ; S;
where xs, takes values of 0 or 1, where 1 represents that software s
is selected and 0 represents otherwise; Pl, represents priorities assigned to the main objectives (l = 1, suitability; l = 2, project factors); ki , represents the weights attached to the deviation
P
variables associated with suitability objectives, ( 2i1 ki 1); ki,
represents the weights attached to the deviation variables associated with project factors, (i = 1, total cost of ownership; i = 2,
P2
implementation time and
i1 ki 1); fs, represents functional
suitability score of software s; nfs, represents non-functional suitability score of software s; TCOs, represents total cost of ownership
of software s; ts, represents estimated completion time for imple
menting software s; dsi , represents the deviation variables associated with suitability objectives (i = 1, functional suitability; i = 2,


non-functional suitability); dpi ; dpi , represents the deviation variables associated with project factors (i = 1, total cost of ownership;
i = 2, implementation time); TCO*, represents the best value for the
total cost of ownership across all software alternatives; T*, represents the best value for implementation time across all software
alternatives.
The resulting solution represents the best choice for the decision maker.
3. An actual evaluation
The proposed approach was applied to Audio Electronics of Turkeys electronic industry to prove the applicability and validity of
this method in an actual industrial environment. Audio Electronics
is an ISO 9001:2000-certied manufacturer of audio and video
intercom systems for apartments and villas. Since 1979, Audio
has grown from a modest origin to a company with a nationwide
reputation for reliable products of innovative design. It has more
than 1500 different products and exports to at least 20 countries.
In recent years, increased product variety, accelerated technological innovation, and a shortened product life cycle have forced
the subject company to improve intra-rm processes. After initial
talks with several consulting rms, top management brought in
one of them to help the company with their improvements. The

5279

company began to reorganize into a cross-functional, matrix organization structure in terms of its hierarchical structure. Meanwhile,
a cross-functional team was charged with reengineering the companys supply chain processes to better meet customer needs. One
of the key conclusions from these efforts was that the organization
could not prosper with its current information system. The companys most recent major investments in information technology
had been made over ve years earlier. Since, the LOGO Classic
has been used as a commercial automation solution in the company. This system is provided by Turkeys largest independent software vendor Logo Business Solutions. It is the most commonly
used small business solution in Turkey, which was developed
according to customer requests and regularly collected directions
since the rst version. Logo Classic includes accounting, inventory,
invoice, bank, safe deposit, xed assets, ination accounting, additional tax, payroll, fast production, and navigator modules. However, in Audio, all modules of the system were not implemented,
and some of the existing modules have not been operated effectively. Logo Classic System has been used for only storing bill of
material (BOM) data, routing, cost, and accounting information.
The consensus among Audios management team was that the
company was information poor and needed to be cut loose from
its existing system. There were also major concerns about being
able to grow the company and become more global without an
integrated information capability. In these circumstances, in order
to meet the companys new business needs, the consulting rms
team and Audios management team have decided to replace the
existing system with a new enterprise resource planning (ERP) system that would lead to an overall organizational improvement.
Consequently, an effective software selection process has become
necessary.
First of all, in order to conduct ERP selection and project implementation, three cross-functional teams are formed according to
the companys organizational structure. These teams are: (i) technical team, (ii) nancial team, and (iii) logistics team. The technical
team is responsible for all customization and development projects, as well as hardware infrastructures; the nancial team is
responsible for all nance-related activities in the ERP project.
The logistics team consists of the representatives of production,
purchasing, R&D, product management, sales, and marketing
departments. They are responsible for all issues related to their
operations, such as master planning, production planning, forecasting, scheduling, all material movements, BOM validation, and
project management. These teams are guided by Audios management team and supported by consultants.
Next, the members of these three teams interview with the
users in all departments to learn their expectations and the perceived needs of the software. In this step, we assist team members
by providing a detailed functional requirements checklist that consists of three levels and contains more than 600 requirements for
different functional areas. This checklist has been developed based
on our experience as consultants to help different companies select
an ERP, and our experience as an implementer of ERP systems. The
team members consider the requirements listed in this checklist
and, in the event that a different requirement occurs during the
interviews in each department, it is added to the checklist. This
checklist is found to be very useful in collecting data. Since the detailed BPR analyses have already been performed, users can easily
decide what kinds of requirements should be met by ERP software.
Team members record these requirements and categorize them by
the seven main business units, such as logistics (LOG), production
planning (PP), sales management (SM), nance management (FM),
human resources management (HRM), information technology (IT)
and the others (OTH).
Regarding non-functional criteria, the team members reached
consensus on using our proposed criteria hierarchy template rep-

5280

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

resented in Fig. 3. After brainstorming sessions, they believe that

each of the three main criteria, namely quality characteristics,
technology factors, and socio-economic factors are important,
and these criteria form the rst level of the hierarchy. All of the
sub-criteria associated with quality characteristics, except installability, are chosen. There are ten sub-criteria related to technology factors, namely platforms, database management systems,
languages and development tools, user management tools, user
documentation, external connectivity, versioning, interface standard, transparency, and module completion. Only one criterion,
vendor capability, is selected as a socio-economic main criterion.
Five sub-criteria are identied to be related to vendor capability,
namely training and support, vendor reputation, R&D technology,
consulting service, and service ability.
Thereafter, information on 15 ERP vendors and systems are initially collected. Unfavourable alternatives are eliminated by asking
a few questions about vendor size, domain, information technology, consulting service, service maintenance etc. At the same time,
the list of system requirements is sent to the vendors via e-mail.
After preliminary screening, seven ERP vendors, IAS (s = 1), Microsoft Axapta (s = 2), Obje (s = 3), Uyumsoft (s = 4), Set (s = 5), MAPICS
(s = 6) and LogoSoft (s = 7) remain under consideration.
In the functional suitability analysis step, the system requirements are translated into the hierarchical structure. The leaders
of each of the three teams, the MIS manager (DM1), the nance
manager (DM2), and the production manager (DM3) are evaluated
weights of the each primary (wi, i = 7 in our case) and secondary
functional requirement (wij) on the basis of hierarchical structure
by using linguistic terms provided in Section 2.4 (very strong-VS,
strong-S, medium-M, weak-W, none-N). The linguistic variables
are translated into fuzzy sets by dening appropriate tness functions, as dened in Table 1. The RAGE defuzzication procedure is
then performed for each response of each decision maker. This procedure requires transforming each fuzzy set into a probability distribution. Therefore, a random experiment of one thousand values
of the membership function corresponding to one thousand randomly selected points in the domain for each fuzzy set is carried
out. The probability distribution was obtained by the normalization of the membership function values associated with the fuzzy

set (Eq. (3)). All these calculations are easily implemented on

spreadsheets using VBA macros developed.
Eqs. (6) and (7) are used to aggregate decision makers assessments and to calculate the weights of all criteria (w0i s, w0ij s and
w0ijk s). Suppose all three team members think that the weight of
the logistics is very strong. Then, wlogistics is calculated as

wlogistics

3
X

Defuzz LV m
VS 9:04 9:86 9:47 28:37:

m1

The values 9.04, 9.86 and 9.47 are obtained from a random
experiment using the triangle distribution for very strong. These
weights are then normalized by using Eqs. (8) and (9). Example
calculations for logistics main requirement are illustrated in
Table 2.
Team members assign numerical values (0, 1, 2 or 3) to each
tertiary system requirement indicating the expected performance
value of the resulting software system (eijk, k = 509 in our case)
by reaching consensus. Afterwards, each vendor is given about
3 h to address the system requirements and formally demonstrate
their product. During these demonstrations, the breadth of coverage for the s. candidate product (psijk, s = 7 in our case) are assigned
by using 03 scale. The heuristic algorithm is performed as depicted in Fig. 5 to obtain functional suitability score for each secondary (fsij) and primary (fsi) system requirement, and the overall
functional suitability score of s. alternative (fs). The results of the
algorithm are depicted in Table 3.
By the team members, acceptable level of functional coverage, f,
is determined as 0.75. Since there exist alternatives with the
functional suitability score that is higher than or equal to f, team
members decide to continue the software selection process. Consequently, IAS, MS Axapta, Uyumsoft and Mapics are selected for the
detailed evaluation (s = 1, 2, 4, 6).
Afterwards, the leaders of each of the three teams, the MIS manager (DM1), the nance manager (DM2), and the production
manager (DM3) are asked to perform non-functional suitability
analyses. Each of them establishes the weights of each non-functional criterion in the form of linguistic variables. These opinions
are recorded in separate spreadsheets for each of the three decision

Table 2
Linguistic assessments and aggregated weights for logistics functional requirement
Functional requirements (i = 1, j = 1, 2, . . . 7)

Decision makers

LOG
Organizational structure
Logistics master data
Purchasing
Vendor evaluation
Stock management
Warehouse management
Reporting

DM 1

DM 2

DM 3

VS
VS
M
S
M
VS
S
VS

VS
M
VS
VS
VS
VS
VS
VS

VS
M
VS
S
S
VS
VS
VS

Weights (wi, wij)

Normalized weights (Nwi, Nwij)

28.37
20.77
26.27
23.50
23.11
27.29
26.41
27.22

0.16
0.12
0.15
0.13
0.13
0.16
0.15
0.16

Table 3
The obtained results from functional suitability analysis
Functional requirements (i = 1, 2 , . . . ,7)

Normalized weights

fsi (s = 1,2, . . . ,7)

Alternatives

(Nwi)

s=1

s=2

s=3

s=4

s=5

s=6

s=7

LOG
PP
SM
FM
HRM
IT
OTH
fs (s = 1, 2, . . . ,7, f = 0.75)

0.16
0.17
0.16
0.16
0.10
0.15
0.10

0.97
0.93
1.00
0.98
0.49
0.99
0.00
0.83

0.88
0.98
0.98
0.95
0.91
1.00
0.88
0.94

0.84
0.31
0.82
0.68
0.89
0.91
0.00
0.65

0.90
0.74
0.81
0.90
0.86
1.00
0.12
0.79

0.56
0.66
0.64
0.66
0.82
0.49
0.00
0.57

1.00
1.00
0.97
0.99
0.78
0.94
0.74
0.94

0.73
0.77
0.84
0.68
0.76
0.94
0.00
0.71

5281

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

Table 4
Example calculations for the weights of socio-economic factors
Non-functional criteria (i = 3, j = 1, k = 1, 2, . . . ,5)

Decision makers

Socio-economic factors
Vendor capability
Training and support
Vendor reputation
R&D technology
Consulting service
Service ability

DM1

DM2

DM3

S
VS
M
VS
VS
M
VS

S
VS
S
VS
VS
M
VS

VS
S
M
VS
VS
M
S

makers. Afterwards, the weights of all criteria (w0i s, w0ij s and w0ijk s)
on the hierarchical structure are calculated. These calculations
are illustrated for socio-economic factors main criterion in
Table 4. Suppose two of three decision makers think that the
importance degree of the socio-economic factors (c3) is strong
and one of them thinks it is very strong. Then w03 is calculated:

w03

2
X

Defuzz

Weights of criteria w0i ; w0ij ; w0ijk

Normalized weights Nw0i ; Nw0ij

23.45
28.36
17.62
27.56
26.00
16.60
27.79

0.30
1.00
0.15
0.24
0.22
0.14
0.24

linguistic variables and defuzzication procedure. These calculations are illustrated for socio-economic factors main criterion
in Table 5. Suppose all of three decision makers think that rating
of IAS (s = 1) under sub-criteria R&D technology (c313) is medium.
Then, R1313 is calculated using equation (13) as:

P3
R1313

R
m
LV m
S Defuzz LV VS 6:98 8:10 8:37 23:45:

m1

R
m
m1 Defuzz LV M

6:62 6:20 5:13

5:98:
3

This process is repeated for each product and all criteria on the
hierarchical structure. Once all these data are transferred to
another spreadsheet, the nfsijs, nfsis and nfss are now easily computed by using Eqs. (14)(17). Table 6 gives the results for each
product.
Team members obtain estimates of total cost of ownership and
implementation time for selected software alternatives through

The values 6.98 and 8.10 are obtained from a random experiment using the triangle distribution for strong, 8.37 is obtained
from a random experiment using the triangle distribution for very
strong. These ratings are then normalized by using Eqs. (10)(12).
Thereafter, the product ratings under the lowest level criteria
(Rsijk) on the hierarchical structure are computed by using the same
Table 5
Example calculations for the product ratings under the socio-economic factors
Alternatives
Non-functional criteria
(i = 1, j = 1, k = 1, 2, . . . ,5)

s=1
Decision makers

s=2

s=4

s=6

Product ratings (Rsjk)

DM1

DM2

DM3

DM1

DM2

DM3

DM1

DM2

DM3

DM1

DM2

DM3

R131k

R231k

R431k

R631k

Training and sup.

Vendor reputation
R&D technology
Consulting service
Service ability

VS
S
M
M
M

S
S
M
S
S

M
S
M
VS
VS

VS
M
VS
S
VS

S
S
VS
VS
S

VS
VS
S
S
S

VS
VS
S
S
S

S
VS
VS
VS
VS

VS
S
VS
S
VS

S
VS
S
S
M

S
VS
M
S
VS

M
S
VS
S
VS

7.75
7.38
5.98
7.43
8.00

9.96
7.81
9.85
6.55
7.78

8.76
7.96
8.63
9.67
9.49

7.28
7.64
5.65
9.64
8.63

Table 6
Non-functional suitability scores of software alternatives
Non-functional criteria

Normalized
weights
Nw0i
Nw0ij

Alternatives

Product ratings
Rsijk
s=1

Quality characteristics (i = 1, j = 1, 2, . . . ,6)

Functionality
Reliability
Usability
Efciency
Maintainability
Portability

0.36

Technology factors (i = 2, j = 1, 2, . . . ,10)

Platforms
DB management systems
Languages and dev. tools
User management tools
External connectivity
User documentation
Versioning
Interface standard
Transparency
Module completion

0.34

Socio-economic factors (i = 3, j = 1)
Vendor capability
Overall non-functional suitability score nfs (s = 1, 2, 4, 6)

0.30

Non-functional suitability scores

nfsij

s=2

s=4

s=6

0.18
0.17
0.18
0.15
0.17
0.15
0.10
0.11
0.09
0.10
0.10
0.09
0.10
0.09
0.11
0.11
1

s=1
7.82
6.00
6.59
7.57
5.82
6.00

5.24
6.96
5.40
6.92
7.78
7.56
6.81
5.30
7.88
5.55

9.45
9.46
8.39
7.75
8.29
8.38
9.41
9.22
8.06
9.53

7.98
7.05
5.63
8.04
7.85
7.11
9.38
7.67
7.60
7.41

nfsi
s=2
8.83
7.59
9.02
7.23
7.57
7.89

s=4
8.47
7.99
7.37
7.94
6.16
7.47

s=6

s=1

s=2

s=4

s=6

6.64

8.06

7.57

7.85

6.56

8.80

7.59

6.96

7.28

8.41

8.84

7.66

6.80
0.68

8.42
0.84

7.96
0.80

7.49
0.75

8.02
7.79
8.82
8.55
6.51
7.37

6.97
7.48
5.30
7.69
6.20
7.71
5.74
8.56
6.61
7.37
7.28

8.41

8.84

7.66

5282

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

Table 7
Quantitative measurements of software alternatives

4. Conclusions

Quantitative measurements

This paper is presented an integrated decision support system

to assist decision makers in enterprise software selection process
by allowing both qualitative and quantitative objectives to be considered in a multiobjective mathematical programming model. The
method provides an objective hierarchy structure of enterprise
software selection problem that contains having complete functional and non-functional suitability, and minimizing total cost of
ownership and implementation time. A hierarchical structure
including non-functional criteria such as quality, technology factors and socio-economic factors is also provided by organizing
the factors addressed in prior studies. The identied criteria are
used in the proposed method as the non-functional selection criteria template. The presented ESSM provides the exibility to include
any specic criteria that the team may wish to consider in any
other situation. Similarly, any new member can be included in
the evaluation team to consider his or her input.
In the presented ESSM, when the qualitative data is in the
form of linguistic variables, a fuzzy set theory and random experiment based solution is used to transform this data into numerical values. The use of fuzzy set theory improves the decisionmaking procedure by considering the vagueness and ambiguity
prevalent in real world systems. Functional and non-functional
suitability analyses are presented to convert this qualitative
information into a format that can be used as parameter in a
multiobjective model. Hence, decision makers can assess various
attributes of an enterprise software system by using linguistic or
quantitative values and all decision makers have direct input into
the mathematical model. To verify the nal decision, the proposed method allows analyzing the changes in the nal outcomes
by substituting different orders of preemptive priorities and various values of k and k.
The applicability of this method is illustrated through a case
study of ERP software selection project in Audio Electronics of Turkeys electronic industry. By using ESSM, decision makers of the
company select the right software to t its business processes instead of adapting its business processes to t the software.
Although the case study is related to a specic software system
and industry, the same concept can be applied to other enterprise
software products and industrial sectors. During the application of
this approach, software support needs to carry out all calculations.
We develop VBA-based macros in Microsoft Excel and transfer the
results to spreadsheets for easy computations. However, when the
number of requirements increases, the method becomes more
complex to solve, even using Excel. For future study, we can generate guidelines and develop an expert system to facilitate decisionmaking process.

ERP software alternatives

x1

Licence cost ($0000)

Maintenance cost ($0000)
Hardware cost($0000)
Training and consulting cost($0000)
Total additional costs($0000)
Installation
Data transformation
Conguration
Additional module (CRM)

8.5
1.5
1
4
1
0.4
0.3
0.3
0

Total cost of ownership, TCOs($0000)

Implementation time, Ts (month)

16
12

x2
8.4
0.6
1
4
1
0.4
0.3
0.3
0
15
16

x4
5.5
1.5
1
3
4
0.4
0.3
0.3
3
15
6

x6
9.5
0.5
1
4
1
0.4
0.3
0.3
0
16
12

Table 8
Results of scenarios [obtained vs. (targeted)]
k1

k2

k1

k2

fs

nfs

TCOs ($0000)

Ts (month)

xs

0.70
0.70

0.30
0.30

0.70
0.70

0.30
0.30

0.94 (1)
0.79 (1)

0.84 (1)
0.80 (1)

15 (15)
15 (15)

16 (6)
6 (6)

x2
x4

interviews with four vendors. Table 7 provides data on the estimated time and total cost of implementing each ERP software.
Team members are asked also to identify the weights attached to
the deviation variables associated with suitability objectives and
project factors. These values are determined as k1 0:70; k2
0:30; k1 0:70; k2 0:30 by reaching consensus.
Based on these data and the previously computed functional
and non-functional suitability scores, we can formulate the goal
constraints for this problem. This 01 goal programming model
is solved using LINDO (version 6.1) in a few seconds of computer
time. This version of LINDO includes the capability to prioritize a
set of objectives using the preemptive goal option. The preemptive goal command performs Lexico-optimization on the model.
The problem is solved using two different orders of preemptive
priorities as requested by the decision making team. In the rst set
of scenarios, the ordering is Psuitability, Pproject, respectively. The second set of scenarios is associated with viewing project factors as
most important. The results are summarized in Table 8. The rst
row is associated with the rst scenario. For this scenario, the


results are summarized as follows: x2 1; ds1 0:06; ds2


0:16; dp1 dp1 dp2 0; dp2 10. MS Axapta is chosen consuming the total cost of $150,000. We will use 10 more months of

implementation time than the initial 6 months, so dp2 = 10 is

shown. The second row is related to the second scenario. Since this
scenario is the project factors focused, this time Uyumsoft (x4) is
chosen satisfying overall project factors related goals x4




1; ds1 0:21; ds2 0:20; dp1 dp1 dp2 dp2 0. The model is
also solved using different k and k weights that are varied from 0
to 1. However, the selected alternatives are not changed in these
scenarios with any values of k and k.
After these scenario analyses, the project team recommended
MS Axapta (x2) as the most suitable selection for the company.
The decision making team is impressed with the analysis that we
presented. They can integrate qualitative objectives like maximizing functional and non-functional suitability, and quantitative
objectives like minimizing implementation time and total cost of
ownership. The decision makers of the company believe that proposed approach is a useful tool providing the ability to examine
various scenarios and assess the sensitivity of the solutions to
changes in priorities. By using our process grounded in mathematics and decision theory, the selection time, as well as the cost of
ERP software implementation and maintenance is reduced.

References
Ayag, Z., & R.G.zdemir (2006). An intelligent approach to ERP software selection
through fuzzy ANP. International Journal of Production
Research, 126
(preview article).
Bosch, J., & Molin, P. (1999). Software architecture design: Evaluation and
transformation. In Proceedings of the IEEE engineering of computer based
systems symposium (ECBS99) (pp. 410).
Buschmann, F., Meunier, R., Rohnet, H., Sommerland, P., & Stal, M. (1996). Patternoriented software architecture a system of patterns. Chichester: John Wiley &
Sons.
Brito, I., & Moreira, A. (2004). Integrating the NFR framework in a RE model. In
Workshop of the 3rd international conference on aspect-oriented software
development (pp. 2833). Lancaster, UK.
Chen, K., & Gorla, N. (1998). Information system project selection using fuzzy logic.
IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans,
28(6), 849855.
Detyniecki, M., & Yager, R. R. (2001). Ranking fuzzy numbers using alpha-weighted
valuations. International Journal of Uncertainty, Fuzziness and Knowledge-Based
Systems, 8(5), 573592.

C.G. Sen et al. / Expert Systems with Applications 36 (2009) 52725283

_ & Ferrell, W. G. (2003). A methodology for selection problems with multiple,
Erol, I.,
conicting objective and both qualitative and quantitative criteria. International
Journal of Production Economics, 86, 187199.
Glinz, M. (2005). Rethinking the notion of non-functional requirements. In
Proceedings of the 3rd world congress for software quality (pp. 5564).
Holland, C., & Light, B. (1999). A critical success factors model for ERP
implementation. IEEE Software, 16(3), 3036.
Illa, X. B., Franch, X., & Pastor, J. A. (2000). Formalising ERP selection criteria. In
Proceedings of the 10th international workshop on software specication and
design (IWSSD00) (pp. 115123).
ISO/IEC Standards 9126 (1991). International Standards Organisation, Information
Technology Software Product Evaluation Quality Characteristics and
Guidelines for their use.
Karlsson, J. (1997). Managing software requirements using quality function
deployment. Software Quality Journal, 6, 311325.
Kontio, J., Chen, S., Limperos, K., Tesoriero, R., Caldiera, G., & Deutsch, M. (1995). A
COTS selection method and experiences of its use. In Twentiethannual software
engineering workshop (pp. 189215). Greenbelt, Maryland, USA.
Kunda, L., & Brooks, D. (1999). Applying social-technical approach for COTS
selection. In 4th UKAIS conference proceedings: Information systems the
next generation (pp. 552565).
Lai, V. S., Wong, B. K., & Cheung, W. (2002). Group decision making in multiple
criteria environment: A case using the AHP in software selection. European
Journal of Operational Research, 137, 134144.
Lawlis, P., Mark, K., Thomas, G., & Courtheyn, T. (2001). A formal process for
evaluating COTS software products. IEEE Computer, 34(5), 5863.
Lin, H. Y., Hsu, P. Y., & Sheen, G. J. (2007). A fuzzy-based decision-making procedure
for data warehouse system selection. Expert Systems with Applications, 32(3),
939953.
Liu, X. F. (1998). A quantitative approach for assessing the priorities of software
quality requirements. The Journal of Systems and Software, 42, 105113.

5283

Maiden, C. N., & Ncube A. M. (1999). PORE: Procurement-oriented requirements

engineering method for the component-based systems engineering
development paradigm. In 2nd international workshop on component-based
software engineering (pp. 112).
Mylopoulos, J., Chung, L., & Nixon, B. (1992). Representing and using non-functional
requirements: A process oriented approach. IEEE Transactions on Software
Engineering, 18(6), 483497.
Ochs, M. A., Pfahl, D., Chrobok, G., & Nothhelfer-Kolb, B. (2000). A COTS acquisition
process: Denition and application experience. In Proceedings of the 11th
ESCOM conference (pp. 335343).
Roychowdhury, S., & Pedrycz, W. (2001). A survey of defuzzication strategies.
International Journal of Intelligent Systems, 16, 679695.
Sherer, S. A. (1993). Purchasing software systems: Managing the risk. Information
and Management, 24(5), 257266.
Tekinerdogan, B., Moreira, A., Araujo, J., & Clements, P. (2004). Early aspects: Aspectoriented requirements engineering and architecture design. In Workshop
Proceedings of the 3rd international conference on aspect-oriented software
development (AOSD04) (pp. 515). Lancaster, UK.
Teltumbde, A. (2000). A framework for evaluating ERP projects. International Journal
of Production Research, 38(17), 45074520.
Tran, V., & Liu, D. B. (1997). A risk-mitigating model for the development of reliable
and maintainable large-scale COTS integrated software systems. In Proceedings
of the annual reliability and maintainability symposium (pp. 361367).
Wei, C. C., Chien, C., & Wang, M. J. (2005). An AHP-based approach to ERP system
selection. International Journal of Production Economics, 96, 4762.
Wei, C. C., & Wang, M. J. (2004). A comprehensive framework for selecting an ERP
system. International Journal of Project Management, 22, 161169.
Yager, R. R., & Filev, D. P. (1994). Essentials of fuzzy modeling and control. USA: John
Wiley and Sons.
Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8, 338353.