World Congress on Software Engineering

A Software Project Risk Analysis Model Based on Evidential Reasoning Approach

Nan Feng, Minqiang Li, Hong Gao School of Management, Tianjin University, Tianjin, China E-mail: fengnan_1978@yahoo.com.cn Abstract
In this paper, a software project risk analysis model based on evidential reasoning approach is presented. The modeling process consists of four phases: specification of the model structure, assessment of evidence strength, determination of software project risks, and risk monitoring and analysis. Using the changes of strength of evidences obtained from the process of software project development, the model can continually estimate the state of risk, and identify the sources of risk. The model provides a rigorous, structured manner to incorporate relevant software project risk factors and their interrelationships when estimating software project risks. the impact of each risk or avoid it completely. The final step is risk monitoring, which ensures that the riskreducing methods are implemented effectively and also determines whether or not the risk-reducing tactics are in fact reducing risks [5]. The crucial step in any risk management process is the analysis of identified risks in order to determine the degree of attention each risk need be paid, since this will ultimately affect the success of a project. In this paper, we develop a software project risk analysis model based on evidential reasoning approach. And the paper elaborates the theoretical concepts and provides operational guidance for implementing the model. In the next section, we discuss the main limitations with existing approaches for software project risk analysis. After that, the process of developing a software project risk analysis model is presented. Finally, we demonstrate and validate the model via a case study.

1. Introduction
There exist many uncertainties in software development processes and products; for instance, the uncertainties in estimating project size, schedule, quality, and in determining resource allocation. Current software engineering techniques cannot eliminate such uncertainties. Thus, risk management is critical [1]. According to reference [2], studies indicate that about 85% of all projects end in failure. Furthermore, it is estimated that 31.1% of projects will be cancelled before they are ever completed [3]. Software development, given its diverse and abstract nature, offers unique challenges and risks. A formal risk management program takes a structured way to evaluate risks in the software development process. A typical risk management framework involves identifying and analyzing the risks to a project and then implementing and monitoring measures to reduce them [4]. The first step in a typical risk management program is the identification of risk, usually involving checklists, questionnaires or brainstorming sessions. The next step is the analysis of the risks identified in such a way that they can be ranked in a meaningful manner. This is followed by the development of contingency measures to decrease

2. Background Research
Software project risk analysis has been given serious consideration by both academics and practitioners for quite some time. Because of space limitations, we do not present a complete review of software risk analysis literature. Instead, we discuss the major limitations of the existing approaches and how our approach mitigates some of these limitations. Reference [6] and [7] presented comprehensive reviews of this literature. Before we discuss the weaknesses of the existing approaches, we discuss the risk exposure estimating. As risk implies a potential loss, there are two elements at issue here: firstly the probability of an unsatisfactory outcome and secondly the consequences of such an outcome [8]. These two measurements are estimated using risk analysis techniques, the multiplicity of which forms the risk exposure metric that is defined by the formula below: Risk Exposure = Prob (UO) Loss (UO) (1) Prob (UO) is the probability of an unsatisfactory outcome. Loss (UO) is the loss to the parties involved if
224

978-0-7695-3570-8/09 $25.00 2009 IEEE DOI 10.1109/WCSE.2009.228

Authorized licensed use limited to: RMK Engineering College. Downloaded on August 03,2010 at 03:16:06 UTC from IEEE Xplore. Restrictions apply.

the outcome is unsatisfactory. And the probabilistic estimation of risks is the key element of risk analysis. For the probabilistic estimation of risks, there are mainly three limitations with existing approaches. First, in software project, the existing methods use the probability of a negative outcome to represent risk. This representation of risk covers only one subcomponent of software project risk and leaves out another important subcomponent (i.e., the level of uncertainty concerning whether the outcome will be positive or negative). This study defines software project risk as the plausibility of a negative outcome under the DempsterShafer theory of belief functions[9]. The notion of risk using plausibility is a more conservative measure of risk than the probability measure of a negative outcome. Second, regarding how to estimate the probability of a risks occurrence, existing methods only provide general suggestions. For example, the estimation can be obtained through discussions among relevant parties. Unfortunately, such discussions are quite limited for guiding users to properly estimate software project risk. This study utilizes evidential reasoning approach to provide objective support for the estimation of risks. Whenever the changes of strength of evidences are obtained, they are plugged into the model to update estimates. Third, the existing methods either focus on the graphical relationships among software project risk factors using flow charts or diagrams, or emphasize the quantitative computation of risk probabilities, but not both. The proposed approach consists of both the quantitative computation of risks and the graphical representation of relevant constructs through an evidential diagram.

probability associated with the answers to a related question. In the following paragraphs, we provide the mathematical definitions of three important functions, mvalues, beliefs, and plausibilities, under DempsterShafer theory that are relevant to the formulations developed in this paper. Suppose we have a decision problem with n possible elements or states of nature forming a mutually exclusive and collectively exhaustive set represented by {a1, a2, a3, . . ., an}. Call this entire set a frame represented by the symbol . In the belief function formalism, uncertainty is not only assigned to the single elements of the frame but also to all other proper subsets of the frame and to the entire frame . These uncertainties are called m-values, the basic belief assignment function. Similar to probabilities, all these m-values add to one:
A

m( A) = 1,

(2)

where A represents all the subsets of the frame , and m( ) = 0, that is, the m-value for the empty set is 0. The m-value pertaining to a statement measures the degree of belief directly assigned to the statement based on the evidence. Basically, belief in a statement represents the total belief that the statement is true. Mathematically, the belief function for a subset of elements A of a frame , is defined as the sum of all the m-values for the individual elements in the subset A, and the m-values for any subsets contained in the subset A. In terms of symbols:

Bel ( A) =

A B

m( B),

(3)

3. Evidential Reasoning Approach

The evidential reasoning approach under the DempsterShafer theory of belief functions has been widely used in a broad range of disciplines. An evidential reasoning approach to risk analysis is a process where several variables (assertions), when combined together, inform the analyst about a variable of interest, such as software project risk. The DempsterShafer theory of belief functions is a generalization of Bayesian theory of subjective probability [10]. The current form of belief function theory is based on the work of Dempster and Shafer. Under Bayesian theory, the answer to the question of interest is represented in terms of probability. That is, under Bayesian theory, we associate objective or subjective probabilities to the possible answers to the question of interest. Under DempsterShafer theory, the degree of belief that can be associated with the possible answers to a question of interest may depend on the

where B is any subset of A. For example, belief in the subset {a1, a2} is Bel({a1, a2}) = m(a1) + m(a2) + m({a1, a2}). Intuitively, the plausibility in a statement is the degree to which the statement is plausible in light of the evidencethe degree to which we do not disbelieve the statement. Mathematically, the plausibility function for a subset of elements A of a frame , is defined to be the maximum possible belief that could be assigned to A if all future evidence were in support of A. In terms of symbols, plausibility is defined as
Pl ( A) =
A B

m(B).

(4)

The plausibility function can also be defined in terms of belief function as

Pl ( A) = 1 Bel ( A). (5)

225

Authorized licensed use limited to: RMK Engineering College. Downloaded on August 03,2010 at 03:16:06 UTC from IEEE Xplore. Restrictions apply.

Assuming A is an assertion that software project risk is low, the plausibility of A , Pl ( A) , represents maximum possible risk that software project risk is high.

4. Evidential Reasoning Model for Software Project Risk Analysis

The process of developing an evidential reasoning model consists of four phases: specification of the model structure, assessment of evidence strength, determination of software project risks, and risk monitoring and analysis. And, the procedure of the model is given in Figure 1 (Risk treatment is not involved in the model discussed in this paper).
1 Specification of model structure

E1.1 Software size is small. directly pertains to assertion A1. Specification quality is good and thus it is connected to that assertion. The software project risk assessor can use existing qualitative methodologies, such as scenarios [11], and questionnaires to develop the evidential diagram. Although costly, methods such as Delphi techniques can be used to refine the completeness and accuracy of an evidential diagram.
A1. Specification quality is good. (0.978, 0.011)

E1.1. Software size is small. (0.9, 0)

E1.2. The complexity of software is low. (0.8, 0.1)

Figure 2. Hypothetical evidential diagram for the

2 Assessment of evidence strength

assertion Specification quality is good

4.2. Assessment of Evidence Strength

New Evidence 3 Determination of software project risks

4 Risk Monitoring and Analysis

Risk Database

Risk Treatment

Figure 1. Model procedure

4.1. Specification of Model Structure

An evidential diagram consists of assertions, evidence, and their interrelationships. Assertions include the main assertion and subassertions. The main assertion is the highest-level assertion; the subassertions are lower-level assertions. Relationships between assertions (e.g., between the main assertion and subassertions, and between higher-level subassertions and lower-level subassertions) need to be defined using logical relationships such as and and or. And evidence represents the information that supports or negates assertions. In Figure 2, the rounded boxes represent assertion nodes. And evidence nodes are represented by rectangular boxes in the evidential diagrams. Evidence nodes are connected to the corresponding assertion(s) that they directly pertain to. For instance, in Figure 1, the evidence

In this step, users assess strength of evidence, which indicates the level of support that a piece of evidence provides in favor of or against the assertion it pertains to. Strength of evidence is represented by m-values. In Figure 2, numbers in parentheses represent m-values or belief inputs provided by the software project risk assessor. The first m-value represents the level of support the evidence provides to the assertion it pertains to, and the second m-value represents the level of support for the negation of the assertion it pertains to. To avoid being heavily affected by individual subjective judgment, evidence strength can be elicited from multiple experts. Group meetings or Delphi techniques can be used to help achieve consensus among experts.

4.3. Determination of Software Project Risks

In this step, beliefs on assertions are computed by combining the strength of all the items of evidence (mvalues), based upon the models structure. If m1 and m2 are the m-values representing two independent items of evidence pertaining to a frame , then the combined m-values (basic belief assignment function) for a subset A of frame using Dempsters rule is given by m( A) = K 1 {m1 ( B1 )m2 ( B2 ) B1 B2 = A, A } , (6)

where K = 1 {m1 ( B1 )m2 ( B2 ) B1 B2 = } , which

represents the renormalization constant. The second term in K represents the conflict.

226

Authorized licensed use limited to: RMK Engineering College. Downloaded on August 03,2010 at 03:16:06 UTC from IEEE Xplore. Restrictions apply.

As figure 2, let us consider a hypothetical example where we have two independent items of evidence pertaining to a binary variable A with values a that the variable is true and a that it is not true. Suppose that we obtain the following beliefs in terms of m-values from the two items of evidence: Evidence 1: m1 (a)=0.9, m1 ( a )=0, m1 ({a, a })=0.1 Evidence 2: m2 (a)=0.8, m2 ( a )=0.1, m2 ({a, a })=0.1. We first cross multiply the two sets of m-values and collect all the resulting m-values and assign them to the common element of the cross product. Next, we renormalize these m-values by dividing by renormalization constant. The cross multiplication and renormalization yield the following m-values and K: K = 1 [m1 (a)m 2 (a ) + m1 (a )m 2 (a)]

= 1 [(0.9)(0.1) + (0.0)(0.8)] = 0.91 m(a) = [m1 (a)m2 (a) + m1 (a)m2 ({a, a}) + m1 ({a, a})m2 (a)] / K = [(0.9)(0.8) + (0.9)(0.1) + (0.1)(0.8)] / 0.91 = 0.978 m(a ) = [m1(a )m2 (a ) + m1(a )m2 ({a, a}) + m1({a, a})m2 (a )]/ K = [(0.0)(0.1) + (0.0)(0.1) + (0.1)(0.1)] / 0.91 = 0.011 m({a, a }) = [m1 ({a, a })m2 ({a, a })]/ K = [(0.1)(0.1)] / 0.91 = 0.011 So, the belief supporting the assertion that Specification quality is good is 0.978, the belief negating the assertion is 0.011, with 0.011 remaining as ambiguity. This suggests that the risk involved in specification quality is 0.022that is, the plausibility that specification quality is not good is 0.022.

Once the software project starts, a continuously monitoring loop starts. Whenever the changes of strength of evidences are obtained, they are plugged into the model to update estimates. A chronological record of such inputs and estimates are kept as a risk profile saved in the software project risk database. Ai represents an assertion in the evidential diagram. In the process of risk monitoring, if the average belief of the previous N time units of the profile records for Ai is under the threshold for it, i.e., AVG (Bel(Ai))N < threshold (Ai), the risk involved in Ai should be treated with. And then the risk sources will be traced. Since an evidential diagram can visually model cause consequence relations, the observed Ai can be traced to its descendant interactively with the user to identify the sources of the risk. The procedure of this phase is given in Figure 3. After the risk treatment, new evidences are plugged into the model to update previous estimates.

5. A Case study
The above model was verified by a financial management system developed by UFIDA Software Company. In this case, A1.Defects rate is low was the assertion observed. For risk monitoring, the threshold of A1 was set as 75% by expert experience.
A1. Defects rate is low. (0.820, 0)

4.4. Risk Monitoring and Analysis

2 Assessment of evidence strength

E1.1.1. Efforts estimation is accurate. (0.8, 0)

&

A1.1. Schedule pressure is small. (0.910, 0)

A1.2. Specification quality is good. (0.880, 0)

3 Determination of software project risks Risk Database AVG (Bel(Ai))N< threshold (Ai) Yes Tracing the Sources of A New Evidence Risk Treatment E1.1.1.1. Expertise level is high. (0.9, 0) E1.1.1.2. Requirements change is not frequent. (0.7, 0) No A1.1.1. Productivity is high. (0.926, 0) E1.2.1. Software size is small. (0.6, 0) E1.2.2. The complexity of software is low.(0.9, 0)

Figure 4. Evidential diagram for the assertion

Figure 3. Risk monitoring and analysis

Defects rate is low

227

Authorized licensed use limited to: RMK Engineering College. Downloaded on August 03,2010 at 03:16:06 UTC from IEEE Xplore. Restrictions apply.

Based upon the process suggested previously, an evidential diagram (see Figure 4) for the Defects rate is low assertion was developed. It consists of one main assertion, two second-level assertions (subassertions). Belief inputs (m-values) on strength of evidence were elicited from project experts. The following fuzzy metric was also provided to facilitate the input of the strength of evidence: very low strength = [00.1); low strength = [0.10.3); median strength = [0.30.7); high strength = [0.70.9); very high strength = [0.91.0]. DELIEF software [12] was used to aggregate m-values form all the items of evidence and to compute the risk of defects rate, which is the plausibility that defects rate is high. Based upon our evidential reasoning model with the and relationship and strength of evidence assessments provided by expert experience, the overall belief that the Defects rate is low is calculated to be 0.820, 0. This means that project manager has 82 percent confidence that the overall assertion Defects rate is low is true and has zero belief that the assertion is not true since no negative evidence was obtained with respect to this assertion. With the changes of strength of evidences, the model kept monitoring the assertion A1. After a period of time, it was found that AVG (Bel(A1))N < threshold (A1). Based on the evidential diagram, the project manager traced the potential source of this problem to be the evidence E1.1.1.2. Requirements change is not frequent. And then, risk treatment was performed to reduce the defects rate. This case shows that it is possible to disclose potential software project risks using the evidential reasoning model.

References
[1] C. F. Fan, Y. C. Yu, BBN-based software project risk management, Journal of Systems and Software, Elsevier Inc., New York, vol. 73, January 2004, pp. 193-203 [2] G. Klein, J. J. Jiang, Seeking consonance in information systems, Journal of Systems and Software, Elsevier Inc., New York, vol. 56, March 2001, pp. 195-202 [3] B. W. Boehm, Project termination doesnt equal project failure, IEEE Computer, IEEE Computer Society, vol. 33, September 2000, pp. 94-96 [4] K. Padayachee, An interpretive study of software risk management perspectives, Proc. 2002 Annual Research Conf. the South African Institute of Computer Scientists and Information Technologists, South Africa, 2002, pp. 118-127 [5] K. Bandyopadhyay, P.P. Mykytyn, and K. Mykytyn, A framework for integrated risk management in information technology, Management Decision, MCB University Press, UK, vol. 37, 1999, pp. 437-444 [6] C. G. Pan, Y. W. Chen, and H. Wang, Overview of the study on theories and methods of software project risk management, Control and Decision, Northeastern University, China, vol. 22, May 2007, pp. 481-486+493 [7] S. Alter, S. Sherer, A general, but readily adaptable model of information system risk, Communications of the Association for Information Systems, Association for Information Systems, Atlanta, vol. 14, 2004, pp. 128. [8] R. Fairley, Risk Management for Software Projects, IEEE Software, IEEE Computer Society, vol. 11, May 1994, pp. 57-67 [9] L. L. Sun, R. P. Srivastava, and T. J. Mock, An information systems security risk assessment model under the Dempster-Shafer theory of belief functions, Journal of Management Information Systems, Elsevier Inc., New York, vol. 22, Spring 2006, pp. 109-142 [10] G. Shafer, Perspectives on the theory and practice of belief functions, International Journal of Approximate Reasoning, Elsevier Inc., New York, vol. 4, 1990, pp. 323-362 [11] W. M. Han, S. J. Huang, An empirical analysis of risk components and performance on software projects, Journal of Systems and Software, Elsevier Inc., New York, vol. 80, January 2007, pp. 42-50 [12] D. Zarley, Y. T. Hsia, and G. Shafer, Evidential reasoning using DELIEF, Proc. 7th national conference of artificial intelligence, AAAI Press, MN, USA, 1988, pp. 205-209

6. Conclusions
This paper proposed a software project risk analysis model based on evidential reasoning approach. The main limitations with existing approaches for software project risk analysis were first discussed. We then presented the process of developing a software project risk analysis model. Finally, we demonstrated and validated the model via a case study. The model provides a rigorous, structured manner to incorporate relevant software project risk factors, and their interrelationships when estimating software project risks. Of course, the research has its limitations. First, we are not able to provide much empirical evidence of the advantages of this approach over others. Thus, field research is needed to empirically compare our approach with other methods. Second, clearly, the modeling approach should be tested in other practice situations in addition to the financial management system discussed in this paper.

228

Authorized licensed use limited to: RMK Engineering College. Downloaded on August 03,2010 at 03:16:06 UTC from IEEE Xplore. Restrictions apply.