Modeling Software Quality with Classification

Trees

265

and expense wasted trying to enhance a module that is already not fault-prone. Let us consider the cost of a Type I misclassification to be a "cost unit", and let us suppose that there is a benefit of 100 units for identifying a fault-prone module at the beginning of testing compared to waiting until a fault is discovered in that module by a customer. Even though a cost ratio of 100:1 is only an illustration here, we believe that costs associated with customer-discovered faults for telecommunications systems are so high that even more than 100:1 is plausible. Each project must determine its own value of the cost ratio. We define the Cost of using a model as the extra effort entailed beyond the budgeted costs of normal development. Our focus is on using a model, and thus, we assume the costs of building a model, and data collection are already budgeted; we do not include them in the cost-benefit analysis. An expanded definition of Cost is a topic for future research. We define Benefit as the benefit due to using the model. From these quantities we can calculate Profit and return on investment, ROI. Profit = Benefit - Cost _
R01
T

(5)
(6

Profit

= c^T

>

The cost of enhancements when acting on this model's predictions is 295 units. The benefit of using the quality model is 5,300 = 53 x 100 units for identifying some fault-prone modules correctly. Figure 7 shows profit as a function of c for several values of Cu/Ci-h For our example, a hypothetical cost ratio of C\\/C\ = 100, profit is maximized when all modules are enhanced, i.e., when c is so large that the Type II misclassification rate is zero. In other words, the value of discovering faults early, which would otherwise be discovered by customers, is so high that it pays to enhance as many modules as possible. However, it is usually not practical to enhance all modules. In this example, we supposed that 29.5% of the modules is the practical limit, and thus, the profit of using the model is 5,005 = 5, 300 295 units. Figure 8 shows return on investment (ROI) as a function of c for several values of Cn/Ci. For small values of c, the Type II misclassification rate is so high that very few modules were classified as fault-prone (see Fig. 2), and therefore ROI is also high. However, a model based on a low value of c finds so few fault-prone modules that it is not very useful to the project. At a value of c = 0.95, the return on investment is ROI = 17.0 = 5,005/295. Most investors would consider this an excellent return on investment. For comparison, suppose we randomly select 295 modules for reliability enhancement. We expect 22 = (0.074) (295) modules to be actually fault-prone. Direct cost is 295 units. The benefit is 2, 200 = 22 x 100 units. The profit is 1,905 = 2, 200-295 units. The return on investment is ROI = 6.5 = 1,905/295. Thus, using the model almost triples the ROI of randomly selecting the same number of modules.
The figure uses a logarithmic scale for profit.