Eclipse Bug Predection Dataset

The process of development of base-level prediction models, extraction of

knowledge, and
construction of the ensemble-based prediction system is illustrated using the
dataset from
the Eclipse Bug Predection Dataset. The details
regarding the origin of the data and their attributes are presented next. For the
experimental
purposes, the dataset has been divided into three sets: a training set containing
data points
that are used to develop base-level data models, a validation set containing data
that are used
to validate confidence in rules extracted from the models, and a testing set that
contains data
points used to evaluate prediction capabilities of the models.3

The original dataset consists of 211 bugs reports dealing with 143 bugs, which were
isolated and corrected during development. The dataset also includes 253 records describing
each component in the project.
a number of components that have to be examined during elimination
of a bugs; and the effort needed to correct a defect. The modified dataset contains only
data related to components with bugs. There are 189 data points.
The data points have been divided into three sets: a training
set of 32 data points, a validation set containing 43 data points, and a testing set which
has
63 data points.

Input

Type of Bug Incorrect_Parameters,

Design_defect,
Clerical_mistakes,
Others
Phases of bugs Inialization of System
Starting Inventory,
Program Execution,
On_Calculation

Line of Codes <0,2562>

Output
 Result System Failure
Compunents examoimed Single,
Multipal

The second point of interest is related directly to a process of elimination of defects. In this
case, the attention is put on the time needed to remove a single defect. The attributes of data
points used in this case are shown in Table 7. As it can be seen, the attributes chosen here
are almost identical to the ones from the previous investigation. The only difference is that
the attribute Number of Examined Components has been added to the first group of INPUT
attributes. This time, the OUTPUT attribute is Effort needed to Eliminate a Defect

The first step in data analysis and modeling is dedicated to the development of base-level
data models that can be used for prediction purposes. The base-level models used here
are rule-based models (see the Rule-Based Models and Ensemble System section). Three
methods of building these models are selected in order to illustrate the idea of utilization of
very different model development techniques. The first rule-based model is generated using
See5/C5.0 (RuleQuest). The See5/C5 tool is an updated version of the well-known C4.5
algorithm (Quinlan, 1993). It has the capability of generating classifiers that are expressed
as decision trees or sets of IF-THEN rules. The commercially-available tool called 4cRule-
Builder (4cData) is used to build the second data model. This tool uses supervised learning
techniques to generate a data model from discrete numerical or nominal data, and has built-in
discretization schemes for continuous attributes

Components Examined During Elimination of Bugs

Three base-level prediction models have provided the prediction
rates that are as follow

See5/C5.0
4cRuleBuilder
GAGP

Validation Process
The bbm values assigned to the rules are presented in Table 9. The table contains only the
rules that are best after the development stage (with the highest values of bbmT) and the ones
that are the best after the validation stage (with the highest value of bbmUPDATE).
The rules generated by See5/C5.0 that are the best after the development process are also the
best after the validation process (see bold entries in Table 9). For the rules NC_See5_c1_15
and NC_See5_c_2, their bbm values have increased after the validation phase. As it can be
seen on the basis of other rules, none of the updated bbm values is smaller that the original
one (there are some cases where bbm values do not change—this means that a rule has not
been satisfied by any validation data point).
The same validation process has been performed on rules generated by 4cRuleBuilder.
In the case of two rules NC_4cRB_c1_1 and NC_4cRB_c2_3, their bbm values have increased
and the rules stay among the best ones. For the other two rules—NC_4cRB_c1_1
and NC_4cRB_c2_2—their bbm values have not changed. However, the bbm values of
the other two rules NC_4cRB_c1_2 and NC_4cRB_c2_1 have increased and have become
higher than the bbm values of NC_4cRB_c1_1 and NC_4cRB_c2_2. As the result, a set of
best rules has been modified.
The validation process performed on rules generated by GAGP method has brought a single
change in the category—multiple components examined (see Table 9). The rule NC_GAGP_
c2_3 has been replaced by NC_GAGP_c2_7, which has the higher value of bbm. The bbm
values of the rules from the category single component examined have improved.

Analysis of Best Rules

For the category single component examined, the range of bbmUPDATE values is from
0.950 to 0.988. The rules with highest bbmUPDATE values are generated by 4cRuleBuilder:
Rule: NC_4cRB_c1_5:
if DEFECT_TYPE is Multiple_Design_Error or Clerical_Error
then SINGLE COMPONENT is examined
Rule: NC_4cRB_c1_9:
if DEFECT_TYPE is Clerical_Error
then SINGLE COMPONENT is examined

Another pair of rules that draws attention is the pair containing the rules NC_4cRB_c1_2
and NC_GAGP_c1_2 (see the following rules). The rule generated by GAGP method uses
a subset of values of the attribute defect type that are used by the rule NC_4cRB_c1_2. Also
in the case of the attribute lines of code, there is some substantial overlapping. Overall, the
rule generated by 4cRuleBuilder seems more specific. Once again, high values of bbmUPDATE
make these rules good candidates for prediction activities.

Bug elimination
The prediction rates for the models representing effort needed to eliminate a defect are
presented in Table 10. In this case, the highest prediction rate is obtained using the model
generated by GAGP method.