See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221556012

Evaluation of mutation testing for object-oriented programs

Conference Paper · January 2006

DOI: 10.1145/1134285.1134437 · Source: DBLP

CITATIONS READS

33 107

3 authors, including:

Mary Jean Harrold

Georgia Institute of Technology
188 PUBLICATIONS   11,038 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Regression Test Selection View project

Test Augmentation View project

All content following this page was uploaded by Mary Jean Harrold on 28 May 2014.

The user has requested enhancement of the downloaded file.

 Evaluation of Mutation Testing
for Object-Oriented Programs
Yu-Seung Ma Mary Jean Harrold
Electronics and Georgia Institute of
Telecommunication Research Technology
Institute, Korea U.S.A.
ysma@etri.re.kr harrold@cc.gatech.edu
ABSTRACT
The effectiveness of mutation testing depends heavily on the
types of faults that the mutation operators are designed to
represent. Thus, the quality of the mutation operators is
key to mutation testing. Although, mutation operators for
object-oriented languages have previously been presented,
little research has been done to show the usefulness of the
class mutation operators. To assess the usefulness of class
mutation operators, we conducted two empirical studies. In
the first study, we examine the number and kinds of mutants
that are generated for object-oriented programs. In the sec-
ond study, we investigate the way in which class mutation
operators model faults that are not detected by traditional
mutation testing. We conducted our studies using a well-
known object-oriented system, BCEL.
Categories and Subject Descriptors
D.2.5 [Software Engineering]: Testing and Debugging
General Terms
Experimentation
1. INTRODUCTION
Object-oriented (OO) programming has many useful fea-
tures, such as information hiding, encapsulation, inheritance,
polymorphism, and dynamic binding. Although these OO
features enable developers to construct systems in a more
systematic and flexible way, they introduce new kinds of
faults [9]. To detect these faults, researchers have proposed
approaches that apply mutation testing to OO features [2,
3, 4, 6].
Mutation testing [5] is a fault-based testing technique that
measures the effectiveness of test suite. Faults are intro-
duced into the program by creating a set of faulty versions,
called mutants. These mutants are created from the origi-
nal program by applying mutation operators, which describe
syntactic changes to the programs. Test cases are used to
Copyright is held by the author/owner.
ICSE’06, May 20–28, 2006, Shanghai, China.
ACM 1-59593-085-X/06/0005.
869
Yong-Rae Kwon
Korea Advanced Institute of
Science and Technology
Korea
kwon@cs.kaist.ac.kr
execute these mutants with the goal of causing each mutant
to produce incorrect output.
To apply mutation testing to object-oriented programs,
researchers have adapted existing mutation operators, which
were developed for procedural-language programs, to OO
programs. Researchers have also developed additional mu-
tation operators, called class mutation operators, to detect
OO specific faults. However, little research has been done to
show the usefulness of the class mutation operators. For ex-
ample, there is no evidence that the class mutation operators
generate realistic faults or that they generate an acceptable
number of mutants.
To address these issues, we conducted a set of empirical
studies using an object-oriented mutation system, MuJava
[8]. The objectives of our studies are (1) to examine how
many mutants and what kinds of mutants are generated for
object-oriented programs and (2) to investigate how many
class mutation operators model faults that are not detected
with traditional mutation testing.
The next section briefly describes the mutation operators
and the mutation tool used for our study. Then, Section 3
presents the empirical study. Section 4 presents our conclu-
sions and future work.
2. OO MUTATION SYSTEM
This section briefly describes the mutation operators and
the mutation tool used for our studies.
2.1 Mutation Operators
There are two types of mutation operators for OO lan-
guages: (1) those adapted from procedural languages and
(2) those developed to handle OO-specific features. We refer
to these mutation operators as traditional mutation opera-
tors and class mutation operators, respectively. We imple-
mented both types in our mutation tool and used them for
our study.
Traditional mutation operators. Due to the consid-
erably high execution costs of mutation testing, researchers
have proposed a selective mutation technique, which uses a
subset of the mutation operators instead all mutation op-
erators. For traditional mutation operators, five selective
mutation operators, listed in Table 1, have been shown em-
pirically to provide almost the same effectiveness as using
the entire set of mutation operators, with cost reduction,
for the programs used in the study, of at least four times
that of using the entire set [7]. Thus, we used this set of
mutation operators for our studies.
 Operator Description gram of significant size and it uses OO features extensively.
ABS Absolute value insertion
Also, our familiarity with BCEL lets us generate a mutation-
AOR Arithmetic operator replacement
LCR Logical connector replacement adequate test suite for it.
ROR Relational operator replacement Except for special kinds of classes—abstract and interface—
UOI Unary operator insertion BCEL is composed of 266 Java classes. Table 3 summarizes
the programs in terms of the number of BCEL’s classes and
Table 1: Selective Traditional Mutation Operators the ratio of the number of classes of that category to the
total number of classes.
Class mutation operators. No research has been done
to develop a selective mutation technique for class mutation Program Size # of Classes Percentage
operators. Thus, we implemented the entire set of class 0 ∼ 50 lines 171 64.28%
mutation operators, which are listed in Table 2. Reference 50 ∼ 100 lines 44 16.54%
[6] presents a detailed description of these operators. 100 ∼ 200 lines 28 10.53%
200 ∼ 400 lines 13 4.89%
Operator Description More than 400 lines 10 3.76%
IHD Hiding variable deletion Total 266 100%
IHI Hiding variable insertion
IOD Overriding method deletion Table 3: Program Size of BCEL Classes
IOP Overridden method calling position change
IOR Overridden method rename
ISK super keyword deletion 3.2 Study 1: Distribution of Mutants
IPC Explicit call of a parent’s constructor deletion One of the barriers to the practical use of mutation testing
PNC new method call with child class type is the unacceptable computational expense of generating and
PMD Instance variable declaration with parent class type running enormous numbers of mutants. In this study, to
PPD Parameter variable declaration with child class type
approximate the execution cost of the mutation testing, we
PRV Reference assignment with other compatible type
OMR Overloading method contents change
examine the number and kinds of mutants generated for
OMD Overloading method deletion BCEL.
OAO Argument order change
OAN Argument number change 3.2.1 Experimental Setup
JTD this keyword deletion For the 266 classes of BCEL, we generated two sets of
JSC static modifier change mutants by applying (1) class mutation operators of Table 2
JID Member variable initialization deletion
and (2) traditional operators of Table 1. We then analyzed
JDC Java-supported default constructor create
EOA Reference and content assignment replacement
the distribution of the mutants for each set. In particular,
EOC Reference and content assignment replacement for the traditional mutation operators, we examine whether
EAM Accessor method change mutation testing of BCEL generates numbers of mutants
EMM Modifier method change that are similar to those generated for procedural programs.
Note that our analyses of traditional and class mutants are
Table 2: Class Mutation Operators conducted with identical data, 266 BCEL classes, and the
mutation tool, MuJava.

2.2 Mutation Tool 3.2.2 Result and Analysis

For object-oriented programs, there are few mutation tools.
As yet, only MuJava is a publicly available object-oriented
mutation tool. MuJava (Mutation System for Java) [8],
which we developed, is a mutation system that supports
the entire mutation process for Java programs. MuJava im-
plements both traditional mutation operators (adapted for
OO programs) and class mutation operators. For the oper-
ators, it automatically generates mutants, runs the mutants
against a suite of tests, and reports the mutation score of
the test suite.
3. EMPIRICAL STUDIES
Using MuJava, we conducted two empirical studies. The
first study investigates the distribution of mutants for object-
oriented programs and the second study investigates the ef-
fectiveness of class mutation operators. This section first de-
scribes the subject we used for our studies, and then presents
the two studies.
3.1 Experimental Data
We used the BCEL (Byte Code Engineering Library) [1]
system, an open source project implemented in Java, for
our studies. We chose BCEL because it is widely-used pro-
Mutants from Class Mutation Operators
Table 4 shows information about mutants generated from
class mutation operators. For procedural programs, the
number of generated mutants is typically large. For exam-
ple, 951 mutants are generated for commonly studied 30-line
triangle classification program TRITYP [7]. However, in
BCEL, 2996 mutants are generated for 266 classes and the
average number of mutants per class is 11.35. This illus-
trates that, for our subject program, the number of mutants
generated from class mutation operators is small, and thus
does not impose a high execution cost.
One interesting result is that each class mutation opera-
tor is applied to a small number of classes. Table 4 shows
that most operators are applied to approximately 10% of
the classes. Only the IPC operator is applied to more than
half of the classes. However, when considering all class mu-
tation operators, most classes (97%) use at least one class
mutation operator. This result suggests that class mutation
operators are not dependent on each other, and that the
BCEL system uses object-oriented features diversely.
Mutants from Traditional Mutation Operators
Table 5 shows the results of applying traditional mutation
operators. The total number of mutants is seven times the

870
 Total # Average # Applied # 3.3.1 Experimental Setup
Operator of mutants of mutants of classes
IHD 0 0 0(0%) Among the 266 classes, we chose Java classes that gener-
IHI 96 0.36 48(18%) ated the maximum number of mutants for each class muta-
IOD 73 0.28 27(10%) tion operator. In this study, we did not consider four muta-
IOP 44 0.17 32(12%) tion operators, IHD, ISK, PMD and PPD, because they do
IOR 74 0.28 27(10%) not generate any mutants for our subject (see Section 3.2).
ISK 0 0 0(0%) We selected 11 classes; information about them is described
IPC 173 0.66 153(58%) in Table 6. The numbers of traditional mutants and class
PNC 243 0.92 12(5%) mutants for each program are listed in Table 7.
PMD 0 0 0(0%)
For each class, we generated a test suite that kills all tra-
PPD 0 0 0(0%)
PRV 307 1.16 34(30%) ditional mutants. We then ran that test suite against class
OMR 59 0.22 14(5%) mutants, and determined the number of class mutants that
OMD 1 0 1(0.4%) were killed by the test suite.
OAN 57 0.22 21(8%)
JTD 209 0.80 66(25%) 3.3.2 Result and Analysis
JSC 20 0.08 11(4%) Table 8 shows the killed rate of the class mutants of the
JID 110 0.42 33(13%) 11 classes. Over 50% of class mutants are killed by the test
JDC 106 0.40 106(40%)
suite. IPC, PNC, OMD, EAM and EMM operators show a
EOA 4 0.02 2(0.8%)
EOC 2 0.01 2(0.8%)
killed rate greater than 95%. This high killed rate means
EAM 1031 3.90 53(20%) that these mutant operators may not be useful in mutation
EMM 387 1.47 8(3%) testing of OO programs because they model the easily de-
all 2996 11.35 255(97%) tectable faults. Conversely, some mutation operators such
as EOA and EOC shows 0% killed rate. They can be consid-
Table 4: Class Mutants of BCEL Classes ered to model OO faults that are difficult to detect. There-
fore, they can be thought as good mutation operators for
Total # Average # Applied # OO programs.
Operator of mutants of mutants of classes Although the study uses a small number of sample classes
ABS 5939 22.33 123(47%) and does not consider the entire set of class mutation op-
AOR 738 2.77 53(20%) erators, it shows that some class mutation operators model
LCR 484 1.82 39(15%) faults that can be detected by traditional mutation testing.
ROR 3831 14.40 84(32%)
UOI 8634 32.46 129(19%)
all 19626 73.89 135(51%) 4. CONCLUSION AND FUTURE WORK
This paper presents the results from two empirical studies
Table 5: Traditional Mutants of BCEL Classes of mutation testing with a real object-oriented system. The
first study shows that the number of class mutants is rela-
tively small compared to traditional mutation. This small
number of class mutants, but less than the number of mu- number can promote the application of mutation testing
tants generated for procedural programs. Also, only 51% of to object-oriented programs. The second study shows that
classes generate mutants from at least one traditional muta- some class mutation operators model faults that are detected
tion operator. This occurs because those BCEL classes are easily by tradition mutation testing. Thus, when creating
implemented without any arithmetic or relational operators. selective mutation operators for object-oriented programs,
Additionally, the result shows a distribution that is differ- these mutation operators can be omitted.
ent from procedural programs. In procedural programs, the However, our study used one sample program and did
number of mutants from AOR is greater than from ROR, not consider mutation operators for exception-handling. In
and the number of mutants from ABS is greater than from future work, we will perform experimentation on more sub-
UOI. However, OO programs (based on our subject) show jects, implement mutation operators for exception handling,
an opposite result. and determine a set of selective mutation operators for class
3.3 Study 2: Effectiveness of Class Mutation mutation operators.
Operators
Although object-oriented features introduce new kinds of
5. REFERENCES
faults, to be detected, these faults may not require a special [1] BCEL : Byte Code Engineering Library WWW page.
testing method; some faults may be detected easily with any [2] R. T. Alexander, J. M. Bieman, S. Chosh, and B. Ji.
testing method or some faults may detected with a testing Mutation of Java objects. In 13th International
method adapted from traditional testing. In the former case, Symposium on Software Reliability Engineering, pages
mutation operators generate easily killed mutants. In the 341–351, November 2002.
latter case, class mutation operators generate mutants that [3] P. Chevalley and P. Thévenod-Fosse. A mutation
can be killed by a test suite generated by traditional muta- analysis tool for Java programs. Journal on Software
tion method. Mutation operators that model these kinds of Tools for Technology Transfer (STTT), pages 1–14,
faults are useless because they raise execution cost without December 2002.
increasing test quality. In this empirical study, we examine [4] S. Kim, J. Clark, and J. McDermid. Class mutation:
whether class mutation operators generate mutants that are Mutation testing for object-oriented programs. OOSS:
killed by the traditional mutation method. Object-Oriented Software Systems, October 2000.

871
 Class Line # Constructor # Var. # Method # Parent #
p1 org.apache.bcel.classfile.Field 65 3 0 5 2
p2 org.apache.bcel.generic.BranchHandle 79 1 2 9 1
p3 org.apache.bcel.verifier.statics.StringRepresentation 190 1 2 14 1
p4 org.apache.bcel.verifier.statics.Pass2Verifier 998 7 5 13 1
p5 org.apache.bcel.generic.ConstantPoolGen 404 3 13 37 0
p6 org.apache.bcel.classfile.LocalVariable 145 3 6 18 0
p7 org.apache.bcel.util.ClassPath 256 2 6 13 0
p8 org.apache.bcel.generic.InstructionList 558 5 5 76 0
p9 org.apache.bcel.classfile.JavaClass 464 2 22 52 1
p10 org.apache.bcel.generic.CodeExceptionGen 118 1 4 13 0
p11 org.apache.bcel.verifier.structurals.LocalVariables 95 1 1 10 0

Table 6: Description of the 11 Experimental Classes

Operator p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 total

ABS 10 8 0 398 302 70 26 94 160 12 66 1146
traditional AOR 0 0 0 4 4 0 0 28 10 4 0 50
mutation LCR 0 0 0 92 0 0 2 6 4 4 14 122
operators ROR 16 1 1 297 127 0 21 84 151 7 46 751
UOI 41 6 0 461 396 100 40 150 311 0 132 1637
total 67 25 1 1252 829 170 89 362 636 25 258 3705
IHI 5 2 0 0 0 0 0 0 1 0 0 8
IOD 0 5 32 0 1 1 3 1 0 2 2 47
IOP 0 2 0 0 0 0 0 0 0 0 0 2
IOR 0 5 32 0 1 1 3 1 0 2 2 47
IPC 3 0 0 0 0 0 0 0 0 0 0 3
PNC 0 0 0 83 40 0 0 0 0 0 0 123
PRV 0 3 0 1 35 40 0 0 0 6 0 85
class OMR 0 0 1 0 0 0 8 32 0 0 1 42
mutation OMD 0 0 0 0 0 0 0 0 1 0 0 1
operators OAN 0 0 0 0 0 2 12 0 0 0 1 15
JTD 0 0 0 1 0 12 1 0 0 5 2 21
JSC 0 0 0 0 5 0 3 0 0 0 0 8
JID 0 1 0 4 8 0 0 0 0 0 0 13
EOA 0 0 0 0 3 0 0 0 0 0 0 3
EOC 0 0 0 0 0 0 0 0 0 0 1 1
EAM 3 0 0 179 38 22 6 0 0 0 0 248
EMM 0 0 0 0 0 0 0 0 0 12 0 12
total 15 18 65 268 139 78 36 34 2 27 9 691

Table 7: Number of Mutants of the 11 Experimental Classes

[5] R. J. Lipton, R. A. DeMillo, and F. G. Sayward. Hints

Total # Equivalent # Killed # on test data selection: help for the practicing
Operator of mutants of mutants of mutants programmer. IEEE Computer, 11(4):34–41, November
IHI 8 0 3(38%) April.
IOD 47 0 11(23%)
[6] Y. S. Ma, Y. R. Kwon, and J. Offutt. Inter-class
IOP 2 1 0(0%)
IOR 47 0 11(23%) mutation operators for Java. Proceedings of 13th
IPC 3 0 3(100%) International Symposium on Software Reliability
PNC 123 0 123(100%) Engineering, November 2002.
PRV 85 0 67(79%) [7] A. J. Offutt and S. D. Lee. An empirical evaluation of
OMR 42 0 24(57%) weak mutation. IEEE Transactions on Software
OMD 1 0 1(100%) Engineering, 20(5):337–344, May 1994.
OAN 15 0 6(40%)
[8] A. J. Offutt, Y. S. Ma, and Y. R. Kwon. An
JTD 21 2 14(74%)
JSC 8 0 6(75%) experimental mutation system for java. ACM
JID 13 0 11(85%) SIGSOFT Software Engineering Notes (SECTION:
EOA 3 0 0(0%) Workshop on empirical research in software testing
EOC 1 0 0(0%) papers), 29(5), Sep. 2004.
EAM 248 0 240(97%) [9] J. Offutt, R. Alexander, Y. Wu, Q. Xiao, and
EMM 12 0 12(100%) C. Hutchinson. A fault model for subtype inheritance
total 691 3 349(51%) and polymorphism. In Proceedings of the 12th
International Symposium on Software Reliability
Table 8: Killed Rate of Class Mutants for the 11 Engineering, pages 84–93, November 2001.
Experimental Classes

872

View publication stats