See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/324685240

Jaguar: a spectrum-based fault localization tool

for real-world software

Conference Paper · April 2018

DOI: 10.1109/ICST.2018.00048

CITATIONS READS

0 14

5 authors, including:

Marcos Lordello Chaim Higor Amario de Souza

University of São Paulo University of São Paulo
33 PUBLICATIONS 119 CITATIONS 5 PUBLICATIONS 4 CITATIONS

SEE PROFILE SEE PROFILE

Fabio Kon
University of São Paulo
267 PUBLICATIONS 2,985 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

POKE-TOOL View project

InterSCity - Enabling the Future Internet for Smart Cities View project

All content following this page was uploaded by Higor Amario de Souza on 22 April 2018.

The user has requested enhancement of the downloaded file.

 Jaguar: a spectrum-based fault localization tool for
real-world software
Henrique L. Ribeiro, Roberto P. A. de Araujo, Marcos L. Chaim
Software Analysis and Experimentation Group — SAEG
School of Arts, Sciences and Humanities
University of São Paulo
{henriquelemos,roberto.araujo,chaim}@usp.br
Abstract—We present the Jaguar fault localization tool, which
uses Spectrum-based Fault Localization techniques (SFL) to
indicate faulty code excerpts. Jaguar supports both control-
flow and data-flow spectra. It also provides visualization of lists
containing suspicious program elements that are more likely to
be faulty. These lists can be used to inspect suspicious methods,
lines, and variables to locate bugs. Although data-flow spectrum
provides more comprehensive information than control-flow, it
is rarely used in SFL due to its high execution costs. Jaguar
uses the ba-dua coverage tool, which gathers data-flow spectrum
using a lightweight approach. Thus, Jaguar can be used in large
programs at affordable execution costs. Jaguar is an open source
tool for Java programs, which is available as an Eclipse plug-in
and as a command line tool.
Index Terms—Fault localization; Spectrum-based; Debugging;
Tool; Structural testing; Data-flow; Control-flow; Tests
I. I NTRODUCTION
Testing and debugging are among the most expensive tasks
during the development cycle [1]. Debugging consists of
locating and fixing program bugs (faults). These activities are
accomplished with the help of static (e.g., source code, bug
reports) and dynamic information (e.g., variable values, test
results). Developers may spend too much time to locate bugs.
Indeed, fault localization is still a manual, time-consuming and
error-prone task until today [2].
Spectrum-based fault localization (SFL) is a promising auto-
mated debugging approach due to its relatively low overhead in
test execution time. SFL techniques use data collected during
a test suite execution, called program spectra, to infer which
source code elements are more likely to contain a bug [3],
[4]. Program spectra comprise program elements1 such as
statements, blocks, predicates, and definition-use associations
(duas) that are associated with test cases.
SFL techniques use ranking metrics to pinpoint faulty
program entities [3], [5], [6]. In general, these metrics calculate
the frequency that each entity is executed in failing test cases
and divides it by the frequency that this entity is executed
in all test cases. Entities that are more executed in failing
test cases are more likely to be faulty. The output of an SFL
This project received financial support from the São Paulo Research
Foundation (FAPESP), processes 2013/24992-2 and 2014/23030-5.
1 We use the terms the terms element and entity indistinctly in this paper.
Higor A. de Souza, Fabio Kon
Department of Computer Science
Institute of Mathematics and Statistics
University of São Paulo
{hamario,kon}@ime.usp.br
technique is a list of program entities sorted in descending
order of suspiciousness.
Most SFL techniques use only control-flow spectra [5], [7]–
[9], such as statements and branches. Data-flow spectrum may
improve SFL since data-flow subsumes control-flow and, thus,
can provide more detailed information to locate faults. Indeed,
the few studies that addressed the use of data-flow spectrum
for fault localization suggested that data-flow spectra perform
better than control-flow spectra [4], [10].
There is a reason why most SFL studies use only control-
flow spectra: cost. A program’s source code or object code
is instrumented to obtain the spectra for fault localization,
which leads data-flow to have a higher run-time overhead
compared to control-flow. Recent results, though, show that
dua spectrum can be obtained with moderate run-time and
memory overheads [11], [12]. They allow the use of data-flow
spectrum in SFL for large and long-running programs.
Few tools provide information to support fault localization,
as shown in [13]. Moreover, most of the available tools use
only control-flow spectrum.
This paper presents the Jaguar open source tool [14]. Jaguar
implements SFL techniques based on data- and control-flow
spectra. We assessed the tool in real open source programs,
which sizes vary from 10 to 96 KLOC. The results show
that Jaguar can be used to support fault localization on
those programs at an affordable execution cost. Moreover, we
assessed the tool in a user study [15] and the participants
deemed the tool as useful for their debugging activities.
In what follows, we present the main features of the Jaguar
tool (Section II), how we assessed it and its limitations
(Section III). We describe and compare Jaguar to related tools
(Section IV) and draw our conclusions (Section V).
II. JAGUAR
Jaguar (JAva coveraGe faUlt locAlization Ranking)2 3 [14]
is an open source SFL tool for Java. Jaguar is composed of
Jaguar Runner, which collects and generates lists of suspi-
cious program elements, and Jaguar Viewer, which presents
the visual information for debugging. Jaguar supports both
2 github.com/saeg/jaguar
3A video demonstration of Jaguar is available at youtu.be/j4l1Mh56g5o.
 Jaguar
Jaguar Runner Jaguar Viewer
Lines
Methods
Program Control-flow Jaguar
JaCoCo
under test spectra list Duas
CH+Lines
JUnit tests ba-dua
Data-flow Ranking
spectra metric CH+Duas
Methods+Lines
Methods+Duas
Fig. 1. Jaguar’s architecture.
control-flow and data-flow spectra (i.e., source code lines and
definition-use associations, respectively). Figure 1 shows the
Jaguar’s architecture.
A. Jaguar Runner
Jaguar Runner collects control-flow spectrum using the
JaCoCo coverage tool (eclemma.org/jacoco). To collect data-
flow spectra, Jaguar uses the ba-dua coverage tool (github.com/
saeg/ba-dua) [12]. The ba-dua tool uses the Bitwise Algorithm
(BA) [11], which tracks duas at low execution costs by using
inexpensive data structures. BA enables the tracking of only
those duas that are potentially covered at each node visited
during a program execution. Jaguar invokes unit tests of a
subject program and collects spectra data for each element
covered during the test execution. It iterates over the classes,
methods, and then lines or duas.
Jaguar then calculates the suspiciousness score of each
element (line or dua) according to a chosen ranking metric.
There are ten ranking metrics available for the suspiciousness
calculation: DRT [16], Jaccard [17], Kulczynski2 [6], McCon
[6], Minus [18], Ochiai [17], Op [19], Tarantula [3], Wong3
[20], and Zoltar [21]. Each metric has its range of suspicious-
ness scores. Jaguar normalizes the scores of all metrics in
a range between 0 and 1. Entities with the lowest score are
assigned to 0 and entities with the highest score are assigned to
1. Jaguar assigns suspiciousness scores to packages, classes,
and methods based on the suspiciousness scores of lines or
duas. The score of each package, class, and method is the
highest score of their internal lines or duas.
The Jaguar Runner’s output is a list of lines or duas executed
by the tests and their respective suspiciousness scores. A
Jaguar’s SFL list is an XML file in which each line or
dua is within its code hierarchy structure (package, class,
and method). Figure 2 shows an example of a Jaguar file
with line spectrum. We removed some attributes for a better
visualization of the report file. Jaguar also provides another
XML format, which is composed of only lines or duas. Jaguar
Runner is available both as an open source plug-in for the
Eclipse IDE and a command line tool.
B. Jaguar Viewer
Jaguar Viewer provides visual information of the SFL lists
for debugging within the Eclipse IDE. It colors the suspicious
program entities according to their suspiciousness scores.
The colors represent four score levels: red (danger) are the
most suspicious entities, orange (warning) are those with
high suspiciousness, yellow (caution) are the entities with
a moderate suspiciousness level, and green (safety) are the
least suspicious ones. The color schema was based on that of
the Tarantula tool [3]. Additionally, we included the orange
color, as done in [22], to increase the distinction among the
program elements. The suspiciousness range for the colors is:
red ≥ 0.75, 0.75 > orange ≥ 0.5, 0.5 > yellow ≥ 0.25, and
green < 0.25.
Clicking on a program entity in Jaguar leads to the source
code on the Eclipse’s editor view. The Jaguar Viewer also has
two filtering widgets to facilitate debugging: a text search box
to filter by terms of the source code and a slider to narrow
down entities by suspiciousness score. Figure 3 shows the
Jaguar View’s list of methods within the Eclipse IDE. The
slider and text search widgets are below the method list.
We created seven views for the Jaguar results. Three views
are lists with a single program entity type: methods, lines, or
duas. Two views have the complete code hierarchy structure
for lines or duas, i.e., a list is ordered by package and its
respective classes, methods, and lines or duas. These two
views have two frames: one frame for packages, classes, and
methods; and another frame that shows lines or duas that
belong to the method set in the previous frame. The two other
views also have two frames: one with the most suspicious
methods and another with their respective lines or duas.
Figure 4 shows details of the Jaguar’s view composed
of code hierarchy with duas. This example shows a fault
of the jtopas program, obtained from the Software-artifact
Repository Infrastructure (SIR – sir.unl.edu) [23]. The values
in the Def and Use columns are the line numbers where the
definitions and uses occur for a variable in the Var column.
Through these views, Jaguar provides information about
the most suspicious entities in a simple visualization scheme,
which is adherent to other Eclipse’s views, such as the JUnit
and the Package explorer views. A developer using Jaguar
can go straight to the entity s/he wants to inspect without the
need of understanding the meaning of a sophisticated graphical
model.
Jaguar also allows tracking user actions performed within
Eclipse for experimental purposes. These actions include
mouse and keyboard events, such as clicking and hovering on
the plug-in area, clicking and code editing in the editor view,
breakpoint settings, and test running. Moreover, the Jaguar
XML report files can also be used in other visualization tools.
III. A SSESSMENT AND LIMITATIONS
A. Effectiveness and efficiency
Jaguar can be used for debugging in real and large systems.
We assessed the tool in programs of the Defects4J database
(github.com/rjust/defects4j) [24], namely, JFreeChart, Com-
mons Lang, Commons Math, and Joda-Time, whose sizes vary
from 22 to 96 KLOC. The test suites of these programs vary
<?xml v e r s i o n = ” 1 . 0 ” e n c o d i n g =”UTF−8” s t a n d a l o n e =” y e s ”?>
<H i e r a r c h i c a l F a u l t C l a s s i f i c a t i o n m e t r i c =”OCHIAI” r e q u i r e m e n t T y p e =” LINE ” t i m e S p e n t =” 11741 ”>
<p a c k a g e name=” de . s u s e b o x . j a v a . u t i l ” p o s i t i o n =” 1 ” s u s p =” 1 . 0 ”>
<c l a s s l i n e =” 105 ” name=” de . s u s e b o x . j a v a . u t i l . A b s t r a c t T o k e n i z e r ” p o s i t i o n =” 1 ” s u s p =” 1 . 0 ”>
<method i d =” 43 ” l i n e =” 837 ” name=” addKeyword ( S t r i n g , O b j e c t , i n t ) ” p o s i t i o n =” 1 ” s u s p =” 1 . 0 ”>
<r e q u i r e m e n t l i n e =” 837 ” c e f =” 18 ” c e p =” 1 ” c n f =” 0 ” cnp =” 102 ” p o s i t i o n =” 1 ” s u s p =” 1 . 0 ” />
<r e q u i r e m e n t l i n e =” 839 ” c e f =” 18 ” c e p =” 1 ” c n f =” 0 ” cnp =” 102 ” p o s i t i o n =” 1 ” s u s p =” 1 . 0 ” />
<r e q u i r e m e n t l i n e =” 866 ” c e f =” 18 ” c e p =” 1 ” c n f =” 0 ” cnp =” 102 ” p o s i t i o n =” 1 ” s u s p =” 1 . 0 ” />
<r e q u i r e m e n t l i n e =” 867 ” c e f =” 18 ” c e p =” 1 ” c n f =” 0 ” cnp =” 102 ” p o s i t i o n =” 5 ” s u s p =” 0 . 9 7 ” />
<r e q u i r e m e n t l i n e =” 863 ” c e f =” 18 ” c e p =” 1 ” c n f =” 0 ” cnp =” 102 ” p o s i t i o n =” 5 ” s u s p =” 0 . 9 7 ” />
<r e q u i r e m e n t l i n e =” 864 ” c e f =” 18 ” c e p =” 1 ” c n f =” 0 ” cnp =” 102 ” p o s i t i o n =” 5 ” s u s p =” 0 . 9 7 ” />
</ method>
<method i d =” 71 ” l i n e =” 1555 ” name=” i s K e y w o r d ( i n t , i n t ) ” p o s i t i o n =” 2 ” s u s p =” 0 . 9 8 ”>
<r e q u i r e m e n t l i n e =” 1561 ” c e f =” 18 ” c e p =” 0 ” c n f =” 0 ” cnp =” 103 ” p o s i t i o n =” 4 ” s u s p =” 0 . 9 8 ” />
<r e q u i r e m e n t l i n e =” 1565 ” c e f =” 18 ” c e p =” 0 ” c n f =” 0 ” cnp =” 103 ” p o s i t i o n =” 4 ” s u s p =” 0 . 9 8 ” />
<r e q u i r e m e n t l i n e =” 1564 ” c e f =” 18 ” c e p =” 0 ” c n f =” 0 ” cnp =” 103 ” p o s i t i o n =” 4 ” s u s p =” 0 . 9 8 ” />
<r e q u i r e m e n t l i n e =” 1558 ” c e f =” 18 ” c e p =” 68 ” c n f =” 0 ” cnp =” 35 ” p o s i t i o n =” 171 ” s u s p =” 0 . 4 5 ” />
<r e q u i r e m e n t l i n e =” 1555 ” c e f =” 18 ” c e p =” 68 ” c n f =” 0 ” cnp =” 35 ” p o s i t i o n =” 171 ” s u s p =” 0 . 4 5 ” />
<r e q u i r e m e n t l i n e =” 1569 ” c e f =” 1 ” c e p =” 68 ” c n f =” 17 ” cnp =” 35 ” p o s i t i o n =” 450 ” s u s p =” 0 . 0 2 ” />
</ method>
<method i d =” 7 ” l i n e =” 234 ” name=” a d d S t r i n g ( S t r i n g , S t r i n g , S t r i n g ) ” p o s i t i o n =” 5 ” s u s p =” 0 . 7 8 ”>
<r e q u i r e m e n t l i n e =” 235 ” c e f =” 17 ” c e p =” 9 ” c n f =” 1 ” cnp =” 94 ” p o s i t i o n =” 20 ” s u s p =” 0 . 7 8 ” />
<r e q u i r e m e n t l i n e =” 234 ” c e f =” 17 ” c e p =” 9 ” c n f =” 1 ” cnp =” 94 ” p o s i t i o n =” 20 ” s u s p =” 0 . 7 8 ” />
</ method>

Fig. 2. Excerpt of a Jaguar’s XML report file.

from 2k to 4k test cases. We also used Ant (obtained from TABLE II

SIR), and jsoup (github.com/jhy/jsoup) in the assessment. T IME TO RUN JAGUAR FOR DATA - AND CONTROL - FLOW.
Table I shows the number of faults ranked among the top-5,
Program JUnit (s) DF (s) CF (s) KLOC # Tests
top-10, top-20, and top-30 entities ranked by Jaguar for dua
Ant 71 97 78 79 986
(DF) and line (CF) spectra using the Ochiai metric. The # JFreeChart 22 88 36 96 2205
Faults column shows the number of faulty versions of each jsoup 4 22 11 10 468
program. These results show data-flow spectrum ranks more Commons Lang 18 89 33 22 2245
Commons Math 144 515 265 85 3602
faults among the top entities compared to control-flow, except Joda-Time 4 165 47 28 4130
for the top-5. Currently, we are carrying out more experiments
to compare the performance of data- and control-flow using
multiple-fault programs and other ranking metrics. We intend
The overall time to execute Jaguar using data-flow spectrum
to publish these results soon.
varied from 22 seconds (jsoup) to 8:35 minutes (Commons
Math). However, executing the tests without Jaguar in Com-
TABLE I
FAULTS RANKED AMONG THE TOP ENTITIES FOR DATA - AND mons Math programs takes 2:24 minutes. The use of control-
CONTROL - FLOW USING O CHIAI . flow spectrum also increased the overall execution time, but it
took less time than data-flow spectrum.
# Faults top-5 top-10 top-20 top-30
Program
DF CF DF CF DF CF DF CF B. Limitations
Ant 10 2 2 4 4 8 6 8 6 In the current version, Jaguar cannot collect dua spectrum in
JFreeChart 26 1 2 6 6 10 11 11 11 two cases: (1) local duas; and (2) non-handled exceptions that
jsoup 38 11 8 20 12 23 19 29 21
Commons Lang 30 2 2 9 7 15 12 18 17 are thrown during an execution. The first case occurs because
Commons Math 43 6 6 11 9 21 13 24 20 the ba-dua tool does not mark duas when the definition and
Joda-Time 26 0 3 1 3 4 5 4 5 use of a variable occur in the same block. The second case
Total 173 22 23 51 41 81 66 94 80 occurs because ba-dua marks the exercised duas whenever a
method is exited. Thus, a non-handled exception ends in a
Table II shows the average time in seconds to execute of non-predictable way that is not covered by the tool.
Jaguar for dua (DF) and line (CF) spectra. The JUnit column Jaguar and ba-dua cover only intra-procedural duas. How-
shows the execution time to run the tests without Jaguar. The ever, to the best of our knowledge, the few available coverage
KLOC and # Tests columns show the size and the number of tools that support inter-procedural data-flow coverage (DUA-
tests of the programs, respectively. FORENSICS – sourceforge.net/projects/duaf [25] and JaBUTi
At each execution, Jaguar runs the tests, collects the spectra – ccsl.icmc.usp.br/pt-br/projects/jabuti [26]) do not work well
information, calculates the suspiciousness of all executed for large and long-running programs due to the high execution
program entities for the ten ranking metrics, and creates a costs to track duas. Instead, ba-dua was devised to collect data-
report file for each metric. We measured the execution time flow spectrum in a lightweight manner, which allows its use
by running each program at least ten times. on such programs.
 Fig. 3. Jaguar SFL tool with a method list.
C. Assessment with users
We assessed Jaguar in a user study that we submitted for
publishing [15]. In this study, the participants performed two
debugging tasks, one using Jaguar and the other without using
it. As our goal was to evaluate the use of SFL in practice,
we chose to use only control-flow spectrum. The use of both
control- and data-flow would include another independent
variable in the study.
The results suggest that Jaguar (and, thereby, SFL) can
improve fault localization. From the 26 participants that took
part in the study, 12 participants (46.15%) found a fault using
Jaguar, while only 5 participants (19.23%) found a bug without
using Jaguar. Moreover, the tool was evaluated by most of
the participants as useful and ease of use. They also showed
interest in using the tool in their future debugging activities.
By analyzing the participants’ behavior while using Jaguar,
we noted that most of them use the suspiciousness lists along
with the inspection of the failing test cases, navigating through
both resources the lists and the test cases when searching for
the faults. Thus, there is a need to provide more information to
help developers in understanding where and why faults occur.
In the first semester of 2018, Jaguar will be used by graduate
students of the Software Testing course in the São Paulo State
Technological College – East Campus, São Paulo, Brazil. We
will keep contact with the students to gather their impressions
on using the tool and assess how they use it.
IV. R ELATED WORK
There are some fault localization tools proposed by the
research community. Most of them have been used only for
experimental purposes, while others are freely accessible for
use in real programs. Some of them are presented below.
Tarantula [3] is a standalone fault localization tool for C
programs. The tool has a visualization scheme that shows the
whole source code of a program. Tarantula colors the state-
ments according to their suspiciousness scores, from higher
(red) to lower (green) scores. The tool also presents differ-
ent brightness levels according to the frequency in which a
statement is executed by the tests. Statements more frequently
executed are the brightest ones.
Tarantula does not work with unit testing frameworks; it
receives a file that contains the result and spectrum of each
test. Moreover, the only metric available in the tool is the
homonym Tarantula metric. To the best of our knowledge, the
largest program used to evaluate Tarantula was Space [23],
which has 9.5 KLOC and 13,585 test cases.
VIDA [27] is a fault localization tool implemented as
an Eclipse plug-in for Java programs. It collects statement
spectrum from JUnit tests and calculates suspiciousness using

Fig. 4. List of code hierarchy with duas.
the Tarantula metric. VIDA presents a list of the top ten
statements, presented as breakpoint candidates.
Also, the tool has a view that shows the history of break-
points. This view includes the developers’ previous estimations
of the correctness of the breakpoint candidates and the current
suspiciousness of statements. The aim is to help developers
in setting breakpoints. VIDA uses colors to differentiate the
developers’ estimations, from red (wrong) to green (correct)
and suspiciousness, from black (high suspiciousness) to light
gray (low suspiciousness). Finally, VIDA provides a static
dependency graph to help developers in the estimation process.
Whyline (github.com/andyjko/whyline) [28] is a standalone
and interactive debugging tool for Java programs. The tool
provides why and why not questions to assist developers in
understanding a program’s behavior. Whyline records com-
plete executions traces, which include trace events and all java
classes (executed or not) to reproduce such an execution. The
tool uses static and dynamic slicing to formulate questions,
presenting such questions in a graphical and interactive way.
Using Whyline, a user loads the execution trace, chooses a
point of interest in the execution timeline. Then, s/he clicks
on the code to open a pop-up with the questions. The questions
contain information about properties of a selected object,
including data values gathered from the execution.
Ko and Myers [29] evaluated the Whyline tool in a user
study. More participants that used Whyline found the bugs
than those that did not use the tool. They were also faster
using the tool. The tool was assessed in the ArgoUML program
(argouml.tigris.org) (150 KLOC). Whyline is not related to
SFL; it does not depend on tests or ranking metrics. However,
the tool does not indicate possible code excerpts.
GZoltar (gzoltar.com) [30] is a fault localization tool for
Java programs available both as an Eclipse plug-in and as a
command line tool. The plug-in version provides three distinct
visualizations to report SFL results. These visualizations aim
to facilitate the investigation of suspicious code excerpts in the
structure of OO Java projects, showing the most suspicious
workspaces, projects, packages, files, classes, methods, and
statements. The tool colors the executed entities from red
to green according to their suspiciousness scores, which are
calculated using the Ochiai metric.
Gouveia et al. [22] performed a user study to evaluate
GZoltar, in which the participants that used the tool were
more effective and efficient than those that did not use it. The
tool was assessed in the XStream (x-stream.github.io) program,
which has 17.4 KLOC and 1,457 test cases. The tool received
positive evaluations regarding its usability.
Jaguar differs from other tools by providing data-flow
information through dua spectrum. Thus, users can search not
only for suspicious lines, but they can also inspect suspicious
variables when looking for bugs. Whyline also provides data
information, storing values collect during an execution.
Different from GZoltar, Jaguar colors the source code in the
Eclipse’s editor view, which highlights the suspicious code.
We opted by presenting the fault localization results in single
suspiciousness lists, which is similar to most existing views
provided by the Eclipse IDE. Jaguar has seven views, three of
them are lists with single program entities, two views are lists
with the code structure (as the GZoltar views), and the other
two views are lists composed of methods their lines or duas.
Jaguar was assessed in large and long-running programs.
The cost to execute for data-flow is not prohibitive for such
programs. Another difference of Jaguar is that it provides
results for ten ranking metrics instead of a single metric
such as for Tarantula and GZoltar. Table III depicts the
main features of each tool. The columns are, respectively, the
tools, support to control-flow spectrum (CF), support to data-
flow spectrum (DF), availability as an IDE’s plug-in (IDE),
availability as a command line tool (CL), integration with a
testing framework (Test), number of available ranking metrics
(Met), how the tool is available (Avail), supported language,
and size of the programs used in their assessments (PS) in
KLOC. The Avail column indicates that a tool is available via
authors, via an open source license, or free for use. The “–”
symbol refers to features that are not related to a tool.
TABLE III
M AIN FEATURES OF FAULT LOCALIZATION TOOLS .
Tool CF DF IDE CL Test Met Avail Lang PS
Tarantula 3 1 Author C 9.5K
VIDA 3 3 3 1 Author Java
Whyline 3 3 – – Open Java 150K
GZoltar 3 3 3 3 1 Free Java 17.4K
Jaguar 3 3 3 3 3 10 Open Java 96K
 V. C ONCLUSION AND FUTURE WORK
This paper presents Jaguar, an open source fault localiza-
tion tool for Java programs. Jaguar provides information of
program entities that are more likely to contain faults.
Jaguar allows researchers to assess the use of data-flow
strategies for fault localization. It is possible to carry out both
user studies and empirical studies with large programs using
the tool. Thus, Jaguar can be used to perform experiments
in realistic scenarios. Moreover, the tool is available for the
software development community.
We have assessed Jaguar in real programs and experiments
with developers. The results suggest that the tool is feasible
for use in commodity computers, even for data-flow spectrum
and large programs with thousands of test cases. Moreover,
users evaluated the tool as useful and ease of use.
As future work, we will improve the tool to allow it to track
duas in single-block methods and exception flows. Coping
with this issues depends on changes in the ba-dua coverage
tool. We are currently working on these changes. Also, we
plan to modify Jaguar to reduce its execution costs and to
improve its usability. Another future work is to provide new
visualizations with additional information, such as relation-
ships among suspicious entities and test cases, data and control
dependencies, and other spectra. Finally, we intend to conduct
more user studies to understand how developers use the data-
flow information to locate faults and to assess how the tool is
used in real development settings.
ACKNOWLEDGMENT
The authors acknowledge the contribution of Danilo Mutti,
Luiz H. Baptistão, Felipe de A. Albuquerque and Mario
Concı́lio during the development of the tools that support
Jaguar. Also, we would like to thank Dr. Alessandro Orso
for its insights and contribution to the tool’s development.
R EFERENCES
[1] G. Tassey, “The economic impacts of inadequate infrastructure for
software testing,” National Institute of Standards and Technology, RTI
Project, vol. 7007, no. 011, 2002.
[2] G. Dandan, S. Xiaohong, W. Tiantian, M. Peijun, and Y. Wang, “State
dependency probabilistic model for fault localization,” Information and
Software Technology, vol. 57, no. 0, pp. 430–445, 2014.
[3] J. A. Jones, M. J. Harrold, and J. Stasko, “Visualization of test informa-
tion to assist fault localization,” in Proceedings of the 24th International
Conference on Software Engineering, ser. ICSE ’02, 2002, pp. 467–477.
[4] R. Santelices, J. A. Jones, Y. Yu, and M. J. Harrold, “Lightweight fault-
localization using multiple coverage types,” in Proceedings of the 31st
International Conference on Software Engineering, ser. ICSE ’09, 2009,
pp. 56–66.
[5] R. Abreu, P. Zoeteweij, and A. J. C. van Gemund, “On the accuracy
of spectrum-based fault localization,” in Proceedings of the Testing:
Academic and Industrial Conference Practice and Research Techniques
- MUTATION, ser. TAICPART-MUTATION ’07, 2007, pp. 89–98.
[6] L. Naish, H. J. Lee, and K. Ramamohanarao, “Spectral debugging with
weights and incremental ranking,” in Proceedings of the 16th Asia-
Pacific Software Engineering Conference, ser. APSEC’09, 2009, pp.
168–175.
[7] M. Renieris and S. P. Reiss, “Fault localization with nearest neighbor
queries,” in Proceedings of the 18th IEEE International Conference on
Automated Software Engineering, ser. ASE ’03, 2003, pp. 30–39.
[8] J. A. Jones, J. F. Bowring, and M. J. Harrold, “Debugging in parallel,” in
Proceedings of the ACM SIGSOFT International Symposium on Software
Testing and Analysis, ser. ISSTA ’07, 2007, pp. 16–26.
View publication stats
[9] T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani,
“Holmes: Effective statistical debugging via efficient path profiling,” in
Proceedings of the 31st International Conference on Software Engineer-
ing, ser. ICSE’09, 2009, pp. 34–44.
[10] W. Masri, “Fault localization based on information flow coverage,”
Software Testing, Verification and Reliability, vol. 20, no. 2, pp. 121–
147, 2010.
[11] M. L. Chaim and R. P. A. d. Araujo, “An efficient bitwise algorithm
for intra-procedural data-flow testing coverage,” Information Processing
Letters, vol. 113, no. 8, pp. 293–300, 2013.
[12] R. P. A. de Araujo and M. L. Chaim, “Data-flow testing in the large,”
in Proceedings of the 7th IEEE International Conference on Software
Testing, Verification and Validation, ser. ICST’14, 2014, pp. 81–90.
[13] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on
software fault localization,” IEEE Transactions on Software Engineering,
vol. 42, no. 8, pp. 707–740, 2016.
[14] H. L. Ribeiro, “On the use of control- and data-flow in fault localization,”
Master thesis, School of Arts, Sciences and Humanities, University of
São Paulo, São Paulo, Brazil, 2016.
[15] H. A. de Souza, M. de S. Lauretto, M. L. Chaim, and F. Kon, “Practical
evaluation of spectrum-based fault localization,” unpublished.
[16] M. L. Chaim, J. C. Maldonado, and M. Jino, “A debugging strategy
based on requirements of testing,” in Proceedings of the 7th Euro-
pean Conference on Software Maintenance and Reengineering, ser.
CSMR’03, 2003, pp. 160–169.
[17] R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, “A
practical evaluation of spectrum-based fault localization,” Journal of
Systems and Software, vol. 82, no. 11, pp. 1780–1792, 2009.
[18] J. Xu, Z. Zhang, W. K. Chan, T. H. Tse, and S. Li, “A general
noise-reduction framework for fault localization of java programs,”
Information and Software Technology, vol. 55, no. 5, pp. 880–896, 2013.
[19] L. Naish, H. J. Lee, and K. Ramamohanarao, “A model for spectra-
based software diagnosis,” ACM Transactions on Software Engineering
and Methodology, vol. 20, no. 3, pp. 1–32, 2011.
[20] W. E. Wong, V. Debroy, and B. Choi, “A family of code coverage-
based heuristics for effective fault localization,” Journal of Systems and
Software, vol. 83, no. 2, pp. 188–208, 2010.
[21] A. Gonzalez-Sanchez, “Automatic error detection techniques based on
dynamic invariants,” Master’s thesis, Delft University of Technology,
2007.
[22] C. Gouveia, J. Campos, and R. Abreu, “Using HTML5 visualizations in
software fault localization,” in 1st IEEE Working Conference on Software
Visualization, ser. VISSOFT ’13, September 2013, pp. 1–10.
[23] H. Do, S. Elbaum, and G. Rothermel, “Supporting controlled experi-
mentation with testing techniques: An infrastructure and its potential
impact,” Empirical Software Engineering, vol. 10, no. 4, pp. 405–435,
2005.
[24] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing
faults to enable controlled testing studies for java programs,” in Pro-
ceedings of the ACM SIGSOFT International Symposium on Software
Testing and Analysis, ser. ISSTA ’14, 2014, pp. 437–440.
[25] R. Santelices and M. J. Harrold, “Efficiently monitoring data-flow
test coverage,” in Proceedings of the 22nd IEEE/ACM International
Conference on Automated Software Engineering, ser. ASE’07, 2007, pp.
343–352.
[26] A. M. R. Vincenzi, J. C. Maldonado, W. E. Wong, and M. E. Dela-
maro, “Coverage testing of java programs and components,” Science of
Computer Programming, vol. 56, no. 1-2, pp. 211–230, 2005.
[27] D. Hao, L. Zhang, T. Xie, H. Mei, and J.-S. Sun, “Interactive fault
localization using test information,” Journal of Computer Science and
Technology, vol. 24, no. 5, pp. 962–974, 2009.
[28] A. J. Ko and B. A. Myers, “Debugging reinvented,” in Proceedings of the
30th International Conference on Software Engineering, ser. ICSE’08,
2008, pp. 301–310.
[29] ——, “Finding causes of program output with the java whyline,” in
Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, ser. CHI ’09, 2009, pp. 1569–1578.
[30] J. Campos, A. Riboira, A. Perez, and R. Abreu, “Gzoltar: an eclipse
plug-in for testing and debugging,” in Proceedings of the 27th
IEEE/ACM International Conference on Automated Software Engineer-
ing, ser. ASE’12, 2012, pp. 378–381.