9/26/13

The JMockit Tutorial - Measuring code coverage

Measuring code coverage with JMockit Coverage

1.
2.
3.
4.
5.
6.

7.
8.
9.
10.

Line coverage
Path coverage
Data coverage
Types of coverage output
1. Call points
Configuring the coverage tool
Generating aggregate reports for separate test runs
1. Generating a single aggregate report for a grouping of test runs
2. Generating an aggregate report from a set of data files
Checking minimum coverage
Activating coverage in a Maven project
Turning coverage output off
Standalone mode

Code coverage consists of a set of software metrics that can tell you how much of the production code is
covered by a given test suite. It's purely quantitative, and does not say anything about the quality of either
the production code or the test code. That said, the examination of code coverage reports will
sometimes lead to the discovery of unreachable code which can be eliminated. But more importantly,
such reports can be used as a guide for the discovery of missing tests. This is not only useful when
creating tests for existing production code, but also when writing tests first, such as in the practice of
TDD (Test Driven Development).

JMockit Coverage provides three different and complementary code coverage metrics: line coverage,
path coverage, and data coverage. An example coverage report showing all metrics can be found here.

Other coverage tools for Java include EMMA,

Clover, Cobertura, and JaCoCo. (The latter being
more often used through the EclEmma Eclipse
plugin.) These tools provide two separate
coverage metrics, namely statement and branch
coverage. The first one is also called line
coverage, even though they don't actually attempt
to measure how much of the executable code in
individual lines has been covered. The second
measures how many of the alternative branches
resulting from decision points (an ifor switch
statement) have been taken during a test run.
JMockit Coverage uses a different, but related,
set of coverage metrics.

Line coverage
The line coverage metric tells us how much of the executable code in a source file has been exercised by tests. Each executable line of code can be
uncovered, covered, or partially covered. In the first case, none of the executable code in it was executed at all. In the second, all of the code was fully
executed at least once. In the third case, only part of the executable code in the line was executed. This can happen, for example, with lines of code containing
multiple logical conditions in a complex boolean expression. JMockit Coverage identifies all three cases, computing the coverage percentage for each
executable line of code accordingly: 0% for an uncovered line, 100% for a covered line, or some value in between for a partially covered line.
A branching point exists wherever the program makes a decision between two possible execution paths to follow. Any line of code containing a logical
condition will be divided in at least two executable segments, each belonging to a separate branch. An executable line of source code with no branching
points contains a single segment. Lines with one or more branching points contain two or more executable segments, separated by consecutive branching
points in the line.
Lets say that NS >= 1is the number of executable segments on a given line. If NEis the number of segments in that line which were executed at least once
during a test run (ie, they are covered segments), then we can calculate the coverage percentage for the line as 100 * NE / NS.
Similarly, the line coverage percentage for a whole source file is calculated from the total number of executable segments and the total number of covered
segments, considering all executable lines of code in the file. The percentage for a package, in turn, is calculated from the total and covered numbers of
segments in the whole set of source files belonging to the package. Finally, the total code coverage percentage is computed by the same formula on the totals
for all packages.

Path coverage
A completely different metric is path coverage, which is computed for method and constructor bodies, not for lines or segments of code. It tells us how many of
the possible execution paths through a method or constructor, from entry to exit, have been executed at least once during the test run.
Note that each method or constructor has a single point of entry, but can have multiple exits. An exit occurs when a returnor throwstatement is executed.
These are normal exits, of course. A method/constructor execution can also terminate abruptly, by propagating an exception (or error) thrown as a result of a
method call, an attempt to access a nullreference, or some other action which caused an unintended program failure.
Each possible path can be either fully executed (covered) or not (uncovered). Paths that execute only partially (ie, they were terminated abruptly) are simply
considered as uncovered.
The path coverage percentage for a method or constructor body is computed in a way similar to the line coverage computation. If NPis the number of possible
paths through the implementation body and NPEis the number of paths executed from entry to exit, then the metric is computed as 100 * NPE / NP. Also in
the same way as the line coverage metric, we extend this formula to the whole source file, the whole package, and the whole set of packages touched by the
test run.

Data coverage
jmockit.googlecode.com/svn/trunk/www/tutorial/CodeCoverage.html

1/5

9/26/13

The JMockit Tutorial - Measuring code coverage

Measures how many of the instance and static non-final fields were fully exercised by the test run. To be fully exercised, a field must have the last value
assigned to it read by at least one test. The percentages are calculated as 100 * NFE / NF, where NFis the number of non-final fields and NFEthe number
of fully exercised fields.

Types of coverage output

The JMockit Coverage tool can generate the following types of output:
1. HTML reports: a multi-page HTML report is written in the "coverage-report" directory, under the current working directory (a different output directory
can be specified if needed). The directory is created if it doesn't yet exist; its contents are overwritten if previously generated. The report will include
pages containing all Java source files covered by the test suite. By default, the tool looks for ".java" source files inside all directories of name "src"
found directly or indirectly under the current working directory; any intermediate sub-directories between "src" and the top-level package directory, such
as "src/java" for example, are also searched.
2. Coverage data files: a single serialized file of name "coverage.ser" is written under the current working directory or an specified output directory. If
the file already exists, its contents are either overwritten or merged with the in-memory results of the current test run, as specified.
These files can be read and processed by external tools. The mockit.coverage.data.CoverageData.readDataFromFile(File)method will
create a new CoverageDatainstance with all the coverage data available in a given serialized file. For more on this, refer to the API documentation
available in jmockit-coverage.jar.
Call points
When running a test suite with the coverage tool, there is optional "call point" information which can be gathered, as selected by the user. A call point is the
point in the source test code from which an specific line of production code was exercised.
Generating coverage with this extra information takes more time and produces significantly larger output; on the other hand, it can be useful to know which
lines of test code caused a given line of production code to be executed during the test run. When included in the HTML report, the list of call points appears
hidden at first but can be easily viewed by clicking on each executable line of code.

Configuring the coverage tool

To enable the JMockit Coverage tool in a JUnit/TestNG test run, add both jmockit.jarand jmockit-coverage.jarto the runtime classpath. With JUnit,
make sure that jmockit.jarappears first in the classpath. If running on a Java 5 JRE, however, add "-javaagent:<proper path>/jmockit.jar" as a
JVM initialization parameter. (For more details on running tests with JMockit, see this page.)
When not using the JMockit mocking APIs, code coverage can still be activated without adding any jar to the classpath. Instead, run with "-javaagent:
<proper path>/jmockit-coverage.jar" as a JVM initialization parameter.
In most cases, the coverage tool does not require any additional configuration to be used. There are, however, several aspects of the tool's behavior which
can optionally be configured for a given test run. This is done by setting one or more of several "jmockit-coverage-xyz" system properties for the JVM
instance running the test suite. Note that you should be able to easily do this inside an Ant target, a Maven surefireplugin configuration, or a test run
configuration for your Java IDE of choice, using either JUnit or TestNG; no JMockit-specific plugin is needed.
The available configuration properties are:
1. jmockit-coverage-output: one or more comma-separated values between html, html-nocp("nocp" stands for "no call points"), serial, and
merge, which select the kind of output to be generated at the end of the test run. The default if none is specified is to generate the basic HTML report
(html-nocp).
The "html" and "html-nocp" values are mutually exclusive, just like "serial" and "merge". However, it is valid to have one of each pair specified at the
same time. In such a case, at the end of the test run both kinds of output will be written.
The presence of "serial" or "merge" causes a serialized data file of name "coverage.ser" to be generated; in the case of "merge", the contents of a
previously existing data file (if any) will be merged with the coverage data gathered by the current test run (otherwise, it has the same effect as
"serial").
2. jmockit-coverage-outputDir: absolute or relative path to the output directory, to be used for writing any "coverage.ser" or "index.html" files
(plus the remaining ".html" files of the HTML report, in automatically created sub-directories). By default, the current working directory of the running
JVM is used, with all ".html" files of the HTML report generated inside a "coverage-report" sub-directory.
3. jmockit-coverage-srcDirs: comma-separated list of Java source directories to be searched when generating an HTML report. (This is not relevant
for the serialized data file.) Each directory is specified by an absolute or relative path. If no such directory is specified, all "src" directories under the
current working directory are searched.
4. jmockit-coverage-classes: a java.util.regex-conformable regular expression which will be used to select the classes from production code which
should be considered for coverage. By default, all classes in production code loaded during the test run and which are not inside jar files are considered.
5. jmockit-coverage-excludes: a java.util.regex-conformable regular expression for class names which should be excluded from consideration when
instrumenting classes for coverage. This property can be used together with jmockit-coverage-classesor on its own. By default, no classes
between those selected for coverage are excluded from consideration.
6. jmockit-coverage-metrics: one or more comma-separated words between line, path, data, and all(the default), which select the specific set of
code coverage metrics to gather coverage information for.
7. jmockit-coverage-check: one or more semicolon-separated rules specifying minimum coverage checks to be performed at the end of a test run. By
default, no such checks are performed. For details, see the specific section about this feature below.
The class selection regular expressions should specify the fully qualified names of classes in production code. Such classes can be either inside directories or
jmockit.googlecode.com/svn/trunk/www/tutorial/CodeCoverage.html

2/5

9/26/13

The JMockit Tutorial - Measuring code coverage

jarfiles. For example, "orderMngr\.domain\..+" selects all classes in the orderMngr.domainpackage as well as in any sub-packages.

Generating aggregate reports for separate test runs

Suppose you have multiple test suites or test run configurations, and you want to generate a single HTML report for the code covered by the full set of tests.
Normally, when JMockit Coverage generates a report at the end of a test run, it overwrites any previous report. Then, instead of getting a merged report as
desired, you get the report for the last test run only. Here is where the "coverage.ser" serialized data files come in.
To activate the generation of this file, we simply set the jmockit-coverage-outputsystem property to a value containing "serial" or "merge". As these
two values suggest, there are different ways to combine multiple coverage data files. The following sub-sections provide the details for each case.
Generating a single aggregate report for a grouping of test runs
Lets say we have multiple test run configurations (for example, multiple junitor testngtasks in the same Ant target) to be executed at once, and want to
produce a single aggregate coverage report for the full set of tests. That is, we don't want to produce a separate report for each individual test run. The easiest
way to achieve this is to use the same working directory for all test runs, with jmockit-coverage-output=mergespecified as a system property for each
test run configuration. Since the HTML report is only desired at the end, the last test run configuration in the sequence should have the appropriate value
specified in the jmockit-coverage-outputproperty. The "coverage.ser" file will be shared between all test runs. The first test run must not read data
from this file; therefore, either the file should be deleted before the first test run, or ignored by having the first test run set jmockit-coverageoutput=serial.
Generating an aggregate report from a set of data files
Suppose we want to generate a separate coverage report for each test run configuration and also an aggregate report merging together the results from all
test runs. In this case we would have ntest runs but n + 1coverage reports. Only one HTML report can be produced from a single test run, so an extra step
which doesn't execute any tests is necessary. In order to preserve the original coverage.seroutput files generated by each test run, they will need to be
written or copied into different output directories.
Assuming that two or more coverage.serfiles are available in separate directories, an aggregate report can be generated from them by executing the
mockit.coverage.CodeCoverage.mainmethod (a regular Java "main" method). To facilitate this, the jmockit-coverage.jarfile is executable. Using
Ant, the following task could be used:
<java fork="yes" dir="myBaseDir" jar="jmockit-coverage.jar">
<arg line="-Djmockit-coverage-output=html module1-outDir anotherOutDir"/>
</java>

The example above uses "myBaseDir" as the base directory where a separate JVM instance will run. Two output directories containing "coverage.ser"
data files are specified, as command line arguments. Other configuration parameters can be specified through the "jmockit-coverage-xyz" system
properties. This separate JVM instance will read each of the "coverage.ser" data files, merge the coverage data in memory, and then generate the
aggregate HTML report before exiting.

Checking minimum coverage

If desired, JMockit Coverage can check that the final coverage percentages at the end of a test run satisfy arbitrary minimum values. Such checks can be
specified through one or more checking rules assigned to the "jmockit-coverage-check" system property (when more than one, they must be separated
by ";" characters).
Each checking rule must be in the form "[scope:]min line percentage[,min path percentage[,min data percentage]]". There are three types of scopes:
Total: The default when no scope is specified. It refers to the total percentage for each metric. For example, the rule "80" specifies that the total line
coverage should be at least 80%, with no minimum percentages for the other metrics. An example specifying thresholds for all three metrics could be
"70,60,85". Note that a value of "0" can also be used to specify no minimum.
perFile: Specifies minimum percentages that each source file must satisfy. If one or more files end up with a lower percentage, the check fails. An
example: "perFile:50,0,40", meaning that each source file must have at least 50% of line coverage and at least 40% of data coverage.
Package: Specifies minimum total percentages for the set of source files in a given package, including sub-packages. For example, the rule
"com.important:90,70" specifies that total line coverage for files under "com.important" should be at least 90%, while total path coverage should
be at least 70%.
All checks (if any) are performed at the end of the test run (at JVM shutdown, actually). Other forms of output (HTML report, serialized file) are not affected.
When an individual check fails, a descriptive message is printed to standard output. If one or more checks have failed, two final actions are taken to have the
fact reported: first, an empty file of name "coverage.check.failed" is created in the current working directory; second, an error (specifically, an
AssertionError) is thrown. When checks are performed but they all pass, the "coverage.check.failed" file, if present in the current directory, is deleted.
The use of a file to mark the success or failure of coverage checks is meant to allow build tools to react accordingly, typically by failing the build when the file is
present. For example, we can do the following in an Ant build script:
<fail message="Coverage check failed">
<condition><available file="coverage.check.failed"/></condition>
</fail>

jmockit.googlecode.com/svn/trunk/www/tutorial/CodeCoverage.html

3/5

9/26/13

The JMockit Tutorial - Measuring code coverage

Activating coverage in a Maven project

If you run tests with Maven's "test" goal, you will need the following dependencies in the pom.xmlfile (assuming the "version" properties have been properly
defined):
<dependency>
<groupId>com.googlecode.jmockit</groupId><artifactId>jmockit</artifactId>
<version>${jmockit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.googlecode.jmockit</groupId><artifactId>jmockit-coverage</artifactId>
<version>${jmockit.coverage.version}</version>
<scope>runtime</scope>
</dependency>

In Maven 2/3, the surefireplugin is the one usually responsible for actually running tests. To configure the coverage tool, specify values for the appropriate
"jmockit-coverage-xyz" system properties. For example, the output directory for generated files can be specified through the jmockit-coverageoutputDirproperty.
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration><argLine>
-Djmockit-coverage-outputDir=target/my-coverage-report
<!-- other "jmockit-coverage" properties, if needed -->
</argLine></configuration>
</plugin>

If the tests are executed on a Java SE 5 JVM, the "-javaagent" JVM parameter should be used as shown below. When running on Java 6 or newer, this
parameter is not necessary.
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration><argLine>
-javaagent:"${settings.localRepository}"/com/googlecode/jmockit/jmockit/${jmockit.version}/jmockit-${jmockit.version}.jar
<!-- "jmockit-coverage" properties, if any are needed -->
</argLine></configuration>
</plugin>

Finally, if the tests don't actually use the JMockit mocking APIs, it's still possible to use the coverage tool. In this case, the only dependency needed is the one
on "jmockit-coverage". Additionally, it's necessary to configure the surefireplugin as follows:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration><argLine>
-javaagent:"${settings.localRepository}"/com/googlecode/jmockit/jmockit-coverage/
${jmockit.coverage.version}/jmockit-coverage-${jmockit.coverage.version}.jar
<!-- "jmockit-coverage" properties, if any are needed -->
</argLine></configuration>
</plugin>

Turning coverage output off

Sometimes we want to turn coverage output off for a particular test run, without having to remove the coverage jar from the classpath. This can be done in two
different ways.
For one, we can manipulate the read-only attribute of the relevant output file, when one has already been generated. The particular file to be manipulated,
always in the working directory, is "coverage.ser" for serialized output or "coverage-report/index.html" for HTML output. The file attribute is checked
by JMockit at startup; when marked as read-only it cannot be overwritten, so JMockit avoids the attempt entirely.
Note that the working directory can usually be selected separately for each test run configuration in the Java IDE. Also, a Java IDE usually provides an easy
mechanism to toggle the read-only status of a file in the project: in IntelliJ IDEA it is done by double clicking the status bar, with the desired file opened in the
editor; in Eclipse there is a "Read only" check box in the "Properties" screen (which can be opened by typing "Alt+Enter") for the text file selected in the editor.
Another way to switch coverage off is to simply set the jmockit-coverage-outputsystem property to an unknown output format, such as "-Djmockitcoverage-output=none".

Standalone mode

jmockit.googlecode.com/svn/trunk/www/tutorial/CodeCoverage.html

4/5

9/26/13

The JMockit Tutorial - Measuring code coverage

In previous sections we described the most typical way to use the coverage tool, by enabling it during a JUnit/TestNG test run to measure test coverage. The
tool can also be used in a more general context, though: the standalone mode, where it can attach to any Java 5+ process to measure line and path coverage
metrics on specified classes, regardless of which code is making calls into said classes.
To activate standalone mode, the target JVM instance must be started with the "-javaagent:<proper/path/>jmockit-coverage.jar" command line
argument. That's it; none of the JMockit toolkit jars need to be present in the classpath of the target process. Initial configuration settings for the coverage tool
can be specified through the "jmockit-coverage-xyz" system properties previously described, but this is entirely optional; the configuration properties can
be modified later through a dedicated UI.
Once the target process is running with the JMockit Coverage Java agent, the user should connect to it with a JMX client which can access arbitrary
"MBeans". Usually, the standard JConsole tool available in a Java 5+ JDK will be used. The JMockit Coverage MBean provides several configuration
properties (the same ones which can be set with "-D" on the command line), and one operation through which the desired output can be generated. The user
interface provided by JConsole is shown below, where the process that is running with the coverage tool is a Tomcat 7 server instance.

The configuration properties (shown as "Attributes" of the "CoverageControl" MBean above) are as before, except in the case of "SrcDirs" (which
corresponds to jmockit-coverage-srcDirs). If this property is not specified, no attempt is made to find source files for the classes considered for
coverage.

jmockit.googlecode.com/svn/trunk/www/tutorial/CodeCoverage.html

5/5