ECE453/SE465/CS447/ECE653/CS647:

Syntax-based Testing
(Mutation Testing)
Lin Tan
February 12, 2014
16-Syntax - February 12, 2014
Mutation Testing

We will see two types of mutation testing:

Input-space grammars: to create inputs (both valid

and invalid)

Program-based grammars: to modify programs

(valid)
2
16-Syntax - February 12, 2014
Regular Expression

Consider this Perl Regular expression:

^(deposit|debit) [0-9]{3} \$[0-9]+\.[0-9]{2}$

What are some valid strings?

3
16-Syntax - February 12, 2014
Grammar

What is the limitation of regular expressions?

Grammar

action = dep | deb

dep = "deposit" account amount

deb = "debit" account amount

account = digit{3}

amount = "$"digit+"."digit{2}

digit = ["0"-"9"]
4
16-Syntax - February 12, 2014
Using Grammar

Two ways to use input grammars for software

testing and maintenance:

Recognizer: can include them in a program to

validate inputs;

Generator: can create program inputs for testing.

5
16-Syntax - February 12, 2014
Grammar Terminology

Grammar

actions = action*

action = dep | deb

dep = "deposit" account amount

deb = "debit" account amount

account = digit{3}

amount = $digit+"."digit{2}

digit = ["0"-"9"]
6
Start symbol
Non-terminals
Production rule
Terminals
16-Syntax - February 12, 2014
Coverage Criteria

Terminal Symbol Coverage (TSC). TR contains each terminal

of grammar G.

Production Coverage (PDC). TR contains each production of

grammar G.

Derivation Coverage (DC). TR contains every possible string

derivable from G.

PDC subsumes TSC.

DC often generates innite test sets, and even if you limit to

xed-length strings, you still have huge numbers of inputs.
7
16-Syntax - February 12, 2014
Grammar Terminology

Grammar

actions = action*

action = dep | deb

dep = "deposit" account amount

deb = "debit" account amount

account = digit{3}

amount = $digit+"."digit{2}

digit = ["0"-"9"]
8
Start symbol
Non-terminals
Production rule
Terminals
How many terminals?
How many productions?
How many derivable strings?
16-Syntax - February 12, 2014
Terminology

Ground String: A (valid) string belonging to the language of

the grammar.

Mutation Operator: A rule that species syntactic

variations of strings generated from a grammar.

Mutant: The result of one application of a mutation

operator to a ground string.

Mutants may be generated either by modifying existing

strings or by changing a string while it is being generated.
9
16-Syntax - February 12, 2014
Mutation Testing

We will see two types of mutation testing:

Input-space grammars: to create inputs (both valid

and invalid)

Program-based grammars: to modify programs

(valid)
10
16-Syntax - February 12, 2014
Testing Invalid Inputs

Your program accepts strings described by a regex or a

grammar.

Was it implemented correctly?

Reasons to test invalid inputs:

Often implementers overlook invalid inputs;

Undened behaviour not acceptable (e.g. buffer overruns).

Solution: Mutate grammars and generate test strings

from the mutated grammars.
11
16-Syntax - February 12, 2014
Some Grammar Mutation Operators

Nonterminal Replacement

dep = "deposit" account amount =>

dep = "deposit" amount amount

Use your judgement to replace nonterminals with

similar nonterminals.

Terminal Replacement

amount = $digit+"."digit{2} =>

amount = $digit+"$"digit{2}
12
deposit $9.22 $12.35
ground string:
deposit 123 $12.35
deposit 123 $12$35
16-Syntax - February 12, 2014
More Grammar Mutation Operators

Terminal and Nonterminal Deletion; e.g.

dep = "deposit" account amount =>

dep = "deposit" amount

Terminal and Nonterminal Duplication; e.g.

dep = "deposit" account amount =>

dep = "deposit" account account amount

13
deposit $12.35
deposit 123 123 $12.35
16-Syntax - February 12, 2014
Killing Mutants

Test case t kills m if running t on m produces

different output than running t on m0,

where m is a mutant for an original ground string

program m0.
14
16-Syntax - February 12, 2014
Recall the Terminology

Ground String: A (valid) string belonging to

the language of the grammar.

Mutation Operator: A rule that species

syntactic variations of strings generated
from a grammar.

Mutant: The result of one application of a

mutation operator to a ground string.
15
16-Syntax - February 12, 2014
Recall Production

Grammar

actions = action*

action = dep | deb

dep = "deposit" account amount

deb = "debit" account amount

account = digit{3}

amount = $digit+"."digit{2}

digit = ["0"-"9"]
16
Production rule
16-Syntax - February 12, 2014
Coverage Criteria

Mutation Coverage (MC). For each mutant m, TR contains a

requirement to kill m".

Mutation score is the percentage of mutants killed.

Mutation Operator Coverage (MOC). For each mutation

operator op, TR contains requirement to create a mutated
string m derived using op.

Mutation Production Coverage (MPC). For each mutation

operator op and each production p that op can be applied to,
TR contains requirement to create a mutated string from p.
17
16-Syntax - February 12, 2014
Mutation Testing

We will see two types of mutation testing:

Input-space grammars: to create inputs (both valid

and invalid)

Program-based grammars: to modify programs

(valid)
18
16-Syntax - February 12, 2014
Mutant Testing: Program-based Grammars

Generating mutants by modifying programs according

to the language grammar, using mutation operators.

Mutants are valid programs (not tests) which ought to

behave differently from the ground string.

The goal of mutation testing is to create tests which

distinguish mutants from originals.
19
16-Syntax - February 12, 2014
Example
20
int foo(int x, int y) { // original
if (x > 5)
return x + y;
else
return x;
}
int foo(int x, int y) { // mutant
if (x > 5)
return x - y;
else
return x;
}

What test case can kill the mutant?

Once we nd a test case that kills a mutant, we can

forget the mutant and keep the test case. The mutant
is then dead.
16-Syntax - February 12, 2014
Uninteresting Mutants

Three kinds of mutants are uninteresting:

Stillborn: such mutants cannot compile (or immediately

crash);

Trivial : killed by almost any test case;

Equivalent: indistinguishable from original program.

The usual application of program-based mutation is

to individual statements in unit testing.
21
16-Syntax - February 12, 2014
Example Mutants
22
Original Method

int Min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min
Mutant

int Min (int A, int B)
{
int minVal;
minVal = A;
! 1 minVal = B;
if (B < A)
! 2 if (B > A)
! 3 if (B < minVal)
{
minVal = B;
! 4 Crash ();
! 5 minVal = A;
! 6 minVal = failOnZero (B);
}
return (minVal);
} // end Min
6 mutants
Each represents a
separate program
Replace one variable
with another
Changes operator
Immediate runtime
failure if reached
Immediate runtime
failure if B==0 else
does nothing
16-Syntax - February 12, 2014
Goals of Mutation Testing

1. Mimic (and hence test for) typical mistakes

2. Encode knowledge about specic kinds of

effective tests in practice

Statement coverage (!4)

Checking for 0 values (!6)

23
16-Syntax - February 12, 2014
Process of Using Mutation Testing

Goal

Kill mutants

Desired side effect

Good tests which kill the mutants.

The tests will help nd faults.

We nd these tests by intuition and analysis.

24
16-Syntax - February 12, 2014
Strong and Weak Mutation

Strong mutation: A fault must be reachable, infect the

state, and propagate to output.

Weak mutation: A fault which kills a mutant need only

be reachable and infect the state.

The book claims that experiments show that weak and

strong mutation require almost the same number of
tests to satisfy them.
25
16-Syntax - February 12, 2014
Strong Mutation Coverage (SMC)

Strongly Killing Mutants: Given a mutant m for a

program P and a test t, t is said to strongly kill m iff
the output of t on P is different from the output of t
on m.

Strong Mutation Coverage (SMC). For each mutant

m ! M, TR contains a test which strongly kills m.
26
16-Syntax - February 12, 2014
Weak Mutation Coverage (WMC)

Weakly Killing Mutants: Given a mutant m that

modies a source location L in program P and a test
t, t is said to weakly kill m iff the state of the
execution of P on t is different from the state of the
execution of m on t, immediately after some
execution of L.

Weak Mutation Coverage (WMC). For each mutant

m ! M, TR contains a test which weakly kills m.
27
16-Syntax - February 12, 2014
Condition Analysis
28
Original Method
int Min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min
Mutant
int Min (int A, int B)
{
int minVal;
! 1 minVal = B;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min

Reachability:

Infection:

Propagation:
Unavoidable;
Need B " A;
Need B > A;
Wrong minVal needs to return to the caller; that is, we can't
execute the body of the if statement.
16-Syntax - February 12, 2014
Conditions

Reachability: unavoidable;

Infection: need B ! A;

Propagation: need B > A.

Wrong minVal needs to return to the caller; that is, we
can't execute the body of the if statement.

Condition for weakly killing mutant: B " A

Condition for strongly killing mutant: B > A

29
16-Syntax - February 12, 2014
WMC vs SMC
30
Original Method
int Min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min
With Embedded Mutants
int Min (int A, int B)
{
int minVal;
! 1 minVal = B;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min

Condition for strongly killing mutant B > A

A test case is A = 5, B = 7 (return value =7 , expected = 5).

Condition for weakly killing mutant: B " A

A test case is A = 8, B = 2 (return value = 2, expected = 2).

16-Syntax - February 12, 2014
Another Example
31
Original Method

int Min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min
With Embedded Mutants

int Min (int A, int B)
{
int minVal;
minVal = A;
! 3 if (B < minVal)
{
minVal = B;
}
return (minVal);
} // end Min

This mutant is an equivalent mutant, since A = minVal. (The

infection condition boils down to false.)

Equivalence testing is, in its full generality, undecidable, but we

can always estimate.
16-Syntax - February 12, 2014
Workow of Mutation Testing
32
16-Syntax - February 12, 2014
Mutation Operators
33
int mutationTest ( int a , b) {
int x = 3 * a , y ;
if (a > b) {
y = -x ;
}
else if ( ! ( a > -b ) ) {
x = a * b ;
}
return x ;
}
What (intra-procedural) mutation operators can
you come up with?

Absolute Value Insertion

Operator Replacement

Arithmetic, Relational, ...

Scalar Variable Replacement

Crash Statement replacement

16-Syntax - February 12, 2014
Interface/Integration Mutation

Change calling method by

changing actual parameter values;

Change calling method by

changing callee; or

Change callee by modifying the

return statement.
34
class M {
int f, g;
void c(int x) {
foo (x, g);
bar (3, x);
}
int foo(int a, int b) {
return a + b * f;
}
int bar(int a, int b) {
return a * b;
}
}
16-Syntax - February 12, 2014
Summary of Syntax-based Testing

Program-based testing has notion of strong and

weak mutants.

Sometimes we mutate the grammar, not strings, and

get tests from the mutated grammar.
35
16-Syntax - February 12, 2014
PIT Mutation Testing Tool

Conditionals Boundary Mutator

Negate Conditionals Mutator

Remove Conditionals Mutator

Math Mutator

Increments Mutator

Invert Negatives Mutator

Inline Constant Mutator

Return Values Mutator

Void Method Calls Mutator

Non Void Method Calls Mutator

Constructor Calls Mutator

Experimental Inline Constant Mutator

Experimental Member Variable Mutator

Experimental Switch Mutator

36

http://pitest.org/
16-Syntax - February 12, 2014