Example: Execution 2 (The Opposite

Path)
NotSet(page) page2 1337 login = 1
Constraint solver may get page2 0; login
1
HTML validation tool discovers failure and
generates bug report added to output set
true
of bug reports

true

Minimization on Path Constraints

Find shorter path constraint for a given bug
report
Eliminates irrelevant constraints better assist
programmer to detect location of the fault
Solution for a shorter path constraint is often
a smaller input
Does not guarantee returned path constraint
is shortest that exposes failure

Minimization Example
HTML malformation from previous example could
have been reached from different execution
paths
NotSet(page) page2 1337 login = 1
page2 1337 login = 1

Set(page) page = 0 page2 1337 login = 1

page2 1337

login = 1 (login 1)

Path Constraint Minimization

Algorithm

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

1.

parameters: Program P, oracle O, bug report b

result : Short path constraint that exposes b.failure
c1 . . . cn intersect(b.pathConstraints);
pc true;
foreach i = 1, . . . , n do
pci c1 . . . ci1 ci+1 . . . cn;
input solve(pci);
if input not equals then
output executeConcrete(P, input);
failures getFailures(O, output);
if b.failure not belongs to failures then
pc pc ci;
input pc solve(pc);
if input pc not equals to then
outputpc executeConcrete(P, input pc );
failurespc getFailures(O, outputpc );
if b.failure failurespc then
return pc;
return shortest(b.pathConstraints);

Apollo
User Input Simulator
Executor
Bug Finder
Oracle
Bug Report Repository
Input minimizer

Input Generator
Symbolic Finder
Constraint Solver
Value Generator

Apollo

Executor: Shadow
Interpreter
Shadow Interpreter
Modified Zend PHP interpreter 5.2.2 to record
path constraints and information associated with
output
Performs symbolic execution along with concrete
execution
Records conditions for PHP-specific comparison
operations such as isset and empty

Executor: Database Manager

Database Manager
(Re) initializes DB used by a PHP application.
Restores DB before each execution
Supply additional information about
username/password pairs

Bug Finder
Bug Report = Failure + Path constraint + Input
inducing failure
Failure = Type of Failure + Corresponding
Message + PHP statement generating bad
HTML
Oracle HTML validation tool (WDG and WC3)
Input Minimizer uses the path constraints
minimization algorithm

Input Generator
Symbolic Driver generates new path
constraints and select next path constraint
Constraint Solver computes an assignment
of values to input parameters that satisfies a
given path constraint.
Choco constraint solver

Value Generator generates value for

parameters
Combines random value generation and constant
values mined from source code

Experimentation
Program

#files

LOC

PHP LOC

# DLs

faqforge

19

1712

734

14164

webchess

24

4718

2226

32352

schoolmate

63

8181

4263

4466

phpsysinfo

73

16634

7745

492217

179

31245

14968

543199

total

faqforge = Tool for creating and managing documents

webchess = Online chess game
schoolmate = PHP/MySQL solution for administering schools
phpsysinfo = Displays system info

Generation Strategies
Compared to two other approaches
Halfond and Orso (Randomized)
Random values to the parameters
Proposed for JavaScript

Minamides static analysis

Approximates the string output of program with a
context-free grammar
Discovers malformed HTML faults

Apollos test input generation previously

discussed

Methodology
10-minute runs on each program
Generation of hundreds of inputs

Ran on both Apollo and Random test input

generation strategies
WDG offline HTML validation tool

Results Classification
Execution crash: PHP interpreter terminates
with exception
Execution error: PHP interpreter emits
warning visible in generated HTML
Execution warning: PHP interpreter emits
warning invisible to HTML output
HTML error: program generates HTML for
which validation tool produces error report
HTML warning: program generates HTML for
which validation produces a warning report

Results Analysis
Resulted in Malformed HTML
Tries to load two missing files

Database related
Apollo

Unset Time-zone

Average line coverage 58.0%

Faults Found on Subject Apps 214

Randomized

Average line coverage 15.0%

Faults Found on Subject Apps 59

Line Coverage = Number of executed lines / Total lines with executable PHP code in application

Results Analysis
Apollo Vs Randomized
58% line coverage Vs 15.2% line coverage
214 faults Vs 59 faults

Apollo Vs Minamides tool

2.7 more HTML validation faults (120 Vs 45)
83 additional execution faults
104 faults (10 minutes) Vs 14 faults (126 minutes)

Apollo is more effective and efficient than

both

Results Analysis: Path Constraint

Minimization
Path Constraints
Program

Success rate %

Orig. Size

Inputs

Reduction Orig. Size

Reduction

faqforge

64

22.3

0.22

9.3

0.31

webchess

91

23.4

0.19

10.9

0.40

schoolmate

51

22.9

0.38

11.5

0.58

Reduces82size of inputs24.3
by up to factor
for
0.18 of 0.1817.5
more than 50% of faults

0.26

phpsysinfo

Success rate Percentage of faults whose exposing input was minimized

Orig. size Average size of original path constraints (# of conjuncts) and inputs (# of key-value
pairs)
Reduction columns Ratio of minimized to un-minimized size. The lower the ratio, the more
successful the minimization

Limitations
Simulating user inputs statically
JavaScript code in the generated HTML not
tracked
Limited line coverage for native C methods
Limited sources of input parameters
Only inputs from global arrays (_POST, _GET
and _REQUEST)

Thank you