Testing objectives

In an excellent book on software testing, Glen

Myers states a number of rules that can serve
.well as testing objectives:
Testing is a process of executing a program
with the intent of finding an error.
A good test case is one that has a high
probability of finding an as-yet undiscovered
error.
A successful test is one that uncovers an
as-yet-undiscovered error.

Testing principles

All tests should be traceable to customer requirements

Tests should be planned long before testing begins

The Pareto principle applies to software testing

Testing should begin in the small and progress toward testing in
the large
Exhaustive testing is not possible
To be most effective, testing should be conducted by an

independent third party

Attributes of a good test

Kaner, Falk, and Nguyen suggest the following attributes of a good test:

A good test has a high probability of finding an error.

To achieve this goal, the tester must understand the software
and attempt to develop a mental picture of how the software might
fail. Ideally, the classes of failure are probed.

A good test is not redundant.

Testing time and resources are limited. There is no point in
conducting a test that has the same purpose as another test. Every
test should have a different purpose (even if it is subtly different)

Types of testing
The various types of testing which are often used
are listed below:

White Box Testing

Black Box Testing
Integration Testing
System and Acceptance Testing
Performance Testing
Regression testing
Internationalization Testing and
Ad hoc Testing

White box testing

White box testing is a way of testing the external
functionality of the code by examining and testing the
program code that realizes the external functionality.
This is also known as clear box, or glass box or open
box testing.
White box testing takes into account the program code,
code structure, and internal design flow.
White box testing is classified into static and structural
testing.

White box testing

Using white-box testing methods, the software engineer
can derive test cases that
guarantee that all independent paths within a module
have been exercised at least once,
exercise all logical decisions on their true and false
sides,
execute all loops at their boundaries and within their
operational bounds, and
exercise internal data structures to ensure their validity.

Classification of white box testing

White box testing

White box testing

Static testing

Structural testing

Desk checking

Code
walkthrough
Code
Inspection

Unit/code
Functional
testing

Code
coverage
Statement
Coverage
Path
coverage
Condition
coverage
Function
coverage

Code
complexity

Cyclomatic
complexity

Static testing
Static testing requires only the source code of the product,
not the binaries or executables.
Static testing does not involve executing the programs on
computers but involves select people going through the
code to find out whether
The code works according to the functional requirement
The code has been written in accordance with the design
developed
Earlier in the project life cycle
The code for any functionality has been missed out
The code handles errors properly
Static testing can be done by humans or with the help of
specialized tools.

Static testing by human

These methods rely on the principle of humans reading the program code to
detect errors rather than computers executing the code to find errors.
This process has several advantages.

Sometimes human can find errors that computers can not

By making multiple humans read and evaluate the program, we can get
multiple perspectives and therefore have more problems identified
upfront than a computer could.
A human evaluation can detect many problems at one go and can even
try to identify the root causes of the problems.
By making humans test the code before execution, computer resources
can be saved.
A proactive method of testing like static testing minimizes the delay in
identification of the problems.
From a psychological point of view, finding defects later in the cycle
creates immense pressure on programmers. They have to fix defects
with less time to spare.

Methods for static testing by human

There are multiple methods to achieve static testing by humans. They
are

Desk checking of the code

Code walk through

Code review

Code inspection

Desk checking

Normally done manually by the author of the code

to verify the portions of the code for correctness.
Such verification is done by comparing the code with
the design or specifications to make sure that the code
does what it is supposed to do and effectively.
Whenever errors are found the author applies the
correction for errors on the spot. This method of
catching and correcting errors is characterized by
No structured method or formulation to ensure
completeness and
No maintaining of a log or check list.

Disadvantages of desk checking

Some of the disadvantages of this method of testing are as

follows:
A developer is not the best person to detect problems in his own
code.
Developers generally prefer to write new code rather than any
form of testing.
This method is essentially person dependent and informal.

Code walkthrough

Walkthroughs are less formal than inspection.

The advantage that walkthrough has over desk checking
is that it brings multiple prospective.
In walkthroughs, a set of people look at the program
code and raise questions for the author.
The author explains the logic of the code and answers
the questions.

Formal inspection

Code inspection also called Fagan inspection is a

method normally with a high degree of formalism.

The focus of this method is to detect all faults, violations,

and other side effects.

Code review checklist

Data item declaration related
Data usage related
Control flow related
Standards related
Style related
Documentation related

Structural testing
Structural testing takes into account the code, code structure, internal
design, and how they are coded.
In structural testing tests are actually run by the computer on the built
product, whereas in static testing the product is tested by humans
using just the source code and not the executables or binaries.
Structural testing can be further classified into

Unit/code functional testing,

Code coverage and
Code complexity testing.

Unit/code functional testing

Initially the developer can perform certain obvious tests, knowing the
input variables and the corresponding expected output variables.
This can be a quick test that checks out any obvious mistakes

By repeating these tests for multiple values of input variables, the

confidence level of the developer to go to the next level increases.

For modules with complex logic or conditions, the developer can

build a debug version of the product by putting intermediate print
statements

Another approach to do the initial test is to run the product under a

debugger or an Integrated Development Environment (IDE).

Code coverage testing

Code coverage testing involves designing and executing test cases
and finding out the percentage of code that is covered by testing.
The percentage of code that is covered by a test is found by adopting
a technique called instrumentation of code.
There are specialized tools available to achieve instrumentation.
The tools also allow reporting on the portions of the code that are covered
frequently, so that the critical or most-often portions of code can be
identified.
Code coverage testing is made up of the following types of coverage.
Statement coverage
Path coverage
Condition coverage
Function coverage

Statement coverage
Program constructs in most conventional programming languages can be
classified as

Sequential control flow

Two-way decision statements like if then else

Multi-way decision statements like switch

Loops like while do, repeat until and for

Object-oriented languages have all of the above and, in addition, a number of other
constructs and concepts. Statement coverage refers to writing test cases that
execute each of the program statements.

Statement coverage = (total statements exercised/

total number of executable statements in program) * 100

Path coverage

In path coverage, we split a program into a number of distinct paths.

A program can start from the beginning and take any of the paths to
its completion.

Path coverage provides a stronger condition of coverage than

statement coverage as it relates to the various logical paths in the
program rather than just program statements.

Path coverage= ( total paths exercised/

total number of paths in program) * 100

Condition coverage

In the path coverage testing though we have covered all the paths
possible, it would not mean that the program is fully covered.

Condition coverage = ( Total decisions exercised /

Total number of decisions in program)*100

The condition coverage as defined by the formula alongside in the

margin gives an indication of the percentage of conditions covered
by a set of test cases.

Condition coverage is a much stronger criteria than statement

coverage.

Function coverage
This is a new addition to structural testing to identify how many
program functions are covered by test cases.
The advantages that function coverage provides over the other
types of coverage are as follows:

Functions are easier to identify in a program and hence it is easier to

write test cases to provide function coverage.
Since functions are at a much higher level of abstraction than code,
it is easier to achieve 100 percent function coverage than 100
percent coverage in any of the earlier methods.
Functions have a more logical mapping to requirements and hence
can provide a more direct correlation to the test coverage of the
product.
Function coverage provides a natural transition to black box testing.
Function coverage = (Total functions exercised/
Total number of functions in program) * 100

Code complexity testing

Cyclomatic complexity is a software metric that provides a quantitative
measure of the logical complexity of a program

Cyclomatic complexity defines the number of independent paths in

the basis set of a program and provides us with an upper bound for
the number of tests that must be conducted to ensure that all
statements have been executed at least once.

An independent path is any path through the program that

introduces at least one new set of processing statements or a new
condition.

Cyclomatic complexity
In order to convert a standard flowchart into a flow graph to compute
cyclomatic complexity, the following steps are taken:
Identify the predicates or decision points in the program.
Ensure that the predicates are simple
Combine all sequential statements into a single node
When a set of sequential statements are followed by a simple
predicate, combine all the sequential statements and the predicate
check into one node and have two edges emanating from this one
node. Such nodes with two edges emanating from them are called
predicate nodes.
Make sure that all the edges terminate at some node; add a node to
represent all the sets of sequential statements at the end of the
program

Flow graph translation

A predicate with a Boolean OR

true
If( A or B)

false

Flow graph translation of an OR to a simple

predicate
An equivalent set of simple predicates

True
True
IF ( B)

IF(A)

False

False

Cyclomatic complexity
A hypothetical program with no decision node
# of independent paths = 1
# of nodes, N = 2
# of edges, E = 1
Cyclomatic complexity = E-N+2 = 1
# of predicate nodes, P=0
End

Cyclomatic complexity = P + 1 = 1

Cyclomatic complexity
Converting a conventional flowchart to a flow graph;

Cyclomatic complexity
Adding one decision node

# of independent paths = 2

# of nodes, N=4
# of edges, E=4
Cyclomatic complexity=E-N+2=2
# of predicate nodes, P=1
Cyclomatic complexity = P +1 = 2

End

Cyclomatic complexity

Complexity

What it means

1 - 10

Well-written code, testability is high, cost/effort to maintain is low

10-20

Moderately complex, testability is medium, cost/effort to maintain is

medium

20-40

Very complex, testability is low, cost/effort to maintain is high

>40

Not testable, any amount of money/effort to maintain may not be

enough

Cyclomatic complexity
Cyclomatic complexity = number of predicate nodes + 1
Cyclomatic complexity = E N + 2
Where, E is the number of edges and N is the number of nodes

Challenges in white box testing

White box testing requires a sound knowledge of the program
code and the programming language. This means that the
developers should get intimately involved in white box testing

Human tendency of a developer being unable to find the defects in

his code.

Fully tested code may not correspond to realistic scenarios

These challenges do not mean that white box testing is
ineffective. But when white-box testing is carried out and these
challenges are addressed by other means of testing, there is a
higher likelihood of more effective testing.

Black box testing

Black box testing involves looking at the
specifications and does not require examining the
code of the program.
This testing is done from the customers view
The test engineer engaged in black box testing only
knows the set of inputs and expected outputs.
This testing is done without the knowledge of the
internals of the system under test.
It is convenient to administer because they use the
complete finished product and do not require any
knowledge of its construction.

Why black box testing

Black box testing helps in the overall functionality verification
of the system under test.
Testing is done based on the requirements
This testing addresses stated as well as implied requirements
This testing encompasses the end user perspectives
Black box testing handles valid and invalid inputs

When to do black box testing

Black box testing activities require involvement
of the testing team from the beginning of the
project life cycle
Testers can get involved right from the
requirements gathering and analysis phase for
the system under test.
Once the code is ready and delivered for testing,
test execution can be done
All the test scenarios developed during the
construction phase are executed

How to do black box testing

Black box testing exploits specifications to generate test cases in a methodical
way to avoid redundancy and to provide better coverage.
The various techniques used for black box testing are as follows:
Requirements based testing
Positive and Negative testing
Boundary Value analysis
Decision tables
Equivalence partitioning
State based testing
Compatibility testing
User documentation testing
Domain testing

Requirements Based testing

Requirements testing deals with validating the requirements
given in the Software Requirements Specification (SRS) of the
system.
The precondition for requirements testing is a detailed review
of the requirements specification.
Requirements review ensures that they are consistent, correct,
complete, and testable.
All explicit requirements and implied requirements are
converted and documented as explicit requirements, there by
bringing better clarity to requirements.
All explicit and implied requirements are collected and
documented as Test Requirements Specification (TRS).

Sample Requirements Traceability Matrix

(RTM)
Req.ID

Description

Priorit
y

Test
conditions

Test case
IDs

Phase of
testing

BR-01

Inserting the key

numbered 123 and
turning it
clockwise should
facilitate locking

Use key 123

Lock_001

Unit,
Component

BR-02

Inserting the key

numbered 123 and
turning it
anticlockwise
should facilitate
unlocking

Use key 123

Lock_002

Unit,
Component

BR_03

Only key number

123 can be used to
lock and unlock

Use key 123

to lock
Use key 123
to unlock

Lock_003
Lock_004

Component

BR-04

No other object
can be used to lock

Use key 456

Use Hairpin
Use
Screwdriver

Lock_005
Lock_006
Lock_007

Integration

BR-05

No other object
can be used to
unlock

Use key 456

Use Hairpin

Lock_008
Lock_009

Integration

Sample test execution data

S.no.

Req.ID

Priority

Test
cases

Total
test
cases

Test
cases
passed

Test
cases
failed

% pass

No. of
defects

BR-01

Lock-01

100

BR-02

Lock_02 1

100

BR-03

Lock_03,
04

50

BR-04

Lock_05,
06,07

67

BR-05

Lock_08,
09,10

100

BR-06

Lock_11

100

BR-07

Lock_12

100

BR-08

Total

12

10

83

12

Positive and negative testing

Positive testing tries to prove that a given product does
what it is supposed to do.
When a test case verifies the requirements of the
product with a set of expected output, it is called positive
test case.
The purpose of positive testing is to prove that the
product works as per specification and expectations.
A product delivering an error when it is expected to give
an error, is also a positive testing.
Positive testing can thus be said to check the products
behavior for positive and negative conditions as stated in
the requirement.

Example of positive test case

Req.no.

Input 1

Input 2

Current
state
Unlocked

Expected
output
locked

BR-01

Key 123

BR-01

Key 123

Turn
clockwise
Turn
clockwise

Locked

No
change
No
Change
Unlock

BR-02

Key 123

Turn
anticlockwise

Unlocked

BR-02

Key 123

Turn
anticlockwise

Locked

BR-04

Hairpin

Turn
clockwise

Locked

No
change

Negative testing
Negative testing is done to show that the product does not fail
when an unexpected input is given
The purpose of negative testing is to try and break the system.
Negative testing covers scenarios for which the product is not
designed and coded
A negative test would be a product not delivering an error
when it should or delivering an error when it should not.
In positive testing if all documented requirements are covered,
then coverage can be considered to be 100 percent.
In contrast, there is no end to negative testing, and 100 percent
coverage in negative testing is impractical.
Negative testing requires a high degree of creativity among the
testers to cover as many unknowns as possible to avoid
failure at customer site.

Example for negative test case

S.No.

Input 1

Input 2

Current
state

Expected
output

Some other Turn

locks key clockwise

Lock

Lock

Some other Turn

locks key clockwise

Unlock

Unlock

Thin piece
of wire

Unlock

unlock

Hit with a
stone

Lock

Lock

Turn
anticlockwise

Boundary value analysis

Conditions and boundaries are two major sources of defects in
a software product
By boundaries, we mean limits of values of the various
variables
BVA, a method useful for arriving at tests that are effective in
catching defects that happen at boundaries