Writing a Test Strategy

Tor Stlhane

Why a testing strategy

We need a testing strategy to help
System and software testers plus Test-andEvaluation staff to determine the overall
test strategy for development or
modification of a software intensive system
Project stakeholders customers and senior
management approve the test strategy
Testers and system and software analysis to
determine
Test objectives
Qualification requirements
Verification and validation criteria

Testing strategy concepts

We will discuss the following concepts:
Purpose of a test strategy
Testing focus
Contents of a test strategy
Software integrity levels
Test objectives and priorities

Purpose of a test strategy

1
The test strategy is important to
Obtain consensus on test goals and
objectives from stakeholders e.g.
management, developers, testers,
customers and users
Mange expectations right from the start
Be sure that we are heading in the right
direction
Identify the type of tests to be
conducted at all test levels

Purpose of a test strategy

2
When we write a test strategy is important to
remember that:
Whatever we do some kind of test strategy
will emerge. Thus, we might as well specify
the one we think is best
A documented strategy is the most
effective way to get an early agreement on
goals and objectives
We need to address:
Human factors usability
Interoperability, except for stand-alone
systems

Testing focus
Our focus will depend on which stakeholder we
are considering at the moment:
Users acceptance test and operational tests
Analysis systems test, qualification tests
Designer integration tests
Programmer unit tests
The main point is that we need to define the
stakeholder first then the tests to be run.

Contents of a test strategy

1

The following is a list of what can be

specified in a test strategy. Not all of
it is needed in all cases only use
what is necessary.
Project plan, risks and activities
Relevant regulations depending of
application area
Required processes and standards
Supporting guide lines

Contents of a test strategy

2

Stakeholders e.g. users, testers,

maintainers and their objectives
Necessary resources people,
computers
Test levels and phases
Test environment
Completion criteria for each phase
Required documentation and review
method for each document

Software integrity level

There are several ways to define
software integrity levels. When we
choose an integrity level this will
strongly influence the way we do
testing.
We will look at three definitions of
integrity levels:
IEEE 1012 general software
ISO 26262 automotive software
IEC 61508 general safety critical
software

IEEE 1012 general

software
4, High some functions affect critical
system performance.
3, Major some functions affects
important system performance
2, Moderate some functions effect
system performance but workarounds
can be implemented to compensate.
1, Low some functions have noticeable
effect on system performance but
creates only inconveniences

V&v Activities
V&V activity
SW Integrity Level

Development
requirements
level
4

Design
level

3 2 1 4 3 2 1

Implementation
level

Test level

4 3 2

Acceptance test execution

X X X

Acceptance test plan

X X

Interface analysis

X X

X X X

X X

Management and review

support

X X

Management review of
V&V

X X

X X X X X

X X

X X
X

ISO 26262 automotive

software
The ASIL level A, B, C or D is the
outcome of the combination of three
factors:
S Severity. How dangerous is a
event
E Probability. How likely is the event
C Controllability. How easy the
event to control if it occurs

Finding the ASIL level

Severity

S1

S2

S3

Probability

C1

C2

C3

E1

QM

QM

QM

E2

QM

QM

QM

E3

QM

QM

E4

QM

E1

QM

QM

QM

E2

QM

QM

E3

QM

E4

E1

QM

QM

E2

QM

E3

E4

Methods for software

integration testing
Methods and Measures

According
to req.

ASIL
A

Requirements based test

9.4.4

++

++

++

++

External interface test

9.4.4

++

++

++

Fault injection test

9.4.4

++

++

Error guessing test

9.4.4

++

++

Methods for software unit

testing
Methods and Measures

According
to req.

ASIL
A

Functional tests

8.4.2

See table 8.2

Structural coverage

8.4.2

See table 8.3

Resource usage measurement

8.4.2

Back-to-back test between

simulation model and code, if
applicable

8.4.2

++ ++

++

IEC 61508 safety critical

software
Safety
integrity
level

High demand or continuous mode of

operation
(Probability of a dangerous failure per
hour)

10 -9 to 10 -8

10 -8 to 10 -7

2
1

10 -7 to 10 -6
10 -6 to 10 -5

PFDavg = Ft / Fnp . The table above, together with this

value decides the SIL level.

Detailed design
Technique/Measure

SIL1

SIL2

SIL3

SIL4

HR

HR

HR

HR

Table B.7

HR

HR

HR

1c Formal methods including for example,

CCS, CSP, HOL, LOTOS, OBJ, temporal
logic, VDM and Z
2 Computer-aided design tools

C.2.4

---

HR

B.3.5

HR

HR

Defensive programming

C.2.5

---

HR

HR

Modular approach

Table B.9

HR

HR

HR

HR

Design and coding standards

Table B.1

HR

HR

HR

Structured programming

C.2.7

HR

HR

HR

HR

Use of trusted/verified software modules

and components (if available)

C.2.10
C.4.5

HR

HR

HR

1a Structured methods including for example,

JSD, MASCOT, SADT and Yourdon
1b Semi-formal methods

Ref
C.2.1

Appropriate techniques/measures shall be selected according to the safety integrity level.

Alternate or equivalent techniques/measures are indicated by a letter following the number. Only
one of the alternate or equivalent techniques/measures has to be satisfied.

Module testing and integration

Technique/Measure
1
Probabilistic testing

Ref
C.5.1

SIL1
---

SIL2
R

SIL3
R

SIL4
HR

Dynamic analysis and testing

B.6.5
Table B.2

HR

HR

HR

3
4

Data recording and analysis

Functional and black box testing

C.5.2
B.5.1
B.5.2
Table B.3

HR
HR

HR
HR

HR
HR

HR
HR

Performance testing

C.5.20
Table B.6

HR

HR

Interface testing

C.5.3

HR

HR

a) Software module and integration testing are verification activities (see table A.9).
b) A numbered technique/measure shall be selected according to the safety integrity level.
c) Appropriate techniques/measures shall be selected according to the safety integrity level.

Test objectives and priorities

Only in rather special cases can we test all
input binary input output and few
parameters. Thus, we need to know
The overall objective of testing
The objective of every test case
The test case design techniques needed
to achieve our goals in a systematic way.
The test objectives are our requirements
specification for testing.

Test data selection

One of the important decisions in selecting
a test strategy is how to select test data.
We will look at five popular methods
Random testing
Domain partition testing
Risk based testing
User profile testing
Bachs heuristic risk-based testing

Random testing
The idea of random testing is simple:
1.Define all input parameters e.g. integer,
real, string
2.Use a random test / number generator to
produce inputs to the SUT
The main problem with this method is lack of
an oracle to check the results against. Thus,
manual checking is necessary.
The method is mostly used for crash testing
(robustness testing) will the system
survive this input?

Domain partition testing 1

Definitions:
A domain is a set of input values for
which the program performs the same
computation for every number of the
set. We want maximal domains so that
the program performs different
computations on adjacent domains
A program is said to have a domain error
if the program incorrectly performs input
classification

Domain testing simple

example
aX

bX c 0

If a >< 0, this equation has the following solution

b
X

2a

b
c

2
4a
a

Otherwise we have that

c
X
b

Testing domains
Aa ><
=0
Aa =
= 00

D3

b2
c

A = 0 0
4a 2
a

D1
D2
b2 c
A = 0 0
4a 2 a

Risk based testing

The idea of risk based testing is to
1.Identify the risk or cost of not
delivering a certain functionality.
2.Use this info to prioritize tests. We
will cover this in more details later
under Test prioritation

User profile testing

The main idea with this type of testing is
to generate tests that mirror the users
way of using the system.
Consider a situation where we know that
the users in 80% of all case
Fetch a table from the database
Update one or more info items
Save the table back to the database

Then 80% of all tests should test these

three actions.

Bachs risk-based testing

Bachs heuristics is based on his
experience as a tester. Based on this
experience he has identified
A generic risk list things that are
important to test
A risk catalogue things that often go
wrong
We will give a short summary of the first
of Bachs lists.

Bachs generic risk list 1

Look out for anything that is:
Complex large, intricate or convoluted
New no past history in this product
Changed anything that has been tampered
with or improved
Upstream dependency a failure here will
cascade through the system
Downstream dependency sensitive to failure in
the rest of the system
Critical a failure here will cause serious
damage

Bachs generic risk list 2

Precise must meet the requirements exactly
Popular will be used a lot
Strategic of special importance to the users
or customers
Third-party developed outside the project
Distributed spread out over time or space
but still required to work together
Buggy known to have a lot of problems
Recent failure has a recent history of
failures.

Test and system level 1

Test and system level 2

From the diagram on the previous slide
we see that we can test on the
Electronics level e.g. DoorActuator
sends the right signal
State / signal level e.g. door is closed
iff DoorStateClosed
Logical level e.g. the door remain
closed as long as the speed is non-zero
Safety level e.g. the door remain
closed as long as the train is moving

Acknowledgement
The first part of this presentation is
mainly taken from Gregory T. Daichs
presentation Defining a Software
Testing Strategy, 30 April 2002.