You are on page 1of 52

1

Testing Awareness
3
Agenda
Testing?
A Sample Test Process
A few Testing Definitions
Test Process Flow
Test roles
Testing realities



4
What is Testing ?
Three integer values representing the lengths of the sides of a triangle, are
the input of the application. The output or result of the application is a
message that the triangle is scalene or equilateral or isocele

Define a set of tests that, to your feeling, would test this application
properly. Enrich your test descriptions by specifying the used set of data
?
A B
C
Scalene : A < B + C
Isocele : B = C and A < B + C
Equilateral : A=B=C and A < B + C


5
Test cases
1. Do you have a test that represents a valid scalene triangle ?
2. Do you have a test that represents a valid equilateral triangle ?
3. Do you have a test that represents a valid isoceles triangle ?
4. Do you have all three permutations of the two equal sides of test 3 (e.g. 3,3,4; 3,4,3; 4,3,3) ?
5. Do you have a test in which one input value is zero ?
6. Do you have a test in which one input is a negative integer ?
7. Do you have a test with all three inputs greater than zero, but with the sum of two of the
input values equal to the third (if the program sais that 1,2,3 represents a scalene triangle, it
would contain a bug) ?
8. Do you have at least all permutations of the previous test ?
9. Do you have a test with three integers greater that zero but with the sum of two of the
numbers less then the third ?
10. Do you have at least all permutations of the previous test ?
11. Do you have a test with all input values being zero ?
12. Do you have at least one test with non-integer inputs ?
13. Do you have a test with the number of input values (two rather then three) ?
14. Did you specify for each test the expected output ?
6
What is testing ?

Testing is executing a software with the
intent to find defect

Exercice feed Back
- Expected result must be define in advance
- Program Algorithm is different from test description
- Valid and Invalid statement must be tested
- Exhaustive testing can be expensive
-




7
But where to stop ?
EXTENT OF TESTING
NUMBER OF
DEFECTS OPTIMUM
TEST
COST OF
TESTING
OVER
TEST
UNDER
TEST
8
Why is Test important ?
1
2,5
5
10
25
100
0
20
40
60
80
100
120
R
e
q
u
i
r
e
m
e
n
t
D
e
s
i
g
n
C
o
d
i
n
g
U
n
i
t

T
e
s
t
A
c
c
e
p
t
a
n
c
e
M
a
i
n
t
e
n
a
n
c
e
Cost to remove an error
9
Errors are unavoidable

What the client wanted What the contract said What the analyst designed
What the developer wrote What was delivered
What was really needed
10
Why Testing?
Defects are discovered sooner
Reduced correction costs!
Quality is evaluated early and iteratively in the
project
Increased visibility!

11
Agenda
Testing ?
A Sample Test Process
A few Testing Concept
Test Process Flow
Test roles
Testing realities



12
Valid for : Corporate Development
(Finance, Governement, distribution,)
1 application is composed of 800 modules imbeded in a
system of 80 000 modules.
A project involve in average between 10 & twenty
people.
Lot of activities on the same system in the same time
User involve in project and elaborate detailled
requirement while you develop
You start always late but you must deliver on time.
13
Test concept 1 :Test stages
Test stages and code validation
System Tests
CODE
Integration Tests
Unit Tests
Acceptance Tests
Software
Architecture (SAD)
Components
Design
Software
Requirements (SRS)
Business needs
(Vision)
VALIDATION
VALIDATION
VALIDATION
VALIDATION
14
Test stages and Test scenarios
creation
System Tests
Integration Tests
Unit Tests
Acceptance Tests
Software
Architecture (SAD)
Components
Design
Software
Requirements (SRS)
Business needs
(Vision)
SCENARIOS CREATION
SCENARIOS CREATION
SCENARIOS CREATION
SCENARIOS CREATION
15
Test stages
The test stages are the key to validate the code along
different point of view and different level of details
Business
Technical
The test stages have a huge impact on how the scenario
are written
Business oriented
Technically oriented
The test stages are related to the maturity of the
software only and independent of the phase
The 4 test stages may be present in each and every iteration

16
Test concepts 2 : Test types
Most common test types
Performance tests (Mainframe response time, Scalability)
Functional tests (Conformance to functional
requirements)
Configuration tests (Different software and hardware on
which the application will run)
Security tests (Role Based Security)
Non-regression tests (No new defects caused by new
code)
Stress tests (Operate the system to its limit)
Recovery tests (Behavior in case of system break-down)
Usability tests (Help on-line, ergonomy,)

17
Test types
Test types are the key to translate test
expectations in a concrete test strategy :

What tests will be performed.
When will which tests be performed.
What skills are needed.
What tools are needed.

18
Test concepts 3 : Test Strategy
Test Strategy is

Best combination off test stages, test types and
application feature to test in order to :

- Anticipate Risk High risk first
- Address test requirement based on priority
- Show quality achievement regularly during
project execution


19
Agenda
Testing ?
A Sample Test Process
A few Testing Concept
Test Process Flow
Test roles
Testing realities



20
Test process overview
Test Strategy & plan



[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of the
project]
Test Strategy and
Planning
Test Design and
Build
Test
Execution
Test
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
Plan

Design
and Build

Execute
and Report

Define test
scenarios
Test model



Frequent
reports




Completed during each
iteration to gain
maturity

21
RUP Lifecycle - Test discipline
22
Test Strategy and Planning
During inception and
elaboration
Define a test strategy able to
meet stakeholders test
expectations.
Define the resources needed to
undertake this strategy
Deliverables
An approved test plan
Optionally a Test Initiation
Checklist
During construction and
transition
Update and detail initial plan
Detailed schedule
Updated scope

[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of the
project]
Test Strategy and
Planning
Test Design and
Build
Test
Execution
Test
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
23
Test Strategy and Planning
Output = Test Plan

Summary of quality
expectations.

Test strategy

Test exit criteria

Test schedule

Reporting facilities

Resources needed by
the strategy






Which test types in which test
stages for which features?
When is testing finished?
Skills, metrics reported, audience,
periodicity
People, budget, tools, test
environments.
Quality Objectives
24
Test Design and Build
Prepare test scenarios needed
during execution.
Structure this material
Deliverables
Tests scenarios
Test model
Automation scenarios
[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of the
project]
Test Strategy and
Planning
Test Design and
Build
Test
Execution
Test
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
25
Each test scenario describes

Unique identifier
Detailed objective for the
scenario
Steps to be executed
Corresponding expected
results
Scenario priority

Unambiguously identify the scenario
for further reference (e.g. defect
reporting)
Assess success or failure of the
scenario
Procedure to be followed by the tester
To be compared against the actual
behavior, base for defect reporting.

Test Design and Build
Output =Test scenarios
Importance of the scenario
26
Test Execution and Reporting
Execute test scenarios
Register the defects
Report execution results
Deliverables
Defects
Reports
[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of the
project]
Test Strategy and
Planning
Test Design and
Build
Test
Execution
Test
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
27
A defect is reported in a defect tracking tool
Difference
Status in the defect life-cycle
Unique identifier
eg. Test tracker, QAA, test director
Link with failed scenario Id
Defect tracking tool
Communication !
Registration
Statistics
A defect is linked to a particular scenario



Test Execution and Reporting
Output = Defects
Defect ARE Key :
-Communication
-Quality Measurment
-Process Improvment
28
Test Execution and Reporting
Output = Report
Defect Status Nb of defects
Open 15
Under correction 14
To be re-tested 35
Under re-test 3
Closed 45
#Defects per status
Open
Under correction
To be re-tested
Under re-test
Closed
There is a
bottleneck there
+ Written conclusions of the analysis
29
Close Test Session
Publish in the test sign-off
document to which extend
the quality expectations are
met
Defects pending
Features tested/ not tested
Acceptance criteria

[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of the
project]
Test Strategy and
Planning
Test Design and
Build
Test
Execution
Test
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
30
Activities workload in each phase
Test strategy
and planning
Test Design and
Build
Test Execution
and Reporting
Close test
session



31
Agenda
Testing ?
A Sample Test Process
A few Testing Concept
Test Process Flow
Test roles
Testing realities



32
Test Roles
Test coordinator
Test analyst
Tester
33
Roles - Test Coordinator
Plan and coordinate test activities and resources.
Control the test execution and report on a regular basis, the
test status and progress.
Assertive - Independent minded
Owner of the test plan and the test model
Good relational skills
High workload during
Strategy and planning
Execution and reporting
34
Roles - Test Analyst
Delivers the test model and test scenarios.
Deep understanding and knowledge of the project technical
specifications and business requirements.
Skilled in Test Specification Techniques (Boundary analysis,
decision tables, equivalence classes).
Owner of the test scenarios
Details oriented
High workload during design and build
35
Roles - Tester
Execute scenarios, report execution results, and register defects.
Correct understanding and knowledge of the application
technical and functional specifications and business
requirements.
Eye for details and rigorous in his/her execution of the test
scenario and defect reporting.
Good relational skills (will have to report defects and work in
good cooperation with their authors).
High workload during execution and reporting

36
Summary: The building blocks
A test plan
The project test objectives, the strategy to meet
them and the means required by this strategy
Test scenarios
A detailed description of the steps to execute
and pre-defined expected behavior for each of
them
A test model
The link between the features and the test
scenarios
Frequent reporting
A clear description of what does and does not
work in the system
Role definition
Who does what and when

Test strategy and test
organization
Detailed test execution
procedures and
corresponding expectations
Tracability and input
to reporting

Visibility
Ownership
37
Some Testing realities !!
38
1. Quality versus deadlines
what if deadlines were expressed in the same way
quality often is?
Yes, were really committed to deadlines. We need to finish
early - we mustnt be late. Exactly when? Well, Ill know it when
the date arrives, but you cant tie it down like that. Oh yes, we
have a plan, but we dont actually put dates on it - but its very
important to achieve timely delivery.
Deception: I cant see it, so it isnt important
Need to make quality visible (quantified attributes)
Need to evaluate projects (and project managers) on
quality achieved, not just on deadlines met
39
Symptoms: Test Planning
2 weeks?
Content of test plans
6 weeks
Shorter one looks exactly the same inside
therefore it must not be any different in nature
2 weeks is better for my schedule
Cant see any difference, therefore there is none
Solution: make detailed test activities visible
Not shorten a piece of string, cut off arm or leg!
testing
testing
40
2. What does testing produce?
Development versus testing
can tell when development is finished
it produces something concrete and demonstrable
positive effect
testing doesnt change the product at all
neutral effect: still the same product
negative effects
adds extra work when were pushed
didnt want to know about bugs anyway
Why testing is a hard sell
Illusion - testing
adds confidence
- or opportunity to
improve quality
41
Sample: Y2K a waste of time
Nothing happened (much)
Therefore, there was never a real serious problem
The doom merchants were wrong
All that effort was a waste of money
just like immunisation programmes
house/car/health insurance
national defense
Does this remind you of anything?
Software testing in general?
42
3. Relationship problem
What causes stress?
Developers
Have I done a good job? (want to show off)
Will it work (properly)/(forefront of technology)
Project managers
Will we meet our deadlines? (promised resources)
How do good testers affect these concerns?
Find lots of bugs, makes everything worse!
worse for developers: find bugs, have to fix not move on
worse for manager: delay project, need more resources
43
Symptoms: developers
What do you mean, it doesnt work?
It works on my machine
It worked yesterday
I see no bugs
Defensiveness: thats not a bug, its a feature
People who find bugs are not nice
Dont find any [more], it will slow us down
Why dont you find something important instead?
You mis-used the system - a real user would never do
that
44
What causes stress for testers?
Will we find all the bugs?
How much testing is enough?
Have we tested everything?
Not enough time to do a proper job
Why should I have to fight the developers
I just report their own bugs, why should they hate me?
Will I get any recognition
How can anyone tell Ive done a good job?
Guilt about bugs not found
What if bugs I found werent fixed - is it my fault?
Found it
Found it
Found it
Im sure
this is the
last one!
Oops!
Oh dear!
#@&*%!
The last bug
Oh No!
46
Symptoms: testers
I want to be your friend
feel guilty about finding bugs in my friends work
Its never enough
want to keep testing forever / Im not done yet
Its all my fault
feel guilty about any bugs missed in testing
we should continually improve, but cant get them all
remember, you didnt put them in!
47
What causes stress for test
managers?
Test manager
Will we finish testing on time?
Will we find the worst of the bugs?
Will it be noticed if I do a good job?
How to control the uncontrollable
No credit where credit is due
if system is good, developers are praised
if system is bad, testers are blamed

48
Symptoms: test managers
Conflict of interest
want to encourage my team to find faults
do a good job if we find lots
my manager doesnt want us to delay the project
do a good job if we dont find many
paradox
if you do your job properly, youre not a team player
if youre not a team player, youre not doing your job
properly
dont want to challenge developers (too much)
49
And finally ...
Some recent research into brain function

testers brain

developers brain

managers brain
The Managers Brain
Deadline
awareness
hemisphere
Golf
centre
Listening skills
Quality awareness cell
Testing awareness micron
Unknown
AMB
Concern for staff
Belief in
teams ability
Belief in own ability
Logical deduction particle
Sales hype resistance
Meet-
ings
Neurological Research
Grove Consultants,
2000
www.grove.co.uk
Molehill to
mountain
converter
Stupid
question
generator
Alarmist area
Unlikely input
generator
Nit-
picking
zone
Diplomacy neurone
Positive
attitude
particle
New feature enthusiasm zone
Appreciation of
developer genius
Accepting
criticism
cell
Detailed technical
understanding
of code
Deadline
awarenes
micron

The Testers Brain
Enjoying
bad news
Bug
sniffing
segment
Boredom
nucleus
Neurological Research
Grove Consultants,
2000
www.grove.co.uk
The Developers Brain
New
feature
area
Over-complication
centre
Bug to feature
conversion
hemisphere
Bug
insertion
lobe
Magnetic
attraction to
newest coolest
development
tools
Optimism gland
Documentation
particle
Testing skill nucleus
Users wouldnt
do that nerve centre
Bug removal
neurone
Belief in own fallibility
Accurate
estimation
nerve
Desire to learn from
previous project
Interpersonal skills
synapse
Neurological Research
Grove Consultants,
2000
www.grove.co.uk
53
Questions & Answers
Thanks You