Testing Foundation Level Training

Program
Prepared By
Radhesyam Yarramsetty
Testing Development Practices

7/11/2014 1
Its Your Course
Please join in
Asking questions
Making appropriate comments
Share Experiences
Second opinions and disagreements are
welcomed.








7/11/2014 2
What is Testing?
Software testing is the process used to help identify
the correctness, completeness, security, and quality
of developed computer software.

Software testing is a process of technical
investigation, performed on behalf of stakeholders,
that is intended to reveal quality-related information
about the product with respect to the context in which
it is intended to operate.

Software Testing furnishes a criticism or comparison
that compares the state and behavior of the product
against a specification.

7/11/2014 3
Continues.
Software testing proves presence of error. But
never its absence.
Software testing is a constructive destruction
process.
Software testing involves operation of a
system or application under controlled
conditions and evaluating the results.
7/11/2014 4
Software Systems Context
Software Systems are an increasing part of life, from
business applications (e.g. banking) to consumer
products ( e.g. cars).
Most people have had an experience with software
that did not work as expected.
Software that does not work correctly can lead many
problems, including loss of money, time or business
reputation, and could even cause injury or death.

7/11/2014 5
Percentage of Defect Found in
different stages

Others
10%
Coding
7%
Design
27%
Requirem
ents
56%
7/11/2014 6
Causes of software defects
Error: a human action that produces an
incorrect result
Fault: a manifestation of an error in software
- also known as a defect or bug or fault
- if executed, a fault may cause a failure
Failure: deviation of the software from its
expected delivery or service
- (found defect)

Failure is an event; fault is a state of
the software, caused by an error
7/11/2014 7
Error - Fault - Failure
A person makes
an error ...
that creates a
fault in the
software ...
that can cause
a failure
in operation
7/11/2014 8
Reliability versus faults
Reliability: the probability that software will
not cause the failure of the system for a
specified time under specified conditions
- Can a system be fault-free? (zero faults, right first
time)
- Can a software system be reliable but still have
faults?
- Is a fault-free software application always
reliable?
7/11/2014 9
Why do faults occur in software?
software is written by human beings
- who know something, but not everything
- who have skills, but arent perfect
- who do make mistakes (errors)
under increasing pressure to deliver to strict
deadlines
- no time to check but assumptions may be wrong
- systems may be incomplete


7/11/2014 10
Why do faults occur in software?
Complexity of the code, infrastructure
Changing and meshing technologies.
Many system interactions
Environmental conditions
Misuse ( deliberate and accidental)
7/11/2014 11
What do software faults cost?
huge sums
- Ariane 5 ($7billion)
- Mariner space probe to Venus ($250m)
- American Airlines ($50m)
very little or nothing at all
- minor inconvenience
- no visible or physical detrimental impact
software is not linear:
- small input may have very large effect
7/11/2014 12
Safety-critical systems
software faults can cause death or injury
- radiation treatment kills patients (Therac-25)
- train driver killed
- aircraft crashes (Airbus & Korean Airlines)
- bank system overdraft letters cause suicide
7/11/2014 13
So why is testing necessary?
because software is likely to have faults
to learn about the reliability of the software
to fill the time between delivery of the software and
the release date
to prove that the software has no faults
because testing is included in the project plan
because failures can be very expensive
to avoid being sued by customers
to stay in business
7/11/2014 14
General Testing Principles
Exhaustive testing is impossible

Testing everything( all combinations of inputs
and preconditions) is not possible.
Instead of exhaustive testing, we can use
risks and priorities to focus testing efforts.
7/11/2014 15
Why not just "test everything"?
system has
20 screens
Average: 10 loop / option
2 types input / field
(date of birth as Jan 3 or 3/1)
(number as integer or decimal)
Around 10^2 possible values
Total for 'exhaustive' testing:
20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
If 1 second per test, 8000 mins, 133 hrs, 17.7 days
(not counting finger trouble, faults or retest)
Avr. 4 menus
3 options / menu
10 secs = 34 wks, 1 min = 4 yrs, 10 min = 40 yrs
7/11/2014 16
Exhaustive testing?
What is exhaustive testing?
- when all the testers are exhausted
- when all the planned tests have been executed
- exercising all combinations of inputs and
preconditions
How much time will exhaustive testing take?
- infinite time
- not much time
- impractical amount of time
7/11/2014 17
The testing paradox
Purpose of testing: to find faults
The best way to build confidence
is to try to destroy it
Purpose of testing: build confidence

Finding faults destroys confidence



Purpose of testing: destroy confidence
7/11/2014 18
Pesticide paradox
If the same tests are repeated over and again,
eventually the same set of test cases will no
longer find any new bugs.
To overcome this Pesticide Paradox, the test
cases need to be regularly reviewed and
revised, and new and different test cases need
to be written.

7/11/2014 19
Testing shows presence of defects
Testing can show that defects are present, but
cannot prove that there are no defects.
It reduces the probability of undiscovered
defects remaining in the system.
If no defects are found, it is not a proof of
correctness.
7/11/2014 20
Early testing
Testing activities should start as early as
possible in the software or system
development life cycle and should be focused
on defined objectives.



7/11/2014 21
Defect clustering
A small number of modules contain most of
the defects discovered during pre-release
testing or show the most operational failures.
7/11/2014 22
Testing is the context dependent
Testing is done differently in different
Contexts. For example, safety-critical software
is tested differently from an eCommerce site.
7/11/2014 23
Absence of Error Fallacy
Finding and fixing defects does not help if the
systems built is unusable and does not fulfill
the users needs and expectations.
7/11/2014 24
Exercise:
Which general testing principles are characterized by the descriptions below?
W. Early Testing
X. Defect Clustering
Y. Pesticide Paradox
Z. Absence-of-error fallacy

1. Testing should start at the beginning of the project
2. Conformance to requirements and fitness for use.
3. Small number of modules contain the most defects.
4. Test Cases must be regularly reviewed and revised.

A. W1, X2, Y3 and Z4
B. W1, X3, Y4 and Z2
C. W2, X4, Y2, and Z3
D. W1, X4,Y2 and Z3

7/11/2014 25
How much testing is enough?
- its never enough
- when you have done what you planned
- when your customer/user is happy
- when you have proved that the system works
correctly
- when you are confident that the system works
correctly
- it depends on the risks for your system
7/11/2014 26
How much testing?
It depends on RISK
- risk of missing important faults
- risk of incurring failure costs
- risk of releasing untested or under-tested software
- risk of losing credibility and market share
- risk of missing a market window
- risk of over-testing, ineffective testing
7/11/2014 27
So little time, so much to test ..
test time will always be limited
use RISK to determine:
- what to test first
- what to test most
- how thoroughly to test each item
}
i.e. where to
place emphasis
7/11/2014 28
Terms to Remember
Bugs,defect, or fault
Error or Mistake( I EEE 610):
Failure
Quality
Risk
Test/ Test Case
Code
Debugging
Software Development
Review
Requirements
Test Objective



7/11/2014 29
Terms to Remember continues.
Test basis
Debugging
Software Development
Review
Requirements
Testware
Test plan
Test Strategy
Test log
Exit Criteria
Test Summary Report
Test Coverage
Test Condition

7/11/2014 30
Alpha and beta testing
Alpha Testing: Services tested by potential user/customers or an
independent test team at developers site, but outside the
development organization. Alpha testing is often employed for
off-the-self software as a form of internal acceptance testing.
Beta Testing: Operations tested by potential and/or existing
users/customers at an external site not otherwise involved with
the developers, to determine whether or not a component or
system satisfies the user/customer needs and fits within the
business processes. Beta testing is often employed as a form of
external acceptance testing for off-the-self software in order to
acquire feedback from the market.
7/11/2014 31
Testing and quality(K2)
testing measures software quality
testing can find faults; when they are
removed, software quality (and possibly
reliability) is improved
what does testing test?
- system function, correctness of operation
- non-functional qualities: reliability, usability,
maintainability, reusability, testability, etc.
- Software Product Quality (ISO 9126)
7/11/2014 32
What is Quality ?
Meeting the customer requirements first time
and every time.
Quality is much more higher than absence of
defects.
Quality can only seen through the eyes of the
customers.
In simple words exceeding customer
expectations assures meeting all the
definitions of quality.


7/11/2014 33
What are software quality factors?
Correctness: Extent to which a program
satisfies and fulfills the users' mission
objectives.
Reliability: Extent to which a program can be
expected to perform its intended function.
Efficiency: Amount of computing resources
and code required by a program to perform a
function.
Integrity: Extent to which access to software or code
can be controlled.

7/11/2014 34
Quality factors continue
Usability: Effort required learning, operating,
preparing input, and interpreting output of the
program.
Maintainability: Effort required locating and fixing an
error in operational program.
Testability: Effort required testing a program to
ensure that it performs its intended function.
Flexibility: Effort required modifying an operational
program.


7/11/2014 35
Quality factors continue.
Portability: Effort required to transfer from
one configuration to another.
Reusability: Extent to which a program can
be used in other applications related to the
packaging and scope of the functions that
programs.
Interoperability: Effort required to couple one
system with another.

7/11/2014 36
Other factors that influence testing
contractual requirements
legal requirements
industry-specific requirements
- e.g. safety-critical or safety-related such as railroad
switching, air traffic control
7/11/2014 37
Test Planning and control (K1)
Test Planning is an activity of verifying the missing of
testing, defining the objectives of testing and the
specification of the test activities in order to meet the
objective and mission.
Test Control is ongoing activity of comparing actual
against the plan and reporting the status, including
deviations from the plan. It should be monitored
through the project.
7/11/2014 38
Test Planning - different levels
Test
Policy
Test
Strategy
Company level

High Level
Test Plan
High Level
Test Plan
Project level (IEEE 829)
(one for each project)
Detailed
Test Plan
Detailed
Test Plan
Detailed
Test Plan
Detailed
Test Plan
Test stage level (IEEE 829)
(one for each stage within a project,
e.g. Component, System, etc.)
7/11/2014 39
The test process
specification execution recording
check
completion
Planning (detailed level)
7/11/2014 40
Test specification
specification execution recording
check
completion
Identify conditions
Design test cases
Build tests
Planning (detailed level)
7/11/2014 41
A good test case
effective

exemplary

evolvable

economic
Finds faults
Represents others
Easy to maintain
Cheap to use
7/11/2014 42
Test specification
test specification can be broken down into three
distinct tasks:
1. identify: determine what is to be tested (identify
test conditions) and prioritise
2. design: determine how the what is to be tested
(i.e. design test cases)
3. build: implement the tests (data, scripts, etc.)
7/11/2014 43
Task 1: identify conditions
list the conditions that we would like to test:
- use the test design techniques specified in the test plan
- there may be many conditions for each system function
or attribute.
prioritise the test conditions
- must ensure most important conditions are covered
(determine what is to be tested and prioritise)
7/11/2014 44
Task 2: design test cases
design test input and test data
- each test exercises one or more test conditions
determine expected results
- predict the outcome of each test case, what is
output, what is changed and what is not changed
design sets of tests
- different test sets for different objectives such as
regression, building confidence, and finding faults
(determine how the what is to be tested)
7/11/2014 45
Designing test cases
Importance
Time
Most important
test conditions
Least important
test conditions
Test cases
7/11/2014 46
Task 3: build test cases
prepare test scripts
- less system knowledge tester has the more detailed
the scripts will have to be
- scripts for tools have to specify every detail
prepare test data
- data that must exist in files and databases at the start
of the tests
prepare expected results
- should be defined before the test is executed
(implement the test cases)
7/11/2014 47
Test execution
specification execution recording
check
completion
Planning (detailed level)
7/11/2014 48
Execution
Execute prescribed test cases
- Execute the test suites and individual test cases

- would not execute all test cases if
testing only fault fixes
too many faults found by early test cases
time pressure
- can be performed manually or automated
7/11/2014 49
Test recording
specification execution recording
check
completion
Planning (detailed level)
7/11/2014 50
Test recording 1
The test record contains:
- identities and versions of
software under test
test specifications
Follow the plan
- mark off progress on test script
- document actual outcomes from the test
- capture any other ideas you have for new test cases
- note that these records are used to establish that all
test activities have been carried out as specified
7/11/2014 51
Test recording 2
Compare actual outcome with expected
outcome. Log discrepancies accordingly:
- software fault
- test fault (e.g. expected results wrong)
- environment or version fault
- test run incorrectly
Log coverage levels achieved (for measures
specified as test completion criteria)
After the fault has been fixed, repeat the
required test activities (execute, design, plan)
7/11/2014 52
Check test completion
specification execution recording
check
completion
Planning (detailed level)
7/11/2014 53
Check test completion
Test completion criteria are specified in the
test plan
If not met, need to repeat test activities, e.g.
test specification to design more tests
specification execution recording
check
completion
Coverage too low
Coverage
OK
7/11/2014 54
Test completion criteria
Completion or exit criteria apply to all levels
of testing - to determine when to stop
- coverage, using a measurement technique, e.g.
branch coverage for unit testing
user requirements
most frequently used transactions
- faults found (e.g. versus expected)
- cost or time

7/11/2014 55
Why test?
build confidence
prove that the software is correct
demonstrate conformance to requirements
find faults
reduce costs
show system meets user needs
assess the software quality
7/11/2014 56
Fault found Faults found
Confidence
Time
Confidence
No faults found = confidence?
7/11/2014 57
A traditional testing approach
Show that the system:
- does what it should
- doesn't do what it shouldn't
Fastest achievement: easy test cases
Goal: show working
Success: system works
Result: faults left in
7/11/2014 58
A better testing approach
Show that the system:
- does what it shouldn't
- doesn't do what it should
Fastest achievement: difficult test cases
Goal: find faults
Success: system fails
Result: fewer faults left in
7/11/2014 59
Who wants to be a tester?
A destructive process
Bring bad news (your baby is ugly)
Under worst time pressure (at the end)
Need to take a different view, a different
mindset (What if it isnt?, What could go
wrong?)
How should fault information be
communicated (to authors and managers?)
7/11/2014 60
Testers have the right to:
accurate information about progress and changes
insight from developers about areas of the software
delivered code tested to an agreed standard
be regarded as a professional (no abuse!)
find faults!
challenge specifications and test plans
have reported faults taken seriously (unreproducible)
make predictions about future fault levels
improve your own testing process
7/11/2014 61
Testers have responsibility to:
follow the test plans, scripts etc. as documented
report faults objectively and factually (no abuse!)
check tests are correct before reporting s/w faults
remember it is the software, not the programmer,
that you are testing
assess risk objectively
prioritise what you report
communicate the truth
7/11/2014 62
Levels of independence
None: tests designed by the person who
wrote the software
Tests designed by a different person
Tests designed by someone from a different
department or team (e.g. test team)
Tests designed by someone from a different
organisation (e.g. agency)
Tests generated by a tool (low quality tests?)
7/11/2014 63
Re-testing after faults are fixed
Run a test, it fails, fault reported
New version of software with fault fixed
Re-run the same test (i.e. re-test)
- must be exactly repeatable
- same environment, versions (except for the software
which has been intentionally changed!)
- same inputs and preconditions
If test now passes, fault has been fixed
correctly - or has it?
7/11/2014 64
Re-testing (re-running failed tests)
x
x
x
x
New faults introduced by the first
fault fix not found during re-testing
Re-test to check
Fault now fixed

7/11/2014 65
Regression test
to look for any unexpected side-effects
x
x
x
x

Cant guarantee
to find them all
7/11/2014 66
Regression testing 1
misnomer: "anti-regression" or "progression"
standard set of tests - regression test pack
at any level (unit, integration, system,
acceptance)
well worth automating
a developing asset but needs to be
maintained
7/11/2014 67
Regression testing 2
Regression tests are performed
after software changes, including faults fixed
when the environment changes, even if application
functionality stays the same
for emergency fixes (possibly a subset)
Regression test suites
evolve over time
are run often
may become rather large
7/11/2014 68
Regression testing 3
Maintenance of the regression test pack
- eliminate repetitive tests (tests which test the same
test condition)
- combine test cases (e.g. if they are always run
together)
- select a different subset of the full regression suite
to run each time a regression test is needed
- eliminate tests which have not found a fault for a
long time (e.g. old fault fix tests)
7/11/2014 69
Regression testing and automation
Test execution tools (e.g. capture replay) are
regression testing tools - they re-execute
tests which have already been executed
Once automated, regression tests can be run
as often as desired (e.g. every night)
Automating tests is not trivial (generally takes
2 to 10 times longer to automate a test than to
run it manually
Dont automate everything - plan what to
automate first, only automate if worthwhile
7/11/2014 70
Expected results
Should be predicted in advance as part of the
test design process
- Oracle Assumption assumes that correct outcome
can be predicted.
Why not just look at what the software does
and assess it at the time?
- subconscious desire for the test to pass - less work
to do, no incident report to write up
- it looks plausible, so it must be OK - less rigorous
than calculating in advance and comparing
7/11/2014 71
Prioritising tests
We cant test everything
There is never enough time to do all the
testing you would like
So what testing should you do?
7/11/2014 72
Most important principle
Prioritise tests
so that,
whenever you stop testing,
you have done the best testing
in the time available.
7/11/2014 73
How to prioritise?
Possible ranking criteria (all risk based)
test where a failure would be most severe
test where failures would be most visible
test where failures are most likely
ask the customer to prioritise the requirements
what is most critical to the customers business
areas changed most often
areas with most problems in the past
most complex areas, or technically critical
7/11/2014 74
Evaluating exit criteria and reporting
Checking test logs against the exit criteria
specified in test planning.
Assessing if more tests are needed or if the
exit criteria specified should be changed.
Writing a test summary report for
stakeholders.
7/11/2014 75
Test Closure activities
Test closure activities collect data from complete test activities
to consolidate experience, i.e when a software system is
released, a test project is completed.
Test closure activities include the following major tasks;
1) Checking which planned deliverables have been delivered,
the closure of incident report or raising of change records for
any that remain open, and the documentation of the acceptance
of the system.
2) Finalizing and archiving test ware, the test environment and
the test infrastructure for later reuses.
3) Handover of testware to the maintenance organization.
4) Analyzing lessons learned for future releases and projects,
and the improvement of the test maturity.

7/11/2014 76