ISTQB Certified Tester

ISTQB Certified Tester Foundation Level
Basics for Software Testing
Basics of Software Testing

ISTQB Certified Tester, Foundation Level
2. Introduction
Introduction
In this chapter we will introduce

the concept of certified tester Training (modules, curriculum, examination, and certification)
Institutions and international connections Examination/ certification bodies, Training and accreditation providers, ISTQB and national boards. The materials for this course
Introduction Certified Tester
Background
Software become ubiquitous Problems in software boil down to quality (or the lack of it) Testing is the way to prove it works as it ahould Almost annything can be tested Products Prototypes documentation
Training need
Testers need to know
The test object Their trade: process, techniques, tools
Managers need to know that testers:

Are good at what they do Know what to do
Certification provides reasonable level of confidence
Expected abilities
Analyze information and evaluate business risks to design and plan inspections and tests Know the limitation:
Framework, technology, process, people, risks, etc. to define Test targets, techniques, tasks, and schedules within them.
Manage the testing process:

Execute tests, file and track faults, measure results, and report
Expected abilities
Plan, conduct, and follow on reviews and inspecions
Of documents and code
Identify and use appropiate tools Select test cases depending on

Probability of finding faults, associated risks, and requied coverage.
Introduction Training
Curriculum
Training is a way to achieve adequate skills and knowledge Two levels
Foundation (this course) Advanced (three courses Test manager, Functional Tester and Technical Tester)
Standardized training program for education of testers International recognition
Introduction Training
Syllabus, Examination and Certification

Main chapters in Syllabus prepared by ISTQB
Fundamentals of testing Testing throughout the software lifecycle Static techniques Test design techniques Test management Toll support for testing
Courses delivered by accredited training providers Examination delivered by certification body iSQl, GASQ
Introduction Organizations
ISTQB
International Software Testing Qualification Board www.istqb.org An international organization of software testers Mission = organize and support the ISTQB Certified Tester ualification scheme for testers.
Provides the core syllabi and sets guidelines for accreditation and examination.
ISTQB (continued)
Members are national and supra national qualification testing boards
ISTQB
Work Groups Syllabi Exam Questions
German Testing Board
American Testing Board
Testing Board
Bulgaria participates in the SEETB South East European Testing Board
GASQ
Global Association for Software Quality www.gasq.org Provides services related to software quality from personnel certification to international standardization.
Delivers exams Accredits trainers and courses
Mission = adance the field by coordinating efforts aiming at new, higher industry standards
Other
ISEB = Information Systems Examinations Board
Of the British Computer Society or BCS www.bcs.org/BCS/Products/Qualifications/ISEB/
iSQI = International Software Quality Institute

The German training and certification body www.isqi.org
IEEE = Institute of Electrical and Electronics Engineers

An American organization with 40% international members. www.ieee.org
For this course

The Handouts
Course Schedule Slides Student Textbook Evaluation Sheet
Additional
Glossary of terms used in Software Testing download v.1.2 from: www.istqb.orgdownload.php British Testing Standard BS7925-2 download the draft from: http://www.testingstandards.co.uk/

3. Fundamentals of testing
3.1 Why is testing necessary?
Why is testing necessary
In this session we will

Understand what testing is and why it is necessary Look at the cause and cost of errors Talk about quality measurements Mention stndards and models for improuvement
Why is testing becoming more important?

The Y2K problem EMU (Economic and Monetary Union)
e Commerce
Increased user base Increased complexity Speed to Market
How Error Occurs

No one is perfect! We all make mistakes or omissions
The more pressure we are under the more likely we are to make mistakes
In IT development we have time and budgetary deadlines to meet
Poor Training
How Errors Occur

Poor Communication Requirements not clearly defined Requirements change & are not properly documented Data specifications not complete
ASSUMPTIONS!
The Cost Of Failures

A single failure may incur little cost or millions Report layouts may be wrong little true cost
Or a significant failure may cost millions

Ariane V, Venus Probe, Mars Explorer and Polar Lander

In extreme cases a software or systems failure may cost LIVES
Usually safety critical systems are tested exhaustively

Aeroplane, railway, nuclear power systems, etc
Unfortunately there are exceptions to this

London Ambulance Service
Example for huge errors
A small town in Illinois received an unusually large monthly electric bill of $7 million in March of 1999. This was about 700 times larger than its normal bill. It turned out to be due to faults in new software that had been purchased by the local power company to deal with Y2K software issues.
Software faults caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered. Software faults in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983, according to news reports in early 1999. the software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so. Disaster was averted when a Soviet commander, based on what he said was a funny feeling in my gut, decided the apparent missile attack was a false alarm. The filtering software code was rewritten.

The cost of failures increases proportionately (tenfold) with the passing of each succesive stage in the system development process before they are detected
To correct a problem at requirements stage may cost $1

To correct the problem post-implementation may cost $000s
The model of cost escalation
The cost of correcting a defect during progressively later phases in the SDLC rises alarmingly (tenfold for each phase).
Cost
UR
FS
TS Code Unit Phase of testing
System
UAT
UR = User Reuquirements FS = Functional Specification TS = Technical Specification UAT = User Acceptance Testing
Why test?
Identifies faults
Reduces live defect Improves the quality of the users application Increases reliability To help ensure live failures dont impact costs & profitability To help ensure requirements are satisfied To ensure legal reqiurements are met To help maintain the organizations reputation
Provides a measure of quality
How much test is enough?

How we know when to stop? If we stop testing too soon we risk errors in the live system If we carry on testing we can delay the product launch
This may lose revenue And damage the corporate image
Exhaustive Testing
Find all faults by testing everything?
To test everything is rarely possible
The time required makes it impractical

The resource commitment required makes it impractical
If we cant test everything, what can we do?

Managing and reducing Risk Carry out a Risk Analysis of the application Prioritise tests to focus on the main areas if risks Apportion time relative to the degree of risk involved Understand the risk to the business of the software not functioning correctly
Risk Reduction Profile: The risk can be defined as a combination of two factors, the likelihood of a problem to occur and the impact that this problem will inflict The decision as to which areas to test is normally based upon the risk involved. The higher the risk to the business the greater priority the testing of a given area will be.
Risk
Business Functional and Non Functional Atributes
Time
Should you stop testing when the product goes live?

Continuing testing beyond the implementation date should be considered
Better that you find the errors than the users
Testing & Quality

The purpose of testing is to find defects. The benefits:
Reduce the number of defects in the released product Improve the quality of the software Increase reliability fewer faults means more reliable Provide a measure of quality of the software
To demonstrate the level of quality for the SUT, testing must be carried out throughout the SDLC. Quality starts at the beginning of the project,
Quality Measurements Quality can be measured by testing for

Correctness Reliability Usability Maintainability Reusability Testability Legal requirements
Contractual requirements Regulatory requirements
ISO 9000
Defines the overall quality management system with document control, training, best practices
ISO 9000 compliance is not a guarantee of quality, it merely provides for reproducibility
ISO 9126
Defines the quality characteristics of software at the lifecycle stages. Four parts
Quality Model-criteria for good software: functionality, reliability, efficiency, maintainability and portability/applicability External Use external metrics (the behavior of software) Internal Use internal metrics (the software itself) Quality in Use metrics the effects of using the software in context
Process Improvement Models

ISO/IEC 15504 Software Process Improvement and Capability dEtermination (SPICE) Software Engineering Institute (SEI) Capability Maturity Model (CMM) SEI Capability Maturity Model Integration (CMMI) Illinois Institute of Technology (IIT) Testing Maturity Model (TMM) IQUIP Informatica B.V. Test Process Improvement (TPI)
Process Improvements Models (continued)

Improvement models have two objectives in common:
Assessment of an existing process Improvement of an existing process
Some models SPICE assess the Capabilities of a process, and identify areas for improvement Others CMM assess the Maturity of a process, give it a Level, and define what to achieve to progress to a next level
Summary
The purpose of testing is to find faults
Faults can be fixed, making better software Better software is more reliable, less prone to failures
Establish the relationship between the software and its specification through testing Testing enables us to measure the quality of the software
Thus understand & manage the risk to the business
Several models for process measurement and improvement

3.2. What is testing?
What is testing

Define some of the words used in software testing
All the terms in this course are derived from the Glossary of terms used in Software Testing version 1.2 (dd. June, 4th 2006) produced by the ISTQB Glossary Working Party
Terms - What is testing
Some Definitions of Testing

the process of executing a program in order to certify its Quality the process of executing a program with the intent of finding errors the process of exercising software to detect errors & to verify that it satisfies specified functional & non-functional requirements
The ISTQB Definition

The process consisting of
All life cycle activities, both static and dynamic, Concerned with planning, preparation and evaluation of software products and related work products
To determine that they satisfy specified requirements, To demonstrate that they are fit for purpose and To detect defects.
ERROR
A human action that produces an incorrect result
DEFECT = FAULT = BUG = PROBLEM

A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
Also called: error condition
FAILURE
Actual deviation of the component or system from its expected delivery, service or result. Also called external fault
ANOMALY
Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, etc. or from someones perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation.
DEFICIENCY
An absence of some necessary quality or element or the lack of fulfillment of a justified expectation. Deficiencies include missing data, incomplete data, or incomplete reports *Note: The term is not the ISTQB glossary
DEFECT MASKING An occurrence in which one defect prevents the detection of another. Also called fault masking
TEST
A set of one more test cases
TEST CASE
A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement.
QUALITY
the totally of the characteristics of an entity that bear in its ability to satisfy stated or implied needs According to DIN-ISO 9126 software quality includes: reliability , usability, efficiency, maintainability and portability/aplicability.
RELIABILITY The ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations.
USABILITY The capability of the software to be understood, learned, used and attractive to the user when used under specified conditions.
EFFICIENCY The capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions.
MAINTAINABILITY The ease with which a software product can be modified to correct defects, modified to met new requirements, modified to make future maintenance easier, or adapted to a changed environment.
PORTABILITY The ease with which the software product can be transferred from one hardware or software environment to another.
Does testing improve quality?

Testing does not build Quality into the software Testing is a means of determining the Quality of the Software under Test
QUALITY ASSURANCE All those planned actions used to fulfill the requirements for quality QUALITY CONTROL Overseeing the software development process to ensure that QA procedures and standards are being followed

3.3. General testing principles
General testing principles

Look at the general testing principles Look at the level of testing maturity
General testing principles
A number of testing principles have been suggested over the past 40 years and offer general guidelines common for all testing Principle 1 Testing shows presence of defects Principle 2 Exhaustive testing is impossible
Principle 3 Early testing

Principle 4 Defect clustering Principle 5 Pesticide paradox Principle 6 Testing is context dependant Principle 7 Absence-of-errors fallacy
Testing Maturity
A different view
Dr. Boris Beizer describes five levels of testing maturity:
Level 0 - There is no difference between testing and debugging. Other than in support of debugging, testing has no purpose.
Level 1 - The purpose of testing is to show that software works.

Level 2 - The purpose of testing is to show that software doesnt work. Level 3 - The purpose of testing is not to prove anything, but to reduce the perceived risk of not working to an acceptable value. Level 4 - Testing is not an act. It is a mental discipline that results in lowrisk software without much testing effort.

3.4. The Fundamental Test Process
The Fundamental Test Process

Look at the steps involved in The Fundamental Test Process Look in detail at each step
What is the objective of a test?

A successful test is one that does detect a fault
TEST MANAGEMENT
PLAN SPECIFY EXECUTE RECORD CHECK COMPLETE
TEST ASSET MANAGEMENT TEST ENVIROMENT MANAGEMENT
There are five defined steps in the Test Process

Test planning and control Test analysis and design Test implementation and execution Evaluating exit criteria and reporting Test closure activities
The Fundamental Test Process Test planning and control

The Test Plan describes how the Test Strategy is implemented
A project plan for testing

Defines what is to be tested, how it is to be tested, what is needed for testing etc. Defined the different types of testing functional, nonfunctional, used automation tools, etc.

The most critical stage of the process Effort spent now will be rewarded later The foundation on which testing is built

The major tasks concerning test control are:
Measuring and analyzing results; Monitoring and documenting progress, test coverage and exit criteria; Assign extra resources; Re-allocate resources; Adjust the test schedule; Arrange for extra test environments; Refine the completion criteria;
The Fundamental Test Process Test analysis and design

Three step progress
Preparation & analysis Building test cases Defining expected results

Test preparation
Analyze the Application Identify test conditions Identify test cases Document thoroughly Cross-referencing

Build Test Cases Test cases comprise
Standing data Transaction data Actions Expected results

Identify Expected Results The outcome of each action The state of the application during & after The state of the data during & after

Cross-Reference & Classify test cases Enables maintainability of Test Assets Allows testing to be performed in a focussed manner directed at specific areas
The Fundamental Test Process Test implementation and execution

Test execution checklist
Test execution schedule/log Identify which tests are to be run Test environment primed & ready Resources ready, willing % able Back-up & recovery procedures in place Batch runs planned & scheduled
Then we are ready to run the tests
The Fundamental Test Process Execution

If planning and preparation is sufficiently detailed this is the easy part of testing
The test is run to verify the application under test

The test itself either passes or fails
Recording
The test log should record Software and test versions Specifications used as test base Testing timings Test results Actual results Expected results Defect details (if applicable)
Test Script <Name> Seq No. Detailed Instruction Test Criteria Expected Results Pass/Fail
Tested by: Date:
The Fundamental Test Process Evaluation exit criteria & reporting

Have we fulfilled the Test Exit Criteria? Used to determine when to stop this phase of testing
Key functionality tested Test coverage Budget used? Defect detection rate Performance satisfacatory
Test closure activities

When testing is over the testing project can be closed. Documents should be updated and put under version control. The extract way depends on the specifics of your organization
Summary
There are five steps in The Fundamental Test Process
Test planning and control Test analysis and design Test implementation and execution Evaluating exit criteria and reporting Test closure activities
The Fundamental Test Process Expected Outcome and Test Oracle
In this subchapter we will

Understand the need to define Expected Outcome Understand where expected results can be found Understand what a Test Oracle is
Expected Results = expected outcomes

Identify required behavior If the expected outcome of a test is not defined the actual output may be misinterpreted
Running a test with only a general concept of the outcome is fatal It is vital the expected results are defined with the tests, before they are used You cannot decide whether a test has passed just because it looks right
Test Oracle
A source to determine expected result In broader terms, a principle or mechanism to recognize a problem Our oracle can be an existing system, a document, a person but not a code itself
Oracle Heuristics or Possible Oracles

History: Present behavior is consistent with past behavior Our Image: behavior is consistent with an image the organization wants to project Comparable Products: behavior is consistent with that of similar functions in comparable products Claims: behaves as claimed or advertised (as seen on TV)
Oracle Heuristics or Possible Oracles (continued)

Specifications or Regulations: behaves as documented User Expectations: behaves the way we think users want The Product (within): comparable to functions patterns within the product
Purpose: behavior is consistent with apparent purpose
Oracles in use
Simplification of Risk
Do not think pass fail Think instead problem no problem
Oracles and Automation

Our ability to automate testing is fundamentally constrained by our ability to create and use oracles
Possible Pitfalls
False alarms Missed bugs
Summary
Without defining expected results do you know if a test has passed or failed?
Expected results must be based on oracles.
The Fundamental Test Process Prioritizing the Tests

Understand why we need to prioritize tests Understand how we decide the priority of individual tests
It is not possible to test everything

Therefore errors will get through to the live system We must do the best testing possible in the time available This means we must prioritize and focus testing on the priorities

Aspects to Consider
Severity Probability Visibility Priority of requirements Frequency of change Vulnerability to error
Technical criticality
Complexity
Customer / User Requirements
Time & Resources
Business Criticality
What elements of the application are essential to the success of the organization?
Customer factors
How visible would a failure be? What does the customer want?
Technical Factors
How complex is it? How likely is an error? How often does this change?
Summary
There will never be enough time to complete all tests Therefore those tests that cover those areas deemed most important (to the business, highest risk etc) must be tested first

3.5. The Psychology of Testing
The Psychology of Testing

Understand what qualities make good testers Look at a testers relationship with developers Look at a testers relationship with management Understand the issues with testing independence
What makes a Tester?

Testing is primarily to find faults
Can be regarded as destructive Development is constructive Testing ask questions Testers need to ask questions
A tester needs many qualities

Intellectual qualities
Can absorb incomplete facts Can work with incomplete facts Can learn quickly on many levels Good verbal communication Good written communication Ability to prioritize Self-organization

Knowledge
How projects work How computer systems and business needs interact IT technology IT commercial aspects Testing techniques Testing best practice To be able to think inside and outside of a system specification

More skills to acquire
Ability to find faults planning, preparation & execution Ability to understand systems Ability to read specifications Ability to extract testable functionality Ability to work efficiently Ability to focus on essentials
Reporting Faults
Faults need to be reported to
Developers to enable them to fix them Management so they can track progress
Communication to both groups is vital
Communication with Developers

A good relationship is vital Developers need to keep testers up to date with changes to the application Testers need to inform developers of defects to allow fixes to be applied
Communication with Management

Managers need progress reports The best way is through metrics
Number of tests planned & prepared Number of tests executed to date Number of defects raised & fixed How long planning, preparation and execution stages take
Testing independence
It is important that testing is separate from development
The developer is likely to confirm adherence not deviation The developer will make assumptions the same when testing as developing
Levels of Independence
Low Developers write their own tests Medium tests are written by another developer High Tests written by an independent body
Tests written by another section Tests written by another organization
Utopia Tests generated automatically
Summary
Testers require a particular set of skills
The desire to break things The desire to explore and experiment Communication Questioning
Testing requires a different mentality to development

destroying things rather than creating them Testing should be separate from development
References:
References
Black, 2001, Kaner, 2002 Beizer, 1990, Black, 2001, Myers, 1979 Beizer, 1990, Hetzel, 1998 Craig, 2002

4. Testing throughout the Software Life Cycle 4.1. Software development models
Software development models

Look at the most commonly accepted models for software development:
The V-Model Iterative Development Model V, V&T (Validation, verification and Testing)
There are many approaches for software development

The most commonly used model is:
The V Model
The main activities in software testing are:

V,V&T (Validation, Verification and Testing)
There are various other models, but they are not part of this course:
Sequential model (Waterfall) Iterative Development Model
Validation
Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled.
Verification
confirmation by examination and through the provision of objective evidence that specified requirements have been fulfilled
Testing
The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.
V-model
The most commonly used model in software development It represents the Software Development Life Cycle Shows the various stages in development and testing Shows the relationships between the various stages
high level
A simple V-model T UAT

design
Requirements
T T
code
System test
Unit test
low level
time
A typical development life-cycle V - Model

Business Requirements User Acceptance testing
Technical Specification Functional Specification Design Specification
Integration testing (large)
System Testting
Integration testing (small) Component Testing
level
Code
time
V- model where do we start testing? T
Requirements
UAT
Design
System Test
Code
level
Unit test
time
V-Model Start Testing Early
R
Reqs
T
UAT
Spec
T
System Test
Code
T
Unit Test
Legend:
level
R Review P Plan T - Test
time
Iterative Development Models

Breaking down the project into a series of small projects Each subproject may follow a mini V-model Related to RAD and agile development Benefits:
Early analysis of risks Revisiting plans Quality increments (when needed) High motivation
A potential pitfall is that the project duration seem to increase. This is more of a psychological problem, though.
Summary
The V- Model is the most common model for software development There are other models used for testing, but these are less widely used
Including V, V & T Iterative Development Model Other models have been created
Basics of Software testing

4.2. Test Levels 4.2.1. Component testing
Test Levels Component Testing

Understand what Component Testing is Look at the Component Testing Process Look at the myriad types of Component Testing
What is component testing?

the testing of individual software components
What is a component?
a minimal software item that can be tested in isolation.
Component testing is also known as:

Unit testing Module testing Program testing
Component testing is the lowest form of testing

At the bottom of the V-model Each component is tested in isolation
Prior to being integrated into the system
Involves testing the code itself

Test the code in the greatest detail Usually done by the components developer
Component Test Process

Component test Planning Component Tests Specification Component Test Execution
BEGIN
Component test planning
Component test specification
Component test execution
Component test recording
Component Test Recording

Component test completion
Checking for Component Tests Completion
END
Testing techniques
Equivalence Partitioning Boundary Value Analysis State transition testing Cause-Effect Graphing Syntax Testing Statement Testing Branch/Decision testing Data Flow Testing Branch Condition Testing Branch Condition Combination Testing Modified Condition Decision Testing LCSAJ Testing Random Testing Other Testing Techniques
What about Playtime ?

Often a short unscripted session with a component can be of benefit
More common at later stages (once formal test techniques have been completed)
Not done too well? Some basics would quickly improve it:
Checklist Standards Procedures Code & version control Sign off (check from independent actor for example, Technical Lead)
Summary
Component testing is intended to test a software component before it is knitted into the system
There are lots of different approaches

ISTQB Certified tester, Foundation Level
4.2. Test Levels 4.2.2. Integration Testing
Test Levels Integration Testing in the Small

Understand what Integration in the Small is Look at how & why we do Integration Testing in the Small
Integration testing
Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems.
Integration testing in the small = component integration testing

testing performed to expose defects in the interfaces and interaction between integrated components
A sample system
databases
External systems
All components of a computer application work on only one thing DATA

Data is passed from one component to another
Data is passed from one system to another Software components will be required to add, amend, & delete information, validate the information and report on the information
Integration testing
Testing should start with the smallest, definable component Building tested components up to the next level of definition Eventually the complete business processes is tested End to End
Testing Components
Each component needs to be tested individually to ensure it processes the data as required
This is Component Testing
Once components have been tested we can link them together to see if they can work together
This is integration Testing in the Small
Linking components together to ensure that they communicate correctly Gradually linking more and more components together Prove that the data is passed between components as required Steadily increase the number of components and create & test sub-systems and then the complete system
Planning testing
We need to determine when and at what levels Integration testing will be performed
Identify the boundaries

A similar process will need to be followed for all elements
Once these elements have been tested individually they are further integrated to support the business process
Stubs and drivers

Used to emulate the missing bits
External systems Sub systems Missing components
May need to be written specifically for the SUT
Approaches to integration testing

Incremental
Top down Bottom-up Functional
Non-incremental
Big bang
Summary
Integration testing is needed to test how components interact with each other and how the system under test works with other systems Integration testing in the small is concerned with the interaction between the various components of a system
Test Levels Integration Testing in the Large

Understand what Integration Testing is Understand what Integration testing in the Large is Look at how & why we test system integration
Integration testing
testing performed to expose defects in the interfaces and in the interactions between integrated components or systems.
Integration testing in the large = system integration testing

Testing he integration of systems and packages; testing interfaces to external organizations (e.g. Electronic Data Interchange, Internet)
Why do we need to do integration testing in the large

Just because the SUT works in the test lab doesnt mean it will work in the real world
All components of a computer system work on only one thing DATA

Data is manipulated within the SUT It may then be passed to another system It may be stored in a database
This is done to satisfy a business process requirement
A typical system
Env
SUT
Env
SUT
Env
SUT
Data Model
Databases
Data
External applications
Integration Testing on a Large Scale

Test the way the SUT interfaces with external systems The SUT and the external systems may run on the same and / or different platforms (Windows, Linux, MacOS) They may be located is separate locations and use communication protocols to pass data back & forth
Additional challenges posed by large scale

Multiple Platforms Communications between platforms Management of the environments Coordination of changes Different technical skills required
Typical approach
Test the system
In a test environment Use stubs & drivers where external systems arent available
Test the system in situ

In a replica of the production / live environment Use stubs & drivers where external systems arent available
Test the system and its interfaces with other systems

In a replica of the production / live environment Access test versions of the external systems
Fundamentally we are still looking at DATA
& how that data is obtained, processed, manipulated, stored and made available or transported for use elsewhere within or outside the organization
Planning
We need to determine when and at what levels Integration testing will be performed
Identify the boundaries

A similar process will need to be followed for all elements
Once these elements have been tested individually they are further integrated to support the business process
Approaches to integration testing

Incremental
Top down Bottom-up Functional
Non-incremental
Big bag
Summary
Integration testing is testing performed to expose faults in the interfaces and in the interaction between integrated components Large-scale integration testing is needed to test how the system under test works with other systems Just because a system works in a test environment doesnt mean it will work in a live environment

4.2. Test Levels 4.2.3. System Testing
Test Levels System Testing
What is System Testing?

The process of testing an integrated system to verify that it meets specified requirements.
Testing of the complete system

May be the last step in integration testing in the small May be the first time that enough of the system has been put together to make a working system
Ideally done by an independent test team Two types functional and non-functional

4.2. Test Levels 4.2.3.1.Functional System Testing
Test Levels Functional System Testing

Understand what functional system testing is Understand the benefits of functional system testing
A Functional Requirement is
A Requirement that specifies a function that system component must perform
Functional System testing is geared to checking the function of the system against specification May be requirements based or business process based
Testing Based on Requirements

Requirements specification used to derive test cases System is tested to ensure the requirements are met
Testing carried out against Business processes

Based on expected use of the system Builds use cases test cases that reflect actual or expected use of the system
Security Testing
testing to determine the security of the software product
How easy is it for an unauthorized user to access the system? How easy is it for an unauthorized person to access the data?
Summary
Functional system testing allows us to test the system against the specifications, user requirements and business processes Two approaches:
From the functional requirements and From a business process

4.2. Test Levels 4.2.3.2. Non-Functional System Testing
Test Levels Non- Functional System Testing

Understand what Non-Functional System testing is Understand the need for Non-Functional System Testing Look at the various types of Non-Functional System Testing
Non-Functional System Testing is defined as:

testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability. Users, generally, focus on the Functionality of the system
Functional testing may show that the system performs as per the requirements Will the system work if now deployed? There are other factors beyond functionality that are critical to the successful use of system
Usability testing
testing to determine the extent to which the software product is understood, easy to learn, easy to operate and attractive to the users under specified conditions.
Employ people to use the application as a user and study how they use it often it is required that virgin users are brought in to test the system A beta test may be a cheap way of doing Usability testing There is no simple way to examine HOW people use the product Testing for ease of learning is not the same as testing for ease of use
Storage Testing = resource utilization testing

The process of testing to determine the resource-utilization of software product.
Study how memory and storage is used by the product Predict when extra capacity may be needed
Instability Testing
The process of testing the instability of a software product Does the installation work? Does installation effect other software Is it easy to carry out?
Does it uninstall correctly?
Documentation testing Testing the quality of the documentation, e.g. user guide or installation guide. Is the documentation complete? Is the documentation accurate? Is the documentation available?
Recovery Testing = Recoverability testing

The process of testing to determine the recoverability of a software product.
Should include recovery from system back-ups Related to reliability
Load testing
Testing geared to assessing the applications ability to deal with the expected throughput of data & users
Load test
A test type concerned with measuring the behavior of a component or system with increasing load, e. g. number of parallel users and/or numbers of transactions to determine what load can be handled by the component or system.
Stress Testing
testing conducted to evaluate a system or component at or beyond the limits of its specified requirements
Each major component in an application or system is tested to its limits

Guides support personnel to prepare for situations when problems are likely to occur Also, Mean Time to Failure MTTF is calculated
Performance Testing
The process of testing to determine the performance of a software product
Related to Efficiency testing
Volume Testing
Testing where the system is subjected to large volumes of data
Related to Resource-utilization testing
Summary
Just because a systems functions have been tested doesnt mean that testing is complete
There are a range of non-functional tests that need to be performed upon a system

4.2. Test Levels 4.2.4. Acceptance Testing
Test Levels Acceptance Testing

Understand what acceptance testing is
Why you would want to do it How you would plan it and prepare for it What you need to actually do it
Understand the different types of acceptance testing
What is Acceptance Testing?
Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.
What is User Acceptance Testing (UAT)?

Exactly what it says it is Users of end product conducting the tests The set-up will represent a working Environment Covers all areas of a project, not just the system A.K.A. Business Acceptance Testing or Business Process Testing
A typical development life-cycle V-Model

User Acceptance Testing
Business Requirements
Technical Specification System Testing Integration Testing (small) Component Testing
Integration Testing (large)
Functional Specification Design Specification
level
Code
time
Planning UAT
Why plan? Things to consider Preparing Tests
Manual Automated
Why Plan?
If you dont, then how do you know you have achieved what you set out to do
Avoids repetition
Test according to code releases
Makes efficient and effective use of time and resources
Things to consider
Timescales Resources Availability
Preparing your tests

Take logical approach Identify the business processes Build into everyday business scenarios
Data Requirements
Copied Environments Created Environments
Running the Tests

Order of tests Confidence Checks Automated and Manual test runs
Contract Acceptance
A demonstration of the acceptance criteria Acceptance criteria will have been defined in the contract Before the software is accepted, it is necessary to show that it matches its specification as defined in the contract
Alpha Testing
Letting your customers do your testing Requires a stable version of the software People who represent your target market use the product in the same way(s) as if they had bought the finished version
Alpha testing is conducted in-house (at the developers site)
Beta Testing
As Alpha testing but users perform tests at their site
Summary
Before software is released it should be subjected to Acceptance Testing
User representation in testing is VITAL

If the product does not pass UAT then a decision about implementation needs to be made

4.3. Test Types the targets of testing
Test Types retesting & regression testing

Understand how fault fixing and re-testing are key to the overall testing process
Look at how test repeatability helps

Understand what regression testing is and where it fits in
Understand how to select test cases for regression testing
Testing is directed at finding faults

These faults will need to be corrected and a new version of the software released
The tests will need to be run again to prove that the fault is fixed
This is known as Re-Testing
The need for re-testing needs to be planned & designed for

Schedules need to allow for re-testing
Tests need to be easily re-run

Test data needs to be re-usable
The environment needs to be easily restored
Regression Testing
Re-testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered as a result of the changes made. It is performed when the software or its environment is changed.

Load
Report function
Program 1
Program 2
Program 3
Printing function
How can you fix this bug break that ?

Program 1 Program 2
Report Function
Program 3 Program 4
Printing function
Program 4
Data Input
Program 5 Program 6
Quote Generation
Tests will also need to be re-run when checking software upgrades

Regression tests should be run whenever there is a change to the software or the environment Regression tests are executed to prove aspects of a system have not changed Regression Testing is a vital testing technique
Selecting cases for regression

Tests that cover safety or business critical functions Tests for areas that change regularly Tests of functions that have a high level of faults
Regression testing is the ideal foundation for Automation

Selecting the right test cases is vital, and requires a degree of knowledge of the application and their expected evolution
Summary
Once a fault has been fixed the software MUST be re-tested
New faults may have been introduced by the fix Existing faults may have been uncovered by the fix Tests need to be written to enable their re-use
Re-testing is the rerunning of failed test once a fix has been implemented to check that the fix has worked Regression testing is running of a wider test suite to check for unexpected errors in unchanged code

4.4. Maintenance testing
Maintenance Testing

Look at the challenges that testers face when an application changes post implementation
See how to ensure that maintenance applied to the system does not cause failures
Maintenance Testing
What is Maintenance Testing?

testing the changes to an operational system or the impact of a changed environment to an operational system.
Testing of changes to existing, established systems

Where maintenance has been performed on the (old) code
Checking that the fixes have been made and that the system has not regressed
Maintenance Testing
The Challenge
Old code No documentation Out of date documentation (worse) How much testing to do and when to stop testing
Mitigation of risk vs. lost revenue
Maintenance Testing
Often application will need maintenance

In doing this minor changes will be required These need to be tested quickly & effectively to allow service to be restored These systems may have been in place and working for years
They are likely to be vital to the running of the business They may be in use 24/7
Maintenance Testing
Main causes for maintenance testing

Migration of the program products Transition of the program products Changing in the production environments Retirement of the software or systems
Maintenance Testing
You need to be able to test the changes quickly and effectively

As well as being able to prove that the change introduced does not impact the other functions A Regression test is in order
Maintenance Testing
Other Risks
Reactive, high risk when changing Impact analysis is vital but is difficult Stopping testing too soon errors may be missed Stopping testing too late missed business through delayed implementation
Maintenance Testing
Summary
All established systems need maintenance from time to time The changed code / function will need to be tested A regression test will need to be executed to ensure that the change(s) have been made correctly and that they have not affected some other part of the system Impact analysis is key, but difficult
Testing throughout the Software Life Cycle
References
CMMI, Craig, 2002, Hetzel, 1998, IEEE 12207 Copeland, 2004, Myers, 1979 Beizer, 1990, Black, 2001, Copeland, 2004 ISO 9126, IEEE 829 Copeland, 2004, Hetzel, 1998

5. Static techniques
Static techniques
Over the next three sessions we will

Discover the various approaches to Static Techniques (Static Testing)
Discover the benefits of Static Testing

Discover the issues with Static Testing
Static techniques
What is Static Testing

Testing of a component or system at specification or implementation level without execution of that software, e.g. reviews or static code analysis
How is this done

By reviewing system deliverables
Static techniques
Session Topics 1. Reviews and the Test Process 2. Types of Review 3. Static Analysis

5.1. Review and the Test Process
Reviews and the Test Process
Session Objectives
To identify what and when to review To cover the Costs & Benefits of Reviews
Why review?
To identify errors as soon as possible in the development lifecycle Reviews offer the chance to find errors in the system specifications This should lead to
Development productivity improvements Reduced development time-scales Lifetime cost reductions Reduced failure levels
When to review?
As soon as an object is ready, before it is used as a product or the basis for the next step in development
What to review?
Anything and everything can be reviewed Requirements, System and Program Specifications should be reviewed prior to publication System Design deliverables should be reviewed both in terms of functionality & technical robustness Code can be reviewed
What should be reviewed?

Program specifications should be reviewed before construction
Code should be reviewed before execution

Test Plans should be reviewed before creating tests
Test Results should be reviewed before implementation
Reviews can be hazardous

If misused they can lead to friction The errors & omissions found should be regarded as a good thing The author(s) should not take errors & omissions personally
It is a positive step to identify defects and issues before the next stage
Building a Quality culture

In order for them to work, Reviews must be regarded as a positive step and must be properly organized
This is often a part of the companys Quality culture

Define the review panel Define the roles of its members Define the review procedures Sell the quality culture to staff Company culture is enhanced to include Quality Control
There are other dangers in a regime of regular reviews

Lack of Preparation
familiarity breeds contempt

No follow up to ensure correction has been made
The wrong people doing them

Used as witch-hunts when things are going wrong
Systems development is very complex

This means that it is difficult to build a system with all elements completely & accurately specified
By reviewing a deliverable, its deficiencies can be identified and remedied earlier saving time and money
What else we can Tester review?

All development and design documentation Test plans Test cases Test results Causes of defect Test metrics Development metrics Operational defects Defects
We can measure how well we do
The costs of reviews

On-going reviews cost approximately 15% of development budget
This includes activities such as the review itself, metric analysis & process improvement
Reviews are highly cost effective
Finding and fixing faults in reviews is significantly cheaper than finding and fixing faults in later stages of testing
The benefits of reviews

Reviews save time and money Development productivity improvements
People take greater care They have more pride in doing good work They have a better understanding of what they are required to deliver
Remove defects / errors earlier Reduced development costs

Reduced fault levels Reduced lifetime cost
Summary
Reviews enable us to ensure that the systems specification is correct and relates to the users requirements
Anything generated by a project can be reviewed

In order for them to be effective, reviews must be well managed

5.2. Review in general
Review in general

Look at the types f review & the activities performed Look at goals & deliverables Look at roles & responsibilities Examine the suitability of each type of review Discover review pitfalls
Review in general Types of Review
Types of review
Authors review Informal Review Walkthroughs Presentations Technical Review Inspections
Review instruments
Prototypes Checklists
Authors review
Before issuing a document , the author should always review his / her work
Conformance to company standards Fitness for purpose Presentations Spelling
Informal Review
It is very helpful to ask a colleague to read through the deliverable to
Identify jargon to eliminate or translate Make sure it all makes sense Rationalize the sequence Identify where the authors familiarity with the subject assumes the reader knows more than he or she actually does
This s the least effective from of review as there is no recognized way of measuring it
Walkthrough
Usually led by the author, although the Project Manager or team Leader may conduct them
For some technical deliverables or complex processes it is very useful for the author to walk through the document
Reviewers are asked to comment and query the concepts and processes that are to be delivered
Presentations
Used to explain to users your understanding of their Requirements
To explain how you will test those requirements i.e. your approach
To gain agreement for the next step
Technical (Peer) Review

Your Peers can also perform more formal technical reviews in larger groups
Often used by workgroups as a self-checking mechanism

Its format is defined, agreed and practised with documented processes and outputs Requires technical expertise
Inspections
Focus on reviewing code or documents Formal process based on rules and checklists
Chaired by Moderator Every attendee has a defined role Includes metrics and entry & exit criteria Training is required
Reviewers must be prepared prior to attendance

Inspections last 2 hours max
Fagan Inspections
Are very effective to the point where they can replace unit testing
Defects are reported no debate

Contentious issues and queries are resolved outside the inspections and the result reported back to the Moderator After Michael Fagan founder of the method www.mfagan.com.
Prototypes (review instrument)

Allows the non-technician to see what is going to be built Enables the business to refine their business flows or adjust sequences on input screens Is used to demonstrate and display features of the [proposed] system
GUI screens, Database designs, Network architecture etc
Checklists (review instrument)

Extremely simple and useful method of ensuring that all elements of a deliverable have been considered
Tends to ensure conformity of content, sequence and style

Can be devised for almost any activity
Review in general
Fundamental procedures and phases of a review

Planning Organizational preparation Individual preparation Review meeting Follow-up
Review in general
Goals
Validation and verification against specifications and standards
Achieve consensus before moving on to next stage
Review in general
Activities
Reviews need to be planned They may require overview meetings Preparation The review meeting Follow-up
Review in general
For each type of review (especially the formal ones) roles and responsibilities must be clearly defined Training may be required to understand and perform the respective roles Roles & responsibilities
Manager Moderator Author Reviewer(s) Secretary
Review in general
Manager decides on the execution of a review. Responsible also to promote and encourage the review process Moderator a neutral person. Ensures that the review stays focused on the goals. This is usually a specially trained person Author writer or person with chief responsibility for the document(s). Authors sometimes take the review personally and do see it as personal criticism Reviewer(s) technical experts who take part in the meeting. Must be open to new ideas. Has to be aware that the author might take any criticism personally Secretary documents the deviations, problems, points and suggestions. Should take as little part in the discussion as possible in order to stay focused
Review in general
Deliverables
Any project deliverable can (and should) be reviewed Reviewing these items can lead to
Product changes Source document changes Improvements (to both reviews and development)
Review in general
Success factors for reviews include:

Each review has a clear predefined objective The right people for the review objectives are involved Defects found are welcomed, and expressed objectively People issues and psychological aspects are dealt with (e.g. making it a positive experience for the author) Review techniques are applied that are suitable to the type and level of software work products and reviewers
Review in general
Success factors for reviews include (cont.):

Checklists or roles are used if appropriate to increase effectiveness of defect identification Training is given in review techniques, especially the more formal techniques, such as inspection Management supports a good review process (e.g. by incorporating adequate time for review activities in project schedules) There is an emphasis on learning and process improvement
Review in general
Pitfalls
Lack of training
And therefore understanding of role, responsibilities and purpose of review
Lack of documentation Lack of management support
Review in general
Review Management
Formal reviews require proper management
Allow people time to prepare Issue guidance notes (if not in Standards) Give plenty of notice Practice Presentation & Walkthroughs Sound out contentious issues ALWAYS document outcomes & Actions
Review in general
Walkthrough
Informal Review
Everything
Technical reviews
Everything
Inspections
What
Req.s & Designs
Specs & Code
Approach
Informal
Informal
Formal
Formal
Why / when
Before detail
Always*
Before detail
Before sign off
By Who
Author(s)
Peer
Group(s)
Moderator
To Whom
IT & Business
Author
IT & Business
Reviewers
Always* : can be done at all points when a document is released
Review in general
Summary
There are various types of reviews

Companies must decide which one(s) are best for them
In order to gain maximum benefit from them they must be organized and implemented

5.3. Static Analysis by tools
Static Analysis by tools
In this session we will:

Understand what Static Analysis is Look at some of the elements of Static Analysis
Compilers Static Analysis tools Data flow analysis Control flow analysis Complexity analysis
What is Static Analysis?

Analysis of software artifacts, e.g. requirements or code, carried out without execution of these software artifacts.
Essence of Static Analysis:

Examine the logic and syntax of the code Attempting to identify errors in the code Provides metrics on flow through the SUT and its complexity A form of automation usually done with tools
Benefits of static analysis:

Early detection of defects prior to test execution. Early warning about suspicious aspects of the code or design, by the calculation of metrics, such as a high complexity measure. Identification of defects not easily found by dynamic testing. Detecting dependencies and inconsistencies in software models, such as links. Improved maintainability of code and design. Prevention of defects, if lessons are learned in development.
What do we hope to find?

Errors in the code
Unreachable code Undeclared variables Parameter type mismatches Uncalled functions and procedures Possible array boundary violations
What else will we get?

Information on the code
Its complexity The flow of data through it The control flow through it Metrics and measurements
Compilers
Convert program language code into Machine Code for faster execution
Will perform some basic Static Analysis

Usually produces a source code list Often produces a memory map Often produces a cross-reference of labels and variables Will do a syntax check on the code prior to compilation
Static Analysis Tools

A Static Analyzer is used to parse the Source Code & identify errors, such as
Missing branches and links Non returnable performs Variables not initialized Standards not met Unreachable code
Will also find any errors found by a Compiler
Data Flow Analysis

Technique to follow the path that various specific items of data take through the code of a program
Looking for possible anomalies in the way the code handles the data items
Variables used without being first created Variables being used after they have been killed
Control Flow Analysis

Examines the flow of control through the code Can identify
Infinite loops Unreachable code Cyclomatic complexity and other metrics
Control Flow Graph
B1
B2
B3
B4
B8
B9
B5
B6
B7
Metrics
Static Analyzers generate metrics based on the code Cyclomatic Complexity
Identifies the number of decision points in a control flow Can be calculated by counting the number of decision points in a control flow graph and adding 1
Other complexity Metrics

Number of Lines of Code Number of nested statements
Summary
Static Analysis in the inspection of program code and logic for errors
The code is analyzed without being executed by the tool

Errors are highlighted and metrics are produced
References:
van Veenendaal, 2004
IEEE 1028
Gilb, 1993

6. Test Design Techniques
Test Design Techniques
In this chapter we will

Understand the differences between Black & White Box testing and where they feature in a testing lifecycle
Understand how a systematic approach provides confidence

Understand how tools can be used to improve and increase productivity

6.1. Identifying test conditions and designing test cases
The following steps are required for the successful process of identifying test conditions and designing test cases:
Designing tests by identifying test conditions Specifying test cases Specifying test procedures
During test cases design, the project documentation is analyzed in order to determine what to test, i.e. to identify the test conditions During test specification the test cases are developed and described in details with the relevant test data by using test design techniques
Expected results should be produced as part of the test case specification and should be included outputs, changes to data, states, etc.

6.2. Categories of test design techniques
Black & White Box Testing
Black Box testing

Testing, either functional or non-functional, without reference to the internal structure of the component or system.
White Box testing

Testing based on an analysis of the internal structure of the component or system.
Black Box Testing
Concentrates on the Business Function

Can be used throughout testing cycle Dominates the later stages of testing
Although is relevant throughout the development life cycle
Little / No knowledge of the underlying code needed
Black Box Testing
We have all done it!

What do you do when changing a light bulb?
Remove Old Bulb

Insert New Bulb
Switch light / lamp on

Switch light / lamp off
White Box Testing
Testing at a detailed level

Normally used after an initial set of tests has been derived using black box test techniques
Appropriate for component testing but becomes less useful as testing moves towards system and acceptance testing
Not aimed at testing the functionality of a component or the interaction of a series of components Needs to be planned
White Box Testing
Testing at a detailed level cont.

A.K.A. Glass Box testing or Structural testing Focuses on Lines of Code Looks at specific conditions Looks at the mechanics of the Application Useful in the early stages of testing
White Box Testing
Requires a detailed knowledge of the code

Business Function / Process not a prime consideration Aim to achieve code coverage, not just functional coverage Plays a lesser role later in the testing cycle
White Box Testing
Code Coverage objective

To devise tests that exercise each significant line of program code at least once
This does not mean every combination of paths and data which is normally unattainable
Techniques and measurements

Systematic techniques exist for black and white box testing
Give us a systematic approach to testing Test cases are developed in a logical manner Enable us to have a sub-set of tests that have a high probability of finding faults
By using defined techniques we should be able to produce corresponding measurements

Means we can see how testing is progressing
Software Testing Tools

Are useful for both Black & White Box Testing Can increase productivity and quality of tests Particularly useful for white box testing
Summary
Black Box testing focuses on functionality White Box testing focuses on code A systematic approach is needed for both
Tests need to be planned, prepared, executed and verified Expected results need to be defined and understood Tools can help increase productivity and quality

6.3. Specification-based or Black Box Techniques
Black Box Techniques

Understand what black box testing is Look at some of the different types of black box testing
Black box testing is

Testing, either functional or non-functional. Without reference to the internal structure of the component or system .
Why do we need black box techniques?

Exhaustive testing is not possible
Due to the constraints of time, money and resources
Therefore we must create a sub-set of tests

These must be achievable, but should not reduce coverage
We should also focus on areas of likely risk

Those places where mistakes may occur
Each black box test techniques has

A method
i.e. how to do it
A test case design approach

How to create test cases using the approach
A measurement technique
Except Random & Syntax
See BS7925-2 for detailed information
BS7925-2 lists all the black box test techniques

Equivalence Partitioning Boundary Value Analysis State Transition Cause & Effect Graphing Syntax Testing Random Testing
Equivalence Partitioning
Uses a model of the component to partition Input & Output values into sets
Such that each value within a set can be reasonably expected to be treated in the same manner
Therefore only one example of that set needs to be input to or resultant from the component
Equivalence Partitioning Test Case Design

Inputs to the component Partitions exercised:
Valid partitions Invalid partitions
Expected results
Example
Enter car details
Engine size 800 5500cc Year of Manufacture Make
Enter car details
Engine Size: Year of manufacture Make
Identify
Valid partition and invalid partitions Test cases including expected results
OK
Boundary Value Analysis

Uses a model of the component to identify the values close to and on partition boundaries
For input and output , valid & invalid data

Chosen specifically to exercise the boundaries of the area under test Most useful when combined with Equivalence Partitioning
Boundary Value Analysis Test Case Design

Inputs to the component Boundaries to be exercised:
Just below On Just above
Expected results
Example
Enter car details
Engine size 800 5500cc Year of Manufacture Make
Identify
Boundaries Test cases Test case category Expected results
Black Box Techniques What are the valid and invalid values for Equivalence Partitioning and Boundary Value Analysis?
Q1 To be eligible for a mortgage you must be between the ages of 18 and 64 (inclusive). The age input field will only accept two digits and will not accept minus figures (-)
Q2 An input field on a mortgage calculator requires a value between 15,000 and 2,000,000. The field only allows numerical values to be entered and has a maximum length of 9 digits.
Q3 The term of a mortgage can be between 5 and 30 years.
Q4 The font formatting box in a word processing package allows the user to select the size of the font ranging from 6 point to 72 point (in 0.5 steps).
Black Box Techniques What are the valid and invalid values for Equivalence Partitioning and Boundary Value Analysis?
Q5 A screen for entering mortgage applications requires information on both peoples wages and will generate the maximum amount available for the mortgage (based on 3 times larger wage , 1 times lower wage). If the mortgage is less than 250,000 then the interest rare is 4.5%, if the amount is 250,000 to 1,000,000 then the interest rate is 4%. Q6 Personal loan of between 1000 to 25000. for loans between 1,000 and 10,000 there is an interest rate of 8.5%, loans between 10,001 and 25,000 have an interest rate of 8%. Q7 A grading system takes students marks (coursework 0 75 and exam 0 25) and generates a grade based on those marks (0 40 Fail, 41 60 C, 61 80 B and 81 100 A).
Decision Table Testing

Uses a model of the relationship between causes and effects for the component Each can cause or effect is expressed as a Boolean condition Represented as a Boolean graph, from which a decision table is produced
Good for:
Capturing system requirements that contain logical conditions Documenting internal system design Recording complex business rules that a system is to implement Serving as a guide to creating test cases that otherwise might not be exercised
Example
A check debit function has inputs: Debit amount Account type P = postal C = counter Current balance And outputs are: New balance Action code D&L = process debit and send out letter D = process debit only S&L = suspend account and send letter L = send out letter only

C1 A1
Example cont
The conditions are:
C2
A2 A3
C3 C1 New balance in credit C2 New balance overdraft, but within authorized limit C3 Account is postal
The actions are:

A1 Process debit A2 Suspend account A3 Send out letter
State transition
Uses analysis of the specification of the component to model its behavior by state transitions
Identifies
The various states that the software may be in The transitions between these points The events that cause the transactions Actions resulting from the transactions
State Transition test Case Design

Identify possible transitions Identify initial state and the data / events that will trigger the transition Identify the expected results
Example
Displaying Time (S1)
Reset (R) Alter time (AT)
Time set (TS) display time (T)
Changing Time (S3)

Test Case 1 2 3 4 5 6
Change
Change
mode (CM)
Display Time (T)
mode (CM)
Display date (D)
Start State
Input Expected Output Finish State
S1
CM D S2
S1
R AT S3
S3
TS T S1
S2
CM T S1
S2
R AD S4
S4
DS D S2
Reset (R) Alter date (AD)
Displaying Date (S2)
Date set (DS) Display date (D)
Changing Date (S4)
Other techniques
Decision Table Testing (Cause & Effect Tabling)
Analysis of the specification to produce a model of relationship between causes and their effects Cases expressed as Boolean conditions in the form of a decision table
Use Case Testing

The tests are designed based on business activities as opposed to based on project requirements
Syntax Testing
Uses the syntax of the components inputs as the basis for the design of the test case
Random Testing
Means of testing functionality with a randomly selected set of values
Summary
Black box testing concentrates on testing the features of the system
Techniques enable us to maximize testing

Create an achievable set of tests that offer maximum coverage Ensure possible areas of risk are tested
Black box testing is relevant throughout the testing process

6.4. Structured-based or White Box Techniques
White Box Techniques

Understand what White Box testing is Look at some of the different types of White Box testing
White Box testing

Testing based on an analysis of the internal structure of the component or system
Also known as Glass Box testing, Clear Box testing
White Box Techniques Why do we need White Box techniques?

Provide formal structure to testing code Enable us to measure how much of a component has been tested Example
<100 lines of code 100,000,000,000,000 possible paths At 1,000 tests per second would still take 3,170 years to test all paths
Loop 20 timeout
To plan and design effective cases requires a knowledge of the

Programming language used
Databases used
Operating system(s) used
And ideally knowledge of the code itself
BS7925-2 lists all the white box test techniques

Statement Testing Branch / Decision Testing Branch Condition Testing Branch Condition Combination Testing Modified Condition Decision Testing Linear Code Sequence & Jump Data Flow Testing
Statement Testing
A white box test design technique in which test cases are designed to execute statements.
Test cases are designed and run with the intention of executing every statement in a component
Statement Testing example

a; if (b)
c;
d;
Any test case with b TRUE will achieve full statement coverage
NOTE: Full statement coverage can be achieved without exercising with b FALSE
Branch / Decision testing

A technique used to execute all branches the code may take based on decision made
Test Cases designed to ensure all branches & decision points are covered
Branch / Decision testing example

a; if (b) c; d; 100% statement coverage requires 1 test case (b = true) 100% branch / decision coverage requires 2 test cases (b = true & b = false)

Exercises Q1 How many tests are required to achieve:
100% statement coverage 100% branch coverage enter user ID IF user ID is valid THEN display enter password IF password is valid THEN display account screen ELSE display wrong password ELSE display wrong ID END IF display time & date

100% statement coverage 100% branch coverage
READ AGE IF AGE >0 THEN IF AGE=21 THEN PRINT 21st ENDIF ENDIF PRINT AGE

100% statement coverage 100% branch coverage READ AGE READ BIRTHDAY IF AGE>0 THEN IF BIRTHDAY = 0 THEN PRINT No values ELSE PRINT BIRTHDAY IF AGE>21 THEN PRINT AGE ENDIF ENDIF ENDIF READ BIRTHMONTH

100% statement coverage 100% branch coverage
READ HUSBANDAGE READ WIFEAGE IF HUSBANDAGE>65 PRINT Husband not retired ENDIF IF WIFEAGE>65 PRINT Wife retired ELSE PRINT Wife not retired ENDIF

Exercises How many tests are required to achieve:
100% statement coverage 100% branch coverage Exercises:Q5
Exercises:Q6
READ HUSBANDAGE READ WIFEAGE IF HUSBANDAGE>65 PRINT Husband retired ENDIF IF WIFEAGE>65 PRINT Wife retired ENDIF
READ HUSBANDAGE IF HUSBANDAGE<65 PRINT Below 65 ENDIF IF HUSBANDAGE>64 PRINT Above 64 ENDIF
Other techniques
Branch Condition testing, Branch Condition Combination testing & Modified Condition Decision testing
Condition testing based upon an analysis of the conditional control flow within the component
LCSAJ
Linear Code Sequence And Jump
Data flow testing

Tests the flow of data through a component
Summary
White box testing can be done immediately after code is written
Doesnt need the complete system Does need knowledge of the code
A combination of all techniques are required for a successful test

Dont rely on just one technique

6.5.Experience-based Technique 6.5.1. Error Guessing
Error Guessing

Understand how you can see experience to predict where errors may occur
Understand how this can benefit testing
Error Guessing
What is error Guessing?

A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made, and to design tests specifically to expose them.
Error Guessing
Used to guess where errors may be lurking

Based on information about how the system has been put together and previous experience of testing it
Use to complement more systematic techniques not instead of

Not ad-hoc testing, but testing that targets certain parts of the application
Error Guessing
Based on the users or testers experience of the commercial aspects of the system
Table based calculations such as calculating benefits payments Where the user is allowed a high degree of flexibility in GUI navigation using multiple windows
Error Guessing
Experience of the developers or the development cycle

Knowledge of an individuals style
Knowledge of the development lifecycle and especially the change management process
Error Guessing
Experience of the operating system used

It is known that Windows NT is better at storage management than Windows 95
Error Guessing
Should still be planned

Is part of the test process
Should be used as a supplement to systematic techniques
Ad-Hoc testing can be encouraged if

The tester maintains a log of those tests performed & is able to reasonably recollect what actions were performed
If you have the time to run extra (Ad-Hoc) test that passes there is little loss
If it finds defect then thats a bonus

6.5.2. Exploratory Testing
Exploratory Testing
Can be powerful approach to testing

Sometimes is much more productive than testing using scripts
Involves tacit constraints concerning which parts of the product are to be tested, or what strategies are to be used
Still this are not test scripts
Every test is influenced by the result of the previous one

6.6. Choosing Test Techniques
Choosing Test Techniques
The choice of a test technique may be made based on:

Regulatory standards Experience and / or Customer / contractual requirements
Some techniques are more applicable to certain situations and test levels; others are applicable to all test levels
Selection factors
The following selection factors are important: Risk level and risk type (thoroughness) Test objective Documentation available Knowledge of the test engineers Time and budget Supporting tools for design / regression testing
Selection factors
Product life cycle status Previous experiences Measurements on test effectiveness International standards many times they are industry specific Customer / contractual requirements Auditability / traceability
A lot of information, skills necessary!
Summary
You can never do enough testing!
The more tests that you can run the more errors you may find
Error guessing can help ensure that areas where defects are likely to occur are fully tested Uses experience and gut feeling to supplement systematic testing techniques Exploratory testing is most useful where there are a few or inadequate specifications or severe time pressure
References:
Craig, 2002, Hetzel, 1998, IEEE 829 Copeland, 2004, Myers, 1979 Kaner, 2002 Beizer, 1990

7. Test Management 7.1. Test Organization
Test Organization

Gain an appreciation of the many different testing structures in place
Understand that it is likely that every organization will have a unique testing set-up
Look at some of the more common, general set-ups and appreciate the impact they may have on the success of the testing carried out
Test Organization
Testing may be the responsibility of the individual developers

Developers are only responsible for testing their own work
Goes against the ideal that testing ought to be independent

Tends to confirm the system works rather than detect errors
Test Organization
Testing may be the responsibility of the developer team

A buddy system
Each developer is responsible for testing his / her colleagues work
Test Organization
Each team has a single tester

Is not involved in the actual development, solely concentrates on testing
Able to work with the team throughout the development process

May be too close to the team to be objective
Test Organization
Dedicated test teams may be in place

These do no development whatsoever Take a more objective view of the system under test
Test Organization
Internal Test Consultants

Provide testing advice to project teams Involvement throughout the lifecycle
Test Organization
Testing outsourced to specialist agencies

Guarantees independence Empathy with the business of the organization?
Test Organization - Main Tasks
Multi disciplined teams needed to cover Production of testing strategies Creation of test plans Production of testing scripts Execution of testing scripts Test asset management Reviewing & reporting Quality assurance Logging results Executing necessary retests Automation expertise User interface expertise Project management Technical support
Database admin, environment admin
Testing terms will typically include:

Test analysts to prepare, execute and analyze tests Test Consultants prepare strategies and plans Test automation experts Load & performance experts Database administrator or designer Test managers / team leaders Test environment managers And others
Test Design Techniques Detailed Tasks
A various tasks and activities hat testers might perform include:

Review and contribute to test plans Analyze, review and assess user requirements, specifications and models for testability Create test specifications Set-up the test environment (often coordinating with system administration and network management) Prepare and acquire test data Implement tests on all test levels, execute and log the tests, evaluate the results and document the deviations from expected results Use test administration or management tools and test monitoring tools as required Automate tests (may be supported by a developer or a test automation expert) Measure performance of components and systems (if applicable ) Review tests developed by others
Test Organization Personnel Qualification
Besides technical skills and knowledge testers

Socially competent and good team players Political and diplomatic skills Assertive, confident, exact and creative, analytical Willing and ready to query in depth Quickly familiarize themselves with complex implementations and applications.
Test Organization Tasks of the test Manager
Test manager = engine and mind of the testing effort

Estimate the time and cost of testing Determine the needed personnel skills Plan for and acquire the necessary resources. Negotiate and set test completion criteria. Define which test cases, by which tester, in which sequence and at which point in time are to be carried out.
Test Organization tasks of the Test Manager
Test manager (continued)

Adapt the test plan to the results and progress of the testing Introduce suitable compensatory measures Report test results and progress/ Receive reports Decide when the tests can be completed based on test completion criteria Introduce and use metrics
Test Organization
Summary
Testing structure will vary between organizations Testing terms require a range of skills

7.2. Test Planning and Estimation
Test Planning and Estimation

Understand how we estimate how long we need for testing Understand how we monitor the progress of testing once it has started Understand how we monitor the progress of testing once it has started Understand what steps we take to ensure that the effort progresses as smoothly as possible

7.2.1. Test Planning

Look at how a test plan is put together Understand how it should be used and maintained Understand why they are so important to a testing project
Test Planning and Estimation What is a test plan? A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst other test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and test measurement techniques to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process.
A project plan for testing
Covering all aspects of testing A living document that should change as testing progresses
Before you plan

Accord with the QA Plan A test strategy in place Put down the test concept
Test plan identifier Introduction Test items Features to be tested Approach Item pass / fail criteria Suspension criteria & resumption criteria
Test deliverables Testing tasks Environment Responsibilities Staffing and training needs Schedules Risks and contingencies Approvals
Based upon IEEE 829-1998 standard for software test documentation
Test Plan Identifier

A unique identifier for the test plan
Introduction
Overview of the plan
A summary of the requirements Discuss what needs to be achieved Detail why testing is needed
Reference to other documents

Quality assurance and configuration management
Test Items
The software items to be tested Their versions numbers / identifiers How they will be handed over to testing References to relevant documents
Features to Be Tested
List all features of the SUT that will be tested under this plan
Features Not to Be Tested

List all features of the SUT that will not be tested under this plan
Between the test items, features to be tested and features not to be tested we have scope of the project
Approach
Describes the approach to testing the SUT This should be high level, but sufficient to estimate the time and resources required
What this approach will achieve Specify major activities Testing techniques Testing tools / aids Constraints to testing Support required environment & staffing
Item Pass / Fail Criteria

How to judge whether a test item has passed
Expected vs. Actual results Certain % of tests pass Number of faults remaining (know and estimated) Should be defined for each test item
Suspension criteria & resumption requirements

Reasons that would cause testing to be suspended Steps necessary to resume testing
Test deliverables
Everything that goes to make up the tests All documentation specification, test plans, procedures, reports Data Testing tools Test management tools, automation tools, Excel, Word etc Test systems, Manual and automated test cases
Testing Tasks
Preparation to perform testing
Test case identification Test case design Test data storage Baseline application
Special skills needed

Spreadsheet skills, test analysis, automation etc
Intertask dependencies
Environment
Requirements for test environment Hardware & software
PCs, servers, routers etc SUT, interfaced applications, databases
Configuration
May be operating systems or middleware to test against
Facilities
Office space, desks, internet access
Responsibilities
Who is responsible? For which activities For which deliverables For the environment
Staffing and Training Needs

Staff required
Test managers, team leaders, testers, test analysts
Skill levels required

Automation skills, spreadsheet skills
Training requirements
Tool specific training, refresher courses
Schedule
Timescales, dates and milestones Resources required to meet milestones Availability of software and environment Deliverables
Risks and Contingencies

What might go wrong? Actions for minimizing impact on testing should things go wrong
Approvals
Who has approved the test plan Names and dates of approval Why is it so important
Evidence that the document has been viewed Shows that the approach has been agreed and has the backing of those who matter You have commitment, now make them stick to it!
Exit Criteria
Should be used to determine whether the software is ready to be released There is never be enough time, money or resource to test everything so testers must focus on business critical function Some examples of test completion
Has Key Functionality been tested Has test Coverage been achieved Has the Budget been used What is the Defect detection rate Is Performance acceptable Are any defects outstanding
Approaches differ based on when the intensive test design work begins
Preventative tests are designed as early as possible
Reactive test design comes after the software or system has been produced
There are some specific approaches

Can be combined or used separately
Model based Methodical Process Dynamic and heuristic
Consultative
Regression - averse

7.2.4. Test Estimation
Test estimation
The same estimating for any project We need to identify the number of tasks (tests) to be performed, the length of each test, the skills and resources required and the various dependencies Testing has a high degree of dependency
You cannot test something until it has been delivered Faults found need to be fixed and re-tested The environment must be available whenever a test is to be run
Factors to Consider
Risk of Failure Complexity of Code / Solution Criticality Coverage Stability Of SUT
Testing is a tool to aid Risk Management

Consider the cost to the business of failure of a feature Test the most important features as soon as possible Be prepared to repeat these tests
Consider the complexity of the supporting code

Estimate & plan tests for these ASAP These are likely to be re-run many times, allow for this in plans
Data is critical to testing

You can in certain situations load the data for testing & test in one pass
These tests will be run first but only once unless

The testing environment may need to be re-set before a test can be re-run
Use previous experience or best estimates to predict the reliability of components

Plan to run tests for the unreliable elements sooner rather than later
Allow time for re-testing

Time must be included for identifying, investigating & logging faults
Faults that have been fixed must be re-tested

Whenever a change is made to the SUT or the environment a regression test should be run How do we build these factors into the estimation?
Summary
Test plans are project plans for testing They identify why testing is needed, what will be tested, the scope of this phase of testing, what deliverables testing will provide and what is required to enable testing to succeed They are living documents that must evolve as the project progresses

7.3. Test Progress Monitoring and Control
Test Progress Monitoring and Control
Test preparation
Estimate number of tests needed Estimate time to prepare
Fit for purpose
Refine these figures

Test Criteria Passes
Track percentage of tests prepared
implementation
time
Why monitor
The purpose of test monitoring is to give feedback and visibility about test activities Information may be collected manually or automatically
Test Execution
Assess the coverage achieved to date by testing
Coverage amount of tests completed, amount of code exercised by tests, amount of features tested so far
Estimate time required to run the test suite
Whilst testing, you can monitor

The number of tests run The number of tests passed & failed The number of defects raised
These can be categorized by Severity, Priority & Probability
The number of re-tests that pass & that fail
The status of the project should be regularly reported

Any deviations from the schedule raised ASAP Any critical faults found should be raised immediately
Describes any guiding or corrective actions taken as a result of information and metrics gathered and reported Controlling measures
Assign extra resource Re-allocate resource Adjust the test schedule
Arrange for extra test environments

Refine the completion criteria
Test Reporting concerned with summarizing information about the testing

What happened during a period of testing
Analyzed information and metrics supporting future actions

According to the Standard for Software Test Documentation (IEEE 829), reporting has a defined structure
Test reporting a test summary report shall have the following structure:
Test summary report identifier Summary Variances Comprehensive assessment Summary of results Evaluation Summary of activities Approvals
Summary
Multiple factors must be considered when estimating the length of time we need to perform testing
Once testing has started it is necessary to monitor the situation as it progress

Careful control must be kept to ensure project success

7.4. Configuration Management
Configuration Management

Gain an understanding of Configuration Management Look at the potential impact of poor Configuration Management on testing And understand how introducing Configuration Management helps address those problems
What is Configuration Management

Generally, ensuring that no-one is able to change a part of the system without proper procedures being in place
The IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to:
Identify and document the functional and physical characteristics of a configuration item, Control changes to those characteristics, Record and report change processing and implementation status, and Verify compliance with specified requirements.
The BS 6488 Code of Practice for Configuration Management

Configuration Management identifies in detail
The total configuration (i.e. hardware , firmware, software, services and supplies) current at any time in the life cycle of each system to which it is applied, Together with any changes or enhancements that are proposed or are in course of being implemented It provides traceability of changes through the lifecycle of each system and across associated systems or groups of systems. It therefore permits the retrospective reconstruction of a system whenever necessary.
Closely linked to Version Control

Version Control looks at each component Holds the latest version of each component What versions of components works with others in a configuration
Without it, testing can be seriously impeded

Developers may not be able to match source to object code Simultaneous changes may made to the same source Unable to identify source code changes made in a particular version of the software
Critical to successful testing

It associates programs with specifications, with test plans, with test data
When a component is successfully tested test results can be included
It enables you to understand what versions of components work with each other
It allows you to understand the relationship between test cases, specifications, and components
If any configuration component is out of step Results are unpredictable

This applies throughout the project lifecycle and beyond
Everything intended to last must be subject to Configuration Management

Anything that isnt, shouldnt exist as part of the product
Other Configuration Management terms Configuration identification

Selecting the configuration items for a system and recording their functional and physical characteristics in technical documentation That is, all Configuration Items and their versions are known
Other Configuration Management terms Configuration control Evaluation, co-ordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. That is, Configuration Items are kept in a library and records are maintained on how Configuration Items change over time
Other configuration management terms Status accounting Recording and reporting of information needed to manage a configuration effectively. This information includes: A listing of the approved configuration identification The status of proposed changes to the configuration, and The implementation status of the approved changes. That is, all actions on Configuration Items are recorded and reported on
Other Configuration Management terms Configuration auditing

The function to check on the contents of libraries of configuration items for standards compliance
Summary
Configuration management enables us to store all information on a system, provides traceability and enables reconstruction Configuration Management is a necessary part of any system development All assets must be known and controlled

ISTQB Certified Tester, Foundation Levels
7.5. Risk and Testing
Risk and Testing
What is a risk:
Can be defined as the chance of an event, hazard, threat or a potential problem
The harm resulting from that event determines the level of the risk
When analyzing, managing and mitigating risks, the test manager follows well established project management principles
Risk and Testing
Project Risks
Relate to the projects capability to deliver its objectives:
Supplier issues: failure of the third party, contractual issues Organizational factors:
Skill and staff shortages Personal and training issues Political issues
Technical issues:
Problems is defining the right requirements The extent that requirements can be met given existing constraints The quality of the design, code and tests
Risk and Testing
Product Risks and risk management activities

Potential failure areas in the software or system They are a risk to the quality of the product
Risk management activities provide a disciplined approach to:

Assess what can go wrong (risks) Determine what risks are important to deal with Implement actions to deal with those risks Identify new risks
Risk and Testing
Risk analysis used to maximize the effectiveness of the overall testing process
A risk factor should be allocated to each function in order to differentiate between
Critical functions
Required functions
Risk and Testing
Critical functions must be fully tested and available as soon as the changes go live. The costs to the business is high if these functions are unavailable for any reason
Required functions not absolutely critical to the business. Usually possible to find adequate methods to work around these problems using other mechanisms

7.6. Incident Management
Incident Management
In this session we shall

Understand what an incident is Understand the impact they have on the testing process Understand how they are logged, and tracked Understand why & how they should be analyzed to help ensure they dont happen again
Incident Management
An incident is when testing cannot progress as planned

any significant, unplanned event that occurs during testing that requires subsequent investigation and / or correction They may be raised against documentation, the System Under Test, the test environment or the tests theselves
Incident Management
The incident needs to be viewed in terms of its impact on testing

As testing progresses through its lifecycle priorities can change The progress of the testing activity can be monitored by analyzing incidents
Incident Management
Top Priority must be given to any issues that prevent testing from progressing
At the second level we need to assign those that can be worked around In the lower categories we log what are minor issues & questions
Incident Management
As project deadlines get nearer what is considered high priority may well change
The emphasis will move towards the product and what will compromise its success
Incident Management
When logging incidents testers should include

Test ID, System ID, Testers ID Expected and Actual results Environment is use Severity & priority of the incident
Incident Management
Service levels should be set up for each category of incident

Progress monitored accordingly
Involve Development, Testers and Users

Issue regular status reports
Incident Management
STATUS should also be tracked

This ensures that we are able to swiftly establish the situation with an incident
Incident Management
A typical status flow is

When first raised, an incident is OPEN It then passes to development to fix WITH DEVELOPMENT Development may question the incident PENDING Or fix the problem and return it for RETEST Should the retest prove successful the defect is CLOSED
Incident Management
Analyzing incidents
Incidents may be analyzed to help monitor the test process And to aid in improvement of the test process
Incident Management
Summary
An incident is any significant, unplanned event that occurs during testing that requires subsequent investigation and / or correction Incidents should be recorded and tracked Analysis of incidents will enable us to see where problems arose and to aid in test process improvement

Cost and Economic Aspects

Look at the cost of testing (or the cost of not testing) Appreciate the value of good, timely testing
The cost of defects that have remained undetected

Direct expenses for the client
Loss of data, interruption of transactions, wrong orders or equipment damage lead to
Indirect costs due to turnover losses

As employees or customers are dissatisfied with a product they will leave
Further cost is added for correction and retesting
The cost of testing

Maturity of the development and test processes, Testability of the software, Personnel qualification, Quality targets, Test strategy applied.
Testing cheaper that defects

Look for optimal balance between
Test cost, Resources, Expected failure costs
In most cases the costs of testing should stay below the costs incurred if faults or deficiencies remain in the end product.
The earlier a fault is found, the cheaper it is to remedy

Errors in the requirements can lead to major re-engineering of the entire system Many errors can be found using reviews
Reviews of documentation and / or code
Allows testing to find more substantial faults
The average cost of fixing a fault increases tenfold with every step of the development process
In code reviews you will find and fix errors in an average of 1 2 minutes
In initial testing, fault fix times will average between 10 and 20 minutes
In integration testing each fault can cost an hour or more
In system test each fault can cost 10 to 40 or more engineer hours

Watts Humphrey
Inspection and testing should begin as early as possible and be used in every phase of the project Early test design can prevent fault multiplication
Analysis of specification during test preparation often brings errors in the specification to light If errors in documentation are not found then the system may be developed incorrectly
The cost of not Testing

What is the cost of testing? Which is cheaper?
Testing and finding faults before release Not testing and finding faults once the system is live
Companies rarely have figures for either
Summary
The purpose of testing is to find faults The earlier a fault is found is cheaper (and easier) it is to fix Starting testing early will help find faults quicker

Standards for Testing

Understand the various standards, norms, guidelines, and practices that may influence testing
A variety of sources for norms, standards, and guidelines for testing

Company standards, policies, and practices
Quality Assurance standards

Industry- or branch-specific standards
Software testing standards
Company standards, policies, and best practices

Developed in-house or Borrowed from industry or branch Written or verbal
Quality Assurance standards

Only specify that testing should be done
E.g. ISO 9000-3:1991 Quality management and quality assurance standards Part3: Guidelines for the application of ISO 9001 to the development, supply and maintenance of software
Industry- or branch-specific standard

Specify what level of testing to perform
E.g. railway, medical, insurance ASTM F153-95 Standard Test Method for Determining the Yield of Wide Inked Computer Ribbons
Testing standards
Specify how to perform testing
E.g. BS7925-1
BS7925-2 IEEE 829
Software Testing vocabulary

Software component testing Standard for Test Documentation
Summary
There are a range of standards that can influence testing These come from different sources and affect different areas of testing Various industries have legal requirements that will influence testing
Sign of development process maturity

Can bring legal or commercial benefits
References
Black, 2001, Craig, 2002
Hetzel, 1998
IEEE 829, Kaner 2002

8. Tool support for testing 8.1. Testing Tool Classification
Testing Tool Classification

Understand that help is at hand in the form of CAST Tools
CAST Computer Aided Software Testing
Look at the range of tools available in the market today

Identify the areas in which tools can best help the testing process
Techniques as Test Tools Products (CAST Tools)

To automate the most repetitious and time consuming tasks Tools are able to do some things that people cannot easily do
Tool support for management of testing and tests:

Test management tools Requirements testing tools
Requirements management tools Requirements testing tools
Problem reporting and tracking

Incident management tools
Configuration management tools
Tool support for static testing:

Review process support tools
May store information about review processes Provide aid for online reviews
Static Analysis Tools purposes

The enforcement of coding standards The analysis of structures and dependencies Aiding in understanding the code
Modelling tools
Tool support for test specification:

Test design tools (definition)
A tool that supports the test design activity by generating test inputs from a specification that may be held is a CASE tool repository, e.g. requirements management tool, or from specified test conditions held in the tool itself.
Test data preparation tools (definition)

A type of test tool that enables data to be selected from existing databases or created, generated, manipulated and edited for use in testing.
Tool support for test execution and logging:

Test execution tools Simulators
Harnesses Test comparators/robots
Security tools
Capture and replay tools = record and playback tools Web testing tools
Coverage Measurement Tools
Tool support for performance and monitoring:

Dynamic analysis tools Performance/ load/ stress testing tools Monitoring tools
Tool support for specific application areas

Tools classified above can be specialized for use in a particular type of application
Commercial tool suites may target specific application areas
Summary
CAST tools can help automate areas of the testing process There is a wide range of functions that can be automated

8.2. Effective Use of Tools: Potential benefits and risks
Effective Use of Tools

Examine the processes involved in identifying where tools can help the testing process
Learn how to approach the selection and implementation of those tools
Potential benefits and risks of tool support (main tasks and activities)
Before buying a tool
Examine the test process

Examine the testing environment (benefits and risks)
Special consideration for some kind of tool

Test execution tools
Replay designed to implement tests that are stored electronically Often requires significant effort in order to achieve significant benefits
Performance testing tools

Need someone with expertise in performance testing to help design the tests and interpret the results
Special consideration for some kind of tool

Static analysis tools
These tools, when applied to source code can enforce coding standards
Test management tools

Need to interface with other tools or spreadsheets The reports need to be designed and monitored so that they provide benefit
As indicated tools can help many aspects of testing

How do you select where you want to start introducing CAST support?
The selection and implementation of a tool is a major task and should be regarded as a project in its own right
Three aspects
Economics Selection Introduction
If testing is currently badly organized tools may not be the immediate answer
Automating chaos simply speeds the delivery of chaos
The benefits of tools usually depend on a systematic and disciplined test process being in place
Address the test processes and introduce disciplines first
Examine the test process in detail

Identify the weaknesses, and the cause of those weaknesses
Find tools that fit in with your processes and address the weak areas
This may be more important than selecting the tool with the most number of features
Prioritize the areas of greatest importance
Examine the Test Environment(s)

One of the most crucial areas for tool implementation Shared environments will not accept automation unless the SUT allows the data to be entered fresh every time Automation may also fill an environment up and make it run out of space as it allows a large amount of data to be entered in a short space of time Test facilities may be (will be!!!) less than Production facilities and use of the tools with the SUT can cause systems to run out of memory
A 6-step process
Requirements specification of the use of the tool Market analysis Demonstrations with manufacturers Evaluation of a smaller selection of tools
Review
Final tool selection
Market Analysis and Funding

Know the tools out there Direct costs the price of the tool Indirect costs man hours, training It is necessary to budget for both the tool and the resources required to implement
Demonstrations Evaluations and Review

Market Demos
Evaluation
Your choice
Pilot project Review of the experiences from the pilot project Process adaptations (if needed) Tool configuration User training On-the-job coaching.
Plan the roll-out of the tool

Gain commitment from the tools users Ensure new projects are aware of the cost of introducing a tool
Summary
Tools can improve the organization and efficiency of a test process
Before tools can be implemented a sound testing process must be in place

Tool selection and implementation must be carefully considered and managed
References
Buwalda, 2001 , Fewster, 1999 Fewster, 1999

ISTQB Certified Tester

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

ISTQB Certified Tester

Hochgeladen von

Copyright:

Verfügbare Formate

ISTQB Certified Tester Foundation Level

Basics for Software Testing

Basics of Software Testing

In this chapter we will introduce

Introduction Certified Tester

Introduction Certified Tester

Managers need to know that testers:

Certification provides reasonable level of confidence

Introduction Certified Tester

Manage the testing process:

Introduction Certified Tester

Identify and use appropiate tools Select test cases depending on

Standardized training program for education of testers International recognition

Syllabus, Examination and Certification

Work Groups Syllabi Exam Questions

German Testing Board

American Testing Board

Bulgaria participates in the SEETB South East European Testing Board

iSQI = International Software Quality Institute

IEEE = Institute of Electrical and Electronics Engineers

For this course

Basics of Software Testing

Why is testing necessary

In this session we will

Why is testing necessary

Why is testing becoming more important?

Why is testing necessary

How Error Occurs

In IT development we have time and budgetary deadlines to meet

Why is testing necessary

How Errors Occur

Why is testing necessary

The Cost Of Failures

Or a significant failure may cost millions

Why is testing necessary

The Cost Of Failures

Usually safety critical systems are tested exhaustively

Unfortunately there are exceptions to this

Why is testing necessary

Example for huge errors

Why is testing necessary

The Cost Of Failures

To correct a problem at requirements stage may cost $1

The model of cost escalation

TS Code Unit Phase of testing

Why is testing necessary

Provides a measure of quality

Why is testing necessary

How much test is enough?

Why is testing necessary

The time required makes it impractical

Why is testing necessary

If we cant test everything, what can we do?

Why is testing necessary

Business Functional and Non Functional Atributes

Why is testing necessary

Should you stop testing when the product goes live?

Better that you find the errors than the users

Why is testing necessary

Testing & Quality

Why is testing necessary

Quality Measurements Quality can be measured by testing for

Why is testing necessary