You are on page 1of 147

22/08/2011

Software testing
Nguyen Thanh Binh IT Faculty, Danang University of Technology Email: ntbinh@dut.udn.vn Website: www.dut.edu.vn/itf/ntb

Prerequisites
No testing experience needed Some familiarity with the development phases of a project would be helpful. Some mathematics would help

References
1. 2. 3. 4. 5. 6. 7. 8. 9.

10.

Paul Jorgensen, Software Testing-A Craftsman's Approach, CRC Press, 1995. Spyos Xanthakis, Pascal Rgnier, Constantin Karapoulios, Le test des logiciels, Hermes Science, 2000. Hung Q. Nguyen and al., Testing application on the Web, John Wiley & Sons, 2004. Ilene Burnstein, Practical Software Testing, Springer, 2003. Glenford J. Myers, The art of software testing, Wiley, 2004. Cem Kaner, Jack Falk, Hung Q. Nguyen, Testing Computer Software, 2nd Edition, John Wiley & Sons, 1999. Boris Beizer, Software Testing Techniques, International Thomson Computer Press, Second Edition, 1990. Neil Bitzenhofer, Software Testing and Verification, Course, MSSE, 2008. Paul Ammann and Jeff Offutt, Introduction to Software Testing, Cambridge University Press, Cambridge, UK, ISBN 0-52188-038-1, 2008. Mauro Pezz, Michal Young, Software Testing and Analysis: Process, Principles, and Techniques, John Wiley & Sons.
3

22/08/2011

Course description
Cover both theoretical and practical aspects of testing software The student will participate in the entire range of test activities:
Analyze a requirements document for test conditions Write a test plan Design, create and execute test cases using various testing approaches Record defects Write a test report
4

Contents
Session 1: Introductory lecture
Introductions and expectations Course overview Contents

Contents
Session 2: Introduction to Software Testing
Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing

22/08/2011

Contents
Session 3: Requirements analysis
Software Development Life Cycle (SDLC) Software Development stage
Requirements

Testing and requirements


Learn to think like a tester

Some examples

Contents
Session 4
Exercise 1: Examining requirements

Contents
Session 5: Structural Testing
White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code
Testing State Behavior

22/08/2011

Contents
Session 6: Static Testing
Reviews and the test process Types of review Static analysis

10

Contents
Session 7: Functional Testing
Introduction to functional testing Functional testing techniques
Boundary Value testing Equivalence Class testing Special Value testing Decision Tables

11

Contents
Session 8: Test Documentation
Test Plan
The need for test plans The structure of test plans A Test Plan Template A Test Plan example Testing on a large project

Test Cases
Test Case Design Test Case Examples
12

22/08/2011

Contents
Session 9 Exercise 2: Writing a test plan and test cases

13

Contents
Session 10: Levels of Testing
Levels of Testing
Integration Testing System Testing

Additional System Test Categories

14

Contents
Session 11: Defect Reports/Test Reports
Handling Defects Bug Tracking System Test Reports Examples

15

22/08/2011

Contents
Session 12: Test Automation and Tools
Test Automation Test tools

16

Why test?
List of 107 software failures that should have been caught by testing http://www.cs.tau.ac.il/~nachumd/verify/horro r.html One vital consideration from Myers book The Art of Software Testing Mars Climate Orbiter Mars Polar Lander
17

Software Errors
1. 2.

3. 4.

5.

6. 7.

The Mars Climate Orbiter crashed in September 1999 because of a "silly mistake": wrong units in a program. The 1988 shooting down of the Airbus 320 by the USS Vincennes was attributed to the cryptic and misleading output displayed by the tracking software. Death resulted from inadequate testing of the London Ambulance Service software. Several 1985-7 deaths of cancer patients were due to overdoses of radiation resulting from a race condition between concurrent tasks in the Therac-25 software. Errors in medical software have caused deaths. Details in B.W. Boehm, "Software and its Impact: A Quantitative Assessment," Datamation, 19(5), 48-59(1973). An Airbus A320 crashes at an air show. A China Airlines Airbus Industrie A300 crashes on April 26, 1994 killing 264. Recommendations include software modifications.
18

22/08/2011

Software Errors
The Explosion of the Ariane 5
On June 4, 1996 an unmanned Ariane 5 rocket launched by the European Space Agency exploded just forty seconds after its lift-off from Kourou, French Guiana. The rocket was on its first voyage, after a decade of development costing $7 billion. The destroyed rocket and its cargo were valued at $500 million. A board of inquiry investigated the causes of the explosion and in two weeks issued a report. It turned out that the cause of the failure was a software error in the inertial reference system. Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storeable in a 16 bit signed integer, and thus the conversion failed.
19

Software Errors
The Patriot Missile Failure
On February 25, 1991, during the Gulf War, an American Patriot Missile battery in Dharan, Saudi Arabia, failed to track and intercept an incoming Iraqi Scud missile. The Scud struck an American Army barracks, killing 28 soldiers and injuring around 100 other people. A report of the General Accounting office, GAO/IMTEC-92-26, entitled Patriot Missile Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia reported on the cause of the failure. It turns out that the cause was an inaccurate calculation of the time since boot due to computer arithmetic errors. Specifically, the time in tenths of second as measured by the system's internal clock was multiplied by 1/10 to produce the time in seconds. This calculation was performed using a 24 bit fixed point register. In particular, the value 1/10, which has a non-terminating binary expansion, was chopped at 24 bits after the radix point. The small chopping error, when multiplied by the large number giving the time in tenths of a second, led to a significant error. Indeed, the Patriot battery had been up around 100 hours, and an easy calculation shows that the resulting time error due to the magnified chopping error was about 0.34 seconds. A Scud travels at about 1,676 meters per second, and so travels more than half a kilometer in this time. This was far enough that the incoming Scud was outside the "range gate" that the Patriot tracked. Ironically, the fact that the bad time calculation had been improved in some parts of the code, but not all, contributed to the problem, since it meant that the inaccuracies did not cancel.
20

Software Errors
Y2K
Spent some billions dollars

21

22/08/2011

Chapter 1 of the textbook A Perspective on Testing


Basic Definitions
Error a mistake in design, coding, requirements, even testing Fault the representation of the error Failure what happens when the fault executes

22

A Perspective on Testing
More Definitions
Testing the process of finding errors and of validating the program/system Test Case a test case has Inputs Steps Outputs Expected results Process Test plan, Write test cases Run the test cases Evaluate results
23

A Perspective on Testing
Test Cases
Test Cases will be discussed in detail in Session 7 and throughout the course. Testing entails establishing the environment, providing the inputs (running the test case), observing outputs, and comparing to expected outputs. Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!
24

22/08/2011

A Perspective on Testing
Identifying Test Cases
Functional Testing
The program is a function that maps input values to output values The only information used is the software specification In our Venn diagram, the Functional Test Cases are a subset of S Further elaborated on in Part II Math background: Chapter 3 We will discuss in Session 6
25

A Perspective on Testing
Identifying Test Cases
Structural Testing
Uses the information inside the black box the actual implementation In our Venn diagram, the Structural Test Cases are a subset of P. Further elaborated on in Part III of the text Math background: Chapter 4 We will discuss this in Session 4 Main method: Test coverage metrics
26

A Perspective on Testing
Identifying Test Cases
Comparing the two (Functional vs Structural)
We will discuss this in Sessions 4 and 6 If all specified behaviors have not been implemented, structural test cases will never be able to recognize this. Conversely, if the program implements behaviors that have not been specified, this will never be revealed by functional test cases.

27

22/08/2011

A Perspective on Testing
Levels of Testing
Again, this will be covered in detail in Session 2.
Requirements Specification System Testing

Preliminary Design

Integration Testing

Detailed Design

Unit Testing

Coding

28

Testing a Program
A program that we want to test reads in 3 integer values these 3 values are interpreted as the lengths of the sides of a triangle. The program prints a message that states whether the triangle is
Equilateral (all 3 sides equal) Isosceles (exactly 2 of the 3 sides are equal) Scalene (all 3 sides are of a different length)

On a sheet of paper, write specific sets of test data that you feel would adequately test this program.

29

Introduction to software testing

10

22/08/2011

Contents
Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing

31

Definitions, Principles, Axioms

32

Definitions
Testing Verification Validation Error/Defect Black box testing White box testing

33

11

22/08/2011

Definitions of Testing (1)


The process of executing a program (or part of a program) with the intention of finding errors (Myers, Humphrey) The purpose of testing is to find errors. Testing is the process of trying to discover every conceivable fault or weakness in a work product (Myers, Kit) The process of searching for errors (Kaner)
34

Definitions of Testing (2)


Testing the process of finding errors and of validating the program/system (Jorgensen) The purpose of software testing is to find errors and to get them fixed (Bitzenhofer) The purpose of software testing is to reduce risk (E&M)

35

What Testing is Not


Testing is not just another phase of the project Testing is not just finding defects Testing is not debugging Testing is not a final exam

36

12

22/08/2011

Verification and Validation: Myers


Verification: An attempt to find errors by executing a program in a test or simulated environment Validation: An attempt to find errors by executing a program in a real environment

37

Verification and Validation: IEEE


Verification: The process of evaluating a system or component to determine whether the productssatisfy the conditions imposed Validation: The process of evaluating a system or componentto determine whether it satisfies specified requirements.

38

Verification and Validation: Kaner


Verification: Checking a program against the most closely related design documents or specifications (Design Verification Testing) Validation: Checking the program against the published user or system requirements (System Validation Testing)

39

13

22/08/2011

Software Fault Terminology


Error a mistake in design, coding, requirements, even testing Fault the representation of the error Failure what happens when the fault executes

40

Software Fault Terminology: Examples


Error
A wrong loop index Programmer-to-device pointer error

Fault the representation of the error


A loop executes one too many times A programmed parameter gets written to the wrong address

Failure what happens when the fault executes


A table overflows; memory gets overwritten The physician only thinks he/she changed that heart parameter

41

Black and White Box


Black box testing
Designed without knowledge of the programs internal structure and design Based on functional requirements Also called Functional Testing

White box testing


Examines the internal design of the program Requires detailed knowledge of its structure Also called Structural Testing
42

14

22/08/2011

Six Essentials of Software Testing (1)


1.The quality of the test process determines the success of the test effort. 2.Prevent defect migration by using early lifecycle testing techniques. 3.The time for software testing tools is now.
Adapted from Software Testing in the Real World, Edward Kit; AddisonWesley, 1995

43

Six Essentials of Software Testing (2)


4.A real person must take responsibility for improving the testing process. 5.Testing is a professional discipline requiring trained, skilled people. 6.Cultivate a positive team attitude of creative destruction.
Adapted from Software Testing in the Real World, Edward Kit; AddisonWesley, 1995

44

Some Principles and Axioms of Testing (1)


Program testing can be used to show the presence of bugs, but never their absence. (Dijkstra, 1969) One of the most difficult problems in testing is knowing when to stop. It is impossible to test your own program.

45

15

22/08/2011

Some Principles and Axioms of Testing (2)


As the number of detected defects in a piece of software increases, the probability of the existence of more undetected defects also increases. Testing is an extremely creative and intellectually challenging task (Myers again) Exhaustive testing is impossible.

46

Stages of testing

47

The Test Process


Requirements phase Planning phase Design and development phase Execution phase Reporting phase

48

16

22/08/2011

Software Test Execution Stages


Unit Test Design Verification Test (DVT)
Interface, Integration, Exit/Formal

System Validation Test (SVT)


Main, Regression, Acceptance

Customer Acceptance Test

49

Unit Test
Performed by the developers White-box testing Should be done with each release to testing Results should be communicated to the Test Group

50

Typical Entry Criteria (Unit Test)


Module compiles successfully (no errors, approved warnings) Unit Test Plan complete Unit Test Cases complete Tools are available

51

17

22/08/2011

Typical Unit Test Activities


Source level debugging if it wont compile Design and review the Unit Test cases Track test defects (maybe) Measure Unit Test coverage Make sure it builds

52

Typical Exit Criteria (Unit Test)


All tests attempted 99% of the tests executed successfully 85% statement coverage Test cases have been stored under Configuration Management

53

Software Test Stages


Unit Test Design Verification Test
Interface, Integration, Exit/Formal Test Phase

System Validation Test


Main, Regression, Acceptance

Customer Acceptance Test

54

18

22/08/2011

Design Verification Test (DVT)


Performed by the test organization Black box testing Verification of engineering requirements Verification of the software design

55

Three Components of DVT


Interface Testing Integration Testing DVT Exit/Formal Test Phase

56

Typical Entry Criteria (DVT)


Unit Test complete (for all code being tested) Engineering requirements and program design are complete The code is under Configuration Management A current build is available that contains all necessary integrated code Interface and Integration test cases have been written
57

19

22/08/2011

Typical DVT activities


Design, write and review the Test Plan Write the DVT Test Cases Execute Interface/Integration test cases Track defects to the code and to the tests

58

Interface Testing
How does a user interact with the program? What input variables are required? What outputs are produced? Extremely important for e-business applications

59

Integration Testing
Integration between software components Integration with the front-end GUIs Integration between a server and its clients Integration between software and hardware Integration between Web services

60

20

22/08/2011

DVT Exit or Formal Test Phase


This is a requirement for System Test Probably involves running a suite of Regression Tests Defect requirements of some sort must be present The whole point: Are we ready for System Test?

61

Typical Exit Criteria (DVT)


All test cases executed 98% of the test cases executed successfully High severity defects have been fixed and verified

62

Software Test Stages


Unit Test Design Verification Test
Interface, Integration, Exit/Formal

System Validation Test


Main, Regression, Acceptance

Customer Acceptance Test

63

21

22/08/2011

System Validation Test (SVT)


Performed by the test organization Validation of customer requirements Three components
Main Test Regression Test SVT Acceptance Test

64

Typical Entry Criteria (SVT)


DVT Exit completes successfully Test Plan has been updated and frozen All test cases have been finalized No high severity problems are open Code is frozen The install works

65

SVT Main Test


Verify the functional requirements of the product from the Customers point of view Includes usability, reliability, performance, and installation Also has its own Entry and Exit criteria Use Cases form a big part of this testing Marketing issues are addressed here too

66

22

22/08/2011

SVT Regression Test


Verify that all defects identified during SVT have been addressed; i.e. all tests run OK

67

System Acceptance Test


Verify final manufacturing build Verify all installation scenarios Verify delivery of all media

68

Typical Exit Criteria (SVT)


All test cases attempted All test cases executed successfully, OR every defect that has been submitted has been addressed No high severity defects are open Test Cases are under Configuration Management Final Test Report has been written
69

23

22/08/2011

Software Test Stages


Unit Test Design Verification Test
Interface, Integration, Exit

System Validation Test


Main, Regression, Exit

Customer Acceptance Test

70

Customer Acceptance Testing


Usually performed at the customer site Must be well-written and simple, but meaningful May be dictated by the customer

71

Perspectives on Software Testing

72

24

22/08/2011

The Objectives and Limits of Testing (1)


Limits
You cannot test a program completely Even if you do find the last bug, youll never know it It takes more time than you have to test less than youd like You will run out of time before you run out of test cases

73

The Objectives and Limits of Testing (2)


Limits
You cannot You cannot You cannot You cannot isnt!) test every path test every valid input test every invalid input prove a program correct (because it

74

The Objectives and Limits of Testing (3)


Objectives
The purpose of testing is to find problems The purpose of finding problems is to get them corrected The purpose of testing is to reduce identifiable risks

75

25

22/08/2011

Tests and Test Cases (1)


Test a collection of Test Cases Test Case a test case has inputs, steps, and outputs, as well as expected results Process
write the Test Plan write test cases run the test cases evaluate results
76

Tests and Test Cases (2)


We will discuss Test Cases in detail in Session 7 and throughout the course. Testing entails establishing the environment, providing the inputs (running the test case), observing outputs, and comparing to expected outputs. (a paraphrase) Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!

77

What is Quality?
Quality from the Producers viewpoint: Does the product meet the requirements? Quality from the Customers viewpoint: Fitness for use (reliability, usability, portability, etc.) Quality (one strongly held view): The measure of a products quality is the satisfaction of the customers, not whether it meets a specification

78

26

22/08/2011

Quality Assurance vs Quality Control


Quality Assurance is the activity that establishes and evaluates the processes that produce products
QA improves the product by improving the process

Quality Control is the activity that evaluates the product produced


QC improves the product by finding defects

79

A Generic Testing Approach (SPRAE)


Specification
A written statement of expected software behavior

Premeditation
Written test plans, test scripts, test data, test schedules

Repeatability
Same results every time a result of a mature process

Accountability
Test logs; analysis and interpretation of results

Economy
The cost effectiveness of testing
80

The Testers Toolkit


Static testing Structural testing methods Functional testing methods Non-functional techniques

81

27

22/08/2011

Requirements analysis

Contents
Software Development Life Cycle (SDLC) Software Development stage
Requirements

Testing and requirements


Learn to think like a tester

Some examples Writing test requirements


83

Software Development Life Cycle (SDLC)

84

28

22/08/2011

Software Development Life Cycle (SDLC)


SDLC is a disciplined and systematic approach that divides the software development process into various phases, such as requirement, design, and coding The phase-wise software development process helps you track schedule, cost, and quality of the software projects

85

Software Development Life Cycle (SDLC)


There are six phases of SDLC
Feasibility analysis phase includes the analysis of project requirement. Requirement analysis and specification phase includes gathering, analyzing, validating, and specifying requirements. Design phase includes translation of requirements specified in the SRS into a logical structure that can be implemented in a programming language. Coding phase includes implementation of design specified in the design document into executable programming language code. Testing phase includes detection of errors in the software. Maintenance phase includes implementation of changes and new requirements in the software at the customer location.

86

SDLC models
Are a tailored form of SDLC phases. Provide a basis for categorizing and controlling the various activities required to develop and maintain the software system Typical types of SDLC models are
Linear model Iterative model

87

29

22/08/2011

Linear Models
Linear models are suitable for projects where all the requirements are identified and well understood before the design of the software begins. The two types of linear models are:
Waterfall model Prototyping model

88

Waterfall model
Describes the software development process in a linear sequential flow. Defines the software development process in seven phases:
Conception Initiation Analysis Design Construction Integration and testing Implementation and maintenance
89

Waterfall model

90

30

22/08/2011

Prototyping model
Is a sample implementation of the system that shows limited and main functional capabilities of the proposed system. Helps the customer determine how the features will function in the final software.

91

Prototyping model

92

Iterative model
Iterative model is used when the requirements for the software are likely to evolve throughout the development process. The various types of iterative models are
Spiral model Component-based development model Unified process

93

31

22/08/2011

Spiral model
Spiral model
Includes the iterative nature of the prototyping model and the linear nature of the waterfall model. Is ideal for developing software that are released in various versions.

The six phases of spiral model are:


Customer communication Planning Risk analysis Engineering Construction and release Customer evaluation
94

Spiral model

95

Component-Based Development Model


In a component-based development model:
Components are reused and combined with other components in the same or other computers in a distributed network to form an application. All related modules that form a component are tested to ensure that they work together

96

32

22/08/2011

Advantages: Linear Models


Requirements are in the hands of the testers early The project is better designed, with better focus Late changes to requirements or design are limited Test tasks of scheduling, planning, etc., are made easier
97

Advantages: Iterative Models


A shippable product is available early Progress (Where are we?) is much easier to track Many opportunities to reappraise requirements or rethink the design Easy to drop features

98

Disadvantages: Linear Models


Big decisions must be made early It is unknown what will be difficult to implement Changing a specification affects everyone Tremendous pressure on testers to prove that a product is NOT ready for release

99

33

22/08/2011

Disadvantages: Iterative Models


Product design may become a bottleneck Products often ship with half the features missing Big temptation for developers to keep writing new code rather than fix whats broken Lack of early planning may cause problems when later features are added How does Marketing market the product? Constant re-running of the same tests Potentially longer product cycles
100

Effects on Testing: Linear Models


Requirements must be well-reviewed early Test plan can be written early When System Testing does start, you are on the critical path

101

Effects on Testing: Iterative Models


Testing starts as soon as the first level of functionality is reached Test plan is written as you go Requests for design changes can be more easily made

102

34

22/08/2011

Software Development stage


Requirements

103

Software Development Stages


Project/Product Planning Requirements and Design Coding and Documentation Testing and fixing bugs Maintenance and Enhancement

104

Why Projects are Cancelled


Project Impaired Factors % of Responses 1. Incomplete Requirements 13.1% 2. Lack of User Involvement 12.4% 3. Lack of Resources 10.6% 4. Unrealistic Expectations 9.9% 5. Lack of Executive Support 9.3% 6. Changing Requirements & Specifications 8.7% 7. Lack of Planning 8.1% 8. Didn't Need It Any Longer 7.5% 9. Lack of IT Management 6.2% 10. Technology Illiteracy 4.3% Other 9.9%
105

35

22/08/2011

Why Projects are Challenged


Project Challenged Factors % of Responses 1. Lack of User Input 12.8% 2. Incomplete Requirements & Specifications 12.3% 3. Changing Requirements & Specifications 11.8% 4. Lack of Executive Support 7.5% 5. Technology Incompetence 7.0% 6. Lack of Resources 6.4% 7. Unrealistic Expectations 5.9% 8. Unclear Objectives 5.3% 9. Unrealistic Time Frames 4.3% 10. New Technology 3.7% Other 23.0%
106

Why Projects Succeed


Project Success Factors 1. User Involvement 2. Executive Management Support 3. Clear Statement of Requirements 4. Proper Planning 5. Realistic Expectations 6. Smaller Project Milestones 7. Competent Staff 8. Ownership 9. Clear Vision & Objectives 10. Hard-Working, Focused Staff Other % of Responses 15.9% 13.9% 13.0% 9.6% 8.2% 7.7% 7.2% 5.3% 2.9% 2.4% 13.9%
107

Software Requirements
According to software engineering theory, software requirements should contain a finite list of behaviors and features, and each requirement should be written to be verifiable. Given a finite list of requirements and a set of completion criteria, requirements-based testing becomes a feasible process.
108

36

22/08/2011

Typical Requirements (FURPS)


Functionality does it do what I need? Usability is it easy to do what I need? Reliability can it do what I need when I need it? Performance does it do what I need in a reasonable amount of time? Supportability what if is DOESNT do what I need?
109

Functionality Requirements
Correctness
Extent to which a program satisfies its specifications and meets customer expectations

Features/Capabilities Generality Security


Extent to which access to software or data by unauthorized persons can be controlled
110

Usability Requirements
Ease of use
Measures the effort required to learn, operate, prepare input for, and interpret the output of, a program

Intuitive Human factors Consistency Documentation

111

37

22/08/2011

Reliability Requirements
Frequency and severity of failure - MTBF Recoverability Predictability Accuracy

112

Performance Requirements
Speed Efficiency Resource consumption Throughput Response time

113

Supportability Requirements
Testability Adaptability Maintainability
Effort required to locate and fix errors

Serviceability
Effort required to modify/update a program

Installability Portability
114

38

22/08/2011

Contents

Testing and requirements


Learn to think like a tester

115

Testing Requirements
Usually done by review Are they complete? Are they reasonable? Can they be tested?

116

Requirements Testing
Testing of the requirements Determine all testable requirements Determine all non-testable requirements Write and execute the tests

117

39

22/08/2011

Reviewing Software Requirements


Learn to recognize what has not been specified (limits, consequences) Learn to recognize what is vague or ambiguous (sometimes, faster) As a tester:
Make sure testers are included in the reviews! Push back to get requirements clarified Always be thinking And just how am I going to test this?
118

Reviewing the Design


Helps determine how to test Shows what to look for Do you need to read the code?

119

Some examples
Some requirements statements Some examples of Test Cases

120

40

22/08/2011

Requirements Example 1
After a high temperature is detected, an alarm must be raised quickly

121

Testable Requirements: Ex. 1


If the threshold temperature is reached, an alarm is triggered. As long as the temperature remains below the threshold temperature, no alarm is triggered. If the temperature dips below the threshold (having gone above it), the alarm does not cease.
122

Requirements Example 2
Novice users should be able to learn the interface with little training When thinking about a test for this, ask yourself: What would be the criterion for saying the test passed? failed?

123

41

22/08/2011

Requirements Example 3
There was a hidden requirement in this feature change: The menu option 6. Utilities | 1. Security will now display a new submenu rather than a Security dialog box. The submenu will have 3 items - 1. CSM, 2. CIS, and 3. Options.

124

Requirements Example 4
How many testable requirements are here? The Access Profile choices for 2. CIS and 3. Options will be grayed out unless those options have been purchased and installed.

125

Testable Requirements for Ex. 4


The Access Profile choice for CIS is available if and only if the CIS feature has been installed. The Access Profile choice for Options is available if and only if the Options feature has been installed.

126

42

22/08/2011

Some Test Cases for Example 4


1. 2. 3. 4.

If the CIS option is installed, Access Profile choices will be available for CIS If the Options option is installed, Access Profile choices will be available for Options If the CIS option is not installed, Access Profile choices for CIS will not be available If the Options option is not installed, Access Profile choices for Options will not be available
127

Requirements Example 5
An upgraded security system was being installed with enhanced password protection. Here is one requirement: Existing passwords which do not conform to the new restrictions will be reset to the User ID, and a list of those users whose passwords were reset will be generated.
128

Requirements Example 6
Untestable requirement Number of invalid attempts before a user is suspended: This count is reset when a successful logon is completed for the user. The default is that the User will never be suspended. The valid range is from 0 (never) to 10 attempts.

129

43

22/08/2011

Testable Requirements for Example 6


Requirement: The number of invalid attempts before suspension may be set to 2 Test Case 1: After 2 bad attempts to log on, the Users USERID is suspended. Test Case 2: A bad attempt, followed by a good attempt, can then be followed by 2 bad attempts again before suspension. The same holds for 1, 3-10 Requirement: The number of invalid attempts before suspension cannot be set to 11 Test Case 3: Attempt to set invalid attempts to 11, or verify it cant be done.

130

Requirements Example 7
Vagueness in different disguises: The Advanced Color Module will improve the print quality of the module. The Advanced Color Module firmware will increase the number of color shades per panel from 4 - 9 shades to 25 - 30 shades. The Advanced Color Module will reduce the tiling effects associated with previous modules.
131

Requirements Example 8
A lot of test cases from just one sentence: All new and modified screens will provide online help.

132

44

22/08/2011

Requirements Example 9
The user interface must be easy to use. In case of failure, the system must be easy to recover and must suffer minimum loss of important data.

133

Lessons Learned
Watch for words like always, never, every, etc. They are difficult to verify. Words like must, should, shall, will, has to, etc. signal potential requirements. Watch for hidden or unstated requirements. Watch for vague requirements: high, quickly, better, faster, should be able to. For negative testing, ask What happens if the test case fails? Ask yourself the question: How will I know if the requirement is not met?
134

Structural testing

45

22/08/2011

Contents
White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code
Testing State Behavior

136

White box testing / Structural testing

137

What is white-box software testing?


What is white-box software testing?
Basic idea is to test a program based on the structure of a program

Pseudonyms: Glass Box, Clear Box, Crystal Box, Structural Testing To distinguish from functional (requirements-based, black-box testing) What do you need for white-box testing?
A white-box testing model and test criteria A white-box test design and generation method Program source code
138

46

22/08/2011

White-Box Testing Objectives


The major objective of white-box testing is to focus on internal program structure, and discover all internal program errors
Errors made by developers

The major testing focuses


Program structures Program statements and branches Various kinds of program paths Program internal logic and data structures Program internal behaviors and states
139

Traditional White-Based Software Testing Methods


Test Model: control flow graph Test case design
Various white-box testing methods generate test cases based on a given control flow graph for a program

The goal is to
Guarantee that all independent paths within a module have been exercised at least once Exercise all logical decisions one their true and false sides Execute all loops at their boundaries and within their operational bounds Exercise internal data structures to assure their validity Exercise all data define and use paths
140

No guarantees
Executing all control flow elements does not guarantee finding all faults
Execution of a faulty statement may not always result in a failure The state may not be corrupted when the statement is executed with some data values Corrupt state may not propagate through execution to eventually lead to failure

What is the value of structural coverage?


Increases confidence in thoroughness of testing Removes some obvious inadequacies
141

47

22/08/2011

Graph theory

142

Definition of a Graph
A set N of nodes, N is not empty A set N0 of initial nodes, N0 is not empty A set Nf of final nodes, Nf is not empty A set E of edges, each edge from one node to another
( ni , nj ), i is predecessor, j is successor

143

Three Example Graphs


0 0 1 2 0

Not a valid graph

3 N0 = { 0 } Nf = { 3 }

8 N0 = { 0, 1, 2 } Nf = { 7, 8, 9 }

3 N0 = { } Nf = { 144 3}

48

22/08/2011

Paths in Graphs
Path : A sequence of nodes [n1, n2, , nM]
Each pair of nodes is an edge

Length : The number of edges


A single node is a path of length 0

Subpath : A subsequence of nodes in p is a subpath of p Reach (n) : Subgraph that can be reached from n
0 1 2

Paths [ 0, 3, 7 ] 6

Reach (0) = { 0, 3, 4, 7, 8, 5, 1, 9 } Reach ({0, 2}) = G Reach([2,6]) = {2, 6, 9}


145

[ 1, 4, 8, 5, 1 ] [ 2, 6, 9 ]

Test Paths and SESEs


Test Path : A path that starts at an initial node and ends at a final node Test paths represent execution of test cases
Some test paths can be executed by many tests Some test paths cannot be executed by any tests

SESE graphs : All test paths start at a single node and end at another node
Single-entry, single-exit N0 and Nf have exactly one node

1 0 2 3

4 6 5

Double-diamond graph Four test paths [ 0, 1, 3, 4, 6 ] [ 0, 1, 3, 5, 6 ] [ 0, 2, 3, 4, 6 ] [ 0, 2, 3, 5, 6 ]


146

Tests and Test Paths


path (t) : The test path executed by test t path (T) : The set of test paths executed by the set of tests T Each test executes one and only one test path A location in a graph (node or edge) can be reached from another location if there is a sequence of edges from the first location to the second
Syntactic reach : A subpath exists in the graph Semantic reach : A test exists that can execute that subpath
147

49

22/08/2011

Testing and Covering Graphs


We use graphs in testing as follows :
Developing a model of the software as a graph Requiring tests to visit or tour specific sets of nodes, edges or subpaths

Test Requirements (TR) : Describe properties of test paths Test Criterion : Rules that define test requirements Satisfaction : Given a set TR of test requirements for a

criterion C, a set of tests T satisfies C on a graph if and only if for every test requirement in TR, there is a test path in path(T) that meets the test requirement tr
Control flow (Structural) Coverage Criteria : Defined on a

graph just in terms of nodes and edges


Data Flow Coverage Criteria : Requires a graph to be

annotated with references to variables

148

Control flow test criteria

149

Node and Edge Coverage


The first (and simplest) two criteria require that each node and edge in a graph be executed Node Coverage (NC) : Test set T satisfies node coverage on graph G if only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n.
We can abbreviate it in terms of the set of test requirements

Node Coverage (NC) : TR contains each reachable node in G.


150

50

22/08/2011

Node and Edge Coverage


Edge coverage is slightly stronger than node coverage Edge Coverage (EC) : TR contains each reachable path of length up to 1, inclusive, in G.
The length up to 1 allows for graphs with one node and no

edges
NC and EC are only different when there is an edge and

another subpath between a pair of nodes (as in an if-else statement)


0 1 2 Node Coverage : TR = { 0, 1, 2 } Test Path = [ 0, 1, 2 ] Edge Coverage : TR = { (0,1), (0, 2), (1, 2) } Test Paths = [ 0, 1, 2 ] [ 0, 2 ]

151

Covering Multiple Edges


Edge-pair coverage requires pairs of edges, or subpaths of length 2 Edge-Pair Coverage (EPC) : TR contains each reachable Edgepath of length up to 2, inclusive, in G.
The length up to 2 is used to include graphs that have less

than 2 edges
The logical extension is to require all paths

Complete Path Coverage (CPC) : TR contains all paths in G.


Unfortunately, this is impossible if the graph has a loop, so a

weak compromise is to make the tester decide which paths: Specified Path Coverage (SPC) : TR contains a set S of 152 test paths, where S is supplied as a parameter. parameter.

Structural Coverage Example


Node Coverage TR = { 0, 1, 2, 3, 4, 5, 6 } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] 0 1 2 3 4 5 6 Edge Coverage TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ] Edge-Pair Coverage TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6], [2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 5, 4, 6 ] Complete Path Coverage Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ] 153

51

22/08/2011

Loops in Graphs
If a graph contains a loop, it has an infinite number of paths Thus, CPC is not feasible SPC is not satisfactory because the results are subjective and vary with the tester Attempts to deal with loops:
1970s : Execute cycles once ([4, 5, 4] in previous example, informal) 1980s : Execute each loop, exactly once (formalized) 1990s : Execute loops 0 times, once, more than once (informal description) 2000s : Prime paths

154

Simple Paths and Prime Paths


Simple Path : A path from node ni to nj is simple if no node appears more than once, except possibly the first and last nodes are the same
No internal loops Includes all other subpaths A loop is a simple path

Prime Path : A simple path that does not appear as a proper subpath of any other simple path
Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ], [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ], [ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ], [ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ], [0], [1], [2], [3] Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ], [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ], 155 [ 2, 3, 0, 1 ]

0 1 3 2

Prime Path Coverage


A simple, elegant and finite criterion that requires loops to be executed as well as skipped

Prime Path Coverage (PPC) : TR contains each prime path in G.

Will tour all paths of length 0, 1, That is, it subsumes node, edge, and edge-pair coverage

156

52

22/08/2011

Round Trips
Round-Trip Path : A prime path that starts and ends at the same node Simple Round Trip Coverage (SRTC) : TR contains at least one roundround-trip path for each reachable node in G that begins and ends a roundround-trip path path. . Complete Round Trip Coverage (CRTC) : TR contains all round-trip paths for each reachable node in G. round These criteria omit nodes and edges that are not in round trips That is, they do not subsume edge-pair, edge, or node coverage
157

Prime Path Example


The previous example has 38 simple paths Only nine prime paths
0 1 2 3 4 5 6 Prime Paths [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5] [ 0, 2, 4, 6 ] [ 5, 4, 6 ] [ 4, 5, 4 ] [ 5, 4, 5 ] Execute loop 0 times Execute loop once Execute loop more than once

158

! means path terminates

Simple & Prime Path Example


Simple paths 0 1 2 3 4 5 6 Len 4 [0, 1, 2, 3, 6] ! [0, 1, 2, 4, 6] ! [0, 1, 2, 4, 5] ! Len 0 [0] [1] [2] [3] [4] [5] [6] ! Len 1 [0, 1] [0, 2] [1, 2] [2, 3] [2, 4] [3, 6] ! [4, 6] ! [4, 5] [5, 4] Len 2 [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * Len 3 [0, 1, 2, 3] [0, 1, 2, 4] [0, 2, 3, 6] ! [0, 2, 4, 6] ! [0, 2, 4, 5] ! [1, 2, 3, 6] ! [1, 2, 4, 5] ! [1, 2, 4, 6] ! * means path cycles

Prime Paths

159

53

22/08/2011

Data flow test criteria

160

Data Flow Criteria


Goal: Try to ensure that values are computed and used correctly Definition (def) : A location where a value for a variable is stored into memory Use : A location where a variables value is accessed def (n) or def (e) : The set of variables that are defined by node n or edge e use (n) or use (e) : The set of variables that are used by node n or edge e
Z = X*2 X = 42 0 1 3 2 5 Z = X-8 4 6 Defs: def (0) = {X} def (4) = {Z} def (5) = {Z} Uses: use (4) = {X} use (5) = {X}
161

DU Pairs and DU Paths


DU pair : A pair of locations (li, lj) such that a variable v is defined at li and used at lj Def-clear : A path from li to lj is def-clear with respect to variable v if v is not given another value on any of the nodes or edges in the path Reach : If there is a def-clear path from li to lj with respect to v, the def of v at li reaches the use at lj du-path : A simple subpath that is def-clear with respect to v from a def of v to a use of v du (ni, nj, v) the set of du-paths from ni to nj du (ni, v) the set of du-paths that start at ni
162

54

22/08/2011

Data Flow Test Criteria


Three criteria
Use every def Get to every use Follow all du-paths

163

Data Flow Test Criteria


First, we make sure every def reaches a use

AllAll -defs coverage (ADC) : For each set of du du-paths S = du (n, v), TR contains at least one path d in S.
Then we make sure that every def reaches all possible uses

All-uses coverage (AUC) : For each set of duAlldu-paths to uses S = du (ni, nj, v), TR contains at least one path d in S.
Finally, we cover all the paths between defs and uses

AllAll -dudu-paths coverage (ADUPC) : For each set S = du (ni, nj, v), TR contains every path d in S. 164

Data Flow Testing Example


Z = X*2 X = 42 0 1 3 2 5 Z = X-8 4 6

All-defs for X [ 0, 1, 3, 4 ]

All-uses for X [ 0, 1, 3, 4 ] [ 0, 1, 3, 5 ]

All-du-paths for X [ 0, 1, 3, 4 ] [ 0, 2, 3, 4 ] [ 0, 1, 3, 5 ] [ 0, 2, 3, 5 ]
165

55

22/08/2011

Graph Coverage Criteria Subsumption

Complete Path Coverage CPC Prime Path Coverage PPC

All-DU-Paths Coverage ADUP All-uses Coverage AUC All-defs Coverage ADC

Edge-Pair Coverage EPC Edge Coverage EC Node Coverage NC

Complete Round Trip Coverage CRTC Simple Round Trip Coverage SRTC
166

Graph Coverage for Source Code

167

Graph Coverage for Source Code


The most common application of graph criteria is to program source Graph : Usually the control flow graph (CFG) Node coverage : Execute every statement Edge coverage : Execute every branch Loops : Looping structures such as for loops, while loops, etc. Data flow coverage : Augment the CFG defs are statements that assign values to variables uses are statements that use variables
168

56

22/08/2011

Control Flow Graphs


A CFG models all executions of a method by describing control structures Nodes : statements or sequences of statements (basic blocks) Edges : Transfers of control Basic Block : A sequence of statements such that if the first statement is executed, all statements will be (no branches) CFGs are sometimes annotated with extra information
branch predicates defs uses

Rules for translating statements into graphs


169

CFG : The if Statement


if (x < y) { y = 0; x = x + 1; } else { x = y; } 1
x<y y=0 x=x+1 x >= y

2 4

x=y

if (x < y) { y = 0; x = x + 1; }

1
x<y y=0 x=x+1

x >= y

3
170

CFG : The if-Return Statement


if (x < y) { return; } print (x); return; 1
x<y return

x >= y

print (x) return

No edge from node 2 to 3. The return nodes must be distinct.

171

57

22/08/2011

Loops
Loops require extra nodes to be added Nodes that do not represent statements or basic blocks

172

CFG : while and for Loops


x = 0; while (x < y) { y = f (x, y); x = x + 1; }
x=0

1 dummy node 2

x<y

x >= y

3
y =f(x,y) x=x+1

implicitly initializes loop

x=0

2
x<y x >= y

for (x = 0; x < y; x++) { y = f (x, y); }

y = f (x, y)

x=x+1

implicitly increments loop

173

CFG : The case (switch) Structure


read ( c) ; switch ( c ) { case N: y = 25; break; case Y: y = 50; break; default: y = 0; break; } print (y);

1
c == N

read ( c );

c == Y default

y = 25; break;

3
y = 50; break;

y = 0; break;

5
print (y);

174

58

22/08/2011

Example Control Flow


public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println System.out.println System.out.println System.out.println System.out.println } ("length: " + length); ("mean: " + mean); ("median: " + med); ("variance: " + var); ("standard deviation: " + sd);

175

Control Flow Graph for Stats


public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length;

i=0 i >= length

varsum = 0; i < length for (int i = 0; i < length; i++) i++ 4 { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println System.out.println System.out.println System.out.println System.out.println } ("length: " + length); ("mean: " + mean); ("median: " + med); ("variance: " + var); ("standard deviation: " + sd);

5 i=0

6 i < length i >= length 7 8


176

Control Flow TRs and Test Paths PPC


1

3 4 5

Prime Path Coverage TR Test Paths A. [ 3, 4, 3 ] i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] B. [ 4, 3, 4 ] ii. [ 1, 2, 3, 4, 3, 4, 3, C. [ 7, 6, 7 ] 5, 6, 7, 6, 7, 6, 8 ] D. [ 7, 6, 8 ] iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ] E. [ 6, 7, 6 ] iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ] F. [ 1, 2, 3, 4 ] v. [ 1, 2, 3, 5, 6, 8 ] G. [ 4, 3, 5, 6, 7 ] H. [ 4, 3, 5, 6, 8 ] I. [ 1, 2, 3, 5, 6, 7 ] J. [ 1, 2, 3, 5, 6, 8 ]

59

22/08/2011

Data Flow Coverage for Source


def : a location where a value is stored into memory
x appears on the left side of an assignment (x = 44;) x is an actual parameter in a call and the method changes its value x is a formal parameter of a method (implicit def when method starts) x is an input to a program

use : a location where variables value is accessed


x appears on the right side of an assignment x appears in a conditional test x is an actual parameter to a method x is an output of the program x is an output of a method in a return statement

If a def and a use appear on the same node, then it is only a DUpair if the def occurs after the use and the node is in a loop
178

Example Data Flow


public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println System.out.println System.out.println System.out.println System.out.println } ("length: " + length); ("mean: " + mean); ("median: " + med); ("variance: " + var); ("standard deviation: " + sd);
179

Control Flow Graph for Stats


1
( numbers ) sum = 0 length = numbers.length

i=0

3
i < length

i >= length med = numbers [ length / 2 ] mean = sum / (double) length; varsum = 0 i=0 i >= length var = varsum / ( length - 1.0 ) sd = Math.sqrt ( var ) print (length, mean, med, var, sd)

4
sum += numbers [ i ] i++

6
i < length varsum = i++

180

60

22/08/2011

CFG for Stats With Defs & Uses 1 def (1) = { numbers, sum, length }
2
def (2) = { i }

3
use (3, 4) = { i, length }

use (3, 5) = { i, length }

4
def (4) = { sum, i } use (4) = { sum, numbers, i }

def (5) = { med, mean, varsum, i } use (5) = { numbers, length, sum }

6
use (6, 7) = { i, length }

use (6, 8) = { i, length }

7 def (7) = { varsum, i } use (7) = { varsum, numbers, i, mean }

8
181

Defs and Uses Tables for Stats


Node 1 2 3 4 5 6 7 8 { varsum, i } { var, sd } { varsum, numbers, i, mean } { varsum, length, var, mean, med, var, sd } { sum, i } { med, mean, varsum, i } { numbers, i, sum } { numbers, length, sum } Def { numbers, sum, length } {i} Use Edge (1, 2) (2, 3) (3, 4) (4, 3) (3, 5) (5, 6) (6, 7) (7, 6) (6, 8) { i, length }
182

Use

{ i, length } { i, length } { i, length }

DU Pairs for Stats


variable numbers length med var sd mean sum varsum i (1, 4) (1, 5) (1, 7) (1, 5) (1, 8) (1, (3,4)) (1, (3,5)) (1, (6,7)) (1, (6,8)) (5, 8) (8, 8) (8, 8) (5, 7) (5, 8) (1, 4) (1, 5) (4, 4) (4, 5) (5, 7) (5, 8) (7, 7) (7, 8) (2, 4) (2, (3,4)) (2, (3,5)) (2, 7) (2, (6,7)) (2, (6,8)) (4, 4) (4, (3,4)) (4, (3,5)) (4, 7) (4, (6,7)) (4, (6,8)) (5, 7) (5, (6,7)) (5, (6,8)) (7, 7) (7, (6,7)) (7, (6,8)) DU Pairs

61

22/08/2011

DU Paths for Stats No Duplicates


There are 38 DU paths for Stats, but only 12 unique
[ 1, 2, 3, 4 ] [ 1, 2, 3, 5 ] [ 1, 2, 3, 5, 6, 7 ] [ 1, 2, 3, 5, 6, 8 ] [ 2, 3, 4 ] [ 2, 3, 5 ] [ 4, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 5, 6, 8 ] [ 7, 6, 7 ] [ 7, 6, 8 ]

5 expect a loop not to be entered 5 require at least one iteration of a loop 2 require at least two iteration of a loop
184

Test Cases and Test Paths


Test Case : numbers = (44) ; length = 1 Test Path : [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] Additional DU Paths covered
[ 1, 2, 3, 4 ] [ 2, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 7, 6, 8 ] The five stars that require at least one iteration of a loop

Test Case : numbers = (2, 10, 15) ; length = 3 Test Path : [ 1, 2, 3, 4, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 7, 6, 8 ] DU Paths covered
[ 4, 3, 4 ] [ 7, 6, 7 ] The two stars that require at least two iterations of a loop

Other DU paths require arrays with length 0 to skip loops But the method fails with divide by zero on the statement
mean = sum / (double) length; A fault was found
185

Testing State Behavior

186

62

22/08/2011

Testing State Behavior


A finite state machine (FSM) is a graph that describes how software variables are modified during execution Nodes : States, representing sets of values for key variables Edges : Transitions, possible changes in the state
switch up Off switch down
187

On

Finite State Machine Two Variables


Tropical Depression
circulation = yes windspeed < 39mph

Tropical Storm
circulation = yes windspeed = 39..73 mph

Something Else
circulation = no windspeed = 0..38 mph

Major Hurricane
circulation = yes windspeed >= 110 mph

Hurricane
circulation = yes windspeed 74..109 mph

Other variables may exist but not be part of state

188

Finite State Machines are Common


FSMs can accurately model many kinds of software
Embedded and control software (think electronic gadgets) Abstract data types Compilers and operating systems Web applications

Creating FSMs can help find software problems Numerous languages for expressing FSMs
UML statecharts Automata State tables (SCR) Petri nets

Limitations : FSMs are not always practical for programs that have lots of states (for example, GUIs)
189

63

22/08/2011

Annotations on FSMs
FSMs can be annotated with different types of actions
Actions on transitions Entry actions to nodes Exit actions on nodes

Actions can express changes to variables or conditions on variables These slides use the basics:
Preconditions (guards) : conditions that must be true for transitions to be taken Triggering events : changes to variables that cause transitions to be taken

This is close to the UML Statecharts, but not exactly the same
190

Example Annotations
open elevator door Closed pre: elevSpeed = 0 trigger: openButton = pressed Open

191

Covering FSMs
Node coverage : execute every state (state coverage) Edge coverage : execute every transition (transition coverage) Edge-pair coverage : execute pairs of transitions (transition-pair) Data flow:
Nodes often do not include defs or uses of variables Defs of variables in triggers are used immediately (the next state) Defs and uses are usually computed for guards, or states are extended FSMs typically only model a subset of the variables

Generating FSMs is often harder than covering them


192

64

22/08/2011

Deriving FSMs
With some projects, a FSM (such as a statechart) was created during design
Tester should check to see if the FSM is still current with respect to the implementation

If not, it is very helpful for the tester to derive the FSM Strategies for deriving FSMs from a program:
1. 2. 3. 4.

Combining control flow graphs Using the software structure Modeling state variables Using implicit or explicit specifications Class Watch uses class Time

Discussion of these strategies based on a digital watch

193

Class Watch
class Watch
// Constant values for the button (inputs) private static final int NEXT = 0; private static final int UP = 1; private static final int DOWN = 2; // Constant values for the state private static final int TIME = 5; private static final int STOPWATCH = 6; private static final int ALARM = 7; // Primary state variable private int mode = TIME; // Three separate times, one for each state private Time watch, stopwatch, alarm;

class Time ( inner class )


private int hour = 0; private int minute = 0; public void changeTime (int button) public String toString ()

public Watch () // Constructor public void doTransition (int button) // Handles inputs public String toString () // Converts values

194

// Takes the appropriate transition when a button is pushed. public void doTransition (int button) { switch ( mode ) { case TIME: if (button == NEXT) mode = STOPWATCH; else watch.changeTime (button); break; case STOPWATCH: if (button == NEXT) mode = ALARM; else stopwatch.changeTime (button); break; case ALARM: if (button == NEXT) mode = TIME; else alarm.changeTime (button); break; default: break; } } // end doTransition()

// Increases or decreases the time. // Rolls around when necessary. public void changeTime (int button) { if (button == UP) { minute += 1; if (minute >= 60) { minute = 0; hour += 1; if (hour >= 12) hour = 0; } } else if (button == DOWN) { minute -= 1; if (minute < 0) { minute = 59; hour -= 1; if (hour <= 0) hour = 12; } } } // end changeTime()
195

65

22/08/2011

1. Combining Control Flow Graphs


The first instinct for inexperienced developers is to draw CFGs and link them together This is really not a FSM Several problems
Methods must return to correct callsites built-in nondeterminism Implementation must be available before graph can be built This graph does not scale up

Watch example
196

CFGs for Watch


doTransition () 1 2 2 3 4 6 8 11 12 13 10 14 12
197

changeTime () 1 2 4 6 8 10

4 10

2. Using the Software Structure


A more experienced programmer may map methods to states These are really not states Problems
Subjective different testers get different graphs Requires in-depth knowledge of implementation Detailed design must be present

Watch example
198

66

22/08/2011

Software Structure for Watch


inMain
button button==NEXT / change mode button==UP or button==DOWN ??? Should inChangeTime transit back to inMain??? button==UP / change hour, minute

inDoTransition

inChangeTime

??? Not clear what triggers this button==DOWN / change hour, minute
199

3. Modeling State Variables


More mechanical State variables are usually defined early First identify all state variables, then choose which are relevant In theory, every combination of values for the state variables defines a different state In practice, we must identify ranges, or sets of values, that are all in one state Some states may not be feasible
200

State Variables in Watch


Constants NEXT, UP, DOWN Not relevant, really TIME, STOPWATCH, ALARM just values Non-constants int mode Time watch, stopwatch, alarm Time class variables int hour Merge into the three Time variables int minute
201

67

22/08/2011

State Variables and Values


Relevant State Variables
These three ranges actually represent quite a bit of thought and semantic domain knowledge of the program

mode : TIME, STOPWATCH, ALARM watch : 12:00, 12:01..12:59, 01:00..11:59 stopwatch : 12:00, 12:01..12:59, 01:00..11:59 alarm : 12:00, 12:01..12:59, 01:00..11:59
Total 3*3*3*3 = 81 states But the three watches are independent, so we only care about 3+3+3 = 9 states (still a messy graph )

202

State Variable Model for Watch


mode = TIME watch = 12:00
next next next

mode = TIME watch = 12:01.. 12:59


next next next

mode = TIME watch = 01:00..12:59


next next next

mode = STOP stopw = 12:00


next next next

mode = STOP stopw = 12:01.. 12:59


next next next

mode = STOP stopw = 01:00..12:59


next next next

mode = ALARM alarm = 12:00

next nextnext

mode = ALARM alarm = 12:01.. 12:59


next next next next

mode = ALARM alarm = 01:00..12:59


next next
203

NonDeterminism in the State Variable Model


Each state has three outgoing transitions on next This is a form of non-determinism, but it is not reflected in the implementation Which transition is taken depends on the current state of the other watch The 81-state model would not have this non-determinism This situation can also be handled by a hierarchy of FSMs, where each watch is in a separate FSM and they are organized together
204

68

22/08/2011

4. Using Implicit or Explicit Specifications


Relies on explicit requirements or formal specifications that describe software behavior These could be derived by the tester These FSMs will sometimes look much like the implementationbased FSM, and sometimes much like the state-variable model
For watch, the specification-based FSM looks just like the statevariable FSM, so is not shown

The disadvantage of FSM testing is that some implementation decisions are not modeled in the FSM

205

Tradeoffs in Applying Graph Coverage Criteria to FSMs


There are two advantages
1. 2.

Tests can be designed before implementation Analyzing FSMs is much easier than analyzing source

There are three disadvantages


1. 2. 3.

Some implementation decisions are not modeled in the FSM There is some variation in the results because of the subjective nature of deriving FSMs Tests have to mapped to actual inputs to the program the names that appear in the FSM may not be the same as the names in the program
206

Static testing

69

22/08/2011

Contents
Reviews and the test process Types of review Static analysis

208

Reviews and the test process

209

Static testing
Static: Not Dynamic, i.e. you dont execute the code This is the process of testing some documentation products The primary goal of Static Testing is to reduce defects in the software by reducing defects in the documentation from which the software is developed
210

70

22/08/2011

Static techniques
Do not execute code
People techniques

Specific techniques
Desk checking Developer tests his/her own documentation Walkthroughs Developer walks the participants through the documentation Peer Reviews/Inspections More formal documentation is distributed beforehand
211

Benefits of reviews
Development productivity improvement Reduced development timescales Reduced testing time and cost Lifetime cost reductions Reduced fault levels Improved customer relations
212

Reviews are cost-effective


10 times reduction in faults reaching test, testing cost reduced by 50% to 80%
Freedman & Weinberg, Handbook of Walkthroughs, Inspections & Technical Reviews

reduce faults by a factor of 10


Yourdon, Structured Walkthroughs

25% reduction in schedules, remove 80% 95% of faults at each stage, 28 times reduction in maintenance cost, many others
Gilb & Graham, Software Inspection
213

71

22/08/2011

What can be Inspected?


Anything written down can be Inspected
policy, strategy, business plans, marketing or advertising material, contracts system requirements, feasibility studies, acceptance test plans test plans, test designs, test cases, test results system designs, logical & physical software code user manuals, procedures, training material
214

What can be reviewed?


anything which could be Inspected
i.e. anything written down

plans, visions, big picture, strategic directions, ideas project progress


work completed to schedule, etc.

Should we develop this marketing options

215

What to review / Inspect?


Tests Requirements Tests Functions Tests Design Tests Code Unit Test
216

Accept. Test

System Test

Integration T

72

22/08/2011

Costs of reviews
Rough guide: 5%-15% of development effort
half day a week is 10%

Effort required for reviews


planning (by leader / moderator) preparation / self-study checking meeting fixing / editing / follow-up recording & analysis of statistics / metrics process improvement (should!)
217

Types of review

218

Types of review of documents


Informal Review (undocumented)
widely viewed as useful and cheap (but no one can prove it!). A helpful first step for chaotic organisations.

Technical Review (or peer review)


includes peer and technical experts, no management participation. Normally documented, fault-finding. Can be rather subjective.

Decision-making Review
group discusses document and makes a decision about the content, e.g. how something should be done, go or no-go decision, or technical comments
219

73

22/08/2011

Types of review of documents


Walkthrough
author guides the group through a document and his or her thought processes, so all understand the same thing, consensus on changes to make

Inspection
formal individual and group checking, using sources and standards, according to generic and specific rules and checklists, using entry and exit criteria, Leader must be trained & certified, metrics required
220

Reviews in general 1
Objectives / goals
validation & verification against specifications & standards achieve consensus (excluding Inspection) process improvement (ideal, included in Inspection)

221

Reviews in general 2
Activities
planning overview / kickoff meeting (Inspection) preparation / individual checking review meeting (not always) follow-up (for some types) metrics recording & analysis (Inspections and sometimes reviews)

222

74

22/08/2011

Reviews in general 3
Roles and responsibilities
Leader / moderator - plans the review / Inspection, chooses participants, helps & encourages, conducts the meeting, performs follow-up, manages metrics Author of the document being reviewed / Inspected Reviewers / Inspectors - specialised fault-finding roles for Inspection Managers - excluded from some types of review, need to plan project time for review / Inspection Others: e.g. Inspection/ review Co-ordinator

223

Reviews in general 4
Deliverables
Changes (edits) in review product Change requests for source documents (predecessor documents to product being reviewed / Inspected) Process improvement suggestions
to the review / Inspection process to the development process which produced the product just reviewed / Inspected

Metrics (Inspection and some types of review)


224

Reviews in general 5
Pitfalls (they dont always work!)
Lack of training in the technique (especially Inspection, the most formal) Lack of or quality of documentation - what is being reviewed / Inspected Lack of management support Failure to improve processes

225

75

22/08/2011

Inspection is different
the document to be reviewed is given out in advance typically dozens of pages to review instructions are "please review this" not just product, sources chunk or sample training, roles

226

Inspection is different
some people have time to look through it and make comments before the meeting (which is difficult to arrange) the meeting often lasts for hours entry criteria to meeting, may not be worth holding

2 max., often much shorter

227

The Inspection Process


Software Development Stage

Change Request
Process Improvement

. Entry Planning Kick off Ind Chk Meet

Exit Edit

.
Next Software 228 Development Stage

76

22/08/2011

Static analysis

229

What can static analysis do?


Remember: static techniques do not execute the code A form of automated testing
check for violations of standards check for things which may be a fault

Descended from compiler technology


a compiler statically analyses code, and knows a lot about it, e.g. variable usage; finds syntax faults static analysis tools extend this knowledge can find unreachable code, undeclared variables, parameter type mis-matches, uncalled functions & procedures, array bound violations, etc.
230

Data flow analysis


This is the study of program variables
variable defined* where a value is stored into it variable used where the stored value is accessed variable is undefined before it is defined or when it goes out of scope
x is defined, y and z are used x=y+z IF a > b THEN read(S) a and b are used, S is defined
*defined should not be confused with declared
231

77

22/08/2011

Data flow analysis faults


n := 0 read (x) n := 1 while x > y do begin read (y) write( n*y) x := x - n end
232

Data flow anomaly: n is re-defined without being used Data flow fault: y is used before it has been defined (first time around the loop)

Control flow analysis


Highlights:
nodes not accessible from start node infinite loops multiple entry to loops whether code is well structured, i.e. reducible whether code conforms to a flowchart grammar any jumps to undefined labels any labels not jumped to cyclomatic complexity and other metrics
233

Cyclomatic complexity
cyclomatic complexity is a measure of the complexity of a flow graph
(and therefore the code that the flow graph represents)

the more complex the flow graph, the greater the measure it can most easily be calculated as:
complexity = number of decisions + 1
234

78

22/08/2011

Which flow graph is most complex?


1
What is the cyclomatic complexity?

3
235

Example of control flow graph


Pseudo-code:
Result = 0 Right = 0 DO WHILE more Questions IF Answer = Correct THEN Right = Right + 1 ENDIF END DO Result = (Right / Questions) IF Result > 60% THEN Print "pass" ELSE Print "fail ENDIF

init do if r=r+1

end

res

if

pass

fail

end

236

Other static metrics


lines of code (LOC) operands & operators (Halsteads metrics) fan-in & fan-out nesting levels function calls OO metrics: inheritance tree depth, number of methods, coupling & cohesion symbolic evaluation
237

79

22/08/2011

Limitations and advantages


Limitations
cannot distinguish "fail-safe" code from programming faults or anomalies (often creates overload of spurious error messages) does not execute the code, so not related to operating conditions

Advantages
can find faults difficult to "see" gives objective quality assessment of code
238

Summary
Reviews help to find faults in development and test documentation, and should be applied early Types of review: informal, walkthrough, technical / peer review, Inspection Static analysis can find faults and give information about code without executing
239

Functional testing

80

22/08/2011

Contents
Introduction to functional testing Functional testing techniques
Boundary Value testing Equivalence Class testing Special Value testing Decision Tables

241

Introduction to functional testing

242

Functional testing
Functional testing: Deriving test cases from program specifications
Functional refers to the source of information used in test case design, not to what is tested

Also known as:


specification-based testing (from specifications) black-box testing (no view of the code)

Functional specification = description of intended program behavior


either formal or informal
243

81

22/08/2011

Systematic vs Random Testing


Random (uniform):
Pick possible inputs uniformly Avoids designer bias A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person) But treats all inputs as equally valuable

Systematic (non-uniform):
Try to select inputs that are especially valuable Usually by choosing representatives of classes that are apt to fail often or not at all

Functional testing is systematic testing


244

Why Not Random?


Non-uniform distribution of faults Example: Java class roots applies quadratic equation
Incomplete implementation logic: Program does not properly handle the case in which b2 - 4ac < 0 and a=0 Random sampling is unlikely to choose a=0.0 and b=0.0

245

Systematic Partition Testing


Failure (valuable test case) No failure Failures are sparse in the space of possible inputs ... ... but dense in some parts of the space

The space of possible input values (the haystack)

If we systematically test some cases from each part, we will include the dense parts

Functional testing is one way of drawing pink lines to isolate regions with likely failures 246

82

22/08/2011

Functional testing: exploiting the specification


Functional testing uses the specification (formal or informal) to partition the input space
E.g., specification of roots program suggests division between cases with zero, one, and two real roots

Test each category, and boundaries between categories


No guarantees, but experience suggests failures often lie at the boundaries (as in the roots program)

247

Why functional testing?


Timely Often useful in refining specifications and assessing testability before code is written Effective finds some classes of fault (e.g., missing logic) that can elude other approaches Widely applicable to any description of program behavior serving as spec at any level of granularity from module to system testing. Economical typically less expensive to design and execute than structural (code-based) test cases

248

Early functional test design


Program code is not necessary
Only a description of intended behavior is needed Even incomplete and informal specifications can be used Although precise, complete specifications lead to better test suites

Early functional test design has side benefits


Often reveals ambiguities and inconsistency in spec Useful for assessing testability And improving test schedule and budget by improving spec Useful explanation of specification or in the extreme case (as in XP), test cases are the spec
249

83

22/08/2011

Functional versus Structural: Classes of faults


Different testing strategies (functional, structural) are most effective for different classes of faults Functional testing is best for finding design faults Structural testing is best for finding programming faults
250

Functional vs structural test: granularity levels


Functional test applies at all granularity levels:
Unit Integration System Regression
(from module interface spec) (from API or subsystem spec) (from system requirements spec) (from system requirements + bug history)

Structural (code-based) test design applies to relatively small parts of a system:


Unit Integration

251

Steps: From specification to test cases


1. Decompose the specification
If the specification is large, break it into independently testable features to be considered in testing

2. Select representatives
Representative values of each input, or Representative behaviors of a model
Often simple input/output transformations dont describe a system. We use models in program specification, in program design, and in test design

3. Form test specifications


Typically: combinations of input values, or model behaviors

4. Produce and execute actual tests


252

84

22/08/2011

From specification to test cases

253

Simple example: Postal code lookup

Input: ZIP code (5-digit US Postal code) Output: List of cities What are some representative values (or classes of value) to test?
254

Example: Representative values


Simple example with one input, one output

Correct zip code


With 0, 1, or many cities

Malformed zip code

Note prevalence of boundary values (0 cities, 6 characters) and error cases

Empty; 1-4 characters; 6 characters; very long Non-digit characters Non-character data
255

85

22/08/2011

Functional testing techniques


Boundary Value testing Equivalence Class testing Special Value testing Decision Tables

256

Examples

257

The Triangle Program


In order for 3 integers a, b, and c to be the sides of a triangle, we must have c1 a + b > c c2 a + c > b c3 b + c > a A triangle is:
Equilateral if all 3 sides are equal Isosceles if 2 sides are equal Scalene if no two sides are equal
a

258

86

22/08/2011

The Triangle Program


The Triangle Program reads in 3 integers and decides if they form an equilateral triangle, an isosceles triangle, a scalene triangle, or if they dont form a triangle at all The logic of the program is clear, but complex, due to the relationships between the inputs and the outputs We assume that each side length is between 1 and 200
259

NextDate Function
This program reads in a date in a certain format and prints out the next days date. For example, an input of 03 31 1998 gives an output of 04 01 1998. The year is constrained to lie between 1812 and 2012 inclusive

260

The Commission Problem


A salesman sells locks, stocks and barrels On a monthly basis, the salesperson reports how many of each he or she has sold Due to manufacturing constraints, we have: 1 #locks 70 1 #stocks 80 1 #barrels 90 Goal: Determine total sales and the commission

261

87

22/08/2011

The Commission Problem


The parts sell for the following: Locks $45 each Stocks $30 each Barrels $25 each Commission is computed on sales: 10% on sales up to and including $1000; 15% on the next $800; and 20% on any amount over $1800 This program has calculation logic to test inputs and outputs are straightforward
262

Boundary Value testing

263

Boundary/Limits Testing
Test values, sizes or quantities near the design limits Value limits Length limits Volume limits Null strings vs Empty strings Errors tend to occur near the extreme values of inputs Robustness: How does the software react when boundaries are exceeded?
264

88

22/08/2011

Single Fault Assumption


The Single Fault assumption from reliability states: Failures are rarely the result of the simultaneous occurrence of two (or more) faults.

265

Extensions of Boundary Testing


Boundary value testing
Stay within the limits

Robustness testing
Exceed the limits

Worst-case testing
Like boundary value testing above, but discard Single Fault Assumption

Robust worst-case testing


Like robustness testing above, but discard Single Fault Assumption
266

Boundary Value Testing


The basic idea of boundary value analysis is to use input variable values at
their minimum, just above the minimum, at a nominal value, just below the maximum, and at the maximum

267

89

22/08/2011

Input Boundary Value Testing


a b x

x(min)

x(min+)

x(nom)

x(max -)

x(max)

Test cases for a variable x, where a x b Experience shows that errors occur more frequently for extreme values of a variable

268

Input Boundary Value Testing (2 Variables)


x2

c x1 a
Test cases for variables x1 and x2, where a x1 b and c x2 d

As in reliability theory, two variables rarely assume their extreme values simultaneously.

269

Robustness Testing
An extension of Boundary Value testing: Include values below the minimum value and above the maximum value This is a form of what we call negative testing -- how does the program handle errors in the input? This type of testing may not be possible with some strongly-typed languages, GUIs with fixed palette values or drop-down lists, etc.
270

90

22/08/2011

Robustness Testing
a b x

Test cases for a variable x, where a x b

1. Stress input boundaries 2. Acceptable response for invalid inputs? 3. Leads to exploratory testing (test hackers)
271

Robustness Testing (2 Variables)


x2

c x1 a b
272

Worst-Case Testing
Discard the Single Fault Assumption Allow the input values to simultaneously approach their boundaries

273

91

22/08/2011

Worst Case Testing (2 Variables)


x2

c x1 a b Eliminate the "single fault" assumption. Murphy's Law


274

Robust Worst-Case Testing


Discard the Single Fault Assumption Allow the input values to simultaneously approach and exceed their boundaries

275

Robust Worst Case (Paranoid) Testing with 2 Variables


x2

c x1 a b
276

92

22/08/2011

Limitations of Boundary Value Testing


Doesnt require much thought May miss internal boundaries Usually assumes the variables are independent Values at the boundary may not have any more meaning than non-boundary values

277

Boundary Value Test Cases for the

examples

278

Boundary Value Test Cases for the Triangle Program


Side lengths are integers ranging from 1 to 200 Typical boundary value analysis requires 5 values for each variable: min, min+, nominal value, max-, max Input Boundary Value testing for 3 variables requires 4x3 + 3 = 15 test cases. Our values: 1, 2, 100, 199, 200
279

93

22/08/2011

Input Boundary Value Analysis

280

Worst Case Boundary Value Analysis


There should be 53 = 125 test cases Side a = 1, 2, 100, 199, 200 times Side b = 1, 2, 100, 199, 200 times Side c = 1, 2, 100, 199, 200 = 125 test cases
281

Test Cases for the NextDate & Commission


Design boundary value test case for
NextDate function Commission problem

282

94

22/08/2011

Equivalence Classes Testing

283

Equivalence Classes
Idea
divide the input domain of a program into classes of data derive test cases based on these partitions

Determine the boundariesthen determine the equivalencies Motivation


Gain a sense of complete testing and avoid redundancy

If a given set of values all cause the program to behave in exactly the same way, then thats a candidate for an Equivalence Class
284

Equivalence Classes
An equivalence class represents a set of valid or invalid states for input condition An input condition is:
a specific numeric value, a range of values a set of related values, or a Boolean condition

Valid inputs

system invalid inputs partition outputs All Rights Reserved

285

95

22/08/2011

Determining equivalence classes


Equivalence classes can be defined using the following guidelines:
If an input condition specifies a range, one valid and two invalid equivalence class are defined If an input condition requires a specific value, one valid and two invalid equivalence classes are defined If an input condition specifies a member of a set, one valid and one invalid equivalence classes are defined If an input condition is Boolean, one valid and one invalid classes are defined
286

Examples
Area code
input condition, Boolean: the area code may or may not be present input condition, range: value defined between 200 and 900

Password
input condition, Boolean: a password may or may not be present input condition, value: six character string

287

Determining equivalence classes


Look for ranges of numbers or values Look for memberships in groups Some may be based on timing Include invalid inputs Include internal boundaries Dont worry if they overlap each other better to be redundant than to miss something However, these test cases will almost surely overlap with boundary value test cases
288

96

22/08/2011

Types of Equivalence Classes (two variables)


Weak normal
one data point from each valid class for each variable

Strong normal
one data point from each class in the cross product of the valid classes

Traditional Equivalence Class


For valid inputs, use one value from each valid class For invalid inputs, a test case will have one invalid value and the remaining values will all be valid

These usually apply when the variables have internal boundaries


289

Types of Equivalence Classes


Input domain Output range

290

Example for input domain


Suppose a program has 2 input variables, x and y Suppose x can lie in 3 possible domains: a x < b, b x < c, or c x d Suppose y can lie in 2 possible domains: e y < f or f y g

291

97

22/08/2011

Weak Equivalence Class Test Cases


y

g f

e d x
292

Strong Equivalence Class Test Cases


y

g f

e d x
293

Equivalence Classes Test Cases for the examples

294

98

22/08/2011

Triangle Program Input Equivalence Classes


Equivalence classes based on input: We get at least 9 classes (5 triangle, 3 not a triangle, 1 garbage): a=b=c * a b, a c, b c * a=bc * ab+c a=cb * b a+c b=ca * ca+b garbage: (0, 1,2), (-7, 4, 8), (x, &, #) * but still a triangle!
295

Output Equivalence Classes for the Triangle Program


There are 4 output equivalence classes:
Equilateral Isosceles Scalene Not a triangle

296

Equivalence Classes testing for NextDate


The classes for the Day variable could be {1 Day 28, Day = 29, Day = 30, Day = 31} The classes for the Month variable could be {Month has 30 days, Month has 31 days, Month = February} The classes for the Year variable could be {Year is not a leap year, Year is a leap year}

297

99

22/08/2011

Equivalence Classes testing for NextDate


Traditional Equivalence Class Test Cases
Valid inputs are mechanically selected from the approximate middle of the valid range

298

Equivalence Classes testing for NextDate


Weak Equivalence Class Test Cases
Inputs are mechanically selected from the approximate middle of the corresponding class

299

Equivalence Classes testing for NextDate


Strong Equivalence Class Test Cases
36 strong equivalence class test cases

300

100

22/08/2011

Equivalence Classes testing for NextDate


36 strong equivalence class test cases (cond)

301

Equivalence Classes testing for Commission Problem


Variable Input Domain Equivalence Classes

302

Equivalence Classes testing for Commission Problem


Weak Input Domain Equivalence Class Test Cases

303

101

22/08/2011

Equivalence Classes testing for Commission Problem


Traditional Input Domain Equivalence Class Test Cases

304

Equivalence Classes testing for Commission Problem


Traditional Input Domain Equivalence Class Test Cases

305

Equivalence Classes testing for Commission Problem


Output Range Equivalence Classes

306

102

22/08/2011

Notes on Equivalence Classes


Check for boundaries in the inputs or in the outputs Always include a garbage class If a variable can take on the value 0, it is usually best to have {0} as its own Equivalence Class Use your experience and domain knowledge to determine the best classes
307

Notes on Equivalence Classes


The traditional form of equivalence class testing is generally not as thorough as weak equivalence class testing, which in turn, is not as thorough as the strong form of equivalence class testing The only time it makes sense to use the traditional approach is when the implementation language is not strongly type If error conditions are a high priority, we could extend strong equivalence class testing to include invalid classes Equivalence class testing is appropriate when input data is defined in terms of ranges and sets of discrete values 308

Notes on Equivalence Classes


Equivalence class testing is strengthened by a hybrid approach with boundary value testing Equivalence class testing is indicated when the program function is complex Strong equivalence class testing makes a presumption that the variables are independent when the Cartesian Product is taken
If there are any dependencies, these will often generate "error" test cases, as they did in the NextDate function
The decision table technique in Chapter 7 resolves this problem
309

103

22/08/2011

Special Value Testing

310

Special Value Testing


One author calls this the most widely practiced form of functional testing The tester uses his or her domain knowledge, experience, or intuition to probe areas of probable errors Other terms: hacking, out-of-box testing, ad hoc testing, seat of the pants testing, guerilla testing
311

Uses of Special Value Testing


1.

2.

3. 4.

Complex mathematical (or algorithmic) calculations Worst Case situations (similar to robustness) Problematic situations from past experience Second guess the likely implementation

312

104

22/08/2011

Examples of Special Value Testing


When dates are involved, check Feb. 28, Feb. 29 and leap years When strings are involved, check the empty string and the single-character blank string Arithmetic: Always try 0 Any ad hoc hacking

313

Characteristics of Special Value Testing


1. 2. 3. 4. 5. 6.

Experience really helps Done by the customer or user frequently Defies measurement Highly intuitive Seldom repeatable Often, very effective

314

Decision Tables Based Testing

315

105

22/08/2011

Decision Tables
Inputs are interpreted as conditions Outputs are interpreted as actions The columns in a table are rules they show which actions result from which conditions Every rule then becomes a test case Dashes represent dont care conditions

316

Decision Table Terminology


Stub c1 Condition c2 c3 True T F Entry True False True T F False False

a1 a2 Action a3 a4

X X

X X X X X X

317

Rule

One Decision Table for the Triangle Program


c1: a, b, c are a triangle? c2: a = b? c3: a = c? c4: b = c? a1: not a triangle a2: Scalene a3: Isosceles a4: Equilateral a5: Impossible X X X X
318

F T T T F

T F T F T F F T F

T F X

X X X X

106

22/08/2011

Decision Table-Based Testing


Most rigorous functional method enforces logical rigor Used to represent and analyze complex logical relationships Decision Tables are declarative no particular order among the conditions is implied Decision Tables support consistency and completeness Dependencies can yield impossible combinations, so we usually have an impossible action
319

Refined Decision Table for the Triangle Problem


c1: a < b+c? c2: b < a+c? c3: c < a+b? c4: a = b? c5: a = c? c6: b = c? a1: Not a triangle a2: Scalene a3: Isosceles a4: Equilateral a5: Impossible X X X X
320

F T T T T T T T T T T -- F T T T T T T T T T -- -- F T T T T T T T T -- -- -- T T T T F F F F -- -- -- T T F F T T F F -- -- -- T F T F T F T F X X X X X X X

A Redundant Decision Table


Conditions 1-4 5 c1: T F c2: c3: a1: a2: a3: --X -X T T X X -6 F T F X X X 7 F F T -X X 8 F F F --X 9 T F F X -X

Rule 9 is identical to Rule 4 (T, F, F) Since the action entries for rules 4 and 9 are identical, there is no ambiguity, just redundancy.

321

107

22/08/2011

An Inconsistent Decision Table


Conditions 1-4 5 c1: T F c2: c3: a1: a2: a3: --X -X T T X X -6 F T F X X X 7 F F T -X X 8 F F F --X 9 T F F -X --

Rule 9 is identical to Rule 4 (T, F, F) Since the action entries for rules 4 and 9 are different there is ambiguity. This table is inconsistent, and the inconsistency implies non-determinism cant tell which rule to apply!
322

Procedure for Decision-Table Based Testing


1. Determine conditions and actions.

2. Develop the Decision Table, watching for completeness don't care entries redundant and inconsistent rules 3. Each rule defines a test case.
323

Decision Tables Based Testing Test cases for the examples

324

108

22/08/2011

Test Cases for the Triangle Problem

325

Exercises
Test cases for NextDate function Test cases for Commission problem

326

Test documentation

109

22/08/2011

Contents
Test Plan
The need for test plans The structure of test plans A Test Plan Template A Test Plan example Testing on a large project

Test Cases
Test Case Design Test Case Examples
328

Test plan

329

The need for a good test plan


Helps to organize, schedule and manage the testing effort Helps in writing test cases Improves communication with developers and with management If done correctly, it may help minimize the documentation required May be required by the customer
330

110

22/08/2011

Validation Planning Issues


Testing is intended to measure software quality and must be planned It takes careful planning to select a good set of tests Questions that must be answered: What to test and when to stop

331

Benefits of Test Planning


Measure the cost of testing vs the confidence gained Forces you to work out expected results before you run the test cases You can actually find errors before running any tests!

332

Components of a Test Plan


Testing approach Schedule Resources Entry and Exit criteria Requirements Matrix What will NOT be tested Test Cases
333

111

22/08/2011

Other Issues to Consider


Test methods Facilities for test execution Test automation Testing tools Support software Configuration management Risks (budget, schedule, etc.)
334

Test Plan activities


Use a Test Plan template, or design one List what cannot be tested Write only what you need

335

Test Plan activities


Have the Test Plan reviewed Test while you write Make it a living document

336

112

22/08/2011

Concerns of Test Planning


Not enough training Lack of test tools Lack of management support Lack of customer/user involvement Not enough time Rapid change Its hard to say No
337

Example of a Test Plan


Test Plan template Test Plan example

338

A Partial List for Your Test Plans


Title page of some sort Description of the testing to be done Updated Requirements Matrix List of test cases Traceability Matrix 3 detailed test cases

339

113

22/08/2011

Testing on a large project


Roles/Responsibilities
Project Mgr/Test Mgr/Development Test Team Lead/Lead Test Designer Testers Test Technicians Build Manager

340

Test Planning Tips


Meet regularly with the Project Manager Plan for the test team to attend a lot of reviews Make assignments early Assign one tester to know everything Have only one Test Plan Biggest headache: Builds
341

Test case

342

114

22/08/2011

Terminology
Test Case
Documentation specifying inputs, predicted results, and a set of execution conditions for a test item

Test Suite
a collection of test cases that are intended to be used to test a software program

343

Characteristics of good Test Cases


Reasonable probability of catching an error
Recall: Testing is the process of finding errors and of validating the program/system

Non-redundant Efficient Medium complexity Repeatable Specific Always list expected results
344

Test Case caveats


Errors can mask other errors Make sure you know how to tell whether a test case passes or fails Bug fixes can cause more bugs

345

115

22/08/2011

Where do you get your Test Cases?


Test Requirements Matrix Your experience Your learning The developer(s) Marketing The customer: Use Cases/Scenarios Defects from the field or previous releases
346

Test Case Examples


Test Case Template Test Case Example

347

To read more
Cem Kaner, What Is a Good Test Case?

348

116

22/08/2011

Test Case Generation


Case-insensitive Passwords State look-up Simple program

349

Case-insensitive Passwords
Write the test cases to verify that Passwords are case insensitive when logging onto a secure system

350

An example
Test Case: Verify that passwords are case insensitive Test Procedure:
1. 2. 3. 4. 5. 6.

Create a User with Password ABCDE. Log off the system. Log on using Password ABCDE (should be successful) Log off the system. Log on using Password abcDe (should be successful). Repeat Steps 4 and 5 using abcde, ABCDe, etc.
351

117

22/08/2011

State Lookup
Input: 2-character state abbreviation The program looks up the state in a table and displays the full name Discuss all considerations for testing such a function

352

Simple program
The program accepts as input 2 integers and outputs their sum Assume arbitrary limits, say -100 to +100 Design the test cases

353

Levels of Testing

118

22/08/2011

Contents
Levels of Testing Additional System Test Categories

355

Levels of Testing

356

Levels of Testing
Waterfall model
Requirements Specification System Testing

Preliminary Design

Integration Testing

Detailed Design

Unit Testing

Coding

357

119

22/08/2011

Levels of Testing
Unit testing Integration Testing System Testing Acceptance Testing Regression Testing

358

Unit testing
Lowest Level of Testing Individual unit of the software is tested in isolation from other parts of a program What can be a unit?
Function / method / module / class / component Screen / Program Back-end related to a Screen Screen + back end
359

Unit testing
Uncover errors in design and implementation, including
data structure in a unit program logic and program structure in a unit unit interface functions and operations of a unit

Unit testers
developers of the units
360

120

22/08/2011

Integration Testing
Intermediate level of testing Group of dependent components are tested together Test that evaluates the interaction and consistency of interacting components Uncover errors in:
Design and construction of software architecture Integrated functions or operations at sub-system level Interfaces and interactions between them Resource integration and/or environment integration

Integration testers
either developers and/or test engineers
361

System Testing
Test the whole system Test end to end functionality Testing against complete requirement Requirement focus
Test cases derived from specification

System testers
test engineers
362

Acceptance Testing
User (or customer) involved Environment as close to field use as possible Focus on:
Building confidence Compliance with defined acceptance criteria in the contract

363

121

22/08/2011

Regression Testing
Testing activities occur after software changes Regression testing usually refers to testing activities during software maintenance phase Retest changed components (or parts) Check the affected parts (or components) Who perform regression testing
Developers - regression testing at the unit level or integration Test engineers - regression testing at the function level QA and test engineers - regression testing at the system level
364

System Test categories


Functional testing Load/Stress/Volume testing Destructive testing/Recoverability Configuration Security Performance Compatibility Usability Reliability Installability Documentation
365

Stress Testing
Also called Load Testing or Volume Testing Test the extremes of real-world workloads Large data volumes Repetitive operations Operation for extended periods: Endurance Testing

366

122

22/08/2011

Destructive Testing
This is what we have called Error Testing The product should
Detect, diagnose and perhaps inhibit unexpected conditions Correct the conditions where possible Report on the unexpected conditions Recover gracefully in any case

367

Thinking Destructively
The most disastrous failure for most software is the loss or corruption of data Think like a hacker, especially when it comes to security issues Destructive Testing is usually unstructured, but document what you do Be creative! Think about the consequences if you succeed
368

Configuration Testing
Software components Hardware components Software with hardware

369

123

22/08/2011

Security Testing
Password usage Access rights Protection Encryption

370

Performance Testing
Usually pertains to hardware Software can be the bottleneck Generally, how fast a task can be done Can also be measured in volume

371

Compatibility Testing
Shared data Shared memory Compatibility with previous versions
operating system, utilities, setups, etc.

Portability might fall here

372

124

22/08/2011

Usability Testing
Human factors Accessibility Ease of use (product) Availability of documentation Can a non-familiar user perform a task?

373

Reliability Testing
Low if a bug is likely to be found High if bugs are unlikely Failures and Time
Time when the failure occurs Time interval between failures Cumulative failures up to a given time Number of failures in a time interval
374

User Manual Testing


Installation Documentation On-line Help

375

125

22/08/2011

Installability
Installing a product and its options De-installing (or Uninstalling) a product or its options Re-installing a product or its options Doing upgrade installs vs doing clean installs

376

Documentation
Reviews Inspections Walkthroughs

377

Documentation
Keep a copy of your comments This is a possible place for new testers to start Manuals can be task-oriented or featureoriented Tutorials are very helpful, so test them out, too

378

126

22/08/2011

Documentation
Installation instructions If task oriented, do the tasks! Other documentation (README, Utilities, etc.) Ease of use of the documentation Missing features Feature compatibility
379

Documentation
On-line Help
Availability Relevance Accuracy Completeness Links Controls

380

Test Reports

127

22/08/2011

Contents
Handling Defects Bug Tracking System Test Reports Examples

382

Handling Defects

383

Handling a defect
Analyze the defect Report the defect Track the defect Retest the defect Close the defect

384

128

22/08/2011

Analyzing a defect
Is it the test, the tester, or the product? Reproducible vs Repeatable Can it be isolated? Find related problems Look for alternate paths to the problem Is it worth reporting?

385

Handling a defect
Analyze the defect Report the defect Track the defect Retest the defect Close the defect

386

Reporting the defect


Look for duplicates Talk with the developer Enter it into the system Make sure it will get noticed!

387

129

22/08/2011

Characteristics of an effective defect report


Tells how to reproduce the problem
in a minimum number of steps

The report is complete and understandable Written Simple

388

Components of a defect report


Identifying information Description of the problem Various status indicators Comments Supporting information Miscellaneous information

389

Components of a defect report


Identifying information
Unique number Submitter Submit date Program or product this is against Version or revision of the product

390

130

22/08/2011

Components of a defect report


Description of the problem
Title
Must convey in a limited number of words enough information that the problem can be understood Any abbreviations must be generally acknowledged or explained in the Description

Description

391

Components of a defect report


Description of the problem
Contents
The problem itself
What did happen What should have happened

The test case used Any other helpful information whatsoever


In particular, any attachments

Keep it to 3 or 4 sentences if you can Avoid vague or confusing terms (frequently, sometimes) Avoid uncommon abbreviations Use any standard terminology Pay attention to spelling and grammar

392

Components of a defect report


Status indicators
Report status Severity and priority Resolution status

393

131

22/08/2011

Components of a defect report


Status indicators
Report Status

Open Closed

394

Components of a defect report


Status indicators
Severity and Priority

Severity: How bad is it?


Generally related to the defects effect on testing
High -- critical defect; test cannot continue Medium -- critical defect; test can continue with a workaround Low -- its a problem but we can keep testing Really low -- documentation, help files, spelling mistakes

Priority: How urgent is it?


When does this have to be fixed?
395

Components of a defect report


Status indicators
Examples of Resolution status

None In Process Fixed Ready for Test Retest Not repeatable Not a problem Do Not Fix Duplicate Multiple Deferred
396

132

22/08/2011

Components of a defect report


Comments/Notes
Investigation Notes Resolution Notes Tester Notes Modification Notes Whatever-works-for-you Notes

397

Components of a defect report


Miscellaneous fields (1)
Steps to Reproduce
Include setup information Anyone with the appropriate setup should be able to reproduce the problem Could be as simple as a Test Case ID, or a lengthy procedure Pay attention to problems that are merely repeatable or not reproducible

398

Components of a defect report


Miscellaneous fields (2)
Environment Target Release Closed Release Closed Date Owner Tester Discovered by Defect type Software component affected Fix hours Test hours Workarounds
399

133

22/08/2011

Components of a defect report


Supporting information
Error printouts Screen shots The test case itself A diskette with data or files Memory dump Trace files, error logs, etc.

400

Handling a defect
Analyze the defect Report the defect Track the defect Retest the defect Close the defect

401

Tracking a Defect
Must have a process in place Generally done by some sort of Review Board The lead tester should be on that board The purpose is to make sure the defect progresses from being submitted to getting closed

402

134

22/08/2011

Retesting a Defect
Retest can have 3 outcomes
The problem has indeed been fixed The problem remains A different problem shows up

403

Closing a Defect
Add Tester Notes or Verification Notes Make sure that anyone else with an interest in the problems resolution has a chance to test it too Close it, or recommend closure

404

Life cycle of a defect report


Entered by a tester Review Board evaluates and assigns information Assigned programmer fixes the problem A build containing the fix becomes available The tester retests the problem The tester or board closes the defect
405

135

22/08/2011

Bug Tracking System

406

The need for a Defect Tracking System


Facilitates clear communication about defects Allows developers to fix bugs based on priority Allows for analysis and reporting options Helps to manage the bugs Keeps a history

407

Items for your Defect Reports


Identifiers: title, number, submitter, submit date Program name and version Description of the problem The Test Case used

408

136

22/08/2011

Test Reports

409

Test Reports
Your chance to evaluate how testing went List what was tested List what was not tested, and why List still-open defects (some ask for all defects) Show the actual schedule (Test Plan) Lets management know the status of that product, from a testing standpoint, as it went out the door
410

Your Test Report


A brief description of what you did, when An attached list of all the Test Cases with an indication of which ones passed and which ones failed (and by default, then, which ones didnt get run) A list of all the defects found (a list of Brief Descriptions)

411

137

22/08/2011

Examples

412

Examples
Test Report template Test Report example

413

Test Automation and Tools

138

22/08/2011

Contents
Test Automation Testing Tools

415

Test Automation

416

What is Test Automation?


Test Automation
refers to the activities and efforts that intend to automate engineering tasks and operations in a software test process
using well-defined strategies and systematic solutions

assists software testers and software testing managers with different aspects of a testing project

417

139

22/08/2011

Objectives of Test Automation?


to free engineers from tedious and redundant manual testing operations to speed up a software testing process to reduce software testing cost and time during a software life cycle to increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule The major key to the success of software automation
to reduce manual testing activities and redundant test operations using a systematic solution to achieve a better testing coverage

418

Different Maturity Levels of Software Test Automation


Level 4: Optimal Systematic Test Measurement & Optimization Level 3: Automatic Systematic Test Generation Level 2: Repeatable Systematic Test Execution Control Level 1: Initial Systematic Test Information Management
419

Different Maturity Levels of Software Test Automation


Level 1: Initial
A software test process at this level provides engineers with systematic solutions and tools to create, update, and manage all types of software test information, including test requirements, test cases, test data, test procedures, test results, test scripts, and problem reports No systematic solutions and tools are available to support engineers test design, test generation, and test executions

Level 2: Repeatable
A software test process at this level not only provides engineers with tools to manage diverse software testing information, but also provides systematic solutions to execute software tests in a systematic manner. These solutions allow engineers to use a systematic approach to run tests and validate test results However, no systematic solutions and tools are available to assist test engineers in test design, test generation, and test coverage 420 measurement

140

22/08/2011

Different Maturity Levels of Software Test Automation


Level 3: Automatic
Besides the test management and test execution tools, a software test process at this level is supported with additional solutions to generate software tests using systematic methods They could be useful to generate black box or white-box software tests However, no systematic solutions are available to measure the test coverage of a test process

Level 4: Optimal
At this level, systematic solutions are available to manage test information, execute tests, and generate tests, and measure test coverage The primary benefit of achieving this level is to help engineers understand the current coverage of a test process, and identify the test coverage issues
421

Essential Needs of Test Automation


A dedicated work force for test automation The commitment from senior managers and engineers The dedicated budget and project schedule A well-defined plan and strategy Talent engineers and cost-effective testing tools Maintenance of automated software tests and tools
422

Types of Automation
Automated test case generation Automated test data generation Automated test execution Applications
Client-server architecture Networks GUI Regression Testing
423

141

22/08/2011

False expectations (1)


You can go from test planning to test execution with no manual intervention One tool can support all test requirements: Not so as the book points out, no tool offers assistance with all tester activities There will be an immediate reduction in test effort and test schedule Automation will reduce testing resources
424

False expectations (2)


Test automation tools are easy to use Everything can be tested Automation compensates for bad requirements or bad design

425

Benefits of Automation
Reduce manual software testing operations and eliminate redundant testing efforts Produce more systematic repeatable software tests, and generate more consistent testing results Execute much more software tests and achieve a better testing coverage in a very limited schedule.

426

142

22/08/2011

Is Automation right for me?


You must have a Development Process in place You should have a Testing Process in place - mostly manual You have many tests that get executed many times Consider the costs Consider the time it takes to implement
427

Some Costs of Automation


The cost of the tool itself, including licensing Any additional computer equipment or test environment enhancements Test tool installation The time it takes to learn the tool Training costs

428

The Automated Test Lifecycle Management


1. 2. 3. 4. 5. 6.

Decision to automate testing Test tool acquisition Automated testing introduction process Test planning, design and development Execution and management of tests Test program review and assessment

429

143

22/08/2011

Testing Tools

430

Selecting a Testing Tool


Ease of use Customizable Platform support (Hardware and OS) Multi-user Defect tracking Functionality Reporting capability Performance and stress testing Version control Pricing Vendor qualifications
431

Types of tools
Requirements Management
Generate (or capture) requirements quickly and efficiently, and manage them Basic notification maybe even determine which test procedures are affected when a requirement changes Examples
TBIs Caliber-RM, QSSs DOORS, Rationals RequisitePro, Integrated Chipwares RTM Workshop
432

144

22/08/2011

Types of tools
Configuration Management
Controls software code, documentation, hardware, firmware, and system environment configuration Examples
CVS, Subversion, Visual SourceSafe http://www.daveeaton.com/scm/CMTools.html

433

Types of tools
Test Planning and Test Case Management
Create and manage your test projects Import test cases from other file formats Tracks history and status of test cases Examples
TestLog, Salome-TMF, IBM Rational Quality Manager http://www.opensourcetesting.org/testmgt.php
434

Types of tools
Test Data Generators
Generate test data from specifications / design / code Examples DataGen, Tnsgen, forSQL Data Generator

GUI testing
Record and Playback a test Rerun repetitive test cases Regression testing Examples
Segue SilkTest, Compuware QA Run, QuickTestPro
435

145

22/08/2011

Types of tools
Load/Performance/Stress Testing Tools
Simulate thousands of users
Specially client-server systems

Examples
Empirix Eload, XML Load Testing, Segue SilkPerformers, IBM Rational Suite

436

Types of tools
Code Coverage
Determining a quantitative measure of code coverage
Statement, Decision, Condition, Path

Finding areas of a program not exercised by a set of test cases Examples


http://csharp-source.net/open-source/code-coverage http://java-source.net/open-source/code-coverage
437

Types of tools
Rule-Based Analyzers
Compare the code written with coding standards or languages-specific rules Support Review / Inspection Examples
Intelli J, FlexeLint for C++, SSW Code Auditor

438

146

22/08/2011

Types of tools
Defect Tracking
Communicate and mange defect-related issues Examples
BugTracker, TrackGear, Bugtrack http://www.testingfaqs.org/t-track.html

439

Types of tools
Debuggers
Allow inspection of program state Pause execution at selected points (breakpoints) when certain conditions occur (watchpoints) after a fixed number of execution steps Provide display and control at the level of program source code

Examples
DDD, Debuggy, Valgrind
440

147