Testing Computer Software 2nd Edition (1999)

Cem Kaner Jack Falk Hung Quoc Nguyen

Testing Computer Software

Objectives and Limits of Testing
Realistic test planning is dominated by the need to select a few test cases from a huge set of possibilities. You cannot fully/completely test a program The prime benefit of testing is improved quality

Testing Computer Software

Software Errors
User Interface Errors Error Handling (one of the least tested areas) Boundary-Related Errors Calculation Errors Initial and Later Stages (happens only the first time) Control Flow Errors (menus to submenus) Errors in Handling and Interpreting Data (between modules) Race Conditions

Testing Computer Software

Software Errors (continued)
Load Conditions Hardware Source and Version Control Documentation (manual, book) Testing Errors (code, procedures, user error)

Testing Computer Software

Reporting and Analyzing Bugs
The point is to get bugs fixed A good report is written, simple, understandable, reproducible, legible, and non-judgmental

The Bug Tracking System

The system introduces project accountability Users of the system (Testers, Project Manager, Programmer, Product Manager, Technical Support, Writer, Test Manager, Senior Executives, Lawyers)

Testing Computer Software

The Bug Tracking System (continued)
The authors strongly discourage providing personal performance data Generate many useful reports (weekly status, deferred, patches)

Testing Computer Software

Test Types and Test Case Design
Pre-testing stage
Requirements review Design review

Coding stage
Whitebox/Glassbox Testing

Testing stage
Blackbox Testing
Functional Testing System Testing Load Testing Configuration Testing Test only one device per group to eliminate the errors common to that group.

User Manuals

Testing Computer Software

Test Types and Test Case Design (continued)
Equivalence Classes (same result from two different tests)
Functional Equivalence Testing (automation, random input)

Boundary Analysis Testing state transitions (ex. Menus) Race Conditions

Testing Computer Software

Test Types and Test Case Design (continued)
Regression (bug fixes, library of tests)

Testing Tools
Basic Tools
File compare Utility, File Viewers

Automated Regression Tests Standards compliance-checking programs Tools for code coverage, assertion checks, memory

Testing Computer Software

Test Planning and Documentation
Test Plan as a product and Test Plan as tool Goal of Test Plans and Docs
Facilitate technical tasks of testing (study program, more complete coverage) Describe testing tasks and processes (size, depth of testing job) Organize testing project (improve individual accountability, schedule)

Testing Computer Software

Test Planning and Documentation (continued)
White box vs Black box
They are complementary. They find different types of bugs.

Suggest evolutionary method for developing test plan

Start small and build and add new sections as you go along. Avoid huge test plan at the start of testing.

Testing Computer Software

Test Planning and Documentation (continued)
Test Plan (defined by IEEE Standard 829) Acceptance Test Criteria Test Design spec Test Summary Report (complete a cycle of testing) Some Minor docs
Test case spec and Test procedure spec Test item transmittal report Test script Test log

Testing Computer Software

Software Development Model implications for testing
Waterfall
You must review the UI early Start writing test plan as early as possible (risk assess) You cannot start testing until late in the project When you start testing, you are on the critical path

Testing Computer Software

Software Development Model implications for testing
Evolutionary
Start reliability testing as soon as first level of functionality plan to staff early Plan waves of usability tests as project grows more complex Write test plan as you go Do your most powerful testing as soon as possible because project might be stopped and shipped at any time

Testing Computer Software

Managing a testing group
Management is the real quality control group in any company Provide predictability to the project manager Different Types of Testing groups
QC enforces standards QA senior people involves at all stages Testing services find bugs Development Services debug, initial tech support