CSTE Skill Category 4

Test Planning
•Risks
•Prerequisites to Test Planning
•Create the Test Plan

Presented by:
David Thompson
QA Engineer II, Albertsons Inc.

1
 Risk Concepts & Vocabulary
• Test Case – Test cases are how the testers validate that
a software function meets the… Software Specifications
• Test Data – Information used to build a… test case

• Test Script – An Online entry of… test cases

• Risk – The potential loss to an organization

• Threat – Something Capable of exploiting vulnerability

• Vulnerability – A design, implementation, or operations flaw
that may be exploited by a threat
• Control – Anything that causes a reduction of risk
Page 224-225 2
 Risks Associated with S/W development
Risk #1 Prevalent Phase
• Improper Use of • Design
Technology

Ways to Mitigate
• Interviewing Users (to ensure need)
• Prototyping

Page 227 3
 Risks Associated with S/W development
Risk #2 Prevalent Phase
• Repetition of Errors • Code

Ways to Mitigate
• White Box Testing

Page 227 4
 Risks Associated with S/W development
Risk #3 Prevalent Phase
• Cascading of Errors • Release / Maintenance

Ways to Mitigate
• Regression Testing
• System Testing

Page 228 5
 Risks Associated with S/W development
Risk #4 Prevalent Phase
• Illogical Processing • Design

Ways to Mitigate
• Data Validation (Bounds √ing, Field
verification)
• Code Review

Page 228 6
 Risks Associated with S/W development
Risk #5 Prevalent Phase
• Inability to Translate User • Requirements
Needs into Technical • Design
Requirements

Ways to Mitigate
• Prototyping
• Interviewing Users

Page 228 7
 Risks Associated with S/W development
Risk #6 Prevalent Phase
• Inability to Control • Entire SDLC
Technology

Ways to Mitigate
• Structured Configuration Management

Page 229 8
 Risks Associated with S/W development
Risk #7 Prevalent Phase
• Incorrect Entry of Data • Code
• Release / Maintenance

Ways to Mitigate
• Bounds Testing
• White Box Testing
• Data Validation Testing

Page 229 9
 Risks Associated with S/W development
Risk #8 Prevalent Phase
• Concentration of Data • Requirements
• Design

Ways to Mitigate
• Security Testing
• Data Transmission Testing
• Input / Output Testing
• Data Normalization » 3rd Normal Form

Page 230 10
 Risks Associated with S/W development
Risk #9 Prevalent Phase
• Inability to React Quickly • Requirements
• Design
• Code

Ways to Mitigate
• Performance / Load Testing
• Build Management / CM

Page 231 11
 Risks Associated with S/W development
Risk #10 Prevalent Phase
• Inability to Substantiate • Design
Processing

Ways to Mitigate
• Security Testing
• Security Logs
• Transaction Logs
• White Box Testing
• Back up Scheme
• CM
Page 231 12
 Risks Associated with S/W development
Risk #11 Prevalent Phase
• Concentration of • Design
Responsibilities

Ways to Mitigate
• Security Testing
• (Security) Policies & Procedures

Page 232 13
 Risks Associated with S/W development
Risk #12 Prevalent Phase
• Erroneous or Falsified • Design
Input Data • Maintenance

Ways to Mitigate
• Data Validation
• I/O Testing
• Black Box Testing
• Security Testing
• Processes and Procedures

Page 232 14
 Risks Associated with S/W development
Risk #13 Prevalent Phase
• Misuse by Authorized End • Design
User • Maintenance / Release

Ways to Mitigate
• Audit Trails
• Security Testing
• Processes and Procedures

Page 233 15
 Risks Associated with S/W development
Risk #14 Prevalent Phase
• Uncontrolled System • Design
Access • Maintenance

Ways to Mitigate
• Securing Equipment and Access
• Processes and Procedures

Page 234 16
 Risks Associated with S/W development
Risk #15 Prevalent Phase
• Ineffective Security and • Design
Privacy Practices for the • Maintenance / Release
Application

Ways to Mitigate
• Securing Equipment and Access
• Processes and Procedures
• Security Testing
• Security Logs Created
• Automatic Emails Generated

Page 234 17
 Risks Associated with S/W development
Risk #16 Prevalent Phase
• Procedural Errors During • Release
Operation • Maintenance

Ways to Mitigate
• Audits
• Procedures and Policies
• Configuration Management
• Checklists
• CPI

Page 235 18
 Risks Associated with S/W development
Risk #17 Prevalent Phase
• Program Errors • Design
• Code
• Maintenance
Ways to Mitigate
• White Box Testing
• Document Reviews
• Audits
• Code Reviews
• Security Testing

Page 236 19
 Risks Associated with S/W development
Risk #18 Prevalent Phase
• Operating System Flaws • Design

Ways to Mitigate
• Code Reviews
• Security Testing
• Load / Performance Testing

Page 237 20
 Risks Associated with S/W development
Risk #18 Prevalent Phase
• Communication System • Design
Failure • Code

Ways to Mitigate
• Data Transmission Testing
• Document Reviews
• Physical Security

Page 238 21
 Risks Associated with S/W testing
When conducting risk analysis, two major components are taken into
consideration:
• The probability that the negative event will occur
• The potential loss or impact associated with the event
Some Primary Testing
• Risks:
Not Enough Training • Lack of Test Competency
• Us versus Them • Lack of Test Tools
• Lack of Mgmt Understanding • Lack of Customer / User
• Involvement
Not enough schedule or
Budget • Over Reliance on Independent
• Testers
Rapid Change
• Testers in a Lose-Lose Situation
• Having to say “no”
• Test Environment
• New Technology
• New Developmental Process
Page 238-239 22
 Risk Analysis
Testing is a process designed to minimize software risks. To make
software testing most effective, it is important to assure all the high
risks associated with the software, will be tested first.

Risk Analysis Process

1. Form the Risk Analysis Team
a) Knowledge of the Application
b) Understanding of Risk Concepts
2. Identify Risks
a) Risk Checklist
b) Risk Analysis
3. Estimate the Magnitude of the Risk
a) Intuition and Judgment
b) Risk Formula
4. Select Testing Priorities
a) Risks Ranked by Severity
Page 241-246 23
 Risk Management
Risk management is a totality of activities that are used to minimize
both the frequency and the impact associated with risks. After
determining risks; need to determine risk ‘appetite’ (the amount of the
loss) management is willing to accept for a given risk.
Risk Reduction Method
1. Risk Frequency X Loss/Occurrence = Total Loss
2. Apply Controls to Minimize Risk
a) Reduce the Opportunity for Error » Minimize Loss
b) Identify Error prior to Loss » Recover Loss
3. If the Controls Cost less than the Estimated Loss there is
a Good Case to Implement the Controls.

Contingency Planning
1. Action Plans should be established for activation when a
loss is known to occur for a given risk. The testers should
evaluate the adequacy of the contingency plans
Page 246-248 24
 Prerequisites to Test Planning

1. Test Objectives – assures the project directives are met

2. Acceptance Criteria – have the user define this criteria
3. Determine Assumptions – have them clearly documented
4. People Issues – Watch out if the s/w engineering Director is also
the head of QA.
5. Constraints – Obvious constraints are test staff size, test budget,
and test schedule.

Page 249-250 25
 Create the Test Plan
The test plan describes how testing will be accomplished. Its
creation is essential to effective testing, and should take about 1/3 of
the total test effort. If the plan is developed carefully, test execution,
analysis, and reporting will flow smoothly.

A Test Plan should…

1. Provide background information on the s/w being tested, test
objectives, risks, and specific tests to be performed
2. Have tests that are: Repeatable, controllable, and ensure
adequate coverage

“The act of designing tests is one of the most

effective error prevention mechanisms known..”

Page 251 26
 How to Write a Good Test Plan…
Understand the Characteristics of the Software
being developed:
1. Define what it means to meet the project objectives.
2. Understand the core business areas and processes.
3. Assess the severity of potential failures.
4. Identify the components for the system.
5. Assure requirements are testable.
6. Address implementation schedule issues.
7. Address interface and data exchange issues.
8. Evaluate contingency plans for system and activities.
9. Identify vulnerable parts of the system and processes operating
outside the information resource management area.

Page 252 27
 How to Write a Good Test Plan…
Build the Test Plan:
1. Set Test Objectives
a) Referenced by a number
b) Write as a measurable statement
c) Assign a priority
d) Define acceptance criteria
2. Develop Test Matrix
a) Define tests as required
b) Define conceptual test cases to be entered as a test script
c) Define verification tests
d) Prepare software test matrix
3. Define Test Administration
a) Identifies schedule, milestones, and resources needed
b) Cannot be completed until the test matrix is completed

Page 253 - 258 28

 How to Write a Good Test Plan…
Develop the Test Matrix:
1. Define Tests as Required
a) Referenced by a name and number
b) Contains: Objective, Test Inputs, Test Procedure, Acceptance
Criteria, Test Controls – when to stop the test, & Reference to what
is tested
2. Define Conceptual Test Cases to be Entered as Test Scripts
a) Use Case type tests.
3. Define Verification Tests
a) Static test performed on a document developed by the team
responsible for creating software.
4. Prepare the Software Test Matrix
a) A requirements traceability matrix.

Page 254 - 257 29

 How to Write a Good Test Plan…
Write the Test Plan:
1. Start Early
2. Keep the Test Plan Flexible
3. Review the Test Plan Frequently
4. Keep the Test Plan Concise and Readable
5. Calculate the Planning Effort
6. Spend the necessary time to complete the Test Plan

Page 261 30
 How to Write a Good Test Plan…
Test Plan Standard
There is no one universally accepted standard for test planning.
There are many Test Plan Standards available on the Internet from
various organizations, for example: MilStd 498, IEEE829, IEEE/EIA
12207. Most Test Plan Standards consist of the following sections.
1. Test Scope 1. Scope
2. Test Objectives 2. Referenced Documents
3. Assumptions 3. Software Test Environment
4. Risk Analysis 4. Test Identification
5. Test Design 5. Test Schedules
6. Roles & Responsibilities 6. Requirements Traceability
7. Test Schedule & Resources 7. Notes
8. Test Data Management 8. Appendices
9. Test Environment
10. Communication Approach
11. Test Tools
Page 262-263 31