1

Chapter 7.2

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 2 Chapter 7.2
Integrating quality activities in
the project life cycle

Continuing:

•Factors affecting intensity of SQA activities

•Verification, validation and qualification

•Development and quality plans for small and for internal

projects

•A model for SQA defect removal effectiveness and cost

Galin, SQA from theory to implementation © Pearson Education Limited 2004
 3
Factors affecting Intensity of SQA Activities
• SQA Activities are linked to the completion of a project
phase
– Requirements, design, etc.

• The SQA activities need to be integrated into the

development plan that implements one or more software
development models, such as the waterfall, prototyping,
spiral, …

• They need to be activities just like other more traditional

activities – be entered in plan, scheduled, etc.
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 4

Factors affecting Intensity of SQA Activities

• SQA planners need to determine
– A list of SQA activities needed for the project
– And then for each activity, they need to decide on
• Timing
• Type of QA activity to be applied (there are several)
• Who performs the activity and resources required.
– Important to note that many participate in SQA activities
» Development team
» Department staff members
» Independent bodies
• Resources required for the removal of defects and
introduction of changes.

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 5

Factors affecting Intensity of SQA Activities

• Sad testimony that few want to allocate the
necessary time for SQA activities.
– This means time for SQA activities and then
time for subsequent removal of defects.
– Often, there is no time for follow-on work!!
• Activities are not simply cranked in and
absorbed!
• So, time for SQA activities and defect
correction actions needs to be examined.
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 Factors
6 Affecting the Required Intensity of QA Activity

• Project Factors
– Magnitude of the project – how big is it?
– Technical complexity and difficulty Discuss
– Extent of reusable software components – a real factor
– Severity of failure outcomes if project fails – essential!
• Team Factors
– Professional qualifications of the team members
– Team acquaintance w/ project and experience in the area
– Availability of staff members who can professionally support
the team, and
– Percentage of new staff members in the team.

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 7 So, what are the activities??
• Recognize that these activities take days to undertake, and days
to correct. Here are some sample activities:
• (and don’t forget the planning for the reviews, materials, …)
• Design Review of Requirements
• Design Review of xxxxx
• Design Review of ….
• Inspections of ….
• Inspections of …..
• Unit Testing of software code for each interface module …
• Unit testing of ….
• System Testing of ….
• Design Review of User’s Manual…..
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 8
Verification, Validation, and Qualification
Three aspects of Quality Assurance for Software Products:

Verification – The process of evaluating a system or component to

determine whether the products of a given development phase
satisfy the conditions imposed at the start of that phase

Validation - The process of evaluating a system or component

during or at the end of the development process to determine
whether it satisfies specified requirements

Qualification - The process used to determine whether a system or

component is suitable for operational use
IEEE Std 610.12-1990 (IEEE 1990)
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 More on VV and Q
9

• Verification looks at the consistency of the products being developed with

products developed in previous phases.
• Developers verify as we go.

• Validation is a customer thing – customers validate the outputs, etc. Validate

against their original requirements.
• Necessary for customer satisfaction.

• Qualification focuses on operational aspects – maintenance is the main issue.

• Has development taken place IAW professional standards and procedures such
that follow-on maintenance is easier to undertake?

• Planners need to determine which of all these need to be examined in each QA

activity.

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 10

Model for SQA defect removal

effectiveness and cost
for defect detection and removal

The model addresses two quantitative aspects of the SQA

planning addressing several defect detection activities.

a.Want to study the SQA plan’s total effectiveness in

removing project defects, and

b.The total costs of removal of project defects

Note again that SQA activities must be integrated within the

project’s development plan.
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 Model for Defect Removal
11

The data:

Model based on three types of data:

Defect origin distribution

in which phase did the defects occur

Defect removal effectiveness, and

how effective are we at removal of defects?

Cost of defect removal.

how much does it cost per defect per phase!!!

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 12

Defect Origin Distribution

• Very consistent over many years:
• Distribution of Defects:
– Requirements Specs 15%
– Design 35%
– Coding / integration 40%
– Documentation 10%

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 13

Defect Removal Effectiveness

• Generally speaking, the percentage of removed defects is lower
than the percentage of detected defects, because some
corrections are ineffective or inadequate.
• We simply miss some!!
• Others are undetected and uncorrected and passed on to
successive development phases.

• Lingering defects coupled with introduced defects in current

development phase add up!!!
• For discussion purposes, we will assume the filtering
effectiveness of accumulated defects of each quality assurance
activity is not less than 40%, that is, each activity removes at least
40% of the incoming defects.
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 14
Defect Removal Effectiveness
• Removal effectiveness
–QA Activity Average defect filtering
effectiveness rate
requirements specs review50%
design inspection 60%
design review 50%
code inspections 65%
unit test 50%
Unit test > code review 30%
integration test 50%
system tests / acceptance 50%
documentation review 50%

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 15
Cost Removal
• Removal of defects differs very
significantly by development phase.
• Cost are MUCH greater in later
development phases.

• Note: In general, defect removal data is not

commonly available.
• Most agree with the data based on key
studies. (next slide)
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 16

Defects removal effectiveness

for quality assurance plans

Defect removal phase Defect Average relative defect removal cost

removal {cost unit}
effectiveness Defect origination phase
Req Des Uni Int Doc
Requirement specification (Req) 50% 1 --- --- --- ---
Design (Des) 50% 2.5 1 --- --- ---
Unit coding (Uni) 50% 6.5 2.6 1 --- ---
Integration (Int) 50% 16 6.4 2.5 1
System documentation (Doc) 50% 6.4 2.5 1
16
System testing / Acceptance 50% 40 16 6.2 2.5 1
testing (Sys)
Opertion by customer (after 100% 110 44 17 6.9 2.5
release)
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 17 The Model
Our Model is based on following assumptions:
•Development process is linear, sequential following waterfall model
•There is a number of new defects introduced each phase
•Review and test software quality assurance activities serve as filters,
removing a percentage of defects and letting the rest pass to next
development phase as we saw three slides back.
•At each phase, incoming defects are the sum of those not removed plus
new defects in current phase
•Cost of defect removal is calculated for each SQA activity by multiplying
the number of defects removed by the relative cost of removing a defect.
(see table, previous slide)
•Remaining defects are passed to the customer. (this is the heaviest cost for
defect removal)

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 18

Model - parameters

• The models we show use these parameters:

– POD – Phase originated defects
– PD – Passed defects (from former phases)
– %FE - % of filtering effectiveness
– RD – Removed defects
– CDR – Cost of defect removal
– TRC – Total Removal Costs

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 19 First Model (standard)
• This model applies a standard quality assurance plan
(standard defects filtering system) composed of QA
activities, that is, ‘filters’ we have:
QA activity removal effectiveness Cost of removing defect

– Requirements Spec review 50% 1

– Design Review 50% 2.5
– Unit Test 50% 6.5
– Integration tests 50% 16
– Documentation Review 50% 16
– System Test 50% 40
– Operation Phase 100% 110
Galin, SQA from theory to implementation © Pearson Education Limited 2004
 20

• Using the standard quality assurance plan’s

quality assurance activities on previous
slide, a process-oriented illustration of the
standard QA plan model follows:

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 21

POD – phase originated defects

PD – passed defects
%FE – percent filtering
RD – removed defects
CDR - Cost of removing defects
TRC - Total Cost of Defect Removal

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 22 Costs of Defect Removal
• But we can do better using a comprehensive quality assurance
plan with more activities, and hence better filtering.

• The comprehensive quality assurance plan (comprehensive

defects filtering system) accomplishes the following:

• 1. Adds two quality assurance activities so that the two are

performed in the design phase as well as in the coding phase
– We have a Design Inspection and a Design Review vice Design, and
– Code Inspections and unit test vice simple Unit Test
• 2. Improves the ‘filtering’ effectiveness of other quality
assurance activities.

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 23

12.3 and 4.9

Design Inspect and
Design Review

10.5 and 6.3 Code inspection

precedes unit test

Comprehensive Quality
6.5
Assurance Plan:

POD – phase originated defects 6.6

PD – passed defects
%FE – percent filtering
RD – removed defects 2.6
CDR - Cost of removing defects
TRC - Total Cost of Defect Removal

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 24
Comprehensive Plan
• Conclusions:
• The standard plan successfully removes 57.6% of the defects in
requirements and design compared to 90.2% in the comprehensive
plan before coding begins.
– Results from more intensive defect-removal efforts.
• The comprehensive plan as a whole is much more economical than the
standard plan as it saves 41% of the total resources investing in defect
removal, compared with the standard plan
• Compared to the standard plan, the comprehensive plan makes a greater
contribution to customer satisfaction by drastically reducing the rate
of defects detected during regular operations: 6.9% to 2.6%

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 25

Conclusion
• So, in general, the quantitative results of the
comparison comply nicely with the SQA
approach.

• Additional investments in QA activities yield

substantial savings in defect removal costs.

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 26

Homework
• Individually, you are to answer the
following questions and send them to me via
Blackboard Assignment.
• Questions: 7.6 and 7.7

Galin, SQA from theory to implementation © Pearson Education Limited 2004

 27

Discussion Forum
• Team 3: Here are your primary questions:
• These are the ones for you to provide for
class discussion with your team being the
primary participants
• Questions: 7.2, 7.5, 7.4

Galin, SQA from theory to implementation © Pearson Education Limited 2004