IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL.

8, NO 2, FEBRUARY 1990 315

Software Costs of Quality

WILLIAM A. MANDEVILLE, MEMBER, IEEE

Abstracf-Our hardware manufacturing cousins have taken advan- and the costs of noncompliance. The costs of compliance
tage of the costs of quality for manufacturing by using it as a lever to
are the costs which are spent to ensure that software is
get needed prevention resources such as new tools, capital, and train-
ing to improve their productivity, quality, and to strategically reduce developed and maintained in accordance with its require-
costs. It is time for the software development community to recognize ments. These costs include the costs to prevent defects
this powerful quality improvement tool and to take full advantage of from occurring in the first place (prevention costs) and the
it. costs o f finding defects (appraisal). A list of typical pre-
vention costs for software follows:
INTRODUCTION the cost of training
the cost of tools
T HERE are very few costs to reduce in the software
world. We cannot reduce inventories or the costs of
materials, and we have little influence over the cost o f
the cost of methods
the cost of standards, policies, and procedures
the cost of consulting
platforms, and labor costs continue to rise. The significant
the cost of quality planning
savings lie in the costs of quality and our ability to reduce
the cost of team prevention meetings
them (by reducing defects). The biggest cost reductions
the cost of fast prototyping
can be gotten by determining what we are spending now
the cost of quality data gathering, analysis, and use
for the costs associated with defects, finding and fixing
the cost of root cause analysis.
the root causes of these defects, and constantly measuring
As you can, see the costs of prevention are the costs to
and monitoring to determine new cost of quality oppor-
prevent defects from occurring (The costs of putting your-
tunities to reduce. Everyone agrees with this philosophy
self into a position of doing things right the first time.)
(preventing defects reduces costs); the problem is how to
The other cost of compliance is appraisal cost. These
identify the problems, get management’s attention, and
costs are associated with the review of the product to en-
develop the projected return on investments so that pre-
sure that it meets requirements. Appraisal costs are usu-
vention resources can be applied to the roots of the prob-
ally associated with the defect removal process. They are
lem.
insurance costs and include the following activities:
This paper defines the costs of quality for software, pre-
the review of requirement specifications
sents a methodology for the collection of costs, and shows
the review of design specifications
you how to use the costs of quality to point the way to-
the review of component and module specifica-
ward quality improvement and significant cost reductions.
tions
There is a big difference between quality control or as-
the review, walk through, or inspection of code
surance and what this paper decribes as quality improve-
the testing process, including unit test, product
ment. The quality function is usually given the charter to
test, system test, and independent verification and vali-
provide assurance that for the company’s product meets
dation test
the requirements of its users.This is different from quality
Beta test, field trial, or parallel test
improvement. Quality improvement is the deliberate,
all product audits (process audits could be consid-
planned approach to preventing defects from occurring.
ered prevention because they are a method to find poten-
Quality improvement responsibility can reside with the
tial problems and remove them, before they cause de-
Quality Organization or it can be part of the development
fects).
organization’s tools or methods group. This paper con-
The costs of noncompliance are failure costs. Included
centrates on using the costs of quality as part of the im-
are the costs to fix errors in software. Failure costs also
provement process.
include the cost of any rework. Rework can be any re-
What are the Costs of Quality for Sofirware? view, audit, inspection, or test which is conducted more
than once. The first review is considered appraisal cost,
Quality costs are categorized into the costs of confor-
any re-review (rework) is considered a failure cost (cost
mance to specifications (or requirements, if you prefer)
of noncompliance). A partial list of failure costs follows:
the cost to fix an error
Manuscript received May 4, 1989; revised September 25, 1989. the cost to re-review a document
The author is with the Carman Group, P.O. Box 867689, Plano, TX
75086. the cost to retest a procedure, case, or script
IEEE Log Number 8932 100. the cost to re-review a module of code

0733-8716/90/0200-0315$01.OO 0 1990 IEEE

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.
 316 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 8. NO. 2. FEBRUARY 1990
0 the cost of computer usage for tests (except the
first)
the cost of lab allocations for other than the first
tests
all expenses associated with fixing problems with
released systems
the cost of expediting fixes (FAX, overnight mail,
E-Mail, overtime, etc.).
Failure costs are further categorized into the costs of
internal failure and the costs of external failure. The in-
ternal failures are those failures which occur under the
developer’s control (on the company’s premise). These
failures occur in requirements specifications, designs,
code, and the early phases of test. External failures are
those which fail in front of customers or users and are
embarrassing. These usually cause reputation damage and
can cause bad will and loss of sales due to poor quality.
Why Measure the Costs of Quality?
Management (which is responsible for allocating the re-
sources) understands dollars, return on investment, and
profit and loss (P & L). No matter how we educate and
train them, they have difficulty understanding the rela-
tionship of defects per thousand lines of code (or on a
function point base) to real dollar investments in preven-
tion activities. For better or worse, the investment deci-
sion makers were brought up in a business school envi-
ronment. This level of management, however, will
understand that you are spending up to 50% of your op-
erating expenses finding, fixing, and explaining away
software defects. You have great a opportunity to show
them the enormous costs of quality and what can be done
to gain improvements.
To get funding for improvement projects (prevention),
it is essential to show the current cost of defects, the cost
of the solution, and the resulting return on investment
(ROI). The only way to continually gain prevention re-
sources, and thus attack the problems at the source, is to
show the problem cost in dollars, the solution cost in dol-
lars, and ROI. The providers of prevention resources un-
derstand ROI.
COST COLLECTION METHODOLOGY
Where Costs of Quality Must be Measured
The costs of quality must be measured at an activity
level. Activities are defined as a group of tasks relating to
a milestone event. System test would be an activity. Unit
test would be an activity. So would coding, requirements
review, and design review.
Errors occur in each activity. They may either be caused
by the activity itself or may be the result of an error from
a previous activity. If the error is found and fixed in the
same activity, say coding, it is much less expensive than
if it originated in function design and did not manifest
itself until system test. This is because the cost of cor-
recting a code error in the coding activity is a matter of
changing the code, recompiling, and other administrative
activities such as reconfiguration and possible re-review,
Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.
while the cost of fixing a design error found in system test
includes the cost of correcting the design documents,
changing the code, and retesting (unit, functional, and
system).
Determining which activity is contributing the most to
quality costs is important in root cause analysis. It will
give you a leg up on directing prevention resources to the
responsible activity. These can be termed liability costs,
since the originator of the error is responsible for the en-
tire cost to fix the error.
What Should be Measured?
It is important to measure the relationship among the
four costs of quality and the relationship to some base that
top level managers understand. The importance of mea-
suring the costs to each other is that it will dramatically
show how much is spent on failure costs (fixing defects)
and how dramatically little is spent on prevention costs.
Appraisal costs should also be shown so that this relation-
ship can be analyzed. Fig. 1 presents a typical cost of
quality chart.
Notice that as prevention resources are added, total cost
of quality reduces. This is why quality is considered free.
The relationship of the costs should also be shown in ways
that management can understand and clearly see that the
total cost of quality is reducing. This can be accomplished
by measuring the costs of quality to a known base (oper-
ating expenses or budget). Fig. 2 is a typical chart of the
total costs of quality to budget.
Notice that there is cost improvement in the relation-
ship. If there is improvement in the relationship of cost to
budget, then the money that would have been spent on
failures now drops through to the bottom line of the In-
come statement.
A Costing Method
There are two methods for collecting the costs of qual-
ity: through a defect collection and costing system and
through the accounting system. The accounting system
requires that a labor distribution system be in place and
that the time spent on reviews, tests, and audits be coded
as appraisal, that the time spent on training, methods, and
tools be recorded as prevention, and that the time spent
on fixing defects be recorded as failure costs. The advan-
tage to this method is that it is an Accounting Department
function and the numbers are usually not suspect if the
right audits and controls are in place. There are also sev-
eral disadvantages.
1) The Accounting Department has rigid rules and they
sometimes do not lend themselves to incorporating a cost
of quality system or changes of any kind.
2) The Development Department is not in control of
the costs of quality, the Accounting Department is.
Needed changes are sometimes difficult to get because
they are not the Accounting Department’s first priority.
3) It is difficult for developers to accurately record time
spent on different activities.
4) The costs of quality can be shown, but the trail to
MANDEVILLE: SOFTWARE COSTS OF QUALITY
140
120
100
80
60
40
20
0
I 1989
MONTHLY TOTAL EKITRNAL INTERNAL
IAPPRAISAL 0 PREVENTION
Fig. I . Total cost of quality. Summary by cost type for project.
I 1989 I 1990
+ MONTHLY TOTAL * DLTERNAL 8 INTERNAL
-K- APPRAISAL + PREVENTION
Fig. 2 . Percent of quality cost to budget.
which defects are causing the largest costs is dead ended.
This is an important disadvantage. You need a trail to the
defects so that root cause analysis can be performed.
A defect collection and costing system uses actual de-
fects and their causes, the labor rates of the personnel fix-
ing the defects, and the average times to fix defects. These
defect costs are used in the calculations for internal and
external failure costs. The number of reviews, tests, and
inspections are used for appraisal costs.
Again, there are both advantages and disadvantages to
a defect costing system. The advantages are that the de-
velopment activity is in control of its cost of quality sys-
tem, the rules of accounting are relaxed, and the system
will have the capability to point the way toward the most
costly defects. The disadvantage is that the numbers are
likely to be less accurate than an accounting system. But,
the reason you are using a cost of quality system is to
strategically reduce the development costs by getting pre-
vention resources. Accounting accuracy finishes in sec-
ond place behind root cause analysis and quality improve-
ment.
The costs of quality using defects and events (such as
reviews and tests) is not well publicized and deserves ex-
ploration. To start with, a defect collection system can be
a good system for root cause analysis and cost reduction.
By investigating the top ten defects occurring during an
activity and then performing root cause analysis on the
top offenders, the roots of the problems can be deter-
mined. However, to get proper prevention funding, you
must know the cost of the problem, the cost of the solu-
tion, and the return. For example, it would be difficult to
obtain an investment of $200 000 in prevention resources
if the costs of the problem were not known. And it would
be unwise to invest the funds if the return was less than
adequate.
A cost of quality system for software using defects and
events as a base looks like the diagram in Fig. 3.
317
MONTHLY
INPUT 4
TPUT
SET UP 4 CALCULATIONS
Fig. 3. System layout.
The set-up database contains the rates of personnel
working on particular tasks such as functional design re-
views or fixing bugs in a requirements document. These
can be established once and reviewed periodically. The
data entry database is monthly input, where the number
of reviews and tests and the resultant defects are input.
I
The data in the set-up database and the data entry database
must be kept at an activity level (i.e., code, function de-
sign, system test, etc.) so that the costs can be traced to
the activities. This aids in root cause analysis, by getting
to the activity with the highest cost. On a monthly basis,
the calculations should be performed. Monthly is pre-
ferred to quarterly because it is more timely or weekly
because the numbers will have more impact. Besides,
managers are used to looking at costs and their impact on
a monthly basis.
Once the costs of failure are determined (failure cost is
always the largest), we need to look at the defects which
are occurring the most frequently Fig. 4 shows the top ten
defects by type.
This chart is easily obtained from a defect tracking sys-
tem. However, if we are interested in strategic cost re-
ductions, we must look beyond the frequency of defects
and look at the overall cost of the defects. There is a re-
lationship to the frequency of defects and their costs,
however, costs have a stronger relationship to where a
defect was found in the life cycle, versus where it was
inserted. A review of Fig. 5 shows this.
Notice that defect DO0056 on the defect-by-type chart
(Fig. 4) is the most frequent, while defect DO0051 (Fig.
5) is the most costly. To strategically reduce costs, the
most costly defects should be tackled first. This defect
strategy is different than trying to get a release out. In that
case, the most severe defects must be fixed first. But, fix-
ing software errors to get a release out does not improve
the quality of the software development process. It does
affect the quality of the particular release, but since there
is no root cause analysis and corrective action, the same
errors will appear in release after release. The root causes
of defects are rarely paid any attention when you are un-
der release pressure.
The improvement in quality and the resultant reduction
in costs of quality come only when we determine the root
causes of defects and take the appropriate corrective ac-
tion to eliminate the defect, forever.
Using a cost of quality system not only points us toward
the most costly defects, but it will also provide liability
costing. Liability costs are those costs assigned to the
 318
Fig. 4. Total defects by type. Top ten, July 1989
Fig. 5 . Total cost by defect. Top ten, July 1989.
originators of the bugs in software. For example, the cause
of a bug inserted by the systems requirements function,
and found in test, is long forgotten when we are trying to
make a release schedule. However, if the costs to fix the
problem are assigned to the systems requirements func-
tion, a whole new list of management concerns and ques-
tions are likely to be raised. Such as, “How did this prob-
lem arise?” “How did it get so far without being
noticed?” and “What are we going to do about prevent-
ing it from recurring?” These are excellent questions and
the type that quality professionals and designers should
ask themselves. But without the added impact of showing
the defects in costs, the requirements and design errors
are long forgotten. Fig. 6 is a typical liability chart show-
ing the costs of defects assigned to the functions respon-
sible for them.
This chart brings the right emphasis on problem solv-
ing. It also provides the first step in root cause analysis
and corrective action. We try to solve most problems by
getting faster test methods (automated testers, faster test
platforms, etc.). This is needed, but the most costly de-
fects are caused by the early life cycle activities (testers
do not cause defects, designers do).
Root Cause Anulysis and Corrective Action Using
Sofrwure Costs of Quality
Once you have determined a starting place for your root
cause analysis (the most costly defects), you need to de-
termine the cause. After you have determined the root
cause, you need to analyze the solutions. The solutions
then need to be costed. With the cost of the problem in
hand and the cost of the solution, you need to present the
return on investment. This is not a trivial task and should
not be taken lightly. There are many activities competing
for investment funds. You must explain the costs of qual-
ity, what poor quality is costing the company, the costs
of the solution. and the return that can be provided.
Monitoring the cost of quality improvements is a con-
tinual task. It is the only way to ensure that your quality
Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.
IEEE J O U R N A L ON S E L E C T E D A R E A S IN C O M M U N I C A T I O N S . VOL. 8. N O 2. F E B R U A R Y 1990
~~
~
TOTAL COST BY DEFECT
(Top Ten) March/1990
Unique Project Number Goes Here. Proj No.
improvement process is on an even keel and headed in the
right direction. Opportunities for improvement are always
there, and the cost improvements to the bottom line fol-
low along hand in glove with the quality improvement
process.
The three most important things in improving quality
are Management Commitment, Management Commit-
ment, and Management Commitment. The way to get and
maintain management commitment is to provide them a
solid return on investment, just as they must provide a
return to their management and stockholders. Cost of
quality systems show you what you are spending on de-
fects, what the most expensive defects are, and where they
originate.
CONCLUSION
The time has come for software quality professionals to
show their contributions to the company’s well being. We
have always known that our activities have positively
contributed, now we can show it. Establishing a software
cost of quality system is an easy, cost-effective method to
start you down the road of long-term cost reductions and
real quality improvement.
William A. Mandeville (M‘88) received the B . S .
degree from the University of New Haven. New
Haven, CT. and the M.S. degree from the Uni-
versity of Southern California.
He is a nationally recognized expert in the ficld
of software quality assurance and software quality
costing. His experience spans 20 years in software
engineering and quality management with major
companies including General Dynamics. ITT.
ITEC. and Ericsson. He has prehentcd keveral in-
novative papers and has led national discussion
panels o n new methodologies in the area of commercial and military soft-
ware quality costing and defect collection and reporting. He is currently
the Vice President of Development with the Carman Group. Dallas. T X .
The Carman Group provides software quality aswrance tools. training. and
consulting services.