196 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 8. NO. 2.
FEBRUARY 1990
Statistical Quality Control Applied to Code
Inspections
DENNIS A. CHRISTENSON, STEEL T. HUANG,
-i
Abstract-Code inspections have been used on the SESS@ Switch
project since 1983. Beginning with a training program for all the de-
velopers involved in the project, code inspections have improved with
each new SESS@Switch generic. The improvement in code inspections
has been the result of hard work and innovation on the part of the
S E S P Switch software developers, and the use of some Statistical
Quality Control (SQC) techniques.
Variations on a standard SQC technique, the control chart, have
been used to track the metrics indicative of the effectiveness of code
inspections. Parameters used in the computation of these metrics in-
clude the preparation effort, inspection time, number of inspectors, the
size of the inspected unit of code, and the number of errors found at
the inspection. The exact form that these control charts have taken
has evolved and improved with experience.
INTRODUCTION
C ONTROL charts have been used in industry as a
means of statistically evaluating the quality of var-
ious manufacturing processes. First, the capability of a
process and the normal statistical variations of the process
are determined. From this characterization of the process,
abnormal deviations of the process and trends away from
the norm can be detected and some corrective action
taken.
CODEINSPECTION FACTORS
In 1986, code inspection data, collected from more than
900 code inspections over several releases of SESS Switch
software, were analyzed. The data reported on the in-
spection form included the number of noncommentary
source lines (ncsl) of code inspected, the total time spent
by the inspectors preparing for the inspection meeting, the
length of the meeting, and the number of errors found.
Several factors [2], [3] were seen in the inspections which
correlated well with the density of errors discovered: 1)
the amount of preparation effort, 2) the rate at which the
code was inspected, and 3) the size of the unit of code.
Preparation Effort: Prior to the actual code inspec-
tion meeting, the inspectors prepare for the inspection by
studying the code and the design documents. Preparation
Manuscript received May 15, 1989; revised September 12. 1989. This
work was presented as a paper entitled Code inspection management using
statistical control limits. by D. A. Christenson and S. T. Huang, at the
National Communications Forum, Rosemont. IL, September 28-30, 1987,
and as A code inspection model for software quality management and
prediction, at the IEEE Global Telecommunications Conference & Ex-
hibition, Hollywood, FL, November 28-December I , 1988.
The authors are with AT&T Bell Laboratories, Naperville. IL 60566.
IEEE Log Number 8932092.
0733-8716/90/0200-0196$01 .OO 0 1990 IEEE
AND ALFRED J. LAMPEREZ
effort is defined as the total inspector preparation time di-
vided by the number of thousand noncommentary source
lines (kncsl) in the unit of code. Preparation effort was
found to have high positive correlation with the density
of errors found.
Inspection Rare: During the actual inspection meet-
ing, the inspectors read the code and look for errors. The
inspection rate is defined as the number of noncommen-
tary source lines (ncsl) inspected divided by the total du-
ration of the inspection meeting(s). The rate of inspection
is not a completely independent variable. It partially de-
pends upon the rate at which errors are being found, which
in turn depends upon the amount of preparation and the
density of errors present in the code. The density of dis-
covered errors was found to have an inverse relationship
to the inspection rate.
Code Size: The size of the code being inspected was
found to be the major factor influencing the preparation
effort and the inspection rate. Larger units of code tended
to receive proportionately less preparation and were in-
spected at a higher rate. This resulted in a lower density
of discovered errors for larger units of code. The small to
moderate sized units of code had high preparation effort
relative to the code size, and also tended to have a low
rate of inspection during the meeting. This resulted in a
generally higher density of discovered errors for these
units of code.
INSPECTION
LIMITS
It was evident that inspections with low preparation ef-
fort and high inspection rate yielded a low density of dis-
covered errors. We needed to emphasize the importance
of these factors to the development community. We de-
fined a recommended preparation effort level, and rec-
ommended inspection rate consistent with that in [ I ] . A
minimum preparation effort level was defined as half of
the recommended level, and a maximum inspection rate
was defined as twice the recommended rate. Three classes
of inspection were then defined in terms of these levels.
Effective inspections would be those that met the
recommended preparation effort and inspection rate.
Questionable inspections would be those that did
not meet the minimum preparation effort andlor exceeded
the maximum inspection rate.
Marginal inspections would be those that were
between Effective and Questionable.
CHRISTENSON er a l . : STATISTICAL QC APPLIED TO CODE INSPECTIONS 197

Marginal inspections might have preparation effort

above the recommended, but a rate between the recom-
mended and the maximum, for example.
. .. .. . . . .. . . .
... ..... . .. . ... . . . .
ERRORLIMITS
:.................... ..... .......__
:
; ...e<
. . . . . .=. .
We characterized the process of making errors in code
as a random process, and demonstrated the variance of the
error density to be inversely proportional to the size of the
FOUND
ERROR
DENSITY
..
.
: ..

.
_... , ......,. .. ........ .
io....
. .........
...
.... . . . . . . . . . .
.. . . .
........
0

........._
.._
.-._.........
...... ...... ........................
.............
... . . . . . . . . . ...........................................
.
0.
......... ..................
-.: ......................................
+6
+5

unit of code [4].This was borne out by the variation in

.. ................................ *......................
...... ....... ............................ +,:
+4
+3

.......
+2
error densities resulting from effective inspections,
~-_-,,___,____~~- .f.. .?.
shown in Fig. 1. Superimposed on the plot is the mean
and several standard deviation units from the mean, based
.:..,.-,. .....-... ...........
...... -.................. ...................a
. ........................................ -1.
.
on the Poisson distribution. The error density distribu-
.
:
-
......... ...i .........:....
........ . . . . . ......................
0
.......... -4.
. . . . . . .......... . . . ........................... .J-
.. . . . . ..._..
4.

tions for past inspections, in each of the three inspection ... . .. .._.
classes, were characterized in terms of this simple Pois- + ,ir ., . . . .
_..-. ,.............................................. ..__.
,. ....... ,...... , ....,...-6.
.

son error model. Upper and lower limits on the density of
errors were selected based on the distribution from ef-
fective inspections. Use of an upper limit would point
out code with a much greater than average error rate.
Using a lower limit would point out marginal inspec-
tions that may not have been effective. Such upper and
lower error limits, as a function of code size, are shown
in Fig. 2.
CODEINSPECTION IMPROVEMENT
After completing the above analysis, the inspection
process was changed in several ways. The recommended
preparation effort and inspection rate were adopted by the
project as goals. Also, a minimum preparation effort level
was adopted as entry criteria for inspections. This was set
at half of the recommended level. Also adopted was a
maximum inspection rate as exit criteria from inspections.
This was set at twice the recommended rate. These con-
trol limits and recommendations for preparation effort, in-
spection rate, and error density became an integral part of
the code inspection process.
A new set of code inspection forms was created with
accompanying instructions that included the recommen-
dations and limits. A new Meeting Notice form allowed
the inspection moderator to schedule a number of coor-
dinated meetings for larger units of code. A new Code
Inspection Report form, which summarized all the rele-
vant parameters of the inspection, highlighted the prepa-
ration effort, inspection rate, and total number of errors
found relative to the code size. This form incorporated a
control chart for errors similar to that shown in Fig. 2.
Project-wide averages for the critical inspection parame-
ters were reported at project status meetings, and were
thus brought to the attention of the project managers along
with other important project information.
These changes to the process encouraged each manager
to be aware of the preparation time being taken by the
inspectors, of the rate at which the code was being in-
spected, and how the resulting error density compared to
the expected range for that code size. It also allowed the
manager to exercise some judgment about the adequacy
of the inspection, and decide on various courses of action
based on the inspection results.
CODE SIZE
Fig. 1. Error density versus code size for effective inspections.
NUMBER OF
ERRORS
CODE SIZE
(LOG-LOG PLOT)
Fig. 2 . Control chart for number of errors found relative to code size.
IMPROVEMENT RESULTS
The new set of code inspection forms and accompany-
ing instructions were employed on the next software re-
lease. Considerable attention was given to the inspection
results by all levels of management on the project. After
all inspections had been held, an analysis was made of the
resulting data. The preparation effofl, inspection rate, and
error density from the inspections using the new inspec-
tion limits were compared to the set of inspections held
prior to the use of the limits. The results showed that there
was a significant improvement in the level of preparation
effort, and also noticeable slowing of the inspection rate,
for larger units of code. The increased preparation and
slower inspection resulted in a twofold increase in the
density of the errors found for larger units of code.
AN INSPECTION MODEL
Although improvement had been made in the inspection
process, the question remained whether further improve-
ment could be made. Having a model which would relate
198 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS. VOL. 8. NO. 2. FEBRUARY 1990
10
REMAINING CODlN
FOUND
ERROR
DENSITY
(RELATIVE)
INSPECTION EFFORT INTENSITY
Fig. 3. New prediction chart based on inspection model
input parameters, such as the preparation effort and in-
spection rate, to the resulting fraction of errors found in
the code would be very useful. We could then estimate
the effectiveness of inspections. In fact, we could esti-
mate the density of coding errors remaining in the code
after an inspection was completed. This estimate would
be one of a number of factors used to estimate the quality
of the software.
In 1988, using data obtained from the past software re-
leases, plus the newer data from the improved inspection
process, we observed a very definite relationship between
the average effort per thousand noncommentary source
lines (kncsl) of code and the average effectiveness of in-
spections. A model was developed that related the frac-
tion of initial errors found to the total hours of inspection
effort per kncsl, or Inspection Effort Intensity [5]. From
this model, a chart was devised as shown in Fig. 3. Given
the inspection effort per kncsl, and the density of coding
errors found, an estimate for the remaining coding error
density could be made from the chart. For example, the
point shown in Fig. 3 represents 5 errors per kncsl found
by the inspection at a particular effort intensity. The hor-
izontal sloped curves represent the initial coding error
density, in this case the density was 6 errors per kncsl.
The vertical sloped curves represent the remaining error
density, in this case 1 error per kncsl. Based on the model
estimate, and other intuitional factors, one could judge
whether the code should be reinspected.
NEW INSPECTION
GUIDELINES
One of the original questions we sought to answer
throughout the analysis of inspection data was: how
much inspection effort is worthwhile? The model pro-
vided a means to estimate this, given we had some esti-
mate of the cost to find and fix an error in later phases of
development. Below is a simple equation for the total cost
of coding errors in a software release, as a function of
inspection effort intensity:
INITIAL CODING
ERROR DENSIN
(RElA TIVE):
where C, is the total cost of coding errors in dollars per
ncsl, Ch is the time cost of the people involved, h is the
average code inspection effort intensity in person hours
per ncsl, w is the average number of person hours per
error needed to rework inspection errors, eo is the initial
coding error density, A is the average cost of an error
missed by the inspection and found during later phases,
and f ( h ) is the model estimate of the fraction of errors
found in the inspection, as a function of the person hours
of inspection effort per kncsl. Fig. 4 shows a plot of this
equation for total cost of coding errors, versus inspection
effort per kncsl, for several values of initial coding error
density. Estimated cost parameters were used. This figure
shows that more inspection effort is justified when there
is a higher density of errors in the code. Conversely, if
few errors are expected, little or no effort is justified. This
relationship is highly dependent upon the average cost to
find and fix an error later in the software life cycle. More
inspection effort is justified if the later cost is high.
FURTHER
IMPROVEMENT
The project adopted the new prediction chart based on
the inspection model (Fig. 3), replacing the older control
chart (Fig. 2) on the code inspection forms. A target for
density of errors remaining was also selected. Inspections
having a predicted density of remaining errors above this
target would be candidates for reinspection. Concep-
tually, this was a big improvement. Whereas with the
older control chart, a moderate amount of inspection ef-
fort finding a moderate density of errors was acceptable,
the new chart might predict a density of remaining errors
higher than the target, suggesting a reinspection of the
code. With the old chart, a moderate effort finding a very
low density of errors implied that the code was not effec-
tively inspected. Now with the new chart it might imply
that the code was very clean to begin with and few errors
remained. The new chart allowed for improvement in the
coding process, i.e., fewer errors being made, where the
old chart was tied to the historical error density distribu-
CHRISTENSON er al.: STATISTICAL QC APPLIED TO CODE INSPECTIONS
AVERAGE
COST OF
CODING ERRORS
PER KNCSL
AVERAGE INSPECTION EFFORT PER KNCSL
Fig. 4. Cost of errors versus inspection effort for various initial error den-
sities
tion. It was now possible to have very clean code go
through an inspection with moderate effort, and be fairly
confident that the code was of high quality.
Referring to Fig 4,point D shows the estimated cost
of coding errors for the previous software release. The
cost equation implied that, given the expected density of
errors, and the estimated cost levels, doubling the inspec-
tion effort would be cost effective. The project accepted
the challenge, and new inspection guidelines and limits
were incorporated into the inspection entry and exit cri-
teria.
Following these changes, the inspection data were
tracked closely. Fig. 5 illustrates the gradual increase in
both inspection effort intensity and found error density
during the subsequent four months. These are compared
to the levels for the previous four software releases, points
A through D.As the figure shows, the upward trend
followed the model closely.
CHARTREFINEMENT
Continuous process improvement pointed toward fur-
ther refining the predictive capabilities of the inspection
model. Project managers as well as the development com-
munity voiced concern regarding the accuracy of the point
estimates of remaining coding error density which were
given for individual code inspections. The original in-
spection model showed relatively good accuracy where
the average of a large number of inspections was in-
volved, such as all the inspections comprising particular
features of a software release. However, predictions for
individual inspections lacked consistent accuracy when
compared to results of reinspections or results of unit test-
ing the code.
Further analyses of the code inspection models param-
eters and its corresponding predictions with reinspection
results and unit test data provided for the development of
a refined chart which would give the probability that an
individual inspection had fewer than a certain, targeted
199
0
6
FOUND
ERROR 4
DENSITY
(RELATIVE)
2
0 I I I I I I
INSPECTION EFFORT INTENSITY
(HOURS PER KNCSL)
Fig. 5 . Improvement in inspection effectiveness after new guidelines.
density of remaining errors. These criteria would be eval-
uated to help determine the need for reinspection of a par-
ticular code module.
By successfully modifying the models parameters, a
much more accurate prediction of the number of remain-
ing errors, to be found by unit testing, would be available.
The objective was to characterize the distribution of the
ratio of the models predicted remaining error density to
the actual unit testing error density. The natural logarithm
transformation of this quantity can be characterized by a
probability distribution that is approximately normal, in-
dicating that the ratio itself is roughly log-normal. This is
depicted in Fig. 6. Using the mean and standard deviation
derived from this distribution, probability values could be
easily assigned to the point estimates corresponding to the
inspection model curves to yield the likelihood that a given
inspection had a remaining error density (unit test error
density) at or below the selected target. This revised chart,
which gives the probability that the density of remaining
errors exceeds a threshold T, is shown in Fig. 7. Using
 r
200 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS. VOL 8. NO 2. FEBRUARY 1990

0.20 I I I I 121 F.O. Buck, Indicators of quality inspections. 1BM Tech. Rep.
TR21.802. Sys. Commun. Div., Kingston, NY, 1981.
131 A. F. Ackerman. P. J. Fowler. and R. G. Ebenau, Software inspec-
tions and the industrial production of software. in Softnure Vulidu-
tion, H. L. Hausen, Ed. New York: Elsevier Science North-Holland
0.15
1984.
[4] D. A. Christenson, and S . T. Huang, Code inspection management
> using statistical control limits. in Proc. N a t . Com!nun. F<JrUnl- 1987.
V vol. 41, no. 11, p. 1095-1100.
Z [SI -, A code inspection model for software quality management and
W 0.10
r prediction. in Proc. IEEE Global Telecotnmun. Conf. - 1988, vol. 1 ,
i$
W
pp. 468-472.
[6] M. E. Fagan, Advances in software inspections. IEEE Trcins. Sofi-
ware Eng., vol. SE-12. July 1986.
K
LL 0.05

0.0 Dennis A. Christenson received the B.S.E.E. de-

gree in 1965 from Iowa State University. the
0.01 0.05 0.5 1 5 10 50
M.S.E.E. degree in 1966 from the University of
Illinois, and the M.B.A. degree in 1973 from the
R A T I O OF P R E D I C T E D T O ACTUAL University of Chicago.
He joined AT&T Bell Laboratories in 1965.
(Log Scale) and his subsequent experience has been related to
Fig. 6. Distribution of the ratio of predicted remaining errors to actual various aspects of software development for
errors found in unit testing. switching systems. Since 1983 he has worked in
the area of Software Metrics and Software Relia-
bility in conjunction with the development of soft-
ware for the 5ESS@ Switching System. He is a member of Technical Staff
in the Quality Engineering Group of the Switching Systems Performance
Division at AT&T Bell Laboratories in Naperville. IL.

Steel T. Huang received the B.S. degree in math-

FOUND ematics in 1969 from the National Taiwan Uni-
ERROR versity and the Ph.D. degree in statistics in 1975
from the University of North Carolina at Chapel
DENSITY Hill.
He was a Faculty member at the University of
Cincinnati and University of North Carolina from
1975 to 1979. Hejoined Bell Laboratories in 1979.
He is responsible for the planning, definition. and
analysis of software quality and productivity met-
rics for the 5ESS Switch development project.
He is Supervisor of tl. Quality Engineering Group of the Switching Sys-
tems Performance Di, ;ion at AT&T Bell Laboratories in Naperville, IL.

INSPECTION EFFORT INTENSITY

Fig. 7. Revised inspection chart showing probability of remaining error
density exceeding a threshold.
Alfred J. Lamperea received the B.S. degree in
statistics from the University of Southwestern
this new chart, rather than estimating the exact density of Louisiana.
He joined AT&T in 1988 and has been respon-
remaining errors, software developers can assess the risk sible for applying statistical analysis and model-
of having too many undetected errors. ing as well as quality control techniques t o data
and metrics from various stages of the software
REFERENCES development cycle. He is a member of Technical
Staff in the Quality Engineering Group of the
i 8
[ I ] M. E. Fagan, Design and code inspections to reduce errors in pro- Switching Systems Perforniance Division at
gram development. IBM Sysr. J . . vol. 15, no. 3, 1976. AT&T Bell Laboratories in Naperville, 1L.