Beruflich Dokumente
Kultur Dokumente
This is the seventh volume in the series 'Handbook of Statistics' started by the
late Professor P. R. Krishnaiah to provide comprehensive reference books in
different areas of statistical theory and applications. Each volume is devoted to
a particular topic in statistics; the present one is on 'Quality Control and Relia-
bility', a modern branch of statistics dealing with the complex problems in the
production of goods and services, maintenance and repair, and management and
operations. The accent is on quality and reliability in all these aspects.
The leading chapter in the volume is written by W. Edwards Deming, a pioneer
in statistical quality control, who spearheaded the quality control movement in
Japan and helped the country in its rapid industrial development during the post
war period. He gives a 14-point program for the management to keep a country
in the ascending path of industrial development.
Two main areas of concern in practice are the reliability of the hardware and
of the process control software. The estimation of hardware reliability and its uses
is discussed under a variety of models for reliability by R.A. Johnson in
Chapter 3, M. Mazumdar in Chapter 4, L. F. Pan in Chapter 15, H. L. Harter
in Chapter 22, A. P. Basu in Chapter 23, and S. Iyengar and G. Patwardhan in
Chapter 24. The estimation of software reliability is considered by F. B. Bastani
and C. V. Ramamoorthy in Chapter 2 and T. A. Mazzuchi and N. D. Singpur-
walla in Chapter 5.
The main concepts and theory of reliability are discussed in Chapters 10, 12,
13, 14 and 21 by F. Proschan in collaboration with P. J. Boland, F. Guess, R. E.
Barlow, G. Mimmack, E. E1-Neweihi and J. Sethuraman.
Chapter 6 by N. R. Chaganty and K. Joag-dev, Chapter 7 by B. W. Woodruff
and A. H. Moore, Chapter 9 by S. S. Gupta and S. Panchapakesan, Chapter 11
by M . C . Bhattacharjee and Chapter 16 by W . J . Padgett deal with some
statistical inference problems arising in reliability theory.
Several aspects of quality control of manufactured goods are discussed in
Chapter 17 by F. B. Alt and N. D. Smith, in Chapter 18 by B. Hoadley, in
Chapter 20 by M. CsOrg6 and L. Horv6th and in Chapter 19 by P. R. Krishnaiah
and B. Q. Miao.
All the chapters are written by outstanding scholars in their fields of expertise
and I wish to thank all of them for their excellent contributions. Special thanks
are due to Elsevier Science Publishers B.V. (North-Holland) for their patience and
cooperation in bringing out this volume.
C. R. Rao
Contributors
W. Edwards Deming
The decline of Western industry, which began in 1968 and 1969, a victim of
competition, has reached little by little a stage that can only be characterized as
a crisis. The decline is caused by Western style of management, and it will
continue until the cause is corrected. In fact, the decline may be ready for a nose
dive. Some companies will die a natural death, victims of Charles Darwin's
inexhorable law of survival of the fittest. In others, there will be awakening and
conversion of management.
What happened? American industry knew nothing but expansion from 1950 till
around 1968. American goods had the market. Then, one by one, many American
companies awakened to the reality of competition from Japan.
Little by little, one by one, the manufacture of parts and materials moves out
of the Western world into Japan, Korea, Taiwan, and now Brazil, for reasons of
quality and price. More business is carded on now between the U. S. and the
Pacific Basin than across the Atlantic Ocean.
A sudden crisis like Pearl Harbor brings everybody out in full force, ready for
action, even if they have no idea what to do. But a crisis that creeps in catches
its victims asleep.
* Parts of this Chapter are extracts from the author's book Out of the Crisis (Center for Advanced
Engineering Study, Massachusetts Institute of Technology, 1985).
2 w. Edwards Deming
for management of figures that are unknown and unknowable. The legal depart-
ment fights off creditors and predators that are on the lookout for an attractive
takeover. Unfortunately, management by the comptroller and the legal department
only brings further decline.
Anyone could add more inhibitors. One, for example, is the choking of business
by laws and regulations; also by legislation brought on by groups of people with
special interests, the effect of which is too often to nullify the work of standard-
izing committees of industry, government, and consumers.
Still another force is the system of detailed budgets which leave a division
manager no leeway. In contrast, the manager in Japan is not bothered by detail.
He has complete freedom except for one item; he can not transfer to other uses
his expenditure for education and training.
6. Other obstacles
The comptroller runs the company on visible figures. This is a sure road to
decline. Why? Because the most important figures for management are not visible:
they are unknown and unknowable. Do courses in finance teach students the
importance of the unknown and unknowable loss
- from a dissatisfied customer?
- from a dissatisfied employee, one that, because of correctible faults of the
system, can not take pride in his work?
- from the annual rating on performance, the so-called merit rating?
- loss from absenteeism (purely a function of supervision)?
Do courses in finance teach their students about the increase in productivity
that comes from people that can take pride in their work?
Unfortunately, the answer is no.
There is now a theory of management. No one can say now that there is
nothing about management to teach. If experience by itself would teach manage-
ment how to improve, then why are we in this predicament? Everyone doing his
best is not the answer that will halt the decline. It is necessary that everyone know
what to do; then for everyone to do his best.
The 14 points apply anywhere, to small organizations as well as to large ones,
to the service industry as well as to manufacturing.
(1) Create constancy of purpose toward improvement of product and service,
with the aim to excel in quality of product and service, to stay in business, and
to provide jobs.
6 IV. Edwards Deming
(2) Adopt the new philosophy. We are in a new economic age, created by
Japan. Transformation of Western style of management is necessary to halt the
continued decline of industry.
(3) Cease dependence on inspection to achieve quality. Eliminate the need for
inspection on a mass basis by building quality into the product in the first place.
(4) End the practice of awarding business on the basis of price tag. Purchasing
must be combined with design of product, manufacturing, and sales, to work with
the chosen supplier, the aim being to minimizing total cost, not initial cost.
(5) Improve constantly and forever every activity in the company, to improve
quality and productivity, and thus constantly decrease costs. Improve design of
product.
(6) Institute training on the job, including management.
(7) Institute supervision. The aim of supervision should be to help people and
machines and gadgets to do a better job.
(8) Drive out fear, so that everyone may work effectively for the company.
(9) Break down barriers between departments. People in research, design,
sales, and production must work as a team, to foresee problems of production
and in use that may be encountered wJ.th the product or service.
(10) Eliminate slogans, exhortations, and targets for the work force asking for
fewer defects and new levels of productivity. Such exhortations only create adver-
sarial relationships, as the bulk of the causes of low quality and low productivity
belong to the system and thus lie beyond the power of the work force.
(11) Eliminate work standards that prescribe numerical quotas for the day.
Substitute aids and helpful supervision.
(12a) Remove the barriers that rob the hourly worker of his right to pride of
workmanship. The responsibility of supervisors must be changed from sheer
numbers to quality.
(b) Remove the barriers that rob people in management and in engineering of
their right to pride of workmanship. This means, inter alia, abolishment of the
annual or merit rating and of management by objective.
(13) Institute a vigorous program of self-improvement and education.
(14) Put everybody in the company to work in teams to accomplish the trans-
formation. Teamwork is possible only where the merit rating is abolished, and
leadership put in its place.
The first step is for Western management to awaken to the need for change.
It will be noted that the 14 points as a package, plus removal of the deadly
diseases and obstacles to quality, are the responsibility of management.
Management in authority will explain by seminars and other means to a critical
mass of people in the company why change is necessary, and that the change will
involve everybody. Everyone must understand the 14 points, the deadly diseases,
and the obstacles. Top management and everyone else must have the courage to
change. Top management must break out of line, even to the point of exile
amongst their peers.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 t)
© Elsevier Science Publishers B.V. (1988) 7-25
Software Reliability
F. B. Bastani a n d C. V. R a m a m o o r t h y
1. Introduction
Process control systems, such as nuclear power plant safety control systems,
air-traffic control systems and ballistic missile defense systems, are embedded
computer systems. They are characterized by severe reliability, performance and
maintainability requirements. The reliability criterion is particularly crucial since
any failures can be catastrophic. Hence, the reliability of these systems must be
accurately measured prior to actual use.
The theoretical basis for methods of estimating the reliability of the hardware
is well developed (Barlow and Proschan, 1975). In this paper we discuss methods
of estimating the reliability of process control software.
Program proving techniques can, in principle, establish whether the program is
correct with respect to its specification or whether it contains some errors. This
is the ideal approach since there is no physical deterioration or random mal-
functions in software. However, the functions expected of process control systems
are usually so complex that the specifications themselves can be incorrect and/or
incomplete, thus limiting the applicability of program proofs.
One approach is to use statistical methods in order to assess the reliability of
the program based on the set of test cases used. Since the early 1970's, several
models have been proposed for estimating software reliability and some related
parameters, such as the mean time to failure (MTTF), residual error content, and
other measures of confidence in the software. These models are based on three
basic approaches to estimating software reliability. Firstly, one can observe the
error history of a program and use this in order to predict its future behavior.
Models in this category are applicable during the testing and debugging phase. It
is often assumed that the correction of errors does not introduce any new errors.
Hence, the reliability of the program increases and, therefore, these models are
often called reliability growth models. A problem with these models is the dif-
ficulty in modelling realistic testing processes. Also, they cannot incorporate pro-
gram proofs, cannot be applied prior to the debugging phase and have to be
modified significantly in order to be applicable to programs developed using
iterative enhancement.
8 F.B. Bastani and C. V. Ramamoorthy
In this section we first give a formal definition of software reliability and then
present a classification of the models proposed for estimating the reliability of a
program.
2.1. Definition
Software reliability has been defined as the probability that a software fault
which causes deviation from the required output by more than the specified
tolerances, in a specified environment, does not occur during a specified exposure
period (TRW, 1976). Thus, the software needs to be correct only for inputs for
which it is designed (specified environment). Also, if the output is correct within
the specified tolerances in spite of an error, then the error is ignored. This may
happen in the evaluation of complicated floating point expressions where many
approximations are used (e.g., polynomial approximations for cosine, sine, etc.).
It is possible that a failure may be due to errors in the compiler, operating
system, microcode or even the hardware. These failures are ignored in estimating
the reliability of the application program. However, the estimation of the overall
system reliability will include the correctness of the supporting software and the
reliability of the hardware.
Software reliability 9
R = 1 - lim nf
n~oo n
/~=1 nf
n
This method of estimating R is the basis of the Nelson model (TRW, 1976).
2.2. Classification
In this subsection we present a classification of some of the software reliability
models proposed over the past fifteen years. The classification scheme is based
on the three different methods of estimating software discussed in Section 1. The
main features of a model serves as a subclassification.
After a program has been coded, it enters a testing and debugging phase.
During this phase, the implemented software is tested till an error is detected.
Then the error is located and corrected. The error history of the program is defined
to be the realization of a sequence of random variables 1"1, T2, . . . , T,, where Tt
denotes the time spent in testing the program after the ( i - 1)-th error was
corrected till the i-th error is detected. One class of software reliability models
attempts to predict the reliability of a program on the basis of its error history.
It is frequently assumed that the correction of errors does not introduce any new
errors. Hence, the reliability of the program increases, and therefore such models
are called software reliability growth models.
Software reliability growth models can be further classified according to whether
they express the reliability in terms of the number of errors remaining in the
program or not. These constitute error-counting and nonerror-counting models,
respectively.
Error-counting models estimate both the number of errors remaining in the
program as well as its reliability. Both deterministic and stochastic models have
been proposed. Deterministic models assume that if the model parameters are
known then the correction of an error results in a known increase in the reliability.
This category includes the Jelinski-Moranda (1972), Shooman (1972), Musa
(1975), and Schick-Wolverton (1978) models. The general Poisson model (Angus
et al., 1980) is a generalization of these four models. Stochastic models include
Littlewood's Bayesian model (Littlewood, 1980a) which models the (usual) case
where larger errors are detected earlier than smaller errors, and the G o e l -
Okumoto Nonhomogeneous Poisson Process Model (NHPP) (Goel and Okumoto,
1979a) which assumes that the number of faults to be detected is a random
variable whose observed value depends on the test and other environmental
factors. Extensions to the Goel-Okumoto N H P P model have been proposed by
Ohba (1984) and Yamada et al. (Yamada et al., 1983; Yamada and Osaki, 1985).
The number of errors remaining in the program is useful in estimating the
maintenance cost. However, with these models it is d~Aficult to incorporate the
case where new errors may be introduced in the program as a result of imperfect
debugging. Further, for some of these models the reliability estimate is unstable
if the estimate of the number of remaining errors is low (Forman and Sing-
purwalla, 1977; Littlewood and Verall, 1980b).
Software reliability 11
on this approach have been proposed by Mills and Basin (Mills and Basin, 1973;
Schick and Wolverton, 1978) and, more recently, by Duran and Wiorkowski
(1981). The major problem is that it is difficult to select errors which have the
same distribution (such as ease of detectability) as the actual errors in the
program. An alternate approach is to let two different teams independently debug
a program and then estimate the number of errors remaining in the program on
the basis of the number of common and disjoint errors found by them. Besides
the extra cost, this method may underestimate the number of errors remaining in
the program since many errors are easy to detect and, hence, are more likely to
be detected by both the teams. DeMillo, Lipton and Sayward (1978) discuss a
related technique called 'program mutation' for systematically seeding errors into
a program.
In this section we have classified many software reliability models without
describing them in detail. References (Bologna and Ehrenberger, 1978; Dahl and
Lahti, 1978; Schick and Wolverton, 1978; Tal, 1976; Ramamoorthy and Bastani,
1982; Goel and Okumoto, 1985) contain a detailed survey of most of these
models. In the next two sections we discuss a few software reliability growth
models and sampling models, respectively.
In this section we first discuss the concepts of error size and testing process.
We develop a general framework for software reliability growth models using these
concepts. Then we briefly discuss some error-counting and nonerror-counting
models. The section concludes with a discussion on the practical application of
such models.
A program P, maps its input domain,/, into its output space, O. Each element
in I is mapped to a unique element in O if we assume that the state variables (i.e.,
output variables whose values are used during the next run, as in process control
software) are considered a part of both I and O. Software reliability models used
during the development phase are intimately concerned with the size of an error.
This is defined as follows:
An error is easily detected if it has a large size since then it affects many input
elements. Similarly, if it has a small size, then it is relatively more difficult to
detect the error. The size of an error depends on the way the inputs are selected.
Good test case selection strategies, like boundary value testing, path testing and
Software reliability 13
range testing, magnify the size of an error since they exercise error-prone con-
structs. Likewise, the observed (effective) error size is lower if the test cases are
randomly chosen from the input domain.
We can generalize the notion of 'error size' by basing it on the different
methods of observing programs. For example, an error has a large size visually
if it can be easily detected by code reading. Similarly, an error is difficult to detect
by code review if it has a small size (e.g., when only one character is missing).
The development phase is assumed to consist of the following cycle:
(1) The program is tested till an error is found;
(2) The error is corrected and step (1) is repeated.
As we have noted above, the error history of a program depends on the testing
strategy employed, so that the reliability models must consider the testing process
used. This is discussed in the following subsection.
ASSUMPTION. Inputs are selected randomly and independently from the input
domain according to the operational distribution.
This is a very strong assumption and will not hold in general, especially so in
the case of process control software where successive inputs are correlated in time
during system operation. For example, if an input corresponds to a temperature
reading then it cannot change very rapidly. To complicate the issue further, most
process control software systems maintain a history of the input variables. The
input to the program is not only the current sensor inputs, but also their history.
This further reduces the validity of the above assumption. The assumption is
necessary in order to keep the analysis and data requirements simple. However,
it is possible to relax it as follows:
ASSUMPTION. Inputs are selected randomly and independently from the input
domain according to some probability distribution (which can change with time).
This means that the effective error size varies with time even though the
program is not changed. This permits a straightforward modelling of the testing
process as discussed in the following subsection.
14 F. B. Bastani and C. V. Ramamoorthy
i.e.,
where
).j -- failure rate after the j-th failure; 0 <~ ).j ~< ~ ;
Tj(s) = testing process at time s after the j-th failure;
f ( T j ( s ) ) = severity of testing process relative to operational distribution;
0 <~f(Tj(s)) <~ ~ .
REMARKS. (1) As we have noted above, f(Tj(.)) is the severity of the testing
Software reliability 15
process relative to the operational distribution, where the testing severity is the
ratio of the probability that a run based on the test case selection strategy detects
an error to the probability that a failure occurs on a run selected according to the
operational distribution. Obviously, during the operational phase, f(Tj(.)) = 1. In
general it is difficult to determine the severity of the test cases, and most models
assume that f ( T j ( . ) ) = 1. However, for some testing strategies we can quantify
f(Tj(.)). For example, in functional testing, the severity increases as we switch to
new functions since these are more likely to contain errors than functions which
have already been tested.
(2) Even the weaker assumption is difficult to justify for programs developed
using incremental top-down or bottom-up integration (Myers, 1978), since the
input domain keeps on changing. Further, the assumption ignores other methods
of debugging programs, such as code reviews, static analysis, program proofs, etc.
(3) In the continuous case, the time is the CPU time (Musa, 1975).
(4) Software reliability growth models can be applied (in principle) to any type
of software. However, their validity increases as the size of the software and the
number of programmers involved increases.
(5) This process is a type of doubly stochastic process; these processes were
originally studied by Cox in 1955 (Cox, 1966).
f(Tj(s)) = as ~-1 , 2s = ( N - M j ) ~ b ,
where N is the number of errors originally present, Mj = Z ji= 1 mi, and 0~, q~ are
constants.
Hence
R j ( t ) = e - dp(N- Mj)t ~ "
L = fi PD~_,(~)
j=l
log, - (N-
j=l j=l
The MLE's can be computed by numerically solving the equations obtained by
equating the partial derivatives of logL with respect to N, c¢, and ~p to O. The final
equations are as follows:
~ 1 ~ ^ ~
j=l IV-Mj_ 1 j=l~tj =0,
n ^
- - + ~ l o g t j - ~ (p(2V-Mj_,)tTlogtj= 0 ,
~ j=l i=1
n ~ (N - Mj_I)tj ~ = 0.
(~ j=l
~j = ~ j - 1
D
This models the case where there is a possibility that a debugging action may
introduce new errors into the program. For the stochastic input domain based
model (Ramamoorthy and Bastani, 1980) we have:
,~j__l -- ~j-- ~ j _ l X ,
where 2j is the error size and X is a random variable having a piecewise con-
tinuous distribution. This models the case where errors detected later have
(stochastically) smaller sizes than those detected earlier.
In order to illustrate models in this category, we present details of the M u s a -
Okumoto Logarithmic model (i984). The inputs to the model are tl, t2, . . . , tn
where t/ is the time (not interval) at which the j-th error was detected. In this
model:
2o
f(Tj(s))-- 1, 2 ( 0 - - -
20 Ot + 1
Thus, the model assumes that the failure rate decreases continuously over the
testing and debugging phase, rather than at discrete points corresponding to error
correction times. Further, the rate of decrease in 2(0 itself decreases with time,
thus modelling the decrease in the size of errors detected as debugging proceeds
Setting the derivative of logL with respect to 2o and 0 to 0 yields two equations
which can be solved numerically for the maximum likelihood estimates of 2 o and
0, i.e., 2o and 0:
Software reliability 19
n t}~ tj tn -0
A A A
Experience has shown that this model is more accurate than the earlier model
proposed by Musa (1975). Further discussions concerning the application of the
new model appear in (Musa and Okumoto, 1984)
3.6. Summary
We can view 2 as a random walk process in the interval (0, e). Each time the
program is changed (due to error corrections or other modifications) 2 changes.
In the formulation of the general model, 2i denotes the state of 2 after the j-th
change to the program. Let Zj denote the time between failures after the j-th
change. Zj is a random variable whose distribution depends on 2j. In all the above
continuous (discrete) time models, we have assumed that this distribution is the
exponential (geometric) distribution with parameter 2j, provided that f(Tj(.)) = 1.
We do not know anything about the random walk process of 2 other than a
sample of time between failures. Hence, one approach is to construct a model for
2 and fit the parameters of the model to the sample data. Then we assume that
the future behavior of 2 can be predicted from the behavior of the model.
Some of the models for 2 which have been developed are as follows:
General Poisson Model (Angus et al., 1980): The set of possible states are (0, e/N,
2e/N . . . . , e); 2j = ( N - j ) e / N ; the parameters are e and N, there is a finite num-
ber of states.
Geometric De-Eutrophication Model (Moranda, 1975): The set of possible states are
(e, ed, ed 2, ed 3. . . . ), where d < 1; 2j = edJ; the parameters are e and d; there is
an infinite (although countable) number of states.
Stochastic (Input Domain) Model (Ramamoorthy and Bastani, 1980): The state is
continuous over the interval (0, e); 2j = 2j_ 1 + Zig.,where Aj ~ 2j_ 1X, X ~ fl(r, s);
the parameters are r and s.
An alternative approach is the Bayesian approach advocated by Littlewood
(1979). In this method, we postulate a prior distribution for each of 2 l, 22, ..., 2j.
Then based on the sample data, we compute the posterior distribution of 2j+ 1.
Some additional discussions appear in (Ramamoorthy, 1980).
Over 50 different software reliability growth models have been proposed so far.
These models yield widely varying predictions for the same set of failure data
(Abdel-Ghaly et al., 1986). Further, any given model gives reasonable predictions
for one set of data and incorrect predictions for other sets of data. This has led
some researchers to propose that for each project several models should be used
and then goodness-of-fit tests should be performed prior to selecting a model that
is valid for the given set of failure data (Goel, 1985; Abdel-Ghaly et al., 1986).
20 F. B. Bastani and C. V. Ramamoorthy
A basic problem with all software reliability growth models is that their assump-
tion that errors are detected as a result of random testing is not true for modern
software development methods. Models which have been validated using data
gathered over a decade ago are not necesarily valid for current projects that use
more systematic methods and tools. As an analogy, consider the task of reviewing
a technical paper. There are (at least) three major types of errors which can creep
into a manuscript. These are (1) spelling, typographical, and other context inde-
pendent errors, (2) grammatical, organization, style, and other context dependent
errors, and (3) correctness of equations, significance of the contribution, and other
technical errors. Context dependent errors can be detected by random testing (i.e.,
by selecting anyone familiar with the language to review the paper) while three
carefully selected referees are vastly superior to a thousand randomly selected
referees in their ability to detect technical errors. Also, the failure process
observed when all the errors are detected by human beings (testing) is different
from that observed when automated tools such as spelling and grammar checkers
are used. Similarly, in software development we now have tools that can detect
most context independent errors (syntax errors, incorrect procedure calls, etc.)
and context dependent errors (undefined variables, invalid pointers, inaccessible
code segments, etc.). These tools include strongly typed languages and their
compilers, data flow analyzers, etc. The remaining errors are generally the result
of misunderstanding of specifications. These are best detected by formal code
review and walk-through, simulation, verification where possible, and systematic
testing which can be either incremental bottom-up or top-down and which
emphasizes error prone regions of the input domain, such as boundary and
special value points. Again, the failure process when these methods are used is
completely different from that obtained when only random testing is used.
In summary, software reliability growth models treat the program as a black
box. That is, the reliability is estimated without regard to the structure of the
program, number of procedures which have been formally proved/derived, etc.
The validity of their assumption regarding random testing is generally not true for
modern program development methods. Experience shows that with systematic
validation techniques, errors are initially detected in quick succession with an
abrupt transition to an (almost) error free state. Thus, these models can only be
used for obtaining an approximate estimate of the reliability of programs.
4. Sampling models
e(1)= ej(1-
j=l
DISCUSSION. This model is a big improvement over the original model. Some
comments are:
(1) The assignment of values to bj is ad hoc; no theoretical justification is given
for the assignment (Nelson, 1978).
(2) The model uses only one type of complexity measure, namely, number of
paths, functions, etc. However, it does not consider the relative complexity of
each path, function, etc.
22 F. B. Bastani and C. V. Ramamoorthy
Many other interesting aspects of the Nelson model are discussed in (TRW,
1976).
= e-RV
j=l 1 +e -'txj '
Software reliability 23
4.3. Summary
The models discussed in this section are especially attractive for medium size
programs whose reliability cannot be accurately estimated by using reliability
growth models. These models also have the advantage of considering the structure
of the program. This enables the joint use of program proving and testing in order
to validate the program and assess its reliability (Long et al., 1977).
5. Conclusion
(1) During the testing and debugging phase at least two different software
reliability growth models should be used, primarily for helping the manager to
make decisions such as when to stop testing, etc. Goodness-of-fit tests should be
performed in order to select the model which is most appropriate for the failure
data obtained from the project.
(2) After the reliability growth models indicate that the reliability objective has
been achieved, a sampling model is used in order to get a more accurate estimate
of the reliability of the program.
(a) At first equivalence classes are determined based on the paths in the
program using the selection criterion discussed in Section 4.2. Boundary value
and range testing are performed in order to ensure that the classes are chosen
properly.
(b) If the path corresponding to each equivalence class can be verified (e.g., by
using symbolic execution) then the correctness probability of the class is 1.
(c) If the correctness of the path cannot be verified, then the degree of the
equivalence class is estimated. Next, as many test cases as necessary are used so
as to achieve a desired confidence in the correctness of the software.
During the first decade of software reliability research the major emphasis was
on developing models based on various assumptions. This resulted in the pro-
liferation of models, most of which were neither used nor validated. Currently the
consensus appears to be that perhaps there is no single model which can be
applied to all types of projects. Hence, one area of active research is to investigate
whether a set of models can be combined so as to achieve more accurate
reliability estimates for various situations. Other research topics include (1) devel-
oping methods of analyzing the confidence in the predictions of a model, and
(2) using software reliability theory to assist with the management of a project
throughout its life cycle.
References
Forman, E. H. and Singpurwalla, N. D. (1977). An empirical stopping rule for debugging and testing
computer software. J. Amer. Stat. Ass. 72, 750-757.
Goel, A. L. and Okumoto, K. (1979a). A time-dependent error-detection rate model for software
reliability and other performance measures. 1EEE Trans. ReL 28(3), 206-211.
Goel, A. L. and Okumoto, K. (1979b). A Markovian model for reliability and other performance
measures for software systems. In Proc. Nat. Comput. Conf., New York 48, 767-774.
Goel, A. L. (1985). Software reliability models: Assumptions, limitations, and applicability. IEEE
Trans. Softw. Eng. 11(12), 1411-1423.
Jelinski, Z. and Moranda, P. (1972). Software reliability research. In: W. Freiberger, ed., Statistical
Computer Performance Evaluation. Academic Press, New York, 465-484.
Littlewood, B. and Verrall, J. L. (1973). A Bayesian reliability growth model for computer software.
J. Roy. Stat. Soc. 22(3), 332-346.
Littlewood, B. (1979). How to measure software reliability and how not to... IEEE Trans. Rel. 28,
103-110.
Littlewood, B. (1980a). A Bayesian differential debugging model for software reliability. Proc.
COMPSAC "80. Chicago, IL, 511-519.
Littlewood, B. and Verrall, J. L. (1980b). On the likelihood function of a debugging model for
computer software reliability. Dep. Math., City Univ., London.
Long, A. B. et al. (1977). A methodology for the development and validation of critical software for
nuclear power plants. Proc. 1st Int. Conf. Comp. Softw. & Appl. (COMPSAC "77). Chicago, IL.
MacWilliams, W. H. (1973). Reliability of large real-time control software systems. In: Rec. 1973
1EEE Syrup. Comput. Sofiw. Rel. New York, 1-6.
Mills, H. D. (1973). On the development of large reliable software. Rec. IEEE Syrup. Comp. Softw.
Rel. New York, 155-159.
Moranda, P. B. (1975). Prediction of software reliability during debugging. In: Proc. 1975 Annu. Rel.
and Maintainability Symp. Washington, DC, 327-332.
Musa, J. D. (1975). A theory of software reliability and its applications. IEEE Trans. Softw. Eng.
1(3), 312-327.
Musa, J. D. and Okumoto, K. (1984). A logarithmic Poisson execution time model for software
reliability measurement. In: Proc. 7th Int. Conf. Softw. Eng., Orlando, FL, 230-237.
Myers, G. J. (1978). The Art of Software Testing. Wiley, New York.
Nelson, E. (1978). Estimating software reliability from test data. Microelectronics and Reliability 17,
67-74.
Ohba, M. (1984). Software reliability analysis models. IBM J. Res. Develop. 28, 428-443.
Ramamoorthy, C. V. and Bastani, F. B. (1979). An input domain based approach to the quantitative
estimation of software reliability. Proc. Taipei Sere. on Softw. Eng. Taipei.
Ramamoorthy, C. V. and Bastani, F. B. (1980). Modelling of the software reliability growth process.
In: Proc. COMPSAC "80, Chicago, IL, 161-169.
Ramamoorthy, C.. and Bastani, F. B. (1982). Software reliability--Status and perspectives. 1EEE
Trans. Soflw. Eng. 8(4), 354-371.
Schick, G. J. and Wolverton, R. W. (1978). An analysis of competing software reliability models.
IEEE Trans. Softw. Eng. 4(2), 104-120.
Shooman, M. L. (1972). Probability models for software reliability prediction. In: W. Freiberger, ed.,
Statistical Computer Performance Evaluation. Academic Press, New York, 485-502.
Tal, J. (1976). Development and evaluation of software reliability estimators. UTEC SR 77-013, Univ.
of Utah, Elect. Eng. Dep., Salt Lake City, UT.
Trivedi, A. K. and Shooman, M. L. (1975). A many-state Markov model for the estimation and
prediction of computer software performance parameters. In: Proc. 1975 Int. Conf. Rel. Sofiw., Los
Angeles, CA, 208-220.
TRW Defense and Space Systems Group (1976). Software Reliability Study. Rep. No. 76-2260.1-9-5,
RW, Redondo Beach, CA.
Weyuker, E. J. and Ostrand, T. J. (1980). Theories of program testing and the application of revealing
subdomains. IEEE Trans. Softw. Eng. 6(3), 236-246.
Yamada, S., Ohba, M. and Osaki, S. (1983). S-shaped reliability growth modeling for software error
detection. 1EEE Trans. Rel. 32, 475-478.
Yamada, S. and Osaki, S. (1985). Software reliability growth modeling: Models and applications.
1EEE Trans. Softw. Eng. 11(12), 1431-1437.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 "~
© Elsevier Science Publishers B.V. (1988) 27-54 J
Richard A. Johnson
I. Introduction
That is, reliability is the probability that the unit is strong enough to overcome
the stress.
Let the stress X have continuous distribution F(x) and strength Y have con-
tinuous distribution G(y). When X and Y can be treated as independent,
This model, first considered by Birnbaum (1956), has found an increasing number
of applications in civil, mechanical and aerospace engineering.
The following examples help to delineate the versatility of the model.
EXAMPLE 1.1 (Rocket engines). Let X represent the maximum chamber pres-
sure generated by ignition of a solid propellant, and Y be the strength of the
rocket chamber. Then R is the probability of a successful firing of the engine.
EXAMPLE 1.2 (Comparing two treatments). A standard design for the com-
parison of two drugs is to assign Drug A to one group of subjects and Drug B
to another group. Denote by X and Y the remission times with Drug A and
Drug B, respectively. Inferences about R = P [ Y > X ] , based on the remission
time data X l , X 2 . . . . , X m and I11, Y2, " " , Yn, are of primary interest to the
experimenter. Although the name 'stress-strength' is not appropriate in the
present context, our target of inference is the parameter R which has the same
structure as in Example 1.1.
EXAMPLE 1.3 (Threshold response model). A unit, say a receptor in the human
eye, operates only if it is stimulated by a source whose random magnitude, Y, is
greater than a (random) lower threshold for the unit. Here
= Ulmn
where U = number of pairs (X,., Yj) with Yj > Xr Alternatively, we can express/~
as
1~ = Fro(y) dG,(y)
-oo
where Fm(") and Gn(') are the empirical cdfs of the X's and Y's, respectively.
Now
U---mn+m(m+ 1 ) / 2 - ~ rank(Xi).
i=l
Owen, Craswell and Hansen (1964) point out that /~ remains unbiased even if
F(.) and G(.) are not continuous.
It is also possible to obtain a distribution-free lower confidence bound on
R = P[Y<X] based on /~. First note that,
1~ - R = Fro(y) d G . ( y ) - F ( y ) dG(y)
(2.1)
=
X- ~
[Fm(y) - F(y)] d G , ( y ) +
-oo
F ( y ) [dGn(y ) - d G ( y ) ] .
Birnbaum and McCarthy (1958) bound the right hand side of relation (2.1) by
so a conservative lower confidence bound can be obtained for R from the dis-
tribution of D~, + Dff. If c is selected so that 1 - ~ ~< P[D~, + D~+ <~ c], then
n e - 2mc2 m e - 2nc2
-1 (2.4)
m+n m+n
2 x / ~ m n c e - 2mnc2/(m+ n) 1 f 2~c/,/~+~
e - ~/2 d t .
(m + n) 3/2 xf r ~ d-2mc/m~/'~
I/~ - R ] ~ s u p I F . - F ] + s u p l G m - GI = Dm + D,,
x y
that follows directly from (2.1). However, the resulting intervals were very wide
so he suggests employing the large sample normal approximation for R.
/~-R S
x/min (m, n'~ - - , N (0, 1)
where
<y
+-
n -<y
G(x) [ 1 - G(y)] dF(x) d F ( y ) ] .
Stress-strength models for reliability 31
c = ~ - 1(1 - y) (2.5)
2 x / m i n (m, n)
where cb-l(") is the inverse of the standard normal cdf. Then / ~ - c is the
100(1 - 7)Yo lower-confidence bound on R. For the equal sample size situation,
Govindarajulu (1968) shows the 9 5 ~ confidence bound takes x / m + n c = 1.17
whereas ~/-m + n c = 2.93 for the Birnbaum and McCarthy approach.
Alternatively, a 2 in Theorem 2.1 can be estimated by replacing F by Fm and
G b y Gn.
Sen (1967) gives essentially the same result as Govindarajulu although he derives
slightly different estimates of a 2. One of his estimates of a 2 can be described in
terms of ranks.
Let Rl, . . . , R m be the ranks of the X i and S 1. . . . . S n the ranks on the Yj in
the combined sample. Set
sgl = 1
n-1 j=l
(Si - j ) - n S - ( +)]
2
12 .
&2=(Sm°+~)min(m,n).
Given any parametric families {F(x[O1), 01e191} for stress and {G(y[02),
Oze 192} for strength, the reliability
Among the numerous choices for stress and strength distributions, only a few
special cases are amenable to exact small sample inference procedures. We first
treat the normal and then the Weibull stress-strength models before discussing the
general case.
{ ]~2--/"£1 "~
R = PIt> Xl = e l Y - x > Ol = ¢
where (I)(.) is the standard normal cdf with pdf cp(.). Without further assumptions,
it is not possible to obtain exact confidence procedures.
×
f l f d(v) (1 -- /)2)(n - 4)/2(1 _ U2)(m- 4)/2 d u d v
-1 ~/-1
with
(y - X) x//m s2(n - 1) fmm
d(v) =
s l ( m - 1)--+I) sl(m 1) ~]n "
We take
1~=~0 i f d ( v ) < ~ - 1 , all I r i s < l ,
(3.2)
h ifd(v)>f 1, all [v[~<l.
Stress-strength modelsfor reflability 33
When the sample sizes m and n are both large, confidence bounds for R can
be set using the approximate normality of ~, &2 = y ~ ( x _ y ) 2 / m , ~ and
& 2 = y . ( y ~ - 2 ) 2 / n . The maximum likelihood estimator of reliability is
a2
/~ = ~ ( Y - x)/v/~-~ + a2)" Since
x/ 1 ) (3.3)
+ op min (m, n)
^2_Im+n [~+__+&22 ( y _ y ) 2
~rR ~.2 + 8.2 n 2(8 2 + &2z)z
(3.6)
',x/ a~ + a 2
where 1 - e = ~(z~).
/[(1 1~,~,2]1/2
T=(Y-X) +n] "J
whose noncentrality parameter is b = (/~2 -/-~l)/a(1/m + 1/n) I/2. Since P~[ T ~< t] is
monotonically decreasing in 5, a 100(1 - e)~o lower confidence bound for b is
given by _5 where
P ~ [ T o b ~ ~< t] = 1 - (3.7)
(3.9)
where
^2 - _ _
n ^2 +
(y-/~l) 2 ( n - 1)a 2
The fact that/~l and ~r1 are known does not seem to lead to exact inference
procedures. Mazumdar (1970) does obtain an exact, but inefficient, confidence
bound by introducing m pseudo random numbers for the first sample.
Stress-strengthmodelsfor reliability 35
=PIT(bm,,,)<x/2(l+~)-'/2z(~_R_c) 1 (3.10)
Note that the sample sizes m and n enter the non-centrality parameter, degrees
of freedom and the percentile x/~+mz(l_R_c). The values of m and n do,
however, enter (3.10) symmetrically. In an application, the solutions m, n must be
maximized over the range R of interest. Owen, Craswell and Hansen (1964) give
a table of values for the case of equal sample sizes.
Rol.o~,p=l-fo~e-(x/o1'"P(X~] p- e- (x/°:)Pdx
02 \02/
1
1 + (O,/Oz) p (3.11)
( X
1/ 1 + ~ F2rt, 2m(0~ ) ) <R (3.13)
where F2. ' 2.n(~) is the upper ~-th point of the F2. ' 2,.-distribution.
Alternatively, since (n/m)F2." 2m/(1 + (n/m)F2,,, 2m) has a beta distribution with
parameters n and m, the lower confidence bound can also be expressed as
1 -- ~/1--0¢
<R
- m~
where t/1 _~ is the 100(1 - ~)-th percentile of the beta distribution. The case of
known stress parameter, 01, can be treated by the same methods.
Basu (1980) considers the Marshall-Olkin bivariate exponential distribution.
can
A
usual!y be evaluated by numerical methods. Notice that /~ is the MLE if
0~ and 02 are MLE's. Except for the normal and exponential cases, confi-
dence bounds must be based on large sample theory.
Suppose
- , N(0, I 1 1(01) )
A
independent of 0 2 and
where 11(01) and 12(02) are the Fisher information matrices for the stress and strength
distributions, respectively. Then, if the derivatives are smooth
Stress-strength modelsfor reliability 37
= n vym(o,_0,)' ~ Fo,(x)dGo~(x)
? +n ^ ~ f Fo,(x)dGo~(x)
+O(minmn,) 1
(3.15)
~f(ul01)
f~ [1 - Go2(U)] f(ulO,)du,
_ ~ ~ O , / f ( u [ 01 )
bo = f ~ F o , ( X ) ~02/g(xl
8g(x]02)02) g(xl 02) dx. (3.16)
Notice that ao, and bo2 are expressions for the covariance of score functions.
^2 1 ^ 1 1( ^ b
aR'~'=--~ a'°llll(Ol)a'°l+--l- 2 b'~ I f 02) ~ .
R ~ , . ~2 - z ~ R , ~ / , f m + n. (3.18)
38 R. A. Johnson
= 1-~2N[G(x)]dF(x ) (4.1)
where ~ ( ' ) is the cdf of the beta distribution having density oc uk-S(1 - u)s- 1
Stress-strengthmodelsfor reliability 39
m i= 1 (4.2)
where s(~)~< S(2 ) ~ " ' " ~ S(n) are the ordered ranks of the Y's in the combined
sample. Bhattacharyya and Johnson (1975) also derive the UMVE estimator as
a generalized U-statistic based on the kernal
m,:, ,44,
Note that/~s,~ is similar in form t o / ~ * k but that it has the feature of a trimmed
mean.
Bhattacharyya and Johnson (1977) establish the following large sample result.
THEOREM 4.1. Let m, n~oo with m/(m + n)--*2 (0<2< 1). Then pointwise
(m + n) (Rs, ~ - R,,~)
~* = 0(1)
and
,/m + n(~,,k- R,,~) ~ , Y O, 1 - 2
where
a~ = VarF[ ~(G(X))] = f ~ 2 ( G ) d F - [ f ~(G)dFl 2,
(4.5)
~r~ =
--0(3 f2 b[ G(x)] b[ G(y)] {G(min (x, y)) - G(x) G(y)} dF(x) dF(y),
From Theorem 4.1 we conclude that a large sample 100(1 - a)% lower con-
fidence bound for R~, k is given by
, ~ 1_2+ (4.6)
so confidence bounds similar to (4.6) are immediate. When F(.) is known, the
100(1 - e)% confidence bound on R~I, is
Z~
/~,,k ~ &l. (4.10)
k! ~ 1
Rs, k
= 1 s! j=s iJ + 02/01)
1 Z (-1) j k-s () 1 (4.11)
B(s, k s + 1) j=o j (s + j + 02/01)
where the last expression is obtained by expanding the product into partial
Stress-strength modelsfor reliability 41
~m
fractions. Here B(s, k - s + 1) is the beta function. We note that ( ;=7 Xe,
Y~"e=l Y,.) is a complete sufficient statistic and ( s + j ) - l u [ ( s + j ) X 1 - YI],
u(x)--1(0) if x > (~<)0, is an unbiased estimator of (s + k + 02/01) -1. The
Rao-Blackwell method leads to the UMVU estimator but its form is complicated
and depends on the hypergeometric function of the second kind. The maximum
likelihood estimator, /~s, k, has the considerably simpler form
k! k~s 1 (4.12)
/~s.k = 1 S! j=O (j + S + Y/X)
111 41>
As a consequence of Theorem 4.2, lower confidence bounds are obtained using
/~s, k to estimate R and Y/X to estimate 02/01 in the expression for trnz.
The asymptotic relative efficiency of the nonparametric estimator (4.2) or (4.4),
versus the exponential maximum likelihood estimator (4.11), is given by
1 2
2 o22 , +-OJOl)]j
e= (4.14)
(1 - 2)a 2 + 2aft
ponents. Suppose that out of the k components, k~ are of one category and their
strengths can be reasonably assumed to have a common distribution G 1. The
remaining k 2 = k - k I components are of a different category and their common
strength distribution is denoted by G2. All the k components are exposed to a
common stress X having the distribution F, and the system operates successfully
if at least s of the k components withstand the stress. This corresponds to the
same structure function (4.3). Here, however, Y~. . . . , Yg, are iid G~,
Yk, + 1. . . . , Yg are iid G2 and X is distributed as F. The system reliability is a
functional of the triplet (F, G~, G2) and it can be formally expressed as
kl k2
R= (j~)(j2)j~ ~ ~ f(1-G,y' Gf'-J'(1-G2)J2Gkl-J2 dF (4.15)
where the sum extends over 0 ~ j a ~< k l , 0 ~<J2 ~< k2 such that s ~ j a +J2 ~< k.
When F, G1 and G 2 are exponential with the scale parameters 0, fl~ and f12, the
integral in (4.15) can be simplified to a linear function of terms of the form
[alfl I + a2fi2 + 0] - l where the known constants a I and a2 vary from term to
term. With independent random samples {X~, . . . , Xm}, {Y~, ..., Yah,} and
{Y21 . . . . . Y2n2} from F, G~ and G 2 respectively, one can easily obtain the maxi-
mum likelihood estimator of R. The U M V U estimator can also be worked out
along the lines of Section 4.2.
Nonparametric estimators of R can be constructed by either of the two proce-
dures. For instance, a nonparametric estimator/~* is obtained by replacing F, G 1
and G 2 in (4.15) by the empirical cdfs. Alternatively, defining the kernel function
h(X1; Yll, .'., Ylk,; Y2~, "", Y2k:) = 1 if at least s of the (k~ + k2)
Y's exceed X1,
= 0 otherwise, (4.16)
EXAMPLE 4.2. Consider a system with k = 2 and s = 1 where the two com-
ponents have strength distributions G 1 and G 2 and are subjected to common
stress with distribution F.
Stress-strength models for reliability 43
R= f (1-G1)G2dF+ f G I ( 1 - G z ) d F + f ( 1 - G 1 ) ( 1 - G 2 ) d F
= 1 - f GIG2dF.
where T 1, Tz and T3 are the numbers of triplets {X;, Ylj,, Y2j2} satisfying
(Y1j, < X i < Y2j2), (Y2j2 < Xi < Yljl) and (Xi < Ylj,, Xi < Y2j:), respectively. The
estimator based on the empirical cdf's is given by
where Qi is the rank of the i-th order statistic X(o within the combined X and Y~
samples, and Q[ is the rank of X(,.) within the combined X and Y2 samples.
2 out of 3 1 out of 2
The subsystem A functions when at least two of the three components survive
the stress X. The component strengths are iid with distribution G~ and the
common stress X has distribution F 1. Similarly, the subsystem B has the structure
of a 1 out of 2 stress-strength model where the strength and stress distributions
are G2 and F 2 respectively. The system reliability R is given by
R = R 2A, 3 R B
1,2
where the factors on the rhs are the stress-strength reliability functions for the
subsystems and they have the same forms as given in (4.1). Using the methods
of Section 4.1, one can obtain the UMVU estimator for each of R~, 3 and R B 1,2
and, due to independence, their product will give the nonparameter UMVU
estimator /~ of R. The limiting normal distribution o f / ~ and the form of the
asymptotic variance can then be obtained from the subsystem results.
(c) Binomial data on components. Often, components are tested under random
stress conditions that prevail, and only the number of survivors are recorded
rather than the measurements of stresses and strengths. In the context of a single
component stress-strength model where our objective is to estimate the probability
R~ = P [ Y > X] = S (1 - G ) d F , the present sampling process yields the count Z n
which is the number of pairs (X~, Y~.), i = 1, . . . , n, such that Y,. > X i. The numeri-
cal measurements of Y,. and Xi are not recorded. The problem then reduces to
estimating a binomial probability from the number of successes in n trials. More
generally, consider a system consisting of c subsystems where each subsystem has
the structure of a single component stress-strength model. The system reliability
is then a function
where Pi = S (1 - Gi) dF;, G,. and Fi are the strength and stress distribution for the
i-th subsystem, and the functional form o f g is determined by the manner in which
the system is structured. Methods of estimating the system reliability from
binomial count data have been developed by Myhre and Saunders (1968),
Madansky (1965), Easterling (1972), and many others. The stress-strength formu-
lation of the model loses its distinctive features when only the count data are
recorded and the subsystems have single components.
For a k (~>2) component stress-strength system where all the components are
exposed to a common stress X in their natural operating environment, some care
is needed for using binomial count data of the component failures for estimating
the system reliability. Intuitively, one might interpret the reliability of an s out of
k system as the probability of obtaining s or more successes in k Bernoulli trials
and proceed to estimate this binomial probability from the count data. In this
process, one would be estimating the functional
Stress-strength modelsfor reliability 45
Rs, k= ~ (4.18)
Alternative definitions are also plausible. We could only require that current
strength exceed the maximum thus far encountered.
Even less stringent, the requirement could be that current strength exceeds current
stress.
Using definition (5.3), Basu and Ebrahimi (1983) consider the case where X(t)
and Y(t) are brownian motion processes with means /Zl, #a and covariances
tr~ min(s, t), tr22min(s, t). They show that
46 R . A . Johnson
which is of the same form as the normal theory model in Section (3.1). Expression
(5.4) would not be expected to apply for large to since R(to)>_. 0.5 all to, when
['/'2 > /21"
Y = ~z + f12z + e2
where e2 is distributed N(0, ~rzZ). For a specimen whose stiffness is z, the con-
ditional reliability becomes
B.z "~
R(X) = PIE> X[z] = r'z- ~]A1] . (5.5)
EXAMPLE 5.2. Refer to Example 1.2 where the purpose is to compare remission
time X using Drug A with remission time Y using Drug B. Suppose that the age
z of the subject influences the remission time. We postulate the linear regression
relations
X= cq + fllz + e l , Y = c~2 + ~2z + e z
/ + (¢~2 -- fl,)Z'].
r(z) = e[r> Xlz] = ~1 ~ 2 (5.6)
\
The models in Example 5.1 and Example 5.2 were introduced by Bhattacharyya
and Johnson (1981). Initially, we consider the more general model where X and
Y may depend on possible different covariates. Set
and assume
Some exact inference procedures are available when the variances are equal.
Set a 2 = a ~ = a 2 so
is normally distributed with mean ~2 + P~Z2o - ~ - P' zlo and standard deviation
Coa where
= - - "+ -- "1- ( Z l 0 -- ~ 1 ) t
Em
Z j = 1 (Zlj -- ~ I ) ( Z l j -- ~ 1 ) t
]1 (ZlO -- Z l )
m n
1
(Z2o - ~2). (5.8)
^ ^
Here ~1 and f12 a r e the least squares estimators. Also
+ Z (y; - y - ~ ( z 2 j - ~2)) 2 •
j=l
48 R. A. Johnson
At
T + ~2(Z2o - ~2)
T=
CoS
q = ~ + t ~ Z o - ~, - tr, Z~o
CoO"
A lower 95% confidence bound, r/, is obtained by solving Fn(Tobs) 0.95 for =
r/. Consequently, a 95% lower confidence bound for R(z~o, Z2o) is given by
Given the random sample X1 . . . . , Xm from f(" q01) and an independent ran-
dom sample from g(" I 02), together with a prior density p(O,, Oz), in principle one
can obtain the posterior distribution
h(01, 02[Xl . . . . , Xm, Y,, "" ", Yn) = p(01, 02) f i f(xil01) (-~ g(y, 102)
i= I j= 1 (6.1)
for (01, 02)- This distribution could then be transformed to the posterior distribu-
tion of Ro," o2 = ~ F(yl 01) dG(y] 02). Enis and Geisser (1971) obtained analytical
results for negative exponential distributions and for normal distributions with
equal variances.
Combining the likelihood (3.12) of the samples of sizes m and n, we obtain the
joint posterior density
..... 1 ...... l
h(O,, 02I~, Y)~:\~I e-(c'+m~>/°'\~l e-C.... ~>/o2
(6.3)
Stress-strength modelsfor reliability 49
The transformed variable p = (1 - r)/(1 - cr) has a beta distribution with parame-
ters n + s 2 and m + s ~ so
P[r<R]=PIp< 1 - 1 _ r]_
(6.6)
1 f(l -- E)/(1 -- c_r)
u,+S2- 1(1 _ u)m+sl- 1 du.
B(n + 82, m + sl) Jo
A 100(1 - ~)% Bayesian lower bound on R is given by
1 -- ~]1--o:
r - (6.7)
1 - cql _~
where qa - ~ is the 100(1 - ~)-th percentile of the beta distribution with parameters
n + s 2 and m + s I . Comparing (6.7) with the alternative form for the bound below
(3.13), we see that the choice of 'informationless' priors, s a - - s 2 = 0 and
c I = c 2 -- 0, leads to the same bound as the classical procedure.
(6.8)
1 1
(2/1;)TM + n)/2 o.rn + n
.o._lexp { (c I + m ) ( mx+clml~2~
20.2 I'll CZ + m "] .)
• rr 1 exp { 2a 2
#2 . . . . . . .
c2 + n / )
• ,610,
t~
Z = -
0,2
w h i c h is d i s t r i b u t e d 2 + n + b" Setting
as ~(m
m ~ + c~m~-] / . -
c =1 I - -1 + - - 1 1 , k . [. n~+c2m2
. . _ _ _ _ //,,/2v,
m + C1 Fl "[- C 2 L n -J- C 2 m + c~ d /
427zc 2(m+n+b)/2mWH+b
F() 2
Stress-strength models for reliability 51
Ep[R] = e I tb < m2 - ml 1
0 +--
+C 1 I+C
Enis and Geisser (1971) show that the choice of a vague prior oc tr- ~ produces
a posterior distribution whose expectation E ( R ) is closer to 0.5 than is the
maximum likelihood estimator• Finally, it should be remarked that they treat the
slightly more general case of estimating P [ a I X 1 + a 2 X 2 + • • • + apXp > 0] and
that one of their formulations includes paired stress-strength data.
Table 5.1
Expert opinions concerningpercentiles of the strength dis-
tributions (ft/sec2)
Percentile
Since the primary source of information about the random variation in stress
and strength is expert opinion and engineering judgement, it is a more difficult
problem to obtain lower bounds for R. In the context of the nuclear power plant,
the lower bound on R converts to an upper bound on the probability of failure
and subsequent radioactive release. Some attempts have been made to quantify
the uncertainty experts have in formulating their opinions and using this quantified
Stress-strength models for reliability 53
uncertainty to develop bounds for the probability of failure. See Bohn et al. (1983)
for more information.
A risk analysis of a system is considerably more complicated than for a single
component. With a nuclear power plant, failure can occur in numerous ways.
From a fault-tree analysis, each separate failure path is determined. Data are
typically available on some component strengths but it is mostly expert opinion
that must be combined in order to obtain an estimate of the failure path proba-
bilities and, ultimately, the system reliability. The calculation of an estimate of
system reliability can involve as many as 300 to 400 components and the proba-
bility of an accident sequence is calculated from, say, a multivariate normal
distribution. In this setting it is possible to include a stress such as an earthquake,
as a common stress to numerous components.
References
Basu A. (1980). The estimation of P[X< Y] for distributions useful in life testing. Navel Res. Log.
Quart. 3, 383-392.
Basu, A. and Ebrabimi, N. (1983). On computing the reliability of stochastic systems. Statistics and
Probability Letters 1, 265-268.
Bhattacharyya, G. K. (1977). Reliability estimation from survivor count data in a stress-strength
setting. IAPQR Transactions--Journal of the Indian Association for Productivity, Quality and Reliability
2, 1-15.
Bhattacharyya, G. K. and Johnson, R. A. (1974). Estimation of reliability in a multicomponent
stress-strength model. J. Amer. Statist. Assoc. 69, 966-70.
Bhattacharyya, G. K. and Johnson, R. A. (1975). Stress-strength models for system reliability. Proc.
Syrup. on Reliability and Fault-tree Analysis, SIAM, 509-32.
Bhattacharyya, G. K. and Johnson, R. A. (1977). Estimation of system reliability by nonparamatric
techniques. Bulletin of the Mathematical Society of Greece (Memorial Volume), 94-105.
Bhattacharyya, G. K. and Johnson, R. A. (1981). Stress-strength models for reliability: Overview
and recent advances. Proc. 26th Design of Experiments Conference, 531-546.
Bhattacharyya, G. K. and Johnson, R. A. (1983). Some reliability concepts useful in materials testing.
Reliability in the Acquisitions Process. Marcel Dekker, New York, 115-131.
Birnbaum, Z. W. (1956). On a use of the Mann-Whitney statistic. Proc. Third Berkeley Symp. Math.
Statist. Prob. 1, 13-17.
Birnbaum, Z. W. and McCarthy, R. C. (1958). A distribution free upper confidence bound for
P(Y < X) based on independent samples of X and Y. Ann. Math. Statist. 29, 558-62.
Bohn, M. P. et al. (1983). Application of the SSMRP methodology to the seismic risk at the Zion
Nuclear Power Plant, NUREG/CR-3428 Nuclear Regulatory Commission, Nov.
chandra, S. and Owen, D. B. (1975). On estimating the reliability of a component subject to several
different stresses (strengths). Naval Res. Log. Quart. 22, 31-40.
Church, J. D. and Harris, B. (1970). The estimation of reliability from stress-strength relationship.
Technometncs 12, 49-54.
Downton, F. (1973). The estimation of P ( Y < X) in the normal case. Technometrics 15, 551-558.
Easterling, R. (1972). Approximate confidence limits for system reliability. J. Amer. Statist. Assoc. 67,
220-22.
Enis, P. and Geisser, S. (1971). Estimation of the probability that Y < X. J. Amer. Statist. Assoc. 66,
162-68.
Govindarajulu, Z. (1967). Two sided confidence limits for P ( Y < X ) for normal samples of X and
Y. Sankhy-d B 29, 35-40.
54 R . A . Johnson
Govindarajulu, Z. (1968). Distribution-free confidence bounds for P ( X < Y). Ann. Inst. Statist. Math.
20, 229-38.
Guttman, I., Johnson, R. A., Bhattacharyya, G. K. and Reiser, B. (1988). Confidence limits for
stress-strength models with explanatory variables. Technometrics (in press).
Lehmann, E. (1959). Testing Statistical Hypotheses. Wiley, New York.
Kececioglu, D. (1972). Reliability analysis of mechanical components and systems. Nuclear Eng. Des.
9, 257-290.
Lloyd, D. K. and Lipow, M. (1962). Reliability, Management, Methods and Mathematics. Prentice-Hall,
Englewood Cliffs, NJ.
Madansky, A. (1965). Approximate confidence limits for the reliability of series and parallel systems.
Technometrics 7, 495-503.
Mazumdar, M. (1970). Some estimates of reliability using interference theory, Naval Res. Log. Quart.
17, 159-65.
McCarthy, J. F. and Orringer, O. (1975). Some approaches to assessing failure probabilities of
redundant structures. Composite Reliability, ASTM STP 580, American Society for Testing and
Materials, 5-31.
Mensing, R. (1984). Personal communication.
Myhre, J. M. and Saunders, S. C. (1968). On confidence limits for the reliability of systems. Ann.
Math. Statist. 39, 1463-72.
Owen, D. B., Craswell, K. J. and Hanson, D. L. (1964). Nonparametric upper confidence bounds
for P(Y < X) and confidence limits for P(Y < X ) when X and Y are normal. J. Amer. Statist. Assoc.
59, 906-24.
Sen, P. K. (1967). A note. on asymptotically distribution-free confidence bounds-for P(X < Y) based
on two independent samples. Sankhy-d A 29, 95-102.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 h
1
© Elsevier Science Publishers B.V. (1988) 55-72
M. Mazumdar
I. Introduction
The index A measures the fraction of the time that a unit is unavailable for service
during periods when it is not on planned outage. Endrenyi [8] has shown that
this index is meaningful even when maintenance lasting short length of times is
concerned, provided that maintenance itself does not contribute to failure.
In the power system vocabulary, the term used for unavailability is the forced
outage rate (FOR), which unfortunately is a misnomer, since the index represents
a pure number and not a rate. The FOR is defined as
where the times appearing in the numerator and denominator refer to a reasonably
long period of observation. The index (2) is equivalent to (1) when the period
under question is long enough. The above definition of the forced outage rate or
the unavailability assumes that the generating unit has only two states--operating
at 100~o capacity or completely failed. The intermediate capacity states are
usually accounted for by defining an index called the equivalent forced outage rate
(EFOR), which is given by the following equation:
FOH + EFOH
EFOR =
SH + FOH
where FOH, EFOH and SH denote respectively full forced outage hours, equiva-
lent forced outage hours and service hours. The quantity, equivalent forced outage
hours, is obtained by multiplying the actual partial outage horus for each partial
outage event by the fractional capacity reduction and then summing the products.
The introduction of the index EFOR enables one to approximate a unit with
several capacity states by one having only two states. In this two-state equivalent
representation, the index EFOR estimates the long-run probability of being fully
out and the quantity (1-EFOR) estimates the long-run probability that it is availa-
ble at full capacity. Data on EFOR are presented for a variety of sizes and types
of generating units in reports published by the Edison Electric Institute, see, for
example, [7].
Load models
An hourly load duration curve is obtained by first plotting on the vertical axis
the power demand forecasted for each hour in a planning period in a chronologi-
cal order along the horizontal axis. The load duration curve (LDC) is then
obtained by reordering the demands in a descending order of magnitude. Here,
the number of days on which the load exceeds a given value is plotted as an
abscissa with the forecasted load value as the ordinate. Assume that the fore-
casted peak demand occurs for one hour during each of the days in a 20-day
planning. Then, one can say that the peak load occurred in a fraction equal to
1/24 of the planning period. Figure 1 shows that the system load was expected
to be above 100 MW during 50~o of the time. When the abscissa is normalized
to 1, the figure can be read to denote the fraction of the time the load is expected
to be above a given value. Thus it is possible to give a probabilistic interpretation
to the load duration curve. The horizontal axis of the curve yields the survivor
function of the load when it is treated as a random variable. It gives the proba-
bility that the observed load will exceed a specified value as denoted by the
ordinate.
In some studies on generation reliability, notably when unit production costs
are evaluated, it is a practice to merge the individual generating unit failure models
and the load probability distribution by defining the so-called equivalent load
Approximate computation of power generating system reliability indexes 59
150 MW-
100 MW
75 MW
Available capacity = c 1 Jr- c 2 --1- • • • --1- C n - ( X 1 "Jr"X 2 -1- " ' " "~ X n "1- L ) ,
where ci is the installed capacity of unit i, and X; is the capacity on outage for
unit i. Notice that the quantity, (X 1 + X 2 + • • • + X, + L), plays the role of an
equivalent load that confronts the n units of the system. A curve which shows the
proportion of times that the observed equivalent load will exceed given specified
values is called the equivalent load duration curve (ELDC). It is clear from the
foregoing discussion that separate sets of such curves can be drawn for all the
n individual units of the system.
Let L denote the system peak load. Then the loss-of-load probability (LOLP)
index is measured by
In the situation where the LOLP index is being estimated for future time periods,
as is typically done in power generation planning, the forecasted peak load will
be uncertain and regarded as a random variable. We usually regard L as normally
distributed with mean/~ and variance a 2, its distribution being independent of the
X i random variables. If the peak load is regarded as known, a 2 = 0 and L = #,
but otherwise, a 2 > 0, and departures from normality may also be anticipated. Let
Y denote the deviation of the peak load from its mean /~. Then we can also
express (4) as follows:
variance a 2. The electric utilities in the United States plan their operation so as
to meet a targeted value of the LOLP index of the order of 10- 4. Thus, the LOLP
measure represeilts the extreme tail probability in the distribution of
X l + X2 + " " + X . .
P r o d u c t i o n costing i n d e x
For the evaluation of the expected operating costs of a given utility, we assume
somewhat simplifying the real-life situation, that (a) there are n units in the sys-
tem, (b) the units are brought into operation in accordance with somespecified
measure of economy of operation (e.g., marginal cost), and ( c ) t h e unit i, in
decreasing order of economy of operation, has capacity, c i and EFOR, pi,
i = 1, 2, ..., n. Let U denote the system load, and let F(x) = Pr { U > x}. Thus
F(x) represents the load-duration curve or LDC.
Consider now the i-th unit in the loading order and let W~ denote the energy
unserved (i.e., the unmet demand) after it has been loaded. Let, as before, X,.
denote the unavailable capacity for unit i, whose probability distribution is given
by (3) and let U denote the system load. We define
Thus, Z; represents the equivalent load on the first i units. Let ge(.) and G~(.)
denote the probability density and distribution functions, respectively, of Z;.
Clearly,
Now denote by ei the energy produced by unit i. Then it follows from (9) that
Ci- I
Ce --
f/
where
=(l-p;)
I Ci-1
G i_,(z) d z , i = 1, 2 , . . . , n . (10)
E(ei)
CF(i)= , i = 1,2 . . . . . n. (11)
¢i
This index gives the ratio of the expected output to the maximum possible output
for each unit. An accurate estimate of this index is needed by the utilities for the
purposes of evaluating expected system operating costs and optimizing its genera-
tion planning.
Computational difficulties
In its planning process, a given utility needs to compute the LOLP and CF
indexes for various combinations of the system load and mix of generating units.
Thus it is necessary that an inexpensive method of computation be used for the
purpose of computing these indexes. Examining (4), we observe that when the ci's
and the pt's are all different, at least 2 n arithmetic operations will be required to
evaluate one LOLP index. Thus, the total number of arithmetic operations in the
computation of one LOLP index varies exponentially with the number of gener-
62 M. Mazumdar
ating units in the system, and it might become prohibitively large for large values
of n. From (10), we observe that the expected energy output of a given unit is
proportional to an average LOLP value over a range of z between Ce_ ~ and Ci.
Thus, it is not feasible for a power system planner to engage in a direct compu-
tation of (4) or (10), and he has to resort to approximations which require much
less computer time.
3. Approximate procedures
Method of cumulants
From an uncritical application of the central limit theorem, one could have
made the convenient assumption that the distribution of X1 + )(2 + "'" + Xn in (5)
or the survivor function G~_ l(x) in (10) will be approximately normal. While this
assumption may not cause problems while computing probabilities near the cen-
tral region of the probability distribution, the 'tail' probabilities may be in-
accurately estimated. A typical generation mix within a given utility usually
contains several large units and otherwise mostly small units thus violating the
spirit of the Lindeberg condition [ 11 ] of the central limit theorem. An approach
to the problem of near-normality is that of making small corrections to the normal
distribution approximation by using asymptotic expansions (Edgeworth or
Gram-Charlier) based on the central limit theorem. Use of these expansions in
evaluating power generating system reliability indexes has come to be known in
the recent power-system literature as the method of cumulants. For details on the
use of these expansions in computation of LOLP, see [13], and for its use in
computing the capacity factor index, see [5]. In the evaluation of LOLP, one first
obtains the cumulants of X1 + X 2 + • • • + X n + Y by summing the corresponding
cumulants of the Xi's and of Y. These are then used in the appropriate Edgeworth
or Gram-Charlier expansion. Similarly, for the purpose of evaluating E(e~) in
(10), one first obtains the cumulants of Z; for each i = 1, 2 . . . . , n, by summing
up the cumulants of X1, X 2. . . . . X~ and U. Next, one writes the formal expansion
for G~(x) using these cumulants upto a given order. Next, one integrates the series
term by term in (10) to obtain an approximation for E(ei). Caramanis et al. [5]
have made a detailed investigation of this approximation in the computation of
the capacity factor indexes. Their results have cast favorable light on the efficiency
of this method.
eSXF~(dx)
V~(dx) - - - (14)
Further, let V* denote the convolution of V~, V2 . . . . . V,. With these definitions,
it is seen that the LOLP index may be expressed, as follows:
We now choose s such that z equals the mean of V*(.). Thus, although in
practical application, z will lie in the fight hand tall of F*(-), it will now be at the
center of the d.f. V*(.). We also expect the distribution of V*(.) to be much closer
to the normal distribution in the central portion of the distribution (much more
so as compared to the tails). Thus, in the second integral of (15), the integration
is being done in a region where V*(.) could be accurately approximated by a
normal distribution or an expansion of the Edgeworth type. The effect of the
multiplier e-sz for s > 0 is to de-emphasize the contribution of V*(dx) for values
of x in the tail. Esscher's approximation technique consists in replacing V*(dx)
by an appropriate normal distribution or an Edgeworth expansion, and evaluating
(15).
It can be shown [9] that corresponding to a given s, the first four cumulants
of V*(.) are given by
p,(_l- p9c£
O"(s) = , =~"1 [Pi + (1 - P i ) e . . . . 12 , (16b)
~O'(s) = z. (17)
Call this unique root so. We now replace V*(dx) in (15) by the normal density
or an appropriate Edgeworth expansion. For a random variable X, whose first
four cumulants are kl, k2, k 3 and k 4, its density F(dx) is approximated by the
Edgeworth expansion [6] formula as follows:
where
tp(t) 1 ,2/2 d~tp(t) t - x - k1
=~e- , ¢~(t)- dt ~ , g~/2 '
k3 k4
~1 k3/2 , 72 k~
LOLP=Pr{X~+X2+'"+Xn>z}
- LOLP l = eq,¢s0)-~O~Eo(u) (19a)
where
_ LOLP 3 = LOLP 2 + LOLP 1
[;4 uv + -7'lz
-
72
u3v - , (19c)
u2- 1
1) = U 3 -- - - , ~](SO) = logeP*(So) •
W
Pr{X, + X 2 + . . . +Xi_ , >zj} = 1 - Pr{X, +)(2 + " ' " +Xi_, <~zj},
(21)
_~ L O L P 3 = L O L P 2 + L O L P ~ 7~ u v ' + - - uv' -
(24 72 ~7 '
(23c)
66 M. Mazumdar
where
u2 - 1
~o(U) = e "2/2 ~ ( u ) , w' = - , j ~ e o ( U ) , v' = u 3 - - -
w'
For the purpose of evaluating E(ei) in (10), the integration can be done using
an appropriate numerical integration routine after evaluating G,._ l(x) for as many
points as the quadrature formula requires. In the numerical work reported in
Section 4, we used the Trapezoidal rule for numerical integration.
4. Numerical results
This section applies the formulas obtained in the preceding section to two
prototype systems. System A is the prototype generating system provided by the
Reliability Test System Task Force of the I E E E Power Engineering Application
of Probabilistic Methods Subcommittee [12]. Table 1 gives the assumed genera-
tion mix of the 32 units comprising the system--their installed capacities and
Table 1
Unit power ratings for a prototype generating
system, and their assumed FOR's (System A)
12 5 0.02
20 4 0.10
50 6 0.01
76 4 0.02
100 3 0.04
155 4 0.04
197 3 0.05
350 1 0.08
400 2 0.12
32
Table 2
Comparison of algorithms for LOLP estimation (System A)
Esscher's approximation
z Exact a
(MW) value (19a) (19b) (19c) Cumulants
Table 2 and Figure 2 impress one with the accuracy of Esscher's approximation
in the region of our interest, i.e., for values of LOLP in the range between 10-3
and 10-5 and beyond. There is very little difference between the three formulas,
and perhaps the formula (19b) represents the overall best choice. The cumulants
methods does not fare too badly in the probability range 10- 1 to 10- 3; but below
this range, the Esscher approximations appear to be decidedly superior to the
method of cumulants. Similar comparisons for several other systems are given in
a research report [9]. The results of this report as well as those given in [14] show
that Esscher's method, while very accurate, is also speedy enough to be adopted
in routine utility practice.
For the purpose of evaluating the accuracy of Esscher's approximation in
providing production costing expressions, we use the data provided by Caramanis
et al. [15] with respect to a second synthetic system, referred to as the EPRI
system D. Tables 3 and 4 give respectively the capacity mix of the system with
the associated FOR's and the load duration curve. Table 5 gives the derived
probability distribution (Is, rs) obtained from Table 4. Here, ls is the interval
midpoint for the j-th load class interval in Table 4, and rj is the associated
probability mass obtained from differencing. Table 6 gives the estimates capacity
factors using the three versions of Esscher's approximations using the normal,
68 M. Mazumdar
17
~ 8.1 (-6)
i1.6(-6)
18 > - 2.9(-7)
Equation (19a) : 0
Equation (19b) : 0 4.7(-8)
Equation (19c) : []
2O Cumulants : I> 7.2(-9)
21 8.4(-10)
I I I I I I I I I I I I I I I t t i I
Fig. 2. Graph of relative error for the Esscher and cumulants approximations for LOLP (system A).
Table 3
EPRI system D. Unit power ratings in loading order
1200 6 0.85320
800 1 0.85320
800 2 0.75910
600 6 0.78750
400 7 0.87420
200 56 0.92564
50 96 0.76000
a Availability -= 1 - FOR.
Table 4
EPRI system D. Description of the LDC
Table 5
Discrete version of LDC (EPRI system D)
Table 6
Comparison of algorithms for capacity factors (EPRI system D)
Esscher's approximation
Unit
no. SC-16 a P3 ~ LD-1 LD-2 LD-3 b
Table 6
(continued)
Esscher's approximation
Unit
no. SC- 16a P3 a LD- 1 LD-2 LD-3 b
References
[1] Baleriaux, E., Jamoville, E. and Fr. Linard de Guertechin (1967). Simulation de l'exploitation
d'un pare de machines thermiques de production d'61eetricit6 couples a des stations de
pompage. Revue E(SRBE ed.) 5, 3-24.
[2] Barlow, R. E. and Proschan, F. (1975). Statistical Theory of Reliability and Life Testing Probability
Models. Holt, Rinehart and Winston, New York.
[3] Billinton, R. (1970). Power System Reliability Evaluation. Gordon and Breach, New York.
[4] Billinton, R., Ringlee, R. J. and Wood, A. J. (1973). Power System Reliability Calculations. MIT
Press, Cambridge, MA.
[5] Caramanis, M., Stremmel, J. V., Fleck, W. and Daniel, S. (1983). Probabilistic production
costing. International Journal of Electrical Power and Energy Systems 5, 75-86.
[6] Cramer, H. (1946). Mathematical Methods of Statistics. Princeton University Press, Princeton, NJ.
[7] EEI Equipment Availability Task Force (1976). Report on equipment availability for the
ten-year period, 1966-1975. Edison Electric Institute, New York.
[8] Endrenyi, J. (1978). Reliability Modeling in Electric Power Systems. Wiley, New York.
[9] Electric Power Research Institute (1985). Large-deviation approximation to computation of
generating-system reliability and production costs. EPRI EL-4567, Palo Alto, CA.
[10] Esscher, F. (1932). On the probability function in the collective theory of risk. Scandinavian
Actuariedskrift 15, 175-195.
[11] Feller, W. (1971). An Introduction to Probability Theory and its Applications, Vol. II, 2nd ed.
Wiley, New York.
[12] IEEE reliability test system (1979). A report prepared by the Reliability Test System Task
Force of the Application of Probability Methods Subcommittee. IEEE Transactions on Power
Apparatus and Systems 98, 2047-2064.
[13] Levy, D. J. and Kahn, E. P. (1982). Accuracy of the edgeworth expansion of LOLP calculations
in small power systems. IEEE Transactions on Power Apparatus and Systems 101, 986-994.
[14] Mazumdar, M. and Gaver, D. P. (1984). On the computation of power-generating system
reliability indexes. Technometrics 26, 173-185.
[15] Stremmel, J. P., Jenkins, R. T., Babb, R. A. and Bayless, W. D. (1980). Production costing using
the cumulant method of representing the equivalent load curve. IEEE Transactions on Power
Apparatus and Systems 99, 1947-1953.
[16] Sullivan, R. L. (1976). Power Systems Planning, McGraw-Hill, New York.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7
© Elsevier Science Publishers B.V. (1988) 73-98
I. Introduction
In the past ten years or so, there has been considerable effort in what has been
termed software reliability modeling. The generally accepted definition of software
reliability is 'the probability of failure-free operation of a computer program in a
specified environment for a specified period of time' (Musa and Okumoto, 1982).
This area has begun to receive much attention for several reasons. Today the
computer is used in many vital areas where a failure could mean costly, even
catastrophic consequences. Due to the recent advances in hardware modeling and
technology, the main cause for computer system failure would be in the software
sector. At the other end of the spectrum, software production is costly and time
consuming, with much of the time and cost being devoted to testing, correcting
and retesting the software. The software producer needs to know the benefits of
testing and must be able to present some tangible evidence of software quality.
The issues concerning the quality and performance of software which are of
interest to the statistician (see Barlow and Singpurwalla, 1985) are:
(1) The quantification and measurement of software reliability.
(2) The assessment of the changes in software reliability over time.
(3) The analysis of software failure data.
(4) The decision of whether to continue or stop testing the software.
The problem of software reliability is different from that of hardware reliability
for several reasons. The cause of software failures is human error, not mechanical
or electrical imperfection, or the wearing of components. Also, once all the errors
are removed, the software is 100~o reliable and will continue to be so. Further-
more, unlike hardware errors there is no process which generate failures. Rather
software 'bugs' which are in the program due to human error are uncovered by
certain program inputs and it is these inputs which are randomly generated as
part of some operational environment.
73
74 T. A. Mazzuchi and N. D. Singpurwalla
A more formal discussion of the software failure process is given in Musa and
Okumoto (1982). A computer program, is a 'set of complete instructions (opera-
tions with operands specified) that executes within a single computer some major
function', and undergoes several runs, where a run is associated with 'the accom-
plishment of a user function'. Each run is characterized by its input variables
which is 'any data element that exists external to the run and is used by the run
or any externally-initiated interrupt'. The environment of a computer program is
the complete set of input variables for each run and the probability of occurrence
of each input during operation. A failure is 'a departure of program operation
from program requirements' and is usually described in terms of the output
variables which are 'any data element that exists external to the run and is set by
the run or any interrupt generated by the run and intended for use by another
program'. A fault or bug is the 'defect in implementation that is associated with
a failure'. The 'act or set of acts of omission or commission by an implementor
or implementors that results from a fault', is an error.
For a more indepth treatment of software terminology the reader is referred to
Musa and Okumoto (1982). For further clarification of types of software errors
and their causes see Amster and Shooman (1975).
Software reliability models may be classified by their attributes (Musa and
Okumoto, 1982; Shanthikumar, 1983) or the phase of the software life cycle
where they may be used (Ramamoorthy and Bastani, 1982). The later approach
will be used here. There are four main phases of the software lifecycle: testing and
debugging phase, validation phase, operational phase, and maintenance phase.
Currently no models exist for use in the maintenance phase and thus this phase
will not be discussed.
In the testing and debugging phase the software is tested for errors. In this
phase an attempt is made to correct any bugs which are discovered. The
discovery of a software bug is a function of its size (the probability that an input
will result in the bug's dis:zovery) and the testing intensity which reflects the way
in which inputs are selected. Another issue in this phase, is the treatment of the
error correction process. The more simple models assume all errors are corrected
with certainty and without introducing new errors, while other account for
imperfect debugging. Models in this phase may be classified into two main
categories: error counting models, and non-error counting models. Models may
be further classified by their approach (Bayesian or classical), their treatment of
the effect of error removal on reliability (deterministic or stochastic) and their
consideration of the time it takes to find and fix software bugs.
Software reliability models 75
where N is the initial number of software bugs and ~p is the failure rate
contribution of an individual error. This model may be used to make inference
about the software once estimates of N and ~p are obtained. Given the software
is tested and n ~< N software failures have occurred, the parameters N and q5 can
be obtained by maximum likelihood techniques. They are obtained as the simulta-
neous solution to
n
q~ = n ' (2.1.2a)
NT- 5~i=1 ( i - 1)t,
U 1 n
i=12N - i + 1
-
N---
1
~ ( i - 1)t~
, (2.1.2b)
Ti=l
parameter e(t), the exposure rate. This parameter would reflect the testing intensity
at any time. The model could thus be modified by normalizing the time between
failures as
t* =
f t t; e(u) d u
Hi_ 1
(2.1.4a)
n~ C[N/I- e(r,.)]
/-/j_ 1
(2.1.4b)
nj C[N//- e(zj)]
C= Z~=I ni , (2.1.5a)
k
Z j = 1 [ N / I - g(za.)]//j
Zk n i / [ N / I - e(zi) ]
C= i=l (2.1.5b)
Z jk= l n j
Software reliability models 77
~2
var (d) - - (2.1.6a)
2iK1 ni
12
var( ) (2.1.6b)
, n;[R/x - =
Shick and Wolverton (1973) actually specify and incorporate a testing intensity
in their modification of the JM model by assuming the failure rate of the software
is a linear function of testing time. The resulting distribution for the interfailure
times is the Rayleigh distribution
Several alterations of this have appeared. Wagoner (1973) fit a Weibull distribu-
tion to software failure data using least squares estimation for parameters. Lipow
(1974) suggested using a linear term which would be a function of the most recent
failure time. Shick and Wolverton (1978) discuss the use of a parabolic function
to model testing intensity. Sukert (1976) also adapted the model to include the
case of more than one failure occurring in a debugging interval.
Musa (1975) was the first to point out that software reliability models should
be based on execution time rather than calendar time. Musa's model is essentially
the same as the JM model but he attempts to model the debugging process in a
more realistic fashion. The model undergoes some alterations in Musa (1979).
Here, the expected net number of corrected software bugs is expressed as an
exponential function of execution time, and the fault correction occurrence rate is
assumed proportional to the failure occurrence rate.
The reliability of software tested for ~ units of execution time is
R(t) = exp { - t/T} where T, the mean time to failure (in execution time) is given
by
r = TOexp (2.1.9)
78 7". A. Mazzuchi and N. D. Singl~urwalla
In the above, TO is the mean time to failure before debugging, M o is the total
number of possible software failures in the maintained life of the software and C
is a testing compression factor. The value TO is further expressed by TO = 1 / f K N o
where f is a ratio of average instruction execution rate to the number of
instructions, called the linear execution frequency and K is an error exposure ratio
relating error exposure frequency to linear execution frequency. The value N O is
the initial number of software errors in the program and is related to M o by
M o = N o / B . The parameter B is called the fault reduction factor. This gives the
model the additional characteristic of being able to handle the possibility of more
than one error being found at one time or the possibility of imperfect debugging.
The value C is a ratio relating the rate of failures during testing to that during use.
From the parameter relationships, two central measures are obtained. The
additional number of software errors which needs to be corrected to increase the
mean time to failure for the program from T~ to T2 is given as
Am = M o To T1E1 (2.1.10)
and the additional execution time required to increase the mean time to failure
from T 1 to T2 is given as
h Z k = O k h r + l~kAm (2.1.12)
for k = I, F, C, where Az and Am are the additional execution time needed and
the additional errors corrected to bring about the change and Ok and/1~ are the
average resource expenditure rate per execution time and failure respectively.
Assuming resources remain constant throughout testing, the testing phase may be
divided into three distinct phases. In each phase only one of the resources is
limiting and the other two are not fully utilized. Thus the additional calendar time
required to increase the mean time to failure from T 1 to T2 is given as
1 I (1 1) Ok log(Tk2~] (2.1.13)
/~. are as previously defined. The quantities Tk, and Tk2 are the mean time to
failures at the boundaries of each resource limiting phase. These boundaries are
at the present and desired mean time to failure and the transition points which
lie in this range. The mean time to failure for a transition point is derived as
c [ P k , ~ , p~ - P~,,~p~, ]
Tk~,, = (2.1.14)
[P~, p~, Ok - P,p~O~, ]
for (k, k ' ) = (C, F), (F, I), (I, C). M u s a notes that it is generally true that OF = 0
and PI = 1 and discusses a method for obtaining PF by treating the failure
correction process as a truncated M / M / P F queueing model.
Most of the parameters of M u s a ' s model must be obtained from past data on
similar projects. The parameters M o and T O (and thus K and No) m a y be obtained
by using m a x i m u m likelihood techniques. The M L E ' s are obtained by solving
i-1
T o = -- 1- Ze, (2.1.15a)
n i=1 Mo
1 c
- - ze , (2.1.15b)
i=1 M o - i + 1 M o T o i=1
yielding a coefficient of variation of 1/n 1/2. Though an exact expression for the
variance of M o is not available, confidence bounds for M o are obtained using
Chebychev's inequality. Based on the distribution of the failure m o m e n t statistic
7 = M o / n - 1/A~k where A ~ = ~k(Mo + 1) - ~k(Mo + 1 - n) and qJ is the d i g a m m a
function, a (1 - ~)~o confidence interval for M o is obtained by determining the
values of M o which correspond to the values of 7 such that
7= ~ + SD(~) (2.1.17)
where
_~to 1 SD(~)= _ 1 (A~O' +
n A¢ ' (A ~k)2 \ ( A ~k)2
Sukert (1980). This model is also based on the JM assumptions but includes the
additional assumption that the severity of the testing process is proportional to
an unknown power of elapsed test time. In the ith debugging time interval of
length H;, the number of errors observed N t is given by a Poisson distribution with
mean value E [ N i ] = dp(N - M e_ I ) H i ~ where M i_ 1 is the number of errors removed
before the start of the ith debugging interval and ~ is an unknown constant. As
in the debugging scenario of Shooman it is assumed that if bugs are corrected they
are corrected at the conclusion of the debugging interval.
Parameters N, ~p and ~ may be obtained by solving the maximum likelihood
equations. In Ramamoorthy and Bastani (1982) these are given for the case
H i = ti, the time between the ( i - 1)st and ith failure, as
1 ~ ~pt~ = O, (2.1.18a)
i=1 N- Mi_ 1 i=1
n
- + ~ logti- ~ ¢p(N- M i_,)t/~ logt/= 0, (2.1.18b)
0¢ i=1 i=1
- Z ( N - M i_l)t~ = 0. (2.1.18c)
p i=l
The extra parameter gives the GPM flexibility but also difficulties in terms of
parameter estimation. Once the parameter estimates are obtained they may be
used with the model to make conclusion regarding the software. One important
expression obtained in Angus, Schafer and Sukert (1980) is the expected time
until the removal of an additional k ~<N - M faults given M faults have already
been removed. The expression is
M+k
7~k= &-'F(&) Z {~[N-i+ 11} -'/~ (2.1.19)
i=M+ I
A
where F(.) is the gamma function and &, ¢p, and N are the MLE's of ~, ¢p
and N. The use of least squares estimates is also discussed by the aforementioned
authors.
There has been much comparison and criticism of the early models in terms
of their assumptions and their parameter estimation. (See for example Forman
and Singpurwalla (1977), Shick and Wolverton (1978), Forman and Singpurwalla
(1979), Sukert (1979), Musa (1979), Littlewood (1979), Littlewood (1980a), Litt-
lewood (1980b), Angus, Schafer and Sukert (1980), Littlewood (1981a), Littlewood
(1981b), Keiller, Littlewood, Miller and Sofer (1982), Musa and Okumoto (1982),
Ramamoorthy and Bastani (1982), Stefanski (1982), Singpurwalla and Meinhold
(1983), Langberg and Singpurwalla (1985)). The paramete(estimation of the JM
model has been most criticized. Forman and Singpurwalla (1977) and (1979),
Littlewood and Verrall (1981) and Joe and Reid (1983), have all illustrated that
the solution of the maximum likelihood equations for the JM model can produce
unreasonably large even non-f'mite estimate for N. In Forman and Singpurwalla
Softwarereliabilitymodels 81
(1977) the authors found that when n is small relative to N, the likelihood function
of N is very unstable and may not have a finite optimum. Littlewood and Verrall
(1981) found that the estimate of N is finite if and only if
n
2"i=1 (i - 1)ti> Y~1=1t;
(2.1.20)
n
2i= 1 (i - 1) n
The authors note that violation of the above implies that no reliabilitygrowth
is
taking place as a result of the debugging process. In Joe and Reid (1983) the
authors show that g is an unsatisfactory point estimate because its median is
negatively biased and can be infinite with substantial probability. The authors
advocate the use of likelihood interval estimates.
Forman and Singpurwalla (1977) and (1979) develop an estimation procedure
to insure against unreasonably large estimates. They propose a stopping rule
based on the comparison of the relative likelihood function for N, to the
'approximate normal relative likelihood' for N:
where
where the unknown constants a and b represent the expected number of failures
eventually discovered and the occurrence rate of an individual error respectively.
Thus for any t >~ 0
F r o m (2.1.23) the distribution for the total error content is poim(n" a) and the
conditional distribution of the number of remaining errors at time t ' ,
N ( t ' ) : N ( o o ) - N ( t ' ) is
where t i j= l tj is the time until the ith failure. Thus in contrast to JM2,
software interfailure times are not independent. Also note that due to this
dependence, the G o e l - O k u m o t o model is of the stochastic type.
Estimators of a and b are obtained via the solution of the m a x i m u m likelihood
equations
A (1 - ~)~/o confidence region for a and b may be established using the approxi-
mation
Goel and O k u m o t o (1980) also discuss the use of the asymptotic normality of
h and b for constructing confidence intervals. Here, model results are based
Software reliabilitymodels 83
where re(v) is the mean value function for N(~). Given the above the mean value
function is given by
1
m('c) = = log(2oO'r + 1) (2.1.29)
0
and the distribution for N(,) is given by poim(n : (1/0) log(2 o 0z + 1)). Expressions
analogous to (2.1.22)-(2.1.25) are obtained by substituting (1/0)log(2 o 0~ + 1) for
a(1 - e-bt). Musa and Okumoto obtain further functions of interest by exploiting
the relationship between time until the ith failure, T" and the number of failures
in a given time. Using this notion
and
n T~
= 0 (2.1.32)
q~ i=l tp~,' + 1 (q~" + 1)(log(tpr~ + 1))
84 T. A. Mazzuchi and N. D. Singpurwalla
^ ^
In the above model, times are in terms of executime time rather than calendar
time. The conversion to calendar time follows the developments in Musa (1975).
The Musa (1975) model was also one of the first models to address the notion
of imperfect debugging. Goel and Okumoto (1979) suggested the use of a Markov
process to model imperfect debugging. Kremer (1982) uses a multidimensional
birth-death process to account for imperfect debugging and the introduction of
new errors as the result of debugging. Kremer (K) begins by assuming that the
failure rate of the software is a product of its fault content and an exposure rate,
h(t). To account for imperfect debugging he further assumes
(K) When a failure occurs, the repair effort is instantaneous and results in one
of three mutually exclusive outcomes
(i) the fault content is reduced by 1 with probability p;
(ii) the fault content remains unchanged with probability q;
(iii) the fault content is increased by 1 with probability r.
Thus the author defines a birth-death process with birth rate rh(t) and death rate
ph(t).
A multidimensional process is defined with X(t) denoting the fault content of
the software at time t and N ( t ) the number of failures to time t. Though reliability
measures are obtained from N(t), the failure rate of the software is a function of
X(t), which is changing in a stochastic manner.
Given the initial fault content of N, the expected number of faults in the
program by time t is
E [ X ( t ) IN, p, r] = N e - p(o (2.1.34)
N
p-r
E [ N ( t ) IN, p, r] = (2.1.35)
N h(u) d u , p = r.
Software reliability models 85
Thus in the life of the software (if p > r) the expected number of failures will be
N/(p - r). Thus p - r is similar to Musa's constant B. Given n failures obtained
by time to, conditional expectations may be obtained as
where p(t o, t) = p(t o + t) - p(to). Using (2.1.36) the conditional expectation for the
number of failures in (to, to + t] is
From these, the reliability of a program tested for to units of time may be obtained
as
is the reliability attribute of each remaining fault. Given n failures by time to the
reliability may be expressed as
n~
P,,,(to) = ~ - - piqJrk.
i--k=m i!j!k!
1
go(a) = - - , 0 < ~ ~< 7o, (2.1.41a)
~0
r ( k , + k2) (~ - ~ , Y " - 1(~2 - ~y,2-,
fo(/~) -
r(kl)r(k2) (/~2 -/~IY' +k2-1
O~fll "(fl(fl2 ; kl, k2)O- (2.1.41b)
For convenience it is assumed that the prior distributions for ~ and fl are
independent. Posterior inference concerning the number of future failures in an
interval or the time until the next failure may be obtained once the posterior
distributions of ~ and fl are computed. The posterior distribution of fl is of interest
in its own right as it may be used to assess the extent, of reliability growth.
Reliability growth would be taking place if fie (0, 1), by observing the posterior
density one may examine the extent to which this is true.
Posterior analysis is conducted fo~ both the case where only the number of
failures per interval and the case where the actual failure times are recorded. In
both cases the posterior distributions of ~ and/3 are intractable. An approximation
is given for the posterior of ft. Due to the intractability of the posterior
distributions of c~ and r, posterior inference concerning the number of failures in
future intervals and the time next failure are conducted numerically via a computer
code described in Kyparisis, Soyer and Daryanani (1984).
When only the number failed in each interval is recorded over a period [0, Xe]
the posterior distribution of Ark the number of failures in (xk_ 1, xk], k = p + 1,
Software reliability models 87
p + 2, p + 3, ... is given by
where gl(c~,/31nl . . . . , np) is the joint posterior density o f ~ and/3. The approximate
marginal posterior density of/3 is obtained as
where S(/3) = Y~= 1 ( x f - xf_ 1). The approximate posterior distribution for /3 is
based on the approximation
~) exp - d~ (2.1.44)
/35(/3)n~ - 1/fl
ao f12 (t n z) ok-- 1 e - v
P r { Z ~ <<.z l t l . . . . . t'} =
fo f, fo 1
(k i 1)! dv
f l + t , _ l ](N-i+l)~
..... , (2.1.49)
R ( t ils, fl) = fl + t[ 1 + ti
displays a decreasing failure rate and this property can be shown to be indepen-
dent of the prior distribution for the (Pi.
Littlewood discusses the use of (2.1.48) and (2.1.49) in determining other
reliability measures. The author suggests the use of maximum likelihood estimation
Software reliability models 89
k = 1, . . . , N, (2.1.51b)
and
~)(~)#o)kN' ~= (~H (eAN Jt -- e-(bt°+ tll)t
qN- k(t) -
j:o - (A _j -
AN- i)
(eBu-jt _ e-(f,o + u,)t
where
f AN-
ON-
k
k
½ { - [ # o +/~1 + ( N - k)~p]
Once estimates of N, tp, #o and #1 are obtained the availability of the system
is given by Y,~=oPN_k(t). The authors specify no means for estimating the
parameters, however N and tp could be estimated using methods applied to the
JM model, while #o and /~1 could be estimated from past experience or from
correction times.
D = kn k it; . (2.2.1b)
i 1
Moranda also suggests using this formulation in conjunction with the nonhomo-
geneous Poisson process. Sukert (1977) generalizes the model to include more
than on failure per debugging interval.
In Littlewood and Verrall (1973) a stochastic Bayesian model is presented. In
this approach the author attempts to model the debugging behavior of the
programmer or programmers involved. As each error is encountered, it is the
intent of programmer to correct the error and thus increase the reliability of the
software. Though this is always the intent it is not always achieved. Often new
errors are created which reduce the reliability of the software. To model this
situation in a Bayesian context, Littlewood suggests expressing the uncertainty
about 2; by assuming a priori that 2 i ~ G(a, ~(i)) where 0(i) is an increasing
function indicating the complexity of the program and quality of the programmer.
Defining 0(i) as an increasing function of L incorporates the assumption that the
programmer's intent is always to improve the software's reliability since
0(i) > 0 ( i - 1) implies that
Software reliability models 91
for l > 0. The above implies that the ).i are stochastically ordered.
Combining the usual assumption that given 2; the variables T;, i = 1. . . . , n, are
independent exponential random variables, with the prior distributions for 2; the
posterior reliability for T~ can be obtained as
T; = Ti~ , bi (2.2.4)
where 0; reflects the effects of the changes made as a result of the (i - 1)st stage
of testing and bI is an error term to account for uncertainty. Note that reliability
growth will have taken place as a result of changes made in the (i - 1)st stage
of testing if 0; > 1; 0; = 1 indicates no improvement and 0; < 1 indicates reliability
decay.
The model is developed based on the following assumptions:
(HSS1) The variables Ti, i = 1. . . . . n, are lognormally distributed with T;~< 1
assumed for all i.
(HSS2) The values b;, i = 1, . . . , n, are lognormally distributed with known
parameters 0 and a 2.
(HSS3) The quantities 0i, i = 1. . . . , n, are exchangeable and are distributed
according to some distribution G with density g.
Taking the logarithm of both sides of (2.2.4) yields
r, -- o , r ; _ 1 + ( 2.2.5)
92 T. A. M a z z u c h i and N. D. Sin~ourwalla
where Y~= log T~ and e; = log ~ are normally distributed, the latter with mean 0
and variance alz. The sequence { Y,.} is thus given by a first order autoregressive
process with a random coefficient 0,..
By assuming further that 0~ ~ N(2, a22) where a ff is known and 2 ~ N(#, a 2)
with # and a32 known, the following posterior results are obtained.
#n=(~232+i=1
~ Y i Y i -,l/] a f f
__
ay = 1 ÷ Yi- 1 /-'
i= 1 Wi _ 1.t
Wi- 1 = a~Yi- 1 + a~ ;
Note that aft reflects the views about the consistency of policies regarding modifi-
cations and design changes made. Using the above, posterior inference can be
obtained for any relevant quantity. For example Bayes probability intervals can
be constructed for the next failure time or reliability growth at each stage can be
assessed by plotting E[Oilyl, . . . , yi] vs. i. Overall, reliability growth can be
examined via E[2Iy~, . . . , y~], i = 1 . . . . . n. In Singpurwalla and Soyer (1985) this
basic model is extended by assuming various dependence structures for the
sequence {0;}. Three additional models are developed using the structure of the
Kalman Filter Model.
When a decision is made to stop testing the software (see Forman and
Singpurwalla, 1977; Okumoto and Goel, 1979, 1980; Krten and Levy, 1980;
Shanthikumar and Tufekci, 1981, 1983; Koch and Kubat, 1983; Chow and
Schechner, 1985, for decision criteria), the software enters t.he validation phase.
In this phase the software undergoes intensive testing in its operational environ-
ment with a goal of obtaining some measurement of its reliability. Software errors
are not corrected in this phase and, in fact, a software failure could result in the
rejection of the software.
Nelson (1978) introduced a simple reliability estimate based on probabilistic
laws. Letting e r denote the size of the remaining errors in the program and noting
that errors are not removed, the number of runs until a software failure is a
geometric random variable with parameter e r. Thus the maximum likelihood
estimate of e r can be used to determine an estimate of reliability. This is given
as
R = 1 - nf/n (3.1)
where n is the total number of sample runs and nf is the number of sample runs
which ended in failure.
The above model suffers from several drawbacks (Ramamoorthy and Bastani,
1982) stemming from its simplicity.
(1) A large number of sample runs is required to obtain meaningful estimates.
(2) The model is based on the assumption that inputs are randomly selected
from the input domain and thus does not consider the correlation of runs from
adjacent segments of the input domain.
(3) The model does not consider any measure of complexity of the program.
Extensions to the basic model have attempted to reduce the number sample
runs by specifying equivalence classes for the input domain (Nelson, 1978;
Ramamoorthy and Bastani, 1979). This goal is achieved at the cost of an increase
in model complexity.
Crow and Singpurwalla (1984) address the issue of correlation of inputs using
a fourier series model. The authors observe that in many cases software failures
occur in clusters and thus the usual assumption that the times between failures
are independent may not be valid. Rather they assume that the time between
failures is given by
T i = f(i) + ~ (3.2)
where ee is a disturbance term with mean 0 and constant variance and f(i) is some
cyclical trend. To identify the cyclical pattern (if any) with which failures occur,
the authors fit the Fourier series model
1
ao = - ~ ti, (3.4a)
/'/ i = 1
~(kj) = -2 5] t ; c o s -2~
- kfi, j = 1. . . . . q, (3.4b)
ni=l n
]~(kj)= 2 ~ t i s i n 2 n k f i , j= 1. . . . . q. (3.4c)
ni=l n
The spectrogram is used to identify the period of the series, and thus the clustering
behavior. A parsimonious model may also be obtained by using only those
weights ~(kj) and/~(kj) for which p2(kj) = a2(ki) + fl2(kj) is large.
This model was applied to three sets of failure data from each of two software
systems. The model was found to adequately represent the failure behavior. One
potential problem of the model is that due to the relationship of a(kj) and/~(kj)
on trigonometric functions, negative values of f ( i ) may be produced. When such
is the case, the authors interpret this as an implication of a very small time
between failure.
Though the intent of the authors in this paper is data analysis, the model can
be used to predict future time between failures and future failure clusters. Also by
specifying a functional form for ee (such as the usual normal assumption),
inference can be made.
Models in this phase are used to illustrate the behavior of the software in its
operating environment. Both Littlewood (1979) and Cheung (1980) obtain the
software reaiiability by assuming the software program is divided into modules.
Cheung suggests a combination of deterministic properties of the structure of
he software with the stochastic properties of module failure behavior, via a
Markov process. He assumes
(C1) Reliabilities of the modules are independent.
(C2) Transfer of control among program modules is a Markov process.
(C3) The program begins and ends with a single module, denoted N 1 and Nn
respectively.
The state space is divided into N 1. . . . . Nn, C, F where N; are the modules, C
indicates successful completion, and F indicates an encountered failure. States C
and F are absorbing. Transition probabilities from N; to Nj (i # j ) are given by
Software reliability models 95
R~p~j where Ri is the reliability of module i and p~j is the usual transition
probability from module i to module j. The transition probability from Ni to F is
1 - R ; and the transition probability from N n to C is given by R,. Thus the
reliability of the software is obtained as the probability of being absorbed into
state C given that the initial state is N 1. This is obtained as
where S(i, j) is the (i, j)th entry in the matrix S = ( I - Q)-1 and Q is the
transition matrix of the process with the rows and columns of C and F deleted.
The module reliabilities R~ may be determined before system integration by
techniques of Section 2 or 3. Transition probabilities may be estimated by running
test case. Cheung further discusses the use of this module in determining testing
strategies and expected error cost of the software. The latter may be used in place
of system reliability in determining the acceptance of the software.
Littlewood (1979) assumes semi-Markov process and takes into account the
time spent in each module. The model further incorporates two sources of failure:
within module failure with rate 7~, i = 1. . . . , n, and failure associated with the
transfer from module i to module j which is given with rate 2;j (i # j ) . Assuming
that these individual failure rates are small in comparison to the switching rates
between modules, Littlewood states that the failure point process of the integrated
program is asymptotically a Poisson proces s with rate parameter
5. Closing comments
Though there is a large body of literature on software reliability (see Shick and
Wolverton, 1978; Ramamoorthy and Bastani, 1982; Shanthikumar, 1983) several
96 T. A. Mazzuchi and N. D. Singpurwalla
issues remain. First, there is a lack of models for the validation, operational and
m a i n t e n a n c e phase of the software. Additional models are needed to address such
issues as software design and testing criteria for release of software. Furthermore,
the vast n u m b e r of models for the testing and development phase has left the user
somewhat confused. Criteria for c o m p a r i s o n and selection of software models
needs to be developed as is done initially in M u s a and O k u m o t o (1982), Kieffer,
Littlewood, Miller and Sofer (1982) a n d I a n n i n o , Musa, Okumoto, Littlewood
(1984), and Soyer and Singpurwalla (1985).
References
prediction. In: J. K. Skwirzynski, ed., Electronic Systems Effectiveness and Life Cycle Costing.
Springer, New York, 441-460.
Kim, J. H., Kim, Y. H. and Park, C. J. (1982). A modified Markov model for the estimation of
computer software performance. Operations Research Letters 1, 253-257.
Koch, H. S. and Kubat, P. (1983). Optimal Release Time of Computer Software. IEEE Transactions
on Software Engineering 9, 323-327.
Kremer, W. (1983). Birth-death and bug counting. IEEE Transactions on Reliability 32, 37-46.
Krten, O. J. and Levy, J. (1980). Software modeling from optimal field energy. Proceedings of the
Annual Reliability and Maintainability Symposium, 410-414.
Kyparisis, J. and Singpurwalla, N. D. (1984). Bayesian inference for the Weibull process. In: L.
Bilard, ed., Computer Science and Statistics; Proceedings of the 16th Symposium on the Interface.
North-Holland, Amsterdam, 57-64.
Kyparisis, J., Soyer, R. and Daryanani, S. (1984). Computer programs for inference from the Weibull
process. Institute for Reliability and Risk Analysis Technical Report, The George Washington
University, Washington, DC.
Langberg, N. and Singpurwalla, N. D. (1985). Unification of some software reliability models via the
Bayesian approach. SIAM Journal on Scientific and Statistical Computing 6, 781-790.
Lipow, M. (1974). Some variations of a model for software time-to-failure. Correspondence ML-74-
2260.1, TRW Systems Group.
Littlewood, B. (1979). How to measure software reliability and how not to. IEEE Transactions on
Reliability 28, 103-110.
Littlewood, B. (1979). Software reliability model for modular program structure. IEEE Transactions
on Reliability 28, 241-246.
Littlewood, B. (1980). The Littlewood-Verral model for software reliability compared with some
rivals. The Journal of Systems and Software 1,251-258.
Littlewood, B. (1980). Theories of software reliability: How good are they and how can they be
improved. IEEE Transactions on Software Engineering 6, 489-500.
Littlewood, B. (1981). A critique of the Jelinski-Moranda model for software reliability. Proceedings
of the 1981 Annual Reliability and Maintainability Symposium, 357-362.
Littlewood, B. (1981). Stochastic reliability growth: a model for fault-removal in computer-programs
and hardware design. IEEE Transactions on Reliability 30, 313-320.
Littlewood, B. and Veri'all, J. L. (1973). A Bayesian reliability growth model for computer software.
Applied Statistics 22, 332-346.
Littlewood, B. and Verrall, J. L. (1981). Likelihood function of a debugging model for computer
software reliability. IEEE Transactions on Reliability 30, 145-148.
Meinhold, R. J. and Singpurwalla, N. D. (1983). Bayesian analysis of a commonly used model for
describing software failures. The American Statistician 32, 168-173.
Moranda, P. B. (1975). Prediction of software reliability during debugging. Proceedings of the 1981
Annual Reliability and Maintainability Symposium, 327-332.
Moranda, P. B. (1979). Event-altered rate models for general reliability analysis. IEEE Transactions
on Reliability 28, 376-381.
Musa, J. D. (1975). A theory of software reliability and its application. IEEE Transactions on Software
Engineering 1, 312-327.
Musa, J. D. (1979). Validity of execution-time theory of software reliability. IEEE Transactions on
Reliability 28, 181-191.
Musa, J. D. and Okumoto, K. (1982). Software reliability models: Concepts classification, compari-
sons, and practice. In: J. K. Skwirzynski, ed., Electronic Systems Effectiveness and Life Cycle Costing.
Springer, New York, 395-423.
Musa, J. D. and Okumoto, K. (1984). A logarithm Poisson execution time model for software
reliability measurement. Proceedings of the 1984 Reliability and Maintainability Symposium.
Nelson, E. (1978). Estimating software reliability from test data. Microelectron. Reliab. 17, 67-74.
Okumoto, K. and Goel, A. L. (1979). Optimal release time for software systems. Proceedings of
COMPSAC, 500-503.
98 T. A. Mazzuchi and N. D. Singpurwalla
Okumoto, K. and Goel, A. L. (1980). Optimal release time for software systems based on reliability
and cost criteria. Journal of Systems and Software 1, 315-318.
Petroski, C. M. (1984). A survey of software reliability. Student Report, The George Washington
University.
Ramamoorthy, C. V. and Bastani, F. B. (1979). An input domain based approach to the quantitative
estimation of software reliability. Proceedings of the Taipei Seminar on Software Engineering, Taipei,
Taiwan.
Ramamoorthy, C. V. and Bastani ,F. B. (1980). Modeling the software reliability growth process.
Proceedings of COMPSAC, Chicago, IL, 161-169.
Ramamoorthy, C. V. and Bastani, F. B. (1982). Software reliability-status and perspectives. IEEE
Transactions on Software Engineering 8, 354-371.
Schick, G. J. and Wolverton, R. W. (1978). An analysis of competing software reliability models.
IEEE Transactions on software Engineering 4, 104-120.
Schick, G. J. and Wolverton, R. W. (1973). Assessment of software reliability. Proceedings Operations
Research, Physica, Werzberg-Wein, 395-422.
Schneidewind, N. F. (1975). An analysis of computer processes in computer software. Proceedings
of the International Conference on Reliable Software, 337-346.
Shanthikumar, J. G. (1981). A general software reliability model for performance prediction.
Mircoelectron. Reliab. 27, 671-682.
Shanthikumar, J. G. (1983). Software reliability models: A review. Microelectron. Reliab. 23, 903-943.
Shanthikumar, J. G. and Tufekci, S. (1981). Optimal release time using generalized decision trees.
Proceedings of the Fourteenth Annual Hawaii International Conference on System Sciences, 58-65.
Shanthikumar, J. G. and Tufekci (1983). Application of a software reliability model to describe
software release time. Microelectron. Reliab. 23, 41-59.
Shooman, M. L. (1972). Probabilistic models for software reliability prediction. In: W. Freiberger,
ed. Statistical Computer Performance Evaluation. Academic Press, New York, 485-502.
Shooman, M. L. (1973). Operational testing and software reliability estimation during program
development. Record of the 1973 IEEE Symposium on Computer Software Reliability, 51-57.
Shooman, M. L. (1975). Software reliability: Measurement and models. Proceedings of the 1975
Annual Reliability and Maintainability Symposium, 485-489.
Shooman, M. L. and Trivedi, A. K. (1976). A many state Markov model for computer software
performance parameters. IEEE Transactions on Reliability 25, 66-68.
Singpurwalla, N. D. and Soyer, R. (1985). Assessing (software) reliability growth using a random
coefficient autoregressive process and its ramifications. To appear in IEEE Transactions on Software
Engineering.
Sukert, A. N. (1977). An investigation of software reliability models. Proceedings of the Annual
Reliability and Maintainability Symposium, 78-84.
Sukert, A. N. (1979). Empirical validation of three software prediction models. IEEE Transactions on
Reliability 28, 199-205.
Stefanski, L. A. (1982). An application of renewal theory to software reliability. Proceedings of the
Twenty-Seventh Conference on the Design of Experiments in Army Research Development Testing. ARO
Report 82-2, 101-118.
Wagoner, W. L. (1973). The final report on a software reliability measurement study. Report TOR-
0074-(41221)-1, The Aerospace Corp., El Segundo, CA.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 ~,~
© Elsevier Science Publishers B.V. (1988)99-111
1. Introduction
Part I
The first part of our study will consist of the effects of dependence on the
classification of life distributions according to the properties of aging. Most of
these concepts originate in the bivariate case and due to its importance and
simplicity we will study this case in more detail. Major source for the material
covered in this part consists of the articles by Freund (1961), Harris (1970),
Brindly and Thompson (1972), Shaked (1977) and the book by Barlow and
Proschan (1981).
1.1. Definitions
Let (X, Y) be a pair of real valued random variables defined on a fixed proba-
bility space. The joint distribution function and the marginals of (X, Y) will be
denoted by Fx, v, Fx and F r and the corresponding density functions by fx, r, f x ,
f y respectively. We write I(A) for the indicator of an event A. Many of the
concepts of positive and negative dependence can be defined in terms of condi-
tions on covariances of functions restricted to certain classes. Thus conditions
99
100 N. R. Chaganty and K. Joag-dev
The condition of 'association' for X = (X1, ... , Xk) , which is stronger than
P U O D and PLOD is given by
Similar bound can be obtained for a parallel system. These two bounds can be
combined to obtain bounds for a general coherent system.
An analog of TP z dependence given in (e) above is obtained by imposing this
condition on every pair of the arguments of the joint density in •k, while other
arguments are kept fixed. This condition known was MTP z implies association
(see for example Barlow and Proschan, 1981).
Finally, some of these conditions with appropriate changes, may be used to
define negative dependence. For example, see Block, Savits and Shaked (1982)
and Joag-dev and Proschan (1983). The components of a vector
X = (Xa, X2, . . . , Xk) are negatively associated if for every nonempty subset A of
{ 1, 2, ..., k} and every pair of co-ordinatewise nondecreasing functions g and h,
the Cov(f(XA), g(X~)) is nonpositive, where A- denotes the complement of A.
Negative dependence is relevant in systems defined in closed environments. For
example, a given number of species competing in an ecosystem with a fixed
amount of resources, may have their life lengths negatively associated.
m(x) =
f; tf(t) dt/ff(x) = i(t) dt/i(x). (1.8)
The life distribution F is said to be increasing mean residual life (IMRL) if m(x)
is increasing in x >~ 0. We say F is decreasing mean residual life (DMRL) if m(x)
is decreasing in x >~ 0. To obtain the monotone behaviour of the conditional mean
residual life mo(x]y), it can be shown that it suffices to have
h(x, y) =
fx (t - x)fc(tly) dt (1.9)
be TP 2. Again it can be shown that this condition is weaker than that needed for
the monotonicity of r(xlc). These results and some extensions were derived by
Shaked (1977). In the same article, Shaked (1977) also introduced the concept of
dependence by total positivity (DTP) for bivariate distributions. Recently Lee
(1985a) generalized the DTP concepts to the multivariate case and obtained a
number of inequalities and monotonicity properties of conditional hazard rate and
mean residual life functions of some multivariate distributions satisfying the DTP
property. In a subsequent paper Lee (1985b) introduced the concept of depen-
dence by reverse regular (DRR) rule, which is the mirror image of DTP, and
studied the relationship of DRR with other concepts of negative dependence.
Harris (1970) defined IHR (increasing hazard rate) property for a multivariate
distribution by requiring
(a) ff(x + t 1)~if(x) nonincreasing in x, and
(b) P [ X > u IX > x] nondecreasing in x for every fixed vector u. (1.10)
Geometric interpretation of (b) has prompted its name 'right comer set increasing'
(RCSI). Condition (a) is clearly 'wear out' condition, while as we shall see, (b)
describes positive dependence.
Brindley and Thompson (1972) studied the class of distributions where only (a)
is satisfied. In order to distinguish between these two classes based on aging
property, one satisfying (a) is called IFR, while the subclass with the additional
requirement of (b) is called IHR (H is for hazard or Harris!). In both cases the
classes can be seen to be closed under (a) taking subsets (b) unions of indepen-
dent sets of variables (c) taking minimums over subsets. Note that ir,Jportance of
the minimums stems from its role in the series systems. Both definitions, when
restricted to univariate, yield the usual IFR distribution. For the univariate case
(b) is trivially satisfied.
Dependence notions in reliability theory 103
To see that RCSI implies positive dependence, let K and M be arbitrary subsets
(not necessarily disjoint) of { 1, 2 . . . . . n}. Denoting appropriate subvectors by x K
and xM etc., it can be seen readily that (1.10b) implies that
which is PUOD.
It would be worthwhile to mention examples of distributions where the above
dependency concepts are manifested in a natural way. If the components are
independent, then most of the conditions are trivially satisfied and hence we
consider those having dependent components.
Let U, X~, X 2 be independent random variables. Consider Y1 = min(U, X1),
I12 = min(U, X2), such functions determine the life of a system where the com-
ponent corresponding to U is connected in series. These functions are also impor-
tant when U represents the arrival time of a shock which disables components
corresponding to X~, X 2. This model, when U, Xa, X 2 each has exponential
distribution, was studied by Marshall and Olkin (1967). They also studied its
multivariate analog where different shocks disable 2, 3. . . . , n components. It
should be noted that the property of association is preserved due to the fact that
minimum of random variables is a co-ordinatewise increasing functions.
Gumbel (1960) discussed a simple model with bivariate distribution where its
survival function is given by
m
where 0 ~< b ~< 1. It is clear that the marginals are exponential and since X has
negative regression dependence, it is only appropriate when two variables have
such dependence. Freund (1961) describes a bivariate model of a two component
system where the joint survival function is same as that of two independent
exponentially distributed random variables with shape parameters e and B, as long
as both components have not failed. Upon failure of one item the shape
parameter of the life distribution of the other component is changed to e I or
changed to ill. The joint survival probability function can be written as
"-i
e e x p ( - fll(y - x ) ) / , x~y
+ (e +/~ - ¢~') 3
104 N. R. Chaganty and K. Joag-dev
~--CZ 1
= exp(-(o~ + ]~)x)[ioc exp( - (~ + / / ) ( y - x))
q
exp(-eX(y- x))[, y~< x . (1.14)
+ (~ +/~ _ ~1) A
The marginal distributions are not exponential but are certain mixtures of
exponentials and the nature of dependence is determined by the relative magni-
tudes of the parameters. In fact,
(1.16)
It is easy to verify that Fl(x) is IFR if and only if ~ < ~1 and F2(y) is IFR if
and only if fl < ill.
Part II
The second part of our study deals with dependence concepts relevant to the
models which consider repair and replacement of the components of a system.
These dependent concepts arise from the study of the theory of stochastic pro-
cesses. Some of the classical types of stochastic processes characterized by differ-
ent dependence relationships are Markov processes, Renewal processes and
Markov renewal processes. The latter includes the previous two as special cases.
The dependent relations such as total positivity, association, stochastic monotoni-
city studied in Part I, have natural occurrence among these processes. It is
needless to say that the vast number of results in the study of the above processes
have wide applications in reliability theory. In the next few sections we shall
examine some of these processes and their applicability in characterizing the
failure rate of the life distributions of systems, as well as in obtaining bounds of
some other quantities of interest in reliability theory. The organization of this part
is as follows: In Section 2.1, we define totally positive Markov process and
discuss some useful theorems related to this process. A concept weaker than
totally positivity is stochastic monotonicity, that is, all totally positive Markov
processes are stochastically monotone but not vice versa. This is discussed in
Section 2.2.
Dependencenotionsin reliabilitytheory 105
is strictly positive when t > 0 for all n >~ 1. If (2.3) holds for n ~< r, we say that the
Mmkov process is totally positive of order r (TPr).
When the state space S is a countable set and the parameter set is the set of
integers, the Markov process is known as a Markov chain. The Markov chain is
said to be time-homogeneous if the transition function Pn is independent of n, in
which case we simply write P. The Markov chain is totally positive if P satisfies
condition (2.3). Karlin and McGregor (1959a, b) have shown that, indeed several
Markov chains and Markov processes are totally positive, the prominant one
being the birth and death process.
An excellent treatise of totally positive Markov chains and totally positive
Markov processes together with applications in several domains of mathematics,
including reliability theory, is given in Karlin (1964). Typical of the results of
Karlin (1964) are the following theorems regarding inheritance of TP character.
106 N. R. Chaganty and K. Joag-dev
The TP property is also prevalent when the initial state of the Markov chain
is fixed. We state this in the theorem below.
The above Theorem 4 was used by Brown and Chaganty (1983) to show that
the first passage time distribution from an initial state to a higher state in a birth
and death process is IFR. This result was also obtained by Keilson (1979),
Derman, Ross and Schechner (1979) using other methods. Another application of
Theorem 4 is given by Assaf, Shaked and Shanthikumar (1985). They have
shown that the time to failure of some systems which are subject to shocks and
damages, which are not necessarily nonnegative, is IFR.
THEOREM 8. Every I F R A distribution in discrete time & either the first passage
time distribution to an increasing set for a stochastically monotone Markov chain with
monotone paths on a partially ordered finite set, or the limit of a sequence of such
distributions.
Analogous theorems in the continuous time frame also hold. The above
theorems were used by Brown and Chaganty (1983) to show that the convolution
of two I F R A distributions is IFRA. Various other applications of the above
theorems to shock models in reliability theory, sampling with and without replace-
ment can also be found in Brown and Chaganty (1983).
Stochastically monotone Markov chains also take an important place in
obtaining optimum control limit rules. The following formulation is due to Derman
(1963). Suppose that a system is inspected at regular intervals of time and that
after each inspection it is classified into one of (m 4- 1) states denoted by 0, 1,
2 . . . . . m. A control limit rule l simply says that replace the system is the observed
state is one of the states k, k + 1, . . . , m for some predetermined state k. The
state k is called the control limit of l. Let X n denote the observed state of the
system in use at time n >/0. We assume that {X~, n ~> 0} is a stationary Markov
chain. Let c ( j ) denote the cost incurred when the system is in state j. Let L
denote the class of all possible control limit rules. For l ~ L , the asymptotic
108 N.R. Chagan~ and K. Joag-dev
,n
expected average cost is defined as A(I) = l i m , _ ~ 1/n ~,= 1 c(X,). The following
theorem was proved by Derman (1963).
THEOREM 9. Let the Markov chain {X~, n >/0} be stochastically monotone. Then
there exists a control limit rule l* such that
The reverse inequalities in the above theorem are valid for F new worse than
used (NWU), that is, ff(x + y) >1 F(x)F(y), for all x, y/> 0. In a two paper series
Brown (1980, 1981) obtained nice properties for the renewal function M(t) when
the underlying distribution F is assumed to be D F R or IMRL. Let
Z(t) = S N ( t ) + 1 - - t denote the forward recurrence time at time t and A(t) = t - SN,,
the renewal age at t. The following theorem can be found in Brown (1980, 1981).
THEOREM 12. (a) I f the underlying distribution F of the renewal process is DFR,
then the renewal density M ' ( t ) exists on (0, ~ ) and is decreasing, that is, M(t) is
concave. Furthermore, Z(t), A(t) are both stochastically increasing in t >/O.
(b) I f F is I M R L then M ( t ) - t/l~ is increasing in t>~ 0 and E[~b(Z(t))] is
increasing in t >/0 for increasing convex functions ~.
In the case where F is IMRL, Brown (1981) provides counter examples to show
that Z(t) is not necessarily stochastically increasing, E[A(t)] not necessarily
increasing and M(t) need not to be concave. An example of Berman (1978) shows
that the analogous results do not hold for I F R and D M R L distributions. As an
application of Theorem 12, Brown (1980) obtained sharp bounds for the renewal
function M(t) for F I M R L , with improved bounds for F DFR. These results are
given in the next theorem.
THEOREM 13. Let Pn = E(X~'), n ~> 1. Let U(t) = t/Izl +/~2/2#12. Let #K+2 be
finite for some k ~ O. I f F is I M R L then
References
Assaf, D., Shaked, M. and Shanthikumar, J. G. (1985). First passage times with PF r densities.
Journal of Appl. Prob. 22, 185-196.
Barlow, R. E. and Proschan, F. (1964). Comparison of replacement policies, and renewal theory
implications. Ann. Math. Statist. 35, 577-589.
Barlow, R. E. and Proschan, F. (1981). Statistical Theory of Reliability and Life Testing. To Begin With,
Silver Spring, Maryland.
Berman, M. (1978). Regenerative multivariate point processes. Adv. Appl. Probability 10, 411-430.
Block, H. W., Savits, T. H. and Shaked, M. (1982). Some concepts of negative dependence. Ann.
of Probability 10, 765-772.
Brindley, E. C. Jr. and Thompson, W. A. Jr. (1972). Dependence and aging aspects of multivariate
survival. Journal of Amer. Stat. Assoc. 67, 822-830.
Brown, M. (1980). Bounds, inequalities, and monotonicity properties for some specialized renewal
processes. Ann. of Probability 8, 227-240.
Brown, M. (1981). Further monotonicity properties for specialized renewal processes. Ann. of P,oba-
bility 9, 891--895.
Brown, M. and Chaganty, N. R. (1983). On the first passage time distribution for a class of Markov
Chains. Ann. of Probability 11, 1000-1008.
Cox, D. R. (1962). Renewal Theory. Methuen, London.
Daley, D. J. (!968). Stochastically monotone Markov chains. Z. Wahrsch. verw. Gebiete 10, 305-317.
Derman, C. (1963). On optimal replacement rules when changes of state are Markovian. In: Richard
Bellman, ed., Mathematical Optimization Techniques. Univ. of California Press, 201-210.
Derman, C., Ross, S. M. and Schechner, Z. (1979). A note on first passage times in birth and death
and negative diffusion processes. Unpublished manuscript.
Esary, J. D., Marshall, A. W. and Proschan, F. (1973). Shock models and wear processes. Ann. of
Probability 1, 627-649.
Esary, J. D., Proschan, F. and Walkup, D. W. (1967). Association of random variables, with
applications. Ann. Math. Stat. 38, 1466-1474.
Feller, W. (1966). An Introduction to Probability Theory and lts Applications, Vol. II. Wiley, New York.
Freund, J. E. (1961). A bivariate extension of the exponential distribution. Journal of Amer. State.
Assoc. 56, 971-977.
Gumbel, E. J. (1960). Bivariate exponential distributions. Journal of Amer. Star. Assoc. 55, 698-707.
Harris, R. (1970). A multivariate definition for increasing hazard rate distribution functions. Ann.
Math. Statist. 41, 713-717.
Joag-dev, K. and Proschan, F. (1983). Negative association of random variables with applications.
Ann. Statist. 11, 286-295.
Karlin, S. (1964). Total positivity, absorption probabilities and applications. Trans. Amer. Math. Soc.
Dependence notions in reliability theory 111
III, 33-107.
Karlin, S. and McGregor, J. (1959a). Coincidence properties of birth and death processes. Pacific
Journal of Math. 9, 1109-1140.
Karlin, S. and McGregor, J. (1959b). Coincidence probabilities. Pacific Journal of Math. 9, 1141-1164.
Karlin, S. and Taylor, H. M. (1975). A First Course in Stochastic Processes, 2nd edition. Academic
Press, New York.
Kalmykov, G. I. (1962). On the partial ordering of one-dimensional Markov processes, Theor. Prob.
Appl. 7, 456-459.
Kamae, T., Krengel, U. and O'Brien, G. C. (1977). Stochastic inequalities on partially ordered
spaces. Ann. of Probability 5, 899-912.
Keilson, J. (1979). Markov Chain Models--Rarity and Exponentiality. Springer, New York.
Kirstein, B. M. (1976). Monotonicity and comparability of time homogeneous Markov processes with
discrete state space. Math. Operations Forschung. Stat. 7, 151-168.
Lee, Mei-Ling Ting (1985a). Dependence by total positivity. Ann. of Probability 13, 572-582.
Lee, Mei-Ling Ting (1985b). Dependence by reverse regular rule. Ann. of Probability 13, 583-591.
Marshall, A. W. and Proschan, F. (1972). Classes of distributions applicable in replacement, with
renewal theory implications. Proceedings of the 6th Berkeley Symposium on Math. Stat. and Prob.
I. Univ. of California Press, Berkeley, CA, 395-415.
Marshall, A. W. and Olkin, I. (1967). A multivariate exponential distribution. Journal ofAmer. Stat.
Assoc. 62, 30-44.
Obretenov, A. (1974). An estimation for the renewal function of an IFR distribution. In: Colloq.
Math. Soc. Janos Bolyai 9. North-Holland, Amsterdam, 587-591.
O'Brien, G. (1972). A note on comparisons of Markov processes. Ann. of Math. Stat. 43, 365-368.
Shaked, M. (1977). A family of concepts of dependence for bivariate distributions. Journal of Amer.
Stat. Assoc. 72, 642-650.
Shaked, M. and Shanthikumar, J. G. (1984). Multivariate IFRA properties of some Markov jump
processes with general state space. Preprint.
Smith, W. L. (1958). Renewal theory and its ramifications. J. Roy. Statist. Sot., Series B 20, 243-302.
Veinott, A. F. (1965). Optimal policy in a dynamic, single product, nonstationary inventory model
with several demand classes. Operations Research 13, 761-778.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 '7
/
© Elsevier Science Publishers B.V. (1988) 113-120
B. W. W o o d r u f f a n d A. H. M o o r e
1. Introduction
2. Z2 goodness-of-fit tests
113
114 B. W. Woodruff and A. H. Moore
Take a random (or censored) sample from the unknown distribution and divide
the support set into a set of k subsets. Now under the null hypothesis, determine
the expected number of observations in each subset denoted by E i (i = 1. . . . . k).
The observed number of sample observations in each subset is denoted by O,. A
usual rule is to choose the subsets so that the expected number of observations
in each subset is greater than or equal to 5. The test statistic is
k ( O i - Ei)2
i=1 Ei
3. Graphical techniques
To use a modified goodness-of-fit test based on the EDF, one has to choose
a family of cdfs of the form F [ ( x - c)/O] where c is a location parameter and 0
is a scale parameter. The estimators of the nuisance parameters must be scale and
location invariant. Usual estimators having this property are maximum likelihood
estimators. When the estimators are inserted in the cdf, we will denote the
cdf evaluated at each order statistic under the null hypothesis Fo[(X i - d ) / 0 ]
by t0i.
Consider the following test statistics:
(i) The Kolmogorov-Smirnov statistic /£:
/£ = max(D +, D - ) ,
where D + = 1.u.b. (i/n - P i ) ,
l <~ i <~ n .
D = 1.u.b.(F, - [ ( i - 1)/n]),
I?=D+ +D-.
When the parameters are estimated by location and scale estimators, then the
null distribution of the test statistic and hence its percentage points do not depend
on c and 0. However in using the tables, one must use the same estimators as
were used in the construction of the table. The table of critical values and the
power of the test is affected by the invariant estimators chosen.
study was done for seven altemate distributions. It was shown that the CvM test
had the highest power for most of the alternative distributions studied when the
null hypothesis was the two parameter negative exponential.
Woodruff et al. [24] and Bush et al. [2] also derived tables for the Rayleigh
distribution (Weibull shape parameter 2.0) for the same sample sizes and signifi-
cance levels given above.
The papers by Woodruff and Bush also studied a range of other Weibull shape
parameters from 0.5(0.5)4.0. A second power study with seven alternate distribu-
tions showed that the AD statistic was the most powerful when the null distribu-
tion was a Weibull with shape parameter 3.5. A relationship between critical
values and the inverse of the shape parameter was presented for the range of
shape parameters studied.
A power study indicated that for larger sample sizes, the CvM was the most
powerful of the three tests. The equation C = a o + ~l(1/fl 2) describes the form of
the relationship between the critical values C and the shape parameter fl derived
for each of the statistics studied. Again ML estimators were used.
Woodruff et al. [26] derived tables of critical values for the modified KS, AD
and CvM goodness-of-fit statistics for the logistic distribution with unknown
shape and location parameters. ML estimators were used to obtain estimates of
the unknown parameters. The statistics were tabled for sample sizes n = 5(5)30
and significance levels ~ = 0.1, 0.5(0.5)0.20. A power study indicated quite good
power against uniform and exponential alternatives. The modified KS test had
lower power than the other two tests studied.
Porter and Moore [20] derived tables of critical values for the modified KS,
AD, and CvM goodness-of-fit statistics for the Pareto distribution with unknown
location and scale parameters and known shape parameters. Best linear unbiased
estimators were used to obtain the parameter estimates. The critical values were
tabled for sample sizes n = 5(5)30, significance levels ~ = 0.1, 0.5(0.5)2.0 and
Pareto shape parameters 0.5(0.5)4.0. The powers were investigated for eight alter-
native distributions. A functional relation between the critical values of test
statistics and the Pareto shape parameters was derived.
When a goodness-of-fit test fails to reject two families of distributions, one can
use a likelihood ratio test to discriminate between them. Bain [1] ~ves an
extensive coverage to likelihood ratio tests. H e lists the test statistic to be used
to discriminate between normal vs. two-parameter exponential, normal vs. double
exponential, normal vs. Cauchy, Weibull vs. lognormal, and extreme-value vs.
normal. For large samples, the asymptotic likelihood ratio test could be used. For
small samples from other distributions Monte Carlo techniques can be used to
obtain the percentage points of the sample statistic for the likelihood ratio test.
References
[1] Bain, L. J. (1978). Statistical Analysis of Reliability and Life-Testing Models (Theory and Methods).
Marcel Dekker, New York and Basel.
[2] Bush, J. G., Woodruff, B. W., Moore, A. H. and Dunne, E. J. (1983). Modified Cramer-von
Mises and Anderson-Darling tests for Weibull distribution with unknown location and scale
parameters. Commun. Statist.-- Theor. Meth. A 12, 2463-2476.
[3] Chandra, M., Singpurwalla, N. and Stephens, M. A. (1981). Kolmogorov statistics for tests of
fit for the extreme-value and Weibull distributions. J. Amer. Statist. Assoc. 71, 204-209.
[4] Devroye, L. and Gyrrfi, L. (1985). Non-Parametric Density Estimation: the Li View. Wiley, New
York.
[5] Durbin, J. (1975). Kolmogorov-Smirnov tests when parameters are established with application
tests of exponentiality and tests on spacings. Biometn'ka 62, 5-22.
[6] Green, J. R. and Hegazy, Y. A. S. (1976). Powerful modified goodness-of-fit tests. J. Amer.
Statist. Assoc. 71, 204-209.
[7] Harter, H. L., Khamis, H. T. and Lamb, R. E. (1984). Modified Kolmogorov-Smirnov tests
of goodness-of-fit. Commun. Statist.--Simula. Computa. 13, 293-323.
120 B. W. Woodruff and A. H. Moore
[8] Harter, H. L. (1984). Another look at plotting positions. Commun. Statist.--Theor. Method. 13,
1613-1633.
[9] Hatter, H. L. (1984). Asymptotic formulas for critical values of a modified Kolmogorov test
statistic. Communications in Statistics B 13, 719-721.
[10] Harter, H. L. and Wiegand, R. P. (1985). A Monte Carlo study of plotting positions. Commun.
Statist.--Simula. Computa. 14, 317-343.
[11] Krishnaiah, P. R. (1980). Handbook of Statistics I, North-Holland, Amsterdam.
[12] Lawless, J. F. (1982). Statistical Models and Methods for Lifetime Data. Wiley, New York.
[13] Lilliefors, H. W. (1967). On the Kolmogorov test for normality with mean and variance
unknown. J. Am. Statist. Assoc. 62, 143-147.
[14] Lilliefors, H. W. (1969). On the Kolmogorov test for the exponential distribution with mean
unknown. J. Am. Statist. Assoc. 64, 387-389.
[15] Littell, R. D., McClave, J. T. and Often, W. W. (1979). Goodness-of-fit tests for the two-
parameter Weibull distribution. Commun. Statist.--Simula. Computa. B 8, 257-269.
[16] Mann, N. R. (1968). Point and interval estimation procedures for the two-parameter Weibull
and extreme-value distributions. Technometrics 10, 231-256.
[17] Mann, N. R., Scheuer, E. M. and Fertig, K. W. (1973). A new goodness-of-fit test for the two
parameter Weibull or extreme-value distribution with unknown parameters. Communications in
Statistics 2, 383-400.
[18] Moore, A. H., Ream, T. J. and Woodruff, B. W. A new goodness-of-fit test for normality with
mean and variance unknown. (Submitted for publication.)
[19] Nelson, W. (1982). Applied Life Data Analysis. Wiley, New York.
[20] Porter, J. E., Moore, A. H. and Coleman, J. W. Modified Kolmogorov, Anderson-Darling and
Cramer-von Mises tests for the Pareto distribution with unknown location and scale parame-
ters. (Submitted for publication.)
[21] Schuster, E. F. (1975). Estimating the distribution function of a symmetric distribution. Bio-
metrika 62, 631-635.
[22] Stephens, M. A. (1977). Goodness-of-fit for the extreme-value distribution. Biometrika 64,
583-588.
[23] Tapia, R. A. and Thompson, J. R. (1978). Nonparametric Probability Density Estimation. The
Johns Hopkins University Press, Baltimore and London.
[24] Woodruff, B. W;, Moore, A. H., Dunne, E. J. and Cortes, R. (1983). A modified
Kolmogorov-Smirnov test for Weibull distributions with unknown location and scale parame-
ters. IEEE Transactions on Reliability 32, 209-213.
[25] Woodruff, B. W., Viviano, P. J., Moore, A. H. and Dunne, E. J. (1984). Modified goodness-of-fit
tests for gamma distributions with unknown location and scale parameters. 1EEE Transactions
on Reliability 33, 241-245.
[26] Woodruff, B. W., Moore, A. H., Yoder, J. D. and Dunne, E. J. (1986). Modified goodness-of-fit
tests for logistic distribution with unknown location and scale parameters. Commun.
Statist.--Simula. Computa. 15(1), 77-83.
[27] Woodruff, B. W., Woodbury, L. B. and Moore, A. H. A new goodness-of-fit test for the uniform
with unspecified parameters. (Submitted for publication.)
[28] Yen, V. C. and Moore, A. H. Modified goodness-of-fit tests for the Laplace distribution.
(Submitted for publication.)
[29] Yen, V. C. and Moore, A. H. New modified goodness-of-fit tests for the Laplace distribution.
(Submitted for publication.)
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7
v
© Elsevier Science Publishers B.V. (1988) 121-129
I. Introduction
This paper is a sequel to the survey paper of Hollander and Proschan (1984)
who examine univariate nonparametric classes and methods in reliability. In this
paper we will examine multivariate nonparametric classes and methods in relia-
bility.
Hollander and Proschan (1984) describe the various univariate nonparametric
classes in reliability. The classes of adverse aging described include the IFR,
IFRA, NBU, N B U E and D M R L classes. The dual classes of beneficial aging are
also covered. Several new univariate classes have been introduced since that time.
One that we will briefly mention is the H N B U E class, since we are aware of
several multivariate generalizations of this class.
The univariate classes in reliability are important in applications concerning
systems where the components can be assumed to be independent. In this case
the components are often assumed to experience wearout or beneficial aging of
a similar type. For example, it is often reasonable to assume that components
have increasing failure rate (IFR). In making this I F R assumption it is implicit
that each component separately experiences wear and no interactions among
components can occur. However in many realistic situations, adverse wear on one
component will promulgate adverse wear on other components. From another
point of view a common environment will cause components to behave similarly.
In either situation, it is clear that an assumption of independence on the components
would not be valid. Consequently multivariate concepts of adverse or beneficial
aging are required.
Multivariate nonparametric classes have been proposed as early as 1970. For
background and references as well as some discussion of univariate classes with
multivariate generalizations in mind see Block and Savits (1981). In the present
paper we shall only describe a few fundamental developments prior to 1981 and
focus on developments since then. The coverage will not be exhaustive but will
emphasize the topics which we feel are most important.
Section 2 deals with multivariate nonparametric classes. In Section 2.1 multi-
variate IFRA is discussed with emphasis on the Block and Savits (1980) class.
Multivariate N B U is covered in Section 2.2 and multivariate N B U E classes are
mentioned in Section 2.3. New developments in multivariate IFR are considered
in Section 2.4 and in Section 2.5 the topics of multivariate D M R L and H N B U E
are touched on.
Familiarity with the univariate classes is assumed. The basic reference for the
IFR, IFRA, NBU and N B U E classes is Barlow and Proschan (1981). See also
Block and Savits (1981). For information on the D M R L class see Hollander and
Proschan (1984). The H N B U E class is relatively recent and the best references
are the original articles. See for example, Klefsj6 (1982) and the references
contained there.
2.1. Multivariate I F R A
Using a characterization of the univariate IFRA class in Block and Savits
(1976) the following definition can be made.
DEFINITION 2.1.1. Let T = (T1, ..., 7",) be a nonnegative random lifetime. The
random vector T is said to be M I F R A if
E~'[h(T)] <<.E[h~'(T/o~]
for all continuous nonnegative nondecreasing functions h and all 0 < ~ ~< 1.
Multivariate nonparametric classes in reliability 123
This definition as mentioned above implies all of the properties one would
desire for a multivariate analog of the univariate IFRA class. Part of the reason
for this is that the definition is equivalent to many other properties which are both
theoretically and intuitively appealing. The statement and proofs of these results
are given below; the form in which these are presented is influenced by the paper
of Marshall and Shaked (1982) who defined a similar M N B U class.
THEOREM 2.1.2. The following conditions are all equivalent to T being MIFRA.
(i) P~{T~A" 5 <~P{T~c~A) for all open upper sets in R~+, all 0 < oct< 1.
(ii) P ~ { T 6 A ) < ~ P { T ~ A ) for all upper sets in R"+, all 0 < ~ < 1 . (i.e.
E~((o(T)) <~E(gp~(T/~)) for all nonnegative, binary, nondecreasing ~ on ~+ ).
(iii) E~(h(T))<~E(h(T/~)) for all nonnegative, nondecreasing h on R~+, all
0<~<1.
(iv) For all nonnegative, nondecreasing, subhomogeneous h on ~"+, h(T) is IFRA.
(v) For all nonnegative, nondecreasing, homogeneous h on R+, h(T) is IFRA.
PROOF. (i) => (ii). By Theorem 3.3 of Esary, Proschan and Walkup (1967) for
an upper set A and any e > 0 there is an open upper set A~ such that A c A~ and
P { T 6 ~A~} <~P{TE aA} + e. Thus
i-1 if i - 1 /
2~ 2k < , h ( t ) < 2 k ' i= 1 , 2 , . . . , k 2 k
hl,(t) =
k if h(t)>, k ,
i.e.
h~(t)= kZk
E ~1 IA,.~(t)
i=1
where IA,.~ is the indicator function of the upper set Ai,/, = {tih(t) >~i/2k}. Thus
124 H. IV. Block and T. H. Savits
where A1, . . . , A m are upper sets, since the remainder follows by the monotone
convergence theorem. We have
=[~=, {~a~'l,(t/¢)dF(,)~)l/~-]1
<~i=~, f ailAi(t/g)dF(t)
= E ([;=~ ailA,(T/~))~]
where the first inequality follows from (iii) and the second by the subhomogeneity.
(iv) ~ (v). Obvious.
(v) => (vi). Let A be an open upper set and define
sup 0 > 0 : 1
{ o t~A } if 0 O: 1
h(t) =
0 otherwise.
NOTE 2.1.3. The following two alternate conditions could also have been added
to the above list of equivalent conditions (provided F(0) = 1).
(vi) P ~ { T ~ A } <~P{T~ ~A} for each set of the form A = U,."_1A+ where
A + = { x l x > x + } , x+E~+ and for all 0 < c ~ < 1 .
(vii) For each k - - 1 , 2 . . . . . for each a o, i = 1. . . . . k, j = 1. . . . . n,
0 ~< a+/~< 0% and for each coherent life function z of order kn z(allT1,
a~2T~ . . . . . alnT1, a21T2, . . . , ak, T,) is IFRA. (See Block and Savits (1980) for
a definition of coherent life function and for some details of the proof.)
In conjunction with the preceding result the following lemma makes it easy to
demonstrate that a host of different lifetimes are MIFRA.
PROOf. Since coherent life functions are homogeneous this follows easily.
~(T1, . . . , Tn) is N B U
for all ~ in a certain class of life functions; (2.2.1)
where T, T ' , T" are independent and have the same distribution.
For every upper s e t A c N + and for every ~ > 0 , f l > 0
(2.2.4)
P ( T c A + B) <~P ( T c A ) P ( T c B ) . (2.2.5)
References
Barlow, R. E. and Proschan, F. (1981). Statistical Theory of Reliability and Life Testing: Probability
Models. To Begin With, Silver Spring, MD.
Basu, A. P. and Ebrahimi, N. (1981). Multivariate HNBUE distributions. University of Missouri-
Columbia, Technical Report # 110.
Berg, J. and Keston, H. (1984). Inequalities with applications to percolation and reliability.
Unpublished report.
Block, H. W. and Savits, T. H. (1976). The IFRA closure problem. Ann. Prob. 4, 1030-1032.
Block, H. W. and Savits, T. H. (1978). Shock models with NBUE survival. J. AppL Prob. 15,
621-628.
Block, H. W. and Savits, T. H. (1980). Multivariate increasing failure rate average distributions. Ann.
Prob. 8, 793-801.
Block, H. W. and Savits, T. H. (1981). Multivariate classes in reliability theory. Math. of O.R. 6,
453-461.
Block, H. W. and Savits, T. H. (1982). The class of MIFRA lifetimes and its relation to other classes.
NRLO 29, 55-61.
Buchanan, B. and Singpurwalla, N, D. (1977). Some stochastic characterizations of multivariate
survival. In: C. P. Toskos and I. Shimi, eds., The Theory and AppL of Reliability, Vol. I, Academic
Press, New York, 329-348.
Multivariate nonparametric classes in reliability 129
Ebrahimi, N. and Ghosh, M. (1981). Multivariate NBU and NBUE distributions. The Egyptian
Statistical Journal 25, 36-55.
E1-Neweihi, E. (1981). Stochastic ordering and a class of multivariate new better than used distribu-
tions. Comm. Statist.-Theor. Meth. A 10(16), 1655-1672.
EI-Neweihi, E., Proschan, F. and Sethuraman, J. (1983). A multivariate new better than used class
derived from a shock model. Operations Research 31, 177-183.
Esary, J. D. and Marshall, A. W. (1979). Multivariate distributions with increasing hazard rate
averages. Ann. Prob. 7, 359-370.
Esary, J. D., Proschan, F. and Walkup, D. W. (1967). Association of random variables, with
applications. Ann. Math. Stat. 38, 1466-1474.
Ghosh, M. and Ebrahimi, N. (1981). Shock models leading to increasing failure rate and decreasing
mean residual life survival. J. Appl. Prob. 19, 158-166.
Griffith, W. (1982). Remarks on a univariate shock model with some bivariate generalizations. NRLQ
29, 63-74.
Hollander, M. and Proschan, F. (1984). Nonparametric concepts and methods in reliability. In: P.
R. Krishnaiah and P. K. Sen, eds., Handbook of Statistics, Vol. 4, Elsevier, Amsterdam.
Klefsj6, B. (1980). Multivariate HNBUE. Unpublished report.
Klefsj6, B. (1982). NBU and NBUE survival under the Marshall-Olkin shock model. IAPQR Trans-
actions 7, 87-96.
Klefsj6, B. (1982). The HNBUE and HNWUE classes of life distributions. NRLQ 29, 331-344.
Marshall, A. W. and Olkin, I. (1967). A generalized bivariate exponential distribution. J. Appl. Prob.
4, 291-302.
Marshall, A. W. and Shaked, M. (1979). Multivariate shock models for distributions with increasing
hazard rate average. Ann. Prob. 7, 343-358.
Marshall, A. W. and Shaked, M. (1982). A class of multivariate new better than used distributions.
Ann. Prob. 10, 259-264.
Marshall, A. W. and Shaked, M. (1984). Multivariate new better than used distributions. Un-
published report.
Savits, T. H. and Shaked, M. (1981). Shock models and the MIFRA property. Stoch. Proc. Appl.
11, 273-283.
Savits, T. H. (1983). Multivariate life classes and inequalities. In: Y. L. Tong, ed., Inequalities on
Probability and Statistics IMS, Hayward, CA.
Savits, T. H. (1985). A multivariate IFR class. J. Appl. Prob., 22, 197-204.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 0
./
© Elsevier Science Publishers B.V. (1988) 131-156
I. Introduction
* This research was supported by the Office of Naval Research Contract N00014-84-C-0167 at
Purdue University. Reproduction in whole or in part is permitted for any purpose of the United
States Government.
131
732 s. s. Gupta and S. Panchapakesan
In the last three decades and more, the literature on selection and ranking
procedures has grown enormously. Several books have appeared exclusively
dealing with selection and ranking procedures. Of these, the monograph of
Bechhofer, Kiefer and Sobel (1968) deals with sequential procedures with special
emphasis on Koopman-Darmois family. Gibbons, Olkin and Sobel (1977) deal
with methods and techniques mostly under the IZ formulation. Gupta and
Panchapakesan (1979) provide a comprehensive survey of the developments in the
field of ranking and selection, with a special chapter on Guide to Tables. They
deal with all aspects of the problem and provide an extensive bibliography.
BOringer, Martin and Schriever (1980) and Gupta and Huang (1981) have dis-
cussed some specific aspects of the problem. A fairly comprehensive categorized
bibliography is provided by Dudewicz and Koo (1982). For a critical review and
an assessment of developments in subset selection theory and techniques, refer-
ence may be made to Gupta and Panchapakesan (1985).
Section 2 discusses the formulation of the basic problem of selecting the best
population using the IZ and SS approaches. Section 3 deals with selection from
gamma, exponential and Weibull populations. Procedures for different generalized
goals are discussed using both IZ and SS approaches. Nonparametric procedures
are discussed in Section 4 for selecting in terms of ~t-quantiles. This section also
discusses procedures for Bernoulli distributions. These serve as distribution-free
procedures for selecting from life distributions in terms of reliability at an arbi-
trarily chosen time. Procedures for selection from restricted families of distribu-
tions are described in Section 5. These include procedures for IFR and IFRA
families in particular. A brief discussion of selection in comparison with a
standard or control follows in Section 6.
Let 7Zl, ..., 7Zk be k given populations where ni has the associated distribution
function Fo,, i = 1. . . . , k. The 0i are real-valued parameters taking values in the
set O. It is assumed that the 0; are unknown. The ordered 0,. are denoted by
011j ~< 0[2] ~< • • • ~< 0[k] and the (unknown) population ne associated with Oto by n;,
i = 1. . . . . k. The populations are ranked according to their 0-values. To be
specific, nu~ is defined to be better than nti) if i < j . No prior information is
assumed regarding the true pairing between (01 . . . . . 0~) and (0711, ..., 0[k]).
where P(CSIR) denotes the PCS using R, and b(Otk1, 0[k_ 1]) is an appropriate
measure of separation of the best population re(k) from the next best re(k- 1~' The
constants P* and b* are specified by the experimenter in advance. The statistical
problem is to define a selection rule which really consists of a sampling rule, a
stopping rule for sampling, and a decision rule. If we consider taking a single
sample of fixed size n from each population, then the minimum value of n is
determined subject to (2.1). A crucial step involved in this is to evaluate the
infimum of the PCS over 12~. = {0 = 01, . . . , Ok): b(Otl,], Otk_ 11) ~> b*}. Any con-
figuration of 0 where this infimum is attained is called a least favorable con-
figuration (LFC). Between two valid (i.e. satisfying (2.1)) single sample proce-
dures, the sample size n is an obvious criterion for efficiency comparison. The
region f2b. is called the preference zone. No requirement is made regarding the
PCS when 0 belongs to the complement of fib* which, in fact, is the indifference
zone.
where f2 = {0} is the whole parameter space. It should be noted that there is no
indifference zone specification in this formulation. As is to be expected, a crucial
step is the evaluation of the infimum of the PCS over 12. Any subset selection rule
that satisfies (2.2) meets the criterion of validity. Denoting the selected subset by
S and its size by IS I, the expected value of lSI serves as a reasonable measure
for efficiency comparison between valid procedures. Besides E(IS b), possible
performance characteristics include E(IS I) - PCS and E([S ])/PCS. The former
one represents the expected number of nonbest populations included in the
selected subset. As an overall measure, one can also consider the supremum of
E ( / S I) over O.
For details on these aspects of the problem, reference may be made to Gupta and
Panchapakesan (1979).
Suppose that the best population is the one associated with the largest 0,.. A
procedure R is said to be monotone if the probability of selecting ~i is at least as
large as that of selecting rcj whenever 0~> 0j..
Table la
Values of the constant c of Rule R3 satisfying equation (3.3); P* = 0.90
v 2 3 4 5 6 7 8 9 10 11
1 0.111 0.072 0.059 0.052 0.047 0.044 0.041 0.039 0.038 0.036
2 0.244 0.183 0.159 0.145 0.135 0.128 0.123 0.119 0.116 0.113
3 0.327 0.260 0.232 0.215 0.203 0.195 0.188 0.183 0.178 0.174
4 0.386 0.317 0.286 0.268 0.255 0.246 0.239 0.232 0.228 0.223
5 0.430 0.360 0.329 0.310 0.297 0.287 0.279 0.273 0.268 0.263
6 0.466 0.396 0.364 0.345 0.332 0.321 0.313 0.307 0.301 0.296
7 0.494 0.426 0.394 0.374 0.361 0.350 0.342 0.336 0.330 0.325
8 0.519 0.451 0.419 0.400 0.386 0.376 0.367 0.360 0.355 0.350
9 0.539 0.472 0.441 0.422 0.408 0.398 0.389 0.382 0.376 0.371
10 0.558 0.492 0.460 0.441 0.428 0.417 0.409 0.402 0.396 0.391
11 0.573 0.508 0.478 0.459 0.445 0.434 0.426 0.419 0.414 0.408
12 0.588 0.524 0.493 0.474 0.461 0.450 0.442 0.435 0.429 0.424
13 0.600 0.537 0.507 0.488 0.475 0.465 0.456 0.450 0.444 0.439
14 0.612 0.550 0.520 0.502 0.488 0.478 0.470 0.463 0.457 0.452
15 0.622 0.561 0.532 0.514 0.500 0.490 0.482 0.475 0.469 0.464
16 0.632 0.572 0.543 0.525 0.511 0.501 0.493 0.486 0.481 0.476
17 0.641 0.582 0.553 0.535 0.522 0.512 0.504 0.497 0.491 0.486
18 0.649 0.591 0.562 0.544 0.532 0.522 0.514 0.507 0.501 0.496
19 0.657 0.599 0.571 0.553 0.540 0.531 0.523 0.516 0.510 0.506
20 0.664 0.607 0.579 0.562 0.549 0.539 0.531 0.525 0.519 0.514
136 S. S. Gupta and S. Panchapakesan
Table lb
Values of the constant c of Rule R3 satisfying equation (3.3); P* = 0.95
v 2 3 4 5 6 7 8 9 10 11
1 0.053 0.035 0.028 0.025 0.023 0.021 0.020 0.019 0.018 0.018
2 0.156 0.119 0.104 0.095 0.089 0.085 0.082 0.079 0.076 0.074
3 0.233 0.188 0.168 0.156 0.148 0.142 0.138 0.134 0.131 0.128
4 0.291 0.242 0.220 0.206 0.197 0.190 0.184 0.180 0.176 0.173
5 0.336 0.285 0.261 0.247 0.237 0.229 0.223 0.218 0.214 0.210
6 0.372 0.320 0.296 0.281 0.271 0.263 0.256 0.251 0.247 0.243
7 0.403 0.350 0.326 0.310 0.300 0.291 0.285 0.279 0.275 0.271
8 0.428 0.376 0.351 0.336 0.325 0.316 0.310 0.304 0.300 0.296
9 0.451 0.399 0.374 0.358 0.347 0.339 0.332 0.326 0.322 0.317
10 0.471 0.419 0.394 0.378 0.367 0.359 0.352 0.346 0.341 0.337
11 0.488 0.437 0.412 0.396 0.385 0.377 0.370 0.364 0.359 0.355
12 0.504 0.453 0.428 0.413 0.402 0.393 0.386 0.380 0.376 0.371
13 0.518 0.468 0.443 0.428 0.417 0.408 0.401 0.395 0.390 0.386
14 0.531 0.481 0.457 0.442 0.430 0.422 0.415 0.409 0.404 0.400
15 0.543 0.494 0.470 0.454 0.443 0.434 0.428 0.422 0.417 0.413
16 0.554 0.505 0.481 0.466 0.455 0.446 0.439 0.434 0.429 0.424
17 0.564 0.516 0.492 0.477 0.466 0.457 0.450 0.445 0.440 0.436
18 0.574 0.526 0.502 0.487 0.476 0.468 0.461 0.455 0.450 0.446
19 0.582 0.535 0.512 0.497 0.486 0.477 0.470 0.465 0.460 0.456
20 0.591 0.544 0.520 0.506 0.495 0.486 0.480 0.474 0.469 0.465
Xi >~ c m a x X j (3.2)
1 ~<j<~k
fO e Gkv - l ( x / c ) g v ( x ) d x = e * , (3.3)
1
X~ ~< - min X. (3.4)
C 1 <~j~<k J
where 0 < c' < 1 is the largest number for which the P*-condition is met. The
constant c' is given by
where v = n~. The values of the constant c' have been tabulated for v = 1(1)25,
k = 2(1)11, and P* = 0.75, 0.90, 0.95, 0.99 by Gupta and Sobel (1962b) who have
studied rule R 4 in the context of selecting from k normal populations the one with
the smallest variance in a companion paper (1962a).
It is known that the gamma family {F(x, 0)}, with common parameter r, is
stochastically increasing in 0, i.e., F(x, 0~) and F(x, Oj) are distinct for 0,. # 0j, and
F(x, 0~) >1F(x, Oj) for all x when 0~< 0j.. This implies that ranking them in terms
of 0 is equivalent to ranking in terms of a-quantile for any 0 < a < 1.
r
T,= L X,j + (n - r)X,r, i= l, ..., k. (3.5)
j=l
The Ti are the so-called total life statistics. It is well-known that 2Te/Oi has a
chi-square distribution with 2r degrees of freedom. In other words, 7",. has a
gamma distribution with scale parameter 0~ and shape parameter r. Thus for
selecting the population with the largest mean life 0e, the procedure R 3 (stated in
terms of the T~) will be
The density (3.7) provides a model for life length data when we assume a
minimum guaranteed life 0~, which is here a location parameter. It is assumed that
all the k populations have a common scale parameter a. The 0i are unknown and
our interest is in selecting the population associated with the largest 0~. We will
discuss some procedures under the IZ formulation. Consider the generalized goal
of selecting a subset of fixed size s so that the t best populations (1 ~< t ~< s < k)
are included in the selected subset. This generalized goal was introduced by Desu
and Sobel (1968). The special case of t = s, namely, that of choosing t populations
so that they are the t best, was considered originally by Bechhofer (1954). When
s = t = 1, we get the basic goal of selecting the best population. The probability
requirement is that
where 0* and P* are specified in advance and a correct selection occurs when
a subset of s populations is selected consistent with the goal. Also, for a
meaningful problem, we should have 1/(~) < P* < 1. In describing several proce-
dures, we will adopt either the generalized goal or one of its special cases. We
will consider the two cases of known and unknown a separately.
Case A: Known or. We can assume without loss of generality that cr = 1. Let
Xij, j = 1, ..., n, denote a sample of n observations from re;, i = 1, . . . , k. Define
Yi mini <-~j<~nXij , i = 1, ..., k.
=
Raghavachari and Starr (1970) considered the goal of selecting the t best
populations (i.e. 1 ~<s = t < k ) and they stvdied the 'natural' rule
0[l . . . . . O[k_t] ;
O[k-t+ 1] ~--- ' ' " = O[k] ; (3. lo)
O[k-t + 11 O[k_t] = O*.
Selection and ranking procedures in reliability models 139
The minimum sample size required to satisfy (3.8) is the smallest integer n for
which
(1 - v) k - t + (k - t ) v - t I ( v , t + 1, k - t) = P * . (3.14)
Raghavachari and Starr (1970) have tabulated the v-values for k = 2(1)15,
t = l(1)k - 1, and P* = 0.90, 0.95, 0.975, 0.99.
In particular, for selecting the best population, the equation (3.14) reduces to
(vk)-l[1 - (1 - v) k] = P * . (3.15)
For the generalized goal, Desu and Sobel (1968) studied the following rule R 6.
R6: Select the s populations associated with Ytk-s+ 1]. . . . ' Ytk~-
Given n, k, t, 0", and P*, they have shown that the smallest s for which the
probability requirement (3.8) is satisfied is the smallest integer s such that
(~) >~p.(k)
t
e-n,o*, ' (3.16)
It should be pointed out that Desu and Sobel (1968) have obtained general results
for location parameter family. They have also considered the dual problem of
selecting a subset of size s (s ~< t) so that all the selected populations are among
the t best.
Case B: Unknown a. In this case, we consider the basic goal of selecting the
best population. Since a is unknown, it is not possible to determine in advance
the sample size needed for a single sample procedure in order to guarantee the
P*-condition. This is similar to the situation that arises in selecting the population
with the largest mean from several normal populations with a common unknown
variance. For this latter problem, Bechhofer, Dunnett and Sobel (1954) proposed
a non-elimination type two-stage procedure in which the first stage samples are
utilized purely for estimating the variance without eliminating any population from
further consideration. A similar procedure was proposed by Desu, Narula and
Villarreal (1977) for selecting the best exponential population. Kim and Lee (1985)
have studied an elimination type two-stage procedure analogous to that of Gupta
140 S. S. Gupta and S. Panchapakesan
and Kim (1984) for the normal means problem. In their procedure, the first stage
is used not only to estimate a but also to possibly eliminate non-contenders. Their
Monte-Carlo study shows that, when 0tkI - 0tk_ 1] is sufficiently large, the elimi-
nation type procedure performs better than the other type procedure in terms ot
the expected total sample size.
The procedure R 7 of Kim and Lee (1985) consists of two stages as follows.
Stage I ." Take n o independent observations from each rcg (1 ~< i ~< k), and compute
y/.(o = min~ ~j<~noXij, and a pooled estimate ~ of a, namely,
k no
where the symbol a + denotes the positive part of a, and h ( > 0 ) is a design
constant to be determined.
(a) If I has only one element, stop sampling and assert that the population
association with V(1)
--[k] as the best.
(b) If I has more than one element, go to the second stage.
and the symbol ( y ) denotes the smallest integer equal to greater than y. Then
compute, for the overall sample, Y~.= maxl~j~vX~j and choose the population
associated with maxi~ x Y~ as the best.
The constant h used in the procedure R 7 is given by
fO °° {1 -- (1 -- O~(x))k}2/{k20~2(x)}fv(X) d x = P* (3.17)
0;,c~>0; i = 1. . . . , k . (3.18)
Selection and ranking procedures in reliability models 141
The c`. and Oz. are unknown. Kingston and Patel (1980a, b) have considered the
problem of selecting from Weibull distributions in terms of their reliabilities
(survival probabilities) at an arbitrary but specified time L > 0. The reliability at
L for F~ (i = 1. . . . . k) is given by
We can without loss of generality assume that L = 1 because the observed failure
times can be scaled so that L = 1 time unit. Further, letting (0`.)c' = 2;, we get
p`. = exp { - 27 1}. Obviously, ranking the populations in terms of the p; is equiva-
lent to ranking in terms of the 2;, and the best population is the one associated
with 2[k], the largest 2,.. Kingston and Patel (1980a) considered the problem of
selecting the best one under the IZ formulation using the natural procedure based
on estimates of the 2`. constructed from type II censored samples. They also
considered the problem of selecting the best in terms of the a-quantiles for a given
~ (0, 1), ~ 1 - e -1, in the case where 01 . . . . . Ok= 0 (unknown). The
~-quantile of F`. is given by ¢`. = 0[ - l o g ( 1 - ~)]l/ci so that ranking in terms of the
~-quantiles is equivalent to ranking in terms of the shape parameter. It should be
noted that the ranking of the ci is in the same order as that of the associated 4`.
if a < 1 - e-1, and is in the reverse order if a > 1 - e-1. The procedures dis-
cussed above are based on maximum likelihood estimators as well as simplified
linear estimators (SLE) considered by Bain (1978, p. 265). For further details on
these procedures, see Kingston and Patel (1980a).
In another paper, Kingston and Patel (1980b) considered the goal of selecting
a subset of restricted size. This formulation, usually referred to as restricted subset
selection (RSS) approach, is due to Gupta and Santner (1973) and Santner
(1975). In the usual s s approach of Gupta (1956), it is possible that the proce-
dure selects all the k populations. In the RSS approach, we restrict the size of
the selected subset by specifying an upper bound m (1 ~< m ~< k - 1); the size of
the selected subset is still random variable taking on values 1, 2 . . . . , m. Thus it
is a generalization of the usual approach (m = k). However, in doing so, an
indifference zone is introduced. The selection goal can be more general than
selecting the best. We now consider a generalized goal in the RSS approach for
selection from Weibull populations, namely, to select a subset of the k given
populations not exceeding m in size such that the selected subset contains at least
s of the t best populations. As before, the populations are ranked in terms of their
2-values. Note that 1 ~< s ~< min (t, m) ~< k. The probability requirement now is
that
When t = s = m and 2* > 1, the problem reduces to selecting the t best popula-
tions using the IZ formulation. When s = t < m = k and 2*= 1, the problem
reduces to selecting a subset of random size containing the t best populations (the
usual SS approach). Thus the RSS approach integrates the formulations of
Bechhofer (1954), Gupta (1956), and Desu and Sobel (1968). General theory
under the RSS approach is given by Santner (1975).
Returning to the Weibul selection problem with the generalized RSS goal,
Kingston and Patel (1980b) studied a procedure based on type II censored
samples from each population. It is defined in terms of the maximum likelihood
A
There have been some investigations of subset selection rules based on ranks
while still assuming that the distributions associated with the populations are
known. This is appealing especially in situations in which the order of the observa-
tions is more readily available than the actual measurements themselves due,
perhaps, to excessive cost or other physical constraints. Under this setup, Nagel
(1970), Gupta, Huang and Nagel (1979), Huang and Panchapakesan (1982), and
Gupta and Liang (1987) have investigated locally optimal subset selection rules
which satisfy the validity criterion that the infimum of the PCS is P* when the
distributions are identical. They have used different optimality criteria in some
neighborhood of an equiparameter point in the parameter space. An account of
these rules is given in Gupta and Panchapakesan (1985).
Characterizations of life length distributions are provided in many situations by
so-called restricted families of distributions which are defined by partial order
relations with respect to known distributions. Well-known examples of such
families are those with increasing (decreasing) failure rate and increasing (decreas-
ing) failure rate average. Selection procedures for such families will be discussed
in the next section.
In the remaining part of this section, we will be mainly concerned with non-
parametric procedures for selection in terms of a quantile and selection from
several Bernoulli distributions. Though the Bernoulli selection problem could have
been discussed under parametric model, it is discussed here to emphasize the fact
that we can use the Bernoulli selection procedures as distribution-free procedures
for selecting from unknown continuous (life) distributions in terms of reliability at
any arbitrarily chosen time point L.
where c is the smallest integer with 1 ~< c ~< r - 1 for which the P*-condition is
satisfied.
For the procedure R9, the infimum of the PCS is attained when the distribu-
tions F 1. . . . . F k are identical and it is shown by Rizvi and Sobel (1967) that c
144 s. s. Gupta and S. Panchapakesan
where
~0 1 Grk--cl(u) dGr(u) ~> P* (4.2)
n~
Gr(u)= ur - l ( 1 - u ) . . . . 1, 0,N<u~<l. (4.3)
(r- 1)!(n - r)!
Rizvi and Sobel have shown that the maximum permissible value o f P* such that
a c-value satisfying (4.2) exists is P1 = PI( n, ~, k) given by
P1 = (,,(i+ (4.4)
i=0 ". r 1))
A short table of Pl-values is given by Rizvi and Sobel for ~ = 0.5 and k = 2(1)10.
The n-values range from 1 in steps of 2 to a value (depending on k) for which
P1 gets very close to 1. Also given by them is a table of the largest value of r - c
for c~ = 1/2 (which means that r = (n + 1)/2), k = 2(1)10, n = 5(10)95(50)495, and
P* = 0.75, 0.90, 0.95, 0.975, 0.99. For the IZ approach to this selection problem,
see Sobel (1967).
{~ if X ° > L
Y,y= ' i=1 ..... k;j=l ..... n. (4.1)
otherwise,
The Yil ..... Yin are independent and identically distributed Bernoulli r a n d o m
variables with success probability p;, i = 1. . . . . k. We are interested in selecting
the population associated with the largest pi.
G u p t a and Sobel (1960) proposed a subset selection rule based on
Yi = ~nj=l Y/j, i = 1, . . . , k. Their rule is
where D is the smallest nonnegative integer for which the P*-requirement is met.
An interesting feature o f Procedure Rio is that the infimum of the PCS occurs
when Pl . . . . . Pk = P (say) but it is not independent of their c o m m o n value p.
Selection and ranking procedures in reliability models 145
For k = 2, Gupta and Sobel (1960) showed that the infimum takes place when
p = 1/2. When k > 2, the common value Po for which the infimum takes place is
not known. However, it is known that this common value Po ~ 1/2 as n ~ ~ . An
improvement in the situation is provided by Gupta, Huang and Huang (1976)
who investigated conditional selection rules and, using the conditioning argument,
obtained a conservative value of d. Their conditional procedure is
RI~: Select re,. if and only if
Y~>>. m a x Yj-D(t) (4.3)
1 ~<j~< k
given T = ~k;= ~ Y~-= t, where D(t) > 0 is chosen to satisfy the P*-condition. Exact
result for the infimum of the PCS is ~ ~tained only for k = 2; in this case, the
infimum is attained when p~ = P2 = P and is independent of the common value p.
For k > 2, Gupta, Huang and Huang (1976) obtained a conservative value for
D(t) and also for D of Rule Rio. They have shown that infP(CS ]R~I ) >i P * if D(t)
is chosen such that
Sd(t) for k = 2,
D(t) (4.4)
~max{d(r): r = 0, 1, . . . , min(t, 2n)) for k > 2,
for k = 2 ,
N(2; d(r), r, n) >1/.[1 - (1 - P * ) ( k - 1)- l] (zn) (4.5)
for k > 2 ,
and N(k; d(t), t, n) = • ( ~ ) . . . ( ~ ) , with the summation taken over the set of all
. . k
nonnegatlve integers s; such. that ~ i = 1 si = t and s k >>,m a x i <~j<<.k- ~sj - d(t).
A conservative constant d for Procedure Rio is given by d = maxo<.t<~knd(t ).
Gupta, Huang and Huang (1976) have tabulated the smallest value d(t) satisfying
(4.5) for k = 2,4(1)10, n = 1(i)10, t = 1(1)20, and P* = 0.75, 0.90, 0.95, 0.99.
They have also tabulated the d-values (conservative) for Procedure Rio for
P* = 0.75, 0.90, 0.95, 0.99, and n = 1(1)4 when k = 3(1)15, and n -- 5(1)10 when
k = 3(1)5.
Under the IZ formulation, one can use the procedure of Sobel and Huyett
(1957) for selecting the population associated with the largest Pi which guarantees
a minimum PCS P* whenever PtkJ -- Ptg- II >/A* > 0. Based on samples of size
n from each population, their procedure based on the Yi defined in (4.1) is
The sample size required is the smallest n for which the PCS >~ P* when
Pt~] . . . . . P[k-lJ = P t k ] - A*, the LCF in this case. Sobel and Huyett (1957)
have tabulated the sample sizes (exact and approximate) for k = 2, 3, 4, 10;
A* = 0.05(0.05)0.50, and P* = 0.50, 0.60, 0.75(0.05)0.95, 0.99.
146 S. S. Gupta and S. Panchapakesan
f ~
--oO
qtr~- l(x + c)qg(x)dx = P* (4.8)
and, ~ and q~ denote correspondingly the cdf and density of the standard normal
distribution. The c-value can be obtained from tables of Bechhofer (1954), Gupta
(1963b), Milton (1963) and Gupta, Nagel and Panchapakesan (1973) for several
selected values of k and P*.
The Bernoulli selection problem has applications to the drug selection problem
and to clinical trials. This fact has spurred lots of research activity involving
investigations of selection procedures using sampling procedures such as the
play-the-winner (PW) sampling rule (introduced by Robbins, 1952 and 1956) and
vector-at-a-time (VT) rule with a variety of stopping rules. One of the main
considerations in many of these procedures is to design the sampling rule so as
to minimize the expected total number of observations and/or the expected num-
ber of observations from the worst population. Some of these procedures suffer
from one drawback or another. For excellent review/survey/comprehensive assess-
ment of these (and other) procedures, reference should be made to Bechhofer and
Kulkarni (1982), BOringer, Martin and Schriever (1980), Gupta and
Panchapakesan (1979, Sections 4.2 through 4.6), and Hoel, Sobel and Weiss
(1975). For corresponding developments in subset selection theory, see Gupta and
Panchapakesan (1979, Section 13.2).
where c = c(k, P*, n, j) is the largest number in (0, 1) for which the P*-condition
is satisfied. The constant c is given by
where Gj denotes the cdf of the jth order statistic in a sample of n observations
from G, and gj is the corresponding density function. The values of c satisfying
(5.2) are tabulated by Barlow, Gupta and Panchapakesan (1969) in the special
case of exponential G, i.e. for selecting from IFRA populations, for P* = 0.75,
0.90, 0.95, 0.99, and the following values of k, n, and j: (i) j = 1, k = 2(1)11 (in
this case, c is independent of n), (ii) k = 2(1)6, j = 2(1)n, and n = 5(1)10 or 12 or
15 depending on k. Table 2a is excerpted from the tables of Barlow, Gupta and
Panchapakesan (1969). It gives the values of c for P* = 0.90, 0.95, k = 2(1)5,
Table 2a
Values of the constant c of Rule R13 satisfying equation (4.2) for selecting
the IFRA distribution with the largest median; G(x)= 1 - e -x, x~>0,
j~< (n + 1)/2 < j + 1, P * = 0.90 (top entry), 0.95 (bottom entry)
n 2 3 4 5
Table 2b
Values of the constant d of Rule RI4 satisfying equation (5.4) for selecting
the IFRA distribution with the smallest median; G(x)= 1 - e -x, x>~O,
j ~<(n + 1)/2 < j + 1, P* = 0.90 (top entry), 0.95 (bottom entry)
n 2 3 4 5
dTs.,i>~ min
l <~r<~k
Tj, r (5.3)
f o B [1 - G j ( x d ) ] k - l g j ( x ) d x = P * (5.4)
for other k values. Table 2b is excerpted from the tables of Barlow, Gupta and
Panchapakesan (1969). It gives the values of d for P * = 0.90, 0.95, k = 2(1)5,
n = 5(1)12, and j such that j ~< (n + 1)/2 < j + 1 (i.e. appropriate for selection in
terms of median).
Suppose that G is the Weibull distribution with cdf G(x) = 1 - exp { -(x/O)~},
x ~> 0, and 0, 2 > 0. It is assumed that 2 is known. Then it is easy to see that
the new constant c~ is given by c I = c ~/~, where c is the constant in the exponen-
tial case (2 = 1). Another interesting special case of G is the half-normal distribu-
tion obtained by folding N(0, a 2) at the origin, where a is assumed to be known.
The class of distributions which are star-shaped with respect to this folded normal
is a subclass of IFRA distributions. Selection in terms of quantiles in this case
has been considered by Gupta and Panchapakesan (1975), who have tabulated
the constant c associated with RI3 for k--- 2(1)10, n = 5(1)10, j = l(1)n, and
P* = 0.75, 0.90, 0.95, 0.99.
where the T/, r are defined as in the case of the procedure R13 , and the appropriate
constant D = D(k, P*, n) > 0 is given by
Here, Gs and gs are the cdf and the density of the jth order statistic in a sample
of n independent observations from G. The values of D are given by Gupta and
Panchapakesan (1974) in the special case where G is the logistic distribution,
G(x) = [ 1 + e-X] - 1, for k = 2(1)10, n = 5(2)15, and P* = 0.75, 0.90, 0.95, 0.99.
Using the ~-ordering (Definition 5.2) with the functions h satisfying certain
properties, Gupta and Panchapakesan (1974) have discussed a class of proce-
Selection and ranking procedures in reliabilitymodels 151
dures for selecting the best (i.e. the one which is stochastically larger than any
other, assumed to exist) of k distributions F;, i, . . . , k, which are Yr'-ordered with
respect to G. The procedures R13 and R15 are special cases of their procedure.
Hooper and Santner (1979) considered selection of good populations in terms
of c~-quantiles for star- and tail-ordered distributions using the RSS approach. Let
ni have the distribution F; and let Fvl denote the distribution having the ith
smallest c~-quantile. Denoting the c~-quantile of any distribution F by x~(F), ~ is
called a good population if x~(F~) > c*x~(Ftk_,+ 11), 0 < c* < 1, in the case of
star-ordered families, and if x~(F,.)> x~(Ft~,_t+ q ) - d*, d* > 0, in the case of
tail-ordered families. The goal of Hooper and Santner (1979) is to select a subset
of size not exceeding m(1 ~< m ~< k - 1) that contains at least one good popula-
tion. They have also considered the problem of selecting a subset of fixed size s
so as to include at least r good populations (r~< t, r~< s < k - t + r) using the IZ
approach.
Selection of one or more good populations as a goal is a relaxation from that
of selecting the best population(s). A good population is defined suitably to reflect
the fact that it is 'nearly' as good as the best. In some form or other it has been
considered by several authors; mention should be made of Fabian (1962),
Lehmann (1963), Desu (1970), Carroll, Gupta and Huang (1975), and Pancha-
pakesan and Santner (1977). A discussion of this can be found in Gupta and
Panchapakesan (1985, Section 4.2).
T i= ~ a X g )
--y--J~ n ,
i= 1, " " " '
k '
(5.7)
J=l
where
aj=gG-l(J-n 1)-gG-l(~ ), j= 1,...,r- 1,
(5.8)
a~=gG-'(~-),
Now, for selecting a subset containing Fte], Gupta and Lu (1979) proposed the
rule
where c is the largest number in (0, 1) satisfying the P*-condition. They have
shown that, if aj ~> 0 for j = 1. . . . . r, a,/> c, and g(0) ~< 1, then
7. Concluding remarks
References
Bain, L. (1978). Statistical Analysis of Reliability and Life-Testing Models, Theory and Methods. Marcel
Dekker, New York.
Barlow, R. E. and Gupta, S. S. (1969). Selectionprocedures for restricted families of distributions.
Ann. Math. Statist. 40, 905-917.
Barlow, R. E., Gupta, S. S. and Panchapakesan, S. (1969). On the distribution of the maximum and
minimum of ratios of order statistics. Ann. Math. Statist. 40, 918-934.
Barlow, R. E., Marshall, A. W. and Proschan, F. (1963). Properties of probability distributions with
monotone hazard rate. Ann. Math. Statist. 34, 375-389.
154 S. S. Gupta and S. Panchapakesan
Bechhofer, R. E. (1954). A single-sample multiple decision procedure for ranking means of normal
populations with known variances. Ann. Math. Statist. 25, 16-39.
Bechhofer, R. E., Dunnett, C. W. and Sobel, M. (1954). A two-sample multiple-decision procedure
for ranking means of normal populations with a common unknown variance. Biometrika 41,
170-176.
Bechhofer, R. E., Kiefer, J. and Sobel, M. (1968). Sequential Identification and Ranking Procedures
(with special reference to Koopman-Darmois populations). The University of Chicago Press, Chicago.
Bechhofer, R. E. and Kulkarni, R. V. (1982). Closed adaptive sequential procedures for selecting the
best of k >/2 Bernoulli populations. In: S. S. Gupta and J. O. Berger, eds., Statistical Decision
Theory and Related Topics--Ill, Vol. 1, Academic Press, New York, 61-108.
Berger, R. L, (1979). Minimax subset selection for loss measured by subset size. Ann. Statist. 7,
1333-1338.
Berger, R. L. and Gupta, S. S. (1980). Minimax subset selection rules with applications to unequal
variance (unequal sample size) problems. Scand. J. Statist. 7, 21-26.
Bickel, P. J. and Lehmann, E. L. (1979). Descriptive statistics for nonparametric models IV. Spread.
In: Jana Jureckova, ed., Contributions to Statistics: Jaroslav Hajek Memorial Volume, Reidel, Boston,
3-40.
Birnbaum, Z. W., Esary, J. D. and Marshall, A. W. (1966). A stochastic characterization of wear-out
for components and systems. Ann. Math. Statist. 37, 816-825.
Brown, G. and Tukey, J. W. (1946). Some distributions of sample means. Ann. Math. Statist. 7, 1-12.
BiJringer, H., Martin, H. and Schriever, K.-I-I. (1980). Nonparametric Sequential Selection Procedures.
Birkhanser, Boston, MA.
Carroll, R. J., Gupta, S. S. and Huang, D.-Y. (1975). On selection procedures for the t best
populations and some related problems. Comm. Statist. 4, 987-1008.
Desu, M. M. (1970). A selection problem. Ann. Math. Statist. 41, 1596-1603.
Desu, M. M., Narula, S. C. and Villarreal, B. (1977). A two-stage procedure for selecting the best
of k exponential distributions. Comm. Statist. A--Theory Methods 6, 1223-1230.
Desu, M. M. and Sobel, M. (1968). A fixed-subset size approach to a selection problem. Biometrika
55, 401-410. Corrections and amendments: 63 (1976), 685.
Doksum, M. (1969). Starshaped transformations and the power of rank tests. Ann. Math. Statist. 40,
1167-1176.
Dudewicz, E. J. and Koo, J. O. (1982). The Complete Categorized Guide to Statistical Selection and
Ranking Procedures. Series in Mathematical and Management Sciences, Vol. 6, American Sciences
Press, Columbus, OH.
Dunnett, C. W. (1955). A multiple comparison procedure for comparing several treatments with a
control. J. Amer. Statist. Assoc. 50, 1096-1121.
Fabian, V. (1962). On multiple decision methods for ranking population means. Ann. Math. Statist.
33, 248-254.
Gibbons, J. D., Olkin, I. and Sobel, M. (1977). Selecting and Ordering Populations: A New Statistical
Methodology. Wiley, New York.
Gupta, S. S. (1956). On a decision rule for a problem in ranking means. Mimeograph Series No.
150, Institute of Statistics, University of North Carolina, Chapel Hill, NC.
Gupta, S. S. (1963a). On a selection and ranking procedure for gamma populations. Ann. Inst. Statist.
Math. 14, 199-216.
Gupta, S. S. (1963b). Probability integrals of the multivariate normal and multivariate t. Ann. Math.
Statist. 34, 792-828.
Gupta, S. S. (1965). On some multiple decision (selection and ranking) rules. Technometrics 7,
225-245.
Gupta, S. S. and Huang, D.-Y. (1980). A note on optimal subset selection procedures. Ann. Statist.
8, 1164-1167.
Gupta, S. S. and Huang, D.-Y. (1981). Multiple Decision Theory: Recent Developments. Lecture Notes
in Statistics, Vol. 6, Springer, New York.
Gupta, S. S., Huang, D.-Y. and Huang, W.-T. (1976). On ranking and selection procedures and tests
of homogeneity for binomial populations. In: S. Ikeda, T. Hayakawa, H. Hudimoto, M. Okamoto,
Selection and ranking procedures in reliability models 155
M. Siotani and S. Yamamoto, eds., Essays in Probability and Statistics, Shinko Tsusho Co. Ltd.,
Tokyo, Japan, Chapter 33, 501-533.
Gupta, S. S., Huang, D.-Y. and Nagel, K. (1979). Locally optimal subset selection procedures based
on ranks. In: J. S. Rustagi, ed., Optimizing Methods in Statistics, Academic Press, New York,
251-260.
Gupta, S. S., Huang, D.-Y. and Panchapakesan, S. (1984). On some inequalities and monotonicity
results in selection and ranking theory. In: Y. L. Tong, ed., Inequalities in Statistics and Probability,
IMS Lecture Notes--Monograph Series, Vol. 5, 211-217.
Gupta, S. S., Huang, W. T. (1984). On isotonic selection rules for binomial populations better than
a standard. In: A. M. Abuammoh, E. A. Ali, E. A. El-Neweihi and M. Q. E1-Osh, eds.,
Developments in Statistics and lts Applications, King Sand Univ. Library, Riyadh, 89-112.
Gupta, S. A. and Kim, W.-X. (1984). A two-stage elimination type procedure for selecting the largest
of several normal means with a common unknown variance. In: T. J. Santner and A. C. Tamhane,
eds., Design of Experiments: Ranking and Selection, Marcel Dekker, New York, 77-93.
Gupta, S. S. and Leu, L.-Y. (1986). Isotonic procedures for selecting populations better than a
standard: two-parameter exponential distributions. In: A. P. Basu, ed., Reliability and Quality
Control, Elsevier Science Publishers B.V., Amsterdam, 167-183.
Gupta, S. S. and Liang, T.-C. (1987). Locally optimal subset selection rules based on ranks under
joint type II censoring. Statistics and Decisions 5, 1-13.
Gupta, S. S. and Lu, M.-W. (1979). Subset selection procedures for restricted families of probability
distributions. Ann. Inst. Statist. Math. 31, 253-252.
Gupta, S. S. and McDonald, G. C. (1982). Nonparametric procedures in multiple decisions (ranking
and selection procedures). In: B. V. Gnedenko, M. L. Puri and I. Vincze, eds., Colloquia
Mathematica Societatis Janos Bolyai, 32: Nonparametric Statistical Inference, Vol. I, North-Holland,
Amsterdam, 361-389.
Gupta, S. S., Nagel, K. and Panchapakesan, S. (1973). On the order statistics from equally cor-
related normal random variables. Biometrika 60, 403-413.
Gupta, S. S. and Panchapakesan, S. (1972). On a class of subset selection procedures. Ann. Math.
Statist. 43, 814-822.
Gupta, S. S. and Panchapakesan, S. (1974). Inference for restricted families: (a) multiple decision
procedures; (b) order statistics inequalities. In: F. Proschan and R. J. Serfling, eds., Reliability and
Biometry: Statistical Analysis of Lifelength, SIAM, Philadelphia, 503-596.
Gupta, S. S. and Panchapakesan, S. (1975). On a quantile selection procedure and associated
distribution of ratios of order statistics from a restricted family of probability distributions. In: R.
E. Barlow, J. B. Fussell and N. D. Singpurwalla, eds., Reliability and Fault Tree Analysis: Theoretical
and Applied Aspects of System Reliability and Safety Assessment, SIAM, Philadelphia, 557-576.
Gupta, S. S. and Panchapakesan, S. (1979). Multiple Decision Procedures: Theory and Methodology of
Selecting and Ranking Populations. Wiley, New York.
Gupta, S. S. and Panchapakesan, S. (1985). Subset selection procedures: review and assessment.
Amer. J. Management Math. Sci. 5, 235-311.
Gupta, S. S. and Santner, T. J. (1973). On selection and ranking procedures--a restricted subset
selection rule. Proceedings of the 39th Session of the International Statistical Institute, Vol. 45, Book I,
478-486.
Gupta, S. S. and Sobel, M. (1958). On selecting a subset which contains all populations better than
a standard. Ann. Math. Statist. 29, 235-244.
Gupta, S. S. and Sobel, M. (1960). Selecting a subset containing the best of several binomial
populations. In: I. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow and H. B. Mann, eds.,
Contributions to Probability and Statistics, Stanford University Press, Stanford, Chapter 20, 224-248.
Gupta, S. S. and Sobel, M. (1962a). On selecting a subset containing the population with the smallest
variance. Biometrika 49, 495-507.
Gupta, S. S. and Sobel, M. (1962b). On the smallest of several correlated F-statistics. Biometrika 49,
509-523.
Gupta, S. S. and Yang, H.-M. (1984). Isotonic procedures for selecting populations better than a
control under ordering prior. In: J. K. Ghosh and J. Roy, eds., Statistics: Applications and New
156 S. S. Gupta and S. Panchapakesan
Directions: Proceedings of the Indian Statistical Institute Golden Jubilee International Conference,
Indian Statistical Institute, Calcutta, 279-312.
Hoel, D. G., Sobel, M. and Weiss, G. H. (1975). A survey of adaptive sampling for clinical trials.
In: R. M. Elashoff, ed., Perspectives in Biometry, Academic Press, New York, 29-61.
Hooper, J. H. and Santner, T. J. (1979). Design of experiments for selection from ordered families
of distributions. Ann. Statist. 7, 615-643.
Huang, D.-Y. and Panchapakesan, S. (1982). Some locally optimal subset selection rules based on
ranks. In: S. S. Gupta and J. O. Berger, eds., Statistical Decision Theory and Related Topics--III,
Vol. 2, Academic Press, New York, 1-14.
Kim, W.-C. and Lee, S.-H. (1985). An elimination type two-stage selection procedure for exponential
distributions. Comm. Statist.--Theor. Meth. 14, 2563-2571.
Kingston, J. V. and Patel, J. K. (1980a). Selecting the best one of several Weibull populations. Comm.
Statist. A--Theory Methods 9, 383-398.
Kingston, J. V. and Patel, J. K. (1980b). A restricted subset selection procedure for Weibull
distributions. Comm. Statist. A--Theory Methods 9, 1371-1383.
Lawrence, M. J. (1975). Inequalities for s-ordered distributions. Ann. Statist. 3, 413-428.
Lehmann, E. L. (1963). A class of selection procedures based on ranks. Math. Annalen 150, 268-275.
Milton, R. C. (1963). Tables of equally correlated multivariate normal probability integral. Technical
Report No. 27, Department of Statistics, University of Minnesota, Minneapolis, MI.
Nagel, K. (1970). On subset selection rules with certain optimality properties. Ph.D. Thesis (also
Mimeograph Series No. 222), Department of Statistics, Purdue University, West Lafayette, IN.
Panchapakesan, S. and Santner, T. J. (1977). Subset selection procedures for Ap-superior popula-
tions. Comm. Statist. A--Theory Methods 6, 1081-1090.
Patel, J. K. (1976). Ranking and selection of IFR populations based on means. J. Amer. Statist. Assoc.
71, 143-146.
Paulson, E. (1952). On the comparison of several experimental categories with a control. Ann. Math.
Statist. 23, 239-246.
Raghavachari, M. and Starr, N. (1970). Selection problems for some terminal distributions. Metron
28, 185-197.
Rizvi, M. H. and Sobel, M. (1967). Nonparametric procedures for selecting a subset containing the
population with the largest ~-quantile. Ann. Math. Statist. 38, 1788-1803.
Robbins, H. (1952). Some aspects of the sequential design of experiments. Bull. Amer. Math. Soc.
58, 527-535.
Robbins, H. (1956). A sequential design problem with a finite memory. Proc. Nat. Acad. Sci. U.S.A.
42, 920-923.
Santner, T. J. (1975). A restricted subset selection approach to ranking and selection problems. Ann.
Statist. 3, 334-349.
Saunders, I. W. and Moran, P. A. P. (1978). On the quantiles of the gamma and F distributions.
J. AppL Prob. 15, 426-432.
Sobel, M. (1967). Nonparametric procedures for selecting the t populations with the largest
c~-quantiles. Ann. Math. Statist. 38, 1804-1816.
Sobel, M. and Huyett, M. J. (1957). Selecting the best one of several binomial populations. Bell
System Tech. J. 36, 537-576.
Zwet, W. R. van (1964). Convex Transformations of Random Variables. Mathematical Center,
Amsterdam.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics. Vol. 7 | ~'~
© Elsevier Science Publishers B.V. (1988) 157-174
0. Introduction
It is obvious that reliability theory has used a great variety of mathematical and
statistical tools to help achieve needed results. These include: total positivity,
majorization and Schur functions, renewal theory, Bayesian statistics, isotonic
regression, Markov and semi-Markov processes, stochastic comparisons and
bounds, convexity theory, rearrangement inequalities, optimization theory--the
list is almost endless•
The question now arises: Has reliability theory reciprocated--that is, has
reliability theory made any contributions to the development of any of the
mathematical and statistical disciplines listed above? The answer is a definite Yes.
In this article we shall show that in the course of solving reliability problems,
theoreticians have developed new results in some of the disciplines above, of
direct value to the discipline and having application in other branches of statistics
and mathematics•
K
[;::;; ;m] g(xl, Yl)
K(X2, Yl)
K ( X l , Y2) "'" K(x1, Ym) >I
K ( x 2 , Y2) "" " K ( x 2 , Y,,,) O.
* Research supported by the Air Force Office of Scientific Research Grant AFOSR 82-K-0007.
157
158 P. J. Boland and F. Proschan
PROOF. The result follows from the 'Basic Composition Formula' (see Karlin
(1968) for a proof):
The theory of totally positive kernels and P61ya frequency functions has been
extensively applied in several domains of mathematics, statistics, economics, and
mechanics. In particular to give but a few examples in the theory of reliability, the
techniques of total positivity have been useful in developing properties of life
distributions with monotone failure rates and in the study of some notions of
component dependence, P61ya frequency functions of order 2 have been helpful
in determining optimal inspection policies, and the variation diminishing property
has been used in establishing characteristics of certain shock models. Reliability
theory has in turn, however, been the motivating force behind some important
developments in the theory of total positivity itself. A good example is the follow-
ing result (see Karlin and Proschan, 1960):
THEOREM 1.2. Let f l , f2, ... be any sequence of densities of nonnegative random
variables, where each f is PF r Then the n-fold convolution g(n, x) = f l * r E * " " *
fn(x) is TP~ in the variables n and x, where n ranges over 1, 2, ... and x traverses
the positive real line.
A similar total positivity result for the first passage time probabilities of the
partial sum process can be proved in the more general case when the random
variables range over the whole real line.
THEOREM 1.3. Let f l , f z , ... be any sequence of PF~ densities of random variables
X l , X 2 . . . . respectively, which are not necessarily non-negative. Consider the first
passage probability for x positive:
, ..... . 1 ]
for n = 1, 2 . . . . . Then h(n, x') is TPr, where n ranges over 1, 2, . . . , and x traverses
the positive real line.
Theorems 1.2 and 1.3 were initially inspired by certain models in reliability and
inventory theory, and these results in turn motivated Karlin (1964) to characterize
new classes of totally positive kernels and develop new applications in (for ex-
ample) discrete Markov chains. Typical of the results of Karlin (1964) are the
following two propositions (see also Karlin, 1968, p. 43):
PROPOSITION 1.5. Let ~ be a TP r Markov chain. Let Finjo denote the probability
that the first passage into the set of states <<-Jooccurs at the nth transition when the
initial state of the process is i > Jo. Then FT.jo is TP r in the variables n >i 1 and i > Jo.
160 P.J. Boland and F. Proschan
We now briefly trace the development leading to Theorems 1.2 and 1.3,
beginning with a basic problem in reliability theory that Black and Proschan
(1959) consider. (See also Proschan (1960) and Barlow and Proschan (1965) for
related problems).
Suppose that a system is required to operate for the period [0, to]. When a
component fails, it is immediately replaced by a spare component of the same
type if one is available. The system fails if no spare is available. Only the originally
supplied spares may be used for replacement during the period [0, to]. Assume
that the system uses k different types of components. At time 0, for each
i = 1, ..., k there are d r 'positions' in the system which are filled with components
of type i. By 'position (i, j)' we mean the jth location in the system where a
component of type i is used. Components of the same type in different positions
may be subject to varying stresses, and so we assume that the life of a component
in position (i, j ) has density function f j . Each replacement has the same life
distribution as its predecessor, and component lives are assumed to be mutually
independent. Let Pr(nr) be the reliability during [0, to] of the ith part of the system
(that is the subsystem consisting of the di components of type i), assuming that
n; spares of type i are available for replacement. The problem is to determine the
'spares kit' n = ( n ~ , . . . , nk) which will maximize the reliability of the system
P~n) = I~/k=l er(nr) during [0, to] subject to a cost constraint of the form
Y~r= 1 crnr <~ C (where cr > 0 for all i = 1, . . . , k).
A vector n o = (n °, n °, ..., n°) is an undominated spares allocation if whenever
O k k
P(n) > e(n ), then Y.r= l cinr > Y~= l Crn°. Black and Proschan (1959) consider
methods for quickly generating families of undominated spares allocations, which
can then be used to solve (approximately) the above problem. One of their
procedures is to start with the cheapest cost allocation (0, 0 . . . . . 0), and succes-
sively generate more expensive allocations as follows: If the present allocation is
n, determine the index io for which
[logPr(nr + 1) - logP~(n¢)]/G (i = 1. . . . . k)
is a maximum (in the case of ties the lowest such index is taken). The next
allocation is then n' = (n 1. . . . , nio_ j, n;o + 1, n;o + 1. . . . , nk). Black and Proschan
observe that the procedures they describe generate undominated allocations if each
Pc(n) is log concave in n. They are able to verify this directly in the case where
the component lives in the ith part of the system are exponentially distributed with
parameter 2~.
Note that logP;(n) is concave in n if and only if (Pr(n + 1)/Pr(n)) is a decreasing
function of n, or equivalently that Pr(n) is a PF a sequence. Let N o. for j = 1, . . . , d r
be the random variable indicating the number of replacements of type i needed
at position (i, j ) in the interval [0, to]. Proschan (1960) is able to show that iff.j.(t)
satisfies the monotone likelihood ratio property for translations (equivalently that
fj(t) is a PF 2 function), then f~n)(t) is a TPa function in the variables n and t
(where f;~n) is the n-fold convolution of f j with itself). Judiciously using
Theorem 1.1 on convolutions of totally positive functions, one is then able to
The impact of reliability theory on mathematics and statistics 161
The notion of associated random variables is one of the most valuable contribu-
tions to statistics that has been generated as a result of reliability theory con-
siderations.
We consider two random variables to be in some sense associated if they are
positively correlated, that is cov(S, T) >/0. A stronger requirement is c o v ( f ( S ) ,
g ( T ) ) >1 0 for all nondecreasing f and g. Finally if c o v ( f ( S , T), g ( S , T)) >I 0 for
all f and g nondecreasing in each argument, we have still stronger version of
association. Esary, Proschan and Walkup (1967) generalize this strongest version
of association to the multivariate case in defining random variables T1, . . . , T~ to
be associated if c o v ( f ( T ) , g ( T ) ) >>,0 for all nondecreasing functions f and g for
which the covariance in question exists. Equivalent definitions of associated
random variables result if the functions f and g are taken to be increasing and
either (i) binary or (ii) bounded and continuous.
Association of random variables satisfies the following desirable multivariate
properties:
(P1) Any subset of a set of associated random variables is a set of associated
random variables.
(P2) If two sets of associated random variables are independent of one another,
then the union of the two sets is a set of associated random variables.
(P3) Any set consisting of a single random variable is a set of associated random
variables.
(P4) Increasing functions of associated random variables are associated.
(Ps) A limit in distribution of a sequence of sets of associated random variables
is a set of associated random variables.
Note that properties P3 and P2 imply that any set of independent random varia-
bles is associated. This fact, together with property P4 enables one to generate
many practical examples of associated random variables. In the special case when
dealing with binary random variables, one can readily show that the binary
random variables X~, . . . , X n are associated if and only if 1 - X ~ ,
1 - X 2 . . . . . 1 - X n are associated.
Many interesting applications may be obtained as a consequence of the follow-
ing result about associated random variables:
162 P. J. Boland and F. Proschan
k
P[S1 <~ S1 . . . . ' Sk <~"Sk] ~ H P [ S i <'~Si]
i=l
and
k
P[S1 > Sl' " ' ' ' Sk > Sk] >~ H P[Si > si]
i=l
f o r all s = (s 1. . . . , Sk)~ R ~.
k
P [ T v , 1 <~ ti~, . . . , Tu, ] <~ t~] >/ I-I P[Tto] <~ t~]
j=l
and
k
P [ T v , I > til, " " , 'Tvkl > tik] >/ I ] P[T[,)1 > tij]
j=l
f o r every choice o f 1 <~ i~ < • • • < ik <~ n and ti, < • • • < tik.
F
F(s, . . . . , sin) = 1 - exp [ - £ 2is i - Z 2u max(s,., sj)
k i= 1 i<j
They point out that if this is the distribution function of the random variables
S 1. . . . . S=, then there exist independent exponential random variables
T l . . . . , T , such that Sj = m i n { T i : i e A j } where A j c {1, 2, . . . , n}. The random
variables S 1. . . . , S= are associated and therefore using Theorem 2.1, we can
The impact of reliability theory on mathematics and statistics 163
show that
or in other words that the chance of no errors of the first kind is greater following
the standard experimental procedure than if separate experiments had been per-
formed. This result is an immediate consequence of Theorem 2.1 once it is
observed that F 1 and F z are nondecreasing functions of the associated random
variables qx, q2, q3 1
The concept of associated random variables has proved to be a useful tool in
various areas of operations research. Shogan (1977) uses properties of associated
random variables to construct bounds for the stochastic activity duration of
PERT network. Heidelberger and Inglehart (1979) use associativity to construct
a set of sufficient conditions which guarantee that the dependent simulations of
a stochastic system produce a variance reduction over independent simulations.
Niu (1981) makes use of association in studying queues with dependent inter-
arrival and service times.
The notion of association of random variables is just one of many notions of
multivariate dependence. Lehmann (1966) introduces several concepts of bivariate
dependence, the strongest of which is TP z dependence ((S, T) are TP 2 dependent
if the joint probability density (or in the discrete case joint frequency function)
f ( s , t) is totally positive of order 2). For a discussion concerning the relationship
164 P. J. Boland and F. Proschan
Therefore h ( p ) is s-shaped (crosses the diagonal at most once and always from
below), a property which is crucial in constructing relay circuits of arbitrarily high
reliability. Birnbaum, Esary and Saunders (1961) generalize this result of Moore
The impact of reliability theory on mathematics and statistics 165
Esary and Proschan also use Theorem 2.5 to construct upper and lower bounds
for t h e reliability of a coherent structure in terms of the minimal paths and
minimal cut sets of the structure.
3. Renewal theory
Renewal theory has its origins in the study of self-renewing aggregates and
especially in actuarial science. Today we view the subject more generally as the
study of functions of independent identically distributed nonnegative random
variables which represent the successive intervals between renewals of a process.
The theory is applied to a wide variety of fields such as risk analysis, counting
processes, fatigue analysis, inventory theory, queuing theory, traffic flow, and
reliability theory. We will summarize a few of the more important and basic ideas
in renewal theory (for a more complete treatment consult Smith (1958), Cox
(1962), Feller (1966), Ross (1970), or Karlin and Taylor (1975)) and then indicate
some of the contributions to this area arising from reliability theory.
By a renewal process we will mean a sequence of independent identically
distributed nonnegative random variables X1, X 2 . . . . . which are not all zero with
probability one. We let F be the distribution function of X1, and F (k) will denote
the k-fold convolution of F with itself. The kth partial sum S k = X 1 + • • • + X k is
the kth renewal point and has distribution function F (k). For convenience we
define F (°) by F(°)(t) = 1 for t >i 0 and zero otherwise. Renewal theory is primarily
concerned with the number N(t) of renewals in the interval [0, t]. N(t), the
renewal random variable, is the maximum value of k for which Sk <~ t, with the
understanding that N(t)= 0 if X ~ > t . It is clear that P ( N ( t ) = n ) =
F(n)(t) - F (n+ 1)(0 and e ( N ( t ) >>.n ) = F(")(t). The process {N(t): t >/0} is known
as a renewal counting process.
The renewal function M(t) is defined to be the expected number of renewals
in [0, t], that is M(t) = E(N(t)). Since M(t) = E(N(t)) = 2 k~= l k P [ N ( t ) = k] =
oo
~=1~ P[N(t) >t k], it follows that M(t) = Zk= ~ FCk)(t) and moreover that M(t) =
~o- [1 + M ( t - x)] dF(x) (this latter identity being known as the fundamental
renewal equation). In spite of the fact that a closed functional form for M(t) is
known for only a few special distributions F, the renewal function M(t) plays a
central role in renewal theory.
166 P. J. Boland and F. Proschan
THEOREM 3.1. I f F has mean #i, then N ( t ) / t ~ 1/# 1 almost surely as t--* oo.
Note that important as these results may be, they are, with the exception of
Theorem 3.2 (i), asymptotic in nature.
In their comparison of replacement policies for stochastically failing units,
Barlow and Proschan (1964) obtain several new renewal theory inequalities. An
age replacement policy is one whereby a unit is replaced upon failure or at age T,
a specified constant, whichever comes first. Under a block replacement policy a
unit is replaced upon failure and at times T, 2T, 3T, .... It is assumed that
failures occur independently and that the replacement time is negligible. There are
advantages for both types of policy, and hence it is of interest to compare the two
types stochastically with respect to numbers of failures, planned replacements and
removals (a removal is a failure or a planned replacement). In many situations it
will be assumed that the life distribution of a unit belongs to a monotone class
such as the IFR (DFR) class (F is IFR if it has increasing (decreasing) failure
rate). It is clear that the evaluation of replacement policies depends heavily on the
theory of renewal processes.
Suppose we let N(t) indicate the number of renewals in [0, t] due to replace-
ments at failure, N*(t) be the number of failures in [0, t] under a block policy,
and N*(t) the number of failures in [0, t] under an age policy. Barlow and
Proschan (1964) prove the following result stochastically comparing these random
variables:
The following bounds on the renewal function M(t) = E(N(t)) are an immediate
consequence:
The impact of reliability theory on mathematics and statistics 167
By considering the number of failures and the number of removals per unit of
time as the duration of the replacement operation becomes indefinitely large,
Barlow and Proschan (1964) obtain the following simple useful bounds on the
renewal function for any F, and an improvement on these bounds for the IFR
(DFR) case (these bounds were conjectured by Bazovsky (1962)):
THEOREM 3.5. (i) M(t) >~ t/S o i ( x ) d x - 1 >>.t/# 1 - 1 for all t >~ O.
(ii) I f F is IFR (DFR), then M(t) <~( >~) tF(t)/ S o if(x) d x <<.(>1) t/l~ 1for all t >~O.
THEOREM 3.6. I f F is IFR (DFR), then Var(N(t)) ~<(>~)M(t), and this inequality
is sharp.
The renewal theory implications of the work of Barlow and Proschan (1964)
provide the key tool in the probabilistic interpretation of Miner's rule given by
Birnbaum and Saunders (1968) and Saunders (1970). Miner's rule (Miner, 1945)
is a deterministic formula extensively used in engineering practice for the cumula-
tive damage due to fatigue. Prior to the work of Birnbaum and Saunders, Miner's
rule was supported by empirical evidence but had very little theoretical justifi-
cation. Birnbaum and Saunders investigate models for stochastic crack growth
with incremental extensions having an increasing failure rate distribution. The
result that for an IFR distribution function F the inequality t/I21 - 1 <~M(t) <~ t/[.t 1
holds, is used to prove that T/121 -- 1 ~ ~ 1"/121 where ]A1 is the expected crack
increment per cycle, z is the expected crack length at which failure occurs and 7
is the expected number of loading cycles to failure. This in turn is used to show
that under certain conditions of dependence on load, Miner's rule does yield the
mathematical expectation of fatigue life. Saunders (1970) extends some of these
results by weakening the model assumptions, in particular by assuming that the
IFR assumption for the crack growth can be relaxed to assuming that F be new
better than used in expectation (NBUE), that is #l > So ff(t + x ) / i ( t ) d x for all
t >~ 0 such that F(t) > 0.
Marshall and Proschan (1972) determine the largest classes of life distributions
for which age and block replacement policies diminish, either stochastically or in
expected value, the number of failures in service. In doing so, they give the first
168 P. J. Boland and F. Proschan
THEOREM 3.7. N(s) * N(t) <~ (>~)N(s + t)for all s, t >~ 0 ¢~ F is NBU (NWU).
Straub (1970) is interested in bounding the probability that the total amount of
insurance claims arising in a fixed period of time does not exceed the amount t
of premiums collected. Letting F(t) be the distribution function for the individual
claims amount, Straub desires bounds for ff(')(t)= P ( N ( t ) < n). Here we may
interpret N(t) as the maximum value of k such that the first k claims sum to a
total ~<t. Motivated by the use of tools in reliability theory and in particular in
the work of Barlow and Marshall on bounds for classes of monotone distribu-
tions, Straub establishes the following important result (see Barlow and Proschan,
1981):
THEOREM 3.8. Let F be a continuous distribution function with hazard function
R (t) = - logif(t).
(a) I f F is NBU (NWU), then
"-' (R(t))J
P(N(t)<n)>/(<<,) ~ e-n(t) fort>/O,n= 1,2 .....
.j=o j!
, - l [nR(t/n)]j
P(N(t) < n) <~ (>I) F, e-'R(t/n) for t >1 O, n = 1, 2, . . . .
j=o j!
The bounds for the renewal function established by Barlow and Proschan
(1964) motivate Marshall (1973) to investigate the existence of 'best' linear bounds
for M(t) ('best' is interpreted to mean the sharpest bounds which when iterated
in the fundamental renewal equation converge monotonically to M(t) for all t).
Esary, Marshall and Proschan (1973) establish properties of the survival
function of a device subject to shocks and wear. One of their principal tools is
the result that [Ftkl(x)] 1/~" is decreasing in k = 1, 2, ..., for any distribution
function F such that F(x) = 0 for x < 0. This result, which is equivalent to the
following property of the renewal random variable N(t), can be used to demon-
strate monotonicity properties of first passage time distributions for certain
Markov processes.
The impact o f reliability theory on mathematics and statistics 169
THEOREM 3.9. Let N(t) denote the number of renewals in [0, t] for a renewal
process. Then [P(N(t) >~k)] 1/k is decreasing in k = 1, 2 . . . . .
THEOREM 3.10. (a) I f F is DFR, then the renewal function M(t) is concave. (b) I f
F is IMRL, then M(t) - (t/#~ - 1) is increasing in t >~O. (Note however that M(t)
is not necessarily concave.)
Hardy, Littlewood, and P61ya (1952) show that x > m y if and only if there exists
a doubly stochastic matrix H such that y = xlI. Schur functions are real valued
functions which are monotone with repsect to the partial ordering of majorization.
A function h with the property that x > m y ~ h(x)>i ( ~ ) h ( y ) is called Schur-
convex (Schur-concave). A convenient characterization of Schur-convexity (-con-
cavity) is provided by the Schur-Ostrowski condition, which states that a dif-
ferentiable permutation invariant function h defined on R" is Schur-convex
(Schur-concave) if and only if
For an excellent and extensive treatment of the theory of majorization, the reader
should consult Marshall and Olkin (1979).
170 P.J. Boland and F. Proschan
Considering the particular case where R '1 . . . . . R ' , one obtains the useful
bound hl,(pl, . . . , p,)>1 hk(Pc, . . . , P c ) for k = 1, ..., n, where Pc is the geo-
metric mean (!q~ pt) 1/'.
Although a large collection of theory and methods exists for order statistics
from a single underlying distribution, a relatively small set of results is available
for the case of order statistics from underlying heterogeneous distributions. In as
much as the time to failure of a k out of n system of independent components
with respective life distributions F 1. . . . . F, corresponds to the (n - k + 1)th order
statistic from the set of underlying heterogeneous distributions {F 1. . . . , Fn},
results about k out of n systems may be interpreted in terms of order statistics
from heterogeneous distributions.
Let us assume that Y/(Y; ) is an observation from distribution Fi (F;) and that
Ri(x ) = - l o g f f i ( x ) (R~ (x) = -logff" (x)) is the corresponding hazard function for
i = 1. . . . , n. The ordered observations are denoted by YH ~ < ' ' ' ~ < Yt-~
(YI~I ~ < " " ~< YI-I)" A random variable Y is stochastically larger than Y'
( y >~st y , ) if Fr(x) <~Fy, (x) for all x. In the realm of order statistics, Theorem 4.1
yields the following result:
THEOREM 4.2. Let (/~I(X), . . . , Rn(X)) )-m(R11(X), ..., gn(x)) for all x >t O. Then
Pledger and Proschan (1971) obtain further results of this type for the case of
proportional hazards. We say that the distributions Fl, . . . , F,, F'l, . . . , F'n have
proportional hazards with constants of proportionality 21, ..., 2,, 2'1. . . . . 2" if
Ri(x ) = 2~R(x) and R ; ( x ) = 2;R(x) for some hazard function R ( x ) and all
i = 1, . . . , n. A consequence of Theorem 4.2 is the following:
COROLLARY 4.3. Let F 1. . . . . F,, F'I, . . . , F'n have proportional hazard func-
tions with 21 . . . . . 2n, 2 ' 1 , . . . , 2;, as constants of proportionality. I f (21 . . . . . 2n)
>m(2'1, . . . , 2"), then Y[1] =st Y[I] and Y[k] >~st Yil,] for k = 2, . . . , n.
Proschan and Sethuraman (1976) generalize Corollary 4.3 and show that under
the same stated conditions, Y = (Y1, . . . , Yn) > / s t Y ' = (Y'l . . . . . Y'n) ( y > s t y , if
and only if f ( Y ) / > s t f ( y , ) for all real valued increasing functions f of n variables).
For more on stochastic ordering the interested reader should consult Kamae,
Krengel and O'Brien (1977). Proschan and Sethuraman apply their result to study
the robustness of standard estimates of the parameter 2 in an exponential distribu-
tion (F(x) = 1 - e - a x ) when the observations actually come from a set of hetero-
geneous exponential distributions.
Other comparisons for k out of n systems are given by Gleser (1975), and
Boland and Proschan (1983). While investigating the distribution of the number
of successes in independent but not necessarily identical Bernoulli trials,
Hoeffding shows that
n
whenever Y~1Pi >~ k, and
THEOREM 4.4. hk(p) is Schur convex in the region where Z~pe>~ k + 1 and
Schur concave in the region where Z~ pi <~ k - 2.
THEOREM 4.5. h~(p) is Schur convex in [(k - 1)/(n - 1), 1] n and Schur concave
in [0, ( k - 1)/(n - 1)] n.
172 P. J. Boland and F. Proschan
Theorems 4.4 and 4.5 represent inequalities which have practical use in the
study of k out of n systems. However it should be clear that they are of more
general interest and have applications in particular in the areas of order statistics
and independent Bernoulli trials.
Barlow and Proschan (1965) show that the mean life of a series system with
IFR components exceeds (is greater than or equal to) the mean life of a similar
system with exponential components, assuming component mean lives match in
the two systems. The reverse ordering is shown to hold in the parallel case.
Solovyev and Ushakov (1967) extend these results to include comparisons with
systems of degenerate and truncated exponential distributions. Marshall and
Proschan (1970) more generally show that if the life distributions F,. and Gi
of corresponding components of a pair of series systems satisfy
~o P~(x)dx >~ ~o Gi(x) dx for all t~> 0, then the same kind of inequality holds for
the system life distribution. Similarly they show that the domination
~) if(x) dx ~> ~ ~ G(x) dx for all t t> 0 is preserved under the formation of parallel
systems, and that both of these types of domination are preserved under con-
volutions. Marshall and Proschan (1970) are (implicitly) working with the concept
of continuous majorization (see Marshall and Olkin (1979)). We say the life
distribution function F majorizes the life distribution function G (written F >m G)
if #F = ~ o f f ( x ) d x = ~ o - G ( x ) d x = # a and ~ f f ( x ) d x > > , ~ - G ( x ) d x for all
t t> 0. As a by-product of their work on the mean life of series and parallel
systems, Marshall and Proschan establish the following result in the theory of
majorization.
THEOREM 4.6. Suppose that Fi > m G J o r each i = 1. . . . , n where Fe and G~ are life
distribution functions. L e t F(t) = F 1 * • " • * Fn(t) and G(t) = G 1 * • ' • * Gn(t) be n-fold
convolutions, with respective means #F and #G. Then
F>mG.
THEOREM 4.7. L e t F ( x l , . . . , xn) be a joint distribution function. I f xi 1 <~ " " <~ x J
for i = 1,..., n, then
J J
Z F(x{ ..... x~) >1 Z F(x~, x022(j) . . . . , xO~n(j))
j=l j 1
References
Barlow, R. E. and Proschan, F. (1964). Comparison of replacement policies, and renewal theory
implications. Ann. Math. Statist. 35, 577-589.
Barlow, R. E. and Proschan, F. (1965). Mathematical Theory of Reliability. Wiley, New York.
Barlow, R. E. and Proschan, F. (1981). Statistical Theory of Reliability and Life Testing. To Begin With,
Silver Spring, MD.
Bazovsky, I. (1962). Study of maintenance cost optimization and reliability of shipboard machinery.
ONR Contract No. Nonr-374000(00) (FBM), United Control Corp., Seattle, WA.
Birnbaum, Z. W., Esary, J. D. and Saunders, S. C. (1961). Multi-component systems and structures
and their reliability. Technometrics 3, 55-77.
Birnbaum, Z. W. and Saunders, S. C. (1968). A probabilistic interpretation of Miner's rule. S I A M
J. App. Math. 16, 637-652.
Black, G. and Proschan, F. (1959). On optimal redundancy. Oper. Res. 7, 581-588.
Boland, P. J. and Proschan, F. (1983). The reliability of k out of n systems. Ann. Prob. 11, 760-764.
Boland, P. J. and Proschan, F. (1984). An integral inequality with applications to order statistics.
To appear.
Brown, M. (1980). Bounds, inequalities, and monotonicity properties for some specialized renewal
processes. Ann. Probability 8, 227-240.
Brown, M. (1981). Further monotonicity properties for specialized renewal processes. Ann. Proba-
bility. 9, 891-895.
Cox, D. R. (1982). Renewal Theory. Wiley, New York.
Derman, C., Lieberman, G. J. and Ross, S. M. (1972). On optimal assembly of systems. Nay. Res.
Log. Quart. 19, 569-574.
Esary, J. D., Marshall, A. W. and Proschan, F. (1973). Shock models and wear processes. Ann. Prob.
1, 627-649.
Esary, J. D. and Proschan, F. (1963). Coherent structures Of non-identical components. Technometrics
5, 191-209.
Esary, J. D., Proschan, F. and Walkup, D. W. (1967). Association of random variables, with
applications. Ann. Math. Stat. 38, 1466-1474.
Feller, W. (1948). On Probability problems in the theory of counters. Courant Anniversary Volume.
Interscience, New York.
Feller, W. (1966). An Introduction to Probability Theory and Its Applications, Vol. II. Wiley, New York.
Fortuin, C. M., Kastelyn, P. W. a~d Ginibre, J. (1971). Correlation inequalities on some partially
ordered sets. Comm. Math. Phys. 22, 89-103.
Gleser, L. (1975). On the distribution of the number of successes in independent trials. Ann. Prob.
3, 182-188.
Hardy, G. H., Littlewood, J. E. and P61ya. (1952). Inequalities. Cambridge University Press, New
York.
Heidelberger, P. and Inglehart, D. L. (1979). Comparing stochastic systems using regenerative
simulation with common random numbers. Adv. Appl. Prob. 11, 804-819.
Hoeffding, W. (1956). On the distribution of the number of successes in independent trials. Ann.
Math. Stat. 27, 713-721.
Joag-dev, K., Perlman, M. D. and Pitt, L. D. (1983). Association of normal random variables and
Slepian's inequality. Ann. Prob. 11, 451-455.
Kamae, T., Krengel, U. and O'Brien, G. L. (1977). Stochastic inequalities on partially ordered spaces.
Ann. Probab. 5, 899-912.
Karlin, S. (1964). Total positivity, absorption probabilities and applications. Trans. Amer. Math. Soc.
III, 33-107.
Karlin, S. (1968). Total Positivity. Stanford University Press, Stanford, CA.
Karlin, S. and Proschan, F. (1960). P61ya type distributions of convolutions. Ann. Math. Stat. 31,
721-736.
Karlin, S. and Taylor, H. M. (1975). A First Course in Stochastic Processes, 2nd edition. Academic
Press, New York.
174 P. J. Boland and F. Proschan
Keilson, J. (1975). Systems of independent Markov components and their transient behavior. In: R.
E. Barlow, J. B. Fussel and N. D. Singpurwalla, eds., Reliability and Fault Tree Analysis. SIAM,
Philadelphia, PA, 351-364.
Kemperman, J. H. B. (1977). On the FKG-inequality for measures on a partially ordered space.
lndag. Math. 39, 313-331.
Kimball, A. W. (1951). On dependent tests of significance in the analysis of variance. Ann. Math.
Star. 22, 600-602.
Lehmann, E. L. (1966). Some concepts of dependence. Ann. Math. Stat. 37, 1137-1153.
Lorentz, G. G. (1953). An inequality for rearrangements. Amer. Math. Mon. 60, 176-179.
Marshall, A. W. and Olkin, I. (1967). A multivariate exponential distribution. J. Amer. Stat. Assoc.
62, 30-44.
Marshall, A. W. and Olkin, I. (1979). Inequalities: Theory of Majorization and Its Applications.
Academic Press, New York.
Marshall, A. W. and Proschan, F. (1970). Mean life of series and parallel systems. J. App. Prob. 7,
165-174.
Marshall, A. W. and Proschan, F. (1972). Classes of distributions applicable in replacement, with
renewal theory implications. In: L. LeCom, J. Neyman and E. L. Scott, eds., Proceedings of the 6th
Berkeley Symposium on Mathematical Statistics and Probability, Vol. I, University of California Press,
Berkeley, CA, 395-415.
Marshall, K. T. (1973). Linear bounds on the renewal function. SIAM J. App. Math. 24, 245-250.
Miner, M. A. (1945). Cumulative damage in fatigue. J. AppL Mech. 12, A159-A164.
Moore, E. F. and Shannon, C. E. (1956). Reliable circuits using less reliable relays. J. Franklin
Institute 262, part I 191-208 and part II 281-297.
Newman, C. M. and Wright, A. L. (1981). An invariance principle for certain dependent sequences.
Ann. Prob. 9, 671-675.
Niu, S. C. (1981). On queues with dependent interarrival and service times. Nay. Res. Log. Quart.
28, 497-501.
Pitt, L. D. (1982). Positively correlated normal random variables are associated. Ann. Prob. 10,
496-499.
Pledger, G. and Proschan, F. (1971). Comparisons of order statistics and of spacings from hetero-
geneous distributions. In: J. S. Rustagi, ed., Optimizing Methods in Statistics. Academic Press, New
York, 89-113.
Proschan, F. (1960). P6lya Type Distributions in Renewal Theory, with an Application to an Inventory
Problem. Prentice-Hall, Englewood, NJ.
Proschan, F. and Sethuraman, J. (1976). Stochastic comparisons of order statistics from hetero-
geneous populations, with applications in reliability theory. J. Mult. Anal 6, 608-616.
Robbins, H. (1954). A remark on the joint distribution of cumulative sums. Ann. Math. Stat. 25,
614-616.
Ross, S. M. (1970). Applied Probability Models with Optimization Applications, Holden-Day, San
Francisco.
Saunders, S. C. (1970). A probabilistic interpretation of Miner's rule. II. SlAM J. App. Math. 19,
251-265.
Shogan, A. W. (1977). Bounding distributions for a stochastic PERT network. Networks 7, 359-381.
Solovyev, A. D. and Ushakov, I. A. (1967). Some estimates for systems with components 'wearing
out'. (In Russian). Avtomat. i Vycisl. Tehn. 6, 38-44.
Smith, W. L. (1968). Renewal theory and its ramifications. J. Roy. Statist. Soc., Series B 20, 243-302.
P. R. Krishnaiah and C, R. Rao, eds., Handbook of Statistics, Vol. 7 1 1
.lk 1
© Elsevier Science Publishers B.V. (1988) 175-213
M. C. Bhattacharjee*
0.1. In recent times, Reliability theoretic ideas and methods have been used
successfully in several other areas of investigation with a view towards exploiting
concepts and tools, which have their roots in Reliability Theory, in other
settings to draw useful conclusions. For a purely illustrative list of some of these
areas and corresponding problems which have been so addressed, one may
mention: demography (bounds on the 'Malthusian parameter', reproductive value
and other related parameters in population growth models--useful when the
age-specific birth and death-rates are unknown or subject to error: Barlow and
Saboia (1973)), queueing theory (probabilistic structure of and bounds on the
stationary waiting time and queue lengths in single server queues: Kleinrock
(1975), Bergmann and Stoyan (1976), KollerstrOm (1976), Daley (1983)) and
economics ('inequality of distribution' and associated problems: Chandra and
Singpurwalla (1981), Klefsj6' (1984), Bhattacharjee and Krishnaji (1985)). In each
of these problems, the domain of primary concern and immediate reference is not
the lifelengths of physical devices/systems of such components or their failure-
logic structure per se but some phenomenon, possibly random, evolving in time
and space. Nevertheless, the basic reason behind the success of cross-fertilization
of ideas and methods in each of the examples listed above is that the concepts
and tools which owe their origin to traditional Reliability theory are in principle
applicable to non-negative (random) variables and (stochastic) processes generated
by such variables.
* Work done while the author was visiting the University of Arizona.
175
176 M. C. Bhattacharjee
bility researchers. Central to this is the idea of a preference ordering R among the
elements x, y,, ... of a finite set F. R is a relation among the elements of F such
that for any x, y ~ F, we say: x R y iff x is at least as preferred as y. Such a relation
R is required to satisfy the two axioms:
(A1) Transitivity: For all x, y, z t F; x R y and y R z ~ x R z.
(A2) Connectedness: For all x, y 6 F; either x R y or y R x or both.
Technically R is a complete pre-order on F; it is analogous to a relation such as
'at least as tall as' among a set of persons. Notice that we can have both x R y
and y R x but x ~ y. For a given F, it is sometimes easier to understand the
relation R through two other relations P, I defined as x P y ~*~ x is strictly
preferred to y; while x I y ,~ x and y are equally preferred (indifference). Then
note, (i) x R y ~:~ y ~ x, i.e., x R y is the negation o f y P x and (ii) the axiom (A2)
says: either x P y or y P x or x I y .
Now consider a society S = { 1, 2 . . . . . n} of n-individuals (voters), n >I 2 and
a finite set A of alternatives consisting of k-choices (candidates/policies/actions),
k > 2. Each individual i t S has a personal preference ordering R i on A satisfying
the axions (A1) and (A2). The problem is to aggregate all the individual
preferences into a choice for S as a whole. To put it another way, since R;
indicates how i 'votes', an 'election' 8 is a complete set of 'votes' {formally,
= {Ri:i~ S}) and since the result of any such election must amalgamate its
elements (i.e., the individual voter-preferences) in a reasonable manner into a
well-defined collective preference of the society S; such a result can be thought
of as another relation R* on A which, to be reasonable, must again satisfy the
same two axioms (A1) and (A2) with F = A.
Arrow conceptualizes the definition of a "voting system" as the specification of
a social preference ordering R* given S, A. There are many possible R* that one
can define including highly arbitrary ones such as R* = R~ for some i ~ S (such
an individual i, if it exists, is called a 'dictator'). To model real-world situations,
we require to exclude such unreasonable voting systems and confine ourselves to
those R* which satisfy some intuitive criteria of fairness and consistency. Arrow
visualized four such conditions, namely:
(C1) (Well-definedness). A voting system R * must be capable of a decision. For
any pair of alternatives a, b; there exists an 'election' for which the society prefers
a to b. [R* must be defined on the set of all n-tuples B = (R~ . . . . . Rn) of
individual preferences and is such that for all a, b in A, either a R* b or a ~ * b,
there exists an B such that b $ * a . ]
(C2) (Independence of Irrelevant Alternatives). R* must be invariant under
addition or delition of alternatives. [ I f A ' c A and o~ = {Ri: i t S} is any election,
then RI*, should depend only on {Ril A, : i t S} where Rtl A, (RI*, 1, respectively) is
the restriction of R; (R* respectively) to A ' . ]
(C3) (Positive Responsiveness). An increasing (i.e., nondecreasing) preference for
an alternative between two elections does not decrease its social preference.
[Formally, given S and A, let g = {R~:i~S} and g ' = { R ' ' i ~ S } be two
elections. If there exists an a t A such that
178 M. C. Bhattacharjee
xi(a,b)= 1 i f a R i b ,
= 0 if aI~ib. (1.1)
FA(a,b)= 1 i f a R * b ,
=0 if a ~ * b . (1.2)
IMPOSSIBILITY THEOREM (Arrow). There does not exist a social choice function
FA satisfying (A1), (A2) and (C1)-(C4).
Reliability applications in economics 179
To argue that the two axioms and four conditions are collectively inconsistent,
the first step is to show:
Thus the conditions (C1)-(C3) imply F = 4(x) for some monotone structure
function 4. The converse is trivial. []
The axioms (A1) and (A2) for voting systems translated to requirements on the
social choice function F(a, b) = 4(x(a, b)) become
(A1) Transitivity: F(a, b) = 1 = F(b, c) =~ F(a, c) = 1.
(A2) Connectedness: F(a, b)= 1 or 0.
Consider a pair of alternatives (a, b ) ~ A 2 such that F(a, b)= 4(x(a, b))= 1.
Borrowing the terminology of reliability theory, we will say
as an (a, b)-cut. Thus an (a, b)-path ((a, b)-cut, respectively) is any coalition, i.e.,
subset of individuals whose common 'non-preference of b relative to a' ('pre-
ference of b over a', respectively) is inherited by the whole society S. Obviously
such paths (cuts) always exist since the whole society S is always a path as well
as a cut for every pair of alternatives. When the relevant pair of alternatives (a, b)
is clear from the context, we drop the prefix (a, b) for simplicity and just refer
to (1.3) and (1.4) as path and cut. A minimal path (cut) is a coalition of which
no proper subset is a path (cut).
180 M. C. Bhattacharjee
To return to the main proof, notice that Lemma 1 limits the search for social
choice functions F = ~(x) to those monotone structure functions tp which satisfy
(A1), (A2) and (C4). A social choice function satisfies the connectedness axiom
(A2) iff for every pair of alternatives (a, b); there exists either a path or a cut,
according as F(a, b) = 1 or 0, whose members' common vote agrees with the
social choise F(a, b). The transitivity axiom (A1) that F(a, b)= 1 =
F(b, c) =~ F(a, c ) = 1 for each triple of alternatives (a, b, c) can be similarly
translated as: for each of the pairs (a, b), (b, c), (a, c); there exists a path, not
necessarily the same, which allow the cycle of alternatives a, b, c, to pass.
Let ~ ' be the class of monotone structure functions and set
= : { ~ J g : no two paths are disjoint},
~ * =: {q~ J / : intersection of all paths is nonempty},
=
(~-* respectively) are those monotone structures for which there is at least one
common component shared by any two paths (all paths, respectively). ~ is the
class of self-dual monotone structures for which every path (cut) is also a cut
(path).
Clearly i f * ~ ~. Also ~ c ~ ; for if not, then there exists two paths P~, /'2
(which are also cuts by self-duality) which are disjoint so that we then have a cut
P1 disjoint from a p a t h / 2 . This contradicts the fact that any two coalitions of
which one is a path and the other a cut must have at least one common com-
ponent, for otherwise it would be possible for a structure tp to fail (tp(x) = 0) and
not-fail ((p(x)~ 0) simultaneously violating the weU-definedness condition (C1).
Thus
c~ ~ * ~ ~ . (1.5)
(A2) ~ ~ ~ (1.6)
and hence q ~ ~ by (1.5). Which are the structures in (A2) that satisfy (A1)? We
show this is precisely ~ * , i.e., claim
~ (A1) = ~ * (1.7)
so that any admissible F = q~(x)~ ~ * . The final step is to show the property
defining ~ * and the no-dictator hypothesis (C4) are mutually inconsistent.
Reliability applications in economics 181
The following outlines the steps of the argument. For any pair (a, b) of alter-
natives, the society S obeying axiom (A2) must either decide 'b is not preferred
to a' (F(a, b)= q)(x(a, b))= 0) or its negation 'b is preferred to a'
(F(a, b) = ¢(x(a, b)) = 1). If the individual votes x(a, b) result in either of these
two social choices as it must, the dual response 1 - x(a, b) (which changes every
individual vote in x(a, b) to its negation) must induce the other; i.e., for each x,
Table 1
viz., xi(a, b) = 1 - xt(b, a), all i~ S, all (a, b); hence F(a, b) = qb(x(a, b)) =
~d(x(a, b) = 1 - ~p(1 - x(a, b)) = 1 - ~(x(b, a)) = 1 - F(b, a). Hence, for the
cycle of alternatives (a, b, c); from the last column of the above table, we have:
F(b, c) = 1 = F(c, a), but F(b, a) = 0; thus contradicting the transitiveness axiom
(A1). Hence all three paths must share a common component.
In the spirit of the above construction, an inductive argument can now similarly
show that if there are (j + 1) paths in all and if every set of j paths have a
common component, then so does the set of all (j + 1) paths; j = 1, 2 . . . . if (A1)
is to hold. Thus there is a component common to all paths, i.e., q ~ if*. Let i*
be such a component. Since i* belongs to every path, it is a one-component cut.
It is also a one component path, but the self-duality of qk That {i*} is both a path
and a cut says,
x,.=l(o) ~ ~(x)=l(0),
While unless there are at least two individual components (n >~ 2) the problem
of aggregation is vacuous, notice the role of the assumption that there are at least
three choices ( k > 2 alternatives) which places the transitiveness axiom in
perspective. There are real-life voting systems (social choice functions) which do
not satisfy (A1). One such example is the majority system R * such that
where
N ( a , b ) = {# of voters i ~ S with aRab} = ~ x~(a,b).
i=1
m = [½n] + 1 i f n o d d ,
= ½n i f n even.
Reliability applications in economics 183
where (0;, x):= (x I . . . . x,._ l, 0, xi+ 1. . . . . xn) and (li, x) is defined similarly.
Hence tp(.;, x) is the social choice given i's vote, i e S. Thus,
when relevance is translated in terms of social choice given i's vote; while
i ~ S is a dictator
q~(le, x(a, b) = 1, qb(Oi, x(a, b)) = O, all (a, b).
Let
2.1. The model and basic results. Denote a committee of n persons by N. Ele-
ments of N are called players. We can take N = {1, 2 . . . . . n} without loss of
generality. A coalition is any subset S of players, S ~ N. Each player votes yes
or no, i.e., for or against the proposition. A winning (blocking) coalition is any
Reliability applications in economics 185
coalition whose individual yes (no)-votes collectively ensure the committee passes
(falls) the proposition Let W be the set of winning coalitions and v: 2Jv~ {0, 1},
t h e binary coalition-value function
v(S) = 1 if S ~ W (S winning),
= 0 if s~ W (S is not winning). (2.1)
(if everyone votes 'no' ('yes'), the proposition fails (wins); and any coalition
containing a winning coalition is also a winning coalition) or, equivalently by an
ordered pair (N, v) where
where s = :[ S] = the number of voters in S and the sum is over all s such that
S is a swing for i.
186 M. C. Bhattacharjee
_ 7+ _ 7+ , (2.3)
Y~+~N7+ 2 n-1
where 7+ is the number of swings for i. The Banzhaff power index also has a
probability interpretation that we shall see later (Section 2.4).
If the indicator variable,
denotes i's vote and C l ( x ) = {x: x+ --- i} is the coalition of assenting players for
a realization x = (x 1, . . . , xn) of 2 n such voting configurations, then the outcome
function ¢: {0, 1}n~ {0, 1} of the voting game is
q,(x) = v ( C , ( x ) ) ,
where v is as defined in (2.1) and tells us whether the proposition passes or fails
in the committee. Note q/models the decision structure in the committee given its
rules, i.e., given the winning coalitions. In the stochastic version of a simple game,
the voting configuration X = (X 1, . . . , Xn) is a random vector whose joint distri-
bution determines the voting-function
v =:E~O(X) = P { $ ( X ) = 1},
the win probability of the proposition in the voting game. Sensitivity of v to the
parameters of the distribution of X captures the effects of individual players' and
their different possible coalitions' voting attitudes on the collective committee
decision for a specified decision structure ft.
When the players act independently with probabilities p = (Pl . . . . . Pn) of voting
'yes', the voting function is
v = h(p) (2.5)
for some h: [0, 1 ] n ~ [0, 1]. The function h is called Owen's multilinear extension
and satisfies (Owen, 1981):
h ( p ) = p~h(l~, p) + (1 - p+)h(O~, p ) ,
Oh
he(p) = : - - = h(l+, p) - h(0+, p ) , (2.6)
k ! ( n - k)!
P(Xr:(1 ) ..... X . ( k ) = 1, X . ( k + 1) . . . . . X u ( n ) = O) -
(n + 1)!
(2.8)
for any permutation (n(1), ..., n(n)) of the players. (2.8) is a description of
homogeneity of the players which Straffin uses to formulate (i) a power index and
(ii) an agreement index which is a measure of the extent to which a player's vote
and the outcome function coincide. He also considers the relationship between
these indices corresponding to the uniform prior and the prior
f ( p ) = constp(1 - p ) ; results we will fred more convenient to describe in a more
general format in the next section.
2.2. Implications of the reliability framework for voting games. F r o m the above
discussions, it is clear that voting games are conceptually equivalent to systems
of components in reliability theory. Table 2 is a list o f the dual interpretations of
several theoretical concepts in the two contexts:
Table 2
player component
committee system
winning (loosing) coalition patch (cut)
blocking coalition complement of a cut
outcome function structure function
voting function reliability function
multilinear extension reliability function with independent components
188 M. C. Bhattacharjee
Thus every voting game has an equivalent reliability network representation and
can consequently be analysed using methods of the latter. As an illustration
consider the following:
EXAMPLE. The simple game (N, IV) with a five player committee
N = {1, 2, 3, 4, 5} and winning coalitions IV as the sets
1 3
I O 5
2 4
of two parallel subsystems of two components each and a fifth component all in
series. We see that to win in the corresponding voting game, a proposition must
pass through each of two subcommittees with '50~o majority wins' voting rule and
then also be passed by the chairperson (component 5). The voting function of this
game when committee members vote 'yes' independently with a probability p (i.e.,
the version of Owen's multilinear extension in the i.i.d, case) is thus given by
the reliability function
of the above coherent structure. The minimal path sets of this structure are the
smallest possible winning coalitions, which are the four 3-player coalitions in IV.
Since the minimal cut sets are (1, 2), (3, 4) and (5), their complements
are the minimal blocking conditions which are the smallest possible coalitions B with
veto-power in the sense that their complements N \ B are not winning coalitions.
since ~b is self-dual. Thus ~O= (p and hence qJ is also self-dual. The latter in
particular implies the existence of a player who must be present in every winning
coalition (viz. (1.7)).
With the choice set restricted to two alternatives; Arrow's condition (C1) is
trivial, condition (C2) of irrelevant alternatives is vacously true and so is the
transitivity axiom (A1). Since ~O= tp, the condition (C1) says ~k(x) must be defined
for all x while axiom (A2) says ~k is binary. The condition of positive respon-
siveness (C3) holds ¢~- all supersets of winning coalitions are winning, built in the
definition of a voting game. Lemma 1 thus implies:
The first part of the above result is due to Ramamarthy and Parthasarathy
(1984). The social choice function analogy of the outcome function and its
coherence in the absence of dummies is new.
A dummy player is one whose exclusion from a winning coalition does not
destroy the winning property of the reduced coalition, i.e.,
theory, as they must. The voting funcion h(p) being a monotone (coherent)
structure's reliability function must be coordinatewise monotone: p<~p'
=~ h(p)<~ h(p') which has been independently recognized in the voting game
context (Owen, 1982). The Banzhaffpower index (2.3) is none other than the
structural importance of components in ~. Since research in voting games and
reliability structures have evolved largely independent of each other, this general
lack of recognition of their dualism has been the source of some unnecessary
duplication of effort. Every result in either theory has a dual interpretation in the
other, although they may not be equally meaningful in both contexts. The following
are some further well known reliability ideas in the context of independent or i.i.d.
components which have appropriate and interesting implications for voting games.
With the exception of 2.2.1 below, we believe the impact of these ideas have not
yet been recognized in the literature on voting games with independent or i.i.d.
players.
measures how crucial is i's vote in a game with outcome function ~k and random
voting probabilities. As an index of i's voting power, v; is defined for any
stochastic voting configuration X and has been used by Straffin within the homo-
geneity framework ((X~, . . . , X,) conditionally i.i.d, given p). We may call v; the
voting importance of i. If the players are independent, then
Vi = h i ( p )
in the notation of Section 2.1 (viz. (2.6)). Thus e.g., in the stochastic unanimity
game where all players must vote yes to pass a proposition, the player least likely
to vote in favor has the most voting importance. Similarly in other committee
decision structures, one can use vi to rank the players in order of their voting
importance. For a game with i.i.d, players, i's voting importance becomes the
function v; = hi(p) where he(p) = h(1 i, p) - h(O;, p) and h('i, o), h(p) denote the
corresponding versions of h(.i, p), h(p) respectively when p = (p . . . . . p). Since
in this case h'(p) = Y,i~Nhi(P), one can also use the proportional voting impor-
tance
v~* - vi _hi(P)
E j ~ N Vj h' ( p )
algorithm will similarly produce all minimal winning coalitions, the Banzhaff
power indices and the voting importances.
2.2.3. S-shapedness of the voting function for i.i.d, players with no dummies. This
follows from the M o o r e - S h a n n o n inequality (Barlow and Proschan, 1965)
dh
p(1 - p) ~ >~ h(p)(1 - h(p))
dp
for the reliability function of a coherent structure with i.i.d, components. Implica-
tions of this f a c t in the voting game context is probably not well known. In
particular the S-shapedness of the voting function implies that among all com-
mittees of a given size n, the k-out-of-n structure (lOOk~n% majority voting
games) have the sharpest rate of increase of the probability of a committee of n
i.i.d, players passing a bill as the players' common yes-voting probability in-
creases.
(M, V) is called the master-game and (Nj, Wj) the modules of the compound game
(N, W). The combinatorial aspects of compound voting games have been exten-
sively studied. Considering the equivalent reliability networks it is clear however
that if the component games (Nj, Wj) have structures ~, j = 1, ..., k, and the
master game (M, V) has structure tp; then the compound voting game (N, W) has
structure
Conversely the existence of some tp, ~k~, ..., ~bk satisfying this representation for
a given ~k can be taken as an equivalent definition of the corresponding master
game, component subgames and the accompanying player sets as the modular
sets of the original voting game. E.g., in the 5-player example at the beginning of
this section, clearly both subcommittees J1 = { 1, 2}, J2 - {3, 4} are modular sets
and the corresponding parallel subsystems are the subgame modules. Ramamur-
thy and Parthasarathy (1983) have recently exploited the results on modular
decomposition of coherent systems to investigate voting games in relation to its
component subgames (modules) and to decompose a compound voting game into
its modular factors (player sets obtained by intersecting maximal modular sets or
their complements with each other). Modular factors decompose a voting game
into its largest disjoint modules. The following is typical of the results which can
be derived via coherent structure arguments (Ramanurthy and Parthasarathy,
1983).
results are necessarily stronger than those that can be derived under the asso-
ciatedness hypothesis alone.
The remarkable insight behind Straffin's homogeneity assumption is that it
amounts to the voting configuration X being a finite segment of a special sequence
of exchangeable variables. The effect of this assumption is that the probability of
any voting pattern x -- (x~, . . . , x,) depends only on the size of the assenting and
dissenting coalitions and not on the identity of the players, as witness (2.8). One
can reproduce this homogeneity of players through an assumption more general
than Strattin's. Ramamurthy and Parthasarathy (1984) exploit appropriate relia-
bility ideas to generalize many results of Straffin and others, by considering the
following weakening of Straffin's assumption.
Since X l , 2 2 , . . . are binary; by the Finnetti's well known theorem, the voting
configuration's joint distribution has a representation
for some prior distribution F on [0, 1]; and the votes X 1 . . . . . X n are conditionally
independent given the 'yes' voting probability p. Straffin's homogeneity assump-
tion corresponds to an uniform prior for p, leading to (2.8). For a stochastic
voting game defined by its outcome (structure) function ~k, consider the power-
index
Ai = : e { x , = ¢ ( x ) } ,
pi =:cov(x;, q4x)),
t5 =: cov(X, q l ( X ) l p ) d F ( p ) .
)
Also, let
b = :cov(P, H ( P ) ) .
Here P is the randomized probability of voting 'yes' with prior F in (2.10). Note
b, tri are defined only under the general homogeneity assumption, while vi, A t and
Pi are well defined for every joint distribution of the voting configuration X. Recall
194 M. C. Bhattacharjee
that a power index measures the extent of change in the voting game's outcome
as a consequence of a player's switching his vote and an agreement index
measures the extent of coincidence of a player's vote and the final outcome. Thus
any measure of mutual dependence between two variables reflecting the voting
attitudes of a player and the whole committee respectively qualifies as an
agreement index. An analysis of the interrelationships of these indices provides an
insight into the interactions between players' individual level of command over the
game and the extent to which they are in tume with the committee decision and
ride the decisive bandwagon.
The agreement index A i is due to Rae (1979). Under (2.8), ve becomes Straffin's
power index and a e is proportional to an agreement index also considered by
Straffin. Note all the coefficients are non-negative. This is clear for ve and A e, and
follows Pc, ere and b from standard facts for associated r.v.s. (Barlow and
Proschan, 1975) which is weaker than the general homogeneity (GH) hypothesis.
The interesting results under the assumption of general homogeneity (Ramamurthy
and Parthasarathy, 1984) are
pe=ai+b,
2 b s ~ ) ~ tri >/
i~N ~0 h(p)(1 - h(p)) d F ( p ) ,
EXe=½ ~ A e = 2 o - j + 2 b + 1. (2.11)
The equality in the second assertion holds only under StralTm's homogeneity (SH)
assumption. This assertion follows by noting tre = ~ o1 P ( 1 - h(p))dF(p) under
GH, h'(p) = Y'e hi(P), termwise integration by parts in Y~etre with uniform prior to
conclude the equality and invoking the S-shapedness of h(p) for the bound.
The above relations in particular imply
(i) Under GH, i is dummy ¢~ a~ = 0. If the odds of each player voting yes and
no are equal under GH, i.e., if the marginal probability P(X e = 1) = ½; then we
also have, i dummy ¢:~ Pc--- b ~ A i = 2b + ½. Thus since ~5 is in a sense the
minimal affinity between a player's vote and the committee's decision, Straffin
suggests using 2a e (Ae - 2b - 1) as an agreement index.
(ii) Let w, l = 2 n - w be the number winning and losing coalitions. Since
hi(½) = fli (structural importance = Banzhaff power index) and h(1) = w/2"; taking
F as a point-mass at ½, (2.11) gives
Without the equal odds condition, the last relation in (2.11) has a more general
version that we may easily develop. Let n; = : .[ 1 p dF(p) = E X~ be the marginal
probability of i voting yes under general homogeneity. Then
Reliability applications in economics 195
1
A i = ~ P(X i = ~b(X) = j ) = E X~k(1., X ) + E((1 - X~)(1 - ~b(0e, X))
j=0
2.4. Influence indices and stochastic compound voting games. There are some
interesting relationships among members of a class of voting games via their
power and agreement indices. In the spirit of (2.10), consider a compound voting
game consisting of the two game modules
i.e., any majority (at least (m + 1) players) coalition wins. Replacing the player
- ( n + 2m + 1) in the majority game by the game G = (N, W), define the
compound game G~* = (N*, W*), where
N*=NwN,,= {1 . . . . . n , n + 1. . . . . n + 2 m } ,
W* = {S c N*" either ] S \ N I ~ m + 1 or/and
I S \ N I >~m, S n N ~ W}. (2.13)
a(k) b(n - k)
/'(X~ . . . . . X k = 1, Xk+~ . . . . = X. = 0)- (a + b) (") ' (2.15)
ab
a/(G: a, b ) = vi(G: a + 1, b + 1),
(a+b)(a+b+ 1)
power index), while the third assertion shows a relationship between voting impor-
tances in the compound game in (2.13) and the corresponding constituency game
under appropriate choice of voter-dependence in the two games.
Notice v~(G: m + 1, m + 1)--}fl;, the Banzhaff power-index in the constituency
game, since the case of players voting yes or no independently with equal odds
(p = ½) can be obtained by letting m ~ oo in the prior Fm+ ~.m + 1" Hence by
(2.16), in the composite game G* with (2m + 1) players,
The right hand side can be used as an agreement index. These results are due to
Straffin (1978).
2.5. While we have argued that several voting game concepts and results are
variants of system reliability ideas in a different guise; others and in particular the
general homogeneity assumption and its implications may contain important
lessons for reliability theory. For example; in systems in which the status of some
or all components may not be directly observable except via perfect or highly
reliable monitors--such as hazardous components in a nuclear installation, the
agreement indices can serve as alternative or surrogate indices of reliability
importance of inaccesible components. The general homogeneity assumption in
system reliability would amount to considering coherent structures of exchange-
able components, a strengthening of the concept of associatedness as a measure
198 M. C. Bhattacharjee
Table 3
Landholding in the State of W. Bengal, India (1961-1962) and model estimates
g(t) : = E ( X - t IX > t)
correspond respectively to the notions of the residual-life and the mean residual
life in reliability theory. In particular the extent of wealth which the 'rich' com-
mand is described by the behavior of g(t) for large values of t. More generally,
the nature of/7, and the excess average holding g(t) over an affluence threshold
t as a function of the threshold provides a more detailed description of the pattern
of ownership across different levels of affluence in the population.
Using the above interpretations of F, and g(t); the notion of skew and heavy
tailed distributions of wealth as being symptomatic of the social disparity of
ownership can be captured in fairly pitcuresque ways with varying degrees of
strength by the different anti-aging classes (DFR, IMRL, NWU, NWUE) of 'life
distributions' well known in reliability theory. For example a holding distribution
F is D F R (decreasing failure rate: F,i"st stochastically increasing in t) if the pro-
portion of the progressively 'rich' with residual holding in excess of any given
amount increases with the level of affluence. The other weaker anti-aging hypo-
theses: IMRL (increasing mean residual life: g(t)'r ), NWU (new worse than used:
Ft >~StF, all t) and N W U E (new worse than used in expectation: g(t)>~ g(0+)) can
be similarly interpreted as weaker descriptions of disparity.
Motivated by these considerations, Bhattacharjee and Krishnaji (1985) have
suggested using
when they exist, as indices of inequality in the distribution of wealth. They also
consider a related measure Io = g* - # =/~(I1 - 1) which is a variant of I~, but
200 M. C. Bhattacharjee
is not dimension free as 11, 12 are. The assumption that the limits in (3.1) exist
is usually not a real limitation in practice. In particular the existence of g* ~< oo
is free under IMRL and DFR assumptions, with g* finite for reasonably nice
subfamilies such as the D F R gammas. More generally, the holding distributions
for which g* ~< oo (g* < oo respectively) exists is the family of 'age-smooth' life
distributions which are those F for which the residual-life hazard function
- l n f f t ( x ) converges on [0, oo] ((0, ~ ] respectively) for each x as t ~ o o
(Bhattacharjee, 1986).
11 and 12 are indicators of aggregate inequality of the distribution of wealth in
two different senses. 11 measures the relative prepondrance of the wealth of the
super-rich, while 12 indicates in a sense how rich they are. The traditional index
of aggregate inequality, on the other hand, as measured by the classical Gini-index
(Lorenz measure) G can be expressed as
Fl(X ) = : # - 1 f o t dF(t),
the share of the population below x. A somewhat pleasantly surprising but not
fully understood feature of the three indices 11, I 2 and G is that they turn out to
be monotone increasing in the coefficient of variation for many holding distribu-
tions F. Such is the case with G under log-normal, 11 under gamma and I 2 under
log-gamma (Bhattacharjee and Krishnaji, 1985). Note also that whenever the
holding distribution is anti-aging in DFR, IMRL, NWU or NWUE sense, the
coefficient of variation (c.v.) is at least one (Barlow and Proschan, 1975); a
skewness feature aptly descriptive of the disproportionate share of the rich.
Recently the author has considered other inequality indices which share this
monotonicity in c.v. under weak anti-aging hypotheses and have re-examined the
appropriateness of 11, 12 and measures of aggregate inequality to show
(Bhattacharjee, 1986a):
(i) The non-trivial case 1 < 12 < m, implies I~ = ~ necessarily and then
(ii) The ratio of the hazard functions of the holding and share distributions
converge to 12:
3.2. The Lorenz curve and TTT-transform. While 11, 12 and the classical Gini
index are all aggregate measures of inequality, it is also useful to have a more
dynamic measure of inequality which will describe the variation of the disparity
of ownership with changing levels of affluence. This is classically modeled by the
Lorenz curve
where # is the average holding and F - J(u) = inf{t: F(t) >1 u} measures the pro-
portion of total wealth owned by the poorest 100p ~o of the population, and is thus
202 M. C. Bhattacharjee
is the scaled total time on test (TTT) transform of the holding distribution F viewed
as a life distribution with mean # and the cumulative TTT-transform,
V:= So1 W ( p ) d p , then
L ( p ) = W ( p ) - (1 - p)/~- i F - l(p),
V=I-G,
G= 1-2
fo F,(t) d F ( t ) = 1 - 2 L ( p ) dp
= 2
fo'
{ p - L ( p ) } d? (3.5)
is scale-equivalent to the area bounded by the diagonal and the Lorenz curve, as
is well known. Based on a random sample with order statistics X(1), X(2). . . . , X(,)
from F, the estimated sample Lorenz curve and the Gini-statistic
~'wl / n
G.=: j=,j(n -j)(X(j+I)- X(j))
n
(n - 1) Z j _ , X(j)
are similarly related to the total time on test statistic and its cumulative version
L. = W. - (n - i) i) X(:) ,
j i
Go=I-V n.
Chandra and Singpurwalla (1981), Klefsj0 (1984) and Taillie (1981) have used
partial orderings of life distributions to compare the Lorenz curves of holding
distributions which are so ordered. For the partial ordering notions
Reliability applications in economics 203
can be used as generalized indices of inequality. The Gini-index is the special case
G = J~ = L 2. Notice in view of (3.7), we have Jk >t O, L k >~ 0 for all anti-aging
holding distributions F or their 'aging' duals; and J~ = L k = 0 only in the
egaliterian case L ( p ) = p where everybody owns the same amount of wealth (F
is degenerate). By expressing Jk as
Klefsj6 (1984) implicitly notes that Jk can be interpreted as the excess over k - 1
of the ratio of the mean life of a parallel system of (k + 1) i.i.d, components with
life distribution F to that of a similar system with exponential lives. Similarly, we
note
204 M. C. Bhattacharjee
Lk = k ; ( l - u ) ~ - 1 ( 1 - W(u))du= 1 - # 1 ffk(t) dt
measures the relative advantage of a component with life F against a series system
of k such i.i.d, components as measured by the difference of the corresponding
mean lives as a fraction of the component mean life. These interpretations bring
to a sharper focus the relationships of the notion of 'inequality of distribution' in
economics to measures of system effectiveness in reliability.
L,(t)=:~ L, -L(t) } if j -
n
1 < t ~ < -j,
n
=:0 if t = 0,
4.2. The presence of identified or potential rivals who are in the race to be the
first to innovate constitutes the major source of uncertainty for an entrepreneur.
It is this aspect of innovational ( R & D ) rivalry on which reliability ideas can be
brought to bear that is of interest to us. Even within the context of such applica-
tions, there are a host of issues in modeling the economics of innovation which
can be so addressed within the Schumpeterian framework. Kamien and Schwartz
(1982) provide a definitive account of contemporary research on the economics
of technical advance, where reliability researchers will recognize the potential to
exploit reliability ideas through modeling the uncertainty associated with
206 M. C. Bhattacharjee
= PI('~ - T) if z > / T ;
Reliability applications in economics 207
r , , P ,
ro Po Po
x [ z < T: rival imitates
z T
ro rl P1
z >/T: rival precedes
T z
The expected net present value of the entrepreneur's returns, with a market
interest rate i, as a consequence of the decision to choose an introduction time
z is
oo
U(z) =
L E { e - ( i - ~ ) ' r ( t ; z, T)} dt + E { e - ( ' - r) max(z. T)p(.c, r)}
The optimal introduction time z* is of course the solution which maximizes the
expected value of profit
4.3. Intensity of rivalry as a reliability idea and its implications. What interests us
more is how the speed of development, as reflected by the economic z*, is
affected by the extent of innovational rivalry which is built-in in the rivals' intro-
duction time distribution H. Kamien and Schwartz (1982) postulate
m
i.e., the rival introduction time d.f. H belongs to a family of distributions with
proportional hazards which are of considerable interest in reliability. We may
think of F as the distribution of rivals' development time under unit rivalry (h = 1)
for judging how fast may the rivals complete development as indicated by H.
Since the hazard function A n ( t ) = : - i n H ( t ) is a measure of time-varying in-
novational risk of rival pre-emption, the proportional hazards hypothesis
A~(t) = hA(t) in (4.3) says the effects of time and rivalry on the entrepreneur's
innovational hazards are separable and multiplicative. If F has a density and
correspondingly a hazard rate (i.e., 'failure rate') 2(0, the so does H with failure
rate h2(t). It is the innovational rate of hazard at time t from the viewpoint of
our entrepreneur; and by standard reliability theoretic interpretation of failure
rates, the conditional probability of rivals' completion soon after t given com-
pletion has not occurred within time t is
As the intensity of rivalry increases by a factor from h to ch; this probability, for
each fixed t and small b, also increases essentiall by the same factor c.
To examine the effect of the intensity of rivalry on the speed of development,
assume that having imitators is preferable to being one (Po > P~) and that the
corresponding rewards are independent of 'innovation-imitation lag' (P'1 = P~ = 0)
as a simplifying assumption. By (4.1) and (4.2), the optimal introduction time z*
is then the implicit solution of
OV
- e-(i-~)~[{ro _ Po + h(P, - Po)2(z)}F(z)
&
satisfying the second derivative condition for a maximum at z*. (4.4) defines
z* = z*(h) implicitly as function of the rivalry intensity. Kamien and Schwartz
(1982) show that if
then either (i) z*(h) 1" or (ii) z*(h) is initially ~ and then t in h. The crux of their
argument is the following. If ro(h) is implicitly defined by the equation
i.e., the condition for the left hand side of (4.4) to have a local extremum as a
function of h; then z*(h) is decreasing, stationary or increasing in h according as
z*(h) > , = or < zo(h). Accordingly, since (4.5) implies that zo(h) is decreasing in
h; either z*(h) behaves according to one of the two possibilities mentioned, or
(iii) r*(h) < zo(h) for all h >~ 0. The last possibility can be ruled out by the con-
tinuity of V= V(z, h) in (4.2), V(0, h ) < 0, V(z*, h ) > 0 and the condition
P1 > Po. Which one of the two possibilities obtains of course depends on the
model parameters. In case (i), the optimal introduction time z*(h) increases with
increasing rivalry and the absence of rivalry (h = 0) yields the smallest such
optimal introduction time. The other case (ii), that depending on the rates of
return and other relevant parameters, there may be an intermediate degree of
rivalry for which the optimal development is quickest possible, is certainly not
obvious a-priori and highlights the non-intuitive effects of rivalry on decisions to
innovate.
F ~ {IFR} c3 ~ (4.7)
and hence so does H; where ~( is the set of life distributions with a log-concave
hazard function. The IFR hypothesis is easy to interpret. It says; the composite
rivals' residual time to development is stochastically decreasing so that if they
have not succeeded so far, then completion of their development within any
additional deadline becomes more and more likely with elapsed time. This reflects
the accumulation of efforts positively reinforcing the chances of success in future.
The other condition that F, and thus H, also has a log-concave hazard function
is less apparent to such interpretation; it essentially restricts the way in which the
time-dependent component of the entrepreneur's innovational hazard from com-
peting rivals grows with time t.
The proportional hazard model (4.3) can accomodate different configurations
of market structure as special cases, an argument clearly in its favor. By (4.3), as
210 M. C. Bhattacharjee
h --, O, P(T > t) ~ 1 for all t > 0 and in the limiting case T is an improper r.v. witb
all its mass at infinity. Thus h = 0 corresponds to absence of rivalry. Similarly as
h ~ 0% P ( T > t)---,O for all t > 0; in the limit the composite rivals' appearance is
immediate and this prevents the possibility of entreprenunial precedence. If our
entrepreneur had a head start with no rivals until a later time when rivals appear
with a very large h, then even if our entrepreneur innovates first; his supernormal
profits from innovation will very quickly be eliminated by rival imitation with high
probability within a very short time as a consequence of high rivalry intensity h,
which shrinks to instantaneous imitation as h approaches infinity. In this sense
the case h = oo reflects the traditional economists' dream of 'perfect competition'.
Among the remaining possibilities 0 < h < oo that reflect more of a realism, Barzel
(1968) distinguishes between moderate and intense rivalry, the latter corresponding
to the situation when the intensity of rivalry exceeds the market growth rate
( h > 7). If rivalry is sufficiently intense, no development becomes best
(h >>~, ~ z*(h) = ~ ) . In other cases, the intense rivalry and non-rivalous solu-
tions provide vividly contrasting benchmarks to understand the innovation pro-
cess under varying degrees of moderate to intense rivalry.
Our modeling to illustrate the use of reliability ideas has been limited to a
relatively simplified situation. It is possible to introduce other variations and
features of realism such as modification of rivals' effort as a result of entre-
preneur's early success, budget constraints, non-contractual development which
allows the option of stopping development under rival precedence, and game
theoretic formulations which incorporate technical uncertainty. There is now sub-
stantial literature on these various aspects of innovation as an economic process
(DasGupta and Stiglitz, 1980, 1980a; Kamien and Schwarz, 1968, 1971, 1972,
1974, 1975, 1982; Lee and Wilde, 1980; Lowry, 1979). It appears to us that there
are many questions, interesting from a reliability application viewpoint which can
be profitably asked and would lead to a deeper understanding of the economics
of innovation. Even in the context of the present model which captures the
essence of the innovating proces under risk of rivalry, there are many such
questions. For example, what kind of framework for R & D rivalry and market
mechanisms lead to the rival entry model (4.3)? Stochastic modeling of such
mechanisms would be of obvious interest. Note the exponential: H ( t ) = e -m,
2(0 = 1; Weibull: H(t) = e -h'~, 2(0 = ~t ~- 1 and the extreme-value distributions:
H(t) = e x p { - h ( e ~ ' - 1)}, 2(t)= 0~e~t all satisfy (4.3) and (4.7), the latter for
~>1.
A related open question is the following. Suppose the rival introduction time
satisfies (4.3) but its distribution F under unit rivalry (h = 1) is unknown. Under
what conditions, interesting from a reliability point of view with an appropriate
interpretation in the context of rivalry, does there exist a finite maximin intro-
duction time ~*(h) and what, if any, is a least favorable distribution F* of time
to rival entry? Such a pair (z*(h), F*), for which
would indicate the entrepreneur's best economic introduction time within any
specified regime of rivalry when he has only an incomplete knowledge of the
benchmark distribution F. Here V(v, h; F) is the total expected reward (4.2) and
(4.1) under (4.3).
The proportional hazards model (4.3) aggregates all sources of rivalry, from
existing firms or potential new entrants. This is actually less of a criticism than
it appears because in the entrepreneur's preception, only the distribution of com-
posite rival entry time matters. It is possible to introduce technical uncertainty in
the model by recognizing that the effort, usually parametrized through cost,
required to successfully complete development is also subject to uncertainties
(Kamien and Schwartz, 1971). Suppose there are n competetors including our
entrepreneur, the rivals are independent and let G(z) be the probability that any
rival completes development with an effort no more than z. If z(t) is the cumula-
tive rival effort up to time t, then the probability that none of the rivals will
succeed by time t is
This leads to (4.3) with F--- G(z), H = P and intensity h = (n - 1) the number of
rivals. We note this provides one possible answer to the question of modeling
rivalry described by (4.3). What other alternative mechanisms can also lead to
(4.3)? If the effort distribution G has a 'failure rate' (intensity of effort) r(z), then
the innovational hazard function and rates are
which show how technical uncertainty can generate market uncertainty. If our
entrepreneur's effort distribution is also G(z) and independent of the rivals; then
note the role of each player in the innovation game is symmetric and each faces
the hazard rate (4.8) since from the perspective of each competitor, the other
(n - 1) rivals are i.i.d, and in series. It would clearly be desirable to remove the
i.i.d, assumption to reflect more of a realism in so far as a rival's effort and
spending decisions are often dictated by those of others.
Some of the effects of an innovation may be irreversible. Computers and
information processing technology which have now begun to affect every facet of
human life is clearly a case in point. Are these impacts or their possible irreversi-
bility best for the whole society? None of the above formulations can address this
issue, a question not in the perview of economists and quantitative modeling
alone; nor do they dispute their relevance. What they can and do provide is an
understanding of the structure and evolution of the innovating process as a risky
enterprise and it is here that reliability ideas may be able to play a more significant
role than hitherto in explaining rivalry and their impacts on the economics of
212 M. C. Bhattacharjee
i n n o v a t i o n . In t u r n the m e a s u r a b l e p a r a m e t e r s o f s u c h m o d e l s a n d their c o n s e -
q u e n c e s c a n t h e n serve as s i g n p o s t s for an i n f o r m e d d e b a t e o n the w i d e r
q u e s t i o n s o f social r e l e v a n c e o f an i n n o v a t i o n .
References
Arrow, K. J. (1951). Social Choice and Individual Values. Wiley, New York.
Arrow, K. J. (1962). Economic welfare and the allocation of resources for invention. In: R. R. Nelson,
ed., The Rate and Direction of Inventive Activity. Princeton University Press, Princeton, NJ.
Barlow, R. E. and Campo, R. (1975). Total time on test processes and applications to failure data
analysis. In: R. E. Barlow, J. Fussell and N. D. Singpurwalla, eds., Reliability and Fault Tree
Analysis, SIAM, Philadelphia, PA, 451-481.
Barlow, R. E. and Saboia, J. L. M. (1973). Bounds and inequalities in the rate of population growth.
In: F. Proschan and R. J. Serfling, eds., Reliability and Biometry, Statistical Analysis of Lifelengths,
SIAM, Philadelphia, PA, 129-162.
Barlow, R. E. and Proschan, F. (1965). Mathematical Theory of Reliability. Wiley, New York.
Barlow, R. E. and Proschan, F. (1975). Statistical Theory of Reliability and Life Testing: Probability
Models. Holt, Rinehart and Winston, New York.
Barzel, Y. (1968). Optimal timing of innovation. Review of Economics and Statistics 50, 348-355.
Bergmann, R. and Stoyan, D. (1976). On exponential bound for the waiting time distribution in
GI/G/1. J. AppL Prob. 13(2), 411-417.
Bhattacharjee, M. C. and Krishnaji, N. (1985). DFR and other heavy tail properties in modeling the
distribution of land and some alternative measures of inequality. In: J. K. Ghosse, ed., Statistics:
Applications and New Directions, Indian Statistical Institute, Eka Press, Calcutta; 100-115.
Bhattacharjee, M. C. (1986). Tail behaviour of age-smooth failure distribution and applications. In:
A. P. Basu, ed., Reliability and Statistical Quality Control, North-Holland, Amsterdam, 69-86.
Bhattacharjee, M. C. (1986a). On using Reliability Concepts to Model Aggregate Inequality of Distribu-
tions. Technical Report, Dept. of Mathematics, University of Arizona, Tucson.
Brains, S. J., Lucas, W. F. and Straffin, P. D., Jr. (eds.) (1978). Political and Related Models. Modules
in Applied Mathematics: Vol. 2, Springer, New York.
Chandra, M. and Singpurwalla, N. D. (1981). Relationships between some notions which are
common to reliability and economics. Mathematics of Operations Research 6, 113-121.
Daley, D. (ed.) (1983). Stochastic Comparison Methods for Queues and Other Processes. Wiley, New
York.
Deegan, J., Jr. and Packel, E. W. (1978). To the (Minimal Winning) Victors go the (Equally Divided)
Spoils: A New Power Idex for Simple n-Person Games. In: S. J. Brahms, W. F. Lucas and P.
D. Straffin, Jr. (eds.): Political and Related Models. Springer-Verlag, New York, 239-255.
DasGupta, P. and Stiglitz, J. (1980). Industrial structure and the nature of innovative activity.
Economic Journal 90, 266-293.
DasGupta, P. and Stiglitz, J. (1980a). Uncertainty, industrial structure and the speed of R& D. Bell
Journal of Economics 11, 1-28.
Feller, W. (1966). Introduction to Probability Theory and Applications. 2nd ed. Wiley, New York.
Gail, M. H. and Gatswirth, J. L. (1978). A scale-free goodness-of-fit test for the exponential distribu-
tion based on the Lorenz curve. J. Amer. Statist. Assoc. 73, 787-793.
Galbraith, J. K. (1952). American Capitalism. Houghton and Mifflin, Boston.
Goldie, C. M. (1977). Convergence theorems for empirical Lorenz curves and their inverses. Advances
in Appl. Prob. 9, 765-791.
Guess, F., Hollander, M. and Proschan, F. (1983). Testing whether Mean Residual Life Changes Trend.
FSU Technical Report #M665, Dept. of Statistics, Florida State University, Tallahassee.
Hicks, J. R. (1932). The Theory of Wages. Macmillan, London.
Hollander, M. and Proschan, F. (1975). Tests for the mean residual life. Biometrika 62, 585-593.
Kamien, M. and Schwartz, N. (1968). Optimal induced technical change. Econometrika 36, 1-17.
Reliability applications in economics 213
Kamien, M. and Schwartz, N. (1971). Expenditure patterns for risky R & D projects. J. Appl. Prob.
8, 60-73.
Kamien, M. and Schwartz, N. (1972). Timing of innovations under rivalry. Econometrika 40, 43-60.
Kamien, M. and Schwartz, N. (1974). Risky R & D with rivalry. Annals of Economic and Social
Measurement 3, 276-277.
Kamien, M. and Schwartz, N. (1975). Market structure and innovative activity: A survey. J.
Economic Literature 13, 1-37.
Kamien, M. and Schwartz, N. (1982). Market Structure and Innovation. Cambridge University Press,
London.
Kelfsj/J, B. (1982). The HNBUE and HNWUE class of life distributions. Naval Res. Logist. Qrtly. 29,
331-344.
Kelfsj/5, B. (1983). Testing exponentiality against HNBUE. Scandinavian J. Statist. 10, 65-75.
Kelfsj~, B. (1984). Reliability interpretations of some concepts from economics. Naval Res. Logist.
Qrtly. 31,301-308.
Kleinrock, L. (1975). Queueing Systems, Vol. 1. Theory. Wiley, New York.
KSllerstrSm, J. (1976). Stochastic bounds for the single server queue. Math. Proc. Cambridge Phil.
Soc. 80, 521-525.
Lucas, W. F. (1978). Measuring power in weighted voting systems. In: S. J. Brahms, W. F. Lucas
and P. D. Straffin, Jr., eds., Political Science and Related Models. Springer, New York, 183-238.
Lee, T. and Wilde, L. (1980). Market structure and innovation: A reformulation. Qrtly. J. of
Economics 194, 429-436.
Loury, G. C. (1979). Market structure and innovation. Qrtly. J. of Economics XCIII, 395-410.
Macdonald, J. B. and Ransom, M. R. (1979). Functional forms, estimation techniques and the
distribution of income. Ecometrika 47, 1513-1525.
Mukherjee, V. (1967). Type III distribution and its stochastic evolution in the context of distribution
of income, landholdings and other economic variables. Sankhy-d A 29, 405-416.
Owen, G. (1982). Game Theory. 2nd edition. Academic Press, New York.
Pechlivanides, P. M. (1975). Social Choice and Coherent Structures. Unpublished Tech. Report # ORC
75-14, Operations Research Center, University of California, Berkeley,
Rae, D. (1979). Decision rules and individual values in constitutional choice. American Political
Science Review 63.
Ramamurthy, K. G. and Parthasarathy, T. (1983). A note on factorization of simple games. Opsearch
20(3), 170-174.
Ramamurthy, K. G. and Parthasarathy, T. (1984). Probabilistic implications of the assumption of
homogeneity in voting games. Opsearch 21(2), 81-91.
Salem, A. B. Z. and Mount, T. D. (1974). A convenient descriptive model of income distribution.
Econometrika 42, 1115-1127.
Schumpeter, J. A. (1961). Theory of Economic Development. Oxford University Press, New York.
Schumpeter, J. A. (1964). Business Cycles. McGraw-Hill, New York.
Schumpeter, J. A. (1975). Capitalism, Socialism and Democracy. Harper and Row, New York.
Seneta, E. (1976). Regularly Varying Functions. Lecture Notes in Math. 508, Springer, New York.
Straffin, P. D., Jr. (1978). Power indices in politics. In: S. J. Brams, W. F. Lucas and P. D. Straffin,
Jr., eds., Political Science and Related Models. Springer, New York, 256-321.
Straffin, P. D., Jr. (1978a). Probability models for power indices. In: P. C. Ordershook, ed., Game
Theory and Political Science, University Press, New York.
TaiUie, C. (1981). Lorenz ordering within the generalized gamma family of income distributions. In:
C. Taillie, P. P. Ganapati and B. A. Baldessari, eds., Statistical Distributions in Scientific Work. Vol.
6. Reidel, Dordrecht/Boston, 181-192.
Taussig, F. W. (1915). Innovation and Money Makers. McMillan, New York.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 "1'~
.]k g ~
© Elsevier Science Publishers B.V. (1988) 215-224
The mean residual life (MRL) has been used as far back as the third century
A.D. (cf. Deevey (1947) and Chiang (1968)). In the last two decades, however,
reliabilists, statisticians, and others have shown intensified interest in the MRL
and derived many useful results concerning it. Given that a unit is of age t, the
remaining life after time t is random. The expected value of this random residual
life is called the mean residual life at time t. Since the MRL is defined for each
time t, we also speak of the M R L function. (See Section 2 for a more formal
definition.)
The M R L function is like the density function, the moment generating function,
or the characteristic function: for a distribution with a finite mean, the MRL
completely determines the distribution via an inversion formula (e.g., see Cox
(1962), Kotz and Shanbhag (1980), and Hall and Wellner (1981)). Hall and
Wellner (1981) and Bhattacharjee (1982) derive necessary and sufficient condi-
tions for an arbitrary function to be a M R L function. These authors recommend
the use of the M R L as a helpful tool in model building.
Not only is the M R L used for parametric modeling but also for nonparametric
modeling. Hall and Wellner (1981) discuss parametric uses of the MRL. Large
nonparametric classes of life distributions such as decreasing mean residual life
(DMRL) and new better than used in expectation (NBUE) have been defined
using MRL. Barlow, Marshall and Proschan (1963) note that the D M R L class
is a natural one in reliability. Brown (1983) studies the problem of approximating
increasing mean residual life (IMRL) distributions by exponential distributions.
He mentions that certain IMRL distributions, '... arise naturally in a class of first
passage time distributions for Markov processes, as first illuminated by Keilson'.
See Barlow and Proschan (1965) and Hollander and Proschan (1984) for further
comments on the nonparametric use of MRL.
A fascinating aspect about M R L is its tremendous range of applications. For
example, Watson and Wells (1961) use MRL in studying burn-in. Kuo (1984)
* Research sponsored by the Air Force Office of Scientific Research, AFSC, USAF, under Grant
AFOSR 85-C-0007.
215
216 F. Guess and F. Proschan
2. T h e o r y o f m e a n r e s i d u a l life
Let F be a life distribution (i.e., F(t) = 0 for t < 0) with a finite first moment.
Let i ( t ) = 1 - F(t). X is the random life with distribution F. The mean residual
life function is defined as
~ i(x + t) f o~ i(u)
m(t) - - dx = ff-~ du
L r(t)
Like the failure rate function (recall that it is defined as r(t)= f(t)/F(t) when
F(t) > 0), the MRL function is a conditional concept. Both functions are condi-
tioned on survival to time t.
While the failure rate function at t provides information about a small interval
after time t ('just after t', see p. 10 Barlow and Proschan (1965)), the M R L
function at t considers information about the whole interval after t ('all after t').
This intuition explains the difference between the two.
Note that it is possible for the M R L function to exist but for the failure rate
function not to exist (e.g., consider the standard Cantor ternary function, see
Chung (1974), p. 12). On the other hand, it is possible for the failure rate function
Mean residual life: theory and applications 217
to exist but the M R L function not to exist (e.g., consider modifying the Cauchy
density to yield f ( t ) = 2/n(1 + t 2) for t >f 0). Both the M R L and the failure rate
functions are needed in theory and in practice.
When m and r both exist the following relationship holds between the two:
m'(t) = m ( t ) r ( t ) - 1. (2.2)
See Watson and Wells (1961) for further comments on (2.2) and its uses•
If the failure rate is a constant ( > 0 ) the distribution is an exponential. If the
MRL is a constant ( > 0 ) the distribution is also an exponential.
L e t / t = E(X). If F(0) = 0 then m(0) = #. If F(0) > 0 then m(0) = #/F(0) ~ #.
For simplicity in discussions and definitions in this section, we assume F(0) = 0.
Let F be right continuous (not necessarily continuous). Knowledge of the MRL
function completely determines the reliability function as follows:
See Hall and Wellner (1981) for a proof. See Bhattacharjee (1982) for another
characterization. Note that condition (ii) rules out the degenerate at 0 distribution•
218 F. Guess and F. Proschan
For (iv) note that d(t) is simply the expected time of death (failure) given that a
unit has survived to time t. Theorem 2.1 delineates which functions can serve as
MRL functions, and hence, provides models for lifelengths.
We restate several bounds involving MRL from Hall and Wellner (1981). Recall
a + = a if a >i 0, otherwise a + = 0.
DEFINITION 2.3. DMRL. A life distribution F has decreasing mean residual life
if its MRL m is a decreasing function.
DEFINITION 2.4. NBUE. A life distribution F is new better than used in expec-
tation if m(0) >1 m(t) for all t >t 0.
Each of these classes above has an obvious dual class associated with it, i.e.,
increasing mean residual life, new worse than used in expectation (NWUE), and
decreasing then increasing mean residual life (DIMRL), respectively.
The D M R L class models aging that is adverse (e.g., wearing occurs). Barlow,
Marshall and Proschan (1963) note that the D M R L class is a natural one in
reliability. See also Barlow and Proschan (1965). The older a D M R L unit is, the
shorter is the remaining life on the average. Chen, Hollander and Langberg (1983)
contains an excellent discussion of the uses of the D M R L class.
Burn-in procedures are needed for units with IMRL. E.g., integrated circuits
have been observed empirically to have decreasing failure rates; and thus they
satisfy the less restrictive condition of IMRL. Investigating job mobility, social
scientists refer to IMRL as inertia. See Morrison (1978) for example. Brown
(1983) studies approximating IMRL distributions by exponentials. He comments
that certain IMRL distributions, '... arise naturally in a class of first passage time
distributions for Markov processes, as first illuminated by Keilson'.
Note that D M R L implies NBUE. The N B U E class is a broader and less
Mean residual life: theory and applications 219
restrictive class. Hall and Wellner (1981) show for NBUE distributions that the
coefficient of variation a/it ~< 1, where a z = Var(X). They also comment on the
use of NBUE in renewal theory. Bhattacharjee (1984b) discusses a new notion,
age-smoothness, and its relation to NBUE for choosing life distribution models
for equipment subject to eventual wear. Note that burn-in is appropriate for
NWUE units.
For relationships of DMRL, IMRL, NBUE, and N W U E with other classes
used in reliability see the survey paper Hollander and Proschan (1984).
The IDMRL class models aging that is initially beneficial, then adverse. Si-
tuations where it is reasonable to postulate an IDMRL model include:
(i) Length of time employees stay with certain companies: An employee with a
company for four years has more time and career invested in the company than
an employee of only two months. The M R L of the four-year employee is likely
to be longer than the M R L of the two-month employee. After this initial IMRL
(this is called 'inertia' by social scientists), the processes of aging and retirement
yield a D M R L period.
(ii) Life lengths of human." High infant mortality explains the initial IMRL.
Deterioration and aging explain the later D M R L stage.
See Guess (1984) and Guess, Hollander, and Proschan (1983) for further
examples and discussion. Bhattacharjee (1983) comments that Gertsbakh and
Kordonskiy (1969) graph the MRL function of a lognormal distribution that has
a 'bath-tub' shaped M R L (i.e., DIMRL).
Hall and Wellner (1981) characterize distributions with MRL's that have linear
segments. They use this characterization as a tool for choosing parametric
models. Morrison (1978) investigates linearly IMRL. He states and proves that
if F is a mixture of exponential then F has linearly IMRL if and only if the mixing
distribution, say G, is a gamma. Howell (1984) studies and lists other references
on linearly DMRL.
In renewal theory M R L arises naturally also. For a renewal process with
underlying distribution F, let G(t) = ( ~ if(u)du)/#. G is the limiting distribution
of both the forward and the backward recurrence times. See Cox (1962) for more
details. Also if the renewal process is in equilibrium then G is the exact distribu-
tion of the recurrence times. G(t) = (m(t)ff(t))/#. The failure rate of G, r 6, is
inversely related to the MRL of F, m F. I.e., re(t ) = 1/mF(t ). Note, however, that
rF(t) ~ 1/mF(t ) is USually the case. See Hall and Wellner (1981), Rolski (1975),
Meilijson (1972), and Watson and Wells (1961) for related discussions.
Kotz and Shanbhag (1980) establish a stability result concerning convergence
of an arbitrary sequence of M R L functions to a limiting MRL function. (See also
Bhattacharjee (1982).) They show an analogous stability result for hazard
measures. (When the failure rate for F exists and vF is F's hazard measure, then
VF(B) = ~B rF(t) dt for B a Borel set.) Their results imply that MRL functions can
provide more stable and reliable information than hazard measures when
assessing noncontinuous distributions from data.
In a multivariate setting, Lee (1985) shows the effect of dependence by total
positivity on M R L functions.
220 F. Guess and F. Proschan
2 ni = k + l (Sin -- t)
mn(t ) = for te [Xk,, X(k + l),) , (3.2)
n-k
i
n; = number of observed failures at time ~';z, se=n- ~ nj (3.5)
j=0
m.(t) l
= ~i=k+ 1
ni(Xil- t) for t~ [~'kZ, X(k+ ,),),
Sk (3.6)
= 0 for t >~/~'u,
EXAMPLE 3.1. Bjerkedal (1960) studies the lifelengths of guinea pigs injected
with different amounts of tubercle bacilli. Guinea pigs are known to have a high
Mean residual life." theory and applications 221
Table 3.1
Empirical m e a n residual life in days at the unique times of death for the 72 guinea pigs under
regimen 5.5. We include the empirical M R L at time 0 also.
to 2.2 × 104 bacillary units per 0.5 ml (loglo(2.2 × 104)=4.342). Table 3.1
presents the data from regimen 5.5 and the empirical MRL.
Graphs of MRL provide useful information not only for data analysis but also
for presentations. Commenting on fatigue longevity and on preventive main-
tenance, Gertsbakh and Kordonskiy (1969) recommend the MRL function as
another helpful tool in such analyses. They graph the MRL for different distribu-
tions (e.g., Weibull, lognormal, and gamma). Hall and Wellner (1979) graph the
empirical MRL for Bjerkedal's (1960) regimen 4.3 and regimen 6.6 data. Bryson
and Siddiqui (1969) illustrate the graphical use of the empirical MRL on survival
data from chronic granulocytic leukemia patients. Using the standard Kaplan-
Meier estimator (e.g., see Lawless (1982), Nelson (1982), or Miller (1980)), Chen,
Hollander, and Langberg (1983) graph the empirical MRL analogue for censored
lifetime data.
Gertsbakh and Kordonskiy (1969) note that estimation of MRL is more stable
than estimation of the failure rate. Statistical properties of estimated means are
better than those of estimated derivatives (which enter into failure rates).
Yang (1978) shows that the empirical MRL is uniformly strongly consistent.
She establishes that mn, suitably standardized, converges weakly to a Gaussian
process. Hall and Wellner (1979) require less restrictive conditions to apply these
results. They derive and illustrate the use of simultaneous confidence bands for
m. Yang (1978) comments that for t > 0, ran(t) is a slightly biased estimator.
Specifically, E(mn(t))= m(t)(1 -Fn(t)). Note, however, that l i m ~ E(m~(t))=
re(t). Thus, for larger samples rn,(t) is practically unbiased. See also Gertsbakh
and Kordonskiy (1969).
Yang (1977) studies estimation of the MRL function when the data are ran-
domly censored. For parametric modeling Hall and Wellner (1981) use the empiri-
cal MRL plot. They observe that the empirical MRL function is a helpful addition
to other life data techniques, such as total time on test plots, empirical (cumula-
tive) failure rate functions, etc. The MRL plot detects certain aspects of the
distribution more readily than other techniques. See Hall and WeUner (1981), Hall
and WeUner (1979), and Gertsbakh and Kordonskiy (1969) for further comments.
When a parametric approach seems inadvisable, the MRL function can still be
used as a nonparametric tool. Broad classes defined in terms of MRL allow a
more flexible approach while still incorporating preliminary information. For ex-
ample, to describe a wear process, a DMRL is appropriate. When newly
developed components are initially produced, many may fall early (such early
failure is called infant mortality and this early stage is called the debugging stage).
Another subgroup tends to last longer. Depending on information about this latter
subgroup, we suggest IMRL (e.g., lifelengths of integrated circuits) or IDMRL
(e.g., more complicated systems where there are infant mortality, useful life, and
wear out stages).
Objective tests exist for these and other classes defined in terms of MRL. E.g.,
see Hollander and Proschan (1984) and Guess, Hollander and Proschan (1983).
To describe 'burn-in' the MRL is a natural function to use. Kuo's (1984)
Appendix 1 presents an excellent brief introduction to burn-in problems and
applications of MRL.
Mean residual life: theory and applications 223
Actuaries apply M R L to setting rates and benefits for life insurance. In the
biomedical setting researchers analyze survivorship studies by M R L . For example,
see E l a n d t - J o h n s o n and J o h n s o n (1980) and Gross and Clark (1975).
Social scientists use I M R L for studies on job mobility, length o f wars, duration
of strikes, etc. See Morrison (1978).
In economics M R L arises also. Bhattacharjee and Krishnaji (1981) present
applications of M R L for investigating landholding. Bhattacharjee (1984a) uses
N B U E for developing optimal inventory policies for perishable items with r a n d o m
shelf life and variable supply.
Bhattacharjee (1982) observes M R L functions occur naturally in other areas
such as optimal disposal of an asset, renewal theory, dynamic programming, and
branching processes.
Acknowledgements
References
Barlow, R. E., Marshall, A. W. and Proschan, F. (1963). Properties of probability distributions with
monotone hazard rate. Ann. Math. Statist. 34, 375-389.
Barlow, R. E. and Proschan, F. (1965). Mathematical Theory of Reliability. Wiley, New York.
Barlow, R. E. and Proschan, F. (1981). Statistical Theory of Reliability and Life Testing. To Begin With,
Silver Springs, MD.
Bhattacharjee, M. C. (1984a). Ordering policies for perishable items with unknown shelf life/variable
supply distribution. Indian Institute of Management, Calcutta, Technical Report.
Bhattacharjee, M. C. (1984b). Tail behavior of age-smooth failure distributions and applications.
Indian Institute of Management, Calcutta, Technical Report.
Bhattacharjee, M. C. (1983). Personal communication.
Bhattacharjee, M. C. (1982). The class of mean residual lives and some consequences. S l A M J.
Algebraic Discrete Methods 3, 56-65.
Bhattacharjee, M. C. and Krishnaji, N. (1981). DFR and other heavy tail properties in modelling the
distribution of land and some alternative measures of inequality. Proceedings of the Indian Statisti-
cal Institute Golden Jubilee International Conference.
Bjerkedal, T. (1960). Acquisition of resistance in guinea pigs infected with different doses of virulent
tubercle bacilli. Amer. J. Hygiene 72, 130-148.
Brown, M. (1983). Approximating IMRL distributions by exponential distributions, with applications
to first passage times. Ann. Probab. 11, 419-427.
Bryson, M. C. and Siddiqui, M. M. (1969). Some criteria for aging. J. Amer. Statist. Assoc. 64,
1472-1483.
Chen, Y. Y., Hollander, M. and Langberg, N. A. (1983). Tests for monotone mean residual life,
using randomly censored data. Biometrics 39, 119-127.
Chiang, C. L. (1968). Introduction to Stochastic Processes in Biostatistics. Wiley, New York.
224 F. Guess and F. Proschan
Chung, K. L. (1974). A Course in Probability Theory, 2nd ed. Academic Press, New York.
Cox, D. R. (1962). Renewal Theory. Methuen, London.
Deevey, E. S. (1947). Life tables for natural populations of animals. Quarterly Review of Biology 22,
283-314.
Elandt-Johnson, R. C. and Johnson, N. L. (1980). Survival Models and Data Analysis. Wiley, New
York.
Gertsbakh, I. B. and Kordonskiy, K. B. (1969). Models of Failure. Springer, New York.
Gross, A. J. and Clark, V. A. (1975). Survival Distributions: Reliability Applications in the Biomedical
Sciences. Wiley, New York.
Guess, F. (1984). Testing whether mean residual life changes trend. Ph.D. dissertation, Department
of Statistics, Florida State University.
Guess, F., Hollander, M. and Proschan, F. (1983). Testing whether mean residual life changes trend.
Florida State University Department of Statistics Report M665. (Air Force Office of Scientific
Research Report 83-160).
Hall, W. J. and Wellner, J. A. (1979). Estimation of mean residual life. University of Rochester
Department of Statistics Technical Report.
Hall, W. J. and Wellner, J. A. (1981). Mean residual life. In: M. CsSrgS, D. A. Dawson, J. N. K.
Rao and A. K. Md. E. Saleh, eds., Statistics and Related Topics, North-Holland, Amsterdam,
169-184.
Hollander, M. and Proschan, F. (1984). Nonparametric concepts and methods in reliability. In: P.
R. Krishnaiah and P. K. Sen, eds., Handbook of Statistics, Vol. 4, Nonparametric Methods, North-
Holland, Amsterdam.
Howell, I. P. S. (1984). Small sample studies for linear decreasing mean residual life. In: M. S.
Abdel-Hameed, J. Quinn and E. ~inlar, eds., Reliability Theory and Models, Academic Press, New
York.
Keilson, J. (1979). Markov Chain Models--Rarity and Exponentiality. Springer, New York.
Kotz, S. and Shanbhag, D. N. (1980). Some new approaches to probability distributions. Adv. in
Appl. Probab. 12, 903-921.
Kuo, W. (1984). Reliability enhancement through optimal burn-in. IEEE Trans. Reliability 33,
145-156.
Lawless, J. F. (1982). Statistical Models and Methods for Lifetime Data. Wiley, New York.
Lee, M. T. (1985). Dependence by total positivity. Ann. Probab. 13, 572-582.
Meilijson, I. (1972). Limiting properties for the mean residual lifetime function. Ann. Statist. 1,
354-357.
Miller, R. G. (1981). Survival Analysis. Wiley, New York.
Morrison, D. G. (1978). On linearly increasing mean residual lifetimes. J. Appl. Probab. 15, 617-620.
Nelson, W. (1982). Applied Life Data Analysis. Wiley, New York.
Rolski, T. (1975). Mean residual life. Bulletin of the International Statistical Institute, Book 4 (Pro-
ceedings of the 40th Session), 266-270.
Swartz, G. B. (1973). The mean residual lifetime function. IEEE Trans. Reliability 22, 108-109.
Watson, G. S. and Wells, W. T. (1961). On the possibility of improving the mean useful life of items
by eliminating those with short lives. Technometrics 3, 281-298.
Yang, G. L. (1978). Estimation of a biometric function. Ann. Statist. 6, 112-116.
Yang, G. (1977). Life expectancy under random censorship. Stochastic Process. AppL 6, 33-39.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 1
i J
© Elsevier Science Publishers B.V. (1988) 225-249
O. Introduction
1. Likelihood
In this section we present a unified way of analyzing incomplete data for a large
number of failure distribution models. We often assume that the failure distribu-
tion F is absolutely continuous with density f and failure rate
* This research was supported by the Air Force Office of Scientific Research (AFSC), USAF, under
Grant AFOSR-77-3179 with the University ef California. Reproduction in whole or in part is
permitted for any purpose of the United States Government.
225
226 R. E. Barlowand F. Proschan
x to be transformed by the function Ro('). For this reason (1.7) is called the
time-transformed exponential model
Let x~, x 2, ..., x, be n independent observations given 2 from this model. The
likelihood is
We conclude that Y,"i=l Ro(x~) and n are jointly sufficient for 2. If we use the
gamma prior for 2,
b a,~a - 1 e - b2
~(,~) --
r(a)
Inference preceeds exactly as for the exponential model, except that observation
x i of the exponential model is replaced by its time-transformed value Ro(x~). This
is valid assuming only that Ro(" ) is continuous.
7t(21 x) oc ;t e - zx n(2)
where n is the prior density for ~. However, if ). and ~ are judged dependent
with joint prior rt(2, ~p), then the posterior density is
zt(21x)oc)~e-~Xf~e-~X~(2,(p)d(o.
The factor ~o e-¢Xrc( 2, q~) dq~, contributed by the stopping rule, depends on ~..
There is an important case not covered by the General Sampling Plan--namely
when it is known that a unit has failed within some time interval but the exact
time of failure is unknown.
The following simple example illustrates the way in which incomplete data can
arise.
Life distribution models and incomplete data 229
EXAMPLE 1.2. Operating data are collected on an airplane part for a fleet of
airplanes. A typical age history for several engines is shown in Figure 1.1. The
crosses indicate the observed ages at failure. Ordered withdrawal times (nonfailure
times) are indicated by short vertical lines. In our example, units 2 and 4 fail at
respective times xco and x~2~ while observation on units 1 and 3 is terminated
without failure at times l~2~ and l~1~ respectively.
Unit
number
Age u
X(1) l(t) x(2) l(2)
It is important to note that all data are plotted against the age axis. Figure 1.2
illustrates how events may have occurred in calendar time. For example, units 1
and 3 had not failed at the end of the calendar record.
The total time on test is an important statistic for the exponential model.
Unit
number
2
L ×
3
I
4
I
Start of End of
calendar record calendar record
DEFINITION 1.3. The total time on test T is the total of the periods of observa-
tion of all the units undergoing test. Excluded from this statistic are any periods
following death or withdrawal or preceding observation. Specifically, the periods
being totalled include only those in which a death or a withdrawal of a unit under
observation can be observed.
n(u)
Age u
x(,) l(,) x(2 ) 1(2)
Let n(u) be the number of units observed to be operating at age u. The observed
function n(u) u >~ O, for Example 1.2 is displayed in Figure 1.3. From Figure 1.3
we may readily calculate the total time on test T(t) corresponding to any
t, 0 <~ t ~ l(2):
For example, for t such that x(2) < t </(2), we obtain from Figure 1.3:
Note that the resulting expression, given in (1.11), can be obtained directly, since
X(l ) and x(2) represent the observed lifetimes of the 2 units that are observed to
fail, l(1) represents the observed age of withdrawal of the unit first withdrawn from
observation, and finally t represents the age of the second unit at the instant t
specified.
Although in this small example, the directly calculated expression (1.11) for
total time on test is simpler, Equation (1.10) is an important identity, since it
Life distribution models and incomplete data 231
yields the total time on test accumulated by age t in terms of the (varying) number
of units on test at each instant during the interval [0, t] for any data set in which
the ages at death or withdrawal are observed. Thus it is a general formula applicable
in a great variety of problems in which data may be incomplete.
Although n(u) is a step function, the integral representation in (1.10) is advan-
tageous, since it is compact, mathematically tractable, and applicable in a great
variety of incomplete data situations. Of course, So n ( u ) d u < ~ in practical
problems since observation ultimately ceases in order to analyze the data in hand.
k
p(DIO) = ~ p(x,[O) f i P(I,[0), (1.12)
s=l t=l
Now for observed data D = {x l, ..., x k, l 1, ... lm}, the probability of the observed
outcome given parameter 0 is
k k
k
L(OID)o¢ I-I p(x~]O) f i P(ltlO ). (1.15)
s=l t=l
From Bayes' Theorem, it is clear that we need not know g(x, !) in order to
compute the posterior density of 0.
In this subsection, we have thus far confined our discussion to the case of
discrete time life distributions since the basic concepts are easier to grasp in this
case. However, in the case of continuous time life distributions, the likelihood
concept is equally relevant, and in fact the expression for the likelihood L(OID)
assumes a rather elegant form if we use n(u), the number on test function. In the
continuous case, p(x[O) is replaced by the probability density element f(xlO).
THEOREM 1.5. Given the failure rate, independent observations are made under the
General Sampling Plan. Let Xl, x 2, ..., x k denote the k observed failure ages. Let
n(u) denote the number of units under observation at age, u, u >i O, and r(u) denote
Life distribution models and incomplete data 233
the failure rate function of the unit at age u. Then the likelihood of the failure rate
function r(u), having observed data D described above, is given by
L(r(u), u >101D)
PROOF. To justify (1.16), we first note that the underlying random events are
the ages at failure or withdrawal. Thus the likelihood of the observed outcome is
specified by the likelihood of the failure ages and survivals until withdrawal. By
Assumption (3) of the General Sampling Model, we need not include any factor
contributed by the stopping rule, since the stopping rule does not depend on the
failure rate function r(-).
To calculate the likelihood, we use the fact that given r(.),
;or'U'U]
(See (1.4).) Specifically, if a unit is observed from age 0 until it is withdrawn at
age l, without having failed during the interval [0, lt], a factor e x p [ - S~ r(u)du]
is contributed to the likelihood. Thus, if no units fail during the test (i.e., k = 0),
the likelihood of the observed outcome is proportional to the expression given in
(1.16) for k = 0.
On the other hand, if a unit is observed from age 0 until it fails at age x~, a
factor
where the first sum is taken over units that failed while the second sum is taken
over units that were withdrawn. The upper limit ' ~ ' is for simplicity and intro-
duces no technical difficulty, since n(u)=-0 after observation ends. []
The likelihood (1.16) applies for any absolutely continuous life distribution. In
the important special case of an exponential life distribution model,
f(xl2) = 2 e-~x, the likelihood of the observed outcome takes the simpler form
L(AID) oc
[;o
2 kexp -2
]
n(u) du , k>~ 1,
(1.17)
[fo
exp - 2
]
n(u) du , k=O.
THEOREM 1.6. Assume that the test plan satisfies Assumptions (1), (2) and (3) of
the General Sampling Plan. Assume that k failures and the number of units operating
at age u, n(u), u >~O, are observed and that the model is the exponential density
f(x]2) = 2 e- ~x. Then
(a) k and T = So n(u) du together constitute a sufficient statistic for 2;
(b) kiT is the MLE for 2.
Note that the MLE, k/T, for 2 represents the number of observed failures divided
by the total time of test.
The maximum likelihood estimator is the mode of the posterior density corre-
sponding to a uniform prior (over an interval containing the MLE). A uniform
prior is often a convenient reference prior. Under suitable circumstances, the
analyst's actual posterior distribution will be approximately what it would have
been had the analyst's prior been uniform. To ignore the departure from uni-
formity, it is sufficient that the analyst's actual prior density changes gently in the
region favored by the data and also that the prior density not too strongly favors
some other region. This result is rigorously expressed in the Principle of Stable
Estimation [see Edwards, Lindman and Savage (1963)]. DeGroot (1970), pages
198-201, refers to this result under the name of precise measurement.
EXAMPLE 1.7. The exact likelihood can be calculated explicitly for specified
stopping rules. Suppose that withdrawal times are determined in advance. Then
the likelihood is
where n(Xs ) is the number surviving just prior to the observed failure at age x s.
To see this consider the airplane engine data in Example 1.2. Using Figure 1.3 as
a guide, the likelihood will have the following factors:
Life distributionmodelsand incompletedata 235
4r(xo))expI-~o"~4r(u)du]
corresponding to the probability that all 4 units survive to x(S) and the first failure
occurs at x(1).
2. For the interval (x(l), l(1)] we have the contribution
ex,I
corresponding to the probability that the remaining 3 units survive this interval.
3. For the interval (l(1), x(2)] we have the contribution
corresponding to the probability that the remaining 2 units survive to x(~) and the
failure occurs at x(z).
4. For the interval (x(2), l(2)] we have the contribution
expf
corresponding to the conditional probability that the remaining unit survives to
age l(2). Multiplying together these conditional probabilities, we obtain a likelihood
having the form shown in (1.18).
In the previous section we saw how to calculate the likelihood function for
general life distributions. This is required in order to calculate the posterior
distribution. Calculation and possibly graphical display of the posterior density
would conceivably complete our data analysis.
If we assume a life density p(xlO) and n(O) is the prior, then
p(x, O) = p(x] 0)~(0) is the joint density and p(x) = ~op(x[O)~(0) dO is the mar-
ginal or predictive density. Given data D and the posterior density r~(0[D), the
predictive density is
p(xlD) = foP(XlO)zr(OID)dO.
236 R. E. Barlow and F. Proschan
EXAMPLE 2.1. For the exponential density 2 e-xx, k ovserved failures, T total
time on test, and the General Sampling Plan, the likelihood is proportional to
2/` e - a t . For the natural conjugate prior,
b a 2a - 1 e - oa
~(2) =
r(a)
the posterior density is
P ( X > thk, T) =
f: e-'t/Tz(2]k, T ) d 2
(2.1)
+t+ T/
Of course, E[ t?] is the mean of the prior distribution while E[ OlD] is the
mean of the posterior distribution.
We wish to select a single value as representing our 'best' estimator of the
unknown parameter 0. To define the best estimator we must specify a criterion
of goodness (or equivalently, of poorness). Statisticians measure the poorness of
an estimator 0 by the expected 'loss' resulting from their estimator 0. One
very popular loss function is squared error loss: specifically, having observed data
D and determined the posterior density ~z(0[D), the expected squared error loss
is given by
E l ( 0 - 0)2ID] ; (2.2)
To find the minimizing value t), we add and subtract E ( O I D ) in the loss
function to obtain
THEOREM 2.2. The Bayes estimator of a parameter 0 with respect to squared loss
is the mean E ( 0 1D) of the posterior density.
Another loss function in popular use is the absolute value loss function:
It is easy to show:
THEOREM 2.3. The Bayes estimator of a parameter 0 with respect to the absolute
value loss function is the median of the posterior density. Specifically, the estimator
0 satisfies
Of course, the prior density and the loss function enter crucially in determining
a 'best' estimator. However, no matter what criterion is used, all the information
concerning the unknown parameter 0 is contained in the posterior density. Thus,
a graph of rc(0[D) is more informative than any single parameter of the posterior
density, whether it be the mean, the median, the mode, a quartile, etc.
bao-(a+ 1) e-b/O
~(o) =
r(a)
(See Diaconis and Ylvisaker (1979) for a proof of this result and for more general
results of this kind.) The corresponding Bayes estimator with respect to squared
error loss is
E[ OIk, T] _ (b + T) (2.7)
( a + k - 1)
g(O) = e x p [ - ~ ] . (2.8)
(2.9)
is the Bayes estimator. If n(O) is the natural conjugate prior, then it is easy to
verify that
b+ T ]a+k
g= b+t+ Tl '
i.e., this is the Bayes estimator of the probability of survival to age t given total
time on test T and k observed failures. Note that this ~ is precisely the marginal
probability of survival until time t.
Specifically, we choose a set C on the positive axis (since we are dealing with
lifetime) such that
f rr(OID)dO= 1 - a. (2.10)
C
Such a set C is called a Bayesian (1 - a) 100 percent credible set (or credible
interval if C is an interval) for 0.
Obviously, the set C is not uniquely determined. It would seem desirable to
choose the set C to be as small (e.g., least length, area, volume) as possible. To
achieve this, we seek a constant c 1 _ ~ and a corresponding set C such that
f re(OlD)dO= 1 - ~. (2.12)
C
A set C satisfying (2.11) and (2.12) is called a highestposterior density credible set
(Box and Tiao, 1973). In general, C would have to be determined numerically with
the aid of a computer.
For the exponential model 2 e-ax, the natural conjugate prior is the gamma
density. Since the gamma density is a generalization of the chi-square density, we
recall the definition of the latter so that we can make use of it to determine
credible intervals for the failure rate of the exponential.
THEOREM 2.6. Let k failures and total time on test T be observed under sampling
assumptions (1), (2) and (3) (Section 1)for the exponential model 2e -zx. Let )~
have the posterior density corresponding to the natural conjugate prior
b a )a - 1 e- b2
~(~) -
r(a)
where z~(n) is the lOOfl percentage point of a chi-square distribution with n d.f.; i.e.,
PROOF. It is easy to verify that (b + T)). given the data has a gamma density,
1~a+k-1 e 2
F(a + k)
COROLLARY 2.7. For 2(a + k) large (say 2(a + k ) > 30), the normal approxima-
tion provides the approximate credibility statement
where YI, Y2. . . . , Yn are independent unit exponentials, the Central Limit
Theorem (e.g., Hoel, Port and Stone, 1971) applies. Note that EX2(2n) = 2n and
Var[z2(2n)] = 4n. Thus,
Z2(2n) - 2n
COROLLARY 2.8. Let k failures and T total time on test be observed under the
General Sampling Plan assumptions (1), (2) and (3) (Section 1), for the exponential
model O-l e-~/o. Let 0 have the natural conjugate prior with integer a, then
PROOF. Since 0 has the natural conjugate prior distribution for the model
0 - 1 e -x/°, then ,~ = 1/0 has the natural conjugate prior for the model 2 e -~x.
(2.16) follows from (2.14). []
lim P { 2 n ( ~ . , - a,) ~ x}
n~o~
or
1 f f e x p [)t(x - a)]
- | e - uuk - 1 du ,
(k 1)! ~o
-oo<x<~, -oo<a<~,A>O (3.2)
For all parameter values, (3.4) is increasing in x. Hence, if we wish to allow the
possibility that the failure rate may be decreasing we must choose the Weibull
model, (3.1'), with e < 1.
The Weibull model appears to furnish an adequate fit for some strand lifetime
data with estimated values of e less than 4. On the other hand, it has been
empirically observed that for strength data, estimates for e using the Weibull
model are often large ( > 27 in some cases). This suggests that (3.2') may provide
a better model for strand strength data.
Usually we wish to quantify our uncertainty about a particular aspect of the life
distribution, such as the probability of surviving x hours. For the three parameter
Weibull model, this is given by
f f ( x l a , 2, ~) = e x p { - [ 2 ( x - a)]~}. (3.6)
L ( a , ~, ).ID)
oc c~2 k~
i=l
(x i - a)
l 'I Iraexp - 2~ an(u) (u - a) ~ - 1 du
1t
for a ~< xi and ~, 2 > 0. Suppose there are m withdrawals and we pool observed
failure and loss times and relabel them as
f
a°o k +m F t(O
n(u) (u - a) ~ - ' du = Z (n-i+ 1) (u-a)~ ,du
i= 1 ,I t(i_ 1)
+ (n - k - m) f/ (k+m)
(u - a) ~ - I d u . (3.7)
Table 3.1
Ordered failure ages of pressure vessels life
tested at 68~o of mean rupture strength (n = 21,
observation to 13488 hours)
1 4000
2 5376
3 7320
4 8616
5 9120
Table 3.2
Ordered failure ages of pressure vessels life
tested at 68~o of mean rupture strength (failures
between 13488 hours and 20568 hours)
1 14400
2 16104
3 20231
4 20233
Life distribution models and incomplete data 245
Figure 3.1 displays credibility contours for ct and 2 after 13488 hours of testing
and again after 20 568 hours of testing. The posterior densities were computed
relative to uniform priors. The posterior density computed after 20568 hours
could also be haterpreted as the result of using the posterior (calculated on the
basis of Table 3.1 and a fiat prior) as the new prior for the data in Table 3.2. A
qualitative measure of the information gained by an additional year of testing can
be deduced by comparing the initial (dark) contours and the tighter (light)
contours in Figure 3.1.
2 , O0
~after 13 4 8 8 h o u r s
1 . 5 0 --
1. O0 --
(D
0
0
O. 5 0 --
O. O0 -- ~ i !f] J I I r i i ~ ~i ,: i i I I r ]1 J I [ J J J I f t r I I I I Ii I i i Ir
0.'t0 0.80 1.20 1.60 2.00 2.40 2.80 3.20 3.60 4.00 ~t.40 4 . 8 0
Fig. 3.1. Highest probability density contours for ~ and 2 for Kevlar/epoxy pressure vessel life test
data, T h e pressure vessels w e r e tested at 68~o stress level.
246 R. E. Barlow and 1:. Proschan
To predict pressure vessel life at the 68~o stress level, we can numerically
compute
P[X>tlD]=fo~fo°~e-(X°°~(a, AlD)d=d2
where rt(~, 2[D) must be numerically computed using the given data, D.
If the mean life
,(1+:)
O-
or the standard deviation of life are of interest, their posterior densities can be
computed by making a change of variable and integrating out the nuisance
parameter. For example, if a = 0 in the Weibull model and we are interested in
the mean life, 0, we can use the Weibull density in terms of c~ and 0.
1+ F 1+-
f ( x l a , O) = a x ~- 1 exp - a
0 0
to compute the joint posterior density rc(~t, 0[ D). The prior for a and 2 must be
replaced by the induced prior for a and 0. This may be accomplished by a change
of variable and by computing the appropriate Jacobian. The marginal posterior
density of 0 is then
4. I. Section 1
In the General Sampling Plan we needed to assume that any stopping rules
used were noninformative concerning the failure distribution. The need for this
assumption was pointed out by Raiffa and Schlaiffer (1961). Examples of infor-
mative stopping rules were given by Roberts (1967) in the context of two stage
sampling of biological populations to estimate population size (so-called capture-
recapture sampling).
Life distribution models and incomplete data 247
E~[ b(D)l 0] = 0
To find this c, consider Y = O(D)/O and note E Y = 1. Then we need only find
c such that
ElJ(Cr- 1)210]
is minimum. This occurs for co = E Y / E Y 2 which is clearly not 1. Hence 0(D)
is sample theory inadmissible. Sample theory unbiasedness is not a viable
criterion.
For ~_arge k, 0 ( D ) = T/k will be approximately the same as our Bayes
estimator. However, T/k is not recommended for small k.
Since tables of the chi-square distribution have in the past been more accessible
than tables of the gamma distribution, we have given the chi-square special
treatment. However with modern computing facilities, we really only need to use
the more general gamma distribution.
P [ "~<(ln2)
lT T'rc(A)=~l=0"50" (4.1)
Suppose now that T is observed and we accept the probability statement (4.1).
Consider the following hypothetical bet.
(i) If ~. < (ln2)/T we lose the amount e- r;
(ii) If 2 >/(In 2)/T we win e- r.
We can pretend that the true 2 is somehow revealed and bets are paid off. If
we believe statement (4.1), then given T such a bet is certainly fair.
Now let us compute our expected gain before T is observed (preposterior
analysis). This is easily seen to be (conditional on 2)
- f on 2~/~2 e - ~ t e - t d t + f ~ 2 e - ~ t e - ~ d t = - 2[ 2- ~/~- 1]
,d 0 ,)(ln 2)/Z 1+ 2
which is negative for all 2 > 0. Note that this is what we subjectively expect, since
as (improper) Bayesians, every probability (and presumably even an improper
prior) is subjective.
The contradiction lies in the observation that
1. conditional on 2 and prior to observing T, our expected winnings are nega-
tive for all 2;
2. conditional on T, our expected loss is zero (using the improper prior
~ ( A ) = 1/2).
The source of the contradiction is that we have not measured our uncertainty
for all events by probability. For example, we have assigned the value ~ to the
event 2 < 2 o for all 2 0 > 0 ; i.e., ~ r c ( 2 ) d 2 = S ~ ( 1 / A ) d 2 = ~ . We can
prove that for any set of uncertainty statements that are not probabilistically
based (relative to proper distributions), a system of bets can be constructed which
will result in the certain loss of money. A bet consists of paying pz < z dollars
to participate with the understanding that if an event E occurs you win z dollars
and otherwise you win nothing.
4.4. Section 3
The Weibull distribution is one of several extreme value distributions. See
Barlow and Proschan (1975), Chapter 8, for a more advanced discussion of
extreme value distributions.
Life distribution models and incomplete data 249
Acknowledgements
References
Barlow, R. E. and Proschan, F. (1975). Statistical Theory of Realibility and Life Testing. Holt, Rinehart
and Winston, New York.
Bickel, P. J. and Blackwell, D. (1967). A note on Bayes estimates, Ann. Math. Statist. 38, 1907-1911.
Box, G. E. P. and Tiao, T. C. (1973). Bayesian Inference in Statistical Analysis. Addison-Wesley,
Reading, MA.
De Groot, M. H. (1970). Optimal Statistical Decisions. McGraw-Hill, New York.
Diaconis, R. and Ylvisaker, D. (1979). Conjugate priors for exponential families. Ann. Statist. 7,
269-281.
Edwards, W., Lindman, H., and Savage, L. J. (1963). Bayesian statistical inference for psychological
research. Psychological Rev. 70, 193-242.
Hoel, P. G., Port, S. C., and Stone, C. J. (1971). Introduction to Probability Theory. Houghton Mifflin,
Boston, MA.
Lindley, D. V. (1978). The Bayesian approach. Scandinavian J. Statist. 5, 1-26.
Pearson, E. S. and Hartley, H. O. (1958). Biometnka Tables for Statisticians. Vol. 1. The University
Press, Cambridge, England.
Raiffa, H. and Schlaiffer, R. (1961). Applied Statistical Decosion Theory. Harvard Business School,
Boston, MA.
Roberts, H. V. (1967). Informative stopping rules and inferences about population size. J. Amer.
Statist. Assoc. 62, 763-775.
Smirnov, N. V. (1952). Limit distributions for the terms of a variational series. Trans. Math. Soc.
Ser. 1, 1-64.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 l/1
/
© Elsevier Science Publishers B.V. (1988) 251-280
* Research supported by the Air Force Office of Scientific Research, AFSC, USAR, under Grant
AFOSR 82-K-0007.
251
252 G. M. Mimmack and F. Proschan
(1976, 1978), Kitchin, Langberg and Proschan (1983), Nelson (1972), Fleming
and Harrington (1979), and Chen, Hollander and Langberg (1982).
One of the by-products of the estimation process is an estimate of the failure
rate function: here, another issue is raised. It is evident that survival function
estimators that are step functions do not provide useful failure rate function
estimators: Miller (1981) mentions smoothing the Kaplan-Meier estimator for
this reason and summarizes the development of other survival function estimators
that may be obtained by considering a special case of the regression model of Cox
(1972). These estimators generally correspond to failure rate function estimators
that are step functions and utilize at most part (but not all) of the interval
information contained in the data. Whittemore and Keller (1983) give several
more refined failure rate function estimators that are step functions and utilize full
interval information. They also describe even more complex estimators that utilize
full interval information: however, these are not computationally convenient com-
pared with their simpler estimators. It seems, from their work, that a successful
rival of the Kaplan-Meier estimator should be only marginally more complex than
it (so as to be computationally convenient and yet yield a useful failure rate
function estimator) and also should utilize more than ordinal information.
In Section 2, we propose an estimator that not only provides a reasonable
failure rate function estimator but also utilizes interval information. Moreover, it
is computationally simple. Our estimator is a discrete counterpart of two versions
of a continuous estimator proposed independently by Kitchin, Langberg and
Proschan (1983) and Whittemore and Keller (1983). The motivation for the
construction of our estimator is the same as that of the former authors, and our
model is the discrete version of theirs: in contrast, the latter authors assume the
more restrictive model of random censorship and obtain their estimator by the
method of maximum likelihood. This provides an alternative method of deriving
our estimator.
The remaining sections are concerned with properties of our estimator. As this
presentation is expository, proofs are omitted: Mimmack (1985) provides proofs.
In Section 3, we explore the asymptotic properties of our estimator under
increasingly restrictive models. Our estimator is strongly consistent and asymptoti-
cally normal under conditions more general than those typically assumed.
Section 4 deals with the relationships among our estimator, the Kaplan-Meier
estimator, and the above-mentioned estimator of Kitchin et al. and Whittemore
and Keller. The section ends with an example using real data.
In Section 5, we continue the comparison of the new estimator and the
Kaplan-Meier estimator: since the properties of the new estimator are expected
to resemble those of its continuous counterparts, we discuss the implications of
simulation studies designed to investigate the small sample behaviour of these
estimators. We also present the results of a Monte Carlo pilot study designed to
investigate the small sample properties of our estimator.
Piecewise geometric estimation of a survival function 253
2. Preliminaries
In this section we formulate the problem in statistical terms and define our
cstimator.
Let X denote the lifelength of a randomly chosen unit, where X has distribution
function G. Suppose that n identical items are placed on test. The resultant
sample consists of the pairs (Z1, bl) . . . . . (Z~, b~), where Z; represents the time
for which unit i is observed and b; indicates whether unit i fails while under
observation or is removed from the test before failure. Symbolically, for
i = 1, ..., n, we have
ai = I(X;<... Y~).
(Xt, Yt), - . . , (Xn, Yn) are assumed to be independent random pairs. Elements
of a pair X; and Y~, where i = 1, . . . , n, are not assumed to be independent.
We assume that the lifelength and censoring random variables are discrete. Let
5f = {x~, x 2. . . . } denote the set of possible values of X and Y¢ = {Yl, Y2. . . . }
denote the union of the sets of possible values of Y~, Yz . . . . , where ~¢ ___ &r. The
survival probabilities of interest are denoted P ( X > xk), k = 1, 2 . . . . . where
P ( X > xk) = G(x~) = 1 - G(xk), k = 1, 2 , . . . .
It is evident that this formulation differs from that of the model of random
censorship which is generally assumed in the literature, and in particular, by
Whittemore and Keller (1983). These authors assume that the lifelength and
censoring random variables are continuous, that the corresponding pairs X; and
Y~, where i = 1, 2, ..., are independent, and that the censoring random variables
are identically distributed. Although Kaplan and Meier (1958) assume only inde-
pendence between corresponding lifelength and censoring random variables,
Breslow and Crowley (1974), Petersen (1977), Aalen (1976, 1978), and others--all
of whom describe the properties of the K a p l a n - M e i e r e s t m a t o r - - a s s u m e also
that the censoring random variables are identically distributed. Our formulation
is the discrete counterpart of that of Kitchin, Langberg and Proschan (1983):
likewise, our estimator is the discrete counterpart of theirs.
Before describing our estimator, we give the notation required.
Let nl be the random number of distinct uncensored observations in the sample
and let t I < t2 < • • • < tn, denote these distinct observed failure times, with to = 0.
Let n 2 be the random number of distinct censored observations in the sample and
let s~ < s 2 < • • • < sn: denote these times, with s o --- 0.
Let D; be the number of failures observed at time t;:
C i = ~ I ( Z j = s i, b j = 0 ) for i = 1. . . . . n 2.
j=l
Let fin(t) --- 1 - Fn(t ) denote the proportion of observations that exceed t:
Let T~. be a measure of the total time on test in the interval (t;_ 1, ti]:
k
P(X>Xk): 1-I [ 1 - P ( X = x + I X > ~ x ; ) ] for k = 1 , 2 . . . . . (2.1)
j=l
It is evident from (2.1) that we may estimate our survival function at x k from
estimates of the failure rates at xl, x2 . . . . , x k. In the experimental situation,
failures are not observed at all the times x 1, x 2 . . . . so specific information about
the failure rates at m a n y of the possible failure times is not available. Having
observed failures at q, t 2, . . . , t,~, we find it simple to estimate the failure rates
P ( X = te]X >~ ti), i = 1. . . . . n 1. However, the question of how to estimate the
failure rates at the intervening possible failure times requires special consideration.
One a p p r o a c h - - t h a t of Kaplan and Meier (1958), Nelson (1969) and o t h e r s - - i s
to estimate the failure rates at these intervening times as zero since no failures are
observed then. However, not observing failures at some possible failure times may
be a result of being in an experimental situation rather than evidence of very small
Piecewise geometric estimation of a survival function 255
failure rates at these times, so we discard this approach and consider nonzero
estimates.
It is reasonable to assume that the underlying process possesses an element of
continuity in that adjacent failure rates do not differ radically from one another.
Thus we consider using the estimate of the failure rate at t,. to estimate the failure
rate at each of the possible failure times between t;_ 1 and ti, where i = 1, ..., n~.
We are therefore assuming that our approximating distribution has a constant
failure rate between the times at which failures are o b s e r v e d - - t h a t is,
Inspection of (2.3) and (2.4) reveals that our estimating function is constructed
from the geometric survival functions G1, . . . , Gn,, where G; is used in the inter-
val (t i_ 1, ti], i = 1 . . . . . n l . Consequently, the estimator (2.3) is called the Piece-
wise G e o m e t r i c E s t i m a t o r (PEGE).
It remains to define estimators of the failure rates ql . . . . , qm" This was origi-
nally done by separately obtaining the maximum likelihood estimators of the
parameters of n 1 truncated geometric distributions: the procedure is outlined at
a later stage because it utilizes the geometric structure of (2.3) and therefore
provides further motivation for the name ' P E G E ' . A more straightforward but less
appealing approach is to obtain the maximum likelihood estimates of q~, . . . , q,,
directly: denoting by L the likelihood of the sample, we have
Substituting (2.3) into this expression and differentiating yields the unique maxi-
mum likelihood estimates
qi = Di/Ti, i = 1. . . . , n1.
256 G. M. Mimmack and F. Proschan
Substituting 01, . . . , an, into (2.3), we finally obtain our estimator, formally defined
as follows.
1 forxk<O or n l = O ,
i--1
(1 - D f f r , ) # { ~ : , . . . . . . ~Xk} IX (1 - D j / T j ) # { m : t , - ' < x " ~ ' J }
j=l
: ( x > xk) = for ti_ 1 < Xk <~ ti, i = 1, . . . , hi, n I > 0 ,
nl
I-1(1 - Dj/Tj) #(m:t' ..... <~t,}
j=1
where the possible values of X* are assumed to be 1, 2 . . . . , N~, N,.+ . The distribu-
tion G* of X* is then given by
Moreover, we note that this is the form of the failure rate estimators in the
intervals (t o, q ] . . . . . (t,l, oo) defined for the Piecewise Exponential Estimator
Piecewise geometric estimation of a survival function 257
1 for t < 0 or nl = 0 ,
i--1
exp[-(t- ti_,)2i] I-[ e x p [ - ( t j . -
j=1
tj_ 1),~j]
P * ( X > t) = for ti_ l < t <~ t/, i = l . . . . , n l , nl > O , (2.5)
n!
I-'[ e x p [ - ( t j - tj_ 1),~j]
j=l
7/=
f t ti nFn(u) du for i-- 1. . . . . n l .
i-1
For i = 1, ..., n 1, 2/is the failure rate in the interval (ti_ 1, t/] and 7,- is the total
time on test in this interval.
The PEXE is a piecewise exponential function because its construction is based
on the assumption of constant failure rate between observed failures: just as a
constant discrete failure rate characterizes a geometric distribution so a constant
continuous failure rate characterizes an exponential distribution. Thus the P E G E
is the discrete counterpart of the PEXE.
Returning to our introductory discussion about the desirable features of survival
function estimators, we now compare the P E G E with other estimators in terms
of these and other features.
First, the P E G E is intuitively pleasing because it reflects the continuity inherent
in any life process. The Kaplan-Meier and other estimators that are step
functions do not have this property.
Second, we note that the P E G E utilizes interval information from both cen-
sored and uncensored observations. It is therefore more sophisticated than the
Kaplan-Meier and Nelson estimators. Moreover, none of the estimators of
Whittemore and Keller utilizes more information than does the PEGE.
Third, the P E G E provides a simple, useful estimator of the failure rate function.
While this estimator is naive compared with the nonlinear estimators of Whitte-
more and Keller, the P E G E has the advantage of being simple enough to calculate
by hand--moreover it requires only marginally more computational effort than
does the Kaplan-Meier estimator.
Regarding the applicability of the PEGE, we note that use of the P E G E is not
restricted to discrete distributions because it can be easily modified by linear
interpolation or by being defined as continuous wherever necessary. This is
theoretically justified by the fact that the integer part of an exponential random
variable has a geometric distribution: by defining the P E G E to be continuous, we
258 G. M. Mimmack and F. Proschan
are merely defining a variant of the PEXE. The properties of this estimator follow
immediately from those of the PEXE.
Finally, apart from being intuitively pleasing, the form of the P E G E allows
reasonable estimates of both the survival function and its percentiles. The
Kaplan-Meier estimator is known to overestimate because of its step function
form. We show in a later section that the P E G E tends to be less than the
Kaplan-Meier estimator, and therefore the P E G E may be more accurate than the
Kaplan-Meier estimator. Whittemore and Keller give some favourable indications
in this respect. They define three survival function estimators that have constant
failure rate between observed failure times. One of these is the PEXE, modified
for ties in the data: the form of the failure rate estimator is the same as the form
of the P E G E failure rate estimator--specifically, for i = 1. . . . , nl,
The third of these estimators is obtained from the average of the two failure rate
estimators described by (2.6) and (2.7).
In a simulation study to investigate the small sample properties of these three
estimators, Whittemore and Keller find that the first estimator tends to under-
estimate the survival function while the second tends to overestimate the survival
function. From these results, we expect the P E G E to underestimate the survival
function and its percentiles. Whittemore and Keller do not record further results
for the first two estimators: however, they do indicate that, in terms of bias at
extreme percentiles, variance and mean square error, the third estimator tends to
be better than the Kaplan-Meier estimator.
The implications for the discrete version of the third estimator are that, in terms
of bias, variance and mean square error, it will compare favourably with the
Kaplan-Meier estimator. An unanswered question is whether the performance of
this estimator is so superior to the performance of the P E G E as to warrant the
additional computational effort required for the former.
Piecewise geometric estimation of a survival function 259
This section treats the asymptotic properties of the P E G E and of the cor-
responding failure rate function estimator. The properties of primary interest are
those of consistency and asymptotic normality: secondary issues are asymptotic
bias and asymptotic correlation.
Initially considering a very general model, we obtain the limiting function of the
PEGE and show that the s e q u e n c e s {Pn(X>Xk)}~°=l and {Pn(X=x/,p
X >/ X k)}~oo= ~ converge in distribution to Gaussian sequences. We then explore the
effects of making various assumptions about the lifelength and censoring random
variables. Under the most general model, the PEGE is not consistent and the
failure rate estimators are not asymptotically uncorrelated: a sufficient condition
for consistency is independence between corresponding lifelength and censoring
random variables, and a sufficient condition for asymptotically independent failure
rate estimators is that the censoring random variables be identically distributed.
However, it is not necessary to impose both of these conditions in order to ensure
both consistency and asymptotic independence of the failure rate estimators:
relaxing the condition of independent lifelength and censoring random variables,
we give conditions under which both desirable properties are obtained.
Before investigating the asymptotic properties of the PEGE, we describe the
theoretical framework of the problem, give some notation, and present a prelimi-
nary result that facilitates the exploration of the asymptotic properties of the
PEGE.
The probability space (f2, ~, P) on which all of the lifelength and censoring
random variables are defined is envisaged as the infinite product probability space
that may be constructed in the usual way from the sequence of probability spaces
corresponding to the sequence of independent random pairs (X1, Yl),
0(2, II2). . . . . Thus 1"2 consists of all possible sequences of pairs of outcomes
corresponding to pairs of realizations in 5f x Y¢: the first member of each pair
corresponds to failure at a particular time and the second member of each pair
corresponds to censorship at a particular time--that is, for each co in f2,
k = 1 , 2 , . . . and j = 1,2 . . . . .
(Xi, Yt)(co) = (Xi(co), Y~.(co)) = (Xk, yj) if the ith element of the infinite
sequence co is the pair of outcomes corresponding to failure at xg and
censorship at yj.
1
lira ~ P(Zi<~xk)=F(xk) for k = 1 , 2 , . . . .
n~°° H i=1
260 G. M. Mirnmack and F, Proschan
1
lim - ~ P ( Z ~ < x k , 6;= 1 ) = F l ( x k ) for k = 1,2 . . . . .
n ~ o o iv/ i= 1
It is evident that a sufficient condition for (A1) and (A2) is that the censoring
random variables be identically distributed.
Definitions of symbols used in this section are given below. Assumptions (A1)
and (A2) ensure the existence of the limits defined. Let
P k i = P ( Z i = X k , ~i= 1) f o r k = 1 , 2 , . . . andi--- 1. . . . . n,
R k t = P ( Z i = x k, bi = 0 ) fork= 1,2,... andi= 1,...,n,
1 n
Pk = lim -~ Z P k ; = F I ( x k ) - F ' ( X k - , ) f o r k = 1,2 . . . . .
n~°° l'l i=1
1
R k = lim - Z Rk, f o r k = 1,2 . . . . .
n~oo n i= 1
DEFINITION 3.1. Let t2* c f2 be the set of infinite sequences which contain, for
each possible failure time, at least one element corresponding to the outcome of
observing failure at that time--that is,
The proposition is proven by showing that the set of infinite sequences that do
not contain at least one element corresponding to the outcome of observing failure
at each possible failure time x, has probability z e r o - - t h a t is,
P ( nlim°°~ ~=1
~ {Xi= xk' Yi>/xk}C) =0 for k = 1 , 2 .....
As the pairs (X1, Y1), (X2, Y2), ... are independent, this is equivalent to proving
the following equality:
n
lira l~ ( 1 - P ( X i = x k, Y,.>tXk))=O for k = 1,2 . . . . . (3.1)
n~ i=l
Piecewise geometric estimation of a survival function 261
Since [I i=
°° 1 (1 - p~) = 0 if and only if ~ i=
o~ 1 Pi = OO, where {Pi}~= 1 is any sequence
of probabilities, and since (A2) implies that
we have (3.1).
The importance of the preceding proposition lies in the simplifications it allows.
It turns out that, on 12" and for n large enough, the P E G E may be expressed in
simple terms of functions that have well-known convergence properties. Since
P(12*) = 1, we need consider the asymptotic properties of the P E G E on O*
alone: these properties are easily obtained from those of the well-known functions.
In order to express the P E G E in this convenient way, we view the estimation
procedure in an asymptotic context.
Suppose co is chosen arbitrarily from f2*. Then, for each k, there is an N
(depending on k and co) such that X;(co) = xj and }',.(co)>~ xj for j = 1. . . . . k and
some i ~< N. Consequently, for n >~ N, the smallest k distinct observed failure times
tl, . . . , tk are merelY x l , . . . , x k, and, since the set of possible censoring times is
contained in f , the smallest k distinct observed times are also x l , . . . , x k. T h e
first k intervals between observed failure times are simply (0, x~],
(Xl, x2] . . . . . (Xk- 1, Xk], and the function T~,~ defined on the ith interval is given
by the number of units on test just before the end of the ith interval--that is,
Likewise, we express the function D~, n defined on the ith interval in terms of the
empirical subdistribution function F2 as follows:
Consequently, taking the limit of each side and using Proposition 3.2, we have
P
n
lim/~,(X> Xk)= lim
n~oo i= 1
1-
F~ (xi) - F~ (x i_ l )
~,,-(x~.-5
_
')
f o r k = 1,2 . . . . ] = 1.
J
262 G. M. M i m m a c k and F. Proschan
{i~l (1 Fln(Xi)-Fl(xi-1) °°
- ffn(X~--l) )}k=l
The proofs of the results that follow are omitted in the interest of brevity. The
most general model we consider is that in which only conditions (A1) and (A2)
are imposed. The following theorem identifies the limits of the sequences
{P.(X = x~lX>~ x~)}~=, and {/3~(X> Xk)}~= ~ for k = l, 2 . . . . and establishes
that the sequences {/S.(X= XkIX>~Xk)}~=~ and {/S.(X> Xk)}ff=l converge to
Gaussian sequences.
FI(Xk) - Fl(xk - 1)
lim P . ( X = x k l X >>,x k ) = fork=l, 2,....
n~o~
fi(Fl(xi)-Fl(xi-1))
lira /~.(X > xk) = 1- ~_ for k = 1, 2 . . . . .
~~ i= 1 F ( x i - l)
where
~,* = (P~,/~(xk, _ , ) . . . . . P,~,JF(xkM- ,)) ,
q-1 r--,
~q~ = PkqPkr 2 2 (~kinkj ~- ~kM+ki, kj ~[- aki, kM+kj -~ ~kM+ki, kM+k j)
i=1 j=l
/(~(x,~q_ ~ ) ~ ( ~ , _ ,)y
r--1
r= 1,...,M,
r = 1, . . . , M ,
.... 2M,
where
p** =
\i=l
(1 - P,./F(x,_ ,)1 . . . . . l-I (1 - P / i f ( x , _ 1)) ,
i=1
)
Z** /
].tJqr f q = l ..... M;r=l ..... M,
kq kr
Cry** ---- 1--[ (1 -- P / i f ( x , _ 1)) ~ I (1 -- e j / ~ ' ( x j _ 1))
i=1 j=l
kq kr
' Z 20"/*m/[(1 -- e l / f f ( X l _ 1)(1 - Pro~if(x,,,_ 1))1 for q <~ r.
l=1 m=l
F l ( x k ) - F l ( x k - 1) _ P ( X : xk)
for k = 1, 2 . . . . . (3.4)
ff(Xk_l) P(X>/x~)
The theorems below give conditions under which this equality holds. As for
correlation, it is evident from the structure of the P E G E that any two elements
of the sequence {Pn(X> xg))ff= 1 are correlated. Consequently the matrix 2~**
264 G. M . M i m m a c k and F. Proschan
cannot be reduced to a diagonal matrix under even the most stringent conditions.
However it turns out that, under certain conditions, the asymptotic correlation
between pairs of the sequence {/Sn(X = xklX>~ x~)}ff= 1 is z e r o - - t h a t is, 1;* is a
diagonal matrix.
The following theorem shows that independence between lifelength and cen-
soring random variables results in strongly consistent (and therefore asymptoti-
cally unbiased) estimators. However any pair in the sequence
{/~n(X= xklY>~xk)}2= 1 is asymptotically correlated in this case. Since the
matrices ~2" and Z** have the same form as in the theorem above, they are not
explicitly defined below.
THEOREM 3.4. Suppose (i) the random variables X i and Y,. are independent for
i = 1, 2 , . . . , and
(ii) there is a distribution function H such that
1 n
lim -- Z P(Y¢<<-x~)=H(x*) fork= 1, 2 , . . . .
n~°° n i=1
Then
k
(iii) F l ( x k ) = ~ P ( X = x i ) H ( x i_ 1) and ff(x~) = P ( X > x k ) H ( x ~ ) for k = 1, 2 . . . .
i=1
where k~ < k2 < "'" < k M are arbitrarily chosen integers and
where k~ < k 2 < " ' < k M are arbitrarily chosen integers and
A sufficient condition for (A1), (A2) and assumption (ii) of the preceding
theorem is that the censoring random variables be identically distributed. In this
case the failure rate estimators are asymptotically independent and the matrix Z**
Piecewise geometric estimation of a survival function 265
is somewhat simplified: The conditions of the following corollary define the model
of random censorship widely assumed in the literature.
COROLLARY 3.5. Suppose (i) the random variables X i and Y,. are independent for
i = 1, 2 , . . . , and
(ii) the random variables Y1, Y2 . . . . are identically distributed.
Then
(iii) with probability 1,
where
l~* = (P(X = Xk~lX>~ xk, ) . . . . . P ( X = XkM]X>~ Xk~)),
O.j:r={Po(X=Xkq'X~Xkq)P(X~Xkq'X~xkq)/F(Xkq-1) for q = r,
for q # r.
where
~,** = (P(X> x~,) . . . . . P(X > x~..)).
r q=l,...,M;r=l,...,M'
kq
aS** = P ( X > Xk,)P(X > Xkr) 2 P ( X = x i l X >~ x,)/[ff(x i_ ,)
i=1
Having dealt with the most restrictive case in which the lifelength and censoring
random variables are assumed to be independent, we now consider relaxing this
condition. It turns out that independence between corresponding lifelength and
censoring random variables is not necessary for asymptotic independence between
pairs of the sequence of failure rate estimators: if the censoring random variables
are assumed to be identically distributed but not necessarily independent of the
corresponding lifelength random variables, then the failure rate estimators are
asymptotically independent. However both the survival function and failure rate
estimators are asymptotically biased. The following corollary expresses these facts
formally.
266 G. M. Mimmack and F. Proschan
COROLLARY 3.6. Suppose (i) the random variables Y1, Y2 . . . . are identically dis-
tributed.
Then
(ii) P k = P ( Z = x k, b = 1) and F ( x k ) = P ( Z > x k )
#*, I2" ,
n
where
#* = (Pk~/?(Xk, 1 ) , ' ' . , Pk,/F(xk~ 1)),
Z* = {l~i~ ) i _ 1..... M;j--1 ..... M'
where
#** = (1 - P,/ff(x i_1)), . . ' , l~ (1 - PJF(x i_1)) ,
\i=1 i=1
kl
aS;* = I~ (1 - Pi/?(x~_ ,)) 1~ (1 - Pm/ff(Xm_ 1))
i--1 m=l
gj
• ~ Pr/[(F(xr- 1))2( 1 - Pr/ff(Xr-1))] forj<~ l.
r 1
The corollaries above give sufficient (rather than necessary) conditions for the
two desirable properties of (i) consistency and (ii) asymptotic independence
between pairs of the sequence of failure rate estimators {fi,(X = x k l X >1 Xk)}k~_ 1.
The final corollaries show that both of the conditions of Corollary 3.5 are not
necessary for these two desirable properties: the conditions specified in these
corollaries are not so stringent as to require that corresponding censoring and
lifelength random variables be independent (as in Corollary 3.5), but rather that
they be related in a certain way.
if and only if
COROLLARY 3.8. Suppose (i) the random variables Y1, Y2, " " are identically dis-
tributed, and
(ii) P(Y,>~ XklX = Xk) = P(Y,>~ XkIX>>- Xk) for k = 1, 2 . . . . and i = 1, 2, . . . .
Then
where
t~* = (P(X = x~, IX >t x~,) . . . . , P(X = x~, Ix >>-XkM)),
Z* = {G;j },~, ..... ~;j=, ..... ~,
where
p** : ( P ( X > xk,) . . . . . P ( X > x k , ) ) , '
• P(X>x~lX>/x~)] forj<<.l.
The last two corollaries are of special interest because they deal with con-
sistency and asymptotic independence in the case of dependent lifelength and
censoring random variables--a situation that is not generally considered despite
its obvious practical significance. Desu and Narula (1977), Langberg, Proschan
and Quinzi (1981) and Kitchin, Langberg and Proschan (1983) consider the
continuous version of the model specified in the last two corollaries.
The condition specifying the relationship between lifelength and censoring
random variables is in fact a mild one: re-expressing it, we have the following
condition:
This condition specifies that the failure rate among those under observation at any
particular age is the same as the failure rate of the whole population of that age.
268 G. M. Mimmack and F. Proschan
that the KME consistently overestimates suggests that its form is inappropriate.
Some indications about the bias of the PEGE are given by considering the
relationship between the PEGE and the KME.
Under certain conditions (for example, if there are no ties among the uncen-
sored observations), the PEGE and the K M E interlace: within each failure inter-
val, the PEGE crosses the K M E once from above. This is not true in general,
however. It turns out that the K M E may have large steps in the presence of ties.
In the case of the PEGE, however, the effect of the ties is damped and the PEGE
decreases slowly relative to the KME. In general, therefore, it is possible to relate
the PEGE and the KME only in a one-sided fashion: specifically, the PEGE at
any observed failure time is larger than the K M E at that time. Examples have
been constructed to show that, in general, the PEGE cannot be bounded from
above by the KME. The following theorem relates /s (the PEGE) and P (the
KME).
It is evident that the condition in (ii) is met if there are no ties among the
uncensored observations: this is likely if the sample is small. From the relation-
ships in the theorem, we infer that the bias of the PEGE is likely to be of the
same order of magnitude as that of the KME. Further indications about bias are
given later.
Having considered some of the practical and physical features of the PEGE
and the KME, we turn briefly to asymptotic properties--briefly because the
PEGE and the K M E are asymptotically equivalent--that is,
The practical implication of this is that there is little reason for strong preference
of either the PEGE or the K M E if the sample is very large.
We now compare the models assumed in using the K M E and the PEGE. In
the many studies of the KME, the most general model includes the assumption
of independence between corresponding life and censoring random variables. Our
most general model does not include this assumption. However this difference is
not important because the assumption of independence is used only to facilitate
the derivation of certain asymptotic properties of the KME: in fact, the definition
of the K M E does not depend on this assumption, and the K M E and the PEGE
are asymptotically equivalent under the conditions of the most general model of
the PEGE. Therefore this assumption is not necessary for using the KME.
The other difference between the models assumed is that the PEGE is designed
specifically for discrete life and censoring distributions while the Kaplan-Meier
model makes no stipulations about the supports of these distributions. However,
Piecewise geometric estimation o f a survival function 271
THEOREM 4.2. Let P * * ( X > t) denote the modified PEXE of the survival proba-
bility P(X > t) for t > O.
(i) P ( x > O < e * * ( x > t ) fort>O.
(ii) l f nF,(tj_l)/(nT"n(tj_a) + Wj_I)<~Dj/Dj_ 1 for j = 2, ..., i, where Wj de-
notes the number of censored observations at tj for j = 1, ..., n,, then
e * * ( x > t,) ~ P ( x > t,_ ,) for i= 1. . . . , n 1.
Consequently, if the condition in (ii) above is met (as it is when there are no ties
among the uncensored observations), both the P E G E and the PEXE interlace
with the KME: in each interval of the form (tt_ ,, t~], the P E G E and the PEXE
cross the K M E once from above. Practical experience suggests that the condition
in (ii) above is not a stringent one: even though this condition is violated in many
of the data sets considered to date, the P E G E and the PEXE still interlace with
the K M E in the manner described. Another indication from practical experience
is that the difference between the PEXE and the P E G E is negligible, even in small
samples.
Finally, we present an example using the data of Freireich et al. (1963). The
272 G. M. Mimmack and F. Proschan
t ×
4-×
i"
I /
x4 ,..a,,
1/
I/
x+
I/
+,
LJA ,/I
+x
k~
Jf
× + x ,+
I i I, /
×Jr x+
,t'
i..,
x -.I-
I,/
×+
0
I" CD
×/+/~
z
I,/ _o
×+ co
r'c
,Y o '-';
..~+× -~o t.u
1 I
X I I.--
,+ -LD
I
I
x -J-
I /
× 4- co
1 /
C~
I /
x~-,
0
?o
Piecewise geometric estimation of a survival function 273
data are the remission times of 21 leukemia patients who have received 6 MP (a
mercaptopurine used in the treatment of leukemia). The ordered remission times
in weeks are: 6, 6, 6, 6 + , 7, 9 + , 10, 10+, 11+, 13, 16, 17+, 19+, 2 0 + , 22,
23, 2 5 + , 32+, 3 2 + , 3 4 + , 3 5 + . The P E G E and the K M E are presented in
Figure 1. (Since the P E G E and the PEXE differ by at most 0.09, only the
PEGE appears.) The graphs illustrate the smoothness of the P E G E in contrast
with the jagged outline of the KME. The K M E and the PEGE interlace even
though the condition in Theorems 4. l(ii) and 4.2(ii) is violated. Since the PEGE
is only slightly above the K M E at the observed failure times and the PEGE
crosses the K M E early in each failure interval, the K M E is considerably larger
than the P E G E by the end of each interval. This behaviour is typical. We infer
that the PEGE certainly does not overestimate: it may even tend to under-
estimate.
We conclude that the PEGE (and the modified PEXE) have significant
advantages over the KME, particularly in the cases of large samples containing
many ties and small samples. It is only in the case of a large sample spread over
a large range that the slight increase in computational effort required for the
PEGE might merit using the K M E because the P E G E and the K M E are likely
to be very similar.
In this section we give some indications of the small sample properties of the
PEGE by considering three simulation studies. In the first study, Kitchin (1980)
compares the small sample properties of the PEXE with those of the KME. In
the second study, Whittemore and Keller (1983) consider the small sample
behaviour of a number of estimators: we extract the results for the K M E and a
particular version of the PEXE. In the third study, we make a preliminary
comparison of the K M E and the PEGE. We expect the behaviour of the piece-
wise exponential estimators to resemble that of the PEGE because piecewise
exponential estimators are continuous versions of the PEGE and, moreover,
piecewise exponential estimators and the PEGE are similar when the underlying
life distribution is continuous.
The pi_ecewise exponential estimator considered by Whittemore and Keller is
denoted FQ4" It is constructed by averaging the PEXE failure rate function estima-
tor with a variant of the PEXE failure rate function estimator--that is, ~Q4 is the
same as the PEXE except that the PEXE failure rate estimators 2/- . . . . , 2 ~ are
replaced by the failure rate estimators 2", ..., 2*, defined as follows:
2* = 5(2;
~ - + 2t+- l ) f o r / = 1, .. ., n l ,
where
2;- = D;/total time on test in (t;_ 2 , ti] for i = 1, . . . , n 1 ,
A_lthough Whittemore and Keller include in their study the two estimators
FQ, and FQ2 constructed from 2 f . . . . . 2~-, and 2~- . . . . . 2,~] respectively, they
present the results for the hybrid estimator FQ, alone because they find that FQI
tends to be negatively biased and ffQ: tends to be positively biased.
The same model is assumed in all three studies. The model is that of random
censorship: corresponding life and censoring random variables are independent
and the censoring random variables are identically distributed. Whittemore and
Keller generate 200 samples in each of the 6 x 3 x 4 = 72 situations that result
from considering six life distributions (representing failure rate functions that are
constant, linearly increasing, exponentially increasing, decreasing, U-shaped, and
discontinuous), three levels of censoring (P(Y<X)~ O, 0.55, 0.76), and four
sample sizes (n = 10, 25, 50, 100). Kitchin obtains 1000 samples in each of a
variety of situations: he considers four life distributions (Exponential, Weibull with
parameter 2, Weibull with parameter ½ and Uniform), three levels of censoring
(P(Y<X) = 0, 0.5, 0.67), and four sample sizes (n = 10, 20, 50). Kitchin's study
is broader than that of Whittemore and Keller in that Kitchin considers Exponen-
tial, Weibull and Uniform censoring distributions while Whittemore and Keller
consider only Exponential censoring distributions. Kitchin apparently produces
the greater variety of sampling conditions because his results vary slightly accord-
ing to the model, while Whittemore and Keller find so much similarity in the
results from the various distributions that they record only the results from the
Weibull distribution.
The conclusions we draw from the two studies are similar. Regarding mean
squared error (MSE), both Kitchin and Whittemore and Keller find that, in
general:
(i) The MSE of the exponential estimator is smaller than that of the KME.
(ii) As the level of censoring increases, the increase in the MSE is smaller for
the exponential estimator than for the KME.
Kitchin reports than (i) and (ii) are not always true of the PEXE and the KME:
the exceptional cases occur in the tails of the distributions.
The conclusions about bias are not so straightforward. Whittemore and Keller
find that the PEXE tends to be negatively biased while Kitchin reports that the
bias of the PEXE is a monotone increasing function of time: examining his
figures, we find that the bias tends to be near zero at some point between the 40th
and 60th percentiles except when the life and censoring distributions are Uniform.
(In this case, the bias is positive only after the 90th percentile.) We conclude that
Whittemore and Keller merely avoid detailed discussion of bias. Regarding the
hybrid estimator, we find in the figures recorded some suggestions of the tenden-
cies observed in the PEXE--specifically, monotone increasing bias and a tendency
for underestimation when the sample size is small and censoring is heavy.
Whether this behaviour is typical of the PEGE also remains to be seen.
Piecewise geometric estimation of a survival function 275
In considering the magnitude of the bias of the estimators, we find the fol-
lowing.
(i) Both Kitchin and Whittemore and Keller report that the bias of the KME
is negligible except in the right tail of the distribution and in the case of a very
small sample (n = 10) and heavy censoring.
(ii) The PEXE i_s considerably more biased than the KME.
(iii) The bias of FQ4 is negligible except in the case of a very small sample and
heavy censoring.
(iv) The bias of each estimator increases as the censoring becomes heavier and
it decreases as the sample size increases.
In view of these two studies, we conclude, firstly, that the PEGE is likely to
compare favourably with the K M E in terms of MSE, and secondly, that the
PEGE is likely to be considerably more biased than the KME. We expect that
the discrete counterpart of FQ4 performs well in terms of both MSE and bias.
Since the bias of this estimator is likely to be small, adjustment for its presumed
tendency to increase monotonically is deemed an unnecessary complication.
In the pilot study we generate three collections of data, each consisting of 100
samples of size 10, from independent Geometric life and censoring distributions.
In each case the life distribution has parameter p = exp(-0.1). The censoring
distributions are chosen so as to produce three levels of censoring: setting
p = e x p ( - 2 ) , where 2=0.00001, 0.1, 0.3, yields the censoring probabilities
P(Y<X) = 0, 0.475, 0.711 respectively.
The conventions followed for extrapolation in the range beyond the largest
observed failure time are as follows:
ff(X>k)={Po(X>t., ) fort.,<~k<s~:,
for k~> s~2 ~> tm ,
fi(X>k)=fi(X>tnl)(1-O~,) k-t"' for k ~> t~,.
This definition of the K M E rests on the assumption that the largest observation
is uncensored, while the definition of the PEGE results from assuming that the
failure rate after the largest observed failure time is the same as the failure rate
in the interval (tn,_ l, t,l ]-
Our conventions for extrapolation differ from those of Kitchin and of Whitte-
more and Keller. Consequently our results involving fight-hand tail probabilities
differ from theirs: a preliminary indication is that our extrapolation procedures
result in estimators that are more realistic than theirs.
Although the size of the study precludes reaching more than tentative con-
clusions, we observe several tendencies.
Tables l(a), 2(a) and 3(a) contain the estimated bias and mean squared error
(MSE) for the K M E and the P E G E of P(X > k) for k = ~p, where ~p is the pth
percentile of the underlying life distribution and p = 1, 5, i0, 20, 30, 40, 50, 60,
70, 80, 90, 95, 99. From these tables we make the following observations.
(i) The MSE of the P E G E is generally smaller than that of the KME. The
276 G. M. Mimmack and F. Proschan
Table 1
Results of pilot study using 100 samples of size 10, Geometric (p = e x p ( - 0 . 1 ) ) life distribution,
Geometric (p = e x p ( - 0.00001)) censoring distribution and P ( Y < X ) ~ - 0
Table 2
Results of pilot study using 100 samples of size 10, Geometric (p = exp(-0.1)) life distribution,
Geometric (p = e x p ( - 0.1)) censoring distribution and P ( Y < X) ~ 0.475
(vi) T h e m a g n i t u d e o f t h e b i a s o f t h e K M E is c o n s i s t e n t l y s m a l l e r t h a n t h a t
o f t h e P E G E o n l y w h e n t h e r e is n o c e n s o r i n g . U n d e r c o n d i t i o n s o f m o d e r a t e a n d
h e a v y c e n s o r i n g , t h e K M E is less b i a s e d t h a n t h e P E G E o n l y a t p e r c e n t i l e s t o
t h e left o f t h e m e d i a n : t o t h e r i g h t o f t h e m e d i a n , t h e P E G E is c o n s i d e r a b l y less
biased than the KME.
(vii) A s c e n s o r i n g i n c r e a s e s , t h e m a g n i t u d e o f t h e b i a s o f t h e K M E i n c r e a s e s
faster than does that of the PEGE.
Tables l(b), 2(b) and 3(b) contain the estimated bias and MSE for the
Kaplan-Meier (KM) and piecewise geometric (PG) estimators of the percentiles
~p, p = 1, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 95, 99. F r o m t h e s e t a b l e s w e m a k e
the following observations.
278 G. M. Mimmack and F. Proschan
Table 3
Results of pilot study using 100 s a m p l e s of size 10, G e o m e t r i c ( p = e x p ( - 0 . 1 ) ) life distribution,
G e o m e t r i c ( p = exp ( - 0.3)) censoring distribution a n d P ( Y < X ) --- 0.711
(i) With a few exceptions, the PG percentile estimator is less biased than the
KM percentile estimator.
(ii) Both estimators tend to be negatively biased.
(iii) At each level of censoring, the bias of the PG percentile estimator is
negligible for percentiles smaller than the 70th, and it is acceptably small for larger
percentiles, except perhaps the 99th percentile. In contrast, the KM percentile
estimators are almost unbiased only for percentiles smaller than the 60th: to the
right of the 60th percentile the bias tends to be very much larger than that of the
PG estimators. This tendency is particularly noticeable in the case of heavy
censoring.
(iv) The MSE of the PG percentile estimator is smaller than that of the KM
percentile estimator only in certain ranges, viz.: p ~< 70 for heavy censoring,
Piecewise geometric estimation of a survival function 279
p ~< 40 for moderate censoring, and 5 ~<p ~< 95 for no censoring. Since the PG
percentile estimator is almost unbiased outside these ranges, the large MSE must
be the result of having large variance.
On the basis of the observations involving the survival function estimators, we
conclude that the small sample behaviour of the P E G E resembles that of the
PEXE: specifically, when there is little or no censoring, the PEGE compares
favourably with the K M E in terms of MSE but not in terms of bias. We expect
that this is true irrespective of the level of censoring when the sample size is
larger. It remains to be seen whether inversion of this general behaviour is typical
when the sample size is very small and censoring is heavy. It is evident that
increased censoring affects the bias and the MSE of the PEGE less than it affects
the bias and the MSE of the KME.
Our conclusions about the percentile estimators are even more tentative
because of the lack of results involving the behaviour of percentile estimators. The
fact that the PG percentile estimator is almost unbiased even in the presence of
heavy censoring, and even as far to the right as the 95th percentile, is of con-
siderable interest because the KM extrapolation procedures are clearly inadequate
for estimating extreme right percentiles.
Regarding the MSE, we note that, under conditions of moderate or heavy
larcensoring, any estimator of the larger percentiles is expected to vary considerably
because there are likely to be very few observations in this range. The ad hoc
extrapolation procedure for the KM is expected to cause the estimators of the
extreme right percentiles to exhibit large negative bias and little variation. In view
of these considerations and the accuracy of the PG percentile estimators, we
conclude that the fact the MSE of the PG percentile estimator of the larger
percentiles is greater than that of the KM percentile estimator is not evidence of
a breakdown in the reliability and efficiency of the PG percentile estimator.
The general indications of our pilot study are that the PEGE and the discrete
version of ffQ4
are attractive alternatives to the KME. In view of the resemblan__ce
between the properties of the P E G E and those of the PEXE, the results for PQ4
portend well for the new discrete estimator: we expect it to be almost unbiased
and to be not only more efficient than the K M E but also more stable under
increased censoring. Moreover, we expect the corresponding percentile estimator
to have these desirable properties also because it is likely to behave at least as
well as the PG percentile estimator.
The properties involving relative efficiency are of considerable importance
because relative efficiency is a measure of the relative quantities of information
utilized by the estimators being compared. This interpretation of relative effi-
ciency, and the fa__ct that heavy censoring is often encountered in engineering
problems, makes FQ4 and its discrete counterpart even more attractive.
References
Aalen, O. (1976). Nonparametric inference in connection with multiple decrement models. Scan-
dinavian J. Statist. 3, 15-27.
280 G. M. Mimmack and F. Proschan
L. F. Pau
1. Introduction
FAILURE-FREE SYSTEM E°
(SAFE and AVAILABLE STATE)
Fig. 1. Relation between failure diagnosis, reliability or degradation processes, safety and mainten-
ance. If the repair is instantaneous (/% = + oo), if there is no detection delay (tin + td = 0), and if the
diagnostic system itself never fails, the asymptotic system availability in the stationary case is:
i=N--I
A = Prob(UUT not failed)t_ ÷ oo = lqi= 1 ei/(2i + ee). More general formulae may be derwed, espe-
cially for finite repair times, and more general degradation processes.
281
282 L. F. Pau
<
D
a~
D [--,
0
m
r~
o
¥
H rn N
r~
II Z
[-~ .r'n
Z H
O
N r~
H
> _ _
0 0
~ .,e-- O
Z
m o r~ E~ m .N
0
Z
Z ~
m o
I ~Z
©
H
b
f
._1_ ~
g
z
o
z
H
o
g
H
o
e,i
Applications of pattern recognition in failure diagnosis and quality control 283
- production yield;
- quality;
- failure tolerance;
- system activation delays;
- costs (lifetime, operation);
- maintenance;
- warranties.
We define here technical diagnostics as the field dealing with all methods,
processes, devices and systems whereby one can detect, localize, analyse and
monitor failure modes of a system, i.e., defects and degradations (see Section 2).
It is at this stage essential to stress that, whereas system reliability and safety
theories are concerned with a priori assessments of the probability that the system
will perform a required task under specified conditions, without failure, for a
specified period of time, the field of failure diagnosis is essentially focusing on
a posteriori and on-line processing and acquisition of all monitoring information
for later decision making.
Failure diagnosis has itself evolved from the utilization of stand-alone tools (e.g.
calibers), to heuristic procedures, later codified into maintenance manuals. At a
later stage, automatic test systems and non-destructive testing instruments, based
on specific test sequences and sensors, have assisted the diagnosis; examples are:
rotating machine vibration monitoring, signature analysis, optical flaw detection,
ultrasonics, ferrography, wear sensors, process parameters, thermography, etc.
More recently, however, there has been implementations and research on evolved
diagnostic processes, with heavier emphasis on sensor integration, signal/image
processing, software and communications. Finally, research is carried out on
automated failure diagnosis, and on expert systems to accumulate and structure
failure symptoms and diagnostic strategies (e.g. avionics, aircraft engines, soft-
ware).
Although the number of application areas and the complexicity of diagnostic
systems have increased, there is still a heavy reliance on 'ad-hoc' or heuristic
approaches to basing decisions on diagnostic information. But a number of funda-
mental diagnostic strategies have emerged from these approaches, which can be
found to be common to these very diversified applications.
After having introduced in Section 2 a number of basic concepts in technical
diagnostics, we will review in Section 3 some of the measurement problems. The
basic diagnostic strategies will be summarized in Section 4. Areas for future
research and progress will be proposed in Section 5.
2. B a s i c c o n c e p t s in t e c h n i c a l d i a g n o s t i c s
Although they may have parts in common, we will essentially make the
difference between the system for which a diagnosis is sought (system/unit/process
under test: UUT), and the diagnostic system. The basic events in technical
diagnostics are well defined in terminology standards; they are: failure, defect,
degradation, condition.
284 L. F. Pau
( CONFUSION MATRIX )
t
A C T U A L F A I L U R E M O D E E.
1
i=0,1,..,N-1
FAILURE
FAILURE DETECTION - /
I
I FAILURE DIAGNOSIS
Fig. 3.
2.1.3. Failure diagnosis: The act or process of identifying a failure mode E upon
an evaluation of its signs and symptoms, including monitoring information. The
diagnostic process carries therefore out a breakdown of failure detection into
individual failure modes.
Applications of pattern recognition in failure diagnosis and quality control 285
2.1.4. Failure analysis: The process of retrieving via adequate sensors all possible
information, measurements, and non-destructive observations, alltogether called
diagnostic information, about the life of the system prior and up to the failure;
it is also a method whereby to correlate these informations.
2.1.5. Failure monitoring: This is the act of observing indicative change of equip-
ment condition or functional measurements, as warnings for possible needed
corrections.
2.2.3. Probability of false alarm: The probability of diagnosing that a failure mode
is present, when in fact none is present (except the normal condition Eo).
2.2.5. Failure coverage: This is the conditional probability that, given there exists
a failure mode of the UUT, this system is able to recover automatically and
continue operations. The process of automatic reconfiguration, and redundancy
management has the purpose of improving the coverage and making the system
fault-tolerant.
2.2.6. Measurement time tin: This is the total time required for acquiring all
diagnostic iinformation (except a priori information) required for the failure
detection, localization and diagnosis, This time may be fractioned into sub-
sequences, and estimated in expected value.
2.2.7. Detection (of diagnosis) delay td: This is the total time required to process
and analyze the diagnostic information, and also to display or transmit the failure
286 L. F. Pau
2.2.8. Forecasting capability tf: This is the lead time with which the occurrence of
a specific failure mode (E o excepted) can be forecasted, given a confidence
interval or margin.
2.2.9. Risks and costs: Costs/risks are attached to each diagnosed failure mode/~,
obtained as a result of the decision process; they include: testing costs,
maintenance/repair costs, safety risks, lost usage costs, warranties, yields.
DEGRADATION PROCESSES
~ DESIGN ~" .~ __
I\
T
OPERATIONS ~ HUMAN FACTORS PROCEDURES
area into which much sensor development research is going, and refer the reader
to the References for other fields.
3.3.2. Feature extraction: The features are then those combined symptoms derived
jointly from d~erent sensors, these measurements being combined together by an
288 L. F. Pau
Fig. 5. Sensors and measurement processes for the diagnosis of integrated circuits.
3.4.3. Effect of control elements and protective elements: The observability is further
reduced for some parts of the UUT because of:
- physical protection: hybrid/composite structures, coatings, multilayers, casings;
- p r o t e c t i o n and failure recovery systems: protection networks, fault-tolerant
parts, active spares, sating modes;
- control elements: feedback controllers, limiters, and measurement effects due to
the detection delay td.
3.4.5. Support structure: The support structure, casing or board may, by its
properties or behavior, interfere with both the sensor and UUT, e.g. because of
mechanical impedance, electromagnetic interference (EMI), etc.
- Sensor/measurement type
- Location
Diversity by: - Design
-Environment
-Data acquisition (bandwidth, gain, wavelength, data rate)
- Software
Feature
Sensor measurement type 2: extraction
Images, electromagnetic waves of diagnostic
information g
>
3.4.7. Sensor reliability: Failure analysis and diagnosis are only possible if the
sensors of all kinds survive to system failures; this m a y require sensor redundancy
(physical or analytical), separate power supplies, and different technologies and
computing resources. Besides sensor and processor reliability, short reaction time
t m and good feature extraction are conflicting hardware requirements, all of which
contribute to increased costs which in turn limit the extent of possible implemen-
tations. Any diagnostic subsystem, and any U U T subsystem which can be
activated separately, should be equipped with a time meter, unit or cycle counter.
r~
•. ~
D
© H ~
I-.-I 0
121 0
M r~
m N 0
r~
tr~
\
0
r~
,7.
i--t
o ~
r~
r~ H r~ d
0 W
0 t~
Z
o m rJ~
H o
o~
N
<
0
0 o
t~
Z
0 E~ r~
eZ
D Z
H
Z H
0 Z
292 L. F. Pau
4. Diagnostic processes
4.1.2.1. Failure mode removal by analysis and inspection: The detection, diagnosis,
localization and removal o f the failure mode which has occurred, are carried out
in sequence; the removal affects, a m o n g others: requirements, design, control,
usage, parts, repair, programs, etc.
4.1.2.3. Exploring the operational envelope: The external specifications define the
operational envelope within which the U U T must perform correctly in mode E o.
These performance limits, while representative o f the realworld process, are not
necessarily accurate, and quite different system states m a y occur. These strategies
S therefore explore the behavior under circumstances not given as performance
requirements, including 'severe' operating environments.
Fault seeding
-
- Code analyzers
- Dynamic test probes, injection of test patterns of bits
2. Validation: - Proof-of-correctness
- Program verification by predicate testing
- Proof-of-loops
- Validation using a representation in a specification language
- Validation by simulation
)
)
ca<
C> 1 '7
z
)
o 8
) u3
r~
u ~
Z
o ~
~ z N
~q~q < Z
O~ ~-m < ._~
pq •
H~Z~
0 ~ U 0
OqJ
g] ~[4
~q~
P40
o ~
z ~
H~
r~
Z ~
O ~
~ 8
~
~3 N
Applications of pattern recognition in failure diagnosis and quality control 295
4.2.2. Failure model unit: For a given UUT configuration D, operational environ-
ment, and set of other learning information/, this unit identifies and prioritizes
the possible failure modes Eo, E 1. . . . , E N - 1 (e.g. critical parts, active routines,
fracture locations). A failure mode effect model (FMEA analysis) is then adjusted
to a usage model of the UUT (incorporating e.g. fatigue, ductility, heating, cumu-
lative failures, cumulative contents of registers) to derive predicted parameter values
for all possible failure modes Eo, E l , E N_ 1, and the potential effects on the
...,
UUT performances.
EXAMPLE. S n e a k circuit analysis (failure mode identification). This is, for elec-
tronic circuits, a systematic review of electric current and logic paths down to the
components and logic statements, to detect latent paths, timing errors, software
errors, hardware failures. It uses essentially the specifications and nodal/topologi-
cal network analysis, in addition to state diagrams for the logic.
4.2.3. Diagnostic decision unit (Figure 11). This decision logic determines the likely
failure mode /~ among Eo, El, ..., EN_I, from the estimated and predicted
parameters, with account for the cost/risk/time factors. This process, which may
also derive classification features from these data, is essentially a pattern
recognition process (signals, images, coded data, text, symbols, logic invariants);
the simplest case is straightforward comparison (template matching) between
estimated and predicted parameters (including event counts).
When the diagnostic decosion is used for the prediction of the remaining U U T
life, and passive sensors only are used, one would use the term non-destructive
evaluation (NDE) instead of technical diagnostics.
Extensions to be above are required within the context of knowledge based
systems or expert systems for diagnostics (Pan, 1986).
H1
r
rJ3 1
m i
H i
ul i
e~
o
"3
@
e~
d=
[ .<
o
rfl
~z
r~
r~
>
i...t
[.-t
U
i
298 L. F. Pau
Fig. 12. Features for data communications network tests and monitoring.
Failure detection
- Sequential hypothesis testing (Wald, 1947).
- Non-parametric sequential testing (Pau, 1978; Fu, 1968; Wald, 1947).
- Hypothesis testing (shift of the mean, variance) (Clark et al., 1975; Sebestyen,
1962).
- Bayes classification (Fukunaga, 1972).
- Discriminant analysis (Fukunaga, 1972; Sebestyen, 1962).
- Nearest neighbor classification rule (Fukunaga, 1972; Devijver, 1979).
- Sensor/observation error compensation (Pau and Kittler, 1980).
Failure localization
- Graph search algorithms (Saeks and Liberty, 1977; Rasmussen and Rouse,
1981; Slagle and Lee, 1971).
- Branch-and-bound algorithms (Navendra and Fukunaga, 1977).
Dynamic programming (Pau, 1981a; Bellman, 1966).
-
Failure diagnosis
- Correspondence analysis (Pau, 1981a; Hill, 1974; Section 5).
- Discriminant analysis (Van de Geer, 1971; Benzecri, 1977).
Applications of pattern recognition in failure diagnosis and quality control 299
Failure analysis
- Variance analysis, correlation analysis (Van de Geer, 1971).
- Principal components analysis (Pau, 1981a; Van de Geer, 1971; Chien and Fu,
1967).
- Scatter analysis (Van de Geer, 1971; Everitt, 1974).
- Clustering procedures, e.g. dynamic clusters algorithm (Pau, 1981a; Everitt,
1974; Hartigan, 1975).
- Multivariate probability density estimation (Parzen, kernel functions, k-nearest
neighbour estimators) (Fukunaga, 1972; Devijver, 1979; Parzen, 1962).
- Multivariate sampling plans (Pan et al., 1983).
Failure monitoring
- Statistics of level crossings, especially two-level crossings (Saeks and Liberty,
1977; Pau, 1981a).
- Spectral analysis and FFT (Chen, 1982).
- Kalman estimation (Pau, 1981a, 1977).
- Recursive least-squares estimators.
- Linear prediction ARMA, ARIMA estimators (Chen, 1982).
- Knowledge based or expert systems for failure monitoring (Pau, 1986).
5.1. Method
5.1.1. Introduction and problem analys&
(a) The case is set up as a clustering problem, where each of the 33 machines
considered is described by measurement attributes (vibration level, operating time,
electricity consumption, flow). The raw data are given in Figure 13. Some essential
characteristics of this problem are the following:
(i) the answer requested is to reduce the number of alternatives for the
diagnosis and failure location;
(ii) it is obvious, for technical reasons, that the four attributes are correlated;
(iii) the number of attributes measured on each machine is fairly small, and all
observations are real valued and non-negative.
300 L.F. Pau
However, the parameters of these relations are u n k n o w n and they can only be
inferred from the sample of 33 machines.
(b) These characteristics build justifications for the use of multivariate statisti-
cal analysis, a n d of correspondence analysis in particular because of its joint use
of information about the machines and about the diagnostic measurements. The
main steps of correspondence analysis are the following (Pan, 1981a; Chen,
1982):
Step 1. First, infer from the data estimated correlations between machines and
between diagnostic measurements, a reduced set of i n d e p e n d e n t feature measure-
ments, according to which the 33 alternative machines may be ranked. As far as
this step is concerned, and this step only, correspondence analysis is comparable
Applications of pattern recognition in failure diagnosis and quality control 301
to factor analysis (Van de Geer, 1971; Hartman, 1960), although the two differ
in the remaining steps.
Step 2. Next, interpret the nature of these statistically independent feature
measurements, by indicating the contribution to each of these by the original
attribute measurements, and determine the diagnosis in terms of these features.
Step 3. Thereafter, rank the feature measurements by decreasing contributions
to the reconstruction of the original 33 x 4 evaluation measurements; the best
feature measurement (e.g. the first) is, in correspondence analysis, the one
maximizing the variance in that direction; in other words, this is the feature
measurement which produces the ranking with the highest possible discrimination
among the 33 machines, thus reducing the doubt of the repairman.
Step 4. Finally, recommend to the failure location those machines which get
the most favorable ranking (in terms of the interpretation) on the first feature axis,
eventually also on the second axis.
(c) One essential advantage of this approach is that the decision maker, will
be provided with a two-dimensional chart, which he may easily interpret, and on
which he may spot with the eye in a straightforward manner, the final reduced
set of candidate machines. Also, apart from the number of feature measurements
used in step 4, no additional assumption is needed, because unsupervised multi-
variate statistical analysis is used. The effect of linear transformation and rescaling
of the initial data is indicated in Section 5.1.2.6.
5.1.2. Theory and use of correspondence analysis (Chen, 1982; Hill, 1974; Pau,
1981a).
5.1.2.1. Notation. Let k(I, J) be the incidence table of non-negative numbers,
representing the attribute measurements j t 3", j = 1, 2, 3, 4, on the machines i t / ,
i = 1, ..., 33. The marginals are defined as follows:
It is convenient to operate on the contingency table p(I, J), rather than on the
incidence table k(1, J):
will be removed. Equivalent machines will thus appear immediately as having very
close representations on the maps. The machine space I is provided with a
distance measure, called Z2 metric, defined by
x(i, j) a= _ p(',j) 1.
p(;, .)p(., j)
Moreover, each machine i~ I and each measurement j e J are assigned the
weights p(i, .), and p ( . , j), respectively, for all variance computations using the
Z2 metric.
5.1.2.3. Theory of correspondence analysis: summary (Pau, 1981a; Chen, 1982; Hill,
1974).
(a) Correspondence analysis, or as it is also called, Fisher's canonical analysis
of contingency tables, amounts to looking for vectors
where Card(. ) is the number of elements in the set, such that when the functions
f, g of the random variables (Y, X) = ( j , / ) are defined by the relations
then the correlation between the random variables f ( Y ) , g(X) is maximum. Corre-
spondence analysis can be applied to non-negative incidence tables k(L J), as
well as to contingency tables p(I, J); the former will be considered in the
following.
(b) Let k(L ' ) and k ( ' , J ) be the diagonal matrices of row and column totals,
assuming none to be zero. The sequence of operations
in which new vectors F ('m, G (m) are successively derived from an initial vector
G (1), is referred to here as the Co(k((L J)) algorithm corresponding to the tableau
I,(i, J).
(c) Its eigenvectors, as defined below, are the solutions of the correspondence
analysis problem, and the coordinates of the individuals and measurements in the
feature space are simply:
F* = ( 1 / x / ~ ) ( p ( . , J ) ) - ' tp(I, J ) G * ,
1 1
(d) G(i, n) - ~ p(i, j ) F ( j , n) . .i e .I , . n .= 1,
. r
p(i, • ) jT"J
304 L. F. Pau
5.1.2.5. Contributions, and interpretations of the factor axes representing the feature
measurements. On a map, the squared Euclidean distance D between rows and/or
columns, has the same value as the Z2 distance between the corresponding
profiles, and
5.I.2.6. Lffect of rescaling the data k(L J). If the attribute measurement k(i, j) is
rescaled by a factor aj > 0, and if the modified x coordinates are noted xa, then
1. C o o r d i n a t e s F M 1 2 3
o f the m e a s u r e m e n t s
F(PRIC) - 0.03785 - 0.00886 0.00010
F(CONS) 0.04187 0.02053 - 0.08758
F(WATR) 0.05526 0.06180 0.00058
F(TIME) 0.17734 0.05025 0.00032
2. C o o r d i n a t e s G M 1 2 3
o f the m a c h i n e s
G(L26) - 0.11505 0.02421 - 0.00150
G(L13) - 0.10726 0.00762 0.00304
G(L18) -0.09264 0.00778 - 0.00159
G (L15) - 0.08407 - 0.03511 - 0.00472
G(L19) - 0.07633 - 0.00993 0.00033
G (L 5) - 0.06924 0.05048 - 0.00186
G(L 4) - 0.06350 - 0.03973 0.00142
G(L33) - 0.05833 0.03013 0.00587
G(Lll) - 0.05656 0.04041 - 0.00033
G(L14) - 0.05310 -0.01005 0.00656
G(L 8) - 0.04395 0.00195 -0.00599
G (L22) - 0.03896 - 0.03516 0.00438
G (L31) -0.03345 - 0.01782 - 0.01013
G (L20) - 0.00388 0.00380 0.00537
G(L24) - 0.00200 - 0.01753 0.00005
G(L 1) 0.00459 - 0.05232 0.00285
G(L10) 0.01442 - 0.04054 - 0.00104
G(L 7) 0.01917 0.02893 0.00087
G(L25) 0.02446 0.03182 - 0.00243
G (L16) 0.03331 0.01714 0.00617
G (L12) 0.03458 - 0.05806 0.00130
G(L28) 0.03717 -0.01573 - 0.00833
G (L 9) 0.04593 0.02954 0.00064
G(L29) 0.04765 -0.02400 0.00103
G(L17) 0.05156 0.02187 - 0.00978
G(L 6) 0.05731 0.04714 0.00146
G(L21) 0.06003 - 0.04308 0.00087
G(L 3) 0.07663 0.03911 0.00179
G(L27) 0.08129 0.03521 - 0.00539
G(L 2) 0.10110 - 0.04922 -0.00005
G(L23) 0.11189 0.02595 0.00689
G (L30) 0.11780 - 0.00506 -0.00246
G (L32) 0.12948 0.00688 0.00054
3. E i g e n v a l u e s r 1 2 3
and inertia
0.4629 E - 0 2 0.9931E -03 0.1817 E - 0 4
82.07~ 17.61~o 0.32~o
Z 82 ~o 99.68~0 100 00~o
5.2.3. Step 3: Ranking the feature measurements. The numlerical results from
Figure 14 yields:
).1 eigenvalue of G* = 0.4629 E - 02 or z I -- 82.07~o ,
22 eigenvalue of G* = 0.993 E - 0 3 or z2 --- 1 7 . 6 1 ~ ,
23 eigenvalue of G* = 0.181 E - 0 4 or z3 = 0.32~o .
Here, it is obvious that the machine diagnosis would essentially rely on the first
feature measurement (vibration level per unit of operating time) and eventually
somehow on the second (flow). Our three-criteria problem has been reduced to
a two-criteria problem with G* as a leading diagnostic criteria to be maximized.
l l l l l l l i l l I [ 1 1 1 [ l l i ; l l l l l l l l i l l l i IIII
I l l i l l l i l l l f l l i
000000000000000000000000000000000 0000
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZ
.=_
l l i l l i l l l l l l l l l l l l l l i l l i l i t i l l i l l III~ rn
I I I I I I l l l l l l l l l II
000000000000000000000000000000000 0000
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZ
0
[.)
I I I I I I I t l l l i l l l l l l ~ l i f l l l l l l l l ~ l l l i l l
l i l i l l l i l l l l l l l I
000000000000000000000000000000000 0000
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZ
~ ~ - ~ - - ~ - - ~ . . . . . . . . . ~ ~o~
308 L. F. Pau
. . . . . . . . . . . . . . . . . . . . . . WATR . . . . . . .
L5
L6
LII
L3
L27
L25
L33 L9
L7
L23
IL26
L17
CONS
L16
L2O
L8
(+)
L30
PRIC
LI9 LI4
L28
L24
L31
L29
Ll5 L22
L4
LIO
L21
L2
TI~E
(b) Because we want the criterion (a) to dominate, we will have to make an
ordering within these non-dominated solutions. Figure 15, which contains the
contributions of the machines to G*, Figure 14 which contains their coordinates,
and last but not least the map of Figure 16, give us, according to the rule (~), the
solution:
Diagnose as defective machine # 3 2 ; if not: # 3 0 ; if not: # 2 3 ; if not: # 2 ; if
not: # 2 7 ; if not: # 3 ; etc.
However, the first machine in this sequence also to have a large positive
contribution to G* (flow) according to criterion (fl), is Machine 27, and the next
Machine 3, or Machine 6. Machines 30 and 2 have negative contributions to G*,
and should be eliminated.
(c) By visual clustering, one could select right away the machines by the original
criteria of minimizing the vibration level, the operating time, the electricity con-
sumption, or the flow p e r s e , by looking at the factor map Figure 16, for which
machines are close to the points representing these criteria/measurements:
(i) Max vibration level: Machines 14, 19, 31, 24, 8, 20, 13, 18, close to PRIC.
(ii) Min operating time: Machines 2, 21, 1, 10, close to TIME.
(iii) Min electricity consumption: Machines 17, 16, close to CONS.
(iv) Min flow: Machines 6, 3, 27, 25, 11, close to WATR.
Notice the large differences between the previous selections (a), (b) according
to criteria (~) and (fl), and the latter ones (c).
5.2.5. Conclusion. Because of the significant contributions of G~* and G*, and
because of the removal of correlated effects, we recommend the following reduced
diagnosis of defect machines:
Machines 32, 23, 27, 3 (in that order, the first being the most likely to have
failed).
References
Beliman, R. (1966). Dynamic programming, pattern recognition and location of faults in complex
systems. J. AppL Probab. 3, 268-280.
Benzecri, J. P. (1977). L'Analyse des Donn~es, Vol. 1 & 2. Dunod, Paris.
Chen, C. H. (1982). Digital Waveform Processing and Recognition. CRC Press, Boca Raton, FL.
Chien, Y, T. and Fu, K. S. (1967). On the generalized Karhunen-Lorve expansion. IEEE Trans.
Inform. Theory 13, 518-520.
Clark, R. N. et al. (1975). Detecting instrument malfunctions in control systems. IEEE Trans.
Aerospace Electron. Systems 11 (4).
310 L. F. Pau
Collacott, R. A. (1976). Mechanical Fault Diagnosis and Condition Monitoring. Chapman & Hall,
London.
Devijver, P. A. (1979). New error bounds with the nearest neighbor rule. IEEE Trans. Inform. Theory
25, 749-753.
Everitt, B. (1974). Cluster Analysis. Wiley, New York.
Fu, K. S. (1968). Sequential Methods in Pattern Recognition and Machine Learning. Academic Press,
New York.
Fukunaga, K. (1972). Introduction to Statistical Pattern Recognition. Academic Press, New York.
Hartigan, J. A. (1975). Clustering Algorithms. Wiley, New York.
Hartman, H. (1960). Modern Factor Analysis. University of Chicago Press, Chicago, IL.
Hill, M. O. (1974). Correspondence analysis: a neglected multivariate method. AppL Statist. Ser. C
23 (3), 340-354.
IEEU Spectrum (1981). Special issue on reliability, October 1981.
IMEKO (1980). TC-10: Glossary of terms and definitions recommended for use in technical
diagnostics and condition-based maintenance. IMEKO Secretariat, Budapest.
Narendra, P. M. and Fukunaga, K. (1977). A branch and bound algorithm for feature subset
selection. IEEE Trans. Comput. 26, 917-922.
Parzen, E. (1962). On estimation of a probability density function and mode. Ann. Math. Statist. 33,
1065-1076.
Pan, L. F. (1977). An adaptive signal classification procedure: application to aircraft engine
monitoring. Pattern Recognition 9 (3), 121-130.
Pau, L. F. (1978). Classification du signal par tests s6quentiels non-param&riques. In: Proc. Conf.
Reconnaissance des formes et traitement des images. INRIA, Rocquencourt, pp. 159-168.
Pau, L. F. (1981a). Failure Diagnosis and Performance Monitoring. Marcel Dekker, New York.
Pau, L. F. (1981b). Applications of pattern recognition to failure analysis and diagnosis. In: K. S. Fu,
ed, Applications of Pattern Recognition. CRC Press, Boca Raton, FL, Chapter 5.
Pau, L. F. (1984). Failure detection processes by an expert system and hybrid pattern recognition.
Pattern Recognition Lett. 2, 419-425.
Pau, L. F. (1986). A survey of expert systems for failure diagnosis, test generation and maintenance.
Expert Systems J. 3 (2), 100-111.
Pau, L. F. (1987a). Knowledge representation approaches in sensor fusion. In: Proc. IFAC World
Congress. Pergamon Press, Oxford.
Pau, L. F. (1987b). System availability in presence of an imperfect test and monitoring system. IEEE
Trans. Aerospace Electron. Systems, 23(5), 625-633.
Pau, L. F. and Kittler, J. (1980). Automatic inspection by lots in the presence of classification errors.
Pattern Recognition 12 (4), 237-241
Pau, L. F., Toghrai, C. and Chen, C. H. (1983). Multivariate sampling plans in quality control: a
numerical example. IEEE Trans. Reliability 32 (4), 359-365.
Rasmussen, J. and Rouse, W. B. (Editors) (1981). Human Detection and Diagnosis of System Failures.
NATO Conference series, Vol. 15, Series 3. Plenum Press, New York.
Saeks, R. and Liberty, S. (1977). Rational Fault Analysis. Marcel Dekker, New York.
Sebestyen, G. (1962). Decision Making Processes in Pattern Recognition. MacMillan, New York.
Slagle, J. R. and Lee, R. C. T. (1971). Application of game tree searching techniques to sequential
pattern recognition. Comm. ACM 14 (2), 103-110.
Van de Geer, J. P. (1971). Introduction to Multivariate Analysis for the Social Sciences. Freeman, San
Francisco, CA.
Wald, A. (1947). Sequential Analysis. Wiley, New York.
Conferences
IEEE Automatic Testing Conference (AUTOTESTCON).
IEEE International Test Conference (Cherry Hill).
IEEE/AIAA Annual Reliability and Maintainability Conferences.
Applications of pattern recognition in failure diagnosis and quality control 311
W. J. Padgett
I. Introduction
* This work was supported by the U.S. Air Force Office of Scientific Research and Army Research
Office.
313
314 w. J. Padgett
investigation have been similar to those for the complete sample case, except that
the problems are generally more difficult mathematically.
The various types of estimators from right-censored samples that have been
proposed in the literature will be indicated and briefly discussed here. They
include histogram-type estimators, kemel-type estimators, maximum likelihood
estimators, Fourier series estimators, and Bayesian estimators. In addition, since
the hazard rate function estimation problem is closely related to the density
estimation problem, various types of nonparametric hazard rate estimators from
right-censored data will be briefly mentioned. Due to their computational sim-
plicity and other properties, the kernel-type density estimators will be emphasized,
and some examples will be given in Section 7.
Before beginning the discussion of the various estimators, in the next section
the required definitions and notation will be presented.
Let X °, X 2,
° ..-, X~o denote the true survival times of n items or individuals
which are censored on the right by a sequence Ua, U2. . . . . Un which in general
may be either constants or random variables. It is assumed that the )(O's are
nonnegative independent identically distributed random variables with common
unknown distribution function F °. For the problem of density estimation, it is
assumed that F ° is absolutely continuous with density fo. The corresponding
hazard rate function is defined by r ° = f°/(1 - F ° ) .
The observed right-censored data are denoted by the pairs (X~, A~), i = 1, . . . , n,
where
X; = min{X °, Us},
Ae={~ if Xg° ~< U;'
if X ° > U~.
Thus, it is known which observations are times of failure or death and which ones
are censored or loss times. The nature of the censoring mechanism depends on
the Us's:
(i) If U1, . . . , Un are fixed constants, the observations are time-truncated. If
all U,.'s are equal to the same constant, then the case of Type I censoring results.
(ii) If all f i -- X~°~, the rth order statistic of X ° . . . . , X °, then the situation is
that of Type II censoring.
(iii) If UI . . . . . Un constitute a random sample from a distribution H (which is
usually unknown) and are independent of X °, . . . , X°, then (Xi, A;),
i = 1, 2, . . . , n, is called a randomly right-censored sample.
The random censorship model (iii) is attractive because of its mathematical
convenience. Many of the estimators discussed later are based on this model.
Assuming (iii), A1. . . . . A,, are independent Bernoulli random variables and the
distribution function F of each X;, i = 1. . . . . n, is given by 1 - F =
Nonparametric estimation of density and hazard rate functions 315
(1 - F °) (1 - H). Under the Koziol and Green [24] model of random censorship,
which is the proportional hazards assumption of Cox [7], it is assumed that there
is a positive constant fl such that 1 - H = (1 - F°) ~. Then by a result of Chen,
Hollander and Langberg [6], the pairs (X°, Ue), i-- 1, . . . , n, follow the pro-
portional hazards model if and only if (X1 . . . . . An) and (A1. . . . . An) are inde-
pendent. This Koziol-Green model of random censorship arises in several
situations (Efron [11], Cs0rg6 and Horvfith [8], Chen, Hollander and Land-
berg [6]). Note that fl is a censoring coefficient since a = P ( X ° ~< Ue) = (1 + fl)- 1,
which is the probability of an uncensored observation.
Based on the censored sample (X;, Ae), i = 1, . . . , n, a popular estimator of the
survival probability S ° ( t ) = 1 - F ° ( t ) at t >/0 is the product-limit estimator, pro-
posed by Kaplan and Meier [22] as the 'nonparametric maximum likelihood
estimator' of S °. This estimator was shown to be 'self-consistent' by Efron [11].
Let (Z e, A; ), i = 1. . . . , n, denote the ordered X;'s along with their corresponding
Ae's. A value of the censored sample will be denoted by the corresponding lower
case letters (x e, be) or (ze, hi) for the unordered or ordered sample, respectively.
The product-limit estimator of S O is defined by [11]
/sn(t) =
i=l
1
-
i+ 1
i
) a,.'
, t E ( Z k _ l, Z ~ ] , k 2 ..... n,
O, t>Z~.
1 -/Sn(Z2), j = 1,
sj: / P,(zj)-Pn(Zj+l), j:2, ..., n - 1, (2.1)
[ fin(Zn), j = n.
simple to compute, histogram estimators are not smooth and are generally not
suited to sophisticated inference procedures.
Estimation of the density function and hazard rate of survival time based on
randomly right-censored data was apparently first studied by Gehan [29]. The life
table estimate of the survival function was used to estimate the density f o as
follows: The observations (x,, b,), i = 1. . . . . n, were grouped into k fixed intervals
[ q , /2), [t2, t3), . . . , [tg, ~ ) , with the finite widths denoted by h,= t,+ 1 -te,
i = 1, . . . , k - 1. Letting n[ denote the number of individuals alive at time t, L;
be the number of individuals censored (lost or withdrawn from the study) in the
interval [t,, t,+ 1), and d, be the number of individuals dying or failing in the ith
interval (where time to death or failure is recorded from time of entry into the
study), define ~, = de/n , and/~, = 1 -Oe, where n, = n~r - 5 1L r Therefore, qe is an
estimate of the probability of^dying or failing in the ^ith interval, given exposure
risk in the ith interval. Let 1I, = p,_ 1H,_ 1, where //1 - 1. Gehan's estimate of
f o at the midpoint tm~ of the ith interval is then
A A A
AnA
expression for estimating the large sample approximation to the variance of
f(tmi ) was also given in [19].
Using the product-limit estimator P. of F °, F01des, Rejt0, and Winter [16]
defined a histogram estimator of f o on a specified interval [0, T], T > 0. For
integer n > 0, let 0 = t~o") < tc1") < • • • < t~,~} = T be a partition of [0, T] into n
subintervals I~"), where
i~m=~[t~_),,t~m), l~<i<v,,
[.[t(")
t vn--1 , T ] , i = v,
L,(t7 °) -
= , x6I} '°. (3.1)
t7 '> -
A
If x ¢ [0, T], f ( x ) is either undefined or defined arbitrarily. Notice that if none
of the observations are censored,/~, reduces to the empirical distribution function,
and (3.1) becomes the usual histogram estimator with respect to the given parti-
tion. The strong uniform consistency of f on [0, T] was proven by F01des,
Rejt0, and Winter [16] under some conditions on the partition, provided that f o
was continuous on [0, T] and H ( T - ) < 1, where H ( T - ) denotes the limit from
the left of H at T. This last condition is common in obtaining consistency
properties under random right-censorship and insures that uncensored observa-
tions can be obtained from the entire interval of interest.
Nonparametric estimation of density and hazard rate functions 317
Burke and Horv/lth [5] defined general density estimators which included
histogram-type and kernel-type estimators with appropriate choices of the defining
functions. They also obtained asymptotic distribution results for these estimators.
In fact, their results were obtained for the more general situation of the k inde-
pendent competing risks model. When k = 2, this reduces to the random right-
censorship model.
The histogram estimator can be obtained as a special case of the kernel density
estimators. The kernel-type estimators have been perhaps the most popular esti-
mators in practice due to their relative computational simplicity, smoothness, and
other properties. Kernel-type estimators from randomly right-censored data have
been studied only since around 1978, beginning with the work of Blum and
Susarla [3]. The investigation of kernel estimators for right-censored samples has
been attempted along the same lines as for the complete sample case. However,
due to mathematical difficulties introduced by the censoring, some of the
analogous theory to the complete sample case has not yet been obtained.
Blum and Susarla [3] generalized the complete sample results of Rosen-
blatt [45] concerning maximum deviation of density estimates by the kernel
method. To define the Blum-Susarla density estimator, let {hn} be a positive
sequence, called the bandwidth sequence, such that limn~o~ h, = 0, and let
N + (x) denote the number of observed X;'s that are greater than x. Define
where [A] denotes the indicator function of the event A. By a modification of the
product-limit estimator, it can be shown that H* is a good estimate of
H* = 1 - H. For a kernel function K satisfying certain conditions, the
Blum-Susarla density estimator is given by
For example, K can be a bounded density function with support in the interval
[ -A, A ] for some A > 0 and absolutely continuous on [ - A , A ] with derivative
K' which is square integrable on [ -A, A]. By following standard arguments,
and H*(x) can be shown to be good estimators off°(x)H*(x) and H*(x), respec-
tively. This motivates the use of (3.1) as an estimator of f°(x). Blum and Susarla
also obtain limit theorems for the maximum over a finite interval of a normalized
deviation of the density estimator (3.2). These results are useful for goodness-of-fit
tests and tests of hypotheses about the unknown lifetime density fo.
318 w. J. Padgett
It was conjectured by Blum and Susarla [3] that the kernel-type estimator
)~(x)=h£1f~K((x-t)/h~)dF*(t)
last condition insured that observed lifetimes in the entire support of F ° would
be available. It should be noted that if no censoring is present, then
where si is given by (2.1). They considered the mean, variance, and mean squared
error of (3.4) under the K o z i o l - G r e e n model of random Acensorship described in
Section 2. This model allowed the expected value of fn(X) to be evaluated by
using the independence of (X 1. . . . . An) and (A 1. . . . , A,). In particular, if K is a
Borel function such that sup IK(t) l < ~ , ~ _~ooqK(t) l dt < o0, lim,~oo [tK(t) l = 0,
and ~ ~_to K(t) dt = 1, then
where
n-1
a= (l+fl) 1, b= 1 - a, Pn( a) : 1~ [ ( n - i + b ) / ( n - i + 1)],
i=l
~(x)
r.(x) = , x~>O,
1 - F,,(x) + 1/n
320 W. J. Padgett
P(x)= ~ ( n - j + 1)-IAjKh.(x-Zj),
j=l
where K was a symmetric integrable kernel with Kh(y ) --K(y/h). They derived
expressions for E[f(x)] and var[~(x)] and proved under the conditions on K
stated by Watson and Leadbetter [55, 56] that r(x) was asymptotically unbiased
if h, ~ 0 and nh, ~ oo. The conditions assumed here were essentially the same as
those required by McNichols and Padgett [34], except for the proportional
hazards (Koziol-Green) model assumption which gave somewhat different
expressions for the mean and variance. The asymptotic variance was also
obtained, and Hajek's projection method was used to establish asymptotic nor-
mality under conditions on K, F °, H, and h,. Tanner and Wong [51] studied a
class of estimators of the same general form as ¢(x) with K h replaced by K s, were
0 was a positive-valued 'smoothing vector' chosen to maximize a likelihood
function. Hence, for this estimator the smoothing parameters were chosen based
on the observed data.
Tanner [49] considered a modified kernel-type estimator of r ° in the form
n
A;
~,,(x) = (2Rk) - ~ ~ I':((x - Z,)/2R,¢),
i=~ n - i + 1
where R k was the distance from x to the kth nearest of the uncensored observa-
tions among X 1. . . . . Xn. This estimator allowed the data to play a role in deter-
mining the degree of smoothing that would occur in the estimate. Assuming that
S o and f o were continuous in a neighborhood about x, k = [n~], ½ < ~ < 1, where
[ • ] was the greatest integer function, that K had bounded variation and compact
Nonparametric estimation of density and hazard rate functions 321
support on the interval [ - 1, 1], and that r ° was continuous at x, it was shown
that ~n(x) was strongly consistent.
Blum and Susarla [3] considered the estimator (in the notation of
Equation (3.2))
(f°H*),,(x)
~n(x) - , x >~ O ,
S*(x)
where S*(x) = (number of Zj's > x)/n. This estimator was also of the kernel type,
and limiting results similar to those stated for the density estimator (3.2) were
obtained for ~,.
Ramlau-Hansen [44] used martingale techniques to treat the general multiplica-
tive intensity model. His results are very general and include the kernel estimators
of hazard rate functions of FOldes, Rejt6 and Winter [16] and Yandell [61]. The
martingale techniques yielded local asymptotic properties of many of the hazard
rate estimators in a simpler manner than classical procedures.
Finally, in a recent paper Liu and Van Ryzin [26] obtained a histogram
estimator of the hazard rate function from randomly right-censored data based on
spacings in the order statistics. They showed the estimator to be uniformly
consistent in a bounded interval and asymptotically normal under suitable con-
ditions. An efficiency comparison of their estimator with the kemel estimator of
hazard rate was also given. Also, Liu and Van Ryzin [27] gave the large sample
theory for the normalized maximal deviation of a hazard rate estimator under
random censoring which was based on a histogram estimate of the subsurvival
density of the uncensored observations.
4. Likelihood methods
be the set of distributions with decreasing left-continuous densities on [0, oo). For
the ordered censored observations (z;, b" ), i = 1. . . . . n, the likelihood function
was written as
n
L ( f °) = l-[ [f°(zi)]a; [S°(zi)] 1-~; ,
i=1
subject to (i) Y l > > ' Y 2 > t ' ' ' > I Y n > I O ,
O, x>~0,
f*(x) = y*,
I O,
zj_l < x < ~ z j , j = l . . . . .
X~Zn+ 1 ,
was a maximum likelihood estimator of fo, where y*+ ~, some value less than or
equal to y*, and z,+ 1 ( > z , ) were chosen so that
n+l,
the largest of the n likelihood values, analogous to Wegman's [57, 58] procedure
for complete samples.
Another approach to the problem of nonparametric maximum likelihood esti-
mation of a density from complete samples was proposed by Good and
Gaskins [20]. This method allowed any smooth integrable function on the interval
of interest (a, b) (which may be finite or infinite) as a possible estimator, but
added a 'penalty function' to the likelihood. The penalty function penalized a
density for its lack of smoothness, so that a very 'rough' density would have a
smaller likelihood than a 'smooth' density, and hence, would not be admissible.
De Montricher, Tapia, and Thompson [9] proved the existence and uniqueness
of the maximum penalized likelihood estimator (MPLE) for complete samples.
Lubecke and Padgett [30] assumed that the sample was arbitrarily right-censored,
(Xi, Ai), i = 1, ..., n, and showed the existence and uniqueness of a solution to
the problem:
maximize L(g)
subject to g(t) >/0 for all t e O,
(4.1)
f g(t) dt= 1,
and g e H(f2),
where
(4.2)
maximize L(fi)= f i
[fx
[~(x,.)] ~i 2 - ~2(t)dt
]1/21 a,-)
Iil=l -oo
(4.4)
where the forcing function is of the form
where
j=l
j#k
logLl(h)=-(k~ b~)log h
(6.3)
j#k
z, ~; z, ~" z, b; z; b;
1.151 0 1.667 1 2.119 0 2.547 1
1.170 0 1.695 1 2.135 1 2.548 1
1.248 0 1.710 I 2.197 1 2.738 0
1.331 0 1.955 0 2.199 0 2.794 1
1.381 0 1.965 1 2.227 1 2.883 0
1.499 1 2.012 0 2.250 0 2.883 0
1.508 0 2.051 0 2.254 1 2.910 1
1.543 0 2.076 0 2.261 0 3.015 I
1.577 0 2.109 I 2.349 0 3.017 1
1.584 0 2.116 0 2.369 1 3.793 0
328 W. J. Padgett
0.74
,L
0.52
0.44
4-- , i
0.37
0.29
0.22
0.15
0.07
0.00 I I
0.00 0.62 1.25 1.88 2.50 3.13 3.75 4.38 5.00 5.63 6.25
Fig. 1. D e n s i t y e s t i m a t e s for s w t i c h d a t a .
References
[1] Barlow, R. E., Bartholomew, D. J., Bremner, J. M., and Brunk, H. D. (1972). Statistical Inference
Under Order Restrictions. Wiley, New York.
[2] Bean, S. J. and Tsokos, C. P. (1980). Developments in nonparametric density estimation. Intern.
Statist. Rev. 48, 215-235.
[3] Blum, J. R. and Susarla, V. (1980). Maximal derivation theory of density and failure rate
function estimates based on censored data. In: P. R. Krishniah, ed., Multivariate Analysis V.
North-Holland, Amsterdam, New York, 213-222.
[4] Breslow, N. and Crowley, J. (1974). A large sample study of the life table and product limit
estimates under random censorship. Ann. Statist. 2, 437-453.
[5] Burke, M. and Horvfith, L. (1982). Density and failure rate estimation in a competing risks
model. Preprint, Dept. of Math. and Statist., University of Calgary, Canada.
[6] Chen, Y. Y., Hollander, M. and Langberg, N. A. (1982). Small sample results for the
Kaplan-Meier estimator. J. Amer. Statist. Assoc. 77, 141-144.
[7] Cox, D. R. (1972). Regression models and life-tables. J. Roy. Statist. Soc. Ser. B 34, 187-220.
[8] Cs6rg6, S. and Horv/tth, L. (1981). On the Koziol-Green model for random censorship. Biometrika
68, 391-401.
[9] De Montricher, G. F., Tapia, R. A. and Thompson, J. R.(1975). Nonparametric maximum
likelihood estimation of probability densities by penalty function methods. Ann. Statist. 3,
1329-1348.
[10] Dykstra, R. L. and Laud, P. (1981). A Bayesian nonparametric approach to reliability. Ann.
Statist. 9, 356-367.
[11] Efron, B. (1967). The two sample problem with censored data. In: Proc. Fifth Berkely Symp.
Math. Statist. Prob. Vo14, 831-853.
[12] Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist. 1,
209-230.
[13] Ferguson, T. S. (1974). Prior distributions on spaces of probability measures. Ann. Statist. 2,
615-629.
[14] Ferguson, T. S. and Phadia, E. G. (1979). Bayesian nonparametric estimation based on
censored data. Ann. Statist. 7, 163-186.
[15] F61des, A., Rejt6, L. and Winter, B. B. (1980). Strong consistency properties of nonparametric
estimators for randomly censored data, I: The product-limit estimator. Periodica Mathematica
Hungarica 11, 233-250.
[16] F61des, A., Rejt6, L. and Winter, B. B. (1981). Strong consistency properties of nonparametric
estimators for randomly censored data, Part II: Estimation of density and failure rate. Periodica
Mathematica Hungarica 12, 15-29.
[17] Friedman, M. (1982). Piecewise exponential models for survival data with covariates. Ann.
Statist. 10, 101-113.
[18] Fryer, M. J. (1977). A review of some non-parametric methods of density estimation. J. Inst.
Math. Appl. 20, 335-354.
[19] Gehan, E. (1969). Estimating survival functions from the life table. J. Chron. DIS. 21, 629-644.
[20] Good, U. J. and Gaskins, R. A. (1971). Nonparametric roughness penalties for probability
densities. Biometrika 58, 255-277.
[21] Kalbfleisch, J. D. and Prentice, R. L. (1980). The StatisticalAnalysis of Failure Time Data. Wiley,
New York.
[22] Kaplan, E. L. and Meier, P. (1958). Non parametric estimation from incomplete observations.
J. Amer. Statist. Assoc. 53, 457-481.
[23] Kimura, D. K. (1972). Fourier Series Methods for Censored Data, PhD. Dissertation, Uni-
versity of Washington.
[24] Koziol, J. A. and Green, S. B. (1976). A Cram6r-von Mises statistic for randomly censored
data. Biometrika 63, 465-473.
[25] Lagakos, S. W. (1979). General right censoring and its impact on the analysis of survival data.
Biometrics 35, 139-156.
330 W. J. Padgett
[26] Liu, R. Y. C. and Van Ryzin, J. (1984). A histogram estimator of the hazard rate with censored
data. Ann. Statistics .
[27] Liu, R. Y. C. and Van Ryzin, J. (1984). The asymptotic distribution of the normalized maximal
deviation of a hazard rate estimator under random censoring. Colloquia Mathematica Societatis
Janos Bolyai, Debrecen (Hungary).
[28] Lo, A. Y. (1978). On a class of Bayesian nonparametric estimates. I: Density estimates. Dept.
of Math. and Statist. Tech. Rep., University of Pittsburgh.
[29] Lo, A. Y. (1978). Bayesian nonparametric method for rate function. Dept. of Math. and Statist.
Tech. Rep., University of Pittsburgh.
[30] Lubecke, A. M. and Padgett, W. J. (1985). Nonparametric maximum penalized likelihood
estimation of a density from arbitrarily right-censored observations. Comm. Statist.-Theory
Meth. .
[31] Marshall, A. W. and Proschan, F. (1965). Maximum likelihood estimation for distributions with
monotone failure rate. Ann. Math. Statist. 36, 69-77.
[32] McNichols, D. T. and Padgett, W. J. (1981). Kernel density estimation under random censor-
ship. Statistics Tech. Rep. No. 74, University of South Carolina.
[33] McNichols, D. T. and Padgett, W. J. (1982). Maximum likelihood estimation of unimodal and
decreasing densities on arbitrarily right-censored data. Comm. Statist.-Theory Meth. 11,
2259-2270.
[34] McNichols, D. T. and Padgett, W. J. (1983). Hazard rate estimation under the Koziol-Green
model of random censorship. Statistics Tech. Rep. No. 79, University of South Carolina.
[35] McNichols, D. T. and Padgett, W. J. (1984). A modified kernel density estimator for randomly
right-censored data. South African Statist. J. 18, 13-27.
[36] Miller, R. G. (1981). Survival Analysis. Wiley, New York.
[37] Mykytyn, S. and Santner, T. A. (1981). Maximum likelihood estimation of the survival function
based on censored data under hazard rate assumptions. Comm. Statist.-Theory Meth. A 10,
1369-1387.
[38] Nair, V. N. (1984). Confidence bands for survival functions with censored data: A comparative
study. Technometrics 26, 265-275.
[39] Padgett, W. J. and McNichols, D. T. (1984). Nonparametric density estimation from censored
data. Comm. Statist.-Theory Meth. 13, 1581-1611.
[40] Padgett, W. J. and McNichols, D. T. (1984). Small sample properties of kernel density
estimators from right-censored data. Statistics Tech. Rep. No. 102, University of South
Carolina.
[41] Padgett, W. J. and Wei, L. J. (1980). Maximum likelihood estimation of a distribution function
with increasing failure rate based on censored observations. Biometn'ka 67, 470-474.
[42] Padgett, W. J. and Wei, L. J. (1981). A Bayesian nonparametric estimator of survival probability
assuming increasing failure rate. Comm. Statist.-Theory Meth. A 10, 49-63.
[43] Parzen, E. (1962). On estimation of a probability density function and mode. Ann. Math. Statist.
33, 1065-1076.
[44] Ramlau-Hanse, H. (1983). Smoothing counting process intensities by means of kernel functions.
Ann. Statist. 11, 453-466.
[45] Rosenblatt, M. (1976). On the maximal deviation of k-dimensional density estimates. Ann.
Probab. 4, 1009-1015.
[46] Scott, D. W. and Factor, L. E. (1981). Monte Carlo study of three data-based nonparametric
probability density estimators. J. Amer. Statist. Assoc. 76, 9-15.
[47] Susarla, V. and Van Ryzin, J. (1976). Nonparametric Bayesian estimation of survival curves
from incomplete observations. J. Amer. Statist. Assoc. 71, 897-902.
[48] Susarla, V. and Van Ryzin, J. (1978). Large sample theory for a Bayesian nonparametric
survival curve estimator based on censored samples. Ann. Statist. 6, 755-768.
[49] Tanner, M. A. (1983). A note on the variable kernel estimator of the hazard function from
randomly censored data. Ann. Statist. 11, 994-998.
[50] Tanner, M. A. and Wong, W. H. (1983). The estimation of the hazard function from randomly
censored data by the kernel method. Ann. Statist. 11, 989-993.
Nonparametric estimation of density and hazard rate functions 331
[51] Tanner, M. A. and Wong, W. H. (1983). Data-based nonparametric estimation of the hazard
function with applications to model diagnostics and exploratory analysis. J. Amer. Statistc.
Assoc..
[52] Tapia, R. A. and Thompson, J. R. (1978). Nonparametric Probability Density Estimation. The
Johns Hopkins Univ. Press, Baltimore, MD.
[53] Tarter, M. E. (1979). Trigonometric maximum likelihood estimation and application to the
analysis of incomplete survival information. J. Amer. Statist. Assoc. 74, 132-139.
[54] Wagner, T. (1975). Nonparametric estimates of probability densities. IEEE Trans. Inform.
Theory 21, 438-440.
[55] Watson, G. S. and Leadbetter, M. R. (1964). Hazard Analysis I. Biometrika 51, 175-184.
[56] Watson, G. S. and Leadbetter, M. R. (1964). Hazard analysis II. Sankhy~ Ser. A 26, 110-116.
[57] Wegman, E. J. (1970). Maximum likelihood estimation of a unimodal density function. Ann.
Math. Statist. 41, 457-471.
[58] Wegman, E. J. (1970). Maximum likelihood estimation of a unimodal density, II. Ann. Math.
Statist. 41, 2160-2174.
[59] Wellner, J. (1982). Asymptotic optimality of the product limit estimator. Ann. Statist. 10,
595-602.
[60] Wertz, W. and Schneider, B. (1979). Statistical density estimation: A bibliography. Internat.
Statist. Rev. 47, 155-175.
[61] Yandell, B. S. (1982). Nonparametric inference for rates and densities with censored serial data.
Biostatistics Program Tech. Rep., University of California, Berkeley.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 | "7
© Elsevier Science Publishers B.V. (1988) 333-351 .Ik/
Introduction
A sample of, say, ten items is drawn from the process at regular intervals and
measurements are obtained on both variables.
For the time being, attention will be focused on monitoring the process means.
One approach would be to ignore the correlation between the characteristics and
monitor each process mean separately. For each sample of size ten, an estimate
of #o~, denoted by Ya, is obtained and plotted against time on an Y-chart with
the following control limits:
Since 3-sigma limits were used in determining (1), the type I error for this chart
equals 0.0027. Another Y-chart would be set up to monitor the process mean of
333
334 F. B. Alt and N. D. Smith
UCL 2 = 32.85,
CL 2 = 30, (2)
LCL 2 = 27.15.
If both sample means plot within their respective control limits, the process is
deemed to be in control. The use of separate 2-charts is equivalent to plotting
(Xl, 52) on a single chart formed by superimposing one 2-chart over the other,
as shown in Figure 1. If the pair of sample means plots wihtin the rectangular
control region, the process is considered to be in control.
~2
, I
UCL 2
-- - ~ Region A
i \ \°. \L Regio~B
LCL 2
_ i __~ R o g i o o c
I
I
LCL 1 UCL 1
The use of separate control charts or the equivalent rectangular region can be
very misleading. It will be shortly demonstrated that the true control region is
elliptical in nature, and the process is judged out of control only if the pair of
means (Y1, 52) plots outside this elliptical region. However, if the rectangular
region is used, it may be erroneously concluded that both process means are in
control (Region A), one is out or control and the other is in (Region B), and both
are out of control (Region C). The degree of correlation between the two variables
affects the size of these regions and their respective errors. Furthermore, the
probability that both sample means will plot within the elliptical region when the
process is in control is exactly l - e , whereas with the rectangular region, this
probability is at least 1-e.
Although the use of separate charts to individually monitor each process mean
suffers from the weakness of ignoring the correlation between the variables, these
Y-charts can sometimes assist in determining which process mean is out of
control. When 5-charts are used in this supplemental fashion, it is recommended
that the type I error rate of each one be set equal to e/p, where p is the number
of variables and e is the overall type-I error. When p = 2 and e = 0.0054, the
Multivariate process control 335
type-I error of each chart would be set at 0.0027 which implies 3-sigrna limits as
used in equations (1) and (2).
In the sequel, control charts will be presented for both Phase I and Phase II
with the presentation for Phase II being first. In both cases, the charts are
referred to as multivariate Shewhart charts.
In some instances, estimates of/~o and '~o may be derived from such a large
amount of past data that these values may be treated as parameters and not their
corresponding estimates. Duncan [10] states that the values for the parameters
could also have been selected by management to attain certain objectives. These
are referred to as standard or target values. Phase II comprises both scenarios.
2
Xo2 = n ( i - / t o ) ' ,~o- I(X _/to) > Zp. ~, (4)
where X denotes the (p x 1) vector of sample means and 2;0 ~ is the inverse of
the (p x p) variance-covariance matrix. For the case of two quality characteristics,
336 F. B. Alt and N. D. Smith
where Xp,,
,2 denotes the noncentral chi-square random variable with p degrees of
freedom and noncentrality parameter 2 = n(/, -/*o)' ~g 1(/~ _/1o)" For p = 2,
The power is strictly increasing in 2 for fixed significance level ~ and fixed sample
size n. Wiener [25] presents tables of the noncentrality parameter 2 for signifi-
cance levels (e) of 0.100, 0.050, 0.025, 0.010, 0.005 and 0.001, degrees of freedom
equal to 1(1)30(2)50(5)100, and for power values (n) of 0.10(0.02)0.70(0.01)0.99.
For example, suppose ~ = 0.005, n = 10, Po = - 0 . 4 , and it is important to detect
a shift of magnitude 0.5 standard deviations in the mean of each variable. Then
r~(2) = 0.42. If Po = - 0 . 2 , the power decreases to 0.28. When there are two
positively correlated characteristics and one of the process standard deviations
(crl) can be adjusted, Alt et al. [6] found that the power is not a monotonically
decreasing function of crI as it is in the univariate case.
A fundamental assumption in the development of the Z2-chart is that the
underlying distribution of the quality characteristics is multivariate normal. In the
univariate case, the effect of nonnormality on the control limits of Y-charts was
studied by Schiling and Nelson [24].
By minimizing the average run length of an out-of-control process for a large
Multivariate process control 337
fixed value of the average run length of an in-control process, Alt and Deutsch [4]
determined the sample size (n) and control chart constant (Xj;2, , ) w h e n there are
two quality characteristics. T h e y f o u n d t h a t (i) f o r a r e l a t i v e l y l a r g e p o s i t i v e c o r r e -
Table 1
Control chart data for the mean-standards given
Sample xl xz Zo2
number
lation, a larger sample size is needed to detect large positive shifts in the means
than small positive shifts, and (ii) a larger sample size is needed to detect shifts
for p > 0, than when p < 0. Montgomery and Klatt [21, 22] present a cost model
for a multivariate quality control procedure to determine the optimal sample size,
sampling frequency, and control chart constant.
Although Hotelling [15, 16] proposed the use of the Z2 random variable in a
control chart setting for the testing of bombsights, he did not actually use Z2-con -
trol charts since the variance-covariance matrix (Zo) was unknown. His papers
are primarily devoted to the case for 2;o unknown.
To illustrate the use of the Z2-control chart, consider the data listed in Table 1
for the plastic film extruding plant described in the Introduction. The sample size
is ten. To assess the impact of changes in either one or both process means, note
that #o~ was increased by increments of 0.125trOl for data sets 12 through 19 while
#ol and #02 were each increased by increments of 0.125ao;, i = 1, 2, for data sets
20 through 27. Since type I error was set equal to 0.0054, UCL = ~ 22, 0.0054
2
XO
24
21
18
15
12
UCL
9
i i i i i i + i
5 10 15 20 25
Sample No.
Fig. 2.
= 10.44. The X2-control chart is illustrated in Figure 2. When only #Ol was
changed (sample numbers 12 to 19), the value of the test statistic (Zo2) exceeded
the UCL as soon as #Ol was increased by at least 0.5 standard deviations (sample
numbers 15 to 19). Furthermore, when #ol and #o2 were simultaneously altered
(sample numbers 20 to 27), Zoz > UCL as soon as each process mean had been
increased by 0.375 standard deviations (sample numbers 22 to 27). The control
limits for the individual control charts were presented in equations (1) and (2).
For sample numbers 12 to 19, the 2-chart for transparency (X1) performed as well
as the zZ-chart. This result is not surprising since the process mean for this
variable alone increased. However, when both process means were increased
(sample numbers 20 to 27), the individual charts did not perform as well as the
Multivariate process control 339
i(2-chart. Specifically, the Z-chart for transparency did not detect an increase until
/~ol had increased by at least 0.5 standard deviations and the 2-chart for tear
resistance did not plot out-of-control until #02 had increased by at least 0.875
standard deviations.
Values of d 2, d3, D1, and D 2 are presented in Table M of Duncan [ 10] for n = 2
to n -- 25. Duncan [ 10] also gives details for constructing a percentage point chart
based on the distribution of W = R/a o.
Another chart that makes use only of the first two moments of the sample
statistic is the S-chart, where S denotes the sample standard deviation with a
divisor of (n - 1). It is known that E(S z) = ag and E(S) = aoc4, where
=[ 2 11/2 F(n/2)
c4 I_n 11_1 r((n- 1)/2) (9)
Thus, Var(S) = E(S a) [ E ( S ) ] 2 = ag(1 - c2). Since most of the probability dis-
-
tribution of S is within 3 standard deviations of E(S), the control limits for the
S-chart are as follows:
CL = O'oC4, (10)
L C L = 00[c4 - 3 , / 1 - c4 1 -- 500-
340 F. B. Alt and N. D. Smith
Table 2
C o n t r o l c h a r t d a t a for p r o c e s s d i s p e r s i o n - - s t a n d a r d s given
U n i v a r i a t e c o n t r o l limits
T r a n s p a r e n c y (x 0 Tear resistance
(xz)
R-chart UCL 1 = 49.22 U C L 2 = 16.41
C L 1 = 27.70 C L z = 9.23
L C L 1 = 6.18 L C L 2 = 2.06
Multivariate control
limits
IS I '/Z-chart U C L = 51.95
(Probability L C L = 6.60
limits)
[S r'/2-chart U C L = 47.17
( 3 - s i g m a limits) C L = 22.90
L C L = 0.00
W*-chart U C L = 12.38
(a = 0.01)
U C L = t76,~,
2 n2_ i , = / 2 / ( n - 1) , LCL = Oz
' 6 ~2
n - 1, i_(=/2)/(n- 1) . (11)
However, Guttman, Wilks, and Hunter [ 11 ] point out that is is customary to use
only an upper control limit; specifically, U C L = a~Z~_ 1, J ( n - 1). Note that the
S2-chart is equivalent to repeated tests of significance of the form Ho: a z = a~ vs.
HI: a 2 ¢ a~ where the critical region for this test is equivalent to the regions
above the U C L and below the LCL, as stated in equation (11). The power of the
test is given by:
g ( ~ ) = 1 -- p [ ) ] - 2 ~ 2 _ 2
1, 1-(~/2)~ Zn-I ~ ,'~-2 ~2_ 1,=/2],
S=
[ ] S12
s21
Sip
S12 • . . Slp
s22
S2p • . . s2p
S2p
where the diagonal elements are the sample variances and the off-diagonal
elements are the sample covariances. For the ease of two quality characteristics:
342 F. B. Alt and N. D. Smith
Recall that the sample correlation coefficient for the ith and jth variables, denoted
by r~j, is defined as r,y = so~sisj.
The sample generalized variance, denoted by IS I, is a widely used scalar
measure of multivariate dispersion. For two variables, ISI =
s21s~- s~2 = s~s~(1 - r 2 2 ) . A geometrical interpretation of IS I for two variables
will now be presented. Let D denote the (2 x n) data matrix after centering:
UCL = 1~011/2~2n
2 - 4, ~/2/2( n -- 1), LCL = ]I;ol 1/2 22n--4,
2 1--C~/2)/2(n -- 1),
(12)
Multivariate process control 343
where I fop 1/2 = 0-01 0"02 N / / ~ - -- ]02). For the plastic film example, J27o11/2 = 25.76.
Thus, for each r a n d o m sample of size n, IS] 1/2 = (s?s~ - s22) 1/2 = s , s 2 x / ~ - r~2
is computed. If IS [ ,/2 > U C L or iS] 1/2 < LCL, the dispersion of the process is
deemed to be out of control and assignable causes are sought. Although the exact
distribution of pS 11/2 for p > 2 is unknown, several approximations are available
and discussed in Alt [ 1 ]. The second approach utilizes only the first two m o m e n t s
of IS] '/2 and the property that most of the probability distribution of IS[ ~/2 is
within three standard deviations of its expected value. Since
Isl = (n - 1)-" 12701 I-i"k =, g 2 - ~, where the chi-square r a n d o m variables are inde-
pendent, it follows that
P
E ( ] S I r) = (n - 1)-Pr 2 pr 127olr [I r(r + (n - k ) / 2 ) / F ( ( n - k)/2).
Thus k=l
E ( [ S I '/2) = 1~.o11/2(2/(n - 1)) p/z r ( n / 2 ) / r ( ( n -p)/2) = 127o1~/2b3 (13)
and
P
E ( I S I ) = I~,ol (n - 1) - p 1-I (n - k) = 12ol 61. (14)
k~l
Now
b, = (n - 2)/(n - 1)
and
b3 = (2/(n - 1)) [r(n/2)/r((n - 2 ) / 2 ) ] .
Anderson [7] shows that the likelihood ratio test of Ho: $ = ~7o vs. H1: , ~ '~o,
modified to be unbiased (the power of the rest is greater than or equal to the
significance level), is based on the following statistic:
In Phase II, control charts are used to determine whether the process is in
control at the standard values (/~o, ~o). During the initial stages of process sur-
veillance,/~o and ,~o are usually unknown and must be estimated from preliminary
samples taken when the process is believed to be in control. These preliminary
samples are referred to as rational subgroups and m is used to denote the number
of subgroups. When there is one quality characteristic, the procedure ordinarily
used to construct the Phase I control chart limits is to replace the standard values
in the Phase II charts by unbiased estimates obtained from the m rational
subgroups. For example, #ol in equation (1) would be replaced by the average of
the sample means obtained from each rational subgroup, and any one of several
measures of variability would be used in place of trol. However, Hillier [13] and
Multivariate process control 345
Yang and Hillier [26] have developed a two-stage procedure using probability
limits for determining whether the data for the first m subgroups came from a
process that was in control (Stage I) a n d whether future subgroup data from this
process exhibit statistical control. This was extended to the multivariate case by
Alt et al. [ 5].
respectively. F o r the plastic film extrusion process, m was chosen to be 10. The
elements of the sample m e a n vectors a n d covariance matrices are recorded in
Table 3.
W h e n standard values for/~ a n d • are available, the test statistic is stated in
equation (4). If the s t a n d a r d values are replaced by their unbiased estimates, the
Table 3
Statistics for control charts for the mean (Phase I--Stage I)
To determine whether the process was in control when the first m subgroups
were obtained, the m values o f T.o. 2 1 are plotted on a chart with U C L =
c l ( m , n , p ) F p , mn m p + l , , a n d L C L = 0- If T'2o,1 for one or more o f the m initial
subgroups plots out o f control, the c o r r e s p o n d i n g subgroups are d i s c a r d e d and
the first stage control limits are recalculated on the basis o f the remaining
subgroups. This p r o c e d u r e is illustrated for the plastic film extrusion p r o c e s s ; the
s u m m a r y statistics are r e c o r d e d in Table 3. To simulate an out-of-control process,
each process mean was increased by one s t a n d a r d deviation for subgroups 4 and
8. N o t e that T 02, 1 exceeded the U C L (with ~ = 0.001) for these two subgroups.
As a consequence, these subgroups were discarded, x and S were recomputed,
and new control limits were determined using m = 8. F o r the remaining eight
subgroups, the recalculated values of T o,, 2 are less than the revised U C L . The
process a p p e a r s to be in control with respect to its mean.
F o r the case when p = 1, U C L = ( ( m - 1 ) / m ) F a , m ( , _ l ) " and T,02, 1 =
n ( x i - ~ ) 2 / s 2 where s 2 was previously defined as the average o f the sample
variances obtained from each subgroup. Since F,, . , ( n - ,), ~ = t,~cn- 2 1), ~/2, it follows
that 1 - ~ = P[(X i - ~)2/~2 ~ ((m - 1)/m)F,,m(,, ,), ~]
= P [ IXi - ~1 ~ x / S 2 ( ( m - l ) / m ) t m ( n - , ) , ~/21
= P[~-A4x/~5<~X,<~+A4x/SS], (21)
where A 4 = x / ( m - 1)/m tr~¢~- 1), ~/2" Thus, the multivariate result reduces to the
univariate result previously o b t a i n e d by Yang and Hillier [26]. Furthermore,
Bonferroni intervals for the individual characteristics are obtained by using
A 4 = x / ( m - 1)/m tm( n _ 1), a/2p" F o r p = 2, the upper and lower control limits for
each variable are given by 2 + A 4 x / / ~ . Setting m = 10, n = 10, and a = 0.001
yields the following control limits:
U C L , = 124.81, U C L 2 = 39.67,
L C L 1 = 59.57, L C L z = 20.77.
Multivariate process control 347
Although each process mean had increased by one standard deviation for
subgroups 4 and 8 and this was detected by T,02, 1 ~ these increases failed to show
up on the univariate charts.
where x and S are obtained from Stage I. It is shown in Alt et al. [5] that T02, 2
is distributed as c2(m, n, p)Fp . . . . . . p + 1 where
In order to determine whether the mean remains in control during Stage II, values
of T,02, 2 for each future subgroup are plotted on a control chart with
UCL = c 2 ( m , n, p)Fp . . . . . . p+ ~, ~ and LCL = 0. If Z2o,z exceeds the UCL, an
assignable cause is sought. Yang and Hillier [26] suggest that 2, S and the UCL
be updated fairly often in the beginning, with less updating after the process has
stabilized. The To2,2-chart can be supplemented by charts for each quality charac-
teristic. For p = 2, the upper and lower control for limits for each variable are
x + A* ~/fi, where A * = x / ( m + 1)/m tmc,- 1), cx/2p.
The summary statistics for the plastic film extrusion process are recorded in
Table 4. Since each mean was increased by one standard deviation for f--- 4 and
Table 4
Statistics for control charts for the mean (Phase I - - S t a g e II)
f
1 90.38 29.55 0.03
2 86.98 29.81 1.05
3 92.39 29.98 1.03
4 88.91 34.22 26.40*
5 89.61 29.44 0.13
6 91.02 29.44 O.14
7 87.96 29.16 1.20
8 103.31 33.11 50.88*
9 86.13 29.35 2.39
10 85.76 30.69 2.37
8, it is not surprising that ToE,2 > U C L (with ~ = 0.005) for these two subgroups.
Since the test statistics for eight more subgroups have plotted in control, ~ and
may be recomputed using the sixteen subgroups where T,02, 1 and T,02, 2 plotted
in control.
It is customary to use only an upper control limit where the percentage point is
2
Zn- l, at"
By using the summary statistics recorded in Table 3, control limits can be
determined for the S and S2-charts. For the transparency variable (X1), 31 = 9.15;
for the tear resistance variable (Xa), 32 = 2.68. From Table M of Duncan, it is
seen that, for n = 10, B 3 = 0.284, and B 4 = 1.716. For X1, the control limits are
U C L 1 = 15.70 and LCL 1 = 2.60; for X 2, U C L a = 4.60 and LCL a = 0.76. Since
none of the points falls outside the S-control limits for either variable, the
variability of the process is deemed to be under control during this preliminary
period of 10 subgroups. For the sa-charts, U C L 1 = (90.98)(25.16)/9 = 254.34
Multivariate process control 349
and UCL 2 --- (7.64)(25.16)/9 = 21.36. Again, none of the points exceed the upper
control limits and the same conclusion would be reached.
When there are multiple quality characteristics, two variations of the IS[ 1/2_
chart were presented for monitoring orocess dispersion during Phase II. The first
was a probability limit chart with control limits as stated in equation (12). These
particular limits are applicable only when p = 2. Let IS*[ 1/2 denote the average
of the square roots of the generalized sample variances. That is,
I S , I 1/2 = (l/m) E m
,'= 1 IS;[ 1/2. Since IS * 1!/2/b3 is an unbiased estimate of I1~011/2,
Phase I control limits are as follows:
When p = 1, the above control limits for a [S*[ 1/2-chart are identical to those for
the S-chart with the B 3 and B 4 factors stated in equation (23). Another procedure
that could be used for investigating process dispersion during Phase I is obtained
from equation (17), which was the likelihood ratio statistic for testing Ho: Z = 27o.
To obtain the corresponding Phase I procedure, unbiased estimates of [I;o[ and
270 1 are needed. Let PSol denote the average oftrl the generalized sample variances
from the m subgroups. That is, ISol = (i/m) ~i= 1 ISil- By using the result stated
in equation (14), it can be shown that 1Sol~b1 is an unbiased estimate of 127ol-
Let SF 1 denote the inverse of the sample variance-covariance matrix for subgroup
i, i = 1. . . . , m. Kshirsagar [ 19] shows that (n - p - 2)S 7 l/(n - 1) is an unbiased
estimate of 270 1• Thus, if S , 1 = ( /1 m ) ~ =m 1 $ 7 1, then (n - p - 2 ) S , l/(n - 1) is
an unbiased estimate of 270 1 obtained from the m rational subgroups. The Phase I
procedure is obtained by substituting ISol~b1 for t 27ot and (n - p - 2 ) S , 1/(n - 1)
for Zo-1 in equation (17). The revised values of W*, i = 1. . . . , m, would still be
plotted on a control chart with UCL = )~p(p 2 + 1)/2-
T h e control limit factors used during Phase I for both one and more than one
quality characteristic were independent of the number of subgroups. Some authors
argue that these factors should also be a function of m, the number of subgroups.
Such factors are presented in Alt [ 1].
The 3-sigma IS*ll/2-chart will be used to investigate the variability of the
plastic film process. The value of ]Se[ 1/2 for each of the initial ten subgroups can
be obtained from the summary statistics presented in Table 3. For example,
350 F. B. Alt and N. D. Smith
Table 5
Statistics for 3-sigma [S*[l/2-chart
Subgroup 1 2 3 4 5 6 7 8 9 10
iStl J/2 22.29 16.85 30.13 18.50 22.97 30.91 22.21 18.15 24.66 7.96
IS*I 1/2 = 21.46
UCL = 44.21 LCL = 1.29
Other approaches
S~=O if C.<<.k,
S,,=-(S,,_~ + X n - / J o ) ( 1 - k / C , ) if C , , > k ,
where So = 0 and k > 0. Crosier's scheme signals when S ,n ~ - 1Sn > h 2.
Healy [ 12] developed a C U S U M p r o c e d u r e b a s e d on the sequential probability
ratio test. Let 6 denote the shift from /~o that is i m p o r t a n t to detect. Define
D = x / ~ , ~ - 1 ~ and a ' = ~t' ~ - l i D . Then H e a l y ' s scheme has
S n -- m a x { S n _ l + a'(x,, - / 1 o ) - 0.5 D, 0} .
References
[1] Alt, F. B. (1973). Aspects of multivariate control charts. M.S. thesis, Georgia Institute of
Technology, Atlanta.
[2] Alt, F. B. (t982). In S. Kotz and N. L. Johnson, eds., Encyclopedia of Statistical Sciences, Vol. 1,
Wiley, New York, 294-300.
[3] Alt, F. B. (1985). In S. Kotz and N. L. Johnson, eds., Encyclopedia of Statistical Sciences, Vol. 1,
Wiley, New York, 110-122.
[4] Alt, F. B. and Deutsch, S. J. (1978). Proc. Seventh Ann. Meeting, Northeast Regional Conf. Amer.
Inst. Decision Sci., 109-112.
[5] Alt, F. B., Goode, J. J., and Wadsworth, H. M. (1976). Ann. Tech. Conf. Trans. ASQC, 170-176.
[6] Aft, F. B., Walker, J. W., and Goode, J. J. (1980). Ann. Tech. Conf. Trans. ASQC, 754-759.
[7] Anderson, T. W. (1984). An Introduction to Multivariate Statistical Analysis, 2nd ed., Wiley, New
York.
[8] Bowker, A. H. and Lieberman, G. J. (1959). Engineering Statistics. Prentice-Hall, Englewood
Cliffs, NJ.
[9] Crosier, R. B. (1986). Technometrics 28, 187-194.
[10] Duncan, A. J. (1974). Quality Control and Industrial Statistics. 4th ed. Richard D. Irwin,
Homewood, IL.
[11] Guttman, I. and Wilks, S. S. (1965). Introductory Engineering Statistics. Wiley, New York.
[12] Healy, J. D. (1987). Technometrics. To appear.
[13] Hillier, F. S. (1969). J. Qual. Tech. 1, 17-26.
[14] Hoel, P. G. (1937). Ann. Math. Stat. 8, 149-158.
[15] Hotelling, H. (1947). In: C. Eisenhart, H. Hastay, and W. A. Wallis, eds., Techniques of
Statistical Analysis, McGraw-Hill, New York, 111-184.
[16] Hotelling, H. (1951). Proceedings of the Second Berkeley Symposium on Mathematical Statistics and
Probability. University of California Press, Berkeley, CA, 23-41.
[17] Jackson, J. E. (1985). Commun. Statist.-Theor. Meth. 14, 2657-2688.
[18] Johnson, R. A. and Wichern, D. W. (1982). Applied Multivariate Statistical Analysis. Prentice-
Hall, Englewood, NJ.
[19] Kshirsagar, A. M. (1972). Multivariate Analysis, Marcel Dekker, New York.
[20] Montgomery, D. C. (1985). Introduction to Statistical Quality Control, Wiley, New York.
[21] Montgomery, D. C. and Klatt, P. J. (1972). Manag. Sci. 19, 76-89.
[22] Montgomery, D. C. and Klatt, P. J. (1972). AIIE Trans. 4, 103-110.
[23] Pignatiello, J. J., Runger, G. C. and Korpela, K. S. (1986). Truly multivariate CUSUM charts.
Working Paper # 86-024, College of Engineering, University of Arizona, Tucson, AZ.
[24] Schilling, E. G. and Nelson, P. R. (1976). J. Qual. Tech. 8.
[25] Wiener, H. L. (1975). A Fortran program for rapid computations involving the non-central
chi-square distribution. NRL Memorandum Report 3106, Washington, DC.
[26] Yang, C.-H. and Hillier, F. S. (1970). J. Qual. Tech. 2, 9-16.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 1
1 u
© Elsevier Science Publishers B.V. (1988) 353-373
Bruce Hoadley
353
354 B. Hoadley
The audit affects the second quantity in this expression via a feedback mecha-
nism, Figure 1. Under this feedback model, the production process is a controlled
stochastic process. When the process is at the standard level (one on an index
scale), there is a small probability per period that the process will change to a
substandard level. Given this substandard level, there is a probability per period
"x TIME
STANDARD: I
• - I I I l i i ~ I
QUALITY:
"~(e)
SUBSTANDARD: b
DEFECT f/DEPEN~DS
~ ONe,l
PLANJ
INDEX
that the audit will detect this change. This probability depends on the audit
sample size. When detection occurs, management acts and forces the process
back towards the standard. This phenomenon is empirically observed in audit
data. So, sample size affects average long run quality, because it affects detection
probability.
In this feedback model, we ignore the possibility of false detection when the
process is at standard. There is an implicit assumption that the cost of false
detection is large and therefore, the producer's risk is small. When detection
occurs, management is supposed to act. Such action frequently involves the
expenditure of substantial resources on a quality improvement program. So the
whole audit strategy is founded on the integrity of exception reporting. Otherwise,
management would pay little attention to the results.
Our model for the total audit cost per period per product is
The USP sample sizes for all products are determined jointly by minimizing the
audit costs plus the field maintenance costs subject to an inspection budget
constraint. The approximate solution to this problem is the ollowing formula for
e = [expected number of defects in the audit sample per period]:
where
e = Expected number of defects in this audit sample, given standard quality (called
expectancy)
= nS,
n = Sample size,
s = Standard defects per unit,
N = Production,
r = G/Ca,
Cr = Field maintenance cost per defect,
C a = Audit cost per unit of expectancy,
P = Probability per period that the process will change to a substandard level,
b
= Process control factor,
B = Budget control factor (monotonically related to the Lagrange multiplier asso-
ciated with the budget constraint).
Note that we express the solution in terms of the expectancy in the sample--not
the sample size itself. Expectancy is the natural metric for describing the size of
an audit; because, the detection power depends on the sample size through
356 B. Hoadley
expectancy. Five switching frames with a standard of one defect per unit generate
as much information as 5000 transistors with a standard of 0.001 defects per unit.
For most applications, the expectancy in the sample ranges from 1 to 10, whereas
sample sizes range from 5 to 10000.
EXAMPLE. Test audit for a small business system. This example illustrates USP in
its simplest form. For a test audit, the quality measurement is total test defects.
The standard is s = 0.06 defects per unit. An analysis of past audit data yielded
P = 0.04. Economic analyses yielded Ca = $430 and Cr = $280, so
r = $280/$430 = 0.65. If B = 2 (used by Western Electric), then the expectancy
formula is
The sample size version of this formula is n ~fJ_~0.06) = (0.93) x / ~ . If the produc-
tion is N = 2820, then e = (0.056)~/2820 = 3.0. The sample size is
3.0/(0.06) = 50. Under the traditional plan [8], the sample size would have been
x/2 x/N = ~ = 75, a 50 percent increase.
t977 1978 1
3
I
(b)
÷2
-2
I I - ~ - "
'1 ~ 3 4 5 6 78"1 2345 67 8
and the value two means twice as many defects as expected under the quality
standard. The posterior probability that the population index is larger than the top
whisker is 0.99. The top of the box, the bottom of the box and the bottom whisker
correspond to probabilities of 0.95, 0.05, 0.01 respectively.
For the Western Electric application of QMP, exceptions are declared when
either the top of the box or the top of the whisker are below standard (i.e., greater
than one on the index scale). This makes the producer's risk small, as explained
in Section 1.1.
The posterior distribution of current population quality is derived under the
assumption that population quality varies at random from period to period. This
random process has unknown process average and process variance. These two
unknown parameters have a joint prior distribution, which describes variation
across product.
The heavy 'dot' is a Bayes estimate of the process average; the ' x ' is the
observed value in the current sample; and the 'dash' is the posterior mean of the
QMP/USP--A modern approach to statistical quality auditing 359
current population index and is called the Best Measure of current quality. This
is like an empirical Bayes estimate--a shrinkage towards the process average. The
process averages ('dots') are joined to show trends.
Although the T-rate chart and the QMP chart sometimes convey similar
messages, there are differences. The QMP chart provides a measure of quality;
the T-rate chart does not. For example, in period 6, 1978 both charts imply that
the quality is substandard, but the QMP chart also implies that the population
index is somewhere between one and three. Comparing period 6, 1977 with
period 4, 1978 reveals similar T-rates, but QMP box plots with different messages.
The QMP chart is a modern control or feedback chart for defect rates. However,
the important outputs of QMP are the estimated process distribution (sometimes
called the prior distribution) and the posterior distribution of current quality. In
other decision making contexts, such as Bayesian acceptance sampling [ 11 ], these
distributions could be used to optimally inspect quality into the product via the
screening of rejected lots [10]. So QMP provides a practical tool for applying
Bayesian acceptance sampling plans.
2. U S P details
We assume:
(i) S' (e) and A' (e) > 0 exist for e > 0.
(ii) (dS/dA)(e) is monotonically decreasing for e > 0.
We deal with m a n y products simultaneously; so, for product i, we use the
subscript i. The general U S P problem is to select e,., i = 1. . . . . I, to minimize
Y.i [Ai(ei) - Si(ei)] subject to the constraints: (i) e i >>,O, (ii) •iAi(ei) <~M.
From K u h n - T u c k e r theory [9], there exists a Lagrange multiplier, 2, so that the
optimal et's satisfy:
(4)
(5) 2/>0, ei>~O, i=1 ..... I.
A(e) : Cae.
The audit savings function: S(e). Let F(e) denote the field maintenance cost
associated with an audit of size e. Then
F(e) = O(e)sCfN,
where
Recall that 0(e) arises from the quality behavior model described in Figure 1.
When the process is at the standard level, there is a probability, P, per period of
a change to the substandard level (b). So, the expected waiting time until this
change is E[Y] = 1/P. When the process is at the substandard level (b), we
assume that in each period, the audit detects the substandard level if the number
of defects, x, exceeds an acceptance number', c. We assume that x has a Poisson
distribution with mean n . s . b = e" b; so, the expected waiting time until detection
depends on e and we define D(e) by E [ Z ] = liD(e). Note that D(e) can be
interpreted as an average detection power. Hence,
0(e)=[ E(Y)
LE(Y) +
1
E(Z)] (1)
F
+ LE(Y) + E(Z)_]
P
=1 +[p +D(ei](b-1).
t When QMP is used for detection, the acceptance number is a random variable.
362 B. Hoadley
S'(e)
-1+2.
A'(e)
E[ , ( b - l) -]1/2
LD (0)31+ 2)lPrNsJ <P/D'(O).
e
D(e) -
e+a
where
a(b - 1)
B - - -
(1 + ~)
otherwise.
3. QMP details
= SHt
where
It xt/e t .
(3) 0 and 72 have a joint prior distribution. The physical interpretation of this
prior is that each product has its own value of 0 and 72 and these vary at random
across products.
Assumption (3) makes this a Bayes empirical Bayes model. We never specify
the form of this joint prior; because, in our heuristic derivation, only its moments
are used.
This is now a full Bayesian model. It specifies the joint distribution of all
variables. The quality rating in QMP is based on the conditional or posterior
distribution of Or given x = (xl, . . . , xr).
E [ a r l x ] = Or = 6jrO + (1 + 6Jr)IT,
where
O/er
b = E(O[ x ) , 6JT = E(coT I x ) , cot =
O/er + ?2
If the process is stable, relative to the sampling variance, then the process
variance is relatively small and the weight is mostly on the process average; but
if the process is unstable, then the process variance is relatively large and the
weight is mostly on the current sample index. The reverse is true of the sampling
variance. If it is large (e.g., small expectancy), then the current data is weak and
the weight is mostly on the process average; but, if the sampling variance is small
(e.g., large expectancy), then the weight is mostly on the current sample index. In
other words, cor, is monotonically increasing with the ratio of sampling variance
to process variance.
QMP/USP--A modern approach to statistical quality auditing 365
If the process average and variance were known, then the posterior variance of
Or would be (1 - < n r ) 0 r / e r , which is estimated by the first term in Vr. But
since the process average and variance are unknown, the posterior variance has
two additional terms. One contains the posterior variance of the process average
and the other contains the posterior variance of the weight.
The first term dominates. A large 6)r (relatively stable process), a small Or
(good current quality) and a large e r (large audit) all tend to make the posterior
variance of Or small. If 697- is small, the the second term is negligible. This is
because the past data is not used much, so the uncertainty about the process
average is irrelevant. If the current sample index is far from the process average,
then the third term can be important. This is because outlying observations add
to our uncertainty.
If the process average and variance were known, then the posterior distribution
would be Gamma, so we approximate the form of the posterior distribution by
a Gamma. The parameters of the fitted Gamma distribution are ~ = shape param-
^2 t,
eter = O r / V r, z = scale parameter = V r / O r. And the approximate posterior
distribution function is
/~ 1
P r [ O r < ~ z l x ] = G~(z/z) =
ff -- x ~-1 e - X d x .
(see the Appendix for detailed formulas). If the process average and variance were
known, then the conditional mean and variance of Or+ 1 would be simply 0 and
72. But since they are unknown, we use Bayes estimates of O and ~,2 and add the
term V[01 x] to the conditional variance to account for the uncertainty in our
estimate of O.
The^ parameters of the fitted G a m m a distribution are ~1 = shape parame-
ter = 02/Vr+l, z I = scale parameter = VT+1/O. SO,
P = 1 - G~,(b/zl).
INDEX SCALE
2 tO 0 2I 3I 4I
) (b)
o ~,Z~--I,,
_~ -I ×lJc
~ " I ×t I°
~ ~ I × I IE
5 5
7=51 Z ( I t - i ) 2 , ~
, ~ . 2 ..
~1 E ( I t - ] ) 2 ,
t=l t=l
where I, is the sample index in past period t. For given values of i and S 2, we
used a standard pattern of It's to compute the Bogie. The results are insensitive
to the pattern. The dashed curve is an upper bound for S 2.
To see how the contour plot works, consider an example. Suppose 7 = 0.8 and
S 2 = 0.7. The point (0.8, 0.7) falls on the contour labeled 2.6. This means that
if the current sample index exceeds 2.6, then the product will be Below Normal.
The contour labeled 2.6 is the set of all pairs (i, S 2) that yield a Bogie of 2.6.
368 B. Hoadley
5.0
4.0
.9
laJ
¢..)
Z
n-
~> 3.0
.8
t--
u)
13-
.7
2.0
.6
.5
t.0
.4
.3
.2
This contour plot summarizes the Below Normal behavior of QMP for an
expectancy of five. As i gets larger than one, the Bogie gets smaller. If i exceeds
1.6, then the Bogie is smaller than 2.34, which corresponds to a T-rate of - 3 .
In this case, QMP Below Normal triggers earlier than a T-rate of - 3 .
For ] less than 1.4, as S 2 gets larger, the Bogie gets smaller. This is because
large S 2 implies large process variance, which makes an observed deviation more
likely to be significant.
For very small S 2, as you move from 7 = 0 to i = 1, the Bogie increases from
2.6 (T-rate = - 3 . 6 ) to 2.9 (T-rate = -4.2). This is an apparent paradox. The
better the process average, the less cushion the producer gets. This is n o t a
paradox, but an important characteristic of QMP. With QMP we are making an
QMP/USP--A modern approach to statistical quality auditing 369
inference about current quality, not long-run quality. If we have a stable past with
i = 0.2, and we suddenly get a sample index of 2.7, then this is very strong
evidence that the process has changed and is worse than standard. If we have
a stable past with i = 1, and we suddenly get a sample index of 2.7, then the
evidence of change is not as strong as with I = 0.2. The weight we put on the
past data depends on how consistent the past is with the present.
The Bogie contour plots provide the engineer with a manual tool to forecast the
number of defects that will be allowed by the end of a period.
Statistical fitter
Figure 5 illustrates jitter statistical in the T-rate. The expectancies are about 0.1,
so the T-rate jitters every time a defect occurs. The small expectancies are revealed
by the long box plots. Period 8, 1977 was Below Normal for the T-rate, but
normal for QMP.
1977 ~978
~ 2 3 4 5 6 7 8 ~ 2 5 4 5 6 7 8
I. I I, I.
I I
\
I \
"i
F-
(b)
+2
o-----q ~q ~
0
-2
-2.7 - .
v
t i i [ ~ I I 1 I i I I I I L
2 5 4 5 6 7 8 -1 2 3 4 5 6 7 8
Appendix
Q M P formulas
The Q M P formulas are derived in [1 ]. In this appendix, we state the formulas
in the notation o f [1, Section 4.5].
F o r rating period 2 t [t = 1, . . . , T (current period)], the r a w audit d a t a is of the
form:
n t = sample s i z e ,
x t = Defects o b s e r v e d in the audit s a m p l e .
The mean and variance o f x t given s t a n d a r d quality (Est and Vst ) are the same,
because x t is Poisson. So,
x t = Equivalent defects = d e f e c t s ,
e t = Equivalent expentancy = e x p e c t a n c y .
It xt/e t ,
ft et
el
-- , gt
1 + et/4 2.5 + (1.5)e t + (0.22)et2
Corresponding weights:
P, = ft, qt = gt gt"
t t
b = (2 p,/t).
Degrees o f freedom:
df= 2 [ 2 qt(1/e,)] 2 _ 1.
qt2(1/e 3 + 2/e if)
a 2 = E q,(It/e,).
Variance ratio:
R = S2/G 2 .
F, G, and H"
l,
B
F-
B-1
1
a = -
RF
d = bleT.
Sampling variance ratio:
rT = a~.la 2 .
Process variance:
Weights:
07- = ~ T ~ + (1 - & r ) I r .
where
G=(y) = f Yo r(~)
1- - X a- 1 e-X dx = Gamma c.d.f.
References
[1] Hoadley, B. (1981). The Quality Measurement Plan. Bell System Technical J. 60 (2), 215-271.
[2] Hoadley, B. (1981). Empirical Bayes analysis and display of failure rates. In: Proceedings of the
1EEE 31st Electronic Components Conference, May 11-13, 1981, Atlanta, GA, pp. 499-505.
[3] Hoadley, B. (1986). Quality Measurement Plan. Encyclopedia of Statistical Sciences, Vol. 7.
Wiley, New York.
[4] Hoadley, B. (1981). The Universal Sampling Plan. In: 35th Annual Quality Congress Transactions
of ASQC, May 27-29, 1981, San Francisco, CA, pp. 80-87.
[5] Shewhart, W. A. (1958). Nature and Origin of standards of quality. Bell System Technical J. 37
(1), 1-22.
QMP/USP--A modern approach to statistical quality auditing 373
[6] Shewhart, W. A. (1931). Economic Control of Quality of Manufactured Product. Van Nostrand,
New York.
[7] Dodge, H. F. (1928). A method of rating manufactured product. Bell System Technical J. 7,
350-368.
[8] Dodge, H. F. and Torrey, M. N. (1956). A check inspection and demerit rating plan, Indust.
Qual. Control 13 (1), 1-8.
[9] Hillier, F. S. and Lieberman, G. J. (1974). Operations Research. Holden-Day, San Francisco,
CA, Chapter 18.
[10] Buswell, G. and Hoadley, B. (1985). QMP/USP: A modern alternative to MIL-STD-105D.
Naval Logistics Quart. 32 (1), 95-111.
[11] Guthrie, D. Jr. and Johns, M. V. Jr. (1959). Bayes acceptance sampling procedures for large
lots. Ann. Math. Statist. 30, 896-925.
[12] Bell Communications Research (1985). BELLCORE-STD-100 and STD-200 inspection
resource allocation plans. Technical Reference TR-TSY-000016 Issue 1.
[13] Brush, G. G., Guyton, D. A., Hoadley, B., Huston, W. B. and Senior, R. A. (1984). BELL-STD-100:
An inspection resource allocation plan. IEEE Communications Society Global Telecommunications
Conference, November 26-29, 1984, Atlanta, GA.
[14] Guyton, D. A. and Hoadley, B. (1985). BELLCORE-STD-200 system reliability test sampling
plan. In: Proceedings of the Annual Reliability and Maintainability Symposium, January 22-24,
1985, pp. 426-431.
[15] Hoadley, B. (1984). QMP theory and algorithms. Bell Communications Research Released
Technical Memorandum TM-TSY-000238, October 26, 1984 (available from author).
[16] Hoadley, B. (1986). The theory of BELLCORE-STD-100: An inspection resource allocation
plan. In: Transactions of the International Conference on Reliability and Quality Control. North-
Holland, Amsterdam.
[17] Guyton, D. A. and Tang, J. (1986). Reporting Current Quality and Trends: T Plots. In:
Proceedings of the 40th Annual Quality Congress, Anaheim, CA, May 1986.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 | Q
.Ik , J
© Elsevier Science Publishers B.V. (1988) 375-402
1. Introduction
are related to the change. For example, the magnitude of the jump of the mean.
In a way, the classical two-sample and multi-sample problem can be considered
as a special case of the general change point problem described above. The
important difference lies in that in the classical case the possible positions of
change are precisely known in advance, while in the above formulation, the most
important question is to determine these possible positions. Page (1954) first
proposed and studied such a formulation.
One frequently-used formulation of the change point problem is as follows:
Consider
x(t)=#(t)+e(t), 0<t~<l, (1.1)
where x(t) is the observation taken from time t, e(t) is the random error,
E e(t) = 0, and #(t) is an unknown left-continuous and piecewise smooth function.
A point to E (0, 1] satisfying
Miao (1987) obtained the asymptotic distributions of these estimators for some
simple types of change points.
The MLE ,) of the intersection 7 of two regression curves is discussed in
Section 2, and various non-Bayesian estimators about jump change model are
presented in Section 3. Section 4 is devoted to Bayesian methods, and in the last
section the estimates about the positions and the number of change points in large
sample case are discussed.
Some other methods are proposed to study the estimates of change points.
Among them are dynamic program and smooth approximation, to name a few.
plays the role of a constraint under which the parameters are estimated. Let wk,
k = 1. . . . , N, be a given set of positive real numbers, called weights. Set
"c(cQ N
where z(a) is an integer such that t~ ~< c~< t~+ l" Suppose that the unknown c~
belongs to a given set, say A. For convenience, we set
where H z ( m ) is the m x Pl design matrix with rows h'l(tk), 1 ~< k ~< m, and H2(m )
is the (N - m) × P2 design matrix with rows h'2(tk), m x k <~ N,
W = diag(w I . . . . . WN) ,
where /~(z) denotes the unconstrained W L S E of fl and QI(z), Q2(z) are the sums
of first z and the last N - z squares of weighted residuals. Set
Then Q(z) is the total weighted residual sums of squares when z is fixed.
The next step is to find that & c A minimizing Q(a) defined by (2.5):
2.2. Classification of
Let
d d
z(~)--
-d-u
'
(illhl(U))[u = ~ - - - (fl2h2(u))[u
'
=ct (2.8)
du
Table 1
Classification of WLSE
~ ti &= ti
THEOREM 2.1 (Hudson, 1966). (i) I f ~ is of Type One, and ~ ( q , ti+~) , then
A
flj=flj*(i), i= 1,2,
and
= ~*(i)
i.e.
fl*(i)h,(o~*(i)) = fl*(i)h2(~*(i)).
2.3. Determination of
If i is unknown, but we know the joint is of Type One, we can find & as
follows.
For a value of i, fig//*(t~) as before. Find whether or not the curves join at least
one ~*(tj) in the right place, i.e.
where Pj*(i) j = 1, 2, are the local residual sums of squares. If the curves do not
join, or if (2.9) is not satisfied, put
T(i) = oo.
Carry out the computation for all relevant values of i. Finally, choose the critical
value of i for which T(i) is minimized. This procedure is based on the definition
of Type One and the theorem above.
If we know that the joint is of Type Two, i.e. ~ = t; for some i, we can easily
find the remaining parameters since we now have a model which is linear in these
parameters. We get the estimate by solving the problem of least square with the
linear constraint
One way of doing this is to find the unconstrained local least square estimates
Review about estimation of change points 381
fl*(i), and then make the relevant adjustment (see Gallant and Fuller, 1973;
Hudson, 1966). If i is unknown, we have to carry out the above computation for
all possible values of i, and choose the critical value of i for which the overall
residual sum of squares is minimized.
Finally, suppose the joint is of Type Three. That is
Now we assume that et's in (1.1) are normal. The logarithm of the likelihood
function, logL(fl, 0"2, 0.2, c0' can easily be written down. We want to find the
maximizing point of it subjecting to the restriction (2.1). Since the function
logL(fl, 0-2, 0-22, ~) is not differentiable with respect to ~, we consider first to
maximize logL(fl, 0-2, 0"~, ~) for a given ~. We have
logL - sup {logL(fl, 0-1z, a 2, ~), fie R m +p2, a.z > 0, i = 1, 2, B(a)fl = 0}
A
-= logL(fl, 0-1,
^ 2 0"2,
A2 00
N N z N-z "2 ) .
- - - - log27r . . . . log b l 2 ( ~ ) - - log az(~
2 2 2 2
~ A ( 2 l°g °'2(°0 + - -
g(a) = arg rain log ~'22(~) . (2.10)
2
^ 2 ) and ~rz(e
Finally, (2.10) is not different from general MLE except az(e ^2 )
are restricted by (2.1). When A is an infinite set, e.g. an interval in [tl, tu],
complications arise. Robison (1964) decomposed A into a finite number of
disjoint sets
and used a reparametrization of the model based on constraint (2.1) and poly-
nomial functions in order to solve (2.10).
If a 2 = o:2,
2 a priori, then & is the same as ordinary least squares estimates.
where the arguments u l , . . . , u N are ordered, i.e. u 1 < U 2 < . . . < UN, the error
terms el . . . . . eU are independent N(0, a z) and the parameters el, e2,/~l,/~2 and
z are unknown. Let 7 be the abscissa of the intersection of two regression lines
of model (2.11), then it is easy to see that
Further, it is assumed here that u s < 7 < u,+ 1. Thet is to say, the overall regres-
sion function is smooth, only a change in tangent occurs in the two sides of 7.
Important problems to be considered are estimating 7 and making inferences
about ~. Generally, the MLE ~) of ~ can be obtained more easily. In order to
estimate and make inferences for 7, one way is to calculate the distribution of
~. Unfortunately, the explicit form of the distribution of ~ is hard to obtain.
So we turn to the asymptotic distribution of ~. Feder (1975a) proved the
asymptotic normality of ~) under more general conditions that et's are iid.
r.v.'s with mean 0, variance a 2, finite (2 + b)-moment for some b > 0 and the
number of unknown change points is finite, but may be larger than one.
Although ~) is asymptotically normally distributed, an empirical study of the
distribution of ~ suggests that for moderate sample size, the normal approxi-
mation is inadequate. So it is necessary to find some other approximate distribu-
tion which fits better in case the sample size is not large. This was considered
by Hinkley (1969). He discussed the relation between the two maximum likelihood
estimates ~) and } of ~, the first with constraint (2.1) and the second
unconstrained. From asymptotic normality of ~, he deduced the asymptotic
normality and approximate expression for the distribution of ~).
If the two regression lines in the model (2.11) are parallel, especially/~1 = r2 = 0
and c~~ c~2, the MLE ~) of change point ~ can be deduced. An asymptotic
Review about estimation of change points 383
z(~) N
Q(/3, c~) = ~ wk(xk -/3'lhl(tk)) 2 + ~ wk(x~ -/3~hl(t~))2 , (3.1)
k= 1 k= T(ct)+ 1
A t
is the weighted sun of squares. (/3 , "~)' is called a W L S E of/3 and z is
where/3-- (/3'1,/3~)'-
If IV, . 1. for
. . t . 1, , N, then (/~,
^, ~ ) , is the ordinary least squares
estimator.
A WLSE can be calculated in two steps. First fix ~ and find /3(z) such that
In the next step take ~ as the solution which minimises Q(z), i.e.,
where QI(z) and Q2(z) are the first ~ and the last N - ~ weighted residual sums
of squares, respectively. That is,
1 t
-5[trl 2 ( x t - fl, hl(tt,)) 2 + a f 2 Z ( x t - fl2hz(tk))2] , (3.4)
k=l k=z+l
Proceeding along the same lines as (3.3), we can obtain a M L E (/~, blz,
^2
0"2, ~) of (/~, a 2, a~, z) in two steps. The first step is to fix 'c, find /~(~), a,.z,
i = 1, 2, such that
L ( ~ ) = max L(z)
"r~J
or, equivalentiy,
where Sl('c) is the sample variance of the first 'c observations, and $2(~) is that
of the last (N - ~) observations. This procedure extend easily to multi-dimensional
observations and to the case of more than one change point.
Suppose the integers kl, . . . , kq satisfy 0 -= k o < k I < " ' " < kq < kq + ~ = N.
Then n = ( k l , . . . , kq) is called a partition of [1, N ] , and
K~N) = {Tr = (k,, . . . , kq), 0 < k, < ' . . < kq <( N} . (3.6)
= _ _ _ N 1 q+l
logL(X, ~, A) Np c - -- log IA] - ~ ~ trace(A- 1Aj(N)), (3.8)
2 2 j=l
1
~fl - ~ xt, (3.10)
fl-- ~ t=a+l
Let
q+l
A,~ = Z A j ( N ) . (3.11)
j=l
~1 = A,~(N), (3.12)
and
Np Np
logL(u) = s u p IogL(X, u, A) = _ N log IA(N)I C.
A>0 2 2 2
(3.13)
#j##j+l, J = 1, . . . , q ,
logL(lr) = sup {logL(X, lr, A1 . . . . . Aq+~), A~>O, i = 1. . . . . q + 1}
Nq+l
- ~ ~xjloglAj(N)l Np Np c . (3.15)
2j=1 2 2
386 P. R. Krishnaiah, B. Q. Miao
where
o~ = (kj - k j _ 2 ) / U (3.16)
and
ks - k j _ 2 > p .
l o g L ( ~ ) = max l o g L 0 z ) . (3.17)
neKu
i.e.
fi = arg m a x l o g L ( n ) .
~EKq
=1"//~+g, t = 1. . . . . z,
Xt (3.18)
#2 + e,, t = z + l, . . . , N ,
where z is unknown, //1 ~//2, and et, t = 1. . . . , N, are iid. with the c o m m o n
distribution N(0, a2). a 2 is unknown.
Consider the null hypothesis Ho: X, = #2, t = 1, . . . , N, against the alternative
H~:
J'#x, t= 1,..., z,
x, )
l#2, t= z+ 1,...,N,
N
x , = z-2 xt ' x*=(N- z)-2 ~ xt, (3.19)
t=l t=z+l
The standard difference between the observations before and after the change
point is
y~ = ( z ( N - z ) l N ) l / 2 ( 2 ~ - ~ * ) , (3.21)
i.e.
= arg max V(z).
l ~ z~<N-- I
z(N- z)
H('~) = max (2~ - 2 * ) ' W~- 1(5~ - 5 " ) = max H ( z ) . (3.23)
N
i.e.
= arg m a x H ( z ) .
3.4. C u m u l a t i v e s u m estimation ( C U S U M E )
i= 1
P ( x I = 1 ) = ~" 0 ° ' (3.24)
( 0l, i= z+ 1,...,N,
S t = ~ x i, t = 1. . . . . N, (3.25)
i=1
vt = N S , - tS u , (3.26)
Similar estimates are then defined when it is known in advance that 0o < 01,
or 0o ¢ 01 as follows:
= inf{to:
to
Iv/ol/> vii, t = 1, . . . , N } , for 0o4= 01 . (3.29)
388 P.R. Krishnaiah, B. Q. Miao
Under the null hypothesis Ho: (there is no change in probability) versus the
alternative H~: (there is a change in probability at unknown time), the following
statistics Vm
+ , V~ and Vm have the same distributions as the null distributions of
m ( N - m)O+,N_m, m ( N - m)D/n.N_ m and m ( N - m)Om,N_m, respectively, the
multi-dimensional extension of the Kolmogorov-Smirnov two-sample statistics,
where
4. Bayesian estimate
In this section we discuss the change point problem from the Bayesian
view-point. We only discuss the case of jump model, since the treatment of
continuous model is similar. Suppose we have a jump model with at most one
jump point:
where u I < u 2 < "" • < UN, ( ~ 1 ' i l l ) # ( g 2 ' /~2) a n d ~ is unknown. A more general
form is
~10 -}- ~ l l U t l -[- ' ' ' "~ ~lqlUtql -j- ~'t' t = 1. . . . . z,
Xt (4.2)
~ 0C20 + 0~2 lUt 1 -1- q-~2q2Utq2+~t, t=z+l,...,N,
For the sake of simplicity, we only consider the model (4.1). Further, we
assume z obeys some specified prior probability distribution and et's are inde-
pendent and normally distributed.
= N(0, t= 1,...,
~t (N(0, a~),
2 t=z+ l,...,N.
The Bayesian estimators of z are usually defined by (i) the posterior mode of
z or (ii) the value minimizing the expected posterior loss of the quadratic loss
functions (z -/1) 2 with respect to the set J of admissible values of z.
Review about estimation of change points 389
Now we give a more detailed description of the results of Schulze (1982), which
generalizes the results of Smith (1975), Frereira (1975), Holbert and Broemeling
(1977) and Chin Choy and Broemeling (1980). We also mention that Chemoff and
Zacks (1964), Kander and Zacks (1966), Bhattacharyya and Johnson (1968),
Gardner (1969), MacNeill (1971), Sen and Srivastava (1973) and Booth and Smith
(1982) also investigated these problems within a Bayesian framework.
First, Schulze (1982) considered improper prior distribution by assuming:
(i) The parameters 0 - ( ~ , fl), a 2 and z are all independently distributed.
(ii) The parameters 01, 02 are uniformly distributed over E2.
(iii) The variances alz, az2 are independently distributed with improper densities
po(a 2) = (a~) v~ and po(a22) = (a2)% where vl and vz are given integers, for
example, vl = v2 = - 1.
(iv) Specified a priori probabilities p0(z), z e a r, are given.
THEOREM 4.1 (Schulze, 1982). Under (i)-(iv) the prior densities are proportional
to
and the corresponding posterior probabilities px(z) for the change point
Sl(z) and S2(z) denote the residual sums of squares of the least square estimate
Oj, j = 1,2, based on the observations t= 1,..., r and t= z+ 1. . . . . N, re-
spectively.
To define proper prior distribution with respect to the parameters 0j, aj.2, j = 1,
2, Schulze (1982) made another assumption:
(v) Conditional on z = t, the parameters 0j, ~2, j = 1, 2, are independently
distributed as norrnalgamma variables NF(2, Tl(t)) and NF(2, T2(t) ) with param-
eters
where rs(t) >>.3, Gs(t) is the positive definite matrices of order 2. (For definition
of NF(p, T) distribution see Humak (1977, A, 2.38, p491)).
390 P. R. Krishnaiah, B. Q. Miao
THEOREM 4.2 (Schulze, 1982). Under (iv) and (v), prior densities
X;(27) = (X 1. . . . , Xz)' , X~('£) ----( X z + l , " ' ' ' XN)' "
Note that we have to calculate all px(Z) and thus all values Cffx, z), j = 1, 2,
according to previous formula (4.8). But for each fixed z c J , the calculation of
CS(x, z), especially for C2(x, z), is quite complicated. Most of the effort is devoted
in searching for an optimal ~.
Another choice of the estimate @ is defined by minimizing the expected
posterior loss function.
/I = arg min R ( # ) ,
i.e., the estimate is the point in J which is nearest to the posterior expectation of
Review about estimation of change points 391
z. Notice that
It requires to calculate px(Z) and thus CJ(x, z) for all z s J with po(Z)> 0.
Therefore, the two methods presented above are comparable.
In the sequel we consider the multivariate jump change model. Let X(t) be an
independent p-dimensional process on (0, 1] such that
E X(i/N) = #j,
VarX(i/N) = Aj, for k}U_)l < i <~k) N), j = 1,2 . . . . . q + 1, (5.2)
where k(oN) = 0 , &q+IL(N)__--N , and #j ~ ~j+l. The number q of change points may be
assumed known or unknown, but it is known that q is less than a given constant
L. Let
/~(LN) = {7["(N) = ( k ? ) , ..., k~N); 0 < k~N) < " " < k~N) < N, l = 1 , . . . , L}
L
= U K(N)
--q (5.3)
q=l
where K(qN) is defined by (3.6). For simplicity, we omit all the superscript N of
X, k, n etc., for example, k j - k ) u), n - n ( N ) , K q _ K ( N ) q . Further, define
X~ = X ( j / N ) . We must always keep in mind that these qualities are dependent on
N.
By (3.3), for given n = (kl . . . . . kq) e Kq, under model M , , we get
N q~l
p_ ___
2 j = l c~/l°g[Ai(N)l + b 2 ~ G ~ ) + b2" (5.5)
where b I and b 2 are constants independent of rr and Av, and A,~(N), A/(N) and
aj are defined by (3.11), (3.9) and (3.16), respectively.
Then we have
THEOREM 5.1 (Krishnaiah, Miao and Zhao, 1986). (kl/N, .... ]¢q/N) is a
strongly consistent estimate of (t 1. . . . , tq).
Q~) N q + 1
= --- ~ ~tjloglA,(N)l- # ( n ) C ~ ), r t ~ g L. (5.12)
2j=,
A ^
In both cases k 1. . . . . kz can be grouped into sets M1, M2, ..;, by the
follow!ng procedure. Let k, be an element of M,. For k2, if k2 kx < D~,
then k2~M1, otherwise k 2 e M 2, where i = 1, 2 corresponds to cases (i) or
(ii), respectively. Continuing this procedure, we get
Thus, k,, . . . , k h are grouped into sets M 1. . . . . M 4. Here we note that M;,
are all dependent on N. Krishnaiah, Miao, Subramanyam and Zhao (1987) proved
this result:
ON = {k: k = m, m + 1. . . . , U - m, - G u ( j ) > C ~ ) } ,
1
tj = 2N {kj,~ + max(kj, k j e D j N } , j = 1. . . . . 4. (5.23)
THEOREM 5.3 (Krishnaiah, Miao and Zhao, 1987b). (q, t 1, . . . , to) is a strong-
ly consistent estimate o f (q, q , . . . , tq).
Proceeding along the same lines as above, we can obtain a number of results
concerning this estimate• For example, we have
G u ( k ) = 2m log IA,k(N)] + ~-
m log I A 2 k ( N ) - m log rBk(N)[ (5.24)
Var X ( i / U ) = )~j,
kj. L < i < ~ k j , j= 1. . . . . q+l.
and
where Xij is defined by (3.10) and I(A) denotes the indicator of a set A, and A k ( N )
and Bk(N ) are defined by (5.14) and (5.17).
Take m m N and ~ur(l~ to satisfy (5.8). Define DN, Dis, tj, j = 1, . . . , q by
=
Case (ii). The only thing known about 2's is that 2e > 0, i = 1, ..., q. By the
same methodology, we have
THEOREM 5.6. Let m = m N and a positive number C ~ ~ satisfy (5.9). Define DN,
DjN, 2j, j = i, . . . , 0 by (5.19)-(5.23). Then (0, t l . . . . . 2o) obtained from above
procedure is a strongly consistent estimate of (q, q, . . . , tq).
l ~x(k-m" ] ...+X(~-)--
DNk = -- m ~ \ ~ / +
A
Based Aon this theorem, if we choose CN = I h N l o g h N I , pick up those integers
kl, ..., kq, such that [DNk, I < C N , l<~i<<.(t, then with probability one, the
number ~ of such integers tends to q - - t h e number of change points, and
k l / N , . . . . k ~ / N (for such k) tend to the change points t I . . . . . tq.
Chen (1987) considered the case where at most one change is allowed, and f
is a step function:
where a, 0 and to are unknown. In this simple case Chen derived the asymptotic
distribution of the test statistic under the null hypothesis that no change point
exists.
THEOREM 5.8 (Chert, 1987). Suppose that Xl, . . . , x N are iid with a c o m m o n
normal distribution N(a, a2). L e t m = raN, N = 1, . . . , 2 . . . be positive integers such
that
lim m / N = 0 , lim ( l o g N ) 2 / m = O.
N ~ S~oc~
Let
Yk = (2m) 1/2 t 2m
I~i=k+ 1
xi - ~
i=k-m+l
}
xi , k = m, m + 1. . . . , N - m,
This theorem suggests a simple way to test the null hypothesis Ho: no change
point exist. Chen made a fairly accurate estimate of the power of this test, and
also obtained a point estimator and interval of the change point t o. In particular,
he gave a simple formula to determine (approximately) the sample size N and mN
needed in order that the confidence interval of t o has a prescribed confidence
coefficient and prescribed length. H e also considered the non-normal case.
Krishnaiah and Miao (1987) extended Chen's work to the continuous change
point case:
f(t)=~# + i l l ( t - to), 0<t<t o,
( #+fl2(t-to), (5.27)
to<t< 1,
where /t, ill, flz and t o are unknown.
398 P. R. Krishnaiah, B. Q. Miao
x~=a+(k/N)fl+ek, k= 1,...,N,
where el, . . . , t N are iid with distribution N(0, o.2). L e t m = m N be a positive integer
such that
1
Y~-zw/m {( x k - 4 " + l + ' ' " +Xk-3m)--(Xk-3,~+, +'''+xk-2m)
~u = max I Y~I
4m~k~N
1 ( 5 ~ ) 1 }
+sloglog -5 -Elogzr .
Then
^2
In case that o.2 is unknown, it can be replaced by some estimate G ,
without invalidating the above theorem, only that 8.2 must satisfy the
condition
This theorem can be used to test the null hypothesis that no change point
exists, to estimate the power of this test and to get a point and interval estimator
of the change point. We can also get the jump/~1-/~2, in much the same way as
Chen's theorem is applied to the discontinuous case.
References
Bacon, D. W. and Watts, D. G. (1971). Estimating the transition between two intersecting straight
lines. Biometrika 58, 524-534.
Bandemer, H. and Schulze, U. (1985). On estimation in multiphase regression models with several
regressors using prior knowledge. Math. Operationsforsch. Statist. Ser. Statistics 16, 3-13.
Review about estimation of change points 399
Bartholomew, D. J. (1959a). A test of homogeneity for ordered alternatives. Biometrika 46, 35-48.
Bartholomew, D. J. (1959b). A test of homogeneity for ordered alternatives II. Biometrika 46,
328-335.
Bhattacharyya, G. K. and Johnson, R. A. (1968). Non-parametric tests for shift at an unknown time
point. Annals Math. Statist. 39, 1731-1743.
Bissell, A. F. (1969). CUSUM techniques for quality control (with discussion). Applied Statistics 18,
1-30.
Booth, N. B. and Smith, A. F. M. (1982). A Bayesian approach to retrospective identification of
change points. Journal of Econometrics 19, 7-22.
Box, G. E. P. and Tiao, G. C. (1965). A change in level of a non-stationary time series. Biometrika
52, 181-192.
Broemeling, L. D. (1974). Bayesian inferences about a changing sequence of random variables.
Commun. Statist. 3, 243-255.
Brown, R. L. and Durbin, J. (1968). Methods of investigating whether a regression relationship is
constant over time. European Meeting, Selected Statistical Papers, L Mathematisch Centrum,
Amsterdam, 37-45.
Brown, R. L., Durbin, J. and Evans, J. M. (1975). Techniques for testing the constancy of regression
relationships over time, with discussion. J. Royal Statistical Society, Set. B 37, 149-192.
Bunke, H. and Schulze, U. (1983). Approximation of change points in regression models. Proc. First
Tampere Sem. Linear Models, 161-177.
Buse, A. and Lira, L. (1977). Cubic splines as a special ease of restricted least squares. J. Amer.
Statist. Assoc. 72, 64-68.
Chambers, J. M. (1971). Regression updating. J. Amer. Statist. Assoc. 66, 744-748.
Chert, X. R. (1987). Testing and interval estimation in a change-point model allowing at most one
change. Technical Report No. 87-25. Center for Multivariate Analysis, University of Pittsburgh.
Chernoff, H. and Zacks, S. (1964). Estimating the current mean of a normal distribution which is
subjected to changes in time. Annals Math. Statist. 35, 999-1018.
Chin Choy, J. H. and Broemeling, L. D. (1980). Some Bayesian inferences for a changing linear
model. Technometrics 2, 71-78.
Cobb, G. W. (1978). The problem of the Nile: Conditional solution to the change-point problem.
Biometrika 65, 243-251.
Eder, F. X. (1968). Moderne Meflmethoden der Physik, T. 1. V.E.B. Deutscher Verlag der Wissen-
schaften, Berlin.
Ertel, J. E. and Fowlkes, E. B. (1976). Some algorithms for linear spline and piecewise multiple linear
regression. J. Amer. Statist. Assoc. 71, 640-648.
Farley, J. U. and Hinich, M. J. (1970). A test for a shiffing slope coefficient in a linear model. J.
Amer. Statist. Assoc. 65, 1320-1329.
Farley, J. U., Hinich, M. J. and McGuire, T. W. (1975). Some comparisons of tests for a shift in
the slopes of a multivariate linear time series model. J. Econometn'cs 3, 297-318.
Feder, P. I. (1975a). On asymptotic distribution theory in segmented regression problem--Identified
case. Annals Statist. 3, 49-83.
Feder, P. I. (1975b). The log likelihood ratio in segmented regression. Annals Statist. 3, 84-97.
Ferreira, P. E. (1975). A Bayesian Analysis of a switching regression model: Know number of
regimes. J. Amer. Statist. Assoc. 70, 370-374.
Gallant, A. R. and Fuller, W. A. (1973). Fitting segmented polynomial regression model whose join
points have to be estimated. J. Amer. Statist. Assoc. 68, 144-147.
Gardner, L. A. (1969). On detecting change in the mean of normal variates. Annals Math. Statist.
40, 116-126.
Ginsburgh, V., Tishler, A. and Zhag, I. (1980). Alternative estimation methods for two-regime
models: A mathematical programming approach. European Economic Review 13, 207-228.
Goldfeld, S. M. and Quandt, R. E. (1972). Non-linear methods in Econometrics. North-Holland,
Amsterdam.
Goldfeld, S. M. and Quandt, R. E. (1973). A Markov model for switching regression. J. Econometrics
1, 3-16.
400 P. R. Krishnaiah, B. Q. Miao
Page, E. S. (1955). A test for a change in a parameter occurring at an unknown point. Biometrika
42, 523-527.
Page, E. S. (1957). On problems in which a change in parameter occurs at an unknown point.
Biometrika 44, 248-252.
Pettitt, A. N. (1979). A non-parametric approach to the change-point problem. Appl. Statist. 28,
126-135.
Pettitt, A. N. (1980). A simple cumulative sum type statistic for the change-point problem with
zero-one observation. Biornelrika 67, 79-84.
Poirier, D. J. (1973). Piecewise regression using cubic splines. J. Amer. Statist. Assoc. 68, 515-524.
Quandt, R. E. (1958). The estimation of the parameter of a linear regression system obeying two
separate regimes. J. Amer. Statist. Assoc. 53, 873-880.
Quandt, R. E. (1960). Tests of the hypotheses that a linear regression system obeys two separate
regimes. J. Amer. Statist. Assoc. 55, 324.
Quandt, R. E. (1972). A new approach to estimating switching regressions. J. Amer. Statist. Assoc.
67, 306-310.
Quandt, R. E. and Ramsey, . B. (1978). Estimating mixtures of normal distributions and switching
regression. J. Amer. Statist. Assoc. 73, 730-738.
Robison, D. E. (1964). Estimates for the points of intersection of two polynomial regressions. J.
Amer. Statist. Assoc. 59, 214-224.
Schechtman, E. (1982). A non-parametric test for detecting changes in location. Commun. Statist.
A 11, 1475-1482.
Schulze, U. E. (1975). Estimation of the unknown change-point between regression regimes. I K M 7,
199-202.
Schulze, U. E. (1976). Consistency of the Hotelling, Wilks, Roy and PiUai-tests for a general linear
hypothesis in multivariate linear non-normally distributed models. Math. Operations forsch, u.
Statist. 7/1, 11-21.
Schulze, U. E. (1982). Estimation in Segmented Regression: known Number of Regimes. Math.
Operationsforsch. Statist. Ser. Statistics, 13, 295-316.
Schweder, T. (1976). Some optimal methods to detect structural shift or outliers in regression. J.
Amer. Statist. Assoc. 71, 491-501.
Sen, A. K. and Srivastava, M. S. (1973). On multivariate tests for detecting change in mean. Sankhy~
A 35, 173-186.
Sen, A. K. and Srivastava, M. S. (1975). On tests for detecting change in mean. Annals Statist. 3,
98-108.
Sen, A. K. and Srivastava, M. S. (1975). On ests for detecting change in mean when variance is
unknown. Ann. of the Institute of Statistical Mathematics 27, 479-486.
Sen, A. K. and Srivastava, M. S. (1975). Some one-sided tests on change in level. Technometrics 17,
61-64.
Smith, A. F. M. (1975). A Bayesian approach to inference about a change-point in a sequence of
random variables. Biornetrika 62, 407-416.
Sprent, P. (1961). Some hypotheses concerning two-phase regression lines. Biometrics 17, 634-645.
Srivastava, M. S. and Worsley, K. J. (1986). Likelihood ratio tests for a change in the multivariate
normal mean. J. Amer. Statist. Assoc. 81, 199-205.
Tishler, A. and Zang, I. (1981). A new maximum likelihood algorithm for piecewise regression. J.
Arner. Statist. Assoc. 76, 980-987.
Tishler, A. and Zang, I. (1981). A maximum likelihood method for piecewise recession models with
a continuous dependent variable. Applied Statist. 30, 116-124.
Tsurumi, H. (1982). A Bayesian and maximum likelihood analysis of a gradual switching regression
in a simultaneous equation framework. J. Econometncs 19, 165-182.
Vostrikova, L. Ju. (1981). Detecting 'disorder' in multi-dimensional random processes. Soviet Math.
Dokl. 24, 55-59.
Wold, S. (1974). Spline functions in data analysis. Technometrics 16, 1-11.
Worsley, K. J. (1979). On the likelihood ratio test for a shift in location of normal populations. J.
Amer. Statist. Assoc. 74, 365-367.
402 P. R. Krishnaiah, B. Q. Miao
Worsley, K. J. (1983). The power of likelihood ratio and cumulative sum tests for a change in a
binomial probability, Biometrika 70, 455-464.
Yin, Y. Q. (1986). Detection of the number, locations and magnitudes of jumps. Technical Report,
University of Arizona.
Yin, Y. Q. (1986). An effective method to estimate the number and locations of jumps. Technical
Report, University of Arizona.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, VoL 7 ")('~
~,4V
© Elsevier Science Publishers B.V. (1988) 403-425
1. Introduction
Research supported t~y NSERC Canada and EMR Canada grants of M. Csrrg6 at Carleton
University, Ottawa.
403
404 M. Cs6rgd and L. Horvhth
tistics type processes are considered for the nonparametric A M O C problem. Their
asymptotics yields a Gaussian process whose variance is that of a Brownian
The literature of the changepoint problem is quite extensive. This review reflects
only our own point of view. For other surveys we refer to Bhattacharyya (1984),
and Wolfe and Schechtman (1984).
bridge, though it is not a Brownian bridge process as it was assumed to be in the
literature. The asymptotic distributions of several of its functionals are given.
Section 2.4 deals with detecting change in the intensity parameter of a renewal
process. In Chapter 3 we consider the problem of sequential detection of change
in a random sequence.
We let
14(0 = e {xtn~] <~ t} , 6(0 = e +1 t}
and
F(t) = + (1 -
Thus here we are testing for 2 = 1 (no changepoint) versus 2 ~ (0, 1), and under
the null hypothesis we have F = H. For further use we introduce also
Q(y)=inf{t:F(t)>~y}, 0<y< 1,
process (i.e. the mean of X1) is known a priori. Letting S* = 0, S* = y kj=l Vj,
k = 1, . . . , n, and
~ ifXj>0o,
Vj= _ if XJ ~< 0o,
is too large. Special emphasis was given to the choice of a = b = 1, in which case
the Vs are simply the signs associated with the (Xj - 0o). The resulting changepoint
test is nonparametric distribution free over the class of continuous random
variables. Indeed under H(J ), on choosing a and b so that E Vj = 0, j = 1. . . . . n,
where it is not necessary any more that 0o should be EX1, we have E (S*) 2 = nab,
and Donsker's invariance principle (cf. for example Breiman, 1968) yields
{ W(t) - o ~ ,inf
~, W(s),O~t<~l}~={IW(t)l, O~<t~<l}
n~oo 1.0~< t ~ 1
Tn= max
O<~k<~n
pSk-kOol, (2.2)
where W is a Wiener process, and a z = E(X 1 - 0o)z. Hence with any consistent
sequence of estimators {an} for a we get
just like in (2.1). Thus Page's T* and Tn of (2.2) are asymptotically equivalent in
distribution.
When the mean 0o is unknown in (2.2) then estimating it by the sample mean
we obtain
max
O~k~n
PSk- kSn/nl , (2.3)
whose large values would reject H~ol): A = 0 in favour of H~I): A ~ 0. Pettitt (1980a)
also considered the statistic of (2.3) for the changepoint problem with zero--one
observations. Under the null assumption Donsker's invariance principle again
yields
The latter distribution function is well known as the limiting distribution of the
two-sided Kolmogorov-Smirnov statistic (cf. Kolmogorov, 1933), and it has been
widely tabulated. We note also that the aforesaid rationalization and discussion
of (2.2) and (2.3) are valid also for HCo2): E X 1 . . . . . EXn = 0o versus HI2):
Nonparametric methods for ehangepoint problems 407
THEOREM 2.1. Under the null hypothesis H o and on asusming E JXll v< ~ for
some v > 2, there exists a sequence o f Brownian bridges {Bn(t), 0 ~< t ~< 1} such that,
as n--, oo
This theorem can be deduced from the Koml6s, Major and Tusnhdy (1975,
1976) and Major (1976) approximation theorem (cf. Theorem 2.6.3 in Cs0rg6 and
R6v6sz, 1981), using the method of proof of Theorem of Csorg6 and Horvhth
(1986a).
The statement of this result can be summarized by saying that the process
(Zn(t), 0 ~< t ~< 1) has the same asymptotic behaviour as the uniform quantile and
empirical processes. As a consequence of this, corresponding theorems proven in
Cs6rg6, S. CsOrg6, Horvfith and Mason (1986), CsOrg6, and Horvfith
(1985a, b,c), CsOrg6, Horvfith and Steinebach (1987), CsOrg6 and Mason (1985),
and Mason (1985) for these processes hold also true for Z n. In order to illustrate
our point we mention only two corollaries here.
Let Q* be the class of functions q: (0, 1 ) ~ ( 0 , ~ ) which are monotone non-
decreasing near zero and monotone nonincreasing near one, and
inf~t~< l _ ~ q ( t ) > 0 for all b e ( 0 , 1/2). If q~ Q*, we define the integral
1
l(q, c) =
~0 (t(1 - t)- ' exp( - cq2(t)/(t(1 - t))) d t , c > O.
(b) if q~ Q*
@
sup {[Zn(t)l/q(t) } , sup {IB(t)lq(t)}
O<t< 1 O<t< 1
Given Theorem 2.1, the proof of (a) resp. (b) of this corollary is the same as
that of Theorem 4.2.1. resp. Theorem 4.2.3. of Cs0rg6 et al. (1986). We note that
(a) does not imply (b), and mention that (b) holds for example with
1 ,]1/2
q(t)= t(1-t)loglog~j_t) , 0<t<l.
t(1
Thus one arrives at the following modification of (2.3) and (2.4): reject H(o° in
favour of H(1° (i = 1, 2) for large values of
1 \ --
R n = max {ISk-kSn/n]/(nl/Z(~( 1 -~)l°gl°gk/n(1-k/n)) 1/2)}.
l<k<n
(2.6)
with any sequence {%} of consistent estimators of ~r. It seems clear that (2.3) is
more powerful than (2.6) for detecting changes that occur near n/2, and just as
in the case of (2.4), the converse is true for changes occurring near the endpoints
0 and n.
Just like the above asymptotic result for (2.6), the next corollary also relates to
(2.4). Let { V ( s ) , - ~ < s < o v } be an Ornstein-Uhlenbeck process, i.e., a
Gaussian process with mean zero and covariance function e x p ( - I t - sl).
lira P{(2 log log n) ~/2 sup {]Z n(t)[/(t(1 - t)) ~/2} - (2 log log n
n~ 0<t<l
+ ½log log log n - i log re) ~< x} = exp ( - 2 exp ( - x)). (2.8)
410 M. Csrrg6 and L. Horv~th
The distribution function of the right hand side random variable of (2.7) was
tabulated by De Long (1981). Assuming that the X~ have a normal distribution,
(2.7) was extensively studied by Siegmund (1986). The result of (2.8) was proved
by Yao and David (1984) under assuming normality of the X~, and by Haccou
et al. (1985) when the Xe are assumed to be exponential. Corollary 2.2 does not,
of course, require these parametric assumptions. From (2.8) it is clear that the
statistic of (2.4) tends to infinity in probability even if HCoi} (i = 1, 2) are true, and
that the normalizing sequences of (2.8) are crucial for the sake of proving this
asymptotic result.
Fn(t) = n - l # { 1 <~i<<.n:X~<~ t}
denote the empirical distribution function of X1, ..., X~, and let
Q ~ ( y ) = i n f { t : F n ( t ) > ~ y }, 0<y<l,
J" - ( l - t ) if x~<0,
(x)
t if x > 0 ,
THEOREM 2.2. Assuming that H o holds, one can define a sequence of two-
parameter Gaussian processes {F,(s, t), 0 ~<s, t < 1) such that
12 ~l/Z fo 1 (2.14)
to(1 ~ to)/ Yn(s, to) ds 9 N(0, 1),
( 1 2 ~1/2 f01
So(1 _ So)] Y~(so,t) dt 9 N(0, 1), (2.15)
(12)l/a ~ ( i - a) sign(Xi-Qn(1))
and
2 ~ ~ sign(Xi_Qn(½))(Xj_Q,(1))+l(l+~)
//2 1 ~k<~n 1 <~i<j<~k
=4 ;oI Y~(s,l) ds ~,
)
B2(s) ds, (2.17)
where 0 < to, s o < 1, and N(0, 1) stands for the standard normal random variable.
It is easy to see that ~ tp(t, s)F(s, t)dddt is a normal random variable with
expectation zero and variance
THEOREM 2.3. Assuming that H 1 holds and Q is continuous at to e (0, 1), we have
(1 - s ) 1 ~ 2 (t o - H(Q(to))) tf2~<s<l.
For the proof of this result we refer to CsOrg6 and Horvfith (1987a). Applying
Theorem 2.3 we can immediately find conditions for the consistency of tests
based on Corollary 2.3. For example, if there is a t o e ( 0 , 1) such that
H ( Q ( t o ) ) v~ t o and Q is continuous at to, then
It is also clear that if H(t) = G(t - A) and H is strictly monotone, then tests based
on (2.9)-(2.17) are consistent against the alternative A ~ 0.
Next we consider the problem of estimating 2. With 0 < a ~< b < 1 let
and
= inf{s : M . ( s ) = M * } .
we have
A a.s.
lim 2n = 2.
n~oo
First we study C k under the null hypothesis H o and then under the alternative H 1.
Given Ho, X 1. . . . . X, are independent identically distributed r a n d o m variables.
We assume
and let E h ( X , X2) = #, h (t) = E {h(X 1, t) - g}. Condition (2.18) implies that
E h2(X1)< oo, and we assume
Here we investigate
U~ = C ~ - k(n - k ) ~ , l<.k<n.
THEOREM 2.5. Assuming that H o holds, and (2.18), (2.19) are satisfied, one can
define a sequence of Gaussian processes {O,(t), 0 ~< t~< 1} such that, as n ~ o o ,
n - 3/2 (t)
sup Ut(, + 1),1 - O. = %.(1), (2.21)
0~<t< 1 (7
Assuming the existence of more than two moments for h, we also have a
weighted version of (2.21).
Nonparametn'c methods for changepoint problems 415
)
sup { n - a / 2 U[(.+ - O.(t) /(t(1 - t))1/2~ = Op(1),
1)t]
l/(n + l)~t<~n/(n + 1)
(2.24)
and we have (2.22)for each n >1 1.
From Theorem 2.5 it follows immediately that, in general, with any sequence
of consistent estimators {an} for a, we have
Hence testing H o versus H~ can be based on large values of the left hand side
random variable of (2.25).
The test statistic of (2.25) is not sensitive on the tails. Whence the next corollary
of Theorem 2.6 is of interest.
COROLLARY 2.5. Assume that H o holds, and (2.18), (2.19) are satisfied. Then, as
n --~ o 0 ,
(a) if q e Q*
n- 3/2 }
sup U[(.+1),1 - O,(t) /q(t) = oe(1 )
O<t<l G
(b) if q e Q*
n - 3/2 }
sup Ig[(,,+l)tll/q(t ) , sup {[O(t)lq(t)}
0<t< 1 ff 0<t< 1
1 )1/2
t (1 - t) log log t (1 t~ '
416 M. Cs6rg3 and L. Horv[tth
and a result like that following (2.6) holds also here. The variance of O(t) is
t ( l - t). Hence another natural choice of a weight function is ( t ( 1 - t)) 1/2.
However I((t(1 - t)) 1/2, c) = ov for every c > 0, and Corollary 2.5 is not appli-
cable. This case is studied in our next corollary (cf. Theorem 2.3 of CsOrg6 and
Horv~tth, 1986b).
COROLLARY 2.6. Assume that H 0 holds, and (2.18), (2.19) are satisfied. Then
+ (1/2) log log log n - (1/2) log n) ~< x} = exp( - 2 exp( - x)).
COROLLARY 2.7. Assume that H o holds, and (2.18), (2.19) are satisfied. I f q ~ Q*,
then as n ~ ov
lfol
n 3 ~2 { U~(. + o,Jq(t)} dt
)
{ 02(t)/q(t)} dt
The next theorem studies the asymptotic behaviour of C k under the alternative
hypothesis H 1. Let
Then
{ #(°t(2 - t) + t#{3)(1 - 2) , 0<t~<2,
limo~ n-2Ci(~+ l)t] = /./(2)(t- 2) (1 - t) + / . / ( 3 ) J . ( 1 - t), 2~<t<l,
& probability.
THEOREM 2.8. Under the null hypothesis H * there exists a sequence of Brownian
bridges {B,(t),0~< t~< 1) such that
For a proof of this theorem we refer to CsOrg6 and Horv~ith (1987b), where
it is deduced from the Koml6s, Major and Tusnfidy (1975, 1976) approximation.
A similar construction was used by CsOrg6 and Horv~tth (1986a) in the weighted
approximation of the uniform quantile process. Just like in proving Theorem of
the latter paper, here too one could use the Skorohod embedding scheme instead
of the Koml6s, Major and Tusn~tdy approximation, and then we would get
Theorem 2.8 with 0 ~< z < 1/4.
Just like that of Theorem 2.1, the statement of this result can be also summar-
ized by saying that the process {K,(t), 0 ~< t~< 1} has the same asymptotic
behaviour as the uniform quantile and empirical processes. This, in turn, results
in similar corresponding conclusions for K n which are now well known for these
empirical processes. The ones which require new proofs can be found in CsOrg6
and Horvfith (1987b). Here we mention only two corollaries. Let {B(t), 0 ~< t ~< 1}
be a Brownian bridge.
@
sup { l ~ - ' / 2 [ K , ( t ) ] q ( t ) ) , sup { I B ( t ) l / q ( t ) )
O<t< 1 O<t< 1
Here, for example, (b) results in a procedure like that of (2.6) with a similar
corresponding asymptotic distribution for the hinted at test.
Let { V(s), - ~ < s < ~ } be an Ornstein-Uhlenbeck process as in Section 2.1.
Nonparametric methods for changepointproblems 419
lim P{(2 log log n) 1/2 sup { IX- ~/2[K, (t) l/(t(1 - 0) 1/2} - (2 log log n
n~ O<t<l
limP{(81ogn)-l/2{ix-lfolK~(t)/(t(1-t))2dt-21ogn}<~x}=dP(x ),
(2.29)
where • is the standard normal distribution function.
The result of (2.27) was proved in Kendall and Kendall (1980), and those of
(2.28) and (2.29) were found by CsOrg6 and Horv~tth (1987b).
The problem of the consistency of tests which can be based on Corollaries 2.8
and 2.9 is open. Also, let N ( x ) be a renewal process which has jumps at
Sl, S 2. . . . . Assume that X 1 = X l , X 2 = $2 - $1 . . . . . X k = Sk - S k _ 1 are indepen-
dent random variables. Suppose we want to test that {Xk, k >~ 1} are identically
distributed. Do the results of this section hold true under these assumptions, i.e.,
when the waiting times of N(.) are not necessarily exponential random variables?
We conjecture that the answer to this question is yes.
Using these sequentially defined random variables we define their empirical distri-
bution functions
1
ffk(t) = k # { 2 ~ < i ~ < k ' ~ i ~ < t } ' k >~ 2, (3.3)
where m is an initial value, and w and g are positive weight functions. When z
is finite we stop sampling and conclude that X~, X 2. . . . are not identically distri-
buted. The stopping time z of (3.4) is related to tests of power one of Robbins
and Siegmund (1970).
Bhattacharya and Frierson (1981) proved that the random variables 4; are
independent with distribution
P { ~ i : k / i } = 1/i, 0<k<i-1.
Using this observation CsOrg6 and Horv~tth (1986c) proved the next approxima-
tion theorem which plays a crucial role in the sequel.
THEOREM 3.1. Assuming that the null hypothesis holds true, we can define a Kiefer
process {K(t, s), 0 <~ t <~ 1, s >~ 0} such that
A Kiefer process is a two-time parameter Gaussian process with mean zero and
covariance function E K(t, s)K(t', s') = (s ^ s') (t ^ t' - tt'). For further prop-
erties of this process we refer to CsOrg6 and R6v6sz (1981).
Based on this theorem one can easily prove
Nonparametric methods for changepoint problems 421
THEOREM 3.2. Assume that the null hypothesis is true. Let w and g be positive
functions and assume that
Now we return to the changepoint problem. For the probability that we stop
under the null hypothesis we have under the conditions of Theorem 3.2
THEOREM 3.3. Assuming that the null hupothesis is true, we can define a Wiener
process { W(t), t >~ 0} such that, as n ~ 0%
Let
THEOREM 3.4. Assuming that the null hypothesis is true and (3.8) holds, we have
THEOREM 3.5. Assume that the density functions f and g are positive over their
open support, g is bounded and there is an x o such that F(xo) v~ G(xo). Then there
is a positive constant c o such that
Pt sup k l f f k ( t ) - t ] / ( k - N ) > l C o } ~ l ,
10~<t~< 1
Now we can discuss the consistency of our procedure. We assume that the
changepoint N = N ( m ) is such that N ( m ) ~ ~ as m --* oo.
Nonparametric methods for changepoint problems 423
First let g(x) = c(x log(log x v 3)) 1/2, where c is a positive constant. Applying
Theorem 3.5, we can find a positive constant c such that
Roughly speaking, the latter statement says that we will stop either at the initial
value m or in c ( N l o g l o g N ) 1/2 steps after the changepoint N.
Let now g(x) = cx ~, 1/2 < / ~ < 1, where c is a positive constant. Then there
exists a sequence r(N) = r(N, ~) --+0 (N + oo) such that
This amounts to saying that we will either stop at the initial value m or in o(N a)
steps after the changepoint N. If cl(x log(log x v 3)) 1/2 ~< g(x) <~c2 xt3, with con-
stants c 1, c 2 > 0, 1/2 </~ ~< 1, then the consistency of our procedure follows from
(3.9) without any further conditions on m and N. Otherwise, if cxt~<~g(x), ~> 1,
then in order to have consistency, we have to assume some conditions on m and
N.
References
Bhattacharya, P. K. and Frierson, F. Jr. (1981). A nonparametric control chart for detecting small
disorders. Ann. Statist. 9, 544-554.
Bhattacharyya, G. K. (1984). Tests of randomness against trend or serial correlations. In: P.K.
Krisnaiah and P. K. Sen, eds., Handbook of Statistics Vol. 4, Nonparametric Methods, North-Holland,
Amsterdam.
Bhattacharyya, G. K. and Johnson, R. (1968). Nonparametric tests for shifts at an unknown time
point. Ann. Math. Statist. 39, 1731-1743.
Blum, J. R., Kiefer, J. and Rosenbtatt, M. (1961). Distribution free tests of independence based on
the sample distribution function. Ann. Math. Statist. 32, 485-498.
Borell, C. (1975). The Brunn-Minkowski inequality in Gauss space. Invent. Math. 30, 207-216.
Breiman, L. (1968). Probability. Addison-Wesley, Reading, MA.
Chung, K. L. and Williams, R. J. (1983). Introduction to Stochastic Integration. Birkh~iuser, Boston.
Cotterill, D. S. and Cs6rg6, M. (1985). On the limiting distribution of and critical values for the
Hoeffding, Blum, Kiefer, Rosenblatt independence criterion. Statistics & Decisions 3, 1-48.
Cs/3rg6, M., Cs6rg6, S., Horv~th, L., and Mason D. M. (1986). Weighted empirical and quantile
processes. Ann. Probab. 14, 31-85.
Cs6rg6, M. and Horv/tth, L. (1985a). On the distributions of the supremum of weighted quantile
processes. Technical Rep. Ser. Lab. Res. Statist. Probab. Carleton University-University of
Ottawa, Canada, No. 55.
Cs6rg6, M. and Horvfi.th, L. (1985b). On the distributions of L 1 and L 2 norms of weighted uniform
empirical and quantile processes. Technical Rep. Ser. Lab. Res. Statist. Probab. Carleton Univer-
sity-University of Ottawa, Canada, No. 60.
Cs6rg6, M. and Horv~th, L. (1985c). On the distributions of L1 and L 2 norms of weighted quantile
processes. Technical Rep. Ser. Lab. Res. Statist. Probab. Carleton University-University of
Ottawa, Ottawa, Canada, No. 61.
Cs~rg6, M. and Horv~.th, L. (1986a). Approximations of weighted empirical and quantile processes.
Statist. Probab. Lett. 4, 275-280.
CsOrg6, M. and Horv/tth, L. (1986b). Invariance principles for changepoint problems. Technical Rep.
424 M. Cs6rglJ and L. Horv[tth
Ser. Lab. Res. Statist. Probab. Carleton University-University of Ottawa, Ottawa, Canada,
No. 80.
Cs6rg6, M. and Horvfith, L. (1987). Detecting change in a random sequence. J. Multivariate Anal
23, 119-130.
Cs6rg6, M. and Horv~tth, L. (1987a). Nonparametric tests for the changepoint problem. J. Statist.
Planning Inference 17, 1-9.
Cs6rg6, M. and Horvfith, L. (1987b). Asymptotic distributions of pontograms. Math. Proc. Cambridge
Phil. Soc. 101, 131-139.
Cs6rg6, M., Horv~tth, L., and Steinebach, J. (1987). On the distribution of the supremum of weighted
empirical processes. In: W. Sendler, ed., Contributions to Stochastics, Physica, Heidelberg.
Cs6rg6, M. and Mason, D. M. (1985). On the asymptotic distribution of weighted uniform empirical
and quantile processes in the middle and on the tails. Stochastic Process. Appl. 21, 119-132.
Cs6rg6, M. and R6v6sz, P. (1981). Strong Approximations in Probability and Statistics. Academic Press,
New York.
Cs6rg6, S. and Horv~tth, L. (1984). The baboons come down from the trees quite normally. In: W.
Grossman, G. Pflug, I. Vincze and W. Wertz, eds., Proc. of the 4th Pannonian Symp. on Math. Stat.,
Bad Tatzrnannsdorf, Austria 1983. Akad6miai Kiad6, Budapest.
Darkhovskh, B. S. (1976). A nonparametric method for the a posteriori detection of the 'disorder'
time of a sequence of independent random variables. Theory Prob. Appl. 21, 178-183.
DeLong, D. M. (1981). Crossing probabilities for a square root boundary by a Bessel process. Comm.
Statist. A--Theor. Methods 10, 2197-2213.
Feller, W. (1966). An Introduction to Probability Theory and its Applications. Vol. 2. Wiley, New York.
Haccou, P., Mellis, E., and van der Geer, S. (1985). On the likelihood ratio test for a change point
in a sequence of independent exponentially distributed random variables. Centre for Mathematics
and Computer Science, Amsterdam, Netherland, Report MS-R8507.
Huse, V. (1986a). Cram6r-von Mises type statistics for the changepoint problem. Preprint, Carleton
University, Ottawa, Canada.
Huse, V. (1986b). Asymptotic properties for the sequential CUSUM procedure. Preprint, Carleton
University, Ottawa, Canada.
Kendall, D. G. and Kendall, W. S. (1980). Alignments in two-dimensional random sets of points.
Adv. Appl. Prob. 12, 380-424.
Kolmogorov, A. N. (1933). Sulla determinazione empirica di une legge di distribuzione. Giorn. Inst.
Ital. Attuari 4, 83-91.
Koml6s, J., Major, P. and Tusnfidy, G. (1975). An approximation of partial sums of independent
R.V.'s and the sample D.F.I.Z. Wahrscheinlichkeitstheorie verw. Gebiete 32, 111-131.
Koml6s, J., Major, P. and Tusn~tdy, G. (1976). An approximation of partial sums of independent
R.V.'s and the sample D.F.II. Z. Wahrscheinlichkeitstheorie re,re. Gebiete 34, 33-58.
Lombard, F. (1981). Asymptotic distributions of rank statistics in the changepoint problem. South
African Statist. Jour. 15, 129-152.
Lombard, F. (1983). Asymptotic distributions of rank statistics in the change-point problem. South
African Statist. Jour. 17, 83-105.
Lombard, F. (1986). The changepoint problem for angular data: A nonparametric approach.
Technometrics 28, 391-397.
Lombard, F. (1987). Rank tests for changepoint problems. Biometrika. To appear.
Lorden, G. (1971). Procedures for reacting to a change in distribution. Ann. Math. Statist. 42,
1897-1908.
Major, P. (1976). The approximation of partial sums of independent r.v.'s. Z. Wahrscheinlichkeits-
theorie verw. Gebiete 35, 213-220.
Martynov, G. V. (1978). Omega-Square Criteria. Nauka, Moscow (Russian).
Mason, D. M. (1985). The asymptotic distribution of generalized R6nyi statistics. Acta Sci. Math.
(Szeged) 48, 315-323.
Matthews, D. A., Farewell, V. T., and Pyke, R. (1985). Asymptotic score statistic processes and tests
for constant hazard against a change-point alternative. Ann. Statist. 13, 583-591.
McDonald, D. (1985). A CUSUM procedure based on sequential ranks. Technical Rep. Ser. Lab.
Res. Statist. Probab. Carleton University-University of Ottawa, Ottawa, Canada, No. 78.
Nonparametric methods for changepoint problems 425
1. Introduction
The theory of multistate systems has been developed to describe more ade-
quately the performance of systems and their components each of which operates
at more than the usually assumed two levels of performance: 'functioning' or
'failed'. (For a recent survey of the subject see [3].)
In this paper we present some results in the optimal allocation of multistate
components to k series systems so that some performance characteristic like
expected number of systems functioning at level ~ or higher, the probability that
at least one of the systems functions at level c~ or higher, etc., is maximized. Our
basic mathematical tools are majorization and Schur functions; the methods used
and some of the theorems obtained are those of [2]. In addition, we show how
these results may be used to obtain useful applications in reliability theory. The
present paper and reference [2] are the only papers exploiting the elegance and
power of majorization and Schur functions to solve optimal allocation problems
in reliability as far as we know.
Preliminaries
For vector x = ( x 1. . . . . xn), let xtl~>~ . . . >~xEnI denote the decreasing
rearrangement of the coordinates of x. We say vector x majorizes vector y if
k k
Xlil>l~Yfil for k = 1. . . . , n - 1
1 1
and
~ X[q = ~ Y[i]"
1 1
J Research sponsored by the Air Force Office of Scientific Research, Air Force Systems Command,
USAF, under Grant Numbers AFOSR 85-0320 and AFOSR 85-C-0007. The U.S. Government is
authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any
copyright notation thereon.
2 Research supported by the U.S. Army Research Office under Grant DAAL 03-86-K-0094.
427
428 E. El-Neweihi, F. Proschan and J. Sethuraman
THEOREM 2.1. L e t P1. . . . . Ptc be the disjoint min path sets o f a parallel-series
system having path length n 1. . . . . n k. Without loss o f generality, assume that
n~ <<. " " <~ ng. Suppose that there are n = n l + n z + " " " + n k independent com-
ponents with realibilities p~ . . . . . Pn (at time to, say) to be allocated among the path
sets• Then the reliability o f the system (at time to) is m a x i m i z e d when the nx most
reliable components are allocated to P1, the n 2 n e x t most reliable components are
allocated to P2, . . . , and finally, the ng least reliable components are allocated to Pk.
PROOF. Let x,. = log I]j~p, pj, i = 1. . . . . k. The allocation described in the hypo-
thesis maximizes x = (xl, .... xk) in the sense of majorization. The reliability
• k
function of the system, 1 - 1-Ii= 1 ( 1 -- e -xi) is a Schur-convex function of x, and
hence the reliability is maximized by the allocation described. []
The following theorem shows that the allocation described above also maximizes
the expected number of working min path sets which are now viewed as k
separate series systems•
THEOREM 2.2. L e r $1, . . . , S k be k series systems o f sizes n 1 <~ • • • <~ nk, respec-
tively. Suppose that there are n = n I + ' " + n k independent components with
reliabilities Pl, . . . , Pn (at time t o, say) to be allocated among the series systems• Then
the expected number o f working subsystems (at time to) is m a x i m i z e d when the nl
most reliable components are allocated to S~, the n 2 next most reliable components
Optimal allocation of multistate components 429
are allocated to S2, ..., and finally, the n k least reliable components are allocated
toSk.
PROOF. Let x = (x 1, ..., Xk) be as defined in the proof of Theorem 2.1. The
expected number of working systems is Yfi = l e x' which is a Schur-convex function
of x. The result thus follows. []
Note that the optimal allocation in Theorem 2.1 and Theorem 2.2 does not
depend on the actual values of the reliabilities Pl . . . . . Pn, but only on their
ordering.
The above results can be extended to cover the case of multistate components
and systems. Consider a parallel-series system as described in Theorem 2.1,
except that now each component has a common state space S ~ [0, ~ ) ; common
choices for S are {0, 1. . . . , M} and the unit interval [0, 1]. Let X l ( t ) . . . . . Xn(t)
denote the states of the components 1, . . . , n and X ( t ) denote the state of the
system at time t, t >~ 0. Suppose that
for each t >i 0. Theorem 2.3 just below describes the best allocation of com-
ponents to path sets to maximize the system state X ( t ) in the stochastic sense
uniformly in t.
PROOF. Fix a time t > 0 and state c~/> 0. Let P~(t) = P(Xi(t ) >1 a), 1 ~< i ~< n.
These probabilities are in increasing order in i for any fixed t and a in view of
condition (2.1). Since the optimal allocation in Theorem 2.1 depends only on the
ordering among such probabilities, the same allocation maximizes the system state
stochastically, uniformly in t. []
Now let us view the above min path sets as k separate multistate series systems
and call them $1 . . . . . Sk. For ~>~0, let N~(t) be the number of systems
functioning at a level ~ or higher. We have the following theorem.
THEOREM 2.4. Let S 1. . . . , S k and N~(t) be as described above. Then the allo-
cation described in Theorem 2.3 maximizes E N d ( t ) uniformly in ~ and t.
In the results we have considered so far, all the components are tacitly assumed
to be of the same type and thus can be interchanged freely. We now consider a
situation in which we have multistate components of m different types. Let X~(t)
<stX~(t ) < s t . . . <stXj~(t) be the states of k components of type j at time t,
j = 1. . . . . m, t >/0. We wish to build k multistate series systems each containing
one component of each type. Let N~(t) be the number of systems that function
at level e or higher, where e >f 0 corresponds to an arbitrary but fixed state. We
wish to maximize either E ( N ~ ( t ) ) or P(N~(t)>i 1). The following theorem gives
the optimal allocation.
Let x ~ = (x~'. . . . . x~). The assembly described in the hypothesis of the theorem
maximizes x ~ in the sense of majorization. Since both E(N~(t)) and P(N~(t) >1 1)
can be expressed as Schur-convex functions of x ~, the result n o w follows. []
REMARK 2.6. The model discussed just above is related to a binary model given
in [1].
3. Applications in reliability
Note that for each pair (i, t), the distribution of Xe(t) depends on the order in
which we have been using the spares. However, a little reflection shows that the
following order maximizes Xe(t ) stochastically, uniformly in t for each i: Start with
the spare whose parameter is 2~1). U p o n its failure, replace it with the spare whose
parameter is 2~a). . . . , and finally use the spare whose parameter is 2~. Let X,.*(t)
be the r a n d o m state of c o m p o n e n t i corresponding to the above order. Then
clearly X*(t) < ~ t X * ( t ) < s t . . . < s t X , ( t ) for each t. The allocation described in
Theorem 2.3 maximizes stochastically the system state uniformly in t.
Thus, employing the order described above for using the spares corresponding
to each socket, together with the optimal allocation of T h e o r e m 2.3 yields the best
stochastic performance of a parallel-series system formed from the n multistate
components.
REMARK 3.3. In Model 3.1, it is assumed that for each i, T~. . . . , T~t have
exponential distributions with parameters satisfying certain conditions. Other dis-
tributions can be used provided the following conditions hold: (1) For each i, the
r a n d o m variables T~. . . . . T~t are stochastically ordered. (2) For i < j , the smallest
stochastically a m o n g T~. . . . , T~t is stochastically less than the smallest among
T~. . . . . T~t, the second smallest among T~. . . . . T~t is stochastically less than the
second smallest among T~. . . . . T~, . . . . , and finally the largest stochastically
among T~. . . . . T ~ is stochastically less than the largest stochastically among
ZJ~ . . . , T j .
References
[1] Derman, C., Lieberman, G. J. and Ross, S. M. (1972). On optimal assembly of systems. Nay.
Res. Log. Quart. 19, 569-574.
[2] E1-Neweihi, E., Proschan, F. and Sethuraman, J. (1984). Optimal allocation of components in
parallel-series and series-parallel systems. Submitted for publication.
[3] E1-Neweihi, E. and Proschan, F. (1984). Degradable Systems: A survey of multistate system
theory. Comm. Statist. 13(4), 405-432.
P. R. Krishnaiah and C. R. Rao, eds., Handbook of Statistics, Vol. 7 t) m)
© Elsevier Science Publishers B.V. (1988) 433-466
H. Leon Harter
I. Introduction
(1.1)
and its cumulative distribution function (cdf) is given by
and cdf
433
434 H. L. Harter
G(x;O,K)=~og(y)dy=X-exp[-(o)Kl • (1.4)
where u = In 0 is the location parameter (mode) and b = 1/K is the scale parameter.
The corresponding pdf is given by
The integral in equation (1.8) cannot be expressed in closed form, but it can be
evaluated with the aid of a table of the incomplete Gamma function (Pearson,
1922).
Weibull, log-Weibull and Gamma order statistics 435
and cdf
G(x; O,~)= I ~ g ( y ) d y . (1.10)
3o
The integral in equation (1.10), like the one in equation (1.8), cannot be expressed
in closed form, but it can be evaluated with the aid of a table of the incomplete
Gamma function (Pearson, 1922).
Special cases of the Gamma distribution include the exponential distribution
(Gamma with shape parameter ~ = 1) and the Erlang distribution (~ any positive
integer). The two-parameter Gamma distribution with scale parameter 0 = 2 is the
chi-square distribution with 2c~ degrees of freedom. The Gamma distribution is
positively skewed for all finite values of the shape parameter ~, but approaches
normality (and hence symmetry) as ~--* oe. Bain (1983) has written an excellent
encyclopedia article on the Gamma distribution.
i=r\I/
and
1
fr(x ) = er- l(x ) [1 - P(x)]"-rp(x), (1.12)
B(r,n - r+ 1)
where
436 H. L. Harter
and
B(a,b)=
~0 t"-l(1-t)b-ldt, a>0, b>0, (1.13)
are the Beta function and the incomplete Beta function ratio, respectively. The
joint pdff~s(x, y) and the joint cdf F~s(x, y) of the r-th and s-th order statistics x~
and x~ (1 ~< r < s ~< n) of a sample of size n from a distribution with p d f p ( x ) and
cdf P(x) may be written (see David, 1981, pp. 10-11) in the forms
n!
L,(x,y) = pr- l(x)p(x ) [e(y)
( r - 1 ) ! ( s - r - 1)I(n - s)I
- e(x)] . . . . lp(y) [1 - P(y)]'-s (1.15)
and
J n! pi(x)[P(y )
F~,(x,y) = j = , i=r
Z i! ( j - - i)!(n --j).
_ p ( x ) l J - i[ 1 - P(y)]" - : . (1.16)
The results in the preceding paragraph may be used to obtain expressions for
the moments of x r and the product moments of xr and x s. Of particular interest
are the mean and the variance of x~ and the covariance of x~ and x s. The mean
or expected value of x r may be written (see David, 1981, p. 33) as
# :n=f xfr(x) dx
and the variance of x r and the covariance of x r and x s (r > s) may be written (see
David, 1981, p. 34) as
a~:. = (x - # r : . ) 2 f ~ ( x ) d x (1.18)
--o0
and
~rs:n =
f fy
- oo - oo
(X -- [~r:n) (Y - #s:,,)fr,(X,Y) dxdy. (1.19)
Harter (1964a) (see also Harter, 1970, Table C3) tabulated (to 5 DP) the expected
values of Weibull order statistics for n - 1 ( 1 ) 4 0 , r = l(1)n, 0 = 1, and
K = 0.5(0.5)4.0(1.0)8.0, using equation (2.1) for n = 40 and a recurrence relation
g= ~ Wl~Xi (2.3)
i=1
and
= ~ w2ixi (2.4)
i=1
statistics were all equal, ordinary least squares could be used. However, the order
statistics of samples from the Weibull (or any other) distribution have unequal
variances and positive covariances. If some of the order statistics are not available
(because of censoring) or are deliberately not used, their weights in equations (2.3)
and (2.4) are taken to be zero.
Govindarajulu and Joshi (1962) [1968] tabulated the coefficients of the order
statistics in the best linear unbiased estimators of the location parameter c and
the scale parameter 0, based on a sample of size n from a Weibull distribution
with shape parameter K, for K = 1.5(0.5)3(1)8 and n = 2(1)12. Their estimators
are based on all available order statistics from complete samples and from some
types of censored samples. The tabular values are given in the original version
(1962) to 5 DP for n = 2(1)5, 4 D P for n = 6(1)9, and 3 D P for n = 10(1)(12),
but they are useless because of the gross errors in the authors' tables of means,
variances and eovariances of Weibull order statistics mentioned earlier (Subsection
2.1). More accurate coefficients are given (to 3 DP) in the revision (1968) for
complete samples of size n = 2(1)10 from Weibull distributions with shape
parameter K = 3, 5, 8, 10.
If both the shape parameter K and the location parameter c are known, the
scale parameter 0 of a Weibull distribution can be estimated from a single order
statistic. The one-order-statistic estimator is a degenerate case of the best linear
unbiased estimator. If a table of expected values of the order statistics of a sample
of size n from a Weibull distribution with shape parameter K is available one can
easily find an unbiased one-order-statistic estimator of the scale parameter 0 from
the r-th order statistic (r = 1, 2 . . . . , n). If a table of variances of the order
statistics is available, one can also find the best such estimator, that is, the one
for the value of r which gives the estimator with minimum variance. Quayle (1963)
tabulated the coefficients (to 6 DP) in the best one-order-statistic estimators of 0
for K = 0.5(0.5)4.0(1.0)8.0 and n = 1(1)20(2)40 after computing the expected
values of the order statistics from equation (2.1) and their variances from the
equation
(2.6)
Clark (1964) used his tables of expected values, variances and covariances of
Weibull order statistics discussed earlier (Subsection 2.1) to tabulate the
coefficients (to 5 DP) in m-order-statistic estimators of the scale parameter 0 of
a Weibull distribution with known shape and location parameters K and c,
respectively, for K = 0.5(0.25)2.0(0.5)4.0, m = 2(1)n, and n = 2(1)15. His m-order-
statistic estimators are the best linear unbiased estimators based on the first m
order statistics of a sample of size n (all available order statistics of a right-
440 H. L. Harter
censored sample). He compared their efficiency with that of the best one-order-
statistic estimators based on xr, where r ~< m. Musson (1965) tabulated (to 5 DP)
the coefficients in simultaneous best linear unbiased estimators of the location
parameter c and the scale parameter /9, based on two order statistics x r and xs,
r < s ~< m, and on the first m order statistics of a sample of size n from a Weibull
distribution with known shape parameter K, for K = 0.5(0.25)2.0(0.5)4.0,
m = 2(1)n and n = 2(1)15.
Carlton (1965) tabulated the coefficients (to 5 DP) in the best linear unbiased
estimators of the scale parameter 0 of a Weibull distribution with known shape
and location parameters, K and c, respectively, based on l of the first m order
statistics of a sample of size n, for K = 0.5(0.25)2.0(0.5)4.0, m = 2(1)n and
n = 2(1)10, where l is the smallest integer such that the/-order-statistic estimator
has efficiency at least Emi. = 0.80, 0.85, 0.90, 0.95, 0.98, 0.99 relative to the
m-order-statistic estimator. Salisberry (1966) tabulated the coefficients (to 5 DP)
in the simultaneous best linear unbiased estimators of the location and scale
parameters c and 0, respectively, of a Weibull distribution with known shape
parameter K, based on l of the first m order statistics of a sample of size n, for
the same values of K, m and n and the same definition of l as used by Carlton.
Herman (1968) tabulated the coefficients (to 6 DP) in the conditional best linear
unbiased estimators c I 0 and of 0[ c, for the same values of K, m and n, as well
as for n = 11(1)13, and l defined the same as above. He showed that if the same
l order statistics are used for both conditional estimators, the coefficients of the
conditional estimators can be used to obtain those of the simultaneous estimators,
so that the latter need not be tabulated.
They obtained the likelihood equations by equating to zero the partial derivatives
of L = Lr+ 1,m with respect to each of the three parameters, which are given by
m
1
8L/OO = - K ( m - r)/O + K ~ (x i __ c)K/OK+ I + K ( n - r n ) (xm - c)K/o K+
i=r+ 1
+ (n - m ) K O K ( x m - c ) K - 1 - Kr(xr+ l - c) K - 1
e x p [ - (Xr+l-c)K/OK]/OK{ 1 -- e x p [ - (xr+ l -C)K/oK]}. (2.10)
Harter and Moore (1965b) (see also Harter, 1970, p. 104) described an iterative
procedure for solving the likelihood equations, and thus obtaining the maximum-
likelihood estimates 0,/( and ~ of the parameters. A listing of the FORTRAN program
is available from the author. Zanakis (1977, 1979) compared this algorithm with
various other giving maximum-likelihood estimates for the parameters of the three-
parameter Weibull distribution.
Only a few authors have given the information matrix for maximum-likelihood
estimators of the parameters of the Weibull distribution. Dubey (1963) and Ravenis
(1964) gave it for complete samples from the three-parameter Weibull distribution.
Cohen (1965) gave it for complete, singly censored, and progressively censored
samples from the two-parameter Weibull distribution. Hatter and Moore (1967b)
(see also Harter, 1970, pp. 106, 108) gave the information matrix for doubly censored
samples from the three-parameter Weibull distribution, inverted it to obtain expres-
sions for the asymptotic variances and covariances of 0,/( and ~, and tabulated the
442 H. L. Harter
based on the first m order statistics of a sample of size n (m ~< n). Moore and Harter
(1965, 1966) (see also Harter, 1970, pp. 75-76) found the one-order-statistic
estimator
0r=Xr,,
/{r--i [ 1 / ( n - j )]}I/K, r = 1,2 . . . . . n. (2.12)
J
Both of these estimators are biased except for K = 1 (the exponential distribution).
order statistics tabulated by Harter (1964a) can easily be used to tabulate nearly best
linear unbiased estimators up through n = 40. Theory and tables for nearly best linear
unbiased estimators from samples of size up to 40 have been given by Stump (1968)
(simultaneous estimators) and R. I. Davis (1969) (conditional estimators). Niles
(1970) and Elkins (1970) have given similar results for best (simultaneous) and nearly
best (conditional) linear invariant estimators (see Subsection 3.2.3). (If the shape
parameter K is unknown, it must be estimated first; for one method, see Murthy
(1968).)
For n > 40, one must either use a different method, say maximum likelihood
(Subsection 2.2.2) or the method of moments (not discussed here because it does not
involve order statistics), or transform to the log-Weibull distribution (first asymptotic
distribution of smallest values), estimate the parameters (see Section 3.2), and
transform back to the Weibull distribution.
II/K
r(n) ~ xK . (2.13)
O- r(n + l/K) I r= , ~'"
Moore and Harter (1966) (see also Harter, 1970, p. 77) have obtained exact upper
and lower confidence bounds, based on one order statistic, in addition to the point
estimator discussed earlier (Subsection 2.2.2), for the scale parameter 0 of a
two-parameter Weibull distribution with known shape parameter K, by raising the
corresponding results for the one-parameter exponential distribution to the 1/K
power. Since this method makes use of tables for the exponential distribution (see
Harter, 1964c; or Hatter, 1970, pp. 580-588), which are identical with Quayle's
tables for K = 1, it requires less extensive (and more readily accessible) tables than
Quayle's method. The two methods give identical results (to within rounding errors).
444 H. L. Harter
H arter and Moore^(1965a) (see also Harter, 1970, p. 57) pointed out that if one
sets 0 = 0,,n, where 0ran, given by equation (2.11), is amaximum-likelihood estimator
of the scale parameter 0 of a two-parameter Weibull distribution with known shape
parameter K, based on the first m order statistics of a sample of size n, then 2m t~K/0 K
has a chi-square distribution with 2m degrees of freedom. They used this fact to show
that an upper confidence bound with confidence level 1 - P (lower confidence bound
with confidence level P) on 0 is given by
where the first subscript on Z2 is the number of degrees of freedom and the second
is the cumulative probability. If the sample values (first m order statistics of a sample
of size n) are known, numerical values of the confidence bounds can be found with
the aid of a table of percentage points of the chi-square distribution. Harter (1964d)
gave such a table (to 6 SF) for degrees of freedom v = 1(1)150(2)330 and cumulative
probability P--- 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1(0.1)0.9, 0.95,
0.975, 0.99, 0.995, 0.999, 0.9995, 0.9999.
Harter and Moore (1967b) (see also Harter, 1970, p. 132) discussed interval
estimation of the parameters of the three-parameter Weibull distribution, using the
maximum-likelihood point estimates obtained by iterative solution of the likelihood
equations (equations (2.8)-(2.10)) and the asymptotic variances and covariances
tabulated by Harter and Moore (1967b) (see also Harter, 1970, Table F3). If
estimation is regular (see Harter (1971) for a discussion of regularity conditions), so
that the maximum-likelihood estimator of a single parameter is normally distributed,
and if the sample size is large enough so that the asymptotic theory is valid, the
asymptotic variances can be used to obtain confidence bounds for individual
parameters. When the location (threshold) parameter c is one of the parameters being
estimated, estimation is non-regular except when certain restrictive conditions are
satisfied, and even under these conditions Monte Carlo studies have shown that the
asymptotic theory is valid only for extremely large samples.
Harter (1970, p. 132), also discussed a method of finding confidence ellipsoids for
sets of parameters. Let p be the number of unknown parameters. Let V (p x p) be
the asymptotic variance-covariance matrix, and let the maximum-likelihood estima-
tors of the parameters be represented by the p components of the vector Y. Then,
if estimation is regular, the asymptotic joint distribution of the estimators is p-variate
normal and is given by
f ( Y ) = e x p ( - ~1 y , V - 1
Y)l(2rc)PlZl V1112. (2.15)
If the sample size is large enough so that the asymptotic theory is valid, one can use
equation (2.15), together with the tabulated asymptotic variances and covariances,
to obtain p-dimensional confidence ellipsoids for the set of p parameters.
Weibull, log-Weibull and Gamma order statistics 445
asymptotic distribution of largest values, which is the negative of the m-th smallest
value (m-th order statistic) in a sample of size n from the first asymptotic distribution
of smallest values (log-Weibull distribution). Lieblein (1953) developed explicit
closed formulas (in terms of tabulated functions) for the variances and covariances
of the order statistics from the first extreme-value distribution. Lieblein (1954) (see
also Lieblein and Zelen, 1956) tabulated (to 7 or 8 DP) the means, variances and
covariances of the order statistics for n = 1(1)6. White (1964) tabulated the expected
values (7 DP) and the variances and covariances (8 DP) of all order statistics of
samples of size n = 1(1)20 from the first asymptotic distribution of smallest values
(log-Weibull distribution). Mann (I 965) independently tabulated the expected values
(7 DP) for n = 1(1)25 and the variances and covariances (7 DP) for n = 1(1)20 (later
extended through n = 25--see Mann, 1968a). White's and Mann's expected values
and variances agree to within a unit in the seventh decimal place, as do their
covariances for n ~< 14. There are, however, serious disagreements in their
covariances for larger values of n, the maximum discrepancies ( x 10 - 7) being 2, 8,
34, 148, 711 and 2067 for n = 15, 16, 17, 18, 19 and 20. The author believes that
Mann's values, which have smooth differences, are much more accurate than
White's, which do not. White (1967) tabulated the expected values and variances
(both to 7 DP) for n = 1(1)50(5)100, using multiple-precision arithmetic. For
n = 21(1)25, his results agree with Mann's to within 10 -7. So far as the author is
aware, the covariances have not been tabulated for n > 25.
U = ~ w3iYi (3.1)
i=1
and
= ~ w4iYi (3.2)
i=1
can be found by the method of generalized least squares or by the method of Lagrange
multipliers. As in the case of the Weibull distribution, linear estimators found by
ordinary (or even weighted) least squares are unbiased but not best (in the sense of
having minimum variances), since the log-Weibull order statistics have unequal
Weibull, log-Weibull and Gamma order statistics 447
variances and positive covariances. If some of the order statistics are not available
(because of censoring) or are deliberately not used, their weights in equations (3.1)
and (3.2) are taken to be zero.
Lieblein and Zelen (1956) tabulated (to 7 D P ) the coefficients o f the best linear
unbiased estimators, based on the first m order statistics of a sample o f size n, where
m = 2 (1)n, n = 2 (1)6, for the parameters u and b of a log-Weibull distribution. White
(1964) extended the table of coefficients up through n = 20, and M a n n (1965)
independently did the same. White's and M a n n ' s coefficients agree to within 1 0 - 7
for n ~< 12 and to within 2 x 1 0 - 7 for n = 13, except for the following misprints in
White's table: A(2, 4, 9) = - 0.1989815, not - 0.9898152, and the values for J = 2
and J = 3 are interchanged in A ( I , J, 13) and B ( I , J, 13), I = 1, 2. As in the case o f
the covariances o f the order statistics, which are used in calculating the coefficients,
there are serious disagreements in their coefficients for n > 13, the maximum
discrepancies ( × 10 - 7 ) being 15, 74, 400, 1958 and 8914 for n = 14, 15, 16, 17 and
18, and even larger for n = 19 and 20. Again, the author believes that M a n n ' s results,
which have smooth differences, are much more accurate than White's, which do not.
Nevertheless, estimators using White's coefficients are unbiased and their variances
are only slightly larger than those of estimators using more accurate coefficients, since
the variances (and covariances) o f linear unbiased estimators are not very sensitive
to moderate changes in the coefficients, so long as the estimators remain unbiased.
Both White and M a n n tabulated the variances and covariances of their estimators,
which agree to within 10- 6 for n ~< 19, with a maximum discrepancy o f 6 x 10 - 6 for
n = 20.
Richardson (1966) tabulated (to 5 D P ) the coefficients in the best simultaneous
linear unbiased estimators o f the parameters of the first asymptotic distribution of
smallest values (log-Weibull distribution), based on l out o f the first m order statistics
o f a sample of size n, for n = 3(1)20, m = 2(1)n, and l = 2(1)1s, where ls is the
smallest value of l that results in an effÉciency of at least 999/o relative to the
m-order-statistic estimator tabulated by White (1964) and M a n n (1965). Shelnutt
(1966) (see also Shelnutt, Moore and Hatter, 1973) tabulated (to 5 D P ) the
coefficients in the best conditional linear estimators of the scale parameter b when
the location parameter u is known, based on l of the first m order statistics of a sample
of size n, for n = 1(1)20, m = l(1)n, l = m and l = 2(1)I c, where lc ( ~ l s ) is the
smallest value of l that results in an efficiency o f at least 99~o relative to the
m-order-statistic estimator. Elwer (1967) tabulated (to 5 D P ) the coefficients in the
best conditional linear unbiased estimators of the scale parameter b when the location
parameter u is known, based on lm of the ( m 2 -- m I + I) available order statistics,
Xm,, Xm, + ~. . . . . Xm~ o f a complete, singly censored or doubly censored sample of
size n, f o r n = 1(1)20, ml ~> 1, m 2 ~< n , l m = m 2 - m 1 + 1 a n d l m = l(1)/m¢,where line
is the smallest value o f lm that results in an efficiency of at least 999/0 relative to the
( m 2 - m I + 1)-order-statistic estimator. Fratzke (1967) tabulated (to 5 D P ) the
coefficients in the best simultaneous linear unbiased estimators o f u and b, based on
l o f the m = n - r I - r 2 available order statistics xrl + 1, Xrl + 2 . . . . , x . _ r : of a
complete, singly censored, or doubly censored sample o f size n, for n = 2(1)18,
r 1 >1 O, r 2 >i 0 and l = 2(1)/ms, where lm~ is the smallest value of l that results in an
448 H. L. Hatter
n -- r 2 n -- r 2
- r l z r , + l f ( z r , + ~ ) l b F ( z r l + 1)
with the asymptotic variances and with the mean square errors of the best linear
invariant estimators (to be discussed in Subsection 3.2.3).
Harter and Moore (1967a) (see also Harter, 1970, pp. 59-60) derived the
maximum-likelihood estimator of the location parameter u, given the scale
parameter b, based on the first m order statistics of a random sample of size n
from a log-Weibull distribution, by transforming the corresponding estimator for
the Weibull distribution. They made use of the fact that if the random variable
X has the two-parameter Weibull distribution with scale parameter 0 and shape
parameter K, then the random variable Y = lnX has the log-Weibull distribution
450 H. L. Hatter
Moore and Harter (1966) (see also Harter, 1970, pp. 79-80) also found the
one-order statistic estimator
Mann (1963, 1965) proposed the use of best linear invariant estimators, which have
minimum mean square error in the class of linear estimators, and can be obtained
from the best linear unbiased estimators by a simple linear transformation. Mann
(1967) tabulated (to 6 DP) the coefficients in the best linear invariant estimators of
the location parameter u and the scale parameter b, based on the first m order
statistics of a sample of size n from the log-Weibull distribution, for m = 2(1)n and
n = 2(1)25. Studies by Harter and Moore (1968) (see also Hatter, 1970, pp. 128-129
and 802-805) and by Mann (1968b) have shown that if one wishes to minimize the
mean square errors of the parameter estimators, best linear invariant estimators are
competitive with maximum-likelihood estimators and superior to others, including
best linear unbiased estimators. For sample size n > 25, the largest value for which
tables of coefficients in the best linear invariant estimators are available, Mann (1967)
suggested an estimation procedure based on randomly dividing the sample into
subsamples of 25 or less, and then finding a weighted average of the best linear
invariant estimators for the subsamples. This method is analogous to a method first
proposed by Lieblein (1954) for best linear unbiased estimators when n > 6.
For large samples, Ogawa (1951) proposed asymptotically best linear estimators,
which are based on optimum quantiles, and are sometimes called quantile estimators.
Assume that one has an ordered sample of size n, where n is large. Consider k real
numbers 2; (i= 1 , 2 , . . . , k ) such that 0 = 2 o < 2 ~ < 2 2 < . ' ' < 2 k < 2 k + l = 1,
where k is an integer < n. Let the selected sample quantiles be the order statistics
Y-I, Y-2, . . . . Ynk, where ni = [n2;] + 1 (i = 1, 2, ..., k) and [n2i] is the greatest
integer not exceeding n2 i. Hassanein (1969) tabulated the optimum spacings 2,
Weibull, log-Weibull and Gamma order statistics 451
coefficients, variances, covariances and efficiencies (to 4 DP, except that the 2; are
given to only 2 DP for k > 2) of the asymptotically best linear estimators, based on
k = 2(1)4 quantiles, of the parameters u and b of the first asymptotic distribution of
smallest values (log-Weibull distribution). Chan and Kabir (1969) extended the
theory and tables (to 4 DP throughout) to the case of singly and doubly censored
samples, with k = 2(1)5, and the case of complete samples with k = 1(1)10. Chan
and Mead (1971) compared the efficiencies of the best linear unbiased estimators,
based on k = 1(1)4 order statistics from complete samples of size n = 10(5)20(1)25,
with those of the corresponding asymptotically best linear estimators based on ranks
n~ = [n2~] + 1 and n,.' = [n2i], i = 1. . . . . k. The efficiencies are consistently lower for
n" than for n~. Further discussion of the problem of how best to define the 2 quantile
of a sample of size n has been given by Harter (1971).
White (1967) showed how to use his table of expected values of log-Weibull order
statistics and ordinary least squares to obtain nearly best linear unbiased estimators
of the parameters u and b, based on the first k order statistics of a sample of size n.
He also showed how to obtain still better estimators by using his tables of expected
values and variances oflog-Weibull order statistics and weighted least squares. These
methods are useful for 25 < n ~< 100, where tables of the covariances of log-Weibull
order statistics are unavailable, and hence best linear unbiased estimators cannot be
found. Theory and tables for nearly best linear unbiased estimators from samples of
size up to 40 have been given by Eickrnann (1968) and Widenhouse (1970) (simul-
taneous estimators) and by Kingcaid (1969) (conditional estimators).
Mann (1968b) discussed several other methods of parameter estimation, including
the method of moments, graphical techniques (plotting on probability paper),
asymptotically efficient linear estimators (see Bennett, 1952; Jung, 1955), linearized
maximum likelihood estimators (see Plackett, 1958), and other linear estimators
proposed by McCool (1965) and by Downton (1966).
The reliable life t R (the mission time for which the reliability is R) is given by
tR = O [ l n ( 1 / R ) ] b (3.12)
xR = u + b i n [ I n ( l / R ) ] . (3.13)
Mann (1965) tabulated values for obtaining a 1 - ~ level lower confidence bound for
tR, based on the first and n-th order statistics of a sample of size n from the
two-parameter Weibull distribution for n = 7, 10, and 20, ~ = 0.10 and 0.50 and for
n = 5, ~ = 0.10. She also tabulated LC(~), the ~ quantile of the distribution of
b l n ( t r / q ) , where t I and tr are the first and r-th order statistics of a sample of size n,
r ~< m, from a two-parameter Weibull distribution, for ~ -- 0.01, 0.05, 0.10, 0.20, 0.50,
0.80, 0.90, 0.95 and n = 2(1)20. An upper bound for b can be obtained by evaluating
the natural logarithm of the ratio of two failure times and then dividing by a tabled
value of LC(~). Mann (1967) tabulated values of r, s and LC for obtaining most
efficient exact confidence bounds of the form (In ts - In tr)/LC, on the parameter b,
based on two of the first m o f n Weibull order statistics, for n --- 2(1)20, m = 2(1)n,
and confidence levels 0.99, 0.95 and 0.90. Mann (1968a) tabulated values for
obtaining a 90~o lower confidence bound, based on two or three of the first order
m statistics of a sample of size n, on the 100(1 - R) percent point of a Weibull
distribution, for n = 2(1)12, m = 2(1)n, and R = 0.90, 0.95. Mann (1968b) sum-
marized her earlier results on exact small-sample confidence bounds on t n , x R a n d
b, and also discussed large-sample confidence bounds on x n a n d b.
Kimball (1946, 1959) considered four methods of finding a confidence region for
the parameters of the first asymptotic distribution of largest values, one based on
statistical estimation functions that he developed and the others on maximum-
likelihood point estimators. His methods apply equally to the first asymptotic
distribution of smallest values (log-Weibull distribution). His confidence intervals,
like his point estimators, are based on complete samples, and it is not necessary to
order the observations, but again this becomes necessary when the samples are
censored.
Since estimation is regular for the first asymptotic distribution of smallest values
(log-Weibull distribution), the maximum-likelihood estimators of the parameters u
and b from complete or censored samples are asymptotically bivariate normal, with
asymptotic variances and covariances as tabulated by Harter and Moore (1968) (see
also Harter, 1970, p. 782). Hence the asymptotic variances can be used to set
confidence bounds on u and b separately, and the asymptotic variances and
covariances can be used to find elliptic joint confidence regions on u and b.
Harter and Moore (1965a) (see also Harter, 1970, pp. 61-62) have shown that if
l~lmn I b is the maximum-likelihood estimator of the location parameter u when the scale
parameter b is known, based on the first m order statistics of a sample of size n, then
and hence an upper confidence bound with confidence level 1 - P (lower confidence
bound with confidence level P) on u is given by
U l - - e = Up = b ln(2m/z~m,p) + ~ l m n l b , (3.15)
where the first subscript on Z2 is the number of degrees of freedom and the second
is the cumulative probability. Moore and Harter (1966) (see also Harter, 1970, p. 80)
have shown that an exact central confidence interval on u, when b is known, is given
by
b nDtm + Ym < u < b lnDum + Ym , (3.16)
where Ym is the m-th order statistic of a sample of size n and where Dim and Dum are
the coefficients in the corresponding confidence bounds on the parameter of an
exponential distribution, tabulated by Harter (1964c) (see also Harter, 1970,
pp. 580-588) for n = 1(1)20(2)40 and the value(s) of m which maximize the
efficiency of the upper confidence bound and/or that of the central confidence
interval.
from equation (1.8) for P(x) in equation (1.12), and then substituting fr(x) from
equation (1.12) and #r:n from equation (1.17) in equation (1.18). Similarly, an
expression for the covariance of the r-th and s-th order statistics of a sample of size n
may be obtained by substituting F(x; 0, ~, c) and F(y; O, ~, c) from equation (1.8)
in equation (1.15), and then substituting #~:n and /~s:n from equation (1.17) in
equation (1.19). As in the case of the expected values, it is sufficient to tabulate the
variances and covariances of the reduced order statistics (those for c = 0 and 0 = 1),
since the variances and covariances are independent of c, and since the variances and
covariances for any other value of 0 can be obtained by multiplying by 02.
The above expressions for the means, variances, and covariances of Gamma order
statistics cannot, in general, be written in closed form. Gupta (1960, 1962), Karns
(1963), and Breiter and Krishnaiah (1967, 1968) developed computational formulas
(involving numerical integration) for the moments of Gamma order statistics. Gupta
used his formulas (valid only when 0 is an integer) to tabulate the first four moments
of the r-th order statistic of a sample of size n for r = l(1)n, n = 1(1)10 as well as
r = 1, n = 11(1)15 when ~ = 1(1)5. Although the results are given in his 1960 paper
to six digits, some of them are in error by as much as 6 units in the fifth digit, probably
because of his reliance on a recurrence relation with altemating signs, which can
result in severe loss of accuracy due to heavy cancellation. In his 1962 paper the
results have been rounded to three decimal places (a maximum of four significant
digits), perhaps indicating that he had some doubt as to the accuracy in the fourth
and fifth decimal places of the table in the earlier paper. Harter (1964a) used Kams'
expression, which is valid for nonintegral values of 0, to tabulate the expected values
of Gamma order statistics for r = l(1)n, n = 1(1)40, and ~ = 0.5(0.5)4.0. The results
were given to five decimal places, but Breiter and Krisnaiah found errors as large as
two units in the fifth decimal place when r = 1, ~ = 0.5 for n ~< 16. Harter recalculated
his tabular values for r = 1, ~ = 0.5, n = 1(1)40. He verified the results of Breiter and
Krisnaiah for n ~< 16, corrected the errors they discovered, and made even larger
corrections (up to 5 units in the fifth decimal place) for n > 16. The corrected tables
were reprinted by Harter (1970, Table C4).
In connection with work on their AFIT Masters theses, Bruce (1964) and Hill (1965)
independently computed, by different methods, the expected values, variances and
covariances of Gamma order statistics for samples of size n = 1(1)15 when
-- 1(1)6. Their results agree to 8 decimal places. Unfortunately, neither included
these tables in his thesis. Breiter and Krishnaiah (1967, 1968) tabulated the first four
moments about the origin, from which one can readily obtain the variances, of
Gamma order statistics, for ~ = 0.5(1.0)5.5, 6.0(1.0)10.0 and n = 1(1)16 in the 1967
report [~ = 0.5(1.0)10.5 and n = 1(1)9 in the 1968 journal article]. Krishnaiah and
Sen (1984) recently reprinted the 1968 tables. The author is not aware of any
published tables of covariances of Gamma order statistics.
Simple closed form expressions are known for the means, variances and
covariances of the order statistics of the exponential distribution, which is a special
case of both the Weibull distribution and the Gamma distribution, with shape
parameter K and ~, respectively, equal to 1. For the standard one-parameter
Weibull, log-Weibull and G a m m a order statistics 455
David (1981) gave the following formulas for the mean #r:n and the variance a~:n of
the r-th order statistic and the covariance ~r~s:, of the r-th and s-th order statistics
(r < s) (see Epstein and Sobel (1953) for a derivation):
Harter (1964a) (see also Harter, 1970, Table C2) tabulated the expected values
(means) to 5 D P for n = 1(1)120, r = l(1)n from equation (4.2). Similar tables of
variances and covariances could easily be computed by use of equation (4.3). Sarhan
and Greenberg (1958) (see also Greenberg and Sarhan, 1962, Table l lA.2) com-
puted such tables (to 8 DP) for n = 1(1)10, r = l(1)n.
a = ~ wsix i (4.4)
i=1
and
= ~ W6iXi (4.5)
i=1
can be found by the method of generalized least squares or by the method of Lagrange
multipliers. If the covariances of the order statistics were all zero, weighted least
squares could be used; if, in addition, the variances of the order statistics were all
equal, ordinary least squares could be used. However, as in the case of the Weibull
distribution, the order statistics of samples from the Gamma distribution have
456 H. L. Harter
unequal variances and positive covariances. If some of the order statistics are not
available (because of censoring) or are deliberately not used, their weights in
equations (4.4) and (4.5) are taken to be zero.
Karns (1963) used his computational formulas for the first two moments of
Gamma order statistics to tabulate best linear unbiased estimators (which are also
maximum likelihood estimators) of the scale parameter 0 of a two-parameter Gamma
distribution with known shape parameter ~, based on one order statistic of a sample
of size n = 1(1)50 for ~ = 0.5(0.5)4.0. He tabulated the coefficients to 6 DP, the
variances of the estimators to 5 DP and their percentage efficiencies to 2 DP. Bruce
(1964) used his unpublished tables of expected values, variances and covariances of
Gamma order statistics to tabulate the best linear unbiased estimators of the scale
parameter 0 of a two-parameter Gamma distribution with known shape parameter
~, based on the first m order statistics of a sample of size n, for ~ -- 1(1)3, n = 1(1)20
and ~ = 4(1)6, n = 1(1)15 with m = l(1)n. He tabulated both the coefficients and the
variances of the m-order-statistic estimators to 5 DP. He also tabulated the
percentage efficiencies (to 2 DP) for the one-order-statistic estimators relative to (1)
the m-order-statistic estimators and (2) the maximum likelihood estimators for the
complete sample. Hill (1965) used his unpublished tables of expected values,
variances and covariances of Gamma order statistics (which agree with Bruce's to
8 DP) to tabulate the best linear unbiased estimators of the scale parameter 0 of a
two-parameter Gamma distribution with known shape parameter ~, based on l of the
firstm order statistics of a sample of size n, for ~ = 1(1)6, n -- 1(1)15, m = l(1)n, and
l equal to the smallest number of order statistics required to attain a desired efficiency
(0.99, 0.98, 0.95, 0.90, 0.85, or 0.80) relative to the m-order-statistic estimators. He
tabulated the coefficients to 5 DP and the relative efficiencies to 3 DP.
Musson (1965) tabulated (to 5 DP) the coefficients in simultaneous best linear
unbiased estimators of the location parameter c and the scale parameter 0, based on
two order statistics x r and xs, r < s ~< m, and on the first m order statistics of a sample
of size n from a Gamma distribution with known shape parameter ~, for ~ = 1(1)6,
m = 2(1)n and n -- 2(1)15. Salisberry (1966) tabulated (to 5 DP) the coefficients in
simultaneous best linear unbiased estimators of the location and scale parameters,
based on l of the first m order statistics of a sample of size n from a Gamma
distribution with known shape parameter ~, for ~ = 1(1)6, m = 3(1)n, n = 3(1)15,
and l defined as by Hill. He tabulated the actual relative efficiencies to 3 DP.
For the one-parameter exponential distribution, a special case (~ = 1) of the
two-parameter Gamma distribution, Sarhan and Greenberg (1957, 1958) tabulated
the exact coefficients of the order statistics in the best linear unbiased estimators of
the parameter 0 (a in their notation) from complete, singly censored, and doubly
censored samples for n = 1(1)10, together with their exact variances and their
percentage efficiencies (to 2 DP) relative to the best linear unbiased estimator based
on the complete sample. They also tabulated similar results for the best linear
unbiased estimators of the location and the scale parameters c and 0 ( # and ain their
notation) and the mean (# + a) for the two-parameter exponential distribution, a
special case (~ = 1) of the three-parameter Gamma distribution. If ~ is known (not
necessarily equal to 1), the best linear unbiased estimator of 0 based on a complete
Weibull, log-Weibulland Gamma orderstatistics 457
0 = ~ xi,./n~. (4.6)
i=l
Harter (1961) tabulated (to 6 DP) the coefficients of the best linear unbiased
estimators of the parameter of a one-parameter exponential distribution based on k
order statistics (k = 1, 2) and of the parameters of a two-parameter exponential
distribution based on k = 2 order statistics for n = k(1)100. Kulldorff (1963a, b)
extended the tables to include the cases k = 3, 4, 5. Both authors also tabulated the
variances of the estimators and their efficiencies relative to the best linear unbiased
estimators based on all n order statistics.
L = L~+ 1,m with respect to each of the three parameters, which are given by
n
OL/OO : - ( m - r)c~/O + ~ ( x i - c)/O 2
i=r+l
where the primes in equation (4.9) indicate differentiation with respect to ~. Harter
and Moore (1965b) (see also Harter, 1970, p. 104) described an iterative procedure
for solving the likelihood equations, and thus obtaining the maximum-likelihood
estimates 0, c2and d of the parameters. A listing of the FORTRAN program is available
from the author.
HarterandMoore(1967b)(seealsoHarter, 1970,pp. 107-108)gavetheinformation
matrix for doubly censored samples from the three-parameter Gamma distribution,
inverted it to obtain expressions for the asymptotic variances and covariances of 0,
and d, and tabulated the latter for ~ = 1, 2, 3 and sample proportions
ql = 0.00(0.05)0.25 and q2 = 0.00(0.25)0.75 censored on the left and on the right,
respectively.
Although iterative procedures are required to find the maximum-likelihood
estimators of the parameters of the Gamma distribution in the general case, closed
form expressions are available for estimation of the parameter of the one-parameter
exponential distribution (the two-parameter Gamma distribution with shape parame-
ter c~= 1). The maximum-likelihood estimator based on the first m order statistics
of a sample size n was given by Epstein and Sobel (1953) as
This estimator is identical with the best linear unbiased estimator given by Sarhan
and Greenberg (1957, 1958).
what limited because of the unavailability of tables of the. expected values of Gamma
order statistics for n > 40 and of variances and covariances of Gamma order
statistics for n > 15, and the use of estimators such as nearly best linear unbiased
estimators has been explored. Morgan (1968) developed the necessary theory and
used Harter's table of expected values of Gamma order statistics to tabulate
coefficients (to 5 DP) of the order statistics in simultaneous nearly best linear
unbiased estimators of the scale and location parameters of a Gamma distribution
with known shape parameter e based on m order statistics, for e = 0.5(0.5)0.4,
n = 15(1)40, and selected values of m < n. He also tabulated the efficiencies (to
3 DP). R. L. Davis (1970) (Gray, 1970) tabulated similar results for conditional
nearly best linear unbiased (invariant) estimators (of 0] c and el 0).
For n > 40, one must use a different method, say maximum likelihood (Subsection
4.2.2) or the method of moments (not discussed here because it does not involve
order statistics and since Fisher (1922) showed that it is inferior to the method of
maximum likelihood). Another possibility is the use of quantile estimators (see
Subsection 3.2.3). S ~ n d a l (1964) and Hassanein (1977) gave theory and tables for
quantile estimators of the parameters of a Gamma distribution. S~'ndal concentrated
on conditional estimation of 0] c, with brief discussions of conditional estimation of
c]O and simultaneous estimation of 0 and c, while Hassanein dealt mainly with
simultaneous estimation. Several authors have given theory and tables for quantile
estimators of the parameters of one- and two-parameter exponential distributions
(special cases with c~ = 1 of the two- and three-parameter Gamma distributions,
respectively), including Dalcher (1955), Ogawa (1957), Sarhan and Greenberg
(1958), Kulldorff (1962, 1963c), Siddiqui (1963), and Saleh (1966).
5. Applications
References
Anonymous (1951). Tables of the first moment of ranked extremes. NBS Report 1167, National
Bureau of Standards, Washington DC.
Bain, Lee (1983). Gamma distribution. In: Samuel Kotz and Norman L. Johnson, eds., Encyclopedia of
Statistical Sciences, Volume 3, Wiley, New York, 292-298.
Bennett, Carl Allen (1952). Asymptotic properties of ideal linear estimators, Ph.D. dissertation,
University of Michigan, Ann Arbor. Publication No. 3716, University Microfilms, Ann Arbor, MI.
Blom, Gunnar (1956). On linear estimates with nearly minimum variance. Ark. Mat. 3, 365-369.
Biom, Gunnar (1958). Statistical Estimates and Transformed Beta-Variables. Wiley, New York/Almqvist
& Wiksell, Stockholm.
Breiter, M. C. and Krishnaiah, P. R. (1967). Tables for the moments of Gamma order statistics. ARL
67-0166, Aerospace Research Laboratories, Wright-Patterson AFB, OH. AD 639729.
Breiter, M. C. and Krishnaiah, P. R. (1968). Tables for the moments of Gamma order statistics.
Sankhy~ B 30, 59-72.
Bruce, Richard A. (1964). Estimation of the scale parameter of the Gamma distribution by use ofm order
statistics. Unpublished M.S. thesis (GRE/Math/64-2), Air Force Institute of Technology,
Wright-Patterson AFB, OH. AD 619774.
Carlton, Howard F. (1965). Estimation of the scale parameter of the Weibull probability density function
by the use of l order statistics. Unpublished M.S. thesis (GRE/Math/65-2), Air Force Institute of
Technology, Wright-Patterson AFB, OH.
Chan, Lai K. and Kabir, A. B. M. Lutful (1969). Optimum quantiles for the linear estimation of the
parameters of the extreme value distribution in complete and censored samples. Naval Res. Logist.
462 H. L. Harter
Kao, John H. K. (1956b). Weibull distribution in life-testing of electron tubes (abstract). J. Amer. Statist.
Assoc. 51, 514.
Kao, John H. K. (1958). Computer methods for estimating Weibull parameters in reliability studies. IRE
Transactions PGRQC 13, 15-22.
Karns, Robert Clay (1963). Scale parameter estimation of the Gamma probability function based on one
order statistic. Unpublished M.S. thesis (GRE/Math/63-8), Air Force Institute of Technology,
Wright-Patterson AFB, OH. AD 425223.
Kimball, Bradford F. (1946). Sufficient statistical estimation functions for the parameters of the
distribution of maximum values. Ann. Math. Statist. 17, 299-309.
Kimball, Bradford F. (1949). An approximation to the sampling variance of an estimated maximum value
of a given frequency based on fit of doubly exponential distribution of maximum values. Ann. Math.
Statist. 20, 110-113.
King, James R. (1971). Probability Chartsfor Decision Making. Industrial Press, New York. Second edition
(1981), TEAM Publications, Tamworth, NH.
Kingcaid, Milton D. (1969). Unbiased nearly best linear conditional estimates of the scale and location
parameters of the extreme value distributions by use of order statistics. Unpublished M.S. thesis
(GRE/Math/69-5), Air Force Institute of Technology, Wright-Patterson AFB, OH. AD 767874.
Krishnaiah, P. R. and Sen, P. K. (1984). Tables for order statistics. In: P. R. Krishnaiah and P. K. Sen,
eds., Handbook of Statistics, Vol. 4: Nonparametric Methods. North-Holland, Amsterdam, 837-935.
Kulldorff, Gunnar (1962). On the asymptotically optimum spacings for the exponential distribution.
Mimeographed report, University of Lund, Sweden.
Kulldorff, Gunnar (1963a). Estimation of one or two parameters of the exponential distribution on the
basis of suitably chosen order statistics. Ann. Math. Statist. 34, 1419-1431.
Kulldorff, Gunnar (1963b). Estimation of one or two parameters of the exponential distribution on the
basis of suitably chosen order statistics. Part II. Mimeographed report, University of Land, Sweden.
Kulldorff, Gunnar (1963c). On the optimum spacing of sample quantiles from an exponential distribu-
tion. Mimeographed report, University of Lund, Sweden.
Laplace, P. S. (1820). Th$orie Analytique des ProbabilitOs (supplement to the third edition).
Gauthier-Villars, Paris.
Lehman, Eugene H., Jr. (1963). Shapes, moments and estimators for the Weibull distribution. IEEE
Trans. Reliability 12 (3), 32-38.
Leone, F. C., Rutenberg, Y. H., and Topp, C. W. (1960). Order statistics and estimators of the Weibull
distribution. Case Statistical Laboratory Publication No. 1026, Case Institute of Technology,
Cleveland, OH. AD 237042.
Lieblein, Julius (1953). On the exact evaluation of the variances and covariances of order statistics in
samples from the extreme-value distribution. Ann. Math. Statist. 24, 282-287.
Lieblein, Julius (1954). A new method of analyzing extreme-value data. Technical Note 3053, National
Advisory Committee for Aeronautics, Washington DC.
Lieblein, Julius (1955). On moments of order statistics from the Weibull distribution. Ann. Math. Statist.
26, 330-333.
Lieblein, Julius and Salzer, Herbert E. (1957). Table of the first moment of ranked extremes. J. Res. Nat.
Bur. Standards 59, 203-206.
Lieblein, Julius and Zelen, Marvin (1956). Statistical investigation of the fatigue life of deep-groove ball
bearings. J. Res. Nat. Bur. Standards 57, 273-316.
Mann, N. R. (1963). Optimum estimates of parameters of continuous distributions. Research Report
No. 63-41, Rocketdyne Division, North Americal Aviation, Inc., Canoga Park, CA.
Mann, Nancy Robbins (1965). Point and interval estimates for reliability parameters when failure times
have the two-parameter Weibull distribution. Ph.D. dissertation, University of California at Los
Angeles. University Microfilms, Ann Arbor, MI.
Mann, Nancy R. (1967). Results on location and scale parameter estimation with application to the
extreme-value distribution. ARL 67-0023, Aerospace Research Laboratories, Wright-Patterson
AFB, OH. AD 653575.
Mann, Nancy R. (1968a). Results on statistical estimation and hypothesis testing with application to the
Weibull and extreme-value distributions. ARL 68-0068, Aerospace Research Laboratories,
Weibull, log-Weibull and Gamma order statistics 465
A s i t P. B a s u l
1. Introduction
1 This research has been supported in part by a Weldon Spring grant and in part by a grant from
the Research Council of the Graduate School at the University of Missouri, Columbia. It is also
sponsored by the Air Force Office of Scientific Research, Air Force Systems command, USAF,
under grant number AFOSR-87-0139.The US Government is authorized to reproduce and distribute
reprints for Governmental purposes notwithstanding any copyright notation thereon.
* This paper is dedicated to the memory of Professor P. R. Krishnaiah.
467
468 A.P. Basu
P3. F(x + t) = F(x)F(t). That is, it possesses the lack (or loss) of memory prop-
erty (LMP).
P4. Let X 1 and X 2 be independently distributed with X; ~ e(2;).
Then min(X1,X2)~e(21 + £2).
P5. Let X l and X 2 be iid e(2). Then min(X1,X2) and max(X1,X2) - min(X1,X2)
are independently distributed.
P6. If the underlying failure process is Poisson with intensity function 2 then the
times between successive failures are iid e(2).
Because of the usefulness of the univariate exponential distribution it is natural
to consider multivariate exponential distributions as models for multicomponent
systems. However, unlike the normal distribution, there is no natural extension
available in a unique way. The object of this paper is to present a survey of some
recent results obtained to this end. In Section 2 we present a survey of bivariate
exponential distributions. Section 3 lists the important multivariate extensions.
Finally, in Section 4 inference problems relating to Marshall-Olkin and
Block-Basu exponential distributions are considered.
Since it is well known that r(x) is a constant, from P1 and P2 it seems that a
natural bivariate exponential distribution should be absolutely continuous with
exponential marginals and constant failure rate r(x,y). However, Basu (1971)
shows that, except in the case of independence, there does not exist any absolutely
continuous bivariate exponential distribution with constant failure rate and margi-
nal exponential distributions.
Marshall and Olkin (1967) prove a similar result when they consider the follow-
ing extension of P3:
470 A.P. Basu
f f ( x , y) = P ( X > x, Y > y)
= P { Z I ( x ; 2 0 = O, Z 2 ( y ; y 2 ) = O, Z 1 2 ( m a x ( x , y ) ; 212) = 0}
= e-alx 22Y--212max(x,y) (2.11)
2.4. B l o c k - B a s u model
A distribution closely related to BVE is the ACBVE of Block and Basu (1974).
The BVE of Marshall-O!kin has exponential marginals and satisfies LMP.
However, it is not absolutely continuous. In many cases it is desirable to have
a bivariate distribution with exponential marginals that is absolutely continuous
and satisfies LMP. Block and Basu (1974) show that, except for the case of
independence, it is not possible to have a bivariate distribution satisfying all three
above mentioned properties. By considering marginals which are mixtures or
weighted averages of exponentials Basu and Block obtain the following distribu-
tion which we call ACBVE.
Multivariate exponential distributions 471
Here 2 = 21 + 22 + 212.
Note ACBVE, though not a special case, is the absolutely continuous part of
BVE.
By suitable reparametrization, (2.12) is also seen to be a special case of
Freund's model.
Because of physical justifications both BVE and ACBVE are considered useful
models. Note in both cases min(X, Y ) ~ exp(2). Other properties are given in
Block and Basu (1974).
{Xij}; 1, denote the interarrival time of the ith process, i = 1, 2. Assume the ith
component fails after N; shocks where Ni follows the geometric distribution with
parameter 1 - p . The time to failure of the two components are then given by
(YI, Y2) =
j
E X2j
i=1
) "
Conditioning on N and using characteristic functions, the joint density of (Y~, II2)
is obtained as given below.
f(Y"Y2)= ~I#2
- p exp - ( ~ + N2Y2°)l(24Pq'l~2Y/YlZ)p \ -l ~ p I" (2.15)
Here 10 is the modified Bessel function of the first kind of order 0. If instead of
ifidependence, (N~, br2) assumes various other bivariate geometric distributions,
the joint density of (I+1, Y2) is shown by Downton (1970) to be still of the form
(2.15).
F(xl, x2, x3) = P(Xl > x~, X2 > x~, X3 > x3)
= exp[- ,~lXl - 22x 2 - /].3x3 - /~12 m a x ( x l , x2) - 213 max(x1,x3)
-- •23 m a x ( x 2 , x3) - "~123 max(x1, x2, X3)]" (3.1)
Multivariate exponential distn'butions 473
if(x,, x2,..., xp) = e x p [ - ~ 2ix i - Z 2omax(x,, xj) - 2;j~ max (xe, xj, x~)
1 i<j i<j<k
= e x p [ ~ s 2~ max(xisi) ] . (3.2)
4. Inference problems
Whereas following explicit expressions are obtained for the method of moments
estimators
474 A.P. Basu
Proschan and Sullo (1976) consider ML estimation in the p-variate case. They
obtain a simple estimator (INT), as the first iterate in solving the likelihood
equations iteratively. In the bivariate case, their INT estimator is shown to be
more efficient than the estimators given by Arnold (1968) and Bemis, Bain and
Higgins (1972). For stress strength model Basu (1981) has considered the ML
estimate of the reliability function P(x < y) for BVE.
Bhattacharya and Johnson (1973) consider the problem of testing independence
(Ho: 212 = 0 ) in the BVE case assuming identical marginal distributions
(21 = ;~2). They obtain the uniformly most powerful test and provide expressions
for power function.
For the ACBVE expressions for ML equations are given in Block and Basu
(1974). Gupta, Mehrotra, and Michalek (1984) obtain the likelihood ratio test for
independence assuming that the marginals are identically distributed. Gross and
Lam (1981) use ACBVE as a model to investigate the equality of mean survival
time when the observations in populations come in pairs. They apply the results
to study bivariate relapse times for patients receiving different treatments. Weier
and Basu (1980, 1981) have considered tests for independence under both BVE
and ACBVE. They consider Ho: 212 = 0 against alternative hypothesis
H1: 212> 0. They show that for testing above hypothesis the power functions
under Marshall-Olkin and Block-Basu distributions are related by
where PMo and P B B a r e the power functions under BVE and ACBVE respectively,
N denotes the sample size and p = 2~2/(21 + 22 + 212) is the correlation coefficient
under BVE. Let
T denote Kendall's r,
R denote Spearman's correlation coefficient,
V = UMP test statistic for independence,
M = maximum likelihood statistic.
The following results for Pitman's asymptotic relative efficiency hold.
ARE(U,M) = 1,
ARE(T, U) = ARE(R, U) = ARE(T, M) = ARE(R, M) = 0.5.
Weier and Basu also carry out powers under BVE and ACBVE alternatives. Wei
(1980) has also studied the power of sign test and a two-sample test for paired
observations that are subject to arbitrary right censoring under the ACBVE model.
Multivariate exponential distributions 475
References
Freund, J. (1961 ). A bivariate extension of the exponential distribution. Journal of American Statistical
Association 56, 971-977.
Friday, D. S. and Patil, G. P. (1977). A bivariate exponential model with applications to reliability
and computer generation of random variables. In: C. P. Tsokos and I. Schimi, eds., Theory and
Applications of Reliability Pre, Vol. L Academic Press, New York, 527-549.
Gajjar, A. V. and Khatri, G. C. (1969). A trivariate extension of Freund's bivariate distribution.
Vidya 12, 126-144.
Gross, A. J. (1973). A competing risk model: a one organ subsystem plus a two organ subsystem.
IEEE Transactions on Reliability 22, 24-27.
Gross, A. J,, Clark, ¥. A. and Liu, V. (1971). Estimation of survival parameters when one of two
organs must function for survival. Biometrics 27, 369-377.
Gross, A. J. and Lam, C. F. (1981). Paired observations from a survival distribution. Biometrics,
505-511.
Gumbel, E. J. (1980). Bivariate exponential distribution. Journal American Statistical Association 55,
698-701.
Gupta, R. C., Mahrotra, K. G. and Michalek, J. E. (1984). A small test for an absolutaly continuous
• bivariate exponential model, Communications in Statistics--Theory Method 13, 1735-1740.
Hawkes, A. G. (1972). A bivariate exponential distribution with applications to reliability. Journal of
the Royal Statistical Society, Ser. B 34, 129-131.
Johnson, N. L. and Kotz, S. (1977). Distributions in Statistics: Continuous Multivariate Distributions.
Wiley, New York.
Kibble, W. F. (1941). A two-variate gamma type distribution. SankhyTt 5, 137-150.
Klein, J. P. and Basu, A. P. (1985). Estimating reliability for bivariate exponential distributions,
Sankhy-d B 47, 346-353.
Krishnaiah, P. R. (1977). On generalized multivariate gamma type distributions and their applica-
tions in reliability• In: C. P. Tsokos and I. Shini, eds., Theory and Applications of Reliability, Vol. I.
Academic Press, New York, 475-494•
Krishnaiah, P. R. (1983)• Multivariate gamma distributions and their applications in reliability•
Technical Report No. 83-09, University of Pittsburgh.
Krishnamoorthy, A. S. and Parthasarathy, M. (1951). A multivariate gamma type distribution. Annals
of Mathematical Statistics 22, 549-557.
Marshall, A. W. and Olkin, I. (1967a). A multivariate exponential distribution. Journal of American
Statistical Association 62, 30-44.
Marshall, A. W. and Olkin, I. (1967b). A generalized bivariate exponential distribution. Journal of
Applied Probability 4, 291-302.
Morgenstern, D. (1956). Einfache beispiele zweidimensionaler verteilungen, MitteilungsblattJ~r Mathe-
matische Statistik 8, 234-235.
Paulson, A. S. (1973). A characterization of the exponential distribution and a bivariate exponential
distribution. Sankhya, Ser. A 35, 69-78.
Proschan, F. and Sullo, P. (1974). Estimating the parameters of a bivariate exponential distribution
in several sampling situations. In: F. Proschan and R.J. Settling, eds., Reliability and Biometry.
SIAM, Philadelphia, 423-440.
Proschan, F. and Sullo, P. (1976). Estimating the parameters of a multivariate exponential distribu-
tion. Journal of American Statistical Association 71, 465-472.
Raftery, A. E. (1984). A continuous multivariate exponential distribution, Communications in Statis-
tics-Theory and Method 13, 947-965.
S arkar, S. K. (1987). A continuous bivariate exponential distribution. Journal of the American Statistical
Association 82, 667-675.
Tosch, T. J. and Holmes, P. T. (1980). A bivariate failure model. Journal of the American Statistical
Association 75, 415-417•
Wei, L. J. (1980). A generalized Gehan and Gilbert test for paired observations that are subject to
arbitrary right censorship. Journal of the American Statistical Association 75, 634-637.
Weier, D. R. (1981). Bayes estimation for a bivariate survival model based on exponential distribu-
Multivariate exponential distributions 477
1. Introduction
In their review paper, Folkss and Chhikara [23] brought together the widely
scattered literature on the inverse Gaussian distribution, demonstrated its many
attractive properties, and pointed out problems and areas that needed more
research. Since then, investigators have solved some of these problems, further
developed these areas, and consequently have raised many new issues. In this
paper, we describe the principal developments in inverse Gaussian theory since
that review.
An important application of this model is in reliability theory, which has
spurred considerable progress in regression for the inverse Gaussian model; after
Section 2 establishes the notation, Section 3 describes the details of this develop-
ment. The widely applicable physical basis for this model and its analytical
tractability make the search for a useful multidimensional generalization impor-
tant; Section 4 describes and compares the various proposals for a multivariate
inverse Gaussian. Section 5 collects a variety of special results such as estimation,
characterization theorems, and Bayesian methods. Section 6 briefly considers the
generalized inverse Gaussian distribution, and Section 7 describes various appli-
cations of the inverse Gaussian. The paper then concludes with a bibliography,
which complements that of Folks and Chhikara.
Let {W(t): t t> 0} be a Brownian motion with positive drift 0, and unit diffusion
parameter, so that W(t) is a N(Ot, t) variate. Define T = inf{t >1 0: W(t) = ~} for
a positive constant ~. The density of T is
479
480 S. lyengar and G. Patwardhan
Tweedie's [41] original notation uses the mean hitting time/~ = a/0 and 2 = 0~2 as
the parameters; it is slightly more convenient to use a = 1/2, so that (1) becomes
(t-P)2/t ~
f(t) = (2nat3)-l/2exp -2#-~a J' for t>_-O. (2)
We use I G ( # , a) to denote the distribution with density (2); note that a is not the
standard deviation. Only in Section 4, where we describe Barndorff-Nielsen's
proposal for a bivariate inverse Gaussian, do we use different notation. In
Table 1, we contrast the properties of the inverse Gaussian with those of the
Gaussian. Its evident tractability has made the inverse Gaussian distribution a
useful example for various general techniques: see, for example, Barndorff-Nielsen
and Cox [12].
Table 1
3. Regression
The attractive properties of the inverse Gaussian distribution have led several
investigators to develop regression techniques for it. A one-way analysis of
variance for different means/x; and constant a was developed by Tweedie [41],
who used the following algebraic identity for positive numbers {a;j: j = 1. . . . . nt;
i = 1. . . . . k}:
where ai and ~ denote the appropriate averages. When {a;j} are independent
I G ( # , a) variates, the terms in (3) are each a times X2 variates with (Y~n,.) - 1,
k - 1, a n d ( ~ n i ) - k degrees of freedom, respectively; the two terms on the right
are independent. Early attempts to extend this 'analysis of reciprocals' had limited
Recent developments in the inverse Gaussian distribution 481
success. Schuster and Miura [36] studied a similar problem for the two-way
layout. Davis [21] concentrated on the zero-intercept model, with/~ = flxi, where
x i is a covariance, and with the mean proportional to the variance. She also
considered the model #; = r0 + fll x~, but only provided some ad hoc procedures.
That these approaches do not exploit the underlying physical process was noted
by Bhattacharya and Fries [14], who proposed a reciprocal linear model. This
model was also used by Whitmore [45].
Consider the following situation: n similar objects are subject to stresses at
levels xl . . . . . x n until they break. Each object has a characteristic breaking
threshold, which we take to be the same. The different stress levels, however,
imply that the accumulation of fatigue proceeds at different objects. If this
accumulation follows a Brownian motion, we may then model the drift as a
function of the covariate, x; and when this drift is linear in x, the time to breaking
for the ith object follows an inverse Gaussian with mean #~, where
1/#~ = to + fll x . and, because the breaking threshold is the same for all objects,
constant a. The appropriate generalization of this 'reciprocal linear model' has
several advantages [24]: (i) it arises out of a possible physical model, (ii) the
hitting time distribution acquires a wide variety of shapes, depending upon the
parameter values, and (iii) the relevant sampling theory is, in large part, tractable.
The inverse Gaussian is thus a competitor to other well-known parametric
families, such as the gamma or the Weibull; however, it does not supplant either
because, for instance, it does not adequately fit data from the exponential distribu-
tion, and the theory for the inverse Gaussian with censoring is not well developed.
Furthermore, the strong resemblance between the inverse Gaussian and Gaussian
theories (as demonstrated in Table 1) becomes somewhat weaker in the regression
context. For example, in the Gaussian linear model, maximum likelihood esti-
mates of parameters coincide with the least squares estimates. However, in the
inverse Gaussian reciprocal model, the two are not the same; and whereas the
(finite sample) sampling properties of the least squares estimates are easier to
obtain than those of the maximum likelihood estimates, the former are less effi-
cient for large samples.
We now turn to the details of this regression model and primarily follow the
development of Bhattacharya and Fries [15]. Suppose that at each design point
x,.~ ~P, i = 1, ..., k, there are independent observations {To: j = 1. . . . , n,}. The
suppose that To is an IG(/z, tr) variate with 1/#; = x" r, or (with some abuse of
notation)/~-1 = Xfl. Assume also that k >~p + 1. With the following notation,
(4b)
482 S. lyengar and G. Patwardhan
which is similar to the Gaussian theory normal equations. When X has full rank,
so does S and (6) has unique roots (which provide a maximum of the likelihood)
These roots, however, need not to be the maximum likelihood estimates since
~'x; may be negative; however, unless N is very small, this is not a serious
problem, because of the asymptotic results sketched below.
Let N--, oo so that ni/N--+hi> O, define diagonal matrices M and H with
m;; = #; and h;/= h;, for i = 1, ..., k, and let A = X ' H M X , then the strong con-
sistency of the estimates given in (7) is straightforward. Bhattacharya and Fries
[15] also show that
N1/2(/~ - fl)-+ Np(0, a A - l ) , (8a)
Thus, approximate testing procedures and confidence sets for parameters of this
regression model are available.
When there is a replicated design, and the model includes a constant term, so
that n; >~ 2 for all i, X = (1, X1), and fl = (rio, fl'0', an analysis of reciprocals is
developed by Bhattacharya and Fries in [15], who use the identity
T/j1 _ ~-1 = (x;/7 - 7"-') + (7',.-- ' - x,'/7) + (T;j' - ~'7 1). (9)
Summing over i and j, we get (10), which decomposes the sum of deviations from
the grand mean into contributions due to the regression of/71 (given rio), Qreg, due
to the lack of fit of the model, Q1, and due to the error, Qe:
Bhattacharya and Fries [ 15] show that the likelihood ratio test for O~ rejects for
large values of
Ql/(k - p)
FL - , (12)
Q e / ( N - k)
can be written as
4. Multivariate theory
Folks and Chhikara [23] noted that very little work had been done in the area
of multivariate generalizations of the inverse Gaussian, and that fact still holds
today. There are at least two interesting approaches here: one generalizes the
underlying physical process to, say, a multidimensional correlated Brownian
motion, and studies certain first passage times that arise; another concentrates on
the nice sampling theory exhibited in Table 1, and seeks a multidimensional
version that has properties similar to that of the multivariate normal (or failing
that, is otherwise tractable). These approaches are exemplified in the papers by
Iyengar [27] and Barndorff-Nielsen and Blaesild [11], respectively. Their pro-
posals for a bivariate inverse Gaussian indicate that the two approaches are
conflicting: that is, those that arise out of a physical process have complicated
sampling properties, and those with nicer sampling properties have an artificial
physical interpretation. Since there are infinitely many multivariate distributions
with inverse Gaussian marginals, still other proposals are bound to appear.
In Iyengar [27], the Gerstein-Mandelbrot [25] random walk model for the
spontaneous activity of a neuron is generalized to a pair of neurons (see
Section 7). In particular, let Wi(t) denote the electrical state of the ith neuron at
time t, for i = 1, 2. If both neurons are driven by the same external source of
stimulation and are subject to random input, then IV(t)= (Wl(t), W2(t))' is
approximately a correlated two-dimensional Brownian motion with mean Ot and
variance tR. Here, 0 is the drift parameter indicating the strength of external
stimulation, and R is a correlation matrix indicating the strength of coupling
between the neurons. Each neuron fires when its electrical state reaches its
characteristic threshold, so that the firing times of interest are Ti = inf{t >i 0:
Recent developments in the inverse Gaussian distribution 485
W,.(t) = ~e}, for i = 1, 2. It is clear that the marginal distribution of T,. is inverse
Gaussian; and since the underlying Brownian motion is correlated, T = (T1, T2)'
is a correlated bivariate inverse Gaussian vector. When 0 = 0, Iyengar [27]
derives the joint density of T, and gives a more complicated expression when there
is a drift towards each barrier. The statistical properties of this proposal for a
bivariate inverse Gaussian are not yet studied.
Barndorff-Nielsen [9] and Barndorff-Nielsen and Blaesild [11] use statistical
criteria to propose a bivariate inverse Gaussian. In [9], Bamdorff-Nielsen
discusses a formula for the approximate distribution (conditional on a maximal
ancillary) of the maximum likelihood estimator, &, of a parameter o~; he
derived this formula by using a saddlepoint approximation. He notes that the
univariate inverse Gaussian with known a is one of only three (the others being
the gamma with known shape parameter, and the Gaussian with known variance)
one-parameter exponential models for which the formula is exact. He then
describes a method of combining univariate exact cases to get a bivariate exact
case and proposes the following: define the bivariate inverse Gaussian
T = ( T 1, Z2)' by starting with T1, and IG(#, a) variate and T2IT ~ an
IG(7, 1/~T~) variate, for some positive 7 and ~. In terms of an underlying
Brownian motion W(t), TI is the first hitting time for Wl of a certain level, and
given 7"1, T2 is the first hitting time for W2 of the level c T 1, for some positive
constant c. Note, however, that T2 is not a stopping time; thus the physical basis
of this proposal is somewhat artificial. Further, the marginal distribution of Tz is
not inverse Gaussian, so the name 'bivariate inverse Gaussian' is not entirely
appropriate.
Barndorff-Nielsen's proposal does have attractive properties, however. In (1)
above, if we write ~/= ~2 and ~k= 0 2, the density becomes
f ( t ) = (~/2~t3) a/2 exp((q~) 1/2) exp[ - (~kt + q/t)/2], for t I> 0, (17)
Another proposal is due to Wasan and Buckholtz [42], who derive a partial
differential equation for a class of bivariate inverse Gaussians, but the only
examples that they provide are the following: (T1, T2)' and (T I, T 1 + 7"2)' where
T1 and T2 are independent inverse Gaussian variates.
6. Extensions
where K s is the modified Bessel function of the third kind with index V. This
family is very broad, as it includes the inverse Gaussian, the gamma, and their
reciprocals. Jorgensen [29] gives an extensive treatment of its basic properties,
estimation and testing procedures, investigates its reliability function, and provides
examples of its use. Barndorff-Nielsen and Halgren [10] have shown that (18) is
not just and ad hoc proposal, by constructing diffusions on [0, oo) for which the
generalized inverse Gaussian with ~ ~< 0 is a first passage time.
7. Applications
Because of its appealing physical basis, the inverse Gaussian enjoys wide
applicability. In short, whenever a Brownian motion or random walk drifting
towards a threshold is a plausible mechanism., this first passage time can arise.
It has been suggested by some authors that the analytical tractability of the inverse
Gaussian is sufficient reason to use it for curve fitting, but that is probably of
limited scientific interest. In several applications, the simple Brownian motion
model is only a starting point: it describes only the baseline behavior; and when
the drift and the barrier have physical meaning, they can be manipulated in an
attempt to replicate more complicated behavior.
Since the hazard function for the inverse Gaussian increases and then decreases,
it serves as a good model for accelerated life tests. Padgett [34] gives confidence
bounds for the reliability function; Bhattacharya and Fries [13] argue that the
inverse Gaussian is more appropriate than the Birnbaum-Saunders fatigue life
488 S. lyengar and G. Patwardhan
distribution; and Chhikara and Guttman [19] give frequentist and Bayesian
prediction limits. Thus, the inverse Gaussian is becoming a widely used tool in
reliability theory.
Many physiological processes evolve according to a diffusion (in fact, that is
how Tweedie [41] started his investigations). For instance, as mentioned before,
Gerstein and Mandelbrot [25] modelled a neuron's membrane potential as a single
number which moves towards or away from a threshold according to whether it
receives an excitatory or inhibitory signal, respectively. If the signals are of roughly
equal magnitude and come at high rates, then a Brownian motion approximation
is appropriate. They showed that this model provides a good fit for the spon-
taneous activity of several neurons in the auditory cortex of a cat. They could
interpret the parameters: the drift is a measure of the difference between the
excitatory and inhibitory rates, and the barrier is the firing threshold. They also
showed that by introducing a time-varying drift for the Brownian motion, they
could replicate the observed behavior of one of the neurons subjected to periodic
stimuli of various frequencies. Other more sophisticated diffusion models are
described in Fienberg [22]. Another important physiological application is the
modelling of tracer dilution curves in cardiology: see Wise [47]. A review of
various types of random walk models for physical systems and an extensive
bibliography is given in Weiss [43].
One of the earliest applications of Brownian motion is due to Bachelier [7],
who modelled stock prices. Since then, investigators have applied it to a wide
variety of phenomena in (among others) finance, marketing, and labor. Recent
examples in which the inverse Gaussian arises are the work of Banerjee and
Bhattacharya [8], who model interpurchase times as a renewal process with
inverse Gaussian components, and Whitmore [44], who models labor turnover
with this model. Whitmore has found that some of his data is very long-tailed,
so he also uses a drift away from the barrier; thus, if T is the hitting time, then
P(T = ~ ) is positive; this idea is similar to that of terminating renewal processes
and gives rise to censored observations, which Whitmore [46] has studied in a
regression context.
8. Conclusion
In spite of the recent work on the inverse Gaussian model, many issues remain
unresolved; they include a fuller treatment of sequential methods, of censored
data, a useful multivariate theory, and a more detailed study of inverse Gaussian
regression. Some questions raised by the discussants of the Folks and Chhikara
[23] paper have not yet been answered; for instance, the application of inverse
Gaussian methods to data generated from multiplicative models has not yet been
studied. Also, the reasons for the tractability of the inverse Gaussian theory is not
well understood; the answer may lie in a geometrical study, and the hyperbolic
geometry studied by Barndorff-Nielsen and his colleagues may well be the key in
this direction. Finally, in order to facilitate the use of this model, inverse Gaussian
Recent developments in the inverse Gaussian distribution 489
Acknowledgement
W e t h a n k an a n o n y m o u s referee, w h o s e c o m m e n t s h e l p e d to i m p r o v e the
quality o f this p a p e r . F o r the first author, this r e s e a r c h w a s partly s u p p o r t e d by
N I M H c o n t r a c t 5 + 32 M H 1 5 7 5 8 - 0 7 at C a r n e g i e - M e l l o n U n i v e r s i t y .
References
[1] AI-Hussaini, E. et al. (1981). A bivariate inverse Gaussian distribution. Ann. Inst. Stat. Math.
33, 57-66.
[2] Al-Hussaini, E. et al. (1981). On the identifiability of finite mixtures of distributions. IEEE
Trans. lnf Th. 27, 664-668.
[3] A1-Hussaini, E. et al. (1984). Information matrix for the mixture of two inverse Gaussian
distributions. Comm. Stat. SirnuL and Comp. 13, 785-800.
[4] Ahsanullah, M. et al. (1984). A characterization of the Wald distribution. Nay. Res. Logistics
Quart. 31, 155-158.
[5] Amoh, R. (1984). Estimation of parameters in mixtures of inverse Gaussian distributions.
Comm. Stat. Th. and Meth. 13, 1031-1043.
[6] Amoh, R. (1984). Estimation of the discriminant function from a mixture of two inverse
Gaussian distributions when sample size is small. Jour. Stat. Comp. and Simul. 20, 275-286.
[7] Bachelier, L. (1900). Theorie de la Speculation. Ann. Sci. Ec. Norm. Super. 17, 21-86.
[8] Banerjee, A. et al. (1983). Bayesian results for the inverse Gaussian distribution with an
application. Tech. 21, 823-829.
[9] Barndorff-Nielsen, O. (1983). On a formula for the distribution of maximum likelihood esti-
mators. Biometrika 70, 343-365.
[10] Barndorff-Nielsen, O. et al. (1977). Infinite divisibility of the hyperbolic and generalized inverse
Gaussian distributions. Zeit. Wahr. 38, 309-312.
[11] Barndorff-Nielsen, O. et al. (1983). Reproductive exponential families. Ann. Star. 11, 770-782.
[12] Barndorff-Nielsen, O. et al. (1984). The effect of sampling rules on likelihood statistics. Int. Stat.
Rev. 52, 309-326.
[13] Bhattacharya, G. et al. (1982). Fatigue failure models: Birnbaum-Saunders vs. inverse
Gaussian. IEEE Trans. Rel. 31, 439-441.
[14] Bhattacharya, G. et al. (1982). Inverse Gaussian regression and accelerated Life Tests. Proc.
Spec. Topics Mtg. on Survival Analysis, 101-117.
[15] Bhattacharya, G. et al. (1984). On the inverse Gaussian multiple regression and model checking
procedures. Tech. Rep., Univ. of Wisconsin.
[16] Chan, M. et al. (1984). Generation of inverse Gaussian distribution tables. Comm. Stat. Th. and
Meth. 12, 423-442.
[17] Cheng, R. (1984). Generation of inverse variates with given sample mean and dispersion. Appl.
Stat. 33, 309-316.
[18] Cheng, R. et al. (1981). Maximum likelihood estimation of parameters in the inverse Gaussian
distribution with unknown origin. Tech. 23, 257-263.
[19] Chhikara, R. et al. (1982). Prediction limits for the inverse Gaussian distribution. Tech. 24,
319-324.
[20] Cohen, A. et al. (1985). Modified estimation fot the three-parameter inverse Gaussian distribu-
tion. Jour. Qual. Tech. 17, 147-154.
[21] Davis, A. (1977). Linear statistical inference as related to the inverse Gaussian distribution.
Ph.D. Dissertation, Okla. State Univ.
490 S. lyengar and G. Patwardhan
[22] Fienberg, S. (1974). Stochastic mpdels for single-neuron firing trains: A survey. Biometrics 30,
399-427.
[23] Folks, L. et al. (1978). The inverse Ganssian distribution and its statistical applications: A
review. JRSS-B. 40, 263-289.
[24] Fries, A. et al. (1983). Analysis of two-factor experiments under an inverse Gaussian model.
JASA 78, 820-826.
[25] Gerstein, G. et al. (1964). Random walk models for the spike activity of a neuron. Biophys. Your.
4, 41-68.
[26] Iwase, K. et al. (1983). UMVU extimation for the inverse Gaussian distribution. JASA 78,
660-663.
[27] Iyengar, S. (1985). Hitting with two-dimensional Brownian motion. S l A M Your. Appl. Math. 22,
983-989.
[28] Jones, G. et al. (1984). On the asymptotic efficiency of the moment and maximum likelihood
estimators in the three-parameter inverse Gaussian distribution. Comm. Star. Th. and Meth. 13,
2307-2314.
[29] Jorgensen, B. (1981). Statistical Properties of the Generalized Inverse Gaussian Distribution.
Springer, New York-Berlin.
[30] Khatri, C. (1962). A characterization of the inverse Gaussian distribution. Ann. Math. Stat. 33,
800-803.
[31] Korwar, R. (t980). On UMVU extimators of the variance and its reciprocal of an inverse
Gaussian distribution. JASA 75, 734-735.
[32] Letac, G. et al. (1983). A characterization of the generalized inverse Gaussian distribution by
continued fractions. Zeit. Wahr. 62, 485-489.
[33] Lombard, F. (1978). A sequential test for the mean of the inverse Gaussian distribution. S. Afr.
Stat. Your. 12, 107-115.
[34] Padgett, W. (1981). Bayes estimation of reliability for the inverse Gaussian model. IEEE Trans.
Relia. 30, 84-85.
[35] Padgett, W. et al. (1979). Estimation for the three-parameter inverse Gaussian distribution.
Comm. Stat. Th. Meth. 8, 129-137.
[36] Schuster, J. et al. (1972). Two-way analysis of reciprocals. Biometrika 59, 478-481.
[37] Seshadri, V. (1980). A note on the inverse Gaussian distribution. In: C. Taillie et al., eds., Star.
Dist. in Scientific Work.
[38] Seshadri, V. (1983). The inverse Gaussian distribution: Some properties and characterizations.
Canad. Your. Star. 11, 131-136.
[39] Seshadri, V. et al. (1983). A Cochran-type decomposition theorem for inverse Gaussian variates.
Tech. Rep., McGill Univ.
[40] Seto, N. et al. (1985). UMVU estimation of the mode and limits of an interval for the inverse
Gaussian distribution. Comm. Stat. Th. Meth. 14, 1151-1162.
[41] Tweedie, M. (1957). Statistical properties of the inverse Gaussian distribution I. Ann. Math.
Stat. 28, 362-377.
[42] Wasan, M. et al. (1973). Differential representation of a bivariate inverse Gaussian process.
Your. Mult. Analysis 3, 242-247.
[43] Weiss, G. (1983). Random walks and their applications. Amer. Scientist 71, 65-71.
[44] Whitmore, G. (1979). An inverse Gaussian model for labor turnover. JRSS-A 142, 468-478.
[45] Whitmore, G. (1980). An exact multiple regression model for inverse Gaussian data. Tech. Rep.,
McGill Univ.
[46] Whitmore, G. (1983). A regression method for censored inverse Gaussian data. Can. Your. Star.
11, 305-315.
[47] Wise, M. (1966). Tracer dilution curves in cardiology and random walks and lognormal distribu-
tions. Acta Phys. Pharm. Neerland. 14, 175-204.
Subject index
Subject Index
491
492 Subject index
V o l u m e 3. T i m e Series in the F r e q u e n c y D o m a i n
E d i t e d by D. R. Brillinger a n d P. R. K r i s h n a i a h
1983 xiv + 485 pp.
Volume 6. Sampling
Edited by P. R. Krishnaiah and C. R. Rao
1988 xvi + 594 pp.