Uncertainty Quantification in Prognostic Health
Management Systems
H. Heath Dewey Derek R. DeVries
Northrop Grumman Northrop Grumman
Innovation Systems Innovation Systems
9160 N Hwy 83 9160 N Hwy 83
Corinne, UT 84307 Corinne, UT 84307
435-863-2375 435-863-6693
Heath.Dewey@ngc.com Derek.DeVries@ngc.com
Abstract – Uncertainty plays a role in nearly all aspects of
prognostic health management (PHM) systems. Aleatory
uncertainty from inherently-variable inputs such as material
properties, epistemic uncertainty from a lack of knowledge
about the system and its inputs, and ontological uncertainty due
to completely unknown factors must all be accounted for in
order to provide the most accurate assessment of the health of
the monitored system. Northrop Grumman Innovation Systems
(NGIS) develops, produces, and provides sustainment of solid
rocket motor systems for the aerospace and defense industry
and has extensive experience in applying uncertainty
quantification (UQ) principles to complicated numerical
simulations and analyses. In this paper, lessons learned by
NGIS on UQ simulations and analyses are presented, and their
applicability to PHM systems is explored. Methods for
measuring and tracking uncertainty through the PHM
predictive train are presented, as is a Monte-Carlo-based
method for performing prognostic numerical calculations,
which accounts for and quantifies epistemic, ontological, and
aleatory uncertainty.
TABLE OF CONTENTS
1. INTRODUCTION ....................................................... 1
2. ARCHITECTURE OF A PHYSICS-BASED PHM
SYSTEM ....................................................................... 2
3. UNCERTAINTY QUANTIFICATION .......................... 2
4. SOURCES OF UNCERTAINTY.................................. 3
5. CATEGORIES OF UNCERTAINTY .......................... 4
6. UNCERTAINTY IN PHM SYSTEMS ........................ 5
7. CALCULATION WITH UNCERTAINTY ................... 8
8. CONCLUSIONS ..................................................... 10
REFERENCES............................................................. 11
BIOGRAPHY .............................................................. 12
1. INTRODUCTION
Northrop Grumman Innovation Systems (NGIS) has long
understood that collecting accurate health state information
about a product is crucial to the success of our products.
978-1-5386-6854-2/19/$31.00 ©2019 IEEE
1
Scott R. Hyde
Northrop Grumman
Innovation Systems
9160 N Hwy 83
Corinne, UT 84307
435-863-6307
Scott.Hyde@ngc.com
Systems engineering plays a vital role in this technical
process. The systems engineering processes “are used to
define the requirements for a system, to transform the
requirements into an effective product, to permit consistent
reproduction of the product where necessary, to use the
product to provide the required services, to sustain the
provision of those services, and to dispose of the product
when it is retired from service.” [1]
These processes define the activities to optimize the designs
and reduce the risks associated with technical decisions
during the development, manufacturing, fielding, operation,
and removal from service of the products. These processes
also define, verify, validate, and control the “the timeliness
and availability, the cost effectiveness, and the functionality,
reliability, maintainability, producibility, usability and other
qualities” [1] of the system. An extremely important part of
this process is to accurately ascertain the ability of the
product to meet the required performance throughout its life,
including the capture of information about the product
through testing, component and system model analysis, and
collection of information from the system’s performance.
The captured system information contains key performance
information or parameters as defined by the system’s
components and material performance models.
If information is collected incorrectly or inaccurately or the
method of collection does not provide the information to
accurately model the behavior, then the performance of the
model will have increased uncertainty (may not be
understood or cannot be quantified) or may not represent the
expected behavior at all.
In today's environment of customers demanding more
efficient and affordable designs while maintaining or
increasing reliability, with lower mass and lower overall cost
of ownership, it is imperative that we improve and
standardize our technical assessment approach. This paper
discussed the approaches that are being used to obtain the
necessary information and quantify the uncertainty as part of
a Prognostic Health Management (PHM) system used to
ascertain the health state information of the system and its
components.

2. ARCHITECTURE OF A PHYSICS-BASED PHM
SYSTEM
The architecture of a general physics-based PHM system is
shown in Figure 1. It is referred to as the predictive train; a
sequence of models and data sources that start with the causal
state variables on the left and end with predicted asset
performance on the right. The physics-based models can be
categorized into three groups: (1) the evolution models that
describe how the state variables evolve under the influence
of the history operator, (2) the conversion models that convert
from state variables to the properties required by the
performance assessment models, and (3) the performance
assessment models that predict how the asset will operate
with that given set of properties. The right-most activity is the
uncertainty roll-up that outputs the desired results of the PHM
system: when will the asset fail, what component will fail
first, and what are the uncertainties and confidence limits in
these predictions?
The five numbered boxes constitute the predictive train. The
uncertainty roll-up on the right is the critical activity that
provides the desired output – probability of failure with
confidence limits.
Figure 1. Architecture of a physics-based PHM system
In Box 1, the causes of property evolution are found. If an
asset component is evolving due to chemical
reaction/diffusion processes, chemists will define the causes
of evolution. If the evolution is due to cumulative exposure
to electromagnetic fields or to mechanical fatigue, other
scientists are needed to define the causes of evolution. For
most materials, there is a rich literature describing causal
processes. The literature is a good starting point, but must
usually be adapted and augmented for the specific assets of
interest.
2
In Box 2, carefully designed experiments are conducted,
sometimes on idealized as well as real systems, to measure
the parameters in the evolution model built from the causal
information of Box 1. These experiments must often span
several years, since the evolution of many parameters is very
slow.
In Box 3, the evolution model is exercised to predict the
evolution in time of the causal state variables given an
applied set of boundary condition histories (the history
operator). If some assets have seen a harsher boundary
condition history, they age out sooner than other assets from
the same set that have not seen the harsher boundary
conditions.
In Box 4, the state variables are converted into the properties
needed by the performance assessment models.
In Box 5, the properties from any point in time are fed into
the performance assessment models that predict asset
performance.
Lastly, all uncertainties from all data sources and all models
along with the ascertained uncertainty for each critical
parameter are rolled into a final analysis called the
uncertainty roll-up, which provides probability of failure (or
conversely, reliability) versus time or usage. [2] [3] [4]
3. UNCERTAINTY QUANTIFICATION
Uncertainty quantification (UQ) is the combination of
verification (assessment of mathematical accuracy) and
validation (assessment of applicability) of mathematical
models of real-world phenomena. Understanding this
requires quantifying the uncertainty of the numerical
solutions of those models with assessments of the variation
in those models, the solutions, the model inputs (boundary
conditions, physical parameters, etc.), and in the phenomena
in order to quantify the variations in the numerical model
outputs.
When one applies uncertainty quantification to an
engineering analysis, they hope to assess the uncertainty in
the real-world physical phenomena and the uncertainty in the
mathematical models of those phenomena and their
numerical implementations in order to better understand the
variability in the real-world physical responses that are being
analyzed.
Uncertainty quantification is now required during product
analysis in many industries for several reasons. First, the
globalization of all industries, as well as increased
competition in specific industries including aerospace and
defense, is forcing companies to reduce product development
costs in order to stay competitive. This reduction in costs
manifests itself in many ways including fewer tests of real-
world physical assets and more virtual tests via analysis. In
order for the results of these virtual tests to be applicable to
the physical products being tested, the uncertainty in the
analyses must be quantified.
Secondly, as many higher-performance products are
developed, normalization of margins-of-safety or safety
factors across the asset are required in order to keep mass to
a minimum while still maintaining the required reliability of
the asset. The uncertainty in the performance of the product
must be quantified in order to enable this normalization.
A significant amount of research has been performed on
uncertainty quantification in recent years. This paper extends
the work of Ewing, Liechty, and Black [5] from general
engineering analysis to the field of PHM and provides details
and examples of uncertainty sources that are often
encountered with PHM systems.
Specific to PHM, uncertainty quantification has always been
a requirement since the purpose of a PHM system is to
ascertain the reliability of an asset via probabilistic methods.
Those working in the field of PHM have traditionally
quantified sources of uncertainty from the aleatory risk of a
component in their analyses while completely ignoring other
sources of uncertainty that may be occur from epistemic
risks. Previous PHM work at Northrop Grumman Innovation
Systems has shown that some sources of uncertainty,
typically on very sensitive parameters such as activation
energy material properties, have previously been ignored but
may dominate all other uncertainty in the system, including
asset-to-asset variation. In order to accurately predict the
performance of an asset, PHM systems must quantify all
sources of uncertainty and utilize those uncertainties in all
aspects of the prognostic calculations.
4. SOURCES OF UNCERTAINTY
In engineering model analyses, there are many sources of
uncertainty that factor into the overall uncertainty in the
result. All of those sources of uncertainty can be
characterized as being a subset of one of three uncertainty
sources: model input uncertainty, model discretization
uncertainty, and model form uncertainty [5].
3
Model input uncertainty is the uncertainty in any input to the
mathematical models used to perform the analysis. In
engineering analyses, input quantities such as material
properties, initial conditions, and boundary conditions fall
under the umbrella of model input uncertainty. Model input
uncertainties are estimated by a variety of methods typically
involving a form of validation testing of the input parameter.
As an example, materials may be tested repeatedly in a
laboratory in order to estimate the probabilistic distribution
of the material properties under differing conditions.
Model discretization uncertainty is the uncertainty in the
implemented mathematical model due to the finite resources
available on computer systems. Specifically, sources of
discretization uncertainty in engineering analyses typically
include mesh (spatial) discretization uncertainty, time step
(temporal) discretization uncertainty, and uncertainties due to
finite floating-point number precision. Estimation of model
discretization uncertainty sources typically involves a
verification process where the results of an analysis are
compared to an analytical solution (e.g., method of
manufacturer solutions) if possible or another numerical
analysis performed with parameters that utilize a different
level of discretization (e.g., grid refinement studies).
Model form uncertainty is the uncertainty in the degree to
which the implemented mathematical model represents the
real-world behavior of the physical objects that are
represented in the model. Simplifications in theory and
implementation in order to make the numerical solutions
tractable (assumptions) are an example of a source of model
form uncertainty. This form of uncertainty is typically
estimated by validation, comparison of the output of the
mathematical model to the real-world response that the model
is attempting to represent. This validation is often challenging
as measurements of the real-world response are subject to
additional measurement uncertainties that must be accounted
for and subtracted from the calculated model form
uncertainty.
One should note that the chosen numerical model has a great
effect on the level of uncertainty from each of the
aforementioned sources. As an example, machine learning
algorithms have very low model form uncertainty (at least
after sufficient learning has been done) since they operate by
building functions that match measured data. However, the
same machine learning algorithms may have undefined or
hard to calculate model discretization uncertainty because the
underlying functions that have been generated may be
arbitrarily discretized by the algorithm. As such, care must be
taken when choosing an algorithm to utilize one that has
measureable or estimable uncertainties from each source.
Specific to PHM systems, additional care must be taken to
choose models that have uncertainties that can be quantified
far into the future. For example, a PHM system may be tasked
with monitoring a fleet of assets over an expected lifetime of
several decades. In that instance, models must be utilized that
have inputs, discretization, and forms that can be measured
or estimated for a significant span of time. Further detail on
this problem and possible solutions to it are given below.
5. CATEGORIES OF UNCERTAINTY
Uncertainty in engineering and PHM analyses can further be
categorized into three classes based on the underlying cause
of uncertainty: aleatory uncertainty, epistemic uncertainty,
and ontological uncertainty [5].
Aleatory uncertainty is uncertainty due to real-world physical
variation. Uncertainties that are categorized as aleatory
cannot be reduced or decreased with additional knowledge
about the physical system being analyzed. An example of
aleatory uncertainty is inherent variation in material
properties that are caused by inhomogeneity or random
distribution of molecules in the structure of the material.
Aleatory uncertainties can be statistically estimated through
testing of the real-world physical systems bounded by
statistical distributions of the varying parameters. Note that
these statistical distributions will contain uncertainty due to
sampling of the parameters, and this uncertainty must also be
captured. For example, if the distribution of the underlying
physical parameter can be represented as a normal
distribution,
𝑉̃ → 𝑁(𝜇, 𝜎 2 ) (1)
and a finite-number of measurements of that parameter are
taken, the form of the normal distribution can be estimated
using further statistical distributions. In this case, the sample
distribution of the underlying parameter can be estimated
from the individual, repeated measurements and the number
of measurements as
𝑉̃𝑠 → 𝑁(𝜇𝑠 , 𝜎𝑠2 ) (2)
where the sample mean and unbiased sample variance are
1 (3)
𝜇𝑠 = ∑ 𝑉𝑖
𝑛
𝑛 unknown factors; Donald Rumsfeld’s “unknown unknowns”
and
1 (4)
𝜎𝑠2 = ∑(𝑉𝑖 − 𝜇𝑠 )2
𝑛−1
𝑛
From Cochran’s Theorem [6], the distribution of the unbiased
sample variance due to the finite number of samples is
𝜎2 (5)
𝜎̃𝑠2 → 𝜒2
(𝑛 − 1) 𝑛−1
2
where 𝜒𝑛−1 is the Chi-Squared Distribution with degrees of
freedom n-1. The actual variance, 𝜎 2 , is not known, only a
sample of the sample variance distribution, 𝜎𝑠2 , has been
obtained. However, the above equation can be inverted to
calculate the probability density function for the actual
variance of the underlying distribution based on this sample
4
of the sample variance distribution. This results in
𝜎̃ 2 → 𝜎𝑠2 (𝑛 − 1)𝐼𝑛𝑣𝜒𝑛−1
2 (6)
2
where 𝐼𝑛𝑣𝜒𝑛−1 is the Inverse Chi-Squared Distribution with
degrees of freedom n-1.
Similarly, from the Central Limit Theorem, the distribution
of the sample mean as a function of the number of samples is
given by
𝜎2 (7)
𝜇̃𝑠 → 𝑁 (𝜇, )
𝑛
This equation can similarly be inverted to produce the
probability distribution function for the mean of the
underlying data as a function of a sample of the sample mean
distribution as
𝜎2 (8)
𝜇̃ → 𝑁 (𝜇𝑠 , )
𝑛
Epistemic uncertainty is uncertainty due to a lack of
knowledge, which can be reduced or eliminated completely
by incorporating additional knowledge into the model. An
example of an epistemic uncertainty is the initial state of the
system being analyzed. The system has a definite initial state,
but that state may not be adequately measured or completely
known when the analysis model is exercised on the numerical
model of the system. If additional measurement were
performed to identify the correct initial state prior to
execution of the analysis, this epistemic uncertainty would be
eliminated. Epistemic uncertainty should be estimated to
produce a statistical distribution. This becomes somewhat
difficult as one is asked to estimate something that they, by
definition, have no knowledge of. As such, often uniform
distributions are utilized so the uncertain parameters are
assumed to fall somewhere within the specified bounding
box.
Ontological uncertainty is uncertainty due to completely
[7] applied to engineering analyses. Such uncertainty may be
due to phenomena that are unexpected and thus not accounted
for in the engineering model but are nonetheless present in
the physical system. This is especially problematic for PHM
systems where the phenomena that are well-understood may
only be dominant in the near term, and unknown or poorly
categorized phenomena dominate the response of the system
in the far term. Such uncertainties are notoriously difficult to
quantify, but some possible methods for roughly estimating
them is discussed below.
One of the difficulties in categorizing an uncertainty source
is that there is an apparent overlap in the definitions of
aleatory and epistemic uncertainty, which is best illuminated
via example. Material properties, as is discussed above, are
classified as aleatory due to the inherent inhomogeneity in the
material structure within the bounds of the measured
information. However, hypothetically one could measure the
complete microstructure of the material and incorporate this
new knowledge into the engineering model, which would
instead define the uncertainty as epistemic because only the
lack of knowledge about the actual microstructural variation
across the entire model is causing uncertainty. To avoid this
overlap of terms, one must define the knowledge that is
missing when dealing with epistemic uncertainty must be
relative to the engineering model [5]. This means that if the
model, as defined and implemented, is capable of
incorporating the additional knowledge about the physical
system without modification, the uncertainty is defined as
epistemic; if the model must be modified in order to
incorporate the additional knowledge about the physical
system, the uncertainty is defined as aleatory; and if the
model cannot be modified because the phenomena or
response are unknown, the uncertainty is defined as
ontological. So, if the engineering model associated with the
analysis used in the example above assumed homogeneous
materials, as is commonplace in engineering analyses, the
material property variation due to inhomogeneity is aleatory
uncertainty.
6. UNCERTAINTY IN PHM SYSTEMS
Prognostic health management systems have many of the
same sources of uncertainty as other engineering disciplines,
some of which are associated with the use of the same design
models for application into prognostic systems; however,
there are other uncertainty sources that are unique and of
particular interest (or concern) with PHM systems. Several of
these uncertainty sources are discussed in this section.
While typical engineering analyses deal with models of
systems with varying inputs based on engineering knowledge
and assumptions, PHM systems deal with inputs to the
models that come from physical sensing devices. These
sensors introduce several types of uncertainty that are not
present in other engineering analyses.
One such sensor-based source of uncertainty is quantization
error. Sensors are typically analog devices that detect and
measure some varying parameter in the physical world. These
analog data are captured by sensors that are measuring an
electrical representation of a physical parameter and the
capture analog information if conditioned and filtered. This
conditioned data is then converted into digital data via
analog-to-digital converters (ADC). These ADCs have a
fixed precision, typically in the range of 8 to 16 bits, and thus
a significant amount of precision in the analog value will be
lost by quantizing the conditioned value.
Because the system is capable of reading the actual analog
sensor value, but knowledge about that value is lost, this
model input uncertainty source is categorized as an epistemic
uncertainty. Quantification of this uncertainty source can be
straightforward by tracking uncertainty sources of each of the
elements of the measurement system. For example, the ADC
that is used in the sensor itself (if digital) or the external ADC
will be documented with the number of bits of precision that
5
it uses. That number of bits and the digital sensor value can
then be used to create a uniform distribution of the value of
the actual underlying physical parameter.
Similar to sensor quantization uncertainty, floating-point
numerical truncation and round-off errors introduce
uncertainty into all numerical calculations. This model
discretization epistemic uncertainty source is due to the finite
precision with which floating-point numbers are represented
in computer systems. As an example, one may be surprised
to find that the sum of 1,000,000 and 1 in single-precision
floating point arithmetic does not yield 1,000,001 but instead
yields 1,000,000. This is because a single-precision floating
point number on the order of a million does not have
sufficient precision to differentiate between the numbers
1,000,000 and 1,000,001; so the summation has no effect
whatsoever.
This uncertainty source is present in all numerical
calculations, but typically does not have a large effect unless
one deals with both large and small numbers simultaneously.
Quantification of this uncertainty source can be done via
inspection. Inspection of the numerical models must be made
by one familiar with floating-point arithmetic and who can
evaluate where in the model such uncertainty may arise. They
can then estimate the distribution of the error based on the
numerical precision.
Using our example above, the summation of a number on the
order of a million with a number on the order of one using
single-precision floating point arithmetic will introduce
uncertainty in the result that can be represented by a uniform
distribution with a width of approximately four. Thankfully,
when using double-precision floating point arithmetic, as is
common and recommended for engineering analyses, these
uncertainties are many orders of magnitude smaller;
however, they still exist and should be accounted for.
Sensors also introduce uncertainty due to natural variation of
the readings they provide. For example, a relative humidity
sensor may only be accurate to within 3%, so if a humidity
sensor gives a reading of 30% there is no way to know
whether the actual relative humidity was 27%, 33%, or
anywhere in between. Similar to sensor quantization
uncertainty, this model input uncertainty is epistemic in
nature. Quantification of this uncertainty source is simple, as
nearly all sensor manufacturers state the accuracy (or
variation) of their sensors in their product documentation and
data sheets.
One method to reduce the uncertainty caused by sensor
variations is to utilize multiple redundant sensors at each
location. Because the sensors are in the same location, they
should be measuring approximately the same underlying
physical value. The different sensor readings can then be
interpreted as statistical samples of a single value. With a
sensor with a fixed accuracy, such as the humidity sensor
example above, the values from the redundant sensors will
fall within a uniform distribution centered on the actual
physical value. Again, as per the Central Limit Theorem, by
averaging these multiple sensor readings, a better estimate of
this physical value can be obtained.
Uncertainty caused by sensor variation has an additional
complexity that is especially prevalent in PHM systems: the
sensors age and their variations will change with time. Over
the potentially long service lives of PHM systems, these
sensor variations can increase by orders of magnitude.
Unfortunately, few sensor manufacturers measure the
degradation of their sensors over such long periods of time
(potentially decades), and so quantification of this
uncertainty source is problematic and can introduce
ontological uncertainty risk into the measure information.
Manufacturers may state an expected lifetime or mean time
between failures of their sensors or components, but this does
not give sufficient information to determine the sensor
variation with time.
It is suggested that builders and operators of PHM systems
work with the sensor manufacturers to quantify the variation
of the sensors over long periods of time. This can be time-
consuming though, as decades of testing may be required in
order to quantify the variation of a sensor before the sensor
can be utilized within a PHM system. Alternately, a PHM
system builder should consider incorporating into their
systems multiple redundant sensors of each type and, in
addition to averaging their readings to reduce the sensor
variation uncertainty as described above, measure the
variation between each redundant sensor as time passes.
While this will not necessarily allow an accurate assessment
of the sensor variation uncertainty to be projected into the
future because such variation is typically nonlinear, it will
allow for the current sensor variation to be measured and a
rough approximation of that uncertainty to be projected (with
high additional uncertainty) into the future. This will amount
to the statistical distributions of sensor variation uncertainty
(e.g., the uniform distribution between 27% and 33%
discussed above) to be projected out into the future, yielding
distributions that expand with time.
Because sensor variation uncertainties can become large and
dominate the uncertainty in the PHM analysis, it is
recommended that sensors also be a part of the PHM analysis
and tracking, allowing for condition-based maintenance or
replacement of the sensors when variation uncertainty
becomes large as measured by the redundant sensors. A
discussion on the logistics of this is not made here, but the
implications on the statistical measures of the uncertainty are
important to recognize and track. Replacing sensors once
sensor variation uncertainty has the effect of limiting the
spread of the statistical distributions of the sensor variation
uncertainty. This will keep the sensor variation uncertainty
from dominating the PHM analysis, and thus it is highly
recommended that such sensor replacement strategies be
incorporated into any PHM system that must be operational
for a significant amount of time.
Another source of epistemic uncertainty that is important in
PHM systems is that of initial conditions. Many PHM aging
6
algorithms take into account both mechanical and chemical
aging of the materials in the system, both of which are very
sensitive to the early environmental conditions that the
materials experience. If the environment around these
materials is not monitored by the PHM system, as is likely
since the materials begin to age as they are being
manufactured and continue aging during storage, shipment,
and assembly with the monitored asset, this can present a
significant amount of uncertainty in the subsequent PHM
analyses.
Unless the environment that the materials see from their
beginning of life can be monitored by the PHM system,
which will likely require a significant expense as well as
coordination with material manufacturers and logistics
companies, the best way to quantify the uncertainty in the
initial conditions of the materials is to utilize material batch
testing. This involves measuring the different batches of
materials that will be utilized in the system as they are
delivered and utilized in manufacturing the system. Once the
initial state of and the uncertainty in the material properties
are known, the PHM system can utilize these values as the
starting point for the prognostic analyses. In addition to
making the question of the environmental conditions prior to
monitoring moot, this technique will also quantify the
uncertainty in the homogeneity and batch-to-batch
consistency of the materials, both of which introduce further
model input epistemic uncertainty.
Just like when utilizing multiple sensors to reduce the sensor
variation uncertainty, when testing material properties one
must take multiple measurements of each property in order to
quantify the uncertainty in both the material and in the
measurement process. The process of calculating the
underlying distributions of those parameters is identical, and
for the case that the underlying distribution is normal results
in a normal distribution of the mean and a Chi-Squared
distribution of the variance, again as per Cochran’s Theorem
[6] (Equations 1-8). This means that after measuring the
material properties multiple times, one does not know the
exact underlying distribution of the physical property, but
they do know the statistical distribution of that underlying
physical distribution, which can be utilized within the
subsequent PHM analyses.
Similar to initial material property variability, the initial
geometry of the monitored system is likely to vary for a
variety of reasons including manufacturing tolerances and
processes. For many systems, this relatively small variability
in geometry will not have a large impact on the uncertainty
in the result of the PHM analysis, but in other cases it may,
so it is recommended that an estimate of this uncertainty be
incorporated into the PHM analysis. Whether this model
input uncertainty source is aleatory or epistemic depends
upon the model being utilized in the analysis. For most
models, the geometry of the system is assumed to be fixed,
and thus the uncertainty is aleatory; however, other models
may treat the geometry as variant, and thus the uncertainty is
epistemic. In either case, the best method of quantifying this
uncertainty is to measure critical dimensions on each asset
and account for the measurement uncertainty in the
uncertainty rollup.
Many PHM systems monitor all assets in the fleet of assets;
however, some systems monitor only a subset of the fleet
assets in order to determine the overall health of the fleet.
This can be done statistically, but it does introduce further
epistemic model input uncertainty to the analyses. Just like
the sampling of material properties discussed above, the state
of the individual assets in the fleet can be estimated from the
state of the assets that were sampled. If the distribution of the
states of the assets is assumed to be normal, then equations 1
through 8 can again be utilized to calculate distributions of
the underlying distributions. In practice, the distribution of
the states of the assets is highly unlikely to be a normal
distribution though, so additional statistical methods not
discussed here should be employed.
For large assets being monitored by PHM systems such as
aircraft, ships, and rocket motors, the variation of
environmental conditions across the asset can be a significant
source of epistemic uncertainty. Quantification of this
uncertainty is a two-fold process. First, the effects of varying
boundary conditions on the model must be ascertained by
running several analyses with differing variations of
conditions across the asset. Then the actual variation of
conditions across the asset must be measured using multiple
sensors positioned on and around the asset. The latter
uncertainty can then be quantified and used as an input into
the former uncertainty to calculate an overall quantification
of this uncertainty source.
In a PHM system, large amounts of data including sensor data
values and calculated asset health data must be transferred
across networks and stored in a data warehouse. There are
many different methods of transferring and storing this data,
and several of those methods involve compressing the data in
order to reduce network bandwidth and storage space [8] [9]
[10].
This data compression may have the effect of introducing a
form of epistemic discretization uncertainty into the data.
Determining the uncertainty that is introduced due to this
compression can be problematic because internal details of
the compression algorithm and how it operates on specific
data values must be known. Because storage space is
relatively cheap in comparison with the costs associated with
introducing significant uncertainty into an analysis and the
resulting potential over-design of a product, lossless
compression algorithms can be utilized in PHM system data
warehouses, which can assist in eliminating this source of
uncertainty.
Similar to the above, one must consider uncertainty that
enters into the model due to unknown boundary conditions
and physical mechanisms. These model input and model form
ontological uncertainties are typically due to the asset being
in an environment or encountering new physics that were not
considered when building the model. For example, the health
7
of a propeller may be dependent on many factors that are
incorporated into the PHM model, but if that propeller is
allowed to operate in a condition that was not considered, like
a sandstorm, there will be a great deal of uncertainty between
the predicted health of the propeller and the observed health.
Like any other ontological uncertainty, estimating this source
of uncertainty is very difficult. One method of coming up
with a very rough estimation is to take all of the physics that
are built into the PHM model and quantify the change in the
response of the model when combinations of those physics
are disabled. Extending the propeller example above, if the
propeller model included phenomena such as rain impact and
icing, then differences in the model when these phenomena
are included and excluded will give a rough order-of-
magnitude estimation of the uncertainty due to other similar
phenomena, such as sand impacts, not being included in the
model.
One additional model form uncertainty source should be
discussed; that of human errors in the development of the
models and their numerical implementations. The estimation
of this uncertainty source is nearly impossible, so it is best to
strive to eliminate this source instead of quantifying it. While
total elimination of unknown human errors is not a feasible
goal, utilizing several different processes while developing
the models and implementations can reduce the likelihood of
these sorts of errors significantly.
It is recommended that formal peer reviews be held at all
levels of development, from theory and model development
to algorithm development to software development. Many
errors can be discovered simply by having one or more peers
step through and review the development process. Specific to
software development, the use of test-driven development is
another method to reduce errors. This development
methodology uses requirements to build test cases, which
must be peer reviewed, and then development is done in order
to make the numerical models successfully run the test cases,
and thus successfully implement the numerical methods and
fulfil the requirements of the program. This is backwards
from the usual software development methodology where
test cases are written after the fact to test the software that
was built to the requirements. Formalizing the requirements
of the numerical implementation and adding the additional
layer of peer review of these requirements ensures that the
software is developed correctly and in alignment with the
developed models.
One final area of uncertainty needs to be accounted for, that
of ontological uncertainty associated with the unknown
unknowns. One method of accounting for this is by
understanding margins in the design and using these to
account for unknown effects. This method can compensate
for the ontological risk in evolutionary systems, but to
account for these risks is difficult for new revolutionary
systems due to the lack of baseline data.
 7. CALCULATION WITH UNCERTAINTY
Once all of the uncertainty sources in an engineering or PHM
analysis have been identified and quantified, those
uncertainties must be utilized within the engineering
calculations and models in order to roll up the uncertainty and
determine the variability and repeatability of the analysis due
to those uncertainties.

The traditional method for utilizing such uncertainties in an

engineering analysis is to sample all of the variables based on
random variation for each statistical distribution within a
single Monte Carlo loop, as seen in Figure 2. The underlying
models are then run repeatedly with the different sets of
sampled inputs, and the outputs from the models can be
analyzed in order to determine the statistical variation of
those outputs. While this method does indeed provide the
needed quantification of the variation of the outputs, it does
not separate the epistemic uncertainty sources from the
aleatory uncertainty sources. This means that the variation of
the outputs due to natural variation of the inputs (aleatory
uncertainty), which are real variations that will be observed
in the physical system, is not separated from the variation of
the outputs due to lack of knowledge (epistemic uncertainty),
which will not manifest itself in the physical system and will
only affect the repeatability of the analysis.

Figure 3. Two-tier Monte Carlo algorithm that gives

information about the variability and repeatability of
the uncertainty quantification analysis

In a PHM system, the underlying model typically provides a

Figure 2. Typical single-tier Monte Carlo algorithm used
in uncertainty quantification analyses
In an attempt to address these shortcomings of a single Monte
Carlo loop, it has been suggested that a two-tier Monte Carlo
loop be utilized in order to separate the epistemic and aleatory
uncertainties [5]. In such an analysis, an inner Monte Carlo
loop is used to sample aleatory uncertainty sources in order
to capture the variation that is expected to be observed in the
physical system. A second outer Monte Carlo loop is then
utilized to sample the epistemic uncertainty sources in order
to capture the lack of knowledge about the physical system
and quantify the repeatability of the analysis. Figure 3 gives
a representation of such a two-tier Monte Carlo analysis.
pass/fail of the asset. For example, a load may be calculated
based on boundary conditions and compared to a material
allowable to calculate whether the material fails. This simple
calculation of pass/fail is typically run forward in time to
create a Boolean pass/fail curve. The inner Monte Carlo loop
then samples the aleatory uncertainty sources, runs this
simple calculation many times, and statically analyzes the
resulting Boolean curves to determine a single probability of
failure that varies with time. The outer Monte Carlo loop is
then used to sample epistemic uncertainty sources and
generate many different probability of failure curves that
represent the variability in the analysis due to a lack of
knowledge about the physical system.
Sample results from such an analysis can be seen in Figure 4.
The first step in analyzing these results is to generate a P-box
[11] around the probability of failure curves with a specific
confidence. This is done by determining the area within the
plot that contains the specific percentage of the curves. The
upper and lower limits of this P-box are shown in Figure 4 as
orange curves.

8
 Figure 4. Sample probability of failure curves from a
Figure 5. Identifying the range of probabilities of failure
two-tier Monte Carlo analysis of an asset monitored by a
at four years from credible P-box with 90% confidence
PHM system. The gray curves are individual samples
from the outer Monte Carlo loop; the orange curves
define the 90% confidence P-box.

One can now intersect horizontal and vertical lines in the plot
with these credible P-box curves and draw statistical
conclusions from those intersections. For example, drawing
a vertical line on our sample plot at four years and
intersecting it with the 90% credible P-box shows that the
range of probabilities of failure that we are 90% confident
that the physical system will have at four years is 0% to 25%
(Figure 5). If instead we are interested in replacing or
refurbishing our asset when it reached a probability of failure
of 50%, we could draw a horizontal line on the plot at 50%
probability of failure and intersect that with the credible P-
box to show that the range of the lifetime of the asset prior to
it needing refurbishment or replacement with 90%
confidence is 4.2 to 5.7 years (Figure 6).

Figure 6. Identifying the range of refurbishment ages

from credible P-box with 90% confidence

Until now we have only been varying aleatory and epistemic

uncertainty sources, and ontological sources have not been
considered. One could utilize a three-tier Monte Carlo
algorithm to also sample these ontological uncertainties and
roll them into the analysis, but the difficulty in quantifying
such uncertainty sources renders the benefits of performing
such an analysis negligible. It is recommended that the two-
tier Monte Carlo loops be utilized in all cases.

In the typical PHM model, there are a very high number of

inputs (e.g., material properties, boundary conditions that
vary with time, etc.). This means that a large number of
Monte Carlo iterations must be utilized in both tiers of the
two-tier approach in order to attain proper convergence. This,
combined with the complexity of the underlying models,

9
means that run times for such analyses can easily become
intractable.
There are several methods that can be utilized to reduce the
run times of such analyses. One such method is to increase
the processing power of the computer system that is running
the analyses. This is typically done by one of two methods:
utilizing parallel processing and utilizing general purpose
graphics processing units.
Parallel processing is very simple to implement for a Monte
Carlo algorithm because the algorithm itself is
“embarrassingly parallel”. That is to say that the iterations
within the algorithm are completely independent of one
another, and thus they can be run simultaneously and get
near-perfect speedup without much additional software
development effort. The use of parallel processing can reduce
run times of Monte Carlo algorithms by near a factor of the
number of cores that are utilized. If one has access to a very
large high performance computer system, this can be up to
seven orders of magnitude [12]. However, the costs
associated with running analysis on such large systems can
be prohibitive, especially with a PHM system where the
analyses need to be continuously rerun as new sensor data is
acquired.
Calculation with general purpose graphics processing units
involves leveraging the high-performance graphics cards on
modern computer systems for computation instead of
graphics-related tasks. Table 1 compares the processing
power of a modern central processing unit (CPU) with a
modern graphics processing unit (GPU) by comparing the
number of floating point operations per second (FLOPS) that
each can theoretically achieve, showing that speedups of
nearly two orders of magnitude can be achieved by utilizing
a GPU instead of a CPU. Note that actual FLOPS figures are
likely to be significantly below these theoretical peak figures.
Also note that GPUs are more optimized for lower-precision
calculations, whereas CPUs see no theoretical benefit in
utilization of half-precision floating-point operations because
their hardware does not directly operate on them [13].
Table 1. Comparison of floating-point calculations per
second (FLOPS) for the Intel i7-8700K CPU and the
NVIDIA Quadro RTX 6000 GPU
Peak FLOPS
Double Single Half
Type Model
Precision Precision Precision
Intel
CPU i7-8700K 5.64×1011 1.13×1012 1.13×1012
[14]
NVIDIA
Quadro
GPU 7.40×1012 1.63×1013 3.26×1013
RTX 6000
[15]
Another method to reduce the run times is to reduce the
number of inputs to be varied, and thus reduce the number of
Monte Carlo iterations that will be needed to attain
10
convergence. In order to reduce this number while not greatly
affecting the statistical results, one should seek to eliminate
variation in input parameters that do not have a large effect
on the results when varied within their statistical
distributions. To this end, sensitivity studies must be
performed on the inputs. In such studies, the analysis models
are run while varying individual parameters within their
statistical distributions (e.g., a normally-distributed variable
may be varied +/- 3σ). The variation of the results due to these
input variations are then tabulated, and those parameters that
have a negligible effect on the results can be treated as
constant in the Monte Carlo analysis.
Different sampling methods can also be utilized with the
Monte Carlo algorithm to aid in convergence. The Latin
Hypercube Sampling (LHS) method may provide better
sampling, and thus require fewer iterations to attain
convergence, but the magnitude of the improvement is highly
dependent on the model. If the model has a high number of
input parameters and there are significant nonlinearities
between the input parameter terms (both of which are
common in PHM models), the improvements are
significantly reduced. Furthermore, because additional work
must be done to generate the sampling points with LHS, it is
possible that run times with LHS may be slower than with
standard Monte Carlo random sampling. This is an area
where additional research should be done in order to quantify
any performance improvements when used with PHM
models.
8. CONCLUSIONS
In the aerospace industry, systems are counted on to work
reliability. Most missions involve critical capabilities either
for human life, scientific studies, or defense applications.
Solid rocket motor systems in particular are counted on to
work the first time every time. This means we must
understand the product thoroughly and ensure that the system
capability can meet or exceed the demands of the mission.
Use of PHM systems that accurately capture and quantify
uncertainty are key to gaining this understanding.
Uncertainty quantification is a key aspect of all engineering
analyses, but it is especially important in PHM because the
uncertainty in the outputs of the engineering models are one
of the primary outputs that must be analyzed and acted upon.
In order for the uncertainty in a PHM analysis to be properly
accounted for, all sources of uncertainty must be identified,
categorized, and quantified. A discussion of some common
PHM-related uncertainty sources, the categorization of those
sources, and methods to quantify them has been presented.
It is recommended that the asset health calculations in a PHM
system with proper uncertainty quantification be performed
with a nested two-tier Monte Carlo loop in order to quantify
the variability and repeatability of the analysis based on
epistemic and aleatory uncertainty sources. A variety of
methods have been presented that have the potential to
dramatically reduce the run times of the Monte Carlo
implementation, but additional research needs to be done in
this area specific to PHM analyses in order to quantify the
potential gains.
The output from the two-tier Monte Carlo analysis in a PHM
system is typically a set of reliability curves that vary with
time. The smooth change in reliability with time is due to
aleatory uncertainty sources, whereas the variation between
reliability curves is due to epistemic uncertainty sources.
Post-processing these curves allows for calculation of both an
asset health assessment and the confidence in that
assessment. It is suggested that such a method for quantifying
uncertainty be utilized within all PHM systems, particularly
those monitoring high-value assets, in order to produce the
highest quality actionable quantifications of health for the
assets being monitored.
11
REFERENCES
[1] International Council on Systems Engineering
(INCOSE) SE Handbook Working Group, "INCOSE
Systems Engineering Handbook v. 3.2.2," 2011.
[2] S. Hyde, D. Richardson, B. Allen and B. Goldberg,
"Model Based Design Influence on Program Testing
Programs Part I," in AIAA Missile Sciences
Conference, AIAA Defense and Security Forum,
Laurel, MD, 2015.
[3] R. Smith, "Chapter 1," in Uncertainty Quantification:
Theory, Implementation, and Applications,
Philadelphia, SIAM, 2014.
[4] D. DeVries, S. Hyde and I. L. Davis, "A Systems
Engineering Approach to Evolution of Physics-based
Prognostic Health Management of Aging Solid Rocket
Motor System Assets," AVT-268 RSM, Utrecht,
Netherlands, 2017.
[5] M. Ewing, B. Liechty and D. Black, "A General
Methodology for Uncertainty Quantification in
Engineering Analyses," Journal of Verification,
Validation and Uncertainty Quantification, vol. 3, no.
2, 2018.
[6] W. G. Cochran, "The distribution of quadratic forms in
a normal system, with applications to the analysis of
covariance," in Mathematical Proceedings of the
Cambridge Philosophical Society, Cambridge, 1934.
[7] D. H. Rumsfeld, 12 February 2002 News Briefing on
Weapons of Mass Destruction in Iraq, Washington
D.C., 2002.
[8] B. Furht, "A Survey of Multimedia Compression
Techniques and Standards Part I JPEG Standard,"
Real-Time Imaging, vol. 1, no. 1, pp. 49-67, 1995.
[9] N. Kimura and S. Latifi, "A Survey on Data
Compression in Wireless Sensor Networks," in
International Conference on Information Technology:
Coding and Computing, 2005.
[10] W. Su, O. B. Akan and E. Cayirci, "Compression
Techniques for Wireless Sensor Networks," in
Wireless Sensor Networks, Springer Science+Business
Media, 2006, pp. 207-232.
[11] S. Ferson and W. T. Tucker, "Sensitivity Analysis
Using Probability Bounding," Reliability Engineering
& Systems Safety, vol. 91, pp. 1435-1442, 2006.
[12] "TOP500 June 2018," TOP500.org, June 2018.
[Online]. Available:
https://www.top500.org/lists/2018/06/. [Accessed 28
September 2018].
[13] P. Konsor, "Performance Benefits of Half Precision
Floats," 15 August 2012. [Online]. Available:
https://software.intel.com/en-us/articles/performance-
benefits-of-half-precision-floats. [Accessed
September 2018].
[14] Intel, "Intel Core i7-8700K Processor," [Online].
Available: https://ark.intel.com/products/126684/.
[Accessed 28 September 2018].
[15] E. Kilgariff, H. Moreton, N. Stam and B. Bell,
"NVIDIA Turing Architecture In-Depth," 14
September 2018. [Online]. Available:
https://devblogs.nvidia.com/nvidia-turing-
architecture-in-depth. [Accessed 28 September 2018].
BIOGRAPHY
Mr. Heath Dewey is a Software
Engineer in the Propulsion Systems
Division of Northrop Grumman
Innovation Systems (formerly ATK
and Orbital ATK) at Promontory,
Utah. He received a B.S. from Iowa
State University in 2004 and a M.S.
28 from the University of Illinois at
Urbana-Champaign in 2006, both in
Aerospace Engineering. Since that time, he has worked in
his current position developing modeling and analysis
tools that are used in simulation of complex physics and
phenomena that occur inside solid rocket motors during
firing and flight. In recent years, he has also led several
software engineering teams in developing data collection
and prognostic health management systems for monitoring
strategic solid rocket motor fleets. He and his wife Kendra
have been married for 12 years and have two children.
Mr. Derek R. DeVries, P.E. is a
Senior Fellow and Propulsion
Systems discipline owner of Avionics
and Control Disciplines for Northrop
Grumman Innovation Systems He is a
licensed Professional Engineer,
Senior Member of the Institute of
Electrical and Electronics Engineers,
and U.S. Air Force veteran. He was
awarded the NATO Science and Technology
Organization’s 2016 Scientific Achievement award for
Application of Integrated Munition Health Management.
He was honored as an invited Luminary speaker at the
international PHM conference in 2015. He has over 30
years of experience in the aerospace industry in operation,
integration, and development of space and strategic
launch systems. He received his B.Sc. degree in Electrical
Engineering from the University of Utah and his M.Sc.
degree in Electrical Engineering from Utah State
University. He served as a member of the Industrial
Advisory Board for the University of Utah Department of
Electrical Engineering and an American Institute of
Aeronautics and Astronautics Standards Committee
Member for development of the industry standard “S-122-
2006 Direct Current Power Systems for Earth-Orbiting
Satellites.” He is currently a member of the Society of
Automotive Engineers PHM Standards Committee. He has
14 U.S. and foreign patents and numerous published
papers. He was awarded the AIAA 2001 Joint Propulsion
Conference Arthur D. Rhea Award for Best Paper of the
Year presented in Ordnance Components and Systems.
Mr. DeVries is a leader in developing technologies in
aerospace systems including stewarding the development
of new significant aerospace projects in component and
system development and integration. He has and continues
to support active development, operational, and
sustainment activities of Air Force and Navy Long Range
Missile Systems, NASA Space Shuttle and Deep Space
12
 missions and development of their next generation launch
vehicles. He has led or been an expert member of several
investigation teams for flight failures charged with
determining root cause and then recommending and/or
leading corrective actions for return to flight. He and his
beautiful wife Jodie have been married for over 35 years
and have five children and nine grandchildren.

Mr. Scott R. Hyde is a Senior Staff

Scientist at Northrop Grumman
Innovation System’s Flight Systems
Group. Throughout his career he has
held many different positions including
Design Engineer, Chief Engineer,
Department Manager, Technology
Program Manager, Business
Development Manager and Senior Scientist. He has over 30
technical publications. Scott received his B.S. in 1986 and his
M.S. in 1992 in Structural Engineering from Utah State
University in Logan, Utah. He has a passion for Integrated
Vehicle Health Management technologies and has
contributed to their development and implementation
throughout his aerospace career. He has spent 30 years
working on improving the management of solid rocket motor
systems. He led a multidiscipline team of engineers and
scientists who developed, verified and validated a
fundamental physics based methodology for predicting the
complex behavior of solid rocket motor propellant. He and
his wife Tawni have been married for more than 35 years and
have four children and are enjoying their first grandson.

13