Continuous Markov Processes

A H Chowdhury, PhD
Professor
Dept. of EEE, BUET
 Introduction
• Reliability problems normally concerned with systems that are
– discrete in space  exist in one of a number of discrete and identifiable states
– continuous in time  exist continuously in one of the system states until a
transition occurs which takes them discretely to another state in which they then
exist continuously until another transition occurs
• Stationary Markov processes  conditional probability of failure or
repair during any fixed interval of time is constant
• Limiting probabilities independent of state residence time
distributions, depends only upon their mean values
• Given these conditions Markov approach can be used for a wide
range of reliability problems  systems that are either non-repairable
or repairable, series-connected, parallel redundant or standby
redundant
 General Modeling Concepts

Transition rate concepts

• Consider a single repairable component with constant failure rate and
repair rate  characterized by exponential distribution

Single component
repairable system state
space diagram

Variation of reliability and

time-dependent availability
 General Modeling Concepts contd.

• Define:
Po(t) = probability that the component is operable at time t
P1(t) = probability that the component is failed at time t
λ = failure rate
μ = repair rate
• Failure density function for a component with a constant
hazard rate of λ 
• Density functions for operating and failed states of system

• Parameters λ and μ referred to as state transition rates  rate

at which system transits from one state of system to another
 General Modeling Concepts contd.

• Failure rate λ  reciprocal of mean time to failure (MTTF)

with times to failure counted from moment the component
begins to operate to the moment it fails

• Repair rate μ  reciprocal of mean time to repair (MTTR)

with these times counted from the moment component fails
to the moment it is returned to an operable condition
 General Modeling Concepts contd.

• Failure and repair rates sometimes incorrectly evaluated by

counting number of failures or repairs in a given period of
time, and dividing by elapsed time

• Correct time value to use in denominator is portion of time in

which component was in the state being considered

• Definition of transition rate

transition rate = number of times a transition occurs from a given

state/time spent in that state
 Evaluating Time Dependent Probabilities

• Consider an incremental interval of time dt

– dt sufficiently small so that probability of two or more events occurring
during this increment of time negligible

• Probability of being in operating state after this time interval dt,

i.e., probability of being in state 0 at time (t+dt)
= [Probability of being operative at time t AND of not failing in time dt]
+ [probability of being failed at time t AND of being repaired in time dt]
 Evaluating Time Dependent Probabilities
contd.

• Using a similar approach used to develop Poisson distribution

Then,

Similarly, and

Thus, This is not a stochastic

transitional probability matrix
 rows of this matrix
summate to zero
 Evaluating Time Dependent Probabilities
contd.

linear differential equations with constant

coefficients

• Performing Laplace transform



• Similarly, from 

Pi(s)  Laplace transform of Pi(t)

P0(0)  initial value of P0(t)
 Evaluating Time Dependent Probabilities
contd.

• Solving for P0(s) and P1(s) as linear simultaneous equations 

• Transforming back into time domain and noting that 


 Evaluating Time Dependent Probabilities
contd.

• Most likely state in which system starts is state 0, i.e. System

in an operable condition at zero time  P0 (0) = 1 , P1(0) = 0

• P0(t) and P1(t)  probabilities of system being in operating

state and failed state, respectively, as a function of time given
that system started at time t = 0 in operating state
 Evaluating Limiting State Probabilities
• Limiting state probabilities will be non-zero for a continuous
Markov process provided system is ergodic (i.e. system independent of
initial conditions)

• For the single component system limiting probabilities can be

evaluated by letting t→


• These limiting state probability expressions applicable irrespective

of whether system starts in operating state or in failed state
 Evaluating Limiting State Probabilities contd.

• For exponential distribution

MTTF = m = 1/λ 

MTTR = r = 1/μ 

• Values of P0 and P1 generally referred to as steady-state or

limiting availability A and unavailability U
• The time dependent availability A(t) of the system
 Evaluating Limiting State Probabilities contd.

•  Availability is probability
of being found in operating state at some time t in future
given that system started in operating state at time t = 0

• Quite different from reliability 

• Reliability is the probability of staying in operating state as a

function of time given that system started in operating state
at time t = 0
• Similar conceptual relationship exists between unavailability
U(t) and unreliability Q(t)
 State Space Diagrams

• State space diagram facilitate solution of continuous or

discrete Markov processes

• It includes all relevant states in which system can reside and

all known ways in which transitions between states can occur

• Translates analyst's knowledge of operation of system into a

mathematical model that can be solved by using Markov
techniques
 State Space Diagrams contd.

• No mathematical model can eliminate the need to exercise

engineering judgement and requirement of a thorough and
exhaustive understanding of the physical and logical operation
of system

• Analyst first translate operation of system into a state space

diagram recognizing both states of system, way these states
communicate and values of transition rates
 State Space Diagrams contd.

Single repairable component

Failure rate Repair rate

State space diagram of component

that can be either in operating, or
up state, and failed, or down state

μ2 may not physically exist in practice 

once failed, repair process may return it State space diagram of component with
to full output state partial output state, e.g. a pump
 State Space Diagrams contd.

Two repairable components

State space diagram same irrespective
of whether components are in series
or parallel redundant
Series system state 1 is up state,
states 2, 3, 4 down states
Parallel redundant system states 1, 2,
3 up states and state 4 down state
P1,P2, P3, P4  probabilities of being in
states 1 to 4
 State Space Diagrams contd.

Two components variation

• If one of the components fails in
a series system, the other
component no longer operates
 its failure rate in these
circumstances becomes zero
• In this case, state 4 does not
exist leaving only states 1-3 and
transition rates between these 3
states
 State Space Diagrams contd.

Two identical components

• If both of components are identical,

states 2 and 3 are also identical 
combine to give a reduced 3-state system
• 2λ and 2μ terms indicate that two
components are available for failure or
repair respectively in next increment of
time and that one of two can fail or be
repaired, but not both in that interval
 State Space Diagrams contd.

Three component system U  component up

• Maximum number of states in a D  component down

3 component system, where

each component can exist in
two states  23 or 8

Series system

Parallel
redundant
system

2-out-of-3 system
 State Space Diagrams contd.

Standby redundant systems

• Consider the two component
standby system
• It is assumed that component A
is the normally operating
component
• Whenever A operable, it replaces
B as operating component
• Subscripts 0, f and s represent State space diagram for two component
operating, failed and stand-by standby system
respectively
 State Space Diagrams contd.

• It is most important that to recognize and identify states in which

system can reside and transitions that can occur between these
states

• Example
1. When A and B identical, it may be operating policy to leave B operating and
place A on standby to give a state equivalent to state 1 with status of
components reversed

2. If switch is not perfect, each state of diagram must include status of not
only A and B but also of switch itself
 State Space Diagrams contd.

Mission orientated systems

• State space diagrams can be constructed for non-repairable or
mission orientated systems and can be solved using Markov
techniques
• In non-repairable systems repair transitions do not exist
Two identical non-repairable components

System no longer ergodic since not all

states can communicate and one state
is an absorbing state
Two identical repairable components
 State Space Diagrams contd.

• Time dependent probabilities of non-repairable systems can

be evaluated by using Markov process techniques in same
way as for repairable systems

• Limiting state probabilities do not have any significance since

 in the limit, probability of residing in absorbing state unity
and probability of residing in all non-absorbing states zero