Markov Analysis

Jørn Vatn
NTNU

1
Introduction
 Markov analysis is used to model systems which have
many different states
 These states range from “perfect function” to a total fault
state
 The migration between the different states may often be
described by a so-called Markov-model
 The possible transitions between the states may further be
described by a Markov diagram

2
Purpose
 Markov analysis is well suited for deciding reliability
characteristics of a system
 Especially the method is well suited for small systems with
complicated maintenance strategies
 In a Markov analysis the following topics will be of interest
 Estimating the average time the system is in each state. These
numbers might further form a basis for economic considerations.
 Estimating how frequent the system in average “visits” the various
states. This information might further be used to estimate the need
for spare parts, and maintenance personnel.
 Estimate the mean time until the system enters one specific state,
for example a critical state.

3
Markov Analysis procedure

1. Make a sketch of the system

2. Define the system states
3. Draw the Markov diagram with the transition rates
4. Quantitative assessment
5. Compilation and presentation of the result from the
analysis

4
Make a sketch of the system
 Pump system wit active pump and a spare pump in
standby

Active
pump

Standby
pump

5
 Definition of system states
 x1 = state of active pump
 x2 = state of standby pump

1 if component i is functioning
�
xi = �
0 if component i is in a fault state
�
System state Component state Comments
xS x1 x2

2 1 1 Both pumps functioning

1 0 1 The active pump is in a fault state, the
standby pump is functioning
0 0 0 Both pumps in a fault state
6
State transitions
 For this system we have assumed that if the active pump fails, the
standby pump could always be started
 Further we assume that if both pumps have failed, they will both be
repaired before the system is put into service again
 The following transition rates are defined
1 = failure rate of the active pump
2 = failure rate of the standby pump (while running, 2 = 0 in
standby position)
1 = repair rate of the active pump (1/1 = Mean Down Time when
the active pump has failed)
B = repair rate when both pumps are in a fault state. I.e. we
assume that if the active pump has failed, and a repair with repair
rate 1 is started, one will ”start over again” with repair rate B, if the
standby pump also fails, independent of “how much” have been
repaired on the active pump.

7
Markov state space diagram
 The circles represent the system states, and the arrows
represent the transition rates between the different system
states
 The Markov diagram and the description of states
represent the total qualitative description of the system

8
Quantitative assessment

 We want to assess the following quantities

 Average time the system remain in the various
system states
 The visiting frequencies to each system state

9
 �a00 a01 L a0 r �
� a1r �
Transition matrix �
a10 a11 L
�
A=� �
�M aij �
 The indexing starts on 0, and moves � �
to r, e.g. there are r +1 system states �ar 0 a r1 L arr �
 � �
Each cell in the matrix has two indexes,
where the first (row index) represent the
”from” state, whereas the second
(column index) represent the “to” state.
 The cells represent transition rates from one state to
another
 aij is thus the transition rate from state i to state j
 The diagonal elements are a kind of ”dummy”-elements,
which are filled in at the end, and shall fulfil the condition
that all cells in a row adds up to zero

10
Example transition matrix: (From , To )

0 1 2
0 �-B 0 B �
A= �
1 �2 -2 - 1 1 ��
2 �
�0 1 -1 �
�

11
State probabilities

 Let Pi(t) represent the probability that the system is in

state i at time t
 Now introduce vector notation, i.e.
 P(t) = [P0(t), P1(t),…,Pr(t)]
 From the definition of the matrix diagram it might be
shown that the Markov state equations are given by:

P(t) A = d P(t)/d t

 These equations may be used to establish both the steady

state probabilities, and the time dependent solution

12
Steady state probabilities

 Let the vector P = [P0, P1,…,Pr] represent the average time

the system is in the various system states in the long time
run
 For example, P0 is average fraction of the time the system
is in state 0, P1 is average fraction of the time the system
is in state 1
 The elements P = [P0, P1,…,Pr] are also denoted steady
state probabilities to indicate that in the stationary situation
Pi represents the probability that the system is in state i.

13
The steady state solution
 In the long run when the system has stabilized we must
have that d P(t)/d t = 0, hence
 PA = 0
 This system of equations is over-determined, hence we
may delete one column, and replace it with the fact that
P0+ P1+…+Pr = 1
 Hence, we have

14
The steady state solution

P A1 = b

where
�a00 a01 L 1�
�a10 a11 L 1�
A1 = � �
�M �
� �
�a r 0 ar1 L 1�
and
b = [0,0, …,0,1]

15
Example
�- B 0 1�
[ P0 P1 P2 ] �2 -2 - 1 1�= [ 0 0 1]
� �
�
�0 1 1�
�
which gives

12
P0 =
(2   B )1  (2  1 )  B

 B 1
P1 =
(  2   B ) 1  (  2   1 )  B

 B ( 2   1 )
P2 =
( 2   B )1  ( 2   1 )  B
16
Numerical solution

 To solve the steady state equations P A1 = b is a tedious

task
 Often we therefore solve these equations by numerical
methods
 The Markov.xls program does this, where we have to:
 Define the transition rates
 Assign numerical values to the transition rates
 Specify the Markov state space matrix

17
18
Visiting frequencies
 Often we are interested in evaluating how many times the
system enters the various states, i.e. the visiting
frequencies
 The visiting frequency for state j is denoted j, and could
be obtained by:
 j = -Pjajj
 From our example we obtain the “system failure rate”

B 12
 0 = - P0a00 =
(2   B )1  (2  1 )  B

19
Time dependent solution
 Up to now we have investigated the steady state situation
 In some situations we also want to investigate the time
dependent solution, i.e. the probability that the system is in
e.g. state 0 at time t
 We now let Pi(t) be the probability that the system is in
state i at time t
 The time dependent solution may be found by:
 P(t) A = d P(t)/d t
 Which could be solved by Laplace methods, or numerical
methods
 For numerical methods we apply Markov.xls

20
Spare parts – Simple model
 Assume a spare part regime where there is only one
spare in the stock
 Upon a demand (with demand rate ) a new spare is
ordered
 The intensity of arrival of a new spare is  = 1/ MTA
(Mean Time to Arrival)

�- �
A=�
� - �
�

21
Spare parts 2 spares in stock
 Assume a spare part regime where there are two
spares in the stock
 Upon a demand (with demand rate ) a new spare is
ordered
 The intensity of arrival of a new spares is  = 1/ MTA
(Mean Time to Arrival) independent of how many in
order 0 1 2
0 �- 0 �
A= �
1 � - -   ��
2 �
�0  - �
�

22