Markov Chains

Chia-Ping Chen

Professor
Department of Computer Science and Engineering
National Sun Yat-sen University

Probability
Dependence between Random Variables

In the discussion of Bernoulli or Poisson processes, it is

assumed that random variables at different times are
independent.
In general, random variables at different times are often
not independent.
A Markov process, also called a Markov chain, allows
dependence of random variables.
To maintain tractability, conditional independence
property is assumed.
That is, random variables are independent conditioned on
any random variable between them.

Prof. C. Chen Markov Chains

Discrete-time Markov Chains

Prof. C. Chen Markov Chains

Conditional Independence

X is independent of Y given Z if

P(X = x|Y = y, Z = z) = P(X = x|Z = z)

for the image of X, Y and Z.

This relationship between X, Y , and Z is called conditional

independence, which is denoted by

X
Y |Z

Prof. C. Chen Markov Chains

Markov Assumption

The Markov assumption for a discrete-time Markov chain

S0 , S1 , . . .

is the conditional independence

{Sn+1 , Sn+2 , . . . }
{Sn1 , . . . , S0 } | Sn , nN

The future is independent of the past given now.

Prof. C. Chen Markov Chains

State

The value of a Markov chain at a time is called a state.

The set of states is called state space.

A sample path of a Markov chain is a sequence of states.
A state transition occurs from one time to the next time,
including the case of self-transition.

Prof. C. Chen Markov Chains

Representation
A Markov chain can be represented by a state transition
graph or a transition probability matrix.

State transition graph

node: state
edge: state transition
annotation: transition probability

Transition probability matrix

p11 p12 ... p1m
p21 p22 ... p2m


T=
.. .. .. ,
.. pij = P(Sn+1 = j|Sn = i)
. . . .
pm1 pm2 . . . pmm

Prof. C. Chen Markov Chains

Example 7.1 Catch-up

Alice is taking a probability course.

In each week, she is either up-to-date or fallen-behind.
If she is up-to-date in a given week, the probability that
she will be up-to-date in the next week is 0.8.
If she is fallen-behind in a given week, the probability that
she will be up-to-date in the next week is 0.6.
Construct a Markov chain for Alice.

Prof. C. Chen Markov Chains

Example 7.2 Flies and Spiders

A fly moves along a straight line in unit increments.

At each time period, it moves one unit to the left with
probability 0.3, one unit to the right with probability 0.3,
and stays in place with probability 0.4.
Two spiders are lurking at positions 1 and m: if the fly
lands there, it is captured by a spider.
Construct a Markov chain for the fly.

Prof. C. Chen Markov Chains

Example 7.3 Machines

A machine can be either working or broken down on a given day.

If it is working, it will break down the next day with
probability b, and will continue working with probability
1 b.
If it breaks down on a given day, it will be repaired and be
working on the next day with probability r, and will
continue to be broken down with probability 1 r.
Construct a Markov chain for the machine.

Suppose whenever the machine remains broken down for l days,

it is replaced by a new working machine. What is the new
Markov chain?

Prof. C. Chen Markov Chains

Sequence of States

The probability of a sequence of states

S0 = s0 , . . . , Sn = sn

is
P(S0 = s0 , . . . , Sn = sn )
= P(S0 = s0 )P(S1 = s1 |S0 = s0 ) . . . P(Sn = sn |Sn1 = sn1 , . . . )
= P(S0 = s0 )P(S1 = s1 |S0 = s0 ) . . . P(Sn = sn |Sn1 = sn1 )
" n #
Y
= s0 psk1 sk
k=1

where
i = P(S0 = i)
is the probability of an initial state.

Prof. C. Chen Markov Chains

n-Step State Transition Probability

The n-step state transition probability is the probability of

starting at one state and ending at another state in n steps

rij (n) = P(Sn = j|S0 = i)

The n-step state transition probabilities constitute a matrix

R(n) = {rij (n)}

called the n-step state transition probability matrix.

Prof. C. Chen Markov Chains

Chapman-Kolmogorov Equation
For n 2
m
X
rij (n) = rik (n 1)pkj
k=1

rij (n) = P(Sn = j|S0 = i)

m
X
= P(Sn = j, Sn1 = k|S0 = i)
k=1
Xm
= P(Sn1 = k|S0 = i)P(Sn = j|Sn1 = k, S0 = i)
k=1
Xm
= P(Sn1 = k|S0 = i)P(Sn = j|Sn1 = k)
k=1
Xm
= rik (n 1)pkj
k=1

Prof. C. Chen Markov Chains

1-Step and n-Step Transition Probability Matrix

R(n) = Tn

The Chapman-Kolmogorov equation can be written as

R(n) = R(n 1)T

By definition
R(1) = T
By induction
R(n) = Tn

Prof. C. Chen Markov Chains

Visualization

State sequences of a Markov chain can be visualized through

state transition graph or trellis.

State transition graph

a sequence of states is a path on the state transition graph
each path on the state transition graph has a probability
an n-step transition probability is the sum of probabilities
of the paths with designated start state and end state

Trellis
an expansion of the state transition graph over time

Prof. C. Chen Markov Chains

Classification of States

Prof. C. Chen Markov Chains

Accessibility between States

State j is said to be accessible to state i if

n such that rij (n) > 0

Accessibility between states is denoted by

i j

on state transition graph, there is a directed path from i to

j, and the edge directions are along the path
starting from i, the probability of reaching j is non-zero
starting from i over and over again, j will be reached
eventually

Prof. C. Chen Markov Chains

Recurrent State and Transient State

A state is recurrent if it is accessible to every state that is

accessible to it. That is, i is recurrent if

i j j i

for every j.

A non-recurrent state is transient. That is, i is transient if

there exists j such that
i j
but
j 9 i

Prof. C. Chen Markov Chains

Revisiting Property

A recurrent state, once visited, is certain to be revisited.

Imagine a token traveling in the state transition graph of a

discrete-time Markov chain. The difference between a recurrent
state and a transient state is not whether the token will leave,
but whether the token will return.

Prof. C. Chen Markov Chains

Partition of State Space

The state space of a Markov chain can be partitioned into the

set of recurrent states and the set of transient state.

Furthermore, the set of recurrent states can be partitioned into

one or more recurrent classes, or simply classes.
Two recurrent states in the same class are accessible to
each other.
Two recurrent states in different classes are not accessible
to each other.

Prof. C. Chen Markov Chains

Periodic Class

A class is periodic if it can be partitioned into multiple subsets

C = S1 Sd

so that all valid state transitions are from one subset to the
next subset cyclicly

S1 S2 Sd S1

A class is aperiodic if it is not periodic.

Prof. C. Chen Markov Chains

Long-term Behavior

Prof. C. Chen Markov Chains

Steady-State Probability

For a Markov chain with a single aperiodic recurrent class, the

steady-state probability of a state is defined by

j = lim rij (n)

n

Note that it is independent of the initial state.

Prof. C. Chen Markov Chains

Probability of a State

j is approximately the probability of Xn = j for large n.

Conditioned on the initial state, we have

m
X
P(Xn = j) = P(Xn = j|X0 = i)P(X0 = i)
i=1
Xm
= rij (n)P(X0 = i)
i=1

For a large n
m
X
P(Xn = j) j P(X0 = i) = j
i=1

Prof. C. Chen Markov Chains

Balance Equations

Let be the column vector of steady-state probabilities. Then

m
0 = 0 P, i.e. j =
X
k pkj
k=1

the balance equations for steady-state probabilities.

By the Chapman-Kolmogorov equations

m
X
rij (n) = rik (n 1)pkj
k=1

Let n m
X
j = k pkj
k=1

Prof. C. Chen Markov Chains

Example 7.5 Two-state Markov Chain

Consider 2-state Markov chain with transition probabilities

p11 = 0.8, p12 = 0.2

p21 = 0.6, p22 = 0.4

Find the steady-state probabilities.

Prof. C. Chen Markov Chains

Solution

must satisfy the balance equations

1 = 1 p11 + 2 p21 = 0.81 + 0.62

2 = 1 p12 + 2 p22 = 0.21 + 0.42

and sum to 1
1 = 1 + 2

Prof. C. Chen Markov Chains

Example 7.6 Dont Forget Umbrellas

An absent-minded professor has 2 umbrellas that he uses when

commuting from home to office and back. If it rains and an
umbrella is available in his location, he takes it. If it is not
raining, he always forgets to take an umbrella. Suppose that it
rains with probability p each time he commutes. What is the
steady-state probability that he gets wet during a commute?

Prof. C. Chen Markov Chains

Solution

Let state i represent that i umbrellas are with the professor.

must satisfy the balance equations and sum to 1

0 = 2 (1 p)
1 = 1 (1 p) + 2 p
2 = 0 1 + 1 p

1 = 0 + 1 + 2

The probability that he gets wet during a commute is p0 .

Prof. C. Chen Markov Chains

Example 7.7 Superstition

A superstitious professor works in a circular building with m

doors, where m is odd, and never uses the same door twice in a
row. Instead, he uses with probability p (or probability 1 p)
the door that is adjacent in the clockwise (or counter-clockwise)
direction to the door he used last. What is the probability that
a given door will be used on some particular day far into the
future?

Prof. C. Chen Markov Chains

Solution

Let the doors be numbered from 0 to m 1 along clockwise

direction. Let state i represents that door i is used by the
professor in a given day. must satisfy the balance equations
and sum to 1

i = (i1)%m p + (i+1)%m (1 p), i = 0, . . . , m 1

1 = 0 + + m1
By symmetry
1
i =
m

Prof. C. Chen Markov Chains

Long-term Frequency of a State

For a Markov chain with single aperiodic class, the steady-state

probability j is the long-term frequency of state j.

Let vij (n) be the number of visits to state j within the first n
transitions, starting from state i. Then

vij (n) a. s.
j
n

Prof. C. Chen Markov Chains

Long-term Frequency of a State Transition

For a Markov chain with single aperiodic class, j pjk is the

long-term frequency of state transition from j to k.

Let qjk (n) be the number of transitions from state j to state

k within the first n transitions, starting from state i.

qjk (n) vij (n) qjk (n) a. s.

= j pjk
n n vij (n)

Prof. C. Chen Markov Chains

Frequency Interpretation for Balance Equations

X
j = k pkj
k

left side: the long-term frequency of state j

right side: the sum of long-term frequencies of state
transitions to j
equality: a transition to j means a visit to j

Prof. C. Chen Markov Chains

Birth-Death Process

A birth-death process is a Markov chain with a linear

transition graph, in which only self-transitions or transitions
between neighboring states are allowed.

A transition to a higher-indexed state is called a birth.

The probability of birth at state i is denoted by bi

P(Sn+1 = i + 1|Sn = i) = bi

A transition to a lower-indexed state is called a death.

The probability of death at state i is denoted by di

P(Sn+1 = i 1|Sn = i) = di

Prof. C. Chen Markov Chains

Local Balance Equations

For a birth-death process, the steady-state probabilities satisfy

local balance equations

i1 bi1 = i di , i = 2, . . . , m

For any n, the number of transitions from i to i 1 and the

number of transitions from i 1 to i differ by at most 1

qi,i1 (n) = qi1,i (n) 1

qi,i1 (n) qi1,i (n) 1

=
n n n
i1 pi1,i = i pi,i1
i1 bi1 = i di

Prof. C. Chen Markov Chains

Simplification

Every local balance equation involves only two adjacent states,

so the computation of steady-state probabilities is simplified.

By local balance equations

i1 bi1 = i di

bi1 b1 . . . bi1
i = i1 = = 1
di d2 . . . di

every steady-state probability is expressed in 1

normalize total probability to 1 to find 1 , then find
2 , . . . , m

Prof. C. Chen Markov Chains

Example 7.8 Random Walk

English walks along a straight line and, at each time period,

takes a step to the right with probability b, and a step to the
left with probability 1 b. She starts in one of the positions
1, . . . , m, but if she reaches the position 0 (or position m + 1),
her step is instantly reflected back to 1 (or m, respectively). We
introduce a Markov chain model whose states are the positions
1, . . . , m. What are the steady-state probabilities?

Prof. C. Chen Markov Chains

Example 7.9 Buffer Size

Packets arrive at a node of a communication network, where

they are stored in a buffer and then transmitted. Let the size of
the buffer be m. If m packets are already present, any newly
arriving packets are discarded. We discretize time in very small
periods, and we assume that in each period, at most one event
can happen that can change the number of packets stored in the
node. We assume that at each period, exactly one of the
following occurs:
one new packet arrives, with probability b > 0
one existing packet completes transmission, with
probability d > 0 if there is at least one packet in the node,
and with probability 0 otherwise
no new packet arrives and no existing packet completes
transmission, with probability 1 b d if there is at least
one packet in the node, and with probability 1 b otherwise

Prof. C. Chen Markov Chains

Solution
The state of the buffer can be modeled as a birth-death process,
with birth and death probabilities

b0 = = bm1 = b, bm = 0
d0 = 0, d1 = = dm = d

Local balance equations

i1 b = i d, i = 1, . . . , m

lead to
b
i = i1 = = i 0 , =
d
m
P
Adding normalization condition i = 1, we get
i=0

1 1 1 i
0 = = and i = 0 i =
1 + + + m 1 m+1 1 m+1

Prof. C. Chen Markov Chains

Short-term Behavior

Prof. C. Chen Markov Chains

Absorbing State

An absorbing state is a state making only self transitions.

That is, s is an absorbing state if

pss = 1

A transient state cannot be absorbing.

An absorbing state is in a recurrent class by itself.
Multiple absorbing states in a Markov chain are possible.

Prof. C. Chen Markov Chains

Absorption Probability

The absorption probability is the probability of visiting an

absorbing state. Specifically, starting at state i, the absorption
probability for absorbing state s is
  
ai = P lim Sn = s  S0 = i

n

For a recurrent initial state i

ai = is

Prof. C. Chen Markov Chains

Equation for Absorption Probabilities
For a transient initial state i
X
ai = pij aj

 
ai = P lim Sn = s|S0 = i
n
m
X  
= P lim Sn = s, S1 = j|S0 = i
n
j=1
Xm  
= P (S1 = j|S0 = i) P lim Sn = s|S1 = j
n
j=1
Xm  
= P (S1 = j|S0 = i) P lim Sn1 = s|S0 = j
n
j=1
Xm
= pij aj
j=1

Prof. C. Chen Markov Chains

Example 7.11 Gamblers Ruin

A gambler wins $1 at each round with probability p, and loses

$1 with probability 1 p. He plays continuously until he either
accumulates a target amount of $m, or loses all his money.

What is the probability that he wins?

Prof. C. Chen Markov Chains

Solution
Let state i represent that gambler has i dollars. State m (as
well as state 0) is an absorbing state. For a transient state i
m
X
ai = pij aj = (1 p)ai1 + pai+1
j=0

or
(1 p)(ai ai1 ) = p(ai+1 ai )
Defining
1p
i = ai+1 ai , =
p
we have
i = i1
which leads to
i = i1 = = i 0

Prof. C. Chen Markov Chains

For absorbing states 0 and m

am = 1, a0 = 0

and
am a0 = (am am1 ) + + (a1 a0 )
= m1 + + 0
= 0 (m1 + + 1)
=1
so
1
0 =
m1 + + 1
Thus
i1 + + 0
ai = a0 + 0 + + i1 =
m1 + + 1

Prof. C. Chen Markov Chains

Expected Time to Absorption

The expected time to absorption is defined by

i = E[A|S0 = i]

where
A = arg min {Sn is recurrent}
n>0

is the first time a recurrent state is visited.

The name is a little confusing.

Prof. C. Chen Markov Chains

Recursion
If i is recurrent
i = 0
If i is transient
i = E[A|S0 = i] = E[E[A|S1 ]|S0 = i]
m
X
= P(S1 = j|S0 = i)E[A|S1 = j, S0 = i]
j=1
Xm
= pij E[A|S1 = j]
j=1
Xm
= pij (j + 1)
j=1
m
X
=1+ pij j
j=1

Prof. C. Chen Markov Chains

Example 7.12 Spider-and-Fly

Consider the spider-and-fly model of Example 7.2. This

corresponds to a Markov chain. Assume m = 4. What is the
expected number of steps until the fly is captured?

Prof. C. Chen Markov Chains

Solution

State 1 and 4 are recurrent states. For the expected time to

absorption, we have

1 = 0
2 = 1 + 0.31 + 0.42 + 0.33
3 = 1 + 0.32 + 0.43 + 0.34
4 = 0

Thus
10
2 = 3 =
3

Prof. C. Chen Markov Chains

Continuous-time Markov Chains

Prof. C. Chen Markov Chains

From Discrete-time to Continuous-time

In a discrete-time Markov chain, state transitions happen only

at discrete times.

A continuous-time Markov chain, state transitions can

happen at any instant.

Prof. C. Chen Markov Chains

Representation of a Continuous-time Markov Chain

A CTMC can be represented by

T T T n T Tn+1 Tn+2
S0 1 S1 2 S2 3 . . .
Sn Sn+1 . . .

where Tn+1 is the waiting time for the current state Sn to

transit to the next state Sn+1 .

The notion of self-transition is abandoned in CTMC.

Prof. C. Chen Markov Chains

Markov Property

A CTMC satisfies the Markov property that the future is

independent of the past given the current state.

The random variables in a CTMC are

current state
waiting time for state transition to occur
next state when state transition occurs
Given Sn , both Tn+1 and Sn+1 are independent of the past.

Prof. C. Chen Markov Chains

Conditional Description

Conditioned on the current state Sn = i

Tn+1 , the time to leave i, is an exponential random
variable with some parameter i
Sn+1 , the next state, is j with probability pij

The Markov property is enforced by the memorylessness

of exponential random variables.
A CTMC is completely specified by a set of state transition
probabilities pij s and state transition rates i s.

Prof. C. Chen Markov Chains

Joint Distribution

Given Sn = i, the joint distribution of Sn+1 and Tn+1 is

P(Sn+1 = j, Tn+1 t|Sn = i)

= P(Tn+1 t|Sn = i)P(Sn+1 = j|Sn = i)
= ei t pij

Note that
j 6= i

Prof. C. Chen Markov Chains

Rate of Transitions Out of a State

Suppose that the current state is Sn = i. The expected time to

a state transition is
Z
1
E[Tn+1 |Sn = i] = ti ei t dt =
0 i

On average, it takes i1 to leave state i.

On average, i transitions occur per unit time when in i.
The rate of transitions out of state i is i .

Prof. C. Chen Markov Chains

Rate of Transitions between States

When in state i
a fraction pij of the state transitions out of i go to state j
on average, the number of state transitions from i to j per
unit time is
i pij
the rate of transitions from state i to state j is

qij = i pij

Prof. C. Chen Markov Chains

Example 7.14 Machines

A machine, once in production mode, operates continuously

until an alarm signal is generated. The time up to the alarm
signal is an exponential random variable with parameter 1.
Subsequent to the alarm signal, the machine is in test mode for
an exponentially distributed amount of time with parameter 5.
The test results are positive, with probability 1/2, in which case
the machine returns to production mode, or negative, with
probability 1/2, in which case the machine is taken for repair.
The duration of the repair is exponentially distributed with
parameter 3. Construct a continuous-time Markov chain.

Prof. C. Chen Markov Chains

Solution
The states are production, test, and repair. The state
transition rates are

1 = 1, 2 = 5, 3 = 3

The state transition probabilities are

0 1 0
P = 12 0 21

1 0 0

The rates of transitions between states are

0 1 0
Q = 25 0 52

3 0 0

Prof. C. Chen Markov Chains

Starting with State Transition Rates
The state transition rates and state transition probabilities can
be determined by the rates of transitions between states.

Suppose the rates of transitions between states are

qij , i 6= j

The state transition rates are

X X X
i = i pij = i pij = qij
j j j

The state transition probabilities are

qij qij
pij = =P
i qij 0
j0

Prof. C. Chen Markov Chains

Auxiliary Discrete-time Markov Chain

A discrete-time Markov chain can be constructed based on a

continuous-time Markov chain.

Suppose X(t) is a continuous-time Markov chain. For > 0,

the discrete-time random process

Z = Z0 , Z1 , . . .

where
Zn = X(n)
is an auxiliary discrete-time Markov chain of X(t).

Prof. C. Chen Markov Chains

State Transition Probabilities
The state transition probabilities of Z, denoted by {pij ()}, are
related to the rates of transitions between states of {X(t)}.

For j 6= i
pij () = P(Zn+1 = j|Zn = i)
transition out of i transition to j
z }| {
= (i + o())
z}|{
pij
= i pij + o()
= qij + o()
For j = i X
pii () = 1 pij ()
j6=i
X
=1 qij + o()
j6=i

Prof. C. Chen Markov Chains

Example 7.14 (continued)

Neglecting o() terms, the state transition probability matrix

for an auxiliary DTMC is

1 0


5 5
2 1 5

2


3 0 1 3

Prof. C. Chen Markov Chains

Alternative Characterization of CTMC

Consider a continuous-time random process. If, conditioned on

the current state i, time later the state is in another state j
with probability
qij + o(), j 6= i
for some qij 0, then the process is a CTMC.

Prof. C. Chen Markov Chains

Example 7.15 Communication Network

Packets arrive at a node of communication network according

to a Poisson process with rate . The packets are stored at a
buffer with room for up to m packets, and are then transmitted
one at a time. However, if a packet finds a full buffer upon
arrival, it is discarded. The time required to transmit a packet
is exponentially distributed with parameter . Show that this
can be modeled as a continuous-time Markov chain.

Prof. C. Chen Markov Chains

Solution

Let state i represent i packets in the buffer.

P(X(t + ) = i 1|X(t) = i) = + o(), i = 1, . . . , m

P(X(t + ) = i + 1|X(t) = i) = + o(), i = 0, . . . , m 1

It follows that {X(t)} is a continuous-time Markov chain.

Prof. C. Chen Markov Chains

Steady-State Probabilities

For a CTMC X(t) with single recurrent class, each state has a
steady-state probability

lim P(X(t) = j) = j
t

The steady-state probabilities agree with those of an auxiliary

DTMC
Zn = X(n)

Prof. C. Chen Markov Chains

Balance Equations
The steady-state probabilities satisfy the balance equations
X X
j qjk = k qkj , j
k6=j k6=j

The balance equations for the steady-state probabilities of

Zn = X(n) are X
j = k pkj
k
X
= j pjj + k pkj
k6=j

X X
j = j 1 qjk + o() + k (qkj + o())
k6=j k6=j
X X
j qjk = k qkj
k6=j k6=j

Prof. C. Chen Markov Chains

Example 7.14 Machines

What are the steady-state probabilities?

Prof. C. Chen Markov Chains

Solution

Use balance equations and normalization

5
1 = 2 + 33
2
52 = 1
5
33 = 2
2
1 + 2 + 3 = 1

30
1 =
41
6
2 =
41
5
3 =
41

Prof. C. Chen Markov Chains

Continuous-time Birth-Death Processes

The states in a continuous-time birth-death process are linearly

arranged and only transitions to a neighboring state are allowed.

That is
qij = 0, for |i j| > 1

Prof. C. Chen Markov Chains

Local Balance Equations

The rates of transition from i to j and from j to i must be the

same
i qij = j qji
for j = i 1.

They are called local balance equations.

Prof. C. Chen Markov Chains

Example 7.15 Communication Network

What are the steady-state probabilities?

Prof. C. Chen Markov Chains

Solution

Use local balance equations

 
i = i+1 i+1 = i = i , =

i = 0 i

Normalization
0 + 1 + + m = 1 0 (1 + + + m ) = 1
0 = (1 + + + m )1

i
i =
1 + + + m

Prof. C. Chen Markov Chains