Chapter 4 1

Markov Chains
Definition, Chapman-Kolmogorov Equations,
Classification of States, Limiting Probabilities,
Transient Analysis, Time Reversibility
Chapter 4 2
Stochastic Processes
A stochastic process is a collection of random variables

Typically, T is continuous (time) and we have
Or, T is discrete and we are observing
at discrete time points n that may or may not be evenly spaced.
Refer to X(t) as the state of the process at time t.
The state space of the stochastic process is the set of all
possible values of X(t): this set may be discrete or
continuous as well.
( ) { }
, X t t T e
{ }
, 0,1, 2,...
n
X n =
( ) { }
, 0 X t t >
Chapter 4 3
Markov Chains
In this chapter, consider discrete-state, discrete-time:
A Markov chain is a stochastic process
where each X
n
belongs to the same subset of {0, 1, 2, },
and
for all states i
0
, i
1
,, i
n-1
and all n > 0 .
Say
Then


Let be the matrix of one-step transition probabilities.

{ }
, 0,1, 2,...
n
X n =
{ } { }
1 1 1 1 1 0 0 1
, ,..., ,
n n n n n n
P X j X i X i X i X i P X j X i
+ +
= = = = = = = =
{ }
1 ij n n
P P X j X i
+
= = =
1
0 for all ,
For any , 1
ij
ij
j
P i j
i P

=
>
=

ij
P ( =

P
Chapter 4 4
n-step Transition Probabilities
Given the chain is in state i at a given time, what is the
probability it will be in state j after n transitions? Find it by
conditioning on the initial transition(s).
{ }
{ } { }
{ } { }
1 1
0
1 1
0
1
0
,
n
ij m n m
m n m m m m
k
m n m m m
k
n
kj ik
k
P P X j X i
P X j X i X k P X k X i
P X j X k P X k X i
P P
+

+ + +
=

+ + +
=

=
= = =
= = = = = =
= = = = =
=

Chapter 4 5
Chapman-Kolmogorov Equations
In general, can find the n-step transition probabilities by
conditioning on the state at any intermediate stage:





Let P
(n)
be the matrix of n-step transition probabilities:

So, by induction,
{ }
{ } { }
0
0 0
0
0
,
n m
ij n m
n m n n
k
m n
kj ik
k
P P X j X i
P X j X k X i P X k X i
P P
+
+

+
=

=
= = =
= = = = = =
=

( ) ( ) ( ) n m n m +
= P P P
( ) n
n
= P P
Chapter 4 6
Classification of States
State j is accessible from state i if
If j is accessible from i and i is accessible from j, we say that
states i and j communicate (i j).
Communication is a class property:
(i) State i communicates with itself, for all i > 0
(ii) If i communicates with j then j communicates with i
(iii) If i j and j k, then i k.
Therefore, communication divides the state space up into
mutually exclusive classes.
If all the states communicate, the Markov chain is irreducible.
0 for some 0
n
ij
P n > >
Chapter 4 7
Recurrence vs. Transience
Let f
i
be the probability that, starting in state i, the process will
ever reenter state i. If f
i
= 1, the state is recurrent, otherwise
it is transient.
If state i is recurrent then, starting from state i, the process
will reenter state i infinitely often (w/prob. 1).
If state i is transient then, starting in state i, the number of
periods in which the process is in state i has a geometric
distribution with parameter 1 f
i.

Or, state i is recurrent if and transient if
Recurrence (transience) is a class property: If i is recurrent
(transient) and i j then j is recurrent (transient).
A special case of a recurrent state is if P
ii
= 1 then i is absorbing.
1
n
ii
n
P

=
=

1
n
ii
n
P

=
<

Chapter 4 8
Recurrence, Transience and Other Properties
Not all states in a finite Markov chain can be transient (why?).
All states of a finite irreducible Markov chain are recurrent.
If whenever n is not divisible by d, and d is the largest
integer with this property, then state i is periodic with
period d.
If a state has period d = 1, then it is aperiodic.
If state i is recurrent and if, starting in state i, the expected
time until the process returns to state i is finite, it is
positive recurrent (otherwise it is null recurrent).
A positive recurrent, aperiodic state is called ergodic.
0
n
ii
P =
Chapter 4 9
Limiting Probabilities
Theorem: For an irreducible ergodic Markov chain,
exists for all j and is independent of i. Furthermore, t
j
is
the unique nonnegative solution of




The probability t
j
also equals the long run proportion of
time that the process is in state j.
If the chain is irreducible and positive recurrent but
periodic, the same system of equations can be solved for
these long run proportions.
lim
n
j ij
n
P t

=
0
0
, 0
1
j i ij
i
j
j
P j t t
t

=
= >
=

Chapter 4 10
Limiting Probabilities 2
The long run proportions t
j
are also called stationary
probabilities because if
then
Let m
jj
be the expected number of transitions until the
Markov chain, starting in state j, returns to state j (finite if
state j is positive recurrent). Then
If is an irreducible Markov chain with stationary
probabilities , and r is a bounded function on the state
space. Then with probability 1,


{ }
0 j
P X j t = =
{ }
for all , 0
n j
P X j n j t = = >
1
jj j
m t =
{ }
, 0
n
X n >
( )
( )
1
0
lim
N
n
n
j
N
j
r X
r j
N
t

=
=

Long run
average reward
Chapter 4 11
Transient Analysis
Suppose a finite Markov chain with m states has some
transient states. Assume the states are numbered so that T
= {1, 2, , t} is the set of transient states, and let P
T
be the
matrix of transition probabilities among these states.
Let R be the t x (m-t) matrix of one-step transition
probabilities from transient states to the recurrent states and
P
R
be the (m-t) x (m-t) matrix of transition probabilities
among the recurrent states: the overall one-step transition
probability matrix can be written as
0
(
=
(

T
R
P R
P
P
If the recurrent states are all
absorbing then P
R
= I.
Chapter 4 12
Transient Analysis 2
If the process starts in a transient state, how long does it
spend among the transient states?
What are the probabilities of eventually entering a given
recurrent state?
Define o
ij
= 1 if i = j and 0 otherwise.
For i and j in T, let s
ij
be the expected number of periods that
the Markov chain is in state j given that it started in state i.
1
T
ij ij ik kj
k
s P s o
=
= +

Condition on the first transition,

and note that transitions from recurrent
states to transient states are impossible
Chapter 4 13
Transient Analysis 3
Let S be the matrix of s
ij
values. Then S = I + P
T
S. Or,



For i and j in T, let f
ij
be the probability that the Markov chain
ever makes a transition into j, starting from i.

For i in T and j in T
c
, the matrix of these probabilities is
( )
( )
1
T
T

=
=
I P S I
S I P
ij ij
ij
jj
s
f
s
o
=
( )
1
T

= F I P R
Chapter 4 14
Time Reversibility
One approach to estimate transition probabilities from each
state is by looking at transitions into states and tracking what
the previous state was.
How do we know this information is reliable?
How do we use it to estimate the forward transition probabilities?
Consider a stationary ergodic Markov chain.
Trace the sequence of states going backwards: X
n
, X
n-1
,, X
0
This is a Markov chain with transition probabilities:


If Q
ij
= P
ij
for all i, j, then the Markov chain is time reversible.
{ }
1
j ji
ij m m
i
P
Q P X j X i
t
t
+
= = = =
Chapter 4 15
Time Reversibility 2
Another way of writing the reversibility equation is:

Proposition: Consider an irreducible Markov chain with
transition probabilities P
ij
. If one can find positive numbers
t
i
summing to 1 and a transition probability matrix Q such
that the above equation holds for all i, j, then Q
ij
are the
transition probabilities for the reversed chain and the t
i
are
the stationary probabilities for both the original and the
reversed chain.

Use this, thinking backwards, to guess at transition
probabilities of reversed chain.
i ij j ji
Q P t t =