An introduction to Markov chains

This lecture will be a general overview of basic concepts relating to Markov chains, and some properties
useful for Markov chain Monte Carlo sampling techniques. In particular, well be aiming to prove a Fun-
damental Theorem for Markov chains.
1 What are Markov chains?
Denition. Let {X
0
, X
1
, ..} be a sequence of random variables and Z = 0, 1, 2, ... be the state space,
i.e. all possible values the random variable can take on. Then {X
0
, X
1
, ..} is called a discrete-time Markov
chain if
P(X
n+1
= i
n+1
|X
n
= i
n
), i Z
That is, the state at time step n + 1 is dependent only on the state at time n.
The denition can describe a random walk on a graph where the vertices are the state space Z, and the
edges are weighted by the transition probabilities:
p
ij
(n) = P(X
n+1
= j|X
n
= i), i, j Z
Denition. A homogeneous Markov chain is one that does not evolve in time; that is, its transition
probabilities are independent of the time step n. Then we have the n-step transition probabilities
p
(m)
ij
= P(X
n+m
= j|X
n
= i)
and we have
p
(0)
ij
=
{
1 : i = j
0 : i = j
Now we can dene a theorem.
Theorem. Chapman-Kolmogorov equation.
p
(m)
ij

kZ
p
(r)
ik
p
(mr)
kj
r N {0}
1
Proof.
p
ij
= P(X
m
= j|X
0
= i) =

kZ
P(X
m
= j, X
r
k|X
o
= i)
=

kZ
P(X
m
= j|X
r
= k, X
0
= i)P(X
r
= k|X
0
= i)
=

kZ
P(X
m
= j|X
r
= k)P(X
r
= k|X
0
= i)
=

kZ
p
(r)
ik
p
(mr)
kj
We can write this as a matrix for convenience:
P
(m)
= ((p
(m)
ij
))
Corollary.
P
(m)
= P
m
Proof. Chapman-Kolmogorov in matrix form gives us
P
(m)
= P
(r)
P
(mr)
r N {0}
P
(2)
= PP = P
2
P
(3)
= PP
2
= P
3
P
(m)
= P
m
, m 2, then
P
(m+1)
= PP
m
= P
m+1
2 Several denitions
A Markov Chain is completely determined by its transition probabilities and its initial distribution.
An initial distribution is a probability distribution
{
i
= P(X
0
= i)|i Z}
such that

i
= 1.
A distribution is stationary if it satises = P.
The period of state i is dened as
d
i
= gcd{m Z|p
(m)
ii
> 0}
that is, the gcd of the numbers of steps that it can take to return to the state.
If d
i
= 1, the state is aperiodic it can occur at non-regular intervals.
A state j is accessible from a state i if the system, when started in i, has a nonzero probability of
eventually transitioning to j, or more formally if there exists some n 0 such that
Pr(X
n
= j|X
0
= i) > 0.
2
We write this as (i j).
We dene the rst-passage time (or hitting time) probabilities as
f
(m)
ij
= P(X
m
= j, X
k
= j, 0 < k < m1|X
0
= i), i, j Z.
that is, the time step at which we rst reach state j.
We denote the expected return time as

ij
=

m = 1

mf
(m)
ij
A state is recurrent if

m=1
f
(m)
ij
= 1
(and transient if the sum is greater than 1).
It is positive-recurrent if
ii
< . That is, we expect to return to the state in a nite number of time
steps.
3 Fundamental Theorem of Markov Chains
Theorem. For any irreducible, aperiodic, positive-recurrent Markov chain P there exists a unique station-
ary distribution {
j
, j Z}.
Proof. We know that for any m,
m

i=0
p
(m)
ij

i=0
p
(m)
ij
1.
If we take the limit as m :
lim
m
m

i=0
p
(m)
ij
=

i=0

j
1.
This implies that for any M,

M
i=0

j
1.
Now we can use Chapman-Kolmogorov:
p
(m+1)
ij
=

i=0
p
(m)
ik
p
kj

M

i=0
p
(m)
ik
p
kj
and take the limit again as m, M

k=0

k
p
kj
3
Now for the sake of contradiction, lets assume that strict inequality holds for some state j. Then if we
sum over all of these states, we have

j=0

j
>

j=0

k=0

k
p
kj
=

k=0

j=0
p
kj
=

k=0

k
but this is a contradiction. So, for some state j, equality must hold.

j
=

k=0

k
p
kj
Thus, a unique stationary distribution exists.
We can separately prove that were guaranteed to converge to the stationary distribution, but this proof
is somewhat more involved.
References
[1] Aaron Plavnick. The fundamental theorem of Markov chains. VIGRE REU at UChicago, 2008.
4