Discrete

Time Markov Chains

• Basic definitions of discrete time Markov Chains
• Classification of Discrete Time Markov Chains
• Analysis of Discrete Time Markov Chains
 Discrete Time Markov Chain (DTMC)

Stochastic process

Discrete Continuous event

events

Discrete Continuous time

time

Memoryless
A DTMC is a discrete time and memoriless discrete event
stochastic process.

2
1 Introduction
2 What is a Markov Chain?
3 n-Step Transition Probabilities
4 Classification of States in a Markov Chain

5. Analysis of Discrete Time Markov Chains

6. Steady-State Probabilities and

Mean First Passage Times
 Basic definitions of
Discrete Time Markov Chain (DTMC)

Definition : a stochastic process with discrete state space and discrete time {Xn, n > 0}
is a discrete time Markov Chain (DTMC) iff

P[Xn+1 = j  Xn = in, ..., X0 = i0] = P[Xn+1 = j  Xn = in] = pij(n)

In a DTMC, the past history impacts on the future evolution of the system via the
current state of the system

pij(n) is called transition probability from state i to state j at time n.

4
 Example: a mouse in a maze

1/2 1/2
1 0 start
1/2
1/4 1/2
1/4
1
1/4
2 3 4
1
1/4

exit 5
1

• Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited
• Assume that the mouse does not have any memory of rooms visited previously.

5
 transition matrix ‫مصفوفة االنتقال‬
Stage (t) Stage (t + 1)
State I Pij State j (with prob. Pij)

Convenient to give transition probabilities in matrix for

P = P(m+1) (m+1) = Pij go to ith state

0 1 2 ... j ... m
0 P00 P0j
1 P10 P1j
Rows are given 2 P20 P2j
in this stage = 
Rows sum to 1
i Pi0 Pij

m Pm0 Pmj Pmm
6
 ‫‪Homogeneous‬‬

‫سلسلة ماركوف المتجانسة‬

‫نقول عن سلسلة ماركوف إنها متجانسة ‪homogeneous‬‬
‫إذا كانت الخاصة الماركوفٌة ال تتعلق بالزمن‬
‫من أجل أٌة قٌمتٌن ‪ i,j‬أي إنها مستقلة عن ‪ ،n‬وعندها ٌمكننا أن نعرف احتمال انتقال‬
‫السلسلة من الحالة ‪ i‬إلى الحالة ‪ j‬بنقلة واحدة ‪ transition‬كما ٌلً‪:‬‬
‫‪∀ n,i,j∈I‬‬ ‫𝑖 = 𝑛𝑋 𝑗 = ‪𝑃𝑖𝑗 = 𝑃 𝑋𝑛+1‬‬
‫‪P[Xn+1 = j  Xn = i] = P[X1 = j  X0 = i] = pij‬‬

‫‪• A homogenuous DTMC is then defined by its transition matrix P =[pij]i,jE‬‬

‫‪0‬‬ ‫‪0.5‬‬ ‫‪00‬‬ ‫‪0.5‬‬ ‫‪0‬‬
‫‪0.5‬‬ ‫‪0.5‬‬ ‫‪00‬‬ ‫‪0‬‬ ‫‪0‬‬
‫=‪P‬‬
• Chapman-Kolmogorov equation:
P(l+m) = P(l)P(m)
or
pij
l  m
 
k E
pik  pkj
l m
 Example
• The probability of going from i to j in m steps is
pij(m) = P{Xn+m = j|Xn=i} = P{Xm = j|X0=i}.
• Let P(m) = [pij(m)] be the m-step transition matrix
• P(m) = Pm
1/2 1/2
1 0 start
1/2
1/4 1/2
1/4
1 1/4
2 3 4
1
1/4

exit 5
1

• What is the probability that the mouse is still in room 0 at time 4? (p00(4))
10
‫‪Classification of Discrete Time Markov Chains‬‬
‫نقول عن حالة أنها غير معزولة ‪( Reachable‬ممكنة الوصول (‪) )accessible‬‬ ‫•‬
‫إذا كان هناك حالة ما ‪ j,‬وكان هناك طرٌق من الحالة ‪J‬إلى الحالة ‪i.‬‬
‫وإذا كان هناك طرٌق آخر من ‪i‬عودة إلى ‪ j‬نقول عن هاتٌن الحالتٌن أنهما‬ ‫•‬
‫تتواصالن ‪Communicate‬‬
‫عندما تكون كل حاالت السلسلة غٌر معزولة عن بعضها‪ٌ ,‬كون البٌان الممثل مرتبط‬ ‫•‬
‫بقوة ‪.strongly connected‬‬
‫العقدة الماصة هً عقدة ال ٌخرج منها أي أسهم‬ ‫•‬
‫الحالة الماصة ‪ Absobing‬نقول عن حالة أنها ماصة إذا كانت العقدة التً تمثلها‬ ‫•‬
‫فً البٌان هً عقدة ماصة‬
 ‫‪ ‬نقول عن حالة 𝑖 انها تعاودية‪/‬متكررة (‪ )recurrent‬اذا كان‬
‫‪∀𝑗 ∈ 𝑆, ∃𝑛 ∈ 𝑁 + ; 𝑟𝑖𝑗 𝑛 > 0 ⟹ 𝑟𝑗𝑖 𝑛 > 0‬‬
‫إذا كان زمن العودة للمرة األولى إلى ‪ I‬منذ الخروج منها ‪T‬‬
‫إذا كان التوقع الرٌاضً لهذا الزمن محدودا‪ ,‬فالحالة تعاودٌة موجبة ‪positive recurrent‬‬
‫أما إن كان ال نهائٌا فهً تكاودٌة معدومة ‪null recurrent‬‬
‫‪ ‬نقول عن حالة 𝑖 انها عابرة (‪ )transient‬إن لم تكن حالة تعاودٌة‬
‫• الدورية (‪ )Periodicity‬الصفوف التعاودية‬
‫ممكن للصفوف التعاودٌة ان تكون دورية (‪ )Periodic‬أو ال دورية (‪)Aperiodic‬‬
‫• دور صف هو نمط (‪ )pattern‬التكرار الزمنً لحاالته‬
‫ٌمكن تجمٌع الحاالت التعاودٌة فً صفوف بحٌث ٌحوي كل صف كل الحاالت التعاودٌة‬
‫الممكن الوصول إلٌها من بعضها بعض‬
 Classification of states
• Let fjj be the probability of returning to state j after leaving j.

• A state j is said transient if fjj < 1

• A state j is said recurrent if fjj = 1
• A state j is said absorbing if pjj = 1.

• Let Tjj be the average reccurn time, i.e. time of returning to j

• A recurrent state j is positive recurrent if E[Tjj] is finite.

• A recurrent state j is null recurrent if E[Tjj] = .

13
 Irreducible Markov chain
• A DTMC is said irreducible iff a state j can be reached in a finite number
of steps from any other state i.

• An irreducible DTMC is a strongly connected graph.

14
Irreducble Markov chain

15
 Periodic Markov chain

• A state j is said periodic if it is visited only in a number of steps which is multiple of

an integer d > 1, called period.
• A state j is said aperiodic otherwise

• A state with a self-loop transition (i.e. pii > 0) is always aperiodic.

• All states of an irreducible Markov chain have the same period.

16
 Classification of irreducible sub-chains

• A sub-chain is said absorbing if there is no arc going out of it.

• Otherwise, the sub-chain is transient.

transcient sub-
chain
absorbing sub-
chain

absorbing sub-
chain

17
 Probability distribution of states

• pi(n) : probability of being in state i at time n

pi(n) = P{Xn = i}
• p(n) = (p1(n), p2(n), ...) : vector of probability distribution over the state space at time n
• The probability distribution p(n) depends on
─ the transition matrix P
─ the initial distribution p(0)
• Remark: if the system is at state i for certainty, then pi(0) = 1 and pj(n) = 0, for j ≠i

• What is the relation between p(n), p(0), and P?

18
 Transient state equations

• By conditioning on the state at time n,

Property:
Let P be the transition matrix of a markov chain and p(0) the initial distribution,
then over the state space at time n
p(n+1) = p(n)P
p(n)= p(0)Pn

19
 Theorem Steady state distribution
For a irreducible and aperiodic DTMC (Discrete Time Markov Chains)
with positive recurrent states, the distribution p(n) converges to a limit vector
p which is independent of p(0) and is the unique solution of the system:
p  p P
 p 1

iE
i Normalization equation

p j   p i pij , j  E
 iE balance equation

 p i  1 equilibrium equation
 iE
• pi are also called stationary probabilities (also called steady state or
equilibrium distribution).
• For an irreducible and periodic DTMC, pi are the percentage of time spent in
state i

20