Discrete-Time Markov Chains Review

Chapman-Kolmogorov Equations: , for all

Initial occupancy probability: n-step occupancy probability:

Classification of States Accessibility o If the n-step transition probability from state to state is non-negative for some state is accessible from state . Communication o States that are accessible from each other communicate. Class o o

States that communicate with each other are in the same class. Classes are disjoint, that is, a state cannot be in more than one class.

Irreducibility o A MC with only one class is irreducible. Recurrence o If the probability that the process will reenter state , starting from state , is equal to 1, ( ) state is recurrent. Also, state is recurrent if . o If a state is recurrent, the process will reenter that state infinitely often. o If the expected time to reenter the recurrent state is finite, state is positive recurrent. All recurrent states in finite-state MCs are positive recurrent! Transience o If the probability that the process will reenter state , starting from state , is less than 1, ( ) state is transient. Also, state is transient if . 1

o -

Since a transient state is only visited a finite number of times, in a finite-state MC, not all states can be transient! If all states were transient, after a finite time no states would be visited, which contradicts the fact that the process should be in some state after any finite number of time intervals. Therefore, in a finite-state MC, at least one state should be recurrent! If a state is transient, each time the process enters that state there will be a positive probability ( ) that it will never reenter that state. Starting in state , the probability that the process will be in state for exactly time periods is . Note that this is the pmf of the geometric distribution with parameter . Therefore the average number of time periods that the process will be in state is . In the long-run, the fraction of time spent in a transient state would be 0.

Periodicity o State has period if for values of that are NOT divisible by , and is the largest such integer with this property. is the greatest common divisor of all possible number of transitions to reenter a state. o The fact that a state has period does not necessarily mean that you can reenter that state in transitions! Example: If it is possible to reenter a state in transitions, the period is , but 2 is not in the set of possible number of transitions to reenter that state. o If for state , state is aperiodic. Ergodicity o A positive recurrent and aperiodic MC is ergodic. o The tpm for ergodic MCs converge to the limiting distribution ( ) as the number of transitions converge to infinity ( ). The limiting distribution provides the probability of observing the stochastic process in each state in the long-run. The limiting (stationary) probability value for state ( ) is independent of the initial state.

Identify the classes of the following Markov Chains and specify which ones are transient, recurrent, etc.