Applied Stochastic Processes Math 564

Fall 2010
Homework 1 Solutions
1. Give examples of the following, all dened on Z:
(a) A distribution;
(b) A nite measure that is not a distribution;
(c) A -nite measure that is not nite;
(d) A measure which is not -nite.
Solution: Any function : Z R
0
denes a -nite measure (as we proved in class), so we need
only specify the function for the rst three.
(a) A simple example is to choose (1) = 1, (x) = 0 for x = 1. Clearly

xZ
(x) = 1. A more
complicated example is to choose (n) = 2
(|n|+1)
for n = 0 and (0) = 0. We check that

nZ
1
2
|n|+1
= 2

n=1
1
2
n+1
=

n=1
1
2
n
= 1.
(b) Take any distribution and multiply by a positive number not equal to one, and this will work.
For example, choose (n) = 17 2
(|n|+1)
for n = 0 and (0) = 0. Then

nZ
(n) = 17 <
which is nite but not 1.
(c) Choose (n) = 1 for all n Z. Clearly

nZ
(n) = . On the other hand, if we choose
A
n
= [n, n] Z, then
(A
n
) =
n

i=n
1 = 2n + 1 < ,
but it is clear that

n=1
A
n
= Z.
(d) Choose (A) = for any set A = .
2. (a) Let B
1
, B
2
, . . . be disjoint events with

n=1
B
n
= . Show that if A is another event and
P(A|B
n
) = p for all n, then P(A) = p.
(b) Let B
1
, B
2
, . . . be disjoint events with

n=1
B
n
= . Now choose A to be another event, and all
we know is p
n
:= P(A|B
n
). (Same problem as above except we do not assume the conditional
probabilities are all the same.) Whats the strongest statement that we can say about P(A)?
Solution:
(a) By the Law of Total Probability, we have that
P(A) =

n=1
P(A|B
n
)P(B
n
) =

n=1
pP(B
n
) = p

n=1
P(B
n
) = p.
1
(b) We proceed as before. Denote
n
= P(B
n
). Again using LTP, we have
P(A) =

n=1
p
n

n
This means, for example, that if we write p = sup
n
p
n
, then
P(A)

n=1
p
n
= p

n=1

n
= p.
Similarly, if we write p = inf
n
p
n
, then we have
p P(A) p.
3. (Norris 1.1.2) Suppose that (X
n
)

n=0
is Markov(, P). Dene Y
n
= X
kn
for some k 1. Show that
(Y
n
)

n=0
is Markov(, P
k
).
Solution: We rst have to show that Y
n
is a Markov chain. In particular, this means that we need
to show that
P(Y
n+1
= j
n+1
|Y
0
= j
0
, Y
1
= j
1
, . . . , Y
n
= j
n
) = P(Y
n+1
= j
n+1
|Y
n
= j
n
),
and that this probability depends only on j
n
, j
n+1
. But using the denition, we have
P(Y
n+1
= j
n+1
|Y
0
= j
0
, Y
1
= j
1
, . . . , Y
n
= j
n
) = P(X
k(n+1)
= j
n+1
|X
0
= j
0
, X
k
= j
1
, . . . , X
kn
= j
n
),
and since X
n
is a Markov chain, this is equal to
P(X
k(n+1)
= j
n+1
|X
kn
= j
n
) = P(Y
n+1
= j
n+1
|Y
n
= j
n
).
Furthermore, we know from Theorem 1.1.2 that
P(X
k(n+1)
= j
n+1
|X
kn
= j
n
) = p
(k)
j
n
,j
n+1
and this depends only on j
n
, j
n+1
. Thus (Y
n
) is a Markov chain.
Now that we have shown it is a Markov chain, we need to compute its initial condition and transition
matrix. But we see that the distribution of Y
0
is the distribution of X
0
which is . Moreover, we have
shown above that the one-step transition for Y
n
is
P(Y
n+1
= j
n+1
|Y
n
= j
n
) = P(X
k(n+1)
= j
n+1
|X
kn
= j
n
) = p
(k)
j
n
,j
n+1
,
so therefore the transition matrix for Y
n
is P
k
.
4. (Norris 1.2.1) Identify the communicating classes of the following transition matrix:

1
2
0 0 0
1
2
0
1
2
0
1
2
0
0 0 1 0 0
0
1
4
1
4
1
4
1
4
1
2
0 0 0
1
2

.
Solution: We have from the matrix (or drawing the graph) 1 5, 2 4, 4 2, 4 3, 4 5, 5 1
and of course i i for all i = 1, . . . , 5. From this we see clearly that 1 5 and 2 4. Just as clearly,
we see there is no path from 1 to 4 so 1 4. Moreover there is no path from 1 to 3, or no path from
3 to 4, so the communicating classes must be {{1, 5}, {2, 4}, {3}}.
2
5. Consider a 4-state Markov chain.
(a) What is the largest number of communicating classes this Markov chain can have? The smallest?
(b) Describe the structure of these classes and in each case give an example of a transition matrix.
(c) Are there any other possibilities (i.e. some intermediate number of communicating classes)? Do
the same for these as for above.
Solution: The answer is that a four-state Markov chain can have any number of communicating
classes from 1 to 4. Examples are:
{{1, 2, 3, 4}}, {{1, 2, 3}, {4}}, {{1, 2}, {3}, {4}}, {{1}, {2}, {3}, {4}}.
Example transition matrices are, in order, (for some p > 0)

1 p p 0 0
0 1 p p 0
0 0 1 p p
p 0 0 1 p

1 p p 0 0
0 1 p p 0
p 0 1 p 0
0 0 0 1

1 p p 0 0
p 1 p 0 0
0 0 1 0
0 0 0 1

1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1

.
This answers everything asked in the question.
However, we can say even more: there is exactly one more case, namely {{1, 2}, {3, 4}}; after this,
every case is equivalent to one of these ve after a renumbering of the states. Thus there are only ve
types of communicating classes for a four-state MC, and this is because there are exactly ve ways
to partition the number 4 into non-zero integers, i.e.
4 = 3 + 1 = 2 + 2 = 2 + 1 + 1 = 1 + 1 + 1 + 1.
The motivated student might want to think of how to characterize this number of types for the general
n-state MC.
3