EECE 641 Information Theory October 24, 2016

American University of Beirut, Fall 2016-2017

Midterm

Name:
ID:

Do not open this booklet until you are told to do so

• You have 2 hour 15 minutes to complete the exam.

• Write all answers in the answer booklet. Answers written elsewhere will not be
graded.
• This is a closed-book exam except for three A4 sheets that are allowed.
• Calculators are allowed.
• The problems are not necessarily in order of difficulty.
• A correct answer does not guarantee full credit and a wrong answer does not guar-
antee loss of credit. You should concisely indicate your reasoning and show all
relevant work. The grade on each problem is based on our judgment of your level
of understanding as reflected by what you have written.
• If we can’t read it, we can’t grade it.

Problem Score
#1 / 15

#2 / 15

#3 / 15

#4 / 10

#5 / 30

#6 / 15

Total / 100
Problem M.1
For each of the following statements, answer by indicating whether the statement is true
or false. No justification is necessary.

(a) For any two independent random variables, H(X + Y ) ≥ H(X).

(b) For any two random variables, H(X + Y ) ≥ H(X).

(c) For any two random variables, H(X|Y ) ≥ H(X).

(d) For any two random variables, H(X|Y ) ≤ H(Y ).

(e) For any two independent random variables, H(max{X, Y }) ≤ H(X) + H(Y ).

(e) For any two independent random variables, H(max{X, Y }) ≥ H(X).

Problem M.2
Consider a Discrete Memoryless Channel (DMC) where the channel input alphabet X
has a finite cardinality as well as that of the output alphabet Y.
You are told that there is an element xo ∈ X such that
1
pY |X (y|xo ) = , ∀y ∈ Y,
|Y|

i.e., given input xo , the output is equally likely to be any output symbol.
Intuition indicates that symbol xo should not appear in any codeword. Is this statement correct?
If yes prove it. If not, either prove that it is not necessarily the case, or provide a counter
example where symbol xo is “clearly useful”.
Whether you answer with “yes” or “no” find an expression for capacity.

Problem M.3
Let X be a discrete random variable defined on the (countable) set of natural numbers
with the following Probability Mass Function (PMF):

pX (k) = ae−k , k = 0, 1, 2, · · · .

(a) Find an expression for a so that the PMF is valid.

(b) Derive the mean of variable X.

(c) Now consider any random variable Y , defined on N whose mean is equal to that of
X found in (b).
Prove that the entropy of Y is less or equal to that of X.

1
Problem M.4
Prove that the expected length of a Huffman code is always less that 1 + log |X |.

Problem M.5
In this problem we consider a model of a communication channel between you and your
3aris/3arous. The channel is a Discrete Memoryless Channel (DMC) that model is mood-
dependent. More precisely, if his/her mood is “good”, the channel is a BSC with crossover
probability of 0.11. If the mood is “bad”, the crossover probability is 0.22.
Assume that the mood {Mi }i is a Bernoulli process where at every time step i, the
variables are independent and each has a probability of being good equal to p.

(a) Prove that capacity of this channel is

C = max I(X; Y |M ) = max [pI(X; Y |”g”) + (1 − p)I(X; Y |”b”)]
where the maximization is over all probability distributions on X given M .
You are asked to provide a coding theorem as well as a converse.
Note: One may think along the lines of “time sharing” and make the analysis correct.
However, there are other approaches.
(b) Compute the capacity of the channel at hand function of p.

Note: Hb (0.11) ≈ 0.5, Hb (0.22) ≈ 0.75.

Problem M.6
You are playing a gambling game with a weird-shaped coin where the results are IID with
the following statistics
H with probability:0.6
T with probability:0.3
H & T with probability:0.1.

You are allowed to bet on either “H” or “T” or both, and the odds that are offered are
as follows:
H:2
T:3
H & T:y on H&z on H.

Let b = (bH , bT ) denote a betting vector, where bH is the fraction of your wealth allocated
to “H”. It is assumed that bi ≥ 0, and that all of the wealth is reinvested, so that
bH + bT = 1.

(a) If y = 1 and z = 10, find the optimal strategy b∗ .

(b) If y = 0, find the optimal strategy b∗ as a function of z > 0.