Conditional Distributions and Transformations

Homework 10

Problems
1. Prove the law of total covariance.

Cov(Y1 , Y2 ) − E[Cov(Y1 , Y2 |X)] = Cov(E[Y1 |X]E[Y2 |X]).

Cov(Y1 , Y2 ) = E(Y1 Y2 ) − E(Y1 )E(Y2 ) (1)

E(Cov(Y1 , Y2 |X)) = E(E(Y1 Y2 |X)) − E(E(Y1 |X)E(Y2 |X))
(2)
= E(Y1 Y2 ) − E(E(Y1 |X)E(Y2 |X))
Cov(E[Y1 |X], E[Y2 |X]) = E(E[Y1 |X]E[Y2 |X]) − E(E(Y1 |X))E(E(Y2 |X))
(3)
= E(E[Y1 |X]E[Y2 |X]) − EY1 EY2
E[Cov(Y1 , Y2 |X)] + Cov(E[Y1 |X]E[Y2 |X]) = E(Y1 Y2 ) + EY1 EY2 (4)
So
Cov(Y1 , Y2 ) − E[Cov(Y1 , Y2 |X)] = Cov(E[Y1 |X]E[Y2 |X]) (5)

2. Consider the covariance matrix  

21 8
Σ=
8 9

(a) Find det(Σ).

det(Σ) = 21 × 9 − 8 × 8 = 125 (6)

(b) Find the eigenvalues and eigenvectors for Σ.

Solve det(Σ − λI) = λ2 − 30λ + 125 = 0 and get λ = 25 or 5.
Consider (Σ − 25I)X = 0, general solution is X1 = 2X2 , so eigenvector can be [2, 1]0 .
Consider (Σ − 5I)X = 0, general solution is X2 = −2X1 , so eigenvector can be [1, −2]0 .

(c) Give a linear transformation Y = AZ of two independent standard normal random variables Z1
and Z2 so that Cov(Y ) = Σ.

Cov(Y ) = Cov(AZ) = ACov(Z)A0 = AA0 = Σ (7)

1
 Conduct Cholesky Decomposition on Σ, and get
 
4.5826 1.7457
A= (8)
0 2.4398

3. Let X1 , X2 have density

P 
3
Γ i=1 αi
fX1 ,X2 (x1 , x2 ) = Q3 x1α1 −1 x2α2 −1 (1 − x1 − x2 )α3 −1 , x1 , x2 ∈ [0, 1], x1 + x2 < 1
i=1 Γ(α i )

αi > 0, i = 1, 2, 3.
(a) Find fX1 (x1 ). the density of X1 .
 
Z 1−x1 Γ P3 αi
i=1
fX1 (x1 ) = Q3 x1α1 −1 x2α2 −1 (1 − x1 − x2 )α3 −1 dx2
0 i=1 Γ(α i )
P 
3
Γ i=1 αi
Z 1−x1
x2 α2 −1 x2 α3 −1 x2
= Q3 x1α1 −1 (1 − x1 )α2 +α3 −1 ( ) (1 − ) d( )
i=1 Γ(αi ) 0 1 − x 1 1 − x 1 1 − x1
(9)
Let
x2
U= (10)
1 − x1
P 
3
Γ i=1 αi
Z 1
α1 −1 α2 +α3 −1
fX1 (x1 ) = Q3 x1 (1 − x1 ) U α2 −1 (1 − U )α3 −1 dU
Γ(α i ) 0
i=1
P3 
Γ i=1 αi Γ(α2 )Γ(α3 )
= Q3 x1α1 −1 (1 − x1 )α2 +α3 −1 (11)
i=1 Γ(α i ) Γ(α 2 + α3 )
P 
3
Γ i=1 αi
= xα1 −1 (1 − x1 )α2 +α3 −1
Γ(α1 )Γ(α2 + α3 ) 1

(b) Find fX2 |X1 (x2 |x1 )

f (x1 , x2 )
fX2 |X1 (x2 |x1 ) =
f (x2 )
Γ( 3i=1 αi )
P
1 −1 α2 −1
Q 3
Γ(αi )
xα
1 x2 (1 − x1 − x2 )α3 −1
i=1
= P3 (12)
Γ( i=1 αi ) α1 −1
Γ(α1 )Γ(α2 +α3 ) x1 (1 − x1 )α2 +α3 −1
2 −1
Γ(α2 + α3 )xα 2 (1 − x1 − x2 )α3 −1
=
Γ(α2 )Γ(α3 )(1 − x1 )α2 +α3 −1

2
(c) Find E[X1 X2 ]
Z 1−x1
E(X1 X2 |x1 ) = x1 x2 f (x2 |x1 )dx2
0
2 −1
1−x1
Γ(α2 + α3 )xα (1 − x1 − x2 )α3 −1
Z
2
= x1 x2 dx2
0 Γ(α2 )Γ(α3 )(1 − x1 )α2 +α3 −1
Z 1−x1
Γ(α2 + α3 ) x1 α3 −1
= xα2 (1 − x1 − x2 )
2
dx2
Γ(α2 )Γ(α3 ) (1 − x1 )α2 +α3 −1 0
Z 1 (13)
Γ(α2 + α3 )
= x1 (1 − x1 ) U α2 (1 − U )α3 −1 dU
Γ(α2 )Γ(α3 ) 0
Γ(α2 + α3 ) Γ(α2 + 1)Γ(α3 )
= x1 (1 − x1 )
Γ(α2 )Γ(α3 ) Γ(α2 + α3 + 1)
Γ(α2 + α3 )Γ(α2 + 1)
= x1 (1 − x1 )
Γ(α2 )Γ(α2 + α3 + 1)

E(X1 X2 ) = E(E(X1 X2 |x1 ))

Z 1
Γ(α2 + α3 )Γ(α2 + 1)
= x1 (1 − x1 )dx1
0 Γ(α2 )Γ(α2 + α3 + 1)
Γ(α2 + α3 )Γ(α2 + 1) (14)
=
6Γ(α2 )Γ(α2 + α3 + 1)
α2 Γ(α2 + α3 )Γ(α2 )
=
α1 + α2 + α3 6Γ(α2 )Γ(α2 + α3 )

(d) Find Cov(X1 , X2 ).

P 
3
Z 1 Γ i=1 αi
EX1 = xα1 (1 − x1 )α2 +α3 −1 dx1
0Γ(α1 )Γ(α2 + α3 ) 1
Γ(α1 + α2 + α3 ) Γ(α1 + 1)Γ(α2 + α3 ) (15)
=
Γ(α1 )Γ(α2 )Γ(α3 ) Γ(α1 + α2 + α3 + 1)
α2 Γ(α2 + α3 )
=
α1 + α2 + α3 Γ(α2 )Γ(α3 )

3
 Z 1−x1
E(X2 |x1 ) = x2 f (x2 |x1 )dx2
0
1−x1
Γ(α2 + α3 )x2α2 −1 (1 − x1 − x2 )α3 −1
Z
= x2 dx2
0 Γ(α2 )Γ(α3 )(1 − x1 )α2 +α3 −1
Z 1−x1
Γ(α2 + α3 ) 1 α3 −1
= xα
2 (1 − x1 − x2 )
2
dx2
Γ(α2 )Γ(α3 ) (1 − x1 )α2 +α3 −1 0
Z 1 (16)
Γ(α2 + α3 )
= (1 − x1 ) U α2 (1 − U )α3 −1 dU
Γ(α2 )Γ(α3 ) 0
Γ(α2 + α3 ) Γ(α2 + 1)Γ(α3 )
= (1 − x1 )
Γ(α2 )Γ(α3 ) Γ(α2 + α3 + 1)
Γ(α2 + α3 )Γ(α2 + 1)
= (1 − x1 )
Γ(α2 )Γ(α2 + α3 + 1)

E(X2 ) = E(E(X2 |x1 ))

Z 1
Γ(α2 + α3 )Γ(α2 + 1)
= (1 − x1 )dx1
0 Γ(α2 )Γ(α2 + α3 + 1)
Γ(α2 + α3 )Γ(α2 + 1) (17)
=
2Γ(α2 )Γ(α2 + α3 + 1)
α2 Γ(α2 + α3 )Γ(α2 )
=
α1 + α2 + α3 2Γ(α2 )Γ(α2 + α3 )

Cov(X1 , X2 ) = E(X1 X2 ) − E(X1 )E(X2 )

α2 Γ(α2 + α3 )Γ(α2 ) 1 α2 Γ(α2 + α3 )Γ(α2 ) 2 (18)
= − { }
α1 + α2 + α3 6Γ(α2 )Γ(α2 + α3 ) 2 α1 + α2 + α3 Γ(α2 )Γ(α2 + α3 )

Challenging Problems

1. Consider an urn with ` white, m gray, anf n black balls. Draw k. Let W be the number of black balls,
G be the number of gray balls, and B be the number of black balls.
(a) Find the mass function fW,B (w, b) for the number of white and black marbles.

Clw Cnb Cm
k−w−b
fW,B (w, b) = k (19)
Cl+m+n

(b) Find EW, EG, EB.

n n k−w
X C w C b C k−w−b
n m Clw X Clw Cm+n
fW (w) = l
k
= k
Cnb Cm
k−w−b
= k (20)
b=0
Cl+m+n Cl+m+n b=0
Cl+m+n

4
 l n k−b
X Clw Cnb Cm
k−w−b
Cnb X
w k−w−b
Cnb Cl+m
fB (b) = k
= k
C l Cm = k
(21)
w=0
Cl+m+n Cl+m+n b=0
Cl+m+n
So W and B follow hyper-geometric distribution.
lk
EW = (22)
l+m+n
nk
EB = (23)
l+m+n

(c) Find E[W |G] and E[B|G].

Clw Cn
k−w−G G
Cm
f (w, G) k
Cl+m+n Clw Cnk−w−G
f (w|G) = = G C k−G
= k−G (24)
f (G) Cm l+n Cl+n
k
Cl+m+n

l l
X X Clw Cnk−w−G l(k − G)
E(W |G) = wf (w|G) = w k−G
= (25)
w=0 w=0
Cl+n l+n
b k−b−G G
Cn Cl Cm
f (b, G) k
Cl+m+n Cnb Clk−b−G
f (b|G) = = G C k−G
= k−G (26)
f (G) Cm l+n Cl+n
k
Cl+m+n

n l
X X C b Clk−b−G n(k − G)
E(B|G) = bf (b|G) = b n k−G = (27)
Cl+n l+n
b=0 b=0

(d) Find Cov(W, B|G)

(e) Find Cov(W, B)
2. Let Z1 , Z2 be a bivariate standard normal with correlation ρ. Find the correlation of Z12 and Z22 .
Since Z1 and Z2 are standard normal random variables, Z12 and Z22 follow χ21 distribution, which means

E(Z12 ) = E(Z22 ) = 1 (28)

V ar(Z12 ) = V ar(Z22 ) = 2 (29)

5
 Z ∞ Z ∞
1 1
E(Z12 Z22 ) = z12 z22 p exp(− (z 2 − 2ρz1 z2 + z22 ))dz1 dz2
−∞ −∞ 2π(1 − ρ2 ) 2(1 − ρ2 ) 1
Z ∞ Z ∞
1 1
= z12 z22 p exp(− ((1 − ρ2 )z12 + (z2 − ρz1 )2 )dz1 dz2
−∞ −∞ 2π(1 − 2(1
ρ2 ) − ρ2 )
Z ∞ Z ∞
1 1 1
=p z12 exp(− 2
(1 − ρ 2 2
)z 1 )) z22 exp(− ((z2 − ρz1 )2 )dz1 dz2
2
2π(1 − ρ ) −∞ 2(1 − ρ ) −∞ 2(1 − ρ2 )
Z ∞
1 1 p
=p z12 exp(− 2
(1 − ρ2 )z12 )) 2π(1 − ρ2 )(ρ2 z12 + 1 − ρ2 )dz1
2π(1 − ρ2 ) −∞ 2(1 − ρ )
Z ∞ 2
z
= z12 exp(− 1 ))(ρ2 z12 + 1 − ρ2 )dz1
−∞ 2
Z ∞ Z ∞
z2 z2
= ρ2 z14 exp(− 1 ))dz1 + (1 − ρ2 ) z12 exp(− 1 ))dz1
−∞ 2 −∞ 2
2
√ 2
√
= 3ρ 2π + (1 − ρ ) 2π
√
= (1 + 2ρ2 ) 2π
(30)
2 2
Cov(Z 1 , Z 2 )
Corr(Z12 , Z22 ) = p
V ar(Z12 )V ar(Z22 )
√ (31)
E(Z12 Z22 ) − 1 (1 + 2ρ2 ) 2π − 1
= =
2 2