Chebysev Inequality

Suppose X is a parameter of a manufactured

2
item with known mean X µ and variance σX.

The quality control department rejects the

item if the absolute deviation of X from µ isX

greater than 2σX . What fraction of the

manufacturing item does the quality control
department reject? Can you roughly guess it?

σX2
P { X − µX ≥ ε} ≤ 2
ε
Contd..
∞

σx2 = ∫ (x − µX )2 fX (x )dx
−∞

∫
2
≥ (x − µX ) fX (x )dx
X −µX ≥ε

≥ ∫ ε2 fX (x )dx
X −µX ≥ε

= ε2P { X − µX ≥ ε}
σX2
∴ P { X − µX ≥ ε} ≤ 2
ε
 Markov Inequality
For a random variable X which take only
non-negative values P {X ≥ a} ≤ E (X )
a
∞
where a > 0.
E (X ) = ∫ xf X (x )dx
0
∞ ∞

≥ ∫ xfX (x )dx ≥ ∫ afX (x )dx

a a

= aP {X ≥ a }
E (X )
∴ P {X ≥ a } ≤
a
2
E (X − k )
Result: P {(X − k )2 ≥ a } ≤
a
 Convergence of a sequence of
random variables
Let X1 , X 2 ,..., X n be a sequence n independent and
identically distributed random variables. Suppose we want
to estimate the mean of the random variable on the basis
of the observed data by means of the relation
1 N
µˆX = ∑ X i
n i =1
How closely does µˆX represent µX as n is increased?
How do we measure the closeness between µˆX and µX ?
Notice that µˆX is a random variable. What do we mean by
the statement µˆX converges to µX ?
Contd..
Consider a deterministic sequence x1, x 2,....xn .... The
sequence converges to a limit x if correspond to any
ε > 0 we can find a positive integer m such that
x − xn < ε for n > m.
Convergence of a random sequence X1, X 2,....Xn cannot be
defined as above.

A sequence of random variables is said to converge

everywhere to X if
X (ξ ) − Xn (ξ ) → 0 for n > m and ∀ξ.
 Almost sure (a.s.) convergence or
convergence with probability 1
For the random sequence X1, X 2 ,....Xn ....{Xn → X } this is an
event.
P {X n → X } = 1 as n → ∞,
If P { Xn − X < ε for n ≥ m} = 1 as m → ∞,
then the sequence is said to converge to X almost sure or
with probability 1.
One important application is the Strong Law of Large
Numbers:
If X1, X 2 ,....Xn ....are iid random variables, then
1 N
∑
n i =1
Xi → µX with probability 1 as n → ∞.
Convergence in mean square sense

If E (Xn − X )2 → 0 as n → ∞ we say that the sequence

converges to X in mean square (M.S).

Example: If X1, X 2 ,....Xn ....are iid random variables, then

1 N
∑
n i =1
Xi → µX in the mean square 1 as n → ∞.

We have to show that

1 N
lim
n →∞
E ( ∑
n i =1
X i − µX )2
= 0
 Contd..
Now,
1 N
1 N

E( ∑
n i =1
Xi − µX ) = E( (∑ (Xi − µX ))
2

n i =1
2

1 N 1 n n

= 2 ∑ E(Xi − µX ) + 2 ∑ ∑ E(X
2
i
− µX )(X j − µX )
n i =1 n i=1 j=1,j≠i
2
nσX
= 2
+0 ( Because of independence)
n
2
σX
=
n
1 N

∴ lim E(
n →∞ n
∑ Xi − µX ) = 0 2

i =1
 Convergence in probability
P { X n − X > ε} is a sequence of probability. X n is said to
convergent to X in probability if this sequence of probability is
convergent that is P { Xn − X > ε} → 0 as n → ∞.

If a sequence is convergent in mean, then it is convergent in

probability also because
P { X n − X 2 > ε2 } ≤ E (Xn − X )2 / ε2 (Markov Inequality)

We have P { Xn − X > ε} ≤ E (Xn − X )2 / ε2

If E (Xn − X )2 → 0 as n → ∞, (mean square convergent)

then P { Xn − X > ε} → 0 as n → ∞.
 Convergence in distribution

The sequence X1, X 2 ,....Xn .... is said to converge to X in

distribution if
FXn (x ) → FX (x ) as n → ∞.
The two distribution function coincides.
 Central Limit Theorem
Consider independent and identically distributed random
variables X1, X 2 ,....Xn . Let Y = X1 + X 2 + ...Xn .
Then µY = µX + µX + ...µX
1 2 n
2 2 2 2
σ
And Y = σ X1 + σ X2 + ... + σ Xn

The central limit theorem states that under very general

conditions Y converges to N (µY , σY2 ) as n → ∞. The conditions
are :
1. The random variables X1, X 2 ,..., Xn are independent with
same mean and variance, but not identically distributed.

2. The random variables X1, X 2 ,..., Xn are independent with

different mean and same variance and not identically
distributed.
Jointly Gaussian Random variables
Two random variables X and Y are called jointly Gaussian
if their joint density function is
⎡ ( x −µ )2 ( x −µX )(y−µY ) (y−µY )2 ⎤⎥
− 1 ⎢ X −2ρXY +
⎢ σX σ ⎥
2(1−ρ2 ) ⎢ σ2 Y σ2 ⎥⎦
fX ,Y (x , y ) = Ae X ,Y ⎣ X Y

where A =
1
2 πσx σy 1−ρX2 ,Y

Properties :
(1) If X and Y are jointly Gaussian, then for any
constants a and b, then the random variable Z ,
given by Z = aX + bY is Gaussian with
mean µZ = a µX + bµY
2 2 2 2 2
and variance Z σ = a σ X + b σY + 2abσX σY ρX ,Y
 Contd..
(2) If two jointly Gaussian r.v.s. are uncorrelated, ρX ,Y = 0
then they are statistically independent.
fX ,Y (x , y ) = fX (x )fY (y ).

(3) If fX ,Y (x , y ) is a jointly Gaussian distribution, then the

marginal densities
fX (x ) and fY (y ) are also Gaussian.

(4) If X and Y are joint by Gaussian random variables then the

optimum nonlinear estimator X̂ of X that minimizes the
mean square error ξ = E {[X − Xˆ ]2 } is a linear estimator
X̂ = aY