OCT 2011

Introduction to Audio Signal Processing

I. C ONVERGENCE OF RANDOM VARIABLES C. Law Of Large Numbers We arrive at the concept of relative frequency approach to determining probability,by performing the experiment large number of time and evaluate the frequency of occurrence of the event,as N tends to innity we expect the relative frequency to approach the probability ,but this happens under certain assumptions. Consider the experiment of tossing of a coin. We model the repeated coin toss experiment as sequence of N independent Bernoulli trials. Let Xi be the random variable that represents the output of the ith Bernoulli trial The random variable Xi takes on the following values Xi = 1 if heads occurs Xi = 0 if tail occurs We repeate the experiment N times. We model the overall experiment output by random vector X X=[X1 X2 .. Xn] we assume that successive trials are independent and performed under identical conditions. We assume that the random variables Xi are iid Bernoulli random variables. The marginal PMF can be given by PX [k]=0.5 k=0 PX [k]=0.5 k=1 Consider the sample mean of the N iid random variables obtained by performing the experiment N times. n 1 Xi Sn = N i=0 The sample mean is also a function of random variable. We will observe the behaviour of the sample mean as N We will characterize the behaviour of the sample mean N by observing the behaviour of the moments of the sample means as N . n 1 Expected value of Sample mean EX [Sn] = EX [Xi ] N i=0 1 EX [Sn] = 2 the expected value of the sample mean is not affected by N n 1 Variance of sample mean var(Sn )=var( Xi ) N i=0
n 1 var(Sn )= N 2 var( i=0

I
A.

N this document we look at the concept of convergence of random variable.

For random variables that are the sum of a large number of independent and identically distributed random variables we can derive methods to approximate the random variables. Two theorems in this context are law of large numbers and central limit theorem. Law of large numbers states that the sample mean of IID random variable converges to expected value. Central limit theorem states that the normalized sum of IID random variable converges to Gaussian random variable. B. Convergence and sum of terms 1 + ai . Where N is large N 1 N i=0 The expression can be evaluated in closed form and exact n n 1 1 1 a a( N + 1) value is given by 1+ ai = 1 + N i=0 N i=0 N 1a we can plot the Sn vs N for increasing values of N. From the consider the sum
Fig. 1. plot of Sn v/s N
n

plot we can observe .As N increases Sn tends to values 1. If N is sufciently large,Sn will differ by 1 by very small amount. This is only true as long as values of a 1. This small difference is the error in approximation. The error can in approximation can be expressed as ( 1 |Sn 1| = N aa N +1) In general this will depend on a as 1a well as N. The error will depend on the specic sequence. We can how ever assert that the sequence will converge for large N as long as a1 ie Sn will converge to 1. If a=0.9999 we require a very large N for Sn to be close the 1 so that the error in approximation is small.

Xi )

Since all the Xi are IID. 1 var(Sn )= N var(Xi ) For Bernoulli random variables var(Xi )=p(1-p) For the case of coin tossing experiment we have p=0.5 var(Xi )= 1 4 And as N var(Xi ) 0

OCT 2011

This means that as N increases,the width of the PMF decreases and eventually goes to 0. var(Xi ) = E[(X E(X))2 ] 0 which means X-E(X) 0 which implies XE(X). Thus the random variable is no longer random but a constant. Such a random variable is called degenerate random variable
n

Consider the random variable Xn ==

i=0

Xi

The PMF of the random variable is can be modelled as a binomial random variable. Xn Bin(N, 1/2) N f ork = 1...N The PMF can be given by PXn [k] = N 1 2 k The PMF of random variable Sn = Xn can be obtained by N applying transformation of a discrete random variable and k noting that Sn takes on values uk = N N 1 N k PSn [uk ] = N uk 2 for uk = N k = 1...N we plot the PMF for various values of N
Fig. 2. plot of Sn v/s NN=10

independent fair toss coins. we will obtain the sample mean of 0.5 for N large enough. Thus X 1 =E(X) 2 This is called Bernoulli law of large numbers. In general X p = E(X) This tells us that the relative frequency of probability of occurrence of heads in large number of independent identical trials . This is the justication for using relative frequency approach to determine probability. This also justies the sample mean to estimate the expectation of random variable. In general if we consider the sample mean of function of n random variable Xi we can use the sample mean to estimate any moment of random variable. we can generalize the law of large numbers D. Law Of Large Numbers If X1,X2......,XN are iid random variables with mean E(X) and variance 2 < then limN Sn = E(X) Let us consider the error in approximation of sample mean Sn-E(X) let us calculate the probability that the error in approximation is exceeds certain threshold . We can obtain a bound for the probability using chebyshev inequality Pr(|Sn E(X)| > ) var(Sn)
2

Fig. 3.

plot of N=30

Pr(|Sn E(X)| > )

2 N 2

now we take the limits on both sides as lim Pr(|Sn E(X)| > ) lim 2 N N 2

Fig. 4.

plot of N=100

lim Pr(|Sn E(X)| > ) 0

lim Pr(|Sn E(X)| > ) = 0

as N increases the PMF takes on values more densely in the range [0-1]. Also PMF takes on values more densely about u=0.5 indicating that probability that sample means take on the value of 0.5 increases. if we plot the CDF then as N increases CDF approaches unit step. If N is large enough the sample mean random variable will always yield a constant number 0.5 Every time we perform Bernoulli Trial consisting of N

This statement implies that sample mean random variable converges to expected value of single random variable.If N is large enough the probability of error in approximation exceeding the threshold will be small.This conditions says that Sn = E(X) in probability. This does no imply that all realizations of the random variable will converge.

OCT 2011

There may be some realizations for which error in approximations is greater that for a given large N,however the probability of this happening becomes small as N increases. This is called convergence in probability E. Sample mean of IID Gaussian random variables if we consider a sequence of gaussian iid random variables X1,X2.....,XN with mean E(X) and nite variance 2 then as per the law of large numbers the sample mean converges to E(X). We must note that this is true only of the random variables are IID with nite mean E(X) and nite variance 2 . We do note need the PMF to establish this result.However if we need to determine the error in approximation we need to determine PMF of the sample mean. The Law of large numbers does not describe how PMF varies with increasing N,it only states that the with of the PMF decreases and it becomes more densely concentrate about the mean E(X). F. Central Limit Theorem Consider the continuous random variable U(-1/2,1/2) uniformly distributed random variable we will look at the n PDF of sum of random variables Consider the sum
i=0

Fig. 7.

PDF of sum of random variable S3=X1+X2+X3 and gaussian PDF

if we continue the convolution process the mean will remain at 0 and variance will be N/12; below is the gure with sum of 10 variables
Fig. 8. PDF of sum of random variable S10 and gaussian PDF

Xi

if N=2 S2=X1+X2 PDF of S2 can be found by convolution integral. below is plot of the PDF of random variable S2
Fig. 5. PDF of sum of random variable S2=X1+X2

As the number of terms in the sum hence the number of IID random variables increases the PDF of sum of random variables approaches the PDF of gaussian random variables This is the essence of the central limit theorem in general we can determine the PDF of sum of N random variables using convolution of all the N random variables. Below is the plot of PDF of S3
Fig. 6. PDF of sum of random variable S2=X1+X2

we compare the plots obtained with the PDF of a gaussian random variable with variance zero mean and variance equal to variance of uniform PDF. var(X)=(b-a)/12; we can see that the gaussian PDF resembles the PDF of S3

Now Let us consider the uniformly distribute random variable between 0 and 1.And observe the PDF of sum of these random variables.We expect the mean of sum to approach NE(X) and variance to approach N*var(X) we us see the plot for N=10 Mean and variance both increase as number of terms in the sum increases. Thus we cannot say whether the PDF converges to any other PDF. To analyse the convergence we normalize the sum so that mean and variance are xed as N increases .Thus we x the mean at 0 and variance at 1.if Sn is the sum of the random variables The normalized sum is given by Sn E(Sn) var(Sn)) = Sn N E(X) N var(X))

OCT 2011

Fig. 9.

PDF of sum of U(0,1) random variable S10 and gaussian PDF

Equivalently the CDF of the standardized sum Pr( Sn E(Sn) var(Sn))

x

x)

1 1 exp( t2 ) dt 2 2

Let us consider the sum of squares of independent Gaussian Random Variables N (0, 1) Let Xi N (0, 1) for i=0,1,....N and assume that all Xi be iid . n 2 We need to determine the approximate PDF of Y = Xi as N becomes large. We can use the central limit theorem to nd the approximate 2 PDF of standardized sum of random variable Xi . To apply central limit theorem we must note that since Xi 2 are independent so are Xi any function of independent random variables are independent thus as N
n i=0 2 Xi N E(X 2 ) i=0

Fig. 10. PDF of normalized sum of U(0,1) random variable S10 and gaussian PDF

N var(X 2 ) E(X 2 ) = 1andvar(X 2 ) = 2

n i=0

N (0, 1)

2 Xi N = N (0, 1) 2N

For nite N we can view it as a approximation

n

Y =
i=0

2 Xi N (N, 2N )

This is the central limit theorem which says that PDF of standardized sum of large number of continuous IID random variables will converge to a Gaussian PDF. Thus if we have random variables arising from small similar physical effect.We can make the IID assumption to assert that the PDF is gaussian.There is no need to know the PDF of each random variable to determine the exact PDF of the sum.We only require the mean and variance of the random variable. The CDF of the standardized sum converges to the CDF of the Gaussian random variable. Central Limit Theorem For Continuous Random Variable If X1,X2....XN are continuous IID random variables ,each n with mean E(X) and variance var(X) and sum Xi then as N
i=0

The approximation becomes better as N increases. Characteristic function of standardized X 2 converges to that of standardiz

Central limit theorem for discrete random variable

Sn E(Sn) var(Sn))

Xi N E(X) =
i=0

N var(X)

N (0, 1)