RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

COMS4100/7105: Digital Communications Lecture 3: Random Processes

Mandar Gujrathi
Bldg 78, Room 312

August 2, 2011

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Overview

Random Variables Random Processes Noise Signal to Noise ratio Transmission and Filtering Digital Communications. Formatting analog information.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random Variables

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Discrete Random Variables

Example: S= { Tail, Head}, Sx = {0, 1} A function that maps S into Sx is called as a Random Variable,

denoted by X (.)
Sx is countable: Discrete Random Variable One to one mapping, Many to one mapping Sx is uncountable or innite: Continuous Random Variable

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

One to one mapping

X (s2 )

Tail

x1 1
s1

Head

x2 0

S X {1,0}

X (s1)

Sample Space S

Example: S= { Tail, Head}, Sx = {1,0}

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Probability of Random Variables

The random variable X(.) maps on an event si from a sample space

S to xi in the sample space Sx

In one to one mapping its basically the same event with dierent

names. Hence, P[X (s) = xi ] = P[sj : X (sj ) = xi ] = P{si }

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Probability of Random Variables

For many to one mapping

P[X (s) = xi ] = P[sj : X (sj ) = xi ] =

j:X (sj )=xi

P[sj ]

fX [xi ] = P[X (s) = xi ]

fX [xi ] is called the Probability Mass Function

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Continuous Random Variable

Example: Length of waiting times at an airport Sx is innite and uncountable: Continuous R.V. Dicult to assign a specic probability to each value of X But we can calculate the probability of X lying in an interval.

exponential distribution 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

px(x)

10

15 x

20

25

30

P[a X b] =
a

fX (x)dx

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Probability Density function (PDF)

fX (x) is called as the PDF, fX [xi ] is called the PMF. Properties
PDF/ PMF must be non negative

fX (x) 0
PMF must sum to 1
M

fX [xi ] = 1
i=1

fX [xi ] = 1
i=1

PDF must integrate to 1

fX (x)dx = 1

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Expectation
Expectation E [X ], Mean

E [X ] =
i

xi fX [xi ]

xfX (x)dx g (x)fX (x)dx

=
Expectation is linear

E [a1 X1 + a2 X2 ] = a1 E [X1 ] + a2 E [X2 ]

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

nth moment
Expectation E [X n ].

E [X ] =
i

xin fX [xi ]

x n fX (x)dx

Variance, 2

Var (X ) = E [(X )2 ] 2 = E [(X E [X ])2 ]

(X E [X ])2 fX (x)dx

Variance is a non-linear operation

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Two Random Variables

HH TH TT HT

1
x

S XY
S

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Concept of Multiple Random Variable

Outcome of 2 coin tosses = {HT , TH, TT , HH}

To map these events we require 2 random variables. But we are mapping from the same sample space

X (si ) x = i S Y (si ) yi Two random variables dened on the same sample space are Jointly distributed. 1 0 0 1 SX ,Y = , , , 0 1 0 1 These are discrete events, so joint PMF

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Joint PMF
For a single RV, PMF: fX [xi ] = P[X (s) = xi ] For a multiple RV, Joint PMF:

fX ,Y [xi , yj ] = P[X (s) = xi , Y (s) = yj ],i = 1, . . . , Nx j = 1, . . . , Ny

Properties of Joint PMF

0 fX ,Y [xi , yj ] 1
Nx Ny

fX ,Y [xi , yj ] = 1
i=1 j=1

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Joint distributions

r =1

Sample Space S

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Joint distributions

r =1

(r , )

Sample Space S

New Sample Space Sr , Sr , = {(r , ) : 0 r 1, 0 2}

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random Processes

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random Signals and Process

Random Variables

X (s2 )

Tail

x1 1
s1

Head

x2 0

S X {1,0}

X (s1)

Sample Space S

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random process example

Mapping the outcomes from the original experimental sample space.

S = {(H, H, T , . . . ), (H, T , H, . . . ), (T , T , H, . . . ), . . . } SX = {(1, 1, 0, . . . ), (1, 0, 1, . . . ), (0, 0, 1, . . . ), . . . } = {x1 , x2 , x3 , . . . }

with each outcome of the random process denoted as

x1 = (x[0], x[1], . . . )
The random process is a mapping from S which is a set of innite

and sequential experimental outcomes to SX which is an innte sequence of realisations.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random process
1.5 X[n,s1] = x1[n] 1

x1[n]
0.5 0 0

10 n

12

14

16

18

20

1.5 X[n,s2] = x2[n] 1

x2[n]
0.5 0 0

10 n

12

14

16

18

20

1.5 X[n,s3] = x3[n] 1

x3[n]
0.5 0 0

10 n

12

14

16

18

20

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random process
A sample space or ensemble composed of functions of time is called

Random or stochastic process

Suppose s1 , s2 , . . . , sn form the sample points of a sample space S

X (t, s), T t T xj (t) = X (t, sj )

x1 ( t )
x1 ( t k )

s1
s2
s3
Sample Space

x 2 (t )

x 2 (t k )

x3 (t )

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random process
Is an ensemble composed of functions of time. For a single sample point s1 we get a single function of time x1 (t). If time is held constant t1 we get a random variable. For a given random process, the mean value of X (t) at arbitrary

time t is dened as X (t) = E [X (t)]

E [X (t)] is the ensemble average obtained by averaging over all the

sample functions with t held constant.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Ensemble and Time Averages

The ensemble average at a given time t is dened as

E [X (t)] =

xfX (t) (x)dx

In practice, we often have access to a single realisation of stochastic

process
In this case it is possible to dene the time average of a signal

1 x(t) = lim T T

T 2

x(t)dt.
T 2

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Ergodic Process

A process is said to be ergodic if all time averages are equal to the

corresponding ensemble averages.

As per the denition,

E [X (t)] = x(t) = x
2

X (t) x

2 = (x(t) x )2 = x

Variance

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Random Process: Operations

Autocorrelation

Rx (t1 , t2 ) = E [X (t1 )X (t2 )]

x1 x2 fX (t1 )X (t2 ) (x1 x2 )dx1 dx2

Autocovariance

Cx (t1 , t2 ) = E [{X (t1 ) x (t1 )}{X (t2 ) x (t2 )}] = RX (t1 , t2 ) x (t1 )x (t2 )

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Operations: Random Process

Crosscorrelation

RXY (t1 , t2 ) = E [X (t1 )Y (t2 )]

xyfX (t1 )Y (t2 ) (xy )dxdy

Cross covariance

CXY (t1 , t2 ) = E [{X (t1 ) x (t1 )}{Y (t2 ) y (t2 )}] = RXY (t1 , t2 ) x (t1 )y (t2 )

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Operations
The two random processes are said to be uncorrelated if

CXY (t1 , t2 ) = RXY (t1 , t2 ) x (t1 )y (t2 ) = 0 RXY (t1 , t2 ) = x (t1 )y (t2 ) RXY (t1 , t2 ) = E [X (t1 )]E [Y (t2 )].
For a complex Stochastic Process, autocorrelation is

Z (t) = X (t) + jY (t) RZ (t1 , t2 ) = E [Z (t1 )Z (t2 )]

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Types of Random Processes

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Strict Sense Stationary Process

A random process X (t). At times t1 , t2 , . . . , tn , we observe random

variables X (t1 ), X (t2 ), . . . , X (tn ).

The joint p.d.f is fX (t1 ),X (t2 ),...,X (tn ) x1 x2 . . . xn . A random process is strict sense stationary if

fX (t1 + ),X (t2 + ),...,X (tn + ) (x1 x2 . . . xn ) = fX (t1 ),X (t2 ),...,X (tn ) (x1 x2 . . . xn )
The joint p.d.f of random variables obtained by observing the

random process is invariant to time shift.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

IID random process

So is IID random process stationary? Yes, Marginal PDF is the same for each RV. Therefore

fX (t1 + ),X (t2 + ),...,X (tn + ) (x1 x2 . . . xn ) = fX (t1 ),X (t2 ),...,X (tn ) (x1 x2 . . . xn )
Any process whose mean or variances change with time is NOT

stationary.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Wide sense Stationary Process

Using joint p.d.f, rst order distribution function, i.e. for n = 1 will

be fX (t1 + ) (x) = fX (t1 ) (x)

If t1 = 0, then

fX ( ) = fX (0) (x)

Since the PDF does not depend on a particular time, so should not

mean/expectation. Therefore

E [X (t)] =

xfX (x)dx = x = constant

Mean of a wide-sense stationary process is constant

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Wide sense stationary process

Now, for n = 2 we have:

fX (t1 + )X (t2 + ) (x1 x2 ) = fX (t2 )X (t1 ) (x1 x2 )

If = t1 we have,

t1 , t2

fX (t1 t1 )X (t2 t1 ) (x1 x2 ) = fX (t2 )X (t1 ) (x1 x2 )

This leads to

t1 , t2

E [X (t1 )X (t2 )] = E [X (0)X (t2 t1 )] RX (t1 , t2 ) = RX (t2 t1 )

Such a process is also called weakly stationary.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Covariance

Recall that autocovariance of a random process is

Cx (t1 , t2 ) = E [{X (t1 ) x (t1 )}{X (t2 ) x (t2 )}] = RX (t1 , t2 ) x (t1 )x (t2 )
If it is WSS,

CX (t1 , t2 ) = RX (t2 t1 ) 2 x

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

To summarise
A second order stationary process can be wide-sense stationary but

the converse is not true.

A random process is ergodic if all time averages of sample functions

equal to corresponding ensemble averages.

A random process is wide-sense stationary when the mean is

independent of time and the autocorrelation function depends on the time dierence.
A random process is strictly stationary when the statistics do not

change regardless of any shift in time.

A strict sense stationary process can be wide sense stationary but

the converse is not true.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Properties of correlation for WSS process

An Autocorrelation function is dened as

RX ( ) = E [X (t + )X (t)] t
Mean square value of the process can be obtained by = 0.

RX (0) = E [X 2 (t)] this is second moment

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Properties
Autocorrelation is an even function
RX ( ) = RX ( )
100

80

60

autocorrelation

40

20

20

40 100

80

60

40

20

0 shift

20

40

60

80

100

>> x = randn(1,101); >> [a, shift] = xcorr(x); >> plot(shifts,a);

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Properties of correlation

Autocorrelation function RX ( ) has the maximum magnitude at

=0 E [(X (t + ) + X (t))2 ] 0 E [X 2 (t + )] + E [X 2 (t)] + 2E [X (t + )X (t)] 0 2RX (0) + 2RX ( ) 0 RX (0) |RX ( )|

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Joint stationary

For jointly WSS processes,

RXY ( ) = RYX ( )

If for all , RXY ( ) = 0 we say X(t) and Y(t) are orthogonal If for all , CXY ( ) = 0 we say X(t) and Y(t) are uncorrelated.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Example

Consider a sinusoidal signal with random phase dened as

X (t) = X (t) cos(2fc t + )

where A and fc are constants and is a random variable uniformly distributed over the interval [, ] and independent of X (t). Obtain the Rx ( ) and PSD.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Homework

Consider a pair of quadrature modulated process that are related to

stationary process as X1 (t) = X (t) cos(2fc t + ) X2 (t) = X (t) sin(2fc t + ) where fc is a carrier frequency and is a random variable uniformly distributed over the interval [0, 2] and independent of X (t). Obtain the cross correlation.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Filtering Stochastic Processes

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Filtering Stochastic Processes

Consider a pair of LTI lters and process as shown.

X (t) h1 (t)

V (t)

Y (t) h2 (t)

Z (t)

The cross correlation and cross spectral densities can be written as

Rvz ( ) = v ( ) z( )
= h1 ( ) x( ) h2 ( ) y ( ) Rvz ( ) = h1 ( ) h2 ( ) Rxy ( ) Gvz (f ) = H1 (f )H2 (f )Gxy (f )

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Quadrature Filters

A common tool in communications is the phase shifter, a device that

shifts the phase of its input by some number of degrees, A cos(2f0 t) 90 phase shifter A cos(2f0 t 90 )

Considered in terms of its constituent complex exponentials, this

phase shifter is shifting

the phases of positive frequencies by /2 and the phases of negative frequencies by +/2.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Quadrature Filters
The transfer function of a Quadrature lter can be written as

HQ (f ) = jsgn(f ) = j = +j
Hence,
HQ ( f )

f >0 f <0

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Quadrature Filters
If we want to write the impulse response,

F[sgn(t)] = F[

1 jf

1 ] = sgn(f ) = sgn(f ) jt j 1 = F 1 [jsgn(f )] = jt t If F(x(t)) = X (f ) then F(X (t)) = x(f ) Duality

Hence the impulse response is

hQ (t) =

1 . t

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Quadrature Filters and Hilbert Transforms

Hence, given an input g (t), to produce a 90 -phase shifted output

g (t), we perform the convolution g (t) = 1 1 g (t) = t

g ( ) d. t

The integral formula is known as the Hilbert transform and g (t) and

g (t) as a Hilbert transform pair.

The lter is known as a Hilbert transformer or a quadrature lter.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Quadrature Filters and Hilbert Transforms

If g (t) is the Hilbert transform of g (t) then g (t) is the Hilbert

transform of g (t).
The inverse Hilbert transform is given by

g (t) =

g ( ) d. t

Observe that, if g (t) is real, so is g (t). In this case, pairs are

orthogonal:

g (t) (t)dt = 0. g

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Noise

A communications signal at the receiver is usually modelled as a

stochastic process. We usually identify two components in the process:

the information-bearing component the signal and the component that bears no information the noise. We sometimes identify a third component: a component that bears

unwanted information, that of another user interference.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Thermal Noise
Produced by random motion of charged particles in conducting

media.
In 1928, Johnson & Nyquist studied noise observed in resistors

(Johnsons noise/ resistance noise).

They found that the noise voltage can be modelled as Gaussian. Furthermore, it has a spectral density

GV (f ) =

2Rh|f | e h|f |/kT 1

V 2 /Hz

R is the resistance (), T is the temperature (K), k = 1.38 1023 J/K is Boltzmanns constant, h = 6.62 1034 J s is Plancks constant.

It turns out that this density is almost at up to infra-red fs.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Thermal Noise

This could be written as

Gv (f ) = 2RkT 1

h|f | 2kT

|f | <<

kT h

Hence, we tend to use the (very accurate) approximation

GV (f ) = 2RkT

V 2 /Hz.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

White & Coloured Noise

Thermal noise has (for our purposes) a at spectral density. By analogy to visible light, we call such noise white noise. Its autocorrelation and PSD are

RW ( ) =

N0 ( ) 2 N0 GW (f ) = 2

RW ( ) = 0 t = 0
Hence, any two samples separated in time are uncorrelated. However, a theoretical problem is that white noise has innite

average power.
Filtered white noise is called coloured noise.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Coloured Noise

Suppose we pass a Gaussian white noise with SD

N0 2

through an LTI

lter with transfer function H(f ).

Resulting output is

N0 |H(f )|2 2 N0 1 Ry ( ) = F [|H(f )|2 ] 2 Gy (f ) =

SD of ltered noise takes the shape of |H(f )|2 Hence, ltered white noise is called coloured noise.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Signal-to-Noise Ratio

Typically, the signal and noise components are added together in a

received communication signal.

In additive noise our received process Y (t) is

Y (t) = S(t) + N(t) S(t) is the noise-free signal (or process) and N(t) is the noise.
We usually assume that: the noise is ergodic and zero-mean, the noise is independent of the signal.

RV

PMF

Cont. RV

PDF

Operations

MRV

Joint Dist.

Process

Operations

Types

Quad lts

Noise

DC

DC

Signal-to-Noise Ratio

With E [|S(t)|2 ] = PS and E [|N(t)|]2 = PN we nd that

E [|Y (t)|2 ] = PS + PN
so the signal and noise power add also in the received signal. An important quantity is the signal-to-noise ratio (SNR):

SNR = Ps /PN , often quoted in dB.

A further typical assumption is that the noise is white and Gaussian,

hence Additive White Gaussian Noise (AWGN).