Beruflich Dokumente
Kultur Dokumente
and Entropy
CS414 Spring 2007
By Roger Cheng
(Huffman coding slides courtesy
of Brian Bailey)
Why study noise?
Its present in all
systems of interest,
and we have to deal
with it
By knowing its
characteristics, we
can fight it better
Create models to
evaluate analytically
Communication system
abstraction
Information
source
Encoder Modulator
Channel
Output signal Decoder Demodulator
Sender side
Receiver side
The additive noise channel
Transmitted signal s(t)
is corrupted by noise
source n(t), and the
resulting received signal
is r(t)
Noise could result form
many sources, including
electronic components
and transmission
interference
n(t)
+ s(t) r(t)
Random processes
A random variable is the result of a single
measurement
A random process is a indexed collection of
random variables, or equivalently a non-
deterministic signal that can be described by
a probability distribution
Noise can be modeled as a random process
WGN (White Gaussian Noise)
Properties
At each time instant t = t
0
, the value of
n(t) is normally distributed with mean 0,
variance
2
(ie E[n(t
0
)] = 0, E[n(t
0
)
2
] =
2
)
At any two different time instants, the
values of n(t) are uncorrelated
(ie E[n(t
0
)n(t
k
)] = 0)
The power spectral density of n(t) has
equal power in all frequency bands
WGN continued
When an additive noise channel has a white
Gaussian noise source, we call it an AWGN
channel
Most frequently used model in
communications
Reasons why we use this model
Its easy to understand and compute
It applies to a broad class of physical channels
Signal energy and power
Energy is defined as
Power is defined as
Most signals are either finite energy and zero
power, or infinite energy and finite power
Noise power is hard to compute in time domain
Power of WGN is its variance
2
2
= | ( ) | dt x x t
/ 2
2
/ 2
1
= lim | ( ) |
T
x
T
T
P x t dt
T
N
i
i i s p s p H
1
) ( log ) ( 2
Symbol P(S) Code
A 0.25 11
B 0.30 00
C 0.12 010
D 0.15 011
E 0.18 10
Average Codeword Length
= .25(2) +
.30(2) +
.12(3) +
.15(3) +
.18(2)
L = 2.27 bits
N
i
i i s codelength s p L
1
) ( ) (
Symbol P(S) Code
A 0.25 11
B 0.30 00
C 0.12 010
D 0.15 011
E 0.18 10
Code Length Relative to
Entropy
Huffman reaches entropy limit when all
probabilities are negative powers of 2
i.e., 1/2; 1/4; 1/8; 1/16; etc.
H <= Code Length <= H + 1
N
i
i i s p s p H
1
) ( log ) ( 2
N
i
i i s codelength s p L
1
) ( ) (
Example
H = -.01*log
2
.01 +
-.99*log
2
.99
= .08
L = .01(1) +
.99(1)
= 1
Symbol P(S) Code
A 0.01 1
B 0.99 0
Limitations
Diverges from lower limit when probability of
a particular symbol becomes high
always uses an integral number of bits
Must send code book with the data
lowers overall efficiency
Must determine frequency distribution
must remain stable over the data set