K.L.N.

COLLEGE OF ENGINEERING
DEPARTMENT OF ELECTRICAL AND ELECTRONICS ENGINEERING
Course/Branch
Duration
Semester
Regulation

: B.E / EEE
: Jan 2016-April 2016.
: VI
Section : A & B
: 2013

Subject
: COMMUNICATION ENGINEERING
Subject Code : EC 6651
Staff Handling : P.K.Arun Kumar

UNIT III SOURCE CODES, LINE CODES & ERROR CONTROL

(Qualitative only)
Primary communication entropy, properties, BSC, BEC, source coding : Shaum, Fao, Huffman
coding : noiseless coding theorum, BW SNR trade off codes: NRZ, RZ, AMI, HDBP, ABQ,
MBnB codes : Efficiency of transmissions, error control codes and applications: convolutions &
block codes.
3.1 Primary communication:

3.2 Entropy and information rate:

Consider an information source which has an alphabet of M different symbols. Let X denote the
symbol set (x1, x2, ..... xM). Each symbol occurs with probability p i and self information Ii. Note
M

that

p
i 1

1 . The amount of information output by the source per symbol depends on the

symbol and takes one of the values I1, I2, .... IM.
In a long string of N symbols the total information Itot is given approximately by :

Itot = Np1log(1/p1) + Np2log(1/p2) ........... + Npmlog2(1/pm)

The average info. per symbol (H(X)) is therefore :
M

i 1

i 1

H ( X ) pi .I i pi . log 2 (1 / pi )

bits/symbol

This is known as the source entropy. If all symbols are equally probable pi = 1/M :
M

H (X )
i 1

1
log M log M
M

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 1

If the source outputs n>>1 symbols the total information output is approximately n.H(X). If r is
the rate of symbol output then the average information rate output (R) of the source is :
R r .H ( X )

bits/sec

3.3 Communication Channels:

3.3.1 binary symmetric channel (BSC):

3.4 Source coding Theorem:

i)shannon-fano algorithm
ii)Huffman coding
3.4.1 Shannon-Fano Encoding:

Sources without memory are such sources of information, where the probability of the
next transmitted symbol (message) does not depend on the probability of the previous
transmitted symbol (message).
Separable codes are those codes for which the unique decipherability holds.
Shannon-Fano encoding constructs reasonably efficient separable binary codes for sources
without memory.
Shannon-Fano encoding is the first established and widely used encoding method. This
method and the corresponding code were invented simultaneously and independently of
each other by C. Shannon and R. Fano in 1948.
Let us have the ensemble of the original messages to be transmitted with their
corresponding probabilities:
X x1 , x2 ,..., xn ; P p1 , p2 ,..., pn
Our task is to associate a sequence Ck of binary numbers of unspecified length nk to each
message xk such that:
No sequences of employed binary numbers Ck can be obtained from each other by adding
more binary digits to the shorter sequence (prefix property).
The transmission of the encoded message is reasonably efficient, that is, 1 and 0 appear
independently and with almost equal probabilities. This ensures transmission of
almost 1 bit of information per digit of the encoded messages.
Another important general consideration, which was taken into account by C. Shannon
and R. Fano, is that (as we have already considered) a more frequent message has to be
encoded by a shorter encoding vector (word) and a less frequent message has to be
encoded by a longer encoding vector (word).

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 2

Shannon-Fano Encoding: Algorithm:

The letters (messages) of (over) the input alphabet must be arranged in order from most
probable to least probable.
Then the initial set of messages must be divided into two subsets whose total probabilities
are as close as possible to being equal. All symbols then have the first digits of their codes
assigned; symbols in the first set receive "0" and symbols in the second set receive "1".
The same process is repeated on those subsets, to determine successive digits of their
codes, as long as any sets with more than one member remain.
When a subset has been reduced to one symbol, this means the symbol's code is complete.
Shannon-Fano Encoding: Example:
Message
x1
x2

x3

x4

x5

x6

x7

x8

Probability 0.25 0.25 0.125 0.125 0.0625 0.0625 0.0625 0.0625

x1,x2,x3,x4,x5,x6,x7,x8
1

0
x1,x2
00
x1

x3,x4,x5,x6,x7,x8
01

11

10
x3,x4

x2
100

x5,x6,x7,x8

101

x3

110

x4

111

x5,x6

x7,x8
1101 1110

1100
x5

x6

x7

x8

Shannon-Fano Encoding: Properties:

The Shannon-Fano encoding is the most efficient when the probability of the occurrence of
each message (letter) xk is of the form
Pand
xk 2 nk
N

2 n1 2 n2 ... 2 nN 1
k 1
The prefix property always
holds and
N

nk

L P xk nk P xk log P xk
14 2 43
k 1
k 1
k)
the efficiency is 100%.
1 4 4 4 2 4 4I ( x43
H (X )

It should be taken into account that the Shannon-Fano code is not unique because it
depends on the partitioning of the input set of messages, which, in turn, is not unique.
If the successive equiprobable partitioning is not possible at all, the Shannon-Fano code
may not be an optimum code, that is, a code that leads to the lowest possible average
length of the encoding vector for a given D.

3.4.2 Huffman Encoding:

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 3

For an optimum encoding, the longer encoding vector (word) should correspond to a
message (letter) with lower probability:
P x1 P x2 ... P xN L x1 L x2 ... L xN

For an optimum encoding it is necessary that

L xN 1 L xN
otherwise the average length of the encoding vector will be unnecessarily increased.

It is important to mention that not more than D (D is the number of letters in the encoding
alphabet) encoding vectors could have equal length (for the binary encoding D=2)

For an optimum encoding with D=2 it is necessary that the last two encoding vectors are
identical except for the last digits.
For an optimum encoding it is necessary that each sequence of length digits either must
be used as an encoding vector or must have one of its prefixes used as an encoding vector.

Huffman Encoding: Algorithm:

The letters (messages) of (over) the input alphabet must be arranged in order from most
probable to least probable.
Two least probable messages (the last two messages) are merged into the composite
message with a probability equal to the sum of their probabilities. This new message must
be inserted into the sequence of the original messages instead of its parents, accordingly
with its probability.
The previous step must be repeated until the last remaining two messages will compose a
message, which will be the only member of the messages sequence.
The process may be utilized by constructing a binary tree the Huffman tree.
The Huffman tree should be constructed as follows:
1) A root of the tree is a message from the last step with the probability 1;
2) Its children are two messages that have composed the last message;
3) The step 2 must be repeated until all leafs of the tree will be obtained. These leafs are
the original messages.
The siblings-nodes from the same level are given the numbers 0 (left) and 1 (right).
The encoding vector for each message is obtained by passing a path from the roots child
to the leave corresponding to this message and reading the numbers of nodes (roots
childintermidiatesleaf) that compose the encoding vector.
Huffman Encoding: Example:
Let us construct the Huffman code for the following set of messages: x1, x2, x3, x4, x5 with
the probabilities p(x1)==p(x5)=0.2
1) x1 (p=0.2), x2 (p=0.2), x3 (p=0.2), x4 (p=0.2), x5 (p=0.2)
2) x4,x5x45 (p=0.4)=> x45,x1,x2,x3
3) x2,x3x23 (p=0.4)=>x45, x23, x1
4) x1,x23x123(p=0.6)=> x123, x45
5) x123, 45x12345 (p=1)

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 4

Encoding vectors: x1(00); x2(010); x3(011); x4(10); x5(11)

1
1
1
H ( X ) 5 0.2 log 0.2 5 log log log 5 2.32
Entropy
5
5
5
Average length of the encoding vector

12
1 1
L 3
2 2
3 2.4

5
5 5
The Huffman code gives (2.32/2.4)100% = 97% efficiency

Line Coding
Introduction:
Binary data can be transmitted using a number of different types of pulses. The choice of a
particular pair of pulses to represent the symbols 1 and 0 is called Line Coding and the choice is
generally made on the grounds of one or more of the following considerations:
Presence or absence of a DC level.
Power Spectral Density- particularly its value at 0 Hz.
Bandwidth.
BER performance (this particular aspect is not covered in this lecture).
Transparency (i.e. the property that any arbitrary symbol, or bit, pattern can be
transmitted and received).
Ease of clock signal recovery for symbol synchronisation.
Presence or absence of inherent error detection properties.
After line coding pulses may be filtered or otherwise shaped to further improve their properties:
for example, their spectral efficiency and/ or immunity to intersymbol interference.
Different Types of Line Coding:
Unipolar Signalling:
Unipolar signalling (also called on-off keying, OOK) is the type of line coding in which one
binary symbol (representing a 0 for example) is represented by the absence of a pulse (i.e. a
SPACE) and the other binary symbol (denoting a 1) is represented by the presence of a pulse (i.e.
a MARK).
There are two common variations of unipolar signalling: Non-Return to Zero (NRZ) and
Return to Zero (RZ).
Unipolar Non-Return to Zero (NRZ):
In unipolar NRZ the duration of the MARK pulse ( ) is equal to the duration (To) of the
symbol slot.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 5

In unipolar NRZ the duration of the MARK pulse ( ) is equal to the duration (To) of the
symbol slot.
Advantages:
Simplicity in implementation.
Doesnt require a lot of bandwidth for transmission.
Disadvantages:
Presence of DC level (indicated by spectral line at 0 Hz).
Contains low frequency components. Causes Signal Droop (explained later).
Does not have any error correction capability.
Does not posses any clocking component for ease of synchronisation.
Is not Transparent. Long string of zeros causes loss of synchronisation.
Return to Zero (RZ):
In unipolar RZ the duration of the MARK pulse ( ) is less than the duration (To) of the
symbol slot. Typically RZ pulses fill only the first half of the time slot, returning to zero for the
second half.

Advantages:
Simplicity in implementation.
Presence of a spectral line at symbol rate which can be used as symbol timing clock
signal.
Disadvantages:
Presence of DC level (indicated by spectral line at 0 Hz).
Continuous part is non-zero at 0 Hz. Causes Signal Droop.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 6

Does not have any error correction capability.

Occupies twice as much bandwidth as Unipolar NRZ.
Is not Transparent

Polar Signalling:
In polar signalling a binary 1 is represented by a pulse g1(t) and a binary 0 by the opposite
(or antipodal) pulse g0(t) = -g1(t). Polar signalling also has NRZ and RZ forms.

Fig. Polar NRZ

Fig. Polar RZ
Polar Non-Return to Zero (NRZ):
Advantages:
Simplicity in implementation.
No DC component.
Disadvantages:
Continuous part is non-zero at 0 Hz. Causes Signal Droop.
Does not have any error correction capability.
Does not posses any clocking component for ease of synchronisation.
Is not transparent.
Polar Return to Zero (RZ):
Advantages:
Simplicity in implementation.
No DC component.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 7

Disadvantages:
Continuous part is non-zero at 0 Hz. Causes Signal Droop.
Does not have any error correction capability.
Does not posses any clocking component for easy synchronisation. However, clock
can be extracted by rectifying the received signal.
Occupies twice as much bandwidth as Polar NRZ.
BiPolar Signalling:
Bipolar Signalling is also called alternate mark inversion (AMI) uses three voltage
levels (+V, 0, -V) to represent two binary symbols. Zeros, as in unipolar, are
represented by the absence of a pulse and ones (or marks) are represented by
alternating voltage levels of +V and V.
Alternating the mark level voltage ensures that the bipolar spectrum has a null at DC
And that signal droop on AC coupled lines is avoided.
The alternating mark voltage also gives bipolar signalling a single error detection
capability.
Like the Unipolar and Polar cases, Bipolar also has NRZ and RZ variations.

Fig. Polar NRZ

Fig. Polar RZ
BiPolar / AMI NRZ:
Advantages:
No DC component.
Occupies less bandwidth than unipolar and polar NRZ schemes.
Does not suffer from signal droop (suitable for transmission over AC coupled
lines).

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 8

Possesses single error detection capability.

Disadvantages:
Does not posses any clocking component for ease of synchronisation.
Is not Transparent.
BiPolar / AMI RZ:
Advantages:
No DC component.
Occupies less bandwidth than unipolar and polar RZ schemes.
Does not suffer from signal droop (suitable for transmission over AC coupled
lines).
Possesses single error detection capability.
Clock can be extracted by rectifying (a copy of) the received signal.
Disadvantages:
Is not Transparent.
HDBn Signalling:
HDBn is an enhancement of Bipolar Signalling. It overcomes the transparency
problem encountered in Bipolar signalling. In HDBn systems when the number of
continuous zeros exceeds n they are replaced by a special code.
The code recommended by the ITU-T for European PCM systems is HDB-3 (i.e. n=3).
In HDB-3 a string of 4 consecutive zeros are replaced by either 000V or B00V.
Where,
B conforms to the Alternate Mark Inversion Rule.
V is a violation of the Alternate Mark Inversion Rule
The reason for two different substitutions is to make consecutive Violation pulses
alternate in polarity to avoid introduction of a DC component.
The substitution is chosen according to the following rules:
1. If the number of nonzero pulses after the last substitution is odd, the substitution pattern
will be 000V.
1. If the number of nonzero pulses after the last substitution is even, the substitution pattern
will be B00V.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 9

HDBn RZ:
Advantages:
No DC component.
Occupies less bandwidth than unipolar and polar RZ schemes.
Does not suffer from signal droop (suitable for transmission over AC coupled
lines).
Possesses single error detection capability.
Clock can be extracted by rectifying (a copy of) the received signal.
Is Transparent.
Manchester Signalling:
In Manchester encoding , the duration of the bit is divided into two halves. The voltage
remains at one level during the first half and moves to the other level during the
second half.
A One is +ve in 1st half and -ve in 2nd half.
A Zero is -ve in 1st half and +ve in 2nd half.
Note: Some books use different conventions.

The transition at the centre of every bit interval is used for synchronization at the receiver.
Manchester encoding is called self-synchronizing. Synchronization at the receiving end can be
achieved by locking on to the the transitions, which indicate the middle of the bits.
It is worth highlighting that the traditional synchronization technique used for unipolar,
polar and bipolar schemes, which employs a narrow BPF to extract the clock signal cannot be
used for synchronization in Manchester encoding. This is because the PSD of Manchester

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 10

encoding does not include a spectral line/ impulse at symbol rate (1/To). Even rectification does
not help.
Error Control:
CONVOLUTION CODES:
The probability of error can be reduced by transmitting more bits than needed to represent the
information being sent, and convolving each bit with neighbouring bits so that if one transmitted
bit got corrupted, enough information is carried by the neighbouring bits to estimate what the
corrupted bit was. This approach of transforming a number of information bits into a larger
number of transmitted bits is called channel coding, and the particular approach of convolving
the bits to distribute the information is referred to as convolution coding. The ratio of
information bits to transmitted bits is the code rate (less than 1) and the number of information
bits over which the convolution takes place is the constraint length. For example, suppose you
channel encoded a message using a convolution code. Suppose you transmitted 2 bits for every
information bit (code rate=0.5) and used a constraint length of 3. Then the coder would send out
16 bits for every 8 bits of input, and each output pair would depend on the present and the past 2
input bits (constraint length =3). The output would come out at twice the input speed. Since
information about each input bit is spread out over 6 transmitted bits, one can usually
reconstruct the correct input even with several transmission errors. The need for coding is very
important in the use of cellular phones. In this case, the channel is the propagation of radio
waves between your cell phone and the base station. Just by turning your head while talking on
the phone, you could suddenly block out a large portion of the transmitted signal. If you tried to
keep your head still, a passing bus could change the pattern of bouncing radio waves arriving at
your phone so that they add destructively, again giving a poor signal. In both cases, the SNR
suddenly drops deeply and the bit error rate goes up dramatically. So the cellular environment is
extremely unreliable. If you didnt have lots of redundancy in the transmitted bits to boost
reliability, chances are that digital cell phones would not be the success they are today.
1.

CODE PARAMETERS AND THE STRUCTURE OF THE CONVOLUTIONAL CODE

The convolutional code structure is easy to draw from its parameters. First draw m boxes
representing the m memory registers. Then draw n modulo-2 adders to represent the n output
bits. Now connect the memory registers to the adders using the generator polynomial as shown in
the Fig. 1.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 11

(1,1,1)

v1

u1

u1

u0

u -1

(0,1,1)

v2

v3
(1,0,1)

Figure 1 - This (3,1,3) convolutional code has 3 memory registers, 1 input bit and 3 output bits.
This is a rate 1/3 code. Each input bit is coded into 3 output bits. The constraint length of the
code is 2. The 3 output bits are produced by the 3 modulo-2 adders by adding up certain bits in
the memory registers. The selection of which bits are to be added to produce the output bit is
called the generator polynomial (g) for that output bit. For example, the first output bit has a
generator polynomial of (1,1,1). The output bit 2 has a generator polynomial of (0,1,1) and the
third output bit has a polynomial of (1,0,1). The output bits just the sum of these bits.
v1 = mod2 (u1 + u0 + u-1)
v2 = mod2 ( u0 + u-1)
v3 = mod2 (u1 + u-1)
The polynomials give the code its unique error protection quality. One (3,1,4) code can have
completely different properties from an another one depending on the polynomials chosen.
Punctured Codes
For the special case of k = 1, the codes of rates , 1/3, , 1/5, 1/7 are sometimes called mother
codes. We can combine these single bit input codes to produce punctured codes which give us
code rates other than 1/n.
By using two rate codes together as shown in the figure, and then just not transmitting one of
the output bits we can convert this rate implementation into a 2/3 rate code. 2 bits come and 3
go out. This concept is called puncturing. On the receive side, dummy bits that do not affect the
decoding metric are inserted in the appropriate places before decoding.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 12

Fig. - Two (2,1,3) convolutional codes produce 4 output bits. Bit number 3 is punctured so the
combination is effectively a (3,2,3) code.
This technique allows us to produce codes of many different rates using just one simple
hardware. Although we can also directly construct a code of rate 2/3 as we shall see later, the
advantage of a punctured code is that the rates can be changed dynamically (through software)
depending on the channel condition such as rain, etc. A fixed implementation, although easier,
does not allow this flexibility.
BLOCK CODES
In coding theory, block codes are one of the two common types of channel codes (the other
one being convolutional codes), which enable reliable transmission of digital data over unreliable
communication channels subject to channel noise.
A block code transforms a message m consisting of a sequence of information symbols over
an alphabet into a fixed-length sequence c of n encoding symbols, called a code word. In a linear
block code, each input message has a fixed length of k < n input symbols. The redundancy added
to a message by transforming it into a larger code word enables a receiver to detect and correct
errors in a transmitted code word, and using a suitable decoding algorithm to recover the
original message. The redundancy is described in terms of its information rate, or more simply
for a linear block code in terms of its code rate, k/n.
The error correction performance of a block code is described by the minimum Hamming
distance d between each pair of code words, and is called the distance of the code.

EC 6651/ COMMUNICATION ENGINEERING/ UNIT-3/ P.K.Arun Kumar

Page 13