A Project Report on

A performance study of decoding algorithms for LPDC

codes on a Binary Erasure Channel
Submitted to the Department of Computer Science and Technology

For the partial fulfilment of the degree of B.E. in

Computer Science and Technology
by
Rahul Chowdhury
Roll number: 111205007
Registration number: 110512007 of 2012-13
B.E., 4th year
and
Shreyasee De
Roll number: 111205027
Registration number: 110512027 of 2012-13
B.E., 4th year
Under the supervision of
Abhik Mukherjee
(Associate Professor)

Department of Computer Science and Technology

INDIAN INSTITUTE OF ENGINEERING SCIENCE AND
TECHNOLOGY, SHIBPUR
May, 2016

CERTIFICATE
This is to certify that the work presented in this report entitled A performance study of
decoding algorithms for LPDC codes on a Binary Erasure Channel, submitted by Rahul
Chowdhury (Examination Roll : 111205007) and Shreyasee De (Examination Roll : 111205027),
has been carried out under my supervision for the partial fulfilment of the degree of Bachelor
of Technology in Computer Science and Technology during the session 2015-16 in the
Department of Computer Science and Technology, Indian Institute of Engineering Science and
Technology, Shibpur.

(Prof. Abhik Mukherjee)

Associate Professor
Department of Computer Science and
Technology
Indian Institute of Engineering Science
and Technology, Shibpur

(Prof. Amit Kumar Das)

Dean (Academic)
Indian Institute of Engineering Science
and Technology, Shibpur

(Prof. Biplab Kumar Sikdar)

Head of the Department
Department of Computer Science and
Technology
Indian Institute of Engineering Science
and Technology, Shibpur

Date: 9th May, 2016

Acknowledgements
We would like to take this opportunity to express my sincere gratitude for Dr. Abhik Mukherjee,
Associate Professor, Department of Computer Science and Technology, Indian Institute of Engineering
Science and Technology, Shibpur, for his invaluable advice, resourceful guidance, inspiring
instructions and constant encouragement without which it would not have been possible to
perform this project. Also, we would like to express my gratitude towards all those who helped us
in completing this project, directly or indirectly.
We would also like to thank all the faculty members for their cooperation during the preparation
of this work.

Date: 9 th May, 2016

(Rahul Chowdhury)
Department of Computer Science and Technology
Indian Institute of Engineering Science
and Technology, Shibpur

(Shreyasee De)
Department of Computer Science and Technology
Indian Institute of Engineering Science
and Technology, Shibpur

Abstract
Abundance of low-cost storage media as well as deep space communication is resulting in renewed
interest in codes suitable for Erasure channels. In storage media, an entire sector may get damaged,
leading to erasure data. In deep space communication, there are so many erratic and unpredictable
sources of noise that can erase communication for a an entire span of time.These phenomenon can be
directly mapped to an erasure channel, and the recovery of information in these situations is related to
decoding for an erasure channel.
We shall show the performance of various decoding algorithms for recovery of erased bits. In this
project, we have considered three algorithms :

A brute force guessing algorithm

An improvement on the brute force guessing algorithm
A message passing algorithm

For each of the algorithms, we shall compare their performance, both in terms of speed of decoding and
correctness of decoding. We want to particularly explore the performance of decoders for Random
erasures and Burst erasures.
At this moment, LDPC codes give performance near Shannon Limit for AWGN channel. Thus, to
achieve better performance, we are studying LDPC codes for Binary Erasure Channels (BEC).

Contents
1.
2.

3.

4.

5.
6.

Introduction...6
Preliminaries and Definitions.6
2.1. Binary Erasure Channel...6
2.2. Classification of Erasure Channels.7
2.3. Applications of Decoding Erasure Codes..7
2.4. Channel Capacity of a Binary Erasure Channel...7
2.5. Low density parity check (LDPC) codes....10
2.6. Using LDPC codes for Binary Erasure Channels.10
2.7. Iterative Decoding using Hard-decision Message Passing.11
Implementation of decoders for LDPC codes on a Binary
Erasure Channel..11
3.1. Algorithm A : Steps...12
3.2. Algorithm A : Illustration...12
3.3. Algorithm B : Steps...13
3.4. Algorithm B : Illustration...13
3.5. Algorithm C : Steps...15
3.6. Algorithm C : Illustration...15
Results and Observations..16
4.1. Decoder A v/s Decoder C16
4.2. Decoding time v/s Codeword length..17
4.3. Decoding correctness comparison.18
4.4. Random Erasures v/s Burst Erasures...19
4.4.1. Decoding time v/s Erasure probability for Decoder C..19
4.4.2. Decoding time v/s Codeword length for Decoder C.19
4.4.3. Decoding correctness v/s Erasure probability for Decoder C.20
Conclusion...21
References..22

1. Introduction
A general communication system can be broken into three parts
(i)
Source - Sends information
(ii)
Channel
(iii)
Destination

Source

Channel

Encoder

Destination
Decoder

Source symbols are mapped into sequence of channel symbols, which then produces the output
sequence of the channel. By mapping the source into the appropriate widely spaced input
sequences to the channel, we can transmit a message with very low probability of error and
reconstruct the source message at the output.
A channel is described by a set of transition probabilities. Maximum rate at which this can be done is
called the capacity of the channel. The channel is said to be memoryless if the probability distribution
of the output depends only on the input at that time and is conditionally independent of previous inputs
or outputs.

2. Preliminaries and Definitions

2.1. Binary Erasure Channel
An erasure channel is a common communications channel model used in coding theory and
information theory. In this model, a transmitter sends a bit (a zero or a one), and the receiver either
receives the bit or it receives a message that the bit was not received ("erased").
The binary erasure channel is a binary channel; that is, it can transmit only one of two symbols
(usually called 0 and 1). (A non-binary channel would be capable of transmitting more than two
symbols, possibly even an infinite number of choices.) The channel is not perfect and sometimes the
bit gets "erased"; that is, the bit gets scrambled so the receiver has no idea what the bit was. If the bit
is lost retransmit it until it gets through.
The BEC is, in a sense, error-free. Unlike the binary symmetric channel, when the receiver gets a bit,
it is 100% certain that the bit is correct. The only confusion arises when the bit is erased.
This channel is often used by theorists because it is one of the simplest noisy channels to analyze.
Many problems in communication theory can be reduced to a BEC.
6

Here the probability of the bit getting through is 1-pe.

2.2. Classification of Erasure Channels

Random erasure channel

1011011011

1?11?1?011

Any random number of bits from any random position may get erased.

Burst erasure channel

1011011011

101???1011

A contiguous chunk of bits (of any length) gets erased.

2.3. Applications of Decoding Erasure Codes

The erasure channel is a very useful idealization of situations where the symbols observed by the receiver
have either very high or very low reliability.
Storage media (recovery of bad sectors).
Deep space communication (noise is so high/irregular that calculations for the channel using SNR
cannot be used).
Intermittent failure (due to loose connections).

Jamming

packet-switched store-and-forward networks

magnetic recording

2.4. Channel capacity of a Binary Erasure Channel

Channel capacity is the upper bound on the rate at which information can be reliably transmitted over a
communication channel. In the noisy channel, the channel capacity is the limiting information rate that
can be achieved with arbitrarily small error probability.
The capacity of a discrete memoryless channel is defined as the maximum average mutual
information in any single use of the channel where the maximum is taken over all possible input
probabilities distributions p(x).

Capacity of the channel is defined as the maximum mutual information between the input and output of
the channel.
C = max I(X ; Y) = max [H(Y) H(Y|X)]
p(x)

p(x)

Where X and Y are the input and output symbol respectively and mutual information is defined as the
amount of information transferred, when Y is received and X is transmitted and the information about Y
from X:
I( Y;X) = H(Y) - H(Y/X)
Where H(Y) is the entropy of Y and H(Y/X) is the conditional entropy.
Now for binary erasure channel with erasure probability e,

The binary erasure channel is an extension of the binary symmetric channel, in the sense that there
is one more output symbol, $. If the received signal is such that it is very likely to be correct received
it should be the corresponding binary symbol, while if it is more unsure it should be the symbol $.
The graphical representation of the channel is shown above. The probability distribution for the
erasure channel would be:

The mutual information between X and Y can be written as

I(X; Y ) = H(Y ) H(Y |X) = H(Y ) X x H(Y |X = x)P(X = x)
= H(Y ) h(e)
8

Since H(Y |X = x) = H(1 e, e, 0) = h(e), both when x = 0 and x = 1. Instead of maximizing I(X; Y ) we
can maximize H(Y ), for varying p = P(X = 1).
To derive H(Y ) we first need to get the distribution of Y , expressed in terms of the distribution of X.
Assume that P(X = 1) = p, then with p(x, y) = p(x)p(y|x) and p(y) =p(x)p(x, y) we can get the
distribution of Y as

Hence the entropy of Y can be derived H(Y ) = H((1 p)(1 e), e, p(1 e)). Naturally, this function can
be optimized by solving for p the derivative equal to zero. But, we can also go ahead and see
H(Y ) = (1 p)(1 e) log(1 p)(1 e) e log e p(1 e) log p(1 e)
= (1 e) p log p (1 e) log(1 p) e log e (1 e) log(1 e)
= (1 e)h(p) + h(e)
Hence the capacity is
C = max H(Y ) h(e) = max (1 e)h(p) = 1 e
The capacity for the binary erasure channel is
plotted as a function of the erasure probability e. If
the e = 0 we get the capacity equal to one, since
then we do not have any errors. On the other
hand, if e = 1 all of our received symbols will be
erasures and we do not have any information
about the transmitted sequence.

2.5. Low density parity check (LDPC) codes

Discovered by Robert Gallager in early 1960.

In information theory, a low-density parity-check (LDPC) code is a linear error correcting code,
a method of transmitting a message over a noisy transmission channel.

Although LDPC codes can be defined over any finite field, the majority of research is focused
on LDPC codes over GF(2)

An LDPC code is defined with respect to its parity check matrix. The term low density
signifies a sparse parity check matrix in terms of number of 1s in the matrix.

Due to its sparseness LDPC code has a great advantages in terms of efficient decoder
implementation and storage.

LDPC codes are capacity-approaching codes, which means that practical constructions exist
that allow the noise threshold to be set very close (or even close on the BEC) to the theoretical
maximum (the Shannon limit) for a symmetric memoryless channel.

Low Density Parity Check (LDPC) codes have superior error performance (4 dB coding gain
over convolutional codes).

2.6.

Using LDPC codes for binary erasure channels

LDPC codes are often used in binary erasure channels for the following reasons :

Long codes can contrast long block fading.

The memoryless erasure channel capacity can be asymptotically achieved by LDPC codes under
iterative message passing decoding.
The message passing decoder is very efficient.
The traditional problem concerning the encoding of LDPC codes has been overcome, as several
solutions have been proposed which permit to perform the encoding operation with a complexity
linearly increasing with the codeword length, while preserving good or very good decoding
performance.

10

2.7. Iterative Decoding using Hard-decision Message passing

The parity check matrix of a code is converted into a bipartite graph, called a Tanner graph which
consists of :

message nodes (ci) - represents the bits of the codeword

check nodes (fi) - represents the parity check equations

In the hard-decision message passing algorithm,messages are passed between the message
nodes and the check nodes.
Several iterations of this message passing cycle continues until the decoding algorithm converges to
a valid codeword, which is indicated when all check nodes have a value 0.

3. Implementation of decoders for LDPC codes on a

Binary Erasure Channel
We have implemented three decoding algorithms for LDPC codes over the BEC. Let us call these
algorithms as Algorithm A, Algorithm B and Algorithm C.
Algorithm A - A brute force algorithm, which making guesses for all erased bits.
Algorithm B - An improvement over the brute force algorithm, using guesses to recover some bits and
linear equation solving for others.
Algorithm C - A hard-decision message passing algorithm.

11

3.1.

ALGORITHM A : STEPS

Input : RECEIVED_CODEWORD
Output : a) Decoding success/failure b) DECODED_CODEWORD
1.
2.

Set CODEWORD=RECEIVED_CODEWORD.
A. Find index of first erased bit in the CODEWORD. Name it as ERASED_INDEX.
B. Set XERASED_INDEX ={0,1}
3.
Guess value CODEWORD[ERASED_INDEX]=x, where xXERASED_INDEX .If XERASED_INDEX is
empty, go to
Step 2.
4.
Check if there are any more erased bits in the CODEWORD.
5.
If there are more erased bits, go to step 2.
6.
If there are no more erased bits, calculate SYNDROME of codeword.
7.
If SYNDROME=0, set DECODED_CODEWORD=CODEWORD. Return TRUE.
8.
If SYNDROME 0, XERASED_INDEX = XERASED_INDEX - x. Go to step 3.

3.2.

ALGORITHM A : ILLUSTRATION

Input : Codeword = 1 0 0 1 ? 0 ? 0

12

3.3.

ALGORITHM B : STEPS

Input :
Output :
1.
2.
3.
4.
5.
6.
7.
8.
9.

3.4.

a) RECEIVED_CODEWORD
a) Decoding success/failure

b) PARITY_CHECK_EQUATIONS
b)DECODED_CODEWORD

Set CODEWORD=RECEIVED_CODEWORD.
A. Find index of first erased bit in the CODEWORD. Name it as ERASED_INDEX.
B. Set XERASED_INDEX ={0,1}
Guess value CODEWORD[ERASED_INDEX]=x, where xXERASED_INDEX .If XERASED_INDEX is
empty, go to step 2.
Check all EQUATION PARITY_CHECK_EQUATIONS.
If EQUATION has 1 unknown variable Cz , set CODEWORD[Z] = XOR YEQUATION, Y CZ{Y}
Repeat step 5 until there is no EQUATION remaining with 1 unknown variable.
If all EQUATION have 0 unknown variables, check syndrome of CODEWORD
If SYNDROME=0, set DECODED_CODEWORD=CODEWORD. Return TRUE.
If SYNDROME 0, XERASED_INDEX = XERASED_INDEX - x. Go to step 3.

ALGORITHM B : ILLUSTRATION

Input :

1) Codeword C = 0 1 1 ? ? 0 1 ?
2) Parity check matrix

Parity Check Matrix

Corresponding Tanner Graph

13

Equations for each of the check nodes :

i. c1 + c3 + c4 + c7 = 0
ii. c2+ c3 + c4 + c7 = 0
iii. c0 + c2 + c5 + c 6 = 0
iv. c 0 + c1 + c5 + c 6 = 0
Received Codeword C= 0 1 1 ? ? 0 1 ?
1. Guess value of c3 =0. So, C = 0 1 1 0 ? 0 1 ?
2. The check node equations become :
i. 1 + 0 + c4 + c7 = 0
ii. 1 + 0 +c4 + c7 = 0
iii. 0 + 1 + 0 + 1 = 0
iv. 0 + 1 + 0 + 1 = 0
Since there are no more nodes with 1 variables, guess another bit.
3. Guess c4 =0. So, C = 0 1 1 0 0 0 1 ?
The check node equations become :
i. 1 + 0 + 0 + c7 = 0
ii. 1 + 0 + 0 + c7 = 0
iii. 0 + 1 + 0 + 1 = 0
iv. 0 + 1 + 0 + 1 = 0
4. So, c7 =1, from i and ii. C = 0 1 1 0 0 0 1 1.
5. Calculate the syndrome for codeword C. We find syndrome=0.
Thus, C = 0 1 1 0 0 0 1 1 is the decoded codeword.

14

3.5. ALGORITHM C : STEPS

Input :
Output :
1.
2.
3.
4.
5.
6.
7.

a) RECEIVED_CODEWORD
b) TANNER_GRAPH
a) Decoding success/failure b)DECODED_CODEWORD

Set CODEWORD=RECEIVED_CODEWORD.
For all message nodes Ci and check nodes Fj , Ci sends CODEWORD[i] to Fj if
TANNER_GGRAPH[i][j]=1.
Check all CHECK_NODEi TANNER_GRAPH.
If for all i, CHECK_NODEi has no unknown variables, set
DECODED_CODEWORD=CODEWORD. Return TRUE.
If CHECK_NODEi has 1 unknown variable Cj ,
set CODEWORD[j] = XOR YCHECK_NODEi, Y Cj{Y}
If the state of no CHECK_NODEi changes in one iteration, return FALSE.
Go to step 2.

3.6. ALGORITHM C : ILLUSTRATION

1.
2.

Received codeword C = 1 ? ? 1 0 0 0 0
Message nodes send values to their connected check nodes.
fo =
f1 =
f2 =
f3 =

c1 + 1 + 0 + 0
c2 + 1 + 0 + 0
1 + c2 + 0 + 0
1+ c1 + 0 + 0

?
3.

Calculate c1 = 1 + 0 +0 = 1

4.

Set C = 1 1 ? 1 0 0 0

for check node f0

1
5.

Calculate c2 = 1 + 0 + 0 = 1

6.

Set C = 1 1 1 1 0 0 0

for check node f2

0

0
T

7.

Syndrome S = C.H = 0. So, C is a valid codeword.

8.

Return C.

0
0

15

4.
4.1

Results and Observations

Decoder A v/s Decoder C

GRAPH I

GRAPH II
We see that the average decoding time of Decoder A grows exponentially with increase in erasure
probability, but that of decoder C grows polynomially. But, it is seen from Graph II that Decoder C gives
decoding failure when the erasure probability is increased, while decoder A does not.
Thus, the high decoding time of Decoder A is compensated by the number of further retransmissions and
decoding that Decoder C has to perform, due to decoding failure.
Let us define the following terms :
f -> Average time taken by decoder C for decoding failure
r -> Average retransmission time for a codeword
s -> Average time taken by decoder C for decoding success
a -> Average time taken by decoder A for decoding success
n -> number of retransmissions done by decoder C
Nmax -> Maximum number of retransmissions allowed, for which the overall time taken by decoder C will be
less than or equal to that of decoder A.
16

Decoding
DECODER C : failure 1

Retransmission
1

DECODER A :

...

Decoding
failure n

Retransmission
n

Decoding
success

Decoding success

Thus, the following inequality must be satisfied : n*(f + r) +s <= a

or,

n <= (a-s)/(f+r)

Nmax = (a-s)/(f+r)

We can see that Nmax increase exponentially with Erasure probability.

4.2.

Decoding time v/s Codeword length

17

4.3.

Decoding correctness comparison

Correct Decoding
The brute force decoder gives the best performance for correctness among the 3 decoders. It gives
quite a reasonable performance (40% - 100% correct decoding) in the critical region (0.2 - 0.4
erasure probability).

Incorrect Decoding
The message passing decoder does not give any incorrect results, while the other two decoders give
incorrect results, the rate of which increases with erasure probability.

18

4.4.

Random v/s Burst Erasures

4.4.1.

Decoding time v/s Erasure probability for Decoder C

Critical region

Performance in the region where erasure probability varies from 0.2 to 0.4 is critical for the decoder. Thus,
we call this region as a critical region for erasure decoding.
We can see from the graph that in the critical region, the average time taken to decode random erasure
codes is more than that for burst erasure codes for the message passing decoder (decoder C).
Hence, the decoder C gives very good performance for burst erasures.
4.4.2.

Decoding time v/s Codeword length for Decoder C

The message passing decoder gives better decoding time performance for random erasures upto
codeword length 15. As codeword length increases further, the decoder takes lower average time
for burst erasures.
19

4.4.3. Decoding correctness v/s Erasure probability for Decoder C

Critical region

Critical region

The message passing decoder gives better decoding performance (less failure percentage, more
correctness) for random erasures over burst erasures, in the critical region.

20

5. Conclusion

We arrived at the following conclusions after studying the performance of various decoding algorithms
for LDPC codes for the Binary Erasure Channel

Though the brute force decoding algorithm has a exponential time complexity compared to the
polynomial time complexity of the message passing decoder, the high decoding failure rate of
the message passing decoder warrants a number of retransmissions to correctly decode a
codeword. The maximum number of retransmissions allowed Nmax also increases exponentially
with increase in erasure probability.
The message passing decoding algorithm, when run for both random and burst erasure
channels gives

Lower average decoding time for burst erasure codes, in the critical region.

For codeword length <=15, lower average decoding time for random erasure codes. As,
codeword length increases further, the average decoding time is lesser for burst erasure
codes.

Better decoding performance for random erasures over burst erasure, in the critical
region.
Since the parity check matrix of a LDPC code is sparse, it becomes difficult to recover the
codeword for burst erasures. In case of random erasures, a reasonable distribution of edges
lets the decoder solve for the unknown erased bits. But unless the edges are uniformly
distributed, so that there are adequate edges connected between the region of burst and the
correct region, it becomes difficult to solve the equations in case of burst erasure.

21

6. References

R. G. Gallager, Low Density Parity Check Codes, IEEE Trans. Information Theory,, IT-8: pp. 2128, January 1962.
R. G. Gallager, Low Density Parity Check Codes, MIT Press, Cambridge, 1963.
William E. Ryan, An Introduction to LDPC Codes,The University of Arizona,August 2003.
Hossein Pishro-Nik, Faramarz Fekri, On Decoding of Low-Density Parity-Check Codes Over the
Binary Erasure Channel, IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 50, NO. 3,
MARCH 2004
Michael G. Luby, Michael Mitzenmacher, M. Amin Shokrollahi, and Daniel A. Spielman, Efficient
Erasure Correcting Codes, IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 47, NO. 2,
FEBRUARY 2001
Sample LPDC codes - http://www.inference.phy.cam.ac.uk/mackay/CodesFiles.html

22