EE4-07 Introduction to Coding Theory

Dr. Wei Dai

Imperial College London (IC)
Autumn 2011
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 1
Section 4 Hamming Codes and Performance
The Hamming code: denition and properties
The dual of the Hamming code
Several bounds in coding theory
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 2
Binary Hamming Codes
The 1[7, 4, 3] Binary Hamming code is a linear code with
H =
_
_
0 0 0 1 1 1 1
0 1 1 0 0 1 1
1 0 1 0 1 0 1
_
_
1 2 3 4 5 6 7
(contains all the nonzero columns of length 3)
Let r 2. The parity-check matrix of the binary Hamming code
1[2
r
1, 2
r
1 r, 3] consists of all the nonzero vectors (columns) of
length r. (Also denoted by 1
r
.)
d = 3 because every 2 columns are linearly independent and
there are 3 columns that are linearly dependent.
Correct exactly 1 error.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 3
Decoding a Binary Hamming Code
Syndrome decoding:
Arrange columns of H in the alphabetically increasing order
1
Compute the syndrome s = yH
T
.
2
The syndrome s gives the location of error.
3
Flip the bit located at s. x
1
= y
1
, , x
s
= y
s
+ 1, , x
n
= y
n
.
Easy to decode.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 4
Coding Bounds
Given n and d, what is the maximum r or k or M?
Given n and k, what is the maximum d?
Sphere packing (Hamming) bound
Sphere covering (Gilbert-Varshamov) bound
Improved Gilbert-Varshamov bound for linear codes
Plotkin bound
Singleton bound
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 5
Hamming Bound and Perfect Codes
Spheres in F
n
q
: B (x, t) =
_
y F
n
q
: d (x, y) t
_
.
Volume: Vol (B (x, t)) =

t
i=0
_
n
i
_
(q 1)
i
.
Remark: Volume does not depend on the center. Denote it by V (t)
Hamming bound (sphere-packing bound): For a code of length n and
distance d, the number of codewords satises
M q
n
/V
__
d1
2
__
= q
n
/
_

d1
2
|
i=0
_
n
i
_
(q 1)
i
_
.
Proof: Let t =
_
d1
2
_
. Let ( = c
1
, , c
M
. Clearly, B (c
i
, t)s are
disjoint, 1 i M. Hence, M V (B (e)) q
n
.
Perfect codes: the codes that attain the Hamming bound.
Hamming code is a perfect code.
Perfect codes are rare (Hamming codes & Golay codes).
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 6
Gilbert-Varshamov Bound
Gilbert-Varshamov (sphere covering) bound: For given code length n
and distance d, there exists a code with number of codewords
q
n
/V (d 1) M.
Proof: Its proved by construction. Let M
0
= ,q
n
/V (d 1)| > 1.
Take an arbitrary c
1
F
n
q
. Since F
n
q
B (c
1
, d 1) ,= , take arbitrary
c
2
F
n
q
B (c
1
, d 1). Clearly, d (c
1
, c
2
) d.
Inductively, suppose that we have obtained codewords c
1
, , c
M
0
1
in this way.
Since Vol
_

M
0
1
i=1
B (c
i
, d 1)
_
(M
0
1) V (d 1) < q
n
,
F
n
q

M
0
1
i=1
B (c
i
, d 1) ,= . Take arbitrary
c
M
0
F
n
q

M
0
1
i=1
B (c
i
, d 1) ,= . Clearly, d (c
M
0
, c
i
) d as
c
M
0
/ B (c
i
, d 1) for all 1 i M
0
1.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 7
Illustration for Sphere Packing and Covering
Sphere Packing Sphere Covering
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 8
Improved Gilbert-Varshamov Bound for Linear Codes
Improved G-V bound for linear codes: For given n, d, k such that

d2
i=0
_
n1
i
_
(q 1)
i
< q
nk
, there exists an [n, k] linear code over F
q
with minimum distance at least d.
Proof: We shall show that an (n k) n matrix H such that every
d 1 columns of H are linearly independent. We construct H as
follows.
Let h
1
be any nonzero vector in F
nk
q
. Let h
2
be any vector not in
span(c
1
). Let h
3
be any vector not in span([c
1
, c
2
]). .
Inductively, suppose that we have constructed H

of size
(n k) (n 1) such that every d 1 columns of H

are linearly
independent. Note that the number of vectors in the linear span of
d 2 or fewer columns of H

is given by

d2
i=0
_
n1
i
_
(q 1)
i
. Since

d2
i=0
_
n1
i
_
(q 1)
i
< q
nk
, h F
nk
q
such that every d 1 columns
of the resulting H = [H

, h] are linearly independent.

Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 9
Plotkin Bound
Plotkin bound: For a code of length n. Suppose that rn < d where
r = 1 q
1
. Then the number of codewords M
_
d
drn
_
.
(M
_
2d
2dn
_
when q = 2.)
Proof: 1) Let T =

cC

C
d (c, c

). Since d d (c, c

) for c ,= c

, we
have M (M 1) d T.
2) Let A be the M n matrix whose rows are the codewords. Let n
i,a
denote the number of entries in the i
th
column of A that are equal to a.
Then

aF
n
i,a
= M for all i. We have T =

n
i=1
(

d (c
i
, c

i
)) =

n
i=1

aF
n
i,a
(M n
i,a
) = M
2
n

n
i=1

aF
n
2
i,a
.
3) By Cauchy-Schwarz inequality,
_

aF
n
i,a
_
2
q

aF
n
2
i,a
. Hence,
T M
2
n

n
i=1
q
1
_

aF
n
i,a
_
2
= rM
2
n. Therefore,
M
2
(d rn) Md.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 10
Cauchy-Schwarz Inequality and Proof
Cauchy-Schwarz Inequality: For real numbers x
1
, , x
n
and
y
1
, , y
n
,
(

m
i=1
x
i
y
i
)
2
(

m
i=1
x
i
)
2
(

m
i=1
y
i
)
2
.
Proof: It hold if either x or y is 0.
Assume that both x and y are nonzero. Dene z = rx +y.
Then 0 |z|
2
2
= r
2
|x|
2
2
+ 2r x, y + |y|
2
2
for all r R.
This implies b
2
4ac = 4 x, y
2
4 |x|
2
2
|y|
2
2
0, i.e.,
x, y
2
|x|
2
2
|y|
2
2
.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 11
Dual of Binary Hamming Codes
(1
r
)

= o
r
_
2
r
1, r, (n + 1) /2 = 2
r1

is called simplex code.

A very low rate code with very large distance.
Proof of d (o
r
) = 2
r1
: by induction.
Simplex codes attain Plotkin bound: M = 2
r
;
2d
2dn
=
2
r
2
r
(2
r
1)
= 2
r
.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 12
Singleton Bound and MDS
Theorem (Singleton Bound):
This distance of any code ( F
n
q
with [([ = M satises
M q
nd+1
.
In particular, if the code is linear and M = q
k
, then
d n k + 1.
Proof: For a code with length n and distance d, take arbitrary c
i
,= c
j
.
Let c
i,1:nd+1
be the rst n d + 1 entries of c
i
and c
i,nd+2:n
be the
last d 1 entries of c
i
. Then
d d
H
(c
i
, c
j
) = d
H
(c
i,1:nd+1
, c
j,1:nd+1
) + d
H
(c
i,nd+2:n
, c
j,nd+2:n
) .
But d
H
(c
i,nd+2:n
, c
j,nd+2:n
) d 1. Hence,
d
H
(c
i,1:nd+1
, c
j,1:nd+1
) 1, i.e., the rst n d + 1 coordinates of all
codewords are distinct. Hence, M q
nd+1
.
Codes meet singleton bound are maximum distance separable (MDS).
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 13
Properties of MDS codes
Properties of MDS codes:Let ( [n, k, d] be a linear code over F
q
. Let H
and G be the corresponding parity-check and generator matrices.
Then the following statements are equivalent
1
( is an MDS code.
2
Every set of n k columns of H is linear independent.
3
Every set of k of G is linear independent.
4
(

is an MDS code.
Proof: It is clear that 12 and 34 as G is the parity-check matrix of
(

. We prove that 14. Once we proved that, it is clear that 41.

To show (

is MDS, it sufces to show that the minimum distance d

is
k + 1.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 14
Proof of Properties of MDS codes
Suppose that d

k. Then a nonzero codeword c (

with at most
k nonzero entries and at least n k zeros. Since permuting the
coordinates reserves the codeword weights (i.e., the distance), w.l.o.g.,
assume that the last n k coordinates of c are 0.
Write H = [A, H

] where A F
(nk)k
q
and H

F
(nk)(nk)
q
. From
assumption 1 (and hence 2), the columns of H

are linear
independent. Hence, H

is invertible. The only way to obtain 0 in all

the last n k coordinates of c = sH is s = 0. It contradicts with the
assumption c ,= 0. Hence, d

k + 1. Together with the Singleton

bound, it follows that d

= k + 1. Hence, 14.
Hamming codes are not MDS in general.
Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 15
Summary
Examples of linear codes

Hamming codes

Simplex codes
Coding Bounds

Sphere packing and covering bounds

Plotkin bound

Singleton bound and MDS

Dr. Wei Dai (Imperial College) Introduction to Coding Theory Autumn 2011 16