9/5/2013

Information Theory and Coding

ECE533

Overview
Source Coding Techniques
Huffman Coding
Source Coding

jaj
Run Length
Lempel-ziv
Arithmetic
Shannon-Fano
Nikesh Bajaj
nikesh.14730@lpu.co.in Source Coding Theorem
Asst. Prof., ECE Dept.
Digital Signal Processing
JPEG: Image Compression
Lovely Professional University 2 By Nikesh Bajaj

Source Coding
Source Coding
Aim ??
Ba

Huffman Coding
David A. Huffman
August 9, 1925 October 7, 1999
Ohio State University, MIT
sh
X xi
1952 paper "A Method for the Construction of
Minimum-Redundancy Codes"
Minimum number of bits that completely
represent the symbol.

3 By Nikesh Bajaj 4 By Nikesh Bajaj

ke

Huffman Coding Huffman Coding

Ni

Algorithm

5 By Nikesh Bajaj 6 By Nikesh Bajaj

1
 9/5/2013

Huffman Algorithm Source coding

Observed Parameters
Entropy H

jaj
Average Length R
Efficiency of the code
Classification of Source coding
Fixed Length and Variable Length Coding
Lossless and Lossy Compression.
Prefix Code or Instantaneous Codes
7 By Nikesh Bajaj 8 By Nikesh Bajaj

Examples
Symbol

x1
x2
x3
Probability

0.37
0.33
0.16
Self-Information

1.4344
1.5995
2.6439
Code word

0
10
110
Ba Huffman Coding

VLC
Code is not unique
Efficiency near to 1
sh
x4 0.07 3.8365 1110
x5 0.04 4.6439 11110 Prefix Code
x6 0.02 5.6439 111110
Loss less Coding
x7 0.01 6.6439 111111

H(X)=2.1152
R=2.1700
N=H/R=2.1152/2.1700=0.9747

9 By Nikesh Bajaj 10 By Nikesh Bajaj

ke

Problems Problem
Ni

DAP for Huffman algorithm. Symbol Probability Self-Information Codeword

x1 0.5 1 1
Take a page of english charactors and x2 0.3 1.737 00
compress it using Huffman algorithm then x3 0.2 2.3219 01

check all the quantitive parameters i.e. H, R,

H(X)=1.4855, R =0.9903, =0.9903
efficiency.
Group two symbols, and calculate same

11 By Nikesh Bajaj 12 By Nikesh Bajaj

2
 9/5/2013

Run Length Coding Run length Coding

It is used when long sequence of Ones and Zeros FLC
comes in the signal
Loss less Coding

jaj

Example
Good for certain applications only
0 0 0 0 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 1
(7) (0) (3) (1) (6) (0) (0)
Or
11111111111111100000000000000000001111
(15,1), (19,0), (4,1)
(01111,1) , (10011,0), (00100,1)
13 By Nikesh Bajaj 14 By Nikesh Bajaj

Lemple-Ziv Algorithm (LZW) 1977
No need of symbol probability
Huffman is good for DMS, not for source with memory
No statistics require,
Ba

Lemple-Ziv Algorithm (LZW) 1977
Lets consider
101011011010101011
Dict.
Location
000
001
Dict.
Content

1
-
FL-
Codeword

0001
sh
010 0 0000
011 10 0010
100 11 0011
101 01 0101
110 101 0111
111 010 1010
Abraham Lempel Jacob Ziv Terry A. Welch
- 1011 1101

15 By Nikesh Bajaj 16 By Nikesh Bajaj

ke

Lemple-Ziv Algorithm Source Coding Theorem

Ni

.H ( X ) R H ( X ) 1

Proof:

17 By Nikesh Bajaj 18 By Nikesh Bajaj

3
 9/5/2013

Arithmetic Coding
+
P(A) = 0.5 P(B) = P(C) = 0.25

jaj
Msg = BACA

19 By Nikesh Bajaj 20 By Nikesh Bajaj

Problems: Encoding

P(1) = P(0) =
Msg = 110101011010101
Ba Problem: Decoding

P(A) =0.5 P(B) =0.2 P(C) =0.3
Code =0.21625
sh
21 By Nikesh Bajaj 22 By Nikesh Bajaj
ke

Image Compression
Image: dpi 4x4 image
Ni

Redundancy?
Spatial Correlation
Spectral Correlation
Temporal Correlation

By: Nikesh Bajaj

23 By Nikesh Bajaj 24

4
 9/5/2013

RGB to YCrCb RGB image

jaj
25 By: Nikesh Bajaj

Ba YIQ or YUV color models

(Examples)

original
Y
sh
U V

27 By: Nikesh Bajaj

ke

Examples Comparison Example-1

Ni

Y : Luminance

Cb: Chrominance

Cr: Chrominance

5
 9/5/2013

Comparison Example-2
Transform Coding
Aim of Transforming
To create a representation for the data in which

jaj
there is less correlation among the coefficient
value
To have a representation in which it is possible
to quantize different coordinates with different
precision

By: Nikesh Bajaj

32

Transform Coding Ba

JPEG: Still Image Compression
Standard
JPEG: Joint Photographic Experts Group
Formally: ISO/IEC JTC1/SC29/WG1

International
organization for
Working Group 1
(JBIG,JPEG)
Joint ISO/IEC
sh
Standardization
Technical Sub-committee 29
International Committee (Coding of Audio,
Electro-technical (Information Picture, Multimedia
Commission Technology) and Hypermedia
information
Work commenced in mid-1980s.
Draft international standard 1991.
Widely used for image exchange, WWW, and digital photography.
ke

Image Compression Standard JPEG:Encoder and Decoder

Ni

By: Nikesh Bajaj

35

6
 9/5/2013

JPEG: Image Partitioning JPEG:Basic Algorithm

jaj
JPEG: Quantization Table Ba JPEG: Differential coding of DC
sh
ke

Entropy Coding for AC Comp. Image Compression

Run Length Introduction to image compression
Ni

Huffman DCT Transform
N 1 M 1
k l
y(k , l ) 4 I (i, j ) cos (2i 1) cos (2 j 1)
i 0 j 0 2N 2M
JPEG standard
For Lossless Compression
For Lossy Compression

41 By Nikesh Bajaj 42 By Nikesh Bajaj

7
 9/5/2013

DCT of an Image DCT

jaj
43 By: Nikesh Bajaj 44 By: Nikesh Bajaj

DCT of an Image

DCT
50

100
Ba 250

200

150
JPEG: Example
sh
100
150

50

200

45 By: Nikesh Bajaj

ke

JPEG: Example JPEG: Example

Ni

8
 9/5/2013

JPEG: Decoding
JPEG: Example
The DC coefficient is DPCM coded (difference
between the DC coefficient of the previous block
and current block)

jaj
The AC coefficients are mapped to run-length
pairs: (run,value)
(0,5),(0, -3),(0, -1),(0,- 2),(0, -3),(0,1),
(0,1),(0, -1),(0, -1),(2,1),(0,2),(0,3),(0, -2),
(0,1),(0,1),(6,1),(0,1),(1,1), EOB
These are then Huffman coded (codes are
specified in the JPEG scheme)

JPEG: Decoding Ba JPEG: Original Vs reconstructed

Image
sh
ke

JPEG:
Example Image Compression
Ni

By: Nikesh Bajaj

54

9
 9/5/2013

JPEG ITU T.81 Standard Shannon Fano-Elias Coding

Different coding Pdf
8x8 block Cdf

jaj
artifacts

By: Nikesh Bajaj

55 56 By Nikesh Bajaj

Shannon Fano-Elias Coding Ba Shannon Fano-Elias Coding

Sym Prob F(x) F(x) F(x)bin l(x) codeword Sym Prob F(x) F(x) F(x)bin l(x) codeword
sh
x1 x1 0.5 0.25 0.01 2 01
x2 ^2 x2 ^2 0.75 0.625 0.101 3 101
x3 ^3 x3 ^3 0.875 0.8125 0.1101 4 1101
x4 ^ 4 x4 ^ 3 1 0.9375 0.1111 4 1111

57 By Nikesh Bajaj 58 By Nikesh Bajaj

ke
Ni

10