Faculty of Engineering

ET 351
Information Theory and Coding
Lecture 02
Unit I: Introduction
What is information theory: model of communication system, message encoding and decoding,
applications of information theory

Semester I, 2020|2021
23 November, 2020 ~ 19 February, 2021

26 December, 2020
What is information?
• By definition, information is certain knowledge about certain things,
which may or may not be conceived by an observer.
• Example: music, story, news, etc.
• Information means.
• Information propagates.
• Information corrupts…

 The Information within a source is the uncertainty of the

source.
2
 Example
• Every time we row a dice, we get points from 1 through 6. The information we get
is larger than we throw a coin or row an “infair” dice.

• The less we know, the more the information contained!

• If knowing for sure, nothing gains.

3
 What is Information Theory?
• The mathematical study of the coding of information in the form of
sequences of symbols, impulses, etc. and of how rapidly such information
can be transmitted, for example through computer circuits or
telecommunications channels.

• Information theory studies the quantification, storage, and communication

of information.

• It was originally proposed by Claude Shannon in 1948 to find fundamental

limits on signal processing and communication operations such as data
compression, in a landmark paper titled "A Mathematical Theory of
Communication". 4
 What is Information Theory?
• Applications of fundamental topics of information theory include
 lossless data compression (e.g. ZIP files), l
 lossy data compression (e.g. MP3s and JPEGs), and
 channel coding (e.g. for DSL).

• A key measure in information theory is "entropy".

• Entropy quantifies the amount of uncertainty involved in the value of a random variable or the
outcome of a random process.

• For example, identifying the outcome of a fair coin flip (with two equally likely outcomes) provides
less information (lower entropy) than specifying the outcome from a roll of a die (with six equally
likely outcomes).
5
 What is Information Theory?
• The field is at the intersection of mathematics, statistics,
computer science, physics, neurobiology, information
engineering, and electrical engineering.

• The theory has also found applications in other areas, including

statistical inference, natural language processing, cryptography,
neurobiology, human vision, the evolution and function of
molecular codes (bioinformatics), model selection in statistics,
thermal physics, quantum computing, linguistics, plagiarism
detection, pattern recognition, and anomaly detection 6
What is Information Theory?
• Important sub-fields of information theory include
source coding,
channel coding,
algorithmic complexity theory,
algorithmic information theory,
information-theoretic security, and
measures of information.
7
Motivation for the Study of Information Theory
Information theory tells us…
•What exactly information is
•How they are measured and presented
•Implications, limitations, and applications

8
Historical Notes

• Claude E. Shannon (1916-2001) himself

in 1948, has established almost
everything we will talk about today.

• He was dealing with communication

aspects.

• He first used the term “bit.”

9
Information Measure
:Behavioral Definitions

• H should be maximized when the object is most unknown.

• H(X)=0 if X is determined.

• The information measure H should be additive for independent objects;

i.e., with 2 information sources which has no relations with each other,
H=H1+H2.

• H is the information entropy!

10
Information Measure: Entropy

• Entropy H(X) of a random variable X is defined by

H(X)= -∑ p(x) log p(x)

• We can verify that the measure H(X) satisfies the three criterion
stated.

• If we choose the logarithm in base 2, then the entropy may be

claimed to be in the unit of bits; the use of the unit will be
clarified later.
11
Models of Communication System

12
Models of Communication System

13
Models of Communication System

14
Models of Communication System

15
Models of Communication System

16
Message Encoding and Decoding

17
Applications of Information Theory
Some Applications of Information Theory include: -
Biometric Pattern Recognition
Genetics
Neuroscience
Finance,
Game Theory,
Asset Price Volatility 18