Elements of Information Theory

kfchen@hznu.edu.cn

http://pan.baidu.com/s/1kT1X4Jd
ftp://kfchen:public@public.sjtu.edu.cn

&

1-1613:20-15:50 A-201

(15%)+(35%)+(50%)

http://pan.baidu.com/s/1kT1X4Jd
ftp://kfchen:public@public.sjtu.edu.cn

2014
, ,
4, ,
2008

2014
T. M. Cover and J. A. Thomas,
Elements of Information Theory,
, ,
2012
R. Bose, Information Theory,
Coding and Cryptography,
,
, 2005


1
2

3
4
5

7
8
9
9

--

(
)
(
)

--

10

xi()
1
I ( xi ) log
log p( xi )
p ( xi )

x1

p( x1 )

1
5

I ( x1 ) log

1
5

x2

p( x2 )

4
5

I ( x2 ) log

4
5

H ( X ) p( xi ) log p( xi )
i 1

1
1 4
4
H ( X ) log log
5
5 5
5

RC

:
()

A, B, C, D4
: A 00, B 01, C 10, D 11
AAAABBCD 0000000001011011
16

()

4p(A)=1/2, p(B)=1/4, p(C)=

p(D)= 1/8
A 0, B 10, C 110, D 111
AAAABBCD 00001010110111
14

()

:
Projec Apollo

1961-1972

1969.7.2111

11

0 000; 1
111
Hamming - (7,4)

:
Dancing Men

(Elsie)


()

E

E

EE
, sever, lever, never,
V

Elsie
LSI
4 come C, O, M

: AM HERE
AE SLANE ABE
SLANEY

()

M ERE E SLNE
: A ELRIES
TGAT
ELRIGES

COME ELSIE

: ELSIE REARE TO
MEET THY GOPD
ELSIE PREPARE TO
MEET THY GOD

ELRIE

NEVER

ELSIE REARE O MEE H O

or Key


1948C. ShannonA
Mathematical Theory of Communication

Communication Theory of Secrecy

System

1916 - 2001

/
one-time pad


1.

2.


3.

4.

5.

k
xkxy

1.

()

2.

()

3.


()

*
1938Shannon
(IEEE)Alfred Noble20

1940ShannonAn Algebra for Theoretical

Genetics
1948Shannon
20

32

*Shannon2009
http://www.zgkjcx.com/Article/UploadFiles/201104/20110427102847354.doc

1949Shannon

1945
A Mathematical Theory of Cryptograph
Boston
1949ShannonProgramming a computer for playing
chess,1965

Mikhail Botvinnik
1980Shannon
Linzjuxingde
Belle1997IBM
chatterbot ALICE
Garry Kasparov

1950ShannonTheseus1951
Presentation of a maze solving machine

1953Shannon(Mind reading)

ShannonD. W.
Hagelbarger
1956J.

1959Coding theorems for a discrete source with

fidelity criterion

1960Two-way communication channel

Claude Elwood Shannon

Father of Information Theory
Electrical engineer, Mathematician,
and native son of Gaylord. His creation
of information theory, the mathematical
theory of communication, in the 1940s
and 1950s inspired the revolutionary
advances in digital communications and
information storage that have shaped
the modern world.
This statue was donated by the
Information Theory Society of the
Institute of Electrical and Electronics
Engineers, whose members follow
gratefully in his footsteps.
Dedicated October 6, 2000
Edgene Daub, Sculptor

ShannonIEEE Information Society252000106Claude Shannon

MichiganGaylordShannon

http://202.120.38.173/information-theory.htm

Thanks!