Outline

• Introduction
• Lossless compression
• Lossy compression
– metrics
– general methods
• scalar
• vector
• differential
• transform (JPEG)
• subband (wavelet)
– MPEG video
143
 RGB Color Model for CRT Displays

R,G,B integer triplets encode how much the corresponding phosphor

should be excited in devices such as a CRT display.

144
 RGB vs CMY

• CMY are complementary colors of RGB

• CMY mostly used in printing devices where color pigments on
the paper absorb certain colors (eg, no red light reflected from
cyan ink) 145
 Video Coding
• MPEG - Motion Pictures Expert Group, 1988
– asymmetric application
• single compression; frequent decompression
• compression is complex!!
– standards consist of 3 parts:
up to 200:1 compression

5:1 to 10:1 compression

– similar to IUT-T H.261 videoconferencing standard

Thanks to Prof. Leonardo Chiariglione for MPEG slide material.

146
Video Flow : Delay vs. Jitter

delay

jitter

147
 Categories of media
• One-way pre-recorded media
– e.g., downloading a movie from comcast/blockbuster
– can be transmitted as fast as the network allows (assuming
adequate buffering at the receiver) -- can be viewed as a special
file transfer operation.
• One-way live media
– e.g., live news report from scene of an accident
– e.g., live seminar (where questions are not permitted)
– live media can not be transmitted faster than the rate it is
generated and encoded.
– delays from transmit at the sender to playout at the receiver of
several or even tens of seconds are acceptable
• Two-way interactive media
– e.g., phone conversation (voice over IP)
– e.g., teleconference
– sensitive to delay - applications delays from transmit to playout
of as little as 150 to 200 ms are often problematic

148
 MPEG Standards
Video: 320 X 240 * 24b/p * 30 frames/s = 55.3Mb/s
Audio: 2 tracks * 44K samples/s * 16b/sample = 1.4Mb/s = 787MB/75min

• MPEG – 1 (1992)
– video on CD-ROM --1.5 Mbit/sec (intermediate rate)
• MPEG – 2 (1996)
– HDTV, DVD (Digital Video/Versatile Disc) -- 5-10 Mbit/sec
– MPEG - 3 (intended for HDTV - combined with MPEG-2)

• MPEG – 4 (1999)
– originally multimedia over phone lines -- 64Kbit/sec
• MPEG – 7 (2001)
– multimedia content description interface (video databases)

149
 MPEG Data Representation Pyramid

Objects
features Semantic-based MPEG-7
extraction representation

Object-based
Objects representation MPEG-4
formation
and tracking

Pixel-based MPEG-1
representation MPEG-2

150
 Motivations for MPEG
• key applications
– original design for storing/retrieving video/audio on
disk-based media (e.g., CD-ROM)
– broadcast of digital video (limited electro spectrum)
– switched digital video (asymmetric digital subscriber lines)
– HDTV
– networked multimedia (packet-switched QoS)

R (required) Y Y = .299R + .587G + .114B

Cb
G Cb = B - Y
B Cr
Cr = R - Y
luminance chrominance

151
 MPEG-1 (ISO/IEC 11172)

• Title: “Coding of moving pictures and associated audio

for digital storage media at up to about 1.5 Mbit/s”
– started - May ’88,
– International Standard - November ‘92
• Main goal
– storing interactive movies on CDs (~1.4 Mbit/s)
• 352 x 240 @ 30 frames/sec
• video quality equivalent to VHS
• CD-quality stereo audio

152
 MPEG-1: a sequence of “firsts”
• First integrated audio-visual standard
– ISO/IEC 11172-1 (Systems), -2 (Video), -3 (Audio)
• First audio-visual standard defining the “receiver”
and not the “transmitter”
• First video coding standard independent of video
format (NTSC / PAL / SECAM)
• First standard jointly developed by all industries
interested in audio and video
• First standard developed entirely in software
• First standard to include a software implementation

153
 MPEG-1 - an assessment
• Video CD (several 10’s million players sold in China)
• “The” format of audio and video for PC
– Windows 95/NT/98 contain MPEG-1 software decoder
• MPEG-1 Audio (layer 3) widely used for Web music
• Digital Audio Broadcasting (DAB) uses MPEG-1
Audio (adopted in Europe and Canada)
• Lightweight MPEG-1 video cameras are on sale

154
 MPEG-1 (cont’d)

• needed features not found in H.261 - Teleconferencing

– arbitrary start - random access (eg, for interactive games)
– fast forward / reverse
• key compression idea: temporal prediction

frame i frame i+1

155
 MPEG Motion Compensation -
Search Algorithms
• temporal dependency

156
MPEG Motion Compensation -
Searching

157
 MPEG Motion Compensation -
Searching

• target block size

• larger blocks are harder to match
• larger blocks, if matched well, compress more

• search area size

• larger areas require more computation to search
• larger areas permit better matches; hence more compression

• motion vector (x,y)

• (15,4) - 15 pixels right, 4 pixels up
• can be in units of ½ pixel

158
MPEG Data Hierarchy

8X8
16 X 16
pixels
pixels

159
 MPEG Motion Compensation
• Matching Criteria - Metrics to measure differences between blocks
• mean absolute difference

• mean square difference

• PDC - pel difference classification

160
 MPEG Motion Compensation -
Search Algorithms
• spatial dependency

motion
vectors

Camera panning - spatial or temporal dependency?

161
 MPEG Motion Compensation -
Search Algorithms
• “coarse quantization” or “distance-diluted”
hypothesis: fast moving objects look blurred to HVS even if sharp in all frames

search
area

= center of
potential matching
block

162
 MPEG Motion Compensation -
Search Algorithms
• “locality-based” or “multi-level hierarchical”
hypothesis: once a close match is found, even better ones are nearby

163
 MPEG Block-Based Motion Compensation
GOP - Group of Pictures

1 2 3 4 5 6 7 8 9 10 11 12 13
I B B P B B P B B P B B I

I frame - Independent encoding

P frame - Predictive encoding based on most recent past I or P

B frame - Bidirectional encoding based on past and future I or P

( Each frame is encoded as a series of 16 x 16 pixel macroblocks )

164
 Inter-frame (P-frame) Coding

frame i+1

frame i

165
B frame encoding

166
 GOP – Group of Pictures

P
B
B
P
7
B
6
B
5
I
4
3
2
1

167
 Display vs Transmission Order

1 2 3 4 5 6 7 8 9 10 11 12 13
I B B P B B P B B P B B I

display order

1 4 2 3 7 5 6 10 8 9 13 11 12
I P B B P B B P B B I B B

transmission order

168
 MPEG Decode-ability
MPEG is uniquely decodable
MPEG is NOT uniquely codable
MPEG
• fastest encoding
file • no motion compensation
• less expensive coder
• least compression

MPEG
file

...
MPEG
file

video sequence
...

• slowest encoding
(frames)
• advanced motion compensation
MPEG
file
• more expensive coder
• most compression 169
 MPEG 1 Performance

Type Size (KB) Compression

------------------------------------------------
I 18 7:1
P 6 20:1
B 2.5 50:1

Avg 4.8 27:1

170
 Digital Video Content Analysis:
Automated Scene Change Detection

• if scene changes, no need to look for

interframe correlation

Slides thanks to Dr. Stephan Fischer

German National Research Center for Information Technology
171
 Video Transition - Cut
• Sudden change of image content between continuous
frames

Time

172
 Video Transition - Fade
• fade out
– image content changes to monochrome color

time

• fade in
– changes from monochrome color to image

173
 Video Transition - Dissolve
• one image morphs (overlays) to another

time

174
ASIDE: Automated Cut Detection -
Using Histograms

Cut 175
 Automated Cut Detection -
Using Histograms - Problems
Different images can have same histograms
• simple

• complex example

176
 MPEG-2 (ISO/IEC 13818)
• Title: “Generic coding of moving pictures and audio”
• History
– started - July ‘90
– International Standard - November ‘94
• Main goal - migration of TV from analog to digital

177
 MPEG-2 - an assessment
• Several 10’s of million set top boxes for satellite and cable sold
• Digital TV VHF/UHF broadcasting
• DVD players in U.S. households (2002)
– 26 million (USA Today)
– 38 million (NPDTechworld)
– 56 million (Adams Media Research)
– 17 million shipped in 2003
• MPEG-2 4:2:2 profile is being adopted by the TV production
industry
• MPEG-2 has created the entirely new digital TV industry
worth ~$?? billion

178
 MPEG-2
• designed to be generic, application independent
• "toolkit" approach
– 5 profiles (i.e., algorithms) each with levels (i.e., constraints)
• simple - no B-frames
• main - MPEG-1
• snr - scalable
• spatially scalable use scalable, layered bitstreams
• high

coefficients
original : 29.75 6.10 -6.03 1.93 -2.01 1.23 -0.95 2.11
layer 1
quantized : 28 8. -8. 0. -4. 0. 0. 4.

error : 1.75 -1.90 1.97 1.93 1.99 1.23 -0.95 -1.89

layer 2 quantized : 2 -2 2 2 2 2 0 -2

error : -0.25 0.1 -0.03 -0.07 -0.01 -0.77 -0.95 0.11

179
 MPEG-2 Levels

level frame size (at 30 frames/s)

low 352 x 240 (SIF)
main 720 x 480 (CCIR 601)
high 1440 1440 x 1152
high 1920 x 1080 (digital HDTV)
w/o compression 884 Mbits/s
w/ compression ~20 Mbits/s

180
ITU-R Digital Terrestrial Television
Broadcasting Model RF/Transmission
System

channel
MPEG-2 video M transport coding
video source coding U (packetizing)
and compression L
T modulation
AC3 - audio I
source coding P
audio
and compression
L
ancillary data E
(closed captions, software) X
O
control data R
HDTV
181
ATSC Digital Television
Standard (Annex A) 16 Sep 95

182
 MPEG-4
• original target: very low bit-rate 4.8 - 64Kb/s
• now aims at: video 5Kb - 10Mb/s audio 2Kb - 64Kb/s

MPEG-4 terminal (receiver side)

183
 MPEG 4 (cont’d)

• Good for video composition, segmentation, and

compression; networked VRML, audiovisual
communication systems, e.g., text-to-speech
interface, facial animation
• Standards being developed for
– shape coding
– motion coding
– texture coding
– etc.

184
 MPEG-4: Scaling
• Spatial scalability
– decoder displays textures and visual objects at a reduced spatial
resolution
• by decoding only a subset of the total bit stream
– 32 levels max. for textures and still images
– 3 levels max. for video sequences
• Temporal scalability
– decoder displays video at a reduced temporal resolution
• by decoding only a subset of the total bit stream
– 3 levels max.

185
MPEG-4: Example of Composition

186
 Origin of MPEG “7”
Work was begun on an MPEG-3 standard for high-definition
television, but it became clear that the tools needed were very
similar to those in MPEG-2, so MPEG-3 was quickly
abandoned, and HDTV support was included in MPEG-2.
When the latest work item was started, the first question taken up
was what number to use. One participant recalled that the
conversation was something like, "Shall the number for the next
job be 5, which follows 4, or should it be 8, attractive in its own
binary way, to follow 1, 2 and 4?
After some thought, MPEG members decided that their new work
item was so different from what had gone before that they threw
both ideas overboard and chose 7 as the lucky number.”

187
 MPEG-7

• Adds description to objects

– this movie segment contains “airplanes attacking
spaceships from outer space”
• Permits fast searching in databases
– e.g., “find all movie segments containing
Presidents Clinton and Chirac shaking hands”

188
 Summary
lossless lossy
• statistical (Huffman) • scalar
• dictionary (LZW) • vector
• run length (fax) • differential
• arithmetic • transform (JPEG)
• subband
• HAAR
• wavelet
• MPEG

www.cis.udel.edu/~amer/CISC651/651.html

189