How Computer Works

Introduction
Acquiring Saliency Data

RoI-based Error Resilience
Conclusion
Region-of-Interest Intra Prediction for

H.264/AVC Error Resilience
Fadi Boulos, Wei Chen, Benoît Parrein and Patrick Le Callet
IRCCyN lab., University of Nantes, France
fadi.boulos@univ-nantes.fr
November 10, 2009
ICIP 2009 Fadi Boulos RoI Intra Prediction for H.264/AVC Error Resilience
Introduction
Conclusion
Motivations
Packet loss affects QoE of video services over IP.
Introduction
Conclusion
Motivations
Spatial position of the loss in the picture influences perceptual

quality1 .
1
F. Boulos, D. S. Hands, B. Parrein and P. Le Callet. Perceptual Effects of Packet Loss on
H.264/AVC Encoded Videos. In 4th International Workshop on Video Processing and Quality
Metrics for Consumer Electronics, Scottsdale, AZ, January 2009.
Introduction
Conclusion
Motivations
Spatial position of the loss in the picture influences perceptual

quality1 .
Spatio-temporal error propagation heavily distorts video

quality.
1
F. Boulos, D. S. Hands, B. Parrein and P. Le Callet. Perceptual Effects of Packet Loss on
H.264/AVC Encoded Videos. In 4th International Workshop on Video Processing and Quality
Metrics for Consumer Electronics, Scottsdale, AZ, January 2009.
Introduction
Conclusion
Error Propagation
Intra-prediction from
Lost slice
inter-predicted MB
Inter-prediction
Prediction order > 1
from lost MB
Introduction
Conclusion
Error Propagation
Intra-prediction from
Lost slice
inter-predicted MB
Inter-prediction
Prediction order > 1
from lost MB
Introduction
Conclusion
Objectives
1 Study the impact of packet loss in the RoI.

2 Build an attention-based error resilience model against packet
loss.
Determine the RoI which robustness should be enhanced.

Counter the effect of packet loss, namely spatio-temporal
error propagation.
Introduction
Conclusion
Outline
1 Acquiring Saliency Data

Test Setup
From Eye Tracking to Regions of Interest
2 RoI-based Error Resilience

Does RoI Matter?
Error Resilience
3 Conclusion
Introduction
Acquiring Saliency Data Test Setup
RoI-based Error Resilience From Eye Tracking to Regions of Interest
Conclusion
Eye Tracking
Track and record eye movement of observers while watching

video sequences.
collected by the eyetracker,

ation has the same weight.
The second way is base
spatial location. The salien
given by:
NF
(k)
!
SMF D (x, y) =
j=
where NF P and ∆ have th

duration.
To determine the most v
Fig. 1. Eye tracking apparatus maps are merged yielding t
erage saliency map is given
Provide reliable information about viewer’s visual attention.

2.2. Subjects
SM (x, y)
Twenty unpaid subjects participated to the experiments. All had nor-
mal or corrected to normal vision. All were inexperienced observers
where K is the number of o
(in video processing) and naive to the experiment. Before each trial,
Finally, the average sal
the subject’s head was positioned so that their chin rested on the
ICIP 2009 Fadi Boulos sian filter given a density sa
RoI Intra Prediction for H.264/AVC Error Resilience
Introduction
Conclusion
Test Setup
Test performed in a standardized environment according to

ITU Recommendation BT.500-112 .
Video set: SD and HD sequences, 8s or 10s duration.
JM 14.0 codec, High Profile, IDRBBP... GOP structure of
length 24.
Encoding bitrates chosen such as to obtain good video quality.
37 subjects participated in the test.
2
ITU-R. Methodology for the subjective assessment of the quality of television pictures, June
2002.
Introduction
Conclusion
Some Definitions
Fixation: the status of a region centered around a pixel

position which was stared at for a predefined duration.
Introduction
Conclusion
Some Definitions

Saccade: eye movement from one fixation to another.
Introduction
Conclusion
Some Definitions

Pursuit: eye movement allowing gaze to follow a moving

target.
Introduction
Conclusion
Some Definitions

Pursuit: eye movement allowing gaze to follow a moving

target.
Saliency map: spatial locations of the eye gaze over time.
Introduction
Conclusion
From Eye Tracking to Saliency Maps
Introduction
Conclusion
Introduction
Conclusion
Introduction
Conclusion
From Saliency Maps to Regions of Interest
The encoder needs macroblock level information.
Introduction
Conclusion
From Saliency Maps to Regions of Interest
The encoder needs macroblock level information.
A pixel belongs to an RoI if its saliency value is higher than a

threshold (set empirically).
Introduction
Acquiring Saliency Data Does RoI Matter?
RoI-based Error Resilience Error Resilience
Conclusion
RoI-based FMO
Group RoI macroblocks into one or more slices independently

of other slices in the picture using FMO.
FMO type 6 provides full control over the assignment of
macroblocks to slices.
Different RoI per picture => 1 PPS NALU per picture in the
bitstream.
2 3
7 Slices 1-6: non-RoI.
3
4 4 Slices 7 & 8: RoI.
8
5
Introduction
Conclusion
Loss Simulation
Loss simulator3 operating on a NALU (slice) basis.

One slice fits in one packet.
Slice losses in I-pictures to test the robustness of the model
against error propagation.
JM spatial error concealment: weighted sample averaging.
3
Joint Video Team of ISO/IEC MPEG and ITU-T VCEG. SVC/AVC Loss Simulator. JVT-Q069, October 2005.
Available at http://wftp3.itu.int/av-arch/jvt-site/2005_10_Nice/
Introduction
Conclusion
Loss Patterns
RoI lost. 8 slices around the RoI lost.
Introduction
Conclusion
Subjective Quality Test
The most reliable way to assess video quality.

25 non-expert subjects rate the quality of video sequences.
Absolute Category Rating (ACR) with a 5-level scale.
Introduction
Conclusion
Perceptual Importance of RoI
Introduction
Conclusion
RoI Intra-Prediction
Slice lost in a reference picture allows error drift to

inter-coded pictures.
Attenuate the temporal error drift in the RoI by coding all RoI
macroblocks of P and B-pictures in intra-prediction mode.
At constant bitrate, the model might incur a quality decrease.
I I I I
I I I I I
I I I I I
I I I I
B or P-picture
Introduction
Conclusion
RoI Slices Lost
Introduction
Conclusion
Overhead of RoI Intra-Prediction
Introduction
Conclusion
Example
w/o error resilience. w/ error resilience.
Introduction
Conclusion
Conclusion and Future Work
We proposed an error resilient coding based on ground truth

visual attention data.
We created a database of video sequences with visual
attention data (publically available soon).
Introduction
Conclusion
Conclusion and Future Work
We proposed an error resilient coding based on ground truth

visual attention data.
We created a database of video sequences with visual
attention data (publically available soon).
Use an objective saliency model.

Investigate the tradeoff between RoI size and quality
degradation/robustness.
Introduction
Conclusion
Questions...

How Computer Works

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

How Computer Works

Hochgeladen von

Copyright:

Verfügbare Formate

Introduction

Acquiring Saliency Data

Region-of-Interest Intra Prediction for

Fadi Boulos, Wei Chen, Benoît Parrein and Patrick Le Callet

IRCCyN lab., University of Nantes, France

November 10, 2009

Packet loss affects QoE of video services over IP.

Packet loss affects QoE of video services over IP.

Spatial position of the loss in the picture influences perceptual

Packet loss affects QoE of video services over IP.

Spatial position of the loss in the picture influences perceptual

Spatio-temporal error propagation heavily distorts video

1 Study the impact of packet loss in the RoI.

Determine the RoI which robustness should be enhanced.

1 Acquiring Saliency Data

2 RoI-based Error Resilience

Track and record eye movement of observers while watching

collected by the eyetracker,

where NF P and ∆ have th

Provide reliable information about viewer’s visual attention.

Test performed in a standardized environment according to

Fixation: the status of a region centered around a pixel

Fixation: the status of a region centered around a pixel

Saccade: eye movement from one fixation to another.

Fixation: the status of a region centered around a pixel

Saccade: eye movement from one fixation to another.

Pursuit: eye movement allowing gaze to follow a moving

Fixation: the status of a region centered around a pixel

Saccade: eye movement from one fixation to another.

Pursuit: eye movement allowing gaze to follow a moving

Saliency map: spatial locations of the eye gaze over time.

From Eye Tracking to Saliency Maps

From Eye Tracking to Saliency Maps

From Eye Tracking to Saliency Maps

From Saliency Maps to Regions of Interest

The encoder needs macroblock level information.

From Saliency Maps to Regions of Interest

The encoder needs macroblock level information.

A pixel belongs to an RoI if its saliency value is higher than a

Group RoI macroblocks into one or more slices independently

Loss simulator3 operating on a NALU (slice) basis.

RoI lost. 8 slices around the RoI lost.

Subjective Quality Test

The most reliable way to assess video quality.

Perceptual Importance of RoI

Slice lost in a reference picture allows error drift to

RoI Slices Lost

Overhead of RoI Intra-Prediction

w/o error resilience. w/ error resilience.

Conclusion and Future Work

We proposed an error resilient coding based on ground truth

Conclusion and Future Work

We proposed an error resilient coding based on ground truth

Use an objective saliency model.

Das könnte Ihnen auch gefallen