The MPEG-7 Standard

Why do we need MPEG-7 ?

Support for Advanced Query Fast & Accurate Access Personalized Content Production and Consumption Content Management Automation
2

Need

Visual Audio Sketch

MPEG: A Brief History (1)

MPEG: Moving Picture Experts Group

Established in 1988

ISO / IEC/JTC1/SC29/WG11

A Working Group of ISO/IEC in charge of the Development of Standards for Coded Representation of Digital Audio and Video

MPEG: A Brief History (2)

MPEG-1: Interactive CD and MP3 MPEG-2: DTV, STB, DVD MPEG-4: Web and Mobility

11 / 1992 11 / 1994 ver1: 09 /1998

ver2: 11 /1999
MPEG-7: ???

08 / 2001 11 / 2001

MPEG-21: Multimedia Framework

MPEG-7: What Is It ?
THE MPEG 7 STANDARD

IS NOT a STANDARD for FEATURE EXTRACTION/MATCHING

IS NOT a COMPRESSION Standard similar to MPEG-1/2/4 or their Extension

Content Description of Various Audio Visual Information

Types of Audio Visual Information Audio, Speech Moving video, still pictures, graphics Information on how objects are combined in scenes
5

MPEG-7: Application Areas

Storage and retrieval of audiovisual databases (image, film, radio
archives)

Broadcast media selection (radio, TV programs) Surveillance (traffic control, surface transportation, production chains) E-commerce and Tele-shopping (searching for clothes / patterns) Remote sensing (cartography, ecology, natural resources management) Entertainment (searching for a game, for a karaoke) Cultural services (museums, art galleries) Journalism (searching for events, persons) Personalized news service on Internet (push media filtering) Intelligent multimedia presentations Educational applications Bio-medical applications

MPEG-7 Description Scope for AV Content Description Granularity

Low-level
High-level

Form Access Classification Link Context

7

MPEG-7: Main Elements

Descriptors (D)
syntax and semantics of each feature representation

Description Schemes (DS)

structure and semantics of the relationships between
components

Description Definition Language (DDL)

creation of new DSs modification/extension of existing DSs
8

MPEG-7: Major Functionalities

Systems

(ISO / IEC 15938 - 1)

Description Definition Language (ISO / IEC 15938 - 2) Visual Audio (ISO / IEC 15938 - 3) (ISO / IEC 15938 - 4)

Multimedia Description Schemes (ISO / IEC 15938 - 5) Reference Software (ISO / IEC 15938 - 6)
9

MPEG-7: Main Elements (2)

DDL

Instantiation DS1
<Object> <Label/> <Definition/> . . </Object>

DS2

DS2

Systems DS3 D1 D2 D3 0001100

10

MPEG-7: Systems It defines tools to:

provide for efficient storage and transport

synchronize between content and description

manage and protect intellectual property

11

MPEG-7: DDL and its Components

Description Definition Language:
Creation of the Ds and DSs: XML Schema & MPEG-7 Extensions Instantiation of XML

XML Schema:
Data types
Simple and Complex types Elements, attributes

Inheritance, Abstract types

MPEG-7 extensions:
Array and Matrix data type
12

MPEG-7: Audio

Sound Effects Music Instrument Timbre Spoken Content

Melody Contour

13

MPEG-7: Visual (1)

Color
quantization, dominant, scalable, color-structure, layout, GoF/GoP

Texture

Shape
region-based, contour-based, 3D

Motion
camera motion, motion trajectory, parametric motion, motion activity
14

MPEG-7: Visual (2)

Localization
spatio temporal

Others
face recognition

15

MPEG-7: Basic Visual Structures

Grid Layout 2D-3D Multiple View Time Series

Spatial 2D Coordinates
Temporal Interpolation

16

Low level Audio Visual descriptors

Video segments
Color Camera motion Motion activity Mosaic

Still regions
Color Shape Position Texture

Moving regions
Color Motion trajectory Parametric motion Spatio-temporal shape

Audio segments
Spoken content Spectral characterization Music: timbre, melody
17

MPEG-7: MMDS Basic Elements

Root, Top-level elements, Packages

Basic elements Datatype & structures Schema tools

Time, Duration, Medialocators

Language Annotation,Person, Place

Link & media localization

Basic DSs
18

MPEG 7: Content Management and Description

Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Title, Creator, Creation location & date, Purpose, Classification, Genre, Review, Parental guidance, etc. (Author generated)
Creation & production Media Content management Content description Structural aspects Conceptual aspects Content Usage

Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution)

Viewpoint of the structure: Segments Basic elements Spatial / temporal structure Audio, Schema Datatype &video low-level Ds Elementary semantic information. tools structures

Viewpoint of conceptual notions Events, objects, abstract concepts, and Link & media their relation Basic DSs
localization
19

Example of Segment trees

SR6: Color Histogram Textual annotation

SR1: Creation, Usage meta information Media description Textual annotation Color histogram, Texture SR2: Shape Color Histogram Textual annotation SR5: Shape Textual annotation SR4: Shape Color Histogram Textual annotation 20

Background

Foreground

SR3: Shape Color Histogram Textual annotation

Time Axis Shot1 Segment 1 Sub-segment 1

Segment Tree
Shot2 Shot3

Semantic DS (Events)

Introduction Summary

Sub-segment 2

Program logo

Sub-segment 3

Studio Overview

Sub-segment 4 segment 2

News Presenter

News Items
Segment 3

International Clinton Case

Segment 4

Pope in Cuba National

Segment 5

Twins Sports Closing

Segment 6

Segment 7

21

MPEG 7: Navigation and Access

Creation support of: discovery, Efficient & production browsing, navigation, visualization Media Content management Content description Structural aspects Conceptual aspects Content Usage

Navigation & Access

Summary

Variation

Basic elements Datatype & structures

Substitution of the original content Adaptation to terminal, network, or Schema user preferences Link & media
tools
localization

Basic DSs
22

MPEG 7: Hierarchical summary

Hierarchical Summary

HighlightLevel

HighlightLevel

Highlight Segment A-V

Data

Highlight Segment

Highlight Segment

Highlight Segment

Highlight Segment

Highlight Segment

Highlight Segment

23

MPEG 7: Sequential summary

Frame Property

Frame Property

Frame Property

Sequential Summary

Text Property

Text Property

Sound Property

Sound Property

Sound Property

A-V
Data

24

MPEG 7: Variation
Universal Multimedia Access
Adapt delivery to network and terminal characteristics (QoS)

25

MPEG-7: Content Organization

Content organization

Collection & Classification

Analytic Model

Description and organization of Creation collection of documents &

production Media Content management Content description Structural aspects Conceptual aspects Content Usage

Navigation & Access

Summary

Variation

Basic elements Datatype & structures Schema tools Link & media localization

Basic DSs
26

MPEG-7: Collection

27

MPEG 7: User Interaction

Content organization

Collection & Classification

Analytic Model

Creation & production Media Content management Content description Structural aspects Conceptual aspects Content Usage

Navigation & Access

User Interaction

Summary User preferences Variation

Basic elements Datatype & structures Schema tools

User identification and preferences: Filtering, search and browsing

Link & media localization

Basic DSs
28

MPEG-7: TimeLine - The Work Plan

Competition: Divergence Individual work Definition scope and r 1998

Convergence
1999 2000 2001

1996

Call for proposals

Working draft

Committee draft Final committee draft

International standard Draft international standard

Conclusions on AV Content Description and MPEG-7

MPEG-7:
AV content description for interoperable application

Description Definition Language:

XML Schema (flexibility) + Binary version (efficiency)

Description Schemes:
Library of description tools Covers a wide range of generic needs
30