The MPEG-7 Standard — A Brief Tutorial

The MPEG-7 Standard - A Brief Tutorial - Ali Tabatabai Sony US Research Laboratories February 27, 2001

Outline q Objectives of the MPEG-7 Standard q Main Elements of MPEG-7 q Scope of MPEG-7 q MPEG-7 Application Areas q MPEG 7’s relation with other standards 2

Why do we need MPEG-7 ? Support for Advanced Query • Fast & Accurate Access Need • Personalized Content Production and Consumption • Content Management • Automation + • Visual • Audio • Sketch 3

MPEG: A Brief History (1) § MPEG: Moving Picture Experts Group Established in 1988 § ISO / IEC/JTC 1/SC 29/WG 11 A Working Group of ISO/IEC in charge of the Development of Standards for Coded Representation of Digital Audio and Video 4

MPEG: A Brief History (2) q MPEG-1: Interactive CD and MP 3 11 / 1992 q MPEG-2: DTV, STB, DVD 11 / 1994 q MPEG-4: Web and Mobility ver 1: 09 /1998 ver 2: 11 /1999 q MPEG-7: ? ? ? q MPEG-21: Multimedia Framework 08 / 2001 11 / 2001 5

MPEG-7: What Is It ? THE MPEG 7 STANDARD IS NOT a COMPRESSION Standard similar to MPEG-1/2/4 or their Extension IS NOT a STANDARD for FEATURE EXTRACTION/MATCHING Content Description of Various Audio Visual Information Types of Audio Visual Information • Audio, Speech • Moving video, still pictures, graphics • Information on how objects are combined in scenes 6

MPEG-7: Application Areas q Storage and retrieval of audiovisual databases (image, film, radio archives) q Broadcast media selection (radio, TV programs) q Surveillance (traffic control, surface transportation, production chains) q E-commerce and Tele-shopping (searching for clothes / patterns) q Remote sensing (cartography, ecology, natural resources management) q Entertainment (searching for a game, for a karaoke) q Cultural services (museums, art galleries) q Journalism (searching for events, persons) q Personalized news service on Internet (push media filtering) q Intelligent multimedia presentations q Educational applications q Bio-medical applications

MPEG-7 Description Scope for AV Content q Description Granularity q Low-level q High-level q Form q Access q Classification q Link q Context 8

MPEG-7: Main Elements q Descriptors (D) § syntax and semantics of each feature representation q Description Schemes (DS) § structure and semantics of the relationships between components q Description Definition Language (DDL) § creation of new DS’s § modification/extension of existing DS’s 9

MPEG-7: Major Functionalities q Systems (ISO / IEC 15938 - 1) q Description Definition Language (ISO / IEC 15938 - 2) q Visual (ISO / IEC 15938 - 3) q Audio (ISO / IEC 15938 - 4) q Multimedia Description Schemes (ISO / IEC 15938 - 5) q Reference Software (ISO / IEC 15938 - 6) 10

MPEG-7: Main Elements (2) DDL DS 1 DS 2 DS 3 Instantiation <Object> <Label/> <Definition/>. . </Object> DS 2 D 1 D 2 D 3 Systems 0001100 11

MPEG-7: Systems It defines tools to: Ø provide for efficient storage and transport Ø synchronize between content and description Ø manage and protect intellectual property 12

MPEG-7: DDL and its Components q Description Definition Language: § Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions § Instantiation of XML q XML Schema: § Data types § Simple and Complex types § Elements, attributes § Inheritance, Abstract types q MPEG-7 extensions: § Array and Matrix data type 13

MPEG-7: Audio q Sound Effects q Music Instrument Timbre q Spoken Content q Melody Contour 14

MPEG-7: Visual (1) q Color § quantization, dominant, scalable, color-structure, layout, Go. F/Go. P q Texture q Shape § region-based, contour-based, 3 D q Motion § camera motion, motion trajectory, parametric motion, motion activity 15

MPEG-7: Visual (2) q Localization § spatio temporal q Others § face recognition 16

MPEG-7: Basic Visual Structures q Grid Layout q 2 D-3 D Multiple View q Time Series q Spatial 2 D Coordinates q Temporal Interpolation 17

Low level Audio Visual descriptors Video segments Still regions • • Color Camera motion Motion activity Mosaic Moving regions Color Shape Position Texture Audio segments • • Color Motion trajectory Parametric motion Spatio-temporal shape • Spoken content • Spectral characterization • Music: timbre, melody 18

MPEG-7: MMDS Basic Elements Root, Top-level elements, Packages Time, Duration, Medialocators Language Annotation, Person, Place Basic elements Datatype & structures Schema tools Link & media localization Basic DSs 19

MPEG 7: Content Management and Description Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Title, Creator, Creation location & date, Purpose, Classification, Genre, Review, Parental guidance, etc. (Author generated) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Creation & production Media Content Usage Content management Content description Structural aspects Conceptual aspects Viewpoint of the structure: Segments Basic elements • Spatial / temporal structure • Audio, Schema Datatype &video low-level Ds • Elementary semantic information. tools structures Viewpoint of conceptual notions • Events, objects, abstract concepts, and Link & media their relation Basic DSs localization 20

Example of Segment trees SR 1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR 6: · Color Histogram · Textual annotation Background SR 3: · Shape · Color Histogram · Textual annotation Foreground SR 2: · Shape · Color Histogram · Textual annotation SR 5: · Shape · Textual annotation SR 4: · Shape · Color Histogram · Textual annotation 21

Time Axis Segment Tree Shot 1 Segment 1 Sub-segment 1 Shot 2 Semantic DS (Events) Shot 3 • Introduction • Summary Sub-segment 2 Sub-segment 3 • Program logo • Studio • Overview Sub-segment 4 • News Presenter segment 2 • News Items Segment 3 • International • Clinton Case • Pope in Cuba Segment 4 • National Segment 5 Segment 6 • Twins • Sports • Closing Segment 7 22

MPEG 7: Navigation and Access Creation support of: discovery, Efficient & production browsing, navigation, visualization Media Content Usage Content management Navigation & Access Summary Content description Structural aspects Basic elements Datatype & structures Conceptual aspects Substitution of the original content Adaptation to terminal, network, or Schema user preferences Link & media tools localization Variation Basic DSs 23

MPEG 7: Hierarchical summary Hierarchical Summary Highlight. Level Highlight Segment Highlight Segment A-V Data 24

MPEG 7: Sequential summary Frame Property Sequential Summary Text Property Sound Property Frame Property Text Property Sound Property A-V Data 25

MPEG 7: Variation Universal Multimedia Access Adapt delivery to network and terminal characteristics (Qo. S) 26

MPEG-7: Content Organization Collection & Classification Content organization Description and organization of Creation collection of documents & Analytic Model Navigation & Access production Media Content Usage Content management Summary Content description Structural aspects Conceptual aspects Variation Basic elements Datatype & structures Schema tools Link & media localization Basic DSs 27

MPEG-7: Collection 28

MPEG 7: User Interaction Collection & Classification Content organization Analytic Model Navigation & Access Creation & production Media Content Usage Content management Summary Content description Structural aspects Conceptual aspects User preferences Variation User identification and preferences: Filtering, search and browsing Basic elements Datatype & structures User Interaction Schema tools Link & media localization Basic DSs 29

MPEG-7 Its Relation with other standards q AHG on “Metadata harmonization”: § SMPTE: Metadata dictionary, KLV encoding § Dublin Core Metadata Initiative § European Broadcast Union q AHG on TV Any. Time Application q Large number of Liaisons: § SMPTE § Dublin Core § W 3 C (XML Schema) § etc. 30

MPEG-7: Time. Line - The Work Plan 1996 Competition: Divergence • Individual work • Definition scope and r 1998 Call for proposals Convergence 1999 Working draft 2000 Committee draft Final committee draft 2001 International standard Draft international standard

Conclusions on AV Content Description and MPEG-7 q MPEG-7: § AV content description for interoperable application q Description Definition Language: § XML Schema (flexibility) + Binary version (efficiency) q Description Schemes: § Library of description tools § Covers a wide range of generic needs 32

The MPEG-7 Standard — A Brief Tutorial —