Herwig Rehatschek LREC Workshop Genoa 23 May 2006

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 Cross Media Aspects in the Areas of Media Monitoring and Content Production ISO 9001 certified 1 Institute of Information Systems & Information Management 18/03/2018 JOANNEUM RESEARCH a TRADITION of INNOVATION

Contents Outlook Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 2 Conclusions ISO 9001 certified è in media production (NM 2 project) è in media monitoring (DIRECT-INFO project) 18/03/2018 Cross media aspects

NM 2 (New Media for a New Millennium) - Motivation NM 2 investigates on è non-linear, interactive narratives è tools for production and delivery 3 è have developed their specific compelling media forms è art, science and technology of production, direction and editing in non-linear narrative are poorly formed 18/03/2018 Traditional distribution channels Main Objectives of NM 2 n aesthetically pleasing n truly engaging (30+ minutes) n plausible from a business perspective Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified è proof of concept to produce compelling non-linear narratives è production of compelling media

Basic Technical Concepts è MPEG-7 to describe audiovisual features è OWL to describe semantics of NO 18/03/2018 Narrative Objects (NO) Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified è internal representation of a „story world“ è basic narrative structures 4 Narrative Structure Language

Media Semantic Mapping Utility (Monsoon) Definition of an object by low level features Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Definition of an object by using other objects 5 18/03/2018 Tabs for objects, concepts and relations

System Architecture tools JRS delivery Description Including application preview Authoring Using narrative • Manual structure language • Automated • Ontology-based Session Management For high volumes 18/03/2018 Ingestion End-user Application 6 middleware Middleware API Management Realisation Engine Profile Learning metadata essence Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 stories user info ISO 9001 certified Database interface

Main Project Results Definition of a Narrative Structure Language Bridging semantic gap 18/03/2018 è describing multimedia objects in narrative context 7 è MPEG-7 & OWL è linking lower level features to higher level concepts Six NL productions with compelling media è Windows XP Media Centre and TV card è Game consoles è Mobile phones Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Targeted for three end-user devices

DIRECT-INFO - Background Media Monitoring Companies Find out 8 è how much a company has spent for advertising è on a product level è over different media 18/03/2018 Measurement of advertisement expenditure è advertisement information è creativity (new campaigns) (targeted in Media. Campaign) è sponsorship tracking (targeted in DIRECT-INFO) Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Typical business cases

Sponsorship Tracking è broadcasting & print media è convince sponsors to invest again è currently done manually 9 Measure efficiency of sport sponsorship 18/03/2018 Sport teams need sponsors to finance their activities è context information needed è multimodal analysis necessary Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Technically very complex task

Multimodal event modelling Genre classification 10 Text analysis (pos. / neg. mentions) Semantic correlation & fusion of multi-modal content analysis to semantically meaningful reports Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 18/03/2018 Identification of logos within visual content ISO 9001 certified Combination of Main Technical Innovations

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 11 18/03/2018 Technical Workflow (1/2)

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 12 18/03/2018 Technical Workflow (2/2)

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 13 18/03/2018 Brand. Detector – Detection of Logos

MPEG-7 Result Viewer/Editor Module parameters Basic events Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Key frames shots 14 18/03/2018 Player

Conclusions è otherwise a lot of time will be lost with specification and extensions è MPEG-7, OWL already provided the needed structures è one expert for MPEG-7, OWL is necessary 18/03/2018 Standardized metadata (MPEG-7, OWL) is a good choice è managed vs. unmanaged code è not all SOAP protocol features supported 15 Web services not so straightforward as always thought è take security issues into account è remote access essential Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Do not underestimate integration efforts!

Outlook: Media. Campaign R&D project, Started: 04/2006 Semantic cross-market advertis. interlinking è relate similar advertisements over different media & countries 16 è in different media (Press, TV, Internet) è utilizing different modalities (audio, text, image, video) 18/03/2018 Identification and tracking of new campaigns è Media Presence and Campaign Ontology (MEPCO) è includes media presence information Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Define Ontology for cross-media modeling of media presence and campaigns

Relevant Links NM 2 website DIRECT-INFO website è http: //www. direct-info. net Media. Campaign website è http: //www. media-campaign. eu Available in June 2006! 17 18/03/2018 è http: //www. ist-nm 2. org/ è available for free! è full support for parts 3, 4 and 5 of the MPEG-7 standard è http: //mpeg-7. joanneum. at Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified JOANNEUM RESEARCH MPEG-7 Library

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 18 ISO 9001 certified JOANNEUM RESEARCH Institute of Information Systems & Information Management Steyrergasse 17 A-8010 Graz / AUSTRIA Voice: +43 316 876 -1194 Mobile: +43 699 1876 -1194 Fax: +43 316 876 -1191 E-mail: Herwig. Rehatschek@joanneum. at Web: http: //www. joanneum. at/iis 18/03/2018 Dr. Herwig Rehatschek

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 20 18/03/2018 Delivery

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 21 18/03/2018 MPEG-7 Automatic Detection Preview (UI)

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 22 18/03/2018 System Architecture

Genre Classification commercials news studio/speaker soccer credits movie 23 è è è 18/03/2018 Fully automatic classification of Semantic block classification è ~90% precision è ~90% recall Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Real-time analysis

Multimodal Event Modelling s 3 c 2 c 3 s 4 Event detection in audio and video Integration of basic events è è visual shots audio segments overlaid text visible frontal faces u s 4 s 5 18/03/2018 s 2 c 1 s 2 a 2+s 2 24 s 1 Classified segments Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Pseudo-scene/story segmentation

Text Analysis è brand names è teams SCHUG text analysis engine 18/03/2018 Detect positive & negative mentions of Topic detection 25 è “Shallow and Chunk-based Unification Grammar” è robust chunk parser for linguistic analysis è transcripts and text captions of TV audio è newspaper texts è supported languages: English, Italian Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Sources

Semantic Data Fusion Rule-based derivation of new facts èbasic appearances ècomplex appearances 26 Based on time-stamped analysis results of all analysis modules 18/03/2018 Combing data from various analysis subsystems èconsolidation of appearances èset-up application for delivery system Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified Two main user interfaces

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 27 18/03/2018 Semantic Data Fusion Display of Results

Herwig Rehatschek LREC Workshop, Genoa 23 May 2006 ISO 9001 certified 28 18/03/2018 Delivery system for Customers