XG Multimedia Semantic News Use Case Thierry Declerck

XG Multimedia Semantic News Use Case Thierry Declerck, DFKI Gmb. H Language Technology Lab

Automatic Semantic Analysis of Metada associated with News Videos of Broacasting companies On-going work in the projects K-Space and MESH

Metadata of News Broadcasters • We analysed the metadata available from various Broadcasters • Their data consists of audio/video material and textual metadata. This is a very valuable data set, since the textual metadata consists also in manually annotated scenes descriptions. • This dataset can be used for building a training corpus for automated alignment of video, audio and text data. • In the next slides we see some abstraction over the various types of metadata provided. 3

The Metadata Labels § § Earthquake Iran § Journal F: 4001 C § § § § § § § 4

The Title Tag Extract: “Erdbeben” (keyword for disaster ontology) ; location “Iran” (with NE detection). Other terms, but yet still unclear about their role 5

The Description Tag Ein schweres Erdbeben hat im Iran die Stadt Bam fast völlig zerstört. Linguistic and semantic analysis: [Subj-NP Ein schweres Erbeben] [Vhat] [LOC-PP in Iran] [OBJ-NP die Stadt Bam] [ADV fast völlig] [Vzerstört]. Extraction: Who (causation): Erbeben (Earthquake) What_action: zertören (destroy) What: Stadt Bam (city of Bam). Here the system can infer that Bam is located in Iran. Where: Iran 6

Bam: Menschen sitzen zwischen Trümmern auf Boden verzweifelte" src="https://present5.com/presentation/3cf3d543f4173976af4c2f4745d182e2/image-7.jpg" alt="The Scenes Tag Bam: Menschen sitzen zwischen Trümmern auf Boden verzweifelte" /> The Scenes Tag Bam: Menschen sitzen zwischen Trümmern auf Boden verzweifelte Menschen sitzen am Strassenrand Schuttberge zerstörte Häuser rauchende Trümmer Descriptons of sequences of images displayed. Extracting related entities: People within ruins, desperate people, destroyed houses, smoking ruins etc. All those terms can be seen as “consequences of the earthquake”. Important also: they provide for a description of what is to be seen in the video. 7

The Keywords Tag Naher Osten: Iran; Erdbeben The pattern of the content of this tag allows us to infer that Iran is located in “near-east”. 8

Linguistic Knowledge Structures § Multiple layers and levels § § § Low-level linguistic features (tokenization, morphology, …) Semantic properties of terms and phrases Named Entities Relation Extraction (incl. Grammatical Relations) Semantic linking to domain ontologies § Can involve several abstraction layers connected through reasoning/mapping processes § Semantic linking to other media analysis § Associated to the domain ontology of MESH (natural disasters in the news) 9

Association of Ontologies 10

Semantic annotation of Text extracted from Images (Thierry Declerck, DFKI & Andreas Cobet, TUB)

Background § The data: The German Broadcast news programme „Tagesthemen“ § Extract Text from key frames of shots. Annotate those terms semantically § Analyse of the position of the text and the kind of text extracted. 6 cases detected so far: 12

Case 1: Above the picture, just a normal phrase, mostly a nominal phrase (NP) 13

Case 2: Below the picture: Name of a person and of the function of this person 14

Case 3: Below the picture: Name of a person and of a city/country 15

Case 4: Above the picture, just a normal nominal phrase, and below the picture, name of a person, 16

Case 5: Below the picture the word „Bericht“ (or similar) and name of Person (=> Journalist) 17

Case 6: A location name. No picture of a specific human 18

Cross-Media Ontologies The next slides by courtesy of Paul Buitelaar, Michael Sintek, Malte Kiesel (DFKI Gmb. H) from the Project Smart. Web. Paper „Feature Representation for Cross-Lingual, Cross. Media Semantic Web Applications“, presented at ESWC 2006.

Semiotic Triangle § See (Ogden & Richards, 1923) - based on § Structural Linguistics (de Saussure, 1916) § philosophical work by Peirce (mostly 19 th century) 20

Semiotic Triangle – the real world . . . actual goalkeepers in the real world. . . 21

Semiotic Triangle – concepts . . . actual goalkeepers in the real world. . . 22

Semiotic Triangle – words goalkeeper (EN) Torwart (DE) doelman (NL). . . actual goalkeepers in the real world. . . 23

Semiotic Triangle – images . . . actual goalkeepers in the real world. . . 24

Features § Multilingual Features § Terms with Linguistic Info and Context Models § Example: goalkeeper § § § part-of-speech: noun morphology: goal-keeper context (Google hits stats. ): [ gets: 420000, holds: 212000, shoots: 55900, … ] § Multimedia Features § Images with Feature Models § Example: goalkeeper § § § color: #111111 shape: human texture: “keypatch-set 223” 25

Representation – Proposal § Attach multilingual and multimedia features to classes and properties (and also instances) § use of meta-classes Class. With. Feats and Property. With. Feats with properties ling. Feat and img. Feat (with ranges Ling. Feat and Img. Feat) § The classes Ling. Feat and Img. Feat are used for complex feature descriptions rdfs: Class rdfs: sub. Class. Of feat: Class. With. Feats feat: ling. Feat feat: img. Feat if: Img. Feat if: color if: texture … rdfs: Property rdfs: sub. Class. Of feat: Property. With. Feats feat: ling. Feat feat: img. Feat lf: Ling. Feat lf: term lf: lang … meta-classes 26

Representation – Simplified Example 27

Representation – Ling. Info Ontology ling. Feat Class. With. Feats Ling. Feat ling. Feat Property. With. Feats Ling. Feat case gender number orthographic. Form part. Of. Speech root semantics term lang morph. Synt. Decomp Word. Form Literal {neuter, female, male} {singular, plural} Literal {Adj, Verb, Noun, . . . } Root Resource analysis. Index function Phrase. Or. Word. Form Integer {modifier, head, neg. Modifier} is-a Inflected. Word. Form Affix Word. Form phrase. Analysis phrase. Category Phrase Literal Phrase. Or. Word. Form {S, NP, AP, PP, VG} is. Composed. Of is-a inflection word. Form is-a Ling. Feat Literal {en_US, en_GB, en, de, fr, . . . } Phrase. Or. Word. Form Stem Word. Form orthographic. Form Morpheme Literal is-a Root is-a Affix . . . is-a 28

Example Instance: “Fußballspielers” (“of the football player”) lang morph. Syn. Decomp term case gender number ortographic. Form part. Of. Speech is. Composed. Of inst 0 : Ling. Feat de Fußballspielers inst 2 : Stem nominative male singular Fußballspieler Noun case gender number ortographic. Form part. Of. Speech word. Form root orthographic. Form inst 3 : Stem analysis. Index orthographic. Form. . . is. Composed. Of function root 1 Fußball modifier inst 1 : Inflected. Word. Form genitive male singular Fußballspielers Noun inst 1 : Root Spieler inst 8 : Stem analysis. Index orthographic. Form. . . function root 2 Spieler head inst 7 : Stem (Ball) inst 4 : Root (Ball) inst 5 : Stem (Fuß) inst 6 : Root (Fuß) 29

Features – Interacting Layers 30

Translating XBRL Into Description Logic Thierry Declerck and Hans-Ulrich Krieger DFKI Gmb. H

Motivation § Toward a large intelligent web-based financial information and decision support systems in the MUSING project § Till now a prototype based on XBRL (e. Xtensible Business Reporting Language), as developed within the e. Ten project WINS § There we experienced the limitations of the XBRL schema, due to the lack of reasoning support over XML-based data and information extracted from documents. § Need to translate XBRL into an ontology 32

XBRL Example: Header & Metadata #001 2001 -01 -01 2001 -12 -31 ISO 4217: EUR …. . …. XBRL Deutschland e. V. Düsseldorf 33

1338066 1338066 XBRL: Financial Data …. 1338066 0 0 0 749385 259760 209343 0 … 34

XBRL to OWL-XBRL § XBRL taxonomies make use of XML in order to describe the structure of an XBRL document as well as to define new datatypes and properties relevant to XBRL. § Allows to check whether a concrete (business) document conforms to the syntactic structure, defined in the schema. § But a need for languages and tools that go beyond the expressive syntactic power of XML Schema. § OWL, the Web Ontology Language is the new emerging language for the Semantic Web that originates from the DAML+OIL standardization. OWL still makes use of constructs from RDF and RDFS, such as rdf: resource, rdfs: sub. Class. Of, or rdfs: domain § Two important variants OWL Lite and OWL DL restrict the expressive power of RDFS, thereby ensuring decidability. § What makes OWL unique (as compared to RDFS or even XML Schema) is the fact that it can describe resources in more detail and that it comes with a well-defined model-theoretical semantics, inherited from description logic 35

Actual Experiment with the Sesame DB § The basic idea during our (manual) effort was that even though we are developing an XBRL taxonomy in OWL using Protégé, the information that is stored on disk is still RDF at the syntactic level. We were thus interested in RDF data base systems, wich make sense of the semantics of OWL and RDFS constructs such as rdfs: sub. Class. Of or owl: equivalent. Class § Current experiment with the Sesame open-source middleware framework for storing and retrieving RDF data. Sesame partially supports the semantics of RDFS and OWL constructs via entailment rules that compute “missing" RDF triples (the deductive closure) § From an RDF point of view, additional 62, 598 triples were generated through Sesame's deductive closure. 36