CS 586 Distributed Multimedia Information Management Prof

CS 586 – Distributed Multimedia Information Management Prof. Dennis Mc. Leod

About the paper. . . Towards Ontology-Driven Discourse: From Semantic Graphs to Multimedia Presentations In Proceedings of the 2 nd International Semantic Web Conference, 2003 By: Joost Geurts, Stefano Bocconi, Jacco van Ossenbruggen, and Lynda Hardman Presented by: Gabriela Gutierrez, February 11, 2004

Overview • Introduction • Example Scenario: Rembrandt • Process 1: From Semantic Graph to Structured Progression • Process 2: From Structured Progression to Multimedia Presentation • Conclusion

Introduction • Traditionally – Applying Semantic Web technology to multimedia information systems focuses on using annotations and ontologies to improve retrieval process – Presentation of data is “detail” best left to CSS or XSLT style sheets • In this paper – Claim that importance and complexity of effective presentation design is grossly underestimated – Concentration on improving the presentation of the retrieval results

Introduction • Human professional designers must understand: – Underlying semantics of the client’s information – Most effective order, grouping and priorities for structuring this information – Most effective means of using the chosen medium to convey the information

Introduction • Information presentation design is a knowledge-driven process. It requires: – Sufficient knowledge about domain – Knowledge on ordering, grouping and prioritizing information – Knowledge about media design • Selection of most appropriate medium • Understanding of medium characteristics in order to choose an effective means to achieve the communication goal

Introduction • Problem: – Professional designers can only design data-driven web sites if the underlying data, its semantics and target audience are relatively homogeneous. – Variety of data sources, semantic relations, output devices, and user profiles forces content providers to adopt one-size-fits-all approach. • Automation is needed in order to make the presentation of information knowledge-driven.

Introduction • Assumptions: – Multimedia items are properly annotated – Annotations represent domain relations in a semantic graph (e. g. RDF) – Graph has associated Domain ontology – There is a Discourse ontology containing information about different document genres and building blocks for creating documents for each genre – There is a Design ontology containing media design knowledge

Introduction

Example Scenario: Rembrandt • Web query: “life and work of Rembrandt” • User-selected type of structured progression: disc: Biography • User-selected output medium: non-interactive multimedia presentation • Semantic graph = retrieval component’s results + domain ontology semantics relations • Structured progression = typical facts (name, DOB, …) + career facts + personal life info

Process 1: From Semantic Graph to Structured Progression • CSS and XSLT operate purely on the XML level of RDF’s serialization syntax w/o any understanding or support for semantics of RDF data model • Transformation process needs access to knowledge on RDF Schema level – For querying underlying domain ontology – For access to its own operating knowledge

Process 1: From Semantic Graph to Structured Progression • Several transformations prototyped in Java and Prolog environments • Direct access to a Sesame RDF Schema-based repository • Can use any query language supported by Sesame (RQL, RDQL, Se. RQL) to gain direct access on the RDF instance level and the RDF Schema Level • Transformation process uses (declarative) domain and discourse-specific knowledge, while (procedural) transformation code remains generic

Process 1: From Semantic Graph to Structured Progression • Transformation code uses RQL query to retrieve classes that Rembrandt instance belongs to. . . dom: Artist • Discourse ontology defines instance of disc: Artist. Biography that has disc: Subject property with value dom: Artist • Structured progressions have a disc: narrative. Units property that specifies the disc: Narrative. Units that can be used to construct it (e. g. disc: Personal. Data, disc: Private. Life and disc: Career)

Process 1: From Semantic Graph to Structured Progression • Narrative Units have associated rules used to select matching content • Example: disc: Private. Life – Rules to select information about family relations from semantic graph – Graph includes relation dom: is. Married between Rembrandt and Saskia_Uylenburgh – Rule #3 in following table can use domain relation to select Saskia in the disc: Role of disc: Spouse • Rules can be applied recursively – Rule #3 specifies that Private. Life is the narrative unit that can be used for a subsequent nested story line – Process continues until no more rules can be applied or a rule specifies that no further expansion should happen

Process 1: From Semantic Graph to Structured Progression • After all rules have been applied: – Biography w/ 3 narrative units – disc: Personal. Data (Rembrandt in role of disc: Main. Character) – disc: Career (Chiaroscuro in role of disc: Technique) – Disc: Private. Life (Saskia_Uylenburgh in role of disc: Spouse)

Process 2: From Structured Progression to Multimedia Presentation • Two-step process: 1. Structured progression transformed into Document Structure – Decisions on output medium (e. g. text, interactive hypermedia, passive multimedia) 2. Document Structure transformed into a tree of formatting objects – Detailed layout and formatting decisions (e. g. timing of presentation, transition effects)

Process 2: From Structured Progression to Multimedia Presentation • Advantage: – Mapping discourse-specific narrative units to more general document elements allows for more commonly applicable formatting rules (e. g. disc: Private. Life can be mapped to document section element, relying on common formatting rules for section-level elements) • Disadvantage: – There is always a level that can no longer be specified in terms of document structure (e. g. a figure w/ too much detail) – Solution: detailed structures copied directly into document structure in step 1 in order to define specific rules in step 2 to deal w/ formatting directly

From Structured Progression to Document Structure

Conveying Document Structure • Transforming a document structure into presentation constructs uses Cuypers library – Uses constraint solving techniques to verify that a presentation construct conforms to delivery-context constraints (e. g. screen size) – Allows alternative formatting specification if constraints are violated • A rule that transforms a document structure into presentation construct has 2 discourse parameters: – disc: Narrative. Type – disc: Role – Parameters allow system to adapt formatting of presentation to convey message more effectively

Conveying Discourse Semantics Directly • Depending on their function, we need to define formatting for different media types – Rembrandt self-portrait (disc: Portrait in disc: Personal. Data vs. disc: Painting illustrating Chiarocuro) • Awareness of impact of different media modalities • Fall-back rules – Image not identified as either disc: Portrait or disc: Painting should be applied generic formatting for images since mm: Painting and mm: Portrait are subclasses of mm: Image

Conclusion • Only short presentations have been generated to date, based on restricted domain ontology • Focus has been on single discourse structure (biography) and single document structure (multimedia presentation) • Additional research required to scale the system to more realistic scenarios • Under investigation: how knowledge about the user interacts w/ discourse and design knowledge used in current prototype