Multimedia Presentation Design Anders Petersen petersea uci edu Ariffin

Multimedia Presentation Design Anders Petersen petersea@uci. edu Ariffin Yahaya ariffin@ics. uci. edu

Papers Presented Today • Automating the Generation of Coordinated Multimedia Explanations (S. K. Feiner and K. R. Mc. Keown) – COMET: content selection, media generation, coordinated layout. • Plan-based Integration of Natural Language and Graphics Generation (W. Wahlster et al. ) – IBIS: generation of 3 -D illustrations. • Automated Generation of Intent-Based 3 D Illustrations (D. D. Seligmann and S. Feiner) – Allocation of information to particular media. • Presentation Design Using an Integrated Knowledge Base (Y. Arens et al. ) – WIP: generalization of text-linguistic notions relations to multimedia presentation • The Knowledge Underlying Multimedia Presentations (Y. Arens, Y. ) – Integrated Interfaces: dynamic construction of multimedia displays using rules. 2

Introduction • Typical Steps in multimedia presentations: – Determination of communicative intent. – Selection of content from a base of knowledge. – Grouping/structuring and ordering. – Allocation to particular media. – Layout. Ariffin Yahaya 3

Automating the generation of Coordinated Multimedia Explanation • Pictures and language complement each other to enable highly effective communication. – First generation authoring facility • BUT how does one go about putting together a system that does it dynamically? – COMET (coordinated multimedia explanation testbed) created to overcome these problems. Ariffin Yahaya 4

First generation authoring facilities • Basic facilities – Create presentations • Text, Graphics, Animation, Video • Problems – Requires skills • Medium conventions (i. e. what people expect) • Coherent mix of mediums. – Must be authored in advance Figure 1 First Generation Authoring • Limits the presentation to a known audience set. Ariffin Yahaya 5

Comet • Goal : – Coordinated, interactive generation of explanations that combine text and graphics, all generated by the system on the fly. • Example: – How to repair a radio receiver / transmitter • • Select symptoms from a menu. System consults a rule database. System can request user to perform actions. System explains actions step by step using graphics and text. Ariffin Yahaya 6

COMET is used in radio repair Remove the old holding Battery. Step 1 of 2. Step 1: Remove the holding battery cover plate, highlighted in the right picture: Loosen the captive screws and pull the holding battery cover plate off the radio. Figure 2 Sample of COMET output. Ariffin Yahaya 7

Comet Overview Content Planner User reque st Presentation generated. Domain knowledge • Contains databases for information used in all the components. – Content planner Geometric Knowledge Static objects Diagnostic rule base User model Previous discourse “How” to say it. Presentation generator – Knowledge Source Expert sys. determines explanation’s content. “What” to say Knowledge Source idle • Basic Components Coordinate explanation to medium. Media Coordinator Figure 3 : Comet Architecture & State Diagram • “Answers” the user’s request – Media coordinator • Associates “answers” with the best method of presentation. – Presentation generator • • Text generator. Graphics generator. Media layout Render & typeset 8

Knowledge Source • Static representation of the objects and actions. – Loom language. • Declarative knowledge in Loom consists of definitions, rules, facts, and default rules. • A deductive engine called a classifier utilizes forwardchaining, semantic unification and object-oriented truth maintenance technologies in order to compile the declarative knowledge into a network designed to efficiently support online deductive query processing • Diagnostic Rule Base – Rules pertaining to the application. • Detailed Geometric knowledgebase for graphics generation Ariffin Yahaya 9

Content Planner • Produces full content for the explanations. – Represented as a hierarchy of logical forms. • Logical forms (LF) are required as input to the next stage which uses functional unification grammar (FUF)*. – Text Plans or Schemas • “Blackboard” capability. – Intermediate results can be stored to see other results before committing. – Previous discourse * FUF not covered in this presentation Ariffin Yahaya 10

Media coordinator • Fine grained analysis of an input LF to decide whether each portion should be realized in images, text or both. • Uses FUF grammar that maps 6 different types of information to types of media. • Passes the output to the presentation generators. – Graphics generator. – Text generator. Ariffin Yahaya 11

6 Different Types of Information • Locations attributes. – Graphics alone • Physical attributes. – Graphics alone • Abstract actions. – Text alone • Expressive connectives that indicate relationships among actions. – Text alone • Simple actions. – Both Text and Graphics. • Compound actions. – Both Text and Graphics. Ariffin Yahaya 12

Text Generator • Type and Number of sentences needed. • Lexical chooser. – LF actions verbs. – LF objects nouns. – Chooses words based on multiple constraints • Wider variety and more appropriate output. Example: – Previous discourse: install reinstall – Use words user knows: technical term explain procedure • Sentence generator. – Construct the syntactic structure. – Linearize the resulting tree as a sentence Ariffin Yahaya 13

Graphics Generator • Uses IBIS (Intent based Illustration System) from the paper “Automated Generation of Intent-Based 3 D Illustrations”. • This paper is covered within this presentation set, so we will revisit IBIS later. • Suffices to say that the graphics generator takes in annotated LFs and outputs graphics. Ariffin Yahaya 14

Media Coordination (1) • Common content description language. – Text and Graphics influence each other. • Generators display “cohesive” presentations. – Communicative goal separated from resources. • LFs only specify goals and what is needed to achieve the goals, generator defines resources. – Provides mechanism for Text and Graphic generators to communicate. • Content description (blackboard) is used to coordinate internal text structures with pictures. Ariffin Yahaya 15

Media Coordination (2) • Bidirectional interaction – Certain types of coordination between media can only be provided by incorporating interactive constrains between text and graphics. • Coordinating sentence breaks with picture breaks. – Both Text and Graphics generators annotate their current process and can refer to each others progress and decisions so that it can compensate. • Cross-referencing text and graphics. – Text generator queries an IBIS database that is indexed by LF so that it can cross-reference specific objects and refer to the graphical locations within the generated text. Ariffin Yahaya 16

COMET Figure 4 COMET architecture from the Paper. Ariffin Yahaya 17

Automated Generation of Intent. Based 3 D Illustrations • Exact presentation of a message is available to us through technology, but many people viewing the same presentation may not lead to all the people having the same interpretation. • The intention of the author, and the viewing context (i. e. who is viewing) must be taken into consideration. • SO, how do we make sure that the presentation is appropriate? – IBIS (Intent Based Illustration System) Ariffin Yahaya 18

IBIS • Goals: – Automate the creation of illustrations based on a specific communicative intent. • Used by COMET • Method – Formalize the intent and create an illustration that fulfills the goals set by the intent. • Example: – Communicative Intent: Show a dice • Generated illustration: Whole dice is shown – Communicative Intent: Show the weight in loaded dice • Generated illustration: Transparent dice with weight visible. Ariffin Yahaya 19

Communication Intent makes a difference! weight Intent: Show the dice. Intent: Show the location of weights in loaded dice. Figure 5 Sample of IBIS output Ariffin Yahaya 20

Generate and Test • Every time IBIS generates a stylistic choice, it also associates a set of criteria to which the results are compared to. – System of ratings (criteria). – Thresholds (minimum degree of success). • Stylistic choice are associated with methods, and as each method is “tried”, the results are tested against the criteria. • If the criteria is not fulfilled, a new stylistic choice with a new method is requested. Ariffin Yahaya Figure 6 IBIS Generate and Test Cycles 21

IBIS Overview to design rules. Geometric Information Ineffective Material Information Features Physical Properties e e Drafter ff ne I v cti Match design rules to style rules. – Communicative Goals • Tightly coupled with COMET. – Generate & Test cycle • Method used by IBIS to evaluate appropriateness. – Illustrator • Maps intent to stylistic choice with “Design Rules”. – Knowledgebase • Superset of graphics info. Style rules Illustration tyle Drafter Abstract Properties • Basic components Design Rules Generate & Test Cycle idle Knowledgebase Commu Illustrator oals Match goals icative g n s ie teg ra St S Figure 7 : IBIS Architecture & State Diagram – Drafter • Maps stylistic choice to visual effect with “Style 22 Rules. ”

Communicative Goals • Location – Show the location of an object in a context. • Relative Location – Show the relative location of 2 or more objects in terms of a specified/derived context. • Property – Show objects physical properties of material, color, size or shape. • State* – Show an object’s state • Change* – Show the difference between a set of states. * State and change may further be qualified by concepts that refer to how the object is manipulated or has changed. Ariffin Yahaya 23

Illustrator • Designs illustrations. – Map communicative goals to style strategies with design methods. – Evaluate the success of communicative goals with design evaluators. • An illustrator may split jobs according to need and assign them to subordinate illustrators. • Share design rules database. Ariffin Yahaya 24

Design Rules • Describes on a high level how illustrations should be put together. – Communicative Goal – Set of Style Strategies. • Visual effect • Style rules. • Design Methods specify how to accomplish communication goals. – Specifies what style strategies must be achieved. • Design Evaluators determine how well communicative goals have been accomplished. – Achievement ratings of a collection of style strategies. Ariffin Yahaya 25

Knowledgebase • Concerned with physical objects to be illustrated. • Superset of typical graphics databases – Geometric Information – Material Information • Also includes – Object’s Features • What are the object’s capabilities – Physical Properties • How does an object move (i. e. a hinge). – Abstract Properties • How things fit together (? ) Ariffin Yahaya 26

Drafter • • • Knows nothing about communicative intent. Translates the illustrator's plans into reality. Tightly coupled with the hardware they utilize. Shares a database of style rules. Report back to the illustrators with the achievement rating of the various style strategies they implement • Render the illustrations. Ariffin Yahaya 27

Style Rules • 2 types of Style Rules that specify either – Style Methods • Accomplish visual effects specified by style strategies. • Illustration methods. – Style Evaluators • Determine the success of style strategies in a given illustration. • Illustration evaluator Ariffin Yahaya 28

Illustrative Style • Represented by an ordering of the rules such that the preferred methods are always attempted first. • Illustrators and drafters can be specified with different illustrative styles. • Illustrations can combine different illustrative styles. Ariffin Yahaya 29

Other IBIS Features • Interactive Illustrations – User can change view specifications with IBIS continuously monitoring to make sure that the communicative goal is maintained. • Written in C++ and CLIPS. Ariffin Yahaya 30

Intent based approach to Authoring Figure 8 Typical intent based authoring architecture. Ariffin Yahaya 31

Plan-based integration of natural language and graphics generation. • Multimodal presentations should be generated from a common representation of what is to be conveyed. • BUT – How do we decompose the communicative goal into sub goals. – How do we Integrate multiple AI components to create the presentation. • WIP multimodal presentation system was created to be a prototype to solve these problems. – Computer as a “desktop publisher”. Ariffin Yahaya 32

WIP Design Goals • Generate coordinated multimodal presentations from a common representation. – What should be in text / graphics. – Which kinds of links verbal / non-verbal is necessary. • Adaptation of these presentations to intended audience and situations. – All presentation decisions are postponed until runtime • Incrementality of all processes constituting the design and realization of the multimodal output. – computations for an object are performed not long before the object is output. Ariffin Yahaya 33

WIP • Goal : – Allows the generation of alternate presentations of the same content taking into account contextual factors such as the user’s degree of expertise and preferences for a particular output medium or mode. – Specify information once, but view in infinite ways. • Example: – How to use an espresso machine. – How to assemble a lawnmower. – How to install a modem • Inputs – Stereotypes, Target language, Layout format, and Output modes. Ariffin Yahaya 34

Interleaved Content Planning • Processing is done non linearly. • Cascades are used based on some task/results queues. • Cascade: – Presentation planner and Layout Manager – Design module – Realization module • Purpose: – Leave presentation decisions to the last possible moment to refine the presentation. Ariffin Yahaya 35

WIP Overview Ge idle P ne ra tio res n. P en tat ion Layout Manager go als s Determine contents and mode bus / queue Presentation Strategies Revision Strategies Selection Rules Text Design Graphics Design User Model Basic Ontology TAG Text Realization Graphics D. Strategies Figure 9 : WIP Architecture & State Diagram Incremental Design Cascade Knowledgebase Application Knowledge ete r Presentation Planner Determine layout and Generate. • Basic Components ara m – Presentation Planner • Decides on content and mode combination – Layout Manager • Screen/Output Manager • Last step to rendering presentation – Text & Graphics Cascade • Micro planner. – Application Knowledge • Application specific data. – Knowledgebase • Used internally. 36

Presentation Planner • Tries to find a presentation strategy whose effect (or header) match the presentation goal. • Keeps revising plan until some basic elements of the presentation are formed. • Elements sent to the task queue. • Design modules take task from the queue and begin processing. Ariffin Yahaya 37

Layout Manager • In charge of the screen real estate. • Positions design components. • Interacts with the realization module. Ariffin Yahaya 38

Text & Graphics Cascade • Design Module – Elementary Speech/Pictoral acts. – What to say (intent) micro-planner • Which view of the espresso machine. • What is the micro-message (instruction). • Realization Module – How to say it micro-planner • Natural Language • Geometric shapes Ariffin Yahaya 39

Application Knowledge • Externally coded in RAT • Main source of knowledge – Domain terminology • Used in – Presentation Planner – Generation of text – Generation of graphics Ariffin Yahaya 40

Knowledgebase • Application Knowledge – Unique domain knowledge • Strategies – Used to design/revise • Presentation • Graphics • User Model – Matches the generation parameters. Ariffin Yahaya 41

Comparison to COMET • WIP • COMET – Operator based approach to planning. – Supports incrementality. – Bidirectional communication between Presentation Planner and Layout Manager. Ariffin Yahaya – Schema based content planner – No increments – Layout component combines text and graphics fragments during final steps. 42

Coming up next: Anders with the second half of this talk. Figure 10 Ariffin is smiling ‘cos he’s done! Ariffin Yahaya 43