Dagstuhl Seminar Semantics for the Web Dagstuhl, 22 March, 2000 Generating Virtual Webpages Wolfgang Wahlster German Research Center for Artificial Intelligence, DFKI Gmb. H Stuhlsatzenhausweg 3 66123 Saarbruecken, Germany phone: (+49 681) 302 -5252/4162 fax: (+49 681) 302 -5341 e-mail: wahlster@dfki. de WWW: http: //www. dfki. de/~wahlster © W. Wahlster, DFKI
Three Generations of Web Sites First Generation Second Generation Interactive Web Sites Static Web Sites Third Generation Virtual Webpages Java. Scripts and Applets Fossils cast in HTML Netbots, Information Extraction, Presentation Planners Database Access and Template-based Generation User Modeling, Machine Learning, Online Layout Dynamic Web Sites Adaptive Web Sites © W. Wahlster, DFKI
What is a Virtual Web Page? Virtual Memory, Virtual Relation, Virtual Reality. . . A Virtual Web Page l is generated on the fly as a combination of various media objects from multiple web sites or as a transformation of a real web page. l looks like a real web page, but is not persistently stored. l integrates generated and retrieved material in a coordinated way. l can be tailored to a particular user profile and adapted to a particular interaction context. l has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components. © W. Wahlster, DFKI
Ai. A: Information Integration for Virtual Webpages PAN Travel Agent Andi Car Route Planner Yahoo News Server Yahoo Weather Server Hotel Guide Gault Millau Restaurant Guide © W. Wahlster, DFKI
Virtual Webpage Retrieved from 5 Different Servers © W. Wahlster, DFKI
The Combination of Retrieved and Generated Media Objects for Virtual Webpages Multi-Domain Problem Specs Multiple Data Sources Information Structures l Relations, Lists l KR Terms NETBOT Retrieved Results Distributed Information Media Objects l Texts, Sounds, Videos l Pictures, Maps, Animations © W. Wahlster, DFKI
The Combination of Retrieved and Generated Media Objects for Virtual Webpages Information Structures l Relations, Lists l KR Terms Select Canned Media Objects Design New Media Objects l Icons, Clip Art l Frames, Sounds l Graphics, Animation l Text, Speech, Mimic Select & Design Retrieved Results Media Objects l Texts, Sounds, Videos l Pictures, Maps, Animations Coordinate Media Objects l Temporal Synchronization l Spatial Layout Transform Media Objects l Clip, Convert, Abstract l Zoom, Pan, Transition Effects Reuse & Transform © W. Wahlster, DFKI
Operational Models of Referential Semantics for Robots and Netbots Robot Netbot Set of Subsumption Relations in an Ontology “Screw” “Departure Time” Set of Recognizers Physical Objects Screw 1 . . . Screw N Set of Wrappers WWW Objects DT 1 . . . DT N © W. Wahlster, DFKI
Enhancement of User Interfaces through Personalization System is able to flexibly tailor presentations to the individual user and the current situation. An animated character serves as “Alter Ego” of the presentation system. Personalized Presenters at DFKI © W. Wahlster, DFKI
The Role of Ontological Annotations for the Generation and Analysis of virtual Webpages with Ontological Annotations Webpages without Ontological Annotations Information Extraction Agent Presentation Planner Virtual Webpage Presentation Agent Persona With Ontological Annotations in: SHOE, OML, XOL, OIL and Persona Annotation in PML Information Extraction Agents Tri. AS © W. Wahlster, DFKI
Video Dresdner Bank © W. Wahlster, DFKI
The Generation of Virtual Webpages with PAN and Ai. A Hotel Agent Address Map Agent Ai. A Pictures and Graphics Pieces of Text Trip Data Netbot PAN Presentation Planner Components of virtual Webpages Coordinates for Pointing Gestures Input for Speech Synthesis Icons for Hyperlinks Weather Agent Train & Flight Scheduling Agent Virtual Web Presentation Persona Server Constraintbased Online Layout Major Event Agent © W. Wahlster, DFKI
Persona as a Personal Travel Consultant © W. Wahlster, DFKI
AIA’s presentation planner has been extended to accommodate for various target platforms through the introduction of a mark-up language layer Presentation Planner PETPML PET Persona Player Agent Script MS-Agent Controller SMIL Player WMLBrowser © W. Wahlster, DFKI
PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications Visual Appearances Behaviors l l Presentation Gestures Reactive Behaviors Idle-time actions Navigation actions Hand-drawn Cartoon Bitmaps Persona Server Auditory Characteristics l Sound effects, auditory icons l Voice: male, female Video Bitmaps Generated Bitmaps from 3 D-Models © W. Wahlster, DFKI
Classification of Persona Gestures Gesture Catalogue Talking Posture 1 • cautious, hesitant • appeal for compliance • avoids body-gestures Talking Posture 2 • active, attentive • self-confident • uses body-gestures © W. Wahlster, DFKI
Context-Sensitive Decomposition of Persona Actions High-Level Persona Actions take-position (t 1 t 2 ) Context-Sensitive Expansion (including Navigation Actions) Decomposition into Uninterruptable Basic Postures Bitmaps point-to (t 3 t 4 ) move-to (t 1 t 2 ) r-stick-pointing (t 3 t 4 ) r-turn (t 1 t 21 ) r-hand-lift (t 3 t 31 ) r-step (t 21 t 22 ) f-turn (t 22 t 2 ) . . r-stick-expose (t 31 t 4 ) . . . © W. Wahlster, DFKI
Extensions of the Representation Formalism Distinction between production and presentation acts (i. e. Persona- or display acts) Explicit representation of qualitative and quantitative constraints Production Act Presentation Act Introduce Create. Graphics S-Show S-Position Elaborate-Parts S-Wait Label S-Create. Window S-Depict S-Point S-Speak Qualitative constraints: Create-Graphics meets S-Show, . . . Metric constraints: 1 <= Duration S-Wait <= 1, . . . © W. Wahlster, DFKI
Persona Presents an Automatically Designed Business Chart © W. Wahlster, DFKI
Virtual Webpage with Animation Effects Based on a Single GIF Image © W. Wahlster, DFKI
Transition Effects in a Series of Retrieved Pictures © W. Wahlster, DFKI
PET: Persona-Enabling Toolkit Objective: l Enable non-professional computer users to populate their web pages with lifelike characters PET comes with: l a set of characters and basic gestures l an easy-to-learn Persona markup language Developer’s PET will include: l a character design tool which enables users to build their own characters Technical Realization: l Based on XML and Java © W. Wahlster, DFKI
The Persona Markup Language Specification of Persona actions <html> <head> Specification of the <title> Persona Test </title> character to be used </head> <body> <persona bitmap=“cartoon”. . . > <uselib url=. . . /> <do name=“greet”/> <do name =“speak” args=“hello”/> </persona> </body> </html> Features: – XML-based – easy to learn © W. Wahlster, DFKI
Tasks of PET-Parser l Preprocessing (required for non-standard HTML pages) l Identification of Persona Tags PET-Generator l Replacement of Persona Tag with Java Applet l Determination of parameters for Java Applet l Realization of Persona active elements as Java. Script l Generation of DHTML layers (transparency of Applet box) l Generation of data, e. g. audio files for speech output l Generation of scripts © W. Wahlster, DFKI
Functional View of PET Bitmaps Webpage with Reference to Java Applet URL of Webpage with Persona Tag <html> <head> <title> Persona Test </title> </head> <body> <persona bitmap=“cartoon”. . . > <uselib url=. . . /> <do name=“greet”/> <do name=“standard”/> <do name =“speak” args=“hello”/> </persona> </body> </html> PET Application Server PET Parser PET Generator <html>. . . <APPLET archive=“personaplayer. jar”. . . </APPLET>. . . </html> Persona Scripts waitscreen 4 gesture greet 0 0 null gesture laugh 0 0 null. . . Persona Engine Behavior Monitor Audio Data Event Handler Character Composer © W. Wahlster, DFKI
The Bidirectional Control Flow on Persona-Enabled Webpages Triggers actions of the Persona l Mouse Clicks l Mouse Movements l Text Input l Speech Input l Menu Input l Direct Manipulation Input Web Persona Triggers operations on elements of the webpage © W. Wahlster, DFKI
Video 911 & Boxter © W. Wahlster, DFKI
Persona Active Elements (PAE) l Active Images An active image starts a persona action when clicked. <ACTIVEIMAGE SRC=“image” HREF=“url” NAME=“image name” STATUS=“status message” ALT=“tooltip” CACTION=“persona action on. Click” OACTION=“persona action on Mouse. Over”. . . > l Addressable Objects An addressable object is an object which can be addressed and manipulated by Persona via its name and its position. <PDIV DIVNAME=“name of the element” DVFRAME=“frame name” TOP=“anchor-y” LEFT=“anchor-x”>some HTML elements</PDIV> © W. Wahlster, DFKI
Spatial and Temporal Coordination of Multimedia Presentations • Multimedia coordination in previous systems: – no declarative representation of spatial and temporal layout – no synchronization of Persona actions with other dynamic multimedia objects • Solution: Use Pre. Plan for automated generation of SMIL expressions – Synchronized Multimedia Integration Language – officially recommended by W 3 consortium © W. Wahlster, DFKI
Approach • High-level specification of temporal and spatial constraints within the operators of the Presentation Planner Pre. Plan • Use of the incremental constraint solving toolkit Cassowary – Uniform treatment of spatial and temporal constraints – Transformation of qualitative constraints into metric constraints – Integration of a backtracking mechanism to handle disjunctions © W. Wahlster, DFKI
Processing Steps Decomposition of presentation goals into elementary acts and collection of spatial and temporal constraints Constraint propagation using extended Cassowary constraint solver Representation of spatial and temporal layout as SMIL constructs © W. Wahlster, DFKI
Example of a Plan Operator (define-plan-operator : header (A 0 (Show. Presentation ? topic)) : constraints (*and* (BELP (Illustrates ? video ? topic)) (BELP (Video ? video)) (BELP (Sets-to-Music ? audio ? topic)) (BELP (Audio ? audio)) (BELP (Summarizes ? title ? topic))) : inferiors (A 1 (SAdd. Smil. Code (? video))) (A 2 (SAdd. Smil. Code (? audio))) (A 3 (SAdd. Smil. Code (? title))) : temporal ((A 1 (d) A 3) (2 <= begin A 1 - begin A 2)) : spatial ( (aligntop A 1) (centerh A 3) (20 <= top A 1 - bottom A 3 <= 20)) © W. Wahlster, DFKI
Representation of Spatial and Temporal Constraints • Spatial constraints – Qualitative constraints (Image 1 Left. Of Image 2), (Image 1 Top. Align Image 2) – Quantitative constraints (4 < Top Image 1 - Top Image 2) • Temporal constraints – Qualitative constraints Allen constraints: (meets, before, overlaps, . . . ) – Quantitative constraints Linear inequalities: (Begin Audio 1 - End Audio 2 < 3), (2 <= Duration Audio 1 <= 6) © W. Wahlster, DFKI
Resulting SMIL-Document <smil> <head> <meta content="Smiling Saarland" name="title"/> <layout> <root-layout height="50" width="300"/> <region height="30" id="reg 43463" left="0" top="0" width="300"/> </layout> </head> <body> <par> <audio begin="1. 0 s" end="21. 807 s" src="snowman. wav"/> <textstream begin="0. 0 s" end="22. 807 s" region="reg 43463" src="snowman. rt"/> <video begin="3. 0 s" end="21. 807 s" src="snowman. avi"/> </par> </body> </smil> © W. Wahlster, DFKI
Interactive Web Demo © W. Wahlster, DFKI
Synchronization of Persona Actions with other Media Objects Some Examples: • point to an object when it appears in a video • comment on a video 3 seconds after it has started • Smile when video appears on the screen • repeat lip movements until audio stops © W. Wahlster, DFKI
Synchronization of Persona Actions (define-plan-operator : header (A 0 (Show. Presentation ? topic)) : constraints (*and* (BELP (Describes ? audio ? topic)) (BELP (Audio ? audio)) (BELP (Talking. Gesture ? video)) (BELP (Repeat. Gesture ? video)) : inferiors (A 1 (Present. Picture. Sequence (? topic))) (A 2 (SAdd. Smile. Code (? audio))) (A 3 (SAdd. Smil. Code (? video))) : temporal ((A 2 (d) A 1) (2 <= begin A 2 - begin A 1) (A 2 (e) A 3)) : spatial ( (aligntop A 1) (alignleft A 1) (1 <= bottom A 0 - bottom A 3 <= 1) (1 <= right A 0 - right A 3 <= 1)) © W. Wahlster, DFKI
SMIL Specification for Persona Presentation <smil> <head> <meta name="title" content="Cyberella in Real. Pix Demo" /> <layout> <root-layout width="650" height="385" background-color="#9898 F 8"/> <region id=“reg 471101" top="0" left="12" width="385" height="284" background-color="#9898 F 8"/> <region id=“reg 081591" top="95" left=“ 354" width="295" height="385" background-color="#9898 F 8"/> </layout> </head> <body> <par> <img region= " reg 471101" begin= " 0. 0" src= "schwarzenacker. rp" fill="freeze"/> <animation begin= "2. 0 s" end="15. 1" region="reg 471101" src= "talking-gst. rp"/> <audio begin= "2. 0 s" end="15. 1" src="schwarzenacker. rm" /> </par> </body> </smil> © W. Wahlster, DFKI
Resulting Timeline Diagram . . At archeological finds in Schwarzenacker, . . © W. Wahlster, DFKI
Using SMIL to Synchronize Persona Actions © W. Wahlster, DFKI
Video Otto 1 © W. Wahlster, DFKI
Information Extraction Agents • Information Filtering • Information Retrieval • Information Integration identify relevant documents –. . . – identify and extract relevant pieces of information – transform them into canonical form wrappers • operational descriptions of a target concept • abstract from concrete occurrence within document • robust against modifications © W. Wahlster, DFKI
The Trainable Information Agents Framework specifications results combination of "classical" problem-solving methods and information agents Application info requests User requests training preferences/heuristics planning knowledge user preferences info domain ontology Browser Info. Broker improved dialog guidance PBD dialog info requests site info/ upd ate info or script query Info Extraction Trainer planning, optimization, Web site annotations and execution rma info ite s © W. Wahlster, DFKI
High Degree of Parallelism of Queries © W. Wahlster, DFKI
Knowledge about a Webpage Shared by User and Agent common part (usable for communication) structural visual/semantic Naive User procedural Learning Annotation Agent © W. Wahlster, DFKI
Video PAN © W. Wahlster, DFKI
Example - Ontology Train_Connection [ from =>> Location; to =>> Location; travel_date =>> Date; time =>> Time; depart_time =>> Time; arrive_time =>> Time; cost =>> Price; travel_duration =>> Duration; info_url =>> URL; . . . ] © W. Wahlster, DFKI
Query Planning - I • states: information states – concepts / attributes and instantiations • operators: querying schemes – preconditions (´+´) and effects (´-´) to time arrive_time travel_duration < +, +, -, -, -, u, - > from travel_date depart_time cost info_url © W. Wahlster, DFKI
Query Planning - II • merging of alternative plans with identical intermediate states • (a) op 2 • (a, b) op 3 • op 1 (a, c) op 4 • (a, b, c, d) • • (a, b, c, d, e) • (a, b) op 3 (a, b, c, d, f) • (a, c) op 4 • (a, b, c, d) op 6 op 5 merging • • op 1 • (a)op 2 op 5 op 6 • (a, b, c, d, e) • (a, b, c, d, f) © W. Wahlster, DFKI
Query Planning - III Top State Ontology City. value = München City. language = German . . . String . . . Operators City. Name 1 Language City value – – + + < String, Language, String, Language > babelfish . . . language " c Î pre (op ) $ i : S 0 = i : : c ¢ Ù c c ¢Ù S 0 = Int ( op ) © W. Wahlster, DFKI
Query Plan Visualization • Features – alternative queries – past states – future states – state descriptions – PBD requests – accept / reject PBD request – assessment of plans – expected completion time © W. Wahlster, DFKI
Using the presentation planner to serve mobile users The new challenge accommodate for device-specific display and interaction limitations (e. g. 60*90 pixel displays) Approach - develop new designs for presenting information on mobile devices - identify elementary design elements - define new presentation strategies for the automated composition of designs © W. Wahlster, DFKI
Application example: Route Descriptions over the Mobile Phone Planner a selected path gets translated by the planner into a sequence of WML pages to be displayed on a mobile phone “vertical bar view” “runway view” “isometric view” © W. Wahlster, DFKI
Video Otto 2 © W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce Digital Assistant Selector © W. Wahlster, DFKI
Simulated Dialogues as a Novel Presentation Technique • Presentation teams convey certain rhetorical relationships in a more canonical way – Provide pros and cons • The single presenters can serve as indices which help the user to classify information. – Provide information from different points of view, e. g. businessman versus tourist • Presentation teams can serve as rhetorical devices that allow for a continuous reinforcement of beliefs – involve pseudo-experts to increase evidence © W. Wahlster, DFKI
Presentation Teams I recommend you this SLX limousine. © W. Wahlster, DFKI
Underlying Knowledge Base • Representation of domain – FACT attribute car_1 consumption_car_1 • Value dimensions for cars adopted from a study of the German car market – safety, economy, comfort, sportiness, prestige, family and environmental friendliness – FACT polarity consumption_car_1 economy negative • Difficulty to infer implication of dimension on attribute – FACT difficulty consumption_car_1 economy low © W. Wahlster, DFKI
Example of a Dialogue Strategy Question: How much gas does it consume? Answer: It consumes 8 l per 100 km. Negative Response: I’m worrying about the running costs. Dampening Counter: Forget about the costs. Think of the prestige! Header: (dampening_counter ? agent ? prop ? dim) Constraints: (*and* (positive ? agent) (pol ? prop ? other_dim positive)) Inferiors: (Speak ? agent (“Forget about the ” ? dim “!”)) (Speak ? agent (“Think of the ” ? other_dim “!”)) © W. Wahlster, DFKI
Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support User-Adaptive Search Planning . . . weighted propositions Multiple Decision Support Agents Spare parts for this car are rather expensive! But, it’s fast! © W. Wahlster, DFKI
Conclusion The generation of virtual webpages is heavily based on ontological annotation The plan-based approach to generating virtual webpages guarantees high-level ontological annotations Ontological annotations are needed not only for information extraction agents but also for presentation agents Realization procedures and wrappers form an important part of the referential semantics of objects on the web Using presentation planning with temporal and spatial constraint processing the low-level media synchronization can be done automatically using SMIL annotations see www. dfki. de/~wahlster/semanticweb © W. Wahlster, DFKI
The Web. Persona and Virtual Webpages Team at DFKI Computer Science: Graphics Design: Elisabeth André Mathias Bauer Dietmar Dengler Alexander Kröner Marcus Meyer Jochen Müller Stephan Neurohr Gaby Paul Thomas Rist Wolfgang Wahlster Bernhard Kirsch Renato Orsini Peter Rist Cognitive Psychology: Susanne van Mulken © W. Wahlster, DFKI
Скачать презентацию Dagstuhl Seminar Semantics for the Web Dagstuhl 22
2d4bdae5d656c7ba55acbb9761d32f98.ppt