Third International Conference on Autonomous Agents 99 Seattle

Third International Conference on Autonomous Agents 99 Seattle, May 3. 1999 Personalized Interface Agents for Virtual Webpages Wolfgang Wahlster German Research Center for Artificial Intelligence, DFKI Gmb. H Stuhlsatzenhausweg 3 66123 Saarbruecken, Germany phone: (+49 681) 302 -5252/4162 fax: (+49 681) 302 -5341 e-mail: wahlster@dfki. de WWW: http: //www. dfki. de/~wahlster © W. Wahlster, DFKI

Outline What is a Virtual Webpage? What are Personalized Interface Agents? Using Life-Like Characters for the Personalization of User Interfaces The Combination of Retrieved and Generated Media Objects for the Generation of Virtual Webpages PET: A Persona-Enabling Toolkit Commercial Applications in Advanced Web. Commerce Psychological Effects of Personas Our Research Agenda for Agent-based User Interfaces Multiple Interface Agents Conclusion © W. Wahlster, DFKI

Three Generations of Web Sites First Generation Second Generation Interactive Web Sites Static Web Sites Third Generation Virtual Web Sites Java. Scripts and Applets Fossils cast in HTML Netbots, Information Extraction, Presentation Planners Database Access and Template-based Generation User Modeling, Machine Learning, Online Layout Dynamic Web Sites Adaptive Web Sites © W. Wahlster, DFKI

What is a Virtual Web Page? Virtual Memory, Virtual Relation, Virtual Reality. . . A Virtual Web Page l is generated on the fly as a combination of various media objects from multiple web sites or as a transformation of a real web page. l looks like a real web page, but is not persistently stored. l integrates generated and retrieved material in a coordinated way. l can be tailored to a particular user profile and adapted to a particular interaction context. l has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components. © W. Wahlster, DFKI

Ai. A: Information Integration for Virtual Webpages PAN Travel Agent Andi Car Route Planner Yahoo News Server Yahoo Weather Server Hotel Guide Gault Millau Restaurant Guide © W. Wahlster, DFKI

Virtual Webpage Retrieved from 5 Different Servers © W. Wahlster, DFKI

Virtual Webpage Augmented by Persona © W. Wahlster, DFKI

What are Agentbased User Interfaces? appear as life-like characters plan interactive behavior autonomously can initiate interaction ACTIVE understand the user‘s requests answer clarification questions allow mixed initiative dialogs REINTER- INTERFACE ACTIVE AGENTS ACTIVE respond immediately to interruptions criticism and clarification questions direct manipulation PROACTIVE anticipate the user's needs adopt the user's goals provide unsolicited comments © W. Wahlster, DFKI

Intelligent Web Services Consumer buys Information Goods Services Web Sites Netbot Intelligent Parallel Retrieval Information Extraction and Summarization Personalized Presentation Matchmaking Teleshopping Assistance Telemarketing Assistance Translation Services Data Mining Services Provider sells Information Goods Services Knowledge about: Usage Patterns User Models Consumer Profiles © W. Wahlster, DFKI

Enhancement of User Interfaces through Personalization System is able to flexibly tailor presentations to the individual user and the current situation. An animated character serves as “Alter Ego” of the presentation system. Personalized Presenters at DFKI © W. Wahlster, DFKI

Persona as a Personal Travel Consultant © W. Wahlster, DFKI

The Personal Picture Finder © W. Wahlster, DFKI

A Netbot for Portrait Photos: The Personal Picture Finder http: //finder. dfki. de: 7000 Portrait Photo Netbot: Personal Picture Finder Parallel Meta. Search of Webpages for WWW Parallel Search in Picture Archives Home Pages Extraction of Images Filtering of Logos, Graphics, . . . Knowledge Sources Applications: Journalism, Contact Preparation, Tracing Criminals © W. Wahlster, DFKI

Personalized Package Presentation by an Animated Agent Personalization = adaption of system behavior according to a user model Personalization = the “agent” appears as animated character which presents, explains and comments an offer, and to which the user can talk to. © W. Wahlster, DFKI

Video Character Presents the Interior of a Boeing 757 © W. Wahlster, DFKI

Reactive Behavior of the Persona Agent © W. Wahlster, DFKI

PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications Visual Appearances Behaviors Presentation Gestures Reactive Behaviors Idle-time actions Navigation actions Hand-drawn Cartoon Bitmaps Persona Server Auditory Characteristics Sound effects, auditory icons Voice: male, female Video Bitmaps Generated Bitmaps from 3 D-Models © W. Wahlster, DFKI

The frames of the visual appearance of persona can be cartoon-like images or video-frames which show real persons More than 200 cartoon frames were drawn by a professional artist. A real persona was filmed with a video-camera and the pictures were digitized with a frame grabber. © W. Wahlster, DFKI

The Persona Editor © W. Wahlster, DFKI

Context-Sensitive Decomposition of Persona Actions High-Level Persona Actions take-position (t 1 t 2 ) Context-Sensitive Expansion (including Navigation Actions) Decomposition into Uninterruptable Basic Postures Bitmaps point-to (t 3 t 4 ) move-to (t 1 t 2 ) r-stick-pointing (t 3 t 4 ) r-turn (t 1 t 21 ) r-hand-lift (t 3 t 31 ) r-step (t 21 t 22 ) f-turn (t 22 t 2 ) . . r-stick-expose (t 31 t 4 ) . . . © W. Wahlster, DFKI

PPP System Architecture Multimedia Generation Text Generator Graphics Generator Gesture Generator Animation Generator Music Generator Multimedia Server Production Acts Generated Material Presentation Planner (PREPLAN) KR & Reasoning (RAT) Presentation Acts Layout Manager Signals + Events Persona Server © W. Wahlster, DFKI

Extensions of the Representation Formalism Distinction between production and presentation acts (i. e. Persona- or display acts) Explicit representation of qualitative and quantitative constraints Production Act Presentation Act Introduce Create. Graphics S-Show S-Position Elaborate-Parts S-Wait Label S-Create. Window S-Depict S-Point S-Speak Qualitative constraints: Create-Graphics meets S-Show, . . . Metric constraints: 1 <= Duration S-Wait <= 1, . . . © W. Wahlster, DFKI

Presentation Strategies in PPP contain qualitative and metric constraints (cf. [André/Rist 97]) (define-plan-operator : HEADER (A 0 (INTRODUCE P A ? object ? window)) : INFERIORS ((A 1 (CREATE-GRAPHICS P A ? window ? object)) (A 2 (S-SHOW P A ? window ? object)) (A 3 (S-POSITION P A ? window)) (A 4 (S-WAIT P A)) (A 5 (ELABORATE-PARTS P A ? object ? window))) : QUALITATIVE ((A 1 (m) A 2) (A 3 (s) A 2) (A 3 (m) A 5) (A 5 (m) A 4) (A 4 (f) A 2)) : METRIC ((10 <= DUR A 2) (2 <= DUR A 4 <= 2)) : START A 1 : FINISH A 2) © W. Wahlster, DFKI

The Combination of Retrieved and Generated Media Objects for Virtual Webpages Multi-Domain Problem Specs Multiple Data Sources Information Structures Relations, Lists KR Terms NETBOT Retrieved Results Distributed Information Media Objects Texts, Sounds, Videos Pictures, Maps, Animations © W. Wahlster, DFKI

The Combination of Retrieved and Generated Media Objects for Virtual Webpages Information Structures Relations, Lists KR Terms Select Canned Media Objects Design New Media Objects Icons, Clip Art Frames, Sounds Graphics, Animation Text, Speech, Mimic Select & Design Retrieved Results Media Objects Texts, Sounds, Videos Pictures, Maps, Animations Coordinate Media Objects Temporal Synchronization Spatial Layout Transform Media Objects Clip, Convert, Abstract Zoom, Pan, Transition Effects Reuse & Transform © W. Wahlster, DFKI

Persona Presents an Automatically Designed Business Chart © W. Wahlster, DFKI

Virtual Webpage with Animation Effects Based on a Single GIF Image © W. Wahlster, DFKI

Transition Effects in a Series of Retrieved Pictures © W. Wahlster, DFKI

The Generation of Virtual Webpages with PAN and Ai. A Hotel Agent Address Map Agent Ai. A Pictures and Graphics Pieces of Text Trip Data Netbot PAN Presentation Planner Components of virtual Webpages Coordinates for Pointing Gestures Input for Speech Synthesis Icons for Hyperlinks Weather Agent Train & Flight Scheduling Agent Virtual Web Presentation Persona Server Constraintbased Online Layout Major Event Agent © W. Wahlster, DFKI

© W. Wahlster, DFKI

PET: Persona-Enabling Toolkit Objective: Enable non-professional computer users to populate their web pages with lifelike characters PET comes with: a set of characters and basic gestures an easy-to-learn Persona markup language Developer’s PET will include: a character design tool which enables users to build their own characters Technical Realization: Based on standard XML, Java. Script, Java and DHTML © W. Wahlster, DFKI

The Persona Markup Language Specification of Persona actions Specification of the character to be used Features: – XML-based – easy to learn © W. Wahlster, DFKI

Functional View of PET Bitmaps Webpage with Reference to Java Applet URL of Webpage with Persona Tag PET Application Server PET Parser PET Generator . . . . . . Persona Scripts waitscreen 4 gesture greet 0 0 null gesture laugh 0 0 null. . . Persona Engine Behavior Monitor Audio Data Event Handler Character Composer © W. Wahlster, DFKI

Tasks of PET-Parser Preprocessing (required for non-standard HTML pages) Identification of Persona Tags PET-Generator Replacement of Persona Tag with Java Applet Determination of parameters for Java Applet Realization of Persona active elements as Java. Script Generation of DHTML layers (transparency of Applet box) Generation of data, e. g. audio files for speech output Generation of scripts © W. Wahlster, DFKI

The Bidirectional Control Flow on Persona-Enabled Webpages Triggers actions of the Persona Mouse Clicks Mouse Movements Text Input Speech Input Menu Input Direct Manipulation Input Web Persona Triggers operations on elements of the webpage © W. Wahlster, DFKI

© W. Wahlster, DFKI

Persona Active Elements (PAE) l Active Images An active image starts a persona action when clicked. l Addressable Objects An addressable object is an object which can be addressed and manipulated by Persona via its name and its position. some HTML elements © W. Wahlster, DFKI

Use of a Life-like Character for Electronic Commerce © W. Wahlster, DFKI

Use of a Life-like Character for Electronic Commerce © W. Wahlster, DFKI

Personified Agents Increase the User's Trust in the System's Presentation Experimental evidence for effects of modality on the user's trust (van Mulken, 1999) The system gives recommendations, which turn out to be wrong in some cases. How much does a user trust the system's advice depending on the modality of a presentation? 1. 0 0. 8 0. 7 0. 6 0. 5 Self-animated Persona, Speech, Gesture, Facial Graphical Expression, Pointing Highlighting Text, Graphical Highlighting © W. Wahlster, DFKI

Impact of the modality of a Presentation on the User's Trustfulness Result: Conclusion: Persona > Speech > Text If the presentation is more human-like, recommendations are more readily followed For decision support systems tutoring systems recommendation systems virtual sales agents personified interface agents have a clear advantage: They increase the user's trust in the information presented by the system © W. Wahlster, DFKI

Sending Interface Agents to Clients: Plug-Ins or Applets? Plug-Ins Add features (character players) to browser Download triggered by user Requires disk space on client Unrestricted access to client Less appropriate for Web. Commerce, Guides Agents integrated in 3 D environments Appropriate for Entertainment Examples: Extempo's Jennifer James (Hayes-Roth et al. 98) PFMagic's virtual petz Applets Java animation code sent over the net Automatic loading Requires no disk space on client Restricted access to client Appropriate for Web. Commerce, Guides Agents integrated in 2 D environments Less appropriate for Entertainment Examples: DFKI's Web Persona (Müller et al. 98) ISI's Adele (Johnson et al 98) New in Ai. A/PAN: Balanced combination of Applets and Servelets Efficient distribution of client-side Java and server-side Java for driving the Interface Agent © W. Wahlster, DFKI

Smart. Kom: Intuitive Multimodal Interaction Project Budget: Project Duration: $ 34 M 4 years The Smart. Kom Consortium: Main Contractor Project Management Testbed Software Integration DFKI Saarbrücken Uinv. Of Munich Media. Interface Berkeley Dresden Univ. of Erlangen Saarbrücken Heidelberg DAIMLERCHRYSLER Aachen European Media Lab Ulm Univ. of Stuttgart Munich Stuttgart © W. Wahlster, DFKI

Smart. Kom-Public: A Multimodal Communication Booth Loudspeaker Room microphone Smartcard/ Credit Card for authentication and billing Face-tracking camera Virtual touchscreen protected against vandalism Multipoint video conferencing Docking station for PDA/Notebook/ Camcorder high speed and broad bandwidth Internet connectivity High-resolution scanner © W. Wahlster, DFKI

Smart. Kom-Mobile: A Handheld Communication Assistant GPS GSM for Telephone, Fax, Internet Connectivity Camera Wearable Compute Server Stylus-Activated Sketch Pad Microphone MOBILE Biosensor for Authentication & Emotional Feedback Loudspeaker Docking Station for Car PC © W. Wahlster, DFKI

Smart. Kom-Home/Office: A Versatile Agent-based Interface Speech. Mike Natural Gesture Recognition Virtual Touchscreen © W. Wahlster, DFKI

Our Research Agenda for Agent-based Interfaces (Wahlster, André, Rist, Müller, Graf etc; www. dfki. de/imedia) 1 2 3 Personalized Presentation Agents (limited user interaction) Personalized Interface Agents (full user interaction) WIP: 1989 -1993 PPP: 1994 -1996 Multiple Interface Agents (agent-agent and user interaction) Multiple Presentation Agents in one scene (eg. pros and cons) Multiple Role-Taking (eg. Travel Assistance vs. Comparison Shopper) Multiple Interface Agents (eg. human-computer, human-human interaction) Ai. A: 1997 -2000 © W. Wahlster, DFKI

Use of a Life-like Character for Electronic Commerce Digital Assistant Selector © W. Wahlster, DFKI

Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support User-Adaptive Search Planning . . . weighted propositions Multiple Decision Support Agents Spare parts for this car are rather expensive! But, it’s fast! © W. Wahlster, DFKI

Multiple Interface Agents l Multiple Presentation Agents l Convey different attitudes, beliefs and biases (e. g. proponent vs. opponent) l Generate argumentatives dialogs among a group of agents l Multiple Interface Agents l human and virtual agents in collaborative browsing environments l different dialog roles (e. g. moderator, advisor, sales agent) l Multiple Role-Taking of One Interface Agent l e. g. travel assistant vs. comparison shopper (use of a different outfit of the character to signal a role change) © W. Wahlster, DFKI

Multi-User Interaction with Multiple interface Agents for Ecommerce on a Single Web. TV © W. Wahlster, DFKI

Intelligent Interface Technology is a Prerequisite for Advanced Web. Commerce Virtual Web Pages Shopbots for Automated Comparison Shopping Text Analysis and Generation Multimedia Presentation Planning Information Extraction from HTML/XML Documents Advanced Web. Commerce User Modeling and Language Generation Coordinated Text & Graphics Planning Machine Translation Intuitive, Multilingual Access Multimodal Interfaces Robust Dialogue Understanding Advanced Speech Synthesis One-to-One Marketing Dialogue with Virtual Sales Agents © W. Wahlster, DFKI

Research on Personalized Interface Agents brings disparate subfields in the area of intelligent systems together Planning User Modeling Knowledge Representation Image Understanding Personalized Intelligent Natural Language Interface Web Agents Understanding Services Machine Learning Plan Recognition Information Multimodal Retrieval User Interfaces © W. Wahlster, DFKI

Conclusion ECommerce projects of DFKI have shown that research on personalized interface agents can be transferred to real world applications: Dekra (largest European organization of used car dealers): Fair. Car as an ECommerce platform with NL access and a comparison shopping agent for used cars Daimler. Chrysler: IKP for online user modelling in a one-to-one marketing system for Mercedes cars Otto/Shopping 24/Eddie Bauer (largest European mail order company): virtual sales agents for one-to-one marketing of fashion and computer hardware Lufthansa/Condor: direct marketing of charter flights © W. Wahlster, DFKI

Conclusion The generation of virtual webpages with personalized interface agents leads to innovative applications in: Electronic Commerce, Electronic TV Guides (EPG) Telelearning environments, Call Centers and Help Desks Two Research Challenges: Making the Interface agents sensitive to temporary limitations of the user‘s time and working memory capacity (cf. our READY project, Jameson et al. , p. 79 -85 in IUI 99 Proceedings) Making the agents instructible, so that they can learn from the user in a dialog, how to extract information in a new domain (cf. Mathias Bauer, Dietmar Dengler Tr. IAs: Trainable Information Assistants for Cooperative Problem Solving in Agents'99, on Tuesday) © W. Wahlster, DFKI

The Web. Persona and Virtual Webpages Team at DFKI Computer Science: Graphics Design: Elisabeth André Mathias Bauer Dietmar Dengler Alexander Kröner Marcus Meyer Jochen Müller Stephan Neurohr Gaby Paul Thomas Rist Wolfgang Wahlster Bernhard Kirsch Renato Orsini Peter Rist Cognitive Psychology: Susanne van Mulken © W. Wahlster, DFKI