Automated Generation of Visual Discourse Michelle X Zhou

Automated Generation of Visual Discourse Michelle X. Zhou Department of Computer Science Columbia University New York, NY 10027 Supported in part by DARPA Contract DAAL 01 -94 -K-0119, New York State Science and Technology Foundation, NSF Grant ECD 88 -11111, and ONR Contract N 00014 -97 -1 -0838

Automated Visual Design Problem M Designing effective visual presentations is difficult and costly M Designing customized visual presentations in a timely manner is more difficult Approach Develop computer techniques to automate visual design process 2

Automated Visual Design Input ? ? ? 3

Previous Work l Single displays APT Mackinlay 86 SAGE Roth & Mattis 91 ANDD Marks 91 l discrete A series of displays APEX Feiner 85 IBIS Seligmann 91 WIP Andre et al. 93 . . . 4

Our Goal: Visual Discourse Design A series of connected displays Open Rotate Scale Move. . . Rotate Scale+Move+. . . Open 5

Thesis. . . Thesis Work IMPROVISE Design Foundation • Computer network management System Design • Patient medical record summary • Knowledge base • Inference engine • Visual realizer • Interaction handler • Data characterization • Visual task hierarchy • Presentation design language • Inference paradigm Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach 6

Thesis. . . Thesis Work IMPROVISE Design Foundation • Computer network management System Design • Patient medical record summary • Knowledge base • Inference engine • Visual realizer • Interaction handler • Data characterization • Visual task hierarchy • Presentation design language • Inference paradigm Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach 7

Visual Discourse Modeling l Coherence Continuity Consistency Unity l Versatility Wide range of information Wide variety of visual media/techniques l Interactivity Interruptible Responsive 8

Thesis. . . Thesis Work IMPROVISE • Computer network Design Foundation management System Design • Patient medical record summary • Knowledge base • Inference engine • Visual realizer • Interaction handler • Data characterization • Visual task hierarchy • Presentation design language • Inference paradigm Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach 9

Design Foundation: Design Process Input Output Presentation Data Presentation Context Presentation Intents + Design Engine Design Knowledge 10

Design Foundation Input Presentation Data Presentation Context Presentation Intents Output Data Characterization + Design Engine Design Knowledge 11

Data Characterization Goal Data Visual Elements …. Previous Work …. . Presentation-related data properties Characterizing Quantitative data Mackinlay 86 Roth & Mattis 90 Characterizing Qualitative data Arens et al. 93 12

Our Approach: Characterizing Heterogeneous Data …. …. . Presentation-related data properties Data Dimensions Domain (Jones (is-a PATIENT) Type (type ATOMIC) Attribute (property (Form …). . . ) Relation (rel …). . . ) Role Sense (role LOCATE) (sense SYMBOL)) 13

Type Domain Data Characterization Taxonomy Atomic Composite Entity Concept Measurement Event Attributes Form Material Location Transience Importance Composite Attributes Relation FD Constituency Attribute Enumeration Role Sense Label List Plot Symbol Portrait Ordering Scalability Continuity Associate Background Categorize Cluster Correlate. . . 14

Design Foundation Input Presentation Data Presentation Context Presentation Intents Output Data Characterization Context Modeling + Design Engine Design Knowledge 15

Context Modeling Goal Context Information Visual Techniques Previous Work Situation space Friedell 83 Display categories Mackinlay 86 Our Approach Audience Occasion Environment 16

Design Foundation Input Presentation Data Presentation Context Presentation Intents Output Data Characterization Context Modeling Intent Modeling + Design Engine Design Knowledge 17

Intent Modeling Goal Presentation Intents Visual Techniques Previous Work Presentation Intents Wehrend & Lewis 90 Casner 91 Marks 91 Roth & Mattis 91 Ignatius & Senay 94 Visual Techniques Hunter 87 Levin 87 Maybury 93 Seligmann 93 Sutcliffe et al. 94 18

Our Approach: Visual Task Characterization Achieve Presentation Intents Imply Visual Tasks (Visual Effects) Achieve Imply Search Focus Elaborate Abstract Visual Techniques Enlarge Highlight Zoom 19

Background Associate Colocate Connect Unite Attach Categorize Mark. Distribute Cluster Outline Mark. Distribute Correlate Plot Mark. Compose Compare Differentiate Intersect Distinguish Mark. Distribute Isolate Emphasize Focus Isolate Reinforce Generalize Merge Identify Name Portray Individualize Profile Locate Position Situate Pinpoint Outline Rank Time Reveal Expose Itemize Specify Separate Switch Encode Label Symbolize Quantify Iconify Portray Tabulate Plot Trace Structure Map Visual Task Taxonomy 20

Design Foundation Input Presentation Data Presentation Context Presentation Intents Output Data Characterization Context Modeling + Design Engine Design Knowledge Modeling Intent Modeling Design Knowledge 21

Design Knowledge Modeling Goal Computational representation of design knowledge Previous Work Visual formalisms Marks 91; Lohse et al. 94 Visual techniques Friedell 84; Seligmann 93; Keller&Keller 94 Visual design principles Winn&Holliday 82; Bertin 83; Mullet&Sano 95 Mackinlay 86; Ignatius&Senay 94; Tufte 83, 90, 97 22

Our Approach: Presentation Design Language l Visual objects Represent various visual formalisms l Visual techniques l Visual design principles Compose/manipulate visual objects Guide the visual object composition and manipulation 23

Visual Object Representation l Syntax red Patterns/compositions l Semantics human heart Meanings l + Pragmatics Specific meanings I New York love 24

Visual Object Hierarchy Discourse Static/Dynamic Tables Visual. Charts Frames Diagrams Visual Lexicon (atomic objects only) Visual Structures Boston Visual Unities Shape/Color/Size/Orientation/. . . Visual Primitives 25

Visual Techniques l Categorized by function Formation Transformation Camera l Categorized by usage Primitive Composite Examples (Design. Table. Chart (is-a Formation. Tech) (operands ? data-obj ? table). . . ) (Move (is-a Transformation. Tech) (operands ? obj) (source) (destination) (start. Time) (end. Time)) (Set. Camera (is-a Camera. Tech) (operand ? camera) (position) (orientation). . . ) 26

Visual Design Principles l Expressiveness rules Comprehensiveness & distinctiveness Generality & discreteness à Integrity à à l Effectiveness rules x 27

Visual Design Principles l Expressiveness rules Comprehensiveness & distinctiveness Generality & discreteness à Integrity à à l Effectiveness rules Accuracy & clarity Appropriateness à Immediacy à Continuity à Consistency à Unity à à 28

Design Foundation Output Input Presentation Data Presentation Context Presentation Intents Data Characterization Context Modeling + Inference Modeling Design Engine Design Knowledge Modeling Intent Modeling Design Knowledge 30

Inference Modeling Goal Flexible and efficient design method Previous Work Constructive design Mackinlay 86; Roth & Mattis 90; Marks 91 Parametric design Zdybel et al. 81; Robertson 91; Ignatius&Senay 94 31

Our Approach: Hybrid Inference Paradigm l Constructive synthesis Planning presentations from scratch l Parametric synthesis Efficiently create visual models for atomic data objects A least-commitment constraint-based approach 32

Planning Elements & Features Visual Discourse Collection of frames Plan Collection of actions Visual Techniques Visual/Domain Objects Design Principles Operators Objects Constraints Practical hierarchical-decomposition partial-order planning DPOCL (Young et al. 95) + SIPE (Wilkins 88) à PREVISE (Zhou 97) 33

Extended Features DPOCL (Young et al. 95) + SIPE (Wilkins 88) Top-down strategy Action decomposition ¹ PREVISE (Zhou 97) +Object decomposition Variables +Enriched Variables Temporal reasoning +Spatial reasoning +Dynamic numerical constraints +Domain heuristics 34

Thesis. . . Thesis Work IMPROVISE • Computer network Design Foundation management System Design • Patient medical record summary • Knowledge base • Inference engine • Visual realizer • Interaction handler • Data characterization • Visual task hierarchy • Presentation design language • Inference paradigm Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach 36

System Framework Knowledge Base Domain data l Situation data l Visual data l Meta data l Inference Engine PREVISE l Visual lexical chooser l Interaction Handler Present interactivity l Control interactivity l Visual Realizer l Interface language design 37

Thesis. . . Thesis Work IMPROVISE • Computer network Design Foundation management System Design • Patient medical record summary • Knowledge base • Inference engine • Visual realizer • Interaction handler • Data characterization • Visual task hierarchy • Presentation design language • Inference paradigm Visual Discourse Modeling • Coherence • Versatility • Interactivity System Development General Approach 38

IMPROVISE (Illustrative Metaphor PROduction in VISual Environments) l Stand-alone graphics system Computer network management l Graphics generator in a multimedia presentation system Healthcare 39

Architecture Messenger Task Analyzer Interaction Handler Presentation Planner Content Planner Designer Chooser Stylist Organizer Coordinator Knowledge Base Converter Renderer 40

Example: Present Patient Information to Nurse 41

Sample Visual Discourse Representation (Discourse 1 (is-a VISUAL-DISCOURSE) (frames [Frame 1] [Highlight 2]. . . )) (Frame 1 (is-a VISUAL-FRAME) (frame. Elements [Struct. Diag 1]) (start. Time 0. 0) (end. Time -1. 0)) (Struct. Diag 1 (is-a STRUCTURE-DIAGRAM) (heading [Table 1 -Demo) (core [Unity 1 -body]) (elements [Unity 2 -device 1]. . . ) (Highlight 2 (is-a Highlight) (operands [Table 1 -demo]) (style MARKER) (color. . . ) (start. Time 5. 0) (end. Time 8. 0). . . ) 42

Input: Presentation Data (Jones-info (is-a CONCEPT) (type COMPOSITE) (convey [Jones] [Jones-demo] [Swan-Ganz]. . . ) (Jones (is-a PATIENT) (type ATOMIC) (role LOCATE) (sense SYMBOL). . . ) (Swan-Ganz (is-a DEVICE) (type ATOMIC) (convey [sw-name] [sw-loc] [sw-content]) (sense SYMBOL). . . ) (Jones-demo (is-a CONCEPT) (type COMPOSITE) (convey [Jones-name]. . . ) (role Identify) (sense LIST). . . ) (Jones-name (is-a NAME-ATTR) (value “S. Jones”) (role IDENTIFY) (sense LABEL). . . ) 43

Input: Presentation Context (Context-info (audience (identity NURSE). . . ) (occasion (presentation ON-LINE) (medium VISUAL-SPEECH). . . ) (environment (display (color COLOR) (size. . . ) (platform (cpu SGI R 4400). . . ))) 44

Input: Presentation Intent & Visual Task Formation Intent: Summarize Structure Identify Locate Symbolize Name Itemize Name Associate 45

Fulfill Visual Tasks Intent: Summarize Structure Design. Structure. Diagram Identify Locate Symbolize Itemize Name Name Associate 46

Structure Diagram Patient Info 47

Fulfill Visual Tasks Intent: Summarize Structure Identify Locate Symbolize Design. Structure. Diagram Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Itemize Name Name Associate 48

Heading Identify Demographics Structure Diagram Device Element Symbolize Core Locate Patient Element Symbolize Device Element Symbolize 49

Fulfill Visual Tasks Intent: Summarize Structure Design. Structure. Diagram Identify Locate Symbolize Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Itemize Name Design. Table Design. Vis. Rep Name Associate 50

Heading (Table) Structure Diagram Device Element Symbolize Core Locate Patient Element Symbolize Device Element Symbolize 51

Design. Table Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Design. Table Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Design. Vis. Rep Design. Word Design. Word Design. Word Design. Word Design. Word 52. . .

Heading (Table) Structure Diagram Device Element Symbolize Core (Unity) Element Symbolize Device Element Symbolize 53

Heading (Table) Structure Diagram Name Associate Element (Unity) Core (Unity) Element (Unity) 54

spacing 1 spacing 2 left alignment spacing min. line length Appropriateness Interactivity Clarity Consistency STM (Gleicher 95) 55

Contributions l l Definition and modeling of visual discourse Methodologies for automated visual discourse design Approaches to practical system design Design and implementation of IMPROVISE 56

Future Work l Conversational capability l Explanation capability l Adaptive capability 57

Thesis Committee MAGIC Group Project Students & Graphics Group Mike Gleicher Thank You 58

59

Exploring Network Entity (Node 1 (is-a NETWORK-NODE) (type COMPOSITE) (components [node-ports]) (convey [node 1 -name] [node 1 -loc]) (sense SYMBOL). . . ) (Link 1 (is-a PHY-CONNECTION) (type COMPOSITE) (components [vpath-segments]) (sense SYMBOL) (source [node 1]) (destination [node 2]). . . ) 60

Exploring Network Link (Task (Elaborate (link 23))) 61

Exploring Network Link (Task (Elaborate (link 23))) (Elaborate (link 23)) (Focus (link 23)) (Expose (link 23)) 62

Exploring Network Entity Composite Tech: Focus (? obj. X) Preference: low-intersection(? obj. X) Decomposition 1: Enlarge(? obj. X) Preference: reduce-intersection(? obj. X) Decomposition 3: Separate(? obj. X, ? rest) Decomposition 2: Highlight (? obj. X) 63

Exploring Network Entity Composite Tech: Expose (? obj. X) Preference: Openable (? obj. X) Decomposition 1: Open (? obj. X) Preference: ok-show-all (? obj. X) Decomposition 3: Set. Transparency (? obj. X) Decomposition 2: Cut. Away (? obj. X) 64

Exploring Network Link Focus(link 23) Expose Decomposition 1: Focus Decomposition 3: Separate (link 23, network 1) Open(link 23) Separate (link 23, network 1) Scale (link 23, node 1, node 5) Display (link 23 -vpath. Segs) ink ausal L C Display Decomposition: Display (link 23 -Seg 1) Display (relation (link 23 -Seg 1, Capacity)). . . Separate Decomposition: Open(link 23) Display Decomposition: Align (network 1) Display (link 23 -Seg 1) Move (link 23, node 1, node 5) Display (relation (link 23 -Seg 1, Capacity)) 65 Scale (link 23, node 1, node 5). . .

Exploring Network Link Complete Plan: Align (network 1) Move (link 23, node 1, node 5) Scale (link 23, node 1, node 5) Display (link 23 -Seg 1) Open(link 23). . . Display (relation (link 23 -Seg 1, Capacity)). . . Time 66

Exploring Network Link Align Move + Scale +. . . Open 67