ACAL Active Capture Automation Language Ana Ramírez

ACAL – Active Capture Automation Language Ana Ramírez Advisors: Marc Davis, Jen Mankoff GUIR 25 February 2004 UC Berkeley - Garage Cinema Research - Group for User Interface Research

Overview • What is Active Capture • Challenges in § design § Implementation • Support at toolkit level for § design § implementation 10/21/2003 2

Motivation • Systems that direct human actions § § Keep awake system Sports instruction (golf swing) Automated health screening Video Door locks • Interactions that control timing of interaction 10/21/2003 3

Active Capture Direction/ Cinematography Computer Vision/ Audition Active Capture Interaction Human. Computer Interaction Processing 10/21/2003 4

Active Capture Direction/ Cinematography Computer Vision/ Audition Active Capture Interaction Human. Computer Interaction Processing 10/21/2003 5

Active Capture Direction/ Cinematography Computer Vision/ Audition Active Capture Interaction Human. Computer Interaction Processing 10/21/2003 6

Active Capture Direction/ Cinematography Computer Vision/ Audition Active Capture Interaction Human. Computer Interaction Processing 10/21/2003 7

Active Capture Direction/ Cinematography Computer Vision/ Audition Active Capture Interaction Human. Computer Interaction Processing 10/21/2003 8

Implemented Applications See Video at: www. cs. berkeley. edu/~anar/presentations/Implemented. Apps. mpg 10/21/2003 9

Head Turn Recognizer • Uses § Gross motion detector § Eye detector • Looks for § § No eyes and No motion Motion Eyes No motion followed by • Uses mediation if something goes wrong. 10/21/2003 10

Exercise • Given raw materials § User actions § System actions § Recognizers system uses • Write down a representation of the interaction of the head turn recognizer. • Work in groups of two 10/21/2003 11

Designing Active Capture Applications • Describe path of “righteousness” • Describe what to do if something goes wrong (mediation) • Use good mediation techniques § Progressive assistance § Freshness § Graceful failure 10/21/2003 12

Head Turn Application • Path of “righteousness” is not obvious • Tedious to write • Expression of time flow cumbersome 10/21/2003 13

Challenges • Difficult to represent § Control process with feedback (mediation) § Timing § Strict and non strict ordering 10/21/2003 14

Goals of ACAL • In general § Natural to describe Active Capture applications. § Support mediation strategies § Include time flow primitives • Support brain storming § Visual “language” • Support rapid prototyping § Support for Wizard-of-Oz prototypes • Support implementation § Make path of “righteousness” apparent in code § Be able to prove an implementation will reach the “done” state. 10/21/2003 15

Current Status of ACAL • Visual “Language” § Started with Ka-Ping Yee in Marc Davis’ class on Multimedia Information in Spring 2003 • Toolkit level support § Main focus this semester • Support Wizard-of-Oz protoyping § Future work • Link visual language, toolkit and wizard-of-oz support together. § Future Work 10/21/2003 16

Current Research Areas • Design Guidelines § Jeff Heer, Nathan Good, Ana Ramirez, Marc Davis, Jen Mankoff. “Presiding Over Accidents: System Mediation of Human Action CHI’ 04 ” • Language Support § ACAL • New Application § “Say Cheese” 10/21/2003 17

Visual “Language” • Path of “righteousness” § § Observations Commands Capture Time Constraints 10/21/2003 18

Visual “Language” • Path of “righteousness” § § Observations Commands Capture Time Constraints 10/21/2003 19

Visual “Language” • Path of “righteousness” § § Observations Commands Capture Time Constraints 10/21/2003 20

Visual “Language” • Path of “righteousness” § § Observations Commands Capture Time Constraints 10/21/2003 21

Visual “Language” • Path of “righteousness” § § Observations Commands Capture Time Constraints 10/21/2003 22

Visual “Language” • Path of “righteousness” § § Observations Commands Capture Time Constraints 10/21/2003 23

Visual “Language” • Add mediation for case when actor is looking at camera before turn. 10/21/2003 24

Visual “Language” • Add freshness to mediation 10/21/2003 25

• Add progressive assistance 10/21/2003 26

Language Design Process Two key challenges: 1. Control-oriented vs. time-oriented representation 2. Absolute vs. relative time relationships 10/21/2003 27

Control vs. Time • State machines and procedural programs describe control flow well • . . . but they visualize time poorly • Timeline representation allows concurrency to be fully expressed • . . . but decisions and control flow don’t fit easily on a timeline 10/21/2003 28

Control vs. Time • Hybrid visual representation: timelines with flow arrows 10/21/2003 29

Absolute vs. Relative Time • Horizontal scale on timeline implies particular lengths of intervals • Problem: sometimes want ordering; sometimes want specific intervals • Solution: arrangement on timeline yields ordering; min/max specifiers constrain time intervals 10/21/2003 30

Flexibility in Ordering • Each point on a track specifies “true”, “false”, or “don’t care” waving speaking < 5 sec Example: doesn’t matter whether when waving or speaking begins or ends, as long as both happen at some point within a 5 -second period 10/21/2003 31

Lessons Learned • Difficult to balance between: § § Control-oriented vs. time-oriented representation Absolute vs. relative time relationships • Difficult to manage complexity • Important to be able to see path of “righteousness” • Easy to get mediation wrong. 10/21/2003 32

Future Work • Visual Language § Better solution to absolute vs. relative time relationships challenge § Better support for mediation strategies • Implementation support • “Say Cheese” • Automated health screening 10/21/2003 33

Questions anar@cs. berkeley. edu 10/21/2003 34

System Architecture 10/21/2003 35

ACAL Design Goals • Natural to describe Active Capture applications • Support key strategies for mediation § Progressive Assistance § Graceful Failure § Freshness • Include time flow primitives • Support brainstorming process 10/21/2003 36

ACAL • Toolkit level support for applications with: § Mediation § Complex timing § Rich media input and output 10/21/2003 37

Future Applications • “Say Cheese” • Folk Computing § Support remote, more frequent medical screening. 10/21/2003 38

Methodology • Theoretical § Active Capture Design Space § Mediation strategies / guidelines • Practical § Reverse engineer implemented applications § Design a new application 10/21/2003 39