Inventing Virtual Teachers and Therapists Promises Systems

Inventing Virtual Teachers and Therapists Promises, Systems & Challenges Ron Cole & the CSLR Virtual Human Research Group September 2, 2005 SPACE meeting; Leuven Belgium

Outline • TALK ONE: Vision, Rationale & Systems – The promise of virtual humans – Scientific rationale – System overviews and demos • TALK TWO: Inventing Virtual Humans – Technologies implemented to date – Technical challenges and missing science – Conclusions and recommendations

What is a Virtual Human? A Believable Computer Character • with personality and attitude • that engages users in natural face-to-face conversation • to produce great learning experiences

Marge

The Promise of Virtual Humans • Effective teachers, therapists, assistants – A virtual human is patient and tireless: learning can be more engaging, motivating, personal and effective • Accessible to nearly everyone, anywhere, anytime – Via multilingual natural dialog interaction on networked computers • Awesome benefits to individuals and society – Humans are expensive and often inaccessible, Virtual Humans are inexpensive • Research tools for acquiring missing knowledge; – they can be programmed to behave in predictable ways; whereas people are often guided by unconscious behaviors

Theoretical & Empirical Foundations The media equation: media = real life Cliff Nass & Byron Reeves “We have found that individuals’ interaction with computers are fundamentally social and natural, just like interactions in real life. ” “All these rules come from the world of interpersonal interaction, and from studies of how people interact with the real world. But all of them apply equally well to media. ” “The more a media technology is consistent with social and physical rules, the more enjoyable the technology will be to use. including feelings of accomplishment, competence, and empowerment. ”

The Persona Effect (Lester et al. , 1997) • Hypothesis: Interfaces that use voice and/or face to foster social agency produce more satisfactory and effective experiences. • Students form a social bond with virtual humans and are motivated to learn and succeed • Better experiences reported in hundreds of experiments – Students give top ratings to: “Do you think Marni is a good teacher? ” “Does Marni act like a real human teacher? ” and “How well does Marni help you learn to read? ” • Better outcomes supported by dozens of experiments – Several experiments show benefits of talking head compared to voice alone

The power of one-on-one tutoring • Benjamin Bloom (1982) posed the “two sigma challenge”: – Demonstrated a two sigma benefit of one-onone tutoring relative to classroom instruction • Meta-analysis of 100 s of tutoring studies confirms benefits of tutoring (Cohen, Kulik & Kulik, 2982)

Cognitive Theory of Multimedia Learning Mayer (2001) • Mayer examined how presentation of words and pictures affected learning in transfer tasks • Best learning occurs when voice is used to explain phenomena displayed simultaneously in pictures or animation • Atkinson demonstrated benefits of talking head relative to voice alone

Recipe for a Virtual Human System – Develop a deep understanding of the task • Theory, research and practice – Analyze and model the performance of human experts – Develop the system • In collaboration with experts and users (participatory design) • Design and test, design and test, test – Evaluate the system (formative evaluation / clinical trials) – Scale Up and Sustain the system

Virtual Teachers and Therapists • Theoretical & Empirical Rationale • Demos – Baldi teaches vocabulary to students with hearing loss – Marni teaches children to read – Marni conducts speech therapy for individuals with Parkinson disease and aphasia Research funded by grants from NSF and NIH

A Virtual Teacher for Students with Hearing Loss • 1998 -2001: Baldi teaches vocabulary to students at Tucker Maxon School – Rapid acquisition of vocabulary – >50% retention several months later – Dramatic improvements in speech production – Featured on national TV and NSF Home page

Foundations to Literacy A comprehensive, scientifically-based reading program designed to • Teach children to read and comprehend text • Through conversational interaction with Marni, a virtual teacher • That behaves like a sensitive and effective reading teacher

Cognitive theory & scientificallybased reading research Skilled reading is – Word recognition processes + comprehension processes – This is called the “Simple Model of Reading” (Gough et al. , 1996) • Word recognition processes – Alphabet, Phonological awareness, Encoding, Decoding, Sight words – Reading in context until fluent & automatic – Evidence-based pedagogy: SBRR (Rayner et al, 2001) • Comprehension processes – Train fluent and expressive reading – Train comprehension through “thinking questions”

Main Components of Ft. L • Foundational Skills Tutors – Teach underlying reading skills • Interactive Books – Teach fluent reading & comprehension • Managed Learning Environment – Enrolls students, tracks and displays progress, manages individual study plans for each student

Ft. L Status • Now in 50 Colorado classrooms – Formal assessment in 40 K-2 classrooms; 2 computers per classroom, 20 min per day per designated students – About 1/3 ESL students (one school has ALL native Spanish speaking students) – 10 Special Ed classrooms for remedial instruction of students with cognitive disabilities • Learning gains in kindergarten and first grade classrooms; kids love the program

LSVT Lee Silverman Voice Treatment “If only we could hear and understand her” -- family of Lee Silverman

Parkinson’s disease 1. 5 Million individuals US alone Over 6 million worldwide 89% have a speech or voice problem (Logemann et al. , 1978) 4% receive traditional speech therapy (Hartelius & Swenson, 1994; Oxtoby, 1982) 1990 Consensus: Speech treatment does not work (Sarno, 1968; Allan, 1970; Green, 1980; Aronson, 1990; Weiner & Singer, 1989)

Perceptual Characteristics of Speech Reduced loudness Hoarse voice quality Monotone Imprecise articulation Vocal tremor Some patients report volume, hoarse voice or monotone as the first PD symptom (Aronson, 1990)

“If you don’t talk loud enough, people stop listening” -Individual with Parkinson Disease Boston, May 1996

LOUD SOFT

To a patient……major life impact “My voice is alive again” “I can talk to my grandchildren!” “I feel like my old self” “I am confident I can communicate!”

Ramig et al. , 2001; J Neurol, Neurosurgery, Psychiatry SPL Rainbow (50 cm) 75 70 65 60 -2 0 2 4 6 8 10 12 14 16 18 20 22 24 Months LSVT R

Unexpected outcomes: System-wide spread of effect Benefits to Articulation Speech Rate Speech Motor Stability Swallowing Face PET (Spielman, et al. 2002; El-Sharkawi, 2002; Spielman et al. , 2003; Kleinow et al. , 2001; Liotti et al. , 2003)

To a patient……major life impact “My voice is alive again” “I can talk to my grandchildren!” “I feel like my old self” “I am confident I can communicate!”

Phonation Task - PD N=5 Pre-LSVT SMA R Put +60 DLPF 9 + +4 34 +4 +34 R a Ins +10 Post-LSVT L +60 R +34 +4 +34 +10 z-score -4 ± 2. 25 +4

LSVT® Applications • • Parkinson Plus (Countryman et al. , 1994) Post Surgery, Fetal cell (Countryman, et al. , 1993) Stroke (Fox et al, 2002; Will et al. , 2002) Multiple Sclerosis (Sapir et al. , 2001) Ataxia (Sapir et al. , 2003) Cerebral palsy (Fox, 2002) Down Syndrome (Robinson et al. , 2004) Aging (Ramig et al. , 2001)

LSVTVT-Loud AH LSVTVT-Loud Phrases LSVTVT– Hi-Lo pitch LSVTVT-Read Aloud

LSVT VT Status • System being tested on three individuals with Parkinson disease • Patients use system during 10 of 16 1 hour sessions using system • Patients enjoy using the system • STATUS: Clinical trials begin Sept 14 05

Virtual Therapists for Aphasia The three systems described next were based on speech and language treatments developed for individuals with Aphasia • C-COSTA (Developed by Leora Cherney, Chicago Rehabilitation Institute • ORLA (Leora Cherney) • TUF-T (Developed by Cynthia Thompson, Northwestern University

C COSTA Computerized Oral Scripts for Teach Aphasia • Develop conversational scripts personally relevant to the individual with aphasia – A sequence of sentences that a person typically speaks in routine communication situations • Ordering pizza over the phone • Making a doctor’s appointment • Implement and evaluate the computerized intervention relative to human therapists • STATUS: 5 subjects tested

Oral Reading for Language in Aphasia ORLA • ORLA is a speech-language treatment protocol for patients with aphasia (Cherney et al. , 1986), • A multi-modality stimulation approach that involves • listening to a sentence, tapping along with the rhythm of the sentence, repeated practice saying the sentence in unison with the clinician and then independently. • Studies indicate improvements in oral expression, auditory comprehension, and written expression (Cherney et al. , 1986). • STATUS: 5 subjects tested

Treatment of Underlying Forms (TUF) for Aphasia by Cynthia Thompson at Northwestern University • Individuals with Broca’s aphasia have difficulty comprehending and producing sentences, particularly sentences with complex syntax. • Training simple sentence types fails to generalize to untrained sentence types and contexts. • Research with TUF demonstrated generalization occurs to sentences that involve similar movement properties. – For example, “It was the thief who chased the artist” results in improved production and comprehension of wh-questions such as “Who did the thief chase? ” • STATUS: Testing Prototype