Uncertainty Corpus Resource to Study User Affect in

Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems Kate Forbes-Riley, Diane Litman, Scott Silliman, Amruta Purandare University of Pittsburgh, PA, USA

Outline u Introduction u WOZ-TUT System u Experimental u Uncertainty u Uses Design Corpus Description of the Uncertainty Corpus 2

Overview: Towards Affect-Adaptive Spoken Dialogue Systems u Automatic Detection: promising across affective states and applications, e. g. (Craig et al. , 2006; Litman & Forbes-Riley, 2006; Lee & Narayanan, 2005; Vidrascu & Devillers, 2005; Batliner et al. , 2003) u Larger goal is automatic adaptation, but results are sparser u More public affect-annotated corpora of human-computer dialogues could help, e. g. HUMAINE project u SYMPAFLY, u AIBO (Batliner et al. , 2004) (German) u Communicator (Walker et al. , 2001; Ang et al. , 2002) (English) Uncertainty Corpus: u new complex domain: spoken dialogue tutoring u new affect annotation: student uncertainty 3

Uncertainty Corpus Collection: WOZ-TUT System u WOZ-TUT: Adaptive Wizard of OZ Tutoring System u modified version of ITSPOKE (Litman & Silliman, 2004) u system responses based on combined uncertainty and correctness u human recognizes speech, annotates uncertainty and correctness u Why uncertainty? u Most frequent in ITSPOKE corpora (Forbes-Riley & Litman, 2007) u Most systems respond only to correctness, but literature suggests uncertain and incorrect answers signal learning impasses u What uncertainty adaptation? u Treating uncertain+correct answers as incorrect should provide additional knowledge to bridge impasse 4

WOZ-TUT Screenshot 5

Experimental Design u 3 Conditions: used parameterized WOZ-TUT dialogue manager u Experimental: treat all uncertain+correct turns as incorrect u First Control: ignore uncertainty (logged) u Second Control: ignore uncertainty (logged), but treated a percentage of random correct answers as incorrect 6

Corpus Excerpts u TUTOR: What will the velocity of the object be a second after that (where the initial velocity is 9. 8 m/s and the acceleration is 9. 8 m/s 2)? u STUDENT: Nineteen point six meters per second? ? [uncertain+correct] u TUTOR in First Control Condition moves on: Good. So at every point in time during the fall of the man and his keys, how do their velocities compare with each other? u TUTOR in Experimental Condition remediates: Okay. As we have seen, if a falling object has an acceleration of 9. 8 m/s 2, its velocity changes by 9. 8 m/s every second. So if a second after it began falling its velocity is 9. 8 m/s, a second later its velocity will be 9. 8 m/s + 9. 8 m/s = 19. 6 m/s. So what will its velocity be a second after that? 7

Experimental Procedure u 60 subjects randomly assigned to 3 conditions (gender-balanced) u Native English speakers with no college physics u Procedure: 1) read background material, 2) took pretest, 3) worked training problem with WOZTUT, 4) took posttest, 5) worked isomorphic test problem with non-adaptive WOZ-TUT 8

Corpus Description Student Total Turns Total Uncertain Turns Total Words Average Words per Turn Tutor 2171 796 13533 6. 23 2531 111829 44. 20 120 dialogues from 60 students (. ogg format) u 20 total hours of dialogue u Student turns manually transcribed, including disfluency and non -syntactic question annotation u Tutor turns and Wizard annotations in log files u 9

Student Answer Attributes Training Problem Ave Turns Ave Correct Turns Ave Uncertain+Correct Turns Ave Adapted-To Turns Ave Uncertain+Correct and Adapted-To Turns u EXP 20. 65 13. 80 9. 95 4. 75 100% CTRL 1 CTRL 2 18. 60 19. 75 12. 55 14. 20 8. 60 11. 15 3. 75 6. 10 0 3. 65 0% 36% One-way ANOVAs showed no significant differences: u number of correct, uncertain, or uncertain+correct turns u number adapted-to turns (EXP vs CTRL 2) 10

Uses of the Uncertainty Corpus I u Compare student performance across conditions to isolate impact of uncertainty adaptation u No significant differences in learning u We are comparing dialogue-based metrics in the isomorphic test problem (Forbes-Riley, Litman and Rotaru, 2008) Isomorphic Test Problem Ave Turns Ave Correct Turns Ave Uncertain Turns EXP 16. 50 14. 60 3. 30 CTRL 1 CTRL 2 16. 80 16. 25 14. 35 14. 10 3. 15 3. 65 - Feedback confound identified and rectified in larger follow-on study 11

Uses of the Uncertainty Corpus II u Resource for analyzing linguistic features of naturallyoccurring user affect in human-computer dialogue u Models built from elicited emotions generally transfer poorly to naturally-occurring dialogue (Cowie and Cornelius, 2003; Batliner et al. , 2003) u Uncertainty Corpus provides a new resource for modeling naturally-occurring dialogue u Large number of features in speech, transcript, log files 12

Summary and Current Directions u The Uncertainty Corpus is a collection of tutorial dialogues between students and an adaptive Wizard-of-Oz spoken dialogue system u Corpus (speech, transcripts, uncertainty and correctness annotations) publicly available by request through the Pittsburgh Science of Learning Center: https: //learnlab. web. cmu. edu/datashop/index. jsp u Follow-on experiments and corpora u Larger WOZ study just completed, with learning results! u Fully automated study to begin Fall 2008 13

Thank You! Questions? Further Information? web search: ITSPOKE or PSLC 14