How much data is enough Generating reliable

How much data is enough? – Generating reliable policies w/MDP’s Joel Tetreault University of Pittsburgh LRDC July 14, 2006

Problem n Problems with designing spoken dialogue systems: q q q n How to handle noisy data or miscommunications? Hand-tailoring policies for complex dialogues? What features to use? Previous work used machine learning to improve the dialogue manager of spoken dialogue systems [Singh et al. , ‘ 02; Walker, ‘ 00; Henderson et al. , ‘ 05] n However, very little empirical work [Paek et al. , ‘ 05; Frampton ‘ 05] on comparing the utility of adding specialized features to construct a better dialogue state

Goal n n How does one choose which features best contribute to a better model of dialogue state? Goal: show the comparative utility of adding three different features to a dialogue state 4 features: concept repetition, frustration, student performance, student moves All are important to tutoring systems, but also are important to dialogue systems in general

Previous Work n n n In complex domains, annotation and testing is timeconsuming so it is important to properly choose best features beforehand Developed a methodology for using Reinforcement Learning to determine whether adding complex features to a dialogue state will beneficially alter policies [Tetreault & Litman, EACL ’ 06] Extensions: q q Methodology to determine which features are the best Also show our results generalize over different action choices (feedback vs. questions)

Outline n n Markov Decision Processes (MDP) MDP Instantiation Experimental Method Results q q Policies Feature Comparison

Markov Decision Processes n n What is the best action an agent should take at any state to maximize reward at the end? MDP Input: q q q States Actions Reward Function

MDP Output n n Policy: optimal action for system to take in each state Calculated using policy iteration which depends on: q q n Propagating final reward to each state the probabilities of getting from one state to the next given a certain action Additional output: V-value: the worth of each state

MDP’s in Spoken Dialogue MDP works offline MDP Training data Policy Dialogue System User Simulator Human User Interactions work online

ITSPOKE Corpus n 100 dialogues with ITSPOKE spoken dialogue tutoring system [Litman et al. ’ 04] q q q n All possible dialogue paths were authored by physics experts Dialogues informally follow question-answer format 60 turns per dialogue on average Each student session has 5 dialogues bookended by a pretest and posttest to calculate how much student learned

Corpus Annotations n Manual annotations: q q Tutor Moves (similar to Dialog Acts) [Forbes-Riley et al. , ’ 05] Student Frustration and Certainty [Litman et al. ’ 04] [Liscombe et al. ’ 05] n Automated annotations: q q q Correctness (based on student’s response to last question) Concept Repetition (whether a concept is repeated) %Correctness (past performance)

MDP State Features Values Correctness Correct (C), Incorrect (I) Certainty Certain (cer), Neutral (neu), Uncertain (unc) Concept Repetition New Concept (0), Repeated (R) Frustration Frustrated (F) , Neutral (N) % Correctness 50 -100% (H)igh, 0 -49% (L)ow

MDP Action Choices Action Example Turn SAQ (Short Answer Question) “What is the direction of that force relative to your fist? ” CAQ (Complex Answer Question) “What is the definition of Newton’s Second Law? ” Mix “If it doesn’t hit the center of the pool what do you know about the magnitude of its displacement from the center of the pool when it lands? Can it be zero? Can it be nonzero? No. Q “So you can compare it to my response…”

MDP Reward Function n Reward Function: use normalized learning gain to do a median split on corpus: n 10 students are “high learners” and the other 10 are “low learners” High learner dialogues had a final state with a reward of +100, low learners had one of -100 n

Methodology n Construct MDP’s to test the inclusion of new state features to a baseline: q q q n Develop baseline state and policy Add a feature to baseline and compare polices A feature is deemed important if adding it results in a change in policy from a baseline policy given 3 metrics: n # of Policy Differences (Diff’s) n %Policy Change (%PC) n Expected Cumulative Reward (ECR) For each MDP: verify policies are reliable (V-value convergence)

Hypothetical Policy Change Example 0 Diffs B 1 State 5 Diffs Policy B 1+Certainty State +Cert 1 Policy +Cert 2 Policy 1 [C] CAQ [C, Cer] [C, Neu] [C, Unc] CAQ CAQ Mix 2 [I] SAQ [I, Cer] [I, Neu] [I, Unc] SAQ SAQ Mix CAQ Mix

Tests B 2+ +Concept B 1+ Correctness Baseline 1 +Certainty +Frustration Baseline 2 +%Correct

Baseline n n Actions: {SAQ, CAQ, Mix, No. Q} Baseline State: {Correctness} Baseline network SAQ|CAQ|Mix|No. Q [C] [I] FINAL

Baseline 1 Policies # State Size Policy 1 [C] 1308 No. Q 2 [I] 872 Mix n Trend: if you only have student correctness as a model of student state, give a hint or other state act to the student, otherwise give a Mix of complex and short answer questions

But are our policies reliable? n n n Best way to test is to run real experiments with human users with new dialogue manager, but that is months of work Our tact: check if our corpus is large enough to develop reliable policies by seeing if V-values converge as we add more data to corpus Method: run MDP on subsets of our corpus (incrementally add a student (5 dialogues) to data, and rerun MDP on each subset)

Baseline Convergence Plot

Methodology: Adding more Features n n n Create more complicated baseline by adding certainty feature (new baseline = B 2) Add other 4 features (concept repetition, frustration, performance, student move) individually to new baseline Check V-value and policy convergence Analyze policy changes Use Feature Comparison Metrics to determine the relative utility of the three features

Tests B 2+ +Concept B 1+ Correctness Baseline 1 +Certainty +Frustration Baseline 2 +%Correct

Certainty n n Previous work (Bhatt et al. , ’ 04) has shown the importance of certainty in ITS A student who is certain and correct, may require a harder question since he or she is doing well, but one that is correct but showing some doubt is a sign they are becoming confused, give an easier question

B 2: Baseline + Certainty Policies B 1 State Policy B 1+Certainty State +Certainty Policy 1 [C] No. Q [C, Cer] [C, Neu] [C, Unc] Mix SAQ Mix 2 [I] Mix [I, Cer] [I, Neu] [I, Unc] Mix No. Q Mix Trend: if neutral, give SAQ or No. Q, else give Mix

Baseline 2 Convergence Plots

Baseline 2 Diff Plots Diff: For each subset corpus, compare policy with policy generated with full corpus

Tests B 2+ +Concept B 1+ Correctness Baseline 1 +Certainty +Frustration Baseline 2 +%Correct

Feature Comparison (3 metrics) n # Diff’s q q n Number of new states whose policies differ from the original Insensitive to how frequently a state occurs % Policy Change (%P. C. ) q Take into account the frequency of each state-action sequence

Feature Comparison n Expected Cumulative Reward (E. C. R. ) q q q One issue with %P. C. is that frequently occurring states have low V-values and thus may bias the score Use the expected value of being at the start of the dialogue to compare features ECR = average V-value of all start states

Feature Comparison Results State Feature E. C. R 10 82. 2% 43. 21 Concept Repetition 10 80. 2% 39. 52 Frustration 8 66. 4% 31. 30 Percent Correctness n %P. C. Student Move n #Diff’s 4 44. 3% 28. 47 Trend of SMove > Concept Repetition > Frustration > Percent Correctness stays the same over all three metrics Baseline: Also tested the effects of a binary random feature n n If enough data, a random feature should not alter policies Average diff of 5. 1

How reliable are policies? Frustration Concept Possible data size is small and with increased data we may see more fluctuations

Confidence Bounds n n n Hypothesis: instead of looking at the V-values and policy differences directly, look at the confidence bounds of each V-value As data increases, confidence of V-value should shrink to reflect a better model of the world Additionally, the policies should converge as well

Confidence Bounds n n CB’s can also be used to distinguish how much better an additional state feature is over a baseline state space That is, if the lower bound of a new state space is greater than the upper bound of the baseline state space

Crossover Example More complicated Model ECR Baseline Data

Confidence Bounds: App #2 n Automatic model switching q n If you know a model, at it’s worst (ie. It’s lower bound is better than another model’s upper bound) then you can automatically switch to the more complicated model Good for online RL applications

Confidence Bound Methodology n For each data slice, calculate upper and lower bounds on the V-value q Take transition matrix for slice and sample from each row using direch. statistical formula 1000 times n n q Run MDP on all 1000 transition matrices to get a range of ECR’s n q do this b/c real world data is not exactly approximating what data is like in the real world, but may be close So get 1000 new transition matrices that are all very similar Rows with not a lot of data are very volatile so expect large range of ECR’s, but as data increases, transition matrices should stabilize such that most of the new matrices produce similar policies and values as the original Take upper and lower bounds at 2. 5% percentile

Experiment n Original action/state setup did not show anything promising q q n State/action space too large for data? Not best MDP instantiation Looked at a variety of MDP configurations q q Refined reward metric Adding discourse segmentation

+essay Instantiation with ’ 03+’ 05 data

+essay Baseline 1

+essay Baseline 2

+essay B 2+SMove

Feature Comparison Results State Feature #Diff’s %P. C. E. C. R Student Move 5 43. 4% 49. 17 Concept Repetition 3 25. 5% 42. 56 Frustration 1 0. 03% 32. 99 Percent Correctness 3 11. 19% 28. 50 n Reduced state size: Certainty = {Cert+Neutral, Uncert} n Trend that SMove and Concept Repetition are the best features B 2 ECR = 31. 92 n

Baseline 1 Upper = 23. 65 Lower = 0. 24

Baseline 2 Upper = 57. 16 Lower = 39. 62

B 2+ Concept Repetition Upper = 64. 30 Lower =49. 16

B 2+Percent Correctness Upper =48. 42 Lower = 32. 86

B 2+Student Move Upper = 61. 36 Lower = 39. 94

Discussion n Baseline 2 – has crossover effect and policy stability More complex features (B 2 + X) – have crossover effect, but not sure if polices are stable (some stabilize at 17 students) Indicates that 100 dialogues isn’t enough for even this simple MDP? (but is enough for baseline 2 to feel confident about? )