MITRE Dialog Management Workshop a review Dan

MITRE Dialog Management Workshop – a review Dan Bohus Dialogs on Dialogs reading group CMU, November 2003

The Workshop § MITRE Dialog Workshop § § § @ MITRE, Bedford/Boston October 27 -28, 2003 Idea § § Bring together researchers working on dialog management Give them a homework n § Adapt you dialog manager to a medical diagnosis domain (details in a sec) Discuss, compare, learn MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

The Homework § Implement a dialog system for the medical diagnosis domain § § Task left open-ended (diagnosis, tutoring, etc) No speech, just text in and out Backend provided backend. doc n Java version and web-based interface version n 3 diseases: malaria, coccidioidomycosis, another one n List of symptoms: headache, nausea, muscle pain, etc. n Decision tree involving symptoms and tests (fever, blood tests, travel patterns, etc) Small enough to presumably not be lots of work, but large enough to allow illustration of functionalities, and provide some skeleton to the discussions… MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Participants § § § § MITRE (Carl Burke et al) Gothenburg (Staffan Larsson) USC ICT (David Traum) NTT/CMU (Matthias Denecke) CMU (Dan, Alex) Ames (Beth-Ann Hockey) DFKI (Norbert Reithinger) MERL (Candy Sidner, Charles Rich) Mi. Di. Ki Go. Di. S (TRINDIKit) ICT Dialogue Manager Ariadne Raven. Claw NASA Dialogue Manager DFKI Dialogue Manager COLLAGEN … and others invited but not present MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S § TRINDIKit – information state update dialogue management toolkit § Information state n n § § § Private: dialog plan, beliefs, agenda (short term goals) Shared: established facts, QUD, last utterance information Dialog moves Update rules Go. Di. S: dialog management system implemented in TRINDIKit, handing: § § information oriented dialogue action oriented dialogue MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

TRINDIKit / Go. Di. S architecture control DME input interpret update select generate output TIS DEVICES Connectio n to Java Backend LEXICON DOMAIN Dialog plans Ontology backend interface MITRE Dialog Management Workshop lexicon domain knowledge workshop : godis : ravenclaw : collagen : themes

Go. Di. S: Task Representation § § Plans; propositional logic Dialogue plans for dealing with diagnosis (issues opened at dialogue start) § § ? x. disease(x): ”which disease is diagnosed? ” ? confirmed_by_interview: ”Is the diagnosis confirmed by additional information? ” ? confirmed_by_tests: ”Is the diagnosis confirmed by medical tests? ” Additional plans § § ? x. info(x): ”What information is there about a given disease? ” ? x. treatment(x): ”What treatment is there for a given disease? ” MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S: Alternate Tasks § User-driven dialogue (implemented) § § § Decision trees as dialogue plans § § § Not load issues when resetting; user has to raise all issues User can ask system to n Provide a diagnosis n Confirm whether user has given disease Move backend knowledge into dialogue plans Information conversion could be done automatically Separate genre: expert system dialogue § § Add special purpose update rules Dynamic dialogue planning by expert MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S: Highlights / Lowlights § Highlights: § Reuse, you get for free: n n n § High-level abstraction for dialog plans n § Grounding Accomodation / plan recognition Multiple simultaneous issues & info sharing Rapid prototyping Lowlights § § Not used in this type of domain so far, so not entirely straight-forward (update rule changes) Dynamic dialog plans (backend decides) MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S Raven. Claw MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw § § Captures all domain-specific dialog (task) logic with a hierarchical description The authoring effort is focused entirely here Dialog Task (Specification) Domain-independent Dialog Engine § § Manages dialog by executing the dialog task specification Provides domain-independent conversational strategies MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel general_feeling R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel general_feeling R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel general_feeling R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Welcome Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel general_feeling R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Hi, this is Madeleine, the automated… Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel R: Headache general_feeling R: R: headache R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Hi, this is Madeleine, the automated… Load. Symptoms Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel R: Headache general_feeling R: R: headache R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Hi, this is Madeleine, the automated… Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel R: Headache general_feeling R: R: headache R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Hi, this is Madeleine, the automated… General. Feel Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Architecture chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel R: Headache general_feeling R: R: headache R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda general_feeling: [good], [bad], [soso] How. Are. You General. Feel Madeleine general_feeling: [good], [bad], [soso] have_fever: [fever]. ![yes], ![no] headache: [headache], ![yes], ![no] cough: [cough], ![yes], ![no] … … MITRE Dialog Management Workshop Hi, this is Madeleine, the automated… How are you feeling today? Not so good, I think I have a fever [soso](not so good) [fever](I think I have a fever) workshop : godis : ravenclaw : collagen : themes

Illustrated Features § Dynamic generation of dialog task structure § § § Dynamic dialog control policy § § Symptoms loaded from backend, appropriate structures to “talk about them” created on-the-fly New symptoms – no DM changes The order in which symptoms are addressed is controlled by the backend Conversational skills MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Illustrated Features § Dynamic generation of dialog task structure § § § Dynamic dialog control policy § § Symptoms loaded from backend, appropriate structures to “talk about them” created on-the-fly New symptoms – no DM changes The order in which symptoms are addressed is controlled by the backend Conversational skills MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Backend Decision Tree Dynamic Dialog Control … chart Madeleine I: Welcome E: Load. Symptoms General. Feel R: How. Are. You? I: Glad diagnostic Diagnose I: Sorry Fever Travel R: Headache general_feeling R: R: headache R: Ask. Fever E: Measure. Temp I: Inform. Fever have_fever Dialog Stack Expectation Agenda Hi, this is Madeleine, the automated… How are you today? Not so good, I think I have a headache Diagnose Sorry to hear you’re not feeling so good, Tell me more about your symptoms… Do you have abdominal pain? Madeleine MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Illustrated Features § Dynamic generation of dialog task structure § § § Dynamic dialog control policy § § Symptoms loaded from backend, appropriate structures to “talk about them” created on-the-fly New symptoms – no DM changes The order in which symptoms are addressed is controlled by the backend Conversational skills MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Conversational Skills § Corresponding agencies added automatically to the dialog task tree § § § § Help What Can I Say? Repeat Suspend / Resume Start Over Timeout handling (not illustrated) Still need all the language generation prompts and grammar, but some of those are developonce, too MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Raven. Claw Conclusion § Highlights § Set task posed no challenges to the framework n § § § Easy to implement Dynamic dialog structure and control Automatic use of domain-independent conversational skills Lowlights? § § § Toolkit perspective: how easy would it be for someone else to build it? Asynchronous behaviors? (timing) Couple of bugs / fixes (or is that a highlight? ) MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S Collagen MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

COLLAGEN Collaborative Interface Agent * focus stack plan tree Collagen communicate observe interact MITRE Dialog Management Workshop observe interact workshop : godis : ravenclaw : collagen : themes

COLLAGEN Systems § air travel planning § email reading and responding (w. IBM/Lotus) § GUI design tool operation § car navigation system operation § airport landing path planning (w. MITRE) § gas turbine operator training (w. USC/ISI) § personal video recorder operation § programmable thermostat operation (with Delft U. ) § multi-modal web-based form-filling MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Collagen: Theory and Implementation Java Implementation Shared. Plan Discourse Theory Intentional purposes, contributes focus stack focus spaces, focus stack segments, lexical items Attentional Linguistic purpose tree (Grosz, Sidner, Kraus, Lochbaum 1974 -1998) MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Collagen: Discourse Segments and Purposes (fixing an air compressor, E = expert, A = apprentice) E: Replace the pump and belt please. A: Ok, I found a belt in the back. replace pump and belt replace belt A: Is that where it should be? A: [removes belt] A: It’s done. E: Now remove the pump. … replace pump E: First you have to remove the flywheel. … E: Now take the pump off the base plate. A: Already did. MITRE Dialog Management Workshop (Grosz, 1974) workshop : godis : ravenclaw : collagen : themes

Discourse state representation Focus Stack Purpose Tree replace pump and belt current focus space replace belt replace pump and belt replace pump replace belt E: Replace the pump and belt please. replace pump replace belt and belt A: Ok, I found a belt in the back. A: Is that where it should be? A: [removes belt] A: It’s done (Grosz & Sidner, 1986) MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Discourse interpretation algorithm The current (communication or manipulation) act either: • starts a new segment/focus space (push) focus stack • ends the current segment/focus space (pop) • continues (contributes to) the current segment/. . . (add) An act contributes to the purpose of a segment if it: • directly achieves the purpose • is a step in the plan for the purpose * purpose tree • identifies the recipe used to achieve the purpose • identifies who should perform the purpose or a step in the plan • identifies a parameter of the purpose or a step in the plan * does not include recursive plan recognition (see later topic) MITRE Dialog Management Workshop (Lochbaum, 1998) workshop : godis : ravenclaw : collagen : themes

COLLAGEN … my take § § Separation of task from dialog/discourse engine Recipes / Domain plans / Task tree § § Full-blown HTN n Hierarchical n Preconditions (constraints) n Effects n Completion / failure n Live nodes Stack to keep track of focus and discourse structure Tree explicitly contains agent and user nodes Formalized / descriptive recipe specs (actually Java underneath), with procedure overwrites… MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Go. Di. S Themes … MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Task Representation § Task representation § § § Separation of task representation from dialog engine High-level representations of task Descriptive rather than procedural n n § Procedural will be unavoidable for complex tasks Expressive power Go. Di. S, Raven. Claw, Collagen: plan based representations of task MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Task/Domain/Gendre § The notion of dialog gendre § § Where to fold it in a dialog manager? § § Tutoring Diagnosis Information Access Go. Di. S: update/select rules Ariadne: plugins Raven. Claw: collapsed with task How clear is that separation: task vs. gendre? MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Development time § Systems took on the order of 3 -5 days to develop § Significant effort in the backend connection n § Significant effort in grammar/language generation development n § Some sites shortcut it Everyone that had an implementation: “fixed a couple of bugs, but no major changes required” MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Development tools § Regression testing (Go. Di. S) § § Systems are complex. Change something in a dialog management framework, can you prove that it did not screw up things that used to work? System-wise, very intractable Component-wise, maybe: i. e. DM with DM inputs/outputs System diagnosis / log visualization tools (Collagen) MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Timing § (Micro)timing § § Turn-taking models § § in general, very rudimentary Asynchronous behaviors § § unaddressed Could be accomplished, but no-one seemed to have it Multi-party conversation § unaddressed MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: the important problems § Different people have different views of what those are: § § § Plan / Intention recognition Reference resolution Backup in complex systems Tense problems Negations Grounding; error prevention / recovery MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Reasoning § Dialog Managers vs Backends § § Where to draw the line? Who does the reasoning? Can we avoid duplicating it? How rich is the interaction between them? n n § Dialog systems - use language to act in a domain, so they are generally strongly tied Basic set of conversational skills can be identified Drawing that line is still an “art”, no general agreement or solutions exist MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes

Themes: Science of Dialog? § § How much science do we have? Theory vs. experiment § § § Interesting Collagen / Raven. Claw similarities Representation or not? GUI analogy § Do we have the checkboxes and radio-buttons? MITRE Dialog Management Workshop workshop : godis : ravenclaw : collagen : themes