Heterogeneous Multi-Robot Dialogues for Search Tasks Thomas K

Heterogeneous Multi-Robot Dialogues for Search Tasks Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky AAAI Spring Symposium 2005: Dialogical Robots

Communication among autonomous robots and humans → Embodied (not necessarily robotic, or even anthropomorphic) agents will become ubiquitous → NL dialogue will be a useful modality → Agent’s dialogue and task models will remain tightly coupled, and independent from other agents We will be talking to multiple agents without a (completely) shared dialog system AAAI Spring ’ 05 Symposium: Dialogical Robots 2

Gedankenexperiment • The problem of many robots: sweeper, furniture mover, baby monitor. • Task: clean up house; don’t wake baby up. • Who do you talk to? • What do you say? • Who talks back? AAAI Spring ’ 05 Symposium: Dialogical Robots 3

Issues for polylogical systems • What’s a reasonable architecture? – – – A single system controlling multiple robots A single DM directing autonomous robots Shared understanding/generation Shared microphone and speakers Robots self-contained: Nothing shared • How to communicate with multiple agents? – – Directions to the entire team? (“talk into the air”) Mediation through a designated team leader? (foreman) Instructions to each team member? (direct management) To a proxy agent within/without the team? (personal assistant) AAAI Spring ’ 05 Symposium: Dialogical Robots 4

What do we have today? • A platform that supports multiple (robot) participants – – Sphinx II ASR; Festival TTS Raven. Claw dialogue manager Galaxy-II message-passing architecture Several “back-end” interfaces • A basic corridor movement domain – Commands to move robots along in vectors and towards named locations – Mechanisms for describing robot status and location to human interlocutor AAAI Spring ’ 05 Symposium: Dialogical Robots 5

Current Architecture HUMAN INTERFACE ASR AGENT 1 DM 1 Robot 1 Interpret 2 AGENT 2 DM 2 Robot 2 Multi-voice TTS AAAI Spring ’ 05 Symposium: Dialogical Robots 6

Implementation HUMAN INTERFACE Sphinx II Phoenix Helios AGENT 1 Ravenclaw Bashful Robot Communication System Galaxy AGENT 2 Swift Rosetta AAAI Spring ’ 05 Symposium: Dialogical Robots Ravenclaw Clyde 7

Human-Robot Treasure Hunt • One Human • Two robots – Segway – Pioneer • Treasures “hidden” in large space • TASK: Retrieve all treasures AAAI Spring ’ 05 Symposium: Dialogical Robots 8

Scenario for Search We found it! We are at <x, y> AAAI Spring ’ 05 Symposium: Dialogical Robots 9

Issues in H-R communication 1. How do people decompose the task into sub-tasks? 2. What language do people use to get the tasks performed by the robots? 3. Given a human command, what is the expected robot behavior? § Explore using Wizard of Oz experiments AAAI Spring ’ 05 Symposium: Dialogical Robots 10

WOZ design • 1 experimenter, 2 robot actors, 1 participant. • Experimenter places treasure, simulates robot treasure sensors. • 1 st robot actor is blind, but carries a webcam for the participant’s consumptions. • 2 nd robot actor can only move by crawling. AAAI Spring ’ 05 Symposium: Dialogical Robots Communication takes place by normal speech, walkietalkie, and through the webcam. 11

Annotation and analysis Data transcribed annotated Utterances classified into functional categories 394 utterances 20 utterance categories 8 major categories Carnegie Mellon Mock. Brow annotation tool AAAI Spring ’ 05 Symposium: Dialogical Robots 12

Utterance/Task Breakdown § Controlling team behaviors § Grounding § Positive/negative feedback § Informing robot of it’s state or the world § Explanations of commands § Orientation Grounding § Navigation § Simple Navigation commands § Spatial Referential Navigation § Object Referential Navigation AAAI Spring ’ 05 Symposium: Dialogical Robots § Manipulation § Manipulating the environment § Manipulating treasure § Coverage § Manipulating the webcam view § Object coverage commands § Generic coverage § Asking about the robot’s abilities § Filler § Real-time command modifications 13

Summary / Future Work • Architecture for human-robot teams • Wo. Z study of language requirements • Different Wo. Z scenarios • Implement language for robot team • “Field” testing AAAI Spring ’ 05 Symposium: Dialogical Robots 14

Controlling team behaviors § “you guys get together” § “T- you go first and B- follow” AAAI Spring ’ 05 Symposium: Dialogical Robots 15

Grounding § Positive/negative feedback § “ok that’s better” § Informing robot of state § “so that’s up” § “I don’t see anything there” § Explanations of commands § “so I can see which direction is up” § Orientation Grounding § “What you’re facing now with the camera – is that the vehicle that you just circumnavigated” § “I can tell you’re going in the wrong direction, stop” AAAI Spring ’ 05 Symposium: Dialogical Robots 16

Navigation § Simple Navigation commands § § “so um T- turn to you left” “T- I want you to turn right 90 degrees” “can you go in that general direction” “can you proceed in that direction” § Spatial Referential Navigation § § “go to that open area” “continue around the periphery of that open area” “back out of that alley” “proceed in that direction until you find an opening to turn left” § Object Referential Navigation § “go over by T-” § “can you go on the other side of that vehicle” § “go over by the posters” AAAI Spring ’ 05 Symposium: Dialogical Robots 17

Manipulation § Manipulating the environment § “T- why don’t you move the trash can” § Manipulating treasure § “T- bring the coin to me” § Manipulating the webcam view § “ok B- look to your left” § “B- can you look around with the camera a little” AAAI Spring ’ 05 Symposium: Dialogical Robots 18

Coverage § Object coverage commands § “ok so examine the shelf” § “do you see something on that shelf in front of B-” § “can you look over by that table over there” § Generic Coverage § “do you see anything that looks interesting” AAAI Spring ’ 05 Symposium: Dialogical Robots 19

Asking about the robot’s abilities § “is that possible” AAAI Spring ’ 05 Symposium: Dialogical Robots 20

Filler § “and now um” § “ok um” AAAI Spring ’ 05 Symposium: Dialogical Robots 21

Real-time command modifications § § § “keep going” “stop” “a little more” “change of plans” “other direction” AAAI Spring ’ 05 Symposium: Dialogical Robots 22