Adaptive Automation for Human Performance in Large-Scale Networked

Adaptive Automation for Human Performance in Large-Scale Networked Systems Raja Parasuraman Ewart de Visser George Mason University Kickoff Meeting, Carnegie Mellon University, August 26, 2008. AFOSR MURI: Modeling Synergies in Large-Scale Human-Machine Networked Systems

Research Goals • Develop validated theories and techniques to predict behavior of large-scale, networked human-machine systems involving unmanned vehicles • Model human decision making efficiency in such networked systems • Investigate the efficacy of adaptive automation to enhance human-system performance 2

Collaborations with MURI Team Members od nd M ce a man bot T n-Ro a ea rfor m Pe eling Cornell/ MIT/Pitt Hum GMU Human-Agent Collaboration CMU Sca ling up to L arg e. N etw ork s All 3

George Mason University Approach • Conduct empirical and modeling studies of human decision making performance with multiple robotic assets • Examine human-system performance using the Distributed Decision Making simulation (DDD Version 4) – (with Mark Campbell of Cornell) • Examine efficacy of Adaptive Delegation Interface (ADI) with Machinetta for Human-Agent collaboration – (with Paul Scerri of CMU) • Develop human-robot performance metrics for use in large networks 4

Joint GMU-Cornell Approach • • • Examine human-system performance (1 -4 person teams, multiple unmanned vehicles, using DDD) in simulated reconnaissance missions (GMU) Model human decision-making performance (Cornell) Identify and quantify human “cognitive bottlenecks” (GMU and Cornell) Identify points for “adaptive tasking” or adaptive automation (GMU and Cornell) Scale up to larger networks (more UVs and agents) 5

Adaptable/Adaptive Automation Adaptable Automation Adaptive Automation Invocation method playbook Event Based Performance Based Model Based Teamwork Proxies Proxy Parasuraman (2000); Kaber & Endsley (2004); Scerri et al. (2006); Miller & Parasuraman (2007))

Playbook Interface for Robo. Flag • Playbook: Enables human-automation communication about plans, goals, and methods—akin to calling “plays” from a sports team’s playbook (Miller & Parasuraman, 2007) • Validation experiments with Robo. Flag • (Parasuraman et al. , IEEE SMC-Part A, 2005) • Human operator supervises multiple Blue Team robots using a delegation interface (Playbook) • Adapted from Cornell University • Work done under DARPA MICA Program

Methods Ø Single operator sends a team of 4 -8 robots (blue team) into opponent territory (populated by red team robots) to locate a specified target and return home as quickly as possible Ø User has Playbook of automated tools to direct robots Ø Waypoint (point and click) control (“Manual”) Ø Automated plays (Circle offense; Circle defense; Patrol border) Ø User selects number of robot(s) to which plays are assigned Ø User can intervene in robot execution of a play and apply corrective measures if necessary Ø Red team robot tactics predictable (always offensive or defensive) or unpredictable (either offensive or defensive)

Hypotheses for Efficacy of Playbook Interface • Use of automated plays at times of user’s choosing enhances mission success rate and reduces mission completion time • Flexible use or either automated plays or manual control allows user to compensate for “brittleness” of automation – particularly when opponent tactics are unpredictable • Management workload associated with delegation is only low to moderate 9

Flexible Delegation Enhances System Performance without Increasing User Workload 10 Parasuraman et al. , IEEE-SMC Part A, 2005

Playbook for Pre-Mission UCAV Planning • • User can call high-end play— e. g. , Airfield Denial, or Stipulate the method and procedure for doing Airfield Denial by – filling in specific variable values (i. e. , which airfield to be attacked) – what UAVs to be used – where they should rendezvous – stipulate – which sub-methods and optional task path branches to be used – Etc. Miller & Parasuraman, Human Factors, 2007 11

Simulation Platforms at GMU • DDD 4. 0 – 1 -4 person teams – Large numbers of UVs/agents • Adaptive Delegation Interface (ADI) – Designed for planning, executing, and monitoring UV movements – Adaptable: High level plans can be proposed by the user and modified by the automation – Adaptive: UVs can autonomously adjust to certain events in the scenario 12

Adaptive Delegation For Planning • Delegation Interfaces: Execution – Many Human-Robot interfaces are primarily execution based – Robo. Flag is an example of an execution-based delegation interface • Delegation Interfaces: Planning – Little prior work on real-time planning with robotic vehicles – Related work on route planning for pilots: Layton et al. (1994) • Preliminary research under DARPA's Multiagent Adjustable Autonomy Framework (MAAF) for Multi-Robot, Multi-Human Teams (with Amos Freedy). 13

Adaptive Delegation Concept Plan verification with doctrine Shared task model Robotic Operator Adaptive Interface planning instructions feedback Automated Planning Assistant planning Doctrine Checker Machinetta feedback Sending instructions to vehicles plan execution monitoring Battle Space 14 automated plan generation

Automated Route Planning Post Processing - Task ordering goes through all possible permutations of the given tasks (if requested) and submits to Machinetta a specific task order to be followed. - Machinetta generates the optimized path plan to reach target locations - Post processing makes use of Machinetta generated paths (for the SEARCH task type) and introduces loading/unloading time (for the EXTRACT task type) into plans. Time Steps 0 1 2 3 8 7 6 5 4 3 2 1 0 Machinetta UV 1 Regions Task Ordering UV 2 Time Steps - Given target location, current location of vehicle and time, fuel, task importance and risk avoidance importance; Machinetta iterates through all possible region traversal options and converges on the best (in terms of time, fuel and risk) trajectory possible. (one such trajectory for a vehicle, 4 time steps and 9 regions is shown in figure above) - Machinetta takes into account both user specified parameters (such as task and risk importance), as well as vehicle capabilities (such as speed and fuel), and generates plans that can implement such complex 15 behaviors as delayed action and risk avoidance.

Multiagent Adjustable Autonomy Framework 5 Adjustable Autonomy 1 UGV 2 asks to confirm Human responds by confirming IED presence 4 2 2 2 1 UGV 2 Camera Failure UGV 1 then provides view 2 Autonomous Behavior 2 1 Plan Instantiation 1 2 1 Dynamic Reallocation Plan is given to UVs carry out plan ! Dynamic Reallocation 1 3 2 Obstacle on Path UGV 1 avoids obstacle UGV 1 loses comms UAV assists and functions as relay station

Task library The Adaptive Delegation Interface Mission wizard & compose Mission map Automated planning assistant 17

Mission Planning & Execution Mission Map Task Library Compose View Reconnaissance Rescue & Extract add asset delete asset Mission Parameters + Super Plays + UAV 1 UAV Recon clockwise UGV 1, 2 UAV Recon counterclockwise Plays UAV Recon UGV Recon & Extract vehicle parameters tasks - Tasks - UAV Recon counterclockwise UGV Recon & Extract G 1, B 5 move search - Move G 1 UAV Recon G 1 extract 2: 00 go home Search G 1 Move G 1 Reactions avoid Search G 1 wait stop + Extract G 1 UAV Recon Move B 5 Check plan Mission Wizard Mission Compose Finish plan Submit plan Automated Planning Assistant Plan ID Iter Type Assets Plan A Plan B R&E 2 2 1 UAV, 2 UGV Time 45 55 Damage Victims 20 5 4 35 Overall 60 45 Type Message Content Review You should include a UAV in the plan before submission Status New plans have been generated Mission Execution Standing By 18 Review plan Modify plan Submit plan Finish plan Execute plan

1 2 3 Agent Control Panel Agent Status Panel Units Assets Task Status Talon Unit Alpha 1 Talon 1 3 3 3 14: 05 Talon 2 14: 05 T 1 T 2 Issues Role Action Time Cannot see IED Talon Unit Alpha (2/3) MI Company Recon & clear area of IEDs Role-reallocation… unknown Camera Failure Provide Camera support Re-defining role… None Disarm IED Moving to IED location unknown ~10 min. Talon 3 STOP Options Task Vehicle Sensors Message Center Pop-out Timeline

Advantages of using the Adaptive Delegation Interface • Users can give high-level commands to a set of vehicles – No need to input each task individually – Automation can generate and finish plans – Humans can adjust plans as needed • Users can monitor executed plans and intervene if necessary • Minimal training needed (20 -30 min. ) 20