Addressing Uncertainty in Performance Measurement of Intelligent Systems

Addressing Uncertainty in Performance Measurement of Intelligent Systems Raj Madhavan 1, 2 Elena Messina 1 Hui-Min Huang 1 Craig Schlenoff 1 1 Intelligent Systems Division National Institute of Standards and Technology (NIST) & 2 Institute for Systems Research (ISR) University of Maryland, College Park Commercial equipment and materials are identified in this presentation in order to adequately specify certain procedures. Such identification does not imply recommendation or endorsement by NIST, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose. The views and opinions expressed are those of the presenter and does not necessarily reflect those of the organizations he is affiliated with.

Measuring Performance of Intelligent Systems § Performance Evaluation, Benchmarking, and Standardization are critical enablers for wider acceptance and proliferation of existing and emerging technologies § Crucial for fostering technology transfer and driving industry innovation § Currently, no consensus nor standards exist on – – § key metrics for determining the performance of a system objective evaluation procedures to quantitatively deduce/measure the performance of robotic systems against user-defined requirements The lack of ways to quantify and characterize performance of technologies and systems has precluded researchers working towards a common goal from – – – exchanging and communicating results, inter–comparing robot performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer.

Measuring Performance of Intelligent Systems § The lack of ways to quantify and characterize technologies and systems also hinders adoption of new systems – – § Users may be reluctant to try a new technology for fear of expensive failure: – § Users don’t trust claims by developers There is lack of knowledge about how to match a solution with a problem … Think of the “graveyards” of unused equipment in some places

Challenges in Measuring Performance of IS § § Diversity of applications and deployment scenarios for the IS Complexity of the Intelligent System itself – – – § Software components Hardware components Interactions between components – System of Systems Lack of a well-defined mathematical foundation for dealing with uncertainty in a complex system – – – methods for computing performance measures and related uncertainties techniques for combining uncertainties and making inferences based on those uncertainties approaches for estimating uncertainties for predicted performance

Uncertainty and Complexity § Uncertainty and complexity are often closely related § The abilities to handle uncertainty and complexity are directly related to the levels of autonomy and performance

Autonomy Levels for Unmanned Systems (ALFUS) Framework § Standard terms and definitions for characterizing the levels of autonomy for unmanned systems § Metrics, methods, and processes for measuring autonomy of unmanned systems § Contextual Autonomous Capability § http: //www. nist. gov/el/isd/ks/autonomy_levels. cfm/ (Hui. Min Huang)

Addressing Uncertainty in Performance Measurement via Complexity § In this context, performance that we are trying to measure is taken to mean the successful completion of the mission § Being able to handle higher level of mission and environmental complexities results in higher system performance § We can determine whether program-specific performance requirements are achievable Mobility Example

Test Methods (1) Hurdle Test Method The purpose of this test method is to quantitatively evaluate the vertical step surmounting capabilities of a robot, including variable chassis configurations and coordinated behaviors, while being remotely teleoperated in confined areas with lighted and dark conditions. Metrics • Maximum elevation (cm) surmounted for 10 repetitions • Average time per repetition § Hurdle Test Method Results: Numbers indicating successful repetitions. 10 corresponds to reliability of 80%--probability of success--that the robot can successfully perform the task at the associated apparatus setting. § Measurement Uncertainty (in measuring Obstacle Traverse Capability): One half of the obstacle size increment (5 cm) and the elapsed time unit (30 s)

Comms Example

Test Methods (2) Radio Comms (Lo. S) Test Method The purpose of this test method is to quantitatively evaluate the line of sight (LOS) radio communications range for a remotely teleoperated robot. Metric • Maximum distance (m) downrange at which the robot completes tasks to verify the functionality of control, video, and audio transmissions. Line-of-Sight Radio Comms Test Method : Stations every 100 m for testing two-way communications. Multiple testing tasks at each test station sum up for the repeatability.

SCORE • § SCORE (System, Component and Operationally Relevant Evaluations) • Is a unified set of criteria and software tools for defining a performance evaluation approach for complex intelligent systems • Provides a comprehensive evaluation blueprint that assesses the technical performance of a system, its components and its capabilities through isolating and changing variables as well as capturing enduser utility of the system in realistic use-case environments System – a set of interacting or interdependent components forming an integrated whole intended to accomplish a specific goal • Component – a constituent part or feature of a system that contributes to its ability to accomplish a goal • Capability – a specific purpose or functionality that the system is designed to accomplish • Technical Performance – metrics related to quantitative factors (such as accuracy, precision, time, distance, etc) as required to meet end-user expectations • Utility Assessment – metrics related to qualitative factors that gauge the quality or condition of being useful to the end-user

How SCORE Handles Complexity • The complexity of the “system under test” grows as more components are introduced into the evaluation • Components evaluated in the elemental tests are less complex than sub-systems (which contain multiple components) which are less complex than the while system • SCORE tests at these various levels of complexity • Data in the following slides indicate that the results of the elemental tests can accurately be predictive of the performance of the subsystem test (which is more complex) and so on.

TRANSTAC • GOAL – Demonstrate capabilities to rapidly develop and field free-form, two-way speech-to-speech translation systems enabling English and foreign language speakers to communicate with one another in real-world tactical situations. • NIST was funded over the past three years to serve as the Independent Evaluation Team for this effort. • METRICS (as specified by DARPA) • System usability testing – providing overall scores to the capabilities of the whole system • Software component testing – evaluate components of a system to see how well they perform in isolation

TRANSTAC A QUICK TUTORIAL ON SPEECH TRANSLATION Please open the car door. Automatic Speech Recognition (ASR) Please open the car door. Machine Translation (MT) ﻳﺮﺟﻰ ﻓﺘﺢ ﺑﺎﺏ ﺍﻟﺴﻴﺎﺭﺓ Text To Speech (TTS)

TRANSTAC METRICS Automated Metrics: . For speech recognition, we calculated Word-Error-Rate (WER). For machine translation, we calculated BLEU and METEOR. TTS Evaluation: Human judges listened to the audio outputs of the TTS evaluation and compared them to the text string of what was fed into the TTS engine. They then gave a Likert score to indicate how understandable the audio file was. WER was also used to judge the TTS output. Low-Level Concept Transfer: A directly quantitative measure of the transfer of the low-level elements of meaning. In this context, a low-level concept is a specific content word (or words) in an utterance. For example, the phrase “The house is down the street from the mosque. ” is one high-level concept, but is made up of three low-level concepts (house, down the street, mosque). Likert Judgment: A panel of bilingual judges rated the semantic adequacy of the translations, an utterance at a time, choosing from a seven point scale. High-Level Concept Transfer: The number of utterances that are judged to have been successfully transferred. The high-level concept metric is an efficiency metric which shows the number of successful utterances per unit of time, as well as accuracy. Surveys/Semi-Structured Interviews: After each live scenario, the Soldiers/Marines and the foreign language speakers filled out a detailed survey asking them about their experiences with the TRANSTAC systems. In addition, semi-structured interviews were performed with all participants in which questions such as “What did you like? , What didn’t you like? and What would you change? ” were explored.

TRANSTAC SCORE Level Metric Team 1 Team 2 Team 3 Complexity Elemental BLEU 1 2 2 Elemental METEOR 1 2 2 Elemental TTS Low-level Concept Transfer 1 1 2 2 Likert Judgment High Level Concept Transfer 1 2 2 1 2 3 User Surveys 1 2 3 Sub-System (Qualitative) From this data, it appears that: • the quantitative performance of the elements of the systems have a direct correlation to the quantitative performance of the subsystems; • the quantitative performance of the sub-systems has a direct correlation to the quantitative performance of the overall system; • the quantitative performance of the overall system has a direct correlation to the qualitative perception of the soldiers using the systems.

In Conclusion … Thank you!