Robotics Institute Carnegie Mellon University Sensor Fusion for

Robotics Institute, Carnegie Mellon University Sensor Fusion for Context Understanding Huadong Wu, Mel Siegel The Robotics Institute, Carnegie Mellon University Sevim Ablay Applications Research Lab, Motorola Labs IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 1

Robotics Institute, Carnegie Mellon University sensor fusion • how to combine outputs of multiple sensor perspectives on an observable? • modalities may be “complementary”, “competitive”, or “cooperative” • technologies may demand registration • variety of historical approaches, e. g. : – – statistical (error and confidence measures) voting schemes (need at least three) Bayesian (probability inference) neural network, fuzzy logic, etc IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 2

Robotics Institute, Carnegie Mellon University context understanding • best algorithm for human-computer interaction tasks depends on context • context can be difficult to discern • multiple sensors give complementary (and sometime contradictory) clues • sensor fusion techniques needed • (but best algorithm for sensor fusion tasks may depend on context!) IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 3

Robotics Institute, Carnegie Mellon University agenda • a generalizable sensor fusion architecture for “context-aware computing” – or (my preference, but not the standard term) “context-aware human-computer interaction” • a realistic test to demonstrate usability and performance enhancement • improved sensor fusion approach (to be detailed in next paper) IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 4

Robotics Institute, Carnegie Mellon University background • current context-sensing architectures (e. g. , Georgia Tech Context Toolkit) tightly couple sensors and contexts • difficult to substitute or add sensors, thus difficult to extend scope of contexts • we describe a modular hierarchical architecture to overcome these limitations IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 5

Robotics Institute, Carnegie Mellon University toward context understanding Identification, representation, and understanding of context Adapt behavior to context traditional system sensor Information Separation + Sensor Fusion sensor humans understand context naturally & effortlessly sensor Sensing hardware: cameras, microphones, etc. Environment situation: people in the meeting room, objects around a moving car, etc. IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 6

Robotics Institute, Carnegie Mellon University methodology • top-down • adapt/extend Georgia Tech Context Toolkit (Motorola helps support both groups) • create realistic context and sensor prototypes • implement a practical context architecture for a plausible test application scenario • implement sensor fusion as a mapping of sensor data into the context database • place heavy emphasis on real sensor device characterization and (where needed) simulation IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 7

Robotics Institute, Carnegie Mellon University context-sensing methodology: sensor data-to-context mapping context observations & hypotheses sensory output IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 8

Robotics Institute, Carnegie Mellon University dynamic database • example: user identification and posture for discerning focus-of-attention in a meeting • tables (next) list basic information about environment (room) and parameters, e. g. , – temperature, noise, lighting, available devices, number of people, segmentation of area, etc – initially many details are entered manually – eventually a fully “tagged” and instrumented environment can reasonably be anticipated • weakest link: maintaining currency IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 9

Robotics Institute, Carnegie Mellon University context classification and modeling IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 10

Robotics Institute, Carnegie Mellon University context information architecture: dynamic context information database Room-table: NSH A 417 Area Detected people # Preference-table-user[Hd] Area-table Name Huadong Wu (κ =1. 0) Height History-table-user[Hd] (σ = 0. 5” ) Height 5’ 6” Weight Name Huadong Wu = 4 lb) Weight 144 lb (σ Preference Time Place Preference-table. Preference user[Hd] … 9: 06 AM-10: 55 AM NSH 4102 Name 6 (κ > 0. 5) Inside Area Detected users User-table Of room NSH A 417 Temperature 72 ºF (σ = 3 ºF) Light condition Brightness grade Entrance Area = 6 db) Noise level 60 db (σ = 6 db) Name Of. Devices room NSH A 417 Device-table Devices Device-table Temperature 72 ºF (σ = 3 ºF) Current Hd Detected people # 4 (κ > 0. 5) Light condition Brightness grade Mel Detected user 6 db)User-table Noise level 60 db (σ = … Devices Device-table Detected people # 2 (κ > 0. 5) Detected user User-table … Background-table-user[Hd] … … Alan Chris Preference-table. Preference Background Placeuser[Hd] Confidence … … IMTC’ 2002, Anchorage, AK, USA … 5’ 6” (σ = 0. 5” ) 144 lb (σ = 4 lb) Preference-tableuser[Hd] … User-table … … Backgroundtable-user[Hd] Backgroundtableuser[Mel] Backgroundtableuser[Alan] Backgroundtableuser[Chris] Huadong Wu (κ =1. 0) … Entrance [0. 5, 0. 9] Entrance [0. 3, 0. 7] Inside [0. 9, 0. 98] Inside [0. 4, 0. 9] … … Activitytable-user[Hd] Activitytableuser[Mel] Activitytableuser[Alan] Activitytableuser[Chris] … First detected history 10: 32 AM, History-table 06/06/2001 user[Hd] 11: 48 AM, History-table 06/06/2001 user[Mel] 2: 48 PM, History-table 06/06/2001 user[Alan] 10: 45 AM, History-table 06/06/2001 user[Chris] … IMTC-2002 -1077 Sensor Fusion mws@cmu. edu … 11

Robotics Institute, Carnegie Mellon University implementation • • low-level sensor fusion done sensing and knowledge integration via LAN context maintained in a dynamic database each significant entity (user, room, house, . . . ) has its own dynamic context repository • dynamic repository maintains and serves context and data to all applications • synchronization and cooperation among disparate sensor modalities achieved by “sensor fusion mediators” (agents) IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 12

Robotics Institute, Carnegie Mellon University system architecture to support sensor fusion for context-aware computing sensor smart sensor node user context database sensors appliance Internet sensor smart sensor node gateway applications sensor fusion embedded OS Intranet appliance embedded OS applications lower-level sensor fusion sensor smart sensor appliance node embedded OS sensors IMTC’ 2002, Anchorage, AK, USA context server higher-level sensor fusion database sever site context database IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 13

Robotics Institute, Carnegie Mellon University practical details. . . • context type =>sensor fusion mediator • mediator integrates corresponding sensors, e. g. , by designating some primary others secondary based on observed or specified performance • Dempster-Shafer “theory of evidence” implementation in accompanying paper • (white-icon components in following cartoon inherited from Georgia Tech CT) IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 14

Robotics Institute, Carnegie Mellon University system configuration diagram application Other AI algorithms Interpreter context data Resource Registry Other AI algorithms context data Resource Registry Interpreter Dempster-Shafer rule SF mediator Aggregator AI algorithms Aggregator Discoverer Widget sensor user - obile m computer IMTC’ 2002, Anchorage, AK, USA site context server Dynamic Context Database site context database server IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 15

Robotics Institute, Carnegie Mellon University details inherited from GT TC • JAVA implementation • Base. Object class provides communication functionality: sending, receiving, initiating, and responding via multithread server • context widgets, interpreters, and discovers subclass from the Base. Object, and inherit its functionality • service is part of the widget object • aggregators subclass from widgets, inheriting their functionality IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 16

Robotics Institute, Carnegie Mellon University object hierarchy and subclass relationship in context toolkit Base. Object Widget Service Interpreter Discoverer Aggregator IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 17

Robotics Institute, Carnegie Mellon University focus-of-attention application • neural network estimates head poses • focus of attention estimate based on head pose probability distribution analysis • audio reports speaker, assumed to be focus of other participants’ attention • situation is not easy to analyze due to, e. g. , dependence of behavior on discussion topic • initial results suggests we need more general fusion approach than provided by Bayesian IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 18

Robotics Institute, Carnegie Mellon University next paper. . . • Dempster-Shafer approach. . . • provides mechanism for handling “belief” and “plausibility” • cautiously stated, generalizes Bayes’ Law a priori probabilities to distributions • (difficulty, of course, is that usually neither the requisite probabilities nor the distributions are actually known) IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 19

Robotics Institute, Carnegie Mellon University focus-of-attention estimation from video and audio sensors IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 20

Robotics Institute, Carnegie Mellon University expectations Dynamic Configuration AI rules Time interval: T Sensor list Expected Updating flag …… context Sensor fusion mediator e. g. Dempster-Shafer Belief Combination Observations & Hypotheses Performance Boost 1. Uncertainty & ambiguity representation to user applications 2. Information consolidation & conflict resolving for users 3. Adaptive sensor fusion support switch to suitable algorithms 4. Robust to configuration change — and for some to die gracefully Widget sensor IMTC’ 2002, Anchorage, AK, USA 5. Situational description support — using more & complex context IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 21

Robotics Institute, Carnegie Mellon University conclusions • preliminary experiments demonstrate feasibility of context-sensing architecture and methodology • expect our further improvements via – – – better uncertainty and ambiguity handling fusion of overlapping sensors context-adaptive widget capability sensor fusion mediator coordinates resources context information server supports applications IMTC’ 2002, Anchorage, AK, USA IMTC-2002 -1077 Sensor Fusion mws@cmu. edu 22