Understanding Human Behavior from Sensor Data Henry Kautz University of Washington
Trends
Growing Ubiquitous Sensing Infrastructure § § GPS Wi-Fi localization RFID tags Wearable sensors
Advances in Artificial Intelligence § § Graphical models Particle filtering Belief propagation Statistical relational learning
Crisis in Caring for the Cognitively Disabled § Epidemic of Alzheimer’s § Community integration of 7. 5 million citizens with MR § 100, 000 @ year disabled by TBI § Post-traumatic stress syndrome § Caregiver burnout
. . . An Opportunity § Create methods for modeling and interpreting human behavior from sensor data § In order to develop assistive technologies to support independent living by people with cognitive disabilities o Help people perform activities of daily living o Monitor behavior to prevent health crises
Outline § Learning and reasoning about transportation routines o ACCESS personal guidance system § Understanding activities of daily living o CARE monitoring system § Further directions
ACCESS Assisted Cognition in Community, Employment, & Social Settings
Motivation: Community Access for the Cognitively Disabled
The Problem § Using public transit is cognitively challenging o Learning bus routes and numbers o Transfers o Recovering from mistakes § Point to point shuttle service impractical o Slow o Expensive § Current GPS units hard to use o Require extensive user input o Error-prone near buildings, inside buses o No help with transfers, timing
Goal § A personal guidance system that o Requires no explicit programming by user or caregiver o Proactively assists user in completing transportation plans o Recognizes user errors, and helps user recover
Technical Problem § Given a data stream from a wearable GPS unit. . . o Infer the user’s location and mode of transportation (foot, car, bus, bike, . . . ) o Predict the user’s destination o Detect user errors
GPS Receivers Geo. Stats GPS logger § Data capacity: 3 weeks @ 0. 5 second frequency § Battery life: 72 hours Nokia 6600 Cell Phone § Java § Bluetooth GPS unit § Internet connectivity for off-board processing § Battery life: 8 hours
GIS Data § Street map o Census 2000 § Bus routes & stops o Seattle Metro § Business / residential areas o Map. Point
Challenges of GPS Data § Many dead zones in urban areas § Sparse measurements inside cars and buses § Systematic error 10 meters § Multi-path propagation § Mapping errors
Representation § Map is a directed graph G=(V, E) § Location: o Edge (block) o Distance along edge o Actual GPS reading § Movement: o Mode { foot, car, bus, indoors } influences velocity o Change edges at intersections o Change mode at bus stops, parked car, buildings § Tracking (filtering): Given some prior estimate, o Update position & mode according to motion model o Correct according to next GPS reading
Motion Model for Mode
Dynamic Bayesian Network I mk-1 Transportation mode xk-1 qk-1 mk xk Edge, velocity, position zk-1 Time k-1 qk Data (edge) association zk Time k GPS reading
Rao-Blackwellised Particle Filtering § Particle filtering: o Evolve approximation to state distribution using samples (particles) o Supports multi-modal distributions and discrete variables (mode, edge) § Rao-Blackwellisation o Particles include distributions over variables o Each particle is a Kalman filter (Gaussian along edge) o Improved accuracy with fewer particles
Learning to Predict User § Prior knowledge – general constraints on transportation use o Vehicle speed range o Bus stops § Learning – specialize model to particular user o 30 days GPS readings of one user, logged every second o Unlabeled data o Learn edge and mode transition parameters using expectation-maximization (EM)
Mode & Location Tracking Measurements Projections Green Bus mode Red Car mode Blue Foot mode
Probability of correctly predicting the future Predictive Accuracy How to improve the model’s predictive power? 5 blocks 50% accuracy City Blocks 22
Transportation Routines Home A B Workplace § Goal: intended destination o Workplace, home, friends, restaurants, … § Trip segments: <start, end, mode> o Home to Bus stop A on Foot o Bus stop A to Bus stop B on Bus o Bus stop B to workplace on Foot
Dynamic Bayesian Net II gk-1 Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 qk-1 gk xk Edge, velocity, position zk-1 Time k-1 qk Data (edge) association zk Time k GPS reading
Unsupervised Hierarchical Learning § Use previous model to infer: o Goals - locations where user stays for long periods of time o Transition points - locations with high mode transition probability o Trip segments – paths connecting transition points or goals § Perform EM learning on the hierarchical model o Learn transition parameters: · Between goals · Between trip segments, given the goal · Between edges & modes, given the trip segment
High Probability Trip Segments Conditioned on Goal = Workplace : bus Goal = Home : car : foot
Predicting Goal and Path Predicted goal Predicted path
Improvement in Predictive Accuracy 45 blocks 50% accuracy
Detecting User Errors § Learned model represents typical correct behavior o Model is a poor fit to user errors § We can use this fact to detect errors! § Cognitive Mode o Normal: model functions as before o Error: switch in prior (untrained) parameters for mode and edge transition
Dynamic Bayesian Net III ck-1 gk Cognitive mode { normal, error } Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 qk-1 ck xk Edge, velocity, position zk-1 Time k-1 qk Data (edge) association zk Time k GPS reading
Goal Clamping § The user’s goal may be explicitly known o Ask the user to confirm highest-probability goal o Appointment calendar § Incorporating such information clamps the goal o Distinguishes novelty from errors
Detecting User Errors Untrained Trained Clamped
ACCESS Prototype § Cell phone with GPS, camera, high-speed internet access § Prompts when it infers that user is…
ACCESS Prototype § Cell phone with GPS, camera, high-speed internet access § Prompts when it infers that user is… About to begin a transportation plan • Confirm destination? • Here is your route!
ACCESS Prototype § Cell phone with GPS, camera, high-speed internet access § Prompts when it infers that user is… About to change mode • This is your bus! • Your stop is next!
ACCESS Prototype § Cell phone with GPS, camera, high-speed internet access § Prompts when it infers that user is… Making an error • You missed your stop! • Here is how to get back on track …
ACCESS Prototype § Cell phone with GPS, camera, high-speed internet access § Prompts when it infers that user is… Visiting a new destination • Please take a picture!
Status § Medical partnerships o Funding by National Institute of Disability & Rehabilitation Research (NIDRR) o Partnership with UW Center for Technology and Disability Studies · User & caregiver needs studies (TBI & MR) · Data collection by job coaches § Extension to indoor navigation o Hospitals, nursing homes, assisted care communities o Wi-Fi localization § Multi-modal interface o Speech, graphics, text o Guidance strategies · Proactive / Just in time · Coordinates / Landmarks o WOZ design study
WOZ
Papers Patterson, Liao, Fox, & Kautz, UBICOMP 2003 Inferring High Level Behavior from Low Level Sensors Patterson et al, UBICOMP-2004 Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services Liao, Fox, & Kautz, AAAI 2004 (Best Paper) Learning and Inferring Transportation Routines
CARE Cognitive Assistance in Real-world Environments
Goal § A home monitoring system that o Assists user in performing activities of daily living o Tracks activities, and provides prompts and warnings as needed o Can be deployed in an ordinary home o Does not require the user to learn a different way to perform the activities – the system adapts, not the user
Short-Term Application § Accurate, automated ADL logs o Changes in routine often precursor to illness, accidents o Human monitoring intrusive & inaccurate Image Courtesy Intel Research
Technical Requirements § Sensor hardware that can be practically deployed in a ordinary home § Methods for activity tracking from sensor data § Methods for automated prompting that consider o Probability of user errors o Probability of system errors o Cost / benefit tradeoffs
Object-Based Activity Recognition § Activities of daily living involve the manipulation of many physical objects o Kitchen: stove, pans, dishes, … o Bathroom: toothbrush, shampoo, towel, … o Bedroom: linen, dresser, clock, clothing, … § We can recognize activities from a timesequence of object touches
Sensing Object Manipulation § RFID: Radiofrequency identification tags o o Small Semi-passive Durable Cheap § Near future: use products’ own tags
Wearable RFID Readers § Designed by Intel Research Seattle o Will be shared with other Intel partners later this year § 13. 56 MHz reader, radio, power supply, antenna o 12 inch range, 12 -150 hour lifetime
Experiment: Morning Activities § 10 days of data from the morning routine in an experimenter’s home o 61 tagged objects § 11 activities o Often interleaved and interrupted o Many shared objects Use bathroom Make coffee Set table Make oatmeal Make tea Eat breakfast Make eggs Use telephone Clear table Prepare OJ Take out trash
Methodology § Goal: simplest model that can robustly track activities § Comparison o Hidden Markov Model o Dynamic Bayesian Network with aggregate features o DBN with aggregation and abstraction smoothing
Hidden Markov Model § § Trained on labeled data 10 -fold cross validation 88% accuracy 9. 4 errors per 20 minute episode
Cause of Errors § Observations were types of objects o Spoon, plate, fork … § Typical errors: confusion between activities o Using one object repeatedly o Using different objects of same type § Critical distinction in many ADL’s o Eating versus setting table o Dressing versus putting away laundry
Aggregate Features § HMM with individual object observations fails o No generalization! § Solution: add aggregate variables o Bit-vector maintains history of objects touched o Aggregate distribution nodes sum number of distinct instances o Aggregate nodes treated as pseudo-observations when an activity transitions o DBN encoding avoids explosion of HMM
DBN with Aggregation § Average number of errors reduced from 9. 4 to 6. 5 (31%) § Deterministic nodes add minimal computational overhead
Improving Robustness § Both HMM and DBN fail if novel (but reasonable) objects are used § Solution: smooth parameters over abstraction hierarchy of object types
Abstraction Smoothing § Methodology: o Train on 10 days data o Test where one activity substitutes one object § Change in error rate: o Without smoothing: 26% increase o With smoothing: 1% increase
Experiment: ADL Form Filling § Tagged real home with 108 tags § 14 subjects each performed 12 of 14 ADLs in arbitrary order § Used glove-based reader § Given trace, recreate activities
Results: Detecting ADLs Activity Prior Work RFID SHARP Personal Appearance 92/92 Oral Hygiene 70/78 Toileting Legend 73/73 Washing up 100/75 Use of Heating Point solution 100/33 Appliance Use General solution 84/78 Care of clothes and linen 100/73 Making a snack 100/78 Making a drink 75/60 Use of phone 64/64 Leisure Activity 100/79 Infant Care 100/58 Medication Taking 100/93 Housework 100/82 Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004
Summary: CARE § Activities of daily living can be learned and robustly tracked using RFID tag data o Simple, direct sensors can often replace (or augment) general machine vision o Works for essentially all ADLs defined in healthcare literature
Next Steps
Key Next Problems § Decision-theoretic control of user interfaces o o Prompts helpful or distracting? User error, or user model error? Context-dependent costs Decision-theoretic natural language processing
Key Next Steps § Measures of effectiveness o ACCESS: user studies with potential clients (controlled conditions) this spring o CARE: Intel / UW / Wash Dept Social Services developing trial for caregiver evaluation o Goal: Long-term deployment in naturalistic settings · Homes, nursing homes, assisted care facilities
Future Research § Physiological sensors o Heart rate, respiration, temperature § Heterogeneous sensors o Environmental + wearables + machine vision o Smart homes § Systems for improving self-awareness o Emotional self-regulation o Social pragmatics o Target populations · Autism spectrum disorders · Traumatic brain injury
General Architecture commonsense knowledge cognitive state intentions decision making activities user profile machine learning physical behavior user interface sensors caregiver alerts
Conclusion: Why Now? § An early goal of AI was to create programs that could understand ordinary human experience § This goal proved elusive o Missing tools for probabilistic inference o Systems not grounded in real world o Lacked compelling purpose § Today we have the mathematical tools, the sensors, and the motivation
Other Research
Building Social Network Models from Sensor Data NSF Human Social Dynamics multi-modal sensor board Coded Database code identifier real-time audio feature extraction audio features Wi. Fi strength
Modal Markov Logic Applied to Dialog Understanding DARPA CALO ASK_IF(S, H, P) ~B~p v ~Bp B ~p B(S, B(H, P) v B(H, ~P)) ~B~q v ~Bq Bp Bq B~p & B(p v q) => Bq B ~q B~q & B(p v q) => Bp B(p v q)
Planning as Satisfiability NSF Intelligent Systems § Unless P=NP, no polynomial time algorithm for SAT § But: great practical progress in recent years o 1980: 100 variable problems o 2005: 100, 000 variable problems § Can we use SAT as a engine for planning? o 1996 – competitive with state of the art o ICAPS 2004 Planning Competition – 1 st prize, optimal STRIPS planning o Inspired research on bounded model-checking
Efficient Model Counting NSF Intelligent Systems § SAT – can a formula be satisfied? § #SAT – how many ways can a formula be satisfied? o Compact translation discrete Bayesian networks #SAT § Efficient model counting (Sang, Beame, & Kautz 2004, 2005) o o Formula caching Component analysis New branching heuristics Cachet – fastest model counting algorithm
Credits § Graduate students: o Don Patterson, Lin Liao, Ashish Sabharwal, Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu, Bill Pentney, Brian Ferris § Colleagues: o Dieter Fox, Gaetano Borriello, Dan Weld, Matthai Philipose, Tanzeem Choudhury § Funders: o NIDRR, Intel, NSF, DARPA
Скачать презентацию Understanding Human Behavior from Sensor Data Henry Kautz
7da329ca579be549d63e7ac2a33e3926.ppt