Situational Awareness in Emergency Response Dr Sharad Mehrotra

Situational Awareness in Emergency Response Dr. Sharad Mehrotra Professor of Computer Science Director, RESCUE Project http: //www. itr-rescue. org

Crisis Response SYSTEM LEVELS • A massive, multi-organization operation • Many layers of government n n Federal: FEMA, FBI, CDC, national guard, . . State: Governor’s Office of Emergency Services (OES), highway patrol, … County: county EOC, police, fire personnel, … City: city emergency offices, police, firefighters, … • Volunteer Organizations n Red cross, organized citizen teams • Industry n Gas, electric utilities, telecommunication companies, hospitals, transportation companies, media companies …. POLICY FEDERA L AUTHORIT Y STATE LOCAL RESOURCE COORD EMC C 2 Incident command C 2 FIRST RESPONDERS VICTIMS LAW OPERATION S

Los Angeles County Emergency Management Organization LA County Emergency Management Council Board of Supervisors Chair of the Board Operational Area Coordinator Director of Emergency Operations Sheriff Operational Area Emergency Operations Center Disaster Management Area Coordinators Sheriff Contact Stations Other Entities Emergency Mgmt Information System Cities of Los Angeles County (87) 3

Operational View of Response • Crisis Management n n n Field level operation Command control Usually local government in-charge • Consequence Management n Gather information • Field, Cities, Special districts, County departments, Other EOC sections/branches n n Analyze consequences with focus on the future Develop plan of action • Life safety, Property loss, Environment, Reconstruction n Establish who is responsible

Operations- Consequence Analysis Public Safety Potential need for: • Security for damaged/evacuated structures • Route management • Civil disturbance control • Casualty/Fatality collection points • Fire fighting/HAZMAT support. • • • Care/Shelter requirements Impact on poor Language, other cultural needs • Food/water distribution • Impact on schools • Impact on non-profit agencies

Operations – Consequence Analysis • • Need for building inspections Removal of hazardous materials Demolition/debris removal Transportation network – impact and restoration • Water/sewage/flood control system impacts Logistics Construction CONSTRUCTION & ENGINEERING • Impact of utility outages • Priorities for restoration • Impact on purchasing system • Impact on transportation • Priorities for transportation restoration • Other support

Role of Information in Response Hypothesis: Right Information to the Right Person at the Hypothesis Right Time can result in dramatically better response Response Effectiveness Quality of Decisions • lives & property saved • damage prevented • cascades avoided • first responders • consequence planners • public Quality & Timeliness of Information Situational Awareness • incidences • resources • victims • needs

Challenges in Situational Awareness State of infrastructure Surge demands Diversity of data sources Concerns of privacy & confidentiality Incompleteness & uncertainty in Data Multimodality and Diversity of Data Real time requirements Interorganizational relationships Lack of incentives Privacy & confidentiality concerns, fear of misuse Dynamically evolving needs Diversity of delivery mechanisms Variability in warning times &urgency Scale – size of impacted population Recipient state & characteristics

RESCUE Project The mission of RESCUE is to enhance the ability of emergency response organizations to rapidly adapt and reconfigure crisis response by empowering first responders with access to accurate & actionable evolving situational awareness Funded by NSF through its large ITR program

RESCUE Partners

Cross cutting issue at every level Security, Privacy& Trust RESCUE Research Social & Disaster Science context, model & understanding of process, organizational structure, needs Engineering & Transportation validation platform for role of IT research Information Centric Computing enhanced situational awareness from multimodal data Networking & Computing systems Computing, communication, & storage systems under extreme situations

Situational Awareness Research in RESCUE Extraction, synthesis, Interpretation Situational Data Management Decn. Support Tools

Approach • Multimodal multi-sensor signal processing n Robustness to noise – noise affecting one modality may be independent of the others. • E. g. , multimicrophe speech recognition with background noise n Complementary information in different modalities – certain events easier to detect in some modalities than others. By combining modalities we can build systems that detect complex events • E. g. , Tracking in audio. people is easier in video whereas speaker identification is easier • Exploit semantics & context for signal interpretation n Knowledge of domain can help interpret data, fill missing values, disambiguate.

Exploiting Semantics for Situational Awareness • How does the system obtain & represent semantics? n User specified • Language for specification of semantics, expressibility, completeness n learnt from data • expressibility, training set might not be available for supervised learning, noise in data may skew unsupervised learning • Principled approach to exploiting semantics to interpret data n Probabilistic models? • Efficiency n Most such problems are NP-hard • Generalizability of the approach n Can we design a generalized approach that can be used to work across diverse types of data and for diverse situational awareness tasks.

Event Detection from sensors • 2300 Loop sensors in LA and OC • Goal: Detect events such as “baseball game” from loop sensor count data. • Semantics: n n Historical traffic data both during game night and nongame night Data is, however, unlabelled. • Smyth et. al. -- TRBC 06, SIGKDD 06, ACM TKDD, AAAI 07, UAI 07

car count Detecting Unusual Events Ideal model car count Baseline model Unsupervised learning faces a “chicken and egg” dilemma (and others)

Inference over Time t+1 p Time, Day Event l a True Count q a Sensor State True Count Sensor State q Observed Count Note how many hidden variables are in this model

Detecting Real Events: Baseball Games Total Number Of Predicted Events Graphical Model Detection of the 76 known events Baseline Model Detection of the 76 known events 203 100. 0% 86. 8% 186 100. 0% 81. 6% 134 100. 0% 72. 4% 98 98. 7% 60. 5% Remember: the model training is completely unsupervised, no ground truth is given to the model

Entity Resolution Problem TODS 2005, IQIS 05, SDM 05, JCDL 07, ICDE 07, DASFAA 07, TKDE 07

Two Most Common Entity-Resolution Challenges Fuzzy lookup – reference disambiguation – match references to objects – list of all objects is given 3/19/2018 Fuzzy grouping – group together object representations, that correspond to the same object DASFAA 2007, Bangkok, Thailand 20

Example of the problem: Disambiguating locations DASFAA 2007, Bangkok, Thailand 22

Web Disambiguation Music Composer Football Player UCSD Professor Comedian Botany Professor @ Idaho

Context Attraction Principle (CAP) publication P 1 “J. Smith” if n n then n reference r, made in the context of entity x, refers to an John E. Smith entity yj SSN = 123 but, the description, provided by r, matches multiple entities: y 1, …, yj, …, y. N, P 1 John E. Smith Joe A. Smith Jane Smith x and yj are likely to be more strongly connected to each other via chains of relationships n than x and yk (k = 1, 2, … , N; k j). Can be translated into a graph connectivity analysis which can be interpreted using a probabilisitic interpretation.

Experiments: Quality (web disambiguation) By Artiles, et al. in SIGIR’ 05 By Bekkerman & Mc. Callum in WWW’ 05 25

GDF vs. Traditional (Robustness) 26

GDF vs. Context (Bhattarya & Getoor) 27

Semantics in IE • Extracting relations from free / semistructured text (slot-filling) • Exploiting semantics in IE n declaratively specified • Specified as (SQL) integrity constraints n n On the relation (s) to be extracted Learnt from data • Mine patterns and associations from the data

Declarative Constraints create table researcher-bios ( name: person title: thing employer: organization employer-joined: date doctoral-degree: degree doctoral-degree-alma: organization doctoral-degree-date: date masters-degree: degree masters-degree-alma: organization masters-degree-date: date bachelors-degree: degree bachelors-degree-alma: organization bachelors-degree-date: date previous-employers: organization awards: thing CHECK employer != doctoral-degree-alma CHECK doctoral-degree-date > masters-degreedate

Pattern mining over data Top 10 med unranked in US OUT T 1 PI PD MI MD BI Stanford CSU Tsinghua T 2 PI PD MI MD BI T 3 1989 2002 BI • Represent data as graph (RDF) • Mine interesting patterns n Including “graph associations” • Example above n Mostly people who have a Ph. D degree from a school outside the US also have their bachelors degree from a school out side the US.

Constraints in Action TUPLE (POSSIBLE) INSTANCES John Smith, Ph. D, UCI, 2000, MS, MIT, 1997, BS, UCI, 1995 John Smith, Ph. D, MIT, 1997, MS, MIT, 2000, BS, UCI, 1995 John Smith, Ph. D, MIT, 2000, MS, MIT, 1997, BS, UCI, 1995 CONSTRAINTS 1. Order of degree dates 2. No “toggling” of schools John Smith, Ph. D, UCI, 2000, MS, MIT, 1997, BS, UCI, 1995 John Smith, Ph. D, MIT, 1997, MS, MIT, 2000, BS, UCI, 1995 John Smith, Ph. D, MIT, 2000, MS, MIT, 1997, BS, UCI, 1995

Experimental Results: Improvement CONSTRAINTS ATTRIBUTE LEVEL CD 1. All (CS) Ph. Ds awarded after 1950 CD 2. Current position is from among a fixed list CD 3. Ph. D awarded only by a Ph. D awarding school TUPLE: CT 1. People do not “toggle” between schools CT 2. Dates of doctoral, masters, and bachelors degrees are in order CT 3. People do not work at the same place they graduate from CT 4. More likely that the grad school is US and the undergrad school is outside US (vs other way around) CT 5. The grad school rank is at least as good (or better) than the undergrad school rank n researcher-bios domain n n (upto) 300 training documents (Web bios) Test set > 2000 documents Use RAPIER + Schema (type) information as baseline Add several constraints Improvement in both precision and recall

Challenges • Language for specifying constraints. • Principled approach to exploiting constraints/ patterns for extraction. • Scalability/efficiency n n Naïve approach of enumerating all possible worlds leads to exponential complexity. Problem NP hard even with a single FD (e. g. , Year Best. Movie) Crash, 2005 Crash, 2006 Million Dollar Baby, 2005 The Lord of the Rings, 2004 The Lord of the Rings, 2005 Possible “worlds” (exponential !!) Crash, 2005 Million Dollar Baby, 2005 The Lord of the Rings, 2004 X X Crash, 2006 Million Dollar Baby, 2005 The Lord of the Rings, 2005 Crash, 2006 Million Dollar Baby, 2005 The Lord of the Rings, 2004

Summary • Situational Awareness research in RESCUE n n n Event detection, extraction, and interpretation from multimodal sensor data Situational data management (R. Jain, S. Mehrotra) Tools for decision support (S. Mehrotra) • Two approaches: n Exploiting multimodal and multisensor input • • • n Multimodal speech, multi-microphone recog. B. Rao, Speech enhanced video M Trivedi Bayesian framework for Multi-sensor event detection P Smyth, Exploiting semantics for interpretation • Text, entity disambiguation S Mehrotra • Sensor data P Smyth • Dynamic recalibration of video based event detection system exploiting semantics [MMCN 08] S. Mehrotra, N. Venkatasubramanian • Automated tagging of images using speech input exploiting context and semantics [Tech. Report 08] S, Mehrotra

Summary • Situational awareness applications requires techniques to translate raw multimodal signals into higher level events. • Extensive research on signal processing but much of it studies different modalities in isolation • Multimodal event detection and exploiting semantics to interpret data is a promising direction. • A principled, generalizable, and a comprehensive approach represents a major challenge and an opportunity. • Situational awareness tools built on such tools could bring transformative changes to the ability of first responders and response organizations to respond to crisis.

Connection to Cyber SA Most of this talk focussed on here. Techniques could translate for cyber awareness. Also, through monitoring physical systems they directly could impact cyber SA. interdependencies Physical systems Cyber Systems Adaptation, Security intercepts Adaptation, refinement Situational Awareness Of underlying Of physical cyber systems Systems Awareness of state of physical system helps gain cyber situational awareness and vice versa. I. e. , State of physical systems can serve as sensors for cyber systems and vice versa