Sensing and Actuation End-to-end systems design for safety

Sensing and Actuation: End-to-end systems design for safety critical applications Dr. Elena Gaura, Reader in Pervasive Computing Director of Cogent Computing Applied Research Centre, Coventry University, e. gaura@coventry. ac. uk Dr. James Brusey, Senior Lecturer, j. brusey@coventry. ac. uk Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Cogent Staff and Ph. D students www. cogentcomputing. org Dr Elena Gaura e. gaura@coventry. ac. uk Expertise: Advanced Sensing; Advanced Measurement Systems; Ambient Intelligence; Design and Deployment of Wireless Sensor Networks; Distributed Embedded Sensing; Intelligent Sensors; Mapping Services for Wireless Sensor Networks; MEMS Sensors Dr James Brusey j. brusey@coventry. ac. uk Expertise: Industrial Robotics and Automation; Machine Learning; RFID; Sensing Visualisation Systems. Michael Richards richardsm@coventr y. ac. uk Expertise: 3 D CFD Modelling Costa Mtagbe Dr. Fotis Liarokapis f. liarokapis@coventry. ac. uk Expertise: Mixed reality systems; mobile computing, virtual reality for entertainment and education Gaura, Brusey Dr. James Shuttleworth j. shuttleworth@coventry. ac. uk Expertise: 3 D Graphics; data fusion and feature extraction, information visualization Expertise: Environmental monitoring Tessa Daniel danielt@coventry. ac. uk Expertise: Applicative Query Mechanisms; Information Extraction in Wireless Sensor Networks. Mike Allen allenm@coventry. ac. uk Expertise: Design and Deployment of Wireless Sensor Networks; Distributed Embedded Sensing. Bremen, February 2009. Tony Mo tony. mo@coventry. ac. uk Expertise: Wireless sensing for gas turbine engines Ramona Rednic rednicr@coventry. ac. uk Expertise: Body sensor networks, Posture John Kemp kempj@coventry. ac. uk Expertise: Advanced Sensing; Sensing Visualisation Systems. Dan Goldsmith goldsmitd@coventry. ac. uk Expertise: Middleware design and test-beds for WSNs

Talk Scope • development cycle for a multimodal wearable instrument • system design decisions • embedding actuation and its consequences • hurdles encountered…. Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Pointers • Timeliness: BSNs and WSNs are becoming commercial in their simpler forms; also coming out of research labs in elaborate versions; - Task Difficulty: Designing such systems needs teams of applications specialists, electronics engineers (most often) and definitely Computer Scientists; - Usefulness: proven, but, apart from being very useful, BSNs are a lot of fun to develop! Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Talk Structure • Part 1: Introduction and overview of the application • Part 2 : The deployment environment - a physiological perspective • Part 3 : System design • Part 4 : Enabling actuation - on-body processing • Part 5 : Implementation - software and hardware support • Part 6: Results analysis and evaluation Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Part 1: Introduction and overview of the application Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

WSNs: research motivation Start point: -Smart Dust (1998) – Pister ($35, 000) vision of “millions of tiny wireless sensors (motes) which would fit on the head of a pin” -sharing “intelligent” systems features (self –x) pushed to XLscale – millions of synchronized, networked, collaborative components Today: -Dust Networks - $30 mil venture (2006); -Tiny. OS – the choice for 10000 developers -make the news and popular press - fashion accessory & easy lobbying - big spenders have committed already (BP, Honeywell, IBM, HP) -technologies matured (digital, wireless, sensors) -first working prototypes; -getting towards “out of the lab”ISWC, Pittsburgh, 01/10/2008 Gaura, Brusey -social scientists are getting ready! Attention! Your spatio-temporal activities are recoded analyzed by the 20000 sensors wide campus net

WSNs –reality Market forecast: 2014 - $50 bil. , $7 bil in 2010 (2004) 2014 - $5 -7 bil. sales (conservative) Infineon tyre sensor 2011 -$1. 6 bil. smart metering/ demand response Industrial Markets- old and new; mostly wired replacements; generally continuous monitoring systems with “data-made-easy” features and internet connected Connecting 466 foil strain gages to a wing box Prompted by regulations and drive towards process efficiency or else… the “cement motes” from Xsilogy come with 30 min warranty! Research: mainly newly enabled applications; “macroscopes”/ “microscopes” ; adventurous money savings ideas Gaura, Brusey ISWC, Pittsburgh, 01/10/2008 Invensys asked a Nabisco executive what was the most important thing he wanted to know. The reply came without a moment's delay: "I'd like to know the moisture content at the centre of the cookie when it reaches the middle of the oven. "

WSNs - pushing the frontiers The motivational square Practical, application oriented research and deployments …forget about throwing them from the back of that plane!. . . Visions Making the most out of a bad situation Research space Commercial endeavours Research space Research/Adoption roadblocks Internet able Microclimate, soil moisture, disease monitoring Industrial needs Gaura, Brusey ISWC, Pittsburgh, 01/10/2008 Largest part of community Theoretical research for large scale networks

Why is it all so hard? …the WSN design space (Ray Komer, ETH, 2004) deployment mobility cost, size, resources and energy heterogeneity communications modality infrastructure network topology coverage connectivity network size lifetime other Qo. S requirements Gaura, Brusey Highly theoretical works Vs practical deployments ISWC, Pittsburgh, 01/10/2008

WSN challenges • Application specific (deployment, size, weight, etc) • System specific – the network is the SENSOR – – Distributed processing- system infrastructure Information extraction Scalability Robustness • Node specific – hardware integration/fabrication/packaging Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

WSN – challenges cont’d • • Physical environment is dynamic and unpredictable (Hw&Sw) Small wireless nodes have stringent energy, storage, communication constraints (Hw mainly) In-network processing of data close to sensor source provides (Sw, systems design) – Scalability for densely deployed sensors – Low-latency for in situ triggering and adaptation • Embedded nodes collaborate to report interesting spatio-temporal events (Sytems design) Embeddable Portable Adaptive Low cost Robust Self healing Self configuring Globally query-able Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Application related challenges • User requirements definition – novel technology hence this is hard • Capability/expectations mitigation • Lack of comparison measure at end-toend systems level !!!Consequence!!! Don’t underestimate the role of cyclic requirements/development/demonstration methodology Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Data acquisition phase • Sensors availability – MEMS technologies are just maturing - many physical sensors available • Digital or analogue output - Digitization required • Sensors compatibility with other systems components • SENSORS CALIBRATION, DRIFT AND FAULTS- Mostly uncalibrated, but…very cheap • Integration sometimes a problem Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Processing and comms challenges • Nodes size, weight, energy resources and processing capabilities – contrary constrains which need mitigating • Unreliability of wireless communications • Lack of debugging tools and wireless technology immaturity • Off-the-shelf comms encapsulation; unlexible protocols • Processing with little on much data Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Processors and Motes Hardware Sensor device interface Processor Memory Communications Form factor Renee Mezzanine card Atmel 8 bit 4 MHz 49 k. B 916 MHz, software modulation 484 mm 2 rectangle Mica 2 Mezzanine card (4 sensors) Analog Atmel 8 bit 8 MHz 644 k. B 916/433 MHz hardware modulation 19. 2 kbps 1800 mm 2 rectangle Mica 2 Dot Single sensor Analog Atmel 8 bit 4 MHz 644 k. B 916/433 MHz hardware modulation 19. 2 kbps 255 mm 2 disc Mica. Z Mezzanine card (4 sensors) Analog Atmel 8 bit 8 MHz 644 k. B 2. 4 GHz Zig. Bee 1800 mm 2 rectangle Intel mote Digital interface ARM 32 -bit 12 MHz 586 k. B 2. 4 GHz Bluetooth 900 mm 2 rectangle Mote Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Information extraction challenges • • Timeliness of acquired data Time synchronization Data storage Information extraction at source Co-opertive behaviour Global vs local treatment of the challenge Mitigating energy vs quality/detail vs timeliness vs system cost, size, etc Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Information delivery challenges • Raw data is too much saying too little • Huge range of user requirements motivated by – conservativeness of some engineering fields (ref - Energy sector, aerospace, defence) • Ease of interpretation by human in the loop – hard to accommodate with limited resources • Range of useful options continuously growing presently Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Actuation enablers • Are still in its infancy • Much to be gained from any breakthroughs here Enabling actuation has serious consequences in the overall system design Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

User satisfaction • Usually unknown/unpredictable till the development ends • Trail and error as the favourite methods presently • Huge range of reported work which failed to satisfy for all possible resons • Unreliability of the put-together systems is damaging to the filed Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

The Grand WSN challenge Facilitating the migration of pervasive sensing from future potential to present success Design space • Care for the un-expert user – “beyond data collection systems” VLS networks as Scientific instruments Permanent monitoring fixtures Gaura, Brusey “The network is the sensor” • Robustness, fault tolerance • Long life – across system layers and system components- in network processing &distribution • Maintenance free systems – scalability, remote programming &generic components/ infrastructure ISWC, Pittsburgh, 01/10/2008

Software - design features • designing for information visualization • designing for robustness and long life - Fault Detection and management • designing for practical applications • designing for robust services support • designing for information extraction- Complex Querying Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Designing for practical applications BSN End-to-end system design approach The problems: • Robustness of deployment • Technologies Integration • Fitness for purpose • Non-experts will use it!!! Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Matching application requirements with available technology in a safety critical application Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Project history • Commissioned late 2005 • Externally funded • Client: NP Aerospace Plc - protective clothing manufacturer for Defence - mostly for bomb disposal missions, de-mining, etc • Ph. D student project Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Project aim: Increased safety of missions through remote monitoring Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

The problem: the suit Environment • Increased heat production and reduced ability to remove heat results in storage • Thermoregulatory system becomes unable to correctly regulate core temperature • This may result in physical and psychological impairment • Increased risk of making an avoidable error and jeopardising the mission Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Possible solutions Manufacturer solution: add a cooling system to the suit Inadequate: a) Inefficient use due to human factors b) Distraction Alternative: a) in-suit instrumentation and continuous monitoring b) automated cooling actuation based on state Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Architecture • Sense-model-decide-act architecture • Two control loops – Rapid feedback to autonomously adjust cooling – Support for modifications to mission plans and investigation into the construction of the suit. Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Instrument Requirements • provide detailed physiological measurement - better insight into what is happening • support on-line and real-time thermal sensation estimates • report of useful information (rather than data) to a remote station and the operative • enable rapid assessment of hazardous situations • allow the provision of thermal remedial measures through control and actuation Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Part 2 : The deployment environment - a physiological perspective Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

UHS and Suit Trials • UHS- thermoregulatory system is unable to defend against increases in core body temperature • UHS - associated with significant physical and psychological impairment • Trials activity regime -four 16: 30 min: sec cycles – – – treadmill walking unloading and loading weights from a kit bag crawling and searching arm cranking standing rest seated physical rest Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Experimental data • Measurands- wired instrumentation – Heart rate – rectal temperature – skin temperatures (arm, chest, thigh and calf ) • Assessment – Subjective thermal sensation – twice per cycle, per segment and overall – Comfort – as above • Measurands - wireless – Skin temperature - 12 sites (symetrical + neck +abdomen) – Acceleration - 3 D - 9 sites – Pulse oximetry, heart rate, CO 2, galvanic Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Experimental data Figure 5. Core temperature responses (n=4; error bars are omitted for clarity) FS-NC=full suit, no cooling; NS= no suit Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Experimental data Figure 3. Typical heart rate response to EOD activity simulation (based on a single subject trial). FS-NC=full suit, no cooling; NO-S=no suit; W=walking; U=unloadin/loading weights; C=crawling and searching; A= arm exercise; R= seated rest. NB. Two of four subjects were not able to complete four activity cycles. Figure 4. Mean skin temperature responses (averaged over 4 subjects; error bars are omitted for clarity). FS-NC=full suit, no cooling; NS=no suit Gaura, Brusey Figure 6. Skin and rectal temperature over time for a subject wearing the full suit with no cooling. Note how core temperature rises with thigh temperature after the two merge. This experiment needed to be terminated as the subject could not continue. Figure 7. Self-assessed thermal sensation compared with chest skin temperature for subject 1. ISWC, Pittsburgh, 01/10/2008

Part 3: System design Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Constraints and design choices- I Suit related – – Mix of wired and wireless Multiple sensors to each node Wires in suit Size, power and weight a concern Suit modularity accounted for – multi-node BSN Three tiers of comms Two separate systems for: Sensors to node Node to base station Gaura, Brusey posture monitoring Physiological ? ? ? ISWC, Pittsburgh, 01/10/2008

Constraints and design choices- II Application related Intermittent comms - jammers, obstacles Maintaining autonomous operation - key Two modes of wireless comms In-suit, on body - short range, near field External to mission control - long range Buffering - avoid overflow Priority transmission Information extraction in-suit Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Constraints and design choices-III Safety critical – Cooling actuation – Operative alerts – Mission alerts – Hardware redundancy Information extraction in-network - major design implications Fault isolation and management Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Constraints and design choices-IV Instrument scope-dual – In field – In the lab - for physiological research and manufacturer research User led choice of operation In field max infromation output - thermal sensation, cooling status, trends, alerts x 2 Data on demand - temperature and other selected In the lab Data output - continuous - all including accel Information output - continuous Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Part 4: In-network modeling Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Processing • Basic filtering performed on sensor node – Allows rejection of invalid data and generation of alarms • Additional filtering using a Kalman filter on the processing nodes – Smooths data as well as providing estimates of error • Modelling of thermal sensation Include posture • Operative alerts CO 2 thresholding • Mission control alerts HR Gaura, Brusey Prediction models ISWC, Pittsburgh, 01/10/2008

Temperature and Thermal Comfort Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Temperature, Filters and Fusion – Kalman Filtering • Why filter? – Basic measurements may be too noisy – Can’t estimate gradient meaningfully otherwise • Why fuse measurements? – Two measurements are more reliable than one – Allow for / detect faulty sensors Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Thermal sensation Modelling • Takes skin temperature (and optionally core temperature) readings as input • Provides an estimation of thermal sensation, both per body segment and globally, as output • The main part of the model is a logistic function based on two main parameters: – the difference between the local skin temperature and its “set” point (the point at which the local sensation is neutral) – the difference between the overall skin temperature and the overall set point • Thermal sensation is given in the range − 4 to 4, with − 4 being very cold and 4 being very hot Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Zhang’s model Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Zhang’s model evaluation Figure 8. Overall thermal sensation over time during the activity regime with no suit. Figure 9. Overall thermal sensation over time during the activity regime with the full suit and with no cooling. Figure 10. Overall thermal sensation over time for a habituated subject with the full protective suit and no cooling. Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

HR and CO 2 Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Posture Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Posture Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Follow-up • New model needed • Activity needs monitoring – posture • Other physiological parameters have to be tried out –HR, galvanic response, heat flux • Model needs to predict not estimate/assess Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Part 5: Prototype implelentation Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Platform and sensors Picture of CO 2 and HR JOHN’ New DIAGRAM HERE Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Networking New pic from John and Ramona here • Wireless links between actuation / processing nodes • Wireless link between actuation node and remote monitoring point • Data/information buffered in case of link failure - may be uploaded at future point Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Temperature Component Data Flow Figure 13. Data and information system flow Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Posture Component Data Flow Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Remote Monitoring New pic from John paper New pic from Ramona paper • Main information display panel includes: – a 3 D figure showing the interpolated temperature distribution across the subject’s skin – the current average skin temperature, and – the current thermal sensation level • Other panels show the location and status of the sensors and the history of the incoming data Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Actuation • Reinforcement Learning algorithms (such as SARSA) can be used to develop a “policy” for controlling the cooling fan based on the “state” of the user • Action is to turn fan on or off and regulate volume • Utility is based on maintaining good comfort levels over time • Takes account of battery depletion, likely mission duration, posture, as well as current thermal comfort Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Operative alerts • Framework in place • Data and information processing flows readily available (piggy back on mission control) • Avoid false alarms - link to robustness and fault management Elena to • Sound considered at this stage but tactile change sounds good too • Research into HCI issues badly needed Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Evaluation and results Figure 19. Predicted thermal sensation including dynamic component of model Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

(a) (c) (b) (d) (e) (f) Figure 10. Skin temperature over time for (a) arm, (b) neck, (c) abdomen, (d) chest, (e) thigh, and (f) calf sites. The two leg sensors (thigh and calf positions) were placed on the right leg only. For several skin sites, temperature values were also obtained using a wired-in data logger (denoted "Logger"). The vertical lines in each graph show the start and end of activities. Each activity is represented by a number. Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Enriching the system for larger informational gain - posture monitoring Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Aim and postures • Dual aim – Direct activity information to mission control for • Supervision of mission - health hazards/colapse/restrains • Technical assessment - problems - controller expertise • Inferrence of abstract info by controllers – Parameter for thermal state prediction • 8 postures required: stand, walk, crawl, sitting, lying down (up, down, side x 2) Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Results and evaluation for posture monitoring Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

Review of tutorial and summary • exposition of design techniques and design choices • focus on an example • BSN- neither large nor widely distributed but there a number of fundamental requirements – the size of the nodes, wearability of the instrumentation, robustness, reliability and fault-tolerance, etc • they dictate the majority of the design and implementation choices. • Pursuing application driven design processes will enable the development of industrially strong systems which will increase confidence in the technology and contribute to its adoption in near future. Gaura, Brusey ISWC, Pittsburgh, 01/10/2008

WSN – theoretical wonders - Scoping of large scale applications - Complex problems solved for individual functional components/services - Theoretical proofs and simulation only - Lack of integrative work Visions led 1. Dust size- mm cube 2. Unplanned deployment 3. Distributed 4. Millions of 5. Re-configurable nets 6. Self-healing 7. Scalable 8. Autonomous 9. Information systems Gaura, Brusey 10. Collaborative decisions SENSE and SEND 1. Stack of quarters & miniaturization vs mote life trade-off 2. Planned, carefully measured; ID based 3. Gateway based – centrally controlled; backboned 4. Hundreds at most (Ex. Scal) 5. Hard coded 6. Prone to failure (more than 50% usually) 7. Only through complete re-design 8. Tightly controlled 9. Data acquisition – relay to base ISWC, Pittsburgh, 01/10/2008 10. Central post processing