Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations Blue. Eyes Human-Operator Monitoring System • System design • System implementation Team members • Future improvements • Project work summary • System demonstration Team mentor Paweł Kowalik Piotr Kubiaczyk Krystian Nowak Stanisław Osiński Tomasz Pużak Jan Kniat, Ph. D. Poznań University of Technology, Poland http: //www. put. poznan. pl
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Agenda • • Motivation What is Blue. Eyes (and what is not) ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Motivation • Human error - a frequent reason of catastrophes and ecological disasters – temporal indisposition – weariness – oversight • Monitoring of conscious brain involvement • Capable sensors available • Lack of complex solution exploiting wireless link technology, database systems and AI algorithms
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration What is Blue. Eyes ? • The complex solution for human-operator monitoring: – conscious brain involvement monitoring – triggering user-defined actions – data recording What is Blue. Eyes not? • Doesn’t predict nor interfere with operator’s thoughts • Cannot force directly the operator to work
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Who is Blue. Eyes for ? • Generic control rooms – Power station – Captain bridge – Flight control centers – Operating theatres – anesthesiologists • Common application – A simpler system version for drivers
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Blue. Eyes - benefits • Prevention from dangerous incidents • Minimization of – ecological consequences – financial loss – a threat to a human life • The reconstruction of the course of operator’s work
Blue. Eyes Human Operator Monitoring System Physiological foundations • Motivation • What is Blue. Eyes ? Human brain • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Preconscious brain Conscious brain • alogical • „automatic” • manages the knowledge • logical reasoning • makes decisions
Blue. Eyes Human Operator Monitoring System Conscious brain - operation modes • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Exploratory mode Planning mode
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Eye movements • Retina - central vision (macula) • Central vision and the conscious brain Saccade - a rapid eye jump to a new location assigned by the conscious attention process. Saccades presence Exploratory mode
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary The eye movement sensor • Eye position measuring - direct infrared oculography • Oxy- and deoxyhemoglobin measurment • Two axial accelerometer • Ambient light sensor • System demonstration Jazz Multisensor
Blue. Eyes Human Operator Monitoring System Wires, wires. . . • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration A serious limitation of: – mobility – comfort – safety
Blue. Eyes Human Operator Monitoring System Personal Area Network ! • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration Solution: A wireless Personal Area Network
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives • System overview • DAU • CSU • System implementation • Future improvements • Project work summary • System demonstration Design objectives • A mobile data acquisition device • A reliable real-time buffering, processing and recording • A clear visualization interface • Mechanisms for creating custom data analyzers • To ensure a possibility of distributing the processing among a number of computers
Blue. Eyes Human Operator Monitoring System overview • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives • System overview Data Acquisition Unit Atmel 89 C 52 microcontroller Jazz Multisensor Bluetooth Central System Unit Bluetooth Connection Manager Visualization Module Data Analysis Data Logger • DAU • CSU • System implementation • Future improvements • Project work summary • System demonstration Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system.
Blue. Eyes Human Operator Monitoring System DAU - features • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives • System overview • DAU • CSU • System implementation • Future improvements • Project work summary • System demonstration Data Acquisition Unit Atmel 89 C 52 microcontroller Jazz Multisensor Bluetooth • Lightweight • Runs on batteries - low power consumption • Easy to use - does not disturb the operator working • ID cards for operator authorization • Voice transmission using hardware PCM codec
Blue. Eyes Human Operator Monitoring System CSU - features • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives • System overview • DAU • Connection management • Data processing • Visualization • Data recording Central System Unit Bluetooth • CSU • System implementation • Future improvements • Project work summary • System demonstration • Access verification • System maintenance Connection Manager Visualization Module Data Analysis Data Logger
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives CSU - saccade detector • Saccade detector and conscious brain involvement level calculation Raw eye movement data saccade • System overview • DAU • CSU Differentiation • System implementation • Future improvements • Project work summary • System demonstration Conscious brain involvement level
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives • System overview CSU - Visualization Module • Raw and processed data visualization using: – VU-meters – pie-charts – time series • DAU • CSU • System implementation • Future improvements • Project work summary • System demonstration • audio and video data playback
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • Objectives • System overview • DAU • CSU • System implementation • Future improvements • Project work summary • System demonstration Data security • Only registered mobile devices can connect to the system • Bluetooth connection authentication • Bluetooth connection encryption • Access rights restrictions • Personal and physiological data encryption
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • DAU • CSU • Future improvements • Project work summary • System demonstration Implementation - DAU • Made from scratch by the team • Bluetooth module inside • Jazz sensor interface • Microphone and earphone sockets • 5 -key keyboard • Two lines, 8 characters LCD • Powered from 6 AA batteries
Blue. Eyes Human Operator Monitoring System Implementation - DAU • Motivation • What is Blue. Eyes ? • Physiological foundations • System design PCM codec board Microcontroller board • System implementation • DAU • CSU • Future improvements • Project work summary • System demonstration Batteries and Bluetooth module voltage level monitor
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • DAU • CSU • Future improvements • Project work summary • System demonstration Implementation - CSU • Standard MDI user interface • Custom-built Bluetooth communication routines • Intel USB camera • ID card programmer • Developed using Visual C++ 6. 0 • Deployment environment: Windows 2000, MS SQL Server
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • System demonstration • Future improvements • Project work summary Future improvements • DAU – small CMOS camera to monitor the operator’s point of gaze – single PCB (SMD technlogy) – low voltage ICs - Li. IO batteries power • CSU – data mining algorithms – advanced database encryption using e. g. AES algorithm
Blue. Eyes Human Operator Monitoring System Project work summary • Motivation • What is Blue. Eyes ? • Physiological foundations • System design • System implementation • System demonstration • Future improvements • Project work summary • Bluetooth and PAN technology acquired • New skills on designing software and hardware systems gained • A mobile device implemented • 31 KLOC of C++ code and 4 KLOC of 8051 assembler code written • All parts of the Project Kit efficiently used • New approach to system safety proposed and implemented
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations • System design Blue. Eyes Human-Operator Monitoring System • System implementation • Future improvements • Project work summary • System demonstration
Blue. Eyes Human Operator Monitoring System • Motivation • What is Blue. Eyes ? • Physiological foundations Thank you for your attention • System design • System implementation • System demonstration • Future improvements • Project work summary Blue. Eyes Human-Operator Monitoring System Poznań University of Technology, Poland http: //www. put. poznan. pl
Blue. Eyes Human Operator Monitoring System DAU - components microphone MC 145483 PCM codec earphone Bluetooth module Jazz Multisensor LCD display Atmel 89 C 52 microcontroller UART Beeper UART MAX 232 LED indicators Blue. Eyes ID card interface ID card Simple keyboard Data Acquisition Unit
Blue. Eyes Human Operator Monitoring System CSU - Connection Manager Operator Data Manager Bluetooth . . . Bluetooth Connection Manager • Data demultiplexer Bluetooth • Initializes • Data producer queues modules • Inquires for new Separated Raw Data devices in range streams Operator data Buffer Manager Alerts • Establishes Alert buffer connections. . . • Supports operator. . . roaming • Feeds Operator Raw Data Operator Buffer Manager with Manager Alerts Alert buffer physiological data
Blue. Eyes Human Operator Monitoring System CSU - Data Analysis • Custom data analyzers - decision tree based classifiers Example: Operator gesture recognition (simplified) Vertical Acceleration variance value high Horizontal Acceleration variance high N “YES” Y N Horizontal Acceleration variance high Y N No answer Y “NO”
Blue. Eyes Human Operator Monitoring System CSU - Data Analysis • Custom data analyzers - decision tree based classifiers • Automatic decision tree coefficients generation - C 4. 5 based algorithm Data samples Assigned category “Yes” . . . No answer. . . “Yes” “No” No answer Decision tree coefficients
Blue. Eyes Human Operator Monitoring System CSU - intermodule communication • Uses buffered thread-safe queues and producer-consumer scheme register at Data consumer creates Data producer Data queue Data consumer & Processing consumer producer Data queue Data consumer & Processing consumer producer Data queue
Blue. Eyes Human Operator Monitoring System CSU - intermodule communication Connection Manager Operator manager (OM) Alerts Data Analysis . . . OM OM Separated physiological data streams Visualization Module Processed data Recorded (off-line) data Data Logger
Скачать презентацию Blue Eyes Human Operator Monitoring System Motivation
7fd3af230eec7e89d0ed2c9f3ee643b0.ppt