BRAIN INSPIRED COGNITIVE SYSTEMS BICS 2010 16 TH

BRAIN INSPIRED COGNITIVE SYSTEMS (BICS) 2010 16 TH JULY 2010 MADRID, SPAIN CORRELATION BETWEEN EYE MOVEMENTS AND MOUTH MOVEMENTS TO DETECT DRIVER COGNITIVE DISTRACTION afizan azman : a. azman@lboro. ac. uk qinggang meng : q. meng@lboro. ac. uk eran edirisinghe : e. a. edirisinghe@lboro. ac. uk

FACTS AND STATISTICS q. Department for Transportation (Df. T) in UK has reported in 2007, that 92% of passenger travel is by road. q. BBC News has been reported that in year of 2008, there were 2, 538 people were killed on Britain’s roads a. azman, q. meng, e. edirisinghe

RESEARCH OVERVIEW To propose a comprehensive and effective system to detect drivers’ cognitive distraction in a real time via physiological measurement a. azman, q. meng, e. edirisinghe

RESEARCH OBJECTIVE q. To suggest new features for cognitive distraction detection – lip and eyebrow movements (future work) q. To use data analysis approaches/techniques- Dynamic Bayesian Networks q. To use face. API toolkit for lip and eyebrow movement detection. a. azman, q. meng, e. edirisinghe

RESEARCH PLAN qcollect real-time data on driver visual and cognitive behaviourmodelling process qrecognize what the driver is doing (using contextual information such as manoeuvres, actions and states) qpredict the alertness level of the driver qdesign an interface to assist the driver a. azman, q. meng, e. edirisinghe

DRIVING SAFETY ISSUE Related to distraction q. Driver state affecting factors i. Fatigue ii. Monotony iii. Drugs iv. Alcohol q. Driver trait factors i. Experience ii. Age q. Environmental factors i. Road environment demands ii. Traffic demands Vehicle ergonomics a. azman, q. meng, e. edirisinghe

MANUAL VISUAL DISTRACTION COGNITIVE a. azman, q. meng, e. edirisinghe

COGNITIVE DISTRACTION Cognitive produces(output) distraction or causes(input) distraction COGNITIVE DISTRACTIO N COGNITIVE qsituation that might lead or shift a person from putting his attention doing something q. Harder to learn and measure – internal distraction q. Mind off the road q. Closely related to visual distraction q. Delay respond, slow brake, missed traffic light/signboard, unable to stay in a safe distance a. azman, q. meng, e. edirisinghe

COGNITIVE DISTRACTION EFFECT Any types of distraction can undermine- (a) vehicle control (b) event detection Fixation concentration= narrowing of the visual field scanned by observer. Cognitive load on driver affects- driver eye’s movement and driver’s event detection a. azman, q. meng, e. edirisinghe

DRIVER COGNITIVE MEASUREMENT available measurements which can be used to measure cognitive workload for drivers: q. Performance measure (primary tasks and secondary tasks)primary is continuous(lane keeping), secondary is noncontinuous(looking rare mirror) q. Physiological measurement- major organ, available for real time q. Rating scales- subjective measurers after activity is completed a. azman, q. meng, e. edirisinghe

NORMAL FACE a. azman, q. meng, e. edirisinghe

THINKING FACE Images from Google Image a. azman, q. meng, e. edirisinghe

USED FEATURES Eye movement-blinking, gaze direction, PERCLOS, saccade Head pose a. azman, q. meng, e. edirisinghe Pupil diameter Heart rate

PROPOSED FEATURES FEATURE SUB PARAM EYEBROWS Bright specularity Pink lip Dark aperture LIPS PARAM Rise duration (in ms) Movement magnitude (in mm) Color Motion Point 1 (lip corner position) Point 2 (lip height) Point 3 (lip corner position) Point 4 (lip height) Shape/template Fixation and Blinking a. azman, q. meng, e. edirisinghe Diameter Gaze MOUTH Height Width Pupil EYES Movement Rotation Movement Height Width

AUTOMATED PROCESS Data fusion and data mining, both is complementary processes that contribute automated process. The automated processes are involved with abductiveinductive (learning and discovery) and deductive (detection) process General properties Implementation Abduction Create model hypothesis for specific sets of data to explain that specific set. Induction Extend model hypotheses for representative sets of data to Mining (discovery of models) make a general assertion or explanation Deduction Apply models to create hypotheses to detect and classify (explain) the existence of target a. azman, q. meng, e. edirisinghe Fusing (detection)

ALGORITHMS A few approaches have been used by recent researchers: q. Regression- statistical modelling q. Ada. Boost- for feature selection q. SVM- popular technique q. BN- popular technique; DBN and SBN a. azman, q. meng, e. edirisinghe

DBN SVM SBN ACCURACY Most accurate. Almost similar accuracy rate with SVM-86. 4 Accurate. Almost similar accurate as DBN-85. 3 Significantly less accurate-80. 7 SENSITIVITY Very good. Can capture more differences in driver distraction and can generate more sensitive model Similar as SBN, but better because SVM accuracy is higher Similar as SVM DECISION/RESPOND BIAS Most liberal Similar as SBN. Similar as SVM HIT RATIO 93. 2 91. 1 87. 0 FALSE ALARM 30. 6 28. 9 34. 6 (worst) CONSTRUCTIONAL DIFFICULTIES Very difficult Average Easy a. azman, q. meng, e. edirisinghe

BAYESIAN NETWORK Is an attractive modelling tool for human sensing. It combines an intuitive graphical representation with efficient algorithms for inference and learning. BNs is a reasoning approach which provides a probabilistic approach to inference. q. A set of random variables make up the nodes of the network. Variables may be discrete or continuous. q. A set of directed links or arrows connect pairs of nodes. If there is an arrow from node X to node Y, X is said to be a parent of Y. q. Each node Xi has a conditional probability distribution P(Xi|Parents (Xi)) that quantifies the effect of the parents on the node. q. The graph has no directed cycles (and hence, is a directed, acyclic graph, DAG). a. azman, q. meng, e. edirisinghe

DYNAMIC BAYESIAN NETWORK q. Attractive modelling for human sensing tool. q. Probabilistic graphical modelling to do inference and learning. q. Encode dependencies among variable in an evolving time. q. Can fuse variety of information with contexual information and expert knowledge. q. Examples: Kalmann Filter and HMMs. a. azman, q. meng, e. edirisinghe

DYNAMIC BAYESIAN NETWORK DBN contains several time slices, where at every time slice, the nodes might give a different action as previous time slice. BNs for time series has the directed arcs and they should flow forward in time and not backward. sequence of observation {Y} by assuming that each observation depends on a discrete hidden state X=hidden state variable Y=observation variable a. azman, q. meng, e. edirisinghe

STATIC BAYESIAN NETWORK a. azman, q. meng, e. edirisinghe

DBN MODEL a. azman, q. meng, e. edirisinghe

INITIAL EXPERIMENT a. azman, q. meng, e. edirisinghe

EXPERIMENTAL SETUP 1 With Audio Task- Lab Setup This experiment will be conducted with an audio playing to the subject. Subjects are required to listen to a recorded streaming radio on air. Listen to the song and at the same time watching the video on the screen. The experimenter will ask questions to the subjects. Questions are based on the recorded audio, recorded video and trigger questions (questions to cognitively distracting the subject)- auditory-based questions, visual-based questions, conversation-based questions, arithmetic-based questions. a. azman, q. meng, e. edirisinghe

EXPERIMENTAL SETUP 2 Real environment- The experiment will take place in a real car on a real road: q. Put the facelab cameras on the car’s dashboard (real car) q. A video of the driver driving the car also will be captured q. Questions will be asked to the driver. Use the same questions q. Are going to use lane change keeping test (LCT) This setup needs to consider the contextual information. a. azman, q. meng, e. edirisinghe

PEARSON-R CORRELATION magnitude of the r-values showed the strength of the relationship between those two variables q 0. 0 to 0. 3 = negligible correlation q 0. 3 to 0. 5 = low correlation q 0. 5 to 0. 7 = reasonable correlation q 0. 7 or more = good or strong correlation a. azman, q. meng, e. edirisinghe

INITIAL RESULTS a. azman, q. meng, e. edirisinghe

SCATTER PLOT a. azman, q. meng, e. edirisinghe

THE END THANK YOU a. azman, q. meng, e. edirisinghe