Overview of Life Logging September 4, 2008 Sung-Bae Cho
Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary
Advances of Digital Devices Right now, it is affordable to buy 40 GB (in 2003) In 3 years 1 TB/year is affordable! It is hard to fill a terabyte/year, but you can: Look at 9, 800 pictures a day (300 KB JPEGs) Read 2, 900 documents a day (1 MB files) Listening to audio or view compressed video 24 hours/day (it takes more than 256 kb/s to fill a TB in a year) Watch 1. 5 Mb/s video 4 hours each day. Source: Microsoft 2003 2/48
Advances of Mobile Devices Properties of Mobile Devices Multi-function Mobility Large Memory Personal Device Collect Various Log Data of Digital Convergence User’s Everyday Life 3/48
Everyday Life with Mobile Devices I’m always with you! 4/48
Available Information from Mobile Devices User Data Scheduler User Created Contents Personal Profile Video Photo Details on daily events Demographical information Preference Who is in the media? Human Relationships E-mail Address Book Bluetooth Call Log Duration Receiver/ Caller Time Who is nearby? SMS Log Mp 3 Player GPS Contents Receiver/ Sender How many people in the media? Sensor Log Usage Log Application Usage Where is the place? What is the activity ? Bio-sensor Sleep? Activity Level? Audio Noisy level? Vision Sensor Time Location Contexts of environments 5/48
Life Logging with Personal Digital Devices Personal Digital Device Always Bringing Context Data Contents Creation Life Digital (Experience) Information Life Record Human Memorize Experience 6/48
Necessity of Context-aware Computing Episodic Management of Personal Information Summary of Everyday Life Personal Information Retrieval based on Semantic Structure Context-aware Computing Integration of Various Type of Information Data Mining from Personal Information User Modeling from Personal Information 7/48
Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary
Definition of Context-aware Computing • Context : Dey & Abowd (1999) – Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, physical or computational object • Context-aware computing – The use of context to provide task-relevant information and/or services to a user, wherever they may be • Three important context-aware behaviors – The presentation of information and services to a user – The automatic execution of a service – The tagging of context to information for later retrieval • Importance of context – The context of user is changed frequently and drastically in ubiquitous and mobile environment (Pascoe, et al. , 1998) 9/48
Trends on Context Awareness • TEA: EU project for enabling context awareness – 1998 ~ 2000, with Te. Co, Starlab, Omega and Nokia – scenarios, technologies, market research and demonstrators • Dr. Whats. On concept project – concept and user scenarios for context aware office PDA device – main focus in usability and user interface design • Earlier the focus was in sensor-based context recognition – recognize and utilize information from user's activity and environment properties • Now more directions and possibilities exist – Context-aware computing – Context-aware communication – Context-aware services, connectivity, information retrieval, affective computing, etc. TEA Dr. Whats. On 10/48
Why Context Awareness is Difficult? • Context recognition is never 100% reliable – Contexts are vague, dynamic, overlapping, and ill-defined • Adaptive user interfaces are scary! – May need an adaptive, learning SW • Sensors & algorithms may need constant recalibration – They may also be too CPU intensive • Application development frameworks & support are missing – Security, privacy concerns a lot • Connectivity to environment – Limited I/O through wireless links – What short range connectivity technology should be used? • Productization – Where’s the business? – What everyday problems it really solves? – Where do we get sensors for sensor-based context recognition? 11/48
Some Technical Challenges • Understanding structure and behavior of context information – context are fuzzy, overlapping, and changing in time • Lack of sensing methods for context information from various sources • Lack of methods for fusing context information • Lack of format of context information • Sensor-based context recognition – hard to obtain reliable data – signal processing and recognition consume memory, energy and processing time – results may be ambiguous • Connectivity to environment and other devices – Are profiles available; how about location information? – Any privacy & security risks? 12/48
Why do we Need Context Awareness? Maybe avoid disturbing people at wrong times Right information, right time, right place Better social acceptance of technology Increased user satisfaction New services Receding to background (Location-based services are the obvious ones) (Calm computing) 13/48
Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary
MIT Media LAB (1) • Area: Visual Contextual Awareness in Wearable Computing (1998) • Sensor: Vision • Probabilistic object recognition – Probabilistic dependence analysis based on neighbor vector & object recognition • O: object, M: measurement – Task recognition with HMM 15/48
MIT Media LAB (2) • Activity recognition based on accelerometer (2004) – 20 activities • Sensor: Accelerometer on diverse points of body • Geometric analysis – Mean, energy, frequency-domain entropy, and correlation • Classification method – Decision Tree C 4. 5, IBL, Naive Bayes 16/48
MIT Media LAB (3) the number of subjects collected under laboratory (L) or naturalistic (N) settings 17/48
MIT Media Lab (4) • Aggregate confusion matrix for C 4. 5 classifier – leave-one-subject-out validation for 20 subjects 18/48
e. Watch Sensor Platform • CMU Computer Science Lab in 2005 – Defense Advanced Research Projects Agency (DARPA) • Activity recognition, improving power consumption, location recognition • Hardware – LCD, LED, vibration motor, speaker, Bluetooth for wireless communication – Li-Ion battery with a capacity of 700 m. Ah • Sensors – a two-axis accelerometer (ADXL 202; +/- 2 g) – Microphone, light & temperature sensors • Method – multi-class SVMs – HMM based selective sampling 19/48
Classifier • multi-class SVMs with Gaussian Radial Basis Function kernels – frequency spectrum-based classification – time-domain-based classification with SVM • means, variances, square root of the uncentered second moment, the median absolute differences 20/48
University of Alberta • National ICT Australia Project: University of Alberta, Canada • Human activity & gesture recognition • Sensor – active, magnetic field, acoustic, laser, camera sensor • Method – Coupled hidden Markov model (CHMM) • Extended HMM model for combining and utilizing concurrent information stream more effectively (By M. Brand et al. , 1997) – M. Brand, N. Oliver, and A. Pentland, “Coupled hidden Markov models for complex action recognition, ” in IEEE Intl. Conf. Comp. Vis. Pat. Rec. , pp. 994 -999, 1997. 21/48
University of Bologna • Micrel Lab: University of Bologna, Italy (2004) • Research – Construction of Ubiquitous environments – Sensor-based gesture recognition • Sensor – Wireless MOCA (Motion capture with integrated accelerometers) sensor • Accelerometer, gyroscope • Small size, small power cosume • Wireless operation • Sticking on diverse points of human body • Method – Hidden Markov Model 22/48
Activity Recognition Summary • Mostly focused on static classification for pose recognition (relatively easy) USA MIT Static classification Finland Nokia Japan ATR ○ ○ Aus. ○ Temporal classification Curtin Univ. ○ Belgium China Starlab. SJT Univ. ○ ○ ○ Vision sensor ○ ○ Canada CMU ○ △ ○ Accelerometer ○ ○ ○ ○ △ △ ○ Instant behavior ○ Repeated behavior (Pose) ○ ○ ○ Geometric calculation ○ △ ○ HMM △ SOM Swiss ETH Zurich △ △ △ ○ ○ 23/48
Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary
Context-aware Mobile Device Trends • Increasing demands for the multi-functional and high-performance phone • Prospects – Increasing demands for Smart Phone – Decreasing basic functional phone • Enlarging role as a computing equipment – Internet surfing – External storage • Increasing investment for adding value of phone • Digital convergence with other functionality – Mp 3 player – Personal Media Player – Camera & Camcoder 25/48
Context-aware Phone on Media The official newsletter of the institute for Complex Engineered Systems (CMU) Sep/Oct 2003 New Scientist, Nov 2004 MIT Sloan Management Review Fall 2004 26/48
Sen. Say (Sensing and Say) • Carnegie Mellon University (2003) • A context-aware mobile phone – Adapting to dynamically changing environmental and physiological states – Manipulating ringer volume, vibration, and phone alerts – Providing remote callers with the ability to communicate the urgency of their calls – making call suggestions to users when they are idle – Providing the caller with feedback on the current status of the Sensay user • Sensors – accelerometers, light, and microphones – mounted at various points on the body 27/48
Sen. Say (2) • Context recognition by thresholding 28/48
Sen. Say (3) • Context classification based on self-organizing map 29/48
MIT Reality Mining Group • Utilizing Context application from the University of Helsinki (Raento et al. , 2005) • Capturing mobile phone usage patterns – from one hundred people (MIT students) – for an extended period of time • Providing – insight into both the users – the ease of use of the device itself • Method: Bayesian inference 30/48
Nokia • Nokia 7650 (2002) – Backlight time adjustment based on proximity & light sensor – Speaker-phone mode change by proximity sensor • Cancellation of speaker-phone mode near Proximity Sensor Light Sensor • Nokia 6230, 6820, 7200 – Presence-enhanced chat service • Presenting and sharing the user’s status – SMS messaging based on other’s status 31/48
Microsoft Research (1) 32/48
Microsoft Research (2) • Alarm method for incoming call – Quiet ringing • Volume down with hand touching – Acknowledging and ignoring calls • Calling response: Tilting to body of phone on the pocket • Notification stop: Holding phone with hand without movement on pocket – Target device for notifications • Device selection for alarm: Selecting the recent device if there are several devices of user – Vibration notification • Vibration mode change: Holding phone for a long time 33/48
Tilt Sensor & User Behavior • Upper: forward/back tilt • Below: left-right tilt 34/48
VTT Electronics (Finland) • Supported by Nokia • VTT Electronics – Advanced Interaction Systems – Context Awareness • Fuzzy/Bayesian Approach Backlight level & font size adjustment Summary of Context Recognition Procedure (From VTT Publications 511) 35/48
VTT Technical Research Center of Finland • Context representation Fuzzy logic • Context reasoning Naive Bayes, Markov Chain • Service Fuzzy Control Fuzzification Context Ontology 36/48
VTT Technical Research Center of Finland (2) • Sensor based contexts – Bottom: high-level contexts 37/48
VTT Technical Research Center of Finland (3) • Audio based context – Bottom: high-level context 38/48
VTT Technical Research Center of Finland (4) • Naive Bayes based classification 39/48
TEA Project • High-level context recognition – Method: Rule & SOM http: //www. teco. edu/tea 40/48
Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary
Summary • Key components for context-aware applications (life logging) – Sensor technology to acquire various contextual information – Intelligence technology to model complex contexts – Agent technology to provide seamless services • Future requirements – Context modeling and cognitive technologies to provide users with more advanced services 42/48
Dr. Whats. On (Laerhoven, 2003) • Concept and user scenarios for context aware office PDA device • Main focus in usability and user interface design – Information fusion: Sensor data, device status, personal preference, schedule – Peripheral attention: supporting appropriate information at appropriate time under prepared conditions • Scenario for a context sensitive phone of Nokia – A day of a Finland student Dude 43/48
Dr. Whats. On Scenario (1) • [Nokia’s Dr. Whats. On Concept Video, Urpo Tuomela, Nokia Research Laboratories] – Red mark: Abstracted information from sensor 44/48
Dr. Whats. On Scenario (2) 45/48
Dr. Whats. On Scenario (3) 46/48
Dr. Whats. On Scenario (4) 47/48
Homework #1: Due 9/9 • Survey the state-of-the-art research on the life logging at KIST, Nokia, Microsoft and MIT (10 min presentation per each institute) – KIST – Nokia – Microsoft – MIT 48/48
Скачать презентацию Overview of Life Logging September 4 2008 Sung-Bae
92f59bcc8e43f7c123a28fbd3e02ad93.ppt