智慧型家庭網路之技術與應用 Professor Yau-Hwang Kuo Director Center for Research

智慧型家庭網路之技術與應用 Professor Yau-Hwang Kuo Director Center for Research of E-life Digital Technology (CREDIT) National Cheng Kung University Tainan, Taiwan 1

Outline l l l l l Introduction Structure of Smart Home Network Realization of Device & Network Layers Agent-based Platform Affective HCI Integrated Perception Cognition Layer Smart Home Services Conclusion

Trend of Digital Home l l l House_n (MIT)、 Aware Home (Geogria Tech. )、 Interactive Workspace (Stanford Univ. )、 Mav. Home (UTA)。 Digital Home Working Group: HP, Intel, IBM, . . . ECHONET: Energy Conversation and Homecare Network. CELF: Consumer Electronic Linux Forum. OSGi: Open Service Gateway Initiative Easy Living: Microsoft

Scenarios of Digital Life smart digital housekeeper. 2. ubiquitous digital nursing agent. 3. affective digital tutor. 4. ubiquitous home security monitor. 5. ubiquitous home content service. 6. universal cyber circles. 7. ubiquitous universal messaging service. 8. personal knowledge warehouse/navigation. 9. nomadic personal digital secretary. 10. secure traffic navigator. 1.

Microsoft’s View for Digital Home Solution l Total l connectivity No more islands of functionality l Personalized l Customized entertainment, communications, and control l Ubiquitous l experiences access Your PCs, devices, and content, securely accessible everywhere

Microsoft’s View for Digital Home Solution l Technology “by invitation only”, not imposed l Highly personal and personalized space l Virtually random, unmanaged “build out” l Complex mix of products and services

Issues of Digital Home l 人機互動能否人性化？ robustness、 adaptability、 multi-modal collaboration 人性化互動特質。 l 感官、認知、情緒、協調、合作實現人性化互動的技術要素。 l ubiquitous multi-modal affective humanmachine interaction 數位家庭的人性化互動需求。 l

Issues of Digital Home (cont. ) l 人際互動能否得到提昇擴大？ l 去空間限制、去時間限制、去具限制、去安全限制。 l 家電間的協力合作能力能否得到提昇 ? l connectivity among appliances、 autonomous collaboration of appliances、 interoperability of appliances。

Issues of Digital Home (cont. ) l 人在數位生活空間的自由度是否得到提昇？可移動性、可轉移性、可調整性。 l ubiquitous integration home network、 location-awareness、 universal access、 multimodal human-machine interaction l

Issues of Digital Home (cont. ) l 人在數位生活空間的便利度是否得到提昇？ l 生活機能完整性、設備與網路無縫結合度、生活機能可獲性 (availability)、用戶干預度、操作易度、穩私與安全等。 l 人在數位生活空間所獲得的生活輔助機能能否得到提昇 ? Smart home network is necessary!

Goals: Infrastructure & applications Create a new life space supported by a smart home service network and attached digital appliances. l Develop e-services over the smart home network and digital appliances to realize a new life style. l Develop a service modeling and execution environment over the smart home network to realize various e-services. l

Goals: technologies l Develop l nomadic HCI technology Speech, vision, physiology, sensors. l Develop affective HCI technology l Develop agent-based home service network middleware. l Develop embedded platform & So. C for smart appliances.

Layered Structure of Smart Home Service Network Applications (health care, entertainment, surveillance, etc. ) Application Layer Service Model Execution Platform (script translation, scheduling, Qo. S) Emotion / Semantics / Behavior / Intention Understanding Cognition / Affection Layer Perception Layer Corpus of Knowledge (Ontology) Natural Language Processing (text, spoken) Inference Engine Integrated Perception Speech Vision (face) Vision (gesture) Physiology Smell Agent Layer Mobile Agent Platform Network Layer Home Network (802. 11, Bluetooth, Home. Plug) + Mobile Internet (SIP +3 G) Device Layer Home Comm. Gateway; Home Perception Server; Home Media Center Networked Physiology & Environment Monitoring Appliances Networked Microphones; Cameras; Speakers Wireless A/V Streaming Appliances

Device & Network Layers: types of digital appliances l Client-type devices l l l Gateway-type devices l l 802. 11 g-based multifunctional audio/voice adaptor 802. 11 g/MPEG-4 -based multifunctional video adaptor 802. 11 g/MPEG-4 -based smart IP camera Bluetooth-based ECG device Multimedia communication gateway Server-type devices l l House control server Human-machine interaction server Content server Application server

Device & Network Layers: relationship among server appliances house control & housekeeping devices 主人用戶端設備屋控伺服器 Wi. Fi/Home Plug 應用伺服器 FTTH/ 3 G/ Wi. MAX Wi. Fi/ Home Plug 通訊伺服器 Wi. Fi/ Home Plug 內容伺服器 Internet/ WWW CO Telephony A/V devices data store

Architecture of agent platform ASI_1_1 ASI_1_2 BIS_1_1 BIS_2_1 ASI SDH Scenario Server DB Register BIS PMS_1_1 PMS_1_2 PMS LKN_1_1 LKN_2_1 LKN FEA_1_1 FEA_2_1 User Request Script What to do ? Service Server Scheduling Algorithm FEA XML How to do ? Service Agent Location Server XML Where to do ? Common API Task agent

Agent-based Runtime Environment Execution environment: IBM Aglets system l Common API l

Adaptive Service Provider: architecture

Adaptive Service Provider: functionalities l Functionalities Registry mechanism for subsystem, device and functionalities l Service provider for user requests l Load balanced service scheduling algorithm according to system resources l Agent cooperation mechanism l

Adaptive Service Provider: components l Service server Subsystem and devices functionalities registration l Service portal for users l Monitoring each subsystem and device l l Service agents Provide service for each user request l Service composition l Task assignment and task agent dispatch according to predefined XML-based scenarios l

Adaptive Service Provider: components (cont. ) l Task agent Execute each functionality on each subsystem l Common API l l Service scheduling algorithm Provide a task list for service agent according to registry and pre-defined scenarios in database l A Petri net based & load balanced scheduling algorithm for adaptive service path in each subsystem and device l

Agent-based Middleware: mobility management l Location detection Device-followed type: mobile IP; signal analysis l Device-free type: speech interaction; vision monitoring. l l Seamless handoff and transcoding for ubiquitous service following l Roaming path tracking and prediction

Agent-based Middleware: appliance collaboration management l Collaboration among homogeneous appliances: data fusion, task migration. l Collaboration among heterogeneous appliances: multi-modal HCI. l Scheduling, concurrency control & synchronization of collaborative tasks. l Self-organization for service deployment

Agent-based Middleware: interoperability management l Device bridge l Protocol bridge l Transcryption l Transcoding l Content translation & adaptation

Agent-based Middleware: remote access management l Remote service deployment l remote service access l remote service management l auto-configuration l service re-direction l service aggregation l UI remoting

Agent-based Middleware: other management functions l load management: Client-server load partition l Server load sharing Load scheduling of appliance farm l l availability management Fault tolerance l Just-in-time activation of appliances l l service quality management

Affective Speech Conversation Synthesis ASR Speec h Tex t Emoti on Dialog System Tex t Speech Emoti on

Emotional Speech Synthesis Text Input Text Analysis Emotion Selection Database Selection Syntactic Analysis Unit Selection Sad Happy Neutral Angry User’s Action Emotional Speech Database Speech Smoothing Speech Segmentation

Behavior Understanding by Vision l High-Level behavior understanding from videos l State Machine l Human l Activity Recognition Two-Stage recognition process l Accident/Abnormal l behavior detection Context & domain knowledge Combination

System Architecture

Method – Activity Recognition l Activity l Level 1 - postures l l Recognition Posture Sequence Level 2 – motion/history l History Map Matching

Method – Behavior understanding l Behavior l Normal behavior l State Machine l l Abnormal behavior l l Activity + Contexts Normal behavior + domain knowledge Accident l Unreasonable activity + domain knowledge

Facial Expression Analysis Face Acquisition Segmentation Eye Region Facial Feature Extraction Deformation Extraction Motion Extraction Facial Expression Classification Representation Key frame Selection Recognition Eye Points Displacement YCb. Cr Image Sequence Color Mouth Region Vectors Mouth Points Fuzzy Neural space Region Of Interest Network Invariant Moments Optical Flow Key Frame Results

Integrated Perception: fuzzification of reference perceptual models Manipulate all kinds of perception in a uniform process to ease the perceptual integration. l Due to high vagueness of perception, fuzzy logic based approach is a good choice to establish the reference models of perception. l The reference models which fuzzify perceptual attributes and perceptual decision subspaces will be embedded into the integrated perception model. l

FL-based Acoustic Reference Model for Emotion Recognition speech corpus feature extraction AAU 1 model SVM clustering for emotion 1 SVM clustering for emotion 2 AAU 2 model … … … SVM clustering for emotion V fuzzification of acoustic features (AFs) and construction of acoustic action units (AAUs) AAUS model

FL-based Acoustic Reference Model for Emotion Recognition (cont. ) Adopt SVM clustering approach in the subspace of each emotion type to gather the clusters of acoustic training patterns. l Inspect all produced SVM clusters in the whole feature space and merge the highly overlapped clusters. l Each cluster is modeled as an AAU represented with its fuzzy cluster center where each feature is a fuzzy set whose membership function is determined by the least-square curve fitting approach on the feature values of training samples included in the cluster. l

FL-based Acoustic Reference Model for Emotion Recognition (cont. ) The mapping between AAUs and emotion types is dependent on the SVM clustering result of each emotion type. l Each emotion type is associated with a set of clusters of acoustic samples. The weight of each cluster is determined by the ratio of the number of samples it contains with respect to the total amount of samples of the same emotion. l

FL-based Facial Reference Model for Emotion Recognition graphical head model morphological process to simulate AUs FACS AUs identification process correspondence feature points (FPs) extraction process Membership grade fuzzy logic based reference model for FACS Membership grade FAU 1 FAUi FAU 2 FAUj FP 1 value FAUk FP 2 value

FL-based Facial Reference Model for Emotion Recognition (cont. ) Intend to construct a computational reference model for FACS action units based on the measurable features of facial expression. l An approach similar to the construction of acoustic reference model is adopted. l The training samples are generated from a generic head model with necessary morphological manipulation. l

FL-based Facial Reference Model for Emotion Recognition (cont. ) l The membership functions will be determined by the least-square curve fitting approach according to the sample patterns produced from the morphological process. l Each AU may just represent a partial facial expression and relate to more than one emotion.

Fuzzy Neural Network for Integrated Emotion Recognition {< total ordering of emotion types>, group level of agreement} Fuzzy group decision process emotion type layer epresentative concept layer Fear FAU 1 Anger FAU 2 Surprise Fear Anger FAUK AAU 1 AAU 2 Surprise AAUS scaled feature layer primary feature layer FPn FP 1 Face Features Expression AF 1 AFm Acoustic Features

Fuzzy Neural Network for Integrated Emotion Recognition (cont. ) All kinds of perceptual information are fused by the FNN model to realize emotion recognition. l Each appliance will have an instance of the corresponding FNN to join the emotion recognition job. l A two-layered (emotion type & concept layers) BP learning algorithm is adopted by using the training samples in constructing reference models. The fuzzy group decision process does not join the learning. l Scaling input value to [0, 1] in the second layer is realized by the membership function of the corresponding fuzzy set. l

Fuzzy Neural Network for Integrated Emotion Recognition (cont. ) The links between AUs and scaled features are not fully connected. l The FAU/AAU nodes realize normalized weighted sum for the membership grades of input features weighted by their respective link strength. l Each emotion type node determines output value by the normalized weighted sum of its inputs from the representative concept layer. l

Cognition Layer: understanding and response l Understand the semantics of multimodal expression. l Classify and recognize the intention/ need/emotion of semantic expression. l Summarize the semantics of multimodal expression according to classified result.

Cognition Layer: understanding and response (cont. ) l Predict the user behavior sequence according to the classified result. l Schedule the response sequence according to the prediction result. l Determine the instant response.

Stimulus Perception spoken language Cognition Semantic Expression Speech Features Processing Extraction Concepts Conceptualization Event Detector (Neural Networkbased Approach) Events gesture face expression Vision Processing Personal Contextual Event / Ontology Rules Emotion Log Emotion Attributes Recognition Signal physiological Processing signals text Response Video Processing Speech Processing Application Control Emotion Types Semantic Summary Emotion Event Stimulus Sequence Response Case base Templates Emotion Episode Discovery Stimulus Semantic Summary Extraction User Behavior Prediction (Episode-based Approach) Emotion Episodes Response Prediction Result Instant Roadmap Response Determination Scheduling

Smart Home Services nomadic content services l health care by integrated perception l smart home surveillance l smart e-mail and calendar arrangement l

Conclusion l Life style of human being will be heavily affected by ICT, but the technological gap is still big. l Ubiquitous HCI and OCI technologies will be important to realize digital life style. l Cognitive computing and affective computing are important to improve the effectiveness of HCI technology.

Description of Context-Aware Middleware User Profile Admission Control Personal Agent Context Reasoning Context Aggregator Resource Management Wrapper Device Service Agent

Context-Aware Middleware Architecture Location Detection Speech Recognition Posture Recognition Identification Interface Context Resource JADE UPn. P Wrapper Agent Platform Bundle Management Reasoning Bundle (Service Scenario) OSGi Platform JAVA Virtual Machine Operation System Bundle Repository Context Aggregator and Ontology Reasoning