Interactive Systems Technical Design Seminar work Sensing

Interactive Systems Technical Design Seminar work: Sensing & Sensors Hannu Kaski Jukka-Pekka Laitinen Miika Vahtola

Introduction 1/4 Sensing • Webster: “To perceive by the senses, to detect automatically especially in a response to physical stimulus” • Way to achieve knowledge of the world (temperature, force, acceleration…) • • Human senses: sight, hearing, touch, smell and taste Vision usually seen as the primary sense and hearing secondary Exceptions to general rules like blindness or deafness Touch can also be important when interacting with systems • Haptic systems - systems that use touch (haptic feedback) e. g. force feedback joysticks • Smell and taste generally ignored within computer interfaces ISTD 2003, Sensing & Sensors

Introduction 2/4 • Human sensing capability in active touch • Differences • • • Length and velocity 10% Acceleration 20% Force 7% Mass 21% Viscosity 14% • Resolution • • Pressure 0, 03 N/cm 2 Transient temperature 0, 05 ºC Skin displacement 20 micro meters Surface texture 0, 1 micro meters ISTD 2003, Sensing & Sensors

Introduction 3/4 • Tactile user interface is one type of sensing oriented UIs (Webster defines tactile: “of or relating the sense of touch” ) • Interface which can be controlled by touching and may give tactile output • Input • Handheld / tablet computers • Computer input devices • Information kiosks (touch screens) • Output • Vibrator alarms (cell phones, pagers) • Force Feedback (Entertainment applications e. g. game controllers, robotic surgery) ISTD 2003, Sensing & Sensors

Introduction 4/4 Sensors • Webster: “A device that responds to a physical stimulus (as heat, light, sound, pressure, magnetism, or a particular motion) and transmits a resulting impulse (as for measurements or operating a control)” • Comprised of two basic parts - a sensing element and a transducer • Contact/contactless sensing • Sensors + signal processing and logic (AI) enable “sensing” in machine domain • Nowadays sensors have integrated microcontrollers • Sensor technologies are rapidly evolving • Drivers: miniaturization, cost, processing power, power consumption factors • In particular the fact that sensors are vital technology enablers for new applications ISTD 2003, Sensing & Sensors

Human vs. Machine Characteristics related to sensing Problem Complexity: Human vs. Machine • Object recognition • Linguistics • Extraction of Relevant Features from Sensor Arrays MACHINE HARD EASY • Judging Maximum Potential Benefit • Thresholding • Tallying EASY HUMAN • Arithmetic • Logic HARD © Thad Roppel from Auburn University ISTD 2003, Sensing & Sensors

Motivation • Sensors are vital technology enablers for new applications • When applied in a right way they will probably ease your everyday life (e. g. intelligent environments) • Context-aware computing • “Perception without the context of action is meaningless” àSensors enable context-awareness (sensor fusion important) • Ubiquitous and pervasive computing • Usability of those devices that can “sense” may be better, because sensors enable a more sophisticated user interaction à Possibly better user experience ISTD 2003, Sensing & Sensors

Implementation • Humans and computers “sense” differently à Machines can only emulate real sensing (i. e. human) with the help of different kind of sensors, signal processing, microcontrollers and logic • OBJECTIVE: To intelligently integrate multiple sensors and multiple sensor modalities (i. e. sensor fusion) to serve the needs of Human-Computer Interaction à More natural and intuitive interaction between humans and computer à “Smart interaction” usually requires a network of sensors working in concert à Remember to keep in mind that systems should be build for people not vice versa à Natural interaction as a design paradigm when possible ISTD 2003, Sensing & Sensors

Sensor selection 1 of 2 • According to www. sensors-ez. com there are 1022 sensor manufacturers and tens of sensor categories • Excellent source of sensor related information: www. sensorsmag. com • Capacitance sensors (based on charge sensing) • Cheap, simple, no calibration • Enables touch (position) and proximity sensing • Some issues that should be noticed when implementing: à Good quality ground reference à Low impedance connections à Keep connections short and of low inductance à Stop ringing by adding a series resistor • Photoelectric sensors (color sensing, lasers for distance sensing) • Reliable, versatile • Able to sense objects of almost any material, size and shape ISTD 2003, Sensing & Sensors

Sensor selection 2 of 2 • Other sensor categories • • • Acceleration & speed Acoustic (e. g. ultrasonic sensors) Displacement & motion Force, pressure & tension Light (e. g. IR) Position & tilt Presense & proximity RF Temperature & humidity Torque & vibration Optical imaging based sensors (e. g. cameras) ISTD 2003, Sensing & Sensors

How to ’sense’ using sensors: Sense-Model/Think-Act Loop m e a s u r a n d measure in volts, amps, ohms, henrys, farads, etc. transduce perception to electrical signal SENSOR ADC environment ACTUATOR transduce signal to heat, displacement, illumination, etc DAC © Mel Siegel from CMU ISTD 2003, Sensing & Sensors convert from signal to symbol compute control action convert from symbol to signal

How to implement? • STEPS to systematize the sensing process: 1. Decomposition of relevant context information acquired by sensors à Model of discrete facts and quantitative measurements 2. Build a system based on some sensor fusion system architecture (below is one example) © Mel Siegel from CMU ISTD 2003, Sensing & Sensors

Usual requirements for an implementation • Small & lightweight -> miniaturization (HDP/ASIC/MEMS) • • • Reliable Information security Biocompatibility Low power consumption Shock proof Low cost ISTD 2003, Sensing & Sensors

Application Proactive Furniture Assembly By Stavros Antifakos, Florian Michahelles and Bernt Schiele from ETH Zurich http: //www. vision. ethz. ch/projects/ A subproject of the Smart-Its Project that is funded in part by the Commission of the European Union and the Swiss Federal Office for Education and Science VIDEO: http: //www. vision. ethz. ch/publ/ubicomp 02. mov ISTD 2003, Sensing & Sensors

Application Introduction • an experimental case study with the IKEA PAX wardrobe • PROBLEM: The presentation of plans by today's instructions is neither sufficient nor satisfying • 3 usage modes were identified: Full-walk-through, Assistance-on-demand Rescue-from-trap • OBJECTIVE: To develop Proactive Instructions for Furniture Assembly -> better usability of instructions • Chosen approach was to immerse instructions into the objects of interest (i. e. parts of a wardrobe) ISTD 2003, Sensing & Sensors

IKEA’s assembly instructions ISTD 2003, Sensing & Sensors

Different ways to assembly the IKEA PAX wardrobe ISTD 2003, Sensing & Sensors

Assembly actions and possible sensor configurations to perceive the action ISTD 2003, Sensing & Sensors

Detection of actions • Simple Markov chains were designed for each action • States and state transition probabilities were modeled by hand -> investigations to use Hidden Markov Models in order to train those probabilities automatically are currently ongoing force sensor screwdriver (gyroscope) ISTD 2003, Sensing & Sensors accelerometer

Challenges • Precision vs. cost (sensors aren’t free) • Cheapest and most unobtrusive sensor configuration enabling a high recognition precision should be the goal • How to inform the user (assembler) about the next steps to be taken? • Parts giving notice (flashing leds, beeping) • Guidance through a PDA/wearable computer/smart phone (should be avoided) • Closed world assumption narrows down the possible applications • we have to be able to fully model all tasks ISTD 2003, Sensing & Sensors

Other applications 1 of 4 • Applications needing • • Proximity sensing Presense detection Position sensing New control interfaces etc. • Automotive • Controls and lighting • Safety -> Electronic Stability Program, Acceleration Skid Control, Brake Assistant, Anti-lock Braking system • Alarms and entry access controls • Computers • Peripheral, mouse and joystick controls • Tactile input/output devices (force feedback, in-keyboard ‘mouse’) • Handheld devices (PDAs, phones etc. ) ISTD 2003, Sensing & Sensors

Other applications 2 of 4 • Biomedical/Biometrics • Health care, personal fitness • Wearable, personal health systems like AMON • bio-sensors (pulse, blood pressure, blood oxygen saturation, body temperature, skin perspiration, ECG) • Robotic surgery (with PHANTOM™-like products) ISTD 2003, Sensing & Sensors

Other applications 3 of 4 • Smart environments (e. g. home, office) • Access controls • Room light switches, remote controllers (no push buttons) • Appliance controls (A/V & kitchen) • Hidden controls and alarms (in walls, furniture) • Object sensing (e. g. sense when somebody touches something they shouldn't) • Human presence sensing (e. g. automated lights and doors) • Hand-wave controls -> Make objects sense (e. g. automatic faucet, power-ups) • Wearable computing ISTD 2003, Sensing & Sensors

Other applications 4 of 4 • Disability/elderly Aids • electronic assistance devices àreduce need for pressure or pull strength • Safety • Tool auto-shutoff (dead-man switches) • Child detection in unsafe areas • Intrusion detection • Security • 'Smart Objects' - arbitrary objects as 'smart cards' (e. g. RFID) • Toys • Dolls, SONY’s Aibo, LEGO Mind. Storms ISTD 2003, Sensing & Sensors

Strengths / Advantages 1 of 2 • More natural interaction, unobtrusiveness and zero activation force à Flexible form factors à Better user experience and usability • More intuitive usage à Faster and easier to learn • Sensors can provide/acquire information not possible to perceive by human senses à HC interaction may work better than human-human interaction in some aspects (e. g. machines try to serve you proactively) à People can acquire additional information (e. g. health state) ISTD 2003, Sensing & Sensors

Strengths / Advantages 2 of 2 • Eases the life of people with disabilities • When deployed well, will make life easier, more comfortable and safer ISTD 2003, Sensing & Sensors

Limitations / Weaknesses 1 of 2 • Context-understanding is challenging • Integration of sensors is demanding because sensed information may have overlaps or even conflicts à Sensor fusion techniques (AI algorithms) • Decrease in user’s intentional control à Need for profiles • Increase in SW inferential burden • Fail decisions à Effect on user acceptance • Sensors don’t work in all conditions • Temperature, humidity, EMC and calibration issues ISTD 2003, Sensing & Sensors

Limitations / Weaknesses 2 of 2 • Accuracy vs. Cost à MEMS technology enables So. C implementations that are cheaper • Noise and bandwidth à Local processing of sensor data decreases bandwidth requirements à Better noise filtering techniques • Limited power supply • Processing of sensor data needs power ISTD 2003, Sensing & Sensors

Selected Industrial Players • Microsoft Corp. – Wireless Intelli. Mouse Explorer • Quantum Research Group – QTouch™& QMatrix™ • Sens. Able Technologies Inc. – PHANTOM™ • Sony Electronics Inc. – AIBO product family • The LEGO Group – Mind. Storms™ product family • VTI Technologies Oy – SCA 620 series z-axis accelerometer family ISTD 2003, Sensing & Sensors

Selected International Research Groups and Projects 1 of 3 • Carnegie Mellon University HCI Institute www. hcii. cmu. edu - GM/CMU Project: Driver-Vehicle Interface - Manipulation in a Virtual Haptic Environment Based on Magnetic Levitation - Robotic Assistants for the Elderly • ETH Zurich www. ethz. ch • Perceptual Computing and Computer Vision Group - Smart-Its [with Lancaster University (UK), University of Karlsruhe (GER), Interactive Institute (SWE) and VTT (FIN)] • Wearable Computing Laboratory - Wearable Microsensor Network - Advanced care and alert portable telemedical MONitor (AMON) • Max Planck Institute for Biological Cybernetics www. kyb. tuebingen. mpg. de/bu - Hap. Sys - High-Definition Haptic Systems - Cog. Vis - Cognitive Vision Systems - ECVision ISTD 2003, Sensing & Sensors

Selected International Research Groups and Projects 2 of 3 • MIT Media Lab www. media. mit. edu/research • Context-Aware Computing Chrysler 300 M IT Edition, Context-Aware Tables, Disruptive Interruptions, Electronic Necklace • Human Design Learning Humans, MIThril, Project Zaurus, Shortcuts • Nanoscale Sensing High-Resolution Interferometric Accelerometer • Object-Based Media Smart Architectural Surfaces • Responsive Environments Design Principles for Efficient Smart Sensor System, Functional Integration for Embedded Intelligence, Modular Platform for High Density Wireless Sensing, Wearable Badge • Robotic Life Sensate Skin, Sociable Robots ISTD 2003, Sensing & Sensors

Selected International Research Groups and Projects 3 of 3 • Tangible Media Door Collision Avoidance Sensor, Tangible Bits • Harvard Bio. Robotics Laboratory www. biorobotics. harvard. edu - Remote Palpation Instruments for Minimally Invasive Surgery - Vibrotactile Sensing and Display - Force Feedback in Surgery: An Analysis of Blunt Dissection ISTD 2003, Sensing & Sensors

Selected Finnish Research Groups and Projects • Tampere Unit for Computer-Human Interaction, University of Tampere • Multimodal Interaction Group www. cs. uta. fi/hci/mmig/projects. htm • Tactile User Interfaces • Multimodal Interface for Persons with Low Vision and/or Hearing Impairment • Recognition and Synthesis of Faces, Gestures, and Actions • Tampere University of Technology • Personal Electronics group www. ele. tut. fi/research/personalelectronics • Smart Home • Wearable Computing • Smart Clothing ISTD 2003, Sensing & Sensors

Companies and Research Groups in Oulu • VTT Electronics • University of Oulu/Department of Electrical and Information Engineering • Polar Electro Oy • Idesco Oy • Advanced Interactive Systems www. vtt. fi/ele/research/ais/ • Interactive Intelligent Electronics (IIE) www. vtt. fi/ele/projects/iie/ http: //www. ee. oulu. fi • Machine Vision and Media Processing Unit • Optoelectronics and Measurement Techniques Laboratory • http: //www. polar. fi • http: //www. idesco. fi Proximity/focus sensing Smart Phone interfaces: • J-P Metsävainio Design Oy http: //www. jpmdesign. fi • My. Origo Oy http: //www. myorigo. com ISTD 2003, Sensing & Sensors

Future Developments • In near term we will see “sensing” slowly become a mainstream feature in man-machine interfaces • Nanotechnology will offer new possibilities because then sensors are so unnoticeable • We won’t know if we have drunk or eaten a sensor • People’s acceptance? ISTD 2003, Sensing & Sensors

Thank you! • Any questions? ISTD 2003, Sensing & Sensors