The Strategic Role of Vision in Cognitive Systems

The Strategic Role of Vision in Cognitive Systems and Robotics - A Perspective from the EU Research Programme ECCV 2010, Special session on "Research Funding for Vision", September 9, 2010 Cécile Huet – cecile. huet@ec. europa. eu Unit E 5 - Cognitive Systems, Interaction, Robotics DG Information Society and Media European Commission http: //www. cognitivesystems. eu 1

Outline ¥ ¥ History Projects portfolio Perspectives and future opportunities Conclusion 2

It all started with VISION: “Cognitive Vision” Initiative in 1998 (FP 5): Rationale ¥ Limitation of computer vision systems: ¤ Lack of ROBUSTNESS/VERSATILITY/SPEED ¥ Cognitive vision ¤ combine state-of-the-art image processing with cognitive reasoning: abstract representation, categorisation, memory organisation, self-adaptivity, contextual knowledge, behavioural/scene interpretation, … 3

Projects portfolio ¥ >140 projects - € 480 million EU funding. ¤ FP 5 (1998 -2002) Cognitive vision - 9 projects ¤ FP 6 (2002 -2006) Cognitive Systems + Advanced robotics - 45 projects ¤ FP 7 (2007 -2013) Cognitive systems, Interaction, Robotics 68 running projects + 22 projects under negotiation 4

FP 5 Cognitive Vision - achievements ¤ FP 5 – DETECT Real Time Detection of Motion Picture Content in Live Broadcasts -> detect/track brand appearance in sport videos 5

FP 5 Cognitive Vision - achievements ¤ FP 5 – ACTIPRET Interpreting and Understanding Activities of Expert Operators for Teaching and Education Recognition of hand arm gestures: original image with overlay and 3 D representation Long term goal: giving hints on how to clear the paper jam in a copy machine 6

FP 5 Cognitive Vision - achievements ¤ LAVA - Learning for adaptable visual assistants ¤ Categorization and interpretation of large numbers of objects, scenes and events, in real settings. • Recognition/Categorization: Occlusion – cluttered background – viewpoint-rotation-scale, … invariance, lighting conditions, … • Categorization - Cope with different forms of the same object (shape, colours, size, angle etc) and an open set of instances of the object • Scene understanding – disambiguation –> multi cue/information fusion – context, … Source: www 7

FP 6: Cognitive systems research - motivation Beyond vision…. Achieving naturalness, versatility, robustness at system level ¤ ¤ from constrained to real-world environments from application-specific to general solutions from component methods to a systems approach from a ‘monodisciplinary’ research effort to working across disciplines RESULTS: Integration of perception (attention mechanisms, affordances, multi-sensory/cue…), representation, learning, reasoning, decision-making, action and communication, cooperation (HRI/RRI), anticipation, … Strong multidisciplinarity (neurosc. , cog sci. , engineering, 8 AI, …)

FP 6: Cognitive systems projects - classification ¥ Robotics (navigation – manipulation) - action-perception(COSPALSPARK), affordances (MACS-PACO-PLUS), anticipation(MINDRACES), reasoning - understanding (COSY), attention, knowledge-reasoning (GNOSYS), observation (MATHESIS), decision-making (DECISIONS-INMOTION), reasoning & self-preservation (ICEA) , Joint Action (JAST), Active haptic(SENSOPAC), developmental (ROBOT-CUB) ¥ Interfaces - audio-visual- HHI/HRI(POP), HRI (PACO-PLUS, COSY), HRIRRI (JAST) – ECAS (RASCALLI) ¥ Assistive technologies - Blind (DECISIONS-IN-MOTION- CASBLIP) – driving assistance (BACS) – Augmented human action (PACO-PLUS) ¥ Scene analysis/digital content - audio-visual + attention (POP-DIRAC), language & information discovery (RASCALLI), description image/video + text (CLASS), man-made scenes interpretation (e. Trims), description of human behaviour(movement-facial expression) from video sequences (HERMES) ¥ + eu. Cognition / PASCAL 9

FP 6: Advanced Robotics projects - classification HRI • • FEELIX GROWING: FEEL, Interact, e. Xpress: a Global app. Roach to devel. Opment With INterdisciplinary Grounding INDIGO: Interaction with Personality and Dialogue Enabled Robots COMMROB: Advanced Behaviour and High-level Multimodal Communication With and Among Robots PHRIENDS: Physical Human-Robot Interaction: Dep. ENDability and Safety ROBOT – FOR HUMAN ASSISTANCE (environments: Firefighter/Urban pedestrian area/home) • • • VIEWFINDER: Vision and Chemiresistor Equipped Web-connected Finding Robots / GUARDIANS URUS: Ubiquitous Networking Robotics in Urban Settings Robots@home: An Open Platform for Home Robotics (platform-perception-navigation) SWARM – for human assistance or autonomous tasks • • • IWARD: Intelligent Robot Swarm for Attendance, Recognition, Cleaning and Delivery (+HRI) ROBOSWARM: Knowledge Environment for Interacting ROBOt SWARMs (IT infrastr. /Cleaning Hospital) GUARDIANS: Group of Unmanned Assistant Robots Deployed In Aggregative Navigation supported by Scent detection Network of Excellence / Coordination and Support Actions • • EURON: European Robotics Research Network CARE: Coordination Actions for Robotics in Europe Ro. Sta: Robot Standards and Reference Architectures (+benchmarking) RAWSEEDS: Robotics Advancement through Web-publishing of Data Sets 10

FP 6 – Vision: Examples ¥ CASBLIP: Cognitive Aid System for Blind People ¤ Sensory substitution: from vision to audio information ¥ DECISIONS-IN-MOTION: Neural Decision-Making in Motion ¤ understand the neural mechanisms underlying fast target selection and sensory-guided decisions and actions in humans in motion ¤ transfer this knowledge into an artificial visual system for automated robotic navigation. 11

FP 7 - Challenge 2: Cognitive Systems, Robotics, Interaction Challenges ¥ greatly improving robustness etc. requires rethinking the way systems are engineered -> theories are needed ¥ Scientific approach: understanding of what both natural and artificial systems can and cannot do, and how and why ¥ Equally importantly integration of disciplines: robotics, AI, computer vision, natural language, cognitive science, psychology, … mathematics, … philosophy of mind, … ¥ we need to be able to support claims that progress is being made ¥ identify real applications that would greatly benefit from such systems 12

FP 7 – CALL 1 – PROJECTS CLASSIFICATION Dexterous Manipulation GRASP CHIROPING CHRIS DEXMART ROBOCAST MIMICS EYESHOTS ALEAR ITALK ROSSI DIPLECS Cog. X LIREC HRI SEARISE SEMAINE PINVIEW PROMETHEUS POETICON Co. FRIEND SCOVIS SF SPARK II REPLICATOR Autonomous Robots Cognition for Human Assistance Sensor network Scene understanding CLASSIC EMIME Spoken language PASCAL 2 Machine Learning Multimodal interaction Medical – Rehab. /Surgery Perception & Sensors Cognition & Language emergence in robots 13 HMI

FP 7 – CALL 3 – PROJECTS CLASSIFICATION: SCENE UNDERSTANDING MULTI-ROBOT / SWARM SCANDLE ACOUSTIC SCENE ANALYSIS UNDERWATER Co 3 AUVs SHOAL AERIAL SFLY URBAN EUROPA FILOSE NAVIGATION HRI SCENE UNDERSTANDING CA in Cog Sys EUCOGII 14

FP 7 – CALL 3 – PROJECTS CLASSIFICATION FROM AN APPLICATION ORIENTED PERSPECTIVE INDUSTRY / MANUFACTURING BRICS ROSETTA IM_CLEVER HANDLE ECCEROBOT MOBILE MANIPULATION PROSTHETICS EUROPA STIFF HUMOUR GUIDANCE DELIVERY TRANSPORT FILOSE SHOAL SFLY ENVIRONMENTAL MONITORING ECHORD MANIPULATION SORTING OBJECTS ASSEMBLY SERVICE ROBOTICS & HRI MONITORING SURVEILLANCE SECURITY Co 3 AUVs SCANDLE REHABILITATION MOTOR SKILL LEARNING SEARCH & RESCUE SOCIALCOMMUNICATIVE SKILLS (multimodal interaction – TV-Host) HUMANOBS ROBOSKIN PROGRAMMING BY DEMO/ TOUCH-BASED SOCIAL INTERACTION 15

FP 7 – CALL 4 – PROJECTS CLASSIFICATION MULTIMODAL LANGUAGE BIO-INSPIRED LOCOMOTION MOTOR CONTROL HAPTICS + VISION IURO ROBOSOM RADHAR DEXTEROUS MANIPULATION AMARSI NIFTI THE AIROBOTS ALIZ-e HUMAVIPS TRIDENT GERT COGNITO FIRST-MM ROBOEARTH GARNICS TACO AUDIOVISUAL HRI MASH NAVIGATION eu. ROBOTICS VANAHEIM WWW SHARED DB: Robots actions SENSOR NETWORK/ HUMAN ACTION RECO CA in ROBOTICS 16 PERCEPTION: VISION

Shared resources: Contribute to and benefit from: Robo. Earth – Robots sharing a knowledge base for world modelling and learning of actions - http: //www. roboearth. org/ ¥ World Wide Web for robots: a giant network and database repository where robots around the world access and continually update to share information and learn from each other about their behaviour and their environment. ¥ Improve any robot’s 3 D sensing, acting and learning capabilities. 17

Shared resources: Contribute to and benefit from: MASH - Massive Sets of Heuristics for Machine Learning ¥ Design of complex learning systems integrating large and heterogeneous families of feature extraction modules. ¥ Open web-based platform where contributors will find tools to interact and communicate with each other, and the means to integrate their feature extractors in continuously running experiments. ¥ Quantitative evaluation of submitted modules ¥ Performance will be measured on action selection with a robotic arm and in a simulated 3 d environment, and object detection. ¥ http: //mash-project. eu/ 18

FP 7 – CALL Fo. F – Smart Factories: ICT for agile and environmentally friendly manufacturing – Target outcome c) Robotics-enabled production processes tested and validated in real-world environment PACKAGING CUSTOMPACKER Custom. Packer ROBOFOOT MANIPULATION/ HANDLING SORTING OBJECTS VISUAL INSPECTION VISUAL SERVOING SHOE HRI + HR COOP. : WORKER DETECTION + INTENTION RECOSKILL LEARNING PACKAGING ELECTRONIC CONSUMER GOODS TAPAS NAVIGATION / PLANNING: MOBILE MANIPULATION / MOBILE ASSISANT: LOGISTICS + ASSEMBLY PRECISION FLEXIBILITY/SCALABILITY RELIABILITY ROBUSTNESS VERSATILITY MANIPULATION NON-RIGID / HEAVY OBJECTS 19

FP 7 – CALL 6 – PROJECTS CLASSIFICATION (! Subject to successful negotiation) DEXTEROUS MANIPULATION NAVIGATION GOAL-DIRECTED/ANTICIPATION SENSOR NETWORK VISION + Object/scene understanding HRI/SOCIAL INTERACT 20 CA in ACS

What next? The 2010 Call for Proposals – FP 7 - ICT Call 7 - http: //www. cognitivesystems. eu INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED ¥ TARGET a): Research projects on Robotic systems operating in real-world environments: + functionalities + autonomy, safety, robustness, efficiency, and ease of use AS APPROPRIATE: + new materials and advanced sensor, actuator, effector, memory and control technologies. ->> validation through REAL-WORLD scenarios ¥ TYPE OF PROJECTS: STRe. P (Specific Targeted Research Projects) High risk/Focussing on specific research topic/component IP (Integrated Projects) System oriented ¥ ¥ BUDGET: 70 M€ DEADLINE: 18. 01. 2011 21

What next? The 2010 Call for Proposals – FP 7 - ICT Call 7 - http: //www. cognitivesystems. eu INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED ¥ TARGET d) Coordinated Action(CA) Fostering communication and co-operation between robotics and cognitive systems research communities: + knowledge sharing: EU, national, and international; + open source; R&D roadmaps, market potential, user acceptance, standardisation, education, ethics, socioeconomic impacts; outreach to relevant professional and general audiences. ¥ TYPE OF PROJECTS: CA ¥ ¥ BUDGET: 3 M€ DEADLINE: 18. 01. 2011 22

What next? The 2011/12 Call for Proposals – FP 7 - ICT Call 9 - http: //www. cognitivesystems. eu INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED ¥ Cognition and control in complex systems (IP and STRe. P) Cross-fertilisation between academic and industrial robotics research (IP) Targeted competitions for smarter robots (CA) ¥ INDICATIVE DEADLINE: 17. 04. 2012 ¥ ¥ 23

Perspectives ¥ Outcome so far and next steps ¤ New research questions -> need for further scientific progress ¤ Also technological progress and application motivated research, building on previous results – innovation in integration and implementation of results to improve robotics functionalities and features. ¤ Real world scenarios: source of research questions + validate research results ¤ Demonstrate progress: Benchmarking 24

CONCLUSION FP 7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios » Role of vision? 25

CONCLUSION FP 7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios » Critical!! 26

THANK YOU! http: //www. cognitivesystems. eu 27