Vision Guided Robotics and Applications in Industry and

Vision Guided Robotics and Applications in Industry and Medicine Matthias Rüther SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Contents ¨ Robotics in General ¨ Industrial Robotics ¨ Medical Robotics ¨ What can Computer Vision do for Robotics? ¨ Vision Sensors ¨ Issues / Problems ¨ Visual Servoing ¨ Application Examples ¨ Summary SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Robotics ¨ What is a robot? "A reprogrammable, multifunctional manipulator designed to move material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks" Robot Institute of America, 1979 ¨ Industrial – Mostly automatic manipulation of rigid parts with well-known shape in a specially prepared environment. ¨ Medical – Mostly semi-automatic manipulation of deformable objects in a naturally created, space limited environment. ¨ Field Robotics – Autonomous control and navigation of a mobile vehicle in an arbitrary environment. SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Robot vs Human ¨ Robot Advantages: – – – Strength Accuracy Speed Does not tire Does repetitive tasks Can Measure SSIP 2004 Graz ¨ Human advantages: – – – © Inst. For Computer Graphics and Vision, 2004 Intelligence Flexibility Adaptability Skill Can Learn Can Estimate

Industrial Robot ¨ Requirements: – – – – Accuracy Tool Quality Robustness Strength Speed Price Maintenance SSIP 2004 Graz Production Quality Production Cost © Inst. For Computer Graphics and Vision, 2004

Medical (Surgical) Robot ¨ Requirements – – – – Safety Accuracy Reliability Tool Quality Price Maintenance Man-Machine Interface SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

What can Computer Vision do for Robotics? ¨ Accurate Robot-Object Positioning ¨ Keeping Relative Position under Movement ¨ Visualization / Teaching / Telerobotics ¨ Performing measurements ¨ Object Recognition ¨ Registration SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004 Visual Servoing

Vision Sensors ¨ Single Perspective Camera ¨ Multiple Perspective Cameras (e. g. Stereo Camera Pair) ¨ Laser Scanner ¨ Omnidirectional Camera ¨ Structured Light Sensor SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors ¨ Single Perspective Camera SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors ¨ Multiple Perspective Cameras (e. g. Stereo Camera Pair) SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors ¨ Laser Scanner SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors ¨ Omnidirectional Camera SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors ¨ Structured Light Sensor Figures from PRIP, TU Vienna SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Issues/Problems of Vision Guided Robotics ¨ Measurement Frequency ¨ Measurement Uncertainty ¨ Occlusion, Camera Positioning ¨ Sensor dimensions SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Vision System operates in a closed control loop. ¨ Better Accuracy than „Look and Move“ systems Figures from S. Hutchinson: A Tutorial on Visual Servo Control SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Example: Maintaining relative Object Position Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3 -D Objects with Dynamic Handling of Occlusion SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Camera Configurations: End-Effector Mounted Fixed Figures from S. Hutchinson: A Tutorial on Visual Servo Control SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Servoing Architectures Figures from S. Hutchinson: A Tutorial on Visual Servo Control SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Position-based and Image Based control – Position based: • • • Alignment in target coordinate system The 3 D structure of the target is rconstructed The end-effector is tracked Sensitive to calibration errors Sensitive to reconstruction errors End-effector target – Image based: • • • Alignment in image coordinates No explicit reconstruction necessary Insensitive to calibration errors Only special problems solvable Depends on initial pose Depends on selected features SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004 Image of end effector Image of target

Visual Servoing ¨ EOL and ECL control – EOL: endpoint open-loop; only the target is observed by the camera – ECL: endpoint closed-loop; target as well as end-effector are observed by the camera EOL SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004 ECL

Visual Servoing ¨ Position Based Algorithm: 1. Estimation of relative pose 2. Computation of error between current pose and target pose 3. Movement of robot ¨ Example: point alignment p 1 p 2 SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Position based point alignment p 1 m p 2 m d ¨ Goal: bring e to 0 by moving p 1 e = |p 2 m – p 1 m| u = k*(p 2 m – p 1 m) ¨ pxm is subject to the following measurement errors: sensor position, sensor calibration, sensor measurement error ¨ pxm is independent of the following errors: end effector position, target position SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing ¨ Image based point alignment p 1 u 1 d 1 ¨ Goal: bring e to 0 by moving p 1 e = |u 1 m – v 1 m| + |u 2 m – v 2 m| v 1 p 2 u 2 v 2 d 2 c 1 ¨ uxm, vxm is subject only to sensor measurement error ¨ uxm, vxm is independent of the following measurement errors: sensor position, end effector position, sensor calibration, target position SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004 c 2

Visual Servoing ¨ Example Laparoscopy Figures from A. Krupa: Autonomous 3 -D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Registration ¨ Registration of CAD models to scene features: Figures from P. Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Tracking ¨ Instrument tracking in laparoscopy Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Summary ¨ Computer Vision provides accurate and versatile measurements for robotic manipulators ¨ With current general purpose hardware, depth and pose measurements can be performed in real time ¨ In industrial robotics, vision systems are deployed in a fully automated way. ¨ In medicine, computer vision can make more intelligent „surgical assistants“ possible. SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004