Efficient rigid body motion tracking with applications to

Efficient rigid body motion tracking with applications to human psychomotor performance assessment “Surgeons who trained on simulations performed the operation 29% faster and made 7 times fewer errors”. Annals of Surgery 2002 Haptica Pro. MIS May 2006 Gerard Lacey Computer Vision and Robotics Group Dept. of Computer Science Trinity College Dublin www. haptica. com | tel. 617 342 7270 | email. info@haptica. com

Overview of Talk • • Haptica Pro. MIS May 2006 2 Context for research – Haptica Ltd Laparoscopic Surgery Intro Solutions to Surgical Training Computer vision tracking Live multimedia annotation of training video The results The current position The Future

Inspiration for Pro. MIS • Previous Research • Mobile Robotics for Frail Visually Impaired Elderly • Bayesian Belief Networks for multi-sensor classification / context sensitive reasoning • • Haptica Ltd formed in 2000 Focus on human performance enhancement in critical tasks • • • Haptica Pro. MIS May 2006 3 Guidance for mobility Computer Assisted Docking Fraud detection in credit card payments Semantic Web search of patent databases Laparoscopic Surgery Training

Laparoscopic Surgery • Laparoscopy – Minimally Invasive Surgery • • Key Challenges • • • Loss of dexterity Fulcrum effect Loss of sense of touch Slower than conventional surgery Environment for learning • • Haptica Pro. MIS May 2006 Surgeons operate with long narrow instruments Watch their movements on TV monitors Rapidly growing due to patient demand cost savings in reduced patient stays 4 Apprentice model See one, do one, teach one Patients used for learning No objective assessment

Current Training Methods • Current Training Tools • Patients • Physical simulators - “Box Trainers” • Virtual Reality and Full Haptic Simulators Physical Sim System behaves like abdomen Realistic tactile feedback Realistic visual feedback Use the real tools Team approach to procedure Simulate specific surgical procedure Assess Trainee’s performance Use with computer based training Playback a video of their operation Simulate affect on of errors Simulate unusual pathologies Haptica Pro. MIS May 2006 5 VR Sim

Objective Skills Measurement in Medicine • Human observation of bench tests • Resnick – U Toronto (OSATS) • Track hand movements to measure surgical skills • Darzi – Imperial College (ICSAD) • Linking skills to patient outcomes • VR to OR trials • Key understanding of the skills to be learned • Cognitive X% + Affective Y% + Psychomotor Skill Z% • Surgery mainly 75% X & 25% Z with Z higher in minimally invasive surgery Haptica Pro. MIS May 2006 6

Key Skills in surgery • Key indicators of skill acquisition • The objective is Proficiency • The time proxy – How fast can it be done? • The repetitions proxy – How many times has it been done? • The errors proxy – have any of these listed errors occurred while the task was being completed? • Surgical Finesse – smooth movement / economy of movement • Path of surgical instruments in space Novice Haptica Pro. MIS May 2006 7 Expert

Measuring Economy of Movement Stereo Cameras Track Instruments Haptica Pro. MIS May 2006 8

Design of the Stereo system • • • Background simplification Camera calibration – essential step Tracking algorithm • • • Top 1/3 of the image Instrument identification Image de-warping Icon tracking Complications • Multiple instruments – • Occlusion • Correspondence disambiguation Haptica Pro. MIS May 2006 9

Vision Systems Design • • Off the shelf web cams (firewire) • • • High frame rate (15 fps or higher) • Key design factor in enabling low cost Augmented Reality Calibration of cameras to sub pixel accuracy Task trays manufactured to high accuracy Task trays rigidly mounted to high accuracy Bile duct AR Haptica Pro. MIS May 2006 10

The Augmented Reality system • Aligning parallel 3 D Universe’s • Vision System reference frame • Graphics Engine reference frame • Mechanical mounting reference frame • Registration • Optical Distortion • De-warping of images • Occlusion masks (instruments and objects) • Real-time tracking and AR display (on a laptop) • down and dirty in direct x / direct show Haptica Pro. MIS May 2006 11

Augmented Reality Concept • Annotate live video with Computer Mediated Learning content • Teaching Instructions • Feedback on Progress • Hybrid tasks • Adverse Events AR Demo Haptica Pro. MIS May 2006 12

Results • Pro. MIS Construct Validity Trials • RCSI / AMNCH • Emory University • Imperial College • Very high ability to discriminate novice from expert using economy of movement • Papers published in surgical fora • Other Validations • European IST prize • Irish Software Association Technical Innovation Prize • Society of Laparoendoscopic Surgeons Innovation of the Year award • Irish Design Institute “Highly Commended Design” Haptica Pro. MIS May 2006 13

Idea refinement – The importance of key reference customers • Concept to product • March 02 to December 03 • Patent May 02 • • • Basic concept – cardboard box Stage Gate™ type process Multiple Prototypes • with RCSI, Emory (US), Imperial (UK) • • • Haptica Pro. MIS May 2006 14 Production prototypes June 03 Production process set-up Oct 03 First sales to RCSI December 03

Current Position • Pro. MIS 2 Launched October 2005 • Hand Tracking for “hand assisted” Procedures • Model abdominal organs • Full procedure training for Colectomy • 8 cameras, tilt-sensor, animatronics • Still runs on a laptop… • Haptica Pro. MIS May 2006 15 7 Papers validating Pro. MIS published at most recent surgical congress

Pro. MIS 2. 0 Haptica Pro. MIS May 2006 16

Haptica Pro. MIS May 2006 17

Haptica Pro. MIS May 2006 18

Haptica Pro. MIS May 2006 19

Haptica Pro. MIS May 2006 20

Haptica Pro. MIS May 2006 21

Future of Augmented Reality and Video Indexation • Current Projects • Illicit Content detection • Colonoscopy: Automated Reporting tools - tagging • Colonoscopy: Automated cancer detection • Hand washing quality assessment • Ground Collision Warning System for Aircraft Haptica Pro. MIS May 2006 22