Real-time Structured Light Depth Extraction Kurtis Keller

Real-time Structured Light Depth Extraction Kurtis Keller & Jeremy Ackerman Department of Computer Science University of North Carolina Chapel Hill, North Carolina, USA EI 2000

Divisions of talk · Application and reason for research · Comparisons of various, relevant depth extraction systems (structured light, etc. . ) · First design with large digital projector · Second design with real time custom projector · Improvements. . .

High speed structured light depth extraction for Augmented reality surgical procedures

Application: Medical l Virtual / Augmented Reality view from single incision. This information can be rendered with textures to surgeon’s viewpoint. Must be real-time, small, able to image shiny/curved surfaces.

Comparison of various, relevant Depth Extraction Systems l Stereo laparoscopes l l l Structured light depth Extraction l l l Traditional viewing of images Automated depth extraction poor Too slow with standard projector and camera Can be used on curved surfaces Laser scan l l Too slow for real time use Miniaturization

Surface Texture and Z depth scanning methods for real-time medical use

Structured light (Binary encoded) Start with coarse projected texture, then increase number of bars for finer resolution. 1 2 3

First System

First Design: Structured Light with large digital projector

First System v Not Real time v Utilized off the shelf (modified) components l l TI micromirror projector Custom lens to flex scope to laparoscope v 3 D scans in lab and animal cadaver tests v Bulky

First System (in operation) v Close to Real time (60 Hz frame imaging, 5 Hz updates) v Custom components l l Projector Lens Custom lens to flex scope to laparoscope v Off the shelf componets l l l TI Micromirror projector Laparoscope Pulnics progressive scan camera

Second System (real time)

Second System v Almost Real time (~10 Hz) v Utilized custom components l l Displaytech ferro-reflective LCD engine in custom projector Custom lens to flex scope to laparoscope v Fits in with standard surgical environment l l Custom projector snap onto standard medical laparoscope Light source 2 meters away from patient by fiber light Projector about the size of an orange, fitting to surgeons hand light weight All exposed surfaces surgical steel and autoclaveable.

High-speed Projector v Miniature size (50 mm Dia x 75 mm long) v Custom components l l Displaytech 640 x 480 180 Hz. engine Custom lens to laparoscope

High-speed Miniature Projector Exploded view Attached to Laparoscope

v Real time v High-speed components l l l Custom, 180 Hz projector 180+ Hz 512 x 512 pixel camera High speed graphics processor

Hardware Improvements l l l Color additive scanning by synchronized color wheel Single tube Stereo Laparoscope Reduction of Specular rejection problems l Increased light level bit depth l Image processing/rejection Color Wheel

FUTURE RESEARCH in HIGH SPEED STRUCTURED LIGHT l l l Obtain Ferro reflective (or other) displays which are not hampered by crystal migration If possible, purchase graphics processor and high speed camera from same vendor Full projection imaging is being performed, not micro internals that we emphasized. l l l Virtual Teleconferencing Inspection Surveillance