GM-Carnegie Mellon Autonomous Driving CRL Curb Detector 1

GM-Carnegie Mellon Autonomous Driving CRL Automated Image Analysis for Robust Detection of Curbs Detect and classify features using learning-based method Project Leader Name & Functional Area Wende Zhang (GM R&D / ECS Lab) David Wettergreen (CMU) Timing Initial Midterm Final Date_______ May, 2014 i. May, 2015 May, 2016 Resources 2013 2014 2015 Total Material Cost[US$] Total Headcount (GM) Total Headcount (CMU) 100 k 0. 1 1 Description Curbs are important cues on identifying the boundary of a roadway. Drivers understand an appropriate parking spot as defined by the curbs when reverse or parallel parking. Detecting curbs and providing information to assist drivers is an important task for active safety. Curb location is also crucial to autonomous parking systems. Visual indications of curbs are widely various in the appearance. For example, under perspective imaging, projection of 3 -dimensional curbs into 2 dimensional image plane distorts most of the curbs’ geometry properties, such as its angle, distance, and ratio of angles. Also, all curbs might be seen different because of age, wear, damage and lighting. Methods of detecting, localizing, and classifying curbs must address this diversity. This is to say, there is not a fixed template or set of templates that could be applied to reliably detect curbs through images. Nevertheless visual appearance is how human drivers successfully detect curbs. Although physical structure can be sensed with some ranging sensors it not distinctive (two offset planes) or diagnostic of the roadway edge. Therefore we choose to pursue visual appearance. This new project will develop an automated curb detection through : • Choosing appropriate features, learning those features to detect, and classifying the detected curbs • Utilizing the calibrated camera to fuse the 3 D geometry information • One year development plan: detect, localize, and classify curbs using in-vehicle vision sensor with backward looking view with wide field of view Motivation/Benefits • Identify the boundary of a road way in urban driving • Understand an appropriate parking spot as defined by the curbs when reverse or parallel parking Deliverable / Technology Insertion into GM (What, When, Where) • Problem Definition: Survey of curbs • Data collection: Database of definite curb images and diverse curb images • Application: Detect curb features in perspective imagery • Experimental validation and performance analysis • Annual report GM Confidential

GM-Carnegie Mellon Autonomous Driving CRL Use Case Slides 3

GM-Carnegie Mellon Autonomous Driving CRL Assumptions • • Color monocular camera Known camera motion Known intrinsic parameters Maximum speed dependent upon frame rate 4

GM-Carnegie Mellon Autonomous Driving CRL Use Cases • Parking lots – Backward parking – Parallel parking • Driveways • Roadways – Single lane – Multi-lane 5

GM-Carnegie Mellon Autonomous Driving CRL 6 Parking lots C U R B GM • Scenario : Curbs exist behind of a vehicle; rear-view camera with wide field of view • Success : Detect and localize curbs on images; (Optional) estimate the distance from a vehicle to curbs C U R B

GM-Carnegie Mellon Autonomous Driving CRL 7 Parking lots C U R B GM • Scenario : Parking curbs exist behind of a vehicle; rear-view camera with wide field of view • Success : Detect and localize parking curbs on images; (Optional) estimate the distance from a vehicle to parking curbs C U R B

GM-Carnegie Mellon Autonomous Driving CRL Driveways GM • Scenario : Curbs exist at the side of the entrance of driveway; front-view camera with wide field of view • Success : Detect and localize curbs on images and indicates driveways as traversable path CURB Driveways CURB 8

GM-Carnegie Mellon Autonomous Driving CRL 9 Roadways • Scenario : Curbs exist at the side of the road; wide field of view camera • Success : Detect and localize curbs on images and indicates curbs as the non-traversable path and the boundary of road GM GM CURB

GM-Carnegie Mellon Autonomous Driving CRL Flow Chart Detection : Localize relevant curbs in each image Edge Segmentation Texture Tracking : Localize the detected curbs in remained images 10

GM-Carnegie Mellon Autonomous Driving CRL 11 Edge Detection Distorted Undistorted Bird’s-eye view Edge

GM-Carnegie Mellon Autonomous Driving CRL Segmentation 12

GM-Carnegie Mellon Autonomous Driving CRL Texture Classification 13

GM-Carnegie Mellon Autonomous Driving CRL Development Plan • • • Develop and test simple features Train classifiers to detect and localize curbs Evaluate classifier performance Add complex features Test quantify detection and localization performance • Train color classifiers to interpret appropriate parking spots • Motion Stereo to exploit 3 D geometry 14

GM-Carnegie Mellon Autonomous Driving CRL 15 Scheme Extract Features Edge Detection - Filters • Edges • Intensity differences • Gradients - Geometric consideration Classification - Horizontal Long features Thin features Color Texture Curvature Tracking - Appearance based tracking

GM-Carnegie Mellon Autonomous Driving CRL Data collection • Using 180 degree field of view camera • Install underneath the side mirror, tilt 45 degree down to the ground Sample images 16

GM-Carnegie Mellon Autonomous Driving CRL Camera Calibration • Wider field of view, more distortion • Camera calibration is necessary in order to find geometry constrains (e. g. , edges…) • Using OCam. Calib (Omnidirectional Camera Calibration Toolbox) to calibrate camera Sample undistorted images 17

GM-Carnegie Mellon Autonomous Driving CRL Shape Information • Edge detection • HOG feature 18

GM-Carnegie Mellon Autonomous Driving CRL 19 Edge Detection Input Image at t Undistorted Image Edge Detection Sequential RANSAC Extract Dominant Edges

GM-Carnegie Mellon Autonomous Driving CRL 20 Edge Detection Input Image at t Undistorted Image Edge Detection Sequential RANSAC Extract Dominant Edges

GM-Carnegie Mellon Autonomous Driving CRL 21 Edge Detection Input Image at t Undistorted Image Edge Detection Sequential RANSAC Extract Dominant Edges Two or Three parallel lines with small offsets are important cue for curbs

GM-Carnegie Mellon Autonomous Driving CRL HOG feature 22

GM-Carnegie Mellon Autonomous Driving CRL HOG feature 23

GM-Carnegie Mellon Autonomous Driving CRL HOG feature Input image HOG map Score map Output image 24

GM-Carnegie Mellon Autonomous Driving CRL 25 Prerequisite • The maximum distance of ‘Curb Detection’ from a vehicle should be defined. short medium long • Given extrinsic parameters and the maximum distance, the followings can be estimated. – Different size of HOG model – Region of interest

GM-Carnegie Mellon Autonomous Driving CRL Geometry calculation GM GM Local Area 2. 7 -3. 6 m (9 -12 feet) Maximum detect distance CURB 26

GM-Carnegie Mellon Autonomous Driving CRL 27 Examples Cadillac SRX http: //www. cadillac. com/srx-luxury-crossover/features-specs/dimensions. html Cadillac CTS http: //www. cadillac. com/cts-sport-sedan/features-specs/dimensions. html

GM-Carnegie Mellon Autonomous Driving CRL 28 GM Geometry calculation Maximum detect distance = 2. 1 meter CURB

GM-Carnegie Mellon Autonomous Driving CRL Image Sample with distance measure 1 m 1 m 1 m - The center of the camera is 1. 05 m from the ground. - The angle of the camera is 45 degree down from the horizontal. - ROI will be reduced. (Red transparent rectangle) - ROI will be changed based on the extrinsic parameter. 29

GM-Carnegie Mellon Autonomous Driving CRL Lane Markings • Since lane markings have strong edges, we need to eliminate outputs from lane markings. • Parts of images which contain lane markings can be removed by detecting white blobs. 30

GM-Carnegie Mellon Autonomous Driving CRL Result Video 31

GM-Carnegie Mellon Autonomous Driving CRL Performance Measure • Choose 300 testing images – Positive samples: images which contains full length of curbs – Negative samples: images without curbs • We consider curbs are detected when the horizontal length of the detected curbs are bigger than half of the horizontal length of image. – Since the size of image is 480 by 720, we consider curbs are detected and the sum of the length of the detected curbs are bigger than 360. 32

GM-Carnegie Mellon Autonomous Driving CRL 33 Performance Measure length of detected curb total length of image Groundtruth Positive Negative Positive 80 13 Negative 24 183 > 0. 5

GM-Carnegie Mellon Autonomous Driving CRL Future Works • Features of curb detection – Redundant information through multiple images • Include tracking system to recover false negatives – Continuity • Develop likelihood function to recover false negatives and remove false positives – Height • Front-view camera – Mount 180 degree field of view camera on the front bumper 34

GM-Carnegie Mellon Autonomous Driving CRL Front-view Camera Configuration 35

GM-Carnegie Mellon Autonomous Driving CRL 36