Computer Science Department Technion-Israel Institute of Technology Geometric

Computer Science Department Technion-Israel Institute of Technology Geometric Active Contours Ron Kimmel www. cs. technion. ac. il/~ron Geometric Image Processing Lab

Edge Detection q Edge Detection: u The process of labeling the locations in the image where the gray level’s “rate of change” is high. n OUTPUT: “edgels” locations, direction, strength q Edge Integration: u The process of combining “local” and perhaps sparse and noncontiguous “edgel”-data into meaningful, long edge curves (or closed contours) for segmentation n OUTPUT: edges/curves consistent with the local data

The Classics q Edge detection: u Sobel, Prewitt, Other gradient estimators u Marr Hildreth zero crossings of u Haralick/Canny/Deriche et al. “optimal” directional local max of derivative q Edge Integration: u tensor voting (Rom, Medioni, Williams, …) u dynamic programming (Shashua & Ullman) u generalized “grouping” processes (Lindenbaum et al. )

The “New-Wave” q Snakes q Geodesic Active Contours q Model Driven Edge Detection “nice” curves that optimize a functional of g( ), i. e. nice: “regularized”, smooth, fit some prior information Image Edge Indicator Function Edge Curves

Geodesic Active Contours q Snakes Terzopoulos-Witkin-Kass 88 u Linear functional efficient implementation u non-geometric depends on parameterization q Open geometric scaling invariant, Fua-Leclerc 90 q Non-variational geometric flow Caselles et al. 93, Malladi et al. 93 u Geometric, yet does not minimize any functional q Geodesic active contours Caselles-Kimmel-Sapiro 95 u derived from geometric functional u non-linear inefficient implementations: n Explicit Euler schemes limit numerical step for stability q Level set method Ohta-Jansow-Karasaki 82, Osher-Sethian 88 u automatically handles contour topology q Fast geodesic active contours Goldenberg-Kimmel-Rivlin-Rudzsky 99 u no limitation on the time step u efficient computations in a narrow band

Laplacian Active Contours q q Closed contours on vector fields u Non-variational models Xu-Prince 98, Paragios et al. 01 u A variational model Vasilevskiy-Siddiqi 01 Laplacian active contours open/closed/robust Kimmel-Bruckstein 01 Most recent: variational measures for good old operators Kimmel-Bruckstein 03

Segmentation

Segmentation q Ultrasound images Caselles, Kimmel, Sapiro ICCV’ 95

Segmentation Pintos

Woodland Encounter Bev Doolittle 1985 q With a good prior who needs the data…

Segmentation Caselles, Kimmel, Sapiro ICCV’ 95

Prior knowledge…

Prior knowledge…

Segmentation

Segmentation

Segmentation Caselles, Kimmel, Sapiro ICCV’ 95

Segmentation q With a good prior who needs the data…

Wrong Prior? ? ?

Wrong Prior? ? ?

Wrong Prior? ? ?

Curves in the Plane q C(p)={x(p), y(p)}, C(0. 1) p [0, 1] C(0. 2) C(0. 7) C(0) y C(0. 4) C(0. 95) C(0. 8) C(0. 9) x C p =tangent

Arc-length and Curvature s(p)= | |dp C

Calculus of Variations Find C for which Euler-Lagrange: is an extremum

Calculus of Variations Important Example Þ Euler-Lagrange: Þ Þ Curvature flow , setting

Potential Functions (g) I(x, y) I(x) Image x g(x, y) x g(x) Edges x x

Snakes & Geodesic Active Contours q Snake model Terzopoulos-Witkin-Kass 88 q Euler Lagrange as a gradient descent q Geodesic active contour model Caselles-Kimmel-Sapiro 95 q Euler Lagrange gradient descent

Maupertuis Principle of Least Action p 1 Snake = Geodesic active contour up to some , i. e Þ Snakes depend on parameterization. Þ Different initial parameterizations 0 yield solutions for different geometric functionals Caselles Kimmel Sapiro, IJCV 97 y x

Geodesic Active Contours in 1 D Geodesic active contours are reparameterization invariant I(x) x g(x) x

Geodesic Active Contours in 2 D Gs *I g(x)=

Controlling Smoothness I Cohen Kimmel, IJCV 97 g -max

Fermat’s Principle In an isotropic medium, the paths taken by light rays are extremal geodesics w. r. t. i. e. , Cohen Kimmel, IJCV 97

Experiments - Color Segmentation Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Tumor in 3 D MRI Caselles, Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

Segmentation in 4 D Malladi, Kimmel, Adalsteinsson, Caselles, Sapiro, Sethian SIAM Biomedical workshop 96

Tracking in Color Movies Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Tracking in Color Movies Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Edge Gradient Estimators Xu-Prince 98, Paragios et al. 01, Vasilevskiy-Siddiqi 01, Kimmel-Bruckstein 01

Edge Gradient Estimators q We want a curve with large points and small ‘s so: q Consider the functional q Where is a scalar function, e. g. .

The Classic Connection Suppose We have and we consider a closed contour for C(s). and by Green’s Theorem we have

The Classic Connection q Therefore: q Hence curves that maximize are curves that enclose all regions where q We have that the optimal curves in this case are The Zero Crossings of the Laplacian isn’t this familiar? is positive!

The Classic Connection q It is pedagogically nice, but the MARR-HILDRETH edge detector is a bit too sensitive. q So we do not propose a grand return to MH but a rethinking of the functionals used in active contours in view of this. q INDEED, why should we ignore the gradient directions (estimates) and have every edge integrator controlled by the local gradient intensity alone?

Our Proposal q Consider functional of the form q These functionals yield “regularized” curves that combine the good properties of LZC’s where precise border following is needed, with the good properties of the GAC over noisy regions!

Implementation Details q We implement curve evolution that do gradient descent w. r. t. the functional Here the Euler Lagrange Equations provide the explicit formulae. q For closed contours we compute the evolved curve via the Osher-Sethian “miracle” numeric level set formulation.

Closed contours EL eq. GA C LZ GA LZ C C C Kimmel-Bruckstein IVCNZ 01

Closed contours EL eq. GA C LZ C+ e GA C Kimmel-Bruckstein IVCNZ 01

Open contours Along the curve b. c. at C(0) and C(L) Kimmel-Bruckstein IVCNZ 01

Open contours Kimmel-Bruckstein IVCNZ 01

Geometric Measures Weighted arc-length Weighted area Alignment Robust-alignment e. g. Variational meaning for Marr-Hildreth edge detector Kimmel-Bruckstein IVCNZ 01

Geometric Measures Minimal variance Chan-Vese, Mumford-Shah, Max-Lloyd, Threshold, …

Geometric Measures Robust minimal deviation

Haralick/Canny-like Edge Detector q Haralick suggested as edge detector Alignment Laplace Topological Homogeneity

Haralick/Canny Edge Detector q Haralick co-area h Thus, indicates optimal alignment + topological homogeneity

Closed Contours & Level Set Method y implicit representation of C Then, x C(t) Geodesic active contour level set formulation y C(t) level set Including weighted (by g) area minimization x

Operator Splitting Schemes q Additive operator splitting (AOS) Lu et al. 90, Weickert, et al. 98 u unconditionally stable for non-linear diffusion q Given the evolution write q Consider the operator q Explicit scheme u , the time step, is upper bounded for stability

Operator Splitting Schemes q Implicit scheme u inverting large bandwidth matrix q First order, semi-implicit, additive operator splitting (AOS), or locally one-dimensional (LOD) multiplicative schemes are stable and efficient given by linear tridiagonal systems of equations LOD: that can be solved for AOS: by Thomas algorithm

Operator Splitting Schemes q We used the following relation (AOS) q Locally One-Dimensional scheme (LOD) q Decoupling the axes and the implicit formulation leads to computational efficiency The 1 st order `splitting’ idea is based on the operator expansion

Example: Geodesic Active Contour y q The geodesic active contour model x Where I is the image and f the implicit representation of the curve q If f is a distance, then , and the short time evolution is C(t) y x q Note that and thus can be computed once for the whole image Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Example: Geodesic Active Contour f is restricted to be a distance map: Re-initialization by Sethian’s fast marching method every iteration in O(n). Computations are performed in a narrow band around the zero set Multi-scale approach: process a Gaussian pyramid of the image y x C(t) y x

Tracking Objects in Movies q Movie volume as a spatial-temporal 3 D hybrid space u The AOS scheme is y t u Edge function derived by the Beltrami framework Sochen Kimmel Malladi 98 x y t x q Contour in frame n is the initial condition for frame n+1.

Experiments - Curvature Flow

Experiments - Curvature Flow CPU Time

Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Information extraction Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Thin Structures Holzman-Gazit, Goldshier, Kimmel 2003

Segmentation in 3 D Change in topology Caselles, Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

Gray Matter Segmentation Coupled surfaces EL equations Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Classification (dogs & cats) walk run Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002 gallop cat. . .

Classification (people) walk Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002 run 45

Conclusions q Geometric-Variational method for segmentation and tracking in finite dimensions based on prior knowledge (more accurately, good initial conditions). q Using the directional information for edge integration. q Geometric-variational meaning for the Marr-Hildreth and the Haralick (Canny) edge detectors, leads to ways to design improved ones. q Efficient numerical implementation for active contours. q Various medical and more general applications. www. cs. technion. ac. il/~ron

Edge Indicator Function for Color q Beltrami framework: Color image = 2 D surface in space q The induced metric tensor for the image surface q Edge indicator = largest eigenvalue of the structure tensor I metric. It represents the direction of maximal change in Y X

AOS Proof: The whole low order splitting idea is based on the operator expansion