A New Correspondence Algorithm Jitendra Malik Computer Science

A New Correspondence Algorithm Jitendra Malik Computer Science Division University of California, Berkeley Joint work with Serge Belongie, Jan Puzicha, Alex Berg

Key contributions: Years 1 -4 The FAÇADE system for semiautomated modeling of architectural scenes n High dynamic range image acquisition n Image based lighting n Inverse global illumination for recovering reflectance and lighting properties n Segmented objects from range images n

Contributors Paul Debevec, now at ICT n George Borshukov, recipient of Technical Achievement Award 2001 with colleagues at Manex visual effects n Yizhou Yu, Asst. Prof. , UIUC n

What remains? High quality automated correspondence is essential n 3 D Structure recovery algorithms need to scale up n Geometric and reflectance properties need to be modeled for a much larger range of scenes than previously considered n

Towards better correspondence Humans use contextual information much more effectively than current algorithms. n Features are not robust to changes in viewpoint. n

How big a window?

The solution to the dilemma. Large windows capture more context but suffer from increased distortion. n Goal: Design a similarity measure which can tolerate affine distortion. n l Similarity should decrease linearly with the amount of distortion. l Cross correlation does not have this property

An example • Solution is to blur the signals, but how exactly?

Blurring the right way

Affine Robustness Condition • The similarity function s(f, f T) should be close to a linear function L of the amount of distortion m(T). • We can obtain an s that satisfies this condition: • Where B is a bounded distortion blur…

Affine Robust Feature The bounded distortion blur of a signal f is the Affine Robust Feature B(f). Constructively B is a linear mapping with: And we take 0 1 2

In 2 d Six oriented filters, half-wave rectified to provide 12 channels n Bounded distortion blur applied to each channel n Similarity is the sum of similarities in each channel computed separately n

Bounded Distortion Blur in 2 D

Comparing three techniques

Another example… Given points in one image, find corresponding points.

. . . Another application: Matching shapes model target Find correspondences between points on shape n Estimate transformation n Measure similarity n

Shape Context Count the number of points inside each bin, e. g. : Count = 4. . . Count = 10 F Compact representation of distribution of points relative to each point

Hand-written Digit Recognition n MNIST 60 000: l l l l linear: 12. 0% 40 PCA+ quad: 3. 3% 1000 RBF +linear: 3. 6% K-NN: 5% K-NN (deskewed): 2. 4% K-NN (tangent dist. ): 1. 1% SVM: 1. 1% Le. Net 5: 0. 95% n MNIST 600 000 (distortions): l l l n Le. Net 5: 0. 8% SVM: 0. 8% Boosted Le. Net 4: 0. 7% MNIST 20 000 l K-NN, Shape context matching: 0. 63 %

Conclusion A new image descriptor which is robust to affine image deformations n Preliminary results suggest that this could result in a considerable improvement in quality of correspondence for long baseline multiple view analysis. n

Plans for next 6 months Combine the use of the affine robust window features with the use of epipolar constraints and probabilistic matching. n Test technique on stereo and motion imagery. n Explore this in the context of an end to end system for scene reconstruction. n