Скачать презентацию Over-Parameterized Variational Optical Flow Tal Nir Alfred M Скачать презентацию Over-Parameterized Variational Optical Flow Tal Nir Alfred M

744d81e9b948f429015fa125953bf034.ppt

  • Количество слайдов: 67

Over-Parameterized Variational Optical Flow Tal Nir Alfred M. Bruckstein Ron Kimmel Technion, Israel institute Over-Parameterized Variational Optical Flow Tal Nir Alfred M. Bruckstein Ron Kimmel Technion, Israel institute of technology Haifa 32000 ISRAEL 1

What is optic flow? • Optic flow relates to the perception of motion. • What is optic flow? • Optic flow relates to the perception of motion. • Optic flow – the apparent motion of objects in the scene as seen on the 2 D image plane. 2

An image 3 An image 3

Warped image 4 Warped image 4

The corresponding optical flow 5 The corresponding optical flow 5

Applications of optic flow An important pre-processing for many visual tasks • Tracking. • Applications of optic flow An important pre-processing for many visual tasks • Tracking. • Segmentation. • Compression. • Super-resolution – requires high accuracy. • 3 D reconstruction (structure from motion). 6

Basic equations Brightness constancy equation u, v are the optic flow components between frame Basic equations Brightness constancy equation u, v are the optic flow components between frame t and t+1 Linearized brightness constancy equation 7

The aperture problem Only the flow component in the gradient direction can be determined The aperture problem Only the flow component in the gradient direction can be determined (normal flow). From an algebraic point of view this is an ill-posed problem An image with N pixels: N equations with 2 N unknowns. 8

Going around the aperture problem Looking for locations where the image has • “Multiple” Going around the aperture problem Looking for locations where the image has • “Multiple” gradient directions, • Discontinuous first image derivatives, • “Corners”. 9

The Lucas-Kanade method B. D. Lucas and T. Kanade, “An iterative image registration technique The Lucas-Kanade method B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision, ” Proc. DARPA Image Understanding Workshop, April, 1981. 10

Lucas-Kanade continued Solve the linear 2 x 2 system of equations 1. The “aperture Lucas-Kanade continued Solve the linear 2 x 2 system of equations 1. The “aperture problem” can occur in certain regions (zero eigenvalue). 2. Typically, the aperture problem does not appear in an exact sense. 3. Method may yield a sparse flow field estimate. 11

Neighborhood based methods • The flow in the patch can be described by a Neighborhood based methods • The flow in the patch can be described by a constant, affine, or other model. M. Irani, B. Rousso, S. Peleg, “Recovery of Ego. Motion Using Region Alignment”. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Vol. 19, No. 3, pp. 268 --272, March 1997 • The smoothness within the patch is inherently enforced. • Discontinuities of the model within the patch may cause inaccuracies. • The resulting problem is over-constrained. 12

Motion in a patch – Over constrained solution (Lucas-Kanade) Optical flow estimation – an Motion in a patch – Over constrained solution (Lucas-Kanade) Optical flow estimation – an ill posed problem 13 Our work Over-parameterized Variational

The variational approach B. K. P. Horn and B. G. Schunck, The variational approach B. K. P. Horn and B. G. Schunck, "Determining optical flow, " Artificial Intelligence, vol. 17, pp. 185 --203, 1981. Find the flow which minimizes the functional Composed of a data and smoothness (regularization) term The resulting Euler-Lagrange equations 14

Variational approach. Cont’. • Dense optical flow field (i. e. a vector at each Variational approach. Cont’. • Dense optical flow field (i. e. a vector at each pixel). • The smoothness (regularization) term enables the completion of the flow in locations with insufficient information. • Global solution – incorporates all the available information. • The best results are achieved by modern variational approaches. 15

T. Brox, A. Bruhn, N. Papenberg, J. Weickert “High Accuracy Optical Flow Estimation Based T. Brox, A. Bruhn, N. Papenberg, J. Weickert “High Accuracy Optical Flow Estimation Based on a Theory for Warping”, ECCV 2004. • L 1 non-linear data term with a gradient constancy term • L 1 smoothness term in x, y, t space (3 D) Euler-Lagrange equation for u (Γ=0) 16

Brox et. al. “High Accuracy Optical Flow Estimation”. Cont’. Three loops of iteration • Brox et. al. “High Accuracy Optical Flow Estimation”. Cont’. Three loops of iteration • Outer loop k. • Inner loop fixed point iteration in order to deal with the nonlinearity in Ψ. • Gauss-Seidel iterations are used in order to solve the resulting sparse linear system of equations. 17

Brox et. al. “High Accuracy Optical Flow Estimation”. Advantages • Solution in Multi-scale helps Brox et. al. “High Accuracy Optical Flow Estimation”. Advantages • Solution in Multi-scale helps to avoid being trapped in local minima – large motion (reduction factor of 0. 95). • The 3 D smoothness term solves the problem in the volume in contrast to the 2 D (two frames) solution. • The gradient constancy term reduces the sensitivity to brightness changes. • Choosing Ψ as an approximately L 1 function: • In the smoothness term it allows discontinuities in the flow field. • In the data term it reduces the sensitivity to outliers. • The addition of ε is for numerical reasons. 18

Results – Brox et al. 19 Results – Brox et al. 19

Our motivation stems from the smoothness term Weighted spatio-temporal gradient Penalty for changes in Our motivation stems from the smoothness term Weighted spatio-temporal gradient Penalty for changes in the optical flow Penalty for changes from an optical flow model 20

The proposed overparameterization model • Basis functions of the flow model • Space and The proposed overparameterization model • Basis functions of the flow model • Space and time varying coefficients The optical flow is now estimated via the coefficients The different roles of the coefficients and basis functions • The basis functions are selected a-priori, the coefficients are estimated. • The regularization is applied only to the coefficients. 21

Over-parameterization - one frame Conventional representation u u + v * Basis functions * Over-parameterization - one frame Conventional representation u u + v * Basis functions * * Coefficients v 22 + Basis functions * Over-parameterized representation

Over-parameterized functional The new regularization term penalizes for changes in the model parameters. 23 Over-parameterized functional The new regularization term penalizes for changes in the model parameters. 23

Euler-Lagrange equations The Euler-Lagange equation for the coefficient Aq 24 Euler-Lagrange equations The Euler-Lagange equation for the coefficient Aq 24

Over-Parameterization models Constant motion model The number of coefficients is n=2 • This case Over-Parameterization models Constant motion model The number of coefficients is n=2 • This case reduces to the regular variational approach of solving directly for u and v. 25

Affine over-parameterization model • Six basis functions 26 Affine over-parameterization model • Six basis functions 26

Rigid motion over-parameterization model • The optic flow of a rigid body is the Rigid motion over-parameterization model • The optic flow of a rigid body is the translation vector divided by the depth (z) is the rotation vector 27

Rigid motion, cont’… • In a seminal paper • The optical flow calculation is Rigid motion, cont’… • In a seminal paper • The optical flow calculation is a pre-processing followed by motion and structure estimation. • In our formulation, the rigid motion model is used directly in the optical flow estimation process. 28

Pure translation overparameterization model • Rigid motion with pure translation Use only the first Pure translation overparameterization model • Rigid motion with pure translation Use only the first three coefficients and basis functions of the general rigid motion model. 29

Numerical scheme • Multi-resolution necessary to deal with large displacements. • At each resolution, Numerical scheme • Multi-resolution necessary to deal with large displacements. • At each resolution, three loops of iterations are applied. We adopt parts of the numerical scheme from T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High Accuracy Optical Flow Estimation Based on a Theory for Warping, ” ECCV 2004. to our over-parameterization model 30

Outer loop k Euler-Lagrange equations, q=1. . . n Insert first order Taylor approximation Outer loop k Euler-Lagrange equations, q=1. . . n Insert first order Taylor approximation to the brightness constancy equation 31

Inner loop – fixed point iteration l Solves the nonlinearity of the convex function Inner loop – fixed point iteration l Solves the nonlinearity of the convex function Ψ At each pixel we have n linear equations with n unknowns: the increments of the coefficients - d. Ai 32

Experimental results The parameters were set experimentally to the following values 33 Experimental results The parameters were set experimentally to the following values 33

Synthetic piecewise affine flow example 34 Synthetic piecewise affine flow example 34

Synthetic piecewise affine flow – ground truth 35 Synthetic piecewise affine flow – ground truth 35

Results Our method is better in the AAE by 68% 36 Results Our method is better in the AAE by 68% 36

Piecewise affine test case 37 The estimated affine parameters are approximately piecewise constant Piecewise affine test case 37 The estimated affine parameters are approximately piecewise constant

Ground truth 38 Our method - affine model Ground truth 38 Our method - affine model

Yosemite without clouds sequence 39 Yosemite without clouds sequence 39

40 40

41 41

42 42

43 43

44 44

45 45

The End 46 The End 46

+39% +35% +16% +15% 47 +39% +35% +16% +15% 47

Yosemite without clouds – ground truth 48 Yosemite without clouds – ground truth 48

Images of the angular error 49 Images of the angular error 49

Histogram of the angular error Our method – pure translation model Brox et. al. Histogram of the angular error Our method – pure translation model Brox et. al. 50

Yosemite - Solution of the affine parameters 51 Yosemite - Solution of the affine parameters 51

Noise sensitivity results 52 Noise sensitivity results 52

“Variational Joint optic-flow Restoration” Computation and Video T. Nir, A. M. Bruckstein, R. Kimmel “Variational Joint optic-flow Restoration” Computation and Video T. Nir, A. M. Bruckstein, R. Kimmel Errors in the data term appear for two reasons: • Errors in the computed flow. • Errors in the image data – noise, blur, interlacing, lossy compression, … The proposed functional 53

Variational Joint optic-flow Computation and Video Restoration. Cont’. • Minimization is performed with respect Variational Joint optic-flow Computation and Video Restoration. Cont’. • Minimization is performed with respect to the optical flow u, v and the image sequence I. • The fidelity term requires that the minimization would not deviate too far from the measured sequence, thus avoiding trivial solutions. • If the expected noise is large, a lower choice of λ is appropriate, allowing larger deviations from the measured sequence. • For , the sequence is constrained to be equal to the measurement, resulting in a regular optic flow scheme. 54

Solution strategy Iterations between optic flow and denoising. 1. Initialization: zero optic flow and Solution strategy Iterations between optic flow and denoising. 1. Initialization: zero optic flow and initial sequence. 2. Solve for the optic flow. 3. Perform denoising. 4. Iterate steps 2, 3 until convergence. 55 Denoising Optic flow calculation

The Denoising step • For the denoising step we use the discrete approximation with The Denoising step • For the denoising step we use the discrete approximation with bilinear interpolation: • Minimize with respect to I 1, I 2, I 3, I 4 and I is performed by gradient descent (A, B, C, D are constant – frozen flow). • The denoising step performs smoothing along the optical flow trajectories. • Remark: Smoothing by total variation is not good for optic flow calculation. 56

Office sequence – Frame 7 57 Office sequence – Frame 7 57

Office sequence – Frame 8 58 Office sequence – Frame 8 58

Office sequence – Frame 9 59 Office sequence – Frame 9 59

Office sequence – Frame 10 60 Office sequence – Frame 10 60

Office sequence – Optic flow at frame 9 61 Office sequence – Optic flow at frame 9 61

Experimental results - Office sequence 62 Experimental results - Office sequence 62

Office sequence results - Cont’. 63 Office sequence results - Cont’. 63

A. Borzi, K. Ito, K. Kunisch: “Optimal control formulation for determining optical flow”, SIAM A. Borzi, K. Ito, K. Kunisch: “Optimal control formulation for determining optical flow”, SIAM J. Sci. Comp. 24(3), 818 -847, (2002) • Minimize with respect to u, v, I • Subject to the constraints 64

Comparison with Borzi Our method Borzi Symmetric - all the sequence Constrain the first Comparison with Borzi Our method Borzi Symmetric - all the sequence Constrain the first image to is denoised. equal the measurement. Non-linear brightness constancy penalty. Linearized brightness constancy equation as a constraint Comparison on sequences run by the best results available from the literature. Comparative results reported on simple synthetic examples. 65 First to suggest the idea of changing the images together with the flow.

What is the actual gap between L 1 and L 2? HS No clouds What is the actual gap between L 1 and L 2? HS No clouds AAE STD ~26. 14 ~19. 16 Multiscale + re -linearization σ=0. 8 +Smoothness 3 D L 1 – Brox 66 Cloudy STD 30. 28 11. 26 HS modified. σ =1. 5 Cloudy AAE 32. 43 16. 41 2. 38 6. 25 9. 14 1. 86 1. 96 5. 90 9. 01 0. 98 1. 17 1. 94 6. 02

Summary • Over-parameterized representation of the optic flow introduces better regularization. • The per Summary • Over-parameterized representation of the optic flow introduces better regularization. • The per pixel model allows the functional minimization to decide on the locations of discontinuities in the higher dimensional space. • Significant improvement for both the 2 D and 3 D cases. • Coupling with our joint optic flow and denoising scheme gives excellent results under heavy noise. • Future: The improved accuracy of the method has the potential to improve motion segmentation, video compression, super-resolution… 67