S T A N F O R D

S T A N F O R D Relaxations and Moves for MAP Estimation in MRFs M. Pawan Kumar Vladimir Kolmogorov Philip Torr Daphne Koller

Our Problem Label l 2 2 0 1 Label l 1 5 v 1 2 1 0 0 4 2 v 2 6 4 3 1 1 3 3 1 0 v 3 Random Variables V = {v 1, . . . , v 4} Label Set L = {l 1, l 2} Labeling f: V L (shown in red) 7 v 4

Our Problem Label l 2 2 0 1 Label l 1 5 v 1 2 1 0 0 4 2 v 2 6 4 3 1 1 3 3 1 0 v 3 7 v 4 Random Variables V = {v 1, . . . , v 4} Label Set L = {l 1, l 2} Labeling f: V L (shown in red) Energy of Labeling E(f) = 13 (shown in green)

Our Problem Label l 2 2 0 1 Label l 1 5 v 1 2 1 0 0 4 2 v 2 6 4 3 1 1 3 3 1 0 v 3 7 v 4 Find f* = argminf E(f) Arbitrary topology, discrete label set, potentials (NP-hard) Pairwise energy function: unary and pairwise potentials (still NP-hard)

Outline • Convex Relaxations – – – Integer Programming Formulation LP Relaxation SDP Relaxation SOCP Relaxation Comparing Relaxations • Move Making Algorithms • Some Interesting Open Problems

Integer Programming Formulation Unary Potentials Label l 2 2 0 1 Label l 1 3 0 v 1 Labeling f shown in red Unary Potential u = [ 5 5 2 ; 2 Cost of v 1 = 1 1 = 2 Cost of v 4 2 v 2 4 ]

Integer Programming Formulation Unary Potentials Label l 2 2 0 1 Label l 1 5 4 3 0 v 1 Labeling f shown in red 2 v 2 Unary Potential u = [ 5 2 ; 2 4 ] Label vector x = [ -1 1 ; 1 -1 ]T v to find 1 Recall that the aim 1 is v 1 = 2 the optimal x

Integer Programming Formulation Unary Potentials Label l 2 2 0 1 Label l 1 5 4 3 0 2 v 1 Labeling f shown in red v 2 Unary Potential u = [ 5 2 ; 2 4 ] Label vector x = [ -1 1 ; 1 -1 ]T Sum of Unary Potentials = 1 ∑i ui (1 + xi) 2

Integer Programming Formulation Pairwise Potentials Label l 2 2 0 1 Label l 1 Labeling f shown in red 5 v 1 4 3 0 2 v 2 Pairwise Potential P 0 0 0 3 0 0 0 1 1 0 0 0 3 0 0 0 Cost of v 1 = 1 and v 1 = 1 Cost of v 1 = 1 and v 2 = 2

Integer Programming Formulation Pairwise Potentials Label l 2 2 0 1 Label l 1 Labeling f shown in red Pairwise Potential P 0 0 0 3 0 0 0 1 1 0 0 0 3 0 0 0 5 v 1 4 3 0 2 v 2 Sum of Pairwise Potentials 1 ∑ P (1 + x )(1+x ) ij ij i j 4

Integer Programming Formulation Pairwise Potentials Label l 2 2 0 1 Label l 1 Labeling f shown in red Pairwise Potential P 0 0 0 3 0 0 0 1 1 0 0 0 3 0 0 0 5 v 1 4 3 0 2 v 2 Sum of Pairwise Potentials 1 ∑ P (1 + x +x + x x ) ij ij i j 4 = 1 ∑ij Pij (1 + xi + xj + Xij) 4 X = x x. T Xij = xi xj

Integer Programming Formulation Constraints • Integer Constraints xi {-1, 1} X = x x. T • Uniqueness Constraint ∑ xi = 2 - |L| i va

Integer Programming Formulation x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va Convex xi {-1, 1} X = x x. T Non-Convex

Outline • Convex Relaxations – – – Integer Programming Formulation LP Relaxation SDP Relaxation SOCP Relaxation Comparing Relaxations • Move Making Algorithms • Some Interesting Open Problems

LP Relaxation Schlesinger, 1976 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi {-1, 1} X = x x. T Relax Non-Convex Constraint

LP Relaxation Schlesinger, 1976 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1] X = x x. T Relax Non-Convex Constraint

LP Relaxation Schlesinger, 1976 X = x x. T Xij [-1, 1] 1 + xi + xj + Xij ≥ 0 ∑ Xij = (2 - |L|) xi j vb

LP Relaxation Schlesinger, 1976 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1] X = x x. T Relax Non-Convex Constraint

LP Relaxation Schlesinger, 1976 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1], Xij [-1, 1] 1 + xi + xj + Xij ≥ 0 ∑ Xij = (2 - |L|) xi j vb

Outline • Convex Relaxations – – – Integer Programming Formulation LP Relaxation SDP Relaxation SOCP Relaxation Comparing Relaxations • Move Making Algorithms • Some Interesting Open Problems

SDP Relaxation Lasserre, 2000 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi {-1, 1} X = x x. T Relax Non-Convex Constraint

SDP Relaxation Lasserre, 2000 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1] X = x x. T Relax Non-Convex Constraint

SDP Relaxation 1 x 1 x 2 . . . xn x. T x = 1 X Xii = 1 Convex Non-Convex Positive Semidefinite Rank = 1

SDP Relaxation 1 x 1 x 2 . . . xn x. T x = 1 X Xii = 1 Convex Positive Semidefinite

SDP Relaxation Lasserre, 2000 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1] X = x x. T Relax Non-Convex Constraint

SDP Relaxation Lasserre, 2000 Retain Convex Part x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1] Xii = 1 Accurate X - xx. T Positive Semidefinite 0 Inefficient

Outline • Convex Relaxations – – – Integer Programming Formulation LP Relaxation SDP Relaxation SOCP Relaxation Comparing Relaxations • Move Making Algorithms • Some Interesting Open Problems

SOCP Relaxation Derive SOCP relaxation from the SDP relaxation x* = argmin 1 ∑ u (1 + x ) + 1 ∑ P (1 + x + X ) ij i j ij i i 4 2 ∑ xi = 2 - |L| i va xi [-1, 1] Xii = 1 X - xx. T 0 Further Relaxation

SOCP Relaxation Kim and Kojima, 2000 Choose a sub-graph G Variables x. G and XG Choose a matrix C 1 = UUT 0 C 1 (XG - x. GT) ≥ 0 Choose a matrix C 2 = UUT REPEAT 0

Outline • Convex Relaxations – – – Integer Programming Formulation LP Relaxation SDP Relaxation SOCP Relaxation Comparing Relaxations • Move Making Algorithms • Some Interesting Open Problems

Dominating Relaxation ≥ A B For all MAP Estimation problem (u, P) A dominates B Dominating relaxations are better

SOCP Relaxation Kim and Kojima, 2000 Choose a sub-graph G Variables x. G and XG Choose a matrix C 1 = UUT 0 C 1 (XG - x. GT) ≥ 0 If G is a tree, LP dominates SOCP

Examples Muramatsu and Suzuki, 2003 (MAXCUT) Ravikumar and Lafferty, 2006 (QP Relaxation) Kumar, Torr and Zisserman, 2006 (Equivalent SOCP Relaxation)

SOCP Relaxation Kim and Kojima, 2000 Choose a sub-graph G Variables x. G and XG Choose a matrix C 1 = UUT 0 C 1 (XG - x. GT) ≥ 0 If G is a cycle with non-negative P

SOCP Relaxation Kim and Kojima, 2000 Choose a sub-graph G Variables x. G and XG Choose a matrix C 1 = UUT 0 C 1 (XG - x. GT) ≥ 0 If G is an even cycle with non-positive P

SOCP Relaxation Kim and Kojima, 2000 Choose a sub-graph G Variables x. G and XG Choose a matrix C 1 = UUT 0 C 1 (XG - x. GT) ≥ 0 If G is an odd cycle with 1 non-positive P

SOCP Relaxation Kumar, Kolmogorov and Torr, 2007 What about other cycles? Dominated by linear cycle inequalities Cliques? Dominated by clique inequalities

Outline • Convex Relaxations • Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves • Some Interesting Open Problems

MRFs in Vision Pab(i, k) lk li ua(i) va Pab(i, k) = wab min{ d(i-k), M } wab is non-negative vb ub(k) d(. ) is a semi-metric distance Truncated Linear Truncated Quadratic

Move Making Current Solution Search Neighbourhood Energy Optimal Move Solution Space Slide courtesy of Pushmeet Kohli

Outline • Convex Relaxations • Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves • Some Interesting Open Problems

Expansion Move Variables take label or retain current label Slide courtesy of Pushmeet Kohli Boykov, Veksler, Zabih 2001

Expansion Move Variables take label or retain current label Status: Expand Ground Initialize with Tree Sky House Slide courtesy of Pushmeet Kohli Tree Ground House Sky Boykov, Veksler, Zabih] [Boykov, Veksler, 2001

Outline • Convex Relaxations • Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves • Some Interesting Open Problems

Multiplicative Bounds Expansion Bounds as bad as ICM Bounds LP Move. Making Potts 2 2 Truncated Linear 2 + √ 2 2 M Truncated Quadratic O(√M) 2 M Metric Labeling O(log h) 2 M

Outline • Convex Relaxations • Move Making Algorithms – State of the Art – Comparison with LP Relaxation – Improved Moves • Some Interesting Open Problems

Randomized Rounding yi = (1 + xi)/2 y’i = y 0 + y 1 + … + yi 0 y’i y’k Choose an interval of length L’ y’h = 1

Randomized Rounding yi = (1 + xi)/2 y’i = y 0 + y 1 + … + yi r 0 y’i y’k y’h = 1 Generate a random number r (0, 1]

Randomized Rounding yi = (1 + xi)/2 y’i = y 0 + y 1 + … + yi r 0 y’i y’k y’h = 1 Assign label next to r (if within the interval)

Move Making • Initialize the labeling • Choose interval I of L’ labels • Each variable can • Retain old label • Choose a label from I • Choose best labeling va vb Non-submodular move? Iterate over intervals Submodular overestimation

Truncated Convex Models Pab(i, k) = wab min{ d(i-k), M } d(. ) is convex Truncated Linear d(x+1) - 2 d(x) + d(x-1) ≥ 0 Truncated Quadratic

Move Making • Choose interval I of L’ labels • Each variable can • Retain old label • Choose a label from I • Choose best labeling Large L’ => Non-submodular va vb

Move Making va vb Submodular problem

Move Making va vb Non-submodular Problem

Move Making va vb Submodular problem Ishikawa, 2003; Veksler, 2007

Move Making s am+1 am+2 bm+2 an va bm+1 bn vb t

Move Making LP Bounds Kumar and Torr, NIPS 08 Kumar and Koller, UAI 09 Type of Problem Bound Potts 2 Truncated Linear 2 + √ 2 Truncated Quadratic O(√M) Metric Labeling O(log h)

Outline • Convex Relaxations • Move Making Algorithms • Some Interesting Open Problems

Problem 1 Relationship between rounding and move-making? What happens when n < h ? ? (Should we even use move-making here? ? ) What about semi-metric MRFs? ?

Problem 2 Graph-cuts based image segmentation Vicente, Kolmogorov, Rother, 2008

Problem 2 Image segmentation with connectivity prior Vicente, Kolmogorov, Rother, 2008

Problem 2++ Kumar and Koller, 20? ?

Questions? ? http: //ai. stanford. edu/~pawan