M Pawan Kumar Slides available online http mpawankumarinfo Energy Function V a V b V c V d Label l 0 Label l 1 Random Variables V V a V b Labels L l ID: 783951
Download The PPT/PDF document "Optimization Convex Relaxations" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
OptimizationConvex Relaxations
M. Pawan Kumar
Slides available online http://mpawankumar.info
Slide2Energy Function
V
a
V
b
V
c
V
d
Label
l
0
Label
l
1
Random Variables V
= {V
a
, V
b
, ….}
Labels L
= {l
0
, l
1
, ….}
Labelling f: {a, b, …. }
{0,1, …}
Slide3Energy Function
V
a
V
b
V
c
V
d
Q(f)
=
∑
a
a
(f
(a
))
Unary Potential
2
5
4
2
6
3
3
7
Label
l
0
Label
l
1
Easy to minimize
Neighbourhood
Slide4Energy Function
V
a
V
b
V
c
V
d
E : (a,b)
E iff V
a
and V
b
are neighbours
E = { (a,b) , (b,c) , (c,d) }
2
5
4
2
6
3
3
7
Label
l
0
Label
l
1
Slide5Energy Function
V
a
V
b
V
c
V
d
+∑
(
a,b
)
ab
(f
(a
),f
(b
))
Pairwise Potential
0
1
1
0
0
2
1
1
4
1
0
3
2
5
4
2
6
3
3
7
Label
l
0
Label
l
1
Q(f)
=
∑
a
a
(f
(a
))
Slide6Energy Minimization
V
1
V
2
2
5
4
2
0
1
3
0
+∑
(
a,b
)
ab
(f
(a
),f
(b
))
∑
a
a(f(a))minf
Slide7Integer Program
+∑
(a,b)
ab(f(a),f
(b
))
∑
a a(f(a))
minfDoes
Va take the label li (xa(
i) = 1) or not (xa(i) = 0)?
xa(i) ∈ {0,1}
V
1
V
2
2
5
4
2
0
1
3
0
Slide8Constraint
+∑
(a,b)
ab(f(a),f
(b
))
∑
a a(f(a))
minfConstraint that
Va can take exactly one label
xa(i) ∈ {0,1}
V
1
V
2
2
5
4
2
0
1
3
0
Slide9Constraint
+∑
(a,b)
ab(f(a),f
(b
))
∑
a a(f(a))
minf∑i
xa(i) = 1
xa(i) ∈ {0,1}
V
1
V
2
2
5
4
2
0
1
3
0
Slide10Unary Potentials
+∑
(a,b)
ab(f(a),f
(b
))
∑
a a(f(a))
minf
∑a ∑i a
(i)xa(i)
∑i xa(i) = 1
xa(i
) ∈ {0,1}
V
1
V
2
2
5
4
2
0
1
3
0
Slide11Pairwise Potentials
+∑
(a,b)
ab(f(a),f
(b
))
∑
a a(f(a))
minf
∑(a,b) ∑i,k
ab(i,k)xa(i)xb
(k)∑i
xa(i) = 1
xa(i) ∈ {0,1}
V
1
V
2
2
5
4
2
0
1
3
0
Slide12Integer Program
minx
x
a(
i
) ∈ {0,1}
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xa(i)
xb(k)s.t.
∑a ∑i a
(i)xa(i)
V
1
V
2
2
5
4
2
0
1
3
0
Slide13QP RelaxationLP Relaxation for Potts ModelLP Relaxation for Pairwise Energy
A Hierarchy of Relaxations
Outline
Ravikumar
and Lafferty, 2006
Slide14Unary Potential Vector
Unary
Potential
u = [ 5
Cost of V
1
= 0
2
Cost of V
1
= 1
; 2 4 ]
V
1
V
2
2
5
4
2
0
1
3
0
For
x
, total unary cost?
u
T
x
Slide15Pairwise Potential Matrix
V
1
V
2
2
5
4
2
0
1
3
0
0
Cost of V
1
= 0 and V
1
= 0
0
0
0
0
Cost of V
1
= 0 and V
2
= 0
3
Cost of V
1
= 0 and V
2
= 1
1
0
0
0
0
0
1
0
3
0
Pairwise Potential Matrix
P
For
x
, total Pairwise cost?
½
x
T
Px
Slide16Integer Program
V
1
V
2
2
5
4
2
0
1
3
0
min
x
x
a
(
i
) ∈ {0,1}
∑
i
x
a
(
i
) = 1
s.t.
uTxConvex?
No. Diagonal of P is 0
+ ½ xTPx
Slide17Integer Program
V
1
V
2
2
5
4
2
0
1
3
0
min
x
x
a
(
i
) ∈ {0,1}
∑
i
x
a
(
i
) = 1
s.t.
uTxConsider a vector d
Define D = diag(d
) + ½
xTPx
Slide18Integer Program
V
1
V
2
2
5
4
2
0
1
3
0
min
x
x
a
(
i
) ∈ {0,1}
∑
i
x
a
(
i
) = 1
s.t.
uTxConsider a vector d
Define D = diag(d
) + ½
xT(P+D)x
Slide19Integer Program
V
1
V
2
2
5
4
2
0
1
3
0
min
x
x
a
(
i
) ∈ {0,1}
∑
i
x
a
(
i
) = 1
s.t.
(u-d)TxConsider a vector d
Define D = diag(
d)
+ ½ xT(P+D)x
Equivalent to the old problem
Why?
x
a
(
i
)*
x
a
(
i
) =
x
a
(
i
)
Slide20Integer Program
V
1
V
2
2
5
4
2
0
1
3
0
min
x
x
a
(
i
) ∈ {0,1}
∑
i
x
a
(
i
) = 1
s.t.
(u-d)TxChoose an appropriate d
+ ½ xT(P+D)x
Convex
Why?
Because
P
+
D
≽
0
d(
i
) = Sum of absolute values
of the
i-th
row of
P
Slide21QP Relaxation
V
1
V
2
2
5
4
2
0
1
3
0
min
x
x
a
(
i
) ∈ [0,1]
∑
i
x
a
(
i
) = 1
s.t.
(u-d)TxChoose an appropriate d
+ ½ xT(P+D)x
d(i) = Sum of absolute values
of the i-th row of P
Solver?
Conditional Gradient
(Frank-Wolfe)
Slide22QP RelaxationConditional GradientLP Relaxation for Potts Model
LP Relaxation for Pairwise EnergyA Hierarchy of Relaxations
Outline
Frank and Wolfe, 1956
Slide23Conditional Gradient
s.t.
x ∈ X
minx f(
x
)
Objective f(
x
) is assumed smoothGradients defined everywhere
Feasible region is convex and bounded
Slide24Conditional Gradient
s.t.
x ∈ X
minx f(
x
)
Compute gradient
g
of f(x) at current xt
Compute conditional gradient
Slide25Conditional Gradient
s.t.
x ∈ X
minx
g
T
x
c
t = argUpdate
xt+1 = ηtxt + (1-η
t)ct
xt+1 ∈ XNo need for projection
Why?
Slide26CG for QP
Initialize x
0 and t = 0
While objective can be reduced
g
= (
u
-d) + (P+D)xt
ct = argminx
xa(i) ∈ [0,1]
∑i xa(i) = 1
s.t.
gTxEasy
Why?Update xt+1
= ηtxt + (1-ηt)ct
We can compute optimal ηt Why?
t = t + 1
Slide27QP RelaxationLP Relaxation for Potts ModelLP Relaxation for Pairwise Energy
A Hierarchy of Relaxations
Outline
Kleinberg and
Tardos
, 1999
Slide28Integer Program
minx
x
a(
i
) ∈ {0,1}
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xa(i)
xb(k)s.t.
∑a ∑i a
(i)xa(i)
V
1
V
2
2
5
4
2
0
1
3
0
ab
(
i,k
) =
w
ab, if
i ≠ k
= 0, if
i
= k
Slide29Integer Program
minx
x
a(
i
) ∈ {0,1}
∑
i
xa(i) = 1
+ ∑(a,b) ∑i
wab |xa(i)-x
b(i)|s.t.
∑a ∑i
a(i)xa(i)
V
1
V
2
2
5
4
2
0
1
3
0
ab
(
i,k
) =
w
ab
, if i ≠ k
= 0, if
i
= k
½
Slide30LP Relaxation
minx
x
a(
i
) ∈ [0,1]
∑
i
xa(i) = 1
s.t.
∑a ∑i a(i)x
a(i)
V
1
V
2
2
5
4
2
0
1
3
0
ab
(
i,k
) =
w
ab, if i ≠ k
= 0, if i
= kFor 2 labels, min-cut problem
Integer optimal solutions
+
∑
(
a,b
)
∑
i
w
ab
|
x
a
(
i
)-
x
b
(
i
)|
½
Slide31QP RelaxationLP Relaxation for Potts ModelLP Relaxation for Pairwise Energy
A Hierarchy of Relaxations
Outline
Chekuri
et al., 2001
Slide32Integer Program
minx
x
a(
i
) ∈ {0,1}
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xa(i)
xb(k)s.t.
∑a ∑i a
(i)xa(i)
V
1
V
2
2
5
4
2
0
1
3
0
Slide33Integer Program
minx
x
a(
i
) ∈ {0,1}
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xab(i,k)
s.t.
∑a ∑i a(i)
xa(i)
V
1
V
2
2
5
4
2
0
1
3
0
x
ab
(
i,k
) ∈ {0,1}
∑
k
xab(i,k) =
x
a
(
i
)
Slide34LP Relaxation
minx
x
a(
i
) ∈ [0,1]
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xab(i,k)
s.t.
∑a ∑i a(i)
xa(i)
V
1
V
2
2
5
4
2
0
1
3
0
x
ab
(
i,k
) ∈ [0,1]
∑
k
xab(i,k) =
x
a
(
i
)
Marginalization constraint
UGC Hardness Guarantees
Slide35QP RelaxationLP Relaxation for Potts ModelLP Relaxation for Pairwise Energy
A Hierarchy of Relaxations
Outline
Sherali
and Adams, 1990
Slide36LP Relaxation
minx
x
a(
i
) ∈ [0,1]
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xab(i,k)
s.t.
∑a ∑i a(i)
xa(i)
xab(i,k) ∈ [0,1]
∑k xab(i,k) =x
a(i) + ∑(a,b,c) ∑i,k,m
abc(i,k,m)xab(i,k,m)
∑k,m xabc(i,k,m) = xa(i)
∑m xabc(i,k,m) =
xab(i,k)
xabc(i,k,m) ∈ [0,1]
Slide37LP Relaxation
minx
x
a(
i
) ∈ [0,1]
∑
i
xa(i) = 1
+ ∑(a,b) ∑i,k
ab(i,k)xab(i,k)
s.t.
∑a ∑i a(i)
xa(i)
∑k xab(i,k) =
xa(i)∑k,m
xabc(i,k,m) = xa(i)
xabc(i,k,m) ∈ [0,1]xab(
i,k) ∈ [0,1]∑m xabc(i,k,m) = x
ab(i,k)
Slide38LP Relaxation Hierarchy
Higher and higher orders of marginalizations
Eventually you will find a tight relaxation
But it may be exponential in size
Exponential blow-up very rare in practice
Real-world is full of structure
Slide39Questions?