Univariate optimization x f x Key Ideas Critical points Direct methods Exhaustive search Golden section search Root finding algorithms Bisection More next time Local vs global optimization ID: 174222
Download Presentation The PPT/PDF document "CS B553: Algorithms for Optimization and..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
CS B553: Algorithms for Optimization and Learning
Univariate
optimizationSlide2
x
f
(x)Slide3
Key Ideas
Critical points
Direct methodsExhaustive searchGolden section searchRoot finding algorithmsBisection[More next time]Local vs. global optimizationAnalyzing errors, convergence ratesSlide4
x
f
(x)
Local maxima
Local minima
Inflection point
Figure 1Slide5
x
f
(x)
a
b
Figure 2aSlide6
x
f
(x)
a
b
Find critical points, apply 2
nd
derivative test
Figure 2bSlide7
x
f
(x)
a
b
Figure 2bSlide8
x
f
(x)
a
b
Global minimum must be one of these points
Figure 2cSlide9
x
f
(x)ab
Exhaustive grid search
Figure 3Slide10
x
f
(x)ab
Exhaustive grid searchSlide11
x
f
(x)Two types of errorsx*
x
t
f(
x
t
)
f(x
*
)
Geometric error
Analytical error
Figure 4Slide12
x
f
(x)ab
Does exhaustive grid search achieve
e
/2 geometric error?
e
x
*Slide13
x
f
(x)ab
Does exhaustive grid search
achieve
e
/2 geometric error?
Not necessarily for multi-modal
objective functions
E
rror
x
*Slide14
Lipschitz
continuity
Slope +K
Slope -K
|f(x)-f(y)|
K|x-y
|
Figure 5Slide15
x
f
(x)ab
Exhaustive grid search achieves
K
e
/2 analytical error in worst case
e
Figure 6Slide16
x
f
(x)ab
Golden section search
m
Bracket [
a,b
]
Intermediate point m with f(m) < f(a),f(b)
Figure 7aSlide17
x
f
(x)ab
Golden section search
m
Candidate bracket 1 [
a,m
]
c
Candidate bracket 2 [
c,b
]
Figure 7bSlide18
x
f
(x)ab
Golden section search
m
Figure 7bSlide19
x
f
(x)ab
Golden section search
m
c
Figure 7bSlide20
x
f
(x)ab
Golden section search
m
Figure 7bSlide21
x
f
(x)ab
Optimal choice: based on golden ratio
m
Choose c so that (c-a)/(m-c) =
, where
is the golden ratio
=> Bracket reduced by a factor of -1 at each step
cSlide22
Notes
Exhaustive search is a
global optimization: error bound is for finding the true optimumGSS is a local optimization: error bound holds only for finding a local minimumConvergence rate is linear: with xn = sequence of bracket midpoints Slide23
x
f
(x)
Root finding: find x-value where f’(x) crosses 0
f
’(x)
Figure 8Slide24
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
Figure 9aSlide25
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
m
Figure
9Slide26
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
Figure
9Slide27
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
m
Figure
9Slide28
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
Figure
9Slide29
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
m
Figure
9Slide30
Bisection
g(x)
a
b
Bracket [
a,b
]
Invariant: sign(f(a)) != sign(f(b))
Linear convergence: Bracket size is reduced by factor of 0.5 at each iteration
Figure
9Slide31
Next time
Root finding methods with
superlinear convergencePractical issues