What affects the induced image motion Camera motion Object motion Scene structure Example Flow Fields This lesson estimation of general flowfields Next lesson constrained by global parametric transformations ID: 247102
Download Presentation The PPT/PDF document "Motion Estimation I" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Motion Estimation I
What affects the induced image motion?
Camera motion
Object motion
Scene structureSlide2
Example Flow Fields
This lesson – estimation of general flow-fields
Next lesson – constrained by global parametric transformationsSlide3
The Aperture Problem
So how much information is there locally…?Slide4
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Not enough info in local regionsSlide5
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Not enough info in local regionsSlide6
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.Slide7
The Aperture Problem
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Information is propagated from regions with high certainty (e.g., corners) to regions with low certainty.Slide8
Such info propagation can cause optical illusions…
Illusory cornersSlide9
1.
Gradient-based (differential) methods
(Horn &Schunk, Lucase
&
Kanade
)
2.
Region-based methods
(Correlation, SSD, Normalized correlation)
Direct (intensity-based) Methods Feature-based MethodsSlide10
Image
J
(taken at time
t
)
Brightness Constancy Assumption
Image
I
(taken at time
t+1
)Slide11
Brightness Constancy Equation:
The Brightness Constancy Constraint
Linearizing (assuming small
(u,v)
):Slide12
* One equation, 2 unknowns
* A line constraint in (u,v) space.* Can recover Normal Flow.Observations:
Need additional constraints…Slide13
Horn and Schunk (1981)
Add global smoothness term
Smoothness error
Error in brightness constancy equation
Minimize:
Solve by using calculus of variationsSlide14
Horn and Schunk (1981)
Problems…* Smoothness assumption wrong at motion/depth discontinuities over-smoothing of the flow field.* How is Lambda determined…?Slide15
Lucas-Kanade (1984)
Assume a single displacement (u,v) for all pixels within a small window (e.g., 3x3, 5x5)
Minimize
E(u,v):
Geometrically -- Intersection of multiple line constraints
Algebraically -- Slide16
Lucas-Kanade (1984)
Differentiating w.r.t
u
and
v
and equating to
0:
Solve for (u,v)
[ Repeat this process for each and every pixel in the image ]
Minimize
E(u,v):Slide17
Problems…
* Still smoothes motion discontinuities (but unlike Horn & Schunk, does not propagate error across the entire image)* Singularities (partially solved by coarse-to-fine)
Lucas-Kanade (1984)Slide18
Singularites
Where in the image will this matrix be invertible and where not…?
HomeworkSlide19
Linearization approximation iterate & warp
x
x
0
Initial guess:
Estimate:
estimate updateSlide20
x
x
0
estimate update
Initial guess:
Estimate:
Linearization approximation
iterate & warpSlide21
x
x
0
Initial guess:
Estimate:
Initial guess:
Estimate:
estimate update
Linearization approximation
iterate & warpSlide22
x
x
0
Linearization approximation
iterate & warpSlide23
Revisiting the small motion assumption
Is this motion small enough?Probably not—it’s much larger than one pixel (2nd order terms dominate)
How might we solve this problem?Slide24
==> small
u
and
v
...
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
image I
image J
iterate
refine
+
Pyramid of image J
Pyramid of image I
image I
image J
Coarse-to-Fine Estimation
Advantages:
(i) Larger displacements. (ii) Speedup.
(iii) Information from multiple window sizes.Slide25
Optical Flow ResultsSlide26
Optical Flow ResultsSlide27
1.
Gradient based methods (Horn &Schunk, Lucase & Kanade, …)
2.
Region based methods
(SSD, Normalized correlation, etc.)
Copyright, 1996 © Dale Carnegie & Associates, Inc.
But… (despite coarse-to-fine estimation)
rely on B.C.
cannot handle very large motions (no more than 10%-15% of image width/height) small object moving fast…?Slide28
Region-Based Methods
* Define a small area around a pixel as the region.* Match the region against each pixel within a search area in next image.* Use a match measure (e.g., SSD=sum
of-squares difference, NC=normalized correlation, etc.)* Choose the maximum (or minimum) as the match.
Advantages:
Can avoid B.C. assumption
Can handle large motions (even of small objects)
Disadvantages:
Less accurate (smaller sub-pixel accuracy)
Computationally more expensiveSlide29
SSD Surface – Textured areaSlide30
SSD Surface -- EdgeSlide31
SSD – homogeneous area
[Anandan’89 - Use coarse-to-fine SSD of local windows to find matches.
- Propagate information using
directional
confidence measures
extracted from each local SSD surface]Slide32
B.C. + Additional constraints:
Copyright, 1996 © Dale Carnegie & Associates, Inc.
Increase aperture:
[e.g., Lucas & Kanade]
Local singularities at degenerate image regions.
Increase analysis window to large image regions
=> Global model constraints:
Numerically stable, but requires prior model selection:
Planar (2D) world model
[e.g., Bergen-et-al:92, Irani-et-al:92+94, Black-et-al]
3D world model[e.g., Hanna-et-al:91+93, Stein & Shashua:97, Irani-et-al:1999]
Spatial smoothness: [e.g., Horn & Schunk:81, Anandan:89] Violated at depth/motion discontinuities