Motion Estimation I - PowerPoint Presentation

Presentation on theme: "Motion Estimation I"— Presentation transcript

Slide1

Motion Estimation I

What affects the induced image motion?

Camera motion

Object motion

Scene structureSlide2

Example Flow Fields

This lesson – estimation of general flow-fields

Next lesson – constrained by global parametric transformationsSlide3

The Aperture Problem

So how much information is there locally…?Slide4

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Not enough info in local regionsSlide5

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Not enough info in local regionsSlide6

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.Slide7

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Information is propagated from regions with high certainty (e.g., corners) to regions with low certainty.Slide8

Such info propagation can cause optical illusions…

Illusory cornersSlide9

1.

Gradient-based (differential) methods

(Horn &Schunk, Lucase

&

Kanade

)

2.

Region-based methods

(Correlation, SSD, Normalized correlation)

Direct (intensity-based) Methods Feature-based MethodsSlide10

Image

J

(taken at time

t

)

Brightness Constancy Assumption

Image

I

(taken at time

t+1

)Slide11

Brightness Constancy Equation:

The Brightness Constancy Constraint

Linearizing (assuming small

(u,v)

):Slide12

* One equation, 2 unknowns

* A line constraint in (u,v) space.* Can recover Normal Flow.Observations:

Need additional constraints…Slide13

Horn and Schunk (1981)

Add global smoothness term

Smoothness error

Error in brightness constancy equation

Minimize:

Solve by using calculus of variationsSlide14

Horn and Schunk (1981)

Problems…* Smoothness assumption wrong at motion/depth discontinuities  over-smoothing of the flow field.* How is Lambda determined…?Slide15

Lucas-Kanade (1984)

Assume a single displacement (u,v) for all pixels within a small window (e.g., 3x3, 5x5)

Minimize

E(u,v):

Geometrically -- Intersection of multiple line constraints

Algebraically -- Slide16

Lucas-Kanade (1984)

Differentiating w.r.t

u

and

v

and equating to

0:

Solve for (u,v)

[ Repeat this process for each and every pixel in the image ]

Minimize

E(u,v):Slide17

Problems…

* Still smoothes motion discontinuities (but unlike Horn & Schunk, does not propagate error across the entire image)* Singularities (partially solved by coarse-to-fine)

Lucas-Kanade (1984)Slide18

Singularites

Where in the image will this matrix be invertible and where not…?



HomeworkSlide19

Linearization approximation  iterate & warp

x

x

0

Initial guess:

Estimate:

estimate updateSlide20

x

x

0

estimate update

Initial guess:

Estimate:

Linearization approximation

 iterate & warpSlide21

x

x

0

Initial guess:

Estimate:

Initial guess:

Estimate:

estimate update

Linearization approximation

 iterate & warpSlide22

x

x

0

Linearization approximation

 iterate & warpSlide23

Revisiting the small motion assumption

Is this motion small enough?Probably not—it’s much larger than one pixel (2nd order terms dominate)

How might we solve this problem?Slide24

==> small

u

and

v

...

u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

image I

image J

iterate

refine

+

Pyramid of image J

Pyramid of image I

image I

image J

Coarse-to-Fine Estimation

Advantages:

(i) Larger displacements. (ii) Speedup.

(iii) Information from multiple window sizes.Slide25

Optical Flow ResultsSlide26

Optical Flow ResultsSlide27

1.

Gradient based methods (Horn &Schunk, Lucase & Kanade, …)

2.

Region based methods

(SSD, Normalized correlation, etc.)

Copyright, 1996 © Dale Carnegie & Associates, Inc.

But… (despite coarse-to-fine estimation)

rely on B.C.

cannot handle very large motions (no more than 10%-15% of image width/height) small object moving fast…?Slide28

Region-Based Methods

* Define a small area around a pixel as the region.* Match the region against each pixel within a search area in next image.* Use a match measure (e.g., SSD=sum

of-squares difference, NC=normalized correlation, etc.)* Choose the maximum (or minimum) as the match.

Advantages:

Can avoid B.C. assumption

Can handle large motions (even of small objects)

Disadvantages:

Less accurate (smaller sub-pixel accuracy)

Computationally more expensiveSlide29

SSD Surface – Textured areaSlide30

SSD Surface -- EdgeSlide31

SSD – homogeneous area

[Anandan’89 - Use coarse-to-fine SSD of local windows to find matches.

- Propagate information using

directional

confidence measures

extracted from each local SSD surface]Slide32

B.C. + Additional constraints:

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Increase aperture:

[e.g., Lucas & Kanade]

Local singularities at degenerate image regions.

Increase analysis window to large image regions

=> Global model constraints:

Numerically stable, but requires prior model selection:

Planar (2D) world model

[e.g., Bergen-et-al:92, Irani-et-al:92+94, Black-et-al]

3D world model[e.g., Hanna-et-al:91+93, Stein & Shashua:97, Irani-et-al:1999]

Spatial smoothness: [e.g., Horn & Schunk:81, Anandan:89] Violated at depth/motion discontinuities