Many slides adapted from S Seitz R Szeliski M Pollefeys Motion and perceptual organization Sometimes motion is the only cue Motion and perceptual organization Sometimes motion is the only cue ID: 313822
Download Presentation The PPT/PDF document "Visual motion" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Visual motion
Many slides adapted from S. Seitz, R. Szeliski, M. PollefeysSlide2
Motion and perceptual organization
Sometimes, motion is the only cueSlide3
Motion and perceptual organization
Sometimes, motion is the only cueSlide4
Motion and perceptual organization
Even “impoverished” motion data can evoke a strong percept
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis",
Perception and Psychophysics 14, 201-211, 1973.Slide5
Motion and perceptual organization
Even “impoverished” motion data can evoke a strong percept
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis",
Perception and Psychophysics 14, 201-211, 1973.Slide6
Motion and perceptual organization
Even “impoverished” motion data can evoke a strong percept
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis",
Perception and Psychophysics 14, 201-211, 1973.Slide7
Uses of motion
Estimating 3D structure
Segmenting objects based on motion cuesLearning and tracking dynamical models
Recognizing events and activitiesSlide8
Motion field
The motion field is the projection of the 3D scene motion into the imageSlide9
Motion field and parallax
X
(t) is a moving 3D pointVelocity of scene point:
V = dX/dtx
(t) = (x(t),y(
t)) is the projection of X in the imageApparent velocity v
in the image: given by components
v
x
= d
x
/d
t
and
v
y
= d
y
/d
t
These components are known as the
motion field
of the image
x
(
t
)
x
(
t+dt
)
X
(
t
)
X
(
t+dt
)
V
vSlide10
Motion field and parallax
x
(t)
x
(
t+dt
)
X
(
t
)
X
(
t+dt
)
V
v
To find image velocity
v
, differentiate
x
=(
x
,
y
) with respect to
t
(using quotient rule):
Image motion is a function of both the 3D motion (
V
) and the
depth of the 3D point (
Z
)Slide11
Motion field and parallax
Pure translation:
V is constant everywhereSlide12
Motion field and parallax
Pure translation:
V is constant everywhere
The length of the motion vectors is inversely proportional to the depth ZVz is nonzero:
Every motion vector points toward (or away from) the vanishing point of the translation directionSlide13
Motion field and parallax
Pure translation:
V is constant everywhere
The length of the motion vectors is inversely proportional to the depth ZVz is nonzero:
Every motion vector points toward (or away from) the vanishing point of the translation directionVz is zero:
Motion is parallel to the image plane, all the motion vectors are parallelSlide14
Optical flow
Definition: optical flow is the
apparent motion of brightness patterns in the imageIdeally, optical flow would be the same as the motion fieldHave to be careful: apparent motion can be caused by lighting changes without any actual motion
Think of a uniform rotating sphere under fixed lighting vs. a stationary sphere under moving illuminationSlide15
Estimating optical flow
Given two subsequent frames, estimate the apparent motion field
u(x,y
) and v(x,y) between them
Key assumptions
Brightness constancy:
projection of the same point looks the same in every frame
Small motion:
points do not move very far
Spatial coherence:
points move like their neighbors
I
(
x
,
y
,
t
–1)
I
(
x
,
y
,
t
)Slide16
Brightness Constancy Equation:
Linearizing the right side using Taylor expansion:
The brightness constancy constraint
I
(
x
,
y
,
t
–1)
I
(
x
,
y
,
t
)
Hence,Slide17
The brightness constancy constraint
How many equations and unknowns per pixel?
One equation, two unknowns
Intuitively, what does this constraint mean?
The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknownSlide18
The brightness constancy constraint
How many equations and unknowns per pixel?
One equation, two unknowns
Intuitively, what does this constraint mean?
The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown
edge
(
u
,
v
)
(
u
’,
v
’)
gradient
(
u
+
u
’,
v
+
v
’)
If (
u
,
v
) satisfies the equation,
so does (
u+u’
,
v+v’
) if
Slide19
The aperture problem
Perceived motionSlide20
The aperture problem
Actual motionSlide21
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusionSlide22
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusionSlide23
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusionSlide24
Solving the aperture problem
How to get more equations for a pixel?
Spatial coherence constraint: pretend the pixel’s neighbors have the same (u,v)If we use a 5x5 window, that gives us 25 equations per pixel
B. Lucas and T. Kanade.
An iterative image registration technique with an application to
stereo vision.
In
Proceedings of the International Joint Conference on Artificial Intelligence
, pp. 674–679, 1981.Slide25
Solving the aperture problem
Least squares problem:
B. Lucas and T. Kanade.
An iterative image registration technique with an application to
stereo vision.
In
Proceedings of the International Joint Conference on Artificial Intelligence
, pp. 674–679, 1981.
When is this system solvable?
What if the window contains just a single straight edge?Slide26
Conditions for solvability
“Bad” case: single straight edgeSlide27
Conditions for solvability
“Good” caseSlide28
Lucas-Kanade flow
Linear least squares problem
B. Lucas and T. Kanade.
An iterative image registration technique with an application to
stereo vision.
In
Proceedings of the International Joint Conference on Artificial Intelligence
, pp. 674–679, 1981.
The summations are over all pixels in the window
Solution given bySlide29
Lucas-Kanade flow
Recall the Harris corner detector:
M = AT
A is the second moment matrixWe can figure out whether the system is solvable by looking at the
eigenvalues of the second moment matrixThe eigenvectors and
eigenvalues of M relate to edge direction and magnitude The eigenvector associated with the larger
eigenvalue
points in the direction of fastest intensity change, and the other eigenvector is orthogonal to itSlide30
Interpreting the eigenvalues
1
2
“Corner”
1
and
2
are large,
1
~
2
1
and
2
are small
“Edge”
1
>>
2
“Edge”
2
>>
1
“Flat” region
Classification of image points using eigenvalues of the second moment matrix:Slide31
Uniform region
gradients have small magnitude
small
l
1, small l2
system is ill-conditionedSlide32
Edge
gradients have one dominant direction
large
l
1
, small
l
2
system is ill-conditionedSlide33
High-texture or corner region
gradients have different directions, large magnitudes
large
l
1
, large
l
2
system is well-conditionedSlide34
Errors in Lucas-Kanade
The motion is large (larger than a pixel)
Iterative refinementCoarse-to-fine estimationExhaustive neighborhood search (feature matching)A point does not move like its neighbors
Motion segmentationBrightness constancy does not holdExhaustive neighborhood search with normalized correlationSlide35
Feature tracking
So far, we have only considered optical flow estimation in a pair of images
If we have more than two images, we can compute the optical flow from each frame to the nextGiven a point in the first image, we can in principle reconstruct its path by simply “following the arrows”Slide36
Ambiguity of optical flow
Need to find good features to trackLarge motions, changes in appearance, occlusions, disocclusions
Need mechanism for deleting, adding new featuresDrift – errors may accumulate over timeNeed to know when to terminate a track
Tracking challengesSlide37
Tracking over many frames
Select features in first frame
For each frame:Update positions of tracked features Discrete search or Lucas-Kanade (or a combination of the two)
Terminate inconsistent tracksCompute similarity with corresponding feature in the previous frame or in the first frame where it’s visibleFind more features to trackSlide38
Shi-Tomasi feature tracker
Find good features using eigenvalues of second-moment matrix
Key idea: “good” features to track are the ones whose motion can be estimated reliably
From frame to frame, track with Lucas-KanadeThis amounts to assuming a translation model for frame-to-frame feature movement
Check consistency of tracks by affine registration to the first observed instance of the feature
Affine model is more accurate for larger displacementsComparing to the first frame helps to minimize drift
J. Shi and C. Tomasi.
Good Features to Track
. CVPR 1994. Slide39
Tracking example
J. Shi and C. Tomasi.
Good Features to Track
. CVPR 1994.