Photo by Carl Warner Photo by Carl Warner - PowerPoint Presentation

Slide1

Photo by Carl Warner

Slide2

Photo by Carl Warner

Slide3

Photo by Carl Warner

Slide4

Feature Matching and Robust FittingComputer VisionJames Hays

Acknowledgment: Many

slides from Derek

Hoiem

and

Grauman&Leibe

2008 AAAI Tutorial

Read

Szeliski

4.1

Slide5

Project 2

Slide6

This section: correspondence and alignmentCorrespondence: matching points, patches, edges, or regions across images

≈

Slide7

Overview of Keypoint MatchingK. Grauman, B. Leibe

B

1

B

2

B

3

A

1

A

2

A

3

1. Find a set of

distinctive key-

points

3. Extract and

normalize the

region content

2. Define a region

around each

keypoint

4. Compute a local

descriptor from the

normalized region

5. Match local

descriptors

Slide8

Harris Corners – Why so complicated?Can’t we just check for regions with lots of gradients in the x and y directions?No! A diagonal line would satisfy that criteria

Current Window

Slide9

Harris Detector [Harris88]

Second moment matrix

9

1. Image derivatives

2. Square of derivatives

3. Gaussian

filter

g(

s

I

)

I

x

I

y

I

x

2

I

y

2

I

x

I

y

g(I

x

2

)

g(I

y

2

)

g(I

x

I

y

)

4. Cornerness function

–

both eigenvalues are strong

har

5. Non-maxima suppression

(optionally, blur first)

Slide10

Harris Corners – Why so complicated?What does the structure matrix look here?

Current Window

Slide11

Harris Corners – Why so complicated?What does the structure matrix look here?

Current Window

Slide12

Harris Corners – Why so complicated?

What does the structure matrix look here?

Current Window

Slide13

Affine intensity change

Only derivatives are used => invariance to intensity shift

I



I

+

b

Intensity scaling:

I



a

I

R

x

(image coordinate)

threshold

R

x

(image coordinate)

Partially invariant

to affine intensity change

I



a

I

+ b

Slide14

Image translation

Derivatives and window function are shift-invariant

Corner location is covariant w.r.t. translation

Slide15

Image rotation

Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner location is covariant w.r.t. rotation

Slide16

Scaling

All points will be classified as

edges

Corner

Corner location is not covariant to scaling!

Slide17

Review: Interest pointsKeypoint detection: repeatable and distinctiveCorners, blobs, stable regionsHarris, DoG

, MSER

Slide18

Comparison of Keypoint DetectorsTuytelaars Mikolajczyk 2008

Slide19

Review: Choosing an interest point detectorWhat do you want it for?

Precise localization in x-y: Harris

Good localization in scale: Difference of Gaussian

Flexible region shape: MSER

Best choice often application dependent

Harris-/Hessian-Laplace/

DoG

work well for many natural categories

MSER works well for buildings and printed things

Why choose?

Get more points with more detectors

There have been extensive evaluations/comparisons

[Mikolajczyk et al., IJCV’05, PAMI’05]

All detectors/descriptors shown here work well

Slide20

Review: Local DescriptorsMost features can be thought of as templates, histograms (counts), or combinationsThe ideal descriptor should be

Robust and Distinctive

Compact and Efficient

Most available descriptors focus on edge/gradient information

Capture texture information

Color rarely used

K. Grauman, B. Leibe

Slide21

Feature MatchingSzeliski 4.1.3Simple feature-space methodsEvaluation methodsAcceleration methodsGeometric verification (Chapter 6)

Slide22

Feature MatchingSimple criteria: One feature matches to another if those features are nearest neighbors and their distance is below some threshold.Problems:Threshold is difficult to setNon-distinctive features could have lots of close matches, only one of which is correct

Slide23

Distance: 0.34, 0.30, 0.40

Distance: 0.61, 1.22

How do we decide which features match?

Slide24

Nearest Neighbor Distance Ratio

where NN1 is the distance to the first nearest neighbor and NN2 is the distance to the second nearest neighbor.

Sorting by this ratio puts matches in order of confidence.

 

Slide25

Can we refine this further?

Slide26

Fitting: find the parameters of a model that best fit the data Alignment: find the parameters of the transformation that best align matched points

Slide27

Fitting and AlignmentDesign challengesDesign a suitable goodness of fit measureSimilarity should reflect application goalsEncode robustness to outliers and noise

Design an

optimization

method

Avoid local optima

Find best parameters quickly

Slide28

Fitting and Alignment: MethodsGlobal optimization / Search for parametersLeast squares fitRobust least squaresIterative closest point (ICP)

Hypothesize and test

Generalized Hough transform

RANSAC

Slide29

Simple example: Fitting a line

Slide30

Least squares line fittingData:

(

x

1

,

y

1

), …, (

xn, y

n

)

Line equation:

y

i

= m

x

i + bFind (m,

b) to minimize

(

x

i

,

y

i

)

y=mx+b

Matlab

:

p = A \ y;

Modified from S. Lazebnik

Slide31

Problem with “vertical” least squaresNot rotation-invariantFails completely for vertical lines

Slide from S. Lazebnik

Slide32

Total least squaresIf (a2+b

2

=

1) then

Distance between point

(

x

i

, yi) is

|

ax

i

+

by

i

+ c

|

(

x

i

,

y

i

)

ax+by+c

=0

Unit normal:

N=

(

a, b

)

Slide modified from S. Lazebnik

proof:

http://mathworld.wolfram.com/Point-LineDistance2-Dimensional.html

Slide33

Total least squaresIf (a2+b2=

1) then

Distance between point

(

x

i

,

y

i) is |ax

i

+

by

i

+ c

|

Find

(

a,

b, c) to minimize the sum of squared perpendicular distances

(

x

i

,

y

i

)

ax+by+c

=0

Unit normal:

N=

(

a, b

)

Slide modified from S. Lazebnik

Slide34

Total least squaresFind (a,

b

,

c

)

to minimize the sum of squared perpendicular distances

(

x

i

,

y

i

)

ax+by+c

=0

Unit normal:

N=

(

a, b

)

Solution is eigenvector corresponding to smallest

eigenvalue

of A

T

A

See details on Raleigh Quotient:

http://en.wikipedia.org/wiki/Rayleigh_quotient

Slide modified from S. Lazebnik

Slide35

Recap: Two Common Optimization Problems

Problem statement

Solution

Problem statement

Solution

(

matlab

)

Slide36

Least squares (global) optimizationGoodClearly specified objectiveOptimization is easyBadMay not be what you want to optimize

Sensitive to outliers

Bad matches, extra points

Doesn’t allow you to get multiple good fits

Detecting multiple objects, lines, etc.

Slide37

Least squares: Robustness to noiseLeast squares fit to the red points:

Slide38

Least squares: Robustness to noiseLeast squares fit with an outlier:

Problem: squared error heavily penalizes outliers

Slide39

Fitting and Alignment: MethodsGlobal optimization / Search for parametersLeast squares fitRobust least squaresIterative closest point (ICP)

Hypothesize and test

Generalized Hough transform

RANSAC

Slide40

Robust least squares (to deal with outliers)General approach: minimize

u

i

(

x

i, θ

)

– residual of

i

th

point

w.r.t

. model parameters

θ

ρ – robust function

with scale parameter σ

The robust function

ρ

Favors a configuration

with small residuals

Constant penalty for large residuals

Slide from S. Savarese

Slide41

Choosing the scale: Just right

The effect of the outlier is minimized

Slide42

The error value is almost the same for everypoint and the fit is very poor

Choosing the scale: Too small

Slide43

Choosing the scale: Too large

Behaves much the same as least squares

Slide44

Robust estimation: DetailsRobust fitting is a nonlinear optimization problem that must be solved iterativelyLeast squares solution can be used for initializationScale of robust function should be chosen adaptively based on median residual

Slide45

Other ways to search for parameters (for when no closed form solution exists)Line searchFor each parameter, step through values and choose value that gives best fit

Repeat (1) until no parameter changes

Grid search

Propose several sets of parameters, evenly sampled in the joint set

Choose best (or top few) and sample joint parameters around the current best; repeat

Gradient descent

Provide initial position (e.g., random)

Locally search for better parameters by following gradient

Slide46

Fitting and Alignment: MethodsGlobal optimization / Search for parametersLeast squares fitRobust least squaresIterative closest point (ICP)

Hypothesize and test

Generalized Hough transform

RANSAC

Slide47

Hypothesize and testPropose parametersTry all possibleEach point votes for all consistent parameters

Repeatedly sample enough points to solve for parameters

Score the given parameters

Number of consistent points, possibly weighted by distance

Choose from among the set of parameters

Global or local maximum of scores

Possibly refine parameters using inliers

Slide48

Fitting and Alignment: MethodsGlobal optimization / Search for parametersLeast squares fitRobust least squaresIterative closest point (ICP)Hypothesize and test

Generalized Hough transform

RANSAC

Slide49

Hough Transform: OutlineCreate a grid of parameter values

Each point votes for a set of parameters, incrementing those values in grid

Find maximum or local maxima in grid

Slide50

x

y

b

m

y = m x + b

Hough transform

Given a set of points, find the curve or line that explains the data points best

P.V.C. Hough,

Machine Analysis of Bubble Chamber Pictures,

Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959

Hough space

Slide from S. Savarese

Slide51

x

y

b

m

x

y

m

3

5

3

3

2

2

3

7

11

10

4

3

2

3

1

4

5

2

2

1

0

1

3

3

b

Hough transform

Slide from S. Savarese

Slide52

x

y

Hough transform

Issue : parameter space [

m,b

] is unbounded…

P.V.C. Hough,

Machine Analysis of Bubble Chamber Pictures,

Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959

Hough space

Use a polar representation for the parameter space

Slide from S. Savarese

Slide53

features

votes

Hough transform - experiments

Slide from S. Savarese

Slide54

features

votes

Need to adjust grid size or smooth

Hough transform - experiments

Noisy data

Slide from S. Savarese

Slide55

Issue: spurious peaks due to uniform noise

features

votes

Hough transform - experiments

Slide from S. Savarese

Slide56

1. Image  Canny

Slide57

2. Canny  Hough votes

Slide58

3. Hough votes  Edges

Find peaks and post-process

Slide59

Hough transform example

http://ostatic.com/files/images/ss_hough.jpg

Slide60

Incorporating image gradientsRecall: when we detect an edge point, we also know its gradient direction

But this means that the line

is uniquely determined!

Modified Hough transform:

For each edge point (

x,y

)

θ = gradient orientation at (x,y)

ρ

= x

cos

θ + y sin θ

H(θ,

ρ

) = H(θ,

ρ) + 1

end

Slide61

Finding lines using Hough transformUsing m,b parameterizationUsing r, theta parameterizationUsing oriented gradientsPractical considerationsBin sizeSmoothingFinding multiple lines

Finding line segments

Slide62

Hough TransformHow would we find circles?Of fixed radiusOf unknown radiusOf unknown radius but with known edge orientation

Slide63

Next lectureRANSACConnecting model fitting with feature matching