Corner detection continued - PowerPoint Presentation

1. Corner detection continued

2. The correspondence problem

3. A general pipeline for correspondenceIf sparse correspondences are enough, choose points for which we will search for correspondences (feature points)For each point (or every pixel if dense correspondence), describe point using a feature descriptorFind best matching descriptors across two images (feature matching)Use feature matches to perform downstream task, e.g., pose estimation

4. Characteristics of good feature pointsRepeatability / invarianceThe same feature point can be found in several images despite geometric and photometric transformations Saliency / distinctivenessEach feature point is distinctiveFewer ”false” matches

5. Goal: repeatabilityWe want to detect (at least some of) the same points in both images.Yet we have to be able to run the detection procedure independently per image.No chance to find true matches!Kristen Grauman

6. Repeatability / invarianceThe feature detector should “fire” at consistent places in spite of rotation, translation etc.Changes to the underlying image (rotations, translations, deformations) shouldn’t change where the detector “fires” : invarianceImage credit : L. Fei-Fei

7. Goal: distinctivenessThe feature point should be distinctive enough that it is easy to matchShould at least be distinctive from other patches nearby????

8. DistinctivenessMain idea: Translating window should cause large differences in patch appearance

9. Consider shifting the window W by (u,v)how do the pixels in W change?compare each pixel before and after bysumming up the squared differences (SSD)this defines an SSD “error” E(u,v):We want E(u,v) to be as high as possible for all u, v!Harris corner detection: the mathW

10. Taylor Series expansion of I:If the motion (u,v) is small, then first order approximation is goodPlugging this into the formula on the previous slide…Small motion assumption

11. Corner detection: the mathConsider shifting the window W by (u,v)define an SSD “error” E(u,v):W

12. Corner detection: the mathConsider shifting the window W by (u,v)define an “error” E(u,v):WThus, E(u,v) is locally approximated as a quadratic error function

13. Recall that we want E(u,v) to be as large as possible for all u,vWhat does this mean in terms of M?Interpreting the second moment matrixSecond moment matrix

14. Flat patch: M

15. Vertical edge: MM

16. Horizontal edge: MMM

17. What about edges in arbitrary orientation?

18. Solutions to Mx = 0 are directions for which E is 0: window can slide in this direction without changing appearance

19. Solutions to Mx = 0 are directions for which E is 0: window can slide in this direction without changing appearanceFor corners, we want no such directions to exist

20. uvE(u,v)E(u,v)E(u,v)E(u,v)vvvuuu

21. Eigenvalues and eigenvectors of M: x is an eigenvector of M with eigenvalue 0M is 2 x 2, so it has 2 eigenvalues with eigenvectors (eigenvectors have unit norm) 

22. Eigenvalues and eigenvectors of MEigenvalues and eigenvectors of MDefine shift directions with the smallest and largest change in errorxmax = direction of largest increase in Emax = amount of increase in direction xmaxxmin = direction of smallest increase in E min = amount of increase in direction xmin xmin xmaxMM

23.  E very high in all directionsCornerE remains close to 0 along  Edgeare small;E is almost 0 in all directions Flat patch  Interpreting the eigenvalues

24. Eigenvalues and eigenvectors of MEigenvalues and eigenvectors of MDefine shift directions with the smallest and largest change in errorxmax = direction of largest increase in Emax = amount of increase in direction xmaxxmin = direction of smallest increase in E min = amount of increase in direction xmin xmin xmaxMM

25.  E very high in all directionsCornerE remains close to 0 along  Edgeare small;E is almost 0 in all directions Flat patch  Interpreting the eigenvalues

26. Corner detection: the mathHow are max, xmax, min, and xmin relevant for feature detection?Need a feature scoring function

27. Corner detection: the mathHow are max, xmax, min, and xmin relevant for feature detection?Need a feature scoring functionWant E(u,v) to be large for small shifts in all directionsthe minimum of E(u,v) should be large, over all unit vectors [u v]this minimum is given by the smaller eigenvalue (min) of M

28. Corner detection summaryHere’s what you doCompute the gradient at each point in the imageCreate the M matrix from the entries in the gradientCompute the eigenvalues Find points with large response (min > threshold)Choose those points where min is a local maximum as features

29. Corner detection summaryHere’s what you doCompute the gradient at each point in the imageCreate the H matrix from the entries in the gradientCompute the eigenvalues. Find points with large response (min > threshold)Choose those points where min is a local maximum as features

30. The Harris operatormin is a variant of the “Harris operator” for feature detectionThe trace is the sum of the diagonals, i.e., trace(H) = h11 + h22Very similar to min but less expensive (no square root)Called the “Harris Corner Detector” or “Harris Operator”Actually the Noble variant of the Harris Corner DetectorLots of other detectors, this is one of the most popular

31.  Corner Edge Flat patch  Corner response function

32. The Harris operatorHarris operator

33. Harris Detector [Harris88]Second moment matrix331. Image derivatives2. Square of derivatives3. Gaussian filter g(sI)IxIyIx2Iy2IxIyg(Ix2)g(Iy2)g(IxIy)4. Cornerness function – both eigenvalues are stronghar5. Non-maxima suppression(optionally, blur first)

34. Weighting the derivativesIn practice, using a simple window W doesn’t work too wellInstead, we’ll weight each derivative value based on its distance from the center pixel

35. Harris detector example

36. f value (red high, blue low)

37. Threshold (f > value)

38. Find local maxima of f

39. Harris features (in red)