International Journal of Computer Vision Kluwer Academic Publishers - Pdf

126K - views

International Journal of Computer Vision Kluwer Academic Publishers

Manufactured in The Netherlands Matching Widely Separated Views Based on Af64257ne Invariant Regions INNE TUYTELAARS AND LUC VAN GOOL University of Leuven Kasteelpark Arenberg 10 B3001 Leuven Belgium tinnetuytelaarsesatkuleuvenacbe lucvangoolesatkul

Embed :
Pdf Download Link

Download Pdf - The PPT/PDF document "International Journal of Computer Vision..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

International Journal of Computer Vision Kluwer Academic Publishers






Presentation on theme: "International Journal of Computer Vision Kluwer Academic Publishers"‚ÄĒ Presentation transcript:

InternationalJournalofComputerVision59(1),61–85,20042004KluwerAcademicPublishers.ManufacturedinTheNetherlands.MatchingWidelySeparatedViewsBasedonAfřneInvariantRegionsINNETUYTELAARSANDLUCVANGOOLUniversityofLeuven,KasteelparkArenberg10,B-3001Leuven,Belgiumtinne.tuytelaars@esat.kuleuven.ac.beluc.vangool@esat.kuleuven.ac.be uytelaarsandVanGool igure1(a)Twoimagesofthesameobject.(b)Twoparallelogram-shapedpatchesastheyaregeneratedbythesystem:whentheviewpointchangestheshapesofthepatchesaretransformedautomaticallysuchthattheycoverthesamephysicalpartofthescene.Eachoftheselocalimagepatcheshasbeenextractedbasedonasingleimage.shapestocoverthesame,physicalpartofthescenein-dependentofviewpoint(undertheassumptionoflocalplanarity).Withchangingviewpoint,theseinvariantre-gionschangetheirshapeintheimage.Itisthankstotheviewpoint-dependencyoftheirshapeintheimagethattheregions’scenecontentcanremaininvariant.Asanexample,Fig.1(b)showstwoinvariantregionsforeachofthetwoviewsshowninFig.1(a).Theinvariant MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions63regionsdoindeedrepresentthesamepartofthebox.Thecruxofthematteristhattheywereextractedfromeachoftheviewsseparately,i.e.withoutanyinforma-tionabouttheotherview.Thisisimportantfrombothcomputationalandpracticalpointofview,asnopair-wisecomparisonsbetweenregionsarenecessaryfortheirextraction,andoneisnotlimitedtoapredeřnedsetofviewpoints.Scenescanvarywidely.Inordertomakesurethatasufřcientnumberofinvariantregionscanbeextracted,severaltypeshavebeenimplemented.Itisourintentiontobuildan‘opportunistic’systemthatexploitsseveraltypesofimagestructure,simplydependingonwhatisonoffer.Thisshouldmaximizetheapplicabilityofthemethodandthenumberofinvariantregionsfound.Hereweproposeaconstructionmethodbasedoncor-nersandonebasedonintensityextrema.Othersarecurrentlybeingconsidered.achieveefřcientmatchingoftheinvariantre-gions,theircolorpatternischaracterizedbyafeatureectorofmomentinvariants.Theyareinvariantun-derbothgeometricandphotometricchanges.Findingcorrespondinginvariantregionsthenboilsdowntothecomparisonofthesevectors.Additionaltestsonthemutualconsistencyofmatchesareperformedtoin-creaserobustness.Boththeregionsandtheirfeaturevectorsarein-ariantundergeometricchanges,whicharemodeledbyafřnetransformationsastheregionsaresmall,Theyarealsobothinvariantunderphotometricchanges,modeledbylineartransformationswithdif-ferentscalingsandoffsetsforeachofthethreecolorHence,correspondencescanbefoundunderawiderangeofviewingconditions.Notethat,contrarytotheregiondescription,theregionextractiondoesnotxplicitlyrelyoncolorinformation:regionsareex-tractedbasedonasinglecolorband.HencethesamemethodscanequallywellbeappliedtogreyscaleTheremainderofthepaperisorganizedasfollows.First,anoverviewofrelatedworkisgiveninSection2.Section3describestheselectionofanchorpoints.Thenexttwosectionsdiscusstwodifferentmethodsforextractinginvariantregions:řrstageometry-basedmethod(Section4)followedbyanintensity-basedmethod(Section5).Section6describeshowtheac-tualcorrespondencesearch,basedonafřnemomentinvariantscomputedovertheseregionsiscarriedout.ConsistencychecksthatcanbeusedtorejectfalsematchesareproposedinSection7.Section8discussessomeexperimentalresults.Section9concludesthepaper.2.RelatedWorkAnimportantsourceofinspirationforourapproachhasbeentheworkofSchmidetal.(1997).Theyiden-tifyspecial‘pointsofinterest’(incasucorners)andxtract2Dtranslationand2Drotationinvariantfea-turesfromtheintensitypatterninřxedcircularregionsaroundthesepoints(incasuthelocaljetasdeřnedbyoenderinkandVanDoorn(1987),basedonGaussianderivativesofimageintensity).Invarianceunderscal-ingishandledbyincludingcircularregionsofseveralsizes.Sincethelevelofinvarianceintheirmethodislimited,itisnotreallysuitedforwidebaselinestereoapplications.Nevertheless,theyobtainedremarkableresultsinthecontextofshortbaselinestereo,objectrecognitionanddatabaseretrieval—forlaterversionsoftheirsystemeveninspiteofverylargescalechanges(Dufournaudetal.,2000).SimilarresultshavebeenreportedforcolorimagesbyMontesinosetal.(2000).Someextensionstowardsafřneinvariantregionshavebeenreportedaswell.Lowe(1999)hasextendedtheseideastorealscale-invariance,usingcircularregionsthatmaximizetheoutputofadifferenceofgaussianřltersinscalespace,whileHalletal.(1999)notonlyappliedautomaticscaleselection(basedonLindeberg(1998)),butalsoretrievedtheorientationofthecircularregioninanunambiguousway.ideBaselineTechniquescopewithwiderbaselines,theafřnegeometricde-formationsintheimageshouldfullybetakenintoaccountduringthematchingprocess.Oneapproachistodeformapatchintheřrstimageinaniter-ativeway,untilitmoreorlessřtsapatchinthe uytelaarsandVanGoolsecondimage(Gruen,1985;SuperandKlarquist,1997).However,thesearchthatisinvolvedreducesthepracticalityofthisapproach.Incontrast,ourmethodisbasedontheextractionandmatchingofinvariantre-gions,andhenceworksonthetwoimagesseparately,withoutsearchingovertheentireimageorapplyingThisisakintotheapproachofPritchettandZisserman(1998)whostarttheirwidebaselinestereoalgorithmbyextractingquadranglespresentintheimageandmatchthesebasedonnormalizedcross-correlationtořndlocalhomographies,whicharethenxploitedinasearchforadditionalcorrespondences.However,theyuseshapesthatareexplicitlypresentintheimage,whileoursaredeterminedlocallybasedonthecolorpatternsaroundanchorpoints,sowearelessdependentonthepresenceofspeciřcstructuresinthescene.Hence,theapplicabilityofourmethodiswider.ellandCarlsson(2000)alsoproposedawidebase-linecorrespondencemethodbasedonafřneinvariance.TheyextractanafřneinvariantFourierdescriptionoftheintensityprořlealonglinesconnectingtwocornerpoints.Thenon-localcharacteroftheirmethodmakesitmorerobust,butatthesametimerestrictsitsusetounoccludedplanarobjects,whichlimitstheapplicabil-ityoftheirmethod.Insummary,oursystemdiffersfromotherwidebaselinestereomethodsinthatwedonotapplyasearchbetweenimagesbutprocesseachimageandeachlocalfeatureindividually(Gruen,1985;SuperandKlarquist,1997;SchaffalitzkyandZisserman,2001),inthatwefullytakeintoaccounttheafřnedeformationscausedbythechangeinviewpoint(Lowe,1999;Montesinosetal.,2000;SchmidandMohr,1997;Dufournaudetal.,2000)andinthatwecandealwithgeneral3Dob-jectswithoutassumingspeciřcstructurestobepresentintheimage(PritchettandZisserman,1998;TellandCarlsson,2000).AfřneInvariantRegionsOtherapproachestoextractingafřneinvariantregionsdescribedinliteraturearemainlysituatedinthecontextoftextureanalysis.BallesterandGonzales(1998)havedevelopedamethodtořndafřneinvariantregionsintexturedimages.Implicitly,theyusethefactthatthesecondmomentmatrixremainsmoreorlessconstantwhenvaryingtheregionparameters,whichmaybeareasonableassumptionfortexturesbutclearlydoesnotholdforgeneralimagepatches.LindebergandGķarding(1997)ontheotherhandhavedevelopedamethodtořndblob-likeregionsusinganiterativescheme,inthecontextofshapefromtex-ture.Inthecaseofweakisotropy,theregionsfoundbytheiralgorithmcorrespondtorotationallysymmetricsmoothingandrotationallysymmetricwindowfunc-tionsinthetangentplanetothesurface.However,ingeneral,theirmethoddoesnotnecessarilyconverge,asthereare,inmostcases,atleasttwoadditionalattractionpoints.Similarideashaverecentlybeenusedforwidebase-linestereobySchaffalitzkyandZisserman(2001).First,theyroughlymatchtexturedregionsintheim-age.Then,theyusetextureinformation(thesecondmomentmatrix)toliftsomedegreesoffreedom,fol-lowedbyanexhaustivesearchoverallHarriscornerpointswithinthatspeciřctextureandoverallpos-sible2Drotationstořndpointcorrespondencesun-derwidebaselineconditions.Byexploitingtextureinformation,theyavoidhavingtodelineateinvariantregions,butatthesametimethislimitstheapplica-bilityoftheirmethodtoimagescontainingstationarytextures.Baumberg(2000)proposedawidebaselinesystemthatisbasedonasimpliředversionoftheregionsofLindebergandGķarding(1997).However,there-gionsBaumbergusesareonlyinvariantunderrotation,stretchandskew,whilescalechangesaredealtwithbyapplyingascalespaceapproach.TheerroronthescalealsoinŖuencestheothercomponentsofthetrans-formation,suchthattheresultinginvariantregionsareprobablynotasaccurateasours.Nevertheless,webelievethatitcouldbebeneřcialtoincludetheaboveregionextractionmethodsintooursystemtofurtherimprovetheperformanceofthesystem(i.e.morecorrespondencesandawiderrangeofapplicability).3.SelectionofAnchorPointsTheřrststepintheextractionofafřneinvariantre-gionsconsistsofselecting‘anchorpoints’,thatserveasseedsforthesubsequentregionextraction.Thisallowstoreducethecomplexityoftheproblemandtheneededcomputationtime,sincetheattentioncanbefocussedonregionsaroundthesepointsinsteadofexaminingeverysinglepixelintheimage.Atthesametime,extraassumptionscanoftenbemadeconcerningtheregionsbasedonthetypeofanchor MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions65Goodanchorpointsarepointsthatresultinstablein-ariantregions,arerepeatableandeasy-to-detect.Withrepeatability,wemeanthatthereisahighprobabilitythatthesamepointwillbefoundinanotherviewaswell—oratleast,apointthatwouldresultinthesameregion.Harriscornerpoints(HarrisandStephens,1983)aregoodcandidates.Apartfromthenecessarypropertiesofgoodanchorpointsmentionedabove,theytypicallycontainalargeamountofinformation(SchmidandMohr,1998),resultinginahighdistinctivepower,andtheyarewelllocalized,i.e.thepositionofthecornerpointisaccuratelydeřned(evenuptosub-pixelaccu-racy)(ShiandTomasi,1994).Insteadofusingcorners,localextremaofimagein-tensitycanserveasanchorpointsaswell.Tothisend,weřrstapplysomesmoothingtotheimagetoreducetheeffectofnoise,causingtoomanyunstablelocalxtrema.Then,thelocalextremaareextractedwithnon-maximumsuppressionalgorithm.Thesepointscannotbelocalizedasaccuratelyascornerpoints,sincethelocalextremainintensityareoftenrathersmooth.However,theycanwithstandanymonotonicintensitytransformationandtheyarelesslikelytolieclosetotheborderofanobjectresultinginanon-planarregion.Thislastpropertyisamajordrawbackwhenworkingwithcornerpoints.Ofcourse,whichkindofanchorpointsperformbestalsodependsonthemethodusedfortheregionextrac-tion,andhowgoodthismethoddealswiththeshort-comingsoftheanchorpoints.Forinstance,forthecor-nerpoints,thehighchanceofanon-planarregioncanbealleviatedbyconstructingaregionthatisnotcen-teredaroundthecornerpoint.Similarly,regionsstart-ingfromlocalintensityextremashouldnotdependtoomuchontheexactpositionoftheextremum,toover-cometheinaccuratelocalizationofthesepoints.Othertypesofanchorpointscouldbeusedaswell.instance,Lowe(1999)usesextremaofadifferenceofGaussiansřlter.4.Geometry-BasedMethodTheřrstmethodforafřneinvariantregionextractionstartsfromHarriscornerpoints(HarrisandStephens,1983)andtheedgesthatcanoftenbefoundclosetosuchapoint(extractedusingtheCannyedgedetector(Canny,1986)).Asthismethodsostronglyreliesonthepresenceandaccuratedetectionofthesegeometricentities,wecoineditthemethod.Two igure2Basedontheedgesclosetothecornerpoint,anafřneinvariantregioncanbeconstructed.differentcasesareconsidered:onemethodisdevelopedforcurvededgeswhileaslightlydifferentmethodisappliedincaseofstraightedges.4.1.Case1:CurvedEdgesbeaHarriscornerpointonanedge,asinFig.2.Twopointsmoveawayfromthecornerinbothdirectionsalongtheedge.Theirrelativespeediscoupledthroughtheequalityofrelativeafřneinvariantparametersanarbitrarycurveparameter,theřrstderivativeofwithrespecttoabs()theabsolutealueandthedeterminant.Fromnowon,wesimplywhenreferringtoeachposition,thetwotogetherwiththecornerforthepointasafunctionoftheparal-lelogramspannedbythevectors(seeFig.2).Thisgivesusaonedimensionalfamilyofparallelogram-shapedregions.Thepointsstopatposi-tionswheresomephotometricquantitiesofthetexturecoveredbytheparallelogramgothroughanextremum.typicallygenerateregionsforafewextrema,whichintroducesakindofscaleconceptasnowregionsofdifferentsizescoexistforasinglecorner.Sinceitisnotguaranteedthatasinglefunctionwillreachanex-tremumoverthelimited-intervalwearelookingat,morethanonefunctionistested.Takingextremaofseveralfunctionsintoaccount,wegetabetterguaran-teethatahighnumberofcornerswillindeedgeneratesomeregions.Thankstoagoodchoiceofthefunctions,thewholeprocesscanbemadeinvariantundertheaforemen-tionedgeometricandphotometricchanges.Examples uytelaarsandVanGool igure3hysicalinterpretationofthefunctions(left)andofsuchfunctionsare M000f2()=abs |p1äpgp2äpg| |päp1päp2|◊M100  M200M000ä M100 2f3()=abs |päpgqäpg| |päp1päp2|◊M100  dxdy M100,M101 orderdegreemomentcom-putedovertheregionthecenterofgravityoftheregion,weightedwithintensity(oneofthethreecolorbands),andthecorneroftheparallelogramoppositetothecornerpointFig.2).Theřrstfunction,),representstheaveragein-tensityovertheregion).Itisnotinitselfinvari-antundertheconsideredphotometrictransformations,reachesitsextremainaninvariantway.Wedonotusethisfunctioninourimplementationthough,sincetheminimaoftendtobebetterlo-calizedthantheextremaof),resultinginmorestableregions.Neverthelesscouldbethebetterchoiceiftheapplicationneedshighspeed.consistoftwocomponentseach:řrst,aratiooftwoareas,oneofwhichdependsonthecenterofgravityweightedwithintensityandhenceonthere-gionpattern,andsecond,afactorthatcompensatesforthedependenceoftheřrstcomponenttooffsetsintheimageintensity.Figure3illustratesthegeometricalinterpretationoftheřrstcomponentfor(right)respectively.Itistwicetheratioofthemarkedarea,dividedbythetotalareaofthere-gion.Bylookingforlocalminimaofthesefunctionswefavormoreregions,i.e.regionsforwhichthecenterofgravityliesonorclosetooneofthedi-agonalsoftheparallelogram.Incontrastto),theareinvariant.Nevertheless,westillselecttheregionswherethefunctionreachesaminimuminsteadofselectingregionswherethefunc-tionreachesaspeciřcvalue,henceavoidingthein-troductionofanother(ratherarbitrary)parameter.Forproofofthegeometricandphotometricinvarianceofthelocalminimaofthesefunctions,werefertoAppendixA.Figure4showstwoinvariantparallelogram-shapedregionsfoundforcorrespondingpointsintwowidelyseparatedviewsofthesameobject.Althoughthereislargeimagedistortionbetweenthetwoimages(ge-ometricallyaswellasphotometrically),theafřnein-ariantregions—whichhavebeenfoundforeachim-ageindependently—coversimilarphysicalpartsofthescene.Forclarity,thecurvededgesonwhichtheex-tractionwasbasedareaddedaswell.Notethattheafřneinvariantregionsfoundarenotcenteredaroundtheanchorpoint.Acenteredalterna-tiveistheparallelogramthathasthenon-centeredpar-allelogramasonequadrant.Nevertheless,wepreferthenon-centeredregions,as—andexperimentshavebornethatout—restrictingtheregiontoonequadrant(delin-eatedbytheedges)makestheassumptionofplanaritymuchmorerealistic,duetothefactthattheanchorpointswestartfromarecorners,oftenlyingclosetoadepthdiscontinuity(seeSection3). MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions67 igure4řneinvariantregionsbasedoncornersandcurvededges.4.2.Case2:StraightEdgesInthecaseofstraightedges,themethoddescribedabovecannotbeapplied,sincealongtheentireedge.However,sincestraightedgesoccurquiteoften,wecannotsimplyneglectthiscase.straightforwardextensionoftheprevioustech-niquewouldthenbetosearchforlocalextremaina2Dsearch-spacespannedbytwoarbitraryparametersforthetwoedges,insteadofa1Dsearch-spaceoverer,thefunctionsweusedforthecurved-edgescase,donotshowclear,well-deřnedextremainthe2Dcase.Rather,wehavesomeshallowvalleysoflowvalues(correspondingtocaseswherethecenterofgravityliesonorclosetooneofthediagonals).Insteadoftakingtheinaccuratelocalxtremaofonefunction,wecombinethetwofunctionsandtaketheintersectionsofthetwovalleys,asshowninFig.5.Thespecialcasewherethetwovalleys(al-most)coincidemustbedetectedandrejected,sincetheintersectionisnotaccurateinthatcase.Theregionssoobtainedprovedtobemuchmorestablethanthosebasedona2Dlocalextremum.Figure6showssomeafřneinvariantregionsex-tractedforthesameimagesasinFig.4,butnowusingthemethoddesignedforstraightedges. uytelaarsandVanGool igure5orthestraightedgescase,theintersectionofthe“valleys”oftwodifferentfunctionsisusedinsteadofalocalextremum. igure6řneinvariantregionsbasedoncornersandstraightedges. MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions69Again,theyclearlycoveridenticalpartsofthe5.Intensity-BasedMethoddrawbackofthemethoddescribedintheprevioussectionisthattheedgesitreliesonareoftenasourceoferrors.Edgesthatwerefoundinoneimagemaybeundetected,interruptedorconnectedinadifferentwayinthesecondimage.Thissectionpresentsanalternativemethodforextractinginvariantregions,thatisdirectlybasedontheanalysisofimageintensity,withoutanintermediatestepinvolvingtheextractionoffeaturessuchasedgesorcorners.Itturnsouttocomplementthepreviousmethodverywell,inthatinvariantregionsaretypicallyfoundatdifferentlocationsintheimage.Insteadofstartingfromcornerpoints,thismethoduseslocalextremainintensityasanchorpoints(cfr.Section3).Givensuchalocalextremum,theintensityfunctionalongraysemanatingfromtheextremumisstudied,asshowninFig.7.Thefollowingfunctionisaluatedalongeachray: max t0abs(I(t)äI0) theEuclideanarclengthalongtheray,intensityatpositiontheintensityextremumandsmallnumberwhichhasbeenaddedtopreventvisionbyzero.Thepointforwhichthisfunctionreachesanextremumisinvariantundertheaforemen-tionedafřnegeometricandlinearphotometrictrans-formations(giventheray).Typically,amaximumis igure7Theintensityalong‘rays’emanatingfromalocalex-tremumarestudied.Thepointoneachrayforwhichafunctionreachesamaximumisselected.Linkingthesepointstogetheryieldsanafřneinvariantregion,towhichanellipseisřttedusingmoments.reachedatpositionswheretheintensitysuddenlyin-creasesordecreasesdramaticallycomparedtothein-tensitychangesencounteredonthelineuptothatpoint,forinstanceattheborderofamoreorlesshomogeneousarea.itselfalreadyinvari-ant.Nevertheless,weagainselectthepointswherethisfunctionreachesanextremumforreasonsofrobustness.Notethatintheory,leavingoutthedenominatorintheexpressionforouldyieldasimplerfunctionwhichstillhasinvariantpositionsforitslocalextrema.Inpractice,however,thissimplerfunctiondoesnotgiveasgoodresultssinceitslocalextremaaremoreshallow,resultingininaccuratepositionsalongtheraysandhenceinaccurateregions.Withthedenominatoradded,ontheotherhand,thelocalextremaareinmostcasesmoreaccuratelylocalized.Next,allpointscorrespondingtomaximaofalongraysoriginatingfromthesamelocalextremumarelinkedtoenclosean(afřneinvariant)region(seeagainFig.7).Thisoftenirregularly-shapedregionisre-placedbyanellipsehavingthesameshapemomentsuptothesecondorder.Thisellipse-řttingisagainafřneinvariant.Finally,wedoublethesizeoftheellipsesfound.Thisleadstomoredistinctiveregions,duetoamorediversiředtexturepatternwithintheregionandhencefacilitatesthematchingprocess,atthecostofahigherriskofnon-planarityduetothelesslocalchar-acteroftheregions.Problemsmayarisewhenmorethanonelocalex-tremumcanbefoundalongtheray.Insuchcase,in-steadofchoosingtheglobalextremum,weselectanxtremumbyimposingacontinuityconstraintofmultipleextrema,weselecttheextremumclosesttotheextremafoundalongtheneighbouringrays.Figure8showssomeintensity-basedregions(ellipses)andthelinkedpointsonwhichtheregionxtractionisbased.Notethattheresultingellipticalregionsarenotcen-teredaroundtheoriginalanchorpoint(theintensityex-tremum).Infact,thewholeprocedureisprettyrobusttotheinaccuratelocalizationofthispoint.Inmostcases(i.e.iftheareaenclosedbythelinkedpointsismoreorlessconvex),smallchangesinitspositionhaveonlylimitedeffectontheresultingregioniftheintensityprořleisindeedshowingashallowextremum.ThisisillustratedinFig.9,wherewerepeatedtheregionxtractionstartingfromdifferentanchorpointslyingclosetotheintensityextremumandhavingsimilarin-tensityvalues.Althoughtheellipticalregionsfoundare uytelaarsandVanGool igure8řneinvariantregionsfoundwiththeintensity-basedregionextractionmethodandthelinkedpointsusedtoextractnotidentical,theyaresimilarenoughtobematched.Tohighlightthesourceofthedeviations,wealsoaddedthelinkedpointsfoundalongtherays,usedintheregion6.FindingCorrespondencesOncelocal,invariantregionshavebeenextracted,řnd-ingcorrespondencesbetweentwoviewsbecomesrel-ativelyeasy.Thisisperformedbymeansofanear-estneighbourclassiřcationscheme,basedonfeatureectorsofinvariantscomputedovertheafřneinvari-antregions.Asintheregionextractionstep,wecon-siderinvariancebothunderafřnegeometricchangesandlinearphotometricchanges,withdifferentoffsetsanddifferentscalefactorsforeachofthethreecolor6.1.NormalizationAlthoughitisverywellpossibletoconstructafeatureectorthatisinitselfinvarianttoallthegeometricand MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions71 igure9Robustnessoftheregionextractiontotheinaccuratelocalizationoftheintensityextremum.photometrictransformationsweconsider(e.g.Mindruetal.,1999),ourexperimentsshowthatbetterresultsareobtainedifoneřrstcompensatesfor(partof)thedeformationsthroughanextranormalizationstep,ex-ploitingextraknowledgeabouttheregion.thegeometry-basedcase,weřrsttransformtheparallelogram-shapedregiontoasquarereferencere-gionofřxedsize.Sinceweknowaspeciřccorneroftheparallelogram(fromtheoriginalanchorpoint)andsinceitisreasonabletoassumethattheclockwiseor-derofthecornersispreserved(i.e.theimageisnotbeingmirrored),theentireafřnedeformationcanbecompensatedforinthisway.theintensity-basedcase,thesituationisslightlymorecomplex.Wecantransformtheellipticalregiontoacircularreferenceregionofřxedsize,but(againassumingtheimageisnotbeingmirrored)thisstillleavesonedegreeoffreedomtobedetermined(cor-respondingtoafreerotationofthecirclearounditscenter).Thislastdegreeoffreedomcannotbederivedfrompurelygeometricinformationgatheredduringtheregionextraction.Instead,wedetermineitbasedonaphotometricinvariantversionoftheaxesofinertia.Themajorandminoraxesofinertiaareextractedasthelinespassingthroughthecenterofthecircularregionwithdeřnedbythesolutionsof: thorder,řrstdegreemoment(seeSection4.1)centeredontheregion’sgeometriccen-Thisequationdiffersfromtheusualdeřnitionoftheaxesofinertiabytheuseofthesemomentsin-steadofmomentscenteredonthecenterofgravityweightedwithimageintensity.Thismakesthemin-arianttolinearintensitychanges(includingoffsets).Basedontheseaxesofinertia,onecanapplyanad-ditionalrotation,thatbringsthemajoraxisofinertiaintoahorizontalposition,henceřxingthelastdegreeoffreedom.Insteadofcomputingtheaxesofinertiatocom-pensateforthelastdegreeoffreedom,onecouldalsoxtractfeaturesthatareinvariantunderrotation.Thisouldprobablygivecomparableresults.However,re-trievingthecompleteafřnedeformationnotonlyal-lowstotreatintensity-basedandgeometry-basedre-gionsinthesamewaybutalsoallowstofurthercom-parethecontentoftwomatchedregionsinapixelwisemanner,basedonnormalizedcross-correlation,inde-pendentofthegeometricdistortions(seeSection6.3).Alsotheilluminationvariationscanbecompensatedforinanextranormalizationstep.Thisisachievedbyreplacingeachintensityvalue)bysuchthattheaverageintensityis128andwithaspreadontheintensitiesof50.6.2.RegionDescriptionEachregionisthencharacterizedbyafeaturevectorofmomentinvariants.ThemomentsweuseareGener-alizedColorMomentswhichhavebeenintroducedinMindruetal.(1999)tobetterexploitthemulti-spectralnatureofthedata.TheycontainpowersoftheimagecoordinatesandoftheintensitiesofthedifferentcolorabccR(x,y)]a[G(x,y)]b[B(x,y)]cdxdyorderpdegreeaact,theyimplicitlycharacterizetheshape,theintensityandthecolordistributionoftheregionpatterninauniformmanner.Moreprecisely,weuse18momentinvariants,sum-marizedinTable1.Theseareinvariantfunctionsofmo-mentsuptosecondorderandřrstdegree(i.e.momentsthatuseuptořrstorderpowersofintensities(andsecondorderpowersof(coordinates).Sincewealreadynormalizedtheregionswithrespecttoview-pointandilluminationvariations,measurementcanactuallybeusedasaninvariantmeasure,asallvari-ationshavebeencompensatedforalready.Therea-sonwhywestillsticktomomentsisthatthesearemorerobusttonoise.inv[1]toinv[3]arerelatedtothe uytelaarsandVanGoolable1omentinvariantsusedforcomparingthepatternswithinregionsafternormalizationagainstgeometricandphotomet-ricdeformations. invv=M11000/M00000invv=M01100/M00000invv=M10100/M00000invv=M10010/M10000invv=M01010/M01000invv=M00110/M00100invv=M10001/M10000invv=M01001/M01000invv=M00101/M00100invv=M10011/M10000invv=M01011/M01000invv=M00111/M00100invv=M10020/M10000invv=M01020/M01000invv=M00120/M00100invv=M10002/M10000invv=M01002/M01000inv correlationbetweentwocolor-bands.inv[4]toinvvandinv[7]toinv[9]arethe-coordinatesre-spectivelyofthecentersofgravityweightedwithonecolor-band,whileinv[10]toinv[18]arecombinationsofhigherordermoments.Asanadditionalinvariant,weusetheregionThisvaluereferstothemethodthathasbeenusedfortheregionextraction.Onlyifthetypeoftworegionscorresponds,cantheybematched.6.3.RegionMatchingEachregionintheřrstimageisthenmatchedtotheregioninthesecondimageforwhichtheMahalanobis-distancebetweenthecorrespondingfeaturevectorsisminimalandbelowapredeřnedthresholdThen,allregionsofthesecondimagearematchedinasimi-larwaytotheregionsextractedfromtheřrstimage.Onlyamutualmatchisacceptedasarealcorrespon-dencebetweenthetwoviews.ThecovariancematrixneededtocomputetheMahalanobis-distancehasbeenestimatedbytrackingrepresentativeregionsoverasetofimages.Duetothedifferentnatureofthedifferentregiontypes,betterresultsareobtainedwhendifferentcovariancematricesarecomputedforeachregiontypeseparately.Thecomparisonoffeaturevectorscanbedoneinanefřcientwayusingindexing-techniques.Atthismoment,onlyindexingbasedontheregiontypehasbeenimplemented.Oncecorrespondingregionshavebeenfound,thenormalizedcross-correlationbetweenthemiscom-putedasařnalcheckbeforeacceptingtheregioncor-respondence.Thiscross-correlationcheckisnotper-formedontherawimagedata,butafternormaliza-tionofthetworegionstoařxed-sizesquareorcir-cularreferenceregion(dependingontheregiontype),asdescribedinSection6.1.Inthisway,theeffectofthegeometricdeformationsonthenormalizedcross-correlationisannihilated.7.Robustness—RejectingFalselyMatchedRegionsDuetothewiderangeofgeometricandphotometrictransformationsallowedandthelocalcharacteroftheregions,falsecorrespondencesareinevitable.Thesecanbecausedbysymmetriesintheimage,orsim-plybecausethelocalregion’sdistinctivepowerisin-sufřcient.Semi-localorglobalconstraintsofferawayout:bycheckingtheconsistencybetweencombinationsoflocalcorrespondences(assumingarigidmotion),alsecorrespondencescanbeidentiředandrejected.Thebestknownconstraintischeckingforaconsistentepipolargeometryinarobustway,e.g.using(FischlerandBolles,1981),andrejectingallcorrespon-dencesnotconformwiththeepipolargeometryfound.Althoughthismethodworksřneinmanyapplications,ourexperimentshaveshownthatitmayhavedifřcul-tiesinatypicalwidebaselinestereosetup,wherefalsematchesaboundandmayevenoutnumberthegoodoneswhilethetotalnumberofmatchesisratherlow.Inthatcase,manyoftherandomlyselectedseven-pointsamplescontainoutliers,resultinginlargecomputationtimes(eachtimerejectingthesampleandtryingoutanewcombination),orevenerroneousresults(asamplecontainingoutlierscoincidentallyyieldingareason-ableamountofmatches).Thelattercasehappensmoreoftenthanexpected,sincematchesareingeneralrandomlyspreadovertheimage,buttendtoclutteronlinearorplanarstructuresinthescene.Here,twoothersemi-localconstraintsareproposedthatmaybeusedtorejectoutliers.Bothworkonacom-binationofregioncorrespondencesonly,hencetheamountofcombinatoricsneededislimited.Theřrstoneteststhegeometricconsistency,whilethesecondoneisaphotometricconstraint.Checkingthesecon-straintsřrstbeforetestingtheepipolargeometrywithcanconsiderablyimprovetheresultsunderthehardconditionsofwidebaselinestereo.ThisisakintotheworkofCarlsson(2000),whohasrecentlyproposedviewcompatibilityconstraintforřvepointsintwoviewsbasedonascaledorthographiccameramodel.7.1.AGeometricConstraintEachmatchbetweentwoafřneinvariantregionsde-řnesanafřnetransformation,matchingtheregioninoneimageonthecorrespondingregioninthesecondimage.Suchanafřnetransformationisinactanapproximationofthehomographylinkingtheprojectionsofallpointslyinginthesameplane. MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions73 igure10iewpointinvarianceoftheregionextractionandmatching:numberofcorrect,symmetricandfalsematchesfoundasafunctionoftherotationanglewithrespecttothe0degreesreferenceview.Sinclairetal.(1995)proposedamethodtotestwhethertworigidplanemotionsarecompatiblebasedontheirhomographiesCombiningthemasyieldsaplanarhomology,whoseeigenanaly-sisrevealsoneřxedpoint(theepipole)andonelineofřxedpoints(thecommonlineofthetwoplanes).Theyprojectthiscommonlinetotheotherimageusingandonceagainusingthetwoplanesareindeedinrigidmotion,thetworesultinglinesinthesecondimageshouldcoincide,whichcaneasilybechecked.Thegeometricconstraintweusehereisasimpleal-gebraicdistance.Asitonlyrequirestheevaluationofthedeterminantofa3matrix,itcanbeappliedquiteast.Thismakesitwellsuitedforapplicationslikeours,wheremanyconsistencychecksareperformedondif-ferentcombinationsofplanes(i.e.matches).Tocheckwhethertwocorrespondencesfoundaregeometricallyconsistentwithoneanother,itsufřcestocheckwhetherpredeřnedthreshold,threshold,aij]andB=[bij]theafřnetransformationsmappingtheregionintheřrstimagetotheregioninthesecondimage,fortheřrstandsecondmatchrespectively.Forthederivationofthissemi-localconstraint,werefertoAppendixB. uytelaarsandVanGool igure11Scaleinvarianceoftheregionextractionandmatching:numberofcorrect,symmetricandfalsematchesfoundasafunctionofthescalefactorwithrespecttothereferenceimage.7.2.APhotometricConstraintApartfromgeometricconstraints,photometriccon-straintscanbederivedaswell.Althoughitisnotnec-essarilytruethattheilluminationconditionsarecon-stantovertheentireimage(duetoshadows,multiplelightsources,etc.),itisreasonabletoassumethatatleastsomepartsoftheimageshavesimilarillumina-tionconditions.First,wecomputeforeachregioncorrespondencetheoffsetsandscalefactorsofthephotometrictrans-formationusingmoments.Then,givenapairofregioncorespondences,wecheckfortheirphoto-metricconsistencybycomparingtheirphotometrictransformations.Fortworegioncorrespondencestobeconsistent,onlyanoverallscalefactorisallowed,tocompensateforthedifferentorientationsoftheregions. MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions75 igure12Illuminationinvarianceoftheregionextractionandmatching:numberofcorrectandsymmetricmatchesfoundbetweentheimagesshownalongthehorizontalaxisandthereferenceimageshownontheright.7.3.RejectingFalseMatchesSupposewehavecorrespondences,eachlinkingadifferentlocalregioninimagetoasimilarregioninimagedifferenttransformations.Foreachcombinationoftwosuchcorrespondences,theaboveconsistencyconstraintscanbechecked.Aspeciřcre-gioncorrespondenceisconsideredincorrectifitiscon-sistentwithlessthanothercorrespondences(withtypically8forthegeometricconstraintand4forthephotometricconstraint).Henceeachgoodcorrespon-denceshouldhaveatleastotherconsistentcorre-spondences.Thisproceduremayhavetoberepeatednumberoftimes,sincerejectingacorrespondencemaycauseothercorrespondencestohavetheirnum-berofconsistentcorrespondencesdecreasedbelowthethresholdaswell.Afterhavingrejectedmostfalsematchesamongtheregioncorrespondencesusingthegeometricandphoto-metricconstraintsdescribedabove,weapply(FischlerandBolles,1981)(arobustmethodbasedonrandomsampling)tořndaconsistentepipolargeom-etryandtorejecttheremainingfalsecorrespondences.Sincethenumberoffalsematcheshasalreadyseri-ouslybeenreduced,thisprocessusuallystopsafteralimitednumberofsamples.Onemustnotethoughthatthecomputationofepipolargeometryisverysensitivetosmallmisallignmentsinthedata.Theregionmatcheswehavefoundsofargiveinmostcasesonlyonesta-blepointcorrespondence(e.g.theharriscornerpointincaseofthegeometry-basedmethod).Intheory,twomorelinearlyindependentpointcorrespondencescanbeextractedfromtheinvariantregion.However,theseadditionalpointcorrespondencesareinsufřcientlysta-blefortheepipolargeometrycomputation,mainlyduetodeviationsfromourmodel,suchastheobjectsur-acenotbeingperfectlyplanar.Thisproblemcanbeercomebymappingoneimageontotheotherus-ingtheafřnetransformation,andlookingformoreaccuratepointcorrespondenceswithinthematchedregionsusingsmallbaselinematchingtechniques.isthenappliedtotheresultingsetofpoint8.ExperimentalResults8.1.ViewpointInvariancequantitativelychecktheviewpointinvarianceofourmethod,wetookimagesofanobjectstartingfromheadonandgraduallyincreasingtheviewingangleinstepsof10degrees.AllimagesweretakenwithourSony uytelaarsandVanGool igure13Example1:Finalregioncorrespondences(top)andepipolargeometry(bottom). MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions77digitalcamera,witharesolutionof768576pixels.TheresultsofthisexperimentareshowninFig.10.eachimage,theafřneinvariantregionswerextractedandmatchedtotheregionsfoundinthe0degreesreferenceimage.Next,theregionswereřl- igure14Example2:Finalregioncorrespondences(top)andepipolargeometry(bottom).teredusingthesemi-localgeometricandphotometricconstraints.Finally,weappliedtheepipolartestusingtoautomaticallyselectthegoodmatches,anderiředthesematchesvisually,subdividingthemintothreedifferentcategories:correct,symmetricandfalse. uytelaarsandVanGool igure15Example3:Finalregioncorrespondences(top)andepipolargeometry(bottom).matches,werefertothosematchesthatdonotlinkphysicallyidenticalpoints,butpointsthatcannotbedistinguishedonalocalscaleduetoasymmetryintheimage.Forinstance,thetextonthedrinkcanusedinthisexperimentcontainstwicetheletter‘M’.Moreover,theselettersareexactlybelowoneanother,sotheyliemoreorlessonthesameepipolarlineduetothechosencameramovement.Asaresult,thereisnowayforthesystemtodistinguishbetweentheregionsfoundonthesetwoletters.FromFig.10,onecanseethatthesystemcandealwithchangesinviewpointupto50or60degrees.Onlycorrectandsymmetricmatcheswereleft.Forlargerangles,thegeometricconsistencytestcouldnolongerbeapplied,asthenumberofmatcheswastoolow(re-memberthatweneedatleastconsistentmatchestoclassifythemasgeometrically‘correct’).Thehori-zontallineaddedtotheřgureindicatestheminimumnumberofmatchesneededforthisgeometricřlter-ingstage.Itismainlythechangeinscaleduetotheforeshorteningoftheobjectthatcausesproblems,incombinationwithmoreandmorespecularreŖection.8.2.InvariancetoScaleChangesAsscalechangesseemtobetheweakestpointintheviewpointinvarianceoftheregions,weperformedsomeextraexperimentstospeciřcallytestforthein-ariancetoscalechanges.Forthesametestobject,imageswithdifferentscalesweretakenbyzoominginandoutwithourdigitalcamera.AscanbeseenfromFig.11,thenumberofmatchesfounddecreaseswithincreasingscalechange.Nevertheless,onecanconcludethattheextractionandmatchingofafřne MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions79 igure16Example4:Finalregioncorrespondences(top)andepipolargeometry(bottom).invariantregionsisabletowithstandscalefactorsrang-ingfrom23to32.Iflargerscalechangesaretobexpected,ascalespaceapproachshouldbeadopted.8.3.IlluminationInvarianceSincechangesintheilluminationarehardertoquantifythanchangesinscaleorviewpoint,wedecidedtousetheimagesprovidedbyFuntetal.(1998)totesttheilluminationinvarianceofoursystem,astheyprovideerydetailedinformationonthedifferentilluminantsused.Usingtheseimages,whicharereadilyavailableallowsforeasycomparisonofourre-sultswithothersystems.Figure12showstheresult.Eachoftheimagesshownbelowthehorizontalaxiscomparedwiththereferenceimagetakenunderhalogenilluminationshowntotheright.TheleftpartofeachimageshowsthewhitetoblackrowoftheMac-bethColorChecker,highlightingthelargedifferenceinillumination.Mostofthe‘symmetric’matchesfoundwereactuallymatchesbetweenthesereferencesquares.allimages,plentyofcorrespondenceswerefound,clearlyshowingtherobustnessofourregionextractionandmatchingtochangingilluminationconditions.8.4.WideBaselineStereoExamplesFigures13–17showsomeviewsofscenestakenfromsubstantiallydifferentviewpoints.Notethelargechangesinscaleinsomepartsoftheimages(e.g.Ex-ample3),theseriousocclusions(e.g.Example4)andtheextremeforeshortening(e.g.Example5).Never-theless,inallcasessufřcientmatcheswerefoundfor uytelaarsandVanGool igure17Example5:Finalregioncorrespondences(top)andepipolargeometry(bottom). igure18ativeexamples:Imagepairsoursystemwasnotabletomatch. MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions81anaccuratedeterminationoftheepipolargeometry.Sometimesthenumberofmatchedregionsisprettylow(e.g.Example4).However,onemustnotforgetthatasingleregioncorrespondenceyieldsthreepointcorrespondences.Eachtime,theupperpartoftheřg-ureshowstheregionsthatcontributedtotheepipolargeometry,i.e.thosethatwerematchedandsurvivedboththegeometricandphotometricřlteringaswellasSomecorrespondingepipolarlinesareshowninthelowerhalfoftheřgures.Finally,Fig.18showssomeexamplesofscenesoursystemwasabletoprocess.Althoughthesescenesdonotseemextraordinarilycomplexordif-řcult,thesystemfailed,mainlyduetothedifferentbackgrounds(car-example),thelackoftextureontheobjects(bothexamples),alargeamountofspecularreŖection(car-example)andnon-planarity(simpsons-xample).Theseimagesclearlyshowsomepossiblefutureresearchdirections.9.Conclusionapproachtothewidebaselinestereocorrespon-denceproblemhasbeenproposed,thatextendstheideasofSchmidandMohronlocalinvariantfeaturestowardsmoreinvarianceandhencewiderbaselines.Ineachimage,localimagepatchesareextractedinanafřneinvariantway,suchthattheycoverthesamephysicalpartofthescene(undertheassumptionoflo-calplanarity).Thesepatchesor‘invariantregions’arematchedbasedonfeaturevectorsofmomentinvariantsthatcombineinvarianceundergeometricandphoto-metricchanges.Theconsistencyofthematchesfoundistestedusingsemi-localconstraints,followedbyatestontheepipolargeometryusing.Asshownintheexperimentalresults,thefeasibilityofafřneinvari-anceevenonalocalscalehasbeendemonstrated.Robustmatchingisquiteagenericprobleminvi-sionandseveralotherapplicationscanbeconsid-ered.Objectrecognitionisone,whereimagesofanobjectcanbematchedagainstasmallsetofrefer-enceimagesofthesameobject.Thesamplesetcanbekeptsmallbecauseoftheinvariance.Moreover,asthefeaturesarelocal,recognitionagainstvariablebackgroundsandunderocclusionissupportedbythismethod.Anotherapplicationisgrouping,wheresym-metriescanbefoundasrepeatedstructures.Imagedatabaseretrievalcanalsobeneřtfromtheseregions,whereotherpicturesofthesamesceneorobjectcanbefound.Here,theviewpointandilluminationinvari-ancegivesthesystemthecapacitytogeneralizetoagreatextentfromasinglequeryimage.Finally,be-ingabletomatchacurrentviewagainstlearnedviewscanallowrobotstoroamextendedspaces,withouttheneedfora3Dmodel.Initialresultsforsuchap-plicationscanbefoundinTuytelaarsandVanGool(1999),Tuytelaarsetal.(1999)andTurinaetal.ppendixA:AfřneInvarianceoftheFunctionExtremaSupposewehavethefollowinggeometricandphoto-metricdeformationsbetweentwoviews:with(and(thethreedifferentcolor-bandsand(and(coordinatesofcorrespond-ingpoints.Inthesequel,weusetorefertoeitherofthethreecolor-bandsnowprovethattheextremaofthefunctionsgiveninSection4.1areinvarianttotheabovedefor-mations.Inotherwords,foreachregioninimageforwhichreachesanextremum,theremustbeacorrespondingregioninimageforwhichreachesanextremumaswell,with)orAfřneInvariantExtremaoffAsmentionedalreadyearlier,theřrstfunctionrepre-sentstheaverageintensityovertheregion.Theextremaofthisfunctionbeinginvarianttotheconsideredde-formations,caneasilybeunderstoodintuitively.Here,wegiveamoreformalproof.dxdy dxdy  uytelaarsandVanGooldxdy dxdydxdy dxdyInpractice,isalwayspositive,suchthatHence,extremaofthefunctionarepreservedundertheconsidereddeformations.Evenifhavebeennegative,extremawouldstillbepreserved,althoughmaximawouldbeturnedintominimaandviceversa.EffectsoftheDeformationsontheCenterofGravitytheotherfunctionsmentionedinSection4.1,itisimportanttořrstfullyunderstandtheeffectofthedeformationsonthecenterofgravityxdxdy dxdyydxdy dxdyFirst,letusconsideronlygeometricdeformations.Inthatcase,wegetfor dxdy dxdyxdxdy dxdyydxdy dxdy dxdy dxdyxdxdy dxdyydxdy dxdyHence,thecenterofgravitybehavesasanormalpointundertheafřnedeformations.Now,letusconsidertheeffectofphotometricde-formations.Here,weinvestigatethecoordinatesofthecenterofgravityrelativetothecoordinatesoftheregioncenter M000,M001 Itcanbeshownthattheeffectofthephotometricde-formationsonisashifttowards I(x,y)äx dxdy (I(x,y)+t dxdy M100+t sM000ygäyc=I(x,y)y I(x,y)äy dxdy (I(x,y)+t dxdy M100+t AfřneInvariantExtremaoffandfarebothcomposedoftwofactors,aratiooftwoareas,oneofwhichdependsonthecenterofgravity,andanexpressionofmomentsuptothesecondorder. |päp1päp2|◊M100  M200M000ä M100 2f3()=abs |päpgqäpg| |päp1päp2|◊M100  Theřrstfactorisaratiooftwoareas,deřnedbytheřxedtotheregionandthecenterof MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions83gravityseenintheprevioussection,thecenterofgravitybehavesasanormal,physicalpointundertheafřnegeometricdeformations,suchthatthisřrstfactorclearlyisgeometricallyinvariant.Alsothesecondfactorcaneasilybecheckedtobeinvarianttothegeometricaldeformations.Next,weshowthattheeffectofthephotometricde-formationsonthisřrstfactorissimilartotheireffectonthecoordinatesofthecenterofgravityrelativetotheregioncenter,namelyarescalingwiththesamescale-actor.Thiscanbeunderstoodbythefactthattheregionliesonthediagonalsoftheparallelogram-shapedregion,i.e.onthelineconnectingonehandandthelineconnectingontheotherhand,whichalsoformonesideoftheareasinthenu-merator(seeFig.3).Hencetheshiftinthepositionofthecenterofgravitycausesaproportionalrescalingoftheareainthenumerator: |päp1päp2|=|p1äpgp2äpg| |päp1päp2|M100 M100+t sM000|päpgqäpg| |päp1päp2|=|päpgqäpg| |päp1päp2|M100 M100+t Thisextrascale-factormustbecompensatedforbythesecondcomponentintheexpressionsof).Andindeed,thesecondcomponentseemstohaveexactlytheinversescale-factor: M20 M12=I(x,y)  ∑∑∑= M20M111t sM0 ppendixB:DerivationofaGeometricSemi-LocalConstraintConsidertwoimagesPointsinimagearede-notedwithhomogeneouscoordinateswhilepointsinimagearedenotedwithhomoge-neouscoordinatesorthecoordi-natesofrealworld(3D)points,capitallettersareused,suchas).Ahomographybelongingtodeřnesthefollowingrelationbetweentheprojectionsinimagesof3Dpointslyingontheanarbitrarypointinimagecorrespondingtothe3DpointtwohomographiescorrespondingtotwodifferentplanesThen,bothlieontheepipolarlinecorrespondingtothepointinthesecondimage.Hence,thefollowingformulafortheepipolarlinecorrespondingtothepointcanbederiveddenotesthevectorproduct.AllepipolarlinespassthroughthesamepointFromthisproperty,wecanderiveaconstraintondenotesthe-thcolumnofmatrixthiscanbeworkedoutasfollows:):[(Thisisasecond-orderequationinwithco-efřcientsfunctionsofDxyExzFyzSincethisequationhastobefulřlledforallpossibleallthecoefřcientsintheequationhavetobezero. uytelaarsandVanGoolInorderforalltheaboveequationstohaveasolutionthefollowingmatrix,whichisafunctionmustberank-deřcient.AppliedtoLocalRegionslocalregions,theperspectivedeformationistoosmalltobedetected.Asaresult,onlyanafřnetransformationcanbederived.Inthiscase,thehomographiescanbeapproximatedbyafřnetransformationsofthefollowingTherank-2constraintderivedintheprevioussectionthenbecomes:Rows(1),(2)and(4)forcetheepipoletolieatin-řnity.Thiscorrespondstoanorthographicprojectionmodel,whichindeedleadstoafřnetransformationsbetweentwoviewsofaplanarobject.ButalsowithoutforcingtheepipoletoinřnitythereisoneconstraintTheactualconsistencyconstraintusedinourxperimentsisthenpredeřnedthreshold.AcknowledgmentsaregratefultoRobotVisINRIASophia-AntipolisforprovidingtheValbonneimages(Fig.13)andforřnancialsupportfromtheECprojectVIBESandtheIUAPproject‘AdvancedMechatronicalSystems’.inneTuytelaarsisapostdoctoralresearcherfundedbytheFundforScientiřcResearchFlanders(Belgium).1.Alternatively,onecouldleaveoutthissecondfactor,andcom-pensatefortheoffsetsbyanappropriatenormalizationoftheintensitiesbeforecomputingthemoments.2.Formoreinformationabouttheseimages,see ReferencesBallester,C.andGonzalez,M.1998.Afřneinvarianttextureseg-mentationandshapefromtexturebyvariationalmethods.ofMathematicalImagingandVisionBaumberg,A.2000.Reliablefeaturematchingacrosswidelysepa-ratedviews.InProc.IEEEConf.onComputerVisionandPatternRecognitionpp.774–781.Canny,J.F.1986.Acomputationalapproachtoedgedetection.ans.onPatternAnalysisandMachineIntelligenceCarlsson,S.2000.Recognizingwalkingpeople.InProc.EuropeanConferenceonComputerVisionpp.472–486.Dufournaud,Y.,SchmidC.,andHoraud,R.2000.Matchingimagewithdifferentresolutions.InProc.IEEEConf.onComputerVisionandPatternRecognitionpp.612–618.Fischler,M.A.andBolles,R.C.1981.Randomsamplingconsensus—Aparadigmformodelřttingwithapplicationstoim-ageanalysisandautomatedcartography.Commun.Assoc.Comp.MachFunt,B.,Barnard,K.,andMartin,L.1998.Iscolourconstancygoodenough?InProc.EuropeanConferenceonComputerVi-pp.445–459.Gruen,A.W.1985.Adaptiveleastsquarescorrelation:Apowerfulimagematchingtechnique.ournalofPhotogrammetry,RemoteSensingandCartography MatchingWidelySeparatedViewsBasedonAfřneInvariantRegions85Hall,D.,ColindeVerdi`ere,V.,andCrowley,L.Objectrecognitionusingcolouredreceptiveřelds.InProc.EuropeanConferenceonComputerVisionHarris,C.J.andStephens,M.1983.Acombinedcornerandedgedetector.InProc.AlveyVisionConf.pp.147–151.oenderink,J.J.andVanDoorn,A.1987.Representationoflocalgeometryinthevisualsystem.BiologicalCyberneticsLindeberg,T.1998.Featuredetectionwithautomaticscaleselection.Int.JournalofComputerVisionLindeberg,T.andGķarding,J.1997.Shape-adaptedsmoothinginestimationof3Ddepthcuesfromafřnedistortionsoflocal,2Dbrightnessstructure.ImageandVisionComputingLowe,D.1999.Objectrecognitionfromlocalscale-invariantfea-tures.InProc.Int.Conf.onComputerVisionpp.1150–1157.Mindru,F.,Moons,T.,andVanGool,L.1999.Recognizingcolorpatternsirrespectiveofviewpointandillumination.InProc.IEEEConf.onComputerVisionandPatternRecognitionpp.368–373.Montesinos,P.,Gouet,V.,andPele,D.2000.Matchingcoloruncali-bratedimagesusingdifferentialinvariants.ImageandVisionCom-SpecialIssueBMVC2000,ElsevierScience,18(9):659–Pritchett,P.andZisserman,A.1998.Widebaselinestereo.InProc.Int.Conf.onComputerVisionpp.754–759.Schaffalitzky,F.andZisserman,A.2001.Viewpointinvarianttexturematchingandwidebaselinestereo.InProc.Int.Conf.onComputerpp.636–643.Schmid,C.,Mohr,R.,andBauckhage,C.1997.Localgrey-valueinvariantsforimageretrieval.Int.JournalonPatternAnalysisandMachineIntelligenceSchmid,C.andMohr,R.1998.Comparingandevaluatinginterestspoints.InProc.Int.Conf.onComputerVisionpp.230–235.Shi,J.andTomasi,C.1994.Goodfeaturestotrack.InProc.Int.Conf.onComputerVisionandPatternRecognitionpp.593–Sinclair,D.,Christensen,H.,andRothwell,C.1995.Usingtherela-tionbetweenaplaneprojectivityandthefundamentalmatrix.InProc.ScandinavianConf.onImageAnalysispp.181–188.Super,B.J.andKlarquist,W.N.1997.Patchmatchingandstereop-sisinageneralstereoviewinggeometry.Int.JournalonPatternAnalysisandMachineIntelligenceell,D.andCarlsson,S.2000.Widebaselinepointmatchingusingafřneinvariantscomputedfromintensityprořles.InProc.Euro-peanConf.onComputerVisionpp.814–828.uytelaars,T.andVanGool,L.1999.Content-basedimageretrievalbasedonlocalafřnelyinvariantregions.InProc.Int.Conf.onisualInformationSystemspp.493–500.uytelaars,T.,VanGool,L.,D’haene,L.,andKoch,R.1999.Match-ingafřnelyinvariantregionsforvisualservoing.InProc.Int.Conf.onRoboticsandAutomationpp.1601–1606.uytelaars,T.,Turina,A.,andVanGool,L.2003.Non-combinatorialdetectionofregularrepetitionsunderperspectiveskew.ans.onPatternAnalysisandMachineIntelligence