International Journal of Computer Vision Kluwer Academic Publishers - PDF document

InternationalJournalofComputerVision59(1),61Ð85,20042004KluwerAcademicPublishers.ManufacturedinTheNetherlands.MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegionsINNETUYTELAARSANDLUCVANGOOLUniversityofLeuven,KasteelparkArenberg10,B-3001Leuven,Belgiumtinne.tuytelaars@esat.kuleuven.ac.beluc.vangool@esat.kuleuven.ac.be uytelaarsandVanGool igure1(a)Twoimagesofthesameobject.(b)Twoparallelogram-shapedpatchesastheyaregeneratedbythesystem:whentheviewpointchangestheshapesofthepatchesaretransformedautomaticallysuchthattheycoverthesamephysicalpartofthescene.Eachoftheselocalimagepatcheshasbeenextractedbasedonasingleimage.shapestocoverthesame,physicalpartofthescenein-dependentofviewpoint(undertheassumptionoflocalplanarity).Withchangingviewpoint,theseinvariantre-gionschangetheirshapeintheimage.Itisthankstotheviewpoint-dependencyoftheirshapeintheimagethattheregionsÕscenecontentcanremaininvariant.Asanexample,Fig.1(b)showstwoinvariantregionsforeachofthetwoviewsshowninFig.1(a).Theinvariant MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions63regionsdoindeedrepresentthesamepartofthebox.Thecruxofthematteristhattheywereextractedfromeachoftheviewsseparately,i.e.withoutanyinforma-tionabouttheotherview.Thisisimportantfrombothcomputationalandpracticalpointofview,asnopair-wisecomparisonsbetweenregionsarenecessaryfortheirextraction,andoneisnotlimitedtoapredeÞnedsetofviewpoints.Scenescanvarywidely.InordertomakesurethatasufÞcientnumberofinvariantregionscanbeextracted,severaltypeshavebeenimplemented.ItisourintentiontobuildanÔopportunisticÕsystemthatexploitsseveraltypesofimagestructure,simplydependingonwhatisonoffer.Thisshouldmaximizetheapplicabilityofthemethodandthenumberofinvariantregionsfound.Hereweproposeaconstructionmethodbasedoncor-nersandonebasedonintensityextrema.Othersarecurrentlybeingconsidered.achieveefÞcientmatchingoftheinvariantre-gions,theircolorpatternischaracterizedbyafeatureectorofmomentinvariants.Theyareinvariantun-derbothgeometricandphotometricchanges.Findingcorrespondinginvariantregionsthenboilsdowntothecomparisonofthesevectors.Additionaltestsonthemutualconsistencyofmatchesareperformedtoin-creaserobustness.Boththeregionsandtheirfeaturevectorsarein-ariantundergeometricchanges,whicharemodeledbyafÞnetransformationsastheregionsaresmall,Theyarealsobothinvariantunderphotometricchanges,modeledbylineartransformationswithdif-ferentscalingsandoffsetsforeachofthethreecolorHence,correspondencescanbefoundunderawiderangeofviewingconditions.Notethat,contrarytotheregiondescription,theregionextractiondoesnotxplicitlyrelyoncolorinformation:regionsareex-tractedbasedonasinglecolorband.HencethesamemethodscanequallywellbeappliedtogreyscaleTheremainderofthepaperisorganizedasfollows.First,anoverviewofrelatedworkisgiveninSection2.Section3describestheselectionofanchorpoints.Thenexttwosectionsdiscusstwodifferentmethodsforextractinginvariantregions:Þrstageometry-basedmethod(Section4)followedbyanintensity-basedmethod(Section5).Section6describeshowtheac-tualcorrespondencesearch,basedonafÞnemomentinvariantscomputedovertheseregionsiscarriedout.ConsistencychecksthatcanbeusedtorejectfalsematchesareproposedinSection7.Section8discussessomeexperimentalresults.Section9concludesthepaper.2.RelatedWorkAnimportantsourceofinspirationforourapproachhasbeentheworkofSchmidetal.(1997).Theyiden-tifyspecialÔpointsofinterestÕ(incasucorners)andxtract2Dtranslationand2Drotationinvariantfea-turesfromtheintensitypatterninÞxedcircularregionsaroundthesepoints(incasuthelocaljetasdeÞnedbyoenderinkandVanDoorn(1987),basedonGaussianderivativesofimageintensity).Invarianceunderscal-ingishandledbyincludingcircularregionsofseveralsizes.Sincethelevelofinvarianceintheirmethodislimited,itisnotreallysuitedforwidebaselinestereoapplications.Nevertheless,theyobtainedremarkableresultsinthecontextofshortbaselinestereo,objectrecognitionanddatabaseretrievalÑforlaterversionsoftheirsystemeveninspiteofverylargescalechanges(Dufournaudetal.,2000).SimilarresultshavebeenreportedforcolorimagesbyMontesinosetal.(2000).SomeextensionstowardsafÞneinvariantregionshavebeenreportedaswell.Lowe(1999)hasextendedtheseideastorealscale-invariance,usingcircularregionsthatmaximizetheoutputofadifferenceofgaussianÞltersinscalespace,whileHalletal.(1999)notonlyappliedautomaticscaleselection(basedonLindeberg(1998)),butalsoretrievedtheorientationofthecircularregioninanunambiguousway.ideBaselineTechniquescopewithwiderbaselines,theafÞnegeometricde-formationsintheimageshouldfullybetakenintoaccountduringthematchingprocess.OneapproachistodeformapatchintheÞrstimageinaniter-ativeway,untilitmoreorlessÞtsapatchinthe uytelaarsandVanGoolsecondimage(Gruen,1985;SuperandKlarquist,1997).However,thesearchthatisinvolvedreducesthepracticalityofthisapproach.Incontrast,ourmethodisbasedontheextractionandmatchingofinvariantre-gions,andhenceworksonthetwoimagesseparately,withoutsearchingovertheentireimageorapplyingThisisakintotheapproachofPritchettandZisserman(1998)whostarttheirwidebaselinestereoalgorithmbyextractingquadranglespresentintheimageandmatchthesebasedonnormalizedcross-correlationtoÞndlocalhomographies,whicharethenxploitedinasearchforadditionalcorrespondences.However,theyuseshapesthatareexplicitlypresentintheimage,whileoursaredeterminedlocallybasedonthecolorpatternsaroundanchorpoints,sowearelessdependentonthepresenceofspeciÞcstructuresinthescene.Hence,theapplicabilityofourmethodiswider.ellandCarlsson(2000)alsoproposedawidebase-linecorrespondencemethodbasedonafÞneinvariance.TheyextractanafÞneinvariantFourierdescriptionoftheintensityproÞlealonglinesconnectingtwocornerpoints.Thenon-localcharacteroftheirmethodmakesitmorerobust,butatthesametimerestrictsitsusetounoccludedplanarobjects,whichlimitstheapplicabil-ityoftheirmethod.Insummary,oursystemdiffersfromotherwidebaselinestereomethodsinthatwedonotapplyasearchbetweenimagesbutprocesseachimageandeachlocalfeatureindividually(Gruen,1985;SuperandKlarquist,1997;SchaffalitzkyandZisserman,2001),inthatwefullytakeintoaccounttheafÞnedeformationscausedbythechangeinviewpoint(Lowe,1999;Montesinosetal.,2000;SchmidandMohr,1997;Dufournaudetal.,2000)andinthatwecandealwithgeneral3Dob-jectswithoutassumingspeciÞcstructurestobepresentintheimage(PritchettandZisserman,1998;TellandCarlsson,2000).AfÞneInvariantRegionsOtherapproachestoextractingafÞneinvariantregionsdescribedinliteraturearemainlysituatedinthecontextoftextureanalysis.BallesterandGonzales(1998)havedevelopedamethodtoÞndafÞneinvariantregionsintexturedimages.Implicitly,theyusethefactthatthesecondmomentmatrixremainsmoreorlessconstantwhenvaryingtheregionparameters,whichmaybeareasonableassumptionfortexturesbutclearlydoesnotholdforgeneralimagepatches.LindebergandGúarding(1997)ontheotherhandhavedevelopedamethodtoÞndblob-likeregionsusinganiterativescheme,inthecontextofshapefromtex-ture.Inthecaseofweakisotropy,theregionsfoundbytheiralgorithmcorrespondtorotationallysymmetricsmoothingandrotationallysymmetricwindowfunc-tionsinthetangentplanetothesurface.However,ingeneral,theirmethoddoesnotnecessarilyconverge,asthereare,inmostcases,atleasttwoadditionalattractionpoints.Similarideashaverecentlybeenusedforwidebase-linestereobySchaffalitzkyandZisserman(2001).First,theyroughlymatchtexturedregionsintheim-age.Then,theyusetextureinformation(thesecondmomentmatrix)toliftsomedegreesoffreedom,fol-lowedbyanexhaustivesearchoverallHarriscornerpointswithinthatspeciÞctextureandoverallpos-sible2DrotationstoÞndpointcorrespondencesun-derwidebaselineconditions.Byexploitingtextureinformation,theyavoidhavingtodelineateinvariantregions,butatthesametimethislimitstheapplica-bilityoftheirmethodtoimagescontainingstationarytextures.Baumberg(2000)proposedawidebaselinesystemthatisbasedonasimpliÞedversionoftheregionsofLindebergandGúarding(1997).However,there-gionsBaumbergusesareonlyinvariantunderrotation,stretchandskew,whilescalechangesaredealtwithbyapplyingascalespaceapproach.Theerroronthescalealsoinßuencestheothercomponentsofthetrans-formation,suchthattheresultinginvariantregionsareprobablynotasaccurateasours.Nevertheless,webelievethatitcouldbebeneÞcialtoincludetheaboveregionextractionmethodsintooursystemtofurtherimprovetheperformanceofthesystem(i.e.morecorrespondencesandawiderrangeofapplicability).3.SelectionofAnchorPointsTheÞrststepintheextractionofafÞneinvariantre-gionsconsistsofselectingÔanchorpointsÕ,thatserveasseedsforthesubsequentregionextraction.Thisallowstoreducethecomplexityoftheproblemandtheneededcomputationtime,sincetheattentioncanbefocussedonregionsaroundthesepointsinsteadofexaminingeverysinglepixelintheimage.Atthesametime,extraassumptionscanoftenbemadeconcerningtheregionsbasedonthetypeofanchor MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions65Goodanchorpointsarepointsthatresultinstablein-ariantregions,arerepeatableandeasy-to-detect.Withrepeatability,wemeanthatthereisahighprobabilitythatthesamepointwillbefoundinanotherviewaswellÑoratleast,apointthatwouldresultinthesameregion.Harriscornerpoints(HarrisandStephens,1983)aregoodcandidates.Apartfromthenecessarypropertiesofgoodanchorpointsmentionedabove,theytypicallycontainalargeamountofinformation(SchmidandMohr,1998),resultinginahighdistinctivepower,andtheyarewelllocalized,i.e.thepositionofthecornerpointisaccuratelydeÞned(evenuptosub-pixelaccu-racy)(ShiandTomasi,1994).Insteadofusingcorners,localextremaofimagein-tensitycanserveasanchorpointsaswell.Tothisend,weÞrstapplysomesmoothingtotheimagetoreducetheeffectofnoise,causingtoomanyunstablelocalxtrema.Then,thelocalextremaareextractedwithnon-maximumsuppressionalgorithm.Thesepointscannotbelocalizedasaccuratelyascornerpoints,sincethelocalextremainintensityareoftenrathersmooth.However,theycanwithstandanymonotonicintensitytransformationandtheyarelesslikelytolieclosetotheborderofanobjectresultinginanon-planarregion.Thislastpropertyisamajordrawbackwhenworkingwithcornerpoints.Ofcourse,whichkindofanchorpointsperformbestalsodependsonthemethodusedfortheregionextrac-tion,andhowgoodthismethoddealswiththeshort-comingsoftheanchorpoints.Forinstance,forthecor-nerpoints,thehighchanceofanon-planarregioncanbealleviatedbyconstructingaregionthatisnotcen-teredaroundthecornerpoint.Similarly,regionsstart-ingfromlocalintensityextremashouldnotdependtoomuchontheexactpositionoftheextremum,toover-cometheinaccuratelocalizationofthesepoints.Othertypesofanchorpointscouldbeusedaswell.instance,Lowe(1999)usesextremaofadifferenceofGaussiansÞlter.4.Geometry-BasedMethodTheÞrstmethodforafÞneinvariantregionextractionstartsfromHarriscornerpoints(HarrisandStephens,1983)andtheedgesthatcanoftenbefoundclosetosuchapoint(extractedusingtheCannyedgedetector(Canny,1986)).Asthismethodsostronglyreliesonthepresenceandaccuratedetectionofthesegeometricentities,wecoineditthemethod.Two igure2Basedontheedgesclosetothecornerpoint,anafÞneinvariantregioncanbeconstructed.differentcasesareconsidered:onemethodisdevelopedforcurvededgeswhileaslightlydifferentmethodisappliedincaseofstraightedges.4.1.Case1:CurvedEdgesbeaHarriscornerpointonanedge,asinFig.2.Twopointsmoveawayfromthecornerinbothdirectionsalongtheedge.TheirrelativespeediscoupledthroughtheequalityofrelativeafÞneinvariantparametersanarbitrarycurveparameter,theÞrstderivativeofwithrespecttoabs()theabsolutealueandthedeterminant.Fromnowon,wesimplywhenreferringtoeachposition,thetwotogetherwiththecornerforthepointasafunctionoftheparal-lelogramspannedbythevectors(seeFig.2).Thisgivesusaonedimensionalfamilyofparallelogram-shapedregions.Thepointsstopatposi-tionswheresomephotometricquantitiesofthetexturecoveredbytheparallelogramgothroughanextremum.typicallygenerateregionsforafewextrema,whichintroducesakindofscaleconceptasnowregionsofdifferentsizescoexistforasinglecorner.Sinceitisnotguaranteedthatasinglefunctionwillreachanex-tremumoverthelimited-intervalwearelookingat,morethanonefunctionistested.Takingextremaofseveralfunctionsintoaccount,wegetabetterguaran-teethatahighnumberofcornerswillindeedgeneratesomeregions.Thankstoagoodchoiceofthefunctions,thewholeprocesscanbemadeinvariantundertheaforemen-tionedgeometricandphotometricchanges.Examples uytelaarsandVanGool igure3hysicalinterpretationofthefunctions(left)andofsuchfunctionsare M000f2(
)=abs |p1Špgp2Špg| |pŠp1pŠp2|×M100  M200M000Š M1002f3(
)=abs |pŠpgqŠpg| |pŠp1pŠp2|×M100  dxdy M100,M101 orderdegreemomentcom-putedovertheregionthecenterofgravityoftheregion,weightedwithintensity(oneofthethreecolorbands),andthecorneroftheparallelogramoppositetothecornerpointFig.2).TheÞrstfunction,),representstheaveragein-tensityovertheregion).Itisnotinitselfinvari-antundertheconsideredphotometrictransformations,reachesitsextremainaninvariantway.Wedonotusethisfunctioninourimplementationthough,sincetheminimaoftendtobebetterlo-calizedthantheextremaof),resultinginmorestableregions.Neverthelesscouldbethebetterchoiceiftheapplicationneedshighspeed.consistoftwocomponentseach:Þrst,aratiooftwoareas,oneofwhichdependsonthecenterofgravityweightedwithintensityandhenceonthere-gionpattern,andsecond,afactorthatcompensatesforthedependenceoftheÞrstcomponenttooffsetsintheimageintensity.Figure3illustratesthegeometricalinterpretationoftheÞrstcomponentfor(right)respectively.Itistwicetheratioofthemarkedarea,dividedbythetotalareaofthere-gion.Bylookingforlocalminimaofthesefunctionswefavormoreregions,i.e.regionsforwhichthecenterofgravityliesonorclosetooneofthedi-agonalsoftheparallelogram.Incontrastto),theareinvariant.Nevertheless,westillselecttheregionswherethefunctionreachesaminimuminsteadofselectingregionswherethefunc-tionreachesaspeciÞcvalue,henceavoidingthein-troductionofanother(ratherarbitrary)parameter.Forproofofthegeometricandphotometricinvarianceofthelocalminimaofthesefunctions,werefertoAppendixA.Figure4showstwoinvariantparallelogram-shapedregionsfoundforcorrespondingpointsintwowidelyseparatedviewsofthesameobject.Althoughthereislargeimagedistortionbetweenthetwoimages(ge-ometricallyaswellasphotometrically),theafÞnein-ariantregionsÑwhichhavebeenfoundforeachim-ageindependentlyÑcoversimilarphysicalpartsofthescene.Forclarity,thecurvededgesonwhichtheex-tractionwasbasedareaddedaswell.NotethattheafÞneinvariantregionsfoundarenotcenteredaroundtheanchorpoint.Acenteredalterna-tiveistheparallelogramthathasthenon-centeredpar-allelogramasonequadrant.Nevertheless,wepreferthenon-centeredregions,asÑandexperimentshavebornethatoutÑrestrictingtheregiontoonequadrant(delin-eatedbytheedges)makestheassumptionofplanaritymuchmorerealistic,duetothefactthattheanchorpointswestartfromarecorners,oftenlyingclosetoadepthdiscontinuity(seeSection3). MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions67 igure4Þneinvariantregionsbasedoncornersandcurvededges.4.2.Case2:StraightEdgesInthecaseofstraightedges,themethoddescribedabovecannotbeapplied,sincealongtheentireedge.However,sincestraightedgesoccurquiteoften,wecannotsimplyneglectthiscase.straightforwardextensionoftheprevioustech-niquewouldthenbetosearchforlocalextremaina2Dsearch-spacespannedbytwoarbitraryparametersforthetwoedges,insteadofa1Dsearch-spaceoverer,thefunctionsweusedforthecurved-edgescase,donotshowclear,well-deÞnedextremainthe2Dcase.Rather,wehavesomeshallowvalleysoflowvalues(correspondingtocaseswherethecenterofgravityliesonorclosetooneofthediagonals).Insteadoftakingtheinaccuratelocalxtremaofonefunction,wecombinethetwofunctionsandtaketheintersectionsofthetwovalleys,asshowninFig.5.Thespecialcasewherethetwovalleys(al-most)coincidemustbedetectedandrejected,sincetheintersectionisnotaccurateinthatcase.Theregionssoobtainedprovedtobemuchmorestablethanthosebasedona2Dlocalextremum.Figure6showssomeafÞneinvariantregionsex-tractedforthesameimagesasinFig.4,butnowusingthemethoddesignedforstraightedges. uytelaarsandVanGool igure5orthestraightedgescase,theintersectionoftheÒvalleysÓoftwodifferentfunctionsisusedinsteadofalocalextremum. igure6Þneinvariantregionsbasedoncornersandstraightedges. MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions69Again,theyclearlycoveridenticalpartsofthe5.Intensity-BasedMethoddrawbackofthemethoddescribedintheprevioussectionisthattheedgesitreliesonareoftenasourceoferrors.Edgesthatwerefoundinoneimagemaybeundetected,interruptedorconnectedinadifferentwayinthesecondimage.Thissectionpresentsanalternativemethodforextractinginvariantregions,thatisdirectlybasedontheanalysisofimageintensity,withoutanintermediatestepinvolvingtheextractionoffeaturessuchasedgesorcorners.Itturnsouttocomplementthepreviousmethodverywell,inthatinvariantregionsaretypicallyfoundatdifferentlocationsintheimage.Insteadofstartingfromcornerpoints,thismethoduseslocalextremainintensityasanchorpoints(cfr.Section3).Givensuchalocalextremum,theintensityfunctionalongraysemanatingfromtheextremumisstudied,asshowninFig.7.Thefollowingfunctionisaluatedalongeachray: max t0abs(I(t)ŠI0) theEuclideanarclengthalongtheray,intensityatpositiontheintensityextremumandsmallnumberwhichhasbeenaddedtopreventvisionbyzero.Thepointforwhichthisfunctionreachesanextremumisinvariantundertheaforemen-tionedafÞnegeometricandlinearphotometrictrans-formations(giventheray).Typically,amaximumis igure7TheintensityalongÔraysÕemanatingfromalocalex-tremumarestudied.Thepointoneachrayforwhichafunctionreachesamaximumisselected.LinkingthesepointstogetheryieldsanafÞneinvariantregion,towhichanellipseisÞttedusingmoments.reachedatpositionswheretheintensitysuddenlyin-creasesordecreasesdramaticallycomparedtothein-tensitychangesencounteredonthelineuptothatpoint,forinstanceattheborderofamoreorlesshomogeneousarea.itselfalreadyinvari-ant.Nevertheless,weagainselectthepointswherethisfunctionreachesanextremumforreasonsofrobustness.Notethatintheory,leavingoutthedenominatorintheexpressionforouldyieldasimplerfunctionwhichstillhasinvariantpositionsforitslocalextrema.Inpractice,however,thissimplerfunctiondoesnotgiveasgoodresultssinceitslocalextremaaremoreshallow,resultingininaccuratepositionsalongtheraysandhenceinaccurateregions.Withthedenominatoradded,ontheotherhand,thelocalextremaareinmostcasesmoreaccuratelylocalized.Next,allpointscorrespondingtomaximaofalongraysoriginatingfromthesamelocalextremumarelinkedtoenclosean(afÞneinvariant)region(seeagainFig.7).Thisoftenirregularly-shapedregionisre-placedbyanellipsehavingthesameshapemomentsuptothesecondorder.Thisellipse-ÞttingisagainafÞneinvariant.Finally,wedoublethesizeoftheellipsesfound.Thisleadstomoredistinctiveregions,duetoamorediversiÞedtexturepatternwithintheregionandhencefacilitatesthematchingprocess,atthecostofahigherriskofnon-planarityduetothelesslocalchar-acteroftheregions.Problemsmayarisewhenmorethanonelocalex-tremumcanbefoundalongtheray.Insuchcase,in-steadofchoosingtheglobalextremum,weselectanxtremumbyimposingacontinuityconstraintofmultipleextrema,weselecttheextremumclosesttotheextremafoundalongtheneighbouringrays.Figure8showssomeintensity-basedregions(ellipses)andthelinkedpointsonwhichtheregionxtractionisbased.Notethattheresultingellipticalregionsarenotcen-teredaroundtheoriginalanchorpoint(theintensityex-tremum).Infact,thewholeprocedureisprettyrobusttotheinaccuratelocalizationofthispoint.Inmostcases(i.e.iftheareaenclosedbythelinkedpointsismoreorlessconvex),smallchangesinitspositionhaveonlylimitedeffectontheresultingregioniftheintensityproÞleisindeedshowingashallowextremum.ThisisillustratedinFig.9,wherewerepeatedtheregionxtractionstartingfromdifferentanchorpointslyingclosetotheintensityextremumandhavingsimilarin-tensityvalues.Althoughtheellipticalregionsfoundare uytelaarsandVanGool igure8Þneinvariantregionsfoundwiththeintensity-basedregionextractionmethodandthelinkedpointsusedtoextractnotidentical,theyaresimilarenoughtobematched.Tohighlightthesourceofthedeviations,wealsoaddedthelinkedpointsfoundalongtherays,usedintheregion6.FindingCorrespondencesOncelocal,invariantregionshavebeenextracted,Þnd-ingcorrespondencesbetweentwoviewsbecomesrel-ativelyeasy.Thisisperformedbymeansofanear-estneighbourclassiÞcationscheme,basedonfeatureectorsofinvariantscomputedovertheafÞneinvari-antregions.Asintheregionextractionstep,wecon-siderinvariancebothunderafÞnegeometricchangesandlinearphotometricchanges,withdifferentoffsetsanddifferentscalefactorsforeachofthethreecolor6.1.NormalizationAlthoughitisverywellpossibletoconstructafeatureectorthatisinitselfinvarianttoallthegeometricand MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions71 igure9Robustnessoftheregionextractiontotheinaccuratelocalizationoftheintensityextremum.photometrictransformationsweconsider(e.g.Mindruetal.,1999),ourexperimentsshowthatbetterresultsareobtainedifoneÞrstcompensatesfor(partof)thedeformationsthroughanextranormalizationstep,ex-ploitingextraknowledgeabouttheregion.thegeometry-basedcase,weÞrsttransformtheparallelogram-shapedregiontoasquarereferencere-gionofÞxedsize.SinceweknowaspeciÞccorneroftheparallelogram(fromtheoriginalanchorpoint)andsinceitisreasonabletoassumethattheclockwiseor-derofthecornersispreserved(i.e.theimageisnotbeingmirrored),theentireafÞnedeformationcanbecompensatedforinthisway.theintensity-basedcase,thesituationisslightlymorecomplex.WecantransformtheellipticalregiontoacircularreferenceregionofÞxedsize,but(againassumingtheimageisnotbeingmirrored)thisstillleavesonedegreeoffreedomtobedetermined(cor-respondingtoafreerotationofthecirclearounditscenter).Thislastdegreeoffreedomcannotbederivedfrompurelygeometricinformationgatheredduringtheregionextraction.Instead,wedetermineitbasedonaphotometricinvariantversionoftheaxesofinertia.ThemajorandminoraxesofinertiaareextractedasthelinespassingthroughthecenterofthecircularregionwithdeÞnedbythesolutionsof: thorder,Þrstdegreemoment(seeSection4.1)centeredontheregionÕsgeometriccen-ThisequationdiffersfromtheusualdeÞnitionoftheaxesofinertiabytheuseofthesemomentsin-steadofmomentscenteredonthecenterofgravityweightedwithimageintensity.Thismakesthemin-arianttolinearintensitychanges(includingoffsets).Basedontheseaxesofinertia,onecanapplyanad-ditionalrotation,thatbringsthemajoraxisofinertiaintoahorizontalposition,henceÞxingthelastdegreeoffreedom.Insteadofcomputingtheaxesofinertiatocom-pensateforthelastdegreeoffreedom,onecouldalsoxtractfeaturesthatareinvariantunderrotation.Thisouldprobablygivecomparableresults.However,re-trievingthecompleteafÞnedeformationnotonlyal-lowstotreatintensity-basedandgeometry-basedre-gionsinthesamewaybutalsoallowstofurthercom-parethecontentoftwomatchedregionsinapixelwisemanner,basedonnormalizedcross-correlation,inde-pendentofthegeometricdistortions(seeSection6.3).Alsotheilluminationvariationscanbecompensatedforinanextranormalizationstep.Thisisachievedbyreplacingeachintensityvalue)bysuchthattheaverageintensityis128andwithaspreadontheintensitiesof50.6.2.RegionDescriptionEachregionisthencharacterizedbyafeaturevectorofmomentinvariants.ThemomentsweuseareGener-alizedColorMomentswhichhavebeenintroducedinMindruetal.(1999)tobetterexploitthemulti-spectralnatureofthedata.TheycontainpowersoftheimagecoordinatesandoftheintensitiesofthedifferentcolorabccR(x,y)]a[G(x,y)]b[B(x,y)]cdxdyorderpdegreeaact,theyimplicitlycharacterizetheshape,theintensityandthecolordistributionoftheregionpatterninauniformmanner.Moreprecisely,weuse18momentinvariants,sum-marizedinTable1.Theseareinvariantfunctionsofmo-mentsuptosecondorderandÞrstdegree(i.e.momentsthatuseuptoÞrstorderpowersofintensities(andsecondorderpowersof(coordinates).Sincewealreadynormalizedtheregionswithrespecttoview-pointandilluminationvariations,measurementcanactuallybeusedasaninvariantmeasure,asallvari-ationshavebeencompensatedforalready.Therea-sonwhywestillsticktomomentsisthatthesearemorerobusttonoise.inv[1]toinv[3]arerelatedtothe uytelaarsandVanGoolable1omentinvariantsusedforcomparingthepatternswithinregionsafternormalizationagainstgeometricandphotomet-ricdeformations. invv=M11000/M00000invv=M01100/M00000invv=M10100/M00000invv=M10010/M10000invv=M01010/M01000invv=M00110/M00100invv=M10001/M10000invv=M01001/M01000invv=M00101/M00100invv=M10011/M10000invv=M01011/M01000invv=M00111/M00100invv=M10020/M10000invv=M01020/M01000invv=M00120/M00100invv=M10002/M10000invv=M01002/M01000inv correlationbetweentwocolor-bands.inv[4]toinvvandinv[7]toinv[9]arethe-coordinatesre-spectivelyofthecentersofgravityweightedwithonecolor-band,whileinv[10]toinv[18]arecombinationsofhigherordermoments.Asanadditionalinvariant,weusetheregionThisvaluereferstothemethodthathasbeenusedfortheregionextraction.Onlyifthetypeoftworegionscorresponds,cantheybematched.6.3.RegionMatchingEachregionintheÞrstimageisthenmatchedtotheregioninthesecondimageforwhichtheMahalanobis-distancebetweenthecorrespondingfeaturevectorsisminimalandbelowapredeÞnedthresholdThen,allregionsofthesecondimagearematchedinasimi-larwaytotheregionsextractedfromtheÞrstimage.Onlyamutualmatchisacceptedasarealcorrespon-dencebetweenthetwoviews.ThecovariancematrixneededtocomputetheMahalanobis-distancehasbeenestimatedbytrackingrepresentativeregionsoverasetofimages.Duetothedifferentnatureofthedifferentregiontypes,betterresultsareobtainedwhendifferentcovariancematricesarecomputedforeachregiontypeseparately.ThecomparisonoffeaturevectorscanbedoneinanefÞcientwayusingindexing-techniques.Atthismoment,onlyindexingbasedontheregiontypehasbeenimplemented.Oncecorrespondingregionshavebeenfound,thenormalizedcross-correlationbetweenthemiscom-putedasaÞnalcheckbeforeacceptingtheregioncor-respondence.Thiscross-correlationcheckisnotper-formedontherawimagedata,butafternormaliza-tionofthetworegionstoaÞxed-sizesquareorcir-cularreferenceregion(dependingontheregiontype),asdescribedinSection6.1.Inthisway,theeffectofthegeometricdeformationsonthenormalizedcross-correlationisannihilated.7.RobustnessÑRejectingFalselyMatchedRegionsDuetothewiderangeofgeometricandphotometrictransformationsallowedandthelocalcharacteroftheregions,falsecorrespondencesareinevitable.Thesecanbecausedbysymmetriesintheimage,orsim-plybecausethelocalregionÕsdistinctivepowerisin-sufÞcient.Semi-localorglobalconstraintsofferawayout:bycheckingtheconsistencybetweencombinationsoflocalcorrespondences(assumingarigidmotion),alsecorrespondencescanbeidentiÞedandrejected.Thebestknownconstraintischeckingforaconsistentepipolargeometryinarobustway,e.g.using(FischlerandBolles,1981),andrejectingallcorrespon-dencesnotconformwiththeepipolargeometryfound.AlthoughthismethodworksÞneinmanyapplications,ourexperimentshaveshownthatitmayhavedifÞcul-tiesinatypicalwidebaselinestereosetup,wherefalsematchesaboundandmayevenoutnumberthegoodoneswhilethetotalnumberofmatchesisratherlow.Inthatcase,manyoftherandomlyselectedseven-pointsamplescontainoutliers,resultinginlargecomputationtimes(eachtimerejectingthesampleandtryingoutanewcombination),orevenerroneousresults(asamplecontainingoutlierscoincidentallyyieldingareason-ableamountofmatches).Thelattercasehappensmoreoftenthanexpected,sincematchesareingeneralrandomlyspreadovertheimage,buttendtoclutteronlinearorplanarstructuresinthescene.Here,twoothersemi-localconstraintsareproposedthatmaybeusedtorejectoutliers.Bothworkonacom-binationofregioncorrespondencesonly,hencetheamountofcombinatoricsneededislimited.TheÞrstoneteststhegeometricconsistency,whilethesecondoneisaphotometricconstraint.Checkingthesecon-straintsÞrstbeforetestingtheepipolargeometrywithcanconsiderablyimprovetheresultsunderthehardconditionsofwidebaselinestereo.ThisisakintotheworkofCarlsson(2000),whohasrecentlyproposedviewcompatibilityconstraintforÞvepointsintwoviewsbasedonascaledorthographiccameramodel.7.1.AGeometricConstraintEachmatchbetweentwoafÞneinvariantregionsde-ÞnesanafÞnetransformation,matchingtheregioninoneimageonthecorrespondingregioninthesecondimage.SuchanafÞnetransformationisinactanapproximationofthehomographylinkingtheprojectionsofallpointslyinginthesameplane. MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions73 igure10iewpointinvarianceoftheregionextractionandmatching:numberofcorrect,symmetricandfalsematchesfoundasafunctionoftherotationanglewithrespecttothe0degreesreferenceview.Sinclairetal.(1995)proposedamethodtotestwhethertworigidplanemotionsarecompatiblebasedontheirhomographiesCombiningthemasyieldsaplanarhomology,whoseeigenanaly-sisrevealsoneÞxedpoint(theepipole)andonelineofÞxedpoints(thecommonlineofthetwoplanes).Theyprojectthiscommonlinetotheotherimageusingandonceagainusingthetwoplanesareindeedinrigidmotion,thetworesultinglinesinthesecondimageshouldcoincide,whichcaneasilybechecked.Thegeometricconstraintweusehereisasimpleal-gebraicdistance.Asitonlyrequirestheevaluationofthedeterminantofa3matrix,itcanbeappliedquiteast.Thismakesitwellsuitedforapplicationslikeours,wheremanyconsistencychecksareperformedondif-ferentcombinationsofplanes(i.e.matches).Tocheckwhethertwocorrespondencesfoundaregeometricallyconsistentwithoneanother,itsufÞcestocheckwhetherpredeÞnedthreshold,threshold,aij]andB=[bij]theafÞnetransformationsmappingtheregionintheÞrstimagetotheregioninthesecondimage,fortheÞrstandsecondmatchrespectively.Forthederivationofthissemi-localconstraint,werefertoAppendixB. uytelaarsandVanGool igure11Scaleinvarianceoftheregionextractionandmatching:numberofcorrect,symmetricandfalsematchesfoundasafunctionofthescalefactorwithrespecttothereferenceimage.7.2.APhotometricConstraintApartfromgeometricconstraints,photometriccon-straintscanbederivedaswell.Althoughitisnotnec-essarilytruethattheilluminationconditionsarecon-stantovertheentireimage(duetoshadows,multiplelightsources,etc.),itisreasonabletoassumethatatleastsomepartsoftheimageshavesimilarillumina-tionconditions.First,wecomputeforeachregioncorrespondencetheoffsetsandscalefactorsofthephotometrictrans-formationusingmoments.Then,givenapairofregioncorespondences,wecheckfortheirphoto-metricconsistencybycomparingtheirphotometrictransformations.Fortworegioncorrespondencestobeconsistent,onlyanoverallscalefactorisallowed,tocompensateforthedifferentorientationsoftheregions. MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions75 igure12Illuminationinvarianceoftheregionextractionandmatching:numberofcorrectandsymmetricmatchesfoundbetweentheimagesshownalongthehorizontalaxisandthereferenceimageshownontheright.7.3.RejectingFalseMatchesSupposewehavecorrespondences,eachlinkingadifferentlocalregioninimagetoasimilarregioninimagedifferenttransformations.Foreachcombinationoftwosuchcorrespondences,theaboveconsistencyconstraintscanbechecked.AspeciÞcre-gioncorrespondenceisconsideredincorrectifitiscon-sistentwithlessthanothercorrespondences(withtypically8forthegeometricconstraintand4forthephotometricconstraint).Henceeachgoodcorrespon-denceshouldhaveatleastotherconsistentcorre-spondences.Thisproceduremayhavetoberepeatednumberoftimes,sincerejectingacorrespondencemaycauseothercorrespondencestohavetheirnum-berofconsistentcorrespondencesdecreasedbelowthethresholdaswell.Afterhavingrejectedmostfalsematchesamongtheregioncorrespondencesusingthegeometricandphoto-metricconstraintsdescribedabove,weapply(FischlerandBolles,1981)(arobustmethodbasedonrandomsampling)toÞndaconsistentepipolargeom-etryandtorejecttheremainingfalsecorrespondences.Sincethenumberoffalsematcheshasalreadyseri-ouslybeenreduced,thisprocessusuallystopsafteralimitednumberofsamples.Onemustnotethoughthatthecomputationofepipolargeometryisverysensitivetosmallmisallignmentsinthedata.Theregionmatcheswehavefoundsofargiveinmostcasesonlyonesta-blepointcorrespondence(e.g.theharriscornerpointincaseofthegeometry-basedmethod).Intheory,twomorelinearlyindependentpointcorrespondencescanbeextractedfromtheinvariantregion.However,theseadditionalpointcorrespondencesareinsufÞcientlysta-blefortheepipolargeometrycomputation,mainlyduetodeviationsfromourmodel,suchastheobjectsur-acenotbeingperfectlyplanar.Thisproblemcanbeercomebymappingoneimageontotheotherus-ingtheafÞnetransformation,andlookingformoreaccuratepointcorrespondenceswithinthematchedregionsusingsmallbaselinematchingtechniques.isthenappliedtotheresultingsetofpoint8.ExperimentalResults8.1.ViewpointInvariancequantitativelychecktheviewpointinvarianceofourmethod,wetookimagesofanobjectstartingfromheadonandgraduallyincreasingtheviewingangleinstepsof10degrees.AllimagesweretakenwithourSony uytelaarsandVanGool igure13Example1:Finalregioncorrespondences(top)andepipolargeometry(bottom). MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions77digitalcamera,witharesolutionof768576pixels.TheresultsofthisexperimentareshowninFig.10.eachimage,theafÞneinvariantregionswerextractedandmatchedtotheregionsfoundinthe0degreesreferenceimage.Next,theregionswereÞl- igure14Example2:Finalregioncorrespondences(top)andepipolargeometry(bottom).teredusingthesemi-localgeometricandphotometricconstraints.Finally,weappliedtheepipolartestusingtoautomaticallyselectthegoodmatches,anderiÞedthesematchesvisually,subdividingthemintothreedifferentcategories:correct,symmetricandfalse. uytelaarsandVanGool igure15Example3:Finalregioncorrespondences(top)andepipolargeometry(bottom).matches,werefertothosematchesthatdonotlinkphysicallyidenticalpoints,butpointsthatcannotbedistinguishedonalocalscaleduetoasymmetryintheimage.Forinstance,thetextonthedrinkcanusedinthisexperimentcontainstwicetheletterÔMÕ.Moreover,theselettersareexactlybelowoneanother,sotheyliemoreorlessonthesameepipolarlineduetothechosencameramovement.Asaresult,thereisnowayforthesystemtodistinguishbetweentheregionsfoundonthesetwoletters.FromFig.10,onecanseethatthesystemcandealwithchangesinviewpointupto50or60degrees.Onlycorrectandsymmetricmatcheswereleft.Forlargerangles,thegeometricconsistencytestcouldnolongerbeapplied,asthenumberofmatcheswastoolow(re-memberthatweneedatleastconsistentmatchestoclassifythemasgeometricallyÔcorrectÕ).Thehori-zontallineaddedtotheÞgureindicatestheminimumnumberofmatchesneededforthisgeometricÞlter-ingstage.Itismainlythechangeinscaleduetotheforeshorteningoftheobjectthatcausesproblems,incombinationwithmoreandmorespecularreßection.8.2.InvariancetoScaleChangesAsscalechangesseemtobetheweakestpointintheviewpointinvarianceoftheregions,weperformedsomeextraexperimentstospeciÞcallytestforthein-ariancetoscalechanges.Forthesametestobject,imageswithdifferentscalesweretakenbyzoominginandoutwithourdigitalcamera.AscanbeseenfromFig.11,thenumberofmatchesfounddecreaseswithincreasingscalechange.Nevertheless,onecanconcludethattheextractionandmatchingofafÞne MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions79 igure16Example4:Finalregioncorrespondences(top)andepipolargeometry(bottom).invariantregionsisabletowithstandscalefactorsrang-ingfrom23to32.Iflargerscalechangesaretobexpected,ascalespaceapproachshouldbeadopted.8.3.IlluminationInvarianceSincechangesintheilluminationarehardertoquantifythanchangesinscaleorviewpoint,wedecidedtousetheimagesprovidedbyFuntetal.(1998)totesttheilluminationinvarianceofoursystem,astheyprovideerydetailedinformationonthedifferentilluminantsused.Usingtheseimages,whicharereadilyavailableallowsforeasycomparisonofourre-sultswithothersystems.Figure12showstheresult.Eachoftheimagesshownbelowthehorizontalaxiscomparedwiththereferenceimagetakenunderhalogenilluminationshowntotheright.TheleftpartofeachimageshowsthewhitetoblackrowoftheMac-bethColorChecker,highlightingthelargedifferenceinillumination.MostoftheÔsymmetricÕmatchesfoundwereactuallymatchesbetweenthesereferencesquares.allimages,plentyofcorrespondenceswerefound,clearlyshowingtherobustnessofourregionextractionandmatchingtochangingilluminationconditions.8.4.WideBaselineStereoExamplesFigures13Ð17showsomeviewsofscenestakenfromsubstantiallydifferentviewpoints.Notethelargechangesinscaleinsomepartsoftheimages(e.g.Ex-ample3),theseriousocclusions(e.g.Example4)andtheextremeforeshortening(e.g.Example5).Never-theless,inallcasessufÞcientmatcheswerefoundfor uytelaarsandVanGool igure17Example5:Finalregioncorrespondences(top)andepipolargeometry(bottom). igure18ativeexamples:Imagepairsoursystemwasnotabletomatch. MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions81anaccuratedeterminationoftheepipolargeometry.Sometimesthenumberofmatchedregionsisprettylow(e.g.Example4).However,onemustnotforgetthatasingleregioncorrespondenceyieldsthreepointcorrespondences.Eachtime,theupperpartoftheÞg-ureshowstheregionsthatcontributedtotheepipolargeometry,i.e.thosethatwerematchedandsurvivedboththegeometricandphotometricÞlteringaswellasSomecorrespondingepipolarlinesareshowninthelowerhalfoftheÞgures.Finally,Fig.18showssomeexamplesofscenesoursystemwasabletoprocess.Althoughthesescenesdonotseemextraordinarilycomplexordif-Þcult,thesystemfailed,mainlyduetothedifferentbackgrounds(car-example),thelackoftextureontheobjects(bothexamples),alargeamountofspecularreßection(car-example)andnon-planarity(simpsons-xample).Theseimagesclearlyshowsomepossiblefutureresearchdirections.9.Conclusionapproachtothewidebaselinestereocorrespon-denceproblemhasbeenproposed,thatextendstheideasofSchmidandMohronlocalinvariantfeaturestowardsmoreinvarianceandhencewiderbaselines.Ineachimage,localimagepatchesareextractedinanafÞneinvariantway,suchthattheycoverthesamephysicalpartofthescene(undertheassumptionoflo-calplanarity).ThesepatchesorÔinvariantregionsÕarematchedbasedonfeaturevectorsofmomentinvariantsthatcombineinvarianceundergeometricandphoto-metricchanges.Theconsistencyofthematchesfoundistestedusingsemi-localconstraints,followedbyatestontheepipolargeometryusing.Asshownintheexperimentalresults,thefeasibilityofafÞneinvari-anceevenonalocalscalehasbeendemonstrated.Robustmatchingisquiteagenericprobleminvi-sionandseveralotherapplicationscanbeconsid-ered.Objectrecognitionisone,whereimagesofanobjectcanbematchedagainstasmallsetofrefer-enceimagesofthesameobject.Thesamplesetcanbekeptsmallbecauseoftheinvariance.Moreover,asthefeaturesarelocal,recognitionagainstvariablebackgroundsandunderocclusionissupportedbythismethod.Anotherapplicationisgrouping,wheresym-metriescanbefoundasrepeatedstructures.ImagedatabaseretrievalcanalsobeneÞtfromtheseregions,whereotherpicturesofthesamesceneorobjectcanbefound.Here,theviewpointandilluminationinvari-ancegivesthesystemthecapacitytogeneralizetoagreatextentfromasinglequeryimage.Finally,be-ingabletomatchacurrentviewagainstlearnedviewscanallowrobotstoroamextendedspaces,withouttheneedfora3Dmodel.Initialresultsforsuchap-plicationscanbefoundinTuytelaarsandVanGool(1999),Tuytelaarsetal.(1999)andTurinaetal.ppendixA:AfÞneInvarianceoftheFunctionExtremaSupposewehavethefollowinggeometricandphoto-metricdeformationsbetweentwoviews:with(and(thethreedifferentcolor-bandsand(and(coordinatesofcorrespond-ingpoints.Inthesequel,weusetorefertoeitherofthethreecolor-bandsnowprovethattheextremaofthefunctionsgiveninSection4.1areinvarianttotheabovedefor-mations.Inotherwords,foreachregioninimageforwhichreachesanextremum,theremustbeacorrespondingregioninimageforwhichreachesanextremumaswell,with)orAfÞneInvariantExtremaoffAsmentionedalreadyearlier,theÞrstfunctionrepre-sentstheaverageintensityovertheregion.Theextremaofthisfunctionbeinginvarianttotheconsideredde-formations,caneasilybeunderstoodintuitively.Here,wegiveamoreformalproof.dxdy dxdy 

uytelaarsandVanGooldxdy dxdydxdy dxdyInpractice,isalwayspositive,suchthatHence,extremaofthefunctionarepreservedundertheconsidereddeformations.Evenifhavebeennegative,extremawouldstillbepreserved,althoughmaximawouldbeturnedintominimaandviceversa.EffectsoftheDeformationsontheCenterofGravitytheotherfunctionsmentionedinSection4.1,itisimportanttoÞrstfullyunderstandtheeffectofthedeformationsonthecenterofgravityxdxdy dxdyydxdy dxdyFirst,letusconsideronlygeometricdeformations.Inthatcase,wegetfor dxdy dxdyxdxdy dxdyydxdy dxdy dxdy dxdyxdxdy dxdyydxdy dxdyHence,thecenterofgravitybehavesasanormalpointundertheafÞnedeformations.Now,letusconsidertheeffectofphotometricde-formations.Here,weinvestigatethecoordinatesofthecenterofgravityrelativetothecoordinatesoftheregioncenter M000,M001 Itcanbeshownthattheeffectofthephotometricde-formationsonisashifttowards 

I
(x
,y
)Š
x
dxdy
(I(x,y)+t dxdy M100+t sM000y
gŠy
c=

I
(x
,y
)y


I
(x
,y
)Š
y
dxdy
(I(x,y)+t dxdy M100+t AfÞneInvariantExtremaoffandfarebothcomposedoftwofactors,aratiooftwoareas,oneofwhichdependsonthecenterofgravity,andanexpressionofmomentsuptothesecondorder. |pŠp1pŠp2|×M100  M200M000Š M1002f3(
)=abs |pŠpgqŠpg| |pŠp1pŠp2|×M100  TheÞrstfactorisaratiooftwoareas,deÞnedbytheÞxedtotheregionandthecenterof MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions83gravityseenintheprevioussection,thecenterofgravitybehavesasanormal,physicalpointundertheafÞnegeometricdeformations,suchthatthisÞrstfactorclearlyisgeometricallyinvariant.Alsothesecondfactorcaneasilybecheckedtobeinvarianttothegeometricaldeformations.Next,weshowthattheeffectofthephotometricde-formationsonthisÞrstfactorissimilartotheireffectonthecoordinatesofthecenterofgravityrelativetotheregioncenter,namelyarescalingwiththesamescale-actor.Thiscanbeunderstoodbythefactthattheregionliesonthediagonalsoftheparallelogram-shapedregion,i.e.onthelineconnectingonehandandthelineconnectingontheotherhand,whichalsoformonesideoftheareasinthenu-merator(seeFig.3).Hencetheshiftinthepositionofthecenterofgravitycausesaproportionalrescalingoftheareainthenumerator: |p
Šp
1p
Šp
2|=|p1Špgp2Špg| |pŠp1pŠp2|M100 M100+t sM000|p
Šp
gq
Šp
g| |p
Šp
1p
Šp
2|=|pŠpgqŠpg| |pŠp1pŠp2|M100 M100+t Thisextrascale-factormustbecompensatedforbythesecondcomponentintheexpressionsof).Andindeed,thesecondcomponentseemstohaveexactlytheinversescale-factor: M
2
0 M
12=

I
(x
,y
)  ···= M20M111t sM0 ppendixB:DerivationofaGeometricSemi-LocalConstraintConsidertwoimagesPointsinimagearede-notedwithhomogeneouscoordinateswhilepointsinimagearedenotedwithhomoge-neouscoordinatesorthecoordi-natesofrealworld(3D)points,capitallettersareused,suchas).AhomographybelongingtodeÞnesthefollowingrelationbetweentheprojectionsinimagesof3Dpointslyingontheanarbitrarypointinimagecorrespondingtothe3DpointtwohomographiescorrespondingtotwodifferentplanesThen,bothlieontheepipolarlinecorrespondingtothepointinthesecondimage.Hence,thefollowingformulafortheepipolarlinecorrespondingtothepointcanbederiveddenotesthevectorproduct.AllepipolarlinespassthroughthesamepointFromthisproperty,wecanderiveaconstraintondenotesthe-thcolumnofmatrixthiscanbeworkedoutasfollows:):[(Thisisasecond-orderequationinwithco-efÞcientsfunctionsofDxyExzFyzSincethisequationhastobefulÞlledforallpossibleallthecoefÞcientsintheequationhavetobezero. uytelaarsandVanGoolInorderforalltheaboveequationstohaveasolutionthefollowingmatrix,whichisafunctionmustberank-deÞcient.AppliedtoLocalRegionslocalregions,theperspectivedeformationistoosmalltobedetected.Asaresult,onlyanafÞnetransformationcanbederived.Inthiscase,thehomographiescanbeapproximatedbyafÞnetransformationsofthefollowingTherank-2constraintderivedintheprevioussectionthenbecomes:Rows(1),(2)and(4)forcetheepipoletolieatin-Þnity.Thiscorrespondstoanorthographicprojectionmodel,whichindeedleadstoafÞnetransformationsbetweentwoviewsofaplanarobject.ButalsowithoutforcingtheepipoletoinÞnitythereisoneconstraintTheactualconsistencyconstraintusedinourxperimentsisthenpredeÞnedthreshold.AcknowledgmentsaregratefultoRobotVisINRIASophia-AntipolisforprovidingtheValbonneimages(Fig.13)andforÞnancialsupportfromtheECprojectVIBESandtheIUAPprojectÔAdvancedMechatronicalSystemsÕ.inneTuytelaarsisapostdoctoralresearcherfundedbytheFundforScientiÞcResearchFlanders(Belgium).1.Alternatively,onecouldleaveoutthissecondfactor,andcom-pensatefortheoffsetsbyanappropriatenormalizationoftheintensitiesbeforecomputingthemoments.2.Formoreinformationabouttheseimages,see ReferencesBallester,C.andGonzalez,M.1998.AfÞneinvarianttextureseg-mentationandshapefromtexturebyvariationalmethods.ofMathematicalImagingandVisionBaumberg,A.2000.Reliablefeaturematchingacrosswidelysepa-ratedviews.InProc.IEEEConf.onComputerVisionandPatternRecognitionpp.774Ð781.Canny,J.F.1986.Acomputationalapproachtoedgedetection.ans.onPatternAnalysisandMachineIntelligenceCarlsson,S.2000.Recognizingwalkingpeople.InProc.EuropeanConferenceonComputerVisionpp.472Ð486.Dufournaud,Y.,SchmidC.,andHoraud,R.2000.Matchingimagewithdifferentresolutions.InProc.IEEEConf.onComputerVisionandPatternRecognitionpp.612Ð618.Fischler,M.A.andBolles,R.C.1981.RandomsamplingconsensusÑAparadigmformodelÞttingwithapplicationstoim-ageanalysisandautomatedcartography.Commun.Assoc.Comp.MachFunt,B.,Barnard,K.,andMartin,L.1998.Iscolourconstancygoodenough?InProc.EuropeanConferenceonComputerVi-pp.445Ð459.Gruen,A.W.1985.Adaptiveleastsquarescorrelation:Apowerfulimagematchingtechnique.ournalofPhotogrammetry,RemoteSensingandCartography MatchingWidelySeparatedViewsBasedonAfÞneInvariantRegions85Hall,D.,ColindeVerdi`ere,V.,andCrowley,L.ObjectrecognitionusingcolouredreceptiveÞelds.InProc.EuropeanConferenceonComputerVisionHarris,C.J.andStephens,M.1983.Acombinedcornerandedgedetector.InProc.AlveyVisionConf.pp.147Ð151.oenderink,J.J.andVanDoorn,A.1987.Representationoflocalgeometryinthevisualsystem.BiologicalCyberneticsLindeberg,T.1998.Featuredetectionwithautomaticscaleselection.Int.JournalofComputerVisionLindeberg,T.andGúarding,J.1997.Shape-adaptedsmoothinginestimationof3DdepthcuesfromafÞnedistortionsoflocal,2Dbrightnessstructure.ImageandVisionComputingLowe,D.1999.Objectrecognitionfromlocalscale-invariantfea-tures.InProc.Int.Conf.onComputerVisionpp.1150Ð1157.Mindru,F.,Moons,T.,andVanGool,L.1999.Recognizingcolorpatternsirrespectiveofviewpointandillumination.InProc.IEEEConf.onComputerVisionandPatternRecognitionpp.368Ð373.Montesinos,P.,Gouet,V.,andPele,D.2000.Matchingcoloruncali-bratedimagesusingdifferentialinvariants.ImageandVisionCom-SpecialIssueBMVC2000,ElsevierScience,18(9):659ÐPritchett,P.andZisserman,A.1998.Widebaselinestereo.InProc.Int.Conf.onComputerVisionpp.754Ð759.Schaffalitzky,F.andZisserman,A.2001.Viewpointinvarianttexturematchingandwidebaselinestereo.InProc.Int.Conf.onComputerpp.636Ð643.Schmid,C.,Mohr,R.,andBauckhage,C.1997.Localgrey-valueinvariantsforimageretrieval.Int.JournalonPatternAnalysisandMachineIntelligenceSchmid,C.andMohr,R.1998.Comparingandevaluatinginterestspoints.InProc.Int.Conf.onComputerVisionpp.230Ð235.Shi,J.andTomasi,C.1994.Goodfeaturestotrack.InProc.Int.Conf.onComputerVisionandPatternRecognitionpp.593ÐSinclair,D.,Christensen,H.,andRothwell,C.1995.Usingtherela-tionbetweenaplaneprojectivityandthefundamentalmatrix.InProc.ScandinavianConf.onImageAnalysispp.181Ð188.Super,B.J.andKlarquist,W.N.1997.Patchmatchingandstereop-sisinageneralstereoviewinggeometry.Int.JournalonPatternAnalysisandMachineIntelligenceell,D.andCarlsson,S.2000.WidebaselinepointmatchingusingafÞneinvariantscomputedfromintensityproÞles.InProc.Euro-peanConf.onComputerVisionpp.814Ð828.uytelaars,T.andVanGool,L.1999.Content-basedimageretrievalbasedonlocalafÞnelyinvariantregions.InProc.Int.Conf.onisualInformationSystemspp.493Ð500.uytelaars,T.,VanGool,L.,DÕhaene,L.,andKoch,R.1999.Match-ingafÞnelyinvariantregionsforvisualservoing.InProc.Int.Conf.onRoboticsandAutomationpp.1601Ð1606.uytelaars,T.,Turina,A.,andVanGool,L.2003.Non-combinatorialdetectionofregularrepetitionsunderperspectiveskew.ans.onPatternAnalysisandMachineIntelligence