FastNearestNeighborRetrievalforBregmanDivergences - PDF document

FastNearestNeighborRetrievalforBregmanDivergences Figure1.Thebregmandivergencebetweenxandy.gencebasedonfisdf(x;y)f(x)�f(y)�hrf(y);x�yi:Onecaninterpretthebregmandivergenceasthedis-tancebetweenafunctionanditsrst-ordertaylorex-pansion.Inparticular,df(x;y)isthedierencebe-tweenf(x)andthelinearapproximationoff(x)cen-teredaty;seegure1.Sincefisconvex,df(x;y)isalwaysnonnegative.Somestandardbregmandivergencesandtheirbasefunctionsarelistedintable1.Abregmandivergenceistypicallyusedtoassesssim-ilaritybetweentwoobjects,muchlikeametric.Butthoughmetricsandbregmandivergencesarebothusedforsimilarityassessment,theydonotsharethesamefundamentalproperties.Metricssatisfythreebasicproperties:non-negativity:d(x;y)0;symmetry:d(x;y)=d(y;x);and,perhapsmostimportantly,thetriangleinequality:d(x;z)d(x;y)+d(y;z).Breg-mandivergencesarenonnegative,howevertheydonotsatisfythetriangleinequality(ingeneral)andcanbeasymmetric.Bregmandivergencesdosatisfyavarietyofgeometricproperties,acoupleofwhichwewillneedlater.Thebregmandivergencedf(x;y)isconvexinx,butnotnecessarilyiny.DenethebregmanballofradiusRaroundasB(;R)fx:df(x;)Rg:Sincedf(x;)isconvexinx,B(;R)isaconvexset.Anotherinterestingpropertyconcernsmeans.Forasetofpoints,themeanunderabregmandivergenceiswelldenedand,interestingly,isindependentofthechoiceofdivergence:XargminXx2Xdf(x;)=1 jXjXx2Xx:Thisfactcanbeusedtoextendk-meanstothefamilyofbregmandivergences(Banerjeeetal.,2005). f.Inparticular,fisassumedtobeLegendre.Table1.Somestandardbregmandivergences.f(x)df(x;y) `221 2kxk221 2kx�yk22 KLPxilogxiPxilogxi yi Mahalanobis1 2x�Qx1 2(x�y)�Q(x�y) Itakura-Saito�PlogxiPxi yi�logxi yi�1 2.2.NNSearchBecauseofthetremendouspracticalandtheoreti-calimportanceofnearestneighborsearchinmachinelearning,computationalgeometry,databases,andelse-where,manyretrievalschemeshavebeendevelopedtoreducethecomputationalcostofndingNNs.KD-trees(Friedmanetal.,1977)areoneoftheearli-estandmostpopulardatastructuresforNNretrieval.Thedatastructureandaccompanyingsearchalgo-rithmprovideablueprintforahugebodyoffuturework(includingthepresentone).Thetreedenesahierarchicalspacepartitionwhereeachnodedenesanaxis-alignedrectangle.Thesearchalgorithmisasim-plebranchandboundexplorationofthetree.ThoughKD-treesareusefulinmanyapplications,theirper-formancehasbeenwidelyobservedtodegradebadlywiththedimensionalityofthedatabase.Metricballtrees(Omohundro,1989;Uhlmann,1991;Yianilos,1993;Moore,2000)extendthebasicmethod-ologybehindKD-treestometricspacesbyusingmet-ricballsinplaceofrectangles.Thesearchalgorithmusesthetriangleinequalitytopruneoutnodes.TheyseemtoscalewithdimensionalitybetterthanKD-trees(Moore,2000),thoughhigh-dimensionaldataremainsverychallenging.Somehigh-dimensionaldatasetsareintrinsicallylow-dimensional;variousretrievalschemeshavebeendevelopedthatscalewithanotionofintrin-sicdimensionality(Beygelzimeretal.,2006).Inmanyapplications,anexactNNisnotrequired;somethingnearbyisgoodenough.Thisisespeciallytrueinmachinelearningapplications,wherethereistypicallyalotofnoiseanduncertainty.Thusmanyresearchershaveswitchedtotheproblemofapproxi-mateNNsearch.Thisrelaxationledtosomesigni-cantbreakthroughs,perhapsthemostimportantbe-inglocalitysensitivehashing(Dataretal.,2004).Spilltrees(Liuetal.,2004)areanotherdatastructureforapproximateNNsearchandhaveexhibitedverystrongperformanceempirically. FastNearestNeighborRetrievalforBregmanDivergences ThepresentpaperappearstobethersttodescribeageneralmethodforecientlyndingbregmanNNs;however,somerelatedproblemshavebeenexamined.(Nielsenetal.,2007)exploresthegeometricpropertiesofbregmanvoronoidiagrams.VoronoidiagramsareofcoursecloselyrelatedtoNNsearch,butdonotleadtoanecientNNdatastructurebeyonddimension2.(Guhaetal.,2007)containsresultsonsketchingbreg-man(andother)divergences.Sketchingisrelatedtodimensionalityreduction,whichisthebasisformanyNNschemes.WeareawareofonlyoneNNspeedupschemeforKL-divergences(Spellman&Vemuri,2005).Theresultsinthispaperarequitelimited:experimentswerecon-ductedononlyonedatasetandthespeedupislessthan3x.Moreover,thereappearstobeasignicanttechnical awinthederivationoftheirdatastructure.Inparticular,theycitethepythagoreantheoremasanequalityforprojectionontoanarbitraryconvexset,whereasitisactuallyaninequality.3.BregmanBallTreesThissectiondescribesthebregmanballtreedatastructure.ThedatastructureandsearchalgorithmsfollowthesamebasicprogramusedinKD-treesandmetrictrees;inplaceofrectangularcellsormetricballs,thefundamentalgeometricobjectisabregmanball.Abbtreedenesahierarchicalspacepartitionbasedonbregmanballs.ThedatastructureisabinarytreewhereeachnodeiisassociatedwithasubsetofthedatabaseXiX.Nodeiadditionallydenesabreg-manballB(i;Ri)withcenteriandradiusRisuchthatXiB(i;Ri).Interior(non-leaf)nodesoftreehavetwochildnodeslandr.Thedatabasepointsbelongingtonodeiaresplitbetweenchildlandr;eachpointinXiappearsinexactlyoneofXlorXr.2ThoughXlandXraredisjoint,theballsB(l;Rl)andB(r;Rr)mayoverlap.Therootnodeofthetreeencapsulatestheentiredatabase.Eachleafcoversasmallfractionofthedatabase;thesetofallleavescovertheentirety.3.1.SearchingThissubsectiondescribeshowtoretrieveaquery'snearestneighborwithabbtree.Throughout,X=fx1;:::;xngisthedatabase,qisaquery,anddf(
;
)isa(xed)bregmandivergence.Thepointweare 2Thedisjointednessofthetwopointsetsisnotessential.searchingforistheleftNNxqargminx2Xdf(x;q):FindingtherightNN(argminx2Xdf(q;x))isconsid-eredinsection5.Branchandboundsearchlocatesxqinthebbtree.First,thetreeisdescended;ateachnode,thesearchal-gorithmchoosesthechildforwhichdf(;q)issmallestandignoresthesiblingnode(temporarily).Uponar-rivingataleafnodei,thealgorithmcalculatesdf(x;q)forallx2Xi.TheclosestpointisthecandidateNN;callitxc.Nowthealgorithmmusttraversebackupthetreeandconsiderthepreviouslyignoredsiblings.Anignoredsiblingjmustbeexploredifdf(xc;q)�minx2B(j;Rj)d(x;q):(1)Thealgorithmcomputestherightsideof(1);wecomebackthatinamoment.If(1)holds,thennodejandallofitschildrencanbeignoredsincetheNNcan-notbefoundinthatsubtree.Otherwise,thesubtreerootedatjmustbeexplored.Thisalgorithmiseasilyadjustedtoreturnthek-nearestneighbors.Thealgorithmhingesonthecomputationof(1)|thebregmanprojectionontoabregmanball.Inthe`22(orarbitrarymetric)case,theprojectioncanbecomputedanalyticallywiththetriangleinequality.Sincegeneralbregmandivergencesdonotsatisfythisinequality,weneedadierentwaytocompute|oratleastbound|therightsideof(1).Computingthisprojectionisthemaintechnicalcontributionofthispaper,sowediscussitseparatelyinsection4.3.2.ApproximateSearchAswementionedinsection2.2,manypracticalappli-cationsdonotrequireanexactNN.Thisisespeciallytrueinmachinelearningapplications,wherethereistypicallyalotofnoiseandeventherepresentationofpointsusedisheuristic(e.g.selectinganappropriatekernelforanSVMofteninvolvesguesswork).This exibilityisfortunate,sinceexactNNretrievalmeth-odsrarelyworkwellonhigh-dimensionaldata.Following(Liuetal.,2004),asimplewaytospeeduptheretrievaltimeofthebbtreeistosimplystopaf-teronlyafewleaveshavebeenexamined.ThisideaoriginatesfromtheempiricalobservationthatmetricandKD-treesoftenlocateapointveryclosetotheNNquickly,thenspendmostoftheexecutiontimeback-tracking.WeshowempiricallythatthequalityoftheNNdegradesgracefullyasthenumberofleavesex-amineddecreases.Evenwhenthesearchprocedureisstoppedveryearly,itreturnsasolutionthatisamongthenearestneighbors. FastNearestNeighborRetrievalforBregmanDivergences Sincefisstrictlyconvex,thegradientmappingisone-to-one.Moreover,theinversemappingisgivenbythegradientoftheconvexconjugate,denedasf(y)supxfhx;yi�f(x)g:(4)Symbolically: Thustosolve(P),wecanlookfortheoptimalx0along0+(1�)q0,andthenapplyrftorecoverxp.3Tokeepnotationsimple,wedenex00+(1�)q0and(5)xrf(x0):(6)Nowontothesecondproperty.Claim3.df(xp;)=R|i.e.theprojectionliesontheboundaryofB(;R).Theclaimfollowsfromcomplementaryslacknessap-pliedto(3).Claims2and3implythatndingtheprojectionofqontoB(;R)isequivalenttondsubjectto:df(x;)=R2(0;1]x=rf(0+(1�)q0):Fortunately,solvingthisprogramissimple.Claim4.df(x;)ismonotonicin.Thisclaimfollowsfromtheconvexityoff.Sincedf(x;)ismonotonic,wecanecientlysearchforpsatisfyingdf(xp;)=Rusingbisectionsearchon.Wesummarizetheresultinthefollowingtheorem.Theorem5.Supposekr2fk2isboundedaroundx0p.Thenapointxsatisfyingjdf(x;q)�df(xp;q)j+O(2)canbefoundinO(log1=)iterations.Eachiterationrequiresonedivergenceevaluationandonegradientevaluation.4.1.StoppingEarlyRecallthatthepointofallthisanalysisistoevaluatewhetherdf(xc;q)�minx2B(;R)df(x;q);(7) 3Allofthebasefunctionsintable1haveclosedformconjugates.wherexcisthecurrentcandidateNN.If(7)holds,thenodeinquestionmustbesearched;otherwiseitcanbepruned.Wecanevaluatetherightsideof(7)ex-actlyusingthebisectionmethoddescribedpreviously,butanexactsolutionisnotneeded.SupposewehaveboundsaandAsatisfyingAminx2B(;R)df(x;q)a:Ifdf(xc;q)a,thenodecanbepruned;ifdf(xc;q)&#x-278;A,thenodemustbeexplored.Wenowdescribeupperandlowerboundsthatarecomputedateachstepofthebisectionsearch;thesearchproceedsuntiloneofthetwostoppingconditionsismet.Alowerboundisgivenbyweakduality.ThelagrangedualfunctionisL()df(x;q)+ 1�df(x;)�R:(8)Byweakduality,forany2[0;1),L()minx2B(;R)df(x;q):(9)Fortheupperbound,weusetheprimal.Atanysatisfyingdf(x;)R,wehavedf(x;q)minx2B(;R)df(x;q):(10)Letusnowputallofthepiecestogether.Wewishtoevaluatewhether(7)holds.Thealgorithmperformsbisectionsearchon,attemptingtolocatethesatis-fyingdf(x;)=R.Atstepithealgorithmevaluatesiontwofunctions.First,itchecksthelowerboundboundgivenbythedualfunctionL(i)denedin(8).IfL(i)�df(xc;q),thenthenodecanbepruned.Otherwise,ifxi2B(;R),wecanupdatetheupperbound.Ifdf(xi;q)df(xc;q),thenthenodemustbesearched.Otherwise,neitherboundholds,sothebisectionsearchcontinues.SeeAlgorithm1forpseu-docode.5.LeftandRightNNSinceabregmandivergencecanbeasymmetric,itde-nestwoNNproblems:(lNN)returnargminx2Xdf(x;q)and(rNN)returnargminx2Xdf(q;x).ThebbtreedatastructurendstheleftNN.WeshowthatitcanalsobeusedtondtherightNN. FastNearestNeighborRetrievalforBregmanDivergences Algorithm1CanPrune Input:l;r2(0;1],q;xc;2RD,R2R.Set=l+r 2.Setx=rf(0+(1�)q0)ifL()�df(xc;q)thenreturnYeselseifx2B(;R)anddf(x;q)df(xc;q)thenreturnNoelseifdf(x;)&#x-278;RthenreturnCanPrune(l;;q;xc;)elseifdf(x;)RthenreturnCanPrune(;r;q;xc;)endif Recallthattheconvexconjugateoffisdenedasf(y)supxfhx;yi�f(x)g.Thesupremumisrealizedatapointxsatisfyingrf(x)=y;thusf(y0)=hy;y0i�f(y):Weusethisidentitytorewritedf(
;
):df(x;y)=f(x)�f(y)�hy0;x�yi=f(x)+f(y0)�hy0;xi=df(y0;x0):ThisrelationshipprovidesasimpleprescriptionforadaptingthebbtreetotherNNproblem:buildabb-treeforthedivergencedfandthedatabaseX0frf(x1);:::;rf(xn)g.Onqueryq,q0rf(q)iscomputedandthebbtreendsx02X0minimizingdf(x0;q0).Thepointxwhosegradientisx0isthentherNNtoq.6.ExperimentsWeexaminetheperformancebenetofusingbbtreesforapproximateandexactNNsearch.AllexperimentswereconductedwithasimpleCimplementationthatisavailablefromtheauthor'swebsite.TheresultsarefortheKL-divergence.WechosetoevaluatethebbtreefortheKL-divergencebecauseitisusedwidelyinmachinelearning,textmining,andcomputervision;moreover,verylittleisknownaboutecientNNretrievalforit.Incontrast,therehasbeenatremendousamountofworkforspeedingupthe`22andMahalanobisdivergences|theybothmaybehandledbystandardmetrictreesandmanyothermethods.Otherbregmandivergencesappearmuchlessofteninapplications.Still,examiningtheprac-ticalperformanceofbbtreesfortheseotherbregmandivergencesisaninterestingdirectionforfuturework.Weranexperimentsonseveralchallengingdatasets.rcv-D.Weusedlatentdirichletallocation(LDA)(Bleietal.,2003)togeneratetopichistogramsfor500kdocumentsinthercv1corpus(Lewisetal.,2004).Thesehistogramsweregeneratedbybuild-ingaLDAmodelonatrainingsetandthenper-forminginferenceon500kdocumentstogener-atetheirposteriordirichletparameters.Suitablyscaled,theseparametersgivearepresentationofthedocumentsinthetopicsimplex(Bleietal.,2003).WegenerateddatausingthisprocessforD=8;16;:::;256topics.Corelhistograms.Thisdatasetcontains60kcolorhistogramsgeneratedfromtheCorelimagedataset.Eachhistogramis64-dimensional.Semanticspace.Thisdatasetisa371-dimensionalrepresentationof5000imagesfromtheCorelStockphotocollection.Eachimageisrepresentedasadistributionover371descriptionkeywords(Rasiwasiaetal.,2007).SIFTsignatures.Thisdatasetcontains1111-dimensionalrepresentationsof10kimagesfromthePASCAL2007dataset(Everinghametal.,2007).EachpointisahistogramofquantizedSIFTfeaturesassuggestedin(Nowaketal.,2006).Noticethatmostofthesedatasetsarefairlyhigh-dimensional.WearemostlyinterestedinapproximateNNretrieval,sincethatislikelysucientformachinelearningappli-cations.Ifthebbtreeisstoppedearly,itisnotguar-anteedtoreturnanexactNN,soweneedawaytoevaluatethequalityofthepointitreturns.Onenat-uralevaluationmetricisthis:Howmanypointsfromthedatabaseareclosertothequerythanthereturnedpoint?CallthisvalueNCfor\numbercloser".IfNCissmallcomparedtothesizeofthedatabase,say10versus100k,thenitwilllikelysharemanypropertieswiththetrueNN(e.g.classlabel).4Theresultsareshowningure2.Thesearestrongresults;itisshownthatthebbtreeisoftenordersofmagnitudefasterthanbrute-forcesearchwithoutasubstantialdegradationofquality.Moreanalysisappearsinthecaption. 4Adierentevaluationcriteriaistheapproximationra-tiosatisfyingdf(x;q)(1+)df(xq;q),wherexqisq'strueNN.Wedidnotusethismeasurebecauseitisdif-culttointerpret.Forexample,supposewend=:3approximateNNsfromtwodierentdatabasesAandB.ItcouldeasilybethecasethatallpointsinAare1:3-approximateNNs,whereasonlytheexactNNindatabaseBis1:3-approximate. FastNearestNeighborRetrievalforBregmanDivergences Table2.Exactsearch dataset dimensionality speedup rcv-8 8 64.5 rcv-16 16 36.7 rcv-32 32 21.9 rcv-64 64 12.0 corelhistograms 64 2.4 rcv-128 128 5.3 rcv-256 256 3.3 semanticspace 371 1.0 SIFTsignatures 1111 0.9 Finally,weconsiderexactNNretrieval.Itiswellknownthatndinga(guaranteed)exactNNinmod-eratetohigh-dimensionaldatabasesisverychalleng-ing.Inparticular,metrictrees,KD-trees,andrelativestypicallyaordareasonablespeedupinmoderatedi-mensions,butthespeedupdiminisheswithincreasingdimensionality(Moore,2000;Liuetal.,2004).Whenusedforexactsearch,thebbtreere ectsthisbasicpat-tern.Table2showstheresults.Thebbtreeprovidesasubstantialspeeduponthemoderate-dimensionaldatabases(upthroughD=256),butnospeeduponthetwodatabasesofhighestdimensionality.7.ConclusionInthispaper,weintroducedbregmanballtreesanddemonstratedtheirecacyinNNsearch.Theexper-imentsdemonstratedthatbbtreescanspeedupap-proximateNNretrievalfortheKL-divergencebyor-dersofmagnitudeoverbruteforcesearch.Therearemanypossibledirectionsforfutureresearch.Onthepracticalside,whichideasbehindthemanyvariantsofmetrictreesmightbeusefulforbbtrees?Onthethe-oreticalside,whatisagoodnotionofintrinsicdimen-sionalityforbregmandivergencesandcanapracticaldatastructurebedesignedaroundit?AcknowledgementsThankstoSergeBelongie,SanjoyDasgupta,CharlesElkan,CarolinaGalleguillos,DanielHsu,NikhilRasi-wasia,andLawrenceSaul.SupportwasprovidedbytheNSFundergrantsIIS-0347646andIIS-0713540.ReferencesBanerjee,A.,Merugu,S.,Dhillon,I.S.,&Ghosh,J.(2005).Clusteringwithbregmandivergences.JMLR.Beygelzimer,A.,Kakade,S.,&Langford,J.(2006).Covertreesfornearestneighbor.ICML.Blei,D.,Ng,A.,&Jordan,M.(2003).Latentdirichletallocation.JMLR.Bregman,L.(1967).Therelaxationmethodofndingthecommonpointofconvexsetsanditsapplicationtothesolutionofproblemsinconvexprogramming.USSRComputationalMathematicsandMathematicalPhysics,7,200{217.Datar,M.,Immorlica,N.,Indyk,P.,&Mirrokni,V.S.(2004).Locality-sensitivehashingschemebasedonp-stabledistributions.SCG2004.Everingham,M.,Gool,L.V.,Williams,C.K.,Winn,J.,&Zisserman,A.(2007).ThePASCALVisualObjectClassesChallenge2007Results.Friedman,J.H.,Bentley,J.L.,&Finkel,R.A.(1977).Analgorithmforndingbestmatchesinlogarithmicex-pectedtime.ACMTransactionsonMathematicalSoft-ware,3(3),209{226.Gray,R.M.,Buzo,A.,Gray,A.H.,&Matsuyama,Y.(1980).Distortionmeasuresforspeechprocessing.IEEETransactionsonAcoustics,Speech,andSignalProcess-ing.Guha,S.,Indyk,P.,&McGregor,A.(2007).Sketchinginformationdivergences.COLT.Lewis,D.D.,Yang,Y.,Rose,T.,&Li,F.(2004).Rcv1:Anewbenchmarkcollectionfortextcategorizationre-search.JMLR.Liu,T.,Moore,A.W.,Gray,A.,&Yang,K.(2004).Aninvestigationofpracticalapproximateneighboral-gorithms.NIPS.Moore,A.W.(2000).Usingthetriangleinequalitytosur-vivehigh-dimensionaldata.UAI.Nielsen,F.,Boissonnat,J.-D.,&Nock,R.(2007).Onbregmanvoronoidiagrams.SODA(pp.746{755).Nowak,E.,Jurie,F.,&Triggs,B.(2006).Samplingstrate-giesforbag-of-featuresimageclassication.ECCV.Omohundro,S.(1989).Fiveballtreeconstructionalgo-rithms(TechnicalReport).ICSI.Pereira,F.,Tishby,N.,&Lee,L.(1993).DistributionalclusteringofEnglishwords.31stAnnualMeetingoftheACL(pp.183{190).Puzicha,J.,Buhmann,J.,Rubner,Y.,&Tomasi,C.(1999).Empiricalevaluationofdissimilaritymeasuresforcolorandtexture.ICCV.Rasiwasia,N.,Moreno,P.,&Vasconcelos,N.(2007).Bridgingthegap:querybysemanticexample.IEEETransactionsonMultimedia.Spellman,E.,&Vemuri,B.(2005).Ecientshapeindex-ingusinganinformationtheoreticrepresentation.Inter-nationalConferenceonImageandVideoRetrieval.Uhlmann,J.K.(1991).Satisfyinggeneralproxim-ity/similarityquerieswithmetrictrees.InformationProcessingLetters,40,175{179.Weinberger,K.,Blitzer,J.,&Saul,L.(2006).Distancemetriclearningforlargemarginnearestneighborclassi-cation.NIPS.Yianilos,P.N.(1993).Datastructuresandalgorithmsfornearestneighborsearchingeneralmetricspaces.SODA.