Left gradient 64257eld integration Middle membrane interpolation Right scattered data interpolation The insets show the shapes of the corresponding kernels Abstract We present a novel approach for rapid numerical approximation of convolutions with 6 ID: 23293
Download Pdf The PPT/PDF document "Convolution Pyramids Zeev Farbman The He..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
targetÞltertoapproximate,weÞrstusenumericaloptimizationtodesignasetofsmallkernels,whicharethenusedtoperformtheanalysisandsynthesisstepsofourmultiscaletransform.Oncetheoptimizationhasbeendone,theresultingtransformcanbeappliedtoanysignalinlineartime.WedemonstratethatourmethodiswellsuitedfortaskssuchasgradientÞeldintegration,seamlessim-agecloning,andscattereddatainterpolation,outperformingexist-ingstate-of-the-artmethods.Keywords:convolution,GreenÕsfunctions,Poissonequation,seamlesscloning,scattereddatainterpolation,ShepardÕsmethodLinks:DLPDFWEB1IntroductionManytasksincomputergraphicsandimageprocessinginvolveapplyinglargelineartranslation-invariant(LTI)Þlterstoimages.Commonexamplesincludelow-andhigh-passÞlteringofimages,andmeasuringtheimageresponsetovariousÞlterbanks.SomelessobvioustasksthatcanalsobeaccomplishedusinglargeLTIÞltersaredemonstratedinFigure1:reconstructingimagesbyintegratingtheirgradientÞeld[Fattaletal.2002],Þttingasmoothmembranetointerpolateasetofboundaryvalues[P«erezetal.2003;Agarwala2007],andscattereddatainterpolation[Lewisetal.2010].WhileconvolutionisthemoststraightforwardwayofapplyinganLTIÞltertoanimage,itcomeswithahighcomputationalcost:O(n2)operationsarerequiredtoconvolveann-pixelimagewithakernelofcomparablesize.TheFastFourierTransformoffersamoreefÞcient,O(nlogn)alternativeforperiodicdomains[Brigham1988].Otherfastapproacheshavebeenproposedforcertainspecialcases.Forexample,Burt[1981]describesamultiscaleapproach,whichcanapproximateaconvolutionwithalargeGaussiankernelinO(n)timeathierarchicallysubsampledlocations.Wereviewthisandseveralotherrelatedapproachesinthenextsection.Inthiswork,wegeneralizetheseideas,anddescribeanovelmul-tiscaleframeworkthatisnotlimitedtoapproximatingaspeciÞckernel,butcanbetunedtoreproducetheeffectofanumberofusefullargeLTIÞlters,whileoperatinginO(n)time.SpeciÞcally,wedemonstratetheapplicabilityofourframeworktoconvolutionswiththeGreenÕsfunctionsthatspanthesolutionsofthePoissonequation,inversedistanceweightingkernelsformembraneinter-polation,andwide-supportGaussiankernelsforscattereddatain-terpolation.TheseapplicationsaredemonstratedinFigure1.Ourmethodconsistsofamultiscalescheme,resemblingtheLapla-cianPyramid,aswellascertainwavelettransforms.However,unlikethesemoregeneralpurposetransforms,ourapproachistocustom-tailorthetransformtodirectlyapproximatetheeffectofagivenLTIoperator.Inotherwords,whilepreviousmultiscalecon-structionsaretypicallyusedtotransformtheproblemintoaspacewhereitcanbebettersolved,inourapproachthetransformitself isthelengthofthe1Dkernels.Non-separablekernelsmaybeapproximatedbyseparableonesusingtheSVDdecompositionofthekernel[Perona1995].B-splineÞltersofvariousordersandscalesmaybeevaluatedatconstanttimeperpixelusingrepeatedintegration[Heckbert1986]orgeneralizedintegralimages[Derpanisetal.2007].ManymethodshavebeendevelopedspeciÞcallytoapproximateconvolutionswithGaussiankernels,becauseoftheirimportantroleinimageprocessing[Wells,III1986].OfparticularrelevancetothisworkisthehierarchicaldiscretecorrelationschemeproposedbyBurt[1981].ThismultiscaleschemeapproximatesaGaussianpyramidinO(n)time.SinceconvolvingwithaGaussianband-limitsthesignal,interpolatingthecoarserlevelsbacktotheÞneronesprovidesanapproximationtoaconvolutionwithalargeGaus-siankernelinO(n).However,BurtÕsapproximationisaccurateonlyforGaussiansofcertainwidths.BurtÕsideasculminatedintheLaplacianPyramid[BurtandAdel-son1983],andwerelaterconnectedwithwavelets[DoandVetterli2003].MorespeciÞcally,theLaplacianpyramidmaybeviewedasahigh-densitywavelettransformansformSelesnick2006].Theseideasarealsoechoedin[Fattaletal.2007],whereamultiscalebilateralÞlterpyramidisconstructedinTheLaplacianpyramid,aswellasvariousotherwavelettrans-forms[Mallat2008]decomposeasignalintoitslowandhighfrequencies,whichwasshowntobeusefulforavarietyofanal-ysisandsynthesistasks.Inparticular,itispossibletoapproxi-matetheeffectofconvolutionswithlargekernels.However,eventhoughtheseschemesrelyonrepeatedconvolution(typicallywithsmallkernels),theyarenottranslation-invariant,i.e.,iftheinputistranslated,theresultinganalysisandsynthesiswillnotdifferonlybyatranslation.ThisisduetothesubsamplingoperationstheseschemesuseforachievingtheirfastO(n)performance.Ourschemealsousessubsampling,butinamorerestrictedfashion,whichwas time,usingFFToverperiodicdomains.However,thereareevenfasterO(n)solversforhandlingspeciÞctypesofsuchequationsoverbothperiodicandnon-periodicdomains.Themultigrid[Trottenbergetal.2001]methodandhier-archicalbasispreconditioning[Szeliski1990]achievelinearperfor-mancebyoperatingatmultiplescales.Astate-of-the-artmultigridsolverforlargegradientdomainproblemsisdescribedbyKazhdanandHoppe[2008],andaGPUimplementationforsmallerproblemsisdescribedbyMcCannandPollard[2008].Sincethematricesintheselinearsystemsarecirculant(ornearlycirculant,dependingonthedomain),theirinverseisalsoacircu-lantconvolutionmatrix.Hence,inprinciple,thesolutioncanbeobtained(orapproximated)byconvolvingtheright-handsidewithakernel,e.g.,theGreenÕsfunctionincasesoftheinÞnitePoisson Agarwala[2007]describesadedicatedquad-treebasedsolverforsuchsystems,whileFarbmanetal.[2009]avoidsolvingalinearsystemaltogetherbyusingmean-valueinterpolation.InSection5weshowhowtoconstructsuchmembranesevenmoreefÞcientlybycastingtheproblemasaratioofconvolutions[Carretal.2003].Thisapproachcanalsobeusefulinmoregeneralscattereddatain-terpolationscenarios[Lewisetal.2010],whentherearemanydatapointstobeinterpolatedorwhenadenseevaluationoftheinterpo-latedfunctionisneeded.OurworkprovidesanefÞcientapproxi-mationforscattereddatainterpolationwhereitisimportanttouselargeconvolutionÞltersthatpropagatetheinformationtotheentire quenciesaretypicallyneededatahigherspectralaccuracyand,hence,largelow-passÞltersareused.SubbandarchitecturessuchaswavelettransformsandLaplacianpyramidsrelyonaspectralÒdivide-and-conquerÓstrategywhere,ateveryscale,thespectrumispartitionedviaÞlteringandthenthelowerendofthespectrum,whichmaybesubsampledwithouta Forwardtransform(analysis)}3:a0=a4:foreachlevell=0...L!1do5:al0=al6:al+1="(h1#al)7:endfor8:{Backwardtransform(synthesis)foreachlevell=L!1...0do11:öal=h2#($öal+1)+g#al012:endforFigure2:Oursubbandarchitectureßowchartandpseudocode.andiscapableofisolatinglow-frequencymodesuptothelargestspatialscale(theDCcomponentoftheimage).Whileitmaynotbeamajorobstacleforsomeapplications,thesedecompositionssufferfromtwomainshortcomings.First,there-sultingtransformedcoordinates,andthereforetheoperationsper-formedusingthesecoordinates,arenotinvarianttotranslation.Thus,unlikeconvolution,shiftingtheinputimagemaychangetheoutcomebymorethanjustaspatialoffset.Second,toachievetheO(n)runningtime,itisnecessarytouseÞniteimpulseresponseÞlters.TheseÞlterscanachievesomespatialandspectrallocaliza-tionbutdonotprovideanidealpartitioningofthespectrum.Aswedemonstratehere,thesepropertiesarecriticalforsomeappli- wherethesuperscriptldenotesthelevelinthehierarchy,al0istheunÞltereddatakeptateachlevel,and"denotesthesubsamplingoperator.Thetransformisinitiatedbysettinga0=a,whereaistheinputsignaltobeÞltered.Thebackwardtransformation(synthesis)consistsofupsamplingbyinsertingazerobetweeneverytwosamples,followedbyaconvolu-tion,withanotherÞlter, al0inourcase).Similarlytothehighdensitywavelettransformation[Selesnick2006],thischoiceismadeinordertominimizethesubsamplingeffectandtoincreasethetranslationinvariance.AssumingidealÞlteringwascomputationallyfeasible,h1andh2couldbechosensoastoperfectlyisolateandreconstructprogres-sivelylowerfrequencybandsoftheoriginaldata,inwhichcasetheroleofgwouldbetoapproximatethedesiredÞlteringoperation.However,sincewewanttokeepthenumberofoperationsO(n),theÞltersh fromexplicitana-lyticÞlterdesignmethodologies,wenumericallyoptimizetheseÞl-terssuchthattheirjointactionwillbestachievethedesiredÞlteringoperation.Insummary,ourapproachconsistsofidentifyingandal-locatingacertainamountofcomputationswithreducedamountof whereöa0FistheresultofourmultiscaletransformwiththekernelsFonsomeinputa.Inordertocarryoutthisoptimizationitre-mainstodeterminethetypesofthekernelsandthenumberoftheirunknownparameters.Thechoiceofthetrainingdataadependsontheapplication,andwillbediscussedinthenextsection.Notethatoncethisoptimizationiscompleteandthekernelshavebeenfound,ourschemeisreadytobeusedforapproximatingf#aonanygivensignala.Allofthekernelsusedtoproduceourresultsareprovidedinthesupplementarymaterial,andhencenofurtheroptimizationisrequiredtousetheminpractice.Inordertominimizethenumberoftotalarithmeticoperations,thekernelsinFshouldbesmallandseparable.ThespeciÞcchoicesreportedbelowcorrespond,inourexperiments,toagoodtrade-offbetweenoperationcountandapproximationaccuracy.Usinglargerand/ornon-separableÞltersincreasestheaccuracy,andhencethespeciÞcchoicedependsontheapplicationrequirements.Remark-ably,weobtainratheraccurateresultsusingseparablekernelsin !n=0wherenistheunitboundarynor-malvector,areenforced.GreenÕsfunctionsG(x,x')deÞnethefundamentalsolutionstothePoissonequation,andaredeÞnedby!G(x,x')="( log1%x!x'%.(7) thatusesa5-by-5kernelforh1,h2anda3-by-3kernelforgtobeparticularlyattractive,asitproducesresultsthatarevisuallyveryclosetothegroundtruth,whileemployingcompactkernels.Amoreaccuratesolution(virtuallyindistinguishablefromthegroundtruth)maybeobtainedbyincreasingthekernelsizesto7-by-7/5-by-5(F7,5)inexchangeforamodestincreaseinrunningtime,orevenfurtherto9-by-9/5-by-5(F9,5).Notethatwehavenoevidencethatthebest onvariousimagesizes.gridsizetime(5x5/3x3)time(7x7/5x5)(millions)(sec,singlecore)(sec,singlecore)0.260.00190.002851.040.0100.0154.190.0470.06516.770.190.2667.10.991.38Table1:Performancestatisticsforconvolutionpyramids.Re-portedtimesexcludediskI/Oandweremeasuredona2.3GHzIntelCorei7(2820qm)MacBookPro.Figure4(a)showsareconstructionofthefree-spaceGreenÕsfunc-tionobtainedbyfeedingourmethodacentereddeltafunctionasinput.Acomparisontothegroundtruth(thesolutionof( ableCPU-basedmultigridsolversforsuchreconstruction.Indeed,theplotsinFigure6conÞrmitssuperiorconvergence.However,formostpracticalgraphicsapplications,areconstructionerrorintherangeof0.001Ð0.0001issufÞcient,andourmethodisabletoachievethisaccuracyconsiderablyfasterthantheKazhdan-Hoppesolver,whilebeingsimplertoimplementandtoporttotheGPU.McCannandPollard[2008]describeaGPU-basedimplementationofamultigridsolver,which,atthetime,enabledtointegrateaone-megapixelimageabout20timespersecond,supportinginteractivegradientdomainpainting.OurcurrentsinglecoreCPU-basedim-plementationalreadyenablestointegratesuchanimage33times resolutiongrid. SuchmembraneshaveoriginallybeenbuiltbysolvingtheLaplaceequation[P«erezetal.2003].However,Farbmanetal.[2009]showedthatothersmoothinterpolationschemes,suchasmean-valueinterpolationmaybeusedaswell,offeringcomputationaladvantages.HereweshowhowtoconstructasuitablemembraneevenfasterbyapproximatingShepardÕsscattereddatainterpolationmethod[Shepard1968]usingaconvolutionpyramid.denotearegionofinterest(onadiscreteregulargrid),whoseboundaryvaluesaregivenbyb(x).Ourgoalistosmoothlyin-terpolatethesevaluestoallgridpointsinside#.ShepardÕsmethoddeÞnestheinterpolantratxasaweightedaverageofknownbound- isnon-zero,0otherwise).Intuitively,includingthechar-acteristicfunction$inthedenominatorensuresthattheweightsofthezerosinörarenotaccumulated.Again,weusetheoptimizationtoÞndasetofkernelsF,withwhichourconvolutionpyramidcanbeappliedtoevaluate(12).TodeÞnethetrainingdata,weset ],whichcomputesmeanvaluecoordinatesat edcfFigure8:GaussianÞlters:(a)OriginalImage(b)Exactconvolu-tionwithaGaussianÞlter(%=4)(c)Convolutionusingourap-proximationforthesame%.(d)Exactkernels(inred)withapprox-imatekernels(inblue).(e)exactGaussian(inred),approximationusing5x5kernels(inblue),approximationusing7x7kernels(ingreen).(f)showsamagniÞedpartof(e).6GaussiankernelsInthissectionwedemonstratehowtouseconvolutionpyramidstoapproximateconvolutionswithGaussiankernelse!% TheeffectiveÞltersthatBurtÕsmethodandintegralimagesproducearetruncatedinspace,i.e.,theyhaveaÞnitesupportthatdependsontheGaussianscale%.WhilethisisnotamajorissuewhentheÞlteringisusedforapplicationssuchasimageblurring,thistrun-cationhasasigniÞcanteffectwhentheÞltersareusedforscattereddatainterpolation,suchastheShepardÕsmethodoutlinedinthepre-vioussection.Inthisapplication,weneedthedatatopropagate,inamonotonicallydecreasingfashion,acrosstheentiredomain.Atruncatedkernelmayleadtodivisionsbyzeroin()ortointro-duceabrupttransitionsintheinterpolatedvalues.Figure9showsourapproximationoftwodifferentGaussianÞl-ters,differingintheirwidth.Theseapproximationswerecomputedusingthesmaller5-by-5kernels.TheapproximationofthewiderGaussianin(a)providesagoodÞtacrosstheentiredomain.ForthenarrowerGaussianin(b),theapproximationlosesitsrelativeaccuracyastheGaussianreachesverylowvalues.Thismaybe ACMTrans.Graph.22,3,313Ð318.PERONA,P.1995.Deformablekernelsforearlyvision.IEEETrans.PatternAnal.Mach.Intell.17,5,488Ð499.SELESNICK,I.2006.Ahigherdensitydiscretewavelettransform.IEEETrans.SignalProc.54,8(Aug.),3039Ð3048.SHANNO,D.F.1970.Conditioningofquasi-Newtonmethodsforfunctionminimization.MathematicsofComputation24,111