Deconvolution is an indispensable tool in image processing and computer vision It commonly employs fast Fourier trans form FFT to simplify computation This operator however needs to t ransform from and to the frequency domain and loses spatial infor ID: 74437
Download Pdf The PPT/PDF document "Inverse Kernels for Fast Spatial Deconvo..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
InverseKernelsforFastSpatialDeconvolutionLiXuyXinTaozJiayaJiazyImage&VisualComputingLab,LenovoR&TzTheChineseUniversityofHongKongAbstract.Deconvolutionisanindispensabletoolinimageprocessingandcomputervision.ItcommonlyemploysfastFouriertransform(FFT)tosimplifycomputation.Thisoperator,however,needstotransformfromandtothefrequencydomainandlosesspatialinformationwhenprocessingirregularregions.Weproposeanecientspatialdeconvolu-tionmethodthatcanincorporatesparsepriorstosuppressnoiseandvisualartifacts.Itisbasedonestimatinginversekernelsthatarede-composedintoaseriesof1Dkernels.AnaugmentedLagrangianmethodisadopted,makinginversekernelbeestimatedonlyonceforeachop-timizationprocess.OurmethodisfullyparallelizableanditsspeediscomparabletoorevenfasterthanotherstrategiesemployingFFTs.Keywords:deconvolution,inversekernels,numericalanalysis,optimiza-tion1IntroductionDeconvolutionhasbeenanessentialtoolforsolvingmanyimage/videorestora-tionandcomputervisionproblems.Itwasalsousedinastronomyimaging[24],medicalimaging[9],signaldecoding,etc.Inrecentyears,itisextensivelyappliedtosystemsincomputationalphotographyandimage/videoediting,including\ruttershuttermotiondeblurring[19],generalmotiondeblurring[6,30,22,4,14,10,28,25,29,21],codedapertureanddepth[13,32],andimagesuper-resolution[2,23,17],sincemanytypesofdegradationcanbepartlymodeledorapproxi-matedbyconvolution,wherekernelsaremonotonicallydecayinglow-pass lters.Whileconvolutioniseasytoapply,itsinverseproblemofproperlydeconvolv-ingimagesisnotthatsimple.Band-limitedconvolutionkernelshaveincompletecoverageinthefrequencydomain,whichmakesinversionill-conditioned,espe-ciallyundertheexistenceofunavoidablequantizationerrorsandcameranoise.Regularizationcanremedythisproblem{seeearlyworkofWiener ltering[27]andTikhonovdeconvolution[26].Existingmethodsareintwostreams,whichhavetheirrespectivecharacteristics.SpatialDeconvolutionVeryfewdeconvolutionmethodsareperformedinthespatialdomain,owingtothehighcomputationalcost.Richardson-Lucymethod[20]doesnotinvolveregularizationandthusmaysu erfromthenoiseandring-ingproblems.Progressiveapproach[31]suppressesringingsbyoperationsinimagepyramids.Goodperformanceisyieldedinsparsepriordeconvolution[13], 2L.Xu,X.TaoandJ.Jiawhichrequirestosolvelargelinearsystems.Withthere-weightingnumericalscheme,thecoecientmatrixofthelinearsystemisno-longerToeplitzandcan-notbeacceleratedusingFFTs.Thisindicatesthatsparse-priordeconvolution,albeitusefulforpreservingstructuresandsuppressingringings,isnottranslationinvariant.DeconvolutioninFrequencyDomainTheconvolutiontheoremstatesthatspatialconvolutioncanbecomputedbypoint-wisemultiplicationinfrequencydomain,whichbringsoutpseudo-inversioninthefrequencydomain[16].Shanetal.[22] ttedthegradientdistributionusingtwoconvexfunctions.Thehalf-quadraticimplementation[11]mathematicallylinksgeneral -normstoafamilyofhyper-Laplaciandistributions.TheseiterativemethodsemployafewFFTsineachpass.EachFFTiswithcomplexityO(nlogn)wherenisthepixelnumberintheimage.Althoughfrequencydomaindeconvolutionisfast,itisnon-trivialforfurtherspeedupbyparallelization.Norisitsuitabletohandleirregularregions,whichhoweverarecommoninobjectmotionblur[3]andfocalblur[13].OurContributionInthispaper,weanalyzethemaindicultyofspatialdeconvolutionandproposeanewnumericalschemebasedoninversekernelsto llthegapbetweenrecentfrequency-domainfastdeconvolutionandspatialpseudo-inverse.Theyareinherentlylinkedinoursystembyintroducingkernelsconstructedaccordingtoregularizedoptimization.Thenewrelationshipenablesempiricalstrategiestoinheritthenicepropertiesinthesetwostreamsofworkandtosigni cantlyspeedupspatialdeconvolution.Althoughseveralusefulsparsegradientpriorsmaynotleadtotranslationin-variantprocessfordeconvolution.Wefounditispossibletoapproximatethemwithaseriesofoperatorsthatareindeedspatiallytranslationinvariant.Ac-cordingly,weproposeane ectivenumericalschemebasedontheaugmentedLagrangianmultipliers[15,1]andkerneldecomposition[18].Theresultingoper-ationsarenomorethanestimationofasetof1Dkernelsthatcanberepeatedlyappliedtoimagesiniterations.Unlikeallpreviousfastrobustdeconvolutiontechniques,ourmethodworksspatiallyandhasanumberofadvantages.1)Itiseasytoimplementandparal-lelize.2)ItrunscomparablywithorevenfasterthanFFT-baseddeconvolutionforhigh-resolutionimages.3)Thismethodcandealwitharbitrarilyirregularregionswithoutmuchcomputationoverhead.4)Visualartifactsaremuchre-duced.Weapplyourmethodtoapplicationsofextendeddepthof eld[12],motiondeblurring[29],andimageupscalingusingbackprojection[8].2MotivationandAnalysisTounderstandtheinherentdi erencebetweenspatialandfrequencydomaindeconvolution,webeginwiththediscussionofconvolutionexpressedintheformy=xk+; InverseKernelsforFastSpatialDeconvolution3 (b)(c)(g)(a)(d)(e)()f()h Fig.1.Illustrationofregularizedinverse lters.(f).(a)isaGaussianblurredimage.(b)-(d)aretherestoredimagesbyconvolvingtheregularizedinverse lter,Wienerdeconvolution,and1DseparatedWienerdeconvolution.(e)showstheGaussiankernel.(f)showsthedirectinverse lter,andregularizedinverse lterfromtopdown.(g)contains1Dscanlinesofthetwoinverse ltersin(f).(h)showstheclose-upsof(c)and(d).wherekisthekernel,yisthedegradedobservation,xisthelatentimage,referstotheconvolutionoperator,andindicatesadditivenoise.We rstexplaintheinversekernelproblemusingthesimpleWienerdecon-volutionandthendiscusstheissuesindesigningapracticalspatialsolverusingsparsegradientpriors,whichise ectivetosuppressnoiseandvisualartifacts.2.1SpatialInverseKernelsforWienerDeconvolutionWienerdeconvolutionintroducesapseudo-inverse lterinfrequencydomain,expressedasW= F(k) F(k)2+1 SNR(1)whereF()denotesFouriertransformand F()isitscomplexconjugate.SNRrepresentsthesignaltonoiseratiothathelpssuppressthehighfrequencypartoftheinverse lter.Therestoredimageisthusx=F 1(WF(y))(2)whereF 1istheinverseFouriertransform. 4L.Xu,X.TaoandJ.JiaAlbeitecient,restorationusingFFTslosesthespatialinformationasdis-cussedaboveandcouldbelessfavoredinseveralapplications.Thismotivatesustoapproximatethisprocessusingpseudo-inversewinthespatialdomain,expressedasx=F 1(W)y=wy;(3)wherewisthelatent(pseudo)spatialinversekernel.Itisknowninsignalpro-cessingthatthistaskcannotalwaysbeaccomplishedgivenanarbitraryWTakingthesimple2DGaussian lterforexample(Fig.1(e)),itsdirectspatialinversekernelisa2Din niteimpulseresponse(IIR) lter,asshowninthetopofFig.1(f).Contrarily,wefoundthatthespatialcounterpartofWienerinversion,i.e.F 1(W),hasa nitesupport,asshowninthebottomofFig.1(f).Thedi er-enceisduetotheinvolvementofregularization1=SNR.Itisactuallyageneralobservationthatinverse lterswithregularizationaretypicallywithdecayingspatialresponses.An1DvisualizationisgiveninFig.1(g).Thekernelwithregularization(bottom)decaysquicklyandthushasacompactsupport.AnimagedegradedbyaGaussiankernel(Fig.1(e))isshowninFig.1(a).TherestoredimageusingthespatialinversekernelwithcompactsupportisgiveninFig.1(d),withvisualartifactsnearimageborder,whichcanbeamelioratedbypadding.Tofurtherincreasethesharpnessandsuppressartifacts,weturntoamoreadvancedsparsegradientregularization.2.2SparseGradientRegularizedDeconvolutionState-of-the-artdeconvolutionmakesuseofsparsegradientpriors[13,11],mak-ingtheoverallcomputationmorecomplexthanaWienerone.Inthispaper,weproposeapracticalschemetoachievespatialdeconvolutionevenwiththesechal-lenginghighlynon-convexsparsepriors.Wedescribetwoissuesinthisprocess,whichconcernkernelsizeandnon-separabilityofregularizeddeconvolution.KernelSizeSpatialinversekernelscouldbeofconsiderablesizes.ForaGaussiankernelwithvariance=3,thecorrespondingregularizedinverse lterusingEq.(1)hasa nitesupportof5151.Althoughitisindependentoftheinputimagesize,itstilllaysalargecomputationalburdento2Dconvolution.KernelNon-separabilityManykernelsareinherentlynon-separable.Evenforthosethatareseparable,theirinversionsarenot.Forexample,eachGaussiankernelcanbedecomposedintotwo1D lters,appliedinthehorizontalandverticaldirectionsrespectively.However,itsinversionisnotseparableduetoregularization.Theroadtospeedingupregularizeddeconvolutionbysimplyperforming1D lteringisthusblocked.ThecomparisoninFig.1(c)and(d)illustratesthedi erence.Thereisa2DinversekernelofGaussiancreatedaccordingtoEq.(1)andaseparatedapproximationusingouterproductoftwo1D lters,formedalsofollowingEq. InverseKernelsforFastSpatialDeconvolution5(1).Therestorationresultusingthere-combined1D ltersisshownin(d).Itcontainsobviousoblique-lineartifacts(seetheclose-upsin(h)).WeaddressthesetwoissuesusingkerneldecompositionwithSVD,presentedbelow.3SparsePriorRobustSpatialDeconvolutionSparsegradientregularizeddeconvolutionworksverywellwithahyperLaplacianprior[11].ItminimizesthefunctionofE(x)=nXi=1 2(xk y)2i+c1x i+c2x i(4)whereiindexesimagepixels.c1andc2are nitedi erentialkernelsinhorizontalandverticaldirectionstoapproximatethe rst-orderderivatives. controlstheshapeofthepriorwith05 1.AcommonwaytosolvethisfunctionistoemployapenaltydecompositionE(xz1;z2)=nXi=10@ 2(xk y)2i+Xj2f1;2 2(zj cjx)2i+zj i1A(5)wherez1andz2areauxiliaryvariablestoapproximateregularizers.Theproblemapproachestheoriginaloneonlyif islargeenough.Thesolveristhusformedasiterativelyupdatingvariablesaszt+1j argminzE(xt;zj; t)(6)xt+1 argminxE(x;zt+1j; t)(7) t+1 2 t(8)tindexesiterations.Sincezjhasananalyticalsolution(orcanbefoundinlook-uptables)[11],themaincomputationliesintheFFTinversionsteptocomputex,whichgivesx=F 1 Pj F(cj)F(zj)+ F(k)F(y) PjF(cj)2+ F(k)2!(9)ItinvolvesseveralFFTs.Basically,updateofzjisperformedinspatialdomainasitinvolvespixel-wiseoperations.Sodomainswitchisunavoidable.3.1PenaltyDecompositionInverseKernelsWeexpandEq.(9)bydecomposingthenumeratoranddenominatorandapplyinverseFFTseparately.Ityieldsx=F 1 1 PjF(cj)2+ F(k)2!0@Xjc0jzj+ k0y1A(10) 6L.Xu,X.TaoandJ.Jiawherec0jandk0areadjointkernelsofcjandkbyrotatingthesekernelsby180degree,andjindexesdi erentialkernelsc.Theoperationsc0jzjandk0yarenowinspatialdomain.k0yisaconstantindependentofvariableszandx(PF(cj)2+ F(k)2) 1inEq.(10)istheinversioninthefrequencydo-main.Itsdomainswitchtopixelvalues,infact,correspondstoaspatialinversekernel.Theregularizationmakesits nitesupportexist.Soitispossibletoestimatespatialinversekernelscorrespondingtothisterm,i.e.,w =F 1 1 PjF(cj)2+ F(k)2!(11)Thisprocessraisesatechnicalchallenge.Because variesiniterations,w needstobere-estimatedineachpass.Aseriesofspatialinverse ltersthusshouldbeproduced,whicharenotoptimalandwastemuchtime.3.2AugmentedLagrangianInverseKernelsTo tthespatialprocessingframework,weadopttheaugmentedLagrangian(AL)method[15,5]toapproximatedeconvolution.ALwasoriginallyusedtotransformconstrainedoptimizationtoanunconstrainedonewiththeconven-tionalLagrangianandanadditionalaugmentedpenaltyterm.Speci cally,wetransformEq.(4)intoE(xzj;\rj)=nXi=10@ 2(xk y)2i+Xj2f1;2zj i+Xj2f1;2 2(zj cjx)2i h\rj(zj cjx)ii1A(12)wheretheterminthesecondrowistheaugmentedLagrangianmultiplierspeci cforthisproblem.hiistheinnerproductoftwovectors.Themajordi erencefromtheoriginalpenaltydecompositionoptimizationisthatheretheupdateof\rjprevents fromvaryingwhiletheoptimizationstillproceedsnicely.Theiterativesolverisgivenbyzt+1j argminzE(xt;zj;\rtj)(13)xt+1 argminxE(x;zt+1j;\rtj)(14)\rt+1j \rtj (zt+1j cjxt+1)(15)Fromtheconvergencepointofview,theALmethodhasbasicallynodi erencewithpenaltydecomposition.Butitismuchmoresuitableforourdeconvolutionframework,inwhich canbe xed,resultinginthesameinversekernelinalliterations. InverseKernelsforFastSpatialDeconvolution7 (a) (b)(c)(d)(e) Fig.2.Separating lters.Aspatialinverse ltershownin(a)canbeapproximatedasalinearcombinationofafewsimpleronesasshownfrom(b)-(e).Eachofthemisseparable.The nallyrestoredimagein(a)canbeformedasalinearcombinationofimagesrestoredbythesesimple ltersrespectively.Byre-organizingtheterms,wegetanexpressionforthetargetimage:x=F 1 1 PjF(cj)2+ F(k)2!F 1 Xi F(cj)(F(zj) 1 F(\rj))+ F(k)F(y)!=w 0@Xj2f1;2c0j(zj 1 \rj)+ k0y1A(16)wherew denotesthesamespatialinverse lterde nedinEq.(11).Thedi er-enceisthat inthisformnolongervariesduringiterations.c0j(zj 1 \rj)canbeecientlycomputedusingforward/backworddi erence.k0yisaconstantandcanbecomputedonlyoncebeforetheiteration.w isaspatialinversekernelthatcanalsobepre-computedandstored.Itseemsnowwesuccessfullyproduceworkableinversekernelwithoutheavycomputationspenttore-estimatingitineachiteration.Buttherearestillt-woaforementionedsizetheseparabilityissuesthatmayin\ruencedeconvolutioneciency.Wefurtherproposeadecompositionproceduretoaddressthem.InverseKernelDecompositionKerneldecompositiontechniqueshavebeenwidelyexplored.Steerable lters[7]decomposekernelsintolinearcombination 8L.Xu,X.TaoandJ.Jiaofasetofbasis lters.Anotherkerneldecompositionisbasedonthesingularvaluedecomposition(SVD)ofw bytreatingitasamatrix[18].Comparedtosteerable lter,itisanon-parametricdecompositionforarbitrary lters.Givenourspatialkernelw ,wedecomposeitasw =USV0,whereUandV0areunitaryorthogonalmatrices,V0isthetransposeofV,andthematrixSisaband-diagonalmatrixwithnonnegativerealnumbersinthediagonal.WeusefluandflvtodenotethelthcolumnvectorsofUandV,whichinessenceare1D lters.w isexpressedasw =Xlslfluflv0(17)Convolvingw withanimageisnowequivalenttoconvolvingasetof1Dkernelsfluandflv.Itcanbeecientlyappliedinspatialdomainwherethenumberof ltersiscontrolledbythenon-zeroelementsinthesingularvaluematrix,inlinewiththerankofthekernel.Ifakernelisspatiallysmooth,whichiscommonfornaturalimages,therankcanbeverysmall.Itisthusallowedtouseonlyafew1Dkernelstoperformdeconvolution.Notethatwecanevenlowertheapproximationprecisionbydroppingsmallnon-zerossingularvaluesforfurtheracceleration.OneexampleofthekernelanditsdecompositionisshowninFig.2.The lteredimagesareshowntogetherwiththeirseparable lters.Inthisexamples,7separable ltersareusedtoapproximatetheinverseregularizedGaussian,whichveri esthatmostinversekernelsarenotoriginallyseparable.3.3MoreDiscussionsOurspatialdeconvolutionisaniterativeprocess.Foreachdeconvolutionprocess,weonlyneedtouseSVDtoestimatew asseveral1Dkernelsonce.Ifthekernelwasdecomposedbefore,w isstoredinour lesforquicklookup.Inthisregard,commonkernels,suchasGaussians,canbepre-computedtosavecomputationduringdeconvolution.Thespatialsupportofthe1Dinversekernelsdependsontheamountofregularization,i.e.theweight.Fornoisyimages,issetsmall,correspondingtostrongregularization.Accordingly,thesizeofinversekernelsissmall.Inpractice,thesupportof1Dkernelsisestimatedbythresholdinginsigni cantvaluesinthekernelandremovingboundaryzerovalues,whicharedeterminedautomaticallyonceisgiven.Thepseudo-codeforinversekerneldeconvolutionisprovidedinAlg.1.4ExperimentalValidationWeevaluatethesystemperformancewithregardtorunningtimeandresultquality.Ourmainobjectiveistohandlefocal,Gaussianorevensparsemotionblur.Inourimplementation,primaryparametersinEq.(12)aresetasfollows:2[5003000],dependingontheimagenoiselevel; is xedto10forall InverseKernelsforFastSpatialDeconvolution9 Algorithm1x=FastSpatialDeconvolution(yk) 1w realF 11 PijF(ci)j2+ jF(k)j22fsl;flu;flvg svd(w )3Discardfsl;flu;flvgpairswithslbelowathreshold4x1 y,\r1 05fort=1tomaxIters6dozt+1i argminzE(xt;zi;\rti)7a i2f1;2gc0i(zt+1i 1 \rti)+ k0y8xt+1 09forl=1tolength(fslg)10doxt+1 xt+1+slafluflv011\rt+1i \rti (zt+1i cixt+1) Table1.Runningtime(inseconds)andPSNRsfordi erentmethods ImageSize RL IRLS TVL1 FastPD Ours 325x365 0.91 85.83 7.50 0.59 0.57 1064x694 2.28 241.34 22.20 2.00 3.27 1251x1251 6.19 537.89 54.30 4.61 7.30 PSNRs 20.2 24.3 22.7 23.3 23.7 images;totally5iterationsareenoughinpractice.Wecomparedourmethodwithothers,includingthespatial-domainRichardson-Lucy(RL)deconvolution,IRLS[13](shortfortheiterativere-weightedleastsquares)approach,TVL1deconvolution[28]andthefastdeconvolution[11],denotedasPDfor\penaltydecomposition".TheTVL1methodisimplementedinClanguageandalltheotherfourmethodsareimplementedinMATLAB.Werun20iterationsforthestandardRL.Allothermethodsarebasedontheauthors'implementationwithdefaultparameters.Runningtimeisobtainedondi erentsizesofimages.Intotal,wecollect10naturalimageswithdi erentresolutions.TheyareblurredwithGaussian lterswithvariance2f12345grespectively.SmallGaussiannoiseisaddedtoeachimage.RunningtimeforthreeresolutionsisreportedinTable1.OurmethodissimilarlyfastasPDemployingFFTsandisamagnitudefasterthanIRLSandTVL1.Ourmethodupdateszwithanalyticalsolutions.Itcanbefurtherspedupbyusingalook-uptable.Asw ispre-computed,wedonotincludeitsestimationtimeinthetable.Inourexperiments,a5151kerneliscomputedin0.1second.The nalPSNRsofallthe10examplesareincludedinTable1.WeshowinFig.3avisualcomparisonalongwithclose-upsfordi erentmethods.Ourresultiscomparablewiththesharpestonewhilenotcontainingextravisualartifacts. 10L.Xu,X.TaoandJ.Jia (a)Input(b)Richardson-Lucy(c)IRLS (d)TVL1(e)FastPD(f)OursFig.3.Visualcomparison.Similarqualityresultsmanifestthatourmethoddoesnotintroduceadditionalvisualartifacts. Fig.4.Samplemotionandfocalblurkernelsforvalidation.StatisticsofFiltersWenowpresentthestatisticsofthe1D ltersw learnedfromdi erenttypesofkernels.Wecollectedasetof ltersinrealmotionblur,representativeGaussianconvolution,andnaturalout-of-focus.The8motionblurkernelsarefrom[14].TheGaussianblurkernelsarewithdi erentscales,controlledbyvariance2f12345g.Wealsocollectfrominternettherealfocalblurkernels.Wenormalizeallofthemtosize3535.AfewexamplesareshowninFig.4.ThestatisticsinTable2indicatethatmotiondeconvolutiontypicallyrequiresmore1Dkernelstoapproximatetheinverse lterthanothers,dueprimarilytolargekernelvariationandcomplexshapes.Convolvingtensofkernelsthatap-proximatew isinfactacompletelyparallelprocessandcanbeeasilyacceleratedusingmultiple-coreCPUandGPU.Thenumberof1Dkernelsisdeterminedbythresholdingthesingularvaluesanddroppingoutinsigni cantones.Varyingthethresholdresultsindi erentnumbersof1Dkernelsandthusa ectstheperformance.WeshowinFig.5howthethresholda ectsthequalityofrestoredimages.Onethresholdcanbeappliedtodi erenttypesofkernelstogeneratereasonableresults.Wealsonotebased InverseKernelsforFastSpatialDeconvolution11Table2.Kerneldecompositionstatistics.\Averagenumber"referstotheaveragenumberofnon-zerosingularvalues,i.e.,thenumberof1D ltersused.\Averagelength"isthelengthofeach1Dkernel. Type Avg.number Avg.length Motion 36.4 110.3 Gaussian 8.3 71.2 Out-of-focus 15.7 87.8 19212325272931 Gaussian Out-of-focus Motion PSNRSingularValueThreshold (log) Fig.5.PSNRsversussingularvaluethresholdsfordi erenttypesofkernels.Thesin-gularvaluethresholdsareplottedinalogarithmicscale.onTable2thatonethresholdmaygeneratedi erentnumbersof1Dkernelsdependingonthestructureandcomplexityoftheoriginalconvolutionkernels.5ApplicationsWeapplyourmethodtoafewcomputervisionandcomputationalphotographyapplications.5.1Deconvolution-IntensiveSuper-ResolutionIterativeback-projection[8]isonee ectiveschemetoupscaleimagesandvideos,andisfastingeneral.Inthisprocess,reconstructionerrorsarebackprojectedin-tothehighresolutionimagethroughinterpolationanddeconvolution,expressedasht+1=ht+(l (htG)#)"p;(18)whereGisakernelthatcouldbeGaussian[8]ornon-Gaussian[17],histhetargethigh-resolutionimageandlisitslow-resolutionversion.#and"aresimpledownscalingandupscalingwithinterpolationoperations.pisthepseudo-inverseofthekernel.Agoodppositivelyin\ruenceshigh-qualityimagesuper-resolution.Sowesubstituteourspatialdeconvolutionforp,whichcountsinregularization 12L.Xu,X.TaoandJ.Jia (a) Input(b) Back projection [8](c) Ours Fig.6.Super-resolutionbyback-projection.indeconvolution.ItproducestheresultsshowninFig.6.Theydemonstratetheusefulnessofourinversekernelscheme,asvisualartifactsaresuppressed.5.2ExtendedDepthofFieldTheproposedmethodcanbeappliedtoremovalofpartoffocalblur.Weemployitintheextendeddepthof eldphotography[12],whichgeneratesablurryimageforeachdepthlayerandrestoresitusingdeconvolution.Blurryimagegenerationisachievedbycontrollingthemotionofthedetectorduringimageintegrationorrotatingthefocusring.SincetheresultingblurPSFsbelongtothegeneralizedGaussianfamily,theycanbeecientlycomputedusingourspatialscheme.Fig.7showstwoexamples.Ittakes1.7sbyourmethodonasingleCPUcoretoproducetheresultsshownin(b)withresolution6811032.Incomparison,thefastdeconvolutionmethod[11]takes2stoproducetheresultsin(c).Ourmethodcanbefullyparallelizedtomuchspeedupcomputation.5.3MotionDeblurringMotionblurkernelsareingeneralasymmetric,correspondingtoalargernumberof1Dkernelsinourdecompositionstep.Itrevealsthenon-separablenatureofmotionkernels.Ourinversekernelschemeisstillapplicableherethankstotheindependenceofeach1D lteringpass.WeshowinFig.8theIRLSdeconvolu-tionresultsof[13]andourinverse lterresults.Thegroundtruthclearimagesandmotionblurkernelsarepresentedintheoriginalpaper[14].Whilebothap-proachesworkinspatialdomain,ourstakes0.5stoprocessthe255255images,comparedtothe70secondsbytheIRLSmethod.5.4Real-timePartialBlurRemovalOurmethoddirectlyhelpspartialimagedeconvolution.Fouriertransformre-quiressquareinputsandanyerrorproducedafterdomainswitchwillbeprop- InverseKernelsforFastSpatialDeconvolution13 (a) Input(b) Ours(c) PD [11] Fig.7.Reconstructedpicturesfromextendeddepthof eldcameras.agatedacrosspixelsduetothelackofspatialconsideration.Ourmethoddoesnothavetheseconstraints.Ourcurrentimplementationcanachievereal-timeperformanceon130130patchesonasingleCPUcore.Itisnotablethatanyshapesofregionscanbehandledinthissystem.Ourempiricallyprocessedre-gionsareslightlyexpandedfromtheusermarkedonestoincludemorepixelsinoptimizationinordertoavoidboundaryvisualartifacts.OneexampleisshowninFig.9,whereabookisfocalblurred.Werestoreapatchusingourmethod,whichdoesnotintroduceunexpectedringingartifacts.Ourmethodtakesonly0.07secondtoprocessthecontent,comparedwith0.4secondneededintheFFT-basedmethod[11]toprocessallpixelswithinthetightestboundingboxenclosingtheselectedregion.Theclose-upsareshownin(c)and(d).Thedi erenceiscausedbyprocessingonlythemarkedpixelsbyourmethodandprocessingallpixelsintherectangularboundingboxbytheFFT-involvedmethod.6ConclusionWehavepresentedaspatialdeconvolutionmethodleveragingthepseudo-inversespatialkernelsunderregularization.FixedkernelestimationisachievedusingtheaugmentedLagrangianmethod.Ourframeworkisgeneraland ndsmanyapplications.Itsimpactisthenumericalbridgetoconnectfastfrequency-domain 14L.Xu,X.TaoandJ.Jia (a) Input(b) IRLS [13](c) Ours Fig.8.Motiondeblurringexamples. (a)Input(b)Ourresult(c)Close-up(Ours)(d)Close-up(PD)Fig.9.PartialBlurRemoval.In(a),wemarkafewpixelsfordeconvolution.Theresultisshownin(b)withtheclose-upin(c).TheFFT-basedmethod(PD)yieldstheresultshownin(d)bydevolvingallpixelsintheboundingbox.operationsandrobustlocalspatialdeconvolution.Ourmethodinheritsthespeedandlocation-sensitivityadvantagesinthesetwostreamsofworkandopensupanewareaforfutureexploration.Themethodcouldbeamazinglyecientifthese1Dkernelbasesinvolvedindecompositionarehandledbydi erentthreadsintheparallelcomputingarchitecture.ItworkswellforgeneralGaussianandotherpracticalmotionandfocalblurkernels.Onedirectionforfutureworkistoinvestigatespatiallyvaryinginversekernelsforcomplexblur.AcknowledgementsTheworkdescribedinthispaperwaspartiallysupportedbyagrantfromtheRe-searchGrantsCounciloftheHongKongSpecialAdministrativeRegion(ProjectNo.413113).TheauthorswouldliketothankShichengZhengforhishelpinimplementingpartofthealgorithm. InverseKernelsforFastSpatialDeconvolution15References1.Afonso,M.V.,Bioucas-Dias,J.M.,Figueiredo,M.A.:Anaugmentedlagrangianapproachtotheconstrainedoptimizationformulationofimaginginverseproblems.ImageProcessing,IEEETransactionson20(3),681{695(2011)2.Agrawal,A.K.,Raskar,R.:Resolvingobjectsathigherresolutionfromasinglemotion-blurredimage.In:CVPR(2007)3.Chakrabarti,A.,Zickler,T.,Freeman,W.T.:Analyzingspatially-varyingblur.In:CVPR.pp.2512{2519(2010)4.Cho,S.,Lee,S.:Fastmotiondeblurring.ACMTrans.Graph.28(5)(2009)5.Danielyan,A.,Katkovnik,V.,Egiazarian,K.:Imagedeblurringbyaugmentedla-grangianwithbm3dframeprior.In:WorkshoponInformationTheoreticMethodsinScienceandEngineering.pp.16{18(2010)6.Fergus,R.,Singh,B.,Hertzmann,A.,Roweis,S.T.,Freeman,W.T.:Removingcamerashakefromasinglephotograph.ACMTrans.Graph.25(3),787{794(2006)7.Freeman,W.T.,Adelson,E.H.:Thedesignanduseofsteerable lters.IEEETrans.PatternAnal.Mach.Intell.13(9),891{906(1991)8.Irani,M.,Peleg,S.:Motionanalysisforimageenhancement:Resolution,occlusion,andtransparency.JournalofVisualCommunicationandImageRepresentation4(4)(1993)9.Jerosch-Herold,M.,Wilke,N.,Stillman,A.,Wilson,R.:Magneticresonancequan-ti cationofthemyocardialperfusionreservewithafermifunctionmodelforcon-straineddeconvolution.Medicalphysics25,73(1998)10.Joshi,N.,Zitnick,C.L.,Szeliski,R.,Kriegman,D.J.:Imagedeblurringanddenois-ingusingcolorpriors.In:CVPR.pp.1550{1557(2009)11.Krishnan,D.,Fergus,R.:Fastimagedeconvolutionusinghyper-laplacianpriors.In:NIPS(2009)12.Kuthirummal,S.,Nagahara,H.,Zhou,C.,Nayar,S.K.:Flexibledepthof eldphotography.IEEETrans.PatternAnal.Mach.Intell.33(1),58{71(2011)13.Levin,A.,Fergus,R.,Durand,F.,Freeman,W.T.:Imageanddepthfromacon-ventionalcamerawithacodedaperture.ACMTrans.Graph.26(3),70(2007)14.Levin,A.,Weiss,Y.,Durand,F.,Freeman,W.T.:Understandingandevaluatingblinddeconvolutionalgorithms.In:CVPR.pp.1964{1971(2009)15.Lin,Z.,Chen,M.,Ma,Y.:Theaugmentedlagrangemultipliermethodforexactrecoveryofcorruptedlow-rankmatrices.UIUCTechnicalReportUILU-ENG-09-2215(2010)16.Mathews,J.,Walker,R.L.:Mathematicalmethodsofphysics,vol.271.WABen-jaminNewYork(1970)17.Michaeli,T.,Irani,M.:Nonparametricblindsuper-resolution.In:ICCV(2013)18.Perona,P.:Deformablekernelsforearlyvision.IEEETrans.PatternAnal.Mach.Intell.17(5),488{499(1995)19.Raskar,R.,Agrawal,A.K.,Tumblin,J.:Codedexposurephotography:motiondeblurringusing\rutteredshutter.ACMTrans.Graph.25(3),795{804(2006)20.Richardson,W.:Bayesian-basediterativemethodofimagerestoration.JournaloftheOpticalSocietyofAmerica62(1),55{59(1972)21.Schmidt,U.,Rother,C.,Nowozin,S.,Jancsary,J.,Roth,S.:Discriminativenon-blinddeblurring.In:CVPR.pp.604{611.IEEE(2013)22.Shan,Q.,Jia,J.,Agarwala,A.:High-qualitymotiondeblurringfromasingleimage.ACMTrans.Graph.27(3)(2008) 16L.Xu,X.TaoandJ.Jia23.Shan,Q.,Li,Z.,Jia,J.,Tang,C.K.:Fastimage/videoupsampling.ACMTrans.Graph.27(5),153(2008)24.Starck,J.,Pantin,E.,Murtagh,F.:Deconvolutioninastronomy:Areview.Publi-cationsoftheAstronomicalSocietyofthePaci c114(800),1051{1069(2002)25.Tai,Y.W.,Lin,S.:Motion-awarenoise lteringfordeblurringofnoisyandblurryimages.In:CVPR.pp.17{24(2012)26.Tikhonov,A.,Arsenin,V.,John,F.:Solutionsofill-posedproblems(1977)27.Wiener,N.:Extrapolation,interpolation,andsmoothingofstationarytimese-ries:withengineeringapplications.JournaloftheAmericanStatisticalAssociation47(258)(1949)28.Xu,L.,Jia,J.:Two-phasekernelestimationforrobustmotiondeblurring.In:ECCV(1).pp.157{170(2010)29.Xu,L.,Zheng,S.,Jia,J.:Unnaturall0sparserepresentationfornaturalimagedeblurring.In:CVPR.pp.1107{1114(2013)30.Yuan,L.,Sun,J.,Quan,L.,Shum,H.Y.:Imagedeblurringwithblurred/noisyimagepairs.ACMTrans.Graph.26(3),1(2007)31.Yuan,L.,Sun,J.,Quan,L.,Shum,H.Y.:Progressiveinter-scaleandintra-scalenon-blindimagedeconvolution.ACMTrans.Graph.27(3)(2008)32.Zhou,C.,Lin,S.,Nayar,S.K.:Codedaperturepairsfordepthfromdefocusanddefocusdeblurring.InternationalJournalofComputerVision93(1),53{72(2011)