PSF Estimation using Sharp Edge Prediction Neel Joshi Richard Szeliski David J - PDF document

PSFEstimationusingSharpEdgePredictionNeelJoshiRichardSzeliskiDavidJ.KriegmanUniversityofCalifornia,SanDiegoMicrosoftResearchImagebluriscausedbyanumberoffactorssuchasmo-tion,defocus,capturinglightoverthenon-zeroareaoftheapertureandpixel,thepresenceofanti-aliasing“ltersonacamerasensor,andlimitedsensorresolution.Wepresentanalgorithmthatestimatesnon-parametric,spatially-varying Max Valid Region Figure1.SharpEdgePrediction.Ablurryimage(topleft)andthe1Dpro“lenormaltoanedge(topright,blueline).Wepredictasharpedge(topright,dashedline)bypropagatingthemaxandminvaluesalongtheedgepro“le.ThealgorithmusespredictedandobservedvaluestosolveforaPSF.Onlyobservedpixelswithinaareused.(bottomleft)Predictedpixelsareblueandvalidobservedpixelsaregreen.(bottomright)Thepredictedvalues. Motion Blur Distortion Defocus Blur Aliasing Perspective Transform Sensor Geometric Transformations Point Spread Function Figure2.ImageFormationModel.Theimagingmodelconsistsoftwogeometrictransformsaswellasblurinducedbymotion,defocus,sensoranti-aliasing,and“nite-areasensorsampling.Wesolveforanestimateofthecontinuouspoint-spreadfunctionateachdiscretelysampled(potentiallyblurryandnoisy)pixel.Wealsoshowthatbysolvingforspatiallyvarying,per-colorchannelPSFscombinedwithper-channelradialdistortioncorrections,wecanremovechromaticaberrationsartifacts.2.RelatedWorkTheproblemofblurkernelestimationandmoregener-allyblinddeconvolutionisalongstandingproblemincom-putervisionandimageprocessing.Theentirebodyofpre-viousworkinthisareaisbeyondwhatcanbecoveredhere.Foramoreindepthstudyofmuchoftheearlierworkinblurestimation,wereferthereadertothesurveypaperbyKundurandHatzinakos[6].Inthecomputervisionliterature,classicalshape-from-defocus[10]addressesPSFestimationusingaparametricmodelforblurthatiseitherapillboxŽor2DGaussianfunctionwithasingleparameterforthePSFsize,i.e.,fo-callengthorkernelradius.Formorecomplexblurs,suchasmotionblur,manyrecentsingle-imageestimationtech-niquesmodelblursasacollectionsof1Dor2Dboxblursandusesegmentationtechniquestohandlemultiplemo-tions[7,4,2].Shanetal.[12],ontheotherhand,usealow-parametermodeltoremovemotionblurduetoanob-jecttranslatingandrigidlyrotatingaboutanaxisparalleltothecamerasopticalaxis.Incontrastwiththispreviouswork,wedonotuseaparametricmodelforthePSFandsolveforspatiallyvaryingkernelswithoutperforminganyexplicitsegmentation.Thereissigni“cantlylessworkintheareaofsingleim-ageblurestimationusingnon-parametrickernels.TheworkbyFergusetal.[3]isperhapsthemostnotablemethodofthistype.Fergusetal.usenaturalimagestatisticstoderiveanimagepriorthatisusedinavariationalBayesformula-tion.Incontrast,weleveragepriorassumptionsonimagestodirectlypredicttheunderlyingsharpimage.WeconsiderourapproachcomplementarytothatofFergusetal.,asourmethodexcelsataccuratelycomputingsmallerkernels,anditcanbeusedforlensandsensorcharacterization.Theirmethodisnotaswellsuitedtotheseapplications,butexcelsatcomputinglargekernelsduetocomplexcameramotion,whichisoutsidethescopeofourwork.Ourworkisconceptuallymostsimilartoslant-edgecal-ibration[11,1].Thesemethodsrecover1Dblurpro“lesbyimagingaslantededgefeatureand“ndingthe1Dkernelnormaltotheedgepro“lethatgivesrisetotheblurredob-servationsoftheknownstepedge.Reichenbachetal.[11]notethatonecancombineseveral1Dsectionstoestimatea2DPSF.Wetakeasimilarapproachphilosophicallytoslant-edgetechniques,withthreemajordifferences:weex-tendthemethodtodirectlysolvefor2DPSFs,wesolveforspatiallyvaryingPSFs,andwepresentablindapproachwheretheunderlyingstepedgeisnotknowapriori.Arelatedareaismodulationtransferfunction(MTF)estimationforlensesthatusesimagesofrandomdotpat-terns[8].Intheory,in“nitesimaldotpatternsareusefulforPSFestimation,butinpractice,itisnotpossibletocreatesuchapattern.Incontrast,creatingsharpstepedgesisrela-tivelyeasyandthusgenerallypreferable[11].Anadditionaladvantageofourworkrelativetousingdotpatternsisthatbyusingagrid-likestructurewithregular,detectablecor-nerfeatures,wecancomputearadialdistortioncorrectioninadditiontoestimatingPSFs.3.ImageFormationModelWenowgiveabriefoverviewofrelevantimagingandopticsconceptsneededforPSFestimation.AsillustratedinFigure2,theimagingmodelconsistsoftwogeomet-rictransforms:aperspectivetransform(usedwhenpho-tographingaknownplanarcalibrationtarget)andaradialdistortion.Thereareseveralsourcesofblurinducedbymo-tion,defocus,sensoranti-aliasing,andpixelsamplingarea(“llfactorandactivesensingareashape).Wemodelallblurasaconvolutionalongtheimageplaneandaccountfordepthdependentdefocusblurand3Dmotionblurbyallow-ingforthePSFtobespatiallyvarying.OurmethodestimatesadiscretelysampledversionofthecontinuousPSFbyeithermatchingthesamplingtotheimageresolution(whichisusefulforestimatinglargeblurkernels)orusingasub-pixelsamplinggridtoestimateadetailedPSF,whichcaptureseffectssuchasanti-aliasingofthesensorandallowsustodomoreaccurateimagerestoration.Inaddition,bycomputingasub-pixelPSF,wecanperformsingle-imagesuper-resolutionbydeconvolvingup-sampledimageswiththerecoveredPSF.GeometricTransformations:Theworldtoimagetransformationconsistsofaperspectivetransformanda radialdistortion.Withtheblindmethod,weignoretheperspectivetransformandoperateinimagecoordinates.Withthenon-blindmethod,wherewephotographaknowncalibrationtarget,wemodeltheperspectivetrans-formationasa2Dhomographytomapknownfeatureloca-onthegridpatterntodetectedfeaturepointsfromtheimage.Weuseastandardmodelforradialdistor-x,yx,yx,y istheradiusrelativetotheim-agecenter.Givenaradialdistortionfunctionandwarpfunctionwhichappliesahomography,thefullalignmentprocessis.Wecomputetheparametersthatminimizethenormoftheresidual.Computingtheseparameterscannotbedonesimultaneouslyinclosedform.However,theproblemisbilinear,andthuswesolvefortheparametersusinganiterativeapproach.ModelingtheDiscretePoint-SpreadFunction:equationfortheobservedimageisaconvolutionofakernelandapotentiallyhigherresolutionsharpimage,plusadditiveGaussianwhitenoise,whoseresultispotentiallydown-sampled:down-samplesanimagebym,nsm,snatasamplingrateforintegerpixelcoordinatesm,n.Inourformulation,thekernelmodelsallblurringeffects,whicharepotentiallyspatiallyvaryingandwavelengthdependent.4.SharpImageEstimationTheblurringprocessisformulatedasaninvertiblelinearsystem,whichmodelstheblurryimageastheconvolutionofasharpimagewiththeimagingsystemsPSF.Thus,ifweknowtheoriginalsharpimage,recoveringthekernelisstraightforward.Thekeycontributionofourworkisare-liableandwidelyapplicablemethodforpredictingasharpimagefromasingleblurryimage.Inthefollowingsec-tion,wepresentourmethodsforpredictingthesharpim-age.InSection5,wediscusshowtoformulateandsolvetheinvertiblelinearsystemtorecoverthePSF.Inthefol-lowingdiscussion,weconsiderimagestobesinglechannelorgrayscale;inSection6,wediscusscolorimages.4.1.BlindEstimationForblindsharpimageprediction,weassumeblurisduetoaPSFwithasinglemode(orpeak),suchthatwhenanimageisblurred,theabilitytolocalizeapreviouslysharpedgeisunchanged;however,thestrengthandpro“leoftheedgeischanged,asillustratedinFigure1.Thus,bylocaliz-ingblurrededgesandpredictingsharpedgepro“les,locallyestimatingasharpimageispossible.Weassumethatallobservedblurrededgesresultfromconvolvinganidealstepedgewiththeunknownkernel.Ouralgorithm“ndsthelocationandorientationofedgesintheblurredimageusingasub-pixeldifferenceofGaus-siansedgedetector.Itthenpredictsanidealsharpedgeby“ndingthelocalmaximumandminimumpixelvalues,inarobustway,alongtheedgepro“leandpropagatesthesevaluesfrompixelsoneachsideofanedgetothesub-pixeledgelocation.Thepixelontheedgeitselfiscoloredaccord-ingtotheweightedaverageofthemaximumandminimumvaluesaccordingtothedistanceofthesub-pixellocationtothepixelcenter,whichisasimpleformofanti-aliasing(seeFigure1).To“ndthemaximumvalue,ouralgorithmmarchesalongtheedgenormal,samplingtheimagelookingforalo-calmaximumusinghysteresis.Speci“cally,themaximumlocationisthe“rstpixelthatislessthan90%(asopposedtostrictlylessthan)ofthepreviousvalue.Oncethisvalueandlocationareidenti“ed,westorethemaximumŽvalueasthemeanofallvaluesalongtheedgepro“lethatarewithin10%oftheinitialmaximumvalue.Ananalogousapproachisusedfortheminimum.Sincewecanonlyreliablypredictvaluesnearedges,weonlyuseobservedpixelswithinaradiusofthepredictedsharpvalues.Theselocationsarestoredasvalidpixelsinamask,whichisusedwhensolvingforthePSF,asdiscussedinSection5.Attheendofthepredictionprocess,wehaveapartiallyestimatedsharpimage,asshowninFigure1.4.2.Non-BlindEstimationFornon-blindsharpedgeprediction,wewanttocom-putethePSFgiventhatweknowthesharpimage.Sinceweanticipateusingthistechniqueinacontrolledlabsetup,wedesignedaspecialcalibrationpatternforthispurpose(Figure3).Wetakeanimageofthispatternandaligntheknowngridpatterntotheimagetogetthesharp/blurrypairneededtocomputethePSFaccurately.Thegridhascorner(checkerboard)featuressothatitcanbeautomati-callydetectedandaligned,anditalsohassharpstepedgesequallydistributedatallorientationswithinatiledpattern,sothatitprovidesedgesthatcaptureeveryradialsliceofthePSF.(Alternatively,wecansaythatthecalibrationpat-ternsprovidesmeasurablefrequenciesatallorientations.)Furthermore,werepresentthegridinmathematicalform(thecurvedsegmentsarearcs),whichgivesusaveryprecisede“nitionforthegrid,whichisadvantageousforperformingalignment.Fornon-blindprediction,wecontinuetoassumethatker-nelhasnomorethanasinglepeak.Thusevenwhenthepatternisblurred,wecandetectcornersonthegridwithasub-pixelcornerdetector.Becauseourcornersareactuallybalancedcheckerboardcrossings(radiallysymmetric),theydonotsufferfromshrinkageŽ(displacement)duetoblur- Figure3.Non-BlindEstimation.(left)Thetiledcalibrationpat-tern,(middle)croppedsectionofanimageofaprintedversionofthegrid,and(right)thecorrespondingcroppedpartoftheknowngridwarpedandshadedtomatchtheimageofthegrid.ring.Oncecornersarefound,thegroundtruthpatternisalignedtotheacquiredimage.Toobtainanaccuratealign-ment,wecorrectforbothgeometricandradiometricaspectsoftheimagingsystem.WeperformgeometricalignmentusingthecorrectionsdiscussedinSection3.We“tahomographyandradialdis-tortioncorrectiontomatchtheknownfeaturelocationsonthegridpatterntocornersdetectedwithsub-pixelprecisionontheacquired(blurry)imageoftheprintedgrid.Wealsomustaccountforthelightingandshadingintheimageofthegrid.Wedothisby“rstaligningtheknowngridtotheimage.Then,foreachedgelocation(asknownfrommathematicalformofthegroundtruthgridpattern),thealgorithm“ndsthemaximumandminimumvaluesontheedgepro“leandpropagatesthemjustasinthenon-blindapproach.Weshadethegridforpixelswithintheblurra-diusofeachedge.Byperformingtheshadingoperation,ouralgorithmhascorrectedforshading,lighting,andradialintensityfalloff.Figure3showstheresultsofthegeometricwarpandshadingtransfer.5.PSFEstimationOncethesharpimageispredicted,weestimatethePSFasthekernelthat,whenconvolvedwiththesharpimage,producestheblurredinputimage.Weformulatetheestima-tionusingaBayesianframeworksolvedusingamaximuma(MAP)technique.InMAPestimation,onetriesto“ndthemostlikelyestimatefortheblurkernelgiventhesharpimageandtheobservedblurredimage,usingtheknownimageformationmodelandnoiselevel.WeexpressthisasamaximizationovertheprobabilitydistributionoftheposteriorusingBayesrule.Theresultisminimizationofasumofnegativeloglikelihoods)=argminTheproblemisnowreducedtode“ningthenegativeloglikelihoodterms.Giventheimageformationmodel(Equa-tion1),thedatatermis:(Thedownsamplingtermin(1)willbeincorporatedinSection5.1.)isamaskingfunctionsuchthatthistermisonlyevaluatedforknownŽpixelsin,i.e.,thosepixelsthatresultfromtheconvolutionofwithproperlyesti-matedpixels,whichformabandaroundeachedgepoint,asdescribedinSection4.1.Theremainingnegativeloglikelihoodterm,,mod-elspriorassumptionsontheblurkernelandregularizesthesolution.Weuseasmoothnesspriorandanon-negativityconstraint.Thesmoothnesspriorpenalizeslargegradientsandthusbiaseskernelvaluestotakeonvaluessimilartotheirneighbors:controlstheweightofthesmoothnesspenalty,and=(2+1)nor-malizesforthekernelarea(isthekernelradius).Sincethekernelshouldsumtoone(asblurkernelsareenergyconserving)theindividualvaluesdecreasewithincreased.Thisfactorisneededtokeeptherelativemagnitudeofkernelgradientvaluesonparwiththedatatermvaluesre-gardlessofkernelsize.Weminimizingthefollowingerrorfunction:subjectto,tosolveforthePSFusingnon-negativelinearleastsquaresusingaprojectivegradientNewtonsmethod.WecurrentlyestimatethenoiselevelusingatechniquesimilartothatofLiuetal.[9],andwehaveem-piricallyfoundtoworkwell.5.1.ComputingaSuper-ResolvedPSFBytakingadvantageofsub-pixeledgedetectionforblindpredictionandsub-pixelcornerdetectionfornon-blindprediction,wecanestimateasuper-resolvedblurker-nelbypredictingasharpimageatahigherresolutionthantheobservedimage.Fortheblindmethod,intheprocessofestimatingthesharpimage,itisnecessarytorasterizethepredictedsharpedge-pro“lebackontoapixelgrid.Byrasterizingthesub-pixelsharp-edgepro“leontoanup-sampledgrid,wecanes-timateasuper-resolvedsharpimage.Inaddition,attheac-tualidenti“ededgelocation(asbefore),thepixelcolorisaweightedaverageoftheminimumandmaximum,wheretheweightingre”ectsthesub-pixeledgelocationonthegrid.Forthenon-blindmethod,wealsomustrasterizethegridpatternatasomedesiredresolution.Sincewedetectcornersatsub-pixelprecision,thegeometricalignmentiscomputedwithsub-pixelprecision.Usingthemathemati-caldescriptionofourgrid,wecanchooseanyupsampledresolutionwhenrasterizingthepredictedsharpimage.Wealsoperformanti-aliasing,asdescribedinSection4.2.TosolveforthePSFusingthesuper-resolvedpredictedsharpimageandtheobserved(vectorized)blurryim-,wemodifyEquation4toincludeadown-samplingfunctionaccordingtoourimagemodel(Equation1).Wetobesuper-resolvedsharpimageblurredbythesuper-resolvedkernel,whereisthematrixformof.Equation4isthen degrees45degreesdegrees Ground Ground Ground Pixels Ground Ground Ground Pixels Figure4.RecoveringBlurKernelsofDifferentSizesandOrientations.WeconvolvedthesharporiginalversionoftheimageshowninFigure1withkernelsof13and17pixelsforthreedifferentorientations.Eachsetisasidebysidecomparisonsofthegroundtruth(left),ourrecoveredkernel(middle),andtheresultofrunningFergusetal.s[3]method(right).haveleftoutthemaskingfunctionforreadability).isamatrixre”ectingthedown-samplingfunction:m,nsm,sn5.2.ComputingaSpatiallyVaryingPSFComputingaspatiallyvaryingPSFisstraightforwardgivenourformulation„wesimplyperformtheMAPesti-mationprocessdescribedintheprevioussectionforsub-windowsoftheimage.Theprocessoperatesonanysizesub-windowaslongasenoughedgesatdifferentorienta-tionsarepresentinthatwindow.Inthelimit,wecouldcom-puteaPSFforeverypixelusingslidingwindows.Wehavefound,inpractice,thatsuchadensesolutionisnotneces-sary,asthePSFtendstovaryspatiallyrelativelyslowly.Ourmethodrequiresenoughedgestobepresentatmostorientations.Whenusingtheentireimage,thisisnotusu-allyanissue;however,whenusingsmallerwindows,theedgecontentmayunder-constrainthePSFsolution.Wehaveasimpletestthatavoidsthisproblem.Weensurethat(a)thenumberofvalidpixelsinthemaskdescribedinEquation4isgreaterthanthenumberofunknownsinthekernel,and(b)wecomputeahistogramof10degreebinsofthedetectededgesorientationsandensurethateachbincontainsatleast100edges.Whenthischeckfails,wedonotcomputeakernelforthatwindow.6.ChromaticAberrationIntheprevioussections,wedidnotexplicitlyaddresssolvingforPSFsforcolorimages.Tohandlecolor,onecouldconverttheimagetograyscale.Inmanycasesthisissuf“cient;however,itismoreaccuratetosolveforaPSFforeachcolorchannel.Thisneedariseswhenchromaticaberrationeffectsareapparent.Duetothewavelength-dependentvariationofthein-dexofrefractionofglass,thefocallengthofalensvariescontinuallywithwavelength.Thispropertycauseslongi-tudinalchromaticaberration(blur/shiftsalongtheopticalaxis),whichimpliesthatthefocaldepth,andthusamountofdefocus,iswavelengthdependent.Italsocauseslateralchromaticaberration(blur/shiftsperpendiculartotheopti-calaxis).WereferthereadertothepaperbyKang[5]foramoredetaileddiscussionoftheseartifacts.BysolvingforaPSFpercolorchannel,wecanmodelthelongitudinalaberrations;weuseaper-colorchannelradialdistortioncorrectiontohandlethelateraldistortions.Wecorrectforlateraldistortionsby“rstperformingedgedetec-tiononeachcolorchannelindependentlyandonlykeepingedgesthataredetectedwithin5pixelsofeachotherinR,G,andB.WethencomputearadialcorrectiontoaligntheRandBedgestotheGedgesandthenperformblindsharpimageprediction.Tocorrectforanyresidualradialshifts,weusethegreenedgelocationsforallcolorchannelssothatallcolorbandshavesharpedgespredictedatthesamelocations.Onecouldperformthislaststepwithoutcorrectingradialdistortion“rstandallowtheshiftstobeentirelymodeledwithinthePSF;however,wehavefoundthetwostageapproachisbet-ter,asitremovessomeaberrationartifactsevenwhenthereisnotenoughedgeinformationtocomputeaPSF,andbyremovingthemajorityoftheshift“rst,wecansolveforsmallerkernels.IfwehaveaccesstoRAWcameraimages,wecancom-putemoreaccurateper-channelPSFsbyaccountingfortheBayerpatternsamplingduringPSFcomputationinsteadofusingthedemosaickedcolorvalues.WesolveforaPSFattheoriginalimageresolution,whichis2xtheresolutionforeachcolorchannelandusethepointsamplingfunctiondis-cussedinSection3,wherethesamplingisshiftedaccordingtotheappropriateBayersamplelocation.7.ResultsTovalidateourblindpredictionmethod,wesyntheticallyblurredasharpimagewithorientedGaussiankernelsof13and17pixelsindiameterforthreedifferentorientations,addedGaussianwhitenoisewithstandard-deviation0.01(where0=blackand1=white),andthenestimatedtheblurkernelusingourblindmethod.Figure4,showsacompari-sonofthegroundtruthkernels,ourrecoveredkernels,andtheresultofrunningFergusetal.smethod.Ourblindalgo-rithmrecoversthesizeandshapeofeachkernelaccurately. (a) (b) (c) (d) (e) Figure5.DefocusandSlightMotion-Blur.(a)Theoriginalblurredimageand(b)thedeconvolvedoutputusingourrecoveredPSF.(c…d)Zoomed-inversionsoftheoriginalanddeconvolvedimagerespectively.(e)ThekernelrecoveredusingthemethodofFergusetal.[3]and(f)ourrecoveredkernel.InFigures5and7,weshowresultsforestimatingker-nelsforimageswithreal,unknownblurs,wherethereisbothdefocusandcameramotionblur.Ourmethodpre-dictsslightlyasymmetricdisk-likekernelsthatareconsis-tentwithdefocusandslightmotionblur.Toqualitativelyvalidatethesekernels,wedeconvolvetheinputimagesusingtheLucy-Richardsonalgorithm.Wechosethisoverothermethodsasitproducesresultswithagoodbalanceofsharpnessandnoisereduction.Further-more,themethodislessforgivingthansomenewermeth-ods,whichallowsforbettervalidation.(Deconvolutionwithaincorrectkernelleadstoincreasedringingartifacts,asshowninFigure6).Ourresultingdeconvolvedimagesaresigni“cantlysharperandshowrelativelyminimalring-ingartifacts,whichindicatesthatthekernelsareaccurate.InFigure5,wealsocompareourrecoveredkerneltoaresultfromrunningFergusetal.scode.Thekernelob-tainedbytheirmethodhasmorenoisethanours,doesnothaveashapeconsistentwithdefocusblur,andthesizeofthenon-zeroareaofthekerneldoesnotmatchtheamountofblurseenintheinputimage.Fergusetal.smethodtook21minutes,whileourstook2.5secondsfortheorigi-nalresolutionand9.5secondsat2xsuper-resolution.Ourmethodissigni“cantlyfasterasitsrunningtimescaleswiththenumberofedgesandkernelsize,whiletheFergusetalmethodisamulti-resolutionapproachwhosespeedscalesimagesizeandkernelsize.Ourmethodisacouplesecondsfasterwhenusingregularleast-squaresinsteadofanon-negativeversion;however,moresmoothingisneededtosuppresslargenegativevalues.Thusweprefertoenforcenon-negativityasitproducessharperPSFs.Figure8displaysanimagewithcameramotionblur.Ourrecoveredkernelcorrectlyshowsthediagonalmotionblurthatisapparentintheinputimage.Thedeconvolvedimageismuchsharperwithminimalringing. Figure6.KernelSizeandOrientation.Imagedeconvolvedwith(left)ourkernel,(middle)ourkernelscaledlarger,and(right)ourkernelrotatedby.Themiddleandrightimageshavemoreringing(mostapparentatthebottomofthewordLeicesterŽ).InFigure9,weshowsuper-resolutionresultswherewehavetakenasharpimage,bicubicallydown-sampleditby4x,andthensolvedfora4xsuper-resolvedkernelfromthedown-sampledinput.Wecomparetheoriginalfullresolu-tionimagetoabicubicallyup-sampledversionofthelow-resolutionimageandtotheupsampledimagedeconvolvedwithourrecoveredkernel.Thedeconvolutionresultsshowasharpeningandrecoveryofhigh-frequencytexturethatisconsistentwiththefullresolutionimages.Figure10showsresultsforourcalibrationgridcapturedwithan11mega-pixelCanon1DsusingaCanonEF28-200mmf3.5-5.6lensattwoaperturesandfocallengths.Foreachimage,wecomputedspatiallyvaryingPSFsbycom-putingkernelsfornon-overlapping220-pixel(thesizeofonegridtile)windowsacrosstheimageat2xresolution,i.e.,twotimestheBayersamplingresolution.EachPSFisdisplayedaccordingtothelocationofitscorrespondingimagewindow.TherecoveredPSFsshowsomeinterestingproperties.ThePSFsshouldbeimagesoftheaperture,andsomekernelsdoshowtheshapeoftheaperture,whichweknowfromthelensspeci“cationstohave6blades.They Figure7.DefocusandSlightMotion-Blur.(topleft)Theoriginalblurredimageand(topright)thedeconvolvedoutputwiththere-coveredkerneldisplayedinthetoprightoftheimage(thekernelhasbeenenlargedby10xfordisplay).(bottomrow)Zoomed-inversionsoftheoriginalanddeconvolvedimage,respectively. Figure8.MotionBlur.(toprow)Theoriginalblurredimage(left)andthedeconvolvedoutput(right)withtherecoveredkerneldis-playedinthetoprightoftheimage(thekernelhasbeenenlargedby10xfordisplay).(bottomrow)Zoomed-inversionsoftheorig-inalanddeconvolvedimage,respectively.alsoshowdonutŽartifactsthatcanoccuratsomesettingswithlower-qualitylenses.Perspectivedistortionacrosstheimageplaneandvignetting(clippingoftheaperture)bythelensbarrelarealsovisible.Forcomparisonweimagedback-litpinholesatthesamecamerasettings.Imagingpin-holestomeasurePSFshassomeinherentproblemsduetothepinholeactuallybeingadiskandnotanin“nitesimalpointandduetodiffraction;however,theseimagesvalidateourrecoveredPSFs.Wealsoacquiredaverysharplyfocusedimage,sothatwecouldmeasuresub-pixelblur.Figure11showsanimageofourgridfroma6mega-pixelCanon1D,usingahigh-qualityCanonEF135mmf/2Llens.WeshowrecoveredPSFsat1x,2x,8x,and16xsub-pixelsampling.ThePSFsusinghighersub-pixelresolutionshowaninterestingstruc-turethatresultsfromacombinationofdiffraction,lensim-perfections,andsensoranti-aliasingandsampling.Figure12showsaresultforperformingblindchromatic Figure9.4xSuper-Resolution.(left)Theoriginalimageandzoom-in,(middle)theoriginalimagebi-cubicallydownsampledandre-upsampledby4xandzoom-in,(right)theupsampledimagedeconvolvedusingtherecovered4xsuper-resolvedkernel(dis-playedinthetoprightoftheimage…thekernelhasbeenenlargedby10xfordisplay)andazoom-inonthebottom. (a)150mmf5.6 (b)145mmf10Figure10.DifferentAperturesandFocalLengths.(“rstrow)Croppedportionsoftheobservedblurredimages,(secondrow)recoveredspatiallyvaryingPSFs(greenchannelonly),(thirdrow)imagesofpinholesatthesamedepthsandsettings,and(fourthrow)ourrecoveredPSFsconvolvedwithadiskthesizeofthepin-hole.For(a)eachPSFispixelsand(b)theyarepixels.ThePSFsre”ecttheshapeoftheapertureandshowper-spectivedistortionandvignettingacrosstheimageplane.aberrationcorrectionforaJPEGimagefromaCanonS60usinga5.8mmfocallengthatf8.AfterperformingradialdistortioncorrectionandpiecewisedeconvolutionusingthespatiallyvaryingPSF,theaberrationartifactsaresigni“-cantlyreduced.Figure13showschromaticaberrationcor-rectionforournon-blindmethod.Toviewfullresolutionversionsofourresults,includingadditionalexamples,visithttp://vision.ucsd.edu/kriegman- 8.DiscussionandFutureWorkWehaveshownhowtorecoverspatiallyvaryingPSFsatsub-pixelprecisionthatcaptureblurduetomotion,defo-cus,andintrinsiccameraproperties.Ourmethodisfast,straightforwardtoimplement,andpredictskernelsaccu-ratelyforawidevarietyofimages.Nevertheless,ourmethoddoeshavesomelimitations,andthereareseveralavenuesforfuturework.Theprimarylimitationofourmethodisthatwecanonlysolveforkernelswithasinglepeak.Thislimitationisduetoourrelianceonanedgedetectorto“ndasinglelocationforeveryblurrededge.Inthecaseofamulti-peakedker-nel,ourmethodwillincorrectlyinterprettheghostŽcopies (a) (b) (c) (d) Figure12.BlindChromaticAberration.(a)RecoveredspatiallyvaryingPSFsforred,green,andblueshownasacolorimage.PSFsareonlycomputedwherethereareenoughedgesobserved.(b)Theoriginalimage,(c)afterradialcorrectionanddeconvolutiontheaberrationsaresigni“cantlyreduced,and(d…e)zoomed-inversionsandintensitypro“lesfor(b…c).ofedgesasindependentedges.Whilewehaveshownthatsingle-peakedkernelsmodelmanycommonlyoccur-ringcasesofblur,wewouldliketoextendourmethodtohandlemulti-modalkernels.Oneoptionistogroupeachstrongeredgewithitsweakerghostedgesusingcontourmatching.Oncetheghostedgesareidenti“ed,wecouldperformsharpedgepredictiononlyfortheprimaryedges.Aseachsharpedgepro“legivesinformationaboutara-dialsliceofthePSF,itisnecessaryforanimage,orimagewindow,tohaveedges(oratleasthigh-frequencycontent)atmostorientations.Ifsomeorientationsarelacking,ourregularizationtermscancompensate;however,thereisabreakingpoint,andtheremaynotalwaysbeenoughedgeinformationtoproperlycomputeaPSF.Inthesecases,alowparameterkernelmodelmaybemoreappropriate,butoursharpimagepredictioncouldstillbebeusedtoimprovemoretraditionalparametrickernelestimationprocedures.Wealsoplantotryusingrobustleastsquarestocompensateforerroneousedgedetectionsorpro“le“ts.Lastly,wewouldliketocharacterizemorelensesandcameras.Wewouldliketobuildadatabasethatthevisionandphotographycommunitycouldcontributetobyusingourpatternandcodetotaketheirownmeasurements.9.AcknowledgementsWewouldliketothanktheanonymousreviewersfortheircomments.Thisworkwaspartiallycompletedwhilethe“rstauthorwasaninternatMicrosoftResearch.References[1]P.D.BurnsandD.Williams.Usingslantededgeanalysisforcolorregistrationmeasurement.InIS&TPICSConference (a) 2 1 0 1 2 2 1 0 1 2 (b) 2 1 0 1 2 2 1 0 1 2 (c) 2 1 0 1 2 2 1 0 1 2 (d) 2 1 0 1 2 2 1 0 1 2 Figure11.Sub-PixelPSFs.(a)Croppedsectionofasharpimageofourgrid,(b)PSF(greenchannelonly)attheBayerresolution(1x),(b)2x,(c),8x,and(d)16xsub-pixelsampling.Thesub-pixelPSFsshowblurresultingfromacombinationofdiffraction,lensimperfections,andsensoranti-aliasingandsampling.pages51…53.SocietyforImagingScienceandTechnology,[2]S.Cho,Y.Matsushita,andS.Lee.Removingnon-uniformmotionblurfromimages.InICCV2007,pages1…8,14-21Oct.2007.[3]R.Fegusetal.Removingcamerashakefromasinglepho-ACMTransactionsonGraphics,27(3):787…794,August2006.[4]J.Jia.Singleimagemotiondeblurringusingtransparency.CVPR07,pages1…8,17-22June2007.[5]S.B.Kang.Automaticremovalofchromaticaberrationfromasingleimage.InCVPR07,pages1…8,17-22June2007.[6]D.KundurandD.Hatzinakos.Blindimagedeconvolution.SPMag,13(3):43…64,May1996.[7]A.Levin.Blindmotiondeblurringusingimagestatistics.InAdvancesinNeuralInformationProcessingSystems.MITPress,2006.[8]E.Levy,D.Peles,M.Opher-Lipson,andS.Lipson.Mod-ulationtransferfunctionofalensmeasuredwitharandomtargetmethod.AppliedOptics,38:679…683,Feb.1999.[9]C.Liu,W.T.Freeman,R.Szeliski,andS.B.Kang.Noiseestimationfromasingleimage.InCVPR06,volume2,pages901…908,NewYork,NY,June2006.[10]S.Nayar,M.Watanabe,andM.Noguchi.Real-timefocusrangesensor.InFifthInternationalConferenceonCom-puterVision(ICCV95),pages995…1001,Cambridge,Mas-sachusetts,June1995.[11]S.E.Reichenbach,S.K.Park,andR.Narayanswamy.Char-acterizingdigitalimageacquisitiondevices.OpticalEngi-,30(2):170…177,February1991.[12]Q.Shan,W.Xiong,andJ.Jia.Rotationalmotiondeblurringofarigidobjectfromasingleimage.InICCV2007,pages1…8,14-21Oct.2007. Figure13.ChromaticAberration.(left)TherecoveredspatiallyvaryingPSFsforred,green,andblueshownasacolorimage.Theredandbluefringingisre”ectedinthePSFimageandthePSFsarelargertowardstheedgeoftheimageandspreadalongthedirectionorthogonaltotheopticalaxis.(middle)Zoom-inontheinputimage.(right)Afterradialcorrectionanddeconvolutiontheaberrationsaresigni“cantlyreduced.