Narasimhan and Shree K Nayar Computer Science Dept Columbia University New York NY 10027 Email srinivas nayar cscolumbiaedu Abstract Images of scenes acquired in bad weather have poor con trasts and colors It is known that the degradation of image q ID: 26451
Download Pdf The PPT/PDF document "Interactive DeWeathering of an Image usi..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Interactive(De)WeatheringofanImageusingPhysicalModelsSrinivasaG.NarasimhanandShreeK.NayarComputerScienceDept.,ColumbiaUniversity,NewYork,NY10027E-mail:srinivas,nayar@cs.columbia.eduAbstractImagesofscenesacquiredinbadweatherhavepoorcon-trastsandcolors.Itisknownthatthedegradationofimagequalityduetobadweatherisexponentialinthedepthsofthescenepoints.Therefore,restoringscenecolorsandcontrastsfromasingleimageofthesceneisinherentlyunder-constrained.Recently,ithasbeenshownthat RGB O FigureDichromaticatmosphericscatteringmodel.Theofascenepointonafoggyorhazyday,isalinearcombinationofthedirectionofdirecttransmission(clearday)color,andthedirectionofairlight(fogorhaze)color. Thedichromaticatmosphericscatteringmodel[3]statesthatthecolorofascenepointinfogorhaze,observedbyacolorcamera,isgivenbyalinearcombinationofthedirectionofairlight(fogorhaze)color,andtheofthecolorofthescenepointasseenonaclearday(seeillustrationingure1).Mathematically,(1)istheskybrightness,isradianceofthescenepointonaclearday,isthescatteringcoecientoftheatmosphereandisthedepthofthescenepoint.Notethatthedichromaticmodelassumesthatthescat-teringcoecientisthesameforallthecolorchan-nels.Also,observethattheproblemofdeweatheringanimage,bycomputingcleardaycolorssolelyfromobservedcolorvectorsisseverelyunder-constrained.Thecontrastormonochromemodel[4]givesamathe-maticalexpressionfortheintensityofascenepointinbadweather,asrecordedbyamonochromecamera:(2)Ascanbeseenfromboththemodels,thecolorandcontrastofascenepointdegradeexponentiallywithitsdepthfromtheobserver.Hence,traditionalspacein-varianttechniquesforcolorandcontrastenhancementcannotbeusedtosatisfactorilydeweatherimages.Inthefollowingsections,wedescribeourinteractivetech-niquesforimagedeweatheringusingsimpleinputsfromtheuser. Notethatthesemodelsarebasedonsingle-scatteringandhencearenotvalidforturbulenceandaerosolblurringandscat-teringbypollutants.Iftheatmosphereisnon-homogeneousalongthelineofsight,willbeafunctionofdepth.Thenscaledisreplacedbyopticalthicknessdx.3DichromaticColorTransferConsiderascenewithpointsatdierentdepthsbutwithsimilarcleardaycolors.Forinstance,treesatdier-entdistances,orbuildingsatdierentdepths,mayhavesimilarcolordirections(althoughtheirmagnitudesmayvary)onaclearday.Inthisscenario,thecolorsofnearscenepointsarelesscorruptedbybadweatherascom-paredtothedistantscenepoints.Wenowdescribeanalgorithmtotransfercolorsfromnearbyregionstore-placecolorsofregionsthataremosteectedbybadweather,inaphysicallyconsistentmanner.Inotherwords,weimposeconstraintsbasedonthedichromaticmodel(1)toselectcolorsofnearscenepointstoreplacecolorsoffarscenepoints.3.1InteractiveStepOnlytwomanualinputsarenecessaryforthecolortransferalgorithm.First,weselectanearbygoodregionintheimage,wherecolorsarenotcorrupted(or,minimallyaltered)bybadweather,asshownbythewhiterectangleingure2(a).Then,wemarkaregion(say,sky)thatmostresemblesthecolorofairlight,asshownbytheblackrectangleingure2(a).Theaver-agecolorwithinthisregioniscomputedtoestimatetheofairlightcolor.3.2AutomatedStepForeachpixel,withcolor,intheweathereectedre-gion,wesearchforthebestmatchingcolorinthegoodregion.Thesearchisrestrictedtoasetofpixelsinthegoodregionthatsatisfythedichromaticplanarity(1),Fromthisset,wechooseapixelwhosecolorisfar-thest(intermsofangle)fromthefogcolor,usingInordertocomputethemagnitudeofthecolorusedtoreplacethepixel,weusethedichromaticmodel(1)todecomposethescenecolorintotwocomponents:Finally,wereplacethecolorofthepixelbythedeweatheredcolor,Notethattheambiguitiesinthedichromaticmodelarebrokenduetothepresenceofsimilarcoloredscenepointsatdierentdistances.Thisalgorithmdoesnotrequireanyinformationregard-ingscenedepthsoratmosphericconditions.Further,it Ifsucharegiondoesnotexistintheimage,thentheusermayprovidethehueoftheskyandassumetheskyintensitytobethemaximumintensityintheimage.Anotherwayofcomputingthecolorofairlightisbyintersectingdichromaticplanesoftwodierentuserprovidedscenecolors[3]. (a)Input:Mistyimage(b)Output:Colorcorrectedimage UserselectedGoodcolorregionUserselectedAirlightcolor FigureColorcorrectionbydichromaticcolortransfer.(a)Inputmistyimageconsistingofgreenbushesatdierentdistances.Aregionofgoodcolorismarkedinthewhiterectangle.Aregionthatmostresemblesthecolorofmistismarkedintheblackrectangle.(b)Colorsfromtheneargoodregionaretransferedtofartherregions.Noticethebluishcolorsofthefartherbushesreplacedbygreenishcolors. doesnotassumehomogeneityoftheatmosphereovertheentireeldofview.Theresultofapplyingthismethodisshowningure2(b).Noticethesignicantchangeinthecolorsofthefarbushes.4DeweatheringusingDepthHeuristicsAlimitationofthecolortransfermethodisthatallcol-orsintheweathereectedregionmaynothavecorre-spondingcolorsinthegoodcolorregion.Inthissec-tion,wedescribedeweatheringusingheuristicsonscenedepths.Notethatsubtleweathereectswithinsmalldepthrangesarenotcapturedbyacamerawithlimiteddynamicrange(say,8bits).Therefore,precisedistancesarenotrequiredforeectivedeweathering.Moreover,inmanycases,itmaybepossibletoinputapproximatetrendsinthedepthsofscenepoints(say,thedirectionofincreasingdepths).Forinstance,ascenewithastreetalongtheviewingdirectioniscommoninsurveillanceortrackingscenarios(seegure6).Thedeweatheringal-gorithmisdetailedbelow.4.1InteractiveStepWeselectaregionoftheskytoobtaintheskyintensity(andskycolordirection,iftheinputisacolorim-age).Then,thedepthtrendisinteractivelyspeciedinthefollowingmanner.First,weinputtheapproxi-matelocationofavanishingpointalongthedirectionofincreasingdistanceintheimage(seeredcircleing-ure4).Thedistancesofthescenepointsareinverselyrelatedtotheirimagedistancestothevanishingpoint.Next,weinputtheapproximateminimumandmaxi-mumdistancesandinterpolatedistances(say,usingalinearorquadraticfunction)forpointsinbetween.Forillustrationpurposes,weusedminmin(3)1)isthefractionalimagedistancefromapixeltothevanishingpoint.ForandforminTheresultingdepthtrendisshowningure3(a).4.2AutomatedStepConsiderthemodelgiveninequation2.Ateverypixel,thedepthestimateisknown,andtheskybrightnessismeasured.Generally,theatmosphereconditionremainsconstant(orvariesslowly)oversmalldistancerangesandeldsofviewthatarerelevanttocomputervisionapplications.Ifweassumehomogeneityoftheat-mosphere,thenthescatteringcoecientisconstantforallpixelsintheimage.Then,notethatdierentvaluesofthescatteringcoecientproducetheeectsofdierentdensitiesofbadweather(moderate,heavy,etc.).Thus,bycontinuouslychanging(imagineasliderinAdobePhotoshop),wecanprogressivelyes-timatethecleardayradiancesateachpixelas,s,EE(1ed)]ed.(4)Similarly,notethatthedichromaticmodel(equation1)canbeusedtorestorecolorsinanRGBimage.There-fore,whilethecolortransfermethodcanbeappliedonlytocolorimages,thismethodcanbeappliedtobothcolorandgray-scaleimages.Inthiscase,thehomogeneityoftheatmospherebreakstheambiguityindeweatheringanTheresultsshowningures4,5and6illus-tratethatapproximatedepthinformationcanbeusedeectivelyforimagedeweathering.5RestorationusingPlanarDepthSegmentsIntheprevioussection,wedescribedaninteractivetech-niquewheredepthtrendscanbefollowed.However, 0.5 1 1.5 2 2.5 Sky region 1 1.5 2 2.5 3 1 1.5 2 2.5 3 (a)(b)(c)FigureDepthheuristicsusedtodeweatherimagesshowningures4,5and6respectively.Thevanishingpointcorrespondingtothedirectionofincreasingdistancesismarked.Approximateminimumandmaximumdistancesareinputtothealgorithmandtheintermediatedistancesareinterpolated.Thedepthsarenotusedforskyregions(emptyspaces). (a)Input:Mistyimage(c)Zoomed-inregionsof(a)(b)Output:Deweatheringtodifferentextents(bychoosingdifferentvaluesfor)(d)Contrastandcolorrestorationinzoomed-inregions. UserselectedVanishingPointUserselectedSkyBrightness FigureRestoringcleardayscenecolorsusingdepthheuristics.(a)Inputimagecapturedinmist.Thecolorsandcontrastsofscenepoints,especiallyinfartherregions,arecorruptedseverely.(b)Twoimagesillustratingdierentamountsofmistremovedfromtheimagein(a).Theseimageswerecomputedusingthedepthtrendshowningure3(a).(c)Zoomedinregionsselectedfrom(a)atdierentdepthsshowingdierentamountsofmist.(d)Correspondingzoomedinregionsofthedeweatheredimages.Noticethesignicantcolorandcontrastenhancement. (a)Input:Mistyimage(c)Zoomed-inregionsof(a)(b)Output:Deweatheringtodifferentextents(bychoosingdifferentvaluesfor)(d)Contrastandcolorrestorationinzoomed-inregions. UserselectedVanishingPointUserselectedSkyBrightnessFigureRestoringcleardayscenecolorsusingdepthheuristics.(a)Inputimagecapturedinmist.Thecolorsandcontrastsofscenepoints,especiallyinfartherregions,arecorruptedseverely.(b)Twoimagesillustratingdierentamountsofmistremovedfromtheimagein(a).Theseimageswerecomputedusingthedepthtrendshowningure3(b).(c)Zoomedinregionsselectedfrom(a)atdierentdepthsshowingdierentamountsofmist.(d)Correspondingzoomedinregionsofthedeweatheredimages.Noticethesignicantcolorandcontrastenhancement. urbansceneswithstrongdepthdiscontinuitiesandse-vereocclusions(inducedbydierentbuildings)arenotsuitableforthepreviousapproachwheredepthtrendsweresmoothlyinterpolated.Insuchcases,itisbet-tertoprovidearoughdepthsegmentationofthescene.Recallthatprecisedepthinformationisnotneededtodeweatherimages.Forinstance,thebrightnesslevelsoffogforafrontalplanarsurfaceareapproximatelyequaltothebrightnesslevelsforacurvedsurfaceatthesamedistance.Thus,planardepthsegmentsshouldsucefordeweatheringinurbanscenes(seegure7(b)).Thedeweatheringalgorithmissimilartotheonepre-sentedintheprevioussection.Thedepths,however,areprovidedasapproximateplanes.Inourexperiments,weusedimagesfromtheColumbiaWeatherandIllumina-tionDatabase(WILD)[2].Orthographicdepthswereobtainedfromsatelliteorthophotos(seegure7(b)).Onceagain,theskybrightnesswasmeasuredbyse-lectingaregionofthesky.Theimagescanbedeweath-eredbycomputingcleardaysceneradiancesorcolorsdependingonwhetheragray-scaleoracolorim-ageisinputtothealgorithm.Resultsofdeweatheringamistysceneisshowningure7(c).Noticethesignif-icantincreaseincontrastsofthescenepointsatvariousdepths.Insummary,theaboveresultsdemonstratethatweathereectscanbesucientlyremovedfromimagesevenwhenonlyapproximatedepthsareknown. (a)Input:Foggyimage(c)Zoomed-inregionsof(a)(b)Output:Deweatheringtodifferentextents(bychoosingdifferentvaluesfor)(d)Contrastrestorationinzoomed-inregions. UserselectedVanishingPointUserselectedSkyBrightnessFigureRestoringcleardayscenecontrastsusingdepthheuristics.(a)Inputgray-scaleimagecapturedinfog.Thecontrastsofscenepoints,especiallyinfartherregions,aredegradedseverely.(b)Twoimagesillustratingdierentamountsoffogremovedfromtheimagein(a).Theseimageswerecomputedusingthedepthtrendshowningure3(c).(c)Zoomedinregionsselectedfrom(a)atdierentdepthsshowingdierentamountsoffog.(d)Correspondingzoomedinregionsofthedeweatheredimages.Noticethesignicantcontrastenhancement. (a)Input:Mistyimage (b)Input:Planardepthsegments (c)Output:Contrastenhancedimage.FigureContrastrestorationusingplanardepthseg-ments.(a)Amistyimage.(b)Planardepthsobtainedfromsatelliteorthophotos.(c)Outputoftherestorationalgo-rithm.Noticetheincreaseinclarityinthefartherbuild-ings.TheinputimageandthedepthswereobtainedfromtheColumbiaWeatherandIlluminationdatabase(WILD). 6AddingWeatherEectstoImagesThusfarwepresentedalgorithmstoremoveweatheref-fectsfromasingleimage.However,incomputergraph-icsapplications,itmayberequiredtoaddatmosphericeectstoimages.Wenowdemonstratetheadditionofweathereectstoasinglephotographusingamanuallyprovidedapproximatedepthmap.Weaddtwotypesofscatteringeectstoanimage.Notethattheintensityofalightraydiminisheswhenitisscatteredbyanatmosphericparticle.Hence,thenumberofscatteringeventseachlightraycanundergoandstillbedetectablebythecamera,issmall.Hence,singlescatteringmodels(equations1and2)sucetoaddweathereectstoimagesofsurfacesinthescene.However,weobservedistinctglowsaroundbrightlightsourcesinthescene.Theseglowsareduetothemul-tiplescatteringoflightintheatmosphere.Recently,NarasimhanandNayar[5]developedananalyticmodelfortheglowaroundapointlightsource.Inthissec-tion,weusethismodelandpresentanalgorithmtoaddweathereectstoclearday(ornight)sceneswithbothlightsourcesandsurfaces.6.1InteractiveStepWemanuallyprovidedepthheuristicsandtheskyinten-sities,asdescribedintheprevioussections.Then,wesegmentthelightsourcesintheimage.Thisstepcanalsobedoneautomaticallyusingasimplethresholdingoperationonimageintensities.6.2AutomatedStepThisstepconsistsoftwopartscorrespondingtothead-ditionofsinglescatteringeectstosurfacesandmultiplescatteringeectstolightsources.Thepixelintensitiesofnon-lightsourceregionsofthescenecorrespondtothecleardaysurfaceradiances.Bysubstitutingdierentvaluesforthescatteringcoecientinequation(2),wesimulatedierentdensitiesofweathertotheregioncorrespondingtosurfaces.Toaddmultiplescatteringeects(glow)aroundeachlightsource,weperformthefollowingoperations.TheglowT,µ)aroundapointlightsourcewithradiantintensityisderivedin[5],T,µlog (5)isthecosineoftheraydirection,opticalthick-isaLegendrepolynomialoforder,andd,1]iscalledforwardscatteringpa-rameter.Dierentvaluesofgeneratedierenttypesofweatherconditions(fog,mist,haze,etc.).Theglowaroundapointsourcecanbeviewedsimplyasapointspreadfunction(PSF).Toaddaglowaroundanarealightsource,thecomputedPSFisconvolvedwiththeshapeofthelightsource.Fi-nally,wecombinethecontributionsofbothsingleandmultiplescattering(glows)torenderarealisticappear-anceofasceneinbadweather.Wedemonstratethistechniqueusingascenephotographedintheeveningcontainingabrightlamp(gure8).Thenalexpres-sionusedtorenderweathereectstothisimageis,is,ReT+E(1eT)](1µ)+I(T,µ(6) (a)Input:Cleareveningimage (b)Output:SingleScattering(minT=1.05) (c)Output:MildMist(q=0.8,minT=1.05) (d)Output:DenseFog(q=0.9,minT=2.0) Figure8:Addingweathereectstoaphotograph.(a)OriginalPhotograph.(b)Hazyimagerenderedusingsinglescattering(dichromaticmodel).(c)and(d)Dierentglowsareaddedtothelamp.Notethatmultiplescatteringeectsduetolightsourcesaresignicantascomparedtosinglescatteringeects. denotestheDiracdeltafunction.Twodif-ferentamountsofmistandfog(=[09],min=[10])wereaddedtotheimageingure8(a).Theseresultsareillustratedingures8(c)and(d).No-ticetherealisticglowingappearanceofthelampandthenaturalattenuationofthebackgroundsurfacebright-nesses.Also,comparetheresultwithonlysinglescat-teringshowningure8(b).7SummaryWepresentedsimpleinteractivetoolstoremoveweathereectsfrom,andaddweathereectsto,asingleimage.Ourmethodsdonotrequirepreciseinformationaboutthesceneortheweathercondition,anddonotrequirechangesinweatherconditionsbetweenimageacquisi-tions.Thethreemethodspresentedareeasy-to-useandcaneectivelyrestorecleardaycolorsandcontrastsfrompoorweatherimages.Wealsopresentedanalgorithmtoaddweathereects(bothsingleandmultiplescattering)toimages.Althoughweshowedfourspecicexamples,itisclearthatsuchsimpletechniquescanbeusedformostscenesthatoccurofteninpractice.References[1]N.S.Kopeika.ASystemEngineeringApproachtoImag-ing.SPIEPress,1998.[2]S.G.Narasimhan,C.Wang,andS.K.Nayar.Alltheimagesofanoutdoorscene.InProc.ECCV,2002.[3]S.G.NarasimhanandS.K.Nayar.Visionandtheatmo-sphere.IJCV,48(3):233 254,August2002.[4]S.G.NarasimhanandS.K.Nayar.Contrastrestorationofweatherdegradedimages.PAMI,25(6),June2003.[5]S.G.NarasimhanandS.K.Nayar.Sheddinglightontheweather.InProc.CVPR,2003.[6]J.P.OakleyandB.L.Satherley.Improvingimagequal-ityinpoorvisibilityconditionsusingaphysicalmodelfordegradation.IEEETrans.onImageProcessing,7,February1998.[7]Y.Y.Schechner,S.G.Narasimhan,andS.K.Nayar.In-stantdehazingofimagesusingpolarization.InProc.,2001.[8]K.TanandJ.P.Oakley.Physicsbasedapprachtocolorimageenhancementinpoorvisibilityconditions.JOSA,18(10):2460 2467,October2001.