Proc.IEEEComputerVisionandPatternRecognition(CVPR2008)1OvercomingVisua - PDF document

Proc.IEEEComputerVisionandPatternRecognition(CVPR2008)1OvercomingVisualReverberationsYaronDiamantandYoavY.SchechnerDept.ofElectricalEngineeringTechnion-IsraelInstituteofTechnologyHaifa32000,Israelyarondi@rafael.co.il,yoav@ee.technion.ac.ilAbstractAnimageacquiredthroughaglasswindowisasuper-positionoftwosources:ascenebehindthewindow,andareectionofasceneinfrontofthewindow.Lightraysinci-dentonthewindowarereectedbackandforthinsidetheglass.Suchinternalreectionsaffecttheradianceofbothsources:aspatialeffectiscreatedofdimmedandshiftedreplications.Ourworkgeneralizesthetreatmentoftrans-parentscenestodealwiththiseffect.First,wepresentaphysicalmodeloftheimageformation.Itturnsoutthateachofthetransmittedandreectedscenesundergoesaconvolutionwithaparticularpointspreadfunction(PSF),composedofdistinctdeltafunctions.Therefore,scenere-coveryinvolvesinversionofthesePSFs.Weanalyzethefundamentallimitationsfacedbyanyattempttosolvethisinverseproblem.Wethenpresentasolutionapproach.Theapproachisbasedondeconvolutionbylinearlteringandsimpleoptimization.Theinputtothealgorithmisapairofframes,takenthroughapolarizinglter.Themethodisdemonstratedexperimentally.1.IntroductionTransparentscenesposeachallengetocomputervi-sion.Theyexistinsetupshavingasemireectingwin-dow,whichsuperimposesthescenebehindthewindowtoareectedscene.Thiscreatesconfusingimages.Arangeofmethodsweredevelopedtoattackthisproblem,basedonmotion[4,9,18,25,27]stereo[22],polariza-tion[11,12,20,21],focus[19,23],illuminationmodula-tion[1]andimagepriors[14].Theysuccessfullydemon-stratedseparationofthescenes(layers).However,thepriorstudiesignoredaspatialeffectofinternalreectionsinsuchscenes,whichwedescribenext.Therefore,thepriormeth-odsarevalidonlyinthelimitwherethiseffectisnegligible.Fig.1demonstratesthiseffectinarealphotographtakenviaawindow.Inadditiontothesuperimposedscenes(toysofastarvs.atreeinthesun),ashiftedandweakerreplicaofthesunandtreeareclearlyseen.Thisiscausedbyinter-nalreectionsthattakeplaceinsideawindow.Inadditiontothatclearreplica,thereisalsoareplicaoftheotherscene(star).Additionalhigherorderreplicasexistforbothob- Figure1.Areal-worldframeS?acquiredthroughatransparentwindow.Inadditiontothesuperpositionoftwoscenes,noticethesecondaryreections(replications),e.g.,ofthesunandtree.Forclarity,pleaseviewthecolorimagesonthecomputermonitor.jects,butareoftentoodimtosee.Overall,theacquiredpho-tographcontainsasuperpositionnotonlyofthetwoorigi-nalscenes,butalsoofthosesamescenesdisplacedtovari-ousdistancesandindifferentpowers.Thepriorstudiesontransparentscenesdidnotaccountforthiseffect.There,themodelandalgorithmsfocusedonthelimitcase,inwhichthedisplacementbetweenthereplicaswasnegligible.Thisisnotavalidsituationingeneral.Inthisworkweexplicitlymodelthiseffectanddealwithit,hencegeneralizingthestudyoftransparentscenes.Op-ticalreectionscreatevisualspatialdisplacements.Thisisanalogoustotemporaldisplacementcreatedbyreectionsoftemporalsignalsofsoundandradio-frequency.Inanal-ogytothedisplacedreplicainourstudy,asoundreectioncreatesadelayedecho.Intheeldofacoustics[3],thisef-fectisgenerallyreferredtoasreverberations.Hence,weusethetermvisualreverberationstodescribetheeffectwedealwith.Asimilarechoeffectinradiofrequencyisaf-fectingreceivedtelevisionsignals,creatingshiftedreplicas.There,cancellationoftheeffectistermeddeghosting[8].Wemodeltheeffectbyphysics-basedexpressionsthataccountforpropertiesofopticalreections,includingpo-larization.Then,thepapertranslatesthemodeltothelanguageofsignal-processing.Itformulatestheeffectas &DPHUD 7 57 [D[LV\D[LV :LQGRZ 5 57 G¶ /U Figure2.PrimaryandsecondaryreectionsforLt[solid]andLr[dotted].Thedistancebetweentheemergingraysisd0.convolutionwithpointspreadfunctions(PSFs),whicharegiveninclosedform.Transparentlayersarethusamix-tureofconvolvedscenes.Themodelusedbypriorstudiesisaspecial(limit)case.Thefundamentallimitationsofthisrecoveryproblemareanalyzed.Then,wepresentaphysics-basedmethodforinvertingtheimageformationmodel.Itrecoverstheseparatescenes,whileovercomingthespatialeffectofvisualreverberations.Theparametersoftheprob-lemarederivedfromtherawimages.Themethodisbasedonframestakenwithapolarizinglter.1Itisdemonstratedinvariousexamples,includingrealexperiments.2.ImageFormationModelAcameraobservesasceneviaasemireectingwindow.Theobjectbehindthewindowistransmittedthroughthewindow,thusvariablesassociatedwithitaredenotedby`t'.Inaddition,thereisanobjectonthecamera-sideofthewindow.Itisreected,thusvariablesassociatedwithitaredenotedby`r'.Specically,Ltistheradiance2oftheobjectbehindthewindow,asmeasuredwhenthereisnowindow.Similarly,Lristheradianceofthereectedobject,asmeasuredifthewindowwasreplacedbyaperfectmirror.ConsiderFig.2.AlightrayfromtheobjectLrreachesthewindow.There,itundergoesaseriesofreectionsandrefractions.Theinternalreectionsinsidethewindowcre-ateaseriesofraysemergingfromthewindow.Sincethewindowisat,alltheraysareinthesameplane,termedtheplaneofincidence(POI).Thedistancebetweensuccessiveemergingrays(secondaryreections)isd0.Thepowerofsuccessivesecondaryreectionsrapidlytendstozero.AsimilareffectoccurswitharayfromtheobjectLt,asillus- 1Polarizationhasbeenusedinstudiesofvariouscomputervisionis-sues[2,6,7,16,17,24,26,28,29].2Thereisaconstantproportionbetweentheobjectradianceandtheimageirradiance.Theproportioncoefcientdoesnotdependonthescenesorontheparametersoftheproblem.Thiscoefcientdependsonlyonthecamera,andthuswedisregarditinthecontextofourproblem.tratedinFig.2.Priorstudiesneglectedtheshiftd0,hencespatialeffectsofsecondaryreectionswereignored.Thisassumptionwasvalidaslongasthewindowwasthinandviewedinlowspatialresolution,butitisnottrueingeneral.Reectionissensitivetopolarization.3ThepolarizationcomponentperpendiculartothePOIisdenotedby?,whilekdenotesthecomponentparalleltothePOI.Thereectancecoefcients[21]fromeachinterfaceofthewindowareR?=sin2(g) sin2(+g);Rk=tan2(g) tan2(+g);(1)whereistheangleofincidenceoflightonthewindow(relativetothesurfacenormal).Heregistheangleoftherefractedrayinsidetheglass.ThisangleisderivedusingSnell'slawsin(g)=sin()=n,wherenistheindexofrefractionofthewindow(fortypicalglass,n1:5).Thetransmittancecoefcients[21]oflightpassinganyoneofthewindowinterfacesareT?=1R?;Tk=1Rk:(2)Theimagecoordinatesare(x,y),wherexisthehorizontalcoordinate.HerethehorizontaldirectionintheimageisdenedastheprojectionofthePOIonthedetectorplane.ItisclearfromFig.2thatthesecondaryreectionscre-ateaspatialeffect.Eachobjectpointissensedsimultane-ouslyindifferentpixels,asitsenergyisdissipatedamongthedifferentreectionorders.HencethetransmittedsceneundergoesaconvolutionwithaparticularPSF:asseeninFig.2,thePSFofthetransmittedsceneishkt=T2k[(x)+R2k(xd)+R4k(x2d):::];(3)whenmeasuringonlythepolarizationcomponentparalleltothePOI,whileRkandTkaregiveninEqs.(1,2).HeredindicatesthedistancebetweensuccessivevisualechoesofLt,asreceivedbythecamera(inpixels).Itisgivenbyd=d0,whered0isthephysicaldistance(incentimeters)betweensecondaryreections,depictedinFig.2,andisthecameramagnication.Notethatinthismodel,eachobjectpointcorrespondstoaparallelsetofrays,whichinturncorrespondtoasetequallyinterspacedpixels.Thisisconsistentwithortho-graphicprojection,4whichweuseforsimplicity.Similarly,thePSFofthereectedsceneishkr=Rk[(x)+T2k(xd)+T2kR2k(x2d):::];(4)whenmeasuringonlytheparallelpolarizationcomponent.Theperpendicularcomponentsalsoundergoconvolutions. 3Ref.[15]analyzedthepolarizationininternalreections,inordertorecoverobjectshapes,includingaatslab.4Inperspectiveprojection,chiefraysarenotparallel,butcorrespondtoaslightlyfanningbeamemanatingfromtheobjectpoint.Inperspective,dandthePSFsarenotspatiallyinvariant.Theconsequencesofthenon-orthographicnatureofthecameraarediscussedin[10].2 Figure3.Simulatedobjects.[Left]Lt.[Right]Lr. Figure4.AsimulatedacquiredframeS?.Itexhibitsbothsec-ondaryreectionsandsuperpositionofareectedscenewithatransmittedscene.TheseparatescenesareshowninFig.3.ThecorrespondingPSFsh?randh?tarederivedanalo-gously,byusingR?andT?insteadofRk;TkinEqs.(3,4).Theacquiredimageintensityisalinearsuperpositionofthereectedandtransmittedscenes.InRefs.[11,21],thissuperpositionwaspointwise,sinceintheimagingcondi-tionsthere,spatialeffectswerenotseen.Incontrast,herethesuperpositionisofconvolvedscenes.Specically,letusmountapolarizinglteronthecamera,andorienttheltertopassonlytheparallelpolarizationcomponent.Assum-ingasinRefs.[5,11,12,21]thattheobjectsfLt;Lrgarelargelydepolarized,theacquiredframeisSk=Lt?hkt+Lr?hkr;(5)where?denotesconvolution.Similarly,orientingthepolar-izerperpendiculartothePOIyieldsS?=Lt?h?t+Lr?h?r:(6)Wecanillustratethisusingasimulation.Fig.3repre-sentstheoriginalobjectsLtandLr.Let=70oinaglasswindowandd=30pixels.Basedonthesevalues,there-ectance,transmittanceandPSFsaregiveninclosedformbytheexpressionsabove.Hencethesimulatedacquiredim-agesaregivenbyEqs.(5,6).Specically,Fig.4showsS?.3.FrequencyAnalysisofConditioningSec.2showedthatLtandLrareconvolvedbyPSFsintherawframesSkandS?.Therefore,inordertorestoreLt              )UHTXHQF\ W + )UHTXHQF\ U+ Figure5.Frequencyresponseofhrford=30pixelsand=27o.ThefrequencyisnormalizedbytheNyquistfrequency.andLrweneedtoperformdeconvolution.LetusexaminethefrequencyresponseofthesePSFs.ThePSFhktisgiveninEq.(3).Plottingitsfrequencyresponserevealsthattheresponseisratherat,andtypically1.Thesameappliestoh?t.Henceinverse-lteringofthesePSFsisexpectedtobestable,makingtherecoveryofLtwell-conditioned.ThesituationisdifferentforLr.ItsPSFhkrisgiveninEq.(4).ThecorrespondingfrequencyresponseisplottedinFig.5.Thisresponsehasvaluescloseto0.Thus,somefrequen-ciesoftheoriginalLraregreatlyattenuated.Consequently,recoveryofLrisill-conditionedaroundsuchfrequencies.4.RecoverywithDeconvolutionInthissectionwerecoverthesourceobjectsfLt;Lrg.Weshowthatthiscanbedoneusinglinearltersthatarede-rivedinclosed-form,andinverttheimageformationmodel.Then,wepointtoaproblemofsimplisticinversion.Finally,wedescribehowrecoverywasperformedbyusinpractice.4.1.LinearFilteringForthemoment,assumethatthevaluesoftheparametersdandareknown.TheirestimationisdescribedinSec.5.Basedondand,thecoefcientsR?,Rk,T?andTkarederived,thusthePSFshkt;hkr;h?t;h?rareknown(Sec.2).First,weeliminateLt.Itiseasytoshowthatthelters~hkt(x)=1 T2k[(x)R2k(xd)];(7)~h?t(x)=1 T2?[(x)R2?(xd)](8)satisfy~h?t?h?t=(x);~hkt?hkt=(x):(9)LetusconvolveEq.(5)withEq.(7).ThenbasedonEq.(9)Sk?~hkt=Lt+Lr?hkr?~hkt:(10)Similarly,convolvingEq.(6)withEq.(8)yieldsS??~h?t=Lt+Lr?h?r?~h?t:(11)3 SubtractingEq.(10)fromEq.(11)yieldsU=Lr?p;(12)whereU=S??~h?tSk?~hkt;(13)andph?r?~h?thkr?~hkt:(14)Eqs.(12)and(13)eliminateLt,thusisolatingLr.How-ever,Lrisstillnotrecovered,sinceitisconvolvedwithpinEq.(12).Hence,weneedtodeconvolvetheeffectofp.Inotherwords,weneedthefunctionvthatsatisesv?p=(x):(15)Thisfunctionhasasimpleanalyticalform,directlyinthespatialdomain.WedetailitinSec.4.2.ApplyingvonEq.(12)yieldsanestimate^Lr=U?v;(16)basedonEq.(15).PluggingEq.(13)inEq.(16)yields^Lr=S??q?r+Sk?qkr;(17)whereq?r=~h?t?v;qkr=~hkt?v:(18)WenowsolveforLt.UsingEqs.(13,16)in(11)yields^Lt=S??q?t+Sk?qkt;(19)whereq?t=~h?t~h?t?v?h?r?~h?t;qkt=~hkt?v?h?r?~h?t:(20)Eqs.(17)and(19)arethebasicrecoveryformulae.TheyshowthatfLt;Lrgcanberecoveredbyoperationoflinearltersgiveninclosedformdirectlyinthespatialdomain.TheseltersaregivenbyEqs.(7,8,18,20),andrelyonvthatisgiveninclosedforminSec.4.2.Toseetheformoftheselters,Fig.6plotsq?tcorrespondingto=27oandd=5pixels.Theltersqkt;q?r;qkrhaveasimilarform.4.2.TheFilterspandvWenowderivetheoperatorv.Denethecoefcientsa R? T2?Rk T2k!;b R?2R2? T2?Rk2R2k T2k!:(21)UsingEq.(21)inEqs.(3,4,7,8)and(14),itcanbeshownthatphasasimpleformp(x)a(x)+b(xd):(22)FromEqs.(15)and(22),vshouldsatisfyv?[a(x)+b(xd)]=(x):(23)            [AWTWGUA Figure6.Thelterq?tdenedinEq.(20).Here=27o,d=5pixels.Theeffectivewidthofq?tis?td,basedonEq.(28).Toderivev,weuseFourieranalysis.FollowingEq.(22),thefrequencyresponseofpisP(!)=a+bej!d:(24)where!isthespatialfrequency.FromEqs.(15,23),vistheinverselterofp,hencethefrequencyresponseofvisV(!)=1 P(!)=1 a
1 1b a
ej!d:(25)WenotethatbandadenedinEq.(21)satisfyjb=aj1.Hence,thelefthandsideof(25)isageometricseries,i.e.,V(!)=1 aX1m=0b amej!md:(26)TheinverseFouriertransformofEq.(26)isthesumv(x)=1 a(x)+X1m=1b am(xmd):(27)Thisresultcanbeveriedinthespatialdomain:convolvingEq.(27)withp(Eq.22),theresultcanbeshowntoyieldafunction,asrequiredbyEqs.(15,23).4.3.ProblemsCausedbyBoundaryConditionsTherecoveryproblemseriouslysuffersfromunknownboundaryconditions.Actually,thisproblemissoserious,thatsometimesitispreferablenottoattemptinversionofthereverberations,unlesstheeffectofunknownboundaryconditionsisaddressed.Eqs.(5)and(6)arevalidforim-agesthathaveinnitesupportinthex-axis.Inpractice,therawframeshaveanitesupportx2[0;W].However,theconvolutionkernelsq?t;qkt;q?randqkrrequirevaluesoutsidetheboundariesoftherawframes.Withoutlossofgenerality,letd�0.Then,thesupportofhkt;hkr;h?t;h?r(Eqs.3,4)residesonlyinx0,i.e.,thesearecausallters.Consequently,onlytheunknownvaluesinx0causeaproblemofboundaryconditions.Unknownvaluesatx&#x-5.1;䘘Warenotproblematic.4 ____  Figure7.Theeffectivewidths?t;kt;?r;krasmultiplesofd.Theplotcorrespondstoaglasswindow(n=1:5).Formostan-gles,thevaluesof?randkraresignicantlylargerthan?t;kt.Theltersq?t;qkt;q?randqkrarecausalandhaveanin-nitesupport.Theirinnitesupportstemsfromthepresenceofvinthem(Eq.18,20),asseenbyEq.(27).However,practically,theseltersdecayfast,andwemaydenetheireffectivewidth.Theeffectivewidthofq?tisdenedby?t=1 dvuuut Pxx2q?2t(x) Pxq?2t(x);(28)asmultiplesofd.Similarly,theeffectivewidthskt;?randkrcorrespondingrespectivelytoqkt;q?randqkrcanbecalcu-lated.Fig.7plotsthesevaluesasafunctionof.Formostangles,thevalues?randkraresignicantlylargerthan?tandkt.Recallthat^Lrisaffectedby?randkr(Eq.17),while^Ltisaffectedby?tandkt(Eq.19).Inotherwords,^Lrisaffectedbylterswitheffectivelymuchlonger`tails'than^Lt.Thus,problemsassociatedwithboundarycondi-tionsaregenerallyexpectedtobemoreseverein^Lr.Fig.8showsthissevereeffect,whenreconstructionisbasedsimplyonzero-padding.Inthissimulation,Lt=0,thustheonlytaskoftherecoveryiseliminationofthesec-ondaryreectionsofLr.Insomecases,thecreatedstripsmaybemoredisturbing(subjectivelyandobjectively)thantheoriginalreverberations,underminingtherecovery.4.4.SolutioninPracticeWenowdescribehowweperformedtherecoveryinpractice.First,tosignicantlyreduceproblemsassociatedwithboundaryconditions,weusemirror-paddingatx0.Theresultingimagesstillhaveartifacts,buttheydecaywithx.Then,reconstructionofLtisdoneusingEq.(19).Wefoundthatpractically^Lttoleratestheunknownboundaryconditions(providedthatmirrorpaddingisused).More-over,asdiscussedinSec.3,thePSFsthatactuponLtarewellconditioned.Forthisreason,thefastandsimplelinearlteringinEq.(19)provedsufcient. Figure8.Inversionusingzeropaddingyieldssharpstripeartifacts.ThereconstructionofLrismoredifcult.Itsuffersmorefromunknownboundaryconditions,duetothelargeref-fectivewidthsofq?randqkr.Moreover,somefrequencycomponentsof^Lrarefundamentallyillconditioned,asdis-cussedinSec.3.Therefore,wedonotcalculate^LrbythesimplisticlinearlteringdescribedinSec.4.1.Rather,wepursuedeconvolutionofpasasolutiontoaregularizedop-timizationproblem.BasedonEq.(13),wesolve^Lr=argminLrkULr?pk2+\r\rr2Lr\r\r2:(29)InEq.(29),thetermkULr?pkisattingterm.Itisminimalwhenthedatatsthemodelwell.Theterm\r\rr2Lr\r\rintroducesregularization.Theparametersetstherelativeweightbetweenthesetwoterms.Hereregular-izationleadstoasmoothimage^Lr.However,otherregu-larizationtermsfromtheliterature[13]canbeused.ThecomputationalcomplexityofthelteringoperationisO(#ofimagepixels).Theregularizedsolutionissome-whatslower,sinceEq.(29)issolvediteratively.Eachitera-tionisO(#ofimagepixels),andweobservedthatconver-genceeffectivelyoccurredwithin25normalizedsteepestdescentiterations.5.EstimationofParametersUptothispoint,theparametersoftheproblem(dand)wereassumedtobeknown.Theirestimationisnowde-tailed.Theincidenceangleatthewindowisindependentofthewavelength.Thedisplacementdisalsopracticallyinsensitivetothewavelength.Hence,theseparametersareestimatedbasedonagrayscale(panchromatic)representa-tionoftherawimages,discardingthecolor.Determiningandthexaxis:Estimationofisdoneinthesamemannerasin[21].Furthermore,Ref.[21]de-scribeshowtheaxiscorrespondingtothePOIisdeterminedintheimageplane,basedonthepolarization.5Inourwork,thisdeterminesthexaxis,alongwhichthedisplacementdofthevisualreverberationsoccurs,aswritteninSec.2.Determiningd:Achallengeraisedbythisstudyistheesti-mationofd.First,weestimatejdj.Then,sign(d)isfound. 5Thisisdetermineduptoa90oambiguity.5 0 10 20 30 40 0 20 40 -5 0 5 10x 10 46 x 10 pixels30


Figure9.AinasimulationcorrespondingtoimagesasappearinFigs.3,4.Thepeaksat~d=30aremarkedbysmallredcircles.ThereverberationsinFigs.1and4createdisplacedreplica-tionsoftheimagecontent.Apparently,thisshouldcreateasecondarypeakoftheautocorrelationofarawframe(S?),asafunctionofahypothesizeddisplacement~d.Inprac-tice,oftensuchapeakatjdjdoesnotappear.Nevertheless,theanticipatedpeakatjdjappearswhenautocorrelationisperformedoverthehorizontalderivativeofarawframe,e.g.,j@xS?(x;y)j.Thereisastillaprobleminpractice:localmaximaoftheautocorrelationfunctionappearinad-ditionalvaluesof~d=jdj.Theseincorrectmaximachangeif@xS?(x;y)isblurred(byaGaussianlterofwidth).Ontheotherhand,thecorrectpeakat~d=jdjisconsistentdespitesuchblurringaction.Thisisseen,forinstance,inFig.9.Thisplotisbasedonasimulatedframe,similartoFig.4.TheautocorrelationAfunctionisparameterizedbytheGaussianwidth.Theconsistencyofthecorrectpeakatjdjdespitethechangeinisrevealedbyasimplevotingprocess.Thisyieldsthenalestimate^d.Forexample,inFig.9,thiscorrectlyyielded^d=jdj(whichwas30pixelsinthiscase).TheSignofdAnautocorrelationfunctionissymmetricaroundtheori-gin.Hence,itgivesnoindicationwhetherd�0ord0.Todeterminesign(d),adifferentcriterionisdevel-oped,basedonthefollowingobservation.Considerasin-glehorizontallineproleoftheimagesat~y.There,letthesourceimagesfLr(x;~y);Lt(x;~y)gbeat,exceptforasin-gleedgelat~xinoneofthesources.Thisedgeappearsalsointherawframes,e.g.inS?(~x;~y),withanabsolutederiva-tivej@xS?(~x;~y)j.Duetointernalreections,thisedgere-verberatesandappearsalsoin(~x+d;~y),(~x+2d;~y)etc.However,thestrengthoftheedgeweakensineachorder,asthePSFshkt;hkr;h?t;h?r.Hence,j@xS?(~x;~y)j&#x-5.1;䝥j@xS?(~x+j^dj;~y)j&#x-5.1;䝥j@xS?(~x+2j^dj;~y)j:(30)if0dand^d=d.Consideratypicalimage,havingatypicalcontentandrandomnoise,butnoreverberations.Takeatripletofpixels Figure10.SimulatedreconstructionscorrespondingtoFig.3.[Left]^Lr.[Right]^Lt.f(~x;~y);(~x+j^dj;~y);(~x+2j^dj;~y)g.Eq.(30)shouldholdinsomepositions(~x;~y),andbeviolatedinotherplacesinthisimage.DeneC+asthesetofallpixels(~x;~y)intheimagethatsatisfyEq.(30),foraspecic^d.Similarly,deneCasthesetofallpixels(~x;~y)intheimagethatsatisfyj@xS?(~x;~y)jj@xS?(~x+j^dj;~y)jj@xS?(~x+2j^dj;~y)j:(31)Onaverage,inatypicalimage,Eq.(30)isexpectedtoholdinasimilarnumberofpossibletripletsasthenumbersat-isfyingEq.(31).ForarandomimagehavingNpixels,jC+j=jCj=N=4.However,thepresenceofreverber-ationscreatesabiasinthisrandomness.Forinstance,pix-elssatisfyingEq.(31)complywithareverberationmodelinwhichd0(ordersdecayleftwards).Hence,wesetsign(^d)=sign(jC+jjCj):(32)Weappliedthiscriterionsuccessfullyinvarioussimulationsandintheexperiments.ThebiasofjC+jvs.jCjwas8%.6.Validation6.1.SimulationExampleThereectance(Eq.1)istypicallymuchsmallerthanthetransmittance(Eq.2).Thus,togetanoticeablemixupofthetwolayersintheacquiredimagesS?andSk,Lrshouldtypicallybeverybright.WeusedLtandLrasinFig.3,whereLt(x)2[0;113]andLr(x)2[0;513].ThemaximalvalueofS?andSkwas255.Weusedx2[d;W],whereW=226pixels,d=30pixels,and=27o.Gaussiannoisehavingstandarddevia-tionof3graylevelswasaddedindependentlytoeverypixelinS?andSk.ThesimulatedacquiredimagesS?andSklooksimilartoFig.4.Then,theunavailabilityofboundaryvaluesissimulatedbychoppingoffthewholepartcorre-spondingtox0intheframes,leavingtheirsupporttobex2[0;W].Now,wesimulatedthereconstruction.WeusedmirrorextrapolationasdescribedinSec.4.3.Fig.10depictsthereconstructionsobtainedasdescribedinSec.4.4,using6 /5 /7 ZLQGRZ FDPHUD Figure11.Thesetupusedintheexperiments.=0:01.Thescenesareseparated,andthereverberationsareeliminated,makingtheresultmorevisuallypleasing.QuantitativeAssessmentToquantitativelymeasuretherecovery,weusethemeansquarederror(MSE)insimulations,wherewehaveaccesstothegroundtruth.First,letusignorethespatialeffectofthevisualreverberations,asinthestateoftheart.Inthiscase,wesimplyrunthepointwisemethodof[21].HereweobtainedavalueMSEpointwiser=161.Whenaccount-ingforthespatialeffectofreverberationsusingourmethod,weobtainedMSErecoveryr=53.Hence,quantitatively,themethodgreatlyimprovedtheMSE.Thisquantitativeim-provementisevident,sincereverberationsintherawdataweresignicantquantitativelyandsubjectively(visually).6.2.ExperimentingwithReal­WorldObjectsWeappliedthemethodonarealsetup.WeusedaNikonD100camera,toobtaindatawhichislinearlyrelatedtothesceneradiance(no\rcorrection).A200mmlensandapolarizerwerettedtoit.Thecamerawassetinfrontofaglasswindow,similarlytothewaydepictedinFig.11.Weacquiredafewframesinvariouspolarizerorientations.Basedonthem,wederivedS?andSk,asdescribedin[17].TheimageS?isshowninFig.1.Itclearlydemonstratesthesecondaryreection(reverberation),aswellastheconfu-sioncausedbythesuperpositionofthereectedandtrans-mittedscenes.TheimageSklookssimilartoit.Theestimationoftheparameterswasperformedasde-scribedinSec.5.Theautomaticestimationofdyielded^d=36pixels,whichwasconsistentwithmanualmeasure-ment.Theestimatedis41o.Consequently,therecoverydescribedinSec.4.4wasappliedseparatelytoeachcolorband.Finally,alltheprocessedcolorbandswerecom-binedtotheresultingoutputcolorimages.Thenal6re-constructed^LtisdepictedinFig.12whilethenal^LrisdepictedinFig.13.Thereconstructionsarevisuallypleas- 6Thereconstructedimagescontainresidualedgeartifacts.Anexplana-tionhypothesisandthewayweovercamethemaredescribedin[10]. Figure12.Thereconstructed^LtintheexperimentcorrespondingtoFig.1.IthasneithervisualreverberationsnorapparenttraceofthecomplementarysceneLr,whoseestimateisshowninFig.13. Figure13.Thereconstructed^LrintheexperimentcorrespondingtoFig.1.IthasneithervisualreverberationsnorapparenttraceofthecomplementarysceneLt,whoseestimateisshowninFig.12. Figure14.[Left]Areal-worldimageSk.Itcontainsasuperposi-tionoftwoscenes.Thevisualreverberation,e.g.,ofthebaby,canclearlybeseen.[Right]Thereconstructed^Lrisseparatedwhilethereverberationiseliminated.ing.Theyhavenocrosstalk(goodseparation)andthevi-sualreverberationsareeliminated.Inanotherexperiment,theacquiredimageSkappearsontheleftsideofFig.14.TheacquiredimageS?lookssimilar.Thereconstructed^LrinthisexperimentappearsontherightsideofFig.14.7 7.DiscussionThepresentedclosedformphysicalmodelelucidatesthefundamentallimitationsoftheproblem(conditioningandboundaryconditions),foreachsourcescene.Furthermore,itcreatesthebasisforfuture,improvedrecoveryalgo-rithms.Thetaskisessentiallyoneofsolvingaconvolutivemixture(see[23]).Theworkcanbeextendedtomethodsthatdonotrelyonapolarizer,ashasbeendoneinotherstudiesthatdealtwithtransparentscenes.Notethatthetruedmaybenon-integer.Thiscreatesresidualerrorsthatmayneedtobeassessed.Moreover,thisaspectmaybeincorporatedexplicitlyintothealgorithm.Theanalysismayalsobegeneralizedtonon-planarwindows.AcknowledgmentsWethankEinavNamerforhelpingintheexperimentsandArieYeredorforusefulcomments.YoavSchechnerisaLandauFellow-supportedbytheTaubFoundation.TheworkwassupportedbytheIsraeliMinistryofScience,Cul-tureandSport(Grant3-3426).ItwasconductedintheOl-lendorffMinervaCenter.MinervaisfundedthroughtheBMBF.References[1]A.Agrawal,R.Raskar,S.K.Nayar,andY.Li.Remov-ingphotographyartifactsusinggradientprojectionandash-exposuresampling.ACMTOG,24:828–835,2005.[2]G.A.AtkinsonandE.R.Hancock.Shapeestimationusingpolarizationandshadingfromtwoviews.IEEETrans.PAMI,29:2001–2017,2007.[3]F.Basbug,K.Swaminathan,andS.Nandkumar.Noisere-ductionandechocancellationfront-endforspeechcodecs.IEEETrans.Speech&AudioProcess.,11(1):1–13,2003.[4]E.Be'eryandA.Yeredor.Blindseparationofsuperim-posedshiftedimagesusingparameterizedjointdiagonaliza-tion.IEEETrans.IP,17:340–353,2008.[5]A.M.Bronstein,M.M.Bronstein,M.Zibulevsky,andY.Y.Zeevi.SparseICAforblindseparationoftransmittedandreectedimages.Int.J.Imag.Sys.Tech.,15:84–91,2005.[6]T.Chen,H.P.A.Lensch,C.Fuchs,andH.Seidel.Polar-izationandphase-shiftingfor3dscanningoftranslucentob-jects.Proc.IEEECVPR,1–8,2007.[7]O.G.Cula,K.J.Dana,D.K.Pai,andD.Wang.Polariza-tionmultiplexinganddemultiplexingforappearance-basedmodeling.IEEETrans.PAMI,29:362–367,2007.[8]M.Dadic.InuenceofchanneltransferfunctiononLMSalgorithmbasednonrecursivedeghostinginanalogTV.Int.Conf.TrendsinComm.,EUROCON,2,2001.[9]T.DarrellandE.Simoncelli.Nullingltersandthesepara-tionoftransparentmotions.InProc.IEEECVPR,738–739,1993.[10]Y.DiamantandY.Y.Schechner.Eliminatingartifactswheninvertingvisualreverebrations.Tech.Rep.CCIT-692,Dept.ElectricalEng.,Technion,March2008.[11]H.FaridandE.H.Adelson.Separatingreectionsfromim-agesbyuseofindependentcomponentanalysis.JOSAA,16:2136–2145,1999.[12]Hermanto,A.K.Barros,T.Yamamura,andN.Ohnishi.Sep-aratingrealandvirtualobjectsusingindependentcomponentanalysis.IEICETrans.Inf.&Syst.,E84-D:1–9,2001.[13]R.Kaftory,Y.Y.SchechnerandY.Y.ZeeviVariationaldistance-dependentimagerestorationProc.IEEECVPR2007.[14]A.LevinandY.Weiss.Userassistedseparationofreectionsfromasingleimageusingasparsityprior.Proc.ECCV,602–613,2004.[15]D.MiyazakiandK.Ikeuchi.Shapeestimationoftranpar-entobjectsbyusinginversepolarizationraytracing.IEEETrans.PAMI,29:2018–2029,2007.[16]D.Miyazaki,M.Saito,Y.Sato,andK.Ikeuchi.Determiningsurfaceorientationsoftransparentobjectsbasedonpolariza-tiondegreesinvisibleandinfraredwavelengths.JOSAA,19:687–694,2002.[17]S.K.Nayar,X.S.Fang,andT.Boult.Separationofreec-tioncomponentsusingcolorandpolarization.IJCV,21:163–186,1997.[18]B.SarelandM.Irani.Separatingtransparentlayersthroughlayerinformationexchange.Proc.ECCV,4:328–341,2004.[19]Y.Y.Schechner,N.Kiryati,andR.Basri.Separationoftransparentlayersusingfocus.IJCV,39:25–39,2000.[20]Y.Y.SchechnerandS.K.Nayar.Generalizedmosaicing:Polarizationpanorama.IEEETrans.PAMI,27:631–626,2005.[21]Y.Y.Schechner,J.Shamir,andN.Kiryati.Polarizationandstatisticalanalysisofscenescontainingasemi-reector.JOSAA,17:276–284,2000.[22]M.Shizawa.Directestimationofmultipledisparitiesfortransparentmultiplesurfacesinbinocularstereo.InProc.IEEEICCV,447–454.,1993.[23]S.Shwartz,Y.Y.SchechnerandM.Zibulevsky.Efcientseparationofconvolutiveimagemixtures.InProc.ICA(LNCS3889),246-253,2006.[24]S.Shwartz,E.Namer,andY.Y.Schechner.Blindhazesepa-ration.InProc.IEEECVPR,1984-1991,2006.[25]R.Szeliski,S.Avidan,andP.Anandan.Layerextrac-tionfrommultipleimagescontainingreectionsandtrans-parency.Proc.IEEECVPR,1:246–253,2000.[26]T.TreibitzandY.Y.Schechner.Instant3Descatter.InProc.IEEECVPR,vol.2,1861–1868,2006.[27]Y.Tsin,S.B.Kang,andR.Szeliski.Stereomatchingwithlinearsuperpositionoflayers.IEEETrans.PAMI,28:290–301,2006.[28]S.UmeyamaandG.Godin.Separationofdiffuseandspec-ularcomponentsofsurfacereectionbyuseofpolariza-tionandstatisticalanalysisofimages.IEEETrans.PAMI,26:639–647,2004.[29]L.B.Wolff.Polarizationvision:anewsensoryapproachtoimageunderstanding.Image&VisionComp.,15:81–93,1997.8