R G G B R I G B RGBIRconventional a 400 600 800 1000 0 1000 2000 3000 4000 wavelengthsensor spectral response red Rgreen Gblue Binfrared I bcenterlefttop ccenterlefttop dcenterleft ID: 509995
Download Pdf The PPT/PDF document "HighResolutionPhotographywithanRGB-Infra..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
HighResolutionPhotographywithanRGB-InfraredCameraHuixuanTang1XiaopengZhang2ShaojieZhuo2FengChen2KiriakosN.Kutulakos1LiangShen21UniversityofToronto2QualcommCanadaInc.AbstractAconvenientsolutiontoRGB-InfraredphotographyistoextendthebasicRGBmosaicwithafourthltertypewithhightransmittanceinthenear-infraredband.Unfor-tunately,applyingconventionaldemosaicingalgorithmstoRGB-IRsensorsisnotpossiblefortworeasons.First,theRGBandnear-infraredimagearedifferentlyfocusedduetodifferentrefractiveindicesofeachband.Second,manufac-turingconstraintsintroducecrosstalkbetweenRGBandIRchannels.InthispaperweproposeanovelimageformationmodelforRGB-IRcamerasthatcanbeeasilycalibrated,andproposeanefcientalgorithmthatjointlyaddressesthreerestorationproblemschanneldeblurring,channelseparationandpixeldemosaicingusingquadraticimageregularizers.Wealsoextendouralgorithmtohandlemoregeneralregularizersandpixelsaturation.Experimentsshowthatourmethodproducessharp,full-resolutionim-agesofpureRGBcolorandIR.1.IntroductionThelastfewyearshaveseenawealthofnewcameraandsensortechnologies,withconsumer-levelphotographybeingamajordriveroftheseefforts.Animportantdevel-opmentinthisdirectionaresensorsthatrecordshort-waveinfrared(IR)andcolor(RGB)inoneshot.Althoughin-fraredimaginghasalonghistoryinremotesensingandthephysicalandbiologicalsciences[25],recentworkincom-putationalphotography[8,21,22]suggestsithasgreatun-tappedpotentialinconsumerimagingapplicationsfromashphotographyandreduced-blurimagingto3Dsensingandbiometrics.Theconventionalapproachtojointinfraredandcolorimagingiseithertoswapcolorltersonacamerasensi-tivetoinfraredortouseonecameradedicatedtoinfraredimagingandanotheroneforcolor.Unfortunately,takingsequentialshotsafterswappingltersisproblematicwhenimagingmovingsubjects,andusingtwocamerasraisesahostofproblemsofitsown(e.g.,highercost,hardertominiaturize,misalignedinfraredandcolorimagesduetodifferencesincameraviewpoint). R G G B R I G B RGB-IRconventional (a) 400 600 800 1000 0 1000 2000 3000 4000 wavelength(!)sensor spectral response red (R)green (G)blue (B)infrared (I) (b)centerlefttop (c)centerlefttop (d)centerlefttop (e)centerlefttop (f)Figure1:Pixelmultiplexing,channelcrosstalkandchromaticaberrationofourprototypeRGB-IRcamera.(a)Colorlterar-ray.(b)Spectralresponsecurves.(c),(d)Defocuskernelatthreespatiallocationsfordepths44cmand18cmaway,respectively,withthelensfocusedat20cm.Ineachcase,weoverlaytwoker-nels:acombinedRGBkernel(green)andanIRkernel(red).(e),(f)ResultofttingthePSFmodelofTangandKutulakos[23],andevaluatingitatthesamelocationsasin(c),(d).CamerasutilizingRGB-IRsensors,ontheotherhand,canbethoughtofastwocamerasinone:theyrecordbothphotosonthesamesensorwiththesamelensandthuscanbereadilyusedincellphonesystemswithconventionalop-tics,producingperfectly-alignedRGBandIRimages.AsshowninFigure1a,RGB-IRsensorsextendthebasicRGBmosaicwithafourthltertypewhosetransmittanceishighintheshort-waveIRband(800-950nm).Producingafull-resolutionRGB-IRphotofromtheout-putofsuchasensorrequiresinferringthethreemissingchannelsateverypixel.Althoughindividualsolutionstothiscolordemosaicingproblemforconventionalcam-eras[14]differinmanyrespects,theyallrelyonthefactthatthespectralpowerdistributionoflightarrivingatnearbypixelsisoftenhighlycorrelated.Thesecorrelationsmakeitpossibletoinferapixel'smissingcolorchannelsfromnearbypixelswherethosechannelsaresenseddirectly.1 Unfortunately,applyingconventionaldemosaicingalgo-rithmstodatafromanRGB-IRsensorisnotpossiblefortworeasons.First,manufacturingconstraintscausetheR,GandBltersofacolormosaictotransmitintheIRband.Al-thoughthisiseasilycorrectedinconventionalRGBcam-erasbyplacinganIR-blockinglterintheopticalpath,thiscannotbedoneforRGB-IRcameras.ThesecamerasmustthereforecontendwiththefactthattheirR,G,andBpixelsactuallyrecordasuperpositionoftheIRchannelandapureR,G,andBchannel,respectively(Figure1b).BecausetheRGBltershavehighIRtransmittance,theyarealsomorepronetosaturationandnoise,reducingdynamicrange.Second,itisnotalwayspossibleforboththeRGBandtheIRimageofasubjecttobefocusedonthesensoratthesametime,sincelightrefractsdifferentlydependingonwavelength.Asaresult,amajorchallengeincaptur-inghigh-qualityphotoswithanRGB-IRsensoriscompu-tationalcorrectionoftheinevitabledefocusblur.Thisblurmaybepresenteitherinthecolorcomponentortheinfraredcomponent,orpotentiallyinboth(Figure1c-f).Inprac-tice,thismeansthatforsubjectscloserthanthehyperfocaldistance[20],adjustingthelenstobringtheRGBchan-nelsintofocuswillcausetheIRchanneltobedefocusedandviceversa.NaivedemultiplexingofRGB-IRdataun-dersuchconditionsyieldspoor-qualityphotosinwhichblurfromonechannelistransferredtotheothers.Thisdepen-denceisespeciallyproblematicinhigh-resolutionRGB-IRcameras,wheretheIRbandisfarfromtheotherthreeandwhereeventinylevelsofdefocusblurwillbecapturedbythesensor.Thus,high-qualityRGB-IRphotographyrequiressimul-taneoussolutionofthreebasiccomputationalproblems:(1)channelseparation,wherebythesensor'smeasurementsaredecomposedintopureR,G,BandIRchannels,(2)pixelde-multiplexing,wherethevalueofthesechannelsisestimatedateverysensorpixel,and(3)channeldeblurring,wherewavelength-dependentdefocusblurintheIRand/orRGBchannelsisremoved.Westudytheirjointsolutionfrombothatheoreticalandanalgorithmicperspective.WeproposeanovelimageformationmodelforRGB-IRcamerasandthensimplifyitsoitcanbereadilyusedwithcontemporaryrestorationalgorithms[4].Asasidebenet,ourmodelenablesefcientrestorationaswell.Specically,weshowthat,intheabsenceofsaturation,ourmodelre-ducestoasetof416linearconstraintsintheFourierdomainandthuscanbedirectlyinvertedunderquadraticregularization.Moreover,althoughiterationsarenecessarytohandlenon-quadraticimageregularizersandpixelsatura-tion,thisformulationstillenablesefcientimplementationoftheindividualiterations.Wenallyextendourrestora-tionmethodtohandlepixelsaturation,whichoftenoccurstoRGB-IRcameras.1.1.RelatedworkIndividually,signalseparation[3],demultiplexing[7,13]anddeblurring[11]havereceivedconsiderableatten-tioninthecomputationalphotographyandimageprocess-ingcommunities.Wearenotaware,however,ofsolutionsapplicabletoRGB-IRcameras,whereallthreeproblemsmustbeunderstoodandsolvedjointly.Specically,recentworkinNIRimaginghasproposedcolorlterarraysforone-shotRGBandIRcapturebuteitherdoesnotconsiderimagerestoration[12],orstud-iesdemosaicingorcrosstalkinisolation[6,9].Although[18]addressesbothcrosstalkanddemosaicing,itassumescrosstalkbetweenthegreenandIRchannelsonly.More-over,thesemethodsdonotaccountforchannel-dependentdefocusbluranddonotuseFourier-domainanalysiseither,whichcanoffersignicantcomputationalefciencies.Ourrestorationalgorithmiscloselyrelatedtorecentworkonconvexoptimizationforimagerestoration[2,4].Inparticular,ourworkcanbethoughtofasanextensionofHeideetal.'soptimizationframework[4],designedtoex-ploitthespecialpropertiesofimageformationincameraswithRGB-IRmosaics.2.ImageformationmodelWestartbymodelingtheformationoftheinputRGB-IRimageifromtheideal(i.e.,sharpandfull-resolution)im-agesofthescenelatvariouswavelengths.Wethensim-plifyitfurtherbyassumingthesensor'sspectralresponsesareapproximatelythesuperpositionoffournarrow-bandresponsecurves.Thissimpliedmodelcanbereadilyin-vertedusingexistingrestorationalgorithms.ContinuousimageformationmodelLetusrstconsidertheunsaturatedversionofaninputrawimagej.Represent-ingallimagesascolumnvectors,wecanexpressjastheresultofthreesuccessivelinearoperationsontheidealim-agelj=Xn=R;G;B;ISn Zrn()Kl| {z }wavelength-dependentblurd!| {z }irradianceofn-thlter| {z }resultofRGB-IRmultiplexing+e(1)whereedenotesnoise.Wenowdescribeindetailthethreeoperations:Wavelength-dependentlensblur:subjectsclosetothecamerawillbeoutoffocusinatleastoneoftheR,G,BandIRchannelsnomatterwherethecamera'slensisfo-cused.Thisisbecausetheindexofrefractionofthelensandthusthedistanceofthecamerafromtheplaneofperfectfocusdependsonwavelength.Inaddition,opticalaberra-tionswillproduceblurevenatthein-focusdepth,andthe blurkernelwillvaryaccordingtobothimagepositionanddepth.Wecharacterizethewavelength-dependentblurforeachwavelengthwithaconvolutionmatrixK.Channelcrosstalk:Manufacturingconstraintspreventtheltersofacolormosaicfromblockingotherbands,e.g.,asshowninFigure1b.ThissuperpositionofR,G,BandIRmakesitimpossibletotreateachchannelindividually,andalsolimitstheeffectivedynamicrangeoftheimage.Con-sequently,thesensorirradianceofthen-thlterintegratesthatofallwavelengths,weightedbythesensorresponsecurvern.Pixelmultiplexing:Duetothemosaicpatternofthecolorlterarray,thenarrowR,G,Bbandsareonlymea-suredateveryotherpixel.Therefore,thehighestfrequen-ciesinanindividualchannelcannotbemeasuredevenifperfectchannelseparationanddeblurringwerepossible.Tomodelthis,wemultiplywithabinarymasktheimageir-radianceduetoeachcolorlterandthensumoverallfourlterstoproducethelinearsensorimagej.Thecorrespond-ingmatrixSninEq.(1)isadiagonalmatrixthatstoresthebinaryindicatorofthen-thlterSnnx;x]=(1iflterisnatpixelx;0otherwise(2)Finally,thecapturedimageisaturatesatpixelswheretheirradianceislargerthanthemaximalpixelintensityimaxi=min(j;imax)(3)ApproximationforrestorationpurposesSincethewavelengthiscontinuous,theidealimagelhasanin-nitenumberofchannelsandisdifculttorestore.Inprac-tice,however,wecandiscretizethemulti-spectralimagelintofourpurecolorchannels.Thissimpliesthemodelandmakesitamenabletoimagerestoration.Weassumethefoursensorresponsefunctionsr()=[rR()rG()rB()rI()]�(�indicatesmatrixtranspose)canbemodeledasthesuperpositionoffournarrow-bandresponsefunctionsq()=[qR()qG()qB()qI()]�througha44channelcrosstalkmatrixC:r()=Cq()(4)Becauseweexpectthebandstobenarrow,wecanas-sumethattheblurkernelsareconstantwithinthebandofqn.WedenotethemasKn.Thuswecandiscretizetheidealimageintoahiddenimageh="hRhGhBhI#wherethefourchannelsarehn=Zqn()l()d:(5)ThuswesimplifyEq.(1)asj=Fh+e(6)wherematrixFmodelstheentireimageformationprocessF=S(C\nI)K(7)HeretheblockdiagonalmatrixK=diag(KRKGKBKI)(8)accountsforband-specicdefocus,matrixC\nImodelschannelcrosstalk(\ndenotestensorproductofmatrices)C\nI="cRRIcRGIcRBIcRIIcGRIcGGIcGBIcGIIcBRIcBGIcBBIcBIIcIRIcIGIcIBIcIII#(9)andS=SRSGSBSI(10)isthesubsamplingmatrixthatproducesthenalmosaic.Inthefollowingweassumethatdefocusisaknowncon-stant,somatrixFinEq.(7)isalsoknown.Wecomputethismatrixusingaprocessakintosensorandlenscalibra-tionforconventionalcameras.Thuscomputingthehiddenimagehfrominputimageibecomesanon-blindrestora-tionproblem.3.CalibrationWecomputematrixFbycomputingthematricesKandCthatmodelband-specicblurandcrosstalk.Thesubsam-plingmatrixisxedbythe22mosaicpatternofthecolorlterarray.SinceKdependsonthelensopticswhereasCdependsonthesensorresponses,eachiscalibratedsepa-rately.BlurkernelcalibrationWeusethetechniqueofJoshietal.[5]toestimatethenon-parametricblurkernels.WethentthemtothecompactPSFmodelofTangandKutu-lakos[23]toaccountfordefocusandaberrations.Firstweneedtoeliminatesensorcrosstalktoallowes-timatingtheblurkernelsforeachchannelindividually.Wedothisbyplacingcolorltersinfrontofthecameratoallowonlyanarrowbandofwavelengths.ThechannelcrosstalkforourparticularprototypecameramainlyoccursbetweenRGBandinfrared.Therefore,weuseIR-onlyandRGB-onlyltersforthispurpose.WeperformPSFcalibrationforindividualfocussettingsindependently.Foreachfocussetting,weestimateblurker-nelsat22depthsspanningabout15metersaroundthein-focusposition.Ateachdepth,weestimatenon-parametricblurkernelsat913imagepositionsfortheRGBandtheIRchannelindependently[5].Wethenaccountformonochromaticandchromaticaber-rationsaswellasdefocus[23].WettheRGBandinfraredblurkernelssimultaneously,constrainingallfourchannelstohavethesamemonochromaticaberrationparameters,andtodifferonlyinthedefocusparameter(Figure1c-f). red (R)green (G)blue (B)infrared (I) (a) 400 600 800 1000 0 1000 2000 3000 4000 wavelength(!)sensor spectral response red (R)green (G)blue (B)infrared (I) (b)Figure2:CrosstalkcalibrationresultsforourRGB-IRcamera.(a)Manuallyspeciedapproximateintervalsofsupportforeachchannel.ThesupportfortheR,G,B,andIRchannelis[550;800],[400;650],[0;600],and[800;1000]nmrespectively.(b)Decom-posedsensorspectralresponsefunctions.CharacterizingchannelcrosstalkWeassumethesensorresponsesr()areknownforadensesamplingofwave-length.1OurgoalistondadecompositionD=C 1ofspectralresponsesintofourindependentnarrowbandsq.Wemanuallydeneabinaryfunctionntoindicatetheap-proximatesupportofeachchannel,wheren()=1ifiswithinthesupportofchannelnandn()=0otherwise.Wethenminimizetheamountofdemultiplexedsensorresponseoutsidethesupport.Thisiscomputedbysolvingaleast-squaresproblemforeachrowd�nofDmindnX:n()=0 d�nr() 2s.t.dnnn]=1(11)andthencomputingCbymatrixinversionC=dRdGdBdI �(12)Figure2showsthecrosstalkcalibrationresultsforoursen-sor.Inourimplementation,wemultiplythecrosstalkma-trixCwithamanually-deneddiagonalmatrixsothatthehiddenimageisproperlywhitebalanced.4.RGB-IRimagerestorationIntheabsenceofpixelsaturation,wesolvethefollow-ingoptimizationproblemtoestimatethehiddenimagehinEq.(6)minhj Fh2+(h)(13)whereisaregularizationfunctionofthehiddenimageh.ThestructureofmatrixFallowsefcientsolutionofEq.(13)underaGaussiangradientprioraswellasmoregeneralones.Weconsiderbothcasesbelow. 1Inpracticewecanobtainthisfromtheltermanufacturerorthroughadditionalcalibration[16].4.1.DirectrestorationwithquadraticregularizersWerstconsiderthefollowingminimizationproblemminhj Fh2+Xmwm Rmh tm2(14)Theregularizationfunctionisoftheform(h)=XmwmRmh tm2(15)Theweightswm,circulantmatricesRm,andvectorstmcontrolthespecicchoiceof.Forexample,ifisaGaus-sianprioronimagegradientsthenm2f12g,matricesR1andR2implementthegradientsrxandry,t1andt2arezero,respectively,andw1andw2controlthestrengthofregularization.FromEqs.(7)-(10)itfollowsthatFisabandedmatrixwithdimensionsWH4WHforaninputimageofwidthWandheightH.Becauseofitslargesize,theoptimiza-tioninEq.(14)cannotbesolveddirectlyingeneral.Inthespeciccaseofspatially-invariantblur,however,theopti-mizationcanbeexpressedintheFourierdomainandsolvedforeachfrequencyindependently,inonestep.Specically,let^hR^hG^hB^hIand^jbetheFouriertransformofthehiddenimagecomponentshRhGhBhIandinputimagejrespectively.AkeypropertyoftheFourier-domainimage^jisthattheelementsofthefourspa-tialfrequencies(u;v)(u+;v)(u;v+)(u+;v+)(16)dependonlyonthecorrespondingelementsinthehiddenimageforany(u;v).Inparticular,if^juvand^huvcollecttheelementsof^jand^hforthosefrequencies,wehave^juv=Fuv^huv+^euv(17)where^Fuv=^S(C\nI)^Kuv(18)Thematrix^Kuvisa1616diagonalmatrixthatmodelsper-frequencymodulationduetodefocusblur.Thematrix^Sisa416matrixthatmodelssubsamplinginthefrequencydomainasamixtureoffrequenciesofthehiddenimage:^S=1 41 1 1111 1 111111 11 1 111 111 1 11111 11 11 111 1 1 11111111 11 11 1 11 1 1111111 11 11(19)SeeAppendixAforaderivationofEqs.(17)-(19).WecannowreformulatetheoptimizationofEq.(14)intoWH 4independentsubproblems,eachofwhichinvolvesonlythetupleoffrequenciesinEq.(16)forsome(u;v):min^huv^juv ^Fuv^huv2+Xmwm^Rmuv^huv ^tmuv2(20) where^Rmuvisa1616diagonalmatrixstoringFouriertransformelementsofthelterscorrespondingtoRmatthefourfrequenciesinEq.(16),and^tmuvstorestheFouriertransformelementsoftmatthesefrequencies.SeeAp-pendixBformoredetailsonhowthisoptimizationcanbeperformedefciently.Inpractice,theblurkernelvariesduetoopticalaberra-tions.Accordingly,werelaxthedefocuskernel'sspatialinvariancebytreatingtheinputimageasacollectionoflo-calpatches,eachofwhichisblurredbyadifferentbutknownblurkernel.4.2.RestorationwithnonquadraticregularizersSincequadraticregularizationtendstooversmoothim-agediscontinuities,weuseamoregeneral,robustregular-izer.Specically,weusealinearcombinationofanL1normonimagegradientsandadenoisngterm[4]basedonthegraphLaplacianmatrix(h)=w1(jrxh+jryh)| {z }L1ongradients+w2h�(diag(A1) A)| {z }Laplacianmatrixh(21)where1denotesavectorofallonesandA=diag(ARAGABAI)isapixelafnitymatrixthattakesintoaccountpixelseparationaswellasintensitydifferencesinthehiddenimage:Annx;y]=exp 1 2 2dist(x;y)2 1 2 2hhx hhy2(22)Here and arespatialandintensityvarianceparameters,respectively.TosolveEq.(13)weusetheoptimizationapproachofHeideetal.[4].SeeAlgorithm1fortheexactsequenceofsteps.Thisalgorithmincludestwostepsbeyondthosein[4]thathandlepixelsaturationandarediscussedinSection5.ThemostcomputationallyexpensivestepsinAlgorithm1areSteps2,9and11.ToupdatetheslackvariablestiinStep9wesett0i=ti+ h,lteritwithabilaterallter[24]ofspatialvariance andintensityvariance 1,andnallysubtracttheresultfromt0i.Weusethepermutohedrallat-tice[1]toimplementthebilaterallterefciently.Tocom-putethehiddenimageinSteps2and11,weobservethatthisinvolvesaquadraticoptimizationthatcanbeefcientlysolvedwiththemethodinSection4.1.5.HandlingpixelsaturationBecausetheRGBltersdonotblockIR,theRGBpixelintensitiesincludeanIRcontributionthatreducesthedy-namicrangeoftheimage.Inparticular,saturationoccurs Algorithm1:Ournalrestorationalgorithm. input:inputimagei,imageformationmatrixF,weightsw1;w2parameter=1=400, =40, =5//initialization1setj i2computeh argminhjj Fhj2+w1jrxhj2+w1jryhj23processhbybilaterallteringandinpaintingsaturatedareas4seth h,tx 0;ty 0;ti 0repeat 5hlast h//enforceinequalityconstraint6j Pi+(I P)max(Fh;imax)//regularizationpenalty7tx min(max(tx+ rxh; 1);1)8ty min(max(ty+ ryh; 1);1)9updateti (ti+ (I diag(A1) 1A)h)//datafidelity10updatez h w1r�xtx w1r�yty w2ti11updateh argminhjj Fhj2+1 jh zj2//extrapolation12extrapolateh 2h hlastuntilconvergenceoutput:estimatedhiddenimageh frequentlyandmustbehandled.2Weaddressthisbyturn-ingEq.(13)intoaconstrainedminimizationproblem:minh;jj Fh2+(h)(23)s.t.jiPjPi(24)whereiisaninputimagecontainingsaturatedpixels,thematrixPextractsunsaturatedpixels,inequalityconstraintsareappliedelement-wise,andtheregularizationfunction(h)isgivenbyEq.(21).Theimagejrepresentsanir-radianceimagethatdoesnotsaturate,andbecomesanad-ditionalunknownthatmustbeestimatedjointlywiththehiddenimageh.Theinequalityconstraintsensurethatiislessthanjatallsaturatedpixels,andisequaltojotherwise.FormulatingpixelsaturationasinequalityconstraintsisessentialinourcasebecauseitallowsustoupdatehusingefcientFourier-domainoperations(Section4.1)insteadofsolvingalargelinearsysteminthespatialdomain.WeextendthemethodinSection4.2tosolvetheaboveproblemwithtwostepsinAlgorithm1:Inpaintinglargesaturatedregions(Step3).Weinitial-izehbyinpaintingtheunknowncolorandintensityofsaturatedpixels.ThishelpsAlgorithm1convergefaster.Moredetailsaregivenbelow. 2Ofcourse,ifitwerepossibletobalancethedynamicrangeofRGBandIRbandsbyattenuatingthestrongerofRGBandIR(e.g.,withRGB-orIR-blockinglters),wecouldreducethechanceofsaturation.ThisishardtodoinpracticebecausetherelativemagnitudesofRGBandIRvaryspatially,andfromscenetoscene. Enforcinginequalityconstraint(Step6).WeinitializejwiththeinputimageiinStep1;ateachiterationwethenxhandupdateeachpixelofjtominimizej Fh2withintheirfeasibleintervalimax1)forsaturatedpixelsandiforunsaturatedpixels.Thuswedonotupdateunsaturatedpixels,andsaturatedpixelsareupdatedtomax(Fh;imax).PartialsaturationmodelPartially-saturatedpixels,i.e.pixelswhoseintensityismissinginsomechannelsbutnotothers,arecommoninRGB-IRimages.Althoughwecan-notcompletelyinferthecolorofsuchpixels,theincompletecolorinformationtheycarrycaninformpixelafnityandisthususefulforinpainting.Tomakethemostoftheavail-ablecolorinformation,werepresentpixelcolorsas4DlinesinR,G,BandIR:"hR[x]hG[x]hB[x]hI[x]#=0(25)wherethematrix=2664cos(RG) sin(RG)00cos(RB)0 sin(RB)0cos(RI)00 sin(RI)0cos(GB) sin(GB)00cos(GI)0 sin(GI)00cos(BI) sin(BI)3775(26)iscontrolledbysixparametersRG;RB;RI;GB;GIandBI.Thisisinspiredbythecolorlinemodel[15]whichhasbeenusedtohandlesaturationinRGBimages.Speci-cally,givenknownhmandhn,mncanbeestimatedbymnnx]=arctanhmmx]+ hnnx]+(27)Theconstantbiasesthecoloroflow-intensitypixelsto-wardgray.Inpractice,werstdetectmissingentriesin,theninpaintthem,andnallyrestorehbyxingandusingEq.(25).SupportregionofsaturatedpixelsWecomputeabinarymap!nnxtoindicatewhichpixelsinchannelnofthehid-denimagehavesaturatedpixelsintheirneighborhood:!nnx]=(1ifmaxy2\nx;cmn0:5iiyimax0otherwise.(28)Heresubscriptmofcmndenotestheltertypeatpixelyinthecolorlterarray.Theneighborhood\nxcorrespondstopixelsatadistancelessthantheradiusoftheblurker-nel.Sothemap!markspixelsthatcarrysignicantcolorinformationaboutthesaturatedregion.Thenwecomputethesupportregion!0mnformnusing!0mnnx]=!mmx!nnx(29)sincehmmxandhnnxmustbothbeknowntocomputemnnx.InpaintingcolorandintensityForeachpairofchannels(m;n)wetheninpaintthemissingmnusingthealgorithmof[10].SpecicallywesolvethefollowingoptimizationproblemminmnUmn mn2(30)subjecttotheconstraintthatmnareupdatedonlyatpixelswhere!0mnnx]=0.ThematrixUinEq.(30)isanafn-itymatrixthattakesintoaccountpixelseparationandcolordifferenceinthesupportregionmn:UUx;y/exp 1 2 2dist(x;y)2 Xmn!0mnnx!0mnny 2 2(mnnx mnny])2(31)wheretheafnityweightsarenormalizedtoensureU1=1.Finally,wesolveforhnateachpixelusingEq.(25)byxingandsolvingwithleastsquares.Todealwithpixelsthatsaturateinallchannels,wesimplytreatthemasiftheIRchannelswerenotsaturated.Althoughthismayunderes-timatepixelintensity,itproducesvisuallypleasingresults.6.ResultsTotestourmethod,weranexperimentswithaprototypeRGB-IRcamera.SaturationhandlingFigure3showsrestorationresultsbe-foreandafterrunningStep3ofAlgorithm1.Ourcolorinpaintingalgorithmproducesreasonablecolorvariationinbothtexturelessandtexturedmulti-colorregions.RestorationqualityFigures4-6showrestorationresultsintheabsenceofdefocus.TheseresultsshowthenotablequalityimprovementsachievedwithAlgorithm1.Ground-truthcomparisonsFigure7and8showground-truthcomparisonsbetweenactualandestimatedRGBandIRchannels.Tocapturetheground-truthchannelsweusedIR-andRGB-blockingltersrespectively.Theseltersguaranteedthatthecapturedimageswerefreeofcrosstalkbutdidnotpreventblurduetoaberrations.ComputationtimeMatlabimplementationofthedirectmethod(Section4.1)takesabout1secondtoprocessa672760pixelpatchonadesktopcomputer.Incompari-son,amathematicallyequivalentimage-domainimplemen-tationtakesaboutaminutetonishunderthesamesettings.Fullrestorationisstillfarslowerbecauseofitsiterativena-tureandtheuseofbilateralltering.Ittakesabout14sec-ondsperiteration,withabout10ofthosesecondsspentonbilateralltering,andtypicalimagesrequiring20iterations.Removalofthedenoisingtermfromstep9resultsinanop-erationthatisthreetimesfaster,attheexpenseofnoisierresults. Step2resultStep3result Step2resultStep3result Step2resultStep3result Figure3:Wecompareresultsbeforeandaftercolorinpainting:theestimateofhiddenimagehbyStep2ofAlgorithm1haserrorsinsaturatedregions.Step3correctsthecolorandintensityofsuchregions.WeonlyshowRGBbandssincenoIRsaturationoccursintheseexamples.input RGBIR sequentialrestoration 5RGB-onlyIR-only directrestoration 5RGB-onlyIR-only fullrestoration 5RGB-onlyIR-only Figure4:Restorationresults.Input:Weturnoncameraautofocustominimizedefocusblur.Wevisualizetheimagemosaicwithtwosub-images:thethreechannelsoftheRGBimagestorepixelsundertheR,G,Bcolorlters,andtheIRimagestorespixelsundertheIRlter.Wecomparethreemethods:Sequentialrestorationrstperformsdemosaicingbysplineinterpolation,andthenperformschannelseparationbyinvertingEq.(4).Notethealiasingartifactsoneyelashes.DirectrestorationcorrespondstoSteps1-3ofAlgorithm1.Itproducesover-smoothedresults.FullrestorationcorrespondstofullexecutionofAlgorithm1.Thenalresultcontainsclearskinandirisdetailsthatareinvisibleintheinput,anddoesnotsufferfromnoiseorartifacts.NotethatsincetheinputRGBcontainsIRcontributionsaswell,itismuchbrighterthantheequivalentRGB-onlyimages.Tobettervisualizethose,theirintensitiesarescaled5.7.ConclusionRGB-IRcamerassimultaneouslysufferfromthreeprob-lems:pixelmultiplexing,channelcrosstalkandchromaticaberrations.Thecouplingoftheseproblemsmakesappli-cationofconventionaldemosaicingalgorithmstoRGB-IRimagesdifcult.Ourkeycontributionisanovelimagefor-mationmodelforRGB-IRcamerasthataccountsforallthreeproblems,allowseasycalibration,andenablesef-cientrestorationwithcommonimageregularizationfunc-tions.Webelievethatourapproachpavesthewayforava-rietyofapplicationsbasedonRGB-IRsensors.Fromapracticalperspective,ouralgorithmcanpotentiallybeac-celeratedfurtherbyexploitingGPUsorusingregularizationfunctionsthatcanbeefcientlyimplemented[19].AcknowledgementsWearegratefulforthesupportoftheNaturalSciencesandEngi-neeringResearchCouncilofCanadaundertheRTIandAccelera-torprograms,andtheMITACSAccelerateprogram.References[1]J.Baek,A.Adams,andJ.Davis.Lattice-basedhigh-dimensionalgaussianlteringandthepermutohedrallattice.J.Math.ImagingVision,46(2):211237,2013.[2]A.ChambolleandT.Pock.Arst-orderprimal-dualalgorithmforconvexproblemswithapplicationstoimaging.J.Math.ImagingVision,40(1):120145,May2011.[3]P.CommonandC.Jutten.HandbookofBlindSourceSeparation.AcademicPress,2010.[4]F.Heide,M.Steinberger,Y.-T.Tsai,N.Rouf,D.Pajak,D.Reddy,O.Gallo,J.Liu,W.Heidrich,K.Egiazarian,J.Kautz,andK.Pulli.FlexISP:Aexiblecameraimageprocessingframework.ACMSIG-GRAPHAsia,33:113,2014. input RGB4IR sequentialrestoration 1:5RGB-only4IR-only directrestoration 1:5RGB-only4IR-only fullrestoration 1:5RGB-only4IR-only Figure5:RestorationresultsforanRGB-IRimagewithalow-intensityIRchannel.Input:Weturnoncameraautofocus,soblurisduetoaberrationsbutnotdefocus.Results:ThesequentialanddirectmethodsproducenoisyIRestimates.Ourfullrestoration,ontheotherhand,producescleanandsharpresults.input RGBIR sequentialrestoration 4RGB-only2IR-only directrestoration 4RGB-only2IR-only fullrestoration 4RGB-only2IR-only Figure6:RestorationresultsforanRGB-IRimagewithlow-intensityRGBcontributions.Input:Weturnoncameraautofocus,soblurisduetoaberrationsbutnotdefocus.Results:ThesequentialmethodcausesartifactsnearthespecularhighlightofeyeglassframeandchopsticksintheRGB-onlyestimate.Boththesequentialanddirectrestorationproducenoisyfaceestimates.Incomparison,fullrestorationproducesacleanRGB-onlyimagewithnoartifacts.[5]N.Joshi,R.Szeliski,andD.Kriegman.PSFestimationusingsharpedgeprediction.InProc.CVPR,2008.[6]D.Kiku,Y.Monno,M.Tanaka,andM.Okutomi.Simultaneouscap-turingofRGBandadditionalbandimagesusinghybridcolorlterarray.InProc.SPIE,volume9023,2014.[7]R.Kimmel.Demosaicing:imagereconstructionfromcolorCCDsamples.IEEETrans.ImageProcessing,8(9),1999.[8]D.KrishnanandR.Fergus.Darkashphotography.InACMSIG-GRAPH,2009.[9]G.Langfelder,T.Malzbender,A.F.Longoni,andF.Zaraga.Design-ingcolorlterarraysforthejointcaptureofvisibleandnear-infraredimages.InProc.SPIE,pages37973800,2011.[10]A.Levin,D.Lischinski,andY.Weiss.Colorizationusingoptimiza-tion.ACMSIGGRAPH,23:689694,2004.[11]A.Levin,Y.Weiss,F.Durand,andW.T.Freeman.Understandingblinddeconvolutionalgorithms.IEEETrans.PAMI,33(12),2011.[12]Y.M.Lu,C.Fredembach,M.Vetterli,andS.S¨usstrunk.Designingcolorlterarraysforthejointcaptureofvisibleandnear-infraredimages.InProc.ICIP,pages37973800,2009.[13]J.Mairal,F.Bach,J.Ponce,G.Sapiro,andA.Zisserman.Non-localsparsemodelsforimagerestoration.InProc.CVPR,2009.[14]S.NarasimhanandS.Nayar.Enhancingresolutionalongmultipleimagingdimensionsusingassortedpixels.IEEETrans.PAMI,27(4),2005.[15]I.OmerandM.Werman.Colorlines:Imagespeciccolorrepresen-tation.InProc.CVPR,2004.[16]M.Parmar,F.Imai,S.H.Park,andJ.Farrell.Adatabaseofhighdynamicrangevisibleandnear-infraredmultispectralimages.InProc.SPIE,volume33,2008. inputRGB ground-truth(RGB-only) sequential(RGB-only) direct(RGB-only) full(RGB-only) input2IR ground-truth(3IR-only) sequential(3IR-only) direct(3IR-only) full(3IR-only) Figure7:Recoveringblur-freeIR-onlytextureonbanknotes.Input:Weintroduceddefocusblurbymanuallyfocusingbehindthescene.Ground-truthimages:WecapturedimagesusingIR-(top)andRGB-blockinglters(bottom).TheRGBimagewasthenwhitebalancedtomakevisualcomparisoneasier.Observethattheupper-rightsideofthe0digitiswashedoutintheIRchannel;thefaceandmapleleafarenearlyinvisibleintheIR;andastrongverticaledgeappearsonlyintheIR-onlyimage.Results:AlthoughtheRGBandIRchannelsareseparatedcorrectlyinallcases,onlyfullrestorationprovidescleananddeblurredRGBandIRchannels.inputRGB ground-truth(3RGB-only) sequential(3RGB-only) direct(3RGB-only) full(3RGB-only) input2IR ground-truth(4IR-only) sequential(4IR-only) direct(4IR-only) full(4IR-only) Figure8:Recoveringblur-freeRGB-onlytextures.Input:Weintroduceddefocusblurbymanuallyfocusingbehindthescene.Ground-truthimages:WefollowthesameprocedureasinFigure7.ObservethatthetextureoftheleftbookandthepurplepigmentoftherightbookarebothtransparenttoIR.Results:theRGBandIRchannelsareseparatedcorrectlyinallcases,butonlyfullrestorationproducesasharpandcleanresult.[17]W.H.Press,S.A.Teukolsky,W.T.Vetterling,andB.P.Flannery.NumericalRecipes:TheArtofScienticComputing(3rded.).Cam-bridgeUniversityPress,2007.[18]Z.Sadeghipoor,Y.M.Lu,andS.S¨usstrunk.Anovelcompressivesensingapproachtosimultaneouslyacquirecolorandnear-infraredimagesonasinglesensor.InProc.ICASSP,2013.[19]U.SchmidtandS.Roth.Shrinkageeldsforeffectiveimagerestora-tion.InProc.CVPR,2014.[20]W.Smith.ModernOpticalEngineering.McGraw-Hill,2000.[21]S.S¨usstrunkandC.Fredembach.Enhancingthevisiblewiththein-visible:Exploitingnear-infraredtoadvancecomputationalphotogra-phyandcomputervision.InProc.Symp.InformationDisplay,2010.[22]K.Tanaka,Y.Mukaigawa,Y.Matsushita,andY.Yagi.Descatteringoftransmissiveobservationusingparallelhigh-frequencyillumina-tion.InProc.ICCP,2013.[23]H.TangandK.N.Kutulakos.Whatdoesanaberratedphototellusaboutthelensandthescene?InProc.ICCP,2013.[24]C.TomasiandR.Manduchi.Bilaterallteringforgrayandcolorimages.InProc.ICCV,1998.[25]M.VollmerandK.-P.Mollmann.InfraredThermalImaging:Fun-damentals,ResearchandApplications.Wiley,2010. A.DerivationofEqs.(17)-(19)LetTbethematriximplementingtheFouriertransformofasingle-channelimageofwidthWandheightH,DenoteT=diag(T;T;T;T)thematriximplementingtheFouriertransformofanRGB-IRimageineachband.WerstturnEq.(6)intoaFourier-domainformulationbymul-tiplyingbothsidesofEq.(6)withTnotingthatT�T=I^j=Tj=TFh+Te=(TFT�)Th+Te=^F^h+^e;(32)where^j,^hand^edenotetheFouriertransformof(unsaturated)input,hiddenandnoiseimage,respectively.Thematrix^FmodelsimageformationintheFourierdomain^F=TFT�:(33)BypluggingEq.(7)intoEq.(33),wehave^F=TS(C\nI)KT�=(TST�)(T(C\nI)T�)(TKT�):(34)First,fromEq.(10)theFourier-domainsubsamplingbecomesTST�=TSRT�TSGT�TSBT�TSIT�:(35)ThematricesTSnT�arecirculantsinceSnarediagonalma-trices.EachrowofTSnT�isashiftedversionoftheFouriertransformofthediagonalofSn,whichmarksthepixellocationsunderthen-thcolorlter.Fortheparticular22mosaicpat-ternwediscussinthispaper,therowassociatedwiththefre-quency(u;v)isnon-zeroonlyincolumnscorrespondingtofre-quenciesat(u;v),(u+;v),(u;v+)and(u+;v+),asinEq.(16).Thevaluesoftheseentriesare1 4; 1 4; 1 4;1 4forT�SRT,1 4;1 4; 1 4; 1 4forT�SGT,1 4;1 4;1 4;1 4forT�SBT,and1 4; 1 4;1 4; 1 4forT�SIT,respectively.Second,fromEq.(9),theFourier-domaincrosstalkcausessu-perpositionamongdifferentbandsateachfrequencyT(C\nI)T�=C\n(TT�)=C\nI:(36)Third,fromEq.(8),theFourier-domaindefocusmatrixisTKT�=diag(TKRT�;TKGT�;TKBT�;TKIT�);(37)whereeachmatrixKniscirculantwhentheblurkernelsarespatially-invariant.ConsequentlyTKT�isadiagonalmatrixthatstorestheFouriertransformoftheblurkernelsinthefourbands.Thisissimplytheconvolutiontheorem,i.e.,image-domaincon-volutioncorrespondstoFourier-domainmodulation.Theaboveshowsthatarowin^Fcorrespondingtothefre-quency(u;v)isnon-zeroonlyinthecolumnscorrespondingto(u;v),(u+;v),(u;v+)and(u+;v+),i.e.,Eq.(16).Consequently,wecandecomposeEq.(33)intoWH 4linearsys-tems,eachrelevanttoaparticular(u;v),byextractingrowsandcolumnscorrespondingtoEq.(16).ThisleadstoEq.(17)andEq.(18),with^SdenedbyEq.(19).B.EfcientsolutionofEq.(20)Foreachfrequencytuple(u;v),thequadraticoptimizationprobleminEq.(20)hasaclosed-formsolution^huv=^F�uv^Fuv+^Ruv 1zuv(38)wherethediagonalmatrix^Ruvandthevectorzuvaredenedas^Ruv=Xmwm^R�muv^Rmuv;(39)zuv=^F�uv^juv+Xmwm^R�muv^tmuv:(40)OurkeyideaistousetotheWoodburymatrixidentity[17]^F�uv^Fuv+^Ruv 1=^R 1uv ^R 1uv^F�uvQ 1uv^Fuv^R 1uv(41)tosimplifytheinverseof1616matrices^F�uv^Fuv+^Ruvtotheinverseof44matricesQuv=I+^Fuv^R 1uv^F�uv:(42)ThematricesQuvaresmallenoughtoallowexplicitsolutionofitsinverse,andtoimplementtheinversewithelement-wiseoper-ations.ThuswecancomputeEq.(38)as^huv=^R 1uvzuv (^R 1uv^F�uv)Quv 1(^Fuv^R 1uvzuv):(43)Algorithm2showstheexactstepstocomputeEq.(43)forall(u;v)simultaneously.Specically,allQuvcanbecomputedatthesametimebylinearlycombining16basismatricesQuv=I+Xlquv[l]l:(44)Thefrequency-speciccoefcientsarequv=diag(^Kuv^R 1uv^Kuv)(45)whilethebasismatricesaresharedbyallfrequenciesl[m;n]= [m;l] [n;l](46)where =^S(C\nI):(47)Thisfollowsfromsubstituting^Fuv= ^KuvintoEq.(42). Algorithm2:ComputingEq.(43)forall(u;v) input:inputimagefrequencies^juv,cameraparametersC,^Sandkuv=diag(^Kuv),regularizationparametersrmuv=diag(^Rmuv),^tmuvandwm(and=denoteelementwisemultiplicationanddivision)1precomputematrices andbyEqs.(46),(47)2ruv Pmwmrmuvrmuv3zuv (kuv^juv)+Pmwm(rmuvtmuv)4computequv kuvkuv=ruvandQuvbyEq.(44)5solvexuvforalllinearsystems:Quvxuv=(zuvkuv=ruv)6compute^huv zuv=ruv (kuvxuv=ruvoutput:estimatedhiddenimagefrequencies^huv