CodedRolling Shutter Photography Flexible SpaceTimeSam - PDF document

CodedRollingShutterPhotography:FlexibleSpace-TimeSamplingJinweiGuColumbiaUniversityYasunobuHitomiSonyCorporationTomooMitsunagaSonyCorporationShreeNayarColumbiaUniversityAbstractWeproposeanovelreadoutarchitecturecalledcodedrollingshutterforcomplementarymetal-oxidesemiconduc-tor(CMOS)imagesensors.Rollingshutterhastradition-allybeenconsideredasadisadvantagetoimagequalitysinceitoftenintroducesskewartifact.Inthispaper,weshowthatbycontrollingthereadouttimingandtheexpo-surelengthforeachrow,therow-wiseexposurediscrep-ancyinrollingshuttercanbeexploitedtoexiblysamplethe3Dspace-timevolumeofsceneappearance,andcanthusbeadvantageousforcomputationalphotography.TherequiredcontrolscanbereadilyimplementedinstandardCMOSsensorsbyalteringthelogicofthecontrolunit.Weproposeseveralcodingschemesandapplications:(1)codedreadoutallowsustobettersampletimedimen-sionforhigh-speedphotographyandopticalowbasedapplications;and(2)row-wisecontrolenablescapturingmotion-blurfreehighdynamicrangeimagesfromasingleshot.Whileaprototypechipiscurrentlyindevelopment,wedemonstratethebenetsofcodedrollingshutterviasimu-lationusingimagesofrealscenes.1.IntroductionCMOSimagesensorsarerapidlyovertakingCCDsen-sorsinavarietyofimagingsystems,fromdigitalstillandvideocamerastomobilephonecamerastosurveillanceandwebcameras.Inordertomaintainhighll-factorandread-outspeed,mostCMOSimagesensorsareequippedwithcolumn-parallelreadoutcircuits,whichsimultaneouslyreadallpixelsinarowintoaline-memory.Thereadoutpro-ceedsrow-by-row,sequentiallyfromtoptobottom.Thisiscalledrollingshutter.Rollingshutterhastraditionallybeenconsidereddetrimentaltoimagequality,becausepixelsindifferentrowsareexposedtolightatdifferenttimes,whichoftencausesskewandotherimageartifacts,especiallyformovingobjects[11,13,6].Fromtheperspectiveofsamplingthespace-timevolumeofascene,however,wearguethattheexposurediscrepancyinrollingshuttercanactuallybeexploitedusingcomputa-tionalphotographytoachievenewimagingfunctionalities ThisresearchwassupportedinpartbySonyCorporationandtheNa-tionalScienceFoundation(IIS-03-25867andCCF-05-41259). (a)CMOSimagesensorarchitecture (b)TimingforrollingshutterFigure1.TheaddressgeneratorinCMOSimagesensorsisusedtoimplementcodedrollingshutterwithdesiredrow-resetandrow-selectpatternsforexiblespace-timesampling.andfeatures.Infact,afewrecentstudieshavedemonstratedtheuseofconventionalrollingshutterforkinematicsandobjectposeestimation[1,2,3].Inthispaper,weproposeanovelreadoutarchitectureforCMOSimagesensorscalledcodedrollingshutter.Weshowthatbycontrollingthereadouttimingandexposurelengthforeachrowofthepixelarray,wecanexiblysam-plethe3Dspace-timevolumeofasceneandtakepho-tographsthateffectivelyencodetemporalsceneappearancewithinasingle2Dimage.Thesecodedimagesareusefulformanyapplications,suchasskewcompensation,high-speedphotography,andhighdynamicrangeimaging.AsshowninFig.1,thecontrolsofrow-wisereadoutandexposurecanbereadilyimplementedinstandardCMOSimagesensorsbyalteringthelogicoftheaddressgeneratorunitwithoutanyfurtherhardwaremodication.Forcon-ventionalrollingshutter,theaddressgeneratorissimplyashiftregisterwhichscansalltherowsandgeneratesrow-reset(RST)androw-select(SEL)signals.Forcodedrollingshutter,newlogicscanbeimplementedtogeneratethede-siredRSTandSELsignalsforcodedreadoutandexposure,asshowninFig.2.SincetheaddressgeneratorbelongstothecontrolunitofCMOSimagesensors[9,17],itiseasytodesignandimplementnewlogicsintheaddressgeneratorusinghighleveltools.Wehavebeguntheprocessofdevelopingtheprototypesensor.Weexpecttohaveafullyprogrammablecodedrollingshuttersensorin18months.Meanwhile,inthispa-per,wedemonstratedcodingschemesandtheirapplications1 viasimulations.Thesimulationexperimentsareperformedwithrealimages,i.e.,fullspace-timevolumesofsceneap-pearancerecordedwithhigh-speedcameraswereusedtosynthesizetheoutputimagesofacodedrollingshuttersen-sor.Thesesynthesizedimagesthushavesimilarcharacter-isticsastheimagescapturedwitharealsensor.2.RollingShutterandRelatedWorkWerstintroducesomebackgroundrelatedtorollingshutter.AsshowninFig.1a,theexposureinCMOSimagesensorsiscontrolledbytherow-resetandrow-selectsig-nalssentfromtherowaddressdecoder–eachrowbecomesphotosensitiveafterarow-resetsignal,andstopscollectingphotonsandstartsreadingoutdataafterarow-selectsig-nal.Becausethereisonlyonerowofreadoutcircuits,thereadouttimingsfordifferentrowscannotoverlap.Inrollingshutter,asshowninFig.1b,thereadouttimingsareshiftedsequentiallyfromtoptobottom.Wedenotetheresettime,thereadouttime,andtheex-posuretimeforarowwith
ts,
t,and
t,respectively.FortypicalCMOSsensors,
tsisaround15sand
tisaround1540s.ForanimagesensorwithMrows,wedenotetheresettiming(i.e.,therisingedgeofrow-resetsignals)andthereadouttiming(i.e.,thefallingedgeofrow-readoutsignals)forthey-throw(1yM)withts(y)andt(y),respectively.Forrollingshutter,wehavet(y)=y
tandts(y)=t(y)
t
t
ts.Recentworkshavemodeledthegeometricdistortioncausedbyrollingshutter[1,11],andproposedmethodstocompensateforskewduetoplanarmotion[13,6].Wilburnetal.[24]demonstratedtheuseofrollingshut-terwithacameraarrayforhigh-speedphotography.ManycomponentsofCMOSimagesensorshavealsobeenre-designedforspecicapplications,suchasHDRimag-ing[25,15]andmulti-resolutionreadout[12].Theseideasaregivingrisetoanewbreedofimagesensors,referredtoas“smart”CMOSsensors[20],whichhavespurredsigni-cantinterestamongcameramanufactures.3.CodedRollingShutter:AnOverviewIncodedrollingshutter,bothts(y)andt(y)canbecon-trolledbytheaddressgenerator.Asaresult,theexposuretime,
t(y),canalsobevariedfordifferentrows.LetE(x;y;t)denotetheradianceofascenepoint(x;y)attimet,andS(x;y;t)denotetheshutterfunctionofacamera.ThecapturedimageI(x;y)isI(x;y)=11E(x;y;t)
S(x;y;t)dt:(1)Figure2showsfourtypesofshutterfunctions.Fortheglobalshutter(Fig.2a)widelyusedinCCDimagesensors,S(x;y;t)isa1Drectangularfunction.Raskaretal.[22]proposedtheuttershutter(Fig.2b),whichbreaksthesin-gleintegrationtimeintomultiplechunksandthusintro- (a)Globalshutter (b)Fluttershutter[22] (c)Rollingshutter (d)CodedrollingshutterFigure2.Timingchartsforfourtypesofcamerashutterfunction.duceshighfrequencycomponentsformotiondeblur.Sincethecodingisxedforallpixels,itisineffectacodedglobalshutter,whichisalsoa1Dfunctionoftimet.Forconven-tionalrollingshutter(Fig.2c),S(x;y;t)=S(tt(y))=S(ty
t).Itisstilla1Dfunctionbecauseofthexedsequentialreadoutorder.Incontrast,theproposedcodedrollingshutter(Fig.2d)extendstheshutterfunctionto2DS(y;t)inwhichboththereadouttimingt(y)andtheexposuretime
t(y)canberow-specic.Asmentionedearlier,becausethereisonlyonerowofreadoutcircuits,thereadouttimingsfordifferentrowscannotoverlap,whichimposesaconstraintont(y).Specically,foranimagesensorwithMrows,thetotalreadouttimeforoneframeisM
t.Eachvalidreadouttimingschemewillcorrespondtoaone-to-oneassignmentoftheMreadouttimingslotstotheMrows.Thisisatypicalassignmentproblemincombinatorialoptimization.Figure3showsonesimpleassignment,whichisadoptedintheconventionalrollingshutter.Theremainderofthepaperdemonstratesseveralcodingschemesandtheirapplications.Welimitedthecodingtobewithinoneframetime.1.Codedreadouttimingt(y)forbettersamplingoverthetimedimensionforopticalowandhigh-speedphotography,asdetailedinSec.4.2.Codedexposuretime
t(y)forhighdynamicrange(HDR)imaging.Withthiscontrol,weproposeasim-plerow-wiseauto-exposureinSec.5.1,whichisef-fectiveforoutdoor,naturalscenes.Moreover,ifbotht(y)and
t(y)arecontrollable,weshowinSec.5.2thatmotion-blurfreeHDRimagescanberecoveredfromasinglecodedimage.4.CodedReadoutandItsApplicationsInthissection,weshowhowtousecodedreadouttobet-tersamplethetimedimensionbyshufingthereadouttim-ingst(y)amongrows.Weproposetwocodingschemes. Figure3.Codedrollingshutterisconstrainedasanassignmentproblembecausereadouttimingscannotoverlapbetweenrows.Hereweshowtheassignmentofaconventionalrollingshutter.4.1.InterlacedReadoutFigure4ashowstherstschemecalledinterlacedread-out,inwhichthetotalreadouttimeforoneframeisuni-formlydistributedintoKsub-images(K=2inFig.4).Eachsub-imagehasM=Krowswhilepreservingfullres-olutioninthehorizontal(x)direction.Thisissimilartointerlacinginvideobroadcastsystems[5].Wenotethatinterlacedreadoutisdifferentfromtheskip-readoutmodeinCMOSimagesensors[17]whereonlyaxedsubsetofrowsareusedforimaging—Incontrast,interlacedreadoutusesalltherowsandallowsfull-lengthexposureforeachrow.Specically,forinterlacedreadout,thereadouttimingt(y)forthey-throwissetast(y)=M(y1) Ky1 K
(M1)+1
t;(2)foranimagesensorwithMrows,whereb
cistheoorfunction.Sincethetimelagbetweenthetopandthebot-tomrowofeachsub-imageisM
t=K,theskewinthesesub-imagesis1=Ktimeoftheconventionalrollingshut-ter.Moreover,thetimelagbetweentwoconsecutivesub-imagesisalsoreducedtoM
t=K(i.e.,theframeratewillincreaseKtimes.)Thesub-imagescanbeusedtoestimateopticalowforframeinterpolationandremovingskew,asdepictedinFig.4b.Thegrayandredcirclesrepresentsthesampledpointsfromtheinputcodedimage.First,weusecubicin-terpolationtoresizethetwosub-imagesI1andI2verticallytofullresolution(shownasthegrayandredsolidlines)andthencomputetheopticalowu0betweenthem.Intermedi-ateimageswithintheblueparallelogramcanberecoveredviabidirectionalinterpolation[4]:Iw(p)=(1w)I1(pwuw(p))+wI2(p+(1w)uw(p));(3)where0w1,p=(x;y)representsonepixel,uw(p)istheforward-warpedopticalowcomputedasuw(pwu0(p))=u0(p).Forexample,theblackdot-dashlineinFig.4bshowstheintermediateimageIw=0:5.Moreover,wecanalsointerpolateaskew-freeimage,Iskew-freew=0:5,shownasthebluedashlineinFig.4b,byreplacingthescalarwinEq.(3)withavectorw=1(y1)=(M1). (a)Interlacedreadout(K=2) (b)DiagramofinterpolationFigure4.Diagramsofinterlacedreadoutcodingandinterpolation.Figure5showsanexperimentalresult.Thesceneisrecordedwithahigh-speedcamera(CasioEX-F1)at300fpswithimageresolution512384.Therecordedvideoisusedtosynthesizetheimagecapturedwithconventionalrollingshutter(Fig.5a)andthecodedimagecapturedwiththeinterlacedreadoutrollingshutter(Fig.5b).Figures5(c,d)showthetwointerpolatedsub-images,I1andI2,wheretheskewisreducedbyhalfcomparedtoFig.5a.Figure5eshowsthecomputedopticalowu0,whichisusedtointer-polatetheintermediateimageIw=0:5(Fig.5f)andtheskew-freeimageIskew-freew=0:5(Fig.5g).Figures5(h,i)showtheerrorsbetweenthetwointerpolatedimagesandthetrueskew-freeimage,whichconrmsthatIskew-freew=0:5indeedremovesalmostalltheskewandisclosetothegroundtruth.TheremainingerrorinIskew-freew=0:5iscausedbyocclusionsduringtheestima-tionofopticalow.4.2.StaggeredReadoutFigure6ashowsthesecondcodingscheme,calledstag-geredreadout,whichreversestheorderofreadoutwithineveryKneighboringrows(K=2inFig.6a).Similartothepreviousscheme,Ksub-imagescanbeextractedfromasinglecodedimagewhereeachsub-imagehasM=Krows.Thereadouttimingt(y)inthiscaseissetast(y)=2y1 K+1Ky+1
t:(4)Comparedwiththeinterlacedreadout,therearetwomaindifferences:(1)Thetimelagwithineachsub-imageforstaggeredreadoutis(MK+1)
t.Thisisroughlythesameasconventionalrollingshutter(M
t),whichmeanstheskewremainsunchanged.(2)Thetimelagbetweentwoconsecutivesub-imagesis
t,whichisontheorderof1540s.Thisisthemainbenetofthiscoding—asimplewaytoachieveultra-highspeedphotographyfortime-criticaleventssuchasaspeedingbulletoraburstingballoon.OneexampleisshowninFig.6.TheoriginalclipisrecordedwithaPhantomv7.1cameraat600fps[23],whichisusedtosynthesizethecodedimageofstaggeredreadout(K=8)showninFig.6b.Threeextractedsub-imagesareshowninFigs.6(c,d,e),whichcapturethemomentthefoot (a)Conventionalrollingshutter (b)Input:interlacedreadout(K=2) (c)Interpolatedsub-image1 (d)Interpolatedsub-image2 (e)Opticalowu0 (f)Intermediate:w=0:5 (g)Skew-free:skew-freew=0:5 (h)Errorofw=0:5 (i)Errorofskew-freew=0:5Figure5.Resultsofopticalowbasedinterpolationwithinterlacedreadout. (a)Staggeredreadout(K=2) (b)Input:staggeredreadout(K=8) (c)Sub-image:1 (d)Sub-image:4 (e)Sub-image:8Figure6.Staggeredreadoutforhigh-speedphotography.touchestheground.Thisprecisemomentwouldnotbecap-turedusingaconventionalrollingshutter.Moreresultscanbefoundinthesupplementaryvideo.5.CodedExposureandReadoutforHighDy-namicRange(HDR)ImagingHDRimagingtypicallyrequireseithermultipleimagesofagivenscenetakenwithdifferentexposures[8,16],orspecialhardwaresupports[19,18].Therstrequiresastaticsceneandstaticcameratoavoidghostingandmo-tionblur,whilethelatterisexpensive,thusmakingHDRimaginginconvenientforhand-heldconsumercameras.Re-searchershaverecentlyproposedmethodstoremoveghost-ing[10]andmotionblur[27,14]frommultipleimages.Inthissection,weshowthatcodedrow-wiseexposure
t(y)canbeusedtoalleviatetheseproblemsforpracti-calHDRimaging:(1)Thedynamicrangeofsceneradiancecanbebettercapturedbyeitheradaptivelysettingtheexpo-sureperroworinterlacingmultipleexposuresintoasingleimage,whichavoidstakingmultipleimagesandeffectively (a)Adaptiverow-wiseAE (b)MembershipfunctionsFigure7.Adaptiverow-wiseAE.RefertoSec.5.1fordetails.reducesghostingandmotionblurduetocamerashake.(2)Row-wiseexposureiseasytoimplementwithinstandardCMOSimagesensors,asexplainedinSec.3,andthusthecostislow.Weproposetwomethodsbelow.5.1.AdaptiveRow­wiseAuto­ExposureWhenwetakeapicturewithauto-exposure(AE),thecameraoftenwilltaketwoimages—rstitquicklycapturesatemporaryimagetogaugetheamountoflightanddeter-mineanoptimalexposure,andthenadjuststheexposureandtakesasecondimageasthenaloutput[17].Mostex-istingAEalgorithmsaredesignedtondasingleexposurethatisoptimalforanentireimage,whichishighlylimitingformanyscenes.Inourrstmethod,weimplementasimpleyeteffec-tiveauto-exposuremethodcalledadaptiverow-wiseauto-exposure.AsshowninFig.7a,themethodndsanoptimalexposureforeachrowofthepixelarray,andthentakesasecondimagewhereeachrowisadjustedforbestcapturingthesceneradiance.Thesecondimageisnormalized(i.e.,dividedbytherow-wiseexposure)togeneratethenalout-put.Comparedwithconventionalauto-exposure,row-wiseauto-exposureismoreexibleandeffective,especiallyforsceneswherethedynamicrangeismainlyspannedverti-cally(e.g.,outdoorsceneswheretheskyismuchbrighterthantheground). (a)ConventionalAE (b)Input:(x;y)and
te(y) (c)Output:adaptiverow-wiseAE (d)Insetsof(a)and(c) Figure8.Resultsofadaptiverow-wiseauto-exposureandconventionalauto-exposure.Tondtheoptimalexposureforeachrow,weproposeasimplemethodusingfuzzylogic.Anoptimalexposureforagivenrowshouldminimizethenumberofsaturatedandun-derexposedpixelswithintherowwhilekeepingmostpixelswell-exposed.Thisheuristicisformulatedasfollows.AsshowninFig.7b,werstintroducethreemembershipfunc-tions,s(i),d(i),andg(i)whichdescribethedegreeofbeingoverexposed(i.e.,saturated),underexposed,orwell-exposedforintensityi.LetI0denotethetemporaryimage.Itmeasuresthesceneradianceeverywhereexceptinthesat-uratedregions,wherenoinformationisrecorded.WethusassumethesceneradianceisLI0(1+ss(I0)),wheres0isascalefactorusedtoestimatethesceneradianceinsat-uratedregions.Thesmallersis,themoreconservativetheAEalgorithmwillbe.Theoptimalexposure
t(y)forthey-throwisfoundbymaximizingthefollowingfunctional:max
tl
te(y)
tuX(L(x;y)
t(y));(5)where(i)isdenedas(i)=s(i)+dd(i)+gg(i);(6)withweightsd,gandlowerandupperboundsofexpo-sureadjustment
tland
tu.Inourexperiments,s=4,d=0:2,g=0:05,
tl=0:1,
tu=10:0,andthethreemembershipfunctionsaredesignedass(i)=1 1+245i,d(i)=1 1+i10,andg(i)=1=(1+i128 10060).Oncetheoptimalexposuresarefoundforallrows,1theyareusedtocapturethesecondimageI.Thenaloutputimageiscom-putedasI(x;y)=I(x;y)=
t(y).Theexperimentsareperformedasfollows.Foreachscene,weuseaCanonEOS20Dtotake30imageswithexposuresrangingfrom1 6400to1:5secondsinthemanual 1ThiscalculationcanbedonewithinaFPGAbuiltincameras. Figure9.StaggeredreadoutandmultipleexposurecodingforHDRimagingwithhand-heldcameras.mode,aswellasanimageintheAEmode(denotedasI0).TocreatethecodedimageI,foreachrowfromthecaptured30imageswechoosetheonewhoseexposureistheclos-esttotheestimatedoptimalexposure.Figure8showstwosetsofexperimentalresults–Fig.8ashowstheimageswithconventionalAE,Fig.8bshowsthecodedimagesIandtherow-wiseexposures,andFig.8cshowsthenaloutputsI.AsshowninFig.8d,theadaptiverow-wiseAEproduceshigherqualityphotographs,inwhichthesaturation(e.g.,thecloudsandthetext)aswellasthenoiseindarkregions(e.g.,thestatuesandthetoys)aresignicantlyreduced.Thismethodrequiresalmostnoimageprocessing.Iffur-therpost-processing(e.g.,denoising)isneeded,noiseam-plicationalongtheverticaldirection(whichisknownfromtheexposurepatterns)canbeconsidered.Moreover,forsceneswherethedynamicrangeispredominantlyspannedhorizontally(e.g.,adarkroomviewedfromoutside),thismethodrevertsbacktoconventionalauto-exposure.5.2.StaggeredReadoutandCodedExposureforHDRImagingwithHand­heldCamerasThegoalofthesecondmethodistorecoverHDRfromasingleimageforhand-heldcameras.Weshowthatwithstaggeredreadout(showninSec.4.2)androw-wiseexpo-sure,notonlycanwecodemultipleexposuresintooneim-age,butwecanalsoremoveimageblurduetocamerashakebyestimatingplanarcameramotion. (a)Input:codedimage (b)Sub-image:1 (c)Sub-image:2 (d)Sub-image:3 (e)Opticalow (f)Blurimages&kernels (g)Output:recoveredHDRr (h)InsetsFigure10.ResultsofstaggeredreadoutandcodedexposureforHDRimagingforhand-heldcameras. (a)Input:codedimage (b)Output:recoveredHDRr (c)InsetsFigure11.AnotherresultforHDRimagingwithcodedrollingshutterforhand-heldcameras.AsshowninFig.9,thepixelarrayofaCMOSimagesensoriscodedwithstaggeredreadout(K=3)andthreeexposures,
t1,
t2,and
t3.Thus,fromasinglein-putimage,I,wecanextractthreesub-images,I1,I2,andI3.Thesesub-imagesareresizedverticallytofullresolu-tionusingcubicinterpolation.Forstaticscenes/cameras,thesesub-imagescanbedirectlyusedtocomposeaHDRimage.Forhand-heldcameras,however,camerashakeisinevitable,especiallyforlongexposures.Becausethesub-imagesarecapturedwithstaggeredreadout,thetimelagbe-tweenthemissmall.Wecanthusassumecameramotionastranslationonlywithaxedvelocitybetweensub-images.Themotionvector~u=[u;uy]canbeestimatedfromI1andI2usingopticalow:~umean(computeFlow(I1;I2I1)):(7)Thecomputedowisusedtoestimatethetwoblurker-nelsforI2andI3,respectively.InsteadofdeblurringI2andI3directly,wefoundthatdeblurringtwocomposedim-ages,I1I2andI1I2I3,willeffectivelysuppresstheringing,2wheretheoperatormeanstheimagesarerstcenter-alignedusingthemotionvector~uandthenaddedtogether.WedenotethetwodeblurredimagesasIb1deblur(I1I2;~u;
t1;
t2),andIb2deblur(I1I2I3;~u;
t1;
t2;
t3).Finally,theoutputHDRimageis:II1
t1Ib1
t1+
t2Ib2
t1+
t2+
t3=3:(8)Theoptimalexposureratios
t3:
t2:
t1shouldbedeterminedbyconsideringboththedesiredextendeddy-namicrangeaswellasthenoiseamplicationduetothemotiondeblurring.Intuitively,thelarger
t3:
t1is,thelargertheextendeddynamicrangeshouldbe,butalargerratiocanalsoamplifymorenoiseduringmotiondeblurringandinturnlowertheeffectivedynamicrange.Ananalysisofthenoiseamplicationandtheselectionoftheexposureratioscanbefoundinthesupplementarydocument.Inourexperiments,weset
t2=2
t1and
t3 2MorediscussionisinSection6. 8
t1,andthustheimprovementindynamicrangewillbe20log(
t3=
t1)=20log8=18:06dB.Wesetthecameramotiontobe~u0=[1;1]pixelsper
t1time.Weusethedeblurringalgorithmpresentedin[7].Simulationex-perimentsareperformedusingasetoftenhigh-resHDRimages,collectedfrommultiplesourcesonline[21].QuantitativeevaluationisshowninTable1.Wecom-paredourmethod(i.e.,I)withthethreeothersingle-shotmethodsusingaconventionalrollingshutter(i.e.,shortex-posureI1=
t1,medialexposureI2=
t2,orlongexpo-sureI3=
t3).Theperformanceforeachmethodismea-suredastheNormalizedRootMeanSquareError(NRMSE)betweentherecoveredHDRimageandtheoriginalscene,takingintoaccountthedynamicrangeoftheoriginalscene:NRMSE(I0;^I0)=p kI0^I0k2=N max(I0)min(I0),whereI0istheorigi-nalscene,^I0istheoutputimageforagivenmethod(i.e.,IorIi=
tei,i=1;2;3),andNisthenumberofpixels.Weranthesimulationwithtwotypesofimagenoise.First,weassumedGaussianadditivenoise(i.e.,scenein-dependentnoise),andperformedthesimulationwithsevenlevelsofnoise.Second,weconsideredGaussianphotonnoise.WemeasuredthephotonnoiseparametersforaCanonEOS20DcameraatveISOvalues,andusedthemtosimulatethephotonnoiseinthecapturedimages.Foreachmethodandeachlevelofimagenoise,wesimulatedthecapturedimagewithmotionblurduetocamerashake,imagenoise,andsaturationduetolimiteddynamicrange.ThesimulatedimageswereusedtorecoverHDRimageI.WerepeatedthesimulationonthetenHDRimagesandtooktheaverage.TheresultsarelistedinTable1.Ourmethod(thecodedrollingshutter)performsbestacrossalllevelsofnoise.Moreover,asexpected,amongthethreeexposuresusingaconventionalrollingshuttercamera,forlowimagenoise,theshortexposurerecoverstheHDRimagewell.Asim-agenoiseincreases,themedialexposureyieldsbetterresult.Withextensivenoise,despiteofsaturationandmotionblur,thelongexposureisbetter.Withacodedrollingshutter,ourmethodcombinesthemeritsofthesethreeexposuresettingsandperformsconsistentlybetterthantheothers.Figure10showsoneexample.Thesimulatedinputim-ageIisshowninFig.10a,generatedaccordingtothecod-ingpatterninFig.9.Gaussiannoise(=0:005)isaddedinI.Thethreesub-images,I1,I2,andI3,areshowninFigs.10(b,c,d).ComparedwiththenaloutputimageI,thesesub-imagesareeithertoodarkandnoisyortooblurryandsaturated,asshowninFigs.10(g,h).Figure11showsanothersetofexperimentalresults.6.ConclusionandDiscussionSummaryInthispaper,weproposedanewreadoutarchi-tectureforCMOSimagesensors,calledcodedrollingshut-ter.Bycontrollingthereadouttimingandexposureperrow,wedemonstratedseveralcodingschemesthatcanbeap-pliedwithinoneframeandtheirapplications.TherequiredcontrolscanbereadilyimplementedinstandardCMOSim-agesensors.AssummarizedinTable2,weachievebene-tssuchaslessskew(i.e.,timelagwithinasub-image)orhighertemporalresolution(i.e.,timelagbetweentwocon-secutivesub-images)orhigherdynamicrange,atthecostofreducedverticalresolution.Onefuturedirectionistodesigncodingschemesformultipleframes,whereexistingde-interlacingmethodscouldbeleveragedtoincreasever-ticalresolution.VerticalResolutionandAliasingAsmentioned,alltheotherapplicationstradeoffverticalresolutionforotherfea-tures(exceptfortheadaptiverow-wiseauto-exposureinSec.5.1).Aliasingduetothecubicinterpolationmightcausenoticeableartifacts,especiallyforthemotiondeblur-ringinSec.5.2.Basedon[26],weanalyzedthealias-ingcausedbyimagedown-samplingandup-sampling,andfoundthatbysimplycombiningthedown-sampledimagesatdifferentphases(e.g.,theimageofalloddrowsandtheimageofallevenrows),thealiasingwillbeeffectivelyal-leviated(whentheblurkernelsarethesameforthedown-sampledimages,aliasingcanbecompletelyavoided.)—thisiswhyweusedthecombinedimagesforHDRimag-inginSec.5.2.Wenotethathorizontalresolutionisalwaysfullyretained.Oneinterestingfuturedirectionistotransferthehighfrequencydetailsfromthehorizontaldirectiontotheverticaldirection.RandomCodingPatternandSparseReconstructionIfwemodelthescenebrightnessforonepixel(x;y)overtimetasa1-Dsignal,thecorrespondingpixelintensityinthecapturedimageisalinearprojectionofthis1-Dsignalwiththeexposurepattern.Thus,with(random)codedex-posurepatterns,weattemptedtoreconstructthespace-timevolume(withzeroskew)fromasingleshotbyexploitingthesparsityinsignalgradients.Insimulation,wefoundthatalthoughthemethodcouldeffectivelyremoveskew,manyhigh-frequencyartifactswouldbepresent,especiallyaroundstrongverticaledges.Removaloftheseartifactswillbethesubjectofourfutureresearch.Pixel-wiseExposureControlCMOSimagesensorsareabletoaddressindividualpixels[17],providedthatthereisenoughbandwidthfordatatransmissiononthechip.Onefutureworkistolookintopossibleimplementationsofpixel-wiseexposurecontrolonchipandachieveevenmoreexibilityforspace-timesampling.References[1]O.Ait-Aider,N.Andreff,J.M.Lavest,andP.Martinet.Simultane-ousobjectposeandvelocitycomputationusingasingleviewfromarollingshuttercamera.InProceedingsofEuropeanConferenceonComputerVision(ECCV),2006. Table1.Performance(NRMSE)ofHDRimagingwithcamerashakeusingcodedrollingshutter Methods GaussianAdditiveNoise() GaussianPhotonNoise 0.00050.0010.0020.0050.010.020.05 ISO100ISO200ISO400ISO800ISO1600 1=
te1 0.01840.02010.02500.04170.06410.09620.1579 0.01810.01840.01900.02020.02222=
te2 0.02240.02260.02350.02800.03750.05500.0940 0.02240.02250.02270.02300.02363=
te3 0.06440.06440.06440.06450.06500.06660.0739 0.06440.06440.06440.06450.0645r 0.01760.01780.01830.02050.02550.03730.0736 0.01760.01770.01780.01800.0182 *Thebestperformerforeachnoiselevelisshowninbold.Table2.SummaryofCodedRollingShutterPhotography ControlSchemes Sub-imageRows(BeforeInterpolation)OutputImagesTimeLagwithinoneSub-imageTimeLagbetweentwoconsecutiveSub-imagesApplications none M1M
trM
trconventionalrollingshutter interlacedreadout M=KKMtr=KMtr=Kslowmotion,skewcompensation staggeredreadout M=KK(MK+1)
trtrhighspeedphotography row-wiseexposure M1(HDR)M
trM
tradaptiverow-wiseauto-exposure staggeredreadout,row-wiseexposure M=K1(HDR)(MK+1)
tr
trHDRimagingforhand-heldcameras *Thebenetsareshowninbold.[2]O.Ait-Aider,A.Bartoli,andN.Andreff.Kinematicsfromlinesinasinglerollingshutterimage.InProceedingsofIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2007.[3]O.Ait-AiderandF.Berry.Structureandkinematicstriangulationwitharollingshutterstereorig.InProceedingsofIEEEInternationalConferenceonComputerVision(ICCV),2009.[4]S.Baker,D.Scharstein,J.P.Lewis,S.Roth,M.J.Black,andR.Szeliski.Adatabaseandevaluationmethodologyforopticalow.InProceedingsofIEEEInternationalConferenceonComputerVi-sion(ICCV),2007.[5]E.B.BellersandG.deHaan.De-Interlacing:AKeyTechnologyforScanRateConversion.ElsevierScienceB.V.,2000.[6]D.Bradley,B.Atcheson,I.Ihrke,andW.Heidrich.Synchronizationandrollingshuttercompensationforconsumervideocameraarrays.InIEEEInternationalWorkoponProjector-CameraSystems(PRO-CAMS),2009.[7]K.Dabov,A.Foi,V.Katkovnik,andK.Egiazarian.Imagedenoisingbysparse3Dtransform-domaincollaborativeltering.IEEETrans-actionsonImageProcessing,16(8):2080–2095,August2007.[8]P.E.DebevecandJ.Malik.Recoveringhighdynamicrangeradiancemapsfromphotographs.InProceedingsofSIGGRAPH,1997.[9]E.R.Fossum.CMOSimagesensors:Electroniccamera-on-a-chip.IEEETransactionsonElectronDevices,44(10):1689–1698,1997.[10]O.Gallo,N.Gelfand,W.-c.Chen,M.Tico,andK.Pulli.Artifact-freehighdynamicrangeimaging.InProceedingsofIEEEInternationalConferenceonComputationalPhotography(ICCP),2009.[11]C.Geyer,M.Meingast,andS.Sastry.Geometricmodelsofrolling-shuttercameras.InIEEEWorkshoponOmnidirectionalVision,2005.[12]S.E.Kemeny,R.Panicacci,B.Pain,L.Matthies,andE.R.Fossum.Multiresolutionimagesensor.IEEETransactionsonCircuitsandSystemsForVideoTechnology,7(4):575–583,August1997.[13]C.-K.Liang,L.-W.Chang,andH.H.Chen.Analysisandcompensa-tionofrollingshuttereffect.IEEETransactionsonImageProcess-ing,17(8):1323–1330,2008.[14]P.-Y.Lu,T.-H.Huang,M.-S.Wu,Y.-T.Cheng,andY.-Y.Chuang.Highdynamicrangeimagereconstructionfromhand-heldcameras.InProceedingsofIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2009.[15]M.Mase,S.Kawahito,M.Sasaki,Y.Wakamori,andM.Furuta.AwidedynamicrangeCMOSimagesensorwithmultipleexposure-timesignaloutputsand12-bitcolumn-parallelcyclicA/Dconverters.IEEEJournalofSolid-StateCircuits,40(12):2787–2795,December2005.[16]T.MitsunagaandS.K.Nayar.Radiometricselfcalibration.InProceedingsofIEEEConferenceonComputerVisionandPatternRecognition(CVPR),volume1,pages374–380,June1999.[17]J.Nakamura.ImageSensorsandSignalProcessingforDigitalStillCameras.CRCPress,2006.[18]S.K.NayarandV.Branzoi.Adaptivedynamicrangeimaging:Op-ticalcontrolofpixelexposuresoverspaceandtime.InProceedingsofIEEEInternationalConferenceonComputerVision(ICCV),vol-ume2,pages1168–1175,October2003.[19]S.K.NayarandT.Mitsunaga.Highdynamicrangeimaging:Spa-tiallyvaryingpixelexposures.InProceedingsofIEEEConferenceonComputerVisionandPatternRecognition(CVPR),volume1,pages472–479,June2000.[20]J.Ohta.SmartCMOSImageSensorsandApplications.CRCPress,2007.[21]PFSTools.http://pfstools.sf.net/.[22]R.Raskar,A.Agrawal,andJ.Tumblin.Codedexposurephotogra-phy:Motiondeblurringusinguttershutter.ACMTransactionsonGraphics(SIGGRAPH),25(3):795–804,July2006.[23]VisionResearch.http://www.visionresearch.com/.[24]B.Wilburn,N.Joshi,V.Vaish,M.Levoy,andM.Horowitz.Highspeedvideousingadensecameraarray.InProceedingsofIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2004.[25]O.Yadid-PechtandE.R.Fossum.WideintrascenedynamicrangeCMOSAPSusingdualsampling.IEEETranscationsonElectronDevices,44(10):1721–1723,October1997.[26]A.Youssef.Imagedownsamplingandupsamplingmethods.InInter-nationalConferenceonImaging,Science,Systems,andTechnology(CISST'99),pages132–138,LasVegas,June1999.[27]L.Yuan,J.Sun,L.Quan,andH.-Y.Shum.Imagedeblurringwithblurredandnoisyimagepairs.ACMTransactionsonGraphics(SIG-GRAPH),26(3):1:1–1:10,July2007.