Gortler Laboratory for Computer Science Massachusetts Institute of Technology Division of Engineering and Applied Sciences Harvard University Abstract This research further develops the light 64257eld and lumigraph image based rendering methods and ID: 80909
Download Pdf The PPT/PDF document "Dynamically Reparameterized Light Fields..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
(s0,t0)(s',t')(s'',t'')(f,g)FDs',t'Ds'',t''(u',v')(u'',v'')(s',t',u',v')=(s',t',f,g)F(s'',t'',u'',v'')=(s'',t'',f,g)F CF r Figure3:Givenaray ,weÞndtheraysinthedatacameraswhichintersectatthesamepoint stantforeachdatacamera.Foradynamicfocalsurface,wewillmodifythemappingwhichchangestheplacementofthefocalsurface.Astaticwithafocalsurfacethatconformstothescenegeometrygivesusadepth-correctionidenticaltothelumi-graph[7].Toreconstructarayfromtheraydatabase,weuseagener-alizeddepth-correction.WeÞrstÞndtheintersectionsof.Thisgivesusthe4-Draycoordinates inFigure3.Usingcamerasnear,sayweapply ,givingus,respectively.Thisgivesustworayswhicharestoredasthepixelinthedatainthedatacamera.WecanthenapplyaÞltertocombinethevaluesforthesetworays.Inthediagram,wehaveusedtworays,althoughinpractice,wecanusemorerayswithappropriateÞlterweights.4VariableApertureandFocusWecanuseourdynamicparameterizationtoefÞcientlycreateim-agesthatsimulatevariablefocusandvariabledepth-of-Þeld.Thisallowsustocreatefocusedimagesofmoderatelysampledsceneswithlargedepthvariationwithoutrequiringgeometricinformation.Inaddition,thisnewparameterizationprovidestheusersigniÞcantartisticexpressionwhencomposingnovelimages.4.1VariableApertureInatraditionalcamera,theaperturecontrolstheamountoflightthatcanentertheopticalsystem.Italsoinßuencestheextentofdepth-of-Þeldpresentintheimages.Withsmallerapertures,moreofthesceneappearsinfocus;largeraperturesproduceimageswithanarrowrangeoffocus.Wesimulatesyntheticaperturesnottoaffectexposure,buttocontroltheamountofdepth-of-Þeldpresentinanimage.Wecaninteractivelyemulateadepth-of-Þeldeffectbycombin-ingraysfromseveralcameras.InFigure4,wearetryingtore-constructtworays,.Inthisexample,theextentofoursyntheticaperturesisfourdatacameras.Wecenterthesyntheticaperturesattheintersectionofwiththecam-erasurface.Wethenrecallraydatabasesamplesbyapplyingforallsuchthatlieswithintheaperture.Thesesamplesarecombinedtocreateasinglereconstructedray.Notethatnearthesurfaceofthevirtualobject,doesnot.Oursyntheticaperturereconstructionwill F desiredcamera A'A'' objectvirtualDs,t Figure4:Foreachdesiredray,oursyntheticaperturesystemcen-terstheapertureattheintersectionoftheraywiththecamerasur-face.Thustherayusestheaperture.Thewillappearinfocus,whilewillnot. toappearinfocus,whilewillnot.Thesizeofthesyn-theticapertureaffectstheamountofdepth-of-Þeld.Itisimportanttonotethatourmodelisnotnecessarilyequiva-lenttoanapertureattachedtothedesiredcamera.Forexample,ifonerotatesthedesiredcamera,oureffectiveapertureremainspar-alleltothecamerasurface.ModelingtheapertureonthecamerasurfaceandnotonthedesiredcameramakestherayreconstructionmoreefÞcientandstillproducesthedesireddepth-of-Þeldeffect(SeeFigure5).Amorerealisticandcompletelensmodelisgivenin[11],althoughthisislessefÞcienttorenderandimpracticalforcapturedlightÞelds.Wedonotweightthequeriedsamplesthatfallwithinthesyn-theticapertureequally.UsingadynamicÞlterthatcontrolstheweighting,wecanimprovethefrequencyresponseofourrecon-struction.InFigure6,weattempttoreconstructthepinkdottedray .Weuseatwo-dimensionalfunctiondescribethepoint-spreadfunctionofthesyntheticaperture.Typ-hasamaximumatandisboundedbyasquareofwidth.TheÞlterisdeÞnedsuchthat)=0whenever,or.TheÞltersshouldalsobedesignedsothatthesumofsampleweightswilladdupto1.Thatis,)=1forallWeusetheapertureÞlterontheray asfol-lows.ThecenteroftheapertureÞlteristranslatedtothepoint.Then,foreachcamerathatisinsidetheaperture,wewillconstructaray andthencalculateusingtheappropriatemapping.Theneachrayisweightedbyandallweightedrayswithintheaperturearesummedtogether.Oursystemcancreatearbitrarilylargesyntheticapertures.Thesizeoftheapertureisonlylimitedtotheextenttowhichtherearecamerasonthecamerasurface.WithsufÞcientlylargeapertures,wecanseethroughobjects,asinFigure7.Oneproblemwithmak-inglargeaperturesoccurswhentheaperturefunctionfallsoutsidethepopulatedregionofthecamerasurface.Whenthisoccurs,theweightedsampleswillnotadduptoone.Thiscreatesavignettingeffectwheretheimagedarkenswhenusingsamplesneartheedgesofthecamerasurface.Thiscanbesolvedbyeitheraddingmoredatacamerastothecamerasurfaceorbyreweightingthesamplesonapixelbypixelbasissotheweightsalwaysadduptoone. Onecouldalsousetheaperturefunctionx,yasabasisfunctionateachsampletoreconstructthecontinuouslightÞeld,althoughthisisnotcomputationallyefÞcient. Figure8:Byvaryingtheplacementofthefocalsurface,onecaninteractivelycontrolwhatappearsinfocus. 5Ray-SpaceAnalysisItisinstructivetoconsidertheeffectsofdynamicreparameteriza-tiononlightÞeldswhenviewedfromrayspace[13,7]and,inpar-ticular,withinepipolarplaneimages(EPIs)[2].Itiswell-knownthat3-DstructuresoftheobservedscenearedirectlyrelatedtofeatureswithintheseparticularlightÞeldslices.Thetwo-parallelplaneparameterizationisparticularlysuitableforanalysisunderthisregimeasshownby[8].Oursystemcanalsobeanalyzedinrayspace,especiallywhenthefocalsurfaceisplanar.Inouranal-ysis,weconsidera2-DsubspaceofrayscorrespondingtoÞxedvaluesofonadynamicfocalplane.Whenthefocalsur-faceisparalleltothecamerasurface,thesliceisidenticaltoanAdynamicallyreparameterizedlightÞeldwithfourpointfea-turesisshowninFigure10a.ThedottedpointisapointatinÞn-ity.AlightÞeldparameterizedwiththefocalplanewillhavearay-spaceslicesimilartoFigure10b.Eachpointfeaturecor-respondstoalinearfeatureinrayspace,wheretheslopeofthelineindicatestherelativedepthofthepoint.Verticalfeaturesintheslicerepresentpointsonthefocalplane;featureswithpositiveslopesrepresentpointsthatarefurtherawayandnegativeslopesrepresentpointsthatarecloser.PointsinÞnitelyfarawaywillhaveaslopeof1(forexample,thedashedline)Althoughnotshownin (s,t)(s',t')(s'',t'') CF3 r F2F1 Figure9:Bychangingtheshapeorplacementfocalsurfaces,wecandynamicallycontrolwhichsamplesineachdatacamerawillcontributetothereconstructedray. theÞgure,variationincoloralongthelineinrayspacerepresentstheview-dependentradianceofthepoint.Ifthesamesetofraysisreparameterizedusinganewfocalplanethatisparalleltotheplane,thesliceshowninFigure10cresults.Thesetwoslicesarerelatedbyasheartransformationalongthedashedline.Ifthefocalplaneisorientedsuchthatitisnotparallelwiththecamerasurface,aswith,thenthesliceistransformednon-linearly,asshowninFigure10d.However,eachhorizontallineofinFigure10disalinearlytransformed(i.e.scaledandshifted)versionofthecorrespondinghorizontallineofconstantFigure10b.Insummary,dynamicreparameterizationoflightÞeldsamountstoasimpletransformationofrayspace.Whenthefocalsurfaceremainsperpendiculartothecamerasurfacebutitspositionischanging,thisresultsinasheartransformationofrayspace.Changingthefocalplanepositionthusaffectswhichray-spacefeatureswillbeaxis-aligned.Thus,wecanuseaseparable,axis-alignedreconstructionÞlteralongwiththefocalplanetoselectwhichfeatureswillbeaxis-aligned,allowingustodynamicallyse-lectwhichfeatureswillbeproperlyreconstructed.Equivalently,onecaninterpretfocalplanechangesasaligningthereconstruc-tionÞltertoaparticularfeatureslope,whilekeepingtherayspaceparameterizationconstant.Undertheinterpretationthatafocalplaneshearsrayspaceandkeepsusesaxis-alignedreconstructionÞlters,ourapertureÞlteringmethodsamounttovaryingtheextentofthereconstructionÞltersalongthedimension.InFigure10e,thedashedhorizontallinesdepicttheextentofthreedifferentapertureÞlters(weassumetheyareinÞnitelythininthedimension).Whencreatingalineimagefromtheray-spaceusingthethreeÞlters,weconstructlineimagesasshowninFigure10f.VaryingtheextentoftheapertureÞlterhastheeffectofÒblurringÓfeatureslocatedfarfromthefocalplanewhilefeatureslocatednearonthefocalplanewillberela-tivelysharp.However,theÞlterwillreducetheamountofview-dependentradianceforfeaturesalignedwiththeÞlter.IfweshearrayspacetoproducetheparameterizationofFigure10candusethesamethreeÞlters,weproducethelineimagesofFigure10g.6FrequencyDomainAnalysisRayspacetransformationshaveothereffectsonthereconstructionprocess.Sinceshearscanarbitrarilymodifytherelativesamplingfrequenciesbetweendimensionsinrayspace,theypresentconsid-erabledifÞcultieswhenattemptingtobandlimitthesourcesignal.Furthermore,anyattempttobandlimitthesampledfunctionbasedonanyparticularparameterizationwillseverelylimittheÞdelityofthereconstructedsignalsfromthelightÞeld. us UUU SSS us(a)(b)(d)(e) s(f)u Figure12:sliceoftwofeatures.Frequencydomainpowerspectrumofthefeatures.Frequencydomainpowerspec-trum,theredboxrepresentsapossiblereconstructionÞlter.WideaperturereconstructioncorrespondstothethinnerÞltersinfrequencydomain.Resultofsmallaperturereconstruction.Resultoflargeaperturereconstruction. structedimageasghosting(seeFigure12e).Onemethodforreducingthisartifactistoincreasethesizeofthesyntheticaperture.Inthefrequencydomain,thisreducesthewidthofthereconstructionÞltersasshowninFigure12d.Usingthisapproach,wecan,inthelimit,reducethecontributionofspu-riousfeaturestoasmallfractionofthetotalextractedsignalen-ergy.Thepartwecannotextractistheresultofthepre-aliasing.BychoosingsufÞcientlywidereconstructionapertures(ornarrowinthefrequencydomain),theeffectofthepre-aliasingcanbemadeimperceptible(belowourquantizationthreshold).Figure12fisre-constructedbyusingawideraperturethanthatinFigure12e.NotethatthealiasinginFigure12fhaslessenergyandismorespreadoutthaninFigure12e.Thisleadstoageneraltrade-offthatmustbeconsideredwhenworkingwithmoderatelysampledlightÞelds.Wecaneither1)ap-plypreÞlteringatthecostoflimitingtherangeofimagesthatcanbesynthesizedfromthelightÞeldandenduretheblurringandattenua-tionartifactsthatareunavoidableindeepscenesor2)enduresomealiasingartifactsinexchangeforgreaterßexibilityinimagegener-ation.Thevisibilityofaliasingartifactscanbeeffectivelylimitedbyselectingappropriateaperturesforagivendesiredimage.7RenderingAsinthelumigraphandlightÞeldsystems,wecanconstructade-siredimagebyqueryingraysfromtheraydatabase.Givenarbitrarycameras,camerasurfaces,andfocalsurfaces,onecanray-tracethedesiredimage.Ifthedesiredcamera,datacameras,camerasurface,andfocalsurfaceareallplanar,thenatexturemappingapproachcanbeusedsimilartothatproposedbythelumigraphsystem.Weextendthetexturemappingmethodusingmulti-texturingforren-deringwitharbitrarynon-negativeapertureÞlters,includingbilin-earandhigherorderÞlters.7.1MemoryCoherentRayTracingWeÞrstdescribeamemorycoherentraytracingmethodwiththefollowingpseudo-code.Insteadofrenderingpixelbypixelinthedesiredimage,wecanrenderthecontributionofeachdatacamerasequentially.Thiscausesustowritetoeachpixelinthedesiredimagemanytimes.Theintersectiontechniquesarethoseusedinstandardraytrac-ing.Inthefollowingdescription,arayhasacolor.Likewise,apixelinthedesiredimagehasacolor.Letbethedesiredcamerawithacenterofandpixelsonitsimageplane.Letbetheapertureweightingfunction,whereisthewidthoftheaperture.InitializetheframebuffertoblackForeachdatacameraapolygonondeÞnedbyprojectionofontothedesiredimageplaneForeachpixeltheraythroughtoget ):= ):= 7.2TextureMappingAlthoughtheraytracingmethodissimpletounderstandandeasytoimplement,therearemoreefÞcientmethodsforrenderingwhenthecamerasurface,imagesurface,andfocalsurfaceareplanar.Weextendthelumigraphtexturemappingapproach[7]tosupportdy-namicreparameterization.Werenderthecontributionofeachdatausingmulti-texturingandanaccumulationbuffer[9].Ourmethodworkswitharbitrarynon-negativeaperturefunctions.Multi-texturing,supportedbyMicrosoftDirect3D7Õstexturestages[14],allowsasinglepolygontohavemultipletexturesandmultipleprojectivetexturecoordinates.Ateachpixel,twosetsoftexturecoordinatesarecalculated,andthentwotexelsareaccessed.Thetwotexelsaremultiplied,andtheresultisstoredintheframebuffer.WewritetotheframebufferusingtheDirect3DalphamodeÒsource+destination,Ówhichmakestheframebufferactasa8-bit,full-coloraccumulationbuffer.OurrenderingtechniqueisillustratedinFigure13.Foreach,wecreatearectangularpolygononthecamerasurfacewithcoordinates.WethenprojectthispolygonontothedesiredcameraÕsimageplaneusingaprojec-tionmatrix,givingusapolygon.Thispolygonrepresentstheregionoftheimageplanewhichusessamplesfromthedatacamera.Thatis,onlypixelsinsidepolygonusetexturefromdatacameraWethenprojectontothefocalplaneusingaplanarho-,a3x3matrixwhichchangesoneprojective2-Dbasistoanother.Thisprojectionisdonefromthedesiredcam-Õspointofview.Theresultingpolygonliesonthefo-calplane.Finally,weusethemappingtocalculatethepixelvaluesforthepolygon.ThisgivesusapolygonwhichrepresentsthetexturecoordinatesforpolygonWecancomposemanyoftheseoperationsintoasinglema-trix,whichtakesusdirectlyfrompolygontotexturecoor-.Thismatrixcanbewrittenas LeftEyeRightEye LensArray F Figure16:AlightÞeldcanbereparameterizedintoadirectlyviewedlightÞeld,whichoperatesontheprincipalsofintegralphotography. Figure15:AnautostereoscopiclightÞeldisdrawnintheback-ground.Thesceneisathree-dimensionalversionofthesmallinsetpictureintheupperleftcorner.Thezoomis400%magniÞcation. lensarrayisplacedinfrontoftheobject,thentheobjectwillappearbehindthedisplay.BecausethelensarrayimageisrenderedfromalightÞeldandnotdirectlyfromanintegralcamera,wecanplacethelensarrayimagebehindthecapturedobject,andtheobjectwillappeartoßoatinfrontofthedisplay.9ResultsThelightÞelddatasetsshowninthispaperwerecreatedasfollows.ThetreedatasetwasrenderedinPovray3.1.Itiscomposedof256(16x16)imageswithresolutionsof320x240.ThecaptureddatasetswereacquiredwithanElectrimEDC1000ECCDcamera(654x496)withaÞxed-focallength16mmlensmountedonanX-YmotionplatformfromArrickRobotics(30Óx30Ódisplacement).Foreachsetwecapturedeither256(16x16)or1024(32x32)pictures.Tocalibratethecamera,weoriginallyusedaFaroArm(asub-millimeteraccuratecontactdigitizer)tomeasurethespatialcoordinatesoftargetsonathree-dimensionalcalibrationpattern.Usinganimageofthetargets,weusedtheTsai-Lenzcameracal-ibrationalgorithm[26]whichreportedfocallength,CCDsensorelementaspectratio,principlepoint,andextrinsicrotationalorien-tation.Theradiallensdistortionreportedforourlenswaslessthan1pixelper1000pixels,andwedecidednottocorrectforit.Finally,weresampledtheraw654x496imagesdownto327x248beforeusingthemasinputtotherenderer.Recently,wehaveexperimentedwithanalternativecalibrationmethodthatrequiresno3DmeasurementsandusestheactuallightÞeldimagesratherthanimagesofaspecialcalibrationobject.ThisapproachhassomesimilaritieswithafamilyoftechniquesknownasÒself-calibrationÓ[5].However,ourlightÞeldcapturesystem,whereanon-rotatingcameraistranslatedinaplane,isadegener-atecasefortrueself-calibration[25].Instead,wealigntheepipolargeometriesofthesourceimagesratherthancomputeanactualcal-ibration.Ourmethodamountstoatwo-axisrectiÞcation[1].IfthecameraÕsopticalaxisisroughlyperpendiculartotheplaneofmotionandtheX-YplatformÕsmotionisreasonablyaccurate,thensucharectiÞcationiseasilyfound.WeÞndtheepipolarplanesin-ducedbythehorizontalandverticalcameramotionbytrackingafewcorrespondingimagepoints.Thiscanbedoneeitherautomat-icallyorbyhand.TheimagesarethenrectiÞedbyarotatingtheirepipolesontothelineatinÞnity.TheÞnalrectiÞedimageswillhavevalidhorizontalandverticalEPIstructuresandcanbeimme-diatelyusedinourlightÞeldviewer.However,whenausernavi-gateswithinthescenetheymightnoticeaprojectivedistortionofthespace.Thisdistortioncanbeamelioratedbyallowingtheusertointeractivelyadjustthefocallengthofthedatacameras.Other-wise,thefocallengthcanbeestimatedbymeasuringafewpointsinthescene.Inourexperiments,ithasbeeneasytocreatelightÞeldsusingthismethod,andtheÞnalresultsarecomparabletoourstrictlycalibrateddatasets.Ourautostereoscopicimageswereprintedat300dpionaTek-tronicsPhaser440dye-sublimationprinteranduseaFresnelTech-nologies#300HexLensArray,withapproximately134lensespersquareinch[6].10FutureWorkLightÞeldscontainalargeamountofredundancywhichwewouldliketoexploit.Wewouldliketodevelopanalgorithmforopti-mallyselectingafocalplane,perhapsusingauto-focustechniquessimilartothoseusedinconsumercamcorders.Currently,thefo-calplanemustbeplacedmanually.Inaddition,webelievethereissomepromiseinusingourreparameterizationtechniquesforpassivedepth-from-focusordepth-from-defocusvisionalgorithms[18].IntheleftcolumnofFigure17,wehavecreatedtwoimageswithdifferentfocalsurfacesandalargeaperture.WethenapplyagradientmagnitudeÞltertotheseimages,whichgiveustheout-puttotheright.Theseedgeimagestelluswherein-focus,high-frequencyenergyexists.Wewouldalsoliketoexperimentwithdepth-from-defocusbycomparingtwoimageswithslightlydiffer-entfocalplanesorapertures.