Micro Phase Shifting Mohit Gupta Columbia University New York NY mohitgcs

Presentation on theme: "Micro Phase Shifting Mohit Gupta Columbia University New York NY mohitgcs"— Presentation transcript:

MicroPhaseShiftingMohitGuptaColumbiaUniversityNewYork,NY10027mohitg@cs.columbia.eduShreeK.NayarColumbiaUniversityNewYork,NY10027nayar@cs.columbia.eduAbstractWeconsidertheproblemofshaperecoveryforrealworldscenes,whereavarietyofglobalillumination(in-terre\rections,subsurfacescattering,etc.)andillumi-nationdefocuseectsarepresent.Theseeectsin-troducesystematicandoftensignicanterrorsintherecoveredshape.Weintroduceastructuredlighttech-niquecalledMicroPhaseShifting,whichovercomestheseproblems.Thekeyideaistoprojectsinusoidalpatternswithfrequencieslimitedtoanarrow,high-frequencyband.Thesepatternsproduceasetofim-agesoverwhichglobalilluminationanddefocuseectsremainconstantforeachpointinthescene.Thisen-ableshighqualityreconstructionsofsceneswhichhavetraditionallybeenconsideredhard,usingonlyasmallnumberofimages.WealsoderivetheoreticallowerboundsonthenumberofinputimagesneededforphaseshiftingandshowthatMicroPSachievesthebound.1.IntroductionPhaseshiftingisoneofthemostwidelyusedshapemeasurementtechniques[ 16 ].Becauseofitsprecisionandlowcost,itisappliedinsurgery,factoryautoma-tionanddigitizationofculturalheritage.Likeallactivescenerecoverytechniques,phaseshiftingassumesthatscenepointsareonlydirectlyilluminatedbythelightsource.Asaresult,itrecoverserroneousshapesforscenesthathaveglobalilluminationduetointerre\rec-tionsandsubsurfacescattering.Furthermore,conven-tionalphaseshiftingalgorithmsassumethatillumina-tioncomesfromaperfectpointsourcewithinnitedepthofeld.Inpractice,allsourceshavealimiteddepthofeld,resultingindefocus.Inordertoaccountfordefocus,existingtechniquesneedtocapturealargenumberofinputimages.Itisworthnotingthatglobalilluminationanddefocuseectsareubiquitous-theyariseinvirtuallyanyrealworldscene.Inthispaper,weintroduceashaperecoverytech-niquecalledMicroPhaseShifting(MicroPS)whichaddressestheproblemsofglobalilluminationandillu-minationdefocus 1 .Whiletheseproblemshaveseena 1Wedonotconsidertheeectsofcameradefocus.Cameradefocusresultsinblurringofcapturedimages,resultinginincor-rectdepths,speciallyatdepthedges.lotofresearchactivityinthelastfewyears[ 7 , 4 , 8 , 19 , 5 ],forthersttime,wepresentatechniquewhichisfast,accurateandwidelyapplicable.Thekeyideaistoprojectsinusoidalpatternswithspatialfrequencieslimitedtoanarrowhigh-frequencyband.ThewordMicroreferstothesmallvaluesforboththebandwidthandperiodsofthepatterns.Thebandwidthaswellastheperiodsarechosentobesmallenoughsothatforeachscenepoint,bothglobalilluminationanddefocuseectsremainconstantoveralltheinputimages.Incontrast,conventionalphaseshiftinghaspatternswithabroadrangeofspatialfrequencies.Figure 1 showscomparisonsbetweenconventionalandMicroPS.Gettingaroundglobalillumination:Nayaretal.showedthathigh-frequencypatternscanbeusedtoseparateglobalilluminationfromthedirectcom-ponent[ 14 ].Severalsubsequentphaseshiftingtech-niques[ 4 , 7 ]havebuiltuponthismethod.Thesetech-niquesmodulatethelowfrequencypatternswithhigh-frequencysinusoidstoremovetheglobalilluminationcomponent.Incontrast,MicroPSavoidstheproblemofglobalillumination.Sinceallthepatternsarehighfrequencysinusoids,globalilluminationforanygivenscenepointremainsconstantoverallthecapturedim-agesandhencedirect-globalseparationisnotrequired.Asweshowinourresults,thissignicantlyimprovesthereconstructionquality.Invariancetodefocus:Illuminationdefocusmani-festsasalow-passlterontheprojectedimages.Iftheimagesaresinusoids,asinphaseshifting,defocusreducestheiramplitude.Theamplitudevariessignif-icantlywiththesinusoidfrequency,aswellasacrossthesceneduetovaryingamountofdefocus,asshowninthesecondrowofFigure 1 .InconventionalPS,theamplitudesfordierentfrequenciesneedtobetreatedasseparateunknowns.InMicroPS,sinceallthefre-quenciesareinanarrowband,theamplitudesforallthefrequenciesareapproximatelythesame,andcanbetreatedasasingleunknown.Asaresult,thenumberofinputimagesrequiredforMicroPSisconsiderablyreduced.Wederivealowerboundonthenumberofinputimagesrequiredforphase-shiftingandshowthatMicroPSachievesthisbound.Resolvingdepthambiguities:InconventionalPS,highfrequencysinusoidsprovidehigh-resolutionphase(depth)information,albeitinasmalldepthrange. ConventionalPhaseShiftingMicroPhaseShifting Projectedimages !1!2!3!1!2!3Spatialfrequencies:Broadband.Spatialfrequencies:Narrowhigh-freq.band.Numberofimages:3F(forFfrequencies)Numberofimages:F+2(forFfrequencies) Eectofprojectordefocus 0 1 2 3 4 0 0.5 1 TimeIntensity w1 w2 w3 AmplitudeVariation 0 1 2 3 4 0 0.5 1 TimeIntensity w1 w2 w3 LargedepthsceneScenePointPLargedepthsceneScenePointPAmplitudes:Dierentacrossfrequencies.Amplitudes:Sameforallfrequencies. Eectofglobalillumination 0 1 2 3 4 0 0.5 1 TimeIntensity Measured Ground Truth Phase Error 0 1 2 3 4 0 0.5 1 TimeIntensity Measured Ground Truth V-groove!3(Lowfreq.)V-groove!3(Highfreq.)Phaseerrorduetoglobalillumination.Resistanttoerrorsduetoglobalillumination. Figure1.ConventionalversusMicrophaseshifting.Thetoprowshowsprojectedimages.Thesecondrowshowstheeectofprojector(source)defocus.ScenepointPreceivesdefocusedilluminationduetothelargedepthofthescene.TheplotsshowthetemporalintensityprolesatPastheprojectedpatternsareshifted.ForconventionalPS,theproleshavedierentamplitudesfordierentfrequencies.ForMicroPS,sinceallthefrequenciesareinanarrowband,alltheamplitudesaresimilar.Thethirdrowshowstheeectofglobalillumination.ThesceneconsistsofaV-groove.Preceivesglobalilluminationduetointerre\rections.InconventionalPS,interre\rectionsresultinalargephaseerrorforlow-frequencysinusoids.MicroPSisresistanttosucherrorssinceallthefrequenciesarehigh.Thelow-frequencysinusoidsareusedtodisambiguatethephaseinformationoveralargerrange.Thispro-cessiscalledphaseunwrapping[ 20 ].Onemightwon-der:HowcanweperformphaseunwrappingwithMi-croPS,whereonlyhighfrequenciesareused?For-tunately,itispossibletoemulatealow-frequencysi-nusoidwithaperiodequaltotheproductofthepe-riodsofseveralhigh-frequencysinusoids.Therearealgorithmsininterferometryliteraturewhichcombineseveralhigh-frequencyphasevaluesintoasingle,un-ambiguousphase[ 10 , 17 ].Whileoriginallyproposedforinterferometry,thesealgorithmsareeasilyadoptedforphaseshifting.AnexampleisshowninFigure 2 Practicalimplications:WithMicroPS,itisnowpossibletoachievehighqualityreconstructionsofsceneswhichhavetraditionallybeenconsideredhard,whilerequiringonlyasmallnumberofinputimages.Forexample,scenesexhibitingavarietyofglobalillu-minationeects(scattering,interre\rections),orcombi-nationsofmultipleeects-canbehandled.MicroPSdiersfromexistingtechniquesinonlytheprojectedpatterns.Asaresult,itcanbereadilyintegratedwithexistingstructuredlightsystems.1.1.RelatedWorkIn1991,Nayaretal.[ 13 ]addressedtheproblemofinterre\rectionsinthecontextofshaperecovery.Theproblemhasgainedrenewedinterestinthelast fewyears.Severalmethodshavebeenproposedforhandlingdefocusandglobalilluminationeects,e.g.,Chandrakeretal.[ 2 ]andLiaoetal.[ 12 ]forphotomet-ricstereo,andGuptaetal.[ 9 ]forshapefromillumi-nationdefocus.Forstructuredlighttriangulation,techniquesforhandlingglobalilluminationhavebeenproposedtoo.However,mostcurrentmethodsrequireseveraltensorhundredsofimages[ 8 , 6 , 19 , 4 ],requiremovingcam-erasorlightsources[ 11 , 5 , 15 ],andareoftenlimitedtoscenarioswhereonlyoneglobalilluminationeectisdominant[ 8 , 19 ].Chenetal.[ 3 ]usepolarizerstoremovesubsurfacescattering.Thismethoddoesnotaddresstheproblemofinterre\rections.Incomparison,MicroPSrequiresasmallnumber(57)ofinputim-ages,doesnotrequireanymovingpartsandcanhandlesceneswheremultipleglobalilluminationanddefocuseectsarepresentatthesametime.1.2.BackgroundPhaseshiftingbelongstotheclassofactivestereotriangulationtechniques.Thesimplestsetupforthesetechniquesconsistsofaprojectorandacamera.Depthofascenepointiscomputedbyintersectingraysfromthecorrespondingprojectorandcamerapixels.Apro-jectorandacamerapixelcorrespondiftheprojectorpixeldirectlyilluminatesthescenepointimagedatthecamerapixel.Correspondencesbetweencameraandprojectorpixelsareestablishedbyprojectingcodedin-tensitypatternsonthescene.Inphase-shifting,theprojectedpatternsarespatialsinusoidswithdierentfrequenciesf!1;!2;:::;!f;:::;!Fg 23 .Foreachfre-quency,thesinusoidisspatiallyshiftedN(N3)times,andanimageiscapturedforeachshift.Consideranidealizedmodelofimageformation,i.e.,scenepointsreceiveperfectlyfocusedilluminationandtherearenoglobalilluminationeects.Then,Rnf(c),theintensityatacamerapixelcforthenthshiftoffrequency!fisgivenas:Rnf(c)=Of(c)+Af(c)cosf(p)+2n N(1)wherepistheprojectorpixelthatilluminatesthescenepointimagedatc.Foreachfrequency!f,thesetofintensitiesRnf(c)formasinusoidasafunctionofthetimeinstantn.Thesinusoidhasthreeunknownpa-rameters-theosetOf(c),theamplitudeAf(c)andthephasef(p).TheamplitudeAf(c)encapsulatestheBRDFofthescenepoint,theintensityfall-ofromtheprojector,foreshorteningandthecameraintensityresponse.TheosetOf(c)includesthecontribution 2ThenumberoffrequenciesFisdeterminedbythedesiredaccuracyandacquisitionspeed.Morethefrequenciesused(moreinputimages),higherthereconstructionquality.3Inthispaper,frequenciesarerepresentedbytheperiodofthesinusoid,inprojectorpixelunits.ofambientilluminationduetoother(temporallysta-ble)lightsourcesilluminatingthescene.Thephaseprovidesthecorrespondenceinformation.Thus,thegoalistorecoverthephasesf1;2;:::;f;:::;FgTherecoveredphasesarethencombinedintoasingleunambiguousphaseusingtemporalphaseunwrap-ping[ 20 ].Equation 1 canbewrittenasasystemoflinearequations:Rf=MUf(2)whereRfisthevectorofrecordedintensitiesforthefrequency!fMisanN3matrix,thenthrowofwhichisgivenas[1cos(2n N)sin(2n N)].Theun-knownvectorUf=[Of(c)Af(c)cos(f(p))Af(c)sin(f(p))]issolvedusinglinearleast-squares.Thephaseinformationfisobtainedas:Af(c)=p Uf(2)2+Uf(3)2(3)f(p)=acosUf(2) Af(c)(4)2.GlobalilluminationandDefocusInthissection,weanalyzetheeectsofglobalil-luminationandilluminationdefocusonphaseshifting.Theintensityatcamerapixelcinthepresenceofglobalilluminationcanbewrittenas:Rn(c)=XqqcO(c)+A(c)cos(q)+2n N(5)whereqcisthelighttransportcoecientbetweenpro-jectorpixelqandcamerapixelc.Forbrevity,wedropthesubscriptf.Intuitively,becauseofglobalillumina-tion,thescenepointimagedatcreceivesilluminationoriginatingfrommultipleprojectorpixels.Aftersim-plication,weget:Rn(c)=O0(c)+A0(c)cos0(c)+2n Nwhere(6)O0(c)=XqqcO(c)(7)A0(c)=A(c)p P(c)2+Q(c)2(8)0(c)=atanQ(c) P(c)(9)P(c)=Xqqccos((q))(10)Q(c)=Xqqcsin((q))(11)Notethattheabovemodelisvalidforallformsofgloballighttransport-multi-bounceinterre\rections,subsurfacescattering,etc.Duetoglobalillumination,thephase0(c)ofthesinusoid(Eq. 9 )ischanged.Thephaseerrorr=(p)0(c)resultsinsystematicerrorsintherecoveredshape[ 8 , 7 ].AnexampleisshowninFigures 2 (d,f). (a)V-groove(b)Recoveredphasemapsfor3(outof5)frequencies(c)Unwrappedphase 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 PixelsDepth (mm) Conventional PS Multiplexed Modulated PS Micro PS 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 PixelsDepth (mm) Conventional PS Multiplexed Modulated PS Micro PS 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 PixelsDepth (mm) Micro PS: Frequency Set 1 Micro PS: Frequency Set 2 Micro PS: Frequency Set 3 [Optimized] (d)7inputimages(e)10inputimages(f)7inputimagesFigure2.MicroPhaseShiftinginthepresenceofinterre\rections.(a)V-groovescene.(b)Recoveredphasemapsfor3outof5frequencies.Sinceallthefrequenciesbelongtoanarrowhigh-frequencyband,thephasemapshaveambiguities.(c)AsingleunambiguousphasemapiscomputedusingtheG-Salgorithm[ 10 ].Depthcomparisonsfordierentschemesusing(d)7imagesand(e)10images,alongtheredlinein(a).ConventionalPSresultsinlargesystematicerrors.ModulatedPSreducesthesystematicbiasduetointerre\rections,butsuersfromlowSNR.DepthusingMicroPSisnearlyerrorfree.(f)ComparisonsfordierentfrequencysetsforMicroPS.Thefrequencyselectionprocedure(Section 3.2 )isusedtodeterminetheoptimalfrequencyset.Illuminationdefocusmanifestsasalocalblurontheprojected,andhence,observedsinusoids,thusloweringtheiramplitude.Thereducedamplitudeisafunctionofboththefrequencyofthesinusoidandtheamountofdefocus.Asaresult,theobservedamplitudesaredif-ferentfordierentscenepoints.AnexampleisshowninFigure 3 (b).Pointsreceivingglobalanddefocusedilluminationshowlargervariationsintheamplitudewithrespecttotheprojectedfrequency.Becauseofthis,inconventionalphaseshifting,theamplitudesfordierentfrequenciesneedtobetreatedasseparateun-knowns.Theosetsateachscenepoint(Eq. 7 )areindependentofthefrequency,andcanbetreatedasasingleunknown,asshowninFigure 3 (c).3.MicroPhaseShiftingWenowpresentourtechnique,whichaddressestheproblemsofglobalilluminationanddefocus.Thekeyideaissimple-usepatternssothatallthespatialfrequenciesarewithinanarrowhigh-frequencyband.Letthefrequencybandbecharacterizedbythemeanfrequency!m,andthewidthoftheband.Allthefrequencies\n=f!1;:::;!Fgliewithintheband!m 2;!+ 2.ForMicroPS,thefrequencyset\nmustmeetthefollowingtwoconditions:(a)!missu-cientlyhigh(periodissmall)sothatglobalilluminationdoesnotintroduceerrorsintherecoveredphase,and(b)thebandwidthissucientlysmallsothattheam-plitudesforallthefrequenciesareapproximatelythesame,i.e.,A1A2:::AFAThereisatradeowhileselectingthemeanfre-quency!m.Theoretically,thehigher!mis,themoreresistantitistoglobalillumination.However,duetoopticalaberrations,projectorscannotprojectarbitrar-ilyhighfrequenciesreliably.Fortypicalcurrentlyavail-ableprojectorswitharesolutionof1024768pixels,wefoundthatthemeanfrequency!m=16pixelsperiodissucientlyhightopreventglobalilluminationerrorsforalargecollectionofscenes.Similarfrequencieswerechosenin[ 14 ]toseparatethedirectandglobalcompo-nentsofanimage.Whenprojectorswithsmallopticalaberrationsareavailable,patternswithhigherfrequen-ciescanbeused.Forananalyticalexpressionforthesinusoidfrequenciesforwhichglobalilluminationdoesnotin\ruencethephase,pleaseseethetechnicalreportavailableatthefollowinglocation[ 1 Similarly,thereisatradeowhileselectingthebandwidth.Theoretically,thenarrowertheband-width,thefewertheunknowns(duetoinvariancetodefocus).However,duetonitespatialresolutionoftheprojectorsandniteintensityresolutionofthepro-jectorsandcameras,twofrequenciesthatareveryclosecannotbedistinguishedreliably.Thisimposesalower-boundon.Ifistheminimumdierencebetweentwofrequenciessothattheycanbedistinguishedreliably,(F1)Wecomputeempiricallybymeasuringtheampli-tudesfordierentfrequenciesaroundthemeanfre- quency!m.Theamplitudesareaveragedoverseveralscenepointsreceivingdierentamountsofglobalil-luminationanddefocus.Then,ischosentobethelargestvaluesothattheamplitudesforallthefrequen-ciesintheresultingfrequencybandareapproximatelythesame(themaximumdierenceintheamplitudesbetweenanypairoffrequenciesinthebandislessthan1%).Forourprojectorwitharesolutionof1024768pixels,wascomputedtobeapproximately3pixelsfor!m=16pixels.Theresultingfrequencybandis[145175]pixels.Algorithm:Usingtheseproperties,wedesignaphaserecoveryalgorithmwhichrequirescapturingonlyF+2imagesforFfrequencies.Threeimagesarecapturedfortherstfrequency.Subsequently,oneimageiscap-turedforeachoftheremainingF1frequencies:Rn(c)=8�����&#x-3.3;〱&#x-3.3;〱&#x-3.3;〱&#x-3.3;〱&#x-3.3;〱:O(c)+A(c)cos1(p)+(n1)2 3
if1n3O(c)+A(c)cos(n2(p))if4nF+2(12)Theabovesystemofequationscanbesolvedjointlyasasinglelinearsystem:Rmicro=MmicroUmicro(13)whereRmicroisthevectorofrecordedintensities.MmicroisasquarematrixofsizeF+2,andisgivenasMmicro=2666666641a100:::01a1cos(2 3)a1sin(2 3)0:::01a1cos(4 3)a1sin(4 3)0:::0100IF1100377777775(14)whereIF1isanidentitymatrixofsizeF1F1.Theunknownvector,Ufact=26666664O(c)A(c)cos(1(p))A(c)sin(1(p))A(c)cos(2(p))A(c)cos(F(p))37777775(15)iscomputedbysolvingthelinearsystemgiveninEq. 13 .Theindividualphasesarecomputedby:A(c)=p Ufact(2)2+Ufact(3)2(16)f(p)=8:acosUfact(f+1) A(c)iff=1acosUfact(f+2) A(c)if2fF:(17) (a)Scenewithglobalilluminationanddefocus 102 103 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Periodf (1/wf) [pixels]Amplitude [0-1] 102 103 0.1 0.2 0.3 0.4 0.5 0.6 Periodf (1/wf) [pixels]Offset [0-1] (b)Normalizedamplitudes(c)OsetsFigure3.Variationinamplitudewithsinusoidfre-quency(period).(a)Scenewithglobalillumination(sub-surfacescattering,diusion)anddefocus.(b)Amplitudesforpointsmarkedin(a).Becauseofglobalilluminationanddefocus,thevariationinamplitudesisdierentfordierentscenepoints.(c)Theosetsareinvariantacrossfrequen-cies,andcanbetreatedasasingleunknown.3.1.PhaseunwrappingThesetoffrequenciesusedinMicroPSbelongtoanarrowhigh-frequencyband.Thereisnounitfre-quency,whichisconventionallyusedfordisambiguat-ingthehigh-frequencyphases.Thus,conventionalphaseunwrappingalgorithmsarenotapplicablehere.Theproblemofdisambiguatingthephaseswithouttheunitfrequencyhasbeenstudiedintheeldofinter-ferometry[ 10 , 17 ].WiththeGushov-Solodkin(G-S)algorithm[ 10 ],itispossibletocombineseveralhigh-frequencyphasesintoasinglelow-frequencyphase.Iftheperiodsofthehigh-frequencysinusoidsarepairwiseco-prime(nocommonfactors),thenalow-frequencysi-nusoidwithaperiodequaltotheproductoftheperiodsofallthehigh-frequencysinusoidscanbeemulated.ThedetailsrequiredforimplementingthealgorithmandtheMATLABcodearegivenon[ 1 ].Figure 2 showsanexamplewiththewrappedhigh-frequencyphasesandthenalunwrappedphase.3.2.FrequencySelectionforMicroPhaseShiftingHowshouldthespatialfrequenciesoftheprojectedpatternsbeselectedforMicroPS? 4 .Inthissection,weoutlineourfrequencyselectionalgorithmforMicroPS.See[ 1 ]formoredetailsandanalysis.Thefrequenciesshouldbechosensothatdepther-rorsduetoincorrectphasecomputations(resulting 4TheoptimalsetofspatialfrequenciesforconventionalPSformageometricseries,withthesmallestfrequency(largestpe-riod)beingtheunitfrequency[ 18 ]. fromnoiseandglobalillumination)areminimized.Foracamerapixel,supposethecorrectcorrespondenceisprojectorcolumnp,butthecomputedcorrespondenceiscolumnq.Theresultingdeptherrorisproportionaltothephaseerror4=pq.ForF-frequencyMicroPS,eachprojectorcolumnisencodedwithauniqueF+2dimensionalintensityvector.Inordertominimizetheprobabilityofaphaseerror,theop-timalsetoffrequenciesshouldmaximizethedistancedpqbetweenvectorscorrespondingtodistinctprojec-torcolumns.Foragivenfrequencyset\n,theaverageweighteddistancebetweenintensityvectorsis:E(\n)=1 N2NXp;q=1pqdpq(18)whereNisthetotalnumberofprojectorcolumns.Fordpq,weusedthenorm-2Euclideandistance.Then,theoptimalsetoffrequenciesinthefrequencybandd!min;!max]istheonethatminimizesE(\n):\n=argmin\nE(\n);!f22!min;!max8f:(19)ThisisaconstrainedF-dimensionaloptimizationproblem.Weusedthesimplexsearchmethodim-plementedintheMATLABoptimizationtoolboxtosolvethis.Forthefrequencybandof[145175]pixelsandF=5,theaboveprocedurereturnsthefollow-ingfrequencyset:[14571609162416471660]pix-els.Figure 2 (e)showsacomparisonofreconstructionsoftheoptimizedsetversustworandomlyselectedfre-quencysets.Noticethespikesinthedepthmapsforthetwounoptimizedsets(rsttwo).Incontrast,thereconstructionfortheoptimizedsetiserror-free.3.3.ProofofOptimalityWenowshowthatMicroPhaseShiftingistheoret-icallyoptimalintermsofthenumberofinputimages.Lemma1TheminimumnumberofimagesrequiredforF-frequencyphase-shiftingisF+2.Proof1Theproofinvolvescountingthenumberofunknownsforeachcamerapixel.Forapixel,theun-knownsare(a)thedirectilluminationcomponent,(b)theambient+globalilluminationcomponent,and(c)aphasevalueforeachfrequency.GivenFfrequencies,thetotalnumberofunknownsisF+2.Sincethesystemofequationsislinear,theminimumnumberofequa-tions(images)requiredforF-frequencyphase-shiftingisF+2.Corollary1MicroPhaseShiftingistheoreticallyop-timalintermsofthenumberofinputimages.ThisdirectlyfollowsfromLemma1andthefactthatMicroPSrequiresF+2imagesforF-frequencyphaseshifting 5 . 5ThelowerboundofFforMicroPSis2,sinceaminimumof2high-frequencysinusoidsarerequiredtoemulatealow-frequency3.4.ComparisonwiththeStateoftheArtModulatedphaseshiftingneedstoexplicitlysep-arateglobalilluminationbymodulatingthelow-frequencypatternswithhigh-frequencysinusoids[ 4 , 7 Eachlow-frequencypatternismodulatedbyatleast3high-frequencyshiftingsinusoids.Sinceaminimumof3low-frequencypatternsarerequiredtocomputethephase,atleast9imagesareneededperlowfrequency.Recently,Guetal.[ 7 ]showedthatbyusingmulti-plexedillumination,thenumberofinputimagesforeachlowfrequencycanbereducedto7.LetFlbethenumberoflowfrequenciesused,andFhbethenumberofhighfrequencies.Sincethehigh-frequencypatternsdonotrequireexplicitsepa-ration[ 14 ],theminimumnumberofimagesrequiredwiththecurrentstateoftheartis7Fl+3Fh.ForMicroPS,explicitdirect-globalseparationisnotre-quired.Asaresult,MicroPSneedsonlyFl+Fh+2inputimages.Forexample,inordertomeasurephaseforonehighfrequencyandonelow-frequency,modu-latedphaseshiftingwouldrequire7+3=10images.Incontrast,givenabudgetof10images,MicroPScanuse8dierentfrequencies,resultinginsignicantlyhigherqualityreconstructions.4.ExperimentsForourexperiments,weusedaSanyoPLC-XP18Nprojector.Thenumberofprojectorcolumnsis1024.Hence,unitfrequencypatternshaveaperiodof1024pixels.FormodulatedPS,atleast7inputimagesareneeded(Section 3.4 ).Givenabudgetofseveninputimages,onlyasingleunitfrequencycanbeused.Incaseof10inputimages,onehighfrequencyandoneunitfrequencycanbecaptured.ForconventionalPS,abudgetof7and10inputimagesallows2and3fre-quencies,respectively.Cameradefocuswasminimizedbycapturingimageswithasmallcameraaperture.Figure 2 showsdepthrecoveryresultsforaV-groovewithinterre\rections.AsshowninFigures 2 (d,e),con-ventionalPS(redcurve)resultsinsignicantaswellassystematicerrors.ModulatedPSreducesthesystem-aticbiasbyseparatingtheeectsofinterre\rections.However,theresultingdepthestimatessuerduetolowSNRoftheseparateddirectilluminationcompo-nent.Incontrast,MicroPSproducesnearlyerror-freeresultswiththesamenumberofinputimages.Figures 4 (a-e)showshaperecoveryresultsforsceneswithdierentglobalilluminationanddefocuseects.Theceramicbowlhasstronginterre\rections.Thelemonskinistranslucent,resultinginsubsurfacescat-tering.Thedepth-rangefortheRussiandollssceneislarge,resultinginilluminationdefocus.Thewaxbowlischallengingbecauseithasbothstronginterre\rectionsandsubsurfacescattering sinusoid.Inpractice,Fdependsonthecomplexityofthescene.Inourexperiments,3F5wasfoundtobesucient. ConventionalPSresultsinlargeandsystematicer-rorsduetointerre\rectionsandlowSNRinlowalbedoregions(e.g.,regionsontheRussiandolls).ModulatedPSrequiresalargenumberofimagesperfrequency.Withabudgetof7images,onlytheunitfrequencycanbeacquired,whichisnotsucientforaccuraterecon-struction.Moreover,theexplicitseparationofdirectcomponentfromthecapturedimagesfurtherenhancesthenoise.Theproblemisespeciallysevereforregionsoflowalbedoandhighlytranslucentmaterials(waxbowlandlemon),forwhichthedirectcomponentisasmallfractionofthetotalradiance.MicroPSdoesnotrequireexplicitseparationandcapturesmanymorespatialfrequenciesgiventhesameimagebudget,thusresultinginhighqualityreconstructions.Themetalbowlillustratesafailurecase.Duetohigh-frequencyinterre\rections,thereconstructedshapesforalltheschemeshavelargeholesanderrors.Formoreresults,see[ 1 5.DiscussionScopeandlimitations:WhileMicroPSreducestheerrorsduetoglobalillumination,itmaynotcompletelyremovethem.Forexample,inthepresenceofhigh-frequencylighttransportsuchasmirrorinterre\rec-tions,MicroPSispronetoerrors(seeFigure 4 (e)foranexample).However,forsceneswherethefre-quencyoflighttransportislessthanthefrequencyofthesinusoidsused,MicroPSwillmitigatetheeectsofglobalillumination.Errorandresolutioncharacteristics:AlthoughtheerrorcharacteristicsofMicroandConventionalPSaredierentduetodierentprojectedpatternsanddecodingalgorithms,bothcanresolvecorrespondenceswithsub-pixelaccuracy,asbothbelongtotheclassofcontinuouscodingschemes.However,incaseoflowSNR,theresolutionisoftenlimitedbylightsourceandcameranoise.Thereisatradeobetweendepthresolu-tionandthenumberofinputimages.Intheexamplesshowninthepaper,arelativelysmallnumberofinputimagesareused,resultinginalowSNR.Ifmoreim-agesareused,noiseceasestobethelimitingfactorandsub-pixelresolutioncanbeachieved.PolarizationandMicroPS:Polarizationhasbeenusedtoreducetheeectofsub-surfacescatteringinphaseshifting[ 3 ].ThisapproachcanalsobeusedinconjunctionwithMicroPSbyplacingpolarizersinfrontofthecameraandtheprojector.Frequencyselection:Thefrequencyselectionalgo-rithm(Section 3.2 )doesnotnecessarilyyieldtheopti-malfrequencyset.Theoptimalsetoffrequenciesde-pendsonthenoiselevelsandresolutionsoftheprojec-torandthecamera,scene'salbedosandlighttransportcharacteristics.Whilehighfrequenciesarepreferredforcopingwithglobalillumination,theyarealsomorepronetoamplitudeloss,andhence,lowSNR.Infuture,weenvisageincorporatingtheprojectorandcameracharacteristicsandcoarselighttransportcharacteris-ticsofthescene(acquiredwithafewinitialmeasure-ments)inthefrequencyselectionalgorithm.Acknowledgments:ThisresearchwassupportedinpartsbyNSF(grantnumberIIS09-64429)andONR(grantnumberN00014-11-1-0285).References[1]Projectwebpage.http://www.cs.columbia.edu/CAVE/projects/MicroPhaseShifting/. 4 , 5 , 7 [2]M.Chandraker,F.Kahl,andD.Kriegman.Re\rectionsonthegeneralizedbas-reliefambiguity.InCVPR,2005. 3 [3]T.Chen,H.P.A.Lensch,C.Fuchs,andH.peterSeidel.Po-larizationandphase-shiftingfor3Dscanningoftranslucentobjects.InCVPR,2007. 3 , 7 [4]T.Chen,H.-P.Seidel,andH.P.A.Lensch.Modulatedphase-shiftingfor3Dscanning.InCVPR,2008. 1 , 3 , 6 [5]C.Hermans,Y.Francken,T.Cuypers,andP.Bekaert.Depthfromslidingprojections.InCVPR,2009. 1 , 3 [6]V.Couture,N.Martin,andS.Roy.Unstructuredlightscanningtoovercomeinterre\rections.InICCV,2011. 3 [7]J.Gu,T.Kobayashi,M.Gupta,andS.K.Nayar.Mul-tiplexedilluminationforscenerecoveryinthepresenceofglobalillumination.InICCV,2011. 1 , 3 , 6 [8]M.Gupta,A.Agrawal,A.Veeraraghavan,andS.Narasimhan.Structuredlight3Dscanningthepresenceofglobalillumination.InCVPR,2011. 1 , 3 [9]M.Gupta,Y.Tian,S.G.Narasimhan,andL.Zhang.Acombinedtheoryofdefocusedilluminationandgloballighttransport.InIJCV,Toappear. 3 [10]V.I.GushovandY.N.Solodkin.Automaticprocessingoffringepatternsinintegerinterferometers.OpticsLasersEngineering,14,1991. 2 , 4 , 5 [11]M.Holroyd,J.Lawrence,andT.Zickler.Acoaxialopti-calscannerforsynchronousacquisitionof3Dgeometryandsurfacere\rectance.ACMTrans.Graph.,29(3),2010. 3 [12]M.Liao,X.Huang,andR.Yang.Interre\rectionremovalforphotometricstereobyusingspectrum-dependentalbedo.InCVPR,2011. 3 [13]S.K.Nayar,K.Ikeuchi,andT.Kanade.ShapefromInter-re\rections.IJCV,6(3),1991. 2 [14]S.K.Nayar,G.Krishnan,M.D.Grossberg,andR.Raskar.Fastseparationofdirectandglobalcomponentsofasceneusinghighfrequencyillumination.ACMTrans.Graph.,25(3),2006. 1 , 4 , 6 [15]J.ParkandA.C.Kak.3Dmodelingofopticallychallengingobjects.IEEETVCG,2008. 3 [16]J.Salvi,S.Fernandez,T.Pribanic,andX.Llado.Astateoftheartinstructuredlightpatternsforsurfaceprolometry.PatternRecognition,43,2010. 1 [17]M.Takeda,Q.Gu,M.Kinoshita,H.Takai,andY.Taka-hashi.Frequency-multiplexfourier-transformprolometry:Asingle-shotthree-dimensionalshapemeasurementofob-jectswithlargeheightdiscontinuitiesand/orsurfaceisola-tions.AppliedOptics,36(22),1997. 2 , 5 [18]C.E.Towers,D.P.Towers,andJ.D.C.Jones.Absolutefringeordercalculationusingoptimisedmulti-frequencyse-lectioninfull-eldprolometry.OpticsandLasersinEn-gineering,43,2005. 5 [19]Y.XuandD.Aliaga.Anadaptivecorrespondencealgo-rithmformodelingsceneswithstronginterre\rections.IEEETVCG,2009. 1 , 3 [20]H.Zhao,W.Chen,andY.Tan.Phase-unwrappingal-gorithmforthemeasurementofthree-dimensionalobjectshapes.AppliedOptics,33(20),1994. 2 , 3 SceneMicroPSModulatedPSConventionalPS (a)Ceramicbowl:Interre\rections (b)Lemon:Subsurfacescattering (c)Russiandolls:Illuminationdefocus (d)Waxbowl:Interre\rections+subsurfacescattering (e)Metalbowl:High-frequencyinterre\rectionsFigure4.Shaperecoveryforsceneswithdierentglobalilluminationanddefocuseects.Foreveryscene,7inputimageswereusedforeachscheme.(a)Ceramicbowlwithinterre\rections.(b)Lemonwithsubsurfacescattering.(c)RussianDolls.Largedepth-rangeofthesceneresultsinilluminationdefocus.(d)Waxbowlwithbothinterre\rectionsandstrongsubsurfacescattering.Withonly7inputimages,MicroPSachievesahighqualityreconstruction.Incontrast,conventionalPSresultsinsystematicerrorsduetointerre\rections,andbothconventionalandmodulatedPSsuerduetolowSNRofthedirectirradiancecomponentinlowalbedoandhighlytranslucentregions.(e)Metalbowl(failurecase).Duetohigh-frequencyspecularinterre\rections,reconstructedshapesusingalltheschemeshavelargeholesanderrors.