columbiaedu Shree K Nayar Columbia University New York NY 10027 nayarcscolumbiaedu Abstract We consider the problem of shape recovery for real world scenes where a variety of global illumination in terre64258ections subsurface scattering etc and illu ID: 25367
Download Pdf The PPT/PDF document "Micro Phase Shifting Mohit Gupta Columbi..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
MicroPhaseShiftingMohitGuptaColumbiaUniversityNewYork,NY10027mohitg@cs.columbia.eduShreeK.NayarColumbiaUniversityNewYork,NY10027nayar@cs.columbia.eduAbstractWeconsidertheproblemofshaperecoveryforrealworldscenes,whereavarietyofglobalillumination(in-terre\rections,subsurfacescattering,etc.)andillumi-nationdefocuse ectsarepresent.Thesee ectsin-troducesystematicandoftensigni canterrorsintherecoveredshape.Weintroduceastructuredlighttech-niquecalledMicroPhaseShifting,whichovercomestheseproblems.Thekeyideaistoprojectsinusoidalpatternswithfrequencieslimitedtoanarrow,high-frequencyband.Thesepatternsproduceasetofim-agesoverwhichglobalilluminationanddefocuse ectsremainconstantforeachpointinthescene.Thisen-ableshighqualityreconstructionsofsceneswhichhavetraditionallybeenconsideredhard,usingonlyasmallnumberofimages.WealsoderivetheoreticallowerboundsonthenumberofinputimagesneededforphaseshiftingandshowthatMicroPSachievesthebound.1.IntroductionPhaseshiftingisoneofthemostwidelyusedshapemeasurementtechniques[ 16 ].Becauseofitsprecisionandlowcost,itisappliedinsurgery,factoryautoma-tionanddigitizationofculturalheritage.Likeallactivescenerecoverytechniques,phaseshiftingassumesthatscenepointsareonlydirectlyilluminatedbythelightsource.Asaresult,itrecoverserroneousshapesforscenesthathaveglobalilluminationduetointerre\rec-tionsandsubsurfacescattering.Furthermore,conven-tionalphaseshiftingalgorithmsassumethatillumina-tioncomesfromaperfectpointsourcewithin nitedepthof eld.Inpractice,allsourceshavealimiteddepthof eld,resultingindefocus.Inordertoaccountfordefocus,existingtechniquesneedtocapturealargenumberofinputimages.Itisworthnotingthatglobalilluminationanddefocuse ectsareubiquitous-theyariseinvirtuallyanyrealworldscene.Inthispaper,weintroduceashaperecoverytech-niquecalledMicroPhaseShifting(MicroPS)whichaddressestheproblemsofglobalilluminationandillu-minationdefocus 1 .Whiletheseproblemshaveseena 1Wedonotconsiderthee ectsofcameradefocus.Cameradefocusresultsinblurringofcapturedimages,resultinginincor-rectdepths,speciallyatdepthedges.lotofresearchactivityinthelastfewyears[ 7 , 4 , 8 , 19 , 5 ],forthe rsttime,wepresentatechniquewhichisfast,accurateandwidelyapplicable.Thekeyideaistoprojectsinusoidalpatternswithspatialfrequencieslimitedtoanarrowhigh-frequencyband.ThewordMicroreferstothesmallvaluesforboththebandwidthandperiodsofthepatterns.Thebandwidthaswellastheperiodsarechosentobesmallenoughsothatforeachscenepoint,bothglobalilluminationanddefocuse ectsremainconstantoveralltheinputimages.Incontrast,conventionalphaseshiftinghaspatternswithabroadrangeofspatialfrequencies.Figure 1 showscomparisonsbetweenconventionalandMicroPS.Gettingaroundglobalillumination:Nayaretal.showedthathigh-frequencypatternscanbeusedtoseparateglobalilluminationfromthedirectcom-ponent[ 14 ].Severalsubsequentphaseshiftingtech-niques[ 4 , 7 ]havebuiltuponthismethod.Thesetech-niquesmodulatethelowfrequencypatternswithhigh-frequencysinusoidstoremovetheglobalilluminationcomponent.Incontrast,MicroPSavoidstheproblemofglobalillumination.Sinceallthepatternsarehighfrequencysinusoids,globalilluminationforanygivenscenepointremainsconstantoverallthecapturedim-agesandhencedirect-globalseparationisnotrequired.Asweshowinourresults,thissigni cantlyimprovesthereconstructionquality.Invariancetodefocus:Illuminationdefocusmani-festsasalow-pass lterontheprojectedimages.Iftheimagesaresinusoids,asinphaseshifting,defocusreducestheiramplitude.Theamplitudevariessignif-icantlywiththesinusoidfrequency,aswellasacrossthesceneduetovaryingamountofdefocus,asshowninthesecondrowofFigure 1 .InconventionalPS,theamplitudesfordi erentfrequenciesneedtobetreatedasseparateunknowns.InMicroPS,sinceallthefre-quenciesareinanarrowband,theamplitudesforallthefrequenciesareapproximatelythesame,andcanbetreatedasasingleunknown.Asaresult,thenumberofinputimagesrequiredforMicroPSisconsiderablyreduced.Wederivealowerboundonthenumberofinputimagesrequiredforphase-shiftingandshowthatMicroPSachievesthisbound.Resolvingdepthambiguities:InconventionalPS,highfrequencysinusoidsprovidehigh-resolutionphase(depth)information,albeitinasmalldepthrange. ConventionalPhaseShiftingMicroPhaseShifting Projectedimages !1!2!3!1!2!3Spatialfrequencies:Broadband.Spatialfrequencies:Narrowhigh-freq.band.Numberofimages:3F(forFfrequencies)Numberofimages:F+2(forFfrequencies) E ectofprojectordefocus 0 1 2 3 4 0 0.5 1 TimeIntensity w1 w2 w3 AmplitudeVariation 0 1 2 3 4 0 0.5 1 TimeIntensity w1 w2 w3 LargedepthsceneScenePointPLargedepthsceneScenePointPAmplitudes:Di erentacrossfrequencies.Amplitudes:Sameforallfrequencies. E ectofglobalillumination 0 1 2 3 4 0 0.5 1 TimeIntensity Measured Ground Truth Phase Error 0 1 2 3 4 0 0.5 1 TimeIntensity Measured Ground Truth V-groove!3(Lowfreq.)V-groove!3(Highfreq.)Phaseerrorduetoglobalillumination.Resistanttoerrorsduetoglobalillumination. Figure1.ConventionalversusMicrophaseshifting.Thetoprowshowsprojectedimages.Thesecondrowshowsthee ectofprojector(source)defocus.ScenepointPreceivesdefocusedilluminationduetothelargedepthofthescene.Theplotsshowthetemporalintensitypro lesatPastheprojectedpatternsareshifted.ForconventionalPS,thepro leshavedi erentamplitudesfordi erentfrequencies.ForMicroPS,sinceallthefrequenciesareinanarrowband,alltheamplitudesaresimilar.Thethirdrowshowsthee ectofglobalillumination.ThesceneconsistsofaV-groove.Preceivesglobalilluminationduetointerre\rections.InconventionalPS,interre\rectionsresultinalargephaseerrorforlow-frequencysinusoids.MicroPSisresistanttosucherrorssinceallthefrequenciesarehigh.Thelow-frequencysinusoidsareusedtodisambiguatethephaseinformationoveralargerrange.Thispro-cessiscalledphaseunwrapping[ 20 ].Onemightwon-der:HowcanweperformphaseunwrappingwithMi-croPS,whereonlyhighfrequenciesareused?For-tunately,itispossibletoemulatealow-frequencysi-nusoidwithaperiodequaltotheproductofthepe-riodsofseveralhigh-frequencysinusoids.Therearealgorithmsininterferometryliteraturewhichcombineseveralhigh-frequencyphasevaluesintoasingle,un-ambiguousphase[ 10 , 17 ].Whileoriginallyproposedforinterferometry,thesealgorithmsareeasilyadoptedforphaseshifting.AnexampleisshowninFigure 2 Practicalimplications:WithMicroPS,itisnowpossibletoachievehighqualityreconstructionsofsceneswhichhavetraditionallybeenconsideredhard,whilerequiringonlyasmallnumberofinputimages.Forexample,scenesexhibitingavarietyofglobalillu-minatione ects(scattering,interre\rections),orcombi-nationsofmultiplee ects-canbehandled.MicroPSdi ersfromexistingtechniquesinonlytheprojectedpatterns.Asaresult,itcanbereadilyintegratedwithexistingstructuredlightsystems.1.1.RelatedWorkIn1991,Nayaretal.[ 13 ]addressedtheproblemofinterre\rectionsinthecontextofshaperecovery.Theproblemhasgainedrenewedinterestinthelast fewyears.Severalmethodshavebeenproposedforhandlingdefocusandglobalilluminatione ects,e.g.,Chandrakeretal.[ 2 ]andLiaoetal.[ 12 ]forphotomet-ricstereo,andGuptaetal.[ 9 ]forshapefromillumi-nationdefocus.Forstructuredlighttriangulation,techniquesforhandlingglobalilluminationhavebeenproposedtoo.However,mostcurrentmethodsrequireseveraltensorhundredsofimages[ 8 , 6 , 19 , 4 ],requiremovingcam-erasorlightsources[ 11 , 5 , 15 ],andareoftenlimitedtoscenarioswhereonlyoneglobalilluminatione ectisdominant[ 8 , 19 ].Chenetal.[ 3 ]usepolarizerstoremovesubsurfacescattering.Thismethoddoesnotaddresstheproblemofinterre\rections.Incomparison,MicroPSrequiresasmallnumber(5 7)ofinputim-ages,doesnotrequireanymovingpartsandcanhandlesceneswheremultipleglobalilluminationanddefocuse ectsarepresentatthesametime.1.2.BackgroundPhaseshiftingbelongstotheclassofactivestereotriangulationtechniques.Thesimplestsetupforthesetechniquesconsistsofaprojectorandacamera.Depthofascenepointiscomputedbyintersectingraysfromthecorrespondingprojectorandcamerapixels.Apro-jectorandacamerapixelcorrespondiftheprojectorpixeldirectlyilluminatesthescenepointimagedatthecamerapixel.Correspondencesbetweencameraandprojectorpixelsareestablishedbyprojectingcodedin-tensitypatternsonthescene.Inphase-shifting,theprojectedpatternsarespatialsinusoidswithdi erentfrequenciesf!1;!2;:::;!f;:::;!Fg 23 .Foreachfre-quency,thesinusoidisspatiallyshiftedN(N3)times,andanimageiscapturedforeachshift.Consideranidealizedmodelofimageformation,i.e.,scenepointsreceiveperfectlyfocusedilluminationandtherearenoglobalilluminatione ects.Then,Rnf(c),theintensityatacamerapixelcforthenthshiftoffrequency!fisgivenas:Rnf(c)=Of(c)+Af(c)cos f(p)+2n N(1)wherepistheprojectorpixelthatilluminatesthescenepointimagedatc.Foreachfrequency!f,thesetofintensitiesRnf(c)formasinusoidasafunctionofthetimeinstantn.Thesinusoidhasthreeunknownpa-rameters-theo setOf(c),theamplitudeAf(c)andthephase f(p).TheamplitudeAf(c)encapsulatestheBRDFofthescenepoint,theintensityfall-o fromtheprojector,foreshorteningandthecameraintensityresponse.Theo setOf(c)includesthecontribution 2ThenumberoffrequenciesFisdeterminedbythedesiredaccuracyandacquisitionspeed.Morethefrequenciesused(moreinputimages),higherthereconstructionquality.3Inthispaper,frequenciesarerepresentedbytheperiodofthesinusoid,inprojectorpixelunits.ofambientilluminationduetoother(temporallysta-ble)lightsourcesilluminatingthescene.Thephaseprovidesthecorrespondenceinformation.Thus,thegoalistorecoverthephasesf 1; 2;:::; f;:::; FgTherecoveredphasesarethencombinedintoasingleunambiguousphase usingtemporalphaseunwrap-ping[ 20 ].Equation 1 canbewrittenasasystemoflinearequations:Rf=MUf(2)whereRfisthevectorofrecordedintensitiesforthefrequency!fMisanN3matrix,thenthrowofwhichisgivenas[1cos(2n N) sin(2n N)].Theun-knownvectorUf=[Of(c)Af(c)cos( f(p))Af(c)sin( f(p))]issolvedusinglinearleast-squares.Thephaseinformation fisobtainedas:Af(c)=p Uf(2)2+Uf(3)2(3) f(p)=acosUf(2) Af(c)(4)2.GlobalilluminationandDefocusInthissection,weanalyzethee ectsofglobalil-luminationandilluminationdefocusonphaseshifting.Theintensityatcamerapixelcinthepresenceofglobalilluminationcanbewrittenas:Rn(c)=Xq qcO(c)+A(c)cos (q)+2n N(5)where qcisthelighttransportcoecientbetweenpro-jectorpixelqandcamerapixelc.Forbrevity,wedropthesubscriptf.Intuitively,becauseofglobalillumina-tion,thescenepointimagedatcreceivesilluminationoriginatingfrommultipleprojectorpixels.Aftersim-pli cation,weget:Rn(c)=O0(c)+A0(c)cos 0(c)+2n Nwhere(6)O0(c)=Xq qcO(c)(7)A0(c)=A(c)p P(c)2+Q(c)2(8) 0(c)=atanQ(c) P(c)(9)P(c)=Xq qccos( (q))(10)Q(c)=Xq qcsin( (q))(11)Notethattheabovemodelisvalidforallformsofgloballighttransport-multi-bounceinterre\rections,subsurfacescattering,etc.Duetoglobalillumination,thephase 0(c)ofthesinusoid(Eq. 9 )ischanged.Thephaseerrorr = (p) 0(c)resultsinsystematicerrorsintherecoveredshape[ 8 , 7 ].AnexampleisshowninFigures 2 (d,f). (a)V-groove(b)Recoveredphasemapsfor3(outof5)frequencies(c)Unwrappedphase 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 PixelsDepth (mm) Conventional PS Multiplexed Modulated PS Micro PS 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 PixelsDepth (mm) Conventional PS Multiplexed Modulated PS Micro PS 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 PixelsDepth (mm) Micro PS: Frequency Set 1 Micro PS: Frequency Set 2 Micro PS: Frequency Set 3 [Optimized] (d)7inputimages(e)10inputimages(f)7inputimagesFigure2.MicroPhaseShiftinginthepresenceofinterre\rections.(a)V-groovescene.(b)Recoveredphasemapsfor3outof5frequencies.Sinceallthefrequenciesbelongtoanarrowhigh-frequencyband,thephasemapshaveambiguities.(c)AsingleunambiguousphasemapiscomputedusingtheG-Salgorithm[ 10 ].Depthcomparisonsfordi erentschemesusing(d)7imagesand(e)10images,alongtheredlinein(a).ConventionalPSresultsinlargesystematicerrors.ModulatedPSreducesthesystematicbiasduetointerre\rections,butsu ersfromlowSNR.DepthusingMicroPSisnearlyerrorfree.(f)Comparisonsfordi erentfrequencysetsforMicroPS.Thefrequencyselectionprocedure(Section 3.2 )isusedtodeterminetheoptimalfrequencyset.Illuminationdefocusmanifestsasalocalblurontheprojected,andhence,observedsinusoids,thusloweringtheiramplitude.Thereducedamplitudeisafunctionofboththefrequencyofthesinusoidandtheamountofdefocus.Asaresult,theobservedamplitudesaredif-ferentfordi erentscenepoints.AnexampleisshowninFigure 3 (b).Pointsreceivingglobalanddefocusedilluminationshowlargervariationsintheamplitudewithrespecttotheprojectedfrequency.Becauseofthis,inconventionalphaseshifting,theamplitudesfordi erentfrequenciesneedtobetreatedasseparateun-knowns.Theo setsateachscenepoint(Eq. 7 )areindependentofthefrequency,andcanbetreatedasasingleunknown,asshowninFigure 3 (c).3.MicroPhaseShiftingWenowpresentourtechnique,whichaddressestheproblemsofglobalilluminationanddefocus.Thekeyideaissimple-usepatternssothatallthespatialfrequenciesarewithinanarrowhigh-frequencyband.Letthefrequencybandbecharacterizedbythemeanfrequency!m,andthewidthoftheband.Allthefrequencies\n=f!1;:::;!Fgliewithintheband!m 2;!+ 2.ForMicroPS,thefrequencyset\nmustmeetthefollowingtwoconditions:(a)!missu-cientlyhigh(periodissmall)sothatglobalilluminationdoesnotintroduceerrorsintherecoveredphase,and(b)thebandwidthissucientlysmallsothattheam-plitudesforallthefrequenciesareapproximatelythesame,i.e.,A1A2:::AFAThereisatradeo whileselectingthemeanfre-quency!m.Theoretically,thehigher!mis,themoreresistantitistoglobalillumination.However,duetoopticalaberrations,projectorscannotprojectarbitrar-ilyhighfrequenciesreliably.Fortypicalcurrentlyavail-ableprojectorswitharesolutionof1024768pixels,wefoundthatthemeanfrequency!m=16pixelsperiodissucientlyhightopreventglobalilluminationerrorsforalargecollectionofscenes.Similarfrequencieswerechosenin[ 14 ]toseparatethedirectandglobalcompo-nentsofanimage.Whenprojectorswithsmallopticalaberrationsareavailable,patternswithhigherfrequen-ciescanbeused.Forananalyticalexpressionforthesinusoidfrequenciesforwhichglobalilluminationdoesnotin\ruencethephase,pleaseseethetechnicalreportavailableatthefollowinglocation[ 1 Similarly,thereisatradeo whileselectingthebandwidth.Theoretically,thenarrowertheband-width,thefewertheunknowns(duetoinvariancetodefocus).However,dueto nitespatialresolutionoftheprojectorsand niteintensityresolutionofthepro-jectorsandcameras,twofrequenciesthatareveryclosecannotbedistinguishedreliably.Thisimposesalower-boundon.Ifistheminimumdi erencebetweentwofrequenciessothattheycanbedistinguishedreliably,(F 1)Wecomputeempiricallybymeasuringtheampli-tudesfordi erentfrequenciesaroundthemeanfre- quency!m.Theamplitudesareaveragedoverseveralscenepointsreceivingdi erentamountsofglobalil-luminationanddefocus.Then,ischosentobethelargestvaluesothattheamplitudesforallthefrequen-ciesintheresultingfrequencybandareapproximatelythesame(themaximumdi erenceintheamplitudesbetweenanypairoffrequenciesinthebandislessthan1%).Forourprojectorwitharesolutionof1024768pixels,wascomputedtobeapproximately3pixelsfor!m=16pixels.Theresultingfrequencybandis[145175]pixels.Algorithm:Usingtheseproperties,wedesignaphaserecoveryalgorithmwhichrequirescapturingonlyF+2imagesforFfrequencies.Threeimagesarecapturedforthe rstfrequency.Subsequently,oneimageiscap-turedforeachoftheremainingF 1frequencies:Rn(c)=8�����&#x-3.3;〱&#x-3.3;〱&#x-3.3;〱&#x-3.3;〱&#x-3.3;〱:O(c)+A(c)cos 1(p)+(n 1)2 3if1n3O(c)+A(c)cos( n 2(p))if4nF+2(12)Theabovesystemofequationscanbesolvedjointlyasasinglelinearsystem:Rmicro=MmicroUmicro(13)whereRmicroisthevectorofrecordedintensities.MmicroisasquarematrixofsizeF+2,andisgivenasMmicro=2666666641a100:::01a1cos(2 3) a1sin(2 3)0:::01a1cos(4 3) a1sin(4 3)0:::0100IF 1100377777775(14)whereIF 1isanidentitymatrixofsizeF 1F 1.Theunknownvector,Ufact=26666664O(c)A(c)cos( 1(p))A(c)sin( 1(p))A(c)cos( 2(p))A(c)cos( F(p))37777775(15)iscomputedbysolvingthelinearsystemgiveninEq. 13 .Theindividualphasesarecomputedby:A(c)=p Ufact(2)2+Ufact(3)2(16) f(p)=8:acosUfact(f+1) A(c)iff=1acosUfact(f+2) A(c)if2fF:(17) (a)Scenewithglobalilluminationanddefocus 102 103 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Periodf (1/wf) [pixels]Amplitude [0-1] 102 103 0.1 0.2 0.3 0.4 0.5 0.6 Periodf (1/wf) [pixels]Offset [0-1] (b)Normalizedamplitudes(c)O setsFigure3.Variationinamplitudewithsinusoidfre-quency(period).(a)Scenewithglobalillumination(sub-surfacescattering,di usion)anddefocus.(b)Amplitudesforpointsmarkedin(a).Becauseofglobalilluminationanddefocus,thevariationinamplitudesisdi erentfordi erentscenepoints.(c)Theo setsareinvariantacrossfrequen-cies,andcanbetreatedasasingleunknown.3.1.PhaseunwrappingThesetoffrequenciesusedinMicroPSbelongtoanarrowhigh-frequencyband.Thereisnounitfre-quency,whichisconventionallyusedfordisambiguat-ingthehigh-frequencyphases.Thus,conventionalphaseunwrappingalgorithmsarenotapplicablehere.Theproblemofdisambiguatingthephaseswithouttheunitfrequencyhasbeenstudiedinthe eldofinter-ferometry[ 10 , 17 ].WiththeGushov-Solodkin(G-S)algorithm[ 10 ],itispossibletocombineseveralhigh-frequencyphasesintoasinglelow-frequencyphase.Iftheperiodsofthehigh-frequencysinusoidsarepairwiseco-prime(nocommonfactors),thenalow-frequencysi-nusoidwithaperiodequaltotheproductoftheperiodsofallthehigh-frequencysinusoidscanbeemulated.ThedetailsrequiredforimplementingthealgorithmandtheMATLABcodearegivenon[ 1 ].Figure 2 showsanexamplewiththewrappedhigh-frequencyphasesandthe nalunwrappedphase.3.2.FrequencySelectionforMicroPhaseShiftingHowshouldthespatialfrequenciesoftheprojectedpatternsbeselectedforMicroPS? 4 .Inthissection,weoutlineourfrequencyselectionalgorithmforMicroPS.See[ 1 ]formoredetailsandanalysis.Thefrequenciesshouldbechosensothatdepther-rorsduetoincorrectphasecomputations(resulting 4TheoptimalsetofspatialfrequenciesforconventionalPSformageometricseries,withthesmallestfrequency(largestpe-riod)beingtheunitfrequency[ 18 ]. fromnoiseandglobalillumination)areminimized.Foracamerapixel,supposethecorrectcorrespondenceisprojectorcolumnp,butthecomputedcorrespondenceiscolumnq.Theresultingdeptherrorisproportionaltothephaseerror4 =p q.ForF-frequencyMicroPS,eachprojectorcolumnisencodedwithauniqueF+2dimensionalintensityvector.Inordertominimizetheprobabilityofaphaseerror,theop-timalsetoffrequenciesshouldmaximizethedistancedpqbetweenvectorscorrespondingtodistinctprojec-torcolumns.Foragivenfrequencyset\n,theaverageweighteddistancebetweenintensityvectorsis:E(\n)=1 N2NXp;q=1p qdpq(18)whereNisthetotalnumberofprojectorcolumns.Fordpq,weusedthenorm-2Euclideandistance.Then,theoptimalsetoffrequenciesinthefrequencybandd!min;!max]istheonethatminimizesE(\n):\n=argmin\nE(\n);!f22!min;!max8f:(19)ThisisaconstrainedF-dimensionaloptimizationproblem.Weusedthesimplexsearchmethodim-plementedintheMATLABoptimizationtoolboxtosolvethis.Forthefrequencybandof[145175]pixelsandF=5,theaboveprocedurereturnsthefollow-ingfrequencyset:[14571609162416471660]pix-els.Figure 2 (e)showsacomparisonofreconstructionsoftheoptimizedsetversustworandomlyselectedfre-quencysets.Noticethespikesinthedepthmapsforthetwounoptimizedsets( rsttwo).Incontrast,thereconstructionfortheoptimizedsetiserror-free.3.3.ProofofOptimalityWenowshowthatMicroPhaseShiftingistheoret-icallyoptimalintermsofthenumberofinputimages.Lemma1TheminimumnumberofimagesrequiredforF-frequencyphase-shiftingisF+2.Proof1Theproofinvolvescountingthenumberofunknownsforeachcamerapixel.Forapixel,theun-knownsare(a)thedirectilluminationcomponent,(b)theambient+globalilluminationcomponent,and(c)aphasevalueforeachfrequency.GivenFfrequencies,thetotalnumberofunknownsisF+2.Sincethesystemofequationsislinear,theminimumnumberofequa-tions(images)requiredforF-frequencyphase-shiftingisF+2.Corollary1MicroPhaseShiftingistheoreticallyop-timalintermsofthenumberofinputimages.ThisdirectlyfollowsfromLemma1andthefactthatMicroPSrequiresF+2imagesforF-frequencyphaseshifting 5 . 5ThelowerboundofFforMicroPSis2,sinceaminimumof2high-frequencysinusoidsarerequiredtoemulatealow-frequency3.4.ComparisonwiththeStateoftheArtModulatedphaseshiftingneedstoexplicitlysep-arateglobalilluminationbymodulatingthelow-frequencypatternswithhigh-frequencysinusoids[ 4 , 7 Eachlow-frequencypatternismodulatedbyatleast3high-frequencyshiftingsinusoids.Sinceaminimumof3low-frequencypatternsarerequiredtocomputethephase,atleast9imagesareneededperlowfrequency.Recently,Guetal.[ 7 ]showedthatbyusingmulti-plexedillumination,thenumberofinputimagesforeachlowfrequencycanbereducedto7.LetFlbethenumberoflowfrequenciesused,andFhbethenumberofhighfrequencies.Sincethehigh-frequencypatternsdonotrequireexplicitsepa-ration[ 14 ],theminimumnumberofimagesrequiredwiththecurrentstateoftheartis7Fl+3Fh.ForMicroPS,explicitdirect-globalseparationisnotre-quired.Asaresult,MicroPSneedsonlyFl+Fh+2inputimages.Forexample,inordertomeasurephaseforonehighfrequencyandonelow-frequency,modu-latedphaseshiftingwouldrequire7+3=10images.Incontrast,givenabudgetof10images,MicroPScanuse8di erentfrequencies,resultinginsigni cantlyhigherqualityreconstructions.4.ExperimentsForourexperiments,weusedaSanyoPLC-XP18Nprojector.Thenumberofprojectorcolumnsis1024.Hence,unitfrequencypatternshaveaperiodof1024pixels.FormodulatedPS,atleast7inputimagesareneeded(Section 3.4 ).Givenabudgetofseveninputimages,onlyasingleunitfrequencycanbeused.Incaseof10inputimages,onehighfrequencyandoneunitfrequencycanbecaptured.ForconventionalPS,abudgetof7and10inputimagesallows2and3fre-quencies,respectively.Cameradefocuswasminimizedbycapturingimageswithasmallcameraaperture.Figure 2 showsdepthrecoveryresultsforaV-groovewithinterre\rections.AsshowninFigures 2 (d,e),con-ventionalPS(redcurve)resultsinsigni cantaswellassystematicerrors.ModulatedPSreducesthesystem-aticbiasbyseparatingthee ectsofinterre\rections.However,theresultingdepthestimatessu erduetolowSNRoftheseparateddirectilluminationcompo-nent.Incontrast,MicroPSproducesnearlyerror-freeresultswiththesamenumberofinputimages.Figures 4 (a-e)showshaperecoveryresultsforsceneswithdi erentglobalilluminationanddefocuse ects.Theceramicbowlhasstronginterre\rections.Thelemonskinistranslucent,resultinginsubsurfacescat-tering.Thedepth-rangefortheRussiandollssceneislarge,resultinginilluminationdefocus.Thewaxbowlischallengingbecauseithasbothstronginterre\rectionsandsubsurfacescattering sinusoid.Inpractice,Fdependsonthecomplexityofthescene.Inourexperiments,3F5wasfoundtobesucient. ConventionalPSresultsinlargeandsystematicer-rorsduetointerre\rectionsandlowSNRinlowalbedoregions(e.g.,regionsontheRussiandolls).ModulatedPSrequiresalargenumberofimagesperfrequency.Withabudgetof7images,onlytheunitfrequencycanbeacquired,whichisnotsucientforaccuraterecon-struction.Moreover,theexplicitseparationofdirectcomponentfromthecapturedimagesfurtherenhancesthenoise.Theproblemisespeciallysevereforregionsoflowalbedoandhighlytranslucentmaterials(waxbowlandlemon),forwhichthedirectcomponentisasmallfractionofthetotalradiance.MicroPSdoesnotrequireexplicitseparationandcapturesmanymorespatialfrequenciesgiventhesameimagebudget,thusresultinginhighqualityreconstructions.Themetalbowlillustratesafailurecase.Duetohigh-frequencyinterre\rections,thereconstructedshapesforalltheschemeshavelargeholesanderrors.Formoreresults,see[ 1 5.DiscussionScopeandlimitations:WhileMicroPSreducestheerrorsduetoglobalillumination,itmaynotcompletelyremovethem.Forexample,inthepresenceofhigh-frequencylighttransportsuchasmirrorinterre\rec-tions,MicroPSispronetoerrors(seeFigure 4 (e)foranexample).However,forsceneswherethefre-quencyoflighttransportislessthanthefrequencyofthesinusoidsused,MicroPSwillmitigatethee ectsofglobalillumination.Errorandresolutioncharacteristics:AlthoughtheerrorcharacteristicsofMicroandConventionalPSaredi erentduetodi erentprojectedpatternsanddecodingalgorithms,bothcanresolvecorrespondenceswithsub-pixelaccuracy,asbothbelongtotheclassofcontinuouscodingschemes.However,incaseoflowSNR,theresolutionisoftenlimitedbylightsourceandcameranoise.Thereisatradeo betweendepthresolu-tionandthenumberofinputimages.Intheexamplesshowninthepaper,arelativelysmallnumberofinputimagesareused,resultinginalowSNR.Ifmoreim-agesareused,noiseceasestobethelimitingfactorandsub-pixelresolutioncanbeachieved.PolarizationandMicroPS:Polarizationhasbeenusedtoreducethee ectofsub-surfacescatteringinphaseshifting[ 3 ].ThisapproachcanalsobeusedinconjunctionwithMicroPSbyplacingpolarizersinfrontofthecameraandtheprojector.Frequencyselection:Thefrequencyselectionalgo-rithm(Section 3.2 )doesnotnecessarilyyieldtheopti-malfrequencyset.Theoptimalsetoffrequenciesde-pendsonthenoiselevelsandresolutionsoftheprojec-torandthecamera,scene'salbedosandlighttransportcharacteristics.Whilehighfrequenciesarepreferredforcopingwithglobalillumination,theyarealsomorepronetoamplitudeloss,andhence,lowSNR.Infuture,weenvisageincorporatingtheprojectorandcameracharacteristicsandcoarselighttransportcharacteris-ticsofthescene(acquiredwithafewinitialmeasure-ments)inthefrequencyselectionalgorithm.Acknowledgments:ThisresearchwassupportedinpartsbyNSF(grantnumberIIS09-64429)andONR(grantnumberN00014-11-1-0285).References[1]Projectwebpage.http://www.cs.columbia.edu/CAVE/projects/MicroPhaseShifting/. 4 , 5 , 7 [2]M.Chandraker,F.Kahl,andD.Kriegman.Re\rectionsonthegeneralizedbas-reliefambiguity.InCVPR,2005. 3 [3]T.Chen,H.P.A.Lensch,C.Fuchs,andH.peterSeidel.Po-larizationandphase-shiftingfor3Dscanningoftranslucentobjects.InCVPR,2007. 3 , 7 [4]T.Chen,H.-P.Seidel,andH.P.A.Lensch.Modulatedphase-shiftingfor3Dscanning.InCVPR,2008. 1 , 3 , 6 [5]C.Hermans,Y.Francken,T.Cuypers,andP.Bekaert.Depthfromslidingprojections.InCVPR,2009. 1 , 3 [6]V.Couture,N.Martin,andS.Roy.Unstructuredlightscanningtoovercomeinterre\rections.InICCV,2011. 3 [7]J.Gu,T.Kobayashi,M.Gupta,andS.K.Nayar.Mul-tiplexedilluminationforscenerecoveryinthepresenceofglobalillumination.InICCV,2011. 1 , 3 , 6 [8]M.Gupta,A.Agrawal,A.Veeraraghavan,andS.Narasimhan.Structuredlight3Dscanningthepresenceofglobalillumination.InCVPR,2011. 1 , 3 [9]M.Gupta,Y.Tian,S.G.Narasimhan,andL.Zhang.Acombinedtheoryofdefocusedilluminationandgloballighttransport.InIJCV,Toappear. 3 [10]V.I.GushovandY.N.Solodkin.Automaticprocessingoffringepatternsinintegerinterferometers.OpticsLasersEngineering,14,1991. 2 , 4 , 5 [11]M.Holroyd,J.Lawrence,andT.Zickler.Acoaxialopti-calscannerforsynchronousacquisitionof3Dgeometryandsurfacere\rectance.ACMTrans.Graph.,29(3),2010. 3 [12]M.Liao,X.Huang,andR.Yang.Interre\rectionremovalforphotometricstereobyusingspectrum-dependentalbedo.InCVPR,2011. 3 [13]S.K.Nayar,K.Ikeuchi,andT.Kanade.ShapefromInter-re\rections.IJCV,6(3),1991. 2 [14]S.K.Nayar,G.Krishnan,M.D.Grossberg,andR.Raskar.Fastseparationofdirectandglobalcomponentsofasceneusinghighfrequencyillumination.ACMTrans.Graph.,25(3),2006. 1 , 4 , 6 [15]J.ParkandA.C.Kak.3Dmodelingofopticallychallengingobjects.IEEETVCG,2008. 3 [16]J.Salvi,S.Fernandez,T.Pribanic,andX.Llado.Astateoftheartinstructuredlightpatternsforsurfacepro lometry.PatternRecognition,43,2010. 1 [17]M.Takeda,Q.Gu,M.Kinoshita,H.Takai,andY.Taka-hashi.Frequency-multiplexfourier-transformpro lometry:Asingle-shotthree-dimensionalshapemeasurementofob-jectswithlargeheightdiscontinuitiesand/orsurfaceisola-tions.AppliedOptics,36(22),1997. 2 , 5 [18]C.E.Towers,D.P.Towers,andJ.D.C.Jones.Absolutefringeordercalculationusingoptimisedmulti-frequencyse-lectioninfull- eldpro lometry.OpticsandLasersinEn-gineering,43,2005. 5 [19]Y.XuandD.Aliaga.Anadaptivecorrespondencealgo-rithmformodelingsceneswithstronginterre\rections.IEEETVCG,2009. 1 , 3 [20]H.Zhao,W.Chen,andY.Tan.Phase-unwrappingal-gorithmforthemeasurementofthree-dimensionalobjectshapes.AppliedOptics,33(20),1994. 2 , 3 SceneMicroPSModulatedPSConventionalPS (a)Ceramicbowl:Interre\rections (b)Lemon:Subsurfacescattering (c)Russiandolls:Illuminationdefocus (d)Waxbowl:Interre\rections+subsurfacescattering (e)Metalbowl:High-frequencyinterre\rectionsFigure4.Shaperecoveryforsceneswithdi erentglobalilluminationanddefocuse ects.Foreveryscene,7inputimageswereusedforeachscheme.(a)Ceramicbowlwithinterre\rections.(b)Lemonwithsubsurfacescattering.(c)RussianDolls.Largedepth-rangeofthesceneresultsinilluminationdefocus.(d)Waxbowlwithbothinterre\rectionsandstrongsubsurfacescattering.Withonly7inputimages,MicroPSachievesahighqualityreconstruction.Incontrast,conventionalPSresultsinsystematicerrorsduetointerre\rections,andbothconventionalandmodulatedPSsu erduetolowSNRofthedirectirradiancecomponentinlowalbedoandhighlytranslucentregions.(e)Metalbowl(failurecase).Duetohigh-frequencyspecularinterre\rections,reconstructedshapesusingalltheschemeshavelargeholesanderrors.