Layered Shape Synthesis Automatic Generation of Control Maps for NonStationary Textures - PDF document

ACM Reference Format Rosenberger, A., Cohen-Or, D., Lischinski, D. 2009. Layered Shape Synthesis: Automatic Generation of Control Maps for Non-Stationary Textures. ACM Trans. Graph. 28 , 5, Article 107 (December 2009), 9 pages. DOI = 10.1145/1618452.1618453 http://doi.acm.org/10.1145/1618452.1618453. Copyright Notice Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro Þ t or direct commercial advantage and that copies show this notice on the Þ rst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speci Þ c permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701, fax +1 (212) 869-0481, or permissions@acm.org. © 2009 ACM 0730-0301/2009/05-ART107 $10.00 DOI 10.1145/1618452.1618453 http://doi.acm.org/10.1145/1618452.1618453 LayeredShapeSynthesis:AutomaticGenerationofControlMapsforNon-StationaryTextures AmirRosenberger TelAvivUniversity DanielCohen-Or TelAvivUniversity DaniLischinski TheHebrewUniversity Figure1: Aninhomogeneoustexture,exhibitinganon-uniformmixtureofpeelingpaint,baremetal,andrust.Fromlefttoright:inputtexture exemplar,controlmapextractedfromtheexemplar,alargercontrolmapsynthesizedbyourapproach,andtheresultingnewtexture. Abstract Manyinhomogeneousreal-worldtexturesarenon-stationaryand exhibitvariouslargescalepatternsthatareeasilyperceivedbya humanobserver.Suchtexturesviolatetheassumptionsunderly- ingmoststate-of-the-artexample-basedsynthesismethods.Con- sequently,theycannotbeproperlyreproducedbythesemethods, unlessasuitablecontrolmapisprovidedtoguidethesynthesis process.Suchcontrolmapsaretypicallyeitheruserspeciedor generatedbyasimulation.Inthispaper,wepresentanalternative: amethodforautomaticexample-basedgenerationofcontrolmaps, gearedatsynthesisofnatural,highlyinhomogeneoustextures,such asthoseresultingfromnaturalagingorweatheringprocesses.Our methodisbasedontheobservationthatanappropriatecontrolmap formanyofthesetexturesmaybemodeledasasuperpositionof severallayers,wherethevisiblepartsofeachlayerareoccupied byamorehomogeneoustexture.Thus,givenadecompositionof atextureexemplarintoasmallnumberofsuchlayers,weemploy anovelexample-basedshapesynthesisalgorithmtoautomatically generateanewsetoflayers.Ourshapesynthesisalgorithmisde- signedtopreservebothlocalandglobalcharacteristicsoftheexem- plar'slayermap.Thisprocessresultsinanewcontrolmap,which thenmaybeusedtoguidethesubsequenttexturesynthesisprocess. Keywords: controlmaps,example-basedtexturesynthesis,non- stationarytextures,shapesynthesis 1Introduction Computergeneratedimageryreliesheavilyontexturestoachieve realism.Oneeasywaytoacquirerealistictexturesisbyscanningor takingphotographsofsurfacesandmaterialsthatsurroundusinthe realworld.Therefore,alargenumberofmethodshavebeenpro- posedforsynthesizingtexturesfromexamples,inthelastdecade [Weietal.2009].Manyofthesemethodsareabletoproduceim- pressiveresultswhenappliedtohomogeneoustexturesthatmaybe describedbystationaryMarkovrandomeld(MRF)models.Yet manyrealworldtexturesarehighlyinhomogeneous,andarenot modeledwellbyastationarystochasticprocess. Consider,forexample,therustymetalsurfaceshownontheleftin Figure1.Thetextureonthissurfaceisclearlynon-stationary,andit maybeseenasahighlynon-uniformmixture,orsuperposition,of severaldifferenttextures:peelingpaint,baremetal,andrust.While eachofthesethreetexturesisroughlyhomogeneous,thetexture asawholeisnot.Thisisatypicalsituationformanyrealworld surfaces,whosetextureoftenresultsfromnaturalprocesses,suchas weathering,corrosion,colorcrackingandpeeling,growthofmoss, etc.[DorseyandHanrahan1996;Dorseyetal.1999;Boschetal. 2004;Desbenoitetal.2004;Dorseyetal.2008]. Acommonremedytocopewithsuchtexturesistoguidethesyn- thesisprocessbyacontrolmapthatencodesthelargescalevaria- tionsandthenon-localfeaturesofthedesiredoutputtexture(e.g., [Ashikhmin2001;Hertzmannetal.2001;Zhangetal.2003;Wang etal.2006;Weietal.2008]).However,suchcontrolmapsaretyp- icallyeitheruser-speciedorproducedbyacustomtailoredsimu- lation(e.g.,biologicalorphysically-based). Inthisworkweproposeanewmethodforautomaticallygenerat- ingcontrolmapsfromexamples,gearedatnaturaltexturessuchas theoneinFigure1.Asobservedabove,suchtexturesoftenlook likeasuperpositionofseverallayers,whereeachvisibleregionsof eachlayerareoccupiedbyamorehomogeneoustexture.Theshape ofthetexture-occupiedregionsineachlayerisfarfromarbitrary. Rather,itistheconsequenceofthespecicnaturalprocessthatpro- ducedthistexture,aswellastheshapeofthelayerunderneath.Nei- therglobalstatistics,norsmallneighborhoodsarecapableoffaith- fullycapturingsuchhigherlevelstructures.Suchappearancesmay begeneratedbyspecializedshadersorbyphysically-basedsimu- lations.However,wearenotawareofanygeneralfullyautomatic wayforgeneratingsuchashaderfromaspecicexample. Ourapproachbeginsbydecomposingtheinputexemplarintoa numberoflayers,whichweorderbottomtotop.Anovelexample- based shapesynthesis algorithmisthenusedtogenerateanewset oflayers,whoselocalandglobalcharacteristicsvisuallyresemblethoseoftheexemplar'slayers.Thisalgorithmmakesuseofabidi-rectionalmeasureofsimilaritybetweentheshapesofthelayers,whichisbasedontheshapes'boundaries.Startingfromsomeini-tialoutputshape,weiterativelyoptimizetheshapewithrespecttothissimilaritymeasure.Oncethenewlayersareavailable,atexturetransferprocessbasedon“texture-by-numbers”[Hertzmannetal.2001]isinvoked,resultinginthenaloutputtexture,suchastheresultshowninFigure1.Insummary,themainnoveltyinourapproachliesinexample-basedsynthesisofasuitablecontrolmap,ratherthanworkingdirectlyonthetexture,oronsomeassociatedappearancespace[LefebvreandHoppe2006].Toourknowledge,suchanapproachhasnotbeenexploredbefore.2RelatedWorkExample-basedtexturesynthesishasenjoyedconsiderableresearchattentioninrecentyears.Mostoftherelevantpreviousmeth-odsmayberoughlyclassiedtoparametricmethods[HeegerandBergen1995],andnon-parametricmethods,whichincludepixel-basedmethods[EfrosandLeung1999;WeiandLevoy2000],patch-basedmethods[EfrosandFreeman2001;Kwatraetal.2003],optimization-basedmethods[Kwatraetal.2005],andappearance-spacetexturesynthesis[LefebvreandHoppe2006].Parametricmethodsattempttoconstructaparametricmodelofthetexturebasedontheinputsample,whichhasproventobeachal-lengingtask,andaremostlysuccessfulwithstructurelessstation-arytextures.Non-parametricmethodshavedemonstratedtheabil-itytohandleawidervarietyoftextures,bygrowingthetextureonepixel/patchatatime.Optimization-basedmethodsevolvethetextureasawhole,furtherimprovingthequalityoftheresultsandmakingthesynthesismorecontrollable.Wereferthereaderto[Weietal.2009]foramorecomprehensiveoverviewofexample-basedtexturesynthesis.Whilenon-parametricmethodsaretypicallyabletoreproducesmallscalestructure,theyhaveadifcultycopingwithhighlyin-homogeneoustextures,sincesuchtexturescannotbemodeledbyastationaryMarkovRandomField(MRF)model,whichprovidesthetheoreticalbasisformostofthesemethods.Inordertohandlesuchtexturesandcontrollargescalestructure,Ashikhmin[2001]proposedtoguidethesynthesisprocessbyauser-providedtargetimage,whichspeciesthelocalaveragecolorsacrossthetargettexture.Texture-by-Numbers[Hertzmannetal.2001]extendsthisideafurtherbyaugmentingtheinputexemplarwithalabelmap,whereregionswithdistincttexturearedistinguishedbydifferentlabels.Asuitablelabelmapmaybepaintedmanuallybytheuser,orcreatedautomaticallyusingunsupervisedimagesegmentation.Tosynthesizeanewimage,atargetlabelmapisprovided,whichindicateshowthedifferenttexturesshouldbearrangedinthere-sultingimage.However,thatworkaddressedneithertheissueofautomaticallygeneratingalabelmapfornaturalinhomogeneoustextures,northeautomaticsynthesisofthetargetlabelmap,aswedoinourwork.Manyotherworkssincemadeuseofcontrolmapswhensynthesiz-ingnon-stationarytextures,forexample[Zhangetal.2003;Wangetal.2006;Guetal.2006;Luetal.2007;Weietal.2008].How-ever,inalloftheseworksthecontrolmapforthetargettextureiseitherprovidedbytheuser,orderivedfromaspecicmodeloftex-tureformationacrossa3Dsurface(e.g.,[Luetal.2007]),andwearenotawareofanypreviousattemptsofexample-basedcontrolmapgeneration.Ourshapesynthesisapproachisrelatedtotextureoptimizationtechniques[Wexleretal.2004;Kwatraetal.2005],whichsynthe-sizetexturesbyminimizingatextureenergyfunction.Thisfunc-tionconsistsofasumoflocaltermsmeasuringhowcloseeachsynthesizedtexturepatchistoanexemplarpatch.However,thisformulationdoesnotaccountforthepossibilitythattheremaybemanyotherpatchesintheexemplarthatarenotrepresentedatallinthesynthesizedresult.Whilethismaybeadequateforhomo-geneoustextures,wheremostpatchesaresimilartoeachother,thequalityoftheresultsforinhomogeneoustexturesisoftencompro-mised.Whileitispossibletoinjectsomeglobalstatisticsintotheoptimization[Kopfetal.2007],theresultingprocessstillfailstocapturethelargescaleappearanceofhighlyinhomogeneousnatu-raltexturesthatarethetargetofthiswork.Incontrast,weperformshapesynthesiswithabidirectionalsimilaritymeasure(inspiredbySimakovetal.[2008]andWeietal.[Weietal.2008]),anddemon-stratemorefaithfulreproductionofappearanceinthecomparisonswepresentinSection4.Appearance-spacetexturesynthesis[LefebvreandHoppe2006]isanotheroptimizationmethodthatoperatesinafeaturespace,ratherthanusingthevaluesofpixelsorsmallpatchesdirectly.Apointcorrespondingtoapixelinamoregeneralfeaturespacemayen-codemoreinformation,allowingstructuretobereproducedbetter.Thelayermapthatweassociatewiththeinputexemplarinourapproachcouldbeviewedasafeaturespacecustom-tailoredforsynthesisoflayeredinhomogeneoustextures.Avarietyofmethodsgeneratetexturesofweatheredsurfacesbyassumingandsimulatingaphysicalmodel[DorseyandHanrahan1996;Dorseyetal.1999;Merillouetal.2001;Boschetal.2004;Desbenoitetal.2004;Dorseyetal.2008].Whilesuchmethodshaveproducedsomehighlyrealisticresults,theyarenotgearedto-wardsmatchingaparticularappearancegivenbyanexample.Also,controllingtheresultsofthesynthesistypicallyinvolvesspecify-ingalargenumberofparameters,whicharenotalwaysintuitive.Incontrast,ourapproachisexample-based,ratherthanphysically-Ourapproachsynthesizestheboundariesofthelayershapesbyex-ample.Thus,itisrelatedtotheCurveAnalogiesworkofHertz-mannetal.[2002],whereasimilarframeworkwasappliedtoreproducethestyleofcurvedshapes.However,ourworkusesadifferentsimilaritymeasureandoperatesonadiscretepatch-basedrepresentationofashape'sboundary,ratherthanavector-basedrep-resentation.AlsorelatedistheworkofBahtetal.[2004],whichusesbinaryvoxelgridsinordertosynthesizegeometricdetailsonvolumesurfaces.Thesevoxelgridsaresimilartothebinaryneigh-borhoodsthatweusetooptimizetheshapeboundaries.However,theirgoalistoaddsmaller-scaledetailtoanexistingglobalshape,whilewefocusonsynthesizingtheentireshapefromscratch.3LayeredShapeSynthesisThisworkdealswithexample-basedgenerationofcontrolmapsrepresentedaslayermaps.Alayermapisanimagewherediffer-entpixelvaluesindicatetodifferentlayers.Let:::bethevaluesoflayermappixels,sortedinascendingorder.Then,alayerisdenedasthesetofallpixelswhosevalueisgreaterthanorequalto.Notethatapixelwithvalueactuallybelongstoalllayers;:::;.Onecanthinkofthelayersasstackedontopofeachother,withlayershigherinthestackpartially“conceal-ing”lowerlayers.Eachlayerhasanassociatedforegroundshape,whichweencodeasabinaryimageofthesamedimensionsasthelayermap.Notethattheshapeisalwayscontainedin.AsmaybeseeninFigures1and6,theboundariesofthesenestedshapesarehighlycorrelated,butnotaligned.Intheguresinthispaper,wedisplayvaluescorrespondingtodifferentlayersusinguniquecolors. abcde abcde a 0.01.33.05.36.0 0.02.75.05.4 0.05.05.3 0.04.6 Figure2:Similaritymeasurebetweenpairsofvedifferentshapes.Givenasetofsuchshapes,ourgoalistosynthesizeanewsetofshapes,whilemaintainingbothglobalandlocalsimilaritytotheoriginalones.Forthispurpose,itisimportantthateachshapeoc-cupiesthesamerelativeamountofpixelsinthesynthesizedmapasitdidintheexemplar,andthattheboundariesofthesynthe-sizedshapeslocallyresemblethoseintheexemplar.Wefoundthatrepresentingtheshapebymeansofitsboundarycurvefailstocap-turealloftherelevantinformation.Sucharepresentationcannotpredictthespatialrelationshipbetweendisconnectedcomponentsoftheshape,anddoesnotpreventself-intersections.Instead,werepresentabinaryshapebyacollectionofpatchescenteredontheshape'sboundarypixels(atmultipleresolutions)inordertocap-turesthenecessaryshapeproperties.Ourshapesynthesisapproachemploysoptimizationsimilarlyto[Wexleretal.2004;Kwatraetal.2005],wherethesynthesizedre-sultisiterativelyoptimizedwithrespecttosomemeasureofitssim-ilaritywiththeexemplar.Webeginbyderivingasuitablebidirec-tionalshapesimilaritymeasure,similarlytoSimakovetal.[2008]andWeietal.[2008].Next,wedescribeanovelgreedyoptimiza-tionschemethatiterativelymodiesaninitialshape,soastoin-creaseitssimilaritytoagivenexemplarshape.Finally,wedis-cusshowthismechanismisusedtocreateanentirenewlayermap,whichisasequenceofnestedshapes,fromthelayermapproducedinthelayerdecompositionphasedescribedintheprevioussection.3.1Shapesimilaritymeasurebethesetsofboundarypixelsofshapes,respectfully.Aboundarypixelisapixelinsideashapewithatleastoneofits4-neighborsoutsidetheshape.Letbetwoboundarypixels,andletbetheneighborhoodscenteredaroundthemandrotatedby.Werefertosuchneighborhoodsaspatches.Wedenethesimilarity,,betweentwoboundarypixelsasthedistancebetweentheirneighborhoods(rotatedsuchthatthedistanceisminimized).Formally,)=Sincewedealwithbinaryimages,thenormaboveissimplythenumberofdifferentpixelsbetweentwopatches.Next,wede-nethelocalsimilaritybetweenaboundarypixelandtheboundaryof(another)shapeasthesimilaritybetweenandthepixelmostsimilartoitontheboundary)=Notethatthissimilaritymeasureisnotsymmetric.Whileitensuresthateveryboundarypatchofissimilartoaboundarypatchin Figure3:Iterativeassignmentofboundarypatches.Theexemplarboundarypatches(left)areassignedtothesynthesizedboundarypatches(right).Incaseswheretwopatchesareassignedtothesameone,theassignmentwiththelargerLdifference(redarrow)isdiscardedandwillbeassignedtoanotherpatchinafutureiter-ation(yellowarrow).theremaybeboundarypatchesinthatarenotwellrepresentedin.Forexample,asimpleshapemaybedeemedsimilartoamorecomplexonethatalsohappenstocontainsomesimplefeatures.Thus,werequireabidirectionalsimilaritymeasure,denedas)=)+ B2j;(3)whichistheaveragenumberofdifferentpixelsbetweenabound-arypatchofoneshapetoitsnearestneighborontheother.Figure2showsseveraldifferentshapesandreportstheirpairwisebidirec-tionalsimilarities.3.2ShapeoptimizationArmedwiththesimilaritymeasureabove,weuseanoptimizationprocedurethatiterativelymodiestheboundaryofasynthesizedtomakeitmoresimilartothatoftheexemplarshapeTheoptimizationproceedsfromcoarsetoneresolution.Ateachresolutionwealternatebetweentwomainsteps:(i)matchingeachboundarypatchoftoaboundarypatchof,and(ii)modifyingbyaddingorremovingpixelsbasedontheresultsofthematchingachievedinthepreviousstep.Thisiterativeoptimizationproce-dureresemblesthatofKwatraetal.[2005],buteachofthetwomainstepsdifferssignicantlyfromitscounterpart,becauseweminimizeadifferent(bidirectional)energyfunction,andworkwithbinaryimages,ratherthantextures.Thesetwostepsarediscussedinmoredetailbelow.Boundarypatchmatching.Aspointedoutearlier,wewouldlikeeveryboundarypatchoftoresembleoneof,butwewouldalsolikeeveryboundarypatchoftoberepresentedin.Thus,assumingwehaveanequalnumberofboundarypatchesin,weseekaminimumcostassignment,afundamentalcombinato-rialoptimizationproblem[Schrijver2003].Solvingthisproblemexactlyistooexpensiveforourpurposes(,whereisthenumberofpatches),soweresorttoanapproximatesolutionusingtheiterativegreedyapproachdescribedbelow.denotethesetsofboundarypatchesofrespectively,andassumefornowthatthetwosetshavethesamesize.Eachpatchinisinitiallyassignedtoitsnearestneighbor.Asaresult,somepatchesinmayhavemorethanoneexemplarpatchassignedtothem,whileothersmayhavenone(seeFigure3).Intheformercase,wekeeponlytheassignmentwiththesmallestdifference,anddiscardtherest.Allofthepairsof Figure5:Reningshapeboundarieswithourmulti-resolutionoptimization.Twoinitialshapes(left)areevolvedusingtwodifferentexem-plars(right). Figure4:Left:Shapeadjustment.Boundaryexemplarpatchesaresuperimposedovertheirassignedpositions.Pixelsinregionsofoverlapbetweenthesesuperimposedpatchesmaybeaddedtotheshape(greendot),orremovedfromit(reddot),makingthenewboundarymoresimilartothatoftheexemplar.Right:Matchingpatchesfromthepreviousshape(gold)aresuperimposedagaintoseedthenewshape(blue).patcheswhichhavebeenassignedarethenremovedfromfurtherconsideration,andtheprocessisrepeateduntileverypatchinhasbeenassigned.Ingeneral,differinsize.Typically,thesynthesizedshapeislargerthantheexemplar.Thus,assumingthat,weconstructasetofexemplarpatchesofsizebyincludingeachexemplarpatchtimes,andrandomlyselect-additionalpatchesfrom.Inthiswayweensurethatalltheboundaryfeaturesintheexemplarshapegetanequalchancetoberepresentedinthesynthesizedshape.Shapeadjustment.Afterndingtheassignmentasdescribedabove,ourgoalistomodifytheboundaryofsoastoincreasethesimilarityto(byreducing).Toachievethis,wesu-perimposeeachexemplarpatchoveritscounterpartin.Considerapixel,whichiscoveredbyseveraloverlappingsuper-imposedpatchesfrom.Informally,ifthesepatchesagreethatshouldbepartoftheshape,itisaddedto.Similarly,apixelinsidemightberemovediftheoverlappingpatchesagreethatitshouldnotbelongtotheshape.ThisisillustratedinFigure4(left).Morespecically,considerapixelinthevicinityofthebound-aryof.Itiscoveredbytwogroupsofoverlappingsuperimposedexemplarpatches:onegrouppredictsthatbelongstotheshape,whiletheotheronepredictsthatisoutsidetheshape.Foreachofthesetwopredictionswecomputeascorebysumminguptheweightsofthecorrespondinggroup'spatchesat.Letbeaboundarypixeloftheexemplarpixelassignedtoit.Thentheentireexemplarpatchisassignedthefollowingweight whichisfurthermultipliedbyaGaussianfallofffunction(thustheweightdecreasesawayfromthecenterofthepatch).Thesigmavalueforthisfunctionwaschosentobehalfofthepatchessize.Thegroupwiththehighestscoreatdetermineswhetherbeincludedorexcludedfromtheshape.Whenthesumofweightsaccumulatedateachpixelisbelowathreshold,itsvalueremainsunchanged.Thisisbecausepatchweightsreectadegreeofcertainty,soareasoflowweightaremoresensitivetorandomnessgeneratedbyourapproximatednear-estneighborsearchandthegreedyassignment,suchthatusingthenewvaluesmayproducenoise.Thisthresholdalsodeterminesthenalamountofpixelsintheshapeaftertheiterationisdone.There-fore,itissetdynamicallysothatthe(relative)amountofthepixelsinsidetheshapeisthesameasintheexemplar.Candidatesfromtheintervalal10�2;10�7]aretestedandtheonewhichresultsinthenearestamountischosen.Aftertheupdateiscomplete,theopti-mizationprocedureisrepeateduntilconvergence.Convergenceisreachedwhenthenumberofchangedpixelsfallsbelowathreshold.Asmentionedearlier,theoptimizationproceedsfromcoarsetoneresolution.Theresultcomputedateachresolutionlevelisupsam-pledtoserveasastartingpointforthenext(ner)level.Atcoarserresolutionstheglobalstructuresareformed,whileneresolutionsllinthenedetailsalongtheshape'sboundary.Inourexam-ples,weuse5to6resolutionlevels.Figure5showshowdifferentinitializationsleadtodifferentglobalshapes.However,inalloftheexamplesthesynthesizedshapecontainsboundaryfeaturesthat Figure6:Ourinhomogeneoustexturesynthesisapproach.areverysimilartothosepresentintheexemplar,resultingincloseoverallresemblance.3.3LayermapsynthesisTheshapeoptimizationprocedurepresentedintheprevioussec-tionmaybeuseddirectlytosynthesizetherst(bottom)layer.Arandomlygeneratedshape,withthenumberofforegroundpixelsmatchingthatofthecorrespondingexemplarlayermaybeusedforinitialization.Inordertogeneratethefollowinglayers,however,wemustintroduceanumberofmodications.First,theshapeofeachlayerisnestedinsidetheshapeofthelayerbeneathit.Sec-ond,theboundariesoftwosuccessivelayersaretypicallyhighlycorrelated.Preservingthiscorrelationisimportant,asitisinstru-mentalforfaithfullyreproducingtheappearanceoftheexemplarinthesynthesizedresult.Itisnotobvioushowtoinitializetheshapeofthenextlayersundertheseconditions.Toaddresstheserequirementswebeginthesynthesisofeachlayerbycreatingamaskthatdenesthearea(containedinsidetheshapeofthepreviouslayer),wherethecurrentshapeisallowedtoevolve.Initially,thismaskissettotheentireshape,butweusethemostrecentboundarypatchassignments(fromthelastshapeoptimizationiteration)toshrinkthismaskdowntoabetterinitialguessfortheregioncontaining.Morespecically,weagainsuperimposeboundarypatchesfromtheexemplarovertheirassignedlocationsontheboundaryof,butthistimewetrytopre-dictwhichofthepixelsinsideshouldbelongtothemaskofasillustratedinFigure4(right).Thus,theinteriorofisseededwithpixelswhicharepredictedtobelongtowithasufcientlylargeweight.Themaskisthenshrunktoincludeonlytheseseededpixels.Seededpixelswithhighweightsformtheinitialguessfor,whilethosewithsomewhatsmallerweightsdenetheremain-ingregionofthemask,withinwhichtheshapeisallowedtoevolveinthecourseoftheoptimization.Theinitializationofeachnewlayerisdoneviathisseedingmech-anisminthecoarsestresolution,whereboundarypatchesarelargeenoughtofullycovertheinteriorofthepreviousshape.Asimilarstepisrepeatedatthebeginningofeachresolutionleveltorecreateanaccuratemaskforthecurrentlevelatthenewresolution,andtorenetheshapeboundary.Afterthisstep,shapeoptimizationproceedsasdescribedbefore.Continuouscontrolmaps.Inourexperimentswefoundthatthesubsequenttexturesynthesisprocesscansometimesbeimprovedbyswitchingfromadiscretelayermaptoacontinuouscontrolmap.Specically,foreachpixelinsidetheshapeitscontinuousmapvalueissetto )+arethedistancesfromrespectively(seeFigure7).Thedistancesareobtainedbyperform-ingdistancetransformsover.DistancetransformswerealsousedtocreatecontrolmapsbyLefebvreandHoppe[2006]. Figure7:Thedistancesfromapointinsideashapetotheneigh-boringshapesareusedtoconvertadiscretelabelmap(left)toacontinuousone(right).4ApplicationsandResultsWefoundthelayeredshapesynthesisapproachdescribedintheprevioustexturetobeeffectiveforsynthesisofinhomogeneoustex-tures,suchasthoseresultingfromnaturalagingorweatheringpro-cesses.ThesynthesisprocessforsuchtexturesconsistsofthreesuccessivephasesdepictedinFigure6:layerdecomposition,shapesynthesis,andtexturesynthesis.Thelayerdecompositionphasetakesaninhomogeneoustexturesampleasinput,andgeneratesalayermapwhichencodesthedis-tincthomogeneoustextureregions(layers)presentintheinput,byassigningauniquelabeltoallofthepixelsbelongingtothesamelayer.FollowingthetextureclassicationapproachadvocatedbyVarmaandZisserman[2003],werstsegmenttheexemplar'spix-elsbyperformingK-Meansclusteringonthedimensionalfea-turevectorsformedbyconcatenatingthevaluesofeachpixel'sneighborhood.Wecurrentlyrelyontheusertospecifyasthenumberofdistincttexturesvisibleintheexemplar,typicallybetween3and5.issetto15inallofourexamples.There-sultingclustersshouldroughlycorrespondtothedistincttexturespresentintheexemplar.Pointsclosertotheclustercentersareduetopixelsthatarethemoretypicalrepresentativesofthecorrespond-ingtextures,whilepointsfarawayfromthecentercomefromareasoftransitionbetweentextures.Let;:::;betheresultingclusters,orderedbytheusersuchthatisthebottomlayer(orig-inal“clean”surface),andisthetoplayer(most“weathered”surface).Forthelayermap,wesettheforegroundpixelsofofCi+1[:::.Iftheclustershavebeenorderedproperly,the;:::;expressesapossiblenaturalevolutionandspreadovertimeoftheweatheringphenomenoncapturedbytheexemplar.Inthelastphaseweusethenewlayermapobtainedintheshapesynthesisphase(Section3)tosynthesizeanewinhomogeneoustexture.Thisisdonebyapplyingthe“texture-by-numbers”frame-work[Hertzmannetal.2001].Whileapplyingtexture-by-numbersdirectlyonthelayermapoftenproducessatisfactoryresults,theirvisualqualitymaybefurtherimprovedbyswitchingfromadiscretelayermaptoacontinuousone,asdescribedintheprevioussection.Thecontinuousmapmaybeseenasaheuristicanalogueforthe Figure8:Avarietyofresultsproducedbyourmethod.Left:inputexemplarsandtheirdecompositionstolayers;Middle:synthesizedlayermap;Right:nalsynthesizedresult. Figure9:Terraingenerationbyheightmapsynthesisusingourmethod.Left:inputheightmapanditsdecompositiontolayers;Middle:thesynthesizedlayermap;Right:nalsynthesizedresult. inputourresult[Kopfetal.2007][Kwatraetal.2005]Figure10:Acomparisonofourapproachtotextureoptimizationwithandwithouthistogrammatching.Inthetoprowbothmethodsperformsynthesisdirectlyfromtheexemplar.Inthebottomtworows,weattempttousetextureoptimizationtosynthesizeanewlayermap.weatheringdegreemapof[Wangetal.2006].Weexperimentedwithourmethodonavarietyofnaturalinhomo-geneoustextures.SomeresultsareshowninFigures1,6,and8.Theexamplesarequitevaried,showcasingphenomenasuchascor-rosion,rust,lichen,andpeelingpaint.Theydiffersignicantlynotonlyintheirappearance,butalsointheirunderlyinglayerstructure,asmaybeseenfromthelayermapsextractedbyourmethod.Ourmethodsuccessfullyreproducesthegloballayerstructures,thelo-calnedetailsoftheshapeboundaries,andthenalappearanceofthesetextures.Ourmethodisnotlimitedtosuchtextures,however.Otherinhomogeneoustexturesthatexhibitsalayeredstructurewithnestedshapesmaybesynthesizedaswell.Forexample,wehavesynthesizedaplausiblectionalsatelliteimagefromoneofEarth(bottomrowinFigure8).Sinceweusepatch-basedshapesynthe-sis,somerepetitionsdooccur,buttheyaremostlydifculttospot,astheyareexplicitlylimitedinourapproachbyourfairboundarysamplingandassignmentmechanisms.Anotherapplicationofourapproachisexample-basedterrainsyn-thesis,asdemonstratedinFigure9.Heightmapsusedtorepre-sentterrainsmayalsobeconsideredasnon-stationarytexturesfor Figure11:Inpainting.Left:original,right:ourresult. Figure12:Exampleofusercontrolviaapaintinginterface.whichourlayeredshapesynthesisapproachtsperfectly.Forthegenerationofthelayermap,asimplequantizationoftheheightmapisused.Theboundariesoflayersresemblecontourlinesinatopographicmap.Inthisapplication,whichissimilartotexturesynthesisdescribedbefore,theshapesynthesisphasegeneratesanewtopographiclayoutforthesynthesizedterrainandthetexturesynthesisphaseaddsthenedetails.Thecomputationtimeofourmethodisdominatedbythe“texturebynumbers”phase,whichtakesuptovehoursfora800imageusing55neighborhoods.Thetimeittakestosynthesizeanewlayermapdependsonthetotallengthoftheshapeboundaries.Thecomplexityoftheoptimizationstepislinearinthenumberofnearestneighborcallsforeachboundarypatch.Weuseapproxi-matenearestneighborsearchvialocallysensitivehashing[DatarandIndyk2004].Ittakesourunoptimizedcode20–30secondsonaveragetocompleteoneoptimizationiterationforan800600im-age.Typically,5–10iterationsareusedateachresolutionlevel,sotheexecutiontimeperlayerisupto30minutes.Wecompareourmethodtotwopreviousexample-basedtexturesynthesismethods:textureoptimization[Kwatraetal.2005]andtextureoptimizationwithcolorhistogrammatching[Kopfetal.2007].Figure10showstheresultsofthethreemethodsside-by-side.Thetoprowshowsthenalsynthesisresultonatextureofarustingsurface.Kwatra'smethodissuitedforstationarytextures,andexhibitsmultipleobviousrepetitionsmakingtheresultquitedissimilarfromtheexemplar.Kopf'sresultmatchestheglobalcolorstatisticsoftheexemplar,andproducesabetterresult,butsomerepetitionsarestillapparent,andsomeregionsofthesynthe-sizedtexturedonothaveasimilarcounterpartintheexemplar(suchasthelargeregionoflighterrustnearthecenter).Itisalsointer-estingtoexaminewhetherthesepreviousmethodsareabletosyn-thesizethelayermap,ratherthansynthesizingthetexturedirectly,andthisisdoneontwoexamplesshowninthetwobottomrowsofFigure10.Themiddlerowisalayermapextractedfromacloudyskytexture,whileinthebottomrowthelayermapisfromtheter-raininFigure9.Inbothoftheseexamples,thepreviousmethodsgeneratemorerepetitionsofentireregionsofthelayermap,andinseveralplacestherearedirecttransitionsbetweennon-adjacentlayers,whicharenotpresentintheinputmap.Figure11showsaninpaintingexample,whereaholeislledinsideaninhomogeneoustexture.Whiletheresultisobviouslynotiden-ticaltotheoriginalimage,itisquiteplausible,andthelledregionblendswellwiththeoriginalparts.Thelayermapinsidetheholeisinitializedrandomly.Sinceourmethodmodiestheentirelayermap,aftereachoptimizationstepweresetthelayermapoutsidethe Figure13:Examplesoffailurecases.Top:violationofthelayermodel.Bottom:failuretoreproducespecicshapes.holebacktotheoriginalone.Figure12demonstratesthefeasibilityofcontrollingtheresultofthesynthesisviaapaintinginterface.TheexemplaranditslayermapareshowninthethirdrowofFigure8.Theusersketchesinyellowtheapproximatepositionwhererustshouldbe,andtheresultingsketchservesastheinitializationfortheshapesynthesisphase.Thinstripsofblueandgreenpixelsareautomaticallyaddedbythesystem,sincetheyellowlayershapeissupposedtobenestedinsidetheblueandthegreenlayershapes.Inordertoavoidchang-ingtheusersketchedshapetoomuch,fewerresolutionlevelsareusedbytheshapesynthesismethod,resultinginthemiddleimageinFigure12,whiletherightmostimageisthesynthesizedtexture.Ourapproachmakestwobasicassumptions:(1)thecontrolmapconsistsofanorderedsetlayers,nestedwithineachother;(2)theproposedshapesimilaritymeasurecapturesalltheshapecharacteristicsthatoneaimstoreproduce.Violationofeitheroftheseassumptionsmayleadtoafailure,asdiscussedbelow.(1)Thersttypeoffailureisdemonstratedbythesyntheticexam-pleinthetoprowofFigure13.Herewegreenandblueregionsthatareindependentofeachotherintheinput(e.g.,anaturaltexturewhoseappearanceresultsfromtwoindependentprocesses).Ourapproachassumesalayeredmodelandgeneratesthegreenlayerrst,followedbythebluelayer.Asaresult,therelationsbetweenthegreenandblueregionsarenotpreserved,andseveralbluere-gionsaresynthesizedinsidegreenones.(2)Theproposedsimilaritymeasurecharacterizesshapesbythelo-calappearanceoftheirboundaries,withoutconsideringtheshapeasawhole.Thus,itisbettersuitedforne-scaleunstructuredshapesandfractal-likeboundaries.Morestructuredelementsmightbebetterhandledbyothermodels.Forexample,thelocationsofthedesertsinthebottomrowofFigure8mighthavebeenrepro-ducedbetterusingthecontext-awaretexturesframework[Luetal.2007].ThebottomrowofFigure13showsanothersyntheticexam-plewheretheeasilyrecognizabledistinctshapesintheinputmapappearmixedinthemapgeneratedbyourmethod.5ConclusionWehavepresentedanovelexample-basedmethodforsynthesisofcontrolmapssuitablefornonstationarytextures,suchasthosere-sultingfromweathering.Tothatend,wehavedevelopedanew powerfulexample-basedshapesynthesisalgorithmthatrepresentsshapesasacollectionofboundarypatchesatmultipleresolution,andsynthesizesanewshapefromanexamplebyoptimizingabidi-rectionalsimilarityfunction.Applicationsofourmethodincludesynthesisofnaturaltexturesandterraingeneration.Infutureworkwehopetoextendthemethodofshapesynthesistoalargersetoftextures,forexample,texturesthatdonotexhibitaclearhierarchyoflayers,andtextureswithlargerstructures.Ourcurrentmeasureemphasizesboundarysimilarityoverotherprop-erties,suchasareatoboundarylengthratio,whichismaintainedonlyimplicitly.Wewouldliketogainabetterunderstandingoftherelationsbetweensuchproperties,andexperimentwithvariousextensionsofoursimilaritymeasure.Wewouldalsoliketodiscoveradditionalapplicationsofourshapesynthesisapproach.Inparticular,itwouldbeinterestingtoexploretheapplicabilityofsuchanapproachtothesynthesisof3Dshapes.Acknowledgments:ThisworkwassupportedinpartbygrantsfromtheIsraelMinistryofScience,andfromtheIsraelScienceFoundationfoundedbytheIsraelAcademyofSciencesandHu-manities.Theauthorswouldalsoliketothanktheanonymousre-viewerswhosesuggestionsweregreatlyhelpful.ReferencesSHIKHMIN,M.2001.Synthesizingnaturaltextures.InProc.Symp.Interactive3DGraphics,217–226.HAT,P.,INGRAM,S.,ANDURK,G.2004.Geometrictexturesynthesisbyexample.InSGP'04:Proceedingsofthe2004Eu-rographics/ACMSIGGRAPHsymposiumonGeometryprocess-,ACM,NewYork,NY,USA,41–44.OSCH,C.,PUEYO,X.,MERILLOU,S.,ANDHAZANFARPOUR,D.2004.Aphysically-basedmodelforrenderingrealis-ticscratches.ComputerGraphicsForum23,3(Sept.),361–370.ATAR,M.,ANDNDYK,P.2004.Locality-sensitivehashingschemebasedonp-stabledistributions.InProc.SCG'04,ACMPress,253–262.ESBENOIT,B.,GALIN,E.,ANDKKOUCHE,S.2004.Simu-latingandmodelinglichengrowth.ComputerGraphicsForum,3(Sept.),341–350.ORSEY,J.,ANDANRAHAN,P.1996.Modelingandrenderingofmetallicpatinas.InProc.SIGGRAPH'96,AddisonWesley,ORSEY,J.,EDELMAN,A.,JENSEN,H.W.,LEGAKIS,J.,ANDEDERSEN,H.K.1999.Modelingandrenderingofweatheredstone.InProc.SIGGRAPH'99,ACMPress,225–234.ORSEY,J.,RUSHMEIER,H.,ANDILLION,F.2008.ModelingofMaterialAppearance.ComputerGraphics.MorganKaufmann/Elsevier,Dec..336pages.FROS,A.A.,ANDREEMAN,W.T.2001.Imagequiltingfortexturesynthesisandtransfer.Proc.SIGGRAPH2001,341–346.FROS,A.A.,ANDEUNG,T.K.1999.Texturesynthesisbynon-parametricsampling.Proc.ICCV'992,1033–1038.,J.,T,C.-I.,RAMAMOORTHI,R.,BELHUMEUR,P.,MTUSIK,W.,ANDAYAR,S.2006.Time-varyingsurfaceap-pearance:acquisition,modelingandrendering.ACMTransac-tionsonGraphics25,3(Proc.SIGGRAPH2006),762–771.EEGER,D.J.,ANDERGEN,J.R.1995.Pyramid-basedtextureProc.SIGGRAPH'95,229–238.ERTZMANN,A.,JACOBS,C.E.,OLIVER,N.,CURLESS,B.,ANDALESIN,D.H.2001.Imageanalogies.Proc.SIGGRAPH,327–340.ERTZMANN,A.,OLIVER,N.,CURLESS,B.,ANDEITZ,S.M.2002.Curveanalogies.InProc.13thEurographicsWorkshoponRendering,EurographicsAssociation,233–246.OPF,J.,F,C.-W.,COHEN-O,D.,DEUSSEN,O.,LISCHINSKI,D.,ANDONG,T.-T.2007.Solidtexturesynthesisfrom2dexemplars.ACMTransactionsonGraphics26,3(Proc.SIG-GRAPH2007),2.WATRA,V.,SCHODL,A.,ESSA,I.,TURK,G.,ANDOBICKA.2003.Graphcuttextures:imageandvideosynthesisusinggraphcuts.ACMTransactionsonGraphics22,3(Proc.SIG-GRAPH2003),277–286.WATRA,V.,ESSA,I.,BOBICK,A.,ANDWATRA,N.2005.Textureoptimizationforexample-basedsynthesis.ACMTrans-actionsonGraphics24,3(Proc.SIGGRAPH2005),795–802.EFEBVRE,S.,ANDOPPE,H.2006.Appearance-spacetex-turesynthesis.ACMTransactionsonGraphics25,3(Proc.SIG-GRAPH2006),541–548.,J.,GEORGHIADES,A.S.,GLASER,A.,W,H.,WEIL.-Y.,GUO,B.,DORSEY,J.,ANDUSHMEIER,H.2007.Context-awaretextures.ACMTrans.Graph.26,1,3.ERILLOU,S.,DISCHLER,J.-M.,ANDHAZANFARPOUR,D.2001.Corrosion:simulatingandrendering.InProc.GraphicsInterface2001,CanadianInformationProcessingSociety,167–CHRIJVER,A.2003.CombinatorialOptimization:PolyhedraandEfciency,vol.A.Springer-Verlag,BerlinHeidelberg.IMAKOV,D.,CASPI,Y.,SHECHTMAN,E.,ANDRANI,M.2008.Summarizingvisualdatausingbidirectionalsimilarity.Proc.CVPR2008,IEEEComputerSociety.ARMA,M.,ANDISSERMAN,A.2003.Textureclassication:Arelterbanksnecessary.InProc.CVPR2003,IEEE,691–698.ANG,J.,TONG,X.,LIN,S.,PAN,M.,WANG,C.,BAO,H.,UO,B.,ANDHUM,H.-Y.2006.Appearancemanifoldsformodelingtime-variantappearanceofmaterials.ACMTransac-tionsonGraphics25,3(Proc.SIGGRAPH2006),754–761.EI,L.-Y.,ANDEVOY,M.2000.Fasttexturesynthesisus-ingtree-structuredvectorquantization.Proc.SIGGRAPH2000EI,L.-Y.,HAN,J.,ZHOU,K.,BAO,H.,GUO,B.,ANDHUMH.-Y.2008.Inversetexturesynthesis.ACMTrans.Graph.273,1–9.EI,L.-Y.,LEFEBVRE,S.,KWATRA,V.,ANDURK,G.,2009.Stateoftheartinexample-basedtexturesynthesis.Eurographics2009StateofTheArtReport,April.EXLER,Y.,SHECHTMAN,E.,ANDRANI,M.2004.Space-timevideocompletion.InProc.CVPR2004,vol.1,120–127.HANG,J.,ZHOU,K.,VELHO,L.,GUO,B.,ANDHUM,H.-Y.2003.Synthesisofprogressively-varianttexturesonarbi-trarysurfaces.ACMTransactionsonGraphics22,3(Proc.SIG-GRAPH2003),295–302.