Manufactured in The Netherlands Contour and Texture Analysis for Image Segmentation JITENDRA MALIK SERGE BELONGIE THOMAS LEUNG AND JIANBO SHI Computer Science Division University of California at Berkeley Berkeley CA 947201776 USA Received December ID: 30120
Download Pdf The PPT/PDF document "International Journal of Computer Vision..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
InternationalJournalofComputerVision43(1),7Ð27,20012001KluwerAcademicPublishers.ManufacturedinTheNetherlands.ContourandTextureAnalysisforImageSegmentationJITENDRAMALIK,SERGEBELONGIE,THOMASLEUNGANDJIANBOSHIComputerScienceDivision,UniversityofCaliforniaatBerkeley,Berkeley,CA94720-1776,USA Maliketal. Figure1.Somechallengingimagesforasegmentationalgorithm.Ourgoalistodevelopasinglegroupingprocedurewhichcandealwithallthesetypesofimages.informationfromthewholeimageistakenintoaccountatthesametime.Incontour-basedapproaches,oftentherststepofedgedetectionisdonelocally.Subsequentlyeffortsaremadetoimproveresultsbyagloballinkingprocessthatseekstoexploitcurvilinearcontinuity.Examplesin-cludedynamicprogramming(Montanari,1971),relax-ationapproaches(ParentandZucker,1989),saliencynetworks(ShaashuaandUllman,1988),stochasticcompletion(WilliamsandJacobs,1995).Acriticismofthisapproachisthattheedge/noedgedecisionismadeprematurely.Todetectextendedcontoursofverylowcontrast,averylowthresholdhastobesetfortheedgedetector.Thiswillcauserandomedgesegmentstobefoundeverywhereintheimage,makingthetaskofthecurvilinearlinkingprocessunnecessarilyharderthaniftherawcontrastinformationwasused.Athirddimensiononwhichvarioussegmentationschemescanbecomparedistheclassofimagesforwhichtheyareapplicable.AssuggestedbyFig.1,wehavetodealwithimageswhichhavebothtexturedanduntexturedregions.Hereboundariesmustbefoundus-contourandtextureanalysis.Howeverwhatndintheliteratureareapproacheswhichconcen-trateononeortheother.Contouranalysis(e.g.edgedetection)maybeade-quateforuntexturedimages,butinatexturedregionitresultsinameaninglesstangledwebofcontours.Thinkforinstanceofwhatanedgedetectorwouldre-turnonthesnowandrockregioninFig.1(a).Theforthisprobleminedgedetec-tionistouseahighthresholdsoastominimizethenumberofedgesfoundinthetexturearea.Thisisob-viouslyanon-solutionsuchanapproachmeansthatlow-contrastextendedcontourswillbemissedaswell.ThisproblemisillustratedinFig.2.Thereisnorecog-nitionofthefactthatextendedcontours,evenweakincontrast,areperceptuallysigniWhiletheperilsofusingedgedetectionintexturedregionshavebeennotedbefore(seee.g.Binford,1981),acomplementaryproblemofcontoursconstitutingaproblemfortextureanalysisdoesnotseemtohavebeenrecognizedbefore.Typicalapproachesarebasedonmeasuringtexturedescriptorsoverlocalwindows,andthencomputingdifferencesbetweenwindowdescrip-torscenteredatdifferentlocations.Boundariescanthengiverisetothinstrip-likeregions,asinFig.3.Forspeci-city,assumethatthetexturedescriptorisahistogramoflinearlteroutputscomputedoverawindow.Anyhistogramwindowneartheboundaryofthetworegionswillcontainlargelterresponsesfromltersorientedalongthedirectionoftheedge.However,onbothsidesoftheboundary,thehistogramwillindicateafeature-lessregion.Asegmentationalgorithmbasedon,say, ContourandTextureAnalysis9 Figure2.Demonstrationoftextureasaproblemforthecontourprocess.EachimageshowstheedgesfoundwithaCannyedgedetectorforthepenguinimageusingdifferentscalesandthresholds:(a)nescale,lowthreshold,(b)nescale,highthreshold,(c)coarsescale,lowthreshold,(d)coarsescale,highthreshold.Aparametersettingthatpreservesthecorrectedgeswhilesuppressingspuriousdetectionsinthetexturedareaisnotpossible. Figure3.Demonstrationofthecontour-as-a-textureproblemusingarealimage.(a)Originalimageofabaldeagle.(b)ThegroupsfoundbyanEM-basedalgorithm(Belongieetal.,1998).distancesbetweenhistograms,willinevitablypartitiontheboundaryasagroupofitsown.Asisevident,theproblemisnotconnedtotheuseofahistogramofteroutputsastexturedescriptor.Figure3(b)showstheactualgroupsfoundbyanEM-basedalgorithmusinganalternativecolor/texturedescriptor(Belongieetal.,1.1.DesiderataofaTheoryofImageSegmentationAtthisstage,wearereadytosummarizeourdesiredattributesforatheoryofimagesegmentation.1.Itshoulddealwithgeneralimages.Regionswithorwithouttextureshouldbeprocessedinthesameframework,sothatthecuesofcontourandtexturedifferencescanbesimultaneouslyexploited.2.Intermsofcontour,theapproachshouldbeabletodealwithboundariesdenedbybrightnessstepedgesaswellaslines(asinacartoonsketch).3.ImageregionscouldcontaintexturewhichcouldberegularsuchasthepolkadotsinFig.1(c),stochasticasinthesnowandrockregionin(a)oranywhereinbetweensuchasthetigerstripesin(b).Akeyquestionhereisthatoneneedsanautomaticpro-cedureforscaleselection.Whateveroneschoiceoftexturedescriptor,ithastobecomputedoveralocalwindowwhosesizeandshapeneedtobede-terminedadaptively.Whatmakesscaleselectionachallengeisthatthetechniquemustdealwiththe Maliketal.widerangeoftexturesregular,stochastic,orin-termediatecasesinaseamlessway.1.2.IntroducingTextonsJuleszintroducedthetermtexton,analogoustoaphonemeinspeechrecognition,nearly20yearsago(Julesz,1981)astheputativeunitsofpreattentivehu-mantextureperception.Hedescribedthemqualita-tivelyforsimplebinarylinesegmentstimulisegments,crossingsandterminatorsbutdidnotpro-videanoperationaldenitionforgray-levelimages.Subsequently,textontheoryfellintodisfavorasamodelofhumantexturediscriminationasaccountsbasedonspatiallteringwithorientationandscale-selectivemechanismsthatcouldbeappliedtoarbitrarygray-levelimagesbecamepopular.Thereisafundamental,wellrecognized,problemwithlinearlters.Generically,theyrespondtoanystimulus.Justbecauseyouhavearesponsetoanori-entedodd-symmetriclterdoesntmeanthereisanedgeatthatlocation.Itcouldbethatthereisahighercontrastbaratsomeotherlocationinadifferentorien-tationwhichhascausedthisresponse.Tokenssuchasedgesorbarsorcornerscannotbeassociatedwiththeoutputofasinglelter.Ratheritisthesignatureoftheoutputsoverscales,orientationsandorderofthethatismorerevealing.Hereweintroduceafurtherstepbyfocussingontheoftheseltersconsideredaspointsinahighdimensionalspace(ontheorderof40ltersareused).Weperformvectorquantization,orclustering,inthishigh-dimensionalspacetondprototypes.Callthesetextonswewillndempiricallythatthesetendtocorrespondtoorientedbars,terminatorsandsoon.Onecanconstructauniversaltextonvocabularybyprocessingalargenumberofnaturalimages,orwecouldndthemadaptivelyinwindowsofimages.Ineachcasethe-meanstechniquecanbeused.Bymappingeachpixeltothetextonnearesttoitsvectoroflterresponses,theimagecanbeanalyzedintotextonchannels,eachofwhichisapointset.Itisouropinionthattheanalysisofanimageintotex-tonswillproveusefulforawidevarietyofvisualpro-cessingtasks.Forinstance,inLeungandMalik(1999)weusetherelatednotionof3Dtextonsforrecognitionoftexturedmaterials.Inthepresentpaper,ourobjec-tiveistodevelopanalgorithmforthesegmentationofanimageintoregionsofcoherentbrightnessandtexturewewillndthatthetextonrepresentationwillenableustoaddressthekeyproblemsinaverynaturalfashion.1.3.SummaryofOurApproachWepursueimagesegmentationintheframeworkofNormalizedCutsintroducedbyShiandMalik(1997,2000).Theimageisconsideredtobeaweightedgraphwherethenodesarepixelsandedgeweights,,denotealocalmeasureofsimilaritybetweenthetwopixels.Groupingisperformedbyndingeigenvec-torsoftheNormalizedLaplacianofthisgraph(3).Thefundamentalissuethenisthatofspecifyingtheedge;werelyonnormalizedcutstogofromtheselocalmeasurestoagloballyoptimalpartitionoftheimage.Thealgorithmanalyzestheimageusingthetwocuesofcontourandtexture.Thelocalsimilaritymeasurebetweenpixelsduetothecontourcue,iscomputedintheinterveningcontourframeworkofLeungandMalik(1998)usingpeaksincontourori-entationenergy(2and4.1).Textureisanalysedus-ingtextons(2.1).Appropriatelocalscaleisestimatedfromthetextonlabels.Ahistogramoftextondensi-tiesisusedasthetexturedescriptor.Similarity,ismeasuredusingthetestonthehistograms(Theedgeweightscombiningbothcontourandtex-tureinformationarespeciedbygatingeachofthetwocueswithatexturednessmeasure(In(5),wepresentthepracticaldetailsofgoingfromtheeigenvectorsofthenormalizedLaplacianmatrixofthegraphtoapartitionoftheimage.Resultsfromthealgorithmarepresentedin(6).SomeoftheresultspresentedherewerepublishedinMaliketal.(1999).2.Filters,CompositeEdgels,andTextonsSincethe1980s,manyapproacheshavebeenproposedinthecomputervisionliteraturethatstartbyconvolv-ingtheimagewithabankoflinearspatialtunedtovariousorientationandspatialfrequencies(KnutssonandGranlund,1983;KoenderinkandvanDoorn,1987;FogelandSagi,1989;MalikandPerona,1990).(SeeFig.4foranexampleofsuchaTheseapproacheswereinspiredbymodelsofpro-cessingintheearlystagesoftheprimatevisualsystem(e.g.DeValoisandDeValois,1988).Thelterkernelsaremodelsofreceptiveeldsofsimplecellsinvisual ContourandTextureAnalysis11 Figure4.Left:Filtersetconsistingof2phases(evenandodd),3scales(spacedbyhalf-octaves),and6orientations(equallyspacedfrom0to).Thebasiclterisadifference-of-Gaussianquadraturepairwith3:1elongation.Right:4scalesofcenter-surroundlters.Eachlteris-normalizedforscaleinvariance.cortex.Toarstapproximation,wecanclassifythemintothreecategories:1.Cellswithradiallysymmetricreceptiveelds.TheusualchoiceofisaDifferenceofGaussians(DOG)withthetwoGaussianshavingdifferentval-uesof.Alternatively,thesereceptiveeldscanalsobemodeledastheLaplacianofGaussian.2.Orientedodd-symmetriccellswhosereceptiveeldscanbemodeledasrotatedcopiesofahor-izontaloddsymmetricreceptiveeld.AsuitablepointspreadfunctionforsuchareceptiveeldisaGaussianwithstandarddeviation.Theratioisameasureoftheelongationoftheter.3.Orientedeven-symmetriccellswhosereceptiveeldscanbemodeledasrotatedcopiesofahorizon-talevensymmetricreceptiveeld.AsuitablepointspreadfunctionforsuchareceptiveeldisTheuseofGaussianderivatives(orequivalently,dif-ferencesofoffsetGaussians)formodelingreceptiveeldsofsimplecellsisduetoYoung(1985).OnecouldequivalentlyuseGaborfunctions.OurpreferenceforGaussianderivativesisbasedontheircomputationalsimplicityandtheirnaturalinterpretationasderivatives(KoenderinkandvanDoorn,1987,1988).Theorientedlterbankusedinthiswork,depictedinFig.4,isbasedonrotatedcopiesofaGaussianderivativeanditsHilberttransform.Moreprecisely,equaltheHilberttransformofalongthe 1 exp exp isthescale,istheaspectratiooftheter,andisanormalizationconstant.(TheuseoftheHilberttransforminsteadofarstderivativemakesanexactquadraturepair.)Theradiallysymmet-ricportionofthelterbankconsistsofDifference-of-Gaussiankernels.Eachlteriszero-meanandnor-malizedforscaleinvariance(MalikandPerona,1990).Nowsupposethattheimageisconvolvedwithsuchabankoflinearlters.Wewillrefertothecollectionofresponseimagesasthehypercolumntransformoftheimage.Whyisthisusefulfromacomputationalpointofview?Thevectoroflteroutputschar-acterizestheimagepatchcenteredatbyasetofvaluesata.ThisissimilartocharacterizingananalyticfunctionbyitsderivativesatapointcanuseaTaylorseriesapproximationtondtheval-uesofthefunctionatneighboringpoints.AspointedoutbyKoenderinkandvanDoorn(1987),thisismorethanananalogy,becauseofthecommutativityoftheoperationsofdifferentiationandconvolution,there-ceptiveeldsdescribedaboveareinfactcomputingblurredderivatives.WerecommendKoenderinkandvanDoorn(1987,1988),JonesandMalik(1992),andMalikandPerona(1992)foradiscussionofotherad-vantagesofsucharepresentation.Thehypercolumntransformprovidesaconvenientfrontendforcontourandtextureanalysis: Maliketal..Incomputationalvision,itiscustomarytomodelbrightnessedgesasstepedgesandtode-tectthembymarkinglocationscorrespondingtothemaximaoftheoutputsofodd-symmetric(e.g.Canny,1986)atappropriatescales.However,itshouldbenotedthatstepedgesareaninadequatemodelforthediscontinuitiesintheimagethatre-sultfromtheprojectionofdepthororientationdis-continuitiesinphysicalscene.Mutualilluminationandspecularitiesarequitecommonandtheiref-fectsareparticularlysignicantintheneighbor-hoodofconvexorconcaveobjectedges.Inaddi-tion,therewilltypicallybeashadinggradientontheimageregionsborderingtheedge.Asaconse-quenceoftheseeffects,realimageedgesarenotstepfunctionsbutmoretypicallyacombinationofsteps,peakandroofproles.AswaspointedoutinPeronaandMalik(1990),theorientedenergyapproach(KnutssonandGranlund,1983;MorroneandOwens,1987;MorroneandBurr,1988)canbeusedtodetectandlocalizecorrectlythesecompos-iteedges.Theorientedenergy,alsoknownasthetureenergy,atangle0isdenedas:hasmaximumresponseforhorizontalcon-tours.RotatedcopiesofthetwolterkernelsareabletopickupcompositeedgecontrastatvariousGiven,wecanproceedtolocalizethecom-positeedgeelements(edgels)usingorientednon-maximalsuppression.Thisisdoneforeachscaleinthefollowingway.Atagenericpixel,letargmaxdenotethedominantorientationthecorrespondingenergy.Nowlookatthetwoneighboringvaluesofoneithersidealongthelinethroughperpendiculartothedominantorientation.Thevalueiskeptatthelocationofonlyifitisgreaterthanorequaltoeachoftheneighboringvalues.Otherwiseitisre-placedwithavalueofzero.Notingthatrangesbetween0andinnity,weconvertittoaprobability-likenumberbetween0and1asfollows:conexpisrelatedtoorientedenergyresponsepurelyduetoimagenoise.Weuse02inthispaper.TheideaisthatforanycontourwithconTexture.Asthehypercolumntransformprovidesagoodlocaldescriptorofimagepatches,thebound-arybetweendifferentlytexturedregionsmaybefoundbydetectingcurvesacrosswhichthereisacantgradientinoneormoreofthecompo-nentsofthehypercolumntransform.Foranelab-orationofthisapproach,seeMalikandPeronaMalikandPeronareliedonaveragingwithlargekernelstosmoothawayspatialvariationforresponseswithinregionsoftexture.Thisprocesslosesalotofinformationaboutthedistributionoflterresponses;amuchbettermethodistorep-resenttheneighborhoodaroundapixelbyahis-togramoflteroutputs(HeegerandBergen,1995;Puzichaetal.,1997).Whilethishasbeenshowntobeapowerfultechnique,itleavesopentwoimpor-tantquestions.Firstly,thereisthematterofwhatsizewindowtouseforpoolingthehistogramintegrationscale.Secondly,theseapproachesonlymakeuseofmarginalbinning,therebymissingoutontheinformativecharacteristicsthatjointassem-bliesoflteroutputsexhibitatpointsofinterest.Weaddresseachofthesequestionsinthefollowing2.1.TextonsThoughtherepresentationoftexturesusinglterre-sponsesisextremelyversatile,onemightsaythatitisoverlyredundant(eachpixelvalueisrepresentedbyreal-valuedlterresponses,whereis40forourlterset).Moreover,itshouldbenotedthatwearecharacterizingtextures,entitieswithsomespa-tiallyrepeatingpropertiesbydenition.Therefore,wedonotexpectthelterresponsestobetotallydiffer-entateachpixeloverthetexture.Thus,thereshouldbeseveraldistinctlterresponsevectorsandallothersarenoisyvariationsofthem.Thisobservationleadstoourproposalofcluster-ingthelterresponsesintoasmallsetofprototyperesponsevectors.Wecalltheseprototypestextons.Al-gorithmically,eachtextureisanalyzedusingthebankshowninFig.4.Eachpixelisnowtransformedtoadimensionalvectoroflterresponses.Thesevectorsareclusteredusing-means.Thecriterionforthisalgorithmistosuchthatafteras-signingeachdatavectortothenearestcenter,thesum ContourandTextureAnalysis13 Figure5.(a)Polka-dotimage.(b)Textonsfoundvia-meanswith25,sortedindecreasingorderbynorm.(c)Mappingofpixelstothetextonchannels.Thedominantstructurescapturedbythetextonsaretranslatedversionsofthedarkspots.Wealsoseetextonscorrespondingtofaintorientededgeandbarelements.Noticethatsomechannelscontainactivityinsideatexturedregionoralonganorientedcontourandnowhereelse.ofthesquareddistancefromthecentersisminimized.-meansisagreedyalgorithmthatndsalocalmini-mumofthiscriterion.Itisusefultovisualizetheresultingclustercentersintermsoftheoriginallterkernels.Todothis,recallthateachclustercenterrepresentsasetofprojectionsoflterontoaparticularimagepatch.Wecansolvefortheimagepatchcorrespondingtoeachclustercenterinaleastsquaressensebypremultiplyingthevectorsrepresentingtheclustercentersbythepseudoinverseoflterbank(JonesandMalik,1992).Thematrixrep-resentingthelterbankisformedbyconcatenatingthelterkernelsintocolumnsandplacingthesecolumnssidebyside.ThesetofsynthesizedimagepatchesfortwotestimagesareshowninFigs.5(b)and6(b).Theseareourtextons.Thetextonsrepresentassembliesoflteroutputsthatarecharacteristicofthelocalimagestructurepresentintheimage.Lookingatthepolka-dotexample,wendthatmanyofthetextonscorrespondtotranslatedversionsofdarkAlsoincludedareanumberoforientededgeelementsoflowcontrastandtwotextonsrepresenting Maliketal. Figure6.(a)Penguinimage.(b)Textonsfoundvia-meanswith25,sortedindecreasingorderbynorm.(c)Mappingofpixelstothetextonchannels.Amongthetextonsweseeedgeelementsofvaryingorientationandcontrastalongwithelementsofthestochastictextureintherocks.nearlyuniformbrightness.Thepixel-to-textonmap-pingisshowninFig.5(c).Eachsubimageshowsthepixelsintheimagethataremappedtothecorrespond-ingtextoninFig.5(b).Werefertothiscollectionofdiscretepointsetsasthetextonchannels.Sinceeachpixelismappedtoexactlyonetexton,thetextonchan-nelsconstituteapartitionoftheimage.TextonsandtextonchannelsarealsoshownforthepenguinimageinFig.6.Noticeinthetwoexampleshowmuchthetextonsetcanchangefromoneimagetothenext.Thespatialcharacteristicsofboththede-terministicpolkadottextureandthestochasticrockstexturearecapturedacrossseveraltextonchannels.Ingeneral,thetextureboundariesemergeaspointdensitychangesacrossthedifferenttextonchannels.Insomecases,atextonchannelcontainsactivityinsideapar-ticulartexturedregionandnowhereelse.Bycompari-son,vectorsoflteroutputsgenericallyrespondwithsomevalueateverypixelaconsiderablylesscleanalternative. ContourandTextureAnalysis15Wehavenotbeenparticularlysophisticatedinthechoiceof,thenumberofdifferenttextonsforagivenimage.Howtochooseanoptimalvalueofmeanshasbeenthesubjectofmuchresearchinthemodelselectionandclusteringliterature;weusedaxedchoice36toobtainthesegmentationresultsinthispaper.Clearly,iftheimagesvaryconsiderablyincomplexityandnumberofobjectsinthem,anadaptivechoicemaygivebetterresults.Themappingfrompixeltotextonchannelprovidesuswithanumberofdiscretepointsetswherebeforewehadcontinuous-valuedltervectors.Sucharepre-sentationiswellsuitedtotheapplicationoftechniquesfromcomputationalgeometryandpointprocessstatis-tics.Withthesetools,onecanapproachquestionssuchwhatistheneighborhoodofatextureelement?howsimilararetwopixelsinsideatexturedregion?Severalpreviousresearchershaveemployedcluster-ingusing-meansorvectorquantizationasastageintheirapproachtotextureclassitworepresen-tativeexamplesareMcLean(1993)andRaghuetal.(1997).Whatisnovelaboutourapproachistheidenti-cationofclustersofvectorsoflteroutputswiththeJulesznotionoftextons.Thenrstorderstatisticsoftextonsareusedfortexturecharacterization,andthespatialstructurewithintextonchannelsenablesscaleestimation.Vectorquantizationbecomesmuchmorethanjustadatacompressionorcodingstep.Thenextsubsectionshouldmakethispointclear.2.1.1.LocalScaleandNeighborhoodSelection.textonchannelrepresentationprovidesusanaturalwaytodenetexturescale.Ifthetextureiscomposedofdis-creteelements(texels),wemightwanttodeneano-tionoftexelneighborsandconsiderthemeandistance Figure7.Illustrationofscaleselection.(a)CloseupofDelaunaytriangulationofpixelsinaparticulartextonchannelforpolkadotimage.(b)Neighborsofthickenedpointforpixelatcenter.Thethickenedpointlieswithininnercircle.Neighborsarerestrictedtoliewithinoutercircle.(c)Selectedscalebasedonmedianofneighboredgelengths,shownbycircle,withallpixelsfallinginsidecirclemarkedwithdots.betweenthemtobeameasureofscale.Ofcourse,manytexturesarestochasticanddetectingtexelsreliablyishardevenforregulartextures.Withtextonswehaveawaytodeneneigh-bors.Foragivenpixelinatextonchannel,rstcon-sideritasathickenedpointadiskcenteredatit.Theideaisthatwhiletextonsarebeingassociatedwithpixels,sincetheycorrespondtoassembliesoflterout-puts,itisbettertothinkofthemascorrespondingtoasmallimagediskdenedbythescaleusedintheGaussianderivativelters.RecallKoenderinksapho-rismaboutapointinimageanalysisbeingaGaussianblobofsmallNowconsidertheDelaunayneighborsofallthepix-elsinthethickenedpointofapixelwhichliecloserthansomeouterscale.Theintuitionisthatthesewillbepixelsinspatiallyneighboringtexels.Computethedistancesofallthesepixelsto;themedianoftheseconstitutesarobustlocalmeasureofinter-texeldis-tance.Wedenethelocalscaletobe15timesthismediandistance.InFig.7(a),theDelaunaytriangulationofazoomed-inportionofoneofthetextonchannelsinthepolka-dotdressofFig.5(a)isshownatopabrightenedversionoftheimage.HerethenodesrepresentpointsthataresimilarintheimagewhiletheedgesprovideproximityThelocalscaleisbasedjustonthetextonchan-nelforthetextonat.Sinceneighboringpixelsshouldhavesimilarscaleandcouldbedrawnfromothertex-tonchannels,wecanimprovetheestimateofscalebylteringofthescaleimage.2.1.2.ComputingWindowedTextonHistograms.Pairwisetexturesimilaritieswillbecomputedbycom-paringwindowedtextonhistograms.Wedenethe Maliketal.windowforagenericpixelastheaxis-alignedsquareofradiuscenteredonpixelEachhistogramhasbins,oneforeachtextonchan-nel.Thevalueofthethhistogrambinforapixelfoundbycountinghowmanypixelsintextonchannelfallinsidethewindow.Thusthehistogramrep-resentstextonfrequenciesinalocalneighborhood.WecanwritethisasasTjk](2)(2) ]istheindicatorfunctionandthetextonassignedtopixel3.TheNormalizedCutFrameworkIntheNormalizedCutframework(ShiandMalik,1997,2000),whichisinspiredbyspectralgraphtheory(Chung,1997),ShiandMalikformulatevisualgroup-ingasagraphpartitioningproblem.Thenodesofthegrapharetheentitiesthatwewanttopartition;forex-ample,inimagesegmentation,theyarethepixels.Theedgesbetweentwonodescorrespondtothestrengthwithwhichthesetwonodesbelongtoonegroup;again,inimagesegmentation,theedgesofthegraphcorre-spondtohowmuchtwopixelsagreeinbrightness,color,etc.Intuitively,thecriterionforpartitioningthegraphwillbetominimizethesumofweightsofcon-acrossthegroupsandmaximizethesumofweightsofconnectionsthegroups.beaweightedundirectedgraph,arethenodesandaretheedges.Letbeapartitionofthegraph:.Ingraphtheoreticlanguage,thesimilaritybetweenthesetwogroupsiscalledtheistheweightontheedgebetweennodes.ShiandMalikproposedtouseasimilaritycriteriontoevaluateapartition.TheycallitnormalizedcutNcut assocAVcutBA isthetotalcon-nectionfromnodesintoallthenodesinthegraph.Formorediscussionofthiscriterion,pleaserefertoShiandMalik(2000).Onekeyadvantageofusingthenormalizedcutisthatagoodapproximationtotheoptimalpartitioncanbecomputedveryefciently.betheassociationmatrix,i.e.istheweightbetweennodesinthegraph.Letbethediagonalmatrixsuchthat,i.e.isthesumoftheweightsofalltheconnectionstonode.ShiandMalikshowedthattheoptimalpartitioncanbefoundbycomputing:argminargmin isabinaryindicatorvectorspeci-fyingthegroupidentityforeachpixel,i.e.pixelbelongstogroupifpixelisthenumberofpixels.NoticethattheaboveexpressionisaRayleighquotient.Ifwerelaxtotakeonrealvalues(insteadoftwodiscretevalues),wecanoptimizeEq.(3)bysolvingageneralizedeigenvaluesystem.Efcientalgorithmswithpolynomialrunningtimearewell-knownforsolvingsuchproblems.Theprocessoftransformingthevectorintoadis-cretebipartitionandthegeneralizationtomorethantwogroupsisdiscussedin(4.DeÞningtheWeightsThequalityofasegmentationbasedonNormalizedCutsoranyotheralgorithmbasedonpairwisesim-ilaritiesfundamentallydependsontheweightsthatareprovidedasinput.Theweightsshouldbelargeforpixelsthatshouldbelongtogetherandsmallotherwise.Wenowdiscussourmethodforcomputings.Sinceweseektocombineevidencefromtwocues,wewillrstdiscussthecomputationoftheweightsforeachcueinisolation,andthendescribehowthetwoweightscanbecombinedinameaningfulfashion.4.1.ImagesWithoutTextureConsiderforthemomentthecrackedearthimageinFig.1(e).Suchanimagecontainsnotextureandmaybetreatedinaframeworkbasedsolelyoncontourfeatures.Thedenitionoftheweightsinthiscase,whichwe,isadoptedfromtheinterveningcontourmethodintroducedinLeungandMalik(1998). ContourandTextureAnalysis17 Figure8.Left:theoriginalimage.Middle:partoftheimagemarkedbythebox.Theintensityvaluesatpixelsaresimilar.However,thereisacontourinthemiddle,whichsuggeststhatbelongtoonegroupwhilebelongstoanother.Justcomparingintensityvaluesatthesethreelocationswillmistakenlysuggestthattheybelongtothesamegroup.Right:orientationenergy.Somewherealong,theorientationenergyisstrongwhichcorrectlyproposesthatbelongtotwodifferentpartitions,whileorientationenergyalongisweakthroughout,whichwillsupportthehypothesisthatbelongtothesamegroup.Figure8illustratestheintuitionbehindthisidea.Ontheleftisanimage.Themiddlegureshowsamag-edpartoftheoriginalimage.Ontherightistheorientationenergy.Thereisanextendedcontoursep-.Thus,weexpecttobemuchmorestronglyrelatedto.Thisintuitioncarriesoverinourdenitionofdissimilaritybetweentwopixels:iftheorientationenergyalongthelinebe-tweentwopixelsisstrong,thedissimilaritybetweenthesepixelsshouldbehigh(andshouldbelow).Contourinformationinanimageiscomputedthroughorientationenergy(OE)fromelon-gatedquadraturelterpairs.Weintroduceaslightmod-cationheretoallowforexactsub-pixellocalizationofthecontourbyndingthelocalmaximaintheorien-tationenergyperpendiculartothecontourorientation(PeronaandMalik,1990).Theorientationenergygivesthecondenceofthiscontour.isthendenedasfollows:isthesetoflocalmaximaalongthelinejoin-ingpixels.Recallfrom(2)that1,isnearly1whenevertheorientatedenergymaximumatissufcientlyabovethenoiselevel.Inwords,twopixelswillhaveaweaklinkbetweenthemifthereisastronglocalmaximumoforientationenergyalongthelinejoiningthetwopixels.Onthecontrary,ifthereislittleenergy,forexampleinaconstantbright-nessregion,thelinkbetweenthetwopixelswillbestrong.Contoursmeasuredatdifferentscalescanbetakenintoaccountbycomputingtheorientationen-ergymaximaatvariousscalesandsettingtobethemaximumoverallthescalesateachpixel.4.2.ImagesthatareTextureMosaicsNowconsiderthecaseofimageswhereinalloftheboundariesarisefromneighboringpatchesofdifferenttexture(e.g.Fig.1(d)).Wecomputepairwisetexturesimilaritiesbycomparingwindowedtextonhistogramscomputedusingthetechniquedescribedpreviously2.1.2).Anumberofmethodsareavailableforcom-paringhistograms.Weusethetest,denedas 2Kk1[hikhjk]2 arethetwohistograms.Foranem-piricalcomparisonofthetestversusothertexturesimilaritymeasures,seePuzichaetal.(1997).isthendenedasfollows:expIfhistogramsareverydifferent,islarge,andtheweightissmall.4.3.GeneralImagesFinallyweconsiderthegeneralcaseofimagesthatcontainboundariesofbothkinds.Thispresentsuswiththeproblemofcueintegration.Theobviousapproachtocueintegrationistodenetheweightbetweenpixelsastheproductofthecontributionfromeach.Theideaisthatifeitherofthecuessuggeststhatshouldbeseparated,thecompositeweight,,shouldbesmall.Wemustbecareful,however,toavoidtheproblemslistedinthe Maliketal.Introduction(1)bysuitablygatingthecues.Thespiritofthegatingmethodistomakeeachcueinlocationswheretheothercueshouldbeoperating.4.3.1.EstimatingTexturedness.AsillustratedinFig.2,thefactthatapixelsurvivesthenon-maximumsuppressionstepdoesnotnecessarilymeanthatthatpixelliesonaregionboundary.Considerapixelinsideapatchofuniformtexture:itsorientedenergyislargebutitdoesnotlieontheboundaryofaregion.Con-versely,considerapixellyingbetweentwouniformpatchesofjustslightlydifferentbrightness:itdoeslieonaregionboundarybutitsorientedenergyissmall.Inordertoestimatethethatapixelliesonaboundary,itisnecessarytotakemoresurround-inginformationintoaccount.Clearlythetruevalueofthisprobabilityisonlydeterminedafterthecorrectsegmentation,whichiswhatweseektoAtthisstageourgoalistoformulatealocalestimateofthetexturednessoftheregionsurroundingapixel.Sincethisisalocalestimate,itwillbenoisybutitsobjectivewillbetobootstraptheglobalsegmentationOurmethodofcomputingthisvalueisbasedonasimplecomparisonoftextondistributionsoneithersideofapixelrelativetoitsdominantorientation.Consideragenericpixelatanorientedenergymaximum.Letthedominantorientationbe.Consideracircleofra-(theselectedscale)centeredon.Wedividethiscircleintwoalongthediameterwithori-.Notethatthecontourpassingthroughistangenttothediameter,whichisitsbeststraightlineapproximation.Thepixelsinthediskcanbeparti-tionedintothreesetswhicharethepixelsinthestripalongthediameter,thepixelstotheleft,andthepixelstotherightof,respectively.Tocomputeourmeasureoftexturedness,weconsidertwohalfwindowcomparisonswithassignedtoeachside.Assumewithoutlossofgeneralitythatrstassignedtothehalf.Denotethehistogramsoftively.Nowconsiderthestatisticbetweenthetwo 2Kk1[hLkhRk]2 Werepeatthetestwiththehistogramsofandretainthemaximumofthetworesult-ingvalues,whichwedenote.Wecanconvertthis Figure9.Illustrationofhalfwindowsusedfortheestimationofthetexturedness.Thetexturednessofalabelisbasedonaonthetextonsinthetwosidesofaboxasshownabovefortwosamplepixels.Thesizeandorientationoftheboxisdeterminedbytheselectedscaleanddominantorientationforthepixelatcenter.Withintherockyarea,thetextonstatisticsareverysimilar,leadingtoalowvalue.Ontheedgeofthewing,thevalueisrelativelyhighduetothedissimilarityofthetextonsthatreoneithersideofastepedge.Sinceinthecaseofthecontourthecontouritselfcanliealongthediameterofthecircle,weconsidertwohalf-windowpartitions:onewherethethinstriparoundthediameterisassignedtotheleftside,andonewhereitisassignedtotheother.Weconsiderbothpossibilitiesandretainthemaximumofthetworesultingvalues.toaprobability-likevalueusingasigmoidasfollows:texture expThisvalue,whichrangesbetween0and1,issmallifthedistributionsonthetwosidesareverydifferentandlargeotherwise.Notethatinthecaseofuntexturedre-gions,suchasabrightnessstepedge,thetextonslyingalongandparalleltotheboundarymakethestatisticsofthetwosidesdifferent.ThisisillustratedinFig.9.Roughly,texture1fororientedenergymaximaintex-tureandtexture0forcontours.textureisdenedtobe0atpixelswhicharenotorientedenergymaxima.4.3.2.GatingtheContourCue.Thecontourcueisgatedbymeansofsuppressingcontourenergyaccord-ingtothevalueoftexture.Thegatedvalue,,isde-nedastextureInprinciple,thisvaluecanbecomputedanddealtwithindependentlyateachlterscale.Forourpurposes,wefounditsufcientsimplytokeepthemaximumvalue ContourandTextureAnalysis19 Figure10.Gatingthecontourcue.Left:originalimage.Top:orientedenergyafternonmaximalsuppression,.Bottom:1texture.Right:,theproductof1textureexp.Notethatthiscanbethoughtofasaedgedetectorwhichhasbeenedtonolongerreontextureregions. Figure11.Gatingthetexturecue.Left:originalimage.Top:Textonslabel,showninpseudocolor.Middle:localscaleestimate.Bottom:texture.Darkergrayscaleindicateslargervalues.Right:Localtextonhistogramsatscalearegatedusingtextureasexplainedin4.3.3.withrespectto.ThegatedcontourenergyisillustratedinFig.10,right.Thecorrespondingweightisthengivenby4.3.3.GatingtheTextureCue.Thetexturecueisgatedbycomputingatextonhistogramateachpixelwhichtakesintoaccountthetexturednessmeasuretexture(seeFig.11).Letbethe-bintextonhis-togramcomputedusingEq.(2).Wedenea(binhistogrambyintroducinga0thbin.Theintuitionisthatthe0thbinwillkeepacountofthenumberofpixelswhichdonotcorrespondtotexture.Thesepix-elsariseintwoforms:(1)pixelswhicharenotorientedenergymaxima;(2)pixelswhichareorientedenergymaxima,butcorrespondtoboundariesbetweentwore-gions,thusshouldnottakepartintextureprocessingtoavoidtheproblemsdiscussedin(1).Moreprecisely,isdenedasfollows:textureeTjk]k1 Khi0NBj i1ptexture Maliketal.denotesalltheorientedenergymaximalyinginsidethewindowisthenumberofpixelswhicharenotorientedenergymaxima.4.3.4.CombiningtheWeights.Aftereachcuehasbeengatedbytheaboveprocedure,wearefreetoper-formsimplemultiplicationoftheweights.Morespecif-ically,werstobtainusingEq.(6).ThenweobtainusingEq.(4)withthegatedversionsofthehis-tograms.Thenwesimplydenethecombinedweight4.3.5.ImplementationDetails.Theweightmatrixisnedbetweenanypairofpixels.Naively,onemightconnecteverypairofpixelsintheimage.How-ever,thisisnotnecessary.Pixelsveryfarawayfromtheimagehaveverysmalllikelihoodofbelongingtothesameregion.Moreover,denseconnectivitymeansthatweneedtosolvefortheeigenvectorsofamatrixofsize,whereisclosetoamillionforatypicalimage.Inpractice,asparseandshort-rangedconnectionpatterndoesaverygoodjob.Inourex-periments,alltheimagesareofsize128192.Eachpixelisconnectedtopixelswithinaradiusof30.Fur-thermore,asparsesamplingisimplementedsuchthatthenumberofconnectionsisapproximatelyconstantateachradius.Thenumberofnon-zeroconnectionsperpixelis1000inourexperiments.Forimagesofdifferentsizes,theconnectionradiuscanbescaledap-propriately.Theparametersforthevariousformulaearegiven1.Theimagebrightnessliesintherange[002(Eq.(1)).3.Thenumberoftextonscomputedusing4.Thetextonsarecomputedfollowingacontrastnor-malizationstep,motivatedbyWeberslaw.Letbethenormofthelterresponsesatpixel.Wenormalizethelterresponsesbythefollowingequation: 0 03 025(Eq.(4)).3and04(Eq.(5))Notethattheseparametersarethesameforallthere-sultsshownin(5.ComputingtheSegmentationWithaproperlydenedweightmatrix,thenormal-izedcutformulationdiscussedin(3)canbeusedtocomputethesegmentation.However,theweightma-trixdenedintheprevioussectioniscomputedusingonlylocalinformation,andisthusnotperfect.Theidealweightshouldbecomputedinsuchawaythatregionboundariesarerespected.Moreprecisely,(1)textonhistogramsshouldbecollectedfrompixelsinawindowresidingexclusivelyinoneandonlyonere-gion.Ifinstead,anisotropicwindowisused,pixelsnearatextureboundarywillhaveahistogramcom-putedfromtextonsinbothregions,thusthehistogram.(2)Interveningcontoursshouldonlybeconsideredatregionboundaries.Anyresponsestotheltersinsidearegionareeithercausedbytextureoraresimplymistakes.However,thesetwocriteriameanthatweneedasegmentationoftheimage,whichisexactlythereasonwhywecomputetheweightsintheplace!Thischicken-and-eggproblemsuggestsaniter-ativeframeworkforcomputingthesegmentation.First,usethelocalestimationoftheweightstocomputeaseg-mentation.Thissegmentationisdonesothatnoregionboundariesaremissed,i.e.itisanover-segmentation.Next,usethisintialsegmentationtoupdatetheweights.Sincetheinitialsegmentationdoesnotmissanyregionboundaries,wecancoarsenthegraphbymergingallthenodesinsidearegionintoonesuper-node.Wecanthenusethesesuper-nodestodeneamuchsimplersegmentationproblem.Ofcourse,wecancontinuethisiterationseveraltimes.However,weelecttostopafter1iteration.Theprocedureconsistsofthefollowing4steps:1.Computeaninitialsegmentationfromthelocallyestimatedweightmatrix.2.Updatetheweightsusingtheinitialsegmentation.3.Coarsenthegraphwiththeupdatedweightstore-ducethesegmentationtoamuchsimplerproblem.4.Computeanalsegmentationusingthecoarsened5.1.ComputingtheInitialSegmentationComputingasegmentationoftheimageamountstocomputingtheeigenvectorsofthegeneralized ContourandTextureAnalysis21(Eq.(3)).Theeigenvec-torscanbethoughtofasatransformationoftheimageintoanewfeaturevectorspace.Inotherwords,eachpixelintheoriginalimageisnowrepresentedbyavec-torwiththecomponentscomingfromthecorrespond-ingpixelacrossthedifferenteigenvectors.Findingapartitionoftheimageisdonebyndingtheclustersinthiseigenvectorrepresentation.Thisisamuchsimplerproblembecausetheeigenvectorshaveessentiallyputregionsofcoherentdescriptorsaccordingtoourcueoftextureandcontourintoverytightclusters.Simpletechniquessuchas-meanscandoaverygoodjobndingtheseclusters.Thefollowingprocedureistaken:1.Computetheeigenvectorscorrespondingtothesec-ondsmallesttothetwelfthsmallesteigenvaluesofthegeneralizedeigensystem(Callthese11eigenvectors12.Thecorrespondingeigenvaluesare2.Weighttheeigenvectorsaccordingtotheeigen-values: 12.Theeigenval-uesindicatetheofthecorrespondingeigenvectors.Noweachpixelistransformedtoan11dimensionalvectorrepresentedbytheweightedeigenvectors.3.Performvectorquantizationonthe11eigenvectors-means.Startwith30centers.LetthecorrespondingRMSerrorforthequantizationbe.Greedilydeleteonecenteratatimesuchthattheincreaseinquantizationerroristhesmallest.Continuethisprocessuntilwearriveatwhentheerrorisjustgreaterthan1Thispartitioningstrategyprovidesuswithaninitialsegmentationoftheimage.Thisisusuallyanover-segmentation.Themaingoalhereissimplytoprovideaninitialguessforustomodifytheweights.Callthisinitialsegmentationoftheimage.Letthenumberofsegmentsbe.Atypicalnumberforis10 Figure12isallowedtobenon-zeroonlyatpixelsmarked.Itshouldbenotedthatthisstrategyforusingmulti-pleeigenvectorstoprovideaninitialoversegmentationismerelyoneofasetofpossibilities.Alternativesin-cluderecursivesplittingusingthesecondeigenvectorrstconvertingtheeigenvectorsintobinaryvaluedvectorsandusingthosesimultaneouslyasinShiandMalik(2000).YetanotherhybridstrategyissuggestedinWeiss(1999).Wehopethatimprovedtheoreticalin-sightintospectralgraphpartitioningwillgiveusabet-terwaytomakethis,presentlysomewhatadhoc5.2.UpdatingWeightsTheinitialsegmentationfoundinthepreviousstepcanprovideagoodapproximationtomodifytheweightaswehavediscussedearlier.With,wemodifytheweightmatrixasfollows:Tocomputethetextonhistogramsforapixelintextonsarecollectedonlyfromtheintersectionofandtheisotropicwindowofsizedeterminedbythescale,issettozeroforpixelsthatarenotintheregionboundariesofThemodiedweightmatrixisanimprovementovertheoriginallocalestimationofweights.5.3.CoarseningtheGraphByhypothesis,sinceisanover-segmentationoftheimage,therearenoboundariesmissed.Wedonotneedtorecomputeasegmentationfortheoriginalproblempixels.Wecancoarsenthegraph,whereeachnodeofthenewgraphisasegmentin.Theweightbetweentwonodesinthisnewgraphiscomputedasfollows: Maliketal. Figure13.Initialsegmentationoftheimageusedforcoarseningthegraphandcomputingnalsegmentation. Figure14.Segmentationofimageswithanimals. ContourandTextureAnalysis23 Figure15.Segmentationofimageswithpeople.indicatesegmentsinistheweightmatrixofthecoars-enedgraphandistheweightmatrixoftheorigi-nalgraph.Thiscoarseningstrategyisjustaninstanceofgraphcontraction(Chung,1997).Now,wehavereducedtheoriginalsegmentationproblemwithanweightmatrixtoamuchsimplerandfastersegmentationproblemofwithoutlosingin5.4.ComputingtheFinalSegmentationAftercoarseningthegraph,wehaveturnedthesegmen-tationproblemintoaverysimplegraphpartitioningproblemofverysmallsize.Wecomputethenalseg-mentationusingthefollowingprocedure:1.Computethesecondsmallesteigenvectorforthegeneralizedeigensystemusing2.Thresholdtheeigenvectortoproduceabi-partitioningoftheimage.30differentvaluesuni-formlyspacedwithintherangeoftheeigenvectoraretriedasthethreshold.Theoneproducingapar-titionwhichminimizesthenormalizedcutvalueischosen.Thecorrespondingpartitionisthebestwaytosegmenttheimageintotworegions.3.Recursivelyrepeatsteps1and2foreachofthepartitionsuntilthenormalizedcutvalueislargerthan0 Maliketal. Figure16.Segmentationofimagesofnaturalandman-madescenes.5.5.SegmentationinWindowsTheaboveprocedureperformsverywellinimageswithasmallnumberofgroups.However,incomplicatedimages,smallerregionscanbemissed.Thisproblemisintrinsicforglobalsegmentationtechniques,wherethegoalisndabig-pictureinterpretationoftheimage.Thisproblemcanbedealtwithveryeasilybyperform-ingthesegmentationinwindows.Considerthecaseofbreakinguptheimageintoquadrants.DetobethesetofpixelsintheImage.Ex-tendeachquadrantbyincludingallthepixelswhicharelessthanadistancefromanypixelsin,withbeingthemaximumtexturescale,,overthewholeimage.Calltheseenlargedwindows.Notethatthesewindowsnowoverlapeachother.Correspondingtoeach,aweightmatrixnedbypullingoutfromtheoriginalweightmatrixtheedgeswhoseend-pointsarenodesin.Foreach,aninitialsegmentationisobtained,accordingtotheprocedurein(5.1).Theweightsareupdatedasin(5.2).Theextensionofeachquadrantmakessurethatthearbitraryboundariescreatedbythewindowingdonotaffectthisprocedure:TextonhistogramupgradeForeachpixelin,thelargestpossiblehistogramwindow(aisentirelycontainedinbyvirtueoftheextension. ContourandTextureAnalysis25 Figure17.Segmentationofpaintings.Thismeansthetextonhistogramsarecomputedfromalltherelevantpixels.ContourupgradeTheboundariesinareapropersubsetoftheboundariesin.So,wecansetthevaluesofatapixelintobezeroifitliesonaregionboundaryin.Thisenablesthecorrectcomputationof.TwoexamplecontourupdatemapsareshowninFig.12.Initialsegmentationscanbecomputedforeachtogive.TheyarerestrictedtotoproduceThesesegmentationsaremergedtoformaninitialseg-.Atthisstage,fakeboundariesfromthewindowingeffectcanoccur.TwoexamplesareshowninFig.13.Thegraphisthencoarsenedandnalsegmentationiscomputedasin(5.3)and6.ResultsWehaverunouralgorithmonavarietyofnaturalim-ages.Figures1417showtypicalsegmentationresults.Inallthecases,theregionsarecleanlyseparatedfromeachotherusingcombinedtextureandcontourcues.Noticethatforalltheseimages,asinglesetofparam-etersareused.Colorisnotusedinanyoftheseex-amplesandcanreadilybeincludedtofurtherimprovetheperformanceofouralgorithm.Figure14showsresultsforanimalimages.Resultsforimagescontain-ingpeopleareshowninFig.15whilenaturaland Maliketal.man-madescenesappearinFig.16.Segmentationre-sultsforpaintingsareshowninFig.17.Asetofmorethan1000imagesfromthecommerciallyavail-ableCorelStockPhotosdatabasehavebeensegmentedusingouralgorithm.Evaluatingtheresultsagainstgroundtruthisthecorrectsegmentationoftheimage?isachal-lengingproblem.Thisisbecausetheremaynotbeasinglecorrectsegmentationandsegmentationscanbetovaryinglevelsofgranularity.Wedonotaddressthisproblemhere;astarthasbeenmadeinrecentworkinourgroup(Martinetal.,2000).ComputingtimesforaCimplementationoftheentiresystemareundertwominutesforimagesofsize176pixelsona750MHzPentiumIIImachine.Thereissomevariabilityfromoneimagetoanotherbecausetheeigensolvercantakemoreorlesstimetoconvergedependingontheimage.7.ConclusionInthispaperwehavedevelopedageneralalgorithmforpartitioninggrayscaleimagesintodisjointregionsofcoherentbrightnessandtexture.Thenovelcon-tributionoftheworkisincueintegrationforimagesegmentationthecuesofcontourandtexturediffer-encesareexploitedsimultaneously.Weregardtheex-perimentalresultsaspromisingandhopethatthepaperwillsparkrenewedresearchactivityinimagesegmen-tation,oneofthecentralproblemsofcomputervision.AcknowledgmentsTheauthorswouldliketothanktheBerkeleyvisiongroup,especiallyChadCarson,AlyoshaEfros,DavidForsyth,andYairWeissforusefuldiscussionsduringthedevelopmentofthealgorithm.WethankDoronTalforimplementingthealgorithminC.Thisre-searchwassupportedby(ARO)DAAH04-96-1-0341,theDigitalLibraryGrantIRI-9411334,NSFGraduateFellowshipstoSBandJSandaBerkeleyFellowshiptoTL.1.Formorediscussionsandvariationsofthe-meansalgorithm,thereaderisreferredtoDudaandHart(1973)andGershoandGray(1992).2.Itisstraightforwardtodevelopamethodformergingtranslatedversionsofthesamebasictexton,thoughwehavenotfounditnecessary.Merginginthismannerdecreasesthenumberofchan-nelsneededbutnecessitatestheuseofphase-shiftinformation.3.Thisissetto3%oftheimagedimensioninourexperiments.Thisistiedtotheintermediatescaleoftheltersinthelterset.4.Thisissetto10%oftheimagedimensioninourexperiments.5.FindingthetrueoptimalpartitionisanNP-hardproblem.6.Theeigenvectorcorrespondingtothesmallesteigenvalueiscon-stant,thususeless.7.Sincenormalizedcutcanbeinterpretedasaspring-masssystem(ShiandMalik,2000),thisnormalizationcomesfromtheequipar-titiontheoreminclassicalstatisticalmechanicswhichstatesthatifasystemisinequilibrium,thenithasequalenergyineachmode(BelongieandMalik,1998).8.Whencolorinformationisavailable,thesimilarityaproductof3terms:.Colorsim-ilarity,,iscomputedusingdifferencesovercolorhistograms,similartotexturemeasuredusingtexturehistograms.Moreover,colorcanclusteredinto,analogoustotex-9.Theseresultsareavailableatthefollowingwebpage:http://www.cs.berkeley.edu/projects/vision/Grouping/overview.htmlReferencesBelongie,S.,Carson,C.,Greenspan,H.,andMalik,J.1998.Color-andtexture-basedimagesegmentationusingEManditsappli-cationtocontent-basedimageretrieval.InProc.6thInt.Conf.ComputerVision,Bombay,India,pp.675Belongie,S.andMalik,J.1998.Findingboundariesinnaturalim-ages:Anewmethodusingpointdescriptorsandareacompletion.Proc.5thEuro.Conf.ComputerVision,Freiburg,Germany,pp.Binford,T.1981.Inferringsurfacesfromimages.cialIntelli-gence,17(1Canny,J.1986.Acomputationalapproachtoedgedetection.Trans.Pat.Anal.Mach.Intell.,8(6):679Chung,F.1997.SpectralGraphTheory,AMS.Providence,RI.DeValois,R.andDeValois,K.1988.SpatialVision.OxfordUniversityPress.NewYork,N.Y.Duda,R.andHart,P.1973.PatternClassicationandSceneAnaly-,JohnWiley&Sons.NewYork,N.Y.Elder,J.andZucker,S.1996.Computingcontourclosures.InProc.Euro.Conf.ComputerVision,Vol.I,Cambridge,England,pp.399Fogel,I.andSagi,D.1989.Gaborltersastexturediscriminator.BiologicalCybernetics,61:103Geman,S.andGeman,D.1984.Stochasticrelaxation,Gibbsdistri-bution,andtheBayesianretorationofimages.IEEETrans.PatternAnal.Mach.Intell.,6:721Gersho,A.andGray,R.1992.VectorQuantizationandSignalCom-pression,KluwerAcademicPublishers,Boston,MA.Heeger,D.J.andBergen,J.R.1995.Pyramid-basedtextureanaly-sis/synthesis.InProceedingsofSIGGRAPH,pp.229Jacobs,D.1996.RobustandefcientdetectionofsalientconvexIEEETrans.PatternAnal.Mach.Intell.,18(1):23Jones,D.andMalik,J.1992.Computationalframeworktodeter-miningstereocorrespondencefromasetoflinearspatialImageandVisionComputing,10(10):699 ContourandTextureAnalysis27Julesz,B.1981.Textons,theelementsoftextureperception,andtheirNature,290(5802):91Knutsson,H.andGranlund,G.1983.Textureanalysisusingtwo-dimensionalquadraturelters.InWorkshoponComputerArchi-tectureforPatternAnalysisandImageDatabaseManagementpp.206Koenderink,J.andvanDoorn,A.1987.Representationoflocalge-ometryinthevisualsystem.BiologicalCybernetics,55(6):367Koenderink,J.andvanDoorn,A.1988.Operationalsignicanceofreceptiveeldassemblies.BiologicalCybernetics,58:163Leung,T.andMalik,J.1998.Contourcontinuityinregion-basedimagesegmentation.InProc.Euro.Conf.ComputerVision,Vol.1,H.BurkhardtandB.Neumann(Eds.).Freiburg,Germany,pp.544Leung,T.andMalik,J.1999.Recognizingsurfacesusingthree-dimensionaltextons.InProc.Int.Conf.ComputerVision,Corfu,Greece,pp.1010Malik,J.,Belongie,S.,Shi,J.,andLeung,T.1999.Textons,contoursandregions:Cueintegrationinimagesegmentation.InProc.IEEEIntl.Conf.ComputerVision,Vol.2,Corfu,Greece,pp.918Malik,J.andPerona,P.1990.Preattentivetexturediscriminationwithearlyvisionmechanisms.J.OpticalSocietyofAmerica,7(2):923Malik,J.andPerona,P.1992.Findingboundariesinimages.InralNetworksforPerception,Vol.1,H.Wechsler(Ed.).AcademicPress,pp.315Martin,D.,Fowlkes,C.,Tal,D.,andMalik,J.2000.Adatabaseofhumansegmentednaturalimagesanditsapplicationtoevaluat-ingsegmentationalgorithmsandmeasuringecologicalstatistics.TechnicalReportUCBCSD-01-1133,UniversityofCaliforniaatBerkeley.http://http.cs.berkeley.edu/projects/vision/Grouping/overview.html.McLean,G.1993.VectorquantizationfortextureclassiIEEETransactionsonSystems,Man,andCybernetics,23(3):637Montanari,U.1971.OntheoptimaldetectionofcurvesinnoisyComm.Ass.Comput.,14:335Morrone,M.andBurr,D.1988.Featuredetectioninhumanvision:Aphasedependentenergymodel.Proc.R.Soc.Lond.B,235:221Morrone,M.andOwens,R.1987.Featuredetectionfromlocalen-ergy.PatternRecognitionLetters,6:303Mumford,D.andShah,J.1989.Optimalapproximationsbypiece-wisesmoothfunctions,andassociatedvariationalproblems.Comm.PureMath.,42:577Parent,P.andZucker,S.1989.Traceinference,curvatureconsis-tency,andcurvedetection.IEEETrans.PatternAnal.Mach.In-,11(8):823Perona,P.andMalik,J.1990.Detectingandlocalizingedgescom-posedofsteps,peaksandroofs.InProc.3rdInt.Conf.ComputerVision,Osaka,Japan,pp.52Puzicha,J.,Hofmann,T.,andBuhmann,J.1997.Non-parametricsimilaritymeasuresforunsupervisedtexturesegmentationandimageretrieval.InProc.IEEEConf.ComputerVisionandPatternRecognition,SanJuan,PuertoRico,pp.267Raghu,P.,Poongodi,R.,andYegnanarayana,B.1997.Unsupervisedtextureclassicationusingvectorquantizationanddeterministicrelaxationneuralnetwork.IEEETransactionsonImageProcess-,6(10):1376ashua,A.andUllman,S.1988.Structuralsaliency:Thedetec-tionofgloballysalientstructuresusingalocallyconnectednet-work.InProc.2ndInt.Conf.ComputerVision,Tampa,FL,USA,pp.321Shi,J.andMalik,J.1997.Normalizedcutsandimagesegmentation.Proc.IEEEConf.ComputerVisionandPatternRecognitionSanJuan,PuertoRico,pp.731Shi,J.andMalik,J.2000.Normalizedcutsandimagesegmentation.IEEETrans.PatternAnal.Mach.Intell.,22(8):888Weiss,Y.1999.Segmentationusingeigenvectors:Aunifyingview.Proc.IEEEIntl.Conf.ComputerVision,Vol.2,Corfu,Greece,pp.975Wertheimer,M.1938.Lawsoforganizationinperceptualforms(par-tialtranslation).InASourcebookofGestaltPsychology,W.Ellis(Ed.).HarcourtBraceandCompany,pp.71Williams,L.andJacobs,D.1995.Stochasticcompletionelds:Aneuralmodelofillusorycontourshapeandsalience.InProc.5thInt.Conf.ComputerVision,Cambridge,MA,pp.408Young,R.A.1985.TheGaussianderivativetheoryofspa-tialvision:Analysisofcorticalcellreceptiveeldline-weightingproles.TechnicalReportGMR-4920,GeneralMotors