/
Normalized Cuts and Image Segmentation Jianbo Shi and Jitendra Malik Member IEEE Abstract Normalized Cuts and Image Segmentation Jianbo Shi and Jitendra Malik Member IEEE Abstract

Normalized Cuts and Image Segmentation Jianbo Shi and Jitendra Malik Member IEEE Abstract - PDF document

yoshiko-marsland
yoshiko-marsland . @yoshiko-marsland
Follow
542 views
Uploaded On 2014-12-05

Normalized Cuts and Image Segmentation Jianbo Shi and Jitendra Malik Member IEEE Abstract - PPT Presentation

Rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image We treat image segmentation as a graph partitioning problem and propose a novel global criterion the n ID: 21088

Rather than focusing

Share:

Link:

Embed:

Download Presentation from below link

Download Pdf The PPT/PDF document "Normalized Cuts and Image Segmentation J..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

NormalizedCutsandImageSegmentationJianboShiandJitendraMalik,IEEEÐWeproposeanovelapproachforsolvingtheperceptualgroupingprobleminvision.Ratherthanfocusingonlocalfeaturesandtheirconsistenciesintheimagedata,ourapproachaimsatextractingtheglobalimpressionofanimage.Wetreatimagesegmentationasagraphpartitioningproblemandproposeanovelglobalcriterion,thenormalizedcut,forsegmentingthegraph.Thenormalizedcutcriterionmeasuresboththetotaldissimilaritybetweenthedifferentgroupsaswellasthetotalsimilaritywithinthe æ 75yearsago,Wertheimer[24]pointedouttheimportanceofperceptualgroupingandorganizationinvisionandlistedseveralkeyfactors,suchassimilarity,proximity,andgoodcontinuation,whichleadtovisualgrouping.However,eventothisday,manyofthecomputationalissuesofperceptualgroupinghavere-mainedunresolved.Inthispaper,wepresentageneral J.ShiiswiththeRoboticsInstitute,CarnegieMellonUniversity,5000ForbesAve.,Pittsburgh,PA15213.E-mail:jshi@cs.cmu.eduJ.MalikiswiththeElectricalEngineeringandComputerScienceDivision,UniversityofCaliforniaatBerkeley,Berkeley,CA94720.E-mail:malik@cs.berkeley.edu.Manuscriptreceived4Feb.1998;accepted16Nov.1999.RecommendedforacceptancebyM.Shah.Forinformationonobtainingreprintsofthisarticle,pleasesende-mailto:tpami@computer.org,andreferenceIEEECSLogNumber107618.0162-8828/00/$10.002000IEEE methods,thesegmentationcriteriausedinmostofthemarebasedonlocalpropertiesofthegraph.Becauseperceptualgroupingisaboutextractingtheglobalimpressionsofascene,aswesawearlier,thispartitioningcriterionoftenfallsshortofthismaingoal.Inthispaper,weproposeanewgraph-theoreticcriterionformeasuringthegoodnessofanimagepartitionÐthenormalizedcut.WeintroduceandjustifythiscriterioninSection2.Theminimizationofthiscriterioncanbeformulatedasageneralizedeigenvalueproblem.Theeigenvectorscanbeusedtoconstructgoodpartitionsoftheimageandtheprocesscanbecontinuedrecursivelyasdesired(Section2.1).Section3givesadetailedexplanationofthestepsofourgroupingalgorithm.InSection4,weshowexperimentalresults.Theformulationandminimiza-tionofthenormalizedcutcriteriondrawsonabodyofresultsfromthefieldofspectralgraphtheory(Section5).RelationshiptoworkincomputervisionisdiscussedinSection6andcomparisonwithrelatedeigenvectorbasedsegmentationmethodsisrepresentedinSection6.1.WeconcludeinSection7.Themainresultsinthispaperwerefirstpresentedin[20].ROUPINGASAgraphcanbepartitionedintotwodisjointA;BBBˆV,A\Bˆ;,bysimplyremovingedgesconnectingthetwoparts.Thedegreeofdissimilaritybetweenthesetwopiecescanbecomputedastotalweightoftheedgesthathavebeenremoved.Ingraphtheoreticlanguage,itiscalledtheA;Bu;vTheoptimalbipartitioningofagraphistheonethatminimizesthisvalue.Althoughthereareanexponentialnumberofsuchpartitions,findingtheminimumcutofagraphisawell-studiedproblemandthereexistefficientalgorithmsforsolvingit.WuandLeahy[25]proposedaclusteringmethodbasedonthisminimumcutcriterion.Inparticular,theyseektopartitionagraphintok-subgraphssuchthatthemaximumcutacrossthesubgroupsisminimized.Thisproblemcanbeefficientlysolvedbyrecursivelyfindingtheminimumcutsthatbisecttheexistingsegments.AsshowninWuandLeahy'swork,thisgloballyoptimalcriterioncanbeusedtoproducegoodsegmentationonsomeoftheimages.However,asWuandLeahyalsonoticedintheirwork,theminimumcutcriteriafavorscuttingsmallsetsofisolatednodesinthegraph.Thisisnotsurprisingsincethecutdefinedin(1)increaseswiththenumberofedgesgoingacrossthetwopartitionedparts.Fig.1illustratesonesuchcase.Assumingtheedgeweightsareinverselyproportionaltothedistancebetweenthetwonodes,weseethecutthatpartitionsoutnodewillhaveaverysmallvalue.Infact,anycutthatpartitionsoutindividualnodesontherighthalfwillhavesmallercutvaluethanthecutthatpartitionsthenodesintotheleftandrighthalves.Toavoidthisunnaturalbiasforpartitioningoutsmallsetsofpoints,weproposeanewmeasureofdisassociationbetweentwogroups.Insteadoflookingatthevalueoftotaledgeweightconnectingthetwopartitions,ourmeasurecomputesthecutcostasafractionofthetotaledgeconnectionstoallthenodesinthegraph.WecallthisdisassociationmeasurethenormalizedcutNcutA;B A;BA;V A;BB;VA;Vu;tisthetotalconnectionfromnodesinAtoallnodesinthegraphandB;Vsimilarlydefined.Withthisdefinitionofthedisassociationbetweenthegroups,thecutthatpartitionsoutsmallisolatedpointswillnolongerhavesmallvalue,sincevaluewillalmostcertainlybealargepercentageofthetotalconnectionfromthatsmallsettoallothernodes.InthecaseillustratedinFig.1,weseethattheacrossnodewillbe100percentofthetotalconnectionfromthatnode.Inthesamespirit,wecandefineameasurefortotalnormalizedassociationwithingroupsforagivenpartition:NassocA;B A;AA;V B;BB;VA;AB;Baretotalweightsofedgesconnectingnodeswithin,respectively.Weseeagainthisisanunbiasedmeasure,whichreflectshowtightlyonaveragenodeswithinthegroupareconnectedtoeachother.Anotherimportantpropertyofthisdefinitionofassocia-tionanddisassociationofapartitionisthattheyarenaturallyrelated:NcutA;B A;BA;V A;BB;V A;VA;AA;V B;VB;BB;V A;AA;V B;BB;VNassocA;BHence,thetwopartitioncriteriathatweseekinourgroupingalgorithm,minimizingthedisassociationbetweenthegroupsandmaximizingtheassociationwithinthe SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.1.Acasewhereminimumcutgivesabadpartition. groups,areinfactidenticalandcanbesatisfiedsimulta-neously.Inouralgorithm,wewillusethisnormalizedcutasthepartitioncriterion.Unfortunately,minimizingnormalizedcutexactlyisNP-complete,evenforthespecialcaseofgraphsongrids.Theproof,duetoPapadimitriou,canbefoundinAppendixA.However,wewillshowthat,whenweembedthenormal-izedcutproblemintherealvaluedomain,anapproximatediscretesolutioncanbefoundefficiently.2.1ComputingtheOptimalPartitionGivenapartitionofnodesofagraph,V,intotwosetsAandB,letbeandimensionalindicatorvector,isinAand,otherwise.Leti;jbethetotalconnectionfromnodetoallothernodes.Withthe,wecanrewriteNcutA;BNcutA;B A;BA;V B;AB;V P…xxi�0;xxj0†ÿwijxxixxjPxxi�0ddi‡ beandiagonalmatrixwithonitsdiagonal,beansymmetricalmatrixwith beanvectorofallones.Usingthefact 11‡xx2and areindicatorvectorsfor,respectively,wecanrewriteNcut …1‡xx†T…DÿW†…1‡xx†k1TD1‡ …1ÿxx†T…DÿW†…1ÿxx†…1ÿk†1TD1ˆ …xxT…DÿW†xx‡1T…DÿW†1†k…1ÿk†1TD1‡ wecanthenfurtherexpandtheaboveequationas: … …xx…1ÿ2k† …xx†k…1ÿk†Mˆ … …xx…1ÿ2k† …xx†k…1ÿk†Mÿ 2… …xx†M‡ 2 …xx†M‡ Droppingthelastconstantterm,whichinthiscaseequals0,weget …1ÿ2k‡2k2†… …xx…1ÿ2k† …xx†k…1ÿk†M‡ 2 …xx†Mˆ …1ÿ2k‡2k2†…1ÿk†2… …xx …1ÿ2k†…1ÿk†2 …xx† k1ÿkM‡ 2 …xx†M:Lettingbˆ ,andsince,itbecomes …1‡b2†… …xx…1ÿb2† …xx†bM‡ 2b …xx†bMˆ …1‡b2†… …xx†bM‡ 2…1ÿb2† …xx†bM‡ 2b …xx†bMÿ 2b bMˆ …1‡b2†…xxT…DÿW†xx‡1T…DÿW†1†b1TD1‡ 2…1ÿb2†1T…DÿW†xxb1TD1‡ 2bxxT…DÿW†xxb1TD1ÿ 2b1T…DÿW†1b1TD1ˆ …1‡xx†T…DÿW†…1‡xx†b1TD1‡ b2…1ÿxx†T…DÿW†…1ÿxx†b1TD1ÿ 2b…1ÿxx†T…DÿW†…1‡xx†b1TD1ˆ ,itiseasytoseethat k1ÿkˆ Puttingeverythingtogetherwehave,Ncut withthecondition†2fNotethattheaboveexpressionistheRayleighquotient[11].Ifisrelaxedtotakeonrealvalues,wecanminimize(5)bysolvingthegeneralizedeigenvaluesystem,However,wehavetwoconstraintsonwhichcomefromtheconditiononthecorrespondingindicatorvector.First,considertheconstraint.Wecanshowthisconstraintonisautomaticallysatisfiedbythesolutionofthegeneralizedeigensystem.Wewilldosobyfirst890IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 transforming(6)intoastandardeigensystemandshowingthecorrespondingconditionissatisfiedthere.Rewrite(6)as 12…DÿW†Dÿ 12zzˆzz;…7†wherezzˆD .Onecaneasilyverifythat isaneigenvectorof(7)witheigenvalueof0.Furthermore, 12…DÿW†Dÿ issymmetricpositivesemidefinitesince,alsocalledthematrix,isknowntobepositivesemidefinite[18].Hence,is,infact,thesmallesteigenvectorof(7)andalleigenvectorsof(7)areperpendi-culartoeachother.Inparticular,,thesecondsmallesteigenvector,isperpendicularto.Translatingthisstate-mentbackintothegeneraleigensystem(6),wehave:isthesmallesteigenvectorwitheigenvalueofand2),whereisthesecondsmallesteigenvectorof(6).Now,recallasimplefactabouttheRayleighquotientquotientLetAbearealsymmetricmatrix.Undertheconstraintthatisorthogonaltothej-1smallesteigenvectors...thequotient isminimizedbythenextsmallestanditsminimumvalueisthecorrespondingAsaresult,weobtain: zzTDÿ 12…DÿW†Dÿ and,consequently, Thus,thesecondsmallesteigenvectorofthegeneralizedeigensystem(6)istherealvaluedsolutiontoournormal-izedcutproblem.Theonlyreasonthatitisnotnecessarilythesolutiontoouroriginalproblemisthatthesecondconstraintontakesontwodiscretevaluesisnotautomaticallysatisfied.Infact,relaxingthisconstraintiswhatmakesthisoptimizationproblemtractableinthefirstplace.WewillshowinSection3howthisrealvaluedsolutioncanbetransformedintoadiscreteform.Asimilarargumentcanalsobemadetoshowthattheeigenvectorwiththethirdsmallesteigenvalueistherealvaluedsolutionthatoptimallysubpartitionsthefirsttwoparts.Infact,thislineofargumentcanbeextendedtoshowthatonecansubdividetheexistinggraphs,eachtimeusingtheeigenvectorwiththenextsmallesteigenvalue.How-ever,inpractice,becausetheapproximationerrorfromtherealvaluedsolutiontothediscretevaluedsolutionaccumulateswitheveryeigenvectortakenandalleigen-vectorshavetosatisfyaglobalmutualorthogonalityconstraint,solutionsbasedonhighereigenvectorsbecomeunreliable.Itisbesttorestartsolvingthepartitioningproblemoneachsubgraphindividually.Itisinterestingtonotethat,whilethesecondsmallestof(6)onlyapproximatestheoptimalnormal-izedcutsolution,itexactlyminimizesthefollowing inreal-valueddomain,wherei;i.Roughlyspeaking,thisforcestheindicatorvectortotakesimilarvaluesfornodesthataretightlycoupled(largeInsummary,weproposeusingthenormalizedcutcriterionforgraphpartitioningandwehaveshownhowthiscriterioncanbecomputedefficientlybysolvingageneralizedeigenvalueproblem.Ourgroupingalgorithmconsistsofthefollowingsteps:Givenanimageorimagesequence,setupaweightedgraphandsettheweightontheedgeconnectingtwonodestobeameasureofthesimilaritybetweenthetwonodes.foreigenvectorswiththesmallesteigenvalues.Usetheeigenvectorwiththesecondsmallesteigenvaluetobipartitionthegraph.DecideifthecurrentpartitionshouldbesubdividedandrecursivelyrepartitionthesegmentedpartsifThegroupingalgorithm,aswellasitscomputationalcomplexity,canbebestillustratedbyusingthefollowing3.1Example:BrightnessImagesFig.2showsanimagethatwewouldliketosegment.Thestepsare:Constructaweightedgraphbytakingeachpixelasanodeandconnectingeachpairofpixelsbyanedge.Theweightonthatedgeshouldreflectthelikelihoodthatthetwopixelsbelongtooneobject.Usingjustthebrightnessvalueofthepixelsandtheirspatiallocation,wecandefinethegraphedgeweightconnectingthetwonodes ÿkFF…i†ÿFF…j†k222Ie 0otherwiseSolvefortheeigenvectorswiththesmallesteigen-valuesofthesystemAswesawabove,thegeneralizedeigensystemin(12)canbetransformedintoastandardeigenvalueproblemof 12…DÿW†Dÿ Solvingastandardeigenvalueproblemforalleigenvectorstakesoperations,whereisthenumberofnodesinthegraph.Thisbecomesimpracticalforimagesegmentationapplicationsisthenumberofpixelsinanimage. SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fortunately,ourgraphpartitioninghasthefollow-ingproperties:1)Thegraphsareoftenonlylocallyconnectedandtheresultingeigensystemsareverysparse,2)onlythetopfeweigenvectorsareneededforgraphpartitioning,and3)theprecisionrequire-mentfortheeigenvectorsislow,oftenonlytherightsignbitisrequired.ThesespecialpropertiesofourproblemcanbefullyexploitedbyaneigensolvercalledtheLanczosmethod.TherunningtimeofaLanczosalgorithmisswheremisthemaximumnumberofmatrix-vectorcomputationsrequiredandisthecostofamatrix-vectorcomputationof,where 12…DÿW†Dÿ .Notethatthesparsitystruc-tureofisidenticaltothatoftheweightmatrixissparse,soisandthematrix-vectorcomputationisonlyToseewhythisisthecase,wewilllookatthecostoftheinnerproductofonerowofwithavector.ForafixedisonlynonzeroifnodeisinaspatialneighborhoodofHence,thereareonlyafixednumberofoperationsrequiredforeachandthetotalcostofTheconstantfactorisdeterminedbythesizeofthespatialneighborhoodofanode.Itturnsoutthatwecansubstantiallycutdownadditionalconnec-tionsfromeachnodetoitsneighborsbyrandomlyselectingtheconnectionswithintheneighborhoodfortheweightedgraph.Empirically,wehavefoundthatonecanremoveupto90percentofthetotalconnectionswitheachoftheneighborhoodswhentheneighborhoodsarelargewithoutaffectingtheeigenvectorsolutiontothesystem.Puttingeverythingtogether,eachofthematrix-vectorcomputationscostoperationswithasmallconstantfactor.Thenumberdependsonmanyfactors[11].Inourexperimentsonimagesegmentation,weobservedthatistypicallyless Fig.3showsthesmallesteigenvectorscomputedforthegeneralizedeigensystemwiththeweightmatrixdefinedabove.Oncetheeigenvectorsarecomputed,wecanparti-tionthegraphintotwopiecesusingthesecondsmallesteigenvector.Intheidealcase,theeigenvec-torshouldonlytakeontwodiscretevaluesandthesignsofthevaluescantellusexactlyhowtopartitionthegraph.However,oureigenvectorscantakeoncontinuousvaluesandweneedtochooseasplittingpointtopartitionitintotwoparts.Therearemanydifferentwaysofchoosingsuchasplittingpoint.Onecantake0orthemedianvalueasthesplittingpointoronecansearchforthesplittingpointsuchthattheresultingpartitionhasthebestNcutA;Bvalue.Wetakethelatterapproachinourwork.Currently,thesearchisdonebycheckingevenlyspacedpossiblesplittingpoints,andcomput-ingthebestamongthem.Inourexperiments,thevaluesintheeigenvectorsareusuallywellseparatedandthismethodofchoosingasplittingpointisveryreliableevenwithasmallAfterthegraphisbrokenintotwopieces,wecanrecursivelyrunouralgorithmonthetwopartitionedparts.Or,equivalently,wecouldtakeadvantageofthespecialpropertiesoftheothertopeigenvectorsasexplainedintheprevioussectiontosubdividethegraphbasedonthoseeigenvectors.Therecursionstopsoncethevalueexceedscertainlimit.Wealsoimposeastabilitycriteriononthepartition.Aswesawearlier,andasweseeintheeigenvectorswiththeseventhtoninthsmallesteigenvalues(Fig.3g-h),sometimesaneigenvectorcantakeontheshapeofacontinuousfunction,ratherthatthediscreteindicatorfunctionthatweseek.Fromtheviewofsegmentation,suchaneigenvectorisattemptingtosubdivideanimageregionwherethereisnosurewayofbreakingit.Infact,ifweareforcedtopartitiontheimagebasedonthiseigenvector,wewillseetherearemanydifferentsplittingpointswhichhavesimilarNcutHence,thepartitionwillbehighlyuncertainandunstable.Inourcurrentsegmentationscheme,wesimplychoosetoignoreallthoseeigenvectorswhichhavesmoothlyvaryingeigenvectorvalues.Weachievethisbyimposingastabilitycriterionwhichmeasuresthedegreeofsmoothnessintheeigenvec-torvalues.Thesimplestmeasureisbasedonfirstcomputingthehistogramoftheeigenvectorvaluesandthencomputingtheratiobetweentheminimumandmaximumvaluesinthebins.Whentheeigenvectorvaluesarecontinuouslyvarying,thevaluesinthehistogrambinswillstayrelativelythesameandtheratiowillberelativelyhigh.Inourexperiments,wefindthatsimplethresholdingontheratiodescribedabovecanbeusedtoexcludeunstableeigenvectors.Wehavesetthatvaluetobe0.06inallourexperiments.Fig.4showsthefinalsegmentationfortheimageshowninFig.2.3.2RecursiveTwo-WayNcutInsummary,ourgroupingalgorithmconsistsofthefollowingsteps:892IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 Fig.2.Agraylevelimageofabaseballgame. Givenasetoffeatures,setupaweightedgraph,computetheweightoneachedge,andsummarizetheinformationintoforeigenvectorswiththesmallesteigenvalues.UsetheeigenvectorwiththesecondsmallesteigenvaluetobipartitionthegraphbyfindingthesplittingpointsuchthatNcutisminimized.Decideifthecurrentpartitionshouldbesubdividedbycheckingthestabilityofthecut,andmakesureNcutisbelowtheprespecifiedvalue.RecursivelyrepartitionthesegmentedpartsifThenumberofgroupssegmentedbythismethodiscontrolleddirectlybythemaximumallowedNcut3.3SimultanousK-WayCutwithMultipleEigenvectorsOnedrawbackoftherecursive2-waycutisitstreatmentoftheoscillatoryeigenvectors.Thestabilitycriteriakeepsusfromcuttingoscillatoryeigenvectors,butitalsopreventsuscuttingthesubsequenteigenvectorswhichmightbeperfectpartitioningvectors.Also,theapproachiscomputationallywasteful;onlythesecondeigenvectorisused,whereasthenextfewsmalleigenvectorsalsocontainusefulpartitioningInsteadoffindingthepartitionusingrecursive2-waycutasdescribedabove,onecanuseallofthetopeigenvectorstosimultanouslyobtainaK-waypartition.Inthismethod,topeigenvectorsareusedasdimensionalindicatorvectorsforeachpixel.Inthefirststep,asimpleclusteringalgorithm,suchasthek-meansalgorithm,isusedtoobtainanoversegmentationoftheimageintogroups.NoattemptismadetoidentifyandexcludeoscillatoryeigenvectorsÐtheyexacerbatetheoversegmentation,butthatwillbedealtwithsubsequently.Inthesecondstep,onecanproceedinthefollowingtwoGreedypruning:Iterativelymergetwosegmentsatatimeuntilonlysegmentsareleft.Ateachmergestep,thosetwosegmentsaremergedthatminimizeNcutcriteriondefinedas:Ncut cut…A1;VÿA1†assoc…A1;V†‡ cut…A2;VÿA2†assoc…A2;V†‡ cut…Ak;AÿAk†assoc…Ak;V†;…14† SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.3.Subplot(a)plotsthesmallesteigenvectorsofthegeneralizedeigenvaluesystem(11).Subplots(b)-(i)showtheeigenvectorscorrespondinthesecondsmallesttotheninthsmallesteigenvaluesofthesystem.Theeigenvectorsarereshapedtobethesizeoftheimage. isthethsubsetofwholesetThiscomputationcanbeefficientlycarriedoutbyiterativelyupdatingthecompactedweightmatrix,withi;jGlobalrecursivecut.Fromtheinitialsegments,wecanbuildacondensedgraph,whereeachsegmentcorrespondstoanodeofthegraph.Theweightoneachgraphedgei;jdefinedtobe,thetotaledgeweightsfromelementsintoelementsin.Fromthiscondensedgraph,wethenrecursivelybipartitionthegraphaccordingtheNcutcriterion.Thiscanbecarriedouteitherwiththegeneralizedeigenvaluesystem,asinSection3.2,orwithexhaustivesearchinthediscretedomain.Exhaustivesearchispossibleinthiscasesinceissmall,typicallyWehaveexperimentedwiththissimultanous-waycutmethodonourrecenttestimages.However,theresultspresentedinthispaperareallbasedontherecursive2-waypartitioningalgorithmoutlinedinSection3.2.Wehaveappliedourgroupingalgorithmtoimagesegmentationbasedonbrightness,color,texture,ormotioninformation.Inthemonocularcase,weconstructthegraphbytakingeachpixelasanodeanddefinetheedgeweightbetweennodeastheproductofafeaturesimilaritytermandspatialproximityterm: ÿkFF…i†ÿFF…j†k22Ie 0otherwiseisthespatiallocationofnode,andisafeaturevectorbasedonintensity,color,ortextureinforma-tionatthatnodedefinedas:,inthecaseofsegmentingpointsets,,theintensityvalue,forsegmentingbrightnessimages,v;v‰Š…,whereh;s;varetheHSVvalues,forcolorsegmentation,†ˆj‰Š…,wheretheDOOGfiltersatvariousscalesandorientationsasusedin[16],inthecaseoftexturesegmentation.Notethattheweightforanypairofnodesthataremorethanpixelsapart.Wefirsttestedourgroupingalgorithmonspatialpointsets.Fig.5showsapointsetandthesegmentationresult.Thenormalizedcutcriterionisindeedabletopartitionthepointsetinadesirableway.Figs.4,6,7,and8showtheresultofoursegmentationalgorithmonvariousbrightnessimages.Figs.6and7aresyntheticimageswithaddednoise.Figs.4and8arenaturalimages.NotethattheªobjectsºinFig.8haveratherill-definedboundaries,whichwouldmakeedgedetectionperformpoorly.Fig.9showsthesegmentationonacolorimage,reproducedingrayscaleinthesetransactions.Theoriginalimageandmanyotherexamplescanbefoundatwebsitehttp://www.cs.berkeley.edu/~jshi/Grouping.Notethat,inalltheseexamples,thealgorithmisabletoextractthemajorcomponentsofscenewhileignoringsmallintracomponentvariations.Asdesired,recursivepartition-ingcanbeusedtofurtherdecomposeeachpiece.Fig.10showspreliminaryresultsontexturesegmenta-tionforanaturalimageofazebraagainstabackground.Notethatthemeasurewehaveusedisorientation-variantand,therefore,partsofthezebraskinwithdifferentstripeorientationshouldbemarkedasseparateregions.Inthemotioncase,wewilltreattheimagesequenceasaspatiotemporaldataset.Givenanimagesequence,aweightedgraphisconstructedbytakingeachpixelintheimagesequenceasanodeandconnectingpixelsthatareinthespatiotemporalneighborhoodofeachother.Theweightoneachgraphedgeisdefinedas:894IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 Fig.4.(a)showstheoriginalimageofsize.Imageintensityisnormalizedtoliewithin0and1.Subplots(b)-(h)showthecomponentsofthepartitionwithNcutvaluelessthan0.04.Parametersetting: wijˆe 0otherwisei;jistheªmotiondistanceºbetweentwopixels.Notethatinthiscaserepresentsthespatial-temporalpositionofpixelTocomputethisªmotiondistance,ºwewilluseamotionfeaturecalledmotionprofile.Bymotionprofileweseektoestimatetheprobabilitydistributionofimagevelocityateachpixel.Letdenoteaimagewindowcenteredatthepixelatlocationattime.Wedenotebymotionprofileofanimagepatchatnode,attimecorrespondingtoanotherimagepatchcanbeestimatedbyfirstcomputingthenormalizingittogetaprobabilitydistribution: Therearemanywaysonecancomputesimilaritybetweentwoimagepatches;wewilluseameasurethatisbasedonthesumofsquareddifferences(SSD):iswithinalocalneighborhoodofimage.Theªmotiondistanceºbetweentwoimagepixelsisthendefinedasoneminusthecross-correlationofthemotionprofiles:i;jInFig.11,weshowresultsofthenormalizedcutalgorithmonasyntheticrandomdotmotionsequenceandaindoormotionsequence,respectively.Formoreelaboratediscussiononmotionsegmentationusingnormalizedcut,aswellashowtosegmentandtrackoverlongimagesequences,readersmightwanttorefertoourpaper[21].4.1ComputationTimeAswesawfromSection3.1,therunningtimeofthenormalizedcutalgorithmis,whereisthenumber SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.5.(a)PointsetgeneratedbytwoPoissonprocesses,withdensitiesof2.5and1.0ontheleftandrightclustersrespectively,(b)thepartitionofpointsetin(a).Parametersettings: Fig.6.(a)Asyntheticimageshowinganoisyªstepºimage.Intensityvariesfrom0to1,andGaussiannoisewithisadded.Subplot(b)showstheeigenvectorwiththesecondsmallesteigenvalueandsubplot(c)showstheresultingpartition. Fig.7.(a)Asyntheticimageshowingthreeimagepatchesformingajunction.Imageintensityvariesfrom0to1andGaussiannoisewithadded.(b)-(d)showthetopthreecomponentsofthepartition. ofpixelsandisthenumberofstepsLanczostakestoconverge.Onthetestimagesshownhere,thenormalizedcutalgorithmtakesabout2minutesonIntelPentium200MHzmachines.Amultiresolutionimplementationcanbeusedtoreducethisrunningtimefurtheronlargerimages.Inourcurrentexperiments,withthisimplementation,therunningtimeonimagecanbereducedtoabout20secondsonIntelPentium300MHzmachines.Furthermore,thebottle-neckofthecomputation,asparsematrix-vectormultiplicationstep,canbeeasilyparallelizedtakingadvantageoffuturecomputerchipdesigns.Inourcurrentimplementation,thesparseeigenvaluedecompositioniscomputedusingtheLASO2numericalpackagedevelopedbyScott.4.2ChoiceofGraphEdgeWeightIntheexamplesshownhere,weusedanexponentialfunctionoftheformofontheweightedgraphedgewithfeaturesimilarityof.Thevalueof896IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 Fig.8.(a)showsaweatherradarimage.(b)-(g)showthecomponentsofthepartitionwithNcutvaluelessthan0.08.Parametersetting: Fig.9.(a)showsacolorimage.(b)-(e)showthecomponentsofthepartitionwithNcutvaluelessthan0.04.Parametersettings: Fig.10.(a)showsanimageofazebra.Theremainingimagesshowthemajorcomponentsofthepartition.ThetexturefeaturesusedcorrespondtoconvolutionswithDOOGfilters[16]atsixorientationsandfivescales. typicallysetto10to20percentofthetotalrangeofthefeaturedistancefunction.Theexponentialweightingfunctionischosenhereforitsrelativesimplicity,aswellasneutrality,sincethefocusofthispaperisondevelopingageneralsegmentationprocedure,givenafeaturesimilaritymeasure.Wefoundthischoiceofweightfunctionisquiteadequatefortypicalimageandfeaturespaces.Section6.1showstheeffectofusingdifferentweightingfunctionsandparametersontheoutputofthenormalizedcutalgorithm.However,thegeneralproblemofdefiningfeaturesimilarityincorporatingavarietyofcuesisnotatrivialone.Thegroupingcuescouldbeofdifferentabstractionlevelsandtypesandtheycouldbeinconflictwitheachother.Furthermore,theweightingfunctioncouldvaryfromimageregiontoimageregion,particularlyinatexturedimage.Someoftheseissuesareaddressedin[15].ELATIONSHIPTOThecomputationalapproachthatwehavedevelopedforimagesegmentationisbasedonconceptsfromspectralgraphtheory.Thecoreideaistousematrixtheoryandlinearalgebratostudypropertiesoftheincidencematrix,,andtheLaplacianmatrix,,ofthegraphandrelatethembacktovariouspropertiesoftheoriginalgraph.ThisisarichareaofmathematicsandtheideaofusingeigenvectorsoftheLaplacianforfindingpartitionsofgraphscanbetracedbacktoCheeger[4],DonathandHoffman[7],andFiedler[9].Thisareahasalsoseencontributionsbytheoreticalcomputerscientists[1],[3],[22],[23].Itcanbeshownthatournotionofisrelatedbyaconstantfactortotheconceptofconductancein[22].Foratutorialintroductiontospectralgraphtheory,werecommendtherecentmonographbyChung[5].Inthismonograph,ChungproposesaªnormalizedºdefinitionoftheLaplacian,as 12…DÿW†Dÿ .TheeigenvectorsforthisªnormalizedºLaplacian,whenmultipliedby ,areexactlythegeneralizedeigenvectorsweusedtocomputenormalizedcut.ChungpointsoutthattheeigenvaluesofthisªnormalizedºLaplacianrelatewelltographinvariantsforgeneralgraphinwaysthateigenvaluesofthestandardLaplacianhavefailedtodo.SpectralgraphtheoryprovidesussomeguidanceonthegoodnessoftheapproximationtothenormalizedcutprovidedbythesecondeigenvalueofthenormalizedLaplacian.OnewayisthroughboundsonthenormalizedCheegerconstant[5]which,inourterminology,canbedefinedas A;BA;V;assocB;VTheeigenvaluesof(6)arerelatedtotheCheegerconstantbytheinequality[5]: EarlierworkonspectralpartitioningusedthesecondeigenvectorsoftheLaplacianofthegraphdefinedastopartitionagraph.ThesecondsmallesteigenvalueofissometimesknownastheFiedlervalue.SeveralresultshavebeenderivedrelatingtheratiocutandtheFiedlervalue.AratiocutofapartitionofA;Vwhich,infact,isthestandarddefinitionoftheCheegerconstant,isdefinedas .ItwasshownthatiftheFiedlervalueissmall,partitioningthegraphbasedontheFiedlervectorwillleadtogoodratiocut[1],[23].OurderivationinSection2.1canbeadapted(byreplacingtheinthedenominatorsbytheidentitymatrix)toshowthattheFiedlervectorisarealvaluedsolutiontotheproblemof cut…A;VÿA†jAj‡ ,whichwecancallaveragecutaveragecutlookssimilartothenormalizedcutaveragecutdoesnothavetheimportantpropertyofhavinga SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.11.Subimages(a)and(b)showtwoframesofanimagesequence.Segmentationresultsonthistwoframeimagesequenceareshowninsubimages(c)to(g).Segmentsin(c)and(d)correspondtothepersonintheforegroundandsegmentsin(e)to(g)correspondtothebackground.Thereasonthattheheadofthepersonissegmentedawayfromthebodyisthat,althoughtheyhavesimilarmotion,theirmotionprofilesaredifferent.Theheadregioncontains2Dtexturesandthemotionprofilesaremorepeaked,while,inthebodyregion,themotionprofilesaremorespreadout.Segment(e)isbrokenawayfrom(f)and(g)forthesamereason. simplerelationshiptotheaverageassociation,whichcanbeanalogouslydefinedas …A;A†jAj‡ .Conse-quently,onecannotsimultaneouslyminimizethedisasso-ciationacrossthepartitionswhilemaximizingtheassociationwithinthegroups.Whenweappliedbothtechniquestotheimagesegmentationproblem,wefoundthatthenormalizedcutproducesbetterresultsinpractice.Therearealsootherexplanationswhythenormalizedcutbetterbehaviorfromgraphtheoreticalpointofview,aspointedoutbyChung[5].Ourwork,originallypresentedin[20],representsthefirstapplicationofspectralpartitioningtocomputervisionorimageanalysis.Thereis,however,oneapplicationareathathasseensubstantialapplicationofspectralpartitio-ningÐtheareaofparallelscientificcomputing.Thepro-blemthereistobalancetheworkloadovermultipleprocessorstakingintoaccountcommunicationneeds.Oneoftheearlypapersis[18].Thegeneralizedeigenvalueapproachwasfirstappliedtographpartitioningby[8]fordynamicallybalancingcomputationalloadinaparallelcomputer.Theiralgorithmismotivatedby[13]'spaperonrepresentingahypergraphinaEuclideanspace.ThenormalizedcutcriteriaisalsocloselyrelatedtokeypropertiesofaMarkovRandomWalk.Thesimilaritymatrixcanbenormalizedtodefineaprobabilitytransitionofarandomwalkonthepixels.Itcanbeshownthattheconductance[22]ofthisrandomwalkisthenormalizedcutvalueandthenormalizedcutvectorsof(12)areexactlytherighteigenvectorsof5.1APhysicalInterpretationAsonemightexpect,aphysicalanalogycanbesetupforthegeneralizedeigenvaluesystem(6)thatweusedtoapproximatethesolutionofnormalizedcut.Wecanconstructaspring-masssystemfromtheweightedgraphbytakinggraphnodesasphysicalnodesandgraphedgesasspringsconnectingeachpairofnodes.Furthermore,wewilldefinethegraphedgeweightasthespringstiffnessandthetotaledgeweightsconnectingtoanodeasitsmass.Imaginewhatwouldhappenifweweretogiveahardshaketothisspring-masssystem,forcingthenodestooscillateinthedirectionperpendiculartotheimageplane.Nodesthathavestrongerspringconnectionsamongthemwilllikelyoscillatetogether.Astheshakingbecomesmoreviolent,weakerspringsconnectingtothisgroupofnodewillbeoverstretched.Eventually,thegroupwillªpopºofffromtheimageplane.Theoverallsteadystatebehaviorofthenodescanbedescribedbyitsfundamentalmodeofoscillation.Infact,itcanbeshownthatthefundamentalmodesofoscillationofthisspringmasssystemareexactlythegeneralizedeigenvectorsof(6).bethespringstiffnessconnectingnodestobethestiffnessmatrix,withi;ii;j.Definethediagonali;i.Letbethedescribingthemotionofeachnode.Thisspring-massdynamicsystemcanbedescribedby:Assumingthesolutiontakestheformof,thesteadystatesolutionsofthisspring-masssystemsatisfy:analogousto(6)fornormalizedcutEachsolutionpairof(21)describesaofthespring-masssystem.Theeigenvectorsthesteadystatedisplacementoftheoscillationineachmodeandtheeigenvaluesgivetheenergyrequiredtosustaineachmodeofoscillation.Therefore,findinggraphpartitionsthathavesmallnormalizedcutvaluesis,ineffect,thesameasfindingawaytoªpopºoffimageregionswithminimaleffort.ELATIONSHIPTOPPROACHESTOInthecomputervisioncommunity,therehasbeensomebeenpreviousworkonimagesegmentationformulatedasagraphpartitionproblem.WuandLeahy[25]usetheminimumcutcriterionfortheirsegmentation.Asmentionedearlier,ourcriticismofthiscriterionisthatittendstofavorcuttingoffsmallregions,whichisundesirableinthecontextofimagesegmentation.Inanattempttogetmorebalancedpartitions,Coxetal.[6]seektominimizetheratio ,whereissomefunctionofthe.Whenistakentobethesumoftheelementsin,weseethatthiscriterionbecomesoneofthetermsinthedefinitionofaveragecutabove.Coxetal.useanefficientdiscretealgorithmtosolvetheiroptimizationproblemassumingthegraphisplanar.SarkarandBoyer[19]usetheeigenvectorwiththelargesteigenvalueofthesystemforfindingthemostcoherentregioninanedgemap.UsingasimilarderivationasinSection2.1,wecanseethatthefirstlargesteigenvectoroftheirsystemapproximates andthesecondlargesteigenvectorapproximatesV;B assoc…A;A†jAj‡ .However,theapproxima-tionisnottightandthereisnoguaranteethatAswewillseelaterinthesection,thissituationcanhappenquiteofteninpractice.Sincethisalgorithmisessentiallylookingforclustersthathavetightwithin-groupingsimilarity,wewillcallthiscriteriaaverageassociation6.1ComparisonwithRelatedEigenvector-BasedMethodsnormalizedcutformulationhasacertainresemblancetoaveragecut,thestandardspectralgraphpartitioning,aswellasaverageassociationformulation.Allthreeofthesealgorithmscanbereducedtosolvingcertaineigenvaluesystems.Howaretheyrelatedtoeachother?Fig.12summarizestherelationshipbetweenthesethreealgorithms.Ononehand,boththenormalizedcutandtheaveragecutalgorithmaretryingtofindaªbalancedpartitionºofaweightedgraph,while,ontheotherhand,898IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 normalizedassociationandtheaverageassociationtryingtofindªtightºclustersinthegraph.Sincethenormalizedassociationisexactly,thenormalizedcutvalue,thenormalizedcutformulationseeksabalancebetweenthegoalofclusteringandsegmentation.Itis,therefore,nottoosurprisingtoseethatthenormalizedcutvectorcanbeapproximatedwiththegeneralizedeigenvec-torof,aswellasthatofJudgingfromthediscreteformulationsofthesethreegroupingcriteria,itcanbeseenthattheaverageassociation assoc…A;A†jAj‡ ,hasabiasforfindingtightclusters.Therefore,itrunstheriskofbecomingtoogreedyinfindingsmall,buttight,clustersinthedata.ThismightbeperfectfordatathatareGaussiandistributed.However,fortypicaldataintherealworldthataremorelikelytobemadeupofamixtureofvariousdifferenttypesofdistributions,thisbiasingroupingwillhaveundesiredconsequences,asweshallillustrateintheexamplesbelow.averagecut cut…A;B†jAj‡ ,theoppositeproblemarisesÐonecannotensurethetwopartitionscomputedwillhavetightwithin-groupsimilarity.Thisbecomesparticu-larlyproblematicifthedissimilarityamongthedifferentgroupsvariesfromonetoanother,orifthereareseveralpossiblepartitionsallwithsimilaraveragecutToillustratethesepoints,letusfirstconsiderasetofrandomlydistributeddatain1DshowninFig.13.The1Ddataismadeupbytwosubsetsofpoints,onerandomlydistributedfrom0to0.5andtheotherfrom0.65to1.0.Eachdatapointistakenasanodeinthegraphandtheweightedgraphedgeconnectingtwopointsisdefinedtobeinverselyproportionaltothedistancebetweentwonodes.Wewillusethreemonotonicallydecreasingweightingfunctions,,definedonthedistancefunction,,withdifferentrateoffall-off.ThethreeweightingfunctionsareplottedinFigs.14a,15a,and16a.Thefirstfunction, ,plottedinFig.14a,hasthefastestdecreasingrateamongthethree.Withthisweightingfunction,onlyclose-bypointsareconnected,asshowninthegraphweightmatrixplottedinFig.14b.Inthiscase,averageassociationfailstofindtherightpartition.Instead,itfocusesonfindingsmallclustersineachofthetwomainsubgroups.Thesecondfunction,,plottedinFig.15a,hastheslowestdecreasingrateamongthethree.Withthisweightingfunction,mostpointshavesomenontrivialconnectionstotherest.Tofindacutofthegraph,anumberofedgeswithheavyweightshavetoberemoved.Inaddition,theclusterontherighthaslesswithin-groupsimilaritycomparingwiththeclusterontheleft.Inthisaveragecuthastroubledecidingonwheretocut.Thethirdfunction, ,plottedinFig.16a,hasamoderatedecreasingrate.Withthisweightingfunction,thenearbypointconnectionsarebalancedagainstfar-awaypointconnections.Inthiscase,allthreealgorithmsperformwellwithnormalizedcut,producingaclearersolutionthanthetwoothermethods.Theseproblems,illustratedinFigs.14,15,and16,infactarequitetypicalinsegmentingrealnaturalimages.Thisisparticularlytrueinthecaseoftexturesegmentation.Differenttextureregionsoftenhaveverydifferentwithin-groupsimilarityorcoherence.Itisverydifficulttopredeterminetherightweightingfunctiononeachimageregion.Therefore,itisimportanttodesignagroupingalgorithmthatismoretoleranttoawiderangeofweightingfunctions.Theadvantageofusingnormalizedcutmoreevidentinthiscase.Fig.17illustratesthispointonanaturaltextureimageshownpreviouslyinFig.10. SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.12.Relationshipbetweennormalizedcutandothereigenvector-basedpartitioningtechniques.Comparedtotheaveragecutformulation,normalizedcutseeksabalancebetweenthegoaloffindingclumpsandfindingsplits. Fig.13.Asetofrandomlydistributedpointsin1D.Thefirst20pointsarerandomlydistributedfrom0.0to0.5andtheremaining12pointsarerandomlydistributedfrom0.65to1.0.SegmentationresultofthesepointswithdifferentweightingfunctionsareshowninFigs.14,15,and Inthispaper,wedevelopedagroupingalgorithmbasedontheviewthatperceptualgroupingshouldbeaprocessthataimstoextractglobalimpressionsofasceneandprovidesahierarchicaldescriptionofit.Bytreatingthegroupingproblemasagraphpartitioningproblem,weproposedthenormalizedcutcriteriaforsegmentingthegraph.Normal-izedcutisanunbiasedmeasureofdisassociationbetweensubgroupsofagraphandithasthenicepropertythatminimizingnormalizedcutleadsdirectlytomaximizingthenormalizedassociation,whichisanunbiasedmeasurefortotalassociationwithinthesubgroups.Infindinganefficientalgorithmforcomputingtheminimumnormalizedcut,weshowedthatageneralizedeigenvaluesystemprovidesarealvaluedsolutiontoourproblem.Acomputationalmethodbasedonthisideahasbeendevelopedandappliedtosegmentationofbrightness,color,andtextureimages.Resultsofexperimentsonrealandsyntheticimagesareveryencouragingandillustratethatthenormalizedcutcriteriondoesindeedsatisfyourinitialgoalofextractingtheªbigpictureºofascene.900IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 Fig.14.Aweightingfunctionwithfastrateoffall-off: ,showninsubplot(a)insolidline.ThedottedlinesshowthetwoalternativeweightingfunctionsusedinFigs.15and16.Subplot(b)showsthecorrespondinggraphweightmatrix.Thetwocolumns(c)and(d)belowshowthefirst,andsecondextremeeigenvectorsfortheNormalizedcut(row1),Averagecut(row2),andAverageassociation(row3).Forbothnormalizedcutaveragecut,thesmallesteigenvectorisaconstantvectoraspredicted.Inthiscase,bothnormalizedcutaveragecutperformwell,whiletheaverageassociationfailstodotherightthing.Instead,ittriestopickoutisolatedsmallclusters. NP-CROOFFORProposition1[Papadimitrou97].NormalizedCut(NCUT)foragraphonregulargridsisNP-complete.Proof.WeshallreduceNCUTonregulargridsfromGivenintegersaddingto,isthereasubsetaddingtoWeconstructaweightedgraphonaregulargridthathasthepropertythatitwillhaveasmallenoughnormalizedcutifandonlyifwecanfindasubsetfromaddingto.Fig.18ashowsthegraphandFig.18bshowstheformthatapartitionthatminimizesthenormalizedcutmusttake.Incomparisontotheintegersismuch,andismuchsmaller,.Weaskthequestion SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.15.Aweightingfunctionwithslowrateoffall-off:,showninsubplot(a)insolidline.ThedottedlinesshowthetwoalternativeweightingfunctionsusedinFigs.14and16.Subplot(b)showsthecorrespondinggraphweightmatrix.Thetwocolumns(c)and(d)belowshowthefirst,andsecondextremeeigenvectorsfortheNormalizedcut(row1),Averagecut(row2),andAverageassociation(row3).Inthiscase,bothnormalizedcutaverageassociationgivetherightpartition,whiletheaveragecuthastroubledecidingonwheretocut. IsthereapartitionwithNcutvaluelessthan ishalfthesumofedgeweightsintheWeshallseethatagoodNcutpartitionofthegraphmustseparatetheleftandrightcolumns.Inparticular,ifandonlyifthereisasubsetaddingto,bytakingthecorrespondingedgesinthemiddlecolumntobeinonesideofthepartition,asillustratedinFig.18b,weachieveanNcutvaluelessthan .Forallotherpartitions,theNcutvaluewillbeboundedbelowby First,letusshowthatthecutillustratedinFig.18b,whereeachsidehasasubsetofmiddlecolumnedgesthataddupto,doeshaveNcutvalueless .LetthebetheNcutvalueforthiscut.ByusingtheformulaforNcut(2.2),wecanseethat902IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 Fig.16.Aweightingfunctionwithmediumrateoffall-off: ,showninsubplot(a)insolidline.ThedottedlinesshowthetwoalternativeweightingfunctionsusedinFigs.14and15.Subplot(b)showsthecorrespondinggraphweightmatrix.Thetwocolumns(c)and(d)belowshowthefirstandsecondextremeeigenvectorsfortheNormalizedcut(row1),Averagecut(row2),andaverageassociation(row3).Allthreeofthesealgorithmsperformsatisfactorilyinthiscase,withnormalizedcutproducingaclearersolutionthantheothertwocuts. ˆ 4an2c‡22k1ÿ1†‡ ishalfthetotaledgeweightsinthegraph,,andarethenumberofedgesfromthemiddlecolumnonthetwosidesofthegraphpartition,.ThetermcanbeinterpretedastheamountofimbalancebetweenthedenominatorsinthetwotermsintheNcutformulaandliesbetweenn).Simplifying,weseethat 422k1ÿ1††2 42ÿ1ˆ aswastobeshown.TocompletetheproofwemustshowthatallotherpartitionsresultinaNcutgreaterthanorequalto Informallyspeaking,whatwillhappenisthateitherthenumeratorsofthetermsintheNcutbecometoolarge,orthedenominatorsbecomesignifi-cantlyimbalanced,againincreasingtheNcutvalue.Weneedtoconsiderthreecases:Acutthatdeviatesfromthecutin1(b)slightlybyreshufflingsomeoftheedgessothatthesumsoftheineachsubsetofthegraphpartitionarenolongerequal.Forsuchcuts,theresultingNcutvaluesare,atbest, 2‡x‡ 2ÿxˆ anc.But,,wehave 42ÿ1ˆ Acutthatgoesthroughanyoftheedgeswith.Evenwiththedenominatorsonbothsidescompletelybalanced,theNcutvalue isgoingtobelargerthan .Thisisensuredbyourchoiceintheconstructionthat.Wehavetoshowthat 2Mc 4ÿ1=c;orM2 Thisisdirect,sincebyconstruction, c2c2ÿ1 Acutthatpartitionsoutsomeofthenodesinthemiddleasonegroup.WeseethatanycutthatgoesthroughoneofthescanimproveitsNcutvaluebygoingthroughtheedgeswithweightinstead.So,wewillfocusonthecasewherethecutonlygoesthroughtheweightedges.Supposethatedgesofsaregroupedintooneset,withtotalweightaddingto,where.Thecorrespondingncutvalue, 2x‡ 4M…n‡1k‡1242xˆ 2ÿdm‡ 2‡dm;where SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION Fig.17.NormalizedcutandaverageassociationresultonthezebraimageinFig.10.Subplot(a)showsthesecondlargesteigenvectorof,approximatingthenormalizedcutvector.Subplots(b)-(e)showthefirsttofourthlargesteigenvectorsof,approximatingtheaverageassociationvector,usingthesamegraphweightmatrix.Inthisimage,pixelsonthezebrabodyhave,onaverage,lowerdegreeofcoherencethanthepixelsinthebackground.Theaverageassociation,withitstendencytofindtightclusters,partitionsoutonlysmallclustersinbackground.Thenormalizedcutalgorithm,havingtobalancethegoalofclusteringandsegmentation,findsthebetterpartitioninthiscase. Thelowerboundon isthen .Furtherexpansionofthe 2ÿx2lˆ 42ÿ22;whereBˆ2M…n‡1‡3 4 Onecanchecktoseethatisanon-decreasingfunctionandhasitsminimumat Inordertoprovethat ,weneedtoestablishtheinequality 4c2ÿB24a2 42ÿ1or 1…c2ÿB24a2 1orusingthefactthat.Tocontinue,notethat,sincean,thiswillbetrueiforif.Since,weonlyneedtoshowthatorthat.ThisissobecauseThisresearchwassupportedby(ARO)DAAH04-96-1-0341,andaNationalScienceFoundationGraduateFellowshiptoJ.Shi.WethankChristosPapadimitriouforsupplyingtheproofofNP-completenessfornormalizedcutsonagrid.Inaddition,wewishtoacknowledgeUmeshVaziraniandAlistairSinclairfordiscussionsongraphtheoreticalgo-rithmsandInderjitDhillonandMarkAdamsforusefulpointerstonumericalpackages.ThomasLeung,SergeBelongie,YeirWeiss,andothermembersofthecomputervisiongroupatUCBerkeleyprovidedmuchusefulfeed-backonouralgorithm.[1]N.Alon,ªEigenvaluesandExpanders,ºCombinatorica,vol.6,no.2,pp.83-96,1986.1986.A.BlakeandA.Zisserman,VisualReconstruction.MITPress,1987.1987.R.B.Boppana,ªEigenvaluesandGraphBisection:AnAverage-CaseAnalysis,ºProc.28thSymp.FoundationsofComputerScience,pp.280-285,1987.1987.J.Cheeger,ªALowerBoundfortheSmallestEigenvalueoftheLaplacian,ºProblemsinAnalysis,R.C.Gunning,ed.,pp.195-199,PrincetonUniv.Press,1970.1970.F.R.K.Chung,SpectralGraphTheory.Am.Math.Soc.,1997.1997.I.J.Cox,S.B.Rao,andY.Zhong,ªRatioRegions:ATechniqueforImageSegmentation,ºProc.13thInt'lConf.PatternRecognition,904IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.22,NO.8,AUGUST2000 Fig.18.(a)showsaweightedgraphonaregulargrid.Themissingedgesonthegridshaveweightsof0.Incomparisontotheintegersisalargenumber(),andisverysmallnumber().(b)showsacutthathasaNcutvalueless .Thiscut,whichonlygoesthroughedgeswithweightequalto,hasthepropertythatthesoneachsideofthepartitionsumupto W.E.DonathandA.J.Hoffman,ªLowerBoundsforthePartition-ingofGraphs,ºIBMJ.ResearchandDevelopment,pp.420-425,1973.1973.R.VanDriesscheandD.Roose,ªAnImprovedSpectralBisectionAlgorithmandItsApplicationtoDynamicLoadBalancing,ºParallelComputing,vol.21,pp.29-48,1995.1995.M.Fiedler,ªAPropertyofEigenvectorsofNonnegativeSym-metricMatricesandItsApplicationstoGraphTheory,ºMath.J.,vol.25,no.100,pp.619-633,1975.1975.S.GemanandD.Geman,ªStochasticRelaxation,GibbsDistribu-tions,andtheBayesianRestorationofImages,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.6,pp.721-741,Nov.1984.1984.G.H.GolubandC.F.VanLoan,MatrixComputations.JohnHopkinsPress,1989.1989.A.K.JainandR.C.Dubes,AlgorithmsforClusteringData.Hall,1988.1988.K.Fukunaga,S.Yamada,H.S.Stone,andT.Kasai,ªARepresenta-tionofHypergraphsintheEuclideanSpace,ºIEEETrans.vol.33,no.4,pp.364-367,Apr.1984.1984.Y.G.Leclerc,ªConstructingSimpleStableDescriptionsforImagePartitioning,ºInt'lJ.ComputerVision,vol.3,pp.73-102,1989.1989.J.Malik,S.Belongie,J.Shi,andT.Leung,ªTextons,ContoursandRegions:CueIntegrationinImageSegmentation,ºProc.Int'lConf.ComputerVision,pp.918-925,1999.1999.J.MalikandP.Perona,ªPreattentiveTextureDiscriminationwithEarlyVisionMechanisms,ºJ.OpticalSoc.Am.,vol.7,no.2,pp.923-932,May1990.1990.D.MumfordandJ.Shah,ªOptimalApproximationsbyPiecewiseSmoothFunctions,andAssociatedVariationalProblems,ºPureMath.,pp.577-684,1989.1989.A.Pothen,H.D.Simon,andK.P.Liou,ªPartitioningSparseMatriceswithEigenvectorsofGraphs,ºSIAMJ.MatrixAnalyticalApplications,vol.11,pp.430-452,1990.1990.S.SarkarandK.L.Boyer,ªQuantitativeMeasuresofChangeBasedonFeatureOrganization:EigenvaluesandEigenvectors,ºProc.IEEEConf.ComputerVisionandPatternRecognition,,J.ShiandJ.Malik,ªNormalizedCutsandImageSegmentation,ºProc.IEEEConf.ComputerVisionandPatternRecognition,pp.731-737,1997.1997.J.ShiandJ.Malik,ªMotionSegmentationandTrackingUsingNormalizedCuts,ºProc.Int'lConf.ComputerVision,pp.1,154-1,160,1998.1998.A.J.SinclairandM.R.Jerrum,ªApproximativeCounting,UniformGenerationandRapidlyMixingMarkovChains,ºInformationandComputation,vol.82,pp.93-133,1989.1989.D.A.SpielmanandS.H.Teng,ªDiskPackingsandPlanarSeparators,ºProc.12thACMSymp.ComputationalGeometry,ry,M.Wertheimer,ªLawsofOrganizationinPerceptualForms(partialtranslation),ºASourcebookofGestaltPsycychology,Ellis,ed.,pp.71-88,Harcourt,Brace,1938.1938.Z.WuandR.Leahy,ªAnOptimalGraphTheoreticApproachtoDataClustering:TheoryandItsApplicationtoImageSegmenta-IEEETrans.PatternAnalysisandMachineIntelligence,vol.15,no.11,pp.1,101-1,113,Nov.1993.JianboShistudiedcomputerscienceandmathematicsasanundergraduateatCornellUniversitywhereherecievedhisBAdegreein1994.HereceivedhisPhDdegreeincomputersciencefromtheUniversityofCaliforniaatBerkeleyin1998.Since1999,hehasbeenamemberofthefacultyoftheRoboticsInstituteatCarnegieMellonUniversity,wherehisprimaryresearchinterestsincludeimagesegmentation,grouping,objectrecognition,motionandshapeanalysis,andmachinelearning.JitendraMalikreceivedtheBTechdegreeinelectricalengineeringfromtheIndianInstituteofTechnology,Kanpur,in1980andthePhDdegreeincomputersciencefromStanfordUniversityin1986.InJanuary1986,hejoinedthefacultyoftheComputerScienceDivision,DepartmentofElectricalEngineeringandCom-puterScienceattheUniversityofCaliforniaatBerkeley,whereheiscurrentlyaprofessor.During1995-1998,healsoservedasvice-chairforgraduatematters.HeisamemberoftheCognitiveScienceandVisionSciencegroupsatUCBerkeley.Hisresearchinterestsareincomputervisionandcomputationalmodelingofhumanvision.Hisworkspansarangeoftopicsinvisionincludingimagesegmentationandgrouping,texture,image-basedmodelingandrendering,content-basedimagequerying,andintelligentvehiclehighwaysystems.Hehasauthoredorcoauthoredmorethan80researchpapersonthesetopics.HereceivedthegoldmedalforthebestgraduatingstudentinElectricalEngineeringfromIITKanpurin1980,aPresidentialYoungInvestigatorAwardin1989,andtheRosenbaumfellowshipfortheComputerVisionProgrammeattheNewtonInstituteofMathema-ticalSciences,UniversityofCambridge,in1993.Heisaneditor-in-chiefoftheInternationalJournalofComputerVision.HeisamemberoftheIEEE. SHIANDMALIK:NORMALIZEDCUTSANDIMAGESEGMENTATION