Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov Member IEEE Olga Veksler - PDF document

FastApproximateEnergyMinimizationviaGraphCutsYuriBoykov,,OlgaVeksler,,andRaminZabih,ÐManytasksincomputervisioninvolveassigningalabel(suchasdisparity)toeverypixel.Acommonconstraintisthatthelabelsshouldvarysmoothlyalmosteverywherewhilepreservingsharpdiscontinuitiesthatmayexist,e.g.,atobjectboundaries.Thesetasksarenaturallystatedintermsofenergyminimization.Inthispaper,weconsiderawideclassofenergieswithvarioussmoothnessconstraints.GlobalminimizationoftheseenergyfunctionsisNP-hardeveninthesimplestdiscontinuity-preservingcase.Therefore,ourfocusisonefficientapproximationalgorithms.Wepresenttwoalgorithmsbasedongraphcutsthatefficientlyfindalocalminimumwithrespecttotwotypesoflargemoves,namelymovesand æ INIMIZATIONIN . Wedevelopalgorithmsthatapproximatelyminimizetheforanarbitraryfinitesetoflabelsundertwofairlygeneralclassesofinteractionpenalty:metricandiscalledaonthespaceoflabelsifit;;;;; ;foranylabels;; .Ifsatisfiesonly(2)and(3),itiscalledaNotethatbothsemimetricsandmetricsincludeimpor-tantcasesofdiscontinuitypreservinginteractionpenalties.Informally,adiscontinuitypreservinginteractiontermshouldhaveaboundonthelargestpossiblepenalty.Thisavoidsoverpenalizingsharpjumpsbetweenthelabelsofneighboringpixels;see[46],[30],andourexperimentalresultsinSection8.6.Examplesofdiscontinuitypreservinginteractionpenaltiesforaone-dimensionallabelsetincludethetruncatedquadratic;(asemimetric)andthetruncatedabsolutedistance;(ametric),whereissomeconstant.Ifismultidimensional,wecanreplacej
janynorm,e.g.,jj
jj.Thesemodelsencouragelabelingsconsistingofseveralregionswherepixelsinthesameregionhavesimilarlabelsand,therefore,weinformallycallthempiecewisesmoothmodels.AnotherimportantdiscontinuitypreservingfunctionisgivenbythePottsmodel;(ametric),is1ifitsargumentistrue,andotherwise0.Thismodelencourageslabelingsconsistingofseveralregionswherepixelsinthesameregionhaveequallabelsand,therefore,weinformallycallitapiecewiseconstantmodel.Webeginwithareviewofpreviousworkonenergyminimizationinearlyvision.InSection3,wegiveanoverviewofourenergyminimizationalgorithms.Ourfirstalgorithm,describedinSection4,isbasedonmovesandworksforanysemimetric.Oursecondalgorithm,describedinSection4,isbasedonthemoremovesbutrequirestobeametric.OptimalitypropertiesofouralgorithmsarediscussedinSection6.Forexample,weshowthatourexpansionalgorithmproducesasolutionwithinaknownfactoroftheglobalminimumof.InSection7,wedescribeanimportantspecialcaseofourenergywhicharisesfromthePottsinteractionpenalty.Thisisaverysimpletypeofdiscontinuitypreservingsmoothnesspenalty,yetweprovethatcomputingtheglobalminimumisNP-hard.Experi-mentaldataispresentedinSection8.Theenergyfunctionsthatweareinterestedin,givenin(1),arisequitenaturallyinearlyvision.Energy-basedmethodsattempttomodelsomeglobalimagepropertiesthatcannotbecaptured,forexample,bylocalcorrelationtechniques.Themainproblem,however,isthatinterestingenergiesareoftendifficulttominimize.WeshowintheAppendixthatoneofthesimplestdiscontinuitypreservingcasesofourenergyfunctionminimizationisNP-hard;therefore,itisimpossibletorapidlycomputetheglobalminimumunlessP=NP.Duetotheinefficiencyofcomputingtheglobalminimum,manyauthorshaveoptedforalocalminimum.However,ingeneral,alocalminimumcanbearbitrarilyfarfromtheoptimum.Thus,itmaynotconveyanyoftheglobalimagepropertiesthatwereencodedintheenergyfunction.Insuchcases,itisdifficulttodeterminethecauseofanalgorithm'sfailures.Whenanalgorithmgivesunsatisfactoryresults,itmaybedueeithertoapoorchoiceoftheenergyfunction,ortothefactthattheanswerisfarfromtheglobalminimum.Thereisnoobviouswaytotellwhichoftheseistheproblem.Anothercommonissueisthatlocalminimizationtechniquesarenaturallysensitivetotheinitialestimate.Ingeneral,alabelingisalocalminimumoftheenergyforany}nearto}Inthecaseofdiscretelabeling,thelabelingsneartoarethosethatliewithinasingle.Manylocaloptimizationtechniquesusewhatwewillcallmoves,whereonlyonepixelcanchangeitslabelatatime(seeFig.2b).Forstandardmoves,(5)canbereadasfollows:Ifyouareatalocalminimum,withrespecttostandardmoves,thenyoucannotdecreasetheenergybychangingasinglepixel'slabel.Infact,thisisaveryweakcondition.Asaresult,optimizationschemesusingstandardmovesfrequentlygeneratelowqualitysolutions.Forinstance,considerthelocalminimumwithrespecttostandardmovesshowninFig.1c.AnexampleofalocalmethodusingstandardmovesisIteratedConditionalModes(ICM),whichisagreedytechniqueintroducedin[4].Foreachpixel,thelabelwhichgivesthelargestdecreaseoftheenergyfunctionischosen,untilconvergencetoalocalminimum.Anotherexampleofanalgorithmusingstandardmovesissimulatedannealing,whichwaspopularizedincomputervisionby[19].Annealingispopularbecauseitiseasytoimplementanditcanoptimizeanarbitraryenergyfunction.Unfortunately,minimizinganarbitraryenergyfunctionrequiresexponentialtimeandasaconsequencesimulatedannealingisveryslow.Theoretically,simulatedannealingshouldeventuallyfindtheglobalminimumifrunforlongenough.Asapracticalmatter,itisnecessarytodecreasethealgorithm'stemperatureparameterfasterthanrequiredbythetheoreticallyoptimalschedule.Onceannealing'stem-peratureparameterissufficientlylow,thealgorithmwillconvergetoalocalminimumwithrespecttostandardmoves.Infact,[20]demonstratethatpracticalimplementa-tionsofsimulatedannealinggiveresultsthatareveryfarfromtheglobaloptimumevenintherelativelysimplecaseofbinarylabelings.Tryingtoimprovetherateofconvergenceofsimulatedannealing[39],[3]developedsamplingalgorithmsforthePottsmodelthatcanmakelargermovessimilartoour.Themaindifferenceisthatwefindthebestmoveamongallpossible,while[39],[3]randomlyselectconnectedsubsetsofpixelsthatchangetheirlabelfrom BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS 2.Infact,weonlyassume;;inordertosimplifythepresentation.Wecaneasilygeneralizeallresultsinthispapertoallow;†6ˆ;.Thisgeneralizationrequirestheuseofdirectedgraphs.3.Inspecialcaseswheretheglobalminimumcanberapidlycomputed,itispossibletoseparatetheseissues.Forexample,[20]pointsoutthattheglobalminimumofaspecialcaseofIsingenergyfunctionisnotnecessarilythedesiredsolutionforimagerestoration.Blake[9],andGreigetal.[20]analyzetheperformanceofsimulatedannealingincaseswithaknownglobalminimum. Likesimulatedannealing,thesealgorithmshaveonlyconvergenceªatinfinityºoptimalityproperties.Thequalityofthesolutionsthatthesealgorithmsproduce,inpractice,underrealisticcoolingschedulesisnotclear.Iftheenergyminimizationproblemisphrasedincontinuousterms,variationalmethodscanbeapplied.Thesemethodswerepopularizedby[22].VariationaltechniquesusetheEulerequations,whichareguaranteedtoholdatalocalminimum.Toapplythesealgorithmstoactualimagery,ofcourse,requiresdiscretization.Anotheralternativeistousediscreterelaxationlabelingmethods;thishasbeendonebymanyauthors,including[12],[36],[41].Inrelaxationlabeling,combinatorialoptimi-zationisconvertedintocontinuousoptimizationwithlinearconstraints.Then,someformofgradientdescentwhichgivesthesolutionsatisfyingtheconstraintsisused.Relaxationlabelingtechniquesareactuallymoregeneralthanenergyminimizationmethods,see[23]and[32].Therearealsomethodsthathaveoptimalityguaranteesincertaincases.Continuationmethods,suchasgraduatednonconvexity[8],areanexample.Thesemethodsinvolveapproximatinganintractable(nonconvex)energyfunctionbyasequenceofenergyfunctions,beginningwithatractable(convex)approximation.Therearecircumstanceswherethesemethodsareknowntocomputetheoptimalsolution(see[8]fordetails).Continuationmethodscanbeappliedtoalargenumberofenergyfunctions,butexceptforthesespecialcasesnothingisknownaboutthequalityoftheiroutput.Meanfieldannealingisanotherpopularminimizationapproach.Itisbasedonestimatingthepartitionfunctionfromwhichtheminimumoftheenergycanbededuced.However,computingthepartitionfunctioniscomputation-allyintractable,andsaddlepointapproximations[31]areused.Thereisalsoandinterestingconnectionbetweenmeanfieldapproximationandotherminimizationmethodslikegraduatednonconvexity[17].Thereareafewinterestingenergyfunctionswheretheglobalminimumcanberapidlycomputedviadynamicprogramming[2].However,dynamicprogrammingisrestrictedessentiallytoenergyfunctionsinone-dimen-sionalsettings.Thisincludessomeimportantcases,suchassnakes[26].Ingeneral,thetwo-dimensionalenergyfunctionsthatariseinearlyvisioncannotbesolvedefficientlyviadynamicprogramming.Graphcuttechniquesfromcombinatorialoptimizationcanbeusedtofindtheglobalminimumforsomemulti-dimensionalenergyfunctions.Whenthereareonlytwolabels,(1)isaspecialcaseofthemodel.Greigetal.[20]showedhowtofindtheglobalminimum,inthiscase,byasinglegraphcutcomputation.NotethatthePottsmodelwediscussinSection7isanaturalgeneralizationoftheIsingmodeltothecaseofmorethantwolabels.AmethodoptimaltowithinafactoroftwoforthePottsmodelwasdevelopedin[14];however,theirenergydatatermisveryrestrictive.Recently,[37],[24],[11]usedgraphcutstofindtheexactglobalminimumofacertaintypeofenergyfunctions.However,thesemethodsapplyonlyifthelabelsareone-dimensional.Mostimportantly,theyrequiretobeconvex[25]and,hence,theirenergiesarenotdiscontinuitypreserving,seeSection8.6.Notethatgraphcutshavealsobeenusedforsegmentationbasedonclustering[47],[16],[44].Unlikeclustering,weassumethatthereisanaturalsetoflabels(e.g.,intensitiesordisparities),andadatapenaltyfunctionwhichmakessomepixel-labelassignmentsmorelikelythanothers.Themaincontributionofthispaperaretwonewalgorithmsformultidimensionalenergyminimizationthatusegraphcutsiteratively.Wegeneralizethepreviousresultsbyallowingarbitrarylabelsets,arbitrarydatatermsandaverywideclassofpairwiseinteractionsthatincludesdiscontinuitypreservingcases.WeachieveapproximatesolutionstothisNP-hardminimizationpro-blemwithguaranteedoptimalitybounds.TheNP-hardnessresultgivenintheAppendixeffectivelyforcesustocomputeanapproximatesolution.Ourmethodsgeneratealocalminimumwithrespecttoverylargemoves.Weshowthatthisapproachovercomesmanyoftheproblemsassociatedwithlocalminima.Thealgorithmsintroducedinthissectiongeneratealabelingthatisalocalminimumoftheenergyin(1)withresepcttotwotypesoflargemoves:.IncontrasttothestandardmovesdescribedinSection2,thesemovesallowalargenumberofpixelstochangetheirlabelssimultaneously.Thismakesthesetof1224IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER20014.Notethatincontinuouscases,thelabelsneartoin(5)arenormallydefinedasjj,whereisapositiveconstantandjj
jjisanorm,,oversomeappropriatefunctionalspace.5.Throughoutthispaper,weinformallyusegraphcutstorefertothemin-cut/max-flowalgorithmsthatarestandardincombinatorialoptimiza-tion[1].SeeSection3.3formoredetailsongraphcuts. Fig.1.Comparisonoflocalminimawithrespecttostandardandlargemovesforimagerestoration.(a)Originalimage.(b)Observednoisyimage.(c)Localminimumwithrespecttostandardmoves.(d)Localminimumwithrespecttoexpansionmoves.Weusetheenergyfromequation(1)withquadraticdatatermspenalizingdeviationsfromtheobservedintensities(b).Thesmoothnesstermistruncatedmetric.Bothlocalminimain(c)and(d)wereobtainedusinglabeling(b)asaninitialsolution. labelingswithinasinglemoveofalocallyoptimalexponentiallylarge,andtheconditionin(5)verydemand-ing.Forexample,movesaresostrongthatweareabletoprovethatanylabelinglocallyoptimalwithrespecttothesemovesiswithinaknownfactoroftheglobalminimum(seeSection6).Fig.1compareslocalminimaforstandardmoves(Fig.1c)andformoves(Fig.1d)obtainedfromthesameinitialsolution(Fig.1b).Thisandotherexperimentsalsoshowthat,inpractice,oursolutionsdonotchangesignificantlybyvaryingtheinitiallabelings.Inmostcases,startingfromaconstantlabeling(whereallpixelshavethesamelabel)isgoodenough.InSection3.1,wediscussthemovesweallowwhicharebestdescribedintermsofpartitions.InSection3.2,wesketchthealgorithmsandlisttheirbasicproperties.Themaincomputationalstepofouralgorithmsisbasedongraphcuttechniquesfromcombinatorialoptimization,whichwesummarizeinSection3.3.3.1PartitionsandMoveSpacesAnylabelingcanbeuniquelyrepresentedbyapartitionofimagepixelsˆfP2Lg,where2Pjasubsetofpixelsassignedlabel.Sincethereisanobviousonetoonecorrespondencebetweenlabelings,wecanusethesenotionsinterchangingly.Givenapairoflabels;,amovefromapartition)toanewpartition)iscalledanforanylabel;.Thismeansthattheonlydifferencebetweenisthatsomepixelsthatwerelabeledarenowlabeled,andsomepixelsthatwerelabeledarenowlabeledspecialcaseofanisamovethatgivesthelabeltosomesetofpixelspreviouslylabeled.OneexampleofmoveisshowninFig.2c.Givenalabel,amovefromapartition)toanewpartition)iscalledanforanylabel.Inotherwords,anmoveallowsanysetofimagepixelstochangetheirlabelsto.AnexampleofanmoveisshowninFig.2d.RecallthatICMandannealinguseallowingonlyonepixeltochangeitsintensity.AnexampleofastandardmoveisgiveninFig.2b.Notethatamovewhichassignsagivenlabeltoasinglepixelisbothan.Asaconsequence,astandardmoveisaspecialcaseofbothaandan3.2AlgorithmsandPropertiesWehavedevelopedtwominimizationalgorithms.Theswapalgorithmfindsalocalminimumwhenswapmovesareallowedandtheexpansionalgorithmfindsalocalminimumwhenexpansionmovesareallowed.Findingsuchalocalminimumisnotatrivialtask.Givenalabeling,thereisanexponentialnumberofswapandexpansionmoves.Therefore,evencheckingforalocalminimumrequiresexponentialtimeifperformednaõvely.Incontrast,checkingforalocalminimumwhenonlythestandardmovesareallowediseasysincethereisonlyalinearnumberofstandardmovesgivenanylabelingWehavedevelopedefficientgraph-basedmethodstofindtheoptimalgivenalabeling(seeSections4and5).Thisisthekeystepinouralgorithms.Oncethesemethodsareavailable,itiseasytodesignvariantsoftheªfastestdescentºtechniquethatcanefficientlyfindthecorrespondinglocalminima.OuralgorithmsaresummarizedinFig.3.Thetwoalgorithmsarequitesimilarintheirstructure.WewillcallasingleexecutionofSteps3.1-3.2anandanexecutionofSteps2,3,and4a.Ineachcycle,thealgorithmperformsaniterationforeverylabel(expan-sionalgorithm)orforeverypairoflabels(swapalgorithm),inacertainorderthatcanbefixedorrandom.Acycleissuccessfulifastrictlybetterlabelingisfoundatanyiteration.Thealgorithmsstopafterthefirstunsuccessfulcyclesincenofurtherimprovementispossible.Obviously,acycleintheswapalgorithmtakesiterations,andacycleintheexpansionalgorithmtakesThesealgorithmsareguaranteedtoterminateinafinitenumberofcycles.Infact,undertheassumptionsthatin(1)areconstantsindependentoftheimagesize,wecaneasilyproveterminationin…jPj†cycles[43].Inpractice,theseassumptionsarequitereasonable.However,intheexperimentswereportinSection8,thealgorithmstopsafterafewcyclesandmostoftheimprovementsoccurduringthefirstcycle.WeusegraphcutstoefficientlyfindforthekeypartofeachalgorithminStep3.1.Step3.1usesasinglegraphcutcomputation.Ateachiteration,thecorrespondinggraph…jPj†pixels.Theexactnumberofpixels,topologyofthegraph,anditsedgeweightsvaryfromiterationto BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS Fig.2.Examplesofstandardandlargemovesfromagiveninitiallabeling(a).ThenumberoflabelsisjLjˆ.Astandardmove,(a)(b),changesthelabelofasinglepixel(inthecircledarea).Strongmoves,(c)and-expansion(d),allowlargenumberofpixelstochangetheirlabelssimultaneously. iteration.ThedetailsofthegrapharequitedifferentfortheswapandtheexpansionalgorithmsandaredescribedindetailsinSections4and5.3.3GraphCutsBeforedescribingthekeyStep3.1oftheswapandtheexpansionalgorithms,wewillreviewgraphcuts.LetGˆhVbeaweightedgraphwithtwodistinguishedverticescalledtheterminals.ACEisasetofedgessuchthattheterminalsareseparatedintheinducedgraphG…C†ˆhVEÿCi.Inaddition,nopropersubsetoftheterminalsinG…C†.Thecostofthecut,denotedjCj,equalsthesumofitsedgeweights.Theminimumcutproblemistofindthecheapestcutamongallcutsseparatingtheterminals.Notethatweusestandardterminologyfromthecombina-torialoptimizationcommunity.Sections4and5showthatStep3.1inFig.3isequivalenttosolvingtheminimumcutproblemonanappropriatelydefinedtwo-terminalgraph.Minimumcutscanbeeffi-cientlyfoundbystandardcombinatorialalgorithmswithdifferentlow-orderpolynomialcomplexities[1].Forexam-ple,aminimumcutcanbefoundbycomputingthemaximumflowbetweentheterminals,accordingtoatheoremduetoFordandFulkerson[15].Ourexperimentalresultsmakeuseofanewmax-flowalgorithmthathasthebestspeedonourgraphsovermanymodernalgorithms[10].Therunningtimeisnearlylinearinpractice.INDINGTHEGivenaninputlabeling)andapairoflabels;,wewishtofindalabelingthatminimizesoveralllabelingswithinoneswapof.ThisisthecriticalstepintheswapmovealgorithmgivenatthetopofFig.3.OurtechniqueisbasedoncomputingalabelingcorrespondingtoaminimumcutonagraphˆhV.Thestructureofthisgraphisdynamicallydeterminedbythecurrentpartitionandbythelabels;Thissectionisorganizedasfollows:First,wedescribetheconstructionofforagiven.Weshowthatcutscorrespondinanaturalwaytolabelingswhicharewithinoneswapmoveof.Theorem4.4showsthatthecostofacutisjCjˆplusaconstant.Acorollaryfromthistheoremstatesourmainresultthatthedesiredlabeling,whereisaminimumcutonThestructureofthegraphisillustratedinFig.4.Forlegibility,thisfigureshowsthecaseofa1Dimage.Foranyimage,thestructureofwillbeasfollows:Thesetofverticesincludesthetwoterminals,aswellasimagepixelsthesets(thatis;).Thus,thesetofconsistsof,andand.Eachpixelisconnectedtotheterminalsbyedgesrespectively.Forbrevity,wewillrefertotheseedgesas1226IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER2001 Fig.3.Ourswapalgorithm(top)andexpansionalgorithm(bottom).6.Toavoidconfusion,wewouldliketomentionthatsomeclustering-basedsegmentationtechniquesinvisionusedifferentgraphcutterminol-ogy.Forexample,[47]computesagloballyminimumcut.Theminimumiscomputedamongallcutsthatseverthegraphintotwononemptyparts.Theterminalsneednotbespecified.Recently,[38]introducednormalizedcutsproposinganewdefinitionofthecutcost.Althoughnormalizedcutsareformulatedasagraphpartitioningproblem,theactualapproximateoptimizationisperformedvianoncombinatorialmethods. Fig.4.Anexampleofthegraphfora1Dimage.Thesetofpixelsintheimageisis,wherep;r;s... (terminallinks).Eachpairofpixelsp;qwhichareneighbors(i.e.,p;q)isconnectedbyanedgewewillcallan(neighborlink).Thesetofedges,,thusconsistsofg2N.Theweightsassignedtotheedgesare Anycutmustsever(include)exactlyoneanypixel:ifneitherwerein,therewouldbeapathbetweentheterminals;whileifbothwerecut,thenapropersubsetofwouldbeacut.Thus,anycutleaveseachpixelinwithexactlyone.Thisdefinesanaturallabelingcorrespondingtoacut;p=Inotherwords,ifthepixelisin,thenisassignedwhenthecutfromtheterminalisassignedlabelfromthe.Ifisnotin,thenwekeepitsinitiallabelThisimpliesthefollowing.Lemma4.1.AlabelingcorrespondingtoacutisoneswapawayfromtheinitiallabelingItiseasytoshowthatacutseversanneighboringpixelsonifandonlyifleavesthepixelsconnectedtodifferentterminals.Formally,Property4.2.ForanycutandforanylinkeIftthene62CIftthene62CIfttheneIfttheneProperties(a)and(b)followfromtherequirementthatnopropersubsetofshouldseparatetheterminals.Properties(c)and(d)alsousethefactthatacuthastoseparatetheterminals.ThesepropertiesareillustratedinFig.5.ThenextlemmaisaconsequenceofProperty4.2and(6).Lemma4.3.ForanycutandforanylinkeTherearefourcaseswithsimilarstructure;wewillillustratethecasewhere.Inthiscase,and,therefore,jC\;.Asfollowsfrom(6),Notethatthisproofassumesthatisasemimetric,i.e.,that(2)and(3)hold.Lemmas4.1and4.3plusProperty4.2yieldTheorem4.4.ThereisaonetoonecorrespondencebetweencutsandlabelingsthatareoneswapfromMoreover,thecostofacutjCjˆplusaThefirstpartfollowsfromthefactthatthesevereduniquelydeterminethelabelsassignedtopixelsandthethatmustbecut.Wenowcomputethecostofacut,whichisjCjˆNotethatfor,wehaveLemma4.3givesthesecondtermin(7).Thus,thetotalcostofacutjCjˆporqThiscanberewrittenasjCjˆ,whereisthesameconstantforallcutsCorollary4.5.Thelowestenergylabelingwithinasinglemovefrom,whereistheminimumcutonINDINGTHEGivenaninputlabeling)andalabel,wewouldliketofindalabelingthatminimizesoveralllabelingswithinone.ThisisthecriticalstepintheexpansionmovealgorithmgivenatthebottomofFig.3.Inthissection,wedescribeatechniquethatsolvestheproblemassumingthat(each)isametricand,thus,satisfiesthetriangleinequality(4).Ourtechniqueisbasedoncomputingalabelingcorrespondingtoaminimumcut BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS Fig.5.Propertiesofacutfortwopixelsp;qconnectedby.DottedlinesshowtheedgescutbyandsolidlinesshowtheedgesremainingintheinducedgraphG…C†ˆhVEÿCi onagraphˆhV.Thestructureofthisgraphisdeterminedbythecurrentpartitionandbythelabel.Asbefore,thegraphdynamicallychangesaftereachiteration.Thissectionisorganizedasfollows:First,wedescribetheconstructionofforagiven)and.Weshowthatcutscorrespondinanaturalwaytolabelingswhicharewithinonemoveof.Then,basedonanumberofsimpleproperties,wedefineaclassofcuts.Theorem4.5showsthatelementarycutsareinonetoonecorrespondencewiththoselabelingsthatarewithinoneand,also,thatthecostofanelementarycutisjCjˆ.AcorollaryfromthistheoremstatesourmainresultthatthedesiredlabelingisaminimumcutonThestructureofthegraphisillustratedinFig.6.Forlegibility,thisfigureshowsthecaseofa1Dimage.Thesetofverticesincludesthetwoterminals,aswellasallimagepixels.Inaddition,foreachpairofneighboringp;qseparatedinthecurrentpartition(i.e.,suchthat),wecreateanauxiliarynodeAuxiliarynodesareintroducedattheboundariesbetweenpartitionsets.Thus,thesetofverticesisisfp;qg2Nfp6ˆfqafpqg):Eachpixelisconnectedtotheterminals,respectively.Eachpairofneighboringpixelsp;qwhicharenotseparatedbythepartitionsuchthat)isconnectedbyan.Foreachpairofneighboringpixelsp;qsuchthat,wecreateatripletofedges,wherethecorrespondingauxiliarynode.Theedgesconnectpixelsandtheconnectstheauxiliarynodetotheterminal.So,wecanwritethesetofalledgesasasp2Ptp;tp ;[fp;qg2Nfp6ˆfqEfp;qg;[fp;qg2Nfpˆfqefp;qg):Theweightsassignedtotheedgesare AsinSection4,anycutmustsever(include)exactlyoneforanypixel.Thisdefinesanaturalcorrespondingtoacut.Formally,Inotherwords,apixelisassignedlabelifthecutfromtheterminal,whileisassigneditsold.Notethat,for62P,therepresentslabelsassignedtopixelsintheinitial.Clearly,wehavethefollowing.Lemma5.1.AlabelingcorrespondingtoacutisoneawayfromtheinitiallabelingAlso,itiseasytoshowthatacutseversanbetweenneighboringpixelsp;qsuchthatandonlyifleavesthepixelsconnectedtodifferentterminals.Inotherwords,Property4.2holdswhenwesubstituteªºforªº.WewillrefertothisasProperty4.2(Analogously,wecanshowthatProperty4.2and(8)establishLemma4.3fortheNow,considerthesetofedgescorrespondingtoapairofneighboringpixelsp;qsuchthat.Inthiscase,thereareseveraldifferentwaystocuttheseedgesevenwhenthepairofseveredisfixed.However,aminimumcutisguaranteedtosevertheedgesindependingonwhatarecutatthe.TheruleforthiscaseisdescribedinProperty5.2below.Assumethatisanauxiliarynodebetweenthecorrespondingpairofneighboringpixels.Property5.2.p;q,thenaminimumcutIftC\EIftC\EIftC\EIftC\EProperty(a)resultsfromthefactthatnosubsetofisacut.TheothersfollowfromtheminimalityofjCjandthefact,andsatisfythetriangleinequalityso1228IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER2001 Fig.6.Anexampleoffora1Dimage.ThesetofpixelsintheimageisPˆfp;q;r;sandthecurrentpartitionisˆfP,whereq;r,and.Twoauxiliarynodesareintroducedbetweenneighboringpixelsseparatedinthecurrentpartition.Auxiliarynodesareaddedattheboundaryofsets thatcuttinganyoneofthemischeaperthancuttingtheothertwotogether.ThesepropertiesareillustratedinFig.7.Lemma5.3.p;q,thentheminimumcutjC\ETheequationfollowsfromProperty5.2,(8),andtheedgeweights.Forexample,if,thenjC\E.Atthesametime,(8)im-pliesthat.Notethattherightpenaltyisimposedwhenever,duetotheauxiliarynodeconstruction.Property4.2()holdsforanycut,andProperty5.2holdsforaminimumcut.However,therecanbeothercutsbesidestheminimumcutthatsatisfybothproperties.WewilldefineancutontobeacutthatsatisfiesProperties4.2()and5.2.Theorem5.4.beconstructedasabovegiven.Then,thereisaonetoonecorrespondencebetweenelementarycutsonandlabelingswithinone.Moreover,foranyelementarycut,wehavejCjˆWefirstshowthatanelementarycutisuniquelydeterminedbythecorrespondinglabeling.Theatthepixeldetermineswhichoftheisin.Property4.2()showswhichbetweenpairsofneighboringpixelsp;qsuchthatshouldbesevered.Similarly,Property5.2determineswhichofthelinksincorrespondingtop;qsuchthatshouldbecut.ThecostofanelementarycutjCjˆC\EItiseasytoshowthat,foranypixel,wehavejC\fgjˆ.Lemmas4.3and5.3holdforelementarycutssincetheywerebasedonProperties4.2and5.2.Thus,thetotalcostofaelementarycutjCjˆjCjˆOurmainresultisasimpleconsequenceofthistheoremsincetheminimumcutisanelementarycut.Corollary5.5.Thelowestenergylabelingwithinasingleexpansionmovefrom,whereistheminimumcutHere,wediscussoptimalitypropertiesofouralgorithms.InSection6.1,weshowthatanylocalminimumgeneratedbyourexpansionmovesalgorithmiswithinaknownfactoroftheglobaloptimum.ThisalgorithmworksincaseofmetricTheswapmovealgorithmcanbeappliedtoawiderclassofbut,unfortunately,itdoesnothaveany(similar)guaranteedoptimalityproperties.InSection6.2,weshowthataprovablygoodsolutioncanbeobtainedevenforbyapproximatingsuchwithasimplePottsmetric.6.1TheExpansionMoveAlgorithmWenowprovethatalocalminimumwhenexpansionmovesareallowediswithinaknownfactoroftheglobalminimum.Thisfactor,whichcanbeassmallas2,willdependonSpecifically,let ;;betheratioofthelargestnonzerovalueoftothesmallestnonzerovalueof.Notethatiswell-definedsince;†6ˆaccordingtothepropertyin(2).Ifaredifferentforneighboringpairsp;q,then ;;Theorem6.1.bealocalminimumwhentheexpansionmovesareallowedandbethegloballyoptimalsolution.Letusfixsomeandlet2PjWecanproducealabelingwithinonemovefromasfollows:Thekeyobservationisthatsinceisalocalminimumifexpansionmovesareallowed,beasetconsistingofanynumberofpixelsinandanynumberofpairsofneighboringpixelsin.Wetobearestrictionoftheenergyoflabelingtotheset BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS Fig.7.Propertiesofaminimumcutfortwopixelp;q.DottedlinesshowtheedgescutbyandsolidlinesshowtheedgesintheinducedgraphG…C†ˆhVEÿCi bethesetofpixelsandpairsofneighboringpixels.Also,letbethesetofpairsofneighboringpixelsonthebethesetofpixelsandpairsofneighboringpixelscontained.Formally,Formally,fp;qp;q62PˆPÿP…†[p;q62P62PThefollowingthreefactshold:Equations(13)and(14)areobviousfromthedefini-tionsin(11)and(10).Equation(15)holdsbecause,forp;q,wehave†6ˆˆB[Oincludesallpixelsinandallneighboringpairsofpixelsin,wecanexpandbothsidesof(12)toget:Using(13),(14),and(15),wegetfromtheequationabove:Togettheboundonthetotalenergy,weneedtosum(16)overalllabels.Observethat,foreveryp;q,thep;qappearstwiceontheleftsideof(17),oncein,andoncein.Similarly,everyp;qtimesontherightsideof(17).Therefore,(17)canberewrittentogettheboundofNotethatKleinbergandTardos[27]developanalgorithmforminimizingwhichalsohasoptimalityproperties.ForthePottsmodeldiscussedinthenextsection,theiralgorithmhasaboundof2.ThisisthesameboundasweobtaininTheorem6.1forthePottsmodel.Forageneral,theyhaveaboundofloglog,wherethenumberoflabels.However,theiralgorithmuseslinearprogramming,whichisimpracticalforthelargenumberofvariablesoccurringinearlyvision.6.2ApproximatingaSemimetricAlocalminimumwhenswapmovesareallowedcanbearbitrarilyfarfromtheglobalminimum.ThisisillustratedbyanexampleinFig.8.Infact,wecanusetheexpansionalgorithmtogetananswerwithinafactoroffromtheoptimumofenergy(1)evenwhenisasemimetric.Here,isthesameasinTheorem6.1.Thisisstillwell-definedforasemimetric.Supposethatpenaltyinsidethedefinitionofenergy(1)isasemimetric.Letbeanyrealnumberintheintervalm;M,where;;DefineanewenergybasedonthePottsinteractionmodelTheorem6.2.isalocalminimumofgiventheexpansionmovesandistheglobalminimumof,thenistheglobalminimumof.Then, rME…^fP…^fEP…foEP…f wherethesecondinequalityfollowsfromTheorem6.1.NotethatM=mThus,tofindananswerwithinafixedfactorfromtheglobalminimumforasemimetric,onecantakealocalgiventheexpansionmovesfor,asdefinedabove.Notethatsuchanisnotalocalminimumofgiventheexpansionmoves.Inpractice,however,wefindthatlocalminimumgiventheswapmovesgivesempiricallybetterresultsthanusing.Infact,theestimatecanbeusedasagoodstartingpointfortheswapalgorithm.Inthiscase,theswapmovealgorithmwillalsogeneratealocalminimumwhoseenergyiswithinaknownfactorfromtheglobalAninterestingspecialcaseoftheenergyin(1)ariseswhenisgivenbythePottsmodel[35]Gemanetal.[18]werethefirsttousethismodelincomputervision.Inthiscase,discontinuitiesbetweenanypairoflabelsarepenalizedequally.Thisis,insomesense,thesimplestdiscontinuitypreservingmodelanditisespeciallyuseful1230IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER20017.Infact,itcanbeshownthatanyalgorithmthatiswithinafactoroftwoforthePottsmodeliswithinafactorofforanarbitrarymetric Fig.8.TheimageconsistsofthreepixelsPˆf.TherearetwopairsofneighborsNˆff.ThesetoflabelsisLˆfa;b;cThevaluesofareshownin(c).a;bb;c a;c.Itiseasytoseethattheconfigurationin(a)isalocalminimumwithrespecttoswapmoves.Itsenergyis,whiletheoptimalconfigurationshownin(b)hasenergy4. whenthelabelsareunorderedorthenumberoflabelsissmall.ThePottsinteractionpenaltyisametric;inthiscase,andourexpansionalgorithmgivesasolutionthatiswithinafactoroftwooftheglobalminimum.Notethatbydefinition,sothisistheenergyfunctionwiththebestbound.Interestingly,thePottsmodelenergyminimizationproblemiscloselyrelatedtoaknowncombinatorialoptimizationproblemcalledthemultiwaycutproblem.Inthissection,weinvestigatethisrelationshipanditsconsequences.Wewillfirstshow(Section7.1)thatthePottsmodelenergyminimizationproblemcanbereducedtothemultiwaycutproblem.Moreprecisely,weprovethattheglobalminimumofthePottsmodelenergycanbecomputedbyfindingtheminimumcostmultiwaycutonanappropriatelyconstructedgraph.Weprove(intheAppen-dix)thatifwecouldefficientlycomputetheglobalminimumofwecouldalsosolveacertainclassofmultiwaycutproblemsthatareknowntobeNP-hard.Thisinturn,impliesthatminimizingisNP-hardand,so,isminimizingtheenergyin(1).Themultiwaycutproblemisdefinedonagraphwithnonnegativeedgeweights,withasetofterminal.AsubsetoftheedgesCEiscalledamultiwaycutiftheterminalsarecompletelyseparatedintheinducedgraphG…C†ˆhVEÿCi.WewillalsorequirethatnopropersubsetofseparatestheterminalsinG…C†.ThecostofthemultiwaycutisdenotedbyjCjandequalsthesumofitsedgeweights.Themultiwaycutproblemistofindtheminimumcostmultiwaycut[13].In[13],theyalsoshowthatthemultiwaycutproblemisNP-complete.Notethatthemultiwaycutproblemisageneralizationofthestandardtwo-terminalgraphcutproblemdescribedinSection3.3.7.1ThePottsModelandtheMultiwayCutProblemWenowshowthattheproblemofminimizingthePottsenergycanbesolvedbycomputingaminimumcostmultiwaycutonacertaingraph.WetakeVˆP[L.Thismeansthatcontainstwotypesofvertices:(pixels)and(labels).Notethatwillserveasterminalsforourmultiwaycutproblem.Twoareconnectedbyanedgeifandonlyifthecorrespondingpixelsareneighborsintheneighborhoodsystem.Thesetconsistsoftheedges,whichwewillcall.Eachp;qg2EisassignedaweightisconnectedbyanedgetoeachAnedgep;lthatconnectsawithaterminal(an)willbecalledaandthesetofallsuchedgeswillbedenotedby.Eachp;lisassigneda,whereisaconstantthatislargeenoughtomaketheweightspositive.TheedgesofthegraphareEˆEET.Fig.9ashowsthestructureofthegraphItiseasytoseethatthereisaone-to-onecorrespondencebetweenmultiwaycutsandlabelings.AmultiwaycutcorrespondstothelabelingwhichassignsthelabeltoallwhicharetotheG…C†.Anexampleofamultiwaycutandthecorrespondingimagepartition(labeling)isgiveninFig.9b.Theorem7.1.isamultiwaycuton,thenjCjˆplusaconstant.TheproofofTheorem7.1isgivenin[11].Corollary7.2.isaminimumcostmultiwaycuton,thenWhilethemultiwaycutproblemisknowntobeNP-completeiftherearemorethantwoterminals,thereisafastapproximationalgorithm[13].Thisalgorithmworksasfollows:First,foreachterminal,itfindsanwayminimumcutthatseparatesfromallotherterminals.Thisisjustthestandardgraphcutproblem.Then,thealgorithmgeneratesamultiwaycutCˆ[,whereargmaxjC…istheterminalwiththelargestcost BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS Fig.9.(a)ExampleofthegraphGˆhVwithmultipleterminals....(b)Amultiwaycuton.Thepixelsareshownaswhitesquares.Eachpixelhasantoitsfourneighbors.Eachpixelisalsoconnectedtoallterminalsby(someoftheareomittedfromthedrawingforlegibility).ThesetofverticesVˆP[Lincludesallpixelsandterminals.ThesetofedgesEˆEETconsistsofall.In(b),weshowtheinducedgraphG…C†ˆhVEÿCicorrespondingtosomemultiwaycut.Amultiwaycutcorrespondstoauniquepartition(labeling)ofimagepixels. isolatingcut.Thisªisolationheuristicºalgorithmproducesacutwhichisoptimaltowithinafactorof .However,theisolationheuristicalgorithmsuffersfromtwoproblemsthatlimitsitsapplicabilitytoourenergyminimizationproblem.Thealgorithmwillassignmanypixelsalabelthatischosenessentiallyarbitrarily.Notethattheunionofallisolatingcutsutsl2LC…l†mayleavesomeverticesdisconnectedfromanyterminal.ThemultiwaycutCˆ[connectsallthoseverticestotheWhilethemultiwaycutproducedisclosetooptimal,thisdoesnotimplythattheresultingisclosetooptimal.Formally,letuswriteTheorem7.1asjCjˆ…C†‡resultsfromthe's,asdescribedin[11]).TheisolationheuristicgivesasolutionCj,whereistheminimumcostmultiwaycut.Thus,C†‡C†.Asaresult,theisolationheuristicalgorithmdoesnotproducealabelingwhoseenergyiswithinaconstantfactorofoptimal.Notethattheusedintheconstructiongivenin[11]issolargethatthisboundisnearlyInthissection,wepresentexperimentalresultsonvisualcorrespondenceforstereo,motion,andimagerestoration.Inimagerestoration,weobserveanimagecorruptedbynoise.Thetaskistorestoretheoriginalimage.Thus,thelabelsareallpossibleintensitiesorcolors.Therestoredintensityisassumedtoliearoundtheobservedoneandtheintensitiesareexpectedtovarysmoothlyeverywhereexceptatobjectboundaries.Invisualcorrespondence,wehavetwoimagestakenatthesametimefromdifferentviewpointsforstereoandatdifferenttimesformotion.Formostpixelsinthefirstimagethereisacorrespondingpixelinthesecondimagewhichisaprojectionalongthelineofsightofthesamereal-worldsceneelement.Thedifferenceinthecoordinatesofthecorrespondingpointsiscalledthedisparity.Instereo,thedisparityisusuallyone-dimensionalbecausecorrespondingpointsliealongepipolarlines.Inmotion,thedisparityisusuallytwo-dimensional.Thus,forcorrespondencethelabelsetisadiscretizedsetofallpossibledisparitiesandthetaskistoestimatethedisparitylabelforeachpixelinthefirstimage.Notethatherecontainsthepixelsofthefirstimage.Thedisparityvariessmoothlyeverywhereexceptatobjectboundariesandcorrespondingpointsareexpectedtohavesimilarintensities.Wecanformulatetheimagerestoration(Section8.6)andcorrespondenceproblems(Sections8.3,8.4,and8.5)asenergyminimizationproblemofthetypein(1).WedescribeourdatatermsinSection8.1.Weusedifferentandwestatethemforeachexample.Section8.2explainsstaticcuesthathelptosetThecorrespondingenergiesareminimizedusingourswapandexpansionalgorithmsgiveninFig.3.Optimalswapandexpansionmoves(Step3.1inFig.3)arefoundbycomputingminimumcostcutsongraphsdesignedinSections4and5.Ourimplementationcomputesminimumcutsusinganewmax-flowalgorithm[10].Runningtimespresentedbelowwereobtainedona333MHzPentiumIII.8.1DataTermForimagerestoration,ourdatatermisstraightforward.istheobservedimageandistheintensityobservedatpixel.Then,;constwhichsaysthattherestoredintensitylabelshouldbeclosetotheobservedintensity.Wesetparameteritisusedtomakethedatapenaltymorerobustagainstoutliers,i.e.,pixelswhichdonotobeytheassumednoisemodel.Thealgorithmisverystablewithrespecttowhichsimplyhelpstosmoothoutthefewoutlyingpixels.Forexample,ifwesettoinfinity,theresultsaremostlythesameexcepttheybecomespeckledbyafewnoisypixels.Now,weturntothedatatermforthestereocorrespon-denceproblem.Supposethefirstimageisandthesecondis.Ifthepixelscorrespond,theyareassumedtohavesimilarintensities.However,therearespecialcircumstanceswhencorrespondingpixelshaveverydiffer-entintensitiesduetotheeffectsofimagesampling.Supposethatthetruedisparityisnotanintegerandthedisparityrangeisdiscretizedtoonepixelaccuracy,aswedohere.Ifapixeloverlapsascenepatchwithhighintensitygradient,thenthecorrespondingpixelsmayhavesignificantlydifferentintensities.Forstereo,weusethetechniqueof[6]todevelopaisinsensitivetoimagesampling.First,wemeasurehowwellfitsintotherealvaluedrangeofdisparities 12 fwdp;d 12xd‡ Wegetfractionalvaluesbylinearinterpolationbetweendiscretepixelvalues.Forsymmetry,wealsop;d 12xp‡ fwdp;dp;dcanbecomputedwithjustafewcomparisons.Thefinalmeasureisp;dfwdp;dp;d;constWesetforallexperimentsanditspurposeandeffectisthesameasthosedescribedfortheimagerestoration.Formotion,wedevelopedsimilartostereo,exceptinterpolationisdoneintwodimensionssincelabelsarenowtwo-dimensional.Detailsaregivenin[43].8.2StaticCuesInthevisualcorrespondence,thereiscontextualinforma-tionwhichwecantakeadvantageof.Forsimplicity,wewillconsiderthecaseofthePottsmodel,i.e.,1232IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER20018.Thissimpleapproachdoesnottreattheimagessymmetricallyandallowsinconsistentdisparities.Forexample,twopixelsinthefirstimagemaybeassignedtoonepixelinthesecondimage.Occlusionsarealsoignored.[28]presentsastereoalgorithmbasedonexpansionmovesthataddressestheseproblems. .Theintensitiesofpixelsinthefirstimagecontaininformationthatcansignificantlyinfluenceourassessmentofdisparitieswithoutevenconsideringthesecondimage.Forexample,twoneighboringpixelsaremuchmorelikelytohavethesamedisparityifweknowthat.Mostmethodsforcomputingcorrespondencedonotmakeuseofthiskindofcontextualinformation.Someexceptionsinclude[5],[33],[45].Wecaneasilyincorporatecontextualinformationintoourframeworkbyallowingtovarydependingonthe.Letrepresentsapenaltyforassigningdifferentdisparitiestoneighboringpixels.Thevalueoftheshouldbesmallerforpairsp;qwithlargerintensitydifferences.Inpractice,wefoundthefollowingsimplefunctiontoworkwell:j†ˆisthePottsmodelparameter.Notethatinsteadof(19),wecouldalsosetthecoefficientsaccordingtoanoutputofanedgedetectoronthefirstimage.Forexample,canbemadesmallforpairsp;q,whereanintensityedgewasdetectedandlargeotherwise.Segmentationresultscanalsobeused.Thefollowingexampleshowstheimportanceofcon-textualinformation.Considerthepairofsyntheticimagesbelow,withauniformlywhiterectangleinfrontofablack Thereisaonepixelhorizontalshiftinthelocationoftherectangleandthereisnonoise.Withoutnoise,theproblemofestimatingisreducedtominimizingthesmoothnessundertheconstraintthatpixelcanbeassigneddisparityonlyifisthesameforallpairsofneighborsp;q,thenisminimizedatoneofthelabelingshowninthefigurebelow.Exactlywhichlabelingminimizesdependsontherelationshipbetweentheheightofthesquareandtheheightofthebackground. Supposenowthatthepenaltyismuchsmallerifthanitisif.Inthiscase,theminimumofachievedatthedisparityconfigurationshowninthefigurebelow.Thisresultismuchclosertohumanperception. Staticcueshelpmostlyinareasoflowtexture.Applicationonrealimagesshowthatthestaticcuesgiveimprovement,butnotasextremeastheexampleabove.SeeSection8.3fortheimprovementsthatthestaticcuesgiveonrealimages.8.3RealStereoImagerywithGroundTruthInFig.10,weshowresultsfromarealstereopairwithknowngroundtruth,providedbyDr.Y.OhtaandDr.Y.NakamurafromtheUniversityofTsukuba.TheleftimageisinFig.10aandthegroundtruthisinFig.10b.Themaximumdisparityforthisstereopairis14,soourdisparitylabelsetis.Thegroundtruthimageactuallyhasonlysevendistinctdisparities.Theobjectsinthisscenearefronto-paralleltothecamera,sothePottsmodel,i.e.,workswell.Sincetherearetexturelessregionsinthescene,thestaticcueshelp,andthecoefficientsaregivenby(19)and(20).Wecomparedourresultsagainstannealingandnormal-izedcorrelation.Fornormalizedcorrelation,wechoseparameterswhichgivethebeststatistics.Weimplementedseveraldifferentannealingvariantsandusedtheonethatgavethebestperformance.ThiswastheMetropolissamplerwithalinearlydecreasingtemperatureschedule.Togiveitagoodstartingpoint,simulatedannealingwasinitializedwiththeresultsfromnormalizedcorrelation.Incontrast,forouralgorithms,thestartingpointisunimportant.Theresultsdifferbylessthan1percentofimagepixelsfromanystartingpointthatwehavetried.Also,werun100testswithrandomlygeneratedinitiallabelings.Finalsolutionspro-ducedbyourexpansionandswapalgorithmshadtheaverageenergyof,correspondingly,whilethestandarddeviationswereonly1,308and459.Figs.10cand10dshowtheresultsoftheswapandexpansionalgorithmsfor,whereistheparameterin(20).Figs.10eand10fshowtheresultsofnormalizedcorrelationandsimulatedannealing.Comparisonswithotheralgorithmscanbefoundin[40].Note,however,that[40]confirmsthatforthisimagerythebestpreviousalgorithmissimulatedannealing,whichoutperforms(amongothers)correlation,robustestimation,scanline-baseddynamicprogramming,andmean-fieldtechniques.Fig.12summarizestheerrorsmadebythealgorithms.Inapproximately20minutes,simulatedannealingreducesthetotalerrorsnormalizedcorrelationmakesbyaboutonefifthanditcutsthenumberoferrorsinhalf.Itmakesverylittleadditionalprogressintherestoffourhours.Ourexpansionandswapalgorithmsmakeapproximatelyfivetimesfewererrorsandapproximatelythreetimesfewertotalerrorscomparedtonormalizedcorrelation.Theexpansionandswapalgorithmsperformsimilarlytoeachother.Theobserveddifferenceinerrorsisinsignif-icant,lessthan1percent.Ateachcycle,theorderoflabels BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS toiterateoverischosenrandomly.Anotherrunofthealgorithmmightgiveslightlydifferentresults,andonaverageabout1percentofpixelschangetheirlabelsbetweendifferentruns.Onaverage,theexpansionalgo-rithmconverges1.4timesfasterthantheswapalgorithm.Fig.11showsthegraphofversustime(inseconds)forouralgorithmsandsimulatedannealing.Notethatthetimeaxisisonalogarithmicscale.Wedonotshowthegraphforbecausethedifferenceintheamongallalgorithmsisinsignificant,asexpectedfromthefollowingargument.Mostpixelsinrealimageshavenearbypixelswithsimilarintensities.Thus,formostpixels,thereareseveralforwhichisapproximatelythesameandsmall.Fortherestofisquitelarge.Thislattergroupofdisparitiesisessentiallyexcludedfromconsiderationbyenergyminimizingalgorithms.Theremainingchoicesofaremoreorlessequallylikely.Thus,thetermoftheenergyfunctionhasverysimilarvaluesforourmethodsandsimulatedannealing.Ourmethodsquicklyreducethesmoothnessenergytoaround,whilethebestsimulatedannealingcanproduceinfourhoursisaround,whichistwiceasbad.Theexpansionalgorithmgivesaconvergencecurvesignificantlysteeperthantheothercurves.Infact,theexpansionalgorithmmakes99percentoftheprogressinthefirstiterationwhichtakeseightseconds.FinalenergiesaregiveninFig.13.Staticcueshelpintheupperrighttexturelesscorneroftheimage.Withoutthestaticcues,acornerofsizeapproximately800pixelsgetsbrokenoffandisassigned1234IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER2001 Fig.10.Realimagerywithgroundtruth.(a)Leftimage:384x288,15labels.(b)Groundtruth.(c)Swapalgorithm.(d)Expansionalgorithm.(e)Normalizedcorrelation.(f)Simulatedannealing. tothewrongdisparity.ThisisreflectedintheerrorcountshowninFig.12,whichworsenswithoutthestaticcues.Thepercentageimprovementmaynotseemtoosignificant,however,visuallyitisverynoticeablesincewithoutthestaticcuesalargeblockofpixelsismisplaced.Weomittheactualimageduetospaceconstraints.Theonlyparameterofthethisenergyfunctionis(20).Thealgorithmsappearstableinthechoiceof.ThetableinFig.14givestheerrorsmadebytheexpansionalgorithmfordifferent.Forsmall,therearemanyerrorsbecausethedatatermisoveremphasizedandfor,therearemanyerrorsbecausethesmoothnesstermisoveremphasized.However,foralargeintervalofvalues,theresultsaregood.Anotherimportanttestistoincreasethenumberoflabelsandevaluatetheeffectsontherunningtimeandtheaccuracyofouralgorithms.Fig.15summarizesthetestresultsfortheexpansionalgorithm(thosefortheswapalgorithmaresimilar).Thefirstcolumnshowsthenumberofintegerdisparitiesthatweuse.Thesecondandthirdcolumnsshowthetimeittooktocompleteoneiterationandtoconverge,correspondingly.Thelasttwocolumnsgivetheerrorcountsatconvergence.Thesecondandthirdcolumnsconfirmthattherunningtimeislinearonaverage.Notethatthenumberofcyclestoconvergencevaries,explaininghighervariabilityinthethirdcolumn.Thelasttwocolumnsshowthattheaccuracyworsensslightlywiththeincreaseinthenumberoflabels.8.4SRITreeStereoPairIntheSRIstereopairwhoseleftimageisshowninFig.16a,thegroundisaslantedsurfaceand,therefore,apiecewiseconstantmodel(Pottsmodel)doesnotworkaswell.Forthisimagepair,wechoosewhichisapiecewisesmoothmodel.Itisametricand,so,weusetheexpansionalgorithmforminimization.Thissceneiswell-textured,sostaticcuesarenotused.Fig.16bandFig.16ccomparetheresultsofminimizingwiththePottsandpiecewisesmoothmodel.Therunningtimestoconvergenceare94secondsand79seconds,respectively.NoticethattherearefewerdisparitiesfoundinFig.16bsincethePottsmodeltendstoproducelargeregionswiththesamedisparity. BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS Fig.11.Energyversustime(inseconds)ofexpansion,swap,andsimulatedannealingalgorithmsfortheprobleminFig.10a.Thestartingenergyisthesameforallalgorithms. Fig.12.Comparisonofaccuracyandrunningtimes. Fig.13.Energiesatconvergenceforouralgorithmsandsimulatedannealing. 8.5MotionFig.17showstheoutputfromthewell-knownflowergardensequence.Sincethecameramotionisnearlyhorizontal,wehavesimplydisplayedthecameramotion.Themotioninthissequenceislarge,withtheforegroundtreemovingsixpixelsinthehorizontaldirection.WeusedthePottsmodelinthisexamplebecausethenumberoflabelsissmall.Thisimagesequenceisrelativelynoisy,sowetook.Determiningthemotionoftheskyisaveryhardprobleminthissequence.Evenstaticcuesdonothelp,sowedidn'tusethem.Therunningtimeis15secondstoconvergence.Fig.18ashowsoneimageofamotionsequencewhereacatmovesagainstmovingbackground.Themotionislarge,withmaximumhorizontaldisplacementoffourpixelsandmaximumverticaldisplacementoftwopixels.Weusedeighthorizontalandfiveverticaldisplacements,thusthelabelsethassize40.Thisisadifficultsequencebecausethecat'smotionisnonrigid.Thesceneiswell-textured,sothestaticcuesarenotused.Inthiscase,wechose,wherearehorizontalandverticalcomponentsofthelabel(recallthatthelabelshavetwodimensionsformotion).Thisisnotametric,soweusedtheswapalgorithmforminimization.Figs.18band18cshowthehorizontalandverticalmotionsdetectedwithourswapalgorithm.Noticethatthecathasbeenaccuratelylocalized.Eventhetailandpartsofthelegsareclearlyseparatedfromthebackgroundmotion.Therunningtimewas24secondstoconvergence.8.6ImageRestorationInthissection,weillustratetheimportanceofdiscontinuitypreservingenergyfunctionsonthetaskofimagerestoration.Fig.19showsimageconsistingofseveralregionswithconstantintensitiesafteritwascorruptedbyFig.19bshowsourimagerestorationresultsforthetruncatedabsolutedifferencemodelwhichisdiscontinuitypreserving.Sinceitisametric,weusedtheexpansionalgorithm.Forcomparison,Fig.19cshowstheresultfortheabsolutedifference,whichisnotdiscontinuitypreserving.Fortheabsolutedifferencemodel,wecanfindtheexactsolutionusingthegraph-cutmethodin[37],[24],[11].Forbothmodels,wechoseparameterswhichminimizetheaverageabsoluteerrorfromtheoriginalimageintensities.Theseaverageerrorswere0.34forthetruncatedand1.8fortheabsolutedifferencemodel,andtherunningtimeswere38and237seconds,respectively.TheresultsinFigs.19band19cwerehistogramequalizedtorevealoversmoothinginFig.19c,whichdoesnothappeninFig.19b.Similaroversmoothingfortheabsolutedifferencemodeloccursinstereo,see[43],[7].1236IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER2001 Fig.14.Tableoferrorsfortheexpansionalgorithmfordifferentvaluesoftheregularizationparameter Fig.15.Dependenceoftherunningtimeandaccuracyondifferentnumberoflabels(disparities)fortheexpansionalgorithm.Errorpercentagesaregivenatconvergence. Fig.16.Treestereopair.(a)Leftimage:256x233,29labels.(b)Piecewiseconstantmodel.(c)Piecewisesmoothmodel. Weconsiderawideclassofenergyfunctionswithvariousdiscontinuitypreservingsmoothnessconstraints.WhileitisNP-hardtocomputetheexactminimum,wedevelopedtwoalgorithmsbasedongraphcutsthatefficientlyfindalocalminimumwithrespecttotwolargemoves,namely,movesandmoves.Ouralgorithmfindsalabelingwithinaknownfactoroftheglobalminimum,whileouralgorithmhandlesmoregeneralenergyfunctions.Empirically,ouralgorithmsper-formswellonavarietyofcomputervisionproblemssuchasimagerestoration,stereo,andmotion.Webelievethatcombinatorialoptimizationtechniques,suchasgraphcuts,willprovetobepowerfultoolsforsolvingmanycomputervisionproblems.INIMIZINGTHEInSection7,weshowedthattheproblemofminimizingtheenergyin(18)overallpossiblelabelingscanbesolvedbycomputingaminimummultiwaycutonacertaingraph.Now,wemakethereductionintheoppositedirection.Letdenotetheenergyin(18).ForanarbitraryfixedgraphGˆhV,wewillconstructaninstanceofminimizingwheretheoptimallabelingdeterminesaminimummultiwaycuton.Thiswillprovethatapolynomial-timemethodforfindingwouldprovideapolynomial-timealgorithmforfindingtheminimumcostmultiwaycut,whichisknowntobeNP-hard[13].ThisNP-hardnessproofisbasedonaconstructionduetoJonKleinberg.Theenergyminimizationproblemweaddresstakesasinputasetofpixels,aneighborhoodrelation,anda BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS Fig.17.Flowergardensequence.(a)Firstimage,352x240,8labels.(b)Horizontalmovement. Fig.18.Movingcat.(a)Firstimage:256x223,40labels.(b)Horizontalmovement.(c)Verticalmovement. Fig.19.Imagerestoration.(a)Noisyimage.(b)Truncatedabsolutedifferencemodel.(c)Absolutedifferencemodel.Theresultsin(b)and(c)arehistogramequalizedtorevealoversmoothingin(c),whichdoesnothappenin(b). labelset,aswellasasetofweightsandafunction.Theproblemistofindthelabelingthatminimizestheenergygivenin(18).GˆhVbeanarbitraryweightedgraphwithterminalverticesandedgeweightsWewilldotheenergyminimizationusingPˆVNˆE.Thelabelsetwillbe.Letbeaconstantsuchthat�KE;forexample,wecantobethesumofall.Ourfunction;ifisaterminalvertex,Foranonterminalvertex,wesetforall,whichmeansalllabelsareequallygood.Wedefinealabelingtobeifthesetofpixelslabeledformsaconnectedcomponentthatincludes.Feasiblelabelingsobviouslycorrespondone-to-onewithmultiwaycuts.TheoremA.1.Thelabelingisfeasible,andthecostofafeasiblelabelingisthecostofthecorrespondingmultiwaycut.Toprovethatisfeasible,supposethattherewereaofpixelsthatwhichwerenotpartofthecomponentcontaining.Wecouldthenobtainalabelingwithlowerenergybyswitchingthissettothelabelofsomepixelontheboundaryof.Theenergyofafeasible†6ˆ,whichisthecostofthemultiwaycutcorrespondingtoThisshowsthatminimizingthePottsmodelenergyonanarbitraryisintractable.Itispossibletoextendthisprooftothecasewhenisaplanargrid,see[43].TheauthorswouldliketothankJ.Kleinberg,D.Shmoys,andE.Tardosforprovidingimportantinputonthecontentofthepaper.ThisresearchhasbeensupportedbyDARPAundercontractDAAL01-97-K-0104,bytheUSNationalScienceFoundationawardsCDA-9703470andIIS-9900115,andbyagrantfromMicrosoft.MostofthisworkwasdonewhileYuriBoykovandOlgaVekslerwereatCornellUniversity.[1]R.K.Ahuja,T.L.Magnanti,andJ.B.Orlin,NetworkFlows:Theory,Algorithms,andApplications.PrenticeHall,1993.1993.A.Amini,T.Weymouth,andR.Jain,ªUsingDynamicProgram-mingforSolvingVariationalProblemsinVision,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.12,no.9,pp.855-867,Sept.1990.1990.S.A.BarkerandP.J.W.Rayner,ªUnsupervisedImageSegmenta-Proc.IEEEInt'lConf.Acoustics,SpeechandSignalProcessing,vol.5,pp.2757-2760,1998.1998.J.Besag,ªOntheStatisticalAnalysisofDirtyPictures,º(withdiscussion),J.RoyalStatisticalSoc.,SeriesB,vol.48,no.3,pp.259-302,1986.1986.S.BirchfieldandC.Tomasi,ªDepthDiscontinuitiesbyPixel-to-PixelStereo,ºInt'lJ.ComputerVision,vol.35,no.3,pp.1-25,Dec.Dec.S.BirchfieldandC.Tomasi,ªAPixelDissimilarityMeasurethatIsInsensitivetoImageSampling,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.20,no.4,pp.401-406,Apr.1998.1998.S.Birchfield,ªDepthandMotionDiscontinuities,ºPhDthesis,StanfordUniv.,June1999.Availablefromhttp://vision.stanford.edu/~birch/publications/..A.BlakeandA.Zisserman,VisualReconstruction.MITPress,1987.1987.A.Blake,ªComparisonoftheEfficiencyofDeterministicandStochasticAlgorithmsforVisualReconstruction,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.11,no.1,pp.2-12,Jan.Jan.Y.BoykovandV.Kolmogorov,ªAnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision,ºProc.Int'lWorkshopEnergyMinimizationMethodsinComputerVisionandPatternRecognition,pp.359-374,Sept.2001.2001.Y.Boykov,O.Veksler,andR.Zabih,ªMarkovRandomFieldswithEfficientApproximations,ºProc.IEEEConf.ComputerVisionandPatternRecognition,pp.648-655,1998.1998.P.B.ChouandC.M.Brown,ªTheTheoryandPracticeofBayesianImageLabeling,ºInt'lJ.ComputerVision,vol.4,no.3,pp.185-210,185-210,E.Dahlhaus,D.S.Johnson,C.H.Papadimitriou,P.D.Seymour,andM.Yannakakis,ªTheComplexityofMultiwayCuts,ºSymp.TheoryofComputing,pp.241-251,1992.1992.P.Ferrari,A.Frigessi,andP.deSa,ªFastApproximateMaximumAPosterioriRestorationofMulticolourImages,ºJ.RoyalStatisticalSoc.,SeriesB,vol.57,no.3,pp.485-500,1995.1995.L.FordandD.Fulkerson,FlowsinNetworks.PrincetonUniv.Press,1962.1962.Y.Gdalyahu,D.Weinshall,andM.Werman.,ªSelf-OrganizationinVision:StochasticClusteringforImageSegmentation,Percep-tualGrouping,andImageDatabaseOrganization,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.23,no.10,pp.1053-1074,Oct.2001.2001.D.GeigerandA.Yuille,ªACommonFrameworkforImageSegmentation,ºInt'lJ.ComputerVision,vol.6,no.3,pp.227-243,227-243,D.Geman,S.Geman,C.Graffigne,andP.Dong,ªBoundaryDetectionbyConstrainedOptimization,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.12,no.7,pp.609-628,JulyJulyS.GemanandD.Geman,ªStochasticRelaxation,GibbsDistribu-tions,andtheBayesianRestorationofImages,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.6,pp.721-741,1984.1984.D.Greig,B.Porteous,andA.Seheult,ªExactMaximumAPosterioriEstimationforBinaryImages,ºJ.RoyalStatisticalSoc.,SeriesB,vol.51,no.2,pp.271-279,1989.1989.W.E.L.GrimsonandT.Pavlidis,ªDiscontinuityDetectionforVisualSurfaceReconstruction,ºComputerVision,GraphicsandImageProcessing,vol.30,pp.316-330,1985.1985.B.K.P.HornandB.Schunk,ªDeterminingOpticalFlow,ºvol.17,pp.185-203,1981.1981.R.HummelandS.Zucker,ªOntheFoundationsofRelaxationLabelingProcesses,ºIEEETrans.PatternAnalysisandMachinevol.5,pp.267-287,1983.1983.H.IshikawaandD.Geiger,ªOcclusions,Discontinuities,andEpipolarLinesinStereo,ºProc.EuropeanConf.ComputerVision,pp.232-248,1998.1998.H.Ishikawa,ªGlobalOptimizationUsingEmbeddedGraphs,ºPhDthesis,NewYorkUniv.,May2000.Availablefromhttp://cs1.cs.nyu.edu/phd_students/ishikawa/index.html.ml.M.Kass,A.Witkin,andD.Terzopoulos,ªSnakes:ActiveContourInt'lJ.ComputerVision,vol.1,no.4,pp.321-331,1987.1987.J.KleinbergandE.Tardos,ªApproximationAlgorithmsforClassificationProblemswithPairwiseRelationships:MetricLabelingandMarkovRandomFields,ºProc.IEEESymp.Founda-tionsofComputerScience,pp.14-24,1999.1999.V.KolmogorovandR.Zabih,ªComputingVisualCorrespon-dencewithOcclusionsviaGraphCuts,ºProc.Int'lConf.Computervol.II,pp.508-515,2001.2001.D.LeeandT.Pavlidis,ªOneDimensionalRegularizationwithDiscontinuities,ºIEEETrans.PatternAnalysisandMachineIntelli-vol.10,no.6,pp.822-829,Nov.1988.1988.S.Li,MarkovRandomFieldModelinginComputerVision.Verlag,1995.1995.G.Parisi,StatisticalFieldTheory.Reading,Mass.:Addison-Wesley,esley,M.Pelillo,ªTheDynamicsofNonlinearRelaxationLabelingJ.Math.ImagingandVision,vol.7,pp.309-323,1997.1238IEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.23,NO.11,NOVEMBER2001 T.Poggio,E.Gamble,andJ.Little,ªParallelIntegrationofVisionModules,ºvol.242,pp.436-440,Oct.1988.SeealsoE.GambleandT.Poggio,MITAIMemo970.970.T.Poggio,V.Torre,andC.Koch,ªComputationalVisionandRegularizationTheory,ºvol.317,pp.314-319,1985.1985.R.Potts,ªSomeGeneralizedOrder-DisorderTransformation,ºProc.CambridgePhilosophicalSoc.,vol.48,pp.106-109,1952.1952.A.Rosenfeld,R.A.Hummel,andS.W.Zucker,ªSceneLabelingbyRelaxationOperations,ºIEEETrans.Systems,Man,andCybernetics,vol.6,no.6,pp.420-433,June1976.1976.S.RoyandI.Cox,ªAMaximum-FlowFormulationoftheStereoCorrespondenceProblem,ºProc.Int'lConf.ComputerVision,pp.492-499,1998.1998.J.ShiandJ.Malik,ªNormalizedCutsandImageSegmentation,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.22,no.8,pp.888-905,Aug.2000.2000.R.H.SwendsonandJ.Wang,ªNonuniversalCriticalDynamicsinMonteCarloSimulations,ºPhysicalRev.Letters,vol.58,no.2,pp.86-88,1987.1987.R.SzeliskiandR.Zabih,ªAnExperimentalComparisonofStereoAlgorithms,ºVisionAlgorithms:TheoryandPractice,B.Triggs,A.Zisserman,andR.Szeliski,eds.,pp.1-19,Sept.1999.1999.R.S.Szeliski,ªBayesianModelingofUncertaintyinLow-LevelInt'lJ.ComputerVision,vol.5,no.3,pp.271-302,Dec.Dec.D.Terzopoulos,ªRegularizationofInverseVisualProblemsInvolvingDiscontinuities,ºIEEETrans.PatternAnalysisandMachineIntelligence,vol.8,no.4,pp.413-424,1986.1986.O.Veksler,ªEfficientGraph-BasedEnergyMinimizationMethodsinComputerVision,ºPhDthesis,CornellUniv.,July1999.Availablefromwww.neci.nj.nec.com/homepages/olga.ga.O.Veksler,ªImageSegmentationbyNestedCuts,ºProc.IEEEConf.ComputerVisionandPatternRecognition,vol.1,pp.339-344,339-344,Y.WeissandE.H.Adelson,ªAUnifiedMixtureFrameworkforMotionSegmentation:IncorporatingSpatialCoherenceandEstimatingtheNumberofModels,ºProc.IEEEConf.ComputerVisionandPatternRecognition,pp.321-326,1996.1996.G.Winkler,ImageAnalysis,RandomFieldsandDynamicMonteCarloMethods.Springer-Verlag,1995.1995.Z.WuandR.Leahy,ªAnOptimalGraphTheoreticApproachtoDataClustering:TheoryandItsApplicationtoImageSegmenta-IEEETrans.PatternAnalysisandMachineIntelligence,vol.15,no.11,pp.1101-1113,Nov.1993.YuriBoykovreceivedtheBSdegreeinappliedmathematicsandinformaticsfromtheMoscowInstituteofPhysicsandTechnologyin1992andtheMSandPhDdegreesinoperationsresearchfromCornellUniversityin1996.HedidhispostdoctoralresearchintheComputerScienceDepartmentatCornellUniversitywheremostoftheworkpresentedinthispaperwasdone.Since1999,hehasbeenamemberoftechnicalstaffatSiemensCorp.ResearchinPrinceton,NewJersey.Hiscurrentresearchinterestsareinoptimization-basedtechniquesforimageprocessing,N-Ddatasegmentation,probabilisticmodels,objectrecognition,graphalgorithms,andmedicalapplications.HeisamemberoftheIEEE.OlgaVekslerreceivedtheBSdegreeinmathematicsandcomputersciencefromNewYorkUniversityin1995andtheMSandPhDdegreesfromCornellUniversityin1999.SheiscurrentlyapostdoctoralresearchassociateatNECResearchInstitute.Herresearchinterestsareenergyminimizationmethods,graphalgo-rithms,stereocorrespondence,motion,cluster-ing,andimagesegmentation.SheisamemberoftheIEEE.RaminZabihattendedtheMassachusettsInstituteofTechnologyasanundergraduate,wherehereceivedBSdegreesinmathematicsandcomputerscienceandtheMScdegreeinelectricalengineeringandcomputerscience.AfterearningthePhDdegreeincomputersciencefromStanfordin1994,hejoinedthefacultyatCornell,whereheiscurrentlyanassociateprofessorofcomputerscience.In2001,hereceivedajointappointmentasanassociateprofessorofradiologyatCornellMedicalSchool.Hisresearchinterestslieinearlyvisionandinapplications,especiallyinmedicine.Hehasalsoservedonnumerousprogramcommittees,includingtheIEEEConferenceonComputerVisionandPatternRecognition(CVPR)in1997,2000,and2001andtheInternationalConferenceonComputerVision(ICCV)in1999and2001.HeorganizedtheIEEEWorkshoponGraphAlgorithmsinComputerVision,heldinconjunctionwithICCVin1999,andservedasguesteditorofaspecialissueofIEEETransactionsonPatternAnalysisandMachineIntelligenceonthesametopic.HeisamemberoftheIEEE.Formoreinformationonthisoranyothercomputingtopic,pleasevisitourDigitalLibraryathttp://computer.org/publications/dlib. BOYKOVETAL.:FASTAPPROXIMATEENERGYMINIMIZATIONVIAGRAPHCUTS