Fast Approximately Optimal Solutions for Single and Dynam ic MRFs Nikos Komodaki - PDF document

Fast,ApproximatelyOptimalSolutionsforSingleandDynamicMRFsNikosKomodakis,GeorgiosTziritasUniversityofCrete,ComputerScienceDepartmentfkomod,tziritasg@csd.uoc.grNikosParagiosMAS,EcoleCentraledeParisnikos.paragios@ecp.frAbstractAnewefcientMRFoptimizationalgorithm,calledFast-PD,isproposed,whichgeneralizes-expansion.Oneofitsmainadvantagesisthatitoffersasubstantialspeedupoverthatmethod,e.g.itcanbeatleast3-9timesfasterthan-expansion.ItsefciencyisaresultofthefactthatFast-PDexploitsinformationcomingnotonlyfromtheorig-inalMRFproblem,butalsofromadualproblem.Further-more,besidesstaticMRFs,itcanalsobeusedforboost-ingtheperformanceofdynamicMRFs,i.e.MRFsvaryingovertime.Ontopofthat,Fast-PDmakesnocompromiseabouttheoptimalityofitssolutions:itcancomputeexactlythesameansweras-expansion,but,unlikethatmethod,itcanalsoguaranteeanalmostoptimalsolutionforamuchwiderclassofNP-hardMRFproblems.ResultsonstaticanddynamicMRFsdemonstratethealgorithm'sefciencyandpower.E.g.,Fast-PDhasbeenabletocomputedispar-ityforstereoscopicsequencesinrealtime,withtheresultingdisparitycoincidingwiththatof-expansion.1.IntroductionDiscreteMRFsareubiquitousincomputervision,andthusoptimizingthemisaproblemoffundamentalimpor-tance.Accordingtoit,givenaweightedgraphG(withnodesV,edgesEandweightswpq),oneseekstoassignalabelxp(fromadiscretesetoflabelsL)toeachp2V,sothatthefollowingcostisminimized:Xp2Vcp(xp)+X(p;q)2Ewpqd(xp;xq):(1)Here,cp(
),d(
;
)determinethesingletonandpairwiseMRFpotentialfunctionsrespectively.Uptonow,graph-cutbasedmethods,like-expansion[ 3 ],havebeenveryeffectiveinMRFoptimization,generat-ingsolutionswithgoodoptimalityproperties[ 8 ].However,besidessolutions'optimality,anotherimportantissueisthatofcomputationalefciency.Infact,thisissuehasrecentlybeenlookedatforthespecialcaseofdynamicMRFs[ 5 , 4 ],i.e.MRFsvaryingovertime.Thus,tryingtoconcentrateonbothoftheseissueshere,weraisethefollowingquestions:cantherebeagraph-cutbasedmethod,whichwillbemoreefcient,butequally(orevenmore)powerful,than-expansion,forthecaseofsingleMRFs?Furthermore, ThisworkwaspartiallysupportedfromtheFrenchANR-BlancgrantSURF(2005-2008)andPlaton(2006-2007).canthatmethodalsoofferacomputationaladvantageforthecaseofdynamicMRFs?Withrespecttothequestionsraisedabove,thisworkmakesthefollowingcontributions.EfciencyforsingleMRFs:-expansionworksbysolvingaseriesofmax-owproblems.Itsefciencyisthuslargelydeterminedfromtheefciencyofthesemax-owproblems,which,inturn,dependsonthenumberofaugmentingpathspermax-ow.Here,webuilduponrecentworkof[ 6 ],andproposeanewprimal-dualMRFoptimizationmethod,calledFast-PD.Thismethod,like[ 6 ]or-expansion,alsoendsupsolvingamax-owproblemforaseriesofgraphs.However,unlikethesetechniques,thegraphsconstructedbyFast-PDensurethatthenumberofaugmentationspermax-owdecreasesdramaticallyovertime,thusboostingtheefciencyofMRFinference.Toshowthis,weproveageneralizedrelationshipbetweenthenumberofaugmentationsandtheso-calledprimal-dualgapassociatedwiththeoriginalMRFproblemanditsdual.Furthermore,tofullyexploittheaboveproperty,2newex-tensionsarealsoproposed:anadaptedmax-owalgorithm,aswellasanincrementalgraphconstructionmethod.Optimalityproperties:Despiteitsefciency,ourmethodalsomakesnocompromiseregardingtheoptimalityofitssolutions.So,ifd(
;
)isametric,Fast-PDisaspow-erfulas-expansion,i.e.itcomputesexactlythesamesolu-tion,butwithasubstantialspeedup.Moreover,itappliestoamuchwiderclassofMRFs 1 ,e.g.evenwithanon-metricd(
;
),whilestillguaranteeinganalmostoptimalsolution.EfciencyfordynamicMRFs:Furthermore,ourmethodcanalsobeusedforboostingtheefciencyofdynamicMRFs(introducedtocomputervisionin[ 5 ]).Twoworkshavebeenproposedinthisregardrecently[ 5 , 4 ].ThesemethodscanbeappliedtodynamicMRFsthatarebi-naryorhaveconvexpriors.Onthecontrary,Fast-PDnatu-rallyhandlesamuchwiderclassofdynamicMRFs,andcandosobyalsoexploitinginformationfromaproblem,whichisdualtotheoriginalMRFproblem.Fast-PDcanthusbethoughtofasageneralizationofprevioustechniques.Therestofthepaperisorganizedasfollows.Insec. 2 ,webrieyreviewtheworkof[ 6 ]aboutusingtheprimal-dualschemaforMRFoptimization.TheFast-PDalgorithmisthendescribedinsec. 3 .Itsefciencyforoptimizing 1Fast-PDrequiresonlyd(a;b)0;d(a;b)=0,a=b 1:[x;y] INIT DUALS PRIMALS();xold x2:foreachlabelcinLdo3:y PREEDIT DUALS(c;x;y);4:[x0;y0] UPDATE DUALS PRIMALS(c;x;y);5:y0 POSTEDIT DUALS(c;x0;y0); 6:x x0;y y0;7:endfor8:ifx=xoldthen9:xold x;goto2;10:endif Fig.1:TheprimaldualschemaforMRFoptimization.singleMRFsisfurtheranalyzedinsec. 4 ,whererelatedresultsandsomeimportantextensionsofFast-PDarepresentedaswell.Sec. 5 explainshowFast-PDcanboosttheperformanceofdynamicMRFs,andalsocontainsmoreexperimentalresults.Finally,weconcludeinsection 6 .2.Primal-dualMRFoptimizationalgorithmsInthissection,wereviewverybrieytheworkof[ 6 ].Considertheprimal-dualpairoflinearprograms,givenby:PRIMAL:mincTxDUAL:maxbTys.t.Ax=b;x0s.t.ATycOneseeksanoptimalprimalsolution,withtheextracon-straintofxbeingintegral.ThismakesforanNP-hardprob-lem,andsoonecanonlyhopeforndinganapproximatesolution.Tothisend,thefollowingschemacanbeused:Theorem1(Primal-Dualschema).Keepgeneratingpairsofintegral-primal,dualsolutions(xk;yk),untiltheele-mentsofthelastpair,sayx;y,arebothfeasibleandhavecoststhatarecloseenough,e.g.theirratioisfapp:cTxfapp
bTy(2)Thenxisguaranteedtobeanfapp-approximatesolutiontotheoptimalintegralsolutionx,i.e.cTxfapp
cTx.Theaboveschemahasbeenusedin[ 6 ],forderivingap-proximationalgorithmsforaverywideclassofMRFs.Tothisend,MRFoptimizationwasrstcastasanequivalentintegerprogramandthen,asrequiredbytheprimal-dualschema,itslinearprogrammingrelaxationanditsdualwerederived.BasedontheseLPs,theauthorsthenshowthat,forTheorem 1 tobetruewithfapp=2dmax dmin 2 ,itsufcesthatthenext(so-calledrelaxedcomplementaryslackness)con-ditionsholdtruefortheresultingprimalanddualvariables:hp(xp)=mina2Lhp(a);8p2V(3)ypq(xp)+yqp(xq)=wpqd(xp;xq);8pq2E(4)ypq(a)+yqp(b)2wpqdmax;8pq2E;a2L;b2L(5)Intheseformulas,theprimalvariables,denotedbyx=fxpgp2V,determinethelabelsassignedtonodes(calledactivelabelshereafter),e.g.xpistheactivelabelofnodep.Whereas,thedualvariablesaredividedintobalanceandheightvariables.Thereexist2balancevariablesypq(a);yqp(a)peredge(p;q)andlabela,aswellas1heightvariablehp(a)pernodepandlabela.Variablesypq(a);yqp(a)arealsocalledconjugateand,forthedualsolutiontobefeasible,thesemustbesetoppositetoeachother,i.e.:yqp(
)ypq(
).Furthermore,theheightvariablesarealwaysdenedintermsofthebalancevariablesasfollows: 2dmaxmaxa=bd(a;b);dminmina=bd(a;b)hp(
)cp(
)+Xq:qp2Eypq(
):(6)Notethat,dueto( 6 ),onlythevectory(ofallbalancevari-ables)isneededforspecifyingadualsolution.Inaddition,forsimplifyingconditions( 4 ),( 5 ),onecanalsodene:loadpq(a;b)ypq(a)+yqp(b):(7)Theprimal-dualvariablesareiterativelyupdateduntilallconditions( 3 )-( 5 )holdtrue.Thebasicstructureofaprimal-dualalgorithmcanbeseeninFig. 1 .Duringaninnerc-iteration(lines3-6inFig. 1 ),alabelcisselectedandanewprimal-dualpairofsolutions(x0;y0)isgeneratedbasedonthecurrentpair(x;y).Tothisend,amongallbal-ancevariablesypq(:),onlythebalancevariablesofc-labels(i.e.ypq(c))areupdatedduringac-iteration.jLjsuchitera-tions(i.e.onec-iterationperlabelcinL)makeupanouteriteration(lines2-7inFig. 1 ),andthealgorithmterminatesifnochangeoflabeltakesplaceatthecurrentouteriteration.Duringaninneriteration,themainupdateoftheprimalanddualvariablestakesplaceinsideUP-DATE DUALS PRIMALS,and(asshownin[ 6 ])thisupdatereducestosolvingamax-owprobleminanappropriategraphGc.Furthermore,theroutinesPREEDIT DUALSandPOSTEDIT DUALSsimplyapplycorrectionstothedualvariablesbeforeandafterthismainupdate,i.e.tovariablesyandy0respectively.Also,forsimplicity'ssake,notethatwewillhereafterrefertoonlyoneofthemethodsderivedin[ 6 ],andthiswillbetheso-calledPD3amethod.3.Fastprimal-dualMRFoptimizationThecomplexityofthePD3aprimal-dualmethodlargelydependsonthecomplexityofallmax-owinstances(oneinstanceperinner-iteration),which,inturn,dependsonthenumberofaugmentationspermax-ow.So,fordesigningfasterprimal-dualalgorithms,werstneedtounderstandhowthegraphGc,associatedwiththemax-owproblematac-iterationofPD3a,isconstructed.Tothisend,wealsohavetorecallthefollowingintuitiveinterpretationofthedualvariables[ 6 ]:foreachnodep,aseparatecopyofallla-belsinLisconsidered,andalltheselabelsarerepresentedasballs,whichoatatcertainheightsrelativetoareferenceplane.Theroleoftheheightvariableshp(
)isthentodeter-minetheballs'height(seeFigure 2 (a)).E.g.theheightoflabelaatnodepisgivenbyhp(a).Also,expressionslike“labelaatpisbelow/abovelabelb”implyhp(a)7hp(b).Furthermore,ballsarenotstatic,butmaymoveinpairsthroughupdatingpairsofconjugatebalancevariables.E.g.,inFigure 2 (a),labelcatpisraisedby+(duetoadding+toypq(c)),andsolabelcatqhastomovedownby(duetoaddingtoyqp(c)sothatconditionypq(c)=yqp(c)stillholds).Therefore,theroleofbalancevariablesistoraiseorlowerlabels.Inparticular,thevalueofbalancevari-ableypq(a)representsthepartialraiseoflabelaatpduetoedgepq,while(by( 6 ))thetotalraiseofaatpequalsthesumofpartialraisesfromalledgesofGincidenttop. +
-
cc p q w pq hp(xp)hq(xq)hp(c)hq(c) cap cap xpc hp(xp)hp(c)hp(c)hp(xp) fp
p p xpc fp (a) (b) (c) fp cap
p fp
cap p =a=a Fig.2:(a)Dualvariables'visualizationforasimpleMRFwith2nodesfp;qgand2labelsfa;cg.Acopyoflabelsfa;cgexistsforeverynode,andalltheselabelsarerepresentedbyballsoatingatcertainheights.Theroleoftheheightvariablesh(
)istospecifyexactlytheseheights.Furthermore,ballsarenotstatic,butmaymove(i.e.changetheirheights)inpairsbyupdatingconjugatebalancevariables.E.g.,here,ballcatpispulledupby+(duetoincreasingypq(c)by+)andsoballcatqmovesdownby(duetodecreasingyqp(c)by).Activelabelsaredrawnwithathickercircle.(b)Iflabelcatpisbelowx,then(dueto( 3 ))wewantlabelctoraiseandreachx.Wethusconnectnodeptothesourceswithanedgesp(i.e.pisans-linkednode),andowfsrepresentsthetotalraiseofc(wealsosetcaps=h(x)h(c)).(c)Iflabelcatpisabovex,then(dueto( 3 ))wewantlabelcnottogobelowx.Wethusconnectnodeptothesinktwithedgept(i.e.pisat-linkednode),andowftrepresentsthetotaldecreaseintheheightofc(wealsosetcapt=h(c)h(x)sothatcwillstillremainabovex).Hence,PD3atriestoiterativelymovelabelsupordown,untilallconditions( 3 )-( 5 )holdtrue.Tothisend,itusesthefollowingstrategy:itensuresthatconditions( 4 )-( 5 )holdateachiteration(whichisalwayseasytodo)andisjustleftwiththemaintaskofmakingthelabels'heightssatisfycon-dition( 3 )aswellintheend(whichisthemostdifcultpart,requiringeachactivelabelxptobethelowestlabelforp).Forthispurpose,labelsaremovedingroups.Inparticular,duringac-iteration,onlythec-labelsareallowedtomove.Furthermore,itwasshownin[ 6 ]thatthemovementofallc-labels(i.e.theupdateofdualvariablesypq(c)andhp(c)forallp;q)canbesimulatedbypushingthemaximumowthroughadirectedgraphGc(whichisconstructedbasedonthecurrentprimal-dualpair(x;y)atac-iteration).ThenodesofGcconsistofallnodesofgraphG(theinternalnodes),plus2externalnodes,thesourcesandthesinkt.Inaddition,allnodesofGcareconnectedbytwotypesofedges:interiorandexterioredges.Interioredgescomeinpairspq,qp(withonesuchpairforevery2neighborsp;qinG),andareresponsibleforupdatingthebalancevariables.Inparticular,theowsfpq=fqpoftheseedgesrepresenttheincrease/decreaseofbalancevariableypq(c),i.e.y0pq(c)=ypq(c)+fpqfqp.Also,asweshallsee,thecapacitiesofinterioredgesareusedtogetherwithPREEDIT DUALS,POSTEDIT DUALStoimposeconditions( 4 ),( 5 ).Butfornow,inordertounderstandhowtomakeafasterprimal-dualmethod,itistheexterioredges(whichareinchargeoftheupdateofheightvariables),aswellastheircapacities(whichareusedforimposingtheremainingcondition( 3 )),thatareofinteresttous.Thereasonisthattheseedgesdeterminethenumberofs-linkednodes,which,inturn,affectsthenumberofaugmentingpathspermax-ow.Inparticular,eachinternalnodeconnectstoeitherthesources(i.e.itisans-linkednode)ortothesinkt(i.e.itisat-linkednode)throughoneoftheseexterioredges,andthisisdone(withthegoalofensuring( 3 ))asfollows:iflabelcatpisabovexpduringac-iteration(i.e.hp(c)�hp(xp)),thenlabelcshouldnotgobelowxp,orelse( 3 )willbeviolatedforp.Nodepthusconnectstotthroughdirectededgept(i.e.pbecomest-linked),andowfptrepresentsthetotaldecreaseintheheightofcafterUPDATE DUALS PRIMALS,i.e.h0p(c)=hp(c)fpt(seeFig. 2 (c)).Furthermore,thecapacityofptissetsothatlabelcwillstillremainabovexp,i.e.cappt=hp(c)hp(xp).Ontheotherhand,iflabelcatpisbelowactivelabelxp(i.e.hp(c)hp(xp)),then(dueto( 3 ))labelcshouldraisesoastoreachxp,andsopconnectstosthroughedgesp(i.e.pbecomess-linked),whileowfsprepresentsthetotalraiseofballc,i.e.h0p(c)=hp(c)+fsp(seeFig. 2 (b)).Inthiscase,wealsosetcapsp=hp(xp)hp(c).Thisway,bypushingowthroughtheexterioredgesofGc,allc-labelsthatarestrictlybelowanactivelabeltrytoraiseandreachthatlabelduringUPDATE DU-ALS PRIMALS 3 .Notonlythat,butthefewerarethec-labelsbelowanactivelabel(i.e.thefewerarethes-linkednodes),thefewerwillbetheedgesconnectedtothesource,andthusthelesswillbethenumberofpossibleaugmentingpaths.Infact,thealgorithmterminateswhen,foranylabelc,therearenomorec-labelsstrictlybelowanactivelabel(i.e.nos-linkednodesexistandthusnoaugmentingpathsmaybefound),inwhichcasecondition( 3 )willnallyholdtrue,asdesired.Putanotherway,UPDATE DUALS PRIMALStriestopushc-labels(whichareatalowheight)up,sothatthenumberofs-linkednodesisreducedandthusfeweraugmentingpathsmaybepossibleforthenextiteration.However,althoughUPDATE DUALS PRIMALStriestoreducethenumberofs-linkednodes(bypushingthemaxi-mumamountofow),PREEDIT DUALSorPOSTEDIT DU-ALSveryoftenspoilthatprogress.Asweshallseelater,thisoccursbecause,inordertorestorecondition( 4 )(whichistheirmaingoal),theseroutinesareforcedtoapplycorrec-tionstothedualvariables(i.e.tothelabels'height).ThisisabstractlyillustratedinFigure 3 ,where,asaresultofpush-ingow,ac-labelinitiallymanagedtoreachanactivelabelxp,butitagaindroppedbelowxp,duetosomecorrectionappliedbytheseroutines.Infact,asonecanshow,theonlypointwhereanews-linkednodecanbecreatedisduringeitherPREEDIT DUALSorPOSTEDIT DUALS. 3Equivalently,ifc-labelatpcannotraisehighenoughtoreachx,UPDATE DUALS PRIMALSthenassignsthatc-labelasthenewactivelabelofp(i.e.x0=c),thuseffectivelymakingtheactivelabelgodown.Thishelpscondition( 3 )tobecometrue,andformsthemainrationalebehindtheupdateoftheprimalvariablesxinUPDATE DUALS PRIMALS. c cap hp(xp)hp(c) p xpc hp(xp)hp(c) p xpc f
p cap hp(xp)hp(c) p xp



 \n
\r \n
\r\r

 \n 
 \n Fig.3:(a)Labelcatpisbelowx,andthuslabelcisallowedtoraiseitselfinordertoreachx.Thismeansthatpwillbeans-linkednodeofgraphGc,i.e.caps�0,andthusanon-zeroowfs(representingthetotalraiseoflabelc)maypassthroughedgesp.Therefore,inthiscase,edgespmaybecomepartofanaugmentingpathduringmax-ow.(b)AfterUPDATE DUALS PRIMALS,labelchasmanagedtoraisebyfsandreachx.Sinceitcannotgohigherthanthat,noowcanpassthroughedgesp,i.e.caps=0,andsonoaugmentingpathmaytraversethatedgethereafter.(c)However,duetosomecorrectionappliedtoc-label'sheight,labelchasdroppedbelowxoncemoreandphasbecomeans-linkednodeagain(i.e.caps�0).Edgespcanthusbepartofanaugmentingpathagain(asin(a)).Toxthisproblem,wewillredenePREEDIT DUALS,POSTEDIT DUALSsothattheycannowensurecondition( 4 )byusingjustaminimumamountofcorrectionsforthedualvariables,(e.g.bytouchingthesevariablesonlyrarely).Tothisend,however,UPDATE DUALS PRIMALSneedstobemodiedaswell.Theresultingalgorithm,calledFast-PD,carriesthefollowingmaindifferencesoverPD3aduringac-iteration(itspseudocodeappearsinFig. 4 ):-thenewPREEDIT DUALSmodiesapairypq(c);yqp(c)ofdualvariablesonlywhenabsolutelynecessary.So,whereasthepreviousversionmodiedthesevariables(therebychangingtheheightofac-label)wheneverc=xp,c=xq(whichcouldhappenextremelyoften),amodicationisnowappliedonlyifloadpq(c;xq)�wpqd(c;xq)orloadpq(xp;c)�wpqd(xp;c)(which,inpractice,happensmuchmorerarely).Inthiscase,amodicationisneeded(seecodeinFig. 4 ),becausetheaboveinequalitiesindicatethatcondition( 4 )willbeviolatedifeither(c;xq)or(xp;c)becomethenewactivelabelsforp;q.Onthecontrary,nomodicationisneededifthefollowinginequalitiesaretrue:loadpq(c;xq)wpqd(c;xq);loadpq(xp;c)wpqd(xp;c),becausethen,asweshallseebelow,thenewUP-DATE DUALS PRIMALScanalwaysrestore( 4 )(i.e.evenif(c;xq)or(xp;c)arethenextactivelabels-e.g.,see( 12 )).Infact,themodicationtoypq(c)thatisoccasionallyappliedbythenewPREEDIT DUALScanbeshowntobetheminimalcorrectionthatrestoresexactlytheaboveinequ-alities(assuming,ofcourse,thisrestorationispossible).-Similarly,thenewPOSTEDIT DUALSmodies 4 bal-ancevariablesy0pq(x0p)(withx0p=c)andy0qp(x0q)(withx0q=c)onlyiftheinequalityload0pq(x0p;x0q)�wpqd(x0p;x0q)holds,inwhichcasePOSTEDIT DUALSsimplyhasto 4WerecallthatPOSTEDIT DUALSmaymodifyonlydualsolutiony0.Forthatsolution,wedeneload0pq(a;b)y0pq(a)+y0qp(b),asin( 7 ). [x;y] INIT DUALS PRIMALS():x randomlabels;y 0;8pq;adjustypq(xp)oryqp(xq)sothatloadpq(xp;xq)=wpqd(xp;xq)y PREEDIT DUALS(c;x;y):8pq;ifloadpq(c;xq)�wpqd(c;xq)orloadpq(xp;c)�wpqd(xp;c)adjustypq(c)sothatloadpq(c;xq)=wpqd(c;xq)[x0;y0] UPDATE DUALS PRIMALS(c;x;y):x0 x;y0 y;ConstructGcandapplymax-flowtocomputeallflowsfsp=fpt;fpq8pq;y0pq(c) ypq(c)+fpqfqp8p;ifanunsaturatedpathfromstopexists;thenx0p cy0 POSTEDIT DUALS(c;x0;y0):fWedenoteload0pq(
;
)=y0pq(
)+y0qp(
)g8pq;ifload0pq(x0p;x0q)�wpqd(x0p;x0q)fThisimpliesx0p=corx0q=cgadjusty0pq(c)sothatload0pq(x0p;x0q)=wpqd(x0p;x0q) Fig.4:Fast-PD'spseudocode.reduceload0pq(x0p;x0q)forrestoring( 4 ).However,thisinequalitywillholdtrueveryrarely(e.g.forametricd(
;
),onemayshowthatitcanneverhold),andsoPOSTEDIT DU-ALSwillmodifyac-balancevariable(therebychangingtheheightofac-label)onlyinveryseldomoccasions.-But,toallowfortheabovechanges,wealsoneedtomodifytheconstructionofgraphGcinUPDATE DU-ALS PRIMALS.Inparticular,forc=xpandc=xq,theca-pacitiesofinterioredgespq;qpmustnowbesetasfollows: 5 cappq=wpqd(c;xq)loadpq(c;xq)+;(8)capqp=wpqd(xp;c)loadpq(xp;c)+;(9)where[x]+max(x;0).Besidesensuring( 5 )(bynotlet-tingthebalancevariablesincreasetoomuch),themainra-tionalebehindtheabovedenitionofinteriorcapacitiesistoalsoensurethat(aftermax-ow)condition( 4 )willbemetbymostpairs(p;q),evenif(c;xq)or(xp;c)arethenextlabelsassignedtothem(whichisagoodthing,sincewewillthusmanagetoavoidtheneedforacorrectionbyPOSTEDIT DUALSforallbutafewp;q).Forseeingthis,thecrucialthingtoobserveisthatif,say,(c;xq)arethenextlabelsforpandq,thencapacitycappqcanbeshowntorepresenttheincreaseofloadpq(c;xq)aftermax-ow,i.e.:load0pq(c;xq)=loadpq(c;xq)+cappq:(10)Hence,ifthefollowinginequalityistrueaswell:loadpq(c;xq)wpqd(c;xq);(11)thencondition( 4 )willdoremainvalidaftermax-ow,asthefollowingtrivialderivationshows:load0pq(c;xq)( 10 );( 8 )=loadpq(c;xq)+[wpqd(c;xq)loadpq(c;xq)]+( 11 )=wpqd(c;xq)(12)ButthismeansthatacorrectionmayneedtobeappliedbyPOSTEDIT DUALSonlyforpairsp;qviolating( 11 )(beforemax-ow).However,suchpairstendtobeveryrareinprac-tice(e.g.,asonecanprove,nosuchpairsexistwhend(
;
)isametric),andthusveryfewcorrectionsneedtotakeplace.Fig. 5 summarizeshowFast-PDsetsthecapacitiesforalledgesofGc.Asalreadyexplained,theinteriorcapaci-ties(withthehelpofPREEDIT DUALS,POSTEDIT DUALS 5Ifc=xorc=xq,thencappq=capqp=0asbefore,i.e.asinPD3a. inafewcases)allowUPDATE DUALS PRIMALStoimposeconditions( 4 ),( 5 ),whiletheexteriorcapacitiesallowUP-DATE DUALS PRIMALStoimposecondition( 3 ).Asare-sult,thenexttheoremholds(see[ 1 ]foracompleteproof):Theorem2.Thelastprimal-dualpair(x;y)ofFast-PDsatises( 3 )-( 5 ),andsoxisanfapp-approximatesolution.Infact,Fast-PDmaintainsallgoodoptimalityproper-tiesofthePD3amethod.E.g.,forametricd(
;
),Fast-PDprovestobeaspowerfulas-expansion(see[ 1 ]):Theorem3.Ifd(
;
)isametric,thentheFast-PDalgo-rithmcomputesthebestc-expansionafteranyc-iteration.4.EfciencyofFast-PDforsingleMRFsBut,besideshavingallthesegoodoptimalityproperties,averyimportantadvantageofFast-PDoverallpreviousprimal-dualmethods,aswellas-expansion,isthatitprovestobemuchmoreefcientinpractice.Infact,thecomputationalefciencyforallmethodsofthiskindislargelydeterminedfromthetimetakenbyeachmax-owproblem,which,inturn,dependsonthenumberofaugmentingpathsthatneedtobecomputed.ForthecaseofFast-PD,thenumberofaugmentationsperinner-iterationdecreasesdramatically,asthealgorithmprogresses.E.g.Fast-PDhasbeenappliedtotheproblemofimagerestoration,andg. 7 containsarelatedresultaboutthedenoisingofacorrupted(withgaussiannoise)“pen-guin”image(256labelsandatruncatedquadraticdistanced(a;b)=min(jabj2;D)-whereD=200-hasbeenusedinthiscase).Also,g. 8(a) showsthecorrespondingnum-berofaugmentingpathsperouter-iteration(i.e.pergroupofjLjinner-iterations).Noticethat,forboth-expansion,aswellasPD3a,thisnumberremainsveryhigh(i.e.almostover2
106paths)throughoutalliterations.Onthecontrary,forthecaseofFast-PD,itdropstowardszeroveryquickly,e.g.only4905and7pathshadtobefoundduringthe8thandlastouter-iterationrespectively(obviously,asalsoshowninFig. 9(a) ,thisdirectlyaffectsthetotaltimeneededperouter-iteration).Infact,forthecaseofFast-PD,itisverytypicalthat,afterveryfewinner-iterations,nomorethan10or20augmentingpathsneedtobecomputedpermax-ow,whichreallybooststheperformanceinthiscase.ThispropertycanbeexplainedbythefactthatFast-PDmaintainsbothaprimal,aswellasadualsolutionthrough-outitsexecution.Fast-PDthenmanagestoeffectivelyusethedualsolutionsofpreviousinneriterations,soastore-ducethenumberofaugmentingpathsforthenextinner-iterations.Intuitively,whathappensisthatFast-PDulti-matelywantstoclosethegapbetweentheprimalandthe cappq=[wpqd(c,x)-loadpq(c,x)]capqp=[wpqd(x,c)-loadpq(x,c)] = c= ccappq= 0capqp= 0 cap=[h(x)-h(c)]cap=[h(c)-h(x)] interior capacities exterior capacities Fig.5:CapacitiesofgraphGc,assetbyFast-PD.


 
  
    


\n\r\r \n\r\r (a)High-levelviewoftheFast-PDalgorithm

    

 \n
\r
\r \r (b)High-levelviewofthe-expansionalgorithmFig.6:(a)Fast-PDgeneratespairsofprimal-dualsolutionsiter-atively,withthegoalofalwaysreducingtheprimal-dualgap(i.e.thegapbetweentheresultingprimalanddualcosts).But,forthecaseofFast-PD,thisgapcanbeviewedasaroughestimateforthenumberofaugmentations,andsothisnumberisforcedtoreduceovertimeaswell.(b)Onthecontrary,-expansionworksonlyintheprimaldomain(i.e.itisasifaxeddualcostisusedatthestartofeachnewiteration)andthustheprimal-dualgapcanneverbecomesmallenough.Therefore,nosignicantreductioninthenumberofaugmentationstakesplaceasthealgorithmprogresses.dualcost(seeTheorem 1 ),and,forthis,ititerativelygener-atesprimal-dualpairs,withthegoalofdecreasingthesizeofthisgap(seeFig. 6(a) ).But,forFast-PD,thegap'ssizecanbethoughtofas,roughlyspeaking,anupper-boundforthenumberofaugmentingpathsperinner-iteration.Since,furthermore,Fast-PDmanagestoreducethisgapatanytimethroughoutitsexecution,thenumberofaugmentingpathsisforcedtodecreaseovertimeaswell.Onthecontrary,amethodlike-expansion,thatworksonlyintheprimaldomain,ignoresdualsolutionscompletely.Itis,roughlyspeaking,asif-expansionisresettingthedualsolutiontozeroatthestartofeachinner-iteration,thuseffectivelyforgettingthatsolutionthereafter(seeFig. 6(b) ).Forthisreason,itfailstoreducetheprimal-dualgapandthusalsofailstoachieveareductioninpathaugmentationsovertime,i.e.acrossinner-iterations.ButthePD3aalgorithmaswellfailstomimicFast-PD'sbehavior(despitebeingaprimal-dualmethod).Asexplainedinsec. 3 ,thishappensbecause,inthiscase,PREEDIT DUALandPOSTEDIT DUALtemporarilydestroythegapjustbeforethestartofUPDATE DUALS PRIMALS,i.e.justbeforemax-owisabouttobegincomputingtheaugmentingpaths.(Note,ofcourse,thatthisdestructionisonlytemporary,andthegapisrestoredagainaftertheexecutionofUPDATE DUALS PRIMALS).Theabovementionedrelationshipbetweenprimal-dualgapandnumberofaugmentingpathsisformallydescribedinthenexttheorem(see[ 1 ]foracompleteproof):Theorem4.ForFast-PD,theprimal-dualgapatthecur-rentinner-iterationformsanapproximateupperboundforthenumberofaugmentingpathsateachiterationthereafter.Sketchofproof.Duringac-iteration,itcanbeshownthatdual-costPpmin(hp(c);hp(xp)),whereasprimal-cost=Pphp(xp),andsotheprimal-dualgapupper-boundsthefollowingquantity:Pp[hp(xp)hp(c)]+=Ppcapsp. Fig.7:Left:“Tsukuba”imageanditsdisparitybyFast-PD.Mid-dle:a“SRItree”imageandcorrespondingdisparitybyFast-PD.Right:noisy“penguin”imageanditsrestorationbyFast-PD.Butthisquantityobviouslyformsanupper-boundonthemaximumow,which,inturn,upper-boundsthenumberofaugmentations(assumingintegralows). Duetotheabovementionedproperty,thetimeperouter-iterationdecreasesdramaticallyovertime.ThishasbeenveriedexperimentallywithvirtuallyallproblemsthatFast-PDhasbeentestedon.E.g.Fast-PDhasbeenalsoappliedtotheproblemofstereomatching,andg. 7 containstheresultingdisparity(ofsize384288with16labels)forthewell-known“Tsukuba”stereopair,aswellastheresultingdisparity(ofsize256233with10labels)foranimagepairfromthewell-known“SRItree”sequence(inbothcases,atruncatedlineardistanced(a;b)=min(jabj;D)-withD=2andD=5-hasbeenused,whiletheweightswpqwereallowedtovarybasedontheimagegradientatp).Figures 9(b) , 9(c) containthecorrespondingrunningtimesperouteriteration.Noticehowmuchfastertheouter-iterationsofFast-PDbecomeasthealgorithmprogresses,e.g.thelastouter-iterationofFast-PD(forthe“SRI-tree”example)lastedlessthan1msec(since,asitturnsout,only4augmentingpathshadtobefoundduringthatiteration).Contrastthiswiththebehaviorofeitherthe-expansionorthePD3aalgorithm,whichbothrequireanalmostconstantamountoftimeperouter-iteration,e.g.thelastouter-iterationof-expansionneededmorethan0.4secstonish(i.e.itwasmorethan400timesslowerthanFast-PD'siteration!).Similarly,forthe“Tsukuba”example,-expansion'slastouter-iterationwasmorethan2000timesslowerthanFast-PD'siteration.Max-owalgorithmadaptation:However,forfullyexploitingthedecreasingnumberofpathaugmentationsandreducetherunningtime,wehadtoproperlyadaptthemax-owalgorithm.Tothisend,thecrucialthingtoobservewasthatthedecreasingnumberofaugmentationswasdirectlyrelatedtothedecreasingnumberofs-linkednodes,asalreadyexplainedinsec. 3 .E.g.g. 8(b) showshowthenumberofs-linkednodesvariesperouter-iterationforthe“penguin”example(withasimilarbehaviorbeingobservedfortheotherexamplesaswell).Ascanbeseen,thisnumberdecreasesdrasticallyovertime.Infact,as 1 4 7 10 13 16 19 22 0 0.5 1 1.5 2x 106 outer iterationNo. of augmentations PD3aa-expansionFast-PD (a) 1 4 7 10 13 16 19 22 0 0.5 1 1.5 2x 106 outer iteration No. of (b)Fig.8:(a)Numberofaugmentingpathsperouteriterationforthe“penguin”example(similarresultsholdfortheotherexamplesaswell).OnlyinthecaseofFast-PD,thisnumberdecreasesdramat-icallyovertime.(b)ThispropertyofFast-PDisdirectlyrelatedtothedecreasingnumberofs-linkednodesperouter-iteration(thisnumberisshownhereforthesameexampleasin(a)). 1 4 7 10 13 16 19 22 0 2 4 6 8 outer iterationtime (secs) PD3aa-expansionFast-PD (a)“penguin” 1 2 3 4 5 6 7 0 1 2 outer iterationtime (secs) PD3aa-expansionFast-PD (b)“Tsukuba” 1 2 3 4 5 6 0 0.1 0.2 0.3 0.4 outer iterationtime (secs) PD3aa-expansionFast-PD (c)“SRItree”
    \n\r\n  


    \r  

   (d)TotaltimesFig.9:Totaltimeperouteriterationforthe(a)“penguin”,(b)“Tsukuba”and(c)“SRItree”examples.(d)Totalrunningtimes.Forallexperimentsofthispaper,a1.6GHzlaptophasbeenused.impliedbycondition( 3 ),nos-linkednodeswillnallyexistuponthealgorithm'stermination.Anyaugmentation-basedmax-owalgorithmstrivingforcomputationalefciency,shouldcertainlyexploitthispropertywhentryingtoextractitsaugmentingpaths.Themostefcientofthesealgorithms[ 2 ]maintains2searchtreesforthefastextractionofthesepaths,asourceandasinktree.Here,thesourcetreewillstartgrowingbyexploringnon-saturatededgesthatareadjacenttos-linkednodes,whereasthesinktreewillgrowstartingfromallt-linkednodes.Ofcourse,thealgorithmterminateswhennoadjacentunsaturatededgescanbefoundanymore.However,inourcase,maintainingthesinktreeiscompletelyinefcientanddoesnotexploitthemuchsmallernumberofs-linkednodes.Wethusproposemaintainingonlythesourcetreeduringmax-ow,whichwillbeamuchcheaperthingtodohere(e.g.,inmanyinneriterations,therecanbefewerthan10s-linkednodes,butmanythousandsoft-linkednodes).Moreover,duetothesmallsizeofthesourcetree,detectingtheterminationofthemax-owprocedurecannowbedonealotfaster,i.e.with- 20 40 60 80 100 1 100 200 300 inner iteration suboptimality 1000 3000 5000 1 3000 6000 9000 inner iteration suboptimality Fig.10:Suboptimalityboundsperinneriteration(for“Tsukuba”and“penguin”).Theseboundsdropto1veryfast,meaningthatthecorrespondingsolutionshavebecomealmostoptimalveryearly.outhavingtofullyexpandthelargesinktree(whichisaverycostlyoperation),thusgivingasubstantialspeedup.Inadditiontothat,forefcientlybuildingthesourcetree,wekeeptrackofalls-linkednodesanddon'trecomputethemfromscratcheachtime.Inourcase,thistrackingcanbedonewithoutcost,since,asexplainedinsec. 3 ,ans-linkednodecanbecreatedonlyinsidethePREEDIT DUALSorthePOSTEDIT DUALSroutine,andthuscanbeeasilydetected.Theabovesimplestrategyhasbeenextremelyeffectiveforboostingtheperformanceofmax-ow,especiallywhenasmallnumberofaugmentationswereneeded.Incrementalgraphconstruction:Butbesidesthemax-owalgorithmadaptation,wemayalsomodifythewaygraphGcisconstructed.I.e.insteadofconstructingtheca-pacitatedgraphGcfromscratcheachtime,wealsoproposeanincrementalwayofsettingitscapacities.Thefollowinglemmaturnsouttobecrucialinthisregard:Lemma1.LetGc,Gcbethegraphsforthecurrentandpreviousc-iteration.Letalsop;qbe2neighboringMRFnodes.If,duringtheintervalfromtheprevioustothecur-rentc-iteration,nochangeoflabeltookplaceforpandq,thenthecapacitiesoftheinterioredgespq;qpinGcandoftheexterioredgessp;pt;sq;qtinGcequaltheresidualcapacitiesofthecorrespondingedgesinGc.Theprooffollowsdirectlyfromthefactthatifnochangeoflabeltookplaceforp;q,thennoneoftheheightvariableshp(xp);hq(xq)orthebalancevariablesypq(xp);yqp(xq)couldhavechanged.Duetolemma 1 ,forbuildinggraphGc,wecansimplyreusetheresidualgraphofGcandonlyre-computethosecapacitiesofGcforwhichtheabovelemmadoesnothold,thusspeeding-upthealgorithmevenfurther.Combiningspeedwithoptimality:Fig. 9(d) containstherunningtimesofFast-PDforvariousMRFproblems.Ascanbeseenfromthatgure,Fast-PDprovestobemuchfasterthaneitherthe-expansion 6 orthePD3amethod,e.g.Fast-PDhasbeenmorethan9timesfasterthan-expansionforthecaseofthe“penguin”image(17.44secsvs173.1secs).Infact,thisbehaviorisatypicalone,sinceFast-PDhasconsistentlyprovidedatleasta3-9timesspeedupforalltheproblemsithasbeentestedon.However,besidesitsefciency,Fast-PDdoesnotmakeanycompromisere-gardingtheoptimalityofitssolutions.Ononehand,thisisensuredbytheorems 2 , 3 .Ontheotherhand,Fast-PD,like 6Since-expansioncannotbeusedifd(
;
)isnotametric,themethodproposedin[ 7 ]hadtobeusedforthecasesofanon-metricd(
;
). [x;y] INIT DUALS PRIMALS(x;y):x x;y y;8pq;ypq(xp)+=wpqd(xp;xq)wpqd(xp;xq);8p;hp(
)+=cp(
) cp(
); Fig.11:Fast-PD'snewpseudocodefordynamicMRFs.anyotherprimal-dualmethod,canalsotellforfreehowwellitperformedbyalwaysprovidingaper-instancesub-optimalityboundforitssolution.Thiscomesatnoextracost,sinceanyratiobetweenthecostofaprimalsolutionandthecostofadualsolutioncanformsuchabound.E.g.g. 10 showshowtheseratiosvaryperinner-iterationforthe“tsukuba”and“penguin”problems(withsimilarresultsholdingfortheotherproblemsaswell).Asonecannotice,theseratiosdropto1veryquickly,meaningthatanalmostoptimalsolutionhasalreadybeenestimatedevenafterjustafewiterations(anddespitetheproblembeingNP-hard).5.DynamicMRFsBut,besidessingleMRFs,Fast-PDcanbeeasilyadaptedtoalsoboosttheefciencyfordynamicMRFs[ 5 ],i.e.MRFsvaryingovertime,thusshowingthegeneralityandpoweroftheproposedmethod.Infact,Fast-PDtsper-fectlytothistask.TheimplicitassumptionhereisthatthechangebetweensuccessiveMRFsissmall,andso,byini-tializingthecurrentMRFwiththenal(primal)solutionofthepreviousMRF,oneexpectstospeedupinference.Asig-nicantadvantageofFast-PDinthisregard,however,isthatitcanexploitnotonlypreviousMRF'sprimalsolution(sayx),butalsoitsdualsolution(sayy).Andthis,forinitializ-ingcurrentMRF'sbothprimalanddualsolutions(sayx;y).Obviously,forinitializingx,onecansimplysetx=x.Regardingtheinitializationofy,however,thingsareslightlymorecomplicated.FormaintainingFast-PD'soptimalityproperties,itturnsoutthat,aftersettingy=y,aslightcorrectionstillneedstobeappliedtoy.Inparticular,Fast-PDrequiresitsinitialsolutionytosatisfycondition( 4 ),i.e.ypq(xp)+yqp(xq)=wpqd(xp;xq),whereasysatisesypq(xp)+yqp(xq)=wpqd(xp;xq),i.e.condition( 4 )withwpqd(
;
)replacedbythepairwisepotentialwpqd(
;
)ofthepreviousMRF.Thesolutionforxingthatisverysimple:e.g.wecansimplysetypq(xp)+=wpqd(xp;xq)wpqd(xp;xq).Finally,fortak-ingintoaccountthepossiblydifferentsingletonpotentialsbetweensuccessiveMRFs,thenewheightswillobviouslyneedtobeupdatedashp(
)+=cp(
) cp(
),where cp(
)arethesingletonpotentialsofthepreviousMRF.ThesearetheonlychangesneededforthecaseofdynamicMRFs,andthusthenewpseudocodeappearsinFig. 11 .Asexpected,fordynamicMRFs,thespeedupprovidedbyFast-PDisevengreaterthansingleMRFs.E.g.Fig. 12(a) showstherunningtimesperframeforthe“SRItree”imagesequence.Fast-PDprovestobebemorethan10timesfasterthan-expansioninthiscase(requiringonaverage0.22secsperframe,whereas-expansionrequired2.28secsonaverage).Fast-PDcanthusrunonabout5 40 50 60 70 80 90 0 0.5 1 1.5 2 2.5 frametime (secs) a-expansionFast-PD (a)Runningtimesperframeforthe“SRItree”sequence 40 50 60 70 80 90 0 1 2 3x 105 frameNo. of augmentations a-expansionFast-PD (b)Augmentingpathsperframeforthe“SRItree”sequenceFig.12:Statisticsforthe“SRItree”sequence.frames/sec,i.e.itcandostereomatchingalmostinrealtimeforthisexample(infact,ifsuccessiveMRFsbeargreatersimilarity,evenmuchbiggerspeedupscanbeachieved).Furthermore,g. 12(b) showsthecorrespondingnumberofaugmentingpathsperframeforthe“SRItree”imagesequence(forboth-expansionandFast-PD).Ascanbeseenfromthatgure,asubstantialreductioninthenumberofaugmentingpathsisachievedbyFast-PD,whichhelpsthatalgorithmtoreduceitsrunningtime.ThissamebehaviorhasbeenobservedinallotherdynamicproblemsthatFast-PDhasbeentestedonaswell.Intuitively,whathappensisillustratedinFig. 13 (a).Fast-PDhasalreadymanagedtoclosethegapbetweenthenalprimal-dualcostsprimalx,dualyofthepreviousMRF.However,duetothepossiblydifferentsingleton(i.e.cp(
))orpairwise(i.e.wpqd(
;
))potentialsofthecurrentMRF,thesecostsneedtobeperturbedtogeneratethenewinitialcostsprimalx,dualy.Nevertheless,asonlyslightperturbationstakeplace,thenewprimal-dualgap(i.e.betweenprimalx,dualy)willstillbeclosetothepreviousgap(i.e.betweenprimalx,dualy).Asaresult,thenewgapwillremainsmall.FewaugmentingpathswillthereforehavetobefoundforthecurrentMRF,andthusthealgorithm'sperformanceisboosted.Putotherwise,forthecaseofdynamicMRFs,Fast-PDmanagestoboostperformance,i.e.reducenumberofaug-mentingpaths,acrosstwodifferent“axes”.Therstaxisliesalongthedifferentinner-iterationsofthesameMRF(e.g.seeredarrowsinFig. 13 (b)),whereasthesecondaxisextendsacrosstime,i.e.acrossdifferentMRFs(e.g.seebluearrowinFig. 13 (b),connectingthelastiterationofMRFt1totherstiterationofMRFt).

   (a)(b)
 \n\r\n … \n \r\r\r \r\r\r
 \n\n \r \r\r\r
 \n\n \r \n \r\r\r \r\r\r
 \n\n \r \r\r\r
 \n\n \r  Fig.13:(a)Thenalcostsprimalx,dualyofthepreviousMRFareslightlyperturbedtogivetheinitialcostsprimalx,dualyofthecurrentMRF.Therefore,theinitialprimal-dualgapofthecurrentMRFwillbeclosetothenalprimal-dualgapofthepreviousMRF.Sincethelatterissmall,sowillbetheformer,andthusfewaugmentingpathswillneedtobecomputedforthecurrentMRF.(b)Fast-PDreducesthenumberofaugmentingpathsin2ways:internally,i.e.acrossiterationsofthesameMRF(seeredarrows),aswellasexternally,i.e.acrossdifferentMRFs(seebluearrow).6.ConclusionsInconclusion,anewgraph-cutbasedmethodforMRFoptimizationhasbeenproposed.Itgeneralizes-expansion,whileitalsomanagestobesubstantiallyfasterthanthisstate-of-the-arttechnique.Hence,regardingoptimizationofstaticMRFs,thismethodprovidesasignicantspeedup.Inadditiontothat,however,itcanalsobeusedforboostingtheperformanceofdynamicMRFs.Inbothcases,itsefciencycomesfromthefactthatitexploitsinformationnotonlyfromthe“primal”problem(i.e.theMRFoptimizationproblem),butalsofroma“dual”problem.Moreover,despiteitsspeed,theproposedmethodcanneverthelessguaranteealmostoptimalsolutionsforaverywideclassofNP-hardMRFs.Duetoalloftheabove,andgiventheubiquityofMRFs,westronglybelievethatFast-PDcanprovetobeanextremelyusefultoolformanyproblemsincomputervisionintheyearstocome.References[1]N.Komodakis,G.TziritasandN.Paragios.FastPrimal-DualStrategiesforMRFOptimization.Technicalreport,2006. 5 [2]Y.BoykovandV.Kolmogorov.Anexperimentalcomparisonofmin-cut/max-owalgorithmsforenergyminimizationinvision.PAMI,26(9),2004. 6 [3]Y.Boykov,O.Veksler,andR.Zabih.Fastapproximateenergyminimizationviagraphcuts.PAMI,23(11),2001. 1 [4]O.JuanandY.Boykov.Activegraphcuts.InCVPR,2006. 1 [5]P.KohliandP.H.Torr.Efcientlysolvingdynamicmarkovrandomeldsusinggraphcuts.InICCV,2005. 1 , 7 [6]N.KomodakisandG.Tziritas.Anewframeworkforapprox-imatelabelingviagraph-cuts.InICCV,2005. 1 , 2 , 3 [7]C.Rother,S.Kumar,V.Kolmogorov,andA.Blake.Digitaltapestry.InCVPR,2005. 7 [8]R.Szeliski,etal.Acomparativestudyofenergyminimizationmethodsformarkovrandomelds.InECCV,2006. 1