unisaarlandde httpwwwmiaunisaarlandde Abstract We study an energy functional for computing optical 64258ow that com bines three assumptions a brightness constancy assumption a gradient constancy assumption and a discontinuitypreserving spatiotemporal ID: 26121
Download Pdf The PPT/PDF document "High Accuracy Optical Flow Estimation Ba..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
severalmodelscouldworktogether.Furthermore,variationalformulationsofmodelsgaveaccesstothelongexperienceofnumericalmathematicsinsolvingpartlydif-cultoptimisationproblems.Findingtheoptimalsolutiontoacertainmodelisoftennottrivial,andoftenthefullpotentialofamodelisnotusedbecauseconcessionstoimple-mentationaspectshavetobemade.Inthispaperweproposeanovelvariationalapproachthatintegratesseveralofthebe-forementionedconceptsandwhichcanbeminimisedwithasolidnumericalmethod.Itisfurthershownthatacoarse-to-nestrategyusingtheso-calledwarpingtechnique[7,16],implementsthenon-linearisedopticalowconstraintusedin[19,2]andinimageregistration.Thishastwoimportanteffects:Firstly,itbecomespossibletointegratethewarpingtechnique,whichwassofaronlyalgorithmicallymotivated,intoavariationalframework.Secondly,itshowsatheoreticallysoundwayofhowimagecorrespondenceproblemscanbesolvedwithanefcientmulti-resolutiontechnique.ItshouldbenotedthatapartfromaverynicepaperbyLef´ebureandCohen[14]notmanytheoreticalresultsonwarpingareavailablesofar.Finally,thegreyvalueconstancyassumption,whichisthebasicassumptioninopti-calowestimation,isextendedbyagradientconstancyassumption.Thismakesthemethodrobustagainstgreyvaluechanges.Whilegradientconstancyassumptionshavealsobeenproposedin[23,22]inordertodealwiththeapertureprobleminthescopeofalocalapproach,theirusewithinvariationalmethodsisnovel.Theexperimentalevaluationshowsthatourmethodyieldsexcellentresults.Comparedtothoseintheliterature,theiraccuracyisalwayssignicantlyhigher,sometimeseventwiceashighasthebestvalueknownsofar.Moreover,themethodprovedalsotoberobustunderaconsiderableamountofnoiseandcomputationtimesofonlyafewsec-ondsperframeoncontemporaryhardwarearepossible.Paperorganisation.Inthenextsection,ourvariationalmodelisintroduced,rstbydiscussingallmodelassumptions,andtheninformofanenergybasedformulation.Section3derivesaminimisationschemeforthisenergy.ThetheoreticalfoundationofwarpingmethodsasanumericalapproximationstepisgiveninSection4.Anexperi-mentalevaluationispresentedinSection5,followedbyabriefsummaryinSection6.2TheVariationalModelBeforederivingavariationalformulationforouropticalowmethod,wegiveanintu-itiveideaofwhichconstraintsinourviewshouldbeincludedinsuchamodel.Greyvalueconstancyassumption.Sincethebeginningofopticalowestimation,ithasbeenassumedthatthegreyvalueofapixelisnotchangedbythedisplacement.I(x;y;t)=I(x+u;y+v;t+1)(1)HereI: R3!Rdenotesarectangularimagesequence,andw:=(u;v;1)isthesearcheddisplacementvectorbetweenanimageattimetandanotherimageattimet+1.Thelinearisedversionofthegreyvalueconstancyassumptionyieldsthefamousopticalowconstraint[11]Ixu+Iyv+It=0(2) constancyassumptionaremeasuredbytheenergyEData(u;v)=Z jI(x+w)I(x)j2+ jrI(x+w)rI(x)j2dx(4)with beingaweightbetweenbothassumptions.Sincewithquadraticpenalisers,out-liersgettoomuchinuenceontheestimation,anincreasingconcavefunction (s2)isapplied,leadingtoarobustenergy[7,16]:EData(u;v)=Z jI(x+w)I(x)j2+ jrI(x+w)rI(x)j2dx(5)Thefunction canalsobeappliedseparatelytoeachofthesetwoterms.Weusethefunction (s2)=p s2+2whichresultsin(modied)L1minimisation.Duetothesmallpositiveconstant, (s)isstillconvexwhichoffersadvantagesintheminimisa-tionprocess.Moreover,thischoiceof doesnotintroduceanyadditionalparameters,sinceisonlyfornumericalreasonsandcanbesettoaxedvalue,whichwechoosetobe0:001.Finally,asmoothnesstermhastodescribethemodelassumptionofapiecewisesmoothoweld.Thisisachievedbypenalisingthetotalvariationoftheoweld[20,8],whichcanbeexpressedasESmooth(u;v)=Z jr3uj2+jr3vj2dx:(6)withthesamefunctionfor asabove.Thespatio-temporalgradientr3:=(@x;@y;@t)indicatesthataspatio-temporalsmoothnessassumptionisinvolved.Forapplicationswithonlytwoimagesavailableitisreplacedbythespatialgradient.ThetotalenergyistheweightedsumbetweenthedatatermandthesmoothnesstermE(u;v)=EData+ESmooth(7)withsomeregularisationparameter0.Nowthegoalistondthefunctionsuandvthatminimisethisenergy.3Minimisation3.1EulerLagrangeEquationsSinceE(u;v)ishighlynonlinear,theminimisationisnottrivial.Forbetterreadabil-itywedenethefollowingabbreviations,wheretheuseofzinsteadoftemphasisesthattheexpressionisnotatemporalderivativebutadifferencethatissoughttobeminimised.Ix:=@xI(x+w),Iy:=@yI(x+w),Iz:=I(x+w)I(x),Ixx:=@xxI(x+w),Ixy:=@xyI(x+w),Iyy:=@yyI(x+w),Ixz:=@xI(x+w)@xI(x),Iyz:=@yI(x+w)@yI(x).(8) whereuk+1=uk+dukandvk+1=vk+dvk.Sowesplittheunknownsuk+1,vk+1inthesolutionsofthepreviousiterationstepuk;vkandunknownincrementsduk;dvk.Forbetterreadabilitylet( 0)kData:= 0(Ikz+Ikxduk+Ikydvk)2+ (Ikxz+Ikxxduk+Ikxydvk)2+(Ikyz+Ikxyduk+Ikyydvk)2;( 0)kSmooth:= 0(jr3(uk+duk)j2+jr3(vk+dvk)j2),(10)where( 0)kDatacanbeinterpretedasarobustnessfactorinthedataterm,and( 0)kSmoothasadiffusivityinthesmoothnessterm.Withthistherstequationinsystem(9)canbewrittenas0=( 0)kDataIkxIkz+Ikxduk+Ikydvk+ ( 0)kDataIkxx(Ikxz+Ikxxduk+Ikxydvk)+Ikxy(Ikyz+Ikxyduk+Ikyydvk)div( 0)kSmoothr3(uk+duk);(11)andthesecondequationcanbeexpressedinasimilarway.Thisisstillanonlinearsystemofequationsforaxedk,butnowintheunknownincrementsduk;dvk.Astheonlyremainingnonlinearityisdueto 0,and hasbeenchosentobeaconvexfunction,theremainingoptimisationproblemisaconvexproblem,i.e.thereexistsauniqueminimumsolution.Inordertoremovetheremainingnonlinearityin 0,asecond,inner,xedpointiterationloopisapplied.Letduk;0:=0,dvk;0:=0beourinitialisationandletduk;l;dvk;ldenotetheiterationvariablesatsomestepl.Furthermore,let( 0)k;lDataand( 0)k;lSmoothdenotetherobustnessfactorandthediffusivitydenedin(10)atiterationk,l.Thennallythelinearsystemofequationsinduk;l+1;dvk;l+1reads0=( 0)k;lDataIkxIkz+Ikxduk;l+1+Ikydvk;l+1+ Ikxx(Ikxz+Ikxxduk;l+1+Ikxydvk;l+1)+ Ikxy(Ikyz+Ikxyduk;l+1+Ikyydvk;l+1)div( 0)k;lSmoothr3(uk+duk;l+1)(12)fortherstequation.Usingstandarddiscretisationsforthederivatives,theresultingsparselinearsystemofequationscannowbesolvedwithcommonnumericalmethods,suchasGauss-SeidelorSORiterations.ExpressionsoftypeI(x+wk)arecomputedbymeansofbilinearinterpolation.4RelationtoWarpingMethodsCoarse-to-newarpingtechniquesareafrequentlyusedtoolforimprovingtheperfor-manceofopticowmethods[3,7,17].Whiletheyareoftenintroducedonapurelyexperimentalbasis,weshowinthissectionthattheycanbetheoreticallyjustiedasanumericalapproximation. YosemitewithcloudsYosemitewithoutclouds TechniqueAAESTDTechniqueAAESTD Nagel[5]10.2216.51Juetal.[12]2.162.00HornSchunck,mod.[5]9.7816.19Bab-HadiasharSuter[4]2.052.92Urasetal.[5]8.9415.61LaiVemuri[13]1.991.41Alvarezetal.[2]5.537.40Ourmethod(2D)1.591.39Weickertetal.[24]5.188.68M´eminP´erez[16]1.581.21M´eminP´erez[16]4.696.89Weickertetal.[24]1.461.50Ourmethod(2D)2.467.31Farneb¨ack[10]1.142.14Ourmethod(3D)1.946.02Ourmethod(3D)0.981.17 Table1.Comparisonbetweentheresultsfromtheliteraturewith100%densityandourresultsfortheYosemitesequencewithandwithoutcloudysky.AAE=averageangularerror.STD=standarddeviation.2D=spatialsmoothnessassumption.3D=spatio-temporalsmoothnessassumption.YosemitewithcloudsYosemitewithoutclouds nAAESTDnAAESTD 01.946.0200.981.17102.505.96101.261.29203.126.24201.631.39303.776.54302.031.53404.377.12402.401.71 Table2.ResultsfortheYosemitesequencewithandwithoutcloudysky.Gaussiannoisewithvaryingstandarddeviationsnwasadded,andtheaverageangularerrorsandtheirstandarddeviationswerecomputed.AAE=averageangularerror.STD=standarddeviation.correspondingoweldspresentedinFig.1giveaqualitativeimpressionoftheserawnumbers:Theymatchthegroundtruthverywell.Notonlythediscontinuitybetweenthetwotypesofmotionispreserved,alsothetranslationalmotionofthecloudsisesti-matedaccurately.Thereasonforthisbehaviourliesinourassumptions,thatareclearlystatedintheenergyfunctional:Whilethechoiceofthesmoothnesstermallowsdiscon-tinuities,thegradientconstancyassumptionisabletohandlebrightnesschangeslikeintheareaoftheclouds.BecauseofthepresenceofsecondorderimagederivativesintheEuler-Lagrangeequa-tions,wetestedtheinuenceofnoiseontheperformanceofourmethodinthenextexperiment.WeaddedGaussiannoiseofmeanzeroanddifferentstandarddeviationstobothsequences.TheobtainedresultsarepresentedinTab.2.Theyshowthatourapproachevenyieldsexcellentowestimateswhenseverenoiseispresent:ForthecloudyYosemitesequence,ouraverageangularerrorfornoisewithstandarddeviation40isbetterthanallresultsfromtheliteratureforthesequencewithoutnoise.Inathirdexperimentweevaluatedtherobustnessofthefreeparametersinourap-proach:theweight betweenthegreyvalueandthegradientconstancyassumption,andthesmoothnessparameter.OftenanimagesequenceispreprocessedbyGaus- Yosemitewithclouds AAE 0.8801001:940.4801002:101.6801002:040.8401002:670.81601002:210.880502:070.8802002:03 Table3.Parametervariationforourmethodwithspatio-temporalsmoothnessassumption.3D-spatio-temporalmethod reductionouterxedinnerxedSOR computationAAEfactorpointiter.pointiter.iter. time/frame 0.9577510 23.4s1:940.9038210 5.1s2:090.8018210 2.7s2:560.7514110 1.2s3:44 Table4.ComputationtimesandconvergenceforYosemitesequencewithclouds.Althoughourpaperdoesnotfocusonfastcomputationbutonhighaccuracy,theim-plicitminimisationschemepresentedhereisalsoreasonablyfast,especiallyifthere-ductionfactorisloweredoriftheiterationsarestoppedbeforefullconvergence.TheconvergencebehaviourandcomputationtimescanbefoundinTab.4.Computationshavebeenperformedona3.06GHzIntelPentium4processorexecutingC/C++code.Forevaluatingtheperformanceofourmethodforreal-worldimagedata,theEttlingerTortrafcsequencebyNagelwasused.Thissequenceconsistsof50framesofsize512512.Itisavailableat http://i21www.ira.uka.de/image sequences/ .InFig.2thecomputedoweldanditsmagnitudeareshown.Ourestimationgivesveryrealisticresults,andthealgorithmhardlysuffersfrominterlacingartifactsthatarepresentinallframes.Moreover,theowboundariesarerathersharpandcanbeuseddirectlyforsegmentationpurposesbyapplyingasimplethresholdingstep.6ConclusionInthispaperwehaveinvestigatedacontinuous,rotationallyinvariantenergyfunctionalforopticalowcomputationsbasedontwoterms:arobustdatatermwithabright-nessconstancyandagradientconstancyassumption,combinedwithadiscontinuity-preservingspatio-temporalTVregulariser.Whileeachoftheseconceptshasproveditsusebefore(seee.g.[22,26]),wehaveshownthattheircombinationoutperformsallmethodsfromtheliteraturesofar.Oneofthemainreasonsforthisperformanceistheuseofanenergyfunctionalwithnon-lineariseddatatermandourstrategytocon-sequentlypostponealllinearisationstothenumericalscheme:Whilelinearisationsinthemodelimmediatelycompromisetheoverallperformanceofthesystem,linearisa- 9.R.Deriche,P.Kornprobst,andG.Aubert.Optical-owestimationwhilepreservingitsdis-continuities:avariationalapproach.InProc.SecondAsianConferenceonComputerVision,volume2,pages290295,Singapore,Dec.1995.10.G.Farneb¨ack.Veryhighaccuracyvelocityestimationusingorientationtensors,parametricmotion,andsimultaneoussegmentationofthemotioneld.InProc.EighthInternationalConferenceonComputerVision,volume1,pages171177,Vancouver,Canada,July2001.IEEEComputerSocietyPress.11.B.HornandB.Schunck.Determiningopticalow.ArticialIntelligence,17:185203,1981.12.S.Ju,M.Black,andA.Jepson.Skinandbones:multi-layer,locallyafne,opticalowandregularizationwithtransparency.InProc.1996IEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,pages307314,SanFrancisco,CA,June1996.IEEEComputerSocietyPress.13.S.-H.LaiandB.C.Vemuri.Reliableandefcientcomputationofopticalow.InternationalJournalofComputerVision,29(2):87105,Oct.1998.14.M.Lef´ebureandL.D.Cohen.Imageregistration,opticalowandlocalrigidity.JournalofMathematicalImagingandVision,14(2):131147,Mar.2001.15.B.LucasandT.Kanade.Aniterativeimageregistrationtechniquewithanapplicationtostereovision.InProc.SeventhInternationalJointConferenceonArticialIntelligence,pages674679,Vancouver,Canada,Aug.1981.16.E.M´eminandP.P´erez.Amultigridapproachforhierarchicalmotionestimation.InProc.SixthInternationalConferenceonComputerVision,pages933938,Bombay,India,Jan.1998.NarosaPublishingHouse.17.E.M´eminandP.P´erez.Hierarchicalestimationandsegmentationofdensemotionelds.InternationalJournalofComputerVision,46(2):129155,2002.18.H.-H.Nagel.Extendingthe'orientedsmoothnessconstraint'intothetemporaldomainandtheestimationofderivativesofopticalow.InO.Faugeras,editor,ComputerVisionECCV'90,volume427ofLectureNotesinComputerScience,pages139148.Springer,Berlin,1990.19.H.-H.NagelandW.Enkelmann.Aninvestigationofsmoothnessconstraintsfortheesti-mationofdisplacementvectoreldsfromimagesequences.IEEETransactionsonPatternAnalysisandMachineIntelligence,8:565593,1986.20.L.I.Rudin,S.Osher,andE.Fatemi.Nonlineartotalvariationbasednoiseremovalalgo-rithms.PhysicaD,60:259268,1992.21.C.Schn¨orr.Segmentationofvisualmotionbyminimizingconvexnon-quadraticfunctionals.InProc.TwelfthInternationalConferenceonPatternRecognition,volumeA,pages661663,Jerusalem,Israel,Oct.1994.IEEEComputerSocietyPress.22.M.Tistarelli.Multipleconstraintsforopticalow.InJ.-O.Eklundh,editor,ComputerVisionECCV'94,volume800ofLectureNotesinComputerScience,pages6170.Springer,Berlin,1994.23.S.Uras,F.Girosi,A.Verri,andV.Torre.Acomputationalapproachtomotionperception.BiologicalCybernetics,60:7987,1988.24.J.Weickert,A.Bruhn,andC.Schn¨orr.Lucas/KanademeetsHorn/Schunck:Combininglocalandglobalopticowmethods.TechnicalReport82,Dept.ofMathematics,SaarlandUniversity,Saarbr¨ucken,Germany,Apr.2003.25.J.WeickertandC.Schn¨orr.AtheoreticalframeworkforconvexregularizersinPDE-basedcomputationofimagemotion.InternationalJournalofComputerVision,45(3):245264,Dec.2001.26.J.WeickertandC.Schn¨orr.Variationalopticowcomputationwithaspatio-temporalsmoothnessconstraint.JournalofMathematicalImagingandVision,14(3):245255,May2001.