/
High Accuracy Optical Flow Estimation Based on a Theory for Warping Thomas Brox Andr es High Accuracy Optical Flow Estimation Based on a Theory for Warping Thomas Brox Andr es

High Accuracy Optical Flow Estimation Based on a Theory for Warping Thomas Brox Andr es - PDF document

yoshiko-marsland
yoshiko-marsland . @yoshiko-marsland
Follow
541 views
Uploaded On 2014-12-18

High Accuracy Optical Flow Estimation Based on a Theory for Warping Thomas Brox Andr es - PPT Presentation

unisaarlandde httpwwwmiaunisaarlandde Abstract We study an energy functional for computing optical 64258ow that com bines three assumptions a brightness constancy assumption a gradient constancy assumption and a discontinuitypreserving spatiotemporal ID: 26121

unisaarlandde httpwwwmiaunisaarlandde Abstract study

Share:

Link:

Embed:

Download Presentation from below link

Download Pdf The PPT/PDF document "High Accuracy Optical Flow Estimation Ba..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

severalmodelscouldworktogether.Furthermore,variationalformulationsofmodelsgaveaccesstothelongexperienceofnumericalmathematicsinsolvingpartlydif-cultoptimisationproblems.Findingtheoptimalsolutiontoacertainmodelisoftennottrivial,andoftenthefullpotentialofamodelisnotusedbecauseconcessionstoimple-mentationaspectshavetobemade.Inthispaperweproposeanovelvariationalapproachthatintegratesseveralofthebe-forementionedconceptsandwhichcanbeminimisedwithasolidnumericalmethod.Itisfurthershownthatacoarse-to-nestrategyusingtheso-calledwarpingtechnique[7,16],implementsthenon-linearisedopticalowconstraintusedin[19,2]andinimageregistration.Thishastwoimportanteffects:Firstly,itbecomespossibletointegratethewarpingtechnique,whichwassofaronlyalgorithmicallymotivated,intoavariationalframework.Secondly,itshowsatheoreticallysoundwayofhowimagecorrespondenceproblemscanbesolvedwithanefcientmulti-resolutiontechnique.Itshouldbenotedthat–apartfromaverynicepaperbyLef´ebureandCohen[14]–notmanytheoreticalresultsonwarpingareavailablesofar.Finally,thegreyvalueconstancyassumption,whichisthebasicassumptioninopti-calowestimation,isextendedbyagradientconstancyassumption.Thismakesthemethodrobustagainstgreyvaluechanges.Whilegradientconstancyassumptionshavealsobeenproposedin[23,22]inordertodealwiththeapertureprobleminthescopeofalocalapproach,theirusewithinvariationalmethodsisnovel.Theexperimentalevaluationshowsthatourmethodyieldsexcellentresults.Comparedtothoseintheliterature,theiraccuracyisalwayssignicantlyhigher,sometimeseventwiceashighasthebestvalueknownsofar.Moreover,themethodprovedalsotoberobustunderaconsiderableamountofnoiseandcomputationtimesofonlyafewsec-ondsperframeoncontemporaryhardwarearepossible.Paperorganisation.Inthenextsection,ourvariationalmodelisintroduced,rstbydiscussingallmodelassumptions,andtheninformofanenergybasedformulation.Section3derivesaminimisationschemeforthisenergy.ThetheoreticalfoundationofwarpingmethodsasanumericalapproximationstepisgiveninSection4.Anexperi-mentalevaluationispresentedinSection5,followedbyabriefsummaryinSection6.2TheVariationalModelBeforederivingavariationalformulationforouropticalowmethod,wegiveanintu-itiveideaofwhichconstraintsinourviewshouldbeincludedinsuchamodel.–Greyvalueconstancyassumption.Sincethebeginningofopticalowestimation,ithasbeenassumedthatthegreyvalueofapixelisnotchangedbythedisplacement.I(x;y;t)=I(x+u;y+v;t+1)(1)HereI: R3!Rdenotesarectangularimagesequence,andw:=(u;v;1)�isthesearcheddisplacementvectorbetweenanimageattimetandanotherimageattimet+1.Thelinearisedversionofthegreyvalueconstancyassumptionyieldsthefamousopticalowconstraint[11]Ixu+Iyv+It=0(2) constancyassumptionaremeasuredbytheenergyEData(u;v)=Z �jI(x+w)�I(x)j2+ jrI(x+w)�rI(x)j2dx(4)with beingaweightbetweenbothassumptions.Sincewithquadraticpenalisers,out-liersgettoomuchinuenceontheestimation,anincreasingconcavefunction (s2)isapplied,leadingtoarobustenergy[7,16]:EData(u;v)=Z �jI(x+w)�I(x)j2+ jrI(x+w)�rI(x)j2dx(5)Thefunction canalsobeappliedseparatelytoeachofthesetwoterms.Weusethefunction (s2)=p s2+2whichresultsin(modied)L1minimisation.Duetothesmallpositiveconstant, (s)isstillconvexwhichoffersadvantagesintheminimisa-tionprocess.Moreover,thischoiceof doesnotintroduceanyadditionalparameters,sinceisonlyfornumericalreasonsandcanbesettoaxedvalue,whichwechoosetobe0:001.Finally,asmoothnesstermhastodescribethemodelassumptionofapiecewisesmoothoweld.Thisisachievedbypenalisingthetotalvariationoftheoweld[20,8],whichcanbeexpressedasESmooth(u;v)=Z �jr3uj2+jr3vj2dx:(6)withthesamefunctionfor asabove.Thespatio-temporalgradientr3:=(@x;@y;@t)�indicatesthataspatio-temporalsmoothnessassumptionisinvolved.Forapplicationswithonlytwoimagesavailableitisreplacedbythespatialgradient.ThetotalenergyistheweightedsumbetweenthedatatermandthesmoothnesstermE(u;v)=EData+ ESmooth(7)withsomeregularisationparameter �0.Nowthegoalistondthefunctionsuandvthatminimisethisenergy.3Minimisation3.1Euler–LagrangeEquationsSinceE(u;v)ishighlynonlinear,theminimisationisnottrivial.Forbetterreadabil-itywedenethefollowingabbreviations,wheretheuseofzinsteadoftemphasisesthattheexpressionisnotatemporalderivativebutadifferencethatissoughttobeminimised.Ix:=@xI(x+w),Iy:=@yI(x+w),Iz:=I(x+w)�I(x),Ixx:=@xxI(x+w),Ixy:=@xyI(x+w),Iyy:=@yyI(x+w),Ixz:=@xI(x+w)�@xI(x),Iyz:=@yI(x+w)�@yI(x).(8) whereuk+1=uk+dukandvk+1=vk+dvk.Sowesplittheunknownsuk+1,vk+1inthesolutionsofthepreviousiterationstepuk;vkandunknownincrementsduk;dvk.Forbetterreadabilitylet( 0)kData:= 0(Ikz+Ikxduk+Ikydvk)2+ �(Ikxz+Ikxxduk+Ikxydvk)2+(Ikyz+Ikxyduk+Ikyydvk)2;( 0)kSmooth:= 0(jr3(uk+duk)j2+jr3(vk+dvk)j2),(10)where( 0)kDatacanbeinterpretedasarobustnessfactorinthedataterm,and( 0)kSmoothasadiffusivityinthesmoothnessterm.Withthistherstequationinsystem(9)canbewrittenas0=( 0)kDataIkx�Ikz+Ikxduk+Ikydvk+ ( 0)kDataIkxx(Ikxz+Ikxxduk+Ikxydvk)+Ikxy(Ikyz+Ikxyduk+Ikyydvk)� div�( 0)kSmoothr3(uk+duk);(11)andthesecondequationcanbeexpressedinasimilarway.Thisisstillanonlinearsystemofequationsforaxedk,butnowintheunknownincrementsduk;dvk.Astheonlyremainingnonlinearityisdueto 0,and hasbeenchosentobeaconvexfunction,theremainingoptimisationproblemisaconvexproblem,i.e.thereexistsauniqueminimumsolution.Inordertoremovetheremainingnonlinearityin 0,asecond,inner,xedpointiterationloopisapplied.Letduk;0:=0,dvk;0:=0beourinitialisationandletduk;l;dvk;ldenotetheiterationvariablesatsomestepl.Furthermore,let( 0)k;lDataand( 0)k;lSmoothdenotetherobustnessfactorandthediffusivitydenedin(10)atiterationk,l.Thennallythelinearsystemofequationsinduk;l+1;dvk;l+1reads0=( 0)k;lDataIkx�Ikz+Ikxduk;l+1+Ikydvk;l+1+ Ikxx(Ikxz+Ikxxduk;l+1+Ikxydvk;l+1)+ Ikxy(Ikyz+Ikxyduk;l+1+Ikyydvk;l+1)� div( 0)k;lSmoothr3(uk+duk;l+1)(12)fortherstequation.Usingstandarddiscretisationsforthederivatives,theresultingsparselinearsystemofequationscannowbesolvedwithcommonnumericalmethods,suchasGauss-SeidelorSORiterations.ExpressionsoftypeI(x+wk)arecomputedbymeansofbilinearinterpolation.4RelationtoWarpingMethodsCoarse-to-newarpingtechniquesareafrequentlyusedtoolforimprovingtheperfor-manceofopticowmethods[3,7,17].Whiletheyareoftenintroducedonapurelyexperimentalbasis,weshowinthissectionthattheycanbetheoreticallyjustiedasanumericalapproximation. YosemitewithcloudsYosemitewithoutclouds TechniqueAAESTDTechniqueAAESTD Nagel[5]10.2216.51Juetal.[12]2.162.00Horn–Schunck,mod.[5]9.7816.19Bab-Hadiashar–Suter[4]2.052.92Urasetal.[5]8.9415.61Lai–Vemuri[13]1.991.41Alvarezetal.[2]5.537.40Ourmethod(2D)1.591.39Weickertetal.[24]5.188.68M´emin–P´erez[16]1.581.21M´emin–P´erez[16]4.696.89Weickertetal.[24]1.461.50Ourmethod(2D)2.467.31Farneb¨ack[10]1.142.14Ourmethod(3D)1.946.02Ourmethod(3D)0.981.17 Table1.Comparisonbetweentheresultsfromtheliteraturewith100%densityandourresultsfortheYosemitesequencewithandwithoutcloudysky.AAE=averageangularerror.STD=standarddeviation.2D=spatialsmoothnessassumption.3D=spatio-temporalsmoothnessassumption.YosemitewithcloudsYosemitewithoutclouds nAAESTDnAAESTD 01.946.0200.981.17102.505.96101.261.29203.126.24201.631.39303.776.54302.031.53404.377.12402.401.71 Table2.ResultsfortheYosemitesequencewithandwithoutcloudysky.Gaussiannoisewithvaryingstandarddeviationsnwasadded,andtheaverageangularerrorsandtheirstandarddeviationswerecomputed.AAE=averageangularerror.STD=standarddeviation.correspondingoweldspresentedinFig.1giveaqualitativeimpressionoftheserawnumbers:Theymatchthegroundtruthverywell.Notonlythediscontinuitybetweenthetwotypesofmotionispreserved,alsothetranslationalmotionofthecloudsisesti-matedaccurately.Thereasonforthisbehaviourliesinourassumptions,thatareclearlystatedintheenergyfunctional:Whilethechoiceofthesmoothnesstermallowsdiscon-tinuities,thegradientconstancyassumptionisabletohandlebrightnesschanges–likeintheareaoftheclouds.BecauseofthepresenceofsecondorderimagederivativesintheEuler-Lagrangeequa-tions,wetestedtheinuenceofnoiseontheperformanceofourmethodinthenextexperiment.WeaddedGaussiannoiseofmeanzeroanddifferentstandarddeviationstobothsequences.TheobtainedresultsarepresentedinTab.2.Theyshowthatourapproachevenyieldsexcellentowestimateswhenseverenoiseispresent:ForthecloudyYosemitesequence,ouraverageangularerrorfornoisewithstandarddeviation40isbetterthanallresultsfromtheliteratureforthesequencewithoutnoise.Inathirdexperimentweevaluatedtherobustnessofthefreeparametersinourap-proach:theweight betweenthegreyvalueandthegradientconstancyassumption,andthesmoothnessparameter .OftenanimagesequenceispreprocessedbyGaus- Yosemitewithclouds  AAE 0.8801001:940.4801002:101.6801002:040.8401002:670.81601002:210.880502:070.8802002:03 Table3.Parametervariationforourmethodwithspatio-temporalsmoothnessassumption.3D-spatio-temporalmethod reductionouterxedinnerxedSOR computationAAEfactorpointiter.pointiter.iter. time/frame 0.9577510 23.4s1:940.9038210 5.1s2:090.8018210 2.7s2:560.7514110 1.2s3:44 Table4.ComputationtimesandconvergenceforYosemitesequencewithclouds.Althoughourpaperdoesnotfocusonfastcomputationbutonhighaccuracy,theim-plicitminimisationschemepresentedhereisalsoreasonablyfast,especiallyifthere-ductionfactorisloweredoriftheiterationsarestoppedbeforefullconvergence.TheconvergencebehaviourandcomputationtimescanbefoundinTab.4.Computationshavebeenperformedona3.06GHzIntelPentium4processorexecutingC/C++code.Forevaluatingtheperformanceofourmethodforreal-worldimagedata,theEttlingerTortrafcsequencebyNagelwasused.Thissequenceconsistsof50framesofsize512512.Itisavailableat http://i21www.ira.uka.de/image sequences/ .InFig.2thecomputedoweldanditsmagnitudeareshown.Ourestimationgivesveryrealisticresults,andthealgorithmhardlysuffersfrominterlacingartifactsthatarepresentinallframes.Moreover,theowboundariesarerathersharpandcanbeuseddirectlyforsegmentationpurposesbyapplyingasimplethresholdingstep.6ConclusionInthispaperwehaveinvestigatedacontinuous,rotationallyinvariantenergyfunctionalforopticalowcomputationsbasedontwoterms:arobustdatatermwithabright-nessconstancyandagradientconstancyassumption,combinedwithadiscontinuity-preservingspatio-temporalTVregulariser.Whileeachoftheseconceptshasproveditsusebefore(seee.g.[22,26]),wehaveshownthattheircombinationoutperformsallmethodsfromtheliteraturesofar.Oneofthemainreasonsforthisperformanceistheuseofanenergyfunctionalwithnon-lineariseddatatermandourstrategytocon-sequentlypostponealllinearisationstothenumericalscheme:Whilelinearisationsinthemodelimmediatelycompromisetheoverallperformanceofthesystem,linearisa- 9.R.Deriche,P.Kornprobst,andG.Aubert.Optical-owestimationwhilepreservingitsdis-continuities:avariationalapproach.InProc.SecondAsianConferenceonComputerVision,volume2,pages290–295,Singapore,Dec.1995.10.G.Farneb¨ack.Veryhighaccuracyvelocityestimationusingorientationtensors,parametricmotion,andsimultaneoussegmentationofthemotioneld.InProc.EighthInternationalConferenceonComputerVision,volume1,pages171–177,Vancouver,Canada,July2001.IEEEComputerSocietyPress.11.B.HornandB.Schunck.Determiningopticalow.ArticialIntelligence,17:185–203,1981.12.S.Ju,M.Black,andA.Jepson.Skinandbones:multi-layer,locallyafne,opticalowandregularizationwithtransparency.InProc.1996IEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,pages307–314,SanFrancisco,CA,June1996.IEEEComputerSocietyPress.13.S.-H.LaiandB.C.Vemuri.Reliableandefcientcomputationofopticalow.InternationalJournalofComputerVision,29(2):87–105,Oct.1998.14.M.Lef´ebureandL.D.Cohen.Imageregistration,opticalowandlocalrigidity.JournalofMathematicalImagingandVision,14(2):131–147,Mar.2001.15.B.LucasandT.Kanade.Aniterativeimageregistrationtechniquewithanapplicationtostereovision.InProc.SeventhInternationalJointConferenceonArticialIntelligence,pages674–679,Vancouver,Canada,Aug.1981.16.E.M´eminandP.P´erez.Amultigridapproachforhierarchicalmotionestimation.InProc.SixthInternationalConferenceonComputerVision,pages933–938,Bombay,India,Jan.1998.NarosaPublishingHouse.17.E.M´eminandP.P´erez.Hierarchicalestimationandsegmentationofdensemotionelds.InternationalJournalofComputerVision,46(2):129–155,2002.18.H.-H.Nagel.Extendingthe'orientedsmoothnessconstraint'intothetemporaldomainandtheestimationofderivativesofopticalow.InO.Faugeras,editor,ComputerVision–ECCV'90,volume427ofLectureNotesinComputerScience,pages139–148.Springer,Berlin,1990.19.H.-H.NagelandW.Enkelmann.Aninvestigationofsmoothnessconstraintsfortheesti-mationofdisplacementvectoreldsfromimagesequences.IEEETransactionsonPatternAnalysisandMachineIntelligence,8:565–593,1986.20.L.I.Rudin,S.Osher,andE.Fatemi.Nonlineartotalvariationbasednoiseremovalalgo-rithms.PhysicaD,60:259–268,1992.21.C.Schn¨orr.Segmentationofvisualmotionbyminimizingconvexnon-quadraticfunctionals.InProc.TwelfthInternationalConferenceonPatternRecognition,volumeA,pages661–663,Jerusalem,Israel,Oct.1994.IEEEComputerSocietyPress.22.M.Tistarelli.Multipleconstraintsforopticalow.InJ.-O.Eklundh,editor,ComputerVision–ECCV'94,volume800ofLectureNotesinComputerScience,pages61–70.Springer,Berlin,1994.23.S.Uras,F.Girosi,A.Verri,andV.Torre.Acomputationalapproachtomotionperception.BiologicalCybernetics,60:79–87,1988.24.J.Weickert,A.Bruhn,andC.Schn¨orr.Lucas/KanademeetsHorn/Schunck:Combininglocalandglobalopticowmethods.TechnicalReport82,Dept.ofMathematics,SaarlandUniversity,Saarbr¨ucken,Germany,Apr.2003.25.J.WeickertandC.Schn¨orr.AtheoreticalframeworkforconvexregularizersinPDE-basedcomputationofimagemotion.InternationalJournalofComputerVision,45(3):245–264,Dec.2001.26.J.WeickertandC.Schn¨orr.Variationalopticowcomputationwithaspatio-temporalsmoothnessconstraint.JournalofMathematicalImagingandVision,14(3):245–255,May2001.