brkicsinisasegvic ferhr Institute of Electrical Measurement and Measurement Signal Processing Graz University of Technology Austria axelpinztugrazat Abstract We study the problem of traf64257c sign detection in the context of traf64257c infrastructur ID: 27278
Download Pdf The PPT/PDF document "Trafc sign detection as a component of a..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1.automaticmappingoftrafcsignsbyusingsigndetectionandrecognitioningeoreferencedvideo2.vericationofanewlyacquiredvideoagainstapreviouslyrecordedstateoftrafcinfrastructureThedevelopmentofthissystemisanongoingprojectcarriedoutincooperationwithanindustrialpartner1.Inthisstageoftheprojectweareprimarilyconcernedwiththeproblemoftrafcsigndetection,asitisacrucialcomponentofthesystem.Thispaperdescribesanearlystageofourprojectandaimsatexploringourdataandpresentingsomesolutionsontheproblemoftrafcsigndetection.2.RelatedworkThevastmajorityofpublishedmethodsfortrafcsigndetectiontakeasmuchuseofaprioriinfor-mationaspossible.Appearanceofatrafcsignisstrictlyconstrained:thesignisalwaysaregularpolygonoracircleanditscolorsarewellknown.Theexactcolorsusedgenerallydependonthecountry,butusuallyincludewhite,yellow,red,blueandblack.Therefore,atypicalsigndetectionmethodusessomecombinationofcolorandshapeconstraintstodetectasign.Colorinformationisusuallyexploitedbyperformingcolor-basedsegmentationoftheimage.SuchsegmentationisdifculttoperforminRGBspace.RGBcolorsareverysensitivetoilluminationchangesandtrafcscenestendtohavevaryingillumination.Someauthors[4,9,25,5,7]trytoover-comethisbydevisingsimpleformulasrelatingred,greenandbluecomponentsandexperimentingwithappropriatethresholds.OthersworkinHSI[21,12]orL*a*b[23]colorspaces.Therehavebeenapproacheswithtrainingneuralnetworks[20]orsupportvectormachines[28]forcolorlabeling.Shapeinformationcanbeobtainedbyvariousstrategies:Houghtransform,fastradialtransform,cornerdetection,patternmatching,geneticalgorithmsetc.Houghtransformisusedtolocatelinesorcirclescorrespondingtoasign[12,10].RecentlyLoyandZelinsky[16]proposedatechniquesimilartoHoughtransformcalledfastradialtransform.Itwassuccesfullyusedforsigndetectionin[21,15].Shapeissometimesdeterminedbyusingcornerinformationofcandidateregions[21,12,7].Someresearchersusepatternmatchingwithsimpleshapetemplates[5].Atechniquebasedongeneticalgorithmswasusedfordetectionofcirculartrafcsignsin[24].Shapedetectionoftenfailsincasesofinsufcientedgecontrast,somostresearcherschoosetoaugmentitwithcolorinformation.Adifferentapproachtotheproblemoftrafcsigndetectionistouseageneralpurposeobjectdetectorinsteadofdevisingapplication-specicalgorithms.AfewresearchersreportsuccesswithusingtherobustdetectorofViolaandJones[26].Forinstance,ChenandSieh[6]usetheViola-Jonesdetectortodetectcircularsigns.Bahlmannetal.[2]extendthefeaturesetdescribedbyViolaandJonesinordertousecolorinformation.BaroandVitria[3]detectsignsusingthemethodofViolaandJonesaugmentedbyafeaturesetproposedbyLienhartandMaydt[14].Theyfurthereliminatefalsepositivesbyusingfastradialsymmetrytransformandprincipalcomponentanalysis.TheirresultsareusedbyEscaleraandRadeva[8].TheinterestedreadercanndamoredetailedreportoncurrentresearchinsigndetectioninarecentlypublishedreviewpaperbyNguwiandKouzani[19]. 1TheInstituteofTransportandCommunications,Zagreb,Croatia (a) (b) (c) (d) (e)Figure2.ClassesoftrafcsignsinCroatia:(a)warningsign,(b)explicitordersign,(c)informationsign,(d)directionsign,(e)supplementalpanel.specialregulationsigns,asdenedbytheViennaConvention,arebyCroatianregulationscontainedinasingleclass–informationsigns.Atthispoint,ourcollectioncontains2352annotatedsignimages.Thiscorrespondstoabout590physicalsigns,aswecollectedfourimagesofeachsign.Whenasignappearsinavideo,itisvisibleforafewseconds.Thevideoframerateis24framespersecond,sothereareonehundredormoreframesinwhichthesigncouldbeannotated.Ourpolicywastoannotatethefollowingfourdistinctiveframes:rstwhenthesignbecomesrecognizablelastwhenthesignisclosesttothecameratheremainingtwoinbetween,roughlyequallyspacedHowever,therewereexceptionstothispolicy.Incasesofverypoorqualityofimagesonlyoneortwoframeswereannotated.AnexampleofannotatingasigninfourframesinwhichitappearsisshowninFigure3.3.2.ConstraintsandproblemsAsthevideosweareusingwerelmedforthepurposeofroadmaintenance,theyweregenerallylmedwhentheweatherwasreasonablygood.Ofcourse,thetaskoflmingforroadmaintenancehastobenishedinreasonabletime(usuallyonemonth),soitispossibleforsomevideostobelmedduringbadweather.However,thevastmajorityofvideosinourcurrentcollectionwerelmedwhentheweatherwassunny,andallofthemwerelmedduringdaytime.Thereareseveralfactorsinuencingsignappearanceinthevideos:ShadowsAstheweatherinourvideosisgenerallysunny,lotsofsignsarepartiallyshaded.ColorchangesDependingonthesunpositionrelativetothesign,somecolorsmayappeardarkenedorlightened.InterlacingeffectsOneofthecameraswithwhichthevideosweretakenusesinterlacing.MotionblurDuetothemotionofthelmingvehicle,lotsofvideossufferfrommotionblur. Ourdecisionwasthereforetorsttrytodetectsignswitharobust,generalpurposedetector.Thepossibilityofusingshapeandcolorinformationwasleftforfutureexploration.ThedetectorwechosewastheoneproposedbyViolaandJones[26].4.SigndetectionmethodandresultsIftwosignsbelongtothesamesemanticgroup,itdoesnotnecessarilymeanthattheyarevisuallyalike.Forexample,Figure5showssixinformationsigns.Althoughallareusedforconveyingnon-criticalinformationtothedriver,noneisgraphicallysimilartotheothers.Theirshapesandcolorsvarytremendously. Figure5.Examplesofinformationtrafcsigns.Althoughtheirsemanticuseisthesame–conveyinginformationtothedriver–theirappearancecanbeverydifferent.Ourgoalisdevelopingatrafcinfrastructureinventorysystemwhichshould,byitsnature,becapableofdetectingandrecognizingalargevarietyofdifferenttrafcsigns.Obviously,inthenalsystemeachvisuallysimilargroupofsignswillhavetohaveadedicateddetector.Itwillbenecessarytoanalyzetheappearanceofallsignsindetailandgroupthemaccordingtosomeruledependingonthedetectorused.Forinstance,ifusingacircularshapedetector,allcircularsignswouldbeinonegroup,regardlessoftheirsemanticmeaning.Thereis,however,asemanticclassofsignswhoseappearanceisquiteconsistent:warningsigns.Warningsignsarealwaystriangularinshapeandhaveathickrededgeandwhitebackground.Theydifferonlyintheideogramsused.This,alongwiththefactthatourdatacollectioncurrentlyhasmorewarningsignimagesthananyother,madethemidealcandidatesfortestingtheViola-Jonesdetector.Thedetectorwastrained2on824imagesthatcontain898warningsigns,assometimestwowarningsignsappeartogether.Thebaseresolutionofthedetectorwassetto2424pixels,sotheannotatedsignswereextractedfromimagesandscaledtomatchthatsize.Asmentionedpreviously,whileannotating,aboundingboxwasplacedtightlyaroundthesign.3ThetrainingprocessusedasetoffeaturesshowninFigure6.WeexperimentedwithusinganextendedsetoffeaturesproposedbyLienhartandMaydt[14],butthebasicsetoffeaturesprovedtobebetter.Thefeaturepoolcontainedatotalof85848features.TheboostingalgorithmusedwasGentleAdaBoost,asexperimentsbyLienhartetal.[13]indicateitoutperformsbothDiscreteAdaBoostandRealAdaBoost. Figure6.Haar-likefeaturesusedintrainingthedetector. 2TheimplementationfromtheOpenCVlibrarywasused.3Weperformedequivalentexperimentsusingwiderboundingboxes.Thehitrateremainedthesame,butthenumberoffalsepositivesdoubled. TheViola-Jonesdetectorisacascadeofboostedclassiers.Someconstraintsweresetonthetrainingprocessfortheindividualstagesofthecascade:Minimumhitrate.Minimumhitrateforacascadestagewassetto0.995.Thismeansthat99.5%ofallpositiveexamplespassingthroughthecascadestageshouldbedetected.Thetrainingprocesscontinuestoaddmorefeaturestothestageclassieruntilminimumhitrateisreached.Maximumfalsealarm.Maximumfalsealarmrateperstagewassetto0.5.Outofalldetectionsmadebyastageofthecascade,only50%maybefalsepositives.Numberofstages.20stagesofthecascadeweretobetrained.Settingthefalsealarmrateto0.5perstageandthetotalnumberofstagesto20yieldsatotalfalsealarmrateof0:520=9;5310�7.Duringtraining,thisfalsealarmratehasbeenreachedbeforethe20thstage,sothetrainingwasterminated.Thetrainingtookthreedaysonaserverwithtwo3GHzdual-coreCPUs,withbothCPUsbeingutilized.Theresultingdetectorconsistsof17cascadestagesandusesatotalof299features. Figure7.TopfourfeaturesoftherstvestagesoftheViola-Jonesdetectorcascade.Thestagesaredistributedonthehorizontalaxis,whiletherstfourfeaturesusedbyeachstageclassierareshownontheverticalaxis.ItisinsightfultoanalyzemainHaar-likefeaturesusedbythedetector.Figure7showstherstfewfeaturesselectedintrainingtheearlystagesofthecascade.Therststageofthecascadeusesfeatureseasilyinterpretablebyahuman:afeaturesensitivetothebottomedgeofasignandthreefeaturessensitivetostructurechangesnearthetopvertexofthesign.Thisfactcanbevisualisedbysuperimposingthefeaturesonimagesoftypicalsigns,asshowninFigure8.Thetraineddetectorwastestedontwoimagesets.Testset1consistsof91imageswhichcontain101warningsigns.Testset2consistsof68imagesof72warningsigns.Noneoftheimagesfromthetestsetshavebeenusedintrainingthedetector. Figure8.Therstfourfeaturesofcascadestage1superimposedonsignimages.TheViola-Jonesdetectorworksbyslidingadetectionwindowacrosstheimage.Afterreachingtheendoftheimage,thewindowisenlargedbythescalefactorandtheprocessrepeats.Wetestedthedetectorfortwodifferentscalefactors:1.05and1.20.TheresultsaresummarizedinTable1.Overall,thedetectorperformswell,achievingmorethan90%hitsonbothtestingsets.Itcanbeseenthatasmallerscalefactorinducesmodestgainsinthehitrateandsignicantlymorefalsepositives.Ofcourse,usingasmallerscalefactoralsoimpactsdetectionspeed-onaverage,ourbestdetectorranat3fpswithscalefactor1.05,andatabout9fpswithscalefactor1.20. Testset Scalefactor Signs Hits Misses Falsepositives [%testset] [%testset] [%testset] 1 1.05 101 96% 4% 84% 1 1.20 101 93% 7% 42% 2 1.05 72 93% 7% 163% 2 1.20 72 90% 10% 53% Table1.Experimentalresultsfortestingthetraineddetectorontwodifferenttestsets.Performancewastestedfordetectorscalefactorsof1.05and1.20.Themissesindetectionusuallyocurredwhentheobservedsignsweretoofarawayfromthecameraorwhenonlyapartialdetectionofasignwasmade.However,asstatedpreviously,eachsignwasan-notatedinfourframes,whichmeansthatforeachphysicalsigntherearefourimagesinthedatabase.Wenoticedthatthesignsthatweremissedinoneimageweremostoftendetectedinatleastoneoftheremainingthree.Consideringthatoursystemfortrafcinfrastructureinventorywillworkwithvideos,itwillhavetheopportunitytodetectasigninmorethanonehundredframes.Therefore,wendtheobservedrateofmissesindetectionacceptable.Ofmuchgreaterconcernisthelargenumberoffalsepositivedetections.Succesfulremovaloffalsepositivesrequiresunderstandingoftheirorigin,sofalsepositiveswereanalyzedindetail.Itwasfoundthatthemajorityoffalsepositivesbelongtooneofthefollowingcategories:detectionsintreesorgrass,linesonaroad,triangularstructuresinthetrafcscene,partsoftrafcsignsandroofstructures(seeFigure9).Proportionsofdifferentfalsepositivecategoriesintestset1areshowninFigure10.5.DiscussionandfutureworkTheViola-Jonesdetectorshowspromisingresults.However,thenumberoffalsepositivedetectionsisaconcern.Afteranalyzingcausesforfalsepositives,severalstrategiesfortheirremovalseempromising.Ofcourse,everyemployedstrategywillhavetousetheaprioriknowledgeoftheproblem. (a) (b) (c) (d) (e)Figure9.Falsepositivesoutlinedbythedetector.Oneimagecorrespondstoonedetectedboundingbox.Cate-gories:(a)tree,(b)roadline,(c)triangularstructure,(d)apartofasign,(e)roofresemblingatriangle Figure10.Causesforfalsepositivesintestset1,withscalefactorsetto1.05Asstatedpreviously,appearanceofatrafcsignisstrictlyconstrained.Modelingthatappearanceinsomeway,preferablythroughuseofshapeandcolor,couldeliminatemostofthefalsepositives.Forinstance,Figure10showsthat33%offalsepositivesintestset1occurintreesorgrass.Inspiteofundesirablecolorchangesduetolighting,thetreesandgrassinourimagesstilllookpredominantlygreen.Therefore,simplylteringpredominantlygreendetectionsmightbeenoughtoeliminate33%offalsepositives.Similarly,theproblemofdetectinglinesontheroadcouldbeeliminatedbylteringgray-coloreddetections,astheroadalwaysappearsgray.Thiswouldreducethefalsepositiveratebyanother13%.Asthesizeofasignisxed,detectionswiderthansomespecicthresholdcouldalsobedisregarded.However,therearemorecomplexcasesoffalsepositives.Triangularstructuresareperhapsthemostdifculttoeliminate.Generally,theycanbepositionedanywhereinthesceneandhaveanyappearance.Therefore,itisimpossibletorecognizethemasfalsebyusingsimplecolorconstraints.Weplantoeliminatethembydevisingamodelofasignthattakesintoaccountbothshapeandappearance.Attemptingtotthismodelwouldtheneasilydetectallothersimplerfalsepositivesliketheaforementionedtrees,asthemodelwouldcertainlyincludecolorconstraints.Also,usingthismodelindependentlyoftheViola-Jonesdetectorandthenfusingtheresultsofthetwomightinducelargerhitrates.Falsepositivescouldalsobeeliminatedbyintroducingcontextualconstraints.Notonlyistheappear-anceofasigndenedveryprecisely,butitslocationinthesceneisalsodenedveryprecisely.Asigncanappeareitherbythesideoftheroadorsomewhereabovetheroad.Thesizeofasignrelativetothedistancetothecameraisalsoconstrained:asignthatisfarfromthecamerashouldappearsmallerthantheoneclosetothecamera.Toenforcetheseconstraints,werstneedtoestablishsomebasicgeometryofthescene.Thesimplestwaytodoitistodetecttheroadintheimage.Knowingtheroad position,wecoulddisregarddetectionsontheroad,detectionstoofarawayfromtheroad,aswellastoolargedetectionstooclosetothevanishingpoint.Introductionofthisgeometrytotheproblemisthesubjectofourcurrentresearch.Instudyingtheeliminationoffalsepositivesitisnecesarrytokeepinmindthatwearenotlimitedtooneimageofasign.Thesystemfortrafcinfrastructureinventorywearedevelopingwillworkwithvideos,whichmeanshundredsofframescontainingasinglesignwillbeavailable.Thesystemwillhaveatrackingmodulewhichwillenabletrackingthedetectionthroughmultipleframes.Itislikelythatthistrackingwilldecreasethefalsedetectionsthatareresultsofrandomnoise.Toconclude,wehaveobtainedpromisingresultsinusingtheViola-Jonesobjectdetectorfordetectingtriangulartrafcsigns.Ourfurtherworkwillgoindirectionsofaddingcontextualconstraintsandmodelingsignappearanceinordertoreducethenumberoffalsepositivesandincreasethedetectionrate.WebelievethatthegeneralpurposeobjectdetectorproposedbyViolaandJonesaugmentedwithasignmodelmightbethesolutionfortrafcsigndetectioninvideoslmedinadverseilluminationconditionswithlow-qualitycameras.References[1]P.Arnoul,M.Viala,J.P.Guerin,andM.Mergy.Trafcsignslocalisationforhighwaysinventoryfromavideocameraonboardamovingcollectionvan.IntelligentVehiclesSymposium,1996.,Proceedingsofthe1996IEEE,pages141–146,Sep1996.[2]C.Bahlmann,Y.Zhu,VisvanathanRamesh,M.Pellkofer,andT.Koehler.Asystemfortrafcsigndetection,tracking,andrecognitionusingcolor,shape,andmotioninformation.IntelligentVehiclesSymposium,2005.Proceedings.IEEE,pages255–260,June2005.[3]X.BaroandJVitria.Fasttrafcsigndetectionongreyscaleimages.RecentAdvancesinArticialIntelligenceResearchandDevelopment,pages69–76,October2004.[4]M.BenallalandJ.Meunier.Real-timecolorsegmentationofroadsigns.ElectricalandCom-puterEngineering,2003.IEEECCECE2003.CanadianConferenceon,3:1823–1826vol.3,May2003.[5]A.Broggi,P.Cerri,P.Medici,P.P.Porta,andG.Ghisio.Realtimeroadsignsrecognition.IntelligentVehiclesSymposium,2007IEEE,pages981–986,June2007.[6]Sin-YuChenandJun-WeiHsieh.Boostedroadsigndetectionandrecognition.MachineLearn-ingandCybernetics,2008InternationalConferenceon,7:3823–3826,July2008.[7]A.delaEscalera,L.E.Moreno,M.A.Salichs,andJ.M.Armingol.Roadtrafcsigndetectionandclassication.IndustrialElectronics,IEEETransactionson,44(6):848–859,Dec1997.[8]S.EscaleraandP.Radeva.Fastgreyscaleroadsignmodelmatchingandrecognition.RecentAdvancesinArticialIntelligenceResearchandDevelopment,pages69–76,2004.[9]L.EstevezandN.Kehtarnavaz.Areal-timehistographicapproachtoroadsignrecognition.ImageAnalysisandInterpretation,1996.,ProceedingsoftheIEEESouthwestSymposiumon,pages95–100,Apr1996. [25]S.Varun,SurendraSingh,R.SanjeevKunte,R.D.SudhakerSamuel,andBinduPhilip.Aroadtrafcsignalrecognitionsystembasedontemplatematchingemployingtreeclassier.InICCIMA'07:ProceedingsoftheInternationalConferenceonComputationalIntelligenceandMultimediaApplications(ICCIMA2007),pages360–365,Washington,DC,USA,2007.IEEEComputerSociety.[26]PaulViolaandMichaelJones.Robustreal-timeobjectdetection.InInternationalJournalofComputerVision,2001.[27]J.Ph.AndreuW.Benesova,Y.Lypetskyy,L.Paletta,A.Jeitler,andE.H¨odl.Amobilesystemforvisionbasedroadsigninventory.InProc.5thInternationalSymposiumonMobileMappingTechnology,Padova,Italy,May2004.[28]ShuangdongZhuandLanlanLiu.Trafcsignrecognitionbasedoncolorstandardization.In-formationAcquisition,2006IEEEInternationalConferenceon,pages951–955,Aug.2006.