DeepDual-ChannelNeuralNetworkforImage-BasedSmokeDetectionKeGu,Member,I - PDF document

DeepDual-ChannelNeuralNetworkforImage-Ba
DeepDual-ChannelNeuralNetworkforImage-BasedSmokeDetectionKeGu,Member,IEEE,ZhifangXia,JunfeiQiao,Member,IEEE,andWeisiLin,Fellow,IEEE—Smokedetectionplaysanimportantroleinindus-trialsafetywarningsystemsandreprevention.Duetothecomplicatedchangesintheshape,textureandcolourofsmoke,identifyingthesmokefromagivenimagestillremainsasubstan-tialchallenge,andthishasaccordinglyarousedaconsiderableamountofresearchattentionrecently.Toaddresstheproblem,wedeviseanewdeepdual-channelneuralnetwork(DCNN)forsmokedetection.Incontrasttopopulardeepconvolutionalnetworks,e.g.,Alex-Net,VGG-Net,Res-Net,andDense-Net,andtheDNCNNthatisspecicallydevotedtodetectingsmoke,ourproposedend-to-endnetworkismainlycomposedofdualchannelsofdeepsubnetworks.Intherstsubnetwork,wese-quentiallyconnectmultipleconvolutionallayersandmax-poolinglayers.Then,weselectivelyappendthebatchnormalizationlayertoeachconvolutionallayerforover-ttingreductionandtrainingacceleration.Therstsubnetworkisshowntobegoodatextractingthedetailinformationofsmoke,suchastexture.Inthesecondsubnetwork,inadditiontotheconvolutional,batchnormalizationandmax-poolinglayers,wefurtherintroducetwoimportantcomponents.Oneistheskipconnectionforavoidingthevanishinggradientandimprovingthefeaturepropagation.Theotheristheglobalaveragepoolingforreducingthenumberofparametersandmitigatingtheover-ttingissue.Thesecondsubnetworkcancapturethebaseinformationofsmoke,suchascontours.Wenallydeployaconcatenationoperationtocombinetheaforementionedtwodeepsubnetworkstocomple-menteachother.Basedontheaugmenteddataobtainedbyrotatingthetrainingimages,ourproposedDCNNcanpromptlyandstablyconvergetotheperfectperformance.ExperimentalresultsconductedonthepubliclyavailablesmokedetectiondatabaseverifythattheproposedDCNNhasattainedaveryhighdetectionratethatexceeds99.5%onaverage,superiortostate-of-the-artrelevantcompetitors.Furthermore,ourDCNNonlyemploysapproximatelyone-thirdoftheparametersneededbythecomparativelytesteddeepneuralnetworks.ThesourcecodeofDCNNwillbereleasedathttps://kegu.netlify.com/.IndexTerms—Smokedetection,deeplearning,convolutionalnetwork,dual-channelnetwork,classicationThisworkwassupportedinpartbytheNationalScienceFoundationofChinaunderGrant61703009,theYoungEliteScientistSponsorshipProgrambyChinaAssociationforScienceandTechnologyunderGrant2017QN-RC001,theYoungTop-NotchTalentsTeamProgramofBeijingExcellentTalentsFundingunderGrant2017000026833ZK40,andtheNationalScienceandTechnologyMajorProjectoftheMinistryofScienceandTechnologyofChinaunderGrant2018ZX07111005.(Correspondingauthor:KeGu.)K.GuandJ.QiaoarewiththeBeijingAdvancedInnovationCen-terforFutureInternetTechnology,BeijingKeyLaboratoryofComputa-tionalIntelligenceandIntelligentSystem,FacultyofInformationTech-nology,BeijingUniversityofTechnology,Beijing100124,China(email:guke.doctor@gmail.com;junfeiq@bjut.edu.cn).Z.XiaiswithBeijingAdvancedInnovationCenterforFutureInternetTechnology,FacultyofInformationTechnology,BeijingUniversityofTech-nology,Beijing100124,China,andalsowiththeStateInformationCenterofP.R.China,Beijing,China(email:spidergirl21@163.com).W.LiniswiththeSchoolofComputerScienceandEngineering,NanyangTechnologicalUniversity,Singapore,639798(email:wslin@ntu.edu.sg).I.INTRODUCTIONTisanurgenttasktopromptlyandeffectivelydetectthesmokeforindustrialautomationandresafetywarningsystems,suchastorchblacksmokedetectioninpetrochem-icaleldsandforestrewarnings.Existingapproachesfortorchblacksmokedetectionandpyrotechnicdetectionmainlydependonmanualobservationorsensors.Becauseoflimitedhumanresources,popularmanualobservation-basedmethodscannotbeusedtomonitorsmokerapidlyandvalidlyinthelongterm,particularlygivenintermittentinterruptionsordistractions.Ontheotherhand,smokesensorsthatarebasedonsmokeparticlesamplingorrelativehumiditysamplingareverylikelytocauseaseveretimedelay;moreover,theycan-notsimultaneouslyandcompletelycoverthedetectionareaswhenappliedtodetectingsmokebecauseoftheinuencesofenvironmentalvariations.Overall,existingsmokedetectionmethodsmeetwithdifcultiesinsatisfyingtherequirementsoftoday'sindustrialprocessesandsafetywarnings.Duringthelastseveralyears,image-basedsmokedetectionmethodshavebeenbroadlyexploredtosolvesuchproblems

.In[1],motivatedbyanobservationthatsmoke
.In[1],motivatedbyanobservationthatsmokeaffectsthehigh-frequencyinformationofanimage'sbackgroundarea,Toreyinetal.proposedamethodthatappliedthespatialwavelettransformtomeasurethehigh-frequencyenergylossofthesceneforsmokedetection.In[2],byintroducingwaveletdecompositionsandasupportvectormachine(SVM),Gubbietal.developedasmokecharacterization-basedtechniquetoidentifysmokefromavideosequence.In[3],Yuandevisedafastaccumulativemotionorientationmodelbasedonanintegralimageforvideosmokedetection.In[4],Yuandevisedavideo-basedsmokedetectionapproachusinghistogramsofthelocalbinarypattern(LBP)andlocalbinarypatternvariance(LBPV).In[5],Yuanputforwardasmokedetectionmethodbylearningshapeinvariantfeaturesonmulti-scalepartitionswithAdaBoost.Notethatthisisthersttimethatthemulti-scalestrategywasusedtoamendtheperformanceofsmokedetectionbyasizablemargin.In[6],Yuanetal.aclassicationalgorithmforsmokeimagesbasedonahigh-orderlocalternarypattern(LTP)withlocalpreservationpro-jection,andthisalgorithmhasledtoanoticeableperformancegaincomparedwiththeexistingrelevantmethods.In[7],Yuanetal.incorporatedtheLBP,kernelprincipalcomponentanalysis(KPCA)andGaussianprocessregressionfordetect-ingsmoke.Forthereaders'convenience,wesummarizetheimage-basedsmokedetectionalgorithmsillustratedaboveinTableI.ItisnotdifculttodeterminethatmostoftheexistingsmokedetectiontechnologiesareonlybasedontheanalysisTABLEI:Summaryofimage-basedsmokedetectionmethods.ReferenceDescription[1]Spatialwavelettransform+High-frequencyenergylossMulti-scale+LBP+LBPV+HistogramsofpyramidsHigh-orderLTP+LocalpreservationprojectionLBP+KPCA+GaussianprocessregressionCNN+Batchnormalization+Fullconnectionandsynthesisoftexturalinformation.Todate,themajorityofexistingsmokedetectionmodelshavebeendevelopedbyusinghand-craftedfeaturesforiden-tifyingsmoke.Nonetheless,suchstudiesmightencounterabottleneckbecausethemanuallyextractedimagefeaturesarestillofinsufcientabilitytocharacterizethecomplicatedvari-ationsinthesmokeimages[1]-[7].Incontrast,deeplearningisverypossiblyabettersolutionforsmokedetectionsincerecentyearshaveseenanunmatchedperformanceattainedbydeeplearning,particularlyinpatternrecognitionapplicationssuchasimagerecognitionandimageclassication[8]-[9].Multipleprevailingdeepconvolutionalnetworksparticipatedinthewell-knownILSVRC(ImageNetLargeScaleVisualRecognitionChallenge[10])competitionclassicationprojectandhavemadesignicantbreakthroughs.Forexample,sometypicalnetworks,suchasAlex-Net[11],ZF-Net[12],VGG-Net[13],GoogLe-Net[14],Xception[15],andRes-Net[16],illustratethatneuralnetworkshavebecomeincreasinglydeep-er,fromafewlayerstomorethanonehundredlayersduringthepastseveralyears.Moreover,deeplearninghasalsobeensuccessfullyappliedtonumerousmultimediaapplicationsre-cently[17]-[21].Despitethegreatachievementsobtainedbydeeplearning,verylimitedefforthasbeendevotedtothesmokedetectiontask.Tothebestofourknowledge,onlyonedeeplearning-basedmodelexistsfordetectingsmoke.Morespecically,in[22],Yinetal.developedadeepnormalizedconvolutionalneuralnetwork(DNCNN)forsmokedetectionfromimages.TheDNCNNimposestwomainimprovementsonasequentialconvolutionalneuralnetwork.OneisthattheDNCNNembedsthebatchnormalization(BN)[23]intotheconvolutionallayerforalleviatingthegradientdispersionandover-ttingproblemswhentrainingthenetwork.TheotheristhattheDNCNNadoptsthedataaugmentationtechnologytosettletheproblemofpositiveandnegativesampleimbalanceandinsufciencythatoccursinthetrainingsamples.ThesetwoimprovementshavepromotedtheperformanceofDNCNNtoahighlevelbeyond97%.However,suchdetectionaccuracyisunabletomeetourrequirementsbecausethepoisonoussmokeemittedintotheairduetotheimperfectsmokedetectiontechnologyisharmfultosafeguardinglifeandtheenvironment;inotherwords,achievingaperfectdetectionperformanceof100%isouruniqueandnever-endingpursuit,similartothegoalforautonomousvehicles[24].Tofurtherenhancetheperformanceandrobustnessofsmokedetection,inthispaperweputforwardanoveldeepdual-channelneuralnetwork,dubbedDCNN.ForagivenWealsoincludeDNCNNasanimage-basedsmokedetectorinTableI.TABLEII:Importantsymbolsandimplications.image,werstdividetheinputimageintopatches,andthenseparatelyidentifyeachpatchbasedontheproposedDCNN.Bysuch

processes,wecanconvertthetaskofdetecting
processes,wecanconvertthetaskofdetectingsmokeintoatwo-categoryclassicationproblem,i.e.,smokepatchandsmoke-freeversion.Theproposedend-to-endDCNNismainlyestablishedbyemployingdualchannelsofdeepsubnetworks.Multipleconvolutionallayersandmax-poolinglayersaresequentiallyconnectedtogeneratetherstchannelofthesubnetwork.Toalleviatetheover-ttingproblemandacceleratethetrainingprocess,weintroducetheBNopera-tions.WeselectivelyappendtheBNlayertoeachofthelastfourconvolutionallayerssinceitwasfoundthattheBNlayerisverylikelytorestrictthefreedomofextractedfeatures[25]-[26].Therstchannelofthesubnetworkisshowntobegoodatextractingthedetailinformationofsmoke.Next,thesecondchannelofthesubnetworkisconstructedbyincorporatingtwonewsignicantcomponentswiththeconvolutional,BN,andmax-poolinglayers.Onecomponentistheskipconnection,whichcontributestopreventingthegradientvanishingandenhancingthefeaturepropagation.Theothercomponentistheglobalaveragepooling,whichisbenecialindecreasingthenumberofparametersandmitigatingtheproblemofover-tting.Itwasfoundthatthesecondchannelofthesubnetworkiscapableofcapturingthebaseinformationofsmoke.Eventually,weconstructtheDCNNbyintroducingaconcatenationoperationtofusethefeaturesextractedusingtheaforementionedtwodeepsub-networks.Bycomplementingeachother,theconcatenationoperationcancondensetheextractedfeaturesandenabletheirstrongerrepresentationability.OurDCNNislearnedbasedontheaugmentedtrainingdata,whicharegeneratedbyrotating1520-9210 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMM.2019.2929009, IEEETransactions on Multimedia3Fig.1:ThebasicarchitectureofSBNN,includingsixconvolutionallayers,threemax-poolinglayers,fournormalizationlayers,andthreefully-connectedlayers.TheimplicationsofsymbolscanbefoundinTableII.`Conv',`MaxP',`NaC',and`FC'standforconvolution,max-pooling,normalizationandconvolution,andfullconnectionoperations,respectively.Fig.2:ThebasicarchitectureofSCNN,includingelevenconvolutionallayers,threemax-poolinglayers,sevennormalizationlayers,andoneglobalaveragepoolinglayer.TheimplicationsofsymbolscanbefoundinTableII.`Conv',`MaxP',`NaC',and`GAP'standforconvolution,max-pooling,normalizationandconvolution,andglobalaveragepoolingoperations,respectively.trainingimagepatches.ExperimentsdemonstratethatourproposedDCNNleadstonoticeableimprovementbyboostingtheperformanceofsmokedetectionandloweringthenumberofmodelparameterscomparedwithrecentlyproposeddeepnetworksincludingAlex-Net[11],ZF-Net[12],VGG-Net[13],GoogLe-Net[14],Xception[15],Res-Net[16],Dense-Net[27],andDNCNN[22].Wehighlightthemainnoveltyandcontributionofthisworkcomparedwithexistingimage-basedsmokedetectionmethodsasfollows.First,fromtheviewpointofthedesignprinciple,thispaperistherstworkthatintegratesthelow-levellocaltexturalcharacteristicsandthehigh-levelglobalcontourinformationfordetectingsmokefromimages.Further,byusingimagedecomposition,westraightforwardlyillustratethenecessityoffusingtheaforesaidtwocomponentsinsmokedetection.Second,fromtheaspectofnetworkstructure,thispaperistherstworkthatdesignsadual-channeldeepneuralnetworkforeffectiveandefcientsmokedetection.Specically,toimprovethedetectioneffectiveness,weinserttheskipconnectionintoasequentialconvolutionalnetworkforcapturingcontourinformationandintroducethefeaturefusionlayerforcomprehensivelysynthesizingtexturalcharacteristicsandcontourinformation.Wereplacethefully-connectedlayerswithasimpleglobalaveragepoolingtolargelyreducethenumberofmodelparameters,andthusenhancetheefciencyduringtrainingandtesting.Last,fromtheperspectiveofdetectionperformance,ourproposedDCNNattainsveryhighaccuracybeyond99.5%onaverage,resultinginarelativeperformancegainofapproximately1%comparedwiththesecond-rankmodel.Thestructureofthispaperisoutlinedasfollows.SectionIIillustratesthedetailsconcerningthenetworkarchitecture,parametersettings,etc.InSectionIII,thesu

periorityofourproposedDCNNisvalidatedbyc
periorityofourproposedDCNNisvalidatedbycomparisonwithstate-of-the-artdeeplearningmodelsandrecentlyproposedsmokedetectionmethods.Furthermore,wespecicallydiscusshowthetwodeepsubnetworkscomplementeachother.SectionIVsummarizesthewholepaper.II.PROPOSEDEEPEURALETWORKRecentyearshaveseenagrowingnumberofmultimediatechnologiesthatwereappliedforresolvingenvironmentalproblems,e.g.smokedetection[22],PM2.5monitoring[28]-[29],andairqualityforecast[30]-[32].TheproposedDCNN,particularlydevisedfordetectingsmoke,willbedescribedindetail.Asillustratedearlier,ourDCNNiscomposedofdualchannelsofdeepsubnetworks.WearrangethewholeSectionIIbyrstintroducingthetwodeepsubnetworks,namely,theselective-basedbatchnormalizationnetwork(SBNN)andskipconnection-basedneuralnetwork(SCNN).Then,wepresenthowtoreasonablycombinetheabovesubnetworkstobuildtheDCNN.Last,weillustratethedetailsofnetworktraining.Fortheconvenienceofreaders,wepresentimportantsymbolsandtheassociatedimplicationsinTableII.A.SBNN'sArchitectureBuiltontheconvolutionalneuralnetworkandinspiredbytherecentlyproposedDNCNN,weestablishtheSBNN,asgiveninFig.1.First,wesequentiallyconnectsixconvolutionallayersandthreemax-poolinglayersforfeatureextraction.Convolutionisacommonlyusedoperationforcapturinglocalinformationtogenerateatensorofoutputs.Theconvolutionallayerconsistsoffeaturemaps,denotedas=1


;n.Eachfeaturemapintheconvolutionallayer,=1


;n,isconvolved1520-9210 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMM.2019.2929009, IEEETransactions on Multimedia4Fig.3:ThebasicarchitectureofDCNN,includingthedualchannelsofdeepsubnetworksforfeatureextraction,aconcatenationoperationforfeaturefusion,aswellasaconvolutionlayerandaglobalpoolinglayerforclassication.`GAP'standsfortheglobalaveragepoolingoperation.Forthesakeofclarity,weremovetheoperationsymbolsfromtheoriginalsubnetworksandonlykeepthefeatureblocks.ThetopchannelistheSBNNandthebottomchannelistheSCNNwiththelterqpandaddedtothebias,followedbyanon-linearactivationfunction=qp;p=1


;nwhere“”indicatestheconvolutionoperation.Theactivationfunctionleveragestherectiedlinearunit(ReLu)functionsinceitismoreconsistentwiththecharacteristicsofbiologicalneurons[33].Themax-poolingtargetstolearnbiologicallyplausiblefeaturesbyactivatingthelocalmaximumresponse.Themainmeritsofmax-pooingaretheinvarianceoftransla-tion,rotationandscale,aswellasthereductioninthenumberofnetworkparameters.Inourimplementation,weselectthemaximumactivationvalueoverasmallpoolingregion.Then,weselectivelyappendtheBNlayertoeachofthelastfourconvolutionallayers.Whentrainingdeepconvolu-tionalneuralnetworks,themostcommonlyusedoptimizationmethodisthemini-batchstochasticgradientdescent(SGD).However,theinternalcovariateshift,namely,thevariationsofinternalinputdistributionsduringtraining,usuallyreducesthetrainingefciencyseriously.TheBNwasproposedtoresolvesuchlimitationsofconvolutionallayers[23].Bytransforminginternalinputswithascaleandshiftsteppriortonon-linearactivation,theBNcanvalidlyspeedupthenetworktrainingandpreventparameterover-tting.Morespecically,basedonthemini-batchmeanandvariance,eachfeaturenormalizedasfollows:jq2j+(2)wherej=1j;ij;irespectivelythemini-batchmeanandvariance,withthesizeofamini-batchandj;ibeingthe-thfeatureofthethsampleinthemini-batch.isaxedsmallpositivenumberusedforpromotingnumericalstability.However,normalizingtheinputfeaturesmightdecreasetheirrepresentationcapabil-ity.TwofreeparametersarethusintroducedtosettlesuchproblemsbytransformingnormalizedfeaturesviaascaleandshiftstepintheBN:)=\f:ThereasonwhywedonotappendtheBNlayerstothersttwoconvolutionallayersoriginatedfrominsightsinarecentstudy,whichimpliedthattheBNmakestheextractedfeaturesfreelyconstrained[25].Therefore,weremovetheBNintherstandsecondconvolutionallayerstofacilitatebetterprotectionofthes

mokecharacteristicsoftheimagepatch.Weext
mokecharacteristicsoftheimagepatch.Weextractthefeaturesfromagivenimagepatchbasedontheabove-mentionedoperations.Then,weappendthreefully-connectedlayersattheendofthelastmax-poolinglayerSufcientexperimentsindicatedthatthefullconnectionop-erationcaneasilycauseover-ttingsincethefully-connectedlayersoftencontainasubstantialnumberoflearnableparam-eters.Atypicalsolutiontoovercomesuchaproblemistointroducethedropouttechnique[34].Inourimplementation,asshowninFig.1,therstfully-connectedlayerreceivesallfeaturemapsofastheinputneuronstoyieldthefeaturemapsofLikewise,wecanderivetheoutputofthesecondandthirdfully-connectedlayers,namely,thefeaturemapsTheoutputlayerconsistingoftwoneuronsproducestwoclassesofprobabilities,=[^.Theoutputprobabilityofthe-thneuronforthe-thclassiscalculatedusingthesoftmaxfunction:;u=1Furthermore,itisnoteworthythat,comparedwiththeDNC-NN,ourproposedSBNNhastwodominantimprovements:1)amorecompactstructure,and2)aselective-basedbatchB.SCNN'sArchitectureOnthebasisofSBNN,theSCNNinthesecondchannelfur-therintroducestheskipconnectionandglobalaveragepooling,asshowninFig.2.First,weconnectelevenconvolutionallay-ers,sevenBNlayers,andtwomax-poolinglayerstoconstructasequentialnetworkforextractingfeatures.Notethat,akintotheSBNN,theBNlayerisnotappendedtotheformertwoconvolutionallayersforbetterfeatureprotection.Thekernelsizeoftherstconvolutionlayerisassignedasnineforextractingricherimagefeatureswithoutseriouslyincreasingnetworkparametersandthatofthesecondconvolutionlayerissetasoneforthepurposeofmergingthefeaturesextractedfromthefrontlayerwithoutchangingthefeatures'structure.Further,theBNlayerisalsonotappendedtothesixthandeleventhconvolutionallayers(i.e.,sincethesetwolayersareusedtolowerthedimensionalitybyproperlyfusingthefeaturemaps.Apartfromtheoperationsmentionedabove,theSCNNhopsandconnectstherstfeaturetothefthfeaturemap(beforethemax-pooling)throughtheskipconnection.Thesetwofeaturemapsaremergedtogetherwithaconcatenationoperationfollowedbyaconvolutionallayerwithitskernelsizeofone:=max(0max(0G1;G5]+wheretheoperationoperationG1;G5]concatenatesthetwofeaturetogether.representtheweightsandbiasesofthesixthlayerforconvolution.Itisnotedthatthemergedfeaturemapcontaintheinitialsimplefeaturesandthecomplicatedfeaturesaftermultipleconvolutionlayers.Next,bythefollowingconvolution,BNandmax-poolingoperations,theredundantfeaturesintheabovemergedfeaturemapcanberemoved.ComparedwiththeSBNN,thesecondchangeinSCNNistheglobalaveragepooling,whichreplacesthethreecumber-somefully-connectedlayers.Specically,insteadofthefully-connectedlayer,thispaperadoptsoneconvolutionlayer(withkernelsizeofoneandkernelnumberoftwo)followedbyasimpleglobalaveragepoolinglayertoyieldapairofaveragenumbers.Theglobalaveragepoolingiscomputedbythefollowing:standsforthe-thpixelvalueinthe-thfeaturearethewidthandheightof.Basedonthesoftmaxfunction,wecanderivetheoutputprobabilityofthe-thneuronforthe-thclasstobeasfollows:;u=1Suchareplacementcansecuretwoobviousadvantages:oneistolargelyreducemodelparametersandtherebymitigatetheover-ttingproblem;theotheristhattheSCNNisavailabletoaccommodatevarioussizesoftheinputimagepatch.(a)(b)(c)Fig.4:Examplesofdataaugmentation:(a)Pristineimagepatchesinthedatasetfortrainingthenetwork;(b)imagepatchesrotatedby90degrees;(c)imagepatchesrotatedby180degrees;and(d)imagepatchesrotatedby270degrees.C.DCNN'sArchitectureThroughsubstantialexperiments,itwasfoundthatboththeproposedSBNNandSCNNhaveattainedhighperformance.TheSBNNisgoodatextractingthedetailinformationofsmoke,whiletheSCNNcannicelycapturebaseinformationofsmoke.ItisnaturaltointegratetheadvantagesofSBNNandSCNNtoconstructthedual-channelDCNNforsmokedetection.Moreconcretely,weextractapartofSBNNbyeliminatingallthreefully-connectedlayers.TheextractedpartofSBNN,dubbedSBNN,isleveragedforfeatureextraction.Similarly,weextractSCNNbyremovingtheglobalaveragepoolinglayerfromtheSCNN.NotethatthesizeofSBNN'soutputisnotmatchedwiththatofSCNN'soutput.Hence,wefurthermodifySBNNbydeletingthethirdmax-pooling.Then,weincorporateSBNNandSCNNbymeansofaconcatenationoperationfollowedbyaconvolutionallayerwithitskernelsizeofone:=max(0max(0F8;G12]+Sofar,wehaveprovidedtheproposeddual-channelnetworkstructureforfeaturee

xtractionandfeaturefusion,asshowninthele
xtractionandfeaturefusion,asshownintheleftsideofFig.3.ItstillrequiressomelayersforclassicationinourDCNN.Itisapparentthatthefully-connectedlayersincludemuchmorelearnableparametersthantheglobalaveragepoolinglayer.Consequently,weappendtheglobalaveragepoolingtothelastconvolutionallayertocomputetwoMorediscussionsabouttheseconclusionswillbeillustratedinthenextTABLEIII:IllustrationofnetworkparametersofSBNN.Intheleftmostcolumn,arehighlightedinredinkinFig.1.LayerTypeNetworkparametersL1,L2ConvolutionFiltersize:33Filternumber:32Stride:11Padding:SameActivationfunction:ReLUL3PoolingPoolingregionsize:33Stride:22Padding:SamePoolingmethod:Max-poolingL4,L5NormalizationandconvolutionFiltersize:33Filternumber:64Stride:11Padding:SameActivationfunction:ReLUL6PoolingPoolingregionsize:22Stride:22Padding:ValidPoolingmethod:Max-poolingL7,L8NormalizationandconvolutionFiltersize:33Filternumber:384Stride:11Padding:SameActivationfunction:ReLUL9PoolingPoolingregionsize:22Stride:22Padding:ValidPoolingmethod:Max-poolingL10,L11FullConnectionNeuronsnumber:2048Dropout:0.5L12OutputNeuronsnumber:2TABLEIV:IllustrationofnetworkparametersofSCNN.Intheleftmostcolumn,arehighlightedinredinkinFig.2.LayerTypeNetworkparametersM1ConvolutionFiltersize:99Filternumber:32Stride:11Padding:SameActivationfunction:ReLUM2ConvolutionFiltersize:11Filternumber:64Stride:11Padding:SameActivationfunction:ReLUM3,M4,M5NormalizationandconvolutionFiltersize:33Filternumber:64Stride:11Padding:SameActivationfunction:ReLUM6ConcatenationandconvolutionFiltersize:11Filternumber:64Stride:11Padding:SameActivationfunction:ReLUM7PoolingPoolingregionsize:33Stride:22Padding:SamePoolingmethod:Max-poolingM8,M9NormalizationandconvolutionFiltersize:33Filternumber:128Stride:11Padding:SameActivationfunction:ReLUM10PoolingPoolingregionsize:33Stride:22Padding:SamePoolingmethod:Max-poolingM11,M12NormalizationandconvolutionFiltersize:33Filternumber:256Stride:11Padding:SameActivationfunction:ReLUM13ConvolutionFiltersize:11Filternumber:2Stride:11Padding:SameActivationfunction:ReLUM14PoolingPoolingmethod:GlobalaveragepoolingActivationfunction:Softmaxaveragenumbers.Then,weusethesoftmaxfunctiontoyieldtwooutputprobabilityvalues.TheaboveprocessescanbeimplementedbyreferringtoEqns.(7)-(8).Fig.3presentsthewholearchitectureofDCNN.D.NetworkTrainingDuringnetworktraining,rst,weindependentlytraineachofthedualchannels,namely,SBNNandSCNN.Forillustra-tion,considerthetrainingofSBNN.Wetakeadvantageofthetrial-and-errormethodtondtheoptimizednetworkstructure,astabulatedinTableIII.Wethenintroducetheglorotuniformmethodtoinitializethenetworkweights[35]andapplythemomentumandlearningratedecaytoadvancethetrainingeffectandpreventitfromfallingintothelocaloptimum[36].Morespecically,thestochasticgradientdescentisexploitedtotraintheSBNNbyassigningthemomentumcoefcientas0.9,theinitiallearningrateas0.01,andthelearningratedecaycoefcientas0.0001[37].Analogoustothemajorityofclassicationtasks,one-hotencodingisdeployedduringthetrainingofSBNNwiththelossfunctionofcrossentropy:)=log^=[;xrepresentsthevectoroftheclasslabel=[^representsthevectorofthecategoryprobability.Themini-batchsizeandthetrainedepocharesettobe96and300,respectively.Intheabove-mentionedenvironment,weadjusttheSBNN'smodelparametersbasedonthetrainingsetanddeterminetheoptimalmodelparametersbymakingthenetworkobtainthebestaccuracyonthevalidationset.ThesameprocessiscarriedouttotraintheSCNNbyminimizingthelossfunctionlog^.ItsoptimizednetworkstructureisshowninTableIV.,weincorporatetheSBNNandSCNNtoconstitutethewholeDCNN,asexhibitedinFig.3.WetraintheDCNNbyoptimizingpartofthenetworkparameters(i.e.,theconvo-lutionallayer)andfreezingtheothers(i.e.,SBNNThird,wene-tunetheoverallparametersofourproposedDCNNtosearchfortheoptimalparameters.Duringtheabove-mentionedtwosteps,weminimizethelossfunctionlog^Todecreasethevarianceofimagepatchesandimprovethenetwork'srobustness,wefurtherintroducetwoimagepreprocessingmethods,namely,patchnormalizationanddataaugmentation.Normalizationcaneffectivelydiminishthein-uenceofbrightnesschangesonsmokedetection.Thispaperusesthepixelwisemin-maxnormalizationmethodforimagepatchnormalization[3

8],whichiscalculatedbythefollowing:isth
8],whichiscalculatedbythefollowing:isthenormalizationvalueofapixel,theintensityvalueofapixel,andaretheminimumandmaximumvaluesofthepixelsintheimagepatch,respectively.Intheclassicationtask,therelativebalanceofdatabe-tweenthecategorieshasasignicantimprovementontheperformanceofthealgorithm[22].Forexample,inthedatasetfortrainingthenetwork,thereareapproximately2200smokeimagepatchesandapproximately8500smoke-freeimagepatchesintotal.By90-degree,180-degreeand270-degreerotations,thenumberofsmokeimagepatchesisincreasedtoasimilarnumberofthesmoke-freeimagepatches.Duetothecharacteristicsofsmoke,theseimagepatchescanbeconsiderednewsmokeimagepatchesacquiredbyrotationWetransferthewell-trainednetworkparametersinSBNNandSCNNtoandSCNNintheDCNN.TABLEV:Comparisonwithmodelsbasedonhand-craftedfeatures.MethodsHLTPMC[6]MCLBP[41]DCNN(Prop.)Set1AR96.4%96.9%99.7%DR97.7%97.6%99.5%FAR4.57%3.68%0.12%Set2AR98.4%97.8%99.4%DR98.5%98.4%99.0%FAR2.44%2.86%0.24%TABLEVI:ComparisonoftheproposedDCNNwitheightmainstreamorstate-of-the-artdeepconvolutionalneuralnetworks.NetworksAlex-NetZF-NetVGG-NetGoogLe-NetXceptionRes-NetDense-NetDNCNNDCNN[11][12][13][14][15][16][27][22]Prop.Set1AR95.6%96.0%96.8%97.0%97.9%97.2%98.6%97.8%99.7%DR94.9%93.6%95.2%95.8%96.7%95.1%98.3%95.2%99.5%FAR3.85%2.41%2.16%2.17%0.13%1.44%1.08%0.48%0.12%Set2AR96.9%97.6%97.9%98.1%98.4%98.1%98.4%98.0%99.4%DR96.5%97.9%97.9%97.2%98.0%97.4%98.2%96.3%99.0%FAR2.69%2.57%2.08%1.22%1.10%1.22%1.10%0.48%0.24%Numberofparameters60million60million120million7million20million60million7million20million2.7millionoperations.Thesmokeimagepatchesgeneratedbytheabovedataaugmentationtechnologyareassociatedwiththedifferentowdirectionsofthesmoke.Fortheconvenienceofreaders,Fig.4providesaugmentationeffectsofseveralrepresentativesmokeimagepatches.III.EXPERIMENTALESULTSThissectionwillconrmtheperformanceofourproposedDCNNfordetectingsmokeanddemonstrateitssuperioritycomparedwithstate-of-the-artrelevantcompetitors.Thissec-tioniscomposedoftheexperimentalprotocol,performancecomparison,implementationspeed,featuremapvisualization,discussion,andtestingofrealapplications.A.ExperimentalProtocolTensorFlow[39]andKeras[40]areusedinourexperimentfortrainingtheproposedDCNNforsmokedetection.TheexperimentalenvironmentistheWindows10operationsystemrunningonaserverwithanIntel(R)Corei7-7820XCPUat3.60GHzandanNVIDIAGeForceGTX1080.Inthistest,wedeploythepubliclyavailablesmokedetectiondatabase[22],whichiscomposedoffoursubsets,namely,Set-1,Set-2,Set-3andSet-4.Specically,Set-1(including831smoke-freeimagepatchesand552smokeimagepatches)andSet-2(including817smoke-freeimagepatchesand688smokeimagepatches)areutilizedforcheckingthedetectionperfor-manceofthenetwork.Set-3consistsof8804smokeimagepatches,whichiscreatedbyexertingthedataaugmentationontheoriginal2201smokeimagepatches,and8511smoke-freeimagepatchesfortrainingthenetwork.Set-4contains9016smokeimagepatches,whichwereproducedbyaugmentingtheoriginal2254smokeimagepatches,and8363smoke-freeimagepatchesforvalidatingthenetwork.TheleftmostcolumninFig.4presentsfourtypicalsmokeimagepatchescontainedinthesmokedetectiondatabase.Forquantifyingtheperformanceofourproposednetworkwithothers,weapplythreetypicalevaluationindicatorsthatincludeaccuracyrate(AR),detectionrate(DR)andfalsealarmrate(FAR),asdenedbythefollowing:AR=DR=FAR=arethenumbersofpositivesamplesandnegativesamples,respectively;standforthenumberofcorrectlydetectedtruepositivesamples,thenumberofnegativesamplesfalselyclassiedaspositivesamples,andthenumberofcorrectlydetectedtruenegativesamples.AgoodmodelisexpectedtoachieveahighvalueinARandDRbutalowvalueinFAR.B.PerformanceComparisonFirst,weexaminetheperformanceoftheproposedDCNNandtabulateitsresultsinTableV.Asseen,ourDCNNhasachievedveryhighperformance,evengreaterthan99.5%onaverage.ToverifythesuperiorityoftheDCNN,wecompareitwithtwopopularmodels,HLTPMC[6]andMCLBP[41],whichweredevelopedbasedonhand-craftedfeaturesfollowedbytheradialbasisfunction(RBF)kernel-basedSVM.Viathegridsearch,thebestSVMparameterscanbeobtainedbytrainingonSet-3(17315patches)andvalidationonSet-4(17379patches).Specically,wesetboththepenaltycoef-cientandgammacoefcientinthe

SVMas1forHLTPMC,andsetthemas798and102for
SVMas1forHLTPMC,andsetthemas798and102forMCLBPrespectively.TheirresultsaretabulatedinTableV.ItcanbeobservedthatourDCNNhasgivenrisetonoticeablygreaterperformancethanHLTPMCandMCLBP.Moreconcretely,consideringtheARTABLEVII:ComparisonofDCNNwithitstwocomponentsandDNCNN.MethodsDNCNN[22]SBNN(Prop.)SCNN(Prop.)DCNN(Prop.)Set1AR97.8%98.3%98.6%99.7%DR95.2%97.3%97.6%99.5%FAR0.48%0.96%0.84%0.12%Set2AR98.0%98.7%98.5%99.4%DR96.3%98.4%97.2%99.0%FAR0.48%0.98%0.48%0.24%(a)Trainingprocess(b)ValidationprocessFig.5:PlotsofaccuracycurvesofXception,Res-Net,Dense-NetandDCNN,duringthetrainingandvalidationprocesses.(a)Trainingprocess(b)ValidationprocessFig.6:PlotsofaccuracycurvesofDNCNN,SBNN,SCNNandDCNNduringthetrainingandvalidationprocesses.astheevaluationindicator,therelativeperformancegainsofourproposedDCNNovertheHLTPMCandMCLBPmodelsare,respectively,3.4%and2.9%onSet-1,aswellas1.0%and1.6%onSet-2respectively.Second,wecomparetheproposedDCNNwitheightpopularorstate-of-the-artdeepneuralnetworks,whichincludeAlex-Net[11],ZF-Net[12],VGG-Net[13],GoogLe-Net[14],Xception[15],Res-Net(152layers)[16],Dense-Net[27],andDNCNN[22].TheperformanceindicesoftheseeightnetworksareillustratedinTableVI.WecaneasilyndthattheDCNNhasacquiredtheoptimalperformance.InviewoftheARindex,theproposedDCNNhasintroducedarelativeperformancegainof1.9%onSet-1and1.4%onSet-2incomparisonwiththefourth-performingDNCNN,whichisarecentlydeviseddeepconvolutionalnetworkspecictosmokedetection.Incontrasttothethird-placeXception,therelativeperformancegainsachievedbyourDCNNare1.8%onSet-1and1.0%onSet-2.TherelativeperformancegainsbetweentheDCNNandthesecond-rankDense-Netare1.1%onSet-1and1.0%onSet-2.Wealsocomparethenumberofparametersusedinthenetworkbecauseitisalsoasignicantindicator.Toachieveanexcellentnetwork,itisdesirablethatthenetworkcontainsfewerparametersandthushasstronggeneralizationability.AslistedinTableVI,ourDCNNjustinvolves2.7millionparameters,lessthanone-thirdoftheparametersusedinthestate-of-the-artdeepnetworksconsideredinthispaper.Furthermore,thestandarddeviationsoftheeightdeepnetworkstestedandourDCNNacrosstwentyiterationsarecheckedandcompared.ThestandarddeviationsofAlex-Net,ZF-Net,VGG-Net,GoogLe-Net,Xception,Res-Net,Dense-Net,DNCNNandDCNNarerespectively0.2382,0.1436,0.1948,0.0049,0.0031,0.0063,0.0123,0.1014and0.0020onSet-1,and0.2179,0.1338,0.1882,0.0034,0.0034,0.0036,0.0058,0.0502and0.0012onSet-2.Fromtheseresults,wecanascertainthattheproposedDCNNhasaconsiderablystableperformance,superiortotheothercompetitors.Third,theproposedDCNNiscomparedwithitstwocom-ponents,namely,SBNNandSCNN.WetabulatetheresultsofSBNN,SCNNandDCNNinTableVII.WealsoincludetherecentlydevelopedDNCNNforcomparison,sincetheSBNNTABLEVIII:ComparisonofablationanalysisoneachcomponentoftheproposedDCNN.NetworksDCNN1DCNN2DCNN3DCNN4DCNN5DCNN6DCNNSet1AR97.3%98.6%98.4%98.7%98.7%97.6%99.7%DR96.0%98.7%96.9%98.0%98.0%96.4%99.5%FAR1.07%1.44%0.60%0.84%0.84%1.56%0.12%Set2AR99.1%99.3%98.4%98.7%98.9%98.5%99.4%DR99.0%99.0%98.3%97.5%98.3%98.3%99.0%FAR0.85%0.49%1.29%0.37%0.49%1.22%0.24%TABLEIX:ImplementationspeedcomparisonofourproposedDCNNwitheightdeepconvolutionalneuralnetworks.NetworksAlex-NetZF-NetVGG-NetGoogLe-NetXceptionRes-NetDense-NetDNCNNDCNN[11][12][13][14][15][16][27][22]Prop.Speed(millisecond/patch)0.3661.0270.3840.7010.6101.5231.1310.3690.453isinspiredbytheDNCNN.ItcanbereadilyfoundthatourSBNNissuperiortotheDNCNN,whichmightbeattributedtotheintroductionofselectivelyappendingtheBNlayeraftertheconvolutionallayer.Inaddition,SCNNperformsbetterthanSBNN,whichispossiblyduetotheuseoftheskipconnectionforpreventingthevanishinggradientandenhancingthefeaturepropagationaswellastheglobalaveragepoolingfordecreas-ingthenumberofparametersandmitigatingtheproblemofover-tting.Last,wecanndthattheproposedDCNNisbetterthanSCNN.ThismightbeduetotheappropriatefusionofSBNNandSCNNtocomplementeachother,sincetheyaregoodatextractingdetailinformationandbaseinformationofsmoke,respectively.InSectionIII-D,wewilldiscussindetailthecomplementarityofthesetwonetworksincapturingthecharacteristicsofsmoke.Fourth,wevisualizethecurvesoftrainingaccuracyandvalidationaccuracytofurthercomparethewholetrainingpro

cessofthestate-of-the-artXception,Res-Ne
cessofthestate-of-the-artXception,Res-Net,Dense-NetandDNCNN,aswellastheproposedSBNN,SCNNandDCNN.Forclarity,wedividetheabovesevennetworksintotwogroups.OnegroupiscomposedofXception,Res-Net,Dense-NetandDCNN,asshowninFigs.5(a)-(b),andtheothergroupiscomposedofDNCNN,SBNN,SCNNandDCNN,asshowninFigs.6(a)-(b).LetusrstconsiderFig.5.From(a),itcanbeviewedthatincomparisonwithXception,Res-NetandDense-Net,theproposedDCNNconvergesmorequickly,anditstrainingaccuracyvaluescanapproachoneastheepochincreasestosurpass30.From(b),wecanseethatthevalidationaccuracyvaluesarequitedifferentamongthefourtestingnetworks.Accordingtotheaccuracyvalues,weareabletoderivethefollowingrank:DCNNRes-Net.Moreover,itcanbefoundthatthevalidationaccuracyvaluesofXceptionandRes-Netarequiteoscillatory,whichispossiblybecausetheyhaveadeeperstructureandasmallchangeinmodelparametersmaylargelyaffectthevalidationaccuracy.WethenobserveFig.6.Twoimportantobservationscanbeestablished:1)theconvergencespeedofDCNNisremarkablyfasterthanitstwocomponents(namely,SBNNandSCNN)andtherecentlyproposedDNCNNduringnetworktraining;and2)DCNNhassuperiorandstableaccuracyvaluescomparedwiththeotherthreenetworkstestedduringthevalidationprocess.Insummary,theintroductionoffusingSBNNandSCNNforfeatureextractioncancontributesubstantiallytoourDCNN.Fifth,weconducttheablationanalysisoneachcomponentoftheproposedDCNN.WeremovealltheBNlayersfromDCNNandcallsuchanetworkDCNN.NotethatinSBNNandSCNN,severalconvolutionlayersexistthattheBNlayersarenotappendedto.Therefore,weaddtheBNlayersafterinSBNNinSCNN.SuchamodiednetworkisnamedDCNN.Further,weseparatelyeliminatetheskipconnectioninSCNN,themomentum,aswellasthelearningratedecay,anddubthosethreenetworksas,DCNNandDCNN.Finally,wereplacetheglobalaveragepoolinglayerwiththetypicallyusedfully-connectedlayers.Inparticular,inSBNNareusedtoinourproposedDCNN.ThismodiednetworkiscalledDCNN.WetraintheDCNN,DCNN,DCNN,DCNNandDCNNbasedonthesamemethodappliedinDCNNandtabulatetheirdetectionperformancesinTableVIII.Accordingtotheresults,twodominantconclusionscanbedrawn.First,byintroducingtheselective-basedBN,skipconnection,momentum,learningratedecay,andglobalaveragepooling,ourproposedDCNNhasattainedthebestclassicationperformance.Second,incontrasttotheothers,andDCNNhavelowtestingaccuracyvalues,andthisimpliesthatBNandglobalaveragepoolinghaveprovidedgreatercontributionstotheDCNN.Sixth,wecarryoutthecomparisonwithotherfusionstrate-giesofSBNNandSCNN.Sincethetaskofsmokedetectioninthisworkisabinaryclassicationproblem,thedirectfusionsofdecisionsofSBNNandSCNNaretheirunionandintersection,namely,`SBNN`SBNNSCNN'and`SBNNTheAR,DRandFARresultsoftheirunionare98.3%,96.0%and0.24%onSet-1,and98.4%,97.1%and0.48%onSet-2respectively.TheAR,DRandFARresultsoftheirintersectionare98.6%,98.9%and1.58%onSet-1,and98.8%,98.6%and0.98%onSet-2respectively.Clearly,thefusionofdecisionsofSBNNandSCNNisappreciablyinferiortotheproposedDCNN,whichcombinesSBNNandSCNNintermsoffeature1520-9210 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMM.2019.2929009, IEEETransactions on Multimedia10Fig.7:IllustrationoftypicalintermediatefeaturemapsextractedusingSBNNandSCNNpriortotwopoolinglayers.C.ImplementationSpeedImplementationefciencyisalsoasignicantindicator.AsillustratedinTableIX,wecomparetheimplementationtimeofourproposedDCNNandtheeightdeepnetworkstested.Specically,weruneachnetworkonall2888testingRGBimagepatchesofsize483(1383patchesfromSet-1and1505patchesfromSet-2)andthencomputetheaveragetimeforeachimagepatch.Inthistest,acomputerwasconguredwithaCPUprocessorof2.1GHz,anNVIDIATITANXpGPUof43.9GB,and64.0GBofRAM.OnecaneasilyndthattheproposedDCNNonlyconsumeslessthan0.5millisecondforeachpatch,obviouslyfasterthanthestate-of-the-artRes-NetandDense-Net.D.FeatureMapVisualizationWefurtherdiscussthenecessityoffusingthefeaturemapsextractedbyusingSBNNandSCNNtocomplementeachother.Morespec

ically,weexhibitinFig.7asampleimagep
ically,weexhibitinFig.7asampleimagepatchanditsassociatedvisualizedintermediatefeaturemapsextractedfromSBNNandSCNN.Thevisualizedfeaturemapsbeforeeachpoolingoperationareconsideredasexamples.First,letusobservethefeaturemapspriortotherstpoolingoperation.AlargegapoffeaturemapsbetweenandSCNNcanbefound.Then,wecomparethefeaturemapsofSBNNandSCNNbeforethesecondpoolingoperation.Wecanalsondamuchlargerdistanceexistsbetweenthefeaturemapsofthetwosubnetworks.Obviously,theSBNNandSCNNarequitedifferent.Furthermore,weleveragethepopularguidedimagelter(GIF)[42]todecomposethesamplesmokeimagepatchintoFig.8:FalselydetectedsamplesfromSet-1andSet-2.abasemapandadetailmap,asshowninFig.7.Suchanoperationhasbeenwidelyappliedinnumerousmultimediaapplications[43]-[45].Asshown,thebasemapcontainsthelarge-scalevariationsinintensity,whereasthedetailmapcontainsthesmall-scaledetails.Comparingthebaseanddetailmapswiththefeaturemapsmentionedabove,wecanndthatthefeaturemapsofSBNNaresimilartothedetailmap,whilethefeaturemapsofSCNNaresimilartothebasemap.Thatis,theSBNNismainlydevotedtoextractingthedetailedfeaturesfromsmokeimagepatchesandtheSCNNmostlyfocusesonextractingthebasicfeatures.TheSBNNandSCNNcannicelycomplementeachotherandeffectivelyboosttheDCNN'sperformance.E.DiscussionAnalysingthefalselydetectedsamplesisaconsiderablybenecialinsightforimprovingtheperformanceofourpro-posednetwork.Thus,wetakeintoaccountfourtypicalsam-pleswhereintheproposedDCNNfails,asshowninFig.8.ItisnotdifculttoobservethatourDCNNisnotgoodatdetectingsmokethathasfewertextures,justasthesamplesshowinFig.1520-9210 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMM.2019.2929009, IEEETransactions on Multimedia11(a)(b)(c)(d)(e)(f)(g)(h)Fig.9:Typicalameimagesinpetrochemicalenterprises.8.Toaddresssuchadifculty,ourfutureworkwillconsiderenhancingimagetexturespriortodetection.Inaddition,itisworthwhiletoemphasizethattheproposedDCNNsystemat-icallyintegratescontourinformationandtextureinformation,bothofwhichareveryimportantforsmokedetection.Incontrast,thewell-knowndeepnetworks,suchasRes-NetandDense-Net,weredevelopedspecicallyforimagerecognitiontasks,inwhichthesemanticinformation(e.g.,contour)playsthemostcrucialfunction.Insummary,ourDCNNisproposedparticularlyforextractingsmokecharacteristicsanddetectingsmokeandistherebysuperiortothesefamousdeepnetworksthatwehavetested.F.TestingofRealApplicationsInthissection,wewillexaminetheproposedDCNNintwoimportantrealapplications.Therstapplicationistodetectwhetherthereissmokeemittedfromaarethatburnswastegasproducedbypetrochemicalenterprises,wheresucharesareemployedtomaintainsafetyandpreventharmtotheenvironment.Importantly,blacksmokewillbegeneratedfromtheareiftheexhaustgasisnotsufcientlyburned.Insuchacase,somewatervapourshouldbesenttotheameforsmokeabatement.However,determininghowtoautomaticallyadjustthevolumeofwatervapourisasignicantproblem.TheproposedDCNNcanbeusedtosolvethisproblembydetectingblacksmokefromthecameraimage,whichisthenfollowedbycontrollingthevolumeofwatervapour.InFig.9,wedisplaysometypicalintermediateameimages.TheimagenumberanditsassociatedresultofourproposedDCNNareasfollows:(a)smoke-free,(b)smoke,(c)smoke,(d)smoke,(e)smoke,(f)smoke,(g)smoke-free,(h)smoke-free,and(i)smoke-free,whicharethesameastherealresults.Thesecondapplicationistoapplycameraimagestodetectcigarettesmoke,asshowninFig.10.TenthousandRGBimagepatchesofsize483arerandomlyselectedfromthesetwocameraimages,andthenlabelledbyvegraduatestudents.Accordingtothelabelresults,wepreserved8919imagepatches,eachofwhichhastheexactsamelabelresults(a)Fig.10:Twocameraimagesofcigarettesmokedetection.TABLEX:Comparisononcigarettesmokedetection.ARXceptionDense-NetDNCNNDCNNFig.10(a)72.4%78.5%57.5%Fig.10(b)65.4%75.8%59.1%Overall68.6%77.1%58.3%79.8%providedbyallvegraduatestudents.Thedetectio

naccuracyvaluesofourDCNNandstate-of-the-
naccuracyvaluesofourDCNNandstate-of-the-artXception,Dense-NetandDNCNNaretabulatedinTableX.Weareabletoderivetwocrucialconclusions:1)theproposedDCNNisslightlysuperiortoDense-NetandobviouslybetterthanXceptionandDNCNN;and2)allthedeepnetworksarenotgreatlyadeptindetectingcigarettesmokefromthecameraimages.Inthefuture,weplantofocusondetectingblacksmokefromameimagesandlightsmokefromcigaretteimages.Specically,wewillrstbuildtwolarge-sizeimagedatasetsforblacksmokedetectionandcigarettesmokedetection.Sec-ond,wewilldesignspecicdeepneuralnetworksthatconsiderthecharacteristicsofblacksmokeandcigarettesmokeforIV.CONCLUSIONSInthispaper,wehaveinvestigatedtheproblemofimage-basedsmokedetectionbydevisinganoveldeepdual-channelneuralnetworkdubbedDCNN.Incontrasttotherecent-lyproposeddeepneuralnetworks,includingAlex-Net,ZF-Net,VGG-Net,GoogLe-Net,Xception,Res-Net,Dense-Net,andthesmoke-specicDNCNN,theproposedDCNNisestablishedmainlybasedonthefusionoftwochannelsofdeepsubnetworks.Therstchannel'ssubnetworkisbuiltbyrstconnectingmultipleconvolutionallayersandmax-poolinglayerssequentially,andthenappendingtheBNlayertopartofthelastconvolutionallayersselectively.Thesecondchannel'ssubnetworkisconstructedbyincorporatingtheskipconnectionandglobalaveragepoolingwiththeconvolutional,BN,andmax-poolinglayers.Theskipconnectioncanhelptopreventthevanishinggradientandenhancethefeaturepropa-gation.Theglobalaveragepoolingisbenecialindecreasingthenumberofnetworkparametersandmitigatingtheproblemofover-tting.TheproposedDCNNisnallydesignedtocombinetheaforementionedtwodeepsubnetworksbyacon-catenationoperation.Weimplementcomparativeexperimentsonthepubliclyavailablesmokedetectionimagedatabasetoconrmtheeffectivenessofourdeepnetwork.Comparedwiththerecentlydevelopedsmokedetectionmodelsandstate-of-the-artdeepneuralnetworks,ourDCNNhasachievedoptimalperformance,beyond99.5%onaverage,withtheleastnetworkparameters.Furthermore,throughanumericalcomparisonandavisualizedcomparison,weil-lustratethatthesuperiorityoftheproposeddeepnetworkisprimarilyachievablebecausethedualdeepsubnetworksmentionedabovecancomplementeachother.EFERENCES[1]B.U.Toreyin,Y.Dedeoglu,andA.E.C¸etin,“Waveletbasedreal-timesmokedetectioninvideo,”inProc.Eur.SignalProcess.Conf.,pp.1-4,Sep.2005.[2]J.Gubbi,S.Marusic,andM.Palaniswami,“Smokedetectioninvideousingwaveletsandsupportvectormachines,”FireSafetyJ.,vol.44,no.8,pp.1110-1115,Nov.2009.[3]F.Yuan,“Afastaccumulativemotionorientationmodelbasedonintegralimageforvideosmokedetection,”PatternRecognit.Lett.,vol.29,no.7,pp.925-932,May2008.[4]F.Yuan,“Video-basedsmokedetectionwithhistogramsequenceofLBPandLBPVpyramids,”FireSafetyJ.,vol.46,no.3,pp.132-139,Apr.[5]F.Yuan,“Adoublemappingframeworkforextractionofshape-invariantfeaturesbasedonmulti-scalepartitionswithAdaBoostforvideosmokedetection,”PatternRecognit.,vol.45,no.12,pp.4326-4336,Dec.2012.[6]F.Yuan,J.Shi,X.Xia,Y.Fang,Z.Fang,andT.Mei,“High-orderlocalternarypatternswithlocalitypreservingprojectionforsmokedetectionandimageclassication,”Inf.Sci.,vol.372,pp.225-240,Dec.2016.[7]F.Yuan,X.Xia,J.Shi,H.Li,andG.Li,“Non-lineardimensionalityreductionandGaussianprocessbasedclassicationmethodforsmokedetection,”IEEEAccess,vol.5,pp.6833-6841,Apr.2017.[8]H.WangandB.Raj,“Ontheoriginofdeeplearning,”arXivpreprint,Mar.2017.[9]I.HadjiandR.P.Wildes.“Whatdoweunderstandaboutconvolutionalnetworks?”arXivpreprintarXiv:1803.08834,Mar.2018.[10]J.Deng,W.Dong,R.Socher,L.Li,K.Li,andF.-F.Li,“Imagenet:Alarge-scalehierarchicalimagedatabase,”inProc.IEEEConf.Comp.Vis.andPatternRecognit.,pp.248-255,Jun.2009.[11]A.Krizhevsky,I.Sutskever,andG.E.Hinton,“ImageNetclassicationwithdeepconvolutionalneuralnetworks,”inProc.Adv.NeuralInf.Process.Syst.,vol.25.pp.1097-1105,2012.[12]M.D.ZeilerandR.Fergus,“Visualizingandunderstandingconvolution-alnetworks,”inProc.Eur.Conf.Comp.Vis.,pp.818-833,Sep.2014.[13]K.SimonyanandA.Zisserman,“Verydeepconvolutionalnetworksforlarge-scaleimagerecognition,”arXivpreprintarXiv:1409.1556,Sep.[14]C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,andA.Rabinovi

ch,“Goingdeeperwithconvolutions,

ch,“Goingdeeperwithconvolutions,”Proc.IEEEConf.Comp.Vis.andPatternRecognit.,pp.1-9,Jun.2015.[15]F.Chollet,“Xception:Deeplearningwithdepthwiseseparableconvo-lutions,”inProc.IEEEConf.Comp.Vis.andPatternRecognit.,pp.1251-1258,Jul.2017.[16]K.He,X.Zhang,S.Ren,andJ.Sun,“Deepresiduallearningforimagerecognition,”inProc.IEEEConf.Comp.Vis.andPatternRecognit.,pp.770-778,Jun.2016.[17]J.Guan,S.Yi,X.Zeng,W.-K.Cham,andX.Wang,“Visualimportanceanddistortionguideddeepimagequalityassessmentframework,”Trans.Multimedia,vol.19,no.11,pp.2505-2520,Nov.2017.[18]N.Takahashi,M.Gygli,andL.V.Gool,“Aenet:Learningdeepaudiofeaturesforvideoanalysis,”IEEETrans.Multimedia,vol.20,no.3,pp.513-524,Mar.2018.[19]Z.Qiu,T.Yao,andT.Mei,“Learningdeepspatio-temporaldependenceforsemanticvideosegmentation,”IEEETrans.Multimedia,vol.20,no.4,pp.939-949,Apr.2018.[20]G.Ning,Z.Zhang,andZ.He,“Knowledge-guideddeepfractalneuralnetworksforhumanposeestimation,”IEEETrans.Multimedia,vol.20,no.5,pp.1246-1259,May2018.[21]S.Zhang,S.Zhang,T.Huang,andW.Gao,“Speechemotionrecognitionusingdeepconvolutionalneuralnetworkanddiscriminanttemporalpyramidmatching,”IEEETrans.Multimedia,vol.20,no.6,pp.1576-1590,Jun.2018.[22]Z.Yin,B.Wan,F.Yuan,X.Xia,andJ.Shi,“Adeepnormalizationandconvolutionalneuralnetworkforimagesmokedetection,”IEEEAccessvol.5,pp.18429-18438,Aug.2017.[23]S.IoffeandC.Szegedy,“Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift,”arXivpreprint,Feb.2015.[24]D.J.FagnantandK.Kockelman,“Preparinganationforautonomousvehicles:Oportunities,barriersandpolicyrecommendations,”Trans-portationResearchPartA:PolicyandPractice,vol.77,pp.167-181,Jul.2015.[25]S.Nah,T.H.Kim,andK.M.Lee,“Deepmulti-scaleconvolutionalneuralnetworkfordynamicscenedeblurring,”inProc.IEEEConf.Comp.Vis.PatternRecognit.,vol.1,no.2,pp.3883-3891,Jul.2017.[26]B.Lim,S.Son,H.Kim,S.Nah,andK.Lee.“Enhanceddeepresidualnetworksforsingleimagesuper-resolution.”inProc.IEEEConf.Comp.Vis.PatternRecognit.Workshops,vol.1,no.2,pp.136-144,Jul.2017.[27]G.Huang,Z.Liu,L.vanderMaaten,andK.Q.Weinberger,“Denselyconnectedconvolutionalnetworks,”inProc.IEEEConf.Comp.Vis.PatternRecognit.,pp.4700-4708,Jul.2017.[28]C.Liu,F.Tsow,Y.Zou,andN.Tao,“Particlepollutionestimationbasedonimageanalysis,”PloSone,vol.11.no.2,pp.e0145955,2016.[29]K.Gu,J.Qiao,andX.Li,“Highlyefcientpicture-basedpredictionofPM2.5concentration,”IEEETrans.Ind.Electron.,vol.66,no.4,pp.3176-3184,Apr.2019.[30]P.-W.Soh,J.-W.Chang,andJ.-W.Huang,“Adaptivedeeplearning-basedairqualitypredictionmodelusingthemostrelevantspatial-temporalrelations,”IEEEAccess,vol.6,pp.38186-38199,2018.[31]K.Gu,J.Qiao,andW.Lin,“Recurrentairqualitypredictorbasedonmeteorology-andpollution-relatedfactors,”IEEETrans.Ind.Informat.vol.14,no.9,pp.3946-3955,Sep.2018.[32]K.Gu,Z.Xia,andJ.Qiao,“StackedselectiveensembleforPM2.5forecast,”IEEETrans.Instrum.Meas.,2019.[33]X.Glorot,A.Bordes,andY.Bengio,“Deepsparserectierneuralnetworks,”inProc.Conf.ArticialIntelligenceandStatistics,pp.315-323,Jun.2011.[34]N.Srivastava,G.Hinton,A.Krizhevsky,I.Sutskever,andR.Salakhut-dinov,“Dropout:Asimplewaytopreventneuralnetworksfromover-tting,”J.Mach.Learn.Res.,vol.15,no.1,pp.1929-1958,Jan.2014.[35]X.GlorotandY.Bengio,“Understandingthedifcultyoftrainingdeepfeedforwardneuralnetworks,”J.Mach.Learn.Res.,vol.9,pp.249-256,Mar.2010.[36]Y.Bengio,“Practicalrecommendationsforgradient-basedtrainingofdeeparchitectures,”Neuralnetworks:Tricksofthetrade,Springer,Berlin,Heidelberg,pp.437-478,2012.[37]Y.Lecun,L.Bottou,Y.Bengio,andP.Haffner,“Gradient-basedlearningappliedtodocumentrecognition,”Proc.IEEE,vol.86,no.11,pp.2278-2324,Nov.1998.[38]R.C.GonzalezandR.E.Woods,DigitalImageProcessing,3rded.EnglewoodCliffs,NJ,USA:Prentice-Hall,2006.[39]M.Abadi,A.Agarwal,P.Barham,etal.,“TensorFlow:Large-scalemachinelearningonheterogeneousdistributedsystems,”arXivpreprint,Mar.2016.[40]F.Chollet.Keras,2015.[Online].Available:http://keras.io[41]S.R.Dubey,S.K.Singh,andR.K.Singh,“Multichanneldeco

dedlocalbinarypatternsforcontent-basedim
dedlocalbinarypatternsforcontent-basedimageretrieval,”IEEETrans.ImageProcess.,vol.25,no.9,pp.4018-4032,Sep.2016.[42]K.He,J.Sun,andX.Tang,“Guidedimageltering,”IEEETrans.PatternAnal.Mach.Intell.,vol.35,no.6,pp.1397-1409,Jun.2013.[43]Z.Farbman,R.Fattal,D.Lischinski,andR.Szeliski,“Edge-preservingdecompositionsformulti-scaletoneanddetailmanipulation,”ACMTrans.Graphics,vol.27,no.3,pp.249-256,Aug.2008.[44]S.Li,X.Kang,andJ.Hu,“Imagefusionwithguidedltering,”Trans.ImageProcess.,vol.22,no.7,pp.2864-2875,Jul.2013.[45]G.A.Kordelas,D.S.Alexiadis,P.Daras,andE.Izquierdo,“Content-basedguidedimageltering,weightedsemi-globaloptimization,andefcientdisparityrenementforfastandaccuratedisparityestimation,”IEEETrans.Multimedia,vol.18,no.2,pp.155-170,Feb.2016.1520-9210 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMM.2019.2929009, IEEETransactions on Multimedia13KeGu(M'19)receivedtheB.S.andPh.D.degreesinelectronicengineeringfromShanghaiJiaoTongUniversity,Shanghai,China,in2009and2015,re-spectively.HeiscurrentlyaProfessorwiththeBei-jingUniversityofTechnology,Beijing,China.Hisresearchinterestsincludeenvironmentalperception,imageprocessing,qualityassessment,andmachinelearning.HereceivedtheBestPaperAwardfromtheIEEETransactionsonMultimedia(T-MM),theBestStudentPaperAwardattheIEEEInternationalConferenceonMultimediaandExpo(ICME)in2016,andtheExcellentPh.D.ThesisAwardfromtheChineseInstituteofElectronicsin2016.HewastheLeadingSpecialSessionOrganizerintheVCIP2016andtheICIP2017,andservesasaGuestEditorfortheDigitalSignalProcessing(DSP).HeiscurrentlyanAssociateEditorfortheIEEEACCESSandIETImageProcessing(IET-IPR),andanAreaEditorfortheSignalProcessingImageCommunication(SPIC).HeisaReviewerfor20topSCIjournals.ZhifangXiareceivedtheB.S.degreeinmeasur-ingandcontrolinstrumentfromAnhuiUniversity,Hefei,Chinain2008andreceivedtheMasterdegreeincontrolscienceandengineeringfromTsinghuaUniversity,Beijing,Chinain2012.Sheiscurrentlyanengineerandaregisteredconsultant(investment)withStateInformationCenter,Beijing,China,andiscurrentlypursuingthePh.D.degreewithBeijingUniversityofTechnology,Beijing,China.Herinter-estsincludeimageprocessing,qualityassessment,machinelearningande-government.ShewonthesecondprizeofNationalexcellentengineeringconsultationawardin2016.JunfeiQiao(M'11)receivedtheB.E.andM.E.degreesincontrolengineerfromLiaoningTech-nicalUniversity,Fuxin,China,in1992and1995,respectively,andthePh.D.degreefromNortheastUniversity,Shenyang,China,in1998.HewasaPost-DoctoralFellowwiththeSchoolofAutomatics,TianjinUniversity,Tianjin,China,from1998to2000.HejoinedtheBeijingUniversityofTech-nology,Beijing,China,whereheiscurrentlyaProfessor.HeistheDirectoroftheIntelligenceSystemsLaboratory.Hiscurrentresearchinterestsincludeneuralnetworks,intelligentsystems,self-adaptive/learningsystems,andprocesscontrolsystems.Prof.QiaoisamemberoftheIEEEComputa-tionalIntelligenceSociety.HeisaReviewerformorethan20internationaljournals,suchastheIEEETransactionsonFuzzySystemsandtheIEEETransactionsonNeuralNetworksandLearningSystems.WeisiLin(F'16)receivedthePh.D.degreefromKingsCollegeLondon.HeiscurrentlyaProfessorwiththeSchoolofComputerScienceandEngineer-ing,NanyangTechnologicalUniversity,Singapore.Hisresearchinterestsincludeimageprocessing,visualqualityevaluation,andperception-inspiredsignalmodeling,withmorethan340refereedpaperspublishedininternationaljournalsandconferences.HehasbeenontheEditorialBoardoftheIEEET-IP,T-CSVT,T-MM,SPL,andJVCI.HehasbeenelectedasAPSIPA(2012/2013)DistinguishedLec-turers.HeservedasaTechnical-ProgramChairforPacic-RimConferenceonMultimedia2012,theIEEEInternationalConferenceonMultimediaandExpo2013,andtheInternationalWorkshoponQualityofMultimediaExperience2014.HeisafellowofInstitutionofEngineeringTechnology,anHonoraryFellowoftheSingaporeInstituteofEngineeringTechnologists,andaFello