The weights on the connec tions between neurons mediate the passed values in both dire ctions The Backpropagation algorithm is used to learn the weights o f a multilayer neural network with a 64257xed architecture It performs gradient descent to try ID: 22415
Download Pdf The PPT/PDF document "Derivation of Backpropagation Introduct..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
DerivationofBackpropagation1IntroductionFigure1:NeuralnetworkprocessingConceptually,anetworkforwardpropagatesactivationtoproduceanoutputanditbackwardpropagateserrortodetermineweightchanges(asshowninFigure1).Theweightsontheconnec-tionsbetweenneuronsmediatethepassedvaluesinbothdirections.TheBackpropagationalgorithmisusedtolearntheweightsofamultilayerneuralnetworkwitha xedarchitecture.Itperformsgradientdescenttotrytominimizethesumsquarederrorbetweenthenetwork'soutputvaluesandthegiventargetvalues.Figure2depictsthenetworkcomponentswhicha ectaparticularweightchange.Noticethatallthenecessarycomponentsarelocallyrelatedtotheweightbeingupdated.Thisisonefeatureofbackpropagationthatseemsbiologicallyplausible.However,brainconnectionsappeartobeunidirectionalandnotbidirectionalaswouldberequiredtoimplementbackpropagation.2NotationForthepurposeofthisderivation,wewillusethefollowingnotation:Thesubscriptkdenotestheoutputlayer.Thesubscriptjdenotesthehiddenlayer.Thesubscriptidenotestheinputlayer.1 Figure2:Thechangetoahiddentooutputweightdependsonerror(depictedasalinedpattern)attheoutputnodeandactivation(depictedasasolidpattern)atthehiddennode.Whilethechangetoainputtohiddenweightdependsonerroratthehiddennode(whichinturndependsonerroratalltheoutputnodes)andactivationattheinputnode.wkjdenotesaweightfromthehiddentotheoutputlayer.wjidenotesaweightfromtheinputtothehiddenlayer.adenotesanactivationvalue.tdenotesatargetvalue.netdenotesthenetinput.3ReviewofCalculusRulesd(eu)dx=eududxd(g+h)dx=dgdx+dhdxd(gn)dx=ngn 1dgdx4GradientDescentonErrorWecanmotivatethebackpropagationlearningalgorithmasgradientdescentonsum-squarederror(wesquaretheerrorbecauseweareinterestedinitsmagnitude,notitssign).Thetotalerrorinanetworkisgivenbythefollowingequation(the12willsimplifythingslater).E=12Xk(tk ak)2Wewanttoadjustthenetwork'sweightstoreducethisoverallerror.W/ @E@WWewillbeginattheoutputlayerwithaparticularweight.2 wkj/ @E@wkjHowevererrorisnotdirectlyafunctionofaweight.Weexpandthisasfollows.wkj= "@E@ak@ak@netk@netk@wkjLet'sconsidereachofthesepartialderivativesinturn.NotethatonlyonetermoftheEsummationwillhaveanon-zeroderivative:theoneassociatedwiththeparticularweightweareconsidering.4.1Derivativeoftheerrorwithrespecttotheactivation@E@ak=@(12(tk ak)2)@ak= (tk ak)Nowweseewhythe12intheEtermwasuseful.4.2Derivativeoftheactivationwithrespecttothenetinput@ak@netk=@(1+e netk) 1@netk=e netk(1+e netk)2We'dliketobeabletorewritethisresultintermsoftheactivationfunction.Noticethat:1 11+e netk=e netk1+e netkUsingthisfact,wecanrewritetheresultofthepartialderivativeas:ak(1 ak)4.3DerivativeofthenetinputwithrespecttoaweightNotethatonlyonetermofthenetsummationwillhaveanon-zeroderivative:againtheoneassociatedwiththeparticularweightweareconsidering.@netk@wkj=@(wkjaj)@wkj=aj4.4WeightchangeruleforahiddentooutputweightNowsubstitutingtheseresultsbackintoouroriginalequationwehave:wkj="kz}|{(tk ak)ak(1 ak)ajNoticethatthislooksverysimilartothePerceptronTrainingRule.Theonlydi erenceistheinclusionofthederivativeoftheactivationfunction.Thisequationistypicallysimpli edasshownbelowwherethetermrepesentstheproductoftheerrorwiththederivativeoftheactivationfunction.wkj="kaj3 4.5WeightchangeruleforaninputtohiddenweightNowwehavetodeterminetheappropriateweightchangeforaninputtohiddenweight.Thisismorecomplicatedbecauseitdependsontheerroratallofthenodesthisweightedconnectioncanleadto.wji/ [Xk@E@ak@ak@netk@netk@aj]@aj@netj@netj@wji="[Xkkz}|{(tk ak)ak(1 ak)wkj]aj(1 aj)ai="jz}|{[Xkkwkj]aj(1 aj)aiwji="jai4