Rob J Hyndman OB H YNDMAN MONASH EDU Department of Econometrics and Business Statistics Monash University Clayton VIC 3800 Australia Abstract Multistep forecasts can be produced recursively by iterating a onestep model or directly using a speci64257 ID: 29050
Download Pdf The PPT/PDF document "Boosting multistep autoregressive foreca..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Boostingmulti-stepautoregressiveforecasts addedbyallowingseveralnonlinearboostingcomponentsateachforecasthorizon;(ii)itallowsnonlinearitieswiththeboostingcomponentswithoutsacricingmuchvariancethankstothereducedvarianceoftheweaklearners;and(iii)itavoidsthedifcultchoicebetweenrecursiveanddi-rectforecasts.Weevaluatethebooststrategyintwosteps.Intherststep,wedecomposethemeansquarederror(MSE)ofthefore-castsandanalyzethebiasandvariancecomponentsoverthehorizon.Webeginbyatheoreticalanalysisofthebiasandvariancecomponentsfortwostepsahead.Thenweconductasimulationstudywithtwodatageneratingpro-cesses(DGP)forthegeneralcaseofhstepsahead.Then,weconsiderreal-worldtimeseriesandcomparetheper-formanceofthebooststrategywiththerecursiveanddirectstrategiesonroughly500timeseriesfromtheM3andNN5forecastingcompetitions.Overall,thebooststrategycon-sistentlyproducesbetterout-of-sampleforecastsandthusisveryattractiveformulti-stepforecastingtasks.2.Multi-stepforecastingstrategiesWebeginbydiscussingtheproblemofmulti-stepforecast-inganddescribetherecursiveandthedirectstrategiesforproducingmulti-stepforecasts.ConsideraunivariatetimeseriesYT=fy1;:::;yTgcom-prisingTobservations.WewouldliketoforecasttheHfutureobservationsfyT+1;:::;yT+Hg.Weassumethatthedataaredescribedbyapossiblynon-linearautoregres-siveprocessoftheformyt=f(xt�1)+"twithxt�1=[yt�1;:::;yt�d]0;(1)wheref"tgisaGaussianwhitenoiseprocesswithzeromeanandvariance2.Thetimeseriesisthereforespec-iedbyafunctionf,anembeddingdimensiond,andanoiseterm"t.Weassumethatwedonotknowford.IfweconsidertheMSEastheerrormeasuretobemin-imized,thentheoptimalforecastathorizonhistheconditionalmeant+hjt=E[yt+hjxt]andthegoalofforecastingistoestimateit.Therecursivestrategy.Onestrategyforproducingmulti-stepforecasts,calledrecursive,centersonbuildingatimeseriesmodelofthesameformas(1),aimingtomin-imizetheone-step-aheadpredictionerrorvariance.Theunknownfuturevaluesarethenobtaineddynamicallybyrepeatedlyiteratingthemodelandbyreplacing(pluggingin)theunknownfuturevalueswiththeirownforecasts.Inotherwords,itentailsamodeloftheformyt=m(zt�1; )+et(2)withzt�1=[yt�1;:::;yt�p]0wherepisanestimationoftheembeddingdimensiondandE[et]=0.Notethatet=f(xt�1)�m(zt�1; )+"tistheforecasterrorofthemodelm.Theparameters areestimatedby^ =argmin Xt[yt�m(zt�1; )]2:Then,forecastsareobtainedrecursively,^T+hjT=m(h)(zT;^ )wherem(h)istherecursiveapplicationofmandh=1;:::;H.Thedirectstrategy.Asecondstrategy,calleddirect,tailorstheforecastingmodeldirectlytotheforecasthori-zon.Thatis,differentforecastingmodelsareusedforeachforecasthorizon:yt=mh(rt�h; h)+et;h;(3)wherert�h=[yt�h;:::;yt�h�ph]0.Foreachmodel,theparameters hareestimatedasfollows^ h=argmin hXt[yt�mh(rt�h; h)]2:Thenforecastsareobtainedforeachhorizonfromthecorrespondingmodel,^T+hjT=mh(rT;^ h)withh=1;:::;H.Ifweknewfandd,thentherecursivestrategyandthedirectstrategywouldbeequivalentwhenfislinear,butnotwhenfisnonlinear.Becauseofminimizingtheone-steppredictionerror,whenfisnonlineartherecursivestrategyisbiasedwhilethedirectstrategyachievestheoptimalerrorinameansquarederrorsense(Fan&Yao,2005;Atiya,El-shoura,Shaheen,&El-sherif,1999).Inpractice,whichstrategyisbetterisanempiricalmatteraswemustestimateunknownfunctionsm(Eq.(2))andmh(Eq.(3))fromanite-sampledataset.Theperformanceofbothstrategiesdependsnotablyonthenonlinearityoff,theembeddingdimensiond,thelevelofnoise2,thesizeofthetimeseriesT,theestimationalgorithmandtheforecasthorizonh.So,thechoicebetweentherecursiveandthedirectstrategyisnotaneasytaskinapplications.3.ThebooststrategyWeproposetoboosttherecursiveforecastsfromasimpleautoregressive(AR)linearmodelbyusingadirectstrategy,allowingseveralsmallandnonlinearadjustmentsateachhorizonh.Eachadjustmenttriestocatchthediscrepancybetweenthelinearrecursiveforecastsandthetruecondi-tionalmeanathorizonh.Inotherwords,webeginwithasimpleautoregressivelin-earmodel,yt=c+ 1yt�1++ pyt�p| {z }m(zt�1; )+etandproduceforecastsfromitusingtherecursivestrategym(h)(zt;^ ).Atthisstage,ourforecastsareequivalentto Boostingmulti-stepautoregressiveforecasts applications.4.RelatedworkThisworkconsidersboostinginthecontextsofmulti-stepforecasting.Inthemachinelearningliterature,boostingiswellknownforclassicationwithAdaBoost(Freund&Schapire,1996),butmuchlessattentionhasbeenpaidtoregressionsettings.SomeextensionsofAdaBoosttoregressionincludeDrucker(1997)andShresthaandSolo-matine(2006).Agradientboostingapproachhasalsobeenproposedforregression(Friedman,2001).Intheforecastingcommunity,boostinghasreceivedevenlessattentionandtheliteratureisrathersparse.Assaad,Bon´e,andCardot(2008)consideredrecurrentneuralnet-worksasweaklearnerswithanadaptedAdaBoostandcomparedtheirmethodwithlocalapproachesontwotimeseries.AudrinoandB¨uhlmann(2003,2009)usedagra-dientboostingapproachtomodelvolatilityinnancialapplications.Boostinghasonlyrecentlybeenconsideredinthemacroeconometricliteraturewith(Shak&Tutz,2009;Bai&Ng,2009;Buchen&Wohlrabe,2011).Eco-nomicforecastingisalsoconsideredinRobinzonov,Tutz,andHothorn(2012)withaboostingproceduretoestimatenonlinearadditiveautoregressivemodels.Finally,agra-dientboostingapproachhasbeenusedrecentlyinaloadforecastingcompetitionandrankedamongthetopvecompetitors(BenTaieb&R.Hyndman,2013).5.BiasandvarianceanalysisAperformanceanalysisoftheforecastingstrategiescanbeaccomplishedthroughanexaminationoftheerrordecom-positionintothebiasandvariancecomponents(Geman,Bienenstock,&Doursat,1992).Letg(zt;^YT;h)denotetheforecastsofagivenstrategyathorizonhusingthein-putvectorztandusingthesetofparameters^YT.TheseparametersareestimatedusingYT,atimeserieswithTobservations.So,theinputvectorztandthesetofparam-eters^YTcanchangeforeachsampleYT.Inaddition,theinputvectorztcanbedifferentfromxt,the“real”inputvectordenedin(1).Letusalsodeneg(zt;T;h)=EYThg(zt;^YT;h)i.Assumingtheprocessdenedin(1)isstationary,theMSEofthegivenstrategyathorizonhisdecomposedasfollows.MSEh=Ext2664E";YTh(yt+h�g(zt;^YT;h))2jxti| {z }MSEh(xt)3775=Ext;"(yt+h�t+hjt)2jxt| {z }NoiseNh+Ext(t+hjt�g(zt;T;h))2| {z }BiasBh(5)+Ext;YTh(g(zt;^YT;h)�g(zt;T;h))2jxti| {z }VarianceVhwhereExandE[jx]denotetheexpectationoverxandtheexpectationconditionalonx,respectively.WecanseethattheMSEoftheforecastsg(zt;^YT;h)athorizonhcanbedecomposedintothreedifferentcompo-nents,namelythenoisetermNh,thesquaredbiastermBhandtheestimationvariancetermVh.So,thisdecomposi-tionisidenticaltotheusualdecompositionusedinmachinelearning(Gemanetal.,1992).However,incontrastwithusualregressionproblems,multi-stepforecastingisdealingwithtime-dependentdataandrequireslearningdependenttaskswithdifferentnoiselevelchangingwiththeforecast-inghorizonh.Athorizonh=1,theproblemofmulti-stepforecastingreducesforallstrategiestotheestimationofthefunctionfsincewehavethesimpleexpressiont+1jt=f(xt).Inthefollowing,weconsiderotherhorizons.Tosimplifythederivations,wewillperformatheoreticalanalysisfortwostepsaheadusingsimilarargumentstoBenTaiebandAtiya(2014).Then,wewillperformMonteCarlosimulationstoanalyzebiasandvarianceforthegeneralcaseofhstepsahead.5.1.TheoreticalanalysisAssumethatthetimeseriesisgeneratedbythenonlin-earautoregressiveprocessdenedin(1).First,wecancomputeyt+2usingaTaylorseriesapproximationuptosecond-orderterms,whichgivesusyt+2=f(f(xt)+"t+1;yt;:::;yt�d+2)+"t+2f(f(xt);:::;yt�d+2)+"t+1fx1+1 2("t+1)2fx1x1+"t+2;wherefx1andfx1x1aretherstandsecondderivativesoffwithrespecttoitsrstargument,respectively.Theconditionalexpectationt+2jtisthengivenbyt+2jt=E[yt+2jxt]=f(f(xt);yt;:::;yt�d+2)+1 22fx1x1Inordertocomputethebiasandvariancetermsath=2,B2(xt)andV2(xt)asdenedin(5),weconsiderthattheforecastsofeachstrategycanbemodeledasasumofthreeterms:thetruefunctionvaluewearetryingtoestimate,thatistheconditionalmeant+2jt=E[yt+2jxt],anoff-settermdenotedby(zt;)andavariabilitytermdenotedby(zt;)"where(zt;)isadeterministiccompo-nentgivingthestandarddeviationoftheterm,and"isastochasticcomponentwithE["]=0andE["2]=1. Boostingmulti-stepautoregressiveforecasts B2(xt)+V2(xt)=(t+2jt�g(zt;;2))2+EYTh(g(zt;^;2)�g(zt;;2))2jxtiFortherecursivestrategy,wehaveBREC2(xt)+VREC2(xt)=ht+2jt�f(f(xt);:::;yt�p+2)(6)+(f(xt);:::;yt�p+2; )+(zt; )mz1+1 2[(zt; )]2mz1z1+1 2[(zt; )]2mz1z1i2(7)+[(f(xt);:::;yt�p+2; )]2+[(zt; )mz1]2+1 2[(zt; )]4m2z1z1(8)+2(f(xt);:::;yt�p+2; )(zt; )mz1E["1"2]+(zt; )2(f(xt);:::;yt�p+2; )mz1z1E["21"2](9)whereweusedthefactthatE["3]=0andE["4]=3forthestandardnormaldistribution.Forthedirectstrategy,wehaveBDIRECT2(xt)+VDIRECT2(xt)=t+2jt�m2(rt; )2(10)+(rt; )2(11)Forthebooststrategy,wehaveBBOOST2(xt)+VBOOST2(xt)=t+2jt��(c+ 1c)+( 21+ 2)yt+( 1 2+ 3)yt�1++( 1 p�1+ p)yt�p+2+( 1 p)yt�p+1+XjkMjk(yt�j;yt�k; jk)1A352(12)+( 1+1)2( )2+I2Xi=1i(yt�j;yt�k; i)2 2| {z }Pjkjk(yt�j;yt�k; jk)2;where=1 andweassumed"0?"jkand"ab?"jk.Letusnowcomparethebooststrategywiththerecursiveanddirectstrategies,beginningwiththebiascomponent.Fortherecursivestrategy,sincethemodelmisusedrecur-sively,wecanseein(6)and(7)thattheoffset(; )ath=1ispropagatedtoh=2.Inaddition,theoffsetisam-pliedwhenthemodelmproducesafunctionthathaslargevariations(i.e.mz1andmz1z1arelargeinmagnitude).Forthedirectstrategy,theoffsetofthemodelath=1doesnotappearin(10).Soprovidedthatthemodelm2isexibleenoughtoestimatetheconditionalmeanandenoughdataisavailable,thebiascanbearbitrarilysmall.Forthebooststrategy,becausewerequiretherecursiveARmodeltobelinear,thepropagationoferrorsislimitedsincemz1iscon-stant,mz1z1=0,(; )=( )and(; )=( ).Evenifthelinearrecursiveforecastsarebiasedatsomehorizon,thenonlinearboostingcomponentscanadjustthebiasascanbeseein(12).Wenowturntothevariancecomponents.Fortherecursivestrategy,wecanseein(8)–(9)that,similartotheoffsetinthebias,thevariancetermsgetamplied.Forthedirectstrategy,wecanseein(11)thatthevariancewilldependonthevariabilityinducedbytheinputrt,thesetofparame-ters andthesizeofthetimeseriesT.Thisvariabilitycanbeparticularlylargeforcomplexnonlinearmodelswhichcontainmanyinteractionsinrtorhavealargesetofpa-rameters .Forthebooststrategy,thevarianceislimited,ontheonehandbytherecursiveARmodelbeinglinear,andontheotherhandbecausetheboostingcomponentsal-lowonlybivariateinteractionsandareshrunktowardszerowiththeshrinkagefactor.Limitinginteractionstotwoisnotastronglimitationsinceweexpectreal-worldtimeseriestodependonlower-orderinteractions.Furthermore,consideringthefactthatthedirectstrategyselectsthemodelateachhorizonindependently,theerrorset;hfromthedifferentmodelsin(3)canbeautocorrelated;thatisinformationisleftintheerrors.Withthebooststrat-egy,adirectapproachisusedafterextractingtherecursivelinearforecastsfromtheobservations.Bydoingso,select-ingthedirectmodelsindependentlyhasasmallereffectcomparedtoapuredirectstrategy.Finally,ifweconsiderthecaseofaninnitelylongtimeseries,andwhenfisnonlinear,thedirectstrategydomi-natestherecursivestrategywhichisbiased(seep.348ofTer¨asvirtaetal.(2010)).Whenfislinear,thedirectandtherecursivestrategyareequivalent(seep.118ofFanandYao(2005)).Inthesamecase,thebooststrategyisequivalenttothedirectstrategyifthemaximumorderofinteractioninthefunctionfistwo.Ifitismorethantwo,thebooststrategywillbebiased.5.2.AnalysisbyMonteCarlosimulationsWeconductasimulationstudytoshedsomelightontheperformanceofthebooststrategyintermsofbiasandvariancecomponentsovertheforecastinghorizon.ThemethodologyissimilartotheoneperformedinBerardiandZhang(2003)exceptthatweconsiderforecastingmulti-stepaheadinsteadofone-stepahead.Datageneratingprocesses.WeconsideranonlinearandalinearARprocessinthesimulationstudy(seeAppendixA.1ofthesupplementarymaterial).Thenonlinearpro-cesshasbeenconsideredin(Medeiros,Ter¨asvirta,&Rech,