InternationalJournalofForecasting19 (2003) 87 - PDF document

InternationalJournalofForecasting19 (2003) 87–94www.elsevier.com/locate/ijforecastForecastingcombinationandencompassingtests 1.IntroductionexaminationofcompetingforecastsshouldbetakentoassesswhetherthesmallerRMSEofthecombined anacceptablespecicationcanbefound,thenopti- FangInternationalJournalofForecastingandBunn(1998)andTaylorandBunn(1999)forquarterlyUKconsumptiondatawithashortertimediscussionsoncombinedforecasterrordistributions.period(from1957(3)to1967(4)).ThethreeARIMAInthispaperwedemonstratethatforecasten-models(TS1–TS3)arecompassingtestsarevaluabletoolsingettingan,(1)insightintowhycompetingforecastsmaybecom-binedtoproduceacompositeforecastwhichisTS2:(1,(2)superiortotheindividualforecasts.Wealsoarguethatresultsfromforecastencompassingtestsarepotentiallyusefulinmodelspecication.WedothisTS3:(1)(1,(3)148forforecastsofquarterlyUKconsumptionexpendi-turedatafrom1957(1)to1975(4).ThemodelsisthenaturallogarithmofUKquarterlyincludevariousversionsofautoregressiveintegratedseasonallyunadjustednon-durableconsumptionex-moving-average(ARIMA)models(Box,Jenkins,&penditureinmillionsofpoundsat1970prices(see,Reinsel,1994),modelsbasedontheworkofDavid-e.g.,CharemzaandDeadman,1997)andisason,Hendry,SrbaandYeo(1978)(DHSYhereafter)Gaussianwhitenoiseprocess.Thesymbolsandvectorautoregression(VAR)models(Sims,denotedifferenceoperators(1980).Becauseallforecastingmodelsarelikelyto)andistheback-shiftoperatorbemisspecied—theyareapproximationstoamuchmorecomplexreality—theresultsinthisThesecondclassofmodelsarebasedonthepapershouldhavepracticalvalue.restrictedDHSYmodel(Davidsonetal.,1978):2.Forecastingmodelsforconsumption,(4)istheUKpersonaldisposableincomein1970prices,measuredinthenaturallogarithms.TheThemodellingofconsumptionisatthecentreofisa‘specialeffects’dummyvariabletheKeynesianeconomictheoryandtheconsumer’sreectingchangesinindirecttaxation(1968(1)(2)expenditureisoneofthemostimportantaggregatesand1973(1)(2)).topredictinmacroeconomics.AgreatdealofWeconsiderfourversionsofDHSYmodelsbasedresearchhasbeendevotedtounderstandingtheon(4).Thedifferenceliesinthewaytheexogenousaggregateconsumer’sexpenditureandprogresscon-ishandled.Ex-anteforecastsoftinuesatarapidpaceaftermorethanahalfcenturyobtainedusingpredictedfromfourARIMAsincetheadvancesinKeynesiantheoryinthe1930sprocessesfortheUKpersonaldisposableincomeand1940s.Forexample,wehavewitnessedthe(Prothero&Wallis,1976):evolutionofthenonstructuralandstructuralap-proachestomacroeconomicforecasting(Diebold,,(5)1998);recentmethodologicaldevelopmentsineconometricssuchasVARmodels,cointegrationand,(6)Grangercausality;andtheongoingdebateaboutwhetheroneshouldorshouldnotbeconcernedabout)(1,(7)144the‘atheoretical’natureoftheVARapproach(see,e.g.,Canova,1999).HereweconsideratotalofninemodelsforUK quarterlyconsumptionexpendituredatafromTheDHSYmodelhadanimportantinuenceonrecentdevelop-1957(1)to1975(4):threeARIMAmodels,fourmentsinappliedeconometricanalysisanddelineatedtheframe-DHSYmodelsandtwoVARmodels.Thespecica-workformanysubsequentempiricalresearch.SeeHendry,tionsofthethreeARIMAmodelsarebasedontheMuellbauerandMurphy(1990)andCharemzaandDeadmanresultsofProtheroandWallis(1976)usingthe(1997)fordiscussionsontheeconometricsofDHSY. FangInternationalJournalofForecasting.(8)becausetheyimposedifferentrestrictionsonthereducedform.If(4)isused,butfortheforecastingperiodisToprovideacommonperspectiveonforecastestimatedfrom(5)–(8),thenwerefertotheDHSYcombinationfortheseninemodels,wecombineeachmodelsasDHSY1,DHSY2,DHSY3andDHSY4.pairofforecasts(usingequalweights)andcomputeWealsoconsidertheVARmethodwhichallowsRMSEs(MAEsandRMSPEs)forthecombinedforcross-variabledynamics.Sincethetimeseriesforecasts(off-diagonalentriesinTable1),regardlessexaminedinthispaperisrelativelyshort,weconsi-ofwhetheroneforecastissignicantlybetterthandertworestrictedVARmodelsoforder5focusingtheothers.Tomakeaneasyevaluation,wemarktheonlyonlags1,4and5,denedasfollows:combinedforecastswithasterisksiftheirRMSEinpanelA(MAEinpanelBorRMSPEinpanelC)areVAR1:011141atleast5%(say)lessthanthoseoftwocorre-spondingindividualforecasts.,(9)ResultsfromTable1indicatethatthecombinedforecastsfromtheARIMAandDHSYmodelsVAR2:011141appeartohavesmallerforecasterrors;asallcasesin,(10)panelsAthroughCdemonstrated,theRMSE(MAEandRMSPE)ofthecombinedforecastsfromtheisthettttttttARIMAandDHSYmodelsareatleast5%lessthanconsumers’expendituredeatorindexin1970thoseofthecorrespondingindividualforecasts.prices,measuredinthenaturallogarithms.SomecombinedforecastsfromVARsandARIMAareerrorterms.Thereareatotalof14parametersin(orDHSY)modelsappeartobebetterthancorre-(9)and30parametersin(10).spondingindividualforecasts.Theviewthatpoolingforecastsfromthreeclassesofmodels(i.e.ARIMAmodels,DHSYmodelsandVARmodels)mayleadtobetterforecastsisconrmedbytheforecast3.ForecastsandtheircombinationsencompassinganalysisinSection4.Wedemonstratetherethatallninemodelsaremisspecied.MoreWeusetherst10years(1957(1)–1966(4))asthespecically,severalmodelsmaynoteffectively‘tting’periodandthelast9years(1967(1)–predictthedynamicsof,andsomeARIMA1975(4))asthe‘forecasting’period.Forillustrativemodels,DHSYmodelsandinparticular,twoVARpurposes,weconsideronlyone-quarter-aheadfore-modelsfailincapturingallinformationrelevanttocastsbasedonmodelswhichareestimatedusingalldataavailableattheforecastingtime;thisrollingapproachallowseachmodeltobeestimated36timesandyields36one-quarter-aheadforecasts.ThediagonalentriesinpanelAofTable1are4.ForecastencompassingRMSEsofone-quarter-aheadforecastsoverthe9yearsforecastingperiodforeachoftheninemodels.ForecastencompassingtestsseektoevaluateToobtainaperspectiveontherobustnessofthewhethercompetingforecastsmaybefruitfullycom-resultstodifferentevaluationmeasures,wealsobinedtoproduceaforecastsuperiortoindividualreportthemeanabsoluteerrors(MAEs)androotforecasts.Suchtestscanbeimplementedbyregres-meansquaredpercenterrors(RMSPEs)forindi-singtheactuallevelof(orthechangein)onthevidualforecastsinpanelsBandC(diagonalentries),s(orthepredictedchanges)fromtworespectively.Ascanbeseen,thethreeARIMA,fourmodels.Forexample,onemayconsidertheregres-DHSYandtwoVARmodelshavesimilarforecastingsionmodel(Chong&Hendry,1986;Ericsson,1993)performances(measuredbyeitherRMSE,MAEorRMSPE).Theslightdifferencesinforecastsariseeitherbecausethemodelsusedifferentdataor(11) FangInternationalJournalofForecastingTable1Forecastaccuracyofindividualandcombinedforecasts TS1TS2TS3DHSY1DHSY2DHSY3DHSY4VAR1VAR2 PanelA:RMSETS10.01310.01240.01250.0115*0.0119*0.0120*0.0120*0.0121*0.0124*TS20.01300.01270.0115*0.0123*0.0123*0.0120*0.0120*0.0121*TS30.01350.0116*0.0124*0.0124*0.0121*0.0120*0.0122*DHSY10.01230.01270.01280.01280.0113*0.0117DHSY20.01340.01350.01330.0118*0.0121*DHSY30.01370.01350.0118*0.0121*0.01350.0117*0.0121*VAR10.01260.0127VAR2PanelB:MAETS10.01100.0099*0.01050.0087*0.0089*0.0090*0.0090*0.00940.0098TS20.01080.01040.0087*0.0093*0.0093*0.0090*0.00940.0093*TS30.01110.0090*0.0095*0.0096*0.0092*0.00950.0095*DHSY10.00960.00930.01000.00990.0088*0.0091*DHSY20.01030.01030.01020.0089*0.0091*DHSY30.01040.01030.0089*0.0092*0.01020.0089*0.0093*VAR10.00950.0098VAR2PanelC:RMSPETS10.001470.001390.001400.00129*0.00134*0.00134*0.00134*0.001360.00134*TS20.001460.001420.00129*0.00137*0.00137*0.00134*0.00134*0.00135*TS30.001510.00130*0.00139*0.00139*0.00136*0.001350.00137*DHSY10.001370.001420.001430.001440.00127*0.00131DHSY20.001500.001510.001490.00132*0.00136*DHSY30.001530.001510.00132*0.00135*0.001510.00132*0.00136*VAR10.001420.00143VAR2 InpanelA*indicatesthattheRMSEofthecombinedforecastisatleast5%lessthanthoseoftwoindividualforecasts.InpanelB*indicatesthattheMAEofthecombinedforecastisatleast5%lessthanthoseoftwoindividualforecasts.InpanelC*indicatesthattheRMSPEofthecombinedforecastisatleast5%lessthanthoseoftwoindividualforecasts.andtestfor1(or0)conditionalonindependentinformationforone-quarter-aheadfore-121istheforecastofmadefrommodelcastingofshouldbothbenonzero.tttusinginformationavailableattimeTheset-upof(12)isdifferenttothatin(11):isthesamethingformodel.andarenotsubjecttotheconstraintthatThisisavariantofthetestduetoFairandShillersumtoone.Itisusuallypreferablenottoforce(1989,1990),whichhadtheformof:andtosumtounityforasFairandShillerargued,(12)maybemoresensitivebecausetherearecasesinwhichtheconstraint(1)doesnotmakesense.Forexample,ifforecastsfrombothmodels.(12)arejustnoise,theestimatesofboth0and0,thesecondmodelforecastbezero.FairandShillerconsideredinsteadofencompassestherst.Onthecontrary,therstbecausethetimeseriesofinterestintheirmodelforecastencompassesthesecondif0andempiricalstudyisnonstationary.Theinclusionofan0.Inthecasethatbothforecastscontaininterceptisalsodesiredsinceitfacilitatesbias FangInternationalJournalofForecastingcorrectionandallowsbiasedforecaststobeevalu-forecastencompassingbyremovingmorecommonated.componentsfromtworegressors.Inaset-uplikeSimilartoFairandShiller’s(1989,1990)ap-(13),forecastencompassinghypothesescanbetestedproach,weconsiderforecastencompassingtestsusingthestandardregressionmethods.basedon(12)with1,andinadditiontheTables2and3reporttheestimationresultsof(12)followingregressionmodel:and(13)foreachpairoftheninemodels,respective-ly.Theridgeregressionestimatesof012
aregiveninTable2.Table3representsordinaryleastsquaresestimateswithasterisksindicatingthat
thecorrespondingtestsarestatisticallydifferentfrom(13)zeroatthe5%levelofsignicance.WecorrectforbothheteroskedasticityandthemovingaverageWenotethat(12)and(13)usedifferentre-processintheestimationofthestandarderrorsofthegressands.ThetwoproceduresprovidedifferentcoefcientestimatesinTable3usingtheprocedureinsightsintowhichinformation,relevanttoforecast-givenbyHansen(1982)andWhite(1980).SeeFairinforecastsfromonemodel,isnotinandShiller(1990)fordetailedformulae.forecastsfromanothermodel.InexaminingtheresultsfromTables2and3,fourIn(12),theinformationcontainedinonemodel’sgeneralconclusionsemerge.Firstly,threeARIMAforecastcomparedtothatinanotherisassessedfrommodels,fourDHSYmodelsandtwoVARmodelsaregressionoftheactualchangesonpredictedeachcontainindependentinformationrelevanttochangesfromtwomodels.Anadvantageofusing.Neitherclassencompassestheother.incomparisonto,say,asaregressand,isForexample,bothestimatedvaluesofitssimplicity.Notethatifthesecondmodelforecast(12)appearnottobezeroforforecastsfromanyencompassestherst(i.e.0),asshownbelow,ARIMAmodels(asmodelI)andDHSYmodels(as0and1.Hence,saccountforalmostmodelII);asthecaseofTS2vis-a-visfourDHSYallinformationcontainedinout-of-samples.Ifmodelsdemonstrates,thevaluesof-coefcientforbothforecastscontainindependentinformationforTS2rangesfrom0.360to0.568,whereasthoseforone-quarter-aheadforecastingof,thenvaluesoffourDHSYmodelsareboundedby0.338and0.543.buildvaluableintuition—therelativeThisimpliesthatforone-quarter-aheadforecastingimportanceofforecastsfromtwomodelsinexplain-horizon,forecastcombinationoftheARIMA,theinginformationcontainedin.Wenotethatal-DHSYandtheVARmodelsisdesirable.Thisresultthoughtheconstraint(1)isnotimposed,isconsistentwiththendingfromTable1.theestimatedvaluesofoftensumnicelySecondly,theforecastencompassingapproachisintoanumberclosetooneforourdata(seeTable2).complementarytotheRMSE(orothercriteriasuchOnereasonforthisisthatallregressorshavestrongasMAEandRMSPE),since,asdemonstratedincorrelationswiththeregressand(thecorrelationresultsinTables1–3,theforecastencompassingcoefcientsarearound0.9).approachcanoftendiscriminatetwomodelsevenThemaindisadvantageoftestsbasedon(12)iswhentheRMSEsareclosefortwoforecastsandthatiftwomodelscontainthesameinformation,determinewhethertheforecastwiththehigherthentheforecastsarehighlycorrected,andsoandRMSEcontainsinformationnotintheotherforecast.arenotseparatelyidentiedduetothesevere collinearityproblem.AsanadhocsolutiontotheThebiasingconstantintheridgeregressionischosentobefromcollinearityproblem,weuseridgeregressionto0.005to0.01,whichproducestableregressioncoefcientsandsufcientlysmallVIFvalues.Becausetheordinaryinferenceobtainestimationof(12).proceduresarenotapplicableinridgeregressionanalysisduetoIncontrast,theregression(13)usesasthethelackofknowledgeoftheexactdistributionalpropertiesoftestregressand,whichisobtainedusingboththerstandstatistics,wedonotgivesignicancelevelsofestimatedco-the(quarterly)seasonaldifferences.Focusingonefcients.SeeNeter,Kutner,andNachtsheim(1996)foraallowsonetogoastepfurthertostudytextbooklevelintroductiontoridgeregression. FangInternationalJournalofForecastingTable2Comparisonofforecasts:estimatesof(12) CONSTTS1TS2TS3DHSY1DHSY2DHSY3DHSY4VAR1VAR2 10.0040.0940.80720.0040.0250.87230.0030.2550.65240.0010.1480.75350.0020.1790.72060.0030.3640.54770.00580.00390.0030.3550.549100.0030.4780.428110.0020.3600.543120.0020.3810.521130.0030.5680.338140.0050.1100.788150.004160.0030.5270.379170.0020.4160.487180.0020.4320.470190.0030.5680.338200.0040.1720.727210.0040.0090.942220.0010.2710.628230.0010.2940.604240.0020.6320.269250.0040.1530.748260.0040.0160.968270.0010.4730.424280.0010.7980.098290.0040.2560.646300.0030.0870.863310.0010.7880.105320.0040.2610.641330.0030.1020.847340.0040.0950.804350.0040.0581.010360.0040.0830.866 Forexample,DHSY3andDHSY4encompassTS1ifsets.Onlyafewstudiesoncomparisonsbetweentwoonefocuseson(Table3),althoughDHSY3classescan,however,befoundintheliteratureandandDHSY4haveslightlyhigherRMSEsthanthatoftheresultsaremixed.Forexample,Nelson(1972)TS1(Table1).(SeealsoArmstrongandCollopyconsideredone-quarter-aheadforecastsof14econ-(1992),andClementsandHendry(1993)onlimita-omicvariablesandfoundthatthetimeseriesmodeltionsofMSEerrormeasures.)performsbetterinex-anteforecastcomparison,butFurthermore,ourresultsextendandsupportnd-withinthesampletheeconometricmodelisahead.ingsreportedinpreviousstudiesontheforecastOntheotherhand,Christ(1975)reportedcom-accuracycomparisonbetweendifferentmodelspri-parisonsofforecastsofrealandnominalGNPinmarilybasedoncriteriasuchasRMSE,MAEorwhichARIMAforecastsare‘uniformlythepoorest’.RMSPE.Thereweremanystudiesonforecastac-InboththeNelsonandChriststudies,asimplelosscuracycomparisonwithintheclassofARIMAfunction,RMSE,isusedtoprovideanoverallmodelsorofeconometricmodelsusingsimilardataforecastingaccuracymeasure.Ourresultsshowthat FangInternationalJournalofForecastingTable3Comparisonofforecasts:estimatesof(13) CONSTTS1TS2TS3DHSY1DHSY2DHSY3DHSY4VAR1VAR2 0.9010.14720.0010.743*0.345*0.717*0.355*0.6990.336*60.0010.6910.339*70.00180.0000.0011.0540.4140.380*0.591*0.478*0.5620.440*130.0010.4040.386*140.0010.591*0.406*0.739*0.512*0.696*0.461*190.0010.4040.386*0.5120.033220.0010.2560.122230.0010.1530.2270.0590.4250.0010.408*0.0010.395*270.0010.0790.4350.0130.3800.0010.442*0.0020.432*0.0010.0650.308320.0010.390*0.0020.389*350.001 *Statisticallysignicantatthe5%level.ARIMA,DHSYandVARclassesofmodelscontainnotethatitmaybeimportanttoconsiderallmostindependentinformationonforecastingquarterlyUKrelevantforecastencompassingtestsbecauseonetestconsumptionexpenditure.Theforecastencompas-maybecomplementarytoanother.Forexample,thesingtestscandiscriminatewellbetweenthem.estimationresultsof(12)suggestthatVAR2en-Finally,thendingfromforecastencompassingcompassesallothermodels.TheresultsfromTable3testscanbeviewedasprimafacieevidenceofmodelbasedontheestimationof(13)indicate,however,misspecication.Inparticular,onemaytestmis-thatTS1andfourDHSYmodelsencompassVAR2.specicationviadifferentformsofregressionssuchThecontrastisclear,dependingonwhetheroneas(12)and(13).Forexample,theresultsfrom(13)shouldfocuson.SincebothseemhighlysuggestiveofthepossibilitythatneitherplayakeyroleinmodelingthequarterlyUKtheARIMAmodels,DHSYmodels,norVARsareconsumptionexpendituredata,anyinformationonacceptableinmodelingthecomponent.Wepossiblemisspecicationonthosetwotermsshould FangInternationalJournalofForecastingDiebold,F.(1989).Forecastcombinationandencompassing:provetobevaluableinimprovingmodelspecica-reconcilingtwodivergentliteratures.InternationalJournalof,589–592.Diebold,F.(1998).Thepastpresent,andfutureofmacro-economicforecasting.JournalofEconomicPerspectives(Spring),175–192.Ericsson,N.R.(1989).Parameterconstancy,meansquaredforecastserrors,andmeasuringforecastperformance:anThepaperhasbenetedfromcommentsfromexpositionandextensions.InEricsson,N.,&Irons,J.(Eds.),WilpenL.Gorr(aneditor),theassociateeditor,andTestingExogeneity.Oxford:OxfordUniversityPress.theanonymousreferee,aswellasfromseminarEricsson,N.R.(1993).Commenton‘Onthelimitationsofparticipantsatthe2000ASAannualmeeting.comparingmeansquaredforecasterrors’byM.P.ClementsandD.F.Hendry.JournalofForecast,644–651.Fair,R.C.,&Shiller,R.(1989).Theinformationalcontextofexanteforecasts.TheReviewofEconomicsandStatistics325–331.Fair,R.C.,&Shiller,R.(1990).ComparinginformationinArmstrong,J.S.,&Collopy,F.(1992).Errormeasuresforforecastsfromeconometricmodels.TheAmericanEconomicgeneralizingaboutforecastingmethods:empiricalcomparisons.,375–389.InternationalJournalofForecasting,69–80.Hansen,L.(1982).LargesamplepropertiesofgeneralizedmethodBox,G.,Jenkins,G.,&Reinsel,G.(1994).In3rded,TimeSeriesofmomentsestimators.,1029–1054.ForecastingandControl.NewJersey:PrenticeHall.Hendry,D.F.,Muellbauer,J.,&Murphy,A.(1990).TheCanova,F.(1999).Vectorautoregressivemodels:specication,econometricsofDHSY.InHey,J.,&Winch,D.(Eds.),estimation,inference,andforecasting.InPesaran,M.H.,&CenturyofEconomics:100yearsoftheRoyalEconomicWickens,M.R.(Eds.),HandbookofAppliedSocietyandtheEconomicJournal.Oxford:BasilBlackwell.,vol.1.Oxford:Blackwell,pp.73–138.Nelson,C.R.(1972).ThepredictionperformanceoftheFRB-Charemza,W.W.,&Deadman,D.F.(1997).In2nded,MIT-PENNmodeloftheUSeconomy.TheAmericanEconDirectionsinEconometricPractice.Cheltenham:EdwardomicReview,902–917.Neter,J.,Kutner,M.,Nachtsheim,C.,&Wasserman,W.(1996)).Chong,Y.Y.,&Hendry,D.F.(1986).EconometricevaluationofIn3rded,AppliedLinearRegressionModels.Chicago:linearmacroeconomicmodels.ReviewofEconomicStudiesIRWIN.,671–690.Prothero,D.L.,&Wallis,K.F.(1976).Modellingmacro-Christ,C.F.(1975).Judgingtheperformanceofeconometriceconomictimeseries.JournalofRoyalStatisticalSocietyAmodelsoftheUSeconomy.InternationalEconomicReview,468–499.,54–74.Sims,C.A.(1980).Macroeconomicsandreality.Clemen,R.(1989).Combiningforecasts:areviewandannotated,1–48.InternationalJournalofForecasting,559–Taylor,J.,&Bunn,D.(1999).Investigatingimprovementsintheaccuracyofpredictionintervalsforcombinationsofforecasts:aClements,M.P.,&Hendry,D.F.(1993).Onthelimitationsofsimulationstudy.InternationalJournalofForecastingcomparingmeansquaredforecasterrors.JournalofForecast325–339.,617–637.White,H.(1980).AheteroskedasticityconsistentcovarianceClements,M.P.,&Hendry,D.F.(1998).ForecastingEconomicmatrixestimatorandadirecttestforheteroskedasticity.TimeSeries.Cambridge:CambridgeUniversityPress.,817–837.Davidson,J.E.H.,Hendry,D.F.,Srba,F.,&Yeo,S.(1978).Econometricmodellingoftheaggregatetimeseriesrelation-shipbetweenconsumers’expenditureandincomeintheUnitedYueFANGisassistantprofessorofDecisionSci-EconomicJournal,661–692.encesattheLundquistCollegeofBusiness,UniversityofOregon.deMenezes,L.,&Bunn,D.(1998).Thepersistenceofspe-HereceivedhisPh.D.fromMITandisauthorofanumberofcicationproblemsinthedistributionofcombinedforecastacademicpublicationsontimeseriesmodeling,economicfore-InternationalJournalofForecasting,415–426.castingandnancialeconometrics.