Anne Morse Huércanos PhD Estimates and Projections Area Population Division This presentation is released to inform interested parties of ongoing research and to encourage discussion of work in progress Any views expressed are those of the authors and not necessarily those of the US C ID: 1027538
Download Presentation The PPT/PDF document "1 SIMPLE VERSUS COMPLEX MODELS" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. 1SIMPLE VERSUS COMPLEX MODELSAnne Morse [Huércanos], PhDEstimates and Projections AreaPopulation DivisionThis presentation is released to inform interested parties of ongoing research and to encourage discussion of work in progress. Any views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.
2. 2OutlineConsiderations other than accuracyTheoretical reasons for the [possible] superiority of simplicityThe nature of the future/time-span Decomposition/causal modelsOverfitting/mistaking noise for signal
3. 3Considerations other than accuracyInternal consistencyUse of most recent dataConsideration of relevant variablesCost of developmentEase of explanationUsefulness for policymakingSource: Smith, Stanley K. "Further thoughts on simplicity and complexity in population projection models." International Journal of Forecasting 13.4 (1997): 557-565.
4. 4Considerations other than accuracy (cont.)Journals favor complexityClients may be reassured by incomprehensibilitySources: Smith, Stanley K. "Further thoughts on simplicity and complexity in population projection models." International journal of forecasting 13.4 (1997): 557-565.Green, Kesten C., and J. Scott Armstrong. "Simple versus complex forecasting: The evidence." Journal of Business Research 68.8 (2015): 1678-1685.
5. 51. The reality is simplePant, P. Narayan, and William H. Starbuck. "Innocents in the forest: Forecasting and research methods." Journal of Management 16.2 (1990): 433-460.Rogers, Andrei. "Population forecasting: Do simple models outperform complex models?." Mathematical Population Studies 5.3 (1995): 187-202.Simple models: No-change modelsLinear extrapolation modelsNo-change models may be hard to beatKnowledge is insufficientThe situation is stableSources: Green, Kesten C., and J. Scott Armstrong. "Simple versus complex forecasting: The evidence." Journal of Business Research 68.8 (2015): 1678-1685.
6. 61. The reality is simple (cont.)Pant, P. Narayan, and William H. Starbuck. "Innocents in the forest: Forecasting and research methods." Journal of Management 16.2 (1990): 433-460.Rogers, Andrei. "Population forecasting: Do simple models outperform complex models?." Mathematical Population Studies 5.3 (1995): 187-202.Sources: Green, Kesten C., and J. Scott Armstrong. "Simple versus complex forecasting: The evidence." Journal of Business Research 68.8 (2015): 1678-1685. Why might no-change or linear extrapolation models produce more accurate results than complex models? The reality (especially in the short run) is often simple Time series are highly autocorrelatedMore recent data are typically more relevantUse exponential smoothing
7. 72. Decomposition and causal modelsSimple models: directly project the variable of interestComplex models: Decompose the variable into parts orProject using a casual variable
8. 82. Decomposition and causal models (cont.)Why might decomposing or using a causal model produce less accurate results?Basic causal factors may act more directly on the total than on the partsSource: O'neill, Brian C., et al. "A guide to global population projections." Demographic Research 4 (2001): 203-288.
9. 92. Decomposition and causal models (cont.)Why might decomposing or using a causal model produce less accurate results?The data may be worseOf the causal variableOf the decomposed components The data may be ShortNoisyNon-representativeSource: O'neill, Brian C., et al. "A guide to global population projections." Demographic Research 4 (2001): 203-288.
10. 102. Decomposition and causal models (cont.)Sources: Vaupel, James W., and Anatoli I. Yashin. "Heterogeneity's ruses: some surprising effects of selection on population dynamics." The American Statistician 39.3 (1985): 176-185.
11. 112. Decomposition and causal models (cont.)Why might decomposing or using a causal model produce less accurate results?Input variables may be harder to project than the output variableSource: O'neill, Brian C., et al. "A guide to global population projections." Demographic Research 4 (2001): 203-288.0-485+5-9…25-29Current (t)Future (t+5)5-9…10-1410-140-4
12. 122. Decomposition and causal models (cont.)Why might we decompose even if the input variable is harder to project than the output variable?Stakeholder needsSource: O'neill, Brian C., et al. "A guide to global population projections." Demographic Research 4 (2001): 203-288.
13. 132. Decomposition and causal models (cont.)We expect the cohort-component method to be more accurate than a simple method when the trends of the input variables are relatively stableWe expect the cohort-component method to be less accurate than a simple method when there is a major turning point in the input trendThis makes the piece more difficult to predict than the wholeSource: O'neill, Brian C., et al. "A guide to global population projections." Demographic Research 4 (2001): 203-288.
14. 142. Decomposition and causal models (cont.)We expect the cohort-component method to be less accurate than a simple method when there is a major turning point in the input trendPant, P. Narayan, and William H. Starbuck. "Innocents in the forest: Forecasting and research methods." Journal of Management 16.2 (1990): 433-460.Rogers, Andrei. "Population forecasting: Do simple models outperform complex models?." Mathematical Population Studies 5.3 (1995): 187-202.Sources:Historical Population Change Data (1910-2020), U.S. Census BureauProjection with a simple (average rate) model
15. 152. Decomposition and causal models (cont.)We expect the cohort-component method to be less accurate than a simple method when there is a major turning point in the input trendSources:Historical Population Change Data (1910-2020), U.S. Census BureauZitter, Meyer and Siegel Jacob. Illustrative Projections of the Population of the United States by Age and Sex 1960 to 1980. Current Population Reports. 25.187 (1958) Projection with a complex (cohort-component) model
16. 162. Decomposition and causal models (cont.)We expect the cohort-component method to be less accurate than a simple method when there is a major turning point in the input trendSources:Hamilton BE, Lu L, Chong Y, et al. Natality trends in the United States, 1909–2018. National Center for Health Statistics. 2020.Zitter, Meyer and Siegel Jacob. Illustrative Projections of the Population of the United States by Age and Sex 1960 to 1980. Current Population Reports. 25.187 (1958) Projection with a complex (cohort-component) model
17. 172. Decomposition and causal models (cont.)Why might using a causal model produce less accurate results?Source:Vigen, Tyler. Spurious correlations. Hachette UK, 2015. https://tylervigen.com/spurious-correlationsA weak causal relationship
18. 182. Decomposition and causal models (cont.)Decomposition is most useful when:There is valid and reliable information about each element Elements are subject to different causal forces Elements are easier to predict than the wholeSources: Green, Kesten C., and J. Scott Armstrong. "Simple versus complex forecasting: The evidence." Journal of Business Research 68.8 (2015): 1678-1685.
19. 193. OverfittingComplex forecasting methods mistake random noise for informationThe key is in determining whether the patterns represent noise or signalNeed for theory!Predictions may be more prone to failure in the era of Big dataSources:Silver, Nate. The signal and the noise: Why so many predictions fail-but some don't. Penguin, 2012.
20. 203. Overfitting (cont.)Overfitting occurs when the model is fit so closely to the training data that it loses its generalizabilityMore likely to overfit when:Data are noisyPoor understanding of fundamental relationshipsSources:Silver, Nate. The signal and the noise: Why so many predictions fail-but some don't. Penguin, 2012.
21. 213. Overfitting (cont.)More likely to overfit when:Data are noisy Make data less noisy Smooth and de-seasonalize!Sources:Makridakis, Spyros, Evangelos Spiliotis, and Vassilios Assimakopoulos. "Statistical and Machine Learning forecasting methods: Concerns and ways forward." PloS one 13.3 (2018): Pant, P. Narayan, and William H. Starbuck. "Innocents in the forest: Forecasting and research methods." Journal of Management 16.2 (1990): 433-460.Mirzavand, Mohammad, and Reza Ghazavi. "A stochastic modelling technique for groundwater level forecasting in an arid environment using time series methods." Water Resources Management 29.4 (2015): 1315-1328.
22. 223. Overfitting (cont.)“There is no reason for such complex methods to be less accurate than simple statistical benchmarks, at least once their shortcoming of over-fitting is corrected.”Sources:Makridakis, Spyros, Evangelos Spiliotis, and Vassilios Assimakopoulos. "The M4 Competition: 100,000 time series and 61 forecasting methods." International Journal of Forecasting 36.1 (2020): 54-74.
23. 233. Overfitting (cont.)Sources:Makridakis, Spyros, Evangelos Spiliotis, and Vassilios Assimakopoulos. "The M4 Competition: 100,000 time series and 61 forecasting methods." International Journal of Forecasting 36.1 (2020): 54-74.“Men may construe things after their fashion/ Clean from the purpose of the things themselves.” —Shakespeare’s Cicero
24. 24ConclusionConsciously articulate all considerations that influence your choicesCarefully consider when to include a causal model or when to decomposeIs the causal theory strong?Are the data usable?Is the element easy to predict?
25. 25ConclusionConsciously articulate all considerations that influence your choicesCarefully consider when to include a causal model or when to decomposeGet rid of unnecessary noise in your data: Smooth and deseasonalized!Don’t overfit your data
26. 26Thank you!Anne Morse [Huércanos]anne.morse@census.govEstimates and Projections AreaPopulation DivisionU.S. Census Bureau