Machine Learning is Better – But When? Keith Ord

Transcript:Machine Learning is Better – But When? Keith Ord:
Machine Learning is Better – But When? Keith Ord Professor Emeritus of Business Statistics When? A recent study by Efron (2020) provides a detailed assessment of the strengths and weaknesses of traditional regression-type methods and pure prediction algorithms. Efron states “When they are [his italics] suitable, the pure prediction methods can be stunningly successful.” The key question is “When?” 9/9/2222/229/21/2022 Federal Forecasters Conference 2 Outline M5 Uncertainty Competition: Overview Issues with Data Analysis The Weight of Numbers Are the criteria used in the study appropriate? Case study using Box-Jenkins Airline data What are the take-home lessons? 9/21/2022 Federal Forecasters Conference 3 M5 Uncertainty Competition [See Makridakis, Spiliotis and Assimakopoulos (2022b) – report on the M5 Uncertainty Competition[refer to as MSA22b] Data refer to day-by-day product sales by Walmart The data are hierarchical by state, store, category, department, product. The most granular levels (numbered 10-12) refer to Product, Product*State and Product*Store 42,840 series relating to 3,049 products; all but 154 series in Levels 10-12 Additional information was available on holidays and prices 1913 days for model development and estimation; 28 days of forecasts (1 to 28 days ahead) generated for evaluation 892 entries in Kaggle competition 6 statistical benchmarks: ARIMA and exponential smoothing methods; no consideration of holidays or price data. No combinations of benchmark methods 9/21/2022 Federal Forecasters Conference 4 Published Conclusions "The most important finding of the M4 competition was that all of the top-performing methods, in terms of both PFs (Point Forecasts) and PIs (Prediction Intervals) were combinations of mostly statistical models, with such combinations being more accurate numerically than either pure statistical or pure ML methods.“ Makridakis, Spiliotis and Assimakopoulos (2020) – report on the M4 Competition Two principal conclusions in the M5 Competition: The superior performance of relatively simple ML (machine learning) methods… as well as the significantly worse performance of statistical methods which did not make it to the top ranks. The [substantial] improvement […] of the winning method over the most accurate statistical benchmark… . Makridakis, Spiliotis and Assimakopoulos (2022a) – report on the M5 Accuracy Competition WHAT CHANGED? 9/21/2022 Federal Forecasters Conference 5 Data Analysis Many of the series extend to nearly 2,000 observations; ML methods typically require long series for fitting, whereas statistical methods do not. Conclusions do not extend to other contexts with “short” series. No use of combinations of statistical methods, despite recommendations from M4. No reporting of preliminary data