Lessons Learned from Backcasting and Forecasting Exercises 16th TRB National Transportation Planning Applications Conference May 16 2017 Thomas Rossi Cambridge Systematics Inc Sarah Sun Federal Highway Administration ID: 764122
Download Presentation The PPT/PDF document "Lessons Learned from Backcasting and For..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Lessons Learned from Backcasting and Forecasting Exercises 16th TRB National Transportation Planning Applications Conference May 16, 2017 Thomas Rossi, Cambridge Systematics, Inc. Sarah Sun, Federal Highway Administration
Backcasting and Forecasting in Validation Base year validation demonstrates a model’s ability to estimate travel behavior for a single point in time A second point would help demonstrate sensitivity of the model to changing conditions that affect travel demand 2 Backcasting and forecasting: Compare model results to observed data for a year other than the base year For forecasting, choose year after base year but still in the past
Study Approach Use two different model versions with different base yearsBackcast or forecast the base year scenario for the “other” model version Use models from two urban areas: Baltimore (BMC) Cincinnati (OKI) 3
Key Observations Changes in input data may have effects that dominate the resultsExamples: Higher network speeds in updated model, lower trip rates in updated modelResults for base year scenarios match observed data better than forecasts/backcasts No surprise here, as validation was done mainly considering the base year There is more consistency between scenarios run using the same model than between scenarios run for the same travel conditions (analysis year) 4
What We Learned Impacts of changes in model parameters between model versionsBehavior changes, errors fixed, improved model form, better hardware, expanded analytical capabilities Effects of changes can propagate, models cannot anticipate everything, updating a model does always make it more “correct” Accuracy of data inputs is important Forecasting/backcasting can help identify “hard to find” errors Changes in assumptions can have unanticipated effects Effects of changes must be considered in validation 5
What Else We Learned Calibration changes should be made only to improve the model’s predictive ability Earlier components show greater accuracy in forecasting (maybe)May reflect error propagation from earlier components downstream Use of fixed factors can make models insensitive to changes over time When a model is updated, the fixed factors are reestimated…and results change 6
Recommendations Validation should always include temporal validation and sensitivity testingTemporal model validation should include a backcast and/or a forecast year application Recognize that changes in model procedures, assumptions, and input data can change model resultsModel inputs need to be thoroughly checked during model development and validation Estimate effects of changes in calibrated parameters Recognize effects of error propagation Test sensitivity effects of fixed factors 7
Resources Project report link:https:// www.fhwa.dot.gov/planning/tmip/publications/other_reports/predictive_tool/index.cfmWebinar (December 2016) Summary: https ://www.fhwa.dot.gov/planning/tmip/community/webinars/summaries/20161214 /Recording: https://connectdot.connectsolutions.com/p609mzn3nyf/Slides:https:// content.govdelivery.com/attachments/USDOTFHWAHEP/2016/12/14/file_attachments/711336/TMIP%2Bwebinar%2B12%2B14%2B2016%2BFinal%2BComp.pdf 8
Acknowledgments Sarah Sun, FHWAKazi Ullah, Cambridge Systematics And especially…Baltimore Metropolitan Council (BMC)Charles Baber Matt de Rouville Ohio-Kentucky-Indiana Council of Governments (OKI) Andrew Rohne 9