1734729 - PowerPoint Presentation

Transition plan for MAPP/CTB Funded Proposal: Improved probabilistic forecast products for the NMME seasonal forecast system Anthony Barnston (PI), Huug van den Dool , Emily Becker, Michael Tippett , Shuhua Li   Demonstration The project aims to improve the NMME probabilistic forecasts by addressing systematic biases in both forecast anomalies and categorical probabilities. The corrected NMME forecasts will have improved reliability and accuracy . The i mprovements will come about due to (1) corrections in pattern placement and p attern characteristics (developed at IRI) , and (2) refinements to local probability a nomalies (developed at CPC) . The development work toward (1) and (2), currently i n progress, will be done during year 1 and the first half of year 2. Operational Deployment During the last quarter of the last year, the software associated with the NMME corrections /improvements will be amalgamated at NOAA/CPC in order that they be implemented in real-time on a monthly basis there. Part of this software consolidation may require rewriting some codes in a different programming language, such as converting Matlab scripts to Fortran code.

The CPC component of the proposal Probabilistic forecast products for the NMME seasonal forecast system Emily Becker and Huug van den Dool also Li- Chuan Chen

THIS IS WHAT WE DO IN REAL TIME

In the same way that traditional MSE can be minimized by a regression, we here attempt to minimize the BS, i.e. the MSE for probability forecasts. The meaning/interpretation of traditional anomaly correlation (AC) By extension the meaning/interpretation of the probability anomaly correlation (PAC)

Unanswered question: To what extent should we be ruled by verification metrics?

CFSv2 JanIC SST forecasts for February, Northern Hemisphere Hindcasts 1982-2010 BOTTOM LINE CONCLUSIONS . The PAC trick works : lower BS = higher accuracy .It cleans up – ve skill in N class, embarrassment avoided. .It tones down too bold forecasts, particularly in A&B class .The gain is appreciable. Brier Score A N Bunadj0.1220.1540.132Adj+corr[0,100]0.1070.1330.114 Brier Skill Sc. A N B unadj 0.146 -0.068 0.115 Adj+corr [0,100] 0.257 0.074 0.232

BS = Reliability minus Resolution plus Uncertainty

CFSv2 JanIC SST forecasts for February, Northern Hemisphere Brier Score A N B unadj 0.122 0.154 0.132 Adj+corr [0,100] 0.107 0.133 0.114 Brier Skill Sc. A N B unadj 0.146 -0.068 0.115 Adj+corr[0,100]0.2570.0740.232 BS = Reliability minus Resolution plus Uncertainty

Brier Score A N B unadj 0.122 0.154 0.132 adj 0.107 0.135 0.116 Brier Skill Sc. A N B unadj 0.146 -0.068 0.115 adj 0.251 0.065 0.221 CFSv2 JanIC SST forecasts for February, Northern Hemisphere BS BS = Reliability minus Resolution plus Uncertainty Brier Skill Sc. A N Bunadj0.146-0.0680.115Adj+corr[0,100]0.2570.0740.232 Brier Score A N B unadj 0.122 0.154 0.132 Adj+corr [0,100] 0.107 0.133 0.114

In conclusion (CPC part) Probability Anomaly Correlation approach yields a lower BS (as expected) for any model and even for NMME collection. This can be implemented! PAC approach is being at CPC compared to ensemble regression, to have some reference. PAC approach has a large impact on the reliability-resolution diagram. Both reliability and resolution improve. PAC has an interesting application in verification of ENSO composites (Li- Chuan Chen) An adjustment to our most beloved conclusion

.How much improvement to be expected by (any) calibration? . Correctibility ? Inherent skill>=threshold .Individual Models may improve more t han the NMME collection. Managing expectations

Quad Chart to Track Progress/Issues of MAPP-CTB Demonstration Project in FY16 Improved probabilistic forecast products for the NMME seasonal forecast system Issues/Risks Issues : **. Mitigation : Finances Scheduling Project Information and Highlights Funding Sources : FY14: $271K FY15: $91K Status : Management Attention Required Potential Management Attention Needed On Target Updated on 11/06/2015 Lead : Columbia University: Anthony Barnston , Michael Tippett, Shuhua Li NCEP/CPC: Huug van den Dool , Emily Becker Scope : Improve NMME-based probabilistic forecast products Estimated Benefits : Improved NMME probabilistic forecasts by addressing systematic biases in both forecast anomalies and categorical probabilities. Software will be deployed at CPC Milestone Date Status Corrections in pattern placement and pattern characteristics FY16Q1 Refinements to local probability anomalies (developed at CPC). FY16Q3 Post-project review FY16Q4 G Y R G G G Y

(A. Barnston, M. Tippett , S. Li at IRI) Use of multivariate statistical methods to correct systematic errors in placement of forecast patterns of each model. Calibration steps: Spatial corrections by IRI may precede local probability adjustments by CPC. Uncorrected model forecast Corrected model forecast Example: Precipitation forecast during El Nino, DJF, from one atmospheric model

Disruption of progress at IRI in 2015 During year 1 of this CTB project, Mike Tippett took a 75%-time teaching position at Columbia University , leaving only summers to do research. So much of his funding was transferred to Shuhua Li. Mike has been training Shuhua to use Matlab in the intended context of this project. Learninghas been steady but slow. In the last month, Shuhua Li gave notice to leave IRI, taking an operationalf orecasting job at BoM in Australia. So his training in this project will not b ear any fruit. Due to the above delay and then unexpected loss of Shuhua Li, IRI presentlyhas no concrete results to show regarding model spatial corrections.Possible fixes for these setbacks: (1) Barnston will do all of the hands-onwork using CPT instead of Matlab (a quicker, but possibly less permanent, software package); (2) We will request a 1-year no-cost extension of the project, and find a new person to do the hands-on model correction trials.