Tom Hopson Luca Delle Monache Yubao Liu Gregory Roux Wanli Wu Will Cheng Jason Knievel Sue Haupt Army Test and Evaluation Command Dugway Proving Ground Dugway Proving Grounds Utah eg T Thresholds ID: 591118
Download Presentation The PPT/PDF document "Comparing and Contrasting Post-processin..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts
Tom Hopson
Luca
Delle
Monache
,
Yubao
Liu, Gregory Roux,
Wanli
Wu, Will Cheng, Jason
Knievel
, Sue
HauptSlide2
Army
Test and Evaluation Command:
Dugway
Proving GroundSlide3
Dugway
Proving Grounds, Utah e.g. T Thresholds
Includes random and systematic differences between members.
Not an actual chance of exceedance unless calibrated.Slide4
Xcel Energy Service Areas
Wind Farms (50+)
~3200 MW
Northern States Power (NSP)
Public Service of Colorado (PSCO)Southwestern Public Service (SPS)3.4 million customers (electric)Annual revenue $11B
Copyright 2010 University Corporation for Atmospheric ResearchSlide5
WRF RTFDDA Model Domains
Ensemble System (30 members)
D1 = 30 km
D2 = 10 km
0-48
hrs
0-48
hrs
Real Time Four Dimensional Data Assimilation (RTFDDA)
41 vertical levels
Vary:
Multi-models
Lateral B.Cs.
Model Physics
External forcing
Yubao
Liu --
yliu@ucar.edu
for further questionsSlide6
Goals of an EPS
Predict the observed distribution of events and atmospheric states
Predict uncertainty in the day’s prediction
Predict the extreme events that are possible on a particular day
Provide a range of possible scenarios for a particular forecastSlide7
Outline
Brief overview of
: 1)
quantile
regression (QR), 2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog
Kalman
filter” (ANKF)
2
nd
moment calibration via rank histograms
Skill score comparisons and improvements with increased
hindcast
data
Example of blending approaches
ConclusionsSlide8
Example of Quantile Regression (QR)
Our application
Fitting T quantiles using QR conditioned on:
Ranked forecast ens
ensemble mean
ensemble median
4) ensemble stdev
5) Persistence
Hopson and Hacker 2012Slide9
Logistic Regression for probability of
exceedance
(
climatological thresholds)Slide10
Probability/°K
Temperature [K]
Probability/°K
Temperature [K]
Forecast
PDF
climatological
PDF
Step I: determine
climatological
quantiles
Probability
Temperature [K]
Step 3: use conditioned CDF to
interpolate desired
quantiles
1.0
0.5
.75
.25
prior
posterior
Step 2: calculate conditional
probs
for each
climat
quan
Probability
Temperature [K]
0.5
.75
.25
1.0
Final result: “sharper” posterior PDF
represented by interpolated
quans
Hopson and Hacker 2012Slide11
T [K]
Time
forecasts
observed
Regressor set
:
1. reforecast ens
2. ens mean
3. ens stdev
4. persistence
5. LR quantile
(not shown)
Probability/°K
Temperature [K]
climatological
PDF
Step I: Determine
climatological quantiles
Step 2: For each quan
, use forward step-wise
cross-validation to select best regress set
Selection requires: a) min QR cost function,
b) binomial distrib at 95% confidence
If requirements not met, retain climatological “prior”
1.
3.
2.
4.
Step 3:
segregate forecasts based on ens dispersion; refit models (Step 2) for each range
Time
forecasts
T [K]
I.
II.
III.
II.
I.
Probability/°K
Temperature [K]
Forecast
PDF
prior
posterior
Final result: “sharper” posterior PDF
represented by interpolated quans
Hopson and Hacker 2012Slide12
National Security Applications Program Research Applications Laboratory
Significant calibration regressors
3hr Lead-time
42hr Lead-time
Station DPG S01Slide13
National Security Applications Program Research Applications Laboratory
RMSE of ensemble members
3hr Lead-time
42hr Lead-time
Station DPG S01Slide14
Time
t = 0
day
-1
day
-2
day
-6
day
-5
day
-4
day
-3
day
-7
OBS
PRED
KF-weight
KF
Delle
Monache
et al. 2010
Analog
Kalman
Filter (ANKF)
Deterministic method applied to each individual ensemble
KF weighting run in analog spaceSlide15
Time
t = 0
day
-1
day
-2
day
-6
day
-5
day
-4
day
-3
day
-7
OBS
PRED
KF-weight
KF
ANKF
AN
“Analog” Space
day
-4
day
-7
day
-5
day
-3
day
-2
day
-1
day
-6
PRED
OBS
farthest analog
closest analog
NOTE
This procedure is applied independently at each observation location and for a given forecast time
Delle
Monache
et al. 2010Slide16
Outline
Brief overview of
: 1)
quantile
regression (QR), 2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog
Kalman
filter” (ANKF)
2
nd
moment calibration via rank histograms
Skill score comparisons and improvements with increased
hindcast
data
Example of blending approaches
ConclusionsSlide17
42-hr dewpoint time series
Before Calibration
After
Calibration (QR)
Station DPG S01Slide18
Original ensemble
QR
LR
ANKF
Rank Histograms
15hr lead wind forecastsSlide19
Skill measures
u
sed
:
Rank histogram (converted to scalar measure)
Root Mean square error (RMSE)
Rank
Probability Score (RPS)
Relative Operating Characteristic (ROC)
curve
Skill Scores
Comparing to original ensemble forecast, but with bias removed => “reference forecast”Slide20
Blue - QR
Red - ANKF
Green - LR
Skill Score Comparison
For wind farm
CEDC, 3hr lead forecasts
Reference forecast: original wind speed ensemble
w
/ bias removed
Data size:
900ptsSlide21
Rank Histogram scalar
QR
QR
LR
LR
RMSE
Skill
Scores Dependence on
T
raining Data Size
Upper dashed – 900pts
Solid – 600pts
Lower dashed – 300pts
Reference Forecast:
Original wind speed ensemble
w
/ bias removedSlide22
ROC
QR
QR
LR
LR
RPSS
Skill
Scores Dependence on
T
raining Data Size (cont)
Reference Forecast:
Original wind speed ensemble
w
/ bias removed
Upper dashed – 900pts
Solid – 600pts
Lower dashed – 300ptsSlide23
RPS
ROC
RMSE
Brier Score
Wind farm TWBT
1
1
1
1
12
12
12
12
24
24
24
24
36
36
36
48
48
48
48
36
ANKF
QR
QR + ANKFSlide24
original
ANKF
QR
QR + ANKF
TWBT
6-hSlide25
Summary
“step-wise cross-validation”-based post-processing framework provides a method to ensure forecast skill no worse than
climatological
and persistence
Also provides an umbrella to blend together multiple post-processing approaches as well as multiple
regressors
, and to diagnose their utility for a variety of cost functions
Quantile
regression and logistic regression useful tools for improving 2
nd
moment of ensemble distributions
See significant skill gains with increasing “
hindcast
data” amount for a variety of skill measures
Blending of post-processing approaches can also further enhance final forecast skill (e.g. ANKF and QR) by capturing “best of both worlds”
Further questions:
hopson@ucar.edu
or
yliu@ucar.eduSlide26