Evaluation and Comparison of Multiple Convection-Allowing E - PowerPoint Presentation

Slide1

Evaluation and Comparison of Multiple Convection-Allowing Ensembles Examined in Recent HWT Spring Forecasting Experiments

Israel Jirak, Steve Weiss, and Chris Melick Storm Prediction Center

WoF Workshop, April 3, 2014Slide2

Convection-allowing ensembles (

~4-km grid spacing) can provide important information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. on the outlook to watch scaleThese ensembles will play an important role in the ability of SPC to provide a more continuous flow of probabilistic hazard information in support of WoF

2 March 2012

29 June 2012

24-h neighborhood

prob

UH ≥25 m

2

/s

2

24-h ensemble max 10-m Wind Speed

Convection-Allowing Ensembles

OverviewSlide3

An experimental real-time Storm-Scale Ensemble Forecast (SSEF) system has been produced for the NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE) by OU CAPS since 2007 through CSTAR funding

Comprised of 4-km convection-allowing WRF and ARPS members:2007: 10 members; 2008: 10 members; 2009: 20 members; 2010: 26 members (full CONUS to 30 h); 2011: 50 members (full CONUS to 36 h); 2012: 28 members; 2013: 27 members at 00Z (to 48 hours) and 8 members at 12Z (to 18 hours)Primarily examine explicit storm attributes, especially hourly maximum fields (HMFs): Updraft Helicity, Updraft Speed, 10-m AGL Wind Speed, 1-km AGL Reflectivity

Ensemble display approaches include spaghetti plots, ensemble max, and neighborhood probabilities (more on next slide)SPC began processing deterministic high-resolution runs from EMC and NSSL as the Storm-Scale Ensemble of Opportunity (SSEO) in 2011

The 4-km AFWA ensemble was made available to SPC in 2012

Convection-Allowing Ensembles

HistorySlide4

Traditional ensemble probabilities of HMFs from high-resolution models are not especially useful, owing to poor agreement among members at the grid point of these fields.

Applying a binary neighborhood approach to a storm-scale ensemble improves the statistical results of HMFs in forecasting severe weatherROI=20-40 kmSigma=30 grid points

Same approach can also be applied to observations (e.g., radar reflectivity) for verification purposes

03 May 2008 (

Harless

2010)

Convection-Allowing Ensembles

Neighborhood ProbabilitiesSlide5

Convection-Allowing Ensembles

Neighborhood Probabilities

HM Updraft Helicity > 25 m

2

s

-2

SSEO 24-hr

fcst

valid 00Z 28 April Slide6

Convection-Allowing Ensembles

Neighborhood Probabilities

Grid-Point Probability

HM Updraft Helicity >25 m

2

s

-2

SSEO 24-hr fcst valid 00Z 28 April Slide7

Convection-Allowing Ensembles

Neighborhood Probabilities

40-km Neighborhood Probability HM Updraft Helicity >25 m

2

s

-2

SSEO 24-hr

fcst valid 00Z 28 April Slide8

Convection-Allowing Ensembles

Neighborhood Probabilities

40-km Neighborhood Smoothed

Prob

HM Updraft Helicity >25 m

2

s

-2

SSEO 24-hr fcst valid 00Z 28 April Slide9

OU/CAPS

Storm-Scale Ensemble Forecast (SSEF) System

Since 2007; 36-hr forecasts from 00z; 12z runs began in

2013

Primarily WRF-ARW; 4-km grid spacing; forecasts

to

60hrs in 2014 Multi-physics, multi-initial conditions: applies SREF perturbations to NAM

ICsAdvanced physics, 3DVAR & radar data assimilation; available for HWT/SFE SPC Storm-Scale Ensemble of Opportunity (SSEO)Since 2011; 36-hr forecasts at 00z & 12z; 7 members (2 time-lagged)

Multi-model (ARW, WRF-NMM & NMM-B), multi-physics; ~4-km grid spacingUses available deterministic models at SPC to process as an ensembleBasic data assimilation through NDAS; available year-round in SPC Air Force Weather Agency (AFWA) Ensemble

Since 2012; 60-hr forecasts at 00z & 12z; 10 members; 4-km grid spacingSingle model (WRF-ARW), multi-physics, multi-initial conditionsCold start from downscaled global model forecasts (GFS, UM, CMC)No data assimilation; available year-round in SPC

Convection-Allowing EnsemblesSystem ComparisonSlide10

The Fractions Skill Score (FSS) was calculated for neighborhood probability (ROI=40 km; σ=40 km) of updraft helicity ≥ 25 m

2s-2 for the SSEO/SSEF versus practically perfect hindcasts of preliminary severe weather reports (ROI=40 km; σ=120 km) during SE2011The SSEO had higher fractions skill score (FSS) for neighborhood probabilities of UH ≥25 m

2/s2 during SFE2011 than the SSEF

The number of members included in the SSEF did not seem to have a strong impact on the statistical results for neighborhood probabilities of UH during SE2011 when verified against severe weather reports

Convection-Allowing Ensembles

SFE2011 Results

Nprob

UH ≥25 m2/s2

SSEO

SSEF – 24

mem

FSS = 0.84

FSS = 0.68

3-hr [NPRS]:UH ≥25

m

2

s

-2

valid 06Z on 02 June

2011

w/ verifying reports and practically perfect

hindcast

FSS 3-h periods SFE2011Slide11

Even for 6-h QPF, the SSEO received the highest subjective ratings relative to other operational and experimental models and ensembles during SFE2011

Statistically, the probabilistic QPF forecasts (>0.5”) from the SSEO were typically as good as (if not better than) the SSEF during SFE2011 at various lead times

Convection-Allowing Ensembles

SFE2011 Results

from

Tara

Jensen, DTC

from Dave Novak,

WPC

SSEO favored over CAPS ensemble Slide12

During SFE2012, the SSEO outperformed the SSEF and AFWA in terms of FSS for neighborhood probabilities of reflectivity ≥40

dBZ (bug later found in SSEF)Subjective ratings by the SFE2012 participants of HMF ensemble forecasts tended to favor the SSEO forecasts of UH over the SSEF and AFWA forecasts

Convection-Allowing Ensembles

SFE2012 Results

Hourly FSS

Nprob

Refl

≥40 dBZ 3-hr ensemble forecast ratings (max, nprob) of UHSlide13

The quality of the AFWA forecasts was less consistent than the SSEO forecasts

Some UH forecasts from the AFWA ensemble were very good (bottom left) while others were poor (bottom right)

Convection-Allowing Ensembles

SFE2012 ResultsSlide14

The impact of radar data assimilation in the CAPS SSEF was evident in the first 4 hours of the 12 UTC-initialized forecast.

Otherwise, there was little statistical difference in the FSS among the 00 and 12 UTC SSEO and SSEF.Subjective ratings of 00Z ensemble HMFs were again favorable for the SSEO during SFE2013

Convection-Allowing Ensembles

SFE2013 Results

Hourly FSS

Nprob

Refl ≥40 dBZ

3-hr ensemble HMF forecast ratings (max, nprob)Slide15

Why is a “poor man’s ensemble” (i.e., SSEO) performing as well as formally designed ensembles? Let’s consider some aspects of configuration for convection-allowing ensembles

Single model vs. multi-modelNumber of membersInitial conditions and IC/LBC perturbationsPhysics

Convection-Allowing Ensembles

ConfigurationSlide16

Even with the same initial conditions, clustering often occurs by model core

Generally more confident in a solution if different model cores are in agreementIs a multi-model approach a good way to address uncertainty in a convection-allowing ensemble?

Convection-Allowing Ensembles

Configuration: Single model vs. multi-model

WRF-NMM

WRF-ARW

SSEO

21Z on 16 April 2011

3-hr spaghetti plot of UH

≥25

m

2

s

2Slide17

Is the success of the SSEO a fortuitous balance of members with an

underforecast bias and those with an overforecast bias (Row and Correia, 2014 AMS); not necessarily a result of using multiple model cores?Neighborhood verification of radar reflectivity reveals members with lower biases (e.g., NAM Nest) versus those with higher biases (e.g., NSSL-WRF)

Biases will change with upcoming HiResW upgrade, so we may learn more this spring

Convection-Allowing Ensembles

Configuration: Single model vs. multi-modelSlide18

For the way convection-allowing ensembles are currently configured, there does not appear to be a huge benefit to running more than ~10 members

Clark et al. (2011) objectively identified the “point of diminishing returns” at 9 members for 6-hr QPF at f30 and 2-km scale

Convection-Allowing Ensembles

Configuration: Number of members

Could run additional members to more effectively sample the forecast PDF, but is it worth the additional computational cost? Use a larger neighborhood?

from Clark et al. (2011) Slide19

Currently, all members of the SSEO are initialized from the NAM (including two time-lagged members), so diversity primarily arises from multi-model/physics

AFWA approach (single model, 3 different IC/LBCs) often leads to higher, overconfident probabilities SSEF approach not an obvious improvement over single, unperturbed IC (i.e. SSEO), suggesting ICs not properly perturbed at this scale

Convection-Allowing Ensembles

Configuration: Initial conditions and IC/LBC perturbations

AFWA

OBS

high probs

nothing observedSlide20

In four test runs for May 2013, Kong et al. (2014) found larger domain-average ensemble spread for multiple fields by directly using LBCs from SREF members rather than extracting perturbations and applying to the NAM (current strategy)

NSSL-WRF ensemble this spring will directly utilize IC/LBCs from selected SREF members

Convection-Allowing Ensembles

Configuration: Initial conditions and IC/LBC perturbations

Ensemble Spread

from Kong et al. (2014) Slide21

Though spread in an ensemble with physics-only diversity is less than that from an ensemble that also includes IC/LBC perturbations, the contribution to spread from physics diversity can be large, including for instability fields (Clark et al. 2010)

Convection-Allowing Ensembles

Configuration: Physics diversity

from Clark et al. (2010) Slide22

Convection-allowing ensembles (~4-km grid spacing) can provide useful information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. in generating outlooks on Day 1; setting the stage for the continuous flow of probabilistic hazard information down

toward the warning scaleThe SPC SSEO has proven to be as useful/skillful as formally designed convection-allowing ensembles, which raises questions about effective/proper configuration of these types of systemsNSSL is running eight 00Z members this spring with only IC/LBC diversity directly from 21Z SREF members

CAPS is planning to run an experimental 4-km EnKF SSEF system this year in near real-time for comparison with traditional SSEF forecasts

Convection-Allowing Ensembles

Summary