Preliminary Skill Comparisons Among IRI-Multi-Model Ensemble- CDC-CCA- Hindcasts

Outline • US-average skill of categorical and probabilistic seasonal forecasts of US surface temperature is examined. • Hindcasts of IRI multi-model ensemble (MM) from 19982003 and CDC-CCA (CCA) from 1995 to 2003 are compared with CPC-Official forecasts (OFF) from 1995 to 2003. • This comparison serves as a sanity check on the verification system and points out advantages and disadvantages of the several techniques. • Among the recommendations are: *CPC forecasts may be too conservative (encouraged by the skill metric we use) in areal coverage and in probabilities, *MM and CCA probabilities appear too large, *MM and CCA categorical forecasts may be better than 1/3 in cl regions, *MM and CCA are sufficiently skillful to serve as a first-guess forecast for CPC’s seasonal forecast system.

Cautionary Notes • CPC (OFF) forecasts are the original operationally- issued forecasts, while those from IRI (MM) and CDC (CCA) are hindcasts. • We do not know for certain whether these hindcasts would be as good as they are if they were issued operationally. • MM occasionally predicts CL (EC), CCA does not, OFF uses CL alot. • MM and CCA predict the near normal category, OFF almost never does. • The sample size of these datasets is small and results are likely not statistically significant. • CL and EC are used interchangeably.

Model Descriptions • MM – IRI’s fully-automated multi-model ensemble of GCM forecasts. Data begins in February, 1998. Hindcasts. • CCA – CDC’s fully-automated technique which uses CCA to predict what a combination of several models would predict as a function of sst. Forced by presisted ssts. Data begins January, 1995. Hindcasts. • OFF – CPC’s subjective official forecasts. Based on statistical and dynamical forecast tools, including MM and CCA, along with Constructed Analogs, individual model forecasts, Trend, Soil Moisture and ENSO-neutral composites. Data from Jan 1995. Operational Forecasts. • These are available at: http: //www. cpc. ncep. noaa. gov/products/predictions/90 day/tools/briefing/index. pri. html

Skill Scores • Non-EC stations: s=((c-e)/(t-e))*100 • All Stations : s=((c+(1/3)*cl-e)/(t-e))*100 c = # stations correct e = # stations correct by chance t = # stations in total cl= # stations predicted equal chances • rpss = 1 - rps/rpsclimate M m m M=1 u=1 • rps = (1/(M-1)) 3[(3 pu) – (3 ou)] 2

TEMP A sample forecast: JFM 1999 CCA MM OFF OBS PREC

CCA cl MM cl 95 96 97 98 99 00 01 02 03 MM No-cl 95 96 97 98 99 00 01 02 03 CCA No-cl 95 96 97 98 99 00 01 02 03

0. 06 vs 0. 13 OFF (+), better 13 MM (+), better 17 OFF (+) MM (-) 6 6 6 MM (+) OFF (-) 16

0. 12 vs 0. 12 OFF (+), better 22 MM (+), better 8 OFF (+) MM(-) 7 3 10 MM (+) OFF(-) 15

0. 08 vs 0. 12 OFF (+), better 18 OFF (+) CCA (-) 16 CCA (+), better 28 15 3 CCA (+) OFF (-) 20

0. 17 vs 0. 12 OFF (+), better 33 CCA (+), better 14 OFF (+) CCA(-) 16 7 CCA (+) OFF(-) 20 10

RPSS=0. 02 CCA OFF RPSS=0. 00 MM RPSS=0. 00 CCA and MM probabilistic forecasts appear too liberal, OFF too conservative. RPSS of all have declined over the last 2 years

Conclusions, Recommendations • CPC uses non-Cl T skill as its metric. This encourages forecasting small areas. This is bad only if generally increased area is justified. We did better before 1997. Is a better set of skill masks needed? • Purely numerical MM does as well in non-Cl T forecasts as combined statistical/numerical CPC. • CPC does better than CCA in non-Cl T forecasts. • Both CCA and MM do better than CPC in Cl T forecasts. This appears to indicate CCA and MM score better than 1/3 in those areas. • CCA Cl T forecasts have high skill when CPC skill is low. • Should areas of EC should be reduced in CPC forecasts (CPC too conservative)? • The decline of skill in all the tools over the last 2 years indicates reduced predictability. • All three probabilistic forecast systems can be improved by calibration. • Differences among forecast skills likely not statistically significant. • MM and CCA should be combined and used by CPC as a first guess.

Скачать презентацию Preliminary Skill Comparisons Among IRI-Multi-Model Ensemble- CDC-CCA- Hindcasts

0820f003072ab1e143ab0bf9b3810ef9.ppt

Количество слайдов: 19