Verification of forecasts of Cbs Bob Lunnon Aviation

Verification of forecasts of Cbs Bob Lunnon, Aviation Outcomes Manager, Met Office WAFS Science meeting, Washington, April 21 st 2009 © Crown copyright Met Office

Acknowledgement Most of this work has been done by Andrew Mirza (some help from Lauren Reid) Thanks also due for use of lightning data (Sferics), and for availability of US products © Crown copyright Met Office

Summary of presentation • Scientific background on product • Background on lightning detection system (Sferics) used to perform verification • Results of verification • Conclusions/Recommendations • Work still to be done © Crown copyright Met Office

FLYSAFE Thunderstorm (Cb) hazard object Target Weather Object "Cb" Cb top volumes: convective turbulence, lightning Cb bottom volumes: hail, icing, lightning, heavy rain, wind shear, turbulence © Crown copyright Met Office

Prediction of Cbs by Numerical Weather Prediction Models – convective parameterisation schemes I • If a NWP model is run with no convective parameterisation scheme, convective instability can occur at individual grid boxes giving rise to “grid-box storms” which are unrealistic and can cause the model to fail due to computational instability • Convective parameterisation schemes redistribute energy (in various forms) and moisture in the vertical so that • Model is stable • Vertical distribution of these variables approximately reflects how they might be distributed in the real atmosphere after one or more Cb have occurred. © Crown copyright Met Office

Prediction of Cbs by Numerical Weather Prediction Models – convective parameterisation schemes II • Convective parameterisation schemes diagnose many atmospheric variables – indeed some diagnose turbulent kinetic energy (TKE), which might appear to be an appropriate predictor of severity of hazard • However, TKE is an internal diagnostic and it is unlikely that two schemes will generate consistent diagnoses • At the February 2007 WAFS coordination meeting in Boulder significant differences between Cb products were noted • In contrast, convective rainfall rate is used (a) to predict expected rainfall, and (b) to diagnose latent heating, so there is likely to be consistency between different models • After the February 2007 meeting, it was agreed to base prediction of Cb severity on a non-linear function of convective rainfall rate © Crown copyright Met Office

Prediction of Cbs by Numerical Weather Prediction Models – convective parameterisation schemes III • Convective parameterisation schemes also diagnose base and top of convection • Hong Kong Observatory highlighted difference in cloud top diagnosed by the two WAFCs in July 2008 • The UK convective parameterisation scheme was adjusted in late 2008 to correct this difference • More work needed in this area © Crown copyright Met Office

Met Office Sferics system • System detects the vertical component of electromagnetic radiation generated by a lightning discharge at around 13 k. Hz (VLF) received at a number of `sensor’ sites • Strong EM emissions around 13 k. Hz • Caused by rapid neutralisation of charge, in the lowest few hundred metres of C-G stroke • Atmospheric attenuation minima close to 13 k. Hz • The atmo. SFERIC wave can propagate over thousands of kilometres along the earth / ionosphere waveguide • 12 sensors in the current network connected to Exeter by communication links • Received digitised waveforms sent to the Flash Location Processor (FLP) via a Data Collection System (DCS) located in a `fire-walled’ DMZ • Location of a `flash’ or `fix’ determined by the Arrival Time Differences (hence ATD) • Differences between at least 4 sensor out-stations are used to calculate lightning position by iterative method © Crown copyright Met Office

Total number of strikes recorded in July 2007 © Crown copyright Met Office

Locally observed lightning strikes in S America, case in August 06 © Crown copyright Met Office

Data from UK sferics system for same day © Crown copyright Met Office

Severe US storms – 13 th March 2006 © Crown copyright Met Office

Sferics – area of reliable data • between 40 S and 80 N and between 110 W and 80 E • Verification data generated in latitude bands (and crude longitude divisions) • South of 20 s • 20 South to 20 North • 20 North to 50 North • North of 50 North © Crown copyright Met Office

Verification against Sferics method • For UK “product”, we verified against convective rainfall rate [current disseminated product is “smeared” to make it look better but we plan to remove the smearing] • We create truth data set on regular global latitude/longitude grid (640 * 480) • Forecast data were up-scaled onto this grid where necessary © Crown copyright Met Office

Results • ROC curves for automated and manual products, two WAFCs, November 08 and January 09 © Crown copyright Met Office

Southern Hemisphere extratropics, November 08 © Crown copyright Met Office

Tropics, November 08 © Crown copyright Met Office

Northern Hemisphere midlatitudes, November 08 © Crown copyright Met Office

Northern Hemisphere, polar area, November 08 © Crown copyright Met Office

All latitudes, November 08 © Crown copyright Met Office

Southern Hemisphere extratropics, January 09 © Crown copyright Met Office

Northern Hemisphere midlatitudes, January 09 © Crown copyright Met Office

Northern Hemisphere, polar regions, January 09 © Crown copyright Met Office

All latitudes, January 09 © Crown copyright Met Office

Rankings by area and season NO VE MB ER J AN UA RY UK man UK auto US man US auto 2 4 1 3 3 4 1 2 NH 1 polar 2 4 3 3 1 4 2 NH 3 midlat Trop- 3 ics 1 2 4 3 1 4 2 4 1 2 3 4 1 2 SH 3 extra Met Office 2 1 4 2 3 4 1 All lats © Crown copyright

Automated results – Eastern Hemisphere (Jan 09) © Crown copyright Met Office

Automated results – Western Hemisphere (Jan 09) © Crown copyright Met Office

Comments on rankings by area and season • Overall “winner” appears to be US manual product (ranked first for combined latitudes for both Nov 08 and Jan 09) • However this ranking reflects the number of grid -points in each latitude zone • Number of flights per year is a function of latitude, strongly peaking in 20 -50 north • UK Automated product is ranked first in this zone in both Nov 08 and Jan 09 • The conclusion is “there is no conclusion” © Crown copyright Met Office

Further work • Look at diurnal variations of forecasts (manual and automatic), particularly in tropics. Histograms of time development, both forecast and measured, would be good • Look at forecast accuracy as a function of data time – 00 Z and 12 Z forecasts may be better than 06 Z and 18 Z forecasts • US auto product is better than UK auto product in tropics – we need to understand why • UK auto product is better than US auto product in NH mid latitudes – we need to understand why • Need to look at cloud base and cloud top – former using METARs, latter using satellite data (CLOUDSAT or METEOSAT) © Crown copyright Met Office