TIGGE Data Archive at NCAR 8 th GIFS-TIGGE

TIGGE Data Archive at NCAR 8 th GIFS-TIGGE Working Group World Meteorological Organization Geneva 22 -24 February, 2010 Doug Schuster Steven Worley Dave Stepaniak Others

Archive Summary l Online Data Ø Period, most recent two weeks Ø ~ 5 TB , public products Ø ~ 1 TB, data preparation, subsetting Ø ~ 65 GB Tropical Cyclone Track Data (CXML) l Offline Data Ø Full period of record Ø ~ 300 TB, NCAR MSS system

Provider Summary l Archive Status l Service Improvements l Special Projects l Metrics l Future Service Additions

Data Delivery (02/2010) • Data exchanges using UNIDATA LDM, HTTP and FTP • 440 GB/day (ECMWF = 332 GB/day) • 1. 7+ million 2 -D gridded fields/day

Archive Inhomogeneities

Parameter vs Provider Matrix Variable Lv. L Geopotential Z PL Specific H PL T PL U-velocity PL V-velocity PL Potential Vor PT Potential T PV U-velocity PV V-Velocity PV U 10 m SL V 10 m SL CAPE SL Conv. Inhib. SL Land-sea SL Mean SLP SL Orog. SL Skin T SL Snow D. H 20 SL Snow F. H 20 ECWF UKMO JMA NCEP CMA CMC BOM Met. F KMA SL PL = Pressure Level, PT = 320 K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level CPTC

Parameter vs Provider Matrix Variable Lv. L Soil Moist. SL Soil T SL Sunshine D. SL Surf. DPT SL Surf. ATmax SL Surf. ATmin SL Surf. AT SL Surf. P SL LW Rad. Out SL LH flux SL Net Rad SL Net Therm. Rad SL Sensible Rad. SL Cloud Cov SL Column Water SL Precipitation SL Wilt. Point SL Field Cap. ECWF UKMO JMA NCEP CMA CMC BOM Met. F KMA SL PL = Pressure Level, PT = 320 K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level CPTC

2009 Service Improvements Ø Improved Web Portal Performance Ø User data selection interface runs smoother. Ø Automated capability to stage offline data for download. Ø Added “Client for URLs” (c. URL) based parameter subsetting capability. Ø Directly extract selected 2 D grids from files across multiple data providers. Ø Downloads managed from user’s workstation. Ø Script templates are provided to drive “subscription style” parameter download capability. Ø Strengthened data ingest processing. Ø System monitoring (disk capacity, LDM status, etc) Ø Failover mechanisms for LDM downtime. Ø Data integrity checking.

2009/2010 Supported Special Projects Ø NCAR Research Applications Laboratory (RAL) Google African Meningitis Forecast Initiative Ø Dr. Thomas Hopson Ø Integrate weather and health data to manage African meningitis outbreaks (in Ghana as starting point). Ø Looking for 2 -week lead time to position vaccines. Ø Processing ECMWF data from present backwards through the archive for model calibration. Ø Total Precipitation, 2 m Temperature, 2 m Dew Point. Ø Global Subarea 40 deg N to 40 Deg S. Ø 0. 5 x 0. 5 degree resolution. Ø Assist RAL and Unidata in providing “Real-time” access to TIGGE based Multi-Model Ensemble Forecast data. Ø http: //www. ral. ucar. edu/projects/ghana/ Ø http: //www. google. org/predict. html

NCAR TIGGE Full Model Data Archive Usage TB NCAR TIGGE Data Access Metrics 16 14 12 10 8 6 4 2 0 2006 c. URL impact 2007 2008 TB Accessed TB Downloaded 2009 Year YEAR GB Accessed GB Downloaded Files Downloaded Active Users / Unique Users 2006 5 5 10 0/5 2007 900 32, 000 10 / 50 2008 1, 300 1, 100 40, 000 10 / 20 2009 13, 700 1, 900 68, 000 10 / 30 2010 80 20 500 3/5 Total 15, 985 3, 925 140, 510 33 / 80

NCAR TIGGE CXML Archive Usage YEAR GB Downloaded Files Downloaded # Active Users / # Unique Users 2008 1 860 7 / 30 2009 26 12, 300 17 / 35 2010 0. 5 500 3/6 Total 27. 5 13, 660 27 / 81

Planned Service Improvements Ø Additional Online Storage Ø 45 TB data online (3 months of most recent data). Ø 15 TB for data processing request output. Ø Interface driven data processing and immediate download capability extended from 2 weeks to 3 months. Ø Improved “Offline” Storage. Ø Data being transitioned from current NCAR MSS system to new HPSS system. Ø Improved access speeds to offline data. Ø Add data processing capability to offline data? Ø Expanded/Updated Computing. Ø Improved LDM ingest reliability. Ø Faster “turn-around” time on user sub-set requests. Ø Potential for subscription services?

Potential Service Improvements Ø Model Validation Data Portal (Pending Funding)? Ø Wide variety of observational and analyzed data sources. Ø In Situ Observation data (NCEP operationally collected data). Ø Ocean based observations (ICOADS). Ø Global tropical cyclone position and intensity data. Ø Atmospheric Reanalysis (JRA, ERA, NNRR, CFSR). Ø Global PCP datasets. Ø Suggested TWG sets to support FDP’s. Ø Will need discussions to set priorities for development. Ø Data sub-set and output format selection capability. Ø Integrated With TIGGE model data portal Ø Data analysis tools and codes. Ø Model Evaluation Tools offered by the Developmental Testbed Center at NCAR Øhttp: //www. dtcenter. org/mrt/users

http: //tigge. ucar. edu/