DATA MANAGEMENT UPDATE Steve Williams Linda Cully and

DATA MANAGEMENT UPDATE Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7. 5) Meeting 28 October 2005

NAME Data Management Timeline 2003 2004 2005 Complete Operations Summary Operational (WWW) Data Staged to CODIAC Finalize Operations Plan Finalize Data Management Plan 8 9 10 11 12 1 2 Final Specs Planning from PIs Meeting 3 4 5 6 7 8 9 10 11 12 1 2 3 Data Analysis Workshop 4 5 6 Preliminary data staged Draft DM Plan OPS Planning Meeting Field Ops and Data Collection On-line Catalog Operational (CODIAC) 7 8 9 Final Data Staged QC’d data from PIs 10 11 12 1 Integrated Data Staged

http: //www. joss. ucar. edu/name/dm

NAME Data Flow • • • Special Research Data Ingest • Aircraft • Ships • Profiler • ISS • • • “Preliminary” Products Operational Data Ingest • GTS • Satellite • Radar • Model Delayed Operational NAME PI Data and “Preliminary” Data Sets Products NAME Online Field Catalog NAME Data Archive Center Catalog Information UCAR (JOSS) “Final” PI Research Data SMEX-04 Data Center (NSIDC) Distributed Data Centers • CICESE • IMTA • NASA/DAACs • NASA/GSFC • NASA/JPL • NASA/La. RC • NOAA/ETL • NOAA/NCDC • NOAA/NODC • NOAA/NSSL • NSIDC • SMN • Univ. of AZ • • •

EOL Data Archives Data Sources EOL Interactive Data System metadata quality control special processing Archive and Distribution Research Community

NAME DATA BASE STATUS (as of 26 October 2005) • 239 of 266 Data sets (90%) have been submitted or linked. • 11 Data sets are in progress • Field Catalog (w/reports and “browse” products) is available on-line • Precipitation (hourly and daily) and Upper Air “Composites” are under preparation • Data Access: http: //www. joss. ucar. edu/ihop/archive/

http: //www. joss. ucar. edu/name/namap 2/

www. joss. ucar. edu/name/catalog/ .

NAME FIELD CATALOG SAMPLE PRODUCTS

NAME 2004 Field Catalog Statistics http: //www. joss. ucar. edu/name/catalog/ • Reports/Summaries (Status, Mission, and Operations) 3, 785 documents/imagery (0. 27 GB) • Research Platform Products (Aircraft, Surface, Upper Air, Ship) 17, 606 Imagery files (1. 25 GB) • Operational Products (Satellite, Surface, Upper Air) 367, 092 Imagery files (17. 83 GB) • Model Output Imagery (Analysis and Forecast Fields) 99, 387 imagery files (6. 76 GB) • TOTALS: 487, 870 Files (26. 11 GB)

Composite Data Sets at NCAR/EOL A composite dataset is a collection (over some time period and region) of similar data (e. g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does NCAR/EOL develop composites? - Provides data in a uniformat with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

“Composite” Data Sets Being Developed by NCAR/EOL for NAME • Surface Meteorological – Datasets being converted • Precipitation - Hourly, and Daily (June-Sept 2005) • Soundings - Highest Res & 5 h. Pa (June – Sept 2005) • Complete Station List for all Composites

NCAR/EOL Atmospheric Sounding Processing Procedures Observing System Output Convert to JOSS Quality-control Format (ASCII) e. g. NWS Micro-ART Calculate Derived Parameters e. g. , u, v, dz/dt Gross Limit Checks Automated Checks QC Flags Error Files Vertical Consistency Checks Visual Examination QC Flags High-resolution and/or Interpolated Vertical Composite Inventory, Archive, and Develop Metadata Data and Metadata Available Online e. g. , 5 h. Pa

NAME TIER-1 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

NAME TIER-2 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

NAME TIER-3 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

NAME ARCHIVED SOUNDINGS BY SOURCE Belize - 140 soundings R/V Ulloa - 91 soundings R/V Altair - 154 soundings R/V El Puma - 32 tethersonde SMN - 2468 soundings ISS - 462 soundings Loreto GLASS - 176 soundings NWS - 16079 soundings Phoenix - 360 soundings San Jose - 324 soundings Yuma - 938 soundings pibal - 3235 pibals Total of 24, 459 soundings, pibals, & tethersondes to be included in the Final “composite” data set for Tier 3*. * Need NSSL Tezopaco and Pibal data to complete composite

CSU UPPER AIR COMPOSITE DATASET (V 2. 0) (1 July - 15 August 2004) Enhanced sites: Number Type of sites 10 USA-NWS enhanced 6 ISS, Loreto, 2 ships 7 SMN enhanced 1 Yuma (4 sites comb. ) 1 San Jose 25 subtotal enhanced Number of sondes 1380 815 1057 377 173 3802 Operational sites: 33 USA-NWS operational 8 SMN operational 1 Belize 42 subtotal 3030 402 75 3507 Estimate of number missing 17 10 30* 9 12 78 41 57** 17 115 67 Total 7309 193 * 14 missed due to unpaid electric bill at Empalme ** 11 missed due to broken generator at Isla Socorro percent missing 1. 2 2. 8 2. 3 6. 9 2. 1 1. 3 14. 2 22. 7 3. 3 2. 6

Embedded Accumulative Precipitation

“Preliminary” Tier-1 Hourly Precipitation (872 stations) ALERT (+ 405) Army Range (o 40) AZMET (o 27) ISS (* 3) MADIS (+ 84) METAR (o 18) NCDC Coop (* 58) NCEP (* 59) NERN/SMN/Sonora (+ 119) NMSU (+ 59)

“Preliminary” Tier-2 Hourly Precipitation (3913 stations) ALERT (+ 414) Army Range (o 40) AZMET (o 28) ISS (* 3) MADIS (+ 891) METAR (o 327) NCDC Coop (* 736) NCEP (* 1163) NERN/SMN/Sonora (+ 182) NMSU (+ 129)

“Preliminary” Tier-3 Hourly Precipitation (10328 stations) ALERT (+ 414) Army Range (o 43) AZMET (o 28) ISS (* 3) MADIS (+ 2855) METAR (o 1089) NCDC Coop (* 2328) NCEP (* 3249) NERN/SMN/Sonora (+ 190) NMSU (+ 129)

“Preliminary” Tier-1 Daily Precipitation (2050 stations) GASIR (* 195) Hourly Stations (+ 872) NCDC Daily Coop (+ 208) NCEP (* 557) Simple Raingage (o 218)

“Preliminary” Tier-2 Daily Precipitation (11955 stations) GASIR (* 1072) Hourly Stations (+ 3913) NCDC Daily Coop (+ 2544) NCEP (* 4208) Simple Raingage (o 218)

“Preliminary” Tier-3 Daily Precipitation (30232 stations) GASIR (* 1106) Hourly Stations (+ 10328) NCDC Daily Coop (+ 7326) NCEP (* 11254) Simple Raingage (o 218)

DATA ISSUES FOR DISCUSSION • Mexican Navy SEMAR Data approval letter received; Request to collaborate with data processing. • SMEX-04 Archives are being merged Priorities? • Are there any datasets not included in NAME Master List? • Are there other “composites” that are needed? • Should U. S. and Mexican Streamflow data be “composited”? Period of Record? • Need re-organization (and cleanup) of NAME web pages. • Need to establish “cutoff” date for datasets to be included in the “composites”. • Others?