The Quality Control of the Integrated Surface Hourly

The Quality Control of the Integrated Surface Hourly (ISH) Database • • Project Background Quality Control of the Data Lessons Learned Future Plans

Why Bother with the Project? ? • • • Multiple data sources difficult to use -- Differing formats -- Differing time conventions (LST, UTC) -- Differing station numbering systems Need to make servicing more efficient Need for additional quality control

Project Tasks • • Proposal and funding Research and preparatory Programming Data processing Quality control phases 1, 2, and 3 CDROM product Online implementation

Proposal and Funding • • ESDIM and OGP Federal Climate Complex datasets -- NCDC & US Navy surface hourly -- USAF global surface hourly & synoptic -- NCDC US hourly precipitation Full period of digital record “Merge” into one global database Quality Control

Programming Thus Far • • Time conversions (ISH in UTC time) Format conversions Station history preparation Data merging Inventories Quality Control Phases 1 and 2 Online Access System

Data Processing • • • Time and format conversions Merging where applicable Produce full period of record archive Inventories and inventory comparison Quality control (detail to follow…. )

Quality Control Phase 1 • Data comparisons as part of merge process to ensure records are for same station-time • Inventory system to ensure no data loss from beginning to end of process • Test data to verify all software is working properly • Random checks of “Version 1” output data • Reprocessing as necessary

Quality Control Phase 2 • • 57 quality control algorithms Validity checks for all data fields Extreme value checks to eliminate gross errors Temporal continuity checks (2 -sided) for elements such as temperature, dew point, wind, & pressure Internal consistency checks between elements Checks for several known systematic problems Test data verification Random checks of “Version 2” output data

Lessons Learned • Thorough test data are critical, and allow for the establishment of baselines for future comparison • Peer review is important—don’t be an “island” • Use phased approach—don’t insist on the “big house” initially; start with the foundation • Have a long-term plan, but be flexible • Expect to reprocess, but limit its frequency by following good design and testing principles

The Results…. ~ 20, 000 stations historically. As early as 1900 to present. Available data increases in 1948, and further increases in 1973.

Data Access Via Climate Data Online System - Access for U. S. and global digital data - Surface hourly, daily, monthly, with others planned http: //cdo. ncdc. noaa. gov

Future Plans • Place full period of record online (now have most recent data online (http: //cdo. ncdc. noaa. gov) • Additional online capabilities, summaries, etc • Integrated Surface Data Processing System—now in development for near real-time QC of all data • Merge additional data sources into ISH • Improve and enhance the metadata • Phase 3 quality control: more sophisticated temporal checks, spatial checks, homogeneity adjustments, etc • Partnerships with other agencies and groups