EMC NOAA Operational Model Archive Distribution System na

EMC NOAA Operational Model Archive Distribution System na Natio “ l” NOMADS Toward an Operational service orientated architecture NCEP/Environmental Modeling Center Jordan C. Alpert/NCEP/EMC Glenn Rutledge/NCDC Jun Wang/SAIC jordan. alpert@noaa. gov 5 th GO-ESSP Earth Science Portal Workshop, June 20, 2006 at Lawrence Livermore National Laboratory where the nation’s climate and weather services begin

NOMADS Web Page • Completely re-designed. • Uses “php” • Easier navigation • Better Documentation • NOMADS is renamed: NOAA National…

What is service oriented architecture? “NOMADS Vision is a distributed service oriented architecture, a system-ofsystems integration based on using loosely coupled connections among independent systems to create a scalable, extensible, interoperable, reliable, and secure framework. ” … reliable?

What is NOMADS? (con’t) Real Time systems at NCEP and NCDC and archives at NCDC make for seamless past to present data availability. n n A digital archive of NOAA’s operational weather models, and an “innovative data access philosophy (read free) to promote interoperable access across the geosciences” (BAMS, Rutledge et. al. , 2006). A Pilot project at NCEP that advances integration (read access) of real time Operational model data. NOMADS: An integrator of common web services infrastructure to support the discovery, access and transport of data (NOAA GEO-IDE Document) User/client applications developed at EMC, NCEP’s development division, and continued development at NCDC.

The NOAA Operational Model Archive and Distribution System NOMADS Goals • provide distributed access to models and associated data, • promote model evaluation and product development, • foster research within the geo-science communities (ocean, weather, and climate) to study multiple earth systems using collections of distributed data, • develop institutional partnerships via distributed open technologies.

Service oriented Goals (DMIT) Goals • To take advantage of internet technology opportunities. • Improve efficiency and reduce costs by bridging the barriers between existing, independent “stove pipe” systems. • Integrate the data management activities of NOAA projects. • Individual components retain responsibility and authority within the context of a systematic set of principles. • Develop and adopt standards for metadata, data discovery and data transports, formats and protocols.

The Operational (reliability) problem To provide Operational Services where security, timeliness (time critical) and reliability are paramount. Operational public access services for public access (includes university/research/scientist) to data, products and information services. Scientific services where efficient and flexible discovery and access to data sets is required. Commercial value-added services and user client applications.

NOAA/NCEP Operational ftp services for model data (What is Operational today). • Two operational centers (ftpprd… and tgftp…) • Standard ftp servers • Load balance, Fail over, State of the art hardware ($) and 24/7 support. • A little difficult to navigate (e. g, file naming convention) • Entire file (0. 25 GB) download necessary. • Holdings not complete (0. 5 degree GFS, NAM hourly, ensembles…? ) Complete and “fast ftp” (Partial ftp transfers) for all data set holdings is imminent.

What the user sees at the operational ftp site: Example of tgftp location for GFS model (first entry) files from June 15: ftp: //tgftp. nws. noaa. gov/SL. us 008001/ST. opnl/MT. gfs_CY. 00/RD. 20060615/PT. grid_DF. gr 1 An Excerpt from the list of 500 files, note name and size: File: fh. 0030_tl. press_gr. onedeg File: fh. 0030_tl. sflux 26562 KB 42981 KB 06/15/2006 Excerpt from the ftpprd inventory description: Inventory of File gfs. t 00 z. pgrbanl Model: GFS Cycle: 00 UTC Forecast: 0 HRS Number of Records: 267 Grid Identification : 3 Number 0001 0002 0003 Level/Layer 1000 ISBL 975 ISBL 950 ISBL Parameter Description HGT 1000 h. Pa isobaric level Geopotential height gpm] HGT 975 h. Pa isobaric level Geopotential height [gpm] HGT 950 h. Pa isobaric level Geopotential height [gpm]

Toward Operational services • We know how to do the hardware part. $ = load balance, fail over, 24/7, redundancy • What about users usage of the applications, e. g. , DAP and GDS and others • Operationalize data flow to NCDC from NCEP. • GRIB 2 with jpeg packing increases CPU resources needed. • Software to handle sub-seting and GRIB 2 • GDS, OPe. NDAP, and other applications need GRIB 2 software strategy. • Implement with partial http transfers and index files (first step).

Highest month: 4. 3 Tb w/3. 8 million downloads – NCEP has ~1 -5 Tb /month User Statistics NCDC Only BAMS Paper

Real Time (NCEP) nomad 3 server • GDS/OPe. N(DAP) DODS: typically 250, 000 queries/day from the GDS log (already /2). • pdisp (“Great Displays”) 2694/day. • http access_log ~200, 000 hits/day • ftp 2 u (fast ftp index download) over 5/11 – 6/15 8, 623* repackaged GRIB downloads/day. Need better ways of evaluating use for good health and for comparison of servers.

Real Time (NCEP) NOMADS Server • Users query files that are not present in tight loops. • During the 12 hours [00 Z-12 Z] on June 1, of the 371 different users who queried the httpd server error_log, the top 7 users with the most queries are listed: #queries User 1579 hawaii. edu 1616 abo. wanadoo. fr 1870 natpool. mwn. de 3293 accuweather. com 4039 saildocs. com 7864 hmg. inpg. fr 100465 labsolar. ufsc. br A solution: Check log file (error_log) user access rate. If the rate criteria (6/minute) is exceeded then place user’s IP address in IP tables to block access for that IP address for a period of time (10 minutes). Placing users in a “penalty box” when they are repeatedly accessing the server in tight loops is the same as users placing a “sleep 600” command or wait 10 minutes in their unix script for loop. This represents a throttle that can be applied to other situations such as many GDS queries when one is needed.

Multiple paths to format independent data access The NOMADS System Design Direct Client Access NCDC Archive Gr. ADS Data Server NCEP Dual Ingest & QC Web Browse & Compute: Sister servers Live Access Other Server Portals Gr. ADS, Ferret, Mat. Lab, IDL, IDV, Web browsers or any OPe. NDAP enabled client THREDDS Data Server Project ACCESS: NASA, GMU, NOAA (NCDC), GMU & OPe. NDAP. org Web Plotter gds, http, binary ftp sub-setting CEOS-Grid Exploratory Grid Projects w/ Globus

Search, Discovery, Access, and Analysis A metadata server, DIMES (Distributed Metadata Server) integrated with GDS to form MIDAS (Kafatos, Yang, Zhao) : Metadata Integrated Data Analysis Server • • • Content consistency between data server and metadata server. Interactive metadata search and data access and analysis. Potential for more effective and efficient searching. Potential for better interfaces. More seamless transition from data search to data access. Uses XML to represent, store, retrieve and interoperate metadata with minimum semantic enforcement. • Contains a metadata model, XML query engine, web-based prototype interface. • Enables the application power of Grads and GDS server side data analysis.

One can search by Time resolution (forecast time), Spatial resolution, Search_Space (region), Search_Time (Cycle), and Text (by variable and other Title text in the Metadata file).

After selection of the Time and Space Box, the user can generate an “sdfopen” URL command including functionality, e. g, time or space averaging, for further processing or generate the DODS URL and run Gr. ADS client script and interactive commands.

Conclusions and Future Plans • Incorporate NOMADS functionality into CLASS (Comprehensive Large Array Stewardship system) • Enhance the NCEP real time servers at NCDC and place in NCDC operations. • Operationalize the data flow from NCEP to NCDC. • Existing Operational NOAA servers can be enhanced with NOMADS service orientated architecture and accepted by operational environment administrators. • Support working prototypes for proof of concept and continued development toward improved services.