Meteorological Assimilation Data Ingest System MADIS Patty Miller

Meteorological Assimilation Data Ingest System (MADIS) Patty Miller (OAR/ESRL/GSD) Dave Helms (NWS/OS&T) NOAA System of Systems Workshop October 6 and 7, 2009 1 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Background History MADIS was established in 2001 to prototype new observation ingest, integration, quality control, and distribution techniques for real time and saved real-time data Goal To integrate and quality control NOAA and other-agency observations and make them easily accessible and usable for operations, research, and commercial purposes 2 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Background (continued) Overall Benefits A more usable, complete, accurate, timely, and higher density observational infrastructure for use in local weather warnings and products, model predictions, and hazardous situations NWS-Specific Benefits Improved observational functionality for… 1. enhancing forecaster situational awareness 2. reducing data access costs for Forecast Offices 3. supporting higher-resolution global and regional data assimilation systems 4. improving the National Digital Forecast Database (NDFD) 3 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Function Observation access, integration, quality control, and distribution system with software support Features • • Access to real-time and saved real-time data sets Observational quality control Application Program Interface (API) Multiple network-enabled data distribution mechanisms (ftp, http, ldm) • Documentation and user support, including customization packages for NWS’s Advanced Weather Interactive Processing System (AWIPS) 4 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Current Status System Capabilities Seamless access to real-time and saved datasets Continuous database updates triggered by arriving observations Uniform observation formats, units, and time stamps Quality control capabilities Station monitoring for network maintenance Authentication for proprietary data Web-enabled push/pull distribution capabilities, with server-side slice and dice capabilities On-the-fly data reformatting, variable transformation, and sounding generations 5 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Current Status (continued) Observational Datasets Scope MADIS supports the collection, integration, quality control, and distribution of thousands of NOAA and non-NOAA observations, including over 61 K surface stations from local, state, and federal agencies, and private networks, as well as upper-air datasets including multi-agency profiler, radiosonde, radiometer, selected satellite observations, and commercial aircraft observations. 61, 172 Surface Stations from 160 networks producing over 12, 800, 000 observations/day 154 Profiler Sites (> 200, 000 obs/day) Over 450, 000 aircraft observations/day Plus global radiosonde and satellite obs Profiler data includes NOAA Profiler Network and Cooperating Agency Profilers Aircraft data includes MDCRS, AMDAR, and TAMDAR Surface data includes METAR, maritime, snow, HCN-M, Urba. Net, and other mesonet 6 MADIS Users Include: NWS Forecast Offices & National Centers NOAA Research NESDIS NOS, other NOAA NCAR NASA DOE Laboratories FAA & EUROCONTROL International meteorological centers Over 200 Universities Hundreds of private companies October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Current Status (continued) Data Sets • Surface Observations – – – – – Meteorological Aviation Reports (METAR) Maritime High-Frequency METAR Historical Climatology Network – Modernization (HCN-M) New England Pilot Project (NEPP) Network Urba. Network Climate Reference Network (CRN) Integrated Meteorological Mesonet* Integrated Hydrological Mesonet* Integrated Snow Mesonet* * - Multi-Agency Data Set 7 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Current Status (continued) Data Sets - Continued • Upper Air Observations – – – Radiosonde NOAA Profiler Cooperating Agency Profiler (CAP)* Next Generation NOAA Profiler (NGNPN) Radiometer* Satellite Radiances Satellite Soundings Satellite Winds Automated Aircraft* Automated Aircraft Profiles at Airports* Experimental Weather In-Situ Deployment Optimization Method (WISDOM) Balloon * - Multi-Agency Data Set 8 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Northeast Standard Surface Network Standard Surface Observations Meteorological Aviation Reports (METARs) Maritime 9 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Northeast Surface Network Additional Surface Observations - New England Pilot Project (NEPP) - AWS Convergence Technologies, Inc. - Citizen Weather Observer Program - Remote Automated Weather Stations - ESRL Ground-Based GPS Meteorology - Weather for You. com - Anything Weather - Soil Climate Analysis Network (SCAN) - Gulf of Maine Ocean Observing System - National Ocean Service Physical Oceanographic Real-Time System (PORTS) and National Water Level Observation Network (NWLON) - Aberdeen Proving Grounds (APG) - OAR – DCNet - Urba. Net - Hydrometeorological Automated Data System (HADS) - North Carolina ECONet - New Jersey Weather and Climate Network - Do. Ts: GA, KY, ME, MD, NH, OH, VA, VT 10 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Northeast Regional Domain Observations 11 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Midwest Standard Surface Network Standard Surface Observations Meteorological Aviation Reports (METARs) Maritime 12 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Midwest Surface Network Additional Surface Observations - AWS Convergence Technologies, Inc. - Citizen Weather Observer Program - Remote Automated Weather Stations - ESRL Ground-Based GPS Meteorology - Weather for You. com - Anything Weather - Soil Climate Analysis Network (SCAN) - National Ocean Service Physical Oceanographic Real-Time System (PORTS) and National Water Level Observation Network (NWLON) - Urba. Net - Oklahoma Mesonet - DOTs: IA, IN, KS, KY, MN, ND, OH, WI - Marquette Mesonet - Union Pacific Railroad - Non-Federal AWOS - NERRS (National Estuarine Research Reserve System) - Co. Ra. HS - Hydrometeorological Automated Data System (HADS) - North Carolina ECONet 13 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Midwest Regional Domain Observations 14 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS West Coast Standard Surface Network Standard Surface Observations Meteorological Aviation Reports (METARs) Maritime 15 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS West Coast Surface Network Additional Surface Observations - AWS Convergence Technologies, Inc. - Citizen Weather Observer Program - Remote Automated Weather Stations - Weather for You. com - Anything Weather - Soil Climate Analysis Network (SCAN) - National Ocean Service Physical Oceanographic Real-Time System (PORTS) and National Water Level Observation Network (NWLON) - Union Pacific Railroad - Snow Information and Technology (SNOTEL) - CA River/Nevada River Forecast Center - Co. Ra. HS - U. S. Bureau of Reclamation Agrimet - Pacific Northwest National Laboratory - CO River Basin Forecast Center - Dugway Proving Grounds - Non-Federal AWOS - Urba. Net - DOTs: CA, CO, ID, MT, NV, UT, WY and many more mesonets… 16 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS West Coast Regional Domain Observations 17 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS – Research to Operations Major Milestones April 3, 2008 NOAA Independent Review Team (IRT) formed to assist NOAA management in how best to transition MADIS to NOAA operations April 29, 2008 Business case analyses submitted to IRT May 22, 2008 IRT meets with subject matter experts to examine and select recommended solution May 29, 2008 IRT issues final report October 22, 2008 Jack Hayes (NWS Assistant Administrator) and Rick Spinrad (OAR Assistant Administrator) sign Letter of Agreement 18 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Research to Operations (continued) The NOAA MADIS Independent Review Team unanimously selected a joint OAR/NWS distributed processing solution Transition Goals Expedite the transition of current GSD capabilities to operations Maintain the continuity of MADIS data streams and services before, during, and after the transition Pre-plan for product improvements and technology infusion Summary Statement “The partnership between OAR and NWS led to a solid technical solution and provided a smoother transition from research to operations. ” 19 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Research to Operations (continued) The NOAA MADIS Independent Review Team Technical Recommendation – IT Architecture Port the existing GSD MADIS software to an integrated NWS TOC and NCO distributed environment, with a supporting backup and research-to-operation test environment at GSD MADIS Compute Environment NCEP TOC Decoders Ingest Internet Distribution 20 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009 Integration and Quality Control

MADIS Transition Plans Time Table to IOC FY 2009 October 1, 2008 October 1, 2009 FY 2010 October 1, 2010 Port Real-Time Processing to NCEP Port I/O Software and Hardware to the TOC Install I/O Software and Hardware at the TOC October 1, 2009 End-to-End Testing Training / C&A OT&E Resolve OT&E Issues 21 October 5, 2009 April 2002 April 1, 2009 NOAA System of Systems Workshop – October 6 and 7, 2009 April 1, 2010 Initial Operating Capability June 3, 2010

MADIS Transition Plans Time Table to FOC FY 2010 FY 2011 June 1, 2010 FY 2012 April 1, 2011 April 1, 2012 Port Data Recovery Process to NCEP Port Web Displays to TOC MADIS Servers CAP Port to TOC Install CAP Hardware and Software January 14, 2011 Prepare Archive for Transfer to NCDC End-to-End Testing OT&E (4/14/11 – 4/29/11) Resolve OT&E 22 October 1, 2010 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009 Full Operating Capability June 14, 2011 October 1, 2011 …

MADIS Ingest and Distribution – IT Architecture Internet Firewall Load Balancer MADIS - Ingest NCO - Processing 23 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009 MADIS - Distribution

MADIS Processing – IT Architecture (continued) NCEP/NCO CCS Computing • MADIS processing code loaded on development and production sides of both primary and backup CCS computers • 16 IBM Power 6 4. 7 GHz shared processors with shared memory • Memory: 38 GB RAM • Storage: 3 TB disk storage tuned for small file I/O • Operating System: AIX – Software • Leverage existing NCO decoders whenever possible • Port MADIS GSD decoder software for other agency observations 24 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

Planned Product Improvement Product improvements such as: 1) advanced data query and web services; 2) expanded metadata fields; 3) additional datasets; and 4) improved and expanded observation QC and station monitoring will serve: NWS Operations • Next. Gen – Weather Information Database (WIDB)/FAA support • National Surface Weather Observing System (NSWOS)/FHWA support • Next Generation NOAA Profiler Network (NGNPN) • Historic Climate Network – Modernized (HCN-M) • Urba. Net, National Mesonet • Improved web-based product generation/Meso. West partnership NOAA Research 25 • Hydrometeorological and Severe Weather Testbeds • Fire weather, renewable energy, and DHS research support (Geo-Targeted Alerting System [GTAS], WISDOM) October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS National Mesonet Plans • Accelerate Mesonet Ingest • Add Mobile Vehicle Observations • Extend and Enhance Metadata Infrastructure 26 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

Supported Data Formats Current • • • AWIPS 1 -compliant net. CDF CF-compliant net. CDF Text MADIS XML Joint METOC Broker Language (JMBL) BUFR Future • OGC Web Feature Service • ISO Metadata • Others? 27 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009

MADIS Team Members Integrated Work Team Extended Integrated Work Team Dave Helms (NWS Co-Chair) Patty Miller (OAR Co-Chair) Brent Gordon (NCEP/NCO) Sue Murphy (OCIO/TOC) OAR/ESRL/GSD Mike Barth Dennis Day Josip Edgett NWS/NCEP/NCO Becky Cosgrove Allan Darling Ben Kyger Michelle Mainelli Dan Starosta Steve Delgreco GSD Developers Mike Barth Leon Benjamin Randall Collander Joanne Edwards Tom Kent 28 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009 NWS/OCIO/TOC Kevin Conaty Craig Hegemann Jackie Lord Francis Yang NESDIS/NCDC Neal Lott

Summary • MADIS supports a large community of multi-agency data providers and data users • Current capabilities include extensive (and growing) net-enabled data services for observations • Transition to NOAA operations underway • Next. Gen FY 2010 – FY 2012 plans include WIDB net -enabled data services for surface data (METAR, maritime, high-frequency METAR, Urba. Net, HCNM, CRN, mesonet), automated aircraft, PIREP, lightning, and profiler observations 29 October 5, 2009 April 2002 NOAA System of Systems Workshop – October 6 and 7, 2009