Recent Updates of the Architecture of MADIS Data

Recent Updates of the Architecture of MADIS Data Processing and Distribution at GSD NOAATECH 2006 Chris Mac. Dermaid Data Systems Group Lead Robert Lipschutz Production Control Manager

Agenda MADIS Background ● Current System Architecture ➢Data Ingest ➢Data Processing ➢Data Saving ➢System Monitoring ● Future Plans ●

Meteorological Assimilation Data Ingest System (MADIS) Goal To integrate other-agency observations with NOAA observations and make them easily accessible and usable to the greater meteorological community Features

MADIS Surface Mesonet System Current mesonet sites = >18, 000 ● ● Current mesonets = 114 ● Largest mesonets ● ● ● AWS Convergence Tech. Inc. Citizen Weather Observing Program (CWOP) Interagency Fire Center’s Remote Automated Weather System (RAWS)

MADIS in GSD's Central Facility MADIS Requirements Acquiring data from many sources ● Moving data between process nodes ● Distribution to MADIS's many users ● Needs to run reliably with little downtime ● Online and near-line collection of saved data dating back to 2001 ●

MADIS Processing Red Hat Enterprise Linux v. 3 and 4 ● 8 Dell dual Xeon 1750's ● 4 Dell Powervaults 220 s's ● Netapp FAS 940 ● Linux Heartbeat ● System. Imager ● Concurrent Versions System (CVS) ●

LDAD/RSAS/MADIS Mesonet “puts” Mesonet “gets” LDAD net. CDF LDM FTP MADIS & Raw LDAD FTP & HTTP MADIS ftp and Open. DAP Raw LDAD QC ACARS, METAR, Maritime SAO, RAOB, & Profiler (Point data from NOAAPORT, DD, . . . ) ODS LDAD net. CDF LDM farm MADIS External LDM Behind Firewall MADIS core MADIS & LDAD net. CDF DMZ hosts MADIS Outside FSL RSAS Point data TOC RSAS & MADIS RSAS Backup to NCEP Point data Net. App LDAD net. CDF LDAD, RSAS & MADIS to MSS MDR MADIS to ftp

MADIS Data Ingest LDM (Unidata's Local Data Manager) ● Perl scripts for FTP gets ● 'wget' for HTTP gets ● FTP and HTTP servers for pushes ● NOAAPORT ● ACARS (ARINC Communications and Address Reporting System) ●

MADIS Ingest Data Flow FTP & LDM MQ client Sat Winds/Soundings, Profiler, . . . ACARS ODS Data Processing METAR, SAO, RAOB, . . . noaaport Ingest Methods FTP “put” FTP “get” HTTP (wget) HTTP “put” (cgi) LDM Satellite (SBN) MQ Client (SBN) HADS LDAD Acq Ingest Surface Data Sets Hydro Maritime METAR Mesonet NERON (COOP) SAO Upper Air Data Sets Radiosonde Aircraft (ACARS) NOAA Profiler Multi-Agency Profiler Radiometer Satellite Winds Satellite Soundings MADIS Processing LDAD Processing Mesonet, Hydro, COOP, . . .

Aircraft Observations TAMDAR from Air. Dat (ldm) LH & BUFR MDCRS from TOC (ftp) FTP & HTTP MADIS LDM External FTP Filtered ACARS MADIS A/C MADIS ACARS QC'd A/C net. CDF LDM farm AMDAR A/C net. CDF Raw ACARS, AMDAR, TAMDAR, LH & MDCRS to MSS net. CDF to MSS External LDM

MADIS Distribution ~375 FTP users and ~150 LDM users ● 7 levels of access ● New data every 5 minutes ● FTP server ● e. Directory LDAP service ● OPe. NDAP ● LDM Farm ● Surface Observation web page ●

MADIS Distribution Flow ftp http Open. D AP http Sfc Obs

Data Saving Data starting July, 2001 (>300 GB) ● 1 TB Snap Appliance File System ● Duplicate Snap server for GSD ● Mass Store System (MSS) for backup ●

MADIS Monitoring Operator coverage from 7 AM to 7 PM daily, 11 AM to 7 PM on weekends ● The Facility Information and Control System (FICS) ● Bugzilla help/log ticketing system ● Cacti Performance Monitor ●

MADIS Future

Questions?