Earth Science Data and Information System ESDIS Project

Earth Science Data and Information System (ESDIS) Project Update Karen. Michael@nasa. gov, EOSDIS System Manager AMSR Science Team Meeting Sept 23 -24, 2014 Huntsville, AL

ESDIS Project

EOSDIS Facilities Data centers, collocated with centers of science discipline expertise, archive and distribute standard data products produced by Science Investigator-led Processing Systems (SIPSs) ASF DAAC SAR Products Sea Ice, Polar Processes CDDIS Crustal Dynamics Solid Earth NCAR, U of Col. HIRDLS, MOPITT, SORCE Key Data Center LP DAAC GES DISC LAADS/ MODAPS La. RC ACRIM Ocean Circulation Air-Sea Interactions GHRC Hydrological Cycle & Severe Weather Atmosphere CERES, SAGE III San Diego PO. DAAC OBPG Ocean Biology & Biogeochemistry ORNL DAAC Biogeochemical Dynamics, EOS Land Validation SIPSs ECS Sites GLAS, MODIS, OMI, OBPG Atmos Composition & Dynamics, Global Modeling, Hydrology, Radiance NSIDC DAAC JPL GSFC Land Processes & Features Cryosphere, Polar Processes MLS, TES SEDAC Human Interactions in Global Change ASDC GHRC AMSR-E, LIS, AMSR 2 Radiation Budget, Clouds, Aerosols, Tropo Chemistry 3

EOSDIS Key Metrics EOSDIS Metrics FY 2013 (Oct 1, 2012 to Sept 30, 2013) Unique Data Products 6, 861 Distinct Users of EOSDIS Data and Services 1. 7 M Average Daily Archive Growth 8. 5 TB/day Total Archive Volume 9. 8 PB End User Distribution Products 839 M End User Average Daily Distribution Volume 22 TB/day EOSDIS Science Data Volume Growth EOSDIS Science Data Products Distribution FY 2000 through FY 2013 4

REVERB - Data and Service Access Client Primary web-based client for discovering and ordering crossdiscipline data from all of EOS Clearing House (ECHO) metadata holdings. Allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets, view browse images, and submit orders via ECHO to appropriate data providers. Users can submit queries using spatial and temporal criteria and examine search results for relevance using built-in tools. https: //earthdata. nasa. gov/reverb

REVERB - Data and Service Access Client Free Text Search GCMD Science Keywords The GCMD and ECHO repositories will eventually merge to form the Common Metadata Repository (CMR), a single, shared metadata repository for EOSDIS collection and granule level metadata.

LANCE – A Component of EOSDIS The Land Atmosphere Near-real time Capability for EOS (LANCE) is a component of EOSDIS that generates and distributes near real-time (NRT) products from the following instruments: • Terra: MODIS • Aqua: MODIS, AIRS • Aura: OMI, MLS • AMSR-E (Aqua) – ended 2011 • AMSR 2 (GCOM-W 1) – future • MISR (Terra) – being added LANCE Objectives: • To leverage science processing expertise to create high quality NRT products • To provide data to applications community with a maximum latency of 3 hours from observation (The standard, science-quality products are typically available with a latency of 20 -48 hours) • To provide data products with high reliability using redundant systems • To provide an umbrella environment with uniform high level requirements to better serve user needs (Consistency, Collaboration, Coordination) 7

LANCE Metrics 2014 8

ESDIS Data Discovery Capabilities Worldview (https: //earthdata. nasa. gov/labs/worldview) • Supports time-critical application areas such as wildfire management, air quality measurements, and weather forecasting. • Allows users to interactively browse satellite imagery in near real-time, add layers, and also provides a time series for comparison. GIBS (Global Imagery Browse Services) • GIBS are a set of standard services to deliver global, full-resolution browse satellite imagery in a highly responsive manner • Full Resolution mosaicked layers that are chopped into imagery tiles stored at predefined zoom levels Worldview/GIBS currently consists of: • Routine NRT production of: MODIS (C 5), AIRS, MLS, OMI, Ocean Color • AMSR-E historical data in process of being added – Just added 5 Cryospheric Layers in Arctic, Antarctic and Geographic projections from 2002 -2011 » : Sea Ice Brightness Temperatures (89 H GHz, 89 V GHz), Sea Ice Concentration (12 km, 25 km), Snow Depth over Ice (12 km) • GIBS/Worldview upcoming plans consist of adding: • Routine NRT production of GPM from PPS • Standard SMAP products 9 • MODIS Collection 6 (Forward + Reprocessed)

Continuing Browse Enhancements GIBS and Worldview • Data granules, used to create the imagery are also searchable and can be downloaded through the Worldview interface. • Delivery via Standard Services OGC WMTS / TWMS / KML Clients can be built to use and display images in GIBS – World. View is an example 10

All GIBS Product Downloads vs LANCE NRT

File Transfer Protocol Testing Compared file transfer protocol performance across different network environments in ESDIS Network lab and to Alaska • FTP, FTPS, HTTP, SCP, SFTP, HTTPS, bb. FTP, Grid. FTP • Varied file size, packet loss rate, round-trip time to emulate different network environments Test findings: • Multiple stream file transfer protocols perform best sending large files over networks with high packet loss and long roundtrip times (WAN) • bb. FTP, Grid. FTP • Single stream file transfer protocols perform best sending small files over networks with short round-trip times (LAN) • FTP, FTPS, HTTP • Protocols that encrypt the data are consistently the slowest. For best performance, data encryption should be avoided unless it is a requirement. • SCP, SFTP, HTTPS

Future File Transfer Protocol Work Continue working with DAACs and EOSDIS partners choose the best performing protocols for file transfers • Some DAACs have started incorporating multi-stream protocols, bb. FTP and Grid. FTP, for sending large files Test additional protocols, including Aspera and MMC ftp including tests with JAXA

Recent and Upcoming Missions Launch Date DAAC SNPP Oct 28, 2011 Multiple GPM Feb 27, 2014 GES DISC Jul 2, 2014 Dec 15, 2014 or Jan 29, 2015 GES DISC ASF / NSIDC Jan 2015 ASDC SAGE-III on ISS Jun 30, 2015 ASDC OCO-3 on ISS Aug 1, 2016 GES DISC CYGNSS (EV-M) Oct 28, 2016 PO. DAAC GRACE FO Jul 8, 2017 PO. DAAC ICESat-2 Jul 27, 2017 NSIDC OCO-2 SMAP DSCOVR 14

Future Suomi National Polar-orbiting Partnership Plans for NASA SDS • Current architecture – SNPP data is processed by NOAA IDPS – ESDIS manages the Science Data Segment (SDS) of SNPP and was responsible for product evaluation for VIIRS, Cr. IS, ATMS, OMPS Nadir as well as the production and distribution of CERES, and OMPS Limb – Archive and distribution of SNPP is at NOAA CLASS • Future Architecture – NASA will be producing and distributing it’s own products from all SNPP instruments and potentially future JPSS missions including NRT – Science Investigator-led Processing Systems (SIPS) and Science Teams have recently been selected for the following disciplines: Land, Ocean, Atmosphere, Ozone, Sounder 15

Soil Moisture Active Passive • SMAP Soil Moisture Active Profile • ORR Nov 4 -5, 2014 • Launch Dec 15, 2014, or January 29, 2015 • Downlink to Alaska, Mc. Murdo, Svalbard, Wallops (backup) • 15 contacts per day; 135 GB/day • 3 year mission timeline • SMAP data will be processed to L 0 by EDOS • SMAP science processing will take place at JPL and GSFC/GMAO • Ingest, archive and distribution of SMAP data will be at ASF for Radar data (L 1) and NSIDC for Radiometer data (L 1 -L 4) 16

Preservation NASA is not a “permanent archive” agency Must maintain “researchive” for as long as data are used for scientific research and/or transition responsibility to permanent archives • Researchive responsibilities persist well beyond lives of missions • NASA works with USGS and NARA for long-term preservation • NASA has to ensure data and other critical items are preserved and made available to permanent archival agencies NASA has developed Preservation Content Specifications for Earth Science Data Categories of Data to be preserved include: • Preflight/Pre-Operations • Science Data Products • Mission Data Calibration • Science Data Product Software • Science Data Product Algorithm Input • Science Data Product Validation • Science Data Software Tools • 17

Acronyms AIRS AMSR ASDC ASF ATMS bb. FTP CERES CMR Cr. IS CTGNSS DAAC DSCOVR ECHO EDOS EOSDIS ESIP FTPS GB GCMD GCOM GES DISC GHRC GIBS GMAO GPM GRACE-FO Grid. FTP HTTP ICESat IDPS Atmospheric Infrared Sounder Advanced Microwave Scanning Radiometer Atmospheric Science Data Center Alaska Satellite Facility Advanced Technology Microwave Sounder Ba. Bar File Transfer Protocol Clouds and the Earth’s Radiant Energy System Common Metadata Repository Cross-track Infrared Sounder Cyclone Global Navigation Satellite System Distributed Active Archive Center Deep Space Climate Observatory EOS Clearing House EOS Data and Operations System Earth Observing Center Earth Observation System Data and Information System Earth Science Information Partners File Transfer Protocol Secure Gigabyte Global Change Master Directory Global Change Observation Mission – 1 st Water Satellite in Series GSFC Earth Sciences Data and Information Services Center Global Hydrology Resource Center Global Imagery Browse Services Global Modeling and Assimilation Office Global Precipitation Measurement Gravity Recovery And Climate Experiment – Follow-on Grid File Transfer Protocol Hypertext Transfer Protocol Ice, Cloud, and Land Elevation Satellite-2 Interface Data Processing Segment ISS JAXA JPL JPSS KML LANCE MISR MLS MODIS MSFC NARA NASA NOAA NRT NSIDC OCO OGC OMI OMPS ORR PPS SAGE SCP SFTP SIPS SMAP SNPP TWMS USGS VIIRS WAN WMS WMTS International Space Station Japan Aerospace Exploration Agency Jet Propulsion Laboratory Joint Polar Satellite System Keyhole Markup Language Local Area Network Land Atmosphere Near real-time Capability for EOS Multi-angle Imaging Spectro. Radiometer Microwave Limb Sounder Moderate Resolution Imaging Spectrometer Marshall Space Flight Center National Archives and Records Administration National Aeronautics and Space Administration National Oceanic and Atmospheric Administration Near Real-Time National Snow and Ice Data Center Distributed Active Archive Center Global Modeling and Assimilation Office Open Geospatial Consortium Ozone Monitoring Instrument Ozone Mapper/Profiler Suite Operational Readiness Review Precipitation Processing System Stratospheric Aerosol Gas Experiment Secure Copy Secure Shell File Transfer Protocol Science Investigator-led Processing System Soil Moisture Active Passive Suomi National Polar-orbiting Partnership Tile Web Map Service United States Geological Survey Visible Infrared Imager Radiometer Suite Wide Area Network Web Map Service Web Map Tile Service