Management of AMSR-E and GLAS Data by NSIDC

Management of AMSR-E and GLAS Data by NSIDC Melinda Marquis AMSR/AMSR-E Data Management Team Lead GLAS/ICESat Data Management Team Lead National Snow and Ice Data Center Po. DAG Meeting April 25 & 26, 2001

Outline • • • Data Flows Data Distribution Services Documentation Post-launch communications Validation Operations Testing Progress & Milestones Subsetting Products tables

AMSR-E Data Flow L 0 EDOS L 0

GLAS Data Flow

Distribution of data by ECS Science Data Server Database EOS Data Gateway (EDG) Search and Order Data Management Subsystem Data Archive (Powderhorn) Granules Distributed User Data Server Subsystem

Data Distribution Services • • Data centers provide different levels of service for different products NSIDC distributes data at the same granularity as it is archived 1. We plan to limit distribution of level 0 AMSR -E and GLAS data 2. We intend to offer additional services, such as subsetting

Data Support Services Provided by NSIDC • DIFs (submitted to the Global Change Master Directory) • Guide Documents (User guides) • Web Site • Newsletter • DAAC Yearbook • Outreach

Directory Interchange Formats (DIFs) • A DIF is a metadata document • Submitted to the NASA’s Global Change Master Directory (GCMD) http: //gcmd. gsfc. nasa. gov/ • GCMD is a comprehensive directory of descriptions of data sets relevant to global change research. • A DIF is a collection of fields that allow user to decide whether a particular data set would be useful to them. • Compatible with U. S. federally mandated Federal Geographic Data Committee’s (FCDC) Content Standard on Digital Geospatial Metadata.

Guide Documents • Is a technical document (user’s guide) for data products funded by NASA. • Must follow a specific template • Template at (http: //harp. gsfc. nasa. gov/v 0 ims/DOCUMENTATION/GU IDE-VALID/GUIDES/Template. ECS 2. htm). • Contains detailed information about the data product, allowing an uninitiated user to understand use the data fully and accurately.

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC • Communication will continue to be important after launch • Either document protocol in Operations Agreement (OA) between NSIDC and SIPS or in Memorandum of Understanding (MOU) between NSIDC and SCF

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC • NSIDC’s web site will provide information on: – Availability of products – Status of Aqua and of AMSR-E and of ICESat and of GLAS – Known errors • Any changes to processing will need to be conveyed so our documentation can be updated – So we can inform users, including registered users

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC USER NSIDC USO NSIDC IN-HOUSE RESOURCES Science Operations Support Staff Programmers Ops Documentation Specialist Science Staff System Engineering/ Admin. As needed OUTSIDE RESOURCES Science Team SIPS Others

Validation – AMSR-E • Richard Armstrong, Jim Maslanik and Julienne Stroeve are actively involved in validation efforts in support of the AMSR-E science team • Therefore, they will perform very high level QA of AMSR-E data products

Archive Migration • Old drives and media will no longer be supported by Storage Tek. • Installation of new drives and media began in April 2001. • All data, including ASMR-E and GLAS will be archived on the new media, which will be the 9940 tapes from Storage Tek.

PDS (Product Distribution System) • Installation completed by the second week in April 2001. • The PDS system capabilities: – CD-R – 8 MM tape – DV • Automated QC of all media (not data) • Automated label printing

Mission Operations and Science Systems (MOSS) Tests – Aqua/AMSR-E • Formal test exercise to demonstrate Aqua Science Operations Concurrent Aqua test load with nominal Terra operations

Mission Operations and Science Systems (MOSS) Tests Aqua science systems end-to-end test from EDOS through science data processing facilities (AMSR-E SIPS) to assess readiness for: • Demonstration of system performance and concurrent operations ( ingest, archive, processing, user access, and distribution) for Terra, Landsat-7 and Aqua • Verify compatibility of AMSR data products with ECS data types • Ability to support early post-launch science operations with DAAC/SIPS production using simulated data • Demonstrate subscriptions, user pull and distribution (ftp and tape) • Demonstrate nominal operations personnel and procedures • Identify critical liens, problems, or defects that must be resolved prior to Aqua launch

MOSS 1 Summary Totals • Data Provider Data Type • • • • • • EDOS GHCC PO. DAAC AE_PMSCI PM 1 GBAD 1 PM 1 GBAD 4 PM 1 GDAB 8 AE_L 2 A AE_Ocean AE_Land AE_Rain AE_Rn. Gd AE_Dy. Ocn AE_Wk. Ocn AE_M 0 Ocn AE_Dy. Sno AE_5 DSno AE_Mo. Sno AE_S 16 AE_SI 12 AE_S 125 AE_Land 3 Browse AMSREL 1 A Granules Ingested 21 19 2 47 48 2 2 46 53

MOSS 2 Summary Totals • Data Provider Data Type Granules Ingested • • • • EDOS AE_PMSCI PM 1 GBAD 1 PM 1 GBAD 4 PM 1 GDAB 8 32 33 6 GHCC PO. DAAC AE_L 2 A 56 AE_Ocean 56 AE_Land 56 AE_Rain 56 AMSREL 1 A 56

Future MOSS Tests (Aqua/AMSR-E) • MOSS 3 : May 14 - May 18 • MOSS 4 : July 16 - July 20 • MOSS 5 : September 17 - September 21

GLAS Test Schedule • April 24: ICT -- went well. We received PDRs and PDSs for 17 L 0 APIDs. All successfully ingested. • May 14 -16: MOSS 3 (part of Aqua MOSS) • July 16 -18: MOSS 4 (part of Aqua MOSS) • July 2 -31: IGST-4 • July 10: IGST-6 • Sept. 3 - Oct. 5: IGST-7

Test Schedule: AMSR-E and GLAS Date AMSR-E GLAS Dec. 4 -7, 00 MOSS-1 X Feb. 20 -22, 01 MOSS 2 X May 14 -18 MOSS 3 X X July 2 -31 IGST-4 X July 10 IGST-6 X July 16 -20 MOSS 4 Sept. 17 -Sept. 21 MOSS 5 X X Sept. 3 - Oct. 5 IGST-7 Target launch date X X NET Dec. , 2001 Dec. 15, 2001

Progress – AMSR-E • ECS System Upgrades needed for AMSR-E –ECS 5 B and 55 Integration and Test • Install and check out Release 5 B (this was completed by July 2000, i. e. , before FY 01) • Install and check out Release 55 has been split into 2 parts: –a) 5 B 06 (Dec. 2000) –b) IRIX upgrade/SGI (2 nd quarter FY 01) • 5 B 07 (March 12 in TS 1; March 19 in OPS) o. System Tests –Perform Regressions Tests on Release 55 (2 nd quarter FY 01) –ECS 55 Site Readiness Acceptance (2 nd quarter FY 01) –System Confidence Tests (SCTs) (One already during 1 st quarter; maybe one or more in future. ) –MOSS (Mission Operations and Science Systems) Tests (First MOSS was in Dec. , 2000. Second was in Feb. , 2001, i. e. , 2 nd quarter FY 01) • Interface Documents –OAs and ICDs are completed, except for those with NASDA.

Milestones – AMSR-E • 1 st Quarter (December 31, 00) – 1. Installation and Checkout 55: –a. 5 B 06 (Dec. 2000) – 2. System Confidence Test – 3. MOSS Test (Dec. 2000) • 2 nd Quarter (March 31, 01) 4. IRIX upgraded 5. Second MOSS Test (Feb. ) • 3 rd Quarter (June 30, 01) – 6. DIFs completed – 7. Add additional Hardware (getting PDS, another rimage system, RAID) • 4 th Quarter (September 30, 01) – 8. Guide documents (summaries) completed – 1 st Quarter (December 31, 01) – 9. Operations Readiness Review (Oct. 15) – 10. Launch (NET 12/01) Level 0 ingest begins

Milestones -- GLAS • 1 st Quarter (by December 31) – 1. Write DIFs (drafts) • 2 nd Quarter (by March 31) – 2. Polish/finish (have reviewed) DIFs • 3 rd Quarter (by June 31) – – – 3. ICD b/w ECS at NSIDC DAAC and ISIPS 4. OA b/w NSIDC DAAC and ISIPS 5. OA b/w NSIDC DAAC and EDOS 6. OA or MOU b/w NSIDC DAAC and SCF ? 7. Interface Confidence Tests

Milestones – GLAS (cont. ) • 4 th Quarter (by September 30) – – – – – 9. Guide doc – L 1 Alt. (GLA 01, 05 & 06) 10. Guide doc – L 2 Alt. (GLA 12 -15) 11. Upgrades to ECS for support of ICESat 12. Installation and Checkout 13. Acceptance and Regression Tests 14. Installation and Checkout 15. Regression Tests 16. Interface Tests 17. Mission End-to-End Tests (IGSTs) • Launch (Dec. 15, 01) • FY 02, 1 st Quarter – Guide doc – L 1 &L 2 Atmos. (GLA 02, 7 -11) – Guide doc – GLA 16

Subsetting • ESDIS is working to provide subsetting for data managed in the ECS – MODIS – AMSR-E – GLAS • For data in HDF-EOS format, the University of Alabama’s HEW subsetter may be used • GLAS data products will be in binary format, with exceptions (higher level products) – NSIDC is pursuing becoming an r. SCF – NSIDC hopes to develop subsetter tool unique to GLAS

Backup Slides

AMSR-E Products

Products

GLAS Data Products and Granularity

GLAS Data Granularity and Potential User Issues • User A: – – GLA 12 is 49 MB/day * 183 days = 8, 967 MB Greenland Ice Sheet occupies 15% of GLA 12 0. 15*8967 MB = 1345 MB • User B: – – GLA 16 is 117 MB/day * 183 days = 21, 411 MB Greenland Ice Sheet occupies 5% of GLA 16 0. 05*21411 MB = 1071 MB

GLAS Data Granularity and Data Delivery Issues • A CD-ROM holds approx. 650 MB of data, an 8 mm Tape holds approx. 5 GB of data. • Our typical GLA 12 user would require 2 x 8 mm tapes or 15 CD-ROMs. • Our GLA 16 user would require 5 x 8 mm tapes or 36 CD-ROMs. • If the data were staged to an ftp server, then the user would be required to download large amounts of data, which would require large periods of time. – @56 kb/sec, the GLA 12 user would take ~44. 5 hrs – @56 kb/sec, the GLA 16 user would take ~106. 2 hrs

GLAS Data Granularity and Data Delivery Issues • However, if the data were subsetted for the Greenland Ice Sheet alone, then the GLA 12 user would require – 1 x 8 mm tape, 2 CD-ROMs or ~6. 6 hrs of @download time – similarly the GLA 16 user would require 1 x 8 mm tape, 2 CD-ROMs or ~5. 3 hrs of download time. • This reduction in media use will reduce media distribution costs and can encourage use of ftp to deliver the data to the user. Subsetting becomes essential for efficient data distribution and is being addressed.

Spatial search via NOSE ECS Science Data Server (SDSRV) EDG Query Parameters Spatial Query Server Granule List Spatial Indicies AMSR-E Granule Inventory

Implementation of Nominal Orbital Spatial Extent (NOSE) for AMSR-E ECS Science Data Server (SDSRV) AMSR-E pre-defined orbit data AMSR-E DLL Spatial Query Server AMSR-E Granule Inventory AMSR-E NOSE Table

Insertion of AMSR-E granule into ECS Science Data Server (SDSRV) AMSR-E granule metadata AMSR-E DLL Spatial Query Server AMSR-E Granule Inventory Spatial Indicies

Subscriptions 5. Email and/or Granule push to user 4. Event is recognized as “belonging” to this user Communications Subsystem (CSS) 1. USO enters subscription for user through GUI NSIDC User Services 3. DSS tells subscription Server about each event Data Server Subsystem (DSS) 2. Event occurs Ingest of data

QA at NSIDC • Automatic QA: – If can’t ingest -> no good – We plan to exercise media: pull data off randomly to inspect quality of media – still readable (new PDS does this, too) • Manual QA: – Inspect media when we distribute data on media (not via ftp) • We will support QA updates if provider wishes to utilize that functionality (hope to make it automated)

AMSR-E L 1 A Data -- availability Proposal: • One orbit of data will be available to team members only, three weeks after the AMSR-E instrument is turned on. • One (1) day's worth of data will be available to team members only, four weeks after the AMSR-E instrument is turned on. • All Level 1 A data will be continuously available to team members after 2 months.