Net. CDF 4 Reformatting Toolkit: BUFR and GRIB Tailoring Preliminary Design Review April 14, 2008 Prepared By: Tom King 1, Walter Wolf 3, Lihang Zhou 1, Yi Song 2, Larisa Koval 1, 1 PSGS, 2 IMSG, 3 NOAA/NESDIS/STAR 1
Review Agenda Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 9: 00 am – 9: 20 am – 10: 00 am – 10: 45 am – 11: 05 am – 11: 20 am – 11: 30 am Wolf King Wolf 2
Review Outline · · · Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 3
Introduction Presented by Walter Wolf NOAA/NESDIS/STAR 4
Introduction · Project Background » » Terminology IJPS NPP/NPOESS NDE · Project Objectives · Integrated Product Team · Project Plan · Entry and Exit Criteria 5
Project Background Terminology · Raw Data Records (RDRs) · Sensor Data Records (SDRs) · Temperature Data Records (TDRs) · Environmental Data Records (EDRs) · Intermediate Products (IPs) 6
Project Background Terminology · Raw Data Records (RDRs) » Full resolution, digital sensor data, time-referenced and locatable in earth coordinates with absolute radiometric and geometric calibration coefficients appended, but not applied, to the data. 7
Project Background Terminology · Sensor Data Records (SDRs) » Data record produced when an algorithm is used to convert RDRs to geolocated, calibrated detected fluxes with associated ephemeris data. Calibration, ephemeris, and any other ancillary data necessary to convert the sensor units back to sensor raw data (counts) are included. 8
Project Background Terminology · Temperature Data Records (TDRs) » Calibrated antenna temperature records from a microwave instrument such as ATMS. · Environmental Data Records (EDRs) » Data records produced when an algorithm is used to convert SDRs to geophysical parameters (including ancillary parameters, e. g. , cloud clear radiation, etc. ). 9
Project Background Terminology · Intermediate Products (IPs) » The NPOESS products that are produced as an intermediate step between the SDR and EDR. These are available from the IDPS upon request. 10
Project Background NPP/NPOESS • NPP and NPOESS, a joint Military/NOAA/NASA effort, is the next series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the NPOESS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (1: 30 pm). • Instrument packages on NPOESS: » Cr. IS, ATMS, VIIRS, OMPS, SEM, CERES • NPP is the first of five missions with launch dates of ≈2011, ≈2013, ≈2016, ≈2018, ≈ 2020, respectively. 11
Project Background NDE · Disseminate NPOESS Data Records to customers. · Generate and disseminate tailored NPOESS Data Records (versions of NPOESS Data Records in previously agreed alternative formats and views). · Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPOESS Data Records). · Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving. · Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA. · Provide software for NPOESS Data Record format translation and other data manipulations. 12
Project Objectives · To build a software package that will tailor NPOESS and NDE products from net. CDF 4 into BUFR and GRIB 2 formats in support of NDE’s overall tailoring efforts. · The Net. CDF 4 Reformatting Toolkit (N 4 RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products. » Flexible » Extendable · The software must be able run within the NDE system architecture and operate within the NDE functional guidelines. · Output product formats and content must meet the needs of NOAA customers. 13
Project Objectives Phase 1 Products · Products to Reformat » » » » ATMS Radiances Cr. IS Radiances Nadir Profile Ozone (OMPS) and OMPS Radiances VIIRS Radiances Snow Cover Vegetation Index Aerosol Optical Thickness Sea Surface Temperature 14
Integrated Product Team · IPT Lead: Walter Wolf (STAR) · IPT Backup Lead: AK Sharma (OSDPD) · NESDIS team: » STAR: Walter Wolf, Hank Drahos, Jaime Daniels, Yi Song, Thomas King, Larisa Koval » OSDPD: Dave Benner, AK Sharma, Ricky Irving » OSD: Tom Schott, Jim Yoe » Data Center: Lei Shi (NCDC) · User team » Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber (NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard (JCSDA), Tony Mc. Nally (ECMWF), Fiona Hilton (UK-Met) » Others: International NWP users, NWP FOs, Climate Users · Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP 15
Project Stakeholders · NOAA National Weather Service » Weather Forecast Offices » National Center for Environmental Prediction · Department of Defense » NRL » FNMOC » AFWA · Global NWP » » EUMETSAT UK Meteo France CMC 16
Project Plan · Year 1 – Design and Development (2008 – 2009) » Verify Requirements – Work with customers to verify product requirements – Discuss with the current developers of similar translators to determine what is required in their output files » Design the Net. CDF 4 reformatting toolkit; » Conduct PDR » Develop BUFR tables and GRIB formats with the product teams for Phase 1 products » Work with NDE to determine the interface between the Level 1 B and the Level 2 NPP products and the reformatter » Conduct CDR 17
Project Schedule – Phase 1 18
Project Plan · Year 2 –Transition to Pre-Operations of Phase 1 Products (2009 – 2010) » Set up infrastructure to implement the readers and writers for the data formats » Implement BUFR tables and GRIB formats for the Phase 1 products on the NDE hardware » Conduct Test Readiness Review for Phase 1 products » Transition and test system within the NDE environment » Conduct Code Review for Phase 1 products 19
Project Plan · Year 3 – Transition to Operations of Phase 1 Products (2010 – 2011) » Evaluate with NDE and OSDPD the implementation of the Reformatting Toolkit within the NDE data handling system » Conduct System Readiness Review for Phase 1 products » Transition pre-operational Phase 1 product reformatting system to operations 20
Phase 1 Transition to Operations 21
Project Plan – Schedule · Schedule (Milestones) » » » Project begins – 07/01/08 Preliminary Design Review – 04/14/09 (10/21/08) Critical Design Review – 09/25/09 (03/19/09) Test Readiness Review – 06/09/09 (02/25/09) Code Unit Test Review – 09/10/10 (01/29/10) System Readiness Review – 01/31/11 (04/20/10) – Waive or shift to NDE 22
PDR Report · The PDR Report (PDRR) is a standard artifact of the STAR EPL process. » The PDR report will be produced after the PDR. » The report will be a critical artifact for the Critical Design Review. · Guidelines for the PDRR are found in STAR EPL process assets 23
PDR Entry Criteria · Requirements Document · Review of Net. CDF 4 Reformatting Toolkit: BUFR and GRIB » » Requirements Software Architecture Quality Assurance Risks and Actions 24
PDR Exit Criteria · Preliminary Design Review Report » The PDR Report (PDRR), a standard artifact of the STAR Enterprise Process Lifecycle (EPL), will be compiled before the CDR » The report will contain: – Actions – Comments – PDR presentation 25
Review Objectives · Review the Requirements · Review Software Architecture · Review Quality Assurance · Identify risks and actions 26
Review Outline · · · Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 27
Requirements Presented by Thomas King NOAA/NESDIS/STAR 28
Requirements Overview · SPSRB Requirements were presented to the developers in a document entitled: “Level 1 Requirements for a Net. CDF 4 Reformatting Tool” (Version 1. 5). · Text in yellow are those requirements derived from the original SPSRB requirements. · Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, EMC (the customer) and the heritage product teams. · Using all of this information a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project. 29
Original Phase I Product List Prioritized Product BUFR GRIB 2 ATMS Radiances X Cr. IS Radiances X -Thinned radiances assumed and are okay - If concurrent delivery w/Cr. IS radiances, include cloud mask Nadir Profile Ozone (OMPS) and OMPS Radiances X Include OMPS radiances in nadir profile product VIIRS Radiances X Snow Cover X Vegetation Index Comments X Aerosol Optical Thickness X Sea Surface Temperature Incorporate into current snow product X 30
Original Phase I User/Heritage Mapping Prioritized Product EMC User Contacts Heritage Product ATMS Radiances Dennis Keyser, John Derber AMSU, AMSR-E Cr. IS Radiances Jim Jung, Dennis Keyser Jack Woollen, Simon Elliott IASI, AIRS Nadir Profile Ozone (OMPS) and OMPS Radiances Dennis Keyser Larry Flynn, Donna Mc. Namara SBUV, GOME VIIRS Radiances Dennis Keyser, John Derber AVHRR GAC Snow Cover Michael Ek Sean Helfrich IMS Products Vegetation Index Michael Ek Le Jiang, Felix Kogan AVHRR Veg. Aerosol Optical Thickness Jeff Mc. Qeen Paul Haggerty MODIS Aerosols Sea Surface Temperature Bert Katz, William Gemmill Shasha Ignatov, John Sapper, Robert Grumbine AVHRR derived SST (ACSPO) 31
Functional Requirements: Reformatting Toolkit Software · Requirement: STAR shall deliver to NDE a reformatting toolkit capable of translating NESDIS Net. CDF 4 data products into NCEP-accepted data formats (i. e. , BUFR and/or GRIB 2). » Requirement: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list. 32
Functional Requirements: Reformatting Toolkit Software » Requirement: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment. – The Toolkit shall compile and run on the NDE IBM AIX P 5 series hardware. – The Toolkit shall interact with the NDE Data Handling System (DHS). – The Toolkit shall be able to read a Production Control File (PCF). – The Toolkit shall handle and return errors according to NDE/STAR standard codes. – The Toolkit shall be able to write a PSF. » Requirement: The toolkit shall consist of modular components that can be tested independently. – The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family hierarchically structured tailoring subroutines. – Data shall be stored in allocatable data structures. 33
Functional Requirements: Reformatting Toolkit Software » Requirement: STAR shall include one update to the reformatting toolkit within its initial project plan. » Requirement: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products. » Requirement: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB 2 in the reformatting toolkit. » Requirement: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB 2 libraries – Updates shall be made when there are updates to the versions of the net. CDF 4 library being used by NDE. » Requirement: NDE and OSDPD shall implement the updated reformatting toolkit into operations once STAR has updated it with BUFR and GRIB library updates from NCEP. 34
Functional Requirements: Reformatting Toolkit Software » Requirement: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development. » Requirement: The output from the toolkit shall be compared with the input. » Requirement: The translation toolkit shall convert from the new format back into Net. CDF 4. » Requirement: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes. – The Reformatting Toolkit software shall generate run logs and return NDE/STAR standard (agreed upon) error codes to the DHS. – Run times shall be monitored by the NDE DHS. 35
Functional Requirements: Reformatting Toolkit Software » Requirement: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e. g. , dissemination). » Requirement: The toolkit parameters (e. g. , how to use the service) shall be well documented. – Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP). » Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators. – Reformatting Toolkit shall use the standard set of error return codes developed by NDE for code running within the DHS. The NDE system shall direct this information to operators. 36
Functional Requirements: Reformatting Toolkit Software » Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting. – All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP. » Requirement: STAR shall coordinate development of the reformatting toolkit Application Program Interface with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system. – The Reformatting Toolkit shall work with the NDE integration team to integrate the Reformatting Toolkit software into the NDE DHS. » Requirement: Toolkit code shall adhere to the STAR coding standards. » Requirement: Performance shall be measured on a product level. 37
Program Requirements: Reformatting Toolkit Project · Requirement: STAR shall provide monthly project status reports to OSDPD and OSD. · Requirement: Earned Value Management shall be performed on the project. · Requirement: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration. · Requirement: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness. 38
Product Requirements: Cr. IS Radiances · Requirement: The Reformatting Toolkit shall tailor the NUCAPS thinned Cr. IS Radiances from Net. CDF 4 into BUFR for EMC. » The Reformatting Toolkit developers shall work with EMC to create a BUFR table for the NUCAPS thinned radiances based on AIRS and IASI. » The table shall use delayed replication for storing the radiances. » BUFR messages shall be smaller than 50 KB. » The BUFR format shall allow for the storage of negative radiances. » The file shall contain the following data fields (see table next slide): 39
Product Requirements: Cr. IS Radiances Variable Name Satellite ID Latitude ID of Originating Center Satellite Instrument Longitude Height of Land Surface Satellite Height Satellite Zenith Angle Land Fraction Satellite Classification Satellite Azimuth Land/Sea Qualifier Year Solar Zenith Cloud Cover Start Channel (per band) End Channel (per band) Calibration Quality Flags Field of View Quality Flags Geolocation Quality Month Solar Azimuth Height of Cloud Top NUCAPS Quality Day Radiance Type Flags Channel Number Hour Ascending/Descending flag Scan Line Number Channel Radiance Minute Field of Regard Scan-Level Quality Flags Type of Band Second Field of View Location of Platform Orbit Number Starting Wavenumber (per band) Ending Wavenumber (per band) 40
Product Requirements: ATMS Radiances · Requirement: The Reformatting Toolkit shall tailor the NPOESS ATMS Radiances from Net. CDF 4 into BUFR for EMC. » The ATMS BUFR file shall contain the TDR (Antenna Temperatures), the associated Quality Flags, and the Geolocation data at native resolution (not resampled) data. » The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is current provided for AMSU and MHS. » BUFR messages shall be smaller than 50 KB. » The file shall contain the following data fields (see table next slide): 41
Product Requirements: ATMS Radiances Variable Name Satellite ID Latitude Bandwidths ID of Originating Center Satellite Instrument Longitude Antenna Temperatures ATMS Quality Flags 123 Satellite Zenith Angle Satellite Classification Satellite Azimuth Year Solar Zenith Month Solar Azimuth Day Scan Line Number Hour Field of View Minute ATMS Channels Second ATMS Central Frequencies Polarization Location of Platform 42
Product Requirements: OMPS Ozone · Requirement: The Reformatting Toolkit shall tailor NPOESS OMPS Ozone products from Net. CDF 4 into BUFR for EMC. » The product shall contain OMPS Nadir Profile and Total Column (this would be the version 8 ozone algorithm for both products). » The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV. » BUFR messages shall be smaller than 50 KB. Note: The project originally planned for OMPS radiances to be included. John Derber (EMC) says that they are not needed. 43
Product Requirements: VIIRS SST · Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS SST products from Net. CDF 4 into BUFR for EMC. » Product shall contain Skin SST, Bulk SST, Quality Flags, Cloud Mask, and geolocation data. » Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS SST product. » The VIIRS SST BUFR table shall be derived from that currently being used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans). » BUFR messages shall be smaller than 50 KB. 44
Product Requirements: VIIRS Radiances · Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Radiances from net. CDF 4 into BUFR for EMC. » The product shall contain VIIRS radiances for 22 channels with associated quality flags and geolocation. Coverage shall be global. » The product shall contain the land cloud mask if it doesn’t take too long for the IDPS to generate those EDRs. » Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS radiance product. This table shall be derived from that currently being used for the GAC AVHRR. » BUFR messages shall be smaller than 50 KB. 45
Product Requirements: Aerosol Optical Thickness · Requirement: The Reformatting Toolkit shall tailor NPOESS Aerosol Optical Thickness (AOT) from net. CDF 4 into BUFR for EMC. » The product shall contain the AOT, wavelength of AOT, and Aerosol Size. » Reformatting Toolkit developers shall work with EMC to develop the AOT BUFR table based on what has already been done for MODIS. » BUFR messages shall be smaller than 50 KB. 46
Product Requirements: VIIRS Snow Cover · Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Snow Cover products from net. CDF 4 to GRIB 2 format for EMC. · After discussions with Ken Mitchell and Michael Ek, it was determined that EMC only wants NPOESS snow products that have been passed through the IMS (Interactive Multisensor Snow and Ice Mapping System). · According to Tom Schott the Reformatting Toolkit project is not currently funded to tailor products for IMS. This is anticipated in the future although IMS may want a format other than GRIB 2. · The Reformatting Toolkit development team recommends that this requirement be removed from the current phase of this project. 47
Product Requirements: Vegetation Index · Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Vegetation Index products from net. CDF 4 to GRIB 2 format for EMC. · After discussions with Ken Mitchell and Michael Ek, it was determined that EMC only wants a Green Vegetation Fraction that is like that currently being produced from AVHRR (Le Jiang, Felix Kogan). · The NPOESS VIIRS Vegetation EDR does not have GVF was present in the VIIRS Surface Type EDR, but was removed in the last version of the Common Data Format Control Books (Feb 2009). According to Andy Heidinger, that algorithm was based on the MODIS vegetation algorithms anyway. · The Reformatting Toolkit development team recommends that this requirement be removed for the current phase of this project. 48
Interface Requirements · Requirement: The Reformatting Toolkit shall receive thinned Cr. IS radiances (~300 channels) files from NUCAPS as an input for generating the Cr. IS radiance BUFR files. These files shall be in Net. CDF 4 and shall already contain the thinned SDR and geolocation information. · Requirement: The Reformatting Toolkit shall receive the NPOESS ATMS TDR files and associated Geolocation files tailored into Net. CDF 4 as an input for generating the ATMS radiance BUFR files. · Requirement: The Reformatting Toolkit shall receive the NPOESS OMPS Total Column Ozone EDR files and OMPS Nadir Profile IP files tailored into Net. CDF 4 as an input for generating the OMPS Ozone BUFR files. 49
Interface Requirements · Requirement: The Reformatting Toolkit shall receive the NPOESS VIIRS radiance SDR files and associated Geolocation files tailored into Net. CDF 4 as an input for generating the VIIRS radiance BUFR files. » Requirement: The Reformatting Toolkit shall receive the NPOESS VIIRS IP Cloud Mask Product to obtain the cloud and land mask for the VIIRS radiance BUFR files. · Requirement: The Reformatting Toolkit shall receive the NPOESS Aerosol Optical Thickness EDR files and associated Geolocation files tailored into Net. CDF 4 as an input for generating the Aerosol Optical Thickness EDR BUFR files. · Requirement: The Reformatting Toolkit shall receive the NPOESS SST EDR files and associated Geolocation files tailored into net. CDF 4 as an input for generating the SST BUFR files. 50
Requirements Summary · The Reformatting Toolkit developers have met with NDE and all the users of the original Phase I NDE tailored products. · All Reformatting Toolkit project, functional, and product requirements have been captured and documented in the Reformatting Toolkit RAD. · As development continues the detailed product requirements shall continue to be refined. · NDVI and Snow Cover are recommended to be waived from the phase 1 requirements. 51
Review Outline · · · Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 52
Software Architecture Presented by Thomas King PSGS 53
Reformatting Toolkit Architecture · Hardware Environment » » · Data Files » » · Development and Unit testing Test Product Distribution System Testing and Integration Production Input Files Static/Ancillary Files Output Files Run Files Software Description » External Interfaces » System Level » Unit Level 54
Reformatting Toolkit Development Hardware · Reformatting Toolkit Development Hardware - Unit tests, and Cr. IS/ATMS simulation data generation will be conducted on the IASI development machine at NSOF. This machine is maintained under the ESPC maintenance contract. » » » IBM P 570 (AIX 5. 3) 7 TB disk space 16 CPU 2 GB memory/CPU IBM XL Version 7. 0 C/C++ IBM XL Version 10. 1 Fortran 77/90 55
Reformatting Toolkit Development Hardware · Additional Development Hardware - Reformatting Toolkit testing may be conducted at STAR on AIX. This machine was purchased with ground systems money. It will be located at NCWCP, and maintained by STAR IT. » » IBM P 570 3 TB disk space 16 CPU 2 GB memory/CPU · Configuration management of Reformatting Toolkit code will be conducted in the STAR Collaborative Environment on STAR Linux hardware running IBM Clear Case. 56
Reformatting Toolkit Test Product Distribution · STAR Data Server - Reformatting Toolkit test products will be available on a distribution server at STAR (ftp 2. orbit. nesdis. noaa. gov). » » Linux 3. 2 TB disk space Access via anonymous ftp Products available on a near real time basis · This server will make available test BUFR and GRIB 2 files containing simulated data, BUFR tables, and any additional resources. 57
Reformatting Toolkit Integration and Production Hardware · NDE SADIE – The Reformatting Toolkit system testing and integration will be conducted on the SADIE integration platform working with NDE integration personnel. This hardware is located at NSOF and is maintained under the ESPC maintenance contract. » » » IBM P 561 (AIX 5. 3) 50 TB disk space 16 CPUs 2 GB/CPU IBM XL Version 9. 0 C/C++ IBM XL Version 11. 1 Fortran 77/90 58
Reformatting Toolkit Integration and Production Hardware · NDE Test/Production Hardware - After successful system tests, NDE will check the code into configuration management and then the code will be promoted to the Test/Production machine by NDE system integration staff. This machine will be located at NSOF. It has not yet been acquired. » » IBM P 570 (AIX 5. 3) TBD Disk space TBD CPUs TBD GB/CPU 59
Reformatting Toolkit Data · The tables on the following slides show all the Reformatting Toolkit input and output. The input data files are those required by the Reformatting Toolkit software to produce the Phase I tailored products. · All input data must be net. CDF 4 format. · Ancillary/Static files, such as the BUFR tables are listed. » They will be delivered as part of the DAP » Described more thoroughly in the CDR · All output will be in either BUFR or GRIB 2 format. 60
Reformatting Toolkit Input Data Files File Format Source Description Purpose Cr. IS Thinned SDR Radiances net. CDF 4 NUCAPS The NUCAPS Cr. IS thinned ~300 channel radiances for all FOVs. Cr. IS BUFR ATMS TDR net. CDF 4 NDE NPOESS ATMS full resolution file tailored into net. CDF 4 from the NPOESS HDF 5. ATMS BUFR ATMS TDR Geo net. CDF 4 NDE NPOESS ATMS Geolocation file that is associated with the ATMS TDR. ATMS BUFR VIIRS M-Band SDR net. CDF 4 NDE NPOESS VIIRS Moderate resolution band SDR. VIIRS BUFR VIIRS I-Band SDR net. CDF 4 NDE NPOESS VIIRS Imagery resolution band SDR VIIRS BUFR VIIRS DNB SDR net. CDF 4 NDE NPOESS VIIRS Day/Night Band SDR. VIIRS BUFR VIIRS M-Band SDR Geo net. CDF 4 NDE VIIRS Moderate resolution band SDR Geo file. VIIRS BUFR VIIRS I-Band SDR Geo net. CDF 4 NDE VIIRS Imagery resolution band SDR Geo file. VIIRS BUFR VIIRS DNB SDR Geo net. CDF 4 NDE VIIRS Day/Night Band SDR Geo file. VIIRS BUFR VIIRS Cloud Mask IP net. CDF 4 NDE VIIRS granule files containing cloud data. VIIRS BUFR VIIRS Cloud Mask IP Geo net. CDF 4 NDE VIIRS granule files containing the geolocation information for the cloud product. VIIRS BUFR 61
Reformatting Toolkit Input Data Files File Format Source Description Purpose OMPS Nadir IP net. CDF 4 NDE NPOESS OMPS Nadir Profile ozone Intermediate Product file tailored into net. CDF 4. Geolocation information is currently in the file. NGAS documentation indicates that much of the content of this product is TBD. OMPS BUFR OMPS Total Column EDR net. CDF 4 NDE NPOESS OMPS Total Column ozone EDR file tailored into net. CDF 4. OMPS BUFR OMPS Total Column SDR Geo net. CDF 4 NDE NPOESS OMPS Total Column ozone SDR Geo file tailored into net. CDF 4. This same Geo file is the used with the OMPS Total Column EDR. OMPS BUFR Aerosol Optical Thickness EDR net. CDF 4 NDE NPOESS AOT EDR file tailored into net. CDF 4. AOT BUFR Aerosol Optical Thickness EDR Geo net. CDF 4 NDE NPOESS AOT EDR Geo file tailored into net. CDF 4. AOT BUFR Sea Surface Temperature EDR net. CDF 4 NDE NPOESS SST EDR file tailored into net. CDF 4. SST BUFR Sea Surface Temperature EDR Geo net. CDF 4 NDE NPOESS Moderate Resolution Terrain-Corrected Geolocation file tailored into net. CDF 4. This is to be used with the NPOESS SST EDR. SST BUFR 62
Reformatting Toolkit Ancillary/Static Data Files File Format Source Description Purpose Cr. IS SDR BUFR Table ASCII N 4 RT Cr. IS thinned radiances (~300 channels, all FOVs) BUFR Template ATMS SDR BUFR Table ASCII N 4 RT ATMS radiances full resolution BUFR Template VIIRS SDR BUFR Table ASCII N 4 RT VIIRS radiances full resolution BUFR Template OMPS EDR BUFR Table ASCII N 4 RT OMPS Nadir Profile and Total Column Ozone BUFR Template AOT EDR BUFR Table ASCII N 4 RT Aerosol Optical Thickness and Particle Size BUFR Template SST EDR BUFR Table ASCII N 4 RT Sea Surface Temperatures, Cloud Mask BUFR Template 63
Reformatting Toolkit Output Data Files File Format Description Users Cr. IS SDR BUFR Cr. IS thinned radiances (~300 channels, all FOVs) JCSDA, NCEP, EUMETSAT, UKMet, ECMWF ATMS SDR BUFR ATMS radiances full resolution JCSDA, NCEP VIIRS SDR BUFR VIIRS radiances full resolution JCSDA, NCEP OMPS EDR BUFR OMPS Nadir Profile and Total Column Ozone JCSDA, NCEP AOT EDR BUFR Aerosol Optical Thickness and Particle Size JCSDA, NCEP SST EDR BUFR Sea Surface Temperatures, Cloud Mask JCSDA, NCEP 64
Reformatting Toolkit Run Data Files File Format Source Description Purpose PCF ASCII NUCAPS The PCF file containing all the required input parameters for the N 4 RT driver script. Driver input N 4 RT Run Log ASCII N 4 RT The N 4 RT log file containing all the standard error and standard output from a given run. Diagnostic N 4 RT Main Resource ASCII N 4 RT The internal control file required by the N 4 RT executable main program. Executable input PSF ASCII N 4 RT The Process Status File containing only the successfully generated output files. Driver output list 65
Reformatting Toolkit Software: External Interfaces · The Reformatting Toolkit system will be run via the execution of a single driver script that will be invoked, monitored, and managed by the NDE DHS Product Generation Manager. · Execution of the script will be event-driven (i. e. the arrival of a required input file whose type is predefined in the Reformatting Toolkit production rules given to NDE). · An invocation of Reformatting Toolkit by the NDE DHS will be used to do a single granule-to-granule file translation (as opposed to reformatting many files as once). · NDE will run the script in a working directory. All Reformatting Toolkit output will be produced in this same working directory. · The driver script will require an input PCF file containing parameters such as: » » » The input and output file names Input and output product type IDs (e. g. ATMS radiances) The type of conversion required (e. g. net. CDF 4 to BUFR) 66
Reformatting Toolkit Software: External Interfaces · The driver script will run, parse the PCF content, run the compiled conversion code, handle program output and errors, direct required NDE error codes to the DHS, generate an output log, and generate a PSF. · If there are errors, NDE will save the contents of the run in a forensics repository. · NDE will manage and direct error status to the operators from the DHS system. · NDE will manage all distribution through the NDE DDS and the short term storage of tailored data on the SAN. 67
Reformatting Toolkit External Interfaces to NDE Reformatting Toolkit External Interfaces NDE DHS Boundary Systems Configurations Product Generation Specifications Process Req. NDE Product Generation Manager Invocation Return Code Reformatting Toolkit Driver Script Rule Sets Output Files & PSF Working Directory Output DAP Specifications Forensics Repository PSF (N 4 RT output) Working Directory PCF (N 4 RT input) BUFR & GRIB 2 Output Files Input Files (net. CDF 4) Data Areas Configurations Info N 4 RT System NDE Production Manager Input Files & PCF Input Files (net. CDF 4) SAN NDE DDS 68
Reformatting Toolkit Software: System Level · The Reformatting Toolkit driver script will be a single Perl script that acts as a wrapper for the compiled Reformatting Toolkit code. » There will be no hard coded paths in the script. All needed information regarding locations of files will come through the PCF. » All system calls have their return values checked so the exits are graceful and informative. » All standard out and standard error will be directed to a single log file. » The driver script will translate the low-level program errors into the high -level numerical error codes expected by the PGM. » The Perl script will generate an internal control file for the main Reformatting Toolkit program. 69
Reformatting Toolkit Software: System Level · The PCF content (nominal mode): » » » Input net. CDF 4 file Output BUFR file Output GRIB 2 file Direction of conversion (eg. NC to BF) Input and output product types Input BUFR table (if BUFR is an output) · The PCF content (test mode): » » » Input BUFR or input GRIB 2 Output net. CDF 4 file Direction of conversion (e. g. BF to NC) Input and output product types NC templates if (net. CDF 4 is an output type) 70
Reformatting Toolkit System Level Data Flow Execution from PGM PCF Return Value to PGM N 4 RT Driver Script BUFR/GRIB 2 net. CDF 4 NC Template Resource BUFR table N 4 RT Main Converter net. CDF 4 Nominal Mode Test Mode Working directory N 4 RT log BUFR file GRIB 2 file Working directory PSF Working directory 71
Reformatting Toolkit Software: Unit Level · The Reformatting Toolkit main program will be a single main C++ program containing all the predefined data structures required by the code. Lowerlevels will be in Fortran 90. Why this approach? » C++ main program will give the system flexibility for having to plug into future types of architectures. » APIs for BUFR are only Fortran 77. » Much of the code being reused is already in Fortran 90. » Many users tend to request readers in Fortran 90. · In the Reformatting Toolkit main program, all data structures will be declared and a series of cases statements will direct the program flow based on the type of input files and the direction of conversion. · The Reformatting Toolkit subroutines at the next level down (and at all subsequent levels) will be in Fortran 90. This next level will manage: » Allocation, initialization, deallocation of data structures. » They are designed for overseeing specific product definitions and conversions. 72
Reformatting Toolkit Software: Unit Level · The lowest-level Reformatting Toolkit subroutines: » Perform the actual reading and writing of the specific file types. » Directly call the library APIs for BUFR, GRIB 2, and net. CDF 4. · General Reformatting Toolkit compiled code characteristics: » The status of all functions are checked to allow for graceful and informative exits. » No paths are hard coded in the compiled code. » The are no system calls from within the compiled code. » All code will be compiled as 64 bit to utilize the IBM architecture. 73
Reformatting Toolkit Unit Level Data Flow Reformatting Toolkit UNIT Level Data Flow N 4 RT resource N 4 RT log file N 4 RT Main Prod N - NC 2 BF Prod N - NC 2 GB Prod N - BF 2 NC Prod N - GB 2 NC Allocate Initialize Prod N - Read NC Prod N - Read BF Prod N - Read GB Prod N - Write BF Prod N - Write GB Prod N – Write NC Prod N - Write NC Deallocate Nominal Mode Test Mode Prod N - Read NC Deallocate BUFR Deallocate GRIB 2 Net. CDF 4 Deallocate Net. CDF 4 74
Reformatting Toolkit Software: Libraries · BUFR Library » NCEP BUFRLIB library available on the NCEP website » Code is all Fortran 77 » Compiled as 64 bit · GRIB 2 Library » Available on the NCEP website » Compiled as 64 bit · net. CDF 4 Library version 4. 0 » Available from Unidata website » Compiled as 64 bit · Reformatting Toolkit developers will coordinate with NDE and the product teams to make sure that all parties are using compatible libraries. 75
Software Architecture Summary · Reformatting Toolkit software development, testing, and operation will be conducted on comparable IBM hardware at NSOF and STAR. · The Reformatting Toolkit code will run as a stand-alone unit within the NDE DHS. · The code will be modular to allow for easy reuse of code and expansion to accommodate new products. · The code will use only the official releases and NDE compatible versions of the net. CDF 4, BUFR, and GRIB 2 libraries. 76
Review Outline · · · Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 77
Quality Assurance Presented by Thomas King PSGS 78
Quality Assurance Background · STAR is using the Capability Maturity Model Integrated (CMMI) to improve processes and practices for development and the transfer of research to operations. · The STAR Enterprise Life Cycle (EPL) merges the CMMI and SPSRB standards. · IASI and NUCAPS are both pathfinder projects (CMMI Level 3). The IASI system has been transitioned to operations successfully. · The Reformatting Toolkit algorithm development will follow the same process but tailored to the scale of the project. 79
Quality Assurance – Project Implemented Tailored STAR EPL Reviews · The Reformatting Toolkit Preliminary Design Review (April 2009) » » · The Reformatting Toolkit Critical Design Review (September 2009) » · Will present the unit test plan to demonstrate that the toolkit is ready to be run in the Test Environment. A code unit test review (August 2010) » · To finalize requirements and to verify that the chosen design is able to meet those requirements. An Reformatting Toolkit Test Readiness Review (June 2010) » · Will present the initial draft of the requirements and discuss a proposed design. An Reformatting Toolkit Requirements Allocation Document (RAD) has been made available to Reformatting Toolkit stakeholders. It will be updated throughout the lifecycle of the project. Will be conducted to ensure that the Reformatting Toolkit software is able to fulfill the functional software requirements. The Reformatting Toolkit System Read Review (November 2010) » Has requested to be waived because Reformatting Toolkit will be run within the 80 NDE system.
Configuration Management (CM) · CM Tool (IBM Rational Clear. Case, Version 7. 0 ) » Has been purchased and implemented in the Collaborative Environment. · CM personnel have been identified. · CM training: » Administrator training completed. » Developers will be trained by the CM administrator. · Detailed CM Plan is under development. 81
STAR Coding Standards · Coding standards guidelines and quick references are available. · Provide a common list of abbreviations. · Adhere to the standards throughout the development life cycle. · Have checklists available for developers to keep track of the delivery status of the code. 82
Quality Assurance – Software · The Reformatting Toolkit software will be delivered incrementally as part of the series of prototype DAPs following the NDE Build Content Reviews (BCR). This will allow system testing of the code within the NDE DHS. The BCR schedule is as follows: » » » » Feb 17 – April 17 (Initial integration) April 20 – June 19 (Throttle and Load Balance) June 22 – July 17 (Generate Hourly, Daily Products) July 20 – August 14 (Generate Tailored and Regional Products) August 17 – September 18 (Complex Production Rules) September 21 – October 16 (Weather Event Based Production) October 19 – November 14 (Production Monitoring) 83
Quality Assurance – Software · All code development is being conducted on a platform that is nearly identical to the test and production target platforms using the same compilers and operating system. · Only the official releases of the NCEP BUFRLIB, GRIB 2, and Net. CDF 4 libraries will be used in the software. · STAR code checking tools will be used to minimize coding bugs and to ensure that Reformatting Toolkit code meets STAR coding standards. · The status of all system calls and intrinsic functions are checked. · Unit tests will have the Reformatting Toolkit generate BUFR and GRIB 2 products from Net. CDF 4 files. These BUFR and GRIB 2 files will be directed back into the Reformatting Toolkit to generate new net. CDF 4 files. » » The contents of the new and original net. CDF 4 files will be compared to demonstrate that went in matches with what came back out. Customers will have access to test data products to verify that values appear reasonable and that precision is not being lost in the format conversions. 84
Quality Assurance – Software · An official (tailored) DAP will be delivered: » » » All Reformatting Toolkit code and system files Test plans Test data sets Error messaging/handling PCF format Production rules Product file specifications Data flow diagrams Estimates of resource usage ATBD has been waived Delivery memo 85
Quality Assurance – Products · Reformatting Toolkit developers will work with EMC, NRL, FNMOC, GMAO, EUMETSAT, and UK Met users to ensure that the proper BUFR descriptors are used in the BUFR tables and that, whenever possible, they are WMO approved (as opposed to local). » Chosen descriptors must work with NCEP BUFRLIB. · Reformatting Toolkit developers will work with EMC, heritage product developers, and NPOESS operational algorithm teams to ensure consistency with heritage products with respect to format and content. · Reformatting Toolkit developers will make BUFR tables and test products (BUFR and GRIB 2) available to users before the operational products are made available. This will allow for preliminary product content validation. » Cr. IS and ATMS are currently being simulated at STAR. » For the other products such as VIIRS, we are working with the IPO (Joe Zajic), the NPP Test Data Working Group (TDWG), and NDE to obtain simulated products. 86
Quality Assurance – Archive and Maintenance · Archive Plan » Reformatting toolkit will be integrated into NDE system and made available to CLASS by NDE » Product archive requirements are addressed within product development projects – NPOESS program office works with CLASS to archive x. DRs – NOAA Unique Product (NUP) projects work with CLASS as required · Long Term Maintenance Plan » The reformatting toolkit will be maintained by the OSDPD staff » STAR system developers will be available 87
Quality Assurance – Documentation and Metadata · Documentation/Metadata Plan » The Documentation will include the SPRSB documents and additional CMMI documents » No metadata is required for these products 88
Quality Assurance Summary · Quality assurance plan will consist of: » Project reviews at which stakeholders are encouraged to participate. » Ongoing interaction with customers, heritage product developers, operations, NDE, and the SPSRB. » Adhering to STAR/NDE software standards and use of standard libraries only. » Software unit tests shall be presented in the TRR. » Documentation of the Reformatting Toolkit code operation, production rules, and software tests will be in the DAP. » Documentation of requirements will be in the Reformatting Toolkit RAD. » Early release of BUFR and GRIB 2 products, tables, and additional resources will allow for customer validation of products. 89
Review Outline · · · Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 90
Risks and Actions Presented by Walter Wolf NOAA/NESDIS/STAR 91
Risk and Actions · Risk 1: NPOESS product formats and content are still changing, especially for VIIRS. · Risk Assessment: Low · Impact: Changes in the NPOESS format may cause delays in the development of the output product files. Likelihood: High · · Mitigation: » Work through the NPOESS Data Format Working Group to obtain information on format and algorithm updates. » Monitor the latest copies of the NPOESS Common Data Format Control Books (CDFCB) for any updates. » Maintain contact with customers to inform them of any upstream product changes. » Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision. · Status: Open 92
Risk and Actions · Risk 2: The roles and responsibilities regarding who shall generate the set of required SPSRB documents for NDE has not yet been decided. · Risk Assessment: Low · Impact: Documentation that has not been planned for may cause delays and or cost overruns · Likelihood: Moderate · Mitigation: » This issue as well as decisions regarding the content of the SPSRB documents is being worked by Ken Jensen and STAR, NDE, OSD, and OSDPD personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions. · Status: Open 93
Risk and Actions · Risk 3: There are small variations in the types of platforms and the versions of the compilers. · Risk Assessment: Low · Impact: Compiler differences may cause errors in the output files. · Likelihood: Low · Mitigation: » Reformatting Toolkit developers will work with NDE during system tests in the integration and production phase to ensure that those results match the results from the units. The BCR process (mentioned earlier) should help to resolve these issues early on in development. · Status: Open 94
Risk and Actions Summary · There are currently 3 risks identified for the Reformatting Toolkit project. · The severity of all 3 risks is low. · These risks remain open at this time. 95
Review Outline · · · Introduction Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions 96
Summary and Conclusions Presented by Walter Wolf NOAA/NESDIS/STAR 97
Review Objectives Have Been Addressed · The Project Mission and Schedule has been reviewed. · The Project Requirements have been reviewed » Recommend waiving the NDVI and Snow Cover products. · Software architecture, hardware, data, and interfaces have been reviewed. · Plans for quality assurance have been reviewed. · Risks and Actions have been reviewed. 98
Current Status · A Cr. IS BUFR table has already been created working with Yi Song, Dennis Keyser, Jeff Ator, Jim Jung, Simon Elliott (EUMETSAT), and Nigel Atkinson (UKMet). · A test Cr. IS BUFR file was created from this table using simulated data and was delivered to EMC (mid-March 09). · The new Cr. IS BUFR table descriptors and test data file have been submitted to WMO by Jeff Ator (early April 09). · Test Cr. IS BUFR file has been made available to NCEP, NRL, FNMOC, and the global NWP centers · Work on the ATMS BUFR table is currently ongoing. · Reformatting Toolkit is currently working with Joe Zajic (IPO) to obtain access to simulated VIIRS products at NSOF. 99
Next Steps · Code Development Phase » ATMS BUFR coding and test file generation » VIIRS, OMPS, SST, and AOT BUFR table development » Obtain VIIRS test data » Participate in the NDE BCR process · Critical Design Review (September 2009) 100
Open Discussion · The review is now open for free discussion 101
Скачать презентацию Net CDF 4 Reformatting Toolkit BUFR and GRIB
5bd0c1d5b338d59252eec2849975cfc2.ppt