Atmosphere Agenda • Science Team – TBS I&TSE SD 3 E Ozone Sounder • Atmosphere PEATE – TBS Atmosphere Land Ocean NICSE PSOE 1
NPP Science Team Counterparts for Atmosphere PEATE Bryan Baum (UW): VIIRS Cloud Retrievals Christina Hsu (GSFC): VIIRS Aerosol Retrievals Hank Revercomb (SSEC): SDR Validation Omar Torres (UMBC): Aerosol Validation Paul Menzel (UW): VIIRS and heritage CDRs 2
Atmosphere PEATE Organization Project Management Hank Revercomb (PI) Liam Gumley (Co-I, PM) Algorithms & Validation Bob Holz Richard Frey Bryan Baum Paolo Antonelli Andy Heidinger Mike Pavolonis Dave Tobin Computing Systems Scott Mindock Steve Dutcher Bruce Flynn Rick Jensen Operations Jerry Robaidek Rosie Spangler Dee Wade 3
NPP Environmental Data Records (EDRs) Land PEATE 1. Albedo (Surface) 2. Land Surface Temperature 3. Snow Cover and Depth 4. Surface Type 5. Active Fires (ARP) 6. Ice Surface Temperature 7. Vegetation Index 8. Aerosol Optical Thickness 9. Aerosol Particle Size Ocean PEATE 10. Ocean Color/Chlorophyll 11. Sea Surface Temperature Ozone PEATE 12. Ozone Total Column/Profile Atmosphere PEATE 13. Suspended Matter 14. Cloud Cover/Layers 15. Cloud Effective Particle Size 16. Cloud Top Height 17. Cloud Top Pressure 18. Cloud Top Temperature 19. Cloud Base Height 20. Cloud Optical Thickness Sounder PEATE 21. Atm Vertical Moisture Profile 22. Atm Vertical Temp. Profile 23. Atm Vertical Pressure Profile VIIRS Intermediate Products Cloud Mask 4
Atmosphere Agenda • Science Team – TBS I&TSE SD 3 E Ozone Sounder • Atmosphere PEATE – TBS Atmosphere Land Ocean NICSE PSOE 5
Atmosphere PEATE Design ● ● The NPP Atmosphere PEATE will be implemented within the framework and facilities of the SSEC TBR This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions. TBR 6
Element Overview • Acquire VIIRS RDRs, SDRs, and Atmosphere EDRs from the SD 3 E and ADS/CLASS • Assess the quality of the NPP Atmosphere EDRs for accomplishing NASA’s climate research requirements TBR • Provide suggested algorithm improvements to the IDPS 7
Changes since PDR • TBR 8
Atmosphere PEATE Interface Diagram SD 3 E CLASS (ADS) Ancillary Data Providers NICSE PSOE Analysis Results, Proposed Algorithm Updates x. DRs, IPs, Ancillary Data x. DR Anc s, IPs, illar y Da (if u ta nav ailab SD 3 le from E) Ancillary Data Casa. NOSA Management Direction ht -flig , ms rith lgo , Info A ta Da Pre Ocean PEATE Software, Data x. DR Eval. Results, Algorithm Updates Inte ract Calibration Updates and Evaluations In Situ Data ion In Situ Data I&TSE Algorithm Updates, Test Requests & Results VOST Ocean Science Community Matchups Sea. BASS RSMAS 9
External Interfaces: SD 3 E Messaging: Any request or report requiring email interaction will be handled by SSEC Data Center (DC) Staff (dcstaff@ssec. wisc. edu). Problem reports, system status notices, subscription requests, and transfer errors all fall into this category. DC Staff are available 0730 - 2300 Central Mon-Fri. DC will escalate issues to the PEATE team only if DC can’t solve them. File Transfers: 1. Pull transfers: DC will inaugurate and monitor any regularly scheduled downloads from the SD 3 E 32 -day data store. These downloads will be handled by the PEATE Ingest subsystem (see Part II of this presentation). 2. Push transfers: DC will request and monitor any subscriptions which have been established with the SD 3 E. 3. Checksums and digital signatures generated at SD 3 E will be automatically ingested and verified. Requests for retransmits will be routed through DC staff. The PEATE Ingest subsystem will also generate in-house checksums based on MD 5. 4. SD 3 E file naming conventions for be maintained. 5. Files will be compressed internally using bzip 2 or similar. 10
External Interfaces 11
Network Performance: NASA GSFC to SSEC Source: Andy Germain <Andy. Germain@gsfc. nasa. gov> 12
External Interfaces: NPP Science Team Product Evaluation and Algorithm Testing/Improvement The Atmosphere PEATE will provide a development server for use by the NPP Science Team Atmosphere subgroup for interactive product evaluation (e. g. , Matlab, IDL, ENVI) and testing of improved EDR algorithms (C, C++, FORTRAN). The system will also provide source code version management (SVN) and a modest online disk archive (<10 TB) for algorithm testing. EDR Product Generation The Atmosphere PEATE Science Processing System is currently envisioned as a “super-user” system. This means NPP Science Team members will deliver compiled EDR code to the system, where a system manager will run the code on the dataset requested by the investigator. Products will be made available on a PEATE FTP site. Data Search and Order The Atmosphere PEATE will provide a web-based interface for the NPP Science Team to search current data holdings and order files for FTP push or pull. Products not available online will be recreated as necessary. 13
Atmosphere PEATE Interfaces • Bullets for previous slide 14
Atmosphere PEATE: Success Criteria 1. Provide environment for pre-launch testing and evaluation of operational atmosphere EDR algorithms 2. Allow rapid post-launch evaluation and comparison of NPP atmosphere EDRs 3. Create infrastructure using available validation data to allow rapid assessment of NPP EDR products 4. Assist NPP Science Team in assessing the suitability of NPP atmosphere EDRs for continuing the climate record of cloud observations from space 5. Provide environment where NPP Science Team can test alternative EDR algorithms on climatologically significant samples of global proxy data 15
Atmosphere PEATE Science Processing System • Assess Cloud EDRs for their ability to support the system long term data trending • Enable creation of consistent long-term cloud property datasets for EDR evaluation • To evaluate climate quality of atmosphere EDRs, must use a consistent version of the calibration and science algorithms for a long-term dataset (e. g. , one month, one year, entire mission). • EDRs must be created rapidly in order for Science Team to give timely feedback to NPP project on algorithm performance. • NPP Science Team members do not have the individual resources to host large datasets, integrate operational NPP algorithms, and test improved/alternative algorithms. • The NPP cloud products must be put into context with historical and ongoing global cloud property datasets (e. g. , PATMOS-X, UW-HIRS, MODIS Terra/Aqua, ) to create self-consistent climate data records (CDRs). 16
Processing System Trade Studies and Key Decisions Examined NASA Ocean SDPS and MODIS Land Processing Systems. Lessons learned: • Recipe-based approach to running science algorithms (system doesn’t care what the algorithm is, as long as it knows how to assemble the ingredients to make the recipe) • Cluster of compute resources (no need for a large shared memory computer) • Decouple the components of the processing system (store, compute, distribute) • Use commodity hardware/software (e. g. , Rackmount Intel/AMD servers, Linux) Key Design Decisions: • Create a system where individual components have loose dependencies on each other. • Leverage existing cluster processing hardware infrastructure and knowledge base. • Create a system which is scalable, efficient, and cost effective. 17
Need Logical Design 1 st 18
Atmosphere PEATE Science Processing System (APSPS) 19
APSPS, Atmosphere PEATE Science Processing System DMS: Data Management System CRG: Computational Resource Grid ARM: Algorithm Rule Manage ING : Ingest System Stores Data Processes Data Applies Product Rules to Data Brings Data into System 20
Science Processing System Preliminary Design Ingest Data Manage Processing Process Data 21
Science Processing System: Lines of Code Why so small? 1. Java libraries 2. Open Source code contribtuions are not counted (e. g. , AXIS Web Services, Hibernate, Tomcat, Log 4 J) 22
Algorithm Lifecycle Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM. 23
Algorithm Ingest Algorithm entered into subversion Product created in bugzilla Algorithm is ported and wrapped Tests are created 24
Algorithm Qualification Write a script to execute algorithm Script manages execution environment Algorithm name, inputs and outputs entered into ARM 25
LEOCAT: Low Earth Orbit Cloud Algorithm Testbed LEOCAT History • Developed by Mike Pavolonis at UW under VIIRS IGS funding to investigate differences in the operational VIIRS cloud algorithms and heritage algorithms in a manner that isolates algorithmic differences. • The best way to accomplish this is to apply each algorithm to the same Level 1 B and ancillary data sets using the same radiative transfer model. • A secondary use of LEOCAT is to serve as an algorithm development tool and global EDR processing system. • LEOCAT approach is also being used for GOES-R AWG work (GEOCAT). LEOCAT Features • Handles multiple imaging sensors (e. g. , MODIS, VIIRS, AVHRR) • Multiple algorithms with the same and/or different parameters can be executed in one instance • Allows for the addition of new algorithms with minimal programming effort (can be added as shared libraries) • Produces HDF 4 output • Can use CRTM or PLOD forward models • Can use GFS, NCEP, or ECMWF ancillary data • Optimized for efficiency • Process a single or multiple granules in one instance 26
LEOCAT Architecture LEOCAT Core Science Algorithm 27
EDR Evaluation Demonstration Goal: To demonstrate the workflow necessary to evaluate a VIIRS atmosphere EDR for climate product quality. Proxy Data: Aqua MODIS is the best available spectral simulation of VIIRS. Products to be Compared: • VIIRS OPS Cloud Mask (versions 1. 3 and 1. 4) • MODIS operational cloud mask (collection 5) • MODIS operational cloud mask with VIIRS bands only Components of Demonstration 1. Obtain Products (from archive, or generate from scratch) 2. Run Quality Control process on each product 3. Intercompare products (internally) 4. Validate products (using external data) 28
Demonstration Work Plan 1. Process one global month of Aqua MODIS proxy data (day/night), starting with Level 1 A data (RDR). 2. Run MODISL 1 DB algorithms for geolocation and calibration. 3. Run DAAC operational algorithm for Level 1 B destriping. 4. Run Cloud Mask algorithms. 5. Examine quality of each individual product (e. g. , algorithm success/failure and retrieval yield statistics; processing summaries (per granule, per day); granule based images; clear radiance composites (daily, 8 -day, monthly). 6. Intercompare the products (compute and map differences in clear sky radiance statistics for final retrieval and intermediate spectral tests). 7. Validate each product by collocating with CALIPSO lidar and comparing to CALIPSO cloud mask. Each step in the process must be straightforward and well documented. 29
MODIS Band 2 (0. 87 micron) 30
MODIS C 5 Cloud Mask 31
VIIRS OPS v 1. 4 Cloud Mask (Preliminary/Unverified) 32
VIIRS OPS v 1. 4 Cloud Mask (Preliminary/Unverified) VIIRS OPS v 1. 4 Cloud Mask (Verified) 33
EDR Evaluation: Satellite and Ground Measurements • Evaluate the effectiveness of proposed cloud algorithms and the resulting global cloud products generated from MODIS (proxy for VIIRS), AIRS (proxy for Cr. IS), Cloudsat, and CALIPSO • A subsequent test of algorithm robustness will be to apply the cloud algorithms to METOP data (AVHRR, HIRS, and IASI) for concurrent time period as A-Train data analyses • Build the capability to assess instrument issues, such as out-of-band response, channels that perform out of spec, detector striping, etc • When VIIRS is launched, it is unlikely that a space-based lidar/radar will be in operation and there will not be continuous coincident lidar/radar measurements with VIIRS • A combined satellite and ground measurement plan provides a comprehensive evaluation capability to assess the VIIRS products 34
EDR Evaluation Measurement Plan • The NASA A-Train measurement platform using MODIS as a proxy for VIIRS will provide: • a platform to compare the VIIRS algorithms directly with MODIS, CALIPSO and Cloud. Sat cloud retrievals (global) • a “baseline” for our global performance expectations for VIIRS • The assessment using ground measurements will provide wellcalibrated point measurements that will be available at VIIRS launch • The combined ground/satellite evaluation using MODIS will provide a measure of how representative the ground evaluation will be in determining the global performance of the VIIRS retrievals at launch 35
EDR Evaluation Goals • The PEATE will be designed to identify algorithm/instrument issues from the physical sensitivity differences between the evaluation and VIIRS products • The VIIRS science team will be enlisted to establish protocols so that cloud product inter-comparisons are performed similarly for all algorithms • The goal is to automate the product inter-comparison process • The evaluation results will be compiled for each VIIRS processing run using established protocols • Graphics (figures) and comparison statistics will be automatically generated for review allowing for instantaneous feedback on changes to the VIIRS algorithms • When new evaluation measurements/retrievals become available they can be easily integrated into the evaluation system • Well-documented evaluation protocols for each VIIRS product will be created 36
EDR Evaluation: Cloud Height Global Images (August 2006) 37
Pre-launch Evaluation Approaches, by EDR • Cloud mask (CALIPSO) • Aerosol/cloud discrimination (CALIPSO) • Aerosol layer height (CALIPSO) • Cloud layering (CALIPSO/Cloud. Sat) • Cloud top height (CALIPSO/Cloud. Sat) • Cloud base (CALIPSO/Cloud. Sat) • Cloud thermodynamic phase (CALIPSO) • Cloud optical thickness (for < ~3 from CALIPSO; for higher from Cloud. Sat) • Cloud particle radius (ongoing research: CALIPSO/Cloud. Sat) 38
Evaluation: Post NPP Launch Evaluation Flow Diagram Post NPP Launch 39
Atmosphere EDR Evaluation Summary At VIIRS launch, the Atmosphere PEATE EDR evaluation system will have the capability to: • Ingest and store global VIIRS RDRs, SDRs and atmosphere EDRs • Regenerate self-consistent SDR and EDR long-term datasets for evaluating climate quality of atmosphere EDRs • Ingest, process, and store the evaluation measurements (ground and satellite) • Collocate (space and time) the VIIRS SDRs and EDRs with the evaluation measurements (ground and satellite) • Produce quantitative comparisons between the VIIRS SDR/EDRs and the evaluation products (global, long-term) • Produce quick-look images of the VIIRS SDR/EDRs, evaluation, and collocated products • Distribute results to NPP Science Team 40
Atmosphere PEATE Development Schedule 41
Atmosphere PEATE Development Schedule 42
Atmosphere PEATE Development Schedule 43
Atmosphere PEATE Development Schedule 44
Atmosphere PEATE: Prototype Hardware (January 2007) Based on commodity hardware: Dual/Quad core servers, SATA RAID, Gigabit Ethernet. By NPP launch: 250 CPU cores, 215 TB disk. Aiming for 100 x processing rate for NPP Atmosphere EDRs. 50 CPU cores 40 TB disk 45
At-Launch PEATE Hardware: Notional Configuration Compute Resources: 250 CPU cores, AMD or Intel, 2 GB RAM per core. Estimate $250 K. Justification: 50 CPU cores yielded 50 x EDR generation rate in prototyping studies. We will have 10 Atmosphere EDRs from NPP, and we desire 100 x EDR generation rate. • 50 CPU = 4 EDRs at 50 x • 100 CPU = 4 EDRs at 100 x • 250 CPU = 10 EDRs at 100 x Storage: 215 Terabytes (TB). Estimate $200 K. Justification: Aqua MODIS L 0 volume is 18. 25 TB/year compressed (ref: Ocean SDPS). We desire complete Aqua MODIS L 0 archive online (7. 5 years), plus 3 years of space for NPP RDR (complete) and SDR+EDR (subset). Estimate NPP at 25 TB/year. • (18. 25 TB/year x 7. 5 years) + (25 TB/year x 3 years) = 215 TB. Networking: Cisco Catalyst Stackable Gigabit Switch Infrastructure (32 Gbps stack interconnect) with 4 x 48 ports. Estimate $50 K. 46
Atmosphere PEATE: Achievements to Date 1. Processed one month of global Aqua MODIS proxy data using MODIS, MODIS VIIRS-like, and VIIRS OPS cloud mask algorithms. 2. Collected Aqua MODIS global Level 1 A data since Jan. 2006. 3. Adapted existing software (LEOCAT) to create infrastructure for running VIIRS OPS EDR code on Linux (32, 64 -bit) and OS X. 4. Demonstrated 50 x processing rate on first generation computing system. 5. Created validation plan and demonstrated a validation approach for cloud mask using CALIPSO CALIOP. 6. Completed System Requirements Review for NASA. 7. Demonstrated process for evaluating a VIIRS EDR (Cloud Mask). 8. Completed successful Preliminary Design Review for NASA. 9. Started deployment and testing of Science Processing System. 47
Gap Analysis EDR Algorithm Qualification/Verification A major concern is the problems we have encountered in obtaining any meaningful test suites to verify that EDR algorithms are operating in the Atmosphere PEATE environment as intended by the algorithm designers. We must make a best guess as to what Gridded IPs and Ancillary datasets should be used as proxies in pre-launch testing. Others? 48
Скачать презентацию Atmosphere Agenda Science Team TBS I TSE
fc930d4f7786ff666d2193781719a1d9.ppt