GSICS Data and Products Servers Volker Gärtner EUMETSAT

GSICS Data and Products Servers Volker Gärtner (EUMETSAT) For GSICS Data Working Group

EUMETSAT GSICS Data and Products Servers GSICS Data Working Group (GDWG) V. Gaertner (Chair), Y. Hwang, A. Jelenak, J. Lafeuille, Y. Li, P. Miu, R. Nakayama, H. Owada, Z. Rong, H. Rothfuss CMA, EUMETSAT, JMA, KMA, NOAA/NESDIS, WMO GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 3

GDWG and the GSICS Data and Products Servers What is the GDWG? The GSICS Data Working Group (GDWG) is a body of the GSICS organisation which consists of data management experts from the participating GSICS Processing and Research Centres (GPRC’s). In close cooperation with the GSICS Research Working Group (GRWG) it advises the GSICS Executive Panel on the planning and implementation of data management activities, infrastructure to provide data, data formats, data retention etc. What are the GSICS Data and Products Servers? These are data management servers, to be implemented by several GPRC’s (CMA, JMA, NOAA, EUMETSAT), which follow a defined set of architecture principles and provide a set of services for the GDWG and the wider GSICS user community to support exchange and access to relevant data. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 4

GSICS Data and Products Servers Concept EUMETSAT took the lead in the development of this server as a result of an agreed GSICS Action (Ref: GDWG-II WG 2/01) from the annual Joint Meeting of the GSICS Research and Data Working Groups. The purpose of the server is to provide a platform where the GRWG can access and exchange comparable source data sets for their research and upload resultant products back to the server for validation, publishing and/or archiving. To facilitate data handling, the net. CDF format was chosen for both the comparable data sets and the resultant products the WMO Global Telecommunication System (GTS) filename convention was selected Once the development has been finally approved, it is envisaged that federation of these servers is created to ensure data redundancy as well as offer a platform for different calibration activities (e. g. CNES SADE desert site calibration project). GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 5

GSICS Data and Products Servers Services The server provides an upload service for the GSICS partners to upload their comparable net. CDF data sets in the WMO file naming convention. These data sets form a rolling archive accessible via HTTP or FTP download. Resulting products from the GRWG are also uploaded to the server for validation, publishing and/or archiving. For the EUMETSAT Data and Products Server, these products are archived in the EUMETSAT Data Centre Archive (URL - http: //archive. eumetsat. int). The EUMETSAT Data and Product Server is implemented on a Sun Fire X 4440 x 64 server under the Sun Solaris v 10 OS. The Unidata THREDDS Data Server, deployed in the Apache Tomcat webserver, catalogues the uploaded data sets and offers the HTTP and FTP download services. A Pro. FTP server is installed for FTP file transfers. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 6

GSICS Data and Products Servers High Level Design GEO-AIRS/IASI is one of many GSICS supports inter-calibration on other orbits, for other instruments, in other spectral regions All are similar in some ways Each pair is unique in some ways Instrument characteristics Agency priority Hierarchical Structure General ATBD Purpose, Options, Selection, Implementation Details for individual pairs on http: //www. star. nesdis. noaa. gov/smcd/spb/calibration/icvs/GSIC S/index. php GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 7

EUMETSAT GSICS Data and Products Server Comparable Data Sets available on the EUMETSAT Data and Products Server: MFG 15 net. CDF data set MSG 15 net. CDF data set IASI 1 c net. CDF data set Members of GRWG perform comparisons between MFG 15 and IASI 1 c data sets as well as MSG 15 and IASI 1 c data sets Currently uploaded products to the Server are the prototype GSICS Correction Coefficients for the intercalibration of the IR channels of Meteosat-9 using IASI as a reference GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 8

Access to the EUMETSAT GSICS Data and Products Server Downloading source data and products The EUMETSAT Data and Product Server went live on the 1 st April, 2009. It can be accessed via any web browser using the following URL: http: //gsics. eumetsat. int Uploading source data and products FTP accounts have been created for the upload of data and products to the server. Only validated and accepted products – according to GSICS Procedure for Product Acceptance – can be uploaded. Requests for ftp accounts can be forwarded to the EUMETSAT User Service Helpdesk (Email: ops@eumetsat. int). The server is currently pre-operational awaiting finalisation of the first set of GSICS data and products, i. e. changes can be made without user announcement. More GSICS information can be found on the GSICS website: http: //gsics. wmo. int/ GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 9

GSICS Data and Products Servers Documentation The following documents have been presented on the GSICS mailing lists (http: //groups. google. com/group/gsics-data-wg? hl=en): GSICS Data Management Server User Guide, Draft. GSICS Data Management Server Operations Service Specification (and supporting documentation) Action GDWG 03_22, Draft. EUMETSAT Archive Net. CDF Format Proposal, Draft. All documents are in draft form but will be released once the Data and Products server is operational. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 10

Data and Products Servers: Planned Activities Planned activities: Other GPRC’s to fully implement Data and Products Server Implement the mechanism for transfer of source data sets between GPRC data and product servers. Support to the GRWG in creation of the first validated calibration products. Analyse how best to improve the server’s services in terms of the GSICS products created from the source data sets (e. g. automated processes for data access). Examine the porting of EUMETSAT Net. CDF data sets from version 3 to version 4. GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 11

BACKUP GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 12

EUMETSAT GSICS Data and Products Server - Screenshots Access: http: //gsics. eumetsat. int GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 13

EUMETSAT GSICS Data and Products Server - Screenshots Source data Uploaded products GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 14

EUMETSAT GSICS Data and Products Server - Screenshots Source data EUMETSAT GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 15

EUMETSAT GSICS Data and Products Server - Screenshots Source data EUM MFG 15 GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 16

EUMETSAT GSICS Data and Products Server - Screenshots Metadata of source data collection Access to Dataset GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 17

EUMETSAT GSICS Data and Products Server - Screenshots Access: dataset Dataset Metadata GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 18

EUMETSAT GSICS Data and Products Server - Screenshots First 10 cols / rows of MFG 1. 5, channel 1 GSICS Users’ Workshop, Bath, UK, 22 Sep 2009 19