Trieste 4 -5 June 2013 GROOM Annual meeting

Trieste 4 -5 June 2013 GROOM Annual meeting Data Management issues Quoi de Neuf à Coriolis en 2008 ? GMMC 13 -15 Octobre 2008 S Pouliquen & Coriolis team Developing a Data Management System for Glider data in Europe Sylvie Pouliquen GROOM Annual meeting 4 -5 June 2013

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 The Objectives of an integrated data system • Data accessible easily from a unique point • Data coherent in term of : ÞData format ÞData Quality ÞProcessing chain ( clearly documented) • Serve both Operational and Research Users ÞData are available in near real time ( within less than 24 hours) ÞData are available in delayed mode after calibration and /or validation

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 What About Gliders • Glider activities are presently driven by individual research drivers • Gliders can deliver real-time data for core parameters ( T, S Current , Chl, O 2) that are useful for both research an operational users • Benefit from what has been developed by Argo and Ocean. Sites Þ Developing integrated Data Management system Þ Common Data format to users Þ Real –time QC of core parameters • Gliders are complementary to other platforms and synergy should be developed Þ Developing a deployment strategy for other needs than pure research ( ie. GMES Marine Core Service in Europe)

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 What is the starting point for GROOM • Data management is done by the different research communities using their own methods • Some harmonization activities started in FP 5 -MFSTEP and FP 6 -MERSEA projects and FP 7 My. Ocean • Data exchange in Real time is working on a best effort schema through EGO but without any commitment nor from providers or data managers • Link with GMES MCS is done though Coriolis providing Glider data as profiles • No agreement neither on RTQC or DMQC but best practices on RTQC through My. Ocean INSTAC not widely known by the glider teams

GROOM Annual meeting Data Management issues Groom Data Management activities Task 3. 2 Trieste 4 -5 June 2013

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 GROOM Data Management Objectives • Data accessible easily from a unique point • Data coherent in term of : ÞData format ÞData Quality ÞProcessing chain ( clearly documented) • Serve both Operational and Research Users ÞData are available in near real time ( within less than 24 hours) ÞData are available in delayed mode after calibration and /or validation

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Objective of task 3. 2 • Improve coherency of Glider dataset in Europe • Facilitate access to Glider data by defining a Data system for Glider data • The system is built to exchange glider data ÞMust first be useful for Glider users and Glider operators and therefore provide access to as much as possible information (metadata, scientific, technical data) provided by Gliders ÞDelivering of Glider Data either on GTS or FTP for Operational users ( i. e. My. Ocean) will be carried on as specific delivery system that will convert the GDAC data into products usable by these operational users

GROOM Annual meeting Data Management issues Who does what ? Trieste 4 -5 June 2013

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 GROOM Data flow PIS Final correction Data to DAC - GDAC Scientific Users Operational Centers GTS FEED GDAC France RT within hours RT DATA QUALITY CONTROL Data assembly Centres DAC DAC Within hours DAC Glider Operators

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Roles of Glider Actors • PI : Principal Investigators ÞTeam or scientists who define the glider mission, deploy the glider and carry out post-recovery delayed mode QC • Glider Operators ÞTeam in charge of steering the glider, collecting all the metadata and the deployment information required for processing, collect all the data transferred in realtime by the glider. Collect the postrecovery high resolution data

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Roles of Glider Actors • the Dacs : the DAC is the facility set up by one or more nations/institutes to provide Real. Time and Delayed mode glider data to the users. It Þcollects the data from the Glider Operator, Þconverts to standard exchange format, Þapplies standardized real-time quality control, Þdelivers data to the GTS and GDACs within few hours of the surfacing and to PIs , Þcoordinates glider data handling for the gliders under their control.

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 List of DACs Country DAC Glider Operator Delayed Mode PI UK BODC UEA , SAMS, NERC (NOC, POL), CEFAS, BAS France CORIOLIS UEA , SAMS, NERC (NOC, POL) CEFAS, BAS LOCEAN, LOV, MIO, LEGOS, LPO, DT-INSU Italy CMRE To Be Defined OGS Germany OGS/CORIOLI S HZG HZG CORIOLIS GEOMAR, AWI SPAIN CORIOLIS PLOCAN SOCIB, CSIC Norway Ui. B/IMR Ui. B Cyprus OC-UCY Greece CORIOLIS HCMR ? To Be Defined Finland To Be Defined Ireland To Be Defined MI MI Poland CORIOLIS IOPAS LOCEAN, LOV, MIO, LEGOS, LPO

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Roles of Glider Actors • GDAC : The GDAC operates the data services where the master copies of the data resides. It doesn’t perform any additional individual glider QC activities. ÞCentral point for data distribution on Internet for all GROOM gliders ÞCan perform data format transformation, of set up additional services ( OGC viewing service, Open. Dap/Oceanotron download services , …) to fulfil additional needs.

GROOM Annual meeting Data Management issues Common data format Trieste 4 -5 June 2013

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Improve Coherency of Glider dataset in Europe • Worked on the definition of a Glider , of a deployment for a glider. Þ A glider is moving platform that is steerable. It can have a propeller and this information must be recorded in the metadata. • Define Level of processing: Þ Level 0 : Data provided by the glider without any unit transformation or geophysical interpretation. Þ Level 1: Geophysical parameters with a quality indicator set up by automatic QC procedures together with the data acquired by the glider. This is the level shared in Near Real Time. Þ Level 2 : Geophysical parameters calibrated after glider recovery together with quality flag information, if possible error estimation together with the non-corrected data provided at Level 1. This is the level shared in Delayed Mode Þ Level 3 and after : Product derived from glider data ( gridded fields, additional parameters calculated …) This is not addressed in the present GROOM data management activities.

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 How to Describe a Glider Takes on board Sensor S Sensor S Performs Deployment N Description of the glider for this deployment Is described by • What are the sensors on board • What is the configuration Transmits for each dive Technical information measurements Technical information measurements Scientific measurements Scientific measurements

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Improve Coherency of Glider dataset in Europe • Worked on common data format to share the Glider data based on Ocean. Sites Net. CDF format already used in Euro. GOOS/ My. Ocean/ Sea. Data. Net ÞDefined how to store Metadata to register the mission description and its evolution in time when changed through down link ÞEnhanced Ocean. Sites format for Scientific information by adding if necessary the new parameters sampled ÞDefined how to store the technical information by definition common vocabulary ÞFirst version of the user manual was delivered ÞCommon tools to produce these Netcdf files are under way

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 RT Processing chain modules Source data Generic Description - Platform - Sensors Slocum Seaglider Other gliders Common Readers in matlab that convert Manufacturers Data into. mat files My Glider description Real Time quality control routines Processing modules Conversion of. mat output to EGO format Net. CDF for transmission Data delivery PI GDAC GTS

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Data Stream in Real time Manufacturer Format D A T A Common Format Glider Operator Customize to meet other needs D A T A DAC My. Ocean Users ata i D Sc Sci and Tec data Sci and Tec GDAC GTS Users users

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Define Real Time QC procedure • Action taken to work on common Near Real Time QC procedures in agreement with what exist already within Euro. GOOS/My. Ocean/Sea. Data. Net ÞAdopt Euro. GOOS/My. Ocean NRT QC procedures for T&S, Chl and O 2 ÞEnhance if necessary these procedures to take into account Glider specific behavior ÞDevelop new recommendations in Partnership with Myo/SDN for additional parameters available in real time

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Data Stream after recovery Manufacturer Format Common Format High Res Manufacturer Format ta da H R fic nti ie sc n tio u -- --- ata at n d h R g rm io Hi fo olut on m res om gh Customize C Hi in de a o to meet at D d m other needs e im laye l T e ea ---D R a l o es Glider Operator H R T E C at i D Sc Common Format DAC Common Format GDAC users GTS Users My. Ocean Users

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Post recovery procedures • The Goal : correct the data transmitted in real time ÞCheck RTQC again. Additional run taking into account past and future data is possible (future data not available in RT !) ÞIf necessary Merge RT data with flash card data in order to fill the gaps left by the RT data transmission in terms of resolution and/or parameters measured ÞCross-calibration along the whole deployment with reference to in-lab water samples measurements when possible. ÞWorking group to develop recommendations for T&S, Chla, Oxygen, Current

GROOM Annual meeting Data Management issues Possible extention outside Europe through the EGO Cost action Trieste 4 -5 June 2013

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 GROOM Data flow PIS Final correction Data to DAC - GDAC Scientific Users Operational Centers GTS FEED GDAC France RT within hours RT DATA QUALITY CONTROL Data assembly Centres DAC DAC Within hours DAC Glider Operators

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Possible extention outside Europe PIS Final correction Data to DAC - GDAC Scientific Users Operational Centers WWW FTP GDAC France GDAC ? GTS FEED Archiv e? Glider Info Center? RT within hours RT DATA QUALITY CONTROL Data assembly Centres Monitoring DAC DAC Within hours Glider Operators

GROOM Annual meeting Data Management issues Trieste 4 -5 June 2013 Data Format and QC procedures • Data Format : based on Ocean. Sites netcdf format with CF convention also used by IMOS /Australia ÞCompare the format metadata to be sure that we are coherent (link with ODIP EU project): • name the same thing the same way • list of mandatory fields that should be in all files • Share tools to generate netcdf files • NRT QC procedures : Both IMOS and QUARTOD procedures are taken as input for defining the EU RTQC procedures for core parameters. • Post-recovery procedure : still and R&D activity but collaboration with international partners should be encouraged

Trieste 4 -5 June 2013 GROOM Annual meeting Data Management issues Quoi de Neuf à Coriolis en 2008 ? GMMC 13 -15 Octobre 2008 S Pouliquen & Coriolis team Thank You Questions ?