DAPPER An OPENDAP Server for In-Situ Data Joe

DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald Denbo, John Osborne, Willa Zhu University of Washington PMEL/NOAA 1

What is Dapper? Web server that provides distributed access to in -situ data via OPe. NDAP protocol Clients include Gr. ADS, Matlab, Nc. Browse, Java Ocean Atlas Available data: – PMEL EPIC database – ARGO profiles – NODC World Ocean Database 2001 Includes database loader for adding/removing profiles from Dapper database 2

What does Dapper do? Efficiently aggregates thousands of in-situ data files (>2000000 for WODB 2001) Handles several in-situ data formats: – Argo GDAC net. CDF – PMEL EPIC net. CDF – MADIS (Meteorological Assimilation Data Ingest System) net. CDF – NODC Argo/COARDS net. CDF Supports time series or profiles Supports space/time queries 3

What is OPe. NDAP? OPe. NDAP (Open source Project for a Network Data Access Protocol) is a protocol for remotely accessing scientific data Uses HTTP as transport 4

Dapper OPe. NDAP Example (space/time query) Client HTTP request: http: //foo. org/. . . /dset. cdp. dods? lat, temp, pres&lat>0&lat<20 Dapper HTTP response: Dataset { Sequence { Float 32 lat; Float 64 _id; Sequence { Float 32 pres, temp; } profile; } } location = { { -4. 01, 1, { {4. 3, 26. 808}, {100, 24. 982}, …}, { -52, 2, {{8. 3, 3. 77}, {100, 3. 743}, …} 5

Dapper OPe. NDAP Example (single profile) Client HTTP request: http: //foo. org/. . . /dset. cdp. dods? lat, lon, time, profile&_id=20 Dapper HTTP response: Dataset { Sequence { Float 64 time; Float 32 lat, lon; Sequence { Float 32 pres, temp; } profile; } } location = { {1034140577999, -52. 9, 310. 144}, {{8. 3, 3. 77}, {9. 1, 3. 77}, …}}; 6

OPe. NDAP (cont. ) Sequences accessible to clients via Java or C++ libraries Dapper specific _id field in “outer” sequence allows unique identification of an individual profile 7

Dapper architectural overview Uses relational database to store metadata and coordinate boundaries for each profile/time series file Data maintained in original format All data delivered as OPe. NDAP sequences 8

Dapper architecture diagram OPe. NDAP client OPe. NDAP Protocol (HTTP) Dapper HTTP Server CDP service net. CDF service CORBA (IIOP) Climate Data Portal My. SQL Java net. CDF library Database loader net. CDF profile 9

Climate Data Portal (CDP) CORBA based Metadata in database (My. SQL) Variables (measurements) in individual net. CDF files Reads EPIC, COARDS, and Argo GDAC net. CDF files Originally developed by Don Denbo and Willa Zhu 10

Dapper architecture diagram OPe. NDAP client Dapper HTTP Server CDP service net. CDF service CORBA (IIOP) Climate Data Portal My. SQL Java net. CDF library Database loader net. CDF profile 11

Dapper HTTP Server Consists of configurable services – Climate Data Portal service – net. CDF service Services route OPe. NDAP requests Implemented as Java servlet 12

Dapper architecture diagram OPe. NDAP client Dapper HTTP Server CDP service net. CDF service CORBA (IIOP) Climate Data Portal My. SQL Java net. CDF library Database loader net. CDF profile 13

Database loader Automatically adds or removes profile info from database Transactional loader – database won’t be left in invalid state Uses Hibernate object relational mapping software 20 -30 ms to load one profile 14

Dapper performance For World Ocean Database 2001 (~2 e 6 profiles): ~600 us/profile for coordinate info ~5 ms to read one profile Faster for smaller datasets Don’t use My. SQL default configuration! 15

Dapper clients from PMEL nc. Browse (Don Denbo, John Osborne) 16

Dapper clients from PMEL Java Ocean. Atlas (John Osborne) 17

CDFsync Rapid synchronization of net. CDF in-situ files over a network Only compressed file differences transmitted across network Based on rsync, but optimized for net. CDF and file lists consisting of thousands or millions of files Available at http: //www. epic. noaa. gov/epic/software/cdfsync/ 18

Current work “Thin” web client for Dapper Query and visualization of profile/time series data Downloads in alternate formats (spreadsheet, GIS, gzipped net. CDF) Based on AJAX (Asynchronous Java. Script and XML) – like Google Maps or Gmail. 19

More info PMEL Dapper Server http: //www. epic. noaa. gov/dapper Downloads, propaganda http: //www. epic. noaa. gov/epic/software/dapper/ Joe. Sirott@noaa. gov 20