Session 4 Using OPe NDAP-enabled Applications to Access

Session 4: Using OPe. NDAP-enabled Applications to Access Australian Data Services and Repositories e. Research Australasia 2011, ½ Day Morning Workshop, Thursday 10 th November 2011

GENERAL INFORMATION • This is a half-day workshop (9 am to 12: 30 pm) • 9: 00 am Introductions and Participants Goals • 9: 15 am Session 1: Discovering OPe. NDAP data access services • 10: 00 am Session 2: Applicable use cases of OPe. NDAP data services − 10: 30 am Tea Break for 15 minutes • 11: 00 am Session 3: OPe. NDAP service protocols and features • 11: 45 am Session 4: Accessingcomplementary features and services 11: 45 am Session 4: Accessing complementary features and services • 12: 30 pm End of Workshop

Session 4 • Accessing OPe. NDAP servers with geospatial services, aggregation services, virtual datasets and data libraries. • A short tutorial on data access to these services using an OPe. NDAP - and OGC-enabled client application or web servlet. • 45 minutes in length

OPe. NDAP crawler • Both Hyrax and TDS support THREDDS catalogs • THREDDS catalogs are hierarchical: • Each catalog describes a set of local resources and • Provide links to child nodes • Analogous to a files system directory tree • Our crawling code • • • Uses simple ‘rules’ to form aggregations Optimizes reads to actual data Can write NCML and/or EML Can also exhaustively read DDX responses Is ‘pipelined’ and stores stages’ results in Postgres

OPe. NDAP crawler availability Our crawler is made up of six shell programs: • ddx_crawler. sh: Crawl the THREDDS catalogs (NB: The misnomer) • ddx_retriever. sh: Get DDX responses • print_cached_urls. sh: Extract TCs and DDXs from the cache • url_classifier. sh: Use TCs and URL characters to form ‘logical groups’ • eml_writer. sh: Write EML for groups • ncml_writer. sh: Write NCML for groups

Nc. ML can provide two basic features: • Augmenting/Modifying data sets with new • Attributes • Values • Combining two or more data sets (i. e. , files) in an aggregation • Three kinds of aggregation are aupported: • Tile files • Join files along an existing axis • Join files along a new axis • The creation of these logical data sets can replace the old idea of an ‘inventory’ of files in some important cases • While very powerful, these aggregations are not applicable to every data set made up of multiple files

Nc. ML demo Access aggregation of data files on a service: • Access an aggregation of http: //satdat 1. gso. uri. edu/thredds/dods. C/NWAtlantic. Raw_6 km • • Look at the DDS and DAS This is an aggregation of 18, 000 discrete files The times run from 473452108 to 987018422 Lets get data from a given latitude over time (Hoffmüller plot) Here’s the Constraint: ? dsp_band_1[17108: 1: 17208][600][0: 1023] Really cool, but gads that’s hard to write. The Nc. ML Documentation: http: //www. unidata. ucar. edu/software/netcdf/ncml/ Using the grid function is much easier, but this particular server does not support it. Here’s the function version: ? grid(dsp_band_1, "lat=30", "954794345<time<987018422")

Providing certificates to clients • There a variety of ways to get PKI certificates • The general process is as follows: • Make a private key • Use that to request a certificate from some ‘Authority’ • When the authority sends you the requested cert, install it. • • How do you do this with OPe. NDAP clients? Right now, the answer is ‘it depends’ We hope it will become more uniform in the future For clients made using the net. CDF C library, version 4: • Add some lines to the. dodsrc file • For clients made with the net. CDF Java library: • Add the cert to a Java Keystore

net. CDF C case # DODS client configuation file. See the DODS # users guide for information. USE_CACHE=0 … # This is the 'verify peer' function of curl. For this to work, the # server's certificate must be signed by a real CA, not the mock CA # we use for demos, hacks, etc. CURL. SSL. VALIDATE = 0 [https: //localhost: 8443/opendap/] CURL. SSL. CERTIFICATE = /Users/jimg/certtest/client. crt [https: //localhost: 8443/opendap/] CURL. SSL. KEY = /Users/jimg/certtest/client. key [https: //localhost: 8443/opendap/] CURL. SSL. CAPATH = /Users/jimg/certtest/

net. CDF Java case When you invoke Java, you need to set four flags: -Dkeystore=<path to the keystore> -Dkeystorepassword=<password for the keystore>

Data Discovery and Access Data discovery services • NASA’s Global Change Master Directory − http: //gcmd. nasa. gov • IMOS e. MII portal − http: //imosmest. aodn. org. au/geonetwork/srv/en/main. home − Help --> http: //emii 1. its. utas. edu. au/drupal/? q=node/25 • TERN Aus. Cover portal − http: //data. auscover. org. au/ • My Ocean portal − http: //www. myocean. eu/web/24 -catalogue. php • TPAC Digital Library − http: //dl. tpac. org. au Data access services • Unidata’s THREDDS Data Service − http: //www. unidata. ucar. edu/projects/THREDDS/ • OPe. NDAP’s Hyrax Data Service − http: //opendap. org/download/hyrax. html • NOAA’s ERDDAP Data Service − http: //coastwatch. pfeg. noaa. gov/erddap

Web services and GIS Data analysis/visualization services • NOAA PMEL’s Live Access Server − http: //ferret. pmel. noaa. gov/Ferret/LAS/home • COLA’s Gr. ADS Data Service − http: //www. iges. org/grads/gds/ Geospatial Information Services (GIS) • Unidata THREDDS WCS (Web Coverage Service) − http: //www. unidata. ucar. edu/projects/THREDDS/ • Reading e-Science/TPAC’s nc. WMS (Web Map Service) − http: //www. resc. rdg. ac. uk/trac/nc. WMS/ • Reading e-Science Centre’s Godiva 2 − http: //www. resc. rdg. ac. uk/twiki/bin/view/Resc/Godiva. Two

Some of the Technology in the TDS 1. THREDDS Dataset Inventory Catalogs provide virtual directories of available data and associated metadata. 2. The Netcdf-Java/CDM library reads Net. CDF, Open. DAP, and HDF 5 datasets, as well as other binary formats such as GRIB and NEXRAD, essentially an (extended) net. CDF view of the data. 3. TDS can use the Net. CDF Markup Language (Nc. ML) to modify and create virtual aggregations of datasets. 4. An integrated server provides OPe. NDAP access with subsetting data access method. 5. An integrated server provides bulk file access through the HTTP protocol. 6. An integrated server provides data access through the Open. GIS Consortium (OGC) Web Coverage Service (WCS) protocol, for any "gridded" dataset whose coordinate system information is complete. 7. An integrated server provides data access through the Open. GIS Consortium (OGC) Web Map Service (WMS) protocol, for any "gridded" dataset whose coordinate system information is complete. 8. The integrated nc. ISO server provides automated metadata analysis and ISO metadata generation.

Some of the Technology in Hyrax 1. THREDDS Dataset Inventory Catalogs provide virtual directories of available data and associated metadata. 2. Supports many formats and data stores: net. CDF 3, net. CDF 4, HDF 5, Free. Form, SQL data bases 3. Uses a plug-in based architecture and includes tools to write custom handlers 4. Net. CDF Markup Language (Nc. ML) to modify and create virtual aggregations of datasets. 5. OPe. NDAP access with subsetting data access method. 6. bulk file access through the HTTP protocol. 7. nc. ISO server provides automated metadata analysis and ISO metadata generation. 8. RDF output 9. Code that has passed a formal security audit 10. A true multi-system architecture that can fit in a variety of enterprise settings 11. An administrator’s interface

A quick look at Digital Library services The TPAC Digital Library, see e. g. https: //dl. tpac. org. au/tpacportal/ the Oceans and Climate Digital Portal • Searchable, aggregatable OPe. NDAP community research portal • Includes text descriptions of data sets with rich and structured metadata The IMOS or Aus. Cover Portals should be explored. • http: //imosmest. aodn. org. au/geonetwork/srv/en/main. home • http: //data. auscover. org. au/

Open Data through the digital library Tim Pugh 03 9669 4345 t. pugh@bom. gov. au

Exercise Open the web browser and visit the following sites for a couple of minutes. • NASA’s Global Change Master Directory − http: //gcmd. nasa. gov − Find a reference to sea surface temperature data in the Southern Ocean • Visit the IMOS, Aus. Cover, or TPAC portal • • Are the site similar or different? NOAA PMEL’s Live Access Server − http: //mynasadata. larc. nasa. gov/data. html − Select “+ Live Access Server (Intermediate Edition)” and view some data • Reading e-Science Centre’s Godiva 2 − http: //www. resc. rdg. ac. uk/twiki/bin/view/Resc/Godiva. Two − Access TDS server with wms available, launch the Godiva 2 java applet

LIST OF OPENDAP SERVERS • OPe. NDAP servers located at • Bureau of Meteorology ( http: //opendap. bom. gov. au: 8080/thredds ) • CSIRO ( http: //opendap. csiro. au/thredds ) • ANU/NCI ( http: //opendap. nci. org/thredds ) • OPe. NDAP, inc ( http: //test. opendap. org: 8080/hyrax ) • TPAC ( http: //opendap-tpac. arcs. org. au/thredds )

Thank you Authors: Tim F. Pugh 1, James Gallagher 2, Dave Fulker 3 1 Australian Bureau of Meteorology, Melbourne, Australia, t. pugh@bom. gov. au 2 OPe. NDAP, Butte, Montana, USA, jgallagher@opendap. org 3 OPe. NDAP, Boulder, Colorado, USA, dfulker@opendap. org