Performance Measures x x and x x OPe

Performance Measures x. x, and x. x OPe. NDAP Services for ESG 15 March 2018 Peter Fox, Patrick West, Stephan Zednik RPI Earth System Grid Center for Enabling Technologies 1

In ESG II (in regard to data) § § Server side aggregation DAP object transfer via HTTP and Grid. FTP GSIFTP integration, my. Proxy support OPe. NDAP-g -> led to architecture of OPe. NDAP/ Hyrax (primarily BES is from OPe. NDAP-g) Earth System Grid Center for Enabling Technologies: (ESG-CET) 2

Requirements leading to OPe. NDAP-g § Separation of the core Data Access Protocol (DAP) from the transport protocol (HTTP). § High Performance Computing. The previous CGI based servers did not have the capacity required by ESG. Error and memory handling added. § Security. Once the OPe. NDAP was independent of the transport protocol, adding security was possible by relying on the Globus gsi. FTP system. § Aggregation. OPe. NDAP 3. 0 did not operate on aggregated datasets. OPe. NDAP-g does. § Transport protocol independence and HPC were incorporated back into OPe. NDAP leading to the current version. Security and aggregation remain an ESG only feature. Earth System Grid Center for Enabling Technologies: (ESG-CET) 3

OPe. NDAP-g Architecture Dispatcher BES Data Ø OPe. NDAP-g Dispatcher (e. g. ESG Front-end Server) Ø Receives requests and asks the BES to fill them Ø Uses Apache Modules Ø Does not directly ‘touch’ data, handles URLs Ø Back End Server (BES) Ø Reads data files, Databases, et c. , returns info Ø May return DAP objects or other data (net. CDF) Earth System Grid Center for Enabling Technologies: (ESG-CET) 4

OPe. NDAP Hyrax Architecture Client OLFS BES Data Ø OPe. NDAP Lightweight Front end Server (OLFS) Ø Receives requests and asks the BES to fill them Ø Uses Java Servlets Ø Does not directly ‘touch’ data Ø Multi-protocol Ø Back End Server (BES) Ø Reads data files, Databases, et c. , returns info Ø May return DAP 2 objects or other data Ø Does not require web server 5 5

OPe. NDAP Lightweight Front end Server Request Formulation** HTTP DAP 2 RDF, OWL, JSON (HTTP) BES Request from client Grid. FTP DAP 2 (Grid. FTP, HTTP) SOAP-DAP (HTTP) Response to client THREDDS** Info output HTML form ASCII output PML output 6 6

Hyrax/ Back-end Server BES Framework Data Catalogs DAP 2 Access BES Commands/ XML Documents Network Protocol and Process start/stop activities Initialization/ Termination PPT/PPTS* Commands** e. g. server-side, Ferrett Provenance Net. CDF 3 HDF 4 RDF/ SPARQL Data 7 … Data Store Interfaces e. g. IOSP *PPT is built in (other protoco **Some commands are built in 7

OPe. NDAP-g services for ESG § Data access via Dataportals. In this Use Case users interact with the portal, browse the catalogs and decide what data to download. The portal passes the request to OPe. NDAP-g which executes it and returns the data to the portal. The portal returns a URL to the user to download the data. § Data access via net. CDF library. In this Use Case users link their applications with OPe. NDAP-g client library for net. CDF. The user can “open” a URL that refers to ESG data. The library fetches the data for the user and makes it a local net. CDF file § The performance requirements for ESG II were met. However, ESG-CET scales up these requirements. Earth System Grid Center for Enabling Technologies: (ESG-CET) 8

Status of the Community OPe. NDAP Server Software § Together the OLFS and the BES are known as Hyrax § Hyrax 1. 6 provides support for Nc. ML-based aggregation § Data response streamed back as net. CDF file § RDF response type § Updated DDX response type (Data Definition XML) § Beginning development of Data. DDX – multi-mime response with data and DDX § Full security audit and static code analysis certification to comply with NOAA and NASA requirements Earth System Grid Center for Enabling Technologies: (ESG-CET) 9

ESG-CET and data § Large data sets, numbers and sizes • High performance • Flexible architecture, both client and several types and numbers of servers • Aggregation • Server side operations • Multiple transport protocol options § Full ESG security support as well as loose federation § Read-only client access via API (net. CDF/CDM) ü To satisfy the new goals, the OPe. NDAP services for ESG have been re-architected. ü We now use parts of the standard OPe. NDAP framework Hyrax, focusing on high performance for the client side and extended flexibility. Earth System Grid Center for Enabling Technologies: (ESG-CET) 10

ESG-CET and Products (server side functions) § Goal: drop in replacement for the TDS part of FTDS in LAS § Requires net. CDF-Java Input-Output Service Provider (IOSP) adapter for Hyrax/BES § Use case examination will be required Earth System Grid Center for Enabling Technologies: (ESG-CET) 11

Security Infrastructure status § OPe. NDAP BES Security • SSL authentication between grid. FTP middle tier and the BES. No persistent SSL connections are maintained § RNI Integration with ESG Security Infrastructure • RNI client supports gsi. FTP connections to ESG Grid. FTP servers • ESG Grid. FTP server handles authentication of user • Neill's ESG Grid. FTp authz callout plugin handles authorization of data request • ESG/RNI Grid. FTP DSI module handles data request, forwards request to ESG BES server running RNI module Earth System Grid Center for Enabling Technologies: (ESG-CET) 12

Client status § RNI Version 0. 1 implemented using net. CDF version 3 and OPe. NDAP’s libnc-dap § In communication with Unidata regarding integration of RNI client with new Net. CDF version 4 § Developing ncml aggregation in both client and server RNI § Full ESG security support Earth System Grid Center for Enabling Technologies: (ESG-CET) 13

The Remote Net. CDF Invocation (RNI) § The client is the net. CDF library. It has exactly the same API as the standard C library net. CDF, but it can deal with local files or files reachable via HTTP, PPT or grid. FTP. § The third tier, the BES server can be reached only via PPT. Net. CDF services for all Net. CDF calls are implemented a a BES module. § The middle tier, acts like a proxy between the RNI client and server and deals with security. Earth System Grid Center for Enabling Technologies: (ESG-CET) 14

RNI Architecture Net. CDF CLIENT connection acts like Library DATA RNI Module RNI Library Grid. FTP OPe. NDAP BES Earth System Grid Center for Enabling Technologies: (ESG-CET) 15

Characteristics of the RNI as part of a data access system § Full Support of standard OPe. NDAP URLs. RNI is being developed with the integrated Unidata/OPe. NDAP net. CDF library (and CDM) § Transparent access to either standard net. CDF files and aggregated datasets via the Net. CDF Markup Language (NCML). § For remote containers, all write operations are disable for security. That is, for HTTP/HTTPS, PPT and grid. FTP/gsi. FTP the RNI system is a read only API. § RNI utilizes Just in Time access. Caching is only for metadata. No pre-fetching of data. § RNI transparently accesses secure (gsi. FTP, HTTPS) or insecure (grid. FTP, HTTP) remote data. Earth System Grid Center for Enabling Technologies: (ESG-CET) 16

Status of the RPI work (OPe. NDAP) § The primary accomplishments for this subproject in the past year has been • The complete request-response for all net. CDF API calls has been completed, that is, the two ends have been developed. • We have highly optimized core components of the standard OPe. NDAP framework to support the performance goals. • We have established the knowledge of how the middle tier (the proxy) will be incorporated into the complete system. • Our work was presented in AGU fall 2007 and EGU 2008. § So far all of the goals established for the first stage of the project has been completed. § Next stage • integration with product server for gateway and data node data access • Functional (with enhancements) replacement for TDS Earth System Grid Center for Enabling Technologies: (ESG-CET) 17

Future, will not elaborate § Storage Resource Manager / DMLite as a client § Return as RDF § Return as PML (Provenance – Proof Markup Language) Earth System Grid Center for Enabling Technologies: (ESG-CET) 18