NOAA s National Ocean Service Office of Response

NOAA’s National Ocean Service • Office of Response and Restoration Serving unstructured grids using OPe. NDAP: Using server-side operations to subset and subsample data Christopher Barker NOAA Office of Response & Restoration Emergency Response Division James Gallagher OPen. DAP, inc.

NOAA Emergency Response Division • National Contingency Plan specifies NOAA’s role in supporting the Coast Guard: “Provide scientific expertise to support an incident response for Oil and Chemical Spills”

Key Role: Trajectory Modeling • Where is the oil (or chemical) going?

Primary Tool: GNOME (General NOAA Operational Modeling Environment) • Lagrangian element (particle) model • Forcing from external sources: – Winds – Currents • Currents: – In house model – External operational models

GOODS GNOME Online Operational Data Server

Example: Deepwater Horizon • Ocean models utilized: – NOAA CSDL: NGOM – Navy models: NCOM, HYCOM, IASNFS – USF: West Florida Shelf ROMS – TGLO/TAMU: TX shelf ROMS – NC State: SABGOM – All structured grid models

Unstructured Grid Models? • Unstructured Grids: – Allow resolution to vary spatially – Conform to boundaries • Nice for oil spills and particle tracking • Many more UGRID models coming online – Many papers at this conference

Some Models of Interest • FVCOM: – n. GOMOFS (NOAA CSDL) – Gulf of Maine/Mass Bay (UMASS) – Salish Sea (PNNL) • SELFE: – Columbia River (OHSU) – Texas Estuaries models (UT) • ADCIRC: – Gulf of Mexico / Southern LA and Texas grid 9, 108, 128 nodes--18, 061, 765 elements

n. GOMOFS (NOAA CSDL) V 6 90, 310 Nodes 174, 550 Elements

What if I just need Mobile Bay? Mobile Bay, AL detail grid. About 300 m grid resolution along a 13 m deep navigation channel

FVCOM-Go. M/GB for Mass Bay and Nantucket Sounds/Shoals Boston Inner Harbor

ADCIRC: Gulf of Mexico / Southern LA and Texas grid (SL 18 TX) • Gulf of Mexico / Southern LA and Texas grid 9, 108, 128 nodes--18, 061, 765 elements • Just surface currents: – 275 MB per time step (plus the grid specs)

Obstacles to using UGRID models: • No standard for data/results on UGRIDS: – Informal working group for (quite!) a few years – Recent draft standard (netcdf 3) – Work on Java. Netcdf lib to support it (SURA modeling test bed project) • Big Grids: – Need server side subsetting

How to get it done? • NOAA/ORR post-DWH funding: – Better able to response to large spills • We started talking to folks about server-side subsetting options • But we’re clients: – We’re not going to run a server • We needed something that would become an excepted standard/tool.

How to get it done? • NOAA/NESDIS noted assorted issues: – Netcdf/Open. DAP development funding limited – Multiple diverging implementations: “Unfunded Mandate” • NESDIS coordinated funding from: – Technology, Planning and Integration for Observations (TPIO) Program – OR&R – National Climatic Data Center (NCDC)

OPe. NDAP-Unidata Linked Servers (OPULS) • NOAA/BAA grant supports this important collaboration between Unidata & OPe. NDAP • First goal: conformance between OPe. NDAP & Unidata servers, through which access is gained to growing amounts of NOAA & related data. Other short-term goals include: – Asynchronous modes, such as are needed for (delayed) access to nearline data, perhaps stored on tape, e. g. – Improved access (with server-side subsetting) to data organized on nonrectangular meshes, such as in coastal modeling • Work began in Boulder during October & will be influenced by an advisory committee (yet to be appointed)

OPe. NDAP: the Data Access Protocol • DAP 2 combines simple data model with a general set of operators. – Data Model: Atomic types (e. g. , ‘Integer’); Arrays; Structures; Grids; and Sequences. – Operators: These provide ways to subset all but the atomic types. – Domain neutral: By keeping the semantics of the model clean, we ensure that it can be applied to many different types of data.

But how is it used? • DAP is generally used as a ‘web service’ • DAP requests are made using a URL • DAP responses are ‘documents’: – Text that contains metadata – Combination of text/metadata and binary data. • Applications read these responses and use them it whatever ways they see fit: – the net. CDF client library makes legacy applications believe they are reading from a local file

About Array and Grid Selection • In addition to requesting a Grid or Array, the Selection can be used to subset in indicial space.

About Functions • Constraint Expression can contain functions • These functions can perform any operation that can be programmed. • Thus they provide a good way to extend a data server to perform new operations • These include operations that are not domain neutral • In Hyrax they are written in C++

Example URLs • The base URL: “http: //test. opendap. org/opendap/data/nc/fnoc 1. nc” • To get metadata: – Dataset variables: http: //test. opendap. org/opendap/data/nc/fnoc 1. nc. dds – … attributes: http: //test. opendap. org/opendap/data/nc/fnoc 1. nc. das – Or less readable in XML: http: //test. opendap. org/opendap/data/nc/fnoc 1. nc. ddx • To get data: – Just the variables u and v: http: //test. opendap. org/opendap/data/nc/fnoc 1. nc. dods? u, v – … in ASCII so it’s easy to read: http: //…/opendap/data/nc/fnoc 1. nc. asc? u, v • With subsetting: – http: //test. opendap. org/opendap/data/nc/fnoc 1. nc. asc? u[0][3: 6][5: 8] • Here’s a function: – http: //…/nc/coads_climatology. nc. ascii? geogrid(SST, 45, -80, 20, 60, ” 1000

Challenges • Unstructured Grids are not a specific type in DAP • We must choose a way, or set of ways, to represent these data • Datasets are often too large to download – subsetting must be done server-side. • Because the subsetting operations are complex, we will need to use server-side functions to implement them

Requirements • Must enable subsetting by polygonal regions • The result must be an unstructured grid itself • A subset must preserve the topological and geometric relationships present in the whole: – we can’t just regrid everything to a more convenient form.

Proposed Solution • Server-side function to add subsetting • Adopt the proposed unstructured grid encoding using net. CDF 3 • Result of the function will be a DAP 2 response – Input is net. CDF 3 with some additional ‘conventions’: it can be represented in DAP 2 – There are existing clients that can read DAP 2 • If they understand netcdf in the new convention, they will understand the results

The server-side function • Ugrid(Mesh, ) – is a comma separated list of latitude and longitude points – However, there is an arbitrary limit to the number of characters in a URL, so • We will also support POST when OPULS makes the transition to DAP 4 – It will likely take more than a year for all of DAP 4 to be realized, but POST for constraint expressions will be set in the first year.

Example ugrid() calls • http: //…/model. nc? ugrid(SST, 45, -80, 20, -60) – When ugrid() is called with two points, it will assume the polygon is a box. • http: //…/model. nc? ugrid(SST, 45, -80, 45, -60, 20, -80) – Here the polygon the same box as above. – There’s an understood edge connecting the first and last points – Point order is important – self-intersecting polygons will raise an error.

http: //…/model. nc? ugrid(SST, -71. 03, 42. 38, -71. 06, 42. 37, 71. 06, 42. 36, -71. 06, 42. 35, -71. 04, 42. 33 -71. 01, 42. 34, 71. 01, 42. 35, -71. 03, 42. 38)

Implementation • We will use the Gridfields library [Howe 05] • The library will be extended to work with the new net. CDF 3 file format: “Deltares CF proposal for Unstructured Grid data model” • And to work with DAP [Howe 05] Bill Howe, David Maier, “Algebraic Manipulation of Scientific Datasets, ” VLDB Journal, 14(4) 2005

Progress so far • Gridfields has already been used to build a simpler server-side demonstration function • The Gridfields code has adopted GNU’s autotools to streamline its build. • We will factor out the C++ code into its own project, separate from the Python layer • This will simplify moving gridfields into the Linux community builds

Summary • Ugrid models are seeing wide deployment • Subsetting UGrids on the server is critical to the wide use of model results • UGrids will be encoded in net. CDF 3 • We will use a widely available open-source library to perform the actual operations • The results will be valid UGrids, in DAP • The work has begun

Use for Curvilinear grids, too? • Capture arbitrary polygon subset. • Rectangle in geo-coordinates not a rectangle in grid coordinates – We generally over sample. - But that’s not always a good solution for highly deformed grids. - What would the result look like? - A new structured grid? - An unstructured grid?

Further Discussion, etc. • Meet here at ECM: – Lunch Wed? • Discussion on UGRID Google group: https: //groups. google. com/group/ugrid-interoperability • OPe. NDAP Wiki: http: //docs. opendap. org/index. php/Projects