THREDDS Data Server TDS and Data Discovery John

THREDDS Data Server (TDS) and Data Discovery John Caron Unidata/UCAR May 15, 2006

THREDDS Data Server HTTP Tomcat Server OAI Provider Catalog. xml OAI Harvester DL Records THREDDS Server Net. CDF-Java (CDM) library • OPe. NDAP • HTTPServer • WCS Application otherhost. gov Datasets hostname. edu OPe. NDAP Server

Collection vs Inventory Datasets Catalog Dataset. Scan Dataset Dataset http: //motherlode. ucar. edu: 8080/thredds/dods. C /model/NCEP/DGEX/CONUS_12 km/file. grib 2 /models/ncep/NAM/ File 1. grib Dataset File 2. grib Dataset File 3. grib Dataset

DL Harvesting Catalog Dataset. Scan Dataset Metadata Record Dataset is. Harvest = true inherit = true Dataset /models/ncep/NAM/ File 1. grib Dataset File 2. grib Dataset File 3. grib Dataset

Metadata Information • • Title / Summary Publisher / Creator / Rights Lat/Lon bounding box Time range – Relative time: “latest 7 days” • Variable names – DLESE : no (not dataset oriented) – GCMD: controlled list, required • Unique ID/ Resource URL

Why not harvest Inventory? • Too many of them, eg in IDD: – NCEP models: 28 collections, 6000 files – NEXRAD level 3 files: ~8 M files • Real-time datasets are never current • DLs (GCMD, DLESE) don’t want them – Collection search in DL, browse inventory on server.

Current Work: Aggregation • • • Make many files into single logical dataset: Make Collection Dataset = Inventory Uses Nc. ML to read into CDM, works at the “syntactic” level. Replaces older “Aggregation Server” – Union – Join on existing dimension – Join on new dimension

TDS/Nc. ML Aggregation

Next: Data. Type Aggregation • • Work at the CDM Data. Type level, know (some) data semantics Forecast Model Collection – – • Combine multiple model forecasts into single dataset with two time dimensions With NOAA/IOOS (Steve Hankin) Point/Station/Trajectory/Profile Data – – Allow space/time queries, return nested sequences Start from / standardize “Dapper conventions”

Forecast Model Collections

Web services for discovery • • • “Latest dataset” Resolver service Dataset Query Capability (DQC) : accept query, return results as a collection of datasets in a catalog Future: Dynamic dataset creation based on user query ? ?

Summary • Expect discovery to be 2 phased: 1. Search for collections in DL with browser 2. Use an application like the IDV (OPe. NDAP) or GIS client (WCS) to drill down to the actual data. • Expect aggregation / query will (eventually) tame the “inventory problem”

Dataset Query Capability Document • XML document that describes the set of valid queries for a dataset. Queries are URLS: http: //www/dqc/radar? stn=ABR&product=NOR&time=1 hour • Selectors: – – – List of choices List of stations Numeric range (point or subrange) Date. Range Latitude/Longitude Bounding Box • Orthogonal selections (except Lists can be nested) • Returns a catalog containing inventory datasets.

Example DQC . 5 u reflectivity . 5 u storm rel. velocity end="2004 -04 -15 T 12: 00" />

Issues • DQC itself doesn’t deal with the query http: //www/dqc/radar? stn=ABR&product=NOR&time=1 hour • Queries are expressible as param=value – Extend to arbitrary URLs (token substitution), eg dods – SOAP RPC? • Returns a catalog, might be the data itself. • Prototype/non-standard, need buy-in from clients to bother continuing.