SEEGRID Roadshow Technical Workshop Stuart Girvan Geoscience

SEEGRID Roadshow Technical Workshop Stuart Girvan – Geoscience Australia Rob Atkinson – Social Change Online CSIRO - Andy Dent, Simon Cox, Robert Woodcock www. seegrid. csiro. au

Disclaimer Participants should only act on the information and advice in these presentations after independent consideration of their circumstances SEEGrid Roadshow 2005

Program Demonstrator Architecture OGC's WFS Specification Introduction to Geoserver GML Application Schemas Implementing Geoserver - Database and Data Mapping Issues Use Cases and Service Structure for Clients in an Open Ended Architecture Further Technical Issues and Discussion SEEGrid Roadshow 2005

Demonstrator Project Participants SEEGrid Roadshow 2005

Contributors Rob Atkinson, Peter Barrs – Social Change Online Joan Esterle (UQ), Simon Cox, Rob Woodcock, Andy Dent – CSIRO EM Greg Jenkins, Mark Jolly, Terry Hannant – PIRSA Brendon Ward, Stephen Bandy – DOIR (GSWA) Lesley Wyborn, Aaron Sedgmen, Tim Mackey – GA Stuart Girvan – GA, pmd*CRC SEEGrid Roadshow 2005

Demonstrator Architecture www. seegrid. csiro. au

Demonstrator Did everyone see the demonstrator this morning? http: //cgsrv 3. arrc. csiro. au/seegrid/savedapps/filter http: //www. ga. gov. au/wfs/reader/wfs. Geochem. Reader. Home. jsp SEEGrid Roadshow 2005

Web Map Composer DOIR PIRSA GA SEEGrid Roadshow 2005

Web Feature Services (OGC) Web Map Composer XML - GMLXMML DOIR PIRSA GA SEEGrid Roadshow 2005

GA Reports Application Web. Map Composer CLIENT APPLICATIONS Common Interface Binding – GML/XMML DATA ACCESS SERVICES DOIR Translation Web Feature Service (WFS) Geoserver (Open Source) DATA SOURCES DOIR Geochemistry or Little Feature Data Source Post. GIS (Open Source) GA to standards here Web Feature Service (WFS) PIRSA Web Feature Service (WFS) no PIRSA change required Geochemistry Feature Data Source Post. GIS (Open Source) here GA Geochemistry Feature Data Source Oracle SEEGrid Roadshow 2005

GA Reports Application ? CLIENTS Web. Map Composer ? ? Common Interface Binding – GML/XMML DOIR WFS GA WFS PIRSA WFS NRM WFS MRT WFS DATA SERVICES NSWDPI WFS NTGS WFS VICDPI WFS DATA SOURCES SEEGrid Roadshow 2005

Simple Web Services Architecture Registry publish find Client Applications bind Services Data Access Services SEEGrid Roadshow 2005

Simple Web Services Architecture • For the moment not using registries to find or publish services • Using “Magic Knowledge”, ie applications will know the data services available Services Client Applications bind GA WFS GSWA WFS PIRSA WFS SEEGrid Roadshow 2005

OGC's WFS Specification www. seegrid. csiro. au

Web Feature Service (WFS) Specification https: //portal. opengeospatial. org/files/? artifact_id=8339 (only 131 pages!) Uses http Accepts requests as either key-value pairs or XML Responds in XML (GML) SEEGrid Roadshow 2005

Web Feature Service (WFS) Uses request/response not SOAP (although it can be wrapped in SOAP) Three main calls: § Get. Capabilities § Describe. Feature. Type § Get. Feature SEEGrid Roadshow 2005

Web Feature Service (WFS) Application (web based or desktop) XML/ KVP Get Capabilities Request Get Capabilities Response XML/ KVP Describe Feature Type Request Describe Feature Type Response XML/ KVP Get Feature Request Get Feature Response XML GML Schema GML Web Feature Service Config Files Data Source http protocol SEEGrid Roadshow 2005

Features ≈ geographic features Features are Geography Markup Language (GML) encoded GML provides a standard for representing geometries (points, lines, polygons) and properties of geographic features SEEGrid Roadshow 2005

Features Borehole § § § § collar location shape Fault collar diameter § shape length § surface trace operator Basin? § displacement logs § § age related observations formations § shape – time dependent § … … § resource estimate § … Ore-body Observation § § § § location subject/specimen/station property/theme/measurand method operator date/time result (+ type/reference § § § commodity deposit type host formation shape resource estimate … system/scale/classification) § … SEEGrid Roadshow 2005

Features Example: Instance of bridge feature encoded as GML attributes 50 51 10 -10 -1996 12435. 2, 6756. 34 SEEGrid Roadshow 2005

WFS Demo Get. Capabilities Describe. Feature. Type Get. Feature SEEGrid Roadshow 2005

Filter Encoding Specification Defines how to write WFS queries when making a Get. Feature call http: //portal. opengeospatial. org/files/? artifact_id=8340 SEEGrid Roadshow 2005

Filter Encoding Example DEPTH 30 Geometry 13. 0983 31. 5899 35. 5472 42. 8143 SEEGrid Roadshow 2005

How would you use a WFS? Using http calls from your programming language to access the WFS and native parsers Using http calls and style sheets to transform the XML SEEGrid Roadshow 2005

Eg GA Report Pages Java reads list of input URLs Subs in search values to pre-prepared XML chunks Calls URL of WFS(s) with XML parameters Parses XML document returned by WFS Data from XML document held in Java objects and sent to jsp pages for display SEEGrid Roadshow 2005

Introduction to Geoserver www. seegrid. csiro. au

Why Geoserver? It’s the reference standard for implementation Open Source – we could influence and directly upgrade if necessary Open Source – no license fees/purchase cost SEEGrid Roadshow 2005

Other WFS Products • Deegree (open source), Cadcorp SIS Map Server, CARIS Spatial Fusion Enterprise, Red. Spider Web 3. 0, Map Manager, Gena. Ware, Scl. Feature. Server, JCarnac. GIS, Geo. Media Web. Map, Map. Xtreme (Map. Info), Map. Server(UMN) 4. 2 + more * Source: OGC Registered Products SEEGrid Roadshow 2005

Geoserver Java based, servlets Runs using Tomcat (or embedded Jetty) Open source community size? Reliant on and closely connected to Geotools Lots of modules SEEGrid Roadshow 2005

Setting up Geoserver Install into Tomcat using WAR file or use inbuilt Jetty servlet container Works with Post. GIS, Oracle, ARCSDE, Shapefiles, mysql as data sources Config files catalog. xml, info. xml, schema. xml directory structure SEEGrid Roadshow 2005

GML Application Schemas www. seegrid. csiro. au

Community/Application Schemas Community schemas provide the common or shared model The target community schema is essential SEEGrid Roadshow 2005

GML Community Schemas Geological XMML - Exploration data (Example Geochemistry) ADX – Assay data GPML – plate tectonics XMML/Geo. Time – stratigraphy CGI International Model Other Land. GML Marine. XML SEEGrid Roadshow 2005

GML Application Schemas Design methodology can be learnt to develop community specific GML application schemas (Simon Cox, Andy Dent CSIRO) Equally important is who will be the custodian and manager of a community schema Also see GML 3. 1 specification , Chapter 8 Rules for Application Schemes SEEGrid Roadshow 2005

Informational Requirements Information models impact on real world performance Ontology Registries SEEGrid Roadshow 2005

Why XML? Extensibility Self describing Ability to be (remotely) validated against schema XML Schema provides “loose tolerances” All software languages have tools to deal with XML SEEGrid Roadshow 2005

Implementing a WFS Using Geoserver – Database and Data Mapping Issues www. seegrid. csiro. au

Traditional GIS and Standard WFS The WFS spec states the minimum requirements for a service interface It says nothing about private schemas vs public (community) schemas Hence most WFS software will only produce “flat” GML SEEGrid Roadshow 2005

Flat GML Example Sample ID Coordinates Sample Type Si. O 2% Ti. O 2% Al 2 O 3% Fe 2 O 3% Mg. O% 11111 127, -32 Whole Rock 50 1 14 14 3 118. 78531, -21. 122948 whole-rock 50 1 14 14 3 SEEGrid Roadshow 2005

Flat GML Example 50 1 Proc 1 Blah 1 X 1 0. 01 1 < Ti. O 2%Procedure. ID>2 < Ti. O 2%Name> Proc 2 < Ti. O 2%Description> Blah 2 < Ti. O 2%Method> X 2 < Ti. O 2%Detection. Limit>0. 01 SEEGrid Roadshow 2005

50 Si. O 2 50 Si. O 2 50 Si. O 2 Proc 1 Blah 1 X 1 Nested XML Example Si. O 2 0. 01 1 Ti. O 2 Proc 2 Blah 2 X 2 Ti. O 2 0. 01 SEEGrid Roadshow 2005

XML Structures There are potentially many structures or information views that may be appropriate for describing data These may be influenced by use cases and architectural considerations SEEGrid Roadshow 2005

SEEGrid Roadshow 2005

Data source to community schemas Community schemas provide the common or shared model All data providers have their own local data model All data providers must map data from local source (database) to community schema, irrespective of technology implementation SEEGrid Roadshow 2005

Requirement We need to be able to map from private schemas to public in as flexible a manner as possible SEEGrid Roadshow 2005

WFS Spec Problem WFS spec allows any query based on the feature description (community schema) Comes from the point of view that you’d be able to use, for example, xpath against a ‘notional’ XML document In reality almost all data sources will be databases or proprietary formats of some kind In practise want to limit the potential queries that can be made against the data source SEEGrid Roadshow 2005

Query Profiles Why you should only allow certain queries § efficiency § security § user friendliness § legal requirements Eg phone books, amazon. com SEEGrid Roadshow 2005

Requirement Limit the possible queries that can be made against the feature data source SEEGrid Roadshow 2005

Implementation At beginning of project Geoserver (nor any other software we were aware of) could not perform mapping from private database to community schema Geoserver could only serve “flat” GML No restrictions on queries (follows the WFS spec) SEEGrid Roadshow 2005

Geoserver Enhancements Initial stages of project included upgrading Geoserver to include plug in mapping to complex schemas and ability to configure pre-defined queries Info. xml and Schema. xml files Catch – only works from database source that you can run an SQL query against SEEGrid Roadshow 2005

info. xml SEEGrid Roadshow 2005

schema. xml SEEGrid Roadshow 2005

Mapping & Query Limitations Limited amount of branching Restricted in the positioning of branching No data dependent tags Only one predefined query per feature type SEEGrid Roadshow 2005

Coping with Limitations The impacts of the limitations of Geoserver and WFS were accommodated by § agreeing to slightly modified target schemas § agreeing to commonly defined input parameters and patterns § a small number of technical hacks SEEGrid Roadshow 2005

Geoserver Development The Geotools community is considering a comprehensive solution to the problems of complex objects. See http: //docs. codehaus. org/display/GEOTOOLS/Community+Schema+Support+and+Complex+ Types If you’re very keen get involved http: //geoserver. sourceforge. net/html/index. php SEEGrid Roadshow 2005

Technical Lessons https: //www. seegrid. csiro. au/twiki/pub/Infosrvices/MCAProject. Outputs/Technical. Learningfromthe. MCA. pd f SEEGrid Roadshow 2005

Baseline Software Requirements* Compliant with current WFS specification Separation of back end system from common interface Ability to restrict incoming queries *For equivalent of current project SEEGrid Roadshow 2005

Baseline Software Requirements Ability to access back end data source of choice (potentially through firewall) Ability to produce GML documents with at least single branching complexity, element attributes, and nestable single elements SEEGrid Roadshow 2005

Software Wishlist Ability to map to a GML document of any complexity and a GUI interface to configure the mapping Ability to run various queries against a single defined feature type Ability to map to an element based on a data source value SEEGrid Roadshow 2005

WFS Spec Changes As a result of the project there is a formal change request in with OGC to allow for query profiles for a feature This will also require a minor change in the Filter Encoding Specification SEEGrid Roadshow 2005

Moving forward • The SEEGRID Roadshow will extend the test bed to the remaining state and territory surveys for Geochemistry • Considering a 3 year ICIP (Industry Co-operative Innovation Program) grant which will take it from a test bed to a production system (network) § The Industry in ICIP is critical SEEGrid Roadshow 2005

Moving forward • The establishment of a production level network of services for one data type will greatly smooth the way for other types of info, not just Geological • Opportunities to cooperate with SEEGRID to progress technological and standards solutions § As part of ICIP proposal or independently • SEEGRID is free, contributions to common goals come from sub sets of the community SEEGrid Roadshow 2005

Use Cases and Service Structure for Clients in an Open Ended Architecture www. seegrid. csiro. au

Scenario 1 User wants to select sample data by some bounding box Sample data must be queryable by bounding box SEEGrid Roadshow 2005

Scenario 2 User wants to see details of data selected in bounding box Sample data must be queryable by bounding box or by id numbers of selected records SEEGrid Roadshow 2005

Scenario 3 User wants to select sample data by some bounding box and some threshold value Sample data must be queryable by bounding box and threshold value property SEEGrid Roadshow 2005

Client Requirements Availability, SLA’s Support Response times Data quality, metadata SEEGrid Roadshow 2005

Server Considerations Not all use cases/query profiles can be known But you can probably make a reasonable guess as to what may be required Most importantly, like any project, need to involve clients (if possible) SEEGrid Roadshow 2005

Client Strategies for Using WFS Querying everything vs multiple queries Schema dependency Caching vs going back to the web service SEEGrid Roadshow 2005

Further Technical Issues and Discussion www. seegrid. csiro. au

Technical Lessons https: //www. seegrid. csiro. au/twiki/pub/Infosrvices/ MCAProject. Outputs/Technical. Learningfromthe. MCA. pdf SEEGrid Roadshow 2005

WFS in the Notional Architecture is the end game perfect world A brief note – what are registries about? Imagine you’re an application! SEEGrid Roadshow 2005

Notional Architecture Ontology Feature Type Catalog Service Metadata Dataset Metadata Persistent Bindings publish find Registry Client Applications Processing Services bind Model Management Services Data Access Services Features Coverages Real-time data Models SEEGrid Roadshow 2005

Notional architecture Yellow Pages + Semantics Dictionary Feature Type Catalog Service Metadata MANAGED Dataset Metadata Persistent Bindings Managed Feature Type Catalogue Registry publish Feature Type Catalogue – List of Feature Types eg Spark Plugs, find Petrol Caps, Oil Filters Managed Service Metadata Feature Type - The agreed set of attributes a sparkplug must have Someone has vouched that a spare parts dealer service Thread Diameter, Thread. Metadata Managed Dataset Reach, Gap etc Client exists that supplies sparkplugs Model Someone – Add in the that from this service you can Service. Applications has vouchedyellow pages for the Management dealer Metadata spare parts Processing Services get a sparkplug that has a certain pedigree/quality Including details of how to contact (use) that Services service bind Metadata - it was made in Germany by Mercedes Benz Managed Persistent Bindings Data Access Services Someone has created instructions to use this service in the following ways Persistent Binding (ie always true) - only ever put the sparkplug the right way up For a Commodore you need sparkplug X Real-time Features Coverages data Models SEEGrid Roadshow 2005