SERVOGrid Basics • SERVOGrid is our project to build a distributed computing infrastructure to support earthquake simulation codes. – We use Web Services for this. – We build portal interfaces to these web services out of portlets. • Portlets for Java portals are standardized (JSR 168) • All major commercial vendors, many open source projects support. • GEON, NEESGrid, LEAD portal use also. • Distributed Computing Infrastructure – Services for remotely executing applications, managing distributed files in logical units, monitoring applications, archiving user projects, and so on. – Variously called “Grids” (DOE), “Cyberinfrastructure” (NSF), “e-Science” (UK e. Science program), …. • The above is sometimes referred to as an “Execution Grid”. – We designed this around a web service architecture. – See links at the end for more information. • But what about the data? – Need ways to access remote data sources through programming interfaces. – Practice often is to download (via FTP) entire catalogs and hand edit to get data you want in the format you want. – We need to automate this. 1
Some SERVO Apps and Their Data • Disloc: handles multiple arbitrarily dipping dislocations (faults) in an elastic half-space. – Relies upon geometric fault models. • Geo. FEST: Three-dimensional viscoelastic finite element model for calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, material properties, and body forces. – Relies upon fault models with geometric and material properties. • Virtual California: Program to simulate interactions between vertical strike-slip faults using an elastic layer over a viscoelastic half-space. – Relies upon fault and fault friction models. • RDAHMM: Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities for transitioning from one class to another. – Used to analyze GPS and seismic catalogs. 2
Where Is the Data? • Quake. Tables Fault Database – SERVO’s fault repository for California. – Compatible with Geo. FEST, Disloc, and Virtual. California – http: //infogroup. usc. edu: 8080/public. html • GPS Data sources and formats (RDAHMM and others). – JPL: ftp: //sideshow. jpl. nasa. gov/pub/mbh – SOPAC: ftp: //garner. ucsd. edu/pub/timeseries – USGS: http: //pasadena. wr. usgs. gov/scign/Analysis/plotdata/ • Seismic Event Data (RDAHMM and others) – SCSN: http: //www. scec. org/ftp/catalogs/SCSN – SCEDC: http: //www. scecd. scec. org/ftp/catalogs/SCEC_DC – Dinger-Shearer: http: //www. scecdc. org/ftp/catalogs/dingershearer/dinger-shearer. catalog – Haukkson: http: //www. scecdc. scec. org/ftp/catalogs/hauksson/Socal 3
What Do We Want to Do? • • • Web Services provide a programming interfaces to data so that you can build remote clients. SERVO portal work houses these Web Service clients. Portal also houses the clients for the remote simulation services (RDAHMH, Geo. FEST, Potts model codes, etc). – As we have seen, these codes work directly with data catalogs • Typically the codes must process and reformat the data catalog entries in code-specific ways. – Done by hand by most developers, which is fine for development. – But not fine for mature codes or automated applications. • Currently, our solution is to harvest data from online catalogs and transform into GIS compatible formats in DBs. – Discuss a couple of examples • But we would like to experiment with building services that are closer to the source. – Finer resolution data products – Academic exercise in implementing reliable sensor network services and streaming applications. • Sensor. ML family from Open Geospatial Consortium provides the standards. 4
Open GIS Services • GML abstract data models can encode data but you need services to interact with the remote data. • Some example OGC services include – Web Feature Service: for retrieving GML encode features, like faults, roads, county boundaries, GPS station locations, …. – Web Map Service: for creating maps out of Web Features – Sensor Grid Services: for working with streaming, time-stamped data. • Problems with OGC services – Not (yet) Web Service compliant • “Pre” web service, no SOAP or WSDL • Use instead HTTP GET/POST conventions. – Often define general Web Service services as specialized standards • Information services • Notification services in sensor grids 5
Using GML to Encode GPS • Collected GPS data for each site is made available through online catalogs. – Using various text formats. • We have developed GML descriptions of GPS using Feature. xsd schema. • Full GML GPS schema: http: //www. crisisgrid. org/files/g ps. xsd. • Supports JPL, SOPAC, and USGS formats. • Other efforts: Feature encodings of Seismic and Fault catalogs. – http: //www. crisisgrid. org/html/s ervo_-_gps. html 6
Anatomy of Web Feature Service • GML provides the data model, but we must still provide a remote programming interface to the data. • Web Feature Service provides three major services as described in OGC specification: • Get. Capabilities: The clients (WMS servers or users) starts with requesting a document from WFS which describes it’s abilities. When a get. Capabilities request arrives, the server dynamically creates a capabilities document and returns this. • This is OGC’s formalization of metadata, so important to GEON, ESG, etc. • Describe. Feature. Type: After the client receives the capabilities document he/she can request a more detailed description for any of the features listed in the WFS capabilities document. • The WFS returns an XML schema that describes the requested feature. • Metadata about a specific entry. • Get. Feature: The client can ask the WFS to return a particular portion of any feature data. • Get. Feature requests contain some property names of the feature and a Filter element to describe the query. • The WFS extracts the query and bounding box from the filter and queries the feature databases. • The results obtained from the DB query are converted the feature’s GML format and returned to the client as a Feature. Collection. 7
Web Map Services • OGC Web Map Servers create digital maps from abstract data sets retrieved from Web Feature Services. – Layer features are overlayed. • WMS servers can be daisy chained. • Our research efforts here are to build Web Service version of the WMS as well as portlet clients. • We are developing backward compatibility with existing WMS. – Purdue CAAGIS for Indiana maps. – NASA JPL On. Earth WMS. • Future plans are to replace current GMT portal tools with WMS, combine maps with Geo. FEST and Virtual California application output. 8
Next Steps and Questions (Jan) • We are very interested in starting a new Sensor Grid project. • – Implement Open Geospatial Consortium standards for Sensor Grids (see supplemental slides). – We are outlining a more ambitious integration of Robert Granat’s RDAHMM with GPS data. • • • I will be visiting JPL in early February. • Some questions we have are on the right. • • • Direct (or at least closer) access to GPS stations? Any documentation on the filtering process? What kind of filters are applied on the raw data? Can it be automated? How is data stored, in a DB or just in files? How do the stations communicate with the servers? What protocols are being used for transport, etc? Are there any web-based interfaces to stations or to the data stores? Are the GPS stations configurable? How often do they send out data, can you change the interval? What are the most important data formats for you? (JPL, SOPAC, USGS) Are there any other formats we need to support? What (other) kinds of applications are there to use the GPS data? 9
HPSearch Overview • HPSearch is a system for scripting and controlling web services and controlling data streams. – Uses Java. Script (Rhino) for scripting – Uses Narada. Brokering for controlling communications between services. – Name is historical • Basic premise is that it can be used to hook together streams of data to successive filters (services) that are controlled through a client environment. 10
Basic HPSearch Parts • Kernel Components – Shell: processes the extended Java. Script – Service Manager: this is a scheduler for load balancing between several services. – Request. Handler: monitors individual remote resources (may include web services but also DBs or plain old URLs). • WSProxy facilitates wrapping legacy applications as streaming web services – Manages both the control invocations from the shell and the NBbased streaming data connections – Comes with two interfaces • Runnable: you extend this Java interface for new services. • Wrapped: this is a legacy/non-WS service such as a perl or matlab script. 11
HPSearch Architecture HPSearch Kernel Data is read / written from / to various data resources Files, Sockets, Topics Request Handler Javascript Shell URIHandler Service Manager Data. Base DBHandler Web Service EP Other Objects Web Service WSDLHandler WSProxy Servic e Broker Network WSProxy Service Data Flows as a stream of messages between WSProxies through the brokering network Stream Proxy Kernel connects to nearest broker and communicates with other kernels with messages HPSearch Kernel Handlers steer web services via simple SOAP requests 12
HPSearch for RDAHMM + GPS Sensor Network / GPS Database Start Activity, Web Service Virtual Data Flow between various components Narada Broker Data Filter Links between Brokers WSProxy wrapped service HPSearch Kernel Actual Data Flow between components takes place through the brokering network Kernel manages (steers) the various services RDAHMM Matlab Script Service distribution 13
More Information • My email: – mpierce@cs. indiana. edu • Overview (submitted to ACES special issue of PAGEOPH). – http: //www. servogrid. org/slide/GEM/SERVO/ISERVO_ACES_PAGEOPH. doc • Quake. Sim Project Page: – http: //quakesim. jpl. nasa. gov/ • Quake. Sim Portal: – http: //www. complexity. ucs. indiana. edu: 8282. • Quake. Tables Fault Database: – http: //infogroup. usc. edu: 8080/public. html • Community Grids Lab GIS Grid Development: – – http: //www. crisisgrid. org/. See particularly http: //www. crisisgrid. org/html/servo. html for GML information. See http: //www. crisisgrid. org/html/wfs. html for more on Web Feature Service. WMS and Information Services information coming soon. 14
Supplemental Slides: Sensor Web Work Slides Prepared by Galip Aydin 15
Sensor Web • Goal: To create Web-based sensor networks by exploiting web-connected sensors (flood gauges, air pollution monitors etc. ). • This would make all sensors and repositories of sensor data discoverable, accessible and where applicable controllable via the WWW. • OGC describes 5 important encoding and service standards for a Sensor Web Implementation (see next slide). 16
OGC-Sensor Web Enablement 1. 2. 3. 4. 5. Sensor. ML: XML encoding language for sensors. Used to discover, query and control Web-resident sensors. Observations & Measurements: The general models and an XML encoding for what a sensor observes or measures (The value returned by or derived from a sensor observation -e. g. quantity, count, boolean, category, ordered category, position-). Sensor Collection Service: A service to fetch observations from a sensor or constellation of sensors. Provides real time or archived observed values. Clients can also obtain information that describes the associated sensors and platforms. Sensor Planning Service: A service by which a client can determine collection feasibility for a desired set of collection requests for one or more mobile sensors/platforms, or the client may submit collection requests directly to these sensors/platforms. SPS enables sensor tasking, acquisition requests, processing and simulation requests, and registration for alert notification. Web Notification Service: A service by which a client may conduct a dialog with one or more other services. Provides a means for Sensor Planning Services to alert people, software, or other sensor systems of SPS results or alerts regarding phenomena of interest. 17
OGC-SWE Pieces 18
Sensor Web Enhancements 1 • We think that with some changes and additions a minimal and high performance Sensor Web with live sensor data capability can be realized. • We realize that some of these service specifications are trying to solve problems common to the Web Services community such as notification. • There are several Notification specifications for Web Services such as WS-Notification and WS-Eventing. Considering the industrial backing and huge WS community support we can expect these specifications to evolve into standards. • Community Grids lab has considerable experience on Notification issues and currently a WS-Notification enabled version of Narada. Brokering is being developed. 19
Sensor Web Enhancements 2 • Sensor Planning service is designed as an asset management service. For our purposes existence of such a service is not crucial instead an Information Service with search and discovery capabilities is preferable. • We are developing a High Performance Fault Tolerant Information Service for GIS applications and with little modifications this service can handle the functionality expected from Sensor Planning Service. • The Information Service will provide search and discovery interfaces and WSDL addresses for Sensor Collection Services. 20
Sensor Web Enhancements 3 • In some cases immediate access to the sensor data is crucial. In such cases we expect to have some sort of a streaming data channel for live sensor data available to the clients. • We plan to use Narada Brokering to provide us such a capability. • Narada Brokering allows message producers to publish messages to topics and subscribers of these topics immediately receives the published messages. • We can create topics according to the types of the sensors and subscribe clients to these topics according to their preferences. Thus Narada Brokering can provide a streaming channel between sensors and the clients. 21
A sample Sensor Web that utilizes Information Service and Narada Brokering. 22
Narada. Brokering Overview • NB is software for routing messages that has been developed by Shrideep Pallickara at CGL for several years. – Topic-based publish/subscribe system • Implements JMS, JXTA, and its own APIs. • Allows you to construct hierarchical broker networks for efficient routing. • Can route “anything” – – – HTTP, TCP/IP, UDP Binary data for collaborative shared display (www. anabas. com) Grid. FTP via proxy SOAP messages WS-Eventing and WS-Reliable. Messaging • Moving toward supporting SOAP 1. 2 -style messaging. 23
Скачать презентацию SERVOGrid Basics SERVOGrid is our project to
fe7de5fc1d6bda38c17981d51ffc6548.ppt