OGSA-DAI Architecture EPCC, University of Edinburgh Amy Krause a. krause@epcc. ed. ac. uk International Summer School on Grid Computing - July 2003 Using OGSA-DAI Release 3 OGSA-DAI Training Workshop, Release 3
Overview 4 Grid. Services recap 4 OGSA-DAI overview 4 Scenarios 4 Components: – Design – Configuration 4 Component Interaction 2 OGSA-DAI Training Workshop, Release 3
OGSI Recap 4 Exploits existing web services properties – Interface abstraction (GWSDL resp. WSDL v 1. 2) – Protocol, language, hosting platform independence 4 Enhancement to web services – – – State Management Event Notification Referenceable Handles Lifecycle Management Service Data Extension See: The OGSI Specification (version 1. 0 at GGF 8) 3 OGSA-DAI Training Workshop, Release 3
Globus OGSI Implementation 4 Globus Toolkit 3 Release – June 03 4 OGSA-DAI Training Workshop, Release 3
The GT 3 Java Container WSDD J 2 EE wrappers also included with JBoss as EJB container 5 OGSA-DAI Training Workshop, Release 3
Globus Server Side Model!? You don’t have to be able to read this but understand that there is a set of classes that Globus define that support Grid Service instances 6 OGSA-DAI Training Workshop, Release 3
Anatomy Of A Grid Service • Service Data Access • Lifetime Management Grid. Service (required) Other Interfaces (Optional) • Service creation (Factory) • Service discovery (Registry) • Notification • Handle Management • Other functions e. g. Handle Element Service Data Element Grid Service • Workflow • Auditing • Resource Management Implementation Hosting Environment 7 OGSA-DAI Training Workshop, Release 3
OGSA Port Types 8 OGSA-DAI Training Workshop, Release 3
OGSA-DAI Port Types 9 OGSA-DAI Training Workshop, Release 3
Java Services 4 Service (Component) is implemented as a Java class 4 Implements the port. Type interfaces and extends some base class public class GDSService extends implements Grid. Service. Impl GDSPort. Type 4 Here GT 3. 0 Grid. Service. Impl implements common Grid. Service interface function 4 Other common functions are reused through delegation 4 This class is instantiated in order to create a service instance 10 OGSA-DAI Training Workshop, Release 3
The OGSA-DAI Project 4 OGSA - Data Access and Integration – Jointly funded by the UK DTI e. Science Programme and industry 4 Provides data access and integration functions for computing Grids using the OGSI framework. 4 Closely associated with GGF DAIS working group 4 Project team members drawn from – Commercial organisations and – Non-commercial organisations 4 Project runs until July 2003 – Support DB 2, Oracle, My. SQL, Xindice 11 OGSA-DAI Training Workshop, Release 3
Phase 1 4 Phase 1 – March to September 2002 – GGF DAIS Workgroup Grid Database Spec – Architectural Framework – Release 0 - Software Prototypes • EPCC (XML Database) – OGSI compliant • IBM UK (Relational Database) – non-OGSI – Functional Scope for Phase 2 12 OGSA-DAI Training Workshop, Release 3
Phase 2 4 Release 1 – Jan 2003 – Basic infrastructure and services. Combine the efforts of Phase 1 and get the team going in one direction 4 Release 2 – Apr 2003 – More functionality and changes to match Grid Service Specification as was then (now OGSI) 4 Release 3 – July 2003 – Final release of Phase 2 to coincide with the full Globus GT 3 release 13 OGSA-DAI Training Workshop, Release 3
Timeline A 2002 M J J A S O N D J F M A 2003 M J J Grid Services Spec – Draft 4 Globus Tech Preview 4 Grid Services Spec – Draft 5 Globus Tech Preview 5 Globus Toolkit 3 - Alpha OGSA-DAI Release 1 - Alpha OGSI Spec – v 1. 0 - Significant changes to OGSI OGSA-DAI Release 2 – Alpha update Globus Toolkit 3 - Beta Globus Toolkit 3 - Release OGSA-DAI Release 3 - Release A S O 14 OGSA-DAI Training Workshop, Release 3
Grid Technology Repository 4 Place for people to publish and discover work related to Grid Technologies 4 International community-driven effort 4 OGSA-DAI registered with the GTR – Visible UK contribution – Free publicity 4 More information from: – http: //gtr. globus. org 15 OGSA-DAI Training Workshop, Release 3
“Buy not Build” 4 OGSA/OGSI 4 Query Language 4 Data Format 4 Data transport 4 Data Description Schema 4 Replication 4… 16 OGSA-DAI Training Workshop, Release 3
10000 Feet Grid Data Resources Client Consumer 17 DBMS OGSA-DAI Training Workshop, Release 3
10000 Feet With OGSA-DAI Services Grid Data Resources GDSF Client GDS DBMS Consumer DAISGR 18 OGSA-DAI Training Workshop, Release 3
1 a. Request to Registry for sources of data about “x” Registry DAISGR 1 b. Registry responds with Factory handle 2 a. Request to Factory for access to database Analyst OR 3 d. Results of query delivered to consumer via FTP, GFTP, … Consumer service creation API interactions Factory GDSF 2 c. Factory returns handle of GDS to client 2 b. Factory creates Grid. Data. Service to manage 3 a. Client queries GDS with SQL, XPath, XQuery access etc 3 c. Results of query returned to client as XML 19 SOAP/HTTP Grid Data Service GDS 3 b. GDS interacts with database Database (Xindice My. SQL Oracle DB 2) OGSA-DAI Training Workshop, Release 3
OGSA-DAI Basic Services OGSA-DAI Distributed Query OGSA-DAI Basic Services Delivery Data Format Query GDS (Create Retrieve Drivers Update Delete) OGSA GDSF Meta Data Notification DAISGR Lifetime Location Database, Communication, OS… Technology 20 OGSA-DAI Training Workshop, Release 3
Location Registry DAISGR find. Service. Data register. Service Factory Analyst find. Service. Data GDSF 4 Data resource publication through registry 4 Data location hidden by factory 4 Data resource meta data available through Service Data Elements 21 OGSA-DAI Training Workshop, Release 3
Heterogeneity Grid Data Service Xindice My. Sql Oracle DB 2 4 Data source abstraction behind GDS instance – Plug in “data resource implementations” for different data source technologies – Does not mandate any particular query language or data format 22 OGSA-DAI Training Workshop, Release 3
Scale Analyst Producer/ Consumer Request Deliver Grid Data Service 4 Delivery configured as part of request 4 Asynchronous delivery with varying modes/transports – “Zero copy deliver” 4 OGSA-DAI will not specify transport mechanism but support existing 23 OGSA-DAI Training Workshop, Release 3
Flexibility Request Doc Grid Data Service Analyst Response Doc Query Update Delivery 4 Data source abstraction behind GDS instance – Document based interface • Document sharing, operation optimization – Combines statement with other, plugin, operations/activities • delivery, data transformation, data caching – Ongoing activity is represented in state of the service • running query, cached data, referenced data 24 OGSA-DAI Training Workshop, Release 3
Dynamism Registry Analyst Notification Factory Grid Data Service 25 OGSA-DAI Training Workshop, Release 3
Management, Ownership, Accounting etc. 4 We rely on OGSA/I for much common distributed computing function 4 Any OGSA-DAI specific function will be compatible with OGSA/I approach 4 Not much has been done to date 26 OGSA-DAI Training Workshop, Release 3
GDS Composition 1 Client GDS Operation GDS 3 Client GDS Operation Client Operation GDS DB Operation GDS 4 Operation GDS 2 GDS DB Operation GDS 5 DB GDS Operation GDS Client Operation DB Operation 27 OGSA-DAI Training Workshop, Release 3 DB
Release 1 4 Simple synchronous interaction with a data source using a GDS as a proxy. SGR – Service. Group. Registration port. Type GS – Grid. Service port. Type F – Factory port. Type GDS – GDS port. Type Registry SGR GS F 1 Client & Consumer Factory GS 2 Q Rs 3 GDS Instance DB GS 28 OGSA-DAI Training Workshop, Release 3
Release 3 4 Asynchronous delivery – Pull Q Client 2 GDS Instance GDS 1 Rs DT D + GDH GSH/R + data id DB 3 Ra Consumer GDT 4 Asynchronous delivery – Push Q + D + GSH/R Client 1 GDS Rs 2 DT GDT Consumer 29 GDS Instance DB GSH/R 3 Ra OGSA-DAI Training Workshop, Release 3
Notation 30 OGSA-DAI Training Workshop, Release 3
Overview – Release 3 (R 3) 31 OGSA-DAI Training Workshop, Release 3
Scenario 1 (synchronous delivery) 4 An analyst wants to perform a SQL query across a dataset with a known name and schema – – – Container starts Analyst Starts Analyst identifies factory that supports required statement type Analyst uses factory to create GDS instance and obtains GSH Analyst maps GSH to GSR using factory Analyst formulates a GDS perform document containing the query – Analyst passes GDS perform document to GDS instance – GDS instance returns data in response – Analyst removes GDS instance 32 OGSA-DAI Training Workshop, Release 3
Scenario 2 (asynchronous delivery) 4 An analyst wants to perform an XPath query across a dataset with a known name and schema – – – – – 33 Container starts Analyst Starts Analyst identifies factory that supports required statement type Analyst uses factory to create GDS instance and obtains GSH Analyst maps GSH to GSR using factory Analyst formulates a GDS perform document containing the query and the URL of the consumer Analyst passes GDS perform document to GDS instance returns report to analyst GDS instance delivers data to specified consumer Analyst removes GDS instance OGSA-DAI Training Workshop, Release 3
Container Start create DSGR 1 GS Container create SGR SG GDSF 1 GS NSrc RDBMS (My. SQL) F HR Northern Hemisph ere. IR C A create C 1 GS GDT NSnk 34 GDSF 2 GS F HR XMLDB (Xindice) Southern. He misphere. IR OGSA-DAI Training Workshop, Release 3
DAIService. Group. Registry 4 Allows OGSA-DAI services to: – Make clients aware of their existence. – Make clients aware of their capabilities, services or the data resources they manage. – Be shared amongst multiple clients. 4 Allows clients to: – Search for DAI services meeting their requirements. 35 OGSA-DAI Training Workshop, Release 3
Port. Types 4 Most-derived port. Type: – DAIService. Group. Registry. 4 Aggregates OGSI port. Types: – Grid. Service: • Query registered services via find. Service. Data. – Notification. Source: • Subscribe to changes in DAISGR state via subscribe. – Service. Group: • Group together DAI services. – Service. Group. Registration: • Add and remove DAI services to and from the DAISGR via add and remove. 36 OGSA-DAI Training Workshop, Release 3
Grid. Data. Service. Factory 4 Exposes a data resource to clients. 4 Allows clients to request creation of Grid Data Services which can be used to interact with the data resource. 37 OGSA-DAI Training Workshop, Release 3
Grid. Data. Service. Factory Port. Types 4 Most-derived port. Type: – Grid. Data. Service. Factory. 4 Aggregates OGSI port. Types: – Grid. Service: • Query the data resource exposed by the GDSF via find. Service. Data. – Factory: • Create a GDS to allow interaction with a data resource via create. Service. – Notification. Source: • Subscribe to changes in DAISGR state via subscribe. 38 OGSA-DAI Training Workshop, Release 3
Grid. Data. Service Port. Types 4 Most-derived port. Type: – GDSPort. Type – Grid. Data. Service 4 Aggregates OGSI and OGSA-DAI port. Types: – Grid. Service: • Query the data resource exposed by the GDSF via find. Service. Data. – Grid. Data. Perform: • Interact with the data resource represented by the GDS via perform. – Grid. Data. Transport • Give data to or receive data from the GDS data either in one complete chunk or in separate sub-chunks via put. Fully, put. Block, get. Fully and get. Block. 39 OGSA-DAI Training Workshop, Release 3
Behind the scenes: Data Resources 4 Data Resources in OGSA-DAI represent a data source/sink 4 Data Resources are typified by: – Way of communicating with the data resource – Location, i. e. properties about the container managing access to the data source/sink and information about its capabilities – The actual data source/sink – The resource, an instantiation/view/sample obtained from the data source/sink 40 OGSA-DAI Training Workshop, Release 3
Data Resources in OGSA-DAI 4 An OGSA-DAI Factory is configured with exactly one data resource – Done in the factory configuration file – Data resource confined to a static named object defined in the Factory configuration file – In the future hope to make this more dynamic 4 A GDS created by a factory – Can only be associated with the data resource known to the factory – Can only be associated with one data resource 41 OGSA-DAI Training Workshop, Release 3
WSDD Container Config 4 Creates persistent registry 4 Creates persistent factory – Defines configuration files to read in OGSA-DAI Container DAISGR GDSF DR 42 GDSF Config file DR GDSR Regist. list OGSA-DAI Training Workshop, Release 3
WSDD Container Config <service name="ogsadai/Grid. Data. Service. Factory" provider="Handler" style="wrapped" use="literal"> <parameter name="ogsadai. gdsf. config. xml. file" value="data. Resource. Config. Rel. xml"/> <parameter name="ogsadai. gdsf. registrations. xml. file“ value="registration. List. xml"/> <parameter name="name" value="Grid Data Service Factory"/> <parameter name="operation. Providers“ value="org. globus. ogsa. impl. ogsi. Factory. Provider"/> <parameter name="persistent" value="true"/> <parameter name="instance-schema. Path" value="schema/ogsadai/gds_service. wsdl"/> <parameter name="instance-base. Class. Name" value="uk. org. ogsadai. service. gds. Grid. Data. Service"/> <parameter name="base. Class. Name" value="uk. org. ogsadai. service. gdsf. Grid. Data. Service. Factory"/> <parameter name="schema. Path" value="schema/ogsadai/gdsf/grid_data_service_factory_service. wsdl"/> <parameter name="handler. Class" value="org. globus. ogsa. handlers. RPCURIProvider"/> <parameter name="instance-name" value="Grid Data Service"/> <parameter name="class. Name" value="uk. org. ogsadai. wsdl. gdsf. Grid. Data. Service. Factory. Port. Type"/> <parameter name="allowed. Methods" value="*"/> <parameter name="factory. Callback" value="uk. org. ogsadai. service. gdsf. Grid. Data. Service. Factory. Callback"/> <parameter name="activate. On. Startup" value="true"/> </service> 43 OGSA-DAI Training Workshop, Release 3
Factory Configuration XML 4 Defines components that constitute a data resource – Data. Resource. Manager: contains DBMS specifics, such as driver class and physical location, and can implement connection pooling – Role. Maps: maps grid credentials to database roles – Data. Resource. Metadata: metadata such as product information and relational or XMLDB specific information – Activity. Maps: activities i. e. operations supported by the data resource; each activity is mapped to its implementing class and a schema 44 OGSA-DAI Training Workshop, Release 3
Factory Configuration XML Skeleton <data. Resource. Config xmlns="http: //ogsadai. org. uk/namespaces/2003/07/gdsf/config"> <documentation> A sample config file. </documentation> <activity. Map name="sql. Query. Statement>. . . </activity. Map> <data. Resource. Metadata> . . . </data. Resource. Metadata> <role. Map name="Name". . . /> <driver. Manager. . . > <driver>. . . </driver> </driver. Manager> </data. Resource. Config> 45 OGSA-DAI Training Workshop, Release 3
Driver Manager 4 Driver. Manager objects encapsulate the data resource, e. g. – Provide connection pooling to databases – Allows a single collection of objects to be shared across any number of GDS instances – GDS connection capabilities to generate dynamic information capabilities, e. g. obtain the database schema 4 GDSF constructs and populates these objects 4 The Driver. Manager mapping element relates the data resource defined in the GDSF configuration file to a Java implementation class 4 Currently have generic classes for – JDBC databases – XML: DB databases (i. e. Xindice) 46 OGSA-DAI Training Workshop, Release 3
Data Resource Implementation Mapping 47 OGSA-DAI Training Workshop, Release 3
Factory Configuration: Driver. Manager <driver. Manager driver. Manager. Implementation="uk. org. ogsadai. porttype. gds. dataresource. Simple. JDBCData. Resource. Implementation"> <driver> <driver. Implementation>org. gjt. mm. mysql. Driver</driver. Implementation> <driver. URI> jdbc: mysql: //localhost: 3306/ogsadai </driver. URI> </driver> </driver. Manager> 48 OGSA-DAI Training Workshop, Release 3
Factory Configuration: Data. Resource. Metadata <data. Resource. Metadata> <product. Info> <!-- This element and its contents are optional. --> <product. Name>My. SQL</product. Name> <product. Version>4</product. Version> <vendor. Name>My. SQL</vendor. Name> </product. Info> <relational. Meta. Data> <database. Schema callback="uk. org. ogsadai. porttype. gds. dataresource. Simple. JDBCMeta. Data. Extractor" /> </relational. Meta. Data> <!-- User can define own metadata --> </data. Resource. Metadata> 49 OGSA-DAI Training Workshop, Release 3
Activities 4 Activities are tasks/operations that can be performed by a GDS on a data resource – Clearly data resources can support subset of activities, e. g. cannot run an SQL query on a Xindice database – The Factory identifies the activities supported by the data resource at configuration time 50 OGSA-DAI Training Workshop, Release 3
Activity Mapping 4 The Activity Map file relates each named activity to – a Java implementation class – XML Schema that corresponds to activity 4 Maps activities to data resources – Unless you are writing your own activity you should not need to modify this file 51 OGSA-DAI Training Workshop, Release 3
Activity Mapping II 52 OGSA-DAI Training Workshop, Release 3
Activity Map Example <activity. Map name="sql. Update. Statement" implementation="uk. org. ogsadai. …. SQLUpdate. Statement. Activity" schema. File. Name="http: //localhost: 8080/…/sql_update_statement. xsd"/> <activity. Map name="sql. Stored. Procedure" implementation="uk. org. ogsadai. …. SQLStored. Procedure. Activity" schema. File. Name="http: //localhost: 8080/…/sql_stored_procedure. xsd"/> <activity. Map name="deliver. From. URL“ class="uk. org. ogsadai. …. Delivery. From. URLActivity“ schema. File. Name="http: //localhost: 8080/…/deliver_from_url. xsd" /> <activity. Map name="deliver. To. URL" class="uk. org. ogsadai. …. Delivery. To. URLActivity“ schema. File. Name=" http: //localhost: 8080/…/ deliver_to_url. xsd" /> 53 OGSA-DAI Training Workshop, Release 3
Factory Configuration: Role. Maps 4 Rolemapper maps grid credentials to database roles 4 Java implementation Simple. Rolemapper is provided with the release: – maps the distinguished name of the user to a username and password – Username and password are provided in a separate file <role. Map name="Simple. Rolemapper" implementation="uk. …. Simple. File. Role. Mapper“ configuration="examples/Example. Database. Roles. xml" /> 54 OGSA-DAI Training Workshop, Release 3
Factory Registration 4 Through meta-data (SDEs) factory exposes – details from the configuration file, i. e. • data manager information • activities supported • relational metadata: database schema – Metadata about components (not shown earlier) 4 Registration file allows GDSF to register with a DAISGR 55 OGSA-DAI Training Workshop, Release 3
Factory Registration. List <gdsf: gdsf. Registration. List … > <gdsf: gdsf. Registration name="default. Registration“ gsh="http: //localhost: 8080/ogsa/services/ogsadai/Gri d. Data. Service. Registry"/> <!-- can have more entries here --> </gdsf: gdsf. Registration. List> 56 OGSA-DAI Training Workshop, Release 3
Analyst Starts and Identifies Factory DAISGR 1 GS SGR SG NSrc C A read C 1 GS Analyst Configuration has GSH of DAISGR GDT NSnk 57 OGSA-DAI Training Workshop, Release 3
Registry Query 4 Query for registered – Grid. Services – Grid. Data. Service. Factories 4 XPath queries possible, for example – //path/data[@name=“Northern. Hemisphere. IR”] 4 Registry must be able to apply this and resolve it to a matching factory instance 4 Factory registers its GSH on startup (if specified in the configuration) 58 OGSA-DAI Training Workshop, Release 3
Analyst Uses Factory Instance To Create GDS Instance GDSF 1 RDBMS (my. SQL) GS GSH create. Service (termination. Time, creation. Parameters) F create GDS 1 GS Northerm Hemisphe re. IR GDS A 1 C 1 GDT GS GDT NSnk 59 OGSA-DAI Training Workshop, Release 3
GDSF Creation Parameters 4 In Release 3 the creation parameters are empty 4 GDSF is associated with exactly one Data Resource 4 GDSF will create a GDS configured for this Data Resource 60 OGSA-DAI Training Workshop, Release 3
GDSF Configures GDS Instance 4 GDS is configured using information from the GDSF configuration 4 Interfaces used to configure GDS are not exposed – They are particular to the implementation of GDSF and GDS 4 Client requests actions to be taken by the GDS on the data resource by using a GDS-Perform document 61 OGSA-DAI Training Workshop, Release 3
Analyst maps GDS GSH GDSF 1 GS RDBMS (my. SQL) F GSH GDS 1 DB 1 GS GDS A 1 C 1 GDT GS GDT NSnk 62 OGSA-DAI Training Workshop, Release 3
GDS-Perform document 4 GDS Perform document contains activities and an optional documentation element 4 Output from one activity can be used by another activity 4 Any hanging outputs will be delivered with the SOAP response (synchronous) 4 Using delivery activities, the output of a query can be delivered asynchronously (via HTTP, FTP, Grid. FTP) 63 OGSA-DAI Training Workshop, Release 3
Analyst Formulates Query As GDS Perform Document <grid. Data. Service. Perform xmlns="http: //ogsadai. org. uk/namespaces/2003/07/gds/types“ > <documentation> Select with data delivered with the response request stored then executed. </documentation> <sql. Query. Statement name="statement"> <expression> select * from littleblackbook where id=10 </expression> <web. Row. Set. Stream name="statementresult"/> </sql. Query. Statement> </grid. Data. Service. Perform> 64 OGSA-DAI Training Workshop, Release 3
GDS Perform Document Schema 4 The WSDL for the GDS port. Type specifies the general schema that the perform method accepts 4 The complex type Activity. Type forms a base for extension by all activities 4 The GDS configuration defines the operations that a GDS will perform 4 The GDS will generate the GDS perform document schema on request based on the specified configuration 65 OGSA-DAI Training Workshop, Release 3
Analyst Passes Request to GDS and Retrieves Data From Response RDBMS (my. SQL) GDS 1 FRED GS GDS A 1 C 1 GDS perform (perform. Document) GDT GS GDT NSnk 66 OGSA-DAI Training Workshop, Release 3
GDS Response Documents GDS response document contains: 4 A named response element referencing a request 4 For each activity in the request, a result element, referencing the name of the activity, which contains the result data – – 67 sql. Query. Statement x. Path. Statement zip. Archive … OGSA-DAI Training Workshop, Release 3
The Data In The Response <grid. Data. Service. Response xmlns="http: //ogsadai. org. uk/namespaces/2003/07/gds/types"> <result name="statement" status="COMPLETE"/> <result name="statementresult" status="COMPLETE"> <
Analyst Removes GDS Instance 4 This is done either – by the GDS instance itself when the lifetime expires, i. e. • the container removes any Grid services whose lifetimes have expired – directly through the “Destroy” method 69 OGSA-DAI Training Workshop, Release 3
To Date 4 Have assumed that OGSA/OGSI is a good thing – OGSA-DAI – Have adopted the OGSI approach 4 Have first concentrated on data access – Data integration, for example, distributed query, pipelines, comes later 4 Working Closely with GGF DAIS Working Group on Grid Database Service Specification 4 Intentions to be a reference implementation 70 OGSA-DAI Training Workshop, Release 3
OGSA-DAI http: //ogsadai. org. uk/ 4 Releases 4 Support from the UK Grid Support Centre 71 OGSA-DAI Training Workshop, Release 3
Скачать презентацию OGSA-DAI Architecture EPCC University of Edinburgh Amy Krause
52a62d066a3f29b65bc6aece8b0649ee.ppt