Скачать презентацию 1 GT 4 WSRF Core and MDS 4 Скачать презентацию 1 GT 4 WSRF Core and MDS 4

7e7b0d53427adb535739aef6333f73d6.ppt

  • Количество слайдов: 37

1 GT 4 WSRF Core and MDS 4 Gabor Kecskemeti • MTA SZTAKI, Hungary 1 GT 4 WSRF Core and MDS 4 Gabor Kecskemeti • MTA SZTAKI, Hungary • Univ. Westminster, UK kecskemeti@sztaki. hu

Globus Toolkit: Open Source Grid Infrastructure Data Replication Globus Toolkit v 4 www. globus. Globus Toolkit: Open Source Grid Infrastructure Data Replication Globus Toolkit v 4 www. globus. org Credential Mgmt Replica Location Grid Telecontrol Protocol Delegation Data Access & Integration Community Scheduling Framework Web. MDS Python Runtime Community Authorization Reliable File Transfer Workspace Management Trigger C Runtime Authentication Authorization Grid. FTP Grid Resource Allocation & Management Index Java Runtime Security Data Mgmt Execution Mgmt Info Services Common Runtime 2

3 Monitoring and Discovery • “Every service should be monitorable and discoverable using common 3 Monitoring and Discovery • “Every service should be monitorable and discoverable using common mechanisms” – WSRF/WSN provides those mechanisms • A common aggregator framework for collecting information from services, thus: – MDS-Index: Xpath queries, with caching – MDS-Trigger: perform action on condition – (MDS-Archiver: Xpath on historical data) • Deep integration with Globus containers & services: every GT 4 service is discoverable – GRAM, RFT, Grid. FTP, CAS, …

4 GT 4 Monitoring & Discovery Clients (e. g. , Web. MDS) WS-Service. Group 4 GT 4 Monitoring & Discovery Clients (e. g. , Web. MDS) WS-Service. Group GT 4 Container Registration & WSRF/WSN Access GT 4 Container MDSIndex Automated registration in container GRAM MDSIndex adapter GT 4 Cont. Custom protocols for non-WSRF entities MDSIndex Grid. FTP User RFT

5 Monitoring and Discovery System (MDS 4) • Grid-level monitoring system – Aid user/agent 5 Monitoring and Discovery System (MDS 4) • Grid-level monitoring system – Aid user/agent to identify host(s) on which to run an application – Warn on errors • Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification – WS-Resource. Properties, WS-Base. Notification, WSService. Group • Functions as an hourglass to provide a common interface to lower-level monitoring tools

6 Information Users : Schedulers, Portals, Warning Systems, etc. WS standard interfaces for subscription, 6 Information Users : Schedulers, Portals, Warning Systems, etc. WS standard interfaces for subscription, registration, notification Standard Schemas (GLUE schema, eg) Cluster monitors (Ganglia, Hawkeye, Clumon, and Nagios) Services (GRAM, RFT, RLS) Queuing systems (PBS, LSF, Torque)

7 MDS 4 Components • Information providers – Monitoring is a part of every 7 MDS 4 Components • Information providers – Monitoring is a part of every WSRF service – Non-WS services are also be used • Higher level services – Index Service – a way to aggregate data – Trigger Service – a way to be notified of changes – Both built on common aggregator framework • Clients – Web. MDS • All of the tools are schema-agnostic, but interoperability needs a well-understood common language

8 Higher-Level Services • Index Service – Caching registry • Trigger Service – Warn 8 Higher-Level Services • Index Service – Caching registry • Trigger Service – Warn on error conditions • Archive Service – Database store for history (in development) • All of these have common needs, and are built on a common framework

9 Common Aggregator Framework • Basic framework for higher-level functions – Subscribe to Information 9 Common Aggregator Framework • Basic framework for higher-level functions – Subscribe to Information Provider(s) – Do some action – Present standard interfaces

10 Aggregator Framework Features 1) Common configuration mechanism – Specify what data to get, 10 Aggregator Framework Features 1) Common configuration mechanism – Specify what data to get, and from where 2) Self cleaning – Services have lifetimes that must be refreshed 3) Soft consistency model – Published information is recent, but not guaranteed to be the absolute latest 4) Schema Neutral – Valid XML document needed only

11 MDS 4 Index Service • Index Service is both registry and cache – 11 MDS 4 Index Service • Index Service is both registry and cache – Datatype and data provider info, like a registry (UDDI) – Last value of data, like a cache • In memory default approach – DB backing store currently being developed to allow for very large indexes • Can be set up for a site or set of sites, a specific set of project data, or for user-specific data only • Can be a multi-rooted hierarchy – No *global* index

12 MDS 4 Trigger Service • Subscribe to a set of resource properties • 12 MDS 4 Trigger Service • Subscribe to a set of resource properties • Evaluate that data against a set of preconfigured conditions (triggers) • When a condition matches, action occurs – Email is sent to pre-defined address – Website updated • Similar functionality in Hawkeye

13 Web. MDS User Interface • Web-based interface to WSRF resource property information • 13 Web. MDS User Interface • Web-based interface to WSRF resource property information • User-friendly front-end to Index Service • Uses standard resource property requests to query resource property data • XSLT transforms to format and display them • Customized pages are simply done by using HTML form options and creating your own XSLT transforms • Sample page: – http: //mds. globus. org: 8080/webmds? info=inde xinfo&xsl=servicegroupxsl

14 Example Web. MDS screenshot 14 Example Web. MDS screenshot

15 Information Providers • GT 4 information providers collect information from some system and 15 Information Providers • GT 4 information providers collect information from some system and make it accessible as WSRF resource properties • Growing number of information providers – Ganglia, Clu. Mon, Nagios – SGE, LSF, Open. PBS, PBSPro, Torque • Many opportunities to build additional ones – E. g. , network monitoring, storage systems, various sensors, see GEMLCA Monitoring Tool (GMT)

Globus Toolkit: Open Source Grid Infrastructure Data Replication Globus Toolkit v 4 www. globus. Globus Toolkit: Open Source Grid Infrastructure Data Replication Globus Toolkit v 4 www. globus. org Credential Mgmt Replica Location Grid Telecontrol Protocol Delegation Data Access & Integration Community Scheduling Framework Web. MDS Python Runtime Community Authorization Reliable File Transfer Workspace Management Trigger C Runtime Authentication Authorization Grid. FTP Grid Resource Allocation & Management Index Java Runtime Security Data Mgmt Execution Mgmt Info Services Common Runtime 16

17 “Stateless” vs. “Stateful” Services File. Transfer Service move (A to B) Client • 17 “Stateless” vs. “Stateful” Services File. Transfer Service move (A to B) Client • Without state, how does client: – – Determine what happened (success/failure)? Find out how many files completed? Receive updates when interesting events arise? Terminate a request? • Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state

18 File. Transfer. Service (without WSRF) File. Transfer Service move (A to B) : 18 File. Transfer. Service (without WSRF) File. Transfer Service move (A to B) : transfer. ID Client what. Happen state tell. Me. When cancel • Developer reinvents wheel for each new service – Custom management and identification of state: transfer. ID – Custom operations to inspect state synchronously (what. Happen) and asynchronously (tell. Me. When) – Custom lifetime operation (cancel)

19 WSRF in a Nutshell • Service - permanent • State representation Service EPR 19 WSRF in a Nutshell • Service - permanent • State representation Service EPR EPR Get. RP Get. Mult. RPs Resource Set. RP Query. RPs Subscribe Set. Term. Time Destroy – Resource - transient - no creation method in standards – Resource Property • State identification – Endpoint Reference • State Interfaces – Get. RP, Query. RPs, Get. Multiple. RPs, Set. RP • Lifetime Interfaces – Set. Termination. Time, Immediate • Notification Interfaces – Subscribe, Notify • Service. Groups - e. g. MDS 4 resource document aggregation

20 Modeling State in Web Services Resource allocation Authentication & Authorization are applied to 20 Modeling State in Web Services Resource allocation Authentication & Authorization are applied to all requests Entity eate Stateful Cr Factory service s State Addres State inspection Lifetime mgmt Notifications Stateful Entities (Resources) Register Stateful Entity Interactions standardized using WSDL and SOAP Service requestor (e. g. , user application) Discovery Registry

21 GT 4 Web Services Runtime • How does WSRF meets Grid? GT 4 21 GT 4 Web Services Runtime • How does WSRF meets Grid? GT 4 gives a hand with the WSRF Core, the WS-GRAM, RFT and other GT 4 services are implemented with its help. • Redesign to enhance scalability, modularity, performance, usability • Leverages existing WS standards – WS-I Basic Profile: WSDL, SOAP, etc. – WS-Security, WS-Addressing • Adds support for emerging WS standards – WS-Resource Framework, WS-Notification • Java, Python, & C hosting environments – Java is standard Apache

22 GT 4 Web Services Runtime Custom Web Services Custom GT 4 WSRF Web 22 GT 4 Web Services Runtime Custom Web Services Custom GT 4 WSRF Web Services WS-Addressing, WSRF, WS-Notification WSDL, SOAP, WS-Security Registry Administration GT 4 Container User Applications

23 GT 4 WS Core in a Nutshell Implementation of WSRF: Resources, Endpoint. References, 23 GT 4 WS Core in a Nutshell Implementation of WSRF: Resources, Endpoint. References, Resource. Properties Example WS resource document from GEMLCA: /home/gkecskem 335462 DONE Operation Providers: pre-build implementations of WSRF operations Service EPR EPR Get. RP Get. Mult. RPs Resource Set. RP Query. RPs Subscribe Set. Term. Time Destroy Implementations of Resources (Reflection. Resource, Persistent. Reflection. Resource) and Resource. Properties (Simple. Resource. Property, Reflection. Resource. Property) Notification implementation: Topics, Topic. Set, Embedded Notification Consumer service

24 WSDL Resource document definition <definitions> <types> <xsd: schema>. . . <xsd: element name= 24 WSDL Resource document definition . . . . . . . . . . . . Service EPR EPR Get. RP Get. Mult. RPs Resource Set. RP Query. RPs Subscribe Set. Term. Time Destroy

25 Defining WSRF Standard Interfaces with GT 4 <definitions>. . . <port. Type name= 25 Defining WSRF Standard Interfaces with GT 4 . . . . . . . . . Service EPR EPR Get. RP Get. Mult. RPs Resource Set. RP Query. RPs Subscribe Set. Term. Time Destroy

26 WSDL Resource Creation Acquiring the EPR <definitions> <types> <xsd: schema>. . . Service 26 WSDL Resource Creation Acquiring the EPR . . . Service EPR EPR EPR Resource RPs Create. Resource . . . Get. RP Get. Mult. RPs Set. RP Query. RPs Subscribe Set. Term. Time Destroy Not standardised method

27 Modeling State in Web Services Resource allocation Authentication & Authorization are applied to 27 Modeling State in Web Services Resource allocation Authentication & Authorization are applied to all requests Entity eate Stateful Cr Factory service s State Addres State inspection Lifetime mgmt Notifications Stateful Entities (Resources) Register Stateful Entity Interactions standardized using WSDL and SOAP Service requestor (e. g. , user application) Discovery Registry

28 GT 4 WS Core in a Nutshell Service Container Service Get. RP Get. 28 GT 4 WS Core in a Nutshell Service Container Service Get. RP Get. Mult. RPs EPR Get. Mult. RPs Set. RP EPRResource EPR Set. RP EPRResource Query. RPs RPs Query. RPs Subscribe Set. Term. Time Resource. Home Destroy Service Container: host multiple services in container; one JVM process …more details: based on AXIS service container, processes SOAP messages, Resource. Context extension.

29 GT 4 WS Core in a Nutshell Service Container Service Get. RP Get. 29 GT 4 WS Core in a Nutshell Service Container Service Get. RP Get. Mult. RPs EPR Get. Mult. RPs Set. RP EPRResource EPR Set. RP EPRResource Query. RPs RPs Query. RPs Subscribe Set. Term. Time Resource. Home Destroy PIP PDP Secure Communication: Transport, Message, Conversation (Transport demonstrates best performance) Configurable Security Policies: Policy Information Points (PIPs), Policy Decision Points (PDP) -- chained Example authorization PDPs: Grid. Map, SAML implementations, XACML policies

30 GT 4 WS Core in a Nutshell Service Container Service PIP Get. RP 30 GT 4 WS Core in a Nutshell Service Container Service PIP Get. RP Get. Mult. RPs EPR Get. Mult. RPs Set. RP EPRResource EPR Set. RP EPRResource Query. RPs RPs Query. RPs Subscribe Set. Term. Time Resource. Home Destroy Work. Manager DB Conn Pool PDP JNDI Directory Work. Manager: “thread pool”, site independent “work” manager Apache Database Connection Pool library (JDBC “Data. Source” implementation) JNDI Directory: manages internal, shared objects (Resource. Homes, Work. Manager, Configuration objects, …)

31 GT 4 WS Core in a Nutshell Apache Tomcat Service Container Service PIP 31 GT 4 WS Core in a Nutshell Apache Tomcat Service Container Service PIP Get. RP Get. Mult. RPs EPR Get. Mult. RPs Set. RP EPRResource EPR Set. RP EPRResource Query. RPs RPs Query. RPs Subscribe Set. Term. Time Resource. Home Destroy Work. Manager DB Conn Pool PDP JNDI Directory Deploy Service Container “standalone” or within Apache Tomcat

32 WSRF & WS-Notification • Naming and bindings (basis for virtualization) – Every resource 32 WSRF & WS-Notification • Naming and bindings (basis for virtualization) – Every resource can be uniquely referenced, and has one or more associated services for interacting with it • Lifecycle (basis for fault resilient state mgmt) – Resources created by services following factory pattern – Resources destroyed immediately or scheduled • Information model (basis for monitoring, discovery) – Resource properties associated with resources – Operations for querying and setting this info – Asynchronous notification of changes to properties • Service groups (basis for registries, collective svcs) – Group membership rules & membership management • Base Fault type

WSRF/WSNs Compared (HPDC 2005) 33 GT 4 -Java GT 4 -C py. Grid. Ware WSRF/WSNs Compared (HPDC 2005) 33 GT 4 -Java GT 4 -C py. Grid. Ware WSRF: : Lite WSRF. NET Languages supported Java C Python Perl C#/C++/VBasic, etc. WS-Security password profile Yes No In progress Yes WS-Security X. 509 profile Yes In progress Yes WS-Secure. Conversation Yes No Yes TLS/SSL Yes Yes Yes Multiple Callout None Yes Not default Yes Yes Memory Footprint JVM + 10 M 22 KB 12 MB Depends Memory size per WS-Resource Depends on resource state 70 B Depends on resource state 0 (file/DB) or 10 B (process) Depends on resource state Unmodified hosting environment Yes No Yes (Apache) Yes Compliance with WS-I Basic Profile Yes Yes In progress Yes Compliance with WS-I Basic Security Profile Yes Yes No Yes Log 4 J Yes Yes WSE diagnostics WS-Resource. Lifetime Yes Yes Yes WS-Resource. Properties Yes Yes Yes WS-Service. Group Yes Yes Yes WS-Base. Faults Yes Yes Yes WS-Base. Notification Yes Consumer Yes No Yes WS-Brokered. Notification Partial No No No Yes WS-Topics Partial No Partial Authorization Persistence of WS-Resources Logging

34 Get. RP Test Distributed client and service on same LAN (times in milliseconds) 34 Get. RP Test Distributed client and service on same LAN (times in milliseconds) No Security 25. 57 X 509 Signing 149. 67 HTTPS 181. 96 17. 1 140. 5 55. 6 81. 39 10. 05 8. 23 2. 34 GT GT py W W 4 4 Gri SR SR - J - C d. W F: F. av ar : Lite NET a e N/A 14. 8 11. 46 2. 85 12. 91 GT GT py W W Gr SR SR 4 4 Gri SR SR 4 4 - J - C d. W F: F. - J - C id. W F: F. : L NE av av ar ite T ar : Lite NET a a e e

Java examples: Acquiring the endpoint reference Endpoint. Reference. Type localendp = new Endpoint. Reference. Java examples: Acquiring the endpoint reference Endpoint. Reference. Type localendp = new Endpoint. Reference. Type(); localendp. set. Address(new Address("https: //grid-computews. cpc. wmin. ac. uk: 8443/wsrf/services/my. WSRFService")); My. WSRFService. Port. Type service. Contact = Service. Locator. get. My. WSRFService. Port. Type. Port(localendp); Set. Sec. Props((Stub) service. Contact, Identity, cred); Create. Resource. Response create. Resource. Response = service. Contact. create. Resource(new Create. Resource()); localendp=create. Resource. Response. get. Endpoint. Reference(); // Now localendp holds the resource’s EPR, lets use it! My. WSRFService. Port. Type resource. Contact = Service. Locator. get. My. WSRFService. Port. Type. Port(localendp); Set. Sec. Props((Stub) resource. Contact, Identity, cred); Legend: Important, Third party (Axis, GT 4), Generated from WSDL 35

Java Examples: Setting a Resource Property 36 WSResource. Properties. Service. Addressing. Locator locator = Java Examples: Setting a Resource Property 36 WSResource. Properties. Service. Addressing. Locator locator = new WSResource. Properties. Service. Addressing. Locator(); Set. Resource. Properties_Port. Type rp. Port = locator. get. Set. Resource. Properties. Port(endpoint); Set. Sec. Props((Stub) rp. Port, Identity, cred); Update. Type update = new Update. Type(); Message. Element msg = new Message. Element(GLCProcess. Namespaces. RP_LASTJOBSTATECHANGED, ”-1 GEMLCA_PRESTATE"); update. set_any(new Message. Element[] { msg }); Set. Resource. Properties_Element request = new Set. Resource. Properties_Element(); request. set. Update(update); rp. Port. set. Resource. Properties(request); Legend: Important, Third party (Axis, GT 4), Generated from WSDL

37 Example Security setup public static void Set. Sec. Props(Stub On. Me, String Identity, 37 Example Security setup public static void Set. Sec. Props(Stub On. Me, String Identity, GSSCredential cred) { On. Me. _set. Property(Constants. GSI_TRANSPORT, Constants. SIGNATURE); On. Me. _set. Property(GSIConstants. GSI_MODE, GSIConstants. GSI_MODE_FULL_DELEG); On. Me. _set. Property(Constants. AUTHORIZATION, new Identity. Authorization(this. Identity == null? default. Identity: Identity)); if (cred != null) On. Me. _set. Property(GSIConstants. GSI_CREDENTIALS, cred); } Legend: Important, Third party (Axis, GT 4), Generated from WSDL