Grid Dynamics Ian Foster Argonne National Laboratory University of Chicago Univa Corporation
2 Acknowledgements l Carl Kesselman, with whom I developed many ideas (& slides) l Bill Allcock, Charlie Catlett, Kate Keahey, Jennifer Schopf, Frank Siebenlist, Mike Wilde @ ANL/UC l Ann Chervenak, Ewa Deelman, Laura Pearlman @ USC/ISI l Karl Czajkowski, Steve Tuecke @ Univa l Numerous other fine colleagues in NESC, EGEE, OSG, Tera. Grid, etc. l NSF & DOE for research support
3 What is the Grid? ? “Resource sharing & coordinated problem solving in dynamic, multiinstitutional virtual organizations” “When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” (George Gilder) “The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001
System-Level Problem Decomposition Implementation Facilities Computers Storage Networks Services Software People U. Colorado UIUC Experimental Model Grid technology COORD. NCSA Computational Model
Grid-enabled Business Intelligence Application Provision New Worker Process 2 BI Server Grid backend Dispatcher Managed Pool of Shared Resources BI server applications started and decommissioned by a Grid-enabled dispatcher 5
Grid Dynamics: Vision vs. Reality l Vision: On-demand access to computing u u On-demand resources from providers u l New communities form easily Adapt easily to new missions, requirements Reality: Much manual configuration, e. g. : u Manually deployed services on dedicated hardware u Manually maintained access control lists u Sysadmin-maintained allocation policies u Human-mediated resource reservation 6
Grid Dynamics: A Two-Dimensional Problem 7 Function Resource l Decompose across network Users Discovery tools Clients integrate dynamically u u Select “best of breed” providers u l Select & compose services Analysis tools Publish result as new services Data Archives Decouple resource & service providers Fig: S. G. Djorgovski
Service-Oriented Systems: The Role of Grid Infrastructure 8 Users l u u l Composition Service-oriented applications Wrap applications as services Compose applications into workflows Service-oriented Grid infrastructure u Workflows Invocation Appln Service Provisioning Provision physical resources to support application workloads “The Many Faces of IT as Service”, ACM Queue, Foster, Tuecke, 2005
Grid Dynamics: Forming & Operating Communities l Define membership & roles; enforce laws & community standards u u l I. e. , policy for service-oriented architecture Addressing dynamic membership & policy Build, buy, operate, & share infrastructure u u u Decouple consumer & provider For data, programs, services, computing, storage, instruments Address dynamics of community demand 9
Grid Dynamics: Forming & Operating Communities l 10 Define membership & roles; enforce laws & community standards u u l I. e. , policy for service-oriented architecture Addressing dynamic membership & policy Build, buy, operate, & share infrastructure u u u Decouple consumer & provider For data, programs, services, computing, storage, instruments Address dynamics of community demand
11 Defining Community: Membership and Laws Identify VO participants and roles l u For people and services Specify and control actions of members l u Empower members delegation u Enforce restrictions federate policy Access B A 1 1 10 A 1 Effective 2 Policy of site to community 10 1 B 16 1 2 Access granted by community to user Site admissioncontrol policies
12 Policy Challenges in VOs l Restrict VO operations based on requestor characteristics u l VO dynamics create challenges Intra-VO u u l VO-specific roles Mechanisms to specify/enforce VO-level policy Inter-VO u l Different VOs define different entities/roles Different sorts of policy need to be enforced u Access, usage, accounting, audit, …
13 Evolution of Grid Security & Policy 1) Grid security infrastructure u Public key authentication & delegation u Access control lists (“gridmap” files) Limited set of policies can be expressed 2) Utilities to simplify operational use, e. g. u My. Proxy: online credential repository u VOMS, ACL/gridmap management Broader set of policies, but still ad-hoc 3) General, standards-based framework for authorization & attribute management
14 Core Security Mechanisms l Attribute Assertions u l Authentication and digital signature u l C asserts that S can perform O on behalf of C Attribute mapping u l Allows signer to assert attributes Delegation u l C asserts that S has attribute A with value V {A 1, A 2… An}vo 1 {A’ 1, A’ 2… A’m}vo 2 Policy u Entity with attributes A asserted by C may perform operation O on resource R
15 Security Services for VO Policy l Attribute Authority (ATA) u l Issue signed attribute assertions (incl. identity, delegation & mapping) Authorization Authority (AZA) u Decisions based on assertions & policy Delegation Assertion VO Resource Admin User A User B can use Service A Attribute VO ATA Mapping ATA VO Me mber Attribu te VO Member Attribute VO User B VO AZA VO A Service VO-A Attr VO-B Attr VO B Service
Closing the Loop: GT 4 Security Toolkit Authz Callout: SAML, XACML 17 SSL/WS-Security with Proxy Services (running Certificates on user’s behalf) Access Compute Center Rights CAS or VOMS issuing SAML or X. 509 ACs Users Rights Local policy on VO identity or attribute authority My. Proxy VO Rights’ Shib KCA
Grid Dynamics: Forming & Operating Communities l Define membership & roles; enforce laws & community standards u I. e. , policy for service-oriented architecture u l 18 Addressing dynamics of membership & policy Build, buy, operate, & share infrastructure u u u Decouple consumer & provider For data, programs, services, computing, storage, instruments Address dynamics of community demand
Bootstrapping a VO by Assembling Services 1) Integrate services from other sources u Virtualize external services as VO services Content Services Capacity Community Services Provider Capacity Provider 2) Coordinate & compose u Create new services from existing ones “Service-Oriented Science”, Science, Foster, 2005 19
20 Providing VO Services: (1) Integration from Other Sources l Negotiate service level agreements l Delegate and deploy capabilities/services l Provision to deliver defined capability l Configure environment l Host layered functions Community A … Community Z
21 Virtualizing Existing Services into a VO l Establish service agreement with service u l E. g. , WS-Agreement Delegate use to VO user User A VO User VO Admin Existing Services User B
22 Deploying New Services Policy Client Allocate/provision Configure Initiate activity Monitor activity Control activity Interface Activity Environment Resource provider WSRF (or WS-Transfer/WS-Man, etc. ), Globus GRAM, Virtual Workspaces
23 Activities Can Be Nested Client Policy Client Environment Interface Resource provider
Embedded Resource Management: E. g. , EGEE & OSG Client-side VO Admin Deleg GRAM Headnode Resource Manager VO User Deleg GRAM Cluster Resource Manager Monitoring and control Deleg VO Scheduler • • • . . . Other Services VO Job GRAM Cluster Resource Manager VO admin delegates credentials to be used by downstream VO services. VO admin starts the required services. VO jobs comes in directly from the upstream VO Users VO job gets forwarded to the appropriate resource using the VO credentials Computational job started for VO VO Job 24
Virtual Workspaces (Kate Keahey et al. ) l GT 4 service for the creation, monitoring, & management of virtual workspaces l High-level workspace description l WSRF mechanisms to monitor & manage l Multiple implementations u u Xen virtual machines u (VMware virtual machines) u l Dynamic accounts … Virtual clusters as a higher-level construct 25
26 How do Grids and VMs Play Together? request VM EPR VM Factory create new VM image Client use existing VM image inspect & manage Create VM image VM Repository deploy, suspend start program VM Manager Resource VM
27 Virtual OSG Clusters OSG cluster Xen hypervisors Tera. Grid cluster “Virtual Clusters for Grid Communities, ” Zhang et al. , CCGrid 2006
Providing VO Services: (2) Coordination & Composition l Take a set of provisioned services … … & compose to synthesize new behaviors l This is traditional service composition u u But must also be concerned with emergent behaviors, autonomous interactions See the work of the agent & Planet. Lab communities “Brain vs. Brawn: Why Grids and Agents Need Each Other, " Foster, Kesselman, Jennings, 2004. 28
The Globus-Based LIGO Data Grid 29 LIGO Gravitational Wave Observatory Birmingham • §Cardiff AEI/Golm Replicating >1 Terabyte/day to 8 sites >40 million replicas so far MTBF = 1 month www. globus. org/solutions
30 Data Replication Service l Pull “missing” files to a storage system Data Location Data Movement Data Replication List of required Files Replica Location Index Grid. FTP Reliable File Transfer Service Grid. FTP Local Replica Catalog Replica Location Index Data Replication Service “Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System, ” Chervenak et al. , 2005
31 Composing Resources … Composing Services Deploy service Deploy container Deploy virtual machine Deploy hypervisor/OS Procure hardware DRS JVM VM Grid. FTP LRC Grid. FTP VO Services VM Hypervisor/OS Physical machine State exposed & access uniformly at all levels Provisioning, management, and monitoring at all levels
Dynamic Service Deployment (Argonne + China Grid) l 32 Interface u Upload-push u Upload-pull u Deploy u Undeploy u Reload “HAND: Highly Available Dynamic Deployment Infrastructure for GT 4, ” Li Qi et al. , 2006
33 Decomposition Enables Separation of Concerns & Roles S 1 User D S 3 “Provide access to data D at S 1, S 2, S 3 with performance P” Service Provider “Provide storage with performance P 1, network with P 2, …” Resource Provider S 2 S 1 D S 2 S 3 Replica catalog, User-level multicast, … S 1 D S 2 S 3
34 Community Commons l What capabilities are available to VO? u l Membership changes, state changes Require mechanisms to aggregate and update VO information The age of information A A S VO-specific indexes S Information S MORE A S FRESH
GT 4 Monitoring and Discovery Services (Uniform Treatment of State is Wonderful!) WS-Service. Group GT 4 Container Clients (e. g. , Web. MDS) MDSIndex Registration & WSRF/WSN Access GT 4 Container MDSIndex Automated registration in container GRAM adapter GT 4 Cont. Custom protocols for non-WSRF entities MDSIndex Grid. FTP User RFT
System-Level Problem Decomposition Implementation Facilities Computers Storage Networks Services Software People U. Colorado UIUC Experimental Model Grid technology COORD. NCSA Computational Model
Grid-enabled Business Intelligence Application Provision New Worker Process 2 BI Server Grid backend Dispatcher Managed Pool of Shared Resources BI server applications started and decommissioned by a Grid-enabled dispatcher 37
38 The Integrating Role of Grid Infrastructure Multiple applications and workload types Coarse Grained Fine Grained Data Driven Workflow Consistent & open management interface End-to-end Quality of Service Consistent & open enactment interface Multiple resource types and instances Grid Infrastructure Dev / Test
Summary: Grid Dynamics and You l Grid = dynamic behaviors & environments u u u l Dynamic communities & activities Decoupling of service consumption from service production Dynamic provisioning of services We have tools to realize dynamic scenarios u Uniform state representation & access u Flexible security & policy framework u l Virtual machines, dynamic services, & other building blocks We now need much experimentation 39
40 For More Information l Globus Alliance u l www. globus. org Background u www. mcs. anl. gov/~foster Come to GT 4 workshop, 8: 30 -12: 00 Wednesday u Overview of features u User experiences u Future directions 2 nd Edition www. mkp. com/grid 2
Available in High-Quality Open Source Software … Data Replication 41 Globus Toolkit v 4 www. globus. org Credential Mgmt Replica Location Grid Telecontrol Protocol Delegation Data Access & Integration Community Scheduling Framework Web. MDS Python Runtime Community Authorization Reliable File Transfer Workspace Management Trigger C Runtime Authentication Authorization Grid. FTP Grid Resource Allocation & Management Index Java Runtime Security Data Mgmt Execution Mgmt Info Services Common Runtime I. Foster, Globus Toolkit Version 4: Software for Service-Oriented Systems, LNCS 3779, 2 -13, 2005
GT 4 & Web Services: Uniform State, Security, Mgmt Custom WSRF Custom Services GT 4 WSRF Web Services Registry & Admin GT 4 Container (e. g. , Apache Axis) User Applications WS-A, WSRF, WS-Notification WSDL, SOAP, WS-Security 42
Glob. Dev Guidelines (Apache) Infrastructure (CVS, email, bugzilla, Wiki) Projects Include … http: //dev. globus. org
Скачать презентацию Grid Dynamics Ian Foster Argonne National Laboratory University
a960f86da91fd77e11e2d1e98f3a8ba3.ppt