Globus Virtual Workspaces An Update SC 2007 Reno

Globus Virtual Workspaces An Update SC 2007, Reno, NV Kate Keahey Argonne National Laboratory University of Chicago keahey@mcs. anl. gov

Motivation and Background

Why Virtual Workspaces? l Quality of Service u We get: batch-style provisioning l l u We need: advance reservations, urgent computing, periodic, best-effort, and others l l l One size fits all Side-effect of job scheduling Separation of job scheduling and resource management E. g. workflow-based apps and batch apps have different needs Quality of Life u u We have: “a 100 nodes we cannot use” Complex applications l l u Hard to install Require validation Separation of environment preparation and resources leasing SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

What are Virtual Workspaces? l A dynamically provisioned environment u u l Environment definition: we get exactly the (software) environment we need on demand. Resource allocation: Provision the resources the workspace needs (CPUs, memory, disk, bandwidth, availability), allowing for dynamic renegotiation to reflect changing requirements and conditions. Implementation u u Traditional means: publishing, automated configuration, coarse-grained enforcement Virtual Machines: encapsulated configuration and fine -grained enforcement Paper: “Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid” SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Virtual Machines App App App Guest OS (Linux) Guest OS (Net. BSD) Guest OS (Windows) VM VM VM Parallels Xen VMWare Virtual Machine Monitor (VMM) / Hypervisor UML Hardware KVM etc. l l Bring your environment with you Fast to deploy, enables short-term leasing Excellent enforcement, performance isolation Very good isolation SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Globus Virtual Workspaces: How Do They Work?

Virtual Workspaces: Vital Stats l The GT 4 Virtual Workspace Service (VWS) allows an authorized client to deploy and manage workspaces on-demand. u GT 4 WSRF front-end (one per site) l u u Follows WS-Agreement provisioning model Currently implements workspaces as Xen VMs l u u u Other implementations could also be used Implements multiple deployment modes l u Leverages GT core and services, notifications, security, etc. Best-effort, leasing, etc. Current release 1. 3 (November ‘ 07) Globus incubator project More information at: http: //workspace. globus. org SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Deploying Workspaces Remotely Pool node VWS Service Pool node Pool node Pool node Workspace -Workspace metadata -Pointer to the image -Logistics information -Deployment request -CPU, memory, node count, etc. SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Interacting with Workspaces The workspace service publishes information on each workspace as standard WSRF Resource Properties. Pool node Users can interact directly with their workspaces the same way the would with a physical machine. SC 07, Reno, NV Pool node Pool node Users can query those properties to find out information about their workspace (e. g. what IP the workspace was bound to) Pool node VWS Service Pool node Trusted Computing Base (TCB) Virtual Workspaces: http: //workspace. globus. org

Workspace Service Components Workspace WSRF front-end that allows clients to deploy and manage virtual workspaces VWS Service Pool node Pool node Pool node Workspace back-end: Resource manager for a pool of physical nodes Deploys and manages Workspaces on the nodes Each node must have a VMM (Xen) installed, as well as the workspace control program that manages individual nodes Contextualization creates a common context for a virtual cluster SC 07, Reno, NV Trusted Computing Base (TCB) Virtual Workspaces: http: //workspace. globus. org

Workspace Service Components l GT 4 WSRF front-end u u Leverages GT core and services, notifications, security, etc. Follows the OGF WS-Agreement provisioning model l u l Provides lease descriptions Workspace Service back-end u u u l Publishes available lease terms Currently focused on Xen Works with multiple Resource Managers Workspace Control Contextualization u Put the virtual appliance in its deployment context SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Managing Resources with Virtual Workspaces

Workspace Back-Ends l Default resource manager (basic slot fitting) u l Commercial datacenter technology would also fit Challenge: finding Xen-enabled resources u u u Amazon Elastic Compute Cloud (EC 2) Selling cycles as Xen VMs Software similar to Workspace Service l u u SC 07, Reno, NV No virtual clusters, contextualization, fine-grain allocations, etc. Solution: develop a back-end to EC 2 Grid credential admission -> EC 2 charging model Virtual Workspaces: http: //workspace. globus. org

Virtual Workspaces for STAR l STAR image configuration u l A virtual cluster composed of an OSG headnode and STAR worker nodes Using the workspace service over EC 2 to provision resources u u Allocations of up to 100 nodes Dynamically contextualized for out-of-the-box cluster SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

with thanks to Jerome Lauret and Doug Olson of the STAR project Running jobs : : : 94 Running jobs 142 Running jobs 124 109 42 73 0 Running jobs : 150 230 VWS/EC 2 Running jobs : : : 76 Running jobs 282 Running jobs 243 140 195 221 0 Running jobs : 300 PDSF BNL WSU Running jobs : : : 96 Running jobs 195 Running jobs 183 136 152 37 54 0 Running jobs : 200 150 Fermi Running jobs : : 50 Running jobs 39 15 21 27 34 0 9 Running jobs : 42 50 Job Completion : SC 07, Reno, NV File Recovery : Virtual Workspaces: http: //workspace. globus. org

with thanks to Jerome Lauret and Doug Olson of STAR project thanks to Jerome Lauret and Doug Olson of the STAR project Nersc PDSF EC 2 (via Workspace Service) WSU SC 07, Reno, NV Accelerated display of a workflow job state Y = job number, X = job state Virtual Workspaces: http: //workspace. globus. org

Workspace Back-Ends l Default resource manager (basic slot fitting) u l Commercial datacenter technology would also fit Challenge: finding Xen-enabled resources u u u Amazon Elastic Compute Cloud (EC 2) Selling cycles as Xen VMs Software similar to Workspace Service l u u l No virtual clusters, contextualization, fine-grain allocations, etc. Grid credential admission -> EC 2 charging model Solution: develop a back-end to EC 2 Challenge: integrating VMs into current provisioning models u SC 07, Reno, NV Solution: gliding in VMs with the Workspace Pilot Virtual Workspaces: http: //workspace. globus. org

Providing Resources: The Workspace Pilot l l Challenge: find the simplest way to integrate VMs into current provisioning models Glide-ins (Condor): poor man’s resource leasing u Best-effort semantics: submit a job “pilot” that claims resources but does not run a job l The Workspace Pilot u Resources booted to dom 0 u Pilot adjusts memory u VWS leases “slots” to VMs u SC 07, Reno, NV Functional closure: kill-all facility, etc. Virtual Workspaces: http: //workspace. globus. org

Workspace Control l VM control u u l Integrating into the network u u u l l Assigning MAC addresses and IP addresses DHCP Delivery tool Building up a trusted networking layer VM image propagation Image management and reconstruction u l Starting, stopping etc. To be replaced by Xen API creating blank partitions Talks to the workspace service via ssh u SC 07, Reno, NV To be replaced Virtual Workspaces: http: //workspace. globus. org

Workspace Back-Ends l Default resource manager (basic slot fitting) u l Commercial datacenter technology would also fit Challenge: finding Xen-enabled resources u u u Amazon Elastic Compute Cloud (EC 2) Selling cycles as Xen VMs Software similar to Workspace Service l u u l Grid credential admission -> EC 2 charging model Solution: develop a back-end to EC 2 Challenge: integrating VMs into current provisioning models u l No virtual clusters, contextualization, fine-grain allocations, etc. Solution: gliding in VMs with the Workspace Pilot Long-term solutions u u u SC 07, Reno, NV Interleaving soft and hard leases Providing better articulated leasing models Developed in the context of existing schedulers Virtual Workspaces: http: //workspace. globus. org

So -- you’ve deployed* some VMs… Now What? *Do they have public IP addresses? Do they actually represent something useful? (BTW, I need an OSG cluster) Can the VMs find out about each other? Can they share storage? How do they integrate into the site storage/account system? Do they have host certificates? And gridmapfile? And all the other things that will integrate them into my VO?

Virtual Clusters l Challenge: what is a virtual cluster? u A more complex virtual machine l u u Available at the same time and sharing a common context Example: l l Networking, shared storage, etc. that will be portable across sites and implementations A set of worker nodes with some edge services in front and NFS-based shared storage Solution: management of ensembles and sharing u Configurable cluster deployment l l u Exporting and sharing a common context l u A set of worker nodes A few Edge Services enabling access to those nodes Configuring and joining context Networking l l Edge Services have public IPs Worker nodes are on a private network shared with the Edge Services Paper: “Virtual Clusters for Grid Communities”, CCGrid 2006 SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Contextualization l Challenge: Putting a VM in the deployment context of the Grid, site, and other VMs u l Assigning and sharing IP addresses, name resolution, applicationlevel configuration, etc. Solution: Management of Common Context u contextualization agent Common Context IP hostname pk Configuration-dependent l u u provides&requires Common understanding between the image “vendor” and deployer Mechanisms for securely delivering the required information to images across different implementations Paper: “A Scalable Approach To Deploying And Managing Appliances”, Tera. Grid conference 2007 SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Where do VM images come from?

Appliance Management l Short term solution: Marketplaces u u u l The Workspace Marketplace http: //workspace. globus. org/vm/marketplace. html Providing described images for scientific community Appliance providers and marketplaces Long-term solution: Appliance Providers u Automated image production, attestation and signing l u Automated management Collaboration with configuration management communities and projects l l l SC 07, Reno, NV r. Path company: the r. Builder project (DOE SBi. R) Bcfg 2, adopted on many ANL resources Osfarm @ CERN Open. Lab, serving the scientific community Appliance providers Virtual Workspaces: http: //workspace. globus. org

Workspace Ecosystem Appliance Providers: OSFarm, r. Path, Cohesive. FT, bcfg 2, etc. marketplaces of all kinds Virtual Organizations: configuration, attestation, maintenance Resource Providers: Local clusters, Grid resource providers (Tera. Grid, OSG) Commercial providers: EC 2, Sun, slicehost, Provisioning a resource, not a platform Middleware: appliances --> resources manage appliance deployment Combining networks and storage VWS SC 07, Reno, NV EC 2 In-Vigo Virtual Workspaces: http: //workspace. globus. org

Parting Thoughts l VMs are the raw materials from which a working system can be built u u u l Division of labor u u u l But we still have to build it! Technical challenges: taking one step at a time Social/procedural challenges Resource providers Appliance providers Can we build trust between these two groups? If you think we can help you out, give us a call: u http: //workspace. globus. org SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org

Acknowledgements l Workspace team: u u u l Funding u u u l Kate Keahey Tim Freeman Borja Sotomayor NSF SDCI “Missing Links” NSF CSR “Virtual Playgrounds” DOE CEDPS Project With thanks to many collaborators: u Jerome Lauret (STAR, BNL), Doug Olson (STAR, LBNL), Marty Wesley (r. Path), Stu Gott (r. Path), Ken Van Dine (r. Path), Predrag Buncic (Alice, CERN), Haavard Bjerke (CERN), Rick Bradshaw (Bcfg 2, ANL), Narayan Desai (Bcfg 2, ANL), Duncan Penfold-Brown (Atlas, uvic), Ian Gable (Atlas, uvic), David Grundy (Atlas, uvic), Ti Leggit (University of Chicago), Greg Cross (University of Chicago), Mike Papka (University of Chicago/ANL) SC 07, Reno, NV Virtual Workspaces: http: //workspace. globus. org