Globus Grid Middleware Basics Components and Services Source

Globus Grid Middleware: Basics, Components, and Services Source: The Globus Project Argonne National Laboratory & University of Southern California http: /www. globus. org Compiled/Compacted for 433 -678: R. Buyya and S. Venugopal 1

Agenda n n The evolution of Globus Security Services n n GRAM(Grid Resource Allocation and Management) n n GSI (Grid Security Infrastructure) MDS (Monitoring and Discovery System) Execution Management Services Information Services Data Services n n n Grid. FTP RFT (Reliable File Transfer) Summary/Conclusion 2

The Globus Project n Basic research in grid-related technologies n n Focused on creating a core Grid middleware n n n Core services for grid-enabled tools & apps Developed Globus Toolkit (1. 0 in 1998, 2. 0 in 2001, 3. 0 in 2004 based on web services/WSRF, 4. 0 in 2008, and 5. 0 in 2011) Construction of production grids & testbeds n n Resource & data management, security, Qo. S, policy, communication, adaptation, etc. Multiple deployments to distributed organizations for production & prototyping Impacts n The Southern California Earthquake Center uses Globus software to visualize earthquake simulation data. n n Scientists in the Earth System Grid (ESG) are producing, archiving, and providing access to climate data that advances our understanding of global climate change. Find more on http: //www-unix. globus. org/alliance/impact/ 3

Grid and Web Services: Convergence Grid GT 1 Started far apart in apps & tech Web GT 2 OGS I Have been converging HTTP L, WSD WS-* WSRF L 2 WSD The definition of WSRF(Web Services Resource Framework) means that Grid and Web communities can move forward on a common base Ack: Prof. Ian Foster, Globus Project

Open Grid Services Architecture Standardization Domain-Specific Services Program Execution Data Services Core Services Open Grid Services Infrastructure WS-Resource Framework Web Services Messaging, Security, Etc. Ack: Prof. Ian Foster, Globus Project

Globus Toolkit 4 6

Globus Toolkit Components Applications Third Party User-Level Middleware Globus Grid Resource Management (GRAM, GASS) (GRAM) Grid Information Services (MDS) Grid Data Management (Grid. FTP, Replica Catalog) GSI Security Layer Grid Resources and Local Services 7

Security Services Review of Security Services 8

Grid Security Infrastructure (GSI) n Based on n Public key cryptography X. 509 certificate Certificate/Proxy - Single Sign-on n Gives information about the proxy n n n $ grid-proxy-info Create proxy at /tmp/x 509 up_u<USER ID> Destroy proxy certificate n $ grid-proxy-destroy 9

PKI: Creating Private & Public Keys n n n To request a certificate a user starts by generating a key pair. The private key is stored encrypted with a pass phrase the user gives Encrypted The public key is put On local disk into a certificate request USER Certificate Request Public Key 10

PKI: Certificate Issuance n n The user then takes the certificate to the CA The CA usually includes a Registration Authority (RA) which verifies the request: n n n The name is unique with respect to the CA It is the real name of the user Etc. Certificate Request ID Public Key 11

PKI: Certificate Issuance n The CA then signs the certificate request and issues a certificate for the user. Certificate Request Public Key Sign Rajkumar Buyya 111, Barry Street Carlton State of Victoria Seal BD 01 -04 -1971 Male 165 cms, 65 Kg B&W Eyes Name Issuer Public Key Signature 12

Mutual Authentication (between A and B) n n n A and B are two parties: Both need to trust each others’ CA. A B (A establishes connection to B and gives his certificate (name, pub. Key) to B). B makes sure that it can trust CA of A. B generates random message A and asks it encrypt it. A encrypts it and send to B B decrypts using A’s public key. If the msg. is same as what B has sent, then A is who it is 13 claiming to be.

Example: Secure Remote Startup in Globus 1. Exchange certificates, authenticate, delegate 2. Check gridmap file 3. Lookup service 4. Run service program (e. g. jobmanager) cert jobmanager 4. map services 2. 3. 1. key client cert key gatekeeper 14

Execution Management Services Grid Resource Allocation and Management (GRAM) 15

What is GRAM? n Grid resource access & job execution services: n n n Handles placement, provisioning and lifetime management of jobs GRAM is a unifying remote interface for remote job submission and resource management GRAM is for stateful job control n Reliable operation n Asynchronous monitoring and control 16

Grid Job Management Goals Provide a service to securely: n Create an environment for a job n Stage files to/from environment n Cause execution of jobs n Monitor execution of jobs n Signal important state changes to client n Enable client access to output files 17

GRAM Command Line Tools n n Pre-WS GRAM (GT 2) Command Line Tools Interactive job submission n n Batch job submission n n globus-job-run host: port/jobmanager-pbs command globus-job-submit command Checking job status when done output n globus-job-status job. ID globus-job-output –out|err job. ID n globus-job-cancel job. ID n n Kill a job 18

Grid Information Service MDS (Monitoring and Discovery Service) 19

The Need for Information n System information is critical to operation of the grid and construction of applications n n How does an application determine what resources are available? What is the “state” of the computational grid? How can we optimize an application based on configuration of the underlying system? We need a general information infrastructure to answer these questions 20

Using Information for Resource Brokering “ 10 GFlops, EOS data, 20 Mb/sec -- for 20 mins” Info service: location + selection Resource Broker “ 20 Mb/sec” GRAM Globus Resource Allocation Managers GRAM Fork LSF EASYLL Condor etc. Metacomputing “What computers? ” Directory “What speed? ” Service “When available? ” “ 50 processors + storage from 10: 20 to 10: 40 pm” GRAM 21

Examples of Useful Information n Characteristics of a compute resource n n Characteristics of a network n n IP address, software available, system administrator, networks connected to, OS version, load Bandwidth and latency, protocols, logical topology Characteristics of the Globus infrastructure n Hosts, resource managers(PBS, Condor etc) 22

Data Services Overview of Data Services 23

Data Services n n Handle movement, access, replication and updates of data Example services in Globus Toolkit n n Grid. FTP Reliable File Transfer (RFT) 24

Data Services - Grid. FTP n Basic tool provided with Globus Toolkit n n n Supported protocols n n globus-url-copy [-tcp-bs TCPBuf. Size. Bytes] [-p parallelism] source. URL dest. URL format = protocol: //[host]: [port]/path gsiftp: //, file: //(on a local machine only), ftp: //, https: // It is legal to specify a user name and password in the URL as follows n gsiftp: //myname: [mypassword]@myhost. mydomai n. com/file. dat 25

Data Services - Grid. FTP n local -> local n n remote -> local n n $ globus-url-copy gsiftp: //localhost/etc/hosts file: ///tmp/hosts_copy $ globus-url-copy gsiftp: //ng 2. vpac. org/etc/hosts file: ///tmp/hosts_copy local -> remote n $ globus-url-copy file: ///tmp/hosts_copy gsiftp: //ng 2. vpac. org/tmp 26

Data Services - Reliable File Transfer (RFT) n n n Manages a set of third-party Grid. FTP transfers Uses a database to checkpoint transfer state Recovers from n n n Source/destination server failures Network failures Container failures Transfers retried with exponential backoff Resumes transfers where they left off 27

Summary n Globus as a core Grid middleware solution n Four major services n Security Service n n n Execution Management Service n GRAM(Grid Resource Allocation and Management ) Information Service n n MDS (Monitoring and Discovery Service) Data Service n n n GSI Grid. FTP Reliable File Transfer Very prominent (during 1998 -2005) – used in several collaborative scientific computing and national grid projects 28

Some Issues with Globus n n n Very complex system, need a strong administration skills to install/maintain Deep learning curve for developers who need to work with APIs More on Globus, visit: n http: //www. globus. org/ 29