Grid Tutorial Cameron Kiddle Grid Research Centre University

Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary Western Grid Summit - 2005 Grid Tutorial

Acknowledgments Rob Simmonds Mark Fox Phil Rizk Abhishek Gaurav Andrey Mirtchovski Nayden Markatchev Western Grid Summit - 2005 Grid Tutorial 2

Outline Grid Computing Overview Basic Grid Services n security, data management, resource management, information services Advanced Grid Services n meta schedulers, workflow managers, replica location service Future of Grid Computing Western Grid Summit - 2005 Grid Tutorial 3

Grid Computing Aimed at providing users with a single system view of a set of distributed resources that span administrative domains Aids in solving large scale problems Aids in collaborative research efforts Western Grid Summit - 2005 Grid Tutorial 4

Current Grid Projects CERN LCG (LHC Computing Grid) Gri. Phy. N (Grid Physics Network) NEES (Network for Earthquake Engineering Simulation) Bio. Sim. Grid IVOA (International Virtual Observatory Alliance) Western Grid Summit - 2005 Grid Tutorial 5

West. Grid and Grid Is West. Grid a computational grid? Provides grid enabled resources n n n Security services Data transfer tools Job submission services West. Grid resources can be part of computational grids Western Grid Summit - 2005 Grid Tutorial 6

Global Grid Forum (GGF) Leads the global standardization effort for grid computing Open Grid Services Architecture (OGSA) Western Grid Summit - 2005 Grid Tutorial 7

Open Grid Services Architecture Infrastructure Services Resource Management Services Data Services Context Services Information Services Self-Management Services Security Services Execution Management Services Western Grid Summit - 2005 Grid Tutorial 8

Grid Middleware Globus Toolkit n n Globus Alliance Open source Legion n University of Virginia Now commercial product developed by Avaki Focuses on data grid / enterprise integration Western Grid Summit - 2005 Grid Tutorial 9

Globus Toolkit Collection of grid services to aid in development of grid applications and environments GT-2 – pre standards (defacto standard) GT-3 – Open Grid Services Infrastructure (OGSI) GT-4 – Web Services Resource Framework (WSRF) – beta version Western Grid Summit - 2005 Grid Tutorial 10

Basic Grid Services Security Services n GSI (Grid Security Infrastructure) Data Management n Grid. FTP Resource Management n GRAM (Grid Resource Allocation and Management) Information Services n MDS (Monitoring and Discovery Service) Western Grid Summit - 2005 Grid Tutorial 11

Grid Security Infrastructure (GSI) Based on n Public key cryptography X. 509 certificates Secure Socket Layer (SSL) Provides n n Mutual authentication Single sign-on Western Grid Summit - 2005 Grid Tutorial 12

Certificates Used to identify and authenticate users and services Includes four primary pieces of information n n Subject name Public key Identity of Certificate Authority (CA) Digital signature of the Certificate Authority Provides global name space Encoded in X. 509 certificate format (IETF) Western Grid Summit - 2005 Grid Tutorial 13

Mutual Authentication Process of two parties proving to each other who they say they are Both parties must trust CAs that signed each other’s certificates Each party sends certificate to other n n Make sure certificate is valid Make sure other party really is the party identified in the certificate Western Grid Summit - 2005 Grid Tutorial 14

Single Sign-On Sign-on only once to computational grid Use a proxy to avoid re-entering password n n Consists of new certificate and private key Certificate signed by owner, not CA Less secure – limited lifetime Both proxy certificate and owner’s certificate sent for mutual authentication grid-proxy-init Western Grid Summit - 2005 Grid Tutorial 15

GSI Demo Western Grid Summit - 2005 Grid Tutorial 16

My. Proxy Developed by NCSA (National Center for Supercomputing Applications) Credential repository Allows proxy credential to be received from any machine Can allow trusted servers to renew proxy credentials myproxy-init, myproxy-get-delegation Western Grid Summit - 2005 Grid Tutorial 17

My. Proxy Demo Western Grid Summit - 2005 Grid Tutorial 18

File Transfers gsiscp n n scp version based on GSI Easy to use, but provides poor performance for large file transfers Grid. FTP n n n Based on FTP (File Transfer Protocol) GSI security on control and data channels Modification of TCP buffer sizes Parallel transfers Third party transfers Western Grid Summit - 2005 Grid Tutorial 19

File Transfer Performance Western Grid Summit - 2005 Grid Tutorial 20

File Transfer Demo Western Grid Summit - 2005 Grid Tutorial 21

Grid Resource Allocation and Management (GRAM) Provides a single standard interface for remote job submission and resource management Requires users and application developers to learn only one method to gain access to a large variety of local management systems globus-job-run, globus-job-submit Western Grid Summit - 2005 Grid Tutorial 22

Job Submission Demo Western Grid Summit - 2005 Grid Tutorial 23

Monitoring and Discovery Service (MDS) Provides information about availability and status of resources Uses LDAP (Lightweight Directory Access Protocol) to access this information Grid Resource Information Service (GRIS) n Provides resource information Grid Index Information Service (GIIS) n Provides caching services and search capabilities Western Grid Summit - 2005 Grid Tutorial 24

Advanced Grid Services Meta-schedulers Workflows Replica Location Service (RLS) Abstract Workflows Western Grid Summit - 2005 Grid Tutorial 25

Meta-Schedulers Condor-G (Condor) n University of Wisconsin-Madison Moab Grid Scheduler (Silver) n Cluster Resources Inc. Western Grid Summit - 2005 Grid Tutorial 26

Condor-G Can submit and manage jobs at multiple sites Uses Globus mechanisms to communicate with resources at different sites Can advertise sites that jobs can be submitted to through Class Ads Western Grid Summit - 2005 Grid Tutorial 27

Class Ad Used to represent characteristics and constraints of resources and jobs Consists of a list of attributes (name/value pairs) Used in matchmaking of jobs to resources Western Grid Summit - 2005 Grid Tutorial 28

Condor-G Demo Western Grid Summit - 2005 Grid Tutorial 29

Workflows C 1 E 1 A B C 2 D E 2 C 3 Western Grid Summit - 2005 Grid Tutorial 30

DAGMan Directed Acyclic Graph Manager Node = program Edge = dependency Monitors Condor log files to enforce ordering of dependencies Western Grid Summit - 2005 Grid Tutorial 31

DAGMan Input File Programs in the DAG Pre/post processing scripts Dependencies Number of times to retry Macro definitions Western Grid Summit - 2005 Grid Tutorial 32

Sample DAGMan Input File JOB A A. condor A JOB B B. condor JOB C C. condor JOB D D. condor B C PARENT A CHILD B C D PARENT B C CHILD D Western Grid Summit - 2005 Grid Tutorial 33

Rescue DAGs Aids in resubmission of uncompleted portions of workflows Produced if DAGMan can not make any further progress in a DAG due to node failures Same as original DAG but nodes that successfully completed before are marked as DONE Western Grid Summit - 2005 Grid Tutorial 34

Life 3 D Workflow Animation Life 3 D Simulation Rendering Western Grid Summit - 2005 Grid Tutorial 35

Life 3 D Workflow - II blackhole (SFU) 2. Life 3 D Simulation 1. lattice (Uof. C) Data Storage Grid Research Center West. Grid 3. grc 15 7. octarine 5. 4. Rendering 6. Animation Western Grid Summit - 2005 Grid Tutorial 36

Life 3 D Workflow - III grc 15 blackhole Stage Data octarine lattice Animation Stage Data Life 3 d Simulation Stage Data Western Grid Summit - 2005 blackhole Rendering Grid Tutorial Stage Data 37

DAGMan Demo Western Grid Summit - 2005 Grid Tutorial 38

Life 3 D Animation Western Grid Summit - 2005 Grid Tutorial 39

Replica Location Service (RLS) Co-developed by Globus and Data. Grid projects Provides a mechanism for registering and locating replicas Maintains mappings between logical names and target names n n Logical names – unique identifiers for data content that may have one or more physical replicas Target name – physical location of a replica or other logical names representing the data Western Grid Summit - 2005 Grid Tutorial 40

Chimera Virtual Data System Part of the Gri. Phy. N project Distributed with the Virtual Data Toolkit Keeps track of how data products are derived Partial Workflow (VDL) Chimera Virtual Data Language Abstract Workflow (DAX) Pegasus Concrete Workflow (DAG) DAG XML RLS TC MDS (Transformation Catalog) Western Grid Summit - 2005 Grid Tutorial 41

Chimera Input - Virtual Data Language (VDL) n n Transformation – an executable program Derivation – an execution of the transformation along with logical input and output file names Output – Abstract Workflow (DAX) n Workflow activities independent of specific grid resources Western Grid Summit - 2005 Grid Tutorial 42

Pegasus Input – Abstract Workflow Output – Concrete Workflow (DAG) n Workflow activities assigned to specific Gird resources Uses RLS to determine data file locations Uses Transformation Catalog to map logical transformations to physical locations Uses MDS to determine available resources Western Grid Summit - 2005 Grid Tutorial 43

Chimera Example infile (blackhole) transitfile add 10 add 20 (lattice) outfile (blackhole) (nexus) Western Grid Summit - 2005 Grid Tutorial 44

Chimera Demo Western Grid Summit - 2005 Grid Tutorial 45

Summary Basic Grid Services Advanced Grid Services Security Services (GSI) n n n Meta-scheduler grid-proxy-init myproxy-get-delegation n Concrete Workflows Data Management (Grid. FTP) n n globus-url-copy n globus-job-run globus-job-submit Abstract Workflows n Information Services n DAGMan Replica Location Service (RLS) Resource Management (GRAM) n Condor-G Chimera/Pegasus MDS Western Grid Summit - 2005 Grid Tutorial 46

Future of Grid Computing Integration of Web Services and Grid Services Web Services Resource Framework (GT-4) Standards are more established n n Involvement of more developers Better tools Grid-enabled applications n n Common library interface “run_job_on_grid” Wider spread use Western Grid Summit - 2005 Grid Tutorial 47

Contact Information Cameron Kiddle kiddlec@cpsc. ucalgary. ca http: //grid. ucalgary. ca/ http: //pages. cpsc. ucalgary. ca/~kiddlec/ Western Grid Summit - 2005 Grid Tutorial 48