Basic Grid Job Submission Alessandra Forti 28 March

Basic Grid Job Submission Alessandra Forti 28 March 2006

Outline n n n Grid components Job submission Documentation

Grid Components RB and LB n RB=Resource Broker n n the equivalent of a batch server. Jobs land on a RB which decides depending on the User specifications where to send the Job LB=Logging and Bookkeeping n Handles the information about the jobs submitted through a specific RB. Normally a RB and a LB resides on the same machine but it is not necessary. A user can submit a job to an RB and log them on an LB at another site

Grid Components UI and CE n UI=User Interface n n The front end where the grid clients accessible to the users reside. It has login access. It can be located anywhere. A laptop with UI software on it can access grid resources. CE=Computing Element n The gateway to a local batch system. It hanldes final authentication and authorization to access the local batch system. It can be on the same machine as the batch server but is not required.

Grid Components SE n SE=Storage element n n Gateway to the data. In the simplest form it is basically a Grid. FTP server, but this type is considered obsolete now. It handles authentication and authorization to access the local data. SRM=Storage Resource Manager n n SRM is a protocol designed to hide the implementation actually used as a backend from the user. Backends are now more sophisticated and have storage management tools and will support access policies based on the DN of the certificate rather than simple unix IDs It is the current version of a SE

Grid Componenents IS n IS=Information system n n n Each site publishes an amount of information about the resources available in the IS. IS has a hierarchical structure (top level BDII, site BDII, service GIIS). The user only sees the top level BDII and this is what the RB, WN and UI see as well. Generic top level BDII will contain all the same information (they are replicas). However a VO might want to run its own BDII containing only information about resources open to it. This gives control to the VO to select good sites in a way transparent to the user.

Work Load Management

Job Submission n First step login to a UI n n A desktop in the department, your laptop, lxplus at CERN… if correctly configured they should be equivalent You need your certificate and key in $HOME/. globus n n If they have the wrong permissions the software will complain. This is to remind users to protect their certificates. grid-proxy-init Your identity: /C=UK/O=e. Science/OU=Manchester/L=HEP/CN=alessandra forti Enter GRID pass phrase for this identity Creating proxy. . . . Done Your proxy is valid until: Wed Mar 29 02: 00: 20 2006

Job submission: JDL language n JDL=Job Description Language n n n To submit a job you’ll need to write what is called a jdl file in which you specify the type of resources the job needs and what files the job needs to find. Unfortunately not everything can be specified. Only what can be retrieved by the IS and the catalogs and sometimes not even that if the RB can’t handle it. Most simple jdl is cat test. Job. jdl Executable = “test. sh"; Std. Output = "test. Job. out"; Std. Error = "test. Job. err"; Input. Sandbox = {". /test. sh"}; Output. Sandbox = {"test. Job. out", "test. Job. err"};

How to list resources for a job edg-job-list-match --vo dteam test. Job. jdl Selected Virtual Organisation name (from --vo option): dteam Connecting to host lcgrb 01. gridpp. rl. ac. uk, port 7772 ****************************** COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found: *CEId* ce 01. tier 2. hep. manchester. ac. uk: 2119/jobmanager-pbs-dteam ******************************

Submit a Job edg-job-submit --vo dteam test. Job. jdl Selected Virtual Organisation name (from --vo option): dteam Connecting to host lcgrb 01. gridpp. rl. ac. uk, port 7772 Logging to host lcgrb 01. gridpp. rl. ac. uk, port 9002 *************************************** JOB SUBMIT OUTCOME The job has been successfully submitted to the Network Server. Use edg-job-status command to check job current status. Your job identifier (edg_job. Id) is: - https: //lcgrb 01. gridpp. rl. ac. uk: 9000/rx. Naz 6 l. EWskg. J 5 cxw 9 wn. HQ ***************************************

Check the job status edg-job-status https: //lcgrb 01. gridpp. rl. ac. uk: 9000/rx. Naz 6 l. EWskg. J 5 cxw 9 wn. HQ ******************************* BOOKKEEPING INFORMATION: Status info for the Job : https: //lcgrb 01. gridpp. rl. ac. uk: 9000/rx. Naz 6 l. EWskg. J 5 cxw 9 wn. HQ Current Status: Scheduled Status Reason: Job successfully submitted to Globus Destination: lcg-ce. ecm. ub. es: 2119/jobmanager-pbs-dteam reached on: Tue Mar 28 13: 16: 12 2006 *******************************

Cancel a Job edg-job-cancel https: //lcgrb 01. gridpp. rl. ac. uk: 9000/rx. Naz 6 l. EWskg. J 5 cxw 9 wn. HQ Are you sure you want to remove specified job(s)? [y/n]n : y ======== edg-job-cancel Success ======== The cancellation request has been successfully submitted for the following job(s): - https: //lcgrb 01. gridpp. rl. ac. uk: 9000/rx. Naz 6 l. EWskg. J 5 cxw 9 wn. HQ ======================== n This command works only for jobs that have been already scheduled or are already running

Check status of all jobs edg-job-status --all --vo dteam Selected Virtual Organisation name (from --vo option): dteam Retrieving Information from LB server lcgrb 01. gridpp. rl. ac. uk: 9000 Please wait: this operation could take some seconds. ******************************* BOOKKEEPING INFORMATION: Status info for the Job : https: //lcgrb 01. gridpp. rl. ac. uk: 9000/8 WQJwag-9 YMC 4 Lk 2 SHye 5 g Current Status: Scheduled Status Reason: Job successfully submitted to Globus Destination: ce 01. tier 2. hep. manchester. ac. uk: 2119/jobmanager-pbs-dteam reached on: Tue Mar 28 13: 32: 44 2006 ************************************************************* BOOKKEEPING INFORMATION: Status info for the Job : https: //lcgrb 01. gridpp. rl. ac. uk: 9000/p. Csb-7 y. Wvz. SGHfr_j. Cq. Tq. Q Current Status: Running Status Reason: Job successfully submitted to Globus Destination: grid-ce. physik. uni-wuppertal. de: 2119/jobmanager-lcgpbs-large reached on: Tue Mar 28 13: 44: 49 2006 *******************************

Retrieve the output edg-job-get-output https: //lcgrb 01. gridpp. rl. ac. uk: 9000/p. Csb-7 y. Wvz. SGHfr_j. Cq. Tq. Q Retrieving files from host: lcgrb 01. gridpp. rl. ac. uk ( for https: //lcgrb 01. gridpp. rl. ac. uk: 9000/p. Csb-7 y. Wvz. SGHfr_j. Cq. Tq. Q ) ********************************** JOB GET OUTPUT OUTCOME Output sandbox files for the job: - https: //lcgrb 01. gridpp. rl. ac. uk: 9000/p. Csb-7 y. Wvz. SGHfr_j. Cq. Tq. Q have been successfully retrieved and stored in the directory: /tmp/job. Output/aforti_p. Csb-7 y. Wvz. SGHfr_j. Cq. Tq. Q **********************************

The Output less /tmp/job. Output/aforti_p. Csb-7 y. Wvz. SGHfr_j. Cq. Tq. Q total 32 drwxrwxr-x 2 aforti 4096 Mar 28 14: 49. / -rw-rw-r-- 1 aforti 0 Mar 28 14: 49 test. Job. err -rw-rw-r-- 1 aforti 22052 Mar 28 14: 49 test. Job. out

Documentation n LCG Main page n n http: //lcg. web. cern. ch/LCG/ Latest version of LCG User Manual n https: //edms. cern. ch/file/454439//LCG-2 -User. Guide. html