Скачать презентацию SEE-GRID-SCI Hands-On Session Using Grid www see-grid-sci eu Скачать презентацию SEE-GRID-SCI Hands-On Session Using Grid www see-grid-sci eu

28a801c98312f73715123d492a9de200.ppt

  • Количество слайдов: 40

SEE-GRID-SCI Hands-On Session: Using Grid www. see-grid-sci. eu Regional SEE-GRID-SCI Meteo Training Faculty of SEE-GRID-SCI Hands-On Session: Using Grid www. see-grid-sci. eu Regional SEE-GRID-SCI Meteo Training Faculty of Electrical Engineering Banja Luka July 2 -3, 2009 Vladimir Slavnic Institute of Physics, Belgrade Serbia slavnic@scl. rs The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP 7 Research Infrastructures contract no. 211338

Setting up the user account Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Setting up the user account Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Setting up the user account User Interface (UI) Access point to Grid User must Setting up the user account User Interface (UI) Access point to Grid User must have a local account on machine Basic grid operations: list resources suitable for job submit jobs cancel jobs retrieve job output show job status file operations … Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Setting up the user account Certificates Grid passport X. 509 Subject Name (SN) /C=RS/O=AEGIS/OU=Institute Setting up the user account Certificates Grid passport X. 509 Subject Name (SN) /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Vladimir Slavnic Certification Authority (CA) Obtaining a certificate $ grid-cert-request userkey. pem usercert_request. pem usercert. pem Virtual Organisation (VO) Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Setting up the user account Setting up . globus Right permissions: – 444 usercert. Setting up the user account Setting up . globus Right permissions: – 444 usercert. pem – 400 userkey. pem Checking a certificate $ grid-cert-info -subject (-enddate) (-issuer) $ openssl verify -CApath /etc/grid-security/certificates/ ~/. globus/usercert. pem $ grid-proxy-init -verify Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Setting up the user account Proxies Standard proxies $ grid-proxy-init $ grid-proxy-info $ grid-proxy-destroy Setting up the user account Proxies Standard proxies $ grid-proxy-init $ grid-proxy-info $ grid-proxy-destroy VOMS proxies – Group – Role $ voms-proxy-init -voms $ voms-proxy-init –voms : [Role=] $ voms-proxy-info (-all) Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Setting up the user account Proxy renewal My. Proxy server $ echo $MYPROXY_SERVER $myproxy-init Setting up the user account Proxy renewal My. Proxy server $ echo $MYPROXY_SERVER $myproxy-init -s -d -n $myproxy-info -s -d $myproxy-destroy -s -d Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Information service Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, Information service Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

INFORMATION SERVICE lcg-infosites & lcg-info $lcg-infosites --vo <vo> <option> -v <verbosity> -f <site> --is INFORMATION SERVICE lcg-infosites & lcg-info $lcg-infosites --vo

INFORMATION SERVICE ldap search examples # RESOURCE ldapsearch -x -H ldap: //ce 64. phy. INFORMATION SERVICE ldap search examples # RESOURCE ldapsearch -x -H ldap: //ce 64. phy. bg. ac. yu: 2170/ -b “Mdsvoname=resource, o=Grid” # s. BDII ldapsearch -x -H ldap: //ce 64. phy. bg. ac. yu: 2170 -b mds-voname=AEGIS 01 -PHY-SCL, o=grid # top-level BDII ldapsearch -x -H ldap: //bdii. phy. bg. ac. yu: 2170/ -b 'Mds-voname=local, o=Grid' Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Wat. G Browser Web based Grid Information System browser Visualization of BDII information Highly Wat. G Browser Web based Grid Information System browser Visualization of BDII information Highly responsive tool because it supports partial refreshes and desynchronization of a web page. Developed with Google Web Toolkit (GWT) open source Java software development framework. Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Wat. G Browser http: //watgbrowser. scl. rs: 8080/ Regional SEE-GRID-SCI Grid Meteo. Training , Wat. G Browser http: //watgbrowser. scl. rs: 8080/ Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Data management Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, Data management Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT Storage element Classic SE LCG Disk pool manager CASTOR Storage Resource Manager DATA MANAGEMENT Storage element Classic SE LCG Disk pool manager CASTOR Storage Resource Manager (SRM) LFC Concept lfc and lcg commands Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT Grid file LFN – Logical File Name GUID - Grid Unique Identifier DATA MANAGEMENT Grid file LFN – Logical File Name GUID - Grid Unique Identifier SURL – Storage URL Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT LFC scheme Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering DATA MANAGEMENT LFC scheme Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT LFC commands • echo $LFC_HOST lfc-ls - List file/directory entries in a DATA MANAGEMENT LFC commands • echo $LFC_HOST lfc-ls - List file/directory entries in a directory lfc-mkdir - Create directory lfc-ln - Make a symbolic link to a file/directory lfc-ls - List file/directory entries in a directory lfc-chmod- Change access mode of a LFC file/directory lfc-chown - Change owner and group of a LFC file/directory lfc-getacl - Get file/directory access control lists lfc-setacl - Set file/directory access control lists … Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT LFC commands examples (1) Listing the entries of a LFC directory $ DATA MANAGEMENT LFC commands examples (1) Listing the entries of a LFC directory $ lfc-ls /grid/seegrid/vlada new_file result Creating directories in the LFC $ lfc-mkdir /grid/seegrid/vlada/new_dir $ lfc-ls -l /grid/seegrid/vlada drwxrwxr-x 0 165 101 0 Feb 19 00: 30 new_dir -rw-rw-r-- 1 165 101 27 Feb 18 01: 08 new_file -rw-rw-r-- 1 165 101 83 Feb 15 00: 30 result Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT LFC commands examples (2) Creation of symbolic links $ lfc-ln -s /grid/seegrid/vlada/result DATA MANAGEMENT LFC commands examples (2) Creation of symbolic links $ lfc-ln -s /grid/seegrid/vlada/result /grid/seegrid/vlada/result_link $ lfc-ls -l /grid/seegrid/vlada/result_link lrwxrwxrwx 1 165 101 0 Feb 19 00: 42 /grid/seegrid/vlada/rezult_link -> /grid/seegrid/vlada/rezult Adding metadata information to LFC entries $ lfc-setcomment /grid/seegrid/vlada/result "best result" $ lfc-ls --comment /grid/seegrid/vlada/result best result Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT lcg_utils-Replica Management echo $LCG_GFAL_INFOSYS lcg-cp Copies a Grid file to a local DATA MANAGEMENT lcg_utils-Replica Management echo $LCG_GFAL_INFOSYS lcg-cp Copies a Grid file to a local destination (download) lcg-cr Copies a file to a SE and registers the file in the catalogue (upload) lcg-del Deletes one file (either one replica or all replicas) lcg-rep Copies a file from one SE to another SE and registers it in the catalogue (replicate) lcg-gt Gets the TURL for a given SURL and transfer Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT lcg_utils-File Catalogue Interaction lcg-aa Adds an alias in the catalogue for a DATA MANAGEMENT lcg_utils-File Catalogue Interaction lcg-aa Adds an alias in the catalogue for a given GUID lcg-ra Removes an alias in the catalogue for a given GUID lcg-rf Registers in the catalogue a file residing on an SE lcg-uf Unregisters in the catalogue a file residing on an SE lcg-la Lists the aliases for a given LFN, GUID or SURL lcg-lr Lists the replicas for a given LFN, GUID or Regional. SURL SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT lcg_utils examples(1) Upload a file $ lcg-cr file: /home/vlada/banja_luka/mpi. jdl guid: 830 DATA MANAGEMENT lcg_utils examples(1) Upload a file $ lcg-cr file: /home/vlada/banja_luka/mpi. jdl guid: 830 e 7 a 82 -f 9 fb-4 c 04 -aafd-e 24 a 5 f 25 acdb specify lfn: $ lcg-cr -l lfn: /grid/seegrid/vlada/mpi file: /home/vlada/banja_luka/mpi. jdl guid: 25 fdcd 16 -bc 76 -4458 -9 c 1 a-8 c 597 fb 33194 Replicate a file (one replica per SE) $ lcg-rep -d grid 15. rcub. bg. ac. yu lfn: /grid/seegrid/vlada/mpi Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT lcg_utils examples(2) List replicas $ lcg-lr lfn: /grid/seegrid/vlada/mpi srm: //grid 15. rcub. DATA MANAGEMENT lcg_utils examples(2) List replicas $ lcg-lr lfn: /grid/seegrid/vlada/mpi srm: //grid 15. rcub. bg. ac. yu/dpm/rcub. bg. ac. yu/h ome/ae gis/generated/2009 -02 -18/file 8 f 99889 c-c 00 a 4 b 7 b 89 d 3 -d 9 abf 66 ba 103 srm: //se. phy. bg. ac. yu/dpm/phy. bg. ac. yu/home/a egis/generated/2009 -02 -18/file 576 fbae 0 -ba 6 a 4 b 21 -a 205 -3 d 115 b 2 cf 7 fb List guid $ lcg-lg lfn: /grid/seegrid/vlada/mpi Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DATA MANAGEMENT lcg_utils examples(3) Listing files and directories $ lcg-ls -l lfn: /grid/seegrid/vlada/mpi -rw-rw-r-- DATA MANAGEMENT lcg_utils examples(3) Listing files and directories $ lcg-ls -l lfn: /grid/seegrid/vlada/mpi -rw-rw-r-- 1 165 101 349 lfn: /grid/seegrid/vlada/mpi Copying files out of the Grid (download) $ lcg-cp lfn: /grid/seegrid/vlada/mpi file: /home/vlada/banja_luka/down/mpi Deleting replicas lcg-del srm: //grid 15. rcub. bg. ac. yu/dpm/rcub. bg. ac. yu/ home/aegis/generated/2009 -0219/file 36 d 25 d 0 d-ee 61 -477 eb 66740 b 3791786 a 4 Regional$ lcg-del -a lfn: /grid/aegis/vlada/mpi 2 -3, 2009 SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July

Data Management Web Portal Developed by Belgrade University Computer Centre Usage Data Management Logical Data Management Web Portal Developed by Belgrade University Computer Centre Usage Data Management Logical Files Development Environments and Portals Description Enables easy interaction with grid file catalogue using simple Web-based user interface. Based on SEE-GRID File Management Java API. Can be integrated with other Web applications. Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Data Management Web Portal Available features Directory browsing Directory management operations: creating new subdirectories, Data Management Web Portal Available features Directory browsing Directory management operations: creating new subdirectories, renaming/moving and deleting directories, replicating directory content, reading directory attributes and permissions. Copying and registering of new files to the catalog as well as file deletion. Reading file attributes and permissions. Replica & alias management operations. Upload and download of files with possible integration of application-defined upload handlers Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Data Management Web Portal • • http: //grid 02. rcub. bg. ac. yu: 8080/New. Data Management Web Portal • • http: //grid 02. rcub. bg. ac. yu: 8080/New. Data. Management/l ogin-REFS. jsp http: //grid 02. rcub. bg. ac. yu: 8080/New. Data. Manageme nt/login-WRF-ARW. jsp Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

Job operations Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, Job operations Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

JDL (Job Description Language) file attribute = expression Executable =

SUBMITING JOBS Generating proxy $ voms-proxy-init -voms <VO_NAME> Myproxy $ myproxy-init -d –n -s SUBMITING JOBS Generating proxy $ voms-proxy-init -voms Myproxy $ myproxy-init -d –n -s Delegate proxy to WMS $ glite-wms-job-delegate-proxy -d Job matching $ glite-wms-job-list-match -a test. jdl Job submission $ glite-wms-job-submit -d -i test. jdl Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

MONITORING JOBS View status of job $ glite-wms-job-status <job_ID> $ glite-wms-job-status -i <job_id_file> View MONITORING JOBS View status of job $ glite-wms-job-status $ glite-wms-job-status -i View logging details $ glite-wms-job-logging-info -v $ glite-wms-job-logging-info -v -i Verbosity level can be from 0 to 3. Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

CANCELING JOBS AND RETRIEVING OUTPUT Cancel a job $ glite-wms-job-cancel <job_ID> $ glite-wms-job-cancel -i CANCELING JOBS AND RETRIEVING OUTPUT Cancel a job $ glite-wms-job-cancel $ glite-wms-job-cancel -i Retrieve files from Output Sandbox $ glite-wms-job-output $ glite-wms-job-output -i $ glite-wms-job-output --dir Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

ADVANCED JOB TYPES Job Collection - submission of a set of jobs whose description ADVANCED JOB TYPES Job Collection - submission of a set of jobs whose description files are placed in one directory Parametric Jobs - submission of a set of jobs having the same descriptions apart from the values of the parametric attributes. Use when your jobs differs only in argument values or input/output files. DAG jobs (Direct Acyclic Graphs) - submission of a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs. Interactive Jobs MPI Jobs Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

JOB COLLECTION Type = JOB COLLECTION Type = "Collection"; Input. Sandbox = (“file. A”); Requirements = Reg. Exp("ce 64. phy. bg. ac. yu*", other. Glue. CEUnique. ID); nodes = { [ Job. Type = "normal"; Executable = "/bin/hostname"; Arguments = "-i"; Std. Output = "out"; Input. Sandbox = (root. Input. Sandbox, “file. B”); Std. Error = "err"; Output. Sandbox ={"out", "err"}; ], [ Job. Type = "Normal"; Executable = "/bin/date"; Std. Output = "date"; Std. Error = "err"; Output. Sandbox= {"date"}; ] }; Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

JOB COLLECTION submission Collection defined in one file $ glite-wms-job-submit -a -o id collection. JOB COLLECTION submission Collection defined in one file $ glite-wms-job-submit -a -o id collection. jdl • • Folder contains jdl files $ glite-wms-job-submit -a --collection jdl Collection status view Collection output $ glite-wms-job-output -i id --dir kolekcija $ ls kolekcija ids_nodes. map Node_0 Node_1 Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

PARAMETRIC JOBS Type =

PARAMETRIC JOBS Type = PARAMETRIC JOBS Type = "Job"; Job. Type = “Parametric"; Executable = "exec"; Std. Input = "input_PARAM_. txt"; Std. Output = "output_PARAM_. txt"; Parameters = 10; Parameter. Start = 1; Parameter. Step = 1; Input. Sandbox = {"", ""}; Output. Sandbox = {"output_PARAM_. txt"}; Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

DAG JOBS [ Type = DAG JOBS [ Type = "dag"; Input. Sandbox = {"job. sh"}; Nodes = [ node. A = [ file ="A. jdl" ; ]; node. B = [ file =“b. jdl" ; ]; node. C = [ file =“C. jdl" ; ]; node. D = [ Description = [ Job. Type = "Normal"; Executable = "job. sh"; Std. Output = "std. out"; Std. Error = "std. err"; Input. Sandbox = {root. Input. Sandbox[0]}; Output. Sandbox = {"std. out", "std. err"}; ]; ]; ]; Dependencies = { {node. A, node. B}, {node. A, node. C}, {{node. B, node. C}, node. D} }; ] Regional SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009

MPI JOBS JDL file: Type =

LINKS http: //aegis-ca. rcub. bg. ac. yu/ https: //edms. cern. ch/file/722398//g. Lite-3 User. Guide. LINKS http: //aegis-ca. rcub. bg. ac. yu/ https: //edms. cern. ch/file/722398//g. Lite-3 User. Guide. pdf http: //wiki. egeesee. org/index. php/Quick_User_Guide_for_Subm itting_Jobs https: //edms. cern. ch/document/590869/1/ http: //wiki. egee-see. org/index. php/SEEGRID_MPI_User_Guide http: //wiki. egee-see. org http: //wiki. egeesee. org/index. php/Data_Management_Web_Por Regionaltal_-_User_Guide SEE-GRID-SCI Grid Meteo. Training , Faculty of Electrical Engineering Banja Luka, July 2 -3, 2009