Скачать презентацию The EPIKH Project Exchange Programme to advance e-Infrastructure Скачать презентацию The EPIKH Project Exchange Programme to advance e-Infrastructure

31d2f907c32d7713f1e8a5b19b16d104.ppt

  • Количество слайдов: 15

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) GISELA Additional Services Diego Scardaci The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) GISELA Additional Services Diego Scardaci (diego. [email protected] infn. it) INFN Dept. of Catania Joint CHAIN/GISELAEPIKH Application Porting School Valparaiso (Chile), 29. 11 -2010 -09. 12. 2010 www. epikh. eu

Outline • The Secure Storage Service for the g. Lite Middleware • Watchdog • Outline • The Secure Storage Service for the g. Lite Middleware • Watchdog • LCG-REC • Our. Grid Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 2

Secure Storage • Provides g. Lite users with suitable and simple tools to store Secure Storage • Provides g. Lite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way. • The service is composed by the following components: • Command Line Applications: commands integrated in the g. Lite User Interface to encrypt/upload and decrypt/ download files. • Application Program Interface: allows the developer to write programs able to manage confidential data. • Keystore: a new grid element used to store and retrieve the users’ keys. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 3

Using Secure Storage • Secure Storage is a GISELA infrastructure service; • The Secure Using Secure Storage • Secure Storage is a GISELA infrastructure service; • The Secure Storage libraries are been installed in the GISELA Central UIs and in the GISELA sites; • A central Keystore has just been installed in the machine securestorage-01. ct. infn. it; • To use Secure Storage: • Read the documentation in the wiki site: http: //grid. ct. infn. it/twiki/bin/view/EELA 2/Secure. Storage; • See the code samples; • Contact: Diego Scardaci (mail: diego. [email protected] infn. it) Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 4

Using Secure Storage • • Set the following environment variables in the UI or Using Secure Storage • • Set the following environment variables in the UI or in the WN: • SS_GRID_KEYSTORE_HOST=: 25406 • SS_GRID_KEYSTORE_DN= In the main script of your application, set the following environment variables to be able to use Secure Storage in a WN: • PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorageclient/bin/: ${PATH} • LD_LIBRARY_PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestora ge-client/lib/: ${LD_LIBRARY_PATH} #!/bin/sh … MY JOB export SS_GRID_KEYSTORE_HOST=: 25406 [ export SS_GRID_KEYSTORE_DN= Type = "Job"; export LCG_CATALOG_TYPE=lfc Job. Type = "Normal"; export LFC_HOST=lfc. eela. ufrj. br Executable = "/bin/sh"; Arguments = "My. SSApplication. sh"; Std. Output = "My. SSApplication. out"; Std. Error = "My. SSApplication. err"; Input. Sandbox = {“My. SSApplication. sh”, …}; Output. Sandbox = {"My. SSApplication. err", " My. SSApplication. out"}; export LCG_RFIO_TYPE=dpm export PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorage -client/bin/: ${PATH} export LD_LIBRARY_PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/secure storage-client/lib/: ${LD_LIBRARY_PATH} #run application ] Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 5

An example of Secure Storage Job #!/bin/sh export SS_GRID_KEYSTORE_HOST=securestorage-01. ct. infn. it: 25406 export An example of Secure Storage Job #!/bin/sh export SS_GRID_KEYSTORE_HOST=securestorage-01. ct. infn. it: 25406 export SS_GRID_KEYSTORE_DN="/C=IT/O=INFN/OU=Host/L=Catania/CN=securestorage-01. ct. infn. it“ export LCG_CATALOG_TYPE=lfc export LFC_HOST=lfc. eela. ufrj. br export LCG_RFIO_TYPE=dpm export PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorage-client/bin/: ${PATH} export LD_LIBRARY_PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorageclient/lib/: ${LD_LIBRARY_PATH} echo "SECURE STORAGE TEST" > text_file_5. txt lcg-scr --vo prod. vo. eu-eela. eu -d lnx 097. eela. if. ufrj. br --vo_permission /C=IT/O=INFN/OU=Personal Certificate/L=Catania/CN=Diego Scardaci -l lfn: /grid/prod. vo. eu-eela. eu/text_file_6. enc text_file_5. txt lcg-scp --vo prod. vo. eu-eela. eu lfn: /grid/prod. vo. eu-eela. eu/text_file_6. enc file: $PWD/text_file_copy_dec. txt echo "That's all folks!“ exit $? [ Type = "Job"; Job. Type = "Normal"; Executable = "/bin/sh"; MY SCRIPT Arguments = ". /securestorage_test. sh"; Std. Output = "out-securestorage_test. out"; Std. Error = "err-securestorage_test. err"; Input. Sandbox = {". /securestorage_test. sh"}; MY JOB Output. Sandbox = {"text_file_copy_dec. txt", "errsecurestorage_test. err", "out-securestorage_test. out"}; Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 ] 6

An example of Secure Storage API usage inside a C program #include An example of Secure Storage API usage inside a C program #include "securestorage. h“ … int main (int argc, char *argv[]) { int fd; int read_byte=0; unsigned char *buffer; FILE *fdout; if (argc != 3) { fprintf(stderr, "Usage: %s n", argv[0]); exit(1); } if ( (fdout = fopen(argv[2], "wb")) == NULL ) { perror("fopen"); exit(1); } if ( (fd=securestorage_open(argv[1], O_RDONLY, 0644)) < 0 ) { /*manage the error*/ } buffer = calloc(1, BUFLEN+1); do { if ((read_byte=securestorage_read(fd, buffer, BUFLEN))<0){ /*manage the error*/} if (read_byte>0) { if (fwrite(buffer, 1, read_byte, fdout) < 0) {/*manage the error*/} } } while (read_byte>0); if ( securestorage_close (fd) < 0 ) {/*manage the error*/ } fclose(fdout); free(buffer); exit(0); } Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 7

Watchdog Why – Especially long term jobs require to be monitored and controlled during Watchdog Why – Especially long term jobs require to be monitored and controlled during their execution. How – Perform job control and monitoring using grid services in the less invasive way. Observations – Almost all jobs submitted on the grid are piloted by shell scripts § Shell scripting allow to get precious info in case of faults § Shell scripting can pilot more complex batch execution – Both AMGA and SE+LFC can be used as the simplest IS on the grid. § lfc-* and lcg-* tools already available for file creation and retrieve § The latency of CLI tools for the storage is very low compared to long term jobs Requirements: – Monitor job execution watching snapshot of files produced by the job execution § File snapshot will be reported on LFC+SE or AMGA servers – It would be useful to configure the monitoring tool accordingly to the user needs § Few shell environment variables can be used to configure the watchdog tool – Control the job execution accessing directly on the WN § It is possible to send commands on the WN Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 8

Watchdog UI UI WN The Watchdog is a shell script to be included in Watchdog UI UI WN The Watchdog is a shell script to be included in the JDL main script. – Some watchdog features: § It starts in background before to UI run the long term job § The watchdog runs as long as Commands the main job or Scripts § The main script can control, stop and wait until the watchdog has finished § Easily and highly configurable Amga and customizable § The watchdog does not compromise the CPU power of the WN JDL WN File snapshots LFC+SE Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 9

Using the Watchdog • Download Watchdog from GISELA forge: https: //forge. eueela. eu/frs/? group_id=103 Using the Watchdog • Download Watchdog from GISELA forge: https: //forge. eueela. eu/frs/? group_id=103 • Read the documentation in the wiki http: //grid. ct. infn. it/twiki/bin/view/EELA 2/Watch. Dog site: • See the code samples distributed inside the packages: • script. jdl • script. sh • Add Watchdog capabilities to your application! • Contact: Riccardo Bruno (mail: riccardo. [email protected] infn. it) Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 10

Using the Watchdog • Configure the Watchdog setting the watchdog. conf file (see the Using the Watchdog • Configure the Watchdog setting the watchdog. conf file (see the wiki); • Applications using Watchdog MUST include the files watchdog. sh, watchdog. ctrl, watchdog. conf. MY JOB My. Script. sh #!/bin/sh [ … Type = "Job"; # prepare and start the watchdog Job. Type = "Normal"; Executable = "/bin/bash"; PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/ : ${PATH} Std. Output = "file. out"; chmod +x watchdog. sh Std. Error = "file. err"; chmod +x watchdog. ctrl Input. Sandbox = {"watchdog. sh", "watchdog. ctrl", "watchdog. co nf", “My. Script. sh"}; chmod +x watchdog. conf Output. Sandbox = {“My. Application. out", "My. Application. err", "wat chdog. log”}; . /watchdog. ctrl start. . . #run application … Arguments = "script. sh"; #stop and wait the watchdog completes ] . /watchdog. ctrl stop Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 11

LCG-REC The new tools have the following prefix: 'lcg-rec-*' to enphatize the recursive interaction LCG-REC The new tools have the following prefix: 'lcg-rec-*' to enphatize the recursive interaction with the calalog/SE. These tools are described briefly below: lcg-rec-cr - Exactly like the corresponding lcg-cr command, this tool copy and register a whole directory structure from a local filesystem in the UI to the file catalog. The same file hierarchy will be kept in the catalog and the related file contents will be stored on a storage element. • lcg-rec-cp - Exactly like the corresponding lcg-cp command, this tool copy from catalog a directory structure downloading the related file content from the storage elements. The same file hierarchy will be kept in the local filesystem. • lcg-rec-del - Exactly like the corresponding lcg-del command, this tool removes a entire directory structure from the catalog and removes each replica of related file from the storage elements. • lcg-rec-rep - Exaclty like the corresponding lcg-rep command, this tool adds recursively a replica to all files of a directory. https: //grid. ct. infn. it/twiki/bin/view/GILDA/Lcg-rec-tools Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 12

Our. Grid • Production opportunistic infrastructure powered by Our. Grid Valparaiso, Joint CHAIN/GISELA/EPIKH Grid Our. Grid • Production opportunistic infrastructure powered by Our. Grid Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 13

Our. Grid • Production gateway allowing the service and the opportunistic infrastructure to interoperate Our. Grid • Production gateway allowing the service and the opportunistic infrastructure to interoperate Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 14

Thank you very much! Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 Thank you very much! Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01 -12 -2010 15