FJPPL meeting Lyon 17 th of February 2010

FJPPL meeting Lyon, 17 th of February 2010 Sylvain Reynaud

SAGA specification (overview) JSAGA 2

SAGA specification / JSAGA impl. Security urit sec se r r nito n to mo mo trol con co JSAGA Data ol ol toc t pro Execution 3

Goal: seamless job submission job desc. JSAGA EO EE GP last infrastructures heterogeneity JSAGA (e. g. EGEE, OSG, DEISA) • which security context to use? • which transfer path (according to firewalls, protocols, security…) ? • which environment variables ? • how to use commands available on workers (wget, srmcp, Sget…) ? g. Lite plug-ins Globus plug-ins RSL staging graph de leg at based on e file ss ta gin g JDL WMS SRM input data SAGA Grid. FTP (e. g. g. Lite, Globus, Unicore) • how to use the legacy APIs ? LCG-CE job JSAGA firewall middleware heterogeneity WS-GRAM job 4

Both implementer & user of SAGA n A single interface for using all middlewares n As many interfaces as design approaches n As many interfaces as used technologies r velope s plug-in per develo plug-ins legacy APIs JSAGA A single interface for using all grid infrastructures core engine JSAGA plug-ins interfaces JSAGA n er JSAGA Ready to use software, adapted to targeted scientific field tion de SAGA n applica JSAGA jobs collection implementation end us user applications interface 5

Plug-ins interfaces in JSAGA n Close to application developer needs JSAGA n – object-oriented – high-level – uniform interface for all the supported technologies n design objectives – easy to use … but << certainly not simple to implement >> (T. Kielmann) plug-ins interfaces Close to existing middleware APIs – service-oriented – low-level – as many interfaces as design approaches + optional interfaces n design objectives – easy to implement – enable efficient usage of middleware APIs • engine code = 2 x plug-ins code JSAGA 6

Plug-ins interfaces JSAGA 7

Plug-ins interfaces - job (streams) Streaming Plug-in interfaces: direct/buffered/redirected streams used before/during/after execution set stream set get for non- for interactive get. Input get. Output get. Error SAGA user interface: get. Input / get. Output naregi cream local ssh unicore 6 wsgram g. Lite-WMS gatekeeper done construction JSAGA planned Job control 8

Plug-ins interfaces - job (monitoring) n Several ways to monitor jobs Monitoring – API mode Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs query listen query • poll job status • listen to notifications about job status changes status for individual filtered job jobs – API granularity • individual jobs • list of jobs • jobs filter (e. g. by user, by date, by tag…) get. State wait. For SAGA user interface: get. State / wait. For Plug-in implements several interfaces n User see a single interface: SAGA n naregi cream local ssh unicore 6 wsgram g. Lite-WMS gatekeeper done construction JSAGA planned Job monitoring 9

Plug-ins interfaces - data Optimizations: n Optional interfaces for plug-ins – e. g. third-party transfer, recursive find with meta-data n Cache management of – opened connections – meta-data – content • explicit via plug-in cache zip file ftp sftp https http Physical irods srb srm gsiftp rbyteio catalog irods srb lfn rns done construction JSAGA planned Logical cache 10

Plug-ins interfaces - security plug-in supported attributes /etc/grid-security/certificates CA jsaga-context-init command line /tmp/x 509 up_u_$UID pass phrase proxy path proxy ~/. globus/user*. pem p 12 key Login / pwd VOMS My. Proxy G. RFC 820 G. Legacy Globus In. Mem. Cred SSH X 509 done construction JSAGA planned cert 11

JSAGA plug-ins Ot rs he id Gr no. h tec local ssh naregi cream unicore 6 wsgram g. Lite-LB gatekeeper local ssh naregi cream unicore 6 wsgram g. Lite-WMS he Ot rs id Gr no. h tec cache zip file ftp sftp https http irods srb srm gsiftp rbyteio catalog irods srb lfn rns rs he Ot id Gr no. h tec X 509 Login / pwd SSH JKS VOMS My. Proxy G. RFC 820 G. Legacy Globus In. Mem. Cred 12 done construction JSAGA planned gatekeeper Security Physical Logical Data Job monitoring Job control Execution

JSAGA plug-ins n SAGA hides most of the differences between the technologies n Information on remaining differences is available for each plug-in module… – on JSAGA web site • Frequently Asked Questions • Configuration example – with command line jsaga-help • supported protocols • associated security context • usage, default attributes JSAGA usage: jsaga-help -a <ctx. Id>. <attr> | --config | -d <mode> | -h | -j <mode> | -s <mode> | -v [-D <ctx. Id>. <attr>=<value> where: -a, --attribute <ctx. Id>. <attr> --config -d, --data <mode> -D <ctx. Id>. <attr>=<value> -h, --help -j, --job <mode> -s, --security <mode> -v, --version Output the value of security context attribute Output the effective configuration Information about data protocols. <mode> = service | context Set context instance attribute (e. g. -DVOMS[0]. User. VO=dteam) Display this help and exit Information about job services. <mode> = service | context Information about security context instances. <mode> = usage | default | missing Output version information and exit 13

JSAGA command line interfaces n JSAGA provides command line interfaces for… – security • jsaga-context-init • jsaga-context-info • jsaga-context-destroy – execution management • jsaga-job-run • jsaga-job-status • jsaga-job-cancel n – data management • • • jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rmdir jsaga-stat jsaga-test jsaga-logical Also useful as code examples to start with SAGA JSAGA 14

Software quality n Build process fully automated – download and install • build tools • external libraries – generate source code – execute test-suites • unitary tests • integration tests – generate project web site • documentation • reports – generate installer GUI n Plug-ins – minimal external dependencies • e. g. g. Lite-UI no needed • OS-independent – maven 'archetype' to generate skeleton of new plug-in project – validated by a middlewareindependent SAGA test suite # SAGA protocols test-suite configuration gsiftp. base=gsiftp: //ccrugceli 01. in 2 p 3. fr/tmp/ gsiftp. base 2=gsiftp: //agena. c-s. fr/grid/tmp/ gsiftp. context=Open. Plast_proxy https. base=http: //grid. in 2 p 3. fr/html/Private/ https. context=Web_X 509 file. base=file: ///c: /tmp/ file. base 2=file: ///c: / JSAGA 15

Installer GUI JSAGA 16

Licenses n LGPL license – for the core engine and for most plug-ins n Optional licenses – for plug-ins having external dependencies, which license is not compatible with LGPL – then, end-user must… • either accept the terms of the license agreement, • or go back to previous screen and uncheck the plug-in(s) JSAGA 17

Contributions n elis@ n • a web portal for submitting jobs to industrial and research grid infrastructures • uses JSAGA to hide grid infrastructure heterogeneity n contribution from – many plug-ins / • Job: g. Lite-WMS, Unicore 6, Globus WS-GRAM, SSH, local • Data: RByte. IO, SFTP – automated SAGA execution management test-suite JSAGA JUX (Java Universal e. Xplorer) • a multi-protocols file browser n contribution from P. Calvat – plug-ins for SRB – plug-ins for i. RODS n logical/physical plug-ins JJS (Java Job Submission) • a tool for running efficiently short-life jobs on EGEE n contribution from P. Calvat – compare overhead with JSAGA / with direct calls to Globus API – scalability tests by JJS users 18

Contributions n Sim. Explorer n • a set of tools, including a workflow engine providing distributed computing facilities, for managing simulation experiments n contribution from – scalability tests • thousands jobs on many WMS – tests with long-run workflows • request + test: auto-reset job service when context renewed j. CAE (Java Computer Aided Engineering) • Computer Aided Engineering on distributed computing environment n contribution from – currently working on SSH plug-in • enable controlling and monitoring a SSH job from another JVM – maybe other plug-ins later… • GRIA, LSF, torque ? – wiki pages JSAGA 19

Contributions n (not a user of JSAGA) • Open source technologies to manage, preserve, and link digital content n contribution from – access to their grid services deployed for D-Grid • Unicore 6 • Globus – v 2 components (Gatekeeper) – v 4 components (WS-GRAM) n (upcoming) contribution from P. Y. Jallud – data plug-in to browse content from Fedora Commons repository – will be included in JUX • g. Lite JSAGA 20

Other ideas for contributions n New users => new use-cases – new tests • bug-fix requests • new wiki pages – new ideas • feature requests • help for testing these features – need support for additional technologies • contribute with new plug-ins n New plug-ins – for job • • • batch systems (with DRMAA) grids clouds peer-to-peer desktop grids pilot job systems – for data • physical/logical file protocols • application level protocols – for security • SSO (Shibboleth, Open. SSO…) JSAGA 21

Conclusion n JSAGA plug-in interfaces designed to – minimize amount of code needed to support new technologies – minimize impact of API uniformity on efficiency and scalability n JSAGA plug-in implementations minimize dependencies – on external tools and libraries (e. g. g. Lite-UI) – on operating system (tested on ) n JSAGA is opened to external contributions JSAGA 22

ank Th u! yo JSAGA 23

Backup slides JSAGA 24

C C' Transfer path depends on… n grid or site R 1 E 1 n execution service – network filtering policy – protocols supported for staging – commands available on workers n transfer protocol – services available from workers (close – access mode (RO, WO, RW) Storage Element, shared FS) – third-party transfer – supported context instances – supported data protection level SMTP SRB R 1 OPla st EG GSIFTP EE CA OP GSIFTPlast C' n data to stage – – common result std-error shared by several jobs installed on some worker nodes file size required data protection level HTTP OPla st job VOMS eper srb: // gatekesr m: / CC-IN 2 P 3 C EGEE WMS eper gateke m wsgra Open. Plast Grid eper gateke calhost lo E 1 t Open. Plas JSAGA / lfn: // g si ft Globus OPla st p: // OPl GSIFTPast job http: // Globus tar: // OPla st World job 25

C C'C'' common E E src executable Transfer path depends on… n grid or site – network filtering policy – protocols supported for staging – commands available on workers n transfer protocol – services available from workers (close – access mode (RO, WO, RW) Storage Element, shared FS) – third-party transfer – supported context instances – supported data protection level SRB R 1 OPla st EG GSIFTP EE CA C' E src OP GSIFTPlast D 1 result E 1 n execution service SMTP input data D 1 R 1 std-error n data to stage – – shared by several jobs installed on some worker nodes file size required data protection level HTTP C" OPla st job OPla st C OPl GSIFTPast job TAR E s. Get OPla st E 1 JSAGA job 26