Status of Tsukuba KEK 21 September 2010 Sylvain

Status of Tsukuba, KEK, 21 September 2010 Sylvain Reynaud

The SAGA specification n SAGA : the specification – a recommendation of Open Grid Forum – a. S Simple A API – for using G middleware Grid • execution management – job submission, monitoring. . . • data management – transfer, list files, search… • service discovery • … – in user A Applications • implementations (or wrappers) exist for C++, Java and Python JSAGA n …in a uniform way use middlewares in a uniform way (independent of middlewares) use functionalities in a uniform way (consistent "look-and-feel") use language bindings in a similar way (language-independent specification) 2

SAGA vs. g. Lite User Interface g. Lite-UI VOMS, My. Prox y user JSAGA WMS, LCG-CE, CREAM-CE SRM, gsiftp LFC, AMG A BDII Source: S Jha, T Kielmann, A Merzky 3

Simple API for Grid Applications n Java code example (data management) Session session = Session. Factory. create. Session(true); URL url = URLFactory. create. URL("gsiftp: //cclcgseli 01. in 2 p 3. fr/t mp/"); // create SAGA object with a factory NSDirectory dir = NSFactory. create. NSDirectory(session, url); t of list conten directory tmp/ // use the created SAGA object gsiftp: //…/ List<URL> result = dir. list(); for (URL r : result) System. out. println(r); JSAGA 4

Simple API for Grid Applications n Java code example (execution management) Session session = Session. Factory. create. Session(true); URL url = URLFactory. create. URL("wms: //lappwms 02. in 2 p 3. fr/…"); // create SAGA object with a factory Job. Service svc = Job. Factory. create. Job. Service(session, url); b and submit a jo nd of // use the created SAGA object it for the e wa Job job = svc. run. Job("/bin/date --utc"); its execution job. wait. For(); System. out. println(job. get. State()); JSAGA 5

The JSAGA implementation n a Java implementation of the SAGA specification n focuses on – uniform usage of middleware…. support) – ease of extension………………. – efficiency and scalability………. – control of the API behavior. . . . trying) – operating-system independency n Under LGPL license JSAGA (beyond what they natively (thanks to the design of adaptor interf (configure binding rather than (tested on ) 6

Example – uniform usage SAGA job user applications JSAGA Globus Gatekeep. gsiftp WMS VOMS Grid. FTP t t t. tx tt ou ou JSAGA 7

Example – what happens behind SAGA job user applications n RSL wrapper script Globus JDL Gatekeep. n EGEE job gsiftp n WMS n modified JSAGA VOMS n EGEE proxy + attributes translate submit job poll status delegate transfer D-Gri d n n n DGrid proxy adapt + translate submit job receive status transfer – to any protocol VOMS WMS Grid. FTP LCG-CE Globus GK t xt t. . tx ou o t t t. tx tt ou ou JSAGA CREAM-CE Unicore 8

interface Layered software architecture implementation JSAGA 9

interface Layered software architecture n user applications SAGA Java binding JSAGA n JSAGA Used by application developers – a single uniform interface – object oriented, high-level n Implemented by adaptors – each "way" to implement a given feature has 1 interface – service oriented, low-level adaptors middleware APIs Used by end users – web portals, GUI, CLI… core engine JSAGA adaptors interfaces implementation n Implemented by middleware – each technology has its own interfaces 10

Layered software architecture n user applications – web portals, GUI, CLI… us fer tat s s s ran tput sten nge t u li cha o JSAGA adaptor interfaces n Used by application developers – a single uniform interface – object oriented, high-level core engine JSAGA Used by end users adaptor Globus GK upload data get output get status Implemented by adaptors – each "way" to implement a given feature has 1 interface – service oriented, low-level adaptor Grid. FTP JSAGA n n Implemented by middleware – each technology has its own interfaces 11

Layered software architecture n user applications – web portals, GUI, CLI… us fer tat s s s ran tput sten nge t u li cha o JSAGA adaptor interfaces n Used by application developers – a single uniform interface – object oriented, high-level core engine JSAGA Used by end users adaptor Globus GK upload data get output get status Implemented by adaptors – each "way" to implement a given feature has 1 interface – service oriented, low-level adaptor Grid. FTP JSAGA n n Implemented by middleware – each technology has its own interfaces 12

Layered software architecture user applications us fer tat s s s ran tput sten nge t u li cha o n JSAGA core engine JSAGA adaptor interfaces adaptor Globus GK upload data JSAGA – engine selects best interface for each user request adaptor Grid. FTP Middleware API can be used more efficiently get output get status n Developing new adaptors is easier – just wrap functionalities supported by middleware API 13

Design of adaptors interfaces As many interfaces as ways to implement each functionality n Example: ways to monitor jobs n SAGA get. State wait. For – API mode • poll job status • listen to notifications about job status changes – API granularity • individual jobs • list of jobs • jobs filter (e. g. by user, by date, by tag…) individual filtered job jobs local ssh naregi cream unicore 6 wsgram Job monitoring g. Lite-WMS gatekeeper done construction JSAGA planned query listen query adaptors JSAGA interfaces for status 14

Design of adaptors interfaces Optional interfaces for optimization n Example: ways to copy a file n SAGA copy – data read/write methods • either stream methods • or get/put methods – data copy • e. g. enable third-party transfer – delegated transfer dataadaptors data JSAGA reader writer copy interfaces cache zip file ftp sftp https http irods srb srm gsiftp rbyteio done construction JSAGA planned Physical 15

Supported technologies Ot rs he id Gr no. h tec local ssh naregi cream unicore 6 wsgram g. Lite-WMS he Ot rs id Gr no. h tec cache zip file ftp sftp https http irods srb srm gsiftp rbyteio catalog irods srb lfn rns rs he Ot id Gr no. h tec – e. g. no g. Lite-UI In. Mem. Cred Globus G. Legacy G. RFC 820 My. Proxy VOMS JKS SSH Login / pwd X 509 operating system n – tested on 16 done construction JSAGA planned Minimal dependencies on n external libraries n external tools Security Physical Logical Data gatekeeper Execution

Latest developments n New JSAGA adaptors n – for data management – reference Python binding for SAGA (as user interface) – reference Java binding for SAGA version 1. 1 (as implementation interface) • g. Lite-LFC, d. Cache (SRM) – for execution management • OGSA-BES (Unicore, ARC…) n JPy. SAGA is now compliant with Improvements in some g. Lite adaptors and their libraries – – JSAGA features thread-safety memory leaks … n JSAGA has a new configuration engine – enable dynamic configuration – support 1 configuration/session – detect conflicts at config. -time 17

Security context selection Try all security contexts Bind contexts with URL patterns n attractive for beginners n predictable/controllable behavior n can be inefficient n requires some knowledge about – waste of time (while trying) – waste of computing resource (risk of output staging failure) n can be unreliable – may create files with wrong ownership – may block an account with too many failed connection attempts JSAGA – used infrastructures – JSAGA configuration capabilities n JSAGA has a new configuration engine – enable dynamic configuration – support 1 configuration/session – detect conflicts at config. -time 18

Software quality n Adaptors – validated by a middlewareindependent SAGA test suite # SAGA protocols test-suite configuration gsiftp. base=gsiftp: //ccrugceli 01. in 2 p 3. fr/tmp/ gsiftp. base 2=gsiftp: //agena. c-s. fr/grid/tmp/ gsiftp. context=Open. Plast_proxy https. base=http: //grid. in 2 p 3. fr/html/Private/ https. context=Web_X 509 file. base=file: ///c: /tmp/ file. base 2=file: ///c: / – maven 'archetype' to generate skeleton of new adaptor project JSAGA n Automated build process – download and install • build tools • external libraries – generate source code – execute test-suites • unitary tests • integration tests – generate project web site • documentation • reports – generate installer GUI • extract library dependencies from maven project description 19

Installer GUI JSAGA 20

Licenses n LGPL license – for the core engine and for most adaptors n Optional licenses – for adaptors having external dependencies, which license is not compatible with LGPL – then, end-user must… • either accept the terms of the license agreement, • or go back to previous screen and uncheck the adaptor(s) JSAGA 21

Example applications n Pandora Gateway – the grid platform of the company • a leading Grid/Cloud Computing Business Services Provider – used in several grid projects, including JSAGA n Sim. Explorer • software for simulation experiment management • includes a workflow engine that uses JSAGA to run simulations on grids • used by experiences in – – biology computer science physics food industry 22

Example applications n j. CAE (Java CAE) • Computer Aided Engineering with support for distributed computing facilities n Calcul Intensif pour les LOgicie de CAO Electronique et les applications embarquées Bio. Informatique (protein sequence alignement. Elis@ • web portal for industrial and academic production grids / JSAGA • Elis@ is used in several grid projects, including Plasturgy (simulation of injectio 23

Other interoperability tools – JJS n Java Job Submission n uses JSAGA in order to hide middleware evolutions for – security • globus proxy VOMS proxy – data management n tool to submit jobs on EGEE/EGI grid infrastructure – efficient and scalable – resource selection based on the Qo. S observed while submitting jobs n • grid. FTP SRM http: //cc. in 2 p 3. fr/docenligne/269 JSAGA – execution • LCG-CE CREAM-CE • WMS n used by D 0 experiment to submit 15 000 jobs/day on – EGEE/EGI (mainly Europe) – Open Science Grid (USA) 24

Other interoperability tools – JUX n Java Universal e. Xplorer n multi-protocols file explorer n extensible n http: //cc. in 2 p 3. fr/docenligne/821 – for protocols • just put the JSAGA plug-ins into the directory "lib/". – for viewers JSAGA 25

Example applications n JSAGA provides command line interfaces for… – security • jsaga-context-init • jsaga-context-info • jsaga-context-destroy – execution management • • n jsaga-job-run jsaga-job-status jsaga-job-cancel jsaga-job-output – data management • • • jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rmdir jsaga-stat jsaga-test jsaga-logical Also useful as code examples to start with SAGA JSAGA 26

Interoperability of SAGA implementations n several implementations of SAGA – different languages (not necessarily) – different characteristics • features (e. g. automatic/configurable context selection, timeout…) • optimizations (e. g. caching of connections, meta-data, states…) • design choices (e. g. high-level / low-level adaptor interfaces) – different supported API extensions (e. g. SD, Advert, Messaging) – different supported technologies (adaptors) n how to use all these API & technologies simultaneously ? C++ Bindings SAGA-C++ JSAGA Java Bindings for SAGA Java SAGA 27

Interoperability of Java implementations System. set. Property("saga. factory", IMPL_NAME); File f = File. Factory. create. File(url); // old code File f = File. Factory. get. Instance(). create. File(url); // new code + get. Instance() File f = File. Factory. get. Instance(IMPL_NAME). create. File(url); Application // new (thread-safe) C++ Bindings SAGA-C++ JSAGA Java Bindings for SAGA + get. Instance() Java SAGA + get. Instance() 28

Interoperability of Python implementations C Python Jython Application Python Bindings for SAGA (Py. SAGA) Cppy. SAGA JPy. SAGA Jy. SAGA Cpp. SAGA (legacy) C++ Bindings SAGA-C++ JSAGA Java Bindings for SAGA Java SAGA 29

Interoperability : all together… C Python Jython Application Python Bindings for SAGA (Py. SAGA) Cppy. SAGA S A G A JPy. SAGA Cpp. SAGA (legacy) C++ Bindings SAGA-C++ JSAGA Jy. SAGA + get. Instance() Java Bindings for SAGA + get. Instance() Java SAGA + get. Instance() 30

Perspectives n Develop plug-ins for batch systems – DRMAA-based • Grid Engine n Develop API extensions – Grid-RPC (from SAGA core) – Service Discovery API (a SAGA extension) • with plug-ins for technologies – CLI-based, with ssh • torque • Grid Engine JSAGA – BDII • with plug-ins for schemas – GLUE – CIM 31

Perspectives job n Job submission – principle • describe user's job • submit it to the grid – middleware select the execution host – OGF standards • BES, DRMAA, SAGA – examples JSAGA Grid-RPC n Remote Procedure Call (RPC) – principle • deploy user's service on grid • call a procedure – middleware select the service instance – OGF standards • Grid-RPC, SAGA – examples 32

ns ? estio Qu JSAGA 33