SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky University

SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky, University of Westminster Summer School Budapest 02 -06 July 2012 SHIWA is supported by the FP 7 Capacities Programme under contract No. RI-261585

Workflow Interoperability Challenge 2

SSP Objectives and Services Objectives: • To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: • • To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse- and fine-grained workflow interoperability solution To offer Distributed Computing Infrastructure interoperability solution To provide desktop computer and portal interfaces to manage workflows Key actors: • researchers workflow engine developers workflow developers 3

SHIWA Data Structure Workflow description abstract workflow Workflow implementations concrete workflow • plain text which describes the inputs and outputs and explains what the workflow does • It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. Workflow configurations • It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. Workflow engines • It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them 4

Coarse-Grained Interoperability: Single Non-native Workflow of Workflow Engine B Workflow Engine A WF Submission client Submission Service Workflow Engine B DCI 5

Coarse-Grained Interoperability: Meta-Workflows of Workflow Engine A Workflow of Workflow Engine B J 1 J 2 WF 4 Submission client Submission Service Workflow Engine B Workflow Engine A J 3 DCI • native workflows: J 1, J 2, J 3 • non-native workflows: WF 4 - black boxes which are managed as legacy code applications 6

SSP: CGI Infrastructure DCIs SHIWA Science Gateway SHIWA Portal ARC DCI SHIWA Repository WF 1 GEMLCA admin WFn g. Lite DCI Unicore DCI Globus DCI GEMLCA Repository WS-PGRADE Workflow editor WF 1 WFm WE 1 WEp ASKALON WE Galaxy WE GWES WE Kepler WE GEMLCA Service Pegasus WE MOTEUR WE Proxy Server Pro. Active WE PGRADE WE WS-PGRADE Workflow engine SHIWA Science Gateway native WE WS-PGRADE v 3. 4. 1 repository submitter proxy management GEMLCA with GIB SHIWA Proxy Server WS-PGRADE GEMLCA + SHIWA repo GEMLCA with GIB SHIWA Proxy Server Resources Triana WE Taverna WE Workflow Engines local resources: invocation of locally deployed WEs portal WE submission to local cluster remote resources: through remotely pre-deployed WEs to ARC, g. Lite, Globus Unicore DCIs 7

SHIWA Portal: Abstract Workflow Jobs Ports 8

SHIWA Portal: Configuring Workflow 9

SHIWA Portal: Executing Workflow 10

SHIWA Repository: Browse View search domain 11

SHIWA Repository: Browse View outputs inputs data sets implementation 12

SHIWA Repository: Table View Browse Workflows Browse Implementations Advanced Search Sort 13

SHIWA Repository: Execution Support Execution data 14

SHIWA Portal: Legacy Code Explorer Taverna workflow 15

SHIWA Portal: Repository Portlet Workflow Repository 16

CGI Developer Scenario: Specifying WE 17

CGI Developer Scenario: Specifying WF through portal 18

CGI Developer Scenario: Specifying WF through desktop 19

CGI User Scenario: Native WE DCIs ARC DCI SHIWA Science Gateway g. Lite DCI Globus DCI Unicore DCI SHIWA Repository WF 1 WE 1 WS-PGRADE Workflow editor WF list WE + WF step 5 retrieve WF Pro. Active WE PGRADE WE Triana WE Taverna WE Workflow Engines user GEMLCA with GIB GEMLCA Service step 4 submit WF MOTEUR WE WEp step 2 edit WF GEMLCA Repository WS-PGRADE Workflow engine Kepler WE Pegasus WE WFm Galaxy WE GWES WE step 1 search WF step 3 retrieve WF data SHIWA Portal ASKALON WE WFn step 6 retrieve proxy step 7 run WF Proxy Server SHIWA Proxy Server 20

CGI User Scenario: Non-native WE DCIs ARC DCI SHIWA Science Gateway g. Lite DCI Globus DCI Unicore DCI SHIWA Repository WF 1 ASKALON WE GEMLCA Repository WF 1 SHIWA Portal WS-PGRADE Workflow Editor WS-PGRADE Workflow Engine WE 1 MOTEUR WE Pro. Active WE WFm Kepler WE Pegasus WE step 1 search WF Galaxy WE GWES WE WFn PGRADE WE WEp step 5 retrieve WF user step 2 edit WF WE + WF GEMLCA with GIB step 4 forward WF GEMLCA Service step 6 retrieve proxy Proxy Server SHIWA Proxy Server GEMLCA Client Workflow Engines MOTEUR Workflow editor MOTEUR GEMLCA Workflow UI Engine step 7 run WF step 3 submit WF Triana WE Taverna WE

CGI User Scenario: WF Execution via Desktop step 1 search WF SHIWA Science Gateway SHIWA Repository WF 1 step 2 retrieve WF step 4 edit WF user WFn SHUWA Desktop GEMLCA Repository WF 1 SHIWA Portal WFm WE 1 step 5 ask run WF step 3 forward WF bundle WEp WE step 6 submit WF WS-PGRADE Workflow Editor WS-PGRADE Workflow Engine GEMLCA with GIB GEMLCA Service DCIs ARC DCI g. Lite DCI Unicore DCI Globus DCI Proxy Server SHIWA Proxy Server 22

SSP DCI & FGI Infrastructure DCIs SHIWA Science Gateway ARC DCI SHIWA Portal SHIWA Repository WF 1 GEMLCA admin WFm WE 1 WFp GEMLCA submitter JSDL Translator Proxy Server ASKALON WE DCI Bridge Meta Broker Resources Galaxy WE GWES WE WS-PGRADE Workflow engine SHIWA Science Gateway Globus DCI GEMLCA Repository WF 1 WS-PGRADE Workflow editor g. Lite DCI WFn Unicore DCI Kepler WE Pegasus WE MOTEUR WE Pro. Active WE PGRADE WE Triana WE Taverna WE Workflow Engines native WE ASKALON, MOTEUR, WS-PGRADE WEs: Kepler, MOTEUR , Taverna & Triana WEs portal WS-PGRADE v 3. 4. 1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEs GEMLCA g. Lite, Globus and Unicore DCIs proxy server local ARC,

DCI Interoperability Scenario SHIWA Science Gateway SHIWA Portal SHIWA Repository WF 1 WFn step 1 search WF GEMLCA Repository step 3 retrieve WF data WF 1 WE 1 WS-PGRADE Workflow editor WS-PGRADE Workflow engine step 4 translate WF WFm WFp step 2 edit WF DCIs step 5 forward WF JSDL Translator step 6 select DCI step 7 retrieve certificate ARC DCI Bridge step 8 submit WF g. Lite DCI Unicore DCI Globus DCI Meta Broker Proxy Server 24

SHIWA Simulation Platform: Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) • joint portal and repository account given by the science gateway administrator DCI resources • certificate given by the relevant DCI administrator, for example the VO sysadmin Access to the SHIWA Portal ssp. shiwa-workflow. eu Access to the SHIWA Repository repo. shiwa-workflow. eu 25

Conclusions • the simulation platform supports the whole lifecycle of workflows, i. e. creating, testing, uploading, browsing, downloading and running workflows • users can use workflows of their own and of other workflow systems through the same user interface, i. e. they can re-cycle or share workflows created by other research teams or even other research communities • workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform • researchers (astro physicists, earth scientists, bio scientists, etc. ) can browse the workflow repository, select and run workflows through the simulation platform • the simulation platform provides access to major European einfrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources 26