Building an European Research Community through Interoperable Workflows and Data ER-flow Prof. Gabor Terstyanszky, University of Westminster, UK Heildelberg, 16 May 2012 SHIWA is supported by the FP 7 Capacities Programme under contract No. RI-261585
Key Players and Challenges E-scientists - researchers of one particular research field, i. e. Astrophysicists, Computational Chemists, Heliophysicists etc. with basic computing knowledge - 10 or 100 thousands or millions Challenges: they are not familiar with the technology to run experiments on computing infrastructures and probably they will never learn it Workflow developers - they are familiar with both Computer Science and a particular research field - up to a few thousands Workflow engine developers - Computer Scientists with knowledge about data and compute technologies - up to a few hundreds 2
Technology Challenges 3
What WE developers want and need They need - to publish WEs somewhere - to test access to WEs somewhere They want - to provide access to WEs Workflow Engine Repository Supercomputer based SGs (DEISA, Tera. Grid) Cluster based service grids (SGs) (EGEE, OSG, etc. ) Local clusters Clouds Supercomputers Desktop grids (DGs) (BOINC, Condor, etc. ) Grid systems E-science infrastructure 4
What WF developers want and need They need - to develop WFs somewhere - to publish WFs somewhere - to manage WFs somewhere - to execute WFs somewhere They want - to develop & publish WFs Workflow Repository Science Gateway Supercomputer based SGs (DEISA, Tera. Grid) Cluster based service grids (SGs) (EGEE, OSG, etc. ) Local clusters Clouds Supercomputers Desktop grids (DGs) (BOINC, Condor, etc. ) Grid systems E-science infrastructure 5
What e-scientists want and need They need - to run experiments seamlessly i. e. executing workflows which access to data and compute resources hiding all technical details They want - to run experiments Workflow Repository Science Gateway Supercomputer based SGs (DEISA, Tera. Grid) Cluster based service grids (SGs) (EGEE, OSG, etc. ) Local clusters Clouds Supercomputers Desktop grids (DGs) (BOINC, Condor, etc. ) Grid systems E-science infrastructure 6
SHIWA Simulation Platform: Development Platform DCIs SHIWA Science Gateway SHIWA Portal GEMLCA admin ARC DCI SHIWA Repository WF 1 WFn g. Lite DCI Unicore DCI Globus DCI GEMLCA Repository WS-PGRADE Workflow editor WS-PGRADE Workflow engine WF 1 WFm WE 1 WEp ASKALON WE Galaxy WE GWES WE Kepler WE GEMLCA Service Pegasus WE MOTEUR WE Proxy Server Pro. Active WE PGRADE WE GEMLCA with GIB SHIWA Proxy Server Workflow Engines Triana WE Taverna WE
SHIWA Portal: Editing Workflow 8
SHIWA Portal: Configuring Workflow 9
SHIWA Portal: Executing Workflow 10
SHIWA Workflow Repository 11
SHIWA Workflow Repository 12
WE Developer Scenario: step 1 specify WE data SHIWA Science Gateway SHIWA Portal workflow engine developer GEMLCA admin SHIWA Repository WF 1 WFn GEMLCA Repository step 2 upload WE binary, dependencies WS-PGRADE Workflow editor WS-PGRADE Workflow engine WF 1 WFm WE 1 WEp step 3 deploy WE GEMLCA with GIB GEMLCA Service Proxy Server SHIWA Proxy Server 13
WF Developer Scenario: step 1 specify WF data SHIWA Science Gateway SHIWA Repository WF 1 WFn SHIWA Portal WF 1 WFm WS-PGRADE Workflow editor WE 1 WEp step 3 deploy WF WS-PGRADE Workflow engine GEMLCA Repository workflow developer step 2 upload WF GEMLCA with GIB GEMLCA Service Proxy Server SHIWA Proxy Server 14 14
WF Developer Scenario: DCIs ARC DCI SHIWA Science Gateway g. Lite DCI Globus DCI Unicore DCI SHIWA Repository WF 1 ASKALON WE step 3 retrieve WF data Kepler WE Pegasus WE MOTEUR WE Pro. Active WE step 1 search WF Galaxy WE GWES WE WFn PGRADE WE Triana WE Taverna WE GEMLCA Repository SHIWA Portal WFm WE 1 WS-PGRADE Workflow editor WF list WF 1 WEp WE + WF WS-PGRADE Workflow engine step 5 retrieve WF GEMLCA with GIB GEMLCA Service step 4 submit WF step 2 edit WF Workflow Engines WF developer step 6 retrieve proxy step 7 run WF Proxy Server SHIWA Proxy Server 15
Execution Infrastructure DCIs ARC DCI Astrophysics Science Gateway SHIWA Repository WF 1 g. Lite DCI Unicore DCI Globus DCI WFn Heliophysics Science Gateway ASKALON WE Galaxy WE GWES WE Kepler WE Pegasus WE Computational Chemistry Science Gateway MOTEUR WE Pro. Active WE PGRADE WE Triana WE Taverna WE Workflow Engines Life sciences Science Gateway 16
End User View 17 17
End User View 18
End User View 19
End User View 20
User Tailored View 21
User Tailored View 22
E-scientist Scenario: DCIs ARC DCI SHIWA Science Gateway g. Lite DCI Globus DCI Unicore DCI SHIWA Repository ASKALON WE Kepler WE Pegasus WE MOTEUR WE Pro. Active WE WFn Galaxy WE GWES WE WF 1 PGRADE WE Triana WE Taverna WE GEMLCA Repository SHIWA Portal WFm WE 1 WS-PGRADE End User View WF list WF 1 WEp WE + WF WS-PGRADE Workflow engine retrieve WF GEMLCA with GIB GEMLCA Service submit WF Select and run WF Workflow Engines WF developer run WF retrieve proxy Proxy Server SHIWA Proxy Server 23
ER-flow Project Partners: University of Westminster Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete Centre National de la Recherche Scientifique Stichting European Grid Initiative Academic Medical Center of the University of Amsterdam Technische Universität Dresden Ludwig-Maximilians-Universität München University College London Trinity College Daudublin Istituto Nazionale di Astrofisica Uo. W United Kingdom MTA-SZTAKI Hungary CNRS EGI. eu AMC France The Netherlands TUD LMU UCL TCD INAF Germany United KIngdom Ireland Italy Technology providers: CNRS, EGI. eu, MTA-SZTAKI, Uo. W Research Communities: Astro-Physics Computational Chemistry Helio-Physics Life Science INAF LMU + TUD TCD + UCL AMC Duration: September 2012 – August 2014 24
ER-flow Research Communities name ASTRO PHYSICS COMPCAT FRANEC/BASTI Las. Mo. G MESSTREAM PLANCK Vis. IVO short description capture comets from the interstellar space evolutionary stellar models modified gravity models modelling the dynamic evolution of meteoroid stream simulation of the ESA Planck mission Computational Cosmology COMPUTATIONAL CHEMISTRY GROMACS Molecular Dynamic application Energy Minimisation Equilibration Single TPR CADDSuite docking application Docking with ligand generation Docking without ligand generation NWChem Quantum Chemistry/Molecular Dynamics application Geometry optimisation = basic WF Opt+freq Freq WF TD-DFT WF Mulliken WF Solvation WF Spectroscopic analysis = Metaworkflow Parameter sweep = Parameter Workflow Transition state seach Transition state analysis = Meta. WF of basic, transition state search and freq WF High throughput HELIO PHYSICS Monthly Event counts Detection of fastest type II CME Retrieve data for HEC event data Associate flares with active regions Counts the number of events for a period by querying the HEC catalogue identifying the fastest type II CME in time finding events in Xray and radio retrieving solar monitor urls for votable events Retrieves all instrument data for an event defined in the HEC catalogue and instrument defined in ICS () associating flares with active regions Origin of solar wind events on Earth LIFE SICENCE free surfer double cross validation protein docking SNP annotation Finds the origins on the Sun of the Solar Wind detected on the Earth medical imaging pattern classification autodock vina with filtering DNA sequencing with Annovar SNP calling genome data re-sequencing sequence alignment aligning sequence fragments to a reference database indel calling sequence assembly of small genomes call insertions and deletions from sequence alignments reconstruction of genomes by building contig sequences simulation of low coverage sequence experiments exome sequencing dti processing random samples alignments from a sequence alignment file Retrieve images for an event WE WS-PGRADE WS-PGRADE g. LITE VO Astro + VOCE VO A&A VO Astro + VOCE VO Astro + Planck VO A&A VO WS-PGRADE g. Li. TE/Uni. Core Mo. SGrid VO WS-PGRADE g. Li. TE/Uni. Core Mo. SGrid VO WS-PGRADE WS-PGRADE WS-PGRADE g. Li. TE/Uni. Core g. Li. TE/Uni. Core Mo. SGrid VO Mo. SGrid VO Taverna 2 web service none Taverna 2 web service none Taverna 2 WS-PGRADE/ /MOTEUR WS-PGRADE/ /MOTEUR WS-PGRADE web service g. LITE g. LITE, PBS g. LITE, PBS none vlemed VO vlemed VO vlemed VO WS-PGRADE /MOTEUR g. LITE, PBS vlemed VO middleware 25
SHIWA Simulation Platform: Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) • joint portal and repository account given by the science gateway administrator DCI resources • certificate given by the relevant DCI administrator, for example the VO sysadmin Access to the SHIWA Portal shiwa-portal. cpc. wmin. ac. uk/liferay-portal-6. 05 Access to the SHIWA Repository shiwa-repo. cpc. wmin. ac. uk 26
Скачать презентацию Building an European Research Community through Interoperable Workflows
baad3c862fff8d22a4e313aef366b582.ppt