SCI-BUS use case http www sci-bus eu Peter

SCI-BUS use case http: //www. sci-bus. eu Peter Kacsuk, Sandor Acs, Mark Gergely MTA SZTAKI Start date: 2011 -10 -01 Duration: 36 months SCI-BUS is supported by the FP 7 Capacities Programme under contract nr RI-283481 1

Main objectives of SCI-BUS • Create a generic-purpose gateway technology • Elaborate an application-specific gateway building technology and customisation methodology • Establish production gateway services both for NGIs (horizontal user communities) and various science communities (vertical user communities) • Provide seamless access to major computing, data and networking DCIs and services including supercomputers, clusters, grids and clouds • Provide gateway development and application development support • Develop business models to guarantee the sustainability of the gateway services and to enable the commercial exploitation 2

Project partners No Participant organisation name Part. short name Country Expertise & Experience 1 Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezet MTA SZTAKI Hungary Coordinator in EDGe. S, EDGI, SHIWA, developer of WS-PGRADE 2 Academisch Medisch Centrum bij de Universiteit van Amsterdam AMC The Netherlands Gateway developer for the medical user community 3 Simsoft Bilgisayar Teknolojileri Ltd. Sti. SIMSOFT Turkey Gateway developer for business process modeling 4 E-GROUP Ict Software Informatikai Zrt EG Hungary Gateway developer for Web 2. 0 5 Eidgenössische Technische Hochschule Zürich ETH Zurich Switzerland Gateway developer for bioscience 6 Middle East Technical University METU Turkey Gateway developer for seizmology 7 Scaletools Ltd. ST Switzerland Gateway developer for clouds 8 Eberhard Karls Universität Tübingen EKUT Germany Gateway developer for computational chemistry 9 University of Westminster Uo. W UK Developer of UK NGS P-GRADE 10 Universidad de Zaragoza Unizar Spain Gateway developer for Pire. Grid SMEs 11 Cloud. Broker Gm. Bh CB Switzerland Gateway developer for clouds 12 4 D SOFT SZAMITASTECHNIKAI KFT 4 D SOFT Hungary Gateway developer for ETICS-2 13 Istituto Nazionale di Astrofisica INAF Italy Gateway developer for astrophysics 14 Laurea-ammattikorkeakoulu oy Laurea Finnland Gateway developer for Blender 15 The Provost Fellows & Scholars of the College of the Holy and Undivided Trinity of Queen Elizabeth near Dublin TCD Ireland Gateway developer for helio 3 physics

Re-usability and adaptability to different communities and needs • Create 11 gateways in the 1 st project year for the following user communities: – – – International seismology community Helio-physics communityc Swiss systems biology community of the Systems. X. ch project German Mo. SGrid computational chemistry and bioinformatics community Biomedical researchers community of the Academic Medical Centre of the University of Amsterdam Astrophysics community Pire. Grid SMEs community Business process modelling community Blender rendering community Citizen web-2 community Public application developer community (based on ETICS-2 results) • Create another 16 user community gateways (including 6 subcontractors) 4

The SCI-BUS Infrastructure Create 11 gateways in the 1 st project year 5

SCI-BUS and EGI Cloud Federation (ECF) What SCI-BUS will provide for ECF users? • SCI-BUS gateway to access ECF Iaa. S cloud resources ondemand for parameter sweep (high-throughput computation) applications (like many simulation applications) • Access the ECF Iaa. S cloud resources for Saa. S services using the Cloud. Broker platform • Images of the SCI-BUS gateways to enable their easy deployment in the ECF (and other) clouds 6

SCI-BUS and EGI Cloud Federation (ECF) What SCI-BUS requires from ECF? • DCI images based on which DCIs can be easily and quickly deployed in the Clouds • Cloud resources where these images can be deployed Use scenario • To quickly test new gateway versions with various DCIs (use case for ECF) • SCI-BUS partners are not DCI experts so they would need the already existing DCI service images based on which they could deploy easily and quickly a certain DCI in the cloud for the time of testing if the gateway runs correctly with this DCI Why? • DCI services are frequently updated or changed • It is difficult to find existing DCIs where gateway developers 7 can test the gateway with different versions of DCIs

Use case objectives SCI-BUS uses the ETICS-2 systematic build and test technology for quality assurance of the gateways. The objectives of using the clouds of ECF are: • To quickly test new generic purpose gateway versions (WSPGRADE/g. USE) with various DCIs by SZTAKI developers – Required DCI images: ARC, g. Lite, UNICORE, GT 2, GT 4, GT 5, PBS, LSF, BOINC • To quickly test new science gateways customized from WSPGRADE/g. USE with various DCIs by all 20 project partners and subcontractors – Required DCI images: ARC, g. Lite, UNICORE, GT 2, GT 4, GT 5, PBS, LSF, BOINC 8

SZTAKI Cloud experiences • Production laboratory-level Cloud – Opennebula – Size: 112 cores, 3 Tbyte disk – Used by the 25 lab members in daily practice for example developing SCI-BUS gateways • Based on the positive experiences of the laboratory-level Cloud, SZTAKI decided to set up an institute-level cloud to be used as production cloud by all 300 staff members in daily practice – Opennebula – Size: 462 cores, 72 Tbyte disk 9

SZTAKI CLOUD Architecture Node 64 Core 256 GB RAM Storage 36 TB Frontend 8 Core, 16 GB RAM Switch 48 port, 4 X 10 G Storage 36 TB Node 64 Core 256 GB RAM Node 64 Core 256 GB RAM

Use case architecture ECF Cloud 1 SZTAKI Cloud (Opennebula) Test access Glite DCI SCI-BUS gateway Test access ECF Cloud 2 ARC DCI 11

First experiment Successful in-house experiment in SZTAKI Cloud (Opennebula) Test access SZTAKI Cloud (Opennebula) EDGI (g. Lite+BOINC) DCI SCI-BUS gateway Component images of EDGI DCI: • 4 g. Lite images: UI, VOMS, WMS, CREAM CE • 1 BOINC image: BOINC server and client All these images have been developed by EDGI and accessible at: http: //www. edgi-grid. eu/downloads/vmimages/v 2. 1/ 12

Second experiment SZTAKI Cloud (Opennebula) SCI-BUS gateway Test access CESNET Cloud (Opennebula) EDGI (g. Lite+BOINC) DCI Many thanks to CESNET and particularly Boris Parak Component images of EDGI DCI: • 4 g. Lite images: UI, VOMS, WMS, CREAM CE • 1 BOINC image: BOINC server and client All these images have been developed by EDGI and accessible at: http: //www. edgi-grid. eu/downloads/vmimages/v 2. 1/ 13

Problems we have found • Access to the sites – Access to the sites first should be asked from the cloud admins (some of them helpful, some of them not) – Slow and unpredictable procedure – No unified access mechanism: • Some service should be accessed by certificate (e. g. CESNET OCCI interface X 509 certificate is needed, Sunstone access with login user/pass) 14

Problems we have found • Use of the sites – The different clouds use different hipervisors but this crucial information is missing in the site descriptions – Network access is different site by site (e. g. CESNET gives public network access but others might not), this information is also missing in the site descriptions – Some short documentation for experts how to deploy the virtual appliances would be useful: • What kind of contextualization is used • What kind of hipervisor to be used 15

Problems we have found • Access to DCI images – We would need a cloud marketplace from where the DCI bridge images could be accessed – First we thought Stratus. Lab’s marketplace to use. However: • There is no complete g. Lite set: APEL, SE, CREAM CE, WE are available but for eample, there is no WMS and UI (even that is available is the old g. Lite 3. 2) • ARC and UNICORE are completely missing • Images are containing lot of unnecessary packages • Stratus. Lab contextualization is different from Opennebula and not so widespread, just 14% in the testbeds of ECF 16

Suggested solution • ECF (or someone else) should provide a marketplace with useful DCI images • Who create the images? Three options: 1. The middleware projects like EMI and EDGI since they are the experts of the middleware services. (These projects soon will be finished) 2. A specialized project like Stratus. Lab 3. A community effort where members of the various communities create the images (no guarantee for the quality of the images) • We recommend Option 2. EGI could organize a new project with this purpose. SZTAKI is happy to contribute based on the current experience of creating and using images. 17

Sustainability of the marketplace • It is not enough to create the marketplace, it should be maintained in a sustainable way. • Who would maintain the images? The three main options are the same as before for creation. • We recommend Option 3. This could be a community effort with some additional services organized by EGI: – Basically we expect that the EGI/NGI community maintains the images (since it is their interest) – However, if an image is not updated after a certain time EGI should find a volunteer (or paid) community member to do the job (maybe some financial resources of EGI could be allocated for this purpose) 18