Creating a Multi-Cloud System with the Cloud Broker

Creating a Multi-Cloud System with the Cloud. Broker Platform Dr. Wibke Sudholt Cloud. Broker Gmb. H Räffelstrasse 25, CH-8045 Zürich, Switzerland Phone: +41 44 515 21 70 Email: info@cloudbroker. com Web: http: //www. cloudbroker. com SCI-BUS is supported by the FP 7 Capacities Programme under contract no. RI-283481

Overview • Cloud computing • Cloud. Broker Platform • SCI-BUS project • Further examples

Cloud Computing SCI-BUS is supported by the FP 7 Capacities Programme under contract no. RI-283481

Cloud Definition • According to the US National Institute of Standards and Technology (NIST) • Essential characteristics – – – On-demand self service Broad network access Resource pooling Rapid elasticity Measured service © Cloud. Broker Gmb. H All rights reserved. • Service models – Software as a Service (Saa. S) – Platform as a Service (Paa. S) – Infrastructure as a Service (Iaa. S) • Deployment models – – Private cloud Community cloud Public cloud Hybrid cloud SCI-BUS Summer School 2013 02. 07. 2013 / 4

What is new? • Combination of previously existing technologies • Change in IT business model, Op. Ex instead of Cap. Ex • Outsourcing, automation, service orientation and economy of scale • Interfaces between providers and users set at the right place • Wholesale instead of handcrafted version of IT © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 5

Advantages of Cloud • • Immediate, on demand access Availability of otherwise unreachable resources Nearly unlimited scalability Dynamic load balancing, cloud bursting No long-term binding, flexibility User self service Pay-per-use with minimal initial investment Focus on core competencies © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 6

Challenges of Cloud • Hype, cloud washing • Complexity and dynamics • Performance limitations, dependency on network • Standardization and vendor lock-in issues • Security, privacy and legal concerns • New service and business model © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 7

Clouds for Scientific Computing • Lots of computing resources and specialized application software needed • Resources often either not existing at all or at capacity limit • Hardware and software expensive, time-consuming and complex to buy, set up, use and maintain • Mostly project-based or periodical usage patterns ⇒Ideal case for cloud computing ⇒Clouds make large-scale scientific computing resources also accessible for SMEs, small research groups, niche applications and short-term projects © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 8

Science Gateways and Clouds • According to the EGI-In. SPIRE Science Gateway Primer • Three principle approaches by which science gateways can interact with cloud computing – Utilize cloud computing resources for the services they offer – Run in the cloud themselves – Adopt the cloud business model © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 9

Scientific Computing Cloud Stack User Gateway Saa. S Paa. S Iaa. S HPC ©

Cloud. Broker Platform SCI-BUS is supported by the FP 7 Capacities Programme under contract no. RI-283481

Cloud. Broker Platform • Web application store for the deployment and execution of compute-intensive scientific and technical software on different clouds • Uses Iaa. S from cloud providers • Offers Paa. S for software vendors and Saa. S for end users • On demand, pay-per-use, browser, programmatic and command-line access, cross-domain • Easy to use, no need for own scientific computing infrastructure, speeds up time to results, Op. Ex instead of Cap. Ex © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 12

Platform Cloud Stack User Gateway Saa. S Paa. S Cloud. Broker Platform Iaa. S

Platform Architecture End Users, Software Vendors, Resource Providers CLI Chemistry Applications Java Client Library REST Web Service API Biology Applications Pharma Applications Engineering Applications … Applications Cloud. Broker Platform Amazon Cloud © Cloud. Broker Gmb. H All rights reserved. IBM Cloud Open. Stack Cloud Open. Nebula Cloud Eucalyptus Cloud. Broker Integration Web Browser UI User Tools … Cloud SCI-BUS Summer School 2013 02. 07. 2013 / 15

Platform Functionality Users Web Browser UI Process Manager Tools Web Service API Application Manager Process Monitor User Manager Queuing System Accounting Module Resource Manager Billing Module Storage Manager Payment Module Image Manager Scalability and Fault Tolerance Handler Cloud Provider Access Manager Amazon Adapter IBM Adapter Open. Stack Adapter Open. Nebula Adapter Eucalyptus Adapter … Adapter Security Frame: Transport Layer Security, Access Rights Security Amazon Cloud © Cloud. Broker Gmb. H All rights reserved. IBM Cloud Open. Stack Cloud Open. Nebula Cloud Eucalyptus Cloud … Cloud SCI-BUS Summer School 2013 02. 07. 2013 / 16

Available Cloud Adapters • Virtual machines – – – Amazon EC 2 IBM Smart. Cloud Enterprise Open. Stack EC 2 and Nova Open. Nebula Eucalyptus • Object storage – – Amazon S 3 IBM Nirvanix Open. Stack S 3 Ceph RADOS ⇒ More cloud adapters continuously to be added ⇒ Users can use and pay for public resources registered by Cloud. Broker and others ⇒ Users can register and charge for their own resources, just cloud compute and storage accounts needed © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 17

Platform Security Customer Client Browser or Application Corporate IT Corporate Security Policies and Standards SSL Secured Connection Authentication Cloud. Broker CBP Platform . Industry Standard Application Security Technology Industry Standard Server Security Technology Industry Standard Secure Data Center SSL Secured Connection SSL secured connection Authentication to Cloud Authentication to VM Cloud Provider © Cloud. Broker Gmb. H All rights reserved. Cloud CBP Instances Dedicated, Secured. and Restricted Virtual Machines Security Certified Compute and Storage Cloud Technology Security Certified Data Center SCI-BUS Summer School 2013 02. 07. 2013 / 18

Interaction of Tiers Object Storage Worker Node 1 Client Platform Master Node Worker Node

User, Resource and Software Management Cloud. Broker Platform Organization 1: Resources, Software, Invoices User 1. 2: Jobs, Data Files, Instances Organization 2: Resources, Software, Invoices User 2. 1: Jobs, Data Files, Instances User 2. 2: Jobs, Data Files, Instances • User roles: admin, advanced, standard • Resource and software visibility: private, protected, public © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 20

Application Requirements Item Requirements / Possibilities Type Scientific and technical applications Domain All License Open source, free, commercial Character Compute-intensive, batch-oriented Interface Command line, non-interactive Runtime Hours, days, weeks Operating system Linux Parallelization Single-threaded / single-core, multi-threaded / multi-core, embarrassingly parallel / high throughput, parallel across nodes (via MPI or other methods) Workflows Single jobs, pipeline jobs, parameter sweeps (via API) © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 21

Ported Applications • Chemistry – Gromacs, Siesta • Biology – BLAST, X! Tandem, Rosetta, TCoffee, Fold. X, Clustal. W, Clustal Omega • Pharma – Auto. Dock • Engineering – Open. FOAM, ASCOMP Trans. AT, Elmer • Rendering – Blender • General – Shell scripts, R, Akka, PHP, wrapper script ⇒ More application software continuously to be added ⇒ Users can use and pay for public software deployed by Cloud. Broker and others ⇒ Users can deploy and charge for their own software, just installation script and software package needed © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 22

Clients and APIs CLI • Command line interface • Job submission Java • Application programming interface • Java client library • Representational state transfer REST • Web service interface © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 23

Public Platform • Public Cloud. Broker Platform version under https: //platform. cloudbroker. com • Free organization and user account after registration and validation • Standard pricing = (resource charges + software charges) × (1 + Cloud. Broker surcharge percentage) • Freemium model: Cloud. Broker surcharges only for commercial and pay-per-use cloud resources and application software • Production version: Monitored, maintained and regularly updated • Helpdesk under platform@cloudbroker. com • Also available as hosted and licensed versions © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 24

© Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07.

SCI-BUS Project SCI-BUS is supported by the FP 7 Capacities Programme under contract no. RI-283481

SCI-BUS Project Overview • Project – SCI-BUS = SCIentific gateway Based User Support – Supported by the EU FP 7 Capacities Programme under contract no. RI-283481 – Running 10/2011 -09/2014 – http: //www. sci-bus. eu • Partners – 15 consortium members from 10 different countries (10 academic, 5 commercial) – 6 subcontractors – Growing number of associated partners (signing of Mo. U) © Cloud. Broker Gmb. H All rights reserved. • Goals – Build science gateways for different communities and domains – Provide researchers seamless access to major computing, data and networking infrastructures and services – Focus on scientific workflows – Develop corresponding gateway technology SCI-BUS Summer School 2013 02. 07. 2013 / 27

Cloud. Broker’s Role in SCI-BUS • Provide connection to public and private, commercial and academic, proprietary and open cloud infrastructures via the Cloud. Broker Platform • Support the integration of the Cloud. Broker Platform as new distributed computing infrastructure (DCI) in WS-PGRADE/g. USE • Enable science gateways to charge for their services via the Cloud. Broker Platform • Lead the project’s efforts towards sustainability and development of business models © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 28

SCI-BUS Cloud Stack User Gateway SCI-BUS Saa. S Paa. S Cloud. Broker Platform Iaa.

SCI-BUS Cloud Stack User Gateway WS-PGRADE/ g. USE Saa. S Paa. S Cloud. Broker

SCI-BUS Architecture © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013

Current Clouds in SCI-BUS • Public commercial clouds – Amazon Web Services – IBM Smart. Cloud Enterprise • Private academic clouds – MTA SZTAKI, Hungary: Open. Nebula and Ceph RADOS – BIFI, University of Zaragoza, Spain: Open. Stack – ETH and University of Zurich, Switzerland: Open. Stack – University of Westminster, UK: Open. Stack © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 32

WS-PGRADE/g. USE Source: http: //www. guse. hu/about/architecture © Cloud. Broker Gmb. H All rights

Cloud. Broker Platform Integration in WS-PGRADE/g. USE • Methodology – Collaboration between MTA SZTAKI, Cloud. Broker and Scale. Tools – Use of the Cloud. Broker Platform Java client library – Coordination of updates • Implemented features – Cloud. Broker DCI Bridge configuration portlet – Cloud. Broker security portlet – Concrete workflow portlet extension for Cloud. Broker jobs – Support for running own applications via a wrapper script – Display of costs • Upcoming features – Display of prices and invoices • Future features – Different types of user mapping – Data Bridge integration Source: http: //www. guse. hu/node/49 © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 34

Current SCI-BUS Gateways • • • Adria Science Portal AEGIS CMPC Portal ag. INFRA Science Gateway Auto. Dock Portal BIFI-Unizar Portal Citizen Web Community Gateway AMC e-Bio. Infra Gateway Etics Portal iportal Mo. SGrid Portal NVG Science Gateway Portal © Cloud. Broker Gmb. H All rights reserved. • • • Public g. USE Portal Render. Farm. fi SHIWA Simulation Platform Sim. Bus. Pro Portal Statistical Seismology Science Gateway TR-Grid g. USE Gateway Vis. IVO Gateway HELIOGate Portal Westminster DG Portal WRF 4 SG SCI-BUS Summer School 2013 02. 07. 2013 / 35

Further Examples SCI-BUS is supported by the FP 7 Capacities Programme under contract no. RI-283481

Protein Modelling • Setup – Collaboration between IBM, Cloud. Broker and the Institute of Molecular Systems Biology (IMSB) at ETH Zurich – Goal is to better understand the mechanisms of infectious diseases – For the example of the Streptococcus bacterium, predict the 3 D structures of important proteins from their 1 D sequences by means of the Rosetta software – Use the Cloud. Broker Platform to manage the calculations on top of IBM Smart. Cloud Enterprise compute and storage resources © Cloud. Broker Gmb. H All rights reserved. Source: Dr. Lars Malmström, IMSB, ETH Zurich • Results – 249 target proteins modeled – More than 1’ 000 CPUs used in parallel – About 36’ 000 jobs automatically managed – Almost 250‘ 000 CPU hours consumed – 2. 3 Mio 3 D protein structure models created – Less than two weeks runtime SCI-BUS Summer School 2013 02. 07. 2013 / 37

Interfacing in KNIME • KNIME = Konstanz Information Miner • Open-source data integration, processing, analysis and exploration framework supported by KNIME. com AG • Experimental Cloud. Broker Platform KNIME node • Computational chemistry workflow showcase: Calculate the dipole moments for a number of molecules © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 38

Web UI for Trans. AT Source: Anita Hollenstein and Patrice Mu ller, http: //eprints. hsr. ch/193/, p. 59 • • • Trans. AT = Transport phenomena Analysis Tool Computational multiphase fluid dynamics (CMFD) software from ASCOMP Gmb. H Collaboration with Hochschule für Technik Rapperswil (HSR), Switzerland Bachelor thesis “Web UI-Front-End for Fluid Dynamics Cloud” Microsoft Silverlight-based Trans. AT web UI © Cloud. Broker Gmb. H All rights reserved. SCI-BUS Summer School 2013 02. 07. 2013 / 39

Uber. Cloud HPC Experiment Heat Transfer Use Case • Collaboration with Biscarri Consultoria SL, Spain, CSC IT Center for Science Ltd. , Finland, and CAELinux, Switzerland • Team 30 of Uber. Cloud HPC Experiment • Application of Elmer finite element software for multiphysical problems on a heat transfer use case • Using Amazon Cluster Compute instances via the Cloud. Broker Platform • Results show that scalability is determined by network backbone performance © Cloud. Broker Gmb. H All rights reserved. Source: Biscarri Consultoria SL, see also http: //tci. taborcommunications. com/Uber. Cloud_HPC_Experiment SCI-BUS Summer School 2013 02. 07. 2013 / 40

Cloud. SME Project • EU FP 7 project • Led by University of Westminster, involving 12 SMEs • Part of the “ICT for Manufacturing SMEs” (I 4 MS) initiative in the Public Private Partnership “Factories of the Future” (PPP Fo. F) • Starting from July 2013 onwards • Cloud-based one-stop-shop for simulations in manufacturing and engineering • Saa. S and Paa. S solutions © Cloud. Broker Gmb. H All rights reserved. Source: http: //cordis. europa. eu/fp 7/ict/computing/home-i 4 ms_en. html SCI-BUS Summer School 2013 02. 07. 2013 / 41

Thank you! – Questions? For more information Email: info@cloudbroker. com Web: http: //www. cloudbroker. com Public platform: https: //platform. cloudbroker. com Helpdesk: platform@cloudbroker. com SCI-BUS is supported by the FP 7 Capacities Programme under contract no. RI-283481