DIET a Cloud Middleware for Seed 4 C

Скачать презентацию DIET a Cloud Middleware for Seed 4 C Скачать презентацию DIET a Cloud Middleware for Seed 4 C

90509843d92f92bf12b45c9db6545c9e.ppt

  • Количество слайдов: 16

DIET a Cloud Middleware for Seed 4 C Eddy Caron Join work with Jonathan DIET a Cloud Middleware for Seed 4 C Eddy Caron Join work with Jonathan Rouzaud-Cornabas, Frédéric Desprez, Rajesh Palanichamy and the DIET Team Ecole Normale Supérieure de Lyon AVALON Research Team Sys. Fera

DIET’s Goals http: //graal. ens-lyon. fr/DIET/ Our goals To develop a toolbox for the DIET’s Goals http: //graal. ens-lyon. fr/DIET/ Our goals To develop a toolbox for the deployment of environments using the Application Service Provider (ASP) paradigm with different applications Use as much as possible public domain and standard software To obtain a high performance and scalable environment Implement and validate our more theoretical results Scheduling for heterogeneous platforms, data (re)distribution and replication, performance evaluation, algorithmic for heterogeneous and distributed platforms, … Based on CORBA and our own software developments FAST for performance evaluation, Log. Service for monitoring, Viz. DIET for the visualization, Go. DIET for the deployment Several applications in different fields (simulation, bioinformatics, …) Release 2. 8 available on the web since november ACI Grid ASP, RNTL GASP, ANR LEGO CIGC-05 -11, ANR Gwendia, Celtic-plus Project SEED 4 C 6/03/12 DIET a Cloud Middleware for Seed 4 C

RPC and Grid-Computing: Grid-RPC • One simple idea – Implementing the RPC programming model RPC and Grid-Computing: Grid-RPC • One simple idea – Implementing the RPC programming model over the grid – Using resources accessible through the network – Mixed parallelism model (data-parallel model at the server level and task parallelism between the servers) • Features needed – Load-balancing (resource localization and performance evaluation, scheduling), – IDL, – Data and replica management, – Security, – Fault-tolerance, – Interoperability with other systems, – … Design of a standard interface – within the OGF (Grid-RPC and SAGA WG) – Existing implementations: Net. Solve/Grid. Solve, Ninf, DIET, Omni. RPC 6/03/12 DIET a Cloud Middleware for Seed 4 C

RPC and Grid Computing: Grid-RPC Request Client S 2 ! B, C Op(C, A, RPC and Grid Computing: Grid-RPC Request Client S 2 ! B, C Op(C, A, B) er sw An A, ) (C S 1 6/03/12 AGENT(s) S 2 DIET a Cloud Middleware for Seed 4 C S 3 S 4

Client and server interface Client side So easy … Multi-interface (C, C++, Fortran, Java, Client and server interface Client side So easy … Multi-interface (C, C++, Fortran, Java, Python, Scilab, Web, etc. ) Grid-RPC compliant Server side Install and submit new server to agent (LA) Problem and parameter description Client IDL transfer from server Dynamic services 6/03/12 new service new version security update outdated service Etc. DIET a Cloud Middleware for Seed 4 C

Architecture overview 6/03/12 DIET a Cloud Middleware for Seed 4 C Architecture overview 6/03/12 DIET a Cloud Middleware for Seed 4 C

Workflow Management Workflow representation Direct Acyclic Graph (DAG) Each vertex is a task Each Workflow Management Workflow representation Direct Acyclic Graph (DAG) Each vertex is a task Each directed edge represents communication between tasks Goals Build and execute workflows Use different heuristics to solve scheduling problems Extensibility to address multi-workflows submission and large grid platform Manage heterogeneity and variability of environment ANR Gwendia Language definition (MOTEUR & MADAG) Idle time Data transfert Comparison on Grid’ 5000 vs EGI (Glite) Grid’ 5000 (DIET) 6/03/12 32. 857 s 0. 214 s Execution time 132. 143 s 274. 643 s 3. 371 s 540. 614 s DIET a Cloud Middleware for Seed 4 C

DIET Scheduling: Plug-in Schedulers Se. D level Performance estimation function Estimation Metric Vector - DIET Scheduling: Plug-in Schedulers Se. D level Performance estimation function Estimation Metric Vector - dynamic collection of performance estimation values Performance measures available through DIET FAST-NWS performance metrics Time elapsed since the last execution Co. RI (Collector of Resource Information) Developer defined values Aggregation Methods Defining mechanism to sort Se. D responses: associated with the service and defined at Se. D level Tunable comparison/aggregation routines for scheduling Priority Scheduler 6/03/12 Performs pairwise server estimation comparisons returning a sorted list of server responses; Can minimize or maximize based on Se. D estimations and taking into consideration the order in which the request for those performance estimations was specified at Se. D level. DIET a Cloud Middleware for Seed 4 C

Data Management DAGDA Joining task scheduling and data management Standardized through Grid. RPC OGF Data Management DAGDA Joining task scheduling and data management Standardized through Grid. RPC OGF WG. Data Arrangement for Grid and Distributed Applications Explicit data replication: Using the API. Implicit data replication. Data replacement algorithm: LRU, LFU AND FIFO Transfer optimization by selecting the more convenient source. Storage resources usage management. Data status backup/restoration. Join work with Gaël Le. Mahec (UPJV/MIS) 6/03/12 DIET a Cloud Middleware for Seed 4 C

Parallel and batch submissions Parallel & sequential jobs transparent for the user system dependent Parallel and batch submissions Parallel & sequential jobs transparent for the user system dependent submission Se. DBatch Many batch systems Batch schedulers behaviour Se. D// Internal scheduling process Monitoring & Performance prediction Simulation (Simbatch) NFS Se. D OAR Se. DBatch SLURM OGE 6/03/12 PBS LSF Loadleveler DIET a Cloud Middleware for Seed 4 C

DIET Cloud: Prototype Inside the Cloud DIET platform is virtualized inside the cloud. (as DIET Cloud: Prototype Inside the Cloud DIET platform is virtualized inside the cloud. (as Xen image for example) Very flexible and scalable as DIET nodes can be launched Scheduling is more complex DIET as a Cloud manager Eucalyptus interface Eucalyptus is treated as a new Batch System Provide a new implementation for the Batch. System abstract class 6/03/12 DIET a Cloud Middleware for Seed 4 C

DIET Cloud: New Architecture 6/03/12 DIET a Cloud Middleware for Seed 4 C DIET Cloud: New Architecture 6/03/12 DIET a Cloud Middleware for Seed 4 C

Grid’ 5000 Building a nation wide experimental platform for Grid & P 2 P Grid’ 5000 Building a nation wide experimental platform for Grid & P 2 P researches (like a particle accelerator for the computer scientists) 9 geographically distributed sites hosting clusters with 256 CPUs to 1 K CPUs) All sites are connected by RENATER (French Res. and Edu. Net. ) Design and develop a system/middleware environment for safely test and repeat experiments Use the platform for Grid experiments in real life conditions 4 main features: A high security for Grid’ 5000 and the Internet, despite the deep reconfiguration feature Single sign-on High-performance LRMS: OAR A user toolkit to reconfigure the nodes and monitor experiment: Kadeploy 6/03/12 DIET deployment over a maximum of processors 1 MA, 8 LA, 540 Se. Ds 1120 clients on 140 machines DGEMM requests (2000 x 2000 matrices) Simple round-robin scheduling DIET a Cloud Middleware for Seed 4 C

Conclusions Grid-RPC Interesting approach for several applications Simple, flexible, and efficient Many interesting research Conclusions Grid-RPC Interesting approach for several applications Simple, flexible, and efficient Many interesting research issues (scheduling, data management, resource discovery and reservation, deployment, fault-tolerance, …) DIET Scalable, open-source, and multi-application platform Concentration on several issues like resource discovery, scheduling (distributed scheduling and plugin schedulers), deployment (Go. DIET), performance evaluation (Co. RI), monitoring (Log. Service and Viz. DIET), data management and replication (DAGDA), Cloud support Large scale validation on the Grid’ 5000 platform A middleware designed and tunable for different applications http: //www. grid 5000. org/ 6/03/12 DIET a Cloud Middleware for Seed 4 C

Results A complete Middleware for heterogeneous infrastructure DIET is light to use Dedicated to Results A complete Middleware for heterogeneous infrastructure DIET is light to use Dedicated to many applications Designed for Grid and Cloud Efficient even in comparison to commercial tools DIET is high tunability middleware Used in production The DIET Team Sys. Fera Compagny (16 persons today) http: //www. sysfera. com 6/03/12 DIET a Cloud Middleware for Seed 4 C

Questions ? http: //graal. ens-lyon. fr/DIET Questions ? http: //graal. ens-lyon. fr/DIET




  • Мы удаляем страницу по первому запросу с достаточным набором данных, указывающих на ваше авторство. Мы также можем оставить страницу, явно указав ваше авторство (страницы полезны всем пользователям рунета и не несут цели нарушения авторских прав). Если такой вариант возможен, пожалуйста, укажите об этом.