High Performance Cluster Computing Architectures and Systems Book

High Performance Cluster Computing: Architectures and Systems Book Editor: Rajkumar Buyya Slides: Hai Jin and Raj Buyya Internet and Cluster Computing Center

Cluster Computing at a Glance Chapter 1: by M. Baker and R. Buyya n n n n Introduction Scalable Parallel Computer Architecture Towards Low Cost Parallel Computing Windows of Opportunity A Cluster Computer and its Architecture Clusters Classifications Commodity Components for Clusters Network Service/Communications SW Middleware and Single System Image Resource Management and Scheduling Programming Environments and Tools Cluster Applications Representative Cluster Systems Cluster of SMPs (CLUMPS) Summary and Conclusions http: //www. buyya. com/cluster/

Resource Hungry Applications n Solving grand challenge applications using computer modeling, simulation and analysis Aerospace Life Sciences CAD/CAM Digital Biology Internet & Ecommerce Military Applications

Application Categories

How to Run Applications Faster ? n There are 3 ways to improve performance: Work Harder n Work Smarter n Get Help n n Computer Analogy Using faster hardware n Optimized algorithms and techniques used to solve computational tasks n Multiple computers to solve a particular n

Scalable (Parallel) Computer Architectures n Taxonomy n n n based on how processors, memory & interconnect are laid out, resources are managed Massively Parallel Processors (MPP) Symmetric Multiprocessors (SMP) Cache-Coherent Non-Uniform Memory Access (CC-NUMA) Clusters Distributed Systems – Grids/P 2 P

Scalable Parallel Computer Architectures n MPP n n A large parallel processing system with a sharednothing architecture Consist of several hundred nodes with a high-speed interconnection network/switch Each node consists of a main memory & one or more processors n Runs a separate copy of the OS SMP n n 2 -64 processors today Shared-everything architecture All processors share all the global resources available Single copy of the OS runs on these systems

Scalable Parallel Computer Architectures n CC-NUMA n n n Clusters n n a scalable multiprocessor system having a cache-coherent nonuniform memory access architecture every processor has a global view of all of the memory a collection of workstations / PCs that are interconnected by a high-speed network as an integrated collection of resources have a single system image spanning all its nodes Distributed systems n n n considered conventional networks of independent computers have multiple system images as each node runs its own OS the individual machines could be combinations of MPPs, SMPs, clusters, & individual computers

Rise and Fall of Computer Architectures n Vector Computers (VC) - proprietary system: n n provided the breakthrough needed for the emergence of computational science, buy they were only a partial answer. Massively Parallel Processors (MPP) -proprietary systems: n n suffers from scalability n n high cost and a low performance/price ratio. difficult to use and hard to extract parallel performance. Symmetric Multiprocessors (SMP): Distributed Systems: Clusters - gaining popularity: n n High Performance Computing - Commodity Supercomputing High Availability Computing - Mission Critical Applications

Top 500 Computers Architecture (Clusters share is growing)

The Dead Supercomputer Society http: //www. paralogos. com/Dead. Super/ n n n n ACRI Alliant American Supercomputer Ametek Applied Dynamics Astronautics BBN CDC Convex Cray Computer Cray Research (SGI? Tera) Culler-Harris Culler Scientific Cydrome n n n n Dana/Ardent/Stellar Elxsi ETA Systems Evans & Sutherland Computer Division Floating Point Systems Galaxy YH-1 Convex C 4600 Goodyear Aerospace MPP Gould NPL Guiltech Intel Scientific Computers Intl. Parallel Machines KSR Mas. Par n n n n Meiko Myrias Thinking Machines Saxpy Scientific Computer Systems (SCS) Soviet Supercomputers Suprenum

Vendors: Specialised ones (e. g. , TMC) disappeared, new emerged

Computer Food Chain: Causing the demise of specialized systems • Demise of mainframes, supercomputers, & MPPs

Towards Clusters The promise of supercomputing to the average PC User ?

Technology Trends. . . n Performance of PC/Workstations components has almost reached performance of those used in supercomputers… n n n Microprocessors (50% to 100% per year) Networks (Gigabit SANs) Operating Systems (Linux, . . . ) Programming environments (MPI, …) Applications (. edu, . com, . org, . net, . shop, . bank) The rate of performance improvements of commodity systems is much rapid compared to specialized systems

Towards Commodity Cluster Computing n n Since the early 1990 s, there is an increasing trend to move away from expensive and specialized proprietary parallel supercomputers towards clusters of computers (PCs, workstations) From specialized traditional supercomputing platforms to cheaper, general purpose systems consisting of loosely coupled components built up from single or multiprocessor PCs or workstations n Linking together two or more computers to jointly solve computational problems

History: Clustering of Computers for Collective Computing PDA Clusters 1960 1980 s 1990 1995+ 2000+

What is Cluster ? n n n A cluster is a type of parallel and distributed processing system, which consists of a collection of interconnected standalone computers cooperatively working together as a single, integrated computing resource. A node n a single or multiprocessor system with memory, I/O facilities, & OS A cluster: generally 2 or more computers (nodes) connected together in a single cabinet, or physically separated & connected via a LAN n appears as a single system to users and applications n provides a cost-effective way to gain features and benefits n

Cluster Architecture Parallel Applications Sequential Applications Parallel Programming Environment Cluster Middleware (Single System Image and Availability Infrastructure) PC/Workstation Communications Software Network Interface Hardware Cluster Interconnection Network/Switch

So What’s So Different about Clusters? n n n Commodity Parts? Communications Packaging? Incremental Scalability? Independent Failure? Intelligent Network Interfaces? Complete System on every node n n n virtual memory scheduler files … Nodes can be used individually or jointly. . .

Windows of Opportunities n Parallel Processing n n Network RAM n n Use memory associated with each workstation as aggregate DRAM cache Software RAID n n Use multiple processors to build MPP/DSM-like systems for parallel computing Redundant Array of Inexpensive/Independent Disks Use the arrays of workstation disks to provide cheap, highly available and scalable file storage Possible to provide parallel I/O support to applications Multipath Communication n Use multiple networks for parallel data transfer between nodes

Cluster Design Issues • Enhanced Performance (performance @ low cost) • Enhanced Availability (failure management) • Single System Image (look-and-feel of one system) • Size Scalability (physical & application) • Fast Communication (networks & protocols) • Load Balancing (CPU, Net, Memory, Disk) • Security and Encryption (clusters of clusters) • Distributed Environment (Social issues) • Manageability (admin. and control) • Programmability (simple API if required) • Applicability (cluster-aware and non-aware app. )

Scalability Vs. Single System Image UP

Common Cluster Modes n n High Performance (dedicated). High Throughput (idle cycle harvesting). High Availability (fail-over). A Unified System – HP and HA within the same cluster

High Performance Cluster (dedicated mode)

High Throughput Cluster (Idle Resource Harvesting) Shared Pool of Computing Resources: Processors, Memory, Disks Interconnect Guarantee at least one workstation to many individuals (when active) Deliver large % of collective resources to few individuals at any one time

High Availability Clusters

HA and HP in the same Cluster Best of both Worlds: world is heading towards • this configuration)

Cluster Components

Prominent Components of Cluster Computers (I) n Multiple High Performance Computers PCs n Workstations n SMPs (CLUMPS) n Distributed HPC Systems leading to Grid Computing n

System CPUs n Processors n Intel x 86 -class Processors n n n Digital Alpha – phased out when HP acquired it. n n n Alpha 21364 processor integrates processing, memory controller, network interface into a single chip IBM Power. PC Sun SPARC n n Pentium Pro and Pentium Xeon AMD x 86, Cyrix x 86, etc. Scalable Processor Architecture SGI MIPS n Microprocessor without Interlocked Pipeline Stages

System Disk n Disk and I/O n n Overall improvement in disk access time has been less than 10% per year Amdahl’s law n n Speed-up obtained from faster processors is limited by the slowest system component Parallel I/O n Carry out I/O operations in parallel, supported by parallel file system based on hardware or software RAID

Commodity Components for Clusters (II): Operating Systems n 2 fundamental services for users n make the computer hardware easier to use n n share hardware resources among users n n Processor - multitasking The new concept in OS services n support multiple threads of control in a process itself n n create a virtual machine that differs markedly from the real machine parallelism within a process multithreading POSIX thread interface is a standard programming environment Trend n n Modularity – MS Windows, IBM OS/2 Microkernel – provides only essential OS services n high level abstraction of OS portability

Prominent Components of Cluster Computers n State of the art Operating Systems n n n n Linux (MOSIX, Beowulf, and many more) Windows HPC (HPC 2 N – Umea University) SUN Solaris (Berkeley NOW, C-DAC PARAM) IBM AIX (IBM SP 2) HP UX (Illinois - PANDA) Mach (Microkernel based OS) (CMU) Cluster Operating Systems (Solaris MC, SCO Unixware, MOSIX (academic project) OS gluing layers (Berkeley Glunix)

Operating Systems used in Top 500 Powerful computers AIX

Prominent Components of Cluster Computers (III) n High Performance Networks/Switches n Ethernet (10 Mbps), n Fast Ethernet (100 Mbps), n Gigabit Ethernet (1 Gbps) n SCI (Scalable Coherent Interface- MPI- 12µsec latency) n ATM (Asynchronous Transfer Mode) n Myrinet (1. 28 Gbps) n Qs. Net (Quadrics Supercomputing World, 5µsec latency for MPI messages) n Digital Memory Channel n FDDI (fiber distributed data interface) n Infini. Band

Prominent Components of Cluster Computers (IV) n Fast Communication Protocols and Services (User Level Communication): Active Messages (Berkeley) n Fast Messages (Illinois) n U-net (Cornell) n XTP (Virginia) n Virtual Interface Architecture (VIA) n

Prominent Components of Cluster Computers (V) n Cluster Middleware n n n Hardware n n DEC Memory Channel, DSM (Alewife, DASH), SMP Techniques Operating System Kernel/Gluing Layers n n Single System Image (SSI) System Availability (SA) Infrastructure Solaris MC, Unixware, GLUnix, MOSIX Applications and Subsystems n n n Applications (system management and electronic forms) Runtime systems (software DSM, PFS etc. ) Resource management and scheduling (RMS) software n Oracle Grid Engine, Platform LSF (Load Sharing Facility), PBS (Portable Batch Scheduler), Microsoft Cluster Compute Server (CCS)

Advanced Network Services/ Communication SW n Communication infrastructure support protocol for n n Communication service provides cluster with important Qo. S parameters n n n Bulk-data transport Streaming data Group communications Latency Bandwidth Reliability Fault-tolerance Network services are designed as a hierarchical stack of protocols with relatively low-level communication API, providing means to implement wide range of communication methodologies n n n RPC DSM Stream-based and message passing interface (e. g. , MPI, PVM)

Prominent Components of Cluster Computers (VI) n Parallel Programming Environments and Tools n Threads (PCs, SMPs, NOW. . ) n POSIX Threads Java Threads n Linux, Windows, on many Supercomputers n n n MPI (Message Passing Interface) Parametric Programming Software DSMs (Shmem) Compilers n C/C++/Java Parallel programming with C++ (MIT Press book) n GUI based tools for PP modeling n n n RAD (rapid application development) tools Debuggers Performance Analysis Tools Visualization Tools

Prominent Components of Cluster Computers (VII) n Applications Sequential n Parallel / Distributed (Cluster-aware app. ) n n Grand Challenging applications n n n Weather Forecasting Quantum Chemistry Molecular Biology Modeling Engineering Analysis (CAD/CAM) ………………. PDBs, web servers, data-mining

Key Operational Benefits of Clustering High Performance n Expandability and Scalability n High Throughput n High Availability n

Clusters Classification (I) n Application Target n High Performance (HP) Clusters n n Grand Challenging Applications High Availability (HA) Clusters n Mission Critical applications

Clusters Classification (II) n Node Ownership Dedicated Clusters n Non-dedicated clusters n Adaptive parallel computing n Communal multiprocessing n

Clusters Classification (III) n Node Hardware n Clusters of PCs (Co. Ps) n Piles of PCs (Po. Ps) Clusters of Workstations (COWs) n Clusters of SMPs (CLUMPs) n

Clusters Classification (IV) n Node Operating System Linux Clusters (e. g. , Beowulf) n Solaris Clusters (e. g. , Berkeley NOW) n AIX Clusters (e. g. , IBM SP 2) n SCO/Compaq Clusters (Unixware) n Digital VMS Clusters n HP-UX clusters n Windows HPC clusters n

Clusters Classification (V) n Node Configuration n Homogeneous Clusters n n All nodes will have similar architectures and run the same OSs Heterogeneous Clusters n Nodes will have different architectures and run different OSs

Clusters Classification (VI) n Levels of Clustering n Group Clusters (#nodes: 2 -99) n n n Nodes are connected by SAN like Myrinet Departmental Clusters (#nodes: 10 s to 100 s) Organizational Clusters (#nodes: many 100 s) National Metacomputers (WAN/Internetbased) International Metacomputers (Internetbased, #nodes: 1000 s to many millions) n n Grid Computing Web-based Computing

Single System Image [See SSI Slides of Next Lecture]

Cluster Programming

Levels of Parallelism PVM/MPI Threads Compilers CPU Task i-l func 1 ( ) {. . . . } a ( 0 ) =. . b ( 0 ) =. . + Task i func 2 ( ) {. . . . } a ( 1 )=. . b ( 1 )=. . x Task i+1 func 3 ( ) {. . . . } a ( 2 )=. . b ( 2 )=. . Load Code-Granularity Code Item Large grain (task level) Program Medium grain (control level) Function (thread) Fine grain (data level) Loop (Compiler) Very fine grain (multiple issue) With hardware

Cluster Programming Environments n Shared Memory Based n n n Message Passing Based n n PVM (Parallel Virtual Machine) MPI (Message Passing Interface) Parametric Computations n n DSM (Distributed Shared Memory) Threads/Open. MP (enabled for clusters) Java threads (IBM c. JVM) Aneka Threads Nimrod-G, Gridbus, also in Aneka Automatic Parallelising Compilers Parallel Libraries & Computational Kernels (e. g. , Net. Solve)

Programming Environments and Tools (I) n Threads (PCs, SMPs, NOW. . ) n n n In multiprocessor systems n Used to simultaneously utilize all the available processors In uniprocessor systems n Used to utilize the system resources effectively Multithreaded applications offer quicker response to user input and run faster Potentially portable, as there exists an IEEE standard for POSIX threads interface (pthreads) Extensively used in developing both application and system software

Programming Environments and Tools (II) n Message Passing Systems (MPI and PVM) n n n Allow efficient parallel programs to be written for distributed memory systems 2 most popular high-level message-passing systems – PVM & MPI PVM n n both an environment & a message-passing library MPI n n n a message passing specification, designed to be standard for distributed memory parallel computing using explicit message passing attempt to establish a practical, portable, efficient, & flexible standard for message passing generally, application developers prefer MPI, as it became the de facto standard for message passing

Programming Environments and Tools (III) n Distributed Shared Memory (DSM) Systems n Message-passing n n n Shared memory systems n n n the most efficient, widely used, programming paradigm on distributed memory system complex & difficult to program offer a simple and general programming model but suffer from scalability DSM on distributed memory system n n alternative cost-effective solution Software DSM n n Usually built as a separate layer on top of the comm interface Take full advantage of the application characteristics: virtual pages, objects, & language types are units of sharing Tread. Marks, Linda Hardware DSM n n Better performance, no burden on user & SW layers, fine granularity of sharing, extensions of the cache coherence scheme, & increased HW complexity DASH, Merlin

Programming Environments and Tools (IV) n Parallel Debuggers and Profilers n Debuggers n n Very limited HPDF (High Performance Debugging Forum) as Parallel Tools Consortium project in 1996 n n Developed a HPD version specification, which defines the functionality, semantics, and syntax for a commercial-line parallel debugger Total. View n n n A commercial product from Dolphin Interconnect Solutions The only widely available GUI-based parallel debugger that supports multiple HPC platforms Only used in homogeneous environments, where each process of the parallel application being debugged must be running under the same version of the OS

Functionality of Parallel Debugger n n n n n Managing multiple processes and multiple threads within a process Displaying each process in its own window Displaying source code, stack trace, and stack frame for one or more processes Diving into objects, subroutines, and functions Setting both source-level and machine-level breakpoints Sharing breakpoints between groups of processes Defining watch and evaluation points Displaying arrays and its slices Manipulating code variables and constants

Programming Environments and Tools (V) n Performance Analysis Tools n n n Help a programmer to understand the performance characteristics of an application Analyze & locate parts of an application that exhibit poor performance and create program bottlenecks Major components n n A means of inserting instrumentation calls to the performance monitoring routines into the user’s applications A run-time performance library that consists of a set of monitoring routines A set of tools for processing and displaying the performance data Issue with performance monitoring tools n n Intrusiveness of the tracing calls and their impact on the application performance Instrumentation affects the performance characteristics of the parallel application and thus provides a false view of its performance behavior

Performance Analysis and Visualization Tools Tool Supports URL AIMS Instrumentation, monitoring library, analysis http: //science. nasa. gov/Software/AIM S MPE Logging library and snapshot performance visualization http: //www. mcs. anl. gov/mpich Pablo Monitoring library and analysis http: //wwwpablo. cs. uiuc. edu/Projects/Pablo/ Paradyn Dynamic instrumentation running analysis http: //www. cs. wisc. edu/paradyn Sv. Pablo Integrated instrumentor, monitoring library and analysis http: //wwwpablo. cs. uiuc. edu/Projects/Pablo/ Vampir Monitoring library performance visualization http: //www. pallas. de/pages/vampir. htm Dimenma s Performance prediction for message passing programs http: //www. pallas. com/pages/dimemas. htm Paraver Program visualization and analysis http: //www. cepba. upc. es/paraver

Programming Environments and Tools (VI) n Cluster Administration Tools n Berkeley NOW n n n SMILE (Scalable Multicomputer Implementation using Low-cost Equipment) n n Gathers & stores data in a relational DB Uses Java applets to allow users to monitor a system Called K-CAP Consists of compute nodes, a management node, & a client that can control and monitor the cluster K-CAP uses a Java applet to connect to the management node through a predefined URL address in the cluster PARMON n n n A comprehensive environment for monitoring large clusters Uses client-server techniques to provide transparent access to all nodes to be monitored parmon-server & parmon-client

Cluster Applications

Cluster Applications n n Numerous Scientific & engineering Apps. Business Applications: n E-commerce Applications (Amazon, e. Bay); n Database Applications (Oracle on clusters). Internet Applications: n ASPs (Application Service Providers); n Computing Portals; n E-commerce and E-business. Mission Critical Applications: n command control systems, banks, nuclear reactor control, star-wars, and handling life threatening situations.

Early Research Cluster Systems Project Platform Communications OS/Manag ement Other Beowulf PCs Multiple Ethernet with TCP/IP Linux + PBS MPI/PVM. Sockets and HPF Berkeley Now Solaris-based Myrinet and Active PCs and Messages workstations Solaris + GLUnix + x. FS AM, PVM, MPI, HPF, Split-C HPVM PCs NT or Linux connection and global resource manager + LSF Java-fronted, FM, Sockets, Global Arrays, SHEMEM and MPI Solaris MC Solaris-based Solaris-supported PCs and workstations Myrinet with Fast Messages Solaris + C++ and Globalization CORBA layer

Cluster of SMPs (CLUMPS) n Clusters of multiprocessors (CLUMPS) n n n To be the supercomputers of the future Multiple SMPs with several network interfaces can be connected using high performance networks 2 advantages n n Benefit from the high performance, easy-to-useand program SMP systems with a small number of CPUs Clusters can be set up with moderate effort, resulting in easier administration and better support for data locality inside a node

Many types of Clusters n High Performance Clusters n n GFS; parallel filesystems; same view of data from each node n n load-level tcp connections; replicate data n n Move processes around to borrow cycles (eg. Mosix) n n Linux Cluster; 1000 nodes; parallel programs; MPI Oracle Parallel Server; n Service. Guard, Lifekeeper, Failsafe, heartbeat, failover clusters Load-leveling Clusters Web-Service Clusters Storage Clusters Database Clusters High Availability Clusters

Summary: Cluster Advantage n n n Price/performance ratio of Clusters is low when compared with a dedicated parallel supercomputer. Incremental growth that often matches with the demand patterns. The provision of a multipurpose system n n Scientific, commercial, Internet applications Have become mainstream enterprise computing systems: n n As Top 500 List, over 50% (in 2003) and 80% (since 2008) of them are based on clusters and many of them are deployed in industries. In the recent list, most of them are clusters!

Backup

Key Characteristics of Scalable Parallel Computers