The opencluster in openlab A technical overview Sverre

The “opencluster” in “openlab” A technical overview Sverre Jarp IT Division CERN SJ – Mar 2003 1

Definitions n n n The “CERN openlab for Data. Grid applications” is a framework for evaluating and integrating cutting-edge technologies or services in partnership with industry, focusing on potential solutions for the LCG. The openlab invites members of the industry to join and contribute systems, resources or services, and carry out with CERN large-scale high-performance evaluation of their solutions in an advanced integrated environment. “opencluster” project The openlab is constructing a pilot ‘compute and storage farm’ called the opencluster, based on HP's dual processor servers, Intel's Itanium Processor Family (IPF) processors, Enterasys's 10 -Gbps switches and, at a later stage, a highcapacity storage system. SJ – Mar 2003 2

LHC Computing Grid LCG n n 02 03 Project Selection, Integration (technology) and deployment (full scale) Full Production service Worldwide scope 04 05 06 SJ – Mar 2003 07 08 09 3

CERN openlab LCG CERN openlab n n Framework for collaboration Evaluation, integration, validation (of cutting-edge technologies) n 02 03 3 -year lifetime 04 05 06 SJ – Mar 2003 07 08 4

Technology onslaught n Large amounts of new technology will become available between now and LHC start-up. A few HW examples: n Processors n n Memory Not all, b Interconnect this ut so PCI-X 2 PCI-Express (serial) will me o Infiniband defin be u f Computer architecture sed itely Chipsets on steroids by L Modular computers HC n n n SMT (Symmetric Multi-Threading) CMP (Chip Multiprocessor) Ubiquitous 64 -bit computing (even in laptops) DDR II-400 (fast) Servers with 1 TB (large) n n n ISC 2003 Keynote Presentation Building Efficient HPC Systems from Catalog Components Justin Rattner, Intel Corp. , Santa Clara, USA n n Disks n n Serial-ATA Ethernet n n 10 Gb. E (NICs and switches) 1 Terabit backplanes SJ – Mar 2003 5

Vision: A fully functional GRID cluster node Remote Fabric WAN Gigabit long-haul link Multi-gigabit LAN CPU Servers Storage system SJ – Mar 2003 6

opencluster strategy n Demonstrate promising technologies n n LCG and LHC on-line Deploy the technologies well beyond the opencluster itself n n 10 Gb. E interconnect in the LHC Test-bed Act as a 64 -bit Porting Centre n n Storage subsystem as CERN-wide pilot Focal point for vendor collaborations n n CMS and Alice already active; ATLAS is interested CASTOR 64 -bit reference platform For instance, in the “ 10 Gb. E Challenge” everybody must collaborate in order to be successful Channel for providing information to vendors n Thematic workshops SJ – Mar 2003 7

The opencluster today n Three industrial partners: n Enterasys, HP, and Intel n A fourth partner to join n Data storage subsystem n n Technology aimed at the LHC era n n Which will “fulfill the vision” Network switches at 10 Gigabits Rack-mounted HP servers 64 -bit Itanium processors Cluster evolution: n n n 2002: Cluster of 32 systems (64 processors) 2003: 64 systems (“Madison” processors) 2004/05: Possibly 128 systems (“Montecito” processors) SJ – Mar 2003 8

Activity overview n Over the last few months n n n Cluster installation, middleware Application porting, compiler installations, benchmarking Initialization of “Challenges” Planned first thematic workshop Future n n n Porting of grid middleware Grid integration and benchmarking Storage partnership Cluster upgrades/expansion New generation network switches SJ – Mar 2003 9

opencluster in detail n Integration of the cluster: n n n Fully automated network installations 32 nodes + development nodes Red. Hat Advanced Workstation 2. 1 Open. AFS, LSF GNU, Intel, ORC Compilers (64 -bit) n n n CERN middleware: Castor data mgmt CERN Applications n n n ORC (Open Research Compiler, used to belong to SGI) Porting, Benchmarking, Performance improvements CLHEP, GEANT 4, ROOT, Sixtrack, CERNLIB, etc. Database software (My. SQL, Oracle? ) SJ – Mar 2003 10

The compute nodes n HP rx 2600 n n Rack-mounted (2 U) systems Two Itanium-2 processors n n n 2 or 4 GB memory (max 12 GB) 3 hot pluggable SCSI discs (36 or 73 GB) On-board 100 Mbit and 1 Gbit Ethernet 4 PCI-X slots: n n 900 or 1000 MHz Field upgradeable to next generation full-size 133 MHz/64 -bit slot(s) Built-in management processor n Accessible via serial port or Ethernet interface SJ – Mar 2003 11

Opencluster - phase 1 n Perform cluster benchmarks: n Parallel ROOT queries (via PROOF) n n n “ 1 GB/s to tape” challenge n n Observed excellent scaling: n 2 4 8 16 32 64 CPUs To be reported at CHEP 2003 Network interconnect via 10 Gb. E switches opencluster may act as CPU servers 50 Storage. Tek tape drives in parallel “ 10 Gbit/s network Challenge” n Groups together all openlab partners n n Enterasys switch HP servers Intel processors and n/w cards CERN Linux and n/w expertise SJ – Mar 2003 12

Enterasys extension 1 Q 2003 13 -24 1 -12 4 Disk Servers 37 -48 25 -36 4 4 2 E 1 OAS 13 -24 1 -12 2 2 E 1 OAS 2 32 2 E 1 OAS 4 49 - 60 32 node Itanium cluster E 1 OAS 4 4 61 -72 73 -84 4 85 -96 200+ node Pentium cluster SJ – Mar 2003 1 -96 Fast. Ethernet Gig copper Gig fiber 10 Gig 13

Why a 10 Gb. E Challenge? n Demonstrate LHC-era technology n n All necessary components available inside the opencluster Identify bottlenecks n n And see if we can improve We know that Ethernet is here to stay n 4 years from now 10 Gbit/s should be commonly available n n n Backbone technology Cluster interconnect Possibly also for i. SCSI and RDMA traffic We want to advance the state-of-the-art ! 14 SJ – Mar 2003

Demonstration of openlab partnership n Everybody contributes: n Enterasys n n Hewlett-Packard n n Server with its PCI-X slots and memory bus Intel n n n 10 Gbit switches 10 Gbit NICs plus driver Stop Pre Processors (i. e. code optimization) ss u CERN n n Linux kernel expertise Network expertise Project management IA 32 expertise n p an : d ru We are with bac nning k-to conn -bac ectio k ns CPU clusters, disk servers on multi-Gbit infrastructure SJ – Mar 2003 15

Opencluster time line Install 32 nodes Start phase 1 - Systems expertise in place Complete phase 1 Start phase 2 Order/Install G-2 upgrades and 32 more nodes open. Cluster integration Jan 03 Complete phase 2 Start phase 3 Order/Install G-3 upgrades; Add nodes EDG and LCG interoperability Jan 04 Jan 05 SJ – Mar 2003 Jan 06 16

Opencluster - future n n Port and validation of EDG 2. 0 software Joint project with CMS n n Integrate opencluster alongside EDG testbed Porting, Verification n n Interoperability with EDG Testbeds and later with LCG-1 n n Relevant software packages (hundreds of RPMs) Understand chain of prerequisites Exploit possibility to leave control node as IA-32 Integration into existing authentication scheme GRID benchmarks n To be defined later SJ – Mar 2003 17

Recap: opencluster strategy n Demonstrate promising IT technologies n n File system technology to come Deploy the technologies well beyond the opencluster itself n Focal point for vendor collaborations n Channel for providing information to vendors SJ – Mar 2003 18

The Workshop SJ – Mar 2003 19

IT Division n 250 people About 200 are at engineering level Groups have both a development and a service responsibility n n 10 groups: n n n n n Advanced Projects’ Group (DI) (Farm) Architecture and Data Challenges (ADC) Communications Services (CS) Fabric Infrastructure and Operations (FIO) Grid Deployment (GD) Databases (DB) Internet (and Windows) Services (IS) User Services (US) Product Support (PS) Most of today’s speakers Controls (CO) are from ADC and DB SJ – Mar 2003 20

High Energy Physics Computing Characteristics n n n Independent events (collisions) n Trivial parallel processing Bulk of the data is read-only n versions rather than updates Meta-data in databases linking to files Chaotic workload – n research environment - physics extracted by iterative analysis, collaborating groups of physicists à Unpredictable unlimited demand Very large aggregate requirements: n computation, data, i/o SJ – Mar 2003 21

SHIFT architecture n Three tiers n Interconnected via Ethernet Tape robots CPU servers (no permanent data) DISK servers (cached data) SJ – Mar 2003 Storage. Tek Powderhorn 6, 000 1/2” tape cartridges 22

detector event filter (selection & reconstruction) reconstruction Data Handling and Computation for Physics Analysis raw data event reprocessing processed data event summary data analysis batch physics analysis event simulation CERN simulation interactive physics analysis les. robertson@cern. ch analysis objects (extracted by physics topic)

Backup SJ – Mar 2003 24