Planning the LCG Fabric at CERN openlab TCO

Planning the LCG Fabric at CERN openlab TCO Workshop November 11 th 2003 Tony. Cass@CERN. ch

Fabric Area Overview Installation Configuration + monitoring Fault tolerance Automation, Operation, Control Infrastructure Electricity, Cooling, Space Batch system (LSF, CPU server) Storage system (AFS, CASTOR, disk server) Network Benchmarks, R&D, Architecture Prototype, Testbeds Purchase, Hardware selection, Resource planning GRID services !? Coupling of components through hardware and software Tony. Cass@CERN. ch 2

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation u Monitoring u Hardware and Configuration and control Purchase Tony. Cass@CERN. ch 3

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation u Monitoring u Hardware and Configuration and control Purchase Tony. Cass@CERN. ch 4

Building Fabric — I u B 513 was constructed in the early 1970 s and the machine room infrastructure has evolved slowly over time. – Like the eye, the result is often not ideal… Tony. Cass@CERN. ch 5

Current Machine Room Layout Problem: Normabarres run one way, services run the other…. Services Tony. Cass@CERN. ch 6

Building Fabric — I u B 513 was constructed in the early 1970 s and the machine room infrastructure has evolved slowly over time. – Like the eye, the result is often not ideal… u With the preparations for LHC we have the opportunity to remodel the infrastructure. Tony. Cass@CERN. ch 7

Future Machine Room Layout 9 m double rows of racks for critical servers Aligned normabarres 18 m double rows of racks 12 shelf units or 36 19” racks 528 box PCs 105 k. W 1440 1 U PCs 288 k. W 324 disk servers 120 k. W(? ) Tony. Cass@CERN. ch 8

Building Fabric — I u B 513 was constructed in the early 1970 s and the machine room infrastructure has evolved slowly over time. – Like the eye, the result is often not ideal… u With the preparations for LHC we have the opportunity to remodel the infrastructure. – Arrange services in clear groupings associated with power and network connections. » Clarity for general operations plus ease of service restart should there be any power failure. – Isolate critical infrastructure such as networking, mail and home directory services. – Clear monitoring of planned power distribution system. u Just “good housekeeping”, but we expect to reap the benefits during LHC operation. Tony. Cass@CERN. ch 9

Building Fabric — II u Beyond good housekeeping, though, there are building fabric issues that are intimately related with recurrent equipment purchase. – Raw power: We can support a maximum equipment load of 2. 5 MW. Does the recurrent additional cost of blade systems avoid investment in additional power capacity? – Power efficiency: Early PCs had power factors of ~0. 7 and generated high levels of 3 rd harmonics. Fortunately, we now see power factors of 0. 95 or better, avoiding the need to install filters in the PDUs. Will this continue? – Many sites need to install 1 U or 2 U rack mounted systems for space reasons. This is not a concern for us at present but may become so eventually. » There is a link here to the previous point: the small power supplies for 1 U systems often have poor power factors. Tony. Cass@CERN. ch 10

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation u Monitoring u Hardware and Configuration and control Purchase Tony. Cass@CERN. ch 11

Fabric Architecture Physical and logical coupling Level of complexity Hardware CPU Disk Storage tray, NAS server, SAN element PC Cluster World wide cluster Software Motherboard, backplane, Bus, integrating devices (memory, Power supply, controller, . . ) Operating system, driver Network (Ethernet, fibre channel, Myrinet, …. ) Hubs, switches, routers Batch system, load balancing, Control software, Hierarchical Storage Systems Wide area network Tony. Cass@CERN. ch Grid middleware 12

Tony. Cass@CERN. ch 13

Tony. Cass@CERN. ch 14

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – – – The load characteristics The batch scheduler Chip technology Processors/box The operating system Others? Tony. Cass@CERN. ch 15

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – The load characteristics » Not much we in IT can do here! – – – The batch scheduler Chip technology Processors/box The operating system Others? Tony. Cass@CERN. ch 16

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – The load characteristics – The batch scheduler » LSF is pretty good here, fortunately. – – Chip technology Processors/box The operating system Others? Tony. Cass@CERN. ch 17

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – The load characteristics – The batch scheduler – Chip technology » Take hyperthreading, for example. Tests have shown that, for HEP codes at least, hyperthreading wastes 20% of the system performance running two tasks on a dual processor machine. There are no clear benefits to running with hyperthreading enabled when running three tasks. What is the outlook here? – Processors/box – The operating system – Others? Tony. Cass@CERN. ch 18

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – – The load characteristics The batch scheduler Chip technology Processors/box » At present, a single 100 base. T NIC would support the I/O load of a quad processor CPU server. Quad processor boxes would halve the cost of networking infrastructure—but they come at a hefty price premium (XEON MP vs XEON DP, heftier chassis, …). What is the outlook here? u And total system memory becomes an issue. – The operating system – Others? Tony. Cass@CERN. ch 19

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – – – The load characteristics The batch scheduler Chip technology Processors/box The operating system » Linux is getting better, but things such as processor affinity would be nice. u Relationship to hyperthreading… – Others? Tony. Cass@CERN. ch 20

Batch Subsystem u Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? – – – The load characteristics The batch scheduler Chip technology Processors/box The operating system Others? Tony. Cass@CERN. ch 21

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation u Monitoring u Hardware and Configuration and control Purchase Tony. Cass@CERN. ch 22

Storage subsystem u Simple building blocks: CPU server + Fiber Channel Interface + tape drive == Tape server Processors “desktop+” node == CPU server + larger case + 6*2 disks == Disk server Tony. Cass@CERN. ch 23

Tony. Cass@CERN. ch 24

Tony. Cass@CERN. ch 25

Storage subsystem — Disk Storage u TCO: Maximise available online capacity within fixed budget (material & personnel). – IDE based disk servers are much cheaper than high end SAN servers. But are we spending too much time on maintenance? » Yes, at present, but we need to analyse carefully the reasons for the current load. u Complexities of Linux drivers seem under control, but numbers have exploded. And are some problems related to batch of hardware? – Where is the optimum? Switching to fibre channel disks would reduce capacity by factor of ~5. » Naively, buy, say, 10% extra systems to cover failures. Sadly, this is not as simple as for CPU servers; active data on the servers must be reloaded elsewhere. » Always have duplicate data? => purchase 2 x required space. Still cheaper than SAN? How does this relate to … Tony. Cass@CERN. ch 26

Storage System — Tapes u The first TCO question is “Do we need them? ” u Disk storage costs are dropping… Tony. Cass@CERN. ch 27

Disk Price/Performance Evolution Tony. Cass@CERN. ch 28

Storage System — Tapes u The first TCO question is “Do we need them? ” u Disk storage costs dropping… But – Disk servers need system administrators, idle tapes sitting in a tape silo don’t. – With disk only solution, we need storage for at least twice the total data volume to ensure no data loss. – Server lifetime of 3 -5 years; data must be copied periodically. » Also an issue for tape, but the lifetime of a disk server is probably still less than the lifetime of a given tape media format. u Assumption required. today is that tape storage will be Tony. Cass@CERN. ch 29

Storage System — Tapes u Tape robotics is easy. – Bigger means better cost/slot. Tony. Cass@CERN. ch 30

Tony. Cass@CERN. ch 31

Storage System — Tapes u Tape robotics is easy. – Bigger means better cost/slot. u Tape drives: High end vs LTO – TCO issue: LTO drives are cheaper than high end IBM and STK drives, but are they reliable enough for our use? » c. f. the IDE disk server area. u Real problem, though is tape media. – Vast portion of the data is accessed rarely but must be stored for long period. Strong pressure to select a solution that minimises an overall cost dominated by tape media. Tony. Cass@CERN. ch 32

Storage System — Managed Storage u Should CERN build or buy software systems? u How to measure the value of a software system? – Initial cost: » Build: Staff time to create required functionality » Buy: Initial purchase cost of system as delivered plus staff time to install and figure for CERN. – Ongoing cost » Build: Staff time to maintain system and add extra functionality » Buy: License/maintenance cost plus staff time to track releases. u u Choice: Extra functionality that we consider useful may or may not arrive. – Batch system: Buy LSF. – Managed storage system: Build CASTOR. u Use this model as we move on to consider system management software. Tony. Cass@CERN. ch 33

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation u Monitoring u Hardware and Configuration and control Purchase Tony. Cass@CERN. ch 34

Installation and Configuration u Reproducibility and guaranteed homogeneity of system configuration is a clear method to minimise ongoing system management costs. A management framework is required that can cope with the numbers of systems we expect. u We faced the same issues as we moved from mainframes to RISC systems. Vendor solutions offered then were linked to hardware—so we developed our own solution. u Is a vendor framework acceptable if we have a homogeneous park of Linux systems? – Being honest, why have we built our own again? Tony. Cass@CERN. ch 35

Installation and Configuration u Installation and configuration is only part of the overall computer centre management: Tony. Cass@CERN. ch 36

ELFms architecture Fault Mgmt System Monitoring System Node Configuration System Installation System Tony. Cass@CERN. ch 37

Installation and Configuration u Installation and configuration is only part of the overall computer centre management: u Systems provided by vendors cannot (yet) be integrated into such an overall framework. u And there is still a tendency to differentiate products on the basis of management software, not raw hardware performance. – This is a problem for us as we cannot ensure we always buy brand X rack mounted servers or blade systems. – In short, life is not so different from the RISC system era. Tony. Cass@CERN. ch 38

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation u Monitoring u Hardware and Configuration and control Purchase Tony. Cass@CERN. ch 39

Monitoring and Control u Assuming that there are clear interfaces, why not integrate a commercial monitoring package into our overall architecture? u Two reasons: – No commercial package meets (met) our requirements in terms of, say, long term data storage and access for analysis. » This could be considered self serving: we produce requirements that justify a build rather than buy decision. – Experience has show, repeatedly, that monitoring frameworks require effort to install and maintain, but don’t deliver the sensors we require. » Vendors haven’t heard of LSF, let alone AFS. » A good reason! Tony. Cass@CERN. ch 40

Hardware Management System u A specific example of the integration problem. Workflows mu Tony. Cass@CERN. ch 41

Console Management u Done poorly now: Tony. Cass@CERN. ch 42

Console Management u We will do better: TCO issue: Do the benefits of a single console management system outweigh costs of developing our own? How do we integrate vendor supplied racks of preinstalled systems? Tony. Cass@CERN. ch 43

Agenda u Building u Batch Fabric Subsystem u Storage subsystem u Installation and Configuration u Monitoring and control u Hardware Purchase Tony. Cass@CERN. ch 44

Hardware Purchase u The issue at hand: How do we work within our purchasing procedures to purchase equipment that minimises our total cost of ownership? u At present, we eliminate vast areas of the multidimensional space by assuming we will rely on ELFms for system management and Castor for data management. Simplified[!!!] view: – CPU: White box vs 1 U vs blades; install or ready packaged – Disk: IDE vs SAN; level of vendor integration u HELP! u Can we benefit from management software that comes with ready built racks of equipment in a multi -vendor environment? Tony. Cass@CERN. ch 45