Procuring a Machine Room The RAL experience HEPi

Procuring a Machine Room The RAL experience HEPi. X – Genome Sequencing Centre, Washington University in St Louis Martin Bly, Graham Robinson, John Gordon

So you want a new machine room? • • • What is a machine room? How big is it? Where can we put it? When do you want it? How long before it’s too small? Is there enough power on site for it? 7 November 2007 Procuring a Machine Room

A Machine Room is… • • • A (very) large room With power (lots) And air conditioning (needs a lot of power too) And networking And a test area And delivery access And security But also offices for staff and operators And conference rooms And telecoms And a kitchen (with all the toys) And not forgetting the rest rooms 7 November 2007 Procuring a Machine Room

RAL requirement • Need to provide space for the FY 2008/9 Tier-1 procurements and various HPC clusters • Not enough space, power or sufficient air conditioning in existing machine rooms • Predicted requirement is for space for 300 racks and 4 tape silos up to 2012. • Space to expand later if new building not big enough • Modular so easier to expand • Flexibility… 7 November 2007 Procuring a Machine Room

Machine Room 2 • Raised floor with… – – Enough room for the cable ducts Air con circulation Power delivery Chilled water delivery • Ceiling space for … – – Network cabling Power cabling Air con circulation Chilled water delivery • So you need to decide what will go where • Use fluid dynamics models to simulate the options – Expensive but worth it – Consider buying the software and the model to run it your self 7 November 2007 Procuring a Machine Room

First hurdle • Management approval – Try to avoid organisational reconfiguration periods – Other organisational priorities and projects • Competition for resources (money, effort, attention) – Helps to have a show-stopper reason for the project – Consistent forward looks of capacity requirements from users • Finance – Always difficult to obtain – Tends to add riders • Project Manager • Project Sponsor 7 November 2007 Procuring a Machine Room

RAL method • STFC has a standard building procurement method – No say over architects, consultants (mechanical, electrical, structural), quantity surveyors – Use first meetings to sound out their experience and expertise, point out deficiencies to PM • More experienced consultants are generally more senior and cost more, worth it • Pushing consultants outside their experience is a challenge • Machine rooms are very specialist • Unless you get SUN, HP, IBM, etc to do it for you 7 November 2007 Procuring a Machine Room

Complications • At RAL, funding constraints imposed three storey building with offices on top two floors in order to get funding to build a machine room with sufficient floor space – At least computing staff have offices close to machine room • Location of power feeds • Existing underground ducts, drains and voids – Must interface with drains, telecoms ducts etc. • Existing computing buildings – How do they fit in the operations model? – Existing building is the centre of the site network • Fast networking between existing and new building – Staff access and accommodation 7 November 2007 Procuring a Machine Room

Regulations and Rules • Regulatory controls – Building regulations change – New ones difficult to interpret • Carbon footprint rules effect everything – Particularly when you mention 5 MW power feed! 7 November 2007 Procuring a Machine Room

Distractions • Office layout – – Open plan, cubicle farm, separate offices Office air con Styling Drain on effort • ‘Make it pretty!’ – Building location can make this important – at RAL it will be the first building visitors see after they enter site so it ‘must look good’ – Actually quite easy: fancy staircase at the entrance 7 November 2007 Procuring a Machine Room

Project Specifications / Client Requirements 1 • Design starts here • Document defines what YOU want • Tight specification – Stick to beat consultants with - don’t leave any ‘wriggle room’ • Make sure they read it carefully – don’t assume they will • Don’t forget the restrooms and showers – Standard numbers for facilities for number of staff on a floor which may not be enough 7 November 2007 Procuring a Machine Room

Project Specifications / Client Requirements 2 • Check all drawings for conformity to spec • Make sure all spec changes agreed in design meetings make it into the spec document and on to the plans. Note changes in meeting records. • Check and recheck drawing and insist errors and omissions are fixed immediately – don’t wait for it to be ‘fixed later’. • The later you make changes, the more difficult it is – you can end up going to Project Board level to get trivial items added – like the toilets the architect left out. 7 November 2007 Procuring a Machine Room

Project Specifications / Client Requirements 3 • Try to be as informed and knowledgeable as possible about technical issues: power, air conditioning, earth meshes, fire prevention etc. – May know more than the consultant in some specialist areas (EMR, earthing) • Bring in additional specialists as required. • Consultants don’t like ‘amateurs’ telling them their job – be prepared to be sidelined in the latter stages as the design progresses (what do you have to contribute? ) • But still attend the meetings – you could stop some disastrous compromise in some crucial area that constrains a key element of the design 7 November 2007 Procuring a Machine Room

Design 1 • Driven by RAL • 300 racks plus four tape silos (SL 8500 type) • From estimates of Tier-1 and STFC HPC and corporate requirements – Tier-1 planning out to 2012 and beyond – Power requirements extrapolated from existing equipment and ASHRAE data (~5% increase per annum): indicate ~2. 5 MW by 2013. – Dual and quad core chips change footprint and power density data – Machine room less densely packed to allow cooling to work efficiently 7 November 2007 Procuring a Machine Room

Design 2 • Design started with in-house architect to work up basic spec before formal design process started. • Useful and cheap learning process, gives a clear idea of what is required and why. • An education for the PM • Beware of know-all architects who think they no better – easy to lose key elements if they think you don’t need it (toilets!). 7 November 2007 Procuring a Machine Room

Design 3 • Make sure you understand the key elements of the design and the compromises they entail – Chilled water supply to computer room via a ring main which air handling units tap off. Provision also for water cooled racks fed from same supply at a later date. But cooling for racks is at a higher temp than for air con units to prevent condensation in racks. Reduces efficiency of the air handling units 10%. Need water cooled racks sooner! 7 November 2007 Procuring a Machine Room

Room data sheets • Project spec eventually evolves into Room Data Sheets • Describe in detail every room, corridor, stairway, cubbyhole and space in the building (and outside too – external security…) • Includes number and location of power points, UTP ports, phone sockets etc. • Define this yourself because architects won’t have a clue how many is enough • Then double the numbers 7 November 2007 Procuring a Machine Room

Checking • Check the drawings again and again for conformity to the description in the RDS – If it can be misunderstood, it will be – ‘You can’t possibly need that many UTP sockets!’ • (want to bet? ) • Always check revisions to RDS and drawings are incorporated into the next issue – Check the meeting records 7 November 2007 Procuring a Machine Room

Building the building • Building contract type defined by professionals • Based on time constraints • Single Design and Build – quick but high risk. • Separate Design and Build contracts – less risky but much slower • Start well before the building is needed • Two years is too tight for single design and build 7 November 2007 Procuring a Machine Room

Contractor selection • Contractor selection should be left to the pros • In our case, based on need to meet the deadline, with no reference to client. • Usual criteria are price and technical assessment of bid – only pros can do this 7 November 2007 Procuring a Machine Room

Cooling • Noted that air cooling would not meet requirements at some stage in future – design for water cooled racks. • Chilled water ring main with taps for both air handling and water cooled racks • More efficient than direct exchange units – But ring main is a single point of failure 7 November 2007 Procuring a Machine Room

Noise • Servers are noisy! • Require silos and low power systems to be in separate room from the data processing and storage systems • Allows fire-prevention barrier wall between the two areas • Put the quieter room next to the operators control room 7 November 2007 Procuring a Machine Room

UPS • Consider the need for UPS capacity • Why? – What do you want to achieve? • How much power? • How long – depends on what you need to achieve – Time to shutdown (disk) systems cleanly? – Continue operation of vital systems? • Including the networking – Spec it to match • Diesel backup? • RAL: Permanent UPS room: power and air con on UPS • Short term UPS for selected racks to enable a clean shutdown. 7 November 2007 Procuring a Machine Room

Combined Heat and Power • Reduces carbon footprint – Local generation more efficient – reduces losses in transmission from power station • Usually lease CHP units – Pay for natural gas used to generate electricity plus a charge per k. Wh for the electricity – Aiming for total cost less than cost from grid to save money – Some saving in condensers for chilled water • Potential to provide a UPS facility 7 November 2007 Procuring a Machine Room

CHP 2 • Good for high stable loads such as a hospital • Not so good for fluctuating loads like computer installations • Lease agreement spread over many years – Forecasting gas and electricity prices is not easy – UK govt and education establishments can get good deals on utility prices through consortium deals – Reduces potential saving, makes prediction of saving difficult 7 November 2007 Procuring a Machine Room

CHP 3 • Computers need a continuous uninterrupted supply – sophisticated load switching required to allow switch over to grid supply and back when needed • To provide a UPS function, generation must match exact load • Leasing company needs to make money on the deal – Best achieved by running the CHP full out from day one – But load in (a new) computer building builds as capacity rises, lease price initially high to compensate – Need accurate power predictions to set up lease • Penalty clauses if don’t meet the load • If exceed the capacity of a single unit (1. 5 MW), need to wait until load is closer to 3 MW to make leasing a second unit an economic proposition – No UPS in capacity in the interim • Could be better to wait until building in use so unknowns are less so. 7 November 2007 Procuring a Machine Room

7 November 2007 Procuring a Machine Room

7 November 2007 Procuring a Machine Room