Improving Computer Room efficiency with freecooling National

Improving Computer Room efficiency with freecooling – National Centre case study M W Brown CEng MIEE EPCC, University of Edinburgh Facility Manager: Advanced Computing Facility June 2008

§ § § Overview of the Advanced Computing Facility The problem "hector" – outline of requirements The solution Initial results Summary June 2008 2

Advanced Computing Facility § Constructed 1976 for the University of Edinburgh: • 1 x 600 m² Computer Room • 24 -stage DX-based cooling (R 12!) servicing the room through 4 vast walk-in air-handling units • "conventional" downflow system § Refurbished 2004 as the Advanced Computing Facility: • 2 x 300 m² Computer Rooms (one active, one empty concrete shell) • all new chilled-water based plant services, with capacity of 1. 2 MW § Major expansion 2007 for "hector" (UK national service): • 2 nd Computer Room brought into operation • new-build external plant room to support massive uplift in required capacity • new HV electrical provision (up to 7 MW) June 2008 3

Computer Room 2, ACF § General-purpose Computer Room § laid out with 10 x 6 m equipment rows, with alternating "hot/cold" aisles § 500 mm subfloor § 4 m from floor level to ceiling § 10 x 60 k. W capacity CCU's arranged along "long walls", supplied from 8° flow/14° return chilled-water system § dual 3 -ph underfloor busbars supply power to each row § Large mix of equipment from many suppliers § designed for approx 400 k. W heat-rejection to air June 2008 4

General-purpose computer room layout § A typical computer room may be arranged with alternating hot/cold aisles 2 x 600 mm tiles wide § Chilled air is supplied through vented floor tiles § Rack-mounted equipment draws in air from the cold aisle through the front, and vents out the back § Room A/C units (chilled-water or DX) arranged along the side walls, typically taking in return air about 2 m from floor June 2008 5

Problems with this layout § Supply air gets mixed with room air raising its temperature prior to being captured by the inlet fans § Incomplete rows allow leakage from cold aisle to hot aisle, thereby wasting chilled air § Racks at the ends of the aisles may suffer from: • leakage of warmer air from the side aisles • starvation of chilled air as the underfloor air is forced into the centre by the CCU fans § Return air into the CCU's has mixed with high-level room air and has thus cooled: • this means that the return air onto the coil is cooler, hence narrower (and less efficient) Δt across the coil • the returning air has transferred some of its heat directly to the room air, thus contributing to the inefficient pre-warming of the supply air June 2008 6

Problems with this layout § Recent measurements at ACF Computer Room 2 (conventional layout): § § Cold aisle temps (midway) in the range: 16. 4° to 18. 5° Hot aisle temps (midway) in the range: 26. 5° to 31. 2° Side aisle just 1 tile (600 mm) off end of cold aisle: 20. 4° CCU inlet temps (2. 2 m off ground): mean of 24° June 2008 7

Problems with this layout § To maximise the efficiency of air-side cooling, you need to separate as far as possible supply and return air § However this is not easy in a general-purpose room designed for flexibility - and thus which may contain a variety of equipment with different loads, different rack designs and dimensions, and from a range of suppliers § A general-purpose room is by definition a compromise, but recent developments in water-assisted racking systems should go far towards enabling that supply/return air separation June 2008 8

Improvements § Replacing multiple independent DX-based room-units with chilled-water units serviced from remote central plant § Having an effective BMS system that can measure room conditions as a whole and adjust local plant (CCU's) and remote plant (chillers etc) without the inefficiencies of multiple independent room units hunting against each other § Improving airflow: • • avoiding short-circuits into and between aisles careful selection of placement of vented floor tiles good underfloor depth with a minimum of obstructions reduction of return-air mixing by increasing height of CCU inlets June 2008 9

Improvements § Selection of CCU's with VSD control of their fans reduces energy when the preference is to run all units concurrently § Selection of cooling towers with VSD control of their fans allows towers to ramp up and down according to load without big fans kicking in and out § Careful selection of chilled-water flow/return temps, and also condenser water temps – allowing a lower condenser water inlet temp to the chillers may increase fan power to the towers, but compressors then may not have to work so hard in compensation June 2008 10

Air versus water cooling § However, power/space density is going up. . . § RCO Building, University of Edinburgh (1976): • designed round a power/space density of approx 0. 5 k. W/m² § Daresbury Laboratory C Block refurbishment (2002): • designed round a power/space density of approx 2. 5 k. W/m² § ACF (phase 1), University of Edinburgh (2004): • designed round a power/space density of approx 2. 5 k. W/m² § ACF (phase 2), "Hector" UK National Service (2007): • designed round a power/space density of approx 7 k. W/m² June 2008 11

Air versus water cooling § Rack power is going up: • 2002: IBM p 690 (HPC-X UK National Service at Daresbury): 10 k. W per rack • 2007: Cray XT 4 ("hector" UK National Service at Edinburgh): 18 k. W per rack • 2008: Cray XT 5 (various HPC sites in US and elsewhere): 38 k. W per rack § This is now at (or beyond) the effective limits of direct aircooling § Suppliers now must either move towards efficient packaging with water-assisted cooling directly in the racking, or more radical methods of direct liquid cooling June 2008 12

Air versus water cooling § Water is a far more efficient heat-transfer medium than air § Why try and cool the entire volume of a Computer Room when most of that air is not being used in the cooling of the equipment ? § Huge amounts of energy are used just moving air around. . . June 2008 13

Air versus water cooling But. . . § Water-cooling infrastructure requires central plant with high capital cost both in plant and physical external space for that plant § Water and expensive electronics are not a good mix, nor are water and high-power electrical supplies. . . June 2008 14

"hector" § UK national HPC service, Oct 2007 – Oct 2013 § Funded by central Government, with EPSRC as the managing agent § § £ 113 M project (capital & recurrent) in 3 x 2 -yr phases Technology (phase 1 & 2) provided by Cray Science Support provided by NAG Ltd Facility operations by partnership of University of Edinburgh and STFC (Daresbury Laboratory) § Physical location: secure site operated by Uo. E June 2008 15

"hector" § Phase 1 (accepted: Sep 07): • 60 TFlop Cray XT 4 • approx max input power of 1. 2 MVA • approx cooling load of 1. 2 MW (heat rejection directly to air) § Phase 2 (installation: summer 09): • • • ~60 Tflop Cray XT 4 (quadcore upgrade) ~200 TFlop Cray (tba) approx input power of 1. 8 MVA approx cooling load of 300 k. W (heat rejection directly to air) approx cooling load of 1500 k. W (to water via R 134 a loop) § Phase 3 (installation: summer 11): • technology supplier subject to future tender • anticipate infrastructure requirements approx as per Phase 2 June 2008 16

"hector" § We were given a very short time to prepare a computer room specifically to support the three phases of "hector" § Energy efficiency was an obvious requirement – even though as an operator we were unable to accept the risk on energy pricing – wisely as it has turned out. . . § Maximising efficiency became a key design goal in order to: 1. meet University requirements regarding energy efficiency 2. be compliant with Government policy regarding energy efficiency in public-sector projects 3. reduce recurrent expenditure thereby saving tax-payer's money 4. common sense! June 2008 17

The solution § § § § Phase 1 infrastructure requirements Outline design for specialised Computer Room Specification of plant services Project timeline Computer Room design details Chilled-water system design details Free cooling design and operation June 2008 18

Phase 1 infrastructure requirements § 60 x Cray XT 4 (dualcore) systems • • input power: in the range 18 -> 20 k. VA each all heat rejected to air chilled air (recommended on-temp of 13°) drawn in directly from subfloor by large 3 -phase variable-speed blower heated air ejected directly out of the top of the cabinet (typically at 42°) June 2008 19

Phase 2 infrastructure requirements § 16 x Cray XT 4 (upgraded to quadcore) systems • • input power: in the range 14 -> 20 k. VA each all heat rejected to air chilled air (recommended on-temp of 13°) drawn in directly from subfloor by large 3 -phase variable-speed blower heated air ejected directly out of the top of the cabinet (typically at 42°) § 24 x New Generation Cray cabinets • • input power: expected to be ~40 k. VA each phase-change evaporative cooling – air within each cabinet drawn across evaporator pipework containing R 134 a and returned to room 1 x XDP (HX) per 4 cabinets R 134 a condensed by chilled water (planning assumption: 10°/16°) June 2008 20

Computer room – outline design § Required infrastructure must be able to cope with both Phase 1 and Phase 2 cooling requirements § High-capacity chilled-water main supplying water at 8° to 14 x 80 k. W capacity CCU's set to supply air off-coil at 13° (+/0. 4°) § Valved connections installed for 12 x XDP HX units for Phase 2 § Install lowered ceiling designed to capture exhaust air from XT 4's, with inlets to CCU's ducted directly from ceiling void § Aim to maximise return air temp to widen Δt across coil and minimise interaction/mixing with room air June 2008 21

Computer Room - outline design § 700 mm between top of cabinets and ceiling void – to minimise mixing of exhaust air and room air § VFD control on CCU's, modulated to supply 60 m³/sec into the floor void (capability: 120 m³/sec) § At normal operation, chilled-water flow rate is around 40 l/s with 8° flow and 14° return § No room conditioning – control only the supply air into the sub-floor. Room ambient maintained at a comfortable level through minor leakage via cable-ways June 2008 22

Specification of plant services § Central plant was required to provide cooling of up to 2. 6 MW (with at least N+1 redundancy in all key elements) § Security of electrical supplies and protection against their diminished quality required significant enhanced electrical provision § Maximising of operating efficiency was a key objective June 2008 23

Chilled water system design details § 3 x parallel 1. 2 MW capacity chillers (duty, standby, reserve) with triple chilled-water circulation pumps (VSD-controlled) always running. 8° flow/14° return § Variable-flow through CCU's and chillers § 6 dry cooling towers for condenser water, with triple condenser water circulation pumps (VSD-controlled) always running. VSD-controlled fans on towers. 32° flow/27° return § 2 x 27, 000 lit capacity buffer-vessels June 2008 24

Chilled water schematic June 2008 25

Plant Room B § New 470 m² Plant Room constructed Jan-Jul 07 to supply services solely for the "hector" services § In prospective: the Plant Room is 1. 5 x the area of the room it services! § Contains all HV switchgear, 4 x transformers, 3200 k. VA UPS modules, chillers, condenser water/chilled water pumps and main controls § "Lights out" operation – no plant operators June 2008 26

Project timeline § § § 27 Jan 07: cut ground for construction of 470 m² Plant Room B mid Mar 07: walls to full height 24 Mar 07: steelwork for roof structure completed 08 May 07: Computer Room 1 refurbishment completed 25 May 07: HV switchroom commissioned mid Jun 07: Cooling towers installed 02 Jul 07: Plant set to work – final commissioning tests (1 MW loadbanks) 26 Jul 07: Start of delivery/installation of Cray XT 4 Aug 07: Cray XT 4 installation/commissioning 12 Sep 07: Entered final acceptance 01 Oct 07: Service commenced June 2008 27

HV infrastructure, 2007 June 2008 28

Protection against power instability § UPS (static, 10 -20 mins autonomy) for Computer Room loads only. Principally for providing clean high-quality 3 ph/50 Hz § Multiple 400 k. VA (2004) and 800 k. VA (2007) units supplied from different sides of their LV boards § MUST keep cooling running when the UPS is maintaining power to the Computer Room § Standby 500 k. VA generators supply power to "essential" services only (pumps, CCU’s, MCC panel etc). Load shed everything else June 2008 29

Electrical provision § 2 incomers to dedicated 11 k. V HV sub-network for the facility § 6 x transformers • 2 x 1. 5 MVA supply original (phase 1) parts of building • 2 x 1. 6/2. 4 MVA supply "hector" UPS switchboard and hence Computer Room connected loads • 2 x 1. 6 MVA supply all mechanical services for "hector" § 3 x dual-section LV boards, each supplied by 2 x TX § 2 x 500 k. VA diesel generators § 8 x static UPS modules: • 2 x 100 k. VA for "hector" MCC panel and chilled-water circ. pumps) • 2 x 400 k. VA (for Computer Room 2) • 4 x 800 k. VA (for Computer Room 1) June 2008 30

Cooling system performance § The average off-coil air temperature is maintained with ease in the range: 12. 7° - 13. 3° (in excess of design spec) § The average chilled-water flow temperature is maintained in the range: 7. 7° - 8. 3° (load independent) § The average chilled-water return temperature is maintained in the range: 13. 7° - 14. 3° § 60 m³ per sec of air at mean 13° is supplied into the sub-floor § Chilled-water flow rate is maintained at 40 lit per sec June 2008 31

Free cooling design and operation § Stage 1: (when OAT < 13°) • • • valves open to allow return chilled-water to divert via secondary cooling towers fan-speeds on all towers set to 30% mechanical services power drops by about 10% (200 k. W to 180 k. W) § Stage 2: (when return chilled-water off towers < 9. 7°) • • • fans modulate between 30% and 70% (aim to achieve 8°) duty chiller backs right off unless chiller entering temp > 9. 7° further power reduction of about 15% (180 k. W to about 150 k. W) § Stage 3: (when return chilled-water off towers < 8. 5°) • • duty chiller setpoint raised to 11. 5° to keep chiller off max power reduction down to around the 60 k. W baseload required to maintain flows of air and water June 2008 32

Free cooling design and operation § Stage 1 freecooling commences when the OAT is < 13° § Stage 2 freecooling is load-dependent but appears to take over from Stage 1 when OAT is around 6° § On observed loadings, the chiller appears to shut down when the OAT is around 2. 5°, but typically the chiller is held off until the temperature has risen to around 4° § Despite this being the week with mid-summer, Stage 3 freecooling was engaged between 17/2145 and 18/0815 June 2008 33

Freecooling opportunity (at 56°N) June 2008 34

Snapshot: Thu 12 June (Computer Room 1) Output from TX 3/TX 4 (input to UPS): 1022 k. W Output from TX 5/TX 6 (mech. services): 199 k. W Duty chiller (no 3): 128 k. W Room CCU's: 23 k. W Cooling towers, fans and pumps: 48 k. W Total input power: UPS losses: 1221 k. W 62 k. W (5%) Mechanical services loads: Computer Room connected load: 960 k. W (79%) Total overhead (%ge of connected load): June 2008 199 k. W (16%) 261 k. W (27%) 35

Snapshot: Thu 12 June (Computer Room 2) Duty chiller (no 3): 80 k. W Room CCU's: 63 k. W Cooling towers, fans and pumps: 41 k. W Total input power: UPS losses (estimated): 601 k. W 57 k. W (9%) Mechanical services loads: Computer Room connected load: 360 k. W (60%) Total overhead (%ge of connected load): June 2008 184 k. W (31%) 241 k. W (67%) 36

Dec 07 – input power to UPS June 2008 37

Dec 07 – input power to mech. services June 2008 38

Snapshot: 16 Dec (Computer Room 1) Chiller ON Total input power: UPS losses: Mechanical services Chiller OFF 1050 k. W 58 k. W 960 k. W (6%) 58 k. W (6%) 150 k. W (14%) 60 k. W (16%) 842 k. W (80%) 842 k. W (88%) 208 k. W (25%) 118 k. W (14%) loads: Computer Room connected load: Total overhead: June 2008 39

Projected annual savings Proportion Power for Cost of year cooling Stage 3 component: 9% 60 k. W £ 47 K Stage 2 component: 17% 150 k. W £ 15 K Stage 1 component: 46% 180 k. W £ 3 K No freecool component: 28% 200 k. W £ 32 K Connected load of 960 k. W June 2008 40

Projected annual savings unoptimised stages 1 -3 design freecooling (72%) Connected load: 960 k. W 67% 27% 14% - 21% 14, 044, 032 10, 680, 192 10, 421, 203 6. 5 p £ 912, 862 £ 694, 202 £ 677, 378 Unit savings per year: 3. 36 GWhr 3. 63 GWhr Cost savings per year: £ 218, 650 £ 235, 303 Overhead: Units per year: Cost per unit: Cost per year: June 2008 41

"Hector" Phase 2 § Planning underway for the technology refresh due in mid 2009 § Ongoing discussions with Cray on the operating parameters for their XDP heat-exchanger unit – we are hoping to influence their design such that the chilled-water off temperatures can be maximised, thereby increasing the possibility of "free cooling" June 2008 42

Conclusions § Annual savings of energy in Gigawatt hours are projected § "Hector" efficiencies are due to: • extensive use of VSD on pumps and fan motors • maximising the separation of supply/room air through direct injection into the base of the cabinets and effective capture of the exhaust air • careful selection of chilled water flow/return temperatures that maximises changes of being able to "free cool" • optimising the design for the specific (albeit perhaps unusual) requirements of the Cray XT 4 system • provision of secondary loops through the cooling towers giving efficient mode of "free cooling" • being at 56 degrees North ! June 2008 43

Acknowledgements § People too numerous to mention have supplied me with information for this presentation, but we should acknowledge: § David Barratt (Engineering Services Manager, University of Edinburgh) § David Somervell (Energy Manager, University of Edinburgh) § Lawrence Valentine (Crown House Technology) • The bulk of the design of the "hector" cooling infrastructure flowed from the pen of Lawrence Valentine, and significant energy efficiencies have been the direct result of his skills June 2008 44