Minimalist s Linux Cluster Changyoung Choi Jeonghyun Kim Seyong

Minimalist’s Linux Cluster Changyoung Choi, Jeonghyun Kim, Seyong Kim Department of Physics Sejong University

Reported in Lattice’ 99

1. Usual Linux cluster • Easy to build, use and maintain • Off-The-Shelf everything! – Ordinary PC’s – Ordinary network components – Free OS (Linux) – Free parallel programming environment (MPI)

Room for improvement • Occupies too much space • Consumes too much power • Not all the components in an ordinary PC is necessary for computation – Possible further reduction in cost! – Possible reduction in power consumption • Supporting full bisection bandwidth is costly!

Conditions • No custom-made hardwares – If it takes too long to develop -> no advantage – If there is, should be recyclable • • Should be efficient enough Should be do-able with small manpower

2. Construction(hardware) • Node – ATX size standard mother board – Intel Pentium IV-2. 4 GHz – 512 Mbytes DDR SDRAM – 100 Mbps ethernet card (3+1) – 180 W power • Cage for ATX size mother board • 1 front end server

CPU Intel P-4, 2. 4 GHz 236, 500(=$1 98)x 36 motherboard ASUS P 4 PE memory NIC 512 MB PC 2700 DDR 3+1 realtek power Sun ATX 180 W cable 203, 500(=$1 70)x 36 112, 200(=$9 3. 5)x 36 53, 000(=$44 )x 36 25, 300(=$21 )x 36 10, 000(=$8)x 36

cabinet 2000 x 910 x 750 880, 000(= $733) cabinet accessory 365, 200(= $304) Cost per node $534. 5 Total cost $534. 5 x 36+$1037 = $20, 279 Price Includes Tax

Software • WOW Linux 7. 1 (equiv. Red. Hat 7. 1) • Linux Terminal Server Project (LTSP) – Remote booting (DHCP + TFTP) – Diskless node system • 2 -D mesh network configuration – Script developed in-house • MPI programming environment – MPICH/LAM

Setup • Front end wait • F-end send DHCP info • send kernel using TFTP service • send client file system by NFS • Node 1 power-up • NIC listens to DHCP server • IP setup complete and download kernel • setup file system on memory and copy NFS mounted file system to memory

Setup(continued) • Node 1 wait • Node 1 send DHCP info • Node 1 send kernel image • Node 1 act as a pipe line between front end and node 2 for NFS • Node 2 power-up • NIC listens to node 1 DHCP server • IP setup complete and download kernel from node 1 • setup file system on memory and copy NFS mounted file system to memory

Parallel boot

3. Performance • 1 -D ring full QCD code – Not suitable for 2 -D mesh structure • 8^3 x 512 lattice

8^3 x 32 lattice tested on 1 -node 1187 sec, CG = 181. 2 -> 369 MFLOPS 8^3 x 512 lattice tested 5522 sec, CG = 391. 4 -> 1. 24 GFLOPS on 4 -node 8^3 x 512 lattice tested 3029 sec, CG = 390. 9 -> 2. 25 GFLOPS on 8 -node

4. Discussion • recyclable – cabinet takes standard motherboard size -> replacing node is enough for a upgrade • occupies smaller foot-print, consumes less power • need to fine-tune • need to modify the code for 1 -D ring to 2 -D mesh