Moving out of SI 2 K How INFN

Moving out of SI 2 K How INFN is moving out of SI 2 K as a benchmark for Worker Nodes performance evaluation Michele Michelotto at pd. infn. it

INFN Problem • INFN needs to procure computing power for the CNAF Tier 1 and for several Tier 2 • Which benchmark should we use to specify our requirements? • Since at least Hepix Fall 2006 is clear the SI 2000 is close to meaningless • INFN started a working group in charge of proposing a solution Hepix Fall 2007 michele michelotto - INFN 2

What it is SI 2 K • SI 2 K is actually a nickname for SPEC CPU Int 2000 benchmark – The geometric mean of a dozen of benchmark which stress the integer performance of a cpu – Delivered by SPEC in 2000, it came after Spec 89, Spec Int 92 and Spec Int 95 – Declared obsolete by SPEC in 2006 – Replaced by CPU Int 2006 Hepix Fall 2007 michele michelotto - INFN 3

Why is it important for us? • SI 2 K is the benchmark used up to now to measure the computing power of all the HEP experiments – Computing power requested by experiment, e. g. in the Computing TDR – Computing power provided by a Lab or a Tier-x – Computing power a funding agency is committed to pay for • Well, you have seen it yourself this week. Several site reported the Kilo or Mega SI 2 K available at their sites (other talked only of cores available) Hepix Fall 2007 michele michelotto - INFN 4

Transition phase • Impossible to find SPEC Int 2000 pubblished results for the new processors – (e. g. the not so new intel Clovertown 4 core) • Impossible to find pubblished SPEC Int 2006 for old processor (before 2006) – E. g. Old P 4 Xeon, P 4, AMD 2 xx • There is no conversion constant from SI 2000 to SI 2006 – the ratio for x 86 architectures is in the 137 – 172 range Hepix Fall 2007 michele michelotto - INFN 5

SI 2 K/SI 2 K 6 ratio Hepix Fall 2007 michele michelotto - INFN 6

The SI 2 K inflaction • The main problem with SI 2000 in our community is that it not (any more) proportional to HEP codes performance • You can buy new processors with huge SI 2 K number but you gain only a smaller increase in real performances • The face value is not the real value Hepix Fall 2007 michele michelotto - INFN 7

Nominal SI vs real SI • SI 2 K results for the last generation processor affected by inflation • So CERN (and FZK) started to use a new currency: SI 2 K measured with “gcc”, the gnu C compiler and using two flavour of optimization – High tuning: gcc –O 3 –funroll-loops– march=$ARCH – Low tuning: gcc –O 2 –f. PIC –pthread Hepix Fall 2007 michele michelotto - INFN 8

Nominal SI vs real SI • CERN Proposal: Use as site rating the “Real SI” obtained by SI measured with gcc-low and increased by 50% – This make sense but only for a short period of time and for the last generation of processor (2006 -2007) – Actually is has been proposed as a short term solution • Run n copies in parallel – Where n is the number of cores in the worker node – To take in account the drop in performance of a multicore machine when fully loaded. – Preferred to the use of Spec INT 2000 Rate which had syncronized access to memory, unlike our batch farm worker nodes Hepix Fall 2007 michele michelotto - INFN 9

Too many SI 2 K • An example: how to evalutate a worker node with two Intel Woodcrest dual core 5160 at 3. 06 GHz ? • SI 2 K (1 core): 2929 – 3089 (min – max) • SI 2 K on 4 cores: 11716 - 12536 • SI 2 K on 4 cores gcc-low : 5523 • SI 2 K on 4 cores gcc-high: 7034 • SI 2 K on 4 cores gcc-low + 50%: 8284 • SI 2 K on 4 cores gcc-high + 50%: 10551 Hepix Fall 2007 michele michelotto - INFN 10

Even more • Actually all the gcc results in the previous slide are on i 386 (32 bit) • if you would like to know how your code is running on 64 bit machine, you can measure Spec INT 2000 with gcc on x 86_64 • So the worker node with two Intel Woodcrest dual core 5160 at 3. 06 GHz • SI 2 K nominal: 2929 – 3089 (min – max) • SI 2 K on 4 cores: 11716 - 12536 • SI 2 K on 4 cores gcc-low: 6021 • SI 2 K on 4 cores gcc-high: 6409 • SI 2 K pn 4 cores gcc-low + 50%: 9031 Hepix Fall 2007 michele michelotto - INFN 11

A scale factor • All these numbers would be only annoying in a world with a unique architecture, in which only clock improves in time • You would be able to find a fixed ratio between all those number. • But in the real world, the ratio depends on both the CPU producer (intel vs AMD) and processor generation (old xeon vs new “core” Xeon ) Hepix Fall 2007 michele michelotto - INFN 12

The nominal SI 2 K Big Differences between Intel and AMD when SI 2 K/GHz are plotted Hepix Fall 2007 michele michelotto - INFN 13

The nominal SI 2 Krate SI 2 K RATE differentiate the dual core (TOP) from the 4 core (bottom) Smaller differences between Intel and AMD using SI 2 K Rate /GHz / core Hepix Fall 2007 michele michelotto - INFN 14

The “CERN” SI • The intel-amd gap is now smaller • Intel Xeon 5 xxx scores -50% wrt nominal • Amd Opteron scores only -29% • Big differences between 32 and 64 bit on same core • SI 2 K/core/Ghz derived by M. Alef page on the HEPIX site at Caspur Hepix Fall 2007 michele michelotto - INFN 15

SI 2 K 6 rate Hepix Fall 2007 michele michelotto - INFN 16

Which is the better? • At INFN we started to measure performances of HEP codes on several machines • Our goal was to find a “commercial” mantained benchmark to replace SI 2 K • We compared HEP code with – SI 2 K pubblished results – SI 2 K CERN (gcc low) – SI 2006 and SI 2006 rate pubblished results – SI 2006 and SI 2006 with gcc 4 (32 and 64 bit) Hepix Fall 2007 michele michelotto - INFN 17

Babar Tier. A Results Hepix Fall 2007 • If you normalize by core and clock all new processors have the same performance • Doubling the older generation cpu • SI 2006 matches this pattern (pubblished and gcc ratio constant) • SI 2000 -cern better than SI 2 K nominal • SI 2000 clearly does’nt michele michelotto -work INFN 18

CMS MC SIM and Pythia • CMS Montecarlo simulation (32 bit) and Pythia (64 bit) show the same performance once normalized • Both Specint 2006 pubblished and Specint 2006 with gcc show the same behaviour • SI 2 K pubbished does not match HEP sw • SI 2 K cern better but not as good as SI 2006 Hepix Fall 2007 michele michelotto - INFN 19

Atlas • • • Hepix Fall 2007 michele michelotto - INFN Here 100% is Xeon 5160 Few results for SI 2006+gcc but no diff from CMS and babar Few results also from SI 2006 pubblished because of several old architectures SI 2 K+gcc not bad SI 2 K pubblished heavily overstimate new Xeon Atlas simulation normalized performs the same on the new intel “core” or amd “opteron” (confirm the behaviour seen with CMS, Pithya, Babar) 20

What INFN should be do? • The Working Group proposed to infn: • Stop using SI 2000 pubblished because it gives meaningless number • SI 2 k CERN is good as a temporary solution but it is still using a 7 years old benchmark that is not mantained by SPEC (v 1. 3) – You can use it in a tender this year • We should move to a benchmark of the SPEC CPU 2006 suite – Best long term solution Hepix Fall 2007 michele michelotto - INFN 21

SI 2006 Rate Best candidate is Spec INT 2006 RATE measured with gcc – Spec INT 2006 because it’s new, you can find pubblished results to compare, will have bug fixes, and will be portable to new platform and new compiler – RATE because you can take in account of scalability issues with the whole machine architecture – Measured with GCC: to keep the environment as close as possible as the experiments. Hepix Fall 2007 michele michelotto - INFN 22

What is missing • All the official number are still expressed in terms of SI 2 K (nominal) • So funding agencies are still bounded to use SI 2 K for all formal agreements • A general agreements on the next benchmark is still missing but work is in progress • Example: use gcc 4. 1 with CERN “low” tuning and 32 bit • NB SI 2006 rate with gcc 64 bit too big for my machines. It needs more than 1 GB per core Hepix Fall 2007 michele michelotto - INFN 23

Thanks • The working group: A. Crescente, L. Dell’Agnello, D. Salomoni, F. Rosso, A. Sartirana, R. Stroili, D. Salomoni, M. Morandin, A. De Salvo, T. Zani • Thanks to CCR: the INFN Computing Committee • Thanks for input from LCG MB Hepix Fall 2007 michele michelotto - INFN 24