Heterogeneous Computations Cluster Hybri LIT Podgainy D V

Heterogeneous Computations Cluster Hybri. LIT Podgainy D. V. on behalf of the Heterogeneous Computations team Hybri. LIT Laboratory of Information Technologies Joint Institute for Nuclear Research Montenegro, Budva, XXV INTERNATIONAL SYMPOSIUM ON NUCLEAR ELECTRONICS & COMPUTING 28 September – 02 October, 2015 HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

Hybri. LIT Team for heterogeneous computations: Gheorghe Adam Alexander Ayriyan Dmitry Golub Mikhail Matveev Evgeny Aleksandrov Dmitriy Belyakov Margarit Kirakosayn Dmitry Podgainy Alexey Prikhodko Alexej I. Streltsov Tatiana Strizh Lucia Valova Tatiana Zaikina Peter Zrelov Tatiana Sapozhnikova Oksana Streltsova Shushanik Torosyan Martin Vala Elena Zemlyanaya Maxim Zuev HETEROGENEOUS COMPUTATIONS TEAM, Hybri. LIT

The heterogeneous computing cluster may serve • to create own software for investigations demanding resource-intensive computations • to use already developed software products and applied mathematical libraries for calculations on hybrid architectures • to develop parallel algorithms for experimental data processing and analysis using programming paradigms for specialized computing systems consisting of graphic accelerators and co-processors HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

TOP 500 List – JUNE 2015

TOP 500 List – JUNE 2015 Source: http: //www. top 500. org/blog/slideshighlights-of-the-45 th-top 500 -list/

«LOMONOSOV 2 » Supercomputer , MSU > 5000 computation nodes Intel Xeon E 5 -2697 v 3 14 C 2. 6 GHz ~82 Gb DRAM GPU Nvidia K 40 m Infiniband FDR «Tornado SUSU» Supercomputer, SUSU 480 computing units (compact and powerful computing blade-modules) 960 processors Intel Xeon X 5680 384 coprocessors Intel Xeon Phi SE 10 X Linpack Performance, TFLOPS [http: //www. top 500. org/] 1 849 288. 2

What we see: modern Supercomputers are hybrid with heterogeneous nodes • Multicore/many-core CPUs with shared memory • Multiple GPUs • Multicore/many-core CPUs with shared memory • Multiple Coprocessors • Multicore/many-core CPUs • GPUs • Coprocessors The practical decisions concerning the implementation of a heterogeneous cluster Hybri. LIT followed from the world wide trends in the high performance computing HETEROGENEOUS COMPUTATIONS TEAM, Hybri. LIT

Hybri. LIT: heterogeneous computation cluster HETEROGENEOUS COMPUTATIONS TEAM, Hybri. LIT

NVIDIA Tesla K 40 “Atlas” GPU Accelerator Intel® Xeon Phi™ Coprocessor Specifications 2880 CUDA GPU cores Memory 12 GB GDDR 5 Peak precision floating point performance 4. 29 TFLOPS single-precision 1. 43 TFLOPS double-precision Intel Xeon Phi 7120 P Clock Speed 1. 24 GHz L 2 Cache 30. 5 MB Cores 61 More threads 244 Peak precision floating point performance 2. 42 TFLOPS single-precision 1. 21 TFLOPS double-precision HETEROGENEOUS COMPUTATIONS TEAM, Hybri. LIT

Summary of current Hybri. LIT version Operation system: Scientific Linux 6. 7 File systems: EOS and NFS Batch system: SLURM

Testing Hybri. LIT: LINPACK benchmark HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

Software for parallel calculations q Open. MPI 1. 6. 5, 1. 8. 1 q Open. MP GCC, ICC q Intel Cluster Studio 2013 q CUDA 5. 5, CUDA 6. 0, CUDA 7. 5 q CUDA Open. CL V 6. 0, Intel Open. CL 4. 6. 0 HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

GPU-accelerated application Over 300 applications are optimized for the GPUs computation • Computational Finance • Defense and Intelligence • Machine Learning • Manufacturing • Media and Oil and Gas • Research: Higher Education and Supercomputing (COMPUTATIONAL CHEMISTRY; BIOLOGY NUMERICAL ANALYTICS PHYSICS) • Safety and Security NUMERICAL ANALYTICS: MATLAB – Mathworks GPU acceleration for MATLAB (high-level technical computing language) Support for 200+ of most used MATLAB functions (incl. Signal Processing, Image Processing, Communications Systems, etc) PHYSICS QUDA Library for Lattice QCD calculations using GPUs. CUDA supports the following fermion formulations: Wilson, Wilson-clover, Twisted mass, Improved staggered (asqtad or HISQ) and Domain wall. Molecular Dynamics: NAMD - Designed for highperformance simulation of large molecular systems; Source: http: //www. nvidia. com/content/gpu-applications/PDF/GPU-apps-catalog-mar 2015. pdf

Some GPU-accelerated Libraries NVIDIA cu. BLAS NVIDIA cu. RAND NVIDIA cu. SPARSE Vector Signal Image Processing GPU Accelerated Linear Algebra Matrix Algebra on GPU and Multicore IMSL Library Building-block Algorithms for CUDA Sparse Linear Algebra NVIDIA NPP NVIDIA cu. FFT C++ STL Features for CUDA Source: https: //developer. nvidia. com/cuda-education. (Will Ramey , NVIDIA Corporation)

An Instance of MPI+CUDA Hybrid Programming: GIMM FPEIP Complex GIMM FPEIP : package developed for simulation of thermal processes in materials irradiated by heavy ion beams Alexandrov E. I. , Amirkhanov I. V. , Zemlyanaya E. V. , Zrelov P. V. , Zuev M. I. , Ivanov V. V. , Podgainy D. V. , Sarker N. R. , Sarkhadov I. S. , Streltsova O. I. , Tukhliev Z. K. , Sharipov Z. A. (LIT) Principles of Software Construction for Simulation of Physical Processes on Hybrid Computing Systems (on the Example of GIMM_FPEIP Complex) // Bulletin of Peoples' Friendship University of Russia. Series "Mathematics. Information Sciences. Physics". — 2014. — No 2. — Pp. 197 -205.

GIMM FPEIP : package for simulation of thermal processes in materials irradiated by heavy ion beams To solve a system of coupled equations of heat conductivity which are a basis of thermal spike model in cylindrical coordinate system Multi-GPU

GIMM FPEIP Logical Scheme

MPI calculations HETEROGENEOUS COMPUTATIONS GROUP, LIT

Multi-GPU calculations 1, 43 GPU calculations 7, 31 7, 45 1, 47 1, 48 Size N x NR : 10000 x 512 Z Available memory Fermi C 2050 - 2, 5 GB Is needed for calculations 20000 x 4096 - 5 GB ------------------ Tesla K 40 s - 12 GB 20000 x 8192 - 10 GB HETEROGENEOUS COMPUTATIONS GROUP, LIT

Hybrid Programming: MPI+Open. MP, MPI+Open. MP+CUDA Multi. Configurational Time Dependnet Hartree (for) Bosons Ideas, methods, and parallel implementation of the MCTDHB package: Many-body theory of bosons group in Heidelberg, Germany http: //MCTDHB. org MCTDHB founders: Lorenz S. Cederbaum, Ofir E. Alon, Alexej I. Streltsov Since 2013 cooperation with LIT: the development LIT of new hybrid implementations package The Multi. Configurational. Time. Dependnet. Hartree (for) Bosons method: PRL 99, 030402 (2007), PRA 77, 033613 (2008) It solves TDSE numerically exactly – see for benchmarking PRA 86, 063606 (2012)

All the terms of the Hamiltonian are under control and can be manipulated BECs of alkaline, alkaline earth, and lanthanoid atoms (7 Li, 23 Na, 39 K, 41 K, 85 Rb, 87 Rb, 133 Cs, 52 Cr, 40 Ca, 84 Sr, 86 Sr, 88 Sr, 174 Yb, 164 Dy, and 168 Er ) Magneto-optical trap The interatomic interaction can be widely varied with a magnetic Feshbach resonance… (Greiner Lab at Harvard. ) 1 D-2 D-3 D: Control on dimensionality by changing the aspect ratio of the trap

Dynamics N=100: sudden displacement of trap and sudden quenches of the repulsion in 2 D ar. Xiv: 1312. 6174 Two generic rgimes: (i) non-violent (under-a-barrier) and Two generic regimes: (i) non-violent (under-a-barrier) and (ii) Explosive (over-a-barrier)

Many-body theory of bosons group Heidelberg, Germany PD Dr. Alexej I. Streltsov UNIVERSITÄT HEIDELBERG Heterogeneous Computations Hybri. LIT - team Dr. Oksana I. Streltsova LIT JINR, DUBNA http: //QDlab. org 26

MCTDHB-Lab is a free-for-download cross-platform software working on Mac, Windows and Linux http: //QDlab. org

Information-computer environment of Hybri. LIT Hardware Software Tools for work on the cluster Accounting: Hybri. LIT site: wwwinfo. jinr. ru/~gvv/SLURM hybrylit. jinr. ru Services Hybri. LIT user support: Indico: LDAP: on JINR Project Management cloud. jinr. ru indico-hybrilit. jinr. ru pm. jinr. ru on Cloud JINR Service HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

Hybri. LIT site: http: // hybrilit. jinr. ru HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

Hybri. LIT cluster: current usage Currently the total number of users comprise 95 people from which 26 are from JINR member-countries: Armenia, Bulgaria, Mongolia, Romania, Slovakia, Ukraine, etc. 19 people are from the following universities of Russia: MSU, SPSU, PFUR and ‘Dubna’ University. Tutorials on the basis of Hybri. LIT: • Regular tutorials on parallel programming technologies both for the institute staff and for students and young scientists from JINR member-states organized by the UC; • Specialized courses from the leading software developers; Specialized courses and seminars within conferences and schools organized by JINR. In particular within GRID’ 2014, International youth conference MPAMCS’ 2014, The Helmholtz International Summer School "Lattice QCD, Hadron Structure and Hadronic Matter" 2014; Mathematical Modeling and Computational Physics MMCP 2015. HETEROGENEOUS COMPUTATIONS TEAM Hybri. LIT

Thank you for your attention! HETEROGENEOUS