Computational Science and Engineering Department Daresbury Laboratory Beyond Teraflops toward Petaflops Computational Chemistry: Challenges and Opportunities Martyn F. Guest and Paul Sherwood CCLRC Daresbury Laboratory m. f. guest@daresbury. ac. uk Beyond Teraflops toward Petaflops 1* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Outline l From Gigaflops to Teraflops - Computational Chemistry Today n n n l Migration from replicated to distributed data Parallel linear algebra (diagonalisation, FFT etc. ) Exploiting multiple length and time scales From Teraflops to Petaflops n Problem scaling and re-formulation - Four Dimensions l Time to Solution, Problem Size, Enhanced Sampling, Accuracy n Long Term - New horizons for simulation l “Simulation of whole systems, and not just system components” l Scientific Challenges in Key Application areas n Catalytic Processes, Biomolecular Simulations, Heavy Particle Dynamics l “Current Status”, “Towards Petaflops”, “New Horizons” l l The Software Challenge Recommendations and Summary Beyond Teraflops toward Petaflops 2* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory How Today’s codes exploit Today’s Hardware NAMD 10 TF Distributed Memory HPCx 1 TF NWChem GAMESS-UK DL_POLY 3 100 GF CHARMM 10 GF Beyond Teraflops toward Petaflops Shared Memory Gaussian 3* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory High-End Computational Chemistry The NWChem Software l Capabilities (Direct, Semi-direct and conventional): n RHF, UHF, ROHF using up to 10, 000 basis functions; analytic 1 st and 2 nd derivatives. n DFT with a wide variety of local and non-local XC potentials, using up to 10, 000 basis functions; analytic 1 st and 2 nd derivatives. n CASSCF; analytic 1 st and numerical 2 nd derivatives. n Semi-direct and RI-based MP 2 calculations for RHF and UHF wave functions using up to 3, 000 basis functions; analytic 1 st derivatives and numerical 2 nd derivatives. n Coupled cluster, CCSD and CCSD(T) using up to 3, 000 basis functions; numerical 1 st and 2 nd derivatives of the CC energy. n Classical molecular dynamics and free energy simulations with the forces obtainable from a variety of sources Beyond Teraflops toward Petaflops 4* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Memory-driven Approaches: NWChem - DFT (LDA) Performance on the SGI Origin 3800 Time (sec) Zeolite ZSM-5 • DZVP Basis (DZV_A 2) and Dgauss A 1_DFT Fitting basis: AO basis: CD basis: 3554 12713 • MIPS R 14 k-500 CPUs (Teras) Wall time (13 SCF iterations): 64 CPUs = 5, 242 seconds 128 CPUs= 3, 451 seconds Est. time on 32 CPUs = 40, 000 secs • 3 -centre 2 e-integrals = 1. 00 X • Schwarz screening = 5. 95 X 10 10 Number of processors • % 3 c 2 e-ints. In core = 100% 10 12 Beyond Teraflops toward Petaflops 5* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Multiple Time and Length Scales l QM/MM - first step towards multiple length scales n QM treatment of the active site l l l reacting centre problem structures (e. g. transition metal centres) excited state processes (e. g. spectroscopy) n Classical MM treatment of environment l enzyme structure, zeolite framework, explicit and/or dielectric solvent models l Multiple time scale algorithms for MD n Recompute different parts of energy expression at different intervals e. g. variants of the Reference System Propagation Algorithm (RESPA) But to date length / time scales only differ by ~ 1 order of magnitude For an example of an effort to link the atomistic and meso-scales see Reality. Grid: http: //www. realitygrid. org/information. html Beyond Teraflops toward Petaflops 6* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory QM/MM Applications Triosephosphate isomerase (TIM) • Central reaction in glycolysis, catalytic interconversion of DHAP to GAP T 128 (O 3800/R 14 k-500) = 181 secs • Demonstration case Measured Time (seconds) within QUASI (Partners UZH, and BASF) • QM region 35 atoms (DFT BLYP) – include residues with possible proton donor/acceptor roles – GAMESS-UK, MNDO, TURBOMOLE • MM region (4, 180 atoms + 2 link) – CHARMM force-field, implemented in CHARMM, DL_POLY Beyond Teraflops toward Petaflops 7* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory From Teraflops to Petaflops l Short Term - Problem Scaling and Re-formulation n Approaches to efficient exploitation of larger systems l Opportunities for more realistic modelling l Need to avoid dependency on continued scaling of existing algorithms n Scientific Targets: Catalysis, Enzymes and Biomolecules, Heavy Particle Dynamics l Long Term - New Horizons n “Simulation of whole systems, and not just system components” n Automated problem solution l l Focus on parallel supercomputers build with commodity compute servers tied by high performance communication fabric New Peta. OPs architecture Projects n IBM’s Blue Light l cellular architecture with 105 or more CPUs; intended to be general purpose n IBM’s Blue Gene l collocation of 32 CPUs, 8 MB RAM on same chip (may scale to 106 CPUs) l application specific, protein folding Beyond Teraflops toward Petaflops 8* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Same model but exploit faster execution More Accurate Methods • Better forcefields • Increased use of ab-initio methods • Higher level QM • QM/MM, DFT replacing semiempirical methods • Finer numerical grids ! Accuracy of methods tends to increase slowly with cost ! Most major challenges involve larger systems • Longer timescale for simulations • Interactive exploration ! Limited scalability of current algorithms ! Long timescales demand higher accuracy • Distributed data methods can exploit large global memories Time “State of the Art” Accuracy Tera-flop computing Larger problems Size ~ 1000 Processors Sampling ! Many algorithms contain serious bottlenecks, e. g. diagonalisation O(N 3) ! Sampling conformational space becomes harder with system size Study many configurations or systems at once • Better Statistics, Free energies • Combinatorial methods • MC Ensembles ! Often satisfied by cheaper, commodity systems Beyond Teraflops toward Petaflops 9* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory What is Petaflop Computational Chemistry l Consider large classical simulations n n n start from a typical 100, 000 particle biomolecule + water cost per timestep ~50 seconds on 1 GF (peak) processor MD simulations of order 0. 5 s will require approx 100, 000 steps Cost 5, 000 Pflops per state point i. e. 1. 5 hours on a Pflop machine Energy Scales as scales as O(N log (N)) and the equilibration time as O(N 5/3) n 1, 000 particle simulation (for 25 us) will take 840 Pflop hour (complex membrane protein) l Quantum Simulations n Assuming a cost of 5 hours on 1 Gflop processor, 1 day on a 1 Pflop resource will simulate 25 nanoseconds of motion, corresponding to the equilibration time of a 10, 000 atom system. Beyond Teraflops toward Petaflops 10* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Modelling of Catalysis 1. Current Status l Tools n Classical simulation l equilibrium structures and transport properties n QM simulations l reactivity of molecular models, surface structures, supercells n QM/MM methods l solvent, ligand lattice effects on local chemistry l Scientific Drivers n From mechanisms to reaction rates n From simplified models to multi-component systems, defect sites Collaboration QUASI (Quantum Simulation in Industry) see http: //www. cse. clrc. ac. uk/qcg/quasi Beyond Teraflops toward Petaflops 11* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Modelling of Catalysis 2. Towards Petaflops Accuracy l Extended dynamical simulation, solvent, counterions etc l Time Advanced forcefields (polarisabilities, cross-terms etc) More extensive use of quantum methods (large DFT clusters and periodic supercells) l l Sampling Combinatorial approach to catalyst formulation Quantum dynamics & tunnelling via path integral methods Finite difference approach to local surface vibrations l ab-initio treatment of larger surface domains l l Size Beyond Teraflops toward Petaflops 12* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Modelling of Catalysis 3. New Horizons Require integrated models spanning time and length scales. n Initially the interface between the length and time scales will be via construction of parametric models n There is also the possibility that lower-level data might be computed on demand as required to sustain the accuracy of the large-scale simulation Reactor geometries Diffusion of reactants and products Heterogeneous surface structures (films etc) Reaction rates Detailed chemical energetics Beyond Teraflops toward Petaflops 13* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Biomolecular Simulation 1. Current Status l Tools n Classical Simulation l Simple but well established forcefields for proteins, nucleic acids, polysacharrides etc n QM and QM/MM l enzyme reaction energetics, ligand binding n Continuum electrostatics l Poisson-Boltzmann, Generalised Born n Statistical Mechanics l Scientific Targets n n n Accurate free energies for more complex systems Faster and more accurate screening of protein / ligand binding Membrane proteins (e. g. receptors) Complex conformational changes (e. g. protein folding) Excited state dynamics Beyond Teraflops toward Petaflops 14* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Biomolecular Simulation 2. Towards Petaflops Time l Faster simulations to approach real timescales n !! major scalability problems l Accuracy Sampling New forcefields incorporating polarisation, cross-terms, etc Increased use of ab-initio methods l Tremendous potential due to importance of free energies. l n Multiple independent simulations n Replica path - simultaneous minimisation or simulations of an entire reaction pathway n Replica exchange - Monte Carlo exchange of configurations between an ensemble of replicas at different temperatures n Combinatorial approach to ligand binding Size l membranes, molecular assemblies … Beyond Teraflops toward Petaflops 15* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Replica Path Methods l Replica path method - simultaneously optimise a series of points defining a reaction path or conformational change, subject to path constraints. P 0 P 1 P 2 P 3 P 4 l l Suitable for QM and QM/MM Hamiltonians P 32 P 33 P 34 P 35 P 36 Parallelisation per point E l Communication is limited to adjacent points on the path global sum of energy function Reaction Co-ordinate Collaboration with Bernie Brooks (NIH) http: //www. cse. clrc. ac. uk/qcg/chmguk Beyond Teraflops toward Petaflops 16* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Biomolecular Simulation 3. New Horizons l Towards full quantum simulation (e. g. Car Parrinello) CCP 1 Flagship project - Simulation of Condensed Phase Reativity: http: //www. ccp 1. ac. uk/projects. shtml l Towards Whole cell simulation n Mechanical deformation, Electrical behaviour n Diffusion of polymeric molecules (e. g. neuro-transmitters) by DPD n Nanoscale models for supra-molecular structures (e. g. actin filaments in muscles) n Atomistic Molecular Dynamics n Quantum chemistry of reacting sites Beyond Teraflops toward Petaflops 17* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Heavy Particle Dynamics 1. Current Status l l Tools n Many methods require evaluation of energies on a massive multidimensional grid l Use high-level computational chemistry methods (e. g. NWChem, MOLPRO), together with task farming. n Complex parameter fitting l Can exploit interactivity (INOLLS) incorporating experimental data n Dynamical Simulation Methods l Wavepacket evolution on a Grid l Classical path methods (multiple direct dynamics trajectories) l Variational solutions for spectroscopic levels Targets n Larger species (5 atoms and beyond) n Influence of surrounding molecules (master equation) CCP 6 Collaboration, Chem. React consortium on national HPC facilities Beyond Teraflops toward Petaflops 18* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Heavy Particle Dynamics 2. Towards Petaflops l Time l l Accuracy l Interactivity in potential energy surface fitting. Longer time simulations (slower reactions) PE surfaces from large basis set CCSD(T) etc MRCI for excited states and couplings l Sampling Large grids for higher dimensionality systems (5 atoms) Multiple coupled PE surfaces (involvement of excited states) J > 0 - additional angular momentum states l Larger grids for wavepacket evolution l l Size Beyond Teraflops toward Petaflops 19* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Heavy Particle Dynamics 3. New Horizons l Integration with Combustion, detonation, atmospheric models, most likely through detailed parameter and rate constant derivation CFD simulation (transient or steady state, turbulence models etc) Reduced chemical kinetics Reaction rates, sensitivity to pressure and temperature Heavy particle dynamics Beyond Teraflops toward Petaflops 20* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory The Software Challenge l l Lack of sustained focus within the Chemical Sciences for responding to the challenges of Petascale Computing The impact of providing such a focus periodically demonstrated at points on the road to Terascale Computing n The HPCC Grand Challenge projects n NWChem - DOE (PNNL) - principally Electronic Structure; n NAMD - NIH funded initiative in bio-molecular sciences and Classical MD. l Such Initiatives demand: n The successful integration of multi-disciplinary teams including Application and Computational scientists, Computer Scientists and Mathematicians; n A long term commitment to the challenge, with funding in place to respond to the inevitable pace of architecture / hardware change. l Relying on the efforts of individual groups to overcome this software challenge will not work. Beyond Teraflops toward Petaflops 21* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Problem Solving Environments Requirement: A comprehensive problem solving environment (PSE) for molecular modeling and simulation. Key components include: • • Beyond Teraflops toward Petaflops 22* common graphical user interfaces scientific modelling management seamless transfer of information between applications persistent data storage integrated scientific data management tools for ensuring efficient use of computing resources across a distributed network i. e. GRID visualization of multi-dimensional data structures 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Summary and Recommendations 1. Computational chemistry on Tera-scale resources “needs work”, but there are plenty of opportunities to advance collaboratively chemical sciences “towards petaflops” n The short term priority is scaling and adapting current methodologies l Advancing use of distributed data algorithms l O(N) techniques to remove bottlenecks and enhance scalability l Detailed work on parallel scaling – Library developments – Performance analysis and prediction tools l Re-formulation of problems in terms of more weakly interacting ensembles l Parallel implementations of more complex physical models l Automation, data handling, PSEs for combinatorial work n In the longer term by tackling more integrated problems l Modularity of software l Science of the time / length scale interfaces Beyond Teraflops toward Petaflops 23* 2 October 2002
Computational Science and Engineering Department Daresbury Laboratory Summary and Recommendations 2. l Investment in Software: “Code Sharing” n UK has kept a strong applications focus, but has lagged behind the US in the radical re-design of simulation packages l l Sustained investment in Petascale Software Development Current UK and International collaborations n Scalable QC algorithms l NWChem, MOLPRO, GAMESS-UK (PNNL, ORNL, SDSC, DL, CCP 1/5/6); n Replica path methods in CHARMM/GAMESS-UK l DL collaboration with NIH, PSC; n Flexible QM/MM models incorporating classical polarisation l Chem. Shell / GULP / GAMESS-UK / NAMD; n Distributed data classical models, electrostatic models l DL_POLY, NWChem – CCP 1/5. l Stronger Links between UK initiatives and US programs Sci. DAC and NSF PACI, NPACI etc. Beyond Teraflops toward Petaflops 24* 2 October 2002
Скачать презентацию Computational Science and Engineering Department Daresbury Laboratory Beyond
3dfa87c85f6d06a61254ee874e28bbd7.ppt