Computing Requirements for km-Scale Climate Modeling Jim Kinter

Computing Requirements for km-Scale Climate Modeling Jim Kinter World Modeling Summit for Climate Prediction Reading, UK 8 May 2008

Science Requirements § Accuracy - drives resolution, hence computational demand

Hurricane Katrina Intensity at Landfall 29 Aug 2005 14 Z 4 km WRF, 62 h forecast Mobile Radar Courtesy of P. Fox (NCAR)

Longer Runs

Weather Ensembles Courtesy of T. Palmer (ECMWF)

Climate Ensembles CMIP 3 archive at PCMDI:

Feedbacks: need entire physical and biogeochemical Earth System

Need: Computational laboratory to address science challenges: • Dramatically expand scale content and mechanism content of complex systems. • Provide access to previously unexplored parameter regimes. • Build ensembles of runs to measure uncertainty and populate the PDF. • Permit inclusion and coupling of additional physical, chemical, and biological processes.

Resources Tradeoffs Resolution EO, Data Assim. xity e ompl C Computing Resources Dur atio n and /or E nse mb le size

Resolution Computing Resources ty xi mple Co Dur atio na nd/ or E nse mb le size

Computing Realities (it doesn’t take a genius …) § Computing capability has increased by O(109)X in 45 years § Partly due to Moore’s Law (how much longer will it be the “law”? ) § Partly due to exploiting parallelism § To continue on an exponential growth curve, with fixed or slowly increasing clock cycles, will require exponential growth in the number of simultaneous threads § Exponential growth in threads means O(104) cores today, O(105) cores in 3 years, and O(106 -107) cores in 5 -8 years § With current HEC technology, O(106) cores means hectares of space, O(102) megawatts of power, and O(106) BTU/hr of A/C § With current memory configurations, O(106) cores means O(1015) bytes (petabytes) of memory

Model Grid Size (km) & Computing Capability Peak Rate: 10 TFLOPS 100 TFLOPS 1 PFLOPS 100 PFLOPS Cores 1, 400 (2005) 12, 000 (2007) 80 -100, 000 (2009) 300 -800, 000 (2011) 6, 000? (20 xx? ) Global NWP 0: 5 -10 days/hr 18 - 29 8. 5 - 14 4. 0 - 6. 3 1. 8 - 2. 9 0. 85 - 1. 4 Seasonal 1: 50 -100 days/day 17 - 28 8. 0 - 13 3. 7 - 5. 9 1. 7 - 2. 8 0. 80 - 1. 3 Decadal 1: 5 -10 yrs/day 57 - 91 27 - 42 12 - 20 5. 7 - 9. 1 2. 7 - 4. 2 120 - 200 57 - 91 27 - 42 12 - 20 5. 7 - 9. 1 Climate Change 2: 20 -50 yrs/day * Core counts above O(104) are unprecedented for weather or climate codes, so the last 3 columns require getting 3 orders of magnitude in scalable parallelization Range: Assumed efficiency of 10 -40% 0 - Atmospheric General Circulation Model (AGCM; 100 vertical levels) 1 - Coupled Ocean-Atmosphere-Land Model (CGCM; ~ 2 X AGCM) 2 - Earth System Model (with biogeochemical cycles) (ESM; ~ 2 X CGCM) Thanks to Jim Abeles (IBM)

Earth Simulator IBM Blue Gene/L

via David Randall

Need: Attention to many aspects of high-end computing • Computing hardware - architecture, data storage and archival, networking etc. • Software - OS, compilers, data management and visualization tools • Power, cooling and infrastructure • Human resources to support multiple models and modeling groups at remote locations

Central Idea • Model and use noise susceptible switches as architecture building blocks – Behavior of such a switch is probabilistic yielding Probabilistic CMOS (PCMOS) • Use PCMOS building blocks to realize probabilistic primitives of probabilistic applications – Non-deterministic behavior of devices is useful, not harmful – Orders of magnitude improvement in energy and performance – High quality of randomization Noise Models Tunneling Power supply Thermal Cross-talk Integrated noise modeling Probabilistic Switch PCMOS Primitives Inverter, Logic gates, Decoders, Multiplexers, DSP filters, etc. Probabilistic So. C RISC Processor PCMOS Co-processor Courtesy of Krishna Palem, Rice University (Apr’ 08)

Need: Effort across all components of simulationbased research • Application development (e. g. grids, algorithms), porting, optimization for target architecture • Execution rate of simulation - dealing with Amdahl’s Law (load balancing etc. ) • Data management - mass storage access and post-processing support for exabyte datasets • Visualization

Need: Effort across all components of simulationbased research • Application development (e. g. grids, algorithms), porting, optimization for target architecture • Execution rate of simulation - dealing with Amdahl’s Law (load balancing etc. ) • Data management - mass storage access and post-processing support for exabyte datasets • Visualization

Weather & Climate Modeling Data Requirements § Coupled ocean-atmosphere-land model § O(102) vertical levels in ocean and atmos. (each) § O(1 km) grid spacing - O(108) columns § Sub-sample at run time to, say, 1% - O(106) columns § O(103) 2 D grids (variables) § Save data O(103) times per run § Half-hourly for weather prediction § 4 X/day for seasonal prediction § Monthly for climate simulation § Replication § O(102) ensemble members § O(103) cases for NWP (3 years), seasonal (50 years)

With Petaflop Computing Comes Exabyte Data Volume § O(1011) bytes per save … means … § Terabytes of I/O per save (similar for reads if doing DA) § O(103) save times § O(105) replications (cases X ensembles) of the data § … means … § O(1019) bytes or O(10) exabytes per model § Distribution - O(10) modeling groups worldwide Need to plan for O(102) exabytes data volume distributed across multiple data centers worldwide

Inevitabilities? § Exabyte data volumes by 2015 § With quasi-linear WAN bandwidth growth, the networks cannot keep pace § The trend is away from centrally designed, implemented and maintained systems toward integration of independently designed, implemented and maintained system elements § It may be inevitable that § data will be distributed § automated data discovery will be indispensable § sophisticated subsetting, on-demand processing and visualization will be necessary § may need to consider on-the-fly processing/synthesis

Summary • Considerable progress in weather & climate modeling over the past 45 years along with a billion-fold increase in computing • Breakthroughs in the next decade will require huge increases in model resolution & complexity 103+ X increase in computing capability along with work on entire spectrum of issues in highend computing and model & code development