30 Years of Navy Modeling and Supercomputers an

30 Years of Navy Modeling and Supercomputers: an Anecdotal History Tom Rosmond Marine Meteorology Division Naval Research Laboratory Monterey, California USA

Outline • Before my Time: 1961 -1974 • My Early Years: 1974 -1980 • The Golden Years of NWP: late 1970’s-early 1980’s • Up to the Present • Conclusions and Thoughts for the Future

Before my Time: 1961 -1974 THE BEGINNING • 1959: Capt Wolff sets up shop on NPS Campus, Monterey CDC 1604: Seymour Cray’s first design with CDC (first supercomputer? ) • 1961: Establishment of Fleet Numerical Weather Facility (FNWF) • 1964: Routine dissemination of numerical products Successive correction analyses of SLP, upper air heights NH Barotropic and thickness advection models

Before my Time: 1961 -1974 CDC 6500’s and NHPE • 1967: FNWC procurement of CDC 6500 to replace 1604 Dual-processors, comparable to 386 microprocessor in FP performance Supercomputer of time was Seymour Cray’s CDC 6600 (NMC in 1966) • 1969: Second 6500 acquired • 1970: Northern Hemisphere Primitive Equation (NHPE) model operational on 4 processors. World’s first multi-processor production code

Before my Time: 1961 -1974 4 -processor 6500 NHPE • Developed by P. Kessel and F. Winninghoff • Processors shared model data through extended core storage (ECS) • OS modifications to allow processor synchronization • Similarities to both shared memory (Open. MP) and distributed memory (MPI) programming models • Parallel efficiency = ~ 75% • An impressive achievement that is certainly underappreciated • An example of the struggle it was to fit models onto these early systems

My Early Years: 1974 -1980 • 1974: I joined the Environmental Prediction Research Facility (EPRF)1 Little involvement in FNWC 2 model development Access of FNWC computer systems (6500’s, 1604) • 1976: NEPRF Numerical Modeling Department formed at request of FNWC CO Capt. R. Hughes He realized that FNWC was unable to maintain R & D continuity needed for NWP system development. Need for a global NWP capability was major motivation, and he was committed to getting computer system to support it. • 1976: FNWC acquired CDC CYBER 175 (similar to CDC 7600) 1 My organization: EPRF 2 Fleet Numerical: FNWF NEPRF FNWC NOARL NRL-Monterey FNOC FNMOC

My Early Years: 1974 -1980 NEPRF Global NWP System Development • 1976: Navy Operational Global Atmospheric Prediction System (NOGAPS) UCLA General Circulation Model Barnes Successive Corrections Global Analysis Variational Initialization with Balance Equation Constraint • 1977: We also had spectral model dynamical core UCLA GCM physics Sat on shelf for several years in favor of UCLA GCM based system • 1980: Prototype NOGAPS running on CYBER 175

My Early Years: 1974 -1980 • 1977: Benchmarking for FNWC procurement NCAR Cray-1, first system outside Los Alamos? UCLA GCM 2. 40 x 3. 00, 6 levels First introduction to vector programming Crude compiler + non-vectorizable code = poor performance Overly conservative target performance, CYBER 203 could compete Allowed subsequent CDC success with CYBER 205 • 1980: CYBER 203 delivered to FNOC Heroic work by CDC to get UCLA model to run fast • 1982: CYBER 203 replaced with CYBER 205

The Golden Years of NWP 1 st half: late 70’s – mid 80’s • Supercomputers were cheapest computers you could buy (price/performance) Cray: Cray-1, XMP, etc CDC: CYBER 203/205, ETA-10 IBM: 360 -195 VP • Development of global NWP forecast systems Dominance of spectral models Increased realization of importance of data assimilation • Establishment of ECMWF “Raised the stakes” in operational NWP Accelerated progress Provided gold standard

The Golden Years of NWP Navy operational models NOGAPS UCLA-GCM - 2. 40 x 3. 00 x L 9 : 1 -pipe/8 Mbyte C 205 : ~25 min/fcstday Spectral – T 47 L 18 : 2 -pipe/32 Mbyte C 205 : ~6 min/fcstday Spectral – T 79 L 18 : 2 -pipe/32 Mbyte C 205 : ~25 min/fcstday (32 bit) Spectral – T 79 L 24 : 4 -pipe/64 Mbyte C 205 : ~12 min/fcstday (32 bit) NORAPS (developed by Rich Hodur, NEPRF) Globally relocatable regional model 6 -8 areas run operationally Varying resolution, domain and grid sizes e. g. ran over South Atlantic during Falklands war (1980)

Some special comments about the CYBER 205 • CDC aggressively pursued meteorology market, both operations and research FNOC, NMC, UKMO, GSFC, GFDL, etc CRAY systems were better general purpose, but 205 excelled on our applications • Initially 205 was difficult to use, compiler/user software was rudimentary, but Language extension showed how machine worked Rich array of exotic vector hardware instructions • System software matured 32 bit/64 bit floating point support Easy mixing of Fortran and explicit vector instructions, we could get “close to the hardware” Spectacular percentage of peak performance possible

The Golden Years of NWP 2 nd half: mid 80’s – early 90’s • Introduction of multi-processor systems Divide and conquer X/MP, Y/MP, ETA-10 Multi-tasking/vectorization programming model (parallel/vector) Spelled the end of single processor supercomputers, e. g. CYBER 205 • Important changes in technology Price/performance advantage of supercomputers ending Introduction of desktop workstations Many people didn’t need supercomputers, just cheap computing Shrinking supercomputer market

Up to the Present: early 1990’s • 1990: Introduction of CRAY C 90 The “best” supercomputer ever? Run by a higher percentage of NWP operational/research centers than any system before or since. Parallel/vector programming model very user friendly Easy to get high percentage of peak performance • 1991: 8 processor C 90 at FNMOC NOGAPS: T 159 L 24 – 6 processors : 10 min/fcstday • 1996: 8 and 16 processor C 90’s at FNMOC NOGAPS: T 159 L 24 – 12 processors : 6 min/fcstday • 1997: COAMPSTM replaced NORAPS as Navy’s regional forecast system

Up to the Present Mid to Late 1990’s • Beginnings of dramatic changes in supercomputer industry Scalable commodity based architectures appearing, e. g. CRAY T 3 E Powerful workstations replacing supercomputers for many applications • End of domination by American supercomputer vendors CDC/ETA were long gone CRAY sold to SGI MTTB (mean time to bankruptcy) very short for new companies • Rise of Japanese vendors to dominate NWP marketplace, at least outside U. S. Fujitsu NEC

Up to the Present 2000 to today • Proliferation of commodity based, scalable architectures for NWP applications Many T 3 E’s still in use SGI: Origin 2000, Origin 3000 IBM: SP 3, SP 4 Linux clusters are also viable alternatives, especially when price/performance is overriding issue • But, rumors of the demise of vector architectures are exaggerated 2000: NEC only vendor with traditional vector architecture, SX-6, but 2002: Resurrected CRAY, Inc introduced X-1 Price/performance vs ultimate performance is central question concerning long-term prospects of these systems

Up to the Present 2000 to today • 2001: FNMOC replaced C 90’s with O 3000’s 1152 total processors NOGAPS: T 239 L 30 – 120 processors: 6 min/fcstday COAMPSTM : 8 -11 areas – 40 -180 processors: 10 -30 min/fcstday • 2003: Operational NAVDAS/NOGAPS (3 DVAR): 60 processors • 2004: Direct radiance retrievals with NAVDAS • 2004: Currently under development at NRL, Monterey T 479 L 54 semi-Lagrangian NOGAPS NAVDAS-AR: 4 DVAR extension of NAVDAS COAMPSTM/WRF: more areas, higher resolution • Clearly computational requirements are never satisfied!!!

Conclusions and thoughts for the future • Vector architectures will continue to be viable candidates for NWP applications Sharing some features with commodity systems, e. g. caches When ultimate performance overrides price/performance Will be part of heterogeneous computing environments • Scientific computing ( and therefore NWP) is now niche market Be thankful that video game applications share many of our requirements Consumer based industry has driven some hardware costs to astonishingly low levels, e. g. hard drives • We must work with vendors to ensure that our requirements are not forgotten Fortran compilers High-performance interconnects