Status Review of NOAA Implementation of ESMF Architecture

Status Review of NOAA Implementation of ESMF Architecture (i. e. NEMS) Mark Iredell May 2008 • NEMS People • NEMS Projects • NEMS Issues

NEMS People Biweekly UMIG (Unified Modeling Infrastructure Group) meeting • • • Tom Black Atm, NAM Huiya Chuang Post Ed Colon Infrastructure Mike Ek Land Jim Geiger Land (NASA) Bob Grumbine Sea ice Henry Juang Atm, Dyn, GFS Young Kwon Hurricane Sujay Kumar Land (NASA) Sarah Lu AQ Avichal Mehra Ocean • S. Moorthi Atm, Phy, GFS • Ken Mitchell Land • Christa Peters Land (NASA) • Youhua Tang AQ • Hendrik Tolman Wave • Ratko Vasic Infrastructure • Jun Wang Infrastructure • Xingren Wu Ice, Coupling • Weiyu Yang Infrastructure • Mike Young Infrastructure • Shujia Zhou Land (NASA) • Yanqiu Zhu GSI 2

NEMS People Monthly UMIT (Unified Modeling Infrastructure Telecon) meeting Includes local UMIG group plus… • V. Balaji GFDL • Brian Gross GFDL • Niki Zadeh GFDL • • Stan Benjamin GSD Tom Henderson GSD Jin Lee GSD Jacques Middlecoff GSD Occasionally the telecon invites a much wider group, including DTC and Navy. 3

NEMS Projects • • • ESMF NEMS atmosphere Write history and Post processor Nesting Aerosols and Chemistry Land Ocean, waves and sea ice Ionosphere Ensemble Data assimilation

ESMF • Earth System Modeling Framework • Community effort, partially supported by NOAA • Proper ESMF Superstructure required for all NEMS components • ESMF Infrastructure optional • NEMS will require ESMF 3. 1. 0 r at this time

An ESMF “Component” “Parent” Driver (User Code) ESMF library code Import State Export State Run() Finalize() Run() Initialize() Set. Services() An ESMF Component Data flow in an ESMF Component Run method Internal State 6

ESMF Component rules • Only Set. Services is public; Initialize, Run, and Finalize are private but exposed by Set. Services. • Import State is input; Export State is output. Both should be fully described using ESMF metadata (names, grid, decomposition, etc). • Coupler Component likely has Import and Export States on different grids. A Coupler Component is user code. • Internal State is private to the component but persists across calls to Initialize, Run, and Finalize. Even other instances of the same component will have their own private internal state. An access exception may be made to other “friendly” components.

ESMF Infrastructure capabilities ESMF offers or will offer capabilities for decomposed halo update and transposes as well as extensive parallel regridding tools (among other capabilities). The user is free to make use of these or not. Thus far NEMS has taken advantage of existing in-house capability and has minimally made use these ESMF capabilities. This is expected in such transition projects, and ESMF capability will be used as needed.

ESMF Portability, Conventions • ESMF is designed to run on most systems Earth modeling runs on. Plus, ESMF infrastructure will support some ordinary functions that otherwise often hinder portability. • ESMF supports but does not require the Climate. Forecast metadata conventions. Should NEMS adopt them? – Pro: wide community uses it, including NOAA labs – Con: very long names, but we can make local aliases

CF convention examples from http: //cf-pcmdi. llnl. gov/documents/cf-standard-names/ncep-grib-code-cf-standard-name-mapping 10

NEMS Component Library • Create a NEMS component library (in SVN). • Each component will have some documentation. Moreover, each component should have a sample MAIN to run it in stand-alone mode for testing. For instance, Dynamics components may have Held-Suarez, Physics may have a single-column model. • Each type of component will have an assigned NCEP librarian to coordinate the sub-library.

NEMS Atmosphere Color Key Component class Atmosphere Coupler class unified atmosphere including digital filter Completed Instance Under Development Future Development Dynamics ARW NMM-B FVCS Spectral NOGAPS FIM COAMPS FISL Dyn-Phy Coupler Simple Regrid, Redist, Chgvar, Avg, etc Physics NAM Phy GFDL Phy GFS Phy WRF Phy Navy Phy • The goal is one unified atmospheric component that can invoke multiple dynamics and physics. • At this time, dynamics and physics run on the same grid in the same decomposition, so the coupler literally is very simple. 12

Original FIM run() Call Order n FIM run: do timestep = first, last dyn_1 ! 1 st half of dyn calls call physics() dyn_2 ! 2 nd half of dyn calls end do n Must re-order operations without changing model results 13 Slide courtesy Stan Benjamin

FIM run() Call Re-Ordering n Re-order time stepping loop: do timestep = first, last+1 if (timestep > first) & dyn_2(timestep-1) if (timestep <= last) then dyn_1(timestep) call physics(timestep) endif end do 14 Slide courtesy Stan Benjamin

New FIM run() Call Order n Combine dyn_1 + dyn_2 into dyn_run and push “if” statements inside n Push physics() and if statement into phy_run do timestep = first, last+1 call dyn_run() call phy_run() end do MI: Note NMM and GFS have gone through the exact same process. 15 Slide courtesy Stan Benjamin

Write history and Post processor Color Key Component class Coupler class Atmosphere Completed Instance unified atmosphere Under Development Future Development Dynamics Physics Write Binary file NEMSIO Post • The Write component transfers state from model tasks to “quilt” tasks and then writes out from there. • NEMSIO is intended to be a unified optimized parallelized I/O package that can write several formats. Ideally, NEMSIO would be used for all NEMS output data and metadata (not necessarily restart data). • The unified post-processor NCEP_POST will run on the quilt tasks. 16 • Perhaps in the future, the Dynamics and Physics will write their own history files.

Nesting Color Key Component class Atmosphere Coupler class Completed Instance Atmosphere 1 -way conc. NAM-NAM 1 -way conc. GFS-NAM Atmosphere Under Development Future Development Atmosphere Atmosphere 2 -way seq. 2 -way conc. moving nest • Parent creates children and creates proper boundary conditions in their import states. • Recursively, children create grandchildren. • Children run on different tasks from parent in concurrent nesting, same tasks in sequential nesting. 17 • Two-way concurrent nesting would require a different time integration scheme that needs to be tested.

NEMS Nesting • Eventually we will have all forms. ► ► Static / Moving. ► • One-way / Two-way. Grid-associated / Not grid-associated. Begin with 1 -way grid-associated static nests. ► The parent domain can have any number of children. ► Telescoping: Children can have any number of children. ► Domains can run concurrently on unique sets of processors. These three criteria require the need for general and repeated splitting of the MPI Communicators. --Done 18 Slide courtesy Tom Black

Nesting – ATM_RUN(Fcst_time) DO Timeloop over Fcst_time at interval Δtinternal “Mini” timeloop IF(My_Compute_Tasks) PREDICT(Δtinternal) IF N_CHILDREN>0 DO N= 1, N_CHILDREN IF(My_Compute_Tasks) ISend BC’s to child IF(Child_Compute_Tasks(N)) Recv from parent CALL ATM_RUN(Child(N), Fcst_time=Δtinternal) Recursive call ENDDO IF(Two-way) Send children’s state back to me and blend into mine ENDIF ENDDO END ATM_RUN 19 Slide courtesy Tom Black

Aerosols and Chemistry Color Key Component class Atmosphere Coupler class Completed Instance Dynamics Physics Aerosols GOCART Under Development Future Development CBM-5 reduced chem WRFchem • GOCART is Goddard aerosol model. CBM is VOC-NOX-O 3 mechanism (chemistry in CMAQ model). • Aerosol and chemistry components compute sources, sinks, and chemical transformation. • Aerosol and chemistry components may need to expose internal state to avoid memory copies. • Atmosphere must be ready to do convective fluxes and removal as well as advection and diffusion. 20

Land (alternative 0) Color Key Component class Coupler class Atmosphere Completed Instance Under Development Dynamics Physics Land Future Development LIS-Noah others • Land is responsible for computing surface fluxes. • Land may need to be invoked inside physics, since it needs radiation and surface layer needs it. • Implicit solving would require iteration between Land PBL/Moist. • How deep should ESMF go? 21

Land (alternative 0 b) Main Color Key Component class Atmosphere Land LIS-Noah Dynamics Physics others Coupler class Completed Instance Under Development Future Development • Land is responsible for computing surface fluxes. • Land may need to be invoked inside physics, since it needs radiation and surface layer needs it. • Implicit solving would require iteration between Land PBL/Moist. • How deep should ESMF go? 22

Land (alternative 1) Color Key Component class Coupler class Atmosphere Completed Instance Under Development Dynamics Radiation Land PBL/Moist Future Development LIS-Noah others • Land is responsible for computing surface fluxes. • Land may need to be invoked inside physics, since it needs radiation and surface layer needs it. • Implicit solving would require iteration between Land PBL/Moist. • How deep should ESMF go? 23

Land (alternative 1 b) Color Key Component class Coupler class Atmosphere Completed Instance Under Development Dynamics Land PBL/Moist Radiation Future Development LIS-Noah others • Land is responsible for computing surface fluxes. • Land may need to be invoked inside physics, since it needs radiation and surface layer needs it. • Implicit solving would require iteration between Land PBL/Moist. • How deep should ESMF go? 24

Land (alternative 2) Color Key Component class Coupler class Atmosphere Completed Instance Under Development Dynamics Radiation Physics Land Future Development PBL/Moist Aerosols/ Chemistry • Land is responsible for computing surface fluxes. • Land may need to be invoked inside physics, since it needs radiation and surface layer needs it. • Implicit solving would require iteration between Land PBL/Moist. • How deep should ESMF go? 25

Land (alternative 3) Color Key Component class Coupler class Atmosphere Completed Instance Under Development Dynamics Radiation Land Physics Surface Ice Future Development PBL/Moist Waves Aerosols/ Chemistry Ocean • Land is responsible for computing surface fluxes. • Land may need to be invoked inside physics, since it needs radiation and surface layer needs it. • Implicit solving would require iteration between Land PBL/Moist. • How deep should ESMF go? 26

GEOS-5 AGCM COMPONENT STRUCTURE CAP HISTORY AGCM COLUMN PROCESSES OGCM (qv, ql, qi, cl, …) MOIST RADIATION CHEM SOLAR IR (O 3, …, Qdust, …) TURB SURFACE LAND ICE (Ts, Fi. . . ) CATCH (Tc, qc, Td, . . . ) VEGDYN (Ci, . . . ) LAKE (Ts, Fi. . . ) DYNAMICS FVCORE GWD (u, v, T, p) SALTWATER (Tskin, Hskin, . . . ) Slide courtesy Max Suarez

Ocean, waves and sea ice Color Key Component class Earth system model Coupler class Atmosphere GFS NEMS-GFS Atm-Ocn Coupler Completed Instance Waves Ice Ocean WW III H-ice MOM 4 N-ice HYCOM Under Development Future Development NCEP CFS non-ESMF MPMD GFS Atm-Ocn Coupler MOM 4 • Under ESMF NEMS, there should be not much difference between SPMD and MPMD codes. However, the MPMD code might not be as portable. MPMD will allow more flexible optimizations though. • Due to some rapidly interacting physics (sea ice thermodynamics and wave roughness), some coupling 28 would have to be frequent.

Ocean, waves and sea ice (2) Color Key Component class Earth system model Coupler class Atmosphere GFS Atm-Sfc Coupler (Fast/Slow) Completed Instance Surface (Fast/Slow) Under Development Future Development Land NEMS-GFS Waves Ice Ocean WW III H-ice MOM 4 N-ice HYCOM NCEP CFS non-ESMF MPMD GFS Atm-Ocn Coupler MOM 4 • Under ESMF NEMS, there should be not much difference between SPMD and MPMD codes. However, the MPMD code might not be as portable. MPMD will allow more flexible optimizations though. • Due to some rapidly interacting physics (sea ice thermodynamics and wave roughness), some coupling would have to be frequent and possibly iterative. The fast land, ice, and waves components may need to 29 run serially on the atmospheric component’s processors.

Ocean, waves and sea ice (3) Color Key Component class Earth system model Coupler class Atm-Sfc Coupler (Slow) Atmosphere Dynamics Completed Instance Surface (Slow) Under Development Future Development Physics Etc. Rad. Waves Surface (Fast) Ice Turbul ence Ocean Waves Ice Ocean Land • Due to some rapidly interacting physics, some coupling would have to be frequent and possibly iterative. The fast mode components may need to run serially on the atmospheric component’s processors. • The fast and slow “modes” of model components may have different import and output requirements, 30 and notably different VMs, but may share internal states.

processor t i m e current CFS load-balance scenario assuming 3 fast timesteps for every 1 slow timestep AM C OM Atmosphere i Fast ice d idle le Two-way coupling Atmosphere Two-way coupling i Fast ice d Slow ice le Ocean

processor t i m e alternate CFS load-balance scenario assuming 3 fast timesteps for every 1 slow timestep AM C OM Fast ice i Slow ice d le Atmosphere Ocean Two-way coupling Fast ice Atmosphere i d le Two-way coupling Fast ice Atmosphere Two-way coupling i d le

alternate Earth system model load-balance scenario processor t i m e assuming 3 fast timesteps for every 1 slow timestep AM Fast ice C OM Fast land Fast waves Slow ice Sfc-Atm coupling Slow land Atmosphere Slow waves Ocean Atm-Sfc coupling Fast ice Fast land Fast waves Sfc-Atm coupling Atmosphere Atm-Sfc coupling Fast ice Fast land Sfc-Atm coupling Atmosphere Two-way coupling Fast waves

Ionosphere Color Key Component class Whole atmosphere Coupler class Completed Instance Atmosphere GFS-IDEA Atm-Ion Coupler Ionosphere TBD GIP Under Development Future Development • GFS-IDEA is the GFS extended up to 600 km (from 60 km). GIP is Global Ionosphere-Plasmasphere • The coupling will be frequent and three-dimensional, providing optimization challenges. 34

Ensemble Color Key Component class Ensemble Coupler class Completed Instance Atmosphere Atm ens Coupler Atmosphere GFS Under Development Future Development NEMS-GFS NEMS-any • The atmosphere ensemble coupler provides stochastic forcing for the ensemble. • The current non-NEMS GEFS capability will be converted to NEMS. 35

Data assimilation Color Key Data assimilation Component class Coupler class Model GFS Mdl-Anl Coupler Analysis GSI Completed Instance Under Development Future Development NAM Whole Earth System Nested Ensemble • Frequent coupling of atmosphere and analysis will be necessary as hourly analyses are required. • Whole Earth System Nested Ensemble may include any components in NEMS from previous slides. • 4 DVAR is supported in this scenario. The inner loop of the variational system could be entirely within the analysis component, or it could invoke adjoint methods with the NEMS atmosphere. • The model-analysis coupler may change grids and variables. When returning to the model state, only the analysis increments will be interpolated. 36

NEMS Issues • ESMF version – NEMS will require ESMF 3. 1. 0 r, which will be the latest public release in spring 2008. NEMS may later need ESMF 3. 1. 1. – All components must use the same version • MAPL – GSFC ESMF wrapper – Used by MOM 4, GOCART, FVCORE – Compatible with NEMS? • Code Repositories – Operationally secure – Collaboration friendly • Experiment Launcher compatible with operations • Documentation and Support