Earth System Modeling Infrastructure Cecelia De Luca ESMF-NCAR March

Earth System Modeling Infrastructure Cecelia De. Luca/ESMF-NCAR March 31 -April 1, 2009 CHy. MP Meeting www. esmf. ucar. edu

Outline • • • Elements of interoperability platforms Integrating across elements Summary www. esmf. ucar. edu

Elements of interoperability platforms 1. Tight coupling tools and interfaces - hierarchical and peer component relationships - frequent, high volume transfers on high performance computers 2. Loose coupling tools and interfaces - generally peer-peer component relationships - lower volume and infrequent transfers on desktop and distributed systems 3. Science gateways - browse, search, and distribution of model components, models, and datasets - visualization and analysis services - workspaces and management tools for collaboration 4. Metadata conventions and ontologies - ideally, with automated production of metadata from models 5. Governance - coordinated and controlled evolution of systems www. esmf. ucar. edu

Tight coupling tools and interfaces Examples: • Earth System Modeling Framework (ESMF) - NASA, NOAA, Department of Defense, community weather and climate models, U. S. operational numerical weather prediction centers (HPC focus) http: //www. esmf. ucar. edu • Flexible Modeling System (FMS) – NOAA precursor to ESMF, still used at the Geophysical Fluid Dynamics Laboratory for climate modeling http: //www. gfdl. noaa. gov/fms/ • Space Weather Modeling Framework (SWMF) – NASAfunded, used at the University of Michigan for space weather prediction www. esmf. ucar. edu

How coupling tools work: • Users wrap their native data in framework data structures • Users adopt standard calling interfaces for a set of methods that enable data exchange between components • Development toolkits help users with routine functions (regridding, time management, etc. ) www. esmf. ucar. edu

ESMF: Standard interfaces • Three ESMF component methods: Initialize, Run, and Finalize (I/R/F) • Each can have multiple phases • Users register their native I/R/F methods with an ESMF Component • Small set of arguments: call ESMF_Grid. Comp. Run (my. Comp, import. State, export. State, clock, phase, blocking. Flag, rc) www. esmf. ucar. edu

ESMF: Distributed data representation 1. Representation in index space (Arrays) • • • Simple, flexible multi-dimensional array structure Regridding via sparse matrix multiply with user-supplied interpolation weights Scalable to 10 K+ processors - no global Supported Array distributions information held locally 2. Representation in physical space (Fields) • Built on Arrays + some form of Grid • Grids are: logically rectangular, unstructured mesh, or observational data streams • Regridding via parallel on-line interpolation weight generation, bilinear or higher order options • Intrinsically holds significant amounts of metadata dynamic, usable for multiple purposes, limited annotation required www. esmf. ucar. edu

ESMF: Coupling options • Generally single executable for simpler deployment • Push mode of data communication is very efficient • Coupling communications can be set up and called in a coupler, or called directly from within components (for I/O, data assimilation) • Hierarchical components for organization into sub -processes • Recursive components for nesting higher resolution regions • Coupling across C/C++ and Fortran • Ensemble management www. esmf. ucar. edu ESMF-based hierarchical structure of GEOS-5 atmospheric GCM

ESMF: Performance portability • ESMF is highly performance portable, low (<5%) overhead • 3000+ regression tests run on 30+ platform/compiler combinations nightly See http: //www. esmf. ucar. edu/download/platforms • Newer ports include native Windows, Solaris • Using Tera. Grid Build and Test Service to simplify regression Performance at the testing ASMM Run-Time Comparison petascale… Scaling of the ESMF sparse matrix multiply, used in regridding transformations, out to 16 K processors. (ESMF v 3. 1. 0 rp 2) Plot from Peggy Li, NASA/JPL Tested on ORNL XT 4, -N 1 means 1 core per node. www. esmf. ucar. edu msec

ESMF: Higher order interpolation techniques in CCSM • ESMF higher order Interpolation noise in the derivative of the zonal wind stress Interp. noise grid index in latitudinal direction Black = bilinear Red = higher-order ESMF v 3. 1. 1 Green = higher order ESMF v 4. 0. 0 www. esmf. ucar. edu interpolation weights were used to map from a 2 degree Community Atmospheric Model (CAM) grid to a POP ocean grid (384 x 320, irregularly spaced) • 33% reduction in noise globally in quantity critical for ocean circulation compared to previous bilinear interpolation approach • ESMF weights are now the CCSM default

HAF ESMF: Model map SWMF Legend GAIM CCSM 4 Dead atm Data atm GFS Atm Dynamics GFS Atm Phys POP Ocean Dead ocean Data ocean Stub ocean CICE ice Dead ice Data ice Dead land Data land NOAA Department of Defense University NASA Department of Energy National Science Foundation Stub ice CLM Ovals show ESMF components and models that are at the working prototype level or beyond. Stub land GFS I/O FIM NEMS NMM-B Atm Dynamics NMM-B Atm Phys Ice sheet ESMF coupling complete ESMF coupling in progress NMM History Strat Chem Component (thin lines) Model (thick lines) GEOS-5 Param Chem GEOS-5 Atm Dynamics GOCART GEOS-5 GWD GEOS-5 FV Dycore FV Cub Sph Dycore Tracer Advection Land Info System GSI GEOS-5 Atm Physics GEOS-5 Hiistory GEOS-5 Atm Chem GEOS-5 Radiation GEOS-5 Aeros Chem GEOS-5 LW Rad GEOS-5 Solar Rad GEOS-5 Surface GEOS-5 Land GEOS-5 Topology GEOS-5 Veg Dyn GEOS-5 Lake GEOS-5 Catchment GEOS-5 Moist Proc GEOS-5 Turbulence GEOS-5 Land Ice GEOS-5 OGCM Poseidon GEOS-5 Data Ocean GEOS-5 Salt Water GEOS-5 Ocean Biogeo WRF POP ROMS MOM 4 UCLA AGCM COAMPS MITgcm Atm MITgcm Ocean www. esmf. ucar. edu SWAN CICE p. WASH 123 NCOM HYCOM ADCIRC

Loose coupling tools and interfaces Examples: • Open. MI http: //www. openmi. org • Web service approaches Coupling options: • Generally multiple executable • Pull mode of data communication simple but not efficient (ask for a data point based on coordinates) • Generally peer-peer component relationships • Coupling across multiple computer languages (Python, Java, C++, etc. ) www. esmf. ucar. edu

Science gateways – access centers Examples: • Earth System Grid (ESG) – DOE, NCAR, NOAA support, used to distribute Intergovernmental Panel on Climate Change data and for climate research http: //www. earthsystemgrid. org • Hydrologic Information System (HIS) - NSF funded, used to enhance access to data for hydrologic analysis http: //his. cuahsi. org • Object Modeling System (OMS) - USDA effort, used for agricultural modeling and analysis http: //javaforge. com/project/1781 www. esmf. ucar. edu

Metadata conventions and ontologies Examples: • Climate and Forecast (CF) conventions - spatial and temporal properties of fields used in weather and climate http: //cf-pcmdi. llnl. gov • METAFOR Common Information Model (CIM) – large EU-funded project, climate model component structure and properties (including technical and scientific properties) http: //metaforclimate. eu • Water. ML – Schema for hydrologic data developed by the Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI) http: //his. cuahsi. org/wofws. html www. esmf. ucar. edu

Governance Pervasive issue in community modeling Divergent effects of • Multiple institutions • Geographic dispersion • Multiple domains of interest (working groups) Must be balanced by strong integration body - strategies: • Meets frequently enough to affect routine development (quarterly) • Meets virtually to get sufficient representation • Includes user and other stakeholder representatives • Authorized to prioritize and set development schedule • Supported by web-based management tools www. esmf. ucar. edu

Integrating across interoperability elements Examples from the Curator project (NSF and NASA) • Automated output of CF and CIM XML schema from ESMF (tight coupling + ontology) • Ingest of ESMF-generated schema into ESG, propagation into tools for search, browse, intercomparison and distribution of model components and models (tight coupling + ontology + science gateway) • Implementation of dataset “trackback” in ESG that connects datasets with detailed information about the models used to create the data (tight coupling + ontology + science gateway) • Implementation of personal and group workspaces in ESG (science gateway + governance) www. esmf. ucar. edu

Integrating across interoperability elements (cont. ) • Translation of ESMF interfaces into web services to enable invocation of ESMF applications from a science gateway, and enable data and metadata from the run to be stored back to the gateway (tight coupling + loose coupling + science gateway + ontology, new. Web servicefunding) Tera. Grid interface ESMF interface Tightly coupled HPC components www. esmf. ucar. edu Loosely coupled components Issue of switch from push to pull data interactions…

Screenshot: Component trackback www. esmf. ucar. edu

Screenshot: Faceted search www. esmf. ucar. edu

Summary • Cross-domain interoperability platforms have multiple elements • Many of these elements already exist • Integration activities (such as Earth System Curator) are the next focus www. esmf. ucar. edu Image courtesy of Rocky Dunlap, Georgia Institute of technology