Community infrastructure for building and coupling high performance climate, weather, and coastal models Cecelia De. Luca NOAA / CIRES University of Colorado, Boulder cecelia. deluca@noaa. gov ESMF website: www. earthsystemmodeling. org Building Community Codes for Effective Scientific Research on HPC Platforms September 6 -7, 2012
Outline • • • Vision and Context Technical Overview User Community Values, Processes and Governance Future
Vision • Earth system models that can be built, assembled and reconfigured easily, using shared toolkits and standard interfaces. • A growing pool of Earth system modeling components that, through their broad distribution and ability to interoperate, promotes the rapid transfer of knowledge. • Earth system modelers who are able to work more productively, focusing on science rather than technical details. • An Earth system modeling community with cost-effective, shared infrastructure development and many new opportunities for scientific collaboration. • Accelerated scientific discovery and improved predictive capability through the social and technical influence of ESMF.
Computational Context • Teams of specialists, often at different sites, contribute scientific or computational components to an overall modeling system • Components may be at multiple levels: individual physical processes (e. g. atmospheric chemistry), physical realms (e. g. atmosphere, ocean), and members of same or multi-model ensembles (e. g. “MIP” experiments) • Components contributed from multiple teams must be coupled together, often requiring transformations of data in the process(e. g. grid remapping and interpolation, merging, redistribution) • Transformations are most frequently 2 D data, but 3 D is becoming more common • There is an increasing need for cross-disciplinary and interframework coupling for climate impacts • Running on tens of thousands of processors is fairly routine; utilizing hundreds of thousands of processors or GPUs is less common • Modelers will tolerate virtually no framework overhead and seek
Architecture • The Earth System Modeling Framework (ESMF) provides a component architecture or superstructure for assembling geophysical components into applications. • ESMF provides an infrastructure that modelers use to – Generate and apply interpolation weights – Handle metadata, time management, I/O and communications, and other common functions The ESMF distribution Components Layer Gridded Components Coupler Components ESMF Superstructure Model Layer User Code Fields and Grids Layer ESMF Infrastructure Low Level Utilities External Libraries MPI, Net. CDF, …
Components • ESMF is based on the idea of components – sections of code that are wrapped in standard interfaces • Components can be arranged hierarchically, helping to organize the structure of complex models • Different modeling groups may create different kinds or levels of components Some of the ESMF components in the GEOS-5 atmospheric GCM
ESMF as an Information Layer Applications of information layer • • • Parallel generation and application of interpolation weights Run-time compliance checking of metadata and time behavior Fast parallel I/O Redistribution and other parallel communications Automated documentation of models and simulations Ability to run components in workflows and as web services Structured model information stored in ESMF wrappers ESMF data structures Standard metadata Standard data structures Attributes: CF conventions, ISO standards, METAFOR Common Information Model Component Field Grid Clock User data is referenced or copied into ESMF structures Native model data structures modules grids fields timekeeping
Standard Interfaces • All ESMF components have the same three standard methods: – Initialize – Run – Finalize • Each standard method has the same simple interface: Steps to adopting ESMF • Divide the application into components (without ESMF) • Copy or reference component input and output data into ESMF data structures • Register components with ESMF • Set up ESMF couplers for data exchange call ESMF_Grid. Comp. Run (my. Comp, import. State, export. State, clock, …) Where: my. Comp points to the component import. State is a structure containing input fields export. State is a structure containing output fields clock contains timestepping information • Interfaces are wrappers and can often be set up in a non-intrusive way
Data Structures • ESMF represents data as Fields that can be built on four discretization types: – – Logically rectangular grids, which may be connected at the edges Unstructured meshes Observational data streams Exchange grids – grids that are the union of the grids of components being coupled, and reference the data on the original grids • ESMF can transform data among these representations, using an underlying finite element unstructured mesh engine
Component Overhead • Representation of the overhead for ESMF wrapped native CCSM 4 component • For this example, ESMF wrapping required NO code changes to scientific modules • No significant performance overhead (< 3% is typical) • Few code changes for codes that are modular • Platform: IBM Power 575, bluefire, at NCAR • Model: Community Climate System Model (CCSM) • Versions: CCSM_4_0_0_beta 42 and ESMF_5_0_0_beta_snapshot_01 • Resolution: 1. 25 degree x 0. 9 degree global grid with 17 vertical levels for both the atmospheric and land model, i. e. 288 x 192 x 17 grid. The data resolution for the ocean model is 320 x 384 x 60.
ESMF Regridding ESMF offers extremely fast parallel regridding with many options (regional, global; bilinear, higher order, first order conservative methods; logically rectangular grids and unstructured meshes; pole options; 2 D or 3 D; invocation as offline application or during model run) Summary of features: http: //www. earthsystemmodeling. org/esmf_releases/non_pu blic/ESMF_5_3_0/esmf_5_3_0_regridding_status. htm HOMME Cubed Sphere Grid with Pentagons Courtesy Mark Taylor of Sandia FIM Unstructured Grid ESMF supported grids IMPACT: “use of the parallel ESMF offline regridding capability has reduced the time it takes to create CLM surface datasets from hours to minutes” Mariana Vertenstein, NCAR Regional Grid
Regridding Modes • ESMF Offline Regrid. Weight. Gen Application: – Separate application that is built as part of ESMF, can be used independently – Application generates a net. CDF weight file from two net. CDF grid files – Supports SCRIP, GRIDSPEC, UGRID, and custom ESMF unstructured format mpirun –np 32 ESMF_Regrid. Weight. Gen –s src_grid. nc –d dst_grid. nc –m bilinear –w weights. nc • Regridding during model execution: – ESMF library subroutine calls which do interpolation during model run – Can get weights or feed directly into ESMF parallel sparse matrix multiply – Can be used without ESMF components call ESMF_Field. Regrid. Store(src. Field=src, dst. Field=dst, regrid. Method=ESMF_REGRID_METHOD_BILINEAR, routehandle=rh) call ESMF_Field. Regrid(src. Field=src, dst. Field=dst, routehandle=rh)
ESMP – Python Interface ● ● ● ● Flexible way to use ESMF parallel regridding Separate download: http: //esmfcontrib. cvs. sourceforge. net/viewvc/esmfcontrib/python/ESMP/ Requirements: – Python – Numpy – Ctypes Limited platform support: Linux/Darwin, GCC(g++/gfortran), Open. MPI Data type: ESMP_Field Grid types: – Single-tile 2 D logically rectangular type: ESMP_Grid – Unstructured type: ESMP_Mesh Support for all ESMF interpolation options 3/19/2018
ESMF Web Services for Climate Impacts GOAL: Develop a two-way coupled, distributed, service-based modeling system comprised of an atmospheric climate model and a hydrological model, utilizing standard component interfaces from each domain. IDEA: Bring the climate model to local applications • Two way technical coupling completed during 2012 using CAM (with active land) and SWAT (Soil Water Assessment Tool) • Utilized switch in CAM’s ESMF component wrapper that enables web service interface • Coupled system is run using the Open. MI configuration editor, a web service driver developed in the hydrological community • Next steps: using WRF within the CESM framework and an updated hydrological model
Summary of Features • Components with multiple coupling and execution modes for flexibility, including a web service execution mode • • • Fast parallel remapping with many features • Time management utility with many calendars, forward/reverse time operations, alarms, and other features • Metadata utility that enables comprehensive, standard metadata to be written out in standard formats • Runs on 30+ platform/compiler combinations, exhaustive nightly regression test suite (4500+ tests) and documentation • • Couples Fortran or C-based model components Core methods are scalable to tens of thousands of processors Supports hybrid (threaded/distributed) programming for optimal performance on many computer architectures; works with codes that use Open. MP and Open. ACC Open source license
Major Users ESMF Components: • NOAA National Weather Service operational weather models (GFS, Global Ensemble, NEMS) • NASA atmospheric general circulation model GEOS-5 • Navy and related atmospheric, ocean and coastal research and operational models – COAMPS, NOGAPS, HYCOM, Wave. Watch, others • Hydrological modelers at Delft hydraulics, space weather modelers at NCAR and NOAA ESMF Regridding and Python Libraries • NCAR/DOE Community Earth System Model (CESM) • Analysis and visualization packages: NCAR Command Language, Ultrascale Visualization - Climate Data Analysis Tools (UV-CDAT), Py. Ferret users • Community Surface Dynamics Modeling System
Usage Metrics • Updated ESMF component listing at: http: //www. earthsystemmodeling. org/components/ • Includes 85 components with ESMF interfaces, 12 coupled crossagency modeling systems in space weather, climate, weather, hydrology, and coastal prediction, for operational and research use • About 4500 registered downloads
Values and Principles • • Community driven development and community ownership Openness of project processes, management, code and information Correctness Commitment to a globally distributed and diverse development and customer base Simplicity Efficiency User engagement Environmental stewardship Web link for detail: http: //www. esmf. ucar. edu/about_us/values. shtml
Agile Development vs Community Values In general, Agile processes promote to community development. However, there areas of potential conflict: Agile emphasis on co-location: “most agile methodologies or approaches assume that the team is located in a single team room” (from Miller, Dist. Agile Development at Microsoft, 2008) vs ESMF emphasis on distributed co-development Agile emphasis on single product owner vs
Making Distributed Co-Development Work Hinges on asynchronous, all-to-all communication patterns: everybody must have information • Archived email list where all development correspondence gets cc’d • Minutes for all telecons • Web browsable repositories (main and contributions), mail summary on check-ins • Daily, publicly archived test results • Monthly archived metrics • Public archived trackers (bugs, feature requests, support requests, etc. ) Discouraged: IMing, one-to-one correspondence or calls – the medium matters
Change Review Board • CRB established as a vehicle for shared ownership through user task prioritization and release content decisions • Consists of technical leads from key user communities • Not led by the development team! • Sets the schedule and expectations for future functionality enhancements in ESMF internal and public distributions – Based on broad user community and stakeholder input – Constrained by available developer resources – Updated quarterly to reflect current realities • CRB reviews releases after delivery for adherence to release plan
Governance Highlights Management of ESMF required governance that recognized social and cultural factors as well as technical factors Main practical objectives of governance: • Enabling stakeholders to fight and criticize in a civilized, contained, constructive way • Enabling people to make priority decisions based on resource realities Observations: • Sometimes just getting everyone equally dissatisfied and ready to move on is a victory • Thorough, informed criticism is the most useful input a project can get • Governance changes and evolves over the life span of a project
Governance Functions • • Prioritize development tasks in a manner acceptable to major stakeholders and the broader community, and define development schedules based on realistic assessments of resource constraints (CRB) Deliver a product that meets the needs of critical applications, including adequate and correct functionality, satisfactory performance and memory use, . . . (Core) Support users via prompt responses to questions, training classes, minimal code changes for adoption, thorough documentation, . . . (Core) Encourage community participation in design and implementation decisions frequently throughout the development cycle (JST) Leverage contributions of software from the community when possible (JST) Create frank and constructive mechanisms for feedback (Adv. Board) Enable stakeholders to modify the organizational structure as required (Exec. Board) Coordinate and communicate at many levels in order to create a knowledgeable and supportive network that includes developers, technical management, institutional management, and program management (IAWG and other bodies)
Governance Executive Management Executive Board Strategic Direction Organizational Changes Board Appointments Interagency Working Group Stakeholder Liaison Programmatic Assessment & Feedback annuall y Reporti ng Advisory Board Reporting External Projects Coordination General Guidance & Evaluation Working Project quarter ly Change Review Board Development Priorities Release Review & Approval Functionality Change Requests Requirements Definition Resource Constraints Implementation Schedule Joint Specification Team Core Development Team Project Management Software Development Testing & Maintenance daily Distribution & User Support weekl y Design and Code Reviews External Code Contributions Collaborative Design Beta Testing
Evolution Phase 1: 2002 -2005 NASA’s Earth Science Technology Office ran a solicitation to develop an Earth System Modeling Framework (ESMF). A multi-agency collaboration (NASA/NSF/DOE/NOAA) won the award. The core development team was located at NCAR. A prototype ESMF software package (version 2 r) demonstrated feasibility. Phase 2: 2005 -2010 New sponsors included Department of Defense and NOAA. A multi-agency governance plan including the CRB was created: http: //www. earthsystemmodeling. org/management/paper_1004_projectpl an. pdf Many new applications and requirements were brought into the project, motivating a complete redesign of framework data structures (version 3 r). Phase 3: 2010 -2015 (and beyond) The core development team moved to NOAA/CIRES for closer alignment with federal models. Basic framework development completed with version 5 r (ports, bugs, feature requests, user support etc. still require resources). The focus is on increasing adoption and creating a community of
Technical Evolution: National Unified Operational Prediction Capability (NUOPC) • ESMF allows for many levels of components, types of components, and types of connections • In order to achieve greater interoperability, usage and content conventions and component templates are needed • A tri-agency collaboration (NOAA, Navy, Air Force) is building a “NUOPC Layer” that constrains how ESMF is used, and introduces metadata and other content standards, along with inheritable templates for different usage scenarios • A production version of the NUOPC Layer is scheduled for delivery at the end of 2012
Co. G Collaboration Environment Central navigation: Templated content Right side: Services Left side: Auto generated navigation of project freeform content Central Section: Freeform content The Co. G environment exposes and collates the information needed for distributed, multi-project development, including project repositories, trackers, and governance processes. It does this in an environment that’s linked to data search, metadata, and visualization services and set up to enable component comparisons.
DCMIP on Co. G Atmospheric Dynamical Core Model Intercomparison Project http: //www. earthsystemcog. org/projects/dcmip-2012/ Planned MIPs: 2013 Downscaling 2014 Atm-surface hydrology 2014 Frameworks
Future Governance and process largely follow established patterns Active development ● Python API redesign and addition of features ● Preparation for NUOPC Layer major release ● Web services for climate impacts ● Regridding options – higher order conservative , filling in gaps ● Advanced fault-tolerance ● Performance optimizations and GPUs ● Co. G collaboration environment ESMF and related development is supported by NASA, NOAA, NSF, and the Department of Defense. 3/19/2018
Скачать презентацию Community infrastructure for building and coupling high performance
8aeb41574e55749b78bbf4d33e3a4af5.ppt