ESMF 3 rd Community Meeting Boulder July 15

ESMF 3 rd Community Meeting, Boulder, July 15, 2004 PRISM An Infrastructure Project for Climate Research in Europe by Nils Wedi @ ECMWF Contributions by A. Caubel, P. Constanza, D. Declat, J. Latour, V. Gayler, E. Guilyardi, C. Larsson, S. Legutke, R. Redler, H. Ritzdorf, T. Schoenemeyer, S. Valcke, R. Vogelsang and many others in the PRISM community. . .

Overview • • What is PRISM? European Interest, partners PRISM Objectives PRISM Model Components PRISM Approach Architecture and User Interface Current Status and Perspective

What is PRISM? • PRogram for Integrated Earth System Modelling • A European project for climate modelling involving 22 partners, 12/2001 – 12/2004 • Funded by the European Commission (4. 8 M€) • Involves state-of-the-art atmosphere, ocean, seaice, atmospheric chemistry, land-surface and ocean-biogeochemistry models

PRISM partners • MPI-M, Germany (Guy Brasseur, coordinator) • KNMI, The Netherlands (Gerbrand Komen, cocoordinator) • MPI-M&D, Germany • Met. Office, United Kingdom • UREADMY, United Kingdom • IPSL, France • Météo-France, France • CERFACS, France • DMI, Denmark • SHMI, Sweden • NERSC, Norway • • • CSCS/ETH, Switzerland INGV, Italy MPI-BGC, Germany PIK, Germany ECMWF UCL-ASTR, Belgium NEC-ESS, Germany FECIT/Fujitsu, France SGI, Germany SUN, Germany NEC-CCRLE, Germany

Help scientists to spend more time on science Provide software infrastructure to • easily assemble earth system coupled models based on existing state-of-art components models • launch/monitor complex/ensembles earth system simulations • access, analyse and share results across wide community Define and promote technical and scientific standards for Earth System modelling

Technical and scientific standards • Scientific: Global parameters Physical interfaces • Technical: Coupler and I/O Data format and grids Architecture and User Interface Diagnostics and visualization Coding and quality

PRISM model components Atmosphere: Atmospheric Chemistry: Météo-France (ARPEGE), MPG-IMET (ECHAM), IPSL (LMDZ), Met. Office (Unified Model), UREADMY, INGV Land Surface: IPSL (Orchidée), Met. Office, MPGIMET, UREADMY, Météo-France (ISBA) Ocean Biogeochemistry: MPI-BGC, IPSL, MPGIMET, Met. Office MPG-IMET, UREADMY, IPSL, Met. Office, Météo-France, KNMI Coupler: CERFACS, NEC, CCRLE, FECIT, SGI, MPI -MAD Regional Climate: SHMI, DMI, Met. Office Sea Ice: Ocean: UREADMY, Met. Office (FOAM), MPI-M (HOPE), IPSL (OPA/ORCA) NERSC, UCL-ASTR, Met. Office, IPSL, MPG-IMET

ESMF - PRISM Running environment PRISM Superstructure ESMF User code Code Infrastructure

Coupling software and its evolution in PRISM • Coupler: OASIS 3. 0/4. 0 (~10 years experience) • Prism system model interface library: PSMILe • MPI 1 or MPI 2 direct communication between models with same grid otherwise repartitioning using a transformer • modularity: prism_put() and prism_get() to implement in existing models

OASIS coupler: Ocean Atmosphere Sea Ice Soil Historical review: Developed since 1991 in CERFACS to couple existing GCMs. At the time: • Models at relatively low resolution (~10000 -20000 pts) • Small number of 2 D coupling fields (~10) • Low coupling frequency (~once/day) èflexibility was very important, efficiency not so much! 1991 |-- 2001 |--- PRISM OASIS 1 OASIS 2 OASIS 3 OASIS 4

OASIS community • CGAM-Reading (UK) Had. AM 3 - ORCA 2 • Southampton University (UK) Inter. Atm - OCCAM lite • UCL (Belgium) LMDz • SMHI (Sweden) RCA(region. ) – RCO(region. ) • U. of Bergen (Norway) MM 5 - ROMS • KNMI (Netherlands) ECHAM 5 - MPI-OM • DMI (Danemark) ECHAM - HIRLAM • INGV (Italy) ECHAM 5 – MPI-OM • IRI (USA) ECHAM 4 - MOM 3 • JAMSTEC (Japan) ECHAM 4 - OPA 8. 2 BAM 3 - ACOM 2 • U. of Tasmania (Australia) Data Atm. - MOM 4 • CAS, IIT Delhi (India) MM 5 - POM • BMRC (Australia) ECHAM - RCA - CLIO BAM - MOM 4

OASIS community • CERFACS (France) ARPEGE 3 - OPA 8. 1 • METEO-FRANCE (France) ARPEGE 4 ARPEGE 3 - ORCA 2 OPA 8. 1 ARPEGE 3 - ARPEGE medias -OPAmed ARPEGE 2 - OPA TDH • IPSL- LODYC, LMD, LSCE (France) LMDz IFS - ORCA 4 OPA 8. 1 - LMDz ECHAM 4 • MERCATOR (France) ORCA 2 LIM OPA ATL 3/ATL 1 ORCA 2 PAM(OPA) • MPI - M&D (Germany) ECHAM 5 PUMA ECHAM 5 ORCA 2 LIM OPAICE - MPI-OM C-HOPE ECHAM 5 EMAD ECHAM 4 - IFS Cy 23 r 4 - E-HOPE IFS Cy 15 r 8 - C-HOPE E-HOPE • ECMWF (UK) E-HOPE

Oasis 2 and Oasis 3 O • Mono-process coupler • 2 D scalar coupling fields interpolation (SCRIP 1. 4) • PRISM System Model Interface Library PSMILe • coupling fields exchange (MPI 1 & MPI 2) • I/O actions (GFDL mpp_io) A A O Oasis 3 A O A A O file A Flexibility, modularity: Coupler and PSMILe act according to user-defined coupling configuration (text file): • number of models and coupling fields • coupling frequencies and transformations for each field • I/O or coupling mode (transparent for model)

Oasis 4 – new demands • Higher resolution, parallel and scalable models • Higher coupling frequencies desirable • Higher number of models and (3 D) coupling fields • Massively parallel platforms Need to optimise and parallelise the coupler

OASIS 4 is composed of: • a Driver • a Transformer • a new PRISM System Model Interface Library

• Interface and data flow

OASIS 4 • MPI parallel communication including repartitioning • parallel multigrid 3 D neighbourhood search and calculation of communication patterns in each source process PSMILe • extraction of useful part of source field only • parallel I/O: single file, distributed files: GFDL mpp_io parallel file: par. Net. CDF • parallel Transformer: loops over PSMILe requests • flexibility and modularity same as Oasis 3 OB C C C OB C OB C T O 1 C O 2

Driver OCE AD ATM PMIOD V 1: in, metadata V 1 V 2: out, metadata V 2 V 3: out, metadata V 3 user OCE PMIOD V 1: out, metadata V 1 V 2: in, metadata V 2 SCC ATM: . . . OCE: . . . LAND: . . . user OCE SMIOC V 1 : to ATM, T 1 to file V 1 V 2 : from ATM, T 2 OC E AT M V 2 V 1 T V 1 file. V 1 Composition Phase Deployment Phase file. V 4 ATM SMIOC V 1 : from OCE, T 1 V 2: to OCE, T 2 V 3 : to LAND V 3 LAN D V 4 Definition Phase ATM-LAND AD user LAND PMIOD V 3: in, metadata V 3 V 4: in, metadata V 4 user LAND SMIOC V 3 : from ATM V 4 : from file. V 4

Oasis – Current status • OASIS 3_prism_2 -2 available • OASIS 4 prototype available • OASIS 4 final PRISM version due 12/2004

File formats and grids • Net. CDF for grid and restart auxiliary files • CF convention under development, extending the COARDS conventions • XML for model and script meta-data input (Fortran namelist and shell replacement)

System Architecture and User Interface PRISM architecture to provide an efficient climate modelling infrastructure to users and developers through: • • Standardised interfaces Remote functionality Centralised administration Distributed resources

Standard compile (SCE) and run environment (SRE) • Finalizing the SCE and SRE for the PRISM models • The system comprises 15 models (arpege_climat 4 echam 5 hamocc lim lmdz mozart mpi-om oasis 3 opa orchidee pisces toy 4 opa toyatm toyche toyoce) which are adapted to a varying degree to the PRISM standards and can run in several combinations. • Most of the models have been tested on a variety of platforms (NEC (SX), SGI (MIPS or IA 64 SGI), FUJITSU (VPP), IBM (power 4)) A tedious task without flashy graphics but very useful !!!

Prep. IFS

Prep. IFS – earth system modeling via the Internet …

OASIS 4 – GUI support XML designed to be read by machines not humans !!!

SMS/Web. Cdp job scheduling and monitoring • Complex automated scheduling • Macro-parallelism • Flexible interdependencies • Interactive control • Visual structure of large systems • Used for all operational and research activities at ECMWF ( ~10 years)

Further tools … • Diagnostics – Prism processing and visualization software (COCO, CDAT/VCS and VTK), at ECMWF MARS/Vis 5 D/Metview • Web GUI – database and diagnostics web interface using web-access server technology, (DODS and LAS), at ECMWF Web-MARS

A PRISM sustained team • Document by PRISM Steering Group proposing establishment of a PRISM sustained team of 7 people sent to European Climate modelling Community (June 04) • First preparation meeting: August 17 th, 2004 • Target: Signature of Consortium Agreement: 01/2005 • MPI (Germany), CERFACS (France), ECMWF (EU), CNRS (France), Met. Office (UK), NCAS(UK), CCRLE (Germany) already expressed strong interest. • Additional FP 6 funding (March 2005)? Enthusiasm is still high !!!

Further information http: //prism. enes. org

Oasis 4: in the prototype (05/2004) • Access and use of XML information • Coupling and I/O of n parallel applications with m components. • Coupling exchange with repartitioning, direct or through the Transformer • Interpolations: • PSMILe (non-exact) parallel neighbourhood search • 3 D 2 D nearest-neighbour, 3 D linear, 2 D linear • I/O: • single and parallel mode • Coupling and I/O exchange from one source to many targets • Local transformations (scatter, gather, add or mult scalar, statistics) • Basic time transformation (average, accumul, min, max)

Oasis 4: still to be done • PSMILe API for model access to SCC and SMIOC information • Interpolation: • More schemes (conservative, 3 D, etc. ) • Exact parallel neighbour search • Transformer parallelisation (almost completed) • Field reduction, combination • Full support of vector and bundle fields (I/O OK) • I/O: distributed mode (par. Net. CDF) • Adaptive grids • Unstructured grids