GEOS-CHEM: Structure, Documentation, Platforms, and Parallelization Bob Yantosca Software Engineer Atmospheric Chemistry Modeling Group Harvard University GEOS-CHEM Users’ Meeting 02 June 2003
Topics 1. 2. 3. 4. 5. 6. 7. 8. 9. GEOS-CHEM history GMAO met fields used to drive GEOS-CHEM model description GEOS-CHEM standard versions and benchmarks GEOS-CHEM user support and documentation GEOS-CHEM platforms Parallelization: Open. MP and MPI Effective coding practices Chemical forecasting 2 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
In the beginning … Ø GEOS-CHEM was originally created from: § § Ø Pieces that came from the Harvard/GISS-II model: § § Ø The old Harvard/GISS-II 9 -layer model (Larry, Yuhang, et al) GEOS-CTM (Dale, Mian) – ancestor of GOCART Emissions (EMISSDR, BIOBURN, etc) Dry Deposition (DRYDEP, DEPVEL, etc. ) Chemistry (SMVGEAR etc. ) Diagnostics (NDxx) and various input files Pieces that came from the GEOS-CTM model: § § Transport (TPCORE, by Lin & Rood) BL mixing (TURBDAY) and Cloud Convection (NFCLDMX) 3 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
In the beginning … GEOS-CHEM evolution: Ø We did not foresee all of the potential uses of GEOS-CHEM at the time we first created GEOS-CHEM. Ø At first, newer code was “onion-skinned” on top of older code. This resulted in functional, albeit, convoluted code. Ø However, a number of recent developments (new met fields, mechanisms, etc. ) has forced us to make some widespread changes to the GEOS-CHEM source code. Ø These problems are not unique to GEOS-CHEM, but also apply to any software product. 4 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
In the beginning … Ø Philosophical Digression: § § Ø “If builders built buildings the way programmers write programs, then the first woodpecker that came along would destroy civilization. ” – Weinberg’s Law “Inside every large program there is a small program struggling to get out. ” – Hoare’s Law of Large Programs Interpretation: § § If you want to build a new building, you tear down the old one first and start from scratch. With software, usually you start with an existing program and add new stuff on top of that. Your foundations may be shaky… 5 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
In the beginning … Ø Much of the existing GEOS-CHEM code has been recently updated, or brand-new code has been added § § § Ø Biomass and biofuel emissions Dry and wet deposition Date/time functions Error handling and file I/O Sulfate chemistry Main program Some sections of GEOS-CHEM still need improvement § § § Diagnostics (especially timeseries) Input file reading Emissions for full chemistry (incl. Isoprene, Soil NOx) 6 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Ø GEOS-CHEM is driven by assimilated met data from NASA/GMAO (formerly DAO), located at GSFC. Ø GMAO has produced 4 major met data sets: § § GEOS-1 GEOS-STRAT GEOS-3 GEOS-4/fv. DAS (1985 – 1995, 20 layers to 10 h. Pa) (1995 – 1997, 46 layers to 0. 1 h. Pa) (2000 – 2002, 48 layers to 0. 01 h. Pa (2003 –, 55 layers to 0. 01 h. Pa) Ø At present, GEOS-CHEM can run with GEOS-1, GEOSSTRAT, and GEOS-3 met fields. Ø GEOS-4 support is currently being added to GEOS-CHEM. 7 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Ø Native horizontal resolution of GMAO met fields: § § Ø 2 o lat x 2. 5 o lon 1 o lat x 1 o lon 1 o lat x 1. 25 o lon We also can regrid met fields to coarser resolution in order to gain computational advantage for long simulations § § Ø GEOS-1 GEOS-STRAT GEOS-3 GEOS-4/fv. DAS 4 o lat x 5 o lon 2 o lat x 2. 5 o lon AND 4 o lat x 5 o lon Regridding of met fields is done offline, and can be automated 8 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Ø I-6 fields (6 -hr instantaneous “snapshot” quantities) § Ø A-6 fields (6 -hr averaged quantities) § § Ø U, V, P, T, Specific humidity, albedo Cld. Frac. – random & max overlap (GEOS-1, GEOS-S) In-cloud optical depth and total cld frac (GEOS-3) Tendency in specific humidity (i. e. precip or evap) Cloud mass flux and detrainment A-3 fields (3 -hr averaged quantities) § § § 10 m winds, roughness height, friction velocity PBL height, column cloud fraction, surface temperature Convective & total precipitation at ground 9 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Times of day when you need to read met fields Z = “Zulu” = abbreviation for GMT GEOS-1 A-6 fields GEOS-S A-3 fields GEOS-3 A-6 fields GEOS-4 A-3 fields From the fv. DAS file spec on GMAO’s website 10 Start of day GEOS-CHEM: Structure, Platforms, . . . Start of next day Monday, June 2, 2003
GMAO met fields Ø GEOS-1, GEOS-STRAT, GEOS-3 § § Ø A-6 fields are stamped w/ center times: 00, 06, 12, 18 GMT (read them at 21, 03, 09, 15 GMT) A-3 fields are stamped w/ end times: 00, 03, 06, … 21 GMT GEOS-4 / fv. DAS § § Ø A-6 fields are stamped w/ center times: 03, 09, 15, 21 GMT (read them at 00, 06, 12, 18 GMT ) A-3 fields are stamped w/ center times: 1: 30, 4: 30, 7: 30, … 22: 30 GEOS-4 timestamps and averaging periods differ from GEOS-3, GEOS-1, and GEOS-STRAT! 11 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Ø Preparing GEOS-CHEM for GEOS-4/fv. DAS met fields: 1. 2. 3. 4. 5. 6. Download, regrid, and format fv. DAS met fields. (in progress) Adapt code to read in fv. DAS met fields w/ different timestamps and averaging intervals. (in progress) Implement new TPCORE transport code for fv. DAS. This contains new semi-lagrangian vertical advection. (done) Fit Philip Cameron-Smith's pressure fixer to the fv. DAS TPCORE transport scheme. Add additional mixing ratio adjustment for residual mass difference to ensure mass conservation. (done) Implement new convection schemes, J. Hack's scheme for moist convection and Phil Rasch's scheme for convective transport. (in progress) Run GEOS-CHEM with GEOS-4/fv. DAS meteorology. Examine results, compare to observations and previous model simulations (yet to be done) 12 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description Ø GEOS-CHEM is currently 8 models in one… § § § § Ø 4 x 5 GEOS-1 4 x 5 GEOS-STRAT 4 x 5 GEOS-3 2 x 2. 5 GEOS-1 2 x 2. 5 GEOS-STRAT 2 x 2. 5 GEOS-3 1 x 1 GEOS-3 (nested grid over China) 1 x 1 GEOS-3 (nested grid over N. America) … soon to be 12 models in one! § § 1 x 1 GEOS-3 (nested grid over Europe) 4 x 5 GEOS-4 2 x 2. 5 GEOS-4 and (maybe not too soon) 1 x 1. 25 GEOS-4 13 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description Ø GEOS-CHEM can perform several different kinds of chemistry simulations, which are selectable at run time. Ø The main simulation is “full chemistry” (i. e. Ozone, etc. ): § § § Chemistry solver: SMVGEAR package by Mark Jacobson Photolysis code: FAST-J by O. Wild & M. Prather 300+ chemical reactions 50+ photolysis reactions 80+ chemical species 31 tracers (i. e. species which are transported) • NOx, HC’s, DMS, SO 2, SO 4, MSA, NH 3, NH 4, Sulf. Nitrates § § Heterogeneous chemistry on aerosol surfaces Aerosol thermodynamic equlibrium (RPMARES) 14 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description Ø In addition to the “full chemistry” simulation, GEOS-CHEM also can perform other chemistry simulations: § § § § Ø 222 Rn – 210 Pb – 7 Be Tagged CO (loss w/ archived OH) Tagged Ox (with pre-saved P-L rates) Methane Ethane Methyl Iodide CO with parameterized OH (B. Duncan) Offline sulfate aerosol All of the “extra” chemistry simulations listed above come standard with GEOS-CHEM § § However, some simulations may be in need of updating Check w/ author of simulation before using it 15 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description GEOS-CHEM source code structure Ø Ø Input and initialization Dynamic loop § § § § § Ø Read daily and monthly and various monthly quantities Interpolate instantaneous (I-6) fields Strat fluxes and Transport PBL mixing and deep cloud convection Diagnostics Dry deposition Emissions breakpoint Chemistry breakpoint Wet deposition Save output to files and quit 16 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description Ø GEOS-CHEM diagnostic output quantities include: § § § § § Ø Tracer concentrations OH, NO 2, HO 2, fields Photolysis rates (J-values) Emissions Cloud and aerosol optical depths Chemical production and loss Dry deposition fluxes and velocities Methyl chloroform lifetime Wet scavenging and wet deposition losses Surface pressure, air mass, air density, etc. GEOS-CHEM can produce average and timeseries output § You can use timeseries output to create 2 -D or 3 -D animations 17 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description GEOS-CHEM directory structure Ø Source code directory (e. g. Code. v 5 -04) § Ø Run directory (e. g. run. v 5 -04) § § Ø Contains Makefiles and Fortran source code Each user keeps run directories in his/her own disk space Contains input files which must be modified by the user in order to select the parameters for a particular GEOS-CHEM run GEOS-CHEM output files will be created in the run directory Users can keep separate run directories for different simulations Data directory (e. g. /data/ctm/GEOS_2 x 2. 5) § § Located in a common disk space available to all users Contains emissions, aerosol files, LAI, tropopause, etc. 18 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Std versions and benchmarks Ø Bob Yantosca currently maintains the standard GEOS-CHEM code at Harvard, using archive software (RCS, soon CVS? ) Ø Procedure for adding new code into the standard model: 1. 2. 3. 4. 5. 6. 7. 8. Ø User downloads a standard GEOS-CHEM version from Harvard User adds new code into their own local copy of GEOS-CHEM User debugs and thoroughly tests his/her local GEOS-CHEM User submits a list of his/her modifications to Bob Y. incorporates modifications into standard GEOS-CHEM Bob Y. benchmarks the new standard GEOS-CHEM version Daniel examines the new standard GEOS-CHEM version Upon Daniel’s blessing, Bob Y. releases GEOS-CHEM version Single point of code modification 19 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Std versions and benchmarks Ø Pros of single point of code modification : § § § Ø Cons of single point of code modification § § Ø One person is responsible for entire standard model. This prevents divergent versions from propagating among users. Consistency is achieved in both coding style and naming files. Documentation and revision history are consistently maintained. If point person is otherwise occupied, turnaround time for incorporating new revisions can be very long. As the number of potential code developers grows, it is hard for one person to keep up with everyone’s submissions. The challenge: to incorporate many revisions from multiple developers into GEOS-CHEM in a timely fashion while preventing the model from diverging. 20 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Std versions and benchmarks Ø 222 Rn— 210 Pb— 7 Be § § Ø GEOS-3 meteorology w/ 48 layers @ 4 x 5 1 year: 1 Jan 2001 – 1 Jan 2002 Diagnostic quantities § § § Ø Benchmark Tracer ratios w/ previous version (surface, 500 h. Pa) Frequency Distribution of tracer ratios Tracer concentrations (surface, 500 h. Pa) Tracer zonal means Budgets for Rn, Pb, Be Benchmark results are posted at Harvard and on the web site. 21 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Std versions and benchmarks Ø Full chemistry benchmark § § Ø Diagnostic quantities § § § Ø GEOS-3 meteorology w/ 30 layers @ 4 x 5 1 month: 1 Jul 2001 – 1 Aug 2001 Tracer ratios w/ previous version (surface, 500 h. Pa) Frequency distribution of tracer ratios J-Value ratios w/ previous version (surface, 500 h. Pa) Tracer concentrations (surface, 500 h. Pa) and zonal means Budgets for Ox and CO; CH 3 CCl 3 lifetime Emissions table (comparison to previous version) Benchmark results are posted at Harvard and on the web site. 22 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Std versions and benchmarks Ø A TAR file w/ GEOS-CHEM source code and benchmark results is created for each new GEOS-CHEM version and is posted on the GEOS-CHEM website. Ø Benchmark results are compared to that of the previous version. We especially focus on the global budgets, emissions, tracer ratios, and the frequency distribution of tracer ratios. Ø Any major problems with a new GEOS-CHEM version should be evident from the benchmark simulation output. Ø A new GEOS-CHEM version is only released after Daniel approves the benchmark results for that version. 23 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø GEOS-CHEM source code + benchmark output is available via website Ø GEOS-CHEM user packages § Ø GEOS-CHEM documentation § § Ø TESTRUN – script for compiling & running GEOS-CHEM jobs (on website) GEOS-CHEM Users’ Guide and FAQ (on website) GEOS-CHEM Style Guide (on website) Visualization packages § § § GAMAP (needs IDL; available from GAMAP website @ Harvard) GRADS (free; see http: //grads. iges. org/grads. html) Matlab (student versions available; check w/ your IT dept) 24 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø How to download GEOS-CHEM source code and other related files from the web: 1. 2. 3. 4. Visit www-as. harvard. edu/chemistry/trop/geos/ Click on Source Code & Data Files at the left of the page Enter the proper password (contact Bob Y. ) Select the files that you wish to download Ø The following TAR files are available for download: § § § GEOS-CHEM source code w/ benchmark simulation output 2 x 2. 5 GEOS-CHEM run directories 4 x 5 GEOS-CHEM run directories 2 x 2. 5 GEOS-CHEM data directories 4 x 5 GEOS-CHEM data directories 25 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø TESTRUN § Perl script for compiling & running GEOS-CHEM, located at: www-as. harvard. edu/chemistry/trop/geos/documentation/ Ø What TESTRUN does 1. 2. 3. 4. 5. Ø Creates two scripts: one for compiling GEOS-CHEM and another for running GEOS-CHEM Submits compilation script to local batch queue system After compilation has completed, submits run script to local batch queue system. This starts the GEOS-CHEM run. Pipes GEOS-CHEM output to a timestamped log file Appends a timestamp to GEOS-CHEM output files Use TESTRUN to make sure you don’t accidentally overwrite files that you wanted to save! 26 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø GEOS-CHEM Users’ Guide § Is the official GEOS-CHEM manual page, located at www-as. harvard. edu/chemistry/trop/geos/documentation/ Ø The GEOS-CHEM Users’ Guide covers the following topics: 1. 2. 3. 4. 5. 6. 7. 8. 9. Introduction and brief overview of GEOS-CHEM Installing and compiling GEOS-CHEM Coding: practice and style Files contained in the GEOS-CHEM data directories Files contained in the GEOS-CHEM run directories Running and debugging GEOS-CHEM Vertical & horizontal resolution of GEOS-CHEM Met fields used in GEOS-CHEM tracers, chemical species, and diagnostics 27 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø GEOS-CHEM Frequently Asked Questions (FAQ) § Shortcuts to the proper section in the User’s Guide, located at www-as. harvard. edu/chemistry/trop/geos/documentation/ Ø The GEOS-CHEM FAQ covers the following topics: 1. 2. 3. 4. 5. 6. 7. 8. What’s new in GEOS-CHEM? What kind of simulations can we run with GEOS-CHEM? What is the horizontal resolution of GEOS-CHEM? What is the vertical resolution of GEOS-CHEM? What are the met fields used by GEOS-CHEM? What diagnostic quantities are archived by GEOS-CHEM? Which chemical species are used in the GEOS-CHEM “full chemistry” mechanism? Who is using GEOS-CHEM? 28 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø GEOS-CHEM Style Guide § § Ø The GEOS-CHEM Style Guide covers the following topics: 1. 2. 3. 4. 5. 6. Ø Describes GEOS-CHEM coding & documentation practices See www-as. harvard. edu/chemistry/trop/documentation/ Background and brief introduction to Fortran 90 Adding documentation headers, indentation, white space Numeric and character data types New language features of Fortran 90 DO loops Fortran 90 modules GEOS-CHEM Style Guide is continually being updated! 29 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø GAMAP § § Ø Visualization package for GEOS-CHEM (and other models too!) See www-as. harvard. edu/chemistry/trop/gamap/ Advantages of using GAMAP: 1. 2. 3. 4. 5. 6. Exploits IDL’s powerful mapping and image processing features Can read output from both GEOS-CHEM and GISS models Contains a user-friendly menu-driven interface GAMAP’s low-level routines can be called separately You can combine GAMAP’s mapping routines with IDL’s HDF, HDF-EOS, and net. CDF file I/O functions With the IDL ION package (sold separately!) you can develop an interactive web-page interface for GAMAP (cf. ITCT 2 k 2) 30 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
User support & documentation Ø We expect GEOS-CHEM developers and users to be resourceful in troubleshooting and debugging § § Ø Bob Y. can provide support for: § § Ø GEOS-CHEM: coding issues and problems GAMAP: problems, bugs, and extensions Bob Y. cannot provide support for: § Ø Check input files and make sure code is debugged Inform Bob Y. of any major bugs that you discover General IDL or Fortran help (read the manual) Bob Y. will “farm out” questions to other GEOS-CHEM users if he cannot answer them himself 31 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM platforms Ø As of June 2003, GEOS-CHEM has been ported to the following shared-memory architectures § § SGI Power Challenge / Origin Compaq Alpha Linux (w/ Portland Group compiler) Sun / Sparc Ø The standard code ships with Makefiles for each of the four supported platforms. Ø At this time, GEOS-CHEM is being recoded to run on distributed-memory architectures w/ help from NASA/NCCS § Hybrid “distributed-shared” memory Compaq Alpha at GSFC 32 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM platforms Ø Shared Memory: one “box” w/ many processors § Each processor can see all of the memory within the “box” Array in memory Proc 1 33 Proc 2 System Memory Proc 3 GEOS-CHEM: Structure, Platforms, . . . Proc 4 Monday, June 2, 2003
GEOS-CHEM platforms Ø Distributed Memory: standalone “nodes” w/ network cnx § A processor can only see the memory on its own node Mem 1 Mem 2 Mem 3 Proc 3 a Proc 1 Proc 2 Proc 3 b 34 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM platforms Ø Hybrid “distributed-shared” memory (e. g. “Halem” at GSFC) § § 300+ nodes, with 4 processors and 2 GB memory per node Halem is the 18 th fastest machine in the world (see top 500. org) 2 GB Memory per Node 1 Node Cnx. to Proc 1 other Proc 2 Proc 3 nodes Proc 4 Cnx. to other nodes 35 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM parallelization Ø GEOS-CHEM achieves parallelization on shared-memory machines using the Open. MP parallelization commands Ø Open. MP is an open standard for parallelization which is supported by many compiler vendors (see openmp. org) § § § Ø SGI Compaq / HP Sun IBM Portland Group (Linux compiler) NAG, etc. Open. MP commands parallelize individual DO loops 36 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM parallelization !$OMP PARALLEL DO !$OMP+DEFAULT( SHARED ) !$OMP+PRIVATE( I, J ) DO J = 1, 1000 DO I = 1, 1000 A(I, J) = A(I, J) ** 1. 35 ENDDO !$OMP END PARALLEL DO ! We have to do this ! a million times! ! Exponentiation is ! very computationally ! intensive! Ø The blue statements are Open. MP sentinels which tell the compiler that we want this DO loop to be executed in parallel Ø (I, J) pairs are split up among the available processors 37 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM parallelization Ø PRIVATE vs. SHARED § § § Ø Each processor needs to keep a local copy of some variables DO-loop indices (e. g. I, J) should be declared PRIVATE Arrays themselves can be declared SHARED, as long as array indices (I, J) are declared PRIVATE To optimize a parallel loop, make sure that it contains as much work as possible § § § Parallelize the outermost loop Parallelize loops which require many iterations (such as looping over tracers, altitude, latitude, longitude) Parallelize loops containing computationally expensive operations (e. g. divisions, logarithms, exponentiations, etc). 38 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM parallelization Ø Advantages of Open. MP § § § Ø Commands are easy to use Good scalability with increasing # of processors Cross-platform support is guaranteed Disadvantages of Open. MP § § Assumes machine has a shared-memory architecture Cannot use for distributed-memory machines, including: • Beowulfs (i. e. PC’s hooked together running Linux) • Blade servers (IBM) § On hybrid “distributed-shared” machines (e. g. Halem), Open. MP can only be used on individual nodes 39 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM parallelization GEOS-CHEM v 5 -04 fullchem, 1 day 30 levels, 4 x 5 grid Compaq Alpha @ GSFC Sun. Fire 3800 @ Dalhousie 1 Processor 23. 3 min / 1 day (97% utilization) 47. 2 min / 1 day 4 Processors 8. 5 min / 1 day (330% utilization) 14. 6 min / 1 day (396% utilization) Speedup 2. 7 times 3. 2 times Going from 1 4 processors results in a speedup of approximately 3 times on Alpha and Sun 40 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM parallelization Ø Open. MP is not optimal for distributed memory machines, these require MPI (message passing interface) parallelization § § Ø For hybrid “distributed-shared” memory machines (e. g. Halem), you can combine MPI and Open. MP: § § Ø GEOS-CHEM can run as-is on Halem, but only on 1 node To use more than 1 node (4 processors) of Halem, we need to use MPI coding commands Use MPI to distribute memory between nodes Use Open. MP to operate among the processors of a single node Tom Clune & Jack Yatteau will update us on the status of recoding GEOS-CHEM for MPI. 41 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Effective coding strategies Ø Summary of good coding style § § § § Ø Include lots of comments (Bob Y. doesn’t know much chemistry!) Add standard documentation headers to each routine Use lots of white space – readability prevents bugs!! Indent code – makes it easier for eye to follow! Place new code into F 90 modules Use allocatable arrays (in modules) instead of COMMON blocks Use new F 90 language features whenever possible See the GEOS-CHEM Style Guide for more details 42 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Chemical forecasting Ø Past aircraft campaigns for which GEOS-CHEM was used to generate real-time forecasts § § Ø NASA TRACE-P (2001) NOAA ITCT 2 k 2 (2002) Future aircraft campaigns in which GEOS-CHEM will be used to generate real-time forecasts § § NASA INTEX-NA (2004) NOAA ITCT 2 k 4 (2004) Ø For TRACE-P and ITCT 2 k 2, GEOS-CHEM forecasts were generated on computers at Harvard Ø In the future we may run the GEOS-CHEM forecasts offsite (maybe at GMAO? ? ? ) We’re not sure yet. 43 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Marketing strategies (? ) GEOS-CHEM license plates ? ? (photo courtesy P. Palmer) 44 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Extra slides … 45 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Comparison of GEOS met fields up to 3 km 46 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields Comparison of GEOS vertical levels to ~32 km 47 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields 1 x 1 nested grid: China 48 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields 1 x 1 nested grid: North America 49 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GMAO met fields 1 x 1 nested grid: Europe 50 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description Ø New GEOS-CHEM code is written using F 90 modules Ø Advantages of F 90 modules: § § § Ø Arrays and routines can be grouped together in a single “librarylike” unit. This allows for more convenient packaging. Arrays and routines in modules can be referenced by other modules elsewhere in the code Module arrays can be dynamically allocatable – better memory management Internal variables and routines can be “hidden” from view to calling routines outside of the module – black box! Modules can be used to centralize operations which may have previously been computed in several different places Existing routines can be packaged into F 90 modules as well 51 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description Ø More about F 90 modules § § Ø You can use PRIVATE declaration to hide module variables or routines from all other external subroutines or modules You can select which variables or routines to import into another program via: USE MY_MODULE, ONLY : MY_SUB If Module B references Module A, then Module A must be compiled before Module B You can “overload” module routines with INTERFACEs so that they can accept different argument types More references § § A Programmer’s Guide to Fortran 90 GEOS-CHEM Style Guide (on website) 52 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description GEOS-CHEM timesteps Ø Dynamic timestep (transport, wetdep, archive diags) § § § Ø Convective timestep (PBL mixing, cloud convection) § Ø Same as dynamic timestep Emissions and dry deposition timestep § Ø 10 min (1 x 1) 15 min (2 x 2. 5) 30 min (4 x 5) Once per hour Photolysis and chemistry timestep § Once per hour 53 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Std versions and benchmarks Ø GEOS-CHEM v 5 -04 is the last released standard version § Ø Added nested-grid capability and cleaned up lots of old code GEOS-CHEM v 5 -05 -03 is the current development version § SMVGEAR II chemistry solver (cf. M. Jacobson) • Written by Mark Jacobson (then at UCLA) • More recent version of our sparse matrix GEAR solver § New transport module (TPCORE) for GEOS-4/fv. DAS winds • Written by S-J Lin (GSFC) • Now with Mass Conservation! (from LLNL) Ø GEOS-CHEM v 6 -01 will be the first version to contain full support for fv. DAS met fields (release date ? ? ) § New cloud convection scheme (cf. P. Rasch) for fv. DAS 54 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description MODULE MY_MODULE IMPLICIT NONE REAL*8, ALLOCATABLE : : ARRAY(: , : ) CONTAINS ! Force declarations ! Allocatable array SUBROUTINE MY_SUB( X ) REAL*8, INTENT(IN) : : X ARRAY(: , : ) = ARRAY(: , : ) * X END SUBROUTINE MY_SUB ! ! SUBROUTINE INIT_MY_MODULE ALLOCATE( ARRAY( 100, 100 ) ) ARRAY(: , : ) = 0 END SUBROUTINE INIT_MY_MODULE ! Subroutine to allocate ! And initialize the array Subroutine example. Of course, GEOS-CHEM code is much more complex SUBROUTINE CLEANUP_MY_MODULE ! Subroutine to IF ( ALLOCATED( ARRAY ) ) DEALLOCATE( ARRAY ) ! Deallocate the END SUBROUTINE CLEANUP_MY_MODULE ! array END MY_MODULE 55 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
GEOS-CHEM description PROGRAM MODULE_TEST USE MY_MODULE ! References all arrays and ! routines in MY_MODULE ! Force explicit declarations IMPLICIT NONE CALL INIT_MY_MODULE ! Initializes and allocates ! module arrays ARRAY(: , : ) = 100. 0 d 0 ! Assignment using array mask CALL MY_SUB( 2. 0 d 0 ) PRINT*, MINVAL( ARRAY ) PRINT*, MAXVAL( ARRAY ) ! Multiplies ARRAY by 2. 0 ! and prints min & max CALL CLEANUP_MY_MODULE ! Deallocates module arrays END PROGRAM MODULE_TEST 56 GEOS-CHEM: Structure, Platforms, . . . Monday, June 2, 2003
Скачать презентацию GEOS-CHEM Structure Documentation Platforms and Parallelization Bob Yantosca
62b914fe758735327a44d80737e5e0b6.ppt