The Paramesh Grid Implementation in FLASH 3 Klaus

The Paramesh Grid Implementation in FLASH 3 Klaus Weide May 2006

Preliminaries Scope: The PARAMESH 3 (actually now: PARAMESH 4. 0) implementation of the Grid unit in FLASH 3 Ignoring particles (a related but separate topic) Ignoring NO_PERMANENT_GUARDCELLS etc. What users* of FLASH 3 need to know about this Grid. *In particular, writers of non-infrastructure code that will work with FLASH 3.

Preliminaries A Warning that Shall Be Repeated: This presentation includes discussion of internal structures of the Grid implementation, in order to explain the function of the Grid. Please do not access these structures in code outside the Grid unit. You may get away with it for a while, but we don't want to support it! If you break it, you get to keep both parts. Please do use the official interfaces. For example, use Grid_get. Blk. Index. Limits instead of constants NXB, NYB, etc. This will keep your code working if the Grid implementation changes!

Basic Computational Unit : Block The grid is composed of blocks Blocks are composed of interior cells and surrounding guard cells Paramesh: All blocks have the same size (number of cells), can have different resolution PM in Flash 3: Block size is determined at setup time.

First Look at Paramesh Grid Purpose of the Grid: represent data more on UNK variables etc. below Each block resides on exactly one processor (at a given point in time) Limitations imposed by Paramesh: Same number of cells, guard cell layers Resolution (“Delta”) of a block changes by multiples of 2 Resolution of neighbors differs at most by factor of 2

How Blocks are Identified At a given time, a block is globally uniquely identified by a pair (PE, Block. ID), where 0 < PE < numprocs 1 < Block. ID < MAXBLOCKS Locally, Block. ID is sufficient to specify a block User code can't directly access remote blocks anyway Morton Numbers provide another way to identify blocks globally (more later)

How Blocks are Stored Solution data, per-block meta data, tree information (for local blocks!) are stored in F 90 arrays declared like this: real, dimension(, , MAXBLOCKS) : : UNK real, dimension(, MAXBLOCKS) : : bnd_box integer, dimension(, MAXBLOCKS) : : parent MAXBLOCKS is a hardwired constant (from setup time) “Inactive” (non-leaf) blocks also use storage These structures are internal to the Grid unit and should not be accessed directly by other code. Use the appropriate Grid_something subroutine calls instead!

How Blocks are Woven Together I The Tree (1 d: binary tree; 3 d: “oct” tree) parent/child relationships Neighbors geometric relationships Morton Ordering a way to serialize blocks used only internally, implicitly Determines storage of blocks

Connecting Blocks: The Tree (1 d: binary tree; 3 d: “oct” tree) parent/child relationships three types of blocks: LEAF – physics acts only on data of these blocks PARENT – have at least one LEAF child ANCESTOR (inactive) – other, data may be invalid Not all blocks have a parent – there is at least one root node, there may be many! Each block has a refinement level, 1 ≤ lrefine(block. ID) ≤ lrefine_max

Connecting Blocks: Neighbors Neighbor relationships array – neighbors across faces surr_blks – surrounding blocks incl. diagonal regenerated by PARAMESH 3 from other information “wrap around” at periodic boundaries Combined Tree and Neighbor info gr_gid array combines parent/child/neigh info This is how linkage info is stored in checkpoint files neigh

Connecting Blocks: Morton Ordering a way to serialize blocks Morton Function maps blocks to integers, '<' relation between integers => morton order of blocks determines how blocks are ordered in memory in each processor AND how they are distributed across processors Morton Curve: an illustration of the Morton order connect centers of leaf blocks in M. order

How Blocks are Woven Together II Warnings The structures linking blocks should not be accessed directly by user code parent, neigh, etc. may point to remote blocks The implementation of Morton function is internal magic of Paramesh. You don't need to understand it to use FLASH 3/Paramesh. User code generally does not need to know where a block's neighbor is. It can rely on Paramesh to fill guard cells from neighbors correctly.

Per-Block Meta-Information Paramesh keeps information associated with each local block: parent, child, surr_blks, neigh – block linkage lrefine, nodetype – tree-related information coord, bnd_box, bsize – physical coordinates There are public accessor interfaces Grid_get. This. Or. That to get the parts of this information that may be useful to user code. Use them!

Block Data: Fluid Variables I FLASH 3 provides the following kinds of variables: UNK variables – stored in adjacent slots in UNK array for each cell-centered (or 'volume-centered') basic hydrodynamic and other physical continuous variables, aka. solution variables, aka. "unknowns". data maintained by PARAMESH UNK contains three kinds of variables, from different Config declarations: General solution variables VARIABLE name #e. g. , VARIABLE dens -> DENS_VAR Species mass fractions SPECIES NAME #e. g. , SPECIES NI 56 For technical reasons there is always at least MFRAC_SPEC "mass scalars" - additional passiveley advected variables MASS_SCALAR NAME #e. g. , MASS_SCALAR YE

Block Data: Fluid Variables II Additional kinds of variables: GRIDVAR aka. SCRATCH variables GRIDVAR name #e. g. , GRIDVAR otmp cell-centered (or face-centered) for temporary use, data is not meant to survive Grid changes data maintained by FLASH 3 Face variables FACEVAR name #e. g. , FACEVAR Mag face-centered currently only used by MHD data maintained by PARAMESH work space – mentioned for completeness maintained by PARAMESH, not currently used by FLASH 3 Fluxes – mentioned for completeness FLUX name #e. g. , FLUX rho maintained by PARAMESH/FLASH 3, used in Hydro

Applying Boundary Conditions Specify type of Boundaries in flash. par Inner boundaries are now also possible Initialization from checkpoint file if restart is TRUE Non-default boundary condition handling: physics units access and modify block data Usually, provide custom Grid_apply. BCEdge. All. Unk. Vars when needed Not recommended: provide custom amr_1 blk_bcset No. Driver unit call Grid_update. Refinement after each time step Hydrostatic boundary conditions: work in progress Implementation as in FLASH 2 default is now available

Life of The Mesh I What happens to the Grid, from start to finish of a calculation Initialization from scratch if restart is FALSE Initialization from checkpoint file if restart is TRUE Changes during Evolution physics units access and modify block data physics units call Grid_fill. Guard. Cells when they need it Driver unit call Grid_update. Refinement after each time step Data and meta info are periodically stored in checkpoint files.

Life of The Mesh – Initialization from scratch How is the Grid constructed initially? One or more initial blocks are created, to become root block(s) of tree(s) – see runtime parameters nblockx, nblocky, nblockz These are distributed by FLASH 3 across processors from the very beginning (“parallel divide domain”), as evenly as possible Some of the initial blocks can now be turned into internal boundary blocks (obstacles), see Simulation_define. Domain PARAMESH magic machinery is let loose on the remaining initial block(s) repeatedly until desired lrefine_max can be reached. It takes care of creating new (finer) blocks as children of coarser parents redistributing blocks across processors (rebalancing) User can influence by setting runtime parameters (lrefine_max, refine_var_n, etc. ) supplying custom Grid_mark. Refine. Derefine implementation

Life of The Mesh – Initialization from checkpoint How is the Grid reconstructed ? Checkpoint contains Grid blocks in original (PE, block. ID) order (Morton Order) This logically one-dimensional sequence of blocks is chopped up and distributed evenly across available processors. The same applies to meta info (e. g. , gr_gid for tree info) Each processor reads and stores only the chunk assigned to it. No processor stores all the global tree data. PARAMESH 3 is called to initialize additional data structures.

Life of The Mesh – Evolution How does the Grid change during a simulation ? Physics units access and modify block data Grid_fill. Guard. Cells is called when needed PARAMESH takes care of communication interpolation and restriction of data may occur Driver unit calls Grid_update. Refinement periodically PARAMESH magic machinery is let loose on the blocks. It takes care of creating new (finer) blocks as children of coarser parents deleting leaf blocks where reolution is too high redistributing blocks across processors (rebalancing) User can influence by setting runtime parameters (lrefine_max, refine_var_n, etc. ) supplying custom Grid_mark. Refine. Derefine implementation

Life of The Mesh – The End How does the Grid save data ? The state of the blocks and meta-info is saved to checkpoint files Blocks end up stored sequentially in global (PE, block. ID) order, i. e. , Morton Order. Some more on this in IO part of tutorial.