Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Inter-domain vs intra-domain communication for nested domains

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.


New member

What would be the optimal rank to node mapping that would maximize on-node, in memory, communication vs off-node communication? It's easy to figure out a good mapping for intra-domain communication as the processor grid is rectangular. But the inter-domain communication pattern is likely more complex: many-to-few or few-to-many depending on the way the information flows between domains (coarser to finer or finer to coarser).

Any advice would be appreciated. Thank you!
This is a good question that does not have a direct answer. However, you can figure out a good answer yourself.

What is required is for you to conduct several tests on your machine, with the EXACT model configuration that you are likely to be using (physics, dynamics, number of domains, domain sizes, etc).

You conduct two types of tests: the aspect ratio (shape) of the decomposed MPI processes, and the mix of MPI+OpenMP processes.

First, do the MPI+OpenMP testing. Find the best number of threads to run per MPI process. This tends to be a smaller number, and requires a number of system dependent env variable to properly "pin" OpenMP processes to specific processors, or to somehow assign "affinity" for an OpenMP process to stay resident on a single processor. This requires that the WRF model is built with MPI+OpenMP (dm+sm in the configure). Make sure to EXPLICITLY set the number of OpenMP threads to use per MPI process. For examle, setenv OMP_NUM_THREADS 2 (or export OMP_NUM_THREADS=2) would set that value to two.

Once you have the optimal OpenMP to MPI ratio, then work on the shape of the MPI decomposed domains. You can use the namelist options nproc_x and nproc_y to explicitly state how many MPI processes to use in the x- and y-directions, respectively. Remember that the MPI-decomposed domains need to have each dimension (for each domain) to be AT LEAST 10 or more grid cells. Do not try to use too many MPI processes.