How many processors should I use to run WRF?

Moderator: NCAR/MMM

Post Reply
Posts: 1990
Joined: Wed Feb 14, 2018 9:21 pm

How many processors should I use to run WRF?

Post by kwerner » Fri Mar 08, 2019 12:19 am

We have a basic rule-of-thumb when it comes to choosing an appropriate number of processors. You will need to consider the decomposition of the processes in relation to the size of the domains. Decomposition will be determined based on the 2 closest factors for the total number of processors. So if you chose 16 processors, the decomposition would be 4x4, which is nice and even and creates a square grid. Choosing something like 11 processors would likely cause problems as the decomposition would be 1x11 since that is a prime number. We want to stay as close to squares as is possible, but that can be deviated from somewhat.

Depending on the total number of processors you use, your domain is divided up into tiles - 1 per processor. Each tile will have a minimum of 5 rows/columns on each side (called ‘halo’ regions), and are used to pass information from each cell/processor to the neighboring tile. You do not want your entire tile to be halo regions, as you will want some actual space for computation in the middle of each tile. If the computation space does not exist, it can cause the model to crash, or the output to be unrealistic. The simple equations we use to test for this is to take the total number of grid spaces in the west-east direction and divide by the number of tiles in the x-direction [(e_we)/(x-tiles)]. You want the resulting number to be at the very least, greater than 10. Then do the same for the south-north direction [(e_sn)/(y-tiles)], again making sure it’s greater than 10.

The largest number of processors you should use should be based on your smallest domain, and the smallest number of processors you should use should be based on your largest domain. This is why it is important to not have domains that vary too much in size (grid spaces). Obviously you don’t want to use too few processors either, as that can make your run very slow (or impossible), so you’ll need to consider that, as well. A good starting place is to use the following equations:

For your smallest-sized domain:
((e_we)/25) * ((e_sn)/25) = most amount of processors you should use

For your largest-sized domain:
((e_we)/100) * ((e_sn)/100) = least amount of processors you should use

and then play around with it from there to see if you can find a good balance for the domain set-up you’re using, checking the decomposition and number of tiles. Keep in mind this is just a rule-of-thumb, so you may be able to pick something at the far end of one of those 2 values, or somewhere right in the middle. You may also be able to go a bit outside those boundaries. Each run is a little bit different.

If the least number of processors you can use to satisfy the compute requirements of the largest domain is GREATER THAN the most you can use for the smallest domain, then your configuration will not work. It may be necessary to use the ndown program to run your domains separately at the point when that becomes a problem (e.g., if d01, d02, d03 all work okay for a certain number of processors, but d04 is too large, then you can run the first 3 domains as a single run, and then use ndown to run the 4th domain separately).

Post Reply

Return to “FAQ: WRF Run-time Problems & Options”