Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

METIS Decomposition

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

akraman

New member
METIS Decomposition:
We are using the following command to generate the decomposition files. But when we look at the decomposed output (run as below) and compare it with an already decomposed file from one of the mesh downloads (https://mpas-dev.github.io/atmosphere/atmosphere_meshes.html) they do not match

gpmetis -minconn -contig -niter=200 x5.6488066.graph.info

So the main questions are:
- How are the atmosphere meshes generated here (https://mpas-dev.github.io/atmosphere/atmosphere_meshes.html) ?
- There is a comment which says “requires the use of a special, convex mesh decomposition file -- a regular METIS partition file will not give correct results!”
- Does this mean that the decomposition generated with METIS above is not correct?

Thank you for your help

Regards
Karthik
 
akraman said:
METIS Decomposition:
We are using the following command to generate the decomposition files. But when we look at the decomposed output (run as below) and compare it with an already decomposed file from one of the mesh downloads (https://mpas-dev.github.io/atmosphere/atmosphere_meshes.html) they do not match

If I recall correctly, Metis employs heuristics to partition graphs, so it may be that the original partition files included in the mesh downloads were created with a different version of Metis from the version you're running, and the heuristics may have been tuned or otherwise changed between versions. As a test of the validity of your partition files, you could try making two simulations -- one with N partitions and another with M partitions -- and verifying that you get reasonable throughput that scales by a factor of approximately M/N (provided you have at least, say, several hundred grid cells in all partitions to ensure that you're within the linear scaling regime of the model).

akraman said:
- How are the atmosphere meshes generated here (https://mpas-dev.github.io/atmosphere/atmosphere_meshes.html) ?

Here are a few slides that describe the general process by which we generate meshes: https://www2.mmm.ucar.edu/people/duda/files/mpas/talks/mpas_mesh_generation.pdf. Note that we haven't yet released any of the codes that we use to generate meshes.

akraman said:
- There is a comment which says “requires the use of a special, convex mesh decomposition file -- a regular METIS partition file will not give correct results!”
- Does this mean that the decomposition generated with METIS above is not correct?

The comment about convex partition files only applies to the parallel interpolation of static, terrestrial fields (when "config_static_interp = true" in your namelist.init_atmosphere file). Otherwise, the Metis command that you've used to create general mesh partition files should work for running the model itself.
 
Thank you @mgduda. I can confirm that the throughput does increase by M/N for a 15km resolution run. So I assume the graph partitioning with METIS is indeed happening correctly as expected
 
Top