15-km – 3-km mesh

[zmenzo]

I am attempting to use the 15-3km mesh for the first time. Currently, I am on the static.nc step and have followed the special instructions laid out on the mesh website (https://mpas-dev.github.io/atmosphere/atmosphere_meshes.html). However, I am curious what a good node submission values would be ("Note also that when 'config_native_gwd_static = true', each MPI task will read roughly 4 GB of global terrain data, so care must be taken to use enough compute nodes to avoid running out of memory.") For the 60-15km mesh I am successfully using (#PBS -l select=1:ncpus=36:mpiprocs=1) but am assuming that will not be enough for the 15-3km mesh. Do you have any recommended values?

Also, when I get the init.nc step it mentions I need a specific io type ("The 3-d fields that exist in the model at higher resolution can easily exceed the 4 GB limit imposed by the classic netCDF format. When creating atmospheric initial conditions (i.e., the "init.nc" file), and when writing output streams from the model with 3-d fields, it is necessary to use an "io_type" that supports large variables, such as "pnetcdf,cdf5" or "netcdf4". For more information on selecting the "io_type" of a stream, refer to Chapter 5 in the Users' Guide.") I have included my current list of libraries used for past successful runs. Will this satisfy the requirement for the 15-3km mesh?

Thanks as always

Zach

 

[zmenzo]

I've attempted the static.nc step twice now with the above stats and I keep getting an Exit Status = 1 at ~3:30:00hr.

 

[mcurry]

Since you will need to use the static interpolation in parallel you'll need to use the CVT partition file x5.64880066.cvt.part.256, which is distributed with the 15-3km mesh. That partition is for 256. I suggest using the minimum amount of nodes needed, which in this case will be 9 nodes with 28 processors each. Your PBS option will look something like:

#PBS -l select=9:ncpus=28:mpiprocs=28

On another note, you normally want to have the number for both ncpus and mpiprocs to be equal.

 

[mgduda]

Since you will need to use the static interpolation in parallel you'll need to use the CVT partition file x5.64880066.cvt.part.256, which is distributed with the 15-3km mesh. That partition is for 256. I suggest using the minimum amount of nodes needed, which in this case will be 9 nodes with 28 processors each. Your PBS option will look something like:

#PBS -l select=9:ncpus=28:mpiprocs=28

On another note, you normally want to have the number for both ncpus and mpiprocs to be equal.

There are a couple of reasons why 9 nodes with 28 tasks/node may not work:

1. The partition file contains 256 partitions, while 9 * 28 = 252.
2. As noted on the mesh download page, the processing of GWDO fields (when config_native_gwd_static=true) requires each MPI task to read and store about ~4 GB of topography data; with 28 tasks/node and 4 GB/task, approximately 112 GB of memory per node would be required, which is more than is available on either the standard or large-memory batch nodes of Cheyenne. (Note that even more memory is required to store model fields, beyond the ~4 GB/task used to store the global terrain field.)

 

[mgduda]

@zmenzo In your modified mpas_init_atm_cases.F file, it looks like you've used shell/Python-style comments:

               !
               !  Without a convex mesh partition file, interpolating static fields in parallel
               !     will give incorrect results. Since it is very unlikely that typical users
               !     will have convex partitions, it's safer to just stop if multiple MPI tasks are
               !     detected when performing the static_interp step.
               !
#               if (domain % dminfo % nprocs > 1) then
#                 call mpas_log_write('****************************************************************', messageType=MPAS_LOG_ERR)
#                 call mpas_log_write('Error: Interpolation of static fields does not work in parallel.', messageType=MPAS_LOG_ERR)
#                 call mpas_log_write('Please run the static_interp step using only a single MPI task.',  messageType=MPAS_LOG_ERR)
#                 call mpas_log_write('****************************************************************', messageType=MPAS_LOG_CRIT)
#               end if

For free-format Fortran, '!' is the comment character. We didn't state this explicitly in the special mesh instructions, but after commenting-out code in the mpas_init_atm_cases.F, you'll need to recompile the init_atmosphere core.

 

[zmenzo]

Thank you all for the assistance, I'm happy I posted my question on here. So if I'm understanding correctly, I can't use that mesh through Cheyenne.

 

[mgduda]

You can definitely use the 15-3 km mesh for simulations on Cheyenne. When processing the static fields using the CVT partition file with 256 partitions, you'll just need to use more nodes and fewer tasks per node. For example, you could try 32 nodes with 8 MPI tasks per node. After you've processed the static fields, you can then use any Metis partition file for the generation of initial conditions and for the model simulation.

 

[mcurry]

My apologies, mgduda is correct. 9 * 28 does not equal 256 and I did not take into account the memory requirements. Thanks for clarifying mgduda! I apologize about the confusion.