Problems with running variable mesh in a regional domain

[m_a_russo]

Hello all,

I am trying to run the 60km -> 03km variable mesh but only in a regional domain (center of the mesh being Portugal).
If I understood correctly from the lecture presentations I have to use grid_rotate, then create the .init.nc file for the domain and finally use create_region for the regional domain. So, rotate, init and then cut basically?

I have a couple of issues when running the first init_atmosphere_model instance after the grid_rotate tool. If I run init_atmosphere_model using only one core the log stops at "Using option 'linear' for vertical extrapolation of temperature" and I get a segmentation fault error.

However, if I try using 8 cores and the associated graph.info.part.8 file it also gives me a segmentation fault error but writes the following error to the .err files:
ERROR: ****************************************************************
ERROR: Error: Interpolation of static fields does not work in parallel.
ERROR: Please run the static_interp step using only a single MPI task.
CRITICAL ERROR: ****************************************************************
Logging complete. Closing file at 2020/08/31 11:41:54

So, I'm confused, in the manual it says that creating the init.nc file can be done in parallel but the error says otherwise? Am I missing a step somewhere? And if I use only one core I might just not have enough memory maybe? (each blade available tops out at 128GB or 176GB of RAM).

Any ideas?

Thanks,
Michael

 

[mcurry]

Thanks for the great question! Yes I've think are missing the static interpolation step. It appears that you are trying to interpolate initial conditions, but appears you are trying to run the static interpolation, which, at present, can only be ran with a single processor. For reference, here is an overview between the two steps, which are both ran with the init_atmosphere model:

• interpolation of static fields - Interpolation of static fields such as terrain height (ter), dominate soil category (isltyp), dominate vegetation category (ivgtyp), gravity wave drag fields (oa1, oa2, etc.) and others. This step is required before running the interpolation of initial conditions.
  
  The current implementation of the static interpolation in the current release of master of MPAS (v7.0), does not permit the interpolation of these fields in parallel, which I wont go into detail here.
  
• interpolation of real-data initial conditions - In this step, real-data conditions (GFS, CFSR) are interpolated to the grid. Some of these fields are: u, w, theta, skin temperature, surface pressure and others. The static field interpolation will need to be ran before this step.
  
  This step can be ran in parallel.

For regional domains, you'll also need to create lateral boundary conditions also using the init_atmosphere model.

When you say manual, do you mean step 4 on the MPAS Tutorial page? If so, you'll notice that step four does not contain a step for interpolating static fields. The static interpolation takes a significant amount of time, so the walkthrough in step 4 cuts out the regional domain from a grid that already contains static fields. In this case x1.163842.static.nc. Hence why the interpolation of static fields is missing.

So for your use case you'll need to add that step in. I recommend the following steps for your use case:

1. Rotate your grid's (.grid.nc) refinement region over your desired location (in your case Portugal).
2. Interpolate static fields to your rotated, global .grid.nc file. Currently, there are a few 'gotchas' with the static interpolation on regional domains, so its best to interpolate static fields to a global mesh and cut regions from it. It will also save you a lot of time from having to run the static interpolation multiple times as you find the exact region you want. (step 1.3)
3. Use the create_region tool to cut out your region - Noting that this may take a few tries to get correct.
4. Interpolate real-data initial conditions to your region. (step 4.2)
5. Create lateral boundaries conditions for your region. (step 4.3)

Let us know if that works, or if you have any questions!

 

[m_a_russo]

Thank you for the clarification, I was indeed mixing up the static.nc and init.nc procedure and trying to create the static.nc file while running in parallel. So that answers the parallel question perfectly. Apologies for the confusion!

That also answers my question about what the user manual and tutorial say, because I was looking at the init.nc creation part.

So I started over and rotated the grid to the desired location (step 1), then when doing step 2 it stops, gives me a segmentation fault error and the log file stops at "Using option 'linear' for vertical extrapolation of temperature". I am running with one core and 170GB of RAM. Could this be a limitation of my available memory? Since the grid is a 60km to 03km grid it might be too many cells for my system?

Log file is attached if you would like to take a look at it.

 

[m_a_russo]

I'm still stumped with this, and haven't been able to create the static.nc file. Any ideas as to what might be causing the stop at "Using option 'linear' for vertical extrapolation of temperature"?
I checked the remaining memory available on my last simulations and I believe the model never ran out of RAM, so this might not be the problem.

 

[m_a_russo]

It seems I have solved the issue. If I change the output io_type to "netcdf4" it completes correctly!