Segmentation fault when running init_atmosphere

davidl2000

davidl2000

New member
I generated a 15km global to 3km variable resolution grid that centers on a specific region in the U.S. for severe weather modeling using the MPAS-Tools Python library. I then used the MPAS-Limited-Area Python tool to select a rectangular domain for what I would like to model (essentially the CONUS). I attempted to use this grid to run (with WPS files generated from GFS data) init_atmosphere with config_static_interp = true and config_native_gwd_static = true as it gets to the terrain generation part this segmentation fault occurs (this is in debug mode)
davidl2000@davidl2000-Main-PC:~/Model_HUB/MPAS-Model$ ./init_atmosphere_model
At line 724 of file mpas_atm_advection.F
Fortran runtime error: Index '7' of dimension 1 of array 'indx' above upper bound of 6

Error termination. Backtrace:
#0 0x96f82823e59 in ???
#1 0x96f82824a71 in ???
#2 0x96f82825082 in ???
#3 0x5cbf378ea596 in __atm_advection_MOD_elgs
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:724
#4 0x5cbf378eb439 in __atm_advection_MOD_migs
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:655
#5 0x5cbf378f055a in __atm_advection_MOD_poly_fit_2
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:605
#6 0x5cbf378f599c in __atm_advection_MOD_atm_initialize_advection_rk
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:288
#7 0x5cbf3792dbdd in __mpas_init_atm_static_MOD_init_atm_static
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_init_atm_static.F:1260
#8 0x5cbf378e2fe4 in __init_atm_cases_MOD_init_atm_setup_case
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_init_atm_cases.F:216
#9 0x5cbf377eb227 in __init_atm_core_MOD_init_atm_core_run
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_init_atm_core.F:92
#10 0x5cbf377459ca in __mpas_subdriver_MOD_mpas_run
at /home/davidl2000/Model_HUB/MPAS-Model/src/driver/mpas_subdriver.F:416
#11 0x5cbf37743da6 in mpas
at /home/davidl2000/Model_HUB/MPAS-Model/src/driver/mpas.F:20
#12 0x5cbf37743e20 in main
at /home/davidl2000/Model_HUB/MPAS-Model/src/driver/mpas.F:10

I've recompiled this over a dozen times, using different configurations (single vs double precision, gnu and gfortran, etc) and cannot figure out why this is segfaulting. It cannot be a memory issue because System Monitor shows RAM usage, even with mpirun -np 8 to be less than 50% of the total system's RAM (I have 64GB RAM on this machine)

P.S. I'm being told when attaching files that I cannot because the temp. directory is missing on the site, so when that is rectified I will upload my namelist.init_atmosphere, log.init_atmosphere.0000.out, and whatever else someone tells me is needed to help debug this.

Thanks in advance
 
Hi,

I dimly remember that I had a similar issue before. After I set config_native_gwd_static = false, init_atmosphere can run successfully.

Would you please try the option config_native_gwd_static = false? Please let me know whether init_atmosphere works.

Also, which version of MPAS are you running?

Thanks.

( I will look at the issue about the missing .temp directory for files upload. It may take some time. Thanks for your patience.)
 
Last edited:
Ming Chen said:
Hi,

I dimly remember that I had a similar issue before. After I set config_native_gwd_static = false, init_atmosphere can run successfully.

Would you please try the option config_native_gwd_static = false? Please let me know whether whether init_atmosphere works.

Also, which version of MPAS are you running?

Thanks.

( I will look at the issue about the missing .temp directory for files upload. It may take some time. Thanks for your patience.)
Click to expand...
Hello and sorry for the late reply! I attempted with the setting "config_native_gwd_static = false" and it still had a Segmentation Fault with the following error:
(base) davidl2000@davidl2000-Main-PC:~/Model_HUB/MPAS-Model$ ./init_atmosphere_model
At line 724 of file mpas_atm_advection.F
Fortran runtime error: Index '7' of dimension 1 of array 'indx' above upper bound of 6

Error termination. Backtrace:
#0 0x7ca6bf823e59 in ???
#1 0x7ca6bf824a71 in ???
#2 0x7ca6bf825082 in ???
#3 0x57eeb8417596 in __atm_advection_MOD_elgs
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:724
#4 0x57eeb8418439 in __atm_advection_MOD_migs
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:655
#5 0x57eeb841d55a in __atm_advection_MOD_poly_fit_2
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:605
#6 0x57eeb842299c in __atm_advection_MOD_atm_initialize_advection_rk
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_atm_advection.F:288
#7 0x57eeb845abdd in __mpas_init_atm_static_MOD_init_atm_static
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_init_atm_static.F:1260
#8 0x57eeb840ffe4 in __init_atm_cases_MOD_init_atm_setup_case
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_init_atm_cases.F:216
#9 0x57eeb8318227 in __init_atm_core_MOD_init_atm_core_run
at /home/davidl2000/Model_HUB/MPAS-Model/src/core_init_atmosphere/mpas_init_atm_core.F:92
#10 0x57eeb82729ca in __mpas_subdriver_MOD_mpas_run
at /home/davidl2000/Model_HUB/MPAS-Model/src/driver/mpas_subdriver.F:416
#11 0x57eeb8270da6 in mpas
at /home/davidl2000/Model_HUB/MPAS-Model/src/driver/mpas.F:20
#12 0x57eeb8270e20 in main
at /home/davidl2000/Model_HUB/MPAS-Model/src/driver/mpas.F:10
 
Hi,

Thank you for testing the namelist option of config_native_gwd_static. I am sorry to know that the case still failed.

I am suspicious that this model crash is caused by wrong grid structure. Can you specify what tools from the MPAS-Tools Python library you used to create the 15-3km MPAS mesh?

If possible, can you follow the standard steps to create a variable-resolution mesh, i.e., run grid_rotate first, then run limited_area_package to create the regional mesh? This is the approach we recommend and it has been widely tested. I expect this would give you the correct result.

Please try and let me know if you have any issues.
 
Of course!
(I will upload all relevant files into a Google Drive temporarily to bypass the temp. error going on with uploading files to this forum, here's the link and I will mention the exact names of the files/screenshots at the appropriate times to help keep the chronology of the process.)
Google Drive link

So I made a two-step Python script, step 1 was the actual script to generate a variable-resolution grid using numpy, xarray, mpas_tools.mesh.creation and matplotlib.pyplot, named generate_new_grid.py to create, plot and visualize the generated grid to verify it looks accurate. I then use a second script called convert_grid2mpas.py that uses xarray, build_spherical_mesh imported from mpas_tools.mesh.creation.build.mesh, and convert imported from
mpas_tools.mesh.conversion that converts the grid_density.nc file generated from the previous script and transform it using JIGSAW and MPAS_Tools into a grid that is usable by MPAS. Then I use the MPAS_Limited_Area package to set a custom region (which in my case is the default conus.custom.pts that is shown on the README for the Github page) to have a 15km CONUS region with the 3km focus on the NE AR/SE MO/W KY/S IL area.

The reasoning for doing the custom method rather than using the pre-existing 15-3km was due to the fact I am attempting to run MPAS for severe weather analysis purposes, using GFS data from 03/14/2025, and I wanted to use a custom domain, and the pre-generated 15-3km mesh on the MPAS website appeared way too big (and therefore too computationally expensive) for me to run with my computer (which uses an Intel i9-13900k and 64GB DDR4-3600 RAM) so I went with MPAS_Tools as the option to generate a variable resolution grid with a custom-set center in NE Arkansas.

I can still try the grid_rotate method but I'm not sure how to incorporate it with custom grids generated using MPAS_Tools.
 
Hello David and Ming, I have same problem running init_atmosphere which crashes when attempting to generate the static file. I edited the fortran file to add some blocks to find out where this might occur, and found out it breaks when attempting the topography interpolation. Here is what my last lines in the log output file looks like.

nVertLevels = 55 (config_nvertlevels)
nSoilLevels = 4 (config_nsoillevels)
nFGLevels = 38 (config_nfglevels)
nFGSoilLevels = 4 (config_nfgsoillevels)
nVertLevelsP1 = 56
nSoilComps = 8
nGocartLevels = 30 (config_gocartlevels)
nMonths = 12 (config_months)
----- done assigning dimensions from Registry.xml -----
Allocating fields ...
5 MB allocated for fields on this task
653 MB total allocated for fields across all tasks
----- done allocating fields -----
WARNING: Variable nominalMinDc not in input file.
WARNING: File houston.grid.nc does not contain a seekable xtime variable. Forcing a read of the first time record.
real-data GFS test case

--- enter subroutine init_atm_static:
called_config_path
called_landuse_data
called_config_topo
called_noahmp
called_all_mpas_pool
completed_nominalMin
completed_KD-Tree allocation
completed_Coriolis parameter
completed_Advection and deformation calculation
Using GMTED2010 terrain dataset
--- start interpolate TER
If Terrain stops, then I crashed after this level
Processing tile: /data/MPAS/mpas_static/mpas_static/topo_gmted2010_30s/31201-32400.13201-14400


I attempted to also play around with these settings in the namelist but no success.
config_supersample_factor = 3
config_lu_supersample_factor = 1
config_30s_supersample_factor = 1
 
@ Akinleye,

How did you create houston.grid.nc ?

Please also tell me which version of MPAS you are running and upload your namelist.init_atmosphere for me to take a look. Thanks.
 
Hi David,

I have tried to repeat your issues in NCAR HPC. Unfortunately I cannot successfully create the MPAS grid. The error message:

Code: 
terminate called after throwing an instance of 'std::invalid_argument'
  what():  Error putting variable triangleAngleQuality: -101
Traceback (most recent call last):
  File "/glade/u/home/chenming/MyDoc/UserCodes/convert_grid2mpas.py", line 18, in <module>
  File "/glade/work/chenming/conda-envs/mchen-npl2025a/lib/python3.12/site-packages/mpas_tools/mesh/creation/build_mesh.py", line 106, in build_spherical_mesh
    check_call(args=args, logger=logger)
  File "/glade/work/chenming/conda-envs/mchen-npl2025a/lib/python3.12/site-packages/mpas_tools/logging.py", line 66, in check_call
    raise subprocess.CalledProcessError(process.returncode, print_args)
subprocess.CalledProcessError: Command 'MpasMeshConverter.x mesh_triangles.nc base_mesh.nc' died with <Signals.SIGABRT: 6>.

This is coming from MpasMeshConverter.x when it tries to write the variable triangleAngleQuality into base_mesh.nc

I looked at mesh_triangles.nc, and it includes all variables without any NaN. Have you ever seen similar issues before? Thanks.


davidl2000 said:
Of course!
(I will upload all relevant files into a Google Drive temporarily to bypass the temp. error going on with uploading files to this forum, here's the link and I will mention the exact names of the files/screenshots at the appropriate times to help keep the chronology of the process.)
Google Drive link

So I made a two-step Python script, step 1 was the actual script to generate a variable-resolution grid using numpy, xarray, mpas_tools.mesh.creation and matplotlib.pyplot, named generate_new_grid.py to create, plot and visualize the generated grid to verify it looks accurate. I then use a second script called convert_grid2mpas.py that uses xarray, build_spherical_mesh imported from mpas_tools.mesh.creation.build.mesh, and convert imported from
mpas_tools.mesh.conversion that converts the grid_density.nc file generated from the previous script and transform it using JIGSAW and MPAS_Tools into a grid that is usable by MPAS. Then I use the MPAS_Limited_Area package to set a custom region (which in my case is the default conus.custom.pts that is shown on the README for the Github page) to have a 15km CONUS region with the 3km focus on the NE AR/SE MO/W KY/S IL area.

The reasoning for doing the custom method rather than using the pre-existing 15-3km was due to the fact I am attempting to run MPAS for severe weather analysis purposes, using GFS data from 03/14/2025, and I wanted to use a custom domain, and the pre-generated 15-3km mesh on the MPAS website appeared way too big (and therefore too computationally expensive) for me to run with my computer (which uses an Intel i9-13900k and 64GB DDR4-3600 RAM) so I went with MPAS_Tools as the option to generate a variable resolution grid with a custom-set center in NE Arkansas.

I can still try the grid_rotate method but I'm not sure how to incorporate it with custom grids generated using MPAS_Tools.
Click to expand...
 
I have updated the Google Drive to contain the outputs from the generated files from the two scripts running. I included the finished converted file but not the base_mesh file (though it was generated with no errors)
I'm using the Python package of MPAS_Tools version 1.4.0 which installed its dependencies. If I need to use the non-Python package version I can try it as well.

Ming Chen said:
Hi David,

I have tried to repeat your issues in NCAR HPC. Unfortunately I cannot successfully create the MPAS grid. The error message:

Code: 
terminate called after throwing an instance of 'std::invalid_argument'
  what():  Error putting variable triangleAngleQuality: -101
Traceback (most recent call last):
  File "/glade/u/home/chenming/MyDoc/UserCodes/convert_grid2mpas.py", line 18, in <module>
  File "/glade/work/chenming/conda-envs/mchen-npl2025a/lib/python3.12/site-packages/mpas_tools/mesh/creation/build_mesh.py", line 106, in build_spherical_mesh
    check_call(args=args, logger=logger)
  File "/glade/work/chenming/conda-envs/mchen-npl2025a/lib/python3.12/site-packages/mpas_tools/logging.py", line 66, in check_call
    raise subprocess.CalledProcessError(process.returncode, print_args)
subprocess.CalledProcessError: Command 'MpasMeshConverter.x mesh_triangles.nc base_mesh.nc' died with <Signals.SIGABRT: 6>.

This is coming from MpasMeshConverter.x when it tries to write the variable triangleAngleQuality into base_mesh.nc

I looked at mesh_triangles.nc, and it includes all variables without any NaN. Have you ever seen similar issues before? Thanks.
Click to expand...
 
Hi David,

Thank you for uploading all these files. I have tried again but the code failed with the same errors I got yesterday. I apologize that I cannot make it work with your scripts. Due to limited human power in NCAR, I have no time to further explore this issue.

Our software engineers are working on mesh generation using JIGSAW. I will keep you posted once this utility is created successfully. At present, please stay with the available meshes we provide on MPAS homepage.
 
I can try to stick with current meshes, I just don't know how to apply them to my sim with the variable resolution since the pre-existing 15-3km would be way too big to for my PC's computational abilities (especially since MPAS is a bit of a RAM hog). If only I could scale the 3km section of the grid then I could try it instead


1776366846413.png
Ming Chen said:
Hi David,

Thank you for uploading all these files. I have tried again but the code failed with the same errors I got yesterday. I apologize that I cannot make it work with your scripts. Due to limited human power in NCAR, I have no time to further explore this issue.

Our software engineers are working on mesh generation using JIGSAW. I will keep you posted once this utility is created successfully. At present, please stay with the available meshes we provide on MPAS homepage.
Click to expand...
 
You must log in or register to reply here.
Back
Top