Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Segmentation Fault when producing 24km initial conditions

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

mcauliffej12

New member
Hello,

I am trying to run the MPAS model using the 24km mesh, however my attempts have been unsuccessful. I was able to run MPAS using the 120km mesh grid, however I cannot seem to create the static file when using 24km mesh. I have pasted the last two lines of the log.init_atmosphere file where the software crashes:

Using option 'linear' for vertical extrapolation of temperature
WARNING: MPAS I/O: Overwriting existing record 1 in output file x1.1024002.static.nc

It may be a memory issue but wanted to cover all my bases first. I made sure to make sure the limit settings were set to unlimited as well.

I have attached my namelist and streams files for the init_atmosphere model and attached a screenshot of the segmentation fault error message.
 

Attachments

  • namelist.init_atmosphere.txt
    1.4 KB · Views: 22
  • streams.init_atmosphere.txt
    999 bytes · Views: 27
  • screen_capture_qdb_seg_fault.png
    screen_capture_qdb_seg_fault.png
    5.9 KB · Views: 368
Everything looks good to me in your namelist and streams files. I'd agree that this sounds like it may be a memory issue. How much memory is available on the node/machine where you're running the static interpolation step? If you've compiled the init_atmosphere core with double-precision reals, it might be worth cleaning and recompiling with PRECISION=single .
 
We were able to fix the error and generate the static file by recompiling MPAS with pio 2 and netcdf. However, now we are having difficulties creating the init.nc file. The last line in the log file is:

----- done assigning dimensions from Registry.xml -----

We felt that the assigned vertical levels were an issue ( we were using 38 vertical levels for 2022 GFS files ). We tried using an older GFS dataset from Jan 2021 but still failed using 38. We also tried the 2022 GFS files using 42 config_nfglevels, but reached the same conclusion. Comparing to previous runs, we assume that the init_model is crashing when working with the GFS data. There is no error file that is generated. Do you know what could be causing this issue?
 
After the " ----- done assigning dimensions from Registry.xml -----" log message is printed, I think the MPAS framework attempts to allocate fields. Are you running with just a single MPI task? If so, how much memory is available on the node/machine where you're running?
 
Using more nodes/processors fixed the issue. We were able to create the init_file. And move onto the atmosphere_model.

We are running into some a problem when running the atmosphere_model. We are using 144 processors (4 nodes with 36 processors each), but the model seems to crash. It doesn't cease operations, yet remains in the same part of the code for up to 2 hours. We assume that something is wrong, and I have attached our log file (failed.log) to show where the software stops.
 

Attachments

  • failed.log
    8.5 KB · Views: 21
Since the segmentation fault issue when creating ICs has been resolved by using more nodes, it might be good to start a new topic for the problem you're seeing when running the model itself.
 
Top