Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

MPAS Segfault Near End of Simulation

A

aroseman

New member
Information:
I have been running a full MPAS run (20-1 km mesh) which runs successfully until segfaulting -6 hours before end of simulation (model runs from 06-30 to 07-03). Restarting also ends up faulting around the same time, but not exactly the same time (RESTART try 1: 2022-07-02_17:50:00, RESTART try 2: 2022-07-02_17:29:48). Everything seems to work fine up until then and I used convert_mpas and haven’t found anything wrong.

Out Logs:
Attached Below
 

Attachments

  • log.atmosphere.0000_Run_RESTART.out.txt
    2.8 MB · Views: 2
  • log.atmosphere.0000_Run_RESTART2.out.txt
    2.6 MB · Views: 2
Last edited:
The trouble happened in the top 6-7 levels. Large vertical motion, high ke and vorticity, even at the vertical grid right below model top can be seen in those levels starting in the history files after 070216 hours.

(1) Currently restarting model with lower timestep to hopefully remove instability.
(2) Should I change the sponge layer strength in MPAS?
Currently, config_zd = 22000 and config_xnutr = 0.2
(3) Currently using 56 vertical levels up to 30km, should i increase the vertical resolution?
 
Last edited:
It would provide some additional information on the geographic location of the large velocities if you set
config_print_detailed_minmax_vel = true
Click to expand...
in your namelist, and that information could be helpful in determining the best course of action (e.g., if the instabilities are developing around significant topography, or near the boundary of a limited-area domain).

Also, it looks from your log file that you're not running a "standard" release of MPAS-A. Is the code you're using from a GSL fork of MPAS-Model? If so, have you tried running with the v8.3.1 code?

Others may be able to comment on the vertical level configuration and on the w-damping coefficient (or other model filters that may help with the instability).
 
Ok, I will turn on that config, thanks!

We are using a fork of MPAS by NSSL, so that could be the problem. I am currently testing with "convective_permitting" to see if the same problem occurs.

Thanks!
 
Please see my comments below:
aroseman said:
The trouble happened in the top 6-7 levels. Large vertical motion, high ke and vorticity, even at the vertical grid right below model top can be seen in those levels starting in the history files after 070216 hours.

(1) Currently restarting model with lower timestep to hopefully remove instability.
Click to expand...
Please keep us updated whether decreasing time step helps.
aroseman said:
(2) Should I change the sponge layer strength in MPAS?
Currently, config_zd = 22000 and config_xnutr = 0.2
Click to expand...
config_xnutr = 0.2 provides strong damping, and I don't think it is necessary to increase its value.
Please increase config_zd = 30000, and see whether it helps stabilize the integration
aroseman said:
(3) Currently using 56 vertical levels up to 30km, should i increase the vertical resolution?
Click to expand...

What is the current vertical resolution of your case, i.e., max and min layer depth?
I am concerned that a thin layer depth may lead to CFL violation.

Please tell me where your case is located in derecho. Thanks.
 
RESTART 1
1. Begin timestep 2022-07-02_17:45:18
global min, max w -18.4396 48.7913
global min, max u -53.4407 53.4900

2. Begin timestep 2022-07-02_17:45:24
global min, max w NaN NaN
global min, max u NaN NaN


3. Model Stops at 2022-07-02_17:50:00

RESTART 2
1. Begin timestep 2022-07-02_17:27:18
global min, max w -20.5746 54.0888
global min, max u -53.0978 53.1182

2. Begin timestep 2022-07-02_17:27:24
global min, max w NaN NaN
global min, max u NaN NaN

3. Model Stops at 2022-07-02_17:29:54

@mgduda
I am also wondering:
(1) why the restarts stop at different places (how does the model chose to stop in the case of instability)
(2) what does it mean for global value to be NaN (is it an explosively large value)

Thanks!
 
Last edited:
@Ming Chen

(1) Decreasing dt from 6 to 3 has no effect on the model crash. Still looking to see how the instability changes.

(2) Also, running with the default "convection_permitting" to see if that changes anything
Note: we are testing MPAS 8.3.0 with NSSL microphysics, which is not part of standard MPAS. Since the instability pattern we see is not common in MPAS, we suspect one of our schemes that differ from "convection_permitting", likely one of NSSL's is the problem.
 
Last edited:
@aroseman

I do believe that you should turn on horizontal momentum damping, e.g.,

Code: 
config_rayleigh_damp_u = true
config number cam damping levels = 5   
config rayleigh damp u timescale days = 6

The above values of damping levels and timescale are default values (MPAS-v8.3.1). I am thinking that you may need to increase damping levels and decreasing damping timescale for your 1-km case.

Please try and let me know whether these options can help suppressing the instability. Thanks!
 
Currently Running Three Tests:
(1) Checking for changes in instability with decreases timestep (dt=6 ==> dt=3)
(2) Running with convective_permitting to see if any physics params are at fault.
(3) Rayleigh Damping: Will keep defaults. Timescale will be 1 day. Original model settings including dt.
 
You must log in or register to reply here.
Top