MPAS Segfault Near End of Simulation

[aroseman]

Information:
I have been running a full MPAS run (20-1 km mesh) which runs successfully until segfaulting -6 hours before end of simulation (model runs from 06-30 to 07-03). Restarting also ends up faulting around the same time, but not exactly the same time (RESTART try 1: 2022-07-02_17:50:00, RESTART try 2: 2022-07-02_17:29:48). Everything seems to work fine up until then and I used convert_mpas and haven’t found anything wrong.

Out Logs:
Attached Below

 

[aroseman]

The trouble happened in the top 6-7 levels. Large vertical motion, high ke and vorticity, even at the vertical grid right below model top can be seen in those levels starting in the history files after 070216 hours.

(1) Currently restarting model with lower timestep to hopefully remove instability.
(2) Should I change the sponge layer strength in MPAS?
Currently, config_zd = 22000 and config_xnutr = 0.2
(3) Currently using 56 vertical levels up to 30km, should i increase the vertical resolution?

 

[mgduda]

It would provide some additional information on the geographic location of the large velocities if you set

config_print_detailed_minmax_vel = true

in your namelist, and that information could be helpful in determining the best course of action (e.g., if the instabilities are developing around significant topography, or near the boundary of a limited-area domain).

Also, it looks from your log file that you're not running a "standard" release of MPAS-A. Is the code you're using from a GSL fork of MPAS-Model? If so, have you tried running with the v8.3.1 code?

Others may be able to comment on the vertical level configuration and on the w-damping coefficient (or other model filters that may help with the instability).

 

[aroseman]

Ok, I will turn on that config, thanks!

We are using a fork of MPAS by NSSL, so that could be the problem. I am currently testing with "convective_permitting" to see if the same problem occurs.

Thanks!

 

[Ming Chen]

Please see my comments below:

The trouble happened in the top 6-7 levels. Large vertical motion, high ke and vorticity, even at the vertical grid right below model top can be seen in those levels starting in the history files after 070216 hours.

(1) Currently restarting model with lower timestep to hopefully remove instability.

Please keep us updated whether decreasing time step helps.

(2) Should I change the sponge layer strength in MPAS?
Currently, config_zd = 22000 and config_xnutr = 0.2

config_xnutr = 0.2 provides strong damping, and I don't think it is necessary to increase its value.
Please increase config_zd = 30000, and see whether it helps stabilize the integration

(3) Currently using 56 vertical levels up to 30km, should i increase the vertical resolution?

What is the current vertical resolution of your case, i.e., max and min layer depth?
I am concerned that a thin layer depth may lead to CFL violation.

Please tell me where your case is located in derecho. Thanks.

 

[mgduda]

@Ming Chen Note that config_zd is the height MSL at which the damping profile begins, so with a model top of 30km, setting config_zd = 30000 may effectively disable the w-damping entirely.

 

[aroseman]

RESTART 1
1. Begin timestep 2022-07-02_17:45:18
global min, max w -18.4396 48.7913
global min, max u -53.4407 53.4900

2. Begin timestep 2022-07-02_17:45:24
global min, max w NaN NaN
global min, max u NaN NaN


3. Model Stops at 2022-07-02_17:50:00

RESTART 2
1. Begin timestep 2022-07-02_17:27:18
global min, max w -20.5746 54.0888
global min, max u -53.0978 53.1182

2. Begin timestep 2022-07-02_17:27:24
global min, max w NaN NaN
global min, max u NaN NaN

3. Model Stops at 2022-07-02_17:29:54

@mgduda
I am also wondering:
(1) why the restarts stop at different places (how does the model chose to stop in the case of instability)
(2) what does it mean for global value to be NaN (is it an explosively large value)

Thanks!

 

[Ming Chen]

@mgduda

Right, ....Sorry ! --- I have the mindset from WRF.

In this case, we may set config_zd = 18000 ( I am not 100% sure what value is the best, just a guess).

 

[aroseman]

@Ming Chen

(1) Decreasing dt from 6 to 3 has no effect on the model crash. Still looking to see how the instability changes.

(2) Also, running with the default "convection_permitting" to see if that changes anything
Note: we are testing MPAS 8.3.0 with NSSL microphysics, which is not part of standard MPAS. Since the instability pattern we see is not common in MPAS, we suspect one of our schemes that differ from "convection_permitting", likely one of NSSL's is the problem.

 

[aroseman]

@Ming Chen What do you think about using config_rayleigh_damp_u?

 

[Ming Chen]

@aroseman

I do believe that you should turn on horizontal momentum damping, e.g.,

config_rayleigh_damp_u = true
config number cam damping levels = 5   
config rayleigh damp u timescale days = 6

The above values of damping levels and timescale are default values (MPAS-v8.3.1). I am thinking that you may need to increase damping levels and decreasing damping timescale for your 1-km case.

Please try and let me know whether these options can help suppressing the instability. Thanks!

 

[aroseman]

Currently Running Three Tests:
(1) Checking for changes in instability with decreases timestep (dt=6 ==> dt=3)
(2) Running with convective_permitting to see if any physics params are at fault.
(3) Rayleigh Damping: Will keep defaults. Timescale will be 1 day. Original model settings including dt.