WOULD GO OFF TOP (DPTHMX NaN) only with KF(1,10,11) + P3(50,51,52)

[haeun]

Hi,

I am encountering a persistent issue when using the KF-family cumulus schemes together with the P3 microphysics scheme. I have spent quite some time debugging this problem and would appreciate any suggestions or insights.

Below is a summary of what I have found.


------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------

​

Cases that fail / do not fail​

Fails

• WRF v4.4.2
• WRF v4.8
• KF (cu_physics = 1) + P3
• KF Eta (cu_physics = 10) + P3
• MSKF (cu_physics = 11) + P3

Works

• Old KF implementation (cu_physics = 99) + P3
• Any other cumulus scheme with P3
• KF-family schemes with other microphysics schemes

Therefore, the problem only appears when the new KF-family schemes are combined with P3.

Things I have already tried​

I tried several approaches that are commonly suggested for this error:

• decreasing the time step
• increasing the model top
• increasing dzbot
• testing different radiation and physics options

Increasing the model top and dzbot delayed the crash by about 2 hours of simulation time, but the model still stopped after approximately 3 hours from the initial time. Therefore these changes only delay the failure and do not solve the underlying problem.

Debugging results​

Since I need to run KF + P3, I started tracing where the NaNs first appear.

Inside the KF scheme, I found that:

• DPTHMX becomes NaN
• DPTHMX is computed using RHOE
• RHOE becomes NaN because both temperature (T) and water vapor (QV) are already NaN.

I then traced the NaNs further upstream.

1. In module_advect_em.F, the advection flux variables (`fqx`, `fqxl`, `fqy`, `fqyl`, `fqz`, `fqzl`) become NaN.

2. As a result, `advect_tend` becomes NaN in `module_advect_em.F`.

3. The `moist` array then becomes NaN in `module_big_step_utilities_em.F` / `solve_em.F`.

4. `QV` becomes NaN.

5. `T` becomes NaN.

6. `RHOE` becomes NaN.

7. `DPTHMX` becomes NaN inside the KF scheme.

8. The model finally stops with the "WOULD GO OFF TOP" error.

So the KF error appears to be a consequence of NaNs that are generated much earlier during moisture advection....?

​

My question​

At this point, I am not sure what could cause the moisture advection variables (fqx, fqy, fqz, etc.) to become NaN only when using the new KF-family schemes together with P3, while:

• the old KF implementation works,
• other cumulus schemes work,
• and other microphysics schemes also work.

Do you have any suggestions about where else I should look, or whether there are any known compatibility issues between the newer KF implementations and P3?


------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------


I can also provide the restart files (wrfrst*), wrfbdy if that would help reproduce the problem.
Any advice would be greatly appreciated.

Thank you!

 

[haeun]

I have a few updates after additional testing.


I tested WRF v4.4.2, v4.7.1, and v4.8. With WRF v4.7.1, I also tested P3 options 51, 52, and 53. All of them run longer than the default P3 option (50), but they eventually fail with the same segmentation fault. (fault in mskf para)


I also changed the physics configuration so that the MSKF cumulus scheme is applied only to the 9 km parent domain (it is turned off in the 3 km nested domain). In addition, I reduced the time step from 30 s to 20 s. With these changes, the simulation was able to run for about 12 hours before crashing with the same error.


I am currently running another test with an even smaller time step(10s) and epssm = 0.5 to improve numerical stability.


However, I ultimately need to complete a 4-day simulation, and I still do not understand the root cause of this issue. I would really appreciate any suggestions on how to resolve this problem rather than just delaying the crash.