WOULD GO OFF TOP (DPTHMX NaN) only with KF(1,10,11) + P3(50,51,52)

[haeun]

Hi,

I am encountering a persistent issue when using the KF-family cumulus schemes together with the P3 microphysics scheme. I have spent quite some time debugging this problem and would appreciate any suggestions or insights.

Below is a summary of what I have found.


------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------

​

Cases that fail / do not fail​

Fails

• WRF v4.4.2
• WRF v4.8
• KF (cu_physics = 1) + P3
• KF Eta (cu_physics = 10) + P3
• MSKF (cu_physics = 11) + P3

Works

• Old KF implementation (cu_physics = 99) + P3
• Any other cumulus scheme with P3
• KF-family schemes with other microphysics schemes

Therefore, the problem only appears when the new KF-family schemes are combined with P3.

Things I have already tried​

I tried several approaches that are commonly suggested for this error:

• decreasing the time step
• increasing the model top
• increasing dzbot
• testing different radiation and physics options

Increasing the model top and dzbot delayed the crash by about 2 hours of simulation time, but the model still stopped after approximately 3 hours from the initial time. Therefore these changes only delay the failure and do not solve the underlying problem.

Debugging results​

Since I need to run KF + P3, I started tracing where the NaNs first appear.

Inside the KF scheme, I found that:

• DPTHMX becomes NaN
• DPTHMX is computed using RHOE
• RHOE becomes NaN because both temperature (T) and water vapor (QV) are already NaN.

I then traced the NaNs further upstream.

1. In module_advect_em.F, the advection flux variables (`fqx`, `fqxl`, `fqy`, `fqyl`, `fqz`, `fqzl`) become NaN.

2. As a result, `advect_tend` becomes NaN in `module_advect_em.F`.

3. The `moist` array then becomes NaN in `module_big_step_utilities_em.F` / `solve_em.F`.

4. `QV` becomes NaN.

5. `T` becomes NaN.

6. `RHOE` becomes NaN.

7. `DPTHMX` becomes NaN inside the KF scheme.

8. The model finally stops with the "WOULD GO OFF TOP" error.

So the KF error appears to be a consequence of NaNs that are generated much earlier during moisture advection....?

​

My question​

At this point, I am not sure what could cause the moisture advection variables (fqx, fqy, fqz, etc.) to become NaN only when using the new KF-family schemes together with P3, while:

• the old KF implementation works,
• other cumulus schemes work,
• and other microphysics schemes also work.

Do you have any suggestions about where else I should look, or whether there are any known compatibility issues between the newer KF implementations and P3?


------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------


I can also provide the restart files (wrfrst*), wrfbdy if that would help reproduce the problem.
Any advice would be greatly appreciated.

Thank you!

 

[haeun]

I have a few updates after additional testing.


I tested WRF v4.4.2, v4.7.1, and v4.8. With WRF v4.7.1, I also tested P3 options 51, 52, and 53. All of them run longer than the default P3 option (50), but they eventually fail with the same segmentation fault.


I also changed the physics configuration so that the MSKF cumulus scheme is applied only to the 9 km parent domain (it is turned off in the 3 km nested domain). In addition, I reduced the time step from 30 s to 20 s. With these changes, the simulation was able to run for about 12 hours before crashing with the same error.


I am currently running another test with an even smaller time step(10s) and epssm = 0.5 to improve numerical stability.


However, I ultimately need to complete a 4-day simulation, and I still do not understand the root cause of this issue. I would really appreciate any suggestions on how to resolve this problem rather than just delaying the crash.