wrf.exe got stuck when writing the second wrfout_d01 file

[ys38079456]

Dear all

I have 3-domain nested model, this model works well for most common cases，but sometimes wrf.exe gets stuck without reporting any warnings or errors, once the bug occurs, it is always stuck at the time of writing the second wrfout_d01 file.

For the case below,a 7-day-simulation begin with 2025-07-28_00:00:00, when wrfout_d03_2025-07-28_00:00:00 is completed, wrf.exe got stuck at the time of writing wrfout_d01_2025-07-29_00:00:00, then wrfout_d01_2025-07-29_00:00:00 remains to be 33.4Mb.

rsl.out.0000 and rsl.error.0000 heve the same ending reporting no errors:

Timing for main: time 2025-07-28_23:59:42 on domain 2: 0.46001 elapsed seconds
Timing for main: time 2025-07-28_23:59:48 on domain 3: 0.09576 elapsed seconds
Timing for main: time 2025-07-28_23:59:54 on domain 3: 0.08116 elapsed seconds
Timing for main: time 2025-07-29_00:00:00 on domain 3: 0.08099 elapsed seconds
Timing for main: time 2025-07-29_00:00:00 on domain 2: 0.45732 elapsed seconds
Timing for main: time 2025-07-29_00:00:00 on domain 1: 1.67706 elapsed seconds
Timing for Writing wrfout_d01_2025-07-29_00:00:00 for domain 1: 1.87068 elapsed seconds
Timing for Writing restart for domain 1: 9.18702 elapsed seconds

rsl.out.0030 differs in size from other rsl.out files:

d01 2025-07-28_13:39:54 1 points exceeded v_cfl = 2 in domain d01 at time 2025-07-28_13:39:54 hours
d01 2025-07-28_13:39:54 Max W: 1 101 38 W: -0.81 w-cfl: 2.00 dETA: 0.01
d01 2025-07-28_13:39:54 1 points exceeded v_cfl = 2 in domain d01 at time 2025-07-28_13:39:54 hours
d01 2025-07-28_13:39:54 Max W: 1 101 38 W: -0.82 w-cfl: 2.00 dETA: 0.01
d01 2025-07-28_13:39:54 1 points exceeded v_cfl = 2 in domain d01 at time 2025-07-28_13:39:54 hours
d01 2025-07-28_13:39:54 Max W: 1 101 38 W: -0.80 w-cfl: 2.00 dETA: 0.01
d01 2025-07-28_13:40:48 1 points exceeded v_cfl = 2 in domain d01 at time 2025-07-28_13:40:48 hours
d01 2025-07-28_13:40:48 Max W: 1 101 38 W: -0.80 w-cfl: 2.00 dETA: 0.01
d01 2025-07-28_15:00:00 Input data is acceptable to use:
d01 2025-07-28_18:00:00 Input data is acceptable to use:
d01 2025-07-28_21:00:00 Input data is acceptable to use:
mediation_integrate.G 1242 DATASET=HISTORY
mediation_integrate.G 1243 grid%id 2 grid%oid 3
mediation_integrate.G 1242 DATASET=HISTORY
mediation_integrate.G 1243 grid%id 1 grid%oid 1
mediation_integrate.G 1242 DATASET=HISTORY
mediation_integrate.G 1243 grid%id 3 grid%oid 4

but i do not think that matters? for this bug always gets stuck when writing the second wrfout_d01 file.
set debug_level=200

d03 2025-07-28_23:59:54 calling inc/HALO_EM_D_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_EM_DA_inline.inc
d03 2025-07-28_23:59:54 calling inc/HALO_EM_SBM_inline.inc
d03 2025-07-28_23:59:54 microphysics_driver: calling wsm6
d03 2025-07-28_23:59:54 calling inc/HALO_EM_THETAM_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_EM_THETAM_inline.inc
d03 2025-07-28_23:59:54 calling inc/HALO_EM_D3_5_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_BDY_EM_D3_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_BDY_EM_MOIST_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_BDY_EM_CHEM_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_BDY_EM_TRACER_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_BDY_EM_SCALAR_inline.inc
d03 2025-07-28_23:59:54 calling inc/HALO_EM_C_inline.inc
d03 2025-07-28_23:59:54 calling inc/PERIOD_BDY_EM_E_inline.inc
d03 2025-07-28_23:59:54 call set_w_surface
d03 2025-07-28_23:59:54 calling inc/HALO_EM_PHYS_W_inline.inc
d03 2025-07-28_23:59:54 calling inc/HALO_EM_PHYS_HCW_inline.inc
d03 2025-07-28_23:59:54 --> TOP OF AFTER ALL RK STEPS
d03 2025-07-28_23:59:54 --> CALLING DIAGNOSTICS DRIVER
d03 2025-07-28_23:59:54 --> TOP OF DIAGNOSTICS PACKAGE
d03 2025-07-28_23:59:54 --> CALL DIAGNOSTICS PACKAGE: ACCUMULATED AND BUCKET DIAGNOSTICS
d03 2025-07-28_23:59:54 calling inc/HALO_EM_E_5_inline.inc
d03 2025-07-28_23:59:54 calling inc/HALO_EM_MOIST_E_5_inline.inc
d03 2025-07-28_23:59:54 calling inc/HALO_EM_SCALAR_E_5_inline.inc
d03 2025-07-29_00:00:00 module_integrate: back from solve interface
d03 2025-07-29_00:00:00 2 module_integrate: back from recursive call to integrate
d03 2025-07-29_00:00:00 module_integrate: calling med_nest_feedback
d03 2025-07-29_00:00:00 calling inc/HALO_INTERP_UP_inline.inc
d03 2025-07-29_00:00:00 calling inc/HALO_INTERP_UP_inline.inc
d03 2025-07-29_00:00:00 calling inc/HALO_EM_FEEDBACK_inline.inc
d03 2025-07-29_00:00:00 module_integrate: back from med_nest_feedback
d02 2025-07-29_00:00:00 1 module_integrate: back from recursive call to integrate
d02 2025-07-29_00:00:00 module_integrate: calling med_nest_feedback
d02 2025-07-29_00:00:00 calling inc/HALO_INTERP_UP_inline.inc
d02 2025-07-29_00:00:00 calling inc/HALO_INTERP_UP_inline.inc
d02 2025-07-29_00:00:00 calling inc/HALO_EM_FEEDBACK_inline.inc
d02 2025-07-29_00:00:00 module_integrate: back from med_nest_feedback
d01 2025-07-29_00:00:00 open_hist_w : opening wrfout_d01_2025-07-29_00:00:00 for writing.
d01 2025-07-29_00:00:00 calling wrf_open_for_write_begin in open_w_dataset
d01 2025-07-29_00:00:00 module_io.F: in wrf_open_for_write_begin, FileName = wrfout_d01_2025-07-29_00:00:00
d01 2025-07-29_00:00:00 calling outsub in open_w_dataset
d01 2025-07-29_00:00:00 back from outsub in open_w_dataset
d01 2025-07-29_00:00:00 calling wrf_open_for_write_commit in open_w_dataset
d01 2025-07-29_00:00:00 back from wrf_open_for_write_commit in open_w_dataset
d01 2025-07-29_00:00:00 med_hist_out: opened wrfout_d01_2025-07-29_00:00:00 as DATASET=HISTORY
d01 2025-07-29_00:00:00 med_restart_out: opening wrfrst_d01_2025-07-29_00:00:00 for writing
d01 2025-07-29_00:00:00 calling wrf_open_for_write_begin in open_w_dataset
d01 2025-07-29_00:00:00 module_io.F: in wrf_open_for_write_begin, FileName = wrfrst_d01_2025-07-29_00:00:00
d01 2025-07-29_00:00:00 calling outsub in open_w_dataset
d01 2025-07-29_00:00:00 back from outsub in open_w_dataset
d01 2025-07-29_00:00:00 calling wrf_open_for_write_commit in open_w_dataset
d01 2025-07-29_00:00:00 back from wrf_open_for_write_commit in open_w_dataset
d01 2025-07-29_00:00:00 med_restart_out: opening wrfrst_d02_2025-07-29_00:00:00 for writing
d01 2025-07-29_00:00:00 calling wrf_open_for_write_begin in open_w_dataset
d01 2025-07-29_00:00:00 module_io.F: in wrf_open_for_write_begin, FileName = wrfrst_d02_2025-07-29_00:00:00
d01 2025-07-29_00:00:00 calling outsub in open_w_dataset

it is even more puzzling that when I do nothing and re-execute wrf.exe to the same case for serval times, this bug sometimes disappears in a very low probability.

Any help I can get with troubleshooting this problem would be much appreciated!
Thank in advance

 

[kwerner]

Hi,
Your rsl* files show numerous CFL errors, which means the model has become unstable. See Segmentation Faults and CFL Errors for some guidance on resolving those errors.