WRF 4.6.1 segmentation fault after 11 years of integration, what could be possible causes?

[xgfan_wrf]

Hi WRF community,

I am running a long-term downscaling experiment with WRF 4.6.1 using two nested domains (D01 = 20 km and D02 = 4 km). The model has been running successfully for about 12 years continuously with restarts and has already passed several leap years. However, it crashed at: 2092-02-29_15:20:00

I have repeated the run a couple of times, and the crash occurs at essentially the same model time, suggesting that it may be deterministic rather than a random hardware problem.

The debug output indicates that the failure occurs in the D01 radiation calculation, specifically during the RRTM call:

d01 2092-02-29_15:20:00 Top of Radiation Driver
d01 2092-02-29_15:20:00 CALL cldfra1
d01 2092-02-29_15:20:00 CALL rrtm
d01 2092-02-29_15:20:00 CAM-CLWRF interpolated values______ year: 2092 julian day: 59.63889
d01 2092-02-29_15:20:00 CAM-CLWRF co2vmr: 5.989544397547096E-004 n2ovmr: 3.762597260565899E-007 ch4vmr: 1.711569258096154E-006
forrtl: severe (174): SIGSEGV, segmentation fault occurred

I am using the CAM-CLWRF greenhouse-gas forcing file:
CAMtr_volume_mixing_ratio.SSP245
and the corresponding values around 2092 are approximately:

2090 597.735 375.951 1719.030 65.733 239.122
2091 598.477 376.137 1714.570 64.454 236.575
2092 599.150 376.310 1710.340 63.198 234.053
2093 599.776 376.470 1706.320 61.965 231.558
2094 600.355 376.619 1702.510 60.754 229.089

The interpolated values printed by WRF at the time of the crash appear reasonable.

The model is writing history output every 3 hours, and the:
wrfout_*_2092-02-29_15:00:00
files were successfully written before the crash. I have not found any obviously abnormal values in those files.

For additional context, D02 successfully completed its radiation calculation at the preceding 15:19:45 model time step. The crash occurs when D01 enters its 15:20:00 radiation calculation, before D02 reaches its corresponding 15:20:00 radiation calculation.

I have also seen the following NetCDF messages from input_wrf at 6-hourly times in D01:

d01 2092-02-29_12:00:00 NetCDF error: NetCDF: Attribute not found
d01 2092-02-29_12:00:00 NetCDF error in ext_ncd_get_dom_ti.code REAL, line 83 Element GMT
d01 2092-02-29_12:00:00 NetCDF error: NetCDF: Attribute not found
d01 2092-02-29_12:00:00 NetCDF error in ext_ncd_get_dom_ti.code INTEGER, line 83 Element JULYR
d01 2092-02-29_12:00:00 NetCDF error: NetCDF: Attribute not found
d01 2092-02-29_12:00:00 NetCDF error in ext_ncd_get_dom_ti.code INTEGER, line 83 Element JULDAY

These messages occur repeatedly at the 6-hourly input times, but the model has continued successfully through them for many years, so I am not sure whether they are related to the segmentation fault.

My questions are:

1. Could this be a known problem in the RRTM/CAM-CLWRF code path in WRF 4.6.1, possibly related to the leap day or the calculation of the Julian day?
2. Are there particular source-code routines or array bounds/index calculations that I should inspect?
3. Could the CAM-CLWRF greenhouse-gas interpolation be triggering an out-of-bounds memory access even though the interpolated CO2, N2O, and CH4 values are physically reasonable?
4. Are there compiler/debugging options that would be particularly useful for locating the source of this SIGSEGV?
5. If this is a known software issue, is there a recommended fix or workaround that would allow me to continue the long-term integration without introducing a discontinuity?

For reference, I have attached the namelist.input file and the PBS job submission script, where physics settings and compiler environment are shown. It was running on NCAR Derecho system, if anyone (admin) have access to it, it is located at:

/glade/work/xfan/tmp/crashed_job_mCd79rst446_7141520_dbg300

Any suggestions would be greatly appreciated.

Thanks!
Xingang