Unphysical near-surface U-wind and O3 hotspots after about 6 hours in WRF 4.6.1 w/ and w/o chem on newer systems

[Peiyang]

Hello everyone,

I am encountering a reproducibility issue when running WRF and WRF-Chem 4.6.1 on different hardware and software environments.

The same model configuration runs correctly on an older Intel Xeon E5 server with CentOS 7.3. On that system, WRF/WRF-Chem 4.6.1 was compiled using the HPC Stack libraries installed following the instructions at 2. Install and Build the HPC-Stack — HPC-Stack Users Guide documentation (2.2. Non-Container HPC-Stack Installation and Build (General/Linux)). The software environment consists of GNU 9.3, MPICH 3.3.2, HDF5 1.10.6, PnetCDF 1.12.1, and NetCDF-C 4.7.4 / NetCDF-Fortran 4.5.3. The simulation results are physically reasonable, and no abnormal values are observed in either the near-surface wind fields or ozone concentrations.

However, when I compile and run exactly the same WRF/WRF-Chem 4.6.1 case on newer workstations, the results become noticeably different:the simulation starts normally but after approximately six hours of integration, localized and obviously unrealistic hotspot values begin to appear, at least in the first model layer of the U-wind field and O₃ concentration. The systems tested include a Lenovo ThinkStation P920 (dual Intel Xeon Gold 5218R, Ubuntu 24.04 LTS) and an HP mini workstation (AMD Ryzen AI Max+ 395, Ubuntu 24.04 LTS). On both systems, WRF/WRF-Chem was compiled with GNU 12.3, MPICH 4.1.2, HDF5 1.14.3, and NetCDF-C 4.9.2 / NetCDF-Fortran 4.6.1.

To further rule out local software installation issues, I also built WRF/WRF-Chem using the official Ubuntu GNU 9.3 HPC Stack container image provided on the HPC Stack website (2. Install and Build the HPC-Stack — HPC-Stack Users Guide documentation, 2.1. Install and Build the HPC-Stack in a Singularity Container). However, the container build exhibits essentially the same behavior.

At this point, I have not yet been able to identify the root cause of the issue. I would greatly appreciate any suggestions regarding possible causes or recommended debugging approaches.

For reference, I have attached ncview screenshots of the first-layer U-wind and O₃ fields showing the hotspots, together with the namelist.input, configure.wrf, and other relevant files. Please note that the input data (wrfinput, wrfbdy, wrffdda, etc.) used in all experiments are identical.

Thank you very much for your time and assistance.