Hello everyone,
I am running WRF 4.6 with three two-way nested domains, and the model crashes with a segmentation fault shortly after integration begins. Below is my configuration and the troubleshooting I have done so far. I would appreciate any suggestions.
wrf.exe starts normally but crashes shortly after integration begins, following the initial simulation time of 2020-08-10 06:00:00. MPI rank 60 reports:
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Using addr2line, I traced the crash to the following call stack:
sf_sfclayrev_mp_psim_stable_<br>sf_sfclayrev_run<br>module_surface_driver<br>first_rk_step_part1<br>solve_em
The corresponding source file is:
phys/physics_mmm/sf_sfclayrev.F90
nzol = int(zolf*100.)<br>rzol = zolf*100.-nzol
To investigate, I added an IEEE floating-point validity check to this function.
After recompiling and rerunning the model, I obtained the following message in rsl.error.0060:
PSIM_DEBUG: NaN or Inf, zolf= NaN
This confirms that zolf is already NaN when psim_stable() is called.
I suspect that the invalid value originates upstream in the surface-layer calculations, but I have not yet identified which variable first becomes NaN.
The main modified files are:
module_mp_nssl_2mom.F<br>module_microphysics_driver.F<br>solve_em.F<br>Registry.EM_COMMON
The modified code compiles successfully, but I am unsure whether these changes could indirectly affect calculations in other physics schemes.
I have also located another WRF 4.6 executable in which I have not found the relevant custom diagnostic markers. I plan to run a comparison using the same initial conditions, boundary conditions, and namelist settings, but I do not yet have the final comparison results.
Any suggestions on tracing the first occurrence of NaN or identifying potential causes would be greatly appreciated.
the following is my namelist.input
please
I am running WRF 4.6 with three two-way nested domains, and the model crashes with a segmentation fault shortly after integration begins. Below is my configuration and the troubleshooting I have done so far. I would appreciate any suggestions.
1. Model configuration
- WRF version: 4.6
- Compiler environment: Intel ifort 19.0.4 + MPI
- Platform: HPC cluster using Slurm
- MPI processes: 180 across 10 nodes, with 18 processes per node
- Nesting: Three two-way nested domains, feedback=1
- Horizontal grid spacing: 12 km, 4 km, and 1.333 km
- Outer-domain time step: 30 s
- Vertical levels: 50
- Model-top pressure: p_top_requested=5000
- Microphysics: mp_physics=17,17,17 (NSSL)
- Surface layer: sf_sfclay_physics=1,1,1
- Land surface: sf_surface_physics=1,1,1 (SLAB)
- Urban canopy: sf_urban_physics=0,0,0
- Stack size: ulimit -s unlimited
2. Error details
real.exe completed successfully and generated wrfinput_d01, wrfinput_d02, wrfinput_d03, and wrfbdy_d01.wrf.exe starts normally but crashes shortly after integration begins, following the initial simulation time of 2020-08-10 06:00:00. MPI rank 60 reports:
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Using addr2line, I traced the crash to the following call stack:
sf_sfclayrev_mp_psim_stable_<br>sf_sfclayrev_run<br>module_surface_driver<br>first_rk_step_part1<br>solve_em
The corresponding source file is:
phys/physics_mmm/sf_sfclayrev.F90
3. NaN identified at the input to
In psim_stable(), the input variable zolf is used to calculate a lookup-table index:nzol = int(zolf*100.)<br>rzol = zolf*100.-nzol
To investigate, I added an IEEE floating-point validity check to this function.
After recompiling and rerunning the model, I obtained the following message in rsl.error.0060:
PSIM_DEBUG: NaN or Inf, zolf= NaN
This confirms that zolf is already NaN when psim_stable() is called.
I suspect that the invalid value originates upstream in the surface-layer calculations, but I have not yet identified which variable first becomes NaN.
4. Troubleshooting completed so far
- I checked the logs and have not found any explicit CFL violation messages.
- The logs contain:
W-DAMPING BEGINS AT W-COURANT NUMBER = 1.000000
However, this appears to describe the threshold for activating vertical-velocity damping, rather than an actual CFL violation. - p_top_requested is already set to 5000 Pa, so my configuration differs from forum cases involving an unusually low model top.
- The urban canopy model is disabled, so the solution involving changes to TS_SCHEME in URBPARM.TBL does not appear applicable.
- The 30 s time step is relatively conservative for the 12 km outer domain.
- I would like to retain two-way nesting with feedback=1, so I am looking for a solution that preserves this configuration.
5. Important note about custom code modifications
The executable was not built from entirely unmodified WRF 4.6 source code. I added custom diagnostic outputs to the NSSL microphysics scheme, including hydrometeor tendencies and water-vapor budget terms.The main modified files are:
module_mp_nssl_2mom.F<br>module_microphysics_driver.F<br>solve_em.F<br>Registry.EM_COMMON
The modified code compiles successfully, but I am unsure whether these changes could indirectly affect calculations in other physics schemes.
I have also located another WRF 4.6 executable in which I have not found the relevant custom diagnostic markers. I plan to run a comparison using the same initial conditions, boundary conditions, and namelist settings, but I do not yet have the final comparison results.
Any suggestions on tracing the first occurrence of NaN or identifying potential causes would be greatly appreciated.
the following is my namelist.input
please