Segmentation Fault Error on wrf.exe

[hidroman]

Hello everyone,

I know that there are many threads about segmentation faults, but unfortunately none of them helped me resolve my issue. I am confident that the model was compiled successfully and without errors. Since segmentation faults can arise from many different reasons, I would really appreciate your help.

I am attaching my namelist.input, rsl logs, and an example met_em file in an archive for reference.

I run the model using:

mpirun -np X ./wrf.exe

where X is the number of tasks specified in my Slurm script.
The error I receive is:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

The model crashes before reaching the first timestep.

I am running the model on an HPC system, compiled with dmpar. I use 1 node and 1 CPU, and assign the number of tasks using -n in the Slurm script. Regardless of the number of tasks I assign (even with a single core), I still encounter the same segmentation fault.

When I set debug_level = 100, the last messages in the log are:

d02 2023-10-13_00:00:40 Top of Radiation Driver
d02 2023-10-13_00:00:40 calling inc/HALO_PWP_inline.inc
d02 2023-10-13_00:00:40 in SFCLAY

I must use Noah-MP as the land surface model, and also I need to use SFCLAY option 1 or 2. Based on the debug output, it seems that the crash might be related to SFCLAY, but I am not certain.

Any suggestions on what might be causing this issue, or what else I should check, would be greatly appreciated.

Thank you very much for your help.

 

[Ming Chen]

The namelist.input in your wrf.zipfile is empty. There is only a single met_em.d02.2023-10-13_00:00:00.nc included in the zip file.

These incomplete information is not help for me. Can you clarify:

(1) which version of WRF did you run?
(2) what is your input data that drives this case?
(3) please upload your namelist.input for me to take a look.

Thanks.

 

[hidroman]

The namelist.input in your wrf.zipfile is empty. There is only a single met_em.d02.2023-10-13_00:00:00.nc included in the zip file.

These incomplete information is not help for me. Can you clarify:

(1) which version of WRF did you run?
(2) what is your input data that drives this case?
(3) please upload your namelist.input for me to take a look.

Thanks.

Hello Mr. Chen. Sorry for the mistake.

1) WRF V4.7.1 compiled with openmpi 4.1.6 and gcc 12
2) GFS Forecast Data
3) Check attachments please.

Thank you in advance.

 

[Ming Chen]

Hi,
I looked at your namelist.input. Let's change the namelist options below and try again.

time_step =45
radt = 9, 9,
sst_skin = 0
sf_lake_physics =0

delete the following two options:
nproc_x = 8
nproc_y = 7

Please rerun this case (both real.exe and wrf.exe) and let me know whether it works. Thanks.

 

[hidroman]

Hi,
I looked at your namelist.input. Let's change the namelist options below and try again.

time_step =45
radt = 9, 9,
sst_skin = 0
sf_lake_physics =0

delete the following two options:
nproc_x = 8
nproc_y = 7

Please rerun this case (both real.exe and wrf.exe) and let me know whether it works. Thanks.

Hello again, I only turned off the sf_lake_physic then the model ran. I wonder what might be the reason for it? Thank you.

 

[Ming Chen]

To know exactly what is going on, we need to run this case in debug mode.

You can recompile WRF by ./configur -D. Then run this case again. The log file will tell when and where the model crashes first. This will give you some hints what is wrong.

Lake model was not frequently used. It was coupled into WRF in 2010. Since then many updates have been added to the modeling system, and there might be some inconsistencies introduced during the process.

 

[hidroman]

To know exactly what is going on, we need to run this case in debug mode.

You can recompile WRF by ./configur -D. Then run this case again. The log file will tell when and where the model crashes first. This will give you some hints what is wrong.

Lake model was not frequently used. It was coupled into WRF in 2010. Since then many updates have been added to the modeling system, and there might be some inconsistencies introduced during the process.

Will do. Thanks for all the help.