WRF segmentation error

[Molan]

Hello!

I am running WRF 4.5 over parts of China (including the Tibetan Plateau). My current run keeps crashing hours into the simulation. I have checked the CFL, but there is no error.

I am using 48 cores to run the simulation and I am using auto ts with max = 100.

The simulation seems to proceed just fine before abruptly crashing.

I have tried to activate the smoothing of the geography at the boundaries (smooth_cg_topo), yet nothing worked.

Here, I have provided the namelist files.

Please help, and thank you!

 

[William.Hatheway]

Hello!

I am running WRF 4.5 over parts of China (including the Tibetan Plateau). My current run keeps crashing hours into the simulation. I have checked the CFL, but there is no error.

I am using 48 cores to run the simulation and I am using auto ts with max = 100.

The simulation seems to proceed just fine before abruptly crashing.

I have tried to activate the smoothing of the geography at the boundaries (smooth_cg_topo), yet nothing worked.

Here, I have provided the namelist files.

Please help, and thank you!

try these steps.

1. run real.exe and export all the log files to a zip folder, i would rename the real.exe rsl file something like real.rsl*
2. run wrf.exe and export all the log files to a zip folder. i would rename the wrf.exe rsl files something like wrf.rsl*
3. if it fails again try running a clean wrf without the wrf data assimilation, i doubt that's the issue but that might be a source of the problem.

 

[Molan]

try these steps.

1. run real.exe and export all the log files to a zip folder, i would rename the real.exe rsl file something like real.rsl*
2. run wrf.exe and export all the log files to a zip folder. i would rename the wrf.exe rsl files something like wrf.rsl*
3. if it fails again try running a clean wrf without the wrf data assimilation, i doubt that's the issue but that might be a source of the problem.

I did what you suggested, and it is still crashing.
No errors at all, only the segmentation error in core 22
grep -i "segmentation" rsl.error.*
rsl.error.0022rogram received signal SIGSEGV: Segmentation fault - invalid memory reference.

here you mentioned changing the name to wrf.rsl.erro, you mean after the run? i did change the real.exe ouputs (there were no error messages in them as well)

In this run i didn't apply nudging at all yet the model stil crash

 

[meteoadriatic]

You need to smooth the whole domain if your domain includes steep slopes. The basic way to do that is to increase number of passes of smoothing of the HGT field in GEOGRID.TBL and rerun geogrid.exe. Also avoid placing boundary zones around complex terrain as much as possible.

 

[Ming Chen]

Please also increase epssm to a larger value (e.g., 0.9) and turn off fdda and adaptive time step. Let's try to reduce the extra complexity and see how the case works.

 

[Molan]

You need to smooth the whole domain if your domain includes steep slopes. The basic way to do that is to increase number of passes of smoothing of the HGT field in GEOGRID.TBL and rerun geogrid.exe. Also avoid placing boundary zones around complex terrain as much as possible.

i have try to implement 3 smoothing passes but still, it keeps failing
no cfl error, just this one.
rsl.error.0002rogram received signal SIGSEGV: Segmentation fault - invalid memory reference.
when i try to run the same model with my other work station it is going well, no crash so far even without the smoothing degree increased, however that station is way slower than the one im trying to run the model on. Both has wrf 4.5

 

[Molan]

Please also increase epssm to a larger value (e.g., 0.9) and turn off fdda and adaptive time step. Let's try to reduce the extra complexity and see how the case works.

okay, i just tried this, and it still crashed after 20 hours of simulation time. however, i would like to ask why when i run this case with the same configuration using the same wrf version 4.5, same geo static data, even without increasing the smoothing passes on other workstation it works normally. I have run a smaller domain case in the current workstation and it has no problem. I am trying to change to the current workstation since it is 1.4 times faster than my other workstation.

 

[Ming Chen]

Hi,

If the same case works in some other machines, it implies that the model and the data both are fine.

This case failed in the machine you are currently using, suggesting that this failure could be attributed to machine issues and/or libraries installed. Can you try to install the same libs using the same compilers that work in other machines, then try again?

okay, i just tried this, and it still crashed after 20 hours of simulation time. however, i would like to ask why when i run this case with the same configuration using the same wrf version 4.5, same geo static data, even without increasing the smoothing passes on other workstation it works normally. I have run a smaller domain case in the current workstation and it has no problem. I am trying to change to the current workstation since it is 1.4 times faster than my other workstation.

 

[Molan]

Hi,

If the same case works in some other machines, it implies that the model and the data both are fine.

This case failed in the machine you are currently using, suggesting that this failure could be attributed to machine issues and/or libraries installed. Can you try to install the same libs using the same compilers that work in other machines, then try again?

i have tried to replicate the libraries and MPI of the functioning workstation with the major difference being the MPICH changed to Open MPI, however it is still failing after one simulation day. I also have tried to run the outer domain alone on my previous environment it works normally (with nudging), then i introduced one inner domain when i run it on 48 cores it crashed after 3.5 simulation days, and one i run it only on 32 cores it run pass that. Im suspecting that as the processing load increase transition of data between cores seems to face some challenge and one core shut off it is process leading to the crash of the simulation

i keep getting such error

rsl.error.0038rogram received signal SIGSEGV: Segmentation fault - invalid memory reference.

(base) cfd@cy-X-N-T:/media/cfd/DATA/Molan/Downscaling_OMPI/2innderdomain_OMPI$ tail -40 rsl.error.0038
d01 2022-05-01_00:00:00 W-DAMPING BEGINS AT W-COURANT NUMBER = 1.00000000
d01 2022-05-01_00:00:00 ----------------------------------------
d02 2022-05-01_00:00:00 Input data is acceptable to use:
d02 2022-05-01_00:00:00 Input data processed for aux input 4 for domain 2
Tile Strategy is not specified. Assuming 1D-Y
WRF TILE 1 IS 78 IE 115 JS 380 JE 442
WRF NUMBER OF TILES = 1
d03 2022-05-01_00:00:00 Input data is acceptable to use:
d03 2022-05-01_00:00:00 Input data processed for aux input 4 for domain 3
Tile Strategy is not specified. Assuming 1D-Y
WRF TILE 1 IS 123 IE 183 JS 357 JE 416
WRF NUMBER OF TILES = 1
d01 2022-05-01_06:00:00 Input data is acceptable to use:
d01 2022-05-01_06:00:00 Input data processed for aux input 4 for domain 1
d01 2022-05-01_06:00:00 Input data is acceptable to use:
d01 2022-05-01_06:00:00 Input data processed for aux input 10 for domain 1
d01 2022-05-01_06:00:00 Input data is acceptable to use:
d02 2022-05-01_06:00:00 Input data is acceptable to use:
d02 2022-05-01_06:00:00 Input data processed for aux input 4 for domain 2
d03 2022-05-01_06:00:00 Input data is acceptable to use:
d03 2022-05-01_06:00:00 Input data processed for aux input 4 for domain 3

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x14ffd05c92ed in ???
#1 0x14ffd05c8503 in ???
#2 0x14ffcf461f0f in ???
#3 0x55cca76f1b35 in ???
#4 0x55cca76f8eff in ???
#5 0x55cca742fd43 in ???
#6 0x55cca6fefb3d in ???
#7 0x55cca6e87883 in ???
#8 0x55cca5f9a11e in ???
#9 0x55cca5f9a6ff in ???
#10 0x55cca5f21b47 in ???
#11 0x55cca5f214ae in ???
#12 0x14ffcf444c86 in ???
#13 0x55cca5f214e9 in ???
#14 0xffffffffffffffff in ???

 

[meteoadriatic]

Is the number of processes started between two machines the same or different? If it is not the same value, due to different domain decomposition it can still fail on one machine and survive on other if the run is marginally unstable - like I suspect due to terrain steepness when you turn on nest.

For test, go nuclear, set number of smoothing passes over 10, repeat geogrid.exe and simulation and see what happens. If survives - it is too steep terrain, most probably within nest boundary zone.