WRF restart crashing before starting

[Parvathy]

Hi,

I am running a long continuous seasonal WRF simulation (WRF v4.5.1) using two nested domains. Initially, I had completed this simulation by running it in shorter segments (about 3 days each) and then stitching the outputs together. This time, I am attempting to run the entire season as one continuous simulation.

The simulation ran successfully until 2019-06-09 07:00 UTC, where it crashed. Since restart files were being written every 24 hours, I attempted to restart the simulation from:

2019-06-09 00:00 UTC
2019-06-08 00:00 UTC

For the restart, I only modified the &time_control section of the namelist.input: restart = .true., run_days, start_month, start_day, and start_hour to match the restart file and no other namelist options were changed. I need to do a 4 month period simulation and so i am using &fdda as well in this simulation.

The restart files (wrfrst_d01 and wrfrst_d02) are readable with ncdump, and the headers appear normal. During the restart, WRF successfully recognizes the restart time and finds the appropriate lateral boundary conditions (as can be seen in the rsl.out.0000) and suddenly crashes. I kept track of the memory through htop and the system memory was not full.

I am attaching the namelist and rsl files for the reference. can anyone help me in this regard? Although .0000 error file is not showing the segmentation fault, the rsl.error.0026 is showing it, so i am uplaoding that as well as a txt file.

@Ming Chen , @kwerner I would be very grateful if you could kindly take a look and help me with this problem. I had seen a similar issue in this: restart run encountered segmentation fault . However, I am not using any mosaic options either.

Thank you in advance for your time and assistance.

 

[kwerner]

Hi, thanks for sending those files. Is it possible for you to try using additional processors to see if that makes any difference, or is 36 your limit? It's unlikely the problem since you've been able to run the simulation so far, but it's worth a check.

Is it possible that your disk space is full (not memory)? Can you check to make sure that's not the case?

If neither of those change anything, will you package all of the rsl* files into a single *.tar or zipped file and attach that, as well? Thanks!

 

[Parvathy]

The server in which we are running the simulation has 128 threads. I ran the simulation with all the threads and still the same error is there. i did grep the segmentation issue :
rsl.error.0042:128 : program received signal SIGSEGV: Segmentation fault - invalid memory reference.
rsl.error.0046:128: program received signal SIGSEGV: Segmentation fault - invalid memory reference.
rsl.error.0082:128: program received signal SIGSEGV: Segmentation fault - invalid memory reference.

but the system itself has 2.9TB storage available and there is at least 900 GB ram space free. I am starting my simulation where there is a restart file as well. I tried to change the run_days as well but there is no variation in the error. i am attaching the zipped rsl files.