"forrtl: error (78): process killed (SIGTERM)" after running WRF for few days

[tikargrg]

Dear All,
I'm trying to execute a one-year simulation in WRF within a single domain. Initially, the simulation progresses smoothly for a few days. However, it abruptly terminates at a certain point with the error message "forrtl: error (78): process killed (SIGTERM)".
Previously, I had successfully completed a run with the same setup using the "CONUS" suite physics, "time_step =60" and without "sst_update".

In my current setup I changed physics, "sst_update".

&time_control
...
auxinput4_inname = "wrflowinp_d<domain>",
auxinput4_interval = 180,
io_form_auxinput4 = 2,
..
&physics
..
sst_update = 1,

I attempted to address the issue by adjusting the different "time_step" parameter from 60 to 15, but unfortunately, the problem persists.

Attached is the "namelist.input" and "rsl.error.0000" files together.

 

[kwerner]

Hi,
Can you package all of your rsl* files together into a single *.tar file and attach that?
When you ran this previously (and it worked), it seems like there were probably quite a few different factors than this case. Several physics options were different, and if you were using a time_step of 60, then DX/DY must have been different. Is that correct?

 

[tikargrg]

Hi, thanks for the response!
I apologize for the delay in getting back to you. In the previous run, despite having the CONUS suite enabled, it seems that due to other physics settings being turned on, the CONUS suite was overwritten. Hence, I believe there are no issues with the physics configuration. Upon reviewing the output files of both the successful and current runs, I found that they both utilize the same physics settings.

To address the situation, I have recently re-run the simulation by enabling spectral nudging and also increasing the <auxinput4_interval> parameter from 180 to 720 and got terminated after few days of wrf run.

Here are the changes made for sst update in the <namelist.input> file than previous successful run (without sst update):
&time_control
...

auxinput4_inname = "wrflowinp_d<domain>",
auxinput4_interval = 720,
io_form_auxinput4 = 2,
..

&physics
..

sst_update = 1,
..

&fdda
grid_fdda = 2,
..
It's worth noting that I haven't altered the DX/XY settings, and I'm solely focusing on the real and wrf cases.

I've endeavored to attach all the error files you requested, alongside the current <namelist.input> file and the previous successful <namelist.input> configuration (excluding the SST update). However, due to the large file sizes, I've only uploaded <rsl.error.0000>, <namelist.input>, and <successful_namelist.input> files for now.

Please note that I'm currently working on derecho, so you can temporarily access my path at: /glade/derecho/scratch/tgurung/wrf/wrf_HMA12_2016/WRF/test/em_real


Thank you for your time!

 

[kwerner]

Now it's my turn to apologize for the delay. Thank you for your patience. Since you modified two major things in the namelists (added SST-update and spectral nudging), can you do a test where you remove one of those at a time to see if you can pinpoint which specific new option is causing the issue?

It is very helpful to be able to see your Derecho directory, so I appreciate that. However, when I looked there, all of the rsl* files in that specific directory are from March 17th, which don't correspond to the rsl* file you shared in the zipped file. After you run the above tests, will you check the path and let me know where I can find those tests? If you save all the files for each test (including all wrfinput*, wrfbdy_d01, wrflowinp*, and any nudging files, namelists, and rsl files) for each case, that would be great! Thanks!