WRF restart run problem (mpi process killed)

[lslrsgis]

Dear WRF community,

I am runing a simulation for one whole year using WRF with updated fields from auxinputs4 (sst_update=1).

The ./wrf.exe program using 16 cores stopped at wrfout_d02_2015-06-24_06:00:00. Then, I tried a restart run starting from wrfrst_d02_2015-06-20_00:00:00. However, the ./wrf.exe stopped immediately at 2015-06-20_06:00:00.

Time in file: 2015-06-20_00:00:00
Time on domain: 2015-06-20_06:00:00
**WARNING** Time in input file not equal to time on domain **WARNING**
**WARNING** Trying next time in file ...
d01 2015-06-20_06:00:00 Input data processed for aux input 4 for domain 1
d01 2015-06-20_06:00:00 Input data is acceptable to use:
mediation_integrate.G 1943 DATASET=HISTORY
mediation_integrate.G 1944 grid%id 2 grid%oid 3
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpnetcdf.so.3.0 00002AE08DA701D5 for__signal_handl Unknown Unknown
libpthread-2.17.s 00002AE0928E35F0 Unknown Unknown Unknown
libmpi.so.12.0.0 00002AE0918264C7 Unknown Unknown Unknown
libmpi.so.12.0.0 00002AE08FD8CD32 Unknown Unknown Unknown
libmpi.so.12.0.0 00002AE0917D5551 Unknown Unknown Unknown
libmpi.so.12.0.0 00002AE0917D5BB6 PMPI_Wait Unknown Unknown
wrf.exe 0000000002E1D89E Unknown Unknown Unknown
wrf.exe 0000000001436790 Unknown Unknown Unknown
wrf.exe 0000000001BF4EBB Unknown Unknown Unknown
wrf.exe 00000000013722FB Unknown Unknown Unknown
wrf.exe 0000000001202A20 Unknown Unknown Unknown
wrf.exe 000000000055329B Unknown Unknown Unknown
wrf.exe 000000000040C6D1 Unknown Unknown Unknown
wrf.exe 000000000040C68F Unknown Unknown Unknown
wrf.exe 000000000040C622 Unknown Unknown Unknown
libc-2.17.so 00002AE092D16505 __libc_start_main Unknown Unknown
wrf.exe 000000000040C529 Unknown Unknown Unknown

The namelist.input and rsl.out.* rsl.error.* are packaged as attachement.

Thanks.

 

[kwerner]

When looking at your rsl.* files, I see many CFL errors. Take a look at this FAQ that discusses segmentation faults, but also CFL errors and how to overcome them:
https://forum.mmm.ucar.edu/phpBB3/viewtopic.php?f=73&t=133