WRF-Chem stopped writing output but not failed

[wrfchem_beginner]

Hi, I have ran into a strange question of WRF-Chem that I hope someone could shed some light on...

Recently I'm running WRF-Chem v3.7 for year 2014. I have run year 2010 successfully, so I applied the same namelist and parameters to year 2014 run, except that I use 2014 meteorology and boundaries. However, every time I submit a job on the server to run, it stops writing output at the same wrf date and time. wrf.exe is still running so the job does not fail before indicated running time. rsl files also look like normal runs, except that it stops writing in the middle. I set the debug levels, and the rsl files still look normal, the last lines of rsl files show:

d01 2014-04-24_07:06:00 calling inc/HALO_EM_FDDA_SFC_inline.inc
d01 2014-04-24_07:06:00 call fddagd_driver
d01 2014-04-24_07:06:00 in PSU FDDA scheme
d01 2014-04-24_07:06:00 call calculate_phy_tend
d01 2014-04-24_07:06:00 call compute_diff_metrics
d01 2014-04-24_07:06:00 calling inc/HALO_EM_TKE_C_inline.inc
d01 2014-04-24_07:06:00 calling inc/PERIOD_BDY_EM_A1_inline.inc
d01 2014-04-24_07:06:00 call bc for diffusion_metrics
d01 2014-04-24_07:06:00 call cal_deform_and_div
d01 2014-04-24_07:06:00 calling inc/HALO_EM_HELICITY_inline.inc
d01 2014-04-24_07:06:00 calling inc/HALO_EM_TKE_D_inline.inc

However the job just stopped at 2014-04-24_07:06:00. The space on server is not exceeded, and other jobs for others years are running normally. So I'm quite confused. I tried cancel the job and resubmit it, and tried start the job at a different time. However, the job always stops at 2014-04-24_07:06:00.

Could someone share any insights on why this may happen? Thank you very much!!

 

[douglowe]

Usually this happens when the model has hit an instability (usually from an unusual meteorological condition) and created a NaN somewhere in the model domain, which has then propagated through the rest of the domain. The exact cause of such instabilities is hard to diagnose - but what you could try is reducing the meteorology timestep (maybe by 20%-50%), to see if that helps the model cope with unusual condition. If it doesn't you might have to check the met inputs for that day, to see if there are any unrealistic values (negative soil moistures, or anything like that), which could cause this issue.