Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

WRF-Chem stopped writing output but not failed

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

Hi, I have ran into a strange question of WRF-Chem that I hope someone could shed some light on...

Recently I'm running WRF-Chem v3.7 for year 2014. I have run year 2010 successfully, so I applied the same namelist and parameters to year 2014 run, except that I use 2014 meteorology and boundaries. However, every time I submit a job on the server to run, it stops writing output at the same wrf date and time. wrf.exe is still running so the job does not fail before indicated running time. rsl files also look like normal runs, except that it stops writing in the middle. I set the debug levels, and the rsl files still look normal, the last lines of rsl files show:

d01 2014-04-24_07:06:00 calling inc/
d01 2014-04-24_07:06:00 call fddagd_driver
d01 2014-04-24_07:06:00 in PSU FDDA scheme
d01 2014-04-24_07:06:00 call calculate_phy_tend
d01 2014-04-24_07:06:00 call compute_diff_metrics
d01 2014-04-24_07:06:00 calling inc/
d01 2014-04-24_07:06:00 calling inc/
d01 2014-04-24_07:06:00 call bc for diffusion_metrics
d01 2014-04-24_07:06:00 call cal_deform_and_div
d01 2014-04-24_07:06:00 calling inc/
d01 2014-04-24_07:06:00 calling inc/

However the job just stopped at 2014-04-24_07:06:00. The space on server is not exceeded, and other jobs for others years are running normally. So I'm quite confused. I tried cancel the job and resubmit it, and tried start the job at a different time. However, the job always stops at 2014-04-24_07:06:00.

Could someone share any insights on why this may happen? Thank you very much!!
Usually this happens when the model has hit an instability (usually from an unusual meteorological condition) and created a NaN somewhere in the model domain, which has then propagated through the rest of the domain. The exact cause of such instabilities is hard to diagnose - but what you could try is reducing the meteorology timestep (maybe by 20%-50%), to see if that helps the model cope with unusual condition. If it doesn't you might have to check the met inputs for that day, to see if there are any unrealistic values (negative soil moistures, or anything like that), which could cause this issue.