wrf.exe stops running suddenly

[antonianna]

Hello everyone!

I have this nested domains connfiguration. "real.exe" runs successfully, but "wrf.exe" stops running suddenly and very quickly at 00:10:06 without shownig an error or message. I don't know what is wrong..Do you have any idea of why this simulation cannot run? I have attached the "namelist.input", "namelist.wps" and the "rsl.out.0000" and "rsl.error.0000" files for your assistance.

The namelist.input file contains:

&time_control
run_days = 20,
run_hours = 0,
run_minutes = 0,
run_seconds = 0,
start_year = 2019, 2019, 2019,
start_month = 11, 11, 11,
start_day = 10, 10, 10,
start_hour = 00, 00, 00,
start_minute = 00, 00, 00,
start_second = 00, 00, 00,
end_year = 2019, 2019, 2019,
end_month = 11, 11, 11,
end_day = 30, 30, 30,
end_hour = 00, 00, 00,
end_minute = 00, 00, 00,
end_second = 00, 00, 00,
interval_seconds = 21600,
input_from_file = .true., .true., .true.,
history_interval = 60, 60, 60,
frames_per_outfile = 1000, 1000, 1000,
restart = .false.,
restart_interval = 1440,
io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2
debug_level = 5
auxhist3_outname = "wrfxtrm_<domain>_<date>",
auxhist3_interval = 1440,
frames_per_auxhist3 = 7,
io_form_auxinput4 = 2
auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360,360,360
/

&domains
time_step = 54,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 3,
e_we = 538, 526, 514,
e_sn = 293, 283, 271,
e_vert = 50, 50, 50,
p_top_requested = 5000,
num_metgrid_levels = 31,
num_metgrid_soil_levels = 4,
dx = 9000, 3000, 1000,
dy = 9000, 3000, 1000,
grid_id = 1, 2, 3,
parent_id = 1, 1, 2,
i_parent_start = 1, 182, 178,
j_parent_start = 1, 100, 97,
parent_grid_ratio = 1, 3, 3,
parent_time_step_ratio = 1, 3, 3,
feedback = 0,
smooth_option = 0,
/

&physics
mp_physics = 8, 8, 8,
ra_lw_physics = 4, 4, 4,
ra_sw_physics = 4, 4, 4,
radt = 9, 9, 9,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
bl_pbl_physics = 1, 1, 1,
bldt = 0, 0, 0,
cu_physics = 1, 1, 0,
cudt = 5, 5, 5,
isfflx = 1,
ifsnow = 0,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
sf_urban_physics = 0, 0, 0,
maxiens = 1,
maxens = 3,
maxens2 = 3,
maxens3 = 16,
ensdim = 144,
sst_skin = 1,
sst_update = 1,
tmn_update = 1,
/

&fdda
/

&dynamics
w_damping = 0,
diff_opt = 1,
km_opt = 4,
diff_6th_opt = 0, 0, 0,
diff_6th_factor = 0.12, 0.12, 0.12,
base_temp = 290.,
damp_opt = 0,
zdamp = 5000., 5000., 5000.,
dampcoef = 0.2, 0.2, 0.2,
khdif = 0, 0, 0,
kvdif = 0, 0, 0,
non_hydrostatic = .true., .true., .true.,
moist_adv_opt = 1, 1, 1,
scalar_adv_opt = 1, 1, 1,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false., .false.,
nested = .false., .true., .true.,
/

&grib2
/

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/

 

[Ming Chen]

Your namelist.input looks fine. Since this case stops immediately after wrfe.xe starts, I am suspicious this is either a memory issue or a data issue.
To identify possible reason, please run this case with a single domain, i.e., max_dom =1. If this test case works, it indicates that the input data is fine. If not, please clarify what data you ungrib.
If this case failed due to memory issue, you may increase the number of processors, which will give you larger memory, and try again.
Hope this is helpful. Please keep me updated about the case. Thanks.

 

[antonianna]

Thank you for your quick response! I tried running this case with max_dom=1, and it ran without any issues. However, when I increased the number of processors (480 processor cores on a supercomputer), and run it with max_dom=2, the simulation only ran for 1 day before stopping. Could it be possible that more processor cores are needed due to the large number of e_sn and e_we? Could the issue be related to sst_update=1? Please help me, i don't know what's wrong for a long time now!

 

[Ming Chen]

Since this case can run for a single domain and run for more than 1 days before it crashed, I believe it is not a data issue.

I am suspicious this case crashed due to issues in WRF physcis/dynamics. What error message did you get in your RSL files?

 

[antonianna]

The only error/warning that I took in the rsl.out, rsl.error files is as follows:
" *************************************
No physics suite selected.
Physics options will be used directly from the namelist.
*************************************
Domain # 1: dx = 9000.000 m
Domain # 2: dx = 3000.000 m
Domain # 3: dx = 1000.000 m
Setting blank km_opt entries to domain #1 values.
--> The km_opt entry in the namelist.input is now max_domains.
Setting blank diff_opt entries to domain #1 values.
--> The diff_opt entry in the namelist.input is now max_domains.
--- WARNING: traj_opt is zero, but num_traj is not zero; setting num_traj to zero.
--- NOTE: qna_update is 0, setting io_form_auxinput17 = 0 and auxinput17_interval = 0 for all domains
--- NOTE: qna_update is 0, setting io_form_auxinput17 = 0 and auxinput17_interval = 0 for all domains
--- NOTE: qna_update is 0, setting io_form_auxinput17 = 0 and auxinput17_interval = 0 for all domains
--- NOTE: grid_fdda is 0 for domain 1, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 1, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 1, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: grid_fdda is 0 for domain 2, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 2, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 2, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: grid_fdda is 0 for domain 3, setting gfdda interval and ending time to 0 for that domain.
--- NOTE: both grid_sfdda and pxlsm_soil_nudge are 0 for domain 3, setting sgfdda interval and ending time to 0 for that domain.
--- NOTE: obs_nudge_opt is 0 for domain 3, setting obs nudging interval and ending time to 0 for that domain.
--- NOTE: bl_pbl_physics /= 4, implies mfshconv must be 0, resetting
Need MYNN PBL for icloud_bl = 1, resetting to 0
--- NOTE: RRTMG radiation is used, namelist value for o3input (ozone input) is used
--- NOTE: One of the RRTMG radiation schemes is in use, setting: levsiz=59, alevsiz=12, no_src_types=6
--- NOTE: num_soil_layers has been set to 4
WRF V4.4 MODEL "

...and the process stopped at the 00:04:00 minute (during the run of 3 domains), indicating that this simulation can only run for 4 minutes.

Again, the physics options are these:

&physics
mp_physics = 8, 8, 8,
ra_lw_physics = 4, 4, 4,
ra_sw_physics = 4, 4, 4,
radt = 9, 9, 9,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
bl_pbl_physics = 1, 1, 1,
bldt = 0, 0, 0,
cu_physics = 1, 1, 0,
cudt = 5, 5, 5,
isfflx = 1,
ifsnow = 0,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
sf_urban_physics = 0, 0, 0,
maxiens = 1,
maxens = 3,
maxens2 = 3,
maxens3 = 16,
ensdim = 144,
sst_skin = 1,
sst_update = 1,
tmn_update = 1,

Do you have any ideas for solving this problem?

Thank you!