Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

WRF_CHEM Hangs

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

martin

New member
Dear WRF Support,

We are running WRF-CHEM version 3.8.1 on a cluster. The simulation is running fine for the course domain. However, when the namelist is modified to get the simulations for three domains (course, intermediate and fine), the wrf.exe executable hangs after 41 minutes of the processed simulation. Is it as issue related to the Physics parameters?

The executable hangs as the following line:
d03 2015-04-17_00:41:06 calling inc/HALO_CUP_G3_IN_inline.inc

Any guidance would be appreciated.

Timing for main: time 2015-04-17_00:41:06 on domain 3: 9896.66211 elapsed seconds
d03 2015-04-17_00:41:06 in med_latbound_in
d03 2015-04-17_00:41:06 module_integrate: calling solve interface
d03 2015-04-17_00:41:06 call HALO_RK_CHEM
d03 2015-04-17_00:41:06 calling inc/HALO_EM_CHEM_E_5_inline.inc
d03 2015-04-17_00:41:06 calling inc/HALO_EM_SCALAR_E_5_inline.inc
d03 2015-04-17_00:41:06 grid spacing, dt, time_step_sound= 3000.00000 18.0000000 4
d03 2015-04-17_00:41:06 call rk_step_prep
d03 2015-04-17_00:41:06 calling inc/HALO_EM_A_inline.inc
d03 2015-04-17_00:41:06 calling inc/PERIOD_BDY_EM_A_inline.inc
d03 2015-04-17_00:41:06 call rk_phys_bc_dry_1
d03 2015-04-17_00:41:06 call init_zero_tendency
d03 2015-04-17_00:41:06 calling inc/HALO_EM_PHYS_A_inline.inc
d03 2015-04-17_00:41:06 call phy_prep
d03 2015-04-17_00:41:06 DEBUG wrf_timetoa(): returning with str = [2015-04-17_00:41:06]
d03 2015-04-17_00:41:06 call radiation_driver
d03 2015-04-17_00:41:06 Top of Radiation Driver
d03 2015-04-17_00:41:06 calling inc/HALO_PWP_inline.inc
d03 2015-04-17_00:41:06 call surface_driver
d03 2015-04-17_00:41:06 SST_UPDATE is on
d03 2015-04-17_00:41:06 in MYJSFC
d03 2015-04-17_00:41:06 in NOAH DRV
d03 2015-04-17_00:41:06 call pbl_driver
d03 2015-04-17_00:41:06 in MYJPBL
d03 2015-04-17_00:41:06 call cumulus_driver
d03 2015-04-17_00:41:06 calling inc/HALO_CUP_G3_IN_inline.inc
 

Attachments

  • namelist.input
    8.1 KB · Views: 113
  • namelist.wps
    1.6 KB · Views: 68
Probably it's a CFL error - maybe your transport timestep is a little too long, causing instabilities in the solution which create NaNs (and these then crash the halo communication routines, which is where the model hung for you).

I'd recommend reducing your time_step to begin with - maybe start with 300 - and see if that solves the problem for you. If it does then you can play around with the time_step settings (such as the parent_time_step_ratio) to try to find a solution which works for you (is both stable, and doesn't increase the expense of the simulation too much).
 
Hi everyone,

I am facing a similar problem. I am running a 3-domain run for 10 days simulation and WRF-Chem hangs after simulating the first 9 hours. I tried to run with different numbers of processors and reduced the timesteps, but they all hung at the same point (calling inc/HALO_EM_COUPLE_B_inline.inc). Also, it seems that the simulation failed when moving from the coarse domain to the nest domain?

Here is the last few rows of the rsl.error (the file is also compressed and attached)
Code:
d01 2018-06-05_10:00:00  DEBUG wrf_timetoa():  returning with str = [2018-06-05_10:00:00]
d01 2018-06-05_10:00:00  DEBUG wrf_timetoa():  returning with str = [2018-06-05_00:00:00]
d01 2018-06-05_10:00:00  DEBUG wrf_timetoa():  returning with str = [2018-06-15_00:00:00]
d01 2018-06-05_10:00:00  DEBUG wrf_timeinttoa():  returning with str = [0000000000_000:001:030]
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock start time = 2018-06-05_00:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock current time = 2018-06-05_10:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock stop time = 2018-06-15_00:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock time step = 0000000000_000:001:030
d01 2018-06-05_10:00:00 module_integrate: back from solve interface
d02 2018-06-05_09:58:30 module_integrate: calling med_nest_force
d02 2018-06-05_09:58:30 calling inc/HALO_EM_COUPLE_A_inline.inc
d02 2018-06-05_09:58:30 calling inc/PERIOD_EM_COUPLE_A_inline.inc
d02 2018-06-05_09:58:30 calling inc/HALO_EM_COUPLE_B_inline.inc

I am attaching the namelist.input for my last run. I am using WRF-Chem v3.9.1.1 with Intel OneAPI compilers and ran the simulation with 16 processors.

Hope someone can help me with this! Thanks!

Best,
Bessie
 

Attachments

  • namelist.input
    9.4 KB · Views: 56
  • rsl.error.zip
    10.3 MB · Views: 45
Hi Bessie,

I see that you are using chem_opt==1, in addition to the GOCART dust and sea salt options. See these warnings in your rsl.error

WARNING: RADM2SORG_AQ chemistry option is experimental and not yet fully tested.
We recommend contacting wrfchemhelp for assistance.
WARNING: dust option 1 currently works only with the GOCART aerosol option.
WARNING: sea salt option 1 currently works only with the GOCART aerosol option.

I recommend switching to chem_opt == 42 or 109. If you choose option 109, I believe you should also choose phot_opt = 1 to get proper aerosol feedback. Also, in order to run with dust and sea salt, you will need to choose seas_opt = 2, and dust_opt = 3

Jordan
 
Top