WRF_CHEM Hangs

Post Reply
martin
Posts: 8
Joined: Thu Oct 04, 2018 12:39 pm

WRF_CHEM Hangs

Post by martin » Thu Mar 28, 2019 9:00 am

Dear WRF Support,

We are running WRF-CHEM version 3.8.1 on a cluster. The simulation is running fine for the course domain. However, when the namelist is modified to get the simulations for three domains (course, intermediate and fine), the wrf.exe executable hangs after 41 minutes of the processed simulation. Is it as issue related to the Physics parameters?

The executable hangs as the following line:
d03 2015-04-17_00:41:06 calling inc/HALO_CUP_G3_IN_inline.inc

Any guidance would be appreciated.

Timing for main: time 2015-04-17_00:41:06 on domain 3: 9896.66211 elapsed seconds
d03 2015-04-17_00:41:06 in med_latbound_in
d03 2015-04-17_00:41:06 module_integrate: calling solve interface
d03 2015-04-17_00:41:06 call HALO_RK_CHEM
d03 2015-04-17_00:41:06 calling inc/HALO_EM_CHEM_E_5_inline.inc
d03 2015-04-17_00:41:06 calling inc/HALO_EM_SCALAR_E_5_inline.inc
d03 2015-04-17_00:41:06 grid spacing, dt, time_step_sound= 3000.00000 18.0000000 4
d03 2015-04-17_00:41:06 call rk_step_prep
d03 2015-04-17_00:41:06 calling inc/HALO_EM_A_inline.inc
d03 2015-04-17_00:41:06 calling inc/PERIOD_BDY_EM_A_inline.inc
d03 2015-04-17_00:41:06 call rk_phys_bc_dry_1
d03 2015-04-17_00:41:06 call init_zero_tendency
d03 2015-04-17_00:41:06 calling inc/HALO_EM_PHYS_A_inline.inc
d03 2015-04-17_00:41:06 call phy_prep
d03 2015-04-17_00:41:06 DEBUG wrf_timetoa(): returning with str = [2015-04-17_00:41:06]
d03 2015-04-17_00:41:06 call radiation_driver
d03 2015-04-17_00:41:06 Top of Radiation Driver
d03 2015-04-17_00:41:06 calling inc/HALO_PWP_inline.inc
d03 2015-04-17_00:41:06 call surface_driver
d03 2015-04-17_00:41:06 SST_UPDATE is on
d03 2015-04-17_00:41:06 in MYJSFC
d03 2015-04-17_00:41:06 in NOAH DRV
d03 2015-04-17_00:41:06 call pbl_driver
d03 2015-04-17_00:41:06 in MYJPBL
d03 2015-04-17_00:41:06 call cumulus_driver
d03 2015-04-17_00:41:06 calling inc/HALO_CUP_G3_IN_inline.inc
Attachments
namelist.wps
(1.58 KiB) Downloaded 33 times
namelist.input
(8.09 KiB) Downloaded 55 times

kwerner
Posts: 2287
Joined: Wed Feb 14, 2018 9:21 pm

Re: WRF_CHEM Hangs

Post by kwerner » Thu Mar 28, 2019 4:33 pm

This topic has been moved to the WRF-Chem section of the forum.
NCAR/MMM

martin
Posts: 8
Joined: Thu Oct 04, 2018 12:39 pm

Re: WRF_CHEM Hangs

Post by martin » Mon Apr 08, 2019 9:20 am

Any help on this issue?

douglowe
Posts: 46
Joined: Thu Oct 11, 2018 8:31 am

Re: WRF_CHEM Hangs

Post by douglowe » Wed Apr 10, 2019 3:29 pm

Probably it's a CFL error - maybe your transport timestep is a little too long, causing instabilities in the solution which create NaNs (and these then crash the halo communication routines, which is where the model hung for you).

I'd recommend reducing your time_step to begin with - maybe start with 300 - and see if that solves the problem for you. If it does then you can play around with the time_step settings (such as the parent_time_step_ratio) to try to find a solution which works for you (is both stable, and doesn't increase the expense of the simulation too much).

beinnnnn
Posts: 9
Joined: Thu Jan 09, 2020 12:04 pm

Re: WRF_CHEM Hangs

Post by beinnnnn » Thu Apr 01, 2021 9:12 am

Hi everyone,

I am facing a similar problem. I am running a 3-domain run for 10 days simulation and WRF-Chem hangs after simulating the first 9 hours. I tried to run with different numbers of processors and reduced the timesteps, but they all hung at the same point (calling inc/HALO_EM_COUPLE_B_inline.inc). Also, it seems that the simulation failed when moving from the coarse domain to the nest domain?

Here is the last few rows of the rsl.error (the file is also compressed and attached)

Code: Select all

d01 2018-06-05_10:00:00  DEBUG wrf_timetoa():  returning with str = [2018-06-05_10:00:00]
d01 2018-06-05_10:00:00  DEBUG wrf_timetoa():  returning with str = [2018-06-05_00:00:00]
d01 2018-06-05_10:00:00  DEBUG wrf_timetoa():  returning with str = [2018-06-15_00:00:00]
d01 2018-06-05_10:00:00  DEBUG wrf_timeinttoa():  returning with str = [0000000000_000:001:030]
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock start time = 2018-06-05_00:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock current time = 2018-06-05_10:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock stop time = 2018-06-15_00:00:00
 DEBUG domain_clockadvance():  after WRFU_ClockAdvance,  clock time step = 0000000000_000:001:030
d01 2018-06-05_10:00:00 module_integrate: back from solve interface
d02 2018-06-05_09:58:30 module_integrate: calling med_nest_force
d02 2018-06-05_09:58:30 calling inc/HALO_EM_COUPLE_A_inline.inc
d02 2018-06-05_09:58:30 calling inc/PERIOD_EM_COUPLE_A_inline.inc
d02 2018-06-05_09:58:30 calling inc/HALO_EM_COUPLE_B_inline.inc
I am attaching the namelist.input for my last run. I am using WRF-Chem v3.9.1.1 with Intel OneAPI compilers and ran the simulation with 16 processors.

Hope someone can help me with this! Thanks!

Best,
Bessie
Attachments
rsl.error.zip
(10.26 MiB) Downloaded 7 times
namelist.input
(9.37 KiB) Downloaded 11 times

jordanschnell
Posts: 141
Joined: Thu Feb 27, 2020 10:58 pm

Re: WRF_CHEM Hangs

Post by jordanschnell » Thu Apr 01, 2021 4:33 pm

Hi Bessie,

I see that you are using chem_opt==1, in addition to the GOCART dust and sea salt options. See these warnings in your rsl.error

WARNING: RADM2SORG_AQ chemistry option is experimental and not yet fully tested.
We recommend contacting wrfchemhelp for assistance.
WARNING: dust option 1 currently works only with the GOCART aerosol option.
WARNING: sea salt option 1 currently works only with the GOCART aerosol option.

I recommend switching to chem_opt == 42 or 109. If you choose option 109, I believe you should also choose phot_opt = 1 to get proper aerosol feedback. Also, in order to run with dust and sea salt, you will need to choose seas_opt = 2, and dust_opt = 3

Jordan

Post Reply

Return to “Running”