BAD TERMINATION (EXIT CODE: 1)

[Hung Nguyen]

Dear all,

I am trying to run WRFv4.4.2 with GDAS data. However, the model only ran for a few hours and then crashed with the following message:

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

I have tried reducing the timestep to 3dx/2dx or set "smooth_cg_topo = .true." but the same error still occurs and I still have no idea what is wrong.

Please kindly help me to solve the problem. I attached my namelist.input, rsl files, and the error code (out-run.txt) for your reference. If you need additional information please kindly let me know.

Thank you very much.

 

[kwerner]

Hi,
In one of your rsl.* files, the message "TBOUND exceeds table limit: reset 344.519" is printed at the end. Take a look at this FAQ that discusses the error "TBOUND exceeds table limit." You may also find some helpful suggestions in this post. Let us know if this still doesn't help.

 

[Hung Nguyen]

Dear Kwerner,

Thank you very much for your kind support.

I followed the links you sent and made changes to the namelist.input file but I still got the "TBOUND" error. What I have done is:

Reduced radt from 30 to 2
set epssm = 0.5 (before epssm = 0.9)

I would appreciate it very much if you could give me further advice to solve the issue. Attached is the namelist.input and rsl* files for your reference.

Thank you very much.

 

[kwerner]

Thank you for trying that and for sending the updated files. After doing some digging, I've realized that the error "TBOUND exceeds table limit" is probably not actually causing the failure. It is simply a message to let you know it's overwriting that temperature to a lower value. I then searched for 'cfl' in your rsl* files and I can see this:

rsl.error.0026:d03 2018-03-14_07:38:00            6  points exceeded v_cfl = 2 in domain d03 at time 2018-03-14_07:38:00 hours
rsl.error.0026:d03 2018-03-14_07:38:00 Max   W:     61     84      3 W:    1.33  w-cfl:    5.73  dETA:    0.01
rsl.out.0026:d03 2018-03-14_07:38:00            6  points exceeded v_cfl = 2 in domain d03 at time 2018-03-14_07:38:00 hours
rsl.out.0026:d03 2018-03-14_07:38:00 Max   W:     61     84      3 W:    1.33  w-cfl:    5.73  dETA:    0.01

which indicates that the model has become unstable - typically due to complex terrain. Can you take a look at this FAQ, which discusses some options to overcome the issue and see if you can find a group of settings that get you past that issue? I'm still not certain it will run to completeness, but I want to make sure we resolve this issue first. After you run, you can just issue

grep cfl rsl*

to see if you still have the errors. If not, but if it's still failing, please send the new namelist and rsl files. Thanks!

 

[Hung Nguyen]

Dear Kwerner,

Thank you very much for your suggestions. I tried to to follow your post on CFL error (FAQ) but WRF still crashed. What I did was:

1. Set smooth_cg_topo = .true. before running real.exe.
2. Set epssm = 0.2.
3. Set w_damping = 1.
4. Executed the command ulimit -s unlimited.
5. Reduced the domain time step from 6dx to 3dx.

but I still got the "cfl error" in some of the rsl files.

However, I found that turning off sf_urban_physics allows WRF to complete successfully (no TBOUND or CFL errors). So, I think the issue is related to urban physics (I used the default URBPALM.TBL for this run). My simulation period is from 25/12/2017 to 02/01/2019. WRF ran normally with sf_urban_physics = 1 until 19/03/2018 and then crashed. I can only restart from 19/03/2018 if I turn off urban_physics.

Attached is the rsl files for both cases (with and without urban_physics) and the namelist.input for your reference.

Thank you again for you kind support.

 

[kwerner]

Hi,
It may be that when you don't use urban physics, the calculations aren't as complex and sensitive, therefore the CFL errors aren't showing up, but I don't think there is an issue with the urban scheme, itself. Can you try to set epssm to something larger, like 0.5, or you can even go up to 0.9, and let me know if that changes anything?

 

[Hung Nguyen]

Dear Kwerner,

Thank you for your suggestions,

I set epssm to 0.9 and reran WRF. This time, I do not have the cfl error anymore but WRF still crashed after running for a few minutes.

I attached my new namelist.input and rsl files for your reference.

Thank you very much for your kind support.

 

[William.Hatheway]

Dear Kwerner,

Thank you for your suggestions,

I set epssm to 0.9 and reran WRF. This time, I do not have the cfl error anymore but WRF still crashed after running for a few minutes.

I attached my new namelist.input and rsl files for your reference.

Thank you very much for your kind support.

In rsl.error.0005 there this call:

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:     415
  RIBX never exceeds RIC, RIB(i,kte) =             NaN  THETAV(i,1) =             NaN  MOL=            NaN  TCONV =   0.0000000E+00  WST =             NaN  KMIX =            1  UST =   8.2262419E-02  TST =             NaN  U,V =             NaN            NaN  I,J=          51          26
-------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5

Acording to this post: RIBX never exceeds RIC, RIB error & RIBX exceeding error

Could be a few things.

1. Try turning your cu_physics = 1, 1, 0,
It is reccomeneded by NCAR that when dx & dy are less than 5km to turn off cu_physics

2. Double check input data ( also what is your input data you are using and where from?

Also in those links I noted above @Ming Chen says there is usually something with incompatible physics options. Until they get back to you try my suggestion first and then they will look eventually at your physics.

 

[Hung Nguyen]

Dear William,

Thank you very much for your help.

I am using "NCEP GDAS Final Analysis" downloaded from the link below:

NCAR RDA Dataset ds083.3

NCAR RDA

rda.ucar.edu

I will try your suggestions to see if they can help.

Thank you again!