STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Topics specifically related to the wrf.exe program
Post Reply
akgv19
Posts: 9
Joined: Wed Apr 03, 2019 7:53 pm

STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by akgv19 » Mon May 18, 2020 6:42 pm

Hello,

I'm running WRF Version 3.9.1.1 on an HPC system, and I'm using the Bias-Corrected CMIP5 CESM dataset on 3 domains centered over the CONUS. I'm performing a 10-year simulation with Noah-MP as my LSM, and beginning to perform model spin-up at the beginning of 1989. The wrf executable seems to produce output for about the first 40 days without any issue. But during the date of February 12 1989, wrf exits with the following error at the end of one of the rsl error files...

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 1749
STOP in Noah-MP
-------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 111 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
In: PMI_Abort(1, N/A)



I'm not finding any obvious cfl errors in any of the rsl files. Here is a copy of my namelist.input file...
namelist.input
(6.45 KiB) Downloaded 31 times
At your convenience, would you be able to give advice on what the issue might be?

~Ali

akgv19
Posts: 9
Joined: Wed Apr 03, 2019 7:53 pm

Re: STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by akgv19 » Mon May 18, 2020 7:18 pm

Here is the actual rsl output file that pertains to the fatal error...
rsl_out_0111.txt
(45.94 KiB) Downloaded 31 times
I hope that helps to provide additional helpful information. Thanks!

Ming Chen
Posts: 1101
Joined: Mon Apr 23, 2018 9:42 pm

Re: STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by Ming Chen » Mon May 18, 2020 11:46 pm

The rsl file shows that the vegetation leaf temperature, TV = 5153.334, which is apparently wrong.
I suspect that this case went wrong probably long before the time it crashed. The namelist settings below are not correct. Please reset and try again:
cu_physics = 1, 1, 1,
sf_sfclay_physics = 2, 2, 2,
sf_surface_physics = 4, 4, 4,
bl_pbl_physics = 6, 6, 6,

(1) Note that for 3km domain, you should turn off cumulus scheme
(2) MYNN 3rd level TKE PBL scheme only works with MYNNSFC surface layer scheme
(3) Did you modify the codes? What is the option of dveg = 10?
WRF Help Desk

akgv19
Posts: 9
Joined: Wed Apr 03, 2019 7:53 pm

Re: STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by akgv19 » Tue May 19, 2020 5:58 pm

Thanks for your quick response. I will make sure to turn off the Cumulus scheme for my 3 km domain. Also, some lab-mates in my office specified the exact same physics schemes I did. However, they used the regular Noah LSM (not Noah-MP). They have had no apparent issues running MYNN3 with other surface layer schemes. But I will continue to keep your suggestions in mind.

As for (3), a few months ago I had issues running wrf.exe at all, although real.exe ran fine with no issue. Kelly at WRF suggested maybe I was having issues with the bucket option in my namelist...

https://forum.mmm.ucar.edu/phpBB3/viewt ... MMON#p9861

I did what she recommended and recompiled. I also updated a few of the physics options in my namelist, which is what I shared in my post yesterday. After that, I began the simulation and it ran for about the first 40 days, before failing.

I wasn't exactly sure if it was an openMPI issue with the HPC system I was running on or whether it was something to do with WRF. So then I had an HPC administrator test an abbreviated version of my simulation to see if it failed at the exact same point (using the same input data and same namelist options) and it did not. He had to stop the simulation because he ran out of storage space. I suppose it may have failed for him eventually, had he kept running, but I can't say for sure. Also, that administrator never had an issue with the bucket option like I originally had, and never had to change the Registry file and recompile. So I'm still not entirely sure what is causing the "STOP in Noah-MP" error. Do you have any thoughts or ideas, regarding this?

Pertaining to dveg =10, this activates the built-in crop models (related to WRF-Crop or Noah-MP-Crop) inside Noah-MP. I was interested in comparing the output from these built-in crop models to other crop models. I also changed a value in the MPTABLE.TBL file, before running this simulation. I don't believe that requires any recompilation.

Do you know of anyone else who has had similar issues? If you have any other suggestions, I would appreciate hearing back from you at your convenience. Thanks!

Ming Chen
Posts: 1101
Joined: Mon Apr 23, 2018 9:42 pm

Re: STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by Ming Chen » Wed May 20, 2020 11:12 pm

I don't think WRF has the option of dveg=10. Please correct me if I am wrong.
I am not sure whether this is the reason for the failed case.
Please try other dveg options and see if that narrows down the problem to this option.
WRF Help Desk

kluck_2020
Posts: 2
Joined: Tue Aug 25, 2020 3:15 am

Re: STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by kluck_2020 » Thu Aug 27, 2020 7:56 pm

WRF does have a dveg=10 option it can be found in the user guide here https://www2.mmm.ucar.edu/wrf/users/doc ... tents.html. It is an option with in &noah_mp that activates the Noah-MP-Crop model. We have used it before but currently when using it, it can't be found when trying to run WRF. Has this option been taken out recently?

Ming Chen
Posts: 1101
Joined: Mon Apr 23, 2018 9:42 pm

Re: STOP in Noah-MP using Bias-Corrected CMIP5 CESM dataset

Post by Ming Chen » Tue Sep 01, 2020 10:06 pm

Yes this option is deactivated in WRF. Please see the code below (in phys/module_sf_noahmplsm.F)

IF(DVEG == 1 .or. DVEG == 6 .or. DVEG == 7) THEN
FVEG = SHDFAC
IF(FVEG <= 0.05) FVEG = 0.05
ELSE IF (DVEG == 2 .or. DVEG == 3 .or. DVEG == 8) THEN
FVEG = 1.-EXP(-0.52*(LAI+SAI))
IF(FVEG <= 0.05) FVEG = 0.05
ELSE IF (DVEG == 4 .or. DVEG == 5 .or. DVEG == 9) THEN
FVEG = SHDMAX
IF(FVEG <= 0.05) FVEG = 0.05
ELSE
WRITE(*,*) "-------- FATAL CALLED IN SFLX -----------"
CALL wrf_error_fatal("Namelist parameter DVEG unknown")
ENDIF
WRF Help Desk

Post Reply

Return to “wrf.exe”