p_top Fatal Error

[matthewblittle]

Problem:
I have run several em_real successful simulations, but am now running into this problem because I have to go back and re-run simulations from times which the output was lost due to local disk failures.

I retrieved the restart file, generated WRF input from the observations from WPS in the same manner as before, and placed them all in the WRF directory. The simulation is to extend from 0000Z 20 Mar 2009 to 0000Z 26 Mar 2009. real.exe seems to run fine (See attached output); however, the problem arises when wrf.exe is executed. As can be seen in the output, the last lines are:

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 543
start_em: p_top from the namelist does not match p_top from the input file.
-------------------------------------------

I don't understand why this error is occurring when I try to go back to re-run a previous simulation but not when continuing the model simulation onward from where the previous one ended.

Help requested:

If anyone can help me understand why this fatal error is occurring, or things I can do to prevent this error so I may continue my simulations, it would be greatly appreciated.

Attached are the namelist.wps, namelist.input, and program ouput and error files from real.exe and wrf.exe run on the Cheyenne HPC resources.

Details:

I am running 3 domains, 2 nested telescoping on the southeast United States.

I am using the Cheyenne HPC resources, using WRF and WPS Version 3.9, and retrieved pre-compiled model files from:

~wrfhelp/PRE_COMPILED_CODE/WRFV3.9_intel_dmpar/

My wrf working directory is:

/glade/scratch/mlittle/CC/WRFV3.9_intel_dmpar/test/em_real

Currently loaded modules (retrieved from module list command):

Currently Loaded Modules:
  1) ncarenv/1.3    3) ncarcompilers/0.5.0   5) netcdf/4.7.3   7) ncl/6.6.2
  2) intel/18.0.5   4) mpt/2.19              6) idl/8.7.2      8) grads/2.2.1

I am using 0.5° GFS analysis data from:
https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/global-forcast-system-gfs

And using 0.5°RTG-SST data from:
ftp://polar.ncep.noaa.gov/history/sst/rtg_low_res

Please let me know if additional information needed to help understand and solve the problem.

Thanks,
Matthew B. Little

 

[kwerner]

Hi,
I ran a couple of tests with your wrfrst* files, namelist, wrfbdy file, etc. The problem you're running into seems to occur when you are using more than 1 domain. If you check particular reference state values in your wrfrst_d02 file (vs. wrfrst_d01), like P_TOP, T00, P00, TLP, etc., they all have a value of zero for d02, but reasonable values for d01. I'm not really sure how this could have happened. If you still have your wrfout* files from this same run (or from some times prior to this), can you check those same values in those (d01 and d02)? You can do this with the following command - for e.g.:

ncdump -v P_TOP wrfrst_d02_2009-03-20_00:00:00

 

[matthewblittle]

Thank you for helping me understand the error better. I ran a slightly modified version of the command you suggested and noticed the following in the domain restart files:

WRFV3.9_intel_dmpar/test/em_real> ncdump -v P_TOP wrfrst_d03_2009-03-20_00:00:00 | grep P_TOP
        float P_TOP(Time) ;
                P_TOP:FieldType = 104 ;
                P_TOP:MemoryOrder = "0  " ;
                P_TOP:description = "PRESSURE TOP OF THE MODEL" ;
                P_TOP:units = "Pa" ;
                P_TOP:stagger = "" ;
 P_TOP = 0 ;
WRFV3.9_intel_dmpar/test/em_real> ncdump -v P_TOP wrfrst_d02_2009-03-20_00\:00\:00 | grep P_TOP
        float P_TOP(Time) ;
                P_TOP:FieldType = 104 ;
                P_TOP:MemoryOrder = "0  " ;
                P_TOP:description = "PRESSURE TOP OF THE MODEL" ;
                P_TOP:units = "Pa" ;
                P_TOP:stagger = "" ;
 P_TOP = 5000 ;
WRFV3.9_intel_dmpar/test/em_real> ncdump -v P_TOP wrfrst_d01_2009-03-20_00\:00\:00 | grep P_TOP
        float P_TOP(Time) ;
                P_TOP:FieldType = 104 ;
                P_TOP:MemoryOrder = "0  " ;
                P_TOP:description = "PRESSURE TOP OF THE MODEL" ;
                P_TOP:units = "Pa" ;
                P_TOP:stagger = "" ;
 P_TOP = 5000 ;

Whereas when running the command on a different set of restart files resulted in the following:

WRFV3.9_intel_dmpar/test/em_real> ncdump -v P_TOP wrfrst_d03_2009-03-19_00\:00\:00 | grep P_TOP
        float P_TOP(Time) ;
                P_TOP:FieldType = 104 ;
                P_TOP:MemoryOrder = "0  " ;
                P_TOP:description = "PRESSURE TOP OF THE MODEL" ;
                P_TOP:units = "Pa" ;
                P_TOP:stagger = "" ;
 P_TOP = 5000 ;
WRFV3.9_intel_dmpar/test/em_real> ncdump -v P_TOP wrfrst_d02_2009-03-19_00\:00\:00 | grep P_TOP
        float P_TOP(Time) ;
                P_TOP:FieldType = 104 ;
                P_TOP:MemoryOrder = "0  " ;
                P_TOP:description = "PRESSURE TOP OF THE MODEL" ;
                P_TOP:units = "Pa" ;
                P_TOP:stagger = "" ;
 P_TOP = 5000 ;
WRFV3.9_intel_dmpar/test/em_real> ncdump -v P_TOP wrfrst_d01_2009-03-19_00\:00\:00 | grep P_TOP
        float P_TOP(Time) ;
                P_TOP:FieldType = 104 ;
                P_TOP:MemoryOrder = "0  " ;
                P_TOP:description = "PRESSURE TOP OF THE MODEL" ;
                P_TOP:units = "Pa" ;
                P_TOP:stagger = "" ;
 P_TOP = 5000 ;

Using the restart files from 2009-03-19 worked and I am now able to continue the simulations.

I very much appreciate your help and wish you the best.

 

[kwerner]

Great! I'm glad you got past the problem. Thank you for updating the post.