Error in real.exe P array = NaN

[cross]

I'm trying to run real.exe and I keep getting the same problem:
rsl.out:

01 2020-01-19_05:00:00  boundary conditions OK for grid
Using sfcprs  to compute psfc
d01 2020-01-19_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
d01 2020-01-19_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
d01 2020-01-19_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
 all_dim =            2
 order =            2
 i,j =           25         113
 p array =             NaN   4.605170
 f array =   -1.400978       11.77058
 p target=    11.51558       11.51437       11.51316       11.51195
   11.51073       11.50952       11.50830       11.50709       11.50512
   11.50163       11.49662       11.49006       11.48191       11.47213
   11.46069       11.44752       11.43256       11.41501       11.39488
   11.37287       11.34887       11.32273       11.29430       11.26340
   11.22983       11.19334       11.15364       11.11040       11.06322
   11.01160       10.95497       10.89258       10.82353       10.74665
   10.66044       10.56286       10.45114       10.32128       10.16730
   9.979517       9.740760       9.416257       8.915963

rsl.error:

d01 2020-01-19_05:00:00          flag_soil_layers read from met_em file is  1
d01 2020-01-19_05:00:00 flag_soil_layers at end of optional_lsm_levels is  1
d01 2020-01-19_05:00:00 flag_soil_levels at end of optional_lsm_levels is  0
d01 2020-01-19_05:00:00 st_levels_input =   3  17  64 194
d01 2020-01-19_05:00:00 sm_levels_input =   3  17  64 194
d01 2020-01-19_05:00:00 sw_levels_input =   3  17  64 194
d01 2020-01-19_05:00:00 med_sidata_input: back from init_domain
d01 2020-01-19_05:00:00 med_sidata_input: calling init_domain
d01 2020-01-19_05:00:00  checking boundary conditions for grid
d01 2020-01-19_05:00:00  boundary conditions OK for grid
Using sfcprs  to compute psfc
d01 2020-01-19_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
d01 2020-01-19_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
d01 2020-01-19_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
real.exe           0000000003097B4A  Unknown               Unknown  Unknown
libpthread-2.27.s  00001508C21A0980  Unknown               Unknown  Unknown
real.exe           00000000031B694F  Unknown               Unknown  Unknown
real.exe           00000000030FD846  Unknown               Unknown  Unknown
real.exe           00000000008CF969  Unknown               Unknown  Unknown
real.exe           000000000044EB85  Unknown               Unknown  Unknown
real.exe           0000000000456271  Unknown               Unknown  Unknown
real.exe           00000000004A665D  Unknown               Unknown  Unknown
real.exe           00000000004C630D  Unknown               Unknown  Unknown
real.exe           00000000004076BC  Unknown               Unknown  Unknown
real.exe           00000000004064EB  Unknown               Unknown  Unknown
real.exe           0000000000405B42  Unknown               Unknown  Unknown
libc-2.27.so       00001508C1A20BF7  __libc_start_main     Unknown  Unknown
real.exe           0000000000405A2A  Unknown               Unknown  Unknown

I'm running WRF 4.3 and WPS 4.3, I also tried with WRF and WPS 4.1, both with ERA5 data.
The error message it's always when processing the same time, the only thing I notice is the p array = NaN in rsl.out
Anyone knows what could be the problem? I'm out of ideas now.

 

[cross]

I tried again starting from a different date, and it always crashes with the same error when is 18 days after the start, so it's not a problem in the input data.

d01 2020-02-05_05:00:00  boundary conditions OK for grid
Using sfcprs  to compute psfc
d01 2020-02-05_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
d01 2020-02-05_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
d01 2020-02-05_05:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
 all_dim =            2
 order =            2
 i,j =           25         113
 p array =             NaN   4.605170
 f array =  -0.2747966       11.04123
 p target=    11.51753       11.51632       11.51511       11.51390
   11.51269       11.51147       11.51026       11.50904       11.50707
   11.50358       11.49857       11.49200       11.48385       11.47407
   11.46263       11.44945       11.43447       11.41691       11.39676
   11.37473       11.35069       11.32452       11.29605       11.26510
   11.23147       11.19491       11.15514       11.11182       11.06454
   11.01281       10.95606       10.89354       10.82436       10.74734
   10.66097       10.56324       10.45137       10.32139       10.16733

 

[Ming Chen]

HI,

Where did you download ERA5 data? Which Vtable did you use to ungrib the ERA5 data?
Can you upload one met_em file for me to take a look?

 

[cross]

Thanks for your reply, I don't think it's the ERA5 files, because it always get an error when it's processing the hour 437. I tried starting in different times.
If I start at 01/01/2020 00:00 I get an error at 19/01/2020 05:00
If I start at 18/01/2020 00:00 I get an error at 05/02/2020 05:00

The ERA5 data I downloaded it with the CDS api, like this https://dreambooker.site/2019/10/03/Initializing-the-WRF-model-with-ERA5-pressure-level/
And I used the Vtable.ERA-interim.pl for the pressure levels data.

View attachment met_em.d01.2020-01-19_05_00_00.nc.zip

 

[Ming Chen]

What about other error message? In the code, we can see the following printout:

      IF ( all_dim .LT. n+1 ) THEN
print *,'all_dim = ',all_dim
print *,'order = ',n
print *,'i,j = ',i,j
print *,'p array = ',all_x
print *,'f array = ',all_y
print *,'p target= ',target_x
         CALL wrf_message ( 0 , 'Troubles, the interpolating order is too large for this few input values' )
         CALL wrf_message ( 0 , 'This is usually caused by bad pressures' )
         CALL wrf_message ( 0 , 'At this (i,j), look at the input value of pressure from metgrid' )
         CALL wrf_message ( 0 , 'The surface pressure and the sea-level pressure should be reviewed, also from metgrid' )
         CALL wrf_message ( 0 , 'Finally, ridiculous values of moisture can mess up the vertical pressures, especially aloft' )
         CALL wrf_message ( 0 , 'The variable type is ' // var_type // '. This is not a unique identifer, but a type of field' )
         CALL wrf_message ( 0 , 'Check to see if all time periods with this data fail, or just this one' )
         CALL wrf_error_fatal ( 'This vertical interpolation failure is more typically associated with untested data sources to ungrib' )
      END IF

 

[cross]

I didn't have any error message, just the segmentation fault

forrtl: severe (174): SIGSEGV, segmentation fault occurred

 

[cross]

What I'm trying to do is to run ndown.
So first I run the first and 2nd domain and now I want to run a third domain.
So now I run WPS for the 2nd and 3rd domain (now domain 1 and 2) without any problem. The only problem happens when I run real.exe, which was ok when I did the first run.
I thought maybe was a memory problem, but I tried in other server with the same configuration and I get the same error at the same time.

 

[cross]

Well I realized that I was doing the nest down wrong.
So, could it be that the resolution I was using for the first domain was too high (3km) so it had some problem for creating the wrfbdy_d01?

 

[Ming Chen]

No I don't think so. The error message below indicates that something is wrong in the data itself:

 p array =             NaN   4.605170

Checking the code, there should have other error messages because prints out more information:

     IF ( all_dim .LT. n+1 ) THEN
print *,'all_dim = ',all_dim
print *,'order = ',n
print *,'i,j = ',i,j
print *,'p array = ',all_x
print *,'f array = ',all_y
print *,'p target= ',target_x
         CALL wrf_message ( 0 , 'Troubles, the interpolating order is too large for this few input values' )
         CALL wrf_message ( 0 , 'This is usually caused by bad pressures' )

Please look at all variables at (i,j = 25 , 113) in your met_em file.