Error in real.exe with ERA-Interim pressure level data

[parth17]

Hello,

I am working with WRF-ARW (v3.8.1) and facing issues in real.exe program while working with ERA-Interim pressure-level data. I have followed multiple tutorials on how to work with WRF using ERA-Interim pressure level data by downloading the sfc, pres and Invariant data files. Along with this I am using the Vtable.ERA-Interim.pl for ungrib. I get no errors while running ungrib and metgrid.

In case of real.exe, I am using dmpar (gfortran), mvapich2 and running them on parallel. Errors start appearing when I run it with 64 processes I get error as follows in rsl.out.0063:

d01 2011-12-31_06:00:00 t(i,j,k) was 0 at 348 359 28 , setting Qv to 0
d01 2011-12-31_06:00:00 t(i,j,k) was 0 at 349 359 28 , setting Qv to 0
Using sfcprs2 to compute psfc
Converged znw(kte) should be about 0.0 = -1.68278022E-03
Full level index = 1 Height = 0.0 m
Full level index = 2 Height = 53.6 m Thickness = 53.6 m
Full level index = 3 Height = 130.6 m Thickness = 77.0 m
Full level index = 4 Height = 231.7 m Thickness = 101.0 m
Full level index = 5 Height = 357.4 m Thickness = 125.7 m
Full level index = 6 Height = 516.8 m Thickness = 159.4 m
Full level index = 7 Height = 719.6 m Thickness = 202.8 m
Full level index = 8 Height = 960.1 m Thickness = 240.5 m
Full level index = 9 Height = 1199.0 m Thickness = 238.9 m
Full level index = 10 Height = 1443.8 m Thickness = 244.8 m
Full level index = 11 Height = 1694.7 m Thickness = 250.9 m
Full level index = 12 Height = 1952.1 m Thickness = 257.4 m
Full level index = 13 Height = 2448.0 m Thickness = 495.9 m
Full level index = 14 Height = 2943.9 m Thickness = 495.9 m
Full level index = 15 Height = 3439.8 m Thickness = 495.9 m
Full level index = 16 Height = 3935.7 m Thickness = 495.9 m
Full level index = 17 Height = 4431.6 m Thickness = 495.9 m
Full level index = 18 Height = 4927.5 m Thickness = 495.9 m
Full level index = 19 Height = 5423.4 m Thickness = 495.9 m
Full level index = 20 Height = 5919.3 m Thickness = 495.9 m
Full level index = 21 Height = 6415.2 m Thickness = 495.9 m
Full level index = 22 Height = 6911.1 m Thickness = 495.9 m
Full level index = 23 Height = 7407.1 m Thickness = 495.9 m
Full level index = 24 Height = 7903.0 m Thickness = 495.9 m
Full level index = 25 Height = 8398.9 m Thickness = 495.9 m
Full level index = 26 Height = 8894.8 m Thickness = 495.9 m
Full level index = 27 Height = 9390.7 m Thickness = 495.9 m
Full level index = 28 Height = 9886.6 m Thickness = 495.9 m
Full level index = 29 Height = 10382.5 m Thickness = 495.9 m
Full level index = 30 Height = 10878.4 m Thickness = 495.9 m
Full level index = 31 Height = 11374.3 m Thickness = 495.9 m
Full level index = 32 Height = 11870.2 m Thickness = 495.9 m
Full level index = 33 Height = 12366.2 m Thickness = 495.9 m
Full level index = 34 Height = 12862.1 m Thickness = 495.9 m
Full level index = 35 Height = 13358.0 m Thickness = 495.9 m
Full level index = 36 Height = 13853.9 m Thickness = 495.9 m
Full level index = 37 Height = 14349.8 m Thickness = 495.9 m
Full level index = 38 Height = 14845.7 m Thickness = 495.9 m
Full level index = 39 Height = 15341.6 m Thickness = 495.9 m
Full level index = 40 Height = 15837.5 m Thickness = 495.9 m
d01 2011-12-31_06:00:00 calling inc/HALO_EM_VINTERP_UV_1_inline.inc
i,j = 307 316
target pressure and value = 10.7630348 2.26704267E-24
var_type = Z
column of pressure and value = 11.5357113 0.00000000
column of pressure and value = NaN -1.16219580
column of pressure and value = NaN -1.45850277
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 5366
troubles, could not find trapping x locations
-------------------------------------------

As per the error, I checked the particular grid location and also added a debug line to display variable name which is causing errors in vertical interpolation [module_initialize_real]. The variable type is "Z".

Similarly my wrf.exe crashes due to this error. Based on WRF User Forum, I tried adding multiple configuration flags such as sfcp_to_sfcp as true but that doesn't help.

It would be really helpful if I can just get a hint on what can be the possible reason behind this error. The data from metgrid seems correct for all 3-d variables such as U, V, R, Z, Q and T.

I would be keen to share more information on this issue.

Namelist:

&domains
time_step = 50,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 1,
target_cfl = 1.2,
e_we = 350,
e_sn = 360,
e_vert = 40,
p_top_requested = 10000,
num_metgrid_levels = 28,
num_metgrid_soil_levels = 4,
dx = 5000,
dy = 5000,
grid_id = 1,
parent_id = 1,
i_parent_start = 1,
j_parent_start = 1,
parent_grid_ratio = 1,
parent_time_step_ratio = 1,
feedback = 0,
smooth_option = 0,
sfcp_to_sfcp = .true.,
use_levels_below_ground = .false.,

&physics
mp_physics = 8,
ra_lw_physics = 4,
ra_sw_physics = 4,
radt = 10,
sf_sfclay_physics = 1,
sf_surface_physics = 2,
bl_pbl_physics = 1,
bldt = 0,
cu_physics = 0,
cudt = 0,
isfflx = 1,
ifsnow = 1,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
sf_urban_physics = 0,

&dynamics
w_damping = 0,
diff_opt = 1,
km_opt = 4,
diff_6th_opt = 0,
diff_6th_factor = 0.12,
base_temp = 290.
damp_opt = 0,
epssm = 0.2,
zdamp = 5000.,
dampcoef = 0.2,
khdif = 0,
kvdif = 0,
non_hydrostatic = .true.,
moist_adv_opt = 1,
scalar_adv_opt = 1,

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 9,
spec_exp = 0.33,
specified = .true.,
nested = .false.,

 

[Ming Chen]

Where did you download the ERA-I data? If from NCAR RDA, please let me know what data you have downloaded.
It is also helpful to send me your namelist.wps to take a look.

 

[parth17]

Hello Chen,

I have downloaded ERA-I data using the python API from ECMWF. My PL data includes Relative Humidity, U-component of wind, V-component of wind, Geopotential, Specific humidity and Temperature. My FIX contains Land-sea Mask. My SFC includes 2 metre dewpoint temperature, 2 metre temperature, 10 metre U wind component, 10 metre V wind component, Mean sea level pressure, Sea surface temperature, Sea-ice cover, Skin temperature, Snow density, Snow depth, Soil temperature levels 1, 2, 3, 4, Surface pressure, Volumetric soil water layer 1, 2, 3, 4.

Namelist.WPS:

&share
wrf_core = 'ARW',
max_dom = 1,
start_date = '2011-12-31_06:00:00',
end_date = '2013-01-01_18:00:00',
interval_seconds = 21600
io_form_geogrid = 2,
debug_level = 0
/

&geogrid
parent_id = 1,
parent_grid_ratio = 1,
i_parent_start = 1,
j_parent_start = 1,
e_we = 350,
e_sn = 360,

!
geog_data_res = '30s',
dx = 5000,
dy = 5000,
map_proj = 'mercator',
ref_lat = 15.519,
ref_lon = 102.816,
truelat1 = 15.519,
truelat2 = 0,
stand_lon = 102.816,
geog_data_path = '../WPS_data'
/

&ungrib
out_format = 'WPS',
prefix = 'PL', (Change based on data , PL/SFC/FIX)
/

&metgrid
fg_name = 'PL','SFC',
constants_name = 'FIX:1989-01-01_12'
io_form_metgrid = 2,

 

[Ming Chen]

I am concerned of the land-sea mask you created. If it is not consistent with soil/sst information, there may have issues as shown in your error message.
Can you download data from NCAR RDA website, then try again? The website is here:
https://rda.ucar.edu/datasets/ds627.0/
This data is open to public and we have tested it to make sure it works for WPS/WRF.

 

[parth17]

Hi Ming,

I realized the same and that will surely be my next step. I had a doubt regarding which Vtable to use (ECMWF or ERA-I.pl) and whether to keep the invariant fields (Land-sea Mask and Geo potential) as constants while running metgrid or download it within the surface files for each timestep. People in different forums have kept it constant dated on 1989-01-01_12 and few have it in their sfc files for each timestep.

I am not sure if the ds627 keeps those data as constants for each timestep. For ds627, I plan to use Vtable.ECMWF by adding specific humidity field in pressure level data.

Parth

 

[Ming Chen]

Parth,
Please download ERA-I pressure level data from NCAR RDA and use Vtable.ERA-interim.pl to ungrib these data. The invariant fields (Land-sea Mask and Geo potential at the surface) should be kept as constants.
I remember relative humidity field is included in the ERA-I pressure level dataset, so you don't need to include specific humidity.

 

[kwthomas]

I recall having a "trapping" error problem a couple of years ago. I think it was an MPI problem. It wasn't a
WRF bug.

You say the problem occurs with 64 processors. Try a different number, including higher.

I know in the past, every once in a while, I'd come up with a processor configuration that just wouldn't work
right. Changing the core count would fix the problem.

 

[parth17]

Finally I was able to run with ERA-Interim pressure level data. It was because of some issues with the software packages in HPC cluster. I was getting vert_cfl error near the boundary but was able to resolve it by increasing "spec_bdy_width" to 10. I guess this wouldn't affect my results much as I am running at 5km spatial resolution.

One more thing that I was not sure of was the pattern of SWDOWN in my output. I see some blue patches in regions with lower Solar downward flux. My guess is that the irregularity in pattern is due to terrain. I have attached an image. Please let me know you opinion.

Thanks a lot to Ming Chen and kwthomas for their help.

 

[Ming Chen]

Can you examine cloud fraction in your wrfout? I am thinking that the low values of swdown may be related to low, thick clouds above the areas.

I agree that increasing the buffer zone to 10 layers shouldn't have significant impacts on your results.

 

[parth17]

Hello Ming,

Thanks once again for your feedback. I have LCL_OUT and LTOP_OUT in my output. I need to edit my registry to output the cloud fraction. But I can see few clouds based on cloud top and cloud bottom. I guess this might be the reason of getting blue patches in SWDOWN. Please find attached the cloud bottom in the attachments.