REAL failure: flukey data

[simaberson]

I am running WRF using GFS and GFSENS data boundary conditions from 2021 for the first time. I have an error in REAL that I can't explain. The rsl.out file seems fine until I get to

hydro check - should only be for isobaric input
TERRAIN_HGT_T: 25 25 0.0000000E+00
PSFC_IN, TOPO_IN: 25 25 101121.3 0.0000000E+00
PSFC extremes (new style)
3.4028235E+38 -3.4028235E+38
must have flukey situation in trad 72 1
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 3028
HOPELESS PSFC, I QUIT
-------------------------------------------

A search provides no hits for "must have flukey situation in trad" or "HOPELESS PSFC, I QUIT" except the actual code (https://github.com/jswhit/gsisrc/blob/master/get_gefs_for_regional.f90), which looking through doesn't help me figure out what is wrong.

I assume PSFC refers to that variable in the input files, which real is finding to have absolute values matching maximum floating-point values. However, when I look at PSFC in the files, it looks reasonable, and I get values from 83478 to 102545 Pa, and plots of the data look fine. I don't see any variables with TOPO, but all the surface data look fine. In fact, I went through all the data, and nothing looks odd.

So, I'm stumped. Any ideas what I am getting wrong?

Sim Aberson (sim.aberson@noaa.gov)

 

[Ming Chen]

Hi, SIm,
Would you please let me know which version of WRF and WPS are you using? And where did you download GFS data? I am not familiar with GFSENS data, but we do use GFS a lot. So I suppose we can start from GFS and hope it will be helpful for GFSENS.

(1) We usually download GFS data from here: https://rda.ucar.edu/datasets/ds084.1/
(2) Please run newer version of WPS like WPSV4.2 or WPSV4.3 and use Vtable.GFS to ungrib the data
(3) Then run real.exe and wrf.exe

Please let me know if you have any issue during the above process.

 

[simaberson]

Hi Ming,

I am using WRFV3.9 because I wish to complete a large set of runs with the same version of the model. Of course, if that is not possible, I can switch, but prefer to keep the same version right now.

I am using a NOAA computer, so I download the data directly from the NOAA HPSS repository. I am using Vtable.GFSENS, which is appropriate for the ensemble.

I am not sure if I need to run the prep-hybrid step. When I do that, I get a different error in REAL, the dreaded segmentation fault. My rsl file has

STUFF MOVED TO REGISTRY: 2 4 90 90 6 6 1
Dist min=1601779.6250000 max=18215222.0000000 count of >100m = 438902/438913 skipping 3507066
ALLOCATE PREP_HYBRID INIT ARRAYS
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 463
Unable to read MAKBND output from unit 900.
-------------------------------------------

But, I'm not sure if I need to run through prep-hybrid in the first place.

Sim

 

[Ming Chen]

Sim,

I don't think you need to run the prep-hybrid step.
One simple question: Have you run WRF successfully using GFS (not GFSENS) ?
Would you please send me your namelist.wps and namelist.innput to take a look? Also, please tell me the website you download the data and which files you download.
I will see whether I can repeat the errors.

 

[simaberson]

Ming,

Thanks for the help. I have run hundreds of cases using GEFS and GFS data from 2010 to 2020. I have not tried to run using 2021 GFS data.

Namelist.input: &time_control
start_year = 2021, 2021,
start_month = 08, 08,
start_day = 10, 10,
start_hour = 06, 06,
start_minute = 00, 00,
start_second = 00, 00,
tstart = 0.0, 0.0,
end_year = 2021, 2021,
end_month = 08, 08,
end_day = 10, 10,
end_hour = 15, 15,
end_minute = 00, 00,
end_second = 00, 00,
interval_seconds = 10800,
history_interval = 540, 540,
auxhist1_interval = 540, 540,
frames_per_outfile = 1, 1,
frames_per_auxhist1 = 1, 1,
analysis = F, F,
restart = .false.,
restart_interval = 1080,
write_restart_at_0h = .false.,
write_hist_at_0h_rst = .true.,
reset_simulation_start = F,
io_form_input = 2,
io_form_history = 2,
io_form_restart = 2,
io_form_boundary = 2,
io_form_auxinput1 = 2,
io_form_auxhist1 = 2,
auxinput1_inname = "met_nmm.d<domain>.<date>"
debug_level = 0,
override_restart_timers = T,
hwrf_filter = F
/

&fdda
/

&domains
time_step = 6,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 2,
s_we = 1, 1,
e_we = 476, 233,
s_sn = 1, 1,
e_sn = 926, 452,
s_vert = 1, 1,
e_vert = 75, 75,
dx = .0225, .0075,
dy = .0225, .0075,
grid_id = 1, 2,
tile_sz_x = 0,
tile_sz_y = 0,
numtiles = 1,
nproc_x = -1,
nproc_y = -1,
parent_id = 0, 1,
parent_grid_ratio = 1, 3,
parent_time_step_ratio = 1, 3,
i_parent_start = 0, 00309,
j_parent_start = 0, 00173,
feedback = 1,
num_moves = 0,
num_metgrid_levels = 34,
p_top_requested = 1000.,
ptsgm = 15000.,
eta_levels = 1.0, 0.997622, 0.995078, 0.99224, 0.989036,
0.98544, 0.981451, 0.977061, 0.972249, 0.966994,
0.96128, 0.955106, 0.948462, 0.941306, 0.933562,
0.925134, 0.915937, 0.90589, 0.894913, 0.882926,
0.869842, 0.855646, 0.840183, 0.823383,
0.805217, 0.785767, 0.7651, 0.7432, 0.720133,
0.695967, 0.670867, 0.645033, 0.6187, 0.592067,
0.565333, 0.538733, 0.5125, 0.4868, 0.461767,
0.437533, 0.4142, 0.391767, 0.370233, 0.3496,
0.329867, 0.310967, 0.292867, 0.275533,
0.258933, 0.243, 0.2277, 0.213, 0.198867,
0.1853, 0.172267, 0.159733, 0.147633, 0.135967,
0.124767, 0.114033, 0.103733, 0.093867, 0.0844,
0.075333, 0.0666, 0.058267, 0.050333, 0.042833,
0.035733, 0.029, 0.0226, 0.0165, 0.010733,
0.005267, 0.0,
use_prep_hybrid = T,
num_metgrid_soil_levels = 4
/

&physics
num_soil_layers = 4,
mp_physics = 5, 5,
ra_lw_physics = 4, 4,
ra_sw_physics = 4, 4,
sf_sfclay_physics = 88, 88,
sf_surface_physics = 2, 2,
bl_pbl_physics = 3, 3,
cu_physics = 4, 4,
mommix = 1.0, 1.0,
var_ric = 1.0,
coef_ric_l = 0.16,
coef_ric_s = 0.25,
h_diff = 1.0, 1.0,
gwd_opt = 0, 0,
sfenth = 0.0, 0.0,
nrads = 90, 270,
nradl = 90, 270,
nphs = 6, 6,
ncnvc = 6, 6,
ntrack = 6, 18,
gfs_alpha = -1.0, -1.0,
sas_pgcon = 0.2, 0.2,
sas_mass_flux = 0.5, 0.5,
co2tf = 1,
vortex_tracker = 2, 7,
nomove_freq = 6, 6,
tg_option = 1,
ntornado = 6, 18,
ens_cdamp = 0.2,
ens_pblamp = 0.2,
ens_random_seed = 99,
ens_sasamp = 50.0,
icloud = 3,
icoef_sf = 6, 6,
iwavecpl = 0, 0,
lcurr_sf = F, F,
pert_cd = F,
pert_pbl = F,
pert_sas = F,
/

&dynamics
non_hydrostatic = .true, .true,
euler_adv = .false.,
wp = 0, 0,
coac = 1.0, 1.2,
codamp = 6.4, 6.4,
terrain_smoothing = 2,
dwdt_damping_lev = 2000.0, 2000.0,
/

&bdy_control
spec_bdy_width = 1,
specified = .true.
/

&namelist_quilt
poll_servers = .false.
nio_tasks_per_group = 0,
nio_groups = 1
/

&logging
compute_slaves_silent=.true.
io_servers_silent=.true.
stderr_logging=.false.
/

namelist.wps:

&share
wrf_core = 'NMM',
max_dom = 2,
start_date = '2021-08-10_06:00:00',
end_date = '2021-08-10_15:00:00',
interval_seconds = 10800,
io_form_geogrid = 2,
/

&geogrid
parent_id = 1, 1,
parent_grid_ratio = 1, 3,
i_parent_start = 1, 78,
j_parent_start = 1, 96,
e_we = 476, 233,
e_sn = 926, 452,
geog_data_res = '2m','2m',
dx = 0.0225,
dy = 0.0225,
map_proj = 'rotated_ll',
ref_lat = 21.0,
ref_lon = -68.1,
stand_lon = -68.1,
geog_data_path = '/mnt/lfs4/HFIP/hfip-hda/HEDAS_CODE/GEO_DATA_2017'
/

&ungrib
out_format = 'WPS',
prefix = 'GFS',
/

&metgrid
fg_name = 'GFS'
constants_name = 'SKINTEMP:2021-08-10_12',
io_form_metgrid = 2,
/

&mod_levs
press_pa = 201300, 200100, 100000, 95000, 90000, 85000, 80000, 75000, 70000, 65000, 60000, 55000, 50000, 45000, 40000, 35000, 30000, 25000, 20000, 15000, 10000, 5000, 1000, 500, 200,
/

I am getting SKINTEMP from the deterministic run, so the first ungrib namelist has

prefix = 'GFS',

and the second has

start_date = '2021-08-10_12:00:00',
end_date = '2021-08-10_12:00:00',

prefix = 'SKINTEMP',

Sim

 

[simaberson]

I have tried to run this with WRFV4, and get to the same ultimate place in REAL that I got with V3.

cat rsl.error.0000
taskid: 0 hostname: v6
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 463
Unable to read MAKBND output from unit 900.
-------------------------------------------
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor
[v6:mpi_rank_0][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0: Bad file descriptor (9)

My look through the code suggests that this write statement is the only place that MAKBND is even mentioned

grep -ir makbnd *
main/real_nmm.f90:'Unable to read MAKBND output from unit 900.')
main/real_nmm.f90: write(message,*) 'Unable to read MAKBND output from unit ',IUNIT_GFS
main/ideal_nmm.F: call wrf_error_fatal('Unable to read MAKBND output from unit 900.')
main/ideal_nmm.F: write(message,*) 'Unable to read MAKBND output from unit ',IUNIT_GFS
Binary file main/real_nmm.exe matches
main/real_nmm.F: call wrf_error_fatal('Unable to read MAKBND output from unit 900.')
main/real_nmm.F: write(message,*) 'Unable to read MAKBND output from unit ',IUNIT_GFS

So, I'm not sure what MAKBND is. I looked through this forum for all mentions of the Bad file descriptor, and none of the solutions mentioned there applied. I've plotted all the fields in the input files, and they look reasonable.

One interesting thing was that REAL with V4 failed because num_metgrid_level was incorrect, but V3 did not catch that. I'm not sure if that is important.

Sim

 

[Ming Chen]

Sim,
We run WPS/WRF using GFS data downloaded from NCAR CISL (https://rda.ucar.edu/datasets/ds084.1/) and the codes work just fine.
We cannot repeat the errors you have, which makes it hard for us to figure out what is wrong. I am suspicious this is a data issue, --- somehow your GFS data downloaded from NOAA website may be in different coordinate structure, or their GRIB codes are different to those specified in Vtable.GFS?
is it possible that you get GFS data from NCAR CISL, which is open to public for free download, and rerun the case?

 

[simaberson]

Ming,

Thanks for testing this. I don't see the GEFS data at that NCAR URL, and a search through the site didn't point me to the data. I am not sure that GEFS data are available at that site, but you would probably know more. I believe that my setup works fine with GFS data, but I need the 30 ensemble members.

In the meantime, I have placed GEFS data for member 30, plus the deterministic data for SKINTEMP, at ftp://ftp.aoml.noaa.gov/hrd/pub/aberson/NCARforum. wgrib2 shows that the data seem fine.

Thanks,
Sim

 

[simaberson]

I copied over files from the rda.ucar.edu archive, and am getting the same exact result. The only other thing I can think of is that I have been running using the old nemsio files as input, and this is a change to grib. Do I need to do anything to specify the difference? I didn't think so, but there is a parameter gfsinit_type in a config file

gfsinit_type=2 ;; 1=grib2,
2=nemsio,
3=spectral,
4=highres grib2

It isn't clear to me whether this needs to be changed, and whether this is run-time or compile-time parameter.

Sim