GFS 0.25 --still-- gives 0 values of Soil temperature with ungrib versions --higher-- than 3.9.0.1

I am changing to GFS 0.25 to feed my WRF models (about time). I decided to install WRF 4.0.2 as I knew prior versions to 3.9.0.1 had a problem giving Soil temperatures of 0 at some points. I compiled the new model using intel compilers (and intel compiled netcdf libraries).

I tested the new 4.0.2 model and everything went fine but when wrf.exe started, it instantaneously crashed giving a segmentation fault (and no extra info). I attach the last part of the rsl.error (the ones that had an error).

First thing I checked is that the met_em files had the right temperature, and bingo! the soil temperatures still had zero values on some points.

I have tried compiling and running WPS versions 3.9.1 and 3.9.0.1 of WRF to see if that solved the problem but it doesn't.

in case it may be of use I run WRF in a AWS cluster using a C4.x instance with 36 cores (that worked fine for the other version of WRF, 3.6.1).

METGRID.TBL.AWR and Vtable.GFS are the correct ones (the exact same file for the three versions, 4.02,3.9.1,3.9.0.1, the WPSv3.6.1 gave NUM_MET_SOIL_LEV =0 so I had to change)


-----------------------------------------------
rsl.error.00xx (only the tail part):
-----------------------------------------------

INITIALIZE THREE Noah LSM RELATED TABLES
mediation_integrate.G 1943 DATASET=HISTORY
mediation_integrate.G 1944 grid%id 1 grid%oid 1
d01 2018-12-12_00:00:00 Input data is acceptable to use:
d01 2018-12-12_00:00:00 Input data processed for aux input 10 for domain 1
d01 2018-12-12_00:00:00 Input data is acceptable to use:
Tile Strategy is not specified. Assuming 1D-Y
WRF TILE 1 IS 105 IE 156 JS 95 JE 140
WRF NUMBER OF TILES = 1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
wrf.exe 000000000303DE25 Unknown Unknown Unknown
wrf.exe 000000000303BBE7 Unknown Unknown Unknown
wrf.exe 0000000002FDD184 Unknown Unknown Unknown
wrf.exe 0000000002FDCF96 Unknown Unknown Unknown
wrf.exe 0000000002F69026 Unknown Unknown Unknown
wrf.exe 0000000002F6FB00 Unknown Unknown Unknown
libpthread.so.0 00001476F82695E0 Unknown Unknown Unknown
wrf.exe 00000000028F7FA3 Unknown Unknown Unknown
wrf.exe 00000000028F1177 Unknown Unknown Unknown
wrf.exe 00000000023A7A3D Unknown Unknown Unknown
wrf.exe 0000000001BA992F Unknown Unknown Unknown
wrf.exe 0000000001319588 Unknown Unknown Unknown
wrf.exe 00000000011A2570 Unknown Unknown Unknown
wrf.exe 000000000050EB8F Unknown Unknown Unknown
wrf.exe 000000000040C391 Unknown Unknown Unknown
wrf.exe 000000000040C34F Unknown Unknown Unknown
wrf.exe 000000000040C2EE Unknown Unknown Unknown
libc.so.6 00001476F7BAD445 Unknown Unknown Unknown
wrf.exe 000000000040C1E9 Unknown Unknown Unknown

It could be the compiler. Try building WRF with a lower optimization level.

Sometimes, the Intel compile messes up optimization with WRF.

Can you upload one GFS file and one met_em file for me to take a look? It is also helpful to provide your namelist.wps and namelist.input. I would like to repeat your case. Thanks.

kwthomas said:

It could be the compiler. Try building WRF with a lower optimization level.

Sometimes, the Intel compile messes up optimization with WRF.

Click to expand...

You mean WRF or just ungrib.exe/metgrid.exe?

Can you upload one GFS file and one met_em file for me to take a look? It is also helpful to provide your namelist.wps and namelist.input. I would like to repeat your case. Thanks.

Click to expand...

I couldn't attach the GFS as it is so large but its the filtered gfs 0.25 that can be downloaded here:
http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl

!namelist.wps =======================================================================
!-----------------------------------------------------------------------
! Common options:
!-----------------------------------------------------------------------
&share
wrf_core = 'ARW',
max_dom = 1,
start_date = '2018-12-14_00:00:00', '2018-12-14_00:00:00',
end_date = '2018-12-21_00:00:00', '2018-12-21_00:00:00',
interval_seconds = 10800,
io_form_geogrid = 2,
debug_level = 0,
/

!-----------------------------------------------------------------------
! Options for geogrid.exe:
!-----------------------------------------------------------------------
&geogrid
! Model grid definitions:
parent_id = 1, 1,
parent_grid_ratio = 1, 3,
i_parent_start = 1, 180,
j_parent_start = 1, 120,
e_we = 313, 361,
e_sn = 280, 262,
dx = 10000,
dy = 10000,
! Map projection definitions:
map_proj = 'mercator',
ref_lat = 21.5,
ref_lon = 45.5,
truelat1 = 25,
truelat2 = 0,
stand_lon = 45,
!ref_x = 45.0,
!ref_y = 25.0,
! Static data configuration:
geog_data_path = '/share/data/GEOG/WPS_GEOG',
geog_data_res = '10s','30s',
/

!-----------------------------------------------------------------------
! Options for ungrib.exe:
!-----------------------------------------------------------------------
&ungrib
out_format = 'WPS',
prefix = 'GFS',
/

!-----------------------------------------------------------------------
! Options for metgrid.exe:
!-----------------------------------------------------------------------
&metgrid
fg_name = 'GFS',
io_form_metgrid = 2,
/

! namelist.input ============================================================================
!-----------------------------------------------------------------------
! Model run definitions:
!-----------------------------------------------------------------------
&time_control
! Set start and end times:
start_year = 2018,
start_month = 12,
start_day = 14,
start_hour = 0,
start_minute = 00, 00,
start_second = 00, 00,
end_year = 2018,
end_month = 12,
end_day = 21,
end_hour = 0,
end_minute = 00, 00,
end_second = 00, 00,
! Input data settings:
interval_seconds = 10800,
input_from_file = .true., .true.,
! Output settings:
history_interval = 60, 60,
frames_per_outfile = 1, 1,
restart = .false.,
restart_interval = 99999,
io_form_history = 2,
io_form_restart = 2,
io_form_input = 2,
io_form_boundary = 2,
iofields_filename = "iofields_d01.txt", "iofields_d02.txt",
ignore_iofields_warning = .true.,
adjust_output_times = .true.,
! Debug settings:
debug_level = 0,
/

!-----------------------------------------------------------------------
! Model domain definitions:
!-----------------------------------------------------------------------
&domains
! Time step configuration:
time_step = 30,
time_step_fract_num = 0,
time_step_fract_den = 1,
parent_time_step_ratio = 1, 2,
! Adaptive time step configuration:
use_adaptive_time_step = .true.,
step_to_output_time = .true.,
target_cfl = 1.2, 1.2,
target_hcfl = 0.84, 0.84,
max_step_increase_pct = 5, 51,
starting_time_step = -1, -1,
max_time_step = 300, 100,
min_time_step = -1, -1,
adaptation_domain = 1,
! Model domain definitions:
max_dom = 1,
s_we = 1, 1,
s_sn = 1, 1,
s_vert = 1, 1,
e_we = 313, 361,
e_sn = 280, 262,
e_vert = 50, 50
dx = 10000, 3333.33,
dy = 10000, 3333.33,
p_top_requested = 1000,
grid_id = 1, 2,
parent_id = 1, 1,
i_parent_start = 1,180,
j_parent_start = 1,120,
parent_grid_ratio = 1, 3,
! Nest feedback options:
feedback = 1,
smooth_option = 0,
! Shape of incoming data:
num_metgrid_levels = 32,
num_metgrid_soil_levels = 4,
! Time series output locations:
max_ts_locs = 0,

! eta_levels = 1.0, 0.995253, 0.990479, 0.985679, 0.980781, 0.975782, 0.970684, 0.965486, 0.960187, 0.954689, 0.948991, 0.943093, 0.936895, 0.930397, 0.923599, 0.916402, 0.908404, 0.899507, 0.888811, 0.876814, 0.862914, 0.847114, 0.829314, 0.809114, 0.786714, 0.762114, 0.735314, 0.706714, 0.676614, 0.645814, 0.614214, 0.582114, 0.549714, 0.517114, 0.484394, 0.451894, 0.419694, 0.388094, 0.356994, 0.326694, 0.297694, 0.270694, 0.245894, 0.223694, 0.203594, 0.185494, 0.169294, 0.154394, 0.140494, 0.127094, 0.114294, 0.101894, 0.089794, 0.078094, 0.066594, 0.055294, 0.044144, 0.033054, 0.022004, 0.010994, 0.0,


!eta_levels = 1.0000,0.9990,0.9970,0.9950,0.9930,0.9900,0.9870,0.9850,0.9830,0.9810,0.9790,0.9760,
!0.9730,0.9700,0.9660,0.9610,0.9550,0.9490,0.9430,0.9360,0.9290,0.9220,0.9150,0.9070,
!0.8990,0.8890,0.8790,0.8690,0.8490,0.8290,0.8090,0.7890,0.7590,0.7290,0.6990,0.6690,
!0.6390,0.6000,0.5500,0.5000,0.4500,0.4000,0.3500,0.3000,0.2500,0.2000,0.1500,0.1000,
!0.0500, 0
/

!-----------------------------------------------------------------------
! Model physics options:
!-----------------------------------------------------------------------
&physics
mp_physics = 1, 1,
ra_lw_physics = 1, 1,
ra_sw_physics = 2, 2,
radt = 30, 30,
sf_sfclay_physics = 4, 4,
iz0tlnd = 1,
sf_surface_physics = 2, 2,
bl_pbl_physics = 4, 4,
bldt = 0, 0,
cu_physics = 1, 1,
cudt = 0, 0,
isfflx = 1,
ifsnow = 0,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
maxiens = 1,
maxens = 3,
maxens2 = 3,
maxens3 = 16,
ensdim = 144,
/

!-----------------------------------------------------------------------
! Grid nudging:
!-----------------------------------------------------------------------
&fdda
grid_fdda = 2,2,
xwavenum = 3
ywavenum = 5
gfdda_inname = "wrffdda_d<domain>",
gfdda_interval_m = 360,360
gfdda_end_h = 9999,9999,
io_form_gfdda = 2,
fgdt = 0,0,
guv = 0.0003, 0.0003
gt = 0.0003, 0.0003
gq = 0.0003, 0.0003
gph = 0.0003
if_no_pbl_nudging_uv = 1, 1
if_no_pbl_nudging_t = 1, 1
if_no_pbl_nudging_q = 1, 1
if_no_pbl_nudging_ph = 1
/


!-----------------------------------------------------------------------
! Model dynamics options:
!-----------------------------------------------------------------------
&dynamics
rk_ord = 3,
diff_opt = 1,
km_opt = 4,
diff_6th_opt = 2,
diff_6th_factor = 0.12,
damp_opt = 3,
zdamp = 5000., 5000.,
dampcoef = 0.2, 0.2,
w_damping = 1,
base_pres = 100000.,
base_temp = 290.,
base_lapse = 50.,
khdif = 0, 0,
kvdif = 0, 0,
smdiv = 0.1, 0.1,
emdiv = 0.01, 0.01,
epssm = 0.1, 0.1,
non_hydrostatic = .true., .true.,
h_mom_adv_order = 5, 5,
v_mom_adv_order = 3, 3,
h_sca_adv_order = 5, 5,
v_sca_adv_order = 3, 3,
moist_adv_opt = 2, 2,
scalar_adv_opt = 2, 2,
tke_adv_opt = 2, 2,
gwd_opt = 0,
dyn_opt = 2,
/

!-----------------------------------------------------------------------
! Boundary condition options:
!-----------------------------------------------------------------------
&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false.,
nested = .false., .true.,
/

!-----------------------------------------------------------------------
! GRIB2 output options (not used):
!-----------------------------------------------------------------------
&grib2
background_proc_id = 255,
forecast_proc_id = 255,
production_status = 255,
compression = 40,
/

!-----------------------------------------------------------------------
! Parallel I/O quilting options:
!-----------------------------------------------------------------------
&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/

Ming Chen said:

Can you upload one GFS file and one met_em file for me to take a look? It is also helpful to provide your namelist.wps and namelist.input. I would like to repeat your case. Thanks.

Click to expand...

Ming chen were you able to reproduce the error?

I have repeated your case and I cannot reproduce your errors. Below is what I did:
(1) download gfs files from NCAR CISL (gfs.0p25.2018121400.f000.grib2 and gfs.0p25.2018121400.f006.grib2)
(2) run WPS to produce met-em files
(3) run REAL and WRF for a 6-hour simulation
This case can be done successfully without any problem.

I also looked at your met_em file, which apparently includes soil information (SM and ST both have reasonable values).

Can you download GFS data from this website:
https://rda.ucar.edu/datasets/ds084.1/

Then try again?

By the way, please don't run with the grib2 option, i.e., delete the following part in your namelist (you seemed to turn it off. But just in case, please double check)

&grib2
background_proc_id = 255,
forecast_proc_id = 255,
production_status = 255,
compression = 40,
/

Hi Ming Chen,

Thanks for the prompt reply. find below my comments:

This case can be done successfully without any problem.

Click to expand...

Great to hear that, which version of WPS are you using?

(1) download gfs files from NCAR CISL (gfs.0p25.2018121400.f000.grib2 and gfs.0p25.2018121400.f006.grib2)

Click to expand...

I need it to be the same one as the opreative GFS as its an operative forecast. Any chance you may try again with the gfs from the url I posted on my previous post? I will try with the files you suggested nonetheless.

I also looked at your met_em file, which apparently includes soil information (SM and ST both have reasonable values).

Click to expand...

I attached a screenshot of another area (the variable is ST000010) but with the same problems, on the pic you can see there are small areas with 0°K values (I marked them with blue circles), the same happens on the met_em file I attached. In my opinion that is what it's causing the model crash.


Thanks a lot for your help much appreciated.

(2) run WPS to produce met-em files

Click to expand...

also would you be able to give me a copy of your ungrib.exe file, given that it does not run in parallel should ( :lol: ) work fine (i can use both gfortran and intel compiled). It's just to try to discard problems.

Please use WPSv4.0 to ungrib the GFS file. let me know whether the newer version works.

The post it's a bit old But I found What the problem was. And it wans't WRF but the input as I was downloading GFS using the grib Filter. I forgot to tick both LAND and LANDN on the filter web (only LAND was included which is not enough to run the new GFS) so when the postprocess run without any problem but the WRF crashed.

Everything works fine now.

Dear all users,

As some other user from this forum Since November 7th, we have been having trouble implementing our real-time WRF-ARW operating system. I have read that since then NCEP is upgrading GFS from version 15.1 to 15.2. Other users also report errors like mine, but so far I have not met the correct answer and that is why I think the reason is in the data. We use 0.25x0.25 GFS data with filter from

URL="http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25......

and we use this data from some years without problems, we haven't a problem when the GFS Model was upgraded to version 15.1 on June 12, 2019. We use WRF v.6.1 and we use the ungrib program from WPS - V4.0 as recommended. Our INTEL compiler is old, so we can't release a newer version of WRF.

I read that some of the WRF users encountered a problem with the land-sea mask change after Nov.11 the gfs.v15.2 implementation.

We successfully run WPS and real.exe. But since GFS version update to 15.2 (November 7, 12Z), I'm experiencing trouble during wrf.exe. My run stops with a lot of print-outs in the rsl.out* files that look like:

Flerchinger USEd in NEW version. Iterations= 10
Flerchinger USEd in NEW version. Iterations= 10
Flerchinger USEd in NEW version. Iterations= 10

and in rsl.error:
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source

wrf.exe 0000000000405937 Unknown Unknown Unknown
wrf.exe 00000000004058CC Unknown Unknown Unknown
libc.so.6 00000033D8C1D994 Unknown Unknown Unknown
wrf.exe 00000000004057D9 Unknown Unknown Unknown


I read that ""The Flerchinger message in itself is not an error but simply a status message sent by the Noah LSM. However, it appears to occur in cases of abnormal model state. For this reason, you can get rid of the message by changing the LSM but the underlying error will remain and may still cause the model to crash, write NaN or do other bad things. The Flerchinger message does not point towards the source of the problem.""

I change as you recommended the sf_surface_physics to = 4, Noah-MP land-surface model. But the result is the same.

We also have a 5 domain system WRF/CMAQ which use GFS data 1x1 deg without problem with GFS v.15.2.

Any help would be greatly appreciated.
Regards,
Maria
-
*************************************************
Maria Prodanova-Boiadjieva
National Institute of Meteorology and Hydrology
66 Tzarigradsko shaussee, Sofia 1784, Bulgaria
Phone: (+3592) 462 4598
Fax: (+3592) 988 03 80, 988 44 94
e-mail:Maria.Prodanova@abv.bg ; Maria.Prodanova@meteo.bg
**************************************************

Maria,
I believe this is a data issue. Please double check your data to make sure landmask and soil fields are consistent (i.e., no land point with invalid soil data). Please also download GFS data from NCAR CISL for the same date of your failed case, and compare the data with those you download from NCEP.
Since the GFS data archived in CISL works fine for WRF, I would suggest you look at the differences, which kght give you hints what is wrong in the data you get from NCEP.

Hi Ming Chen,

Thank you very much for your reply. I also think that is from the data with the new version of GFS v.15.2. This problem is with the land-sea mask change
after Nov.11. Our system is operative and I use GFS data 0.25 deg from https://nomads.ncep.noaa.gov/ with filter. The system work from many years and also work now with archive with old version of GFS.

I saw about NCAR CISL data, but I can't understand if they are in real time. I don't succeeded to find the Internet page of this data. I will download GFS data from NCAR CISL for the same date, and will compare the data with those you download from NCEP, but I failed to find the page of NCAR CISL archive

My opinion is is - How to resolve this data problem with GFS data from nomads page in WRF model.

Maria

Hi María,

As I said on a previous post in my case it was the grib filter was ionly downloading one of the land masks (LAND) but now there are two LAND and LANDN make sure you download both of them (and of course use the latest ungrib.exe and Vtable versions or it won't work.