Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Spectral nudging error with mradfg in fftpack5

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

M

melpomene23

New member
Hi,

I am trying to run a simulation with spectral nudging in WRFV4. I first encountered a problem with the run after two hours into the simulation and decided to find the problem with WRF in debug mode, compiled after "./configure -D ". The errors that I get are:

forrtl: severe (408): fort: (2): Subscript #1 of the array CH2 has value 722 which is greater than the upper bound of 1

Image PC Routine Line Source
wrf.exe 000000000C8A9396 Unknown Unknown Unknown
wrf.exe 000000000C01F8E7 mradfg_ 103 mradfg.f90
wrf.exe 000000000BDD5A3F mrftf1_ 140 mrftf1.f90
wrf.exe 000000000BDCB415 rfftmf_ 135 rfftmf.f90
wrf.exe 0000000007D9B610 module_fdda_spnud 1269 module_fdda_spnudging.f90
wrf.exe 0000000007D98896 module_fdda_spnud 1163 module_fdda_spnudging.f90
wrf.exe 0000000007D7C201 module_fdda_spnud 608 module_fdda_spnudging.f90
wrf.exe 0000000007DB62FF module_fddagd_dri 598 module_fddagd_driver.f90
wrf.exe 00000000048B5A17 module_first_rk_s 824 module_first_rk_step_part1.f90
wrf.exe 000000000381019E solve_em_ 892 solve_em.f90
wrf.exe 00000000032B920B solve_interface_ 121 solve_interface.f90
wrf.exe 000000000052AA76 module_integrate_ 329 module_integrate.f90
wrf.exe 000000000040A8D3 module_wrf_top_mp 324 module_wrf_top.f90
wrf.exe 0000000000409D65 MAIN__ 29 wrf.f90
wrf.exe 0000000000409D1E Unknown Unknown Unknown
libc-2.19.so 00002AAAABE69B25 __libc_start_main Unknown Unknown
wrf.exe 0000000000409C29 Unknown Unknown Unknown


Here is my namelist.input:

&time_control
start_year = 2017, 2017,
start_month = 07, 07,
start_day = 17, 17,
start_hour = 00, 00,
start_minute = 00, 00,
start_second = 00, 00,
end_year = 2017, 2017,
end_month = 07, 07,
end_day = 17, 17,
end_hour = 06, 06,
end_minute = 00, 00,
end_second = 00, 00,
interval_seconds = 10800,
input_from_file = .true.,.true.,
history_interval = 10, 10,
frames_per_outfile = 30, 500,
restart = .false.,
restart_interval = 5000,
io_form_history = 12
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2
debug_level = 0
iofields_filename = "my_file.txt", "my_file.txt",
ignore_iofields_warning = .true.,
nocolons = .true.
/

&pio_control
usepio = .true.,
pioprocs = 100,
piostart = 0,
piostride = 36,
/


&domains
time_step = 15,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 2,
e_we = 721, 901,
e_sn = 721, 901,
e_vert = 65, 65,
eta_levels = 1.0000, 0.9975, 0.9950, 0.9926, 0.9901, 0.9876, 0.9852,
0.9827, 0.9802, 0.9778, 0.9751, 0.9721, 0.9689, 0.9652,
0.9612, 0.9566, 0.9516, 0.9459, 0.9395, 0.9324, 0.9244,
0.9154, 0.9054, 0.8942, 0.8817, 0.8678, 0.8523, 0.8353,
0.8166, 0.7962, 0.7740, 0.7501, 0.7244, 0.6970, 0.6682,
0.6379, 0.6065, 0.5740, 0.5409, 0.5074, 0.4736, 0.4400,
0.4068, 0.3742, 0.3425, 0.3118, 0.2823, 0.2541, 0.2274,
0.2022, 0.1761, 0.1541, 0.1340, 0.1157, 0.0990, 0.0839,
0.0703, 0.0579, 0.0468, 0.0368, 0.0278, 0.0196, 0.0124,
0.0058, 0.0000,
p_top_requested = 5000,
sfcp_to_sfcp = .true.,
num_metgrid_levels = 32,
num_metgrid_soil_levels = 4,
dx = 4050, 1350,
dy = 4050, 1350,
grid_id = 1, 2,
parent_id = 0, 1,
i_parent_start = 1, 211,
j_parent_start = 1, 211,
parent_grid_ratio = 1, 3,
parent_time_step_ratio = 1, 3,
feedback = 1,
smooth_option = 1,
rh2qv_method = 2,
wif_input_opt = 1,
num_wif_levels = 30,
/

&physics
mp_physics = 28, 28,
use_aero_icbc = .true.,
ra_lw_physics = 4, 4,
ra_sw_physics = 4, 4,
sf_sfclay_physics = 1, 1,
sf_surface_physics = 2, 2,
bl_pbl_physics = 9, 9,
cu_physics = 0, 0,
shcu_physics = 2, 2,
icloud = 1,
isfflx = 1,
ifsnow = 1,
swint_opt = 1,
radt = 5, 5,
bldt = 0, 0,
num_soil_layers = 4,
num_land_cat = 21,
sf_urban_physics = 0, 0,
iz0tlnd = 1,
isftcflx = 2,
grav_settling = 0, 0,
o3input = 2,
scalar_pblmix = 1,
aer_opt = 3,
/

&fdda
grid_fdda = 2, 2,
gfdda_inname = "wrffdda_d<domain>",
gfdda_interval_m = 180, 180,
gfdda_end_h = 6, 6,
io_form_gfdda = 2,
fgdt = 0, 0,
if_no_pbl_nudging_ph = 1, 1,
if_no_pbl_nudging_uv = 1, 1,
if_no_pbl_nudging_t = 1, 1,
if_no_pbl_nudging_q = 1, 1,
gph = 0.0003, 0.0003
guv = 0.0003, 0.0003,
gt = 0.000045, 0.000045
gq = 0.000045, 0.000045
if_ramping = 0,
xwavenum = 4, 2,
ywavenum = 4, 2,
/

&dynamics
w_damping = 1,
diff_opt = 2, 2,
km_opt = 4, 4,
diff_6th_opt = 2, 2,
diff_6th_factor = 0.3, 0.3,
base_temp = 290.
damp_opt = 3,
zdamp = 5000., 5000.,
dampcoef = 0.25, 0.25,
h_mom_adv_order = 5, 5,
h_sca_adv_order = 5, 5,
v_mom_adv_order = 3, 3,
v_sca_adv_order = 3, 3,
non_hydrostatic = .true., .true.,
moist_adv_opt = 4, 4,
scalar_adv_opt = 4, 4,
chem_adv_opt = 2, 2,
tke_adv_opt = 4, 4,
momentum_adv_opt = 3, 3,
gwd_opt = 0,
use_theta_m = 1,
use_q_diabatic = 1,
epssm = 0.2, 0.2,
emdiv = 0.01, 0.01,
smdiv = 0.1, 0.1,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false.,
nested = .false., .true.,
/

&grib2
/

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/


As a note, I am using WRF with PIO, and Thompson aerosol_aware microphysics scheme, using the aerosol data incorporated in met_em files.
I tried the same prodecure with regular netcdf (io_form_history=2 instead of 12), and got the same errors. However, the debug mode ran successfully for 20 minutes in the simulation without any errors without spectral nudging.

It's my first time using spectral nudging, so I was wondering if there's something I'm doing wrong.

I really appreciate help with this problem, the sooner the better.

Thanks,
 
Spectral nudging is often used to control longer wave phases for long simulation. One general rule to choose the wavenumber nudged is that:
(domain size)/(wavenumber) = 1000km in each direction. For your case, the values of xwavenum and ywavenum seem too large. Can you reset these values and try again?
I looked through all other options in your enamelist, and they look fine.

Ming Chen
 
Hi Ming,

Thanks you for your suggestion. I had used wavenumber=2 for the first domain, and turning off spectral nudging for the second domain, and had encountered the same errors. I don't think it has to do with wave number.
I then changed the wave numbers following a suggestion from a paper, and turned on spectral nudging for the second domain, but the error in mradfg.f90 still persists.

Thank you for your help,
 
I'll also have to add:
The error arises when I run wrf in debug mode. But since it's not advancing beyond this point, the problem will take effect some time in the simulation. As I mentioned, I suspected there's a problem with spectral nudging when the run stopped after 2 hours into the simulation, without aborting the run.
 
Please let me know what your input data is (the data you ungrib to produce initial and boundary conditions). Can you also post your namelist.wps?
Thanks.

Ming Chen
 
I'm using 3 hourly NCEP-GFS data (0.25*0.25 degree resolution).

namelist.wps:

&share
wrf_core = 'ARW',
max_dom = 4,
start_date = '2017-07-17_00:00:00','2017-07-17_00:00:00','2017-07-17_00:00:00','2017-07-17_00:00:00',
end_date = '2017-07-20_00:00:00','2017-07-20_00:00:00','2017-07-20_00:00:00','2017-07-20_00:00:00',
interval_seconds = 10800
io_form_geogrid = 2,
active_grid = .true., .true., .false., .false.,
debug_level = 0
/

&geogrid
parent_id = 1, 1, 2, 3,
parent_grid_ratio = 1, 3, 3, 3,
i_parent_start = 1, 211, 311, 281,
j_parent_start = 1, 211, 311, 281,
e_we = 721, 901, 841, 841,
e_sn = 721, 901, 841, 841,
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!
! The default datasets used to produce the MAXSNOALB and ALBEDO12M
! fields have changed in WPS v4.0. These fields are now interpolated
! from MODIS-based datasets.
!
! To match the output given by the default namelist.wps in WPS v3.9.1,
! the following setting for geog_data_res may be used:
!
! geog_data_res = 'maxsnowalb_ncep+albedo_ncep+default', 'maxsnowalb_ncep+albedo_ncep+default',
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
geog_data_res = '5m','2m','30s','30s',
dx = 4050,
dy = 4050,
map_proj = 'lambert',
ref_lat = 39.3,
ref_lon = -28.0,
truelat1 = 30.0,
truelat2 = 60.0,
stand_lon = -28.0,
geog_data_path = '/glade/u/home/wrfhelp/WPS_GEOG/'
/

&ungrib
out_format = 'WPS',
prefix = 'FILE',
/

&metgrid
fg_name = 'FILE'
io_form_metgrid = 2,
constants_name = 'QNWFA_QNIFA_SIGMA_MONTHLY.dat'
/


Thanks a lot,
 
I repeated your case with the spectral nudging. I run the same case using your namelist.wps and namelist.input, i.e., the same grid and same physics/dynamics options, except that I change the wave numbers to 3 and 1 for the parent and child domains respectively. The other difference is that I use GFS analysis data at 6-hr intervals to drive WRF, which I don't think will cause significant differences between your case and the one I repeated. The model has been running successfully for more than 6 hours and the results look fine. Due to the huge computation resource it cost, I stopped the case after it ran into 7 hours of integration.
I attach the namelist.input I use. Can you repeat this case and see how it works for you? I don't think the codes related to spectral nudging is an issue. However, it is hard for me to tell at present what caused the failure on your side. Please keep me updated about this issue if you figure out the reason.

Ming Chen

&time_control
start_year = 2017, 2017,
start_month = 02, 02,
start_day = 01, 01,
start_hour = 00, 00,
start_minute = 00, 00,
start_second = 00, 00,
end_year = 2017, 2017,
end_month = 02, 02,
end_day = 01, 01,
end_hour = 12, 12,
end_minute = 00, 00,
end_second = 00, 00,
interval_seconds = 21600,
input_from_file = .true.,.true.,
history_interval = 360, 360,
frames_per_outfile = 1, 1,
restart = .false.,
restart_interval = 5000,
io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2
debug_level = 0
/
&domains
time_step = 15,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 2,
e_we = 721, 901,
e_sn = 721, 901,
e_vert = 65, 65,
eta_levels = 1.0000, 0.9975, 0.9950, 0.9926, 0.9901, 0.9876, 0.9852,
0.9827, 0.9802, 0.9778, 0.9751, 0.9721, 0.9689, 0.9652,
0.9612, 0.9566, 0.9516, 0.9459, 0.9395, 0.9324, 0.9244,
0.9154, 0.9054, 0.8942, 0.8817, 0.8678, 0.8523, 0.8353,
0.8166, 0.7962, 0.7740, 0.7501, 0.7244, 0.6970, 0.6682,
0.6379, 0.6065, 0.5740, 0.5409, 0.5074, 0.4736, 0.4400,
0.4068, 0.3742, 0.3425, 0.3118, 0.2823, 0.2541, 0.2274,
0.2022, 0.1761, 0.1541, 0.1340, 0.1157, 0.0990, 0.0839,
0.0703, 0.0579, 0.0468, 0.0368, 0.0278, 0.0196, 0.0124,
0.0058, 0.0000,
p_top_requested = 5000,
sfcp_to_sfcp = .true.,
num_metgrid_levels = 32,
num_metgrid_soil_levels = 4,
dx = 4050, 1350,
dy = 4050, 1350,
grid_id = 1, 2,
parent_id = 0, 1,
i_parent_start = 1, 211,
j_parent_start = 1, 211,
parent_grid_ratio = 1, 3,
parent_time_step_ratio = 1, 3,
feedback = 1,
smooth_option = 1,
rh2qv_method = 2,
wif_input_opt = 1,
num_wif_levels = 30,
/
&physics
mp_physics = 28, 28,
use_aero_icbc = .true.,
ra_lw_physics = 4, 4,
ra_sw_physics = 4, 4,
sf_sfclay_physics = 1, 1,
sf_surface_physics = 2, 2,
bl_pbl_physics = 9, 9,
cu_physics = 0, 0,
shcu_physics = 2, 2,
icloud = 1,
isfflx = 1,
ifsnow = 1,
swint_opt = 1,
radt = 5, 5,
bldt = 0, 0,
num_soil_layers = 4,
num_land_cat = 21,
sf_urban_physics = 0, 0,
iz0tlnd = 1,
isftcflx = 2,
grav_settling = 0, 0,
o3input = 2,
scalar_pblmix = 1,
aer_opt = 3,
/

&fdda
grid_fdda = 2, 2,
gfdda_inname = "wrffdda_d<domain>",
gfdda_interval_m = 360, 360,
gfdda_end_h = 6, 6,
io_form_gfdda = 2,
fgdt = 0, 0,
if_no_pbl_nudging_ph = 1, 1,
if_no_pbl_nudging_uv = 1, 1,
if_no_pbl_nudging_t = 1, 1,
if_no_pbl_nudging_q = 1, 1,
gph = 0.0003, 0.0003
guv = 0.0003, 0.0003,
gt = 0.000045, 0.000045
gq = 0.000045, 0.000045
if_ramping = 0,
xwavenum = 3, 1,
ywavenum = 3, 1,
/

&dynamics
w_damping = 1,
diff_opt = 2, 2,
km_opt = 4, 4,
diff_6th_opt = 2, 2,
diff_6th_factor = 0.3, 0.3,
base_temp = 290.
damp_opt = 3,
zdamp = 5000., 5000.,
dampcoef = 0.25, 0.25,
h_mom_adv_order = 5, 5,
h_sca_adv_order = 5, 5,
v_mom_adv_order = 3, 3,
v_sca_adv_order = 3, 3,
non_hydrostatic = .true., .true.,
moist_adv_opt = 4, 4,
scalar_adv_opt = 4, 4,
chem_adv_opt = 2, 2,
tke_adv_opt = 4, 4,
momentum_adv_opt = 3, 3,
gwd_opt = 0,
use_theta_m = 1,
use_q_diabatic = 1,
epssm = 0.2, 0.2,
emdiv = 0.01, 0.01,
smdiv = 0.1, 0.1,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false.,
nested = .false., .true.,
/

&grib2
/

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/
 
Hi Ming,

Thank you so much for reporting on your run. I have repeated the simulation in debug mode with spectral nudging using different wavenumbers, and I still got the following errors before the integration started, after which the run was aborted:

At line 140 of file mradfg.f90
Fortran runtime error: Index '104' of dimension 1 of array 'wa' above upper bound of 103

Error termination. Backtrace:
#0 0x4ad490d in mradfg_
at /gpfs/fs1/(.....)/WRF/external/fftpack/fftpack5/mradfg.f90:140
#1 0x4a1ebcd in mrftf1_
at /gpfs/fs1/(.....)/WRF/external/fftpack/fftpack5/mrftf1.f90:141
#2 0x4a1ad52 in rfftmf_
at /gpfs/fs1/(.....)/WRF/external/fftpack/fftpack5/rfftmf.f90:138
#3 0x32823fd in __module_fdda_spnudging_MOD_spectralnudgingfilterfft2dncar
at /glade/(.....)/WRF/phys/module_fdda_spnudging.f90:1269
#4 0x3283e07 in __module_fdda_spnudging_MOD_spectral_nudging_filter_3dx
at /glade/(.....)/WRF/phys/module_fdda_spnudging.f90:1163
#5 0x328eaaf in __module_fdda_spnudging_MOD_spectral_nudging
at /glade/(.....)/WRF/phys/module_fdda_spnudging.f90:611
#6 0x329f87e in __module_fddagd_driver_MOD_fddagd_driver
at /glade/(.....)/WRF/phys/module_fddagd_driver.f90:644
#7 0x1f2ae06 in __module_first_rk_step_part1_MOD_first_rk_step_part1
at /glade/(.....)/WRF/dyn_em/module_first_rk_step_part1.f90:887
#8 0x1726b8e in solve_em_
at /glade/(.....)/WRF/dyn_em/solve_em.f90:920
#9 0x152b9f1 in solve_interface_
at /glade/(.....)/WRF/share/solve_interface.f90:139
#10 0x4706ed in __module_integrate_MOD_integrate
at /glade/(.....)/WRF/frame/module_integrate.f90:325
#11 0x408293 in __module_wrf_top_MOD_wrf_run
at ../main/module_wrf_top.f90:324
#12 0x40744d in wrf
at /glade/(....)/WRF/main/wrf.f90:29
#13 0x4074ac in main
at /glade/(....)/WRF/main/wrf.f90:6

** I have left out the directory where I run the simulation on cheyenne.**

The first time I ran the simulation with spectral nudging, it ran for several hours, before becoming idle. I gave it another half an hour hoping that the run would continue, before killing the run manually. That was why I decided to run the case in debug mode (./configure -D) to see if any errors appear, which they did. So my guess is that, even though yours ran successfully for six hours, chances are it might crash somewhere down the line. I'm curious to find out if you'll encounter the same errors if you ran in debug mode.
I am using analysis and 3-hour forecast data.

Thanks,
 
Hi Ming,

I just tried WRF Debug with spectral nudging, with the new wave numbers. (I created wrfinput and wrffda files again, with these new wave numbers and for a shorter time span.)
I got the same errors as before.

I appreciate if you could try the same simulation in debug mode and let me know what you make of the errors if you get them, too.

Thanks a lot,
 
This is a big case that costs huge amounts of computation resources, and running in debug mode will require more. I am sorry that I cannot rerun it.
Can you try the same case but with different forcing data. This is to examine whether the problem in the failed case is caused by unreasonable forcing data. I am perplexing about this case. There is no reason I can run it but you cannot.

Ming Chen
 
Ming,

Just for references, I am running in debug mode just for the first 10 minutes. It's not necessary to run the entire simulation in debug mode, of course. What's more, the run doesn't even go through after these errors. I'm also using quilting as to help improve I/O.

The point is, I can run the case in normal mode (as I have before). But since I know there are potential errors with spectral nudging, I don't want to let it run for, say, 5 hours, and then crash, since my resources are limited, too.
I had used GFS analysis and 3-hour forecast before. The run today was with 6-hour FNL forecast, and I still got the same error. I'm going to try FNL 6-hour analysis, too.
 
The same errors with 6_hour FNL analysis:

At line 140 of file mradfg.f90
Fortran runtime error: Index '104' of dimension 1 of array 'wa' above upper bound of 103

Error termination. Backtrace:
#0 0x4ad490d in mradfg_
at /gpfs/fs1/scratch/mk1369/WRFV4_DEBUG/WRF/external/fftpack/fftpack5/mradfg.f90:140
#1 0x4a1ebcd in mrftf1_
at /gpfs/fs1/scratch/mk1369/WRFV4_DEBUG/WRF/external/fftpack/fftpack5/mrftf1.f90:141
#2 0x4a1ad52 in rfftmf_
at /gpfs/fs1/scratch/mk1369/WRFV4_DEBUG/WRF/external/fftpack/fftpack5/rfftmf.f90:138
#3 0x32823fd in __module_fdda_spnudging_MOD_spectralnudgingfilterfft2dncar
at /glade/scratch_new/mk1369/WRFV4_DEBUG/WRF/phys/module_fdda_spnudging.f90:1269
#4 0x3283e07 in __module_fdda_spnudging_MOD_spectral_nudging_filter_3dx
at /glade/scratch_new/mk1369/WRFV4_DEBUG/WRF/phys/module_fdda_spnudging.f90:1163
#5 0x328eaaf in __module_fdda_spnudging_MOD_spectral_nudging
at /glade/scratch_new/mk1369/WRFV4_DEBUG/WRF/phys/module_fdda_spnudging.f90:611
#6 0x329f87e in __module_fddagd_driver_MOD_fddagd_driver
at /glade/scratch_new/mk1369/WRFV4_DEBUG/WRF/phys/module_fddagd_driver.f90:644
#7 0x1f2ae06 in __module_first_rk_step_part1_MOD_first_rk_step_part1
at /glade/scratch/mk1369/WRFV4_DEBUG/WRF/dyn_em/module_first_rk_step_part1.f90:887
 
Would you please recompile WRF in normal mode (not debug mode), then rerun the case?
I just realized that we have some codes that may fail when run in debug mode (esp with -D), but work fine when running with normally compiled code
I am not sure of the reason yet. We will look into this issue.

Ming Chen
 
You must log in or register to reply here.
Top