Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

obs nudging - wrf crashes shortly into the simulation

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

luna_123456

New member
I am running a nested simulation in WRFv3.8.1 and am running into difficulty setting up the obs nudging. The model crashes shortly after getting started with various errors. I have two soundings that I would like to use to nudge the outer domain (d01) at t = 15:05z and t = 18:30z in the simulation. The process through wrf.exe works and I think my OBS_DOMAIN101 file is being read in correctly since I can see the data in the rsl.out files. The simulation runs just fine if I exclude nudging so I think physics and dynamics are setup fine. WRF will generate the first set of wrfout_d0* files and then reads in the observations and crashes a few seconds into the simulation.

Highlighted in bold are the changes that I made to the namelist and below is the error that I am seeing.

Here is my namelist.input:

&time_control
run_days = 0,
run_hours = 24,
run_minutes = 0,
run_seconds = 0,
start_year = 2011,2011,2011,2011,
start_month = 11,11,11,11,
start_day = 26, 26, 26, 26,
start_hour = 00, 00, 00, 00,
start_minute = 00, 00, 00, 00,
start_second = 00, 00, 00, 00,
end_year = 2011,2011,2011,2011,
end_month = 11,11,11,11,
end_day = 27, 27, 27, 27,
end_hour = 00, 00, 00, 00,
end_minute = 00, 00, 00, 00,
end_second = 00, 00, 00, 00,
interval_seconds = 21600,
input_from_file = .true.,.true.,.true.,.true.,
fine_input_stream = 0,0,0,0,
history_interval = 30, 30, 10, 10,
frames_per_outfile = 1,1,1,1,
restart = .false.,
restart_interval = 5000,
io_form_history = 2,
io_form_restart = 2,
io_form_input = 2,
io_form_boundary = 2,
auxinput11_interval = 1,
auxinput11_end_h = 99999,


/

&domains
time_step = 36,
max_dom = 4,
s_we =
e_we =
s_sn =
e_sn =
s_vert =
e_vert = 110, 110, 110, 110,
num_metgrid_levels = 30,
p_top_requested = 10000.0000,
dx =
dy =
grid_id = 1, 2, 3, 4,
parent_id = 1, 1, 2, 3,
i_parent_start =
j_parent_start =
parent_grid_ratio = 1, 5, 5, 5,
parent_time_step_ratio = 1, 5, 5, 5,
feedback = 0,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false.,.false., .false.,
nested = .false., .true., .true., .true.,
/

&fdda
obs_nudge_opt = 1,0,0,0,
max_obs = 150000,
fdda_start = 0., 0., 0., 0.,
fdda_end = 99999., 99999., 99999., 99999.,
obs_nudge_wind = 1,1,1,1,
obs_coef_wind = 6.E-4,6.E-4,6.E-4,6.E-4,
obs_nudge_temp = 1,1,1,1,
obs_coef_temp = 6.E-4,6.E-4,6.E-4,6.E-4,
obs_nudge_mois = 1,1,1,1,
obs_coef_mois = 6.E-4,6.E-4,6.E-4,6.E-4,
obs_rinxy = 25.,5.,1.,0.2,
obs_rinsig = 0.1,
obs_twindo = 0.6666667,0.6666667,0.6666667,0.6666667,
obs_npfi = 10,
obs_ionf = 1, 2, 2, 2,
obs_idynin = 0,
obs_dtramp = 40.,
obs_prt_freq = 10, 10, 10, 10,
obs_prt_max = 10,
obs_ipf_errob = .true.,
obs_ipf_nudob = .true.,
obs_ipf_in4dob = .true.,
obs_ipf_init = .true.,
/


&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 0,
/



I am primarily getting errors of this nature:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x2B31FCA676F7
#1 0x2B31FCA67D3E
#2 0x2B31FD93B26F
#3 0x1E4F433 in __module_fddaobs_rtfdda_MOD_errob
#4 0x1E57072 in __module_fddaobs_driver_MOD_fddaobs_driver
#5 0x16E07AC in __module_first_rk_step_part2_MOD_first_rk_step_part2._omp_fn.13 at module_first_rk_step_part2.f90:?
#6 0x2B31FD088434
#7 0x2B31FD6F1E24
#8 0x2B31FD9FE34C
#9 0xFFFFFFFFFFFFFFFF

I would also like to be able to start d03 and d04, 12 hours into the simulation. Is offsetting the domain start times possible with nudging?

Any help would be much appreciated!
 
Hi,
Can you send your log files (e.g., rsl.error.* and rsl.out.*)? If you have several, you can package them into 1 .tar file and attach that.
Thanks!
 
So I'm making progress but have not solved this problem. I recompiled WRF with smpar and I have reduced the simulation to one domain and am trying to nudge with only one sounding at t = 19:30z. The only changes to the namelist below is the max number of domains (set to 1 or 2 depending on the case).
For max domains = 1, the simulation sometimes crashes and sometimes doesn't with no changes to the setup. The use of "ulimit -s unlimited" before running WRF seems to help but isn't the solution since it will still crash. In the cases of these crashes, it's happening right around the sounding time (~19:30). Sometimes just before and sometimes just after.

After a successful run with one domain, I wanted to try two, so I modified the namelist and the simulation is crashing around the same time.

Attached is my wrf.log file for a simulation with 2 domains.
 
Hi,
The fact that this crashes sometimes, and not other times with an identical set-up seems to indicate some sort of instability with your system or environment. I would recommend discussing this problem with your systems administrator at your institution to see if they are able to help with this.
 
I ran tests for all of the libraries and recompiled and rebuilt WRF per the instructions here: http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php. Adjusting the systems setup appears to have worked. The simulation no longer arbitrarily crashes with observation nudging.
 
Top