wrf.exe error. Segmentation fault - invalid memory reference.

Hello all,

I am doing a simulation on WRFV3.9 but wrf.exe fails 20 days before finishing (I am running 2 months). I get the following error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x7F130EFF9347
#1 0x7F130EFF995E
#2 0x3907C3291F
#3 0x230FAED in __module_sf_sfclayrev_MOD_psim_stable
#4 0x231431C in __module_sf_sfclayrev_MOD_sfclayrev1d
#5 0x2318236 in __module_sf_sfclayrev_MOD_sfclayrev
#6 0x1B82BE2 in __module_surface_driver_MOD_surface_driver
#7 0x1518FD7 in __module_first_rk_step_part1_MOD_first_rk_step_part1
#8 0x10A741A in solve_em_
#9 0xF9DC6F in solve_interface_
#10 0x46B996 in __module_integrate_MOD_integrate
#11 0x407DC3 in __module_wrf_top_MOD_wrf_run

My domains are 27, 9 and 3 km. And even if the recommended time step should be 6*dx, I am running it with 90 because it already failed with 160.
Before I was trying to use version 3.9.1 but it was going very slow and it crashed even before. Now that I am trying with version 3.9 it went beyond and ran faster but still didn't finish.

I am wondering if I can restart the simulation trying with a smaller time step, but I am not sure if that will affect the outputs. Maybe should I start again all the test with another time step?

I would really really appreciate any clue about this error...I don't understand what it's happening since I ran the same case in V3.7 and it went well.

Thanks in advance!.


Just in case, this is the namelist.input I am using:

&time_control
run_days = 100,
run_hours = 0,
run_minutes = 0,
run_seconds = 0,
start_year = 2010, 2010, 2010,
start_month = 09, 09, 09,
start_day = 01, 01, 01,
start_hour = 00, 00, 00,
start_minute = 00, 00, 00,
start_second = 00, 00, 00,
end_year = 2010, 2010, 2010,
end_month = 11, 11, 11,
end_day = 01, 01, 01,
end_hour = 00, 00, 00,
end_minute = 00, 00, 00,
end_second = 00, 00, 00,
interval_seconds = 21600,
input_from_file = .true., .true., .true.,
history_interval = 60, 60, 60,
frames_per_outfile = 1500, 1500, 1,
restart = .false.,
restart_interval = 10000,
io_form_history = 2,
io_form_restart = 2,
io_form_input = 2,
io_form_boundary = 2,
debug_level = 0,
/

&domains
time_step = 90,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 3,
step_to_output_time = .true.
e_we = 132, 100, 55,
e_sn = 100, 88, 61,
e_vert = 50, 50, 50,
p_top_requested = 10286.1,
num_metgrid_levels = 27,
num_metgrid_soil_levels = 4,
dx = 27000, 9000, 3000,
dy = 27000, 9000, 3000,
grid_id = 1, 2, 3,
parent_id = 1, 1, 2,
i_parent_start = 1, 35, 22,
j_parent_start = 1, 31, 25,
parent_grid_ratio = 1, 3, 3,
parent_time_step_ratio = 1, 3, 3,
feedback = 0,
smooth_option = 2,
eta_levels = 1.000, 0.9947, 0.9895, 0.9843, 0.979,
0.9739, 0.9684, 0.9626, 0.9564, 0.9498,
0.9426, 0.9348, 0.9262, 0.9167, 0.9062,
0.8946, 0.8816, 0.8671, 0.8509, 0.833,
0.813, 0.7909, 0.7667, 0.7402, 0.7116,
0.6809, 0.6483, 0.6141, 0.5785, 0.5419,
0.5047, 0.4672, 0.4299, 0.3931, 0.357,
0.322, 0.2883, 0.256, 0.2253, 0.1963,
0.169, 0.1435, 0.1171, 0.0952, 0.0753,
0.0571, 0.0407, 0.0257, 0.0122, 0.000,
/

&physics
mp_physics = 3, 3, 3,
ra_lw_physics = 1, 1, 1,
ra_sw_physics = 1, 1, 1,
radt = 30, 30, 30,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
bl_pbl_physics = 1, 1, 1,
bldt = 0, 0, 0,
cu_physics = 1, 1, 1,
cudt = 5, 5, 5,
isfflx = 1,
ifsnow = 0,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
num_land_cat = 24,
sf_urban_physics = 0, 0, 0,
/

&fdda
/

&dynamics
w_damping = 0,
diff_opt = 1, 1, 1,
km_opt = 4, 4, 4,
diff_6th_opt = 0, 0, 0,
diff_6th_factor = 0.12, 0.12, 0.12,
base_temp = 290.,
damp_opt = 0,
zdamp = 5000., 5000., 5000.,
dampcoef = 0.01, 0.01, 0.01,
khdif = 0, 0, 0,
kvdif = 0, 0, 0,
non_hydrostatic = .true., .true., .true.,
moist_adv_opt = 1, 1, 1,
scalar_adv_opt = 1, 1, 1,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false., .false.,
nested = .false., .true., .true.,
/

&grib2
/

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/

I looked at your namelist.input. I have a few concerns about your options:
(1) One critical issue I notice is that, your integration covers 2-month period, but you didn't set sst_update =1. For long-term simulation, it is recommended to set set_update =1, which updates surface information like SST, LAI, SEAICE, etc. Otherwise these variables will be fixed as values at the initial time, which is physically unreasonable for long-term simulation.
Please set set_update =1 in &physics, and also remember to set the following options in &time_control
auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360 ( minutes generally matches time given by interval_seconds
io_form_auxinput4 = 2 (IO format)

(2) Another issue is that, I notice you turn on cumulus scheme for all the three domains. I would suggest to turn it off in the innermost domain with resolution of 3km. This is because under this resolution, cumulus can be resolved.

(3) Please set w_damping =1. If the domain is located over complex terrain area with large topography gradient, you may also need to increase the value of epssm from the default (0.1) to larger value (e.g. 0.5)

(4) time_step can be 4 x delx, which is kind of conservative.

Please rerun this case with the modification of the namelist.input.

Hello Ming Chen,

Thank you very much for taking a look at my namelist.input and for all the advice regarding this error.
I will do the changes you told me to test again the simulation and I will come back to you to update you about the results.

I really appreciate your help!

Thanks,
Mirce.

Hello Ming Chen,

I am sorry for such a late update about this error.

I made the changes you suggested me and the model ran successfully!
The only change I didn't do was the value for epssm because half of my domain was in a complex area and the rest in a pretty flat one, so that made me doubt if it met the requirements to do the change.

Thank you very much for your support and taking the time to look at my namelist.input, I really appreciate it!

Best,
Mirce.

Mirce,
Thank you for letting me know. I am glad it works!

Thank you MIng Chen,
This solution helps me too!!!

Ming Chen said:

I looked at your namelist.input. I have a few concerns about your options:
(1) One critical issue I notice is that, your integration covers 2-month period, but you didn't set sst_update =1. For long-term simulation, it is recommended to set set_update =1, which updates surface information like SST, LAI, SEAICE, etc. Otherwise these variables will be fixed as values at the initial time, which is physically unreasonable for long-term simulation.
Please set set_update =1 in &physics, and also remember to set the following options in &time_control
auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360 ( minutes generally matches time given by interval_seconds
io_form_auxinput4 = 2 (IO format)

(2) Another issue is that, I notice you turn on cumulus scheme for all the three domains. I would suggest to turn it off in the innermost domain with resolution of 3km. This is because under this resolution, cumulus can be resolved.

(3) Please set w_damping =1. If the domain is located over complex terrain area with large topography gradient, you may also need to increase the value of epssm from the default (0.1) to larger value (e.g. 0.5)

(4) time_step can be 4 x delx, which is kind of conservative.

Please rerun this case with the modification of the namelist.input.

Click to expand...

HI Ming Chen,

I hope you're doing well

I am doing a simulation on WRF 4-3.2 but wrf.exe fails just i am running for 30 hours (6 hours spinup time) I get the following error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x7F94BC9346F7
#1 0x7F94BC934D3E
#2 0x7F94BBE3926F
#3 0x2DCFE8D in __module_sf_sfclayrev_MOD_psim_stable
#4 0x2DD44DC in __module_sf_sfclayrev_MOD_sfclayrev1d
#5 0x2DD93BD in __module_sf_sfclayrev_MOD_sfclayrev
#6 0x23C586A in __module_surface_driver_MOD_sfclayrev_seaice_wrapper
#7 0x2407AA2 in __module_surface_driver_MOD_surface_driver
#8 0x1B044D0 in __module_first_rk_step_part1_MOD_first_rk_step_part1
#9 0x144B8C7 in solve_em_
#10 0x12C32C0 in solve_interface_
#11 0x480C73 in __module_integrate_MOD_integrate
#12 0x481254 in __module_integrate_MOD_integrate
#13 0x481254 in __module_integrate_MOD_integrate
#14 0x408323 in __module_wrf_top_MOD_wrf_run

My domains are 9, 3, and 1 km. I tried with different types of time steps like 90, and 50,45, it just ran upto 4 minutes

kindly help in this regard and suggest to me

Please have a look at the name list :

&time_control
run_days = 0,
run_hours = 30,
run_minutes = 0,
run_seconds = 0,
start_year = 2022, 2022, 2022,
start_month = 07, 07, 07,
start_day = 29, 29, 29,
start_hour = 21, 21, 21,
end_year = 2022, 2022, 2022,
end_month = 07, 07, 07,
end_day = 31, 31, 31,
end_hour = 03, 03, 03,
interval_seconds = 3600
input_from_file = .true.,.true.,.true.,
history_interval = 60, 60, 60,
frames_per_outfile = 1000, 1000, 1000,
restart = .false.,
restart_interval = 5000,
io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2
debug_level = 0
auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360
io_form_auxinput4 = 2
/

&domains
time_step = 45,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 3,
e_we = 325, 271, 325,
e_sn = 325, 271, 325,
e_vert = 45, 45, 45,
dzstretch_s = 1.1
p_top_requested = 5000,
num_metgrid_levels = 34,
num_metgrid_soil_levels = 4,
dx = 9000, 3000, 1000,
dy = 9000, 3000, 1000,
grid_id = 1, 2, 3,
parent_id = 0, 1, 2,
i_parent_start = 1, 117, 81,
j_parent_start = 1, 117, 81,
parent_grid_ratio = 1, 3, 3,
parent_time_step_ratio = 1, 3, 3,
feedback = 0,
smooth_cg_topo = .true.,
smooth_option = 2
eta_levels = 1.0000, 0.9946, 0.9875, 0.9789, 0.9685,
0.9562, 0.9413, 0.9238, 0.9037, 0.8813,
0.8514, 0.8210, 0.7906, 0.7602, 0.7298,
0.6812, 0.6290, 0.5796, 0.5333, 0.4901,
0.4493, 0.4109, 0.3746, 0.3412, 0.3098,
0.2802, 0.2524, 0.2267, 0.2028, 0.1803,
0.1593, 0.1398, 0.1219, 0.1054, 0.0904,
0.0766, 0.0645, 0.0534, 0.0433, 0.0341,
0.0259, 0.0185, 0.0118, 0.0056, 0.0000
/

&physics
physics_suite = 'CONUS'
mp_physics = 6, 6, 6,
cu_physics = 1, 1, 0,
ra_lw_physics = 1, 1, 1,
ra_sw_physics = 1, 1, 1,
bl_pbl_physics = 8, 8, 8,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
radt = 30, 30, 30,
bldt = 0, 0, 0,
cudt = 5, 5, 0,
icloud = 1,
num_land_cat = 21,
sf_urban_physics = 1, 1, 1,
fractional_seaice = 1,
/

&fdda
/

&dynamics
hybrid_opt = 2,
w_damping = 1,
diff_opt = 2, 2, 2,
km_opt = 4, 4, 4,
diff_6th_opt = 0, 0, 0,
diff_6th_factor = 0.12, 0.12, 0.12,
base_temp = 290.
damp_opt = 3,
zdamp = 5000., 5000., 5000.,
dampcoef = 0.2, 0.2, 0.2,
khdif = 0, 0, 0,
kvdif = 0, 0, 0,
non_hydrostatic = .true., .true., .true.,
moist_adv_opt = 1, 1, 1,
scalar_adv_opt = 1, 1, 1,
gwd_opt = 1, 0, 0,
damp_opt = 0,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false.,.false.,
nested = .false., .true., .true.,
/

&grib2
/

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/

Kindly waiting for your reply

Thanks and Regards
Nagaraju Gaddam

Hi Nagaraju

Your namelist.input looks fie except that
(1) please turn off cumulus scheme for D02, that is,
cu_physics = 1, 1, 0,
(2) Please set the radiation time step to be
radt = 9, 9, 9
(3) Please change the PBL scheme to
bl_pbl_physics = 1, 1, 1,

Let me know whether the case can work.

Ming Chen said:

Hi Nagaraju

Your namelist.input looks fie except that
(1) please turn off cumulus scheme for D02, that is,
cu_physics = 1, 1, 0,
(2) Please set the radiation time step to be
radt = 9, 9, 9
(3) Please change the PBL scheme to
bl_pbl_physics = 1, 1, 1,

Let me know whether the case can work.

Click to expand...

Dear Ming Chen,
Thank you for the response I changed it according to your suggestions but it was given the same segmentation error

Please have a look at the attached namelist and rsl. error. tar files

Thanks and regards
Gaddam Nagaraju

Please look at the namelist.input file
&time_control
run_days = 0,
run_hours = 30,
run_minutes = 0,
run_seconds = 0,
start_year = 2022, 2022, 2022,
start_month = 07, 07, 07,
start_day = 29, 29, 29,
start_hour = 21, 21, 21,
end_year = 2022, 2022, 2022,
end_month = 07, 07, 07,
end_day = 31, 31, 31,
end_hour = 03, 03, 03,
interval_seconds = 3600
input_from_file = .true.,.true.,.true.,
history_interval = 60, 60, 60,
frames_per_outfile = 1000, 1000, 1000,
restart = .false.,
restart_interval = 5000,
io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2
debug_level = 0
auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360
io_form_auxinput4 = 2
/

&domains
time_step = 45,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 3,
e_we = 325, 271, 325,
e_sn = 325, 271, 325,
e_vert = 45, 45, 45,
dzstretch_s = 1.1
p_top_requested = 5000,
num_metgrid_levels = 34,
num_metgrid_soil_levels = 4,
dx = 9000, 3000, 1000,
dy = 9000, 3000, 1000,
grid_id = 1, 2, 3,
parent_id = 0, 1, 2,
i_parent_start = 1, 117, 81,
j_parent_start = 1, 117, 81,
parent_grid_ratio = 1, 3, 3,
parent_time_step_ratio = 1, 3, 3,
feedback = 0,
smooth_cg_topo = .true.,
smooth_option = 2
eta_levels = 1.0000, 0.9946, 0.9875, 0.9789, 0.9685,
0.9562, 0.9413, 0.9238, 0.9037, 0.8813,
0.8514, 0.8210, 0.7906, 0.7602, 0.7298,
0.6812, 0.6290, 0.5796, 0.5333, 0.4901,
0.4493, 0.4109, 0.3746, 0.3412, 0.3098,
0.2802, 0.2524, 0.2267, 0.2028, 0.1803,
0.1593, 0.1398, 0.1219, 0.1054, 0.0904,
0.0766, 0.0645, 0.0534, 0.0433, 0.0341,
0.0259, 0.0185, 0.0118, 0.0056, 0.0000
/

&physics
physics_suite = 'CONUS'
mp_physics = 6, 6, 6,
cu_physics = 1, 0, 0,
ra_lw_physics = 1, 1, 1,
ra_sw_physics = 1, 1, 1,
bl_pbl_physics = 1, 1, 1,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
radt = 9, 9, 9,
bldt = 0, 0, 0,
cudt = 5, 5, 0,
icloud = 1,
num_land_cat = 21,
sf_urban_physics = 1, 1, 1,
fractional_seaice = 1,
/

&fdda
/

&dynamics
hybrid_opt = 2,
w_damping = 1,
diff_opt = 2, 2, 2,
km_opt = 4, 4, 4,
diff_6th_opt = 0, 0, 0,
diff_6th_factor = 0.12, 0.12, 0.12,
base_temp = 290.
damp_opt = 3,
zdamp = 5000., 5000., 5000.,
dampcoef = 0.2, 0.2, 0.2,
khdif = 0, 0, 0,
kvdif = 0, 0, 0,
non_hydrostatic = .true., .true., .true.,
moist_adv_opt = 1, 1, 1,
scalar_adv_opt = 1, 1, 1,
gwd_opt = 1, 0, 0,
damp_opt = 0,
/

&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true., .false.,.false.,
nested = .false., .true., .true.,
/

&grib2
/

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,

First, please delete the options below:

auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360
io_form_auxinput4 = 2

This is because you didn't set sst_update =1 and thus you shouldn't specify wrflowinp as input.

Second, How many processors did you use to run this case? With triply-nested domain, it would require relatively large memory. Can you try to run with a larger number of processors, which will give you large memory, and see whether you get the same error?

Also, where is the model domain located? In addition to segmentation fault, did you fid any other error message in your rsl files, such as CFL violation etc?

Ming Chen said:

First, please delete the options below:

auxinput4_inname = "wrflowinp_d<domain>"
auxinput4_interval = 360
io_form_auxinput4 = 2

This is because you didn't set sst_update =1 and thus you shouldn't specify wrflowinp as input.

Second, How many processors did you use to run this case? With triply-nested domain, it would require relatively large memory. Can you try to run with a larger number of processors, which will give you large memory, and see whether you get the same error?

Also, where is the model domain located? In addition to segmentation fault, did you fid any other error message in your rsl files, such as CFL violation etc?

Click to expand...

Dear Ming Chen,

Thank you for the reply, I changed it according to your suggestions but it was giving the same wrf.exe error. Segmentation fault - invalid memory reference.

Please have a look attached files and name lists and WPS configuration domain image
kindly waiting for your reply...

Sincearly
Nagaraju Gaddam

I found many CFL violations in your rsl files. This indicates that this case is numerically unstable. I am suspicious that the finest nested domain (the 1-km d03) might cause the CFL trouble. Would you please do the following changes:
(1) time_step = 36
(2) epssm = 0.5, 0.7, 0.,9 (add this to &dynamics in namelist.input). Then rerun the case.
Please let me know whether it works. Thanks.