Segmentation fault while running wrf.exe and rrtm: TBOUND exceeds table limit: reset 341.054

[Litta]

Hi,
I tried running wrf.exe (WRFV4.6) for a large domain using 32 processors, but encountered a segmentation fault (error [11]). I checked the rsl.error.* files and found some messages related to RRTMG.
Please find the attached namelist.input file and relevant rsl.error.* files.
I tried the suggestion available here. But it was not success yet.
Could you please check and suggest the necessary corrections?

rrtm: TBOUND exceeds table limit: reset 341.054


rrtm: TBOUND exceeds table limit: reset 340.970


rrtm: TBOUND exceeds table limit: reset 340.903


rrtm: TBOUND exceeds table limit: reset 340.779


rrtm: TBOUND exceeds table limit: reset 340.660


rrtm: TBOUND exceeds table limit: reset 340.352


d01 2025-05-21_08:00:00 ----------------------------------------


d01 2025-05-21_08:00:00 W-DAMPING BEGINS AT W-COURANT NUMBER = 1.00000000


d01 2025-05-21_08:00:00 ----------------------------------------


rrtm: TBOUND exceeds table limit: reset 340.545


rrtm: TBOUND exceeds table limit: reset 340.525


rrtm: TBOUND exceeds table limit: reset 340.487


rrtm: TBOUND exceeds table limit: reset 340.310

 

[Ming Chen]

I have a few concerns regarding your namelist.input:

(1) you only have 15 vertical levels, which is way too small. Please increase to at least 41.
(2) set radt = 5
(3) use RRTMG radaition scheme (ra_lw_physics=4, ra_sw_physics=4)

Please modify the above options, then try again. Note that with changes in vertical levels, you need to rerun REAL.

 

[Litta]

Thank you for the message. It is working now with the changes. I would like to reduce the number of upper levels to minimize computation time and memory usage. How many levels can be reduced?

 

[Ming Chen]

Thanks for the update. I am glad it works !

It s recommended that at least 40 levels should be specified in the vertical, ---- I don't think you can save some computation time by reducing vertical levels. But please feel free to try with 35, and I hope it will work for your case.

 

[Litta]

Thank you so much for your suggestion. I could run the model with 35 levels.

 

[Ming Chen]

Thanks for letting me know. I am glad it works!

 

[Litta]

Thank you for your support
I added few more options in namelist.input to get solar radiation outputs (WRF-solar). I got different kind of segmentation fault error (munmap_chunk(): invalid pointer) after generating first file.
Could you please check and suggest the necessary corrections?

YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
This typically refers to a problem with your application.

&physics
mp_physics = 8, 3, 3,
ra_lw_physics = 4, 1, 1,
ra_sw_physics = 4, 1, 1,
radt = 5, 30, 30,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
bl_pbl_physics = 5, 1, 1,
bldt = 0, 0, 0,
topo_wind = 2,
cu_physics = 1, 1, 0,
cudt = 5, 5, 5,
isfflx = 1,
ifsnow = 0,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
sf_urban_physics = 0, 0, 0,
swint_opt = 2,
couple_farms = .true.,
aer_opt = 3,
num_land_cat = 21,


ERROR
INITIALIZE THREE Noah LSM RELATED TABLES
mediation_integrate.G 1242 DATASET=HISTORY
mediation_integrate.G 1243 grid%id 1 grid%oid 1
d01 2025-06-03_00:00:00 Input data is acceptable to use:
Tile Strategy is not specified. Assuming 1D-Y
WRF TILE 1 IS 1210 IE 1451 JS 1 JE 180
WRF NUMBER OF TILES = 1
munmap_chunk(): invalid pointer
Program received signal SIGABRT: Process abort signal.
Backtrace for this error:
#0 0x14ae21823832 in ???
#1 0x14ae21822a05 in ???
#2 0x14ae21454ddf in ???
#3 0x14ae214a157c in ???
#4 0x14ae21454d35 in ???
#5 0x14ae214287f2 in ???
#6 0x14ae2142912f in ???
#7 0x14ae214ab646 in ???
#8 0x14ae214ab8db in ???
#9 0x14ae214af9c9 in ???
#10 0x2fc4f4f in ???
#11 0x284600c in ???
#12 0x217d33d in ???
#13 0x1510896 in ???
#14 0xf9875d in ???
#15 0xdb3e2d in ???
#16 0x4169cb in ???
#17 0x406a51 in ???
#18 0x40604c in ???
#19 0x14ae2143fedf in ???
#20 0x14ae2143ff8f in ???
#21 0x406084 in ???
#22 0xffffffffffffffff in ???

 

[Ming Chen]

The error message doesn't provide any helpful information, which makes it hard to tell what is wrong.

Please recompile WRF in debug mode, then rerun this case. The log file will tell when and where the model crashes first, which may provide some helpful hints about what is wrong.

 

[Litta]

Thank you for your message!
I changed the physics options. Now I got cfl errors. Please check attached namelist.input file and rsl.error. file.
Please provide the recommended physics option for the best solar radiation forecast. Could we use cumulus parameterization option for 5km resolution domain

 

[Ming Chen]

Hi Litta,
I found many CFL violations in your rsl file, which indicates that the model integration becomes unstable. Your physics and dynamics options all look fine to me. However, to overcome the CFL issue, please set the following options in your namelist.input and try again:

(1) time_step = 20
(2) couple_farms = 0
(3) epssm = 0.7

Another issue is, I found that num_metgrid_levels = 26 in your namelist.input. Can you tell me what data did you ungrib to drive this case? The coarse resolution of input data may also cause trouble in some cases. if nothing else is wrong, I would suggest that you try some commomly used data like ERA5 and GFS.

Let me know whether you can run this case successfully.

 

[Litta]

Thank you so much for your support!

I changed the options mentioned above but encountered an error again.
Earlier, I was using ECMWF high-resolution data. Currently, I’m using GFS 0.25° data and have also changed bl_pbl_phys = 1.

It is now working perfectly with these settings.

time_step = 20,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 1,
e_we = 1935, 112, 94,
e_sn = 1435, 97, 91,
e_vert = 35, 30, 30,
p_top_requested = 5000,
num_metgrid_levels = 34,
num_metgrid_soil_levels = 4,
dx = 5000, 10000, 3333.33,
dy = 5000, 10000, 3333.33,
grid_id = 1, 2, 3,
parent_id = 0, 1, 2,
i_parent_start = 1, 31, 30,
j_parent_start = 1, 17, 30,
parent_grid_ratio = 1, 3, 3,
parent_time_step_ratio = 1, 3, 3,
feedback = 1,
smooth_option = 0,
/
&physics
mp_physics = 8, 3, 3,
ra_lw_physics = 4, 1, 1,
ra_sw_physics = 4, 1, 1,
radt = 5, 30, 30,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
bl_pbl_physics = 1, 1, 1,
bldt = 0, 0, 0,
topo_wind = 2,
cu_physics = 1, 1, 0,
cudt = 5, 5, 5,
isfflx = 1,
ifsnow = 1,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
sf_urban_physics = 0, 0, 0,
swint_opt = 2,
aer_opt = 3,

/

 

[Litta]

Currently I am learning satellite radiance assimilation for solar radiation forecast improvement. Could you guide me to set up namelist.input for satellite radiance data assimilation.

 

[Ming Chen]

Thank you so much for your support!

I changed the options mentioned above but encountered an error again.
Earlier, I was using ECMWF high-resolution data. Currently, I’m using GFS 0.25° data and have also changed bl_pbl_phys = 1.

It is now working perfectly with these settings.

time_step = 20,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 1,
e_we = 1935, 112, 94,
e_sn = 1435, 97, 91,
e_vert = 35, 30, 30,
p_top_requested = 5000,
num_metgrid_levels = 34,
num_metgrid_soil_levels = 4,
dx = 5000, 10000, 3333.33,
dy = 5000, 10000, 3333.33,
grid_id = 1, 2, 3,
parent_id = 0, 1, 2,
i_parent_start = 1, 31, 30,
j_parent_start = 1, 17, 30,
parent_grid_ratio = 1, 3, 3,
parent_time_step_ratio = 1, 3, 3,
feedback = 1,
smooth_option = 0,
/
&physics
mp_physics = 8, 3, 3,
ra_lw_physics = 4, 1, 1,
ra_sw_physics = 4, 1, 1,
radt = 5, 30, 30,
sf_sfclay_physics = 1, 1, 1,
sf_surface_physics = 2, 2, 2,
bl_pbl_physics = 1, 1, 1,
bldt = 0, 0, 0,
topo_wind = 2,
cu_physics = 1, 1, 0,
cudt = 5, 5, 5,
isfflx = 1,
ifsnow = 1,
icloud = 1,
surface_input_source = 1,
num_soil_layers = 4,
sf_urban_physics = 0, 0, 0,
swint_opt = 2,
aer_opt = 3,

/

Hi, Litta,
Thank you for the update. I am glad this case finally works !

 

[Ming Chen]

Currently I am learning satellite radiance assimilation for solar radiation forecast improvement. Could you guide me to set up namelist.input for satellite radiance data assimilation.

Litta,
I am not an expert in data assimilation. Unfortunately we no longer provide support for WRFDA due to limited human power.
Hope someone in the community may provide some input.