dawrfvar crashes with double free or corruption

[Feng Liu]

Hi,
I am going to run WRFDA 4DVAR (WRF v4.1.2) using ds337.0 (NCEP ADP Global Upper Air and Surface Weather Observations (PREPBUFR format). The program da_wrfvar.exe always crashes with an error as below:
I am struggling to figure out the issue but in vain. I would like to get your help because I know you are an outstanding expert in WRFDA.

*** Error in `./da_wrfvar.exe': double free or corruption (!prev): 0x000000008c132040 ***
Sometimes it becomes

*** Error in `./da_wrfvar.exe': corrupted double-linked list: 0x0000000074453170 ***

More detail screen output can be found in the wa_wrfvar.log attached.

The rsl.error and rsl.out (see attached) do not show any error messages even though a high debug level is set in namelist.input (attached).

I also used compile option, Mbounds, to check compilation, and the program Valgrind ( (http://valgrind.org/) to detect memory management while running da_wrfvar.exe. No any errors and problems were founded. I do not know if we need to concern the WARNING like

--------------------------- WARNING ---------------------------
WARNING FROM FILE: da_read_obs_bufrgpsro.inc LINE: 115
Error 209 opening PREPBUFR obs file gpsro.bufr
---------------------------------------------------------------


Please also see the namelist.input file attached.

What I did in my work directory WRFDA/workdir are
1. Copy entire WRFDA/run as new directory workdir.
2. Link the ../var/run/be.dat.CV3 as be.dat at the workdir.
3. Link wrfbdy_d01 and wrfinput_d01 at the same directory.
4. Link wrfinput_d01 as fg.
5. Link prepbufr.gdas.20131228.t00z.nr and prepbufr.gdas.20131228.t06z.nr as ob01.bufr and ob02.bufr, respectively.
6. I also link or changed three files’ name
a. ln –sf RRTM_DATA_DBL RRTM_DATA
b. ln –sf RRTMG_LW_DATA_DBL RRTMG_LW_DATA
c. ln –sf RRTMG_SW_DATA_DBL RRTMG_SW_DATA
Because if I do not change above files' names the following error, for example if not change RRTMG_LW_DATA_DBL into RRTMG_LW_DATA
FATAL CALLED FROM FILE: <stdin> LINE: 12932
module_ra_rrtmg_lw: error reading RRTMG_LW_DATA on unit 10


Thank you very much for taking the time to help me.

 

[Feng Liu]

Hi,
For more details, please find debug information on this issue

(gdb) info source
Current source file is ../nptl/sysdeps/unix/sysv/linux/raise.c
Compilation directory is /usr/src/debug/glibc-2.17-c758a686/signal
Located in /usr/src/debug/glibc-2.17-c758a686/nptl/sysdeps/unix/sysv/linux/raise.c
Contains 58 lines.
Source language is c.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.
(gdb) bt
#0 0x00002aaaaee302c7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00002aaaaee319b8 in __GI_abort () at abort.c:90
#2 0x00002aaaaee72e17 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x2aaaaef847f8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00002aaaaee7b609 in malloc_printerr (ar_ptr=0x2aaaaf1c0760 <main_arena>, ptr=<optimized out>,
str=0x2aaaaef84920 "double free or corruption (!prev)", action=3) at malloc.c:5004
#4 _int_free (av=0x2aaaaf1c0760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000273aab4 in a_module_em::a_rk_update_scalar () at module_em_ad.f90:3472
#6 0x00000000025926cf in solve_em_ad () at solve_em_ad.f90:6451
#7 0x00000000021953fb in solve_interface () at solve_interface.f90:221
#8 0x000000000127628a in module_integrate::integrate () at module_integrate.f90:365
#9 0x000000000041466d in module_wrf_top::wrf_run_ad () at ../main/module_wrf_top.f90:1655
#10 0x000000000047fa9c in da_4dvar::da_ad_model () at da_4dvar.f:518
#11 0x0000000000ab224c in da_minimisation::da_transform_vtoy_adj () at da_minimisation.f:3436
#12 0x0000000000a96edb in da_minimisation::da_calculate_gradj () at da_minimisation.f:596
#13 0x0000000000aa5f6c in da_minimisation::da_minimise_cg () at da_minimisation.f:2641
#14 0x00000000004621ca in da_wrfvar_top::da_solve () at da_wrfvar_top.f:3752
#15 0x000000000043a5e0 in da_wrfvar_top::da_wrfvar_interface () at da_wrfvar_top.f:2841
#16 0x000000000043a533 in da_wrfvar_top::da_wrfvar_run () at da_wrfvar_top.f:2811
#17 0x000000000040a204 in da_wrfvar_main () at da_wrfvar_main.f:35
#18 0x000000000040a176 in main ()
#19 0x00002aaaaee1c495 in __libc_start_main (main=0x40a140 <main>, argc=1, argv=0x7fffffffde28, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffde18) at ../csu/libc-start.c:266
---Type <return> to continue, or q <return> to quit---
#20 0x000000000040a079 in _start ()