Segmentation Fault in WRF-Chem: Issues with wrfinput files modified by ETKF in WRFDA hybrid

[RMWang]

Hello everyone,

I currently work on running WRF-Chem after Hybrid 3DVar-ETKF WRFDA, and encountered a persistent segmentation fault (wrfchem-run stopped at 'calls rrtmg_lw').


The Hybrid 3DVar-ETKF process contains the following steps:

1.wrf_3dvar_input_*.e001,wrf_3dvar_input_*.e002: Be generated by setting “write_input = .true.” in WRF-Chem. WRF 3DVAR input files.

2.gen_be_ensmean.exe: Execute for calculating ensemble mean and variance, to obtain wrf_3dvar_input_*.mean and wrf_3dvar_input_*.vari.

3.gen_be_ep2.exe: Execute for generating ensemble perturbations.

4.gen_be_vertloc.exe: Execute for vertical localization.

5.da_wrfvar.exe: Main WRFDA program, assimilating observations for mean file.

6.gen_be_etkf.exe: ETKF, to update the ensemble perturbations.

7.da_wrfvar.exe: ETKF, to create filtered observations and prepare formatted omb files for ETKF.

8.gen_be_addmean.exe: ETKF, to execute for adding ensemble perturbations to the mean.

9.changeMMINLUinwrfout.py: Python script for modifying wrfinput files(change MMINLU to "MODIFIED_IGBP_MODIS_NOAH", in order to solve other error).



The files below are attached:

1) wrfinput_d01_HybrTrue and wrfinput_d02_HybrTrue, which can successfully running WRF-Chem

2) wrfinput_d01_ETKFfalse and wrfinput_d01_ETKFfalse, which can not successfully running WRF-Chem

3) rsl.error.0000, The error file from WRF-Chem.

4) namelist.input and wrfbdy_d01 which can run WRF-Chem.


Firstly, there is no “cfl” string existed in rsl file.

Then, I use one of the wrfinput members (e.g. wrfinput_d01.e001, modified from wrf_3dvar_input_*.e001, by da_wrfvar.exe of step 6), successfully running the WRF-Chem.


So I speculate it is an issue caused by ETKF (step7-9).

However, I ncdump -h files from 1) and 2) and compare those and find no difference between the head contents of true and false files. Thus I can not find any clue from the comparison.



Respectfully wish for your comments with appreciation in advance for your assistance!
----------------------------------------------------------------------------------------------------------------------------------
Because there is a problem with uploading files to this website, I attached it on another website:

input: namelist.input and 6 other files

The files below are attached: 1) wrfinput_d01_HybrTrue and wrfinput_d02_HybrTrue, which can successfully running WRF-Chem 2) wrfinput_d01_ETKFfalse and wrfinput_d01_ETKFfalse, which can not successfully running WRF-Chem 3) rsl.error.0000, The error file from WRF-Chem. 4) namelist.input and...

files.fm

 

[RMWang]

16th July Update:
I checked the input files between true and false, found that the variables modified by ETKF all becomes "NaN" (see the attached txt file) . I am further checking for the reason. If you can give me a clue, I would be appreciate it!

 

[RMWang]

16th July Update:
I checked the input files between true and false, found that the variables modified by ETKF all becomes "NaN" (see the attached txt file) . I am further checking for the reason. If you can give me a clue, I would be appreciate it!

18th July Updated:
Something wrong still exists but I have no idea, please give me a clue for any file or command I can check, thank you!
It seems that something wrong in step "./gen_be_etkf.exe",

• Zero Eigenvalues: All eigenvalues calculated are zero, indicating an issue in the eigenvalue computation.
• NaN Values in Inflation Calculation: Various calculated values, including the inflation factor, are NaN.
• Extreme Ensemble Values: The average and standard deviation of the ensemble values are extremely large or infinite.
• NetCDF Error: The output process encounters a NetCDF dimension bound error.
• SED Command Error: There is an unterminated s command in a sed expression.

the log is as followes:
[1] Initialize information.
Number of ensemble members = 80
Number of prognostic variables = 7
List of prognostic variables = U V W PH T QVAPOR MU
naccumt1 = 20
naccumt2 = 20
nstartaccum1 = 1
nstartaccum2 = 1
nout = 1
tainflatinput = 1.00000
rhoinput = 1.00000
......
infl_fac_TRNK = F
infl_fac_WG03 = F
infl_fac_WG07 = T
infl_fac_BOWL = F
letkf_flag = F
rnd_seed = 2021051400
rnd_nobs = 5000
etkf_erro_max = 20.00000
etkf_erro_min = 0.00001
etkf_inno_max = 20.00000
etkf_inno_min = 0.00001
etkf_erro_flg = T
etkf_inno_flg = F
etkf_wrfda = F

[2] Read observation information.
Number of unfiltered observations = 15015 1
......
Number of unfiltered observations = 15015 80
Number of filtered observations = 0
begin setup of ob.etkf.filt
complete setup of ob.etkf.filt

[3] Set up arrays using input ensemble mean forecast

[4] Extract necessary fields from WRF ensemble forecasts.

[5] Call ETKF:
Enter ETKF with nout = 1
infl_fac_TRNK F
infl_fac_WG03 F
infl_fac_WG07 T
infl_fac_BOWL F
[1] Compute mean(H(xf)) and H(xf)'
[2] Calculate Transform(HZ)(HZ) in Bishop et al. 2001 Eq.(11)
[3] Calculate C and Gamma in Bishop et al. 2001 Eq.(11)
Eigenvalue 1 0.000000000000000E+000
......
Eigenvalue 80 0.000000000000000E+000

Eigenvalue-vector check 1 0.000000000000000E+000
......
Eigenvalue-vector check 80 0.000000000000000E+000
[4] Calculate inflation factor
USING WG07 INFLATION FACTOR ALGORITHM
squareinno_now 0.000000000000000E+000
squareinno_mn 0.000000000000000E+000
nobs 0.000000000000000E+000
trace 0.000000000000000E+000
ainflat NaN
ainflat_old 1.00000000000000
ainflat_tmp NaN
ainflat_mean NaN
squareinno_mn 0.000000000000000E+000
proj_mn 0.000000000000000E+000
nanals-1 79
squareinno_mn 0.000000000000000E+000
nobs 0
rho -Infinity
[5] Calculate the grand transformation matrix
ENS
MAX ENS 7945.16440105438
MIN ENS -1.024190324627713E+024
EIGEN/(EIGEN*RHO+1)
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
......
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000
SUM_EIGNV= 0.000000000000000E+000
SUM_AFTEREIGNV= 0.000000000000000E+000
AVG_ENS= -4.564473183397777E+018
STD_ENS= 4.674889271458658E+042
[6] Calculate the rescaled ETKF perturbations
ENS
MAX ENS -Infinity
MIN ENS Infinity
AFTER AVG_ENS= NaN
AFTER STD_ENS= NaN
v Variable Ensemble Mean Prior Pert StDv Post. Pert. StDv
1 U 12.1333144***************** NaN
2 V 3.8670637***************** NaN
3 W 0.0161680***************** NaN
4 PH 3284.6399180***************** NaN
5 T 38.1140117***************** NaN
6 QVAPOR 0.0060299***************** NaN
7 MU 371.6183953 439.9620819 NaN

[6] Output ETKF analysis ensemble:
NCVPT: : NetCDF: Start+count exceeds dimension bound
sed: -e expression #1, char 37: unterminated `s' command

 

[RMWang]

Hello everyone, I found the reason of errors in ETKF.
The issue is that: the da_wrfvar.exe program in the /omb/working.e001/ directory, when outputting the ob.etkf.* files, replaced the values in the fifth, sixth, and tenth columns with null values. This led to an error when the ob.etkf.* files were subsequently input into the gen_be_etkf.exe program, causing the program to reject all the observational data.
1) Only observation types of "sound, sonde_sfc" (with apm_plc = 120) and "tamdar, tamdar_sfc" (with apm_plc = 220) can be filtered.
apm_plc is the value that will be written in the fifth of ob.etkf.* files.


2) Rules of filtering in gen_be_etkf.f90:

That is why the observtions are all rejected.
More details can be seen in satjemkes's description.

Fundamental issue with etkf: or just a simple not understanding the logic

If you can give me any clue about this issue, I would be appreciated! Thank you in advance!