Segmentation Fault in WRF-Chem: Issues with wrfinput files modified by ETKF in WRFDA hybrid

[RMWang]

Hello everyone,

I currently work on running WRF-Chem after Hybrid 3DVar-ETKF WRFDA, and encountered a persistent segmentation fault (wrfchem-run stopped at 'calls rrtmg_lw').


The Hybrid 3DVar-ETKF process contains the following steps:

1.wrf_3dvar_input_*.e001,wrf_3dvar_input_*.e002: Be generated by setting “write_input = .true.” in WRF-Chem. WRF 3DVAR input files.

2.gen_be_ensmean.exe: Execute for calculating ensemble mean and variance, to obtain wrf_3dvar_input_*.mean and wrf_3dvar_input_*.vari.

3.gen_be_ep2.exe: Execute for generating ensemble perturbations.

4.gen_be_vertloc.exe: Execute for vertical localization.

5.da_wrfvar.exe: Main WRFDA program, assimilating observations for mean file.

6.gen_be_etkf.exe: ETKF, to update the ensemble perturbations.

7.da_wrfvar.exe: ETKF, to create filtered observations and prepare formatted omb files for ETKF.

8.gen_be_addmean.exe: ETKF, to execute for adding ensemble perturbations to the mean.

9.changeMMINLUinwrfout.py: Python script for modifying wrfinput files(change MMINLU to "MODIFIED_IGBP_MODIS_NOAH", in order to solve other error).



The files below are attached:

1) wrfinput_d01_HybrTrue and wrfinput_d02_HybrTrue, which can successfully running WRF-Chem

2) wrfinput_d01_ETKFfalse and wrfinput_d01_ETKFfalse, which can not successfully running WRF-Chem

3) rsl.error.0000, The error file from WRF-Chem.

4) namelist.input and wrfbdy_d01 which can run WRF-Chem.


Firstly, there is no “cfl” string existed in rsl file.

Then, I use one of the wrfinput members (e.g. wrfinput_d01.e001, modified from wrf_3dvar_input_*.e001, by da_wrfvar.exe of step 6), successfully running the WRF-Chem.


So I speculate it is an issue caused by ETKF (step7-9).

However, I ncdump -h files from 1) and 2) and compare those and find no difference between the head contents of true and false files. Thus I can not find any clue from the comparison.



Respectfully wish for your comments with appreciation in advance for your assistance!
----------------------------------------------------------------------------------------------------------------------------------
Because there is a problem with uploading files to this website, I attached it on another website:

input: namelist.input and 6 other files

The files below are attached: 1) wrfinput_d01_HybrTrue and wrfinput_d02_HybrTrue, which can successfully running WRF-Chem 2) wrfinput_d01_ETKFfalse and wrfinput_d01_ETKFfalse, which can not successfully running WRF-Chem 3) rsl.error.0000, The error file from WRF-Chem. 4) namelist.input and...

files.fm

 

[RMWang]

16th July Update:
I checked the input files between true and false, found that the variables modified by ETKF all becomes "NaN" (see the attached txt file) . I am further checking for the reason. If you can give me a clue, I would be appreciate it!

 

[RMWang]

16th July Update:
I checked the input files between true and false, found that the variables modified by ETKF all becomes "NaN" (see the attached txt file) . I am further checking for the reason. If you can give me a clue, I would be appreciate it!

18th July Updated:
Something wrong still exists but I have no idea, please give me a clue for any file or command I can check, thank you!
It seems that something wrong in step "./gen_be_etkf.exe",

• Zero Eigenvalues: All eigenvalues calculated are zero, indicating an issue in the eigenvalue computation.
• NaN Values in Inflation Calculation: Various calculated values, including the inflation factor, are NaN.
• Extreme Ensemble Values: The average and standard deviation of the ensemble values are extremely large or infinite.
• NetCDF Error: The output process encounters a NetCDF dimension bound error.
• SED Command Error: There is an unterminated s command in a sed expression.

the log is as followes:
[1] Initialize information.
Number of ensemble members = 80
Number of prognostic variables = 7
List of prognostic variables = U V W PH T QVAPOR MU
naccumt1 = 20
naccumt2 = 20
nstartaccum1 = 1
nstartaccum2 = 1
nout = 1
tainflatinput = 1.00000
rhoinput = 1.00000
......
infl_fac_TRNK = F
infl_fac_WG03 = F
infl_fac_WG07 = T
infl_fac_BOWL = F
letkf_flag = F
rnd_seed = 2021051400
rnd_nobs = 5000
etkf_erro_max = 20.00000
etkf_erro_min = 0.00001
etkf_inno_max = 20.00000
etkf_inno_min = 0.00001
etkf_erro_flg = T
etkf_inno_flg = F
etkf_wrfda = F

[2] Read observation information.
Number of unfiltered observations = 15015 1
......
Number of unfiltered observations = 15015 80
Number of filtered observations = 0
begin setup of ob.etkf.filt
complete setup of ob.etkf.filt

[3] Set up arrays using input ensemble mean forecast

[4] Extract necessary fields from WRF ensemble forecasts.

[5] Call ETKF:
Enter ETKF with nout = 1
infl_fac_TRNK F
infl_fac_WG03 F
infl_fac_WG07 T
infl_fac_BOWL F
[1] Compute mean(H(xf)) and H(xf)'
[2] Calculate Transform(HZ)(HZ) in Bishop et al. 2001 Eq.(11)
[3] Calculate C and Gamma in Bishop et al. 2001 Eq.(11)
Eigenvalue 1 0.000000000000000E+000
......
Eigenvalue 80 0.000000000000000E+000

Eigenvalue-vector check 1 0.000000000000000E+000
......
Eigenvalue-vector check 80 0.000000000000000E+000
[4] Calculate inflation factor
USING WG07 INFLATION FACTOR ALGORITHM
squareinno_now 0.000000000000000E+000
squareinno_mn 0.000000000000000E+000
nobs 0.000000000000000E+000
trace 0.000000000000000E+000
ainflat NaN
ainflat_old 1.00000000000000
ainflat_tmp NaN
ainflat_mean NaN
squareinno_mn 0.000000000000000E+000
proj_mn 0.000000000000000E+000
nanals-1 79
squareinno_mn 0.000000000000000E+000
nobs 0
rho -Infinity
[5] Calculate the grand transformation matrix
ENS
MAX ENS 7945.16440105438
MIN ENS -1.024190324627713E+024
EIGEN/(EIGEN*RHO+1)
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
......
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000
SUM_EIGNV= 0.000000000000000E+000
SUM_AFTEREIGNV= 0.000000000000000E+000
AVG_ENS= -4.564473183397777E+018
STD_ENS= 4.674889271458658E+042
[6] Calculate the rescaled ETKF perturbations
ENS
MAX ENS -Infinity
MIN ENS Infinity
AFTER AVG_ENS= NaN
AFTER STD_ENS= NaN
v Variable Ensemble Mean Prior Pert StDv Post. Pert. StDv
1 U 12.1333144***************** NaN
2 V 3.8670637***************** NaN
3 W 0.0161680***************** NaN
4 PH 3284.6399180***************** NaN
5 T 38.1140117***************** NaN
6 QVAPOR 0.0060299***************** NaN
7 MU 371.6183953 439.9620819 NaN

[6] Output ETKF analysis ensemble:
NCVPT: : NetCDF: Start+count exceeds dimension bound
sed: -e expression #1, char 37: unterminated `s' command

 

[RMWang]

Hello everyone, I found the reason of errors in ETKF.
The issue is that: the da_wrfvar.exe program in the /omb/working.e001/ directory, when outputting the ob.etkf.* files, replaced the values in the fifth, sixth, and tenth columns with null values. This led to an error when the ob.etkf.* files were subsequently input into the gen_be_etkf.exe program, causing the program to reject all the observational data.
1) Only observation types of "sound, sonde_sfc" (with apm_plc = 120) and "tamdar, tamdar_sfc" (with apm_plc = 220) can be filtered.
apm_plc is the value that will be written in the fifth of ob.etkf.* files.


2) Rules of filtering in gen_be_etkf.f90:

That is why the observtions are all rejected.
More details can be seen in satjemkes's description.

Fundamental issue with etkf: or just a simple not understanding the logic

If you can give me any clue about this issue, I would be appreciated! Thank you in advance!

 

[RMWang]

A later review suggested that the original failure was related to abnormal fields in the preprocessed ob.etkf.* files. These anomalies caused all observations to be filtered out in gen_be_etkf.exe, leading to nobs = 0, zero eigenvalues, and NaN values in the inflation step, which eventually caused the forecast to crash. After switching the inflation scheme from WG07 to BOWL, the ETKF ran stably and no further NaN values appeared. I have not yet confirmed the fundamental cause, so I am sharing this only as a reference for others facing a similar issue.