Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

gen be for ensemble-based perturbations

S

satjemkes

Member
Hi,

The user guide indicates that gen_be can be used to generate background covariance matrix from forecast differences (nmc method) or ensemble-based perturbations. However the guide only provides information how to setup gen_be to run for the forecast difference method.
Has anybody in the community ran gen_be for the ensemble_based perturbations and if yes, is it possible to share some experience.
many thanks
stpehen
 
Hi,
I have the same question with you.I want to generate perturbations, which have to remove ensemble mean first, but I could not find the code and I tried to use the gen_be_ep2.exe, but it seems to be something wrong there.
If you have solved this problem, may I ask for your help? Thanks a lot.
Li Ma
 
Hi Li Ma,
I was using the 3rd version of the gen_be code. I have not checked if this is distributed also with the latest version of WRF, it is not an 'official' part of the code, but it is distributed through the main distribution.
Version 1 of genbe takes too long and the Version 2 (which is distributed separately) i had problems reading the perturbations in the code. Version 2 is nice because it generates also netcdf formatted output.

To run the 3rd version of genbe is pretty simple. if you want i can try to find the scripts i was using. I need to look at the scripts to explain to you how to set it up. (I have a general setup i my mind, but before i say something stupid, i prefer to double check the scripts.)

let me know if this would be helpful.

From my experience what seemed critical is the spread of the ensemble. I experimented a little with it. As i have only a small compute system, i could not test it to my like. I played with various perturbation options of WRF, but none of these generated enough spread. So i settled to have different initial conditions.

stephen
 
Dear Li Ma

I just checked the latest version of wrf (4.6.1) and the version 3 of the gen_be is in the var directory. (not sure which version of wrf you are using).
 
Dear stephen,

Sorry，I just saw your reply. It's been bothering me for a long time, and I haven't solved it yet. So if it's convenient for you, can you help me? How can we communicate more conveniently?
The version of wrf I am using is WRF4.4.There are 12 members ,and when I am using gen_be_ep2.exe to generate perturbations, it's supposed to generate 12 temporary files, right?But there are only two files, tmp.e001 and e002.And there were no errors, but it didn't run anymore.
It would be great if you could help me. I would appreciate it.
My email address is ma_li@fudan.edu.cn
Looking forward to your email～
 
Dear Stephen and Li Ma
i’m currently encountering the same problem and would greatly appreciate your assistance. Hoping you could share your expertise to the relevant solution. Any help or guidance would be highly appreciated.

Thank you !
 
Dear LilyLisa (interesting name)

It is possible to run gen_be for an ensemble of n-members, There is however, some confusion as there are different gen_be versions around. There is a standard gen_be as part of the wrf distribution, which is (extremely) slow. then there is a genbe_2 which is NOT part of the distribution as it is a separate distribution. It is much faster, and it generates netcdf formatted files, which might be handy. And then there is a genbe_v3, which is NOT an official part of the wrf distribution, yet it is distributed as part of the distribution. I am using this version. It is relatively fast but it is not generating a netcdf formatted file.

I can send my script which i used (which is tailored to my situation), and my namelist file used for genbe (which is generated for v3 of genbe).

stephen
 
I am trying to generate my own be.dat file while running the gen_be_wrapper.ksh I am encountering this error
---------------------------------------------------------------
Run Stage 0: Calculate ensemble perturbations from model forecasts.
---------------------------------------------------------------
Beginning CPU time: Mon Jun 30 16:41:25 CDT 2025
gen_be_stage0_wrf: Calculating standard perturbation fields valid at time 2024100706
mv: cannot stat 'pert.2024100706*': No such file or directory
2024100606 /work/uharis1/ARW/WPS/WRF5/run/2024100606/wrfout_d01_2024-10-07_06:00:00 /work/uharis1/ARW/WPS/WRF5/run/2024100618/wrfout_d01_2024-10-07_06:00:00
Ending CPU time: Mon Jun 30 16:41:25 CDT 2025
---------------------------------------------------------------
Run Stage 1: Read standard fields, and remove time/ensemble/area mean.
---------------------------------------------------------------
Beginning CPU time: Mon Jun 30 16:41:25 CDT 2025
Stage 1 failed with error 127


can someone help me with this
 
There could be different reasons for this. IF (capitals) the pert file is generated it might be in a different location than the script think it is. So check IF the file exists and if it does then ensure that all paths are correct. If the pert file is NOT generated, then it could be that one of the settings in the namelist file is creating a confusion, OR one of the wrfout file is corrupted. There is a simple check (ask Google how to check if a netcdf file is corrupted). In that case you simply have to regenerate the wrfout file(s). For some odd reason WRF does generate corrupted netcdf files ones in a while.
The structure of the namelist file is a bit obscure. It took me some time to get the hang of it.
(I assume you are trying to generate the bgerr file from a set of forecasts for different times (e.g.6 hour apart), the socalled nmc method, and not from an ensemble of forecasts for the same validity time.
Which version of bgerr are you using?
 
uharis1 said:
I am trying to generate my own be.dat file while running the gen_be_wrapper.ksh I am encountering this error
---------------------------------------------------------------
Run Stage 0: Calculate ensemble perturbations from model forecasts.
---------------------------------------------------------------
Beginning CPU time: Mon Jun 30 16:41:25 CDT 2025
gen_be_stage0_wrf: Calculating standard perturbation fields valid at time 2024100706
mv: cannot stat 'pert.2024100706*': No such file or directory
2024100606 /work/uharis1/ARW/WPS/WRF5/run/2024100606/wrfout_d01_2024-10-07_06:00:00 /work/uharis1/ARW/WPS/WRF5/run/2024100618/wrfout_d01_2024-10-07_06:00:00
Ending CPU time: Mon Jun 30 16:41:25 CDT 2025
---------------------------------------------------------------
Run Stage 1: Read standard fields, and remove time/ensemble/area mean.
---------------------------------------------------------------
Beginning CPU time: Mon Jun 30 16:41:25 CDT 2025
Stage 1 failed with error 127


can someone help me with this
Click to expand...
There could be different reasons for this. IF (capitals) the pert file is generated it might be in a different location than the script think it is. So check IF the file exists and if it does then ensure that all paths are correct. If the pert file is NOT generated, then it could be that one of the settings in the namelist file is creating a confusion, OR one of the wrfout file is corrupted. There is a simple check (ask Google how to check if a netcdf file is corrupted). In that case you simply have to regenerate the wrfout file(s). For some odd reason WRF does generate corrupted netcdf files ones in a while.
The structure of the namelist file is a bit obscure. It took me some time to get the hang of it.
(I assume you are trying to generate the bgerr file from a set of forecasts for different times (e.g.6 hour apart), the socalled nmc method, and not from an ensemble of forecasts for the same validity time.
Which version of bgerr are you using?
 
lilylisa said:
Dear Stephen and Li Ma
i’m currently encountering the same problem and would greatly appreciate your assistance. Hoping you could share your expertise to the relevant solution. Any help or guidance would be highly appreciated.

Thank you !
Click to expand...
Dear LilyLisa (interesting name)

It is possible to run gen_be for an ensemble of n-members, There is however, some confusion as there are different gen_be versions around. There is a standard gen_be as part of the wrf distribution, which is (extremely) slow. then there is a genbe_2 which is NOT part of the distribution as it is a separate distribution. It is much faster, and it generates netcdf formatted files, which might be handy. And then there is a genbe_v3, which is NOT an official part of the wrf distribution, yet it is distributed as part of the distribution. I am using this version. It is relatively fast but it is not generating a netcdf formatted file.

I can send my script which i used (which is tailored to my situation), and my namelist file used for genbe (which is generated for v3 of genbe).

stephen
 
Hi,
I am currently running gen_be to create a background error matrix based on an ensemble forecast using the standard version.
According to the documentation on this page: WRFDA Online Tutorial
the instructions state that for setting this up, one should edit these values in the file gen_be_wrapper.ksh:
export START_DATE=2007091500 # the start and end date should be the same for ensemble GEN_BE
export END_DATE=2007091500
export BE_METHOD=ENS
export NE=10 # However many ensemble members you have
export FCST_RANGE1=00
However, due to how the script gen_be_stage0_wrf.ksh is implemented, it will exit with code -2 after checking that FCST_RANGE2 is greater than FCST_RANGE1, unless one also changes that value.
I made it work by setting both values to 0 and commenting out the exit statement at line 50 of gen_be_stage0_wrf.ksh (another option could be to modify line 48 like "if [[ $DIFF_FCST -lt 0 ]]; then" still having set both FCST_RANGE to 0).
Another difference in my application compared to the instructions is that, because I want to use all hourly wrfouts from the ensemble forecast, I set END_DATE equal to the last available wrfout date.
This seems to work.
If these changes are correct and reasonable, I would like to suggest adding them to the GEN_BE help page. Otherwise, I would be happy to receive suggestions on what the best strategy would be to achieve my goal.
Additionally, I would like to take this opportunity to ask for suggestions on how to parallelize the computation, at least for stages 0 and 1.
 
mnas said:
Hi,
I am currently running gen_be to create a background error matrix based on an ensemble forecast using the standard version.
According to the documentation on this page: WRFDA Online Tutorial
the instructions state that for setting this up, one should edit these values in the file gen_be_wrapper.ksh:
export START_DATE=2007091500 # the start and end date should be the same for ensemble GEN_BE
export END_DATE=2007091500
export BE_METHOD=ENS
export NE=10 # However many ensemble members you have
export FCST_RANGE1=00
However, due to how the script gen_be_stage0_wrf.ksh is implemented, it will exit with code -2 after checking that FCST_RANGE2 is greater than FCST_RANGE1, unless one also changes that value.
I made it work by setting both values to 0 and commenting out the exit statement at line 50 of gen_be_stage0_wrf.ksh (another option could be to modify line 48 like "if [[ $DIFF_FCST -lt 0 ]]; then" still having set both FCST_RANGE to 0).
Another difference in my application compared to the instructions is that, because I want to use all hourly wrfouts from the ensemble forecast, I set END_DATE equal to the last available wrfout date.
This seems to work.
If these changes are correct and reasonable, I would like to suggest adding them to the GEN_BE help page. Otherwise, I would be happy to receive suggestions on what the best strategy would be to achieve my goal.
Additionally, I would like to take this opportunity to ask for suggestions on how to parallelize the computation, at least for stages 0 and 1.
Click to expand...
Hi,

Not quite clear to me. You write you want to run genbe for an ensemble using the "ENS" method, while at the same time you want to use hourly forecast data. if you use an ensemble, then each esemble member has same basetime and the same forecast hour.
For each set you need to prepare a different namelist file with appropriate time. I was using a simple script which prepared this namelist file for each set.
Then it is also not clear which version of genbe you are using. Unless there has been some changes, but the standard version distributed in the wrf tar bal is slow. I have switched to version 3, also in the wrf tar ball. Much faster.
genbe is compiled using the same setup for wrfda. so if you use mpi it parallisation should be autometic. But never checked this if more than cpu was engaged during runtime.
s.
 
Hi,
I can confirm the standard version of gen_be, which comes with the WRF distribution, is very slow, but I didn't manage to compile version 3. I will commit more to that.
Regarding the hourly forecast data, I'm calculating the perturbation for each ensemble member and for each forecast timestep in a certain time window. The aim is to compare this result with what I obtain with only one timestep. I know this is a specific and unusual application, and I reported it in my post just for completeness.
I would be pleased if you could help me with your script for gen_be v3, once I have built it.
 
mnas said:
Hi,
I can confirm the standard version of gen_be, which comes with the WRF distribution, is very slow, but I didn't manage to compile version 3. I will commit more to that.
Regarding the hourly forecast data, I'm calculating the perturbation for each ensemble member and for each forecast timestep in a certain time window. The aim is to compare this result with what I obtain with only one timestep. I know this is a specific and unusual application, and I reported it in my post just for completeness.
I would be pleased if you could help me with your script for gen_be v3, once I have built it.
Click to expand...
there is a makefile for the version 3, it all depends of course on the cpu used, on my linux box i had no problem. My reference to paralisation was reference to version 1. I have not tried to copy the compilation option for the wrfda into the makefile of the version 3.
will try to find the script, which is embedded in a python script for a cycliing experiment. will keep you posted.
 
satjemkes said:
there is a makefile for the version 3, it all depends of course on the cpu used, on my linux box i had no problem. My reference to paralisation was reference to version 1. I have not tried to copy the compilation option for the wrfda into the makefile of the version 3.
will try to find the script, which is embedded in a python script for a cycliing experiment. will keep you posted.
Click to expand...
how good are your python skills? the enclosed module contains python code which i used to generate the be.dat file. I have code for version 2 and version 3 in the python module. the code generates an flist file, prepares the namelist files and calls the gen_be executable. As i have little disk space, i could only save the be.dat file and all other files are removed (sorry).
I have tried to code the module such that i can still read it after some time. so there are little or no python tricks and or intrinsic funtions.
maatdaf.scratch.project.experiment.analysis.genbe: means just the path name of the genbe directory relative to the maatdaf directory. Meaning i run the genbe in a separate directory, which i remove once the be.dat has been generated.
nothing special. hope it helps a little
 
Last edited:
You must log in or register to reply here.
Top