Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Some questions about the open data of ECMWF？

W

WannaJune

New member
First, I used your official method to process the ECMWF data using the grib _ set -r -w packingType = grid _ ccsds -s packingType = grid _ simple input. Grib2 output.grib2.After processing, WPS can indeed run normally, but when WRF runs, it will prompt such an error, which I put in the attachment.
After that, I used the GFS data and ECMWF data of the same period to perform WPS pre-processing, and got two kinds of.nc intermediate files. I parsed the two files with NCL, and found that the two files were different. (The data I downloaded here are all complete data, but ECMWF does not include stl1.But stl2, stl3, stl4 and swvl1, swvl2, swvl3, swvl4 are all data related to the number of soil layers.) The directory structure of ECMWF and GFS processed by the metgrib program of WPS is attached here, in which the number of soil layers of ECMWF is only one.The number of vertical layers is 14, while the number of GFS soil layers is 4, and the number of vertical layers is 34. That is to say, the method of transcoding ECMWF provided by you does not work for some documents. Is there any other solution?
 

Attachments

  • ecmwf_met.txt
    17.1 KB · Views: 5
  • gfs_met.txt
    23.7 KB · Views: 1
  • rsl.error.txt
    1.9 KB · Views: 5
Hi,
One of the problems with the EC data is that it's not all standardized like GFS data. Depending on where/how the data are obtained, they can be different. We've found that, for e.g., for one user, ungrib is unable to recognize the soil fields. A possible solution for this is to use GFS soil data, in addition to the EC data. We expect the output to still be reasonable, especially because the WRF LSMs are more like that used by GFS. We hope to have a better solution for this issue in the future. For additional information, see this thread in our GitHub code repository (I believe one of you is the one who was having the discussion with my colleague).
 
I get a solution from ECMWF and it's works for me.

Download file from open data ECMWF Data Store (ECPDS), e.g. 20241009000000-6h-oper-fc.grib2

Download and install ecCodes

Remove CCSDS compression which WPS cannot handle:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple 20241009000000-6h-oper-fc.grib2 20241009000000-6h-oper-fc.grib2_simple

Remove wrongly encoded parameter stl1 and its pair swvl1 because WPS requires both temperature and moisture for each soil level:

grib_copy -w shortName!=st,shortName!=swvl1 20241009000000-6h-oper-fc.grib2_simple 20241009000000-6h-oper-fc.grib2_simple_1

Separate depthBelowLandLayer parameters and skt which WPS / ungrib cannot process in GRIB2 format to separate files:

grib_copy -w typeOfLevel=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer
grib_copy -w typeOfLevel!=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_other
grib_copy -w shortName=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_skt
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other_1
The last code give me an error son I change it for this:
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_other_1

skt and depthBelowLandLayer parameters need to be converted to GRIB1 which ungrib can recognise

grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer 20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_skt 20241009000000-6h-oper-fc.grib1_skt
After this, you need to have three GRIB files which will be used as input for ungrib:

20241009000000-6h-oper-fc.grib2_other_1
20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
20241009000000-6h-oper-fc.grib1_skt

The same has to be done for other GRIB files (for other forecasting steps).

ungrib cannot process mix of GRIB1 and GRIB2 in one go, consequently it has to be run twice. Run ungrib first for GRIB1 files, and the second time for GRIB2 file. Between two runs, edit namelist.wps e.g., for the first run you can use:

&ungrib
out_format = 'WPS',
prefix = 'SFILE',

for the second run against GRIB2 file, you can use:

&ungrib
out_format = 'WPS',
prefix = 'OFILE',

after the second run of ungrib, you can run georgid and metgrid. Metgrid section of namelist.wps should look like this:

&metgrid
fg_name = 'SFILE' 'OFILE'
...

to include both files produced in two ungrib runs.

After that, one can normally run real.exe and wrf.exe.

Note: I get an error in wrf.exe, son I add the next in the namelist.input
  • input_from_file = .true.,
  • force_use_old_data = .true.,
 
jmcabrera said:
I get a solution from ECMWF and it's works for me.

Download file from open data ECMWF Data Store (ECPDS), e.g. 20241009000000-6h-oper-fc.grib2

Download and install ecCodes

Remove CCSDS compression which WPS cannot handle:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple 20241009000000-6h-oper-fc.grib2 20241009000000-6h-oper-fc.grib2_simple

Remove wrongly encoded parameter stl1 and its pair swvl1 because WPS requires both temperature and moisture for each soil level:

grib_copy -w shortName!=st,shortName!=swvl1 20241009000000-6h-oper-fc.grib2_simple 20241009000000-6h-oper-fc.grib2_simple_1

Separate depthBelowLandLayer parameters and skt which WPS / ungrib cannot process in GRIB2 format to separate files:

grib_copy -w typeOfLevel=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer
grib_copy -w typeOfLevel!=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_other
grib_copy -w shortName=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_skt
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other_1
The last code give me an error son I change it for this:
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_other_1

skt and depthBelowLandLayer parameters need to be converted to GRIB1 which ungrib can recognise

grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer 20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_skt 20241009000000-6h-oper-fc.grib1_skt
After this, you need to have three GRIB files which will be used as input for ungrib:

20241009000000-6h-oper-fc.grib2_other_1
20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
20241009000000-6h-oper-fc.grib1_skt

The same has to be done for other GRIB files (for other forecasting steps).

ungrib cannot process mix of GRIB1 and GRIB2 in one go, consequently it has to be run twice. Run ungrib first for GRIB1 files, and the second time for GRIB2 file. Between two runs, edit namelist.wps e.g., for the first run you can use:

&ungrib
out_format = 'WPS',
prefix = 'SFILE',

for the second run against GRIB2 file, you can use:

&ungrib
out_format = 'WPS',
prefix = 'OFILE',

after the second run of ungrib, you can run georgid and metgrid. Metgrid section of namelist.wps should look like this:

&metgrid
fg_name = 'SFILE' 'OFILE'
...

to include both files produced in two ungrib runs.

After that, one can normally run real.exe and wrf.exe.

Note: I get an error in wrf.exe, son I add the next in the namelist.input
  • input_from_file = .true.,
  • force_use_old_data = .true.,
Click to expand...
Hi, how did you get the solution from ECMWF, send an email?QAQ
 
jmcabrera said:
I get a solution from ECMWF and it's works for me.

Download file from open data ECMWF Data Store (ECPDS), e.g. 20241009000000-6h-oper-fc.grib2

Download and install ecCodes

Remove CCSDS compression which WPS cannot handle:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple 20241009000000-6h-oper-fc.grib2 20241009000000-6h-oper-fc.grib2_simple

Remove wrongly encoded parameter stl1 and its pair swvl1 because WPS requires both temperature and moisture for each soil level:

grib_copy -w shortName!=st,shortName!=swvl1 20241009000000-6h-oper-fc.grib2_simple 20241009000000-6h-oper-fc.grib2_simple_1

Separate depthBelowLandLayer parameters and skt which WPS / ungrib cannot process in GRIB2 format to separate files:

grib_copy -w typeOfLevel=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer
grib_copy -w typeOfLevel!=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_other
grib_copy -w shortName=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_skt
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other_1
The last code give me an error son I change it for this:
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_other_1

skt and depthBelowLandLayer parameters need to be converted to GRIB1 which ungrib can recognise

grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer 20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_skt 20241009000000-6h-oper-fc.grib1_skt
After this, you need to have three GRIB files which will be used as input for ungrib:

20241009000000-6h-oper-fc.grib2_other_1
20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
20241009000000-6h-oper-fc.grib1_skt

The same has to be done for other GRIB files (for other forecasting steps).

ungrib cannot process mix of GRIB1 and GRIB2 in one go, consequently it has to be run twice. Run ungrib first for GRIB1 files, and the second time for GRIB2 file. Between two runs, edit namelist.wps e.g., for the first run you can use:

&ungrib
out_format = 'WPS',
prefix = 'SFILE',

for the second run against GRIB2 file, you can use:

&ungrib
out_format = 'WPS',
prefix = 'OFILE',

after the second run of ungrib, you can run georgid and metgrid. Metgrid section of namelist.wps should look like this:

&metgrid
fg_name = 'SFILE' 'OFILE'
...

to include both files produced in two ungrib runs.

After that, one can normally run real.exe and wrf.exe.

Note: I get an error in wrf.exe, son I add the next in the namelist.input
  • input_from_file = .true.,
  • force_use_old_data = .true.,
Click to expand...
useful! so similar to the copernicus use of ECMWF and ERA5 post i made
 
jmcabrera said:
I get a solution from ECMWF and it's works for me.

Download file from open data ECMWF Data Store (ECPDS), e.g. 20241009000000-6h-oper-fc.grib2

Download and install ecCodes

Remove CCSDS compression which WPS cannot handle:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple 20241009000000-6h-oper-fc.grib2 20241009000000-6h-oper-fc.grib2_simple

Remove wrongly encoded parameter stl1 and its pair swvl1 because WPS requires both temperature and moisture for each soil level:

grib_copy -w shortName!=st,shortName!=swvl1 20241009000000-6h-oper-fc.grib2_simple 20241009000000-6h-oper-fc.grib2_simple_1

Separate depthBelowLandLayer parameters and skt which WPS / ungrib cannot process in GRIB2 format to separate files:

grib_copy -w typeOfLevel=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer
grib_copy -w typeOfLevel!=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_other
grib_copy -w shortName=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_skt
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other_1
The last code give me an error son I change it for this:
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_other_1

skt and depthBelowLandLayer parameters need to be converted to GRIB1 which ungrib can recognise

grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer 20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_skt 20241009000000-6h-oper-fc.grib1_skt
After this, you need to have three GRIB files which will be used as input for ungrib:

20241009000000-6h-oper-fc.grib2_other_1
20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
20241009000000-6h-oper-fc.grib1_skt

The same has to be done for other GRIB files (for other forecasting steps).

ungrib cannot process mix of GRIB1 and GRIB2 in one go, consequently it has to be run twice. Run ungrib first for GRIB1 files, and the second time for GRIB2 file. Between two runs, edit namelist.wps e.g., for the first run you can use:

&ungrib
out_format = 'WPS',
prefix = 'SFILE',

for the second run against GRIB2 file, you can use:

&ungrib
out_format = 'WPS',
prefix = 'OFILE',

after the second run of ungrib, you can run georgid and metgrid. Metgrid section of namelist.wps should look like this:

&metgrid
fg_name = 'SFILE' 'OFILE'
...

to include both files produced in two ungrib runs.

After that, one can normally run real.exe and wrf.exe.

Note: I get an error in wrf.exe, son I add the next in the namelist.input
  • input_from_file = .true.,
  • force_use_old_data = .true.,
Click to expand...
I tried your method, but/ Real.exe did not generate wrfbodyQAQ1730172870462.png
 
jmcabrera said:
I get a solution from ECMWF and it's works for me.

Download file from open data ECMWF Data Store (ECPDS), e.g. 20241009000000-6h-oper-fc.grib2

Download and install ecCodes

Remove CCSDS compression which WPS cannot handle:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple 20241009000000-6h-oper-fc.grib2 20241009000000-6h-oper-fc.grib2_simple

Remove wrongly encoded parameter stl1 and its pair swvl1 because WPS requires both temperature and moisture for each soil level:

grib_copy -w shortName!=st,shortName!=swvl1 20241009000000-6h-oper-fc.grib2_simple 20241009000000-6h-oper-fc.grib2_simple_1

Separate depthBelowLandLayer parameters and skt which WPS / ungrib cannot process in GRIB2 format to separate files:

grib_copy -w typeOfLevel=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer
grib_copy -w typeOfLevel!=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_other
grib_copy -w shortName=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_skt
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other_1
The last code give me an error son I change it for this:
grib_copy -w shortName!=skt 20241009000000-6h-oper-fc.grib2_other 20241009000000-6h-oper-fc.grib2_other_1

skt and depthBelowLandLayer parameters need to be converted to GRIB1 which ungrib can recognise

grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer 20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_skt 20241009000000-6h-oper-fc.grib1_skt
After this, you need to have three GRIB files which will be used as input for ungrib:

20241009000000-6h-oper-fc.grib2_other_1
20241009000000-6h-oper-fc.grib1_depthBelowLandLayer
20241009000000-6h-oper-fc.grib1_skt

The same has to be done for other GRIB files (for other forecasting steps).

ungrib cannot process mix of GRIB1 and GRIB2 in one go, consequently it has to be run twice. Run ungrib first for GRIB1 files, and the second time for GRIB2 file. Between two runs, edit namelist.wps e.g., for the first run you can use:

&ungrib
out_format = 'WPS',
prefix = 'SFILE',

for the second run against GRIB2 file, you can use:

&ungrib
out_format = 'WPS',
prefix = 'OFILE',

after the second run of ungrib, you can run georgid and metgrid. Metgrid section of namelist.wps should look like this:

&metgrid
fg_name = 'SFILE' 'OFILE'
...

to include both files produced in two ungrib runs.

After that, one can normally run real.exe and wrf.exe.

Note: I get an error in wrf.exe, son I add the next in the namelist.input
  • input_from_file = .true.,
  • force_use_old_data = .true.,
Click to expand...

My friend tried this and he got this error in real.exe:
Code: 
DYNAMICS OPTION: Eulerian Mass Coordinate
   alloc_space_field: domain            1 ,            2041759704  bytes allocated
d01 2024-11-16_00:00:00  Yes, this special data is acceptable to use: OUTPUT FROM METGRID V4.6.0
d01 2024-11-16_00:00:00  Input data is acceptable to use: met_em.d01.2024-11-16_00:00:00.nc
 metgrid input_wrf.F first_date_input = 2024-11-16_00:00:00
 metgrid input_wrf.F first_date_nml = 2024-11-16_00:00:00
d01 2024-11-16_00:00:00 Timing for input          0 s.
Max map factor in domain 1 =  1.06. Scale the dt in the model accordingly.
Using sfcprs  to compute psfc
d01 2024-11-16_00:00:00 No average surface temperature for use with inland lakes
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    2838
Not enough soil temperature data for Noah LSM scheme.
-------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=1 message=application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
:
system msg for write_line failure : Bad file descriptor

It is also worth noting that when I ran this specific command "grib_copy -w typeOfLevel=depthBelowLandLayer 20241009000000-6h-oper-fc.grib2_simple_1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer"
it did not generate a new file so he added "!" in typeOfLevel. Then when he ran this command "grib_set -s edition=1 20241009000000-6h-oper-fc.grib2_depthBelowLandLayer 20241009000000-6h-oper-fc.grib1_depthBelowLandLayer" he gets this error:
Code: 
ECCODES ERROR   :  concept: no match for paramId=260360
ECCODES ERROR   :  concept: input handle edition=1, centre=ecmf
ECCODES ERROR   :  Please check the Parameter Database 'https://apps.ecmwf.int/codes/grib/param-db/?id=260360'
ECCODES ERROR   :  grib_set_values[0] edition (type=long) failed: Concept no match

Any ideas?
 
I proposed a solution a few weeks ago and it works. The problem is related to the new encoding method in grib2, where the results from the IFS model are saved. Unfortunately, this requires changes to the ungrib source code. I have described and explained everything in detail in
forum.mmm.ucar.edu

Change in soil variables from ECMWF Cycle 49r1

On Tuesday (November 12), ECMWF released the 49r1 version of the operational IFS. Despite all the improvements, they modified the soil variables (temperature and humidity) which stopped the ECMWF Vtable that had been released in the 4.6.0 version of the WPS. Based on the new variables made...
forum.mmm.ucar.edu forum.mmm.ucar.edu

Regards
Mariusz Figurski
 
You must log in or register to reply here.
Top