Missing values encountered in interpolated fields when running metgrid with UCAR ERA5 Data

[fcarotenuto]

Dear all,

I'm (desperately) try to run WRF with the ERA5 data available from UCAR through the era5_to_int Python script.
Through extensive research and trial and error I think I've understood the variables needed to run WRF with the --isobaric option of era5_to_int which are the following:

- Dataset "d633000". On pressure levels (mbar): Z/Q/T/U/V
- Dataset "d633000". Surface variables: SOILGEO/MSL/VAR_2T/VAR_2D/VAR_10U/VAR_10V/RSN/SD/SSTK/SKT/SWVL1/SWVL2/SWVL3/SWVL4/STL1/STL2/STL3/STL4/CI
- Dataset "d633006". On model levels: SP
- Invariants: LSM (d633000, 1979/01/01 00:00 - 1979/01/01 00:00), Z (d633006, 2016/01/01 00:00 - 2016/01/01 00:00)

This set of variables allows era5_to_int to run as era5_to_int.py -i -p /home/wrf/Software/Data/ERA5 2023-07-01_00 2023-07-15_23 1 &> era5_to_int.out and produce a complete set of "ERA5:" data (see era5_to_int.out.txt attached).

Still, If I try to run metgrid.exe it complaints about encountering missing values in interpolated fields (PRES, SOILHGT, PSFC, PMSL, GHT) and does not run (see attached metgrid.out.txt and metgrid.log).

The things that I've tried to do up until now were:

1) Increase the domain over which ERA5 data were subsetted, to be sure that my WRF domain was more than fully included. Error is the same.

2) Use model levels data instead than isobaric ones. No matter how I do the request it downloads only the "Q" data and even if the request is from 00:00 of 2023/07/01 the first file is 2023/07/01 06. Thus era5_to_int complains on missing 2023/07/01 00.

Any help would be super appreciated

Thanks!

 

[Ming Chen]

I just run a test case using era5_to_int.py to create intermediate files. All work fine and my case can run through WPS to WRF.

Would you please upload your namelist.wps for me to take a look?

Also, please tell me what files you download from NCAR RDA and your command usage.

Thanks.

 

[fcarotenuto]

Dear Ming Chen,

Thank you for your reply!

In attachment I've put:
1. my namelist.wps
2. the list of files I've downloaded from NCAR RDA
3. the ipynb I use to obtain said files

The command I use is:
era5_to_int.py -i -p /path/to/files 2023-07-01_00 2023-07-15_23 1 &> era5_to_int.out

When I run it for model levels files I remove "-i" from the command.

Thank you again for your support!

 

[Ming Chen]

I just noticed that you didn't download global ERA5 data? Please let me know if I am wrong.

Note that some of the data are on Gaussian grids, and our script can only handle global data.

Please download the entire ERA5 data on global grids and rerun the case. Let me know if you still have issues.

 

[William Oswaldo]

Subject: Issues running era5_to_int.py with ERA5 pressure-level and surface data only — unclear behavior


Dear Ming Chen and team,


Thank you for your support and time.


We are experiencing several issues trying to use the era5_to_int.py script with ERA5 data from NCAR RDA (d633.0). We have not applied any spatial subsetting — the data are global.


We have prepared two sets of data:

1. Pressure-level (pl) variables (37 levels):​

Path: /home/nodo2/Descarga/ERA5/pl/201301/01/


We downloaded the following variables:

• Z (129)
• T (130)
• U (131)
• V (132)
• Q (133)
• R (157)

These are standard 37 pressure levels.

2. Surface and invariant variables:​

Path: /home/nodo2/Descarga/ERA5/sfc/201301/


We downloaded:

2t, 2d, 10u, 10v, sp, msl, skt,
stl1, stl2, stl3, stl4,
swvl2, swvl3, swvl4,
ci, sd,
z (invariant), lsm (invariant)

File examples:

/e5.oper.an.sfc.128_167_2t.ll025sc.2013010100_2013013123.nc
/e5.oper.invariant.128_129_z.ll025sc.1979010100_1979010100.nc
/e5.oper.invariant.128_172_lsm.ll025sc.1979010100_1979010100.nc

We tried different commands, like this command:

python era5_to_int.py --path /home/nodo2/Descarga/ERA5/sfc/201301 \
-v "2t 2d 10u 10v sp msl skt stl1 stl2 stl3 stl4 swvl2 swvl3 swvl4 ci sd z lsm" \
2013-01-01_00 2013-01-01_18
Error: 2t 2d 10u 10v sp msl skt stl1 stl2 stl3 stl4 swvl2 swvl3 swvl4 ci sd z lsm is not a known WPS variable

python era5_to_int.py --path /home/nodo2/Descarga/ERA5/sfc/201301 \
-v "TT DEWPT UU VV PSFC PMSL SKINTEMP ST000007 ST007028 ST028100 ST100289 SM007028 SM028100 SM100289 SEAICE SNOW_EC LANDSEA" \
2013-01-01_00 2013-01-01_18
Missing data file: q from model levels

and the normal situation for the variables using -i:

python era5_to_int.py --path /home/nodo2/Descarga/ERA5/sfc/201301 -i 2013-01-01_00 2013-01-01_18
datetime = 2013-01-01 00:00:00
until_datetime = 2013-01-01 18:00:00
interval_hours = 6:00:00
Processing time record 2013-01-01_00
Error: Could not find file e5.oper.an.pl.128_129_z.ll025sc.2013010100_2013010123.nc needed for ERA5 variable Z


python era5_to_int.py --path /home/nodo2/Descarga/ERA5/pl/201301/01 -i 2013-01-01_00 2013-01-01_18
datetime = 2013-01-01 00:00:00
until_datetime = 2013-01-01 18:00:00
interval_hours = 6:00:00
Processing time record 2013-01-01_00
Traceback (most recent call last):
File "/home/nodo2/Convertir_NC_GRIB_WPS/era5_to_int/era5_to_int.py", line 579, in <module>
idx = find_time_index(e5filename, initdate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nodo2/Convertir_NC_GRIB_WPS/era5_to_int/era5_to_int.py", line 387, in find_time_index
with Dataset(ncfilename) as f:
^^^^^^^^^^^^^^^^^^^
File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: '/home/nodo2/Descarga/ERA5/pl/201301/01/e5.oper.an.pl.128_129_z.ll025sc.2013010100_2013010123.nc'


All we have is error.

This is confusing, because we are not using any model-level (ml) data, and we have already downloaded q at pressure levels (pl).

1. Why does era5_to_int.py expect model-level data when we are working in isobaric mode (-i)? Even though q is present in pl files, the script looks for it in ml.
2. Is there a way to explicitly disable ml dependencies, or a flag to only process pl and/or sfc?
3. Can we use only surface data (e.g. for sea surface temperature, 2m air temp, 10m winds)? The script seems to reject all -v variable names even when using the WPS-compliant internal ones.

Any guidance or clarification would be deeply appreciated. We'd like to use this tool for a case study using only pressure-level and surface data.


Best regards,

William Oswaldo Riaño Acosta