ERA5 surface data not found in WPS

[mfe_et]

Hello,
I have a problem running WPS with ERA5 data. I have downloaded 2 grib files, one for pressure levels (all pressure levels) and the other one for single level data (all variables ecept precipitation type, K index and total totals index). But it seems like metgrid doesn't recognise surface level data, some examples of metgrid.log:
=====
Couldn't find PSFC at level 200100.000000 to fill level 200100.000000 of PRES
Couldn't find SOILT060 at level 200100.000000 to fill level 60.000000 of SOILT
Couldn't find SOILT100 at level 200100.000000 to fill level 100.000000 of SOILT
=====
However it says "Successful completion of metgrid."
But then I run WRF real.exe and there's an error because it needs surface data and is not available in metgrid output's nectcdf files:
=====
Missing surface temp, replaced with closest level, use_surface set to false.
Missing surface RH, replaced with closest level, use_surface set to false.
Missing surface u wind, replaced with closest level, use_surface set to false.
Missing surface v wind, replaced with closest level, use_surface set to false.
ERROR in psfc: flag_psfc = 0, flag_soilhgt = 0, flag_slp = 0, sfcp_to_sfcp = F
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 1403
not enough info for a p sfc computation
-------------------------------------------
=====
I'm using 4.4 version of WRF and Vtable.ECMWF for ungrib.
You can find the namelist.wps and namelis.input files I'm using.
I don't know if the problem is in ungrib step because surface data is not well processed from the grib file, or in metgrid step... but there must be something wrong with my namelist files, Vtable or grib files.
Please any help would be much much much appreciated.
Thank you!

 

[kwerner]

Do you have soil data in your input files? You need at least 2 levels of soil temperature and soil moisture.

 

[mfe_et]

Yes I have, at the single level data grib file there are some soil variables:

I discovered that I must run ungrib twice, one for pressure level data and a second time for single level data.
The problem is that there is an error running ungrib.exe, when I try to process the single level data grib file (I can't attach it because it's too big 43.5MB).
I have tryed using different Vtables: Vtable.ECMWF_sigma, Vtable.ECMWF, ERA-Interim.ml, ERA-Interim.pl
but all of them give the same output, you can find attached the ungrib.log file, just in case the output helps to identify the problem.
As ungrib can't process single level data, metgrid doesn't find surface data and finally real.exe can't run.

 

[Deleted member 3607]

ERA-Interim.ml, ERA-Interim.pl

Those are the vtables for ERA5 that you need to use.

But the log that you are showing says that the input file is tossing this error

ERROR: Data not found: 2022-09-20_09:00:00.0000

Did you download all the ERA5 data as a subset from the RDA database or just the single variable? The vtables most likely need all the data to run ungrib

 

[mfe_et]

I downloaded ERA5 data using CDS API, I did 2 requests (attached file) because there are 2 types of ERA5 data: reanalysis-era5-pressure-levels and reanalysis-era5-single-levls. I can'ts attach those 2 grib files because they are too big:
+ pressure-levels.grib -> 38.7MB
+ single-level.grib -> 43.5MB
but they contain all the variables available in ERA5 (except precipitation type, k index and total totals index) as you can check in the requests.
I have read that ungrib must be run once for pressure level variables using Vtable.ECMWF and then again only for single level data. But when I run it the second time, using only single-level.grib I get the error you mention "ERROR: Data not found: 2022-09-20_09:00:00.0000".
Which Vtable should I use to run ungrib with single-level.grib? or am I doing any other mistake?
If I run ungrib with both grib files at the same run it only precesses pressure level data and then metgrid has not enough info (Couldn't find XXX at level 200100.000000) and then real.exe fails: "not enough info for a p sfc computation"

 

[kwerner]

@mfe_et
Can you use the g1print.exe (or g2print.exe, depending whether your data are grib1 or grib2 format) utility to issue the following. From the WPS directory, issue

./util/g1print.exe <path-to-your-input-data>/<input_data_file> >& g1print.log

Obviously entering your actual path to the data and the actual name of the data file for your surface fields. Then attach the g1print.log file so I can take a look. Thanks!

 

[mfe_et]

Sure! I have attached two files, using g1print.exe with the pressure levels data file and with the single level data file.
Thank you very much!

 

[kwerner]

Thanks for sending those. I notice that the dates in your file are all out of order, which we've found to cause issues. Take a look at this post, which addresses this.

 

[Deleted member 3607]

Thanks for sending those. I notice that the dates in your file are all out of order, which we've found to cause issues. Take a look at this post, which addresses this.

@kwerner @mfe_et

This might be helpful too. Since I am working on ERA5 and finding the same problems.

Initializing the WRF model with ECMWF ERA-Interim reanalysis

Summarization To summarise, you need three separate data files from the reanalysis data: Pressure level data (or model levels) Surface variable data Fixed data.

dreambooker.site

 

[mfe_et]

Thank you so much for your replies!!
At the same blog @Whatheway suggests I found this:
Initializing the WRF model with ERA5 (pressure level)
where ERA5 is used instead of ERA-Interim and pressure levels (same as me). So those are the steps I follow to run WRF.
I tried what @kwerner said to create a grib file with ordered data:
grib_copy -B 'dataDate:i asc, dataTime:i asc, validityDate:i asc, validityTime:i asc, shortName asc' download_sl.grib download_sl_reordered.grib
grib_copy -B 'dataDate:i asc, dataTime:i asc, validityDate:i asc, validityTime:i asc, shortName asc' download_pl.grib download_pl_reordered.grib
Then I run g1print.exe to check the results (you can find the files attached).
But I still have the same problem when I run ungib, metgrid and real.exe, the surface data is not processed. I attach ungrib.log, metgrid.log and the error output of real.exe files for you to check them.
Is the problem related to the Fcst hour of the data??
Example:
4 22 1 0 0 2022-09-22_06:00 + 02
is after
3 23 1 0 0 2022-09-22_06:00 + 03
I tried to order by Fcst hour but I didn't get it, and I don't know how to remove that data and take only Fcst hour = +00 data.

 

[kwerner]

Thanks for sending those, and I'm glad you were able to easily reorder the GRIB files. I believe the issue now is that your surface fields file is still not quite correctly ordered. The initial dates and times are ordered, but the forecast hour (the "+ x" in the far right column) is still out of order.

  1  49            1    0    0  2022-09-22_06:00 + 02
   2 145 BLD        1    0    0  2022-09-22_06:00 + 02
   3  23            1    0    0  2022-09-22_06:00 + 03
   4  22            1    0    0  2022-09-22_06:00 + 02
   5   1            1    0    0  2022-09-22_06:00 + 03
   6 143 CP         1    0    0  2022-09-22_06:00 + 02
   7 218            1    0    0  2022-09-22_06:00 + 03
   8 239 CSF        1    0    0  2022-09-22_06:00 + 02
   9 220            1    0    0  2022-09-22_06:00 + 03
  10  17            1    0    0  2022-09-22_06:00 + 03
  11  24            1    0    0  2022-09-22_06:00 + 03
  12  16            1    0    0  2022-09-22_06:00 + 03
  13  15            1    0    0  2022-09-22_06:00 + 03
  14 182 E          1    0    0  2022-09-22_06:00 + 02
  15  44 ES         1    0    0  2022-09-22_06:00 + 02
  16 180 EWSS       1    0    0  2022-09-22_06:00 + 02
  17  21            1    0    0  2022-09-22_06:00 + 02

 

[mfe_et]

I am sorry for the delay.
I was also thinking that, but I am really not able to order the data correctly, taking into account forecast hour.
I have been searching in different blogs, wikis and users' guides and I can't find how to get it.
I tried typing commands like:
grib_copy -B 'dataDate:i asc, dataTime:i asc, Fcsthour:i asc, validityDate:i asc, validityTime:i asc, shortName asc' download_sl.grib dslr.grib
grib_copy -B 'dataDate:i asc, dataTime:i asc, dataFcsthour:i asc, validityDate:i asc, validityTime:i asc, shortName asc' download_sl.grib dslr.grib
grib_copy -B 'dataDate:i asc, dataTime:i asc, FcstTime:i asc, validityDate:i asc, validityTime:i asc, shortName asc' download_sl.grib dslr.grib
grib_copy -B 'dataDate:i asc, dataTime:i asc, forecastTime:i asc, validityDate:i asc, validityTime:i asc, shortName asc' download_sl.grib dslr.grib
etc.
But nothing works, the grib file is always not correctly ordered when I check it using g1print.exe (example):
----------------------------------------------------
rec GRIB GRIB Lvl Lvl Lvl Time Fcst
Num Code name Code one two hour
----------------------------------------------------
1 49 1 0 0 2022-09-22_06:00 + 06
2 39 1 0 0 2022-09-22_06:00 + 06
3 50 1 0 0 2022-09-22_06:00 + 06
4 40 1 0 0 2022-09-22_06:00 + 06
5 53 1 0 0 2022-09-22_06:00 + 06
6 49 1 0 0 2022-09-22_06:00 + 02
7 33 1 0 0 2022-09-22_06:00 + 06

Can someone help me please??

 

[kwerner]

Unfortunately I'm not sure how to manipulate the data either, but perhaps someone else will see this post and will be able to offer some assistance. Let us know if you figure it out.

 

[Deleted member 3607]

@kwerner @mfe_et

What was the source of the ERA5 data? Was it the RDA database?

 

[mfe_et]

Thank you for your answers.
@Whatheway I downloaded data from cds API, you can find attached a txt file with the queries I did (I can't attach a .py file, but that's exactly the code I run).
This is the page for surface data (the data that I have problems to reorder, and that's why ungrib can't process it): Copernicus Climate Data Store |
And this is for data in pressure levels (the data that ungrib processes correctly): Copernicus Climate Data Store |

 

[Deleted member 3607]

Thank you for your answers.
@Whatheway I downloaded data from cds API, you can find attached a txt file with the queries I did (I can't attach a .py file, but that's exactly the code I run).
This is the page for surface data (the data that I have problems to reorder, and that's why ungrib can't process it): Copernicus Climate Data Store |
And this is for data in pressure levels (the data that ungrib processes correctly): Copernicus Climate Data Store |

@mfe_et @kwerner

Just for trial and error could you please try downloading the ERA5 data from the NCAR RDA database and see if it cannot create met files?

NCAR RDA Dataset d633000

NCAR RDA

rda.ucar.edu

So the steps for using the ERA5 data from NCAR RDA.

Step 1.
link ERA-interim.pl to vtable
link_grib.csh to the pressure files. (make sure the pressure files are in a seperate folder from the surface files)
run ./ungrib.exe

Step 2.
Change wps.namelist
from

&ungrib
out_format = 'WPS',
prefix = 'FILE',
/

to

&ungrib
out_format = 'WPS',
prefix = 'SFILE',
/

then link
cp -f ungrib/Variable_Tables/Vtable.ECMWF ./Vtable
and link the surface files (needs to be in a seperate folder from pressure files) with link_grib.csh
run ungrib.exe

Step 3.
Change wps.namelist again for metgrid to this

&metgrid
fg_name = 'FILE', 'SFILE'
io_form_metgrid = 2,
/

run metgrid.exe

Should work. If not message me and i'll arrange time to help

 

[llsalazardom]

Hi,
I am trying to run the wrf with the ERA5 data but when I run the ungrib it returns this error
Subroutine DATINT: Interpolating 3-d files to fill in any missing data...
Looking for data at time 2022-06-11_00
ERROR: Data not found: 2022-06-11_00:00:00.0000

Do you know what could be the error?

 

[kwerner]

@llsalazardom
Please post your question as a new topic, as your error is different than the one originally posted here. This helps with readability for other users. Thanks!

 

[mfe_et]

Sorry for the long delay @Whatheway, I've been working on something different and I couldn't try what you suggested. Thank you for your help.
I downloaded ERA5 data from NCAR RDA Database and followed the steps you describe. Apparently it works well for ungrib and metgrib but it crashes when I run real.exe. This time the error message is different:
Using sfcprs to compute psfc
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 6241
troubles, could not find trapping x locations
-------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

You can find attached the output files from ungrib, metgrid and real.exe (real.exe output and error files are compressed because they were too big to attach them).
I tried to find that error in other users' questions at the forum and it seems that they corrected it updating WPS version. I am using WPS-4.4 so I don't know what I should do to solve it.