Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

(RESOLVED) Large grib files

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

adaconnnolly

New member
I am trying to use data from a meteorological model that contains many model levels. Because of this, it seems the grib files are too large to be ingested by WPS. When I run ungrib.exe, I get the successful completion message but I can tell from the log that only some of the model levels have been read. This later causes a dimension disagreement and an error when running metgrid.exe.

It seems that ungrib.exe will read no more than 258 grib messages. Similarly, when I run g2print.exe, only 258 grib message are displayed but I know the grib file contains more than twice as many messages (confirmed by running ncl_filedump on the grib files). I could divide the input met data into more grib files. I was hoping there is an easier fix- perhaps some magic number that caps the number of grib message that WPS programs will read?
 
Hi,

Can you let me know what type of input data and what version of WPS you are using? Thanks!
 
The input is the ECMWF, version of WPS is 3.9.1.

Thanks for looking into it. I'll provide some more information on what I've done so far.

I have the surface data and model level data in separate files, in order to have pure grib1 file for the surface fields and pure grib2 file for the atmospheric data. I linked a Vtable with only surface fields and ran ungrib on the surface data, choosing prefix FILE_SFC, and that seems to have worked fine. I linked a Vtable with only atmospheric fields and ran ungrib on the atmospheric data, choosing prefix FILE_ML, and though I do not get an error message, I can tell from the log that only some of the grib messages have been read for U, V, temperature and specific humidity.
Initially, I did not realize this so I proceeded to use calc_ecmwf_p, having constructed an appropriate ecmwf_coeffs file in the same directory. This seemed to have worked, and the prefix PRES files were present afterwards. If I then run metgrid after adding all prefixes, FILE_ML, FILE_SFC, and PRES, to the metgrid section in the namelist,wps, I will get an error from size disagreement in line 2588 of WRF/external/io_netcdf/wrf_io.F90 from a 3.9.1.1 version of WRF. At that line of code, the disagreement comes from Length(j) being equal to the fewer number of model levels that were actually read by ungrib. At the same time, DH%DimLengths(i) is the correct number of levels (# of model levels + 1 for the surface), which I presume comes from the PRES files since all the model levels are represented in the ecmwf_coeffs table. Whatever the case, there does seem to be a legitimate size disagreement that arises from something going wrong during ungrib. Since no error is thrown at that time, I am not having much success identifying the root of the problem.
 
Hi,

Can you send me the following files so that I can take a look at this?
- namelist.wps
- namelist.input
- one of your ECMWF files
- all the Vtables you are using
- your ecmwf_coeffs file

Thanks!
 
The forum is not allowing me to post files other than the namelist.wps with a warning that the file extension is illegal. I can attach them to a private message but not send them to you. The grib file is just over 4 GB anyhow, so how would you like to transfer a file that large?

I was able to successfully run ungrib and metgrid if I split the grib file into three parts, so I'll likely just pursue that solution
 
Hi,

Unfortunately this forum makes us declare every type of extension we will allow. Besides *.input, can you let me know the other extensions so that I can add them for future users?

If you are satisfied with the method you are using, then that is great; however, if you choose to pursue this more, please see the forum homepage (http://forum.mmm.ucar.edu/phpBB3/). The introduction in the top describes how to attach larger files.
 
I'm interested in a better solution to this problem for the future.

With your tip I did find a work-around by appending '.txt' to the end of each file. That might be a good protocol for the future. In my case, given the need to submit a large file, I just included all the text files in a single .tar with the grib file. Thank you for directing me to those instructions, this is my first time posting to this forum.
 
Hi,

With some investigation, we found that it seems that the NCEP g2 library we use to read the grib files is likely incapable of reading files that are larger than 2GB, which probably explains why your work-around to split the files allowed you to run. Unfortunately we don't have any other solution for this at this time.
 
Top