metgrid error (Didn't recognize format version of data in FILE)

[lslrsgis]

Dear WRF community,

I have met a problem when running ./metgrid.exe. It reads as follows:

[****@manager WPSV3.9]$ ./metgrid.exe
Processing domain 1 of 1
Processing 2010-07-14_00
FILE
ERROR: Didn't recognize format version of data in FILE:2010-07-14_00.\nFound version 1919229984 but expected either 3, 4, or 5. This could be an endian problem.

Any indication is appreciated. Thanks.

 

[kwerner]

1) What version of WPS are you using?
2) Can you please attach a copy of your namelist.wps file, your full metgrid.log file, and the FILE:* file the code is complaining about?
Thanks!

 

[lslrsgis]

Thanks for the quick response. Actually, I am running WPS for a test case wrfchem. Exercise 01: https://ruc.noaa.gov/wrf/wrf-chem/tutorialexercises/tutorialexercises001.html

The WPS version I am using is WPS V3.9. The FILE: * files are downloaded (not generated by ungrib) from ftp://aftp.fsl.noaa.gov/divisions/taq/tutorialexercises/exe001/ (link ‘DATA’ on previous https link). In the WPS folder,

(1) I linked chem TBL as follows
[****@manager geogrid]$ ln -svf GEOGRID.TBL.ARW_CHEM GEOGRID.TBL
and adopted the namelist.wps for this specified domain.

(2) run ./geogrid.exe successfully to generate geo_em.d01.nc.

(3) However, when executing ./metgrid.exe, it encountered an error as follows:

[****@manager WPSV3.9]$ ./metgrid.exe
--------------------------------------------------------------------------------
Processing domain 1 of 1
Processing 2010-07-14_00
FILE
ERROR: Didn't recognize format version of data in FILE:2010-07-14_00.\nFound version 1919229984 but expected either 3, 4, or 5. This could be an endian problem.
--------------------------------------------------------------------------------

The namelist.wps and output log files are attached. Thanks in advance.

 

[mgduda]

Could you attach your configure.wps file? One possible cause for the error you're seeing is that the intermediate file was written in little-endian byte order, while the WPS was compiled to read all unformatted files assuming they are in big-endian byte order; or, perhaps the opposite is true (big-endian files with compilation for little-endian reads).

 

[lslrsgis]

Thanks for the quick reply. I have attached the configure.wps.

If it is very possible due to big-endian/little-endian inconsistency, what should I do?

 

[lslrsgis]

BTW, would you look at this post for WPS/NARR: http://forum.mmm.ucar.edu/phpBB3/viewtopic.php?f=39&t=8149? Thanks very much.

 

[mgduda]

Your configure.wps file does contain compiler flags to set unformatted I/O to big-endian byte order and to use 4-byte record markers, both of which are correct:

FFLAGS              = -ffree-form -O -fconvert=big-endian -frecord-marker=4
F77FLAGS            = -ffixed-form -O -fconvert=big-endian -frecord-marker=4

I tried downloading the "FILE:2010-07-14_00" file from ftp://aftp.fsl.noaa.gov/divisions/taq/tutorialexercises/exe001/ , and I was able to successfully use it with metgrid, so I don't think there's any problem with the intermediate files.

Which version of the gfortran compiler are you using (you can check with "gfortran --version")?

 

[lslrsgis]

Thanks for the quick response. I am using gfortran version 7.4.0

***@***-Vostro-3668:~/Downloads$ gfortran --version
GNU Fortran (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 

[mgduda]

It may be worth trying to create your own intermediate file with ungrib.exe, then trying to use that intermediate file with metgrid.exe. If you're using WPS v3.9, you may have problems using current GFS data, but I've uploaded a GFS GRIB2 file from 2016-12-29_00 to https://nextcloud.mmm.ucar.edu/nextcloud/index.php/s/o8nYwN3nHMQr1Ov (look for the file named "gfs_2016122900_0000.grib2"). Could you give this a try to see if the intermediate file you generate works with metgrid?

 

[nepomuzem]

I just had the same problem this morning on the 2020-02-03_t18z cycle.
Ungrib run without problem and then metgrid had the same error:

ERROR: Didn't recognize format version of data in GFS:2020-02-08_18.\nFound version 1079145767 but expected either 3, 4, or 5. This could be an endian problem.

I run WRF operationally every day and never had any problem it failed only tonight, the next cycle run without problem but I would like to know if this is going to happen again.
I use intel compilers and here is what I got on the configure.wps:

DM_FC               = mpiifort -f90=ifort
DM_CC               = mpiicc -cc=icc
FC                  = $(DM_FC)
CC                  = $(DM_CC)
LD                  = $(FC)
FFLAGS              = -O0 -FR -convert big_endian
F77FLAGS            = -O0 -FI -convert big_endian

This is also my ifort version:

ifort (IFORT) 16.0.0 20150815
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.

Any help would be apreciated.
Many thanks.

 

[mgduda]

@nepomuzem That's interesting that it's only a specific GFS time that seems to fail. Is the problem reproducible? I.e., if you download the GFS GRIB file again and re-run ungrib and metgrid, is the failure the same? If so, could you let me know exactly which GFS file you're downloading (a link would be easiest)?

 

[nepomuzem]

@mgduda I downloaded the GFS data from the previously named cycle ( and run all the process again following the same procedure (as once the model has run the run folder is destroyed saving only the logs), I couldn't reproduce the problem.

I run the ungrib once to feed two models, as the ungrib is independent of the topography, doing so I avoid having to run the ungrib two times (with the new GFS resolution the ungrib takes longer and the computational cost is higher). Both model's metgrid.exe failed at the same timestep which makes me think that the problem was in the ungrib.

Nothing showed up on the ungrib.log so it may have been a problem copying the ungrib outputs to it's final destination or downloading a corrupted GFS file (though that is unlikely for we check for consistency on the downloaded GFS). I'll keep an eye on it to see if it happens again. but it looks like it was a one time thing.

Still if you are interested, we download the GFS from the nomads server, instead of using the grib filter we extract the variables we need by downloading the correspondent byte ranges on each files's inventory as that is faster than using the gribfilter. the file in question was from the 2020-02-03T18Z cycle the forecast time that corresponds to the time 2020-02-08T18Z, I guess that corresponds to
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200203/18/gfs.t18z.pgrb2.0p25.f117

thanks for the prompt reply.

 

[kaelel18]

Hi. I'm posting in this thread because I'm experiencing the same problem. I'm trying to write an intermediate file for SST data which I got from Optimum Interpolation Sea Surface Temperature (OISST) at 0.25º × 0.25º resolution. I followed the instruction from this thread https://forum.mmm.ucar.edu/phpBB3/viewtopic.php?f=48&t=8113&p=14108&hilit=SST+update#p14108. I downloaded the script for netcdf to intermediate file and I was able to run the script successfully. Problem is, when I run metgrid, I get

Processing domain 1 of 3
 Processing 2017-12-30_00
    FILE
    SST
ERROR: Didn't recognize format version of data in SST:2017-12-30_00.\nFound version 50331648 but expected either 3, 4, or 5. This could be an endian problem.
application called MPI_Abort(MPI_COMM_WORLD, 22020) - process 0

How do I resolve this? I am attaching relevant files for reference. Thanks

 

[kaelel18]

Hi. I already solved the problem. I placed

convert='big_endian'

in the script and metgrid ran just fine.

 

[ntmanhvn181]

Hi kaelel18,
I have a similar error with you. Can you kindly answer that which script you put the line " " and which line of this script is added?
I check my configure.wps file and the command: convert='big_endian' is included in my file.

FFLAGS              = -FR -convert big_endian
F77FLAGS            = -FI -convert big_endian
FCSUFFIX            =
FNGFLAGS            = $(FFLAGS)
LDFLAGS             =
CFLAGS              = -w
CPP                 = /lib/cpp -P -traditional
CPPFLAGS            = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DIO_BINARY -DIO_GRIB1 -DBIT32
ARFLAGS             =
CC_TOOLS            =

Thank you very much!

 

[mgduda]

@ntmanhvn181 I think you may need to compile the program that was used to create the intermediate file with the "big endian" flag (which will depend on the compiler). Was it the netcdf-to-intermediate.f program or the WPS-ghrsst-to-intermediate that you used (as described in this post)? If it was the WPS-ghrsst-to-intermediate program that was used, could you post the contents of your makefile if it was modified? Essentially, you would need to add something like '-convert big_endian' (for the Intel compilers) to the definition of the FFLAGS variable.

 

[ntmanhvn181]

Hi mgduda,
I'm sorry to reply quite late. My make is attached in the below.
Please help to fix my error.
Thank you very much!
Bests,
Manh,

 

[mgduda]

It looks like the 'make' output attachment might have gotten lost. Could you post it (or attach it at a text file) again? Thanks!

 

[ntmanhvn181]

Hi mgduda,
I've uploaded my makefile in the last post.
I have tried to modify the makefile of WPS-ghrsst-to-intermediate program, however it still failed. Here is my set-up:

FC      = ifort #pgf90
FFLAGS  = -g -convert big_endian
FFLAGS += -tp=istanbul
FFLAGS += -mcmodel=medium
#FFLAGS += -Kieee                  # use exact IEEE math
FFLAGS += -Mbounds                # for bounds checking/debugging
#FFLAGS += -Ktrap=fp               # trap floating point exceptions
FFLAGS += -Bstatic_pgi            # to use static PGI libraries
#FFLAGS += -Bstatic                # to use static netCDF libraries
FFLAGS += -mp=nonuma -nomp        # fix for "can't find libnuma.so"

# NETCDF should be compiled with the same compiler as this program

#NETCDF = /usr/local/src/wrf/LIBS
#HDF5   = /usr/local/src/hdf5-1.8.20.pgi
INCL  = -I$(NETCDF)/include
INCL += -I$(HDF5)/include
LIBS  = -L$(NETCDF)/lib -lnetcdf -lnetcdff
LIBS += -L$(HDF5)/lib -lhdf5_hl -lhdf5 -lm
LIBS += -L/usr/lib64 -lz -ldl

Please help me to address it.
Thank you again.
Bests,
manh,

 

[mgduda]

Assuming the Intel Fortran compiler, I think the attached makefile should work. Note that the environment variable NETCDF will probably need to be set to the path where the NetCDF library has been installed.