metgrid error (Didn't recognize format version of data in FILE)

Topics related to running the metgrid.exe program
Post Reply
lslrsgis
Posts: 123
Joined: Sun Oct 28, 2018 3:35 am

metgrid error (Didn't recognize format version of data in FILE)

Post by lslrsgis » Thu Aug 01, 2019 9:57 am

Dear WRF community,

I have met a problem when running ./metgrid.exe. It reads as follows:

[****@manager WPSV3.9]$ ./metgrid.exe
Processing domain 1 of 1
Processing 2010-07-14_00
FILE
ERROR: Didn't recognize format version of data in FILE:2010-07-14_00.\nFound version 1919229984 but expected either 3, 4, or 5. This could be an endian problem.

Any indication is appreciated. Thanks.

kwerner
Posts: 2070
Joined: Wed Feb 14, 2018 9:21 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by kwerner » Thu Aug 01, 2019 4:46 pm

1) What version of WPS are you using?
2) Can you please attach a copy of your namelist.wps file, your full metgrid.log file, and the FILE:* file the code is complaining about?
Thanks!
NCAR/MMM

lslrsgis
Posts: 123
Joined: Sun Oct 28, 2018 3:35 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by lslrsgis » Fri Aug 02, 2019 1:21 am

Thanks for the quick response. Actually, I am running WPS for a test case wrfchem. Exercise 01: https://ruc.noaa.gov/wrf/wrf-chem/tutor ... es001.html

The WPS version I am using is WPS V3.9. The FILE: * files are downloaded (not generated by ungrib) from ftp://aftp.fsl.noaa.gov/divisions/taq/t ... es/exe001/ (link ‘DATA’ on previous https link). In the WPS folder,

(1) I linked chem TBL as follows
[****@manager geogrid]$ ln -svf GEOGRID.TBL.ARW_CHEM GEOGRID.TBL
and adopted the namelist.wps for this specified domain.

(2) run ./geogrid.exe successfully to generate geo_em.d01.nc.

(3) However, when executing ./metgrid.exe, it encountered an error as follows:

[****@manager WPSV3.9]$ ./metgrid.exe
--------------------------------------------------------------------------------
Processing domain 1 of 1
Processing 2010-07-14_00
FILE
ERROR: Didn't recognize format version of data in FILE:2010-07-14_00.\nFound version 1919229984 but expected either 3, 4, or 5. This could be an endian problem.
--------------------------------------------------------------------------------

The namelist.wps and output log files are attached. Thanks in advance.
Attachments
WRF_LOG_FILES.zip
(4.06 KiB) Downloaded 30 times

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Fri Aug 02, 2019 5:04 am

Could you attach your configure.wps file? One possible cause for the error you're seeing is that the intermediate file was written in little-endian byte order, while the WPS was compiled to read all unformatted files assuming they are in big-endian byte order; or, perhaps the opposite is true (big-endian files with compilation for little-endian reads).
NCAR/MMM

lslrsgis
Posts: 123
Joined: Sun Oct 28, 2018 3:35 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by lslrsgis » Sat Aug 03, 2019 6:32 am

Thanks for the quick reply. I have attached the configure.wps.

If it is very possible due to big-endian/little-endian inconsistency, what should I do?
Attachments
configure.wps.txt
(3.57 KiB) Downloaded 30 times

lslrsgis
Posts: 123
Joined: Sun Oct 28, 2018 3:35 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by lslrsgis » Sat Aug 03, 2019 6:36 am

BTW, would you look at this post for WPS/NARR: viewtopic.php?f=39&t=8149? Thanks very much.

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Sat Aug 03, 2019 5:00 pm

Your configure.wps file does contain compiler flags to set unformatted I/O to big-endian byte order and to use 4-byte record markers, both of which are correct:

Code: Select all

FFLAGS              = -ffree-form -O -fconvert=big-endian -frecord-marker=4
F77FLAGS            = -ffixed-form -O -fconvert=big-endian -frecord-marker=4
I tried downloading the "FILE:2010-07-14_00" file from ftp://aftp.fsl.noaa.gov/divisions/taq/t ... es/exe001/ , and I was able to successfully use it with metgrid, so I don't think there's any problem with the intermediate files.

Which version of the gfortran compiler are you using (you can check with "gfortran --version")?
NCAR/MMM

lslrsgis
Posts: 123
Joined: Sun Oct 28, 2018 3:35 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by lslrsgis » Fri Aug 09, 2019 2:55 am

Thanks for the quick response. I am using gfortran version 7.4.0

***@***-Vostro-3668:~/Downloads$ gfortran --version
GNU Fortran (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Fri Aug 23, 2019 7:45 pm

It may be worth trying to create your own intermediate file with ungrib.exe, then trying to use that intermediate file with metgrid.exe. If you're using WPS v3.9, you may have problems using current GFS data, but I've uploaded a GFS GRIB2 file from 2016-12-29_00 to https://nextcloud.mmm.ucar.edu/nextclou ... N3nHMQr1Ov (look for the file named "gfs_2016122900_0000.grib2"). Could you give this a try to see if the intermediate file you generate works with metgrid?
NCAR/MMM

nepomuzem
Posts: 12
Joined: Wed Dec 12, 2018 1:46 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by nepomuzem » Tue Feb 04, 2020 11:22 am

I just had the same problem this morning on the 2020-02-03_t18z cycle.
Ungrib run without problem and then metgrid had the same error:

Code: Select all

ERROR: Didn't recognize format version of data in GFS:2020-02-08_18.\nFound version 1079145767 but expected either 3, 4, or 5. This could be an endian problem.
I run WRF operationally every day and never had any problem it failed only tonight, the next cycle run without problem but I would like to know if this is going to happen again.
I use intel compilers and here is what I got on the configure.wps:

Code: Select all

DM_FC               = mpiifort -f90=ifort
DM_CC               = mpiicc -cc=icc
FC                  = $(DM_FC)
CC                  = $(DM_CC)
LD                  = $(FC)
FFLAGS              = -O0 -FR -convert big_endian
F77FLAGS            = -O0 -FI -convert big_endian
 
This is also my ifort version:

Code: Select all

ifort (IFORT) 16.0.0 20150815
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.
Any help would be apreciated.
Many thanks.

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Tue Feb 04, 2020 6:09 pm

@nepomuzem That's interesting that it's only a specific GFS time that seems to fail. Is the problem reproducible? I.e., if you download the GFS GRIB file again and re-run ungrib and metgrid, is the failure the same? If so, could you let me know exactly which GFS file you're downloading (a link would be easiest)?
NCAR/MMM

nepomuzem
Posts: 12
Joined: Wed Dec 12, 2018 1:46 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by nepomuzem » Wed Feb 05, 2020 2:37 pm

@mgduda I downloaded the GFS data from the previously named cycle ( and run all the process again following the same procedure (as once the model has run the run folder is destroyed saving only the logs), I couldn't reproduce the problem.

I run the ungrib once to feed two models, as the ungrib is independent of the topography, doing so I avoid having to run the ungrib two times (with the new GFS resolution the ungrib takes longer and the computational cost is higher). Both model's metgrid.exe failed at the same timestep which makes me think that the problem was in the ungrib.

Nothing showed up on the ungrib.log so it may have been a problem copying the ungrib outputs to it's final destination or downloading a corrupted GFS file (though that is unlikely for we check for consistency on the downloaded GFS). I'll keep an eye on it to see if it happens again. but it looks like it was a one time thing.

Still if you are interested, we download the GFS from the nomads server, instead of using the grib filter we extract the variables we need by downloading the correspondent byte ranges on each files's inventory as that is faster than using the gribfilter. the file in question was from the 2020-02-03T18Z cycle the forecast time that corresponds to the time 2020-02-08T18Z, I guess that corresponds to
https://nomads.ncep.noaa.gov/pub/data/n ... .0p25.f117

thanks for the prompt reply.

kaelel18
Posts: 52
Joined: Wed Apr 17, 2019 2:40 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by kaelel18 » Fri Feb 14, 2020 5:37 am

Hi. I'm posting in this thread because I'm experiencing the same problem. I'm trying to write an intermediate file for SST data which I got from Optimum Interpolation Sea Surface Temperature (OISST) at 0.25º × 0.25º resolution. I followed the instruction from this thread https://forum.mmm.ucar.edu/phpBB3/viewt ... ate#p14108. I downloaded the script for netcdf to intermediate file and I was able to run the script successfully. Problem is, when I run metgrid, I get

Code: Select all

Processing domain 1 of 3
 Processing 2017-12-30_00
    FILE
    SST
ERROR: Didn't recognize format version of data in SST:2017-12-30_00.\nFound version 50331648 but expected either 3, 4, or 5. This could be an endian problem.
application called MPI_Abort(MPI_COMM_WORLD, 22020) - process 0
How do I resolve this? I am attaching relevant files for reference. Thanks
Attachments
sst.zip
(1020.76 KiB) Downloaded 25 times

kaelel18
Posts: 52
Joined: Wed Apr 17, 2019 2:40 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by kaelel18 » Fri Feb 14, 2020 6:26 am

Hi. I already solved the problem. I placed

Code: Select all

convert='big_endian' 
in the script and metgrid ran just fine.

ntmanhvn181
Posts: 47
Joined: Thu Jul 04, 2019 4:07 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by ntmanhvn181 » Fri Jan 15, 2021 4:31 am

Hi kaelel18,
I have a similar error with you. Can you kindly answer that which script you put the line " " and which line of this script is added?
I check my configure.wps file and the command: convert='big_endian' is included in my file.

Code: Select all

FFLAGS              = -FR -convert big_endian
F77FLAGS            = -FI -convert big_endian
FCSUFFIX            =
FNGFLAGS            = $(FFLAGS)
LDFLAGS             =
CFLAGS              = -w
CPP                 = /lib/cpp -P -traditional
CPPFLAGS            = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DIO_BINARY -DIO_GRIB1 -DBIT32
ARFLAGS             =
CC_TOOLS            = 
Thank you very much!

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Fri Jan 15, 2021 5:11 pm

@ntmanhvn181 I think you may need to compile the program that was used to create the intermediate file with the "big endian" flag (which will depend on the compiler). Was it the netcdf-to-intermediate.f program or the WPS-ghrsst-to-intermediate that you used (as described in this post)? If it was the WPS-ghrsst-to-intermediate program that was used, could you post the contents of your makefile if it was modified? Essentially, you would need to add something like '-convert big_endian' (for the Intel compilers) to the definition of the FFLAGS variable.
NCAR/MMM

ntmanhvn181
Posts: 47
Joined: Thu Jul 04, 2019 4:07 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by ntmanhvn181 » Mon Jan 25, 2021 3:44 pm

Hi mgduda,
I'm sorry to reply quite late. My make is attached in the below.
Please help to fix my error.
Thank you very much!
Bests,
Manh,
Attachments
makefile.txt
(1.22 KiB) Downloaded 4 times
Last edited by ntmanhvn181 on Tue Jan 26, 2021 2:30 am, edited 2 times in total.

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Mon Jan 25, 2021 6:42 pm

It looks like the 'make' output attachment might have gotten lost. Could you post it (or attach it at a text file) again? Thanks!
NCAR/MMM

ntmanhvn181
Posts: 47
Joined: Thu Jul 04, 2019 4:07 am

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by ntmanhvn181 » Tue Jan 26, 2021 2:32 am

Hi mgduda,
I've uploaded my makefile in the last post.
I have tried to modify the makefile of WPS-ghrsst-to-intermediate program, however it still failed. Here is my set-up:

Code: Select all

FC      = ifort #pgf90
FFLAGS  = -g -convert big_endian
FFLAGS += -tp=istanbul
FFLAGS += -mcmodel=medium
#FFLAGS += -Kieee                  # use exact IEEE math
FFLAGS += -Mbounds                # for bounds checking/debugging
#FFLAGS += -Ktrap=fp               # trap floating point exceptions
FFLAGS += -Bstatic_pgi            # to use static PGI libraries
#FFLAGS += -Bstatic                # to use static netCDF libraries
FFLAGS += -mp=nonuma -nomp        # fix for "can't find libnuma.so"

# NETCDF should be compiled with the same compiler as this program

#NETCDF = /usr/local/src/wrf/LIBS
#HDF5   = /usr/local/src/hdf5-1.8.20.pgi
INCL  = -I$(NETCDF)/include
INCL += -I$(HDF5)/include
LIBS  = -L$(NETCDF)/lib -lnetcdf -lnetcdff
LIBS += -L$(HDF5)/lib -lhdf5_hl -lhdf5 -lm
LIBS += -L/usr/lib64 -lz -ldl
Please help me to address it.
Thank you again.
Bests,
manh,

mgduda
Posts: 422
Joined: Mon Feb 26, 2018 7:35 pm

Re: metgrid error (Didn't recognize format version of data in FILE)

Post by mgduda » Thu Jan 28, 2021 9:08 pm

Assuming the Intel Fortran compiler, I think the attached makefile should work. Note that the environment variable NETCDF will probably need to be set to the path where the NetCDF library has been installed.
Attachments
makefile.txt
(835 Bytes) Downloaded 2 times
NCAR/MMM

Post Reply

Return to “metgrid”