The MPAS output file is not recognized

[yycheng]

Hi everyone, I am new to using MPAS.I tried to run a simulation with a 60-3km variable resolution and got a series of output files.

Output streams I wrote: io_type="pnetcdf,cdf5" for large files.

But I can't use the "CDO sinfon" command, or "ncdump" or other way to view the basic information of the data.

cdo sinfon: Open failed on >ea_cptp.init.nc<
Unsupported file type

ncdump: ea_cptp.init.nc: NetCDF: Unknown file format

There was an error that reports "not a valid cdm file"(from panoply)

I also tried the MPAS_convert program, and it reported an error:

 Error: Problems opening input file /home/yycheng/diag.2012-06-20_00.00.00.nc
        This could result from an input file with no unlimited dimension.

However, static files that can be opened via "cdo sinfon" can be converted by mpas_convert.

Thank you!

 

[mgduda]

It may be that your NetCDF library wasn't compiled with support for reading and writing the CDF-5 format, which is provided by the Parallel-NetCDF library. I think the 60-3 km mesh has few enough grid cells that -- assuming around 55 vertical levels -- the CDF-5 large-variable format isn't necessary, and you could either omit the io_type="pnetcdf,cdf5" attribute from your output streams or change the attribute to, e.g., io_type="pnetcdf", which is the default.

As an alternative, if you anticipate working with larger meshes like the 15-3 km mesh, where large-variable support is required, it might be worth recompiling the NetCDF library with support for reading and writing the CDF-5 format, though this would probably require recompiling CDO and other tools to link with the recompiled NetCDF library. The NetCDF4 library does support large variables in files through the use of the HDF5 format, but my experience has been that writing NetCDF4/HDF5 files in parallel generally offers inferior performance to writing CDF-5 files through the Parallel-NetCDF library.

So, the short-term solution may be to just omit io_type="pnetcdf,cdf5" from the definition of your output streams when working with the 60-3 km mesh. I appreciate that the variety of file formats (CDF-2, CDF-5, HDF5) and libraries (NetCDF, Parallel-NetCDF) can complicate matters when running simulations and post-processing results; unfortunately, it's apparently challenging to find a library and format that offers good parallel write performance along with good portability.

 

[mgduda]

I can add that the 'ncmpidump' command that is provided as part of the Parallel-NetCDF library installation should work with your CDF-5 output files. To confirm that your static files are in another format (perhaps CDF-2, which is supported by default in the NetCDF library), you could run, e.g.,

ncmpidump -k <name of output file>
ncmpidump -k <name of static file>

CDF-5 files will be reported as "64-bit data", while CDF-2 files will be reported as "64-bit offset".

 

[yycheng]

It may be that your NetCDF library wasn't compiled with support for reading and writing the CDF-5 format, which is provided by the Parallel-NetCDF library. I think the 60-3 km mesh has few enough grid cells that -- assuming around 55 vertical levels -- the CDF-5 large-variable format isn't necessary, and you could either omit the io_type="pnetcdf,cdf5" attribute from your output streams or change the attribute to, e.g., io_type="pnetcdf", which is the default.

As an alternative, if you anticipate working with larger meshes like the 15-3 km mesh, where large-variable support is required, it might be worth recompiling the NetCDF library with support for reading and writing the CDF-5 format, though this would probably require recompiling CDO and other tools to link with the recompiled NetCDF library. The NetCDF4 library does support large variables in files through the use of the HDF5 format, but my experience has been that writing NetCDF4/HDF5 files in parallel generally offers inferior performance to writing CDF-5 files through the Parallel-NetCDF library.

So, the short-term solution may be to just omit io_type="pnetcdf,cdf5" from the definition of your output streams when working with the 60-3 km mesh. I appreciate that the variety of file formats (CDF-2, CDF-5, HDF5) and libraries (NetCDF, Parallel-NetCDF) can complicate matters when running simulations and post-processing results; unfortunately, it's apparently challenging to find a library and format that offers good parallel write performance along with good portability.

Thank you very much for your explanation of MPAS output!I solved this problem by using the same netCDF library that I used when compiling MPAS

All relevant post-process programs (like cdo) need to be compiled using such a library.