WRF data file naming structure different depending on options selected.

[William.Hatheway]

I've recently noticed that when changing the options in the WRF namelist, specifically between using NetCDF (option 2) and PNetCDF (option 11), both the file naming structure and the file sizes change.


For instance, with PNetCDF (option 11), the file name follows the format: wrfout_d01_2025-02-11_00_00_00. However, when switching to NetCDF (option 2), the file name is formatted as wrfout_d01_2025-02-11:00:00:00. This suggests that the PNetCDF option automatically enables a "nocolons" setting, which does not seem to be activated with the NetCDF option.


In addition to the naming structure change, there is a noticeable difference in file size. The NetCDF file is approximately 32.1 MB, whereas the PNetCDF file is around 115.2 MB for the same domain and time. I have confirmed that the data content is identical by using ncdump, as there are no additional variables, yet the file sizes differ significantly.


Here is the relevant portion of the namelist:


PNetCDF options (option 11):

io_form_history = 11
io_form_restart = 11
io_form_input = 11
io_form_boundary = 11


NetCDF options (option 2):

io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2


Given that PNetCDF improves I/O performance but also results in larger file sizes, I'm curious as to why the file structure and size are affected in this manner, especially when the data itself remains the same. Could you provide any insights into this behavior, potentially from NCAR's perspective?

 

[William.Hatheway]

I've recently noticed that when changing the options in the WRF namelist, specifically between using NetCDF (option 2) and PNetCDF (option 11), both the file naming structure and the file sizes change.


For instance, with PNetCDF (option 11), the file name follows the format: wrfout_d01_2025-02-11_00_00_00. However, when switching to NetCDF (option 2), the file name is formatted as wrfout_d01_2025-02-11:00:00:00. This suggests that the PNetCDF option automatically enables a "nocolons" setting, which does not seem to be activated with the NetCDF option.


In addition to the naming structure change, there is a noticeable difference in file size. The NetCDF file is approximately 32.1 MB, whereas the PNetCDF file is around 115.2 MB for the same domain and time. I have confirmed that the data content is identical by using ncdump, as there are no additional variables, yet the file sizes differ significantly.


Here is the relevant portion of the namelist:


PNetCDF options (option 11):

io_form_history = 11
io_form_restart = 11
io_form_input = 11
io_form_boundary = 11


NetCDF options (option 2):

io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2


Given that PNetCDF improves I/O performance but also results in larger file sizes, I'm curious as to why the file structure and size are affected in this manner, especially when the data itself remains the same. Could you provide any insights into this behavior, potentially from NCAR's perspective?

@islas any ideas?

 

[islas]

For the difference in file names, it looks like it can be traced back to this commit:

File name bug fix for PNETCDF and NOCOLONS: · wrf-model/WRF@3f712ae

A change/bug was introduced with -r7980 (2015-01-17) that modified the output filenames for the PNETCDF I/O form and also when the NOCOLONS=T flag was set. This is inconsistent with the other I/O ...

github.com

As for the discrepancy in file size, do you know if you are using HDF5 compression with netCDF? You should see something about it in the configuration, at least for the make build.

 

[William.Hatheway]

For the difference in file names, it looks like it can be traced back to this commit:

File name bug fix for PNETCDF and NOCOLONS: · wrf-model/WRF@3f712ae

A change/bug was introduced with -r7980 (2015-01-17) that modified the output filenames for the PNETCDF I/O form and also when the NOCOLONS=T flag was set. This is inconsistent with the other I/O ...

github.com

As for the discrepancy in file size, do you know if you are using HDF5 compression with netCDF? You should see something about it in the configuration, at least for the make build.

netCDF4 I think but I do have HDF5 installed

 

[islas]

Then yes, it will be using compression. If you run with NETCDF_classic=1 (or FORCE_NETCDF_CLASSIC=ON for the CMake build) you should see a closer comparison in file size.

 

[William.Hatheway]

Then yes, it will be using compression. If you run with NETCDF_classic=1 (or FORCE_NETCDF_CLASSIC=ON for the CMake build) you should see a closer comparison in file size.

Could you explain why if both were using netCDF4 with HDF5 one would be bigger then the other?

Because my wrf build is the same for both cases the only variable was the i/o option being either pnetcdf or netcdf

 

[William.Hatheway]

Could you explain why if both were using netCDF4 with HDF5 one would be bigger then the other?

Because my wrf build is the same for both cases the only variable was the i/o option being either pnetcdf or netcdf

@islas

Or is it a pnetcef issue?

 

[islas]

Because my wrf build is the same for both cases the only variable was the i/o option being either pnetcdf or netcdf

Both do not necessarily use netCDF4 + HDF5 even if the same build. Changing the IO option changes the library used (netCDF+HDF5 or PnetCDF). The libraries can take very different approaches to writing netCDF file formats, and will ultimately be limited in capabilities by how they were originally compiled. None of these are issues per se, just differences in implementation of netCDF file writing.