Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

WRF data file naming structure different depending on options selected.

William.Hatheway

William.Hatheway

Active member
I've recently noticed that when changing the options in the WRF namelist, specifically between using NetCDF (option 2) and PNetCDF (option 11), both the file naming structure and the file sizes change.


For instance, with PNetCDF (option 11), the file name follows the format: wrfout_d01_2025-02-11_00_00_00. However, when switching to NetCDF (option 2), the file name is formatted as wrfout_d01_2025-02-11:00:00:00. This suggests that the PNetCDF option automatically enables a "nocolons" setting, which does not seem to be activated with the NetCDF option.


In addition to the naming structure change, there is a noticeable difference in file size. The NetCDF file is approximately 32.1 MB, whereas the PNetCDF file is around 115.2 MB for the same domain and time. I have confirmed that the data content is identical by using ncdump, as there are no additional variables, yet the file sizes differ significantly.


Here is the relevant portion of the namelist:


PNetCDF options (option 11):

io_form_history = 11
io_form_restart = 11
io_form_input = 11
io_form_boundary = 11


NetCDF options (option 2):

io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2


Given that PNetCDF improves I/O performance but also results in larger file sizes, I'm curious as to why the file structure and size are affected in this manner, especially when the data itself remains the same. Could you provide any insights into this behavior, potentially from NCAR's perspective?
 

Attachments

  • namelist_FFS_melb.input
    5.4 KB · Views: 1
William.Hatheway said:
I've recently noticed that when changing the options in the WRF namelist, specifically between using NetCDF (option 2) and PNetCDF (option 11), both the file naming structure and the file sizes change.


For instance, with PNetCDF (option 11), the file name follows the format: wrfout_d01_2025-02-11_00_00_00. However, when switching to NetCDF (option 2), the file name is formatted as wrfout_d01_2025-02-11:00:00:00. This suggests that the PNetCDF option automatically enables a "nocolons" setting, which does not seem to be activated with the NetCDF option.


In addition to the naming structure change, there is a noticeable difference in file size. The NetCDF file is approximately 32.1 MB, whereas the PNetCDF file is around 115.2 MB for the same domain and time. I have confirmed that the data content is identical by using ncdump, as there are no additional variables, yet the file sizes differ significantly.


Here is the relevant portion of the namelist:


PNetCDF options (option 11):

io_form_history = 11
io_form_restart = 11
io_form_input = 11
io_form_boundary = 11


NetCDF options (option 2):

io_form_history = 2
io_form_restart = 2
io_form_input = 2
io_form_boundary = 2


Given that PNetCDF improves I/O performance but also results in larger file sizes, I'm curious as to why the file structure and size are affected in this manner, especially when the data itself remains the same. Could you provide any insights into this behavior, potentially from NCAR's perspective?
Click to expand...
@islas any ideas?
 
For the difference in file names, it looks like it can be traced back to this commit:
github.com

File name bug fix for PNETCDF and NOCOLONS: · wrf-model/WRF@3f712ae

A change/bug was introduced with -r7980 (2015-01-17) that modified the output filenames for the PNETCDF I/O form and also when the NOCOLONS=T flag was set. This is inconsistent with the other I/O ...
github.com github.com

As for the discrepancy in file size, do you know if you are using HDF5 compression with netCDF? You should see something about it in the configuration, at least for the make build.
 
islas said:
For the difference in file names, it looks like it can be traced back to this commit:
github.com

File name bug fix for PNETCDF and NOCOLONS: · wrf-model/WRF@3f712ae

A change/bug was introduced with -r7980 (2015-01-17) that modified the output filenames for the PNETCDF I/O form and also when the NOCOLONS=T flag was set. This is inconsistent with the other I/O ...
github.com github.com

As for the discrepancy in file size, do you know if you are using HDF5 compression with netCDF? You should see something about it in the configuration, at least for the make build.
Click to expand...
netCDF4 I think but I do have HDF5 installed
 
Then yes, it will be using compression. If you run with NETCDF_classic=1 (or FORCE_NETCDF_CLASSIC=ON for the CMake build) you should see a closer comparison in file size.
 
islas said:
Then yes, it will be using compression. If you run with NETCDF_classic=1 (or FORCE_NETCDF_CLASSIC=ON for the CMake build) you should see a closer comparison in file size.
Click to expand...
Could you explain why if both were using netCDF4 with HDF5 one would be bigger then the other?

Because my wrf build is the same for both cases the only variable was the i/o option being either pnetcdf or netcdf
 
William.Hatheway said:
Because my wrf build is the same for both cases the only variable was the i/o option being either pnetcdf or netcdf
Click to expand...
Both do not necessarily use netCDF4 + HDF5 even if the same build. Changing the IO option changes the library used (netCDF+HDF5 or PnetCDF). The libraries can take very different approaches to writing netCDF file formats, and will ultimately be limited in capabilities by how they were originally compiled. None of these are issues per se, just differences in implementation of netCDF file writing.
 
Hi, I don't want to make another thread but just want to add that since the option of PnetCDF make the file has a different naming convention. When I try to restart, it will not recognize this new naming format (It only accepts the netCDF fomat).

I've encountered problem with writing large restart files with quiliting that PnetCDF needs to be used to write restart files. Any ideas on this or should I make a seperate threads?
 
islas said:
I'm not 100% confident, but it may be that you need to use io_form_restart = 11
Click to expand...
Tam Nguyen said:
Hi, I don't want to make another thread but just want to add that since the option of PnetCDF make the file has a different naming convention. When I try to restart, it will not recognize this new naming format (It only accepts the netCDF fomat).

I've encountered problem with writing large restart files with quiliting that PnetCDF needs to be used to write restart files. Any ideas on this or should I make a seperate threads?
Click to expand...
I use pnetcdf exclusively and all the io options need to be set to = 11 for those files to be used
 
Hi, just an update. Thank you for all your reply. Maybe it's a mistake that I did not use the pnetcdf exclusively since the boundary file from real.exe was with option 2.

But still, I'm quite curious how you don't get the error when wrf.exe read the restart file (which was produced with option 11 so have "underscore" instead of "colons").

However, another thing I encountered is a warning on "BAD MEMORY ORDER" later, then the run can die quite early after that.

Here the little part that I've add in the /external/io_pnetcdf/wrf_io.F90.

subroutine ext_pnc_open_for_read_begin( RawFileName, Comm, IOComm, SysDepInfo, DataHandle, Status)
use wrf_data_pnc
use ext_pnc_support_routines
implicit none
include 'wrf_status_codes.h'
# include "pnetcdf.inc"
character*(*) ,intent(IN) :: RawFileName # Changed this line
integer ,intent(IN) :: Comm
integer ,intent(IN) :: IOComm
character*(*) ,intent(in) :: SysDepInfo
integer ,intent(out) :: DataHandle
integer ,intent(out) :: Status
type(wrf_data_handle) ,pointer :: DH
integer :: XType
integer :: stat
integer ,allocatable :: Buffer:))
integer :: VarID
integer :: StoredDim
integer :: NAtts
integer :: DimIDs(2)
integer(KIND=MPI_OFFSET_KIND) :: VStart(2)
integer(KIND=MPI_OFFSET_KIND) :: VLen(2)
integer :: TotalNumVars
integer :: NumVars
integer :: i
character (NF_MAX_NAME) :: Name
character(len=len(RawFileName)) :: FileName # Changed this line

! ADDED ! Clean filename - replace colons with underscores
write(FileName, fmt="(a)") TRIM(ADJUSTL(RawFileName))
do i = 1, len_trim(FileName)
if(FileName(i:i) == ':') FileName(i:i) = '_'
enddo
Click to expand...
 
Last edited:
Tam Nguyen said:
Hi, just an update. Thank you for all your reply. Maybe it's a mistake that I did not use the pnetcdf exclusively since the boundary file from real.exe was with option 2.

But still, I'm quite curious how you don't get the error when wrf.exe read the restart file (which was produced with option 11 so have "underscore" instead of "colons").

However, another thing I encountered is a warning on "BAD MEMORY ORDER" later, then the run can die quite early after that.

Here the little part that I've add in the /external/io_pnetcdf/wrf_io.F90.
Click to expand...
Code: 
 io_form_history                     = 11,                                        ! Format for writing history files (11: pnetCDF)
 io_form_restart                     = 11,                                        ! Format for writing restart files (11: pnetCDF)
 io_form_input                       = 11,                                         ! Format for reading input files (11: pnetCDF)
 io_form_boundary                    = 11,
 
You must log in or register to reply here.
Top