Parallel Netcdf4 I/O support (as opposed to pnetcdf)

Hi,
I am trying to test parallel I/O options in WRF. I have successfully installed WRF with parallel I/O through the pnetcdf library (uncompressed netcdf3 files), by following this guide:
http://www.hpcadvisorycouncil.com/pdf/WRF_v3.8 Installation_Best_Practices.pdf
and also set io_form_history =11 and io_form_restart = 11 to enable the parallel IO in the namelist.input. This works as expected.

BUT, i understand that netcdf (not pnetcdf) also supports parallel I/O using compressed netcdf4 files by using the hdf5 library on the backend.

My question is: how do i properly configure the netcdf/hdf5/wrf to utiliize this parallel I/O backend as opposed to pnetcdf?
What setting do I use for io_form_history and io_form_restart? 2 , 11, something else?

Many thanks.

Hi,
Unfortunately the WRF model is not capable of using parallel I/O. If you are using netCDF-4, you must install it without activating parallel I/O based on HDF5. It is possible to use compression with netcdf-4 and HDF5 (without parallel I/O). See this page for details:
https://www2.mmm.ucar.edu/wrf/users/building_netcdf4.html

Kwerner,
I am a bit confused, i have found many sources stating that WRF can indeed use parallel I/O (including the user guide), and again i have successfully enabled parallel I/O with the pnetcdf library. My question here is about how to enable parallel I/O through the netcdf (not pnetcdf) library.

Sources referring to parallel I/O in WRF:
https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/v4.0/users_guide_chap5.html#RunWRF (search p-netcdf)
https://dl.acm.org/doi/10.1145/3219104.3219120
https://cug.org/proceedings/cug2016_proceedings/includes/files/pap123s2-file1.pdf

From source #2:

WRF provides a rich collection of I/O strategies such as using different parallel I/O libraries (PnetCDF, NetCDF)

Click to expand...

Please let me know what I am missing here.

Many thanks.

To reiterate what Kelly said: The WRF model is not able to support both compression and parallelism from netcdf. For WRF, the HDF5 compression is a file-wide operation, and the parallel aspect would have required the introduction of several separate compressions into a single file. WRF chose to only support compression from netcdf4. In WRF, we do not support "multiple processes writing to a single file" parallelism from netcdf4.

There are a number of I/O options with WRF.

The traditional, serial I/O netcdf option is selected with io_form = 2. If a user has defined the NETCDF env variable to point to a netcdf4 with HDF5 compression, then this option may also be used to do HDF5 compression on that single file.

To allow each of the MPI processes to write out their own file (with 1000 MPI processes, there would be 1000 files written - possibly per time period), the run-time option is io_form = 102. For domains that are medium to large (in excess of 500 horizontal grid cells on a side), the io_form = 102 option is fairly popular. A user is required to stitch the data back together for post-processing (http://www2.mmm.ucar.edu/wrf/users/special_code.html and look for the JOINER tar file).

The WRF model requires the pnetcdf library if a user would like to have multiple processes write to the same file. To extract the full capability of the pnetcdf library, many users utilize the quilting option in WRF, which allows buffered asynchronous I/O to be shunted to a few dedicated WRF I/O processes. This quilting utility is quite fragile.

Side note 1: If a user sees a similar computational performance between MPI-only and MPI+OpenMP parallelism, having 2, 3, or 4 OpenMP threads reduces the files generated with the io_form = 102 option by a factor of 2,3, or 4.

Side note 2: Some of the parallel options have poor performance based on the stream. For example, the boundary file tends to be slower with efforts at parallel I/O.

Side note 3: The restart file is a great candidate for the io_form = 102. If the restart file is input, the smaller pieces of the restart file are reassembled by WRF. A user must always have the same number and orientation of processors to use a restart file created with io_form = 102.

Thank you Dave for the detailed response, it is a big help.

One last question just for final clarification:
So you are saying that the Netcdf4 library (not pnetcdf) can be used solely for serial IO? There is no possiblity to use netcdf4 parallel IO capabilities?

I am so interested in this, becasue there have been some recent advancements in the HDF5 library (backend of netcdf4 library) regarding async IO that look very promising moving forward, and these advancements will not be available for pnetcdf as it is not hdf5 based.
See: https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.5715

If this is indeed the case, and no parallel netcdf4 is supported, I might be interested in looking into implementing this, if it is not currently being worked on.
Please let me know,

Thank you

There is no current work being done with WRF to take advantage of the parallel aspects of netcdf4. The compression makes the output take about 3x longer.

If your only intent is to get data out of the model quickly, we see output speed-ups of 10x on AWS and > 50x on a dedicated HPC when using the io_form_history=102 (vs the vanilla io_form_history = 2). You need to recombine the data, but that runs on a small machine.

Hello Dave,
Quick question here. I was rereading this post and your side note states:

Side note 1: If a user sees a similar computational performance between MPI-only and MPI+OpenMP parallelism, having 2, 3, or 4 OpenMP threads reduces the files generated with the io_form = 102 option by a factor of 2,3, or 4.

Click to expand...

Would this be the right syntax?
For OMP_NUM_THREADS=2 ---> io_form=51
For OMP_NUM_THREADS=3 ---> io_form=34
For OMP_NUM_THREADS=4 ---> io_form=25 (or 26)
For OMP_NUM_THREADS=6 ---> io_form=17
For OMP_NUM_THREADS=8 ---> io_form=12 (or 13)
The tricky part is what happens when it's not an exact multiple: Does it really matter if I choose io_form=25 or 26 for 4 threads? Probably not, but I want to be sure.
Thanks,
Arturo

Arturo,
In Dave's message, io_form refers to the namelist option "io_form_history," which specifies the output format type. Setting it to 102 means the history files are in the format of split netCDF. Take a look at the namelist section of the WRF Users' Guide for additional details.

Hello,
I overthought and misread this post as it doesn't matter whether you're using MPI or MPI+OpenMP, io_form_history and io_form_restart should be set to 102 in order to split the files.