(SOLVED) NetCDF Error when attempting to using PNetCDF Quilting with WRF

Topics specifically related to the wrf.exe program
Post Reply
rfritzen
Posts: 4
Joined: Tue Apr 07, 2020 2:04 pm

(SOLVED) NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by rfritzen » Tue Apr 07, 2020 2:21 pm

Good Morning WRF-Help Forums,

Over the past few months, I have been working on using the WRF model on Argonne's LCRC Cluster but have been running into some issues pertaining to nested domains taking too long to write files. To fix this problem in the past, I have employed the PNetCDF library and file quilting options (namelist io setting 11 and setting the two options in &namelist_quilt). When trying to use these two options on LCRC however, my wrf.exe process aborts as soon as it tries to read in the first file with the following:

Code: Select all

d01 2013-06-19_00:00:00  NetCDF error: NetCDF: Attempt to use feature that was not turned on when netCDF was built.
d01 2013-06-19_00:00:00  NetCDF error in wrf_io.F90, line         984
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    1821
 med_latbound_in: error opening wrfbdy_d01 for reading. IERR =        -1021
-------------------------------------------
This is pointing to wrf_io.F90 in the pnetcdf module.

I built the WRF model using the GNU GCC-7.3.0 and OpenMPI-3.1.3 compilers, the following flags were set for all libraries I compiled:

Code: Select all

export CC=mpicc
export CXX=mpicpc
export F77=mpifort
export FC=mpifort
export MPICXX=mpicpc
export MPIF77=mpifort
export MPICC=mpicc
export MPIFC=mpifort
I compiled the following libraries in the listed order:

Code: Select all

zlib-1.2.11
szip-2.1.1
jpeg-6b
libpng-1.6.37
jasper-1.900.1
hdf5-1.10.6
PnetCDF-1.12.1
netcdf-c-4.7.3
netcdf-fortran-4.5.2
pio-2.5.0
For the WRF model, I enabled GRIB2 output through the Arch/Config.pl script, then I configured using options 34 (GNU (gfortran / gcc), dmpar), and 1 (Basic Nesting), then edited the output configure.wrf file:

Code: Select all

Changed SFC to mpif90
Changed SCC to mpicc
changed CCOMP to mpicc
changed DM_FC to mpif90
changed DM_CC to mpicc

Under ARCHFLAGS added -DPNETCDF_QUILT on the line following -DPNETCDF
I then compiled the model with no errors.

Any pointers on how to address this problem would be greatly appreciated. I've tried to provide as much information to the setup and the problem as possible, but can provide more if needed to help solve this issue.

Thanks!

kwerner
Posts: 1926
Joined: Wed Feb 14, 2018 9:21 pm

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by kwerner » Tue Apr 21, 2020 12:00 am

Hi,
I just want to let you know that we aren't ignoring you. I've sent this inquiry to one of our software engineers, as they will know much more about this. I'm hoping to hear something from them soon. Thank you for your patience.
NCAR/MMM

rfritzen
Posts: 4
Joined: Tue Apr 07, 2020 2:04 pm

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by rfritzen » Fri Apr 24, 2020 8:59 pm

Thanks for the update. I'm still trying a few solutions such as rebuilding libraries, trying different compilers, etc.

If you have any other information, please let me know here or reach out via email.

davegill
Posts: 73
Joined: Mon Apr 23, 2018 9:03 pm

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by davegill » Mon Apr 27, 2020 9:23 pm

The error message that you provide is one that is completely inside of netcdf:

Code: Select all

NetCDF error: NetCDF: Attempt to use feature that was not turned on when netCDF was built.
This could mean that the WRF model is trying to use specific netcdf4 capabilites, and that the pnetcdf is not supporting those (for example, compression) If you turn off pnetcdf (use the "io_form_*=2"), does the model run successfully (but, slowly due to the time to process output)?

Taking a step back from the actual error, let's look at the bigger problem that you cite, long time to write output files, which you are addressing with quilting and pnetcdf.

There are a couple of options to consider.
1. Use a different way to output data. Many users have found that the "io_form_history=102" option to be effective. This uses standard netcdf, but each MPI task writes out its own file. So, 1000 MPI tasks = 1000 output files generated (possibly per time period). The output is very fast since the files sizes are small, and there is no need to drain all of the information to receiving output tasks (but there are a LOT of files generated). There are scripts that will join these split history files. Take a look at the bottom of this web page for information on the joiner program:
https://www2.mmm.ucar.edu/wrf/users/special_code.html

2. The pnetcdf and quilting options are designed to assist the output of the large history files, but with no effect on input. Instead of using pnetcdf for all of the io_form options, maybe just use it for "io_form_history=11" (if you have not already tried this).

3. Reduce the fields that are output. At runtime, you can substantially reduce the output time by simply removing 3d fields that you do not use. Why 3d only? With 50 vertical levels, you would need to remove 50 2d arrays to remove the same amount of data from a history file as a single 3d array. It is simply a matter of effective use of your time.
• Edit the namelist.input file, the time_control namelist record

Code: Select all

&time_control
 iofields_filename = "myoutfields.txt”

Note that this is a MAXDOM namelist variable, and that file name can be named anything you choose.
• Place the fields to remove in the named text file "myoutfields.txt" (located in the same directory as the "namelist.input" file. To remove two pressure arrays (almost redundant with other fields that are available) and the vertical velocity, the file would include this single line:

Code: Select all

-:h:0:W,PB,P
• Where “-” means REMOVE this variable from the output stream, “h” is the output (history) stream, and “0” is the stream number (standard WRF history file)

rfritzen
Posts: 4
Joined: Tue Apr 07, 2020 2:04 pm

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by rfritzen » Wed Apr 29, 2020 3:49 pm

The model does run when io_form is set to 2, however as mentioned slow IO times only allow for about 14 hours of the 24 hour simulation I am trying to run to be completed in my job's 6 hour window.

This error confuses me because I compiled PHDF5 and PNetCDF prior to building NetCDF, and I did use the --enable-parallel flag on NetCDF and the configure did capture the PNetCDF installation I completed.

I can try only setting io_form_history to 11 to see what happens.

In regards to output fields, I am already using the iofields_filename setting for this simulation.

rfritzen
Posts: 4
Joined: Tue Apr 07, 2020 2:04 pm

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by rfritzen » Fri May 15, 2020 8:28 pm

Good Afternoon,

I wanted to provide an update in regards to this issue.

What davegill mentioned in point #2 was indeed the issue, changing the input options back to 2 (Standard NetCDF) while leaving the output options set as PNetCDF (11) allowed the model to successfully run while maintaining the expected PNetCDF I/O increases.

Perhaps for the time being, a new error message / warning can be included in namelist checks for input, restart files set as PNetCDF (11) as not supported by PNetCDF.

Thank you for the assistance on the manner.

Ming Chen
Posts: 1101
Joined: Mon Apr 23, 2018 9:42 pm

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by Ming Chen » Mon May 18, 2020 11:25 pm

Thank you for the update, --- it is a good suggestion to add checks for the io_form of input data. We will take care of this issue.
WRF Help Desk

mlaufer
Posts: 4
Joined: Tue Sep 01, 2020 10:20 am

Re: NetCDF Error when attempting to using PNetCDF Quilting with WRF

Post by mlaufer » Thu Oct 08, 2020 2:17 pm

I too encountered this same error issue when trying to use initially trying to use pnetcdf for the inputs:
io_form_input = 11,
io_form_boundary = 11,

The root cause in my case was that WPS was compiled against a WRF version without pnetcdf, while the actual runs were using a WRF compiled with pnetcdf.

Once i made sure that WPS was compiled against the WRF with pnetcdf support It now works correctly with all io_form* = 11 enabled.

Post Reply

Return to “wrf.exe”