grid_rotate make error on Derecho

[tomerburg]

I'm attempting to run "make" on grid_rotate on Derecho, and am getting the following output and error:

/glade/u/apps/common/23.04/spack/opt/spack/intel-oneapi-compilers/2023.0.0/compiler/latest/linux/bin/intel64/ifort grid_rotate.f90 -o grid_rotate -O3 -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/include -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/include -L/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/lib -lnetcdff -lnetcdf -lnetcdf -lm

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -lnetcdf

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -lnetcdf

make: *** [Makefile:9: grid_rotate] Error 1

For reference, I have the following modules loaded:

Currently Loaded Modules:

1) ncarenv/23.06 (S) 3) intel/2023.0.0 5) cray-mpich/8.1.25 7) hdf5/1.12.2

2) craype/2.7.20 4) ncarcompilers/1.0.0 6) parallel-netcdf/1.12.3 8) netcdf/4.9.2

 

[Ming Chen]

I just compiled grid_rotate in derecho. Below is what I loaded:

Currently Loaded Modules:

  1) ncarenv/23.06 (S)   3) nco/5.1.4   5) ncview/2.1.8     7) ncarcompilers/1.0.0   9) hdf5/1.12.2

  2) craype/2.7.20       4) ncl/6.6.2   6) intel/2023.0.0   8) cray-mpich/8.1.25    10) netcdf/4.9.2

And I simply issue the command 'make'. The code can be successfully compiled.

Can you try again?

 

[tomerburg]

I tried this and got the exact same error...

/glade/u/apps/common/23.04/spack/opt/spack/intel-oneapi-compilers/2023.0.0/compiler/latest/linux/bin/intel64/ifort grid_rotate.f90 -o grid_rotate -O3 -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/include -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/include -L/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/lib -lnetcdff -lnetcdf -lnetcdf -lm

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -lnetcdf

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -lnetcdf

make: *** [Makefile:9: grid_rotate] Error 1

EDIT: I loaded my modules in the same order as yours, and I still got the same errors. Otherwise my next guess would be perhaps related to my .bashrc file, or LD_LIBRARY_PATH.

1) ncarenv/23.06 (S) 3) nco/5.1.4 5) ncview/2.1.8 7) ncarcompilers/1.0.0 9) hdf5/1.12.2


2) craype/2.7.20 4) ncl/6.6.2 6) intel/2023.0.0 8) cray-mpich/8.1.25 10) netcdf/4.9.2

 

[tomerburg]

If this helps in debugging, I was technically able to successfully compile the code when I manually entered the output I get for "nf-config --flibs" for FCLIBS in the makefile:

-L/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/lib -lnetcdff -lnetcdf -lnetcdf -lm

And removed the "lnetcdf" references:

-L/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/lib -lnetcdff -lm

Even though this compiled, when I tried to run "grid_rotate" I still got the following error:

./grid_rotate: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory

 

[Ming Chen]

Please copy the codes saved at

/glade/derecho/scratch/chenming/grid_rotate_derecho.

 

[mgduda]

I think this may be a problem with the nf-config program in the NetCDF-Fortran 4.9.2 library, and which we use for locating the correct NetCDF libraries in the grid_rotate Makefile. On Derecho with the netcdf/4.9.2 module loaded, running nf-config --flibs gives

-L/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/packages/netcdf-fortran/4.6.0/oneapi/2023.0.0/amlu/lib -lnetcdff -lnetcdf -lnetcdf -lm

while the $NETCDF environment variable suggests that the NetCDF-C library (linked with -lnetcdf) is found in /glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/oneapi/2023.0.0/iijr/lib .

Can you try modifying the definition of FCLIBS in the top-level grid_rotate Makefile so that it looks like this?

FCLIBS = $(shell nf-config --flibs) -L${NETCDF}/lib

 

[mgduda]

Also, after compiling grid_rotate using the modification to FCLIBS mentioned in my previous post, you may get the following error when trying to run it on Derecho:

./grid_rotate: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory

It seems that loading the netcdf/4.9.2 module on Derecho doesn't correctly set the $LD_LIBRARY_PATH variable to allow the netcdff shared library to be found. So you may need to set your $LD_LIBRARY_PATH variable to ${NETCDF}/lib:$LD_LIBRARY_PATH .

Hopefully a new NetCDF library and an update to the netcdf module on Derecho will obviate the need for these manual changes.

 

[tomerburg]

Thank you - that worked perfect for grid_rotate, and as an added bonus for convert_mpas as well!

 

[jpiers]

Wanted to mention that these fixes worked for me too (grid_rotate and convert_mpas)

 

[mgduda]

Just to provide an update on this issue, it should no longer be necessary to edit the Makefile or to make changes to the $LD_LIBRARY_PATH environment variable for both the convert_mpas and grid_rotate tools now that MPAS-Dev/MPAS-Tools PR #556 and mgduda/convert_mpas PR #14 have been merged.