Compiling WRF em_real on Derecho

[jamesrup]

Can anyone comment on what my problem might be when I try to compile em_real on Derecho? I haven't had problems doing this on Cheyenne but am running into issues on Derecho. I'll also note that I checked out configure.wrf in /glade/work/wrfhelp/derecho_pre_compiled_code/wrfv4.5.1 and mine looks almost identical (mine is attached).

My module call:

module purge
module load ncarenv/23.06
module load intel/2023.0.0
module load ncarcompilers/1.0.0
module load craype/2.7.20
module load cray-mpich/8.1.25
module load netcdf/4.9.2

Configure option #50, with no moving nest.

I'm using the latest version of WRF cloned from Github. The current error I'm getting from the bottom of my compile write-out (compile.out.txt, attached) is this:

ftn -o module_cumulus_driver.o -c -ip -O3 -w -ftz -fno-alias -align all -FR -convert big_endian -I../dyn_em -I/glade/work/ruppert/tctor/WRF/external/esmf_time_f90 -I/glade/work/ruppert/tctor/WRF/main -I/glade/work/ruppert/tctor/WRF/external/io_netcdf -I/glade/work/ruppert/tctor/WRF/external/io_int -I/glade/work/ruppert/tctor/WRF/frame -I/glade/work/ruppert/tctor/WRF/share -I/glade/work/ruppert/tctor/WRF/phys -I/glade/work/ruppert/tctor/WRF/wrftladj -I/glade/work/ruppert/tctor/WRF/chem -I/glade/work/ruppert/tctor/WRF/inc -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/oneapi/2023.0.0/iijr/include -real-size `expr 8 \* 4` -i4 module_cumulus_driver.f90
make[3]: *** Deleting file 'module_sf_clm.o'
make[3]: [../configure.wrf:370: module_sf_clm.o] Hangup (ignored)
make[1]: [Makefile:1027: physics] Hangup (ignored)
make[2]: [Makefile:240: physics] Hangup (ignored)
make: *** [Makefile:110: wrf] Hangup

 

[Ming Chen]

Please load the modules below:
module load ncarenv/23.06

module load intel-classic/2023.0.0

module load ncarcompilers/1.0.0

module load cray-mpich/8.1.25

module load craype/2.7.20

module load netcdf-mpi/4.9.2

module load netcdf/4.9.2

Then choose the option 50 after ./configure

Hope this is helpful for you.

 

[jamesrup]

Thanks for these suggestions! It worked, compiled successfully!

Although without the last module you listed: netcdf/4.9.2
That module failed to load, with the below message. I assume that's not a problem, since hdf5-mpi uploads with netcdf-mpi. Do you agree?

Lmod has detected the following error: Cannot load module "hdf5/1.12.2" because these module(s) are
loaded:
hdf5-mpi

While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
hdf5/1.12.2 /glade/u/apps/derecho/modules/23.06/oneapi/2023.0.0/hdf5/1.12.2.lua
netcdf/4.9.2 /glade/u/apps/derecho/modules/23.06/oneapi/2023.0.0/netcdf/4.9.2.lua

 

[Ming Chen]

Thanks for the update. You are right that hdf5-mpi includes netCDF lib.

 

[jamesrup]

Hi Ming Chen/WRF support,

I've just tried to compile the latest WRFv4.6.1 to take advantage of one of the bug fixes (reflectivity output) and it failed to compile em_real, even though it was working with v4.6.0 with the same environmental settings and config option (#50).

I'm using the following, which all load successfully:

module load ncarenv/23.09
module load intel-classic/2023.2.1
module load ncarcompilers/1.0.0
module load cray-mpich/8.1.27
module load craype/2.7.23
module load netcdf-mpi/4.9.2

The compile text output surrounding the first instance of "Error" is shown below:

rm -f ../external/RSL_LITE/interp_domain_em_part3.G ../external/RSL_LITE/interp_domain_em_part3.bb
ftn -o ../external/RSL_LITE/interp_domain_em_part3.o -c -ip -O3 -w -ftz -fno-alias -align all -FR -convert big_endian -I../dyn_em -I/glade/work/ruppert/wrf-piccolo/WRF/external/esmf_time_f90 -I/glade/work/ruppert/wrf-piccolo/WRF/main -I/glade/work/ruppert/wrf-piccolo/WRF/external/io_netcdf -I/glade/work/ruppert/wrf-piccolo/WRF/external/io_int -I/glade/work/ruppert/wrf-piccolo/WRF/frame -I/glade/work/ruppert/wrf-piccolo/WRF/share -I/glade/work/ruppert/wrf-piccolo/WRF/phys -I/glade/work/ruppert/wrf-piccolo/WRF/wrftladj -I/glade/work/ruppert/wrf-piccolo/WRF/chem -I/glade/work/ruppert/wrf-piccolo/WRF/inc -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/include -real-size `expr 8 \* 4` -i4 ../external/RSL_LITE/interp_domain_em_part3.f90
rm -f ../external/RSL_LITE/interp_domain_em_small.o
sed -e "s/^\!.*'.*//" -e "s/^ *\!.*'.*//" ../external/RSL_LITE/interp_domain_em_small.F > ../external/RSL_LITE/interp_domain_em_small.G
ifort: error #10106: Fatal error in /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/bin/intel64/../../bin/intel64/fortcom, terminated by kill signal
compilation aborted for ../external/RSL_LITE/interp_domain_em_part1.f90 (code 1)
make[2]: [../configure.wrf:371: ../external/RSL_LITE/interp_domain_em_part1.o] Error 1 (ignored)
/lib/cpp -P -nostdinc -I/glade/work/ruppert/wrf-piccolo/WRF/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DDA_CORE=0 -DWRFPLUS=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 -DNONSTANDARD_SYSTEM_FUNC -DWRF_USE_CLM -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT -DDM_PARALLEL -DNETCDF -DLANDREAD_STUB=1 -DUSE_ALLOCATABLES -Dwrfmodel -DGRIB1 -DINTIO -DKEEP_INT_AROUND -DLIMIT_ARGS -DBUILD_RRTMG_FAST=0 -DBUILD_RRTMK=0 -DBUILD_SBM_FAST=1 -DSHOW_ALL_VARS_USED=0 -DCONFIG_BUF_LEN=65536 -DMAX_DOMAINS_F=21 -DMAX_HISTORY=25 -DNMM_NEST=0 -I. -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT ../external/RSL_LITE/interp_domain_em_small.G > ../external/RSL_LITE/interp_domain_em_small.bb
/glade/work/ruppert/wrf-piccolo/WRF/tools/standard.exe ../external/RSL_LITE/interp_domain_em_small.bb | /lib/cpp -P -nostdinc -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT > ../external/RSL_LITE/interp_domain_em_small.f90
rm -f ../external/RSL_LITE/interp_domain_em_small.G ../external/RSL_LITE/interp_domain_em_small.bb
ftn -o ../external/RSL_LITE/interp_domain_em_small.o -c -ip -O3 -w -ftz -fno-alias -align all -FR -convert big_endian -I../dyn_em -I/glade/work/ruppert/wrf-piccolo/WRF/external/esmf_time_f90 -I/glade/work/ruppert/wrf-piccolo/WRF/main -I/glade/work/ruppert/wrf-piccolo/WRF/external/io_netcdf -I/glade/work/ruppert/wrf-piccolo/WRF/external/io_int -I/glade/work/ruppert/wrf-piccolo/WRF/frame -I/glade/work/ruppert/wrf-piccolo/WRF/share -I/glade/work/ruppert/wrf-piccolo/WRF/phys -I/glade/work/ruppert/wrf-piccolo/WRF/wrftladj -I/glade/work/ruppert/wrf-piccolo/WRF/chem -I/glade/work/ruppert/wrf-piccolo/WRF/inc -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/include -real-size `expr 8 \* 4` -i4 ../external/RSL_LITE/interp_domain_em_small.f90
ar ru ../main/libwrflib.a module_driver_constants.o module_domain_type.o module_streams.o module_domain.o module_integrate.o module_timing.o module_configure.o module_tiles.o module_machine.o module_nesting.o module_wrf_error.o module_state_description.o module_sm.o module_io.o module_comm_dm.o module_comm_dm_0.o module_comm_dm_1.o module_comm_dm_2.o module_comm_dm_3.o module_comm_dm_4.o module_comm_nesting_dm.o module_dm.o module_quilt_outbuf_ops.o module_io_quilt.o module_cpl.o module_cpl_oasis3.o module_clear_halos.o wrf_num_bytes_between.o wrf_shutdown.o wrf_debug.o libmassv.o collect_on_comm.o hires_timer.o clog.o nl_get_0_routines.o nl_get_1_routines.o nl_get_2_routines.o nl_get_3_routines.o nl_get_4_routines.o nl_get_5_routines.o nl_get_6_routines.o nl_get_7_routines.o nl_set_0_routines.o nl_set_1_routines.o nl_set_2_routines.o nl_set_3_routines.o nl_set_4_routines.o nl_set_5_routines.o nl_set_6_routines.o nl_set_7_routines.o ../inc/allocs_0.o ../inc/allocs_1.o ../inc/allocs_2.o ../inc/allocs_3.o ../inc/allocs_4.o ../inc/allocs_5.o ../inc/allocs_6.o ../inc/allocs_7.o ../inc/allocs_8.o ../inc/allocs_9.o ../inc/allocs_10.o ../inc/allocs_11.o ../inc/allocs_12.o ../inc/allocs_13.o ../inc/allocs_14.o ../inc/allocs_15.o ../inc/allocs_16.o ../inc/allocs_17.o ../inc/allocs_18.o ../inc/allocs_19.o ../inc/allocs_20.o ../inc/allocs_21.o ../inc/allocs_22.o ../inc/allocs_23.o ../inc/allocs_24.o ../inc/allocs_25.o ../inc/allocs_26.o ../inc/allocs_27.o ../inc/allocs_28.o ../inc/allocs_29.o ../inc/allocs_30.o ../inc/allocs_31.o ../inc/deallocs_0.o ../inc/deallocs_1.o ../inc/deallocs_2.o ../inc/deallocs_3.o ../inc/deallocs_4.o ../inc/deallocs_5.o ../inc/deallocs_6.o ../inc/deallocs_7.o ../inc/deallocs_8.o ../inc/deallocs_9.o ../inc/deallocs_10.o ../inc/deallocs_11.o ../external/RSL_LITE/feedback_domain_em_part1.o ../external/RSL_LITE/feedback_domain_em_part2.o ../external/RSL_LITE/force_domain_em_part2.o ../external/RSL_LITE/interp_domain_em_part1.o ../external/RSL_LITE/interp_domain_em_part2.o ../external/RSL_LITE/interp_domain_em_part3.o ../external/RSL_LITE/interp_domain_em_small.o
ar: creating ../main/libwrflib.a
ar: ../external/RSL_LITE/interp_domain_em_part1.o: No such file or directory
make[2]: [Makefile:131: framework] Error 1 (ignored)
ranlib ../main/libwrflib.a
ranlib: '../main/libwrflib.a': No such file
make[2]: [Makefile:132: framework] Error 1 (ignored)

Any ideas?

Cheers,
James

 

[islas]

You may need to add `-j 1` to the compile command. The login sessions on Derecho do not provide a lot of extra memory and compilation with intel compilers often uses quite a bit of memory. The default compilation without specifying any `-j` option is `-j 2` which may be consuming more memory than available.