AMD AOCC compiler support for WPS?
- Thread starter leuthold
- Start date
William.Hatheway
Well-known member
I concur with @leuthold that the WRF `configure.defaults` already incorporates support for AMD compilers. However, it appears that the WPS component lacks a dedicated configuration setup for AMD compilers. Given that WPS involves heavy data processing, optimizations tailored for AMD CPUs, aligning its configuration with the AMD compiler infrastructure would be conducive to maximizing performance and compatibility.
AMD Compiler option for WPS · Issue #242 · wrf-model/WPS
https://forum.mmm.ucar.edu/threads/amd-aocc-compiler-support-for-wps.14534/ As stated in the WRF Forum WPS appears to be missing the AOCC compilers in it's configure.defaults options. I think it sh...
github.com
Last edited:
Gabriel Cassol
New member
I have been using WRF operationally with AMD processors for 5 consecutive years, if there is any way to use the AMD compiler to speed up WRF simulation it would be very interesting. Due to the price currently, AMD processors are more worth purchasing than Intel processors, so having an AMD compiler option like there is for Intel would be very important.
William.Hatheway
Well-known member
Code:
########################################################################################################################
#ARCH AMD Linux x86_64, AOCC compilers # serial serial_NO_GRIB2 dmpar dmpar_NO_GRIB2
#
COMPRESSION_LIBS = CONFIGURE_COMP_L
COMPRESSION_INC = CONFIGURE_COMP_I
FDEFS = CONFIGURE_FDEFS
NCARG_LIBS =
NCARG_LIBS2 =
FC = mpif90
SFC = flang
CC = mpicc
SCC = clang
LD = $(FC)
FFLAGS = -Mfreeform -ffree-line-length-none -fopenmp
F77FLAGS = -Mfixed -ffixed-line-length-none -fopenmp
FNGFLAGS = $(FFLAGS)
LDFLAGS = -fopenmp
CFLAGS = -O3 -fopenmp
CPP = /lib/cpp -P -traditional
CPPFLAGS = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DIO_BINARY -DIO_GRIB1 -DBIT32 CONFIGURE_MPI
ARFLAGS =
RANLIB = ranlib
CC_TOOLS = $(SCC)I came up with this with the help of chatgpt and some coding forums. I have no way to test it but perhaps this will get a start for someone.
@Ming Chen @Gabriel Cassol
@leuthold did you ever get it to work on WPS?
William.Hatheway
Well-known member
Can the one I came up with be tested somehow
Almost. geogrid.exe and metgrid.exe compiled, but ungrib did not. I used almost identical configurations as above. AOCC 4.1.
ungrib.exe error.
"
ld.lld: error: duplicate symbol: gbytes_
>>> defined at gbytes.f:1
>>> gbytes.o : (gbytes_) in archive ./ngl/libw3.a
>>> defined at gbytesc.f:15
>>> gbytesc.o : (.text+0x40) in archive ./ngl/libg2_4.a
William.Hatheway
Well-known member
William.Hatheway
Well-known member
William.Hatheway
Well-known member
Where you able to use the AOCC compilers for WRF without having to change anything?
For version 4.1.0, no. The compiler threw an error about "-vectorize-noncontigous-memory-aggressively". It told me to change it to "--vectorize-non-contiguous-memory-aggressively", which allowed the code to compile.
I started running into unusual crashes in MYNN and Thompson schemes. I had to change the optimizations from -Ofast to -O3 and remove -ffast-math. Below is the relevant part of the configure.wrf
# Settings for AMD Linux x86_64, AOCC flang compiler with AOCC clang (dm+sm)
# Supported AMDARCH are znver1, znver2 and znver3 for ZEN1, ZEN2 and ZEN3 respectively
# For optimized AMDFCFLAGS and AMDLDFLAGS, please reach out to toolchainsupport@amd.com
#
DESCRIPTION = AMD ($SFC/$SCC) : AMD ZEN1/ ZEN2/ ZEN3 Architectures
DMPARALLEL = 1
OMPCPP = -D_OPENMP
OMP = -fopenmp
OMPCC = -fopenmp
SFC = flang
SCC = clang
CCOMP = clang
DM_FC = mpif90
DM_CC = mpicc
FC = time $(DM_FC)
CC = $(DM_CC) -DFSEEKO64_OK
LD = $(FC)
RWORDSIZE = $(NATIVE_RWORDSIZE)
AMDARCH = -march=znver3
AMDMATHLIB = -fveclib=AMDLIBM
AMDLDFLAGS = -Wl,-mllvm -Wl,-enable-loop-reversal -Wl,-mllvm -Wl,-enable-gather -Wl,-mllvm -Wl,--vectorize-non-contiguous-memory-aggressively
AMDFCFLAGS =
PROMOTION = #-fdefault-real-8
ARCH_LOCAL = -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM -DRPC_TYPES=2 $(NETCDF4_IO_OPTS)
CFLAGS_LOCAL = -w -c -m64 -O3 $(AMDARCH)
LDFLAGS_LOCAL = -m64 -O3 -Mstack_arrays $(AMDARCH) $(AMDLDFLAGS) $(AMDMATHLIB) -lamdlibm -lomp
CPLUSPLUSLIB =
ESMF_LDFLAG = $(CPLUSPLUSLIB)
FCOPTIM = -O3 $(AMDARCH) -Mbyteswapio -Mstack_arrays -ftree-vectorize -Mbyteswapio -funroll-loops -finline-aggressive -finline-hint-functions $(AMDMATHLIB) $(AMDFCFLAGS)
FCREDUCEDOPT = -O3 $(AMDARCH) -Mstack_arrays -DFCREDUCEDOPT
FCNOOPT = -O0
FCDEBUG = # -g $(FCNOOPT)
FORMAT_FIXED = -Mfixed
FORMAT_FREE = -Mfreeform
FCSUFFIX =
BYTESWAPIO = -Mbyteswapio
FCBASEOPTS_NO_G = -w $(FORMAT_FREE) $(BYTESWAPIO)
FCBASEOPTS = -O3 $(FCBASEOPTS_NO_G) $(FCDEBUG)
MODULE_SRCH_FLAG=
TRADFLAG = -traditional $(NETCDF4_IO_OPTS)
CPP = /lib/cpp -P
AR = llvm-ar
ARFLAGS = ru
M4 = m4
RANLIB = llvm-ranlib
RLFLAGS =
CC_TOOLS = $(SCC)
NETCDFPAR_BUILD = echo SKIPPING
I'd like to install and try out AMD's math libraries, but my time working with AMD compilers and clusters has come to an end as I'm retiring.
Mike
I started running into unusual crashes in MYNN and Thompson schemes. I had to change the optimizations from -Ofast to -O3 and remove -ffast-math. Below is the relevant part of the configure.wrf
# Settings for AMD Linux x86_64, AOCC flang compiler with AOCC clang (dm+sm)
# Supported AMDARCH are znver1, znver2 and znver3 for ZEN1, ZEN2 and ZEN3 respectively
# For optimized AMDFCFLAGS and AMDLDFLAGS, please reach out to toolchainsupport@amd.com
#
DESCRIPTION = AMD ($SFC/$SCC) : AMD ZEN1/ ZEN2/ ZEN3 Architectures
DMPARALLEL = 1
OMPCPP = -D_OPENMP
OMP = -fopenmp
OMPCC = -fopenmp
SFC = flang
SCC = clang
CCOMP = clang
DM_FC = mpif90
DM_CC = mpicc
FC = time $(DM_FC)
CC = $(DM_CC) -DFSEEKO64_OK
LD = $(FC)
RWORDSIZE = $(NATIVE_RWORDSIZE)
AMDARCH = -march=znver3
AMDMATHLIB = -fveclib=AMDLIBM
AMDLDFLAGS = -Wl,-mllvm -Wl,-enable-loop-reversal -Wl,-mllvm -Wl,-enable-gather -Wl,-mllvm -Wl,--vectorize-non-contiguous-memory-aggressively
AMDFCFLAGS =
PROMOTION = #-fdefault-real-8
ARCH_LOCAL = -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM -DRPC_TYPES=2 $(NETCDF4_IO_OPTS)
CFLAGS_LOCAL = -w -c -m64 -O3 $(AMDARCH)
LDFLAGS_LOCAL = -m64 -O3 -Mstack_arrays $(AMDARCH) $(AMDLDFLAGS) $(AMDMATHLIB) -lamdlibm -lomp
CPLUSPLUSLIB =
ESMF_LDFLAG = $(CPLUSPLUSLIB)
FCOPTIM = -O3 $(AMDARCH) -Mbyteswapio -Mstack_arrays -ftree-vectorize -Mbyteswapio -funroll-loops -finline-aggressive -finline-hint-functions $(AMDMATHLIB) $(AMDFCFLAGS)
FCREDUCEDOPT = -O3 $(AMDARCH) -Mstack_arrays -DFCREDUCEDOPT
FCNOOPT = -O0
FCDEBUG = # -g $(FCNOOPT)
FORMAT_FIXED = -Mfixed
FORMAT_FREE = -Mfreeform
FCSUFFIX =
BYTESWAPIO = -Mbyteswapio
FCBASEOPTS_NO_G = -w $(FORMAT_FREE) $(BYTESWAPIO)
FCBASEOPTS = -O3 $(FCBASEOPTS_NO_G) $(FCDEBUG)
MODULE_SRCH_FLAG=
TRADFLAG = -traditional $(NETCDF4_IO_OPTS)
CPP = /lib/cpp -P
AR = llvm-ar
ARFLAGS = ru
M4 = m4
RANLIB = llvm-ranlib
RLFLAGS =
CC_TOOLS = $(SCC)
NETCDFPAR_BUILD = echo SKIPPING
I'd like to install and try out AMD's math libraries, but my time working with AMD compilers and clusters has come to an end as I'm retiring.
Mike
William.Hatheway
Well-known member
Thank you mike, once I get an AMD cpu to test I'll let you know how it goes. Can I reach out to you for help if needed? @leuthold @Gabriel Cassol
Dear all.
I generated all .exe in WPS with this configuration:
#### Architecture specific settings ####
#
# Settings for Linux x86_64, gfortran (dmpar)
#
#
COMPRESSION_LIBS = -L/home/thiago/WRF-SRC/opt/lib -ljasper -lpng -lz
COMPRESSION_INC = -I/home/thiago/WRF-SRC/opt/include
FDEFS = -DUSE_JPEG2000 -DUSE_PNG
# Compiladores AOCC
# Compiladores AOCC
SFC = flang
SCC = clang
DM_FC = mpif90
DM_CC = mpicc
FC = $(DM_FC)
CC = $(DM_CC)
LD = $(FC)
# Arquitetura e libs AMD
AMDARCH = -march=znver3
AMDMATHLIB = -fveclib=AMDLIBM
AMDLDFLAGS = -Wl,-mllvm -Wl,-enable-loop-reversal \
-Wl,-mllvm -Wl,-enable-gather \
-Wl,-mllvm -Wl,--vectorize-non-contiguous-memory-aggressively
# Flags Fortran
FFLAGS = -Ofast -Mfixed -fno-common -ffast-math -w -Mfreeform -Mbyteswapio -Mextend \
$(AMDARCH) $(AMDMATHLIB)
F77FLAGS = -Ofast -Mfixed -ffast-math -march=znver3 -funroll-loops -Mextend \
-ftree-vectorize -fno-common \
-Wno-macro-redefined -Wno-shift-count-overflow
FORMAT_FREE = -Mfreeform
FORMAT_FIXED = -Mfixed
FCCOMPAT =
FCSUFFIX =
FNGFLAGS = $(FFLAGS)
# Flags C
CFLAGS = -w -c -m64 -Ofast -ffast-math $(AMDARCH)
LDFLAGS = -m64 -Ofast $(AMDARCH) \
$(AMDLDFLAGS) $(AMDMATHLIB) -lamdlibm -lomp -Xlinker --allow-multiple-definition
CPP = /bin/cpp -P -C -undef -nostdinc -traditional
CPPFLAGS = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DGEOGRID \
-DBIT32 -DNO_SIGNAL -DMPI -DUSE_JPEG2000 -DUSE_PNG -D__64BIT__
I generated all .exe in WPS with this configuration:
#### Architecture specific settings ####
#
# Settings for Linux x86_64, gfortran (dmpar)
#
#
COMPRESSION_LIBS = -L/home/thiago/WRF-SRC/opt/lib -ljasper -lpng -lz
COMPRESSION_INC = -I/home/thiago/WRF-SRC/opt/include
FDEFS = -DUSE_JPEG2000 -DUSE_PNG
# Compiladores AOCC
# Compiladores AOCC
SFC = flang
SCC = clang
DM_FC = mpif90
DM_CC = mpicc
FC = $(DM_FC)
CC = $(DM_CC)
LD = $(FC)
# Arquitetura e libs AMD
AMDARCH = -march=znver3
AMDMATHLIB = -fveclib=AMDLIBM
AMDLDFLAGS = -Wl,-mllvm -Wl,-enable-loop-reversal \
-Wl,-mllvm -Wl,-enable-gather \
-Wl,-mllvm -Wl,--vectorize-non-contiguous-memory-aggressively
# Flags Fortran
FFLAGS = -Ofast -Mfixed -fno-common -ffast-math -w -Mfreeform -Mbyteswapio -Mextend \
$(AMDARCH) $(AMDMATHLIB)
F77FLAGS = -Ofast -Mfixed -ffast-math -march=znver3 -funroll-loops -Mextend \
-ftree-vectorize -fno-common \
-Wno-macro-redefined -Wno-shift-count-overflow
FORMAT_FREE = -Mfreeform
FORMAT_FIXED = -Mfixed
FCCOMPAT =
FCSUFFIX =
FNGFLAGS = $(FFLAGS)
# Flags C
CFLAGS = -w -c -m64 -Ofast -ffast-math $(AMDARCH)
LDFLAGS = -m64 -Ofast $(AMDARCH) \
$(AMDLDFLAGS) $(AMDMATHLIB) -lamdlibm -lomp -Xlinker --allow-multiple-definition
CPP = /bin/cpp -P -C -undef -nostdinc -traditional
CPPFLAGS = -D_UNDERSCORE -DBYTESWAP -DLINUX -DIO_NETCDF -DGEOGRID \
-DBIT32 -DNO_SIGNAL -DMPI -DUSE_JPEG2000 -DUSE_PNG -D__64BIT__
"Do you have any clear, step-by-step guidance for installing WRF and WPS from scratch using AOCC and OpenMPI?
I'm struggling to compile WPS—I've tried every method I've found on forums or from AI suggestions, but nothing has worked yet. Could you share how you modified the configure.defaults (or configure.wps) file? Any advice would be greatly appreciated!
William.Hatheway
Well-known member
WPS.configure
William.Hatheway
Well-known member
wrf configure
Code:
# Settings for AMD Linux x86_64, AOCC flang compiler with AOCC clang (dm+sm)
# Supported AMDARCH are znver1, znver2 and znver3 for ZEN1, ZEN2 and ZEN3 respectively
# For optimized AMDFCFLAGS and AMDLDFLAGS, please reach out to toolchainsupport@amd.com
#
DESCRIPTION = AMD ($SFC/$SCC) : AMD ZEN1/ ZEN2/ ZEN3 Architectures
DMPARALLEL = 1
OMPCPP = -D_OPENMP
OMP = -fopenmp
OMPCC = -fopenmp
SFC = flang
SCC = clang
CCOMP = clang
DM_FC = mpif90
DM_CC = mpicc
FC = time $(DM_FC)
CC = $(DM_CC) -DFSEEKO64_OK
LD = $(FC)
RWORDSIZE = $(NATIVE_RWORDSIZE)
AMDARCH = -march=znver3
AMDMATHLIB = -fveclib=AMDLIBM
AMDLDFLAGS = -Wl,-mllvm -Wl,-enable-loop-reversal -Wl,-mllvm -Wl,-enable-gather -Wl,-mllvm -Wl,--vectorize-non-contiguous-memory-aggressively
AMDFCFLAGS =
PROMOTION = #-fdefault-real-8
ARCH_LOCAL = -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM -DRPC_TYPES=2 $(NETCDF4_IO_OPTS)
CFLAGS_LOCAL = -w -c -m64 -O3 $(AMDARCH)
LDFLAGS_LOCAL = -m64 -O3 -Mstack_arrays $(AMDARCH) $(AMDLDFLAGS) $(AMDMATHLIB) -lamdlibm -lomp
CPLUSPLUSLIB =
ESMF_LDFLAG = $(CPLUSPLUSLIB)
FCOPTIM = -O3 $(AMDARCH) -Mbyteswapio -Mstack_arrays -ftree-vectorize -Mbyteswapio -funroll-loops -finline-aggressive -finline-hint-functions $(AMDMATHLIB) $(AMDFCFLAGS)
FCREDUCEDOPT = -O3 $(AMDARCH) -Mstack_arrays -DFCREDUCEDOPT
FCNOOPT = -O0
FCDEBUG = # -g $(FCNOOPT)
FORMAT_FIXED = -Mfixed
FORMAT_FREE = -Mfreeform
FCSUFFIX =
BYTESWAPIO = -Mbyteswapio
FCBASEOPTS_NO_G = -w $(FORMAT_FREE) $(BYTESWAPIO)
FCBASEOPTS = -O3 $(FCBASEOPTS_NO_G) $(FCDEBUG)
MODULE_SRCH_FLAG=
TRADFLAG = -traditional $(NETCDF4_IO_OPTS)
CPP = /lib/cpp -P
AR = llvm-ar
ARFLAGS = ru
M4 = m4
RANLIB = llvm-ranlib
RLFLAGS =
CC_TOOLS = $(SCC)
NETCDFPAR_BUILD = echo SKIPPINGWilliam.Hatheway
Well-known member
Hi, I'm dealing with a similar issue where I'm trying to get WPS 4.5.1 to compile on AMD/Zen4 but there is no zen4 option available from ./configure so I'm trying to manually set up the configure.wps file. I'm getting an odd issue where some fields appear to come out scrambled.
See SST's below (met_em file from wps 4.5.1 built with Intel on the left looking very normal, versus what I get from the newly-compiled code on the right):
The configure.wps file is attached here. Note that I picked a gfortran option when running configure and then made modifications to the resulting file.
I have tried lowering the optimization level (with the thought that it is causing issues with precision) from -O3 to -O2 . This version compiled but it generates an error when attempting to read grib files (ERROR: Grib2 file or date problem, stopping in edition_num.) I'm using the same grib files as in prior tests where they were read by ungrib just fine and they are definitely linked to the right directory.
I've also tried modifying the -O3 flag to "-O3 -ffp-model=precise" and got that to compile but the scrambling issue with fields like SSTs remained.
So at this point I'm out of ideas and would really appreciate any tips that could be offered. Thanks!
Megan
See SST's below (met_em file from wps 4.5.1 built with Intel on the left looking very normal, versus what I get from the newly-compiled code on the right):
The configure.wps file is attached here. Note that I picked a gfortran option when running configure and then made modifications to the resulting file.
I have tried lowering the optimization level (with the thought that it is causing issues with precision) from -O3 to -O2 . This version compiled but it generates an error when attempting to read grib files (ERROR: Grib2 file or date problem, stopping in edition_num.) I'm using the same grib files as in prior tests where they were read by ungrib just fine and they are definitely linked to the right directory.
I've also tried modifying the -O3 flag to "-O3 -ffp-model=precise" and got that to compile but the scrambling issue with fields like SSTs remained.
So at this point I'm out of ideas and would really appreciate any tips that could be offered. Thanks!
Megan