Error while using AMD AOCC/AOCL to build ABACUS

[QuantumMisaka]

Details

While using AMD AOCC/AOCL (5.0.0.1 version, accessed today) to build ABACUS, I encounter error in the last linking step

[100%] Built target lr
[100%] Building CXX object CMakeFiles/abacus.dir/source/main.cpp.o
[100%] Linking CXX executable abacus
ld.lld: error: undefined symbol: container::kernels::set_memory<std::complex<double>, container::DEVICE_GPU>::operator()(std::complex<double>*, std::complex<double> const&, unsigned long const&)>>> referenced by propagator.h:94 (/home/mikoto/tests/test2/abacus-develop/source/module_hamilt_lcao/module_tddft/propagator.h:94)
>>>               source/module_hamilt_lcao/module_tddft/CMakeFiles/tddft.dir/propagator_cn2.cpp.o:(container::Tensor module_tddft::create_identity_matrix<std::complex<double>>(int, container::DeviceType))
>>> did you mean: container::kernels::set_memory<std::complex<double>, container::DEVICE_CPU>::operator()(std::complex<double>*, std::complex<double> const&, unsigned long const&)
>>> defined in: source/module_base/module_container/libcontainer.a(memory_impl.cpp.o)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/abacus.dir/build.make:1198: abacus] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1247: CMakeFiles/abacus.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

So:

• How can we try to solve this problem?
• Does ABACUS support compilation like
• • gcc-MKL
• • AOCC-AOCL
• • gcc-AOCL

Other compilation setting:
gcc/g++/gfortran 11.4
clang/clang++:

AMD clang version 17.0.6 (CLANG: AOCC_5.0.0-Build#1377 2024_09_24)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /data/softwares/AMD/5.0.0-aocl-aocc/aocc/bin

LAPACK: OpenBLAS (by toolchain) or AOCL
ScaLAPACK, FFTW：From AOCL
ELPA: compiled by toolchain

Hardware: AMD EPYC 7b12
OS: Ubuntu 22.04

Notice: flang cannot be used, which will lead to error in build ELPA.

flang: error: unsupported option '--whole-archive'
flang: error: unsupported option '--no-whole-archive'
flang: error: unknown argument: '-soname'
flang: error: no such file or directory: 'libelpa_openmp.so.19'

Have you read FAQ on the online manual http://abacus.deepmodeling.com/en/latest/community/faq.html

• Yes, I have read the FAQ part on online manual.

Task list for Issue attackers (only for developers)

• Understand the problem or question described by the user.
• Check if the issue is a known problem or has been addressed in the documentation.
• Test the issue or problem on a similar system or environment, if possible.
• Identify the root cause or provide clarification on the user's question.
• Provide a step-by-step guide, including any necessary resources, to resolve the issue or answer the question.
• If the issue is related to documentation, update the documentation to prevent future confusion (optional).
• If the issue is related to code, consider implementing a fix or improvement (optional).
• Review and incorporate any relevant feedback from users or developers.
• Ensure the user's issue is resolved or their question is answered and close the ticket.

[QuantumMisaka]

GCC-AOCL installation of ABACUS can be done via toolchain, but I need to do lapack link in toolchain for automatic installation

[QuantumMisaka]

GCC-AOCL installation of ABACUS can be done via toolchain, but I need to do lapack link in toolchain for automatic installation

GCC-AOCL version of ABACUS can be compiled while OpenMPI should be compiled by AOCC-AOCL and keep ELPA be compiled by GCC-OpenBLAS. The SCF effciency of this GCC-AOCL version is slighly lower than GCC-OpenBLAS version of abacus. The buliding script is like:

#!/bin/bash
#SBATCH -J build
#SBATCH -N 1
#SBATCH -n 16
#SBATCH -o install.log
#SBATCH -e install.err
# install ABACUS with libxc and deepks
# JamesMisaka in 2023.08.31

# Build ABACUS by gnu-AOCL
module load openmpi/5.0.6-amd5.0.0 cereal cmake

TOOL=$(pwd)
VER=250302

ABACUS_DIR=../../abacus-$VER
#source ./install/setup
cd $ABACUS_DIR

#PREFIX=.
PREFIX=/data/softwares/abacus/$VER-aocl
BUILD_DIR=build_abacus_aocl

APPDIR=/data/softwares
LIBDIR=/data/libraries

LAPACK=/data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib
SCALAPACK=/data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib
ELPA=$LIBDIR/elpa/2025.01.001-gcc/cpu
FFTW3=/data/softwares/AMD/5.0.0-aocl-aocc/aocl
CEREAL=$LIBDIR/cereal/1.3.2
LIBXC=$LIBDIR/libxc/7.0.0-gcc
LIBNPY=$LIBDIR/libnpy/1.0.1/include
LIBRI=$LIBDIR/libri/0.2.0
LIBCOMM=$LIBDIR/libcomm/0.1.0

cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \
        -DCMAKE_CXX_COMPILER=g++ \
        -DMPI_CXX_COMPILER=mpicxx \
        -DLAPACK_DIR=$LAPACK \
        -DSCALAPACK_DIR=$SCALAPACK \
        -DELPA_DIR=$ELPA \
        -DFFTW3_DIR=$FFTW3 \
        -DCEREAL_INCLUDE_DIR=$CEREAL \
        -DLibxc_DIR=$LIBXC \
        -DENABLE_LCAO=ON \
        -DENABLE_LIBXC=ON \
        -DUSE_OPENMP=ON \
        -DENABLE_ASAN=OFF \
        -DUSE_ELPA=ON \
        -DENABLE_LIBRI=OFF \

cmake --build $BUILD_DIR -j `nproc` | tee build.log
cmake --install $BUILD_DIR | tee install.log

but if change the -DCMAKE_CXX_COMPILER to clang++ in AOCC, error occur like above.

It seems that while using clang++, some package related to GPU-ABACUS need to be deal with in some way. However, the hardware in this machine is only AMD-EPYC-7b12 without any GPU

@caic99 @dzzz2001 kindly ask whether you have some idea ?

[caic99]

error in the last linking step

This largely blames to the difference of grammars which is not a part of ISO C++ between clang and gcc. Generally, it only requires little modification on codes.

Notice: flang cannot be used, which will lead to error in build ELPA.

It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.

The SCF effciency of this GCC-AOCL version is slighly lower

With LCAO or PW basis?

[QuantumMisaka]

The SCF effciency of this GCC-AOCL version is slighly lower

With LCAO or PW basis?

LCAO basis in Fe80C36 surface system, run by MPI16-OMP1 in AMD-epyc-7b12

• gcc-openblas: 40s/scf-step
• gcc-aocl (elpa from gcc-openblas): 41s/scf-step
• Intel-OneAPI: 45s/scf-step (after libfakeintel.so crack)

[caic99]

It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.

Or just try ignoring those args if you believe it's OK to do so. Plz see clangd/clangd#662

[caic99]

LCAO basis

I would suggest you checking the underlying math libs used by elpa to see if functions from AOCL is really in use. Plus, it is a common practice to compile with a standard BLAS implementation and use a vendor library in linking stage / at runtime.

[QuantumMisaka]

GCC-AOCL installation of ABACUS can be done via toolchain, but I need to do lapack link in toolchain for automatic installation

Which is done and will be upload in the existing toolchain PR

[QuantumMisaka]

LCAO basis

I would suggest you checking the underlying math libs used by elpa to see if functions from AOCL is really in use. Plus, it is a common practice to compile with a standard BLAS implementation and use a vendor library in linking stage / at runtime.

@caic99 Thanks for your suggestion, after fully support AOCL in toolchain, I've complied another abacus and find that this gcc-AOCL version of ABACUS have better efficiency in LCAO-genelpa SCF calculation. Here is a ranking list after check-out, showing the time comsumed in each scf step (not include the first and the last scf step in each ion step)

Task: MPI16-OMP1 LCAO-genelpa for Fe5C2(510) [Fe80C36], ABACUS commit 1fa5e3a , Hardware: AMD-EPYC-7b12

• gcc-aocl: 38s/scf-step
• gcc-openblas 40s/scf-step
• aocc-openblas-aocl （38-42）s/scf-step （unstable）
• intel-oneapi 46s/scf-step

And here is the link libs of ABACUS in gcc-aocl

        linux-vdso.so.1 (0x00007ffd813fe000)
        /data/libraries/fakeintel/libfakeintel.so (0x000079512e365000)
        libelpa_openmp.so.19 => /data/libraries/elpa/2025.01.001-amd/cpu/lib/libelpa_openmp.so.19 (0x000079512d400000)
        libfftw3.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3.so.3 (0x000079512cc00000)
        libscalapack.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libscalapack.so (0x000079512c400000)
        libfftw3_omp.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3_omp.so.3 (0x000079512e358000)
        libxc.so.15 => /data/libraries/libxc/7.0.0-gcc/lib/libxc.so.15 (0x000079512b600000)
        libmpi.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi.so.40 (0x000079512b200000)
        libomp.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libomp.so (0x000079512ac00000)
        libblis.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis.so.5 (0x000079512a000000)
        libflame.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libflame.so (0x0000795129000000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000795128c00000)
        libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x000079512e26f000)
        libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000079512e24f000)
        libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x0000795128800000)
        libmpi_mpifh.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi_mpifh.so.40 (0x000079512d78a000)
        libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x0000795128400000)
        libaoclutils.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libaoclutils.so (0x000079512d762000)
        libblis-mt.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis-mt.so.5 (0x0000795127800000)
        libmpi_usempif08.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi_usempif08.so.40 (0x000079512d3c7000)
        libmpi_usempi_ignore_tkr.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi_usempi_ignore_tkr.so.40 (0x000079512d755000)
        libflang.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflang.so (0x0000795127200000)
        libflangrti.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflangrti.so (0x000079512d748000)
        libpgmath.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libpgmath.so (0x0000795126e00000)
        libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x000079512cbb8000)
        librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x000079512e246000)
        libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x000079512e241000)
        libopen-pal.so.80 => /data/softwares/openmpi/5.0.6-amd5/lib/libopen-pal.so.80 (0x000079512caf7000)
        libpmix.so.2 => /data/softwares/openmpi/5.0.6-amd5/lib/libpmix.so.2 (0x0000795126a00000)
        libmunge.so.2 => /usr/lib/x86_64-linux-gnu/libmunge.so.2 (0x000079512d73d000)
        libevent_core-2.1.so.7 => /data/softwares/openmpi/5.0.6-amd5/lib/libevent_core-2.1.so.7 (0x000079512d392000)
        libevent_pthreads-2.1.so.7 => /data/softwares/openmpi/5.0.6-amd5/lib/libevent_pthreads-2.1.so.7 (0x000079512d738000)
        libhwloc.so.15 => /usr/lib/x86_64-linux-gnu/libhwloc.so.15 (0x000079512b5a4000)
        /lib64/ld-linux-x86-64.so.2 (0x000079512e36c000)
        libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x000079512caf2000)
        libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x000079512cac8000)

Looking forward for this issue being solved and running AOCC-AOCL version of ABACUS in AMD hardware

[QuantumMisaka]

It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.

Or just try ignoring those args if you believe it's OK to do so. Plz see clangd/clangd#662

I've done this with add a command in toolchain when in build dir of ELPA

          sed -i ./libtool \
          -e 's/\\$wl-soname //g' \
          -e 's/\\$wl--whole-archive\\$convenience \\$wl--no-whole-archive//g' \
          -e 's/\\$wl\\$soname //g'

Then the error is supressed lol

[QuantumMisaka]

It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.

Or just try ignoring those args if you believe it's OK to do so. Plz see clangd/clangd#662

I've done this with add a command in toolchain when in build dir of ELPA

          sed -i ./libtool \
          -e 's/\\$wl-soname //g' \
          -e 's/\\$wl--whole-archive\\$convenience \\$wl--no-whole-archive//g' \
          -e 's/\\$wl\\$soname //g'

Then the error is supressed lol

ABACUS GCC-AOCL after this operation can be compiled, but have much lower efficiency in 52s/scf-step. It seems that there are some problems in flang-compiled ELPA.

ldd information

(base) mikoto@amd-cpu:/data/softwares/abacus/250302-aocl/bin$ ldd abacus 
        linux-vdso.so.1 (0x00007fff74360000)
        /data/libraries/fakeintel/libfakeintel.so (0x00007fdb8df15000)
        /data/libraries/elpa/2025.01.001-amd/cpu/lib/libelpa_openmp.so (0x00007fdb8ce00000)
        libfftw3.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3.so.3 (0x00007fdb8c600000)
        libscalapack.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libscalapack.so (0x00007fdb8be00000)
        libfftw3_omp.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3_omp.so.3 (0x00007fdb8df08000)
        libxc.so.15 => /data/libraries/libxc/7.0.0-gcc/lib/libxc.so.15 (0x00007fdb8ac00000)
        libmpi.so.40 => /data/softwares/openmpi/5.0.6-amd5.0.0/lib/libmpi.so.40 (0x00007fdb8a800000)
        libomp.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libomp.so (0x00007fdb8a200000)
        libblis.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis.so.5 (0x00007fdb89600000)
        libflame.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libflame.so (0x00007fdb88600000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdb88200000)
        libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007fdb8d319000)
        libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdb8dee6000)
        libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fdb87e00000)
        libmpi_usempif08.so.40 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libmpi_usempif08.so.40 (0x00007fdb8deb3000)
        libmpi_usempi_ignore_tkr.so.40 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libmpi_usempi_ignore_tkr.so.40 (0x00007fdb8dea8000)
        libmpi_mpifh.so.40 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libmpi_mpifh.so.40 (0x00007fdb8d2a8000)
        libopen-pal.so.80 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libopen-pal.so.80 (0x00007fdb8c53f000)
        libpmix.so.2 => /usr/lib/x86_64-linux-gnu/libpmix.so.2 (0x00007fdb8bc43000)
        libmunge.so.2 => /usr/lib/x86_64-linux-gnu/libmunge.so.2 (0x00007fdb8de9f000)
        libevent_core-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007fdb8de6a000)
        libevent_pthreads-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fdb8de63000)
        libhwloc.so.15 => /usr/lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007fdb8d24c000)
        libflang.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflang.so (0x00007fdb87800000)
        libflangrti.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflangrti.so (0x00007fdb8de56000)
        libpgmath.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libpgmath.so (0x00007fdb87400000)
        libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fdb8cdb8000)
        libaoclutils.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libaoclutils.so (0x00007fdb8d224000)
        libblis-mt.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis-mt.so.5 (0x00007fdb86800000)
        librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007fdb8de4f000)
        libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdb8de4a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdb8df1c000)
        libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdb8de45000)
        libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007fdb8ab5e000)        libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x00007fdb8c515000)
        libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fdb8c4eb000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fdb8cd97000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fdb8c4cc000)
        libssh.so.4 => /usr/lib/x86_64-linux-gnu/libssh.so.4 (0x00007fdb8a793000)
        libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fdb8c4b8000)
        libnettle.so.8 => /usr/lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fdb8ab18000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fdb8a5a8000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fdb8a554000)        libldap-2.5.so.0 => /usr/lib/x86_64-linux-gnu/libldap-2.5.so.0 (0x00007fdb8a1a1000)
        liblber-2.5.so.0 => /usr/lib/x86_64-linux-gnu/liblber-2.5.so.0 (0x00007fdb8bc32000)
        libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fdb8a0d2000)
        libbrotlidec.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fdb8bc24000)
        libz.so.1 => /usr/lib/x86_64-linux-gnu/libz.so.1 (0x00007fdb8a538000)
        libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fdb88456000)
        libhogweed.so.6 => /usr/lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fdb8a08a000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fdb8817e000)
        libcrypto.so.3 => /usr/lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fdb86200000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fdb88043000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fdb8a520000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fdb87d35000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fdb8a05b000)
        libcom_err.so.2 => /usr/lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fdb8cd91000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fdb8a512000)        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fdb8a040000)
        libbrotlicommon.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fdb88433000)
        libffi.so.8 => /usr/lib/x86_64-linux-gnu/libffi.so.8 (0x00007fdb8a505000)
        libkeyutils.so.1 => /usr/lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fdb8a039000)
        libresolv.so.2 => /usr/lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fdb895ec000)