TiledArray unit tests fail on nVIDIA V100 GPU

[victor-anisimov]

I have compiled the latest, Feb 15 copy of TiledArray (commit 53af746) on Intel Xeon Gold 6152 CPU equipped with nVidia V100 GPU. Compilation successfully finishes. All tests compile as well. However, the execution of tests stalls.

`1: [100%] Built target ta_test
1/3 Test #1: build_ta_test .................... Passed 1724.30 sec
test 2
Start 2: ta_test-np-1

2: Test command: mpirun "-n" "1" "/home/vanisimov/tiledarray/tiledarray/build/tests/ta_test" "--log_level=warning" "--show-progress" "--run_test=!@distributed"
2: Test timeout computed to be: 10000000
2: created 3 CUDA streams + 2 I/O streams
2: Running 1788 test cases...
2:
2: 0% 10 20 30 40 50 60 70 80 90 100%
2: |----|----|----|----|----|----|----|----|----|----|
2: *************************************************!!MADNESS: Hung queue?
2: !!MADNESS: Hung queue?
2: !!MADNESS: Hung queue?
2: !!MADNESS: Hung queue?
2: !!MADNESS: Hung queue?
2: !! ERROR TiledArray: Aborting due to MADNESS exception.
2: !! ERROR TiledArray: ThreadPool::await() timed out after 900.0 seconds
2: !! ERROR TiledArray: rank=0 id={0,21626} 105 of 125 tiles set
2: unknown location(0): fatal error: in "um_expressions_suite/tensor_factories": signal: SIGABRT (application abort requested)
2: /home/vanisimov/tiledarray/tiledarray/tests/expressions_cuda_um.cpp(144): last checkpoint`

The host compiler is gcc/8.2.0. The version of cuda toolkit is 11.2.0.

For comparison, all unit tests successfully complete if TiledArray is compiled in CPU-only mode by using the same version of gcc/8.2.0. What version of TiledArray code should I use to see CUDA tests passing?

[victor-anisimov]

The TiledArray compilation script. Note that TBB is OFF.

`module use /soft/modulefiles
module load gcc/8.2.0
module load cuda/11.2.0
module load mkl
module load mpi
module load cmake/3.19.3

export PROJECT_DIR=$HOME/tiledarray
export PROJECT_INSTALL=$PROJECT_DIR/install

export CC=gcc
export CXX=g++
export FC=gfortran

export ROOT=$PROJECT_DIR
export INTEL_DIR=$A21_SDK_ROOT
export MKLROOT=${INTEL_DIR}/mkl/latest
export TBBROOT=${INTEL_DIR}/tbb/latest

export BUILDTYPE=RelWithDebInfo

export MAD_NUM_THREADS=8
unset OMP_NUM_THREADS
export MAD_BUFFER_SIZE=128MB

cd $PROJECT_DIR
cd tiledarray
rm -rf build; mkdir build; cd build
cmake -D CMAKE_INSTALL_PREFIX=$PROJECT_INSTALL
-D CMAKE_TOOLCHAIN_FILE=$PROJECT_DIR/jlse-mpi-mkl-tbb.cmake
-D ENABLE_CUDA=ON
-D CMAKE_CUDA_ARCHITECTURES=70
-D ENABLE_TBB=OFF
-D BUILD_SHARED_LIBS=OFF
-D TA_ASSUMES_ASLR_DISABLED=OFF
-D CMAKE_POSITION_INDEPENDENT_CODE=ON
-D CMAKE_CXX_COMPILER=g++
-D CMAKE_PREFIX_PATH="${PROJECT_INSTALL}:${CMAKE_PREFIX_PATH}"
-D CMAKE_BUILD_TYPE=$BUILDTYPE
-D TA_BUILD_UNITTEST=TRUE
-D BTAS_BUILD_UNITTEST=TRUE
-D MPIEXEC_EXECUTABLE=mpirun
-D TA_TRACE_TASKS=OFF
..
export VERBOSE=1
make
make check`

[victor-anisimov]

Here is my toolchain file, jlse-mpi-mkl-tbb.cmake

`set(MKL_ROOT_DIR "$ENV{INTEL_DIR}/mkl/latest")
set(TBB_ROOT_DIR "$ENV{INTEL_DIR}/tbb/latest")
if (DEFINED ENV{EIGEN3_DIR})
set(ENV_EIGEN3_DIR "$ENV{EIGEN3_DIR}")
else()
set(ENV_EIGEN3_DIR "$ENV{EIGEN_DIR}")
endif()
set(EIGEN3_INCLUDE_DIR "${ENV_EIGEN3_DIR}" CACHE PATH "Eigen3 library directory")
set(BOOST_ROOT "$ENV{BOOST_DIR}" CACHE PATH "Boost root directory")

set(CMAKE_C_COMPILER gcc)
set(CMAKE_CXX_COMPILER g++)
set(CMAKE_Fortran_COMPILER gfortran)
set(MPI_CXX_SKIP_MPICXX ON)

set(CMAKE_C_FLAGS_INIT "-std=c99" CACHE STRING "Initial C compile flags")
set(CMAKE_C_FLAGS_DEBUG "-O0 -g -Wall" CACHE STRING "Initial C debug compile flags")
set(CMAKE_C_FLAGS_MINSIZEREL "-Os -DNDEBUG" CACHE STRING "Initial C minimum size release compile flags")
set(CMAKE_C_FLAGS_RELEASE "-O3 -DNDEBUG" CACHE STRING "Initial C release compile flags")
set(CMAKE_C_FLAGS_RELWITHDEBINFO "-O2 -g -Wall" CACHE STRING "Initial C release with debug info compile flags")
set(CMAKE_CXX_FLAGS_INIT "-I/home/vanisimov/tiledarray/install/include -std=c++11 -D__TBB_LEGACY_MODE=1 -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1" CACHE STRING "Initial C++ compile flags")
set(CMAKE_CXX_FLAGS_DEBUG "-I/home/vanisimov/tiledarray/install/include -O0 -g -Wall -ftemplate-backtrace-limit=0" CACHE STRING "Initial C++ debug compile flags")
set(CMAKE_CXX_FLAGS_MINSIZEREL "-I/home/vanisimov/tiledarray/install/include -Os -g -DNDEBUG" CACHE STRING "Initial C++ minimum size release compile flags")
set(CMAKE_CXX_FLAGS_RELEASE "-I/home/vanisimov/tiledarray/install/include -O3 -DNDEBUG" CACHE STRING "Initial C++ release compile flags")
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "-I/home/vanisimov/tiledarray/install/include -O2 -g -Wall -ftemplate-backtrace-limit=0" CACHE STRING "Initial C++ release with debug info compile flags")

set(LAPACK_LIBRARIES "-Wl,--start-group" "${MKL_ROOT_DIR}/lib/intel64/libmkl_intel_lp64.a" "${MKL_ROOT_DIR}/lib/intel64/libmkl_core.a" "${MKL_ROOT_DIR}/lib/intel64/libmkl_sequential.a" "-Wl,--end-group" "-lpthread" "-lm" "-ldl" CACHE STRING "BLAS linker flags")
set(LAPACK_INCLUDE_DIRS ${MKL_ROOT_DIR}/include CACHE STRING "LAPACK include directories")
set(LAPACK_COMPILE_DEFINITIONS BTAS_HAS_CBLAS=1;BTAS_HAS_LAPACKE=1;BTAS_HAS_INTEL_MKL=1;MADNESS_LINALG_USE_LAPACKE=1 CACHE STRING "LAPACK preprocessor definitions")
set(INTEGER4 TRUE CACHE BOOL "Set Fortran integer size to 4 bytes")
set(BUILD_SHARED_LIBS OFF CACHE BOOL "Build shared libraries")`

[victor-anisimov]

With SYCL version of cuTT ready for integration with TiledArray, it becomes important to get CUDA version of TiledArray running on JLSE so that the SYCL and CUDA ports of TiledArray could be debugged side by side. The goal is to use TiledArray CUDA unit tests to test the correctness of SYCL port of TiledArray. The current problem is that TiledArray CUDA unit tests hang on JLSE V100 GPU. The same issue happens on JLSE A100 node.

Here is some additional information. TA no longer supports CUDA/10.x. JLSE has the required CUDA/11.2 installed. GNU/8.2 compiles both the library and unit tests for CPU and CUDA versions of TA. The CPU unit tests successfully pass. The CUDA unit tests hang as described earlier.

GNU/9.3 compiles TA library for CPU and GPU but fails compiling unit tests.

GNU/10.2 compiles TA library and unit tests for CPU. The unit tests for CPU successfully complete. However, CUDA/11.2 does not support GNU/10, so GNU/10 cannot be used with CUDA version of TA. The highest version of GNU supported by CUDA is GNU/9.x.

GNU/9.3 gives the following error when compiling TA unit tests:

2: /home/vanisimov/tiledarray/tiledarray/src/TiledArray/range.h:580:17: required from here 2: /soft/packaging/spack-builds/linux-rhel7-x86_64/gcc-9.3.0/gcc-9.3.0-qfmcwfbuvnpn47zxjzfjvodzjl6reerh/include/c++/9.3.0/type_traits:2384:11: error: no type named 'type' in 'struct std::enable_if<false, void>' 2: 2384 | using enable_if_t = typename enable_if<_Cond, _Tp>::type; 2: | ^~~~~~~~~~~ 2: In file included from /home/vanisimov/tiledarray/tiledarray/tests/range.cpp:28: 2: /home/vanisimov/tiledarray/tiledarray/src/TiledArray/range.h:565:12: note: candidate: 'template<class ... Index, typename std::enable_if<TiledArray::detail::is_integral_list<Ts>::value>::type* <anonymous> > TiledArray::Range::Range(const Index ...)' 2: 565 | explicit Range(const Index... extents) 2: | ^~~~~ 2: /home/vanisimov/tiledarray/tiledarray/src/TiledArray/range.h:565:12: note: template argument deduction/substitution failed: 2: /home/vanisimov/tiledarray/tiledarray/src/TiledArray/range.h:564:63: error: no type named 'type' in 'struct std::enable_if<false, void>' 2: 564 | Index...>::value>::type* = nullptr>

The complete error log is attached.
log.cpu.gnu.9.3.TA-May15.txt

There is a discussion of the issue with enable_if in Stackoverflow https://stackoverflow.com/questions/13401716/selecting-a-member-function-using-different-enable-if-conditions . Since the above error happens only with GNU/9.x as I see it on JLSE, whereas GNU/8.x and GNU/10.x are free of this issue, the TA code is fine and the actual fault is in GNU/9.x. However, since CUDA TA unit tests hang when compiled with GNU/8.2, and GNU/9.3 is not able to compile the unit tests, it's a deadlock.

I tried configuring and compiling TA library for CUDA with GNU/9.3 and then compiling the unit tests with GNU/10.2 but the latter step did not work. It looks like CMake inserts GNU/9.3 include files/paths into the generated code during the configuration step that GCC/10.2 does not recognize.

Any suggestion on how to make CUDA TA unit tests running on JLSE when using GNU compilers?

[victor-anisimov]

I'm adding one more evidence that TA unit tests do not work on nVIDIA GPU platform. Following is an extra piece in support of that supposition.

Same as with GNU/8.2, TA unit tests work fine on CPU platform if compiled with Clang++ (icpx from Intel oneAPI SDK). Unlike GNU/9.3, Clang does not have issues with enable_if.

I just compiled TA and the unit tests for CUDA platform by using LLVM/10 as a host compiler and nvcc as a device compiler. Similar to the case with GNU/8.2 the unit tests hang on nVIDIA V100 GPU.

Does anyone have a version of TA that has unit tests running on nVIDIA GPU? I can assist with getting account on JLSE to access V100 nodes. The process should be easy since no NDA is required for that.

[asadchev]

Most likely this task will fall to me. How urgent is to have TA CUDA working? I am planning to refactor much of it over next two months and don't want to spend much time fixing current. On May 18, 2021 9:23 PM, "Victor Anisimov" ***@***.***> wrote: I'm adding one more evidence that TA unit tests do not work on nVIDIA GPU platform. Following is an extra piece in support of that supposition. Same as with GNU/8.2, TA unit tests work fine on CPU platform if compiled with Clang++ (icpx from Intel oneAPI SDK). Unlike GNU/9.3, Clang does not have issues with enable_if. I just compiled TA and the unit tests for CUDA platform by using LLVM/10 as a host compiler and nvcc as a device compiler. Similar to the case with GNU/8.2 the unit tests hang on nVIDIA V100 GPU. Does anyone have a version of TA that has unit tests running on nVIDIA GPU? I can assist with getting account on JLSE to access V100 nodes. The process should be easy since no NDA is required for that. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABH62KB44SMRYVNPMCHPYU3TOMHKPANCNFSM4YUOCW5A> .

[victor-anisimov]

@asadchev, getting CUDA version of TA working is kind of urgent for me. It blocks my work of porting TA to SYCL. If the current plan is to rework the CUDA code in TA then the work on SYCL port will have to be postponed. We may need to discuss the schedule with Theresa Windus.

[asadchev]

Can bring it up Friday? Before that can we meet over videophone, so I have idea what SYCL will require?

…

On May 18, 2021 9:56 PM, "Victor Anisimov" ***@***.***> wrote: @asadchev <https://github.com/asadchev>, getting CUDA version of TA working is kind of urgent for me. It blocks my work of porting TA to SYCL. If the current plan is to rework the CUDA code in TA then the work on SYCL port will have to be postponed. We may need to discuss the schedule with Theresa Windus. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABH62KEFWQ4ACZO7QTKMGGTTOMLDFANCNFSM4YUOCW5A> .

[victor-anisimov]

It's a good plan, @asadchev! Would a meeting at 10 AM Central (11 AM Eastern) work for you? I can send you a BlueJeans link. Otherwise, anytime today during the business hours would be fine with me. Just let me know.

[asadchev]

Yes, that will work. ***@***.***, can you cc Prof Valeev too, svp?

…

On May 19, 2021 10:01 AM, "Victor Anisimov" ***@***.***> wrote: It's a good plan, @asadchev <https://github.com/asadchev>! Would a meeting at 10 AM Central (11 AM Eastern) work for you? I can send you a BlueJeans link. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABH62KF6QLOOQO56RQNJQETTOPADZANCNFSM4YUOCW5A> .

[victor-anisimov]

Done. Sent a BlueJeans invite to all three of us, 10 AM Central (11 AM Eastern).

[victor-anisimov]

If I execute tests on a V100 node manually, e.g. "./ta_test --log_level=all" I can see that many test suites including cutt_suite pass. The first test that fails is um_expressions_suite. To reproduce "./ta_test --log_level=all --run_test=um_expressions_suite".

[evaleev]

superseded by #271