pytorch flags override CUDA_ARCHITECTURES

[njzjz]

Describe the bug

pytorch flags override CUDA_ARCHITECTURES, making the build fail.

Expected behavior

CUDA_ARCHITECTURES set by cmake should not be overridden, as abacus has specific arch requirements.

To Reproduce

See https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=754750&view=logs&j=517fe804-fa30-5dc2-1413-330699242c05&t=c7d0fcb4-3a10-5b26-dfea-e879a93fc2ab&l=283

-- Autodetected CUDA architecture(s): 3.5;5.0;8.0;8.6;8.6+PTX

Environment

See the log for details.

Additional Context

No response

[hongriTianqi]

@njzjz Could you please try to modify the CMakeLists.txt line 84-98 as follows to see if it solves your issue?

if(USE_CUDA)
  target_link_libraries(
    device 
    cusolver 
    cublas 
    cufft
  )
  set_property(
    TARGET device
    APPEND
    PROPERTY CUDA_ARCHITECTURES
    60 # P100
    70 # V100
    75 # T4
    80 # A100
  )
}

If it works, I will submit a PR on this issue.

[caic99]

Hi @njzjz ,
This process happens in the configuring part of PyTorch.

[njzjz]

@hongriTianqi It does not work and even cannot pass the unittest.

[hongriTianqi]

@hongriTianqi It does not work and even cannot pass the unittest.

@njzjz

Sorry about the confusion, another trial is to include the necessary modifications to ABACUS's cmake file to ensure that the required CUDA architectures from PyTorch are included without overriding the existing CUDA_ARCHITECTURES. Again, I cannot make sure that it will work.

You may replace the existing set_property() block in ABACUS's cmake file with the following code:

include("${TORCH_INSTALL_PREFIX}/share/cmake/Caffe2/Caffe2Targets.cmake")
get_property(TORCH_CUDA_ARCH_LIST GLOBAL PROPERTY TORCH_CUDA_ARCH_LIST)
set_property(
TARGET device
PROPERTY CUDA_ARCHITECTURES
${TORCH_CUDA_ARCH_LIST}
60 # P100
70 # V100
75 # T4
80 # A100
)

This might ensure that ABACUS's specific arch requirements are preserved, while also including the required CUDA architectures from PyTorch.

I don't know if abacus will.define the same architecture with different names, in that case, unittests failure can be understood.

[njzjz]

@hongriTianqi I may make the issue clear: ABACUS does not support arch 3.5 but PyTorch may append such architecture. Compilation for 3.5 will fail with errors.

[njzjz]

See also https://stackoverflow.com/questions/75565347/atomicadd-showing-error-no-instance-of-overloaded-function-atomicadd-mathes

[hongriTianqi]

@njzjz thanks for clarifying the issue. In this case, do we need to exclude arch 3.5 from pytorch's settings? For example, adding the following line in pytorch camke file:

list(REMOVE_ITEM TORCH_CUDA_ARCH_LIST 35)