[BUG] Encountering raft::cuda_error for milvus index nodes on p4d.24xlarge AWS instance with 570.86.10 GPU driver

[maldi001]

Milvus Index nodes fail with the following error - W20250416 15:32:52.339228 20 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: raft::cuda_error
terminate called after throwing an instance of 'raft::cuda_error' on p4d.24xlarge AWS instance with 570.86.10 GPU driver.

Exact Error
W20250416 15:32:52.339228 20 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: raft::cuda_error
terminate called after throwing an instance of 'raft::cuda_error'
what(): CUDA error encountered at: file=/workspace/source/cmake_build/thirdparty/knowhere/knowhere-src/src/common/raft/integration/raft_initialization.cc line=53: call='cudaGetDeviceCount(&result)', Reason=cudaErrorSystemNotReady:system not yet initialized
Obtained 6 stack frames
#1 in /milvus/lib/libknowhere.so: raft::cuda_error::cuda_error(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) +0xd5 [0x7fcfacfa77b5]
#2 in /milvus/lib/libknowhere.so(+0x6a6377) [0x7fcfacdf7377]
#3 in /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fcfcda0aee8]
#4 in /milvus/lib/libknowhere.so: raft_knowhere::initialize_raft(raft_knowhere::raft_configuration const&) +0x6d [0x7fcfad4555ad]
#5 in /milvus/lib/libknowhere.so: knowhere::KnowhereConfig::SetRaftMemPool() +0x49 [0x7fcfacf6cd69]
#6 in milvus: runtime.asmcgocall.abi0 +0x64 [0x2045624]

Containers that failed
Name: milvus-querynode0, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode4, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode1, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode32, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode46, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode18, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode5, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode20, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode29, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode39, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode15, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode8, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode14, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode21, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode3, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode16, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode10, Status: Exited (134) 3 minutes ago
Name: milvus-indexnode33, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode44, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode19, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode30, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode36, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode35, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode42, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode11, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode31, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode24, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode12, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode0, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode23, Status: Exited (134) 3 minutes ago
Name: milvus-indexnode17, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode28, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode27, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode2, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode45, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode7, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode26, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode37, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode13, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode34, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode47, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode22, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode43, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode41, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode38, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode6, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode9, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode40, Status: Exited (134) 2 minutes ago
Name: milvus-indexnode25, Status: Exited (134) 2 minutes ago

Steps/Code to reproduce bug

• Launch p4d.24xlarge AWS instance. And install the cuda driver and cuda toolkit

• wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin

• sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

• wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

• sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

• sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

• sudo apt-get update

• sudo apt-get install -y cuda-drivers

• curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

• sudo apt-get update

• sudo apt-get install -y nvidia-container-toolkit

• sudo nvidia-ctk runtime configure --runtime=docker

• sudo systemctl restart docker

• Launch the test with the following variables
  export MILVUS_DOCKER_VERSION='milvusdb/milvus:v2.5.4-gpu'
  export MILVUS_GPU_DEVICES="0,1,2,3,4,5,6,7"
  export MILVUS_INODE_PER_GPU=6
  export MILVUS_NUM_DNODES=48
  export DATASET_NAME=miracl-en
  export NIM_EMBEDDING_DIM=2048
  export MILVUS_MAX_SEGMENT_ROWS=120000

Debug steps

• Restarted the VM instance
• Re-installed the driver
• Install nvidia-fabric manager (sudo apt install nvidia-driver-570-open libnvidia-nscq-570 nvidia-modprobe nvidia-fabricmanager-570 datacenter-gpu-manager-4 nv-persistence-mode) No luck in installing the software.
• nvidia-smi topo -m output
  GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
  GPU0 X PHB NODE NODE SYS SYS SYS SYS 0-23,48-71 0 N/A
  GPU1 PHB X NODE NODE SYS SYS SYS SYS 0-23,48-71 0 N/A
  GPU2 NODE NODE X PHB SYS SYS SYS SYS 0-23,48-71 0 N/A
  GPU3 NODE NODE PHB X SYS SYS SYS SYS 0-23,48-71 0 N/A
  GPU4 SYS SYS SYS SYS X PHB NODE NODE 24-47,72-95 1 N/A
  GPU5 SYS SYS SYS SYS PHB X NODE NODE 24-47,72-95 1 N/A
  GPU6 SYS SYS SYS SYS NODE NODE X PHB 24-47,72-95 1 N/A
  GPU7 SYS SYS SYS SYS NODE NODE PHB X 24-47,72-95 1 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
ubuntu@ip-172-31-8-187:/raid2$ nvidia-smi nvlink --status -i 0
GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-5af94945-3b0e-4d14-6884-165e27c9b45a)
NVML: Unable to retrieve NVLink information as all links are inActive

[singhmanas1]

Tried with 1xA100 on the same p4d.24xlarge AWS instance (as per Milvus team's suggestion) but still running into the same issue-

terminate called after throwing an instance of 'raft::cuda_error'
  what():  CUDA error encountered at: file=/workspace/source/cmake_build/thirdparty/knowhere/knowhere-src/src/common/raft/integration/raft_initialization.cc line=53: call='cudaGetDeviceCount(&result)', Reason=cudaErrorSystemNotReady:system not yet initialized

Attaching the index node log for reference as well.

milvus-indexnode4.log