[Issue]: torch.cuda.mem_get_info function RuntimeError(HIP error: invalid argument)

[jaeyung2]

Problem Description

I'm try to run sample code in container after install gpu-operator helm chart on my cluster(1.32.8 k8s version). my test code is below. How can I solve this problem? I'd appreciate it if you could let me know.

import torch
if torch.cuda.is_available():
    free_memory_bytes, total_memory_bytes = torch.cuda.mem_get_info()

    free_memory_gb = free_memory_bytes / (1024**3)
    total_memory_gb = total_memory_bytes / (1024**3)

    print(f"Free GPU memory: {free_memory_gb:.2f} GB")
    print(f"Total GPU memory: {total_memory_gb:.2f} GB")

else:
    print("CUDA is not available. Cannot get GPU memory info.")

and error message is:

Traceback (most recent call last):
  File "/var/lib/jenkins/test.py", line 4, in <module>
    free_memory_bytes, total_memory_bytes = torch.cuda.mem_get_info()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/memory.py", line 738, in mem_get_info
    return torch.cuda.cudart().cudaMemGetInfo(device)
RuntimeError: HIP error: invalid argument
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

my pod yaml information:

apiVersion: v1
kind: Pod
metadata:
  name: rocm-pytorch-test
  namespace: kube-amd-gpu
  labels:
    app: rocm-pytorch-test
spec:
  restartPolicy: Never
  tolerations:
    - key: "amd.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: rocm-pytorch
      image: rocm/pytorch:rocm6.4.2_ubuntu22.04_py3.10_pytorch_release_2.6.0
      imagePullPolicy: IfNotPresent
      command: ["sleep", "infinity"]
      resources:
        limits:
          "amd.com/gpu": 8

Here is my OS and GPU information

OS: NAME="Ubuntu"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
CPU: 
model name	: AMD EPYC 7413 24-Core Processor
GPU:
  Name:                    AMD EPYC 7413 24-Core Processor    
  Marketing Name:          AMD EPYC 7413 24-Core Processor    
  Name:                    AMD EPYC 7413 24-Core Processor    
  Marketing Name:          AMD EPYC 7413 24-Core Processor    
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
  Name:                    gfx90a                             
  Marketing Name:          AMD Instinct MI250X/MI250          
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-

Operating System

Ubuntu 22.04

CPU

AMD EPYC 7413 24-Core Processor

GPU

AMD Instinct MI250X/MI250

ROCm Version

ROCm 6.4.2

ROCm Component

HIP

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[jaeyung2]

When I run my python script with AMD_LOG_LEVEL=1 and multiple error log like below

:1:hip_fatbin.cpp           :779 : 15176473012 us:  Cannot find CO in the bundle /opt/rocm-6.4.2/lib/libMIOpen.so.1.0.60402 for ISA: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
:1:hip_fatbin.cpp           :564 : 15915461486 us:  No compatible code objects found for: gfx90a:sramecc+:xnack-, value of HIP_FORCE_SPIRV_CODEOBJECT: 0
:1:rocdevice.cpp            :1932: 15918401612 us:  HSA_AMD_AGENT_INFO_MEMORY_AVAIL query failed.

[yansun1996]

Hi @jaeyung2 thanks for reporting this issue.

1. Could you please check whether rocm-smi or amd-smi list could show expected hardware information within the PyTorch image you are using ?
2. Is the driver pre-installed on the node or installed by GPU operator ?

[jaeyung2]

Hi, @yansun1996

1. Could you please check whether rocm-smi or amd-smi list could show expected hardware information within the PyTorch image you are using ?

• run rocm-smi command

root@rocm-pytorch-test:/var/lib/jenkins# rocm-smi


========================================= ROCm System Management Interface =========================================
=================================================== Concise Info ===================================================
Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
              (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
====================================================================================================================
0       2     0x740c,   31957  40.0°C  87.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
1       3     0x740c,   7103   36.0°C  0.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2       4     0x740c,   9752   40.0°C  90.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
3       5     0x740c,   52450  35.0°C  0.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
4       6     0x740c,   49482  44.0°C  93.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
5       7     0x740c,   60436  40.0°C  0.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
6       8     0x740c,   35581  42.0°C  85.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
7       9     0x740c,   20807  39.0°C  0.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
====================================================================================================================
=============================================== End of ROCm SMI Log ================================================

• run amd-smi list command

root@rocm-pytorch-test:/var/lib/jenkins# amd-smi list
GPU: 0
    BDF: 0000:29:00.0
    UUID: 4fff740c-0000-1000-8072-8ebc89268dca
    KFD_ID: 31957
    NODE_ID: 2
    PARTITION_ID: 0

GPU: 1
    BDF: 0000:2c:00.0
    UUID: 5bff740c-0000-1000-80ed-1333a4145366
    KFD_ID: 7103
    NODE_ID: 3
    PARTITION_ID: 0

GPU: 2
    BDF: 0000:2f:00.0
    UUID: 59ff740c-0000-1000-8012-5243f98885f9
    KFD_ID: 9752
    NODE_ID: 4
    PARTITION_ID: 0

GPU: 3
    BDF: 0000:32:00.0
    UUID: 6bff740c-0000-1000-8061-cce40adb54f7
    KFD_ID: 52450
    NODE_ID: 5
    PARTITION_ID: 0

GPU: 4
    BDF: 0000:ad:00.0
    UUID: 8dff740c-0000-1000-8073-71b5f9c86e6b
    KFD_ID: 49482
    NODE_ID: 6
    PARTITION_ID: 0

GPU: 5
    BDF: 0000:b0:00.0
    UUID: 9cff740c-0000-1000-8070-48f08bcb2ea7
    KFD_ID: 60436
    NODE_ID: 7
    PARTITION_ID: 0

GPU: 6
    BDF: 0000:b3:00.0
    UUID: 0eff740c-0000-1000-80d9-ae2ed94f35e4
    KFD_ID: 35581
    NODE_ID: 8
    PARTITION_ID: 0

GPU: 7
    BDF: 0000:b6:00.0
    UUID: 2eff740c-0000-1000-80af-357d3313eac9
    KFD_ID: 20807
    NODE_ID: 9
    PARTITION_ID: 0

2. Is the driver pre-installed on the node or installed by GPU operator ?

It's installed by GPU operator. set deviceConfig.spec.driver.enable: true of values.yaml

[yansun1996]

ok thanks for the information @jaeyung2 , I think this could be some issues on the PyTorch image, or for the image compatibility with the system / hardware, since the error message is complaining it cannot find some libs or objects in the rocm libs.

GPU operator's functionality is to install the driver and bring up the k8s basic components, as for the specific PyTorch workload image issue, could you please also raise an issue in https://github.com/ROCm/ROCm or https://github.com/ROCm/ROCm-docker ?

[jaeyung2]

hi @yansun1996 . Our team just found that the issue occured in ubuntu 22.04. but not in ubuntu 24.04. We tested with hip code like below.

• test_pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rocminfo
  namespace: kube-amd-gpu
spec:
  containers:
  - image: docker.io/rocm/rocm-terminal:latest
    name: rocminfo
    command: ["/bin/sh","-c"]
    args: ["rocminfo && sleep infinity"]
    securityContext:
      runAsUser: 0
    resources:
      limits:
        amd.com/gpu: 8
  restartPolicy: Never

• test.cpp (build in container)

#include <hip/hip_runtime.h>
#include <iostream>

void checkHipError(hipError_t err) {
    if (err != hipSuccess) {
        std::cerr << "HIP Error: " << hipGetErrorString(err) << std::endl;
        exit(EXIT_FAILURE);
    }
}

int main() {
    checkHipError(hipSetDevice(0));
    std::cout << "✅ HIP Runtime and Initialize GPU device success." << std::endl;

    size_t free_bytes, total_bytes;

    hipError_t err = hipMemGetInfo(&free_bytes, &total_bytes);

    if (err != hipSuccess) {
        std::cerr << "❌ Error: Failed to get GPU memory information." << std::endl;
        std::cerr << "HIP Error: " << hipGetErrorString(err) << std::endl;
        return -1;
    }

    std::cout << "🎉 Success: Available to get GPU memory information." << std::endl;

    double free_mb = static_cast<double>(free_bytes) / (1024.0 * 1024.0);
    double total_mb = static_cast<double>(total_bytes) / (1024.0 * 1024.0);
    
    std::cout << "----------------------------------------" << std::endl;
    std::cout << "GPU memory info:" << std::endl;
    std::cout << "  - Total memory (Total): " << total_mb << " MB" << std::endl;
    std::cout << "  - Available memory (Free): " << free_mb << " MB" << std::endl;
    std::cout << "----------------------------------------" << std::endl;

    return 0;
}

• ubuntu 22.04 (kernel version: 5.15.0-153-generic, rocm version: 6.4.2)

root@rocminfo:/home/rocm-user# ./test 
✅ HIP Runtime and Initialize GPU device success.
❌ Error: Failed to get GPU memory information.
HIP Error: invalid argument

• ubuntu 24.04 (kernel version: 6.8.0-83-generic, rocm version: 6.4.2)

root@rocminfo:/home/rocm-user# ./test
✅ HIP Runtime and Initialize GPU device success.
🎉 Success: Available to get GPU memory information.
----------------------------------------
GPU memory info:
  - Total memory (Total):65520 MB
  - Available memory (Free): 65446 MB
----------------------------------------

and also found that difference kernel module load between ubuntu22.04 and ubuntu24.04.

# ubuntu 24.04 (after reboot)
root@mi250-041:~# lsmod | grep -i amd
amdgpu              19730432  10
amddrm_ttm_helper      12288  1 amdgpu
amdttm                114688  2 amdgpu,amddrm_ttm_helper
amddrm_buddy           24576  1 amdgpu
amdxcp                 12288  1 amdgpu
amd_sched              61440  1 amdgpu
amdkcl                 36864  3 amd_sched,amdttm,amdgpu
drm_exec               12288  1 amdgpu
drm_suballoc_helper    16384  1 amdgpu
drm_display_helper    237568  1 amdgpu
cec                    94208  2 drm_display_helper,amdgpu
video                  77824  1 amdgpu
amd64_edac             61440  0
edac_mce_amd           28672  1 amd64_edac
kvm_amd               208896  0
kvm                  1409024  1 kvm_amd
ccp                   143360  1 kvm_amd
i2c_algo_bit           16384  3 igb,ast,amdgpu

# ubuntu 22.04 (after reboot)
root@mi250-042:~/gpu-operator# lsmod | grep -i amd
amd64_edac             53248  0
edac_mce_amd           36864  1 amd64_edac
kvm_amd               155648  0
kvm                  1040384  1 kvm_amd
ccp                   110592  1 kvm_amd
amdgpu               9904128  10
iommu_v2               24576  1 amdgpu
gpu_sched              45056  1 amdgpu
drm_ttm_helper         16384  3 drm_vram_helper,ast,amdgpu
ttm                    86016  3 drm_vram_helper,amdgpu,drm_ttm_helper
drm_kms_helper        315392  5 drm_vram_helper,ast,amdgpu
drm                   622592  8 gpu_sched,drm_kms_helper,drm_vram_helper,ast,amdgpu,drm_ttm_helper,ttm
i2c_algo_bit           16384  3 igb,ast,amdgpu

[jaeyung2]

• In Ubuntu 22.04, the kmm-worker log like below

E0918 11:09:31.777162       1 funcs_kmod.go:15] "Could not get the git commit; using <undefined>" err="vcs.revision not found in build info settings" logger="kmm-worker"
I0918 11:09:31.777305       1 funcs_kmod.go:18] "Starting worker" logger="kmm-worker" version="0.0.1" git commit="<undefined>"
I0918 11:09:31.777320       1 funcs_kmod.go:30] "Reading config" logger="kmm-worker" path="/etc/kmm-worker/config.yaml"
I0918 11:09:31.778845       1 worker.go:76] "preparing firmware for loading" logger="kmm-worker" image directory="/tmp/firmwareDir/updates" host mount directory="/var/lib/firmware"
I0918 11:09:32.942374       1 modprobe.go:33] "Running modprobe" logger="kmm-worker" command="/sbin/modprobe -vd /tmp/opt amdgpu"

• But in Ubuntu 24.04, the kmm-worker log like below

E0918 10:34:02.090162       1 funcs_kmod.go:15] "Could not get the git commit; using <undefined>" err="vcs.revision not found in build info settings" logger="kmm-worker"
I0918 10:34:02.090380       1 funcs_kmod.go:18] "Starting worker" logger="kmm-worker" version="0.0.1" git commit="<undefined>"
I0918 10:34:02.090400       1 funcs_kmod.go:30] "Reading config" logger="kmm-worker" path="/etc/kmm-worker/config.yaml"
I0918 10:34:02.091090       1 worker.go:76] "preparing firmware for loading" logger="kmm-worker" image directory="/tmp/firmwareDir/updates" host mount directory="/var/lib/firmware"
I0918 10:34:02.728754       1 modprobe.go:33] "Running modprobe" logger="kmm-worker" command="/sbin/modprobe -vd /tmp/opt amdgpu"
I0918 10:34:02.739348       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/acpi/video.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585199       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/gpu/drm/display/drm_display_helper.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585228       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/gpu/drm/drm_suballoc_helper.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585263       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/gpu/drm/drm_exec.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585270       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdkcl.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585276       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amd-sched.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585282       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdxcp.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585288       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amddrm_buddy.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585295       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdttm.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585301       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amddrm_ttm_helper.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585308       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdgpu.ko " logger="kmm-worker.modprobe.stdout"

I think somethine went wrong with /sbin/modprobe -vd /tmp/opt amdgpu ? or went wronginsmod /tmp/opt/lib/modules/{UBUNTU_22_04_KERNEL_VERSION}-generic/kernel/drivers/acpi/video.ko(.zst) in ubuntu 22.04?

[yansun1996]

• In Ubuntu 22.04, the kmm-worker log like below

E0918 11:09:31.777162       1 funcs_kmod.go:15] "Could not get the git commit; using <undefined>" err="vcs.revision not found in build info settings" logger="kmm-worker"
I0918 11:09:31.777305       1 funcs_kmod.go:18] "Starting worker" logger="kmm-worker" version="0.0.1" git commit="<undefined>"
I0918 11:09:31.777320       1 funcs_kmod.go:30] "Reading config" logger="kmm-worker" path="/etc/kmm-worker/config.yaml"
I0918 11:09:31.778845       1 worker.go:76] "preparing firmware for loading" logger="kmm-worker" image directory="/tmp/firmwareDir/updates" host mount directory="/var/lib/firmware"
I0918 11:09:32.942374       1 modprobe.go:33] "Running modprobe" logger="kmm-worker" command="/sbin/modprobe -vd /tmp/opt amdgpu"

• But in Ubuntu 24.04, the kmm-worker log like below

E0918 10:34:02.090162       1 funcs_kmod.go:15] "Could not get the git commit; using <undefined>" err="vcs.revision not found in build info settings" logger="kmm-worker"
I0918 10:34:02.090380       1 funcs_kmod.go:18] "Starting worker" logger="kmm-worker" version="0.0.1" git commit="<undefined>"
I0918 10:34:02.090400       1 funcs_kmod.go:30] "Reading config" logger="kmm-worker" path="/etc/kmm-worker/config.yaml"
I0918 10:34:02.091090       1 worker.go:76] "preparing firmware for loading" logger="kmm-worker" image directory="/tmp/firmwareDir/updates" host mount directory="/var/lib/firmware"
I0918 10:34:02.728754       1 modprobe.go:33] "Running modprobe" logger="kmm-worker" command="/sbin/modprobe -vd /tmp/opt amdgpu"
I0918 10:34:02.739348       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/acpi/video.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585199       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/gpu/drm/display/drm_display_helper.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585228       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/gpu/drm/drm_suballoc_helper.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585263       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/kernel/drivers/gpu/drm/drm_exec.ko.zst " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585270       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdkcl.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585276       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amd-sched.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585282       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdxcp.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585288       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amddrm_buddy.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585295       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdttm.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585301       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amddrm_ttm_helper.ko " logger="kmm-worker.modprobe.stdout"
I0918 10:34:39.585308       1 cmdlog.go:70] "insmod /tmp/opt/lib/modules/6.8.0-79-generic/updates/dkms/amdgpu.ko " logger="kmm-worker.modprobe.stdout"

I think somethine went wrong with /sbin/modprobe -vd /tmp/opt amdgpu ? or went wronginsmod /tmp/opt/lib/modules/{UBUNTU_22_04_KERNEL_VERSION}-generic/kernel/drivers/acpi/video.ko(.zst) in ubuntu 22.04?

Hi @jaeyung2 I think that's due to the difference of the kernel version. You are comparing the 5.15 vs 6.8 generic kernels, there are always changes happening in the development of the Linux kernel between major versions, so that the kernel module dependency graph could change as well.

[jaeyung2]

Hi @jaeyung2 I think that's due to the difference of the kernel version. You are comparing the 5.15 vs 6.8 generic kernels, there are always changes happening in the development of the Linux kernel between major versions, so that the kernel module dependency graph could change as well.

Yes, I know. I think Ubuntu 22.04 should also load the 'amdkcl', 'amdxcp' kernel module, but it doesn't work that way. Or do you think only the amdgpu kernel module should load in Ubuntu 22.04?

[yansun1996]

Hi @jaeyung2 I think that's due to the difference of the kernel version. You are comparing the 5.15 vs 6.8 generic kernels, there are always changes happening in the development of the Linux kernel between major versions, so that the kernel module dependency graph could change as well.

Yes, I know. I think Ubuntu 22.04 should also load the 'amdkcl', 'amdxcp' kernel module, but it doesn't work that way. Or do you think only the amdgpu kernel module should load in Ubuntu 22.04?

ok, good point @jaeyung2 this line shows it is using inbox driver, looks like blacklist not applied. otherwise it would use amdttm, that's what I remembered before.

Could you please check whether the inbox amdgpu driver was blacklisted or not on that Ubuntu 22.04 ?

ttm                    86016  3 drm_vram_helper,amdgpu,drm_ttm_helper

[wonkyooh]

Hi @jaeyung2 I think that's due to the difference of the kernel version. You are comparing the 5.15 vs 6.8 generic kernels, there are always changes happening in the development of the Linux kernel between major versions, so that the kernel module dependency graph could change as well.

Yes, I know. I think Ubuntu 22.04 should also load the 'amdkcl', 'amdxcp' kernel module, but it doesn't work that way. Or do you think only the amdgpu kernel module should load in Ubuntu 22.04?

ok, good point @jaeyung2 this line shows it is using inbox driver, looks like blacklist not applied. otherwise it would use amdttm, that's what I remembered before.

Could you please check whether the inbox amdgpu driver was blacklisted or not on that Ubuntu 22.04 ?

ttm                    86016  3 drm_vram_helper,amdgpu,drm_ttm_helper

Hi, @yansun1996 Im WonKyo, @jaeyung2 's coworker and I will answer your question.
On ubuntu 22.04 blacklisting amdgpu does not actually blacklist amdgpu module for it is still in the initramfs

root@mi250-041:/etc/modprobe.d# cat blacklist.conf 
# This file lists those modules which we don't want to be loaded by
# alias expansion, usually so some other driver will be loaded for the
# device instead.

# evbug is a debug tool that should be loaded explicitly
blacklist evbug

# these drivers are very simple, the HID drivers are usually preferred
blacklist usbmouse
blacklist usbkbd

# replaced by e100
blacklist eepro100

# replaced by tulip
blacklist de4x5

# causes no end of confusion by creating unexpected network interfaces
blacklist eth1394

# snd_intel8x0m can interfere with snd_intel8x0, doesn't seem to support much
# hardware on its own (Ubuntu bug #2011, #6810)
blacklist snd_intel8x0m

# Conflicts with dvb driver (which is better for handling this device)
blacklist snd_aw2

# replaced by p54pci
blacklist prism54

# replaced by b43 and ssb.
blacklist bcm43xx

# most apps now use garmin usb driver directly (Ubuntu: #114565)
blacklist garmin_gps

# replaced by asus-laptop (Ubuntu: #184721)
blacklist asus_acpi

# low-quality, just noise when being used for sound playback, causes
# hangs at desktop session start (Ubuntu: #246969)
blacklist snd_pcsp

# ugly and loud noise, getting on everyone's nerves; this should be done by a
# nice pulseaudio bing (Ubuntu: #77010)
blacklist pcspkr

# EDAC driver for amd76x clashes with the agp driver preventing the aperture
# from being initialised (Ubuntu: #297750). Blacklist so that the driver
# continues to build and is installable for the few cases where its
# really needed.
blacklist amd76x_edac

blacklist amdgpu

root@mi250-041:/etc/modprobe.d# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.0-153-generic
W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast
W: Possible missing firmware /lib/firmware/amdgpu/yellow_carp_gpu_info.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/vangogh_gpu_info.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_rlc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_mec2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_me.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_pfp.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_ce.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_sdma1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/cyan_skillfish_sdma.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi10_mes.bin for module amdgpu
root@mi250-041:/etc/modprobe.d# lsinitramfs /boot/initrd.img-5.15.0-153-generic | grep -i amdgpu.ko
usr/lib/modules/5.15.0-153-generic/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko

So, on another node that has amdgpu driver installed via amdgpu-install, i checked the iniramfs to see how the amdgpu module was loaded and it just lists the new amdgpu module that was installed by amdgpu-install

root@mi250-026:~# lsinitramfs /boot/initrd.img-5.15.0-25-generic | grep -i amdgpu.ko
usr/lib/modules/5.15.0-25-generic/updates/dkms/amdgpu.ko

Also, I have tested unloading the amdgpu module by running the following command on both ubuntu 24.04 and 22.04 and it seems that only ubuntu 24.04 unloads it and ubuntu 22.04 cannot unload it saying that the module is being used.

modprobe -r amdgpu

Sorry I do not have the logs of the above command from our severs.

If you have any other suggestions or solutions please feel free to share with us