Add PCI bus fallback for Linux GPU device discovery in containerized environments - #27591
Merged
Conversation
In AKS/Kubernetes environments, the nvidia-drm kernel module may not be loaded, resulting in no /sys/class/drm/cardN entries for NVIDIA GPUs. This adds a fallback that scans /sys/bus/pci/devices/ for GPU-class PCI devices (VGA compatible controllers and 3D controllers) when DRM-based detection finds no GPUs. Co-authored-by: baijumeswani <12852605+baijumeswani@users.noreply.github.com>
…ment for card_idx Co-authored-by: baijumeswani <12852605+baijumeswani@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Analyze potential issues with cuda execution provider in AKS
Add PCI bus fallback for Linux GPU device discovery in containerized environments
Mar 9, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a PCI-bus sysfs fallback to Linux GPU device discovery so GPU devices can still be surfaced (and EPs matched) in containerized environments where /sys/class/drm/cardN is absent (e.g., when nvidia-drm isn’t loaded).
Changes:
- Added PCI scanning of
/sys/bus/pci/devices(filtering display controllers by class/subclass) and creation ofOrtHardwareDevicerecords fromvendor/devicesysfs files. - Updated
GetGpuDevices()to attempt DRM-based discovery first and fall back to PCI scanning only when DRM yields zero GPUs. - Added a new device discovery test that validates basic GPU properties when GPUs are present.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| onnxruntime/core/platform/linux/device_discovery.cc | Implements PCI-bus fallback GPU discovery and integrates it into Linux GetGpuDevices() when DRM discovery returns empty. |
| onnxruntime/test/platform/device_discovery_test.cc | Adds a GPU discovery smoke/validation test to ensure GPU enumeration doesn’t crash and returns sane IDs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…bility The device_id field is not populated on Apple Silicon (defaults to 0), so asserting device_id != 0 would cause test failures on that platform. Co-authored-by: baijumeswani <12852605+baijumeswani@users.noreply.github.com>
…paths and add hermetic tests - Move PCI detection functions (DetectGpuPciPaths, GetGpuDeviceFromPci) from anonymous namespace to onnxruntime::pci_device_discovery namespace - Add device_discovery_linux.h header exposing PCI detection API for testing - Make DetectGpuPciPaths accept sysfs root path parameter for testability - Add comprehensive hermetic unit tests using fake sysfs directory structure: - VGA controller detection (class 0x0300) - 3D controller detection (class 0x0302) - Filtering out non-GPU PCI devices - Nonexistent/empty directory handling - Multiple GPU detection - Missing class file handling - Vendor/device ID reading and metadata population - NVIDIA vs non-NVIDIA discrete GPU metadata Co-authored-by: baijumeswani <12852605+baijumeswani@users.noreply.github.com>
Co-authored-by: baijumeswani <12852605+baijumeswani@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
edgchen1
reviewed
Mar 10, 2026
…m PCI path, use ASSERT_STATUS_OK, remove unnecessary test setup - Add reference link to PCI Code and ID Assignment Specification for class codes - Remove card_idx metadata from PCI fallback since directory_iterator order is unspecified and cannot match DRM's cardN ordering - Replace ASSERT_TRUE(status.IsOK()) with ASSERT_STATUS_OK() macro - In GetGpuDeviceFromPci tests, only create vendor/device files (not class) since those are the only files the function reads Co-authored-by: edgchen1 <18449977+edgchen1@users.noreply.github.com>
Contributor
Author
edgchen1
reviewed
Mar 10, 2026
This was referenced Mar 23, 2026
f-dy
pushed a commit
to f-dy/onnxruntime
that referenced
this pull request
Mar 23, 2026
On AWS EC2 vGPU instances (e.g. g5.xlarge with A10G), /sys/class/drm/card0 exists (simple-framebuffer for console VGA) but lacks device/vendor. GetGpuDeviceFromSysfs fails, and ORT_RETURN_IF_ERROR propagates the error immediately, preventing the PCI fallback at line 289 from ever running. The PCI path has the correct data: /sys/bus/pci/devices/0000:00:1e.0/vendor = 0x10de (NVIDIA) /sys/bus/pci/devices/0000:00:1e.0/class = 0x030200 (3D controller) Change the DRM loop to log a warning and continue instead of returning error, as suggested by @tianleiwu during the microsoft#27591 review. This allows the loop to skip non-GPU DRM entries (like simple-framebuffer) and fall through to the PCI fallback when no valid GPU is found via DRM. Fixes microsoft#27806
This was referenced Mar 30, 2026
This was referenced Apr 20, 2026
This was referenced May 1, 2026
This was referenced May 8, 2026
This was referenced Jul 1, 2026
This was referenced Jul 8, 2026
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GPU device discovery on Linux relies exclusively on
/sys/class/drm/cardNentries (DRM subsystem). In AKS/Kubernetes containers,nvidia-drmis typically not loaded—only the base NVIDIA driver is needed for CUDA compute. No DRM entries means noOrtHardwareDevicewithOrtHardwareDeviceType_GPUis created, soGetEpDevicesnever matches the CUDA EP.Adds a fallback path in
GetGpuDevices()that scans/sys/bus/pci/devices/when DRM yields zero GPUs:DetectGpuPciPaths()— enumerates PCI devices, filters by class code0x0300(VGA) and0x0302(3D controller, used by NVIDIA datacenter GPUs) per the PCI Code and ID Assignment Specification (base class 03h). Accepts an injectable sysfs root path for testability.GetGpuDeviceFromPci()— readsvendor/devicefiles directly from the PCI device sysfs path and populatesOrtHardwareDevicewithpci_bus_idand discrete GPU metadata. Note:card_idxis intentionally omitted from PCI-discovered devices sincedirectory_iteratortraversal order is unspecified and cannot be made consistent with DRM'scardNordering.GetGpuDevices()— tries DRM first; if empty, falls back to PCI scanThe PCI detection functions are exposed via a new
onnxruntime::pci_device_discoverynamespace (declared incore/platform/linux/pci_device_discovery.h) so they can be tested hermetically with fake sysfs directories.The fallback only activates when DRM finds nothing, so no behavioral change on systems where DRM works.
Also adds:
GpuDevicesHaveValidPropertiestest that validates GPU device type and vendor ID when GPUs are present. The test intentionally does not assert ondevice_idsince some platforms (e.g., Apple Silicon) do not populate it.test/platform/linux/pci_device_discovery_test.cc) that create fake sysfs directory structures to exercise the PCI fallback path, covering VGA/3D controller detection, non-GPU filtering, empty/missing paths, multiple GPUs, vendor/device ID reading, and NVIDIA discrete metadata. Tests use theASSERT_STATUS_OK()macro fromtest/util/include/asserts.hand useCreateFakePciDeviceto set up complete fake PCI device directories for bothDetectGpuPciPathsandGetGpuDeviceFromPcitests.Motivation and Context
CUDA EP registration fails on AKS (Azure Kubernetes Service) because the NVIDIA device plugin exposes GPUs via
/dev/nvidia*and the NVIDIA driver, but does not loadnvidia-drm. The existing/sys/class/drm-only detection path returns no GPU devices, blockingGetEpDevicesfrom returning the CUDA EP. The same setup works on bare-metal Linux where DRM is loaded.💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.