fix(vllm-cpp): map the CUDA 12 capabilities at the CPU build - #11388
Closed
localai-bot wants to merge 2 commits into
Closed
fix(vllm-cpp): map the CUDA 12 capabilities at the CPU build#11388localai-bot wants to merge 2 commits into
localai-bot wants to merge 2 commits into
Conversation
vllm.cpp's CUDA kernels need the CUDA 13 toolchain: 12.x nvcc lacks compute_121a and its ptxas rejects the sm_120a NVFP4 MMA kernels, so backend/go/vllm-cpp/Makefile ships no CUDA 12 variant and the arm64 CUDA build targets sm_121a (GB10 / DGX Spark) only. backend/index.yaml declared nvidia-l4t and nvidia-l4t-cuda-13 but no nvidia-l4t-cuda-12, so a Jetson AGX Orin (sm_87, JetPack 6) fell through SystemState.Capability's "default" catch-all and silently installed cpu-vllm-cpp. That is the right build for that host, but it was indistinguishable from an oversight both to a reader of the index and to a user wondering why their GPU box pulled a CPU backend. Map nvidia-cuda-12 and nvidia-l4t-cuda-12 explicitly at the CPU build, state the Blackwell-only constraint in the gallery description and the backend README, and add specs that pin the routing. No behaviour change: these hosts already resolved to the CPU build via the catch-all. The README also records the image-tag trap behind the same symptom on a supported host: /run/localai/capability is baked in at image build time, so a DGX Spark on the CUDA 12 -nvidia-l4t-arm64 image reports nvidia-l4t-cuda-12 and gets the CPU build; -nvidia-l4t-arm64-cuda-13 (or LOCALAI_FORCE_META_BACKEND_CAPABILITY) gets the GPU one. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…da12-fallback Two conflicts, both from master moving under a stale branch: - backend/index.yaml: master rewrote the vllm-cpp gallery description (#11363). Keep that rewrite and re-append this branch's paragraph on the CUDA 13 / Blackwell-only constraint and the CPU fallback for CUDA 12 hosts. - backend/go/vllm-cpp/README.md: master gained the ABI pin/mirror note from #11386. Keep both sections; they cover different things. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
backend/index.yamldeclaresnvidia,nvidia-cuda-13,nvidia-l4tandnvidia-l4t-cuda-13forvllm-cpp, but nothing for CUDA 12. A Jetson AGX Orin(sm_87, JetPack 6) therefore falls through
SystemState.Capability'sdefaultcatch-all and installs
cpu-vllm-cpp.That is the correct build for that host - vllm.cpp's CUDA kernels need the CUDA 13
toolchain, since 12.x nvcc lacks
compute_121aand its ptxas rejects the sm_120aNVFP4 MMA kernels, so no CUDA 12 variant is shipped at all. The problem is that
the right answer arrived by accident: it was indistinguishable from an oversight
both to someone reading the index and to a user wondering why their GPU box pulled
a CPU backend.
Map
nvidia-cuda-12andnvidia-l4t-cuda-12explicitly, state the Blackwell-onlyconstraint in the gallery description and the backend README, and pin the routing
with specs. No behaviour change: these hosts already resolved to the CPU build
through the catch-all.
The image-tag trap
The README now also records a second, separate way to end up on the CPU build,
which produces the same "why is my GPU idle" symptom on a supported host:
/run/localai/capabilityis baked into the image at build time. A DGX Sparkrunning the CUDA 12
-nvidia-l4t-arm64image reportsnvidia-l4t-cuda-12and sogets the CPU build. The fix is the
-nvidia-l4t-arm64-cuda-13image, orLOCALAI_FORCE_META_BACKEND_CAPABILITY=nvidia-l4t-cuda-13.Tests
core/gallery/backend_index_capabilities_test.gois new. Besides the vllm-cpprouting table it adds a repo-wide integrity check - every capability reference in
backend/index.yaml, for every backend, must resolve to an entry that actuallyexists in the index. That catches a typo'd or deleted meta-backend target across
the whole file, not just here.
Note on the branch
The commit is from 2026-07-27 and was never proposed. It has been merged up to
masterwith two conflicts resolved: master rewrote the vllm-cpp gallerydescription in #11363 (kept, with this branch's hardware paragraph re-appended),
and master gained the ABI pin/mirror note in #11386 (kept alongside the new
hardware-coverage section).
Verified after the merge:
make lint LINT_NEW_FROM=origin/masterreports 0 issues,core/galleryandcore/gallery/importersare green, andbackend/index.yamlstill parses with only the two
#amd:comment-spacing warnings that master alreadycarries.