ROCm (AMD GPU) backend

[mudler]

Someone offered to help bring up ROCm support, so this issue tracks the work.

ROCm is listed as roadmap in the README backend table. The codebase was structured for exactly this kind of addition: the engine core (scheduler, KV/block management, persistent batch, sampler, serving) is backend-agnostic, mirroring upstream vLLM, and a new device lands as additive files through three seams (see .agents/backends.md):

1. Platform: a mirror of vllm/platforms/. Upstream already ships platforms/rocm.py, so ours is a port of that. Platforms self-register and expose device probing, memory model, stream semantics and graph-capture capability.
2. Attention backend registry: backends self-register per (DeviceType, name). Adding one is a single self-registering translation unit plus a priority slot.
3. vt:: op tables: per-device kernel registration for GEMM, norms, rope, activations, MoE and sampling.

Adding a platform never touches engine code. The drop-in kernel ABI (.agents/specs/dropin-kernel-abi.md) matches upstream kernel entry-point signatures at the vt-op boundary, and upstream's ROCm kernels mirror the CUDA signatures in csrc/, so kernels lifted from vLLM's ROCm path should bind without redesign.

vLLM's ROCm platform and kernels are the primary upstream to mirror. SGLang has native ROCm support too and is a secondary reference. Build recipes and per-backend state are in docs/BUILD.md.

Milestones

• M0, build: -DVLLM_CPP_HIP=ON CMake path, HIP toolchain detection, portable layer compiles for a target gfx arch.
• M1, platform: platforms/rocm mirroring vllm/platforms/rocm.py, self-registered.
• M2, first model end to end: a small dense model produces coherent output on an AMD GPU via the portable kernel path, token parity vs the CPU reference backend.
• M3, attention: register a ROCm attention backend. Upstream uses Triton/AITER flash attention on ROCm; start from whatever vLLM selects for the target arch.
• M4, correctness gate: greedy token parity vs a vLLM-ROCm oracle on the same hardware, following the project gate methodology.
• M5, speed: benchmark vs vLLM on the same box, quant-matched. The bar is vLLM, not other engines.

If you want to pick this up

Comment with the hardware and stack you have (MI300, consumer RDNA, Strix Halo APU, ROCm version). It changes what the first target model and quantization should be for M2, and whether a vLLM-ROCm oracle can run on the same box for M4. I'm happy to walk you through the codebase and split the milestones into separate issues once work starts.

[VikashLoomba]

GPU(s): 4x 7900 XTX
ROCm Version: 7.14
OS: Linux

[tbrasser]

Nice timing. GTR9 Pro 128GB here on hadron/kairos/localai

Current models.yaml

name: "qwen36-35b-a3b-mtp:latest"
backend: llama-cpp
parameters:
  model: Qwen3.6-35B-A3B-Q4_K_M.gguf
  temperature: 0
draft_model: mtp-Qwen3.6-35B-A3B-Q4_0.gguf
context_size: 32768
threads: 8
f16: true
gpu_layers: 999
mmap: true
flash_attention: true
reasoning_effort: none
pinned: true
known_usecases:
  - chat
  - completion
options:
  - spec_type:draft-mtp
  - spec_n_max:3
  - spec_p_min:0.75
  - draft_gpu_layers:999
  - parallel:1
  - cache_reuse:256
template:
  use_tokenizer_template: true

(Btw I'm just taking my first steps, have been more focused on the cluster beneath until now 😅)