diff --git a/.agents/specs/main-regression-repair-2026-08-08.md b/.agents/specs/main-regression-repair-2026-08-08.md new file mode 100644 index 000000000..92ebf49e3 --- /dev/null +++ b/.agents/specs/main-regression-repair-2026-08-08.md @@ -0,0 +1,284 @@ +# Main Regression Repair Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> `superpowers:subagent-driven-development` to implement this plan task-by-task. +> Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Repair the two integration regressions introduced by merged PR #140 +without reverting its ROCm/Gemma-4 functionality, adding an exception, or +raising a baseline; then reconcile and merge release PR #141. + +**Architecture:** Gemma-4 expert GeGLU must enter the shared +`layers::MlpGateUpMethodBase` family, preserving both host-backed and already +device-resident expert layouts. Runner async eligibility must ask two portable +capabilities—whether the allocation is host-readable and whether the platform +implements the device mirror—instead of naming CUDA in shared code. + +**Tech stack:** C++17, `vt::` op/provider and Platform seams, doctest, Python +policy/mutation gates, CMake/CTest, GitHub row-PR workflow. + +## Global constraints + +- Preserve every #140 ROCm and Gemma-4 BF16/FP8/resident-expert behavior. +- The source calls are `vt::GeluAndMul`, not `SiluAndMul`; Gemma uses GeGLU. +- This is the merged-GEMM/MoE seam, not the residual-norm `vt::FusedChain` glue + seam. Do not add a decorative `FusedChain` token. +- Do not add or retain a `gemma4_moe` entry in + `scripts/merged-gemm-consistency-allowlist.txt`. +- Do not add `DSR-ALLOW`, raise `scripts/device-leakage-baseline.json`, or move + the CUDA test behind another shared-layer helper. +- Make two independently testable production commits: Gemma-4 seam repair + first, runner capability repair second. Each commit carries its tests and any + same-commit public-record projection required by the current policy. +- No performance claim is made. GPU/ROCm runtime evidence is correctness and + inertness evidence only. +- Rebase onto current `main` before implementation and again before review. + Keyed records are taken from `main` wholesale and the narrow repair clause is + reapplied; they are never three-way merged. + +--- + +## Intake and verified gap + +The intake search on 2026-08-08 found merged PR +[#140](https://github.com/mudler/vllm.cpp/pull/140), which introduced both +affected files, and open PR +[#154](https://github.com/mudler/vllm.cpp/pull/154), which overlaps +`gemma4_moe.cpp` but is a separate resident-expert performance campaign. No +open issue or row PR already claimed this repair. This plan is reserved by +draft PR #158 on row `MODEL-TEXT-gemma4-gemma4-for-causal-lm`; that existing +model row owns the Gemma-4 PLE/YOCO/MoE backbone and avoids inventing a row. + +The dispatched base is `1ce0d662`: + +- `src/vllm/model_executor/models/gemma4_moe.cpp:27-47` and `:50-74` + implement host and device expert GeGLU separately. Both end in direct + `vt::GeluAndMul` calls at lines 46 and 73 and reference none of the shared + gate-up constructs. +- `scripts/check-fusion-consistency.py:61-76,112-121,203-226` classifies those + two calls as merged-GEMM drift. `python3 scripts/check-fusion-consistency.py` + reports `gemma4_moe.cpp (2 gated-MLP epilogue hand-call site(s))`. +- `tests/scripts/test_check_fusion_consistency.py:117-192` contains the + merged-GEMM mutation contract; its shipped-tree assertion at lines 156-163 + is red with `['gemma4_moe']`. +- `include/vllm/model_executor/layers/linear.h:82-142` already provides the + shared base and GeGLU arm (`UnquantizedMlpGateUpGeluMethod`). +- `tests/vllm/model_executor/layers/test_linear_method.cpp:342-378` already + proves the owned merged-weight GeGLU method is byte-identical to + `{MatmulBT; GeluAndMul}`. +- `src/vllm/v1/worker/gpu/runner.cpp:88-111` adds + `QueueSupportsAsyncInputCombine`; line 107 names `DeviceType::kCUDA` in the + shared runner. Both constructors consume it at lines 347-348 and 388-389. +- `scripts/device-leakage-baseline.json:12-18` binds `kcuda=0`. Therefore + `python3 scripts/check-device-leakage.py` is red with `kcuda: 1 > baseline + 0`; the baseline is already correct and must not change. +- `include/vllm/platforms/interface.h:120-128` already exposes the composed + backend and host/device address-space query. `runner.cpp:2169-2193` already + implements the CUDA device mirror, but the availability is not represented + as Platform policy. +- `tests/scripts/test_device_leakage.py:155-219` proves a planted shared-layer + `kCUDA`, including one hidden behind a helper, fails the DSR. + +After dispatch, `origin/main` advanced to `1a021b1b` and added a +`gemma4_moe` known-drift allowlist entry. That makes the coarse gate green but +does not repair the execution seam and is explicitly outside this design. The +implementation starts by rebasing and removing that entry in the Gemma-4 +repair commit. PR #154's current head must be inspected again immediately +before editing because it changes the same model file. + +## File map + +| Path | Responsibility | +|---|---| +| `include/vllm/model_executor/layers/linear.h` | Shared GeGLU gate-up method; add only the minimum borrowed/resident expert-weight form needed by Gemma-4. | +| `src/vllm/model_executor/models/gemma4_moe.cpp` | Route both host-backed and device-resident expert paths through that method. | +| `tests/vllm/model_executor/layers/test_linear_method.cpp` | Red-first byte-equivalence and layout/lifetime coverage for the expert-weight form. | +| `tests/scripts/test_check_fusion_consistency.py` | Pin both Gemma-4 call sites to the real method seam so a decorative token cannot satisfy the coarse checker. | +| `scripts/merged-gemm-consistency-allowlist.txt` | Remove the temporary `gemma4_moe` exception if present after rebase. | +| `include/vllm/platforms/interface.h` | Add a default-false `supports_async_device_mirror()` policy capability. | +| `src/vllm/platforms/cuda.cpp` | Override the capability true where the mirror is implemented. | +| `src/vllm/v1/worker/gpu/runner.cpp` | Select async input combine from unified host readability or the mirror capability; contain CUDA implementation details in the existing CUDA arms. | +| `tests/vllm/platforms/test_platform.cpp` | Capability defaults and CUDA override. | +| `tests/vllm/v1/worker/test_runner.cpp` | CPU/unified-memory default-on and rollback/inertness behavior. | +| `scripts/check-device-leakage.py`, `tests/scripts/test_device_leakage.py` | Existing unchanged DSR and mutation proof; no baseline edit. | + +## Task 1: Fold Gemma-4 expert GeGLU onto the shared method family + +**Interfaces** + +- Consumes: `layers::MlpGateUpMethodBase::Apply(Dev, const vt::Tensor&) -> + DBuf`, the existing `UnquantizedMlpGateUpGeluMethod`, and contiguous Gemma-4 + expert weights `[2I,H]` / `[H,I]`. +- Produces: one shared GeGLU method form that can bind either a host-backed + expert slab (staged through `DBuf`) or an already resident `vt::Tensor`, with + both `ExpertGeGLUHost` and `ExpertGeGLUDevice` calling it. + +- [ ] **Step 1: Rebase and repeat intake.** Fetch `main`; inspect PR #154 and + all open row PRs; confirm the two direct calls and allowlist state at the new + head. Resolve any overlap before editing. + +- [ ] **Step 2: Add the red source-routing assertion.** Extend + `test_check_fusion_consistency.py` to inspect comment-stripped + `gemma4_moe.cpp` and require zero direct `vt::GeluAndMul` calls plus two real + shared-method applications in the two expert helpers. Run: + + ```sh + python3 tests/scripts/test_check_fusion_consistency.py + ``` + + Expected before production edits: FAIL naming both direct sites. The test + must still fail if either one of the two method applications is replaced by + the old direct sequence. + +- [ ] **Step 3: Add the red numerical expert-layout test.** Extend + `test_linear_method.cpp` with deterministic BF16 `[2I,H]` expert data and + compare the new host-backed and resident-view method form against the exact + pre-repair sequence: two current projections/packing operations, + `GeluAndMul`, then down projection. Assert raw BF16 bytes, shape `[M,I]`, and + method name. Run: + + ```sh + cmake --build build --target test_linear_method -j2 + ctest --test-dir build -R '^test_linear_method$' --output-on-failure + ``` + + Expected before implementation: compile failure because the expert method + form does not exist. + +- [ ] **Step 4: Implement the minimum shared method form.** Keep device choice + out of the method. A host slab is staged into the queue's backend before + binding its tensor; a resident slab is bound directly. Preserve the existing + activation order and BF16 store boundary. Do not introduce a Gemma-private + fused kernel or route GeGLU through the SwiGLU-only + `MoeGateUpSwiGLUGrouped` descriptor. + +- [ ] **Step 5: Route both production helpers.** Replace both direct + `GeluAndMul` sequences with the shared method. Remove the temporary + `gemma4_moe` allowlist entry after rebasing. Run: + + ```sh + python3 scripts/check-fusion-consistency.py + python3 tests/scripts/test_check_fusion_consistency.py + ctest --test-dir build -R '^(test_linear_method|test_gemma4_paged_engine)$' --output-on-failure + ``` + + Expected: the merged-GEMM drift list is empty; all method and available + Gemma-4 gates pass. An unavailable real-checkpoint/GPU gate remains explicitly + pending rather than being simulated. + +- [ ] **Step 6: Commit independently.** Include required keyed public-record + projections in this same commit, describing a structural correctness repair + with no speed number. + + ```text + fix(gemma4): route expert GeGLU through shared gate-up method + ``` + +## Task 2: Replace the runner's CUDA identity test with capabilities + +**Interfaces** + +- Consumes: `Platform::is_unified_memory()` and the existing CUDA + `GPUModelRunner::async_device_mirror()` implementation. +- Produces: `Platform::supports_async_device_mirror() -> bool`, default false, + CUDA true; `QueueSupportsAsyncInputCombine` returns + `is_unified_memory() || supports_async_device_mirror()`. + +The truth table is binding: + +| Platform allocation/mirror | Async input combine | +|---|---| +| CPU or any genuinely unified-memory backend | true | +| Discrete CUDA with the implemented device mirror | true | +| Discrete ROCm without a HIP mirror | false | +| A future discrete backend after it implements and advertises a mirror | true | + +- [ ] **Step 1: Add red capability tests.** In `test_platform.cpp`, assert the + base/CPU capability is false and the compiled CUDA platform capability is + true. Add a pure truth-table test for the eligibility predicate so the + discrete-ROCm case is exercised without pretending CPU memory is discrete. + +- [ ] **Step 2: Confirm the DSR is red first.** Run: + + ```sh + python3 scripts/check-device-leakage.py + python3 tests/scripts/test_device_leakage.py + ``` + + Expected before the production edit: the real-tree checker fails with + `kcuda: 1 > baseline 0`; the mutation suite itself passes. + +- [ ] **Step 3: Implement capability selection.** Add the Platform virtual and + CUDA override, then rewrite `QueueSupportsAsyncInputCombine` using only those + two capabilities. Keep CUDA headers, launches, and compile guards in the + existing CUDA implementation arms; this task removes only shared policy's + device identity test. + +- [ ] **Step 4: Prove behavior and leakage.** Run: + + ```sh + python3 scripts/check-device-leakage.py + python3 tests/scripts/test_device_leakage.py + cmake --build build --target test_platform test_runner test_loaded_engine_dense -j2 + ctest --test-dir build -R '^(test_platform|test_runner|test_loaded_engine_dense)$' --output-on-failure + ``` + + Expected: DSR remains `kcuda=0` with no baseline edit; CPU/unified async stays + default-on; the sync rollback and scheduler construction matrix remain green. + Where the authorized ROCm host is available, additionally compile the HIP + runner and run `test_rocm_backend`; a discrete board must resolve async input + combine false until a HIP mirror exists. + +- [ ] **Step 5: Commit independently.** Include required keyed public-record + projections in this same commit, with no performance claim. + + ```text + fix(runner): select async combine by platform capability + ``` + +## Task 3: Review, integrate the repair, then reconcile release PR #141 + +- [ ] **Step 1: Run focused and full gates on the repair head.** At minimum: + + ```sh + python3 scripts/check-fusion-consistency.py + python3 tests/scripts/test_check_fusion_consistency.py + python3 scripts/check-device-leakage.py + python3 tests/scripts/test_device_leakage.py + scripts/agent-preflight.sh + python3 scripts/check-doc-checkpoint.py --base origin/main --head HEAD + ``` + +- [ ] **Step 2: Dispatch a fresh mutation reviewer.** It must independently + replace each of the two Gemma-4 method applications with the old direct + activation and reintroduce the runner `kCUDA` branch. Each mutation must make + its named gate red. Findings go to a fresh implementer and then a scoped fresh + re-review. + +- [ ] **Step 3: Rebase and merge the repair PR.** Re-run the operator's full + gate on the exact reviewed head and merge in the same session when CI is green. + +- [ ] **Step 4: Reconcile #141 from the repaired `main`.** For + `docs/STATUS.md`, `docs/BENCHMARKS.md`, `docs/FEATURES.md`, `.agents/NOW.md`, + matrices, and coordination records, take repaired `main` wholesale and reapply + only #141's W5 release-manifest clauses. Union-append append-only evidence and + run `scripts/sort-state-tail.py --apply` if required. Verify unrelated repaired + main lines are byte-identical. + +- [ ] **Step 5: Review and merge #141.** Re-run its manifest suites, full + preflight, exact-head CI, and a fresh mutation review; then mark ready and merge + in the same session. This lands the W5 manifest/static-boundary contract only; + it does not claim that binary artifacts are already published. + +## Success criteria + +- Both Gemma-4 expert paths invoke a real `MlpGateUpMethodBase` family method; + neither contains a direct `vt::GeluAndMul` call. +- The structural mutation test fails when either call site is unfolded. +- `gemma4_moe` is absent from the merged-GEMM allowlist. +- The shared runner contains no code-level `kCUDA` reference for async combine. +- Async eligibility follows the four-row capability truth table and preserves + CPU, CUDA, unified-memory, and discrete-ROCm behavior. +- Device leakage stays at `kcuda=0`; no baseline or waiver changes. +- Focused tests, full preflight, exact-head CI, and independent mutation review + pass for the repair; then reconciled #141 passes its own gates and is merged. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 085d118a9..7b71d4fa0 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -352,7 +352,7 @@ built on it rather than keeping the flattering one. | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box | | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs | | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 24 NATIVE (+8 GDN, BOTH recurrences); 63 host-tier. **27B prefill 21.5x on GB10** (ragged-M reached coopmat). opt-125m e2e token-exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | -| ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM | +| ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **No new number.** W0 green on 4 gfx archs; #140 ran M0/M1 on 2× R9700; #158 repairs are CPU-green (expert GeGLU and async capability truth table). No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist | | Parakeet/FastConformer ASR (P1-P4 + ONE-SURFACE fold ROW 1) | **NO number measured, claimed or owed.** Correctness-gated only, CPU f32; the 2026-08-07 surface fold (`vllm_transcribe`, `/v1/audio/transcriptions`) is transcript-byte-identical plumbing, no speed claim. | Floor is `parakeet.cpp`, same clip and box; needs a CUDA provider and a pretrained checkpoint | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 781a9c950..d645dc9d2 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -108,7 +108,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `GemmaForCausalLM` | google/gemma-1.1-2b-it, unsloth/gemma-2b | near-tie 48/48 vs vLLM 0.25.0 | pending | | `Gemma2ForCausalLM` | google/gemma-2-2b-it | near-tie 48/48 vs vLLM 0.25.0 | pending | | `Gemma3ForCausalLM` | google/gemma-3-1b-it | strict 48/48 vs vLLM 0.25.0 | pending | -| `Gemma4ForConditionalGeneration` | Gemma-4 multimodal (unsloth/gemma-4-E4B-it) | text strict, image mm near-tie; audio pending | pending | +| `Gemma4ForConditionalGeneration` | Gemma-4 multimodal (unsloth/gemma-4-E4B-it) | text strict, image mm near-tie; #158 expert GeGLU host/resident fold CPU byte-exact and runner async eligibility capability-gated; audio pending | pending | | `Gemma4UnifiedForConditionalGeneration` | Gemma-4 "unified" HF export (google/gemma-4-12B-it), no-PLE dense layout | shares the Gemma-4 text+mm forward; loads on the same factory (contributor #140); no separate oracle gate for this arch name yet | pending | | `GraniteForCausalLM` | ibm-granite/granite-3.3-2b-instruct | near-tie 16/16 vs vLLM 0.25.0 | pending | | `StableLmForCausalLM` | stabilityai/stablelm-2-1_6b | near-tie 16/16 vs vLLM 0.25.0 | pending | @@ -205,7 +205,7 @@ the registered engine forward. | CPU (x86, Arm i8mm; A76 assembly correct/default, llama speed gate open) | ✅ | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | -| ROCm | ◐ (W0 community-verified on 4 gfx archs, #41; APU unified-memory fix landed, unverified) | ✅ | ✅ | ✅ | +| ROCm | ◐ (W0 community-verified on 4 gfx archs, #41; APU unified-memory fix landed, unverified; async combine requires unified memory or a mirror) | ✅ | ✅ | ✅ | | XPU / TPU | ☐ | ✅ | ◐ | ☐ | CUDA runtime-verified on GB10 (sm_121a), Jetson Thor (sm_110) and Jetson AGX @@ -282,7 +282,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Architecturally blocked | Vision tower lives outside the registered engine forward | | Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch | -| ROCm | W0 verified by community, model e2e pending | Backend + platform + 1 op, ctest-green on gfx1151/1103/1100/1201 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU UnifiedMemory fix in (managed allocs, unverified); M2 unblocks with it. [ROCM.md](ROCM.md) | +| ROCm | W0 verified by community, model e2e pending | Ctest-green on gfx1151/1103/1100/1201 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU managed allocations are unverified; async combine accepts unified memory, not discrete/no-mirror. [ROCM.md](ROCM.md) | | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends | | Custom logits processors on CUDA | Open, not root-caused | Segfaults in a CUDA build, 232/232 green on CPU | | Memory budgeting (`ROAD-V1-MEM`, #83) | M1+M2 landed (absolute bytes) | `--kv-cache-memory` sizes the KV pool from an absolute byte budget (ABI v16, group-aware divisor); `--num-blocks` overrides; `--gpu-memory-utilization` needs the M3 profile run (dgx-gated). See `specs/kv-sizing.md` | diff --git a/docs/STATUS.md b/docs/STATUS.md index 6fbbd7503..390dc9657 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -416,7 +416,7 @@ recurrences; 27B prefill 21.5x on GB10; qwen3_5 #125 VERIFIED; CUDA build repair [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in — **blind-written, unverified** — M2 unblocks on verification; gfx1201 hipBLAS + -Gemma-4 MoE (#140, contributor) M0/M1 on 2× R9700, CPU-link-verified our side; +Gemma-4 MoE (#140, contributor) M0/M1 on 2× R9700; #158 CPU-green; [guide](ROCM.md)), and the full tool-calling template surface. **Scale-out / distributed execution is scoped, with two legs landed CPU-gated** (2026-07-28): one `vt::` collective / process-group abstraction with backend transports (NCCL / RDMA / MLX-ring) mirrors vLLM's @@ -432,7 +432,7 @@ tp=1). The **real TP-2 GPU run + NCCL build-verify remain HW-blocked** (no parallelism mode is enumerated and ranked in [parallelism-modes](../.agents/specs/parallelism-modes.md), noting vLLM's "sequence parallel" is a TP compilation pass, not an axis). -**Shared layer names no device**: async readback is a `Backend` capability. +**Shared layer names no device**: async combine is capability-gated. **Tensor parallelism is scoped end-to-end at the pin** (#287, [tensor-parallelism-spike](../.agents/specs/tensor-parallelism-spike.md)): diff --git a/docs/USAGE.md b/docs/USAGE.md index 32c3240f3..54433801a 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -27,6 +27,14 @@ build/examples/vllm-cli \ --max-tokens 64 ``` +Gemma-4 MoE expert residency (host-backed or already device-resident) is chosen +automatically from the loaded weights; the shared GeGLU method adds no CLI or +server flag. + +Async input combine is also selected automatically from backend capabilities: +it is enabled for genuinely unified memory or a backend-provided sampled-token +mirror, and stays synchronous on a discrete backend without that mirror. + | Flag | Default | Meaning | |---|---|---| | `--model ` | (required) | Model directory (config.json + tokenizer.json + safetensors) | diff --git a/include/vllm/model_executor/layers/linear.h b/include/vllm/model_executor/layers/linear.h index b660cc2b7..c5a1a09b2 100644 --- a/include/vllm/model_executor/layers/linear.h +++ b/include/vllm/model_executor/layers/linear.h @@ -119,14 +119,38 @@ class UnquantizedMlpGateUpMethod : public MlpGateUpMethodBase { // merged-GEMM descriptor / fused-kernel scheme choice has one home across both // activation families; a future nvfp4 checkpoint gets a GeGLU quant arm the same // way the SwiGLU one gets GateUpFusedMarlinD. +// +// Gemma-4's experts borrow contiguous [2I,H] slabs rather than OwnedTensor +// objects. The host form stages that slab through DBuf on the running backend; +// the resident form binds its existing vt::Tensor view. Both then enter the +// same ApplyWeight implementation, with no device identity in the method. class UnquantizedMlpGateUpGeluMethod : public MlpGateUpMethodBase { public: UnquantizedMlpGateUpGeluMethod(const OwnedTensor* gate_up, int64_t intermediate) : gate_up_(gate_up), I_(intermediate) {} + UnquantizedMlpGateUpGeluMethod(const uint16_t* gate_up_host, int64_t intermediate, + int64_t hidden) + : gate_up_host_(gate_up_host), I_(intermediate), H_(hidden) {} + + UnquantizedMlpGateUpGeluMethod(const vt::Tensor& gate_up_resident, + int64_t intermediate) + : gate_up_resident_(gate_up_resident), I_(intermediate) {} + DBuf Apply(Dev d, const vt::Tensor& x) const override { + if (gate_up_ != nullptr) return ApplyWeight(d, x, ResidentWeight(d, *gate_up_)); + if (gate_up_host_ != nullptr) { + DBuf staged(d, vt::DType::kBF16, {2 * I_, H_}, gate_up_host_); + return ApplyWeight(d, x, staged.t()); + } + return ApplyWeight(d, x, gate_up_resident_); + } + + const char* Name() const override { return "bf16-gate-up-gelu"; } + + private: + DBuf ApplyWeight(Dev d, const vt::Tensor& x, const vt::Tensor& wgu) const { const int64_t M = x.shape[0]; - vt::Tensor wgu = ResidentWeight(d, *gate_up_); // [2I, H] raw-NK DBuf gate_up(d, vt::DType::kBF16, {M, 2 * I_}); vt::MatmulBT(d.q, gate_up.t(), x, wgu); DBuf act(d, vt::DType::kBF16, {M, I_}); @@ -134,11 +158,11 @@ class UnquantizedMlpGateUpGeluMethod : public MlpGateUpMethodBase { return act; } - const char* Name() const override { return "bf16-gate-up-gelu"; } - - private: - const OwnedTensor* gate_up_; + const OwnedTensor* gate_up_ = nullptr; + const uint16_t* gate_up_host_ = nullptr; + vt::Tensor gate_up_resident_; int64_t I_; + int64_t H_ = 0; }; } // namespace layers diff --git a/include/vllm/v1/worker/gpu/runner.h b/include/vllm/v1/worker/gpu/runner.h index ed6f9241b..6de71ed7f 100644 --- a/include/vllm/v1/worker/gpu/runner.h +++ b/include/vllm/v1/worker/gpu/runner.h @@ -83,12 +83,21 @@ #include "vt/device.h" #include "vt/tensor.h" +namespace vt { +class Backend; +} + namespace vllm::v1::kv_offload { class KVConnector; // KV-EXTERNAL-CACHE: worker-side store/load seam (fwd-decl). } // namespace vllm::v1::kv_offload namespace vllm::v1 { +// Backend policy predicate used by the queue-level async-input gate. Kept +// separate from device registration so its complete capability truth table can +// be exercised without pretending a host allocation belongs to a discrete GPU. +bool BackendSupportsAsyncInputCombine(const vt::Backend& backend); + // Decode-first reorder (utils.py::reorder_batch_to_split_decodes_and_prefills @ // e24d1b24, T0 subset). Reorders `input_batch`'s active [0, num_reqs) requests so // all requests with num_scheduled_tokens <= decode_threshold at the front, using diff --git a/include/vt/backend.h b/include/vt/backend.h index cf4ff60ee..3a3b78aa0 100644 --- a/include/vt/backend.h +++ b/include/vt/backend.h @@ -127,17 +127,14 @@ class Backend { // non-CUDA GPU (e.g. ROCm gfx1201) is the hazard: the non-CUDA leg of // sample_tokens_async Synchronizes and then host-dereferences `dev_ids`, a // device Alloc that is garbage off-device (the "!"-token corruption on the lab - // R9700, 2026-08-07), so those queues MUST stay synchronous. Overridden true - // by CPU (host and device memory are the same allocation, so the read is - // always valid) and by CUDA (the sampled id is device-mirrored, - // async_device_mirror()). This is the capability the runner's - // `device == kCUDA` gate actually asked; it lives on Backend (src/vt, off the - // DSR scan) so the device-agnostic shared layer stops naming a device — the - // same move SupportsAuxStream made for the aux-stream gate. - // TODO(rocm): an INTEGRATED non-CUDA GPU reports UnifiedMemory()==true (see - // row/ROCM-UNIFIED-MEMORY-B), where the alias IS valid; such a backend may - // override this true once a HIP sampled-token mirror or a D2H copy of dev_ids - // lands. + // R9700, 2026-08-07), so those queues MUST stay synchronous. CUDA overrides + // true because the sampled id is device-mirrored (async_device_mirror()); CPU + // retains its #159 true override, though the runner also recognizes every + // genuinely unified-memory backend directly through UnifiedMemory(). This is + // the capability the runner's `device == kCUDA` gate actually asked; it lives + // on Backend (src/vt, off the DSR scan) so the device-agnostic shared layer + // stops naming a device — the same move SupportsAuxStream made for the + // aux-stream gate. virtual bool SupportsAsyncSampledTokenReadback() const { return false; } // Optional graph/command capture (CUDA Graphs / Metal ICB / Vulkan CB). diff --git a/scripts/merged-gemm-consistency-allowlist.txt b/scripts/merged-gemm-consistency-allowlist.txt index 2440573c6..bbfc62ceb 100644 --- a/scripts/merged-gemm-consistency-allowlist.txt +++ b/scripts/merged-gemm-consistency-allowlist.txt @@ -30,7 +30,6 @@ minicpm # SwiGLU dense MLP -> UnquantizedMlpGateUpMethod; pending FOLD-M minicpm3 # SwiGLU dense MLP (MLA arch) -> UnquantizedMlpGateUpMethod; pending FOLD-MIGRATE phi3 # SwiGLU dense MLP -> UnquantizedMlpGateUpMethod (also on the glue allowlist); pending FOLD-MIGRATE gemma4_vision # GeGLU vision-tower MLP -> UnquantizedMlpGateUpGeluMethod + a clamp-epilogue hook (Tier C2); pending FOLD-MIGRATE -gemma4_moe # known-drift pending fold: PLE shared expert and routed experts hand-call GeGLU; fold onto the shared merged-GEMM/MoE gate-up family before claiming seam closure laguna # known-drift pending fold: Laguna NVFP4 resident/graph decode hand-rolls the shared-expert SwiGLU epilogue; fold onto layers::MlpGateUp seam is part of the decode/runtime framework-routing port (AGENTS.md 3rd seam) diff --git a/src/vllm/model_executor/models/gemma4_moe.cpp b/src/vllm/model_executor/models/gemma4_moe.cpp index 7aa34ea8d..30169aa8a 100644 --- a/src/vllm/model_executor/models/gemma4_moe.cpp +++ b/src/vllm/model_executor/models/gemma4_moe.cpp @@ -9,6 +9,7 @@ #include #include "vllm/model_executor/model_loader/nvfp4_dequant.h" +#include "vllm/model_executor/layers/linear.h" #include "vllm/model_executor/models/dense_attn_block.h" #include "vllm/model_executor/models/device_pool.h" #include "vt/backend.h" @@ -26,51 +27,19 @@ using vt::Tensor; void ExpertGeGLUHost(Dev d, DBuf& out, const Tensor& x, const uint16_t* gate_up_e, const uint16_t* down_e, int64_t I, int64_t H) { - const int64_t T = x.shape[0]; - DBuf gate_w(d, DType::kBF16, {I, H}, gate_up_e); - DBuf up_w(d, DType::kBF16, {I, H}, gate_up_e + I * H); DBuf down_w(d, DType::kBF16, {H, I}, down_e); - DBuf gate(d, DType::kBF16, {T, I}); - DBuf up(d, DType::kBF16, {T, I}); - vt::MatmulBT(d.q, gate.t(), x, gate_w.t()); - vt::MatmulBT(d.q, up.t(), x, up_w.t()); - DBuf gu(d, DType::kBF16, {T, 2 * I}); - const size_t row = static_cast(I) * sizeof(uint16_t); - for (int64_t t = 0; t < T; ++t) { - d.b.Copy(d.q, static_cast(gu.ptr()) + static_cast(t) * 2 * row, - static_cast(gate.ptr()) + static_cast(t) * row, row); - d.b.Copy(d.q, static_cast(gu.ptr()) + static_cast(t) * 2 * row + row, - static_cast(up.ptr()) + static_cast(t) * row, row); - } - DBuf act(d, DType::kBF16, {T, I}); - vt::GeluAndMul(d.q, act.t(), gu.t()); + DBuf act = layers::UnquantizedMlpGateUpGeluMethod(gate_up_e, I, H).Apply(d, x); vt::MatmulBT(d.q, out.t(), act.t(), down_w.t()); } void ExpertGeGLUDevice(Dev d, DBuf& out, const Tensor& x, const uint16_t* gate_up_e, const uint16_t* down_e, int64_t I, int64_t H) { - const int64_t T = x.shape[0]; const vt::Device dev = d.q.device; - Tensor gate_w = - Tensor::Contiguous(const_cast(gate_up_e), DType::kBF16, dev, {I, H}); - Tensor up_w = Tensor::Contiguous(const_cast(gate_up_e + I * H), DType::kBF16, - dev, {I, H}); + Tensor gate_up_w = Tensor::Contiguous(const_cast(gate_up_e), DType::kBF16, + dev, {2 * I, H}); Tensor down_w = Tensor::Contiguous(const_cast(down_e), DType::kBF16, dev, {H, I}); - DBuf gate(d, DType::kBF16, {T, I}); - DBuf up(d, DType::kBF16, {T, I}); - vt::MatmulBT(d.q, gate.t(), x, gate_w); - vt::MatmulBT(d.q, up.t(), x, up_w); - DBuf gu(d, DType::kBF16, {T, 2 * I}); - const size_t row = static_cast(I) * sizeof(uint16_t); - for (int64_t t = 0; t < T; ++t) { - d.b.Copy(d.q, static_cast(gu.ptr()) + static_cast(t) * 2 * row, - static_cast(gate.ptr()) + static_cast(t) * row, row); - d.b.Copy(d.q, static_cast(gu.ptr()) + static_cast(t) * 2 * row + row, - static_cast(up.ptr()) + static_cast(t) * row, row); - } - DBuf act(d, DType::kBF16, {T, I}); - vt::GeluAndMul(d.q, act.t(), gu.t()); + DBuf act = layers::UnquantizedMlpGateUpGeluMethod(gate_up_w, I).Apply(d, x); vt::MatmulBT(d.q, out.t(), act.t(), down_w); } diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 4bf531bdb..e3f0fcad7 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -86,19 +86,20 @@ static bool AsyncRunnerEnvDefault() { } // Async input-combine reads the sampled token id back on the host between -// steps. Whether that read is VALID is a backend CAPABILITY, not a device name: -// ask the backend (vt::Backend::SupportsAsyncSampledTokenReadback, backend.h), -// which answers true for CPU (host and device memory are one allocation) and -// CUDA (the sampled id is device-mirrored, async_device_mirror()), and false for -// a DISCRETE non-CUDA GPU (e.g. ROCm gfx1201) whose sample_tokens_async leg -// host-dereferences a device Alloc — the root cause of the "!" tokens on the lab -// R9700 (2026-08-07). An absent backend (device not built into this binary) -// yields nullptr and therefore false, which also subsumes the old -// #ifdef VLLM_CPP_CUDA guard. Keeping the question on the backend is what stops -// this device-agnostic shared layer from naming a device (check-device-leakage). +// steps. Whether that read is VALID is a backend capability, not a device name: +// unified memory is directly host-readable; otherwise the backend must advertise +// a sampled-token mirror (CUDA's async_device_mirror()). A discrete non-CUDA GPU +// without a mirror remains synchronous. An absent backend yields false, which +// also subsumes the old #ifdef VLLM_CPP_CUDA guard. Keeping both questions on the +// backend stops this shared layer from naming a device (check-device-leakage). +bool BackendSupportsAsyncInputCombine(const vt::Backend& backend) { + return backend.UnifiedMemory() || + backend.SupportsAsyncSampledTokenReadback(); +} + static bool QueueSupportsAsyncInputCombine(const vt::Queue& queue) { const vt::Backend* backend = vt::TryGetBackend(queue.device.type); - return backend != nullptr && backend->SupportsAsyncSampledTokenReadback(); + return backend != nullptr && BackendSupportsAsyncInputCombine(*backend); } // GDN step-geometry diagnostic (default OFF). When VT_GDN_DIAG_STEP_LOG=1, each diff --git a/tests/scripts/test_check_fusion_consistency.py b/tests/scripts/test_check_fusion_consistency.py index f554fbf1d..59b821204 100644 --- a/tests/scripts/test_check_fusion_consistency.py +++ b/tests/scripts/test_check_fusion_consistency.py @@ -4,6 +4,7 @@ from __future__ import annotations import importlib.util +import re import sys import unittest from pathlib import Path @@ -21,6 +22,26 @@ gemm_merge_drift_models = mod.gemm_merge_drift_models +def _strip_cpp_comments(text: str) -> str: + return re.sub(r"//[^\n]*|/\*.*?\*/", "", text, flags=re.DOTALL) + + +def _cpp_function_body(text: str, name: str) -> str: + match = re.search(rf"\b{name}\s*\([^;]*\)\s*\{{", text) + if match is None: + raise AssertionError(f"missing C++ function {name}") + start = match.end() - 1 + depth = 0 + for pos in range(start, len(text)): + if text[pos] == "{": + depth += 1 + elif text[pos] == "}": + depth -= 1 + if depth == 0: + return text[start + 1 : pos] + raise AssertionError(f"unterminated C++ function {name}") + + class DriftModelTests(unittest.TestCase): def test_adopted_model_passes(self) -> None: # residual sites present, but the file references FusedChain => adopted. @@ -162,6 +183,27 @@ def test_shipped_tree_is_green(self) -> None: ) self.assertEqual(gemm_merge_drift_models(scanned, allowlisted), []) + def test_gemma4_expert_helpers_apply_the_shared_geglu_method(self) -> None: + source = _strip_cpp_comments( + (ROOT / "src/vllm/model_executor/models/gemma4_moe.cpp").read_text( + encoding="utf-8" + ) + ) + failures: list[str] = [] + apply_pattern = re.compile( + r"layers::UnquantizedMlpGateUpGeluMethod\s*\([^;]*\)\s*\.Apply\s*\(" + ) + for helper in ("ExpertGeGLUHost", "ExpertGeGLUDevice"): + body = _cpp_function_body(source, helper) + direct_calls = len(re.findall(r"\bvt::GeluAndMul\s*\(", body)) + if direct_calls: + failures.append(f"{helper}: {direct_calls} direct vt::GeluAndMul call(s)") + if len(apply_pattern.findall(body)) != 1: + failures.append( + f"{helper}: expected exactly one shared GeGLU method application" + ) + self.assertEqual(failures, [], "; ".join(failures)) + def test_a_new_unfolded_model_would_fail(self) -> None: scanned = dict(mod.scan_models_gemm(ROOT / "src/vllm/model_executor/models")) scanned["brand_new_mlp_arch"] = (2, False) diff --git a/tests/vllm/model_executor/layers/test_linear_method.cpp b/tests/vllm/model_executor/layers/test_linear_method.cpp index c7f99914c..6dad26aa3 100644 --- a/tests/vllm/model_executor/layers/test_linear_method.cpp +++ b/tests/vllm/model_executor/layers/test_linear_method.cpp @@ -377,6 +377,89 @@ TEST_CASE("linear_method: fused GeGLU gate-up seam == standalone MatmulBT+GeluAn CHECK(got[i] == ref[i]); // BYTE-IDENTICAL — the fold changes nothing numerically } +TEST_CASE("linear_method: expert GeGLU host slab and resident view preserve split reference bytes") { + const int64_t M = 3, H = 8, I = 5; + OwnedTensor gate_up = MakeBf16({2 * I, H}, 23); + OwnedTensor down = MakeBf16({H, I}, 29); + OwnedTensor xw = MakeBf16({M, H}, 31); + const auto* gate_up_data = + reinterpret_cast(gate_up.bytes.data()); + const auto* down_data = reinterpret_cast(down.bytes.data()); + + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + vt::Backend& b = vt::GetBackend(vt::DeviceType::kCPU); + vllm::dense_attn::Dev d{b, q}; + vllm::dense_attn::DBuf x(d, DType::kBF16, {M, H}, xw.bytes.data()); + + // Pre-repair reference: stage the two I-wide projections independently, + // concatenate their bf16 rows, apply GeGLU, then project down. + vllm::dense_attn::DBuf gate_w(d, DType::kBF16, {I, H}, gate_up_data); + vllm::dense_attn::DBuf up_w(d, DType::kBF16, {I, H}, gate_up_data + I * H); + vllm::dense_attn::DBuf down_w(d, DType::kBF16, {H, I}, down_data); + vllm::dense_attn::DBuf gate(d, DType::kBF16, {M, I}); + vllm::dense_attn::DBuf up(d, DType::kBF16, {M, I}); + vt::MatmulBT(d.q, gate.t(), x.t(), gate_w.t()); + vt::MatmulBT(d.q, up.t(), x.t(), up_w.t()); + vllm::dense_attn::DBuf packed(d, DType::kBF16, {M, 2 * I}); + const size_t row_bytes = static_cast(I) * sizeof(uint16_t); + for (int64_t m = 0; m < M; ++m) { + d.b.Copy(d.q, + static_cast(packed.ptr()) + static_cast(m) * 2 * row_bytes, + static_cast(gate.ptr()) + static_cast(m) * row_bytes, + row_bytes); + d.b.Copy(d.q, + static_cast(packed.ptr()) + static_cast(m) * 2 * row_bytes + + row_bytes, + static_cast(up.ptr()) + static_cast(m) * row_bytes, + row_bytes); + } + vllm::dense_attn::DBuf act_ref(d, DType::kBF16, {M, I}); + vt::GeluAndMul(d.q, act_ref.t(), packed.t()); + vllm::dense_attn::DBuf out_ref(d, DType::kBF16, {M, H}); + vt::MatmulBT(d.q, out_ref.t(), act_ref.t(), down_w.t()); + + layers::UnquantizedMlpGateUpGeluMethod host_method(gate_up_data, I, H); + REQUIRE(std::string(host_method.Name()) == "bf16-gate-up-gelu"); + vllm::dense_attn::DBuf act_host = host_method.Apply(d, x.t()); + CHECK(act_host.t().rank == 2); + CHECK(act_host.t().shape[0] == M); + CHECK(act_host.t().shape[1] == I); + vllm::dense_attn::DBuf down_host(d, DType::kBF16, {H, I}, down_data); + vllm::dense_attn::DBuf out_host(d, DType::kBF16, {M, H}); + vt::MatmulBT(d.q, out_host.t(), act_host.t(), down_host.t()); + + vt::Tensor gate_up_resident = vt::Tensor::Contiguous( + const_cast(gate_up_data), DType::kBF16, q.device, {2 * I, H}); + vt::Tensor down_resident = vt::Tensor::Contiguous( + const_cast(down_data), DType::kBF16, q.device, {H, I}); + layers::UnquantizedMlpGateUpGeluMethod resident_method(gate_up_resident, I); + REQUIRE(std::string(resident_method.Name()) == "bf16-gate-up-gelu"); + vllm::dense_attn::DBuf act_resident = resident_method.Apply(d, x.t()); + CHECK(act_resident.t().rank == 2); + CHECK(act_resident.t().shape[0] == M); + CHECK(act_resident.t().shape[1] == I); + vllm::dense_attn::DBuf out_resident(d, DType::kBF16, {M, H}); + vt::MatmulBT(d.q, out_resident.t(), act_resident.t(), down_resident); + + std::vector act_expected(static_cast(M) * I); + std::vector act_host_bytes(act_expected.size()); + std::vector act_resident_bytes(act_expected.size()); + act_ref.Download(d, act_expected.data()); + act_host.Download(d, act_host_bytes.data()); + act_resident.Download(d, act_resident_bytes.data()); + CHECK(act_host_bytes == act_expected); + CHECK(act_resident_bytes == act_expected); + + std::vector out_expected(static_cast(M) * H); + std::vector out_host_bytes(out_expected.size()); + std::vector out_resident_bytes(out_expected.size()); + out_ref.Download(d, out_expected.data()); + out_host.Download(d, out_host_bytes.data()); + out_resident.Download(d, out_resident_bytes.data()); + CHECK(out_host_bytes == out_expected); + CHECK(out_resident_bytes == out_expected); +} + TEST_CASE("linear_method: bf16 UnquantizedLinearMethod apply == reference MatmulBT") { const int64_t M = 2, K = 16, N = 4; OwnedTensor w = MakeBf16({N, K}, 7); // raw-NK [N=out, K=in] diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index c34713eb4..45fd97e07 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -187,6 +187,49 @@ constexpr int kBlockSize = 8; constexpr int kMaxModelLen = 32; constexpr int kNumBlocks = 8; +class AsyncInputCombineBackend final : public vt::Backend { + public: + AsyncInputCombineBackend(bool unified_memory, bool sampled_token_mirror) + : unified_memory_(unified_memory), + sampled_token_mirror_(sampled_token_mirror) {} + + void* Alloc(size_t) override { return nullptr; } + void Free(void*) override {} + void Memset(vt::Queue&, void*, int, size_t) override {} + void Copy(vt::Queue&, void*, const void*, size_t) override {} + vt::Queue CreateQueue() override { return {}; } + bool UnifiedMemory() const override { return unified_memory_; } + bool SupportsAsyncSampledTokenReadback() const override { + return sampled_token_mirror_; + } + + private: + bool unified_memory_; + bool sampled_token_mirror_; +}; + +TEST_CASE("async input combine follows backend readability capabilities") { + struct Case { + const char* name; + bool unified_memory; + bool sampled_token_mirror; + bool want; + }; + const Case cases[] = { + {"unified memory without a mirror", true, false, true}, + {"discrete memory with a mirror", false, true, true}, + {"discrete memory without a mirror", false, false, false}, + {"unified memory with a mirror", true, true, true}, + }; + + for (const Case& tc : cases) { + CAPTURE(std::string(tc.name)); + const AsyncInputCombineBackend backend(tc.unified_memory, + tc.sampled_token_mirror); + CHECK(vllm::v1::BackendSupportsAsyncInputCombine(backend) == tc.want); + } +} + // A fake KVCacheConfig with the gate group structure: one full-attn group + one // mamba (GDN) group, sharing kNumBlocks blocks. KVCacheConfig MakeKvConfig(const HfConfig& c, diff --git a/tests/vt/test_backend.cpp b/tests/vt/test_backend.cpp index af0152ccc..b66e63d01 100644 --- a/tests/vt/test_backend.cpp +++ b/tests/vt/test_backend.cpp @@ -15,6 +15,7 @@ TEST_CASE("CPU backend is registered and allocates usable memory") { Backend& cpu = vt::GetBackend(DeviceType::kCPU); CHECK(cpu.UnifiedMemory()); CHECK_FALSE(cpu.SupportsGraphCapture()); + CHECK(cpu.SupportsAsyncSampledTokenReadback()); // S7: a single-stream backend declines the MoE shared-expert aux-stream // overlap — base false, exactly what the old `device==kCUDA` gate returned on // CPU (the shared path runs serially, byte-identical, no overlap).