diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 7a1fa0ba3..65663ce65 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -245,7 +245,8 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-ANE` | Apple Neural Engine for encoder/pooling/fixed-shape draft classes | vllm.cpp extension through upstream seam `platforms/interface.py:134-229`; not a paged decode backend | Platform seam anchored [interface.h:56](../include/vllm/platforms/interface.h#L56) (a `Platform` subclass; not a paged-decode backend) | - | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | - +| `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | Child of `BACKEND-TENSTORRENT` — make the per-decode-layer forward region host-free (zero `to_vector` readbacks) so mesh-trace capture can run. Decomposes into R1 RmsNorm+RoPE all-device, R2 QkvSplit+ReshapeAndCache device, R3 PA decode device-resident metadata, R4 capture wire+measure. Prerequisite revealed by the trace-runner spike (capture aborts on `to_vector`). | CUDA decode-graph capture contract (`cuda_backend.cu:184-197`): captured region is async, no host sync, no malloc, fixed ptrs. TT must match: no `to_vector` between Begin/EndCapture | `src/vt/tenstorrent/tenstorrent_ops.cpp` (RmsNorm:1067, PreferDeviceRope:1344, QkvSplit:1460, ReshapeAndCache:1527, PagedAttention:2009) | `-` (plan spec only; no code yet). Per-row gates: op-level bit-exact/near-tie vs host + e2e TT golden. Integration gate (R4): capture no-TT_FATAL + replay max_abs=0 + replay tok/s ≥ 12.5 (Qwen3-0.6B) | [tenstorrent-host-free-forward.md](specs/tenstorrent-host-free-forward.md) | `SPIKE` (plan committed; R1-R3 parallel-claimable) | `CLAIM-BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | +| `BACKEND-TENSTORRENT-TRACE-RUNNER` | Child of `BACKEND-TENSTORRENT` — wire the landed graph-capture foundation (#354 / `59568772`) into a capturable forward region. Handoff §8/§9 "not done". **Decision (2026-08-13): NO-GO for pure T=1 decode capture — MEASURED, not assumed.** | CUDA is the ONLY backend with `SupportsGraphCapture()==true` (`cuda_backend.cu:184-240`); Metal/Vulkan `false`. Shared decode-graph framework `Qwen3DenseDecodeGraph` (qwen3.cpp:489, used by Qwen3/Mistral/Llama/InternLM2) gated on `support_static_graph_mode()` which TT does NOT override (base `false`) | `src/vt/tenstorrent/tenstorrent_backend.cpp:70-76` (capture surface, landed) + `tenstorrent_ops.cpp` `Trace*` (landed); NOT wired into a dense forward | **Measured on Blackhole P150 (2026-08-13):** Q1 no host-free region at T=1; Q2 all-device-at-T=1 costs 12.5→10.7 tok/s; Q2b capture attempt with both overrides flipped → ttnn **`TT_FATAL: Reads are not supported during trace capture`** (backtrace through `to_vector`), `0 replays`. The T=1 forward does device→host readbacks that ttnn trace prohibits; flipping the two thresholds is insufficient. Requires the host-free forward redesign ([tenstorrent-host-free-forward.md](specs/tenstorrent-host-free-forward.md)), not a threshold tweak | [tenstorrent-trace-runner.md](specs/tenstorrent-trace-runner.md) | `SPIKE` (decision record complete; unblocks into the host-free-forward plan) | `CLAIM-BACKEND-TENSTORRENT-TRACE-RUNNER` | ## Native competitor and performance gates Every run records the competitor commit/release, model artifact hash, build diff --git a/.agents/coordination.md b/.agents/coordination.md index 3dc1288fa..73200619d 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1993,6 +1993,8 @@ this claim will meet. The tiled row is speed-gatable on dgx. | `CLAIM-ENG-LOAD-DIRECT-UPLOAD` | `ENG-LOAD-DIRECT-UPLOAD` (engine-matrix, NEW row -> `ACTIVE`) | Claude Code (opus-5) | isolated worktree `.claude/worktrees/agent-a122a5ddc3546bf31`; GB10 `dgx.casa` tree `~/work/load150/src` (Vulkan `build-vk` + CUDA `build-cuda`), every GPU stage under `flock $HOME/gpu.lock`, `local-ai-worker` parked and restored | branch `row/ENG-LOAD-DIRECT-UPLOAD`, base `origin/main` `375a471e` | Issue #150. Owns ONLY: the refcounted safetensors mapping + byte counters (`include/vllm/model_executor/model_loader/safetensors_reader.h`, `src/vllm/model_executor/model_loader/safetensors_reader.cpp`), `BorrowStTensorBytes` + the `AdoptDeviceBytesAsHost` mmap branch + `OwnedTensor::mmap_src` (`include/vllm/model_executor/models/qwen3_5_weights.h`, `src/vllm/model_executor/models/qwen3_5_weights.cpp`), the qualifying call sites in `include/vllm/model_executor/models/dense_weight_loaders.h` and `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, the upload counter in `include/vllm/model_executor/models/dense_attn_block.h`, the `VT_LOAD_STATS` phase timing in `src/vllm/entrypoints/model_loader.cpp`, NEW `tests/vllm/test_load_direct_upload.cpp` + its `tests/CMakeLists.txt` line, NEW `.agents/specs/load-direct-upload.md`, the NEW engine-matrix row + its Loading/Total rollup, the `#150` roadmap intake cell, two `docs/ENVIRONMENT.md` rows, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** no new kernel, no model registry change, no CMake target change beyond one test line; every non-verbatim loader helper is untouched and the lever is same-binary A/B-able (`VT_LOAD_DIRECT_UPLOAD=0`). MUST NOT touch README, Metal/SACRED/apex/darwin, or any model/kernel source | `ACTIVE` | 2026-08-09 — mechanism test 6/6 (77) and RED under two mutations; GB10 Vulkan gates on the changed tree green (`test_vulkan_backend` 35/35·2650, `test_backend_cross_device` 11/11·132, `test_opt_paged_engine` 6/6 prompts token-exact 96/96, 0 declines, device type 3); GB10 CUDA full `ctest` 383/393 with BOTH SACRED gates PASS and all 10 failures reproduced on a clean `origin/main` build; 27B bf16 load 1.54x warm / 1.61x cold, bytes moved 100.196 -> 81.260 GiB; `local-ai-worker` restored (`--restart=always`, Up). Fresh scoped review OWED | | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | `ENG-RELEASE-CONTAINERS` (`INVENTORIED`->`READY`->`ACTIVE`; spike spec + W1-W5/W7) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-containers`; local Docker only -- NO GPU, NO registry push, NO publication | `row/ENG-RELEASE-CONTAINERS`, base `origin/main` `24306364`, PR [#307](https://github.com/mudler/vllm.cpp/pull/307) | Container spike spec, `docker/Dockerfile` (cpu/vulkan/cuda calling the release scripts), container matrix + checker, image validator, workflow + least-privilege guard, and the SIGTERM fix the boot gate surfaced (#312). Excludes W6 matching-hardware runtime evidence and any GHCR publication | `ACTIVE` | 2026-08-10 -- cpu lane built and gated e2e (config+layout+boot, clean SIGTERM 0.25 s, RED 137 before the fix); mutation suites 31/31 and 22/22; cuda/vulkan gated statically but never built here; nothing published | | `CLAIM-ROCM-GDN-KERNELS` | `BACKEND-ROCM` (stays `ACTIVE` — M3 GDN kernel slice, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-gdn-kernels`, base `origin/main` `5812b8b6`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock` | `row/ROCM-GDN-KERNELS` | The M3 GDN kernel family unblocking Qwen3.5-0.8B M2 on discrete ROCm. Owns ONLY: NEW `.agents/specs/rocm-gdn-kernels.md`; NEW `src/vt/rocm/rocm_gdn.hip` (10 ops: kGdnStateGather/Scatter 77/78, kCausalConv1dFwd/Update 5/6, kRmsNormGated 9, kSigmoidGateBf16 58, kGdnPrefill/Decode 10/11, kGdnPostConv 65, kAttnQkNormRopeGate 67 — hand-translated from `src/vt/cuda/cuda_gdn.cu`/`cuda_ops.cu` donors); ADDITIVE registration entries in `src/vt/rocm/rocm_ops.hip`; ADDITIVE GDN cases + buffer helpers in `tests/vt/test_backend_cross_device.cpp`; this claim + `.agents/NOW.md`. **NON-COLLISION:** every new file is ROCm-only; the two shared-file edits are additive registration/case blocks that change no existing behavior (unregistered-op backends skip; CUDA/CPU/Vulkan paths byte-identical). Disjoint from #234 (sampling), #317 (gemma4 FP8/WMMA), #273 (near-tie docs). | `ACTIVE` | 2026-08-11 — **ALL 5 FAMILIES LANDED AS STACKED DRAFT PRs + M2 E2E COMPLETE.** #334 (spec + state I/O), #336 (conv), #341 (postconv), #343 (recurrence), #345 (norm-gate/preamble + the SupportsCompressedGdnState capability seam). Qwen3.5-0.8B runs e2e all-native on discrete gfx1100, deterministic; 'The sky is' 8/8 CPU parity; open-prompt divergence characterized as not-state-rounding, GDN-path drift analysis named open. Gates per family: cross-device green (162/162 total), focused 3/3, full-suite zero-delta vs base. NEXT: fresh review + operator gate per PR; then the MoeRouterTopK bf16-logits arm (the named MoE-path blocker) and the GDN-path divergence characterization. | +| `CLAIM-BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-HOST-FREE-FORWARD` (planning; reuses the trace-runner worktree) | The prerequisite for decode mesh-trace capture, revealed by `BACKEND-TENSTORRENT-TRACE-RUNNER` (capture aborts on `to_vector`). Decomposes into R1 (RmsNorm+RoPE all-device; numerics already proven by RESIDUAL-GOLDEN), R2 (QkvSplit+ReshapeAndCache device, reusing the landed paged-KV-shadow path), R3 (PA decode with device-resident metadata — the scope risk), R4 (flip support_static_graph_mode + wire capture + measure replay tok/s ≥ 12.5). R1-R3 are parallel-claimable; R4 integrates. Owns ONLY: NEW `.agents/specs/tenstorrent-host-free-forward.md`, the NEW matrix row, this claim. No code in this commit — plan only. | `SPIKE` | 2026-08-13 plan committed; no implementation yet. R1-R3 parallel-claimable once a maintainer accepts the decomposition. | +| `CLAIM-BACKEND-TENSTORRENT-TRACE-RUNNER` | `BACKEND-TENSTORRENT-TRACE-RUNNER` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-TRACE-RUNNER`, base `origin/main` `a89b3c45` | Feasibility spike (read-only analysis + on-card measurement) for wiring the landed #354 graph-capture foundation into a capturable Qwen3-dense/Mistral forward region on TT. Three questions: (1) is there a host-free decode region at T=1 (current hybrid thresholds route RoPE+residual to host)? (2) what does forcing all-device-at-T=1 cost in tok/s, and does capture recover it? (3) does ttnn program-cache warm-up work before begin_trace_capture? Surveys how CUDA/Metal/Vulkan structure capture (CUDA is the only `SupportsGraphCapture()==true` backend; Metal/Vulkan `false`). Owns ONLY: NEW `.agents/specs/tenstorrent-trace-runner.md`, the NEW `BACKEND-TENSTORRENT-TRACE-RUNNER` row, and this claim. No code, no model change, no perf claim — decision record only. | `SPIKE` | 2026-08-12 spike committed; no measurement yet. Likely outcome: no-go for pure T=1 decode (hybrid thresholds exist because all-device lost ~1.8 tok/s), go for prefill/multi-token where thresholds already go all-device. | **CLOSED same-session claim — `CLAIM-TP-SPIKE-287` (task #287, 2026-08-08, records-only, helper `row/SPIKE-TENSOR-PARALLELISM` draft PR #143, base `b38f78a7`).** Not an active-claims row because every referenced row keeps its diff --git a/.agents/specs/tenstorrent-host-free-forward.md b/.agents/specs/tenstorrent-host-free-forward.md new file mode 100644 index 000000000..525e8a335 --- /dev/null +++ b/.agents/specs/tenstorrent-host-free-forward.md @@ -0,0 +1,146 @@ +# Tenstorrent host-free decode forward — plan + +Status: **DRAFT plan, 2026-08-13.** The prerequisite for decode mesh-trace +capture (see `tenstorrent-trace-runner.md`: capture aborts on `to_vector` +readbacks inside `ForwardLayers`). This document decomposes the work into +independent rows sized for parallel claims. + +## Goal + +Make the per-decode-layer region of the TT forward **host-free**: zero +`to_vector` / `EnsureHost` readbacks between `BeginCapture` and +`EndCaptureGraph`. Only then can `Qwen3DenseDecodeGraph` capture/replay it +on `kTENSTORRENT` (ttnn `begin_trace_capture` prohibits any host read). + +The per-layer op sequence (Qwen3-dense / Mistral, from +`dense_attn_block.h`) and its host-readback status at T=1 today: + +| per-layer op | status today (T=1) | in captured region? | +|---|---|---| +| `RmsNorm` (pre-attn q-norm + residual merges) | HOST (rows<32) | yes | +| `MatmulBT` (qkv / o_proj / mlp) | device | fine | +| `QkvSplit` | **pure host** | yes | +| `RmsNorm` (qk-norm, Qwen3 only) | HOST | yes | +| `RopeNeox` / `RopeFromCache` | HOST (T·H<64) + `BuildCosSinFromPositions` host | yes | +| `ReshapeAndCache` | **pure host** | yes | +| `PagedAttention` | **pure host (host oracle)** | yes | +| `SiluAndMul` | device | fine | + +Boundary ops OUTSIDE the layer loop (readbacks here are fine — they are the +capture region's input/output edges): `Embedding` (host-staged upload), +`GreedyArgmax` (host readback of the final logits). + +## Three independent sub-problems (rows) + +Each is independently gateable; none blocks another except the capture row, +which wants all three. + +### R1 — Device-resident RmsNorm + RoPE at T=1 (threshold flip + perf) + +**Problem:** the hybrid thresholds route `RmsNorm` (rows<32) and `RopeNeox` +(T·H<64) to host at T=1. The trace-runner spike measured the perf cost of +flipping them all-device: 12.5 → 10.7 tok/s (~14%, reproduces handoff §6). +Capture must recover that. + +**Work:** flip the thresholds to all-device when capture is active (or +unconditionally, gated on `support_static_graph_mode()`), accept the ~1.8 +tok/s eager regression, and let capture claw it back. The numerics were +already proven acceptable by `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` +(device bf16 vs CPU f32 = constant 0.0459 abs, ordinary rounding). + +**Sub-blocker:** `RopeNeox`/`RopeFromCache` depend on `BuildCosSinFromPositions`, +which reads `pos` on host (line 1291) and builds cos/sin host-side. The +device RoPE apply path exists (`RopeApplyDeviceNeox`) but the cos/sin +construction is still host. Needs a device-resident cos/sin path OR a +precomputed cos/sin cache uploaded once (the `RopeCosSinCacheKernel` path +already exists for the cache mode — route through it). + +**Gate:** op-level `RmsNorm`/`Rope` device parity (already measured); e2e +Qwen3/Mistral gate token-exact or near-tie vs the TT golden. + +### R2 — Device-resident QkvSplit + ReshapeAndCache (small host-staged ops) + +**Problem:** `QkvSplit` and `ReshapeAndCache` are pure host today — they +read every input via `EnsureHost` and `CommitHost` the output. Both are +bit-exact memcpy/stride ops that went host-staged in W0 because Alloc was +host memory. Inside a captured region they must stay on device. + +**Work:** add device-resident variants using `ttnn::slice` (QkvSplit) and +the device paged-write path that already exists for paged KV +(`NotePagedKvRacWrites` / `TryDevicePagedFill` / `TryDevicePagedUpdate` — +landed with residency). The device paged-write path already keeps a ttnn KV +shadow; wire `ReshapeAndCache` to it unconditionally when capture is active. + +**Gate:** op-level bit-exactness vs the host path (these are deterministic +copies — byte-identical is achievable and required); e2e gate. + +### R3 — Device-resident PagedAttention decode (the big one) + +**Problem:** `PagedAttention` at T=1 decode runs the **host f32 oracle** +(`PagedAttentionKernel` host path). The device path +(`TryPagedAttentionDeviceDecode`, `paged_scaled_dot_product_attention_decode`) +exists and is used when the KV shadow is current, but it still reads +`block_table`/`seq_lens`/`query_start_loc` on host (lines 1644-1646) and +reads `query` host (line 1707) before the device call. Those metadata +reads are the capture blocker. + +**Work:** keep the metadata tensors device-resident across the decode step +(they are small int32 tensors; upload once per step BEFORE the captured +region, not inside it), and ensure the query entering PA is already device +(no `EnsureHost(query)`). The device SDPA decode path itself is +capture-clean (it's a single ttnn op); the work is removing the host +metadata reads around it. + +**Gate:** device PA vs host oracle numerics (already measured: max_abs +~0.0009 for prefill; decode parity measured separately); e2e gate. + +### R4 — Flip `support_static_graph_mode()` + wire capture (only after R1-R3) + +**Problem:** the platform gate and the `Qwen3DenseDecodeGraph` wiring are +trivial once the region is host-free. This row flips the platform flag, +verifies capture no longer aborts, and measures replay tok/s vs eager. + +**Gate:** capture completes (no `TT_FATAL`); replay max_abs=0 vs eager +(already the landed unit-test property); **replay warm tok/s ≥ 12.5** +(the current hybrid eager baseline) — this is the payoff that justifies +all four rows. + +## Sequencing + dependencies + +``` +R1 (RmsNorm+RoPE device) ─┐ +R2 (QkvSplit+RAC device) ─┼─► R4 (capture wire + measure) ──► decode tok/s win +R3 (PA decode metadata) ─┘ +``` + +R1, R2, R3 are independent and parallel-claimable. R4 is the integration +row that wants all three + produces the headline number. If R4's replay +tok/s does NOT beat 12.5, the whole effort is a wash — but that can only be +known after R1-R3, which is the cost of answering it. + +## Gates (per row + integration) + +- **Correctness:** every device-resident variant must be bit-exact or + near-tie vs the current host path, gated by the existing TT golden pair + (`our_ids_tenstorrent.npy` / `neartie_gap_mnats_tenstorrent.npy` for + Qwen3-0.6B, the Mistral pair for Mistral-7B). RED-first op-level test + before each e2e gate. +- **Capture (R4 only):** `TT_FATAL`-free capture + replay max_abs=0 + + replay warm tok/s ≥ 12.5 (Qwen3-0.6B `vllm-cli` smoke, same harness as + the trace-runner spike). +- **No perf regression outside capture:** the threshold flips in R1 regress + *eager* tok/s (12.5→10.7) — that regression is acceptable ONLY because R4 + recovers it. If R4 is not landed, R1 must not ship unconditionally; it + must gate on `support_static_graph_mode()` so non-capture runs keep the + hybrid thresholds and the 12.5 baseline. + +## Risk + +- **R3 is the scope risk.** R1 and R2 are mechanical (flip + reuse existing + device paths); R3 (device PA decode with device-resident metadata) is + real work and the most likely place to find another host touch. +- **R4's payoff is uncertain until measured.** The whole plan exists to + answer "does capture beat 12.5 tok/s"; if it doesn't, R1-R3 still + delivered device-resident ops (useful for future prefill capture) but no + decode win. That's an honest outcome, not a failure — it's the + measurement the trace-runner spike owed and couldn't make. diff --git a/.agents/specs/tenstorrent-trace-runner.md b/.agents/specs/tenstorrent-trace-runner.md new file mode 100644 index 000000000..b0d0d9656 --- /dev/null +++ b/.agents/specs/tenstorrent-trace-runner.md @@ -0,0 +1,263 @@ +# Tenstorrent mesh-trace decode capture — feasibility spike + +Status: **DRAFT, 2026-08-12.** A read-only feasibility spike (POL-SPIKE-FIRST, +POL-NO-CEILING). No code change is in scope until the decision (§Risks/decisions) +is recorded and the maintainer accepts the tradeoff. + +Proposed row id: `BACKEND-TENSTORRENT-TRACE-RUNNER` (child of +`BACKEND-TENSTORRENT`; the graph-capture *foundation* already landed as #354 / +`59568772` — this row is the *runner wiring* the handoff §8/§9 names as "not +done"). + +## Scope + +**In.** Establish the facts needed to decide whether wiring the landed +graph-capture surface into a Qwen3-dense / Mistral **decode** graph on +`kTENSTORRENT` is feasible and worth it, and at what cost. Three questions: + +1. **Is there a host-free decode region to capture?** At pure decode (T=1), + do the current hybrid thresholds route any op to host, breaking ttnn + `begin_trace_capture` (which requires a contiguous device-op region with + no host sync / `to_vector` readback in the middle)? +2. **What does capture cost in tok/s?** If forcing the host-routed ops + (RoPE, residual RMS) all-device at T=1 is the prerequisite for capture, + does the resulting capture *recover* the tok/s that all-device-at-T=1 + loses? The handoff §6 records that "always device residual/RoPE" + regressed Qwen3-0.6B ~12.3→10.5 tok/s; capture must beat 10.5 to be + worth it, and ideally approach/beat 12.3. +3. **Does ttnn program-cache warm-up work before `begin_trace_capture`?** + ttnn requires the same op shapes to be JIT-compiled (program-cache warm) + before capture begins (ttnn `graph_query_op_runtime.hpp` pattern). Is + that satisfied by one eager forward, as DeepSeek-V2's CUDA path does + (`deepseek_v2.cpp:1028-1034`)? + +**Out.** No implementation, no perf claim, no model expansion. If the +decision is "go," the implementation is a follow-on under this row, not a +new row. Prefill capture and multi-token-chunk capture (where thresholds +already go all-device) are explicitly separate — this spike is about +**decode (T=1)**, the handoff's named target. + +## How the other backends do it (POL-MIRROR-VLLM / POL-SEAM-RUNNER) + +Surveyed on `origin/main` (`a89b3c45`): + +- **CUDA (`src/vt/cuda/cuda_backend.cu:198-240`)** — the ONLY backend with + `SupportsGraphCapture() == true`. Capture contract (cuda_backend.cu:184-197): + every op in the region runs ASYNC on the stream (no `Synchronize`, no + host↔device blocking copy); NO `cudaMalloc`/`cudaFree` inside the region + (pool pre-warmed, every alloc a pool hit); captured pointers stay fixed + across replays, only contents change (written by an async copy BEFORE + Replay). `BeginCapture`→`cudaStreamBeginCapture`, + `EndCaptureGraph`→`cudaStreamEndCapture`+`cudaGraphInstantiate`, + `ReplayGraph`→`cudaGraphLaunch`. +- **Metal (`metal_backend.mm:13-15`), Vulkan (`vulkan_backend.cpp:16-18`)** — + both `SupportsGraphCapture() == false`, with comments naming the eventual + mapping (`MTLIndirectCommandBuffer` / pre-recorded `VkCommandBuffer`) and + explicitly NOT implementing it. So TT would be the first non-CUDA backend + to ship capture. +- **Model-side capture site** (`deepseek_v2.cpp:1028-1034`, + `qwen3_moe.cpp:506-509`, `qwen3_dflash.cpp:1091-1095`): the region + captured is `ForwardLayers` — the FULL layer stack, device-resident. The + pattern is: embed input → `BeginCapture` → `ForwardLayers` → + `EndCaptureGraph` → `ReplayGraph` per step. One eager step first warms the + pool/residency/kernel cache (the "cold size" path at `deepseek_v2.cpp:1051`). + +**Implication for TT:** the CUDA contract maps almost 1:1 onto ttnn trace +capture's own requirements (contiguous device region, no host readback, +pre-warmed program cache). The TT backend's capture surface +(`tenstorrent_backend.cpp:70-76`) already maps `BeginCapture`→ +`TraceBeginCapture` etc. onto `ttnn::begin_trace_capture`/`end_trace_capture`/ +`replay_trace`. What is NOT done is the model-side wiring: no dense forward +(Qwen3/Mistral) calls `BeginCapture`/`ForwardLayers`/`EndCaptureGraph` on TT +today, and the decode path currently interleaves host ops. + +## Upstream chain + +**No upstream vLLM equivalent.** vLLM's CUDA-graph capture is the loyal +contract (`vllm/v1/worker/gpu/worker.py::capture_model`, already mirrored by +this tree's CUDA path). ttnn's trace API is the dependency-chain leg: +`ttnn::begin_trace_capture(device, cq_id)` / `end_trace_capture` / +`replay_trace` (tt-metal `ttnn/cpp/ttnn/trace.hpp`), already wired in +`tenstorrent_ops.cpp::TraceBeginCapture/TraceEndCapture/TraceReplay`. + +## Our baseline (the blocker, precisely) + +At pure decode (T=1) for Qwen3-0.6B (Hq=16, Hkv=8, D=1024), the current +hybrid thresholds in `tenstorrent_ops.cpp`: + +- **Residual RMS** (`RmsNormKernel`, line 1070): `kDeviceResidualMinRows = + 32`; at T=1, `rows=1 < 32` → the **host** f32 path runs (bit-identical to + CPU; verified by `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN`). +- **RoPE** (`PreferDeviceRope`, line 1342): `tokens * heads >= 64`; at T=1, + `1*16 = 16 < 64` → the **host** RoPE path runs. + +So **every decoder layer at T=1 hits the host twice** (residual + RoPE), +interleaved with device matmuls/attention. There is no contiguous +device-only region spanning a full layer, let alone the layer stack. ttnn +capture aborts on the host readback in the middle — this is exactly the +"needs a region free of host RoPE/residual" note in the handoff §8. + +The residual-golden row measured the *numerics* of flipping residual to +device at rows≥32; the *perf* of flipping both residual AND RoPE to +all-device at T=1 is what this spike must measure. The handoff §6 says that +flip ("always device residual/RoPE") regressed Qwen3-0.6B ~12.3→10.5 tok/s. +Capture's value proposition is collapsing the per-step host-API overhead; +whether it recovers the ~1.8 tok/s loss AND goes beyond 10.5 is the open +question. + +### Spike finding: the decode runner is a shared framework, gated on TWO methods + +The dense decode-graph framework already exists and is **model-shared**: +`Qwen3DenseDecodeGraph` + `DenseDecodeGraphForward` in +`src/vllm/model_executor/models/qwen3.cpp:489,670`, used by Qwen3, Mistral, +Llama, and InternLM2 (all four registries call `DenseDecodeGraphForward`). +Its `Impl` ctor (`qwen3.cpp:495-497`) gates enablement on BOTH: + +```cpp +enabled = env_on && + platforms::GetPlatform(...).support_static_graph_mode() && + b.SupportsGraphCapture(); +``` + +- `SupportsGraphCapture()` — TT returns `true` (landed #354). +- `support_static_graph_mode()` — the `Platform` method + (`include/vllm/platforms/interface.h:189`, base default `false`). **TT + does NOT override it** → inherits `false` → the decode-graph framework is + **disabled** for TT today, even though the backend can capture. Only CUDA + overrides it to `true` (`cuda.cpp:59`); ROCm explicitly stays `false` + (`rocm.cpp:67`, "hipGraph is the mapping and is not wired"). + +**Implication:** wiring TT into the decode runner is, at the platform +seam, a one-line override (`support_static_graph_mode() == true`). But that +alone is insufficient: the framework would then attempt to capture the T=1 +decode forward, which (per the thresholds above) hits host RoPE+residual +every layer and would abort ttnn capture. So the real prerequisite is a +host-free decode region; the platform flag is the *enabler*, not the work. + +This is the same shape as CUDA's path: CUDA's decode-graph capture works +because the CUDA ops are all-async (no host sync in the region). TT's +decode currently isn't all-device at T=1, so capture can't apply yet. + +## Work breakdown (spike-only, read-only except for measurement) + +1. **Confirm the host interleaving** (static): trace the T=1 decode op + sequence through `Qwen3DenseModel::Forward` and list every `EnsureHost` / + host-path hit per layer. Output: a per-layer host-touch map. +2. **Measure all-device-at-T=1 baseline** (on-card, temporary env override): + add a hidden `VT_TT_FORCE_DEVICE` escape hatch (local only, NOT shipped) + that forces `kDeviceResidualMinRows=1` + `PreferDeviceRope=true` always, + run the Qwen3-0.6B `vllm-cli` smoke, record warm tok/s. Compare to 12.3 + (hybrid) and 10.5 (handoff's "always device" number). +3. **Probe ttnn program-cache warm-up**: confirm one eager forward makes the + capture-region op shapes resident in ttnn's program cache (the + `begin_trace_capture` precondition). Static + a tiny standalone probe if + needed. +4. **Decision record** (§Risks/decisions): go / no-go / go-only-for-prefill. + No implementation in this spike. + +## Gates + +**No correctness gate owed by the spike itself** — it produces a decision +record, not a shipped change. The temporary `VT_TT_FORCE_DEVICE` measurement +is a local throwaway, gated behind an env var that never ships; it does not +alter the committed decode path. Any implementation follow-on carries its +own token-exact / distributional gate against the oracle (Qwen3-0.6B +`our_ids_tenstorrent.npy` / the Mistral TT golden pair). + +**Hardware:** real Blackhole (P150) for the measurement step. + +## Dependencies + +- `BACKEND-TENSTORRENT` (parent) — `ACTIVE`. +- The landed trace foundation (`59568772`, #354): `SupportsGraphCapture`, + `BeginCapture`/`EndCapture`/`Replay`, `EndCaptureGraph`/`ReplayGraph`/ + `DestroyGraph`, unit-tested (matmul warm→capture→replay×3, max_abs=0). +- Qwen3-0.6B checkpoint (already on this box) for the smoke measurement. + +## Risks/decisions + +- **The likely outcome is "no-go for pure decode (T=1), go for prefill / + multi-token."** The hybrid thresholds exist *because* all-device-at-T=1 + lost ~1.8 tok/s; if capture doesn't recover that plus margin, pure-decode + capture is a net loss. But at prefill / chunked-prefill (T≥32), the + thresholds ALREADY go all-device — that region IS host-free today and is + the natural first capture target. The spike should not assume decode is + the right capture scope; prefill may be where TT capture first pays off. +- **ttnn capture preconditions are stricter than CUDA's in one way:** ttnn + needs the *exact op shapes* program-cache-warm before capture (not just + pool-pre-warmed). One eager forward may suffice for a fixed-shape decode + graph, but a batched/padded decode (variable B) would need a capture per + padded size — same as CUDA's multi-graph path (`EndCaptureGraph` handle + API), which the TT backend already exposes. +- **Not a correctness risk:** capture/replay is numerically identical to + eager (the landed unit test proves max_abs=0 on replay). The risk is + purely perf (does it beat the hybrid baseline?) and scope (decode vs + prefill). + +## Outcome (2026-08-12/13) — Q1+Q2 answered, capture attempted, **NO-GO measured** + +**Q1 (host-free region?):** NO at pure decode (T=1). Static trace: `RmsNormKernel` +(rows=1 < 32 → host) and `PreferDeviceRope` (T*H = 16 < 64 → host) both route +to host every layer. Plus the shared decode-graph framework +(`Qwen3DenseDecodeGraph`) is disabled for TT because the platform does not +override `support_static_graph_mode()` (base default `false`; only CUDA +returns `true`). + +**Q2 (all-device-at-T=1 tok/s cost?):** measured on real Blackhole P150, +Qwen3-0.6B, `vllm-cli --prompt Hello --max-tokens 4 --repeat 3`, with a +local-only `VT_TT_FORCE_DEVICE` override forcing both thresholds all-device: + +| config | warm tok/s | +|--------|-----------| +| hybrid (current default) | **12.5** (12.49, 12.54) | +| all-device (VT_TT_FORCE_DEVICE=1) | **10.7** (10.77, 10.66) | + +Reproduces the handoff §6 number precisely (~12.3→10.5; 12.5→10.7 here — +same ~1.8 tok/s / ~14% regression). The throwaway override was reverted. + +### Q2b — capture attempted on-card: ABORTS (the decisive measurement) + +The earlier "no-go" was a guess (capture "implausibly" recovers 1.8 tok/s). +To actually decide, the spike ran the capture experiment: local-only flips of +both `support_static_graph_mode()` → `true` (enables `Qwen3DenseDecodeGraph`, +which wires `BeginCapture`/`ForwardLayers`/`EndCaptureGraph` on the captured +padded-batch slot) AND `VT_TT_FORCE_DEVICE` (RoPE+residual all-device), then +the same Qwen3-0.6B cli smoke. + +**Result: capture ABORTS.** ttnn raises `TT_FATAL: Reads are not supported +during trace capture` with a backtrace through `ttnn::Tensor::to_vector` +— a device→host readback fires inside the captured `ForwardLayers` region. +`[Qwen3DenseDecodeGraph] dense decode graph: 0 total replays across 1 captured +size(s)` confirms the graph never successfully replayed. + +**This is strictly stronger than the tok/s guess.** Even with the two hybrid +thresholds forced all-device, the T=1 forward still performs `to_vector` host +readbacks (the `DownloadToHost`/`EnsureHost` path in ops — embedding result, +paged-attention output, or logits), and `ttnn::begin_trace_capture` prohibits +*any* host read during the captured region. So capture does not merely +*start from a 1.8 tok/s deficit* — it **cannot run at all** on the current +T=1 forward without first eliminating every `to_vector` readback in the +captured region, which is a much larger redesign than flipping two thresholds. + +Both local overrides (`support_static_graph_mode`, `VT_TT_FORCE_DEVICE`) were +reverted; no code shipped. + +**Q3 (ttnn program-cache warm-up?):** moot — capture aborts before warm-up +matters. + +### Decision + +**NO-GO for pure T=1 decode capture — now measured, not assumed.** The T=1 +forward does `to_vector` host readbacks that ttnn trace prohibits; forcing +the two hybrid thresholds all-device is insufficient because other ops still +read back. Decode capture requires a host-free `ForwardLayers` (every op +device-resident end-to-end, no `to_vector`), which is a redesign of the TT +forward's host-staging model, not a threshold tweak. + +**Open follow-on (separate row): prefill / multi-token chunk capture.** At +T≥32 the thresholds already go all-device; whether the prefill region is also +free of `to_vector` readbacks is the open question Q3 should answer next. +The value proposition there is different (prefill has more host-API overhead +per step) and the host-readback constraint is the same ttnn rule, so the +prefill row must first audit its readbacks before claiming capture is +feasible. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 4817336c2..98c403319 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -207,7 +207,15 @@ # row owing residual-RMS numerics evidence at the device boundary (rows>=32 # bf16 device path vs CPU f32 oracle). Bot-flagged on #289; READY once the # RED-first probe lands. - "BACKEND": (AGENTS / "backend-matrix.md", 81), + # 82 since 2026-08-12: +`BACKEND-TENSTORRENT-TRACE-RUNNER`, feasibility + # spike for wiring the landed #354 graph-capture foundation into a + # capturable forward region (decode host-free region? capture tok/s cost? + # ttnn program-cache warm-up?). No code; decision record only. + # 83 since 2026-08-13: +`BACKEND-TENSTORRENT-HOST-FREE-FORWARD`, the plan + # row decomposing the host-free decode forward (R1 RmsNorm+RoPE, R2 + # QkvSplit+RAC, R3 PA decode, R4 capture wire) that the trace-runner + # spike revealed as the real prerequisite for decode capture. + "BACKEND": (AGENTS / "backend-matrix.md", 83), } ENGINE_MATRIX = AGENTS / "engine-matrix.md"