spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode - #541
Draft
lu-zero wants to merge 2 commits into
Draft
spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode#541lu-zero wants to merge 2 commits into
lu-zero wants to merge 2 commits into
Conversation
BACKEND-TENSTORRENT-TRACE-RUNNER. Replaces the prior "no-go" guess with a measured result from actually attempting the capture on-card. The spike ran the real experiment: local-only flips of both support_static_graph_mode() (enables Qwen3DenseDecodeGraph) AND VT_TT_FORCE_DEVICE (RoPE+residual all-device), then Qwen3-0.6B cli smoke. Both overrides reverted; no code shipped. Result: capture ABORTS. ttnn raises TT_FATAL: Reads are not supported during trace capture with a backtrace through ttnn::Tensor::to_vector<float> -- a device->host readback fires inside the captured ForwardLayers region. 0 replays. The T=1 forward does to_vector readbacks (embedding/PA/logits) that ttnn trace prohibits; forcing the two hybrid thresholds all-device is insufficient. This is strictly stronger than the earlier tok/s-recovery guess: decode capture cannot run AT ALL on the current forward without eliminating every to_vector readback in the captured region (a host-free ForwardLayers redesign, not a threshold tweak). Q1 (no host-free region at T=1) and Q2 (all-device costs 12.5->10.7 tok/s, reproducing handoff §6) stand. Decision: NO-GO for pure T=1 decode, measured. Next: prefill capture (separate row) must first audit its to_vector readbacks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]
lu-zero
force-pushed
the
row/BACKEND-TENSTORRENT-TRACE-RUNNER
branch
from
August 12, 2026 22:16
690b49e to
f0bdc2d
Compare
BACKEND-TENSTORRENT-HOST-FREE-FORWARD (child of BACKEND-TENSTORRENT). The
trace-runner spike measured that decode mesh-trace capture ABORTS
(TT_FATAL: Reads are not supported during trace capture, backtrace through
to_vector<float>) because the per-layer T=1 forward does device->host
readbacks that ttnn trace prohibits. Capture wiring works (it engaged and
tried); the forward is the blocker.
This commit is a PLAN only (no code). It decomposes the host-free forward
into four independent rows, grounded in a full readback map of the current
TT ops:
R1 RmsNorm + RoPE all-device at T=1 (threshold flip; numerics already
proven by BACKEND-TENSTORRENT-RESIDUAL-GOLDEN; eager cost 12.5->10.7
tok/s measured, recovered by capture).
R2 QkvSplit + ReshapeAndCache device-resident (bit-exact memcpy/stride
ops; reuse the landed paged-KV-shadow path).
R3 PagedAttention decode with device-resident metadata (the scope risk --
device SDPA decode exists, but block_table/seq_lens/query_start_loc
are read host-side around it).
R4 flip support_static_graph_mode + wire capture + measure replay tok/s
(integration; the payoff that justifies R1-R3: replay warm tok/s
>= 12.5).
R1-R3 parallel-claimable; R4 integrates. The R1 eager regression (12.5->
10.7) is acceptable ONLY because R4 recovers it; R1 must gate on
support_static_graph_mode() so non-capture runs keep the 12.5 baseline.
The trace-runner row's "next" now points here instead of to a vague
prefill-capture note: the host-free forward is the real unblock.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Maki:glm-5.2 [Maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
BACKEND-TENSTORRENT-TRACE-RUNNER(child ofBACKEND-TENSTORRENT). A read-only feasibility spike (POL-SPIKE-FIRST, POL-NO-CEILING) deciding whether to wire the landed graph-capture foundation (#354 /59568772) into a capturable decode region. No code shipped — decision record only. The spike actually attempted the capture on-card rather than guessing.Findings
Q1 — host-free region at T=1? No. Static trace:
RmsNormKernel(rows=1 < 32) andPreferDeviceRope(T·H = 16 < 64) both route to host every layer. Bonus: the shared decode-graph frameworkQwen3DenseDecodeGraph(qwen3.cpp:489, used by Qwen3/Mistral/Llama/InternLM2) is disabled for TT because the platform doesn't overridesupport_static_graph_mode()(basefalse; only CUDA returnstrue).Q2 — all-device-at-T=1 tok/s cost? Measured on real Blackhole P150, Qwen3-0.6B, local-only
VT_TT_FORCE_DEVICEoverride (reverted, unshipped):Reproduces handoff §6 precisely (~12.3→10.5; 12.5→10.7 here; same ~14% / ~1.8 tok/s regression).
Q2b — capture attempted on-card: ABORTS. With both
support_static_graph_mode()→true(enablesQwen3DenseDecodeGraph, which wiresBeginCapture/ForwardLayers/EndCaptureGraph) ANDVT_TT_FORCE_DEVICEset, capture still fails:A device→host readback fires inside the captured
ForwardLayersregion. ttnnbegin_trace_captureprohibits any host read during capture; forcing the two hybrid thresholds all-device is insufficient because other ops (embedding / paged-attention / logits) still doto_vector.Decision
NO-GO for pure T=1 decode capture — measured, not assumed. Decode capture cannot run at all on the current forward without first eliminating every
to_vectorreadback in the captured region, which is a redesign of the TT forward's host-staging model, not a threshold tweak. The earlier "implausible to recover 1.8 tok/s" reasoning is superseded: capture aborts before perf even enters the picture.Open follow-on (separate row): prefill / multi-token chunk capture. At T≥32 the thresholds already go all-device; whether the prefill region is also free of
to_vectorreadbacks is the open question the next row must answer first (same ttnn host-read prohibition applies).How the other backends do it
CUDA is the ONLY
SupportsGraphCapture()==truebackend (cuda_backend.cu:184-240: contract = async/no host sync/no malloc in region, fixed pointers). Metal and Vulkan arefalsewith the eventual mapping named but unimplemented. Model-side capture sites captureForwardLayersdevice-resident, one eager step warms the pool first.Files
.agents/specs/tenstorrent-trace-runner.md(spec + measured## Outcome+ decision)BACKEND-TENSTORRENT-TRACE-RUNNERrow in.agents/backend-matrix.md(SPIKE, decision record complete).agents/coordination.mdscripts/check-agent-record.pyBACKEND 81→82src/change. Both local overrides (support_static_graph_mode,VT_TT_FORCE_DEVICE) were reverted.