row/SERVE-ASYNC-EXECUTOR: decode-graph slot double-buffer, gated OFF (Option-A groundwork) - #36
Merged
Merged
Conversation
…, default OFF)
Option B of the c16/c32 overlap unlock. The depth-2 moved drain (runner.cpp mirror
path) exists for hazard-C: the Qwen3.5 decode graph bakes its per-step H2D INSIDE the
captured replay (BuildStepDevInputs uploads positions/attn+GDN metadata from the
per-size SizeSlot's persistent HOST vectors), which s.Refresh overwrites off-queue, so
Replay(N) reads s.* while Refresh(N+1) overwrites them.
Ring (gated VT_ASYNC_EXECUTOR, default OFF): each decode-graph driver's slots map goes
map<size,SizeSlot> -> map<size,SlotRing{SizeSlot[2]}>, alternated per step; each slot
records a blocking-sync reuse event on the main queue after its replay, host-waited
before its next Refresh. runner.cpp skips the drain when the previous logits are a
non-owning graph view (ForwardLogits.non_owning_view); eager/mixed steps still own
their logits and drain. backend CreateEvent gains a cudaEventBlockingSync flavor. OFF
routes through the single-slot code with the drain intact — byte-identical by
construction.
Capture-safety fix: the ring needs TWO retained [S,vocab] logits blocks (one per
slot) but the pool was warmed for one -> the 2nd capture hit cudaMalloc mid-capture;
the warm branch now pre-grows the pool. Working scratch is freed and safely shared
(the two graphs replay sequentially on one stream).
RESULT (dgx GB10, commit archived, dual-lock): CPU -Werror + 6 suites pass. Ring
PROVEN correct — async GREEN 5/5 @conc-32, SACRED sync 3/3, memcheck x2 clean. But
hazard-C, though real-by-construction (the graph re-reads its host inputs every
replay), is empirically UNREPRODUCIBLE on GB10 (the baked H2D is a tiny fast copy at
the replay's start; the drain-skipped-no-ring arm passed 3/3 @conc-4 AND @conc-32),
and the c16/c32 speed A/B is NEUTRAL (host-orchestration bound). Per the parity-enabler
rule the default stays OFF with the honest result recorded; the ring is the
provably-safe drain-removal path and the stepping stone to Option A (out-of-graph
device-staged H2D). Test-only knobs VT_ASYNC_EXECUTOR_{NO_DBUF,POISON,TRACE} +
VT_ASYNC_SERVING_CONC retained for reproducing the analysis. Full analysis + evidence
in the benchmark record.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
# Conflicts: # .agents/NOW.md # .agents/benchmark-record.md # .agents/state.md
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
The #36 squash touched qwen3_5.cpp without moving docs/FEATURES.md in the same commit (the pre-merge doc-checkpoint ran on the merge-commit shape, where the diff looked records-only) and CI flagged the drift. The row also lagged reality: async scheduling has been default-ON with the UAF-safe drain since #31, and #36 added the opt-in VT_ASYNC_EXECUTOR slot ring. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
VT_ASYNC_EXECUTOR(default OFF, OFF path byte-identical): two SizeSlots per padded size with per-slot reuse events in BOTH decode-graph drivers (35B MoE + 27B dense), a blocking-event flavor onBackend::CreateEvent, and a drain-skip in the runner when the prior step's logits are a non-owning graph view. This is the provably-safe mechanism for removing the depth-2 drain, and the stepping stone to Option A (H2D staged out of capture into persistent device buffers, vLLMstates.py:64/_prepare_input_ids).Honest verdict
cudaMallocin the pool warm path (the ring needs a 2nd retained logits block).vllm bench serveregime (~50 tok/s output), NOT the binding online_gate grid; a binding-harness A/B is owed before any default flip. Memory delta below noise floor.Default stays OFF per parity-enablers. Test knobs (
VT_ASYNC_EXECUTOR_{NO_DBUF,POISON,TRACE}) retained for Option-A work.Residuals (recorded)
Option A is the real c16 recovery; ignore_eos UAF bracket not re-run (eager path unchanged/drained); binding-harness A/B owed.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys