perf(qwen3_5): SERVE-ASYNC-OPTION-A — decode-graph input H2D staged out of capture (faithful vLLM, gated OFF) - #39
Merged
Conversation
…UT of capture; faithful vLLM, RED reproduces, A/B WASH, gated OFF Option A ports vLLM's actual async decode-input structure (states.py:64, _prepare_input_ids / synchronize_input_prep): under VT_ASYNC_EXECUTOR each decode step's input H2D is enqueued on the main queue BEFORE ReplayGraph into PERSISTENT device buffers the captured graph reads, guarded by an input-staged event recorded right after the tiny staging copy — so the next same-slot Refresh waits only that copy, never the GPU tail. PINNED host staging makes the H2D a true-async DMA (pageable GB10 H2D is host-synchronous, which is why #36's poison never raced). Supersedes #36's in-capture-baked ring (ring retained for the depth-2 logits double-buffer). Both drivers (35B MoE + 27B dense). Persistent s.dev drawn from a dedicated pool via ActivePoolScope so it never starves the captured forward's scratch (fixes a cudaMalloc-during-capture). Default OFF byte-identical. Verified on dgx GB10 sm_121a (both locks, single load/arm, drop_caches): - CPU -Werror + runner/llm_engine/engine_core_proc/async_llm/input_batch 5/5. - RED VT_ASYNC_EXECUTOR_POISON reproduces DETERMINISTICALLY (pinned-source poison -> coherent-but-wrong tokens) — the RED #36 could not construct. - GREEN async 5/5 @conc-32, SACRED x3, memcheck x2 = 0 errors, NO_DBUF PASS, served ignore_eos UAF bracket 4-128 all ALIVE; server drain-skip engagement confirmed. - Binding A/B (online_gate, env-toggle): throughput WASH (c8 +0.14%, c16 +0.02%, c32 +0.06%), TTFT WASH. Removing exactly the drain + in-capture H2D does not move throughput -> the c16/c32 gap is NOT the async input path. DEFAULT OFF. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
# Conflicts: # .agents/NOW.md # .agents/benchmark-record.md # .agents/state.md
localai-bot
marked this pull request as ready for review
August 5, 2026 19:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Option A — stage the decode-graph input H2D OUT of capture (faithful vLLM)
VT_ASYNC_EXECUTORnow ports vLLM's actual async decode-input structure(
gpu_model_runner.py_prepare_input_ids/synchronize_input_prep,states.py:64):each
SizeSlotgets persistent device input buffers the captured graph reads andpinned host staging; per step the input H2D is enqueued on the main queue before
ReplayGraph(out of capture) and an input-staged event is recorded right after thetiny staging copy, so the next same-slot Refresh waits only that copy, never the GPU
tail. Pinned host memory makes the H2D a true-async DMA (pageable GB10 H2D is
host-synchronous, which is exactly why #36's poison could never race). Supersedes #36's
in-capture-baked ring (the 2-slot ring is retained for the depth-2 logits double-buffer).
Both drivers (35B MoE + 27B dense). Default OFF, byte-identical.
A capture-time
cudaMallocbug (the persistents.devpopped-and-held a size-class thecaptured forward's scratch then needed) is fixed by routing
s.devthrough a dedicatedPersistentDecodeInputPoolunderActivePoolScope.Verification (dgx GB10 sm_121a, both locks, single load/arm, drop_caches)
-Werror+ runner/llm_engine/engine_core_proc/async_llm/input_batch 5/5.VT_ASYNC_EXECUTOR_POISONreproduces deterministically (pinned-sourcepoison -> coherent-but-wrong tokens). This is the RED row/SERVE-ASYNC-EXECUTOR: decode-graph slot double-buffer, gated OFF (Option-A groundwork) #36 could not construct, and it
proves the input-staged event is load-bearing.
test_qwen36_async_servingON @conc-32 5/5; SACREDtest_qwen36_paged_engineON x3; compute-sanitizer memcheck x2 = 0 errors;VT_ASYNC_EXECUTOR_NO_DBUFPASS; served ignore_eos UAF bracket 4-128 all ALIVE;drain-skip engagement confirmed in the server.
online_gate.py, ONE binary env-toggled, single load/arm, 3 reps):throughput WASH (c8 1778.9->1781.4, c16 2294.4->2294.8, c32 2914.5->2916.3 tok/s);
TTFT WASH.
Decision
Speed-neutral -> default OFF (parity-enablers). Definitive negative: removing exactly
the depth-2 drain and the in-capture baked H2D leaves throughput/TTFT unchanged, so
the c16/c32/c8 deficit to vLLM is not the decode-graph input-H2D structure. Residual
is prefill glue (task #61). Mechanism kept gated (strictly better than #36: faithful
structure + a working RED). Evidence
dgx:~/work/mirror-ab/option-a/.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys