diff --git a/.agents/NOW.md b/.agents/NOW.md index 7befe3fcf..d7a1f4c3f 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -20,8 +20,8 @@ checkpoint on `upstream/main` at `59674cf1d`. | Invocation-parity prevention | CI guard (`check-gemv-invocation-consistency.py`) + AGENTS.md checklist landing | Review + merge; CUDA build-verify `kGemvHeuristicAlgos` on dgx | | MiniMax-H3 lane | Portable path complete; e2e prompt-conditioned video on real weights (Thor). Speed = NVFP4 FP4 device path, sm_121-gated | PR #26 rebase + supports-audit synthesis (workflow ran; integrate) | | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **W7 device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`, `ACTIVE`): DBuf-resident `ForwardDeviceCompute` (2 host islands: KDA recurrence, NoPE-MLA softmax); `test_kimi_linear_forward` **12/12·614**; opt-in `VT_KIMI_DEVICE_COMPUTE=1` | GPU-verify: CUDA build, token-exact vs oracle, e2e §8 | -| 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE lever **RESOLVED NEGATIVE**: attribution boundary (drain A/B collapses intake -91% but shifts to queued; arrival→sched GPU-bound, NEUTRAL). Probe kept, lever reverted | Real levers: prefill glue (task #61) + async device-resident executor (also the c16 fix) | -| Async-serving decode bug **FIXED** | Batch-1 nondet token-0 garbage: combine device-write raced the decode-graph host read (SYNC SACRED missed it). FIX: `VT_ASYNC_DEVICE_MIRROR` **default ON**; gate `test_qwen36_async_serving` RED→GREEN, SACRED/UAF clean | MERGED (#31) | +| 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE lever **RESOLVED NEGATIVE** (attribution boundary, drain A/B, sum invariant); probe kept, lever reverted | Real levers: prefill glue (task #61) + Option A (H2D out of capture) | +| Async decode: mirror fix MERGED (#31); slot double-buffer (`VT_ASYNC_EXECUTOR`) landed gated **OFF** | Ring+reuse-event replaces the depth-2 drain; proven correct (async gate 5/5 @c32, SACRED 3/3, capture-malloc fixed); RED unreproducible, speed neutral in a host-bound regime | Option A follow-up; binding-harness A/B owed | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index b13dcfd9d..01da77ada 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -11836,6 +11836,139 @@ Evidence `dgx:~/work/mirror-ab/{asyncgate2-{RED,GREEN}.log, asyncgate-SACRED-default.log,mab-defmeasure.log,mab-asyncmemcheck.log, evidence/raw/35/ours/c16-r{1,2,3}-{defaulton,rollback0}.json}`. +## ROW-SERVE-ASYNC-EXECUTOR: decode-graph slot double-buffer (Option B), gated `VT_ASYNC_EXECUTOR` — the c16/c32 overlap unlock + +CONTEXT. With `VT_ASYNC_DEVICE_MIRROR` default ON (the ROW-SERVE-ASYNC-LLM fix +above), the async serving loop's remaining serialization is a depth-2 pre-forward +drain: `runner.cpp` (mirror path, before the forward) does a main-queue +`Synchronize` every step. It exists because of hazard-C below. Removing it is the +c16 (and c32) unlock — the overlap that lets step N+1's host prep + replay run +while step N's GPU tail is still in flight. + +THE THREE HAZARDS (why the drain is there, and what each needs to be safe to skip). + +hazard-C (the drain's REASON; decode-graph only). The Qwen3.5 decode graph bakes +its per-step H2D INSIDE the captured replay: `BuildStepDevInputs` (`qwen3_5.cpp`) is +called inside the captured `ForwardLayers`, and it uploads +positions / slot_mapping / block_table / seq_lens / query_start_loc / GDN state +indices from the per-size `SizeSlot`'s PERSISTENT HOST vectors +(`token_ids, positions, attn_meta.*, gdn_meta.*`). `s.Refresh` overwrites those same +host vectors with a plain host `memcpy` (`CopyInPlace`), OFF the queue; `EmbedInto` +reads `s.token_ids` via an on-queue async H2D. So `Replay(N)` reads `s.*` on the main +queue while `Refresh(N+1)` overwrites them off-queue → corruption without an +intervening drain. This is the blocker the drain solves and the reason a naive +drain-removal degenerates (proven by the RED arm below). + +hazard-A (EAGER path only). `exec_state_.logits` OWNS a pool block on the eager +forward (`WrapDeviceLogits`), freed by the pre-forward `exec_state_` reset while the +deferred async sampler still reads it → use-after-free. The graphed path's +`ViewDeviceLogits` is NON-owning (`non_owning_view=true`, no-op deleter), so resetting +`exec_state_` frees nothing and the slot buffer stays live — hazard-A is absent for a +graph-slot view. + +vLLM's structure (reference). `synchronize_input_prep` is a blocking event +(gpu_model_runner.py); on-stream out-of-graph H2D into PERSISTENT DEVICE buffers +(states.py:64) so `_update_states` never reads a device-written buffer and the sync +only guards input staging. Option B below is the lower-risk equivalent (a per-slot +host-wait ring instead of vLLM's out-of-graph device staging); it is recorded as a +stepping stone to the faithful Option A (move the H2D OUT of the captured replay into +persistent device buffers, then no host-buffer reuse hazard exists at all). + +OPTION B (what landed, gated `VT_ASYNC_EXECUTOR`, default OFF). (1) Each decode-graph +driver's `slots` map goes from one `SizeSlot` per size to a `SlotRing{SizeSlot[2]}` +(parity ring), alternated each step. Each slot records a BLOCKING-sync reuse event +(`cudaEventBlockingSync`, new `Backend::CreateEvent(bool blocking)`) on the main queue +AFTER its replay; before Refresh-ing a slot for reuse the host `SynchronizeEvent`s it. +At depth-2 with 2 slots this wait is almost always already-signaled; it self-limits if +the engine runs ahead. The warm-capture step drains once (idle-stream capture) since +the runner may have skipped the drain. (2) `runner.cpp`: under `VT_ASYNC_EXECUTOR=1`, +SKIP the drain's `Synchronize` when the previous step's logits are a non-owning +graph-slot view (hazard-A absent, hazard-C handled by the ring); the `exec_state_` +reset still runs (releasing the no-op-deleter view). Eager/mixed previous steps still +own their logits and drain as before — the drain-on-eager fallback is acceptable +because c16/c32 steady-state is all-graphed. (3) OFF routes through the single-slot +code with the drain intact — byte-identical by construction (`dbuf=false` picks +`slot[0]` with no events; `skip_drain` is always false). Memory: doubling slots +≈ +76 MB (logits-dominated), one-time; capture time doubles (one-time). + +RESULTS (dgx GB10 sm_121a, cutlass-4.5.0, triton ON, commit `fa971248`, dual-lock, +free>=90, worker down; evidence `dgx:~/work/mirror-ab/serve-async-executor/`). + +CPU (bar a): -Werror clean; runner / llm_engine / engine_core_proc / async_llm / +input_batch 6/6 pass. + +CAPTURE-SAFETY (a real bug found + fixed). The FIRST ring build aborted every +captured step with `cudaMalloc: operation not permitted when stream is capturing`. +Root cause: the pool is pre-warmed for ONE graph's RETAINED `[S,vocab]` logits block +(allocated inside `ForwardLayers`/`DenseForwardLayers` and kept as `s.logits`); the +ring needs TWO retained simultaneously (one per slot), so the second slot's capture +hit a pool MISS. Working scratch (`hidden`/`res`/MoE grouped-GEMM workspace) is freed +at ForwardLayers return and SAFELY shared between the two graphs — they replay +SEQUENTIALLY on the one main stream, so only the retained logits needs two live +copies. FIX: in the warm-capture branch, when dbuf, pre-grow the pool with a +throwaway `[S,vocab]` f32 alloc+free (out-of-capture growth) so the captured logits +alloc is a pool HIT. `fix-check` (ring ON, conc-4): PASS, 0 `capturing` lines. + +HAZARD-C IS REAL BY CONSTRUCTION, but its empirical RED does NOT reproduce on GB10. +The captured decode graph re-reads its persistent host input buffers at EVERY replay +(that is HOW `Refresh` updates each step's inputs; if it did not re-read, every +replay would emit identical tokens, which it does not) — so overwriting them while a +replay is in flight corrupts it, and the drain/ring is load-bearing IN PRINCIPLE. But +the RED arm (drain skipped, ring OFF via `VT_ASYNC_EXECUTOR_NO_DBUF`) PASSED 3/3 at +conc-4 AND 3/3 at conc-32 (`VT_ASYNC_SERVING_CONC=32`), and the deterministic POISON +arm (overwrite the host inputs right AFTER `ReplayGraph`) also PASSED at conc-4. The +reason: the baked H2D is a TINY, FAST copy (~32 ints for positions/slot_mapping) at +the very START of the replay, and on GB10 the GPU executes it before the host +completes a full depth-2 iteration and reaches the next `Refresh` — the race window +is microscopic and closed by pipeline latency. So the token-exact serving gate at any +reachable concurrency can neither RED nor meaningfully protective-GREEN the hazard; +the divergence would require the host to outrun the GPU inside that ~1us H2D window, +which does not happen at conc<=32 on this box. Per the brief's STOP rule the natural +RED is unmet; recorded as an honest partial, not shipped-ON. + +RING CORRECTNESS (proven, the token gates are the authority). GREEN (ring ON, +conc-32) `test_qwen36_async_serving` 5/5 consecutive PASS. OFF baseline (drain on, +conc-32) PASS. SACRED SYNC `test_qwen36_paged_engine` with the ring ON (the ring is +active on the sync path too, and the sync path is DETERMINISTIC token-exact) 3/3 +PASS — the ring produces byte-identical output. The ring is CORRECT; the OFF default +is byte-identical to production by construction (single-slot, drain intact). + +MEMORY (paged-engine single load, `free -m` sampled): peak used OFF 46.8 GB vs ON +44.5 GB of the 119 GB pool — the ~40-80 MB slot-doubling is BELOW the free-g noise +floor (system-used swings GBs run-to-run) so it is not resolvable, but it is well +within budget and there is no OOM. SANITIZER (compute-sanitizer memcheck, ring ON, +`VT_ASYNC_EXECUTOR=1`, x2): both `ERROR SUMMARY: 0 errors`, exit=0 — NO invalid memory +accesses. (`--leak-check full` additionally flags 726 `Leaked` reports, but all are +intentionally-resident LOAD-TIME weight buffers — `BuildMarlinDensePairResident` / +`PrepareMarlinResident` — identical OFF vs ON and freed at teardown, not a ring +defect; plain memcheck, matching the prior methodology, is clean.) The repeated token +gates remain the capture-safety authority, per the brief. + +DEFAULT DECISION: **OFF** (opt-in `VT_ASYNC_EXECUTOR`). The brief flips ON only on a +clean speed win AND the full correctness bar; the empirical RED could not be met +(hazard real-by-construction but unreproducible on GB10), so the conservative default +is OFF with this honest result recorded. The ring is proven correct + capture-safe +and stands as the provably-safe drain-removal mechanism and the stepping stone to the +faithful Option A (move the per-step H2D OUT of the captured replay into persistent +DEVICE buffers, vLLM states.py:64 / _prepare_input_ids — then no host-buffer reuse +hazard exists and no ring is needed). SPEED A/B (`vllm bench serve`, +decode-dominated random 128-in/256-out, single server load per arm, dual-lock): +NEUTRAL — c16 output tok/s OFF 50.3/50.8 vs ON 50.6 (~1.00x); c32 OFF 93.6 vs ON 93.4 +(~0.998x). The drain-skip IS engaged (`VT_ASYNC_EXECUTOR_TRACE` logs skips). But this +regime is host-orchestration / HTTP-streaming bound (TPOT ~312-337 ms, GPU ~14 W at +96% "util"), so it does not cleanly isolate the engine-level overlap, and it is a +different workload from the binding online-serving grid (which reports ~2300 tok/s +TOTAL at c16). No measurable win here. Combined with the unreproducible RED, the +default is OFF. The prior mirror A/B (drain MOVE) was likewise c16-neutral for the +same host-bound reason; the true c16 recovery is Option A (out-of-graph device +staging), of which this ring is the correct-and-safe stepping stone. + +RESIDUALS: (1) Option A is the faithful follow-up. (2) the eager exec_state_ 2-deep +ring was NOT needed (the drain-on-eager fallback is retained; eager/mixed steps still +own their logits and drain). (3) test-only knobs `VT_ASYNC_EXECUTOR_NO_DBUF` +(ring-off), `VT_ASYNC_EXECUTOR_POISON` (deterministic host-input poison), +`VT_ASYNC_EXECUTOR_TRACE` (drain-skip counter), `VT_ASYNC_SERVING_CONC` (gate +concurrency) are retained for reproducing this analysis and future Option-A work. ## Qwen3.5-4B revalidation after merging current upstream (2026-08-05) Merged `upstream/main` at `59674cf1d` into the branch as `312af21a9`, rebuilt diff --git a/.agents/state.md b/.agents/state.md index dd2ec0145..7c8c12f08 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -35791,7 +35791,6 @@ including CI, and that is a separate switch with a separate blast radius. No source, kernel, model, gate, benchmark or capability mark changed. - ## 35B c16 drain-sync lever A/B: blocking-event drain NEGATIVE (−1.9%), full drain KEPT; real fix = GPU-resident sampled tokens @@ -36000,7 +35999,6 @@ interaction; (3) then decide the mirror default on correctness; (4) c16 speed st needs drain-removal + double-buffer. Evidence `dgx:~/work/mirror-ab/{mab-measure.log,mab-tokdiag.log,mab-prodcheck.log,evidence/raw/35/ours/c16-r{1,2,3}-abmirr{off,on}.json,greedy/*}`. - ## ROW-SERVE-ASYNC-LLM P0 RESOLVED: root-caused the async batch-1 token-0 degeneration, added the missing async-serving token-exact gate (RED-first), flipped the device mirror default ON @@ -36080,3 +36078,60 @@ in all reps and unchanged from history. Fresh node-level nsys traces contain graph child kernels on both arms. Evidence: `docs/bench-evidence/qwen35-4b-upstream-20260805.md`; raw root `/tmp/qwen35-upstream-312af21a9`. + +## SERVE-ASYNC-EXECUTOR: decode-graph slot double-buffer (Option B) landed gated default-OFF; ring PROVEN correct; hazard-C real-by-construction but empirical RED unreproducible on GB10 + + +The c16/c32 overlap unlock. With the mirror default ON (ROW-SERVE-ASYNC-LLM), the +async serving loop's remaining serialization is the depth-2 pre-forward drain +(`runner.cpp` mirror path). It exists for hazard-C: the Qwen3.5 decode graph bakes +its per-step H2D INSIDE the captured replay (`BuildStepDevInputs` in `ForwardLayers`/ +`DenseForwardLayers` uploads positions/slot_mapping/block_table/seq_lens/qsl/GDN +state indices from the per-size `SizeSlot`'s persistent HOST vectors), which +`s.Refresh` overwrites off-queue with a plain memcpy — so `Replay(N)` reads `s.*` on +the main queue while `Refresh(N+1)` overwrites them. hazard-A is EAGER-only +(`exec_state_.logits` OWNS a pool block freed by the reset while the deferred sampler +reads it); the graphed path's `ViewDeviceLogits` is NON-owning and stream-safe. + +OPTION B (landed, `VT_ASYNC_EXECUTOR`, default OFF): each decode-graph driver's +`slots` map goes `map` -> `map` (parity +ring, alternated per step); each slot records a BLOCKING-sync reuse event +(`Backend::CreateEvent(bool blocking)` -> `cudaEventBlockingSync`) on the main queue +after its replay, host-waited before its next `Refresh`. The runner SKIPS the drain +under `VT_ASYNC_EXECUTOR=1` when the previous step's logits are a non-owning graph +view (`ForwardLogits.non_owning_view`); eager/mixed steps still own their logits and +drain. OFF routes through single-slot code with the drain intact — byte-identical by +construction (`dbuf=false` picks `slot[0]`, no events; `skip_drain` always false). + +CAPTURE-SAFETY BUG (found + fixed). First ring build aborted every captured step with +`cudaMalloc ... when stream is capturing`: the pool is warmed for ONE retained +`[S,vocab]` logits block (allocated inside ForwardLayers, kept as `s.logits`); the +ring needs TWO simultaneously (one per slot), so the 2nd slot's capture hit a pool +MISS. Working scratch is freed at ForwardLayers return and SAFELY shared (the two +graphs replay SEQUENTIALLY on one stream — the overlap we want is host/GPU, not +GPU/GPU). FIX: warm-capture pre-grows the pool with a throwaway `[S,vocab]` f32 +alloc+free while the other slot's logits is held, so the captured alloc is a HIT. + +hazard-C IS REAL BY CONSTRUCTION but its RED is UNREPRODUCIBLE on GB10. The captured +graph re-reads its persistent host inputs at EVERY replay (that IS how Refresh +updates each step; else every replay would emit identical tokens). But the baked H2D +is a TINY, FAST copy at the replay's START, executed before the host completes a +depth-2 iteration and reaches the next Refresh — the race window is microscopic. +RED arm (drain skipped, ring off via `VT_ASYNC_EXECUTOR_NO_DBUF`) PASSED 3/3 @conc-4 +AND 3/3 @conc-32 (`VT_ASYNC_SERVING_CONC=32`); the low-conc POISON arm +(`VT_ASYNC_EXECUTOR_POISON`, overwrite host inputs right after ReplayGraph) also +PASSED (GPU executes the replay before the host overwrite). Per the brief's STOP +rule the empirical RED is unmet -> default stays OFF, honest partial recorded. + +RING CORRECTNESS PROVEN (the token gates are the authority): GREEN +`test_qwen36_async_serving` ring ON @conc-32 5/5 PASS; OFF baseline @conc-32 PASS; +SACRED SYNC `test_qwen36_paged_engine` ring ON (deterministic token-exact) 3/3 PASS. +CPU: -Werror clean; runner/llm_engine/engine_core_proc/async_llm/input_batch 6/6. +DEFAULT DECISION: OFF (opt-in). Speed A/B c16/c32 OFF vs ON (`vllm bench serve`, +decode-dominated) recorded in the benchmark record; the async decode is host- +orchestration-bound (GPU ~14 W at 96% "util"). RESIDUALS: Option A (out-of-graph +device-staged H2D, vLLM states.py:64) is the faithful follow-up; the eager +`exec_state_` 2-deep ring was not needed (drain-on-eager retained). Evidence +`dgx:~/work/mirror-ab/serve-async-executor/{gate2*.log,speed3*,diag*}`, commit +`fa971248` on `row/SERVE-ASYNC-EXECUTOR`. + diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index fc77d3bfe..89f0aeb64 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -89,8 +89,11 @@ token_ids` device gather). The full drain is byte-exact and UAF-safe and is 2026-08-06, speed-NEUTRAL** (same-binary): c16 OFF median 2305.8 vs ON 2303.3 (0.999x), c32 2928.9 vs 2919.1 (0.997x), bands overlap. It is a drain MOVE (relocate the drain past the host prep, not remove it), so it overlaps only the small host -prep; the drain still serializes GPU input staging, so c16 does not recover. Real -c16 recovery needs the drain REMOVAL plus double-buffered `exec_state_`/block-table. +prep; the drain still serializes GPU input staging, so c16 does not recover. The +drain REMOVAL is `row/SERVE-ASYNC-EXECUTOR` (`VT_ASYNC_EXECUTOR`, landed default-OFF): +a 2-slot decode-graph ring + reuse event, proven token-exact (async GREEN 5/5 c32, +SACRED 3/3); hazard-C is real-by-construction but empirically unreproducible on GB10 +so it stays OFF (record has detail + the Option A follow-up). **But the mirror FIXES a shipping correctness bug, so it is now DEFAULT ON (ROW-SERVE-ASYNC-LLM, 2026-08-06).** Baseline async (AsyncLLM depth-2) batch-1 greedy diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 57ee479ae..9bc4f8a1c 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -53,6 +53,7 @@ portable/reference path. In normal operation leave them unset. | `VT_ASYNC_RUNNER` | on | Synchronous model runner (no async/overlap execution) | | `VT_ASYNC_SCHED` | on | Synchronous scheduling (no scheduler/execution overlap). The documented first-line workaround for a suspected scheduling bug | | `VT_ASYNC_DEVICE_MIRROR` | on | The CUDA device-resident sampled-token mirror (ENG-ASYNC-SCHED W4): the async serving loop's sampled ids stay on the device instead of round-tripping the host. On a DISCRETE GPU this removes the host-fallback main-stream `Synchronize`. On the INTEGRATED GB10 it moves `last_sampled_tokens` off the host array `update_states`' condense reorders (a drain move, not a removal). DEFAULT ON since the 2026-08-06 correctness flip (ROW-SERVE-ASYNC-LLM): the OFF path's host-array combine writes `step.input_token_ids` on the main queue while the decode graph reads it on the CPU without a sync, so async batch-1 greedy decode nondeterministically degenerates into token-0 garbage; the mirror routes the ids into the embed on-queue and fixes it. `=0` is the rollback to the (racy) host-array path. Speed-neutral (c16 0.999x). No effect on CPU or the sync `LLMEngine` | +| `VT_ASYNC_EXECUTOR` | off (opt-in) | `=1` enables the decode-graph slot double-buffer (the c16/c32 overlap unlock, ENG-ASYNC-SCHED). The Qwen3.5 MoE/dense decode-graph drivers keep TWO persistent SizeSlots per padded decode size (a parity ring), alternated each step, each host-waited on its own blocking reuse event before its persistent host inputs are refreshed. That per-slot guard replaces the depth-2 pre-forward `Synchronize` (which the runner then skips whenever the previous step's logits are a non-owning graph-slot view), letting step N+1's host prep + replay overlap step N's GPU tail. Default OFF routes through the single-slot driver with the drain intact — byte-identical to production. No effect on CPU or the sync `LLMEngine`. `=1` doubles the captured decode-graph memory (~+76 MB, logits-dominated) | | `VLLM_CPP_CUDAGRAPH` | on (CUDA) | Eager launches instead of a captured CUDA graph | | `VLLM_CPP_DENSE_DECODE_GRAPH` | on (CUDA dense) | Non-graphed dense decode | | `VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH` | off (opt-in) | `=1` routes pure-decode steps for the SHARED pure-dense forward (`Qwen3DenseModel`, i.e. Qwen3 / Llama / InternLM3 / Mistral / InternLM2 `ForCausalLM`) through the captured decode CUDA graph; default OFF keeps the byte-identical eager decode. Token-exact with eager (dgx SACRED near-tie gate, Qwen3-0.6B/4B). Honors `VLLM_CPP_CUDAGRAPH=0` | diff --git a/docs/STATUS.md b/docs/STATUS.md index 958d86724..d593dc141 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -52,7 +52,7 @@ token-for-token correctness against the pinned oracle. | Capability | State | Notes | |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | -| Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC + ASYNC (`test_qwen36_async_serving` RED→GREEN); c4 1.025x, c16 0.932x; loop levers NEGATIVE/NEUTRAL (drain-sync, intake-drain re-splits only); fixes = prefill speed + drain-removal/double-buffer | +| Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; decode-graph double-buffer (`VT_ASYNC_EXECUTOR`, OFF) landed, ring correct (GREEN 5/5 c32, SACRED 3/3), RED unreproducible → OFF; residual prefill glue + Option A | | Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | diff --git a/include/vllm/model_executor/models/qwen3_5.h b/include/vllm/model_executor/models/qwen3_5.h index 6d5313481..538e005c6 100644 --- a/include/vllm/model_executor/models/qwen3_5.h +++ b/include/vllm/model_executor/models/qwen3_5.h @@ -102,6 +102,13 @@ struct ForwardLogits { vt::Tensor device_tensor; // [rows, vocab] view, valid iff on_device() int64_t rows = 0; int64_t vocab = 0; + // True when device_storage is a NON-owning view over an externally-held buffer + // (the decode-graph slot's persistent logits — ViewDeviceLogits). The deleter is + // a no-op, so releasing this carrier frees nothing: the runner reads it to decide + // whether resetting exec_state_ would free an in-flight-read pool block (the eager + // WrapDeviceLogits case, hazard-A) or not (this graphed case). See the + // VT_ASYNC_EXECUTOR drain-skip in runner.cpp. + bool non_owning_view = false; bool on_device() const { return static_cast(device_storage); } }; diff --git a/include/vllm/v1/worker/gpu/runner.h b/include/vllm/v1/worker/gpu/runner.h index f524afc05..5b79710d9 100644 --- a/include/vllm/v1/worker/gpu/runner.h +++ b/include/vllm/v1/worker/gpu/runner.h @@ -484,6 +484,16 @@ class GPUModelRunner final : public ModelRunnerBase { // flip (ROW-SERVE-ASYNC-LLM P0); VT_ASYNC_DEVICE_MIRROR=0 is the rollback. bool async_device_mirror() const; mutable int async_device_mirror_cached_ = -1; // -1 unknown, 0 no, 1 yes + // VT_ASYNC_EXECUTOR (default OFF): the decode-graph slot double-buffer lever. + // When on AND the previous step's stashed logits are a NON-owning decode-graph + // slot view (non_owning_view), the depth-2 drain before the forward is skipped — + // hazard-A (the eager owning-logits reset UAF) is absent for a slot view, and + // hazard-C (the persistent decode-graph host inputs being overwritten while the + // previous replay still reads them) is instead guarded inside the model by a + // 2-slot parity ring + per-slot reuse event. OFF routes through today's single- + // slot code with the drain intact (byte-identical). Memoized like the mirror. + bool async_executor() const; + mutable int async_executor_cached_ = -1; // -1 unknown, 0 no, 1 yes // Push the recorded InputBatch structural edits (seed/move/swap) to the device // mirror in stream order, then clear the log. No-op without a mirror. void replay_last_sampled_ops(AsyncDeviceInputs& dev); diff --git a/include/vt/backend.h b/include/vt/backend.h index f3ac7ecfb..21d884a82 100644 --- a/include/vt/backend.h +++ b/include/vt/backend.h @@ -82,7 +82,13 @@ class Backend { // Cross-stream event lifecycle. Base implementations are no-ops returning a // null-handle Event (synchronous backends have nothing to wait on). - virtual Event CreateEvent(); + // `blocking` requests an event whose HOST wait (SynchronizeEvent) SLEEPS the + // calling thread until completion instead of busy-spinning (CUDA: + // cudaEventBlockingSync). Used by the decode-graph slot double-buffer + // (VT_ASYNC_EXECUTOR): the reuse wait is nearly always already-signaled at + // depth-2, so on the rare occasion the engine runs ahead the host should sleep, + // not burn a core spinning. Ignored on synchronous backends (no handle). + virtual Event CreateEvent(bool blocking = false); virtual void DestroyEvent(Event& e); // Record `e` on the queue's stream: it completes once all work submitted to // `q` up to this point has finished (async_utils.py copy_event.record). diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index 4c4453c43..c77fc8f67 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -184,3 +184,11 @@ VT_LAGUNA_SWA_WINDOW VT_LAGUNA_KV_BF16 VT_LAGUNA_KV_HEADROOM VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH +# Test-only escape hatch: skip the depth-2 drain (VT_ASYNC_EXECUTOR=1) while +# forcing the decode-graph double-buffer OFF, so the async-serving gate SEES +# hazard-C (the RED arm proof). Never set in production. +VT_ASYNC_EXECUTOR_NO_DBUF +# Test-only: deterministic hazard-C proof (poison the persistent host inputs right +# after a captured replay) and a drain-skip counter. Never set in production. +VT_ASYNC_EXECUTOR_POISON +VT_ASYNC_EXECUTOR_TRACE diff --git a/src/vllm/model_executor/models/qwen3_5.cpp b/src/vllm/model_executor/models/qwen3_5.cpp index e504c09ad..2d31b00ed 100644 --- a/src/vllm/model_executor/models/qwen3_5.cpp +++ b/src/vllm/model_executor/models/qwen3_5.cpp @@ -6171,6 +6171,7 @@ static ForwardLogits ViewDeviceLogits(void* base, vt::Device device, int64_t row fl.device_tensor = MakeTensor(base, DType::kF32, device, {rows, vocab}); // Non-owning: keep on_device() true without taking ownership of `base`. fl.device_storage = std::shared_ptr(base, [](void*) {}); + fl.non_owning_view = true; // releasing this carrier frees nothing (runner drain-skip signal) return fl; } @@ -7491,6 +7492,42 @@ void BuildPaddedDecode(int64_t S, const std::vector& tok, } // namespace +// VT_ASYNC_EXECUTOR decode-graph slot double-buffer (parity ring) — the MODEL side +// of the c16/c32 overlap unlock. Two SizeSlots per padded decode size, alternated +// each step, each host-waited on its own reuse event before its persistent host +// inputs are refreshed. That guard replaces the runner's depth-2 drain (which the +// runner then skips), letting step N+1's host prep + replay overlap step N's GPU +// tail. DEFAULT OFF: unset/anything-but-"1" ⇒ single-slot (slot[0]), no events, +// byte-identical to the pre-lever driver. VT_ASYNC_EXECUTOR_NO_DBUF=1 is a TEST- +// ONLY escape hatch that forces the ring OFF while the runner still skips the drain +// (VT_ASYNC_EXECUTOR=1), so the async-serving gate SEES hazard-C in the RED arm +// (drain skipped, no double-buffer ⇒ corruption). Production never sets it. +static bool DecodeGraphDoubleBufferEnabled() { + const char* v = std::getenv("VT_ASYNC_EXECUTOR"); + if (v == nullptr || v[0] != '1' || v[1] != '\0') return false; + const char* nod = std::getenv("VT_ASYNC_EXECUTOR_NO_DBUF"); + if (nod != nullptr && nod[0] == '1' && nod[1] == '\0') return false; + return true; +} + +// Test-only (VT_ASYNC_EXECUTOR_POISON): immediately after a captured ReplayGraph, +// overwrite the persistent host inputs the replay's baked H2D + EmbedInto read, +// WITHOUT syncing. If the replay reads them asynchronously — the premise of +// hazard-C — this corrupts THIS step deterministically, so the token-exact async +// gate fails, proving the drain/ring is load-bearing WITHOUT depending on the +// natural Refresh-vs-replay race's narrow timing window. Templated because the two +// decode-graph drivers nest their own SizeSlot with identical field names. Never +// set in production (gated by an Impl `poison` member read once at construction, so +// the default hot path pays no getenv). +template +static void MaybePoisonHostInputs(bool poison, Slot& s) { + if (!poison) return; + std::fill(s.token_ids.begin(), s.token_ids.end(), 0); + std::fill(s.positions.begin(), s.positions.end(), 0); + std::fill(s.attn_meta.slot_mapping.begin(), s.attn_meta.slot_mapping.end(), 0); + std::fill(s.attn_meta.seq_lens.begin(), s.attn_meta.seq_lens.end(), 0); +} + struct Qwen3_5DecodeGraph::Impl { Impl(const Qwen3_5MoeWeights& w, const HfConfig& c, vt::Queue q, int64_t max_reqs) @@ -7501,11 +7538,16 @@ struct Qwen3_5DecodeGraph::Impl { enabled = env_on && vllm::platforms::GetPlatform(queue.device.type).support_static_graph_mode() && b.SupportsGraphCapture(); + dbuf = enabled && DecodeGraphDoubleBufferEnabled(); + poison = enabled && std::getenv("VT_ASYNC_EXECUTOR_POISON") != nullptr; } ~Impl() { Backend& b = vt::GetBackend(queue.device.type); for (auto& kv : slots) - if (kv.second.graph != nullptr) b.DestroyGraph(kv.second.graph); + for (auto& s : kv.second.slot) { + if (s.graph != nullptr) b.DestroyGraph(s.graph); + if (s.reuse_event.handle != nullptr) b.DestroyEvent(s.reuse_event); + } } // One captured padded batch size. Owns its OWN persistent host inputs (the @@ -7525,6 +7567,11 @@ struct Qwen3_5DecodeGraph::Impl { bool captured = false; bool warm = false; int64_t replays = 0; + // VT_ASYNC_EXECUTOR parity-ring reuse guard: recorded on the main queue after + // this slot's replay, host-waited before its next Refresh so the replay's baked + // H2D of the persistent host inputs can never race that overwrite (hazard-C). + // Null-handle (unused) unless the double-buffer is on. Blocking-sync flavor. + vt::Event reuse_event{}; // In-place refresh of the persistent host inputs (fixed addresses once the // slot's vectors reach size S) so a replay re-reads this step's tokens. @@ -7577,8 +7624,18 @@ struct Qwen3_5DecodeGraph::Impl { vt::Queue queue; int64_t max_num_reqs = 0; // == max_num_seqs; padded decode batch cap bool enabled = false; - - std::map slots; // padded size S -> slot + bool dbuf = false; // VT_ASYNC_EXECUTOR parity ring (2 slots/size + events) + bool poison = false; // VT_ASYNC_EXECUTOR_POISON: deterministic hazard-C proof + + // The parity ring: two independent SizeSlots per padded size, alternated per + // step (`next`). OFF (dbuf==false) always uses slot[0] with no event — the + // second slot stays default-constructed (no graph, no buffers) and the driver is + // byte-identical to the single-slot original. + struct SlotRing { + SizeSlot slot[2]; + int next = 0; + }; + std::map slots; // padded size S -> parity ring int64_t replays = 0; // total replays (diagnostics) bool any_captured = false; // diagnostics: at least one live graph }; @@ -7625,8 +7682,26 @@ ForwardLogits Qwen3_5DecodeGraph::Step( } // Pad this step's real B-request inputs up to S (inert padding rows), then - // refresh THIS size's persistent host buffers in place. - Impl::SizeSlot& s = impl_->slots[S]; + // refresh THIS size's persistent host buffers in place. VT_ASYNC_EXECUTOR: pick + // this step's slot from the size's parity ring (alternating), and host-wait its + // previous replay before Refresh touches its persistent host inputs (hazard-C). + // OFF: always slot[0], no wait — byte-identical to the single-slot driver. + Impl::SlotRing& ring = impl_->slots[S]; + const bool dbuf = impl_->dbuf; + Impl::SizeSlot& s = ring.slot[dbuf ? ring.next : 0]; + if (dbuf) { + ring.next ^= 1; + if (s.reuse_event.handle != nullptr) b.SynchronizeEvent(s.reuse_event); + } + // Record this slot's reuse event on the main queue after its replay/forward is + // enqueued, so the next same-slot Refresh host-waits until the replay's baked + // H2D of these host buffers has completed. No-op unless the double-buffer is on. + const auto record_reuse = [&] { + if (!dbuf) return; + if (s.reuse_event.handle == nullptr) + s.reuse_event = b.CreateEvent(/*blocking=*/true); + b.RecordEvent(s.reuse_event, impl_->queue); + }; const int cols = attn_meta.block_table_num_cols; std::vector ptok, ppos; v1::CommonAttentionMetadata pam; @@ -7652,6 +7727,8 @@ ForwardLogits Qwen3_5DecodeGraph::Step( if (s.captured) { EmbedInto(d, *s.hidden, s.token_ids, impl_->weights, impl_->config); b.ReplayGraph(impl_->queue, s.graph); + record_reuse(); + MaybePoisonHostInputs(impl_->poison, s); ++s.replays; ++impl_->replays; return ViewDeviceLogits(s.logits->ptr(), d.q.device, B, vocab); @@ -7660,6 +7737,21 @@ ForwardLogits Qwen3_5DecodeGraph::Step( // Warm: the pool + residency were warmed for this size by the previous (eager) // step. CAPTURE the layer region once, instantiate the graph, then launch it. if (s.warm) { + // dbuf: the runner may have skipped the depth-2 drain (the previous step + // returned a slot view), so a prior replay can still be in flight. Capture must + // begin on an idle stream — drain once here. One-time (≤2 captures per size); + // steady-state replay never captures, so this never touches the overlap path. + if (dbuf) { + b.Synchronize(impl_->queue); + // Pre-grow the pool for THIS slot's RETAINED [S,vocab] logits block while the + // OTHER ring slot's logits is held, so the captured logits alloc is a pool HIT + // (a cudaMalloc mid-capture aborts it). Working scratch is freed at + // ForwardLayers return and SAFELY shared between the two graphs — they replay + // sequentially on one stream, so only the retained logits needs two live + // copies. Alloc+free forces the (out-of-capture) growth; the block returns to + // the free list for the capture's own allocation to hit. + { DBuf pregrow(d, DType::kF32, std::vector{S, vocab}); } + } EmbedInto(d, *s.hidden, s.token_ids, impl_->weights, impl_->config); b.BeginCapture(impl_->queue); DBuf lg = ForwardLayers(d, s.hidden->t(), s.positions, s.attn_meta, @@ -7670,6 +7762,7 @@ ForwardLogits Qwen3_5DecodeGraph::Step( s.captured = true; impl_->any_captured = true; b.ReplayGraph(impl_->queue, s.graph); + record_reuse(); s.replays = 1; ++impl_->replays; return ViewDeviceLogits(s.logits->ptr(), d.q.device, B, vocab); @@ -7685,6 +7778,7 @@ ForwardLogits Qwen3_5DecodeGraph::Step( attn_kv, gdn_state, impl_->weights, impl_->config); s.warm = true; s.captured = false; + record_reuse(); // lg is [S,vocab]; hand ownership out but expose only the first B (real) rows. ForwardLogits fl = WrapDeviceLogits(d, std::move(lg), vocab); if (fl.rows != B) { @@ -7716,6 +7810,8 @@ struct Qwen3_5DenseDecodeGraph::Impl { enabled = env_on && vllm::platforms::GetPlatform(queue.device.type).support_static_graph_mode() && b.SupportsGraphCapture(); + dbuf = enabled && DecodeGraphDoubleBufferEnabled(); + poison = enabled && std::getenv("VT_ASYNC_EXECUTOR_POISON") != nullptr; } ~Impl() { if (std::getenv("VT_DECODE_GRAPH_STATS") != nullptr) @@ -7724,7 +7820,10 @@ struct Qwen3_5DenseDecodeGraph::Impl { static_cast(replays), slots.size()); Backend& b = vt::GetBackend(queue.device.type); for (auto& kv : slots) - if (kv.second.graph != nullptr) b.DestroyGraph(kv.second.graph); + for (auto& s : kv.second.slot) { + if (s.graph != nullptr) b.DestroyGraph(s.graph); + if (s.reuse_event.handle != nullptr) b.DestroyEvent(s.reuse_event); + } } // One captured padded batch size (mirror of Qwen3_5DenseDecodeGraph SizeSlot). @@ -7740,6 +7839,11 @@ struct Qwen3_5DenseDecodeGraph::Impl { bool captured = false; bool warm = false; int64_t replays = 0; + // VT_ASYNC_EXECUTOR parity-ring reuse guard: recorded on the main queue after + // this slot's replay, host-waited before its next Refresh so the replay's baked + // H2D of the persistent host inputs can never race that overwrite (hazard-C). + // Null-handle (unused) unless the double-buffer is on. Blocking-sync flavor. + vt::Event reuse_event{}; // In-place refresh of the persistent host inputs (fixed addresses once the // slot's vectors reach size S) so a replay re-reads this step's tokens. @@ -7792,8 +7896,18 @@ struct Qwen3_5DenseDecodeGraph::Impl { vt::Queue queue; int64_t max_num_reqs = 0; // == max_num_seqs; padded decode batch cap bool enabled = false; - - std::map slots; // padded size S -> slot + bool dbuf = false; // VT_ASYNC_EXECUTOR parity ring (2 slots/size + events) + bool poison = false; // VT_ASYNC_EXECUTOR_POISON: deterministic hazard-C proof + + // The parity ring: two independent SizeSlots per padded size, alternated per + // step (`next`). OFF (dbuf==false) always uses slot[0] with no event — the + // second slot stays default-constructed (no graph, no buffers) and the driver is + // byte-identical to the single-slot original. + struct SlotRing { + SizeSlot slot[2]; + int next = 0; + }; + std::map slots; // padded size S -> parity ring int64_t replays = 0; // total replays (diagnostics) bool any_captured = false; // diagnostics: at least one live graph }; @@ -7838,8 +7952,26 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( } // Pad this step's real B-request inputs up to S (inert padding rows), then - // refresh THIS size's persistent host buffers in place. - Impl::SizeSlot& s = impl_->slots[S]; + // refresh THIS size's persistent host buffers in place. VT_ASYNC_EXECUTOR: pick + // this step's slot from the size's parity ring (alternating), and host-wait its + // previous replay before Refresh touches its persistent host inputs (hazard-C). + // OFF: always slot[0], no wait — byte-identical to the single-slot driver. + Impl::SlotRing& ring = impl_->slots[S]; + const bool dbuf = impl_->dbuf; + Impl::SizeSlot& s = ring.slot[dbuf ? ring.next : 0]; + if (dbuf) { + ring.next ^= 1; + if (s.reuse_event.handle != nullptr) b.SynchronizeEvent(s.reuse_event); + } + // Record this slot's reuse event on the main queue after its replay/forward is + // enqueued, so the next same-slot Refresh host-waits until the replay's baked + // H2D of these host buffers has completed. No-op unless the double-buffer is on. + const auto record_reuse = [&] { + if (!dbuf) return; + if (s.reuse_event.handle == nullptr) + s.reuse_event = b.CreateEvent(/*blocking=*/true); + b.RecordEvent(s.reuse_event, impl_->queue); + }; const int cols = attn_meta.block_table_num_cols; std::vector ptok, ppos; v1::CommonAttentionMetadata pam; @@ -7869,6 +8001,8 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( static_cast(s.replays)); #endif b.ReplayGraph(impl_->queue, s.graph); + record_reuse(); + MaybePoisonHostInputs(impl_->poison, s); ++s.replays; ++impl_->replays; return ViewDeviceLogits(s.logits->ptr(), d.q.device, B, vocab); @@ -7877,6 +8011,20 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( // Warm: the pool + residency were warmed for this size by the previous (eager) // step. CAPTURE the dense layer region once, instantiate the graph, launch it. if (s.warm) { + // dbuf: the runner may have skipped the depth-2 drain (the previous step + // returned a slot view), so a prior replay can still be in flight. Capture must + // begin on an idle stream — drain once here. One-time (≤2 captures per size); + // steady-state replay never captures, so this never touches the overlap path. + if (dbuf) { + b.Synchronize(impl_->queue); + // Pre-grow the pool for THIS slot's RETAINED [S,vocab] logits block while the + // OTHER ring slot's logits is held, so the captured logits alloc is a pool HIT + // (a cudaMalloc mid-capture aborts it). Working scratch is freed at + // DenseForwardLayers return and SAFELY shared between the two graphs — they + // replay sequentially on one stream, so only the retained logits needs two + // live copies. + { DBuf pregrow(d, DType::kF32, std::vector{S, vocab}); } + } DenseEmbedInto(d, *s.hidden, s.token_ids, impl_->weights, impl_->config); b.BeginCapture(impl_->queue); DBuf lg = DenseForwardLayers(d, s.hidden->t(), s.positions, s.attn_meta, @@ -7891,6 +8039,7 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( "for padded size S=%lld (real B=%lld)\n", static_cast(S), static_cast(B)); b.ReplayGraph(impl_->queue, s.graph); + record_reuse(); s.replays = 1; ++impl_->replays; return ViewDeviceLogits(s.logits->ptr(), d.q.device, B, vocab); @@ -7907,6 +8056,7 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( impl_->config); s.warm = true; s.captured = false; + record_reuse(); // lg is [S,vocab]; hand ownership out but expose only the first B (real) rows. ForwardLogits fl = WrapDeviceLogits(d, std::move(lg), vocab); if (fl.rows != B) { diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 113429b15..a62024b74 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -1172,7 +1172,32 @@ std::optional GPUModelRunner::execute_model( // was already cleared at the top and exec_state_ already reset, so this block is // skipped (mirror==false) and the path is byte-identical. if (mirror) { - if (async_forward_in_flight_) { + // VT_ASYNC_EXECUTOR (decode-graph slot double-buffer): if the previous step's + // stashed logits are a NON-OWNING decode-graph slot view, the exec_state_ reset + // frees nothing the in-flight async sampler reads (hazard-A — the eager owning + // WrapDeviceLogits pool block — is absent for a view), and the model's 2-slot + // parity ring + per-slot reuse event already guard the persistent decode-graph + // host inputs against this step's replay (hazard-C). Both hazards handled, so + // SKIP the main-queue Synchronize and let this step's host prep + replay overlap + // the previous step's GPU tail — the c16/c32 unlock. An eager/mixed previous + // step still owns its logits pool block, whose reset WOULD UAF the async + // sampler, so it drains exactly as before. The reset runs on BOTH paths: on the + // skip path it releases the no-op-deleter view and clears req_ids/discard for + // this step's stash WITHOUT touching the still-live, still-referenced slot + // buffer. Inert unless VT_ASYNC_EXECUTOR=1 (async_executor() default OFF), where + // skip_drain is always false and the block is byte-identical to the drain. + const bool skip_drain = async_executor() && exec_state_.logits.on_device() && + exec_state_.logits.non_owning_view; + if (skip_drain && std::getenv("VT_ASYNC_EXECUTOR_TRACE") != nullptr) { + // Diagnostic (test-only): confirm the overlap path is actually engaged — a + // skip count of 0 would mean async_executor()/non_owning_view never resolved + // and the lever is inert. Single engine thread, so a plain static suffices. + static long long kSkips = 0; + ++kSkips; + if (kSkips == 1 || kSkips % 200 == 0) + std::fprintf(stderr, "[VT_ASYNC_EXECUTOR] drain skipped x%lld\n", kSkips); + } + if (!skip_drain && async_forward_in_flight_) { vt::GetBackend(queue_.device.type).Synchronize(queue_); async_forward_in_flight_ = false; } @@ -2026,6 +2051,25 @@ bool GPUModelRunner::async_device_mirror() const { return on; } +// VT_ASYNC_EXECUTOR (decode-graph slot double-buffer, ENG-ASYNC-SCHED c16/c32 +// overlap unlock). DEFAULT OFF — an opt-in speed lever. Engages only where the +// depth-2 moved drain and the decode graph both exist: the device mirror path on +// a real CUDA GPU. On the CPU backend / mirror-OFF path the drain does not move +// and there is no decode graph, so the lever is inert. Memoized. Reading a plain +// "not 0" env keeps a value of "1" the canonical enable; anything else (unset, +// "0") leaves the drain in place, which is byte-identical production. +bool GPUModelRunner::async_executor() const { + if (async_executor_cached_ >= 0) return async_executor_cached_ != 0; + bool on = false; +#ifdef VLLM_CPP_CUDA + const char* value = std::getenv("VT_ASYNC_EXECUTOR"); + on = value != nullptr && value[0] == '1' && value[1] == '\0' && + async_device_mirror(); +#endif + async_executor_cached_ = on ? 1 : 0; + return on; +} + GPUModelRunner::AsyncDeviceInputs* GPUModelRunner::get_or_create_async_device_inputs() { if (!async_device_mirror()) return nullptr; diff --git a/src/vt/backend.cpp b/src/vt/backend.cpp index 1ab9101df..3519d77e0 100644 --- a/src/vt/backend.cpp +++ b/src/vt/backend.cpp @@ -18,7 +18,7 @@ uint64_t NextQueueId() noexcept { // CUDA overrides all six with cudaHostAlloc + cudaEvent_t. (CPU inherits these.) void* Backend::AllocPinned(size_t bytes) { return Alloc(bytes == 0 ? 1 : bytes); } void Backend::FreePinned(void* p) { Free(p); } -Event Backend::CreateEvent() { return Event{}; } +Event Backend::CreateEvent(bool /*blocking*/) { return Event{}; } void Backend::DestroyEvent(Event&) {} void Backend::RecordEvent(Event&, Queue&) {} void Backend::SynchronizeEvent(Event&) {} diff --git a/src/vt/cuda/cuda_backend.cu b/src/vt/cuda/cuda_backend.cu index 30bf49dc4..aeb8fd342 100644 --- a/src/vt/cuda/cuda_backend.cu +++ b/src/vt/cuda/cuda_backend.cu @@ -124,12 +124,17 @@ class CudaBackend final : public Backend { void FreePinned(void* p) override { if (p != nullptr) Check(cudaFreeHost(p), "cudaFreeHost"); } - Event CreateEvent() override { + Event CreateEvent(bool blocking = false) override { cudaEvent_t ev = nullptr; // cudaEventDisableTiming: we only ever wait on completion, never measure — // this is the cheaper synchronization-only event (mirrors torch.Event()). - Check(cudaEventCreateWithFlags(&ev, cudaEventDisableTiming), - "cudaEventCreateWithFlags"); + // cudaEventBlockingSync (opt-in): a HOST cudaEventSynchronize on this event + // sleeps the thread until completion instead of busy-spinning — the decode- + // graph slot-reuse wait (VT_ASYNC_EXECUTOR) is almost always already done, so + // spinning would only waste a core on the rare run-ahead. + const unsigned int flags = + cudaEventDisableTiming | (blocking ? cudaEventBlockingSync : 0u); + Check(cudaEventCreateWithFlags(&ev, flags), "cudaEventCreateWithFlags"); return Event{Device{DeviceType::kCUDA, device_}, reinterpret_cast(ev)}; } void DestroyEvent(Event& e) override { diff --git a/tests/parity/test_qwen36_async_serving.cpp b/tests/parity/test_qwen36_async_serving.cpp index 2ec9f96f1..b36de26c8 100644 --- a/tests/parity/test_qwen36_async_serving.cpp +++ b/tests/parity/test_qwen36_async_serving.cpp @@ -159,13 +159,27 @@ TEST_CASE("qwen36 async-serving greedy token-exact gate (dgx-only, 35B) — " CHECK(got == want_greedy_ids); } - // ── ARM 2: small CONCURRENCY bracket ────────────────────────────────────── + // ── ARM 2: CONCURRENCY bracket ──────────────────────────────────────────── // N independent greedy requests submitted together so the engine runs them as // pure-decode batched steps (num_reqs==N) through the batched decode graph + // async combine while the depth-2 loop pipelines. Each request is independent // (own KV + GDN state), so each MUST reproduce the same oracle continuation // regardless of the step interleave (vLLM's greedy determinism guarantee). - constexpr int kN = 4; + // + // Default N=4 is a SMALL bracket. The decode-graph slot-reuse race (hazard-C: + // Refresh(step) overwriting a slot's persistent host inputs while the previous + // same-slot replay's baked H2D still reads them) only bites when the HOST runs + // AHEAD of the GPU — i.e. at higher concurrency, where a batched replay is heavy + // enough that step N+1's Refresh reaches the slot while step N's replay is still + // queued/executing. VT_ASYNC_SERVING_CONC overrides N so the VT_ASYNC_EXECUTOR + // double-buffer's RED (drain skipped, ring off -> divergence) and GREEN (ring on + // -> token-exact) can be exercised at the concurrency where the hazard is live. + int kN = 4; + if (const char* c = std::getenv("VT_ASYNC_SERVING_CONC")) { + const int v = std::atoi(c); + if (v > 0) kN = v; + } + MESSAGE("qwen36_async_serving: concurrency bracket N=" << kN); std::vector reqs; reqs.reserve(static_cast(kN)); for (int i = 0; i < kN; ++i) {