mudler · localai-bot · Aug 5, 2026 · Aug 5, 2026 · Aug 5, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -20,8 +20,8 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Invocation-parity prevention | CI guard (`check-gemv-invocation-consistency.py`) + AGENTS.md checklist landing | Review + merge; CUDA build-verify `kGemvHeuristicAlgos` on dgx |
 | MiniMax-H3 lane | Portable path complete; e2e prompt-conditioned video on real weights (Thor). Speed = NVFP4 FP4 device path, sm_121-gated | PR #26 rebase + supports-audit synthesis (workflow ran; integrate) |
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **W7 device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`, `ACTIVE`): DBuf-resident `ForwardDeviceCompute` (2 host islands: KDA recurrence, NoPE-MLA softmax); `test_kimi_linear_forward` **12/12·614**; opt-in `VT_KIMI_DEVICE_COMPUTE=1` | GPU-verify: CUDA build, token-exact vs oracle, e2e §8 |
-| 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE lever **RESOLVED NEGATIVE**: attribution boundary (drain A/B collapses intake -91% but shifts to queued; arrival→sched GPU-bound, NEUTRAL). Probe kept, lever reverted | Real levers: prefill glue (task #61) + async device-resident executor (also the c16 fix) |
-| Async-serving decode bug **FIXED** | Batch-1 nondet token-0 garbage: combine device-write raced the decode-graph host read (SYNC SACRED missed it). FIX: `VT_ASYNC_DEVICE_MIRROR` **default ON**; gate `test_qwen36_async_serving` RED→GREEN, SACRED/UAF clean | MERGED (#31) |
+| 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE lever **RESOLVED NEGATIVE** (attribution boundary, drain A/B, sum invariant); probe kept, lever reverted | Real levers: prefill glue (task #61) + Option A (H2D out of capture) |
+| Async decode: mirror fix MERGED (#31); slot double-buffer (`VT_ASYNC_EXECUTOR`) landed gated **OFF** | Ring+reuse-event replaces the depth-2 drain; proven correct (async gate 5/5 @c32, SACRED 3/3, capture-malloc fixed); RED unreproducible, speed neutral in a host-bound regime | Option A follow-up; binding-harness A/B owed |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -11836,6 +11836,139 @@ Evidence `dgx:~/work/mirror-ab/{asyncgate2-{RED,GREEN}.log,
 asyncgate-SACRED-default.log,mab-defmeasure.log,mab-asyncmemcheck.log,
 evidence/raw/35/ours/c16-r{1,2,3}-{defaulton,rollback0}.json}`.
 
+## ROW-SERVE-ASYNC-EXECUTOR: decode-graph slot double-buffer (Option B), gated `VT_ASYNC_EXECUTOR` — the c16/c32 overlap unlock
+
+CONTEXT. With `VT_ASYNC_DEVICE_MIRROR` default ON (the ROW-SERVE-ASYNC-LLM fix
+above), the async serving loop's remaining serialization is a depth-2 pre-forward
+drain: `runner.cpp` (mirror path, before the forward) does a main-queue
+`Synchronize` every step. It exists because of hazard-C below. Removing it is the
+c16 (and c32) unlock — the overlap that lets step N+1's host prep + replay run
+while step N's GPU tail is still in flight.
+
+THE THREE HAZARDS (why the drain is there, and what each needs to be safe to skip).
+
+hazard-C (the drain's REASON; decode-graph only). The Qwen3.5 decode graph bakes
+its per-step H2D INSIDE the captured replay: `BuildStepDevInputs` (`qwen3_5.cpp`) is
+called inside the captured `ForwardLayers`, and it uploads
+positions / slot_mapping / block_table / seq_lens / query_start_loc / GDN state
+indices from the per-size `SizeSlot`'s PERSISTENT HOST vectors
+(`token_ids, positions, attn_meta.*, gdn_meta.*`). `s.Refresh` overwrites those same
+host vectors with a plain host `memcpy` (`CopyInPlace`), OFF the queue; `EmbedInto`
+reads `s.token_ids` via an on-queue async H2D. So `Replay(N)` reads `s.*` on the main
+queue while `Refresh(N+1)` overwrites them off-queue → corruption without an
+intervening drain. This is the blocker the drain solves and the reason a naive
+drain-removal degenerates (proven by the RED arm below).
+
+hazard-A (EAGER path only). `exec_state_.logits` OWNS a pool block on the eager
+forward (`WrapDeviceLogits`), freed by the pre-forward `exec_state_` reset while the
+deferred async sampler still reads it → use-after-free. The graphed path's
+`ViewDeviceLogits` is NON-owning (`non_owning_view=true`, no-op deleter), so resetting
+`exec_state_` frees nothing and the slot buffer stays live — hazard-A is absent for a
+graph-slot view.
+
+vLLM's structure (reference). `synchronize_input_prep` is a blocking event
+(gpu_model_runner.py); on-stream out-of-graph H2D into PERSISTENT DEVICE buffers
+(states.py:64) so `_update_states` never reads a device-written buffer and the sync
+only guards input staging. Option B below is the lower-risk equivalent (a per-slot
+host-wait ring instead of vLLM's out-of-graph device staging); it is recorded as a
+stepping stone to the faithful Option A (move the H2D OUT of the captured replay into
+persistent device buffers, then no host-buffer reuse hazard exists at all).
+
+OPTION B (what landed, gated `VT_ASYNC_EXECUTOR`, default OFF). (1) Each decode-graph
+driver's `slots` map goes from one `SizeSlot` per size to a `SlotRing{SizeSlot[2]}`
+(parity ring), alternated each step. Each slot records a BLOCKING-sync reuse event
+(`cudaEventBlockingSync`, new `Backend::CreateEvent(bool blocking)`) on the main queue
+AFTER its replay; before Refresh-ing a slot for reuse the host `SynchronizeEvent`s it.
+At depth-2 with 2 slots this wait is almost always already-signaled; it self-limits if
+the engine runs ahead. The warm-capture step drains once (idle-stream capture) since
+the runner may have skipped the drain. (2) `runner.cpp`: under `VT_ASYNC_EXECUTOR=1`,
+SKIP the drain's `Synchronize` when the previous step's logits are a non-owning
+graph-slot view (hazard-A absent, hazard-C handled by the ring); the `exec_state_`
+reset still runs (releasing the no-op-deleter view). Eager/mixed previous steps still
+own their logits and drain as before — the drain-on-eager fallback is acceptable
+because c16/c32 steady-state is all-graphed. (3) OFF routes through the single-slot
+code with the drain intact — byte-identical by construction (`dbuf=false` picks
+`slot[0]` with no events; `skip_drain` is always false). Memory: doubling slots
+≈ +76 MB (logits-dominated), one-time; capture time doubles (one-time).
+
+RESULTS (dgx GB10 sm_121a, cutlass-4.5.0, triton ON, commit `fa971248`, dual-lock,
+free>=90, worker down; evidence `dgx:~/work/mirror-ab/serve-async-executor/`).
+
+CPU (bar a): -Werror clean; runner / llm_engine / engine_core_proc / async_llm /
+input_batch 6/6 pass.
+
+CAPTURE-SAFETY (a real bug found + fixed). The FIRST ring build aborted every
+captured step with `cudaMalloc: operation not permitted when stream is capturing`.
+Root cause: the pool is pre-warmed for ONE graph's RETAINED `[S,vocab]` logits block
+(allocated inside `ForwardLayers`/`DenseForwardLayers` and kept as `s.logits`); the
+ring needs TWO retained simultaneously (one per slot), so the second slot's capture
+hit a pool MISS. Working scratch (`hidden`/`res`/MoE grouped-GEMM workspace) is freed
+at ForwardLayers return and SAFELY shared between the two graphs — they replay
+SEQUENTIALLY on the one main stream, so only the retained logits needs two live
+copies. FIX: in the warm-capture branch, when dbuf, pre-grow the pool with a
+throwaway `[S,vocab]` f32 alloc+free (out-of-capture growth) so the captured logits
+alloc is a pool HIT. `fix-check` (ring ON, conc-4): PASS, 0 `capturing` lines.
+
+HAZARD-C IS REAL BY CONSTRUCTION, but its empirical RED does NOT reproduce on GB10.
+The captured decode graph re-reads its persistent host input buffers at EVERY replay
+(that is HOW `Refresh` updates each step's inputs; if it did not re-read, every
+replay would emit identical tokens, which it does not) — so overwriting them while a
+replay is in flight corrupts it, and the drain/ring is load-bearing IN PRINCIPLE. But
+the RED arm (drain skipped, ring OFF via `VT_ASYNC_EXECUTOR_NO_DBUF`) PASSED 3/3 at
+conc-4 AND 3/3 at conc-32 (`VT_ASYNC_SERVING_CONC=32`), and the deterministic POISON
+arm (overwrite the host inputs right AFTER `ReplayGraph`) also PASSED at conc-4. The
+reason: the baked H2D is a TINY, FAST copy (~32 ints for positions/slot_mapping) at
+the very START of the replay, and on GB10 the GPU executes it before the host
+completes a full depth-2 iteration and reaches the next `Refresh` — the race window
+is microscopic and closed by pipeline latency. So the token-exact serving gate at any
+reachable concurrency can neither RED nor meaningfully protective-GREEN the hazard;
+the divergence would require the host to outrun the GPU inside that ~1us H2D window,
+which does not happen at conc<=32 on this box. Per the brief's STOP rule the natural
+RED is unmet; recorded as an honest partial, not shipped-ON.
+
+RING CORRECTNESS (proven, the token gates are the authority). GREEN (ring ON,
+conc-32) `test_qwen36_async_serving` 5/5 consecutive PASS. OFF baseline (drain on,
+conc-32) PASS. SACRED SYNC `test_qwen36_paged_engine` with the ring ON (the ring is
+active on the sync path too, and the sync path is DETERMINISTIC token-exact) 3/3
+PASS — the ring produces byte-identical output. The ring is CORRECT; the OFF default
+is byte-identical to production by construction (single-slot, drain intact).
+
+MEMORY (paged-engine single load, `free -m` sampled): peak used OFF 46.8 GB vs ON
+44.5 GB of the 119 GB pool — the ~40-80 MB slot-doubling is BELOW the free-g noise
+floor (system-used swings GBs run-to-run) so it is not resolvable, but it is well
+within budget and there is no OOM. SANITIZER (compute-sanitizer memcheck, ring ON,
+`VT_ASYNC_EXECUTOR=1`, x2): both `ERROR SUMMARY: 0 errors`, exit=0 — NO invalid memory
+accesses. (`--leak-check full` additionally flags 726 `Leaked` reports, but all are
+intentionally-resident LOAD-TIME weight buffers — `BuildMarlinDensePairResident` /
+`PrepareMarlinResident` — identical OFF vs ON and freed at teardown, not a ring
+defect; plain memcheck, matching the prior methodology, is clean.) The repeated token
+gates remain the capture-safety authority, per the brief.
+
+DEFAULT DECISION: **OFF** (opt-in `VT_ASYNC_EXECUTOR`). The brief flips ON only on a
+clean speed win AND the full correctness bar; the empirical RED could not be met
+(hazard real-by-construction but unreproducible on GB10), so the conservative default
+is OFF with this honest result recorded. The ring is proven correct + capture-safe
+and stands as the provably-safe drain-removal mechanism and the stepping stone to the
+faithful Option A (move the per-step H2D OUT of the captured replay into persistent
+DEVICE buffers, vLLM states.py:64 / _prepare_input_ids — then no host-buffer reuse
+hazard exists and no ring is needed). SPEED A/B (`vllm bench serve`,
+decode-dominated random 128-in/256-out, single server load per arm, dual-lock):
+NEUTRAL — c16 output tok/s OFF 50.3/50.8 vs ON 50.6 (~1.00x); c32 OFF 93.6 vs ON 93.4
+(~0.998x). The drain-skip IS engaged (`VT_ASYNC_EXECUTOR_TRACE` logs skips). But this
+regime is host-orchestration / HTTP-streaming bound (TPOT ~312-337 ms, GPU ~14 W at
+96% "util"), so it does not cleanly isolate the engine-level overlap, and it is a
+different workload from the binding online-serving grid (which reports ~2300 tok/s
+TOTAL at c16). No measurable win here. Combined with the unreproducible RED, the
+default is OFF. The prior mirror A/B (drain MOVE) was likewise c16-neutral for the
+same host-bound reason; the true c16 recovery is Option A (out-of-graph device
+staging), of which this ring is the correct-and-safe stepping stone.
+
+RESIDUALS: (1) Option A is the faithful follow-up. (2) the eager exec_state_ 2-deep
+ring was NOT needed (the drain-on-eager fallback is retained; eager/mixed steps still
+own their logits and drain). (3) test-only knobs `VT_ASYNC_EXECUTOR_NO_DBUF`
+(ring-off), `VT_ASYNC_EXECUTOR_POISON` (deterministic host-input poison),
+`VT_ASYNC_EXECUTOR_TRACE` (drain-skip counter), `VT_ASYNC_SERVING_CONC` (gate
+concurrency) are retained for reproducing this analysis and future Option-A work.
 ## Qwen3.5-4B revalidation after merging current upstream (2026-08-05)
 
 Merged `upstream/main` at `59674cf1d` into the branch as `312af21a9`, rebuilt

diff --git a/.agents/state.md b/.agents/state.md
@@ -35791,7 +35791,6 @@ including CI, and that is a separate switch with a separate blast radius.
 
 No source, kernel, model, gate, benchmark or capability mark changed.
 
-
 ## 35B c16 drain-sync lever A/B: blocking-event drain NEGATIVE (−1.9%), full drain KEPT; real fix = GPU-resident sampled tokens
 <!-- state: 2026-08-06T01:00 -->
 
@@ -36000,7 +35999,6 @@ interaction; (3) then decide the mirror default on correctness; (4) c16 speed st
 needs drain-removal + double-buffer. Evidence
 `dgx:~/work/mirror-ab/{mab-measure.log,mab-tokdiag.log,mab-prodcheck.log,evidence/raw/35/ours/c16-r{1,2,3}-abmirr{off,on}.json,greedy/*}`.
 
-
 ## ROW-SERVE-ASYNC-LLM P0 RESOLVED: root-caused the async batch-1 token-0 degeneration, added the missing async-serving token-exact gate (RED-first), flipped the device mirror default ON
 <!-- state: 2026-08-06T14:00 -->
 
@@ -36080,3 +36078,60 @@ in all reps and unchanged from history. Fresh node-level nsys traces contain
 graph child kernels on both arms. Evidence:
 `docs/bench-evidence/qwen35-4b-upstream-20260805.md`; raw root
 `/tmp/qwen35-upstream-312af21a9`.
+
+## SERVE-ASYNC-EXECUTOR: decode-graph slot double-buffer (Option B) landed gated default-OFF; ring PROVEN correct; hazard-C real-by-construction but empirical RED unreproducible on GB10
+<!-- state: 2026-08-06T18:30 -->
+
+The c16/c32 overlap unlock. With the mirror default ON (ROW-SERVE-ASYNC-LLM), the
+async serving loop's remaining serialization is the depth-2 pre-forward drain
+(`runner.cpp` mirror path). It exists for hazard-C: the Qwen3.5 decode graph bakes
+its per-step H2D INSIDE the captured replay (`BuildStepDevInputs` in `ForwardLayers`/
+`DenseForwardLayers` uploads positions/slot_mapping/block_table/seq_lens/qsl/GDN
+state indices from the per-size `SizeSlot`'s persistent HOST vectors), which
+`s.Refresh` overwrites off-queue with a plain memcpy — so `Replay(N)` reads `s.*` on
+the main queue while `Refresh(N+1)` overwrites them. hazard-A is EAGER-only
+(`exec_state_.logits` OWNS a pool block freed by the reset while the deferred sampler
+reads it); the graphed path's `ViewDeviceLogits` is NON-owning and stream-safe.
+
+OPTION B (landed, `VT_ASYNC_EXECUTOR`, default OFF): each decode-graph driver's
+`slots` map goes `map<size,SizeSlot>` -> `map<size,SlotRing{SizeSlot[2]}>` (parity
+ring, alternated per step); each slot records a BLOCKING-sync reuse event
+(`Backend::CreateEvent(bool blocking)` -> `cudaEventBlockingSync`) on the main queue
+after its replay, host-waited before its next `Refresh`. The runner SKIPS the drain
+under `VT_ASYNC_EXECUTOR=1` when the previous step's logits are a non-owning graph
+view (`ForwardLogits.non_owning_view`); eager/mixed steps still own their logits and
+drain. OFF routes through single-slot code with the drain intact — byte-identical by
+construction (`dbuf=false` picks `slot[0]`, no events; `skip_drain` always false).
+
+CAPTURE-SAFETY BUG (found + fixed). First ring build aborted every captured step with
+`cudaMalloc ... when stream is capturing`: the pool is warmed for ONE retained
+`[S,vocab]` logits block (allocated inside ForwardLayers, kept as `s.logits`); the
+ring needs TWO simultaneously (one per slot), so the 2nd slot's capture hit a pool
+MISS. Working scratch is freed at ForwardLayers return and SAFELY shared (the two
+graphs replay SEQUENTIALLY on one stream — the overlap we want is host/GPU, not
+GPU/GPU). FIX: warm-capture pre-grows the pool with a throwaway `[S,vocab]` f32
+alloc+free while the other slot's logits is held, so the captured alloc is a HIT.
+
+hazard-C IS REAL BY CONSTRUCTION but its RED is UNREPRODUCIBLE on GB10. The captured
+graph re-reads its persistent host inputs at EVERY replay (that IS how Refresh
+updates each step; else every replay would emit identical tokens). But the baked H2D
+is a TINY, FAST copy at the replay's START, executed before the host completes a
+depth-2 iteration and reaches the next Refresh — the race window is microscopic.
+RED arm (drain skipped, ring off via `VT_ASYNC_EXECUTOR_NO_DBUF`) PASSED 3/3 @conc-4
+AND 3/3 @conc-32 (`VT_ASYNC_SERVING_CONC=32`); the low-conc POISON arm
+(`VT_ASYNC_EXECUTOR_POISON`, overwrite host inputs right after ReplayGraph) also
+PASSED (GPU executes the replay before the host overwrite). Per the brief's STOP
+rule the empirical RED is unmet -> default stays OFF, honest partial recorded.
+
+RING CORRECTNESS PROVEN (the token gates are the authority): GREEN
+`test_qwen36_async_serving` ring ON @conc-32 5/5 PASS; OFF baseline @conc-32 PASS;
+SACRED SYNC `test_qwen36_paged_engine` ring ON (deterministic token-exact) 3/3 PASS.
+CPU: -Werror clean; runner/llm_engine/engine_core_proc/async_llm/input_batch 6/6.
+DEFAULT DECISION: OFF (opt-in). Speed A/B c16/c32 OFF vs ON (`vllm bench serve`,
+decode-dominated) recorded in the benchmark record; the async decode is host-
+orchestration-bound (GPU ~14 W at 96% "util"). RESIDUALS: Option A (out-of-graph
+device-staged H2D, vLLM states.py:64) is the faithful follow-up; the eager
+`exec_state_` 2-deep ring was not needed (drain-on-eager retained). Evidence
+`dgx:~/work/mirror-ab/serve-async-executor/{gate2*.log,speed3*,diag*}`, commit
+`fa971248` on `row/SERVE-ASYNC-EXECUTOR`.
+
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -89,8 +89,11 @@ token_ids` device gather). The full drain is byte-exact and UAF-safe and is
 2026-08-06, speed-NEUTRAL** (same-binary): c16 OFF median 2305.8 vs ON 2303.3
 (0.999x), c32 2928.9 vs 2919.1 (0.997x), bands overlap. It is a drain MOVE (relocate
 the drain past the host prep, not remove it), so it overlaps only the small host
-prep; the drain still serializes GPU input staging, so c16 does not recover. Real
-c16 recovery needs the drain REMOVAL plus double-buffered `exec_state_`/block-table.
+prep; the drain still serializes GPU input staging, so c16 does not recover. The
+drain REMOVAL is `row/SERVE-ASYNC-EXECUTOR` (`VT_ASYNC_EXECUTOR`, landed default-OFF):
+a 2-slot decode-graph ring + reuse event, proven token-exact (async GREEN 5/5 c32,
+SACRED 3/3); hazard-C is real-by-construction but empirically unreproducible on GB10
+so it stays OFF (record has detail + the Option A follow-up).
 
 **But the mirror FIXES a shipping correctness bug, so it is now DEFAULT ON
 (ROW-SERVE-ASYNC-LLM, 2026-08-06).** Baseline async (AsyncLLM depth-2) batch-1 greedy