chore: retest and rebench on RTX 5070 - #9
Merged
mudler merged 10 commits intoJul 30, 2026
Conversation
…es as contended Discharges the revalidation docs/BENCHMARKS.md recorded as PENDING for the c317237 transplant. That transplant's two commits are now upstream (a131de0, b6f1efc), so the measured tree is plain current main @ 7f620e7. All 18 legs plus the correctness tier ran under one flock /tmp/gpu on the local RTX 5070 Ti. Correctness is unchanged bit for bit: the six suites report exactly the counts the 2026-07-25 record does, and the benchmark output is token-IDENTICAL to that series, 128/128 requests per repetition in both the direct-ON and direct-OFF arms. 109 upstream commits moved no token on this workload. Speed, ours ON vs the vLLM 0.24.0 oracle measured in the SAME series: total throughput 0.9819x (was 0.9864x), TTFT 0.7982x PASS, TPOT/ITL 1.1400x FAIL (was 1.1341x). Repetition spread is 0.13% ours and 0.17% vLLM, so the ratios are not noise. Memory passes: direct loading cuts peak PSS 73.4%, stable PSS 91.1%, and now also mean TTFT 12.7% versus direct-OFF. The 2026-07-25 ABSOLUTE numbers are VOID. Every one of that series' nine performance legs ran with the GPU at 11-13% utilization and 611 MiB of extra resident VRAM; today's nine ran at 0%. Both arms gained ~14% on a genuinely idle box, which is why the ratio barely moved. The cause was a hole in the harness: prepare_leg gated idleness on nvidia-smi --query-compute-apps, which enumerates CUDA contexts only, so a graphics consumer kept the GPU busy invisibly for a whole binding series. Now also gated on utilization.gpu (GPU_IDLE_UTIL_MAX, default 2%); re-parsed it reads 0 for today and 12 for 2026-07-25. That series' ratios, its same-binary component attributions and its profiling attribution survive, each having been internal to one uniformly contended series. Two summarizer defects fixed on the way, either of which made summarizing a series against the previous one impossible: it read historical token legs under a `perf-` prefix the harness has never written, and it required a `vllm_production` key its own current output does not emit. Adds tools/bench/run_qwen35_4b_ab.sh, the same-binary A/B shape the comparison harness cannot express: one lock across the series, arms interleaved with the order flipped on even repetitions so neither arm is systematically the warmer one, per-leg token ids captured so an arm that changed the output is caught rather than celebrated, and the same idle gate. Also corrects the oracle label. The local venv is vLLM 0.24.0, not the 0.25.0 the prior evidence claimed; that series' own recorded vllm-version.txt already said 0.24.0. It is behind the project parity pin (555967922 / 0.26.0.dev0) and is labelled as such rather than treated as a pin-era denominator. The residual is unchanged and now has a spec. TPOT is the only failing latency axis, and the mechanism is specific: this GPU is discrete, so is_integrated_gpu() is false, both ENG-ASYNC-SCHED W3 device call sites take their host fallback, and sample_tokens_async must synchronize the main stream before the host can read the sampled ids. The async scheduler is engaged (max_concurrent_batches=2) and therefore overlaps nothing here. Upstream has no such branch: states.py:64 keeps last_sampled_tokens GPU-resident unconditionally and states.py:132 never condenses slots. Scoped as ENG-ASYNC-SCHED W4 in .agents/specs/async-discrete-device-combine.md, including the second per-step barrier that must land with it. No 4B result implies support or speed for the 27B/35B gates, which remain hardware-unavailable on this host. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…ens (opt-in), and the attribution that motivated it is wrong Gives the discrete-CUDA async path the device-resident last_sampled_tokens that upstream keeps on every platform (states.py:64). The W3 device combine/scatter were gated on is_integrated_gpu() because they operate on the runner's host arrays in place, which is only legal under UMA; a discrete GPU took a host fallback that must synchronize the main stream for the sampled ids. W4 adds runner-owned device buffers for last_sampled / prefill_len / query_start_loc / seq_lens / input_ids, points the existing kernels at them, and threads the patched device ids to the forward through ModelForwardInput::device_token_ids. Our InputBatch condenses, which upstream never does (states.py:132 frees slot indices into a pool), so the host records its structural row edits and a new LaunchApplyLastSampledOps replays them on the device in stream order. W4e removes the other per-step barrier: the CUDA embedding's out-of-range flag no longer does cudaMalloc + cudaStreamSynchronize + cudaFree on every call but uses a persistent ring of slots with a deferred, event-queried check. That reporting contract changed deliberately and its test changed with it: a bad id is now raised no later than the next embedding on the queue, same message and same id. The gather always clamped, so the offending call never read out of bounds either way. Three design corrections the spike did not anticipate, all recorded in the spec. The embed patches a PREFIX rather than embedding the runner's buffer, because the decode graph embeds a PADDED id vector whose real rows come first. The per-step uploads copy from PAGEABLE host memory on purpose: a shared pinned staging buffer is a race, since pinned copies are truly asynchronous and the next upload can overwrite bytes an in-flight DMA has not read, a window that spans steps under a depth-2 scheduler. And the forward reads the ids through an RAII scoped override, consumed on first use, rather than a defaulted parameter on five entry points plus the decode-graph class. THE FINDING. The first gate run failed at 0/128 token identity with garbage from the first decode, because the device mirror stayed all zero: the scatter branch in sample_tokens_async never executed. vllm-bench drives the SYNCHRONOUS LLMEngine::step() loop, which calls sample_tokens(), not AsyncLLM's depth-2 step_with_batch_queue. So on the benchmarked path there is no async sampler and no sample_tokens_async synchronize to remove, and the 2026-07-25 attribution of 497 cudaStreamSynchronize calls (20.975 s, 42.20 ms/call) to that function is wrong for this workload. Those synchronizations are something else and must be re-attributed before another lever is chosen from that trace. Fixed by feeding the mirror from whichever sampler ran. Gates, one flock /tmp/gpu, same binary: token identity 128/128 in both directions; test_qwen35_plain_weights --no-skip 3/3 (1672); test_input_batch 25/25 (183, +3 new W4 cases including a replay-vs-host composition case that catches a misordered op); test_combine_tokens 7/7; test_ops_gdn 66/66 (4242); test_ops_paged_attn 25/25 (454,474); clean -Werror CUDA rebuild, 0 warnings. Speed is NEUTRAL here and cannot be otherwise: with no overlap to unlock, W4 adds four small uploads and two kernels per step and removes nothing. Paired runs gave 6612.31 vs 6600.68 and 6602.22 vs 6612.15 tok/s, equal and opposite. So it lands OPT-IN and DEFAULT OFF (VT_ASYNC_DEVICE_MIRROR=1), the same disposition the W3 device kernels had before their A/B; production is byte-identical. The binding measurement is a serving A/B over AsyncLLM and is PENDING. Also adopts the one real gap an external C++/Rust porting study identified: VLLM_CPP_SANITIZE host lanes (ASan+UBSan, TSan as separate builds, CUDA refused, -fno-sanitize-recover=all) with a sanitize-cpu CI matrix, and VT_POOL_BYPASS=1 so compute-sanitizer can see the tensor boundaries and use-after-free that the caching, size-class-rounding device pool hides. Both default off. The study's other rules were assessed and are already satisfied or exceeded here; the item-by-item decision is recorded rather than left implicit. Records a pre-existing failure found on the way and verified not introduced: test_cuda_ops "CUDA matmul (cuBLASLt) matches CPU on odd sizes" fails 11 elements on the bf16/bf16 17x31x13 case on this discrete sm_120, reproduced on a pristine build of the same commit. The project develops on GB10/sm_121a, so this is an unattributed per-architecture numerics difference on consumer Blackwell. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…PU break they found
The lanes landed in the previous commit configured but did not build, and
verifying them turned up a real break in that same commit.
1. -Werror=unused-function on a CPU build. AsyncDeviceMirrorEnvDefault() is
referenced only inside #ifdef VLLM_CPP_CUDA, so it is dead code when the CUDA
backend is off. That breaks the EXISTING build-test-cpu CI job, not just the
new lanes; the sanitizer lane simply happened to compile the CPU
configuration first. Guarded the definition with the same #ifdef as its use.
This is exactly the failure mode the guide's "compile every cfg branch" rule
describes.
2. -Werror vs sanitizer instrumentation. Instrumentation changes inlining enough
that GCC's range and initialization analyses fire inside libstdc++ on correct
code: a one-element std::vector<int32_t> v = {x} in voxtral.cpp draws
"forming offset 4 is out of the bounds [0, 4]" for a 4-byte read of a 4-byte
array, and <regex> draws 26 -Wmaybe-uninitialized reports out of
std::function internals. Neither is project code. The lane now keeps the
warnings and drops -Werror, done in cmake/CompilerWarnings.cmake where the
per-target flags are set - a global -Wno-error is overridden by those PRIVATE
target options and silently does nothing, which is why the first attempt at
this did not work. The plain build still enforces -Werror and is clean.
Verified end to end rather than by configuring: both lanes configure; both
guards fire (a lane value outside the allowlist, and a lane with CUDA on, are
FATAL_ERROR); the ASan+UBSan lane builds test_input_batch, test_combine_tokens
and test_arena clean and all three pass under detect_leaks=1 with UBSan stack
traces, which covers the new W4 structural-op cases; and the plain CPU build is
green again (test_input_batch 25/25, 183 assertions).
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The new idle gate refused to start the second arm, reporting 14% utilization. It was right about the number and wrong about the moment: run_leg sampled the GPU before its cooldown, so it read the previous leg still draining. Sleep first, then snapshot and check. Found by running the gate rather than by reading it. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…r the async sampler's Binding same-binary A/B for the W4 mirror, three interleaved repetitions per arm with the order flipped on even reps, one lock, idle box: 0.9996x total and output throughput, 1.0004x TPOT, 1.0002x TTFT. Each arm's own repetition spread is an order of magnitude wider than the gap between them, so this is NEUTRAL, which is what the opt-in default-OFF disposition already assumed. Token identity across all three pairs is 384/384 requests. More importantly, a fresh attribution-complete profile of the benchmarked path replaces the premise the row was scoped from. The 2026-07-25 record blamed 497 cudaStreamSynchronize calls (20.975 s, 42.20 ms/call) on sample_tokens_async's discrete host bookkeeping. The new trace shows 112 cudaStreamSynchronize over ~64 decode steps plus prefill and warm-up at 10.12 ms each, which is about one per engine step; scaled to the binding workload (~512 steps at ~38 ms TPOT) that is ~500 calls at ~40 ms, the old numbers almost exactly. The count and the time were right. The attribution was not: it is the depth-1 LLMEngine::step() loop waiting for its own sampling, and sample_tokens_async is never called on that path at all. The per-call embedding barrier is absent from the new trace, which is W4e working; whole-run cudaMalloc is down to 818 calls / 29 ms and cudaFree to 512 / 28 ms. The consequence is a different lever list. There is no per-step synchronize left to delete on the synchronous path, because the wait IS the step. Overlap requires running the async engine loop, which is exactly what W4 now makes legal on a discrete GPU, so the next measurement is a serving A/B over AsyncLLM. That is blocked by the harness, not the engine: run_serve_low.py needs a pinned SGLang container image and accepts only the 27B/35B model keys, and neither is available on this host. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…he device is CUDA The device-leakage ratchet failed on the W4 enable predicate, which tested queue_.device.type == kCUDA && !is_integrated_gpu() and so put a device-type token in the device-agnostic shared layer. The ratchet is right, and the better question already existed: vt::Backend::UnifiedMemory() answers "is device memory addressable from the host", which is precisely what decides whether the mirror is needed. A unified device (GB10, and the CPU backend trivially) keeps the in-place path; a device with separate memory needs the mirror. Same behaviour on every platform, one fewer device-type test, and the DSR bucket returns to its baseline of 0. Re-gated after the change: token identity mirror ON vs OFF 128/128, test_qwen35_plain_weights 3/3 (1672), test_input_batch 25/25 (183), test_combine_tokens 7/7, test_ops_gdn 66/66 (4242), test_ops_paged_attn 25/25 (454,474), clean -Werror rebuild, and the leakage mutation suite green. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
From the same profile that re-attributed the synchronizations: the two largest GPU kernels are SM80 / Ampere-class cutlass tensorop GEMMs, together 59.9% of kernel time, selected by cuBLASLt's heuristic on a Blackwell sm_120 device. Deliberately filed as a LEAD and not a lever. The instance counts and the _relu_ epilogue point at the prefill projections, and prefill is the axis we already pass; the failing axis is TPOT. And no matched vLLM trace exists on this device, so nobody knows yet whether the oracle resolves the same kernels - in which case there is no gap at all. The next step is a matched pair of traces on the identical corpus, diffed by kernel name, before anyone tries to force a tactic. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The harness derived the vLLM arm's CUDA_HOME from the Nix CMake cache. That is right for a venv borrowing Nix's CUDA and wrong for one that carries its own: it would put a 12.9 toolkit ahead of the 13.x the venv's extensions were compiled against, and FlashInfer's JIT would then compile against mismatched headers. VLLM_CUDA_HOME selects the venv's toolkit and skips the symlink farm entirely; unset, behaviour is exactly as before. Needed because the local oracle was two versions behind the parity pin and could not be brought to it by pip: the pin is a vLLM main commit with no release tag and no prebuilt wheel, so it has to be built from source, and such a build brings its own CUDA wheels. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Assisted-by: Codex:gpt-5 [Codex]
… pin
The local oracle was vLLM 0.24.0 against a pin of 555967922, and it could never
have matched by installation: the pin is a main commit with no release tag and no
prebuilt wheel on any platform, so pip only ever reaches an older release. Built
it from source instead - vllm 0.23.1rc1.dev1511+g555967922.cu132, sm_120 only,
with the pinned stack - into a separate .venv-vllm-pin. The 0.24.0 venv is
untouched and kept as a rollback denominator, and ${VLLM_SOURCE} is now a real
checkout at the pin rather than "unavailable".
The result runs the other way from what I expected. vLLM at the pin is 0.9875x
the 0.24.0 release on this workload, so the old denominator was the FASTER vLLM
and the published number was understating us. Against the true pin: total and
output throughput 0.9970x, req/s 0.9969x, TTFT 0.7731x PASS, TPOT 1.1241x FAIL.
The throughput gap is 0.3%, not 1.8%, with no change to our code, and TPOT
(+12.4%) is confirmed as the one real gap worth working.
The control that makes that attributable to the oracle rather than to us: our own
arm reproduces the previous series exactly, 1.0027x and token-identical 128/128
per repetition. Spread is 0.11% (pin) and 0.08% (ours).
Records the CUDA toolkit ceiling, because it cost two full builds and is not
obvious. The usable version is set by what the DRIVER can JIT, not by what is
newest: vLLM ships FlashAttention-2 as 8.0+PTX and the driver JIT-compiles it at
load, so a 13.2-capable driver rejects nvcc-13.3 PTX - and only at runtime, after
a completely clean build. CUDA 13.0 is separately unusable because its headers
predate glibc 2.42's rsqrt. 13.2 is the only version clearing both, the whole
toolkit must match or cccl refuses, and vLLM must be installed --no-deps or pip
re-resolves the runtime downward afterwards and puts the mismatch back.
Also corrects a claim I made earlier in this session: the repo's triton_kernels/
directory does NOT shadow Triton's package in benchmark runs. The harness invokes
the metrics script by path, so sys.path[0] is tools/bench and the repo root is
never on sys.path. The shadowing only happened in an ad-hoc stdin-piped test.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…nothing moved Rebased this branch from main 7f620e7 onto f3ecbe7 and re-ran the 18-leg matched comparison against the same source-built oracle at the parity pin. The pin was re-checked before the run and is unchanged (555967922), so the denominator is still correct. The result is a null: 0.9972x total throughput (was 0.9970x), TTFT 0.7701x PASS, TPOT 1.1247x FAIL. Ours 6610.270 vs pin 6628.651 tok/s, per-rep spread 0.05% and 0.07%. The control is what makes that a measurement rather than an assumption. All three arms drifted down by the same ~0.13% (ours 0.9988x, direct-OFF 0.9983x, pin 0.9986x). The pin arm is the SAME binary on the SAME corpus and cannot have changed, yet it drifted the same amount - so the drift is ambient thermal/clock state, not code, and it is the size of the arms' own repetition spread. Every ratio sits well inside that floor. No movement was the expected outcome. The one upstream commit in the window that names this gap, 2b00866, concluded the decode TPOT/ITL gap is batch composition rather than a decode-kernel deficiency, and changed records rather than code; the rest of the window is breadth (DeepSeek-V4-Flash, Gemma-4, Kimi K3, TP, LoRA, AWQ/GPTQ/MXFP4, fp8 KV, xgrammar) on paths this dense 4B decode workload does not touch. TPOT remains owned by ENG-ASYNC-SCHED. It also validates the rebase itself. Three conflicts were resolved: two append-only record files (kept both sides, upstream first, verified additive with zero lines lost from upstream) and one real collision where ModelForwardInput gained a field on both sides - upstream's mm and our device_token_ids. Both are kept with ours LAST, since it has to be the final member; that is the same constraint that broke positional aggregate initializers when it was first written mid-struct. Beyond the 1672/1672 model gate, our generated output is token-identical 128/128 in every repetition to the pre-rebase series on both the direct-ON and direct-OFF arms, which is the semantic check those resolutions needed. Records-and-evidence only; no engine change. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Collaborator
|
Reviewed and merged into |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some fixes for dGPU and benchmark result updates