Every recent comparison ran against the 0.25.0 ROLLBACK oracle, not the recorded PIN (555967922 / 0.26.0.dev0) — the symlink points at the rollback

[localai-bot]

Found while resolving #370. This is a measurement-integrity issue, not a code defect, and it affects the denominators of the current parity numbers.

The pin, as recorded

.agents/upstream-sync.md:7-8:

Active pin (advanced 2026-07-26): current parity pin 555967922 (vLLM 0.26.0.dev0 + transformers 5.14.1, Torch 2.13.0, FlashInfer 0.6.15.post1, CUTLASS ...)

:13 keeps vllm-oracle-v0.25.0-stage explicitly as a rollback.

What is actually being used

~/venvs/vllm-oracle -> ~/venvs/vllm-oracle-v0.25.0-stage
  vllm              0.25.0
  torch             2.11.0
  flashinfer-python 0.6.13        <-- the recorded mm_fp4 cute-dsl/sm_121 regression

~/venvs/vllm-oracle-next          <-- this is the PIN
  vllm              0.26.0.dev0+g5559679  (/home/mudler/work/vllm-src-5559679)
  torch             2.13.0
  flashinfer-python 0.6.15.post1

Every script in this campaign resolves $HOME/venvs/vllm-oracle, so every recent comparison used the rollback, including:

• the 27B vLLM decode profile (80.4492 ms/step) that 27B gap DECOMPOSED against vLLM's own decode profile: distributed +4.40 ms/step (tower 48%, Marlin 29%, other 23%); vLLM's fp8 tower is cuDNN and it fuses gate_up #365's whole gap decomposition rests on,
• the 0/4 token divergence in 27B is NOT token-exact vs a DETERMINISTIC pinned oracle, and it predates the gate_up row — 0/4 on current main's split path #370,
• and, pending confirmation of when the symlink moved, the canonical grids behind 27B 0.9561x and 35B 0.9719x.

AGENTS.md is explicit: "Comparisons run against the pinned oracle recorded in .agents/upstream-sync.md."

Why this was not obvious

The rollback runs, and it is deterministic (measured: 4/4 self-match on greedy 128-token continuations). A wrong-but-stable reference is indistinguishable from a correct one unless you check its identity. AGENTS.md's rule that an oracle is gateable only once it demonstrably builds and runs the model is necessary but, as this shows, not sufficient -- it must also be the pinned one.

The project record notes the symlink was pointed at the 0.25.0 stage as a post-reboot repair with a "durable 0.26 rebuild" left outstanding. That residual is what bit here: a temporary repair became the silent default.

Consequence for #370

#370's finding stands as measured -- our tree at the byte-identical commit 6603356a differs 0/4 from the CURRENT reference -- but the cause is now clear: the reference is not the one 6603356a was verified against. Our tree is exonerated twice over.

What is running now

Our tree at 6603356a and at current main, both compared against the PINNED stage, plus a determinism check on the pin itself. If 6603356a comes back exact against the pin, the entire 0/4 episode is an artifact of the mispointed symlink.

Owed regardless of that result

1. Repoint ~/venvs/vllm-oracle at the pin, or fix the scripts to resolve the pin explicitly rather than through a mutable symlink. A benchmark harness that silently accepts whatever the symlink points at will do this again.
2. Add an identity assertion to every oracle-touching script: print and check vllm.__version__ and the flashinfer version against the recorded pin, and ABORT on mismatch. This is the same class as the FA2-marker guard that already protects against degraded local builds -- the reference side simply never got one.
3. Re-measure the binding grids against the pin before any parity number is quoted as established.

[localai-bot]

Worse than a mispointed symlink: the PINNED oracle does not exist on disk

Ran the pin. It cannot start:

=== PINNED stage identity ===
vllm               0.26.0.dev0+g5559679 /home/mudler/work/vllm-src-5559679
torch              2.13.0
flashinfer-python  0.6.15.post1

Traceback (most recent call last):
  File "/home/mudler/venvs/vllm-oracle-next/bin/vllm", line 3, in <module>
    from vllm.entrypoints.cli.main import main
ModuleNotFoundError: No module named 'vllm'

vllm-oracle-next holds an editable install pointing at /home/mudler/work/vllm-src-5559679, and that directory is gone:

src dir exists: NO
(not a git repo)

So the recorded parity pin 555967922 has no importable package and no source tree. There is currently no runnable pinned oracle on this box.

Corrected state

stage	runs?	is the pin?
vllm-oracle -> v0.25.0-stage (vllm 0.25.0, FlashInfer 0.6.13)	yes	no -- recorded as a rollback
vllm-oracle-next (vllm 0.26.0.dev0+g5559679)	no -- source deleted	yes

This is exactly the "durable 0.26 rebuild residual" the project record flagged as outstanding after the post-reboot repair. The residual was not cosmetic: the pin was never durably installed, so the rollback silently became the de facto reference.

Consequences

1. No gate can currently be run against the pin as recorded. AGENTS.md: an oracle is gateable only once it demonstrably BUILDS AND RUNS the model. This one does neither.
2. Every current comparison -- 27B 0.9561x, 35B 0.9719x, the 80.4492 ms/step vLLM decode profile underpinning 27B gap DECOMPOSED against vLLM's own decode profile: distributed +4.40 ms/step (tower 48%, Marlin 29%, other 23%); vLLM's fp8 tower is cuDNN and it fuses gate_up #365's decomposition, and 27B is NOT token-exact vs a DETERMINISTIC pinned oracle, and it predates the gate_up row — 0/4 on current main's split path #370's token result -- is against the rollback. Those numbers are provisional, not established. They are not shown to be wrong; they are shown to be against an unrecorded reference.
3. 27B is NOT token-exact vs a DETERMINISTIC pinned oracle, and it predates the gate_up row — 0/4 on current main's split path #370 is fully explained and our tree is exonerated: the reference 6603356a was verified against no longer exists in runnable form.

Fix, in order

1. Rebuild vLLM at 555967922 from source into vllm-oracle-next (needs network + a long build), or, if that revision is no longer practical, advance the pin deliberately per the upstream-sync procedure and reconcile every affected row and gate -- not silently.
2. Then re-verify: the pin must run the 27B and the 35B, and reproduce a recorded golden, before anything is gated against it.
3. Add an identity assertion to every oracle-touching script -- print and check vllm.__version__ plus the flashinfer version against upstream-sync.md and ABORT on mismatch. The local build side has had a FA2-marker guard for exactly this failure mode; the reference side never got one, which is why a deleted pin went unnoticed.

Note on cause

I have not established when or why the source tree was removed, and I am not going to guess. Worth recording that this box hit 100% disk earlier today -- partly my own 150G test build, since removed -- and this project has a standing practice of pruning source-* trees under disk pressure. A prune that caught vllm-src-* would produce exactly this state. Whoever rebuilds should consider putting the oracle source somewhere the pruning conventions do not reach.

[localai-bot]

Today's 27B gap decomposition is downstream of this, and the tower term is the most exposed

Cross-linking because #362 / #365 spent a day attributing the 27B decode gap, and every denominator in that work came from this venv.

What was measured against the 0.25.0 rollback:

• The binding six-point 27B canonical grid (27B c1 is not reproducible across runs: our arm measures 9.37 and 10.76 tok/s at the SAME SHA while vLLM holds within 1% (published 0.8384x does not reproduce; canonical re-run says 0.9561x) #349 / PR bench(27n): the published 27B grid does not reproduce; record the canonical six-point regrid #360): flat 0.9371x-0.9561x.
• vLLM's per-kernel decode profile (27B gap DECOMPOSED against vLLM's own decode profile: distributed +4.40 ms/step (tower 48%, Marlin 29%, other 23%); vLLM's fp8 tower is cuDNN and it fuses gate_up #365): 80.4492 ms/step GPU-busy.
• The full decomposition of the +4.40 ms/step gap: Marlin 29%, fp8 tower 48%, GDN decode kernel 9%, RMSNorm fusion 7%, residue 7%.

What is NOT invalidated. Every A/B and lever measurement is same-binary, one variable, ours-vs-ours: the Marlin gate_up win (+2.1%/+1.7%, 193 -> 129.000 calls), merged-QKV (neutral, 96 -> 64 calls, time unchanged), Phase A+B (+0.5-0.7%). Those do not involve the oracle at all. Our own per-kernel table (#362) is likewise ours-only.

What IS exposed: every ratio and every cross-arm term, i.e. the entire decomposition.

The tower term specifically

The 48% term rests on vLLM running cudnn_generated_fort_native_sm120_matMul_pointwise_pointwise where we run cuBLASLt nvjet_*. Traced to source, that selection is:

vLLM -> FlashInferFP8ScaledMMLinearKernel -> flashinfer bmm_fp8(backend="auto")
     -> fp8_gemm_sm100 -> AutoTuner.choose_one("fp8_gemm", ...) -> CudnnFp8GemmRunner

It is an empirical FlashInfer autotune outcome, not a fixed design choice. Without tuning data the fallback runner is cutlass_sm12x, not cuDNN.

The rollback carries FlashInfer 0.6.13; the pin records 0.6.15.post1. So the one component that chooses the kernel differs by two patch releases between what we measured and what the pin specifies. The cuDNN attribution -- and therefore the recommendation about whether to take on a cuDNN dependency -- could move under the correct pin.

Also relevant to whoever fixes this: .agents/parity-ledger.md:379 records 0.6.13 on the box, and a separate recorded finding notes a flashinfer-0.6.13 mm_fp4 cute-dsl/sm_121 regression. The rollback is not a neutral substitute for the pin on fp4/fp8 paths.

Recommendation

Re-establishing the pinned oracle should outrank further lever work on the 27B. A lever chosen against the wrong reference optimizes toward the wrong target, and the largest open term (the tower, 48%) is precisely the one whose kernel selection is decided by the component that differs.

Concretely owed once the pin is restored: re-run the 27B canonical grid and re-take vLLM's decode profile, then re-derive the decomposition. Our side needs no re-measurement.

[localai-bot]

The PINNED oracle is present and GATEABLE. vllm-oracle-next runs the 27B.

Checked what is actually installed on dgx:

~/venvs/vllm-oracle              -> vllm-oracle-v0.25.0-stage   (the ROLLBACK)
~/venvs/vllm-oracle-v0.25.0-stage:  vllm 0.25.0                 flashinfer 0.6.13
~/venvs/vllm-oracle-next:           vllm 0.23.1rc1.dev1511+g555967922  flashinfer 0.6.15.post1

vllm-oracle-next carries SHA 555967922 and flashinfer 0.6.15.post1 -- both exactly what .agents/upstream-sync.md:7-8 records as the active pin.

And it runs, which is the bar this project sets (a gateable oracle demonstrably builds and RUNS the model; constructing a config proves nothing):

vllm 0.23.1rc1.dev1511+g555967922
flashinfer 0.6.15.post1
RUNS: 64 tokens in 5.179s = 12.36 tok/s
TEXT:  Paris. The capital of France is Paris. The capital of France is Paris

Qwen3.6-27B-NVFP4, greedy, coherent, no errors, VLLM_ENABLE_V1_MULTIPROCESSING=0, gpu_memory_utilization=0.55.

Why this matters beyond bookkeeping

The 27B's largest open term is the fp8 tower (48% of a measured +4.40 ms/step, #365), and that kernel is chosen by FlashInfer's autotuner -- bmm_fp8(backend="auto") -> AutoTuner.choose_one("fp8_gemm", ...) -> CudnnFp8GemmRunner. FlashInfer is precisely what differs between the rollback (0.6.13) and the pin (0.6.15.post1). A different selection on the correct pin could move that 48% figure before anyone commits to a cuDNN dependency, which is currently the single largest open decision on the 27B.

So re-measuring against the pin is not a formality; it may change the target.

One discrepancy worth recording, not glossing

The version string reads 0.23.1rc1.dev1511+g555967922, whereas upstream-sync.md describes the pin as "vLLM 0.26.0.dev0". The SHA and the FlashInfer version match exactly, so this is almost certainly a git describe artifact of the tag base in that checkout rather than a different tree -- but it should be confirmed (vllm.__file__ -> its .git, or pip show) before the pin is declared restored, not assumed.

What is now unblocked

Re-running the canonical grids against vllm-oracle-next instead of the symlink. Every ratio currently in docs/BENCHMARKS.md -- including the six-point 27B grid this session added (0.9371x-0.9561x) and the 35B's 0.9719x -- is measured against the rollback. Our own arm needs no re-measurement; only the denominator changes.

Concretely owed: point the driver's --client at ~/venvs/vllm-oracle-next/bin/vllm (or repoint the symlink once the discrepancy above is confirmed) and re-run 27n and 35.

[localai-bot]

The blocker is the HARNESS, not the oracle: dgx-online-serving.sh enforces expected='0.25.0'

I tried to re-run the canonical 27n grid against the pinned oracle. The driver refused, and its refusal is the answer:

online-gate: vLLM oracle version drift:
  metadata='0.23.1rc1.dev1511+g555967922.precompiled'
  runtime ='0.23.1rc1.dev1511+g555967922'
  expected='0.25.0'

Dry-run rc=0, corpus rc=0, execute rc=2 on the version check. The pinned oracle loads and generates (verified separately: 64 tokens, coherent, no errors), so the oracle is not the problem -- the harness is pinned to expect 0.25.0.

That reframes this issue. It is not "a symlink drifted". The rollback is consistently configured: the vllm-oracle symlink points at 0.25.0 AND the online-gate driver expects 0.25.0. They agree with each other and both disagree with .agents/upstream-sync.md, which records the active pin as 555967922 / FlashInfer 0.6.15.post1.

Why the refusal is correct

The check is doing exactly what an oracle-identity gate should: it compares corpus metadata, runtime version and an expected value, and refuses when they diverge. Without it I would have silently produced a grid labelled with one oracle and measured against another -- which is the failure this issue exists to prevent. Good gate; I would not weaken it.

What restoring the pin actually requires

Not a symlink flip. At minimum:

1. Update the driver's expected-oracle version (and whatever records that constant) from 0.25.0 to the pin.
2. Regenerate the corpora -- the existing manifests carry vllm_commit: 702f4814fe54fabff350d43cb753ae3e47c0c276, which is neither the pin nor 0.25.0's recorded commit and needs reconciling.
3. Re-capture the SACRED goldens and demonstrate ZERO drift before any ratio is accepted, per the pin-advance procedure.
4. Re-run 27n and 35.

That is a governed change to a gate's identity check, so it wants a spec and a review rather than an in-flight edit -- which is why I stopped here rather than editing the expected version to make my run pass. Turning a red gate green by relaxing its assertion is precisely what AGENTS.md forbids.

Practical consequence for the current numbers

Everything in docs/BENCHMARKS.md, including the six-point 27B grid this session added (0.9371x-0.9561x) and the 35B's 0.9719x, is measured against 0.25.0 + FlashInfer 0.6.13, consistently and with a harness that enforces that pairing. The numbers are internally sound; they are simply not against the recorded pin. Given that the 27B's largest open term (the fp8 tower, 48%) is selected by FlashInfer's autotuner, re-measuring after the pin is restored may reprice it -- which is an argument for doing the restoration properly before the cuDNN dependency question is decided.