The benchmark harness ENFORCES the 0.25.0 rollback oracle and would REJECT the recorded pin — nobody could have measured correctly even deliberately

[localai-bot]

The pin advanced to 555967922 on 2026-07-26 (.agents/upstream-sync.md:7-9). The
benchmark harness never followed, and it does not merely default to the old
oracle — it raises on anything else:

tools/bench/online_gate.py:53-54
VLLM_ORACLE_VERSION = "0.25.0"
FLASHINFER_VERSION  = "0.6.13"
tools/bench/serve_low_common.py:28
VLLM_COMMIT = "702f4814fe54fabff350d43cb753ae3e47c0c276"

enforced at online_gate.py:3509-3533 (HarnessError: vLLM oracle version drift).

AGENTS.md: "Comparisons run against the pinned oracle recorded in
.agents/upstream-sync.md." Nobody could have measured against the pin even
deliberately — the harness refuses it. That is the part #375 does not cover:
#375 is a symlink pointing at the rollback, this is the gate demanding it.

Confirmed still live on dgx today:

~/venvs/vllm-oracle -> ~/venvs/vllm-oracle-v0.25.0-stage   (the ROLLBACK)
scripts/dgx-online-serving.sh:35  client="${HOME}/venvs/vllm-oracle/bin/vllm"

so the canonical driver resolves the rollback by construction.

Measured identity of both venvs (2026-08-12, dgx)

	vllm-oracle-v0.25.0-stage (what runs)	vllm-oracle-next (the PIN)
vllm.__version__	0.25.0	0.23.1rc1.dev1511+g555967922
dist metadata	0.25.0	0.23.1rc1.dev1511+g555967922.precompiled
flashinfer-python	0.6.13	0.6.15.post1
torch	2.11.0	2.13.0
transformers	5.13.1	5.14.1
pandas	2.2.3	MISSING
install	site-packages	editable -> ~/work/vllm-src-5559679 @5559679229bc9618

Two traps recorded here because they cost time today:

1. The pin is an editable install, so <venv>/lib/python3.12/site-packages/vllm
   does not exist. Any grep -r under that path returns nothing and reads as
   "the feature is absent from the pin". The real tree is ~/work/vllm-src-5559679/vllm.
2. The runtime version string is 0.23.1rc1.dev1511+g555967922, not the
   0.26.0.dev0 the pin record prose names, and metadata carries an extra
   .precompiled suffix that the runtime string lacks. The existing check
   asserts metadata == runtime == CONST, which cannot hold on the pin.

What is owed

Advance the three constants to the pin and keep the assertion mechanism — a
harness that accepts any oracle is worse than one pinned to the wrong version.
Additionally assert the resolved commit SHA, which is what actually
distinguishes the pin from the rollback (#375's symlink reported a clean
0.25.0 and looked healthy).

Related: #375 (symlink), #417 Finding 2 (first report of this), #414 (denominator config).

[localai-bot]

Constants survey: 1,399 grep hits, exactly 3 live definitions

grep -rn '0.25.0|0.6.13|702f4814|vllm-oracle|VLLM_ORACLE_VERSION|FLASHINFER_VERSION|VLLM_COMMIT' over tools/ scripts/ tests/ .agents/ docs/ returns 1,399 hits. They fall into three classes and only the first was reconciled.

Class A — live comparison constants (RECONCILED)

site	before	after
tools/bench/online_gate.py:53 VLLM_ORACLE_VERSION	0.25.0	0.23.1rc1.dev1511+g555967922
tools/bench/online_gate.py:54 FLASHINFER_VERSION	0.6.13	0.6.15.post1
tools/bench/serve_low_common.py:28 VLLM_COMMIT	702f4814fe54fabff350d43cb753ae3e47c0c276	5559679229bc961848b121ccdeaa8fa5d79bec98
(new) VLLM_DISTRIBUTION_VERSION	did not exist	0.23.1rc1.dev1511+g555967922.precompiled
online_gate.py:55 PANDAS_VERSION	2.2.3	unchanged, now fails closed at the pin (#522)

All four now come from a parity-pin block in .agents/upstream-sync.md. Every other consumer (online_gate_summary.py, gdn_packed_component.py, run_serve_low.py, four tests/tools modules) already imported these names, so there was no fourth copy to chase.

Class B — golden provenance (DELIBERATELY UNTOUCHED)

These state what produced an immutable artifact, which is a true historical fact. Advancing them would make the tree lie about its own goldens; they move on recapture, not on reconcile.

• tools/bench/gdn_ba_projection_oracle.py:24,75
• tools/bench/gdn_packed_decode_oracle.py:28,132
• tests/parity/goldens/gdn-packed-decode-oracle/manifest.json:50-51
• tests/parity/goldens/gdn_ba_projection_bf16_sm121/oracle.json:43-44
• tests/fixtures/nvfp4_flashinfer_v025_gb10/manifest.json:8-9

Owed follow-up: the two component-oracle scripts also carry a LIVE --require-vllm-version default of 0.25.0, so re-capturing either golden still demands the rollback. That is a recapture row, not this one, but it should not be forgotten.

Class C — narrative (UNTOUCHED)

~1,380 mentions in .agents/ records, specs, and capture-script docstrings. A past run that used 0.25.0 keeps saying so; history is git.

--language-model-only EXISTS at the pin

The report that it does not was two checking artifacts, both worth recording:

check	pin (vllm-oracle-next)	rollback
engine/arg_utils.py registration	:1276	:1241
occurrences in vllm serve --help=all	1	1
vllm serve --help=language-model-only	resolves, with help text	resolves, with help text

1. The pin is an editable install: <venv>/lib/python3.12/site-packages/vllm does not exist, so a grep -r under it returns nothing. The tree is ~/work/vllm-src-5559679/vllm.
2. vllm serve --help is a summary whose last line is For full list: vllm serve --help=all.

So no ratio needs re-labelling because of the flag. #414 is unaffected and still stands: the canonical driver never passes it. Note also that run_serve_low.py:549, the repo's only other mention, lives inside build_dry_run_manifest with <VLLM_ORACLE>/<MODEL_SNAPSHOT> placeholders — a recorded recipe, not executed argv, so nothing verifies a serve-low server was launched with the flag it documents.