policy(oracle): admit a pinned secondary oracle where vLLM implements nothing (#647)

.agents/oracles/README.md

@@ -0,0 +1,41 @@
+# The oracle registry — one file per oracle
+
+Every upstream this project compares itself against has exactly one file here,
+named `<id>.md`, and every one of them carries a pin. The rule this surface
+serves is [`AGENTS.md`](../../AGENTS.md) §"When vLLM has no implementation":
+vLLM is the primary reference, a secondary oracle is admissible only where vLLM
+implements nothing, and an upstream with no recorded revision is not an oracle at
+all.
+
+**Why one file per oracle rather than a table.** A shared table is a lock: two
+rows advancing two different pins in the same week conflict on lines neither
+owns, and an automatic three-way merge of a keyed record is exactly what the
+protocol refuses. Files are read by glob (`.agents/oracles/*.md`), so adding an
+oracle touches one new path and nothing else.
+
+**What lives here and what does not.** These files carry *identity*: which
+upstream, which revision, measured when, gateable or not, and the evidence.
+They do not carry methodology or results. The vLLM parity pin keeps its home in
+[`../upstream-sync.md`](../upstream-sync.md) — `vllm.md` points at it rather
+than restating it, because a pin transcribed twice is a pin that drifts.
+
+## The record
+
+Each file carries exactly one fenced ` ```oracle-pin ` block with these keys, in
+any order, one `key = value` per line:
+
+| Key | Meaning |
+|---|---|
+| `id` | matches the filename stem, and the `id` column of the AGENTS.md table |
+| `role` | `primary` (vLLM, and only vLLM) or `secondary` |
+| `upstream` | the canonical repository URL |
+| `scope` | what this oracle may answer for — one line |
+| `pin` | a 40-hex commit, a version string, or the literal `UNPINNED` |
+| `pin_label` | the human name of that revision (`v0.5.15`, `b9892`, `none`) |
+| `pinned_on` | ISO date the pin was recorded or measured |
+| `gateable` | `yes` only once it demonstrably builds and runs the model |
+| `evidence` | a repo path that exists when `gateable = yes`; the issue that owes the measurement (`#N`) when `gateable = no` |
+
+`scripts/check-oracle-pins.py` enforces all of it, including that the AGENTS.md
+table and this directory name the same set of ids. Run it with `--self-test` to
+sweep its own fixture corpus in both directions.

.agents/oracles/diffusers.md

@@ -0,0 +1,28 @@
+# `diffusers` — schedulers, VAEs and diffusion pipelines
+
+The diffusion lane's counterpart to `transformers`: where a model ships a
+`diffusers` pipeline and vLLM-Omni does not implement it, `diffusers` is the
+reference for the flow-matching or denoising schedule, the VAE decode, and the
+pipeline's step-by-step composition. Several checkpoints ship a `diffusers`
+integration before, or instead of, any serving-engine one — `MiniMax-Music3`
+recommends `diffusers` alongside SGLang-Omni, and its flow-matching and Flow-VAE
+stages are the parts a scheduler oracle answers exactly.
+
+Prefer vLLM-Omni wherever it implements the pipeline; reach here when it does
+not, and record which of the two a given stage was gated against, because they
+do not always agree on scheduler details.
+
+**Not gateable yet:** nothing in this repository has executed `diffusers` as an
+oracle, so there is no measured revision to pin. #647 owes both.
+
+```oracle-pin
+id = diffusers
+role = secondary
+upstream = https://github.com/huggingface/diffusers
+scope = schedulers, VAEs and diffusion pipelines vLLM-Omni does not implement
+pin = UNPINNED
+pin_label = none
+pinned_on = 2026-08-13
+gateable = no
+evidence = #647
+```

.agents/oracles/llama-cpp.md

@@ -0,0 +1,26 @@
+# llama.cpp — the CPU and GGUF k-quant floor
+
+Never the mirror source: llama.cpp's structure is not vLLM's, and a behavior
+difference between them is settled by vLLM. What it supplies is a **floor** —
+the CPU and GGUF k-quant speed and memory numbers a user can actually get today,
+which is the honest denominator on every path where vLLM's own CPU support is
+not the thing being compared.
+
+The pin is a **local fork**, not upstream `master`, because the CPU floor
+campaign builds it with a fixed recipe and reads its kernels; the fork is what
+`237ad9b96` names. Measured evidence and the build recipe are in
+[`../specs/cpu-llamacpp-floor-remeasure-2026-07-22.md`](../specs/cpu-llamacpp-floor-remeasure-2026-07-22.md),
+which carries the raw per-run numbers, and the same pin backs the A76 dot-product,
+elementwise-GEMM, GDN-orientation and threadpool specs.
+
+```oracle-pin
+id = llama-cpp
+role = secondary
+upstream = https://github.com/ggml-org/llama.cpp
+scope = CPU and GGUF k-quant speed and memory floors, quant-matched against the same weights
+pin = 237ad9b96
+pin_label = b9892
+pinned_on = 2026-07-22
+gateable = yes
+evidence = .agents/specs/cpu-llamacpp-floor-remeasure-2026-07-22.md
+```

.agents/oracles/sglang.md

@@ -0,0 +1,47 @@
+# SGLang, including SGLang-Omni
+
+Two distinct roles, and one forbidden one — the methodology is
+[`../specs/sglang-parity-oracle.md`](../specs/sglang-parity-oracle.md) and the
+enumerated surface is [`../sglang-matrix.md`](../sglang-matrix.md):
+
+- **Correctness cross-check.** Where SGLang and vLLM agree on greedy tokens for
+  the same model, ours matches both. Where they diverge, **vLLM wins** — SGLang
+  never re-specifies a vLLM-derived behavior.
+- **Performance floor.** Wherever SGLang beats vLLM on an equivalent workload,
+  SGLang is the binding floor.
+- **Model paths vLLM does not implement at all.** This is the case the fallback
+  rule was written for, and today it is SGLang-Omni: `MiniMax-Music3`
+  (`minimax_music3`) is served by SGLang-Omni and registered nowhere in vLLM.
+  Here SGLang is not a cross-check, it is the only available reference.
+- **Forbidden:** porting SGLang's data structures as a second, incompatible
+  abstraction. A SGLang-distinct behavior is an opt-in over our vLLM-derived
+  design, never a fork of the engine.
+
+**Two SGLang revisions already appear in specs, for two different purposes.**
+The oracle pin below is the *source* pin — the tree every `file:line` in
+`sglang-matrix.md` was read from. The benchmark harness in
+[`../specs/cuda-sglang-low-concurrency.md`](../specs/cuda-sglang-low-concurrency.md)
+separately pins tag `v0.5.13` / `28b095c01005d4a3a2a5b637b7d028b07fba31b2` and
+its runtime image, and that pin is never silently substituted into the other's
+evidence.
+
+**SGLang-Omni is a third repository and has no pin of its own yet.** It is
+covered by this record's scope, not by this record's revision; a Music3-class
+lane that needs it owes the pin before it can claim a gate.
+
+**Not gateable yet:** no SGLang run has been recorded on this project's hardware.
+The parity-oracle spec is explicitly a read-only scoping spike ("no engine code,
+no measurement taken here"), and the low-concurrency spec fixes the artifacts
+"before anyone installs SGLang". Source has been read; nothing has been executed.
+
+```oracle-pin
+id = sglang
+role = secondary
+upstream = https://github.com/sgl-project/sglang
+scope = a model or serving path SGLang implements and vLLM does not, plus the SGLang correctness cross-check and performance floor
+pin = f63458b5beaceabbd9d749b9fc956370e1b649e6
+pin_label = v0.5.15
+pinned_on = 2026-07-27
+gateable = no
+evidence = #647
+```

.agents/oracles/transformers.md

@@ -0,0 +1,33 @@
+# HuggingFace `transformers` — the reference vLLM itself mirrors
+
+Where vLLM implements a model, `transformers` is not the authority: vLLM is,
+including where the two differ. What `transformers` answers is the layer beneath
+that — a model's own reference implementation, and the processors, feature
+extractors and tokenizers vLLM delegates to rather than reimplements. That is
+why it is a legitimate oracle for a preprocessing golden even on models where
+vLLM is the behavior spec.
+
+**The pin is the one resolved inside the pinned vLLM environment**, not an
+independently chosen release: `transformers` 5.14.1 is what
+[`../upstream-sync.md`](../upstream-sync.md) records alongside vLLM
+`555967922`. Pinning it separately would let the oracle environment hold two
+different `transformers` at once, which is the drift this registry exists to
+stop.
+
+Precedent for it running as an oracle: the AUDIO track's A0 captured the
+`WhisperFeatureExtractor` log-mel reference, the mel filterbank and the audio
+placeholder expansion directly from it
+([`../specs/audio-track.md`](../specs/audio-track.md) §A0), and the C++ STFT is
+gated against those goldens.
+
+```oracle-pin
+id = transformers
+role = secondary
+upstream = https://github.com/huggingface/transformers
+scope = a model, processor, feature extractor or tokenizer's own reference implementation, at the version the pinned vLLM environment resolves
+pin = 5.14.1
+pin_label = 5.14.1
+pinned_on = 2026-07-26
+gateable = yes
+evidence = .agents/specs/audio-track.md
+```

.agents/oracles/tt-forge.md

@@ -0,0 +1,42 @@
+# Tenstorrent tt-forge — the only reference for Tenstorrent execution
+
+vLLM has no Tenstorrent platform anywhere
+([`../porting-inventory.md`](../porting-inventory.md) §15), so `BACKEND-TENSTORRENT`
+is an extension with no upstream analog to mirror. That is the strongest form of
+the fallback case: not "vLLM disagrees", but "vLLM has nothing to say".
+
+**It qualifies as an oracle because it executes, not merely compiles.** tt-forge
+is an end-to-end stack over TT-Metalium with three frontends (TT-XLA for
+PyTorch/JAX, TT-Forge-ONNX, TT-Lang), running inference and training and testing
+800+ model variants in CI. A stack that only lowered IR would be a source to read,
+not an oracle to run.
+
+**`tt-forge-models` is not a second oracle.** It is the shared model-definition
+repository — a `ForgeModel` interface with `load_model()` / `load_inputs()` — and
+it publishes no reference outputs and no PCC or accuracy comparison against a
+reference framework. It supplies *subjects*, so it is named in the scope below
+rather than given a record of its own.
+
+**ttnn is not an oracle either.** Our backend is a thin `vt::` adapter over
+ttnn's C++ op library, which makes ttnn an implementation dependency. An oracle
+is something we compare against; ttnn is something we call.
+
+**Not gateable, and hardware is the reason.** tt-forge requires Wormhole or
+Blackhole silicon, and nothing in this repository has executed it — the ACTIVE
+Tenstorrent row gates its ops against our own CPU f32 path instead
+([`../backend-matrix.md`](../backend-matrix.md) `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN`).
+The Blackhole P150 that the row's real-hardware gates run on is where a tt-forge
+arm would first become measurable. There are also no release tags, so its pin will
+be a commit, not a version.
+
+```oracle-pin
+id = tt-forge
+role = secondary
+upstream = https://github.com/tenstorrent/tt-forge
+scope = Tenstorrent Wormhole/Blackhole execution, whose subjects come from tenstorrent/tt-forge-models; vLLM has no Tenstorrent platform at all
+pin = UNPINNED
+pin_label = none
+pinned_on = 2026-08-13
+gateable = no
+evidence = #647
+```

.agents/oracles/vllm-omni.md

@@ -0,0 +1,26 @@
+# vLLM-Omni — the diffusion and TTS half of the primary reference
+
+A separate repository from `vllm-project/vllm`, on its own release cadence, and
+the home of the architectures vLLM proper never registers: the MiniMax-H3 joint
+video+audio DiT, LTX-2.5, and the TTS family (`MossTTSDelayModel`,
+`MossTTSRealtime`, `Qwen3TTSForConditionalGeneration`, IndexTTS-2.5). It is
+still *vLLM* for mirroring purposes — its structure is the one we mirror — but
+it is not covered by the vLLM parity pin, which names a commit in the other
+repository.
+
+**It has no pin, and that is tracked debt, not an omission.** #633 owes it.
+Until it is pinned, every omni-only lane is compared against source read at
+whatever revision happened to be checked out, which is why H3 W3+ and LTX-2.5
+carry no oracle-run gate.
+
+```oracle-pin
+id = vllm-omni
+role = secondary
+upstream = https://github.com/vllm-project/vllm-omni
+scope = diffusion, TTS and the omni-only architectures absent from the pinned vLLM registry
+pin = UNPINNED
+pin_label = none
+pinned_on = 2026-08-13
+gateable = no
+evidence = #633
+```

.agents/oracles/vllm.md

@@ -0,0 +1,24 @@
+# vLLM — the primary oracle
+
+The mirror source and the only oracle that outranks the others. Where vLLM
+implements a behavior, it defines it; a secondary oracle disagreeing with vLLM
+loses.
+
+**The pin lives in [`../upstream-sync.md`](../upstream-sync.md)**, in its
+` ```parity-pin ` block, together with the runtime and distribution version
+strings a live oracle reports about itself and the `+g<sha>` constraint
+`assert_oracle_commit` enforces. It is restated below only as identity — the
+sync cycle advances the block over there, and `tools/bench/` reads that block,
+not this file.
+
+```oracle-pin
+id = vllm
+role = primary
+upstream = https://github.com/vllm-project/vllm
+scope = every behavior vLLM implements — defaults, modes, errors, edge cases, and both correctness and speed gates
+pin = 5559679229bc961848b121ccdeaa8fa5d79bec98
+pin_label = 0.26.0.dev0
+pinned_on = 2026-07-26
+gateable = yes
+evidence = .agents/upstream-sync.md
+```

.agents/roadmap_v1.md

@@ -182,6 +182,7 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#606](https://github.com/mudler/vllm.cpp/issues/606) | — | `vllm-serve` aborts on `--enable-auto-tool-choice` (89/157 recipes) and `--trust-remote-code` (82/157), both of which are no-ops for us, so a copy-pasted official recipe command never reaches model load. Needs an accepted-and-inert seam with a per-flag reason; no row owns serve CLI recipe compatibility | feature |
 | [#607](https://github.com/mudler/vllm.cpp/issues/607) | — | No `--language-model-only`: 43 recipes skip the vision encoder to hand its VRAM to the KV cache and we load the tower unconditionally. The flag appears in this repo only in `tools/bench/run_serve_low.py`, which passes it to the ORACLE — a grep reads as coverage and is not | feature |
 | [#608](https://github.com/mudler/vllm.cpp/issues/608) | `TOOLS-PARSER-BREADTH` | Six `--tool-call-parser` names missing (`openai`, `inkling`, `minimax_m3`, `nemotron_json`, `kimi_k3`, `ling3`), closing the last 8 of 90 official-recipe uses; four are portable at the pin, two are post-pin | feature |
+| [#647](https://github.com/mudler/vllm.cpp/issues/647) | — | Oracle policy had no fallback and no pin concept: five upstreams beyond vLLM are already compared against (vLLM-Omni, SGLang, llama.cpp, `transformers`, tt-forge) with their pins scattered across individual specs or absent entirely. AGENTS.md now admits a named secondary oracle where vLLM implements nothing, `.agents/oracles/<id>.md` pins each one file-per-oracle, and `check-oracle-pins.py` enforces both directions. The gateability debts for `sglang`, `diffusers` and `tt-forge` stay open on this issue | feature |
 
 ## Top-level portfolio
 

.github/workflows/ci.yml

@@ -170,6 +170,17 @@ jobs:
           python3 scripts/check-snapshot-pins.py
           python3 scripts/check-snapshot-pins.py --self-test
           python3 tests/scripts/test_check_snapshot_pins.py
+      - name: Every oracle is named, pinned, and honest about gateability (#647)
+        # The fallback rule (AGENTS.md "When vLLM has no implementation") admits
+        # a secondary oracle only at a recorded pin. Ungated, the registry decays
+        # the way the scattered pins it replaced already had: an upstream gets
+        # compared against at whatever revision happened to be checked out, and
+        # the pin that does exist lands in one spec where no other spec can see
+        # it. `--self-test` sweeps the fixture corpus in both directions.
+        run: |
+          python3 scripts/check-oracle-pins.py
+          python3 scripts/check-oracle-pins.py --self-test
+          python3 tests/scripts/test_check_oracle_pins.py
       - name: cuBLASLt GEMM invocation stays dtype-faithful (template parity)
         run: |
           python3 scripts/check-gemv-invocation-consistency.py

AGENTS.md

@@ -160,6 +160,50 @@ establish what actually ran. A GEMM/GEMV invocation-parity claim proves output
 dtype, compute and scale type, entry point, algorithm policy, and resolved
 template dtypes *in the same tool*.
 
+## When vLLM has no implementation
+
+vLLM — including `vllm-project/vllm-omni` — is the primary reference, and
+wherever it implements the behavior it is the only one. Where it implements
+nothing, the work is still not ungated. It runs against a **secondary oracle**,
+which is admissible only if it is one of these upstreams and only at a recorded
+pin:
+
+<!-- oracle-registry:begin -->
+
+| Oracle | Registry id | Reach for it when |
+|---|---|---|
+| vLLM | `vllm` | always, wherever it implements the behavior — the primary |
+| vLLM-Omni | `vllm-omni` | diffusion, TTS and the omni-only architectures vLLM proper never registers |
+| HuggingFace `transformers` | `transformers` | a model, processor or tokenizer's own reference implementation — the source vLLM itself mirrors |
+| `diffusers` | `diffusers` | schedulers, VAEs and diffusion pipelines |
+| SGLang, including SGLang-Omni | `sglang` | a model or serving path SGLang implements and vLLM does not |
+| llama.cpp | `llama-cpp` | CPU and GGUF k-quant floors |
+| Tenstorrent tt-forge | `tt-forge` | Tenstorrent hardware, which vLLM has no backend for at all |
+
+<!-- oracle-registry:end -->
+
+**Pin every one of them.** Each has a file of its own,
+[`.agents/oracles/<id>.md`](.agents/oracles/), carrying the revision, the date it
+was measured, whether it is gateable, and the evidence. An upstream with no pin is
+not an oracle, it is a moving target, and a number measured against it cannot be
+reproduced. The vLLM parity pin keeps its home in
+[`.agents/upstream-sync.md`](.agents/upstream-sync.md); its oracle file points
+there rather than restating it. One file per oracle, read by glob — never a
+shared table every change has to write.
+
+**A secondary oracle answers one question and no others:** what correct output
+looks like on a path vLLM cannot produce at all. It never outranks vLLM, and it
+never becomes the mirror source — behavior, defaults, structure and naming still
+mirror vLLM wherever vLLM speaks, which is why "SGLang does it differently" is
+never on its own a reason to diverge. When vLLM later implements the path, the
+row reconciles onto vLLM and says so in its spec.
+
+**Gateability is measured, not assumed.** The primary's rule holds for all of
+them: an oracle is gateable once it demonstrably *builds and runs* the model.
+Until it does, its file records `gateable = no` and names the issue that owes the
+measurement, so an ungateable lane is visible debt rather than a discovery
+someone makes mid-campaign.
+
 ## Gates
 
 Correctness first, always. Establish the declared token-exact gate — or an

scripts/agent-preflight.sh

@@ -75,6 +75,7 @@ CHECKERS=(
   check-surface-coverage
   check-test-registration
   check-snapshot-pins
+  check-oracle-pins
   check-now-current
   check-gate-commands
 )
@@ -115,6 +116,7 @@ SUITES=(
   test_check_surface_coverage
   test_check_test_registration
   test_check_snapshot_pins
+  test_check_oracle_pins
   test_cpu_x86_llamacpp_floor
   test_audit_live_rows
   test_check_gate_commands