Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/roadmap_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ issue is not yet placed. Keyed record: update in place, never append.
| [#243](https://github.com/mudler/vllm.cpp/issues/243) | — | `vllm-feature-gap-analysis.md` is a stale 2026-07-28 snapshot: 9 of 16 HIGH/MED gaps have since landed | bug |
| [#242](https://github.com/mudler/vllm.cpp/issues/242) | — | `docs/FEATURES.md` drift: arch counts say 30 (registry has 35), multimodal-over-HTTP marked ☐ though W1-W3 landed | bug |
| [#230](https://github.com/mudler/vllm.cpp/issues/230) | — | `test_agent_record.py`: 7 issue-intake tests never run, and error when they do | bug |
| [#222](https://github.com/mudler/vllm.cpp/issues/222) | `SERVE-GATE-ONLINE` | The online-serving gate aborts on current main: harness looks for `examples/server`, the build produces `examples/vllm-server` — paths repaired by `2b262622`/`8fce04d3`; residue is the drift guard, the `27n` key and a model gate that recorded a skip as passed | bug |
| [#224](https://github.com/mudler/vllm.cpp/issues/224) | `ENG-DOCS-SITE` | Publish `docs/` as a GitHub Pages site without owning a second copy | feature |
| [#203](https://github.com/mudler/vllm.cpp/issues/203) | `BACKEND-VULKAN` | Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark | bug |
| [#201](https://github.com/mudler/vllm.cpp/issues/201) | `BACKEND-ROCM` | `hipblasGemmEx` overload mismatch in `rocm_matmul_hipblaslt.hip` | bug |
Expand Down
66 changes: 66 additions & 0 deletions .agents/specs/cuda-online-serving-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,64 @@ The accepted c16 profiler keeps 16 of 48 requests resident. The active c2
diagnostic keeps two of six resident. Both admit one replacement per completion,
matching the online closed-loop client instead of preloading all prompts.

### Model keys

`tools/bench/online_gate.py` `MODEL_REVISIONS` is the only list of keys the
driver admits, and `scripts/dgx-online-serving.sh`'s `--model` guard must name
exactly the same set.

| Key | Repository | Revision | Correctness precondition |
|---|---|---|---|
| `27` | `unsloth/Qwen3.6-27B-NVFP4` | `890bdef7` | `test_qwen27_paged_engine` — a golden FOR THIS checkpoint |
| `27n` | `nvidia/Qwen3.6-27B-NVFP4` | `0893e160` | `test_qwen27_paged_engine` — **build sanity only**, see below |
| `35` | `nvidia/Qwen3.6-35B-A3B-NVFP4` | `491c2f1e` | `test_qwen36_paged_engine` |
| `q3mxfp4` | `Yi30/Qwen3-8B-MXFP4` | `b3e7ab32` | `mxfp4_smoke_battery` (#44) |

`27` and `27n` are different models, not two spellings: the unsloth repo ships a
BF16 `lm_head` and a different mixed-precision layout, the nvidia ModelOpt repo
an NVFP4 MLP + FP8 W8A8 tower + NVFP4 head. They share no goldens and their
ratios are not comparable.

**`27n` scope, stated once and recorded in the evidence.** The committed 27B
goldens belong to `@890bdef7` (`tests/parity/hf_snapshot.h`
`kQwen27NvfP4Revision`), so `test_qwen27_paged_engine` is a BUILD-SANITY
precondition for `27n`, never a golden for `@0893e160`. `record-model-gate`
writes `golden_revision`, `model_revision` and
`golden_covers_benched_checkpoint` into `preflight/model-gate/<key>.json` and
the summary revalidates them, so the distinction is machine-readable rather than
a comment a reader may miss. A `27n` **correctness** claim additionally owes a
greedy continuation against the pinned oracle on `@0893e160` itself; until that
exists, `27n` produces performance evidence with a build-sanity precondition and
no token-exact correctness claim may be made from it.

`27n` is refused by `--trace-only`: `TRACE_PRIMARY_GRAPH_CONTRACTS` holds node
and kernel-family counts captured on `@890bdef7` alone, so a `27n` trace has no
contract to validate against.

**Stop condition.** A model gate that did not compare a token is not a passed
gate. A checkpoint-gated parity test emits a loud MESSAGE and returns 0 when its
snapshot is absent, so the driver captures the gate's own output (`ctest -V`)
and `record-model-gate` fails closed unless the log carries that gate's proof
line. If the `@890bdef7` snapshot is absent on the gate host, `27n` stops here
rather than recording a vacuous pass.

**Artifact name.** The `server` target's `OUTPUT_NAME` is `vllm-server`
(`examples/CMakeLists.txt`, W6), so the built file is `examples/vllm-server`.
Every consumer resolves that name and keeps the pre-rename `examples/server` as
a REPLAY FALLBACK, so an evidence tree recorded before W6 still replays against
the binary it was recorded with. `tests/tools/test_online_gate_server_binary.py`
scans the repository for the stale spelling and admits only that fallback shape,
and only in a file that resolves the declared name too — a fallback whose
primary is deleted is an ordinary hardcode and is reported.

Issues: [#222](https://github.com/mudler/vllm.cpp/issues/222) (the harness
consumed `examples/server` while the build emits `examples/vllm-server`, so
`--execute` aborted before starting a server; the path repair itself landed on
`main` as `2b262622`/`8fce04d3`, leaving this spec to own the drift guard, the
`27n` key and the fail-closed model gate) and
[#213](https://github.com/mudler/vllm.cpp/issues/213), which this gate is the
reproducible measurement path for.

## Upstream chain

- CLI and benchmark orchestration:
Expand Down Expand Up @@ -323,6 +381,14 @@ owns the DGX campaign and its result directory.
- Provisioning, compilation, downloads, and cache warmup stay outside the
measurement lock window.
- The exact same checkpoint is required; converted weights are non-binding.
- A correctness precondition borrowed from a NEIGHBOURING checkpoint (key `27n`
on the `@890bdef7` goldens) is build sanity, and the evidence must say so in a
field, not in prose. The risk it carries is that a `27n` performance number
reads as correctness-preconditioned when only the build was proven; the
mitigation is the recorded `golden_covers_benched_checkpoint=false` plus the
proof-line requirement that makes a skipped gate fail closed.
- The gate host must hold BOTH 27B snapshots to run `27n`. It stops rather than
degrading: a `27n` run on a box without `@890bdef7` is a blocked gate.
- A profiler dependency that is importable but not executable through the
spawned EngineCore `PATH` is a failed preflight, not a reason to omit the
paired trace or reuse a lock-released series.
Expand Down
2 changes: 1 addition & 1 deletion docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -600,7 +600,7 @@ It speaks **OpenAI's Sora video shape**, so an OpenAI client works against it
unmodified, and it keeps the richer native knobs alongside.

```sh
build/examples/server --model /path/to/Qwen3.6-27B \
build/examples/vllm-server --model /path/to/Qwen3.6-27B \
--video-dit /path/to/h3-dit.gguf --video-vae /path/to/video-vae.safetensors \
--audio-vae /path/to/audio-vae.safetensors \
--video-vae-config video_vae/config.json --audio-vae-config audio_vae/config.json \
Expand Down
35 changes: 25 additions & 10 deletions scripts/dgx-online-serving.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ usage() {
cat >&2 <<'EOF'
usage:
dgx-online-serving.sh --dry-run [--claim-root DIR] [--client PATH] [--vllm-cpp-sha SHA]
dgx-online-serving.sh --prepare-corpus --model 27|35 --source-corpus DIR --evidence DIR
dgx-online-serving.sh --prepare-corpus --model 27|27n|35 --source-corpus DIR --evidence DIR
dgx-online-serving.sh --trace-only --model 27 --snapshot DIR --source-corpus DIR \
--evidence DIR --build-dir DIR --configure-log FILE [--client PATH] [--port N] \
[--trace-concurrency 2|16] [--gdn-ba-mode both] [--gdn-packed-mode both]
dgx-online-serving.sh --execute --model 27|35 --snapshot DIR --source-corpus DIR \
dgx-online-serving.sh --execute --model 27|27n|35 --snapshot DIR --source-corpus DIR \
--evidence DIR --build-dir DIR --configure-log FILE [--client PATH] [--port N]
dgx-online-serving.sh --startup-only --model 27|35|q3mxfp4 --snapshot DIR \
dgx-online-serving.sh --startup-only --model 27|27n|35|q3mxfp4 --snapshot DIR \
--source-corpus DIR --evidence DIR --build-dir DIR --configure-log FILE \
[--client PATH] [--port N]
EOF
Expand Down Expand Up @@ -107,10 +107,11 @@ if [[ ${mode} == dry-run ]]; then
exit 0
fi

[[ ${model} == 27 || ${model} == 35 || ${model} == q3mxfp4 ]] || {
echo "--model must be 27, 35 or q3mxfp4" >&2; exit 2; }
# 35B MoE prefills a wider chunk; the 27B NVFP4 dense and the q3mxfp4 MXFP4 dense
# 8B both use the dense 2048 batched-token gate value.
[[ ${model} == 27 || ${model} == 27n || ${model} == 35 || ${model} == q3mxfp4 ]] || {
echo "--model must be 27, 27n, 35 or q3mxfp4" >&2; exit 2; }
# 35B MoE prefills a wider chunk; the 27B NVFP4 dense arms (27 = unsloth,
# 27n = nvidia/ModelOpt) and the q3mxfp4 MXFP4 dense 8B all use the dense 2048
# batched-token gate value.
if [[ ${model} == 35 ]]; then
max_num_batched_tokens=8192
else
Expand Down Expand Up @@ -146,8 +147,12 @@ fi
# on 2026-07-13, and the W1D3 closure (b80663a) authorized the fresh
# binding/exact-grid rerun. Timed grids still require a production
# (profile-control-OFF) build via the recorded configure contract.
# The refusal below is about WHICH CHECKPOINT, not which architecture: 27n is a
# Qwen3.6-27B dense graph too. TRACE_PRIMARY_GRAPH_CONTRACTS holds node and
# kernel-family counts captured on the unsloth @890bdef7 27B alone, so no other
# key has a contract to validate a trace against.
if [[ ${mode} == trace-only && ${model} != 27 ]]; then
echo "H1d trace-only control is defined only for the Qwen3.6-27B dense graph; 35B performance remains held" >&2
echo "H1d trace-only control is defined only for the unsloth Qwen3.6-27B NVFP4 checkpoint (--model 27), the only key with captured graph contracts; 35B performance remains held" >&2
exit 2
fi
if [[ ${mode} == trace-only && ${trace_concurrency} != 2 && ${trace_concurrency} != 16 ]]; then
Expand Down Expand Up @@ -239,7 +244,14 @@ if [[ ${mode} == trace-only ]]; then
else
execution_manifest="${execution_dir}/${model}.json"
fi
if [[ ${model} == 27 ]]; then
if [[ ${model} == 27 || ${model} == 27n ]]; then
# Both dense 27B arms precondition on the same paged-engine suite, whose
# committed goldens are the unsloth @890bdef7 checkpoint's -- so for 27n this
# is BUILD SANITY, not a golden for @0893e160, and a 27n correctness claim
# additionally owes a greedy continuation against the pinned oracle there.
# This comment is NOT the record: record-model-gate writes golden_revision,
# model_revision and golden_covers_benched_checkpoint into
# preflight/model-gate/<key>.json and the summary revalidates them.
test_name=test_qwen27_paged_engine
gate_target=${test_name}
elif [[ ${model} == 35 ]]; then
Expand Down Expand Up @@ -1077,8 +1089,11 @@ if [[ ${model} == q3mxfp4 ]]; then
cat "${gate_log}" >&2
exit 1
fi
# -V, not just --output-on-failure: a checkpoint-gated parity test emits a loud
# SKIP MESSAGE and returns 0, so a PASSING run must also land its own output in
# the log, or record-model-gate cannot tell a real gate from a skipped one.
elif ! "${benchmark_clean_env[@]}" "${h1d_plan_env[@]}" \
ctest --test-dir "${build_dir}" -R "^${test_name}$" --output-on-failure \
ctest --test-dir "${build_dir}" -R "^${test_name}$" -V --output-on-failure \
>"${gate_log}" 2>&1; then
cat "${gate_log}" >&2
exit 1
Expand Down
2 changes: 1 addition & 1 deletion tests/tools/test_gdn_packed_component.py
Original file line number Diff line number Diff line change
Expand Up @@ -562,7 +562,7 @@ def _write_complete_evidence(root: pathlib.Path, *, packed_better: bool = True)
json.dumps(build_component_plan(SOURCE_SHA)), encoding="utf-8"
)
build = root / "build"
artifact = build / "examples" / "server"
artifact = build / "examples" / "vllm-server"
artifact.parent.mkdir(parents=True)
artifact.write_text(
"immutable MatmulNvfp4Cutlass [VT_FP4_CACHE] prepared\n",
Expand Down
18 changes: 14 additions & 4 deletions tests/tools/test_online_gate_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
MAX_NUM_BATCHED_TOKENS,
MAX_MODEL_LEN,
MAX_NUM_SEQS,
MODEL_GATE_CONTRACTS,
MODEL_REPOSITORIES,
MODEL_REVISIONS,
FLASHINFER_VERSION,
Expand Down Expand Up @@ -1002,8 +1003,11 @@ def test_trace_driver_builds_before_requiring_the_server_binary(self) -> None:
'[[ -n ${build_dir} && -f ${build_dir}/CMakeCache.txt ]]'
)
build = script.index('if ! "${build_cmd[@]}" >"${build_log}" 2>&1; then')
# The driver resolves the server artifact by name (`vllm-server`, with
# the pre-rename `server` as a replay fallback) and only then asserts it
# is executable, so the resolution line is the postflight's first token.
executable_postflight = script.index(
'[[ -x ${build_dir}/examples/server ]]'
'server_bin="${build_dir}/examples/vllm-server"'
)
self.assertLess(configured_preflight, build)
self.assertLess(build, executable_postflight)
Expand Down Expand Up @@ -2247,7 +2251,7 @@ def test_execution_model_and_trace_records_hash_their_artifacts(self) -> None:
),
encoding="utf-8",
)
(build / "examples/server").write_bytes(
(build / "examples/vllm-server").write_bytes(
b"MatmulNvfp4Cutlass\0[VT_FP4_CACHE] prepared\0"
b"[VT_CUDA_PROFILE] started\0[VT_BENCH_SHUTDOWN] ready\0"
)
Expand Down Expand Up @@ -2352,7 +2356,13 @@ def record(path: pathlib.Path) -> dict:
self.assertIn("build_command", execution["artifacts"])

gate_log = root / "gate.log"
gate_log.write_text("passed\n", encoding="utf-8")
# ctest -V output: the gate's own proof line, which a
# checkpoint-absent (and therefore token-free) run never prints.
gate_log.write_text(
f"{MODEL_GATE_CONTRACTS['test_qwen27_paged_engine']['proof']}\n"
"passed\n",
encoding="utf-8",
)
gate = record_model_gate(
root / "gate.json",
log=gate_log,
Expand Down Expand Up @@ -2420,7 +2430,7 @@ def record(path: pathlib.Path) -> dict:
"--force-overwrite=true",
"--output",
str(prefix),
str(build / "examples/server"),
str(build / "examples/vllm-server"),
"--model",
str(snapshot),
"--port",
Expand Down
Loading
Loading