From c3db78042ddf9bdcb0df37e36dc821a7c74890a0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 22:28:51 +0000 Subject: [PATCH] record: track published GHCR container images as ENG-RELEASE-CONTAINERS User-directed: alongside the downloadable archives in ENG-RELEASE-BINARIES (#117, PR #141), publish container images to GHCR from GitHub Actions. Issue #170 carries the public description. The image is not a second packaging format. It ships exactly the staged tree the release-binary matrix already defines - bin/vllm-server, VERSION, manifest, SHA256, SBOM, provenance, licenses - with ENTRYPOINT vllm-server and nothing else in it. If the two lanes grow separate layouts that is a bug, so the row records the dependency explicitly: the staging tree and install rule are owed by ENG-RELEASE-BINARIES and do not exist yet. One package, lane in the tag: ghcr.io/mudler/vllm.cpp:-cuda / -vulkan / -cpu, moving :latest-cuda / :latest-vulkan / :latest-cpu, bare :latest aliasing cpu. Version tags immutable. rocm is recorded blocked-preview tracking its binary channel (#41, #132). Every lane is a linux/amd64 + linux/arm64 multi-arch manifest built on native runners rather than QEMU. arm64 is not a courtesy here: the gate hardware this project is measured on is aarch64 throughout - GB10 sm_121a, Thor sm_110 (#168), Orin sm_87 - and an emulated CUDA build does not finish inside a runner's lifetime. Metal and MLX are recorded NOT-CONTAINERIZABLE, not deferred. There is no macOS container runtime and no Metal passthrough into a Linux VM, so those lanes can only ever be static binaries. Writing it as a boundary keeps it from being re-opened as pending work later. The accepted release-binary contract block is deliberately untouched: this is a separate distribution channel with its own artifact format, registry, tag contract and publish flow, so it gets its own row and its own future spike rather than widening a design that has already been reviewed and is guarded by a 30-test mutation suite. Entered at INVENTORIED with planned: specs/container-images.md. Under POL-SPIKE-FIRST the row cannot advance until that spike exists; no image, workflow or registry package is claimed by the ENGINE_ROWS 143 -> 144 bump. docs/STATUS.md and docs/BENCHMARKS.md carry the line the doc-checkpoint gate requires. The STATUS line is paid for out of the page, as its ratchet demands: the Tier-A1 fold paragraph was a run-by-run log of a 2026-07-30 branch, down to a literal "commit , NOT pushed", on a page whose contract is one binding current-state line per capability. It collapses to its binding result plus pointers, and the run detail was already in .agents/benchmark-record.md, .agents/state.md and the fold-plan spec. The unflattering parts stay on the page on purpose - only OLMo-2 has a committed golden, Granite and StableLM skip rather than gate, dflash and deepseek_v2 are build-verified only. Ratchet lowered 279130 -> 276960. Gates: check-agent-record (ENGINE=144), test_agent_record 13/13, test_doc_checkpoint 40/40, check-release-binary-contract, check-readme-structure, check-public-doc-tables, check-policy, check-protocol-consistency, check-commit-trailers, check-doc-checkpoint - all OK on a worktree pinned at 994cd8d4. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 5 +++-- .agents/roadmap_v1.md | 1 + docs/BENCHMARKS.md | 1 + docs/STATUS.md | 7 ++++++- scripts/check-agent-record.py | 9 ++++++++- scripts/check-public-doc-tables.py | 14 +++++++++++++- 6 files changed, 32 insertions(+), 5 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 5ae1d8a34..25aef2918 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger. | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 3 | 0 | 1 | 5 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 21 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 11 | -| Serving, API, CLI, library | 25 | 10 | 2 | 1 | 0 | 5 | 2 | 1 | 4 | +| Serving, API, CLI, library | 26 | 10 | 2 | 1 | 0 | 5 | 2 | 1 | 5 | | LoRA and adapters | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 9 | 3 | 3 | 0 | 0 | 0 | 1 | 1 | 1 | -| **Total** | **143** | **36** | **16** | **4** | **7** | **21** | **8** | **9** | **41** | +| **Total** | **144** | **36** | **16** | **4** | **7** | **21** | **8** | **9** | **42** | ## Engine core and scheduling @@ -208,6 +208,7 @@ claims it. | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp:265`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ANCHOR-BACKFILL` | CLAIM-SERVE-GATE-1 | | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - | | `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | server target only `examples/CMakeLists.txt:54-64`; CPU per-TU/runtime-dispatch baseline `CMakeLists.txt:870-890`, `src/vt/cpu/cpu_matmul_elem.cpp:553-612`, `src/vt/cpu/cpu_quant_dot_arm.cpp:39-77`; cross-family CUDA fat/per-source-gencode and multi-SM AOT gaps remain; no install/archive/publish implementation | help smoke only `examples/CMakeLists.txt:59-63`; issue `#117`; user-reviewed fat-CUDA/adaptive-CPU matrix and gates in [release-binary-matrix.md](specs/release-binary-matrix.md) | [release-binary-matrix.md](specs/release-binary-matrix.md) | `SPIKE` | `CLAIM-ENG-RELEASE-BINARIES-SPIKE` | +| `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | no Dockerfile for the server exists; the only container asset is the arm64 CPU bench cross-check `docker/Dockerfile.arm64`, which builds `vllm-bench` and is not a release artifact; staging tree, install rule and publish workflow are owed by [`ENG-RELEASE-BINARIES`](engine-matrix.md) and unimplemented | issue `#170`; no image or registry evidence | `planned: specs/container-images.md` | `INVENTORIED` | - | | `SERVE-CLI-CHAT` | Interactive `chat` and `complete` commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation | T1 | registration `vllm/entrypoints/cli/main.py:17-37,73-98`; client/model resolution + stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312` at `5559679229` | current in-process completion only `examples/cli/main.cpp:1-207`; remote command implementation absent | C-ABI stream baseline `tests/capi/test_capi.cpp:567-711`; chat-template baseline `tests/capi/test_chat_prompt.cpp:37-89`; command/fake-server tests absent | [cli-chat-complete.md](specs/cli-chat-complete.md) | `ANCHOR-BACKFILL` | `CLAIM-SERVE-CLI-CHAT-SPIKE` | | `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md). **`/v1/embeddings` LIVE 2026-08-08 (ARCH-ONE-SURFACE ROW 6, `CLAIM-EMBEDDINGS-ONE-SURFACE`):** task-conditional registration (embed/api_router.py:22-28 mirror; the route exists ONLY on a pooling-model server, and the generate routes do not — both directions socket-404-pinned), OpenAI request/response shape (string-or-array input; `dimensions`/base64/token-arrays are named-residual 400s), handler drives the ONE engine path (`LoadedEngine -> LLMEngine::embed -> registry forward -> PoolingRunner`) — the same path `vllm_embed` (ABI v15) drives. RESIDUALS: `/pooling`, `/score`, `/rerank`, `/classify` (need a classify arch) | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/embed/protocol.py:34,173-185`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | `src/vllm/entrypoints/openai/api_server.cpp` `handle_embeddings` + the `if (embedder_)` route gate; `examples/server/main.cpp` pooling task dispatch | `tests/vllm/entrypoints/openai/test_api_server.cpp` embeddings section (dispatch shape + socket smoke + BOTH-direction 404 pins) | [embeddings-one-surface.md](specs/embeddings-one-surface.md) | `ACTIVE` | `CLAIM-EMBEDDINGS-ONE-SURFACE` | | `ENG-POOLER-SEQ` | The non-generative POOLER OP — turn hidden states into a pooled embedding/logit row instead of a sampled token. **W1 LANDED + CPU-GATED 2026-07-28 (`CLAIM-POOLING`, NOT pushed):** the sequence pooling methods `CLSPool`/`LastPool`/`MeanPool` (+ `GetSeqPoolingMethod` factory) over a packed `[num_tokens, hidden]` CPU buffer keyed by a minimal `PoolingCursor` (CLS/MEAN reject partial prefill, LAST allows it, MeanPool upcasts to float32) and the activation heads `PoolerIdentity`/`PoolerNormalize` (L2 `F.normalize`)/`PoolerMultiLabelClassify` (sigmoid)/`PoolerClassify` (sigmoid if `num_labels<2` else `softmax`). Unit-gated vs DOUBLE-PRECISION references, RED-first. **W2 LANDED + CPU-GATED 2026-07-29 (`CLAIM-POOLING`, NOT pushed):** the pooler HEADS composite (`EmbeddingPoolerHead` = projector→matryoshka→normalize; `ClassifierPoolerHead` = classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` (method∩head task intersection) + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` groupby-task routing (`ForEmbedding`/`ForSeqCls` + a mixed embed+classify batch + ctor task-support validation), and the `PoolerConfig`/`PoolingParams`/`PoolingParamsUpdate` structs; `test_pooler_heads` 27/27 (240 asserts) vs double-precision refs, RED-first (disable matryoshka slice + logit_mean calibration → 8 cases / 50 asserts fail). RESIDUALS (named, spec §Work breakdown): the endpoints (W4), tokwise `AllPool`/`StepPool` (W5), a concrete pooling MODEL + real-oracle cosine gate (W3-model — see `ENG-POOLING-RUNNER`) | T2 | `vllm/model_executor/layers/pooler/seqwise/methods.py:35-121`; `vllm/model_executor/layers/pooler/activations.py:106-158`; `vllm/model_executor/layers/pooler/seqwise/heads.py:19-196`; `vllm/model_executor/layers/pooler/seqwise/poolers.py:41-138`; `vllm/model_executor/layers/pooler/special.py:23-140`; `vllm/model_executor/layers/pooler/common.py:12-30`; `vllm/pooling_params.py:35-70`; `vllm/config/pooler.py:16-90`; `vllm/v1/pool/metadata.py:13-71`; `tests/model_executor/layers/test_pooler_methods.py`, `tests/model_executor/layers/test_pooler_activations.py`, `tests/model_executor/layers/test_pooler_heads.py` | `include/vllm/model_executor/layers/pooler/{methods,activations,pooling_metadata,common,pooling_params,pooler_config,heads,poolers,dispatch_pooler}.h` + `src/vllm/model_executor/layers/pooler/{methods,activations,heads,poolers,dispatch_pooler}.cpp` — anchor `src/vllm/model_executor/layers/pooler/dispatch_pooler.cpp:13` | `tests/vllm/model_executor/layers/pooler/test_pooler.cpp` (CLS/LAST/MEAN + factory + activations, 50 asserts) + `test_pooler_heads.cpp` (Embedding/Classifier heads + SequencePooler + DispatchPooler, 240 asserts) — anchor `tests/vllm/model_executor/layers/pooler/test_pooler.cpp:81` | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 4bcd95150..b32c4f0b9 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -81,6 +81,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | H3 | `ROAD-V1-H3` | **DIFFUSION generation — a new capability class.** MiniMax-H3 (`MiniMaxH3DiTModel`): omni-modal video+audio generation via a 50-step flow-matching denoise loop, ported from vLLM-Omni. Not autoregressive: no KV cache, sampler or logits. | [`MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`](model-matrix.md) | [minimax-h3 spike](specs/minimax-h3.md) | `PARTIAL` | **W0-W2 landed 2026-08-03**: packed layout (fp64 grid bit-exact), latent packing, scheduler and the full DiT forward parity-gated vs the upstream vLLM-Omni modules at reduced dims (max abs diff 1.6e-7, 10/10 cases). **W2b device-resident forward LANDED (f32) and GPU-VERIFIED 2026-08-03** — the whole DiT graph runs with activations resident in device memory, gated vs the same upstream goldens on a Thor sm_110 GPU at video 1.49e-7 / audio 8.94e-8. Only 3 H3 kernels were needed; the port reuses the tuned shared ops. Next gate: bf16 stream + `vt::FusedChain` glue folds, then the FP4 path — which needs sm_121a, since sm_110 resolves every fp4/cutlass feature DISABLED. **HW verdict CORRECTED 2026-08-03: e2e is NOT blocked** — quantized H3 checkpoints fit (GGUF ~41 GB working set; NVFP4 likewise) and the ComfyUI-GGUF arm's 535-tensor manifest already resolves onto our contract, so e2e + a speed comparison are reachable. W7 `/v1/videos` still needs a NEW MP4/AV-encoder dependency decision. **bf16 13-SHARD RELEASE INDEXES 2026-08-07 (`row/H3-BF16-SHARDED-DIT`)**: the ORIGINAL 66.3 GB bf16 DiT (13 safetensors shards) is now resolvable through its own `model.safetensors.index.json`, with a host-f32 reference loader and `--dit ` working everywhere `--dit ` did; gated CPU-only (72/72, 54497 post-rebase) on index/name mapping and on the REAL 535-tensor geometry read from a sparse 13-shard release. The DEVICE streamer landed 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`, spec §8.14): one tensor at a time, zero host buffer for the bulk, bit-exact vs the non-streamed reference (73/73, 55203). **ENCODER + THE NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`, spec §8.15)**: the 14-shard bf16 text encoder streams too and `--encoder-only` runs it alone; measured over 233 tokens, Q4_K_M-vs-bf16 conditioning is cos 0.99745 mean / 6.85% rel RMS excl. sink / 3.5 deg median rotation — as much as a one-word prompt edit, but DIFFUSE. Whether the RENDER changes is NOT established (75/75, 55609). This unblocks the bf16-vs-quantized quality A/B; no render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the NVFP4 DiT projections now keep FP4 PACKED and route through the shared `dense_nvfp4::MatmulNvfp4W4A16D` (Marlin W4A16 — vLLM's own forced-a16 selection; SAME kernel as Laguna/dense-Qwen3 NVFP4; no new quant code); fp4-vs-bf16 WIRING gate GREEN (62/62·30039, W4A16 dispatcher runs all 11 quantized GEMMs). **GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** Marlin W4A16 RAN on sm_121a (`dense_gemms==11` default / `marlin_gemms==11` VT_MARLIN_DENSE=0, `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT; fp4 is a MEMORY win (~16 vs ~66 GB), ~0.79–0.83× the bf16 arm per diffusion forward (compute-bound large M; 3.47× faster at small decode-like M). Real-checkpoint fp4-resident t2va e2e RUNS end-to-end (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-768×1344-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render-coherence bug (device VAE decode / denoise), separate from the fp4 speed work. vLLM-Omni serves NO quantized H3 (BF16-only) -> HW/loader-forced-indirect (4×B300 209f render 86.964 s vs 1×GB10 209 s/forward). **2026-08-08 ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** the public 0/1 selector is mapped once to generic `DeviceType`; DSR returns 34→32 with the baseline/allowlist unchanged; CPU compile/fold test pending in CI due shared-disk pressure. | | 15 | `ROAD-V1-D6` | **llama.cpp device breadth folded into scope (user-directed 2026-08-05):** the 11 ggml backends vLLM has no platform for — cann, musa, opencl, openvino, rpc, webgpu, zdnn, zendnn, hexagon, blas, virtgpu — inventoried as `BACKEND-GGML-*`. **SPIKES FIRST:** no implementation before each row's `.agents/specs/.md` clears the spike contract, per the standing directive. vLLM stays the mirror source; llama.cpp is the breadth reference. | [backend matrix](backend-matrix.md) | ☐ per-row spike required | `INVENTORIED` | first spike accepted | | REL | `ROAD-V1-RELEASE` | KISS downloads per OS+host ABI: one adaptive CPU binary and one fat CUDA binary covering every supported SM; per-SM CUDA artifacts are optional diagnostics; stable channels require matching runtime evidence and build-only paths stay preview | [`ENG-RELEASE-BINARIES`](engine-matrix.md) | [release binary matrix](specs/release-binary-matrix.md) | `SPIKE` | Fresh review of PR #129, then W1 cross-family per-source gencode; multi-SM AOT and CPU ISA audits precede bundle work; no archive exists | +| IMG | `ROAD-V1-CONTAINERS` | **Published container images on GHCR, built by GitHub Actions (user-directed 2026-08-08).** The same staged bundle `ROAD-V1-RELEASE` defines, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane — and `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image, every supported SM), `vulkan`, `cpu`, plus `rocm` blocked-preview; version tags immutable, every `latest-` moves. Every lane is a `linux/amd64`+`linux/arm64` multi-arch manifest on native runners, because the project's own gate hardware (GB10, Thor, Orin) is arm64. Metal and MLX are NOT-CONTAINERIZABLE and stay static-binary-only — a recorded boundary, not pending work. Depends on the `ROAD-V1-RELEASE` install/stage tree: the image IS the bundle, so the two lanes must not grow separate layouts. No image, workflow or registry package exists. | [`ENG-RELEASE-CONTAINERS`](engine-matrix.md) | none yet: `planned: specs/container-images.md`; the row cannot leave `INVENTORIED` without it | `INVENTORIED` | Write the container-image spike spec (lane matrix, base images, manifest/provenance, retention, least-privilege publish flow) against the accepted release-binary contract; no image work starts before it | An area row cannot enter `READY` without a real spike under `specs/`, and cannot enter `DONE` without exact code and test/evidence anchors. Closed execution diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 15ab81f30..09d11314c 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -5,6 +5,7 @@ | Reference | Workload | Headline | Tokens | |---|---|---|---| | **Binary release matrix (spiked)** | `ENG-RELEASE-BINARIES`: primary host-ABI fat-CUDA + adaptive-CPU static-core bundles; optional per-SM diagnostics; experimental literal-static musl CPU | **PENDING:** pins 10-SM fat CUDA, adaptive no-AVX2 CPU, W1-W13/W10-W12 policy, public pending states; 30 tests GREEN. No archive, staged smoke, runtime, correctness, or performance evidence | n/a | +| **Container images (inventoried)** | `ENG-RELEASE-CONTAINERS`: GHCR images from GitHub Actions; lanes `-cuda`/`-vulkan`/`-cpu`, amd64+arm64 manifests | **No number owed:** record-only. No Dockerfile, workflow, registry package or image exists; the image is the unimplemented `ENG-RELEASE-BINARIES` bundle | n/a | | **Developer agent entry point (implemented)** | `DOCS-AGENT-PROTOCOL-ENTRYPOINT`: public contribution guide + synchronized, mutation-gated pre-claim intake rule | Rebased documentation/protocol only; benchmark void | n/a | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | | **DSR fix: async readback capability (2026-08-08)** | **No number owed**: behavior-neutral (CPU/CUDA async-ON, discrete non-CUDA async-OFF, unchanged); moves a `kCUDA` check onto `Backend`, unblocking red CI on #127/#154/#155 | diff --git a/docs/STATUS.md b/docs/STATUS.md index 090d32d07..f2cbe0dc4 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -36,6 +36,11 @@ Startup-latency axis (2026-08-07): `MEASURED / provisional`. Cold launch to firs 6 legs contended, uncontended repeat died with a host reboot. [Detail](../.agents/specs/startup-latency-axis.md). +Container images (2026-08-08): `INVENTORIED`, nothing built. `ENG-RELEASE-CONTAINERS` records GHCR images +published by GitHub Actions - lanes `-cuda`/`-vulkan`/`-cpu` plus moving `latest-cuda`/`latest-vulkan`/`latest-cpu`, +each an amd64+arm64 manifest. The image is the `ENG-RELEASE-BINARIES` staged bundle, which does not exist yet, so +no Dockerfile, workflow or registry package is claimed. Metal/MLX are NOT-CONTAINERIZABLE. Issue #170. + Protocol (2026-08-08): `agent-start.py` → claim → preflight; review FAIL → fresh implementer → both gates → fresh review until PASS. Budgets cannot stop it; registry/prompt mutation gates bind it. Verified PRs merge; obsolete close. @@ -2151,7 +2156,7 @@ The gate models' W4A4 decode is at the achievable roofline. See `.agents/specs/n **DeepSeek-V4 ds4-gap STUDY — 16.5 IS reachable, ranked lever plan (2026-07-30, research/profiling only, `.agents/specs/deepseek-ds4-gap-lever-plan-2026-07-30.md`).** Fresh nsys-diff (ours 85.35 ms/step = 11.2 t/s vs ds4 54.86 ms = 17.2 t/s; gap 30.5 ms, 98.5% GPU-bound; roofline ours 42–45% vs ds4 65–70% of 240 GB/s). **No irreducible gap** — the 30.5 ms is fully attributed to specific kernels, each with a ds4 file:line precedent. RANKED LEVERS (measured-%-anchored): **L1 dense Q8_0 preq-prologue fusion** (~9 ms; a MEASURED still-present 9.1 ms activation-quant DRAM round-trip — DISTINCT from the refuted Brick 8, which fused into the grouped-MoE kernel; near-tie; → ~13.5 t/s, the go/no-go), **L3 route→warp-topk** (6.5 ms, BIT-EXACT, → ~14.5), **L4 QKV+norm+RoPE+KV fusion** (4.0 ms, near-tie, → ~15.3), **L2 Q8_0 sub-warp GEMV tiling** (speculative occupancy lever, short-K only, 5–8 ms → ~15.8–16.5). Measurement REJECTED as wishful: MHC (measured TIE), routed-MoE (we WIN +2.6 ms via Brick 6), attention/fp8-KV (0.6 ms short-ctx), stream-overlap (98.5% GPU-bound). Honest band: **15.3 t/s sure (L1+L3+L4 bit-exact/near-tie), 16.5–17.2 if L2 tiling lands.** Implementation follows the gated sequence with explicit stop conditions. Row stays `ACTIVE`. -**Tier-A1 fold — 5 bf16 SwiGLU-MLP archs onto the SHARED gate-up seam `layers::UnquantizedMlpGateUpMethod` (2026-07-30, branch `worktree-wf_9054a036-47d-1`, commit ``, NOT pushed).** First EXECUTED tier of the cross-arch merged-GEMM fold plan (`arch-fusion-fold-plan-2026-07-30.md` §A1) AND the empirical REUSE PROOF the keep-quant generalization front (`bb864b0d`) could not run in keep-quant (DeepSeek is the only keep-quant MoE model). Each arch's hand-rolled `{ResidentWeight; MatmulBT[2I,H]; SiluAndMul}` gate-up sequence is REPLACED by the already-shared bf16 gate-up MLP method (exemplar `qwen3.cpp:91`), so multiple real gated archs now route their MLP through ONE tuned method and inherit the nvfp4 `GateUpFusedMarlinD` fused arm FOR FREE the moment a quantized checkpoint ships. **BYTE-FOR-BYTE identical op sequence — no loader concat needed** (every arch already packs a merged `[2I,H] gate_up_proj` `OwnedTensor`). Folded: **OLMo-2** (`olmo2.cpp`), **StableLM** (`stablelm.cpp`), **qwen3_dflash** (`qwen3_dflash.cpp`, was hand-merged but off-seam), **deepseek_v2** (`deepseek_v2.cpp` `DenseMlp` — one fold covers BOTH the dense-layer MLP and the shared-expert epilogue, its two call sites) via the direct `UnquantizedMlpGateUpMethod` arm; **Granite** (`granite.cpp`) via the full `layers::MakeMlpGateUpMethod` factory (its MLP weights ARE `Qwen3DenseMlpWeights`, so it is the exact exemplar and is nvfp4-ready today). The shared method's TU (`linear.h`/`nvfp4.h`) is UNTOUCHED — only new `#include`s in the 5 model TUs — so 27B/35B/qwen3_dense stay strict without re-gate. **EMPIRICAL GATES (DGX GB10 sm_121a, RelWithDebInfo + CUTLASS 4.5.0 + Triton-AOT — the PRODUCTION stack; configure log proves `CUTLASS … enabling sm120a NVFP4 cutlass GEMM` + `Triton AOT … vendored … sm_121a`).** (1) CPU composite-golden RED-first unit case in `test_linear_method` — the fused seam `UnquantizedMlpGateUpMethod::Apply` is BYTE-IDENTICAL to the standalone `{MatmulBT[2I,H]; SiluAndMul}` (RED-first PROVEN on the dev box: a perturbed reference weight fails all 15 byte checks; GREEN after revert) → on the DGX production CUDA stack `test_linear_method` **4/4 cases · 31 assertions** GREEN. (2) **OLMo-2 SACRED paged-engine gate RAN + PASSED: `test_olmo2_paged_engine` 1/1 · 92 assertions, 16/16 prompts (13/16 STRICT token-exact vs vLLM per-prompt greedy + 3/16 near-tie-band, max gap 0.094 nats, 0 forward-divergent) — IDENTICAL to the pre-fold bring-up result (`5c00fc4`), and the test's HARD anchor-drift REQUIRE against the committed pre-fold `our_ids.npy` PASSED, i.e. the folded engine produced the EXACT SAME token sequence → the fold is a proven byte-exact NO-OP on the real GPU forward through the shared seam.** Golden md5 UNCHANGED before/after (no golden overwrite). **HONEST SCOPE:** only OLMo-2 has a committed golden in-tree — Granite and StableLM goldens were NEVER committed (the `granite_greedy_2b`/`stablelm_greedy_1_6b` dirs the test comments name do not exist in the repo or on the box), so their paged-engine tests LINK + LOAD-verify + skip-on-absent-golden (rc=0, build-verified on the production stack) but do NOT run a vs-vLLM token gate THIS pass; capturing their oracle goldens is available follow-up. Granite/StableLM route through the IDENTICAL shared method OLMo-2 just proved token-exact on GPU, and are covered by the CPU composite golden. dflash + deepseek_v2: build-verified on the production stack + covered by the CPU composite golden. **Net: 1 arch empirically token-exact-gated end-to-end on GPU (OLMo-2, the shared-seam anchor) + the reuse proof (CPU composite RED-first) + 5/5 build-verified on the production stack.** No new env flag (the fold is unconditional and bit-exact; the shared method carries the existing nvfp4 arm selection). Next tiers: A2 (MLA A-proj 3→1 merge) / A4 (bf16 grouped-MoE `kMoeGateUpSwiGLU`). +**Tier-A1 fold - 5 bf16 SwiGLU-MLP archs onto the shared gate-up seam `layers::UnquantizedMlpGateUpMethod` (2026-07-30).** OLMo-2, StableLM, qwen3_dflash, deepseek_v2 (dense MLP + shared-expert epilogue) and Granite now route their MLP through one tuned method and inherit the nvfp4 `GateUpFusedMarlinD` fused arm for free; the op sequence is byte-for-byte identical, no loader concat. Gated on the DGX production stack: `test_linear_method` 4/4 (RED-first composite golden) and the OLMo-2 SACRED paged-engine gate 1/1 - 13/16 STRICT token-exact + 3/16 near-tie, identical to the pre-fold result, with the anchor-drift REQUIRE against the committed pre-fold ids passing, so the fold is a proven byte-exact no-op on the real GPU forward. **Honest scope: only OLMo-2 has a committed golden**; Granite and StableLM goldens were never captured, so those tests link, load-verify and skip rather than running a vs-vLLM token gate, and dflash + deepseek_v2 are build-verified plus CPU-composite-covered only. Run detail in [.agents/benchmark-record.md](../.agents/benchmark-record.md) and [.agents/specs/arch-fusion-fold-plan-2026-07-30.md](../.agents/specs/arch-fusion-fold-plan-2026-07-30.md); next tiers A2 (MLA A-proj 3->1) / A4 (bf16 grouped-MoE). **Q8_0 GEMV sudo-ncu root cause (2026-07-30, measurement-only, `.agents/specs/ds4-q8-ncu-2026-07-30.md`).** sudo ncu (now unblocked) on our `QuantDotGemmQ8_0Kernel`: **LATENCY-bound** (long-scoreboard stall 54.4) at 71.9% occupancy, L1 hit 96.7% (34-byte over-fetch L1-absorbed → not the DRAM bottleneck), L2 10.7%. Explains why Bricks 4/11/12 (alignment/occupancy/launch-count) were all flat — the axis is memory-LATENCY, not those. The one untried lever = ILP (multiple output-rows/thread → independent load streams). ds4 ncu side not captured (graph-decode needs --graph-profiling). No production code changed. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index d09c0cf3a..68f6bd2ea 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -308,7 +308,14 @@ # server implementation. Inventoried while its release-matrix spike is written; # no packaging support is claimed by the count bump. # Bumped for a real new row, never to make a failing state transition pass. -ENGINE_ROWS = 143 +# 144 since 2026-08-08: +`ENG-RELEASE-CONTAINERS` (published GHCR container +# images built by GitHub Actions — a distribution channel distinct from the +# downloadable archives in `ENG-RELEASE-BINARIES`: different artifact format, +# registry, tag contract, multi-arch manifest and publish flow, sharing only the +# staged bundle. User-directed, issue #170; inventoried while its spike is +# written, and no image, workflow or registry package is claimed by the bump. +# Bumped for a real new row, never to make a failing state transition pass. +ENGINE_ROWS = 144 MATRIX_PATHS = [ENGINE_MATRIX, *(path for path, _ in MATRICES.values())] REQUIRED = [ diff --git a/scripts/check-public-doc-tables.py b/scripts/check-public-doc-tables.py index 80f38fdb7..eb5657a1c 100755 --- a/scripts/check-public-doc-tables.py +++ b/scripts/check-public-doc-tables.py @@ -389,7 +389,19 @@ def features_errors(text: str) -> list[str]: # community-green, approach-(b) F6 fix in blind-unverified, M2 on # verification — paid for inside the same parenthetical; the board/arch # detail lives in docs/ROCM.md and .agents/backend-matrix.md. Net -12. - "chars": 279130, + # + # 276960 since 2026-08-08 (measured 276945): the container-image lane + # (`ENG-RELEASE-CONTAINERS`, issue #170) owes this page a status line, paid + # for OUT of the page. The Tier-A1 fold paragraph was a run-by-run log of a + # 2026-07-30 branch - down to a literal `commit , NOT pushed` - on a + # page whose contract is ONE binding current-state line per capability. It + # collapsed to its binding result plus pointers, with the run detail already + # present in .agents/benchmark-record.md, .agents/state.md and + # specs/arch-fusion-fold-plan-2026-07-30.md. The unflattering parts were kept + # ON the page deliberately: only OLMo-2 has a committed golden, Granite and + # StableLM skip rather than gate, dflash and deepseek_v2 are build-verified + # only. Nothing was lost, only de-duplicated. Net -2185. + "chars": 276960, "h2_sections": 11, "long_paragraphs": 89, "oversized_cells": 47,