Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/NOW.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# NOW — the one-Read resume surface

<!-- now-updated: 2026-08-07 -->
<!-- now-updated: 2026-08-08 -->

Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live,
the gate being chased, what to do next. Never a log — evidence lives in the
Expand Down Expand Up @@ -28,7 +28,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
| Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — |
| `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning |
| `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable |
| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 IN REVIEW (`row/DEVICE-KNOB`)**: ABI v14 `device` knob, absent-cuda fails loud, `--device` on server+cli; **#135 H3 device-seam repair IN** (ABI 0/1 via `DeviceType`, DSR 32) | Merge after review; CUDA A/B |
| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 LANDED; #138 repair CPU-GREEN**: ABI v14 stable; named platform resolves via registry; DSR 39→32 (`kcuda=0`), H3 untouched | Review #138; CUDA A/B residual |

In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna
bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.
Expand Down
13 changes: 13 additions & 0 deletions .agents/coordination.md
Original file line number Diff line number Diff line change
Expand Up @@ -1670,6 +1670,19 @@ this claim will meet. The tiled row is speed-gatable on dgx.

## Closing and archival

- 2026-08-08: `CLAIM-ARCH-ONE-SURFACE-DEVICE-LEAKAGE` CLOSED on draft PR
#138. Base `b44ad337` inherited PR #136's seven shared-layer CUDA literals
(DSR 39 vs baseline 32). The repair keeps ABI-v14 0/1/2 and the public
`auto`/`cpu`/`cuda` names, represents slot 2 internally as a named-platform
selection, resolves its canonical name through `FindPlatformByName`, and
propagates the registry's `DeviceType` without a backend-specific shared
branch. RED-first compiler failure pinned the new seam; the non-CUDA `kXPU`
resolver assertion kills a hidden constant return. GREEN: DSR 32
(`kcuda=0`) with baseline/allowlist unchanged; checker mutations 25/25;
CPU Release `-Werror` `test_platform` 11/11·85,
`test_loaded_engine_dense` 9/9·65, `test_capi` 45/45·428. H3 source was
untouched; no CUDA execution, download, benchmark or service change.

A block closes only when every row in its declared scope is `DONE`. In the same
change:

Expand Down
1 change: 1 addition & 0 deletions .agents/parity-ledger.md
Original file line number Diff line number Diff line change
Expand Up @@ -926,3 +926,4 @@ Columns:
| 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; MEASUREMENT, lifecycle unchanged) | **MiniMax-H3 - THE NUMBER: what quantizing the TEXT ENCODER to Q4_K_M does to the conditioning.** Same prompt (wuxia, 233 tokens), same tokenizer, same 50-layer truncation, same `MiniMaxH3EncoderTextForwardDevice`, same f32 activations - only the weight bytes differ (Q4_K_M ggml blocks vs the original bf16 14-shard release); both arms self-report IDENTICAL geometry (50/5120/64/8/128/25600), which is what establishes they are the same model. Conditioning `[233, 5120]` f32 via `--encoder-only --save-embeds`. | Not a vLLM-parity change: H3 is BEYOND-PIN (vllm-omni), and vLLM-Omni serves NO quantized H3 at all (BF16-only), so there is no upstream arm to compare against - the bf16 release IS the reference here, and it is the one the loader added in `6d454b00` makes runnable. The quantization-sensitivity premise is ComfyUI PR 15298 (H3's partial split-half RoPE produces channel-wise magnitude outliers that corrupt even INT8), and the measurement CONFIRMS its mechanism concretely: token 0 is an attention sink at norm 15,522 vs a 366 mean (42x), carrying 68% of the total squared error with its DIRECTION intact (cos 0.99962). | **MEASURED on Thor sm_110, build `d1085374` (built and measured as `d1085374`, amended for the row-branch trailer; IDENTICAL tree `dd9283cf`, so the measurement binary IS this commit), GPU idle.** Q4_K_M vs bf16: max|diff| 154.0, RMS 0.5045, rel RMS **0.03403** (0.06849 excluding the sink token), per-token cosine min 0.90916 / mean **0.99745** / median 0.99810, rotation median 3.535 deg / max 24.61 deg, 232 of 233 tokens below cosine 0.999. NOT a scale change: norm ratio mean 0.99010 but the best global rescale only moves 0.03403 -> 0.03280, so it is DIRECTIONAL. CALIBRATION arm (bf16 encoder, ONE-WORD prompt edit `at night`->`at dawn`, also 233 tokens): rel RMS 0.01897 / 0.06666 excl. sink, cosine mean 0.99769 median 0.99963, 172 of 233 tokens above 0.999. So quantization moves the conditioning by the SAME total energy as rewriting a word of the prompt (6.85% vs 6.67%) but with the opposite SHAPE - diffuse over every token instead of concentrated on the words that changed. Cost: Q4 arm 40 s / 18.0 GiB peak; bf16 arm 40 s / 45.41 GiB uploaded / host conversion peak 0.0195 MiB / 51.95 GiB total peak, streamer counters `layers=50 tensors=400 direct=350 converted=200 fused=100` proving the shard path ran. `benchmark_binding=false` (no throughput claim). EXPLICITLY NOT ESTABLISHED: that the RENDER changes - nothing here measures the DiT's sensitivity to a 3.5-degree median rotation; the owed follow-up is a same-DiT/same-seed render A/B over the two saved embeds. |
| 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 ABI-v12 device dispatch follow-up; lifecycle unchanged) | **MiniMax-H3 video engine device selection is backend-parameterized.** The stable public selector remains 0=CPU / 1=CUDA, but shared code maps it once to `vt::DeviceType`, creates one queue through `GetBackend(device_type)`, and uses that queue's device instead of naming CUDA twice. | vLLM-Omni pipeline ownership remains unchanged; this is a vllm.cpp C-ABI/backend-seam correction with no upstream behavioral delta. | RED: DSR 34 (`kcuda=2`) vs baseline 32. GREEN: DSR 32 with baseline/allowlist unchanged; checker mutations 25/25. New fold unit pins 0/1 and invalid selectors; CPU compile/test NOT RUN locally because the shared filesystem reached 100%, pending GitHub CI. No GPU, download, or performance claim. |
| 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 device-dispatch mutation-gate follow-up; lifecycle unchanged) | **Pins the single-queue and device-provenance contract through the real `MiniMaxH3VideoEngine::Load` path.** Adds a read-only engine `device()` query and a counting CUDA-backend fixture whose queue reports a distinctive CPU:7 device. | No vLLM-Omni behavior changes; this is test observability for the vllm.cpp backend seam above. | Reviewer mutation reproduced first: replacing `im.device = stream_queue.device` with a second `CreateQueue().device` remained GREEN at 5 cases / 135 assertions. RED-first test then reported `2 == 1`; restoring reuse is GREEN at 6/137. Independent mutation deleting queue-device provenance fails the CPU:7 assertion. DSR remains 32 and all 25 leakage-checker mutations pass. No GPU, download, generation, or performance claim. |
| 2026-08-08 (`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE`; `ARCH-ONE-SURFACE` ROW 8 follow-up; PR #138; CPU-only; lifecycle unchanged) | **Removes PR #136's seven shared CUDA literals without changing ABI-v14 device selection.** Wire 0/1/2 and public `auto`/`cpu`/`cuda` stay exact; internal slot 2 becomes a named-platform tag, `FindPlatformByName` resolves the canonical name, and shared loading propagates the registered `DeviceType`. Explicit CPU, absent-CUDA-before-I/O and CAPI slot 2 are preserved; H3 is untouched. | vLLM `config/device.py:13,61-66` @ `555967922`: supported device name assigned verbatim and never silently substituted. The registry lookup is the in-tree Platform/Backend portability seam; no upstream behavior delta. | RED: inherited DSR 39 (`kcuda=7`) and focused compile failures for the new enum/API/signature. Mutation pin: resolver input `kXPU` must return `kXPU`. GREEN: DSR 32 (`kcuda=0`) with baseline/allowlist unchanged; checker suite 25/25; CPU Release `-Werror` `test_platform` 11/11·85, `test_loaded_engine_dense` 9/9·65, `test_capi` 45/45·428. No CUDA runtime or performance claim; GPU A/B remains residual. |
23 changes: 22 additions & 1 deletion .agents/specs/one-surface-abi.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ONE SURFACE — every capability ships through the C ABI

Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; the #123 review's three MINOR findings closed in the same change (c_header_compile.c now references the v11+v12 surface + the v14 field; the v12 changelog block moved to chronological position; the fold fixture's flag list gained `--keep-quant`). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.**
Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; PR #138 follow-up removes PR #136's seven shared CUDA literals by resolving the stable public name through the platform registry and propagating its `DeviceType` (DSR 39 -> 32, `kcuda=0`, baseline/allowlist unchanged). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.**

## The defect

Expand Down Expand Up @@ -202,3 +202,24 @@ must report that exact device after exactly one `CreateQueue` call. The original
second-queue mutation was false-GREEN at 5/135, then RED at `2 == 1`; deleting
the queue-device reuse is independently RED. The final CPU fold target is GREEN
at 6 cases / 137 assertions in the isolated `/dev/shm` build.

### ROW 8 shared-device follow-up (`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE`, PR #138)

PR #136 implemented ABI-v14 explicit selection correctly but encoded its CUDA
identity seven times in shared configuration/loading code, regressing DSR from
32 to 39. The repair preserves the wire values and public names exactly:
`Device::kNamedPlatform` remains integer 2 and `DeviceName()` remains `"cuda"`.
Shared loading now passes that canonical name to `FindPlatformByName`, then
propagates the registered platform's `DeviceType`; no CUDA `DeviceType` literal
remains in the shared selector. This is a real abstraction rather than textual
evasion: the pure resolver test supplies `kXPU` and requires `kXPU` back, while
the registry test walks every registered canonical device name. Explicit CPU
still ignores the accelerator lookup, absent CUDA still throws before model
path I/O, the C ABI still maps slot 2, and H3 dispatch is untouched.

RED was the focused compiler failure for the missing enum/API/signature plus
the inherited checker result (`kcuda=7`, DSR 39). GREEN is DSR 32 with
`kcuda=0`, unchanged baseline/allowlist, all 25 checker mutations, and a CPU
Release `-Werror` build: `test_platform` 11/11·85,
`test_loaded_engine_dense` 9/9·65 and `test_capi` 45/45·428. No CUDA runtime was
used; the existing GPU A/B residual remains.
2 changes: 1 addition & 1 deletion .agents/specs/surface-coverage-2026-08-07.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows).
| 5 | Kimi-Linear incremental | expose the incremental decode path through the runner/engine (the recompute forward already routes) | rewrite `kimi_linear_gen` | S–M | `KimiDecodeCache` on the runner |
| 6 | Embeddings/pooling | `vllm_embed`/pooling entry point + live `/v1/embeddings`; register a pooling arch (`is_pooling_model=true`); invoke `PoolingRunner` in the step | — | M | pooler live-wiring |
| 7 | Multimodal input | multimodal-content entry point on `vllm_chat`; run the vision/audio tower in the engine step (`mm_features`→`ModelForwardInput.mm`) | wire `chat_mm` seam into the ABI | L | `MM-SERVE-E2E` engine mm-forward residual |
| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue` explicit arms via `LoadedEngine::ResolveExplicitDeviceType`; explicit cpu never probes, explicit ABSENT cuda fails LOUD before any model I/O (device.py:61-66 never-substitute mirror); `--device` on server + cli | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` |
| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`; leakage follow-up PR #138)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue`; the stable public name resolves through `FindPlatformByName` and its registered `DeviceType` is propagated without a shared CUDA literal; explicit cpu never probes, explicit ABSENT cuda fails LOUD before model I/O; DSR 32 / `kcuda=0` | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` |
| 9 | Voxtral + audio chat seam | register `VoxtralForConditionalGeneration` + fold `VoxtralGenerateGreedy` into the registry forward; audio-capable chat fn + an engine consumer for `AudioKwargs` mm_features | rewrite tests→clients | M | mirror upstream `voxtral.py:309`, `SupportsTranscription` |
| 10 | Gemma-4 audio e2e | bf16 device audio forward + audio→text merge (residual `gemma4_audio.h:41`); USM log-mel front end | fold into the registered mm forward | M | `MM-SERVE-E2E` |
| 11 | Tokenizer/bench ABI + real-load gate | `vllm_tokenize`/`vllm_detokenize`; token-id/count fields on the stream callback for bench; **gate `vllm_engine_load` on a REAL tiny checkpoint at least once** (today bad-path only, `test_capi.cpp:474`) | rewrite `tokenize`/`bench` as clients | S-M | `/tokenize` route exists (`api_server.cpp:432`) |
Expand Down
27 changes: 27 additions & 0 deletions .agents/state.md
Original file line number Diff line number Diff line change
Expand Up @@ -42468,3 +42468,30 @@ records main-wholesale + these rows re-applied. (4) Explicit
names for xpu/vulkan/metal/rocm are additive later; auto reaches them today.
(5) The transcription stack refuses explicit cuda (CPU pipeline) — revisit
when an accelerated transcription path exists.

## 2026-08-08 — ARCH-ONE-SURFACE ROW 8 device leakage repaired through the platform registry (PR #138)
<!-- state: 2026-08-08T13:00 -->

Current main `b44ad337` inherited PR #136 with seven CUDA enum literals in the
shared device config/loader: DSR was 39 vs the immutable baseline 32
(`kcuda=7`). RED-first focused tests required a platform-neutral ABI enum tag,
a canonical-name registry lookup and an optional resolved `DeviceType`; the
old API failed to compile on each missing seam.

The ABI and public behavior are unchanged: values remain 0=auto, 1=cpu,
2=cuda; parsing and `DeviceName()` still expose `"cuda"`; absent CUDA still
fails loudly before model-path I/O; explicit CPU still bypasses accelerator
selection. Internally slot 2 is `Device::kNamedPlatform`; shared loading calls
`FindPlatformByName(DeviceName(device))` and propagates the registered
platform's type. A mutation-sensitive test supplies `kXPU` and requires
`kXPU` back, so replacing the propagation with a hidden CUDA constant is RED.
The platform test walks every registered canonical name and pins missing-name
failure.

GREEN in `/dev/shm/vllm-device-leakage-build`, CPU Release, CUDA OFF,
`-Werror`: `test_platform` 11/11 cases · 85 assertions;
`test_loaded_engine_dense` 9/9 · 65; `test_capi` 45/45 · 428. The leakage
checker reports DSR 32 (`kcuda=0`, `is_cuda=0`, `cuda_inc=0`, `vt_ifdef=32`)
with baseline/allowlist unchanged, and its mutation suite is 25/25. H3 source
is untouched. No CUDA runtime, model download, service action, benchmark or
release artifact occurred; the existing CUDA-build A/B remains pending.
2 changes: 1 addition & 1 deletion docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one.

| Track | Status | Next gate |
|---|---|---|
| Surface coverage (`ARCH-ONE-SURFACE`) | **No number owed**: CPU tooling/plumbing (guard; ROW 8 device knob, ABI v14 `device`, zero value = the probe, byte-identical); no kernel path touched | Fold lanes carry their own gates; ROW 8 CUDA-build A/B = named residual |
| Surface coverage (`ARCH-ONE-SURFACE`) | **CORRECTNESS COMPLETE:** #138 restores DSR 32 (`kcuda=0`) via registry/name resolution; ABI-v14 selection unchanged; no speed claim | CPU selector/platform/C-ABI tests green; CUDA A/B remains residual |
| 35B prefill TTFT | 0.93x to 0.98x at every concurrency (2026-08-05) | Attribute the residual, then close |
| 35B low-batch MoE decode | CLOSED at low batch (c1 0.975x, c4 wins); c16 0.93x. `VT_ASYNC_DEVICE_MIRROR` **default ON for correctness**. `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) A/B'd speed-NEUTRAL | c16 lever is prefill glue (task #61), not the decode drain. `test_qwen36_async_serving` GREEN |
| CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) |
Expand Down
7 changes: 4 additions & 3 deletions docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1563,9 +1563,10 @@ runtime-verified yet.
lanes. Guard `scripts/check-surface-coverage.py` (two axes, preflight + CI):
every `examples/*` unit is a client of `include/vllm.h` or tracked to a fold
row; every `FEATURES.md` C-ABI capability names an entry point or is tracked.
Fold ROW 8: `vllm_model_params.device` (ABI v14, 0=auto/1=cpu/2=cuda);
explicit cpu forces CPU, an absent named device fails loud (never
substituted); `--device` on server+cli. CUDA-build A/B = named residual.
ROW 8 ABI-v14 device selection is behavior-complete; #138 repairs #136's
shared-layer DSR 39->32 through registry/name resolution (`kcuda=0`) with the
baseline and allowlist unchanged. CPU selector/platform/C-ABI gates are
green; CUDA-build A/B remains the named residual.
- **Automatic prefix caching (APC)** is on by default for dense models (hybrid /
GDN and attention-free default off, mirroring vLLM), and it now has an
end-to-end cache-ON gate on `Qwen/Qwen3-4B` (a shared common prefix reused
Expand Down
Loading
Loading