Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .agents/backend-matrix.md

Large diffs are not rendered by default.

26 changes: 26 additions & 0 deletions .agents/benchmark-record.md
Original file line number Diff line number Diff line change
Expand Up @@ -16177,3 +16177,29 @@ the ~30 flushes/token. The declines name exactly three ops: `kCausalConv1dFwd`
(op 5, prefill only), `kRopeCosSinCache` (op 66) and `kAttnQkNormRopeGate`
(op 67). The last two run per full-attention layer, 16 per token, and are the
next lever; they are kernel work, not plumbing.

## Rolled out of the scoreboard on 2026-08-08

Moved verbatim from `docs/BENCHMARKS.md` by `scripts/roll-benchmark-record.py`. Nothing edited or deleted.

## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700)

| Path | Warm tok/s | Notes |
|------|------------|--------|
| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill |
| server `/v1/completions` | ~38 | exclusive |
| server `/v1/chat` thinking off | ~32 | after expert cache |
| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack |

## Rolled out of the scoreboard on 2026-08-08

Moved verbatim from `docs/BENCHMARKS.md` by `scripts/roll-benchmark-record.py`. Nothing edited or deleted.

## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700)

| Path | Warm tok/s | Notes |
|------|------------|--------|
| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill |
| server `/v1/completions` | ~38 | exclusive |
| server `/v1/chat` thinking off | ~32 | after expert cache |
| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack |
38 changes: 19 additions & 19 deletions .agents/engine-matrix.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions .agents/feature-matrix.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions .agents/kernel-matrix.md

Large diffs are not rendered by default.

32 changes: 16 additions & 16 deletions .agents/model-matrix.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions .agents/quantization-matrix.md

Large diffs are not rendered by default.

12 changes: 2 additions & 10 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
| **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only |
| **DSR fix: async readback capability (2026-08-08)** | **No number owed**: behavior-neutral (CPU/CUDA async-ON, discrete non-CUDA async-OFF, unchanged); moves a `kCUDA` check onto `Backend`, unblocking red CI on #127/#154/#155 |
| **`ROAD-V1-MEM` M1+M2 (2026-08-08)** | KV auto-sizing CPU brick: `--kv-cache-memory` sizes the pool from a byte budget via the group-aware `KVBytesPerBlock` divisor (ABI v16, CPU-gated). M3 profile run dgx-gated |
| **Record/checker repair 2026-08-0708** | Restored red record/env gates; made release AST semantic pins Python 3.12/3.13-stable; recorded merged Gemma-4 MoE as known merged-GEMM drift and closed the stale embeddings claim. No runtime/performance change |
| **Record/checker repair 2026-08-07-08** | Restored red record/env gates; made release AST semantic pins Python 3.12/3.13-stable; recorded merged Gemma-4 MoE as known merged-GEMM drift and closed the stale embeddings claim. No runtime/performance change |
| **vLLM** | Qwen3.6-27B NVFP4, GB10 | ahead 4.5% at c1, **tie** at c2 to c32 | identical |
| **vLLM** | Qwen3.6-35B-A3B NVFP4, GB10 | 0.93x to 1.03x: ahead at c4, worst c16 0.93x | identical |
| **vLLM** | DeepSeek-V2-Lite (MLA), GB10 | 0.86x to 0.95x throughput, TTFT wins at c4/c8 | identical |
Expand Down Expand Up @@ -383,12 +383,4 @@ built on it rather than keeping the flattering one.

Build flags, environment variables, and the full gate list are in
[BUILD.md](BUILD.md) and [ENVIRONMENT.md](ENVIRONMENT.md).

## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700)

| Path | Warm tok/s | Notes |
|------|------------|--------|
| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill |
| server `/v1/completions` | ~38 | exclusive |
| server `/v1/chat` thinking off | ~32 | after expert cache |
| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack |
No throughput change. Matrix lifecycle bookkeeping only (2026-08-08).
3 changes: 3 additions & 0 deletions docs/ENVIRONMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ portable/reference path. In normal operation leave them unset.
| `VT_VULKAN_GEMV_UNROLL` | 4 | `=1` forces the un-unrolled decode GEMV body. Four independent accumulators keep four reads per lane in flight instead of one -- memory-level parallelism, not instruction count. It rides a specialization constant, so both arms are the same committed module and A/B in one binary. MEASURED **1.055x, 7 of 8 interleaved pairs**. Worth noting it measured 5/8 and was REVERTED earlier the same day: that test ran while the GPU was only 26% busy, where a 10% GEMV win moves e2e by 1.4% and is unresolvable against this box's noise. After the ring fix made the run GPU-bound the same code reads 7/8. A negative result is regime-dependent. Vulkan-only |
| `VT_VULKAN_RING` | 128 | Descriptor-set ring depth per pipeline, i.e. how many times one shader may be recorded before the batch must flush. This is what actually caps batch length: at 16 it forced a submit every 40-46 dispatches, because `vt_rms_norm` runs 112 times per forward pass and exhausted its ring seven times. At 128 a whole forward pass (368 dispatches) fits in ONE submit. Each avoided flush is a `vkQueueSubmit` plus a blocking `vkWaitForFences`, and a host profile puts 62% of on-CPU time in the kernel and NVIDIA driver against 14% in our own code -- so submits are the host cost. Lower it to bisect or to cut descriptor-pool size. Vulkan-only |
| `VT_SERVER_VERBOSE` | off | `=1` turns on the server's chatty progress output, and is the umbrella switch the individual knobs fall back to: `VT_SERVER_PREFILL_PROGRESS` reads it when unset, so `VT_SERVER_VERBOSE=1` alone enables chunked-prefill progress (rate-limited to ~2 Hz per request). `examples/server` sets it from its own verbosity flag, so it is user-facing rather than a kernel tuning switch |
| `VT_SERVER_MAX_PROMPT_CHARS` | lab default | Cap on templated chat prompt character length before reject. `0` disables. Protects the engine from accidental multi-100k Hermes SOUL payloads |
| `VT_SERVER_MAX_NEW_TOKENS` | lab default | Cap on max new tokens for a server request (clamps runaway client `max_tokens`). `0` disables |
| `VT_SERVER_PREFILL_PROGRESS` | follows `VT_SERVER_VERBOSE` | `=1` forces chunked-prefill progress logs even when verbose is off |
| `VT_VULKAN_BATCH` | on | `=0` forces the per-dispatch submit-and-wait path instead of recording many dispatches into ONE command buffer (with a pipeline barrier between them) and submitting once. MEASURED on GB10: decode **2.62x**, faster in **8 of 8** interleaved pairs (24.8 -> 64.5 tok/s), 40-46 dispatches per submit, opt-125m still 6/6 token-exact. Batching is sound only because all three host-read paths drain the batch first: `Backend::Copy` and `Memset` (host memcpy over the mapped allocation), `Synchronize`, and `Backend::FlushPending`, which op_provider.cpp calls before running a PORTABLE REFERENCE TIER kernel directly over device memory. Without that third one a host kernel would read stale bytes silently. It exists for the same-binary A/B and as the bisect lever. Vulkan-only |
| `VT_VULKAN_GEMV` | on | `=0` forces the decode GEMV onto the portable scalar GEMM instead of `vt_matmul_vec`. The vec tactic gives each output element a WORKGROUP whose lanes stride K, so adjacent lanes read adjacent addresses; the scalar kernel puts one invocation on each output element and loops K there, which for the `MatmulBT` weight layout leaves adjacent lanes `K*2` bytes apart, each pulling its own cache line for 2 bytes of it. Selected only for `MatmulBT` at M=1 with K at least one workgroup wide -- the other orientation is already coalesced and the vec shape would make it worse. MEASURED on GB10: GEMM `ms/call` improves in 7 of 8 interleaved pairs, median 1.8x. Its K reduction is a tree, so unlike the scalar kernel it does NOT share the CPU's accumulation order; the opt-125m STRICT gate (6/6 token-exact) is what clears it. Vulkan-only |
| `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only |
Expand Down
1 change: 1 addition & 0 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,3 +309,4 @@ holds for the 31 architectures inventoried on 2026-08-05. A row's lifecycle stat
are independent: see [STATUS.md](STATUS.md). Parakeet ASR (encoder + CTC/RNN-T/TDT) runs natively on CPU, 4 checkpoints token-exact vs HF.

| Gemma4 MoE ROCm fused helpers (`vt::fused_ops`) | partial | Portable seam; ROCm fast path; CPU/Vulkan link |
| Live-row audit demotion (abandoned ACTIVE->SPIKE) | done | Matrices only; unblocks agent-record |
Loading
Loading