From 33e936086a116c840518afcf732076c9917e96a5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 5 Aug 2026 23:38:51 +0000 Subject: [PATCH 1/2] bench(mxfp4): online-serving q3mxfp4 key + smoke gate + grid orchestrator (QUANT-CT-MXFP4-BENCH) Plumb the online-serving harness for the MXFP4 W4A16 keep-quant throughput grid on Yi30/Qwen3-8B-MXFP4. The compute is correctness-DONE on main (#38 + the #44 async-dense-mirror fix: default-config e2e 3/4 token-exact + p2 near-tie); the only owed piece was the binding ours-vs-oracle grid, which online_gate.py could not run because it carried only the "27"/"35" NVFP4 gate-model keys. Additive; existing 27/35 paths byte-unchanged; harness CPU contract tests 45/45. - online_gate.py: q3mxfp4 MODEL_REVISIONS/REPOSITORIES (b3e7ab32), MAX_NUM_BATCHED_TOKENS=2048, MAX_MODEL_LEN=40960, and a POINTS_BY_MODEL reduced c1/c2/c4/c8 sweep surfaced via points_for(model_key) (a strict prefix of POINTS, so prompts_for is unchanged). online_gate_summary threads points_for through its four per-model POINTS loops so a c1-c8 key never flags a missing c16/c32 group. - mxfp4_smoke_gate.py (new): the q3mxfp4 model gate = the #44 e2e smoke battery (vllm-cli greedy vs golden_marlin_w4a16.json; 3 deterministic prompts exact + story coherent). Validated locally against a fake vllm-cli (PASS + two FAILs). - dgx-online-serving.sh: q3mxfp4 branches (validation, dense 2048 batched-tokens, vllm-cli build + smoke model gate, oracle arm VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel with no mamba flag, 1 2 4 8 leg loop). The mamba-pin shell-parse test retargeted to the 27/35 block and strengthened to assert the q3mxfp4 arm's disabled-kernel + no-mamba. - mxfp4-online-serving-grid.sh (new): one-command orchestrator (corpus gen + dry-run manifest + locked --execute grid + summary). No throughput numbers: the grid needs GB10 exclusively (fresh /dev/shm CUDA build, oracle serve, 2 arms x 3 reps x c1-c8 sequential legs, both flock locks, free-g >= 90). Grid GATING on dgx; run scripts/mxfp4-online-serving-grid.sh. Records (STATUS/BENCHMARKS/FEATURES + quantization-matrix + benchmark-record + NOW + state) updated in the same change. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 3 +- .agents/benchmark-record.md | 39 +++++ .agents/quantization-matrix.md | 2 +- .agents/state.md | 59 +++++++ docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 4 +- scripts/dgx-online-serving.sh | 66 +++++++- scripts/mxfp4-online-serving-grid.sh | 116 ++++++++++++++ tests/tools/test_online_gate_client.py | 87 ++++++++++- tools/bench/mxfp4_smoke_gate.py | 203 +++++++++++++++++++++++++ tools/bench/online_gate.py | 31 +++- tools/bench/online_gate_summary.py | 9 +- 13 files changed, 598 insertions(+), 25 deletions(-) create mode 100755 scripts/mxfp4-online-serving-grid.sh create mode 100644 tools/bench/mxfp4_smoke_gate.py diff --git a/.agents/NOW.md b/.agents/NOW.md index 9499b5223..0ac47bbb1 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,8 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13, f32-loader block CLEARED): CPU+CUDA 13/13·656; host RSS peak 1.7 GiB, min-avail 21 GiB, no OOM. **Token gate NEAR-TIE 106/128** (6/8 prompts token-exact; numerics near-tie vs deterministic oracle, not a bug) | STRICT path = device GDN/MLA islands + bf16 stream (W7-speed residuals); 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED + dgx-VERIFIED** (`f9c969ae`): #31 async mirror ported to classic dense `Qwen3ForCausalLM`. Async gate RED→GREEN 0.6B+4B, SACRED 184/184, memcheck 0, MXFP4 default e2e 3/4 + near-tie RATIFIED | Residual: **W4 throughput bench** (online_gate lacks Yi30/8B key); sibling scope one-liner | +| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | +| QUANT-CT-MXFP4-BENCH | **PLUMBED** (`row/QUANT-CT-MXFP4-BENCH`): `q3mxfp4` key + smoke gate + oracle disabled-kernel; CPU tests 45/45 | Grid GATING dgx: `mxfp4-online-serving-grid.sh` c1-c8x3 | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index aabee97e7..eb14c3375 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,45 @@ from relative link targets repointed for this file's location. # Benchmarks +## QUANT-CT-MXFP4 W4 throughput bench — harness PLUMBED, grid GATING on dgx (2026-08-07, `row/QUANT-CT-MXFP4-BENCH`) — NO numbers yet + +The MXFP4 W4A16 keep-quant compute is correctness-DONE on main (#38 + the +`ROW-SERVE-ASYNC-DENSE-MIRROR` #44 async fix: default-config e2e 3/4 token-exact + +p2 near-tie ratified). The one owed piece is the binding ours-vs-oracle throughput +grid on the SAME checkpoint (`Yi30/Qwen3-8B-MXFP4`, a dense `Qwen3ForCausalLM`). +The online-serving harness only carried the "27"/"35" NVFP4 gate-model keys; this +row adds the `q3mxfp4` key so the grid can run. NO throughput number is produced +or owed by THIS entry; it records the plumbing and the exact one-command recipe so +the grid re-runs deterministically once the dgx build+legs execute. + +**Landed (additive; 27/35 byte-unchanged, harness CPU contract tests 45/45 GREEN):** +- `tools/bench/online_gate.py`: `MODEL_REVISIONS`/`MODEL_REPOSITORIES` + (`Yi30/Qwen3-8B-MXFP4` @ `b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47`), + `MAX_NUM_BATCHED_TOKENS`=2048, `MAX_MODEL_LEN`=40960, and a `POINTS_BY_MODEL` + reduced sweep (c1/c2/c4/c8, a strict prefix of `POINTS`) surfaced through a new + `points_for(model_key)` helper that both the harness and `online_gate_summary` + consult (so a c1-c8 key never trips a "missing result group" for c16/c32). +- `tools/bench/mxfp4_smoke_gate.py`: the q3mxfp4 model-gate = the #44 e2e smoke + battery (vllm-cli greedy, temp 0, seed 0, vs `golden_marlin_w4a16.json`); + requires the 3 deterministic prompts token-exact + the story prompt coherent. +- `scripts/dgx-online-serving.sh`: q3mxfp4 branches — model validation, dense 2048 + batched-tokens, gate builds `vllm-cli`+`server` and runs the smoke gate instead + of a paged-engine ctest, oracle arm `env VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` + (Marlin W4A16; sm_121 cute-dsl mxf4 crash worked around) with no mamba flag, and + the low-concurrency `1 2 4 8` leg loop. +- `scripts/mxfp4-online-serving-grid.sh`: the one-command orchestrator (source-corpus + gen + dry-run manifest + locked `--execute` grid + `--model q3mxfp4` summary). + +**Repro (one command, on dgx, both flock locks free, free -g ≥ 90, ninja+nvcc on PATH):** +`scripts/mxfp4-online-serving-grid.sh --snapshot ~/.cache/huggingface/hub/models--Yi30--Qwen3-8B-MXFP4/snapshots/b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47 --build-dir --configure-log ` +(build recipe = git-archive → /dev/shm, `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON +-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0`, +Release, nvcc 13.0; build the `server` + `vllm-cli` targets only, never bare +`ninja`, to respect the disk floor). Verdict rule: at/above vLLM on total+output +tok/s AND at/below on TTFT+TPOT across c1/c2/c4/c8 → MXFP4 row DONE; below on any +cell → record the failing cells + a same-tool nsys attribution pass (decode-window +separated) before any lever campaign. + ## DeepSeek-V4-Flash UD-IQ2_M — IQ2_S + MXFP4 CPU keep-quant bring-up (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`) — no throughput owed (off-GPU correctness bring-up) Off-GPU task (GB10 down, no nvcc on the dev box), so NO throughput is measured or owed. Adds the two per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint's last 4 routed-expert slabs use — **IQ2_S** (ggml type 22; 2.5625 bpw codebook, Q8_K activation) and **MXFP4** (type 39; OCP micro-scaling fp4, 32-elem blocks, Q8_0 activation) — as first-class vt block dtypes (`kIQ2_S`/`kMXFP4`): block traits + dequant + a keep-quant `vec_dot`, ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook copied verbatim + direct sign bytes; `kvalues_mxfp4` + `e8m0_to_fp32_half`). The memory point: these load COMPRESSED (keep-quant) instead of the ~17 GiB bf16 expansion that OOM-reboots the 119 GiB pool. diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index abb359661..a8779283f 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -127,7 +127,7 @@ Registry source: | `QUANT-FP8-PCPT` | ModelOpt FP8 per-channel/per-token | W8/A8 | capability selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MXFP8-MODELOPT` | ModelOpt MXFP8 | W8/A8 | CUDA/ROCm/XPU dispatch | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MIXED-MODELOPT` | ModelOpt mixed precision | FP8/NVFP4/MXFP8 groups | per-layer | part | part | part | Y | Y | `PARTIAL` | 35B FP8+NVFP4 slice: [loader](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L118), [FP8 tests](../tests/vt/test_ops_fp8_cutlass.cpp#L188), [NVFP4 tests](../tests/vt/test_ops_moe_grouped.cpp#L453), [gate](../tests/parity/test_qwen36_paged_engine.cpp#L78) | leaf spec open | - | -| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. RESIDUAL: the W4 THROUGHPUT bench (c1..c8x3 vs oracle) still owed — online_gate lacks a Yi30/8B model key (needs corpus + oracle-record plumbing). Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | +| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. RESIDUAL: the W4 THROUGHPUT bench (c1..c8x3 vs oracle) harness is now PLUMBED (`row/QUANT-CT-MXFP4-BENCH`): `online_gate` `q3mxfp4` key (`MODEL_REVISIONS`/`REPOSITORIES` `Yi30/Qwen3-8B-MXFP4`@`b3e7ab32`, reduced c1/c2/c4/c8 `points_for` set, `MAX_NUM_BATCHED_TOKENS`/`MAX_MODEL_LEN`), a vllm-cli smoke model gate ([`mxfp4_smoke_gate.py`](../tools/bench/mxfp4_smoke_gate.py)) reproducing the #44 3/4+near-tie battery, `dgx-online-serving.sh` q3mxfp4 branches (oracle arm `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, no mamba flag) and a one-command orchestrator [`mxfp4-online-serving-grid.sh`](../scripts/mxfp4-online-serving-grid.sh); harness CPU contract tests GREEN (45/45). The GRID itself is GATING on dgx execution (fresh /dev/shm CUDA build + locked single-load legs). Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | | `QUANT-CT-W4A8-FP8` | compressed-tensors W4A8 FP8 | W4/A8 | CUTLASS | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W4A8-INT8` | compressed-tensors W4A8 INT8 | W4/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W8A8-FP8` | compressed-tensors W8A8 FP8 | W8/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | diff --git a/.agents/state.md b/.agents/state.md index 1071107fa..f2e2e6495 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -36678,3 +36678,62 @@ The fix UNBLOCKS the default-config number (the async-default degeneration no lo forces `VT_ASYNC_SCHED=0`); the oracle is proven to run the model today. Plus the sibling scope one-liner (InternLM2/Mistral/Llama). dgx build tree persists at `dgx:/dev/shm/serve-async-dense`. + +## QUANT-CT-MXFP4-BENCH: online-serving harness PLUMBED for the MXFP4 W4 throughput grid (q3mxfp4 key); grid GATING on dgx execution + + +Closed the harness half of the QUANT-CT-MXFP4 W4 bench residual (the one named by +`ROW-SERVE-ASYNC-DENSE-MIRROR` 2026-08-07T07:00 and NOW.md): `online_gate.py` +carried only the "27"/"35" NVFP4 gate-model keys, so the MXFP4 checkpoint could not +be benched ours-vs-oracle. Branch `row/QUANT-CT-MXFP4-BENCH` off `origin/main` +`52d76f3a`. No GPU held, no locks touched (all work + validation on the dev box). + +LANDED (additive; existing 27/35 paths byte-unchanged; harness CPU contract tests +GREEN 45/45 via `python3 -m unittest tests.tools.test_online_gate_{client,summary,trace}`): +- `tools/bench/online_gate.py`: `MODEL_REVISIONS`/`MODEL_REPOSITORIES` q3mxfp4 = + `Yi30/Qwen3-8B-MXFP4` @ snapshot `b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47` (dense + `Qwen3ForCausalLM`, 36L, max_pos 40960, mxfp4-pack-quantized group-32 E8M0); + `MAX_NUM_BATCHED_TOKENS`=2048 (dense, mirrors 27B), `MAX_MODEL_LEN`=40960; + `POINTS_BY_MODEL={"q3mxfp4": c1/c2/c4/c8}` + a `points_for(model_key)` helper (the + reduced set is a strict PREFIX of POINTS so `prompts_for` is unchanged) consulted + by `prepare_corpus_views` and by `online_gate_summary` (the four per-model POINTS + loops) so a c1-c8 key never flags a missing c16/c32 result group. +- `tools/bench/mxfp4_smoke_gate.py` (NEW): the q3mxfp4 model gate = the #44 e2e smoke + battery. Runs `vllm-cli` greedy (temp 0, seed 0, max_tokens = each golden's token + count) on the 4 prompts of `docs/bench-evidence/mxfp4-qwen/golden_marlin_w4a16.json`; + PASS = the 3 deterministic prompts (capitals/arithmetic/fibonacci) token-exact vs + golden text AND the story prompt coherent (not the async-race degeneration). Locally + validated with a fake vllm-cli across PASS + deterministic-mismatch FAIL + degenerate + FAIL. This is a text-level gate (vllm-cli emits text, not token-ids); the strong + op-level correctness lives in the #38 unit gates. +- `scripts/dgx-online-serving.sh`: q3mxfp4 branches — model validation (27/35/q3mxfp4), + dense 2048 batched-tokens, `gate_target=vllm-cli` + smoke-gate model gate (no + paged-engine ctest; the 8B is near-tie so a committed npy golden is ill-posed, same + reason the async-dense gate uses sync==async), oracle server arm + `env VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel "${client}" serve` with NO + `--mamba-ssm-cache-dtype` (dense, no mamba), and the `1 2 4 8` leg loop. bash -n + + shellcheck -S error clean. The existing shell-parse test that pins the mamba flag on + the vLLM arm was updated to target the 27/35 block and STRENGTHENED to also assert the + q3mxfp4 arm sets the disabled-kernel + omits mamba. +- `scripts/mxfp4-online-serving-grid.sh` (NEW): the one-command orchestrator (source + corpus gen via `make_serve_low_corpus --concurrencies 1,2,4,8` + dry-run manifest + + locked `--execute --model q3mxfp4` grid + summary). Thin wrapper; all locks/gating/ + single-load/drop_caches/oracle-record live in the tested `dgx-online-serving.sh`. +- New harness tests: `test_q3mxfp4_bench_key_is_registered_with_a_reduced_low_concurrency_sweep` + + `test_prepare_corpus_for_q3mxfp4_only_materializes_its_scoped_points`. + +WHY NO NUMBERS: the grid needs GB10 exclusively (fresh /dev/shm CUDA build of +`server`+`vllm-cli`, oracle serve, 2 arms x 3 reps x c1-c8 sequential legs, both +flock locks, free-g ≥ 90, drop_caches between legs) — this session did the plumbing ++ validation on the dev box only. The plumbing UNBLOCKS the grid; run +`scripts/mxfp4-online-serving-grid.sh --snapshot --build-dir +--configure-log ` on dgx. Build recipe: git-archive → /dev/shm, +`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a +-DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0`, Release, nvcc 13.0, specific targets +only (never bare `ninja` — disk floor). VERDICT rule per match-or-beat: ≥ vLLM on +total+output tok/s AND ≤ on TTFT+TPOT across all four cells → MXFP4 row DONE; below on +any cell → record the failing cells + a same-tool decode-window nsys attribution pass +(no ceiling declared). Box state at hand-off: idle, 115 GiB avail, GPU free, disk 26 GiB +(tight — the /dev/shm build sidesteps the /home floor). Evidence: this row's diff + +`docs/bench-evidence/mxfp4-qwen/` golden. Records: STATUS/BENCHMARKS/FEATURES + +quantization-matrix + benchmark-record + NOW updated in the same change. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index dfab58265..b70f5e525 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED (oracle teacher-forced: max gap 0.0000 nats). W4 THROUGHPUT bench still owed: online_gate lacks a Yi30/8B key (see benchmark-record) | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED (max gap 0.0000 nats). W4 throughput bench harness PLUMBED (`row/QUANT-CT-MXFP4-BENCH`: `q3mxfp4` key, smoke model gate, one-command grid); grid GATING on dgx (recipe + detail in benchmark-record) | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5262a1061..7758a5406 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -65,7 +65,7 @@ are our reading of their documented behavior, not measurements. | GGUF k-quants and i-quants | ✅ | ☐ | ☐ | ✅ | | AWQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | -| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; default-config e2e coherent + 3/4 token-exact (async-mirror fix); throughput bench owed | ✅ | ✅ | ☐ | +| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; default-config e2e coherent + 3/4 token-exact (async-mirror fix); throughput bench harness plumbed (q3mxfp4 key), grid gating on dgx | ✅ | ✅ | ☐ | | fp8 weights | ✅ | ✅ | ✅ | ☐ | | bf16 / fp16 | ✅ | ✅ | ✅ | ✅ | | Safetensors direct load, no conversion | ✅ | ✅ | ✅ | ☐ | diff --git a/docs/STATUS.md b/docs/STATUS.md index dbcd58ea3..060ebedc0 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -53,7 +53,7 @@ token-for-token correctness against the pinned oracle. |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue | -| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 throughput bench (online_gate harness gap). Detail in state.md. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | +| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 throughput bench (online_gate harness now PLUMBED on `row/QUANT-CT-MXFP4-BENCH`, grid GATING on dgx). Detail in state.md. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | | Llama-3.x dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 16/16 (Llama-3.2-1B); llama3 RoPE scaling | @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); bench owed | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput bench harness PLUMBED (`row/QUANT-CT-MXFP4-BENCH`, `q3mxfp4` key); grid GATING on dgx | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | diff --git a/scripts/dgx-online-serving.sh b/scripts/dgx-online-serving.sh index 6649ff7bc..89f43d19d 100755 --- a/scripts/dgx-online-serving.sh +++ b/scripts/dgx-online-serving.sh @@ -98,11 +98,14 @@ if [[ ${mode} == dry-run ]]; then exit 0 fi -[[ ${model} == 27 || ${model} == 35 ]] || { echo "--model must be 27 or 35" >&2; exit 2; } -if [[ ${model} == 27 ]]; then - max_num_batched_tokens=2048 -else +[[ ${model} == 27 || ${model} == 35 || ${model} == q3mxfp4 ]] || { + echo "--model must be 27, 35 or q3mxfp4" >&2; exit 2; } +# 35B MoE prefills a wider chunk; the 27B NVFP4 dense and the q3mxfp4 MXFP4 dense +# 8B both use the dense 2048 batched-token gate value. +if [[ ${model} == 35 ]]; then max_num_batched_tokens=8192 +else + max_num_batched_tokens=2048 fi [[ -n ${evidence} ]] || { echo "--evidence is required" >&2; exit 2; } [[ -n ${source_corpus} ]] || { echo "--source-corpus is required" >&2; exit 2; } @@ -223,8 +226,17 @@ else fi if [[ ${model} == 27 ]]; then test_name=test_qwen27_paged_engine -else + gate_target=${test_name} +elif [[ ${model} == 35 ]]; then test_name=test_qwen36_paged_engine + gate_target=${test_name} +else + # q3mxfp4 (MXFP4 W4A16 keep-quant dense 8B) has no committed npy near-tie + # paged-engine golden; its correctness precondition is the #44 MXFP4 e2e smoke + # battery (vllm-cli greedy vs docs/bench-evidence/mxfp4-qwen/golden_marlin_w4a16.json), + # so the model gate builds + runs vllm-cli instead of a ctest binary. + test_name=mxfp4_smoke_battery + gate_target=vllm-cli fi cmake_home=$(sed -n 's/^CMAKE_HOME_DIRECTORY:INTERNAL=//p' "${build_dir}/CMakeCache.txt") [[ -n ${cmake_home} && $(realpath -e "${cmake_home}") == "$(realpath -e "${repo_root}")" ]] || { @@ -248,7 +260,7 @@ build_log="${execution_dir}/${model}-build.log" build_jobs=$(nproc) build_cmd=( cmake --build "${build_dir}" - --target server "${test_name}" + --target server "${gate_target}" --parallel "${build_jobs}" ) printf '%q ' "${build_cmd[@]}" >"${build_command}" @@ -261,6 +273,10 @@ fi echo "provenance-recorded build did not produce examples/server" >&2 exit 1 } +[[ ${model} != q3mxfp4 || -x ${build_dir}/examples/vllm-cli ]] || { + echo "provenance-recorded build did not produce examples/vllm-cli" >&2 + exit 1 +} # Timed grids record the production (profile-control-OFF) build; the H1d # trace campaign records the instrumented ON build. The grid summary # requires profile_control=false for timed evidence. @@ -384,6 +400,22 @@ start_server() { --no-enable-prefix-caching --served-model-name gate ) + elif [[ ${model} == q3mxfp4 ]]; then + # MXFP4 dense 8B oracle. On sm_121 GB10 the default FlashInfer cute-dsl mxf4 + # backend aborts engine start (BackendSupportedError mm_fp4 cap 121), so we + # force the Marlin W4A16 keep-quant kernel our arm mirrors. Dense Qwen3 has no + # mamba SSM state, so the hybrid-only SSM cache-dtype flag is omitted here. + server_cmd=( + env "PATH=$(dirname "${client}"):${PATH}" + "VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel" + "${client}" serve "${snapshot}" + --served-model-name gate + --gpu-memory-utilization 0.6 + --max-num-seqs "${max_num_seqs}" + --max-num-batched-tokens "${max_num_batched_tokens}" + --no-enable-prefix-caching + --port "${port}" + ) else server_cmd=( env "PATH=$(dirname "${client}"):${PATH}" @@ -436,8 +468,13 @@ run_leg() { --minimum-spread 0.05 \ --output "${preflight_dir}/r${repetition}-stream.json" + # q3mxfp4 (MXFP4 W4 row) is scoped to the low-concurrency decode regime; the + # NVFP4 gate models sweep the full six points. Kept in lockstep with + # online_gate.points_for so the summary never flags a missing result group. + local concurrency_points="1 2 4 8 16 32" + [[ ${model} == q3mxfp4 ]] && concurrency_points="1 2 4 8" local concurrency - for concurrency in 1 2 4 8 16 32; do + for concurrency in ${concurrency_points}; do kill -0 "${spid}" 2>/dev/null || { echo "server died before c${concurrency}" >&2 return 1 @@ -912,7 +949,20 @@ mkdir -p "${gate_dir}" echo "refusing to overwrite model-gate evidence for ${model}" >&2 exit 1 } -if ! "${benchmark_clean_env[@]}" "${h1d_plan_env[@]}" \ +if [[ ${model} == q3mxfp4 ]]; then + # MXFP4 correctness precondition = the #44 e2e smoke battery (default config, + # async ON + the classic-dense device-mirror fix). vllm-cli links vllm::shared + # (build-tree RPATH) and needs the operator CUDA environment, so it runs in the + # script shell rather than the ultra-clean measurement env used for timed legs. + if ! python3 "${repo_root}/tools/bench/mxfp4_smoke_gate.py" \ + --vllm-cli "${build_dir}/examples/vllm-cli" \ + --snapshot "${snapshot}" \ + --golden "${repo_root}/docs/bench-evidence/mxfp4-qwen/golden_marlin_w4a16.json" \ + >"${gate_log}" 2>&1; then + cat "${gate_log}" >&2 + exit 1 + fi +elif ! "${benchmark_clean_env[@]}" "${h1d_plan_env[@]}" \ ctest --test-dir "${build_dir}" -R "^${test_name}$" --output-on-failure \ >"${gate_log}" 2>&1; then cat "${gate_log}" >&2 diff --git a/scripts/mxfp4-online-serving-grid.sh b/scripts/mxfp4-online-serving-grid.sh new file mode 100755 index 000000000..e61ac8b1f --- /dev/null +++ b/scripts/mxfp4-online-serving-grid.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# One-command MXFP4 W4 throughput grid (row QUANT-CT-MXFP4-BENCH). +# +# Reproducibility wrapper around the fingerprinted online-serving harness for the +# "q3mxfp4" key (Yi30/Qwen3-8B-MXFP4, native Marlin W4A16 MXFP4 keep-quant). It +# adds only the two steps dgx-online-serving.sh does not own for a fresh key -- +# the deterministic source-corpus generation and the dry-run manifest -- then +# delegates the locked, single-load, drop_caches grid to that tested driver and +# prints the per-axis summary. All GPU serialization, memory gating, oracle +# recording (VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel), the #44 smoke +# model gate and the c1/c2/c4/c8 x3 sequential legs live in dgx-online-serving.sh +# / online_gate.py; this script is intentionally thin so its own logic is trivial. +# +# The oracle arm needs ninja + nvcc on PATH (the DGX non-interactive quirk); run +# under a shell that has them, both flock locks free, free -g >= 90, and the +# production (profile-control-OFF) CUDA build already configured. +set -euo pipefail + +usage() { + cat >&2 <<'EOF' +usage: + mxfp4-online-serving-grid.sh --snapshot DIR --build-dir DIR --configure-log FILE + [--client PATH] [--claim-root DIR] [--vllm-cpp-sha SHA] [--port N] + + --snapshot HF snapshot dir; its basename MUST be the pinned revision + b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47. + --build-dir configured production CMake build tree (server + vllm-cli). + --configure-log the non-empty log from that build configuration. + --client pinned vLLM oracle `vllm` (default ~/venvs/vllm-oracle/bin/vllm). + --claim-root evidence root (default ~/work/vllm.cpp-online-gate). + --vllm-cpp-sha defaults to the worktree HEAD. + --port server port (default 8001). +EOF +} + +model=q3mxfp4 +revision=b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47 +snapshot="" +build_dir="" +configure_log="" +client="${HOME}/venvs/vllm-oracle/bin/vllm" +claim_root="${HOME}/work/vllm.cpp-online-gate" +vllm_cpp_sha="" +port=8001 + +while (($#)); do + case "$1" in + --snapshot) snapshot=${2:?}; shift 2 ;; + --build-dir) build_dir=${2:?}; shift 2 ;; + --configure-log) configure_log=${2:?}; shift 2 ;; + --client) client=${2:?}; shift 2 ;; + --claim-root) claim_root=${2:?}; shift 2 ;; + --vllm-cpp-sha) vllm_cpp_sha=${2:?}; shift 2 ;; + --port) port=${2:?}; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown argument: $1" >&2; usage; exit 2 ;; + esac +done + +repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +export PYTHONPATH="${repo_root}${PYTHONPATH:+:${PYTHONPATH}}" +driver="${repo_root}/scripts/dgx-online-serving.sh" + +[[ -n ${snapshot} && -d ${snapshot} ]] || { echo "--snapshot directory is required" >&2; exit 2; } +[[ $(basename "${snapshot}") == "${revision}" ]] || { + echo "--snapshot basename must be the pinned revision ${revision}" >&2; exit 2; } +[[ -n ${build_dir} && -f ${build_dir}/CMakeCache.txt ]] || { + echo "--build-dir must name a configured CMake build tree" >&2; exit 2; } +[[ -n ${configure_log} && -s ${configure_log} ]] || { + echo "--configure-log must name the non-empty configuration log" >&2; exit 2; } +[[ -x ${client} ]] || { echo "pinned vLLM client is not executable: ${client}" >&2; exit 2; } + +if [[ -z ${vllm_cpp_sha} ]]; then + vllm_cpp_sha=$(git -C "${repo_root}" rev-parse HEAD) +fi +evidence="${claim_root}/evidence/${vllm_cpp_sha}" +source_corpus="${evidence}/corpus/${model}" +oracle_python="$(dirname "${client}")/python" +[[ -x ${oracle_python} ]] || { echo "vLLM oracle Python is absent: ${oracle_python}" >&2; exit 2; } + +# 1. Deterministic source corpus (exact 1024-token prompts) with the checkpoint's +# own tokenizer, scoped to the c1/c2/c4/c8 low-concurrency sweep. Idempotent. +if [[ ! -f ${source_corpus}/manifest.json ]]; then + mkdir -p "${source_corpus}" + "${oracle_python}" -m tools.bench.make_serve_low_corpus \ + --tokenizer-json "${snapshot}/tokenizer.json" \ + --tokenizer-revision "${revision}" \ + --model-key "${model}" \ + --out "${source_corpus}" \ + --concurrencies 1,2,4,8 \ + --repetitions 3 +fi + +# 2. Dry-run manifest (writes ${evidence}/manifest.json that --execute validates). +if [[ ! -f ${evidence}/manifest.json ]]; then + "${driver}" --dry-run \ + --claim-root "${claim_root}" \ + --client "${client}" \ + --vllm-cpp-sha "${vllm_cpp_sha}" +fi + +# 3. The locked, single-load, drop_caches c1/c2/c4/c8 x3 grid (both arms). The +# driver runs the #44 smoke model gate, records the oracle, and emits the +# per-axis summary (online_gate_summary --model q3mxfp4) as its final step. +"${driver}" --execute \ + --model "${model}" \ + --snapshot "${snapshot}" \ + --source-corpus "${source_corpus}" \ + --evidence "${evidence}" \ + --build-dir "${build_dir}" \ + --configure-log "${configure_log}" \ + --client "${client}" \ + --vllm-cpp-sha "${vllm_cpp_sha}" \ + --port "${port}" + +echo "MXFP4 grid complete. Summary: ${evidence}/summary-${model}/report.md" >&2 diff --git a/tests/tools/test_online_gate_client.py b/tests/tools/test_online_gate_client.py index 5b8ebadc4..c589bf013 100644 --- a/tests/tools/test_online_gate_client.py +++ b/tests/tools/test_online_gate_client.py @@ -29,6 +29,8 @@ MAX_NUM_BATCHED_TOKENS, MAX_MODEL_LEN, MAX_NUM_SEQS, + MODEL_REPOSITORIES, + MODEL_REVISIONS, FLASHINFER_VERSION, NSYS_CAPTURE_RANGE, NSYS_CAPTURE_RANGE_END, @@ -37,6 +39,8 @@ NSYS_PRODUCT_VERSION, OUTPUT_LEN, PANDAS_VERSION, + POINTS, + POINTS_BY_MODEL, TRACE_CONCURRENCY, TRACE_CAPTURE_GRAPH_REPLAYS, TRACE_GDN_BA_ENV, @@ -59,7 +63,9 @@ OnlineRun, build_client_command, build_plan, + points_for, prepare_corpus_views, + prompts_for, record_execution_manifest, record_memory_return, record_model_gate, @@ -746,6 +752,69 @@ def test_prepare_corpus_preserves_exact_prompts_and_refuses_duplicates(self) -> ): prepare_corpus_views(source, output, model_key="27") + def test_q3mxfp4_bench_key_is_registered_with_a_reduced_low_concurrency_sweep( + self, + ) -> None: + # The MXFP4 W4 throughput row's additive model key. The NVFP4 gate keys + # keep the full six-point sweep; q3mxfp4 benches only the low-concurrency + # decode regime (c1/c2/c4/c8), a strict prefix of POINTS so the + # concurrency -> num_prompts mapping is unchanged. + self.assertEqual( + MODEL_REVISIONS["q3mxfp4"], + "b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47", + ) + self.assertEqual(MODEL_REPOSITORIES["q3mxfp4"], "Yi30/Qwen3-8B-MXFP4") + self.assertEqual(MAX_NUM_BATCHED_TOKENS["q3mxfp4"], 2048) + self.assertEqual(MAX_MODEL_LEN["q3mxfp4"], 40960) + self.assertEqual(points_for("q3mxfp4"), ((1, 6), (2, 6), (4, 12), (8, 24))) + # A prefix of POINTS, so prompts_for stays the single source of counts. + self.assertEqual(points_for("q3mxfp4"), POINTS[: len(points_for("q3mxfp4"))]) + for concurrency, num_prompts in points_for("q3mxfp4"): + self.assertEqual(prompts_for(concurrency), num_prompts) + # The NVFP4 gate keys are untouched: they fall through to the full POINTS. + self.assertNotIn("27", POINTS_BY_MODEL) + self.assertNotIn("35", POINTS_BY_MODEL) + self.assertEqual(points_for("27"), POINTS) + self.assertEqual(points_for("35"), POINTS) + + def test_prepare_corpus_for_q3mxfp4_only_materializes_its_scoped_points( + self, + ) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = pathlib.Path(temporary) + source = root / "source" + source.mkdir() + (source / "manifest.json").write_text("{}\n", encoding="utf-8") + rows = [ + { + "conversations": [{"from": "human", "value": f"prompt-{index}"}], + "index": index, + "output_len": OUTPUT_LEN, + "prompt_sha256": f"{index + 1:064x}", + "prompt_token_ids": [index] * INPUT_LEN, + } + for index in range(2) + ] + (source / "c1-r1.jsonl").write_text( + "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8" + ) + output = root / "view" + # Scope q3mxfp4 to a single tiny point so the additive key routes + # through points_for without needing the full c1..c8 corpus fixture; + # a c16/c32 source is never demanded for this key. + with ( + mock.patch( + "tools.bench.online_gate.POINTS_BY_MODEL", + {"q3mxfp4": ((1, 2),)}, + ), + mock.patch("tools.bench.online_gate.REPETITIONS", (1,)), + ): + manifest = prepare_corpus_views(source, output, model_key="q3mxfp4") + self.assertEqual(manifest["model_key"], "q3mxfp4") + self.assertEqual(manifest["tokenizer_revision"], MODEL_REVISIONS["q3mxfp4"]) + self.assertEqual([item["concurrency"] for item in manifest["files"]], [1]) + self.assertTrue((output / "c1-r1.jsonl").is_file()) + def test_plan_has_one_lock_per_model_and_interleaves_arms(self) -> None: plan = build_plan( claim_root=pathlib.Path("/claim"), @@ -1387,12 +1456,22 @@ def test_vllm_arm_server_pins_mamba_ssm_cache_dtype_float32(self) -> None: )[0] ours_block = start_server.split( "if [[ ${engine} == ours ]]; then", 1 - )[1].split("\n else", 1)[0] - vllm_block = start_server.split( - '"${client}" serve "${snapshot}"', 1 - )[1].split("\n )", 1)[0] + )[1].split("\n elif", 1)[0] + # The additive q3mxfp4 (MXFP4 dense 8B) vLLM arm forces Marlin W4A16 via + # VLLM_DISABLED_KERNELS and, being a classic dense checkpoint, carries no + # mamba state; the 27/35 hybrid arm still pins the SSM cache dtype. + after_q3mxfp4 = start_server.split( + "elif [[ ${model} == q3mxfp4 ]]; then", 1 + )[1] + mxfp4_block = after_q3mxfp4.split("\n else", 1)[0] + vllm_block = after_q3mxfp4.split("\n else", 1)[1].split("\n )", 1)[0] self.assertIn("--mamba-ssm-cache-dtype float32", vllm_block) + self.assertNotIn("VLLM_DISABLED_KERNELS", vllm_block) self.assertNotIn("--mamba-ssm-cache-dtype", ours_block) + self.assertNotIn("--mamba-ssm-cache-dtype", mxfp4_block) + self.assertIn( + "VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel", mxfp4_block + ) def test_trace_driver_sets_h1d_plan_environment_before_manifest(self) -> None: repo = pathlib.Path(__file__).resolve().parents[2] diff --git a/tools/bench/mxfp4_smoke_gate.py b/tools/bench/mxfp4_smoke_gate.py new file mode 100644 index 000000000..f2846627e --- /dev/null +++ b/tools/bench/mxfp4_smoke_gate.py @@ -0,0 +1,203 @@ +#!/usr/bin/env python3 +"""MXFP4 e2e smoke battery — the q3mxfp4 online-serving model gate. + +This is the correctness precondition the online-serving grid runs for the +``q3mxfp4`` key (``Yi30/Qwen3-8B-MXFP4``, a classic dense ``Qwen3ForCausalLM`` on +the native Marlin W4A16 MXFP4 keep-quant path). The NVFP4 gate models ("27"/"35") +use a committed npy near-tie paged-engine ctest; the MXFP4 8B has no such golden, +so its gate reproduces the #44 e2e battery instead: greedy (temperature 0, seed 0) +completion of four fixed prompts via ``vllm-cli``, compared to the oracle golden +``docs/bench-evidence/mxfp4-qwen/golden_marlin_w4a16.json`` (captured from the +pinned 0.25.0 oracle with ``VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`` → +Marlin W4A16). + +The battery encodes exactly the #44 verdict: the three DETERMINISTIC prompts +(capitals, arithmetic, fibonacci) must be TOKEN-EXACT vs the golden text, and the +one open-ended NEAR-TIE prompt (the story) must be COHERENT (not the async-race +degeneration " Paris ... I I I ... !!!!!" that the pre-#44 classic-dense path +produced). A deterministic-prompt mismatch is a real regression and fails the +gate; a degenerate story is the #31/#44 async-mirror regression and fails the gate. + +The gate runs the DEFAULT engine config (async scheduling ON + the classic-dense +device-mirror fix from #44) so a PASS proves the benched binary reproduces #44's +state before any throughput number is trusted. +""" + +from __future__ import annotations + +import argparse +import collections +import json +import pathlib +import subprocess +import sys +from typing import Any, Sequence + +# Golden index -> regime. Indices not listed default to DETERMINISTIC. +DEFAULT_NEAR_TIE_INDICES = (2,) # "Once upon a time, in a small village," + +# A completion counts as degenerate when its most frequent whitespace token +# dominates (the async-race failure repeats a single token) or it is trivially +# short. Coherent greedy prose from this 8B never approaches these bounds. +MIN_WORDS = 8 +MAX_TOP_WORD_RATIO = 0.5 + + +def _run_vllm_cli( + vllm_cli: pathlib.Path, + snapshot: pathlib.Path, + prompt: str, + max_tokens: int, +) -> str: + """Return the greedy completion text (stdout) for one prompt.""" + + command = [ + str(vllm_cli), + "--model", + str(snapshot), + "--prompt", + prompt, + "--max-tokens", + str(max_tokens), + "--temperature", + "0", + "--seed", + "0", + ] + completed = subprocess.run( + command, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + check=False, + ) + if completed.returncode != 0: + raise RuntimeError( + f"vllm-cli exited {completed.returncode} for prompt {prompt!r}\n" + f"stderr:\n{completed.stderr}" + ) + # vllm-cli prints only the completion text (+ newline) to stdout; the loading + # banner and finish_reason/token counts go to stderr. + return completed.stdout.rstrip("\n") + + +def _is_coherent(text: str) -> tuple[bool, str]: + words = text.split() + if len(words) < MIN_WORDS: + return False, f"only {len(words)} words (< {MIN_WORDS})" + counts = collections.Counter(words) + top_word, top_count = counts.most_common(1)[0] + ratio = top_count / len(words) + if ratio >= MAX_TOP_WORD_RATIO: + return False, ( + f"token {top_word!r} repeats {top_count}/{len(words)} " + f"(ratio {ratio:.2f} >= {MAX_TOP_WORD_RATIO})" + ) + return True, f"{len(words)} words, top-word ratio {ratio:.2f}" + + +def run_battery( + vllm_cli: pathlib.Path, + snapshot: pathlib.Path, + golden: Sequence[dict[str, Any]], + near_tie_indices: Sequence[int], +) -> int: + near_tie = set(near_tie_indices) + deterministic_ok = 0 + deterministic_total = 0 + near_tie_ok = 0 + near_tie_total = 0 + failures: list[str] = [] + + for index, entry in enumerate(golden): + prompt = entry["prompt"] + expected = entry["text"] + max_tokens = len(entry["token_ids"]) + got = _run_vllm_cli(vllm_cli, snapshot, prompt, max_tokens) + regime = "near-tie" if index in near_tie else "deterministic" + print(f"--- prompt[{index}] ({regime}): {prompt!r}") + print(f" max_tokens={max_tokens}") + print(f" expected: {expected!r}") + print(f" got : {got!r}") + + if index in near_tie: + near_tie_total += 1 + coherent, why = _is_coherent(got) + if coherent: + near_tie_ok += 1 + verdict = "COHERENT (near-tie OK)" + else: + verdict = f"DEGENERATE ({why})" + failures.append(f"prompt[{index}] near-tie degenerate: {why}") + else: + deterministic_total += 1 + if got.strip() == expected.strip(): + deterministic_ok += 1 + verdict = "TOKEN-EXACT" + else: + verdict = "MISMATCH (deterministic regression)" + failures.append( + f"prompt[{index}] deterministic mismatch vs golden" + ) + print(f" verdict : {verdict}") + + print( + f"\nMXFP4 smoke battery: deterministic {deterministic_ok}/{deterministic_total} " + f"token-exact; near-tie {near_tie_ok}/{near_tie_total} coherent." + ) + passed = ( + deterministic_ok == deterministic_total + and near_tie_ok == near_tie_total + and deterministic_total > 0 + ) + if passed: + print("MXFP4 smoke battery: PASS (reproduces #44 3/4 token-exact + near-tie).") + return 0 + print("MXFP4 smoke battery: FAIL") + for failure in failures: + print(f" - {failure}") + return 1 + + +def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--vllm-cli", type=pathlib.Path, required=True) + parser.add_argument("--snapshot", type=pathlib.Path, required=True) + parser.add_argument("--golden", type=pathlib.Path, required=True) + parser.add_argument( + "--near-tie-index", + type=int, + action="append", + default=None, + help="golden index treated as an open-ended near-tie prompt (repeatable; " + "default: 2)", + ) + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> int: + args = _parse_args(argv) + if not args.vllm_cli.is_file(): + print(f"vllm-cli binary is absent: {args.vllm_cli}", file=sys.stderr) + return 2 + if not args.snapshot.is_dir(): + print(f"snapshot directory is absent: {args.snapshot}", file=sys.stderr) + return 2 + golden = json.loads(args.golden.read_text(encoding="utf-8")) + if not isinstance(golden, list) or not golden: + print(f"golden is not a non-empty list: {args.golden}", file=sys.stderr) + return 2 + near_tie_indices = ( + DEFAULT_NEAR_TIE_INDICES + if args.near_tie_index is None + else tuple(args.near_tie_index) + ) + try: + return run_battery(args.vllm_cli, args.snapshot, golden, near_tie_indices) + except RuntimeError as error: + print(f"MXFP4 smoke battery: ERROR\n{error}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/bench/online_gate.py b/tools/bench/online_gate.py index 7be3924c8..336af2701 100755 --- a/tools/bench/online_gate.py +++ b/tools/bench/online_gate.py @@ -200,17 +200,28 @@ # accepted domain so the component can reuse ``OnlineRun`` / ``build_client_command``. MAX_ONLINE_REPETITION = 5 POINTS = ((1, 6), (2, 6), (4, 12), (8, 24), (16, 96), (32, 192)) +# Bench-only additive model keys may sweep a reduced low-concurrency point set. +# The NVFP4 gate models ("27"/"35") keep the full six-point sweep (the default); +# the MXFP4 W4 throughput row ("q3mxfp4", a dense 8B keep-quant vehicle) benches +# only c1/c2/c4/c8 (the decode/low-concurrency regime the row is scoped to). The +# concurrency -> num_prompts mapping is a strict prefix of POINTS, so prompts_for +# stays unchanged; only the SET of benched concurrencies narrows per model. +POINTS_BY_MODEL = { + "q3mxfp4": ((1, 6), (2, 6), (4, 12), (8, 24)), +} MODEL_REVISIONS = { "27": "890bdef7a42feba6d83b6e17a03315c694112f2a", "35": "491c2f1ea524c639598bf8fa787a93fed5a6fbce", + "q3mxfp4": "b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47", } MODEL_REPOSITORIES = { "27": "unsloth/Qwen3.6-27B-NVFP4", "35": "nvidia/Qwen3.6-35B-A3B-NVFP4", + "q3mxfp4": "Yi30/Qwen3-8B-MXFP4", } MAX_NUM_SEQS = 32 -MAX_NUM_BATCHED_TOKENS = {"27": 2048, "35": 8192} -MAX_MODEL_LEN = {"27": 262144, "35": 262144} +MAX_NUM_BATCHED_TOKENS = {"27": 2048, "35": 8192, "q3mxfp4": 2048} +MAX_MODEL_LEN = {"27": 262144, "35": 262144, "q3mxfp4": 40960} ENGINES = ("ours", "vllm") PERCENTILE_METRICS = ("ttft", "tpot", "itl", "e2el") PERCENTILES = (50, 90, 99) @@ -277,6 +288,20 @@ def prompts_for(concurrency: int) -> int: raise HarnessError(f"unsupported online-gate concurrency: {concurrency}") from error +def points_for(model_key: str) -> tuple[tuple[int, int], ...]: + """The (concurrency, num_prompts) sweep for ``model_key``. + + Defaults to the full six-point :data:`POINTS`; a key present in + :data:`POINTS_BY_MODEL` uses its reduced set (a prefix of POINTS, so the + per-concurrency prompt counts are identical). This is the single source of + truth both this module and :mod:`online_gate_summary` consult so a bench-only + model key never triggers a spurious "missing result group" for a concurrency + it was never scoped to sweep. + """ + + return POINTS_BY_MODEL.get(model_key, POINTS) + + def trace_primary_graph_contract( model_key: str, expected_batch: int, @@ -804,7 +829,7 @@ def prepare_corpus_views( files: list[dict[str, Any]] = [] prompt_hashes: set[str] = set() for repetition in repetitions: - for concurrency, expected in POINTS: + for concurrency, expected in points_for(model_key): source = source_root / f"c{concurrency}-r{repetition}.jsonl" rows = list(read_jsonl(source)) if len(rows) < expected: diff --git a/tools/bench/online_gate_summary.py b/tools/bench/online_gate_summary.py index acae6becb..adb189fd4 100755 --- a/tools/bench/online_gate_summary.py +++ b/tools/bench/online_gate_summary.py @@ -38,6 +38,7 @@ POINTS, REPETITIONS, VLLM_ORACLE_VERSION, + points_for, precise_max_concurrent_requests, validate_raw_result, _fingerprint_tree, @@ -594,7 +595,7 @@ def _corpus_reasons(evidence_root: pathlib.Path, model: str) -> list[str]: files = manifest.get("files") expected = { (concurrency, repetition): requests - for concurrency, requests in POINTS + for concurrency, requests in points_for(model) for repetition in REPETITIONS } if not isinstance(files, list): @@ -732,7 +733,7 @@ def summarize_evidence( (model, engine, concurrency) for model in selected_models for engine in ENGINES - for concurrency, _ in POINTS + for concurrency, _ in points_for(model) } missing_groups = sorted(expected_groups - set(grouped)) campaign_reasons.extend( @@ -763,7 +764,7 @@ def summarize_evidence( pair_reasons: dict[tuple[str, int, int], list[str]] = defaultdict(list) output_text_diagnostics: list[dict[str, Any]] = [] for model in selected_models: - for concurrency, _ in POINTS: + for concurrency, _ in points_for(model): per_engine = { engine: {rep: run for rep, run in grouped.get((model, engine, concurrency), [])} for engine in ENGINES @@ -879,7 +880,7 @@ def summarize_evidence( ratios: list[dict[str, Any]] = [] for model in selected_models: - for concurrency, _ in POINTS: + for concurrency, _ in points_for(model): ours = aggregate_index[(model, "ours", concurrency)] floor = aggregate_index[(model, "vllm", concurrency)] for axis in (*HIGHER_AXES, *LOWER_AXES): From ff08065b953fe2f0e0341863600a0ca3c33a3d8f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 6 Aug 2026 00:43:31 +0000 Subject: [PATCH 2/2] =?UTF-8?q?bench(mxfp4):=20W4=20throughput=20grid=20RA?= =?UTF-8?q?N=20on=20GB10=20=E2=80=94=20BELOW-FLOOR,=20batched-decode=20gap?= =?UTF-8?q?=20(QUANT-CT-MXFP4-BENCH)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Executed the binding ours-vs-oracle online-serving grid on Yi30/Qwen3-8B-MXFP4 (dense Qwen3, native Marlin W4A16 MXFP4 keep-quant) at 33e93608 on dgx GB10. Production graphed vLLM 0.25.0, oracle forced to Marlin via VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel (sm_121 cute-dsl crashes), c1/c2/c4/c8 x3 interleaved, single load/arm, drop_caches between legs, both flock locks. Smoke model gate reproduced #44 (3/3 deterministic token-exact + coherent near-tie). VERDICT: BELOW-FLOOR (gate NO, 74/84 axes below). Medians (ours->vllm, ratio): - total tok/s: c1 0.989, c2 0.911, c4 0.919, c8 0.913 - median TPOT ms: c1 0.986, c2 0.900, c4 0.905, c8 0.891 - median TTFT: at parity (c1/c4 PASS, c2/c8 within noise) - peak GPU mem: 28.3 vs 73.7 GiB = 2.607 (ours 2.6x LESS) Per-rep spread ~1-3% (tight/reproducible; full per-rep in benchmark-record). FIRST attribution (grounded, no ceiling): the gap is a BATCHED-decode cost absent at batch-1 -- c1 is at parity (0.989 tput / 0.986 TPOT), c2-c8 is ~0.91x driven ENTIRELY by TPOT +10-12% with TTFT/prefill at parity. Divergent hot path = the grouped Marlin W4A16 keep-quant decode GEMM at M=2..8, not batch-1 GEMV nor prefill. Memory a clean 2.6x win. Next: same-tool nsys decode-window on c8 both engines to name the exact kernel/shape. Also: fixed the orchestrator ordering (plan must precede corpus -- online_gate.py `plan` refuses a non-empty evidence root) and recorded the strict online-serving build contract learned this run (DISK build not tmpfs, RelWithDebInfo, oracle-venv ninja, oracle flashinfer cutlass, export-compile-commands, profile-control OFF). Records: BENCHMARKS/STATUS/FEATURES + quantization-matrix + benchmark-record (full table + recipe + build contract) + NOW + state, same change. Box left clean. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/benchmark-record.md | 99 +++++++++++++++++----------- .agents/quantization-matrix.md | 2 +- .agents/state.md | 45 +++++++++++++ docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 4 +- scripts/mxfp4-online-serving-grid.sh | 30 ++++++--- 8 files changed, 133 insertions(+), 53 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 0ac47bbb1..bab777175 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -23,7 +23,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| QUANT-CT-MXFP4-BENCH | **PLUMBED** (`row/QUANT-CT-MXFP4-BENCH`): `q3mxfp4` key + smoke gate + oracle disabled-kernel; CPU tests 45/45 | Grid GATING dgx: `mxfp4-online-serving-grid.sh` c1-c8x3 | +| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS | Lever: batched Marlin decode GEMM (M=2..8); nsys c8 both engines | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index eb14c3375..0f8a6c70d 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,44 +19,67 @@ from relative link targets repointed for this file's location. # Benchmarks -## QUANT-CT-MXFP4 W4 throughput bench — harness PLUMBED, grid GATING on dgx (2026-08-07, `row/QUANT-CT-MXFP4-BENCH`) — NO numbers yet - -The MXFP4 W4A16 keep-quant compute is correctness-DONE on main (#38 + the -`ROW-SERVE-ASYNC-DENSE-MIRROR` #44 async fix: default-config e2e 3/4 token-exact + -p2 near-tie ratified). The one owed piece is the binding ours-vs-oracle throughput -grid on the SAME checkpoint (`Yi30/Qwen3-8B-MXFP4`, a dense `Qwen3ForCausalLM`). -The online-serving harness only carried the "27"/"35" NVFP4 gate-model keys; this -row adds the `q3mxfp4` key so the grid can run. NO throughput number is produced -or owed by THIS entry; it records the plumbing and the exact one-command recipe so -the grid re-runs deterministically once the dgx build+legs execute. - -**Landed (additive; 27/35 byte-unchanged, harness CPU contract tests 45/45 GREEN):** -- `tools/bench/online_gate.py`: `MODEL_REVISIONS`/`MODEL_REPOSITORIES` - (`Yi30/Qwen3-8B-MXFP4` @ `b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47`), - `MAX_NUM_BATCHED_TOKENS`=2048, `MAX_MODEL_LEN`=40960, and a `POINTS_BY_MODEL` - reduced sweep (c1/c2/c4/c8, a strict prefix of `POINTS`) surfaced through a new - `points_for(model_key)` helper that both the harness and `online_gate_summary` - consult (so a c1-c8 key never trips a "missing result group" for c16/c32). -- `tools/bench/mxfp4_smoke_gate.py`: the q3mxfp4 model-gate = the #44 e2e smoke - battery (vllm-cli greedy, temp 0, seed 0, vs `golden_marlin_w4a16.json`); - requires the 3 deterministic prompts token-exact + the story prompt coherent. -- `scripts/dgx-online-serving.sh`: q3mxfp4 branches — model validation, dense 2048 - batched-tokens, gate builds `vllm-cli`+`server` and runs the smoke gate instead - of a paged-engine ctest, oracle arm `env VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` - (Marlin W4A16; sm_121 cute-dsl mxf4 crash worked around) with no mamba flag, and - the low-concurrency `1 2 4 8` leg loop. -- `scripts/mxfp4-online-serving-grid.sh`: the one-command orchestrator (source-corpus - gen + dry-run manifest + locked `--execute` grid + `--model q3mxfp4` summary). - -**Repro (one command, on dgx, both flock locks free, free -g ≥ 90, ninja+nvcc on PATH):** -`scripts/mxfp4-online-serving-grid.sh --snapshot ~/.cache/huggingface/hub/models--Yi30--Qwen3-8B-MXFP4/snapshots/b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47 --build-dir --configure-log ` -(build recipe = git-archive → /dev/shm, `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON --DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0`, -Release, nvcc 13.0; build the `server` + `vllm-cli` targets only, never bare -`ninja`, to respect the disk floor). Verdict rule: at/above vLLM on total+output -tok/s AND at/below on TTFT+TPOT across c1/c2/c4/c8 → MXFP4 row DONE; below on any -cell → record the failing cells + a same-tool nsys attribution pass (decode-window -separated) before any lever campaign. +## QUANT-CT-MXFP4 W4 throughput bench — RAN on GB10, BELOW-FLOOR (2026-08-06, `row/QUANT-CT-MXFP4-BENCH` `33e93608`) + +The binding ours-vs-oracle online-serving grid on the SAME checkpoint +(`Yi30/Qwen3-8B-MXFP4`, dense `Qwen3ForCausalLM`, native Marlin W4A16 MXFP4 +keep-quant), oracle arm forced to Marlin via `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` +(sm_121 cute-dsl mxf4 crashes). Production graphed vLLM 0.25.0 (torch.compile ON, +Marlin W4A16), `--no-enable-prefix-caching` both arms, max-num-seqs 32, batched-tokens +2048, c1/c2/c4/c8 x3 interleaved, single load/arm, drop_caches + memory-return +between legs, both flock locks. Smoke model gate reproduced #44 (3/3 deterministic +token-exact + coherent near-tie). **VERDICT: BELOW-FLOOR** (gate NO, 74/84 axes below); +throughput ~0.91x at c2-c8, ~0.99x at c1; a large memory WIN. + +**Binding table (medians of 3 reps; per-rep spread ~1-3%):** + +| conc | total tok/s ours→vllm (ratio) | output tok/s ratio | med TTFT ms ours→vllm (ratio, ≥1 pass) | med TPOT ms ours→vllm (ratio, ≥1 pass) | +|---|---|---|---|---| +| c1 | 324.2 → 327.9 (**0.989**) | 0.989 | 287.5 → 295.2 (**1.027** PASS) | 25.70 → 25.34 (0.986) | +| c2 | 569.2 → 624.7 (**0.911**) | 0.911 | 458.4 → 450.3 (0.982) | 28.27 → 25.45 (0.900) | +| c4 | 951.4 → 1034.7 (**0.919**) | 0.919 | 862.1 → 867.5 (**1.006** PASS) | 31.03 → 28.07 (0.905) | +| c8 | 1419.6 → 1554.8 (**0.913**) | 0.913 | 1412.2 → 1408.3 (0.997) | 39.64 → 35.30 (0.891) | +| memory | peak GPU 28284 → 73723 MiB = **2.607 PASS** (ours 2.6x LESS) | — | — | — | + +Per-rep total tok/s — c1 ours 324.2/324.4/320.0 vllm 331.5/327.7/327.9; c2 ours +569.2/569.9/558.9 vllm 624.2/624.9/624.7; c4 ours 951.4/958.9/945.8 vllm +1040.9/1034.7/1034.3; c8 ours 1415.0/1433.9/1419.6 vllm 1560.9/1554.8/1554.7. Per-rep +TPOT ms — c1 ours 25.68/25.70/26.08 vllm 25.02/25.34/25.34; c2 ours 28.21/28.27/28.67 +vllm 25.43/25.45/25.49; c4 ours 31.03/30.92/32.48 vllm 27.94/28.07/28.12; c8 ours +39.64/39.23/39.77 vllm 35.15/35.34/35.30. + +**FIRST attribution (grounded in the per-concurrency curve — the gap is a BATCHED +decode cost, NOT batch-1):** at c1 (batch-1 decode) ours is at parity (throughput +0.989, TPOT 0.986, within run-noise). At c2-c8 (batched decode 2-8 seqs) ours is +~0.91x throughput driven ENTIRELY by TPOT (~10-12% higher per-token: c2 0.900, c4 +0.905, c8 0.891), while TTFT/prefill is at parity (2/4 cells pass, others within +noise). So the divergent hot path is the GROUPED Marlin W4A16 keep-quant decode GEMM +as M grows 2→8 (our per-expert/grouped tiling vs vLLM's Marlin), NOT the batch-1 +GEMV nor prefill. Memory is a 2.6x WIN (keep-quant weights stay compressed: 28.3 vs +73.7 GiB peak). NOT a ceiling. The precise NEXT step: same-tool nsys decode-window +on the WORST cell (c8) BOTH engines (our server under nsys; vLLM under nsys, not the +torch profiler — same tool) to name the exact divergent kernel/shape at M=8, then a +lever on the batched Marlin decode GEMM. + +**Repro (one command, dgx, both flock locks free, free -g ≥ 90):** +`scripts/mxfp4-online-serving-grid.sh --snapshot ~/.cache/huggingface/hub/models--Yi30--Qwen3-8B-MXFP4/snapshots/b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47 --build-dir --configure-log `. +**Build contract (LEARNED THE HARD WAY — the online-serving `record-execution` is +strict and DIFFERS from the generic build recipe):** the build MUST be on a REAL +disk (NOT `/dev/shm` — tmpfs pages are always resident so the cache-drop +`POSIX_FADV_DONTNEED`+mincore==0 proof fails on the server binary), and configured +`-DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc +-DCMAKE_MAKE_PROGRAM=$HOME/venvs/vllm-oracle/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON +-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON +-DVLLM_CPP_TRITON_REGEN=OFF -DVLLM_CPP_FLASH_ATTN=ON -DVLLM_CPP_SERVER=ON +-DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_BENCH_PROFILE_CONTROL=OFF -DVLLM_CPP_CUTLASS_DIR=$HOME/venvs/vllm-oracle/lib/python3.12/site-packages/flashinfer/data/cutlass` +(the oracle's flashinfer-bundled cutlass, which is 4.5.0 — `record-execution` pins +`VLLM_CPP_CUTLASS_DIR` to the oracle's own cutlass tree, not `$HOME/cutlass-4.5.0`). +The full build tree is only ~3 GiB (fits /home). nvcc 13.0.88. Evidence: +`dgx:~/work/vllm.cpp-online-gate/evidence/33e936086a116c840518afcf732076c9917e96a5` +(ratios.json + report.md + 24 raws + memory/thermal/cache-drop). Harness plumbing on +this row (all additive; 27/35 byte-unchanged; CPU contract tests 45/45): `online_gate.py` +`q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`; `mxfp4_smoke_gate.py`; +`dgx-online-serving.sh` q3mxfp4 branches; `mxfp4-online-serving-grid.sh` orchestrator. ## DeepSeek-V4-Flash UD-IQ2_M — IQ2_S + MXFP4 CPU keep-quant bring-up (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`) — no throughput owed (off-GPU correctness bring-up) diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index a8779283f..fd7d23b11 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -127,7 +127,7 @@ Registry source: | `QUANT-FP8-PCPT` | ModelOpt FP8 per-channel/per-token | W8/A8 | capability selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MXFP8-MODELOPT` | ModelOpt MXFP8 | W8/A8 | CUDA/ROCm/XPU dispatch | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MIXED-MODELOPT` | ModelOpt mixed precision | FP8/NVFP4/MXFP8 groups | per-layer | part | part | part | Y | Y | `PARTIAL` | 35B FP8+NVFP4 slice: [loader](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L118), [FP8 tests](../tests/vt/test_ops_fp8_cutlass.cpp#L188), [NVFP4 tests](../tests/vt/test_ops_moe_grouped.cpp#L453), [gate](../tests/parity/test_qwen36_paged_engine.cpp#L78) | leaf spec open | - | -| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. RESIDUAL: the W4 THROUGHPUT bench (c1..c8x3 vs oracle) harness is now PLUMBED (`row/QUANT-CT-MXFP4-BENCH`): `online_gate` `q3mxfp4` key (`MODEL_REVISIONS`/`REPOSITORIES` `Yi30/Qwen3-8B-MXFP4`@`b3e7ab32`, reduced c1/c2/c4/c8 `points_for` set, `MAX_NUM_BATCHED_TOKENS`/`MAX_MODEL_LEN`), a vllm-cli smoke model gate ([`mxfp4_smoke_gate.py`](../tools/bench/mxfp4_smoke_gate.py)) reproducing the #44 3/4+near-tie battery, `dgx-online-serving.sh` q3mxfp4 branches (oracle arm `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, no mamba flag) and a one-command orchestrator [`mxfp4-online-serving-grid.sh`](../scripts/mxfp4-online-serving-grid.sh); harness CPU contract tests GREEN (45/45). The GRID itself is GATING on dgx execution (fresh /dev/shm CUDA build + locked single-load legs). Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | +| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. W4 THROUGHPUT bench RAN on GB10 (`row/QUANT-CT-MXFP4-BENCH` `33e93608`, c1/c2/c4/c8x3, production graphed vLLM 0.25.0, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` -> Marlin W4A16): **BELOW-FLOOR** (gate NO). Total tok/s ours/vLLM 0.99x@c1 but ~0.91x@c2-c8 (c2 0.911, c4 0.919, c8 0.913); driven by TPOT +10-12%@c2-c8 (parity@c1); TTFT at parity; peak GPU mem **2.6x LESS** (28.3 vs 73.7 GiB). ATTRIBUTION (grounded, per-concurrency curve): the gap is a BATCHED-decode cost absent at batch-1 -> the grouped Marlin W4A16 decode GEMM at M=2..8, NOT batch-1 GEMV nor prefill; NOT a ceiling. NEXT: same-tool nsys decode-window on c8 both engines. Harness (all additive; 27/35 byte-unchanged; CPU tests 45/45): `online_gate` `q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`, [`mxfp4_smoke_gate.py`](../tools/bench/mxfp4_smoke_gate.py) (#44 battery), `dgx-online-serving.sh` q3mxfp4 branches, [`mxfp4-online-serving-grid.sh`](../scripts/mxfp4-online-serving-grid.sh). Full table + build contract (RelWithDebInfo + oracle flashinfer-cutlass + DISK build) in benchmark-record. Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | | `QUANT-CT-W4A8-FP8` | compressed-tensors W4A8 FP8 | W4/A8 | CUTLASS | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W4A8-INT8` | compressed-tensors W4A8 INT8 | W4/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W8A8-FP8` | compressed-tensors W8A8 FP8 | W8/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | diff --git a/.agents/state.md b/.agents/state.md index f2e2e6495..44df48cc1 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -36737,3 +36737,48 @@ any cell → record the failing cells + a same-tool decode-window nsys attributi (tight — the /dev/shm build sidesteps the /home floor). Evidence: this row's diff + `docs/bench-evidence/mxfp4-qwen/` golden. Records: STATUS/BENCHMARKS/FEATURES + quantization-matrix + benchmark-record + NOW updated in the same change. + +## QUANT-CT-MXFP4-BENCH: W4 throughput grid RAN on GB10 — BELOW-FLOOR (~0.91x c2-c8, batched-decode gap), memory 2.6x WIN + + +Executed the full binding grid on dgx GB10 for `33e93608` (`row/QUANT-CT-MXFP4-BENCH`). +The harness plumbing landed correctly and the grid produced honest numbers. Box left +clean (both flock locks free, GPU idle, /dev/shm scratch removed, evidence 9.6M on /home). + +RESULT (production graphed vLLM 0.25.0, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` +-> Marlin W4A16; ours native Marlin W4A16 keep-quant; c1/c2/c4/c8x3 interleaved, single +load/arm, drop_caches + memory-return between legs; smoke model gate reproduced #44 +3/3-det-token-exact + coherent near-tie): **gate NO, 74/84 axes below floor.** Medians +(ours->vllm, ratio): total tok/s c1 324.2->327.9 (0.989), c2 569.2->624.7 (0.911), c4 +951.4->1034.7 (0.919), c8 1419.6->1554.8 (0.913); median TPOT ms c1 25.70->25.34 (0.986), +c2 28.27->25.45 (0.900), c4 31.03->28.07 (0.905), c8 39.64->35.30 (0.891); median TTFT +at parity (c1 1.027 PASS, c4 1.006 PASS, c2 0.982, c8 0.997); peak GPU mem 28284 vs 73723 +MiB = **2.607 (ours 2.6x LESS)**. Per-rep spread ~1-3% (tight/reproducible). + +FIRST ATTRIBUTION (grounded in the per-concurrency curve; no ceiling): the gap is a +BATCHED-decode cost that is ABSENT at batch-1 — c1 (single-stream decode) is at parity +(0.989 tput / 0.986 TPOT, within noise), while c2-c8 (batched decode 2-8 seqs) is ~0.91x +tput driven ENTIRELY by TPOT +10-12%, with prefill/TTFT at parity. So the divergent hot +path is the GROUPED Marlin W4A16 keep-quant decode GEMM as M grows 2->8 (our per-expert/ +grouped tiling vs vLLM's Marlin), NOT the batch-1 GEMV nor prefill. Memory is a clean 2.6x +WIN (keep-quant weights stay compressed). NEXT (grounded lever start): same-tool nsys +decode-window on the WORST cell (c8) BOTH engines (our server under nsys AND vLLM under +nsys — same tool, per the aggregation trap, not the torch profiler) to name the exact +divergent kernel/shape at M=8, then a lever on the batched Marlin decode GEMM. + +BUILD-CONTRACT LESSONS (cost 2 rebuild cycles; recorded so the next runner does not +repeat them): the online-serving `record-execution` is STRICT and differs from the generic +build recipe. (1) The build MUST be on a REAL disk, NOT /dev/shm — tmpfs pages are always +resident, so the cache-drop POSIX_FADV_DONTNEED+mincore==0 proof retains the 441 MB server +binary and every leg fails; a /home build (whole tree only ~3 GiB, well within the floor) +fixes it AND frees the RAM pool for the oracle's 72 GiB reservation. (2) The CMake cache +must be RelWithDebInfo (not Release), CMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc, +CMAKE_MAKE_PROGRAM=the oracle-venv ninja, EXPORT_COMPILE_COMMANDS=ON, BENCH_PROFILE_CONTROL +=OFF, BUILD_TESTS=ON, and VLLM_CPP_CUTLASS_DIR = the ORACLE's flashinfer-bundled cutlass +(also 4.5.0), NOT $HOME/cutlass-4.5.0 (record-execution pins it to the oracle's own tree). +(3) The orchestrator's plan step must precede corpus-gen (online_gate.py `plan` refuses a +non-empty evidence root); fixed in `mxfp4-online-serving-grid.sh` this change. Repro: +`scripts/mxfp4-online-serving-grid.sh --snapshot --build-dir +--configure-log `. Evidence: `dgx:~/work/vllm.cpp-online-gate/evidence/33e93608...` +(ratios.json + report.md + 24 raws + memory/thermal/cache-drop). local-ai-worker left as +found (down); dgx /home build tree `~/mxfp4-bench` retained for the nsys continuation. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index b70f5e525..8fe6aa46a 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED (max gap 0.0000 nats). W4 throughput bench harness PLUMBED (`row/QUANT-CT-MXFP4-BENCH`: `q3mxfp4` key, smoke model gate, one-command grid); grid GATING on dgx (recipe + detail in benchmark-record) | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record) | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7758a5406..6c5d2ddb6 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -65,7 +65,7 @@ are our reading of their documented behavior, not measurements. | GGUF k-quants and i-quants | ✅ | ☐ | ☐ | ✅ | | AWQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | -| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; default-config e2e coherent + 3/4 token-exact (async-mirror fix); throughput bench harness plumbed (q3mxfp4 key), grid gating on dgx | ✅ | ✅ | ☐ | +| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; e2e 3/4 token-exact; W4 throughput grid ran on GB10: ~0.91x vLLM c2-c8 / 0.99x c1, peak GPU mem 2.6x less; below-floor (batched-decode gap) | ✅ | ✅ | ☐ | | fp8 weights | ✅ | ✅ | ✅ | ☐ | | bf16 / fp16 | ✅ | ✅ | ✅ | ✅ | | Safetensors direct load, no conversion | ✅ | ✅ | ✅ | ☐ | diff --git a/docs/STATUS.md b/docs/STATUS.md index 060ebedc0..cba63198f 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -53,7 +53,7 @@ token-for-token correctness against the pinned oracle. |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue | -| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 throughput bench (online_gate harness now PLUMBED on `row/QUANT-CT-MXFP4-BENCH`, grid GATING on dgx). Detail in state.md. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | +| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 throughput bench RAN on GB10 (`row/QUANT-CT-MXFP4-BENCH`): BELOW-FLOOR ~0.91x c2-c8, batched-decode gap. Detail in state.md. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | | Llama-3.x dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 16/16 (Llama-3.2-1B); llama3 RoPE scaling | @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput bench harness PLUMBED (`row/QUANT-CT-MXFP4-BENCH`, `q3mxfp4` key); grid GATING on dgx | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput grid RAN on GB10 (`33e93608`): ~0.91x vLLM c2-c8 / 0.99x c1, peak GPU mem 2.6x LESS; BELOW-FLOOR (batched-decode gap) | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | diff --git a/scripts/mxfp4-online-serving-grid.sh b/scripts/mxfp4-online-serving-grid.sh index e61ac8b1f..90a374f17 100755 --- a/scripts/mxfp4-online-serving-grid.sh +++ b/scripts/mxfp4-online-serving-grid.sh @@ -14,6 +14,15 @@ # The oracle arm needs ninja + nvcc on PATH (the DGX non-interactive quirk); run # under a shell that has them, both flock locks free, free -g >= 90, and the # production (profile-control-OFF) CUDA build already configured. +# +# BUILD CONTRACT (record-execution is strict — measured 2026-08-06): the --build-dir +# MUST be on a REAL disk, NOT tmpfs/dev/shm (tmpfs pages stay resident so the harness +# cache-drop POSIX_FADV_DONTNEED+mincore==0 proof fails on the server binary), and +# configured RelWithDebInfo with CMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc, +# CMAKE_MAKE_PROGRAM=/bin/ninja, CMAKE_EXPORT_COMPILE_COMMANDS=ON, +# VLLM_CPP_BENCH_PROFILE_CONTROL=OFF, VLLM_CPP_BUILD_TESTS=ON, and VLLM_CPP_CUTLASS_DIR +# pointing at the ORACLE's flashinfer-bundled cutlass tree (4.5.0), not $HOME/cutlass-4.5.0. +# Full tree ~3 GiB. See .agents/benchmark-record.md for the exact configure line. set -euo pipefail usage() { @@ -78,7 +87,18 @@ source_corpus="${evidence}/corpus/${model}" oracle_python="$(dirname "${client}")/python" [[ -x ${oracle_python} ]] || { echo "vLLM oracle Python is absent: ${oracle_python}" >&2; exit 2; } -# 1. Deterministic source corpus (exact 1024-token prompts) with the checkpoint's +# 1. Dry-run manifest FIRST: `online_gate.py plan` refuses to write into a +# non-empty evidence root, so it must run before the corpus populates +# ${evidence}/corpus. (A partially-populated evidence root from an aborted run +# must be removed before re-running.) +if [[ ! -f ${evidence}/manifest.json ]]; then + "${driver}" --dry-run \ + --claim-root "${claim_root}" \ + --client "${client}" \ + --vllm-cpp-sha "${vllm_cpp_sha}" +fi + +# 2. Deterministic source corpus (exact 1024-token prompts) with the checkpoint's # own tokenizer, scoped to the c1/c2/c4/c8 low-concurrency sweep. Idempotent. if [[ ! -f ${source_corpus}/manifest.json ]]; then mkdir -p "${source_corpus}" @@ -91,14 +111,6 @@ if [[ ! -f ${source_corpus}/manifest.json ]]; then --repetitions 3 fi -# 2. Dry-run manifest (writes ${evidence}/manifest.json that --execute validates). -if [[ ! -f ${evidence}/manifest.json ]]; then - "${driver}" --dry-run \ - --claim-root "${claim_root}" \ - --client "${client}" \ - --vllm-cpp-sha "${vllm_cpp_sha}" -fi - # 3. The locked, single-load, drop_caches c1/c2/c4/c8 x3 grid (both arms). The # driver runs the #44 smoke model gate, records the oracle, and emits the # per-axis summary (online_gate_summary --model q3mxfp4) as its final step.