Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/NOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
| DeepSeek-V4-Flash decode | **Closed: beats ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact); phase-2 residency NEG, default-OFF | — |
| f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
| Invocation-parity prevention | CI guard + AGENTS.md checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
| MiniMax-H3 lane | **RENDER BUG CLOSED** (`H3-RENDER-CLOSE` PR #77): #70/#74 white = t2va on the REF2VA ckpt; the FL2VA GGUF t2va renders COHERENT (adj-cos 0.95) | Follow-up: partition guard + vision tower |
| MiniMax-H3 lane | **RENDER + task/partition guard CLOSED** (#77 render; `H3-TASK-PARTITION-GUARD`: t2va-on-ref2va now RAISES like `_resolve_task`) | Follow-up: vision tower (W3) |
| Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF |
| 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
| Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
Expand Down
45 changes: 45 additions & 0 deletions .agents/benchmark-record.md
Original file line number Diff line number Diff line change
Expand Up @@ -14167,3 +14167,48 @@ noise fix from #70 stands but was already known not to be the render fix.
| NVFP4 dequant | independent torch dequant of `blocks.0.attn.qkv_proj` + Laguna/Qwen3 already prove `DequantNvfp4ToBf16` byte-exact | sane trained weight (rms 0.089, absmax=ws2·6·maxscale=3.61) |
| CUDA kernels at real seq | NEW gate `test_minimax_h3 :: CUDA device forward tracks the host at the REAL render seq (1920)` (RealRatioParams head_dim=128, seq 1920) | **CUDA device == CPU host** (28/28) — no scale-dependent kernel bug (#74 only ran device-vs-host on the CPU backend) |
| forward math | RefDiT restatement vs true upstream source, read side by side (block, attention, AdaLN view(m*3,6H), 3D-RoPE, modulate) | identical |

## MiniMax-H3 TASK/PARTITION GUARD — mirror `_resolve_task`'s raise; the two arms have NO structural discriminator so a stripped file must DECLARE its partition (2026-08-06, `row/H3-TASK-PARTITION-GUARD` PR #84, `ROAD-V1-H3`, CPU-only dev box)

The #70/#74 follow-up. The white grid was `task=t2va` run on the Ref2VA-partition
checkpoint; upstream `pipeline._resolve_task` RAISES on the mismatch
(`pipeline_minimax_h3.py:374-391`, raise at 387-390), and the recipe documents the split
(`recipes/MiniMaxAI/MiniMax-H3.md:50-51,289`). This row mirrors the raise 1:1 and adds the
community-file fallback.

**The discriminator finding (grounded both sides).** Upstream derives the partition from
the release config `model_index.json` → `_minimax_h3` → `{partition, tasks}`
(`pipeline_minimax_h3.py:279-282`). Community GGUF/NVFP4 STRIP that block, and there is
NO structural fallback — MEASURED on the two real captured manifests:

| Arm | Manifest (tensors) | Normalized base names | Key shapes (video/audio/condition/time patch) |
|---|---|---|---|
| Ref2VA | `minimax_h3_nvfp4_manifest.inc` (1051) | 535 after collapsing `{weight,weight_scale,weight_scale_2}` | `[5376,96]`/`[5376,32]`/`[5376,5120]`/`[5376,256]` |
| FL2VA | `minimax_h3_gguf_manifest.inc` (535) | 535 | identical |

`comm -23`/`-13` of the two normalized name sets is EMPTY both ways; all reference-relevant
shapes match. Ref2VA prepends reference rows through the SAME `video/audio_patch_proj`, so
it adds no reference-specific tensor. A name/shape auto-detector is impossible in principle,
so a stripped file must DECLARE `--partition fl2va|ref2va` (server: `--video-partition`);
`MiniMaxH3PartitionFromFlag` maps it to the recipe's served-task set (fl2va→{t2va,fl2va},
ref2va→{ref2va}).

**Guard behavior table (task × partition → pass/refuse):**

| task \ partition | FL2VA {t2va,fl2va} | Ref2VA {ref2va} | unknown/stripped |
|---|---|---|---|
| t2va | pass | **REFUSE (#77 mismatch)** | REFUSE (declare `--partition`) |
| fl2va | pass | REFUSE | REFUSE |
| ref2va| REFUSE | pass | REFUSE |

**RED-first proof (reviewer mutation).** New case `test_minimax_h3 :: "the task/partition
guard refuses the #77 mismatch"`, 38 assertions: the #77 combo throws, correct pairings
pass, stripped refuses every task, `--partition` recovers, `MiniMaxH3TaskOfRequest` maps
the three request shapes, and the two real manifests are asserted to reduce to the same
535-name set. Neutralizing the guard body (`return;` at the top of
`MiniMaxH3CheckTaskPartition`) turned the case RED at **10 failed assertions** (t2va-on-
ref2va, the stripped refusals, the dispatch-level `t2va_req` on Ref2VA); restoring it went
GREEN. Suite **67/67** (66 prior + this), 46549 assertions; `test_video_api` 4/4 (server
wiring). Wired at both loading entry points: `MiniMaxH3GenerateT2va` guards every full
render; the pure pipeline-math tests build `declared=false` requests so the guard is inert
there. No numbers changed — this is a correctness/refusal gate, not a perf lever.
59 changes: 59 additions & 0 deletions .agents/specs/minimax-h3.md
Original file line number Diff line number Diff line change
Expand Up @@ -565,3 +565,62 @@ trailing target rows are the clip; the old code fed unpatchify the full buffer a
guard mirroring upstream (community files strip the release config); the encoder vision
tower (W3) is still unported, so image/video-conditioned ref2va/fl2va renders are not yet
clean (ref2va with a synthetic reference + text-only encoder still grids).

## 8.7 TASK/PARTITION GUARD — mirror `_resolve_task`'s raise (2026-08-06, `row/H3-TASK-PARTITION-GUARD` PR #84)

The #70/#74 white grid cost three campaigns because our driver silently accepted
`task=t2va` on the Ref2VA-partition checkpoint. Upstream `pipeline._resolve_task`
RAISES on the mismatch (`pipeline_minimax_h3.py:374-391`, esp. 387-390); the recipe
documents the split (`recipes/MiniMaxAI/MiniMax-H3.md:50-51,289`: "One server loads one
checkpoint partition … must match the served partition"). This row mirrors the raise 1:1.

**Partition detection — two paths, and the definitive no-discriminator finding.**
Upstream reads the served-task set from the release config
(`pipeline_minimax_h3.py:279-282`):

```
release = model_index.get("_minimax_h3") or {}
self.partition = str(release.get("partition", "")) # "fl2va" | "ref2va"
self.supported_tasks = frozenset(release.get("tasks") or ())
```

`MiniMaxH3PartitionFromModelIndex(model_index)` mirrors those exact keys. But community
GGUF/NVFP4 redistributions STRIP that block, and — measured on the two real manifests
this spec already captured — there is **NO structural fallback**: the Ref2VA NVFP4
(1051 tensors) and FL2VA GGUF (535 tensors) carry the **IDENTICAL DiT**. Normalizing the
NVFP4 `{weight, weight_scale, weight_scale_2}` split, both files reduce to the **SAME 535
base tensor names AND the SAME shapes** (video_patch_proj `[5376,96]`, audio_patch_proj
`[5376,32]`, condition_proj `[5376,5120]`, time_embedder.proj_in `[5376,256]` on both;
`comm -23`/`-13` of the normalized name sets is empty both ways). Ref2VA conditioning is
achieved by PREPENDING reference rows through the SAME `video/audio_patch_proj` weights,
so it introduces no reference-specific tensor to key on. A name/shape auto-detector is
therefore impossible in principle. When the config is stripped the partition must be
**DECLARED** (`--partition fl2va|ref2va`), never guessed; `MiniMaxH3PartitionFromFlag`
maps it to the recipe's served-task set (fl2va→{t2va,fl2va}, ref2va→{ref2va}).

**The refuse.** `MiniMaxH3CheckTaskPartition(task, info)` is the raise half of
`_resolve_task`. The task is what the request ENCODES (`MiniMaxH3TaskOfRequest`:
`ref_blocks`→ref2va, `keyframe_frame_indices`→fl2va, else t2va), and
`MiniMaxH3GenerateT2va` calls the pair before denoising. A declared partition refuses a
task it does not serve; an UNKNOWN partition (stripped file, no `--partition`) refuses
EVERY task as ambiguous and names the recipe lines. A default-constructed
`MiniMaxH3PartitionInfo` (`declared=false`) leaves the guard inactive, so the pure
pipeline-math unit tests are unaffected. Wired at both checkpoint-loading entry points:
the driver (`--partition`) and the server (`--video-partition`).

**Guard behavior table (task × partition → pass/refuse):**

| task \ partition | FL2VA {t2va,fl2va} | Ref2VA {ref2va} | unknown/stripped |
|---|---|---|---|
| **t2va** | pass | **REFUSE (the #77 mismatch)** | REFUSE (declare `--partition`) |
| **fl2va** | pass | REFUSE | REFUSE |
| **ref2va**| REFUSE | pass | REFUSE |

**RED-first proof.** New case `test_minimax_h3 :: "the task/partition guard refuses the
#77 mismatch"` (38 assertions): the #77 combo `MiniMaxH3CheckTaskPartition("t2va",
ref2va)` throws; the correct pairings pass; the stripped case refuses every task and
`--partition` recovers it; `MiniMaxH3TaskOfRequest` maps the three request shapes; and it
asserts the two real manifests reduce to the identical 535-name set (proving the
no-discriminator premise in the harness). Neutralizing the guard body (reviewer mutation)
turned the case RED at 10 assertions, restoring it turned it GREEN — the test has teeth.
Suite: 67/67 (66 prior + this), 46549 assertions. `test_video_api` 4/4 (server wiring).
38 changes: 38 additions & 0 deletions .agents/state.md
Original file line number Diff line number Diff line change
Expand Up @@ -39742,3 +39742,41 @@ partition/supported_tasks guard mirroring upstream (community files strip the re
and the encoder vision tower (W3) for clean image/video-conditioned ref2va/fl2va (ref2va with
a synthetic reference + text-only encoder still grids). dgx assets: `~/h3fp4/ckpt/MiniMax-H3-
FL2VA-Q3_K_M.gguf`, `~/h3fp4/fl2va_t2va_20/`. Box left clean.

## MiniMax-H3 task/partition GUARD landed — mirror `_resolve_task`'s raise (`row/H3-TASK-PARTITION-GUARD` PR #84, helper, DRAFT PR, CPU-only)
<!-- state: 2026-08-06T23:59 -->

The #77 follow-up: the driver silently accepted `task=t2va` on the Ref2VA-partition
checkpoint (the #70/#74 white grid, three campaigns). Upstream `pipeline._resolve_task`
RAISES on the mismatch (`vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py:374-391`,
raise at 387-390); the recipe documents the split (`recipes/MiniMaxAI/MiniMax-H3.md:50-51,289`:
one server serves one partition, FL2VA→{t2va,fl2va}, Ref2VA→{ref2va}). Mirrored 1:1.

DISCRIMINATOR FINDING (definitive, both sides). Upstream reads the served-task set from the
release config `model_index.json` → `_minimax_h3` → `{partition,tasks}` (pipeline:279-282);
`MiniMaxH3PartitionFromModelIndex` mirrors those exact keys. Community GGUF/NVFP4 STRIP that
block, and there is NO structural fallback — MEASURED on the two real captured manifests: the
Ref2VA NVFP4 (`minimax_h3_nvfp4_manifest.inc`, 1051 tensors) and FL2VA GGUF
(`minimax_h3_gguf_manifest.inc`, 535) carry the IDENTICAL DiT. Collapsing the NVFP4
`{weight,weight_scale,weight_scale_2}` split, both reduce to the SAME 535 base tensor names AND
the SAME shapes (`comm` empty both ways; video_patch_proj `[5376,96]`, condition_proj
`[5376,5120]`, etc. equal). Ref2VA prepends reference rows through the SAME
`video/audio_patch_proj`, adding no reference-specific tensor. So a stripped file must DECLARE
`--partition fl2va|ref2va` (server `--video-partition`); `MiniMaxH3PartitionFromFlag` maps it.

CODE. New (pure, gated-on-any-machine, `minimax_h3_planner.cpp`): `MiniMaxH3PartitionInfo`
(`declared`/`partition`/`supported_tasks`), `MiniMaxH3PartitionFromModelIndex`,
`MiniMaxH3PartitionFromFlag`, `MiniMaxH3TaskOfRequest` (ref_blocks→ref2va, keyframes→fl2va,
else t2va), and the raise `MiniMaxH3CheckTaskPartition`. `MiniMaxH3GenerateT2va` calls the
pair before denoising; a default `declared=false` request leaves it inert (pipeline-math tests
unaffected). Driver `--partition` + server `--video-partition` wired. An unknown partition
refuses EVERY task (ambiguous) and names the recipe lines.

GATE. New case `test_minimax_h3 :: "the task/partition guard refuses the #77 mismatch"` (38
assertions): #77 combo throws, correct pairings pass, stripped refuses+`--partition` recovers,
task-of-request maps all three shapes, and the two real manifests are asserted to the same
535-name set. RED-first (reviewer mutation): neutralizing the guard body → 10 failed
assertions; restored → GREEN. Suite 67/67 (66 prior +1), 46549 assertions; `test_video_api`
4/4 (server wiring). No numbers changed (refusal gate, not a perf lever). Records: spec §8.7,
STATUS/BENCHMARKS H3 rows, benchmark-record, NOW. Pre-existing preflight red
(check-fusion-consistency `minimax_h3_video_vae_device`) is not this row.
2 changes: 1 addition & 1 deletion docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ built on it rather than keeping the flattering one.
| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
| vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. Detail: benchmark-record + spec §8 |
| MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920; dequant byte-exact. spec §8.6 |
| MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) |
| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** (+951us). `FLASH-OCCUPANCY` #75: matched-c8 ncu, occupancy IDENTICAL 8.33%; built vLLM's exact flash recipe, matched reg+instr, STILL +10us, gap is ptxas SASS quality, no lever/flip |
| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: no number measured, claimed or owed.** Vulkan registers 8 of the CPU backend's 83 ops and runs no model. The 2026-08-06 campaign spec is PLAN ONLY. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-E`: build llama.cpp `-DGGML_VULKAN=ON` at pin `237ad9b96` on dgx, `llama-bench` on the same GGUF, record three columns (ours-Vulkan, llama.cpp-Vulkan, ours-CUDA). Blocked by `VK-B` and by `glslc` on the gate box |
| SGLang floor arms | Never ran | Both arms of the SGLang comparison |
Expand Down
2 changes: 1 addition & 1 deletion docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ model architecture is wired.
| Image | ✅ correctness-gated | ✅ | ✅ | ◐ |
| Video | ✅ correctness-gated | ✅ | ✅ | ☐ |
| Audio | ✅ correctness-gated | ✅ | ◐ | ◐ |
| Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ t2va renders a COHERENT prompt-matched scene on GB10 (FL2VA-partition GGUF → h264/AAC mp4); render bug CLOSED (was wrong-partition usage); Marlin W4A16 byte-exact | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ |
| Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ t2va renders a COHERENT prompt-matched scene on GB10 (FL2VA-partition GGUF → h264/AAC mp4); render bug CLOSED (was wrong-partition usage); task/partition guard mirrors `_resolve_task`; Marlin W4A16 byte-exact | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ |
| Multimodal over the OpenAI server | ☐ | ✅ | ✅ | ◐ |

Image, video and audio are correct through the CLI and library. Serving them
Expand Down
2 changes: 1 addition & 1 deletion docs/STATUS.md

Large diffs are not rendered by default.

18 changes: 17 additions & 1 deletion examples/minimax_h3_gen/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@
// --audio-vae <audio_vae.safetensors> --audio-vae-config <config.json>
// --prompt-embeds <f32.bin> (rows of text_dim, little-endian f32)
// --out <out.mp4>
// [--partition fl2va|ref2va] (REQUIRED for a full render: the
// served checkpoint partition — community GGUF/NVFP4 strip it and
// the FL2VA/Ref2VA DiTs are indistinguishable. t2va/fl2va need
// fl2va, ref2va needs ref2va; recipe:50-51,289)
// [--keep-quant] [--steps N] [--frames N] [--height N] [--width N]
// [--workdir DIR] [--ffmpeg PATH] [--dry-run]
//
Expand Down Expand Up @@ -170,6 +174,11 @@ int main(int argc, char** argv) {
std::string roundtrip_path; // diagnostic: encode->decode a real image
std::vector<std::string> ref_image_paths;
std::string ref_video_prefix, ref_audio_path;
// The served checkpoint PARTITION. Community GGUF/NVFP4 files strip the release
// model_index.json `_minimax_h3` block, and the FL2VA/Ref2VA DiTs are structurally
// identical, so it cannot be inferred from the weights — it must be DECLARED. Empty
// => the guard refuses a full render and tells the user to pass it (the #77 catch).
std::string partition_flag;
double imgvid_noise_aug = 1.0;
int64_t encoder_max_layers = 0;
int64_t steps = 0, frames = 0, height = 0, width = 0;
Expand Down Expand Up @@ -204,6 +213,7 @@ int main(int argc, char** argv) {
else if (f == "--ref-image") ref_image_paths.push_back(Need(argc, argv, ++i, f));
else if (f == "--ref-video") ref_video_prefix = Need(argc, argv, ++i, f);
else if (f == "--ref-audio") ref_audio_path = Need(argc, argv, ++i, f);
else if (f == "--partition") partition_flag = Need(argc, argv, ++i, f);
else if (f == "--noise-aug") imgvid_noise_aug = std::stod(Need(argc, argv, ++i, f));
else if (f == "--encoder-max-layers") encoder_max_layers = std::stoll(Need(argc, argv, ++i, f));
else if (f == "--steps") steps = std::stoll(Need(argc, argv, ++i, f));
Expand Down Expand Up @@ -234,7 +244,8 @@ int main(int argc, char** argv) {
"[--height N] [--width N] [--device cpu|cuda] [--workdir DIR] [--ffmpeg PATH] "
"[--dry-run] [--denoise-only] [--dump-params] "
"[--first-frame f.ppm] [--last-frame f.ppm] [--noise-aug A] "
"[--ref-image f.ppm ...] [--ref-video DIR] [--ref-audio f.wav]\n";
"[--ref-image f.ppm ...] [--ref-video DIR] [--ref-audio f.wav] "
"[--partition fl2va|ref2va]\n";
return 2;
}

Expand Down Expand Up @@ -578,6 +589,11 @@ int main(int argc, char** argv) {
request.video_latents_std = video_stats.std_dev;
request.audio_latents_mean = audio_stats.mean;
request.audio_latents_std = audio_stats.std_dev;
// Declare the served partition so MiniMaxH3GenerateT2va can refuse a task the
// checkpoint does not serve (the #77 catch: t2va on the Ref2VA arm). An empty
// --partition is declared-but-unknown; the guard then names the recipe lines and
// asks for fl2va|ref2va rather than silently rendering the wrong combination.
request.partition = vllm::MiniMaxH3PartitionFromFlag(partition_flag);

// --- ref2va VIDEO reference: a CLIP prepended to the sequence. Reads
// DIR/frame_%06d.ppm, which is exactly what this example WRITES, so a previous
Expand Down
Loading