From 7ad1dc46d04838136714de84cbb1c8189b3e44ea Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 17:01:59 +0000 Subject: [PATCH 1/2] =?UTF-8?q?record(minimax-h3):=20the=20REF=20canvas=20?= =?UTF-8?q?renders=20COHERENT=20=E2=80=94=201344x768/124f=20measured=20end?= =?UTF-8?q?=20to=20end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §8.4 left the vllm-omni REF canvas (768x1344, 124f) UNRUN on honesty grounds ("a full 50-step render is ~2.85 h, so it was not run"). This row RAN it on Thor (sm_110, portable fallback attention throughout) and measured every stage. The §8.4 white latent does NOT reproduce at the REF canvas on the matching partition. VAE-input latent adjacent-cell cosine 0.8924 (§8.4 white 0.06, real encode 0.789, §8.6 coherent 0.9467); the init-noise control reads 0.0019, white as it must. Decoded frames carry a period-16 seam ratio of 1.15, equal to the known-good 864x480 render. What the earlier grids were is task/partition, measured on one canvas and prompt: ref2va-on-FL2VA 2.28, t2va pre-guard 1.87, t2va on the guarded binary with Gaussian init noise 1.15. The artifact SCALES WITH CANVAS, which is why small canvases looked acceptable and the REF canvas did not. NEW BUG recorded: the pipeline's own decode OOMs at the REF canvas. The 50-step denoise completes and the latent dump lands, then VAE decode exhausts the GPU pool and reboots the box (NV_ERR_NO_MEMORY in the kernel log). The dequantised bf16 DiT stays resident across MiniMaxH3VideoVaeDecodeTemporalDevice; decoding the SAME dumped latent standalone (VAE only) completes at 23 GiB of 122 GiB and produced the 1.15 frames. Generation is correct there; the decode is what does not fit alongside the model. Also corrects a comment that claimed `decoder_tiling` "defaults false" while minimax_h3.h has `bool decoder_tiling = true`. Tiling composes with temporal chunking by default and is required, not optional, because the ViT3D's RoPE is length-normalized over the grid it is handed. Record-only plus a comment; no behaviour change. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] --- .agents/specs/minimax-h3.md | 50 +++++++++++++++++++ .../models/minimax_h3_pipeline.cpp | 9 ++-- 2 files changed, 56 insertions(+), 3 deletions(-) diff --git a/.agents/specs/minimax-h3.md b/.agents/specs/minimax-h3.md index 16c6e59c2..1dd411d6a 100644 --- a/.agents/specs/minimax-h3.md +++ b/.agents/specs/minimax-h3.md @@ -1129,3 +1129,53 @@ the projections never touch a host buffer), total peak **51.95 GiB** of the 122 UNIFIED pool. Streamer counters on the real checkpoint: `layers=50 tensors=400 direct=350 converted=200 fused=100`, i.e. the shard path ran and every projection took the no-host-copy upload. + +## 8.16 The REF CANVAS renders COHERENT — 1344x768/124f measured end to end, and the pipeline's own decode OOMs there (2026-08-08, `row/H3-REF-CANVAS-RENDER`, Thor sm_110) + +§8.4 left the vllm-omni REF canvas (768x1344, 124f) UNRUN — "a full 50-step render is +~2.85 h, so it was not run (largest-fitting-config honesty)". This row RAN it, on Thor +(sm_110, 122.8 GiB unified, no FA2 so the portable fallback attention throughout), and +measured every stage. **The render is COHERENT at the REF canvas** on the matching +partition; the §8.4 white latent does NOT reproduce. + +**Config:** `MiniMax-H3-FL2VA-Q4_K_M.gguf` (`--dequant-bf16`), `--partition fl2va`, +task **t2va** (no reference image), 1344x768, 124 frames, 50 steps, seq_len 38080, +176.3 s/forward, `VT_H3_DUMP_DIR` on. + +| Stage | Metric | Value | Reference | +|---|---|---|---| +| init noise | adj-cell cosine | **0.0019** | white by construction (control) | +| init noise | r_W / r_H / r_T | 0.999 / 1.001 / 0.999 | 1.0 == spatially white | +| VAE-input latent | adj-cell cosine | **0.8924** | §8.4 white 0.06; real encode 0.789; §8.6 coherent 0.9467 | +| VAE-input latent | r_W / r_H / r_T | 0.347 / 0.313 / 0.311 | well below the 1.0 white floor | +| decoded frames | period-16 seam ratio | **1.15** | == the known-good 864x480 render (1.15); ref2va-on-FL2VA was 2.28 | + +`r` is mean|adjacent diff| / E|X-Y| along one axis: 1.0 for spatially white, lower when +correlated. Calibrated before use on synthetic latents (white 1.00, smooth field 0.44, +smooth+30% per-cell noise 0.86), so the instrument is known to separate the two cases. + +**Task/partition is what the earlier grids were.** Renders passing `--ref-image` against +the FL2VA partition are ref2va-on-FL2VA, the combination §8.6 identified and §8.7's guard +now REJECTS at the CLI. Measured on the same canvas and prompt: ref2va-on-FL2VA **2.28**, +t2va pre-guard binary **1.87**, t2va on the guarded binary with Gaussian init noise +**1.15**. The artifact SCALES WITH CANVAS (864x480 showed 1.15 while 1344x768 showed 2.28 +on the same mismatch), which is why small-canvas runs looked acceptable and the REF canvas +did not. + +**NEW BUG — the pipeline decode OOMs at the REF canvas.** The 50-step denoise completed +and the latent dump was written; VAE decode then exhausted the GPU pool and REBOOTED the +box (`NVRM: GPU0 ... Out of memory [NV_ERR_NO_MEMORY] ... _memdescAllocInternal`, kernel +log, ~1 min after the dump, box down ~2 min later). The dequantised bf16 DiT stays +resident through `MiniMaxH3VideoVaeDecodeTemporalDevice`. Decoding the SAME dumped latent +standalone — VAE only, no DiT resident — completes with room to spare and produced the +1.15 frames above. So generation is correct at the REF canvas and the DECODE is what does +not fit alongside the model. A driver that frees the DiT before decode, or decodes in +temporal slices, is the fix; not attempted in this row. + +**Residuals.** (1) No full 124-frame MP4 with audio at the REF canvas yet — blocked on the +OOM above. (2) The audio latent has no `VT_H3_DUMP_DIR` hook, so the audio arm is +unmeasured; it was diagnosed only through the shared-sequence argument. (3) `--roundtrip` +and `--decode-latent` went with the pre-fold binary (documented at +`minimax_h3_gen/main.cpp:31-36`), so replaying a dumped latent through the VAE now needs a +throwaway harness against `MiniMaxH3VideoVaeDecodeTemporalDevice`; a `--decode-latent` on +the ABI would have turned this row's 3 h re-render into a 2 min decode. diff --git a/src/vllm/model_executor/models/minimax_h3_pipeline.cpp b/src/vllm/model_executor/models/minimax_h3_pipeline.cpp index 20026fdf3..c1ade8847 100644 --- a/src/vllm/model_executor/models/minimax_h3_pipeline.cpp +++ b/src/vllm/model_executor/models/minimax_h3_pipeline.cpp @@ -479,9 +479,12 @@ MiniMaxH3T2vaResult MiniMaxH3GenerateT2va(vt::Device device, const MiniMaxH3T2va vt::Queue vq = vt::GetBackend(device.type).CreateQueue(); const MiniMaxH3VideoVaeDeviceWeights staged_vae = StageMiniMaxH3VideoVaeWeights(vq, video_config, video_weights); - // Upstream's video path is decode_base -> decode_temporal: chunked in TIME, - // and NOT spatially tiled (decoder_tiling defaults false and the shipped - // config does not set it). + // Upstream's video path is decode_base -> decode_temporal: chunked in TIME. + // decode_temporal then composes SPATIAL tiling per chunk whenever + // `decoder_tiling` is set, which it is by DEFAULT (minimax_h3.h: `bool + // decoder_tiling = true`) -- required, not optional, because the ViT3D's RoPE + // is length-normalized over the grid it is handed. A real canvas therefore + // decodes as 256-px tiles inside each temporal chunk. result.frames = MiniMaxH3VideoVaeDecodeTemporalDevice( device, video_config, staged_vae, video_latent, request.latent_t, request.latent_h, request.latent_w, request.num_frames, &result.frame_shape); From 2cad6cb9ff209e58a2bf34a9deb6af429732fd92 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 22:31:04 +0000 Subject: [PATCH 2/2] fix(minimax-h3): drain the scratch pool at denoise->decode, unblocking the REF canvas The REF-canvas render (1344x768/124f) completed its 50-step denoise and then took the BOX DOWN in VAE decode: NV_ERR_NO_MEMORY from the driver, then `minimax-h3-gen invoked oom-killer`, twice. The scratch pool is UNCAPPED on this platform (cuda.cpp:98, `device_pool_cap_bytes = 0`), so every block a DBuf returns across 50 denoise steps is retained for the life of the process, keyed by size class. The VAE decode that follows allocates DIFFERENT classes, so it can reuse none of them and cudaMallocs on top of the whole retained set. DevicePool::Drain releases the retained blocks to the driver and reports the bytes freed; MiniMaxH3GenerateT2va calls it once at the denoise -> decode boundary. Nothing live is touched -- `free_` only ever holds blocks a DBuf already returned -- and under VT_POOL_BYPASS the free list is empty, so it is a no-op there. Measured on the REF canvas: 10.25 GiB released, the render COMPLETED (124 frames + 32 kHz audio, no reboot), and the pool still served 99.92% of allocations from the free list (hits=98475 misses=75 distinct-classes=8). Draining at a PHASE boundary does not cost the within-phase reuse the pool exists for. Capping the pool platform-wide was REJECTED: it charges every CUDA model the cudaMalloc/cudaFree sync storm the pool was built to avoid, to fix a problem that only appears at one phase change. `--keep-quant` as a no-code workaround was also tried and is not one -- it OOM-killed the box ~8 min in, during weight staging, before a single denoise step. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] --- .agents/specs/minimax-h3.md | 18 ++++++++++-- .../vllm/model_executor/models/device_pool.h | 28 +++++++++++++++++++ .../models/minimax_h3_pipeline.cpp | 19 ++++++++++++- 3 files changed, 62 insertions(+), 3 deletions(-) diff --git a/.agents/specs/minimax-h3.md b/.agents/specs/minimax-h3.md index 1dd411d6a..f188d9564 100644 --- a/.agents/specs/minimax-h3.md +++ b/.agents/specs/minimax-h3.md @@ -1169,8 +1169,22 @@ log, ~1 min after the dump, box down ~2 min later). The dequantised bf16 DiT sta resident through `MiniMaxH3VideoVaeDecodeTemporalDevice`. Decoding the SAME dumped latent standalone — VAE only, no DiT resident — completes with room to spare and produced the 1.15 frames above. So generation is correct at the REF canvas and the DECODE is what does -not fit alongside the model. A driver that frees the DiT before decode, or decodes in -temporal slices, is the fix; not attempted in this row. +not fit alongside the model. + +**FIXED in this row — `DevicePool::Drain` at the phase change.** The scratch pool is +UNCAPPED on this platform (`cuda.cpp:98`, `device_pool_cap_bytes = 0`), so every block a +`DBuf` returns across 50 denoise steps is retained forever, keyed by size class. The VAE +decode then asks for DIFFERENT classes, cannot reuse any of them, and `cudaMalloc`s on +top. `MiniMaxH3GenerateT2va` now drains the pool once at the denoise -> decode boundary. +Measured on the REF canvas: **10.25 GiB released**, the render completed (124 frames + +32 kHz audio, no reboot), and the pool still served **99.92%** of allocations from the +free list (`hits=98475 misses=75 distinct-classes=8`) — draining at a PHASE boundary does +not cost the within-phase reuse the pool exists for. Capping the pool platform-wide was +REJECTED: it would charge every CUDA model the cudaMalloc/cudaFree sync storm the pool was +built to avoid, for a problem that only appears at one phase change. + +`--keep-quant` was tried first as a no-code workaround and is NOT one: it OOM-killed the +box ~8 min in, during weight staging, before a single denoise step. **Residuals.** (1) No full 124-frame MP4 with audio at the REF canvas yet — blocked on the OOM above. (2) The audio latent has no `VT_H3_DUMP_DIR` hook, so the audio arm is diff --git a/include/vllm/model_executor/models/device_pool.h b/include/vllm/model_executor/models/device_pool.h index 511e814c9..d2abbec77 100644 --- a/include/vllm/model_executor/models/device_pool.h +++ b/include/vllm/model_executor/models/device_pool.h @@ -119,6 +119,34 @@ class DevicePool { free_[key].push_back(p); } + // Release every RETAINED block back to the driver, and report the bytes freed. + // + // The pool earns its keep WITHIN a phase, where the same size classes recur + // every step and cudaMalloc/cudaFree would be a sync storm. Across a PHASE + // CHANGE the retained classes are the wrong shapes for what comes next, so on + // an UNCAPPED pool (`device_pool_cap_bytes == 0`, which is GB10/Thor today) + // they are pure headroom loss at exactly the moment the next phase wants its + // own working set. Draining at that boundary costs one cudaFree per retained + // block, once, and is what keeps a big-canvas MiniMax-H3 VAE decode from + // meeting the driver's OOM on top of 50 steps of denoise scratch. + // + // SAFETY: `free_` only ever holds blocks a DBuf already returned, so nothing + // live is touched. Under VT_POOL_BYPASS the free list is always empty (Put + // frees straight through) and this is a no-op. + size_t Drain(vt::Backend& b) { + std::lock_guard lk(mu_); + size_t freed = 0; + for (auto& entry : free_) { + for (void* p : entry.second) { + b.Free(p); + freed += entry.first; + } + } + free_.clear(); + retained_ = 0; + return freed; + } + ~DevicePool() { if (std::getenv("VT_POOL_STATS") != nullptr) { const uint64_t h = hits_.load(), m = misses_.load(); diff --git a/src/vllm/model_executor/models/minimax_h3_pipeline.cpp b/src/vllm/model_executor/models/minimax_h3_pipeline.cpp index c1ade8847..16d33f97a 100644 --- a/src/vllm/model_executor/models/minimax_h3_pipeline.cpp +++ b/src/vllm/model_executor/models/minimax_h3_pipeline.cpp @@ -32,6 +32,7 @@ #include #include +#include "vllm/model_executor/models/device_pool.h" // ActivePool()/DevicePool::Drain #include "vt/backend.h" #include "vt/dtype.h" @@ -476,7 +477,23 @@ MiniMaxH3T2vaResult MiniMaxH3GenerateT2va(vt::Device device, const MiniMaxH3T2va // scalar reference; at real resolutions it is the stage that does not finish. It // stays the CPU path, and stays the thing the device path is gated against. if (device.type != vt::DeviceType::kCPU) { - vt::Queue vq = vt::GetBackend(device.type).CreateQueue(); + vt::Backend& vae_backend = vt::GetBackend(device.type); + // PHASE CHANGE: denoise is done, the VAE decode is next. The denoise left the + // scratch pool holding every activation size class it touched, and on an + // UNCAPPED pool (GB10/Thor: `device_pool_cap_bytes == 0`) those blocks are + // never returned to the driver. The decode allocates DIFFERENT classes, so it + // cannot reuse any of them -- they are pure headroom loss at the one moment + // the decode needs its own working set. At the REF canvas (1344x768/124f) + // that is the difference between a decode that fits and one that takes the + // BOX DOWN: measured 85 GiB resident at this point against a ~18 GiB decode + // in a 122 GiB unified pool, and the driver OOM (NV_ERR_NO_MEMORY) rebooted + // the machine. Draining costs one cudaFree per retained block, once. + const size_t drained = ActivePool()->Drain(vae_backend); + if (std::getenv("VT_POOL_STATS") != nullptr) { + std::fprintf(stderr, "[h3] drained %.2f GiB of denoise scratch before VAE decode\n", + static_cast(drained) / (1024.0 * 1024.0 * 1024.0)); + } + vt::Queue vq = vae_backend.CreateQueue(); const MiniMaxH3VideoVaeDeviceWeights staged_vae = StageMiniMaxH3VideoVaeWeights(vq, video_config, video_weights); // Upstream's video path is decode_base -> decode_temporal: chunked in TIME.