diff --git a/.agents/specs/minimax-h3.md b/.agents/specs/minimax-h3.md index 16c6e59c..f188d956 100644 --- a/.agents/specs/minimax-h3.md +++ b/.agents/specs/minimax-h3.md @@ -1129,3 +1129,67 @@ the projections never touch a host buffer), total peak **51.95 GiB** of the 122 UNIFIED pool. Streamer counters on the real checkpoint: `layers=50 tensors=400 direct=350 converted=200 fused=100`, i.e. the shard path ran and every projection took the no-host-copy upload. + +## 8.16 The REF CANVAS renders COHERENT — 1344x768/124f measured end to end, and the pipeline's own decode OOMs there (2026-08-08, `row/H3-REF-CANVAS-RENDER`, Thor sm_110) + +§8.4 left the vllm-omni REF canvas (768x1344, 124f) UNRUN — "a full 50-step render is +~2.85 h, so it was not run (largest-fitting-config honesty)". This row RAN it, on Thor +(sm_110, 122.8 GiB unified, no FA2 so the portable fallback attention throughout), and +measured every stage. **The render is COHERENT at the REF canvas** on the matching +partition; the §8.4 white latent does NOT reproduce. + +**Config:** `MiniMax-H3-FL2VA-Q4_K_M.gguf` (`--dequant-bf16`), `--partition fl2va`, +task **t2va** (no reference image), 1344x768, 124 frames, 50 steps, seq_len 38080, +176.3 s/forward, `VT_H3_DUMP_DIR` on. + +| Stage | Metric | Value | Reference | +|---|---|---|---| +| init noise | adj-cell cosine | **0.0019** | white by construction (control) | +| init noise | r_W / r_H / r_T | 0.999 / 1.001 / 0.999 | 1.0 == spatially white | +| VAE-input latent | adj-cell cosine | **0.8924** | §8.4 white 0.06; real encode 0.789; §8.6 coherent 0.9467 | +| VAE-input latent | r_W / r_H / r_T | 0.347 / 0.313 / 0.311 | well below the 1.0 white floor | +| decoded frames | period-16 seam ratio | **1.15** | == the known-good 864x480 render (1.15); ref2va-on-FL2VA was 2.28 | + +`r` is mean|adjacent diff| / E|X-Y| along one axis: 1.0 for spatially white, lower when +correlated. Calibrated before use on synthetic latents (white 1.00, smooth field 0.44, +smooth+30% per-cell noise 0.86), so the instrument is known to separate the two cases. + +**Task/partition is what the earlier grids were.** Renders passing `--ref-image` against +the FL2VA partition are ref2va-on-FL2VA, the combination §8.6 identified and §8.7's guard +now REJECTS at the CLI. Measured on the same canvas and prompt: ref2va-on-FL2VA **2.28**, +t2va pre-guard binary **1.87**, t2va on the guarded binary with Gaussian init noise +**1.15**. The artifact SCALES WITH CANVAS (864x480 showed 1.15 while 1344x768 showed 2.28 +on the same mismatch), which is why small-canvas runs looked acceptable and the REF canvas +did not. + +**NEW BUG — the pipeline decode OOMs at the REF canvas.** The 50-step denoise completed +and the latent dump was written; VAE decode then exhausted the GPU pool and REBOOTED the +box (`NVRM: GPU0 ... Out of memory [NV_ERR_NO_MEMORY] ... _memdescAllocInternal`, kernel +log, ~1 min after the dump, box down ~2 min later). The dequantised bf16 DiT stays +resident through `MiniMaxH3VideoVaeDecodeTemporalDevice`. Decoding the SAME dumped latent +standalone — VAE only, no DiT resident — completes with room to spare and produced the +1.15 frames above. So generation is correct at the REF canvas and the DECODE is what does +not fit alongside the model. + +**FIXED in this row — `DevicePool::Drain` at the phase change.** The scratch pool is +UNCAPPED on this platform (`cuda.cpp:98`, `device_pool_cap_bytes = 0`), so every block a +`DBuf` returns across 50 denoise steps is retained forever, keyed by size class. The VAE +decode then asks for DIFFERENT classes, cannot reuse any of them, and `cudaMalloc`s on +top. `MiniMaxH3GenerateT2va` now drains the pool once at the denoise -> decode boundary. +Measured on the REF canvas: **10.25 GiB released**, the render completed (124 frames + +32 kHz audio, no reboot), and the pool still served **99.92%** of allocations from the +free list (`hits=98475 misses=75 distinct-classes=8`) — draining at a PHASE boundary does +not cost the within-phase reuse the pool exists for. Capping the pool platform-wide was +REJECTED: it would charge every CUDA model the cudaMalloc/cudaFree sync storm the pool was +built to avoid, for a problem that only appears at one phase change. + +`--keep-quant` was tried first as a no-code workaround and is NOT one: it OOM-killed the +box ~8 min in, during weight staging, before a single denoise step. + +**Residuals.** (1) No full 124-frame MP4 with audio at the REF canvas yet — blocked on the +OOM above. (2) The audio latent has no `VT_H3_DUMP_DIR` hook, so the audio arm is +unmeasured; it was diagnosed only through the shared-sequence argument. (3) `--roundtrip` +and `--decode-latent` went with the pre-fold binary (documented at +`minimax_h3_gen/main.cpp:31-36`), so replaying a dumped latent through the VAE now needs a +throwaway harness against `MiniMaxH3VideoVaeDecodeTemporalDevice`; a `--decode-latent` on +the ABI would have turned this row's 3 h re-render into a 2 min decode. diff --git a/include/vllm/model_executor/models/device_pool.h b/include/vllm/model_executor/models/device_pool.h index 511e814c..d2abbec7 100644 --- a/include/vllm/model_executor/models/device_pool.h +++ b/include/vllm/model_executor/models/device_pool.h @@ -119,6 +119,34 @@ class DevicePool { free_[key].push_back(p); } + // Release every RETAINED block back to the driver, and report the bytes freed. + // + // The pool earns its keep WITHIN a phase, where the same size classes recur + // every step and cudaMalloc/cudaFree would be a sync storm. Across a PHASE + // CHANGE the retained classes are the wrong shapes for what comes next, so on + // an UNCAPPED pool (`device_pool_cap_bytes == 0`, which is GB10/Thor today) + // they are pure headroom loss at exactly the moment the next phase wants its + // own working set. Draining at that boundary costs one cudaFree per retained + // block, once, and is what keeps a big-canvas MiniMax-H3 VAE decode from + // meeting the driver's OOM on top of 50 steps of denoise scratch. + // + // SAFETY: `free_` only ever holds blocks a DBuf already returned, so nothing + // live is touched. Under VT_POOL_BYPASS the free list is always empty (Put + // frees straight through) and this is a no-op. + size_t Drain(vt::Backend& b) { + std::lock_guard lk(mu_); + size_t freed = 0; + for (auto& entry : free_) { + for (void* p : entry.second) { + b.Free(p); + freed += entry.first; + } + } + free_.clear(); + retained_ = 0; + return freed; + } + ~DevicePool() { if (std::getenv("VT_POOL_STATS") != nullptr) { const uint64_t h = hits_.load(), m = misses_.load(); diff --git a/src/vllm/model_executor/models/minimax_h3_pipeline.cpp b/src/vllm/model_executor/models/minimax_h3_pipeline.cpp index 20026fdf..16d33f97 100644 --- a/src/vllm/model_executor/models/minimax_h3_pipeline.cpp +++ b/src/vllm/model_executor/models/minimax_h3_pipeline.cpp @@ -32,6 +32,7 @@ #include #include +#include "vllm/model_executor/models/device_pool.h" // ActivePool()/DevicePool::Drain #include "vt/backend.h" #include "vt/dtype.h" @@ -476,12 +477,31 @@ MiniMaxH3T2vaResult MiniMaxH3GenerateT2va(vt::Device device, const MiniMaxH3T2va // scalar reference; at real resolutions it is the stage that does not finish. It // stays the CPU path, and stays the thing the device path is gated against. if (device.type != vt::DeviceType::kCPU) { - vt::Queue vq = vt::GetBackend(device.type).CreateQueue(); + vt::Backend& vae_backend = vt::GetBackend(device.type); + // PHASE CHANGE: denoise is done, the VAE decode is next. The denoise left the + // scratch pool holding every activation size class it touched, and on an + // UNCAPPED pool (GB10/Thor: `device_pool_cap_bytes == 0`) those blocks are + // never returned to the driver. The decode allocates DIFFERENT classes, so it + // cannot reuse any of them -- they are pure headroom loss at the one moment + // the decode needs its own working set. At the REF canvas (1344x768/124f) + // that is the difference between a decode that fits and one that takes the + // BOX DOWN: measured 85 GiB resident at this point against a ~18 GiB decode + // in a 122 GiB unified pool, and the driver OOM (NV_ERR_NO_MEMORY) rebooted + // the machine. Draining costs one cudaFree per retained block, once. + const size_t drained = ActivePool()->Drain(vae_backend); + if (std::getenv("VT_POOL_STATS") != nullptr) { + std::fprintf(stderr, "[h3] drained %.2f GiB of denoise scratch before VAE decode\n", + static_cast(drained) / (1024.0 * 1024.0 * 1024.0)); + } + vt::Queue vq = vae_backend.CreateQueue(); const MiniMaxH3VideoVaeDeviceWeights staged_vae = StageMiniMaxH3VideoVaeWeights(vq, video_config, video_weights); - // Upstream's video path is decode_base -> decode_temporal: chunked in TIME, - // and NOT spatially tiled (decoder_tiling defaults false and the shipped - // config does not set it). + // Upstream's video path is decode_base -> decode_temporal: chunked in TIME. + // decode_temporal then composes SPATIAL tiling per chunk whenever + // `decoder_tiling` is set, which it is by DEFAULT (minimax_h3.h: `bool + // decoder_tiling = true`) -- required, not optional, because the ViT3D's RoPE + // is length-normalized over the grid it is handed. A real canvas therefore + // decodes as 256-px tiles inside each temporal chunk. result.frames = MiniMaxH3VideoVaeDecodeTemporalDevice( device, video_config, staged_vae, video_latent, request.latent_t, request.latent_h, request.latent_w, request.num_frames, &result.frame_shape);