fix(minimax-h3): REF canvas renders COHERENT + drain the scratch pool at denoise→decode - #157
Merged
Merged
Conversation
…easured end to end
§8.4 left the vllm-omni REF canvas (768x1344, 124f) UNRUN on honesty grounds
("a full 50-step render is ~2.85 h, so it was not run"). This row RAN it on Thor
(sm_110, portable fallback attention throughout) and measured every stage.
The §8.4 white latent does NOT reproduce at the REF canvas on the matching
partition. VAE-input latent adjacent-cell cosine 0.8924 (§8.4 white 0.06, real
encode 0.789, §8.6 coherent 0.9467); the init-noise control reads 0.0019, white
as it must. Decoded frames carry a period-16 seam ratio of 1.15, equal to the
known-good 864x480 render.
What the earlier grids were is task/partition, measured on one canvas and prompt:
ref2va-on-FL2VA 2.28, t2va pre-guard 1.87, t2va on the guarded binary with
Gaussian init noise 1.15. The artifact SCALES WITH CANVAS, which is why small
canvases looked acceptable and the REF canvas did not.
NEW BUG recorded: the pipeline's own decode OOMs at the REF canvas. The 50-step
denoise completes and the latent dump lands, then VAE decode exhausts the GPU
pool and reboots the box (NV_ERR_NO_MEMORY in the kernel log). The dequantised
bf16 DiT stays resident across MiniMaxH3VideoVaeDecodeTemporalDevice; decoding
the SAME dumped latent standalone (VAE only) completes at 23 GiB of 122 GiB and
produced the 1.15 frames. Generation is correct there; the decode is what does
not fit alongside the model.
Also corrects a comment that claimed `decoder_tiling` "defaults false" while
minimax_h3.h has `bool decoder_tiling = true`. Tiling composes with temporal
chunking by default and is required, not optional, because the ViT3D's RoPE is
length-normalized over the grid it is handed.
Record-only plus a comment; no behaviour change.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…g the REF canvas The REF-canvas render (1344x768/124f) completed its 50-step denoise and then took the BOX DOWN in VAE decode: NV_ERR_NO_MEMORY from the driver, then `minimax-h3-gen invoked oom-killer`, twice. The scratch pool is UNCAPPED on this platform (cuda.cpp:98, `device_pool_cap_bytes = 0`), so every block a DBuf returns across 50 denoise steps is retained for the life of the process, keyed by size class. The VAE decode that follows allocates DIFFERENT classes, so it can reuse none of them and cudaMallocs on top of the whole retained set. DevicePool::Drain releases the retained blocks to the driver and reports the bytes freed; MiniMaxH3GenerateT2va calls it once at the denoise -> decode boundary. Nothing live is touched -- `free_` only ever holds blocks a DBuf already returned -- and under VT_POOL_BYPASS the free list is empty, so it is a no-op there. Measured on the REF canvas: 10.25 GiB released, the render COMPLETED (124 frames + 32 kHz audio, no reboot), and the pool still served 99.92% of allocations from the free list (hits=98475 misses=75 distinct-classes=8). Draining at a PHASE boundary does not cost the within-phase reuse the pool exists for. Capping the pool platform-wide was REJECTED: it charges every CUDA model the cudaMalloc/cudaFree sync storm the pool was built to avoid, to fix a problem that only appears at one phase change. `--keep-quant` as a no-code workaround was also tried and is not one -- it OOM-killed the box ~8 min in, during weight staging, before a single denoise step. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
This was referenced Aug 8, 2026
mudler
added a commit
that referenced
this pull request
Aug 8, 2026
Landing a PR with `gh pr merge --merge` breaks two gates, and it has now happened three times in a row on main: c3db780 (#172, mine), 3e5072d (#174), and the trailer half on da0a9d8 (#157). check-role-discipline.py inspects every commit in the pushed range and wants each to name its row/<ROW-ID> branch or its PR as (#N). GitHub's squash-merge writes (#N) into the subject for free. A --merge landing leaves the content commit with the subject it had on the branch, which names neither, so POL-PR-REQUIRED reports "reached main without a reviewed row/* PR" about a commit that came from exactly such a PR. check-commit-trailers.py reads the same range, and GitHub's generated "Merge pull request #N from ..." message carries no FOLLOWING_AGENTS_PROTOCOL paragraph and no trailers, so POL-COMMIT-TRAILERS fails on the merge commit itself. Both gates are scoped over github.event.before..github.sha and each run's before is the previous run's sha, so no later run re-covers a range that already went red. That is the part worth writing down: the failure is not repairable after the push except by rewriting published history or waiving it, and it is invisible at PR time because the PR is checked against its own base. Documented in CONTRIBUTING.md rather than .agents/workflow.md because that file is 7 bytes under its 12288-byte procedure budget, and every paragraph of its non-generated prose is load-bearing; buying room by trimming it would cost more than this note is worth. CONTRIBUTING.md is also where a contributor with merge rights actually looks. This is documentation, not a guard. The permanent fix is a repository setting - allow squash only, disable merge commits - which needs admin and is left to the owner. No record row, no capability claim, no code. Every tree-scoped gate on main was already green before this change; main's red runs come entirely from the diff-scoped range gates described above. Gates: check-policy, check-protocol-consistency, check-agent-record, check-readme-structure, check-public-doc-tables, check-role-discipline, check-doc-checkpoint and check-commit-trailers over the range - all OK on a worktree pinned at 80b0880. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
mudler
added a commit
that referenced
this pull request
Aug 8, 2026
test_agent_role.py::test_landed_detached_commit_remains_strict_without_ pending_evidence failed on PR #176 and would fail on many PRs that have nothing to do with what it asserts. The test's subject is main()'s DECISION: a violation on a commit that has landed, with no --pending-pr-head evidence, is strict (return 1) rather than a REPORT (return 0). But it obtained that violation by relying on the REAL HEAD to be one, and that coupling is live under CI. On a pull_request event GitHub checks out the SYNTHETIC merge commit for refs/pull/N/merge. inspect() sees two parents, so arrives_via_row_pr() switches to the merged_messages branch and scans the PR's own commit bodies with PR_REFERENCE = \(#\d+\)|#\d+, which matches a bare #123 anywhere. Any PR whose commit message cites an issue or PR number therefore stops being a violation, main() returns 0, and the assertion fails with 0 != 1. #176's message quotes (#157) and (#174) while explaining these very gates, which is how it tripped. Fixed by feeding main() a fixed violation through inspect/enforced rather than depending on HEAD's message, plus a regression case pinning that a message mentioning (#157) and #174 cannot flip the decision again. Proven on a reconstructed synthetic merge, the exact shape CI checks out (git merge --no-ff of the PR head into the base, detached): RED old test on that checkout: AssertionError: 0 != 1, 41 tests, FAILED (failures=1) -- the same failure CI reported GREEN new test on the SAME checkout: 42 tests, OK Not changed: PR_REFERENCE matching a bare #123 anywhere in a body also means a direct push whose message merely mentions an issue satisfies POL-PR-REQUIRED. That is a real hole, but tightening a policy gate's strictness is a separate reviewed decision, not a CI repair. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 8, 2026
Two fixes for a main branch that has been red for several landings. 1) CONTRIBUTING.md documents the landing rule. Landing a PR with `gh pr merge --merge` breaks two gates, and it happened three times in a row: c3db780 (#172, mine), 3e5072d (#174), and the trailer half on da0a9d8 (#157). check-role-discipline.py inspects every commit in the pushed range and wants each to name its row branch or its PR number, which squash-merge writes into the subject for free and --merge does not. check-commit-trailers.py reads the same range, and GitHub's generated merge-commit message carries no FOLLOWING_AGENTS_PROTOCOL paragraph and no trailers. Both gates are scoped over before..sha, and each run's before is the previous run's sha, so no later run re-covers a range that already went red: the failure is not repairable after the push, and it is invisible at PR time because a PR is checked against its own base. Documented in CONTRIBUTING.md rather than .agents/workflow.md because that file sits 7 bytes under its 12288-byte procedure budget. 2) tests/scripts/test_agent_role.py is unbroken under a pull_request checkout. test_landed_detached_commit_remains_strict_without_pending_ evidence asserts a decision in main() -- a violation on a landed commit with no pending evidence is strict rather than a REPORT -- but it obtained that violation by relying on the real HEAD to be one. Under a pull_request event GitHub checks out the synthetic refs/pull/N/merge commit; inspect() then sees two parents, so arrives_via_row_pr() scans the PR's own commit bodies with a pattern that matches a bare #123 anywhere. Any PR whose message cites an issue or PR number stopped being a violation, main() returned 0, and the test failed 0 != 1. It now feeds main() a fixed violation, with a regression case pinning that a message mentioning a PR number cannot flip the decision again. Proven on a reconstructed synthetic merge, the exact shape CI checks out: RED, the old test on that checkout gave AssertionError: 0 != 1 across 41 tests, the same failure CI reported; GREEN, the new test on the SAME checkout gave 42 tests OK. Confirmed in real CI on this PR: agent-record now passes. Not changed: the reference pattern matching a bare #123 anywhere also means a direct push whose message merely mentions an issue satisfies POL-PR-REQUIRED. That is a real hole, but tightening a policy gate is a separate reviewed decision, not a CI repair. The permanent fix for the merge-method half is a repository setting -- allow squash only, disable merge commits -- which needs admin. Gates: check-agent-record (ENGINE=144), check-role-discipline, check-policy, check-protocol-consistency, check-readme-structure, check-public-doc-tables, test_agent_role 42/42, test_agent_onboard, plus check-commit-trailers and check-doc-checkpoint over the range. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two things, both grounded in a full REF-canvas run (1344x768 / 124f) that §8.4 had left unrun ("~2.85 h, so it was not run").
1. The REF canvas renders COHERENT — measured
The §8.4 white latent does not reproduce at the REF canvas on the matching partition.
The earlier grids were task/partition, measured on one canvas and prompt: ref2va-on-FL2VA 2.28 → t2va pre-guard 1.87 → t2va on the guarded binary with Gaussian init noise 1.15. The artifact scales with canvas, which is why small canvases looked acceptable and the REF canvas did not.
2. The decode OOM — found and fixed
The 50-step denoise completed, then VAE decode took the box down:
NV_ERR_NO_MEMORY, thenminimax-h3-gen invoked oom-killer. Twice.The scratch pool is uncapped on this platform (
cuda.cpp:98,device_pool_cap_bytes = 0), so every block aDBufreturns across 50 denoise steps is retained for the life of the process, keyed by size class. The decode allocates different classes, reuses none of them, andcudaMallocs on top of the whole retained set.DevicePool::Drainreleases them and reports the bytes;MiniMaxH3GenerateT2vacalls it once at the denoise→decode boundary.10.25 GiB released, render completed with audio, no reboot — and the pool still served 99.92% from the free list, so draining at a phase boundary costs nothing of the within-phase reuse the pool exists for. Per-step forward time was unchanged at 176.3 s.
Alternatives rejected
cudaMalloc/cudaFreesync storm the pool was built to avoid, to fix a problem that appears at one phase change.--keep-quantas a no-code workaround — is not one. OOM-killed the box ~8 min in, during weight staging, before a single denoise step.Safety
free_only ever holds blocks aDBufalready returned, so nothing live is touched. UnderVT_POOL_BYPASSthe free list is always empty andDrainis a no-op. The drain is called from the H3 pipeline only; no other model's path changes.Also
Corrects a comment claiming
decoder_tiling"defaults false" whileminimax_h3.hhasbool decoder_tiling = true. Tiling composes with temporal chunking by default and is required, not optional — the ViT3D's RoPE is length-normalized over the grid it is handed. The stale comment cost this row a wrong hypothesis.