fix(ltx2): wire conv_in to correct widths in LTX2VideoUpBlock3d - #14308
fix(ltx2): wire conv_in to correct widths in LTX2VideoUpBlock3d#14308Solaris-star wants to merge 1 commit into
Conversation
Two independent defects in __init__ made any non-nominal decoder_block_out_channels unloadable (huggingface#14307): 1. conv_in projected onto out_channels, but the upsampler that consumes its output expects out_channels * upscale_factor. Fix: project onto out_channels * upscale_factor and gate conv_in creation on in_channels != out_channels * upscale_factor. 2. LTX2VideoDecoder3d passed input_channel = output_channel // upsample_factor[i] to each up block, but the tensor arriving from the previous block is output_channel wide (no division needed). The division produced a fictitious width corresponding to no tensor in the graph. Fix: input_channel = output_channel. Both defects are invisible on released LTX-2 checkpoints because the nominal decoder_block_out_channels happen to make conv_in unnecessary and the division a no-op. They surface immediately with any other decoder width configuration. Adds unit tests covering the conv_in projection, the no-conv_in nominal case, and the no-upsampler case. Fixes huggingface#14307
ErenAta16
left a comment
There was a problem hiding this comment.
Reproduced #14307 and confirmed both halves of the fix on CPU with a tiny AutoencoderKLLTX2Video, plus checked the nominal path is byte-for-byte unchanged.
Three decoder width patterns, latent_channels=4, upsample_factor=(2,2,2), decode() on a (1, 4, 3, 8, 8) latent:
decoder_block_out_channels |
conv_in count |
before | after |
|---|---|---|---|
(16, 32, 64) — nominal, each block 2× the last |
0 | decode OK (1, 3, 17, 64, 64) |
same |
(32, 32, 32) — constant width |
2 | RuntimeError weight of size [16, 8, 3, 3, 3], expected input[1, 16, ...] |
decode OK (1, 3, 17, 64, 64) |
(64, 32, 16) — decreasing |
2 | RuntimeError weight of size [16, 4, 3, 3, 3], expected input[1, 8, ...] |
decode OK (1, 3, 17, 64, 64) |
So the conv_in branch is unreachable on the nominal pattern (0 built) and broken on every other one, exactly as the issue describes.
On the regression risk, since the second hunk changes input_channel for all blocks rather than just the conv_in ones: I diffed every decoder.up_blocks.* parameter shape in the nominal configuration before and after. 18 parameters, all identical. That's the part I'd have wanted checked before merging, and it holds.
The three new tests in tests/models/autoencoders/test_autoencoder_kl_ltx2.py pass (3 passed).
One note on the second hunk. The old input_channel = output_channel // upsample_factor[i] is only equal to the incoming width when block_out_channels[i-1] // upsample_factor[i-1] == block_out_channels[i] // upsample_factor[i] * upsample_factor[i] — i.e. exactly the nominal doubling pattern with a constant factor. Worth saying in the PR body, because it explains why the released checkpoints load: they sit on the one pattern where the wrong expression coincides with the right one.
Measured on main @ 9f6fc2c with the PR applied, torch CPU, Python 3.10.12.
Summary
Fixes #14307
Two independent defects in
LTX2VideoUpBlock3d.__init__made any non-nominaldecoder_block_out_channelsunloadable:1.
conv_inprojected ontoout_channels, but the upsampler expectsout_channels * upscale_factor.The upsampler is built with
in_channels=out_channels * upscale_factor, soconv_inmust project onto that same width. The guard should comparein_channels != out_channels * upscale_factor.2.
LTX2VideoDecoder3ddividedoutput_channelby the current block'supsample_factorto computeinput_channel.The tensor arriving from the previous block is
output_channelwide — no division needed. The division produced a fictitious width corresponding to no tensor in the graph.Both defects are invisible on released LTX-2 checkpoints because the nominal
decoder_block_out_channelshappen to makeconv_inunnecessary and the division a no-op.Tests
Added
tests/models/autoencoders/test_autoencoder_kl_ltx2.pywith 3 tests:test_conv_in_projects_to_upsampler_input_width— non-nominal widths trigger conv_in and forward pass succeedstest_no_conv_in_when_widths_match— nominal case still works without conv_intest_conv_in_with_no_upsampler— no-upsampler case works correctly