From f8f8e6bf3043599ab872ec92ba21c76caaacfb2f Mon Sep 17 00:00:00 2001 From: mabry1985 Date: Mon, 13 Jul 2026 07:46:30 +0000 Subject: [PATCH 1/6] =?UTF-8?q?ltx2-nvfp4:=20LTX-2.3-22B=20distilled=20?= =?UTF-8?q?=E2=86=92=20NVFP4=20(shipped=20to=20HF)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Quantize LTX-2.3-22B-distilled to NVFP4 mixed-precision for Blackwell/sm120. Lightricks ships fp4 only for the 19B; 2.3-22B is bf16-only — this fills the gap. Converter (quantize.py) streams bf16 through comfy_kitchen's own NVFP4 layout, replicating Lightricks' 19B layer policy (fp4 transformer_blocks 1-42; keep block 0 + last 5 + gates + VAE/vocoder bf16) and stamping the _quantization_metadata header ComfyUI's loader needs. Measured (960x544, reproduced 2x): 46.1→22.9 GB, 2.85→1.82 s/it (1.57x), ~60→~37 GB peak VRAM. Ship distilled-decode (full-decode shows mild fp4 artifacting). Card/BLOG/LICENSE + honest numbers. Live: https://huggingface.co/protoLabsAI/LTX-2.3-22B-distilled-NVFP4 Co-Authored-By: Claude Opus 4.8 (1M context) --- experiments/ltx2-nvfp4/BLOG.md | 68 ++++ experiments/ltx2-nvfp4/LICENSE | 349 ++++++++++++++++++ experiments/ltx2-nvfp4/README.md | 117 ++++++ .../ltx2-nvfp4/measurements-multires.json | 58 +++ experiments/ltx2-nvfp4/quantize.py | 91 +++++ 5 files changed, 683 insertions(+) create mode 100644 experiments/ltx2-nvfp4/BLOG.md create mode 100644 experiments/ltx2-nvfp4/LICENSE create mode 100644 experiments/ltx2-nvfp4/README.md create mode 100644 experiments/ltx2-nvfp4/measurements-multires.json create mode 100644 experiments/ltx2-nvfp4/quantize.py diff --git a/experiments/ltx2-nvfp4/BLOG.md b/experiments/ltx2-nvfp4/BLOG.md new file mode 100644 index 0000000..8176dec --- /dev/null +++ b/experiments/ltx2-nvfp4/BLOG.md @@ -0,0 +1,68 @@ +# Quantizing LTX-2.3 22B to NVFP4 on Blackwell (and the cu130 trap everyone hits) + +Lightricks ships an fp4 checkpoint for **LTX-2 19B**, but the newer, better **LTX-2.3 22B is bf16 only**. If you +have a Blackwell card (RTX PRO 6000, RTX 50-series) that means leaving the 4-bit TensorCores idle on your best +video model. So we quantized it ourselves — and reverse-engineered Lightricks' exact mixed-precision policy in +the process. + +## The result + +A 22.9 GB NVFP4 checkpoint (down from 46.1 GB), running the DiT at **1.57× the speed of bf16** and **−38% peak +VRAM**, with the distilled decode path **visually indistinguishable** from bf16. + +| 960×544, 8-step distilled | bf16 | NVFP4 | Δ | +|---|---|---|---| +| Disk | 46.1 GB | 22.9 GB | 2.0× | +| DiT step | 2.85 s/it | 1.82 s/it | 1.57× | +| Peak VRAM | ~60 GB | ~37 GB | −38% | +| Cold load + first clip | 39.2 s | 19.0 s | 2.1× | + +(Reproduced across 2 runs each. Speedup is on the DiT denoising loop; the bf16 VAE decode is unchanged, so +whole-pipeline gain is smaller on very short clips.) + +## The trap: cu130 or bust + +The single biggest source of "NVFP4 is slower than fp8" confusion in the LTX community is the CUDA version. +NVFP4 only hits the fast path on **torch built with CUDA 13**. On cu128 it silently falls back to a slow path +and runs **~2× slower than fp8** — ComfyUI even warns you (`You need pytorch with cu130 or higher to use +optimized CUDA operations`) but it's easy to miss. Same sm120/cu130 story we hit on the vLLM side. One +force-reinstall fixes it: + +```bash +pip install --force-reinstall --no-cache-dir torch==2.11.0 torchvision torchaudio \ + --index-url https://download.pytorch.org/whl/cu130 +``` + +## The method: mirror Lightricks, don't guess + +There's no turnkey "quantize an LTX checkpoint" node. But `comfy_kitchen` (shipped with ComfyUI-LTXVideo) +exposes the exact primitive the official fp4 was built with — `TensorCoreNVFP4Layout.quantize`. The trick is +matching two things so ComfyUI's loader recognizes the result: + +1. **The tensor format.** Each fp4 weight is three tensors: `W` (uint8 packed), `W_scale` (fp8-e4m3 per-block), + `W_scale_2` (fp32 per-tensor). Using `comfy_kitchen`'s own serializer guarantees this. +2. **The `_quantization_metadata` header.** This is the actual load trigger. Without it the loader treats the + packed uint8 as a regular weight and dies on a shape mismatch (`[4096,2048]` vs `[4096,4096]`). It's a JSON + map of `{module_path: {"format": "nvfp4"}}` in the safetensors header. + +And **which** layers to quantize? We diffed the shipped 19B fp4 against its bf16 and recovered the policy: fp4 +the transformer-block Linears, but **keep block 0, the last 5 blocks, all gating projections, and the entire +VAE/vocoder in bf16**. Classic first/last-layer precision retention — the bf16 blocks absorb the ~9% +per-tensor error of the 4-bit middle. We applied the identical policy to the 22B. + +## The honest caveat: distilled decode only + +The 2.3 workflow has two decode branches. On the fp4 model, the **distilled decode is clean**, but the **full +decode shows mild added artifacting**. Not surprising — the DiT is where the 4-bit lives, and the full VAE +decoder is more sensitive to it. Default to distilled decode and it's a clean win. + +## Reproduce + +The converter is ~100 lines, runs in ~40 s on one Blackwell card, and is in +[`protoLabsAI/lab`](https://github.com/protoLabsAI) under `experiments/ltx2-nvfp4/`. The checkpoint is on +[HuggingFace](https://huggingface.co/protoLabsAI). + +--- + +*Part of the protoLabs quant + serving lab — parity-verified low-bit models and the Blackwell serving findings +that make them run. LTX-2.3 · NVFP4 · sm120 · cu130.* diff --git a/experiments/ltx2-nvfp4/LICENSE b/experiments/ltx2-nvfp4/LICENSE new file mode 100644 index 0000000..82c330b --- /dev/null +++ b/experiments/ltx2-nvfp4/LICENSE @@ -0,0 +1,349 @@ + LTX-2 Community License Agreement + License date: January 5, 2026 + + +By using or distributing any portion or element of LTX-2, you agree +to be bound by this Agreement. + + 1. Definitions. + + "Agreement" means the terms and conditions for the license, use, + reproduction, and distribution of LTX-2 and the Complementary + Materials, as specified in this document. + + "Control" means the direct or indirect ownership of fifty percent + (50%) or more of the voting securities or other ownership + interests, or the power to direct the management and policies of + such Entity through voting rights, contract, or otherwise. + + "Data" means a collection of information and/or content extracted + from the dataset used with LTX-2, including to train, pretrain, + or otherwise evaluate LTX-2. The Data is not licensed under this + Agreement. + + "Derivatives of LTX-2" means all modifications to LTX-2, works + based on LTX-2, or any other model which is created or initialized + by transfer of patterns of the weights, parameters, activations or + output of LTX-2, to the other model, in order to cause the other + model to perform similarly to LTX-2, including – but not limited + to - distillation methods entailing the use of intermediate data + representations or methods based on the generation of synthetic + data by LTX-2 for training the other model. For clarity, Derivatives + of LTX-2 include: (i) any fine-tuned or adapted weights, parameters, + or checkpoints derived from LTX-2; (ii) derivative model architectures + that incorporate or are based upon LTX-2's architecture; and + (iii) any modified or extended versions of the Complementary + Materials. All intellectual property rights in Derivatives of LTX-2 + shall be subject to the terms of this Agreement, and you may not + claim exclusive ownership rights in any Derivatives of LTX-2 that + would restrict the rights granted herein. + "Entity" means any individual, corporation, partnership, limited + liability company, or other legal entity. For purposes of this + Agreement, an Entity shall be deemed to include, on an aggregative + basis, all subsidiaries, affiliates, and other companies under + common Control with such Entity. When determining whether an Entity + meets any threshold under this Agreement (including revenue + thresholds), all subsidiaries, affiliates, and companies under + common Control shall be considered collectively. + "Harm" includes but is not limited to physical, mental, + psychological, financial and reputational damage, pain, or loss. + "Licensor" or "Lightricks" means the owner that is granting the + license under this Agreement. For the purposes of this Agreement, + the Licensor is Lightricks Ltd. + "LTX-2" means the large language models, text/image/video/audio/3D + generation models, and multimodal large language models and their + software and algorithms, including trained model weights, parameters + (including optimizer states), machine-learning model code, + inference-enabling code, training-enabling code, fine-tuning + enabling code, accompanying source code, scripts, documentation, + tutorials, examples, and all other elements of the foregoing + distributed and made publicly available by Lightricks (including, + for example, at https://github.com/Lightricks/LTX-2) for the LTX-2 + model released on January 5, 2026. This license is applicable to + all LTX-2 versions released since January 5, 2026, and all future + releases of LTX-2 under this license. + "Output" means the results of operating LTX-2 as embodied in + informational content resulting therefrom. + "you" (or "your") means an individual or legal Entity licensing + LTX-2 in accordance with this Agreement and/or making use of LTX-2 + for whichever purpose and in any field of use, including usage of + LTX-2 in an end-use application - e.g. chatbot, translator, image + generator. + 2. Grant of License. Subject to the terms and conditions of this + Agreement, you are granted a non-exclusive, worldwide, + non-transferable and royalty-free limited license under Licensor's + intellectual property or other rights owned by Licensor embodied + in LTX-2 to use, reproduce, prepare, distribute, publicly display, + publicly perform, sublicense, copy, create derivative works of, + and make modifications to LTX-2, for any purpose, subject to the + restrictions set forth in Attachment A; provided however, that + Entities with annual revenues of at least $10,000,000 (the + "Commercial Entities") are required to obtain a paid commercial + use license in order to use LTX-2 and Derivatives of LTX-2, + subject to the terms and provisions of a different license (the + "Commercial Use Agreement"), as will be provided by the Licensor. + Commercial Entities interested in such a commercial license are + required to contact Licensor. Any commercial use of LTX-2 or + Derivatives of LTX-2 by the Commercial Entities not in accordance + with this Agreement and/or the Commercial Use Agreement is strictly + prohibited and shall be deemed a material breach of this Agreement. + Such material breach will be subject, in addition to any license + fees owed to Licensor for the period such Commercial Entity used + LTX-2 (as will be determined by Licensor), to liquidated damages, + which will be paid to Licensor immediately upon demand, in an + amount equal to double the amount that would otherwise have been + paid by you for the relevant period of time. Such amount reflects + a reasonable estimation of the losses and administrative costs + incurred due to such breach. You agree and understand that this + remedy does not limit the Licensor's right to pursue other remedies + available at law or equity. + 3. Distribution and Redistribution. You may host for third parties + remote access purposes (e.g. software-as-a-service), reproduce + and distribute copies of LTX-2 or Derivatives of LTX-2 thereof in + any medium, with or without modifications, provided that you meet + the following conditions: + (a) Use-based restrictions as referenced in paragraph 4 and all + provisions of Attachment A MUST be included as an enforceable + provision by you in any type of legal agreement (e.g. a + license) governing the use and/or distribution of LTX-2 or + Derivatives of LTX-2, and you shall give notice to subsequent + users you distribute to, that LTX-2 or Derivatives of LTX-2 + are subject to paragraph 4 and Attachment A in their entirety, + including all use restrictions and acceptable use policies; + (b) You must provide any third party recipients of LTX-2 or + Derivatives of LTX-2 a copy of this Agreement, including all + attachments and use policies. Any Derivative of LTX-2 (as + defined in Section 1, including but not limited to fine-tuned + weights, modified training code, models trained on Outputs, or + any other derivative) must be distributed exclusively under + the terms of this Agreement with a complete copy of this + license included; + (c) You must cause any modified files to carry prominent notices + stating that you changed the files; + (d) You must retain all copyright, patent, trademark, and + attribution notices excluding those notices that do not + pertain to any part of LTX-2, Derivatives of LTX-2. + You may add your own copyright statement to your modifications and + may provide additional or different license terms and conditions - + respecting paragraph 3(a) - for use, reproduction, or distribution + of your modifications, or for any such Derivatives of LTX-2 as a + whole, provided your use, reproduction, and distribution of LTX-2 + otherwise complies with the conditions stated in this Agreement, + and you provide a complete copy of this Agreement with any such + use, reproduction and distribution of LTX-2 and any Derivatives + thereof. + 4. Use-based restrictions. The restrictions set forth in Attachment A + are considered Use-based restrictions. Therefore, you cannot use + LTX-2 and the Derivatives of LTX-2 in violation of the specified + restricted uses. You may use LTX-2 subject to this Agreement, + including only for lawful purposes and in accordance with the + Agreement. "Use" may include creating any content with, fine-tuning, + updating, running, training, evaluating and/or re-parametrizing + LTX-2. You shall require all of your users who use LTX-2 or a + Derivative of LTX-2 to comply with the terms of this paragraph 4. + 5. The Output You Generate. Except as set forth herein, Licensor + claims no rights in the Output you generate using LTX-2. You are + accountable for input you insert into LTX-2, the Output you + generate and its subsequent uses. No use of the Output can + contravene any provision as stated in the Agreement. + 6. Updates and Runtime Restrictions. To the maximum extent permitted + by law, Licensor reserves the right to restrict (remotely or + otherwise) usage of LTX-2 in violation of this Agreement, update + LTX-2 through electronic means, or modify the Output of LTX-2 + based on updates. You shall undertake reasonable efforts to use + the latest version of LTX-2. Any use of the non-current version + of LTX-2 is done solely at your risk. + 7. Export Controls and Sanctions Compliance. You acknowledge that + LTX-2, Derivatives of LTX-2 may be subject to export control laws + and regulations, including but not limited to the U.S. Export + Administration Regulations and sanctions programs administered by + the Office of Foreign Assets Control (OFAC). You represent and + warrant that you and any users of LTX-2 are not (i) located in, + organized under the laws of, or ordinarily resident in any country + or territory subject to comprehensive sanctions; (ii) identified + on any U.S. government restricted party list, including the + Specially Designated Nationals and Blocked Persons List; or + (iii) otherwise prohibited from receiving LTX-2 under applicable + law. You shall not export, re-export, or transfer LTX-2, directly + or indirectly, in violation of any applicable export control or + sanctions laws or regulations. You agree to comply with all + applicable trade control laws and shall indemnify and hold + Licensor harmless from any claims arising from your failure to + comply with such laws. + 8. Trademarks and related. Nothing in this Agreement permits you to + make use of Licensor's trademarks, trade names, logos or to + otherwise suggest endorsement or misrepresent the relationship + between the parties; and any rights not expressly granted herein + are reserved by the Licensor. + + 9. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides LTX-2 on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or + conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS + FOR A PARTICULAR PURPOSE. You are solely responsible for + determining the appropriateness of using or redistributing LTX-2 + and Derivatives of LTX-2 and assume any risks associated with + your exercise of permissions under this Agreement. + + 10. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall Licensor be liable + to you for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as + a result of this Agreement or out of the use or inability to use + LTX-2 (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if Licensor has been + advised of the possibility of such damages. + + 11. Accepting Warranty or Additional Liability. While redistributing + LTX-2 and Derivatives of LTX-2, you may, provided you do not + violate the terms of this Agreement, choose to offer and charge + a fee for, acceptance of support, warranty, indemnity, or other + liability obligations. However, in accepting such obligations, + you may act only on your own behalf and on your sole + responsibility, not on behalf of Licensor, and only if you agree + to indemnify, defend, and hold Licensor harmless for any liability + incurred by, or claims asserted against Licensor, by reason of + your accepting any such warranty or additional liability. + + 12. Governing Law. This Agreement and all relations, disputes, claims + and other matters arising hereunder (including non-contractual + disputes or claims) will be governed exclusively by, and construed + exclusively in accordance with, the laws of the State of New York. + To the extent permitted by law, choice of laws rules and the + United Nations Convention on Contracts for the International Sale + of Goods will not apply. For the purposes of adjudicating any + action or proceeding to enforce the terms of this Agreement, you + hereby irrevocably consent to the exclusive jurisdiction of, and + venue in, the federal and state courts located in the County of + New York within the State of New York. The prevailing party in + any claim or dispute between the parties under this Agreement + will be entitled to reimbursement of its reasonable attorneys' + fees and costs. You hereby waive the right to a trial by jury, + to participate in a class or representative action (including in + arbitration), or to combine individual proceedings in court or + in arbitration without the consent of all parties. + 13. Term and Termination. This Agreement is effective upon your + acceptance and continues until terminated. Licensor may terminate + this Agreement immediately upon written notice to you if you + breach any provision of this Agreement, including but not limited + to violations of the use restrictions in Attachment A or + unauthorized commercial use. Upon termination: (a) all rights + granted to you under this Agreement will immediately cease; + (b) you must immediately cease all use of LTX-2 and Derivatives + of LTX-2; (c) you must delete or destroy all copies of LTX-2 + and Derivatives of LTX-2 in your possession or control; and + (d) you must notify any third parties to whom you distributed + LTX-2 or Derivatives of LTX-2 of the termination. Sections 8-13, + and Section 15 shall survive termination of this Agreement. + Termination does not relieve you of any obligations incurred + prior to termination, including payment obligations under + Section 2. In addition, if You commence a lawsuit or other + proceedings (including a cross-claim or counterclaim in a lawsuit) + against Licensor or any person or entity alleging that LTX-2 or + any Output, or any portion of any of the foregoing, infringe any + intellectual property or other right owned or licensable by you, + then all licenses granted to you under this Agreement shall + terminate as of the date such lawsuit or other proceeding is filed. + 14. Disputes and Arbitration. All disputes arising in connection with + this Agreement shall be finally settled by arbitration under the + Rules of Arbitration of the International Chamber of Commerce + ("ICC Rules"), by one (1) arbitrator appointed in accordance with + the ICC Rules. The seat of arbitration shall be New York, NY, USA, + and the proceedings shall be conducted in English. The arbitrator + shall be empowered to grant any relief that a court could grant. + Judgment on the arbitration award may be entered by any court + having jurisdiction thereof. Each party waives its right to a + trial by jury and to participate in any class or representative + action. + 15. If any provision of this Agreement is held to be + invalid, illegal + or unenforceable, the remaining provisions shall be unaffected + thereby and remain valid as if such provision had not been set + forth herein. + END OF TERMS AND CONDITIONS + ATTACHMENT A: Use Restrictions + When using the Outputs, LTX-2 and any Derivatives thereof, you + will comply with the Acceptable Use Policy. In addition, you + agree not to use the Outputs, LTX-2 or its Derivatives in any + of the following ways: + 1. In any way that violates any applicable national, federal, + state, local or international law or regulation; + 2. For the purpose of exploiting, Harming or attempting to + exploit or Harm minors in any way; + 3. To generate or disseminate false information and/or content + with the purpose of Harming others; + 4. To generate or disseminate personal identifiable information + that can be used to Harm an individual; + 5. To generate or disseminate information and/or content (e.g. + images, code, posts, articles), and place the information + and/or content in any context (e.g. bot generating tweets) + without expressly and intelligibly disclaiming that the + information and/or content is machine generated; + 6. To defame, disparage or otherwise harass others; + 7. To impersonate or attempt to impersonate (e.g. deepfakes) + others without their consent; + 8. For fully automated decision making that adversely impacts an + individual's legal rights or otherwise creates or modifies a + binding, enforceable obligation; + + 9. For any use intended to or which has the effect of + discriminating against or Harming individuals or groups based + on online or offline social behavior or known or predicted + personal or personality characteristics; + + 10. To exploit any of the vulnerabilities of a specific group of + persons based on their age, social, physical or mental + characteristics, in order to materially distort the behavior + of a person pertaining to that group in a manner that causes + or is likely to cause that person or another person physical + or psychological Harm; + + 11. For any use intended to or which has the effect of + discriminating against individuals or groups based on legally + protected characteristics or categories; + + 12. To provide medical advice and medical results interpretation; + + 13. To generate or disseminate information for the purpose to be + used for administration of justice, law enforcement, + immigration or asylum processes, such as predicting an + individual will commit fraud/crime commitment (e.g. by text + profiling, drawing causal relationships between assertions + made in documents, indiscriminate and arbitrarily-targeted use); + + 14. To generate and/or disseminate malware (including – but not + limited to – ransomware) or any other content to be used for + the purpose of harming electronic systems; + + 15. To engage in, promote, incite, or facilitate discrimination + or other unlawful or harmful conduct in the provision of + employment, employment benefits, credit, housing, or other + essential goods and services; + + 16. To engage in, promote, incite, or facilitate the harassment, + abuse, threatening, or bullying of individuals or groups of + individuals; + + 17. For military, warfare, nuclear industries or applications, + weapons development, or any use in connection with activities + that may cause death, personal injury, or severe physical or + environmental damage; + + 18. For commercial use only: To train, improve, or fine-tune any + other machine learning model, artificial intelligence system, + or competing model, except for Derivatives of LTX-2 as + expressly permitted under this Agreement; + + 19. To circumvent, disable, or interfere with any technical + limitations, safety features, content filters, or use + restrictions implemented in LTX-2 by Licensor; + + 20. To use LTX-2 or Derivatives of LTX-2 in any product, service, + or application that directly competes with Licensor's + commercial products or services, or is designed to replace or + substitute Licensor's offerings in the market, without + obtaining a separate commercial license from Licensor. \ No newline at end of file diff --git a/experiments/ltx2-nvfp4/README.md b/experiments/ltx2-nvfp4/README.md new file mode 100644 index 0000000..cc155fd --- /dev/null +++ b/experiments/ltx2-nvfp4/README.md @@ -0,0 +1,117 @@ +--- +license: other +license_name: ltx-2-community-license +license_link: LICENSE +base_model: Lightricks/LTX-2.3 +tags: + - text-to-video + - image-to-video + - video-generation + - nvfp4 + - fp4 + - quantization + - blackwell + - comfyui + - ltx-video +library_name: comfyui +pipeline_tag: text-to-video +--- + +# LTX-2.3-22B-distilled — NVFP4 (mixed precision) + +A **native NVFP4** (4-bit) quantization of [`Lightricks/LTX-2.3`](https://huggingface.co/Lightricks/LTX-2.3) +`22b-distilled-1.1`, for **NVIDIA Blackwell** (RTX PRO 6000 / RTX 50-series, sm120) via ComfyUI. + +Lightricks ships an fp4 checkpoint for **LTX-2 19B** but the newer **LTX-2.3 22B is bf16-only**. This fills +that gap: the 22B distilled model at ~half the disk and VRAM, running on Blackwell's native FP4 TensorCores. + +## Results (measured, honest) + +Single-clip, `960×544`, 8-step distilled sampler, distilled decode, RTX PRO 6000 (96 GB), vLLM-adjacent +ComfyUI stack on **torch cu130**. Speed/VRAM reproduced across 2 runs each; quality eyeballed across 4 prompts. + +| Metric | bf16 (upstream) | **NVFP4 (this)** | Δ | +|---|---|---|---| +| Disk size | 46.1 GB | **22.9 GB** | **2.0× smaller** | +| DiT step speed | 2.85 s/it | **1.82 s/it** | **1.57× faster** | +| Peak runtime VRAM | ~60 GB | **~37 GB** | **−38%** | +| Cold load + first clip | 39.2 s | **19.0 s** | **2.1× faster** | +| Distilled-decode quality | baseline | **visually unchanged** | — | + +Higher resolution holds the win (fp4 `1280×704` ≈ 3.85 s/it, scaling with token count as expected). + +## ⚠️ You need torch built with CUDA 13 (cu130) + +This is the single most important thing. NVFP4 only hits the fast TensorCore path on **cu130**. On **cu128, +fp4 silently falls back and runs ~2× *slower* than fp8** (ComfyUI logs `WARNING: You need pytorch with cu130 or +higher to use optimized CUDA operations`). This is the field's #1 LTX-fp4 confusion. If your fp4 is slower than +fp8, this is why. + +```bash +pip install --force-reinstall --no-cache-dir torch==2.11.0 torchvision torchaudio \ + --index-url https://download.pytorch.org/whl/cu130 +``` + +## ⚠️ Use the distilled decode path + +The LTX-2.3 single-stage workflow ships **two decode branches** (a fast "distilled" decode and a "full" +decode). On this quant, the **distilled decode is visually indistinguishable from bf16**, while the **full +decode shows mild added artifacting**. Default to the distilled decode. (The DiT is where the fp4 lives; the +VAE decoders are untouched bf16 — the full decoder is just more sensitive to the small DiT-linear error.) + +## Usage (ComfyUI) + +1. cu130 torch (above) + [`ComfyUI-LTXVideo`](https://github.com/Lightricks/ComfyUI-LTXVideo) + (provides the `comfy_kitchen` NVFP4 kernels: `scaled_mm_nvfp4`, `quantize_nvfp4`). +2. Drop `ltx-2.3-22b-distilled-1.1-fp4.safetensors` into `ComfyUI/models/checkpoints/`. +3. Load the `2.3/LTX-2.3_T2V_I2V_Single_Stage_Distilled_Full` example workflow, point the + `CheckpointLoaderSimple` at this file, keep the **distilled** decode/save branch. +4. Text encoder: any Gemma-3-12B LTX encoder (`gemma_3_12B_it_fp8_scaled.safetensors` works well). + +The checkpoint carries a standard `_quantization_metadata` header, so ComfyUI's LTX loader auto-detects the +NVFP4 layers — no flags, no config edits. + +## What's quantized (mixed precision) + +Replicates Lightricks' own 19B-fp4 layer policy exactly (reverse-engineered from their shipped checkpoint): + +- **NVFP4 (4-bit):** the 2-D Linear weights in `transformer_blocks` **1–42** (attention q/k/v/out + feed-forward, + video + audio + cross-modal). 1,176 weights. +- **Kept bf16 (precision-sensitive):** transformer block **0** and the **last 5 blocks (43–47)**, all + `to_gate_logits` gates, the patchify / proj_out / adaln / caption + embeddings connectors, and the entire + VAE / audio-VAE / vocoder. + +Each fp4 weight is stored as `W` (uint8 packed) + `W_scale` (fp8-e4m3 per-block) + `W_scale_2` (fp32 +per-tensor) — byte-identical to the shipped 19B fp4 format. + +## Reproduce it + +The full converter is `quantize.py` (in this repo / `protoLabsAI/lab`). It streams the bf16 checkpoint +tensor-by-tensor through `comfy_kitchen`'s own `TensorCoreNVFP4Layout.quantize`, so the output format is +guaranteed loader-compatible. ~40 s on one Blackwell card. + +```bash +CUDA_VISIBLE_DEVICES=0 python quantize.py \ + --in ltx-2.3-22b-distilled-1.1.safetensors \ + --out ltx-2.3-22b-distilled-1.1-fp4.safetensors +``` + +## Limitations & honest notes + +- Numbers are **single-clip** on one specific Blackwell + cu130 ComfyUI config; your mileage varies with + resolution, length, sampler, and card. +- Speedup is on the **DiT denoising** loop. End-to-end wall-clock also includes VAE decode (bf16, unchanged) + and text encoding, so the *whole-pipeline* ratio is smaller than 1.57× at short clip lengths. +- **Full-decode** path: mild artifacting (see above). Distilled decode only. +- FP4 is 4-bit — expect a small per-tensor error (~9% relative L1 on a quantized layer), which the first/last + bf16 blocks are there to absorb. If you see quality loss, widen the bf16-kept block set and re-quantize. + +## Attribution & license + +Derivative of [`Lightricks/LTX-2.3`](https://huggingface.co/Lightricks/LTX-2.3), released under the **LTX-2 +Community License** (see `LICENSE`). This quant is redistributed under the same license, including its +Acceptable Use Policy (Attachment A) and paragraph 4 use restrictions, which pass through to you. High-revenue +commercial entities require a separate Commercial Use Agreement from Lightricks for *use* — see the license. + +Quantized by [protoLabsAI](https://huggingface.co/protoLabsAI). Method + measurements: +`protoLabsAI/lab` → `experiments/ltx2-nvfp4/`. diff --git a/experiments/ltx2-nvfp4/measurements-multires.json b/experiments/ltx2-nvfp4/measurements-multires.json new file mode 100644 index 0000000..e4bbafd --- /dev/null +++ b/experiments/ltx2-nvfp4/measurements-multires.json @@ -0,0 +1,58 @@ +[ + { + "label": "fp4_768x512", + "wall_s": 13.5, + "peak_vram_mb": 41069, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00003_.mp4" + }, + { + "label": "fp4_1280x704", + "wall_s": 34.5, + "peak_vram_mb": 43757, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00004_.mp4" + }, + { + "label": "bf16_768x512", + "wall_s": 33.2, + "peak_vram_mb": 63341, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00005_.mp4" + }, + { + "label": "bf16_1280x704", + "wall_s": 48.3, + "peak_vram_mb": 65869, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00006_.mp4" + }, + { + "label": "fp4_dog", + "wall_s": 48.0, + "peak_vram_mb": 59597, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00007_.mp4" + }, + { + "label": "fp4_tokyo", + "wall_s": 36.3, + "peak_vram_mb": 43661, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00008_.mp4" + }, + { + "label": "fp4_bread", + "wall_s": 37.5, + "peak_vram_mb": 43661, + "staged_mb": null, + "best_s_per_it": null, + "out": "output_D_00009_.mp4" + } +] \ No newline at end of file diff --git a/experiments/ltx2-nvfp4/quantize.py b/experiments/ltx2-nvfp4/quantize.py new file mode 100644 index 0000000..85ed9f6 --- /dev/null +++ b/experiments/ltx2-nvfp4/quantize.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""Quantize an LTX-2.3 bf16 checkpoint to NVFP4 (fp4_mixed), replicating Lightricks' +exact layer-selection policy reverse-engineered from the shipped 19B fp4 checkpoint. + +Policy (matches ltx-2-19b-dev-fp4.safetensors): + - Quantize ONLY 2D Linear weights in model.diffusion_model.transformer_blocks + - KEEP block 0 and the last 5 blocks (43..47) in bf16 [first/last precision retention] + - KEEP all to_gate_logits gates in bf16 [tiny, gating-sensitive] + - KEEP everything else in bf16: patchify/proj_out, adaln, caption/embeddings + connectors, VAE, audio_vae, vocoder, and all non-2D tensors. + +Each quantized weight K becomes three keys (comfy_kitchen serialization, verified +identical to the shipped fp4): K (uint8 packed fp4), K_scale (fp8_e4m3 per-block), +K_scale_2 (fp32 per-tensor). ComfyUI's LTX loader auto-detects fp4 from these sidecars. + +Run with ComfyUI's venv (has comfy_kitchen + torch cu130): + CUDA_VISIBLE_DEVICES=1 ~/dev/ComfyUI/venv/bin/python ltx23_nvfp4_quantize.py \ + --in /mnt/data/models-cold/LTX-2.3/ltx-2.3-22b-distilled-1.1.safetensors \ + --out /mnt/data/models-cold/LTX-2.3/ltx-2.3-22b-distilled-1.1-fp4.safetensors +""" +import argparse, re, time, torch +from safetensors import safe_open +from safetensors.torch import save_file +from comfy_kitchen.tensor.nvfp4 import TensorCoreNVFP4Layout as NVFP4 + +KEEP_BLOCKS = {0, 43, 44, 45, 46, 47} # bf16 (matches 19B: block 0 + last 5) +BLOCK_RE = re.compile(r"model\.diffusion_model\.transformer_blocks\.(\d+)\.") + +def is_quant_target(key: str, shape) -> bool: + if not key.endswith(".weight"): + return False + if len(shape) != 2: + return False + m = BLOCK_RE.match(key) + if not m: # only the main transformer_blocks + return False + if int(m.group(1)) in KEEP_BLOCKS: + return False + if "to_gate_logits" in key: # keep gates high-precision + return False + return True + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--in", dest="src", required=True) + ap.add_argument("--out", dest="dst", required=True) + ap.add_argument("--limit", type=int, default=0, help="quantize only first N targets (dry test)") + args = ap.parse_args() + + assert torch.cuda.is_available(), "need CUDA for ck.quantize_nvfp4" + t0 = time.time() + out = {} + quant_layers = {} # module_path -> {"format":"nvfp4"} + n_q = n_kept = n_qbytes = 0 + with safe_open(args.src, framework="pt", device="cpu") as f: + src_meta = f.metadata() or {} + keys = list(f.keys()) + targets = [k for k in keys if is_quant_target(k, f.get_slice(k).get_shape())] + if args.limit: + targets = targets[:args.limit] + tset = set(targets) + print(f"{len(keys)} tensors total | {len(targets)} → NVFP4 | {len(keys)-len(targets)} → bf16 passthrough") + for i, k in enumerate(keys): + t = f.get_tensor(k) + if k in tset: + qdata, params = NVFP4.quantize(t.to("cuda")) + sd = NVFP4.state_dict_tensors(qdata, params) # {"":qdata,"_scale":bs,"_scale_2":s} + for suf, qt in sd.items(): + out[k + suf] = qt.cpu().contiguous() + n_qbytes += out[k + suf].numel() * out[k + suf].element_size() + quant_layers[k[:-len(".weight")]] = {"format": "nvfp4"} # module path, no .weight + n_q += 1 + if n_q % 200 == 0: + print(f" [{n_q}/{len(targets)}] quantized … {time.time()-t0:.0f}s", flush=True) + else: + out[k] = t.contiguous() + n_kept += 1 + del t + # Header the ComfyUI LTX loader reads to build NVFP4 layers (the load trigger). + import json as _json + meta = {k: v for k, v in src_meta.items() if k != "encrypted_wandb_properties"} + meta["_quantization_metadata"] = _json.dumps({"format_version": "1.0", "layers": quant_layers}) + meta["quant_source"] = args.src.split("/")[-1] + print(f"quantized {n_q} weights (~{n_qbytes/1e9:.1f} GB fp4), kept {n_kept} bf16 tensors") + print(f"_quantization_metadata: {len(quant_layers)} layers tagged nvfp4") + print(f"writing {args.dst} …") + save_file(out, args.dst, metadata=meta) + print(f"DONE in {time.time()-t0:.0f}s") + +if __name__ == "__main__": + main() From d9e26b9ddfb39e61be9ff35bacf312f8b60044a7 Mon Sep 17 00:00:00 2001 From: mabry1985 Date: Mon, 13 Jul 2026 07:46:30 +0000 Subject: [PATCH 2/6] infra/video-bridge: OpenAI /v1/videos over ComfyUI+LTX (protoBanana#38 piece 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Standalone video bridge co-located with ComfyUI on protolabs — the shape the team settled on (litellm's native /v1/videos router shadows passthrough routes, so the job layer can't live in the gateway). Implements the AGREED 3-URL contract: POST /v1/videos, GET /v1/videos/{id}, GET /v1/videos/{id}/content. Reuses protoBanana's ComfyUIClient (inherits the #39 cache-nonce fix); protoBanana stays image-only. Restart-survivable JSON-backed job store (job_id→prompt_id, status re-derived from ComfyUI /history); mp4 served off local disk; LTX knobs in extra_body. Piece 1 = parameterized distilled-decode T2V workflow (workflows/ ltx2-t2v.json). Verified end-to-end: POST→poll→206KB mp4 in ~14s. Stubbed/next: I2V (input_reference accepted, needs I2V template variant), fine progress via ws, edge route in homelab-iac. Co-Authored-By: Claude Opus 4.8 (1M context) --- infra/video-bridge/README.md | 55 ++ infra/video-bridge/bridge.py | 218 ++++++++ infra/video-bridge/inject.py | 71 +++ infra/video-bridge/requirements.txt | 5 + infra/video-bridge/smoke.py | 31 ++ infra/video-bridge/workflows/ltx2-t2v.json | 590 +++++++++++++++++++++ 6 files changed, 970 insertions(+) create mode 100644 infra/video-bridge/README.md create mode 100644 infra/video-bridge/bridge.py create mode 100644 infra/video-bridge/inject.py create mode 100644 infra/video-bridge/requirements.txt create mode 100644 infra/video-bridge/smoke.py create mode 100644 infra/video-bridge/workflows/ltx2-t2v.json diff --git a/infra/video-bridge/README.md b/infra/video-bridge/README.md new file mode 100644 index 0000000..8e381d5 --- /dev/null +++ b/infra/video-bridge/README.md @@ -0,0 +1,55 @@ +# protoLabs video bridge (stub) + +OpenAI `/v1/videos` async-jobs API over ComfyUI + **LTX-2.3-22B NVFP4**, co-located with +ComfyUI on `protolabs:8188`. This is **piece 2** of [protoBanana#38](https://github.com/protoLabsAI/protoBanana/issues/38) — +the standalone video bridge the team settled on (litellm's native `/v1/videos` router shadows +passthrough routes, so the job layer can't live inside the gateway). protoBanana stays +image-only; this depends on it (reuses `ComfyUIClient`, inherits the #39 cache-nonce fix). + +## Contract (AGREED — `protoDirector/GATEWAY_CONTRACT.md`) + +``` +POST /v1/videos {model, prompt, seconds, size, negative_prompt?, seed?, extra_body?} + (+ optional multipart input_reference for first-frame/I2V) + -> 201 {id, object:"video", model, status:"queued", progress, created_at} +GET /v1/videos/{id} -> {id, status: queued|in_progress|completed|failed, progress, error?} +GET /v1/videos/{id}/content -> mp4 bytes (video/mp4) +``` + +- **Job id survives restarts** — `job_id -> prompt_id` persisted to `JOB_STORE`; status is + re-derived from ComfyUI `/history` (also persistent), so a client resumes polling after a + bridge crash/redeploy. +- **Bytes off local disk** — `/content` reads ComfyUI's output dir directly; `/view` fallback. +- **LTX knobs in `extra_body`** — `fps`, `seed`, `negative_prompt` (model-agnostic surface stays clean). + +## Run + +```bash +pip install -r requirements.txt +uvicorn bridge:app --host 0.0.0.0 --port 8100 # needs ComfyUI up on :8188 +# smoke: +python smoke.py +``` + +Env: `COMFY_URL` (`http://127.0.0.1:8188`), `COMFY_OUTPUT_DIR` (`/mnt/data/ltx-out`), +`JOB_STORE` (`/mnt/data/ltx-out/video-bridge/jobs.json`), `MODEL_PREFIX` (`protolabs/ltx2`). + +## Edge routing (homelab-iac task) + +Route `/v1/videos*` → this bridge, everything else → the gateway unchanged. Client contract +holds exactly as written (same base URL, key, shapes). When `sora-2`/`veo-*` become real, the +bridge dispatches by `model`: `protolabs/ltx2-*` local, anything else proxied to the gateway's +native `/v1/videos`. + +## What's stubbed / next + +- **`seconds` → frames** snaps to LTX's `8n+1` at `fps` (default 30). Verify the mapping matches + intended clip length on real requests. +- **`input_reference` (I2V)** is accepted + uploaded, but the template is **T2V-only**; wiring + first-frame conditioning needs the I2V workflow variant (un-bypass the "Load Image" group). +- **Fine progress** — currently coarse (queued 0 / in_progress null / completed 100). ComfyUI's + `/ws` gives per-step progress; wire it for a real 0-100. +- **Concurrency** — single ComfyUI instance serializes; `n=1` only (per contract v1). + +Workflow: `workflows/ltx2-t2v.json` (distilled-decode path, Full fork removed). Injection map +in `inject.py`. diff --git a/infra/video-bridge/bridge.py b/infra/video-bridge/bridge.py new file mode 100644 index 0000000..0a1da05 --- /dev/null +++ b/infra/video-bridge/bridge.py @@ -0,0 +1,218 @@ +"""protoLabs video bridge — OpenAI /v1/videos over ComfyUI (LTX-2.3 NVFP4). + +Co-located with ComfyUI on protolabs:8188 (issue protoBanana#38, piece 2). Implements +exactly the three AGREED contract URLs so protoDirector's video runner + any `sora-2`/ +`veo-*` client hit `protolabs/ltx2-*` identically: + + POST /v1/videos {model, prompt, seconds, size, ...} -> {id, status:"queued"} + GET /v1/videos/{id} -> {id, status, progress, error?} + GET /v1/videos/{id}/content -> mp4 bytes + +Job store is JSON-file-backed (restart-survivable job ids, #38 hard req): the job->prompt_id +map persists, and status is re-derived from ComfyUI's /history (which also survives a bridge +restart), so polling resumes cleanly after a crash/redeploy. mp4 bytes are read off ComfyUI's +output dir on local disk — no multi-hundred-MB clips hauled through an extra hop. + +Reuses protoBanana's ComfyUIClient (the #39 cache-nonce fix comes along for free). protoBanana +itself stays image-only; this is a separate service that depends on it. + +Run: uvicorn bridge:app --host 0.0.0.0 --port 8100 +Env: COMFY_URL (default http://127.0.0.1:8188), COMFY_OUTPUT_DIR (/mnt/data/ltx-out), + JOB_STORE (/mnt/data/ltx-out/video-bridge/jobs.json), MODEL_PREFIX (protolabs/ltx2) +""" +from __future__ import annotations +import os, sys, json, time, uuid, asyncio, pathlib +from typing import Any, Optional +from fastapi import FastAPI, HTTPException, UploadFile, File, Form, Request +from fastapi.responses import JSONResponse, Response + +# co-located protoBanana checkout provides the reusable ComfyUI client +sys.path.insert(0, os.path.expanduser("~/dev/protoBanana")) +from protobanana.client import ComfyUIClient # noqa: E402 + +import inject # noqa: E402 + +COMFY_URL = os.environ.get("COMFY_URL", "http://127.0.0.1:8188") +OUTPUT_DIR = pathlib.Path(os.environ.get("COMFY_OUTPUT_DIR", "/mnt/data/ltx-out")) +JOB_STORE = pathlib.Path(os.environ.get("JOB_STORE", "/mnt/data/ltx-out/video-bridge/jobs.json")) +MODEL_PREFIX = os.environ.get("MODEL_PREFIX", "protolabs/ltx2") + +app = FastAPI(title="protoLabs video bridge", version="0.1-stub") +_client: Optional[ComfyUIClient] = None +_lock = asyncio.Lock() + + +# ---- job store (JSON-file-backed; restart-survivable) -------------------------- +def _load_jobs() -> dict[str, Any]: + if JOB_STORE.exists(): + try: + return json.loads(JOB_STORE.read_text()) + except Exception: + return {} + return {} + + +def _save_jobs(jobs: dict[str, Any]) -> None: + JOB_STORE.parent.mkdir(parents=True, exist_ok=True) + tmp = JOB_STORE.with_suffix(".tmp") + tmp.write_text(json.dumps(jobs, indent=1)) + tmp.replace(JOB_STORE) # atomic + + +async def _put(job: dict[str, Any]) -> None: + async with _lock: + jobs = _load_jobs() + jobs[job["id"]] = job + _save_jobs(jobs) + + +def _get(job_id: str) -> Optional[dict[str, Any]]: + return _load_jobs().get(job_id) + + +@app.on_event("startup") +async def _startup() -> None: + global _client + _client = ComfyUIClient(COMFY_URL, poll_interval_s=1.0, default_timeout_s=600.0) + + +@app.on_event("shutdown") +async def _shutdown() -> None: + if _client: + await _client.aclose() + + +# ---- status derivation (re-derived from ComfyUI, so it survives bridge restart) -- +async def _derive(job: dict[str, Any]) -> dict[str, Any]: + """Refresh a job's status/progress/out_file from ComfyUI /history + /queue.""" + if job["status"] in ("completed", "failed"): + return job + pid = job["prompt_id"] + http = _client.http + hist = (await http.get(f"{COMFY_URL}/history/{pid}")).json() + entry = hist.get(pid) + if entry: + st = entry.get("status", {}) + if st.get("completed"): + if st.get("status_str") == "error": + job.update(status="failed", progress=None, + error={"message": str(st.get("messages"))[:500]}) + else: + job.update(status="completed", progress=100, + out_file=_find_output(entry)) + await _put(job) + return job + # not done yet — queued vs in_progress from /queue + q = (await http.get(f"{COMFY_URL}/queue")).json() + running = any(pid == item[1] for item in q.get("queue_running", [])) + job["status"] = "in_progress" if running else "queued" + job["progress"] = None if running else 0 + return job + + +def _find_output(entry: dict[str, Any]) -> Optional[dict[str, str]]: + for _nid, out in entry.get("outputs", {}).items(): + for key in ("videos", "gifs", "images"): + for f in out.get(key) or []: + return {"filename": f["filename"], "subfolder": f.get("subfolder", ""), + "type": f.get("type", "output")} + return None + + +# ---- the three contract endpoints ----------------------------------------------- +@app.post("/v1/videos") +async def create_video(request: Request, + input_reference: Optional[UploadFile] = File(None), + model: Optional[str] = Form(None), + prompt: Optional[str] = Form(None)): + # accept both JSON and multipart (input_reference forces multipart) + if input_reference is None and request.headers.get("content-type", "").startswith("application/json"): + body = await request.json() + else: + body = {"model": model, "prompt": prompt} + prompt = body.get("prompt") + if not prompt: + raise HTTPException(400, "prompt is required") + model = body.get("model") or f"{MODEL_PREFIX}-distilled" + extra = body.get("extra_body") or {} + + # NOTE (stub): input_reference (first-frame / I2V) is accepted and uploaded, but the + # current template is T2V-only. Wiring it needs the I2V workflow variant (un-bypass the + # "Load Image" group). Tracked as the next bridge task. + ref_name = None + if input_reference is not None: + ref_name = await _client.upload_image(await input_reference.read(), + filename=input_reference.filename or "ref.png") + + wf = inject.build_workflow( + prompt, + size=body.get("size"), + seconds=body.get("seconds"), + seed=body.get("seed"), + negative_prompt=body.get("negative_prompt"), + extra_body=extra, + ) + try: + pid = await _client.submit_prompt(wf) + except Exception as e: + raise HTTPException(502, f"ComfyUI submit failed: {e}") + + job = {"id": f"vid_{uuid.uuid4().hex[:24]}", "prompt_id": pid, "model": model, + "status": "queued", "progress": 0, "created_at": int(time.time()), + "params": {"size": body.get("size"), "seconds": body.get("seconds"), + "input_reference": ref_name}, "out_file": None, "error": None} + await _put(job) + return JSONResponse(_public(job), status_code=201) + + +@app.get("/v1/videos/{job_id}") +async def get_video(job_id: str): + job = _get(job_id) + if not job: + raise HTTPException(404, "unknown job id") + job = await _derive(job) + return _public(job) + + +@app.get("/v1/videos/{job_id}/content") +async def get_content(job_id: str): + job = _get(job_id) + if not job: + raise HTTPException(404, "unknown job id") + job = await _derive(job) + if job["status"] != "completed": + raise HTTPException(409, f"job not completed (status={job['status']})") + out = job.get("out_file") + if not out: + raise HTTPException(500, "completed but no output file recorded") + # read off local disk (co-located); fall back to ComfyUI /view + path = OUTPUT_DIR / out["subfolder"] / out["filename"] + if path.exists(): + data = path.read_bytes() + else: + r = await _client.http.get(f"{COMFY_URL}/view", + params={"filename": out["filename"], + "subfolder": out["subfolder"], "type": out["type"]}) + r.raise_for_status() + data = r.content + return Response(content=data, media_type="video/mp4", + headers={"Content-Disposition": f'inline; filename="{out["filename"]}"'}) + + +def _public(job: dict[str, Any]) -> dict[str, Any]: + """The client-facing shape (hide prompt_id / internal fields).""" + out = {"id": job["id"], "object": "video", "model": job["model"], + "status": job["status"], "progress": job.get("progress"), + "created_at": job["created_at"]} + if job.get("error"): + out["error"] = job["error"] + return out + + +@app.get("/healthz") +async def healthz(): + try: + await _client.http.get(f"{COMFY_URL}/system_stats") + return {"ok": True, "comfy": COMFY_URL} + except Exception as e: + raise HTTPException(503, f"comfy unreachable: {e}") diff --git a/infra/video-bridge/inject.py b/infra/video-bridge/inject.py new file mode 100644 index 0000000..56e3d37 --- /dev/null +++ b/infra/video-bridge/inject.py @@ -0,0 +1,71 @@ +"""Inject OpenAI /v1/videos request params into the LTX-2.3 T2V ComfyUI workflow. + +Injection map (ltx2-t2v.json, distilled-decode path): + prompt -> node 2483 CLIPTextEncode.text (positive) + negative_prompt -> node 2612 CLIPTextEncode.text (negative) + size WxH -> node 3059 EmptyLTXVLatentVideo.width/height + seconds -> node 4979 PrimitiveInt.value (frames, snapped to 8n+1) + seed -> node 4832 RandomNoise.noise_seed +LTX-specific knobs (fps, etc.) arrive via extra_body and are applied where present. +""" +from __future__ import annotations +import copy, json, os +from typing import Any + +_TEMPLATE_PATH = os.path.join(os.path.dirname(__file__), "workflows", "ltx2-t2v.json") + +# node ids in ltx2-t2v.json (ComfyUI API format) +N_POS, N_NEG, N_LATENT, N_FRAMES, N_NOISE = "2483", "2612", "3059", "4979", "4832" +DEFAULT_FPS = 30 +DEFAULT_NEG = "pc game, console game, video game, cartoon, childish, ugly" + + +def load_template() -> dict[str, Any]: + return json.load(open(_TEMPLATE_PATH)) + + +def snap_frames(seconds: float, fps: int = DEFAULT_FPS) -> int: + """LTX latent length must be 8n+1. Map seconds*fps to the nearest valid count.""" + raw = max(1, round(float(seconds) * fps)) + n = round((raw - 1) / 8) + return max(9, n * 8 + 1) + + +def parse_size(size: str | None) -> tuple[int, int]: + if not size: + return 1280, 704 + w, h = size.lower().split("x") + return int(w), int(h) + + +def build_workflow( + prompt: str, + *, + size: str | None = None, + seconds: float | str | None = None, + seed: int | None = None, + negative_prompt: str | None = None, + extra_body: dict[str, Any] | None = None, +) -> dict[str, Any]: + extra = extra_body or {} + wf = copy.deepcopy(load_template()) + fps = int(extra.get("fps", DEFAULT_FPS)) + + wf[N_POS]["inputs"]["text"] = prompt + wf[N_NEG]["inputs"]["text"] = negative_prompt or extra.get("negative_prompt") or DEFAULT_NEG + + w, h = parse_size(size) + wf[N_LATENT]["inputs"]["width"] = w + wf[N_LATENT]["inputs"]["height"] = h + + if seconds is not None: + wf[N_FRAMES]["inputs"]["value"] = snap_frames(float(seconds), fps) + + # seed: explicit > extra_body.seed > leave template default. + # NOTE: protoBanana PR #39's per-submission SaveImage nonce defeats ComfyUI's exec + # cache server-side, so a fixed/absent seed no longer returns a stale cached clip. + s = seed if seed is not None else extra.get("seed") + if s is not None: + wf[N_NOISE]["inputs"]["noise_seed"] = int(s) + + return wf diff --git a/infra/video-bridge/requirements.txt b/infra/video-bridge/requirements.txt new file mode 100644 index 0000000..cb31318 --- /dev/null +++ b/infra/video-bridge/requirements.txt @@ -0,0 +1,5 @@ +fastapi>=0.115 +uvicorn[standard]>=0.30 +httpx>=0.27 +python-multipart>=0.0.9 +# + protoBanana on PYTHONPATH (co-located checkout) for protobanana.client.ComfyUIClient diff --git a/infra/video-bridge/smoke.py b/infra/video-bridge/smoke.py new file mode 100644 index 0000000..0840345 --- /dev/null +++ b/infra/video-bridge/smoke.py @@ -0,0 +1,31 @@ +"""End-to-end smoke: POST a job, poll to completion, fetch mp4 bytes.""" +import time, sys, httpx +BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8100" +c = httpx.Client(timeout=30.0) + +r = c.post(f"{BASE}/v1/videos", json={ + "model": "protolabs/ltx2-distilled", + "prompt": "a red fox trotting through fresh snow at dawn, breath visible, cinematic, soft light", + "seconds": 2, "size": "768x512", "seed": 123, +}) +print("POST", r.status_code, r.json()) +r.raise_for_status() +jid = r.json()["id"] + +t0 = time.time() +while True: + time.sleep(2) + s = c.get(f"{BASE}/v1/videos/{jid}").json() + print(f" [{time.time()-t0:4.0f}s] status={s['status']} progress={s.get('progress')}") + if s["status"] in ("completed", "failed"): + break + if time.time() - t0 > 300: + print("TIMEOUT"); sys.exit(1) + +if s["status"] == "failed": + print("FAILED:", s.get("error")); sys.exit(1) + +r = c.get(f"{BASE}/v1/videos/{jid}/content") +print("GET /content:", r.status_code, r.headers.get("content-type"), len(r.content), "bytes") +open("/mnt/data/ltx-out/bridge_smoke.mp4", "wb").write(r.content) +print("saved /mnt/data/ltx-out/bridge_smoke.mp4") diff --git a/infra/video-bridge/workflows/ltx2-t2v.json b/infra/video-bridge/workflows/ltx2-t2v.json new file mode 100644 index 0000000..4979707 --- /dev/null +++ b/infra/video-bridge/workflows/ltx2-t2v.json @@ -0,0 +1,590 @@ +{ + "1241": { + "inputs": { + "frame_rate": [ + "4978", + 0 + ], + "positive": [ + "2483", + 0 + ], + "negative": [ + "2612", + 0 + ] + }, + "class_type": "LTXVConditioning", + "_meta": { + "title": "LTXVConditioning" + } + }, + "2004": { + "inputs": { + "image": "example.png" + }, + "class_type": "LoadImage", + "_meta": { + "title": "Load Image" + } + }, + "2483": { + "inputs": { + "text": "{{PROMPT}}", + "clip": [ + "4960", + 0 + ] + }, + "class_type": "CLIPTextEncode", + "_meta": { + "title": "CLIP Text Encode (Positive Prompt)" + } + }, + "2612": { + "inputs": { + "text": "{{NEGATIVE}}", + "clip": [ + "4960", + 0 + ] + }, + "class_type": "CLIPTextEncode", + "_meta": { + "title": "CLIP Text Encode (Negative Prompt)" + } + }, + "3059": { + "inputs": { + "width": 1280, + "height": 704, + "length": [ + "4979", + 0 + ], + "batch_size": 1 + }, + "class_type": "EmptyLTXVLatentVideo", + "_meta": { + "title": "EmptyLTXVLatentVideo" + } + }, + "3159": { + "inputs": { + "strength": 0.7, + "bypass": [ + "4977", + 0 + ], + "vae": [ + "3940", + 2 + ], + "image": [ + "3336", + 0 + ], + "latent": [ + "3059", + 0 + ] + }, + "class_type": "LTXVImgToVideoConditionOnly", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 LTXV Img To Video Condition Only" + } + }, + "3336": { + "inputs": { + "img_compression": 18, + "image": [ + "4981", + 0 + ] + }, + "class_type": "LTXVPreprocess", + "_meta": { + "title": "LTXV Preprocess" + } + }, + "3940": { + "inputs": { + "ckpt_name": "ltx-2.3-22b-distilled-1.1-fp4.safetensors" + }, + "class_type": "CheckpointLoaderSimple", + "_meta": { + "title": "Load Checkpoint" + } + }, + "3980": { + "inputs": { + "frames_number": [ + "4979", + 0 + ], + "frame_rate": [ + "4985", + 0 + ], + "batch_size": 1, + "audio_vae": [ + "4010", + 0 + ] + }, + "class_type": "LTXVEmptyLatentAudio", + "_meta": { + "title": "LTXV Empty Latent Audio" + } + }, + "4010": { + "inputs": { + "ckpt_name": "ltx-2.3-22b-distilled-1.1-fp4.safetensors" + }, + "class_type": "LTXVAudioVAELoader", + "_meta": { + "title": "Load LTXV Audio VAE" + } + }, + "4528": { + "inputs": { + "video_latent": [ + "3159", + 0 + ], + "audio_latent": [ + "3980", + 0 + ] + }, + "class_type": "LTXVConcatAVLatent", + "_meta": { + "title": "LTXVConcatAVLatent" + } + }, + "4802": { + "inputs": { + "noise": [ + "4814", + 0 + ], + "guider": [ + "4808", + 0 + ], + "sampler": [ + "4967", + 0 + ], + "sigmas": [ + "4966", + 0 + ], + "latent_image": [ + "4528", + 0 + ] + }, + "class_type": "SamplerCustomAdvanced", + "_meta": { + "title": "SamplerCustomAdvanced" + } + }, + "4808": { + "inputs": { + "skip_blocks": "28", + "model": [ + "3940", + 0 + ], + "positive": [ + "1241", + 0 + ], + "negative": [ + "1241", + 1 + ], + "parameters": [ + "4964", + 0 + ] + }, + "class_type": "MultimodalGuider", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 Multimodal Guider" + } + }, + "4814": { + "inputs": { + "noise_seed": 42 + }, + "class_type": "RandomNoise", + "_meta": { + "title": "RandomNoise" + } + }, + "4818": { + "inputs": { + "samples": [ + "4824", + 1 + ], + "audio_vae": [ + "4010", + 0 + ] + }, + "class_type": "LTXVAudioVAEDecode", + "_meta": { + "title": "LTXV Audio VAE Decode" + } + }, + "4819": { + "inputs": { + "fps": [ + "4978", + 0 + ], + "bit_depth": 8, + "images": [ + "4983", + 0 + ], + "audio": [ + "4818", + 0 + ] + }, + "class_type": "CreateVideo", + "_meta": { + "title": "Create Video" + } + }, + "4824": { + "inputs": { + "av_latent": [ + "4802", + 0 + ] + }, + "class_type": "LTXVSeparateAVLatent", + "_meta": { + "title": "LTXVSeparateAVLatent" + } + }, + "4828": { + "inputs": { + "cfg": 1.0, + "model": [ + "3940", + 0 + ], + "positive": [ + "1241", + 0 + ], + "negative": [ + "1241", + 1 + ] + }, + "class_type": "CFGGuider", + "_meta": { + "title": "CFG Guider" + } + }, + "4829": { + "inputs": { + "noise": [ + "4832", + 0 + ], + "guider": [ + "4828", + 0 + ], + "sampler": [ + "4831", + 0 + ], + "sigmas": [ + "4971", + 0 + ], + "latent_image": [ + "4528", + 0 + ] + }, + "class_type": "SamplerCustomAdvanced", + "_meta": { + "title": "SamplerCustomAdvanced" + } + }, + "4831": { + "inputs": { + "sampler_name": "euler_ancestral_cfg_pp" + }, + "class_type": "KSamplerSelect", + "_meta": { + "title": "KSamplerSelect" + } + }, + "4832": { + "inputs": { + "noise_seed": 0 + }, + "class_type": "RandomNoise", + "_meta": { + "title": "RandomNoise" + } + }, + "4845": { + "inputs": { + "av_latent": [ + "4829", + 0 + ] + }, + "class_type": "LTXVSeparateAVLatent", + "_meta": { + "title": "LTXVSeparateAVLatent" + } + }, + "4848": { + "inputs": { + "samples": [ + "4845", + 1 + ], + "audio_vae": [ + "4010", + 0 + ] + }, + "class_type": "LTXVAudioVAEDecode", + "_meta": { + "title": "LTXV Audio VAE Decode" + } + }, + "4849": { + "inputs": { + "fps": [ + "4978", + 0 + ], + "bit_depth": 8, + "images": [ + "4982", + 0 + ], + "audio": [ + "4848", + 0 + ] + }, + "class_type": "CreateVideo", + "_meta": { + "title": "Create Video" + } + }, + "4852": { + "inputs": { + "filename_prefix": "output_D", + "format": "auto", + "codec": "auto", + "video": [ + "4849", + 0 + ] + }, + "class_type": "SaveVideo", + "_meta": { + "title": "Save Video" + } + }, + "4960": { + "inputs": { + "text_encoder": "gemma_3_12B_it_fp8_scaled.safetensors", + "ckpt_name": "ltx-2.3-22b-distilled-1.1-fp4.safetensors", + "device": "default" + }, + "class_type": "LTXAVTextEncoderLoader", + "_meta": { + "title": "LTXV Audio Text Encoder Loader" + } + }, + "4963": { + "inputs": { + "modality": "AUDIO", + "cfg": 7.0, + "stg": 1.0, + "perturb_attn": true, + "rescale": 0.7, + "modality_scale": 3.0, + "skip_step": 0, + "cross_attn": true + }, + "class_type": "GuiderParameters", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 Guider Parameters" + } + }, + "4964": { + "inputs": { + "modality": "VIDEO", + "cfg": 3.0, + "stg": 1.0, + "perturb_attn": true, + "rescale": 0.9, + "modality_scale": 3.0, + "skip_step": 0, + "cross_attn": true, + "parameters": [ + "4963", + 0 + ] + }, + "class_type": "GuiderParameters", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 Guider Parameters" + } + }, + "4966": { + "inputs": { + "steps": 15, + "max_shift": 2.05, + "base_shift": 0.95, + "stretch": true, + "terminal": 0.1, + "latent": [ + "4528", + 0 + ] + }, + "class_type": "LTXVScheduler", + "_meta": { + "title": "LTXVScheduler" + } + }, + "4967": { + "inputs": { + "eta": 0.25, + "sampler_name": "exponential/res_2s", + "seed": 94, + "bongmath": true + }, + "class_type": "ClownSampler_Beta", + "_meta": { + "title": "ClownSampler" + } + }, + "4971": { + "inputs": { + "sigmas": "1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875, 0.0" + }, + "class_type": "ManualSigmas", + "_meta": { + "title": "ManualSigmas" + } + }, + "4977": { + "inputs": { + "value": true + }, + "class_type": "PrimitiveBoolean", + "_meta": { + "title": "bypass_i2v" + } + }, + "4978": { + "inputs": { + "value": 24.0 + }, + "class_type": "PrimitiveFloat", + "_meta": { + "title": "fps" + } + }, + "4979": { + "inputs": { + "value": 121 + }, + "class_type": "PrimitiveInt", + "_meta": { + "title": "number of frames" + } + }, + "4981": { + "inputs": { + "resize_type": "scale longer dimension", + "resize_type.longer_size": 1536, + "scale_method": "lanczos", + "input": [ + "2004", + 0 + ] + }, + "class_type": "ResizeImageMaskNode", + "_meta": { + "title": "Resize Image/Mask" + } + }, + "4982": { + "inputs": { + "horizontal_tiles": 2, + "vertical_tiles": 2, + "overlap": 6, + "last_frame_fix": false, + "working_device": "auto", + "working_dtype": "auto", + "vae": [ + "3940", + 2 + ], + "latents": [ + "4845", + 0 + ] + }, + "class_type": "LTXVTiledVAEDecode", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 LTXV Tiled VAE Decode" + } + }, + "4983": { + "inputs": { + "horizontal_tiles": 2, + "vertical_tiles": 2, + "overlap": 6, + "last_frame_fix": false, + "working_device": "auto", + "working_dtype": "auto", + "vae": [ + "3940", + 2 + ], + "latents": [ + "4824", + 0 + ] + }, + "class_type": "LTXVTiledVAEDecode", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 LTXV Tiled VAE Decode" + } + }, + "4985": { + "inputs": { + "a": [ + "4978", + 0 + ] + }, + "class_type": "LTXFloatToInt", + "_meta": { + "title": "\ud83c\udd5b\ud83c\udd63\ud83c\udd67 Float To Int" + } + } +} \ No newline at end of file From f6d0baef4667bbd62b819299a2a4c31bd5b9d90d Mon Sep 17 00:00:00 2001 From: mabry1985 Date: Mon, 13 Jul 2026 07:57:59 +0000 Subject: [PATCH 3/6] video-bridge: fix multipart POST dropping seconds/size/seed on I2V requests The multipart branch built body from only {model, prompt} Form params, so any input_reference (I2V) request silently dropped seconds/size/seed/negative_prompt/ extra_body. Parse the full form instead (coerce seconds/seed numeric, JSON-decode extra_body). Verified: multipart POST now propagates all contract fields. Reported by protoDirector review on protoBanana#38. Co-Authored-By: Claude Opus 4.8 (1M context) --- infra/video-bridge/bridge.py | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/infra/video-bridge/bridge.py b/infra/video-bridge/bridge.py index 0a1da05..28b1e44 100644 --- a/infra/video-bridge/bridge.py +++ b/infra/video-bridge/bridge.py @@ -23,7 +23,7 @@ from __future__ import annotations import os, sys, json, time, uuid, asyncio, pathlib from typing import Any, Optional -from fastapi import FastAPI, HTTPException, UploadFile, File, Form, Request +from fastapi import FastAPI, HTTPException, UploadFile, File, Request from fastapi.responses import JSONResponse, Response # co-located protoBanana checkout provides the reusable ComfyUI client @@ -122,14 +122,25 @@ def _find_output(entry: dict[str, Any]) -> Optional[dict[str, str]]: # ---- the three contract endpoints ----------------------------------------------- @app.post("/v1/videos") async def create_video(request: Request, - input_reference: Optional[UploadFile] = File(None), - model: Optional[str] = Form(None), - prompt: Optional[str] = Form(None)): - # accept both JSON and multipart (input_reference forces multipart) - if input_reference is None and request.headers.get("content-type", "").startswith("application/json"): + input_reference: Optional[UploadFile] = File(None)): + # accept both JSON and multipart (input_reference forces multipart). + # multipart: pull ALL contract fields from the form (not just model/prompt) so an + # I2V request doesn't silently drop seconds/size/seed/negative_prompt/extra_body. + if request.headers.get("content-type", "").startswith("application/json"): body = await request.json() else: - body = {"model": model, "prompt": prompt} + form = await request.form() + body = {k: v for k, v in form.multi_items() if k != "input_reference"} + # scalars that must be numeric downstream + for k in ("seconds", "seed"): + if k in body and body[k] not in (None, ""): + body[k] = float(body[k]) if k == "seconds" else int(body[k]) + # extra_body arrives as a JSON string over multipart + if isinstance(body.get("extra_body"), str): + try: + body["extra_body"] = json.loads(body["extra_body"]) + except json.JSONDecodeError: + raise HTTPException(400, "extra_body must be valid JSON") prompt = body.get("prompt") if not prompt: raise HTTPException(400, "prompt is required") From 03f663f4250d695e6b453bb46e11165156e6aa90 Mon Sep 17 00:00:00 2001 From: mabry1985 Date: Mon, 13 Jul 2026 10:10:31 +0000 Subject: [PATCH 4/6] =?UTF-8?q?ltx2-lora:=20orchestrated=20video-LoRA=20dr?= =?UTF-8?q?iver=20(clips=20=E2=86=92=20usable=20LoRA)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit make_video_lora.py runs the full chain one-command + resumable: caption → preprocess → fix-conditions (the embeds-land-next-to-clips gotcha) → config → train → wire (symlink LoRA into ComfyUI loras/ + emit a T2V workflow with LoraLoaderModelOnly on the fp4 base). Each stage skips if its output exists. Proves the "make our own LoRAs" path is now point-a-folder → get-a-LoRA. Config/wire stages validated against the toy data; full caption/train run is per-dataset. Runbook (README) documents the one command + the manual stages. Co-Authored-By: Claude Opus 4.8 (1M context) --- experiments/ltx2-lora/README.md | 74 ++++++++++ experiments/ltx2-lora/make_video_lora.py | 174 +++++++++++++++++++++++ experiments/ltx2-lora/toy_smoke.yaml | 88 ++++++++++++ 3 files changed, 336 insertions(+) create mode 100644 experiments/ltx2-lora/README.md create mode 100644 experiments/ltx2-lora/make_video_lora.py create mode 100644 experiments/ltx2-lora/toy_smoke.yaml diff --git a/experiments/ltx2-lora/README.md b/experiments/ltx2-lora/README.md new file mode 100644 index 0000000..1ebec3b --- /dev/null +++ b/experiments/ltx2-lora/README.md @@ -0,0 +1,74 @@ +# LTX-2.3 LoRA training — proven pipeline (Blackwell / sm120 / cu128) + +End-to-end LoRA fine-tuning of **LTX-2.3-22B** on the RTX PRO 6000, verified 2026-07-13. LoRAs train +in bf16/int8 and **apply to our NVFP4 22B at inference**. + +## One command (orchestrated) + +```bash +~/dev/LTX-2/.venv/bin/python make_video_lora.py \ + [--rank 32] [--steps 1000] [--bucket 768x512x49] [--with-audio] [--gpu 1] [--no-caption] [--no-wire] +``` + +`make_video_lora.py` runs the whole chain — **caption → preprocess → fix-conditions → config → train → +wire into ComfyUI** — and is **resumable** (each stage skips if its output exists). Point it at a folder +of clips; get a `