[https://nvbugs/6185248][fix] In model_loader.py, resolve llm_backbone = getattr(model, "llm", model) and - #14389
[https://nvbugs/6185248][fix] In model_loader.py, resolve llm_backbone = getattr(model, "llm", model) and #14389tensorrt-cicd wants to merge 1 commit into
model_loader.py, resolve llm_backbone = getattr(model, "llm", model) and #14389Conversation
…d kwargs in Kimi-K2.5 Kimi-K2.5 is a VLM wrapper that holds the LLM as ``self.llm = DeepseekV3ForCausalLM(...)``, so the speculative-decoding attributes (``draft_config``, ``draft_model``, ``load_draft_weights``) live on ``model.llm`` rather than on ``model``. Two issues: 1. ``ModelLoader._load_weights`` dereferenced ``model.draft_config`` / ``model.draft_model`` / ``model.load_draft_weights`` directly, which raised ``AttributeError: 'KimiK25ForConditionalGeneration' object has no attribute 'draft_config'`` on MTP runs. Resolve the LLM backbone via ``getattr(model, "llm", model)`` for both ``HF`` and ``DUMMY`` load formats. 2. ``KimiK25ForConditionalGeneration.forward`` dropped extra kwargs (``spec_metadata``, ``resource_manager``) when calling ``self.llm.forward(...)`` positionally. Forward through the relevant kwargs so MTP receives ``spec_metadata`` and warmup no longer fails with ``'NoneType' object has no attribute 'gather_ids'``. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR introduces VLM wrapper model support by filtering multimodal-specific parameters during KimiK25 forward execution and adjusting draft-weight loading to route operations through the LLM backbone instead of the wrapper model. ChangesVLM Wrapper Model Support
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Closing in favor of #14379 which has been verified via /bot run (PR_Github #49557 and PR_Github #49659 both SUCCESS, including the original failing stage GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1). Both PRs fix the same root cause. #14379 keeps the fix on the model side; #14389 changes model_loader.py. K2.5 follows the same composition-with-self.llm pattern as Qwen2VL/Qwen3VL/Gemma3VL/ LlavaNext/VILA, but is the first VLM in the codebase to need spec_dec, so it's also the first to need the draft_config/draft_model forwarders that #14379 adds. Containing this within the model class keeps framework code unchanged. |
3 participants
Summary
self.llm, so MTP'sdraft_config/draft_model/load_draft_weightslive onmodel.llm, notmodel; the loader dereferenced them onmodeldirectly.model_loader.py, resolvellm_backbone = getattr(model, "llm", model)and use it fordraft_config/draft_model/load_draft_weights(both HF and DUMMY paths). Inmodeling_kimi_k25.py, forward extra kwargs (filtered to drop multimodal-only keys) intoself.llm.forward(...).Test plan
Links
Summary by CodeRabbit