[https://nvbugs/6182617][fix] Restore K2.5 multimodal dep8 accuracy test on transformers 5.5.x - #14392
Conversation
📝 WalkthroughWalkthroughThe PR addresses tokenizer routing inconsistency in Kimi K2.5 by forcing deterministic use of TikTokenTokenizer, adds dynamic module loading with fallback, adjusts the multimodal test token budget strategy, increases test timeout, and removes the associated skip waiver. ChangesKimi K25 Tokenizer Determinism
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tensorrt_llm/_torch/models/modeling_kimi_k25.py (1)
1187-1206: 💤 Low valueConsider narrowing exception scope for dynamic module loading.
Per coding guidelines, prefer catching specific exceptions over broad
Exception. The dynamic module loading could fail withImportError,ModuleNotFoundError,AttributeError, orTypeError. While the fallback is safe, narrowing the catch would make failure modes more explicit:♻️ Suggested narrowing of exception handling
- except Exception as exc: # pragma: no cover - defensive fallback + except (ImportError, ModuleNotFoundError, AttributeError, TypeError, OSError) as exc: # pragma: no cover - defensive fallbackAs per coding guidelines: "Avoid broad exception handling — catch specific exceptions, not bare
except:"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/models/modeling_kimi_k25.py` around lines 1187 - 1206, The current broad except Exception in the dynamic loading block around get_class_from_dynamic_module / slow_cls.from_pretrained should be narrowed to only the relevant failure types; replace the generic except Exception as exc with a specific tuple like except (ImportError, ModuleNotFoundError, AttributeError, TypeError) as exc so import/factory failures are handled but other exceptions still propagate, keeping the same logging and early return behavior for slow_tok fallback.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tensorrt_llm/_torch/models/modeling_kimi_k25.py`:
- Around line 1187-1206: The current broad except Exception in the dynamic
loading block around get_class_from_dynamic_module / slow_cls.from_pretrained
should be narrowed to only the relevant failure types; replace the generic
except Exception as exc with a specific tuple like except (ImportError,
ModuleNotFoundError, AttributeError, TypeError) as exc so import/factory
failures are handled but other exceptions still propagate, keeping the same
logging and early return behavior for slow_tok fallback.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: fad18cc1-a3c6-4eef-9ab0-84def202119b
📒 Files selected for processing (4)
tensorrt_llm/_torch/models/modeling_kimi_k25.pytests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.pytests/integration/test_lists/test-db/l0_dgx_b200.ymltests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
- tests/integration/test_lists/waives.txt
|
/bot run --disable-fail-fast |
|
PR_Github #49617 [ run ] triggered by Bot. Commit: |
|
PR_Github #49617 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
/bot run --extra-stage DGX_B200-8_GPUs-PyTorch-1 |
|
PR_Github #49710 [ run ] triggered by Bot. Commit: |
|
PR_Github #49711 [ run ] triggered by Bot. Commit: |
|
PR_Github #49710 [ run ] completed with state |
|
PR_Github #49711 [ run ] completed with state
|
|
/bot run --extra-stage DGX_B200-8_GPUs-PyTorch-1 --disable-fail-fast |
|
/bot run --disable-fail-fast |
|
PR_Github #49845 [ run ] triggered by Bot. Commit: |
|
PR_Github #49845 [ run ] completed with state |
|
/bot run --stage-list DGX_B200-8_GPUs-PyTorch-1 |
|
PR_Github #49906 [ run ] triggered by Bot. Commit: |
|
PR_Github #49906 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #50047 [ run ] triggered by Bot. Commit: |
|
PR_Github #50047 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #50085 [ run ] triggered by Bot. Commit: |
|
PR_Github #50085 [ run ] completed with state |
…essor Signed-off-by: Tianyu Xiong <117647511+tianyuxbear@users.noreply.github.com>
Signed-off-by: Tianyu Xiong <117647511+tianyuxbear@users.noreply.github.com>
Signed-off-by: Tianyu Xiong <117647511+tianyuxbear@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #50155 [ run ] triggered by Bot. Commit: |
|
PR_Github #50155 [ run ] completed with state |
…est on transformers 5.5.x (NVIDIA#14392) Signed-off-by: Tianyu Xiong <117647511+tianyuxbear@users.noreply.github.com>
…est on transformers 5.5.x (NVIDIA#14392) Signed-off-by: Tianyu Xiong <117647511+tianyuxbear@users.noreply.github.com>
5 participants
Summary
TikTokenTokenizerin the multimodalInputProcessor. transformers 5.5.x'sAutoTokenizerrouting for K2.5 sometimes returns the Rust fast backend, which BPE-splits K2.5 special tokens (<|media_pad|>,<|im_user|>, ...) instead of preserving their canonical IDs. This previously made the multimodal hashing path fall back withNumber of mm_embeds (N) does not match expected total (0).max_num_tokens=16384override on K2.5 dep8 multimodal: withattention_dp=Trueon dep8 the resolvedmoe_max_num_tokenswasmax_num_tokens * dp_size = 131072, pushing the per-calltorch.ops.trtllm.fused_moeworkspace to ~24 GiB. Combined with PyTorch caching-allocator fragmentation from mid-run attention workspace resizes, this caused intermittent CUDA OOM. Falling back to the LLM defaultmax_num_tokens=8192brings the workspace to ~12 GiB and keeps it within activation headroom.max_num_tokens(twice the prefill steps per long prompt). Matches the precedent set byTestKimiK25::test_nvfp4[tp8]onl0_gb200_multi_nodes(180 min) for K2.5 NVFP4 workloads.TestKimiK25::test_nvfp4[dep8]: both failure modes (tokenizer mismatch, fused_moe OOM) are addressed; re-enable the test in pre-merge CI.Resolves NVBug 6182617.
Test plan
pytest tests/unittest/_torch/modeling/test_modeling_kimi_k25.py— 24/24 PASS (previously 10 failures aroundplaceholder_countand multimodal embeddings).pytest tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4 -k dep8— PASSED, MMMU accuracy 81.44 (within sample noise of transformers 5.3 baseline 80.889), 0 OOM, total runtime ~68 min.Summary by CodeRabbit
Bug Fixes
Tests