[None][feat] Default GLM-5 to the Python KV-cache transceiver - #16524
Conversation
|
/bot run --disable-fail-fast |
📝 WalkthroughWalkthroughAdds GLM-5-specific transceiver runtime selection to ChangesGLM-5 transceiver runtime
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant TorchLlmArgs
participant _resolve_transceiver_runtime_auto
participant DeepseekV3ForCausalLM
TorchLlmArgs->>_resolve_transceiver_runtime_auto: Resolve auto runtime with NIXL configuration
_resolve_transceiver_runtime_auto->>DeepseekV3ForCausalLM: Inspect pretrained GLM-5 configuration
DeepseekV3ForCausalLM-->>_resolve_transceiver_runtime_auto: Return PYTHON
_resolve_transceiver_runtime_auto-->>TorchLlmArgs: Set transceiver_runtime to PYTHON
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tensorrt_llm/_torch/models/modeling_deepseekv3.py (1)
1896-1899: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNarrow the hook’s configuration type.
Anydefeats the new interface contract. UsePretrainedConfig | NoneandLiteral["PYTHON"] | None; add Google-styleArgs/Returnsdocumentation for the hook.Based on coding guidelines: “Annotate every function, use
Nonefor non-returning functions, avoidAny… and use …|.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/models/modeling_deepseekv3.py` around lines 1896 - 1899, Update the classmethod get_preferred_transceiver_runtime signature to accept PretrainedConfig | None and return Literal["PYTHON"] | None, replacing Any and Optional annotations. Add Google-style Args and Returns documentation describing the configuration parameter and optional runtime result.Source: Coding guidelines
tests/unittest/llmapi/test_llm_args.py (1)
3170-3211: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAnnotate the newly added test functions.
Add precise parameter and return annotations to
_pretrained_configand the three test methods, including-> Nonefor tests.Based on coding guidelines: “Annotate every function … [and] avoid
Any.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/llmapi/test_llm_args.py` around lines 3170 - 3211, Annotate `_pretrained_config` with concrete parameter and return types, avoiding `Any`, and add precise parameter annotations plus `-> None` to `test_preference_per_architecture`, `test_no_config_defers_to_cpp`, and `test_glm5_resolves_auto_to_python_on_nixl`. Use types matching the architecture list, model type, expected runtime value, and test fixtures already used.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unittest/llmapi/test_llm_args.py`:
- Around line 3178-3189: Update test_preference_per_architecture in
tests/unittest/llmapi/test_llm_args.py to isolate both production predicates:
add a GLM architecture paired with a non-GLM model_type expecting "PYTHON", and
add a non-GLM architecture paired with model_type="glm_moe_dsa" expecting the
fallback result. The existing coverage is insufficient until these cases are
included; no follow-up outside this test file is needed.
---
Nitpick comments:
In `@tensorrt_llm/_torch/models/modeling_deepseekv3.py`:
- Around line 1896-1899: Update the classmethod
get_preferred_transceiver_runtime signature to accept PretrainedConfig | None
and return Literal["PYTHON"] | None, replacing Any and Optional annotations. Add
Google-style Args and Returns documentation describing the configuration
parameter and optional runtime result.
In `@tests/unittest/llmapi/test_llm_args.py`:
- Around line 3170-3211: Annotate `_pretrained_config` with concrete parameter
and return types, avoiding `Any`, and add precise parameter annotations plus `->
None` to `test_preference_per_architecture`, `test_no_config_defers_to_cpp`, and
`test_glm5_resolves_auto_to_python_on_nixl`. Use types matching the architecture
list, model type, expected runtime value, and test fixtures already used.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c8e06700-376f-4f2c-b544-22493efbc27e
📒 Files selected for processing (15)
tensorrt_llm/_torch/models/modeling_deepseekv3.pytests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4ep4_gentp4ep4_glm5_nvfp4_dp_tllm.yamltests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_1k1k_con1_ctx1_dep4_gen1_tep4_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_1k1k_con4096_ctx1_dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_1k1k_con512_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_8k1k_con1024_ctx1_dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_8k1k_con512_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con4096_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL.yamltests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yamltests/unittest/llmapi/test_llm_args.py
|
PR_Github #59866 [ run ] triggered by Bot. Commit: |
|
PR_Github #59866 [ run ] completed with state
|
03682f8 to
88cae70
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #60250 [ run ] triggered by Bot. Commit: |
|
PR_Github #60250 [ run ] completed with state
|
f59c6a3 to
2eb5072
Compare
|
PR_Github #60549 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*" |
2eb5072 to
461cf84
Compare
|
PR_Github #60688 [ run ] triggered by Bot. Commit: |
|
PR_Github #60688 [ run ] completed with state
|
…in disagg GLM-5 and GLM-5.2 (GlmMoeDsaForCausalLM / glm_moe_dsa) share the DeepseekV3ForCausalLM implementation class; override get_preferred_transceiver_runtime() to opt GLM checkpoints into the Python (v2) transceiver, differentiated per checkpoint so DeepSeek-V3/ V3.2 keep the C++ default. The preference is adopted only when the user leaves cache_transceiver_config.transceiver_runtime at 'auto' and the effective backend is NIXL. Set transceiver_runtime: PYTHON explicitly in the GLM-5 perf-sanity disaggregated NIXL configs, and switch the GLM-5 disagg stress-test config from backend DEFAULT to NIXL + PYTHON so the TRTLLM_USE_UCX_KVCACHE=1 fallback cannot silently revert it to the C++ transceiver. Add unit tests covering the per-architecture preference and the end-to-end 'auto' resolution for GLM-5 on NIXL. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…ansceiver test Address review: the GLM-positive case satisfied both predicates at once, so deleting either production check still passed. Add one case per predicate in isolation (GLM architecture with a non-GLM model_type, and a non-GLM architecture with model_type=glm_moe_dsa). Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
461cf84 to
8bd681c
Compare
|
/bot run --stage-list "A10-PyTorch-1, A10-PyTorch-2" |
1 similar comment
|
/bot run --stage-list "A10-PyTorch-1, A10-PyTorch-2" |
|
PR_Github #60937 [ run ] triggered by Bot. Commit: |
|
PR_Github #60937 [ run ] completed with state |
…string Replace the MagicMock pretrained config in the GLM-5 transceiver-preference tests with a real transformers.PretrainedConfig, and rewrap the preference docstring in modeling_deepseekv3.py at 100 columns. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
|
/bot run --stage-list "A10-PyTorch-1, A10-PyTorch-2" |
|
PR_Github #61239 [ run ] triggered by Bot. Commit: |
|
PR_Github #61239 [ run ] completed with state |
|
/bot skip --comment "required tests have passed" |
|
PR_Github #61297 [ skip ] triggered by Bot. Commit: |
|
PR_Github #61297 [ skip ] completed with state |
…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (83 commits) [None][feat] Bind SourceIdentity to checkpoint artifacts (NVIDIA#16159) [None][infra] Waive 4 failed cases for main in pre-merge 49550 (NVIDIA#16798) [None][fix] Make FlashInfer sampling op wrappers opaque to Dynamo (NVIDIA#16732) [None][feat] top-k: route decode to CuTe DSL GVR top-k in e2e (NVIDIA#16420) [None][feat] Default GLM-5 to the Python KV-cache transceiver (NVIDIA#16524) [None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor (NVIDIA#16422) [https://nvbugs/6426850][test] Unwaive Qwen3.5 397B NVFP4 ADP4 TRTLLM test (NVIDIA#16348) [https://nvbugs/6445456][fix] Restore inplace ops for functionalization v2 (NVIDIA#16410) [None][infra] Waive 1 failed cases for main in pre-merge 49229 (NVIDIA#16786) [None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints (NVIDIA#16433) [None][feat] ADP conversation router: configurable least-queued placement for new conversations (NVIDIA#16294) [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16780) [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16781) [TRTLLMINF-188][infra] Require approval for PerfSanity wildcard runs (NVIDIA#16777) [TRTLLM-13969][feat] Support MiniMax M3 for Disaggregated Serving (NVIDIA#16017) [NVIDIA#11932][fix] Filter CUTLASS MoE GEMM tile configs by device shared memory on SM121 (NVIDIA#12704) [None][chore] Remove attention backend test waivers (NVIDIA#16723) [TRTLLM-14540][perf] Skip fp32 state round-trip in FlashInfer GDN pre… (NVIDIA#16703) [TRTLLM-13694][feat] Add IBDB recipe provenance and refresh configs (NVIDIA#16254) [None][infra] Preview/bump/main (NVIDIA#16758) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>
…#16524) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…in disagg
GLM-5 and GLM-5.2 (GlmMoeDsaForCausalLM / glm_moe_dsa) share the DeepseekV3ForCausalLM implementation class; override get_preferred_transceiver_runtime() to opt GLM checkpoints into the Python (v2) transceiver, differentiated per checkpoint so DeepSeek-V3/ V3.2 keep the C++ default. The preference is adopted only when the user leaves cache_transceiver_config.transceiver_runtime at 'auto' and the effective backend is NIXL.
Set transceiver_runtime: PYTHON explicitly in the GLM-5 perf-sanity disaggregated NIXL configs, and switch the GLM-5 disagg stress-test config from backend DEFAULT to NIXL + PYTHON so the TRTLLM_USE_UCX_KVCACHE=1 fallback cannot silently revert it to the C++ transceiver.
Add unit tests covering the per-architecture preference and the end-to-end 'auto' resolution for GLM-5 on NIXL.
Summary by CodeRabbit
New Features
Configuration
Tests
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.