Skip to content

[None][feat] Default GLM-5 to the Python KV-cache transceiver - #16524

Merged
chuangz0 merged 3 commits into
NVIDIA:mainfrom
chuangz0:glm_model_default_python_transceiver
Jul 23, 2026
Merged

[None][feat] Default GLM-5 to the Python KV-cache transceiver#16524
chuangz0 merged 3 commits into
NVIDIA:mainfrom
chuangz0:glm_model_default_python_transceiver

Conversation

@chuangz0

@chuangz0 chuangz0 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

…in disagg

GLM-5 and GLM-5.2 (GlmMoeDsaForCausalLM / glm_moe_dsa) share the DeepseekV3ForCausalLM implementation class; override get_preferred_transceiver_runtime() to opt GLM checkpoints into the Python (v2) transceiver, differentiated per checkpoint so DeepSeek-V3/ V3.2 keep the C++ default. The preference is adopted only when the user leaves cache_transceiver_config.transceiver_runtime at 'auto' and the effective backend is NIXL.

Set transceiver_runtime: PYTHON explicitly in the GLM-5 perf-sanity disaggregated NIXL configs, and switch the GLM-5 disagg stress-test config from backend DEFAULT to NIXL + PYTHON so the TRTLLM_USE_UCX_KVCACHE=1 fallback cannot silently revert it to the C++ transceiver.

Add unit tests covering the per-architecture preference and the end-to-end 'auto' resolution for GLM-5 on NIXL.

Summary by CodeRabbit

  • New Features

    • Added automatic Python runtime selection for GLM-5 transceiver configurations.
    • GLM-5 deployments using the NIXL backend now default to the Python transceiver runtime.
  • Configuration

    • Updated disaggregated inference and performance configurations to explicitly use the Python transceiver runtime.
  • Tests

    • Added coverage verifying GLM-5 detection, default behavior, and automatic runtime resolution.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds GLM-5-specific transceiver runtime selection to DeepseekV3ForCausalLM, tests automatic resolution to PYTHON, and updates disaggregated NIXL configurations for context and generation workers.

Changes

GLM-5 transceiver runtime

Layer / File(s) Summary
GLM-5 runtime preference hook
tensorrt_llm/_torch/models/modeling_deepseekv3.py
Adds get_preferred_transceiver_runtime, returning PYTHON for GLM-5 architecture or model-type identifiers and None otherwise.
Runtime resolution tests
tests/unittest/llmapi/test_llm_args.py
Tests architecture matching, missing configurations, and automatic NIXL resolution to PYTHON.
Disaggregated runtime configurations
tests/integration/defs/disaggregated/test_configs/*, tests/scripts/perf-sanity/disaggregated/*glm-5*
Sets NIXL and explicit PYTHON transceiver runtimes for context and generation workers.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TorchLlmArgs
  participant _resolve_transceiver_runtime_auto
  participant DeepseekV3ForCausalLM
  TorchLlmArgs->>_resolve_transceiver_runtime_auto: Resolve auto runtime with NIXL configuration
  _resolve_transceiver_runtime_auto->>DeepseekV3ForCausalLM: Inspect pretrained GLM-5 configuration
  DeepseekV3ForCausalLM-->>_resolve_transceiver_runtime_auto: Return PYTHON
  _resolve_transceiver_runtime_auto-->>TorchLlmArgs: Set transceiver_runtime to PYTHON
Loading

Possibly related PRs

Suggested labels: api-compatible

Suggested reviewers: qijune, nv-xtf, bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the main change: defaulting GLM-5 to the Python KV-cache transceiver.
Description check ✅ Passed The description explains the problem, solution, and test coverage, though it is not fully structured into the template sections.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
tensorrt_llm/_torch/models/modeling_deepseekv3.py (1)

1896-1899: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Narrow the hook’s configuration type.

Any defeats the new interface contract. Use PretrainedConfig | None and Literal["PYTHON"] | None; add Google-style Args/Returns documentation for the hook.

Based on coding guidelines: “Annotate every function, use None for non-returning functions, avoid Any … and use … |.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/_torch/models/modeling_deepseekv3.py` around lines 1896 - 1899,
Update the classmethod get_preferred_transceiver_runtime signature to accept
PretrainedConfig | None and return Literal["PYTHON"] | None, replacing Any and
Optional annotations. Add Google-style Args and Returns documentation describing
the configuration parameter and optional runtime result.

Source: Coding guidelines

tests/unittest/llmapi/test_llm_args.py (1)

3170-3211: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Annotate the newly added test functions.

Add precise parameter and return annotations to _pretrained_config and the three test methods, including -> None for tests.

Based on coding guidelines: “Annotate every function … [and] avoid Any.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/llmapi/test_llm_args.py` around lines 3170 - 3211, Annotate
`_pretrained_config` with concrete parameter and return types, avoiding `Any`,
and add precise parameter annotations plus `-> None` to
`test_preference_per_architecture`, `test_no_config_defers_to_cpp`, and
`test_glm5_resolves_auto_to_python_on_nixl`. Use types matching the architecture
list, model type, expected runtime value, and test fixtures already used.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unittest/llmapi/test_llm_args.py`:
- Around line 3178-3189: Update test_preference_per_architecture in
tests/unittest/llmapi/test_llm_args.py to isolate both production predicates:
add a GLM architecture paired with a non-GLM model_type expecting "PYTHON", and
add a non-GLM architecture paired with model_type="glm_moe_dsa" expecting the
fallback result. The existing coverage is insufficient until these cases are
included; no follow-up outside this test file is needed.

---

Nitpick comments:
In `@tensorrt_llm/_torch/models/modeling_deepseekv3.py`:
- Around line 1896-1899: Update the classmethod
get_preferred_transceiver_runtime signature to accept PretrainedConfig | None
and return Literal["PYTHON"] | None, replacing Any and Optional annotations. Add
Google-style Args and Returns documentation describing the configuration
parameter and optional runtime result.

In `@tests/unittest/llmapi/test_llm_args.py`:
- Around line 3170-3211: Annotate `_pretrained_config` with concrete parameter
and return types, avoiding `Any`, and add precise parameter annotations plus `->
None` to `test_preference_per_architecture`, `test_no_config_defers_to_cpp`, and
`test_glm5_resolves_auto_to_python_on_nixl`. Use types matching the architecture
list, model type, expected runtime value, and test fixtures already used.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c8e06700-376f-4f2c-b544-22493efbc27e

📥 Commits

Reviewing files that changed from the base of the PR and between 1228fed and 0e80777.

📒 Files selected for processing (15)
  • tensorrt_llm/_torch/models/modeling_deepseekv3.py
  • tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp4ep4_gentp4ep4_glm5_nvfp4_dp_tllm.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_1k1k_con1_ctx1_dep4_gen1_tep4_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_1k1k_con4096_ctx1_dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_1k1k_con512_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_8k1k_con1024_ctx1_dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb200_glm-5-fp4_8k1k_con512_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con4096_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_1k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml
  • tests/unittest/llmapi/test_llm_args.py

Comment thread tests/unittest/llmapi/test_llm_args.py Outdated
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59866 [ run ] triggered by Bot. Commit: 0e80777 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59866 [ run ] completed with state SUCCESS. Commit: 0e80777
/LLM/main/L0_MergeRequest_PR pipeline #48271 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yufeiwu-nv
yufeiwu-nv removed the request for review from ruodil July 17, 2026 23:31
@chuangz0
chuangz0 force-pushed the glm_model_default_python_transceiver branch 2 times, most recently from 03682f8 to 88cae70 Compare July 20, 2026 03:30
@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60250 [ run ] triggered by Bot. Commit: 88cae70 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60250 [ run ] completed with state SUCCESS. Commit: 88cae70
/LLM/main/L0_MergeRequest_PR pipeline #48612 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@nv-xtf nv-xtf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chuangz0
chuangz0 force-pushed the glm_model_default_python_transceiver branch 2 times, most recently from f59c6a3 to 2eb5072 Compare July 21, 2026 03:40
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60549 [ run ] completed with state FAILURE. Commit: 7a0f5bf
/LLM/main/L0_MergeRequest_PR pipeline #48863 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*"

@chuangz0
chuangz0 force-pushed the glm_model_default_python_transceiver branch from 2eb5072 to 461cf84 Compare July 21, 2026 12:24
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60688 [ run ] triggered by Bot. Commit: 461cf84 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60688 [ run ] completed with state FAILURE. Commit: 461cf84
/LLM/main/L0_MergeRequest_PR pipeline #48981 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chuangz0 added 2 commits July 22, 2026 09:48
…in disagg

GLM-5 and GLM-5.2 (GlmMoeDsaForCausalLM / glm_moe_dsa) share the
DeepseekV3ForCausalLM implementation class; override
get_preferred_transceiver_runtime() to opt GLM checkpoints into the
Python (v2) transceiver, differentiated per checkpoint so DeepSeek-V3/
V3.2 keep the C++ default. The preference is adopted only when the user
leaves cache_transceiver_config.transceiver_runtime at 'auto' and the
effective backend is NIXL.

Set transceiver_runtime: PYTHON explicitly in the GLM-5 perf-sanity
disaggregated NIXL configs, and switch the GLM-5 disagg stress-test
config from backend DEFAULT to NIXL + PYTHON so the
TRTLLM_USE_UCX_KVCACHE=1 fallback cannot silently revert it to the C++
transceiver.

Add unit tests covering the per-architecture preference and the
end-to-end 'auto' resolution for GLM-5 on NIXL.

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…ansceiver test

Address review: the GLM-positive case satisfied both predicates at
once, so deleting either production check still passed. Add one case
per predicate in isolation (GLM architecture with a non-GLM model_type,
and a non-GLM architecture with model_type=glm_moe_dsa).

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
@chuangz0
chuangz0 force-pushed the glm_model_default_python_transceiver branch from 461cf84 to 8bd681c Compare July 22, 2026 01:48
@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "A10-PyTorch-1, A10-PyTorch-2"

1 similar comment
@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "A10-PyTorch-1, A10-PyTorch-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60937 [ run ] triggered by Bot. Commit: 8bd681c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60937 [ run ] completed with state SUCCESS. Commit: 8bd681c
/LLM/main/L0_MergeRequest_PR pipeline #49202 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@chuangz0
chuangz0 requested a review from litaotju July 23, 2026 02:37
Comment thread tensorrt_llm/_torch/models/modeling_deepseekv3.py Outdated
Comment thread tests/unittest/llmapi/test_llm_args.py Outdated
Comment thread tests/unittest/llmapi/test_llm_args.py
…string

Replace the MagicMock pretrained config in the GLM-5 transceiver-preference
tests with a real transformers.PretrainedConfig, and rewrap the preference
docstring in modeling_deepseekv3.py at 100 columns.

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "A10-PyTorch-1, A10-PyTorch-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61239 [ run ] triggered by Bot. Commit: a84ea81 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61239 [ run ] completed with state SUCCESS. Commit: a84ea81
/LLM/main/L0_MergeRequest_PR pipeline #49478 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "required tests have passed"

@chuangz0
chuangz0 enabled auto-merge (squash) July 23, 2026 13:42
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61297 [ skip ] triggered by Bot. Commit: a84ea81 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61297 [ skip ] completed with state SUCCESS. Commit: a84ea81
Skipping testing for commit a84ea81

Link to invocation

@chuangz0
chuangz0 merged commit e16dcc5 into NVIDIA:main Jul 23, 2026
8 checks passed
hhzhang16 added a commit to hhzhang16/TensorRT-LLM that referenced this pull request Jul 23, 2026
…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce

* 'main' of https://github.com/NVIDIA/TensorRT-LLM: (83 commits)
  [None][feat] Bind SourceIdentity to checkpoint artifacts (NVIDIA#16159)
  [None][infra] Waive 4 failed cases for main in pre-merge 49550 (NVIDIA#16798)
  [None][fix] Make FlashInfer sampling op wrappers opaque to Dynamo (NVIDIA#16732)
  [None][feat] top-k: route decode to CuTe DSL GVR top-k in e2e (NVIDIA#16420)
  [None][feat] Default GLM-5 to the Python KV-cache transceiver (NVIDIA#16524)
  [None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor (NVIDIA#16422)
  [https://nvbugs/6426850][test] Unwaive Qwen3.5 397B NVFP4 ADP4 TRTLLM test (NVIDIA#16348)
  [https://nvbugs/6445456][fix] Restore inplace ops for functionalization v2 (NVIDIA#16410)
  [None][infra] Waive 1 failed cases for main in pre-merge 49229 (NVIDIA#16786)
  [None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints (NVIDIA#16433)
  [None][feat] ADP conversation router: configurable least-queued placement for new conversations (NVIDIA#16294)
  [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16780)
  [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16781)
  [TRTLLMINF-188][infra] Require approval for PerfSanity wildcard runs (NVIDIA#16777)
  [TRTLLM-13969][feat] Support MiniMax M3 for Disaggregated Serving (NVIDIA#16017)
  [NVIDIA#11932][fix] Filter CUTLASS MoE GEMM tile configs by device shared memory on SM121 (NVIDIA#12704)
  [None][chore] Remove attention backend test waivers (NVIDIA#16723)
  [TRTLLM-14540][perf] Skip fp32 state round-trip in FlashInfer GDN pre… (NVIDIA#16703)
  [TRTLLM-13694][feat] Add IBDB recipe provenance and refresh configs (NVIDIA#16254)
  [None][infra] Preview/bump/main (NVIDIA#16758)
  ...

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>
yuanjingx87 pushed a commit to yuanjingx87/TensorRT-LLM that referenced this pull request Jul 26, 2026
…#16524)

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants