[None][feat] Default GPT-OSS to the Python KV-cache transceiver by nv-xtf · Pull Request #16479 · NVIDIA/TensorRT-LLM

nv-xtf · 2026-07-16T06:42:53Z

Description

Built on top of #16164 (per-model transceiver runtime auto selection): opt GPT-OSS into the Python KV-cache transceiver by default.

GptOssForCausalLM now overrides get_preferred_transceiver_runtime() to return "PYTHON". Disaggregated deployments that leave transceiver_runtime at its default auto (backend NIXL/DEFAULT) use the Python transceiver (KvCacheTransceiverV2) out of the box. Explicit user settings always take precedence, and incompatible backends fall back to the C++ transceiver per the #16164 resolution semantics. No effect on non-disaggregated deployments.

Test Coverage

New unit test: test_modeling_gpt_oss.py::test_gpt_oss_prefers_python_transceiver pins the model preference.
The resolution chain (auto → model hook → backend compatibility) is covered by the [None][feat] BREAKING: add per-model transceiver runtime auto selection #16164 unit tests (TestTransceiverRuntimeAutoResolution).
Verified end-to-end with test_disaggregated_gpt_oss_120b_harmony: startup logs show Resolved transceiver_runtime='auto' to 'PYTHON' for GptOssForCausalLM and Using KvCacheTransceiverV2; explicit CPP override verified to bypass the preference.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Summary by CodeRabbit

Bug Fixes
- Improved GPT-OSS runtime compatibility by selecting the Python transceiver runtime.
Tests
- Added coverage to verify the preferred transceiver runtime is selected correctly.

coderabbitai · 2026-07-16T06:44:45Z

📝 Walkthrough

Walkthrough

GPT-OSS now declares "PYTHON" as its preferred transceiver runtime, with explicit typing annotations and a unit test asserting the returned value.

Changes

GPT-OSS transceiver runtime

Layer / File(s)	Summary
Declare and validate Python runtime preference `tensorrt_llm/_torch/models/modeling_gpt_oss.py`, `tests/unittest/_torch/modeling/test_modeling_gpt_oss.py`	`GptOssForCausalLM` adds a typed class method returning `"PYTHON"`, and a unit test verifies the preference.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: yechank-nvidia

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title follows the repository template and clearly summarizes the main change.
Description check	✅ Passed	The description includes Description, Test Coverage, and PR Checklist sections and is sufficiently complete.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

nv-xtf · 2026-07-16T06:46:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-16T06:52:26Z

PR_Github #59647 [ run ] triggered by Bot. Commit: a5219f6 Link to invocation

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime. In disaggregated serving, when the user leaves cache_transceiver_config.transceiver_runtime at its default 'auto' and the effective backend is NIXL ("DEFAULT" resolves to NIXL when no legacy backend env var is set), Kimi-K2.5 adopts the Python (v2) transceiver instead of the C++ one. Explicit user settings and non-NIXL backends are unaffected. This lets the Kimi-K2.5 disaggregated tests keep backend "DEFAULT" without hard-coding the transceiver runtime. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime. In disaggregated serving, when the user leaves cache_transceiver_config.transceiver_runtime at its default 'auto' and the effective backend is NIXL ("DEFAULT" resolves to NIXL when no legacy backend env var is set), Kimi-K2.5 adopts the Python (v2) transceiver instead of the C++ one. Explicit user settings and non-NIXL backends are unaffected. This lets the Kimi-K2.5 disaggregated tests keep backend "DEFAULT" without hard-coding the transceiver runtime. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. - Remove the redundant TestKimiK2::test_nvfp4 disaggregated accuracy test; Kimi disaggregated accuracy is covered by TestKimiK25, and Kimi-K2-Thinking accuracy stays covered by the aggregated tests in test_llm_api_pytorch. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime, so disaggregated Kimi-K2.5 adopts the Python (v2) transceiver by default when the user leaves transceiver_runtime at 'auto' and the effective backend is NIXL. Explicit user settings and non-NIXL backends are unaffected. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. - TestKimiK25::test_nvfp4: set cache_transceiver_config explicitly to backend=NIXL + transceiver_runtime=PYTHON (ctx and gen). The disagg test harness injects TRTLLM_USE_UCX_KVCACHE=1 for any non-NIXL backend, so a "DEFAULT" backend would resolve to UCX and force the C++ transceiver; the test sets NIXL explicitly so it actually exercises the Python transceiver rather than relying on the model-default resolution. - Remove the redundant TestKimiK2::test_nvfp4 disaggregated accuracy test; Kimi disaggregated accuracy is covered by TestKimiK25, and Kimi-K2-Thinking accuracy stays covered by the aggregated tests in test_llm_api_pytorch. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

tensorrt-cicd · 2026-07-16T19:44:43Z

PR_Github #59647 [ run ] completed with state FAILURE. Commit: a5219f6
/LLM/main/L0_MergeRequest_PR pipeline #48082 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-xtf · 2026-07-17T01:48:50Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T01:55:06Z

PR_Github #59826 [ run ] triggered by Bot. Commit: a5219f6 Link to invocation

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime, so disaggregated Kimi-K2.5 adopts the Python (v2) transceiver by default when the user leaves transceiver_runtime at 'auto' and the effective backend is NIXL. Explicit user settings and non-NIXL backends are unaffected. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. - TestKimiK25::test_nvfp4: set cache_transceiver_config explicitly to backend=NIXL + transceiver_runtime=PYTHON (ctx and gen). The disagg test harness injects TRTLLM_USE_UCX_KVCACHE=1 for any non-NIXL backend, so a "DEFAULT" backend would resolve to UCX and force the C++ transceiver; the test sets NIXL explicitly so it actually exercises the Python transceiver rather than relying on the model-default resolution. - Remove the redundant TestKimiK2::test_nvfp4 disaggregated accuracy test; Kimi disaggregated accuracy is covered by TestKimiK25, and Kimi-K2-Thinking accuracy stays covered by the aggregated tests in test_llm_api_pytorch. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

nv-xtf · 2026-07-17T03:31:42Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T03:36:58Z

PR_Github #59863 [ run ] triggered by Bot. Commit: 92af2a5 Link to invocation

tensorrt-cicd · 2026-07-17T03:38:45Z

PR_Github #59826 [ run ] completed with state ABORTED. Commit: a5219f6

Link to invocation

tensorrt-cicd · 2026-07-17T13:19:19Z

PR_Github #59863 [ run ] completed with state FAILURE. Commit: 92af2a5
/LLM/main/L0_MergeRequest_PR pipeline #48266 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime, so disaggregated Kimi-K2.5 adopts the Python (v2) transceiver by default when the user leaves transceiver_runtime at 'auto' and the effective backend is NIXL. Explicit user settings and non-NIXL backends are unaffected. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. - TestKimiK25::test_nvfp4: set cache_transceiver_config explicitly to backend=NIXL + transceiver_runtime=PYTHON (ctx and gen). The disagg test harness injects TRTLLM_USE_UCX_KVCACHE=1 for any non-NIXL backend, so a "DEFAULT" backend would resolve to UCX and force the C++ transceiver; the test sets NIXL explicitly so it actually exercises the Python transceiver rather than relying on the model-default resolution. - Remove the redundant TestKimiK2::test_nvfp4 disaggregated accuracy test; Kimi disaggregated accuracy is covered by TestKimiK25, and Kimi-K2-Thinking accuracy stays covered by the aggregated tests in test_llm_api_pytorch. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

nv-xtf · 2026-07-19T03:26:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-19T03:32:56Z

PR_Github #60141 [ run ] triggered by Bot. Commit: 92af2a5 Link to invocation

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime, so disaggregated Kimi-K2.5 adopts the Python (v2) transceiver by default when the user leaves transceiver_runtime at 'auto' and the effective backend is NIXL. Explicit user settings and non-NIXL backends are unaffected. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. - TestKimiK25::test_nvfp4: set cache_transceiver_config explicitly to backend=NIXL + transceiver_runtime=PYTHON (ctx and gen). The disagg test harness injects TRTLLM_USE_UCX_KVCACHE=1 for any non-NIXL backend, so a "DEFAULT" backend would resolve to UCX and force the C++ transceiver; the test sets NIXL explicitly so it actually exercises the Python transceiver rather than relying on the model-default resolution. - Remove the redundant TestKimiK2::test_nvfp4 disaggregated accuracy test; Kimi disaggregated accuracy is covered by TestKimiK25, and Kimi-K2-Thinking accuracy stays covered by the aggregated tests in test_llm_api_pytorch. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

tensorrt-cicd · 2026-07-19T15:56:57Z

PR_Github #60141 [ run ] completed with state SUCCESS. Commit: 92af2a5
/LLM/main/L0_MergeRequest_PR pipeline #48520 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-xtf · 2026-07-19T16:21:48Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-19T16:27:13Z

PR_Github #60182 [ run ] triggered by Bot. Commit: 2df5cdc Link to invocation

nv-xtf · 2026-07-19T16:33:54Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-19T16:39:55Z

PR_Github #60184 [ run ] triggered by Bot. Commit: a9b554c Link to invocation

tensorrt-cicd · 2026-07-19T16:43:16Z

PR_Github #60182 [ run ] completed with state ABORTED. Commit: 2df5cdc

Link to invocation

tensorrt-cicd · 2026-07-19T17:09:51Z

PR_Github #60184 [ run ] completed with state FAILURE. Commit: a9b554c
/LLM/main/L0_MergeRequest_PR pipeline #48557 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-xtf · 2026-07-20T01:28:10Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-20T02:17:36Z

PR_Github #60205 [ run ] triggered by Bot. Commit: a9b554c Link to invocation

Mirror the GPT-OSS approach (NVIDIA#16479): KimiK25ForConditionalGeneration now declares "PYTHON" as its preferred KV-cache transceiver runtime, so disaggregated Kimi-K2.5 adopts the Python (v2) transceiver by default when the user leaves transceiver_runtime at 'auto' and the effective backend is NIXL. Explicit user settings and non-NIXL backends are unaffected. - modeling_kimi_k25.py: add get_preferred_transceiver_runtime() -> "PYTHON". - test_modeling_kimi_k25.py: unit test asserting the preference. - TestKimiK25::test_nvfp4: set cache_transceiver_config explicitly to backend=NIXL + transceiver_runtime=PYTHON (ctx and gen). The disagg test harness injects TRTLLM_USE_UCX_KVCACHE=1 for any non-NIXL backend, so a "DEFAULT" backend would resolve to UCX and force the C++ transceiver; the test sets NIXL explicitly so it actually exercises the Python transceiver rather than relying on the model-default resolution. - Remove the redundant TestKimiK2::test_nvfp4 disaggregated accuracy test; Kimi disaggregated accuracy is covered by TestKimiK25, and Kimi-K2-Thinking accuracy stays covered by the aggregated tests in test_llm_api_pytorch. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

tensorrt-cicd · 2026-07-20T03:07:34Z

PR_Github #60205 [ run ] completed with state SUCCESS. Commit: a9b554c
/LLM/main/L0_MergeRequest_PR pipeline #48574 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Tingfeng Xian <289617005+nv-xtf@users.noreply.github.com>

nv-xtf · 2026-07-20T05:19:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-20T05:25:26Z

PR_Github #60268 [ run ] triggered by Bot. Commit: 624016f Link to invocation

tensorrt-cicd · 2026-07-20T08:32:19Z

PR_Github #60268 [ run ] completed with state SUCCESS. Commit: 624016f
/LLM/main/L0_MergeRequest_PR pipeline #48627 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-xtf · 2026-07-20T08:34:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-20T08:41:28Z

PR_Github #60327 [ run ] triggered by Bot. Commit: 624016f Link to invocation

tensorrt-cicd · 2026-07-20T12:32:40Z

PR_Github #60327 [ run ] completed with state FAILURE. Commit: 624016f
/LLM/main/L0_MergeRequest_PR pipeline #48671 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-xtf · 2026-07-20T13:48:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-20T13:54:26Z

PR_Github #60381 [ run ] triggered by Bot. Commit: 624016f Link to invocation

tensorrt-cicd · 2026-07-20T16:19:01Z

PR_Github #60381 [ run ] completed with state SUCCESS. Commit: 624016f
/LLM/main/L0_MergeRequest_PR pipeline #48720 completed with status: 'SUCCESS'

CI Report

Link to invocation

nv-xtf requested a review from a team as a code owner July 16, 2026 06:42

nv-xtf requested a review from yechank-nvidia July 16, 2026 06:42

github-actions Bot assigned nv-xtf Jul 16, 2026

nv-xtf requested a review from Shixiaowei02 July 16, 2026 06:43

Shixiaowei02 approved these changes Jul 16, 2026

View reviewed changes

QiJune approved these changes Jul 17, 2026

View reviewed changes

nv-xtf force-pushed the dev-tingfengx-gpt-oss-transceiver-v2-default branch from 2df5cdc to a9b554c Compare July 19, 2026 16:31

nv-xtf added 2 commits July 20, 2026 11:10

[None][feat] Default GPT-OSS to the Python KV-cache transceiver

9f2140b

Signed-off-by: Tingfeng Xian <289617005+nv-xtf@users.noreply.github.com>

[None][feat] Default GPT-OSS to the Python KV-cache transceiver

624016f

Signed-off-by: Tingfeng Xian <289617005+nv-xtf@users.noreply.github.com>

nv-xtf force-pushed the dev-tingfengx-gpt-oss-transceiver-v2-default branch from a9b554c to 624016f Compare July 20, 2026 03:10

nv-xtf merged commit 4e38fb8 into NVIDIA:main Jul 20, 2026
7 checks passed

Uh oh!

Conversation

nv-xtf commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

nv-xtf commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

nv-xtf commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

nv-xtf commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

nv-xtf commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

nv-xtf commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

nv-xtf commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

nv-xtf commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

nv-xtf commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

nv-xtf commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

nv-xtf commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

nv-xtf commented Jul 16, 2026 •

edited

Loading

coderabbitai Bot commented Jul 16, 2026 •

edited

Loading