[TRTLLM-13639][perf] Migrate Kimi perf-sanity tests to Transceiver v2 by Shixiaowei02 · Pull Request #15966 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-07-06T07:23:34Z

Description

This pull request updates several performance sanity test YAML configurations for disaggregated worker setups, primarily to enhance the cache_transceiver_config section. The main changes introduce a new runtime and adjust memory settings for the cache transceiver, which will impact how cache data is managed and transferred during tests.

Cache transceiver configuration updates:

Added transceiver_runtime: PYTHON to all affected YAML files, switching the cache transceiver to use the Python runtime for improved flexibility and compatibility. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24]
Added kv_cache_bounce_size_mb: 384 to selected YAML files, increasing the key-value cache bounce buffer size to 384 MB for improved cache transfer performance in relevant test scenarios. [1] [2] [3] [4] [5] [6] [7] [8]

No other functional or logic changes were made; these updates are limited to configuration improvements for the test scripts.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Summary by CodeRabbit

Tests
- Updated several performance sanity test configurations to use explicit cache and runtime settings.
- Standardized test setup across multiple GB200 and GB300 benchmark scenarios.
- Improved consistency of benchmark runs by removing reliance on implicit defaults.

Shixiaowei02 · 2026-07-06T07:26:15Z

/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*"

coderabbitai · 2026-07-06T07:27:23Z

📝 Walkthrough

Walkthrough

Fourteen disaggregated perf-sanity YAML test configuration files (GB200 and GB300 variants) are updated to explicitly add use_kv_cache_manager_v2: false under kv_cache_config and transceiver_runtime: PYTHON under cache_transceiver_config, applied consistently in both worker_config.gen and worker_config.ctx sections.

Changes

Perf-sanity YAML config updates

Layer / File(s)	Summary
GB200 config updates `tests/scripts/perf-sanity/disaggregated/gb200_kimi-k25-thinking-fp4_*.yaml`	Six GB200 configs add `use_kv_cache_manager_v2: false` and `transceiver_runtime: PYTHON` under `gen` and `ctx` worker_config sections.
GB300 config updates `tests/scripts/perf-sanity/disaggregated/gb300_kimi-k25-thinking-fp4_*.yaml`	Seven GB300 configs add the same `use_kv_cache_manager_v2: false` and `transceiver_runtime: PYTHON` settings under `gen` and `ctx` worker_config sections.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately reflects the main change: migrating Kimi perf-sanity tests to Transceiver v2.
Description check	✅ Passed	The description covers the purpose, scope, and checklist, but the Test Coverage section is left blank.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-06T07:32:20Z

PR_Github #57724 [ run ] triggered by Bot. Commit: 588936c Link to invocation

tensorrt-cicd · 2026-07-06T11:52:34Z

PR_Github #57724 [ run ] completed with state SUCCESS. Commit: 588936c
/LLM/main/L0_MergeRequest_PR pipeline #46434 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-08T09:44:33Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge"

tensorrt-cicd · 2026-07-08T09:51:02Z

PR_Github #58216 [ run ] triggered by Bot. Commit: 588936c Link to invocation

tensorrt-cicd · 2026-07-08T10:15:29Z

PR_Github #58216 [ run ] completed with state FAILURE. Commit: 588936c
/LLM/main/L0_MergeRequest_PR pipeline #46858 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-08T10:35:54Z

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-7"

tensorrt-cicd · 2026-07-08T10:41:50Z

PR_Github #58224 [ run ] triggered by Bot. Commit: 588936c Link to invocation

tensorrt-cicd · 2026-07-08T11:06:18Z

PR_Github #58224 [ run ] completed with state SUCCESS. Commit: 588936c
/LLM/main/L0_MergeRequest_PR pipeline #46866 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Shixiaowei02 · 2026-07-08T11:43:43Z

/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*"

tensorrt-cicd · 2026-07-08T11:49:08Z

PR_Github #58228 [ run ] triggered by Bot. Commit: fc8ad13 Link to invocation

tensorrt-cicd · 2026-07-08T15:32:13Z

PR_Github #58228 [ run ] completed with state FAILURE. Commit: fc8ad13
/LLM/main/L0_MergeRequest_PR pipeline #46870 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-09T07:51:58Z

/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*"

tensorrt-cicd · 2026-07-09T07:58:44Z

PR_Github #58421 [ run ] triggered by Bot. Commit: a76a77a Link to invocation

tensorrt-cicd · 2026-07-09T12:17:46Z

PR_Github #58421 [ run ] completed with state FAILURE. Commit: a76a77a
/LLM/main/L0_MergeRequest_PR pipeline #47037 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-10T03:11:47Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-11,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-5,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5"

tensorrt-cicd · 2026-07-10T03:18:07Z

PR_Github #58570 [ run ] triggered by Bot. Commit: d68cc48 Link to invocation

tensorrt-cicd · 2026-07-10T04:43:32Z

PR_Github #58570 [ run ] completed with state FAILURE. Commit: d68cc48
/LLM/main/L0_MergeRequest_PR pipeline #47165 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-14T09:39:35Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-11,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-5,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5"

tensorrt-cicd · 2026-07-14T09:45:06Z

PR_Github #59172 [ run ] triggered by Bot. Commit: ef44752 Link to invocation

tensorrt-cicd · 2026-07-14T11:26:52Z

PR_Github #59172 [ run ] completed with state SUCCESS. Commit: ef44752
/LLM/main/L0_MergeRequest_PR pipeline #47676 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

…-bounce size The 12 gb200/gb300 Kimi-K2.5 disaggregated perf-sanity cases, flipped to the Python (v2) KV-cache transceiver with a 384 MiB KV-bounce arena, for measuring the v1(C++)-vs-v2(Python) transceiver gap. Bounce min_blocks is env-gated (TRTLLM_KV_CACHE_BOUNCE_MIN_BLOCKS); default keeps bounce off on the tiny con4 contexts, set =1 to force it on. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-07-16T06:08:34Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-11,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-5,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5"

tensorrt-cicd · 2026-07-16T06:15:00Z

PR_Github #59634 [ run ] triggered by Bot. Commit: f0179c8 Link to invocation

tensorrt-cicd · 2026-07-16T07:35:50Z

PR_Github #59634 [ run ] completed with state SUCCESS. Commit: f0179c8
/LLM/main/L0_MergeRequest_PR pipeline #48071 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Shixiaowei02 · 2026-07-16T07:40:07Z

/bot skip --comment "Related tests have passed."

tensorrt-cicd · 2026-07-16T07:46:18Z

PR_Github #59661 [ skip ] triggered by Bot. Commit: f0179c8 Link to invocation

tensorrt-cicd · 2026-07-16T07:53:47Z

PR_Github #59661 [ skip ] completed with state SUCCESS. Commit: f0179c8
Skipping testing for commit f0179c8

Link to invocation

….5 con4096 disagg perf case The gen_only case timed out on the CTX side: 2939 "KV cache transfer timeout: elapsed ~60244ms > kv_transfer_timeout_ms=60000ms" lines across CTX ranks 0-3 and 4096/4096 failed requests. Root cause is the data plane, not the timeout budget. With bounce disabled, each 8k request submits its KV as ~15.6k scattered per-block fragments in a single NIXL transfer request, which UCX cannot stripe across rails, so transfers queue behind one another until the 60s budget expires. Coalescing the request's blocks into one contiguous fabric-VMM buffer restores a single multi-rail write. Reproduced and verified locally on GB300 (20 GPUs / 5 nodes, gen_only), both arms on the same commit, wheel and container: as-is : FAILED - 0/4096 successful, 2879 CTX transfer timeouts + bounce : PASSED - 4096/4096 successful, 0 timeouts, 11580 tok/s output per-transfer latency p50 2.16 ms / p90 2.99 ms "[kv-bounce] coalesced 256 blocks / 274MiB into one region across 1 writer(s)" Bounce engages on the single-writer path here because attention DP is enabled on both sides, so each request has exactly one peer rank (expected_transfers == 1). This mirrors the settings NVIDIA#15966 already applied to the other Kimi disagg cases. The timeout budget is left at the 60s default on purpose: coalesced transfers complete in ~2 ms, four orders of magnitude inside it. Un-waives only the gen_only case, which is the one covered by this bug and the one verified above. The e2e variant stays waived pending its own validation. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

github-actions Bot assigned Shixiaowei02 Jul 6, 2026

Shixiaowei02 force-pushed the user/xiaoweis/kimi-perf branch from 96efe46 to 588936c Compare July 6, 2026 07:25

Shixiaowei02 marked this pull request as draft July 6, 2026 07:26

Shixiaowei02 force-pushed the user/xiaoweis/kimi-perf branch 2 times, most recently from 0837be9 to fc8ad13 Compare July 8, 2026 11:31

Shixiaowei02 marked this pull request as ready for review July 8, 2026 11:31

Shixiaowei02 requested review from chenfeiz0326, chuangz0 and nv-xtf July 8, 2026 11:31

Shixiaowei02 marked this pull request as draft July 8, 2026 11:38

Shixiaowei02 force-pushed the user/xiaoweis/kimi-perf branch from fc8ad13 to a76a77a Compare July 9, 2026 07:50

Shixiaowei02 force-pushed the user/xiaoweis/kimi-perf branch from ef44752 to 2a52cb6 Compare July 16, 2026 06:06

Shixiaowei02 marked this pull request as ready for review July 16, 2026 06:07

Shixiaowei02 requested review from a team as code owners July 16, 2026 06:07

Shixiaowei02 requested a review from nv-ananjappa July 16, 2026 06:07

Shixiaowei02 force-pushed the user/xiaoweis/kimi-perf branch from 2a52cb6 to f0179c8 Compare July 16, 2026 06:08

Shixiaowei02 changed the title ~~[None][perf] Validate Kimi-K2.5 transceiver v2 performance~~ [TRTLLM-13639][perf] Validate Kimi-K2.5 transceiver v2 performance Jul 16, 2026

Shixiaowei02 changed the title ~~[TRTLLM-13639][perf] Validate Kimi-K2.5 transceiver v2 performance~~ [TRTLLM-13639][perf] Migrate Kimi perf-sanity tests to Transceiver v2 Jul 16, 2026

nv-xtf approved these changes Jul 16, 2026

View reviewed changes

chenfeiz0326 approved these changes Jul 16, 2026

View reviewed changes

Shixiaowei02 enabled auto-merge (squash) July 16, 2026 07:40

chuangz0 approved these changes Jul 16, 2026

View reviewed changes

Shixiaowei02 merged commit 4188dbe into NVIDIA:main Jul 16, 2026
14 checks passed

Shixiaowei02 deleted the user/xiaoweis/kimi-perf branch July 16, 2026 07:55

coderabbitai Bot mentioned this pull request Jul 17, 2026

[None][feat] Default GLM-5 to the Python KV-cache transceiver #16524

Merged

1 task

This was referenced Jul 27, 2026

[https://nvbugs/6487038][test] unwaive GB300 Kimi disagg e2e after transceiver fixes #16920

Open

[https://nvbugs/6487039][fix] Generalize ADP dummy lifecycle #16921

Open

Uh oh!

Conversation

Shixiaowei02 commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Summary by CodeRabbit

Uh oh!

Shixiaowei02 commented Jul 6, 2026

Uh oh!

coderabbitai Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

Shixiaowei02 commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Shixiaowei02 commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Shixiaowei02 commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Shixiaowei02 commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

Shixiaowei02 commented Jul 10, 2026

Uh oh!

tensorrt-cicd commented Jul 10, 2026

Uh oh!

tensorrt-cicd commented Jul 10, 2026

Uh oh!

Shixiaowei02 commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

Shixiaowei02 commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

Shixiaowei02 commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Shixiaowei02 commented Jul 6, 2026 •

edited

Loading

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading

Shixiaowei02 commented Jul 16, 2026 •

edited

Loading