Skip to content

[None][test] Un-waive K2.5 Thinking FP4 disagg-NIXL e2e/gen_only tests - #15443

Merged
chenfeiz0326 merged 2 commits into
NVIDIA:mainfrom
chenfeiz0326:chenfeiz/unwaive-k25-thinking-fp4-disagg-nixl
Jun 23, 2026
Merged

[None][test] Un-waive K2.5 Thinking FP4 disagg-NIXL e2e/gen_only tests#15443
chenfeiz0326 merged 2 commits into
NVIDIA:mainfrom
chenfeiz0326:chenfeiz/unwaive-k25-thinking-fp4-disagg-nixl

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Removes 3 SKIP entries from tests/integration/test_lists/waives.txt for the K2.5 Thinking FP4 disagg-NIXL family — the underlying issues are resolved on current ToT main.

Tests removed from waives.txt

Test Bug Validation
disagg_upload-e2e-gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL https://nvbugs/6280649 5/5 PASS, 119242–120884 tok/s, 1.38% spread
disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL https://nvbugs/6221022 5/5 PASS, 92186–94509 tok/s, 2.50% spread
disagg_upload-e2e-gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL https://nvbugs/6179661 Same model+config family as above; unblocked by the same ToT fixes

Both validated cases were run as 5 separate Slurm jobs (per-run output dir) on Lyris against a freshly built ToT main wheel:

  • GB300 (sm_103): partition=gb300, dep4 ctx + dep16 gen
  • GB200 (sm_100): partition=gb200, dep4 ctx + dep16 gen, gen_only mode

Spread is well under the 5% CV threshold used by perf-sanity.

Test plan

  • CI perf-sanity job picks up the un-waived tests
  • All 3 cases PASS in the next post-merge perf-sanity run

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Removed waive entries for performance sanity tests, enabling previously skipped test configurations to run.

@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Removes three SKIP waive entries from tests/integration/test_lists/waives.txt: two disagg_upload-e2e entries for gb200_kimi-k25-thinking and gb300_kimi-k25-thinking, and one disagg_upload-gen_only entry for gb200_kimi-k25-thinking, all under perf/test_perf_sanity.py::test_e2e.

Changes

kimi-k25-thinking waive removal

Layer / File(s) Summary
Remove kimi-k25-thinking disagg_upload SKIP entries
tests/integration/test_lists/waives.txt
Deletes two disagg_upload-e2e SKIP lines for gb200/gb300_kimi-k25-thinking (lines 301–304 region) and one disagg_upload-gen_only SKIP line for gb200_kimi-k25-thinking (line 325 region), leaving the qwen3-235b, deepseek-r1, and glm-5 entries untouched.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#15373: Modifies the same perf/test_perf_sanity.py::test_e2e disagg/gen-only regions in waives.txt, including gb200/gb300_kimi-k25-thinking and nearby model/mode waives.
  • NVIDIA/TensorRT-LLM#15395: Changes disagg_upload-* waives for GB200/GB300 and gen-only/e2e variants in the same perf/test_perf_sanity.py section of waives.txt.
  • NVIDIA/TensorRT-LLM#15269: Also removes/updates perf/test_perf_sanity.py::test_e2e disagg upload waive entries in the same disagg_upload-e2e and disagg_upload-gen_only sections.

Suggested reviewers

  • mzweilz
  • 2ez4bz
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: removing waive entries for K2.5 Thinking FP4 disagg-NIXL tests, with relevant context about test types (e2e/gen_only). It is concise and directly related to the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The PR description clearly explains the purpose, provides detailed validation results, and documents the testing methodology comprehensively.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

Validated 5/5 PASS on ToT main wheel for the GB300 e2e and GB200
gen_only variants on Lyris with the configured perf-sanity threshold:

- disagg_upload-e2e-gb300 K2.5 8k1k con4096 dep16 NIXL:
    119242 - 120884 tok/s, 1.38% spread (5/5)
- disagg_upload-gen_only-gb200 K2.5 8k1k con4096 dep16 NIXL:
    92186 - 94509 tok/s, 2.50% spread (5/5)

The GB200 e2e variant (NVBug 6179661) shares the same model and
config family and is unblocked by the same ToT fixes, so it is
unwaived together with the other two.

NVBugs:
- https://nvbugs/6280649 (gb300 e2e)
- https://nvbugs/6221022 (gb200 gen_only)
- https://nvbugs/6179661 (gb200 e2e)

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@chenfeiz0326
chenfeiz0326 force-pushed the chenfeiz/unwaive-k25-thinking-fp4-disagg-nixl branch from 49c177b to ce1e98c Compare June 17, 2026 04:38
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #54756 [ run ] triggered by Bot. Commit: ce1e98c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #54756 [ run ] completed with state FAILURE. Commit: ce1e98c
/LLM/main/L0_MergeRequest_PR pipeline #43774 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…6 NIXL

The newly un-waived disagg_upload-e2e/gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL test still timed out in pre-merge with the default kv_transfer_timeout_ms. Raise it to 600000 ms to match the deepseek-r1/v32 NIXL perf-sanity configs.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #54859 [ run ] triggered by Bot. Commit: 12dd871 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #54859 [ run ] completed with state SUCCESS. Commit: 12dd871
/LLM/main/L0_MergeRequest_PR pipeline #43863 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@chenfeiz0326
chenfeiz0326 requested a review from mzweilz June 23, 2026 01:11
@chenfeiz0326
chenfeiz0326 enabled auto-merge (squash) June 23, 2026 02:34
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #55127 [ skip ] triggered by Bot. Commit: 12dd871 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #55127 [ skip ] completed with state SUCCESS. Commit: 12dd871
Skipping testing for commit 12dd871

Link to invocation

@chenfeiz0326
chenfeiz0326 merged commit e2c7c4f into NVIDIA:main Jun 23, 2026
8 checks passed
xinhe-nv pushed a commit to tensorrt-cicd/TensorRT-LLM that referenced this pull request Jun 24, 2026
NVIDIA#15443)

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>
BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants