Skip to content

[None][infra] Unwaive deepseek-v32-fp4 dep8 disagg perf sanity test (nvbugs/6085022) - #15160

Open
tensorrt-cicd wants to merge 2 commits into
NVIDIA:mainfrom
tensorrt-cicd:unwaive-6085022-deepseek-v32-dep8
Open

[None][infra] Unwaive deepseek-v32-fp4 dep8 disagg perf sanity test (nvbugs/6085022)#15160
tensorrt-cicd wants to merge 2 commits into
NVIDIA:mainfrom
tensorrt-cicd:unwaive-6085022-deepseek-v32-dep8

Conversation

@tensorrt-cicd

@tensorrt-cicd tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Description

Unwaives the following perf sanity test by removing its entry from tests/integration/test_lists/waives.txt:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL]

The waiver referenced https://nvbugs/6085022.

Test Coverage

Re-enables the above test on GB200.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Re-enabled a previously waived performance test to improve test coverage.

Remove the waiver for
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL]
(https://nvbugs/6085022).

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR removes a single waiver entry from the test skip list, unwaiving a specific disagg_upload perf sanity test case that was previously marked as skipped.

Changes

Test Waiver Maintenance

Layer / File(s) Summary
Remove perf sanity disagg_upload test waiver
tests/integration/test_lists/waives.txt
Deleted the SKIP waiver entry for perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL], making that parameterized test case no longer waived.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related PRs

Suggested reviewers

  • mzweilz
  • ZhanruiSunCh
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description check ✅ Passed The description covers the key aspects (what test was unwaived and why), but doesn't follow the repository's PR template structure with formal ticket/type prefix.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately describes the main change: removing a waiver from the test list for a specific deepseek-v32-fp4 performance test, and references the related bug (nvbugs/6085022).

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@chenfeiz0326
chenfeiz0326 force-pushed the unwaive-6085022-deepseek-v32-dep8 branch from ab084fc to e892394 Compare June 9, 2026 10:24
@tensorrt-cicd tensorrt-cicd changed the title test: unwaive deepseek-v32-fp4 dep8 disagg perf sanity test (nvbugs/6085022) [None][infra] Unwaive deepseek-v32-fp4 dep8 disagg perf sanity test (nvbugs/6085022) Jun 9, 2026
@chenfeiz0326
chenfeiz0326 requested a review from mzweilz June 9, 2026 10:25
@chenfeiz0326

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-*"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53199 [ run ] triggered by Bot. Commit: e892394 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53199 [ run ] completed with state FAILURE. Commit: e892394
/LLM/main/L0_MergeRequest_PR pipeline #42396 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…seek-v32 dep8 disagg test

The disagg-gen_only-gb200_deepseek-v32-fp4_32k4k_con256_ctx1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL
test still failed after unwaiving, hitting the KV cache transfer timeout (60000 ms
default) under the 32k4k input/output workload.

Raise kv_transfer_timeout_ms to 600000 in both the gen and ctx
cache_transceiver_config blocks of the perf-sanity and perf disaggregated
configs, matching the value already used by other gb200/gb300 disagg configs.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@chenfeiz0326

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-*"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53297 [ run ] triggered by Bot. Commit: 539cdfb Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53297 [ run ] completed with state FAILURE. Commit: 539cdfb
/LLM/main/L0_MergeRequest_PR pipeline #42485 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants