Skip to content

[https://nvbugs/6029882][test] Unwaive TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_mtp]#15101

Closed
tianyuxbear wants to merge 1 commit into
NVIDIA:mainfrom
tianyuxbear:fix/6029882
Closed

[https://nvbugs/6029882][test] Unwaive TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_mtp]#15101
tianyuxbear wants to merge 1 commit into
NVIDIA:mainfrom
tianyuxbear:fix/6029882

Conversation

@tianyuxbear

@tianyuxbear tianyuxbear commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Description

Removes the waiver for accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_mtp] (NVBugs 6029882, also tracked by 6261994).

The waiver was added for an intermittent CUDA OOM at executor init (model.to('cuda')) on the GB200-8_GPUs-2_Nodes stage. Investigation shows this is an environment-driven failure: the model weights nearly fill the GB200 and a foreign process occupying GPU 0 in CI tips it over, rather than a model-memory regression. Local reproduction on a 2-node × 4× GB200 setup (50 runs) did not reproduce the OOM, and repair-bot could not reproduce on B200 either. This matches the same fingerprint seen in the Kimi-K2.5 init-OOM bugs (6248837 / 6260890).

Unwaiving to let CI confirm; closing 6029882 and 6261994 on a clean run.

Test Coverage

accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_mtp]

Summary by CodeRabbit

Release Notes

  • Tests
    • Removed a test waiver entry to re-enable testing of a previously skipped scenario.

…ti_gpus[throughput_mtp]

Signed-off-by: Tianyu Xiong <117647511+tianyuxbear@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 017c602d-a63a-4ea5-9e84-b4b916499bc2

📥 Commits

Reviewing files that changed from the base of the PR and between 09c21b6 and 8bedc10.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

The PR removes a single test waiver entry from the integration test skip list. The deleted waiver was for accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_mtp], which means this specific test case will no longer be skipped and will run as part of the test suite.

Changes

Test Waiver Removal

Layer / File(s) Summary
NVFP4 test waiver removal
tests/integration/test_lists/waives.txt
Removed the SKIP waiver for TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_mtp], allowing the test to execute in the standard test run.

Possibly related PRs

  • NVIDIA/TensorRT-LLM#15010: Both PRs remove SKIP entries for DeepSeekR1 NVFP4-related tests from the waiver list.
  • NVIDIA/TensorRT-LLM#14896: Both PRs modify the same waiver list by adding or removing SKIP entries for accuracy/test_llm_api_pytorch.py tests.
  • NVIDIA/TensorRT-LLM#14792: Both PRs modify test waiver entries for TestDeepSeekR1 NVFP4 multi-GPU test variants in the same waiver file.

Suggested reviewers

  • chzblych
  • nvchenghaoz
  • bo-nv

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically identifies the main change: unwaiving a test waiver. It includes the NVBugs ticket reference and is directly related to the changeset's primary purpose.
Description check ✅ Passed The PR description is comprehensive and complete, covering the rationale for removing the waiver, investigation findings, test coverage, and context about the NVBugs tickets involved.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@tianyuxbear

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52749 [ run ] triggered by Bot. Commit: 8bedc10 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52749 [ run ] completed with state SUCCESS. Commit: 8bedc10
/LLM/main/L0_MergeRequest_PR pipeline #42011 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tianyuxbear

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-2" --disable-reuse-test

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #53040 [ run ] triggered by Bot. Commit: 8bedc10 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #53040 [ run ] completed with state SUCCESS. Commit: 8bedc10
/LLM/main/L0_MergeRequest_PR pipeline #42259 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants