[https://nvbugs/6189928][fix] Remove waives for DeepSeek-V3.2 FP4 perf sanity tests by xwang233 · Pull Request #14650 · NVIDIA/TensorRT-LLM

xwang233 · 2026-05-27T22:58:12Z

Description

Removes the two SKIP waives added by #14357 for:

perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_v32_fp4_blackwell-v32_fp4_tep8_mtp3_8k1k] (nvbugs/6189928)
perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_v32_fp4_blackwell-v32_fp4_dep8_mtp1_8k1k] (nvbugs/6190071)

Root cause was the transformers 5.3.0 → 5.5.3 upgrade in #13994: bare
PreTrainedConfig no longer exposes max_position_embeddings, so
AutoTokenizer.from_pretrained(..., trust_remote_code=True) against
model_type=deepseek_v32 fell through to the bare config and raised
AttributeError. The benchmark_serving client subprocess died right
after argparse, before any tokenizer was constructed.

Fixed by #14293 (parent nvbugs/6185234), which registers deepseek_v32
and kimi_k2 against the local DeepseekV3Config in transformers'
CONFIG_MAPPING at TRT-LLM import time. Pre-merge regression coverage
added in tests/unittest/_torch/test_custom_config_registration.py.

Test Coverage

The two perf-sanity stages that own these tests:

DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-1 (TEP8/MTP3)
DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-4 (DEP8/MTP1)

PR Checklist

Summary by CodeRabbit

Tests
- Removed two test waiver entries from the performance test list, allowing previously exempted test scenarios to resume execution.

…f sanity tests Both tests were waived in NVIDIA#14357 after L0_PostMerge NVIDIA#2725 caught the transformers 5.5.3 AutoTokenizer.from_pretrained failure on model_type=deepseek_v32 checkpoints. That failure was fixed by NVIDIA#14293 (register deepseek_v32 / kimi_k2 against CONFIG_MAPPING at import time, parent bug 6185234), which merged ~12h before the waives landed. Removing the waives so the post-merge perf-sanity stages cover the DeepSeek-V3.2 FP4 path again. Also covers nvbugs/6190071 (sibling DEP8/MTP1 variant, same RCA). Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

coderabbitai · 2026-05-27T22:59:42Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1e883339-1786-47e4-bda6-823c76b581d9

📥 Commits

Reviewing files that changed from the base of the PR and between 9fa1358 and 778735f.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

Two test waiver entries for test_e2e[aggr_upload-deepseek_v32_fp4_blackwell] variants are removed from the integration test waiver list, indicating those specific test configurations are no longer being skipped.

Changes

Test Waiver List Cleanup

Layer / File(s)	Summary
Remove Deepseek variant waivers `tests/integration/test_lists/waives.txt`	Two `perf/test_perf_sanity.py::test_e2e` waiver entries for `aggr_upload-deepseek_v32_fp4_blackwell` (`dep8_mtp1_8k1k` and `tep8_mtp3_8k1k` variants) are removed.

Possibly related PRs

NVIDIA/TensorRT-LLM#14283: Removes or adds SKIP/waiver entries in the same tests/integration/test_lists/waives.txt file for test cases.
NVIDIA/TensorRT-LLM#13829: Also modifies tests/integration/test_lists/waives.txt by removing or adding specific test_e2e waiver entries.
NVIDIA/TensorRT-LLM#14461: Removes perf/test_perf_sanity.py::test_e2e waiver entries from the same test waiver mechanism.

Suggested reviewers

jieli-matrix
StanleySun639
crazydemo

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main change: removing waives for DeepSeek-V3.2 FP4 perf sanity tests, and is directly related to the changeset.
Description check	✅ Passed	The PR description provides clear explanation of what was changed and why, specifies the root cause, the fix implementation, and lists test coverage. It follows the repository template structure with all key sections completed.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

xwang233 · 2026-05-27T23:00:32Z

/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-1,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-4"

tensorrt-cicd · 2026-05-27T23:06:15Z

PR_Github #50625 [ run ] triggered by Bot. Commit: 778735f Link to invocation

tensorrt-cicd · 2026-05-27T23:23:24Z

PR_Github #50625 [ run ] completed with state FAILURE. Commit: 778735f
/LLM/main/L0_MergeRequest_PR pipeline #40118 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-05-27T23:30:17Z

/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-1,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-4"

tensorrt-cicd · 2026-05-27T23:35:57Z

PR_Github #50628 [ run ] triggered by Bot. Commit: 778735f Link to invocation

tensorrt-cicd · 2026-05-28T01:52:48Z

PR_Github #50628 [ run ] completed with state FAILURE. Commit: 778735f
/LLM/main/L0_MergeRequest_PR pipeline #40121 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-05-28T21:41:56Z

/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-1,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-4"

tensorrt-cicd · 2026-05-28T21:48:01Z

PR_Github #50895 [ run ] triggered by Bot. Commit: 778735f Link to invocation

tensorrt-cicd · 2026-05-28T22:56:40Z

PR_Github #50895 [ run ] completed with state FAILURE. Commit: 778735f
/LLM/main/L0_MergeRequest_PR pipeline #40361 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-05-29T18:29:26Z

/bot skip --comment "This PR only removes two lines from tests/integration/test_lists/waives.txt and touches no production code. The two unwaived DeepSeek-V3.2 FP4 perf-sanity tests (v32_fp4_tep8_mtp3_8k1k / nvbugs/6189928 and v32_fp4_dep8_mtp1_8k1k / nvbugs/6190071) both PASSED in the PR_Github #50628 run. The only CI failure is an unrelated, pre-existing repo-wide perf regression in k25_thinking_fp4_tep8_8k1k, which is failing on all current PRs and is tracked separately."

tensorrt-cicd · 2026-05-29T18:35:01Z

PR_Github #51090 [ skip ] triggered by Bot. Commit: 778735f Link to invocation

tensorrt-cicd · 2026-05-29T18:40:46Z

PR_Github #51090 [ skip ] completed with state SUCCESS. Commit: 778735f
Skipping testing for commit 778735f

Link to invocation

xwang233 · 2026-06-01T17:04:41Z

Closing as redundant: the two DeepSeek-V3.2 FP4 perf-sanity waives this PR removes (nvbugs/6190071 and nvbugs/6189928) were already removed on main by PR #14664 ("[None][test] Unwaive some Perf Tests", commit 32eda52). The change here is now a no-op against main, so there's nothing left to merge.

github-actions Bot assigned xwang233 May 27, 2026

xwang233 requested review from ZhanruiSunCh, chzblych and jieli-matrix and removed request for ZhanruiSunCh and chzblych May 29, 2026 18:29

xwang233 closed this Jun 1, 2026

Uh oh!

Conversation

xwang233 commented May 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 27, 2026

Walkthrough

Changes

Possibly related PRs

Suggested reviewers

Uh oh!

xwang233 commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 27, 2026

Uh oh!

xwang233 commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

xwang233 commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

xwang233 commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

xwang233 commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xwang233 commented May 27, 2026 •

edited by coderabbitai Bot

Loading