Skip to content

[https://nvbugs/6450333][test] Unwaive DeepSeek V4 Flash auto dtype test - #16890

Merged
lfr-0531 merged 1 commit into
NVIDIA:mainfrom
lfr-0531:user/fanrongl/unwaive-nvbug-6450333-dsv4-flash-base
Jul 28, 2026
Merged

[https://nvbugs/6450333][test] Unwaive DeepSeek V4 Flash auto dtype test#16890
lfr-0531 merged 1 commit into
NVIDIA:mainfrom
lfr-0531:user/fanrongl/unwaive-nvbug-6450333-dsv4-flash-base

Conversation

@lfr-0531

@lfr-0531 lfr-0531 commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Description

NVBug 6450333 is not reproducible on the latest main, so the waiver for the following test is no longer needed:

accuracy/test_llm_api_pytorch.py::TestDeepSeekV4Flash::test_auto_dtype SKIP (https://nvbugs/6450333)

This PR removes only that waiver entry, re-enabling the existing DeepSeek V4 Flash auto-dtype accuracy test in the configured B200 and B300 CI stages. It does not change source or runtime behavior.

Test Coverage

Validated on 4x B200 from main at b91ffdad9113f1d04adbb685b044f9d64759ab88 after a clean SM100 build-install. The test process loaded the current repository source and rebuilt nanobind module; LlmRequest.get_tokens_range was present.

CUDA_VISIBLE_DEVICES=0,1,2,3 \
LLM_MODELS_ROOT=/home/scratch.trt_llm_data/llm-models \
PYTHONPATH="$PWD:$PWD/tests/integration/defs:$PWD/examples" \
python3 -m pytest -vv -s \
  tests/integration/defs/accuracy/test_llm_api_pytorch.py::TestDeepSeekV4Flash::test_auto_dtype \
  --timeout=2400

Result: 1 passed in 431.42s.

  • MMLU: 89.059 (threshold 85.383)
  • GSM8K: 95.451 (threshold 91.907)
  • pre-commit run --files tests/integration/test_lists/waives.txt: passed

The local validation used B200 hardware; removing the waiver restores coverage for both the B200 and B300 stages where this test is already listed.

PR Checklist

  • PR title follows the [JIRA/NVBUG][type] description format
  • PR description clearly explains what and why
  • Relevant test coverage is included above
  • Change is limited to one stale waiver; no API, dependency, CODEOWNERS, documentation, or architecture changes
  • DCO sign-off is included

GitHub Bot Help

To see a list of available CI bot commands, comment /bot help.

Dev Engineer Review

  • Removed the TestDeepSeekV4Flash::test_auto_dtype waiver and its NVBug reference from tests/integration/test_lists/waives.txt.
  • The change is correctly scoped to the intended test-list entry, with no source or runtime behavior changes.
  • Waiver format remains valid, with no unintended configuration changes or duplicate entries identified.

QA Engineer Review

  • No test-db/ or qa/ files were modified.
  • Removed the waiver for accuracy/test_llm_api_pytorch.py::TestDeepSeekV4Flash::test_auto_dtype, re-enabling its configured B200 and B300 CI coverage.
  • The test passed on 4x B200 with MMLU 89.059 and GSM8K 95.451.
  • Verdict: sufficient.

Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ef2359f3-8b92-443d-9163-224d77290389

📥 Commits

Reviewing files that changed from the base of the PR and between 1dd8b97 and 7d14c8f.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

The integration waiver list removes the SKIP entry for TestDeepSeekV4Flash::test_auto_dtype, allowing the test to run without that waiver.

Changes

Integration waiver update

Layer / File(s) Summary
Remove DeepSeekV4Flash waiver
tests/integration/test_lists/waives.txt
Removes the TestDeepSeekV4Flash::test_auto_dtype waiver and its linked nvbugs reference.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: qijune

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the waiver removal change and identifies the target test and NVBug.
Description check ✅ Passed The description includes the required sections and gives clear issue, fix, test coverage, and checklist details.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@lfr-0531

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61854 [ run ] triggered by Bot. Commit: 7d14c8f Link to invocation

@lfr-0531
lfr-0531 enabled auto-merge (squash) July 27, 2026 09:59
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61854 [ run ] completed with state SUCCESS. Commit: 7d14c8f
/LLM/main/L0_MergeRequest_PR pipeline #50047 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@lfr-0531

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-1-cbts"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61935 [ run ] triggered by Bot. Commit: 7d14c8f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61935 [ run ] completed with state FAILURE. Commit: 7d14c8f
/LLM/main/L0_MergeRequest_PR pipeline #50125 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@lfr-0531

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62058 [ run ] triggered by Bot. Commit: 7d14c8f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62058 [ run ] completed with state SUCCESS. Commit: 7d14c8f
/LLM/main/L0_MergeRequest_PR pipeline #50243 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@lfr-0531

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62122 [ run ] triggered by Bot. Commit: 7d14c8f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62122 [ run ] completed with state SUCCESS. Commit: 7d14c8f
/LLM/main/L0_MergeRequest_PR pipeline #50303 completed with status: 'SUCCESS'

CI Report

Link to invocation

@lfr-0531
lfr-0531 merged commit 6219c2e into NVIDIA:main Jul 28, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants