[https://nvbugs/6426860][fix] Stabilize compressor BF16 tolerance

[mingyangHao]

Summary by CodeRabbit

• Bug Fixes

  • Improved numerical tolerance handling for variable-length attention prefill validation, reducing false failures with BF16 results.
  • Re-enabled coverage for a previously skipped sparse attention test.

• Tests

  • Updated validation to account for the output data type’s precision when comparing results.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[mingyangHao]

/bot run

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c2d565ef-58b2-46db-b62a-ac40c792259c

📥 Commits

Reviewing files that changed from the base of the PR and between 02cedf6 and a0a5102.

📒 Files selected for processing (2)

• tests/integration/test_lists/waives.txt
• tests/unittest/_torch/attention/sparse/deepseek_v4/test_compressor_kernel.py

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

---

📝 Walkthrough

Walkthrough

The variable-length DeepSeek V4 prefill test now derives relative tolerance from BF16 epsilon, while retaining the existing absolute tolerance. Its corresponding integration test waiver is removed.

Changes

DeepSeek V4 prefill validation

Layer / File(s)	Summary
BF16-aware prefill validation tests/unittest/_torch/attention/sparse/deepseek_v4/test_compressor_kernel.py, tests/integration/test_lists/waives.txt	The output comparison uses the maximum of 2e-3 and BF16 epsilon for rtol, and the matching skipped-test waiver is removed.

Estimated code review effort: 2 (Simple) | ~5 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16295: Updates DeepSeek attention test waivers in the same integration waiver file.
• NVIDIA/TensorRT-LLM#16356: Adds DeepSeek V4-related test waivers in the same file.
• NVIDIA/TensorRT-LLM#16364: Changes waiver entries for specific failing tests in the same integration list.

Suggested reviewers: heyuhhh

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description matches the template headings but leaves the issue, solution, and test coverage sections essentially empty.	Add a short Description of the bug and fix, and include concrete Test Coverage entries for the changed test path.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise, specific, and accurately reflects the main change to BF16 tolerance handling.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #59210 [ run ] triggered by Bot. Commit: a0a5102 Link to invocation

[tensorrt-cicd]

PR_Github #59210 [ run ] completed with state SUCCESS. Commit: a0a5102
/LLM/main/L0_MergeRequest_PR pipeline #47708 completed with status: 'SUCCESS'

CI Report

Link to invocation