[https://nvbugs/6473397][test] Raise BF16 Laguna DFlash memory skip threshold to prevent OOM - #16627
Conversation
…hreshold TestLagunaXS_2_1::test_bf16_dflash OOMs on 80 GB devices (A100 80GB, H100 80GB): the BF16 Laguna-XS-2.1 checkpoint occupies ~64 GiB of weights, leaving only ~14 GiB for KV cache, workspace, and CUDA graph capture at max_batch_size=128 / max_seq_len=4096, which is insufficient. Reducing batch size would slow evaluation and change what the test exercises without a guarantee of fitting; the model itself is the dominant memory cost. Raise skip_less_device_memory from 80000 to 90000 on the BF16 variant so pytest skips cleanly on 80 GB devices while still running on >=94 GB SKUs (H100 NVL, H20, H200, RTX PRO 6000 Blackwell, B200). Verified passing on RTX PRO 6000 Blackwell (~96 GiB). FP8 and NVFP4 variants are unaffected; they fit on H100 80 GB and remain at 80000. Also drop the now-redundant A100/H100 waives for this test, since the marker handles skipping structurally. Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
|
/bot run |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
💤 Files with no reviewable changes (1)
📝 WalkthroughWalkthroughThe BF16 DFLASH accuracy test now skips on devices with less than 90000 device memory units. Its A100 and H100 waiver entries are removed. ChangesBF16 DFLASH test eligibility
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #60371 [ run ] triggered by Bot. Commit: |
|
PR_Github #60371 [ run ] completed with state |
Summary by CodeRabbit
Description
TestLagunaXS_2_1::test_bf16_dflashOOMs on 80 GB devices: BF16 Laguna-XS-2.1 weights are ~64 GiB, leaving too little headroom for KV cache + CUDA graph captureat
max_batch_size=128,max_seq_len=4096.skip_less_device_memoryfrom 80000 → 90000 on BF16 only. FP8 and NVFP4 variants fit on 80 GB and are unchanged.Test plan
pytest test_bf16_dflashskips cleanly on H100 80 GBPR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.