[None][test] Adjust timeout cases in QA perf test by yufeiwu-nv · Pull Request #16894 · NVIDIA/TensorRT-LLM

yufeiwu-nv · 2026-07-27T08:44:42Z

Signed-off-by: yufeiwu-nv 230315618+yufeiwu-nv@users.noreply.github.com

Dev Engineer Review

Updated tests/integration/test_lists/qa/llm_perf_core.yml:
- Reduced qwen3_30b_a3b benchmark requests from 256 to 64.
- Removed two llama_v3.3_nemotron_super_49b variants from the L40S/H100/H20 group.
- Added TP=4 49B input/output cases for H100/H20, marked too slow for L40S PCIe-only interconnects.
The changes are limited to performance-test configuration entries and preserve the existing test path and grouping.
No public API or exported entity changes.

QA Engineer Review

Modified QA test-list file: tests/integration/test_lists/qa/llm_perf_core.yml.
Removed two 49B cases, retained the 1000,2000 TP=4 case, and added H100/H20-specific coverage with a L40S performance exclusion.
No test functions were added or modified.
Verdict: needs follow-up because CBTS coverage data is unavailable.

Description

local test passed: https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/view/TRT-LLM-Perf-Pipelines/job/TRT-LLM-Perf-Test/1392/allure/#suites/1391e8d00208145d2c262e76f0a53c75

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…just request counts and add new test cases for llama_v3.3. This includes reducing the number of requests for the 30B MoE model and adding additional tests for the 49B model under different configurations. Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

coderabbitai · 2026-07-27T08:46:58Z

Walkthrough

Updated the LLM performance QA configuration to reduce Qwen3 request volume and target slow Nemotron benchmark variants to compatible GPU groups.

Changes

Performance QA matrix

Layer / File(s)	Summary
Benchmark selection and GPU targeting `tests/integration/test_lists/qa/llm_perf_core.yml`	Reduced `qwen3_30b_a3b` requests from 256 to 64, removed slow Nemotron variants from the L40S group, and added them to the H100/H20 group.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14952: Updates the same LLM performance QA test list and related GPU-specific benchmark selections.

Suggested reviewers: qijune

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is mostly empty and is missing the required what/why summary and explicit test coverage details.	Add a short issue/solution summary and fill the Test Coverage section with the specific perf QA cases run; keep the PR checklist complete.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and clearly reflects the perf QA test case adjustments in llm_perf_core.yml.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

yufeiwu-nv · 2026-07-27T09:44:40Z

/bot skip --comment "only test list modify"

tensorrt-cicd · 2026-07-27T09:50:19Z

PR_Github #61886 [ skip ] triggered by Bot. Commit: f14b548 Link to invocation

tensorrt-cicd · 2026-07-27T10:00:22Z

PR_Github #61886 [ skip ] completed with state SUCCESS. Commit: f14b548
Skipping testing for commit f14b548

Link to invocation

…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (54 commits) [NVIDIA#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 (NVIDIA#12705) [None][test] Adjust timeout cases in QA perf test (NVIDIA#16894) [https://nvbugs/6157892][fix] Mistral format refactor (NVIDIA#15123) [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe (NVIDIA#16830) [https://nvbugs/6501376][fix] Test-only fix — drop the `if hidden_size % 2 != 0: with pytest.raises(...)`… (NVIDIA#16844) [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support (NVIDIA#16355) [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert `bfloat16` under… (NVIDIA#16203) [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft (NVIDIA#16831) [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test (NVIDIA#16878) [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests (NVIDIA#16838) [None][infra] Waive 21 failed cases for main in post-merge 2862 (NVIDIA#16882) [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host (NVIDIA#16791) [None][perf] Optimize Blackwell fused MHC half-MMA kernel (NVIDIA#16799) [None][infra] Auto-update test durations from OpenSearch (last 7 days) [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path (NVIDIA#16789) [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks (NVIDIA#16774) [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer (NVIDIA#16505) [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue (NVIDIA#16775) [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats (NVIDIA#16571) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

…nnahz/dep-1082-shared-mnnvl-moe-lifecycle * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (142 commits) [NVIDIA#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 (NVIDIA#12705) [None][test] Adjust timeout cases in QA perf test (NVIDIA#16894) [https://nvbugs/6157892][fix] Mistral format refactor (NVIDIA#15123) [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe (NVIDIA#16830) [https://nvbugs/6501376][fix] Test-only fix — drop the `if hidden_size % 2 != 0: with pytest.raises(...)`… (NVIDIA#16844) [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support (NVIDIA#16355) [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert `bfloat16` under… (NVIDIA#16203) [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft (NVIDIA#16831) [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test (NVIDIA#16878) [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests (NVIDIA#16838) [None][infra] Waive 21 failed cases for main in post-merge 2862 (NVIDIA#16882) [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host (NVIDIA#16791) [None][perf] Optimize Blackwell fused MHC half-MMA kernel (NVIDIA#16799) [None][infra] Auto-update test durations from OpenSearch (last 7 days) [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path (NVIDIA#16789) [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks (NVIDIA#16774) [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer (NVIDIA#16505) [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue (NVIDIA#16775) [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats (NVIDIA#16571) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

yufeiwu-nv requested a review from a team as a code owner July 27, 2026 08:44

yufeiwu-nv requested review from crazydemo and xinhe-nv July 27, 2026 08:44

github-actions Bot assigned yufeiwu-nv Jul 27, 2026

Merge branch 'main' into fp4

f14b548

yufeiwu-nv changed the title ~~[None][test] Update LLM performance test configurations in llm_perf_core.yml~~ [None][test] Adjust timeout cases in QA perf test Jul 27, 2026

yufeiwu-nv enabled auto-merge (squash) July 27, 2026 09:44

crazydemo approved these changes Jul 27, 2026

View reviewed changes

yufeiwu-nv merged commit 9f5b377 into NVIDIA:main Jul 27, 2026
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][test] Adjust timeout cases in QA perf test - #16894

[None][test] Adjust timeout cases in QA perf test#16894
yufeiwu-nv merged 2 commits into
NVIDIA:mainfrom
yufeiwu-nv:fp4

yufeiwu-nv commented Jul 27, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

yufeiwu-nv commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yufeiwu-nv commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

yufeiwu-nv commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yufeiwu-nv commented Jul 27, 2026 •

edited

Loading

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading