[https://nvbugs/6222480][test] fix stress test issue on H100 by xinhe-nv · Pull Request #14721 · NVIDIA/TensorRT-LLM

xinhe-nv · 2026-05-29T02:59:33Z

Summary by CodeRabbit

Tests
- Updated test backend configurations to use Triton instead of CUTLASS
- Removed obsolete test cases from test selection lists
- Cleaned up test waiver entries
- Updated disaggregated stress test configurations
Chores
- Consolidated test configuration and duration tracking

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-05-29T03:03:14Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8bf4ade9-880c-4a67-a30d-8320e596c6e8

📥 Commits

Reviewing files that changed from the base of the PR and between 1bb1a02 and 56a7dd5.

📒 Files selected for processing (9)

tests/integration/defs/.test_durations
tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_eagle_triton.yaml
tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_triton.yaml
tests/integration/defs/disaggregated/test_disaggregated.py
tests/integration/test_lists/qa/llm_function_core.txt
tests/integration/test_lists/qa/llm_function_rtx6k.txt
tests/integration/test_lists/qa/llm_function_stress.txt
tests/integration/test_lists/test-db/l0_l40s.yml
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (5)

tests/integration/test_lists/test-db/l0_l40s.yml
tests/integration/test_lists/qa/llm_function_rtx6k.txt
tests/integration/test_lists/qa/llm_function_core.txt
tests/integration/test_lists/waives.txt
tests/integration/defs/.test_durations

📝 Walkthrough

Walkthrough

This PR updates disaggregated test infrastructure to migrate from CUTLASS to TRITON backend for key-value cache and mixture-of-experts configurations, wires these new configurations into test parametrization, and removes several test cases and corresponding waivers from QA test selection lists.

Changes

Test backend migration and list cleanup

Layer / File(s)	Summary
TRITON backend configuration updates `tests/integration/defs/disaggregated/test_configs/disagg_config_ctxtp2_gentp2_gptoss_tllm_eagle_triton.yaml`, `disagg_config_ctxtp2_gentp2_gptoss_tllm_triton.yaml`	YAML test configurations update `backend` and `moe_config.backend` settings from `CUTLASS` to `TRITON` across context and generation server sections.
Test configuration mapping and parametrization wiring `tests/integration/defs/disaggregated/test_disaggregated.py`	Configuration mapping for stress-test variants wires `gpt_oss_120b_eagle_triton_stress` and `gpt_oss_120b_triton_stress` to the new TRITON configs; `TestConfig` parametrization is updated to reference these new variant names.
Test selection list and waiver cleanup `tests/integration/defs/.test_durations`, `tests/integration/test_lists/qa/llm_function_core.txt`, `llm_function_rtx6k.txt`, `llm_function_stress.txt`, `test-db/l0_l40s.yml`, `waives.txt`	Phi LoRA 1-GPU duration entry removed; test entries for eagle3 tp8, Phi LoRA, and disaggregated stress variants removed from QA test lists; corresponding `SKIP` waivers removed from `waives.txt`; stress test list updated to reference new TRITON variants.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14596: Both PRs modify tests/integration/test_lists/waives.txt by changing which GPTOSS-related integration tests are waived.
NVIDIA/TensorRT-LLM#14454: Updates disaggregated test infrastructure that relates to stress-test configuration mappings modified in this PR.
NVIDIA/TensorRT-LLM#14653: Both PRs directly modify tests/integration/test_lists/waives.txt by changing waived integration tests.

Suggested reviewers

jieli-matrix
niukuo
chenfeiz0326

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is essentially the template with minimal content filled in; the explicit 'Description' and 'Test Coverage' sections are empty, lacking explanation of the issue and solution.	Fill in the 'Description' section explaining the stress test issue being fixed and why switching to TRITON backend resolves it. Add 'Test Coverage' section listing affected test cases.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly references a specific NVBugs ticket and indicates a test fix for stress test issues on H100, which aligns with the changes switching backends from CUTLASS to TRITON in stress test configurations.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2026-05-29T03:03:31Z

/bot run --skip-test

tensorrt-cicd · 2026-05-29T03:10:59Z

PR_Github #50945 [ run ] triggered by Bot. Commit: 5bacdcf Link to invocation

dongfengy

LGTM for the MOE change part

tensorrt-cicd · 2026-05-29T06:10:43Z

PR_Github #50945 [ run ] completed with state SUCCESS. Commit: 5bacdcf
/LLM/main/L0_MergeRequest_PR pipeline #40402 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

tensorrt-cicd · 2026-05-29T06:33:43Z

/bot reuse-pipeline

xinhe-nv · 2026-05-29T06:36:26Z

@dongfengy

the triton + eagle case is still failure on H100, https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/view/TRT-LLM-Function-Pipelines/job/DEBUG_LLM_FUNCTION_CLUSTER_TEST/1566/testReport/H100.disaggregated/test_disaggregated/test_disaggregated_stress_test_input8k_output1k_conc512_gpt_oss_120b_eagle_triton_stress_/, is it expected value?
triton backend is hang on H100, https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/DEBUG_LLM_FUNCTION_TEST/2458/console
trtion backend is hang on B200, https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/view/TRT-LLM-Function-Pipelines/job/DEBUG_LLM_FUNCTION_CLUSTER_TEST/1571/testReport/B200.disaggregated/test_disaggregated/test_disaggregated_stress_test_input8k_output1k_conc512_gpt_oss_120b_triton_stress_/

xinhe-nv · 2026-06-01T09:06:21Z

/bot run --skip-test

tensorrt-cicd · 2026-06-01T09:15:56Z

PR_Github #51345 [ run ] triggered by Bot. Commit: 1d1e5bb Link to invocation

tensorrt-cicd · 2026-06-01T09:33:27Z

PR_Github #51345 [ run ] completed with state FAILURE. Commit: 1d1e5bb
/LLM/main/L0_MergeRequest_PR pipeline #40757 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-06-01T09:40:01Z

/bot run --skip-test

tensorrt-cicd · 2026-06-01T09:46:34Z

PR_Github #51351 [ run ] triggered by Bot. Commit: a1d6a37 Link to invocation

tensorrt-cicd · 2026-06-01T10:10:40Z

PR_Github #51351 [ run ] completed with state FAILURE. Commit: a1d6a37
/LLM/main/L0_MergeRequest_PR pipeline #40762 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv · 2026-06-02T01:29:36Z

/bot run --skip-test

tensorrt-cicd · 2026-06-02T01:36:39Z

PR_Github #51463 [ run ] triggered by Bot. Commit: 3293fb7 Link to invocation

tensorrt-cicd · 2026-06-02T05:28:11Z

PR_Github #51463 [ run ] completed with state SUCCESS. Commit: 3293fb7
/LLM/main/L0_MergeRequest_PR pipeline #40872 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

xinhe-nv · 2026-06-02T05:28:38Z

/bot reuse-pipeline

tensorrt-cicd · 2026-06-02T05:34:14Z

PR_Github #51509 [ reuse-pipeline ] triggered by Bot. Commit: c371f6b Link to invocation

xinhe-nv · 2026-06-02T05:47:21Z

/bot reuse-pipeline

tensorrt-cicd · 2026-06-02T05:50:48Z

PR_Github #51509 [ reuse-pipeline ] completed with state SUCCESS. Commit: c371f6b
Reusing PR_Github #51463 (Partly Tested) for commit c371f6b

Link to invocation

tensorrt-cicd · 2026-06-02T05:52:50Z

PR_Github #51514 [ reuse-pipeline ] triggered by Bot. Commit: 2391872 Link to invocation

tensorrt-cicd · 2026-06-02T06:04:36Z

PR_Github #51514 [ reuse-pipeline ] completed with state SUCCESS. Commit: 2391872
Reusing PR_Github #51463 (Partly Tested) for commit 2391872

Link to invocation

xinhe-nv requested review from a team as code owners May 29, 2026 02:59

github-actions Bot assigned xinhe-nv May 29, 2026

xinhe-nv requested a review from dongfengy May 29, 2026 03:02

xinhe-nv enabled auto-merge (squash) May 29, 2026 03:02

dongfengy approved these changes May 29, 2026

View reviewed changes

xinhe-nv force-pushed the fix branch from 2e5a891 to b5a3e64 Compare June 1, 2026 02:26

StanleySun639 approved these changes Jun 1, 2026

View reviewed changes

xinhe-nv added 2 commits June 1, 2026 17:39

immprove gpt-oss stress tests

3454b2a

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

update stress tests

a1d6a37

Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv force-pushed the fix branch from 85a2690 to a1d6a37 Compare June 1, 2026 09:39

Merge branch 'main' into fix

3293fb7

Merge branch 'main' into fix

14840df

Merge branch 'main' into fix

c371f6b

Merge branch 'main' into fix

2391872

xinhe-nv merged commit f57f4aa into NVIDIA:main Jun 2, 2026
7 checks passed

xinhe-nv deleted the fix branch June 2, 2026 06:09

This was referenced Jun 4, 2026

[https://nvbugs/6222480][fix] Fix stress #14949

Merged

[https://nvbugs/5546507][test] Remove obsolet… #14995

Merged

Uh oh!

Conversation

xinhe-nv commented May 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented May 29, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

dongfengy left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

xinhe-nv commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xinhe-nv commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

xinhe-nv commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

xinhe-nv commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

xinhe-nv commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

xinhe-nv commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xinhe-nv commented May 29, 2026 •

edited by coderabbitai Bot

Loading

xinhe-nv commented May 29, 2026 •

edited

Loading