[None][fix] Add AutoDeploy post-merge stages by hnover-nv · Pull Request #15418 · NVIDIA/TensorRT-LLM

hnover-nv · 2026-06-16T16:09:56Z

Summary by CodeRabbit

Tests
- Enhanced post-merge testing pipeline with additional automated test stages for comprehensive validation.
- Updated performance testing configuration for improved accuracy across hardware variants.

Description

Add missing AutoDeploy post-merge Jenkins stages for test-db blocks that already declare stage: post_merge and backend: autodeploy:

H100_PCIe-AutoDeploy-Post-Merge-1 for l0_h100
DGX_B200-AutoDeploy-Post-Merge-1 for l0_b200
DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1 for l0_dgx_b200

These mirror the existing pre-merge AutoDeploy stage configurations for the same YAML files and GPU counts. Without these stages, the corresponding post-merge AutoDeploy test-db blocks are present but not selected by Jenkins unless the same tests also appear in another reachable block.

Also remove one duplicate 4-GPU B200 AutoDeploy perf sanity entry from the post-merge block. That test remains in the pre-merge AutoDeploy block, which is also run during post-merge, so keeping it in both places would make it run twice after this PR makes the post-merge block reachable.

After enabling the previously unreachable B200 AutoDeploy post-merge blocks, PR CI exposed an existing Nano NVFP4 accuracy failure:

accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[nvfp4-1-attn_dp_off-trtllm]: GSM8K accuracy below threshold.

This PR temporarily waives that test so the stage wiring fix can land while the underlying accuracy regression is investigated separately. The comparable pre-move single-GPU test previously ran as test_accuracy[nvfp4-1-trtllm]; the newer pytest id comes from the later attention-DP parameter split.

Test Coverage

Ran a local one-off reachability check over jenkins/L0_Test.groovy and tests/integration/test_lists/test-db/*.yml for AutoDeploy post-merge blocks.
- Before this fix: 3 missing AutoDeploy post-merge blocks.
- After this fix: 5 matched AutoDeploy post-merge blocks, 0 missing.
Checked pre/post duplicate overlap for the newly covered AutoDeploy stage shapes:
- l0_h100.yml, 1 GPU: 0 overlapping tests.
- l0_b200.yml, 1 GPU: 0 overlapping tests.
- l0_dgx_b200.yml, 4 GPUs: 0 overlapping tests after removing the duplicate perf sanity entry.
Parsed tests/integration/test_lists/test-db/l0_dgx_b200.yml with PyYAML after editing.
Ran ./scripts/check_test_list.py --check-duplicate-waives.
Remote PR CI was run with:
- /bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"
- Build #43691 selected the new post-merge stages and exposed the waived 1-GPU Nano NVFP4 accuracy failure.

No runtime CI was run locally; this is a Jenkins stage mapping, test-list cleanup, and temporary waive change.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

hnover-nv · 2026-06-16T18:31:47Z

/bot run --post-merge

coderabbitai · 2026-06-16T18:35:01Z

📝 Walkthrough

Walkthrough

Three new post-merge AutoDeploy stage mappings are added to launchTestJobs() in the Jenkins pipeline for H100 PCIe and DGX B200 (both single and 4-GPU configurations). Additionally, the 4-GPU DGX B200 AutoDeploy perf sanity test list replaces the super_ad aggregated upload variant with the super_mtp_ad variant.

Changes

AutoDeploy Post-Merge CI Configuration

Layer / File(s)	Summary
New Jenkins post-merge AutoDeploy stage mappings `jenkins/L0_Test.groovy`	Adds `H100_PCIe-AutoDeploy-Post-Merge-1` (x86, `h100-cr`, `l0_h100`, split `1,1`), `DGX_B200-AutoDeploy-Post-Merge-1` (x86 SLURM, `auto:dgx-b200-flex`, `l0_b200`, split `1,1,1,1`), and `DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1` (x86 SLURM, `auto:dgx-b200-flex`, `l0_dgx_b200`, split `1,1,4,1`) to `launchTestJobs()`.
DGX B200 4-GPU perf sanity test variant swap `tests/integration/test_lists/test-db/l0_dgx_b200.yml`	Replaces the `aggr_upload-super_ad_blackwell-super_ad_ws4_1k1k` `test_e2e` variant with `aggr_upload-super_mtp_ad_blackwell-super_mtp_ad_ws4_1k1k` in the 4-GPU `post_merge` AutoDeploy perf sanity block.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly and concisely describes the main change: adding AutoDeploy post-merge stages. It follows the required format [None][fix] and accurately reflects the primary objective of the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR includes clear Description, Test Coverage, and Checklist sections and explains what changed and why.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-06-16T18:38:35Z

PR_Github #54654 [ run ] triggered by Bot. Commit: df21810 Link to invocation

tburt-nv · 2026-06-16T19:06:54Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tburt-nv · 2026-06-16T19:07:49Z

#15418 (comment) would launch the entire post-merge pipeline, but we only need to run these stages.

tensorrt-cicd · 2026-06-16T19:14:04Z

PR_Github #54660 [ run ] triggered by Bot. Commit: df21810 Link to invocation

hnover-nv · 2026-06-16T19:14:12Z

#15418 (comment) would launch the entire post-merge pipeline, but we only need to run these stages.

I wanted to make sure the normal post-merge pipeline picked the stages up. While it should be enough the stages exist, it seemed possible something else was mis-wired so that they could be directly invoked but would not auto-run. but if you think it looks ok and we just need to check that the tests are still green (they were off for a week or two) then that's find with me, we can check manually next week or whatever that they ran.

tensorrt-cicd · 2026-06-16T19:17:35Z

PR_Github #54654 [ run ] completed with state ABORTED. Commit: df21810

Link to invocation

tensorrt-cicd · 2026-06-16T20:52:02Z

PR_Github #54660 [ run ] completed with state FAILURE. Commit: df21810
/LLM/main/L0_MergeRequest_PR pipeline #43691 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-16T23:50:06Z

/bot run --disable-fail-fast

github-actions · 2026-06-16T23:55:45Z

👎 Promotion blocked, new vulnerability found

Vulnerability report

Component	Vulnerability	Description	Severity
pytorch	CVE-2025-3000	A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function torch.jit.script. The manipulation leads to memory corruption. It is possible to launch the attack on the local host. The exploit has been disclosed to the public and may be used.	MEDIUM

hnover-nv · 2026-06-22T19:05:41Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-22T19:11:35Z

PR_Github #55073 [ run ] triggered by Bot. Commit: b0dbcce Link to invocation

tensorrt-cicd · 2026-06-22T22:41:12Z

PR_Github #55073 [ run ] completed with state FAILURE. Commit: b0dbcce
/LLM/main/L0_MergeRequest_PR pipeline #44060 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-23T04:08:06Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

github-actions · 2026-06-23T04:08:12Z

⚠️ Bot command ignored: The /bot command must appear at the very beginning of the comment (no leading blank lines or spaces). Please post a new comment with /bot as the first character.

hnover-nv · 2026-06-23T04:18:27Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-23T04:23:55Z

PR_Github #55152 [ run ] triggered by Bot. Commit: 000fbce Link to invocation

tensorrt-cicd · 2026-06-29T18:44:50Z

PR_Github #56440 [ kill ] completed with state SUCCESS. Commit: 0644801
Successfully killed previous jobs for commit 0644801

Link to invocation

hnover-nv · 2026-06-29T18:51:12Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-29T18:57:32Z

PR_Github #56443 [ run ] triggered by Bot. Commit: 0644801 Link to invocation

tensorrt-cicd · 2026-06-29T20:41:31Z

PR_Github #56443 [ run ] completed with state FAILURE. Commit: 0644801
/LLM/main/L0_MergeRequest_PR pipeline #45285 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-29T21:04:06Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-29T21:12:06Z

PR_Github #56459 [ run ] triggered by Bot. Commit: acb1d19 Link to invocation

tensorrt-cicd · 2026-06-29T22:20:59Z

PR_Github #56459 [ run ] completed with state FAILURE. Commit: acb1d19
/LLM/main/L0_MergeRequest_PR pipeline #45300 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-30T00:06:46Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

hnover-nv · 2026-06-30T00:50:26Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

hnover-nv · 2026-06-30T06:38:27Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-30T06:44:41Z

PR_Github #56524 [ run ] triggered by Bot. Commit: acb1d19 Link to invocation

tensorrt-cicd · 2026-06-30T10:48:49Z

PR_Github #56524 [ run ] completed with state SUCCESS. Commit: acb1d19
/LLM/main/L0_MergeRequest_PR pipeline #45362 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

hnover-nv · 2026-06-30T16:32:04Z

/bot run

tensorrt-cicd · 2026-06-30T16:37:51Z

PR_Github #56662 [ run ] triggered by Bot. Commit: acb1d19 Link to invocation

tensorrt-cicd · 2026-06-30T16:47:35Z

PR_Github #56662 [ run ] completed with state FAILURE. Commit: acb1d19
/LLM/main/L0_MergeRequest_PR pipeline #45486 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-30T17:01:42Z

/bot run

tensorrt-cicd · 2026-06-30T17:11:18Z

PR_Github #56674 [ run ] triggered by Bot. Commit: acb1d19 Link to invocation

tensorrt-cicd · 2026-06-30T19:22:00Z

PR_Github #56674 [ run ] completed with state SUCCESS. Commit: acb1d19
/LLM/main/L0_MergeRequest_PR pipeline #45497 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-30T19:23:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-30T19:29:42Z

PR_Github #56704 [ run ] triggered by Bot. Commit: acb1d19 Link to invocation

tensorrt-cicd · 2026-07-01T04:18:06Z

PR_Github #56704 [ run ] completed with state SUCCESS. Commit: acb1d19
/LLM/main/L0_MergeRequest_PR pipeline #45527 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-07-01T04:19:37Z

/bot run

tensorrt-cicd · 2026-07-01T04:24:57Z

PR_Github #56826 [ run ] triggered by Bot. Commit: acb1d19 Link to invocation

tensorrt-cicd · 2026-07-01T07:44:04Z

PR_Github #56826 [ run ] completed with state SUCCESS. Commit: acb1d19
/LLM/main/L0_MergeRequest_PR pipeline #45642 completed with status: 'SUCCESS'

CI Report

Link to invocation

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

github-actions Bot assigned hnover-nv Jun 16, 2026

hnover-nv changed the title ~~[None][infra] Add AutoDeploy post-merge stages~~ [None][fix] Add AutoDeploy post-merge stages Jun 16, 2026

hnover-nv force-pushed the hnover/ad-post-merge-stages branch 2 times, most recently from 37f9c2a to df21810 Compare June 16, 2026 18:18

hnover-nv marked this pull request as ready for review June 16, 2026 18:32

hnover-nv requested a review from a team June 16, 2026 18:32

hnover-nv requested a review from a team as a code owner June 16, 2026 18:32

hnover-nv requested review from tburt-nv and yiqingy0 June 16, 2026 18:32

govind-ramnarayan approved these changes Jun 16, 2026

View reviewed changes

tburt-nv approved these changes Jun 16, 2026

View reviewed changes

hnover-nv force-pushed the hnover/ad-post-merge-stages branch from df21810 to 0b1f948 Compare June 16, 2026 23:49

hnover-nv merged commit 4cfcc64 into NVIDIA:main Jul 1, 2026
10 checks passed

evezhier pushed a commit to evezhier/TensorRT-LLM that referenced this pull request Jul 2, 2026

[None][fix] Add AutoDeploy post-merge stages (NVIDIA#15418)

4cbf7d7

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

hnover-nv deleted the hnover/ad-post-merge-stages branch July 7, 2026 22:05

coderabbitai Bot mentioned this pull request Jul 8, 2026

[TRTLLM-13696][test] Part1: Add CPU only CI stage #15815

Merged

1 task

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[None][fix] Add AutoDeploy post-merge stages (NVIDIA#15418)

06ddd54

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

coderabbitai Bot mentioned this pull request Jul 27, 2026

[None][perf] Preserve default V2 KV cache pool sizing #16783

Open

1 task

Uh oh!

Conversation

hnover-nv commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

hnover-nv commented Jun 16, 2026

Uh oh!

coderabbitai Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tburt-nv commented Jun 16, 2026

Uh oh!

tburt-nv commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hnover-nv commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hnover-nv commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

👎 Promotion blocked, new vulnerability found

Vulnerability report

Uh oh!

hnover-nv commented Jun 22, 2026

Uh oh!

tensorrt-cicd commented Jun 22, 2026

Uh oh!

tensorrt-cicd commented Jun 22, 2026

Uh oh!

hnover-nv commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

hnover-nv commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 30, 2026

Uh oh!

hnover-nv commented Jun 30, 2026

Uh oh!

hnover-nv commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

hnover-nv commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

hnover-nv commented Jun 16, 2026 •

edited

Loading

coderabbitai Bot commented Jun 16, 2026 •

edited

Loading