[None][test] CI-only: RUN (not skip) gb200 deepseek-v32 gen-only perf-sanity at 33b0a32 (DO NOT MERGE) by tensorrt-cicd · Pull Request #15224 · NVIDIA/TensorRT-LLM

tensorrt-cicd · 2026-06-10T15:16:59Z

CI-only PR — DO NOT MERGE.

Purpose: actually run (not skip) the case
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_1k1k_con2048_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL]
against commit 33b0a32 (DeepGemmFusedMoE Triton fuse, #14592).

Why the base was changed from main:
Pre-merge CI tests the merge result, which inherits the base branch's waives.txt.
main's waives.txt waives this exact case (line ~295, nvbugs/6280649), so a PR based on
main skips it and reports a false "success."

Fix: base is now chenfeiz/test-v32-gen-33b0a32-base, a branch pinned at exactly 33b0a32
that does not waive this case. The head branch is 33b0a32 + a trivial no-op edit, so:

code under test = 33b0a32
merge result's waives.txt does not skip the case → the case actually runs.

33b0a32 Empty commit on top of 33b0a32 (tree unchanged) to open a PR that runs: perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only- gb200_deepseek-v32-fp4_1k1k_con2048_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL] 33b0a32 is a known-good commit for this case (it passed in post-merge history). This PR re-runs the same case in pre-merge CI to check whether the gen-server CUDA-graph capture crash (cudaErrorLaunchFailure) reproduces on known-good code, which would indicate the failure is environmental (machine status / node env) rather than related to TRT-LLM code. Do not merge. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-06-10T15:19:39Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-10T15:27:25Z

PR_Github #53328 [ ] completed with state FAILURE. Commit: ``

Link to invocation

chenfeiz0326 · 2026-06-11T02:24:51Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-11T02:31:31Z

PR_Github #53442 [ run ] triggered by Bot. Commit: 1efaf53 Link to invocation

tensorrt-cicd · 2026-06-11T03:58:52Z

PR_Github #53442 [ run ] completed with state SUCCESS. Commit: 1efaf53
/LLM/main/L0_MergeRequest_PR pipeline #42609 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-06-13T02:04:46Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-13T02:11:00Z

PR_Github #53986 [ run ] triggered by Bot. Commit: 1efaf53 Link to invocation

tensorrt-cicd · 2026-06-13T02:54:20Z

PR_Github #53986 [ run ] completed with state SUCCESS. Commit: 1efaf53
/LLM/main/L0_MergeRequest_PR pipeline #43072 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

…en-only perf-sanity No-op comment in test_perf_sanity.py so the pre-merge stage GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4 re-runs instead of being skipped as already-passed. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

tensorrt-cicd · 2026-06-13T05:46:08Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

chenfeiz0326 · 2026-06-13T09:22:39Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-13T09:29:11Z

PR_Github #54035 [ run ] triggered by Bot. Commit: e319a32 Link to invocation

tensorrt-cicd · 2026-06-13T10:24:52Z

PR_Github #54035 [ run ] completed with state SUCCESS. Commit: e319a32
/LLM/main/L0_MergeRequest_PR pipeline #43120 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

tensorrt-cicd · 2026-06-13T11:43:24Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

chenfeiz0326 · 2026-06-13T11:44:27Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-13T11:50:19Z

PR_Github #54042 [ run ] triggered by Bot. Commit: e319a32 Link to invocation

tensorrt-cicd · 2026-06-13T12:08:07Z

PR_Github #54042 [ run ] completed with state FAILURE. Commit: e319a32
/LLM/main/L0_MergeRequest_PR pipeline #43126 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…v32 gen-only perf-sanity Bumps the no-op marker in test_perf_sanity.py so the pre-merge stage GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4 re-runs instead of being skipped as already-passed. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-06-14T15:02:59Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-14T15:09:42Z

PR_Github #54131 [ run ] triggered by Bot. Commit: cace795 Link to invocation

tensorrt-cicd · 2026-06-14T17:27:18Z

PR_Github #54131 [ run ] completed with state FAILURE. Commit: cace795
/LLM/main/L0_MergeRequest_PR pipeline #43215 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-06-15T06:02:09Z

/bot run --disable-fail-fast --stage-list "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

tensorrt-cicd · 2026-06-15T06:08:21Z

PR_Github #54231 [ run ] triggered by Bot. Commit: cace795 Link to invocation

tensorrt-cicd · 2026-06-15T07:30:37Z

PR_Github #54231 [ run ] completed with state SUCCESS. Commit: cace795
/LLM/main/L0_MergeRequest_PR pipeline #43307 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-06-15T13:35:20Z

/bot run --disable-fail-fast --stage-list "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2"

tensorrt-cicd · 2026-06-15T13:41:31Z

PR_Github #54307 [ run ] triggered by Bot. Commit: cace795 Link to invocation

tensorrt-cicd · 2026-06-15T16:09:01Z

PR_Github #54307 [ run ] completed with state FAILURE. Commit: cace795
/LLM/main/L0_MergeRequest_PR pipeline #43378 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

github-actions Bot assigned tensorrt-cicd Jun 10, 2026

tensorrt-cicd changed the base branch from main to chenfeiz/test-v32-gen-33b0a32-base June 13, 2026 11:42

tensorrt-cicd changed the title ~~[None][test] CI-only: re-run gb200 deepseek-v32 gen-only perf-sanity at 33b0a32 (DO NOT MERGE)~~ [None][test] CI-only: RUN (not skip) gb200 deepseek-v32 gen-only perf-sanity at 33b0a32 (DO NOT MERGE) Jun 13, 2026

chzblych deleted the branch NVIDIA:chenfeiz/test-v32-gen-33b0a32-base June 16, 2026 06:24

chzblych closed this Jun 16, 2026

Uh oh!

Conversation

tensorrt-cicd commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chenfeiz0326 commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

chenfeiz0326 commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

chenfeiz0326 commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

chenfeiz0326 commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

chenfeiz0326 commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

chenfeiz0326 commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

chenfeiz0326 commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

chenfeiz0326 commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tensorrt-cicd commented Jun 10, 2026 •

edited

Loading