[https://nvbugs/6463987][perf] Revert NGC PyTorch 26.05 stack upgrade for GB300 GLM-5-fp4 ctx_only NVFP4 regression by chenfeiz0326 · Pull Request #16601 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-07-20T06:19:50Z

Description

Fixes a ~32% output-token-throughput regression on GB300 for
ctx_only-gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL
by reverting the version pins from PR #15087 (NGC PyTorch 26.04 → 26.05
container upgrade).

Bug: https://nvbugs/6463987
Culprit: [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack #15087 (commit 02cedf6e4e, [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack)
Test case: ctx_only-gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL

Measured perf (3-rep same-node medians, oci-aga GB300)

Wheel	median (tok/s)	reps	CV
good `1662a877`	4.07	[4.04, 4.23, 4.07]	1.8%
bad `46bf21b1`	2.75	[2.84, 2.72, 2.75]	1.8%
ToT `3fe4ceb9` (post-merge 2844)	2.58	[2.58, 2.56, 2.63]	1.3%
fix (this PR)	4.16	[4.15, 4.23, 4.16]	1.0%

Fix clears the (good+bad)/2 = 3.41 threshold by +22% and recovers
+61% over current ToT.

Root cause

PR #15087 bumped cuBLAS 13.4.0.1 → 13.4.1.2, cuDNN 9.21 → 9.22, and
NCCL 2.29.7 → 2.30.4 in a single container-image swap. The
bug_profile module ranking (from the automated triage) points at
cuBLAS 13.4.1.2's NVFP4 heuristic selecting a slower kernel path on
GB300 for this shape (primary); NCCL 2.30's attention-DP allgather
changes contribute a tertiary effect. Startup time actually got FASTER
between good and bad (7m10s → 5m50s), which rules out JIT/warmup and
pins the loss to steady-state prefill GEMMs.

Changes (infra only, three files, no source touched)

docker/common/install_tensorrt.sh: cuBLAS 13.4.1.2 → 13.4.0.1,
cuDNN 9.22.0.52 → 9.21.0.82, NCCL 2.30.4 → 2.29.7. Comment URLs back
to the 26.04 release notes.
jenkins/current_image_tags.properties: container image tags back
to the 26.04 base (202607100544-16206) for x86, sbsa, and both
rockylinux variants.
requirements.txt: torch upper bound 2.13.0a0 → 2.12.0a0,
nvidia-nccl-cu13 upper bound 2.30.4 → 2.29.7.

Test plan

CI: L0_PostMerge GB300 GLM-5-fp4 perf-sanity should recover
to good-commit levels.
Adjacent perf-sanity cases on the same container should be
unaffected or improved (the container revert is symmetric).
Functional CI on x86 + sbsa; verify wheel builds against the
26.04 base image tags.

🤖 Generated with Repair Bot

Summary by CodeRabbit

Compatibility
- Updated NVIDIA CUDA, cuDNN, NCCL, and cuBLAS component versions for improved alignment with the supported PyTorch release.
- Refined PyTorch and NCCL version constraints to ensure compatible installations.
Build and Deployment
- Updated container and wheel image references to the latest validated build set across supported environments.

… (cuBLAS/cuDNN/NCCL) for GB300 GLM-5-fp4 ctx_only NVFP4 regression PR NVIDIA#15087 (commit 02cedf6) upgraded the NGC PyTorch base container 26.04 -> 26.05 and pinned cuBLAS 13.4.0.1 -> 13.4.1.2, cuDNN 9.21.0.82 -> 9.22.0.52, NCCL 2.29.7 -> 2.30.4. On the GB300 ctx_only-gb300_glm-5-fp4_1k1k perf-sanity workload this triggers a ~32% output_token_throughput regression (4.07 -> 2.75 tok/s on oci-aga GB300, 3-rep same-node medians, CV 1.8%). The regression is still present on origin/main HEAD (3fe4ceb, post-merge 2844) measured at 2.58 tok/s (list [2.58, 2.56, 2.63], CV 1.3%) via auto_perf_stabilize's 3-rep reproduce on a clean unpatched wheel. The bug_profile module ranking pins the loss to steady-state prefill GEMMs on the ctx_only NVFP4 workload: cuBLAS 13.4.1.2's NVFP4 heuristic (primary) plus NCCL 2.30 attention-DP allgather changes (tertiary). Startup got FASTER (7m10s -> 5m50s) between good and bad commits which rules out JIT/warmup and pins the loss to steady-state. This diff is a minimal revert of the version pins from PR NVIDIA#15087. No TensorRT-LLM source is touched. Verified on oci-aga GB300 in the auto-perf-fix workflow: 3-rep list [4.15, 4.23, 4.16] median 4.16 tok/s -- clears the (good+bad)/2 threshold 3.41 tok/s by +22% and recovers +61% over current ToT (2.58 tok/s). Changes (infra only, three files): * docker/common/install_tensorrt.sh: cuBLAS 13.4.1.2 -> 13.4.0.1, cuDNN 9.22.0.52 -> 9.21.0.82, NCCL 2.30.4 -> 2.29.7. Aligns comment URLs with the 26.04 release notes. * jenkins/current_image_tags.properties: pins container image tags back to the 26.04 base (202607100544-16206) for x86, sbsa, and both rockylinux variants. * requirements.txt: torch upper bound 2.13.0a0 -> 2.12.0a0, nvidia-nccl-cu13 upper bound 2.30.4 -> 2.29.7. Comment URLs aligned with 26.04. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

coderabbitai · 2026-07-20T06:30:47Z

📝 Walkthrough

Walkthrough

Updated CUDA dependency pins, PyTorch package constraints, release-note references, and five LLM Docker image tags to align with the selected PyTorch and CUDA 13.2 releases.

Changes

CUDA and PyTorch release alignment

Layer / File(s)	Summary
CUDA dependency and package constraints `docker/common/install_tensorrt.sh`, `requirements.txt`	Pinned cuDNN, NCCL, CUBLAS, `torch`, and `nvidia-nccl-cu13` to the updated release versions and adjusted the related release-note references.
LLM image tag updates `jenkins/current_image_tags.properties`	Updated five LLM Docker image properties to the corresponding `pytorch-26.04` image builds and new tag suffixes.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: mlefeb01

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: reverting the NGC PyTorch 26.05 stack upgrade to fix the GB300 NVFP4 regression.
Description check	✅ Passed	It covers the issue, root cause, changes, and test plan, so it mostly satisfies the template despite not using the exact checklist format.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@jenkins/current_image_tags.properties`:
- Around line 16-20: Add the repository-standard NVIDIA copyright header to the
modified current_image_tags.properties file, including the current year 2026,
while preserving all existing image variable definitions unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8522fa0e-aa5b-491b-a2c5-0d1c50eaa123

📥 Commits

Reviewing files that changed from the base of the PR and between 447a628 and 89a81c8.

📒 Files selected for processing (3)

docker/common/install_tensorrt.sh
jenkins/current_image_tags.properties
requirements.txt

coderabbitai · 2026-07-20T06:30:51Z

+LLM_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-x86_64-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206
+LLM_SBSA_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-aarch64-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206
+LLM_ROCKYLINUX8_PY310_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:cuda-13.2.1-devel-rocky8-x86_64-rocky8-py310-trt10.16.1.11-skip-tritondevel-202607100544-16206
+LLM_ROCKYLINUX8_PY312_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:cuda-13.2.1-devel-rocky8-x86_64-rocky8-py312-trt10.16.1.11-skip-tritondevel-202607100544-16206
+LLM_SBSA_WHEEL_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:cuda-13.2.1-devel-ubuntu24.04-sbsa-ubuntu24.04-py312-trt10.16.1.11-skip-tritondevel-202607100544-16206


📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add or update the NVIDIA copyright header for this modified file.

The supplied file has no visible NVIDIA copyright header or current-year notice. Add the repository-standard header and update it for 2026.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/current_image_tags.properties` around lines 16 - 20, Add the repository-standard NVIDIA copyright header to the modified current_image_tags.properties file, including the current year 2026, while preserving all existing image variable definitions unchanged.

Source: Coding guidelines

chzblych · 2026-07-20T06:40:46Z

-CUDNN_VER="9.22.0.52-1"
-NCCL_VER="2.30.4-1+cuda13.2"
-CUBLAS_VER="13.4.1.2-1"
+CUDNN_VER="9.21.0.82-1"


@chenfeiz0326 We need to catch up with the latest NGC PyTorch base image for several reasons, especially because we'll soon need a newer base image to support upcoming hardware. The current approach doesn't scale well if it requires us to remain on an older base image.

If cuBLAS is the main blocker, could we pin cuBLAS to a known-good version while continuing to upgrade the NGC PyTorch base image?

We should also file a cuBLAS issue so the underlying problem can be properly investigated and fixed. Pinning cuBLAS may work as a temporary mitigation, but it isn't a sustainable long-term solution, especially since upcoming hardware may require a newer cuBLAS version.

chzblych · 2026-07-20T06:44:04Z

-LLM_ROCKYLINUX8_PY312_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:cuda-13.2.1-devel-rocky8-x86_64-rocky8-py312-trt10.16.1.11-skip-tritondevel-202607151440-16194
-LLM_SBSA_WHEEL_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:cuda-13.2.1-devel-ubuntu24.04-sbsa-ubuntu24.04-py312-trt10.16.1.11-skip-tritondevel-202607151440-16194
+LLM_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-x86_64-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206
+LLM_SBSA_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-aarch64-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206


Suggested change

LLM_SBSA_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-aarch64-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206

LLM_SBSA_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-sbsa-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206

Let's use sbsa going forward.

chenfeiz0326 · 2026-07-20T11:40:26Z

Superseded by #16623 which applies the reviewer's recommended surgical approach: pin cuBLAS 13.4.0.1 in docker/common/install_tensorrt.sh while keeping the 26.05 base image, cuDNN 9.22, NCCL 2.30, and torch 2.13a0 upper bound. Full details, root-cause evidence, and same-image verification (baseline 2.84 → fix 4.19 on healthy node) are in #16623.

Closing this PR.

…GB300 NVFP4 GEMM heuristic regression cuBLAS 13.4.1.2 (introduced by PR NVIDIA#15087 in the 26.05 stack upgrade) picks a slower NVFP4 GEMM plan on GB300 for the GLM-5 dense-attention prefill shapes, driving output_token_throughput -32% (4.07 -> 2.75) and per-request TTFT +42% (247 ms -> 352 ms) on aggr_upload-ctx_only-gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL. This is a surgical alternative to NVIDIA#16601, which fully reverted the 26.05 -> 26.04 stack (container tag + cuBLAS + cuDNN + NCCL + torch upper bound in a single blob). Per reviewer feedback, we prefer to pin cuBLAS specifically while keeping the newer NGC base image, so downstream fixes in cuDNN 9.22, NCCL 2.30, and torch 2.13a0 are retained. Only docker/common/install_tensorrt.sh changes: CUBLAS_VER="13.4.1.2-1" -> "13.4.0.1-1" The header/lib symlink guards added in NVIDIA#15087 (CUBLAS_HDR_DIR / CUBLAS_LIB_DIR checks) are already conditional (if [ -d ... ]) so they no-op cleanly when the older cuBLAS is installed. No other code paths change. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 requested review from a team as code owners July 20, 2026 06:19

chenfeiz0326 requested review from ZhanruiSunCh and mlefeb01 July 20, 2026 06:19

github-actions Bot assigned chenfeiz0326 Jul 20, 2026

coderabbitai Bot reviewed Jul 20, 2026

View reviewed changes

chzblych reviewed Jul 20, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Jul 20, 2026

[https://nvbugs/6445494][infra] Upgrade Triton to 3.7.0 for Torch 2.12.0 compatibility #16303

Open

3 tasks

chenfeiz0326 mentioned this pull request Jul 20, 2026

[https://nvbugs/6463987][perf] Pin cuBLAS to 13.4.0.1 to work around GB300 NVFP4 GEMM heuristic regression #16623

Closed

4 tasks

chenfeiz0326 closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[https://nvbugs/6463987][perf] Revert NGC PyTorch 26.05 stack upgrade for GB300 GLM-5-fp4 ctx_only NVFP4 regression - #16601

[https://nvbugs/6463987][perf] Revert NGC PyTorch 26.05 stack upgrade for GB300 GLM-5-fp4 ctx_only NVFP4 regression#16601
chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:repair-bot-bug6463987

chenfeiz0326 commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 20, 2026

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 20, 2026

Uh oh!

chzblych Jul 20, 2026

Uh oh!

chzblych Jul 20, 2026

Uh oh!

chzblych Jul 20, 2026 •

edited

Loading

Uh oh!

chenfeiz0326 commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	LLM_SBSA_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-aarch64-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206
	LLM_SBSA_DOCKER_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-26.04-py3-sbsa-ubuntu24.04-trt10.16.1.11-skip-tritondevel-202607100544-16206

Uh oh!

Conversation

chenfeiz0326 commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Measured perf (3-rep same-node medians, oci-aga GB300)

Root cause

Changes (infra only, three files, no source touched)

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 20, 2026

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

chzblych Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

chzblych Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

chzblych Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chenfeiz0326 commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

chzblych Jul 20, 2026 •

edited

Loading