[https://nvbugs/6463987][perf] Pin cuBLAS to 13.4.0.1 to work around GB300 NVFP4 GEMM heuristic regression - #16623
Conversation
📝 WalkthroughWalkthroughThe TensorRT installation script pins cuBLAS to ChangescuBLAS Installation Pin
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…GB300 NVFP4 GEMM heuristic regression cuBLAS 13.4.1.2 (introduced by PR NVIDIA#15087 in the 26.05 stack upgrade) picks a slower NVFP4 GEMM plan on GB300 for the GLM-5 dense-attention prefill shapes, driving output_token_throughput -32% (4.07 -> 2.75) and per-request TTFT +42% (247 ms -> 352 ms) on aggr_upload-ctx_only-gb300_glm-5-fp4_1k1k_con1_ctx1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL. This is a surgical alternative to NVIDIA#16601, which fully reverted the 26.05 -> 26.04 stack (container tag + cuBLAS + cuDNN + NCCL + torch upper bound in a single blob). Per reviewer feedback, we prefer to pin cuBLAS specifically while keeping the newer NGC base image, so downstream fixes in cuDNN 9.22, NCCL 2.30, and torch 2.13a0 are retained. Only docker/common/install_tensorrt.sh changes: CUBLAS_VER="13.4.1.2-1" -> "13.4.0.1-1" The header/lib symlink guards added in NVIDIA#15087 (CUBLAS_HDR_DIR / CUBLAS_LIB_DIR checks) are already conditional (if [ -d ... ]) so they no-op cleanly when the older cuBLAS is installed. No other code paths change. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
d71ef3f to
15b3722
Compare
|
/bot run --stage-list "Build-Docker-Images" |
|
PR_Github #60551 [ run ] triggered by Bot. Commit: |
|
PR_Github #60551 [ run ] completed with state
|
|
Perf verification result: the fix does not recover perf — the regression is 100% GB300 node variance. Re-ran 3 baseline + 3 fix with the same wheel and same 26.05 image on lyris GB300, collecting the runtime cuBLAS version in every job:
Baseline mean 3.75, fix mean 3.29 — both CV ~28%. Neither config is significantly different from the other; the 32% "regression" in the bug is entirely explained by GB300 node identity. Some nodes hit 4.2–4.5 tok/s regardless of cuBLAS version; other nodes hit 2.6–2.8 tok/s regardless of cuBLAS version. cuBLAS 13.4.0.1 vs 13.4.1.2 has no measurable perf effect on this workload. This also invalidates the verification for PR #16601: its 4.16 measurement was probably a fast node with the full-revert stack, but the same node would have measured ~4.2 with the original 26.05 stack too. The verification signal was not real. The original triage's own note (bug_repro's raw data on oci-aga same-node CV 1.8%) flagged this pattern — the tight in-config numbers came from the two commits landing on the SAME node, which the oci-aga scheduler happened to do. On lyris, when the scheduler lands the two configs on different nodes, the ~30% node-variance effect washes out any config-level signal. Closing this PR. The right next step is to route the bug to whoever owns the GB300 node-quality investigation on lyris/oci-aga, not to a cuBLAS version pin. Same for #16601 — it was closed as superseded by this PR, but its own perf claim was similarly node-variance-dominated. |
4 participants
Summary
Surgical alternative to #16601 for the GB300 GLM-5-NVFP4 ctx_only 32% throughput regression (nvbug 6463987).
Changes only one version pin in
docker/common/install_tensorrt.sh:Keeps everything else on the 26.05 stack:
LLM_*_DOCKER_IMAGEtags — unchanged (26.05 image kept)<=2.13.0a0— unchangednvidia-nccl-cu13<=2.30.4inrequirements.txt— unchangedRoot cause
PR #15087 rolled up cuBLAS 13.4.0.1 → 13.4.1.2, cuDNN 9.21 → 9.22, and NCCL 2.29.7 → 2.30.4 in one commit alongside the container bump. The delta on this bug is prefill-GEMM-shaped, not comm- or warmup-shaped:
output_token_throughput: 4.07 → 2.75 (-32%)nv26.04.48445190→nv26.05).The prefill GEMM shape class (GLM-5 dense-attention Q/K/V + MLP at 1024×hidden×dp_slice on GB300 SM100) is exactly what a per-shape cuBLAS heuristic table change would hit. Full triage lives in the bug's history report; the module ranking there is
gemm_quantization(primary),communication(secondary),gpu_kernel(tertiary).Why this is preferred over #16601
Per the reviewer's comment on #16601:
Comparison:
CUBLAS_VERbumpThe header/lib symlink guards added in #15087 (
if [ -d "${CUBLAS_HDR_DIR}" ]atinstall_tensorrt.sh:83and:96) are already conditional, so they no-op cleanly when the older cuBLAS is installed — no additional cleanup needed.Verification (lyris GB300, same wheel
tensorrt_llm-1.3.0rc21, cuda_arch=103-real, targetorigin/mainatc77bc6ed6)All four runs use the same 26.05 pyxis image and the same wheel. The only variable is cuBLAS runtime version — simulated by
apt-get install --allow-downgrades libcublas-13-2=13.4.0.1-1+ symlink repoint inside the container before pytest starts.readlink -f /usr/local/cuda/lib64/libcublas.sowas confirmed to resolve to.../libcublas.so.13.4.0.1in every fix run.output_token_throughputReference numbers from the bug (oci-aga same-node CV 1.8%):
The one fix run on a healthy node (theia0197: 4.19 > good_commit 4.07) shows full recovery. The other two fix nodes hit the same 3-ish range regardless of cuBLAS — consistent with the known GB300 node variance pattern where sub-3-tps/s workloads see wide per-node scatter that dominates any per-attempt measurement. The regression itself only reproduces cleanly on ~1/4 of nodes we've tested — hence the deep same-node methodology used in the original triage.
Baseline TTFT (351.7 ms) matches the bug's bad_commit_perf mean TTFT (352 ms) almost exactly, and fix-run2 TTFT (236.8 ms) is even better than the bug's good_commit_perf mean TTFT (247 ms). This gives the strongest signal that cuBLAS is the operative variable on the healthy node.
Follow-up (in parallel, not blocking)
CUBLAS_VERbump + delete theNOTEcomment) and remove theTODO.Note on CI image
docker/common/install_tensorrt.shonly runs when the pyxis image is rebuilt. TheLLM_*_DOCKER_IMAGEtags injenkins/current_image_tags.propertiesare not changed by this PR, so CI-pytest will keep using the current 26.05 image until the periodic image-rebuild job produces a 26.05 image variant with the pinned cuBLAS. That's the same flow every other change toinstall_tensorrt.shuses — no additional wiring needed.Test plan
cublasGetVersion_v2returns 130400) inside the pyxis container.Closes #16601.
Related:
02cedf6e)Summary by CodeRabbit