[TRTLLM-11715][infra] Fix in upgrade dependencies for dlfw 26.04 stack - #15248
Open
tensorrt-cicd wants to merge 77 commits into
Open
[TRTLLM-11715][infra] Fix in upgrade dependencies for dlfw 26.04 stack#15248tensorrt-cicd wants to merge 77 commits into
tensorrt-cicd wants to merge 77 commits into
Conversation
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
… 2.11.0a0 Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Emma Qiao <qqiao@nvidia.com>
Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
tensorrt-cicd
requested review from
a team,
QiJune,
chang-l,
dpitman-nvda,
niukuo and
yizhang-nv
June 11, 2026 06:17
Contributor
📝 WalkthroughWalkthroughThis PR performs a comprehensive upgrade of NVIDIA's CUDA (13.1.1→13.2.1), PyTorch (2.10.0→2.11.0), and TensorRT (10.15→10.16) frameworks across Docker, CI, and dependencies, while introducing runtime refinements: CUDA memory descriptor initialization normalization, Clang-to-GCC compiler compatibility for ARM64, torch inductor backend switching, PMIx MPI integration for performance testing, and updated CI test waivers for new test variants. ChangesIntegrated Stack Upgrade and Runtime Optimizations
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
tests/integration/test_lists/waives.txt (1)
38-47: Coverage follow-up needed for newly added waivers.Coverage status is insufficient for the newly waived paths in:
accuracy/test_llm_api_pytorch.pyunittest/_torch/modeling/test_modeling_nemotron_h.pyunittest/_torch/visual_gen/test_attention_integration.pyunittest/_torch/visual_gen/test_flux_pipeline.pyunittest/auto_deploy/singlegpu/models/test_decilm_modeling.pyunittest/auto_deploy/singlegpu/models/test_granite_moe_hybrid_modeling.pyunittest/llmapi/test_llm_pytorch.pyPlease track unwaive follow-ups (owner + target milestone) outside this PR so regression coverage is restored.
As per coding guidelines, "tests/**: Act as a QA engineer reviewing test changes and coverage... suggest concrete list file names and whether coverage is sufficient, insufficient, or needs follow-up outside the PR."
Also applies to: 94-97, 388-388, 407-416, 426-427
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/test_lists/waives.txt` around lines 38 - 47, Newly added test waiver entries lack tracked follow-ups to restore coverage; add a follow-up plan outside this PR listing each waived file (accuracy/test_llm_api_pytorch.py, unittest/_torch/modeling/test_modeling_nemotron_h.py, unittest/_torch/visual_gen/test_attention_integration.py, unittest/_torch/visual_gen/test_flux_pipeline.py, unittest/auto_deploy/singlegpu/models/test_decilm_modeling.py, unittest/auto_deploy/singlegpu/models/test_granite_moe_hybrid_modeling.py, unittest/llmapi/test_llm_pytorch.py and the additional ranges referenced) with an assigned owner and target milestone, and update the waiver metadata/comment block in the waiver file (the waives list entries) to include a short “follow-up: OWNER / MILESTONE / tracking-issue-ID” note so CI coverage restoration is tracked; ensure each entry is marked “insufficient” where applicable and add the tracking issue IDs or create them if missing.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/L0_Test.groovy`:
- Line 1: The library reference
bloom-jenkins-shared-lib@emma/update_nsc_login_node on the top-level `@Library`
declaration is pinned to a personal branch; change it to a reviewed immutable
ref (e.g., `@main`, a release tag, or a specific commit SHA) so the pipeline uses
a stable vetted version—update the `@Library` entry that currently names
bloom-jenkins-shared-lib@emma/update_nsc_login_node to point to the chosen
immutable ref.
- Around line 4120-4122: Restore coverage for the Verl lane by either
uncommenting and re-enabling the previously disabled stage entry
("DGX_B200-4_GPUs-Verl-Post-Merge-1") or add a new equivalent Verl stage entry
that runs "l0_verl" on a different available pool; locate the commented array
line for "DGX_B200-4_GPUs-Verl-Post-Merge-1" and either remove the comment to
re-enable it or duplicate it with a new unique name and appropriate pool string
(ensuring it still references "l0_verl"), then add a follow-up ticket reference
in the PR linking to the nvbugs workaround if you must keep it disabled so the
re-enable action is tracked.
In `@requirements.txt`:
- Around line 92-93: Pin the newly added Python dependencies by updating the
requirements.txt entries for "librosa" and "msgpack" to explicit version
constraints (e.g., replace the bare "librosa" and "msgpack" lines with
tested/version-pinned specifiers like "librosa==<tested_version>" and
"msgpack==<tested_version>" or a safe range "librosa>=x.y,<x+1.0" and
"msgpack>=a.b,<a+1.0") so builds are reproducible; choose the exact versions
validated in the 26.04 container and apply those specifiers to the librosa and
msgpack lines.
---
Nitpick comments:
In `@tests/integration/test_lists/waives.txt`:
- Around line 38-47: Newly added test waiver entries lack tracked follow-ups to
restore coverage; add a follow-up plan outside this PR listing each waived file
(accuracy/test_llm_api_pytorch.py,
unittest/_torch/modeling/test_modeling_nemotron_h.py,
unittest/_torch/visual_gen/test_attention_integration.py,
unittest/_torch/visual_gen/test_flux_pipeline.py,
unittest/auto_deploy/singlegpu/models/test_decilm_modeling.py,
unittest/auto_deploy/singlegpu/models/test_granite_moe_hybrid_modeling.py,
unittest/llmapi/test_llm_pytorch.py and the additional ranges referenced) with
an assigned owner and target milestone, and update the waiver metadata/comment
block in the waiver file (the waives list entries) to include a short
“follow-up: OWNER / MILESTONE / tracking-issue-ID” note so CI coverage
restoration is tracked; ensure each entry is marked “insufficient” where
applicable and add the tracking issue IDs or create them if missing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f7aea187-960d-4c94-b190-f8f7d2598727
📒 Files selected for processing (27)
README.mdconstraints.txtcpp/include/tensorrt_llm/runtime/virtualMemory.hcpp/tensorrt_llm/deep_ep/CMakeLists.txtcpp/tensorrt_llm/flash_mla/CMakeLists.txtcpp/tensorrt_llm/runtime/virtualMemory.cppcpp/tests/unit_tests/common/cudaDriverWrapperTest.cppcpp/tests/unit_tests/runtime/virtualMemoryTest.cppdocker/Dockerfile.multidocker/Makefiledocker/common/install_cuda_toolkit.shdocker/common/install_pytorch.shdocker/common/install_tensorrt.shdocs/source/legacy/reference/support-matrix.mdjenkins/Build.groovyjenkins/L0_Test.groovyjenkins/current_image_tags.propertiesjenkins/scripts/perf/README.mdjenkins/scripts/perf/aggregated/slurm_launch_draft.shjenkins/scripts/perf/local/submit.pyrequirements.txtscripts/rename_docker_images.pytensorrt_llm/_torch/compilation/backend.pytensorrt_llm/_torch/compilation/piecewise_optimizer.pytests/integration/defs/verl/verl_config.ymltests/integration/test_lists/waives.txttests/unittest/auto_deploy/singlegpu/custom_ops/moe/test_trtllm_moe.py
💤 Files with no reviewable changes (1)
- constraints.txt
| @@ -1,4 +1,4 @@ | |||
| @Library(['bloom-jenkins-shared-lib@main', 'trtllm-jenkins-shared-lib@main']) _ | |||
| @Library(['bloom-jenkins-shared-lib@emma/update_nsc_login_node', 'trtllm-jenkins-shared-lib@main']) _ | |||
Contributor
There was a problem hiding this comment.
Pin the shared library to a reviewed immutable ref.
Line 1 points bloom-jenkins-shared-lib at a personal branch head. After merge, a force-push or branch deletion can break CI or change the library code Jenkins executes with pipeline credentials independently of this PR. Use @main, a release tag, or an immutable commit instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@jenkins/L0_Test.groovy` at line 1, The library reference
bloom-jenkins-shared-lib@emma/update_nsc_login_node on the top-level `@Library`
declaration is pinned to a personal branch; change it to a reviewed immutable
ref (e.g., `@main`, a release tag, or a specific commit SHA) so the pipeline uses
a stable vetted version—update the `@Library` entry that currently names
bloom-jenkins-shared-lib@emma/update_nsc_login_node to point to the chosen
immutable ref.
Comment on lines
+4120
to
+4122
| // Disable Verl stage due to https://nvbugs/6236818. | ||
| // Please re-enable it after the bug is fixed. | ||
| // "DGX_B200-4_GPUs-Verl-Post-Merge-1": ["auto:dgx-b200-flex", "l0_verl", 1, 1, 4, 1, true], |
Contributor
There was a problem hiding this comment.
Post-merge Verl coverage is now insufficient.
Commenting out DGX_B200-4_GPUs-Verl-Post-Merge-1 removes the only active -Verl- lane in this file, so l0_verl is no longer exercised post-merge here. If the nvbugs block is unavoidable, please track a follow-up outside this PR to re-enable this stage or add a replacement Verl lane on another pool.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@jenkins/L0_Test.groovy` around lines 4120 - 4122, Restore coverage for the
Verl lane by either uncommenting and re-enabling the previously disabled stage
entry ("DGX_B200-4_GPUs-Verl-Post-Merge-1") or add a new equivalent Verl stage
entry that runs "l0_verl" on a different available pool; locate the commented
array line for "DGX_B200-4_GPUs-Verl-Post-Merge-1" and either remove the comment
to re-enable it or duplicate it with a new unique name and appropriate pool
string (ensuring it still references "l0_verl"), then add a follow-up ticket
reference in the PR linking to the nvbugs workaround if you must keep it
disabled so the re-enable action is tracked.
Comment on lines
+92
to
+93
| librosa | ||
| msgpack |
Contributor
There was a problem hiding this comment.
Pin versions for librosa and msgpack.
These dependencies are added without version constraints, which can lead to non-reproducible builds and unexpected breakage when upstream releases introduce changes.
📌 Recommended fix: add version constraints
-librosa
-msgpack
+librosa>=0.10.0,<0.11.0
+msgpack>=1.0.0,<2.0.0(Adjust the specific version ranges based on the actual versions tested in the 26.04 container.)
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| librosa | |
| msgpack | |
| librosa>=0.10.0,<0.11.0 | |
| msgpack>=1.0.0,<2.0.0 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@requirements.txt` around lines 92 - 93, Pin the newly added Python
dependencies by updating the requirements.txt entries for "librosa" and
"msgpack" to explicit version constraints (e.g., replace the bare "librosa" and
"msgpack" lines with tested/version-pinned specifiers like
"librosa==<tested_version>" and "msgpack==<tested_version>" or a safe range
"librosa>=x.y,<x+1.0" and "msgpack>=a.b,<a+1.0") so builds are reproducible;
choose the exact versions validated in the 26.04 container and apply those
specifiers to the librosa and msgpack lines.
The 26.03 upgrade branch had drifted on tests/integration/test_lists/waives.txt relative to main and was conflicting on PR review. Restore the file to origin/main's content so the PR carries no waiver edits; whatever waiver adjustments the dlfw upgrade actually needs will go in a focused follow-up. Signed-off-by: chenfeiz <chenfeiz@nvidia.com>
Collaborator
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-3" |
Collaborator
Author
|
PR_Github #53500 [ run ] triggered by Bot. Commit: |
Collaborator
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-2" |
Signed-off-by: chenfeiz0326 <chenfeiz@nvidia.com>
Collaborator
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-2" |
Collaborator
Author
|
PR_Github #53546 [ run ] triggered by Bot. Commit: |
Collaborator
Author
|
PR_Github #53500 [ run ] completed with state |
Collaborator
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-2" |
Collaborator
Author
|
PR_Github #53556 [ run ] triggered by Bot. Commit: |
Collaborator
Author
|
PR_Github #53546 [ run ] completed with state |
Collaborator
Author
|
PR_Github #53556 [ run ] completed with state |
The aggregated multi-node perf-sanity path runs `srun --mpi=pmix` with the DLFW 26.04 OpenMPI 4.1.7 build. With the default PMIx GDS plugin selection (ds21/ds12 shared-memory) the cross-node UCX worker-address modex fails on GB200 SBSA nodes, and the launcher process aborts before pytest collection with: [host:pid] pml_ucx.c:178 Error: Failed to receive UCX worker address: Not found (-13) [host:pid] pml_ucx.c:482 Error: Failed to resolve UCX endpoint for rank N Forcing the in-memory hash GDS via PMIX_MCA_gds=hash makes the modex use PMIx's hash table backend instead of the fragile shared-memory ones, which matches the workaround documented for this exact symptom in open-mpi/ompi#6981. Patched in two slurm_run wrappers used by the multi-node aggregated path: - jenkins/scripts/slurm_run.sh (CI multi-node L0 path) - jenkins/scripts/perf/local/slurm_run.sh (perf/local QA path) The export is a no-op when PMIx is not used (single-node / non --mpi=pmix runs). Signed-off-by: chenfeiz0326 <chenfeiz@nvidia.com>
Collaborator
|
/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-*" |
Collaborator
Author
|
PR_Github #54210 [ run ] triggered by Bot. Commit: |
Collaborator
Author
|
PR_Github #54210 [ run ] completed with state
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a multi-node MPI initialization failure surfaced by the DLFW 26.03 upgrade in this branch. After upgrading the base image to
nvcr.io/nvidia/pytorch:26.03-py3, the aggregated multi-node perf-sanity stage (e.g.GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1, NVbug 6162581) aborts atMPI_Init_threadwith:Root cause
The 26.03 PyTorch container ships OpenMPI 4.1.7 built with PMIx support only — no classic PMI-2. The aggregated multi-node srun still passes
--mpi=pmi2(inslurm_launch_draft.shandL0_Test.groovy). Under--mpi=pmi2, srun exposes a PMI-2 server but no PMIx server. Whentensorrt_llm.llmapi.mgmn_leader_node/mgmn_worker_nodeimportmpi4py.MPIand callMPI_Init_thread, OMPI'spmix3x_clienttries to attach to a PMIx server it can't find and aborts.The disaggregated path already uses
--mpi=pmix(jenkins/scripts/perf/disaggregated/slurm_launch_draft.sh), which is why only the aggregated multi-node case is broken.Fix
Switch the aggregated multi-node srun from
--mpi=pmi2to--mpi=pmix, matching what the disaggregated path has always used and what the new OMPI build supports.Files touched:
jenkins/scripts/perf/aggregated/slurm_launch_draft.shjenkins/L0_Test.groovy(multi-node sbatch path)jenkins/scripts/perf/local/submit.py(default +--mpi-typehelp string)jenkins/scripts/perf/README.md(documentation note)Test plan
GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-{1..7}is_single_gpu_aggrshort-circuit still drops--mpiSummary by CodeRabbit
Release Notes
New Features
librosaandmsgpackpackage dependencies.Chores