[https://nvbugs/5930934][fix] Cherry-pick fix NCCL OOM hang

[pcastonguay]

…

Summary by CodeRabbit

• Bug Fixes

  • Enhanced stability by preventing potential deadlock scenarios during distributed computation operations under memory-constrained conditions.

• Refactor

  • Optimized memory efficiency by reducing peak memory consumption through improved tensor operation handling in model inference.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

Two optimization changes: fallback behavior for AllReduce tactic selection now defaults to plain NCCL instead of NCCL_SYMMETRIC to avoid potential deadlocks, and in-place tensor additions replace out-of-place operations in the DeepSeek V3 model to reduce peak memory usage.

Changes

Cohort / File(s)	Summary
AllReduce Tactic Selection tensorrt_llm/_torch/custom_ops/torch_custom_ops.py	Cache miss fallback changed from NCCL_SYMMETRIC to plain NCCL (NCCL.value) due to deadlock risk under out-of-memory conditions with ncclMemAlloc.
Memory Optimization tensorrt_llm/_torch/models/modeling_deepseekv3.py	In-place tensor additions replace out-of-place operations in moe_reduce_add_shared_output and _compute_routed_output to reduce temporary allocations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The PR description contains only the template structure with an empty Description section and uncompleted Test Coverage section, lacking substantive details about the changes, rationale, or test safeguards.	Complete the Description section with details about the OOM hang issue and the fix, and provide specific test coverage details that safeguard the changes in the PR.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title mentions fixing NCCL OOM hang, which directly aligns with the primary change: modifying AllReduce fallback behavior from NCCL_SYMMETRIC to NCCL to prevent deadlocks under OOM conditions.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pcastonguay]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #37738 [ run ] triggered by Bot. Commit: a801189 Link to invocation

[tensorrt-cicd]

PR_Github #37738 [ run ] completed with state SUCCESS. Commit: a801189
/LLM/release-1.3.0rc5.post1/L0_MergeRequest_PR pipeline #17 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[peihu-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #37764 [ run ] triggered by Bot. Commit: a801189 Link to invocation

[tensorrt-cicd]

PR_Github #37764 [ run ] completed with state SUCCESS. Commit: a801189
/LLM/release-1.3.0rc5.post1/L0_MergeRequest_PR pipeline #18 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[pcastonguay]

/bot skip --comment "Flaky multi-GPU tests"

[tensorrt-cicd]

PR_Github #37858 [ skip ] triggered by Bot. Commit: a801189 Link to invocation

[tensorrt-cicd]

PR_Github #37858 [ skip ] completed with state SUCCESS. Commit: a801189
Skipping testing for commit a801189

Link to invocation