[https://nvbugs/6426852][fix] Add explicit gc.collect() between del weights and…

[trtllm-agent]

Summary

• Root cause: Per-RPC ipc_collect() couldn't reclaim IPC mappings because del weights alone left the mapped storages in unreachable cycles; many partial-update RPCs on MoE model exhausted CUDA driver's IPC handle table.
• Fix: Add explicit gc.collect() between del weights and torch.cuda.ipc_collect() in the per-RPC branch so IPC-mapped storages are released in the same iteration, complementing the prior producer-side lazy-replica fix.
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6426852

Summary by CodeRabbit

• Bug Fixes

  • Improved weight update reliability during repeated requests by ensuring GPU memory handles are released more consistently.
  • Reduced the chance of failures when updating model weights across multiple GPU operations.

• Tests

  • Updated integration waivers and related GPU test behavior to reflect the new weight-handling flow.
  • Adjusted test helpers to create GPU weight copies more lazily.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 448e27d6-f7eb-42c7-8225-25bccd60b1cf

📥 Commits

Reviewing files that changed from the base of the PR and between 5ce293f and 9821ab7.

📒 Files selected for processing (3)

• tensorrt_llm/llmapi/rlhf_utils.py
• tests/integration/test_lists/waives.txt
• tests/unittest/_torch/ray_orchestrator/single_gpu/test_llm_update_weights.py

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

---

📝 Walkthrough

Walkthrough

Adds an explicit gc.collect() call before torch.cuda.ipc_collect() in the weight update path to release CUDA IPC storages. Removes an associated test waiver and refactors a test helper to lazily replicate model weights per device instead of eagerly copying to all devices.

Changes

CUDA IPC weight update fix

Layer / File(s)	Summary
gc.collect() before IPC sweep tensorrt_llm/llmapi/rlhf_utils.py	Adds an unconditional gc.collect() call after weight reload and before torch.cuda.ipc_collect() to ensure IPC storages are released.
Lazy weight replication and waiver removal tests/unittest/_torch/ray_orchestrator/single_gpu/test_llm_update_weights.py, tests/integration/test_lists/waives.txt	Test helper now only clones weights for the owning device and lazily populates other devices' weights on demand; removes the obsolete waiver for the multi-gpu update-weights test.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16093: Both PRs modify the same waives.txt entry for the multi_gpu test_llm_update_weights_multi_gpu.py -m "part1" case.
• NVIDIA/TensorRT-LLM#16109: Both PRs adjust CI waiver entries in tests/integration/test_lists/waives.txt.

Suggested reviewers: brb-nv, shuyixiong, mingyangHao

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main fix: adding explicit gc.collect() for CUDA IPC handle cleanup, though it is truncated.
Description check	✅ Passed	The description covers summary, root cause, fix, and test plan, but it omits the template's explicit Description and PR Checklist sections.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}