[None][fix] NIXL agent transfer and shutdown race by Shixiaowei02 · Pull Request #14137 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-05-14T09:50:49Z

@coderabbitai summary

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Shixiaowei02 · 2026-05-14T11:39:57Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-14T11:46:25Z

PR_Github #48359 [ run ] triggered by Bot. Commit: 8f2a967 Link to invocation

tensorrt-cicd · 2026-05-14T18:09:07Z

PR_Github #48359 [ run ] completed with state SUCCESS. Commit: 8f2a967
/LLM/main/L0_MergeRequest_PR pipeline #38165 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-05-15T04:22:36Z

/bot run --add-multi-gpu-test --disable-fail-fast

Shixiaowei02 · 2026-05-15T04:27:29Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-15T04:29:16Z

PR_Github #48513 [ run ] triggered by Bot. Commit: 47c757d Link to invocation

tensorrt-cicd · 2026-05-15T04:33:21Z

PR_Github #48515 [ run ] triggered by Bot. Commit: 47c757d Link to invocation

Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-05-15T10:00:01Z

/bot run --add-multi-gpu-test --disable-fail-fast

Shixiaowei02 · 2026-05-15T10:00:10Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-15T10:07:24Z

PR_Github #48582 [ run ] triggered by Bot. Commit: 16041d4 Link to invocation

tensorrt-cicd · 2026-05-15T10:08:16Z

PR_Github #48583 [ run ] triggered by Bot. Commit: 16041d4 Link to invocation

tensorrt-cicd · 2026-05-15T10:08:19Z

PR_Github #48582 [ run ] completed with state ABORTED. Commit: 16041d4

Link to invocation

tensorrt-cicd · 2026-05-15T10:08:42Z

PR_Github #48515 [ run ] completed with state ABORTED. Commit: 47c757d

Link to invocation

tensorrt-cicd · 2026-05-15T14:46:02Z

PR_Github #48583 [ run ] completed with state SUCCESS. Commit: 16041d4
/LLM/main/L0_MergeRequest_PR pipeline #38368 completed with status: 'SUCCESS'

CI Report

Link to invocation

Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…own/UAF race) Cover the remaining reviewer-flagged gaps for the NIXL transfer-agent shutdown/UAF-safety refactor: - test_agent.py (mock-based, no GPU; verified on container w/ nixl binding): * TestBindingsNixlTransferStatus -- wait() success/None-timeout/failure(+log), is_completed, last_status/last_status_str passthrough + <unavailable>/-1 fallback when the cpp status lacks getLastStatus*. * TestBindingsNixlTransferAgentShutdown -- shutdown() idempotency (nulls _cpp_agent first), no-op when never initialized, submit-after-shutdown raises. - transferAgentTest.cpp (C++ gtest, TEST_NIXL_BACKEND-guarded -- runs in CI): * StatusOutlivesAgent -- destroy owning agent before the status; orphaned isCompleted()/wait() report failure (expired weak_ptr, no use-after-free). * ConcurrentSubmit -- 8 threads submit concurrently; exercises the shared_lock + per-request reqParams isolation; all transfers succeed. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths. - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr. - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>