Skip to content

[https://nvbugs/6104831][fix] disagg buffer-pool RAII and NIXL agent lifetime fixes#14739

Closed
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:nvbug6104831-tier1-fixes
Closed

[https://nvbugs/6104831][fix] disagg buffer-pool RAII and NIXL agent lifetime fixes#14739
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:nvbug6104831-tier1-fixes

Conversation

@chienchunhung

@chienchunhung chienchunhung commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Carves out the disagg KV transfer lifetime and buffer-pool hygiene fixes from PR #13713 into a smaller, mergeable change. All three components close pre-existing hazards that exist today independent of in-flight cancellation.

What this PR changes (all ports from PR #13713)

Code name (per PR #13713) What it is Files
C1 Bounded wait_for slice on the buffer-acquire CV: kBufferAcquireSliceMs = 100, with the acquire loop calling mBuffersCV.wait_for(lk, slice) instead of an unbounded wait. Same pattern as PR #13713's "Cap disagg KV polling wait_for slice" applied to the buffer-acquire site. baseTransBuffer.{h,cpp}
A2 BufferIndexHolder RAII class that owns an acquired buffer-pool index and releases it on destruction, plus wiring at the four acquisition sites in the formatters and the recv-side flow in dataTransceiver.cpp (with detach() for the formatter happy-path handoff). baseTransBuffer.{h,cpp}, cacheFormatter.cpp, mlaCacheFormatter.cpp, dataTransceiver.cpp
A8 nb::keep_alive<0, 1>() annotation on the NIXL agent's nanobind binding so the agent stays alive while its returned TransferStatus exists. nixl_utils/agentBindings.cpp

Issues these resolve independent of in-flight cancellation

  • C1 — without a bounded slice, the buffer-acquire CV wait can block indefinitely. A 100ms slice ensures the waiter periodically observes shutdown / wakeup signals so the process can drain cleanly, with negligible CPU overhead from the periodic wake.
  • A2 — transfer buffer-slot indices acquired during setup are not released when a transfer throws partway through (transient I/O error, peer disconnect, allocation failure). Each leaked slot reduces the pool's usable capacity until the pool is effectively exhausted and new transfers refuse or hang. The RAII holder closes the leak class structurally rather than per-branch.
  • A8 — the NIXL agent can be torn down (executor recycle, shutdown) while a pending TransferStatus still holds a non-owning reference to it. Subsequent operations on the TransferStatus would dereference freed memory. The nanobind keep-alive annotation ties the agent's Python-side lifetime to the TransferStatus it returned.

What this PR explicitly excludes

Code name (per PR #13713) Why excluded
A1 shared_ptr<LlmRequest> ownership for mSenderFutures / mRequesterFutures. Touches a wider API surface; planned for a follow-up so this PR stays focused on the buffer-pool / agent lifetime fixes.
A3 Recv-side dedup sets — only needed for cancel-throw retry idempotency.
A4 / A7 Timeout WARN dedup + observe-only WARN sites. Carried in a sibling PR (#14740).
A5 / A6 State-reorder + eager setKvCacheTransferStart in requestAndReceiveAsync. Not in scope.
A9 / A10 Rank-symmetric collective entry. Carried in a sibling PR for separate verification.
G1–G8 All in-flight cancellation logic.

…lifetime fixes

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51013 [ run ] triggered by Bot. Commit: 6f6a7b7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51013 [ run ] completed with state SUCCESS. Commit: 6f6a7b7
/LLM/main/L0_MergeRequest_PR pipeline #40461 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51114 [ run ] triggered by Bot. Commit: 6f6a7b7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51114 [ run ] completed with state SUCCESS. Commit: 6f6a7b7
/LLM/main/L0_MergeRequest_PR pipeline #40552 completed with status: 'SUCCESS'

CI Report

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants