[https://nvbugs/6104831][fix] Enforce request and buffer index lifecycle integrity - #14768
Conversation
…exists Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
…ctx-side checkContextTransferStatus Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
…cle integrity. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
chienchunhung
changed the title
📝 WalkthroughWalkthroughThis PR migrates the cache transceiver request methods from raw pointer arguments to ChangesCache Transceiver and Disaggregated Safety
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp`:
- Around line 555-565: The timeout logging is computed using now - start before
verifying the transfer future completed, causing polling delay to be reported as
a timeout; update the polling/inspection logic (the loop that reads the transfer
future and emits WARNs) so you first check the transfer's completion/readiness
(e.g., future::wait_for(0) or equivalent) and skip timeout calculation/logging
if the future is already ready, then only compute elapsed = now - start and
consult kvTransferTimeoutMs and mTimedOutSenderIds when the transfer remains
incomplete; apply the same fix to the other similar blocks referenced around the
622-640 and 808-834 regions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8433b8e6-f940-404e-99bb-f9f6e4f6d3ae
📒 Files selected for processing (8)
cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.hcpp/tensorrt_llm/batch_manager/baseTransBuffer.cppcpp/tensorrt_llm/batch_manager/baseTransBuffer.hcpp/tensorrt_llm/batch_manager/cacheTransceiver.cppcpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cppcpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cppcpp/tensorrt_llm/nanobind/batch_manager/cacheTransceiver.cpptensorrt_llm/_torch/pyexecutor/py_executor.py
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
…ve entry Restores the pre-A9/A10 behavior in _check_disagg_gen_transfer_status, _prepare_disagg_gen_init, _executor_loop_pp, and _executor_loop to match upstream/main. Reduces this PR's delta vs main and isolates the rank-symmetric collective-entry changes for separate verification. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #51140 [ run ] triggered by Bot. Commit: |
|
PR_Github #51140 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #51196 [ run ] triggered by Bot. Commit: |
|
PR_Github #51196 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #51226 [ run ] triggered by Bot. Commit: |
|
PR_Github #51226 [ run ] completed with state |
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #51237 [ run ] triggered by Bot. Commit: |
|
PR_Github #51237 [ run ] completed with state
|
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #51629 [ run ] triggered by Bot. Commit: |
|
PR_Github #51629 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #51696 [ run ] triggered by Bot. Commit: |
|
PR_Github #51696 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #51752 [ run ] triggered by Bot. Commit: |
|
PR_Github #51752 [ run ] completed with state
|
|
PR#14916 (a cloned version of this PR + rebasing to the latest main) passed the targets tests that failed in this PR, confirming the code changes rebasing to the latest main should pas those tests. |
|
/bot help |
GitHub Bot Help
Provide a user friendly way for developers to interact with a Jenkins server. Run See details below for each supported subcommand. Details
Launch build/test pipelines. All previously running jobs will be killed.
kill
Kill all running builds associated with pull request. skip
Skip testing for latest commit on pull request. reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break. |
…-on-verify Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> # Conflicts: # tests/integration/test_lists/test-db/l0_sanity_check.yml
|
/bot skip --comment "The failed perf regression tests have been verified separately in PR#14916." |
|
PR_Github #51908 [ skip ] triggered by Bot. Commit: |
|
/bot skip --comment "The failed perf regression tests have been verified separately in PR#14916." |
|
PR_Github #51908 [ skip ] completed with state |
|
PR_Github #51911 [ skip ] triggered by Bot. Commit: |
|
PR_Github #51911 [ skip ] completed with state |
5 participants
Summary
This PR is a strict subset of PR #13713 — same files, fewer changes — that carries only the components which (a) fix concrete pre-existing bugs and (b) are safe to land independently of the in-flight cancellation surface.
The goal is a low-risk increment that reviewers of PR #13713 can clear quickly: the included components are byte-equivalent ports of code already reviewed there; the deltas to audit are the exclusion decisions.
Relationship with PR #13713
This PR carries only the changes from PR #13713 that fix concrete pre-existing bugs (
LlmRequestlifetime, buffer-slot leak on exception, NIXL agent lifetime) plus the observe-only timeout WARN. Everything in PR #13713 that exists to support the in-flight cancellation surface — both the surface itself and the always-on auxiliaries that the cancel-throw retry pattern depends on — is left out.In-flight cancellation is default OFF in PR #13713. With that surface dormant, this PR and PR #13713 are conceptually the same bug-fix landing. The cancellation surface, and the auxiliaries needed to support it, are deferred to a follow-up.
What this PR fixes
shared_ptr<LlmRequest>in async transfer ownershipLlmRequestuse-after-free: the async worker can outlive the request's scheduling lifetime; the raw pointer in the futures vector dereferences freed memory.BufferIndexHolderRAIInb::keep_alive<0, 1>()TransferStatusstill references it.Mitigates NVBug 6104831 (and adjacent reports 6043291, 6162328). The three failure modes above are the cleanup-path bugs behind the customer-reported wedges, crashes, and silent corruption in long-running disaggregated deployments — they fire under client-side cancellations + load on
maintoday, regardless of whether the in-flight cancellation feature is on. Merging this PR resolves those NVBugs without behavioral change for healthy traffic.The in-flight cancellation feature itself is default OFF in PR #13713 and not in current production use; redesigning it (per-rank dedup → cross-rank-consistent primitive, retry semantics) is deferred to a follow-up and is not on the critical path for this reliability fix.
Test coverage
Two unit-test files:
cpp/tests/unit_tests/batch_manager/bufferIndexHolderTest.cpp— 10 parametrized test cases × 2 sides (send / recv) = 20 test instantiations. Uses a test-onlyObservableTransBufferManagersubclass that exposes the protected concurrence counter. Covers default-construct, nullopt-index, RAII release on scope exit, explicitrelease()+ idempotency,detach()semantics, move-construct + move-assign ownership transfer, and exception-unwind release.tests/unittest/others/test_kv_cache_transceiver.py— addstest_async_transfer_keeps_llm_request_alive[sender]/[receiver]. Uses a Pythonweakref.refobserver on theLlmRequestto verify that after the external Python reference is dropped, the C++shared_ptrinmSenderFutures/mRequesterFutureskeeps the wrapper alive. Would catch a regression where the futures vector reverted to a raw pointer.Follow-up
In-flight cancellation needs more careful design and is deferred to a follow-up PR on top of this baseline.
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.