[https://nvbugs/6330273][fix] Reserve worst-case SWA slots to avoid single-request deadlock by lowsfer · Pull Request #15588 · NVIDIA/TensorRT-LLM

lowsfer · 2026-06-24T11:12:15Z

Summary by CodeRabbit

Bug Fixes
- Improved cache slot allocation for attention-based workloads, making memory usage more consistent and better aligned with sliding-window behavior.
- Fixed stale-cache window handling so in-flight inputs retain the expected live token range.
Documentation
- Clarified comments to better explain the intended cache lifecycle behavior.

Description

When a sliding-window attention (SWA) layer is configured with window_size <= tokens_per_block, the number of in-window blocks needed oscillates between 1 and 2 as the window slides across block boundaries. The live window for a step is [history_length + 1 - window_size, history_length] (the + 1 accounts for the in-flight input token that always attends within the window). When those tokens straddle a block boundary, two blocks are live simultaneously, even though window_size <= tokens_per_block.

Previously the per-pool-group minimum slot floor was 1 per life cycle. With a tight quota this could size an SWA pool to a single slot, so a lone request could not obtain the 2nd block it needs mid-generation and the manager would deadlock — unable to serve even one request.

This change floors the per-pool-group minimum slots at the worst-case in-window span:

swa_floor_blocks = num_sink_blocks + ceil((window_size - 1) / tokens_per_block) + 1

computed per attention life cycle. Full-attention life cycles use the largest SWA floor among all attention life cycles — a valid lower bound because all attention life cycles see the same seq_len. SSM / non-attention life cycles keep a floor of 1. The floor flows into _min_quota_for_level, so the GPU quota is bumped up when needed to guarantee a single request can always make progress.

A clarifying comment was also added to AttnLifeCycle.get_stale_range documenting why the + 1 in the window boundary is intentional.

🤖 Generated with Claude Code

coderabbitai · 2026-06-24T11:15:38Z

📝 Walkthrough

Walkthrough

Adds SWA-aware minimum-slot floor logic to _compute_min_slots_from_constraints in _storage_manager.py, replacing the flat "1 slot per life cycle" default with per-type calculations using a swa_floor_blocks helper. Imports AttnLifeCycle and unwrap_optional to support the new logic. Adds clarifying comments in AttnLifeCycle.get_stale_range about the + 1 window semantics.

Changes

SWA-aware minimum slot floor

Layer / File(s)	Summary
AttnLifeCycle window semantics clarification `tensorrt_llm/runtime/kv_cache_manager_v2/_life_cycle_registry.py`	Adds inline comments in `AttnLifeCycle.get_stale_range` (lines 45–47) documenting the intentional `+ 1` window behavior and the inclusive handling of the token at `history_length`; no executable logic changed.
SWA-aware min-slot floor implementation `tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py`	Expands imports to include `AttnLifeCycle` and `unwrap_optional`, then rewrites `_compute_min_slots_from_constraints` with an `swa_floor_blocks` helper that computes per-life-cycle minimums differently for non-attention lifecycles (`+1`), windowed-attention lifecycles (`+swa_floor_blocks`), and full-attention lifecycles (`+floor_num_blocks`).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise, specific, and accurately reflects the SWA deadlock fix in KV cache management.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description clearly explains the bug, fix, and impact, though the template’s Test Coverage section is not explicitly filled out.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

lowsfer · 2026-06-24T11:17:12Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-24T11:23:42Z

PR_Github #55487 [ run ] triggered by Bot. Commit: c9e058f Link to invocation

tensorrt-cicd · 2026-06-24T15:16:44Z

PR_Github #55487 [ run ] completed with state SUCCESS. Commit: c9e058f
/LLM/main/L0_MergeRequest_PR pipeline #44412 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

lowsfer · 2026-06-25T10:18:32Z

/bot run --disable-fail-fast

lowsfer · 2026-06-26T01:54:50Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-26T02:01:05Z

PR_Github #55940 [ run ] triggered by Bot. Commit: a501890 Link to invocation

tensorrt-cicd · 2026-06-26T05:46:57Z

PR_Github #55940 [ run ] completed with state FAILURE. Commit: a501890
/LLM/main/L0_MergeRequest_PR pipeline #44822 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-29T11:06:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-29T11:12:57Z

PR_Github #56354 [ run ] triggered by Bot. Commit: b6278fa Link to invocation

tensorrt-cicd · 2026-06-29T14:47:51Z

PR_Github #56354 [ run ] completed with state SUCCESS. Commit: b6278fa
/LLM/main/L0_MergeRequest_PR pipeline #45201 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-29T17:15:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-29T17:22:19Z

PR_Github #56425 [ run ] triggered by Bot. Commit: b6278fa Link to invocation

tensorrt-cicd · 2026-06-29T19:18:25Z

PR_Github #56425 [ run ] completed with state SUCCESS. Commit: b6278fa
/LLM/main/L0_MergeRequest_PR pipeline #45268 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ingle-request deadlock When a sliding-window attention (SWA) layer has window_size <= tokens_per_block, the instantaneous in-window block count oscillates between 1 and 2 as the window slides across block boundaries. Sizing the pool to the minimum (1) could leave a lone request unable to obtain the 2nd block it needs mid-generation, deadlocking. Floor the per-pool-group minimum slots at the worst-case in-window span, ceil((window_size - 1) / tokens_per_block) + 1 plus sink blocks, computed per attention life cycle. Full-attention life cycles use the largest SWA floor (all attention life cycles see the same seq_len, so it is a valid lower bound). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

… SWA min-slots floor The SWA worst-case min_slots floor keeps the in-window blocks resident, so the GPU level can no longer be shrunk below the window's block requirement and in-window pages are not dropped/migrated under memory pressure. Rework the two event-manager tests to use a small window (small floor) with more committed blocks than fit after the shrink, and assert per-layer-group event well-formedness (events emitted for each layer group, correct cache_level diff, hashes previously stored) instead of cross-group symmetry / exact counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

lowsfer · 2026-07-01T15:43:19Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-01T15:49:59Z

PR_Github #56960 [ run ] triggered by Bot. Commit: 2f88c53 Link to invocation

tensorrt-cicd · 2026-07-01T18:48:14Z

PR_Github #56960 [ run ] completed with state SUCCESS. Commit: 2f88c53
/LLM/main/L0_MergeRequest_PR pipeline #45765 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-08T02:48:45Z

/bot run --disable-fail-fast

lowsfer · 2026-07-08T05:54:30Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-08T06:01:17Z

PR_Github #58170 [ run ] triggered by Bot. Commit: 2f88c53 Link to invocation

tensorrt-cicd · 2026-07-08T08:49:39Z

PR_Github #58170 [ run ] completed with state SUCCESS. Commit: 2f88c53
/LLM/main/L0_MergeRequest_PR pipeline #46820 completed with status: 'SUCCESS'

CI Report

Link to invocation

nvpohanh · 2026-07-09T08:07:52Z

@yizhang-nv could you review this?

… KVCacheManagerV2 Translate two main-branch Python changes in runtime/kv_cache_manager_v2/ into the C++ backend under cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/: Translated from: - 17190cf [feat] add commit min snapshot for SSM reuse (NVIDIA#15752) config: drop HelixConfig/ssm_reuse_interval, add commit_min_snapshot; page: add SsmCommittedPage + UncommittedPage::convertToSsmCommitted; blockRadixTree: unlinkPage(expectedPage) guard + snapshot-length-aware SSM match truncation; kvCache: commit(is_end), _commitBlock(commitSsm,moveSsm), rewritten _snapshotSsmToTreeBlock, new _copyPageToTreeBlock and _snapshotPartialBlockToTree, plus commit_min_snapshot assert relaxations; nanobind: commit(is_end) and config/manager surface updates. - 52fc8f4 [fix] Reserve worst-case SWA slots to avoid single-request deadlock (NVIDIA#15588, nvbugs/6330273) storageManager::computeMinSlotsFromConstraints SWA worst-case floor; lifeCycleRegistry getStaleRange comment. Also in this change: - Mirror the Python Block._release_pages() eager reclaim into C++ Block::releasePages()/removeSubtree so page reclamation does not depend on ~Block() destruction timing. - Add kv::AssertionError + a nanobind translator to Python AssertionError so config validation and the commit history-alignment check match the Python backend's exception type. - Add white-box reuse-tree introspection helpers (attention/swa/ssm life-cycle ids, reuse_match_pages) to the C++ _introspection submodule and _introspection.py, and route the shared tests through them instead of Python-only internals. - Adapt the SWA event-manager tests (small window; deeper load for the migration test to force both pool groups to migrate under the new slot floor). - _core/_kv_cache.py: add the same VIRTUAL_STOP loop-break/partial-snapshot guard the C++ commit() uses, keeping the two backends behaviourally aligned. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

github-actions Bot assigned lowsfer Jun 24, 2026

lowsfer force-pushed the bug-6330273 branch from c9e058f to 33936d8 Compare June 25, 2026 10:18

lowsfer force-pushed the bug-6330273 branch from 33936d8 to a501890 Compare June 26, 2026 01:54

lowsfer force-pushed the bug-6330273 branch from a501890 to b6278fa Compare June 29, 2026 11:06

lowsfer and others added 2 commits June 30, 2026 04:01

lowsfer force-pushed the bug-6330273 branch from b6278fa to 2f88c53 Compare June 30, 2026 04:04

lowsfer requested a review from yizhang-nv July 9, 2026 08:05

yizhang-nv approved these changes Jul 13, 2026

View reviewed changes

lowsfer merged commit 52fc8f4 into NVIDIA:main Jul 13, 2026
7 checks passed

Uh oh!

Conversation

lowsfer commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Uh oh!

coderabbitai Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

lowsfer commented Jun 24, 2026

Uh oh!

tensorrt-cicd commented Jun 24, 2026

Uh oh!

tensorrt-cicd commented Jun 24, 2026

Uh oh!

lowsfer commented Jun 25, 2026

Uh oh!

lowsfer commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

lowsfer commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

lowsfer commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

lowsfer commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

lowsfer commented Jul 8, 2026

Uh oh!

lowsfer commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

nvpohanh commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lowsfer commented Jun 24, 2026 •

edited

Loading

coderabbitai Bot commented Jun 24, 2026 •

edited

Loading