Skip to content

[None][fix] kv_cache_v2: fix SWA scratch on deferred-context revert under ADP balance#14934

Closed
qiaoxj07 wants to merge 1 commit into
NVIDIA:feat/deepseek_v4from
qiaoxj07:fix/swa-scratch-adp-revert
Closed

[None][fix] kv_cache_v2: fix SWA scratch on deferred-context revert under ADP balance#14934
qiaoxj07 wants to merge 1 commit into
NVIDIA:feat/deepseek_v4from
qiaoxj07:fix/swa-scratch-adp-revert

Conversation

@qiaoxj07

@qiaoxj07 qiaoxj07 commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

revert_allocate_context() undoes the per-iteration resize_context() capacity grow when attention-DP balance (attention_dp_config.enable_balance) or batch-waiting defers a context request after V2 scheduling. On sliding-window-attention (SWA) layers this collided with SWA scratch reuse and crashed DeepSeek-V4 DEP8 runs once adp_balance was enabled.

Two failure modes:

  1. A freshly-deferred context (history_length == 0) tripped resize()'s SWA-scratch invariant history_length >= old_capacity - max_rewind_len (which only holds for a generation-time window slide) — e.g. AssertionError: SWA scratch requires old_capacity - max_rewind_len (965) <= history_length (0) <= old_capacity (965).
  2. A context whose prefill grew past the sliding window holds scratch slots, and resize() refuses to shrink while scratch slots are live.

Fix

  • _KVCache.resize() gains a keyword-only allow_swa_scratch_reuse (default True). The revert path passes False to perform a scratch-free capacity shrink, leaving the generation-time window-slide path untouched.
  • revert_allocate_context() suspend()s caches that hold scratch slots — freeing the scratch slots and releasing pages back to the pool — and resumes them via prepare_context() / _resume_and_restore() on the next schedule (the same machinery generation and the existing pre_cap > 0 revert already use).

adp_balance stays fully functional and deferred pages are still freed during the wait window (no KV-cache capacity regression).

Test

DSv4 DEP8 (TP8/EP8, attention-DP), adp_balance (timeout_iters=100, batching_wait_iters=20), SWA, MTP0, NVFP4, bs128 / conc256 on GB200:

  • Before: crashes in warmup with SWA scratch requires ....
  • After: the full benchmark completes (Total Token Throughput ~13.6k tok/s) with 0 SWA assertions and 0 errors.

🤖 Generated with Claude Code

…nder ADP balance

revert_allocate_context() undoes resize_context()'s capacity grow when
attention-DP balance (or batch waiting) defers a context request after
V2 scheduling. For sliding-window-attention layers this collided with
SWA scratch reuse in two ways:

- a fresh deferred context (history_length == 0) tripped resize()'s
  scratch invariant (history_length >= old_capacity - max_rewind_len),
  which only holds for a generation-time window slide;
- a context whose prefill grew past the sliding window holds scratch
  slots, and resize() refuses to shrink while scratch slots are live.

Fix:
- _KVCache.resize() gains a keyword-only allow_swa_scratch_reuse (default
  True); revert passes False to perform a scratch-free capacity shrink.
- revert_allocate_context() suspend()s caches that hold scratch slots,
  freeing scratch slots and releasing pages back to the pool; the request
  is resumed via prepare_context()/_resume_and_restore() on reschedule,
  the same machinery generation and the pre_cap>0 revert already use.

adp_balance stays fully functional and pages are still freed during the
defer window (no KV-cache capacity regression).

Validated on DSv4-Pro DEP8 + adp_balance (timeout_iters=100,
batching_wait_iters=20) + SWA + mtp0 + nvfp4: the full benchmark now
completes (Total Token Throughput ~13.6k tok/s) where the unpatched
build crashed during warmup with "SWA scratch requires ...".

Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
@qiaoxj07
qiaoxj07 requested a review from a team as a code owner June 4, 2026 04:31
@qiaoxj07
qiaoxj07 requested a review from jiaganc June 4, 2026 07:07
@jiaganc

jiaganc commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

This issue is fixed in #14849. I suggest you to use that PR.

@qiaoxj07 qiaoxj07 closed this Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants