Skip to content

Fix nonpad_kv_seqlen in static-cache generation example (before-scatter -> after-scatter) - #365

Merged
titaiwangms merged 2 commits into
mainfrom
fix/static-cache-example-nonpad
Jun 19, 2026
Merged

Fix nonpad_kv_seqlen in static-cache generation example (before-scatter -> after-scatter)#365
titaiwangms merged 2 commits into
mainfrom
fix/static-cache-example-nonpad

Conversation

@titaiwangms

Copy link
Copy Markdown
Contributor

What

examples/static_cache_generation.py fed nonpad_kv_seqlen as the before-scatter KV count. A separate nonpad_kv_seqlen accumulator stayed always-equal to write_indices (both initialized to 0, both incremented only after session.run), so the example passed nonpad == write_indices. The spec-correct value is write_indices + cur_seq_len — the valid KV count after the current chunk is scattered into the cache, per the bottom-right is_causal contract (onnx/onnx#8068).

Impact

Wrong in every phase:

  • Prefill: passed 0 instead of prompt_len, so the kernel saw ~0 valid keys and produced wrong logits.
  • Decode: off-by-one on every step.

On the CPU EP the maskless graph runs-but-wrong silently (no error raised). This is example-only — it is the canonical usage reference and does not affect library/graph code or PR #364.

Fix

  • Feed nonpad_kv_seqlen = write_indices + cur_seq_len at the run inputs.
  • Remove the now-dead redundant nonpad_kv_seqlen accumulator (it merely duplicated write_indices).
  • Add an explanatory inline comment + docstring note so the canonical example teaches the correct after-scatter pattern.

Verified

  • Matches the authoritative reference in tests/static_cache_parity_test.py (prefill nonpad = prompt_len = write_indices(0) + query_len, lines 322-327; decode nonpad = valid_len + 1 = write_indices(valid_len) + 1, lines 361-362) and the consumer src/mobius/components/_attention.py:169-183 (the scatter precedes the maskless is_causal Attention's nonpad read).
  • Ran on CPU with Qwen/Qwen2.5-0.5B -> coherent output: The capital of France is -> Paris. It is the largest city in. Pre-fix the all-zero prefill nonpad silently produced wrong logits.
  • Triple-reviewed (code / critical / readability all ship).

Reference

titaiwangms and others added 2 commits June 19, 2026 19:44
…eration

The static-cache generation example fed the BEFORE-scatter valid KV count as
nonpad_kv_seqlen. Both nonpad_kv_seqlen and write_indices were initialized to 0
and only incremented AFTER session.run, so the value fed each step always equaled
write_indices instead of the spec-correct count. This is wrong in prefill (feeds 0
instead of prompt_len) and off-by-one on every decode step.

The model scatters K/V into the cache before the maskless is_causal Attention
reads nonpad_kv_seqlen, so under the bottom-right contract the valid count must
include the just-written chunk: write_indices + cur_seq_len. Feed that directly and
remove the now-redundant nonpad_kv_seqlen accumulator (it duplicated write_indices).

Matches the authoritative reference in tests/static_cache_parity_test.py (prefill
nonpad = prompt_len = write_indices(0) + query_len; decode nonpad = valid_len + 1 =
write_indices(valid_len) + 1) and the consumer src/mobius/components/_attention.py
(scatter precedes the nonpad read). Verified by running the example on CPU with
Qwen/Qwen2.5-0.5B: 'The capital of France is' -> ' Paris. It is the largest city in'
(coherent; pre-fix the all-zero prefill nonpad produced wrong logits).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Update the module-docstring bullet to match the fix's after-scatter semantics:
nonpad_kv_seqlen is the number of valid KV entries after the current chunk is
scattered (write_indices + cur_seq_len), not a generic running count.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes the canonical static-cache generation example to feed nonpad_kv_seqlen as the after-scatter valid-KV count (write_indices + cur_seq_len), matching the bottom-right causal is_causal contract.

Changes:

  • Correct nonpad_kv_seqlen fed into session.run() to write_indices + cur_seq_len (after-scatter valid KV count).
  • Remove the redundant nonpad_kv_seqlen accumulator that previously duplicated write_indices.
  • Add docstring and inline comments clarifying the after-scatter nonpad_kv_seqlen pattern for prefill and decode.

@github-actions

Copy link
Copy Markdown

Performance Comparison

Comparing 6630bd3c598f42

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0%
bert (feature-extraction) num_nodes 60 60 +0.0%
falcon model_size_bytes 364 KB 364 KB +0.0%
falcon num_nodes 68 68 +0.0%
gemma2 model_size_bytes 428 KB 428 KB +0.0%
gemma2 num_nodes 107 107 +0.0%
gpt2 model_size_bytes 388 KB 388 KB +0.0%
gpt2 num_nodes 54 54 +0.0%
llama model_size_bytes 425 KB 425 KB +0.0%
llama num_nodes 62 62 +0.0%
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0%
llama (static-cache) num_nodes 58 58 +0.0%
mamba (ssm-text-generation) model_size_bytes 296 KB 296 KB +0.0%
mamba (ssm-text-generation) num_nodes 98 98 +0.0%
phi3 model_size_bytes 421 KB 421 KB +0.0%
phi3 num_nodes 60 60 +0.0%
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0%
phi3 (static-cache) num_nodes 56 56 +0.0%
qwen2 model_size_bytes 425 KB 425 KB +0.0%
qwen2 num_nodes 62 62 +0.0%
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0%
qwen2 (static-cache) num_nodes 58 58 +0.0%
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB +0.0%
qwen3_5_moe (hybrid-text-generation) num_nodes 275 275 +0.0%
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB +0.0%
qwen3_5_text (hybrid-text-generation) num_nodes 129 129 +0.0%
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB +0.0%
qwen3_5_vl (hybrid-qwen-vl) num_nodes 413 413 +0.0%
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0%
t5 (seq2seq) num_nodes 166 166 +0.0%
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0%
whisper (speech-to-text) num_nodes 128 128 +0.0%

No performance regressions.

@codecov

codecov Bot commented Jun 19, 2026

Copy link
Copy Markdown

The author of this PR, titaiwangms, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at support@codecov.io with any questions.

@titaiwangms
titaiwangms merged commit a49e0f8 into main Jun 19, 2026
22 of 23 checks passed
@titaiwangms
titaiwangms deleted the fix/static-cache-example-nonpad branch June 19, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants