Commit a49e0f8
Fix nonpad_kv_seqlen in static-cache generation example (before-scatter -> after-scatter) (#365)
## What
`examples/static_cache_generation.py` fed `nonpad_kv_seqlen` as the
**before-scatter** KV count. A separate `nonpad_kv_seqlen` accumulator
stayed always-equal to `write_indices` (both initialized to 0, both
incremented only *after* `session.run`), so the example passed `nonpad
== write_indices`. The spec-correct value is `write_indices +
cur_seq_len` — the valid KV count **after** the current chunk is
scattered into the cache, per the bottom-right `is_causal` contract
(onnx/onnx#8068).
## Impact
Wrong in **every** phase:
- **Prefill**: passed `0` instead of `prompt_len`, so the kernel saw ~0
valid keys and produced wrong logits.
- **Decode**: off-by-one on every step.
On the CPU EP the maskless graph **runs-but-wrong silently** (no error
raised). This is **example-only** — it is the canonical usage reference
and does **not** affect library/graph code or PR #364.
## Fix
- Feed `nonpad_kv_seqlen = write_indices + cur_seq_len` at the run
inputs.
- Remove the now-dead redundant `nonpad_kv_seqlen` accumulator (it
merely duplicated `write_indices`).
- Add an explanatory inline comment + docstring note so the canonical
example teaches the correct **after-scatter** pattern.
## Verified
- Matches the authoritative reference in
`tests/static_cache_parity_test.py` (prefill nonpad = prompt_len =
write_indices(0) + query_len, lines 322-327; decode nonpad = valid_len +
1 = write_indices(valid_len) + 1, lines 361-362) and the consumer
`src/mobius/components/_attention.py:169-183` (the scatter precedes the
maskless `is_causal` Attention's `nonpad` read).
- Ran on CPU with `Qwen/Qwen2.5-0.5B` -> coherent output: `The capital
of France is` -> ` Paris. It is the largest city in`. Pre-fix the
all-zero prefill nonpad silently produced wrong logits.
- Triple-reviewed (code / critical / readability all ship).
## Reference
- onnx/onnx#8068 (causal bottom-right errata).
- Relates to the static-cache + Flash enablement in PR #364 — this is a
separate, example-only follow-up.
---------
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent 6630bd3 commit a49e0f8
1 file changed
Lines changed: 13 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | 106 | | |
110 | 107 | | |
111 | 108 | | |
| |||
129 | 126 | | |
130 | 127 | | |
131 | 128 | | |
132 | | - | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
133 | 137 | | |
134 | 138 | | |
135 | 139 | | |
| |||
153 | 157 | | |
154 | 158 | | |
155 | 159 | | |
156 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
157 | 163 | | |
158 | | - | |
159 | 164 | | |
160 | 165 | | |
161 | 166 | | |
| |||
0 commit comments