feat: continuous-batching scheduler (#449 M3 Stage 2a-ii)

[inureyes]

Summary

The second half of Stage 2a (#449 M3): a minimal continuous-batching scheduler over the ragged decode graph (PR #466). B slots serve N > B requests that finish at staggered times, so queued requests are admitted mid-stream into freed slots (slot recycling). Validated by reference-equivalence under dynamic membership. With 2a-i this completes Stage 2a. Spike-only, so CI is unaffected.

What is in it

• Shim xla_llama_refresh_mirror: pulls the live rank-5 KV of all active slots back to the host mirror (d2h). Mid-stream admit is refresh + prefill_slot(freed) + commit so admitting one sequence does not disturb the others (refresh captures active slots' advanced state, prefill_slot overwrites only the freed slot, commit re-uploads).
• Driver src/bin/llama_sched.rs: B slots, a FIFO queue of N requests with varied lengths and varied token caps. The loop admits queued requests into free slots, ragged-decodes all active slots (inactive rows masked and ignored), evicts at the cap or an EOS token, and recycles the slot. Phase 1 captures each request's capped single-seq reference; Phase 2 asserts each request's stream equals its reference.

Validation (reference-equivalence under recycling)

Because caps vary, requests evict at different steps and later requests admit mid-stream while others are still decoding, so a passing run proves refresh/commit preserves the active slots and a recycled slot starts clean.

Device	B	N	caps	admits	result
CPU local-task	3	6	12-14	6 (3 initial + 3 mid-stream)	6/6 reference-exact
CUDA GB10	4	8	12-39	8 (4 initial + 4 mid-stream)	8/8 reference-exact

Findings

• Continuous batching is correct. Admit, evict, slot recycling, and mid-stream join all preserve per-request output: each sequence is identical to its standalone run.
• The host-mirror admit is correct but not the throughput path. Each admit does a full rank-5 KV round-trip (d2h refresh + h2d commit), so the tok/s here is a correctness result, not a throughput one (Stage 1 measured raw batched throughput). 2b will replace it with a device-side write of only the admitted slot's region.
• Static B with masked inactive slots works. A half-full batch produces the same per-request output as a full one.

Full result in spike/iree-ffi/FINDINGS_sched.md; the Stage 2 plan in spike/openxla/STAGE2_DESIGN.md.

Next

2b: productize an XlaBatchEngine in mlxcel-xla (contiguous KV, fixed B_max, a device-side slot write instead of the host-mirror round-trip, greedy, CLI/bench), then 2c the common BatchEngine trait + server integration.

Refs #449