Skip to content

feat(ds4): add monolithic Strix Halo concurrent serving - #598

Draft
Graffioh wants to merge 2 commits into
Luce-Org:mainfrom
Graffioh:codex/ds4-concurrency-strix
Draft

feat(ds4): add monolithic Strix Halo concurrent serving#598
Graffioh wants to merge 2 commits into
Luce-Org:mainfrom
Graffioh:codex/ds4-concurrency-strix

Conversation

@Graffioh

@Graffioh Graffioh commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Scope

This rebuilds the concurrency work for DeepSeek4 only. Qwen source and GGML kernels are unchanged.

It supersedes the experimental direction in #594, #595, #596 and Graffioh#16, while incorporating the MoE concurrency discussion. The supported topology is deliberately narrow: one local HIP gfx1151 Strix Halo with the complete model and all experts resident.

Implementation

  • Adds a model-neutral continuous scheduler and SeqEngine boundary with admission, cancellation, fair prompt progress, slow-client isolation, and reusable slot lifecycle handling.
  • Extends the shared paged allocator for reservations and rolling decode headroom.
  • Adds DeepSeek4 persistent paged raw MLA, compressed MLA, and indexer state using 128-token model pages.
  • Lowers up to 16 independent lanes into an exact gathered DeepSeek4 graph. Live decode rows share a weight pass; selected prompt rows advance one exact token per sequence in that same pass.
  • Keeps sampling history and device block-table state isolated per reusable sequence slot.
  • Auto-enables paged mode for --max-concurrency 2..16 and exposes optional physical-pool sizing through --kv-pool-tokens.

Guardrails

This path fails closed for CUDA, non-gfx1151 HIP devices, layer or remote target splits, DFLASH_DS4_MOE_TP, drafts/DSpark, DDTree, PFlash/KVFlash, fused decode, approximate prefill, windowed attention, and prefix-cache parking. It requires the full model to remain resident on the single Strix Halo.

Strix Halo concurrency screening

Model GGUF: DeepSeek-V4-Flash-ROCMFP2-STRIX.gguf
Device: one local HIP gfx1151 Strix Halo (Radeon 8060S), with the full model and experts resident.
Protocol: one fresh server and same-concurrency warm-up per level; short ragged, disjoint prompts; exact prefill; greedy streaming; 256 forced completion tokens per request; one repeat.

Concurrent requests token/s
1 10.50 tok/s
8 25.88 tok/s
16 24.43 tok/s

No llama.cpp comparison is stated yet: the available llama-server rejects this ROCmFP GGUF at load time (invalid ggml type 101). A compatible llama.cpp/DeepSeek4 ROCmFP build (or a compatible GGUF) is required before publishing paired DS4-vs-llama results. This is a one-repeat screening, not a performance claim; use five paired repeats once that baseline is available.

Validation

  • Built the HIP gfx1151 server and all focused concurrency targets, including test_deepseek4_unit.
  • 358 selected scheduler, server, feature-gate, slot-manager, contract, page-layout, and paged-cache checks pass.
  • Paged allocator suite: 13/13 pass.
  • Model-backed DeepSeek4 screening completed at C=1, C=4, C=8, and C=16 (29/29 requests; 7,424/7,424 completion tokens; zero failures).
  • Final publication commit is byte-identical to the validated worktree.
  • Qwen directories, Qwen tests, and GGML kernels have no diff from upstream.

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant