feat(ds4): add monolithic Strix Halo concurrent serving - #598
Draft
Graffioh wants to merge 2 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
This rebuilds the concurrency work for DeepSeek4 only. Qwen source and GGML kernels are unchanged.
It supersedes the experimental direction in #594, #595, #596 and Graffioh#16, while incorporating the MoE concurrency discussion. The supported topology is deliberately narrow: one local HIP
gfx1151Strix Halo with the complete model and all experts resident.Implementation
SeqEngineboundary with admission, cancellation, fair prompt progress, slow-client isolation, and reusable slot lifecycle handling.--max-concurrency 2..16and exposes optional physical-pool sizing through--kv-pool-tokens.Guardrails
This path fails closed for CUDA, non-
gfx1151HIP devices, layer or remote target splits,DFLASH_DS4_MOE_TP, drafts/DSpark, DDTree, PFlash/KVFlash, fused decode, approximate prefill, windowed attention, and prefix-cache parking. It requires the full model to remain resident on the single Strix Halo.Strix Halo concurrency screening
Model GGUF:
DeepSeek-V4-Flash-ROCMFP2-STRIX.ggufDevice: one local HIP
gfx1151Strix Halo (Radeon 8060S), with the full model and experts resident.Protocol: one fresh server and same-concurrency warm-up per level; short ragged, disjoint prompts; exact prefill; greedy streaming; 256 forced completion tokens per request; one repeat.
No llama.cpp comparison is stated yet: the available
llama-serverrejects this ROCmFP GGUF at load time (invalid ggml type 101). A compatible llama.cpp/DeepSeek4 ROCmFP build (or a compatible GGUF) is required before publishing paired DS4-vs-llama results. This is a one-repeat screening, not a performance claim; use five paired repeats once that baseline is available.Validation
gfx1151server and all focused concurrency targets, includingtest_deepseek4_unit.