Skip to content

llama: refactor fused ops - #24646

Merged
am17an merged 4 commits into
ggml-org:masterfrom
am17an:refactor-fused-ops
Jul 8, 2026
Merged

llama: refactor fused ops#24646
am17an merged 4 commits into
ggml-org:masterfrom
am17an:refactor-fused-ops

Conversation

@am17an

@am17an am17an commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Overview

Refactor op detection to be more streamlined, add llm_graph_fused_node so that we don't rely on tensor names. The intention is to make adding new attention ops (e.g. #24231 + various other new interesting ops in dsv4) easier.

Additional information

Requirements

@github-actions github-actions Bot added the model Model specific label Jun 15, 2026
This was referenced Jul 1, 2026
Merged
Merged
Draft
@ggerganov ggerganov mentioned this pull request Jul 6, 2026
Merged
@am17an
am17an marked this pull request as ready for review July 6, 2026 09:56
@am17an
am17an requested review from CISC and ggerganov as code owners July 6, 2026 09:56
@ggerganov ggerganov self-assigned this Jul 6, 2026
ggerganov
ggerganov reviewed Jul 8, 2026
View reviewed changes

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LLAMA_TENSOR_NAME_s from llama-impl.h are no longer needed - can be removed

Comment thread src/llama-context.cpp Outdated
Comment thread src/llama-graph.h Outdated
ggerganov
ggerganov reviewed Jul 8, 2026
View reviewed changes
Comment thread src/llama-graph.h Outdated
@am17an

am17an commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@ggml-org/maintainers can I get another approval?

@am17an
am17an merged commit 90e0f5c into ggml-org:master Jul 8, 2026
25 checks passed
@ldumancas ldumancas mentioned this pull request Jul 14, 2026
Open
fewtarius added a commit to fewtarius/CachyLLama that referenced this pull request Jul 19, 2026
@fewtarius
merge upstream/master: 135 commits (DFlash, Hy3 MTP, Q2_0, DeepseekV4…
cf4f42e 
…, OpenCL Q6_K/Adreno, CORS, checkpoint min-step, prompt cache refactor, MoE expert API stays)

Upstream highlights since 6be7459:
- model: DFlash speculative with KV rotation (ggml-org#25823)
- model: Hy3 (hy_v3) with MTP speculative decoding (ggml-org#25395)
- model: DeepseekV4 with fused hyper-connection ops (ggml-org#25585)
- ggml: 0.17.0, LIGHTNING_INDEXER, out_prod, f16 set_rows
- vulkan: Q2_0 support, native e2m1/e4m3 conversions, transfer-queue race fix
- CUDA: MMQ kernel config refactor (ggml-org#24127), tighter MMQ src1 buffer for fp4 (ggml-org#25613), CUDA graphs on Volta/Turing, MoE gate/up dedup, CUDA Virtual Devices
- ROCm: hexagon L2 cache rework, native fp4, FP16/INT8 coopmat on AMD
- SYCL: Battlemage flash attention via oneDNN XMX, XIELU op, fp16 conv2d_dw
- OpenCL: Q6_K GEMM/GEMV fix, ragged-tile MoE prefill FP16, Adreno vectorized LD/ST, A7x optimizations, ABS op
- kleidiai: SME2 f32 kernel, SME vs SME2 dispatch
- server: refactor prompt cache state ownership (ggml-org#25649) - new server_prompt_cache_state separates prompt metadata from KV data
- server: evict checkpoints within min-step (ggml-org#25472)
- server: text-only slot save/restore with mtmd (ggml-org#25076)
- server: --cors-* options (ggml-org#25655)
- server: refactored server_stream (ggml-org#25541)
- server: respect min-step when splitting prompt batches (ggml-org#25420)
- server: move chat-template thinking probe inside init try/catch (ggml-org#24093)
- common: auto-download dflash/eagle3 HF sidecars (ggml-org#25811), drop --stdin mutual-exclusion, align tokenize usage
- conversion: BitNetForCausalLM, dflash tokenizer fix, split MTP export for HY V3
- llama-quant: exclude i32 ffn_gate_tid2eid routing table, allow manual tensor types with --pure
- llama-batch: fix allowed decreasing pos in a seq (ggml-org#25449), n_keep_tail in split_equal for recurrent
- llama: refactor fused ops (ggml-org#24646), TP fix for Phi3/Bert/Plamo2/3/ChatGLM
- ui: agentic content UX, reasoning effort on mobile add sheet, MCP panel fixes, thinking menu fix
- vendor: BoringSSL 0.20250713.0
- tests: actually exercise test-recurrent-state-rollback, ds_v4_hc sentinel init, export-graph-ops graceful exit

CachyLLama preservation work (conflict resolution):

1. tools/server/server-task.h: Accept upstream's server_prompt refactor (no data member, clear() method).
   Move our t_last_used field from server_prompt to server_prompt_cache_state (where it now lives
   after the refactor). server_prompt_cache_state already has the size() method, so our old
   size() on server_prompt is no longer needed.

2. tools/server/server-context.cpp (create_checkpoint): Take upstream's min-step eviction
   pre-filter as the FIRST pass, then keep our existing highest-pos_min eviction as the
   capacity overflow fallback. These are complementary: min-step removes redundant checkpoints
   from the same task; highest-pos_min keeps the rec-window-friendly checkpoints when at cap.

3. tools/server/server-context.cpp (handle_completions_impl): Keep our std::vector<server_task>
   tasks batching for multi-prompt requests and per-user concurrency check, AND take upstream's
   res->set_req(&req) for spipe ownership transfer.

4. tools/server/server-task.cpp: Fix references to entry.tokens -> entry.prompt.tokens,
   entry.checkpoints -> entry.prompt.checkpoints, entry.n_tokens() -> entry.prompt.n_tokens().
   Update find_eviction_candidate return type from list<server_prompt>::iterator to
   list<server_prompt_cache_state>::iterator.

5. ggml/src/ggml-cuda/mmq.cuh + new mmq-config-rdna3_5.cuh: Upstream's massive MMQ refactor
   moved per-architecture config into separate files but did NOT add RDNA3.5 (gfx1150/1/2/3,
   Strix Halo). Create mmq-config-rdna3_5.cuh (231 CASE entries) derived from rdna2 with
   nthreads=128 (4 warps) and I=48 (smaller X tile) matching our original Strix Halo tuning.
   Wire into both host and device dispatch paths before the RDNA4 / RDNA2 fallback.

6. README.md and AGENTS.md: Keep CachyLLama-specific links and project context where upstream
   added parallel content.

Verified:
- cmake --build builds clean (Release, CPU-only)
- llama-server starts, --help shows all CachyLLama flags preserved:
  --cache-ssd-hot-ram, --cache-ssd-warm-ram, --cache-ssd-system-prompts,
  --cache-ssd-system-max-days, --cache-ssd-no-fsync, --cache-ssd-max-conversations,
  --max-concurrent-per-user
- /expert-stats and /expert-tracking endpoints preserved
- 55/58 tests pass; 3 failures unrelated to merge:
  - test-tokenizers-ggml-vocabs: missing model downloads
  - test-jinja-py: missing jinja2 Python module
  - test-quant-type-selection: snapshot mismatch on upstream's new MXFP4_MOE heuristic

Custom CachyLLama files untouched (no upstream conflicts):
- common/kv-ssd-cache.{cpp,h}, common/kv-ssd-posix.h, common/kv-ssd-system-cache.{cpp,h}
- common/kv_page_manager.{cpp,h}
- tools/server/server-context-page-manager.{cpp,h}
- tools/server/server-context-ssd-cache.{cpp,h}
- test_kv_page_manager.cpp, tests/test-ssd-cache-caps.cpp
- STRIX_HALO_NOTES.md, docs/development/user-isolation-design.md
- .github/workflows/build-cpu.yml, build-cuda-windows.yml, build-vulkan.yml
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026
@am17an ROCm FP4 Contributor
llama: refactor fused ops (ggml-org#24646)
e4f85ef
gianni-cor pushed a commit to tetherto/qvac-fabric-llm.cpp that referenced this pull request Jul 25, 2026
@am17an @gianni-cor
llama: refactor fused ops (ggml-org#24646)
4fb86eb
@CISC CISC mentioned this pull request Aug 2, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes
@ServeurpersoCom ServeurpersoCom ServeurpersoCom approved these changes
@ngxson ngxson ngxson approved these changes
@CISC CISC Awaiting requested review from CISC CISC is a code owner

Assignees

@ggerganov ggerganov

Labels

model Model specific

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@am17an @ggerganov @ServeurpersoCom @ngxson