Skip to content

ggml-backend-meta: fix device and buffer type names - #25688

Open
Frozenlock wants to merge 1 commit into
ggml-org:masterfrom
Frozenlock:fix-meta-naming
Open

ggml-backend-meta: fix device and buffer type names#25688
Frozenlock wants to merge 1 commit into
ggml-org:masterfrom
Frozenlock:fix-meta-naming

Conversation

@Frozenlock

@Frozenlock Frozenlock commented Jul 14, 2026

Copy link
Copy Markdown

Overview

Fixes the names of meta devices and meta buffer types: the constructors iterate the moved-from constructor parameter instead of the member of the same name, so every meta device and buffer type is named "Meta()".

With a single meta device this is cosmetic (the current -sm tensor path), but any code that distinguishes buffer types by name breaks as soon as more than one meta device exists — e.g. the KV cache context map in llama-kv-cache.cpp collapses the buffer types of distinct meta devices into one entry, allocating all layers' caches on the first device.

Verified: with two meta devices, names render as Meta(CUDA0,CUDA1) / Meta(CUDA2,CUDA3) and the per-layer buffer-type maps no longer collapse.

Additional information

Found while experimenting with tensor parallelism for MLA architectures on a 12x RTX 3090 rig (multiple meta devices in one process). Related follow-up work will be proposed separately.

Requirements

  • I have read and agree with the contributing guidelines[1]
  • AI usage disclosure: YES. the bug was found, the fix authored, and the validation runs executed by an AI coding agent operating my hardware under my direction.

[1] I am NOT a cpp programmer, so I'm not in a position to evaluate the quality as I would with my main programming languages. That's my way on contributing back to this project. Given the contributing guidelines, I'll understand if you don't accept the PR.


Edit:

Results from the follow-up work (for context)

On 12x RTX 3090 (PCIe Gen4), GLM-5.2 744B-A40B at UD-IQ2_M:

config decode (novel text) decode (code rewrite) context
-sm layer (baseline) 23 t/s 23 t/s 384K
MLA TP, hybrid 2x6 + spec 33 t/s 69 t/s 96K
MLA TP, hybrid 4x3 + spec 42 t/s 107 t/s 48K

"spec" = --spec-type ngram-map-k4v,draft-mtp (the GLM MTP head, ported from #24868 to
GLM-DSA). TP output verified byte-identical to -sm layer on GLM-4.7-Flash before any
speculative decoding was added.

Branches:
mla-tensor-parallel
hybrid-tp-pp
mla-mtp

@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Jul 14, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown

Hi @Frozenlock, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

  • AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

The constructors iterated the moved-from constructor parameter instead of
the member of the same name, so every meta device and buffer type was
named "Meta()". With a single meta device this is cosmetic, but code
that distinguishes buffer types by name (e.g. the KV cache context map)
breaks as soon as more than one meta device exists.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant