cuda: AMD RDNA4 Q1_0/Q2_0 — HIP-path vec_dots (+37%/+16% decode), opt-in quant dedup, opt-in hipBLASLt prefill routes - #116
Open
The-Monk wants to merge 3 commits into
Open
Conversation
…in hipBLASLt prefill routes
HIP-guarded vec_dot_q2_0_q8_1 (dynamic-selector __byte_perm lowers poorly on
AMD): Bonsai-27B Q2_0 decode 34.52 -> 47.20 t/s on gfx1201, PPL byte-identical.
Opt-in GGML_HIP_DEDUP_MMVQ_QUANT skips redundant sibling activation quantize
launches (+2-3% decode here, more under spec-decode verify). Opt-in
GGML_HIP_{Q1_0,Q2_0}_HIPBLASLT_PREFILL routes large-M matmuls to hipBLASLt
int8 GEMM with a self-tuning algo cache and buffer-free weight-cache
invalidation: pp1024 +37.5% (Q2_0) / +17.8% (Q1_0) warm. hipBLASLt is an
optional dependency; without it (Windows HIP) and on CUDA builds the routes
compile to inert stubs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ode)
Same treatment as the Q2_0 HIP path: bit-spread raw code bits c in {0,1} into
{0,1}-byte dp4a operands and fold dot(s,u) = 2*dot(c,u) - sum(u) through the
q8_1 stored sum, replacing the select-chain fallback in unpack_q1_0_bytes.
Bonsai-27B Q1_0 tg128 on 1x R9700: 54.53 -> 63.03 t/s (64.93 with
GGML_HIP_DEDUP_MMVQ_QUANT=1). test-backend-ops MUL_MAT: all pass. CUDA path
untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The-Monk
changed the title
Collaborator
|
@The-Monk Thanks for the chnages, will do a more careful review this week. By the way Q1_0 is fully upstreamed to llama.cpp so migth be worth also trying to upstream it there, so check whetehr they have added better AMD support since we merged things. Happy to merge these here but also worst trying to upstream to main llama.cpp as well since most things are already there too (with the caveat of Q2_0 being slightly different there). |
There was a problem hiding this comment.
Pull request overview
Adds RDNA4 optimizations for Q1_0/Q2_0 inference in the shared CUDA/HIP backend.
Changes:
- Optimizes HIP vector-dot decoding.
- Adds optional activation-quantization deduplication.
- Adds optional hipBLASLt prefill routes, tuning, and weight caches.
Policy note: The PR description declares Claude Code generation, conflicting with AGENTS.md:36-40, which prohibits AI-written PR descriptions.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
ggml/src/ggml-hip/CMakeLists.txt |
Adds optional hipBLASLt discovery. |
ggml/src/ggml-cuda/vecdotq.cuh |
Adds HIP/MUSA low-bit vector-dot paths. |
ggml/src/ggml-cuda/mul_mat_q2_0_hipblaslt.cuh |
Declares the Q2_0 prefill route. |
ggml/src/ggml-cuda/mul_mat_q2_0_hipblaslt.cu |
Implements Q2_0 hipBLASLt prefill. |
ggml/src/ggml-cuda/mul_mat_q1_0_hipblaslt.cuh |
Declares the Q1_0 prefill route. |
ggml/src/ggml-cuda/mul_mat_q1_0_hipblaslt.cu |
Implements Q1_0 hipBLASLt prefill. |
ggml/src/ggml-cuda/mmvq.cu |
Adds activation-quantization deduplication. |
ggml/src/ggml-cuda/hipblaslt_wcache.cuh |
Declares cache invalidation APIs. |
ggml/src/ggml-cuda/hipblaslt_wcache.cu |
Implements invalidator registration. |
ggml/src/ggml-cuda/ggml-cuda.cu |
Integrates routes and cache lifecycle. |
ggml/src/ggml-cuda/common.cuh |
Adds FP8 conversion and cache state. |
Suppressed comments (2)
ggml/src/ggml-cuda/mul_mat_q1_0_hipblaslt.cu:415
- The claim that all uses share one stream is false: the backend dispatches concurrent branches and devices independently. Both cache builders publish the entry immediately after enqueueing requantization, so another stream can hit and read it before initialization completes. Publish a completion event and wait on hits, or synchronize before insertion as the existing Hopper cache does at
mmq-hopper-q1.cu:149-153.
// Returns cached int8 weight (building it on first miss if within budget), or
// nullptr -> caller must requant on-the-fly. Build + all uses share the stream,
// so the one-time requant is correctly ordered before any GEMM that reads it.
ggml/src/ggml-cuda/mul_mat_q2_0_hipblaslt.cu:425
- The cache entry is published immediately after requantization is enqueued. A concurrent branch or request on another stream can hit this entry and launch GEMM before the converted weights are initialized. Publish a completion event and wait on hits, or synchronize before insertion as the existing Hopper cache does at
mmq-hopper-q1.cu:149-153.
g_wcache_bytes += need;
auto res = g_wcache.emplace(key, c);
return &res.first->second;
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+786
to
+789
| const int qs0 = bq2_0->qs[offset + 0] | (bq2_0->qs[offset + 1] << 8) | | ||
| (bq2_0->qs[offset + 2] << 16) | (bq2_0->qs[offset + 3] << 24); | ||
| const int qs1 = bq2_0->qs[offset + 4] | (bq2_0->qs[offset + 5] << 8) | | ||
| (bq2_0->qs[offset + 6] << 16) | (bq2_0->qs[offset + 7] << 24); |
Comment on lines
+1230
to
+1231
| const bool dedup_hit = dedup_quant && | ||
| ctx.mmvq_quant_cache_tensor == src1 && ctx.mmvq_quant_cache_buf; |
Comment on lines
+900
to
+903
| int man = (int) (ax * 512.0f + 0.5f); | ||
| if (man > 7) { | ||
| man = 7; | ||
| } |
Comment on lines
+487
to
+491
| if (src0->type != GGML_TYPE_Q1_0) return false; | ||
| if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; | ||
| if (src0->ne[2] != 1 || src0->ne[3] != 1) return false; | ||
| if (src1->ne[2] != 1 || src1->ne[3] != 1) return false; | ||
| if (src0->ne[0] != src1->ne[0] || src0->ne[0] % Q1K != 0) return false; |
Comment on lines
+452
to
+456
| if (src0->type != GGML_TYPE_Q2_0) return false; | ||
| if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; | ||
| if (src0->ne[2] != 1 || src0->ne[3] != 1) return false; | ||
| if (src1->ne[2] != 1 || src1->ne[3] != 1) return false; | ||
| if (src0->ne[0] != src1->ne[0] || src0->ne[0] % Q2K != 0) return false; |
Comment on lines
+398
to
+399
| std::map<const void *, cached_w> g_wcache_i8; | ||
| std::map<const void *, cached_w_f8> g_wcache_f8; |
| // cached (requant paid once), the rest fall back to on-the-fly pool requant. | ||
| // hipMalloc failure also falls back -- never OOM-crash (the Stage-3 lesson). | ||
| struct cached_w { int8_t * q8 = nullptr; float * wscale = nullptr; size_t bytes = 0; }; | ||
| std::map<const void *, cached_w> g_wcache; |
Comment on lines
+560
to
+563
| // ---- activation int8/e4m3 (per-token) + 4-byte accumulator, from the pool ---- | ||
| ggml_cuda_pool_alloc<int8_t> x8 (ctx.pool(), (size_t)K * M); | ||
| ggml_cuda_pool_alloc<float> asc (ctx.pool(), (size_t)M); | ||
| ggml_cuda_pool_alloc<int32_t> acc (ctx.pool(), (size_t)N * M); // i32 (int8) or reinterpreted f32 (fp8) |
Review hardening (Copilot flag on the converted-weight caches): entries are published under the mutex right after the requant kernel is launched, so ordering vs the GEMM that reads them was guaranteed only for same-stream consumers. Record a build_done event on the build stream and make any consumer on a different stream hipStreamWaitEvent on it before using the entry (falling back to on-the-fly requant if the wait cannot be issued). Events are destroyed with their entries in the invalidators. No behavior change on the current single-compute-stream-per-device backend (the wait never fires); the previously documented invariant is now enforced. Verified: build clean, both routes smoke-tested on 1x R9700 (Q1_0 pp1024 1500 t/s, Q2_0 1487 t/s, build+hit paths exercised). CCA (Claude Code Augmented)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The AMD side of the Q1_0/Q2_0 story. This fork's HIP release bundles currently run Prism's binary/ternary formats through kernels tuned for NVIDIA; on RDNA4 hardware that leaves a lot on the table. This PR contributes three self-contained pieces from our RDNA4 fork (The-Monk/llama.cpp
roc8, the kernel base of The Rock8), re-measured on this tree on 2× Radeon AI PRO R9700 (gfx1201, ROCm/TheRock 7.14).All numbers below: Bonsai-27B (
qwen35hybrid), 1 GPU,llama-bench -r 5(tg) /-r 3(pp).1. HIP-path
vec_dotfor Q2_0 (+37%) and Q1_0 (+16%), on by defaultThe existing Q2_0 vec_dot extracts symbols with dynamic-selector
__byte_permchains, which are built around NVIDIA'sPRMT; on HIP they lower poorly and the decode GEMV pays for it end-to-end. This PR adds an AMD path using the same split this file already uses forunpack_q1_0_bytes: bit-spread the raw 2-bit codes into bytes and use the identitydot(s,u) = dot(c,u) − sum(u), applying the offset once via the q8_1 stored sum — plain shift/mask +dp4a, no per-code arithmetic. The CUDA path is untouched.The second commit gives Q1_0 the same treatment with the binary form of the identity (
s = 2c − 1⇒dot(s,u) = 2·dot(c,u) − sum(u)), replacing the select-chain fallback inunpack_q1_0_bytesthat materialized ±1 bytes on a VALU-bound kernel.Perplexity is identical (wikitext-2, 20 chunks:
11.5603 ± 0.47269both builds, Q2_0), and binary now decodes faster than ternary, as its byte ratio says it should.2. Opt-in activation-quant dedup (
GGML_HIP_DEDUP_MMVQ_QUANT), default OFFSibling mmvq matmuls that read the same activation tensor (wq/wk/wv, ffn_gate/ffn_up, wqkv/wqkv_gate) each launch a
quantize_row_q8_1that recomputes byte-identical output. With the flag set, the quantization is computed once per activation tensor and reused; the cache lives on the backend context, is keyed by tensor pointer, and is reset at everygraph_compute, so entries cannot survive a graph rebuild.GGML_HIP_DEDUP_MMVQ_QUANT_BATCHadditionally widens it tone11>1(spec-decode verify batches).Measured here: Q2_0 48.20 (+2.1% on top of #1), Q1_0 64.93 (+3.0% on top of the new Q1_0 dot). On our fork the same lever is worth considerably more under MTP self-speculative decode, where the verify pass multiplies the redundancy — that's why the
_BATCHvariant exists. Lossless: the cached bytes are the bytes the skipped launch would have produced; PPL run with both flags on is byte-identical to stock.3. Opt-in Q1_0/Q2_0 hipBLASLt prefill routes, default OFF, optional dependency
Large-M (prefill) matmuls routed through hipBLASLt int8 GEMM (dequantize once → cached int8 weights + per-channel scale; activations int8 per call), with a self-tuning per-shape algorithm cache — gfx1201 has no hipBLASLt cost model and the heuristic picks badly. Includes the weight-cache invalidation registry (
hipblaslt_wcache): caches are keyed on device addresses, so the backend drops entries when it frees a buffer — without this, multi-model runs and server model swaps can silently serve stale weights from a reused allocation.Knobs per format (
<FMT>∈Q1_0,Q2_0):GGML_HIP_<FMT>_HIPBLASLT_PREFILL=1(enable),_MTHRESH=<n>(default 384 — deliberately conservative because the crossover is model-shape-dependent; override per model),_TUNE_CACHE=<path>,_NOTUNE=1,_WCACHE_MB=<n>,_FP8=1(dequantize to e4m3 instead of int8).Build impact:
find_package(hipblaslt QUIET)— optional. Not found (e.g. Windows HIP, where hipBLASLt doesn't exist) → routes compile as inert stubs and the release bundles are unaffected. CUDA builds compile the same stubs (__HIP_PLATFORM_AMD__-guarded).What we tried and deliberately did NOT include
Our fork runs Q2_0 with
nwarps=8/rows_per_block=3on RDNA4. Transplanting those constants onto this tree's kernel was a disaster (nwarps=8 → 8.2 t/s) to neutral (full sweep: rpb 2/3/4 × nwarps 2/4/8 all ≤ +0.7% over stock). Launch-geometry constants do not transfer between kernel implementations; this tree's geometry is already at its optimum, so no tuning changes are included.Testing
test-backend-ops test -o MUL_MAT: all pass, both with default env and with both dedup flags set.-DCMAKE_HIP_ARCHITECTURES=gfx1201.Reproduce:
cmake -B build -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DCMAKE_BUILD_TYPE=Release cmake --build build -j --target llama-bench test-backend-ops ./build/bin/llama-bench -m Ternary-Bonsai-27B-Q2_0.gguf -ngl 999 -p 0 -n 128 -r 5 GGML_HIP_DEDUP_MMVQ_QUANT=1 ./build/bin/llama-bench -m ... GGML_HIP_Q2_0_HIPBLASLT_PREFILL=1 GGML_HIP_Q2_0_HIPBLASLT_TUNE_CACHE=/tmp/t.bin \ ./build/bin/llama-bench -m ... -p 1024 -n 0 # run twice; first run pays the tuning costStanding offer
The dual-R9700 box these numbers come from is available for validating HIP release candidates or future AMD-touching PRs — happy to run your release CI artifacts against real gfx1201 before they ship.
🤖 Generated with Claude Code