From d827b3336783eb5bd788d0cfaa30fe3541c6737d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 20:31:12 +0000 Subject: [PATCH] perf(vulkan): ragged M reaches coopmat -- 27B prefill 21.5x The largest win of this campaign, and it was self-inflicted. A prefill-dominated 27B run profiled with GPU timestamps put vt_matmul -- the UNTILED SCALAR kernel -- at 433 calls, 409,611.9 ms, 99.9% of GPU time, 945.99 ms/call. That is ~96 GFLOP/s, about 1% of this device. GPU busy was 95% of wall, so prefill was never host-bound and the remaining reference-tier ops were ~0.1%. Three prior structural attributions were wrong; one timestamp profile settled it. WHY COOPMAT DECLINED HAD TO BE MEASURED. Both candidates a reader would guess were excluded by inspection: every dimension is a whole tile (5120, 17408, 256) and activations are bf16. So the predicate was made to report itself under VT_VULKAN_DISPATCH_STATS, and it named the clause immediately: coopmat DECLINED: M is not a multiple of 16 (a.dtype=2 b.dtype=2 m=17 k=5120 n=10240) m = tokens + 1, so a 512-token prompt gives 513 and 513 % 16 == 1. EVERY prefill GEMM at every prompt length fell to the scalar kernel. THE CAUSE WAS THE EARLIER HANG FIX. coopMatLoad reads a full 16x16 tile unmasked, so at M=1 it read ~30 KB past the activation buffer and the fence never signalled. That was fixed by requiring m % 16 == 0 && n % 16 == 0 -- correct, and it is why the 27B runs instead of hanging. The note left at the time said a masked or padded load was the better long-term answer and was not attempted. That deferred work WAS the prefill bottleneck, hidden because the only model then running on Vulkan was opt-125m, whose small GEMMs made the scalar path cheap. THE FIX SHIFTS THE TRAILING TILE BACK rather than masking. A tile that would overrun M slides down to start at M-16 and therefore reads only real rows. Exact, not approximate: a result row depends solely on that row of A and on B, never on which tile computed it, so rows shared with the previous tile recompute to bit-identical values and the duplicate stores write the same bytes. It needs M >= 16, which the predicate now requires in place of M % 16 == 0; below that there is no in-bounds window, and decode (M=1) is served by the GEMV tactic. MEASURED, GB10, Qwen3.6-27B, 512-token prefill: GEMM ms/call 945.99 -> 30.40 31.1x prefill tok/s 1.18 -> 25.41 21.5x E2E ms 433,321 -> 20,192 21.5x vt_matmul is gone from the profile. 21.5x is far outside this box's 2.1x noise band, so it is callable at n=1. A 512-token prefill is now 20.1 s against llama.cpp's ~1.1 s -- 18x behind, down from 244x. Gate 26/26, 1563 assertions ON GB10; the new ragged-M case asserts the TACTIC and exactness at M=17 and short-circuits on llvmpipe, which has no coopmat (local run shows 1020). opt-125m STRICT 6/6 token-exact. Durable lesson: a correctness fix can silently become a performance cliff, and a selection predicate able to route an entire model onto the correctness tier should be able to say so. That diagnostic now ships behind the stats flag. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 69 +++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- src/vt/vulkan/shaders/vt_matmul_coopmat.comp | 19 +- src/vt/vulkan/vulkan_ops.cpp | 56 +- src/vt/vulkan/vulkan_spirv.cpp | 575 ++++++++++--------- tests/vt/test_vulkan_backend.cpp | 70 +++ 7 files changed, 504 insertions(+), 289 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index f9021f2d1..311fa0294 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -15982,3 +15982,72 @@ loads, or caching the K/V slice across the query heads that share a KV head — Recorded because the hypothesis was specific and the refutation is reusable: this is the second kernel this session where a barrier-count argument looked compelling and measured flat (the subgroup GEMV was the first). + +### ★ RAGGED M: a one-line predicate was costing 21.5x on prefill (2026-08-08, GB10) + +The largest win of this campaign, and it was self-inflicted. + +**How it was found.** Two GDN increments moved coverage 11 -> 3 and speed not at +all, so a prefill-dominated run was profiled with GPU timestamps: + + vt_matmul 433 calls 409,611.9 ms 99.9% of GPU 945.99 ms/call + +The UNTILED SCALAR kernel — the portable correctness tier — was carrying every +prefill GEMM at ~96 GFLOP/s, roughly **1% of what GB10 can do**. GPU busy was 95% +of wall, so prefill was never host-bound, and the remaining reference-tier ops +accounted for ~0.1%. **Three prior structural attributions were wrong; the +timestamp profile settled it in one run.** + +**Why coopmat declined, measured not reasoned.** Both obvious candidates were +excluded by reading the source: every dimension is a whole tile (5120, 17408, +256) and activations are bf16 (`DBuf dx(d, DType::kBF16, ...)`). The predicate had +to be made to report itself, gated behind `VT_VULKAN_DISPATCH_STATS`: + + coopmat DECLINED: M is not a multiple of 16 (a.dtype=2 b.dtype=2 m=17 k=5120 n=10240) + +**`m = tokens + 1`.** A 512-token prompt gives m=513, and `513 % 16 == 1`. Every +prefill GEMM, at every prompt length, fell to the scalar kernel. + +**The cause was my own hang fix from the previous day.** `coopMatLoad` reads a +full 16x16 tile unmasked, so at M=1 it read ~30 KB past the activation buffer and +the fence never signalled. That was fixed by requiring `m % 16 == 0 && n % 16 == +0` — correct, and it is why the 27B runs at all instead of hanging. The note left +at the time read *"a masked/padded load is the better long-term answer, not +attempted here."* That deferred work WAS the prefill bottleneck. It went unnoticed +because the only model then running on Vulkan was opt-125m, whose small GEMMs made +the scalar path cheap. + +**The fix: shift the trailing tile back, do not mask.** A tile that would overrun +M slides down to start at `M-16`, so it reads only REAL rows. Exact, not +approximate: a result row depends solely on that row of A and on B, never on which +tile computed it, so the rows shared with the previous tile recompute to +bit-identical values and the duplicate stores write the same bytes. It needs +`M >= 16`, which the predicate now requires in place of `M % 16 == 0`; below that +there is no in-bounds 16-row window, and decode (M=1) is served by the GEMV +tactic. + +**MEASURED, GB10, Qwen3.6-27B, 512-token prefill:** + +| | before | after | | +|---|---:|---:|---:| +| GEMM ms/call | 945.99 | 30.40 | **31.1x** | +| prefill tok/s | 1.18 | 25.41 | **21.5x** | +| E2E ms | 433,321 | 20,192 | **21.5x** | + +`vt_matmul` is gone from the profile; `vt_matmul_coopmat` carries 432 calls at +30.4 ms. 21.5x is far outside this box's 2.1x noise band, so it is callable at +n=1. A 512-token prefill is now **20.1 s against llama.cpp's ~1.1 s — 18x behind, +down from 244x.** + +**Gate: 26/26, 1563 assertions on GB10.** The new ragged-M case asserts the TACTIC +(`PipelineExistsFor("vt_matmul_coopmat")`) and exactness at M=17, and it +short-circuits on llvmpipe, which has no coopmat — the local run shows 1020 +assertions, the GB10 run 1563. A gate that cannot run on the CI device has to be +run on the device that has the feature, or it proves nothing. + +**The durable lesson: a correctness fix can silently become a performance cliff, +and a selection predicate that can route an entire model onto the correctness tier +should be able to SAY SO.** Reading the source excluded the two candidates a human +would guess and pointed at neither; the predicate's own report named it +immediately. That diagnostic now ships behind the stats flag. + diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index f6f568047..d16579665 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -350,7 +350,7 @@ built on it rather than keeping the flattering one. | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config | | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box | | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs | -| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 24 NATIVE (+8 GDN, BOTH recurrences; oracle-gated, no speed); 63 host-tier. opt-125m e2e token-exact on llvmpipe. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | +| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 24 NATIVE (+8 GDN, BOTH recurrences); 63 host-tier. **27B prefill 21.5x on GB10** (ragged-M reached coopmat). opt-125m e2e token-exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist | diff --git a/docs/STATUS.md b/docs/STATUS.md index e7e0dd6ef..7c15c6315 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -411,7 +411,7 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU, Vulkan (opt-125m exact, GEMV 1.8x; 24 native, +8 GDN incl. BOTH -recurrences, oracle-gated, no speed; qwen3_5 #125 VERIFIED on 27B; CUDA build repaired +recurrences; 27B prefill 21.5x on GB10; qwen3_5 #125 VERIFIED; CUDA build repaired [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in — **blind-written, unverified** — M2 unblocks on verification; gfx1201 hipBLAS + diff --git a/src/vt/vulkan/shaders/vt_matmul_coopmat.comp b/src/vt/vulkan/shaders/vt_matmul_coopmat.comp index 2ff5a7395..e2cfc3b86 100644 --- a/src/vt/vulkan/shaders/vt_matmul_coopmat.comp +++ b/src/vt/vulkan/shaders/vt_matmul_coopmat.comp @@ -85,8 +85,8 @@ const uint VT_CM = 16u; // the reported M = N = K tile extent shared float vt_cm_spill[VT_CM * VT_CM]; void main() { - // Output tile coordinates. THE HOST GUARANTEES m % 16 == 0 AND n % 16 == 0, so - // every tile is whole. + // Output tile coordinates. THE HOST GUARANTEES n % 16 == 0 and m >= 16; a + // ragged M is handled by the tile shift below. // // That guarantee is load-bearing and was learned the hard way: `coopMatLoad` // below reads a FULL 16x16 tile with no masking, so a partial tile reads past @@ -100,6 +100,21 @@ void main() { uint tile_m = (tile / tiles_n) * VT_CM; uint tile_n = (tile % tiles_n) * VT_CM; + // RAGGED M, HANDLED BY SHIFTING THE LAST TILE BACK rather than masking the + // load. A trailing tile would read rows past M -- the fault described above -- + // so it is slid down to start at M-16 and therefore reads only REAL rows. + // + // This is exact, not an approximation. A result row depends solely on that row + // of A and on B, never on which tile computed it, so the rows this tile shares + // with its predecessor are recomputed to bit-identical values. The duplicate + // stores write the same bytes to the same addresses; the values are equal, so + // the interleaving cannot matter. + // + // It requires M >= 16, which the host predicate enforces. Below that there is + // no in-bounds 16-row window at all, and the decode path (M=1) is served by the + // GEMV tactic instead. + if (tile_m + VT_CM > p.m) { tile_m = p.m - VT_CM; } + coopmat acc = coopmat(0.0); diff --git a/src/vt/vulkan/vulkan_ops.cpp b/src/vt/vulkan/vulkan_ops.cpp index cf003202e..66fcce503 100644 --- a/src/vt/vulkan/vulkan_ops.cpp +++ b/src/vt/vulkan/vulkan_ops.cpp @@ -32,6 +32,8 @@ #include #include #include +#include +#include #include #include @@ -42,6 +44,12 @@ namespace vt::vulkan { namespace { +// Gated on the same flag as the dispatch profile; costs nothing when unset. +const bool kCoopMatWhy = [] { + const char* v = std::getenv("VT_VULKAN_DISPATCH_STATS"); + return v != nullptr && std::strcmp(v, "0") != 0; +}(); + // Storage dtype -> the shader-side code (vt_common.glsl VT_DT_*). uint32_t DtypeCode(DType d) { switch (d) { @@ -478,6 +486,41 @@ bool CoopMatMatmulUsable(const Tensor& a, const Tensor& b, int64_t k, int64_t m, if (kDisabled) return false; const VulkanContext& ctx = VulkanContext::Get(); + + // WHY IT DECLINED, reported once per distinct reason under + // VT_VULKAN_DISPATCH_STATS. A 27B prefill measured 99.9% of GPU time in the + // UNTILED SCALAR kernel at ~96 GFLOP/s -- roughly 1% of what this device can do + // -- because this predicate was returning false for every GEMM, and reading the + // source did not reveal which clause. Shapes were whole tiles and activations + // were bf16, so the obvious two candidates were both excluded by inspection and + // the answer still had to be measured. A selection predicate that can silently + // route an entire model onto the correctness tier should be able to say so. + if (kCoopMatWhy) { + const char* why = nullptr; + if (!ctx.coopmat_bf16_f32()) why = "device reports no bf16->f32 16x16x16 SUBGROUP config"; + else if (ctx.subgroup_size() != 32) why = "subgroup size is not 32"; + else if (a.dtype != DType::kBF16) why = "operand a is not bf16"; + else if (b.dtype != DType::kBF16) why = "operand b is not bf16"; + else if (k % 16 != 0) why = "K is not a multiple of 16"; + else if (m < 16) why = "M is below one 16-row tile"; + else if (n % 16 != 0) why = "N is not a multiple of 16"; + if (why != nullptr) { + static std::mutex seen_mu; + static std::set seen; + std::string key = std::string(why) + "|" + std::to_string(static_cast(a.dtype)) + + "," + std::to_string(static_cast(b.dtype)); + std::lock_guard g(seen_mu); + if (seen.insert(key).second) { + std::fprintf(stderr, + "[vt vulkan] coopmat DECLINED: %s (a.dtype=%d b.dtype=%d " + "m=%lld k=%lld n=%lld)\n", + why, static_cast(a.dtype), static_cast(b.dtype), + (long long)m, (long long)k, (long long)n); + std::fflush(stderr); + } + } + } + return ctx.coopmat_bf16_f32() && ctx.subgroup_size() == 32 && a.dtype == DType::kBF16 && b.dtype == DType::kBF16 && k % 16 == 0 && // M AND N MUST ALSO BE WHOLE TILES. `coopMatLoad` reads a FULL 16x16 @@ -493,7 +536,18 @@ bool CoopMatMatmulUsable(const Tensor& a, const Tensor& b, int64_t k, int64_t m, // inside the allocation and its garbage rows were discarded by the // bounds-checked store. Raggedness alone was not enough; the read has to // leave the allocation to fault. - m % 16 == 0 && n % 16 == 0; + // + // M NEED ONLY BE AT LEAST ONE WHOLE TILE, not a multiple of one. The + // shader slides a trailing tile back to start at M-16, so every read + // stays in bounds and the shared rows recompute to identical values. + // + // Requiring m % 16 == 0 here is what fixed the original hang, and it + // MEASURED as the entire prefill bottleneck: prompt length gives + // m = tokens + 1, so 513 % 16 == 1 sent every 27B prefill GEMM to the + // untiled scalar kernel -- 99.9% of GPU time at ~96 GFLOP/s, about 1% of + // this device. N stays whole because a ragged N would need the same + // treatment on the B operand and no shape in play needs it. + m >= 16 && n % 16 == 0; } // GEMV TACTIC SELECTION (VK-F). Same shape of contract as the coopmat predicate diff --git a/src/vt/vulkan/vulkan_spirv.cpp b/src/vt/vulkan/vulkan_spirv.cpp index e69f2cd5d..828c08d20 100644 --- a/src/vt/vulkan/vulkan_spirv.cpp +++ b/src/vt/vulkan/vulkan_spirv.cpp @@ -6542,39 +6542,39 @@ constexpr uint32_t kSpv_vt_matmul[] = { }; constexpr uint32_t kSpv_vt_matmul_coopmat[] = { - 0x07230203u, 0x00010300u, 0x0008000bu, 0x000001fdu, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, + 0x07230203u, 0x00010300u, 0x0008000bu, 0x00000207u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, 0x00000009u, 0x00020011u, 0x00001151u, 0x00020011u, 0x000013fcu, 0x00020011u, 0x000013feu, 0x00020011u, 0x000014e1u, 0x00020011u, 0x00001786u, 0x0006000au, 0x5f565053u, 0x5f52484bu, 0x6f6c6662u, 0x36317461u, 0x00000000u, 0x0008000au, 0x5f565053u, 0x5f52484bu, 0x706f6f63u, 0x74617265u, 0x5f657669u, 0x7274616du, 0x00007869u, 0x0008000au, 0x5f565053u, 0x5f52484bu, 0x6b6c7576u, 0x6d5f6e61u, 0x726f6d65u, 0x6f6d5f79u, 0x006c6564u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, 0x00000000u, 0x00000003u, 0x0007000fu, 0x00000005u, 0x00000004u, 0x6e69616du, 0x00000000u, 0x000000f1u, - 0x0000018bu, 0x00060010u, 0x00000004u, 0x00000011u, 0x00000020u, 0x00000001u, 0x00000001u, 0x00030047u, + 0x00000195u, 0x00060010u, 0x00000004u, 0x00000011u, 0x00000020u, 0x00000001u, 0x00000001u, 0x00030047u, 0x000000e3u, 0x00000002u, 0x00050048u, 0x000000e3u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00050048u, 0x000000e3u, 0x00000001u, 0x00000023u, 0x00000004u, 0x00050048u, 0x000000e3u, 0x00000002u, 0x00000023u, 0x00000008u, 0x00050048u, 0x000000e3u, 0x00000003u, 0x00000023u, 0x0000000cu, 0x00050048u, 0x000000e3u, 0x00000004u, 0x00000023u, 0x00000010u, 0x00050048u, 0x000000e3u, 0x00000005u, 0x00000023u, 0x00000014u, - 0x00040047u, 0x000000f1u, 0x0000000bu, 0x0000001au, 0x00040047u, 0x00000121u, 0x00000006u, 0x00000002u, - 0x00030047u, 0x00000122u, 0x00000002u, 0x00040048u, 0x00000122u, 0x00000000u, 0x00000018u, 0x00050048u, - 0x00000122u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x00000124u, 0x00000018u, 0x00040047u, - 0x00000124u, 0x00000021u, 0x00000001u, 0x00040047u, 0x00000124u, 0x00000022u, 0x00000000u, 0x00040047u, - 0x0000013fu, 0x00000001u, 0x00000000u, 0x00040047u, 0x00000146u, 0x00000006u, 0x00000002u, 0x00030047u, - 0x00000147u, 0x00000002u, 0x00040048u, 0x00000147u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000147u, - 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x00000149u, 0x00000018u, 0x00040047u, 0x00000149u, - 0x00000021u, 0x00000003u, 0x00040047u, 0x00000149u, 0x00000022u, 0x00000000u, 0x00040047u, 0x0000018bu, - 0x0000000bu, 0x0000001bu, 0x00040047u, 0x000001b2u, 0x00000001u, 0x00000001u, 0x00040047u, 0x000001b6u, - 0x00000006u, 0x00000004u, 0x00030047u, 0x000001b7u, 0x00000002u, 0x00050048u, 0x000001b7u, 0x00000000u, - 0x00000023u, 0x00000000u, 0x00040047u, 0x000001b9u, 0x00000021u, 0x00000004u, 0x00040047u, 0x000001b9u, - 0x00000022u, 0x00000000u, 0x00040047u, 0x000001d1u, 0x00000006u, 0x00000002u, 0x00030047u, 0x000001d2u, - 0x00000002u, 0x00050048u, 0x000001d2u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x000001d4u, - 0x00000021u, 0x00000005u, 0x00040047u, 0x000001d4u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001f4u, - 0x0000000bu, 0x00000019u, 0x00040047u, 0x000001f5u, 0x00000006u, 0x00000004u, 0x00030047u, 0x000001f6u, - 0x00000002u, 0x00040048u, 0x000001f6u, 0x00000000u, 0x00000018u, 0x00050048u, 0x000001f6u, 0x00000000u, - 0x00000023u, 0x00000000u, 0x00030047u, 0x000001f8u, 0x00000018u, 0x00040047u, 0x000001f8u, 0x00000021u, - 0x00000000u, 0x00040047u, 0x000001f8u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001f9u, 0x00000006u, - 0x00000004u, 0x00030047u, 0x000001fau, 0x00000002u, 0x00040048u, 0x000001fau, 0x00000000u, 0x00000018u, - 0x00050048u, 0x000001fau, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000001fcu, 0x00000018u, - 0x00040047u, 0x000001fcu, 0x00000021u, 0x00000002u, 0x00040047u, 0x000001fcu, 0x00000022u, 0x00000000u, + 0x00040047u, 0x000000f1u, 0x0000000bu, 0x0000001au, 0x00040047u, 0x0000012bu, 0x00000006u, 0x00000002u, + 0x00030047u, 0x0000012cu, 0x00000002u, 0x00040048u, 0x0000012cu, 0x00000000u, 0x00000018u, 0x00050048u, + 0x0000012cu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x0000012eu, 0x00000018u, 0x00040047u, + 0x0000012eu, 0x00000021u, 0x00000001u, 0x00040047u, 0x0000012eu, 0x00000022u, 0x00000000u, 0x00040047u, + 0x00000149u, 0x00000001u, 0x00000000u, 0x00040047u, 0x00000150u, 0x00000006u, 0x00000002u, 0x00030047u, + 0x00000151u, 0x00000002u, 0x00040048u, 0x00000151u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000151u, + 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x00000153u, 0x00000018u, 0x00040047u, 0x00000153u, + 0x00000021u, 0x00000003u, 0x00040047u, 0x00000153u, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000195u, + 0x0000000bu, 0x0000001bu, 0x00040047u, 0x000001bcu, 0x00000001u, 0x00000001u, 0x00040047u, 0x000001c0u, + 0x00000006u, 0x00000004u, 0x00030047u, 0x000001c1u, 0x00000002u, 0x00050048u, 0x000001c1u, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00040047u, 0x000001c3u, 0x00000021u, 0x00000004u, 0x00040047u, 0x000001c3u, + 0x00000022u, 0x00000000u, 0x00040047u, 0x000001dbu, 0x00000006u, 0x00000002u, 0x00030047u, 0x000001dcu, + 0x00000002u, 0x00050048u, 0x000001dcu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x000001deu, + 0x00000021u, 0x00000005u, 0x00040047u, 0x000001deu, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001feu, + 0x0000000bu, 0x00000019u, 0x00040047u, 0x000001ffu, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000200u, + 0x00000002u, 0x00040048u, 0x00000200u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000200u, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00030047u, 0x00000202u, 0x00000018u, 0x00040047u, 0x00000202u, 0x00000021u, + 0x00000000u, 0x00040047u, 0x00000202u, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000203u, 0x00000006u, + 0x00000004u, 0x00030047u, 0x00000204u, 0x00000002u, 0x00040048u, 0x00000204u, 0x00000000u, 0x00000018u, + 0x00050048u, 0x00000204u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x00000206u, 0x00000018u, + 0x00040047u, 0x00000206u, 0x00000021u, 0x00000002u, 0x00040047u, 0x00000206u, 0x00000022u, 0x00000000u, 0x00020013u, 0x00000002u, 0x00030021u, 0x00000003u, 0x00000002u, 0x00030016u, 0x00000006u, 0x00000020u, 0x00040020u, 0x00000007u, 0x00000007u, 0x00000006u, 0x00040015u, 0x00000008u, 0x00000020u, 0x00000000u, 0x00040021u, 0x00000009u, 0x00000008u, 0x00000007u, 0x00040020u, 0x00000010u, 0x00000007u, 0x00000008u, @@ -6597,45 +6597,45 @@ constexpr uint32_t kSpv_vt_matmul_coopmat[] = { 0x00000029u, 0x000000e6u, 0x00000001u, 0x00040020u, 0x000000e7u, 0x00000009u, 0x00000008u, 0x0004002bu, 0x00000008u, 0x000000eau, 0x00000010u, 0x00040017u, 0x000000efu, 0x00000008u, 0x00000003u, 0x00040020u, 0x000000f0u, 0x00000001u, 0x000000efu, 0x0004003bu, 0x000000f0u, 0x000000f1u, 0x00000001u, 0x00040020u, - 0x000000f2u, 0x00000001u, 0x00000008u, 0x0004002bu, 0x00000008u, 0x0000010bu, 0x00000003u, 0x0004002bu, - 0x00000008u, 0x0000010cu, 0x00000002u, 0x00071168u, 0x0000010du, 0x00000006u, 0x0000010bu, 0x000000eau, - 0x000000eau, 0x0000010cu, 0x00040020u, 0x0000010eu, 0x00000007u, 0x0000010du, 0x0004002bu, 0x00000006u, - 0x00000110u, 0x00000000u, 0x0004002cu, 0x0000010du, 0x00000111u, 0x00000110u, 0x0004002bu, 0x00000029u, - 0x00000119u, 0x00000002u, 0x00040016u, 0x0000011du, 0x00000010u, 0x00000000u, 0x00071168u, 0x0000011eu, - 0x0000011du, 0x0000010bu, 0x000000eau, 0x000000eau, 0x00000023u, 0x00040020u, 0x0000011fu, 0x00000007u, - 0x0000011eu, 0x0003001du, 0x00000121u, 0x0000011du, 0x0003001eu, 0x00000122u, 0x00000121u, 0x00040020u, - 0x00000123u, 0x0000000cu, 0x00000122u, 0x0004003bu, 0x00000123u, 0x00000124u, 0x0000000cu, 0x0004002bu, - 0x00000029u, 0x00000125u, 0x00000003u, 0x00040020u, 0x00000130u, 0x0000000cu, 0x0000011du, 0x00040032u, - 0x00000008u, 0x0000013fu, 0x00000000u, 0x00060034u, 0x00000019u, 0x00000140u, 0x000000aau, 0x0000013fu, - 0x00000035u, 0x00071168u, 0x00000143u, 0x0000011du, 0x0000010bu, 0x000000eau, 0x000000eau, 0x00000035u, - 0x00040020u, 0x00000144u, 0x00000007u, 0x00000143u, 0x0003001du, 0x00000146u, 0x0000011du, 0x0003001eu, - 0x00000147u, 0x00000146u, 0x00040020u, 0x00000148u, 0x0000000cu, 0x00000147u, 0x0004003bu, 0x00000148u, - 0x00000149u, 0x0000000cu, 0x0004002bu, 0x00000029u, 0x0000014au, 0x00000004u, 0x0004002bu, 0x00000008u, - 0x00000183u, 0x00000100u, 0x0004001cu, 0x00000184u, 0x00000006u, 0x00000183u, 0x00040020u, 0x00000185u, - 0x00000004u, 0x00000184u, 0x0004003bu, 0x00000185u, 0x00000186u, 0x00000004u, 0x00040020u, 0x00000187u, - 0x00000004u, 0x00000006u, 0x0004002bu, 0x00000008u, 0x00000189u, 0x00000108u, 0x0004003bu, 0x000000f0u, - 0x0000018bu, 0x00000001u, 0x00040032u, 0x00000008u, 0x000001b2u, 0x00000000u, 0x00060034u, 0x00000019u, - 0x000001b3u, 0x000000aau, 0x000001b2u, 0x00000023u, 0x0003001du, 0x000001b6u, 0x00000008u, 0x0003001eu, - 0x000001b7u, 0x000001b6u, 0x00040020u, 0x000001b8u, 0x0000000cu, 0x000001b7u, 0x0004003bu, 0x000001b8u, - 0x000001b9u, 0x0000000cu, 0x0004002bu, 0x00000029u, 0x000001bau, 0x00000005u, 0x00040020u, 0x000001cdu, - 0x0000000cu, 0x00000008u, 0x00040015u, 0x000001d0u, 0x00000010u, 0x00000000u, 0x0003001du, 0x000001d1u, - 0x000001d0u, 0x0003001eu, 0x000001d2u, 0x000001d1u, 0x00040020u, 0x000001d3u, 0x0000000cu, 0x000001d2u, - 0x0004003bu, 0x000001d3u, 0x000001d4u, 0x0000000cu, 0x00040020u, 0x000001eau, 0x0000000cu, 0x000001d0u, - 0x0003002au, 0x00000019u, 0x000001ecu, 0x0004002bu, 0x00000008u, 0x000001edu, 0x00000020u, 0x0004002bu, - 0x00000008u, 0x000001f0u, 0x00000080u, 0x0004001cu, 0x000001f1u, 0x00000006u, 0x000001f0u, 0x00040020u, - 0x000001f2u, 0x00000004u, 0x000001f1u, 0x0004003bu, 0x000001f2u, 0x000001f3u, 0x00000004u, 0x0006002cu, - 0x000000efu, 0x000001f4u, 0x000001edu, 0x00000035u, 0x00000035u, 0x0003001du, 0x000001f5u, 0x00000008u, - 0x0003001eu, 0x000001f6u, 0x000001f5u, 0x00040020u, 0x000001f7u, 0x0000000cu, 0x000001f6u, 0x0004003bu, - 0x000001f7u, 0x000001f8u, 0x0000000cu, 0x0003001du, 0x000001f9u, 0x00000008u, 0x0003001eu, 0x000001fau, - 0x000001f9u, 0x00040020u, 0x000001fbu, 0x0000000cu, 0x000001fau, 0x0004003bu, 0x000001fbu, 0x000001fcu, + 0x000000f2u, 0x00000001u, 0x00000008u, 0x0004002bu, 0x00000008u, 0x00000115u, 0x00000003u, 0x0004002bu, + 0x00000008u, 0x00000116u, 0x00000002u, 0x00071168u, 0x00000117u, 0x00000006u, 0x00000115u, 0x000000eau, + 0x000000eau, 0x00000116u, 0x00040020u, 0x00000118u, 0x00000007u, 0x00000117u, 0x0004002bu, 0x00000006u, + 0x0000011au, 0x00000000u, 0x0004002cu, 0x00000117u, 0x0000011bu, 0x0000011au, 0x0004002bu, 0x00000029u, + 0x00000123u, 0x00000002u, 0x00040016u, 0x00000127u, 0x00000010u, 0x00000000u, 0x00071168u, 0x00000128u, + 0x00000127u, 0x00000115u, 0x000000eau, 0x000000eau, 0x00000023u, 0x00040020u, 0x00000129u, 0x00000007u, + 0x00000128u, 0x0003001du, 0x0000012bu, 0x00000127u, 0x0003001eu, 0x0000012cu, 0x0000012bu, 0x00040020u, + 0x0000012du, 0x0000000cu, 0x0000012cu, 0x0004003bu, 0x0000012du, 0x0000012eu, 0x0000000cu, 0x0004002bu, + 0x00000029u, 0x0000012fu, 0x00000003u, 0x00040020u, 0x0000013au, 0x0000000cu, 0x00000127u, 0x00040032u, + 0x00000008u, 0x00000149u, 0x00000000u, 0x00060034u, 0x00000019u, 0x0000014au, 0x000000aau, 0x00000149u, + 0x00000035u, 0x00071168u, 0x0000014du, 0x00000127u, 0x00000115u, 0x000000eau, 0x000000eau, 0x00000035u, + 0x00040020u, 0x0000014eu, 0x00000007u, 0x0000014du, 0x0003001du, 0x00000150u, 0x00000127u, 0x0003001eu, + 0x00000151u, 0x00000150u, 0x00040020u, 0x00000152u, 0x0000000cu, 0x00000151u, 0x0004003bu, 0x00000152u, + 0x00000153u, 0x0000000cu, 0x0004002bu, 0x00000029u, 0x00000154u, 0x00000004u, 0x0004002bu, 0x00000008u, + 0x0000018du, 0x00000100u, 0x0004001cu, 0x0000018eu, 0x00000006u, 0x0000018du, 0x00040020u, 0x0000018fu, + 0x00000004u, 0x0000018eu, 0x0004003bu, 0x0000018fu, 0x00000190u, 0x00000004u, 0x00040020u, 0x00000191u, + 0x00000004u, 0x00000006u, 0x0004002bu, 0x00000008u, 0x00000193u, 0x00000108u, 0x0004003bu, 0x000000f0u, + 0x00000195u, 0x00000001u, 0x00040032u, 0x00000008u, 0x000001bcu, 0x00000000u, 0x00060034u, 0x00000019u, + 0x000001bdu, 0x000000aau, 0x000001bcu, 0x00000023u, 0x0003001du, 0x000001c0u, 0x00000008u, 0x0003001eu, + 0x000001c1u, 0x000001c0u, 0x00040020u, 0x000001c2u, 0x0000000cu, 0x000001c1u, 0x0004003bu, 0x000001c2u, + 0x000001c3u, 0x0000000cu, 0x0004002bu, 0x00000029u, 0x000001c4u, 0x00000005u, 0x00040020u, 0x000001d7u, + 0x0000000cu, 0x00000008u, 0x00040015u, 0x000001dau, 0x00000010u, 0x00000000u, 0x0003001du, 0x000001dbu, + 0x000001dau, 0x0003001eu, 0x000001dcu, 0x000001dbu, 0x00040020u, 0x000001ddu, 0x0000000cu, 0x000001dcu, + 0x0004003bu, 0x000001ddu, 0x000001deu, 0x0000000cu, 0x00040020u, 0x000001f4u, 0x0000000cu, 0x000001dau, + 0x0003002au, 0x00000019u, 0x000001f6u, 0x0004002bu, 0x00000008u, 0x000001f7u, 0x00000020u, 0x0004002bu, + 0x00000008u, 0x000001fau, 0x00000080u, 0x0004001cu, 0x000001fbu, 0x00000006u, 0x000001fau, 0x00040020u, + 0x000001fcu, 0x00000004u, 0x000001fbu, 0x0004003bu, 0x000001fcu, 0x000001fdu, 0x00000004u, 0x0006002cu, + 0x000000efu, 0x000001feu, 0x000001f7u, 0x00000035u, 0x00000035u, 0x0003001du, 0x000001ffu, 0x00000008u, + 0x0003001eu, 0x00000200u, 0x000001ffu, 0x00040020u, 0x00000201u, 0x0000000cu, 0x00000200u, 0x0004003bu, + 0x00000201u, 0x00000202u, 0x0000000cu, 0x0003001du, 0x00000203u, 0x00000008u, 0x0003001eu, 0x00000204u, + 0x00000203u, 0x00040020u, 0x00000205u, 0x0000000cu, 0x00000204u, 0x0004003bu, 0x00000205u, 0x00000206u, 0x0000000cu, 0x00050036u, 0x00000002u, 0x00000004u, 0x00000000u, 0x00000003u, 0x000200f8u, 0x00000005u, 0x0004003bu, 0x00000010u, 0x000000e2u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x000000eeu, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000101u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000106u, 0x00000007u, - 0x0004003bu, 0x0000010eu, 0x0000010fu, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000112u, 0x00000007u, - 0x0004003bu, 0x0000011fu, 0x00000120u, 0x00000007u, 0x0004003bu, 0x00000144u, 0x00000145u, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x0000018au, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000195u, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x00000198u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001e4u, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x000001e7u, 0x00000007u, 0x00050041u, 0x000000e7u, 0x000000e8u, 0x000000e5u, + 0x0004003bu, 0x00000118u, 0x00000119u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x0000011cu, 0x00000007u, + 0x0004003bu, 0x00000129u, 0x0000012au, 0x00000007u, 0x0004003bu, 0x0000014eu, 0x0000014fu, 0x00000007u, + 0x0004003bu, 0x00000010u, 0x00000194u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x0000019fu, 0x00000007u, + 0x0004003bu, 0x00000010u, 0x000001a2u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001eeu, 0x00000007u, + 0x0004003bu, 0x00000010u, 0x000001f1u, 0x00000007u, 0x00050041u, 0x000000e7u, 0x000000e8u, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x000000e9u, 0x000000e8u, 0x00050080u, 0x00000008u, 0x000000ebu, 0x000000e9u, 0x000000eau, 0x00050082u, 0x00000008u, 0x000000ecu, 0x000000ebu, 0x00000035u, 0x00050086u, 0x00000008u, 0x000000edu, 0x000000ecu, 0x000000eau, 0x0003003eu, 0x000000e2u, 0x000000edu, 0x00050041u, @@ -6652,231 +6652,238 @@ constexpr uint32_t kSpv_vt_matmul_coopmat[] = { 0x00000105u, 0x00000104u, 0x000000eau, 0x0003003eu, 0x00000101u, 0x00000105u, 0x0004003du, 0x00000008u, 0x00000107u, 0x000000eeu, 0x0004003du, 0x00000008u, 0x00000108u, 0x000000e2u, 0x00050089u, 0x00000008u, 0x00000109u, 0x00000107u, 0x00000108u, 0x00050084u, 0x00000008u, 0x0000010au, 0x00000109u, 0x000000eau, - 0x0003003eu, 0x00000106u, 0x0000010au, 0x0003003eu, 0x0000010fu, 0x00000111u, 0x0003003eu, 0x00000112u, - 0x00000023u, 0x000200f9u, 0x00000113u, 0x000200f8u, 0x00000113u, 0x000400f6u, 0x00000115u, 0x00000116u, - 0x00000000u, 0x000200f9u, 0x00000117u, 0x000200f8u, 0x00000117u, 0x0004003du, 0x00000008u, 0x00000118u, - 0x00000112u, 0x00050041u, 0x000000e7u, 0x0000011au, 0x000000e5u, 0x00000119u, 0x0004003du, 0x00000008u, - 0x0000011bu, 0x0000011au, 0x000500b0u, 0x00000019u, 0x0000011cu, 0x00000118u, 0x0000011bu, 0x000400fau, - 0x0000011cu, 0x00000114u, 0x00000115u, 0x000200f8u, 0x00000114u, 0x00050041u, 0x000000e7u, 0x00000126u, - 0x000000e5u, 0x00000125u, 0x0004003du, 0x00000008u, 0x00000127u, 0x00000126u, 0x000500c2u, 0x00000008u, - 0x00000128u, 0x00000127u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x00000129u, 0x00000101u, 0x00050041u, - 0x000000e7u, 0x0000012au, 0x000000e5u, 0x00000119u, 0x0004003du, 0x00000008u, 0x0000012bu, 0x0000012au, - 0x00050084u, 0x00000008u, 0x0000012cu, 0x00000129u, 0x0000012bu, 0x00050080u, 0x00000008u, 0x0000012du, - 0x00000128u, 0x0000012cu, 0x0004003du, 0x00000008u, 0x0000012eu, 0x00000112u, 0x00050080u, 0x00000008u, - 0x0000012fu, 0x0000012du, 0x0000012eu, 0x00060041u, 0x00000130u, 0x00000131u, 0x00000124u, 0x00000076u, - 0x0000012fu, 0x00050041u, 0x000000e7u, 0x00000132u, 0x000000e5u, 0x00000125u, 0x0004003du, 0x00000008u, - 0x00000133u, 0x00000132u, 0x000500c2u, 0x00000008u, 0x00000134u, 0x00000133u, 0x000000e6u, 0x0004003du, - 0x00000008u, 0x00000135u, 0x00000101u, 0x00050041u, 0x000000e7u, 0x00000136u, 0x000000e5u, 0x00000119u, - 0x0004003du, 0x00000008u, 0x00000137u, 0x00000136u, 0x00050084u, 0x00000008u, 0x00000138u, 0x00000135u, - 0x00000137u, 0x00050080u, 0x00000008u, 0x00000139u, 0x00000134u, 0x00000138u, 0x0004003du, 0x00000008u, - 0x0000013au, 0x00000112u, 0x00050080u, 0x00000008u, 0x0000013bu, 0x00000139u, 0x0000013au, 0x00050041u, - 0x000000e7u, 0x0000013cu, 0x000000e5u, 0x00000119u, 0x0004003du, 0x00000008u, 0x0000013du, 0x0000013cu, - 0x00071169u, 0x0000011eu, 0x0000013eu, 0x00000131u, 0x00000076u, 0x0000013du, 0x00000000u, 0x0003003eu, - 0x00000120u, 0x0000013eu, 0x000300f7u, 0x00000142u, 0x00000000u, 0x000400fau, 0x00000140u, 0x00000141u, - 0x00000163u, 0x000200f8u, 0x00000141u, 0x00050041u, 0x000000e7u, 0x0000014bu, 0x000000e5u, 0x0000014au, - 0x0004003du, 0x00000008u, 0x0000014cu, 0x0000014bu, 0x000500c2u, 0x00000008u, 0x0000014du, 0x0000014cu, - 0x000000e6u, 0x0004003du, 0x00000008u, 0x0000014eu, 0x00000106u, 0x00050041u, 0x000000e7u, 0x0000014fu, - 0x000000e5u, 0x00000119u, 0x0004003du, 0x00000008u, 0x00000150u, 0x0000014fu, 0x00050084u, 0x00000008u, - 0x00000151u, 0x0000014eu, 0x00000150u, 0x00050080u, 0x00000008u, 0x00000152u, 0x0000014du, 0x00000151u, - 0x0004003du, 0x00000008u, 0x00000153u, 0x00000112u, 0x00050080u, 0x00000008u, 0x00000154u, 0x00000152u, - 0x00000153u, 0x00060041u, 0x00000130u, 0x00000155u, 0x00000149u, 0x00000076u, 0x00000154u, 0x00050041u, - 0x000000e7u, 0x00000156u, 0x000000e5u, 0x0000014au, 0x0004003du, 0x00000008u, 0x00000157u, 0x00000156u, - 0x000500c2u, 0x00000008u, 0x00000158u, 0x00000157u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x00000159u, - 0x00000106u, 0x00050041u, 0x000000e7u, 0x0000015au, 0x000000e5u, 0x00000119u, 0x0004003du, 0x00000008u, - 0x0000015bu, 0x0000015au, 0x00050084u, 0x00000008u, 0x0000015cu, 0x00000159u, 0x0000015bu, 0x00050080u, - 0x00000008u, 0x0000015du, 0x00000158u, 0x0000015cu, 0x0004003du, 0x00000008u, 0x0000015eu, 0x00000112u, - 0x00050080u, 0x00000008u, 0x0000015fu, 0x0000015du, 0x0000015eu, 0x00050041u, 0x000000e7u, 0x00000160u, - 0x000000e5u, 0x00000119u, 0x0004003du, 0x00000008u, 0x00000161u, 0x00000160u, 0x00071169u, 0x00000143u, - 0x00000162u, 0x00000155u, 0x000000e6u, 0x00000161u, 0x00000000u, 0x0003003eu, 0x00000145u, 0x00000162u, - 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000163u, 0x00050041u, 0x000000e7u, 0x00000164u, 0x000000e5u, - 0x0000014au, 0x0004003du, 0x00000008u, 0x00000165u, 0x00000164u, 0x000500c2u, 0x00000008u, 0x00000166u, - 0x00000165u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x00000167u, 0x00000112u, 0x00050041u, 0x000000e7u, - 0x00000168u, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x00000169u, 0x00000168u, 0x00050084u, - 0x00000008u, 0x0000016au, 0x00000167u, 0x00000169u, 0x00050080u, 0x00000008u, 0x0000016bu, 0x00000166u, - 0x0000016au, 0x0004003du, 0x00000008u, 0x0000016cu, 0x00000106u, 0x00050080u, 0x00000008u, 0x0000016du, - 0x0000016bu, 0x0000016cu, 0x00060041u, 0x00000130u, 0x0000016eu, 0x00000149u, 0x00000076u, 0x0000016du, - 0x00050041u, 0x000000e7u, 0x0000016fu, 0x000000e5u, 0x0000014au, 0x0004003du, 0x00000008u, 0x00000170u, - 0x0000016fu, 0x000500c2u, 0x00000008u, 0x00000171u, 0x00000170u, 0x000000e6u, 0x0004003du, 0x00000008u, - 0x00000172u, 0x00000112u, 0x00050041u, 0x000000e7u, 0x00000173u, 0x000000e5u, 0x000000e6u, 0x0004003du, - 0x00000008u, 0x00000174u, 0x00000173u, 0x00050084u, 0x00000008u, 0x00000175u, 0x00000172u, 0x00000174u, - 0x00050080u, 0x00000008u, 0x00000176u, 0x00000171u, 0x00000175u, 0x0004003du, 0x00000008u, 0x00000177u, - 0x00000106u, 0x00050080u, 0x00000008u, 0x00000178u, 0x00000176u, 0x00000177u, 0x00050041u, 0x000000e7u, - 0x00000179u, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x0000017au, 0x00000179u, 0x00071169u, - 0x00000143u, 0x0000017bu, 0x0000016eu, 0x00000076u, 0x0000017au, 0x00000000u, 0x0003003eu, 0x00000145u, - 0x0000017bu, 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000142u, 0x0004003du, 0x0000011eu, 0x0000017cu, - 0x00000120u, 0x0004003du, 0x00000143u, 0x0000017du, 0x00000145u, 0x0004003du, 0x0000010du, 0x0000017eu, - 0x0000010fu, 0x0006116bu, 0x0000010du, 0x0000017fu, 0x0000017cu, 0x0000017du, 0x0000017eu, 0x0003003eu, - 0x0000010fu, 0x0000017fu, 0x000200f9u, 0x00000116u, 0x000200f8u, 0x00000116u, 0x0004003du, 0x00000008u, - 0x00000180u, 0x00000112u, 0x00050080u, 0x00000008u, 0x00000181u, 0x00000180u, 0x000000eau, 0x0003003eu, - 0x00000112u, 0x00000181u, 0x000200f9u, 0x00000113u, 0x000200f8u, 0x00000115u, 0x0004003du, 0x0000010du, - 0x00000182u, 0x0000010fu, 0x00050041u, 0x00000187u, 0x00000188u, 0x00000186u, 0x00000023u, 0x0007116au, - 0x00000188u, 0x00000182u, 0x00000076u, 0x000000eau, 0x00000028u, 0x0000010cu, 0x000400e0u, 0x0000010cu, - 0x0000010cu, 0x00000189u, 0x00050041u, 0x000000f2u, 0x0000018cu, 0x0000018bu, 0x00000023u, 0x0004003du, - 0x00000008u, 0x0000018du, 0x0000018cu, 0x0003003eu, 0x0000018au, 0x0000018du, 0x000200f9u, 0x0000018eu, - 0x000200f8u, 0x0000018eu, 0x000400f6u, 0x00000190u, 0x00000191u, 0x00000000u, 0x000200f9u, 0x00000192u, - 0x000200f8u, 0x00000192u, 0x0004003du, 0x00000008u, 0x00000193u, 0x0000018au, 0x000500b0u, 0x00000019u, - 0x00000194u, 0x00000193u, 0x00000183u, 0x000400fau, 0x00000194u, 0x0000018fu, 0x00000190u, 0x000200f8u, - 0x0000018fu, 0x0004003du, 0x00000008u, 0x00000196u, 0x0000018au, 0x00050086u, 0x00000008u, 0x00000197u, - 0x00000196u, 0x000000eau, 0x0003003eu, 0x00000195u, 0x00000197u, 0x0004003du, 0x00000008u, 0x00000199u, - 0x0000018au, 0x00050089u, 0x00000008u, 0x0000019au, 0x00000199u, 0x000000eau, 0x0003003eu, 0x00000198u, - 0x0000019au, 0x0004003du, 0x00000008u, 0x0000019bu, 0x00000101u, 0x0004003du, 0x00000008u, 0x0000019cu, - 0x00000195u, 0x00050080u, 0x00000008u, 0x0000019du, 0x0000019bu, 0x0000019cu, 0x00050041u, 0x000000e7u, - 0x0000019eu, 0x000000e5u, 0x00000076u, 0x0004003du, 0x00000008u, 0x0000019fu, 0x0000019eu, 0x000500aeu, - 0x00000019u, 0x000001a0u, 0x0000019du, 0x0000019fu, 0x000400a8u, 0x00000019u, 0x000001a1u, 0x000001a0u, - 0x000300f7u, 0x000001a3u, 0x00000000u, 0x000400fau, 0x000001a1u, 0x000001a2u, 0x000001a3u, 0x000200f8u, - 0x000001a2u, 0x0004003du, 0x00000008u, 0x000001a4u, 0x00000106u, 0x0004003du, 0x00000008u, 0x000001a5u, - 0x00000198u, 0x00050080u, 0x00000008u, 0x000001a6u, 0x000001a4u, 0x000001a5u, 0x00050041u, 0x000000e7u, - 0x000001a7u, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x000001a8u, 0x000001a7u, 0x000500aeu, - 0x00000019u, 0x000001a9u, 0x000001a6u, 0x000001a8u, 0x000200f9u, 0x000001a3u, 0x000200f8u, 0x000001a3u, - 0x000700f5u, 0x00000019u, 0x000001aau, 0x000001a0u, 0x0000018fu, 0x000001a9u, 0x000001a2u, 0x000300f7u, - 0x000001acu, 0x00000000u, 0x000400fau, 0x000001aau, 0x000001abu, 0x000001acu, 0x000200f8u, 0x000001abu, - 0x000200f9u, 0x00000191u, 0x000200f8u, 0x000001acu, 0x000200f9u, 0x000001aeu, 0x000200f8u, 0x000001aeu, - 0x000400f6u, 0x000001b0u, 0x000001b1u, 0x00000000u, 0x000200f9u, 0x000001afu, 0x000200f8u, 0x000001afu, - 0x000300f7u, 0x000001b5u, 0x00000000u, 0x000400fau, 0x000001b3u, 0x000001b4u, 0x000001cfu, 0x000200f8u, - 0x000001b4u, 0x00050041u, 0x000000e7u, 0x000001bbu, 0x000000e5u, 0x000001bau, 0x0004003du, 0x00000008u, - 0x000001bcu, 0x000001bbu, 0x000500c2u, 0x00000008u, 0x000001bdu, 0x000001bcu, 0x00000119u, 0x0004003du, - 0x00000008u, 0x000001beu, 0x00000101u, 0x0004003du, 0x00000008u, 0x000001bfu, 0x00000195u, 0x00050080u, - 0x00000008u, 0x000001c0u, 0x000001beu, 0x000001bfu, 0x00050041u, 0x000000e7u, 0x000001c1u, 0x000000e5u, - 0x000000e6u, 0x0004003du, 0x00000008u, 0x000001c2u, 0x000001c1u, 0x00050084u, 0x00000008u, 0x000001c3u, - 0x000001c0u, 0x000001c2u, 0x0004003du, 0x00000008u, 0x000001c4u, 0x00000106u, 0x0004003du, 0x00000008u, - 0x000001c5u, 0x00000198u, 0x00050080u, 0x00000008u, 0x000001c6u, 0x000001c4u, 0x000001c5u, 0x00050080u, - 0x00000008u, 0x000001c7u, 0x000001c3u, 0x000001c6u, 0x00050080u, 0x00000008u, 0x000001c8u, 0x000001bdu, - 0x000001c7u, 0x0004003du, 0x00000008u, 0x000001c9u, 0x0000018au, 0x00050041u, 0x00000187u, 0x000001cau, - 0x00000186u, 0x000001c9u, 0x0006003du, 0x00000006u, 0x000001cbu, 0x000001cau, 0x00000030u, 0x0000010cu, - 0x0004007cu, 0x00000008u, 0x000001ccu, 0x000001cbu, 0x00060041u, 0x000001cdu, 0x000001ceu, 0x000001b9u, - 0x00000076u, 0x000001c8u, 0x0003003eu, 0x000001ceu, 0x000001ccu, 0x000200f9u, 0x000001b5u, 0x000200f8u, - 0x000001cfu, 0x00050041u, 0x000000e7u, 0x000001d5u, 0x000000e5u, 0x000001bau, 0x0004003du, 0x00000008u, - 0x000001d6u, 0x000001d5u, 0x000500c2u, 0x00000008u, 0x000001d7u, 0x000001d6u, 0x000000e6u, 0x0004003du, - 0x00000008u, 0x000001d8u, 0x00000101u, 0x0004003du, 0x00000008u, 0x000001d9u, 0x00000195u, 0x00050080u, - 0x00000008u, 0x000001dau, 0x000001d8u, 0x000001d9u, 0x00050041u, 0x000000e7u, 0x000001dbu, 0x000000e5u, - 0x000000e6u, 0x0004003du, 0x00000008u, 0x000001dcu, 0x000001dbu, 0x00050084u, 0x00000008u, 0x000001ddu, - 0x000001dau, 0x000001dcu, 0x0004003du, 0x00000008u, 0x000001deu, 0x00000106u, 0x0004003du, 0x00000008u, - 0x000001dfu, 0x00000198u, 0x00050080u, 0x00000008u, 0x000001e0u, 0x000001deu, 0x000001dfu, 0x00050080u, - 0x00000008u, 0x000001e1u, 0x000001ddu, 0x000001e0u, 0x00050080u, 0x00000008u, 0x000001e2u, 0x000001d7u, - 0x000001e1u, 0x0004003du, 0x00000008u, 0x000001e3u, 0x0000018au, 0x00050041u, 0x00000187u, 0x000001e5u, - 0x00000186u, 0x000001e3u, 0x0006003du, 0x00000006u, 0x000001e6u, 0x000001e5u, 0x00000030u, 0x0000010cu, - 0x0003003eu, 0x000001e4u, 0x000001e6u, 0x0003003eu, 0x000001e7u, 0x000001b2u, 0x00060039u, 0x00000008u, - 0x000001e8u, 0x00000014u, 0x000001e4u, 0x000001e7u, 0x00040071u, 0x000001d0u, 0x000001e9u, 0x000001e8u, - 0x00060041u, 0x000001eau, 0x000001ebu, 0x000001d4u, 0x00000076u, 0x000001e2u, 0x0003003eu, 0x000001ebu, - 0x000001e9u, 0x000200f9u, 0x000001b5u, 0x000200f8u, 0x000001b5u, 0x000200f9u, 0x000001b1u, 0x000200f8u, - 0x000001b1u, 0x000400fau, 0x000001ecu, 0x000001aeu, 0x000001b0u, 0x000200f8u, 0x000001b0u, 0x000200f9u, - 0x00000191u, 0x000200f8u, 0x00000191u, 0x0004003du, 0x00000008u, 0x000001eeu, 0x0000018au, 0x00050080u, - 0x00000008u, 0x000001efu, 0x000001eeu, 0x000001edu, 0x0003003eu, 0x0000018au, 0x000001efu, 0x000200f9u, - 0x0000018eu, 0x000200f8u, 0x00000190u, 0x000100fdu, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000000bu, - 0x00000000u, 0x00000009u, 0x00030037u, 0x00000007u, 0x0000000au, 0x000200f8u, 0x0000000cu, 0x0004003bu, - 0x00000010u, 0x00000016u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000031u, 0x00000007u, 0x0004003du, - 0x00000006u, 0x00000017u, 0x0000000au, 0x0004007cu, 0x00000008u, 0x00000018u, 0x00000017u, 0x0003003eu, - 0x00000016u, 0x00000018u, 0x0004003du, 0x00000008u, 0x0000001au, 0x00000016u, 0x000500c7u, 0x00000008u, - 0x0000001cu, 0x0000001au, 0x0000001bu, 0x000500aau, 0x00000019u, 0x0000001du, 0x0000001cu, 0x0000001bu, - 0x000300f7u, 0x0000001fu, 0x00000000u, 0x000400fau, 0x0000001du, 0x0000001eu, 0x0000001fu, 0x000200f8u, - 0x0000001eu, 0x0004003du, 0x00000008u, 0x00000020u, 0x00000016u, 0x000500c7u, 0x00000008u, 0x00000022u, - 0x00000020u, 0x00000021u, 0x000500abu, 0x00000019u, 0x00000024u, 0x00000022u, 0x00000023u, 0x000200f9u, - 0x0000001fu, 0x000200f8u, 0x0000001fu, 0x000700f5u, 0x00000019u, 0x00000025u, 0x0000001du, 0x0000000cu, - 0x00000024u, 0x0000001eu, 0x000300f7u, 0x00000027u, 0x00000000u, 0x000400fau, 0x00000025u, 0x00000026u, - 0x00000027u, 0x000200f8u, 0x00000026u, 0x0004003du, 0x00000008u, 0x00000028u, 0x00000016u, 0x000500c2u, - 0x00000008u, 0x0000002bu, 0x00000028u, 0x0000002au, 0x000500c5u, 0x00000008u, 0x0000002du, 0x0000002bu, - 0x0000002cu, 0x000500c7u, 0x00000008u, 0x0000002fu, 0x0000002du, 0x0000002eu, 0x000200feu, 0x0000002fu, - 0x000200f8u, 0x00000027u, 0x0004003du, 0x00000008u, 0x00000033u, 0x00000016u, 0x000500c2u, 0x00000008u, - 0x00000034u, 0x00000033u, 0x0000002au, 0x000500c7u, 0x00000008u, 0x00000036u, 0x00000034u, 0x00000035u, - 0x00050080u, 0x00000008u, 0x00000037u, 0x00000032u, 0x00000036u, 0x0003003eu, 0x00000031u, 0x00000037u, - 0x0004003du, 0x00000008u, 0x00000038u, 0x00000016u, 0x0004003du, 0x00000008u, 0x00000039u, 0x00000031u, - 0x00050080u, 0x00000008u, 0x0000003au, 0x00000038u, 0x00000039u, 0x000500c2u, 0x00000008u, 0x0000003bu, - 0x0000003au, 0x0000002au, 0x000500c7u, 0x00000008u, 0x0000003cu, 0x0000003bu, 0x0000002eu, 0x000200feu, - 0x0000003cu, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000000eu, 0x00000000u, 0x00000009u, 0x00030037u, - 0x00000007u, 0x0000000du, 0x000200f8u, 0x0000000fu, 0x0004003bu, 0x00000010u, 0x0000003fu, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x00000042u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000047u, 0x00000007u, - 0x0004003bu, 0x0000004du, 0x0000004eu, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000055u, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x00000061u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000084u, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x00000089u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x0000008du, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x00000093u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x000000afu, 0x00000007u, - 0x0004003bu, 0x00000010u, 0x000000b7u, 0x00000007u, 0x0004003du, 0x00000006u, 0x00000040u, 0x0000000du, - 0x0004007cu, 0x00000008u, 0x00000041u, 0x00000040u, 0x0003003eu, 0x0000003fu, 0x00000041u, 0x0004003du, - 0x00000008u, 0x00000043u, 0x0000003fu, 0x000500c2u, 0x00000008u, 0x00000044u, 0x00000043u, 0x0000002au, - 0x000500c7u, 0x00000008u, 0x00000046u, 0x00000044u, 0x00000045u, 0x0003003eu, 0x00000042u, 0x00000046u, - 0x0004003du, 0x00000008u, 0x00000048u, 0x0000003fu, 0x000500c2u, 0x00000008u, 0x0000004au, 0x00000048u, - 0x00000049u, 0x000500c7u, 0x00000008u, 0x0000004cu, 0x0000004au, 0x0000004bu, 0x0003003eu, 0x00000047u, - 0x0000004cu, 0x0004003du, 0x00000008u, 0x0000004fu, 0x00000047u, 0x0004007cu, 0x00000029u, 0x00000050u, - 0x0000004fu, 0x00050082u, 0x00000029u, 0x00000052u, 0x00000050u, 0x00000051u, 0x00050080u, 0x00000029u, - 0x00000054u, 0x00000052u, 0x00000053u, 0x0003003eu, 0x0000004eu, 0x00000054u, 0x0004003du, 0x00000008u, - 0x00000056u, 0x0000003fu, 0x000500c7u, 0x00000008u, 0x00000057u, 0x00000056u, 0x00000021u, 0x0003003eu, - 0x00000055u, 0x00000057u, 0x0004003du, 0x00000008u, 0x00000058u, 0x00000047u, 0x000500aau, 0x00000019u, - 0x00000059u, 0x00000058u, 0x0000004bu, 0x000300f7u, 0x0000005bu, 0x00000000u, 0x000400fau, 0x00000059u, - 0x0000005au, 0x0000005bu, 0x000200f8u, 0x0000005au, 0x0004003du, 0x00000008u, 0x0000005cu, 0x00000042u, - 0x000500c5u, 0x00000008u, 0x0000005eu, 0x0000005cu, 0x0000005du, 0x0004003du, 0x00000008u, 0x0000005fu, - 0x00000055u, 0x000500abu, 0x00000019u, 0x00000060u, 0x0000005fu, 0x00000023u, 0x000300f7u, 0x00000063u, - 0x00000000u, 0x000400fau, 0x00000060u, 0x00000062u, 0x00000069u, 0x000200f8u, 0x00000062u, 0x0004003du, - 0x00000008u, 0x00000065u, 0x00000055u, 0x000500c2u, 0x00000008u, 0x00000067u, 0x00000065u, 0x00000066u, - 0x000500c5u, 0x00000008u, 0x00000068u, 0x00000064u, 0x00000067u, 0x0003003eu, 0x00000061u, 0x00000068u, - 0x000200f9u, 0x00000063u, 0x000200f8u, 0x00000069u, 0x0003003eu, 0x00000061u, 0x00000023u, 0x000200f9u, - 0x00000063u, 0x000200f8u, 0x00000063u, 0x0004003du, 0x00000008u, 0x0000006au, 0x00000061u, 0x000500c5u, - 0x00000008u, 0x0000006bu, 0x0000005eu, 0x0000006au, 0x000200feu, 0x0000006bu, 0x000200f8u, 0x0000005bu, - 0x0004003du, 0x00000029u, 0x0000006du, 0x0000004eu, 0x000500afu, 0x00000019u, 0x0000006fu, 0x0000006du, - 0x0000006eu, 0x000300f7u, 0x00000071u, 0x00000000u, 0x000400fau, 0x0000006fu, 0x00000070u, 0x00000071u, - 0x000200f8u, 0x00000070u, 0x0004003du, 0x00000008u, 0x00000072u, 0x00000042u, 0x000500c5u, 0x00000008u, - 0x00000073u, 0x00000072u, 0x0000005du, 0x000200feu, 0x00000073u, 0x000200f8u, 0x00000071u, 0x0004003du, - 0x00000029u, 0x00000075u, 0x0000004eu, 0x000500b3u, 0x00000019u, 0x00000077u, 0x00000075u, 0x00000076u, - 0x000300f7u, 0x00000079u, 0x00000000u, 0x000400fau, 0x00000077u, 0x00000078u, 0x00000079u, 0x000200f8u, - 0x00000078u, 0x0004003du, 0x00000029u, 0x0000007au, 0x0000004eu, 0x000500b1u, 0x00000019u, 0x0000007cu, - 0x0000007au, 0x0000007bu, 0x000300f7u, 0x0000007eu, 0x00000000u, 0x000400fau, 0x0000007cu, 0x0000007du, - 0x0000007eu, 0x000200f8u, 0x0000007du, 0x0004003du, 0x00000008u, 0x0000007fu, 0x00000042u, 0x000200feu, - 0x0000007fu, 0x000200f8u, 0x0000007eu, 0x0004003du, 0x00000008u, 0x00000082u, 0x00000055u, 0x000500c5u, - 0x00000008u, 0x00000083u, 0x00000082u, 0x00000081u, 0x0003003eu, 0x00000055u, 0x00000083u, 0x0004003du, - 0x00000029u, 0x00000086u, 0x0000004eu, 0x00050082u, 0x00000029u, 0x00000087u, 0x00000085u, 0x00000086u, - 0x0004007cu, 0x00000008u, 0x00000088u, 0x00000087u, 0x0003003eu, 0x00000084u, 0x00000088u, 0x0004003du, - 0x00000008u, 0x0000008au, 0x00000055u, 0x0004003du, 0x00000008u, 0x0000008bu, 0x00000084u, 0x000500c2u, - 0x00000008u, 0x0000008cu, 0x0000008au, 0x0000008bu, 0x0003003eu, 0x00000089u, 0x0000008cu, 0x0004003du, - 0x00000008u, 0x0000008eu, 0x00000055u, 0x0004003du, 0x00000008u, 0x0000008fu, 0x00000084u, 0x000500c4u, - 0x00000008u, 0x00000090u, 0x00000035u, 0x0000008fu, 0x00050082u, 0x00000008u, 0x00000091u, 0x00000090u, - 0x00000035u, 0x000500c7u, 0x00000008u, 0x00000092u, 0x0000008eu, 0x00000091u, 0x0003003eu, 0x0000008du, - 0x00000092u, 0x0004003du, 0x00000008u, 0x00000094u, 0x00000084u, 0x00050082u, 0x00000008u, 0x00000095u, - 0x00000094u, 0x00000035u, 0x000500c4u, 0x00000008u, 0x00000096u, 0x00000035u, 0x00000095u, 0x0003003eu, - 0x00000093u, 0x00000096u, 0x0004003du, 0x00000008u, 0x00000097u, 0x0000008du, 0x0004003du, 0x00000008u, - 0x00000098u, 0x00000093u, 0x000500acu, 0x00000019u, 0x00000099u, 0x00000097u, 0x00000098u, 0x000400a8u, - 0x00000019u, 0x0000009au, 0x00000099u, 0x000300f7u, 0x0000009cu, 0x00000000u, 0x000400fau, 0x0000009au, - 0x0000009bu, 0x0000009cu, 0x000200f8u, 0x0000009bu, 0x0004003du, 0x00000008u, 0x0000009du, 0x0000008du, - 0x0004003du, 0x00000008u, 0x0000009eu, 0x00000093u, 0x000500aau, 0x00000019u, 0x0000009fu, 0x0000009du, - 0x0000009eu, 0x000300f7u, 0x000000a1u, 0x00000000u, 0x000400fau, 0x0000009fu, 0x000000a0u, 0x000000a1u, - 0x000200f8u, 0x000000a0u, 0x0004003du, 0x00000008u, 0x000000a2u, 0x00000089u, 0x000500c7u, 0x00000008u, - 0x000000a3u, 0x000000a2u, 0x00000035u, 0x000500abu, 0x00000019u, 0x000000a4u, 0x000000a3u, 0x00000023u, - 0x000200f9u, 0x000000a1u, 0x000200f8u, 0x000000a1u, 0x000700f5u, 0x00000019u, 0x000000a5u, 0x0000009fu, - 0x0000009bu, 0x000000a4u, 0x000000a0u, 0x000200f9u, 0x0000009cu, 0x000200f8u, 0x0000009cu, 0x000700f5u, - 0x00000019u, 0x000000a6u, 0x00000099u, 0x0000007eu, 0x000000a5u, 0x000000a1u, 0x000300f7u, 0x000000a8u, - 0x00000000u, 0x000400fau, 0x000000a6u, 0x000000a7u, 0x000000a8u, 0x000200f8u, 0x000000a7u, 0x0004003du, - 0x00000008u, 0x000000a9u, 0x00000089u, 0x00050080u, 0x00000008u, 0x000000aau, 0x000000a9u, 0x00000035u, - 0x0003003eu, 0x00000089u, 0x000000aau, 0x000200f9u, 0x000000a8u, 0x000200f8u, 0x000000a8u, 0x0004003du, - 0x00000008u, 0x000000abu, 0x00000042u, 0x0004003du, 0x00000008u, 0x000000acu, 0x00000089u, 0x000500c5u, - 0x00000008u, 0x000000adu, 0x000000abu, 0x000000acu, 0x000200feu, 0x000000adu, 0x000200f8u, 0x00000079u, - 0x0004003du, 0x00000029u, 0x000000b0u, 0x0000004eu, 0x0004007cu, 0x00000008u, 0x000000b1u, 0x000000b0u, - 0x000500c4u, 0x00000008u, 0x000000b3u, 0x000000b1u, 0x000000b2u, 0x0004003du, 0x00000008u, 0x000000b4u, - 0x00000055u, 0x000500c2u, 0x00000008u, 0x000000b5u, 0x000000b4u, 0x00000066u, 0x000500c5u, 0x00000008u, - 0x000000b6u, 0x000000b3u, 0x000000b5u, 0x0003003eu, 0x000000afu, 0x000000b6u, 0x0004003du, 0x00000008u, - 0x000000b8u, 0x00000055u, 0x000500c7u, 0x00000008u, 0x000000bau, 0x000000b8u, 0x000000b9u, 0x0003003eu, - 0x000000b7u, 0x000000bau, 0x0004003du, 0x00000008u, 0x000000bbu, 0x000000b7u, 0x000500acu, 0x00000019u, - 0x000000bdu, 0x000000bbu, 0x000000bcu, 0x000400a8u, 0x00000019u, 0x000000beu, 0x000000bdu, 0x000300f7u, - 0x000000c0u, 0x00000000u, 0x000400fau, 0x000000beu, 0x000000bfu, 0x000000c0u, 0x000200f8u, 0x000000bfu, - 0x0004003du, 0x00000008u, 0x000000c1u, 0x000000b7u, 0x000500aau, 0x00000019u, 0x000000c2u, 0x000000c1u, - 0x000000bcu, 0x000300f7u, 0x000000c4u, 0x00000000u, 0x000400fau, 0x000000c2u, 0x000000c3u, 0x000000c4u, - 0x000200f8u, 0x000000c3u, 0x0004003du, 0x00000008u, 0x000000c5u, 0x000000afu, 0x000500c7u, 0x00000008u, - 0x000000c6u, 0x000000c5u, 0x00000035u, 0x000500abu, 0x00000019u, 0x000000c7u, 0x000000c6u, 0x00000023u, - 0x000200f9u, 0x000000c4u, 0x000200f8u, 0x000000c4u, 0x000700f5u, 0x00000019u, 0x000000c8u, 0x000000c2u, - 0x000000bfu, 0x000000c7u, 0x000000c3u, 0x000200f9u, 0x000000c0u, 0x000200f8u, 0x000000c0u, 0x000700f5u, - 0x00000019u, 0x000000c9u, 0x000000bdu, 0x00000079u, 0x000000c8u, 0x000000c4u, 0x000300f7u, 0x000000cbu, - 0x00000000u, 0x000400fau, 0x000000c9u, 0x000000cau, 0x000000cbu, 0x000200f8u, 0x000000cau, 0x0004003du, - 0x00000008u, 0x000000ccu, 0x000000afu, 0x00050080u, 0x00000008u, 0x000000cdu, 0x000000ccu, 0x00000035u, - 0x0003003eu, 0x000000afu, 0x000000cdu, 0x000200f9u, 0x000000cbu, 0x000200f8u, 0x000000cbu, 0x0004003du, - 0x00000008u, 0x000000ceu, 0x00000042u, 0x0004003du, 0x00000008u, 0x000000cfu, 0x000000afu, 0x000500c5u, - 0x00000008u, 0x000000d0u, 0x000000ceu, 0x000000cfu, 0x000200feu, 0x000000d0u, 0x00010038u, 0x00050036u, - 0x00000008u, 0x00000014u, 0x00000000u, 0x00000011u, 0x00030037u, 0x00000007u, 0x00000012u, 0x00030037u, - 0x00000010u, 0x00000013u, 0x000200f8u, 0x00000015u, 0x0004003bu, 0x00000010u, 0x000000d5u, 0x00000007u, - 0x0004003bu, 0x00000007u, 0x000000d8u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000dcu, 0x00000007u, - 0x0004003du, 0x00000008u, 0x000000d3u, 0x00000013u, 0x000500aau, 0x00000019u, 0x000000d4u, 0x000000d3u, - 0x00000035u, 0x000300f7u, 0x000000d7u, 0x00000000u, 0x000400fau, 0x000000d4u, 0x000000d6u, 0x000000dbu, - 0x000200f8u, 0x000000d6u, 0x0004003du, 0x00000006u, 0x000000d9u, 0x00000012u, 0x0003003eu, 0x000000d8u, - 0x000000d9u, 0x00050039u, 0x00000008u, 0x000000dau, 0x0000000eu, 0x000000d8u, 0x0003003eu, 0x000000d5u, - 0x000000dau, 0x000200f9u, 0x000000d7u, 0x000200f8u, 0x000000dbu, 0x0004003du, 0x00000006u, 0x000000ddu, - 0x00000012u, 0x0003003eu, 0x000000dcu, 0x000000ddu, 0x00050039u, 0x00000008u, 0x000000deu, 0x0000000bu, - 0x000000dcu, 0x0003003eu, 0x000000d5u, 0x000000deu, 0x000200f9u, 0x000000d7u, 0x000200f8u, 0x000000d7u, - 0x0004003du, 0x00000008u, 0x000000dfu, 0x000000d5u, 0x000200feu, 0x000000dfu, 0x00010038u, + 0x0003003eu, 0x00000106u, 0x0000010au, 0x0004003du, 0x00000008u, 0x0000010bu, 0x00000101u, 0x00050080u, + 0x00000008u, 0x0000010cu, 0x0000010bu, 0x000000eau, 0x00050041u, 0x000000e7u, 0x0000010du, 0x000000e5u, + 0x00000076u, 0x0004003du, 0x00000008u, 0x0000010eu, 0x0000010du, 0x000500acu, 0x00000019u, 0x0000010fu, + 0x0000010cu, 0x0000010eu, 0x000300f7u, 0x00000111u, 0x00000000u, 0x000400fau, 0x0000010fu, 0x00000110u, + 0x00000111u, 0x000200f8u, 0x00000110u, 0x00050041u, 0x000000e7u, 0x00000112u, 0x000000e5u, 0x00000076u, + 0x0004003du, 0x00000008u, 0x00000113u, 0x00000112u, 0x00050082u, 0x00000008u, 0x00000114u, 0x00000113u, + 0x000000eau, 0x0003003eu, 0x00000101u, 0x00000114u, 0x000200f9u, 0x00000111u, 0x000200f8u, 0x00000111u, + 0x0003003eu, 0x00000119u, 0x0000011bu, 0x0003003eu, 0x0000011cu, 0x00000023u, 0x000200f9u, 0x0000011du, + 0x000200f8u, 0x0000011du, 0x000400f6u, 0x0000011fu, 0x00000120u, 0x00000000u, 0x000200f9u, 0x00000121u, + 0x000200f8u, 0x00000121u, 0x0004003du, 0x00000008u, 0x00000122u, 0x0000011cu, 0x00050041u, 0x000000e7u, + 0x00000124u, 0x000000e5u, 0x00000123u, 0x0004003du, 0x00000008u, 0x00000125u, 0x00000124u, 0x000500b0u, + 0x00000019u, 0x00000126u, 0x00000122u, 0x00000125u, 0x000400fau, 0x00000126u, 0x0000011eu, 0x0000011fu, + 0x000200f8u, 0x0000011eu, 0x00050041u, 0x000000e7u, 0x00000130u, 0x000000e5u, 0x0000012fu, 0x0004003du, + 0x00000008u, 0x00000131u, 0x00000130u, 0x000500c2u, 0x00000008u, 0x00000132u, 0x00000131u, 0x000000e6u, + 0x0004003du, 0x00000008u, 0x00000133u, 0x00000101u, 0x00050041u, 0x000000e7u, 0x00000134u, 0x000000e5u, + 0x00000123u, 0x0004003du, 0x00000008u, 0x00000135u, 0x00000134u, 0x00050084u, 0x00000008u, 0x00000136u, + 0x00000133u, 0x00000135u, 0x00050080u, 0x00000008u, 0x00000137u, 0x00000132u, 0x00000136u, 0x0004003du, + 0x00000008u, 0x00000138u, 0x0000011cu, 0x00050080u, 0x00000008u, 0x00000139u, 0x00000137u, 0x00000138u, + 0x00060041u, 0x0000013au, 0x0000013bu, 0x0000012eu, 0x00000076u, 0x00000139u, 0x00050041u, 0x000000e7u, + 0x0000013cu, 0x000000e5u, 0x0000012fu, 0x0004003du, 0x00000008u, 0x0000013du, 0x0000013cu, 0x000500c2u, + 0x00000008u, 0x0000013eu, 0x0000013du, 0x000000e6u, 0x0004003du, 0x00000008u, 0x0000013fu, 0x00000101u, + 0x00050041u, 0x000000e7u, 0x00000140u, 0x000000e5u, 0x00000123u, 0x0004003du, 0x00000008u, 0x00000141u, + 0x00000140u, 0x00050084u, 0x00000008u, 0x00000142u, 0x0000013fu, 0x00000141u, 0x00050080u, 0x00000008u, + 0x00000143u, 0x0000013eu, 0x00000142u, 0x0004003du, 0x00000008u, 0x00000144u, 0x0000011cu, 0x00050080u, + 0x00000008u, 0x00000145u, 0x00000143u, 0x00000144u, 0x00050041u, 0x000000e7u, 0x00000146u, 0x000000e5u, + 0x00000123u, 0x0004003du, 0x00000008u, 0x00000147u, 0x00000146u, 0x00071169u, 0x00000128u, 0x00000148u, + 0x0000013bu, 0x00000076u, 0x00000147u, 0x00000000u, 0x0003003eu, 0x0000012au, 0x00000148u, 0x000300f7u, + 0x0000014cu, 0x00000000u, 0x000400fau, 0x0000014au, 0x0000014bu, 0x0000016du, 0x000200f8u, 0x0000014bu, + 0x00050041u, 0x000000e7u, 0x00000155u, 0x000000e5u, 0x00000154u, 0x0004003du, 0x00000008u, 0x00000156u, + 0x00000155u, 0x000500c2u, 0x00000008u, 0x00000157u, 0x00000156u, 0x000000e6u, 0x0004003du, 0x00000008u, + 0x00000158u, 0x00000106u, 0x00050041u, 0x000000e7u, 0x00000159u, 0x000000e5u, 0x00000123u, 0x0004003du, + 0x00000008u, 0x0000015au, 0x00000159u, 0x00050084u, 0x00000008u, 0x0000015bu, 0x00000158u, 0x0000015au, + 0x00050080u, 0x00000008u, 0x0000015cu, 0x00000157u, 0x0000015bu, 0x0004003du, 0x00000008u, 0x0000015du, + 0x0000011cu, 0x00050080u, 0x00000008u, 0x0000015eu, 0x0000015cu, 0x0000015du, 0x00060041u, 0x0000013au, + 0x0000015fu, 0x00000153u, 0x00000076u, 0x0000015eu, 0x00050041u, 0x000000e7u, 0x00000160u, 0x000000e5u, + 0x00000154u, 0x0004003du, 0x00000008u, 0x00000161u, 0x00000160u, 0x000500c2u, 0x00000008u, 0x00000162u, + 0x00000161u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x00000163u, 0x00000106u, 0x00050041u, 0x000000e7u, + 0x00000164u, 0x000000e5u, 0x00000123u, 0x0004003du, 0x00000008u, 0x00000165u, 0x00000164u, 0x00050084u, + 0x00000008u, 0x00000166u, 0x00000163u, 0x00000165u, 0x00050080u, 0x00000008u, 0x00000167u, 0x00000162u, + 0x00000166u, 0x0004003du, 0x00000008u, 0x00000168u, 0x0000011cu, 0x00050080u, 0x00000008u, 0x00000169u, + 0x00000167u, 0x00000168u, 0x00050041u, 0x000000e7u, 0x0000016au, 0x000000e5u, 0x00000123u, 0x0004003du, + 0x00000008u, 0x0000016bu, 0x0000016au, 0x00071169u, 0x0000014du, 0x0000016cu, 0x0000015fu, 0x000000e6u, + 0x0000016bu, 0x00000000u, 0x0003003eu, 0x0000014fu, 0x0000016cu, 0x000200f9u, 0x0000014cu, 0x000200f8u, + 0x0000016du, 0x00050041u, 0x000000e7u, 0x0000016eu, 0x000000e5u, 0x00000154u, 0x0004003du, 0x00000008u, + 0x0000016fu, 0x0000016eu, 0x000500c2u, 0x00000008u, 0x00000170u, 0x0000016fu, 0x000000e6u, 0x0004003du, + 0x00000008u, 0x00000171u, 0x0000011cu, 0x00050041u, 0x000000e7u, 0x00000172u, 0x000000e5u, 0x000000e6u, + 0x0004003du, 0x00000008u, 0x00000173u, 0x00000172u, 0x00050084u, 0x00000008u, 0x00000174u, 0x00000171u, + 0x00000173u, 0x00050080u, 0x00000008u, 0x00000175u, 0x00000170u, 0x00000174u, 0x0004003du, 0x00000008u, + 0x00000176u, 0x00000106u, 0x00050080u, 0x00000008u, 0x00000177u, 0x00000175u, 0x00000176u, 0x00060041u, + 0x0000013au, 0x00000178u, 0x00000153u, 0x00000076u, 0x00000177u, 0x00050041u, 0x000000e7u, 0x00000179u, + 0x000000e5u, 0x00000154u, 0x0004003du, 0x00000008u, 0x0000017au, 0x00000179u, 0x000500c2u, 0x00000008u, + 0x0000017bu, 0x0000017au, 0x000000e6u, 0x0004003du, 0x00000008u, 0x0000017cu, 0x0000011cu, 0x00050041u, + 0x000000e7u, 0x0000017du, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x0000017eu, 0x0000017du, + 0x00050084u, 0x00000008u, 0x0000017fu, 0x0000017cu, 0x0000017eu, 0x00050080u, 0x00000008u, 0x00000180u, + 0x0000017bu, 0x0000017fu, 0x0004003du, 0x00000008u, 0x00000181u, 0x00000106u, 0x00050080u, 0x00000008u, + 0x00000182u, 0x00000180u, 0x00000181u, 0x00050041u, 0x000000e7u, 0x00000183u, 0x000000e5u, 0x000000e6u, + 0x0004003du, 0x00000008u, 0x00000184u, 0x00000183u, 0x00071169u, 0x0000014du, 0x00000185u, 0x00000178u, + 0x00000076u, 0x00000184u, 0x00000000u, 0x0003003eu, 0x0000014fu, 0x00000185u, 0x000200f9u, 0x0000014cu, + 0x000200f8u, 0x0000014cu, 0x0004003du, 0x00000128u, 0x00000186u, 0x0000012au, 0x0004003du, 0x0000014du, + 0x00000187u, 0x0000014fu, 0x0004003du, 0x00000117u, 0x00000188u, 0x00000119u, 0x0006116bu, 0x00000117u, + 0x00000189u, 0x00000186u, 0x00000187u, 0x00000188u, 0x0003003eu, 0x00000119u, 0x00000189u, 0x000200f9u, + 0x00000120u, 0x000200f8u, 0x00000120u, 0x0004003du, 0x00000008u, 0x0000018au, 0x0000011cu, 0x00050080u, + 0x00000008u, 0x0000018bu, 0x0000018au, 0x000000eau, 0x0003003eu, 0x0000011cu, 0x0000018bu, 0x000200f9u, + 0x0000011du, 0x000200f8u, 0x0000011fu, 0x0004003du, 0x00000117u, 0x0000018cu, 0x00000119u, 0x00050041u, + 0x00000191u, 0x00000192u, 0x00000190u, 0x00000023u, 0x0007116au, 0x00000192u, 0x0000018cu, 0x00000076u, + 0x000000eau, 0x00000028u, 0x00000116u, 0x000400e0u, 0x00000116u, 0x00000116u, 0x00000193u, 0x00050041u, + 0x000000f2u, 0x00000196u, 0x00000195u, 0x00000023u, 0x0004003du, 0x00000008u, 0x00000197u, 0x00000196u, + 0x0003003eu, 0x00000194u, 0x00000197u, 0x000200f9u, 0x00000198u, 0x000200f8u, 0x00000198u, 0x000400f6u, + 0x0000019au, 0x0000019bu, 0x00000000u, 0x000200f9u, 0x0000019cu, 0x000200f8u, 0x0000019cu, 0x0004003du, + 0x00000008u, 0x0000019du, 0x00000194u, 0x000500b0u, 0x00000019u, 0x0000019eu, 0x0000019du, 0x0000018du, + 0x000400fau, 0x0000019eu, 0x00000199u, 0x0000019au, 0x000200f8u, 0x00000199u, 0x0004003du, 0x00000008u, + 0x000001a0u, 0x00000194u, 0x00050086u, 0x00000008u, 0x000001a1u, 0x000001a0u, 0x000000eau, 0x0003003eu, + 0x0000019fu, 0x000001a1u, 0x0004003du, 0x00000008u, 0x000001a3u, 0x00000194u, 0x00050089u, 0x00000008u, + 0x000001a4u, 0x000001a3u, 0x000000eau, 0x0003003eu, 0x000001a2u, 0x000001a4u, 0x0004003du, 0x00000008u, + 0x000001a5u, 0x00000101u, 0x0004003du, 0x00000008u, 0x000001a6u, 0x0000019fu, 0x00050080u, 0x00000008u, + 0x000001a7u, 0x000001a5u, 0x000001a6u, 0x00050041u, 0x000000e7u, 0x000001a8u, 0x000000e5u, 0x00000076u, + 0x0004003du, 0x00000008u, 0x000001a9u, 0x000001a8u, 0x000500aeu, 0x00000019u, 0x000001aau, 0x000001a7u, + 0x000001a9u, 0x000400a8u, 0x00000019u, 0x000001abu, 0x000001aau, 0x000300f7u, 0x000001adu, 0x00000000u, + 0x000400fau, 0x000001abu, 0x000001acu, 0x000001adu, 0x000200f8u, 0x000001acu, 0x0004003du, 0x00000008u, + 0x000001aeu, 0x00000106u, 0x0004003du, 0x00000008u, 0x000001afu, 0x000001a2u, 0x00050080u, 0x00000008u, + 0x000001b0u, 0x000001aeu, 0x000001afu, 0x00050041u, 0x000000e7u, 0x000001b1u, 0x000000e5u, 0x000000e6u, + 0x0004003du, 0x00000008u, 0x000001b2u, 0x000001b1u, 0x000500aeu, 0x00000019u, 0x000001b3u, 0x000001b0u, + 0x000001b2u, 0x000200f9u, 0x000001adu, 0x000200f8u, 0x000001adu, 0x000700f5u, 0x00000019u, 0x000001b4u, + 0x000001aau, 0x00000199u, 0x000001b3u, 0x000001acu, 0x000300f7u, 0x000001b6u, 0x00000000u, 0x000400fau, + 0x000001b4u, 0x000001b5u, 0x000001b6u, 0x000200f8u, 0x000001b5u, 0x000200f9u, 0x0000019bu, 0x000200f8u, + 0x000001b6u, 0x000200f9u, 0x000001b8u, 0x000200f8u, 0x000001b8u, 0x000400f6u, 0x000001bau, 0x000001bbu, + 0x00000000u, 0x000200f9u, 0x000001b9u, 0x000200f8u, 0x000001b9u, 0x000300f7u, 0x000001bfu, 0x00000000u, + 0x000400fau, 0x000001bdu, 0x000001beu, 0x000001d9u, 0x000200f8u, 0x000001beu, 0x00050041u, 0x000000e7u, + 0x000001c5u, 0x000000e5u, 0x000001c4u, 0x0004003du, 0x00000008u, 0x000001c6u, 0x000001c5u, 0x000500c2u, + 0x00000008u, 0x000001c7u, 0x000001c6u, 0x00000123u, 0x0004003du, 0x00000008u, 0x000001c8u, 0x00000101u, + 0x0004003du, 0x00000008u, 0x000001c9u, 0x0000019fu, 0x00050080u, 0x00000008u, 0x000001cau, 0x000001c8u, + 0x000001c9u, 0x00050041u, 0x000000e7u, 0x000001cbu, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, + 0x000001ccu, 0x000001cbu, 0x00050084u, 0x00000008u, 0x000001cdu, 0x000001cau, 0x000001ccu, 0x0004003du, + 0x00000008u, 0x000001ceu, 0x00000106u, 0x0004003du, 0x00000008u, 0x000001cfu, 0x000001a2u, 0x00050080u, + 0x00000008u, 0x000001d0u, 0x000001ceu, 0x000001cfu, 0x00050080u, 0x00000008u, 0x000001d1u, 0x000001cdu, + 0x000001d0u, 0x00050080u, 0x00000008u, 0x000001d2u, 0x000001c7u, 0x000001d1u, 0x0004003du, 0x00000008u, + 0x000001d3u, 0x00000194u, 0x00050041u, 0x00000191u, 0x000001d4u, 0x00000190u, 0x000001d3u, 0x0006003du, + 0x00000006u, 0x000001d5u, 0x000001d4u, 0x00000030u, 0x00000116u, 0x0004007cu, 0x00000008u, 0x000001d6u, + 0x000001d5u, 0x00060041u, 0x000001d7u, 0x000001d8u, 0x000001c3u, 0x00000076u, 0x000001d2u, 0x0003003eu, + 0x000001d8u, 0x000001d6u, 0x000200f9u, 0x000001bfu, 0x000200f8u, 0x000001d9u, 0x00050041u, 0x000000e7u, + 0x000001dfu, 0x000000e5u, 0x000001c4u, 0x0004003du, 0x00000008u, 0x000001e0u, 0x000001dfu, 0x000500c2u, + 0x00000008u, 0x000001e1u, 0x000001e0u, 0x000000e6u, 0x0004003du, 0x00000008u, 0x000001e2u, 0x00000101u, + 0x0004003du, 0x00000008u, 0x000001e3u, 0x0000019fu, 0x00050080u, 0x00000008u, 0x000001e4u, 0x000001e2u, + 0x000001e3u, 0x00050041u, 0x000000e7u, 0x000001e5u, 0x000000e5u, 0x000000e6u, 0x0004003du, 0x00000008u, + 0x000001e6u, 0x000001e5u, 0x00050084u, 0x00000008u, 0x000001e7u, 0x000001e4u, 0x000001e6u, 0x0004003du, + 0x00000008u, 0x000001e8u, 0x00000106u, 0x0004003du, 0x00000008u, 0x000001e9u, 0x000001a2u, 0x00050080u, + 0x00000008u, 0x000001eau, 0x000001e8u, 0x000001e9u, 0x00050080u, 0x00000008u, 0x000001ebu, 0x000001e7u, + 0x000001eau, 0x00050080u, 0x00000008u, 0x000001ecu, 0x000001e1u, 0x000001ebu, 0x0004003du, 0x00000008u, + 0x000001edu, 0x00000194u, 0x00050041u, 0x00000191u, 0x000001efu, 0x00000190u, 0x000001edu, 0x0006003du, + 0x00000006u, 0x000001f0u, 0x000001efu, 0x00000030u, 0x00000116u, 0x0003003eu, 0x000001eeu, 0x000001f0u, + 0x0003003eu, 0x000001f1u, 0x000001bcu, 0x00060039u, 0x00000008u, 0x000001f2u, 0x00000014u, 0x000001eeu, + 0x000001f1u, 0x00040071u, 0x000001dau, 0x000001f3u, 0x000001f2u, 0x00060041u, 0x000001f4u, 0x000001f5u, + 0x000001deu, 0x00000076u, 0x000001ecu, 0x0003003eu, 0x000001f5u, 0x000001f3u, 0x000200f9u, 0x000001bfu, + 0x000200f8u, 0x000001bfu, 0x000200f9u, 0x000001bbu, 0x000200f8u, 0x000001bbu, 0x000400fau, 0x000001f6u, + 0x000001b8u, 0x000001bau, 0x000200f8u, 0x000001bau, 0x000200f9u, 0x0000019bu, 0x000200f8u, 0x0000019bu, + 0x0004003du, 0x00000008u, 0x000001f8u, 0x00000194u, 0x00050080u, 0x00000008u, 0x000001f9u, 0x000001f8u, + 0x000001f7u, 0x0003003eu, 0x00000194u, 0x000001f9u, 0x000200f9u, 0x00000198u, 0x000200f8u, 0x0000019au, + 0x000100fdu, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000000bu, 0x00000000u, 0x00000009u, 0x00030037u, + 0x00000007u, 0x0000000au, 0x000200f8u, 0x0000000cu, 0x0004003bu, 0x00000010u, 0x00000016u, 0x00000007u, + 0x0004003bu, 0x00000010u, 0x00000031u, 0x00000007u, 0x0004003du, 0x00000006u, 0x00000017u, 0x0000000au, + 0x0004007cu, 0x00000008u, 0x00000018u, 0x00000017u, 0x0003003eu, 0x00000016u, 0x00000018u, 0x0004003du, + 0x00000008u, 0x0000001au, 0x00000016u, 0x000500c7u, 0x00000008u, 0x0000001cu, 0x0000001au, 0x0000001bu, + 0x000500aau, 0x00000019u, 0x0000001du, 0x0000001cu, 0x0000001bu, 0x000300f7u, 0x0000001fu, 0x00000000u, + 0x000400fau, 0x0000001du, 0x0000001eu, 0x0000001fu, 0x000200f8u, 0x0000001eu, 0x0004003du, 0x00000008u, + 0x00000020u, 0x00000016u, 0x000500c7u, 0x00000008u, 0x00000022u, 0x00000020u, 0x00000021u, 0x000500abu, + 0x00000019u, 0x00000024u, 0x00000022u, 0x00000023u, 0x000200f9u, 0x0000001fu, 0x000200f8u, 0x0000001fu, + 0x000700f5u, 0x00000019u, 0x00000025u, 0x0000001du, 0x0000000cu, 0x00000024u, 0x0000001eu, 0x000300f7u, + 0x00000027u, 0x00000000u, 0x000400fau, 0x00000025u, 0x00000026u, 0x00000027u, 0x000200f8u, 0x00000026u, + 0x0004003du, 0x00000008u, 0x00000028u, 0x00000016u, 0x000500c2u, 0x00000008u, 0x0000002bu, 0x00000028u, + 0x0000002au, 0x000500c5u, 0x00000008u, 0x0000002du, 0x0000002bu, 0x0000002cu, 0x000500c7u, 0x00000008u, + 0x0000002fu, 0x0000002du, 0x0000002eu, 0x000200feu, 0x0000002fu, 0x000200f8u, 0x00000027u, 0x0004003du, + 0x00000008u, 0x00000033u, 0x00000016u, 0x000500c2u, 0x00000008u, 0x00000034u, 0x00000033u, 0x0000002au, + 0x000500c7u, 0x00000008u, 0x00000036u, 0x00000034u, 0x00000035u, 0x00050080u, 0x00000008u, 0x00000037u, + 0x00000032u, 0x00000036u, 0x0003003eu, 0x00000031u, 0x00000037u, 0x0004003du, 0x00000008u, 0x00000038u, + 0x00000016u, 0x0004003du, 0x00000008u, 0x00000039u, 0x00000031u, 0x00050080u, 0x00000008u, 0x0000003au, + 0x00000038u, 0x00000039u, 0x000500c2u, 0x00000008u, 0x0000003bu, 0x0000003au, 0x0000002au, 0x000500c7u, + 0x00000008u, 0x0000003cu, 0x0000003bu, 0x0000002eu, 0x000200feu, 0x0000003cu, 0x00010038u, 0x00050036u, + 0x00000008u, 0x0000000eu, 0x00000000u, 0x00000009u, 0x00030037u, 0x00000007u, 0x0000000du, 0x000200f8u, + 0x0000000fu, 0x0004003bu, 0x00000010u, 0x0000003fu, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000042u, + 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000047u, 0x00000007u, 0x0004003bu, 0x0000004du, 0x0000004eu, + 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000055u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000061u, + 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000084u, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000089u, + 0x00000007u, 0x0004003bu, 0x00000010u, 0x0000008du, 0x00000007u, 0x0004003bu, 0x00000010u, 0x00000093u, + 0x00000007u, 0x0004003bu, 0x00000010u, 0x000000afu, 0x00000007u, 0x0004003bu, 0x00000010u, 0x000000b7u, + 0x00000007u, 0x0004003du, 0x00000006u, 0x00000040u, 0x0000000du, 0x0004007cu, 0x00000008u, 0x00000041u, + 0x00000040u, 0x0003003eu, 0x0000003fu, 0x00000041u, 0x0004003du, 0x00000008u, 0x00000043u, 0x0000003fu, + 0x000500c2u, 0x00000008u, 0x00000044u, 0x00000043u, 0x0000002au, 0x000500c7u, 0x00000008u, 0x00000046u, + 0x00000044u, 0x00000045u, 0x0003003eu, 0x00000042u, 0x00000046u, 0x0004003du, 0x00000008u, 0x00000048u, + 0x0000003fu, 0x000500c2u, 0x00000008u, 0x0000004au, 0x00000048u, 0x00000049u, 0x000500c7u, 0x00000008u, + 0x0000004cu, 0x0000004au, 0x0000004bu, 0x0003003eu, 0x00000047u, 0x0000004cu, 0x0004003du, 0x00000008u, + 0x0000004fu, 0x00000047u, 0x0004007cu, 0x00000029u, 0x00000050u, 0x0000004fu, 0x00050082u, 0x00000029u, + 0x00000052u, 0x00000050u, 0x00000051u, 0x00050080u, 0x00000029u, 0x00000054u, 0x00000052u, 0x00000053u, + 0x0003003eu, 0x0000004eu, 0x00000054u, 0x0004003du, 0x00000008u, 0x00000056u, 0x0000003fu, 0x000500c7u, + 0x00000008u, 0x00000057u, 0x00000056u, 0x00000021u, 0x0003003eu, 0x00000055u, 0x00000057u, 0x0004003du, + 0x00000008u, 0x00000058u, 0x00000047u, 0x000500aau, 0x00000019u, 0x00000059u, 0x00000058u, 0x0000004bu, + 0x000300f7u, 0x0000005bu, 0x00000000u, 0x000400fau, 0x00000059u, 0x0000005au, 0x0000005bu, 0x000200f8u, + 0x0000005au, 0x0004003du, 0x00000008u, 0x0000005cu, 0x00000042u, 0x000500c5u, 0x00000008u, 0x0000005eu, + 0x0000005cu, 0x0000005du, 0x0004003du, 0x00000008u, 0x0000005fu, 0x00000055u, 0x000500abu, 0x00000019u, + 0x00000060u, 0x0000005fu, 0x00000023u, 0x000300f7u, 0x00000063u, 0x00000000u, 0x000400fau, 0x00000060u, + 0x00000062u, 0x00000069u, 0x000200f8u, 0x00000062u, 0x0004003du, 0x00000008u, 0x00000065u, 0x00000055u, + 0x000500c2u, 0x00000008u, 0x00000067u, 0x00000065u, 0x00000066u, 0x000500c5u, 0x00000008u, 0x00000068u, + 0x00000064u, 0x00000067u, 0x0003003eu, 0x00000061u, 0x00000068u, 0x000200f9u, 0x00000063u, 0x000200f8u, + 0x00000069u, 0x0003003eu, 0x00000061u, 0x00000023u, 0x000200f9u, 0x00000063u, 0x000200f8u, 0x00000063u, + 0x0004003du, 0x00000008u, 0x0000006au, 0x00000061u, 0x000500c5u, 0x00000008u, 0x0000006bu, 0x0000005eu, + 0x0000006au, 0x000200feu, 0x0000006bu, 0x000200f8u, 0x0000005bu, 0x0004003du, 0x00000029u, 0x0000006du, + 0x0000004eu, 0x000500afu, 0x00000019u, 0x0000006fu, 0x0000006du, 0x0000006eu, 0x000300f7u, 0x00000071u, + 0x00000000u, 0x000400fau, 0x0000006fu, 0x00000070u, 0x00000071u, 0x000200f8u, 0x00000070u, 0x0004003du, + 0x00000008u, 0x00000072u, 0x00000042u, 0x000500c5u, 0x00000008u, 0x00000073u, 0x00000072u, 0x0000005du, + 0x000200feu, 0x00000073u, 0x000200f8u, 0x00000071u, 0x0004003du, 0x00000029u, 0x00000075u, 0x0000004eu, + 0x000500b3u, 0x00000019u, 0x00000077u, 0x00000075u, 0x00000076u, 0x000300f7u, 0x00000079u, 0x00000000u, + 0x000400fau, 0x00000077u, 0x00000078u, 0x00000079u, 0x000200f8u, 0x00000078u, 0x0004003du, 0x00000029u, + 0x0000007au, 0x0000004eu, 0x000500b1u, 0x00000019u, 0x0000007cu, 0x0000007au, 0x0000007bu, 0x000300f7u, + 0x0000007eu, 0x00000000u, 0x000400fau, 0x0000007cu, 0x0000007du, 0x0000007eu, 0x000200f8u, 0x0000007du, + 0x0004003du, 0x00000008u, 0x0000007fu, 0x00000042u, 0x000200feu, 0x0000007fu, 0x000200f8u, 0x0000007eu, + 0x0004003du, 0x00000008u, 0x00000082u, 0x00000055u, 0x000500c5u, 0x00000008u, 0x00000083u, 0x00000082u, + 0x00000081u, 0x0003003eu, 0x00000055u, 0x00000083u, 0x0004003du, 0x00000029u, 0x00000086u, 0x0000004eu, + 0x00050082u, 0x00000029u, 0x00000087u, 0x00000085u, 0x00000086u, 0x0004007cu, 0x00000008u, 0x00000088u, + 0x00000087u, 0x0003003eu, 0x00000084u, 0x00000088u, 0x0004003du, 0x00000008u, 0x0000008au, 0x00000055u, + 0x0004003du, 0x00000008u, 0x0000008bu, 0x00000084u, 0x000500c2u, 0x00000008u, 0x0000008cu, 0x0000008au, + 0x0000008bu, 0x0003003eu, 0x00000089u, 0x0000008cu, 0x0004003du, 0x00000008u, 0x0000008eu, 0x00000055u, + 0x0004003du, 0x00000008u, 0x0000008fu, 0x00000084u, 0x000500c4u, 0x00000008u, 0x00000090u, 0x00000035u, + 0x0000008fu, 0x00050082u, 0x00000008u, 0x00000091u, 0x00000090u, 0x00000035u, 0x000500c7u, 0x00000008u, + 0x00000092u, 0x0000008eu, 0x00000091u, 0x0003003eu, 0x0000008du, 0x00000092u, 0x0004003du, 0x00000008u, + 0x00000094u, 0x00000084u, 0x00050082u, 0x00000008u, 0x00000095u, 0x00000094u, 0x00000035u, 0x000500c4u, + 0x00000008u, 0x00000096u, 0x00000035u, 0x00000095u, 0x0003003eu, 0x00000093u, 0x00000096u, 0x0004003du, + 0x00000008u, 0x00000097u, 0x0000008du, 0x0004003du, 0x00000008u, 0x00000098u, 0x00000093u, 0x000500acu, + 0x00000019u, 0x00000099u, 0x00000097u, 0x00000098u, 0x000400a8u, 0x00000019u, 0x0000009au, 0x00000099u, + 0x000300f7u, 0x0000009cu, 0x00000000u, 0x000400fau, 0x0000009au, 0x0000009bu, 0x0000009cu, 0x000200f8u, + 0x0000009bu, 0x0004003du, 0x00000008u, 0x0000009du, 0x0000008du, 0x0004003du, 0x00000008u, 0x0000009eu, + 0x00000093u, 0x000500aau, 0x00000019u, 0x0000009fu, 0x0000009du, 0x0000009eu, 0x000300f7u, 0x000000a1u, + 0x00000000u, 0x000400fau, 0x0000009fu, 0x000000a0u, 0x000000a1u, 0x000200f8u, 0x000000a0u, 0x0004003du, + 0x00000008u, 0x000000a2u, 0x00000089u, 0x000500c7u, 0x00000008u, 0x000000a3u, 0x000000a2u, 0x00000035u, + 0x000500abu, 0x00000019u, 0x000000a4u, 0x000000a3u, 0x00000023u, 0x000200f9u, 0x000000a1u, 0x000200f8u, + 0x000000a1u, 0x000700f5u, 0x00000019u, 0x000000a5u, 0x0000009fu, 0x0000009bu, 0x000000a4u, 0x000000a0u, + 0x000200f9u, 0x0000009cu, 0x000200f8u, 0x0000009cu, 0x000700f5u, 0x00000019u, 0x000000a6u, 0x00000099u, + 0x0000007eu, 0x000000a5u, 0x000000a1u, 0x000300f7u, 0x000000a8u, 0x00000000u, 0x000400fau, 0x000000a6u, + 0x000000a7u, 0x000000a8u, 0x000200f8u, 0x000000a7u, 0x0004003du, 0x00000008u, 0x000000a9u, 0x00000089u, + 0x00050080u, 0x00000008u, 0x000000aau, 0x000000a9u, 0x00000035u, 0x0003003eu, 0x00000089u, 0x000000aau, + 0x000200f9u, 0x000000a8u, 0x000200f8u, 0x000000a8u, 0x0004003du, 0x00000008u, 0x000000abu, 0x00000042u, + 0x0004003du, 0x00000008u, 0x000000acu, 0x00000089u, 0x000500c5u, 0x00000008u, 0x000000adu, 0x000000abu, + 0x000000acu, 0x000200feu, 0x000000adu, 0x000200f8u, 0x00000079u, 0x0004003du, 0x00000029u, 0x000000b0u, + 0x0000004eu, 0x0004007cu, 0x00000008u, 0x000000b1u, 0x000000b0u, 0x000500c4u, 0x00000008u, 0x000000b3u, + 0x000000b1u, 0x000000b2u, 0x0004003du, 0x00000008u, 0x000000b4u, 0x00000055u, 0x000500c2u, 0x00000008u, + 0x000000b5u, 0x000000b4u, 0x00000066u, 0x000500c5u, 0x00000008u, 0x000000b6u, 0x000000b3u, 0x000000b5u, + 0x0003003eu, 0x000000afu, 0x000000b6u, 0x0004003du, 0x00000008u, 0x000000b8u, 0x00000055u, 0x000500c7u, + 0x00000008u, 0x000000bau, 0x000000b8u, 0x000000b9u, 0x0003003eu, 0x000000b7u, 0x000000bau, 0x0004003du, + 0x00000008u, 0x000000bbu, 0x000000b7u, 0x000500acu, 0x00000019u, 0x000000bdu, 0x000000bbu, 0x000000bcu, + 0x000400a8u, 0x00000019u, 0x000000beu, 0x000000bdu, 0x000300f7u, 0x000000c0u, 0x00000000u, 0x000400fau, + 0x000000beu, 0x000000bfu, 0x000000c0u, 0x000200f8u, 0x000000bfu, 0x0004003du, 0x00000008u, 0x000000c1u, + 0x000000b7u, 0x000500aau, 0x00000019u, 0x000000c2u, 0x000000c1u, 0x000000bcu, 0x000300f7u, 0x000000c4u, + 0x00000000u, 0x000400fau, 0x000000c2u, 0x000000c3u, 0x000000c4u, 0x000200f8u, 0x000000c3u, 0x0004003du, + 0x00000008u, 0x000000c5u, 0x000000afu, 0x000500c7u, 0x00000008u, 0x000000c6u, 0x000000c5u, 0x00000035u, + 0x000500abu, 0x00000019u, 0x000000c7u, 0x000000c6u, 0x00000023u, 0x000200f9u, 0x000000c4u, 0x000200f8u, + 0x000000c4u, 0x000700f5u, 0x00000019u, 0x000000c8u, 0x000000c2u, 0x000000bfu, 0x000000c7u, 0x000000c3u, + 0x000200f9u, 0x000000c0u, 0x000200f8u, 0x000000c0u, 0x000700f5u, 0x00000019u, 0x000000c9u, 0x000000bdu, + 0x00000079u, 0x000000c8u, 0x000000c4u, 0x000300f7u, 0x000000cbu, 0x00000000u, 0x000400fau, 0x000000c9u, + 0x000000cau, 0x000000cbu, 0x000200f8u, 0x000000cau, 0x0004003du, 0x00000008u, 0x000000ccu, 0x000000afu, + 0x00050080u, 0x00000008u, 0x000000cdu, 0x000000ccu, 0x00000035u, 0x0003003eu, 0x000000afu, 0x000000cdu, + 0x000200f9u, 0x000000cbu, 0x000200f8u, 0x000000cbu, 0x0004003du, 0x00000008u, 0x000000ceu, 0x00000042u, + 0x0004003du, 0x00000008u, 0x000000cfu, 0x000000afu, 0x000500c5u, 0x00000008u, 0x000000d0u, 0x000000ceu, + 0x000000cfu, 0x000200feu, 0x000000d0u, 0x00010038u, 0x00050036u, 0x00000008u, 0x00000014u, 0x00000000u, + 0x00000011u, 0x00030037u, 0x00000007u, 0x00000012u, 0x00030037u, 0x00000010u, 0x00000013u, 0x000200f8u, + 0x00000015u, 0x0004003bu, 0x00000010u, 0x000000d5u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000d8u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000dcu, 0x00000007u, 0x0004003du, 0x00000008u, 0x000000d3u, + 0x00000013u, 0x000500aau, 0x00000019u, 0x000000d4u, 0x000000d3u, 0x00000035u, 0x000300f7u, 0x000000d7u, + 0x00000000u, 0x000400fau, 0x000000d4u, 0x000000d6u, 0x000000dbu, 0x000200f8u, 0x000000d6u, 0x0004003du, + 0x00000006u, 0x000000d9u, 0x00000012u, 0x0003003eu, 0x000000d8u, 0x000000d9u, 0x00050039u, 0x00000008u, + 0x000000dau, 0x0000000eu, 0x000000d8u, 0x0003003eu, 0x000000d5u, 0x000000dau, 0x000200f9u, 0x000000d7u, + 0x000200f8u, 0x000000dbu, 0x0004003du, 0x00000006u, 0x000000ddu, 0x00000012u, 0x0003003eu, 0x000000dcu, + 0x000000ddu, 0x00050039u, 0x00000008u, 0x000000deu, 0x0000000bu, 0x000000dcu, 0x0003003eu, 0x000000d5u, + 0x000000deu, 0x000200f9u, 0x000000d7u, 0x000200f8u, 0x000000d7u, 0x0004003du, 0x00000008u, 0x000000dfu, + 0x000000d5u, 0x000200feu, 0x000000dfu, 0x00010038u, }; constexpr uint32_t kSpv_vt_matmul_vec[] = { diff --git a/tests/vt/test_vulkan_backend.cpp b/tests/vt/test_vulkan_backend.cpp index 24b1618c6..004aec4f6 100644 --- a/tests/vt/test_vulkan_backend.cpp +++ b/tests/vt/test_vulkan_backend.cpp @@ -991,6 +991,76 @@ TEST_CASE("a REFERENCE-TIER op drains the batch before it touches device memory" vk.DestroyQueue(q); } +TEST_CASE("a RAGGED-M GEMM takes coopmat and is exact -- the shifted trailing tile") { + if (!VulkanPresent()) return; + auto& ctx = vt::vulkan::VulkanContext::Get(); + if (!ctx.coopmat_bf16_f32() || ctx.subgroup_size() != 32) return; // scalar device + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Queue q = vk.CreateQueue(); + const Device d{DeviceType::kVULKAN, 0}; + + // M = 17 IS THE SHAPE THAT COST A 100x. Prompt length gives m = tokens + 1, so + // every prefill M is one past a tile boundary: 513 % 16 == 1. The old predicate + // required m % 16 == 0, so every 27B prefill GEMM fell to the untiled scalar + // kernel -- measured at 99.9% of GPU time and ~96 GFLOP/s. + // + // The trailing tile now slides back to start at M-16, so it reads only real + // rows. Rows 1..15 are therefore computed TWICE, by both tiles, and this test + // exists to prove those duplicate writes are bit-identical rather than merely + // close: a row's result depends only on that row of A and on B. + constexpr int64_t kM = 17, kK = 32, kN = 32; + + std::vector a(kM * kK), b(kN * kK); + for (int64_t i = 0; i < kM * kK; ++i) + a[static_cast(i)] = vt::F32ToBF16(0.5f * static_cast((i % 7) - 3)); + for (int64_t i = 0; i < kN * kK; ++i) + b[static_cast(i)] = vt::F32ToBF16(0.25f * static_cast((i % 5) - 2)); + + void* da = vk.Alloc(a.size() * sizeof(uint16_t)); + void* db = vk.Alloc(b.size() * sizeof(uint16_t)); + auto* dout = static_cast(vk.Alloc(kM * kN * sizeof(float))); + vk.Copy(q, da, a.data(), a.size() * sizeof(uint16_t)); + vk.Copy(q, db, b.data(), b.size() * sizeof(uint16_t)); + vk.Synchronize(q); + + Tensor ta = Tensor::Contiguous(da, vt::DType::kBF16, d, {kM, kK}); + Tensor tb = Tensor::Contiguous(db, vt::DType::kBF16, d, {kN, kK}); + Tensor to = Tensor::Contiguous(dout, vt::DType::kF32, d, {kM, kN}); + vt::MatmulBT(q, to, ta, tb); + vk.Synchronize(q); + + // THE TACTIC IS THE LOAD-BEARING ASSERTION. The scalar kernel is equally + // correct, so numbers alone cannot show that a ragged M now reaches the tensor + // cores -- which is the entire point of the change. + CHECK(ctx.PipelineExistsFor("vt_matmul_coopmat")); + + std::vector got(static_cast(kM * kN)); + vk.Copy(q, got.data(), dout, got.size() * sizeof(float)); + vk.Synchronize(q); + + // Exact against a host oracle in f64. Row 16 (only the shifted tile) and rows + // 1..15 (BOTH tiles) must agree; a wrong shift shows up as a wrong or + // duplicated row rather than as noise. + for (int64_t i = 0; i < kM; ++i) { + for (int64_t j = 0; j < kN; ++j) { + double acc = 0.0; + for (int64_t c = 0; c < kK; ++c) { + acc += static_cast(vt::BF16ToF32(a[static_cast(i * kK + c)])) * + static_cast(vt::BF16ToF32(b[static_cast(j * kK + c)])); + } + CAPTURE(i); + CAPTURE(j); + CHECK(got[static_cast(i * kN + j)] == + doctest::Approx(static_cast(acc)).epsilon(1e-3)); + } + } + + vk.Free(da); + vk.Free(db); + vk.Free(dout); + vk.DestroyQueue(q); +} + TEST_CASE("Vulkan float-controls are PROBED and reported, not assumed") { if (!VulkanPresent()) return; // The relaxed-precision knobs Vulkan leaves implementation-defined. We cannot