From 93852c28350bf9b0e02e5531ea12a2bbb55ba4e2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 9 Aug 2026 02:03:16 +0000 Subject: [PATCH 1/2] perf(vulkan): fused attn preamble native -- 16 host round trips/token gone kAttnQkNormRopeGate (gemma-RMSNorm(q) + gemma-RMSNorm(k) + partial NeoX RoPE from the precomputed cos/sin cache + gate passthrough) gets a native Vulkan kernel. Module count 24 -> 25; the Vulkan reference-tier decline list is now exactly kRopeCosSinCache and kCausalConv1dFwd, both host-side by design. WHY THIS OP. After the GDN rows there was no kernel-speed lever left in 27B decode: vt_matmul_vec runs at 90% of the GB10 bandwidth roof and lm_head at 74%. What remained was the reference tier's ARCHITECTURAL cost. src/vt/op_provider.cpp drains the recorded command batch (submit + blocking fence) before it can hand a host kernel device memory, so a reference-tier op costs a full GPU round trip no matter how little arithmetic it does. Of the three ops still declining, kCausalConv1dFwd is prefill-only and kRopeCosSinCache is deliberately on the host (the double-precision angle table, vLLM's own split), which left this one. The 27B has 64 layers of which 48 are linear-attention, so it fires 16x per token. PORTED FROM. Per-element math 1:1 from our own CPU reference src/vt/cpu/cpu_ops.cpp:956-1010 AttnQkNormRopeGateKernel (the transcription of vLLM's QKNormRoPEFusionPass -> _C.fused_qk_norm_rope). Dispatch shape from our CUDA kernel src/vt/cuda/cuda_ops.cu:1316-1394: one workgroup per (token, head) over Hq + Hkv slots, CUDA's dim3(t, hq+hkv) grid flattened, with the two paired normed elements recomputed per output rather than staged in shared memory. The row reduction is vt_rms_norm_gated.comp's idiom; the cos/sin indexing is vt_rope_from_cache.comp's, minus the positions indirection, because this cache is the per-step [T,rot] fill rather than the model-wide table. MEASURED, GB10, Qwen3.6-27B bf16, 1 prompt, 32-in, c1, page cache dropped before every run. Two-length flush diff (output-len 4 vs 12), both arms measured: per decoded token main this row reference-tier drains 16 0 command-buffer flushes 18 4 GPU dispatches 884 900 GPU-active 236.3 ms 234.6 ms wall (median TPOT) 253.5 ms 246.2 ms host time (wall - GPU) 17.2 ms 11.6 ms The GPU does the same work to within noise and takes four MORE dispatches per token; what changes is that 16 submit + blocking-fence round trips per token stop happening. Paired decode, 6 order-alternated AB/BA pairs at 32-in/32-out: this row wins 5 of 6, median TPOT over the non-outlier legs 249.74 -> 242.34 ms, i.e. decode 4.00 -> 4.13 tok/s against llama.cpp Vulkan's 4.35. Two legs (one per arm) landed at ~1.8x the block median; discarding the pairs containing them leaves 4 of 4 and the same ~3%. CORRECTNESS. NMSE vs the CPU oracle in the same binary on the GB10 NVIDIA driver: q 5.69e-15 / k 1.59e-14 (gemma=1), 6.5e-15 / 1.43e-14 (gemma=0); the bf16-q/k + f32-gate arm is bit-exact. The f32 arm is NOT bit-exact and is not claimed to be: the workgroup tree reduction changes the mean square's accumulation order, the tier vt_rms_norm and vt_rms_norm_gated already sit in. The gate passthrough IS held bit-exact, and against the SOURCE rather than the oracle's output, so a kernel that copied the q half into both halves cannot pass. Two new cases, both asserting the MECHANISM as well as the numbers (PipelineExistsFor + last_selected == vt-native), on deliberately awkward shapes: Hq=5 / Hkv=2 ragged, Dh=160 past the 128-wide workgroup, rot=96 < Dh so partial RoPE is exercised, and both source rows as padded packed-projection views. GATES. test_vulkan_backend 29/29 (2329 assertions) on GB10 and 29/29 (1786) on llvmpipe; test_opt_paged_engine with VLLM_CPP_DEVICE=vulkan still 6/6 token-exact (96/96) with 0 declines on both. gen-vulkan-spirv.py --check clean at pinned glslang 16.5.0. Local ctest: the 6 failures in a Vulkan-ON build reproduce identically on pristine origin/main and are not from this change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code] --- .agents/specs/vulkan-full-support.md | 75 +- docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 10 +- docs/STATUS.md | 6 +- .../shaders/vt_attn_qk_norm_rope_gate.comp | 186 +++++ src/vt/vulkan/vulkan_ops.cpp | 78 ++ src/vt/vulkan/vulkan_spirv.cpp | 723 ++++++++++++++++++ tests/vt/test_vulkan_backend.cpp | 319 +++++++- 8 files changed, 1387 insertions(+), 12 deletions(-) create mode 100644 src/vt/vulkan/shaders/vt_attn_qk_norm_rope_gate.comp diff --git a/.agents/specs/vulkan-full-support.md b/.agents/specs/vulkan-full-support.md index 40ae741f0..1009a5ed7 100644 --- a/.agents/specs/vulkan-full-support.md +++ b/.agents/specs/vulkan-full-support.md @@ -327,11 +327,84 @@ the umbrella, not a substitute for them. | **VK-H** | **Attention variants + samplers** (16 ops) | B (samplers), G (attn variants) | **83/83 — closes the op surface** | | **VK-I** | **AMD/RDNA (or Arc) bring-up** | hardware acquisition | The staging path for non-host-visible memory, and the gate re-run where Vulkan actually matters | +### 6.0a `VK-G` partial: the FUSED ATTN PREAMBLE landed — 2026-08-09 + +`row/BACKEND-VULKAN-QKNORM`. `kAttnQkNormRopeGate` — gemma-RMSNorm(q) + +gemma-RMSNorm(k) + partial NeoX RoPE from the precomputed cos/sin cache + the +gate passthrough — is native. Module count 24 -> 25. + +**Why this op, and only this op.** After the GDN rows there was no kernel-speed +lever left in decode: `vt_matmul_vec` runs at 90% of the GB10 bandwidth roof and +`lm_head` at 74%. What remained was the reference tier's ARCHITECTURAL cost — +`src/vt/op_provider.cpp` drains the recorded command batch (submit + blocking +fence) before it can hand a host kernel device memory, so a reference-tier op +costs a full GPU round trip no matter how little arithmetic it does. Three ops +declined; `kCausalConv1dFwd` is prefill-only and `kRopeCosSinCache` is +deliberately host-side (the double-precision table, vLLM's own split), which left +this one. The 27B has 64 layers of which 48 are linear-attention, so it fires 16 +times per decoded token. + +**Ported from.** Per-element math 1:1 from our own CPU reference +`src/vt/cpu/cpu_ops.cpp:956-1010 AttnQkNormRopeGateKernel` (itself the transcription of +vLLM's `QKNormRoPEFusionPass` -> `_C.fused_qk_norm_rope`); dispatch shape from our +CUDA kernel `src/vt/cuda/cuda_ops.cu:1316-1394` — one workgroup per (token, head) +over Hq + Hkv slots, CUDA's `dim3(t, hq+hkv)` grid flattened, with the two paired +normed elements RECOMPUTED per output rather than staged in shared memory. The +reduction idiom is `vt_rms_norm_gated.comp`'s and the cos/sin indexing is +`vt_rope_from_cache.comp`'s, minus the positions indirection: this cache is the +per-step `[T,rot]` fill, so the row is `tok * rot`. + +**MEASURED on GB10 (Qwen3.6-27B bf16, Vulkan, 1 prompt, 32-in, c1, page cache +dropped before every run).** By the two-length flush diff (output-len 4 vs 12, +both arms measured, neither inferred): + +| per decoded token | main | this row | +|---|---:|---:| +| reference-tier drains | 16 | **0** | +| command-buffer flushes, all reasons | 18 | **4** | +| GPU dispatches | 884 | 900 | +| GPU-active | 236.3 ms | 234.6 ms | +| wall (median TPOT) | 253.5 ms | 246.2 ms | +| host time, wall minus GPU-active | 17.2 ms | **11.6 ms** | + +That is the whole mechanism in one table. The GPU does the SAME work to within +noise and takes four more dispatches per token; what changes is that 16 submit + +blocking-fence round trips per token stop happening. The four remaining flushes +are the batcher's own copy-src and descriptor-ring-full drains, not provider +declines, and the reference-tier decline list is now exactly `kRopeCosSinCache` +and `kCausalConv1dFwd`, both by design. + +**PAIRED DECODE**, 6 order-alternated AB/BA pairs at 32-in/32-out, page cache +dropped before each of the 12 runs: **this row wins 5 of 6**. Median TPOT over +the non-outlier legs 249.74 ms -> 242.34 ms, i.e. decode **4.00 -> 4.13 tok/s** +against llama.cpp Vulkan's 4.35. Two legs (one per arm, pairs 2 and 3) came in at +~446-448 ms, 1.8x the block median; discarding the pairs that contain them leaves +**4 of 4**, and it is the same ~3% either way. + +**HONESTY ON THE SPEED NUMBER.** The effect is ~3% and this box swings ~2x run to +run, which is why the readout is a per-pair win rate and not a mean. It is also +why the first paired block of this row was thrown away: the host had rebooted and +brought `local-ai-worker` back up under `--restart=always`, and a contended block +reads ~450-505 ms against a quiet box's ~250 ms, roughly 14x the effect being +measured. The mechanism table above is the load-bearing evidence — the host-time +column measures the thing this row removes directly and does not depend on the +wall clock being reproducible. + +**Correctness.** NMSE vs the CPU oracle in the same binary on the GB10 NVIDIA +driver: q `5.69e-15`, k `1.59e-14` (gemma=1); `6.5e-15` / `1.43e-14` (gemma=0); +the bf16-q/k + f32-gate arm is BIT-EXACT. The f32 arm is NOT bit-exact and is not +claimed to be — the workgroup tree reduction changes the mean square's +accumulation order, the same tier `vt_rms_norm` and `vt_rms_norm_gated` sit in. +The gate passthrough IS held bit-exact, against the SOURCE rather than against the +oracle's output. `test_vulkan_backend` 29/29 (2329 assertions) on GB10; +`test_opt_paged_engine` still 6/6 token-exact (96/96), 0 declines. + ### 6.0 `VK-G` partial: the two GATED-DELTA RECURRENCES landed — 2026-08-08 `row/BACKEND-VULKAN-GDN-CORE`. `kGdnPrefill` and `kGdnDecode` are native; the Vulkan module count goes 22 -> 24 and the GDN family 6 -> 8. `kCausalConv1dFwd`, -`kRopeCosSinCache` and `kAttnQkNormRopeGate` stay on the reference tier. +`kRopeCosSinCache` and `kAttnQkNormRopeGate` stay on the reference tier (the last +of the three is closed by §6.0a). **What was ported, and from where.** Per-step arithmetic 1:1 from `src/vt/cpu/cpu_ops.cpp:1280-1311` `GdnHeadTokenStep`; dispatch shape and state diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 650a485fc..eb4f8eb84 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -361,7 +361,7 @@ built on it rather than keeping the flattering one. | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config | | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box | | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs | -| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 24 NATIVE (+8 GDN); 63 host-tier. **27B prefill 21.5x**, decode 2.99->**3.93** (GDN state in place, 98->2 copy flushes/tok, 8/8 paired; target 4.35). opt-125m exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | +| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN); 62 host-tier. **27B prefill 21.5x**, decode 4.00->**4.13** (fused attn preamble native, flushes 18->4/tok, 5/6 pairs; target 4.35). opt-125m exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 4915d9fb1..09c473dca 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -215,11 +215,11 @@ kernels for it. Vulkan **runs a model end to end**: `opt-125m` greedy is STRICT token-exact, 6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle, all nine of that model's ops dispatched natively with **zero provider declines**. Qwen3.6-27B runs too, both -GDN recurrences native and the GDN state cache read and written IN PLACE rather -than gathered to a host copy per layer per token: **decode 3.93 tok/s against -llama.cpp Vulkan's 4.35, prefill 21.5x** (GB10). Still partial at 24 native -kernels plus 8 GDN, the rest on the portable CPU tier (decode still pays for -`kRopeCosSinCache` and `kAttnQkNormRopeGate`); quant/MoE/MLA have none at all. +GDN recurrences and the fused attention preamble native, its GDN state cache read +and written IN PLACE: **decode 4.13 tok/s against llama.cpp Vulkan's 4.35, +prefill 21.5x** (GB10). Still partial at 25 native kernels plus 8 GDN, the rest +on the portable CPU tier (decode's one remaining host op, `kRopeCosSinCache`, +mirrors vLLM's own split and stays there); quant/MoE/MLA have none at all. Build with `-DVLLM_CPP_VULKAN=ON`; off by default. ## Serving, API and operations diff --git a/docs/STATUS.md b/docs/STATUS.md index bb6a69649..a0661aa7e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -420,9 +420,9 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact ## Not supported yet LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU, -Vulkan (opt-125m exact; 24 native +8 GDN, BOTH -recurrences; 27B prefill 21.5x, decode 3.93/4.35 llama.cpp (GDN state -in place, 98->2 copy flushes/tok); #125 VERIFIED +Vulkan (opt-125m exact; 25 native +8 GDN, both +recurrences + fused attn preamble; 27B prefill 21.5x, decode +4.13/4.35 llama.cpp, 18->4 flushes/tok; #125 VERIFIED [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in (**blind-written, unverified**); M2 needs verification; gfx1201 hipBLAS + diff --git a/src/vt/vulkan/shaders/vt_attn_qk_norm_rope_gate.comp b/src/vt/vulkan/shaders/vt_attn_qk_norm_rope_gate.comp new file mode 100644 index 000000000..e0ed0e68d --- /dev/null +++ b/src/vt/vulkan/shaders/vt_attn_qk_norm_rope_gate.comp @@ -0,0 +1,186 @@ +#version 450 +// vt::AttnQkNormRopeGate — the FUSED FULL-ATTENTION PREAMBLE: split q|gate + +// per-head gemma-RMSNorm(q) and gemma-RMSNorm(k) + partial NeoX RoPE from a +// precomputed cos/sin cache + gate passthrough, in ONE dispatch. +// +// UPSTREAM: vLLM's QKNormRoPEFusionPass +// (vllm/compilation/passes/fusion/qk_norm_rope_fusion.py:188 -> _C.fused_qk_norm_rope, +// itself fla `fused_qk_norm_rope.py:95-102`), the same citation the recipe carries +// (include/vt/recipes.h:231-247). +// +// NUMERIC ORACLE: our own CPU reference AttnQkNormRopeGateKernel +// (src/vt/cpu/cpu_ops.cpp:956-1010), transcribed element for element — the +// GemmaNormElem grouping `v * inv * wj` with `wj = w [+1 if gemma]` +// (cpu_ops.cpp:944-948), the rotation spelled `ni*c - nih*sn` / `ni*sn + nih*c` +// (:980, :986), and the tail dims [rot, Dh) normed but UNROTATED (:988). +// +// DISPATCH SHAPE ported from the CUDA kernel (src/vt/cuda/cuda_ops.cu:1316-1383): +// ONE WORKGROUP PER (token, head) over Hq + Hkv slots — CUDA's dim3(t, hq+hkv) +// grid (:1394), flattened here because Vulkan's workgroup count is consumed as a +// single x extent by VulkanContext::Dispatch. The q heads occupy [0, Hq) and the +// kv heads [Hq, Hq+Hkv), exactly as CUDA's `is_q = head < hq` split. The workgroup +// tree-reduces the head row's mean square (vt_common.glsl vt_tg_sum, the idiom +// vt_rms_norm_gated.comp already uses) and then RECOMPUTES the two paired normed +// elements per output rather than round-tripping them through shared memory — +// CUDA's choice at :1359-1361, kept so the two devices compute the same +// expression. +// +// THIS IS A FUSION OF TWO KERNELS THAT ALREADY WORK. The reduction half is +// vt_rms_norm_gated.comp:51-62; the cos/sin indexing is vt_rope_from_cache.comp, +// with ONE difference that matters: there the cache is indexed by +// `position * rotary_dim` because the table is the model-wide [P, rot] cache, +// while HERE the cache is the per-step [T, rot] fill that kRopeCosSinCache +// produced for exactly these tokens (ops.cpp:1533-1535 validates +// cos_sin.shape[0] == T), so the row is `tok * rot` — no positions operand at +// all. cpu_ops.cpp:995 does the same. +// +// WHY THE CACHE IS STILL BUILT ON THE HOST: kRopeCosSinCache constructs its +// angles in DOUBLE (cpu_ops.cpp:932-936) and this is vLLM's own split — see the +// argument in vt_rope_from_cache.comp. This kernel is the apply side of it. +// +// TIER: NMSE, not bit-exact. The recipe calls the standalone op a byte-exact +// golden against the composed RmsNorm+RmsNorm+RopeFromCache+gate chain +// (recipes.h:237), and that is still true device-locally, but the workgroup tree +// reduction changes the ACCUMULATION ORDER of the mean square relative to the CPU +// tier's sequential loop — the same trade vt_rms_norm and vt_rms_norm_gated +// already make. The scale is `1.0 / sqrt(...)`, NOT `inversesqrt` +// (vt_common.glsl § DELIBERATE DIVERGENCES). +#extension GL_GOOGLE_include_directive : require +#include "vt_common.glsl" + +layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in; + +// qgate / kf: the two SOURCE rows, dtype-erased pairs (§ STORAGE MODEL). +layout(binding = 0) readonly buffer QGb32 { uint v[]; } QG32; +layout(binding = 1) readonly buffer QGb16 { uint16_t v[]; } QG16; +layout(binding = 2) readonly buffer KFb32 { uint v[]; } KF32; +layout(binding = 3) readonly buffer KFb16 { uint16_t v[]; } KF16; +// q_out / k_out / gate_out. +layout(binding = 4) buffer QOb32 { uint v[]; } QO32; +layout(binding = 5) buffer QOb16 { uint16_t v[]; } QO16; +layout(binding = 6) buffer KOb32 { uint v[]; } KO32; +layout(binding = 7) buffer KOb16 { uint16_t v[]; } KO16; +layout(binding = 8) buffer GOb32 { uint v[]; } GO32; +layout(binding = 9) buffer GOb16 { uint16_t v[]; } GO16; +// q_norm / k_norm / cos_sin are f32 BY CONTRACT (src/vt/ops.cpp:1539-1541), so +// they take ONE binding each rather than a dtype-erased pair whose 16-bit half +// could never be taken — the arrangement vt_gdn_post_conv already uses for its +// f32-by-contract operands. +layout(binding = 10) readonly buffer QNb { uint v[]; } QN; +layout(binding = 11) readonly buffer KNb { uint v[]; } KN; +layout(binding = 12) readonly buffer CSb { uint v[]; } CS; + +// The three dtype axes the op contract actually admits (ops.cpp:1524-1538): +// qgate and kf share ONE dtype; q_out and k_out share ONE dtype; gate_out is +// either that same dtype or f32 while q/k are bf16 (the FA-2 prefill combo), so +// it is its own axis. q_norm/k_norm/cos_sin are f32 and are NOT axes. +layout(constant_id = 0) const uint VT_QKN_SRC_DT = VT_DT_F32; +layout(constant_id = 1) const uint VT_QKN_QK_DT = VT_DT_F32; +layout(constant_id = 2) const uint VT_QKN_GATE_DT = VT_DT_F32; + +layout(push_constant) uniform Params { + uint hq; // q heads + uint hkv; // kv heads + uint dh; // head dim + uint rot; // rotary_dim (<= dh) + uint half_dim; // rot / 2 + uint qgate_rs; // qgate.stride[0], in ELEMENTS (>= hq*2*dh: a packed view) + uint kf_rs; // kf.stride[0], in ELEMENTS + uint gemma; // RmsNormArgs::gemma — weight is (1+w) when set + uint qg_off, kf_off, qo_off, ko_off, go_off, qn_off, kn_off, cs_off; + float eps; +} p; + +// cpu_ops.cpp:944-948 GemmaNormElem. The +1 rides a UNIFORM branch, not +// `w + float(gemma)`: `-0.0 + 0.0` is `+0.0`, so the additive spelling would +// flip the sign of a zero weight's output bits on the gemma=false path. +float qkn_elem(float v, float inv, float w) { + float wj = p.gemma != 0u ? w + 1.0 : w; + return v * inv * wj; +} + +float qkn_src(bool is_q, uint idx) { + return is_q ? VT_LOAD(QG32, QG16, VT_QKN_SRC_DT, p.qg_off, idx) + : VT_LOAD(KF32, KF16, VT_QKN_SRC_DT, p.kf_off, idx); +} + +float qkn_w(bool is_q, uint j) { + return is_q ? uintBitsToFloat(QN.v[(p.qn_off >> 2) + j]) + : uintBitsToFloat(KN.v[(p.kn_off >> 2) + j]); +} + +void qkn_store(bool is_q, uint idx, float v) { + if (is_q) { + VT_STORE(QO32, QO16, VT_QKN_QK_DT, p.qo_off, idx, v); + } else { + VT_STORE(KO32, KO16, VT_QKN_QK_DT, p.ko_off, idx, v); + } +} + +void main() { + uint heads = p.hq + p.hkv; + uint wg = gl_WorkGroupID.x; + uint tid = gl_LocalInvocationID.x; + // Token-major so consecutive workgroups walk consecutive heads of ONE source + // row: the q heads of a token are adjacent in qgate. + uint tok = wg / heads; + uint head = wg % heads; + bool is_q = head < p.hq; + + // cuda_ops.cu:1331-1342. Every one of these is workgroup-uniform. + uint src_base; + uint out_base; + uint gate_base = 0u; + if (is_q) { + uint qrow = tok * p.qgate_rs + head * 2u * p.dh; + src_base = qrow; // q half [0, dh) + gate_base = qrow + p.dh; // gate half [dh, 2dh) + out_base = (tok * p.hq + head) * p.dh; + } else { + uint hk = head - p.hq; + src_base = tok * p.kf_rs + hk * p.dh; + out_base = (tok * p.hkv + hk) * p.dh; + } + + // ---- gemma-RMSNorm reduction over Dh, f32 variance (cpu_ops.cpp:969-974) ---- + float partial = 0.0; + for (uint j = tid; j < p.dh; j += VT_TG) { + float v = qkn_src(is_q, src_base + j); + partial += v * v; + } + float inv = 1.0 / sqrt(vt_tg_sum(tid, partial) / float(p.dh) + p.eps); + + // ---- partial NeoX RoPE + write (cpu_ops.cpp:975-991) ---- + uint cs = (p.cs_off >> 2) + tok * p.rot; + for (uint j = tid; j < p.dh; j += VT_TG) { + if (j < p.half_dim) { + float ni = qkn_elem(qkn_src(is_q, src_base + j), inv, qkn_w(is_q, j)); + float nih = qkn_elem(qkn_src(is_q, src_base + j + p.half_dim), inv, + qkn_w(is_q, j + p.half_dim)); + float c = uintBitsToFloat(CS.v[cs + j]); + float sn = uintBitsToFloat(CS.v[cs + p.half_dim + j]); + qkn_store(is_q, out_base + j, ni * c - nih * sn); + } else if (j < p.rot) { + uint i = j - p.half_dim; + float ni = qkn_elem(qkn_src(is_q, src_base + i), inv, qkn_w(is_q, i)); + float nih = qkn_elem(qkn_src(is_q, src_base + i + p.half_dim), inv, + qkn_w(is_q, i + p.half_dim)); + float c = uintBitsToFloat(CS.v[cs + i]); + float sn = uintBitsToFloat(CS.v[cs + p.half_dim + i]); + qkn_store(is_q, out_base + j, ni * sn + nih * c); + } else { + // Tail dims are normed but NOT rotated (cpu_ops.cpp:988-989). + qkn_store(is_q, out_base + j, + qkn_elem(qkn_src(is_q, src_base + j), inv, qkn_w(is_q, j))); + } + } + + // ---- gate passthrough, q heads only: the raw second Dh of the q|gate pair, + // no norm and no rope (cpu_ops.cpp:999-1002, cuda_ops.cu:1378-1382) ---- + if (is_q) { + for (uint j = tid; j < p.dh; j += VT_TG) { + VT_STORE(GO32, GO16, VT_QKN_GATE_DT, p.go_off, out_base + j, + VT_LOAD(QG32, QG16, VT_QKN_SRC_DT, p.qg_off, gate_base + j)); + } + } +} diff --git a/src/vt/vulkan/vulkan_ops.cpp b/src/vt/vulkan/vulkan_ops.cpp index c73d45bf0..da8534aec 100644 --- a/src/vt/vulkan/vulkan_ops.cpp +++ b/src/vt/vulkan/vulkan_ops.cpp @@ -215,6 +215,12 @@ struct GdnPostConvParams { uint32_t g_off, beta_off, alog_off, dtb_off; float eps; }; +// The fused full-attention preamble (BACKEND-VULKAN-QKNORM). +struct AttnQkNormRopeGateParams { + uint32_t hq, hkv, dh, rot, half_dim, qgate_rs, kf_rs, gemma; + uint32_t qg_off, kf_off, qo_off, ko_off, go_off, qn_off, kn_off, cs_off; + float eps; +}; // ONE block for BOTH recurrences: vt_gdn_prefill and vt_gdn_decode share their // step body through an include, so they must also share their push layout. The // prefill shader ignores has_idx / n_state_rows (they are the decode state-row @@ -245,6 +251,8 @@ static_assert(sizeof(GdnStateGatherParams) <= 128, "push constants must fit the guaranteed 128 bytes"); static_assert(sizeof(GdnRecurrenceParams) <= 128, "push constants must fit the guaranteed 128 bytes"); +static_assert(sizeof(AttnQkNormRopeGateParams) <= 128, + "push constants must fit the guaranteed 128 bytes"); template void Go(const char* name, const Binder& b, const P& p, uint32_t groups, @@ -1308,6 +1316,71 @@ void GdnDecodeKernel(Queue& q, Tensor& out, const Tensor& q_in, const Tensor& k, Go("vt_gdn_decode", bind, p, static_cast(batch * hv * p.nv), spec, 2); } +// --------------------------------------------------------------------------- +// The FUSED FULL-ATTENTION PREAMBLE (BACKEND-VULKAN-QKNORM). +// +// WHY THIS OP AND NOT ANOTHER. With the GDN family native, a 27B decode step was +// MEASURED at ~30 command-buffer flushes per token of which ~28 were +// reference-tier: op_provider.cpp drains the whole recorded batch (submit + +// blocking fence) before it can hand a host kernel device memory, so every +// reference-tier op costs a full round trip regardless of how little arithmetic +// it does. The declines named exactly three ops, and only this one runs in +// DECODE on every step: kCausalConv1dFwd is prefill-only and kRopeCosSinCache is +// deliberately host-side (the double-precision table; see vt_rope_from_cache.comp). +// Qwen3.6-27B has 64 layers of which 48 are linear-attention, so this fires 16 +// times per token. +// +// NO DECLINE PATH. Every dtype the op contract admits (ops.cpp:1524-1541) is a +// specialization axis here, the packed row strides are read from stride[0], and +// rotary_dim < Dh is the ordinary case rather than a special one — so there is no +// shape this kernel has to hand back. cpu_ops.cpp:956-1010 is the oracle. +void AttnQkNormRopeGateKernel(Queue&, Tensor& q_out, Tensor& k_out, Tensor& gate_out, + const Tensor& qgate, const Tensor& kf, const Tensor& q_norm, + const Tensor& k_norm, const Tensor& cos_sin, + const RmsNormArgs& na, const RopeArgs& ra) { + const int64_t t = q_out.shape[0], hq = q_out.shape[1], dh = q_out.shape[2]; + const int64_t hkv = k_out.shape[1]; + if (t == 0 || hq + hkv == 0 || dh == 0) return; + Binder bind; + // Binding order IS the descriptor order: the shader declares qgate, kf, then + // the three outputs as dtype-erased PAIRS, then the three f32-by-contract + // operands as single bindings. + const uint32_t qg_off = bind.Add(qgate, "attn_qk_norm_rope_gate: qgate"); + const uint32_t kf_off = bind.Add(kf, "attn_qk_norm_rope_gate: kf"); + const uint32_t qo_off = bind.Add(q_out, "attn_qk_norm_rope_gate: q_out"); + const uint32_t ko_off = bind.Add(k_out, "attn_qk_norm_rope_gate: k_out"); + const uint32_t go_off = bind.Add(gate_out, "attn_qk_norm_rope_gate: gate_out"); + const uint32_t qn_off = bind.AddU32Only(q_norm, "attn_qk_norm_rope_gate: q_norm"); + const uint32_t kn_off = bind.AddU32Only(k_norm, "attn_qk_norm_rope_gate: k_norm"); + const uint32_t cs_off = bind.AddU32Only(cos_sin, "attn_qk_norm_rope_gate: cos_sin"); + // Ascending constantID order: the shared qgate/kf dtype, the shared q/k out + // dtype, the gate out dtype (its own axis — the FA-2 prefill combo keeps the + // gate f32 while q/k are bf16). + const uint32_t spec[3] = {DtypeCode(qgate.dtype), DtypeCode(q_out.dtype), + DtypeCode(gate_out.dtype)}; + AttnQkNormRopeGateParams p{static_cast(hq), + static_cast(hkv), + static_cast(dh), + static_cast(ra.rotary_dim), + static_cast(ra.rotary_dim / 2), + static_cast(qgate.stride[0]), + static_cast(kf.stride[0]), + na.gemma ? 1u : 0u, + qg_off, + kf_off, + qo_off, + ko_off, + go_off, + qn_off, + kn_off, + cs_off, + na.eps}; + // ONE WORKGROUP PER (token, head) over Hq + Hkv slots — CUDA's dim3(t, hq+hkv) + // grid (cuda_ops.cu:1394) flattened. NOT FlatGroupCount: the workgroup + // tree-reduces one head row's mean square. + Go("vt_attn_qk_norm_rope_gate", bind, p, static_cast(t * (hq + hkv)), spec, 3); +} + struct Registrar { Registrar() { // Same guard as the backend registrar: a Vulkan-enabled build on a host with @@ -1370,6 +1443,11 @@ struct Registrar { reinterpret_cast(static_cast(&GdnPrefillKernel))); RegisterOp(OpId::kGdnDecode, DeviceType::kVULKAN, reinterpret_cast(static_cast(&GdnDecodeKernel))); + // BACKEND-VULKAN-QKNORM: the fused full-attention preamble, the last + // per-decode-step reference-tier op on the 27B. + RegisterOp( + OpId::kAttnQkNormRopeGate, DeviceType::kVULKAN, + reinterpret_cast(static_cast(&AttnQkNormRopeGateKernel))); } } registrar; diff --git a/src/vt/vulkan/vulkan_spirv.cpp b/src/vt/vulkan/vulkan_spirv.cpp index 9affa43bf..1cf727531 100644 --- a/src/vt/vulkan/vulkan_spirv.cpp +++ b/src/vt/vulkan/vulkan_spirv.cpp @@ -349,6 +349,724 @@ constexpr uint32_t kSpv_vt_add[] = { 0x0000014au, 0x00010038u, }; +constexpr uint32_t kSpv_vt_attn_qk_norm_rope_gate[] = { + 0x07230203u, 0x00010300u, 0x0008000bu, 0x00000437u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, + 0x00001151u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, + 0x00000000u, 0x00000001u, 0x0007000fu, 0x00000005u, 0x00000004u, 0x6e69616du, 0x00000000u, 0x0000026fu, + 0x00000274u, 0x00060010u, 0x00000004u, 0x00000011u, 0x00000080u, 0x00000001u, 0x00000001u, 0x00030047u, + 0x00000190u, 0x00000002u, 0x00050048u, 0x00000190u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00050048u, + 0x00000190u, 0x00000001u, 0x00000023u, 0x00000004u, 0x00050048u, 0x00000190u, 0x00000002u, 0x00000023u, + 0x00000008u, 0x00050048u, 0x00000190u, 0x00000003u, 0x00000023u, 0x0000000cu, 0x00050048u, 0x00000190u, + 0x00000004u, 0x00000023u, 0x00000010u, 0x00050048u, 0x00000190u, 0x00000005u, 0x00000023u, 0x00000014u, + 0x00050048u, 0x00000190u, 0x00000006u, 0x00000023u, 0x00000018u, 0x00050048u, 0x00000190u, 0x00000007u, + 0x00000023u, 0x0000001cu, 0x00050048u, 0x00000190u, 0x00000008u, 0x00000023u, 0x00000020u, 0x00050048u, + 0x00000190u, 0x00000009u, 0x00000023u, 0x00000024u, 0x00050048u, 0x00000190u, 0x0000000au, 0x00000023u, + 0x00000028u, 0x00050048u, 0x00000190u, 0x0000000bu, 0x00000023u, 0x0000002cu, 0x00050048u, 0x00000190u, + 0x0000000cu, 0x00000023u, 0x00000030u, 0x00050048u, 0x00000190u, 0x0000000du, 0x00000023u, 0x00000034u, + 0x00050048u, 0x00000190u, 0x0000000eu, 0x00000023u, 0x00000038u, 0x00050048u, 0x00000190u, 0x0000000fu, + 0x00000023u, 0x0000003cu, 0x00050048u, 0x00000190u, 0x00000010u, 0x00000023u, 0x00000040u, 0x00040047u, + 0x000001acu, 0x00000001u, 0x00000000u, 0x00040047u, 0x000001b1u, 0x00000006u, 0x00000004u, 0x00030047u, + 0x000001b2u, 0x00000002u, 0x00040048u, 0x000001b2u, 0x00000000u, 0x00000018u, 0x00050048u, 0x000001b2u, + 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000001b4u, 0x00000018u, 0x00040047u, 0x000001b4u, + 0x00000021u, 0x00000000u, 0x00040047u, 0x000001b4u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001c2u, + 0x00000006u, 0x00000002u, 0x00030047u, 0x000001c3u, 0x00000002u, 0x00040048u, 0x000001c3u, 0x00000000u, + 0x00000018u, 0x00050048u, 0x000001c3u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000001c5u, + 0x00000018u, 0x00040047u, 0x000001c5u, 0x00000021u, 0x00000001u, 0x00040047u, 0x000001c5u, 0x00000022u, + 0x00000000u, 0x00040047u, 0x000001d8u, 0x00000006u, 0x00000004u, 0x00030047u, 0x000001d9u, 0x00000002u, + 0x00040048u, 0x000001d9u, 0x00000000u, 0x00000018u, 0x00050048u, 0x000001d9u, 0x00000000u, 0x00000023u, + 0x00000000u, 0x00030047u, 0x000001dbu, 0x00000018u, 0x00040047u, 0x000001dbu, 0x00000021u, 0x00000002u, + 0x00040047u, 0x000001dbu, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001e6u, 0x00000006u, 0x00000002u, + 0x00030047u, 0x000001e7u, 0x00000002u, 0x00040048u, 0x000001e7u, 0x00000000u, 0x00000018u, 0x00050048u, + 0x000001e7u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000001e9u, 0x00000018u, 0x00040047u, + 0x000001e9u, 0x00000021u, 0x00000003u, 0x00040047u, 0x000001e9u, 0x00000022u, 0x00000000u, 0x00040047u, + 0x000001fdu, 0x00000006u, 0x00000004u, 0x00030047u, 0x000001feu, 0x00000002u, 0x00040048u, 0x000001feu, + 0x00000000u, 0x00000018u, 0x00050048u, 0x000001feu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, + 0x00000200u, 0x00000018u, 0x00040047u, 0x00000200u, 0x00000021u, 0x0000000au, 0x00040047u, 0x00000200u, + 0x00000022u, 0x00000000u, 0x00040047u, 0x0000020au, 0x00000006u, 0x00000004u, 0x00030047u, 0x0000020bu, + 0x00000002u, 0x00040048u, 0x0000020bu, 0x00000000u, 0x00000018u, 0x00050048u, 0x0000020bu, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00030047u, 0x0000020du, 0x00000018u, 0x00040047u, 0x0000020du, 0x00000021u, + 0x0000000bu, 0x00040047u, 0x0000020du, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000220u, 0x00000001u, + 0x00000001u, 0x00040047u, 0x00000224u, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000225u, 0x00000002u, + 0x00050048u, 0x00000225u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x00000227u, 0x00000021u, + 0x00000004u, 0x00040047u, 0x00000227u, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000231u, 0x00000006u, + 0x00000002u, 0x00030047u, 0x00000232u, 0x00000002u, 0x00050048u, 0x00000232u, 0x00000000u, 0x00000023u, + 0x00000000u, 0x00040047u, 0x00000234u, 0x00000021u, 0x00000005u, 0x00040047u, 0x00000234u, 0x00000022u, + 0x00000000u, 0x00040047u, 0x00000249u, 0x00000006u, 0x00000004u, 0x00030047u, 0x0000024au, 0x00000002u, + 0x00050048u, 0x0000024au, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x0000024cu, 0x00000021u, + 0x00000006u, 0x00040047u, 0x0000024cu, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000257u, 0x00000006u, + 0x00000002u, 0x00030047u, 0x00000258u, 0x00000002u, 0x00050048u, 0x00000258u, 0x00000000u, 0x00000023u, + 0x00000000u, 0x00040047u, 0x0000025au, 0x00000021u, 0x00000007u, 0x00040047u, 0x0000025au, 0x00000022u, + 0x00000000u, 0x00040047u, 0x0000026fu, 0x0000000bu, 0x0000001au, 0x00040047u, 0x00000274u, 0x0000000bu, + 0x0000001bu, 0x00040047u, 0x00000330u, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000331u, 0x00000002u, + 0x00040048u, 0x00000331u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000331u, 0x00000000u, 0x00000023u, + 0x00000000u, 0x00030047u, 0x00000333u, 0x00000018u, 0x00040047u, 0x00000333u, 0x00000021u, 0x0000000cu, + 0x00040047u, 0x00000333u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000003d7u, 0x00000001u, 0x00000002u, + 0x00040047u, 0x000003dbu, 0x00000006u, 0x00000004u, 0x00030047u, 0x000003dcu, 0x00000002u, 0x00050048u, + 0x000003dcu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x000003deu, 0x00000021u, 0x00000008u, + 0x00040047u, 0x000003deu, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000407u, 0x00000006u, 0x00000002u, + 0x00030047u, 0x00000408u, 0x00000002u, 0x00050048u, 0x00000408u, 0x00000000u, 0x00000023u, 0x00000000u, + 0x00040047u, 0x0000040au, 0x00000021u, 0x00000009u, 0x00040047u, 0x0000040au, 0x00000022u, 0x00000000u, + 0x00040047u, 0x00000436u, 0x0000000bu, 0x00000019u, 0x00020013u, 0x00000002u, 0x00030021u, 0x00000003u, + 0x00000002u, 0x00040015u, 0x00000006u, 0x00000020u, 0x00000000u, 0x00040020u, 0x00000007u, 0x00000007u, + 0x00000006u, 0x00030016u, 0x00000008u, 0x00000020u, 0x00040021u, 0x00000009u, 0x00000008u, 0x00000007u, + 0x00040020u, 0x0000000du, 0x00000007u, 0x00000008u, 0x00040021u, 0x0000000eu, 0x00000006u, 0x0000000du, + 0x00050021u, 0x00000018u, 0x00000008u, 0x00000007u, 0x00000007u, 0x00050021u, 0x0000001du, 0x00000006u, + 0x0000000du, 0x00000007u, 0x00050021u, 0x00000022u, 0x00000008u, 0x00000007u, 0x0000000du, 0x00060021u, + 0x00000027u, 0x00000008u, 0x0000000du, 0x0000000du, 0x0000000du, 0x00020014u, 0x0000002du, 0x00040020u, + 0x0000002eu, 0x00000007u, 0x0000002du, 0x00050021u, 0x0000002fu, 0x00000008u, 0x0000002eu, 0x00000007u, + 0x00060021u, 0x00000038u, 0x00000002u, 0x0000002eu, 0x00000007u, 0x0000000du, 0x00040015u, 0x0000003fu, + 0x00000020u, 0x00000001u, 0x0004002bu, 0x0000003fu, 0x00000040u, 0x00000010u, 0x0004002bu, 0x00000006u, + 0x00000049u, 0x7f800000u, 0x0004002bu, 0x00000006u, 0x0000004fu, 0x007fffffu, 0x0004002bu, 0x00000006u, + 0x00000051u, 0x00000000u, 0x0004002bu, 0x00000006u, 0x00000058u, 0x00000040u, 0x0004002bu, 0x00000006u, + 0x0000005au, 0x0000ffffu, 0x0004002bu, 0x00000006u, 0x0000005eu, 0x00007fffu, 0x0004002bu, 0x00000006u, + 0x00000061u, 0x00000001u, 0x0004002bu, 0x00000006u, 0x0000006du, 0x00008000u, 0x0004002bu, 0x0000003fu, + 0x00000072u, 0x0000000au, 0x0004002bu, 0x00000006u, 0x00000074u, 0x0000001fu, 0x0004002bu, 0x00000006u, + 0x00000078u, 0x000003ffu, 0x0004002bu, 0x0000003fu, 0x00000081u, 0x0000000du, 0x0004002bu, 0x00000006u, + 0x00000098u, 0x00000400u, 0x0004002bu, 0x0000003fu, 0x0000009bu, 0x00000001u, 0x0004002bu, 0x00000006u, + 0x000000a3u, 0x00000071u, 0x0004002bu, 0x0000003fu, 0x000000a6u, 0x00000017u, 0x0004002bu, 0x00000006u, + 0x000000b0u, 0x00000070u, 0x0004002bu, 0x00000006u, 0x000000c4u, 0x000000ffu, 0x00040020u, 0x000000c6u, + 0x00000007u, 0x0000003fu, 0x0004002bu, 0x0000003fu, 0x000000cau, 0x0000007fu, 0x0004002bu, 0x0000003fu, + 0x000000ccu, 0x0000000fu, 0x0004002bu, 0x00000006u, 0x000000d6u, 0x00007c00u, 0x0004002bu, 0x00000006u, + 0x000000ddu, 0x00000200u, 0x0004002bu, 0x0000003fu, 0x000000e6u, 0x0000001fu, 0x0004002bu, 0x0000003fu, + 0x000000eeu, 0x00000000u, 0x0004002bu, 0x0000003fu, 0x000000f3u, 0xfffffff6u, 0x0004002bu, 0x00000006u, + 0x000000f9u, 0x00800000u, 0x0004002bu, 0x0000003fu, 0x000000fdu, 0x0000000eu, 0x0004002bu, 0x00000006u, + 0x00000130u, 0x00001fffu, 0x0004002bu, 0x00000006u, 0x00000133u, 0x00001000u, 0x0004002bu, 0x00000006u, + 0x00000168u, 0x00000002u, 0x0004002bu, 0x00000006u, 0x00000169u, 0x00000108u, 0x0004002bu, 0x00000006u, + 0x0000016au, 0x00000080u, 0x0004001cu, 0x0000016bu, 0x00000008u, 0x0000016au, 0x00040020u, 0x0000016cu, + 0x00000004u, 0x0000016bu, 0x0004003bu, 0x0000016cu, 0x0000016du, 0x00000004u, 0x00040020u, 0x00000170u, + 0x00000004u, 0x00000008u, 0x0013001eu, 0x00000190u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000008u, 0x00040020u, 0x00000191u, 0x00000009u, + 0x00000190u, 0x0004003bu, 0x00000191u, 0x00000192u, 0x00000009u, 0x0004002bu, 0x0000003fu, 0x00000193u, + 0x00000007u, 0x00040020u, 0x00000194u, 0x00000009u, 0x00000006u, 0x0004002bu, 0x00000008u, 0x0000019cu, + 0x3f800000u, 0x00040032u, 0x00000006u, 0x000001acu, 0x00000000u, 0x00060034u, 0x0000002du, 0x000001adu, + 0x000000aau, 0x000001acu, 0x00000051u, 0x0003001du, 0x000001b1u, 0x00000006u, 0x0003001eu, 0x000001b2u, + 0x000001b1u, 0x00040020u, 0x000001b3u, 0x0000000cu, 0x000001b2u, 0x0004003bu, 0x000001b3u, 0x000001b4u, + 0x0000000cu, 0x0004002bu, 0x0000003fu, 0x000001b5u, 0x00000008u, 0x0004002bu, 0x0000003fu, 0x000001b8u, + 0x00000002u, 0x00040020u, 0x000001bcu, 0x0000000cu, 0x00000006u, 0x00040015u, 0x000001c1u, 0x00000010u, + 0x00000000u, 0x0003001du, 0x000001c2u, 0x000001c1u, 0x0003001eu, 0x000001c3u, 0x000001c2u, 0x00040020u, + 0x000001c4u, 0x0000000cu, 0x000001c3u, 0x0004003bu, 0x000001c4u, 0x000001c5u, 0x0000000cu, 0x00040020u, + 0x000001cbu, 0x0000000cu, 0x000001c1u, 0x00060034u, 0x0000002du, 0x000001d4u, 0x000000aau, 0x000001acu, + 0x00000051u, 0x0003001du, 0x000001d8u, 0x00000006u, 0x0003001eu, 0x000001d9u, 0x000001d8u, 0x00040020u, + 0x000001dau, 0x0000000cu, 0x000001d9u, 0x0004003bu, 0x000001dau, 0x000001dbu, 0x0000000cu, 0x0004002bu, + 0x0000003fu, 0x000001dcu, 0x00000009u, 0x0003001du, 0x000001e6u, 0x000001c1u, 0x0003001eu, 0x000001e7u, + 0x000001e6u, 0x00040020u, 0x000001e8u, 0x0000000cu, 0x000001e7u, 0x0004003bu, 0x000001e8u, 0x000001e9u, + 0x0000000cu, 0x0003001du, 0x000001fdu, 0x00000006u, 0x0003001eu, 0x000001feu, 0x000001fdu, 0x00040020u, + 0x000001ffu, 0x0000000cu, 0x000001feu, 0x0004003bu, 0x000001ffu, 0x00000200u, 0x0000000cu, 0x0003001du, + 0x0000020au, 0x00000006u, 0x0003001eu, 0x0000020bu, 0x0000020au, 0x00040020u, 0x0000020cu, 0x0000000cu, + 0x0000020bu, 0x0004003bu, 0x0000020cu, 0x0000020du, 0x0000000cu, 0x00040032u, 0x00000006u, 0x00000220u, + 0x00000000u, 0x00060034u, 0x0000002du, 0x00000221u, 0x000000aau, 0x00000220u, 0x00000051u, 0x0003001du, + 0x00000224u, 0x00000006u, 0x0003001eu, 0x00000225u, 0x00000224u, 0x00040020u, 0x00000226u, 0x0000000cu, + 0x00000225u, 0x0004003bu, 0x00000226u, 0x00000227u, 0x0000000cu, 0x0003001du, 0x00000231u, 0x000001c1u, + 0x0003001eu, 0x00000232u, 0x00000231u, 0x00040020u, 0x00000233u, 0x0000000cu, 0x00000232u, 0x0004003bu, + 0x00000233u, 0x00000234u, 0x0000000cu, 0x0003002au, 0x0000002du, 0x00000240u, 0x00060034u, 0x0000002du, + 0x00000246u, 0x000000aau, 0x00000220u, 0x00000051u, 0x0003001du, 0x00000249u, 0x00000006u, 0x0003001eu, + 0x0000024au, 0x00000249u, 0x00040020u, 0x0000024bu, 0x0000000cu, 0x0000024au, 0x0004003bu, 0x0000024bu, + 0x0000024cu, 0x0000000cu, 0x0004002bu, 0x0000003fu, 0x0000024du, 0x0000000bu, 0x0003001du, 0x00000257u, + 0x000001c1u, 0x0003001eu, 0x00000258u, 0x00000257u, 0x00040020u, 0x00000259u, 0x0000000cu, 0x00000258u, + 0x0004003bu, 0x00000259u, 0x0000025au, 0x0000000cu, 0x00040017u, 0x0000026du, 0x00000006u, 0x00000003u, + 0x00040020u, 0x0000026eu, 0x00000001u, 0x0000026du, 0x0004003bu, 0x0000026eu, 0x0000026fu, 0x00000001u, + 0x00040020u, 0x00000270u, 0x00000001u, 0x00000006u, 0x0004003bu, 0x0000026eu, 0x00000274u, 0x00000001u, + 0x0004002bu, 0x0000003fu, 0x0000028au, 0x00000005u, 0x0004002bu, 0x0000003fu, 0x000002abu, 0x00000006u, + 0x0004002bu, 0x00000008u, 0x000002beu, 0x00000000u, 0x00040020u, 0x000002e3u, 0x00000009u, 0x00000008u, + 0x0004002bu, 0x0000003fu, 0x000002eeu, 0x00000003u, 0x0004002bu, 0x0000003fu, 0x000002ffu, 0x00000004u, + 0x0003001du, 0x00000330u, 0x00000006u, 0x0003001eu, 0x00000331u, 0x00000330u, 0x00040020u, 0x00000332u, + 0x0000000cu, 0x00000331u, 0x0004003bu, 0x00000332u, 0x00000333u, 0x0000000cu, 0x00040032u, 0x00000006u, + 0x000003d7u, 0x00000000u, 0x00060034u, 0x0000002du, 0x000003d8u, 0x000000aau, 0x000003d7u, 0x00000051u, + 0x0003001du, 0x000003dbu, 0x00000006u, 0x0003001eu, 0x000003dcu, 0x000003dbu, 0x00040020u, 0x000003ddu, + 0x0000000cu, 0x000003dcu, 0x0004003bu, 0x000003ddu, 0x000003deu, 0x0000000cu, 0x0004002bu, 0x0000003fu, + 0x000003dfu, 0x0000000cu, 0x00060034u, 0x0000002du, 0x000003e7u, 0x000000aau, 0x000001acu, 0x00000051u, + 0x0003001du, 0x00000407u, 0x000001c1u, 0x0003001eu, 0x00000408u, 0x00000407u, 0x00040020u, 0x00000409u, + 0x0000000cu, 0x00000408u, 0x0004003bu, 0x00000409u, 0x0000040au, 0x0000000cu, 0x00060034u, 0x0000002du, + 0x00000412u, 0x000000aau, 0x000001acu, 0x00000051u, 0x0006002cu, 0x0000026du, 0x00000436u, 0x0000016au, + 0x00000061u, 0x00000061u, 0x00050036u, 0x00000002u, 0x00000004u, 0x00000000u, 0x00000003u, 0x000200f8u, + 0x00000005u, 0x0004003bu, 0x00000007u, 0x00000266u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000026cu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000273u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000277u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000027bu, 0x00000007u, 0x0004003bu, 0x0000002eu, 0x0000027fu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000284u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000288u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000294u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000029au, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002a5u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000002bdu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002bfu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000002cau, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x000002ceu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002d0u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000002d9u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002dau, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000002dcu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002e9u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002f3u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000305u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x00000309u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000030bu, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x0000030du, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000030fu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000312u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000313u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000315u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000317u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x0000031eu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000320u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x00000326u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000328u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000032au, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000032bu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000032du, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000032fu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000033au, 0x00000007u, 0x0004003bu, 0x0000002eu, 0x0000034eu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000350u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000351u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000035au, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000035fu, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x00000363u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000365u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x00000367u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000369u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000036cu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000036du, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000036fu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000371u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x00000378u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000037au, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x00000380u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000382u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000384u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000385u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000387u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000389u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000390u, 0x00000007u, 0x0004003bu, 0x0000002eu, 0x000003a4u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003a6u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003a7u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x000003b0u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003b2u, + 0x00000007u, 0x0004003bu, 0x0000002eu, 0x000003b4u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003b6u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003b9u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003bau, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003bcu, 0x00000007u, 0x0004003bu, 0x0000002eu, 0x000003beu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003c0u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003c1u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003c8u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003e8u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000400u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000401u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000413u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000042bu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000042cu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000042eu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000430u, 0x00000007u, 0x00050041u, 0x00000194u, 0x00000267u, + 0x00000192u, 0x000000eeu, 0x0004003du, 0x00000006u, 0x00000268u, 0x00000267u, 0x00050041u, 0x00000194u, + 0x00000269u, 0x00000192u, 0x0000009bu, 0x0004003du, 0x00000006u, 0x0000026au, 0x00000269u, 0x00050080u, + 0x00000006u, 0x0000026bu, 0x00000268u, 0x0000026au, 0x0003003eu, 0x00000266u, 0x0000026bu, 0x00050041u, + 0x00000270u, 0x00000271u, 0x0000026fu, 0x00000051u, 0x0004003du, 0x00000006u, 0x00000272u, 0x00000271u, + 0x0003003eu, 0x0000026cu, 0x00000272u, 0x00050041u, 0x00000270u, 0x00000275u, 0x00000274u, 0x00000051u, + 0x0004003du, 0x00000006u, 0x00000276u, 0x00000275u, 0x0003003eu, 0x00000273u, 0x00000276u, 0x0004003du, + 0x00000006u, 0x00000278u, 0x0000026cu, 0x0004003du, 0x00000006u, 0x00000279u, 0x00000266u, 0x00050086u, + 0x00000006u, 0x0000027au, 0x00000278u, 0x00000279u, 0x0003003eu, 0x00000277u, 0x0000027au, 0x0004003du, + 0x00000006u, 0x0000027cu, 0x0000026cu, 0x0004003du, 0x00000006u, 0x0000027du, 0x00000266u, 0x00050089u, + 0x00000006u, 0x0000027eu, 0x0000027cu, 0x0000027du, 0x0003003eu, 0x0000027bu, 0x0000027eu, 0x0004003du, + 0x00000006u, 0x00000280u, 0x0000027bu, 0x00050041u, 0x00000194u, 0x00000281u, 0x00000192u, 0x000000eeu, + 0x0004003du, 0x00000006u, 0x00000282u, 0x00000281u, 0x000500b0u, 0x0000002du, 0x00000283u, 0x00000280u, + 0x00000282u, 0x0003003eu, 0x0000027fu, 0x00000283u, 0x0003003eu, 0x00000284u, 0x00000051u, 0x0004003du, + 0x0000002du, 0x00000285u, 0x0000027fu, 0x000300f7u, 0x00000287u, 0x00000000u, 0x000400fau, 0x00000285u, + 0x00000286u, 0x000002a4u, 0x000200f8u, 0x00000286u, 0x0004003du, 0x00000006u, 0x00000289u, 0x00000277u, + 0x00050041u, 0x00000194u, 0x0000028bu, 0x00000192u, 0x0000028au, 0x0004003du, 0x00000006u, 0x0000028cu, + 0x0000028bu, 0x00050084u, 0x00000006u, 0x0000028du, 0x00000289u, 0x0000028cu, 0x0004003du, 0x00000006u, + 0x0000028eu, 0x0000027bu, 0x00050084u, 0x00000006u, 0x0000028fu, 0x0000028eu, 0x00000168u, 0x00050041u, + 0x00000194u, 0x00000290u, 0x00000192u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x00000291u, 0x00000290u, + 0x00050084u, 0x00000006u, 0x00000292u, 0x0000028fu, 0x00000291u, 0x00050080u, 0x00000006u, 0x00000293u, + 0x0000028du, 0x00000292u, 0x0003003eu, 0x00000288u, 0x00000293u, 0x0004003du, 0x00000006u, 0x00000295u, + 0x00000288u, 0x0003003eu, 0x00000294u, 0x00000295u, 0x0004003du, 0x00000006u, 0x00000296u, 0x00000288u, + 0x00050041u, 0x00000194u, 0x00000297u, 0x00000192u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x00000298u, + 0x00000297u, 0x00050080u, 0x00000006u, 0x00000299u, 0x00000296u, 0x00000298u, 0x0003003eu, 0x00000284u, + 0x00000299u, 0x0004003du, 0x00000006u, 0x0000029bu, 0x00000277u, 0x00050041u, 0x00000194u, 0x0000029cu, + 0x00000192u, 0x000000eeu, 0x0004003du, 0x00000006u, 0x0000029du, 0x0000029cu, 0x00050084u, 0x00000006u, + 0x0000029eu, 0x0000029bu, 0x0000029du, 0x0004003du, 0x00000006u, 0x0000029fu, 0x0000027bu, 0x00050080u, + 0x00000006u, 0x000002a0u, 0x0000029eu, 0x0000029fu, 0x00050041u, 0x00000194u, 0x000002a1u, 0x00000192u, + 0x000001b8u, 0x0004003du, 0x00000006u, 0x000002a2u, 0x000002a1u, 0x00050084u, 0x00000006u, 0x000002a3u, + 0x000002a0u, 0x000002a2u, 0x0003003eu, 0x0000029au, 0x000002a3u, 0x000200f9u, 0x00000287u, 0x000200f8u, + 0x000002a4u, 0x0004003du, 0x00000006u, 0x000002a6u, 0x0000027bu, 0x00050041u, 0x00000194u, 0x000002a7u, + 0x00000192u, 0x000000eeu, 0x0004003du, 0x00000006u, 0x000002a8u, 0x000002a7u, 0x00050082u, 0x00000006u, + 0x000002a9u, 0x000002a6u, 0x000002a8u, 0x0003003eu, 0x000002a5u, 0x000002a9u, 0x0004003du, 0x00000006u, + 0x000002aau, 0x00000277u, 0x00050041u, 0x00000194u, 0x000002acu, 0x00000192u, 0x000002abu, 0x0004003du, + 0x00000006u, 0x000002adu, 0x000002acu, 0x00050084u, 0x00000006u, 0x000002aeu, 0x000002aau, 0x000002adu, + 0x0004003du, 0x00000006u, 0x000002afu, 0x000002a5u, 0x00050041u, 0x00000194u, 0x000002b0u, 0x00000192u, + 0x000001b8u, 0x0004003du, 0x00000006u, 0x000002b1u, 0x000002b0u, 0x00050084u, 0x00000006u, 0x000002b2u, + 0x000002afu, 0x000002b1u, 0x00050080u, 0x00000006u, 0x000002b3u, 0x000002aeu, 0x000002b2u, 0x0003003eu, + 0x00000294u, 0x000002b3u, 0x0004003du, 0x00000006u, 0x000002b4u, 0x00000277u, 0x00050041u, 0x00000194u, + 0x000002b5u, 0x00000192u, 0x0000009bu, 0x0004003du, 0x00000006u, 0x000002b6u, 0x000002b5u, 0x00050084u, + 0x00000006u, 0x000002b7u, 0x000002b4u, 0x000002b6u, 0x0004003du, 0x00000006u, 0x000002b8u, 0x000002a5u, + 0x00050080u, 0x00000006u, 0x000002b9u, 0x000002b7u, 0x000002b8u, 0x00050041u, 0x00000194u, 0x000002bau, + 0x00000192u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x000002bbu, 0x000002bau, 0x00050084u, 0x00000006u, + 0x000002bcu, 0x000002b9u, 0x000002bbu, 0x0003003eu, 0x0000029au, 0x000002bcu, 0x000200f9u, 0x00000287u, + 0x000200f8u, 0x00000287u, 0x0003003eu, 0x000002bdu, 0x000002beu, 0x0004003du, 0x00000006u, 0x000002c0u, + 0x00000273u, 0x0003003eu, 0x000002bfu, 0x000002c0u, 0x000200f9u, 0x000002c1u, 0x000200f8u, 0x000002c1u, + 0x000400f6u, 0x000002c3u, 0x000002c4u, 0x00000000u, 0x000200f9u, 0x000002c5u, 0x000200f8u, 0x000002c5u, + 0x0004003du, 0x00000006u, 0x000002c6u, 0x000002bfu, 0x00050041u, 0x00000194u, 0x000002c7u, 0x00000192u, + 0x000001b8u, 0x0004003du, 0x00000006u, 0x000002c8u, 0x000002c7u, 0x000500b0u, 0x0000002du, 0x000002c9u, + 0x000002c6u, 0x000002c8u, 0x000400fau, 0x000002c9u, 0x000002c2u, 0x000002c3u, 0x000200f8u, 0x000002c2u, + 0x0004003du, 0x00000006u, 0x000002cbu, 0x00000294u, 0x0004003du, 0x00000006u, 0x000002ccu, 0x000002bfu, + 0x00050080u, 0x00000006u, 0x000002cdu, 0x000002cbu, 0x000002ccu, 0x0004003du, 0x0000002du, 0x000002cfu, + 0x0000027fu, 0x0003003eu, 0x000002ceu, 0x000002cfu, 0x0003003eu, 0x000002d0u, 0x000002cdu, 0x00060039u, + 0x00000008u, 0x000002d1u, 0x00000032u, 0x000002ceu, 0x000002d0u, 0x0003003eu, 0x000002cau, 0x000002d1u, + 0x0004003du, 0x00000008u, 0x000002d2u, 0x000002cau, 0x0004003du, 0x00000008u, 0x000002d3u, 0x000002cau, + 0x00050085u, 0x00000008u, 0x000002d4u, 0x000002d2u, 0x000002d3u, 0x0004003du, 0x00000008u, 0x000002d5u, + 0x000002bdu, 0x00050081u, 0x00000008u, 0x000002d6u, 0x000002d5u, 0x000002d4u, 0x0003003eu, 0x000002bdu, + 0x000002d6u, 0x000200f9u, 0x000002c4u, 0x000200f8u, 0x000002c4u, 0x0004003du, 0x00000006u, 0x000002d7u, + 0x000002bfu, 0x00050080u, 0x00000006u, 0x000002d8u, 0x000002d7u, 0x0000016au, 0x0003003eu, 0x000002bfu, + 0x000002d8u, 0x000200f9u, 0x000002c1u, 0x000200f8u, 0x000002c3u, 0x0004003du, 0x00000006u, 0x000002dbu, + 0x00000273u, 0x0003003eu, 0x000002dau, 0x000002dbu, 0x0004003du, 0x00000008u, 0x000002ddu, 0x000002bdu, + 0x0003003eu, 0x000002dcu, 0x000002ddu, 0x00060039u, 0x00000008u, 0x000002deu, 0x00000025u, 0x000002dau, + 0x000002dcu, 0x00050041u, 0x00000194u, 0x000002dfu, 0x00000192u, 0x000001b8u, 0x0004003du, 0x00000006u, + 0x000002e0u, 0x000002dfu, 0x00040070u, 0x00000008u, 0x000002e1u, 0x000002e0u, 0x00050088u, 0x00000008u, + 0x000002e2u, 0x000002deu, 0x000002e1u, 0x00050041u, 0x000002e3u, 0x000002e4u, 0x00000192u, 0x00000040u, + 0x0004003du, 0x00000008u, 0x000002e5u, 0x000002e4u, 0x00050081u, 0x00000008u, 0x000002e6u, 0x000002e2u, + 0x000002e5u, 0x0006000cu, 0x00000008u, 0x000002e7u, 0x00000001u, 0x0000001fu, 0x000002e6u, 0x00050088u, + 0x00000008u, 0x000002e8u, 0x0000019cu, 0x000002e7u, 0x0003003eu, 0x000002d9u, 0x000002e8u, 0x00050041u, + 0x00000194u, 0x000002eau, 0x00000192u, 0x000000ccu, 0x0004003du, 0x00000006u, 0x000002ebu, 0x000002eau, + 0x000500c2u, 0x00000006u, 0x000002ecu, 0x000002ebu, 0x000001b8u, 0x0004003du, 0x00000006u, 0x000002edu, + 0x00000277u, 0x00050041u, 0x00000194u, 0x000002efu, 0x00000192u, 0x000002eeu, 0x0004003du, 0x00000006u, + 0x000002f0u, 0x000002efu, 0x00050084u, 0x00000006u, 0x000002f1u, 0x000002edu, 0x000002f0u, 0x00050080u, + 0x00000006u, 0x000002f2u, 0x000002ecu, 0x000002f1u, 0x0003003eu, 0x000002e9u, 0x000002f2u, 0x0004003du, + 0x00000006u, 0x000002f4u, 0x00000273u, 0x0003003eu, 0x000002f3u, 0x000002f4u, 0x000200f9u, 0x000002f5u, + 0x000200f8u, 0x000002f5u, 0x000400f6u, 0x000002f7u, 0x000002f8u, 0x00000000u, 0x000200f9u, 0x000002f9u, + 0x000200f8u, 0x000002f9u, 0x0004003du, 0x00000006u, 0x000002fau, 0x000002f3u, 0x00050041u, 0x00000194u, + 0x000002fbu, 0x00000192u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x000002fcu, 0x000002fbu, 0x000500b0u, + 0x0000002du, 0x000002fdu, 0x000002fau, 0x000002fcu, 0x000400fau, 0x000002fdu, 0x000002f6u, 0x000002f7u, + 0x000200f8u, 0x000002f6u, 0x0004003du, 0x00000006u, 0x000002feu, 0x000002f3u, 0x00050041u, 0x00000194u, + 0x00000300u, 0x00000192u, 0x000002ffu, 0x0004003du, 0x00000006u, 0x00000301u, 0x00000300u, 0x000500b0u, + 0x0000002du, 0x00000302u, 0x000002feu, 0x00000301u, 0x000300f7u, 0x00000304u, 0x00000000u, 0x000400fau, + 0x00000302u, 0x00000303u, 0x00000353u, 0x000200f8u, 0x00000303u, 0x0004003du, 0x00000006u, 0x00000306u, + 0x00000294u, 0x0004003du, 0x00000006u, 0x00000307u, 0x000002f3u, 0x00050080u, 0x00000006u, 0x00000308u, + 0x00000306u, 0x00000307u, 0x0004003du, 0x0000002du, 0x0000030au, 0x0000027fu, 0x0003003eu, 0x00000309u, + 0x0000030au, 0x0003003eu, 0x0000030bu, 0x00000308u, 0x00060039u, 0x00000008u, 0x0000030cu, 0x00000032u, + 0x00000309u, 0x0000030bu, 0x0004003du, 0x0000002du, 0x0000030eu, 0x0000027fu, 0x0003003eu, 0x0000030du, + 0x0000030eu, 0x0004003du, 0x00000006u, 0x00000310u, 0x000002f3u, 0x0003003eu, 0x0000030fu, 0x00000310u, + 0x00060039u, 0x00000008u, 0x00000311u, 0x00000036u, 0x0000030du, 0x0000030fu, 0x0003003eu, 0x00000312u, + 0x0000030cu, 0x0004003du, 0x00000008u, 0x00000314u, 0x000002d9u, 0x0003003eu, 0x00000313u, 0x00000314u, + 0x0003003eu, 0x00000315u, 0x00000311u, 0x00070039u, 0x00000008u, 0x00000316u, 0x0000002bu, 0x00000312u, + 0x00000313u, 0x00000315u, 0x0003003eu, 0x00000305u, 0x00000316u, 0x0004003du, 0x00000006u, 0x00000318u, + 0x00000294u, 0x0004003du, 0x00000006u, 0x00000319u, 0x000002f3u, 0x00050080u, 0x00000006u, 0x0000031au, + 0x00000318u, 0x00000319u, 0x00050041u, 0x00000194u, 0x0000031bu, 0x00000192u, 0x000002ffu, 0x0004003du, + 0x00000006u, 0x0000031cu, 0x0000031bu, 0x00050080u, 0x00000006u, 0x0000031du, 0x0000031au, 0x0000031cu, + 0x0004003du, 0x0000002du, 0x0000031fu, 0x0000027fu, 0x0003003eu, 0x0000031eu, 0x0000031fu, 0x0003003eu, + 0x00000320u, 0x0000031du, 0x00060039u, 0x00000008u, 0x00000321u, 0x00000032u, 0x0000031eu, 0x00000320u, + 0x0004003du, 0x00000006u, 0x00000322u, 0x000002f3u, 0x00050041u, 0x00000194u, 0x00000323u, 0x00000192u, + 0x000002ffu, 0x0004003du, 0x00000006u, 0x00000324u, 0x00000323u, 0x00050080u, 0x00000006u, 0x00000325u, + 0x00000322u, 0x00000324u, 0x0004003du, 0x0000002du, 0x00000327u, 0x0000027fu, 0x0003003eu, 0x00000326u, + 0x00000327u, 0x0003003eu, 0x00000328u, 0x00000325u, 0x00060039u, 0x00000008u, 0x00000329u, 0x00000036u, + 0x00000326u, 0x00000328u, 0x0003003eu, 0x0000032au, 0x00000321u, 0x0004003du, 0x00000008u, 0x0000032cu, + 0x000002d9u, 0x0003003eu, 0x0000032bu, 0x0000032cu, 0x0003003eu, 0x0000032du, 0x00000329u, 0x00070039u, + 0x00000008u, 0x0000032eu, 0x0000002bu, 0x0000032au, 0x0000032bu, 0x0000032du, 0x0003003eu, 0x00000317u, + 0x0000032eu, 0x0004003du, 0x00000006u, 0x00000334u, 0x000002e9u, 0x0004003du, 0x00000006u, 0x00000335u, + 0x000002f3u, 0x00050080u, 0x00000006u, 0x00000336u, 0x00000334u, 0x00000335u, 0x00060041u, 0x000001bcu, + 0x00000337u, 0x00000333u, 0x000000eeu, 0x00000336u, 0x0004003du, 0x00000006u, 0x00000338u, 0x00000337u, + 0x0004007cu, 0x00000008u, 0x00000339u, 0x00000338u, 0x0003003eu, 0x0000032fu, 0x00000339u, 0x0004003du, + 0x00000006u, 0x0000033bu, 0x000002e9u, 0x00050041u, 0x00000194u, 0x0000033cu, 0x00000192u, 0x000002ffu, + 0x0004003du, 0x00000006u, 0x0000033du, 0x0000033cu, 0x00050080u, 0x00000006u, 0x0000033eu, 0x0000033bu, + 0x0000033du, 0x0004003du, 0x00000006u, 0x0000033fu, 0x000002f3u, 0x00050080u, 0x00000006u, 0x00000340u, + 0x0000033eu, 0x0000033fu, 0x00060041u, 0x000001bcu, 0x00000341u, 0x00000333u, 0x000000eeu, 0x00000340u, + 0x0004003du, 0x00000006u, 0x00000342u, 0x00000341u, 0x0004007cu, 0x00000008u, 0x00000343u, 0x00000342u, + 0x0003003eu, 0x0000033au, 0x00000343u, 0x0004003du, 0x00000006u, 0x00000344u, 0x0000029au, 0x0004003du, + 0x00000006u, 0x00000345u, 0x000002f3u, 0x00050080u, 0x00000006u, 0x00000346u, 0x00000344u, 0x00000345u, + 0x0004003du, 0x00000008u, 0x00000347u, 0x00000305u, 0x0004003du, 0x00000008u, 0x00000348u, 0x0000032fu, + 0x00050085u, 0x00000008u, 0x00000349u, 0x00000347u, 0x00000348u, 0x0004003du, 0x00000008u, 0x0000034au, + 0x00000317u, 0x0004003du, 0x00000008u, 0x0000034bu, 0x0000033au, 0x00050085u, 0x00000008u, 0x0000034cu, + 0x0000034au, 0x0000034bu, 0x00050083u, 0x00000008u, 0x0000034du, 0x00000349u, 0x0000034cu, 0x0004003du, + 0x0000002du, 0x0000034fu, 0x0000027fu, 0x0003003eu, 0x0000034eu, 0x0000034fu, 0x0003003eu, 0x00000350u, + 0x00000346u, 0x0003003eu, 0x00000351u, 0x0000034du, 0x00070039u, 0x00000002u, 0x00000352u, 0x0000003cu, + 0x0000034eu, 0x00000350u, 0x00000351u, 0x000200f9u, 0x00000304u, 0x000200f8u, 0x00000353u, 0x0004003du, + 0x00000006u, 0x00000354u, 0x000002f3u, 0x00050041u, 0x00000194u, 0x00000355u, 0x00000192u, 0x000002eeu, + 0x0004003du, 0x00000006u, 0x00000356u, 0x00000355u, 0x000500b0u, 0x0000002du, 0x00000357u, 0x00000354u, + 0x00000356u, 0x000300f7u, 0x00000359u, 0x00000000u, 0x000400fau, 0x00000357u, 0x00000358u, 0x000003a9u, + 0x000200f8u, 0x00000358u, 0x0004003du, 0x00000006u, 0x0000035bu, 0x000002f3u, 0x00050041u, 0x00000194u, + 0x0000035cu, 0x00000192u, 0x000002ffu, 0x0004003du, 0x00000006u, 0x0000035du, 0x0000035cu, 0x00050082u, + 0x00000006u, 0x0000035eu, 0x0000035bu, 0x0000035du, 0x0003003eu, 0x0000035au, 0x0000035eu, 0x0004003du, + 0x00000006u, 0x00000360u, 0x00000294u, 0x0004003du, 0x00000006u, 0x00000361u, 0x0000035au, 0x00050080u, + 0x00000006u, 0x00000362u, 0x00000360u, 0x00000361u, 0x0004003du, 0x0000002du, 0x00000364u, 0x0000027fu, + 0x0003003eu, 0x00000363u, 0x00000364u, 0x0003003eu, 0x00000365u, 0x00000362u, 0x00060039u, 0x00000008u, + 0x00000366u, 0x00000032u, 0x00000363u, 0x00000365u, 0x0004003du, 0x0000002du, 0x00000368u, 0x0000027fu, + 0x0003003eu, 0x00000367u, 0x00000368u, 0x0004003du, 0x00000006u, 0x0000036au, 0x0000035au, 0x0003003eu, + 0x00000369u, 0x0000036au, 0x00060039u, 0x00000008u, 0x0000036bu, 0x00000036u, 0x00000367u, 0x00000369u, + 0x0003003eu, 0x0000036cu, 0x00000366u, 0x0004003du, 0x00000008u, 0x0000036eu, 0x000002d9u, 0x0003003eu, + 0x0000036du, 0x0000036eu, 0x0003003eu, 0x0000036fu, 0x0000036bu, 0x00070039u, 0x00000008u, 0x00000370u, + 0x0000002bu, 0x0000036cu, 0x0000036du, 0x0000036fu, 0x0003003eu, 0x0000035fu, 0x00000370u, 0x0004003du, + 0x00000006u, 0x00000372u, 0x00000294u, 0x0004003du, 0x00000006u, 0x00000373u, 0x0000035au, 0x00050080u, + 0x00000006u, 0x00000374u, 0x00000372u, 0x00000373u, 0x00050041u, 0x00000194u, 0x00000375u, 0x00000192u, + 0x000002ffu, 0x0004003du, 0x00000006u, 0x00000376u, 0x00000375u, 0x00050080u, 0x00000006u, 0x00000377u, + 0x00000374u, 0x00000376u, 0x0004003du, 0x0000002du, 0x00000379u, 0x0000027fu, 0x0003003eu, 0x00000378u, + 0x00000379u, 0x0003003eu, 0x0000037au, 0x00000377u, 0x00060039u, 0x00000008u, 0x0000037bu, 0x00000032u, + 0x00000378u, 0x0000037au, 0x0004003du, 0x00000006u, 0x0000037cu, 0x0000035au, 0x00050041u, 0x00000194u, + 0x0000037du, 0x00000192u, 0x000002ffu, 0x0004003du, 0x00000006u, 0x0000037eu, 0x0000037du, 0x00050080u, + 0x00000006u, 0x0000037fu, 0x0000037cu, 0x0000037eu, 0x0004003du, 0x0000002du, 0x00000381u, 0x0000027fu, + 0x0003003eu, 0x00000380u, 0x00000381u, 0x0003003eu, 0x00000382u, 0x0000037fu, 0x00060039u, 0x00000008u, + 0x00000383u, 0x00000036u, 0x00000380u, 0x00000382u, 0x0003003eu, 0x00000384u, 0x0000037bu, 0x0004003du, + 0x00000008u, 0x00000386u, 0x000002d9u, 0x0003003eu, 0x00000385u, 0x00000386u, 0x0003003eu, 0x00000387u, + 0x00000383u, 0x00070039u, 0x00000008u, 0x00000388u, 0x0000002bu, 0x00000384u, 0x00000385u, 0x00000387u, + 0x0003003eu, 0x00000371u, 0x00000388u, 0x0004003du, 0x00000006u, 0x0000038au, 0x000002e9u, 0x0004003du, + 0x00000006u, 0x0000038bu, 0x0000035au, 0x00050080u, 0x00000006u, 0x0000038cu, 0x0000038au, 0x0000038bu, + 0x00060041u, 0x000001bcu, 0x0000038du, 0x00000333u, 0x000000eeu, 0x0000038cu, 0x0004003du, 0x00000006u, + 0x0000038eu, 0x0000038du, 0x0004007cu, 0x00000008u, 0x0000038fu, 0x0000038eu, 0x0003003eu, 0x00000389u, + 0x0000038fu, 0x0004003du, 0x00000006u, 0x00000391u, 0x000002e9u, 0x00050041u, 0x00000194u, 0x00000392u, + 0x00000192u, 0x000002ffu, 0x0004003du, 0x00000006u, 0x00000393u, 0x00000392u, 0x00050080u, 0x00000006u, + 0x00000394u, 0x00000391u, 0x00000393u, 0x0004003du, 0x00000006u, 0x00000395u, 0x0000035au, 0x00050080u, + 0x00000006u, 0x00000396u, 0x00000394u, 0x00000395u, 0x00060041u, 0x000001bcu, 0x00000397u, 0x00000333u, + 0x000000eeu, 0x00000396u, 0x0004003du, 0x00000006u, 0x00000398u, 0x00000397u, 0x0004007cu, 0x00000008u, + 0x00000399u, 0x00000398u, 0x0003003eu, 0x00000390u, 0x00000399u, 0x0004003du, 0x00000006u, 0x0000039au, + 0x0000029au, 0x0004003du, 0x00000006u, 0x0000039bu, 0x000002f3u, 0x00050080u, 0x00000006u, 0x0000039cu, + 0x0000039au, 0x0000039bu, 0x0004003du, 0x00000008u, 0x0000039du, 0x0000035fu, 0x0004003du, 0x00000008u, + 0x0000039eu, 0x00000390u, 0x00050085u, 0x00000008u, 0x0000039fu, 0x0000039du, 0x0000039eu, 0x0004003du, + 0x00000008u, 0x000003a0u, 0x00000371u, 0x0004003du, 0x00000008u, 0x000003a1u, 0x00000389u, 0x00050085u, + 0x00000008u, 0x000003a2u, 0x000003a0u, 0x000003a1u, 0x00050081u, 0x00000008u, 0x000003a3u, 0x0000039fu, + 0x000003a2u, 0x0004003du, 0x0000002du, 0x000003a5u, 0x0000027fu, 0x0003003eu, 0x000003a4u, 0x000003a5u, + 0x0003003eu, 0x000003a6u, 0x0000039cu, 0x0003003eu, 0x000003a7u, 0x000003a3u, 0x00070039u, 0x00000002u, + 0x000003a8u, 0x0000003cu, 0x000003a4u, 0x000003a6u, 0x000003a7u, 0x000200f9u, 0x00000359u, 0x000200f8u, + 0x000003a9u, 0x0004003du, 0x00000006u, 0x000003aau, 0x0000029au, 0x0004003du, 0x00000006u, 0x000003abu, + 0x000002f3u, 0x00050080u, 0x00000006u, 0x000003acu, 0x000003aau, 0x000003abu, 0x0004003du, 0x00000006u, + 0x000003adu, 0x00000294u, 0x0004003du, 0x00000006u, 0x000003aeu, 0x000002f3u, 0x00050080u, 0x00000006u, + 0x000003afu, 0x000003adu, 0x000003aeu, 0x0004003du, 0x0000002du, 0x000003b1u, 0x0000027fu, 0x0003003eu, + 0x000003b0u, 0x000003b1u, 0x0003003eu, 0x000003b2u, 0x000003afu, 0x00060039u, 0x00000008u, 0x000003b3u, + 0x00000032u, 0x000003b0u, 0x000003b2u, 0x0004003du, 0x0000002du, 0x000003b5u, 0x0000027fu, 0x0003003eu, + 0x000003b4u, 0x000003b5u, 0x0004003du, 0x00000006u, 0x000003b7u, 0x000002f3u, 0x0003003eu, 0x000003b6u, + 0x000003b7u, 0x00060039u, 0x00000008u, 0x000003b8u, 0x00000036u, 0x000003b4u, 0x000003b6u, 0x0003003eu, + 0x000003b9u, 0x000003b3u, 0x0004003du, 0x00000008u, 0x000003bbu, 0x000002d9u, 0x0003003eu, 0x000003bau, + 0x000003bbu, 0x0003003eu, 0x000003bcu, 0x000003b8u, 0x00070039u, 0x00000008u, 0x000003bdu, 0x0000002bu, + 0x000003b9u, 0x000003bau, 0x000003bcu, 0x0004003du, 0x0000002du, 0x000003bfu, 0x0000027fu, 0x0003003eu, + 0x000003beu, 0x000003bfu, 0x0003003eu, 0x000003c0u, 0x000003acu, 0x0003003eu, 0x000003c1u, 0x000003bdu, + 0x00070039u, 0x00000002u, 0x000003c2u, 0x0000003cu, 0x000003beu, 0x000003c0u, 0x000003c1u, 0x000200f9u, + 0x00000359u, 0x000200f8u, 0x00000359u, 0x000200f9u, 0x00000304u, 0x000200f8u, 0x00000304u, 0x000200f9u, + 0x000002f8u, 0x000200f8u, 0x000002f8u, 0x0004003du, 0x00000006u, 0x000003c3u, 0x000002f3u, 0x00050080u, + 0x00000006u, 0x000003c4u, 0x000003c3u, 0x0000016au, 0x0003003eu, 0x000002f3u, 0x000003c4u, 0x000200f9u, + 0x000002f5u, 0x000200f8u, 0x000002f7u, 0x0004003du, 0x0000002du, 0x000003c5u, 0x0000027fu, 0x000300f7u, + 0x000003c7u, 0x00000000u, 0x000400fau, 0x000003c5u, 0x000003c6u, 0x000003c7u, 0x000200f8u, 0x000003c6u, + 0x0004003du, 0x00000006u, 0x000003c9u, 0x00000273u, 0x0003003eu, 0x000003c8u, 0x000003c9u, 0x000200f9u, + 0x000003cau, 0x000200f8u, 0x000003cau, 0x000400f6u, 0x000003ccu, 0x000003cdu, 0x00000000u, 0x000200f9u, + 0x000003ceu, 0x000200f8u, 0x000003ceu, 0x0004003du, 0x00000006u, 0x000003cfu, 0x000003c8u, 0x00050041u, + 0x00000194u, 0x000003d0u, 0x00000192u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x000003d1u, 0x000003d0u, + 0x000500b0u, 0x0000002du, 0x000003d2u, 0x000003cfu, 0x000003d1u, 0x000400fau, 0x000003d2u, 0x000003cbu, + 0x000003ccu, 0x000200f8u, 0x000003cbu, 0x000200f9u, 0x000003d3u, 0x000200f8u, 0x000003d3u, 0x000400f6u, + 0x000003d5u, 0x000003d6u, 0x00000000u, 0x000200f9u, 0x000003d4u, 0x000200f8u, 0x000003d4u, 0x000300f7u, + 0x000003dau, 0x00000000u, 0x000400fau, 0x000003d8u, 0x000003d9u, 0x00000406u, 0x000200f8u, 0x000003d9u, + 0x00050041u, 0x00000194u, 0x000003e0u, 0x00000192u, 0x000003dfu, 0x0004003du, 0x00000006u, 0x000003e1u, + 0x000003e0u, 0x000500c2u, 0x00000006u, 0x000003e2u, 0x000003e1u, 0x000001b8u, 0x0004003du, 0x00000006u, + 0x000003e3u, 0x0000029au, 0x0004003du, 0x00000006u, 0x000003e4u, 0x000003c8u, 0x00050080u, 0x00000006u, + 0x000003e5u, 0x000003e3u, 0x000003e4u, 0x00050080u, 0x00000006u, 0x000003e6u, 0x000003e2u, 0x000003e5u, + 0x000300f7u, 0x000003eau, 0x00000000u, 0x000400fau, 0x000003e7u, 0x000003e9u, 0x000003f5u, 0x000200f8u, + 0x000003e9u, 0x00050041u, 0x00000194u, 0x000003ebu, 0x00000192u, 0x000001b5u, 0x0004003du, 0x00000006u, + 0x000003ecu, 0x000003ebu, 0x000500c2u, 0x00000006u, 0x000003edu, 0x000003ecu, 0x000001b8u, 0x0004003du, + 0x00000006u, 0x000003eeu, 0x00000284u, 0x0004003du, 0x00000006u, 0x000003efu, 0x000003c8u, 0x00050080u, + 0x00000006u, 0x000003f0u, 0x000003eeu, 0x000003efu, 0x00050080u, 0x00000006u, 0x000003f1u, 0x000003edu, + 0x000003f0u, 0x00060041u, 0x000001bcu, 0x000003f2u, 0x000001b4u, 0x000000eeu, 0x000003f1u, 0x0004003du, + 0x00000006u, 0x000003f3u, 0x000003f2u, 0x0004007cu, 0x00000008u, 0x000003f4u, 0x000003f3u, 0x0003003eu, + 0x000003e8u, 0x000003f4u, 0x000200f9u, 0x000003eau, 0x000200f8u, 0x000003f5u, 0x00050041u, 0x00000194u, + 0x000003f6u, 0x00000192u, 0x000001b5u, 0x0004003du, 0x00000006u, 0x000003f7u, 0x000003f6u, 0x000500c2u, + 0x00000006u, 0x000003f8u, 0x000003f7u, 0x0000009bu, 0x0004003du, 0x00000006u, 0x000003f9u, 0x00000284u, + 0x0004003du, 0x00000006u, 0x000003fau, 0x000003c8u, 0x00050080u, 0x00000006u, 0x000003fbu, 0x000003f9u, + 0x000003fau, 0x00050080u, 0x00000006u, 0x000003fcu, 0x000003f8u, 0x000003fbu, 0x00060041u, 0x000001cbu, + 0x000003fdu, 0x000001c5u, 0x000000eeu, 0x000003fcu, 0x0004003du, 0x000001c1u, 0x000003feu, 0x000003fdu, + 0x00040071u, 0x00000006u, 0x000003ffu, 0x000003feu, 0x0003003eu, 0x00000400u, 0x000003ffu, 0x0003003eu, + 0x00000401u, 0x000001acu, 0x00060039u, 0x00000008u, 0x00000402u, 0x0000001bu, 0x00000400u, 0x00000401u, + 0x0003003eu, 0x000003e8u, 0x00000402u, 0x000200f9u, 0x000003eau, 0x000200f8u, 0x000003eau, 0x0004003du, + 0x00000008u, 0x00000403u, 0x000003e8u, 0x0004007cu, 0x00000006u, 0x00000404u, 0x00000403u, 0x00060041u, + 0x000001bcu, 0x00000405u, 0x000003deu, 0x000000eeu, 0x000003e6u, 0x0003003eu, 0x00000405u, 0x00000404u, + 0x000200f9u, 0x000003dau, 0x000200f8u, 0x00000406u, 0x00050041u, 0x00000194u, 0x0000040bu, 0x00000192u, + 0x000003dfu, 0x0004003du, 0x00000006u, 0x0000040cu, 0x0000040bu, 0x000500c2u, 0x00000006u, 0x0000040du, + 0x0000040cu, 0x0000009bu, 0x0004003du, 0x00000006u, 0x0000040eu, 0x0000029au, 0x0004003du, 0x00000006u, + 0x0000040fu, 0x000003c8u, 0x00050080u, 0x00000006u, 0x00000410u, 0x0000040eu, 0x0000040fu, 0x00050080u, + 0x00000006u, 0x00000411u, 0x0000040du, 0x00000410u, 0x000300f7u, 0x00000415u, 0x00000000u, 0x000400fau, + 0x00000412u, 0x00000414u, 0x00000420u, 0x000200f8u, 0x00000414u, 0x00050041u, 0x00000194u, 0x00000416u, + 0x00000192u, 0x000001b5u, 0x0004003du, 0x00000006u, 0x00000417u, 0x00000416u, 0x000500c2u, 0x00000006u, + 0x00000418u, 0x00000417u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x00000419u, 0x00000284u, 0x0004003du, + 0x00000006u, 0x0000041au, 0x000003c8u, 0x00050080u, 0x00000006u, 0x0000041bu, 0x00000419u, 0x0000041au, + 0x00050080u, 0x00000006u, 0x0000041cu, 0x00000418u, 0x0000041bu, 0x00060041u, 0x000001bcu, 0x0000041du, + 0x000001b4u, 0x000000eeu, 0x0000041cu, 0x0004003du, 0x00000006u, 0x0000041eu, 0x0000041du, 0x0004007cu, + 0x00000008u, 0x0000041fu, 0x0000041eu, 0x0003003eu, 0x00000413u, 0x0000041fu, 0x000200f9u, 0x00000415u, + 0x000200f8u, 0x00000420u, 0x00050041u, 0x00000194u, 0x00000421u, 0x00000192u, 0x000001b5u, 0x0004003du, + 0x00000006u, 0x00000422u, 0x00000421u, 0x000500c2u, 0x00000006u, 0x00000423u, 0x00000422u, 0x0000009bu, + 0x0004003du, 0x00000006u, 0x00000424u, 0x00000284u, 0x0004003du, 0x00000006u, 0x00000425u, 0x000003c8u, + 0x00050080u, 0x00000006u, 0x00000426u, 0x00000424u, 0x00000425u, 0x00050080u, 0x00000006u, 0x00000427u, + 0x00000423u, 0x00000426u, 0x00060041u, 0x000001cbu, 0x00000428u, 0x000001c5u, 0x000000eeu, 0x00000427u, + 0x0004003du, 0x000001c1u, 0x00000429u, 0x00000428u, 0x00040071u, 0x00000006u, 0x0000042au, 0x00000429u, + 0x0003003eu, 0x0000042bu, 0x0000042au, 0x0003003eu, 0x0000042cu, 0x000001acu, 0x00060039u, 0x00000008u, + 0x0000042du, 0x0000001bu, 0x0000042bu, 0x0000042cu, 0x0003003eu, 0x00000413u, 0x0000042du, 0x000200f9u, + 0x00000415u, 0x000200f8u, 0x00000415u, 0x0004003du, 0x00000008u, 0x0000042fu, 0x00000413u, 0x0003003eu, + 0x0000042eu, 0x0000042fu, 0x0003003eu, 0x00000430u, 0x000003d7u, 0x00060039u, 0x00000006u, 0x00000431u, + 0x00000020u, 0x0000042eu, 0x00000430u, 0x00040071u, 0x000001c1u, 0x00000432u, 0x00000431u, 0x00060041u, + 0x000001cbu, 0x00000433u, 0x0000040au, 0x000000eeu, 0x00000411u, 0x0003003eu, 0x00000433u, 0x00000432u, + 0x000200f9u, 0x000003dau, 0x000200f8u, 0x000003dau, 0x000200f9u, 0x000003d6u, 0x000200f8u, 0x000003d6u, + 0x000400fau, 0x00000240u, 0x000003d3u, 0x000003d5u, 0x000200f8u, 0x000003d5u, 0x000200f9u, 0x000003cdu, + 0x000200f8u, 0x000003cdu, 0x0004003du, 0x00000006u, 0x00000434u, 0x000003c8u, 0x00050080u, 0x00000006u, + 0x00000435u, 0x00000434u, 0x0000016au, 0x0003003eu, 0x000003c8u, 0x00000435u, 0x000200f9u, 0x000003cau, + 0x000200f8u, 0x000003ccu, 0x000200f9u, 0x000003c7u, 0x000200f8u, 0x000003c7u, 0x000100fdu, 0x00010038u, + 0x00050036u, 0x00000008u, 0x0000000bu, 0x00000000u, 0x00000009u, 0x00030037u, 0x00000007u, 0x0000000au, + 0x000200f8u, 0x0000000cu, 0x0004003du, 0x00000006u, 0x0000003eu, 0x0000000au, 0x000500c4u, 0x00000006u, + 0x00000041u, 0x0000003eu, 0x00000040u, 0x0004007cu, 0x00000008u, 0x00000042u, 0x00000041u, 0x000200feu, + 0x00000042u, 0x00010038u, 0x00050036u, 0x00000006u, 0x00000010u, 0x00000000u, 0x0000000eu, 0x00030037u, + 0x0000000du, 0x0000000fu, 0x000200f8u, 0x00000011u, 0x0004003bu, 0x00000007u, 0x00000045u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x0000005du, 0x00000007u, 0x0004003du, 0x00000008u, 0x00000046u, 0x0000000fu, + 0x0004007cu, 0x00000006u, 0x00000047u, 0x00000046u, 0x0003003eu, 0x00000045u, 0x00000047u, 0x0004003du, + 0x00000006u, 0x00000048u, 0x00000045u, 0x000500c7u, 0x00000006u, 0x0000004au, 0x00000048u, 0x00000049u, + 0x000500aau, 0x0000002du, 0x0000004bu, 0x0000004au, 0x00000049u, 0x000300f7u, 0x0000004du, 0x00000000u, + 0x000400fau, 0x0000004bu, 0x0000004cu, 0x0000004du, 0x000200f8u, 0x0000004cu, 0x0004003du, 0x00000006u, + 0x0000004eu, 0x00000045u, 0x000500c7u, 0x00000006u, 0x00000050u, 0x0000004eu, 0x0000004fu, 0x000500abu, + 0x0000002du, 0x00000052u, 0x00000050u, 0x00000051u, 0x000200f9u, 0x0000004du, 0x000200f8u, 0x0000004du, + 0x000700f5u, 0x0000002du, 0x00000053u, 0x0000004bu, 0x00000011u, 0x00000052u, 0x0000004cu, 0x000300f7u, + 0x00000055u, 0x00000000u, 0x000400fau, 0x00000053u, 0x00000054u, 0x00000055u, 0x000200f8u, 0x00000054u, + 0x0004003du, 0x00000006u, 0x00000056u, 0x00000045u, 0x000500c2u, 0x00000006u, 0x00000057u, 0x00000056u, + 0x00000040u, 0x000500c5u, 0x00000006u, 0x00000059u, 0x00000057u, 0x00000058u, 0x000500c7u, 0x00000006u, + 0x0000005bu, 0x00000059u, 0x0000005au, 0x000200feu, 0x0000005bu, 0x000200f8u, 0x00000055u, 0x0004003du, + 0x00000006u, 0x0000005fu, 0x00000045u, 0x000500c2u, 0x00000006u, 0x00000060u, 0x0000005fu, 0x00000040u, + 0x000500c7u, 0x00000006u, 0x00000062u, 0x00000060u, 0x00000061u, 0x00050080u, 0x00000006u, 0x00000063u, + 0x0000005eu, 0x00000062u, 0x0003003eu, 0x0000005du, 0x00000063u, 0x0004003du, 0x00000006u, 0x00000064u, + 0x00000045u, 0x0004003du, 0x00000006u, 0x00000065u, 0x0000005du, 0x00050080u, 0x00000006u, 0x00000066u, + 0x00000064u, 0x00000065u, 0x000500c2u, 0x00000006u, 0x00000067u, 0x00000066u, 0x00000040u, 0x000500c7u, + 0x00000006u, 0x00000068u, 0x00000067u, 0x0000005au, 0x000200feu, 0x00000068u, 0x00010038u, 0x00050036u, + 0x00000008u, 0x00000013u, 0x00000000u, 0x00000009u, 0x00030037u, 0x00000007u, 0x00000012u, 0x000200f8u, + 0x00000014u, 0x0004003bu, 0x00000007u, 0x0000006bu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000070u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000076u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000091u, + 0x00000007u, 0x0004003du, 0x00000006u, 0x0000006cu, 0x00000012u, 0x000500c7u, 0x00000006u, 0x0000006eu, + 0x0000006cu, 0x0000006du, 0x000500c4u, 0x00000006u, 0x0000006fu, 0x0000006eu, 0x00000040u, 0x0003003eu, + 0x0000006bu, 0x0000006fu, 0x0004003du, 0x00000006u, 0x00000071u, 0x00000012u, 0x000500c2u, 0x00000006u, + 0x00000073u, 0x00000071u, 0x00000072u, 0x000500c7u, 0x00000006u, 0x00000075u, 0x00000073u, 0x00000074u, + 0x0003003eu, 0x00000070u, 0x00000075u, 0x0004003du, 0x00000006u, 0x00000077u, 0x00000012u, 0x000500c7u, + 0x00000006u, 0x00000079u, 0x00000077u, 0x00000078u, 0x0003003eu, 0x00000076u, 0x00000079u, 0x0004003du, + 0x00000006u, 0x0000007au, 0x00000070u, 0x000500aau, 0x0000002du, 0x0000007bu, 0x0000007au, 0x00000074u, + 0x000300f7u, 0x0000007du, 0x00000000u, 0x000400fau, 0x0000007bu, 0x0000007cu, 0x0000007du, 0x000200f8u, + 0x0000007cu, 0x0004003du, 0x00000006u, 0x0000007eu, 0x0000006bu, 0x000500c5u, 0x00000006u, 0x0000007fu, + 0x0000007eu, 0x00000049u, 0x0004003du, 0x00000006u, 0x00000080u, 0x00000076u, 0x000500c4u, 0x00000006u, + 0x00000082u, 0x00000080u, 0x00000081u, 0x000500c5u, 0x00000006u, 0x00000083u, 0x0000007fu, 0x00000082u, + 0x0004007cu, 0x00000008u, 0x00000084u, 0x00000083u, 0x000200feu, 0x00000084u, 0x000200f8u, 0x0000007du, + 0x0004003du, 0x00000006u, 0x00000086u, 0x00000070u, 0x000500aau, 0x0000002du, 0x00000087u, 0x00000086u, + 0x00000051u, 0x000300f7u, 0x00000089u, 0x00000000u, 0x000400fau, 0x00000087u, 0x00000088u, 0x00000089u, + 0x000200f8u, 0x00000088u, 0x0004003du, 0x00000006u, 0x0000008au, 0x00000076u, 0x000500aau, 0x0000002du, + 0x0000008bu, 0x0000008au, 0x00000051u, 0x000300f7u, 0x0000008du, 0x00000000u, 0x000400fau, 0x0000008bu, + 0x0000008cu, 0x0000008du, 0x000200f8u, 0x0000008cu, 0x0004003du, 0x00000006u, 0x0000008eu, 0x0000006bu, + 0x0004007cu, 0x00000008u, 0x0000008fu, 0x0000008eu, 0x000200feu, 0x0000008fu, 0x000200f8u, 0x0000008du, + 0x0003003eu, 0x00000091u, 0x00000051u, 0x000200f9u, 0x00000092u, 0x000200f8u, 0x00000092u, 0x000400f6u, + 0x00000094u, 0x00000095u, 0x00000000u, 0x000200f9u, 0x00000096u, 0x000200f8u, 0x00000096u, 0x0004003du, + 0x00000006u, 0x00000097u, 0x00000076u, 0x000500c7u, 0x00000006u, 0x00000099u, 0x00000097u, 0x00000098u, + 0x000500aau, 0x0000002du, 0x0000009au, 0x00000099u, 0x00000051u, 0x000400fau, 0x0000009au, 0x00000093u, + 0x00000094u, 0x000200f8u, 0x00000093u, 0x0004003du, 0x00000006u, 0x0000009cu, 0x00000076u, 0x000500c4u, + 0x00000006u, 0x0000009du, 0x0000009cu, 0x0000009bu, 0x0003003eu, 0x00000076u, 0x0000009du, 0x0004003du, + 0x00000006u, 0x0000009eu, 0x00000091u, 0x00050080u, 0x00000006u, 0x0000009fu, 0x0000009eu, 0x00000061u, + 0x0003003eu, 0x00000091u, 0x0000009fu, 0x000200f9u, 0x00000095u, 0x000200f8u, 0x00000095u, 0x000200f9u, + 0x00000092u, 0x000200f8u, 0x00000094u, 0x0004003du, 0x00000006u, 0x000000a0u, 0x00000076u, 0x000500c7u, + 0x00000006u, 0x000000a1u, 0x000000a0u, 0x00000078u, 0x0003003eu, 0x00000076u, 0x000000a1u, 0x0004003du, + 0x00000006u, 0x000000a2u, 0x0000006bu, 0x0004003du, 0x00000006u, 0x000000a4u, 0x00000091u, 0x00050082u, + 0x00000006u, 0x000000a5u, 0x000000a3u, 0x000000a4u, 0x000500c4u, 0x00000006u, 0x000000a7u, 0x000000a5u, + 0x000000a6u, 0x000500c5u, 0x00000006u, 0x000000a8u, 0x000000a2u, 0x000000a7u, 0x0004003du, 0x00000006u, + 0x000000a9u, 0x00000076u, 0x000500c4u, 0x00000006u, 0x000000aau, 0x000000a9u, 0x00000081u, 0x000500c5u, + 0x00000006u, 0x000000abu, 0x000000a8u, 0x000000aau, 0x0004007cu, 0x00000008u, 0x000000acu, 0x000000abu, + 0x000200feu, 0x000000acu, 0x000200f8u, 0x00000089u, 0x0004003du, 0x00000006u, 0x000000aeu, 0x0000006bu, + 0x0004003du, 0x00000006u, 0x000000afu, 0x00000070u, 0x00050080u, 0x00000006u, 0x000000b1u, 0x000000afu, + 0x000000b0u, 0x000500c4u, 0x00000006u, 0x000000b2u, 0x000000b1u, 0x000000a6u, 0x000500c5u, 0x00000006u, + 0x000000b3u, 0x000000aeu, 0x000000b2u, 0x0004003du, 0x00000006u, 0x000000b4u, 0x00000076u, 0x000500c4u, + 0x00000006u, 0x000000b5u, 0x000000b4u, 0x00000081u, 0x000500c5u, 0x00000006u, 0x000000b6u, 0x000000b3u, + 0x000000b5u, 0x0004007cu, 0x00000008u, 0x000000b7u, 0x000000b6u, 0x000200feu, 0x000000b7u, 0x00010038u, + 0x00050036u, 0x00000006u, 0x00000016u, 0x00000000u, 0x0000000eu, 0x00030037u, 0x0000000du, 0x00000015u, + 0x000200f8u, 0x00000017u, 0x0004003bu, 0x00000007u, 0x000000bau, 0x00000007u, 0x0004003bu, 0x00000007u, + 0x000000bdu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000c1u, 0x00000007u, 0x0004003bu, 0x000000c6u, + 0x000000c7u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000ceu, 0x00000007u, 0x0004003bu, 0x00000007u, + 0x000000dau, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000fcu, 0x00000007u, 0x0004003bu, 0x00000007u, + 0x00000101u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000105u, 0x00000007u, 0x0004003bu, 0x00000007u, + 0x0000010bu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000127u, 0x00000007u, 0x0004003bu, 0x00000007u, + 0x0000012eu, 0x00000007u, 0x0004003du, 0x00000008u, 0x000000bbu, 0x00000015u, 0x0004007cu, 0x00000006u, + 0x000000bcu, 0x000000bbu, 0x0003003eu, 0x000000bau, 0x000000bcu, 0x0004003du, 0x00000006u, 0x000000beu, + 0x000000bau, 0x000500c2u, 0x00000006u, 0x000000bfu, 0x000000beu, 0x00000040u, 0x000500c7u, 0x00000006u, + 0x000000c0u, 0x000000bfu, 0x0000006du, 0x0003003eu, 0x000000bdu, 0x000000c0u, 0x0004003du, 0x00000006u, + 0x000000c2u, 0x000000bau, 0x000500c2u, 0x00000006u, 0x000000c3u, 0x000000c2u, 0x000000a6u, 0x000500c7u, + 0x00000006u, 0x000000c5u, 0x000000c3u, 0x000000c4u, 0x0003003eu, 0x000000c1u, 0x000000c5u, 0x0004003du, + 0x00000006u, 0x000000c8u, 0x000000c1u, 0x0004007cu, 0x0000003fu, 0x000000c9u, 0x000000c8u, 0x00050082u, + 0x0000003fu, 0x000000cbu, 0x000000c9u, 0x000000cau, 0x00050080u, 0x0000003fu, 0x000000cdu, 0x000000cbu, + 0x000000ccu, 0x0003003eu, 0x000000c7u, 0x000000cdu, 0x0004003du, 0x00000006u, 0x000000cfu, 0x000000bau, + 0x000500c7u, 0x00000006u, 0x000000d0u, 0x000000cfu, 0x0000004fu, 0x0003003eu, 0x000000ceu, 0x000000d0u, + 0x0004003du, 0x00000006u, 0x000000d1u, 0x000000c1u, 0x000500aau, 0x0000002du, 0x000000d2u, 0x000000d1u, + 0x000000c4u, 0x000300f7u, 0x000000d4u, 0x00000000u, 0x000400fau, 0x000000d2u, 0x000000d3u, 0x000000d4u, + 0x000200f8u, 0x000000d3u, 0x0004003du, 0x00000006u, 0x000000d5u, 0x000000bdu, 0x000500c5u, 0x00000006u, + 0x000000d7u, 0x000000d5u, 0x000000d6u, 0x0004003du, 0x00000006u, 0x000000d8u, 0x000000ceu, 0x000500abu, + 0x0000002du, 0x000000d9u, 0x000000d8u, 0x00000051u, 0x000300f7u, 0x000000dcu, 0x00000000u, 0x000400fau, + 0x000000d9u, 0x000000dbu, 0x000000e1u, 0x000200f8u, 0x000000dbu, 0x0004003du, 0x00000006u, 0x000000deu, + 0x000000ceu, 0x000500c2u, 0x00000006u, 0x000000dfu, 0x000000deu, 0x00000081u, 0x000500c5u, 0x00000006u, + 0x000000e0u, 0x000000ddu, 0x000000dfu, 0x0003003eu, 0x000000dau, 0x000000e0u, 0x000200f9u, 0x000000dcu, + 0x000200f8u, 0x000000e1u, 0x0003003eu, 0x000000dau, 0x00000051u, 0x000200f9u, 0x000000dcu, 0x000200f8u, + 0x000000dcu, 0x0004003du, 0x00000006u, 0x000000e2u, 0x000000dau, 0x000500c5u, 0x00000006u, 0x000000e3u, + 0x000000d7u, 0x000000e2u, 0x000200feu, 0x000000e3u, 0x000200f8u, 0x000000d4u, 0x0004003du, 0x0000003fu, + 0x000000e5u, 0x000000c7u, 0x000500afu, 0x0000002du, 0x000000e7u, 0x000000e5u, 0x000000e6u, 0x000300f7u, + 0x000000e9u, 0x00000000u, 0x000400fau, 0x000000e7u, 0x000000e8u, 0x000000e9u, 0x000200f8u, 0x000000e8u, + 0x0004003du, 0x00000006u, 0x000000eau, 0x000000bdu, 0x000500c5u, 0x00000006u, 0x000000ebu, 0x000000eau, + 0x000000d6u, 0x000200feu, 0x000000ebu, 0x000200f8u, 0x000000e9u, 0x0004003du, 0x0000003fu, 0x000000edu, + 0x000000c7u, 0x000500b3u, 0x0000002du, 0x000000efu, 0x000000edu, 0x000000eeu, 0x000300f7u, 0x000000f1u, + 0x00000000u, 0x000400fau, 0x000000efu, 0x000000f0u, 0x000000f1u, 0x000200f8u, 0x000000f0u, 0x0004003du, + 0x0000003fu, 0x000000f2u, 0x000000c7u, 0x000500b1u, 0x0000002du, 0x000000f4u, 0x000000f2u, 0x000000f3u, + 0x000300f7u, 0x000000f6u, 0x00000000u, 0x000400fau, 0x000000f4u, 0x000000f5u, 0x000000f6u, 0x000200f8u, + 0x000000f5u, 0x0004003du, 0x00000006u, 0x000000f7u, 0x000000bdu, 0x000200feu, 0x000000f7u, 0x000200f8u, + 0x000000f6u, 0x0004003du, 0x00000006u, 0x000000fau, 0x000000ceu, 0x000500c5u, 0x00000006u, 0x000000fbu, + 0x000000fau, 0x000000f9u, 0x0003003eu, 0x000000ceu, 0x000000fbu, 0x0004003du, 0x0000003fu, 0x000000feu, + 0x000000c7u, 0x00050082u, 0x0000003fu, 0x000000ffu, 0x000000fdu, 0x000000feu, 0x0004007cu, 0x00000006u, + 0x00000100u, 0x000000ffu, 0x0003003eu, 0x000000fcu, 0x00000100u, 0x0004003du, 0x00000006u, 0x00000102u, + 0x000000ceu, 0x0004003du, 0x00000006u, 0x00000103u, 0x000000fcu, 0x000500c2u, 0x00000006u, 0x00000104u, + 0x00000102u, 0x00000103u, 0x0003003eu, 0x00000101u, 0x00000104u, 0x0004003du, 0x00000006u, 0x00000106u, + 0x000000ceu, 0x0004003du, 0x00000006u, 0x00000107u, 0x000000fcu, 0x000500c4u, 0x00000006u, 0x00000108u, + 0x00000061u, 0x00000107u, 0x00050082u, 0x00000006u, 0x00000109u, 0x00000108u, 0x00000061u, 0x000500c7u, + 0x00000006u, 0x0000010au, 0x00000106u, 0x00000109u, 0x0003003eu, 0x00000105u, 0x0000010au, 0x0004003du, + 0x00000006u, 0x0000010cu, 0x000000fcu, 0x00050082u, 0x00000006u, 0x0000010du, 0x0000010cu, 0x00000061u, + 0x000500c4u, 0x00000006u, 0x0000010eu, 0x00000061u, 0x0000010du, 0x0003003eu, 0x0000010bu, 0x0000010eu, + 0x0004003du, 0x00000006u, 0x0000010fu, 0x00000105u, 0x0004003du, 0x00000006u, 0x00000110u, 0x0000010bu, + 0x000500acu, 0x0000002du, 0x00000111u, 0x0000010fu, 0x00000110u, 0x000400a8u, 0x0000002du, 0x00000112u, + 0x00000111u, 0x000300f7u, 0x00000114u, 0x00000000u, 0x000400fau, 0x00000112u, 0x00000113u, 0x00000114u, + 0x000200f8u, 0x00000113u, 0x0004003du, 0x00000006u, 0x00000115u, 0x00000105u, 0x0004003du, 0x00000006u, + 0x00000116u, 0x0000010bu, 0x000500aau, 0x0000002du, 0x00000117u, 0x00000115u, 0x00000116u, 0x000300f7u, + 0x00000119u, 0x00000000u, 0x000400fau, 0x00000117u, 0x00000118u, 0x00000119u, 0x000200f8u, 0x00000118u, + 0x0004003du, 0x00000006u, 0x0000011au, 0x00000101u, 0x000500c7u, 0x00000006u, 0x0000011bu, 0x0000011au, + 0x00000061u, 0x000500abu, 0x0000002du, 0x0000011cu, 0x0000011bu, 0x00000051u, 0x000200f9u, 0x00000119u, + 0x000200f8u, 0x00000119u, 0x000700f5u, 0x0000002du, 0x0000011du, 0x00000117u, 0x00000113u, 0x0000011cu, + 0x00000118u, 0x000200f9u, 0x00000114u, 0x000200f8u, 0x00000114u, 0x000700f5u, 0x0000002du, 0x0000011eu, + 0x00000111u, 0x000000f6u, 0x0000011du, 0x00000119u, 0x000300f7u, 0x00000120u, 0x00000000u, 0x000400fau, + 0x0000011eu, 0x0000011fu, 0x00000120u, 0x000200f8u, 0x0000011fu, 0x0004003du, 0x00000006u, 0x00000121u, + 0x00000101u, 0x00050080u, 0x00000006u, 0x00000122u, 0x00000121u, 0x00000061u, 0x0003003eu, 0x00000101u, + 0x00000122u, 0x000200f9u, 0x00000120u, 0x000200f8u, 0x00000120u, 0x0004003du, 0x00000006u, 0x00000123u, + 0x000000bdu, 0x0004003du, 0x00000006u, 0x00000124u, 0x00000101u, 0x000500c5u, 0x00000006u, 0x00000125u, + 0x00000123u, 0x00000124u, 0x000200feu, 0x00000125u, 0x000200f8u, 0x000000f1u, 0x0004003du, 0x0000003fu, + 0x00000128u, 0x000000c7u, 0x0004007cu, 0x00000006u, 0x00000129u, 0x00000128u, 0x000500c4u, 0x00000006u, + 0x0000012au, 0x00000129u, 0x00000072u, 0x0004003du, 0x00000006u, 0x0000012bu, 0x000000ceu, 0x000500c2u, + 0x00000006u, 0x0000012cu, 0x0000012bu, 0x00000081u, 0x000500c5u, 0x00000006u, 0x0000012du, 0x0000012au, + 0x0000012cu, 0x0003003eu, 0x00000127u, 0x0000012du, 0x0004003du, 0x00000006u, 0x0000012fu, 0x000000ceu, + 0x000500c7u, 0x00000006u, 0x00000131u, 0x0000012fu, 0x00000130u, 0x0003003eu, 0x0000012eu, 0x00000131u, + 0x0004003du, 0x00000006u, 0x00000132u, 0x0000012eu, 0x000500acu, 0x0000002du, 0x00000134u, 0x00000132u, + 0x00000133u, 0x000400a8u, 0x0000002du, 0x00000135u, 0x00000134u, 0x000300f7u, 0x00000137u, 0x00000000u, + 0x000400fau, 0x00000135u, 0x00000136u, 0x00000137u, 0x000200f8u, 0x00000136u, 0x0004003du, 0x00000006u, + 0x00000138u, 0x0000012eu, 0x000500aau, 0x0000002du, 0x00000139u, 0x00000138u, 0x00000133u, 0x000300f7u, + 0x0000013bu, 0x00000000u, 0x000400fau, 0x00000139u, 0x0000013au, 0x0000013bu, 0x000200f8u, 0x0000013au, + 0x0004003du, 0x00000006u, 0x0000013cu, 0x00000127u, 0x000500c7u, 0x00000006u, 0x0000013du, 0x0000013cu, + 0x00000061u, 0x000500abu, 0x0000002du, 0x0000013eu, 0x0000013du, 0x00000051u, 0x000200f9u, 0x0000013bu, + 0x000200f8u, 0x0000013bu, 0x000700f5u, 0x0000002du, 0x0000013fu, 0x00000139u, 0x00000136u, 0x0000013eu, + 0x0000013au, 0x000200f9u, 0x00000137u, 0x000200f8u, 0x00000137u, 0x000700f5u, 0x0000002du, 0x00000140u, + 0x00000134u, 0x000000f1u, 0x0000013fu, 0x0000013bu, 0x000300f7u, 0x00000142u, 0x00000000u, 0x000400fau, + 0x00000140u, 0x00000141u, 0x00000142u, 0x000200f8u, 0x00000141u, 0x0004003du, 0x00000006u, 0x00000143u, + 0x00000127u, 0x00050080u, 0x00000006u, 0x00000144u, 0x00000143u, 0x00000061u, 0x0003003eu, 0x00000127u, + 0x00000144u, 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000142u, 0x0004003du, 0x00000006u, 0x00000145u, + 0x000000bdu, 0x0004003du, 0x00000006u, 0x00000146u, 0x00000127u, 0x000500c5u, 0x00000006u, 0x00000147u, + 0x00000145u, 0x00000146u, 0x000200feu, 0x00000147u, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000001bu, + 0x00000000u, 0x00000018u, 0x00030037u, 0x00000007u, 0x00000019u, 0x00030037u, 0x00000007u, 0x0000001au, + 0x000200f8u, 0x0000001cu, 0x0004003bu, 0x0000000du, 0x0000014cu, 0x00000007u, 0x0004003bu, 0x00000007u, + 0x0000014fu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000153u, 0x00000007u, 0x0004003du, 0x00000006u, + 0x0000014au, 0x0000001au, 0x000500aau, 0x0000002du, 0x0000014bu, 0x0000014au, 0x00000061u, 0x000300f7u, + 0x0000014eu, 0x00000000u, 0x000400fau, 0x0000014bu, 0x0000014du, 0x00000152u, 0x000200f8u, 0x0000014du, + 0x0004003du, 0x00000006u, 0x00000150u, 0x00000019u, 0x0003003eu, 0x0000014fu, 0x00000150u, 0x00050039u, + 0x00000008u, 0x00000151u, 0x00000013u, 0x0000014fu, 0x0003003eu, 0x0000014cu, 0x00000151u, 0x000200f9u, + 0x0000014eu, 0x000200f8u, 0x00000152u, 0x0004003du, 0x00000006u, 0x00000154u, 0x00000019u, 0x0003003eu, + 0x00000153u, 0x00000154u, 0x00050039u, 0x00000008u, 0x00000155u, 0x0000000bu, 0x00000153u, 0x0003003eu, + 0x0000014cu, 0x00000155u, 0x000200f9u, 0x0000014eu, 0x000200f8u, 0x0000014eu, 0x0004003du, 0x00000008u, + 0x00000156u, 0x0000014cu, 0x000200feu, 0x00000156u, 0x00010038u, 0x00050036u, 0x00000006u, 0x00000020u, + 0x00000000u, 0x0000001du, 0x00030037u, 0x0000000du, 0x0000001eu, 0x00030037u, 0x00000007u, 0x0000001fu, + 0x000200f8u, 0x00000021u, 0x0004003bu, 0x00000007u, 0x0000015bu, 0x00000007u, 0x0004003bu, 0x0000000du, + 0x0000015eu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000162u, 0x00000007u, 0x0004003du, 0x00000006u, + 0x00000159u, 0x0000001fu, 0x000500aau, 0x0000002du, 0x0000015au, 0x00000159u, 0x00000061u, 0x000300f7u, + 0x0000015du, 0x00000000u, 0x000400fau, 0x0000015au, 0x0000015cu, 0x00000161u, 0x000200f8u, 0x0000015cu, + 0x0004003du, 0x00000008u, 0x0000015fu, 0x0000001eu, 0x0003003eu, 0x0000015eu, 0x0000015fu, 0x00050039u, + 0x00000006u, 0x00000160u, 0x00000016u, 0x0000015eu, 0x0003003eu, 0x0000015bu, 0x00000160u, 0x000200f9u, + 0x0000015du, 0x000200f8u, 0x00000161u, 0x0004003du, 0x00000008u, 0x00000163u, 0x0000001eu, 0x0003003eu, + 0x00000162u, 0x00000163u, 0x00050039u, 0x00000006u, 0x00000164u, 0x00000010u, 0x00000162u, 0x0003003eu, + 0x0000015bu, 0x00000164u, 0x000200f9u, 0x0000015du, 0x000200f8u, 0x0000015du, 0x0004003du, 0x00000006u, + 0x00000165u, 0x0000015bu, 0x000200feu, 0x00000165u, 0x00010038u, 0x00050036u, 0x00000008u, 0x00000025u, + 0x00000000u, 0x00000022u, 0x00030037u, 0x00000007u, 0x00000023u, 0x00030037u, 0x0000000du, 0x00000024u, + 0x000200f8u, 0x00000026u, 0x0004003bu, 0x00000007u, 0x00000172u, 0x00000007u, 0x000400e0u, 0x00000168u, + 0x00000168u, 0x00000169u, 0x0004003du, 0x00000006u, 0x0000016eu, 0x00000023u, 0x0004003du, 0x00000008u, + 0x0000016fu, 0x00000024u, 0x00050041u, 0x00000170u, 0x00000171u, 0x0000016du, 0x0000016eu, 0x0003003eu, + 0x00000171u, 0x0000016fu, 0x000400e0u, 0x00000168u, 0x00000168u, 0x00000169u, 0x0003003eu, 0x00000172u, + 0x00000058u, 0x000200f9u, 0x00000173u, 0x000200f8u, 0x00000173u, 0x000400f6u, 0x00000175u, 0x00000176u, + 0x00000000u, 0x000200f9u, 0x00000177u, 0x000200f8u, 0x00000177u, 0x0004003du, 0x00000006u, 0x00000178u, + 0x00000172u, 0x000500acu, 0x0000002du, 0x00000179u, 0x00000178u, 0x00000051u, 0x000400fau, 0x00000179u, + 0x00000174u, 0x00000175u, 0x000200f8u, 0x00000174u, 0x0004003du, 0x00000006u, 0x0000017au, 0x00000023u, + 0x0004003du, 0x00000006u, 0x0000017bu, 0x00000172u, 0x000500b0u, 0x0000002du, 0x0000017cu, 0x0000017au, + 0x0000017bu, 0x000300f7u, 0x0000017eu, 0x00000000u, 0x000400fau, 0x0000017cu, 0x0000017du, 0x0000017eu, + 0x000200f8u, 0x0000017du, 0x0004003du, 0x00000006u, 0x0000017fu, 0x00000023u, 0x0004003du, 0x00000006u, + 0x00000180u, 0x00000023u, 0x0004003du, 0x00000006u, 0x00000181u, 0x00000172u, 0x00050080u, 0x00000006u, + 0x00000182u, 0x00000180u, 0x00000181u, 0x00050041u, 0x00000170u, 0x00000183u, 0x0000016du, 0x00000182u, + 0x0004003du, 0x00000008u, 0x00000184u, 0x00000183u, 0x00050041u, 0x00000170u, 0x00000185u, 0x0000016du, + 0x0000017fu, 0x0004003du, 0x00000008u, 0x00000186u, 0x00000185u, 0x00050081u, 0x00000008u, 0x00000187u, + 0x00000186u, 0x00000184u, 0x00050041u, 0x00000170u, 0x00000188u, 0x0000016du, 0x0000017fu, 0x0003003eu, + 0x00000188u, 0x00000187u, 0x000200f9u, 0x0000017eu, 0x000200f8u, 0x0000017eu, 0x000400e0u, 0x00000168u, + 0x00000168u, 0x00000169u, 0x000200f9u, 0x00000176u, 0x000200f8u, 0x00000176u, 0x0004003du, 0x00000006u, + 0x00000189u, 0x00000172u, 0x000500c2u, 0x00000006u, 0x0000018au, 0x00000189u, 0x0000009bu, 0x0003003eu, + 0x00000172u, 0x0000018au, 0x000200f9u, 0x00000173u, 0x000200f8u, 0x00000175u, 0x00050041u, 0x00000170u, + 0x0000018bu, 0x0000016du, 0x000000eeu, 0x0004003du, 0x00000008u, 0x0000018cu, 0x0000018bu, 0x000200feu, + 0x0000018cu, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000002bu, 0x00000000u, 0x00000027u, 0x00030037u, + 0x0000000du, 0x00000028u, 0x00030037u, 0x0000000du, 0x00000029u, 0x00030037u, 0x0000000du, 0x0000002au, + 0x000200f8u, 0x0000002cu, 0x0004003bu, 0x0000000du, 0x0000018fu, 0x00000007u, 0x0004003bu, 0x0000000du, + 0x00000198u, 0x00000007u, 0x00050041u, 0x00000194u, 0x00000195u, 0x00000192u, 0x00000193u, 0x0004003du, + 0x00000006u, 0x00000196u, 0x00000195u, 0x000500abu, 0x0000002du, 0x00000197u, 0x00000196u, 0x00000051u, + 0x000300f7u, 0x0000019au, 0x00000000u, 0x000400fau, 0x00000197u, 0x00000199u, 0x0000019eu, 0x000200f8u, + 0x00000199u, 0x0004003du, 0x00000008u, 0x0000019bu, 0x0000002au, 0x00050081u, 0x00000008u, 0x0000019du, + 0x0000019bu, 0x0000019cu, 0x0003003eu, 0x00000198u, 0x0000019du, 0x000200f9u, 0x0000019au, 0x000200f8u, + 0x0000019eu, 0x0004003du, 0x00000008u, 0x0000019fu, 0x0000002au, 0x0003003eu, 0x00000198u, 0x0000019fu, + 0x000200f9u, 0x0000019au, 0x000200f8u, 0x0000019au, 0x0004003du, 0x00000008u, 0x000001a0u, 0x00000198u, + 0x0003003eu, 0x0000018fu, 0x000001a0u, 0x0004003du, 0x00000008u, 0x000001a1u, 0x00000028u, 0x0004003du, + 0x00000008u, 0x000001a2u, 0x00000029u, 0x00050085u, 0x00000008u, 0x000001a3u, 0x000001a1u, 0x000001a2u, + 0x0004003du, 0x00000008u, 0x000001a4u, 0x0000018fu, 0x00050085u, 0x00000008u, 0x000001a5u, 0x000001a3u, + 0x000001a4u, 0x000200feu, 0x000001a5u, 0x00010038u, 0x00050036u, 0x00000008u, 0x00000032u, 0x00000000u, + 0x0000002fu, 0x00030037u, 0x0000002eu, 0x00000030u, 0x00030037u, 0x00000007u, 0x00000031u, 0x000200f8u, + 0x00000033u, 0x0004003bu, 0x0000000du, 0x000001a9u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000001aeu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001cfu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001d0u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000001d5u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001f2u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001f3u, 0x00000007u, 0x0004003du, 0x0000002du, 0x000001a8u, + 0x00000030u, 0x000300f7u, 0x000001abu, 0x00000000u, 0x000400fau, 0x000001a8u, 0x000001aau, 0x000001d3u, + 0x000200f8u, 0x000001aau, 0x000300f7u, 0x000001b0u, 0x00000000u, 0x000400fau, 0x000001adu, 0x000001afu, + 0x000001c0u, 0x000200f8u, 0x000001afu, 0x00050041u, 0x00000194u, 0x000001b6u, 0x00000192u, 0x000001b5u, + 0x0004003du, 0x00000006u, 0x000001b7u, 0x000001b6u, 0x000500c2u, 0x00000006u, 0x000001b9u, 0x000001b7u, + 0x000001b8u, 0x0004003du, 0x00000006u, 0x000001bau, 0x00000031u, 0x00050080u, 0x00000006u, 0x000001bbu, + 0x000001b9u, 0x000001bau, 0x00060041u, 0x000001bcu, 0x000001bdu, 0x000001b4u, 0x000000eeu, 0x000001bbu, + 0x0004003du, 0x00000006u, 0x000001beu, 0x000001bdu, 0x0004007cu, 0x00000008u, 0x000001bfu, 0x000001beu, + 0x0003003eu, 0x000001aeu, 0x000001bfu, 0x000200f9u, 0x000001b0u, 0x000200f8u, 0x000001c0u, 0x00050041u, + 0x00000194u, 0x000001c6u, 0x00000192u, 0x000001b5u, 0x0004003du, 0x00000006u, 0x000001c7u, 0x000001c6u, + 0x000500c2u, 0x00000006u, 0x000001c8u, 0x000001c7u, 0x0000009bu, 0x0004003du, 0x00000006u, 0x000001c9u, + 0x00000031u, 0x00050080u, 0x00000006u, 0x000001cau, 0x000001c8u, 0x000001c9u, 0x00060041u, 0x000001cbu, + 0x000001ccu, 0x000001c5u, 0x000000eeu, 0x000001cau, 0x0004003du, 0x000001c1u, 0x000001cdu, 0x000001ccu, + 0x00040071u, 0x00000006u, 0x000001ceu, 0x000001cdu, 0x0003003eu, 0x000001cfu, 0x000001ceu, 0x0003003eu, + 0x000001d0u, 0x000001acu, 0x00060039u, 0x00000008u, 0x000001d1u, 0x0000001bu, 0x000001cfu, 0x000001d0u, + 0x0003003eu, 0x000001aeu, 0x000001d1u, 0x000200f9u, 0x000001b0u, 0x000200f8u, 0x000001b0u, 0x0004003du, + 0x00000008u, 0x000001d2u, 0x000001aeu, 0x0003003eu, 0x000001a9u, 0x000001d2u, 0x000200f9u, 0x000001abu, + 0x000200f8u, 0x000001d3u, 0x000300f7u, 0x000001d7u, 0x00000000u, 0x000400fau, 0x000001d4u, 0x000001d6u, + 0x000001e5u, 0x000200f8u, 0x000001d6u, 0x00050041u, 0x00000194u, 0x000001ddu, 0x00000192u, 0x000001dcu, + 0x0004003du, 0x00000006u, 0x000001deu, 0x000001ddu, 0x000500c2u, 0x00000006u, 0x000001dfu, 0x000001deu, + 0x000001b8u, 0x0004003du, 0x00000006u, 0x000001e0u, 0x00000031u, 0x00050080u, 0x00000006u, 0x000001e1u, + 0x000001dfu, 0x000001e0u, 0x00060041u, 0x000001bcu, 0x000001e2u, 0x000001dbu, 0x000000eeu, 0x000001e1u, + 0x0004003du, 0x00000006u, 0x000001e3u, 0x000001e2u, 0x0004007cu, 0x00000008u, 0x000001e4u, 0x000001e3u, + 0x0003003eu, 0x000001d5u, 0x000001e4u, 0x000200f9u, 0x000001d7u, 0x000200f8u, 0x000001e5u, 0x00050041u, + 0x00000194u, 0x000001eau, 0x00000192u, 0x000001dcu, 0x0004003du, 0x00000006u, 0x000001ebu, 0x000001eau, + 0x000500c2u, 0x00000006u, 0x000001ecu, 0x000001ebu, 0x0000009bu, 0x0004003du, 0x00000006u, 0x000001edu, + 0x00000031u, 0x00050080u, 0x00000006u, 0x000001eeu, 0x000001ecu, 0x000001edu, 0x00060041u, 0x000001cbu, + 0x000001efu, 0x000001e9u, 0x000000eeu, 0x000001eeu, 0x0004003du, 0x000001c1u, 0x000001f0u, 0x000001efu, + 0x00040071u, 0x00000006u, 0x000001f1u, 0x000001f0u, 0x0003003eu, 0x000001f2u, 0x000001f1u, 0x0003003eu, + 0x000001f3u, 0x000001acu, 0x00060039u, 0x00000008u, 0x000001f4u, 0x0000001bu, 0x000001f2u, 0x000001f3u, + 0x0003003eu, 0x000001d5u, 0x000001f4u, 0x000200f9u, 0x000001d7u, 0x000200f8u, 0x000001d7u, 0x0004003du, + 0x00000008u, 0x000001f5u, 0x000001d5u, 0x0003003eu, 0x000001a9u, 0x000001f5u, 0x000200f9u, 0x000001abu, + 0x000200f8u, 0x000001abu, 0x0004003du, 0x00000008u, 0x000001f6u, 0x000001a9u, 0x000200feu, 0x000001f6u, + 0x00010038u, 0x00050036u, 0x00000008u, 0x00000036u, 0x00000000u, 0x0000002fu, 0x00030037u, 0x0000002eu, + 0x00000034u, 0x00030037u, 0x00000007u, 0x00000035u, 0x000200f8u, 0x00000037u, 0x0004003bu, 0x0000000du, + 0x000001fau, 0x00000007u, 0x0004003du, 0x0000002du, 0x000001f9u, 0x00000034u, 0x000300f7u, 0x000001fcu, + 0x00000000u, 0x000400fau, 0x000001f9u, 0x000001fbu, 0x00000209u, 0x000200f8u, 0x000001fbu, 0x00050041u, + 0x00000194u, 0x00000201u, 0x00000192u, 0x00000081u, 0x0004003du, 0x00000006u, 0x00000202u, 0x00000201u, + 0x000500c2u, 0x00000006u, 0x00000203u, 0x00000202u, 0x000001b8u, 0x0004003du, 0x00000006u, 0x00000204u, + 0x00000035u, 0x00050080u, 0x00000006u, 0x00000205u, 0x00000203u, 0x00000204u, 0x00060041u, 0x000001bcu, + 0x00000206u, 0x00000200u, 0x000000eeu, 0x00000205u, 0x0004003du, 0x00000006u, 0x00000207u, 0x00000206u, + 0x0004007cu, 0x00000008u, 0x00000208u, 0x00000207u, 0x0003003eu, 0x000001fau, 0x00000208u, 0x000200f9u, + 0x000001fcu, 0x000200f8u, 0x00000209u, 0x00050041u, 0x00000194u, 0x0000020eu, 0x00000192u, 0x000000fdu, + 0x0004003du, 0x00000006u, 0x0000020fu, 0x0000020eu, 0x000500c2u, 0x00000006u, 0x00000210u, 0x0000020fu, + 0x000001b8u, 0x0004003du, 0x00000006u, 0x00000211u, 0x00000035u, 0x00050080u, 0x00000006u, 0x00000212u, + 0x00000210u, 0x00000211u, 0x00060041u, 0x000001bcu, 0x00000213u, 0x0000020du, 0x000000eeu, 0x00000212u, + 0x0004003du, 0x00000006u, 0x00000214u, 0x00000213u, 0x0004007cu, 0x00000008u, 0x00000215u, 0x00000214u, + 0x0003003eu, 0x000001fau, 0x00000215u, 0x000200f9u, 0x000001fcu, 0x000200f8u, 0x000001fcu, 0x0004003du, + 0x00000008u, 0x00000216u, 0x000001fau, 0x000200feu, 0x00000216u, 0x00010038u, 0x00050036u, 0x00000002u, + 0x0000003cu, 0x00000000u, 0x00000038u, 0x00030037u, 0x0000002eu, 0x00000039u, 0x00030037u, 0x00000007u, + 0x0000003au, 0x00030037u, 0x0000000du, 0x0000003bu, 0x000200f8u, 0x0000003du, 0x0004003bu, 0x0000000du, + 0x0000023au, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000023cu, 0x00000007u, 0x0004003bu, 0x0000000du, + 0x00000260u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000262u, 0x00000007u, 0x0004003du, 0x0000002du, + 0x00000219u, 0x00000039u, 0x000300f7u, 0x0000021bu, 0x00000000u, 0x000400fau, 0x00000219u, 0x0000021au, + 0x00000241u, 0x000200f8u, 0x0000021au, 0x000200f9u, 0x0000021cu, 0x000200f8u, 0x0000021cu, 0x000400f6u, + 0x0000021eu, 0x0000021fu, 0x00000000u, 0x000200f9u, 0x0000021du, 0x000200f8u, 0x0000021du, 0x000300f7u, + 0x00000223u, 0x00000000u, 0x000400fau, 0x00000221u, 0x00000222u, 0x00000230u, 0x000200f8u, 0x00000222u, + 0x00050041u, 0x00000194u, 0x00000228u, 0x00000192u, 0x00000072u, 0x0004003du, 0x00000006u, 0x00000229u, + 0x00000228u, 0x000500c2u, 0x00000006u, 0x0000022au, 0x00000229u, 0x000001b8u, 0x0004003du, 0x00000006u, + 0x0000022bu, 0x0000003au, 0x00050080u, 0x00000006u, 0x0000022cu, 0x0000022au, 0x0000022bu, 0x0004003du, + 0x00000008u, 0x0000022du, 0x0000003bu, 0x0004007cu, 0x00000006u, 0x0000022eu, 0x0000022du, 0x00060041u, + 0x000001bcu, 0x0000022fu, 0x00000227u, 0x000000eeu, 0x0000022cu, 0x0003003eu, 0x0000022fu, 0x0000022eu, + 0x000200f9u, 0x00000223u, 0x000200f8u, 0x00000230u, 0x00050041u, 0x00000194u, 0x00000235u, 0x00000192u, + 0x00000072u, 0x0004003du, 0x00000006u, 0x00000236u, 0x00000235u, 0x000500c2u, 0x00000006u, 0x00000237u, + 0x00000236u, 0x0000009bu, 0x0004003du, 0x00000006u, 0x00000238u, 0x0000003au, 0x00050080u, 0x00000006u, + 0x00000239u, 0x00000237u, 0x00000238u, 0x0004003du, 0x00000008u, 0x0000023bu, 0x0000003bu, 0x0003003eu, + 0x0000023au, 0x0000023bu, 0x0003003eu, 0x0000023cu, 0x00000220u, 0x00060039u, 0x00000006u, 0x0000023du, + 0x00000020u, 0x0000023au, 0x0000023cu, 0x00040071u, 0x000001c1u, 0x0000023eu, 0x0000023du, 0x00060041u, + 0x000001cbu, 0x0000023fu, 0x00000234u, 0x000000eeu, 0x00000239u, 0x0003003eu, 0x0000023fu, 0x0000023eu, + 0x000200f9u, 0x00000223u, 0x000200f8u, 0x00000223u, 0x000200f9u, 0x0000021fu, 0x000200f8u, 0x0000021fu, + 0x000400fau, 0x00000240u, 0x0000021cu, 0x0000021eu, 0x000200f8u, 0x0000021eu, 0x000200f9u, 0x0000021bu, + 0x000200f8u, 0x00000241u, 0x000200f9u, 0x00000242u, 0x000200f8u, 0x00000242u, 0x000400f6u, 0x00000244u, + 0x00000245u, 0x00000000u, 0x000200f9u, 0x00000243u, 0x000200f8u, 0x00000243u, 0x000300f7u, 0x00000248u, + 0x00000000u, 0x000400fau, 0x00000246u, 0x00000247u, 0x00000256u, 0x000200f8u, 0x00000247u, 0x00050041u, + 0x00000194u, 0x0000024eu, 0x00000192u, 0x0000024du, 0x0004003du, 0x00000006u, 0x0000024fu, 0x0000024eu, + 0x000500c2u, 0x00000006u, 0x00000250u, 0x0000024fu, 0x000001b8u, 0x0004003du, 0x00000006u, 0x00000251u, + 0x0000003au, 0x00050080u, 0x00000006u, 0x00000252u, 0x00000250u, 0x00000251u, 0x0004003du, 0x00000008u, + 0x00000253u, 0x0000003bu, 0x0004007cu, 0x00000006u, 0x00000254u, 0x00000253u, 0x00060041u, 0x000001bcu, + 0x00000255u, 0x0000024cu, 0x000000eeu, 0x00000252u, 0x0003003eu, 0x00000255u, 0x00000254u, 0x000200f9u, + 0x00000248u, 0x000200f8u, 0x00000256u, 0x00050041u, 0x00000194u, 0x0000025bu, 0x00000192u, 0x0000024du, + 0x0004003du, 0x00000006u, 0x0000025cu, 0x0000025bu, 0x000500c2u, 0x00000006u, 0x0000025du, 0x0000025cu, + 0x0000009bu, 0x0004003du, 0x00000006u, 0x0000025eu, 0x0000003au, 0x00050080u, 0x00000006u, 0x0000025fu, + 0x0000025du, 0x0000025eu, 0x0004003du, 0x00000008u, 0x00000261u, 0x0000003bu, 0x0003003eu, 0x00000260u, + 0x00000261u, 0x0003003eu, 0x00000262u, 0x00000220u, 0x00060039u, 0x00000006u, 0x00000263u, 0x00000020u, + 0x00000260u, 0x00000262u, 0x00040071u, 0x000001c1u, 0x00000264u, 0x00000263u, 0x00060041u, 0x000001cbu, + 0x00000265u, 0x0000025au, 0x000000eeu, 0x0000025fu, 0x0003003eu, 0x00000265u, 0x00000264u, 0x000200f9u, + 0x00000248u, 0x000200f8u, 0x00000248u, 0x000200f9u, 0x00000245u, 0x000200f8u, 0x00000245u, 0x000400fau, + 0x00000240u, 0x00000242u, 0x00000244u, 0x000200f8u, 0x00000244u, 0x000200f9u, 0x0000021bu, 0x000200f8u, + 0x0000021bu, 0x000100fdu, 0x00010038u, +}; + constexpr uint32_t kSpv_vt_cast[] = { 0x07230203u, 0x00010300u, 0x0008000bu, 0x000001c6u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, 0x00001151u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, @@ -11344,6 +12062,10 @@ constexpr uint32_t kSpv_vt_silu_and_mul[] = { 0x000200feu, 0x00000156u, 0x00010038u, }; +constexpr uint32_t kSpecIds_vt_attn_qk_norm_rope_gate[] = { + 0u, 1u, 2u, +}; + constexpr uint32_t kSpecIds_vt_cast[] = { 0u, 1u, }; @@ -11400,6 +12122,7 @@ constexpr uint32_t kSpecIds_vt_sigmoid_gate_bf16[] = { const SpirvModule kSpirvModules[] = { {"vt_add", kSpv_vt_add, sizeof(kSpv_vt_add) / sizeof(uint32_t), nullptr, 0}, + {"vt_attn_qk_norm_rope_gate", kSpv_vt_attn_qk_norm_rope_gate, sizeof(kSpv_vt_attn_qk_norm_rope_gate) / sizeof(uint32_t), kSpecIds_vt_attn_qk_norm_rope_gate, 3}, {"vt_cast", kSpv_vt_cast, sizeof(kSpv_vt_cast) / sizeof(uint32_t), kSpecIds_vt_cast, 2}, {"vt_causal_conv1d_update", kSpv_vt_causal_conv1d_update, sizeof(kSpv_vt_causal_conv1d_update) / sizeof(uint32_t), nullptr, 0}, {"vt_embedding", kSpv_vt_embedding, sizeof(kSpv_vt_embedding) / sizeof(uint32_t), kSpecIds_vt_embedding, 3}, diff --git a/tests/vt/test_vulkan_backend.cpp b/tests/vt/test_vulkan_backend.cpp index ef6155613..446dbfc29 100644 --- a/tests/vt/test_vulkan_backend.cpp +++ b/tests/vt/test_vulkan_backend.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -57,7 +58,7 @@ TEST_CASE("the committed SPIR-V table is present and well-formed") { // point of the split: at the target shader surface the words must not be // re-parsed by every TU that merely needs the table. const size_t n = vt::vulkan::kSpirvModuleCount; - CHECK(n == 24); + CHECK(n == 25); for (size_t mi = 0; mi < n; ++mi) { const auto& m = vt::vulkan::kSpirvModules[mi]; CAPTURE(m.name); @@ -78,7 +79,9 @@ TEST_CASE("the committed SPIR-V table is present and well-formed") { "vt_gdn_state_gather", "vt_gdn_state_scatter", "vt_rms_norm_gated", "vt_sigmoid_gate_bf16", // BACKEND-VULKAN-GDN-CORE: the two recurrences. - "vt_gdn_prefill", "vt_gdn_decode"}) { + "vt_gdn_prefill", "vt_gdn_decode", + // BACKEND-VULKAN-QKNORM: the fused attn preamble. + "vt_attn_qk_norm_rope_gate"}) { bool found = false; for (size_t mi = 0; mi < vt::vulkan::kSpirvModuleCount; ++mi) { if (std::strcmp(vt::vulkan::kSpirvModules[mi].name, want) == 0) found = true; @@ -179,6 +182,14 @@ TEST_CASE("the committed SPIR-V table records each module's specialization const // share one step body (vt_gdn_recurrence.glsl) and must stay in lockstep. REQUIRE(m.spec_id_count == 2); for (uint32_t want = 0; want < 2; ++want) CHECK(m.spec_ids[want] == want); + } else if (std::strcmp(m.name, "vt_attn_qk_norm_rope_gate") == 0) { + // The shared qgate/kf dtype, the shared q_out/k_out dtype, and the gate + // dtype. THREE, and the third one is the assertion: the gate is a separate + // axis only because the op contract admits an f32 gate alongside bf16 q/k + // (the FA-2 prefill combo, src/vt/ops.cpp:1530-1536). q_norm, k_norm and + // the cos/sin cache are f32 by contract and are therefore NOT axes. + REQUIRE(m.spec_id_count == 3); + for (uint32_t want = 0; want < 3; ++want) CHECK(m.spec_ids[want] == want); } else { CHECK(m.spec_id_count == 0); } @@ -2145,3 +2156,307 @@ TEST_CASE("a GDN recurrence wider than the shared tile DECLINES to the reference vk.DestroyQueue(vq); cpu.DestroyQueue(cq); } + +// =========================================================================== +// BACKEND-VULKAN-QKNORM — the fused full-attention preamble. +// +// SHAPES ARE CHOSEN, NOT DEFAULTED. Hq=5 / Hkv=2 is RAGGED (Hq is not a multiple +// of Hkv, and 5+2=7 workgroups per token exercise the flattened (token, head) +// decomposition rather than a power-of-two one). Dh=160 exceeds the 128-wide +// workgroup, so every lane walks its head row in a strided loop and the mean +// square is a genuine tree reduction over partial sums. rot=96 < Dh is the whole +// point of the op — PARTIAL RoPE, so dims [96,160) must come out normed but +// UNROTATED, and a kernel that rotated the whole head would still pass a +// rot==Dh test. Both source rows are PADDED VIEWS (stride[0] > shape[1]), the +// QKVParallelLinear packed layout the model actually hands the op. +// =========================================================================== +namespace { + +// The op's contract: qgate is [T, Hq*2*Dh] with q at [h*2*Dh, h*2*Dh+Dh) and the +// gate at the second Dh (src/vt/ops.cpp:1516-1523). +constexpr int64_t kQkT = 3, kQkHq = 5, kQkHkv = 2, kQkDh = 160, kQkRot = 96; +constexpr int64_t kQkQRow = kQkHq * 2 * kQkDh, kQkKRow = kQkHkv * kQkDh; +constexpr int64_t kQkQPad = 24, kQkKPad = 8; // packed-view row padding +constexpr int64_t kQkQStride = kQkQRow + kQkQPad, kQkKStride = kQkKRow + kQkKPad; + +// std::to_string gives six DECIMALS, which renders every NMSE this kernel +// produces as "0.000000" — the one number the message exists to carry. +std::string Sci(double v) { + char buf[32]; + std::snprintf(buf, sizeof(buf), "%.3g", v); + return buf; +} + +Tensor QkSrc(void* p, Device dev, vt::DType dt, int64_t cols, int64_t stride) { + Tensor t = Tensor::Contiguous(p, dt, dev, {kQkT, cols}); + t.stride[0] = stride; // inner-contiguous, padded row: the packed projection view + return t; +} + +} // namespace + +TEST_CASE("the fused attn preamble runs NATIVELY on Vulkan: partial RoPE, ragged heads") { + if (!VulkanPresent()) return; + auto& ctx = vt::vulkan::VulkanContext::Get(); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + const std::vector qgate = Spread(kQkT * kQkQStride, 2.0f, 401u); + const std::vector kf = Spread(kQkT * kQkKStride, 2.0f, 409u); + // Norm weights around zero: with gemma the applied weight is (1+w), so this is + // a spread around 1 and a kernel that dropped the +1 changes every output. + const std::vector qn = Spread(kQkDh, 0.4f, 419u); + const std::vector kn = Spread(kQkDh, 0.4f, 421u); + + const int64_t qelems = kQkT * kQkQStride, kelems = kQkT * kQkKStride; + const int64_t qout = kQkT * kQkHq * kQkDh, kout = kQkT * kQkHkv * kQkDh; + + Buf vqg(vk, qelems, 4), vkf(vk, kelems, 4), vqn(vk, kQkDh, 4), vkn(vk, kQkDh, 4); + Buf vcs(vk, kQkT * kQkRot, 4); + Buf vqo(vk, qout, 4), vko(vk, kout, 4), vgo(vk, qout, 4); + Buf cqg(cpu, qelems, 4), ckf(cpu, kelems, 4), cqn(cpu, kQkDh, 4), ckn(cpu, kQkDh, 4); + Buf ccs(cpu, kQkT * kQkRot, 4), cpo(cpu, kQkT, 4); + Buf cqo(cpu, qout, 4), cko(cpu, kout, 4), cgo(cpu, qout, 4); + + // The cos/sin table is built ONCE on the host — that is the op's own split + // (kRopeCosSinCache stays on the portable tier by design) — and the SAME BYTES + // are handed to both devices, so the comparison isolates this kernel. + const std::vector pos = {0, 7, 4096}; // a long position: real angles + std::memcpy(cpo.p(), pos.data(), pos.size() * 4); + Tensor cpot = Tensor::Contiguous(cpo.p(), vt::DType::kI32, cd, {kQkT}); + Tensor ccst = Tensor::Contiguous(ccs.p(), vt::DType::kF32, cd, {kQkT, kQkRot}); + vt::RopeCosSinCache(cq, ccst, cpot, vt::RopeArgs{10000.0f, static_cast(kQkRot)}); + + vk.Copy(vq, vqg.p(), qgate.data(), qelems * 4); + vk.Copy(vq, vkf.p(), kf.data(), kelems * 4); + vk.Copy(vq, vqn.p(), qn.data(), kQkDh * 4); + vk.Copy(vq, vkn.p(), kn.data(), kQkDh * 4); + vk.Copy(vq, vcs.p(), ccs.p(), kQkT * kQkRot * 4); + std::memcpy(cqg.p(), qgate.data(), qelems * 4); + std::memcpy(ckf.p(), kf.data(), kelems * 4); + std::memcpy(cqn.p(), qn.data(), kQkDh * 4); + std::memcpy(ckn.p(), kn.data(), kQkDh * 4); + vk.Synchronize(vq); + + auto out3 = [](void* p, Device dev, int64_t h) { + return Tensor::Contiguous(p, vt::DType::kF32, dev, {kQkT, h, kQkDh}); + }; + Tensor vqgt = QkSrc(vqg.p(), vd, vt::DType::kF32, kQkQRow, kQkQStride); + Tensor vkft = QkSrc(vkf.p(), vd, vt::DType::kF32, kQkKRow, kQkKStride); + Tensor vqnt = Tensor::Contiguous(vqn.p(), vt::DType::kF32, vd, {kQkDh}); + Tensor vknt = Tensor::Contiguous(vkn.p(), vt::DType::kF32, vd, {kQkDh}); + Tensor vcst = Tensor::Contiguous(vcs.p(), vt::DType::kF32, vd, {kQkT, kQkRot}); + Tensor vqot = out3(vqo.p(), vd, kQkHq), vkot = out3(vko.p(), vd, kQkHkv); + Tensor vgot = out3(vgo.p(), vd, kQkHq); + Tensor cqgt = QkSrc(cqg.p(), cd, vt::DType::kF32, kQkQRow, kQkQStride); + Tensor ckft = QkSrc(ckf.p(), cd, vt::DType::kF32, kQkKRow, kQkKStride); + Tensor cqnt = Tensor::Contiguous(cqn.p(), vt::DType::kF32, cd, {kQkDh}); + Tensor cknt = Tensor::Contiguous(ckn.p(), vt::DType::kF32, cd, {kQkDh}); + Tensor cqot = out3(cqo.p(), cd, kQkHq), ckot = out3(cko.p(), cd, kQkHkv); + Tensor cgot = out3(cgo.p(), cd, kQkHq); + + const vt::RopeArgs ra{10000.0f, static_cast(kQkRot)}; + // BOTH gemma arms. The model only ever calls gemma=true, but the weight is + // applied as (1+w) vs w through a UNIFORM branch in the shader, and a branch + // written the other way round would be invisible with only one arm gated. + for (bool gemma : {true, false}) { + const vt::RmsNormArgs na{1e-6f, gemma}; + vt::AttnQkNormRopeGate(cq, cqot, ckot, cgot, cqgt, ckft, cqnt, cknt, ccst, na, ra); + vt::AttnQkNormRopeGate(vq, vqot, vkot, vgot, vqgt, vkft, vqnt, vknt, vcst, na, ra); + vk.Synchronize(vq); + + std::vector gq(qout), gk(kout), gg(qout); + vk.Copy(vq, gq.data(), vqo.p(), qout * 4); + vk.Copy(vq, gk.data(), vko.p(), kout * 4); + vk.Copy(vq, gg.data(), vgo.p(), qout * 4); + vk.Synchronize(vq); + const std::vector rq(cqo.as(), cqo.as() + qout); + const std::vector rk(cko.as(), cko.as() + kout); + const double nq = NmseOf(rq, gq), nk = NmseOf(rk, gk); + // Reported, NOT asserted: on THIS device the tree reduction happens to land + // on the CPU tier's own bits, but that is a property of the shapes and the + // driver, not a promise the op makes across devices — the tier this kernel is + // gated at is NMSE, like every other reducing shader in the backend. + const bool bitwise = std::memcmp(rq.data(), gq.data(), static_cast(qout) * 4) == 0 && + std::memcmp(rk.data(), gk.data(), static_cast(kout) * 4) == 0; + // Assembled OUTSIDE the macro: MESSAGE(x << y) hands the expression to the + // doctest MessageBuilder, so a flag written inside renders as "1". + const std::string line = std::string("attn_qk_norm_rope_gate (gemma=") + + (gemma ? "1" : "0") + ") q NMSE " + Sci(nq) + ", k NMSE " + Sci(nk) + + ", bitwise-equal to the CPU: " + (bitwise ? "YES" : "no"); + MESSAGE(line); + CHECK(nq <= kGdnNmseTol); + CHECK(nk <= kGdnNmseTol); + + // THE GATE IS A PASSTHROUGH, so it is held to the BIT-EXACT tier, not NMSE: + // it is a copy of the raw second Dh of each q|gate pair with no norm and no + // rotation, and it is compared against the SOURCE rather than against the CPU + // oracle's output — a kernel that copied the q half into both would agree + // with a kernel that did the same on the other device. + bool gate_exact = true; + for (int64_t tok = 0; tok < kQkT; ++tok) { + for (int64_t h = 0; h < kQkHq; ++h) { + for (int64_t j = 0; j < kQkDh; ++j) { + const float want = + qgate[static_cast(tok * kQkQStride + h * 2 * kQkDh + kQkDh + j)]; + if (gg[static_cast((tok * kQkHq + h) * kQkDh + j)] != want) gate_exact = false; + } + } + } + CHECK(gate_exact); + // The gate must also be DIFFERENT from q, or "carrying data" is unproven. + CHECK(std::memcmp(gg.data(), gq.data(), static_cast(qout) * 4) != 0); + + // PARTIAL RoPE: dims [rot, Dh) are normed but NOT rotated, so they equal the + // plain gemma-RMSNorm of the source. Checked against a hand-computed f64 row + // rather than against the CPU kernel, which would agree with a Vulkan kernel + // that rotated everything only if the CPU one did too. + for (int64_t h = 0; h < kQkHq; ++h) { + const size_t base = static_cast(h * 2 * kQkDh); // token 0 + double ss = 0.0; + for (int64_t j = 0; j < kQkDh; ++j) { + const double v = qgate[base + static_cast(j)]; + ss += v * v; + } + const double inv = 1.0 / std::sqrt(ss / static_cast(kQkDh) + 1e-6); + for (int64_t j = kQkRot; j < kQkDh; ++j) { + const double w = gemma ? qn[static_cast(j)] + 1.0 : qn[static_cast(j)]; + const double want = qgate[base + static_cast(j)] * inv * w; + const double got = gq[static_cast(h * kQkDh + j)]; + CAPTURE(h); + CAPTURE(j); + CHECK(std::fabs(got - want) <= 1e-4 * (std::fabs(want) + 1e-3)); + } + } + } + + CHECK(ctx.PipelineExistsFor("vt_attn_qk_norm_rope_gate")); + CHECK(RanNative(vt::OpId::kAttnQkNormRopeGate)); + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +TEST_CASE("the fused attn preamble serves the bf16 q/k + f32 gate combo natively") { + if (!VulkanPresent()) return; + auto& ctx = vt::vulkan::VulkanContext::Get(); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // The FA-2 prefill combo the op contract singles out (src/vt/ops.cpp:1530-1536): + // bf16 sources, bf16 q/k (they feed attention and the bf16 KV-cache write) and + // an f32 gate (sigmoid(gate) must see the un-rounded value). It is the ONLY + // shape in which the gate dtype differs from q/k, which is the reason the gate + // is its own specialization axis — a shader that reused the q/k dtype constant + // for the gate would write bf16 into an f32 buffer and pass every same-dtype + // test. + const std::vector qgate_f = Spread(kQkT * kQkQStride, 2.0f, 431u); + const std::vector kf_f = Spread(kQkT * kQkKStride, 2.0f, 433u); + const std::vector qn = Spread(kQkDh, 0.4f, 439u); + const std::vector kn = Spread(kQkDh, 0.4f, 443u); + std::vector qgate_b(qgate_f.size()), kf_b(kf_f.size()); + for (size_t i = 0; i < qgate_f.size(); ++i) qgate_b[i] = vt::F32ToBF16(qgate_f[i]); + for (size_t i = 0; i < kf_f.size(); ++i) kf_b[i] = vt::F32ToBF16(kf_f[i]); + + const int64_t qelems = kQkT * kQkQStride, kelems = kQkT * kQkKStride; + const int64_t qout = kQkT * kQkHq * kQkDh, kout = kQkT * kQkHkv * kQkDh; + + Buf vqg(vk, qelems, 2), vkf(vk, kelems, 2), vqn(vk, kQkDh, 4), vkn(vk, kQkDh, 4); + Buf vcs(vk, kQkT * kQkRot, 4); + Buf vqo(vk, qout, 2), vko(vk, kout, 2), vgo(vk, qout, 4); + Buf cqg(cpu, qelems, 2), ckf(cpu, kelems, 2), cqn(cpu, kQkDh, 4), ckn(cpu, kQkDh, 4); + Buf ccs(cpu, kQkT * kQkRot, 4), cpo(cpu, kQkT, 4); + Buf cqo(cpu, qout, 2), cko(cpu, kout, 2), cgo(cpu, qout, 4); + + const std::vector pos = {1, 33, 900}; + std::memcpy(cpo.p(), pos.data(), pos.size() * 4); + Tensor cpot = Tensor::Contiguous(cpo.p(), vt::DType::kI32, cd, {kQkT}); + Tensor ccst = Tensor::Contiguous(ccs.p(), vt::DType::kF32, cd, {kQkT, kQkRot}); + vt::RopeCosSinCache(cq, ccst, cpot, vt::RopeArgs{10000.0f, static_cast(kQkRot)}); + + vk.Copy(vq, vqg.p(), qgate_b.data(), qelems * 2); + vk.Copy(vq, vkf.p(), kf_b.data(), kelems * 2); + vk.Copy(vq, vqn.p(), qn.data(), kQkDh * 4); + vk.Copy(vq, vkn.p(), kn.data(), kQkDh * 4); + vk.Copy(vq, vcs.p(), ccs.p(), kQkT * kQkRot * 4); + std::memcpy(cqg.p(), qgate_b.data(), qelems * 2); + std::memcpy(ckf.p(), kf_b.data(), kelems * 2); + std::memcpy(cqn.p(), qn.data(), kQkDh * 4); + std::memcpy(ckn.p(), kn.data(), kQkDh * 4); + vk.Synchronize(vq); + + auto qk3 = [](void* p, Device dev, int64_t h) { + return Tensor::Contiguous(p, vt::DType::kBF16, dev, {kQkT, h, kQkDh}); + }; + Tensor vqgt = QkSrc(vqg.p(), vd, vt::DType::kBF16, kQkQRow, kQkQStride); + Tensor vkft = QkSrc(vkf.p(), vd, vt::DType::kBF16, kQkKRow, kQkKStride); + Tensor vqnt = Tensor::Contiguous(vqn.p(), vt::DType::kF32, vd, {kQkDh}); + Tensor vknt = Tensor::Contiguous(vkn.p(), vt::DType::kF32, vd, {kQkDh}); + Tensor vcst = Tensor::Contiguous(vcs.p(), vt::DType::kF32, vd, {kQkT, kQkRot}); + Tensor vqot = qk3(vqo.p(), vd, kQkHq), vkot = qk3(vko.p(), vd, kQkHkv); + Tensor vgot = Tensor::Contiguous(vgo.p(), vt::DType::kF32, vd, {kQkT, kQkHq, kQkDh}); + Tensor cqgt = QkSrc(cqg.p(), cd, vt::DType::kBF16, kQkQRow, kQkQStride); + Tensor ckft = QkSrc(ckf.p(), cd, vt::DType::kBF16, kQkKRow, kQkKStride); + Tensor cqnt = Tensor::Contiguous(cqn.p(), vt::DType::kF32, cd, {kQkDh}); + Tensor cknt = Tensor::Contiguous(ckn.p(), vt::DType::kF32, cd, {kQkDh}); + Tensor cqot = qk3(cqo.p(), cd, kQkHq), ckot = qk3(cko.p(), cd, kQkHkv); + Tensor cgot = Tensor::Contiguous(cgo.p(), vt::DType::kF32, cd, {kQkT, kQkHq, kQkDh}); + + const vt::RmsNormArgs na{1e-6f, true}; + const vt::RopeArgs ra{10000.0f, static_cast(kQkRot)}; + vt::AttnQkNormRopeGate(cq, cqot, ckot, cgot, cqgt, ckft, cqnt, cknt, ccst, na, ra); + vt::AttnQkNormRopeGate(vq, vqot, vkot, vgot, vqgt, vkft, vqnt, vknt, vcst, na, ra); + vk.Synchronize(vq); + + CHECK(ctx.PipelineExistsFor("vt_attn_qk_norm_rope_gate")); + CHECK(RanNative(vt::OpId::kAttnQkNormRopeGate)); + + std::vector gq(qout), gk(kout); + std::vector gg(qout); + vk.Copy(vq, gq.data(), vqo.p(), qout * 2); + vk.Copy(vq, gk.data(), vko.p(), kout * 2); + vk.Copy(vq, gg.data(), vgo.p(), qout * 4); + vk.Synchronize(vq); + std::vector rqf(qout), gqf(qout), rkf(kout), gkf(kout); + for (int64_t i = 0; i < qout; ++i) { + rqf[static_cast(i)] = vt::BF16ToF32(cqo.as()[i]); + gqf[static_cast(i)] = vt::BF16ToF32(gq[static_cast(i)]); + } + for (int64_t i = 0; i < kout; ++i) { + rkf[static_cast(i)] = vt::BF16ToF32(cko.as()[i]); + gkf[static_cast(i)] = vt::BF16ToF32(gk[static_cast(i)]); + } + const double nq = NmseOf(rqf, gqf), nk = NmseOf(rkf, gkf); + const bool bitwise = std::memcmp(cqo.p(), gq.data(), static_cast(qout) * 2) == 0 && + std::memcmp(cko.p(), gk.data(), static_cast(kout) * 2) == 0; + const std::string line = std::string("attn_qk_norm_rope_gate (bf16 q/k, f32 gate) q NMSE ") + + Sci(nq) + ", k NMSE " + Sci(nk) + + ", bitwise-equal to the CPU: " + (bitwise ? "YES" : "no"); + MESSAGE(line); + CHECK(nq <= kGdnNmseTol); + CHECK(nk <= kGdnNmseTol); + + // The f32 gate is a widening passthrough of the bf16 source, so it is EXACT. + bool gate_exact = true; + for (int64_t tok = 0; tok < kQkT; ++tok) { + for (int64_t h = 0; h < kQkHq; ++h) { + for (int64_t j = 0; j < kQkDh; ++j) { + const float want = vt::BF16ToF32( + qgate_b[static_cast(tok * kQkQStride + h * 2 * kQkDh + kQkDh + j)]); + if (gg[static_cast((tok * kQkHq + h) * kQkDh + j)] != want) gate_exact = false; + } + } + } + CHECK(gate_exact); + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} From c6b6904093a8062f5de760485b5ecc5cff20acdc Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 9 Aug 2026 02:28:02 +0000 Subject: [PATCH 2/2] docs: roll the Gemma4 FP8 stream-lab section into the append-only record docs/BENCHMARKS.md is a KEYED TABLE: a checkpoint updates a ROW, it does not append a dated H2 section. #154 appended one, which left main red on check-public-doc-tables for two reasons at once -- the non-canonical section, and the em-dash it carried (house style forbids em-dashes on the public pages). scripts/roll-benchmark-record.py --apply moves it verbatim into .agents/benchmark-record.md, which is where per-change narrative belongs, and both errors go with it. Nothing is lost and nothing is rewritten: the section moves byte-for-byte into the append-only record. This is main's red, not this branch's, and it was blocking the agent-record CI job on every open PR rather than just on the change that introduced it. RESIDUE, NAMED RATHER THAN PAPERED OVER. check-public-doc-tables still fails on one item: docs/STATUS.md is 277213 chars against a 276960 ratchet. This branch SHRINKS that page by 4 chars; main is already 257 over. Clearing it means collapsing superseded narrative, and the only block big enough to matter is a single 33,211-char table cell (the Laguna-S-2.1 MoE row) that is itself well past the 220-char cell rule. Collapsing a cell that size is a deliberate, separately-reviewable change with its own owner, not something to bury in a Vulkan performance PR, so it is left open and stated here instead. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 13 +++++++++++++ docs/BENCHMARKS.md | 9 --------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 70aa73222..9b5fdacf5 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -16177,3 +16177,16 @@ the ~30 flushes/token. The declines name exactly three ops: `kCausalConv1dFwd` (op 5, prefill only), `kRopeCosSinCache` (op 66) and `kAttnQkNormRopeGate` (op 67). The last two run per full-attention layer, 16 per token, and are the next lever; they are kernel work, not plumbing. + +## Rolled out of the scoreboard on 2026-08-09 + +Moved verbatim from `docs/BENCHMARKS.md` by `scripts/roll-benchmark-record.py`. Nothing edited or deleted. + +## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700) + +| Path | Warm tok/s | Notes | +|------|------------|--------| +| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill | +| server `/v1/completions` | ~38 | exclusive | +| server `/v1/chat` thinking off | ~32 | after expert cache | +| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack | diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index eb4f8eb84..180769400 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -383,12 +383,3 @@ built on it rather than keeping the flattering one. Build flags, environment variables, and the full gate list are in [BUILD.md](BUILD.md) and [ENVIRONMENT.md](ENVIRONMENT.md). - -## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700) - -| Path | Warm tok/s | Notes | -|------|------------|--------| -| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill | -| server `/v1/completions` | ~38 | exclusive | -| server `/v1/chat` thinking off | ~32 | after expert cache | -| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack |