vocab: Support tokenizer for LFM2.5-8B-A1B - #23826
Conversation
CISC
left a comment
CISC
left a comment
There was a problem hiding this comment.
Why both in pre_computed_hashes?
CISC
left a comment
CISC
left a comment
There was a problem hiding this comment.
Hmmm, looking more closely the LFM2 pre-tokenizer does not match LFM2.5, it uses a new regex:
'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s
|
Thanks for the feedback @CISC .
I tried to place them both in the table, but then re-running
on regex, the difference is minimal, and testing on real-life use cases didn't show any difference. However, we discovered that tool calling doesn't work with this chat template in llama.cpp (works in other frameworks) and currently debugging it. |
One (the original) should stay in
Still, since there is an actual difference it would be prudent to add an |
964007f to
6f8fa55
Compare
|
@CISC , we reworked the chat template to use a similar regex to lfm2 here https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/discussions/5 . I moved back existing tokenizer to models, hope it looks good to merge! |
That works too I guess. :) |
* origin/master: vocab : support tokenizer for LFM2.5-8B-A1B (ggml-org#23826) graph : ensure DS32 kq_mask_lid is F32 (ggml-org#23864) server: remove obsolete scripts (ggml-org#23870) ci : update macos release to use macos-26 runner (ggml-org#23878) download: add option to skip_download (ggml-org#23059) mtmd: Add DeepSeekOCR 2 Support (ggml-org#20975) CUDA: Check PTX version on host side to guard PDL dispatch (ggml-org#23530) server: bump timeout to 3600s (ggml-org#23842) model : support for DeepseekV32ForCausalLM with generic DeepSeek Sparse Attention (DSA) implementation (ggml-org#23346) llama: use f16 mask for FA to save VRAM (ggml-org#23764) sync : ggml ggml : bump version to 0.13.1 (ggml/1523) ngram-mod : Add missing include (ggml-org#23857) llama: add llm_graph_input_mtp (ggml-org#23643) app : move licences to llama-app (ggml-org#23824) cuda : disables launch_fattn PDL enrollment due to compiler bug (ggml-org#23825) meta : Add missing `buffer` set in allreduce fallback !COMPUTE clear (ggml-org#23480)
* vocab: Support tokenizer for LFM2.5-8B-A1B * Keep liquid6 tokenizer in models
* vocab: Support tokenizer for LFM2.5-8B-A1B * Keep liquid6 tokenizer in models
* vocab: Support tokenizer for LFM2.5-8B-A1B * Keep liquid6 tokenizer in models
The RVQ loader was wired per architecture: llama.cpp and gptneox.cpp each carried their own branch deciding which tensors might be bit-plane encoded. Every new architecture meant another copy, and an architecture without one silently loaded nothing compressed. llama_model_base::create_tensor now offers any 2-D weight to llama_orka_try_create first, so a tensor is served from bit-planes whenever the side tensors exist and dense otherwise. The per-arch branches are deleted (-64 in gptneox, -47 in llama); support is now a property of the artifact, not of a hand-maintained list. 3-D weights, such as LFM2's depthwise short-conv kernels, fall through to the dense path by construction. llama-orka gains variable index widths per stage (idx_bits[3]) so a stage may be 8, 10 or 12 bits, and the CUDA kernel follows. conversion/base.py registers the LFM2.5 tokenizer checksum, matching ggml-org/llama.cpp#23826 which maps this same hash to the existing lfm2 pre-tokenizer. Note that PR's discussion observes the LFM2.5 regex genuinely differs from LFM2 (\s*[\r\n] vs \s*[\r\n]+, trailing \s vs \s+); the merged change reuses lfm2 regardless, and this follows the merged change. Measured on LiquidAI/LFM2.5-2.6B-Base packed to 3.2825 bpw, RTX 3060: GGUF 1.739 GB from 5.39 GB bf16, 3.10x tensors rvq=150 dense=116 dense_transformed=0 llama-bench pp128 38.04 t/s, tg64 33.60 t/s llama-cli correct output, generation 65.1 t/s Known unrelated defect: llama-cli aborts under -fit on LFM2 with GGML_ASSERT(a->ne[d] == b->ne[d]) in ggml_concat, inside the short-conv graph during graph_reserve. It reproduces independently of these changes (src/models/lfm2.cpp is untouched here), does not affect llama-bench, and -fit off is a working alternative.
3 participants
Overview
LFM2.5-8B-A1B shares architecture with LFM2-8B-A1B but comes with a new extended tokenizer.
This PR adds support for it.
GGUFs are uploaded to LiquidAI/LFM2.5-8B-A1B-GGUF
Requirements