feat(tokenizer): recognize and split the Mistral Tekken pre-tokenizer (#168)

[filipsajdak]

Adds SplitPattern::kTekken so mistralai/Mistral-Nemo-Instruct-2407 loads instead of dying at tokenizer: unrecognized pre-tokenizer split regex. Refs #168.

First: why you can trust the generated table

This adds 1,861 generated Unicode ranges. The reasonable first question is whether that table is right, so it was answered before anything was built on it — by regenerating your existing artefacts and requiring byte-identity, twice:

oracle	mine	yours	result
HF tokenizers	0.23.1	0.22.2 (per the goldens header)	regenerated the existing pretokenizer_goldens.inc: byte-identical
Python unicodedata	needed 15.0.0	15.0.0	pinned Python 3.12.12 and regenerated the committed unicode_data.{h,cpp}: byte-identical

Both checks mattered. Nothing I had shipped Unicode 15.0.0 by default — 3.14 gives 16.0.0, 3.13 gives 15.1.0 — so without pinning, this PR would have quietly placed a 16.0.0 table beside your 15.0.0 one and every check would still have looked green. Byte-identity on the artefacts you already have is the only thing that catches that.

Verified with cmp, sha256 and a clean git status, not by eye.

Second: this cannot regress what works

The change is additive by construction:

• UCat is untouched. kCategoryRanges (1,563) and kWhitespaceRanges (10) come out byte-identical after the regeneration. LetterCase is a second, narrow table, not a re-cut of the first — re-cutting would have put four working patterns at risk to serve one new one.
• All 108 pre-existing golden rows keep their exact qwen/llama columns. 8 rows are added, and only added, for the new behaviours.
• No existing SplitPattern changes behaviour. marks_aware splits into marks_in_run + marks_excluded, but every existing pattern sets both to the value it had.
• No vocab fixture is committed. The pre-tokenizer gate needs none, so a 9.26 MB tokenizer.json stays out of a tree whose largest is 1.96 MB.

test_tokenizer_parity_mistral (SentencePiece) and test_tokenizer_parity_deepseek (the seven-stage pipeline) pass unchanged. Those two are the real regression evidence for the shared scanner and the new table.

Third: the arm, and the part worth your attention

Tekken is tiktoken's o200k_base pat_str with exactly two edits: the optional (?i:'s|'t|'re|'ve|'m|'ll|'d)? group deleted from both letter alternatives, and \p{N} instead of \p{N}{1,3}. Everything from the punct run onward is byte-equal. So this puts the GPT-4o-family scanner in the tree; an o200k arm later would be two flags on the same matcher.

Three of the four differences from the shared Qwen/Llama-3 scanner are flags — no contraction alternative, max_digits = 1, and / in the punct run's trailing class. The fourth splits marks_aware, because Tekken carries \p{M} inside both letter classes (like kQwen2) while its punct negation has no \p{M} (like kLlama3), which no other pattern needs.

The fifth is where a bug would hide, so it gets named rather than buried. The two letter classes overlap on {Lm, Lo, M}, so [U]*[L]+ genuinely backtracks and cannot be MatchLetterRun's single-predicate scan. MatchTekkenAlt records the give-back positions and walks them longest-first; MatchTekkenLetterRun mirrors the engine's ordered alternation — alt 1 with prefix, alt 1 without, alt 2 with prefix, alt 2 without — rather than taking longest-match. Ordered, not longest, is the assumption to check.

Also worth flagging, since it is easy to assume otherwise: your existing Mistral gate is Mistral-7B-v0.3, a SentencePiece Metaspace tokenizer with byte-fallback. Different family entirely — Tekken was genuinely unrepresented, not a near-neighbour of something covered. (I got this wrong in #168 and have corrected it there.)

Evidence

• test_pretokenizer: 23/23 cases, 101,714 assertions — standalone and in-suite. Both are stated because test_qwen36_spec_decode: token drift in-suite, passes standalone 11/11 — filed as 'suite-context-dependent' but the decisive arm was never run #247 records a gate that passes standalone and drifts in-suite, so the standalone number alone would not have answered it.
• Round-trip on the real checkpoint, against the HF tokenizers library on Mistral-Nemo-Instruct-2407: 5,000 fuzz strings / 90,659 token ids, plus 45 hand-built edge cases — encode ids and decode both byte-exact, 100%. Both directions, not just "it splits".
• Linux aarch64, gcc 13.3 / libstdc++: tokenizer set 7/7 (test_pretokenizer, test_tokenizer_parity, _mistral, _deepseek, test_unicode_data, test_tokenizer_metaspace_split, test_detokenizer). Run on gcc specifically because libc++ tolerates missing transitive includes that libstdc++ rejects, and this change adds a <vector> dependency.
• Not run: the full 365-test suite on Linux. I have no clean full-suite baseline there, so I am not claiming one.

Full suite on macOS: the delta from this change is zero

365 tests, 97% passed, 11 failed — and all 11 reproduce identically on unmodified 7c7f024. The baseline was re-run in a pristine git archive tree with its own build directory, rather than a stash, so nothing of mine could leak into it:

test	with this change	clean 7c7f024
test_kv_offload_fs	Not Run	Not Run
test_kv_offload_tiering	Not Run	Not Run
test_kv_offload_connector	Not Run	Not Run
test_lmcache_client	Not Run	Not Run
test_lmcache_connector	Not Run	Not Run
test_serve_low_tools	Failed	Failed
test_safetensors	Failed	Failed
test_load_direct_upload	Failed	Failed
test_none_hash_determinism	Failed	Failed
test_ops_quant_dot	SEGFAULT	SEGFAULT
test_cpu_kernel_bench_cli	Failed	Failed

Same 11 red before and after, SEGFAULT included. The five Not Run are macOS build failures in files this change does not touch (<unistd.h>/getpid, ::htonl against the macOS macro, and one -Wunused-function); test_serve_low_tools is already noted as red on main in #233. Recorded here as the regression evidence for a Unicode-table change, not as a bug report — the fixes are not bundled into this PR.

Your own gates, run locally before opening: check-commit-trailers, check-doc-checkpoint, check-now-current, check-pr-size, check-public-doc-tables — all exit 0.

Reproducing the goldens

python3 tools/gen_pretok_goldens.py > tests/vllm/pretokenizer_goldens.inc
python3 tools/gen_unicode_data.py   # requires unidata 15.0.0, i.e. Python 3.12

Happy to split this into the table change and the arm if you would rather review them separately — I kept it as one because the table has no consumer without kTekken.

[filipsajdak]

sanitize-cpu (address,undefined): the delta from this PR is zero

The baseline here is your own CI on unmodified main — same runner, same toolchain — rather than anything from my machine: run 31422786494 on e17e8f8a fails this job already.

	this PR	main @ e17e8f8a
failing tests	9, 44, 59, 234, 238	the same five, same numbers
	test_load_direct_upload, test_llama_embedding_fold, test_laguna_nvfp4_loader, test_openai_api_server, test_capi	identical
UBSan sites	test_load_direct_upload.cpp:177:48, src/vt/cpu/cpu_ops.cpp:33:70, src/vllm/model_executor/models/laguna.cpp:1028:56	identical file:line:column

All are 2-byte-alignment violations on short unsigned int — load of misaligned address at the two cpu_ops.cpp / laguna.cpp sites, and reference binding to misaligned address at test_load_direct_upload.cpp:177 — in files this PR does not touch. test_pretokenizer itself passes clean under ASan/UBSan.

Not fixing them here: that would bundle unrelated work into a tokenizer change.

The other two reds were mine, and are fixed

pr-size and agent-record failed the first run with base must be an ancestor of head. I branched from 7c7f024, main moved 37 commits, and the PR's base.sha was therefore not an ancestor of the head — so both gates aborted before doing any work, which is why their logs looked like checkout noise. Rebased onto 2430636; the diff is unchanged at +1017/-15 across the same 10 files.

To be exact about what I can and cannot claim: the cause is removed — the head is now a descendant of the base — but I have not seen those two gates go green, because the workflows on the current head are sitting at action_required and have not run. That is presumably the fork-PR approval gate; the first push was approved and ran all 15 checks, and the force-push minted a new SHA that needs approving again. Flagging it only so the absent checks are not mistaken for a stuck queue.

Two things that may be worth knowing beyond this PR:

• With main advancing this fast, those two gates can go red purely on base ancestry while nothing is wrong with the change. Rebasing immediately before a merge attempt avoids the confusing red.
• That remedy has a cost here, though: each rebase mints a new SHA and re-gates the workflows behind another approval. Worth weighing before asking a contributor to rebase.

[localai-bot]

Merged — closes #168. This is a genuinely careful port, and the two seam decisions are the right ones.

Your red agent-record and pr-size checks were a fork artifact, not a defect: both refused with base must be an ancestor of head because the branch was never rebased onto current main, so neither checker could compute a commit range at all. Every substantive job — build, cuda-fat-build, build-test-cpu, build-test-cpu-arm64, build-test-vulkan, device-leakage, documentation-checkpoint, commit-protocol-tag — was green, and the sanitize-cpu (address,undefined) failure is a pre-existing main baseline that shows up identically on every open PR. Merging locally supplied the ancestry those two checkers wanted. Nothing for you to fix.

Two things I verified rather than took on trust, because both are the kind of claim that's easy to assert and expensive to be wrong about:

1. The generated Unicode table reproduces byte-for-byte. I re-ran tools/gen_unicode_data.py and md5'd before and after — identical for both include/vllm/tokenizer/unicode_data.h and src/vllm/tokenizer/unicode_data.cpp (unidata 15.0.0, 1861 letter-case ranges). The table isn't hand-edited.
2. The goldens are oracle-derived, not authored. gen_pretok_goldens.py drives the real HF tokenizers Split pre-tokenizer with the regex quoted verbatim from mistralai/Mistral-Nemo-Instruct-2407 tokenizer.json, across ~90 cases including 60 seeded random strings. That's the difference between a test and a transcription of the implementation.

The part I most appreciated: you widened two seams instead of bending them.

• A separate narrow LetterCase table rather than re-cutting UCat — so every existing consumer keeps its exact bytes, and the case split lives where it's actually needed.
• Splitting the single marks_aware flag into marks_in_run and marks_excluded, because Tekken carries \p{M} inside its letter classes (like kQwen2) while its punct negation omits it (like kLlama3). That combination exists in no other pattern, which is exactly why one flag sufficed until now — and both derived flags evaluate identically to the old one for every existing pattern, so it's non-regressive by construction.

MatchTekkenAlt walking the greedy give-back stops longest-first is the correct reading of ordered-alternation backtracking, and the comment explaining why it can't fold into MatchLetterRun's single-predicate scan (the classes overlap on {Lm, Lo, M}) is the kind of thing that saves the next reader an hour.

DetectPattern keying on an exact regex-string match also means a checkpoint that doesn't match falls through to today's behaviour — no existing model can change class. Verified test_pretokenizer, test_tokenizer_metaspace_split, test_detokenizer and test_tokenizer_parity* all green on the merged tree.

[localai-bot]

Landed on main as d8aa32b8a350 — the Mistral Tekken pre-tokenizer is on main and verified present.

Closing manually: it landed as a rebased copy of this branch, so the head SHA here never became reachable from main and GitHub could not auto-close it. Nothing is outstanding; the PR was just left showing OPEN.

Thanks for the contribution.