merge: row/TOKENIZER-TEKKEN -- recognize and split the Mistral Tekken pre-tokenizer (#283)

Closes #168. External contribution from Filip Sajdak (@filipsajdak).

The PR's red agent-record and pr-size checks were a FORK ARTIFACT, not a
defect: both checkers refused with "base must be an ancestor of head"
because the branch had not been rebased onto current main, so neither could
compute a commit range at all. Every substantive job -- build, cuda-fat-
build, build-test-cpu, build-test-cpu-arm64, build-test-vulkan,
device-leakage, documentation-checkpoint, commit-protocol-tag -- was green.
The sanitize-cpu (address,undefined) failure is the pre-existing main
baseline (test_load_direct_upload, test_llama_embedding_fold,
test_laguna_nvfp4_loader, test_openai_api_server, test_capi), identical on
every open PR. Merging locally supplies the ancestry those two checkers
wanted.

Tekken is the first pattern here whose letter rule is CASE-AWARE. It is
tiktoken's o200k_base pat_str with exactly two edits -- no (?i:'s|'t|...)
contraction group, and single-codepoint \p{N} instead of \p{N}{1,3} -- so
an uppercase run ends a piece when a lowercase run follows ("HelloWorld" ->
"Hello" + "World"). Because the two letter classes OVERLAP on {Lm, Lo, M},
this needs genuine ordered-alternation backtracking and could not fold into
MatchLetterRun's single-predicate scan; MatchTekkenAlt walks the greedy
give-back stops longest-first, which is exactly what the regex engine picks.

Two seams widened rather than bent:
  - A separate narrow LetterCase table (Lu/Lt=upper, Ll=lower, Lm/Lo in
    BOTH), because UCat collapses all of L* into kLetter and cannot express
    the split. UCat and every existing consumer keep their exact bytes.
  - The single marks_aware flag splits into marks_in_run and
    marks_excluded. Tekken carries \p{M} inside its letter classes (like
    kQwen2) while its punct negation omits \p{M} (like kLlama3) -- a
    combination no other pattern has, which is why one flag sufficed until
    now. Both derived flags evaluate identically to the old one for kQwen2
    and for every other pattern, so this is non-regressive by construction.

Verified before merging rather than taken on trust:
  - tools/gen_unicode_data.py reproduces the checked-in
    include/vllm/tokenizer/unicode_data.h and src/.../unicode_data.cpp
    BYTE-FOR-BYTE (md5 identical after a local regen, unidata 15.0.0, 1861
    letter-case ranges). The generated table is not hand-edited.
  - The goldens are oracle-derived, not authored: gen_pretok_goldens.py
    drives the real HF tokenizers Split pre-tokenizer with the regex quoted
    verbatim from mistralai/Mistral-Nemo-Instruct-2407 tokenizer.json, over
    ~90 cases including 60 seeded random strings.
  - DetectPattern keys on an exact regex-string match, so a checkpoint that
    does not match falls through to today's behavior. No existing model can
    change class.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]

Author: mudler

docs/USAGE.md

@@ -129,6 +129,31 @@ Two more example binaries ship alongside it:
 - `tokenize` ([`examples/tokenize/main.cpp`](../examples/tokenize/main.cpp)), a
   tokenizer smoke tool taking `<tokenizer.json | model.gguf> <corpus.txt>`.
 
+### Which HF tokenizers load
+
+A checkpoint's `tokenizer.json` is accepted when its `pre_tokenizer` is one this
+build recognises. Recognition is by exact regex or pipeline shape, not by model
+name, so a checkpoint from any vendor loads if it carries one of these:
+
+| family | shape | examples |
+|---|---|---|
+| Qwen3.6 | one `Split` regex, single-codepoint `\p{N}`, `\p{M}` folded into letter runs | Qwen3.6-27B |
+| Qwen2/Qwen3 classic | as above without `\p{M}` awareness | Qwen3-0.6B, Qwen3-Coder |
+| Llama-3 | `\p{N}{1,3}` digit groups, no `\p{M}` awareness | Llama-3 family |
+| Tekken (Mistral) | case-aware letter runs, single-codepoint `\p{N}`, `/` in the punct tail | Mistral-Nemo-Instruct-2407 |
+| GPT-2 byte-level | `ByteLevel(use_regex=true)` with no explicit `Split` | OPT, GPT-2 |
+| DeepSeek | a seven-stage `Sequence` pipeline, not one alternation | DeepSeek-V2/V3 |
+| SentencePiece | `Metaspace` + byte-fallback vocab | Mistral-7B-v0.3 |
+
+An unrecognised one fails loudly at load with `tokenizer: unrecognized
+pre-tokenizer split regex: <regex>`, rather than tokenizing incorrectly. If you
+hit that, the printed regex is what a new pattern would have to match.
+
+Note that Mistral ships **two** unrelated tokenizer families: Mistral-7B-v0.3 is
+SentencePiece, while Mistral-Nemo is Tekken, a byte-level BPE whose regex is
+tiktoken's `o200k_base` with the contraction group removed and `\p{N}{1,3}`
+reduced to `\p{N}`. Support for one says nothing about the other.
+
 ### How much memory a Vulkan load needs
 
 On a unified-memory device (a DGX Spark) the Vulkan heap and system RAM are the

include/vllm/tokenizer/pretokenizer.h

@@ -27,6 +27,19 @@ enum class SplitPattern {
             // case-SENSITIVE contractions, a plain ` ?` space prefix instead of
             // the `[^\r\n\p{L}\p{N}]?` prefix, UNBOUNDED `\p{N}+` digit runs,
             // no `[\r\n]*` punct tail and no `\s*[\r\n]+` rule at all.
+  kTekken,  // Mistral Tekken family (Mistral-Nemo, and the Tekken-v3/v7
+            // checkpoints that share its tokenizer.json shape). The ONLY
+            // pattern here whose letter rule is CASE-AWARE: two alternatives,
+            // [\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+ then
+            // the same pair with the quantifiers swapped, so an uppercase run
+            // ends a piece when a lowercase run follows ("HelloWorld" ->
+            // "Hello" + "World"). It is tiktoken's o200k_base pat_str with
+            // exactly two edits: NO (?i:'s|'t|...) contraction group, and
+            // single-codepoint \p{N} instead of \p{N}{1,3}. Marks sit INSIDE
+            // both letter classes (like kQwen2) while the punct negation is
+            // [^\s\p{L}\p{N}] with no \p{M} (like kLlama3) -- a combination no
+            // other pattern has. Its punct run also ends [\r\n/]*, absorbing a
+            // '/' that follows a newline.
   kDeepSeek,  // DeepSeek family (DeepSeek-V2/V2-Lite/V3). STRUCTURALLY UNLIKE
               // every pattern above: not ONE alternation regex but a HF
               // `Sequence` PIPELINE of seven pre-tokenizers, each further

include/vllm/tokenizer/unicode_data.h

@@ -2,7 +2,7 @@
 // Generator:  tools/gen_unicode_data.py
 // Regenerate: python3 tools/gen_unicode_data.py
 // Unicode data version (Python unicodedata.unidata_version): 15.0.0
-// Category ranges: 1563; whitespace ranges: 10.
+// Category ranges: 1563; whitespace ranges: 10; letter-case ranges: 1861.
 // Semantics mirror HF tokenizers byte-level BPE: categories are the major
 // Unicode general-category classes (unassigned -> kOther); IsWhitespace is
 // python str.isspace().
@@ -29,6 +29,20 @@ enum class UCat : uint8_t {
 
 UCat Category(uint32_t cp);
 
+// CASE class of a letter. Needed only by SplitPattern::kTekken, whose letter
+// alternatives are [\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}] and
+// [\p{Ll}\p{Lm}\p{Lo}\p{M}]; UCat collapses all of L* into kLetter and
+// cannot express that. Deliberately a separate narrow table, so UCat and every
+// existing consumer of it are unchanged. kNotLetter for anything outside L*.
+enum class LetterCase : uint8_t {
+  kNotLetter = 0,
+  kUpper = 1,     // Lu, Lt -- the UPPER class only
+  kLower = 2,     // Ll     -- the LOWER class only
+  kCaseless = 3,  // Lm, Lo -- present in BOTH classes
+};
+
+LetterCase LetterCaseOf(uint32_t cp);
+
 // python str.isspace() semantics (per HF byte-level pretokenization):
 // includes 0x1C-0x1F, NEL (0x85) and NBSP (0xA0); excludes ZWSP (0x200B),
 // Mongolian vowel separator (0x180E) and BOM (0xFEFF).

src/vllm/tokenizer/pretokenizer.cpp

@@ -41,7 +41,10 @@
 
 #include "vllm/tokenizer/pretokenizer.h"
 
+#include <cstddef>
 #include <cstdint>
+#include <string_view>
+#include <vector>
 
 #include "vllm/tokenizer/unicode_data.h"
 
@@ -125,6 +128,87 @@ size_t MatchLetterRun(std::string_view t, size_t pos, bool marks_in_run) {
   return p;
 }
 
+// Tekken rules 1-2, the CASE-AWARE letter alternatives:
+//   [^\r\n\p{L}\p{N}]? [\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]* [\p{Ll}\p{Lm}\p{Lo}\p{M}]+
+//   [^\r\n\p{L}\p{N}]? [\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+ [\p{Ll}\p{Lm}\p{Lo}\p{M}]*
+// The two classes OVERLAP on {Lm, Lo, M}, so this needs real backtracking and
+// cannot be folded into MatchLetterRun's single-predicate scan.
+bool InTekkenUpperClass(uint32_t cp) {
+  if (Category(cp) == UCat::kMark) return true;  // \p{M} is in BOTH classes
+  const LetterCase lc = LetterCaseOf(cp);
+  return lc == LetterCase::kUpper || lc == LetterCase::kCaseless;
+}
+
+bool InTekkenLowerClass(uint32_t cp) {
+  if (Category(cp) == UCat::kMark) return true;  // \p{M} is in BOTH classes
+  const LetterCase lc = LetterCaseOf(cp);
+  return lc == LetterCase::kLower || lc == LetterCase::kCaseless;
+}
+
+// Scans a maximal run of `pred` from `from`, recording every codepoint
+// boundary so the caller can give characters back (regex backtracking).
+void ScanRun(std::string_view t, size_t from, bool (*pred)(uint32_t),
+             std::vector<size_t>& stops) {
+  stops.clear();
+  stops.push_back(from);
+  size_t p = from;
+  while (p < t.size()) {
+    const Cp c = DecodeAt(t, p);
+    if (!pred(c.cp)) break;
+    p = c.end;
+    stops.push_back(p);
+  }
+}
+
+// `upper_first_required`: alternative 2 needs [U]+ then [L]*; alternative 1
+// needs [U]* then [L]+. Returns the match end, or 0 for no match.
+size_t MatchTekkenAlt(std::string_view t, size_t from, bool upper_first_required,
+                      std::vector<size_t>& scratch) {
+  ScanRun(t, from, InTekkenUpperClass, scratch);
+  // [U]* / [U]+ are greedy, so walk the give-back positions longest-first;
+  // the FIRST that lets the second half match is what the regex engine picks.
+  for (size_t i = scratch.size(); i-- > 0;) {
+    const size_t split = scratch[i];
+    const size_t upper_len = i;  // codepoints consumed by the upper run
+    if (upper_first_required && upper_len == 0) break;  // [U]+ needs >= 1
+    size_t p = split;
+    size_t lower_len = 0;
+    while (p < t.size()) {
+      const Cp c = DecodeAt(t, p);
+      if (!InTekkenLowerClass(c.cp)) break;
+      p = c.end;
+      ++lower_len;
+    }
+    if (!upper_first_required && lower_len == 0) continue;  // [L]+ needs >= 1
+    if (p > from) return p;
+  }
+  return 0;
+}
+
+size_t MatchTekkenLetterRun(std::string_view t, size_t pos,
+                            std::vector<size_t>& scratch) {
+  // Alternation is ORDERED and `X?` is greedy, so the engine tries, in order:
+  // alt1-with-prefix, alt1-without, alt2-with-prefix, alt2-without.
+  for (int alt = 0; alt < 2; ++alt) {
+    const bool upper_first_required = alt == 1;
+    for (int use_prefix = 1; use_prefix >= 0; --use_prefix) {
+      size_t start = pos;
+      if (use_prefix != 0) {
+        const Cp c0 = DecodeAt(t, pos);
+        const UCat cat = Category(c0.cp);
+        if (c0.cp == U'\r' || c0.cp == U'\n' || cat == UCat::kLetter ||
+            cat == UCat::kNumber || c0.end >= t.size()) {
+          continue;  // this codepoint cannot serve as the optional prefix
+        }
+        start = c0.end;
+      }
+      const size_t end = MatchTekkenAlt(t, start, upper_first_required, scratch);
+      if (end != 0) return end;
+    }
+  }
+  return 0;
+}
+
 // Rule 3: \p{N} (Qwen, max_digits=1) | \p{N}{1,3} (Llama-3, max_digits=3;
 // greedy, so long digit runs split into groups of three from the left).
 size_t MatchNumbers(std::string_view t, size_t pos, int max_digits) {
@@ -141,10 +225,13 @@ size_t MatchNumbers(std::string_view t, size_t pos, int max_digits) {
 
 // Rule 4: ` ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*` (Qwen3.6; marks_excluded=true)
 //         ` ?[^\s\p{L}\p{N}]+[\r\n]*`      (Llama-3; marks_excluded=false)
+//         ` ?[^\s\p{L}\p{N}]+[\r\n/]*`     (Tekken;  slash_in_tail=true)
 // Optional single literal ASCII space, then >=1 codepoints that are not
 // regex-space/letters/numbers (nor marks, for Qwen), then any trailing \r/\n
-// bytes. If only the space matched, the rule fails as a whole.
-size_t MatchPunctRun(std::string_view t, size_t pos, bool marks_excluded) {
+// bytes -- and '/' too for Tekken. If only the space matched, the rule fails
+// as a whole.
+size_t MatchPunctRun(std::string_view t, size_t pos, bool marks_excluded,
+                     bool slash_in_tail) {
   size_t p = pos;
   if (t[p] == ' ') ++p;
   const size_t run_begin = p;
@@ -159,7 +246,10 @@ size_t MatchPunctRun(std::string_view t, size_t pos, bool marks_excluded) {
     p = c.end;
   }
   if (p == run_begin) return 0;
-  while (p < t.size() && IsNewlineByte(t[p])) ++p;
+  while (p < t.size() &&
+         (IsNewlineByte(t[p]) || (slash_in_tail && t[p] == '/'))) {
+    ++p;
+  }
   return p;
 }
 
@@ -516,15 +606,28 @@ std::vector<std::pair<size_t, size_t>> Pretokenize(std::string_view text,
   // UNIQUE to the Qwen3.6 regex. Classic Qwen2/Qwen3 and Llama-3 both treat
   // marks like ordinary punct-run codepoints. Number grouping is single-digit
   // for BOTH Qwen variants; only Llama-3 groups \p{N}{1,3}.
-  const bool marks_aware = pattern == SplitPattern::kQwen2;
+  // kTekken needs these two APART: marks live inside its letter classes (like
+  // kQwen2) while its punct negation is [^\s\p{L}\p{N}] with no \p{M} (like
+  // kLlama3). No other pattern mixes them, which is why one flag sufficed
+  // until now.
+  const bool tekken = pattern == SplitPattern::kTekken;
+  const bool marks_in_run = pattern == SplitPattern::kQwen2 || tekken;
+  const bool marks_excluded = pattern == SplitPattern::kQwen2;
   const int max_digits = pattern == SplitPattern::kLlama3 ? 3 : 1;
   std::vector<std::pair<size_t, size_t>> spans;
+  std::vector<size_t> scratch;  // reused give-back stops for the Tekken scan
   size_t pos = 0;
   while (pos < text.size()) {
-    size_t end = MatchContraction(text, pos);
-    if (end == 0) end = MatchLetterRun(text, pos, /*marks_in_run=*/marks_aware);
+    // Tekken has NO (?i:'s|'t|...) alternative at all.
+    size_t end = tekken ? 0 : MatchContraction(text, pos);
+    if (end == 0) {
+      end = tekken ? MatchTekkenLetterRun(text, pos, scratch)
+                   : MatchLetterRun(text, pos, marks_in_run);
+    }
     if (end == 0) end = MatchNumbers(text, pos, max_digits);
-    if (end == 0) end = MatchPunctRun(text, pos, /*marks_excluded=*/marks_aware);
+    if (end == 0) {
+      end = MatchPunctRun(text, pos, marks_excluded, /*slash_in_tail=*/tekken);
+    }
     if (end == 0) end = MatchWsNewlines(text, pos);
     if (end == 0) end = MatchWsNotBeforeNonSpace(text, pos);
     if (end == 0) end = MatchWs(text, pos);

src/vllm/tokenizer/tokenizer.cpp

@@ -26,6 +26,12 @@ constexpr const char* kQwen36Regex =
     R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)";
 constexpr const char* kClassicQwen2Regex =
     R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)";
+// Mistral Tekken (mistralai/Mistral-Nemo-Instruct-2407 and the other Tekken
+// checkpoints). Byte-equal to tiktoken's o200k_base pat_str except that the
+// optional (?i:'s|'t|...) group is absent from both letter alternatives and
+// numbers are \p{N} rather than \p{N}{1,3}.
+constexpr const char* kTekkenRegex =
+    R"([^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+)";
 
 [[noreturn]] void Fail(const std::string& msg) {
   throw std::runtime_error("tokenizer: " + msg);
@@ -350,6 +356,7 @@ SplitPattern DetectPattern(const json& doc) {
   }
   if (re == kQwen36Regex) return SplitPattern::kQwen2;
   if (re == kClassicQwen2Regex) return SplitPattern::kQwen2Classic;
+  if (re == kTekkenRegex) return SplitPattern::kTekken;
   if (re.find(R"(\p{N}{1,3})") != std::string::npos) {
     return SplitPattern::kLlama3;
   }