[API] Public way to construct SentencePieceTokenizer (Unigram) from tokenizer.json / in-memory pieces+scores

[ericstj]

[API] Public way to construct SentencePieceTokenizer (Unigram) from tokenizer.json / in-memory pieces+scores

Summary

Microsoft.ML.Tokenizers fully implements the SentencePiece Unigram model
(SentencePieceUnigramModel), but the only public way to obtain a
SentencePieceTokenizer is SentencePieceTokenizer.Create(Stream), which parses a
SentencePiece protobuf (.model / ModelProto). There is no public API to build a
Unigram tokenizer from a Hugging Face tokenizer.json (or from in-memory pieces + scores).

Many modern HF models ship a JSON-only Unigram tokenizer (tokenizer.json with
model.type == "Unigram", model.vocab as [piece, score] pairs, model.unk_id, and a
normalizer/pre_tokenizer) and no sentencepiece.model/spiece.model/tokenizer.model
protobuf. For these models there is currently no supported way to construct the tokenizer.

Current behavior

• SentencePieceTokenizer's only constructor is internal SentencePieceTokenizer(ModelProto modelProto, ...).
• The Sentencepiece.* generated protobuf types (ModelProto, TrainerSpec,
  NormalizerSpec, …) are internal, so callers can't build a ModelProto directly.
• The public factory SentencePieceTokenizer.Create(Stream, bool addBeginOfSentence, bool addEndOfSentence, ...)
  requires a serialized SentencePiece protobuf stream.
• By contrast, BpeTokenizer / WordPieceTokenizer expose vocab-file/stream factories, so the
  asymmetry is Unigram-specific.

Request

A public factory to construct a Unigram SentencePieceTokenizer without a protobuf, e.g. one of:

1. From tokenizer.json — SentencePieceTokenizer.CreateFromTokenizerJson(Stream json, ...)
   (parse model.vocab pieces+scores, model.unk_id, and the normalizer/pre_tokenizer
   precompiled charsmap + metaspace settings).
2. From in-memory pieces+scores —
   Create(IEnumerable<(string Piece, float Score)> vocab, int unkId, ReadOnlySpan<byte> precompiledCharsMap, bool addDummyPrefix, bool escapeWhitespaces, ...).

Either would let callers load JSON-Unigram models that have no .model protobuf.

Workaround we're using

Since the only public entry is a protobuf stream, we synthesize a SentencePiece ModelProto
on the fly from tokenizer.json and feed the bytes to Create(Stream):

• pieces ← model.vocab [piece, score] (mapping special tokens to CONTROL/UNKNOWN types).
• trainer_spec.model_type = UNIGRAM, trainer_spec.unk_id = model.unk_id (+ bos/eos/pad ids).
• normalizer_spec.precompiled_charsmap ← the Precompiled normalizer's precompiled_charsmap
  bytes from tokenizer.json (gives byte-exact NFKC parity), plus
  add_dummy_prefix / escape_whitespaces from the Metaspace pre-tokenizer.

This works, but it requires hand-writing a SentencePiece protobuf encoder and re-deriving the
wire schema, which is exactly the kind of thing the library could expose directly. It's also
fragile across schema changes.

Why it matters

JSON-only Unigram tokenizers are common (multilingual static-embedding models, several HF
encoder models). Without a JSON/in-memory factory, every consumer must either ship a converted
.model or reimplement the protobuf synthesis above.

Repro context

• Model: a potion-multilingual-128M-style static embedding model — tokenizer.json only,
  model.type == "Unigram", ~500k-entry custom vocab, [PAD]=0 / [UNK]=1, with a
  Sequence→Precompiled normalizer (precompiled_charsmap present) and a Metaspace
  pre-tokenizer. No sentencepiece.model.
• Microsoft.ML.Tokenizers main (commit 901da3e).

[github-actions]

🔍 No similar issues found with high confidence.

---

AI-powered similar issue detection

[ericstj]

/cc @tarekgh

[github-actions]

Triage Notes

Label: enhancement — this is a well-described request for a new public API, not a bug. The library currently works correctly for .model protobuf inputs; the gap is that there is no supported path for HuggingFace JSON-only Unigram models.

Area: Microsoft.ML.Tokenizers — specifically SentencePieceTokenizer and SentencePieceUnigramModel.

---

Implementation Plan

The core blocker is that both SentencePieceTokenizer's constructor and SentencePieceBaseModel's constructor are tightly coupled to ModelProto (the internal generated protobuf type). A minimal, additive fix would involve two layers:

1. Add a raw-data constructor to SentencePieceBaseModel / SentencePieceUnigramModel

SentencePieceBaseModel (src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs) extracts a handful of scalar fields from ModelProto (bos/eos/unk tokens and IDs, addDummyPrefix, escapeWhitespaces, byteFallback, precompiledCharsmap). A second internal constructor that accepts those fields directly — with no ModelProto dependency — would allow the Unigram model to be built from any source.

SentencePieceUnigramModel (src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs) similarly iterates modelProto.Pieces to populate its _vocab/_vocabReverse/_trie. A companion constructor accepting IReadOnlyList<(string Piece, float Score, PieceType Type)> (or a simpler (string, float)[] for normal tokens) alongside the scalar config would complete the internal change.

2. Expose two new public factories on SentencePieceTokenizer

In src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:

// Option A – from HuggingFace tokenizer.json (preferred for the reported use case)
public static SentencePieceTokenizer CreateFromHuggingFaceJson(
    Stream jsonStream,
    bool addBeginningOfSentence = true,
    bool addEndOfSentence = false,
    IReadOnlyDictionary<string, int>? specialTokens = null);

// Option B – from in-memory pieces + scores (building block, and useful on its own)
public static SentencePieceTokenizer Create(
    IEnumerable<(string Piece, float Score)> vocab,
    int unkId,
    ReadOnlySpan<byte> precompiledCharsMap = default,
    bool addDummyPrefix = true,
    bool escapeWhiteSpaces = true,
    bool byteFallback = false,
    bool addBeginningOfSentence = true,
    bool addEndOfSentence = false,
    IReadOnlyDictionary<string, int>? specialTokens = null);

CreateFromHuggingFaceJson would parse the well-known HF schema (model.type, model.vocab as [piece, score] pairs, model.unk_id, normalizer.precompiled_charsmap, pre_tokenizer Metaspace settings) using System.Text.Json and delegate to the in-memory Create overload. Note that SentencePieceNormalizer already accepts a raw precompiledCharsMap span, so no normalizer changes are needed.

Files likely affected:

File	Change
src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs	Add non-ModelProto internal constructor
src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs	Add non-ModelProto internal constructor
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs	Add CreateFromHuggingFaceJson + in-memory Create overload; update existing internal constructor to route through either path
test/Microsoft.ML.Tokenizers.Tests/	Add round-trip tests using a real JSON-only Unigram tokenizer.json

The BPE path (SentencePieceBpeModel) has similar coupling and could follow the same pattern in a follow-up if needed, but the issue is Unigram-specific.

Generated by Issue Triage for issue #7624 · ● 374.7K · ◷

[tarekgh]

@ericstj SentencePiece supports different tokenizer models, such as BPE and Unigram. As a result, Hugging Face tokenizer JSON files can have different formats depending on the underlying tokenizer model.

For example, the Llama 1 tokenizer uses SentencePiece BPE, and its tokenizer JSON follows a format similar to a generic BPE tokenizer (e.g., the Llama tokenizer JSON https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/raw/main/tokenizer.json. However, this format cannot be loaded directly by the SentencePiece tokenizer and must first be converted into the SentencePiece model format. This is why many models ship a tokenizer.model file alongside the tokenizer.json file.

In contrast, if you look at a SentencePiece Unigram tokenizer, such as the one used by paraphrase-multilingual-MiniLM-L12-v2, you'll notice that its tokenizer JSON has a different structure that reflects the Unigram model representation https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/blob/main/tokenizer.json.

[ericstj]

@tarekgh I see - do you think Microsoft.ML.Tokenizers should support all formats? Or maybe we just expose enough to let the caller parse themselves and create it?

[tarekgh]

do you think Microsoft.ML.Tokenizers should support all formats? Or maybe we just expose enough to let the caller parse themselves and create it?

We didn't support json files with SentencePiece in the first place for two reasons, the first is there is no consistent json format for different models, the second is all HuggingFace LLMs using such tokenizer ship with tokenizer.model file which we support. This will narrow down the actual need to the json files. And we have recommended for anyone really need json can parse it. IIRC, we had some test code doing such parsing as example users can use. so I agree more with we just expose enough to let the caller parse themselves and create it.

[ericstj]

The potion multi-lingual model I was using did not have a protobuf file. I'm working around it by constructing the protobuf myself -- ericstj/Model2Vec.Net#5. AFAICT there is no way to construct our SentencePiece tokenizer without providing a protobuf file/stream. I suppose we could force users to understand the JSON and the protobuf format - but that's a gap compared to using those huggingface models in other frameworks.

Do "all HF SentencePiece models also ship tokenizer.model?": true for generative LLMs (SP BPE), not for Unigram — the encoder/reranker ecosystem this targets. HuggingFace Hub survey, 491 SentencePiece-Unigram repos ordered by top downloads:

ship tokenizer.json	also ship SP proto	json-only (no proto)
288	166	122 (43%)

Verified top 40 json-only: 36 are genuine Unigram, no proto (~7.7M monthly downloads) — e.g. mxbai-rerank-*, cross-encoder/nli-deberta-v3-*, MoritzLaurer/mDeBERTa-v3-*, BGE-m3, meta-llama/Prompt-Guard-86M.

This PR supports the feature set used by all of these. Seems like enabling 43% of the top models using SentencePiece-Unigram is worth the effort. A gap analysis of our implementation compared to upstream uncovered the following missing features, which are omitted/deferred as mentioned below.

1. Merged <unk> for out-of-vocabulary runs
   When a multi-character span has no vocabulary match and byte_fallback is off, upstream SentencePiece merges the maximal unknown run into a single <unk> token; Microsoft.ML.Tokenizers instead emits one <unk> per character. This is existing core-Unigram behavior — it lives in the Unigram encoder and applies equally to the protobuf (tokenizer.model) path, not just JSON loading — so changing it would shift existing baselines and is out of scope for this PR. It only manifests when a model is fed script it doesn't cover (e.g. CJK to an English/pruned vocab); the target multilingual models cover their scripts and are unaffected. The Model2Vec.Net workaround shares this exact behavior (it uses the same core encoder), so the JSON loader is no worse on this axis.

2. Trailing/leading-whitespace exactness on Replace-collapse models
   Some models express whitespace collapsing as a normalizer Replace (e.g. " {2,}" → "▁") rather than SentencePiece's remove_extra_whitespaces. These are kept on the fast path and approximated with remove_extra_whitespaces, which differs only at the string ends — it strips a leading/trailing whitespace where the literal Replace would emit a single ▁. Reproducing it exactly would push these (high-traffic: t5, pegasus, bge-m3, NLLB) onto the slower per-call normalizer-chain path for a difference that only appears on inputs that begin or end with whitespace. The fast-path approximation is kept deliberately.

3. Pair-sequence templates and type_ids
   post_processor templates declare both a single and a pair form; the pair form wraps two segments and emits a per-token type_id (segment mask). Only the single template is consumed. Supporting pairs requires new public API — a pair-encode entry point and a richer encoding result carrying type_ids — which is a separate design from this change. Single-sequence encoding (embeddings, classification, generation) is the dominant path; pair encoding concentrates in cross-encoders/rerankers.

4. Non-Unigram models from tokenizer.json (BPE / WordPiece)
   This loader is the SentencePiece Unigram path; BPE and WordPiece tokenizer.json files are different model families served by BpeTokenizer / BertTokenizer and are out of scope here.

5. ByteLevel post-processor (trim_offsets)
   A GPT-2/RoBERTa ByteLevel-BPE construct that only adjusts reported character offsets and adds no special tokens — orthogonal to SentencePiece Unigram and not applicable.

Perf/tests: common path unchanged, no new per-call allocations; 406 tests pass on net8.0 and net48.