[None][fix] Force slow tokenizer for Kimi K2.5 (TikTokenTokenizer) by qiaoxj07 · Pull Request #14416 · NVIDIA/TensorRT-LLM

qiaoxj07 · 2026-05-21T14:17:49Z

Summary

Force use_fast=False when loading the Kimi K2.5 tokenizer so the model's own slow TikTokenTokenizer (tokenization_kimi.py) is used instead of transformers 5.x's auto-converted Fast tokenizer.
In tensorrt_llm/bench/utils/data.py:initialize_tokenizer, gate use_fast=False on tokenizer_class == "TikTokenTokenizer" via a new helper that reads tokenizer_config.json without instantiating, so behavior is unchanged for every other model.
In tensorrt_llm/_torch/models/modeling_kimi_k25.py:KimiK25InputProcessor.__init__, hardcode use_fast=False for AutoTokenizer (the file is K2.5-specific) and pin self._processor.tokenizer = self._tokenizer so any text path through self._processor(text=...) uses the same slow tokenizer. AutoProcessor's use_fast is left as-is because it only controls image processor selection.

Why

Kimi K2.5 ships only a slow TikTokenTokenizer(PreTrainedTokenizer) with a custom CJK-aware pat_str regex; there is no Fast equivalent and no tokenizer.json upstream. transformers >= 5 newly tries to auto-convert it to a Fast tokenizer via TokenizersBackend.convert_to_native_format, which:

Sees vocab_file = tiktoken.model exists.
Tries to extract it as SentencePiece -> fails ("Wrong wire type in tag").
Logs Could not extract SentencePiece model … Falling back to TikToken extractor.
Builds a Fast tokenizer from the BPE merges with a generic pre-tokenization regex (not Kimi's pat_str).

The resulting Fast tokenizer mis-segments multilingual / CJK content, producing roughly +20% more tokens for the same input string compared to the slow path. In one benchmark sweep (kimi-k2-sol-8192-1024-ratio08), this caused dataset prompts that were generated at <=8192 tokens to be re-tokenized by trtllm-bench at up to 9820 tokens, overflowing the server's max_seq_len and rejecting requests with HTTP 400.

Upstream's own constraints corroborate this: moonshotai/Kimi-K2.5 README pins transformers >= 4.57.1, and the sibling Kimi-K2.6 (same architecture) caps to < 5.0.0. This PR makes trtllm work correctly under either transformers 4.x or 5.x without requiring a transformers downgrade.

Test plan

trtllm-bench throughput on a Kimi-K2.5 checkpoint under transformers >= 5: confirm the DATASET DETAILS percentiles match the slow tokenizer (e.g. for the random-v2 ratio-0.8 ISL=8192 dataset, P100 should be <= 8192 + chat-template overhead, not ~9820).
trtllm-bench throughput on a non-Kimi model (e.g. Llama, DeepSeek-V3) under transformers >= 5: confirm use_fast=True is still selected (no perf regression for the Fast-tokenizer path) by inspecting the tokenizer type after init.
Kimi K2.5 multimodal serving smoke test: confirm self._processor(text=...) and self._tokenizer(text) yield the same input_ids for a sample prompt.

References

https://huggingface.co/moonshotai/Kimi-K2.5/discussions/7 — slow tokenizer issue thread
https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/README.md — transformers >= 4.57.1
https://huggingface.co/moonshotai/Kimi-K2.6 — sibling model caps transformers < 5.0.0
https://github.com/huggingface/transformers/blob/v5.1.0/src/transformers/tokenization_utils_tokenizers.py — convert_to_native_format SentencePiece-then-TikToken fallback

Summary by CodeRabbit

Bug Fixes
- Fixed tokenizer initialization to ensure consistent tokenizer behavior across all text and image preprocessing paths, significantly improving model reliability and performance predictability across different scenarios
- Enhanced tokenizer selection logic to automatically determine and apply the correct tokenizer implementation for each specific model, improving system compatibility, stability, and overall robustness

Kimi K2.5 ships only the slow `TikTokenTokenizer` (model repo's `tokenization_kimi.py`). transformers >= 5 tries to auto-convert it to a Fast tokenizer via a SentencePiece-then-TikToken fallback that does not honor the custom CJK-aware `pat_str`. The fallback Fast tokenizer mis-segments multilingual content, producing ~+20% more tokens than the slow path for the same input string. Concretely, benchmark prompts that the dataset generator produced at <= 8192 tokens get re-tokenized at the bench to up to ~9820 tokens. Two fixes: 1. `tensorrt_llm/bench/utils/data.py`: in `initialize_tokenizer`, inspect `tokenizer_class` from `tokenizer_config.json` (without instantiating) and pass `use_fast=False` to `AutoTokenizer` for classes in `_REQUIRES_SLOW_TOKENIZER` (currently `{"TikTokenTokenizer"}`). Behavior is unchanged for all other models. 2. `tensorrt_llm/_torch/models/modeling_kimi_k25.py`: in `KimiK25InputProcessor.__init__`, hardcode `use_fast=False` for the `AutoTokenizer` call (the file is K2.5-specific) and pin `self._processor.tokenizer = self._tokenizer` so any text path through `self._processor(text=...)` uses the same slow tokenizer. `AutoProcessor`'s `use_fast` is left as-is because it only affects image processor selection. References: - https://huggingface.co/moonshotai/Kimi-K2.5/discussions/7 - https://huggingface.co/moonshotai/Kimi-K2.6 (transformers < 5.0.0) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

coderabbitai · 2026-05-21T14:22:29Z

📝 Walkthrough

Walkthrough

The PR enforces slow tokenizer usage in two complementary locations: a general tokenizer initialization utility that detects when slow tokenizers are required via configuration inspection, and the KimiK25 model which explicitly uses slow tokenizers and synchronizes processor and text tokenization paths.

Changes

Slow Tokenizer Enforcement

Layer / File(s)	Summary
Slow tokenizer detection infrastructure `tensorrt_llm/bench/utils/data.py`	Adds `os` import, introduces `_REQUIRES_SLOW_TOKENIZER` set of known tokenizer classes that require slow implementations, and implements `_tokenizer_class_from_config` helper that reads `tokenizer_config.json` from local or HuggingFace Hub sources to extract the `tokenizer_class` field.
Tokenizer initialization with slow-tokenizer support `tensorrt_llm/bench/utils/data.py`	Updates `initialize_tokenizer` to compute a `use_fast` flag by checking the extracted tokenizer class against the slow tokenizer set, then passes this flag to `AutoTokenizer.from_pretrained` to override default fast tokenizer behavior.
KimiK25 tokenizer and processor synchronization `tensorrt_llm/_torch/models/modeling_kimi_k25.py`	Constructs `KimiK25InputProcessor._tokenizer` with explicit `use_fast=False` when loading from model repository; loads `self._processor` via `AutoProcessor.from_pretrained` and pins its `.tokenizer` attribute to `self._tokenizer` to maintain consistent tokenization across text and image preprocessing.

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested reviewers

VALLIS-NERIA

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically identifies the main change: forcing the slow tokenizer for Kimi K2.5 (TikTokenTokenizer). It aligns directly with the primary objective of the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	PR description is comprehensive with detailed summary, motivation, test plan with checkmarks, and references.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/bench/utils/data.py`:
- Around line 46-51: The docstring for the function with signature ending in
"model_name_or_path: str) -> Optional[str]:" must be reformatted to satisfy Ruff
D205: make the summary a single line (one-sentence short description), add one
blank line, then put the expanded description/details on subsequent lines;
adjust the existing triple-quoted string so the first line is the short summary,
followed by a blank line, then the longer explanation about reading
"tokenizer_class" from "tokenizer_config.json" and the return behavior.
- Around line 58-65: The hf_hub_download call can raise several
huggingface_hub-specific exceptions that aren’t covered by the current except
tuple, causing the tokenizer fallback to fail; update the try/except around
hf_hub_download (the call using hf_hub_download(repo_id=model_name_or_path,
filename="tokenizer_config.json")) to also catch
huggingface_hub.utils.EntryNotFoundError, LocalEntryNotFoundError,
RepositoryNotFoundError, RevisionNotFoundError and
huggingface_hub.errors.GatedRepoError (either by importing and adding them to
the except tuple or by catching the broader huggingface_hub errors namespace) so
that any of those failures return None as the fallback.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d268af38-52ee-4d87-89f6-329cb060e019

📥 Commits

Reviewing files that changed from the base of the PR and between 6bf1701 and e7718cc.

📒 Files selected for processing (2)

tensorrt_llm/_torch/models/modeling_kimi_k25.py
tensorrt_llm/bench/utils/data.py

… no-op) In transformers 5.x, ``AutoTokenizer.from_pretrained(..., use_fast=False)`` is silently ignored for Kimi-K2.5's ``TikTokenTokenizer`` -- the returned tokenizer is still ``TokenizersBackend`` with ``is_fast=True``, built via the new generic SentencePiece-then-TikToken fast converter that does not honor the custom CJK-aware ``pat_str``. Bench-side empirical verification on a transformers 5.5.3 container: even with ``use_fast=False`` set, the dataset percentile MAX stayed at 9820 (the inflated number); switching to direct class load brought it back to 8156, matching the transformers 4.57.3 baseline exactly. Instead of toggling ``use_fast``, load the slow class directly via ``transformers.dynamic_module_utils.get_class_from_dynamic_module`` from the model repo's remote code: 1. ``tensorrt_llm/bench/utils/data.py``: replace ``_REQUIRES_SLOW_TOKENIZER`` with ``_DYNAMIC_TOKENIZER_LOAD`` mapping ``tokenizer_class`` -> ``"module.Class"``; for hits, instantiate the slow class directly. Fall back to ``AutoTokenizer`` for everything else and on dynamic-load failure. 2. ``tensorrt_llm/_torch/models/modeling_kimi_k25.py``: same direct-load trick in ``KimiK25InputProcessor.__init__``, with ``AutoTokenizer`` fallback. ``self._processor.tokenizer = self._tokenizer`` pinning is retained so any text path through ``self._processor(text=...)`` uses the same slow tokenizer. Verified on OCI GB200 with ``trtllm-bench throughput`` against the ``kimi-k25-8192-1-20000-ratio-08`` dataset on a transformers 5.5.3 container -- dataset percentiles: MIN 6571, MAX 8156, AVG 7415, P99 8155 (matches the 2026-05-19 transformers 4.57.3 baseline). Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

qiaoxj07 · 2026-05-21T14:52:37Z

Updated approach: use_fast=False is a no-op in transformers >= 5 for this tokenizer (verified empirically — AutoTokenizer.from_pretrained(model_path, use_fast=False) on a transformers 5.5.3 container still returned TokenizersBackend, is_fast=True, and the dataset MAX percentile remained inflated at 9820).

Switched to loading the slow TikTokenTokenizer class directly via transformers.dynamic_module_utils.get_class_from_dynamic_module from the model repo's remote code, bypassing AutoTokenizer's conversion entirely. The hot-patched bench job on OCI GB200 with transformers 5.5.3 now produces dataset percentiles MIN 6571 / MAX 8156 / AVG 7415 / P99 8155 — exactly matching the transformers 4.57.3 baseline from 2026-05-19.

…ixes trtllm-serve) Previous iteration only patched ``tensorrt_llm/bench/utils/data.py`` and ``tensorrt_llm/_torch/models/modeling_kimi_k25.py``. That fixes the bench-side dataset percentile path, but leaves trtllm-serve (and the LLM API runtime in general) still using the broken Fast-converted tokenizer: LLM.__init__ -> _try_load_tokenizer -> ModelLoader.load_hf_tokenizer -> tensorrt_llm.tokenizer.tokenizer.load_hf_tokenizer -> TransformersTokenizer.from_pretrained -> AutoTokenizer.from_pretrained(..., use_fast=True_or_False) -> [broken Fast TokenizersBackend under transformers >= 5] That ``self._tokenizer`` is then passed in to ``create_input_processor`` which calls ``KimiK25InputProcessor(model_path, config, tokenizer, ...)`` with the broken tokenizer; the input processor stores it as ``self._tokenizer`` and uses it for every incoming request's encoding, producing the same +20% token-count inflation we saw in the bench dataset stats. This is what caused the original CTX worker's ``default_max_tokens (-746) = max_seq_len (8232) - splited_prompt_len (8978)`` 400 Bad Request errors at the very top of this thread. Consolidate the fix into one place at the runtime tokenizer layer: 1. ``tensorrt_llm/tokenizer/tokenizer.py``: add public ``DYNAMIC_SLOW_TOKENIZER_MAP`` and ``try_load_dynamic_slow_tokenizer``. Wire the helper into ``TransformersTokenizer.from_pretrained`` so it is tried first; on hit, wrap the slow class in TransformersTokenizer and return; on miss / failure, fall through to the existing AutoTokenizer path unchanged. 2. ``tensorrt_llm/bench/utils/data.py``: replace the inline detect-and- load logic with a call to the shared helper. Returns the raw HF tokenizer (PreTrainedTokenizer subclass) as before. 3. ``tensorrt_llm/_torch/models/modeling_kimi_k25.py``: revert the AutoTokenizer call to its original form (the runtime fix now loads the slow tokenizer upstream). Keep ``self._processor.tokenizer = self._tokenizer`` so any text path through ``self._processor(text=...)`` uses the same tokenizer the rest of the class uses (the processor builds its own internal tokenizer copy via AutoProcessor that is independently mis-converted). Verified on OCI GB200 with the same transformers 5.5.3 container via ``trtllm-bench throughput`` on the Kimi K2.5-NVFP4 ctx-only sweep: dataset MIN/MAX/AVG/P99 = 6571 / 8156 / 7415 / 8155, identical to the 2026-05-19 transformers 4.57.3 baseline, and zero ``400 Bad Request`` errors during the run. References: - https://huggingface.co/moonshotai/Kimi-K2.5/discussions/7 - https://huggingface.co/moonshotai/Kimi-K2.6 (pins transformers < 5.0.0) - https://github.com/huggingface/transformers/blob/v5.1.0/src/transformers/tokenization_utils_tokenizers.py Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

qiaoxj07 · 2026-05-21T15:21:00Z

v3: moved the fix into the runtime tokenizer layer so trtllm-serve is also covered.

The previous v2 only fixed the bench-side dataset prep path (initialize_tokenizer) and the K2.5 multimodal KimiK25InputProcessor.__init__ for the tokenizer is None branch. trtllm-serve was still broken because LLM._try_load_tokenizer -> ModelLoader.load_hf_tokenizer -> TransformersTokenizer.from_pretrained -> AutoTokenizer.from_pretrained still produced the broken Fast tokenizer, which was then handed to KimiK25InputProcessor via create_input_processor -- bypassing the tokenizer is None branch entirely. The original CTX worker's default_max_tokens (-746) = max_seq_len (8232) - splited_prompt_len (8978) HTTP 400 errors at the top of the thread were caused by exactly this path.

v3 consolidates:

New public helper try_load_dynamic_slow_tokenizer(model_dir, **kwargs) in tensorrt_llm/tokenizer/tokenizer.py, driven by DYNAMIC_SLOW_TOKENIZER_MAP = {"TikTokenTokenizer": "tokenization_kimi.TikTokenTokenizer"}.
TransformersTokenizer.from_pretrained now tries the helper before falling back to AutoTokenizer.from_pretrained. This fixes every runtime call site (LLM API, trtllm-serve, PyExecutor model engine) at once.
bench/utils/data.py:initialize_tokenizer calls the same helper for the dataset-prep path.
modeling_kimi_k25.py reverts to the original AutoTokenizer call (the upstream _tokenizer is now slow already); the only retained tweak is pinning self._processor.tokenizer = self._tokenizer so the K2.5 processor's internal tokenizer (built independently by AutoProcessor) doesn't diverge.

Re-verified on OCI GB200 / transformers 5.5.3 via trtllm-bench throughput on Kimi K2.5-NVFP4 with the same 8192/1/ratio-08 dataset: dataset MIN/MAX/AVG/P99 = 6571 / 8156 / 7415 / 8155 (matches the 4.57.3 baseline) and zero HTTP 400 errors during the run.

…rt + yapf Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

…209) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

qiaoxj07 · 2026-05-21T15:57:30Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-21T16:04:22Z

PR_Github #49727 [ run ] triggered by Bot. Commit: 45a6a6b Link to invocation

tensorrt-cicd · 2026-05-21T19:54:45Z

PR_Github #49727 [ run ] completed with state SUCCESS. Commit: 45a6a6b
/LLM/main/L0_MergeRequest_PR pipeline #39334 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

Detect auto_map.AutoTokenizer = [SlowClass, null] after AutoTokenizer returns Fast, then reload the slow class via get_class_from_dynamic_module. Avoids transformers 5.x mis-converting Kimi K2/K2.5's TikTokenTokenizer to a Fast tokenizer with a generic regex that drops the model's pat_str. References NVIDIA#14416. Signed-off-by: Zhenhuan Chen <zhenhuanc@nvidia.com>

…enizer initialize_tokenizer in bench/utils/data.py called AutoTokenizer directly, bypassing the post-load fixes wired into TransformersTokenizer.from_pretrained -- in particular maybe_fix_byte_level_tokenizer, which prevents DeepSeek-V3-family models from loading with a Metaspace pre-tokenizer that silently strips spaces ('hello world' -> 'helloworld') on transformers >= 5.x. Route through the shared wrapper and peel off .tokenizer to return the raw HF tokenizer the rest of bench code expects. Adds a unit test that pins the routing. References NVIDIA#14416. Signed-off-by: Zhenhuan Chen <zhenhuanc@nvidia.com>

…h TransformersTokenizer Two entry points were calling AutoTokenizer.from_pretrained directly, bypassing the post-load fixes wired into TransformersTokenizer.from_pretrained -- in particular maybe_fix_byte_level_tokenizer (DeepSeek-V3 Metaspace bug that silently strips spaces) and _fallback_to_fast_tokenizer (DeepSeek-V3.2 AttributeError on transformers >= 5.x): * bench/utils/data.py:initialize_tokenizer (used by trtllm-bench) * serve/router.py:BlockHashMixin._get_tokenizer (used by trtllm-serve block-hash KV-cache routing -- it hand-inlined the byte-level fix only, missing the DeepSeek-V3.2 fallback) Route both through TransformersTokenizer.from_pretrained(...).tokenizer so they inherit the canonical fix list. Adds two guard tests. References NVIDIA#14416. Signed-off-by: Zhenhuan Chen <zhenhuanc@nvidia.com>

qiaoxj07 requested review from a team as code owners May 21, 2026 14:17

qiaoxj07 requested review from FrankD412 and Wanli-Jiang May 21, 2026 14:17

github-actions Bot assigned qiaoxj07 May 21, 2026

coderabbitai Bot reviewed May 21, 2026

View reviewed changes

Comment thread tensorrt_llm/bench/utils/data.py Outdated

Comment thread tensorrt_llm/bench/utils/data.py Outdated

qiaoxj07 added 3 commits May 21, 2026 23:22

Merge branch 'main' into user/xqiao/fix-kimi-k25-fast-tokenizer

781f0ee

[None][fix] Format tensorrt_llm/tokenizer/tokenizer.py to satisfy iso…

7e8ec84

…rt + yapf Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

[None][fix] Fix docstring style on new tokenizer helpers (ruff D205/D…

45a6a6b

…209) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

Wanli-Jiang approved these changes May 22, 2026

View reviewed changes

qiaoxj07 enabled auto-merge (squash) May 22, 2026 04:44

qiaoxj07 disabled auto-merge May 22, 2026 04:44

dc3671 mentioned this pull request May 22, 2026

[None][fix] Route trtllm-bench and trtllm-serve tokenizer load through TransformersTokenizer #14452

Merged

2 tasks

qiaoxj07 closed this May 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][fix] Force slow tokenizer for Kimi K2.5 (TikTokenTokenizer) - #14416

[None][fix] Force slow tokenizer for Kimi K2.5 (TikTokenTokenizer)#14416
qiaoxj07 wants to merge 6 commits into
NVIDIA:mainfrom
qiaoxj07:user/xqiao/fix-kimi-k25-fast-tokenizer

qiaoxj07 commented May 21, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 21, 2026 •

edited

Loading

Walkthrough

Changes

Suggested reviewers

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

qiaoxj07 commented May 21, 2026

Uh oh!

qiaoxj07 commented May 21, 2026

Uh oh!

qiaoxj07 commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

qiaoxj07 commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Test plan

References

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Suggested reviewers

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

qiaoxj07 commented May 21, 2026

Uh oh!

qiaoxj07 commented May 21, 2026

Uh oh!

qiaoxj07 commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qiaoxj07 commented May 21, 2026 •

edited

Loading

coderabbitai Bot commented May 21, 2026 •

edited

Loading