[TRTLLM-9654][feat] Support DeepSeek-V32 chat template

[chang-l]

Created tensorrt_llm/llmapi/deepseek_v32/for deepseek v3.2 tokenizer/chat_template and encoding functions

Usage:
In addition to DeepSeek-V3.2-Exp (already supported), simply custom tokenizer flag if deploying with HF/DeepSeek-V3.2:

trtllm-eval ... --custom_tokenizer deepseek_v32

trtllm-serve ... --custom_tokenizer deepseek_v32

Evaluation:

• GPQA

trtllm-eval --model ${model_path} \
 --tp_size 4 \
 --ep_size 4 \
 --kv_cache_free_gpu_memory_fraction ${kv_fraction} \
 --extra_llm_api_options /tmp/extra-llm-api-config.yml \
 --custom_tokenizer deepseek_v32 \
 gpqa_diamond \
 --apply_chat_template \
 --chat_template_kwargs '{"thinking": true}' \
 --max_output_length 120000 

[12/10/2025-23:01:12] [TRT-LLM] [I] lm-eval gpqa_diamond_cot_zeroshot_aa results (scores normalized to range 0~100):
|           Tasks            |Version|   Filter   |n-shot|  Metric   |   | Value |   |Stderr|
|----------------------------|------:|------------|-----:|-----------|---|------:|---|-----:|
|gpqa_diamond_cot_zeroshot_aa|      1|strict-match|     0|exact_match|↑  |83.3333|±  |2.6552|

• AIME25

pip install lm-eval==0.4.9.2
lm_eval --model local-chat-completions --model_args "model=deepseek-ai/DeepSeek-V3.2,base_url=http://0.0.0.0:8111/v1/chat/completions,tokenized_requests=False,tokenizer_backend=None,num_concurrent=20,timeout=5000,max_length=72768" --tasks aime25 --apply_chat_template --gen_kwargs '{"temperature":1.0,"max_gen_toks":72768,"top_p":0.95,"chat_template_kwargs":{"thinking":true}}' --log_samples --output_path "aime25_ds32_v2"

local-chat-completions (model=deepseek-ai/DeepSeek-V3.2,base_url=http://0.0.0.0:8111/v1/chat/completions,tokenized_requests=False,tokenizer_backend=None,num_concurrent=20,timeout=5000,max_length=72768), gen_kwargs: ({'temperature': 1.0, 'max_gen_toks': 72768, 'top_p': 0.95, 'chat_template_kwargs': {'thinking': True}}), limit: None, num_fewshot: None, batch_size: 1
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.9333|±  |0.0463|

Summary by CodeRabbit

Release Notes

• New Features
  • Added --tokenizer_mode CLI option (choices: auto, slow, deepseek_v32; default: auto) for eval and serve commands
  • Introduced DeepSeek V32 tokenizer support with specialized chat template handling and tool invocation capabilities

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[coderabbitai]

📝 Walkthrough

Walkthrough

Introduces DeepSeek V32 tokenizer support to the TensorRT LLM framework by adding a new --tokenizer_mode CLI option with "deepseek_v32" variant, creating a dedicated encoding module for DSML-based message formatting and tool invocation, and integrating the custom tokenizer into the existing tokenizer initialization pipeline.

Changes

Cohort / File(s)	Summary
CLI Enhancement tensorrt_llm/commands/eval.py, tensorrt_llm/commands/serve.py	Added --tokenizer_mode option (choices: auto, slow, deepseek_v32; default: auto) to both eval and serve commands; updated function signatures and propagated tokenizer_mode parameter through to llm_args dictionary.
New DeepSeek V32 Module tensorrt_llm/llmapi/deepseek_v32/__init__.py, tensorrt_llm/llmapi/deepseek_v32/tokenizer.py, tensorrt_llm/llmapi/deepseek_v32/encoding.py	Created new package with three components: package initialization re-exporting public API (DeepseekV32Tokenizer, encode_messages, parse_message_from_completion_text); DeepseekV32Tokenizer class wrapping HF tokenizer with specialized chat_template handling and delegated tokenizer methods; comprehensive encoding module with DSML templates, message rendering, tool invocation handling, and completion text parsing utilities.
Tokenizer Initialization Integration tensorrt_llm/llmapi/llm_args.py	Extended tokenizer_mode field from Literal['auto', 'slow'] to include 'deepseek_v32'; added conditional initialization branch in validate_and_init_tokenizer to instantiate DeepseekV32Tokenizer when mode is 'deepseek_v32'.
Chat Template Processing tensorrt_llm/inputs/utils.py	Added import for DeepseekV32Tokenizer and new code path in chat template handling to detect and delegate to DeepseekV32Tokenizer.apply_chat_template when applicable.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

• encoding.py: Review the comprehensive DSML templates, token definitions, message rendering logic, and completion text parsing—ensure correctness of regex patterns and state machine logic for tool call extraction and thinking block handling
• tokenizer.py: Verify that the delegated tokenizer interface correctly implements all required methods and properties; pay attention to the from_pretrained classmethod and apply_chat_template integration
• llmapi/llm_args.py: Confirm that the new tokenizer_mode initialization path properly imports and instantiates DeepseekV32Tokenizer with correct parameter passing
• CLI propagation: Verify that tokenizer_mode is correctly threaded through eval and serve command paths into the downstream llm_args dictionary

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 2.63% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Description check	⚠️ Warning	PR description lacks the required template structure. Missing 'Description' section, 'Test Coverage' section is empty, and PR Checklist items are not substantively addressed.	Fill in the 'Description' section explaining what and why, add specific test coverage details, and address all PR Checklist items with clear documentation of compliance or justification.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly and specifically describes the main change: adding support for DeepSeek-V32 chat template functionality with the appropriate JIRA ticket and feature type.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (6)

tensorrt_llm/llmapi/deepseek_v32/tokenizer.py (3)

19-22: Prefer calling super().__init__ for base-class initialization

__init__ currently re-implements the base TransformersTokenizer initializer instead of delegating. This works today but is brittle if the base class adds fields later.

-    def __init__(self, tokenizer):
-        # tokenizer should be the HF tokenizer
-        self.tokenizer = tokenizer
-        self._all_special_tokens_set = set(self.tokenizer.all_special_tokens)
+    def __init__(self, tokenizer):
+        # tokenizer should be the HF tokenizer
+        super().__init__(tokenizer)

---

25-42: Remove or use the unused download_dir parameter

from_pretrained declares download_dir but never uses it or forwards it to AutoTokenizer.from_pretrained, which is slightly misleading and triggers static analysis complaints.

Either drop the parameter or pass it through (e.g., as cache_dir) if that was the intent. If you don’t plan to support it, simplifying the signature is clearer.

---

44-55: Clarify behavior of apply_chat_template for DeepSeek-specific kwargs

apply_chat_template only consumes a thinking flag from **kwargs and ignores other possible arguments (like add_generation_prompt, chat_template, etc.) that upstream callers might pass for HF chat templates.

The current behavior is fine for the DeepSeek V3.2 path as wired in inputs/utils.apply_chat_template, but it would help future readers to document this explicitly in a short docstring or comment so it’s clear which kwargs are honored and that generation-prompt behavior is entirely encoded in encode_messages.

tensorrt_llm/llmapi/deepseek_v32/encoding.py (1)

208-215: Parenthesize and/or chain for clarity

The assertion mixes or and and:

assert index == 0 or prev_assistant_idx >= 0 and assistant_msg.get("role") == "assistant", ...

Although the current precedence (index == 0 or (prev_assistant_idx >= 0 and role == "assistant")) is likely what you intend, it’s clearer and silences RUF021 to make it explicit:

-        assert index == 0 or prev_assistant_idx >= 0 and assistant_msg.get("role") == "assistant", (
+        assert index == 0 or (
+            prev_assistant_idx >= 0 and assistant_msg.get("role") == "assistant"
+        ), (
             f"Invalid messages at {index}:\n{assistant_msg}"
         )

tensorrt_llm/inputs/utils.py (1)

585-595: DeepSeek V3.2 chat-template branch is coherent

The DeepSeek-specific branch cleanly short-circuits before the generic TransformersTokenizer path, delegates prompt construction to DeepseekV32Tokenizer.apply_chat_template, and mirrors the enable_tokenize behavior by returning encode(prompt) when requested.

Note that add_generation_prompt and HF chat_template resolution are intentionally bypassed for this mode; consider adding a short comment to that effect if you expect future contributors to extend this function.

tensorrt_llm/llmapi/llm_args.py (1)

2052-2063: DeepSeek tokenizer init branch works; document instance-vs-path expectation

The new elif self.tokenizer_mode == 'deepseek_v32' branch correctly wires DeepseekV32Tokenizer.from_pretrained with either the explicit tokenizer path or the model name and preserves trust_remote_code / use_fast semantics.

One nuance: this assumes self.tokenizer is a path/name, not an already-instantiated tokenizer. If a caller passes a tokenizer object while also setting tokenizer_mode='deepseek_v32', from_pretrained will likely fail. If that scenario is possible in your usage, consider:

• Guarding with isinstance(self.tokenizer, (str, Path)) and, if it’s already a tokenizer, wrapping it directly: self.tokenizer = DeepseekV32Tokenizer(self.tokenizer), or
• Explicitly documenting that for deepseek_v32 you must provide a path/name, not a tokenizer instance.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 390391e and fa3fa6e.

📒 Files selected for processing (7)

• tensorrt_llm/commands/eval.py (3 hunks)
• tensorrt_llm/commands/serve.py (5 hunks)
• tensorrt_llm/inputs/utils.py (2 hunks)
• tensorrt_llm/llmapi/deepseek_v32/__init__.py (1 hunks)
• tensorrt_llm/llmapi/deepseek_v32/encoding.py (1 hunks)
• tensorrt_llm/llmapi/deepseek_v32/tokenizer.py (1 hunks)
• tensorrt_llm/llmapi/llm_args.py (2 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: The code developed for TensorRT-LLM should conform to Python 3.8+
Indent Python code with 4 spaces; do not use tabs
Always maintain the namespace when importing in Python, even if only one class or function from a module is used (e.g., use from package.subpackage import foo and then foo.SomeClass() instead of from package.subpackage.foo import SomeClass)
Python filenames should use snake_case (e.g., some_file.py)
Python class names should use PascalCase (e.g., class SomeClass)
Python function and method names should use snake_case (e.g., def my_awesome_function():)
Python local variable names should use snake_case, with prefix k for variable names that start with a number (e.g., k_99th_percentile = ...)
Python global variables should use upper snake_case with prefix G (e.g., G_MY_GLOBAL = ...)
Python constants should use upper snake_case (e.g., MY_CONSTANT = ...)
Avoid shadowing variables declared in an outer scope in Python
Initialize all externally visible members of a Python class in the constructor
For Python interfaces that may be used outside a file, prefer docstrings over comments
Python comments should be reserved for code within a function, or interfaces that are local to a file
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx
Python attributes and variables can be documented inline with type and description (e.g., self.x = 5 followed by """<type>: Description of 'x'""" )
Avoid using reflection in Python when functionality can be easily achieved without reflection
When using try-except blocks in Python, limit the except clause to the smallest set of specific errors possible instead of catching all exceptions
When using try-except blocks in Python to handle multiple possible variable types (duck-typing), keep the body of the try as small as possible and use the else block to implement the logic

Files:

• tensorrt_llm/commands/eval.py
• tensorrt_llm/inputs/utils.py
• tensorrt_llm/commands/serve.py
• tensorrt_llm/llmapi/deepseek_v32/__init__.py
• tensorrt_llm/llmapi/deepseek_v32/tokenizer.py
• tensorrt_llm/llmapi/llm_args.py
• tensorrt_llm/llmapi/deepseek_v32/encoding.py

**/*.{cpp,h,cu,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

All TensorRT-LLM Open Source Software code files should contain an NVIDIA copyright header that includes the current year at the top

Files:

• tensorrt_llm/commands/eval.py
• tensorrt_llm/inputs/utils.py
• tensorrt_llm/commands/serve.py
• tensorrt_llm/llmapi/deepseek_v32/__init__.py
• tensorrt_llm/llmapi/deepseek_v32/tokenizer.py
• tensorrt_llm/llmapi/llm_args.py
• tensorrt_llm/llmapi/deepseek_v32/encoding.py

🧠 Learnings (6)

📓 Common learnings

Learnt from: venkywonka
Repo: NVIDIA/TensorRT-LLM PR: 6029
File: .github/pull_request_template.md:45-53
Timestamp: 2025-08-27T17:50:13.264Z
Learning: For PR templates in TensorRT-LLM, avoid suggesting changes that would increase developer overhead, such as converting plain bullets to mandatory checkboxes. The team prefers guidance-style bullets that don't require explicit interaction to reduce friction in the PR creation process.

Learnt from: samuellees
Repo: NVIDIA/TensorRT-LLM PR: 6974
File: tensorrt_llm/serve/scripts/benchmark_dataset.py:558-566
Timestamp: 2025-08-18T08:42:02.640Z
Learning: In TensorRT-LLM's RandomDataset (tensorrt_llm/serve/scripts/benchmark_dataset.py), when using --random-token-ids option, sequence length accuracy is prioritized over semantic correctness for benchmarking purposes. The encode/decode operations should use skip_special_tokens=True and add_special_tokens=False to ensure exact target token lengths.

📚 Learning: 2025-08-18T08:42:02.640Z

Learnt from: samuellees
Repo: NVIDIA/TensorRT-LLM PR: 6974
File: tensorrt_llm/serve/scripts/benchmark_dataset.py:558-566
Timestamp: 2025-08-18T08:42:02.640Z
Learning: In TensorRT-LLM's RandomDataset (tensorrt_llm/serve/scripts/benchmark_dataset.py), when using --random-token-ids option, sequence length accuracy is prioritized over semantic correctness for benchmarking purposes. The encode/decode operations should use skip_special_tokens=True and add_special_tokens=False to ensure exact target token lengths.

Applied to files:

• tensorrt_llm/commands/eval.py
• tensorrt_llm/commands/serve.py
• tensorrt_llm/llmapi/llm_args.py

📚 Learning: 2025-08-01T15:14:45.673Z

Learnt from: yibinl-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.

Applied to files:

• tensorrt_llm/inputs/utils.py

📚 Learning: 2025-07-28T17:06:08.621Z

Learnt from: moraxu
Repo: NVIDIA/TensorRT-LLM PR: 6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

• tensorrt_llm/inputs/utils.py

📚 Learning: 2025-08-14T15:38:01.771Z

Learnt from: MatthiasKohl
Repo: NVIDIA/TensorRT-LLM PR: 6904
File: cpp/tensorrt_llm/pybind/thop/bindings.cpp:55-57
Timestamp: 2025-08-14T15:38:01.771Z
Learning: In TensorRT-LLM Python bindings, tensor parameter collections like mla_tensor_params and spec_decoding_tensor_params are kept as required parameters without defaults to maintain API consistency, even when it might affect backward compatibility.

Applied to files:

• tensorrt_llm/inputs/utils.py

📚 Learning: 2025-11-27T09:23:18.742Z

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 9511
File: tests/integration/defs/examples/serve/test_serve.py:136-186
Timestamp: 2025-11-27T09:23:18.742Z
Learning: In TensorRT-LLM testing, when adding test cases based on RCCA commands, the command format should be copied exactly as it appears in the RCCA case, even if it differs from existing tests. For example, some RCCA commands for trtllm-serve may omit the "serve" subcommand while others include it.

Applied to files:

• tensorrt_llm/commands/serve.py

🧬 Code graph analysis (6)

tensorrt_llm/commands/eval.py (1)

tensorrt_llm/llmapi/llm.py (2)

• tokenizer (763-767)
• tokenizer (770-771)

tensorrt_llm/inputs/utils.py (1)

tensorrt_llm/llmapi/deepseek_v32/tokenizer.py (3)

• DeepseekV32Tokenizer (16-147)
• apply_chat_template (44-55)
• encode (122-134)

tensorrt_llm/commands/serve.py (4)

tensorrt_llm/llmapi/llm.py (2)

• tokenizer (763-767)
• tokenizer (770-771)

tensorrt_llm/_torch/models/modeling_llava_next.py (1)

• tokenizer (73-74)

tensorrt_llm/_torch/models/modeling_vila.py (1)

• tokenizer (905-906)

tensorrt_llm/_torch/models/modeling_qwen2vl.py (1)

• tokenizer (124-125)

tensorrt_llm/llmapi/deepseek_v32/__init__.py (2)

tensorrt_llm/llmapi/deepseek_v32/encoding.py (2)

• encode_messages (291-309)
• parse_message_from_completion_text (383-425)

tensorrt_llm/llmapi/deepseek_v32/tokenizer.py (1)

• DeepseekV32Tokenizer (16-147)

tensorrt_llm/llmapi/deepseek_v32/tokenizer.py (3)

tensorrt_llm/llmapi/tokenizer.py (11)

• TransformersTokenizer (28-267)
• from_pretrained (75-78)
• apply_chat_template (65-69)
• eos_token_id (40-41)
• pad_token_id (44-45)
• is_fast (91-92)
• get_added_vocab (94-96)
• encode (51-52)
• convert_tokens_to_string (105-135)
• decode (54-55)
• convert_ids_to_tokens (98-103)

tensorrt_llm/llmapi/deepseek_v32/encoding.py (1)

• encode_messages (291-309)

tensorrt_llm/inputs/utils.py (1)

• apply_chat_template (565-617)

tensorrt_llm/llmapi/llm_args.py (1)

tensorrt_llm/llmapi/deepseek_v32/tokenizer.py (2)

• DeepseekV32Tokenizer (16-147)
• from_pretrained (25-42)

🪛 Ruff (0.14.8)

tensorrt_llm/llmapi/deepseek_v32/tokenizer.py

31-31: Unused class method argument: download_dir

(ARG003)

tensorrt_llm/llmapi/deepseek_v32/encoding.py

2-2: Unused noqa directive (non-enabled: E501)

Remove unused noqa directive

(RUF100)

---

35-35: Possible hardcoded password assigned to: "bos_token"

(S105)

---

35-35: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

35-35: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

36-36: Possible hardcoded password assigned to: "eos_token"

(S105)

---

36-36: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

36-36: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

37-37: Possible hardcoded password assigned to: "thinking_start_token"

(S105)

---

38-38: Possible hardcoded password assigned to: "thinking_end_token"

(S105)

---

39-39: Possible hardcoded password assigned to: "dsml_token"

(S105)

---

39-39: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

39-39: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

41-41: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

41-41: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

41-41: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

41-41: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

42-42: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

42-42: String contains ambiguous ｜ (FULLWIDTH VERTICAL LINE). Did you mean | (VERTICAL LINE)?

(RUF001)

---

57-57: Do not catch blind exception: Exception

(BLE001)

---

208-208: Parenthesize a and b expressions when chaining and and or together, to make the precedence clear

Parenthesize the and subexpression

(RUF021)

🔇 Additional comments (10)

tensorrt_llm/llmapi/deepseek_v32/encoding.py (2)

35-52: Unicode token constants and S105/RUF001 warnings

The BOS/EOS/thinking/DSML token strings intentionally use fullwidth characters and unusual delimiters to match the DeepSeek V3.2 tokenizer spec. Static-analysis warnings about ambiguous ｜ and possible “passwords” are expected here and shouldn’t be “simplified” to ASCII |, or behavior will diverge from the model’s encoding.

If these warnings are noisy in CI, consider suppressing them via tool configuration rather than altering the tokens.

---

54-59: Broad exception in to_json is acceptable copy-from-spec behavior

to_json catches a bare Exception to fall back from ensure_ascii=False to ensure_ascii=True. Given this is copied from the model’s reference implementation and the fallback is conservative, I’d keep the broad catch and, if needed, silence BLE001 via linter config instead of narrowing the exception set.

tensorrt_llm/inputs/utils.py (1)

26-29: Import placement and dependency direction look fine

Importing DeepseekV32Tokenizer from tensorrt_llm.llmapi.deepseek_v32 here matches the intended layering (inputs → llmapi) and doesn’t introduce an obvious cycle, given llmapi.tokenizer doesn’t depend on inputs.

tensorrt_llm/llmapi/deepseek_v32/__init__.py (1)

7-14: Re-export surface looks appropriate

Exporting DeepseekV32Tokenizer, encode_messages, and parse_message_from_completion_text via __all__ matches how callers use this package (e.g., imports in inputs/utils.py and llm_args.py) and keeps the public surface focused.

tensorrt_llm/llmapi/llm_args.py (1)

1712-1716: tokenizer_mode extension is consistent with CLI surfaces

Adding 'deepseek_v32' to the tokenizer_mode Literal and documenting its use for DeepSeek V3.2 matches the new CLI options in eval/serve and keeps the JSON schema in sync.

tensorrt_llm/commands/eval.py (2)

42-47: CLI --tokenizer_mode definition is aligned with BaseLlmArgs

The new option exposes exactly the supported values ("auto", "slow", "deepseek_v32") and clearly documents the DeepSeek-specific mode. This keeps the CLI surface in sync with BaseLlmArgs.tokenizer_mode.

---

117-134: Propagating tokenizer_mode into llm_args is correct

Adding "tokenizer_mode": tokenizer_mode when building llm_args ensures the selected mode reaches both PyTorch and TensorRT backends via BaseLlmArgs, matching the new tokenizer initialization logic.

tensorrt_llm/commands/serve.py (3)

81-107: Extending get_llm_args with tokenizer_mode is consistent and backwards compatible

Adding the tokenizer_mode: str = "auto" parameter and threading it into the llm_args dict lets both serve and disaggregated flows control tokenizer initialization without breaking existing callers (they either pass it explicitly or fall back to "auto").

---

253-264: Serve CLI --tokenizer_mode matches eval and llm_args

The new option exposes the same set of choices as eval ("auto", "slow", "deepseek_v32") with a clear help string. This keeps server-side configuration aligned with BaseLlmArgs.tokenizer_mode.

---

421-454: Passing tokenizer_mode through to get_llm_args correctly wires server behavior

Propagating the CLI tokenizer_mode argument into get_llm_args ensures the LLM instance used by the server honors DeepSeek V3.2 tokenization when requested, while leaving existing defaults unchanged.

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #27801 [ run ] triggered by Bot. Commit: fa3fa6e

[tensorrt-cicd]

PR_Github #27801 [ run ] completed with state SUCCESS. Commit: fa3fa6e
/LLM/main/L0_MergeRequest_PR pipeline #21217 completed with status: 'FAILURE'

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #27920 [ run ] triggered by Bot. Commit: c8aa3ee

[lfr-0531]

LGTM!

[tensorrt-cicd]

PR_Github #27920 [ run ] completed with state SUCCESS. Commit: c8aa3ee
/LLM/main/L0_MergeRequest_PR pipeline #21320 completed with status: 'FAILURE'

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #28056 [ run ] triggered by Bot. Commit: c8aa3ee

[tensorrt-cicd]

PR_Github #28056 [ run ] completed with state SUCCESS. Commit: c8aa3ee
/LLM/main/L0_MergeRequest_PR pipeline #21429 completed with status: 'FAILURE'

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #28321 [ run ] triggered by Bot. Commit: c8aa3ee

[tensorrt-cicd]

PR_Github #28321 [ run ] completed with state SUCCESS. Commit: c8aa3ee
/LLM/main/L0_MergeRequest_PR pipeline #21662 completed with status: 'FAILURE'

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #28450 [ run ] triggered by Bot. Commit: c8aa3ee

[tensorrt-cicd]

PR_Github #28450 [ run ] completed with state FAILURE. Commit: c8aa3ee
/LLM/main/L0_MergeRequest_PR pipeline #21779 completed with status: 'FAILURE'

[tensorrt-cicd]

PR_Github #28826 [ run ] triggered by Bot. Commit: c1faa60

[tensorrt-cicd]

PR_Github #28826 [ run ] completed with state FAILURE. Commit: c1faa60
/LLM/main/L0_MergeRequest_PR pipeline #22072 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #28832 [ run ] triggered by Bot. Commit: c1faa60

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #29011 [ run ] triggered by Bot. Commit: a68b52c

[tensorrt-cicd]

PR_Github #29011 [ run ] completed with state SUCCESS. Commit: a68b52c
/LLM/main/L0_MergeRequest_PR pipeline #22233 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

[chang-l]

/bot run

[tensorrt-cicd]

PR_Github #29031 [ run ] triggered by Bot. Commit: a68b52c

[tensorrt-cicd]

PR_Github #29031 [ run ] completed with state SUCCESS. Commit: a68b52c
/LLM/main/L0_MergeRequest_PR pipeline #22252 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

[lfr-0531]

/bot run

[tensorrt-cicd]

PR_Github #29060 [ run ] triggered by Bot. Commit: a68b52c

[tensorrt-cicd]

PR_Github #29060 [ run ] completed with state SUCCESS. Commit: a68b52c
/LLM/main/L0_MergeRequest_PR pipeline #22279 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.