[https://nvbugs/6507080][fix] Override TokenizerBase.__repr__ to return f"{self.__class__.__name__}()"…

[trtllm-agent]

Summary

• Root cause: TokenizerBase inherits transformers __repr__, which reads the abstract added_tokens_decoder property and raises NotImplementedError on user protocol subclasses
• Fix: Override TokenizerBase.__repr__ to return f"{self.__class__.__name__}()" so pydantic/logging never touches transformers-internal state on custom tokenizers
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6507080

Dev Engineer Review

• Updated TokenizerBase in tensorrt_llm/tokenizer/tokenizer.py to override __repr__, returning f"{self.__class__.__name__}()".
• This avoids PreTrainedTokenizerBase.__repr__ accessing tokenizer internals (e.g., added_tokens_decoder) that custom TokenizerBase subclasses may not implement, preventing NotImplementedError when pydantic/logging triggers repr().

QA Engineer Review

• Test-list changes: tests/integration/test_lists/waives.txt
  • Removed the waivers entry for unittest/llmapi/test_llm.py::test_llm_with_customized_tokenizer (no entries added).
  • Verdict: needs follow-up

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7bfc20ab-bd31-43b4-8ebe-41affd4e72b1

📥 Commits

Reviewing files that changed from the base of the PR and between 30b981f and ced2a75.

📒 Files selected for processing (2)

• tensorrt_llm/tokenizer/tokenizer.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

• tensorrt_llm/tokenizer/tokenizer.py

---

Walkthrough

Adds a fallback TokenizerBase.__repr__ returning ClassName() and removes the waiver for the customized tokenizer integration test.

Changes

Tokenizer representation and test enablement

Layer / File(s)	Summary
TokenizerBase representation fallback and waiver removal tensorrt_llm/tokenizer/tokenizer.py, tests/integration/test_lists/waives.txt	Adds a class-name-only __repr__ implementation and removes the customized tokenizer test from the waiver list.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16978: Updates other integration-test waiver entries in the same file.
• NVIDIA/TensorRT-LLM#16982: Updates related QA test waiver entries in the same file.

Suggested reviewers: qijune, schetlur-nv, crazydemo, yiqingy0, dpitman-nvda

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required ticket/type format and clearly summarizes the fix to TokenizerBase.repr.
Description check	✅ Passed	The description covers the root cause, fix, test plan, and bug link, so it is mostly complete despite not matching the template headings exactly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[fredricz-20070104]

Review summary - Approve

Reviewed the full diff; no blocking or major issues found.

Minor, non-blocking notes:

• tensorrt_llm/tokenizer/tokenizer.py: repr loses detail for fully-implemented subclasses

Automated review by NVCortex Lite, run by @fredricz-20070104.

[mikeiovine]

/bot run

[tensorrt-cicd]

PR_Github #63052 [ run ] triggered by Bot. Commit: ced2a75 Link to invocation

[tensorrt-cicd]

PR_Github #63052 [ run ] completed with state FAILURE. Commit: ced2a75
/LLM/main/L0_MergeRequest_PR pipeline #51153 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation