[TRTLLM-12338][feat] Lift TOKENIZER_ALIASES to module level in llmapi.llm_args

[nv-yna]

Summary

Hoist TOKENIZER_ALIASES from a method-local variable inside BaseLlmArgs.validate_and_init_tokenizer to a module-level constant in tensorrt_llm/llmapi/llm_args.py. Mirrors the equivalent change made on main in #12990 ("Support custom_tokenizer in KvCacheAwareRouter for disagg serving"); difference here is that this branch's dict also includes 'deepseek_v4' → DeepseekV4Tokenizer, since DeepSeek-V4 lives on this branch and isn't on main yet.

Why

External code outside TRT-LLM legitimately needs to resolve the same short aliases TRT-LLM uses internally. Concretely, the Dynamo TRT-LLM worker (dynamo.trtllm.workers.llm_worker, added in ai-dynamo/dynamo#8079) does:

```python
from tensorrt_llm.llmapi.llm_args import TOKENIZER_ALIASES
```

so it can build the V4 tokenizer before instantiating `LLM(...)`. Dynamo needs the tokenizer object early to:

• populate `SamplingParams` defaults from model-specific special tokens, and
• wire up its health-check logits processor (it then passes `skip_tokenizer_init=True` to the engine).

On this branch, `TOKENIZER_ALIASES` is defined as a method-local variable, so the import fails at module load time with:

```
ImportError: cannot import name 'TOKENIZER_ALIASES' from 'tensorrt_llm.llmapi.llm_args'
```

…and the Dynamo worker crashes before any runtime work begins (CrashLoopBackOff in K8s deployments).

main already has the dict module-level (#12990, 2026-04-19), so once this branch rebases onto main, the import will work for the V32/GLM aliases. This PR adds the V4 entry now (so DeepSeek-V4 deployments via Dynamo work today on this branch) and trims one local copy of the dict — making the eventual rebase a no-op merge for this surface.

Behavior

No semantic change for existing callers. `validate_and_init_tokenizer` now references the module-level constant instead of redefining its own copy.

Test plan

• Smoke: `python3 -c 'from tensorrt_llm.llmapi.llm_args import TOKENIZER_ALIASES; print(sorted(TOKENIZER_ALIASES))'` prints `['deepseek_v32', 'deepseek_v4', 'glm_moe_dsa']`
• Smoke: `trtllm-serve --custom_tokenizer deepseek_v4 ...` continues to load the V4 tokenizer (validates the local function body still works)
• Verified locally on a Dynamo+TRT-LLM K8s deployment of DeepSeek-V4-Flash on B200×4: `from ... import TOKENIZER_ALIASES` succeeds, worker boots, and `/v1/chat/completions` returns coherent answers.

[Tabrizian]

@lfr-0531 Just wondering would rebase take care of this issue or we need to merge this PR to feat/deepseek_v4?

[nv-yna]

Pre-commit Check is failing on a pre-existing branch issue, not on this PR's diff.

The 5 errors are Bandit B105 ("possible hardcoded password") false positives flagging tokenizer special tokens in tensorrt_llm/tokenizer/deepseek_v4/tokenizer.py:23-27:

BOS_TOKEN = "<｜begin▁of▁sentence｜>"
EOS_TOKEN = "<｜end▁of▁sentence｜>"
USER_TOKEN = "<｜User｜>"
ASSISTANT_TOKEN = "<｜Assistant｜>"
THINKING_END_TOKEN = "</think>"

These lines are already on feat/deepseek_v4 HEAD; this PR doesn't touch that file (git diff is single-file, only tensorrt_llm/llmapi/llm_args.py). Other PRs on this branch are hitting the same wall — e.g. #13563 ("[None][feat] Required changes for DeepSeek‑V4 Disaggregated") also has Pre-commit Check = FAILURE.

Suggested branch fix (separate PR, out of scope here): add # nosec B105 to each of those five lines, or add B105 to the bandit allowlist for that file. Happy to file that as a follow-up if it would help unblock the branch.

cc @lfr-0531 @Tabrizian

[Tabrizian]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #45979 [ run ] triggered by Bot. Commit: bf89349 Link to invocation

[tensorrt-cicd]

PR_Github #45979 [ run ] completed with state FAILURE. Commit: bf89349
/LLM/main/L0_MergeRequest_PR pipeline #36131 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation