[https://nvbugs/5823783][test] add qa test case for trust-remote-code on multinode failure

[crazydemo]

has checked this case can capture the multinode failure:
on bad commit: [FAIL] https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/DEBUG_LLM_FUNCTION_TEST/2327/
on good commit: [PASS] https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/DEBUG_LLM_FUNCTION_TEST/2328

Summary by CodeRabbit

• Tests
  • Added tests for tokenizer serialization and deserialization in distributed scenarios with remote code loading support.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[crazydemo]

/bot run

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR introduces a new unit test for validating pickle serialization of tokenizers with trust_remote_code=True in multi-node distributed environments, and registers this test in the integration test list.

Changes

Cohort / File(s)	Summary
Integration Test Registry tests/integration/test_lists/qa/llm_function_core.txt	Added test reference entry for the new multinode tokenizer pickle roundtrip test in two locations within the test list.
Unit Test Addition tests/unittest/llmapi/test_tokenizer_multinode.py	New test method test_trust_remote_code_tokenizer_pickle_roundtrip_multinode that validates pickle serialization of TransformersTokenizer instances loaded with trust_remote_code=True, simulating rank-0 dynamic module loading and worker node unpickling scenarios.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	PR description lacks required sections: title format missing, Description section empty, Test Coverage section empty, and PR Checklist items unchecked except final checkbox.	Add proper PR title following format [ticket][type] summary, fill Description section explaining what and why, specify relevant tests in Test Coverage section, and address checklist items appropriately.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly specifies the NVBugs ticket, the type [test], and describes the main change: adding a QA test case for trust-remote-code on multinode failure.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

tests/unittest/llmapi/test_tokenizer_multinode.py (1)

10-10: Use module import + namespace access for tokenizer helpers.

Line 10 imports class/function symbols directly from a module, which conflicts with the repo’s Python import policy. Please import the module and access members via namespace on Line 51 and Line 60.

♻️ Proposed fix

-from tensorrt_llm.tokenizer.tokenizer import TransformersTokenizer, load_hf_tokenizer
+from tensorrt_llm.tokenizer import tokenizer
@@
-            tok = load_hf_tokenizer("Kimi-K2-Instruct", trust_remote_code=True)
+            tok = tokenizer.load_hf_tokenizer("Kimi-K2-Instruct", trust_remote_code=True)
@@
-    assert isinstance(restored, TransformersTokenizer)
+    assert isinstance(restored, tokenizer.TransformersTokenizer)

As per coding guidelines: "Python imports must use form from package.subpackage import module (never from module import Class)."

Also applies to: 51-51, 60-60

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/unittest/llmapi/test_tokenizer_multinode.py` at line 10, The test
imports symbols directly—TransformersTokenizer and load_hf_tokenizer—from
tensorrt_llm.tokenizer.tokenizer which violates the import policy; change the
import to the module form (import the tokenizer module, e.g., from
tensorrt_llm.tokenizer import tokenizer) and update usages at the call sites
that reference TransformersTokenizer and load_hf_tokenizer (the occurrences
around the previously noted lines) to use namespace access
(tokenizer.TransformersTokenizer and tokenizer.load_hf_tokenizer).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/unittest/llmapi/test_tokenizer_multinode.py`:
- Around line 44-56: The test mutates sys.modules entries
("transformers_modules" and "transformers_modules.kimi_k2.tokenization_kimi")
without preserving prior values, risking cross-test contamination; modify the
test around load_hf_tokenizer/KimiK2Tokenizer to save the original values (e.g.,
orig_tm = sys.modules.get("transformers_modules") and orig_sub =
sys.modules.get("transformers_modules.kimi_k2.tokenization_kimi")), set the fake
modules, run the mocked AutoTokenizer.from_pretrained and pickle roundtrip as
before, then in the finally block restore the originals (put back
orig_tm/orig_sub if not None, otherwise pop the keys) so existing shared state
is preserved.

---

Nitpick comments:
In `@tests/unittest/llmapi/test_tokenizer_multinode.py`:
- Line 10: The test imports symbols directly—TransformersTokenizer and
load_hf_tokenizer—from tensorrt_llm.tokenizer.tokenizer which violates the
import policy; change the import to the module form (import the tokenizer
module, e.g., from tensorrt_llm.tokenizer import tokenizer) and update usages at
the call sites that reference TransformersTokenizer and load_hf_tokenizer (the
occurrences around the previously noted lines) to use namespace access
(tokenizer.TransformersTokenizer and tokenizer.load_hf_tokenizer).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: deb500d0-8d35-4c3e-8cb1-f0fb5c99fd14

📥 Commits

Reviewing files that changed from the base of the PR and between 2dbd154 and 50379e1.

📒 Files selected for processing (2)

• tests/integration/test_lists/qa/llm_function_core.txt
• tests/unittest/llmapi/test_tokenizer_multinode.py

[tensorrt-cicd]

PR_Github #37670 [ run ] triggered by Bot. Commit: 50379e1 Link to invocation

[tensorrt-cicd]

PR_Github #37670 [ run ] completed with state SUCCESS. Commit: 50379e1
/LLM/main/L0_MergeRequest_PR pipeline #29155 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[crazydemo]

/bot run

[tensorrt-cicd]

PR_Github #37784 [ run ] triggered by Bot. Commit: ff5c788 Link to invocation

[JunyiXu-nv]

The test logic looks good, but the bug should be tested on multinode senario. Is this one running on multi-node?

[crazydemo]

The test logic looks good, but the bug should be tested on multinode senario. Is this one running on multi-node?

This case even does not requires gpu, as I use a clean subprocess to simulate the multinode env. And has checked this case will fail on v1.2.0rc8, with error message:

nittest/llmapi/test_tokenizer_multinode.py:59: in test_trust_remote_code_tokenizer_pickle_roundtrip_multinode
15:12:44      restored = pickle.loads(data)  # nosec B301
15:12:44                 ^^^^^^^^^^^^^^^^^^
15:12:44  E   ModuleNotFoundError: No module named 'transformers_modules'

And on main branch, which includes your fix PR, can pass.

So I think this is enough for check this bug, without running on a real multinode env.

[tensorrt-cicd]

PR_Github #37784 [ run ] completed with state SUCCESS. Commit: ff5c788
/LLM/main/L0_MergeRequest_PR pipeline #29248 completed with status: 'FAILURE'

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[crazydemo]

/bot run

[tensorrt-cicd]

PR_Github #37969 [ run ] triggered by Bot. Commit: afcc26f Link to invocation

[tensorrt-cicd]

PR_Github #37969 [ run ] completed with state SUCCESS. Commit: afcc26f
/LLM/main/L0_MergeRequest_PR pipeline #29405 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

Link to invocation

[crazydemo]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #38181 [ reuse-pipeline ] triggered by Bot. Commit: 29f69a3 Link to invocation

[tensorrt-cicd]

PR_Github #38181 [ reuse-pipeline ] completed with state SUCCESS. Commit: 29f69a3
Reusing PR_Github #37969 for commit 29f69a3

Link to invocation