[TRTLLM-14571][infra] Enable container-local AutoTuner cache in CI by YihuiLu512 · Pull Request #16794 · NVIDIA/TensorRT-LLM

YihuiLu512 · 2026-07-23T09:45:44Z

Wire TLLM_AUTOTUNER_CACHE_PATH into pytest stages to reuse tactic profiling across testcases via a volatile container-local /tmp path.
Harden _deserialize_cache_data to skip non-literal tactics (SyntaxError) instead of crashing load_cache, e.g. Fp4QuantTactic IntEnum reprs.
Add regression test for the skip-on-load behavior.

The goal is to improve GPU utilization during CI runs.

Dev Engineer Review

Updated jenkins/L0_Test.groovy to inject TLLM_AUTOTUNER_CACHE_PATH into pytest runs, pointing to a container-local volatile /tmp/trtllm_autotuner_cache/autotuner_cache.json, with inline rationale to avoid writing to shared/bind-mounted paths due to unreliable locking over NFS.
Hardened AutoTunerProfilingCache._deserialize_cache_data in tensorrt_llm/_torch/autotuner.py to also catch SyntaxError from ast.literal_eval when cached tactic representations are not valid Python literals; it logs once and skips the bad entry instead of failing the entire cache load (preventing reuse of stale/incorrect tactics).
Regression test added to verify non-literal tactic representations are skipped rather than causing load_cache to raise.

QA Engineer Review

Added test: test_load_cache_skips_non_literal_tactic() in tests/unittest/_torch/misc/test_autotuner.py.
Test-list coverage: no changes found under tests/integration/test_lists/, and the new test is not listed in test-db/ or qa/ test lists.
Verdict: insufficient.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

YihuiLu512 · 2026-07-23T09:46:58Z

/bot run

coderabbitai · 2026-07-23T09:49:12Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fa9e1ab8-31c7-4329-8635-2c74096658f7

📥 Commits

Reviewing files that changed from the base of the PR and between 7600b13 and ac34b8e.

📒 Files selected for processing (2)

jenkins/L0_Test.groovy
tensorrt_llm/_torch/autotuner.py

Walkthrough

The PR configures pytest runs to use a container-local AutoTuner profiling cache and updates cache deserialization to skip tactic entries that raise SyntaxError, with a regression test covering valid and invalid entries.

Changes

Autotuner cache reliability

Layer / File(s)	Summary
Container-local cache configuration `jenkins/L0_Test.groovy`	Pytest commands set `TLLM_AUTOTUNER_CACHE_PATH` to a volatile `/tmp` JSON cache path.
Invalid tactic cache handling `tensorrt_llm/_torch/autotuner.py`, `tests/unittest/_torch/misc/test_autotuner.py`	Cache loading catches `SyntaxError`, skips invalid tactic entries, and verifies that valid entries remain loaded.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise, specific, and matches the main change: enabling a container-local AutoTuner cache in CI.
Description check	✅ Passed	It explains the issue, the solution, the added test, and checklist status; only the template headings are omitted.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-23T09:53:35Z

PR_Github #61274 [ run ] triggered by Bot. Commit: 7600b13 Link to invocation

tensorrt-cicd · 2026-07-23T12:40:34Z

PR_Github #61274 [ run ] completed with state FAILURE. Commit: 7600b13
/LLM/main/L0_MergeRequest_PR pipeline #49509 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

YihuiLu512 · 2026-07-27T03:02:56Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-27T03:09:43Z

PR_Github #61786 [ run ] triggered by Bot. Commit: 7600b13 Link to invocation

tensorrt-cicd · 2026-07-27T08:11:59Z

PR_Github #61786 [ run ] completed with state SUCCESS. Commit: 7600b13
/LLM/main/L0_MergeRequest_PR pipeline #49985 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

YihuiLu512 · 2026-07-28T03:11:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-28T03:18:23Z

PR_Github #62074 [ run ] triggered by Bot. Commit: 7600b13 Link to invocation

tensorrt-cicd · 2026-07-28T06:48:17Z

PR_Github #62074 [ run ] completed with state SUCCESS. Commit: 7600b13
/LLM/main/L0_MergeRequest_PR pipeline #50259 completed with status: 'SUCCESS'

CI Report

Link to invocation

- Wire TLLM_AUTOTUNER_CACHE_PATH into pytest stages to reuse tactic profiling across testcases via a volatile container-local /tmp path. - Harden _deserialize_cache_data to skip non-literal tactics (SyntaxError) instead of crashing load_cache, e.g. Fp4QuantTactic IntEnum reprs. - Add regression test for the skip-on-load behavior. Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>

- Drop the pinned Fp4QuantTactic example from the non-literal-tactic skip comment; point at NVIDIA#16782 for the serialize-side fix. - Decouple the regression test from the IntEnum: build the poisoned repr from a generic object and trim its docstring. Signed-off-by: Yihui Lu <269394165+YihuiLu512@users.noreply.github.com>

YihuiLu512 · 2026-07-28T11:20:30Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-28T11:26:30Z

PR_Github #62186 [ run ] triggered by Bot. Commit: ac34b8e Link to invocation

tensorrt-cicd · 2026-07-28T18:39:26Z

PR_Github #62186 [ run ] completed with state FAILURE. Commit: ac34b8e
/LLM/main/L0_MergeRequest_PR pipeline #50356 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

YihuiLu512 · 2026-07-29T02:44:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T02:50:08Z

PR_Github #62384 [ run ] triggered by Bot. Commit: ac34b8e Link to invocation

JunyiXu-nv

Overall LGTM, the added catch sweep is non-blocking.

tensorrt-cicd · 2026-07-29T04:29:25Z

PR_Github #62384 [ run ] completed with state FAILURE. Commit: ac34b8e
/LLM/main/L0_MergeRequest_PR pipeline #50548 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

YihuiLu512 · 2026-07-29T06:03:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T06:09:29Z

PR_Github #62442 [ run ] triggered by Bot. Commit: ac34b8e Link to invocation

tensorrt-cicd · 2026-07-29T09:01:58Z

PR_Github #62442 [ run ] completed with state SUCCESS. Commit: ac34b8e
/LLM/main/L0_MergeRequest_PR pipeline #50597 completed with status: 'SUCCESS'

CI Report

Link to invocation

YihuiLu512 requested review from a team as code owners July 23, 2026 09:45

YihuiLu512 requested review from JunyiXu-nv, asfiyab-nvidia, dpitman-nvda and yiqingy0 July 23, 2026 09:45

github-actions Bot assigned YihuiLu512 Jul 23, 2026

coderabbitai Bot mentioned this pull request Jul 23, 2026

[TRTLLM-14541][fix] VisualGen: deterministic autotuner tactics across runs and ranks #16782

Merged

1 task

yiqingy0 approved these changes Jul 28, 2026

View reviewed changes

JunyiXu-nv reviewed Jul 28, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/autotuner.py Outdated

Comment thread tests/unittest/_torch/misc/test_autotuner.py Outdated

YihuiLu512 added 2 commits July 28, 2026 19:11

YihuiLu512 force-pushed the ci-perf-02_autotuner branch from 7600b13 to ac34b8e Compare July 28, 2026 11:17

YihuiLu512 requested a review from JunyiXu-nv July 28, 2026 12:38

JunyiXu-nv approved these changes Jul 29, 2026

View reviewed changes

YihuiLu512 merged commit e240d4a into NVIDIA:main Jul 29, 2026
10 checks passed

Uh oh!

Conversation

YihuiLu512 commented Jul 23, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

YihuiLu512 commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

YihuiLu512 commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

YihuiLu512 commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

YihuiLu512 commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

YihuiLu512 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

JunyiXu-nv left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

YihuiLu512 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

YihuiLu512 commented Jul 23, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading