[https://nvbugs/6276841][fix] When torch_compile=True, pass kv_cache_config=KvCacheConfig(free_gpu_memory_fra… by tensorrt-cicd · Pull Request #15095 · NVIDIA/TensorRT-LLM

tensorrt-cicd · 2026-06-08T09:33:17Z

Summary

Root cause: Default kv_cache free_gpu_memory_fraction=0.9 leaves no headroom for piecewise CUDA-graph capture workspace under torch_compile=True on the 44 GiB L40S.
Fix: When torch_compile=True, pass kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.6) (mirrors PR [https://nvbugs/6162860][fix] Set free_gpu_memory_fraction=0.6 only when torch_compile=True for test_bfloat16_ #14109 / nvbugs 6162860 for TestDeepSeekV3Lite); also remove the L40S waiver.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6276841

Summary by CodeRabbit

Tests
- Optimized GPU memory allocation for improved test stability in specific execution scenarios
- Re-enabled a previously skipped test case to enhance validation coverage

coderabbitai · 2026-06-08T09:35:35Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 69f31610-945b-48a4-b084-4a8ee27d5444

📥 Commits

Reviewing files that changed from the base of the PR and between 2febb37 and 5a514e8.

📒 Files selected for processing (2)

tests/integration/defs/accuracy/test_llm_api_pytorch.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR adjusts the Qwen 3 30B A3B Instruct test configuration to prevent out-of-memory during fp8 precision test execution when torch_compile is enabled. The KV-cache memory fraction is conditionally reduced from default to 0.6 to provide workspace for CUDA-graph capture, and the corresponding test waiver is removed since the configuration change allows the test to pass.

Changes

KV-cache memory adjustment for torch_compile

Layer / File(s)	Summary
KV-cache memory fraction reduction and waiver removal `tests/integration/defs/accuracy/test_llm_api_pytorch.py`, `tests/integration/test_lists/waives.txt`	`test_fp8` for `TestQwen3_30B_A3B_Instruct_2507` now conditionally overrides `pytorch_config["kv_cache_config"]` with `free_gpu_memory_fraction=0.6` when `torch_compile` is enabled. The corresponding waiver for this test case is removed from the skip list since the fix resolves the executor OOM issue.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#13852: Introduces the KvCacheConfig(free_gpu_memory_fraction=0.6) pattern to avoid KV-cache OOM, directly matching this PR's fix approach.
NVIDIA/TensorRT-LLM#14948: Also removes a previously waived test_fp8 case from tests/integration/test_lists/waives.txt, demonstrating a similar pattern of test waiver cleanup.
NVIDIA/TensorRT-LLM#15077: Modifies tests/integration/test_lists/waives.txt to update the test-skip configuration, operating on the same waiver list as this PR.

Suggested reviewers

Wanli-Jiang
moraxu
jieli-matrix

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: conditionally lowering KV-cache free_gpu_memory_fraction when torch_compile is enabled to resolve OOM issues.
Description check	✅ Passed	The PR description provides clear root cause analysis, the specific fix applied, references to related prior work, and documents the test verification performed.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

byshiue · 2026-06-09T01:14:25Z

/bot run

tensorrt-cicd · 2026-06-09T01:23:39Z

PR_Github #52898 [ run ] triggered by Bot. Commit: 5a514e8 Link to invocation

tensorrt-cicd · 2026-06-09T10:18:24Z

PR_Github #52898 [ run ] completed with state SUCCESS. Commit: 5a514e8
/LLM/main/L0_MergeRequest_PR pipeline #42147 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-06-11T01:34:45Z

/bot run

tensorrt-cicd · 2026-06-11T01:41:20Z

PR_Github #53419 [ run ] triggered by Bot. Commit: 5a514e8 Link to invocation

tensorrt-cicd · 2026-06-11T10:35:08Z

PR_Github #53419 [ run ] completed with state FAILURE. Commit: 5a514e8
/LLM/main/L0_MergeRequest_PR pipeline #42590 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-06-12T00:59:41Z

/bot run

tensorrt-cicd · 2026-06-12T01:05:55Z

PR_Github #53734 [ run ] triggered by Bot. Commit: 5a514e8 Link to invocation

tensorrt-cicd · 2026-06-13T01:05:48Z

PR_Github #53734 [ run ] completed with state ABORTED. Commit: 5a514e8

Link to invocation

byshiue · 2026-06-15T05:32:19Z

/bot run

tensorrt-cicd · 2026-06-15T05:37:58Z

PR_Github #54226 [ run ] triggered by Bot. Commit: 5a514e8 Link to invocation

tensorrt-cicd · 2026-06-15T14:32:53Z

PR_Github #54226 [ run ] completed with state FAILURE. Commit: 5a514e8
/LLM/main/L0_MergeRequest_PR pipeline #43303 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…fp8 with torch_compile Piecewise CUDA-graph capture under torch_compile=True needs additional GPU workspace beyond what the default kv_cache fraction (0.9) reserves on the 44 GiB L40S. As a result, _capture_piecewise_cuda_graphs OOMs in the executor_extra stage during executor creation. Mirror the established fix from NVIDIA#14109 (nvbugs/6162860, TestDeepSeekV3Lite): when torch_compile=True, set free_gpu_memory_fraction=0.6 so the executor has room for graph workspace. The non-compile path keeps the default and already passes. Also remove the now-obsolete L40S waiver for this test. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> # Conflicts: # tests/integration/defs/accuracy/test_llm_api_pytorch.py # tests/integration/test_lists/waives.txt

byshiue · 2026-06-26T01:47:54Z

/bot run

tensorrt-cicd · 2026-06-26T01:53:22Z

PR_Github #55938 [ run ] triggered by Bot. Commit: 6025ae9 Link to invocation

tensorrt-cicd · 2026-06-26T06:41:04Z

PR_Github #55938 [ run ] completed with state FAILURE. Commit: 6025ae9
/LLM/main/L0_MergeRequest_PR pipeline #44821 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-06-26T06:45:55Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-26T06:51:13Z

PR_Github #56018 [ run ] triggered by Bot. Commit: 6025ae9 Link to invocation

tensorrt-cicd · 2026-06-26T08:38:36Z

PR_Github #56018 [ run ] completed with state FAILURE. Commit: 6025ae9
/LLM/main/L0_MergeRequest_PR pipeline #44895 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-06-29T00:55:12Z

/bot run

tensorrt-cicd · 2026-06-29T01:00:54Z

PR_Github #56246 [ run ] triggered by Bot. Commit: 6025ae9 Link to invocation

tensorrt-cicd · 2026-06-29T05:37:06Z

PR_Github #56246 [ run ] completed with state FAILURE. Commit: 6025ae9
/LLM/main/L0_MergeRequest_PR pipeline #45106 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-06-30T01:06:02Z

/bot run

byshiue · 2026-07-01T01:11:59Z

/bot run

tensorrt-cicd · 2026-07-01T01:18:22Z

PR_Github #56773 [ run ] triggered by Bot. Commit: 6025ae9 Link to invocation

tensorrt-cicd · 2026-07-01T05:52:39Z

PR_Github #56773 [ run ] completed with state FAILURE. Commit: 6025ae9
/LLM/main/L0_MergeRequest_PR pipeline #45595 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-07-17T02:10:46Z

/bot run --disable-cbts

tensorrt-cicd · 2026-07-17T02:17:36Z

PR_Github #59832 [ run ] triggered by Bot. Commit: bb79ca7 Link to invocation

tensorrt-cicd · 2026-07-17T02:37:16Z

PR_Github #59832 [ run ] completed with state FAILURE. Commit: bb79ca7
/LLM/main/L0_MergeRequest_PR pipeline #48240 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

byshiue · 2026-07-20T01:34:56Z

/bot run --disable-cbts

tensorrt-cicd · 2026-07-20T02:18:00Z

PR_Github #60206 [ run ] triggered by Bot. Commit: bb79ca7 Link to invocation

tensorrt-cicd · 2026-07-20T02:57:20Z

PR_Github #60206 [ run ] completed with state FAILURE. Commit: bb79ca7
/LLM/main/L0_MergeRequest_PR pipeline #48581 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> # Conflicts: # tests/integration/test_lists/waives.txt

byshiue · 2026-07-21T01:52:05Z

/bot run --disable-cbts

tensorrt-cicd · 2026-07-21T01:58:47Z

PR_Github #60523 [ run ] triggered by Bot. Commit: bdecb96 Link to invocation

tensorrt-cicd · 2026-07-21T02:19:38Z

PR_Github #60523 [ run ] completed with state FAILURE. Commit: bdecb96
/LLM/main/L0_MergeRequest_PR pipeline #48847 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

ruodil · 2026-07-21T05:59:31Z

+        # 44 GiB L40S; lower the fraction in that case to avoid executor OOM.
+        if torch_compile:
+            pytorch_config["kv_cache_config"] = KvCacheConfig(
+                free_gpu_memory_fraction=0.6)


this may affect perf value for other models

tensorrt-cicd requested a review from a team as a code owner June 8, 2026 09:33

github-actions Bot assigned tensorrt-cicd Jun 8, 2026

jieli-matrix approved these changes Jun 9, 2026

View reviewed changes

tensorrt-cicd force-pushed the repair-bot-bug6276841 branch from 5a514e8 to 2e936b6 Compare June 17, 2026 06:36

Merge remote-tracking branch 'github/main' into codex/pr15095-conflicts

6025ae9

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> # Conflicts: # tests/integration/defs/accuracy/test_llm_api_pytorch.py # tests/integration/test_lists/waives.txt

Merge branch 'main' into repair-bot-bug6276841

bb79ca7

byshiue requested review from a team as code owners July 17, 2026 02:10

byshiue requested review from BowenFu, mzweilz, niukuo, ruodil and schetlur-nv July 17, 2026 02:10

mzweilz approved these changes Jul 17, 2026

View reviewed changes

Merge remote-tracking branch 'github/main' into codex/pr15095-conflicts

bdecb96

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> # Conflicts: # tests/integration/test_lists/waives.txt

ruodil reviewed Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

tensorrt-cicd commented Jun 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 8, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

byshiue commented Jun 9, 2026

Uh oh!

tensorrt-cicd commented Jun 9, 2026

Uh oh!

tensorrt-cicd commented Jun 9, 2026

Uh oh!

byshiue commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

byshiue commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

byshiue commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

byshiue commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

byshiue commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

byshiue commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

byshiue commented Jun 30, 2026

Uh oh!

byshiue commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

byshiue commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

byshiue commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

byshiue commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026 •

edited by coderabbitai Bot

Loading