[None][feat] AutoDeploy: Fix hardcoded configs

[taylor-yb-lee]

Summary by CodeRabbit

• Tests

  • Added regression tests for auto-deploy engine configuration handling, validating proper propagation of scheduling and batch settings when provided and fallback to defaults when omitted.

• Chores

  • Enhanced configuration flexibility for the auto-deploy engine to conditionally apply settings from provided configuration or use standard defaults.

Description

• Fixed some hardcoded configs to be set and propagated properly

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

Walkthrough

ADEngine.init now conditionally initializes scheduling and batch-wait settings from ad_config when provided, otherwise applies previous default values. Two regression tests validate both code paths: one with explicit ad_config settings, another with defaults when ad_config is omitted.

Changes

ADEngine scheduling configuration

Layer / File(s)	Summary
ADEngine conditional configuration tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py	ADEngine.__init__ moves timing/DP field initialization (stream_interval, attention_dp_config, batch_wait_* settings) into conditional branches: populated from ad_config when available, otherwise stub defaults are applied.
Regression tests and test helper tests/unittest/auto_deploy/singlegpu/shim/test_engine.py	Module imports LlmArgs and AttentionDpConfig. _make_cache_seq_interface helper centralizes CachedSequenceInterface construction. Two CUDA-gated tests: one validates ADEngine propagates ad_config scheduling settings into llm_args; another validates stub defaults when ad_config is omitted.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The description provides a basic explanation of the fix but lacks detail on the specific issue, the solution approach, and test coverage. The Test Coverage section is empty, and the description does not clearly explain what problem the hardcoded configs caused or how the fix addresses it.	Expand the Description section to explain the root cause of the hardcoded configs issue and how the conditional logic fixes it. Provide explicit details in the Test Coverage section listing the specific tests added (test_ad_engine_propagates_pyexecutor_scheduling_config and test_ad_engine_scheduling_config_defaults_without_ad_config).

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title 'AutoDeploy: Fix hardcoded configs' directly matches the main objective of the PR, which is to fix hardcoded configuration values in ADEngine so they are properly set from ad_config instead of being forced to stub defaults.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (2)

tests/unittest/auto_deploy/singlegpu/shim/test_engine.py (2)

112-123: ⚡ Quick win

Add return type annotations to newly added helper/tests.

Please add explicit return types for _make_cache_seq_interface, test_ad_engine_propagates_pyexecutor_scheduling_config, and test_ad_engine_scheduling_config_defaults_without_ad_config.

Suggested patch

-def _make_cache_seq_interface(device, max_seq_len=64, max_batch_size=8):
+def _make_cache_seq_interface(
+    device: torch.device, max_seq_len: int = 64, max_batch_size: int = 8
+) -> CachedSequenceInterface:
@@
-def test_ad_engine_propagates_pyexecutor_scheduling_config():
+def test_ad_engine_propagates_pyexecutor_scheduling_config() -> None:
@@
-def test_ad_engine_scheduling_config_defaults_without_ad_config():
+def test_ad_engine_scheduling_config_defaults_without_ad_config() -> None:

As per coding guidelines, "Static type checking with mypy is opt-in by submodule; always annotate functions with return types (use None if function does not return)".

Also applies to: 167-168

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/auto_deploy/singlegpu/shim/test_engine.py` around lines 112 -
123, Add explicit return type annotations to the newly added helper and test
functions: annotate _make_cache_seq_interface to return CachedSequenceInterface,
and annotate the tests test_ad_engine_propagates_pyexecutor_scheduling_config
and test_ad_engine_scheduling_config_defaults_without_ad_config to return None;
update any other nearby functions on lines referenced (e.g., the functions
around 167-168) to include explicit return types as well so the module is fully
annotated for mypy opt-in checks.

---

122-185: Test coverage for this regression scope is sufficient.

tests/unittest/auto_deploy/singlegpu/shim/test_engine.py now covers both required paths: propagation from ad_config and fallback defaults when ad_config is omitted.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/auto_deploy/singlegpu/shim/test_engine.py` around lines 122 -
185, Tests added (test_ad_engine_propagates_pyexecutor_scheduling_config and
test_ad_engine_scheduling_config_defaults_without_ad_config) verify ADEngine
propagates scheduling/streaming knobs from an LlmArgs ad_config and falls back
to previous stub defaults when omitted; if these tests fail, update
ADEngine.__init__ (the ADEngine constructor) to copy stream_interval,
attention_dp_config, batch_wait_timeout_ms, batch_wait_timeout_iters, and
batch_wait_max_tokens_ratio from the provided ad_config LlmArgs into
engine.llm_args (or preserve existing defaults when ad_config is None), ensuring
AttentionDpConfig fields (enable_balance, batching_wait_iters, timeout_iters)
are preserved; rerun the unit tests with get_inference_model and the
cache_seq_interface fixtures to confirm passing.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/auto_deploy/singlegpu/shim/test_engine.py`:
- Around line 112-123: Add explicit return type annotations to the newly added
helper and test functions: annotate _make_cache_seq_interface to return
CachedSequenceInterface, and annotate the tests
test_ad_engine_propagates_pyexecutor_scheduling_config and
test_ad_engine_scheduling_config_defaults_without_ad_config to return None;
update any other nearby functions on lines referenced (e.g., the functions
around 167-168) to include explicit return types as well so the module is fully
annotated for mypy opt-in checks.
- Around line 122-185: Tests added
(test_ad_engine_propagates_pyexecutor_scheduling_config and
test_ad_engine_scheduling_config_defaults_without_ad_config) verify ADEngine
propagates scheduling/streaming knobs from an LlmArgs ad_config and falls back
to previous stub defaults when omitted; if these tests fail, update
ADEngine.__init__ (the ADEngine constructor) to copy stream_interval,
attention_dp_config, batch_wait_timeout_ms, batch_wait_timeout_iters, and
batch_wait_max_tokens_ratio from the provided ad_config LlmArgs into
engine.llm_args (or preserve existing defaults when ad_config is None), ensuring
AttentionDpConfig fields (enable_balance, batching_wait_iters, timeout_iters)
are preserved; rerun the unit tests with get_inference_model and the
cache_seq_interface fixtures to confirm passing.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4ea3647f-f265-4c83-bb5a-ea570f151fb4

📥 Commits

Reviewing files that changed from the base of the PR and between 4437cb9 and 4bbea1b.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py
• tests/unittest/auto_deploy/singlegpu/shim/test_engine.py

[galagam]

LGTM. Thanks!

_{Approved in CodeRabbit Change Stack}

[galagam]

/bot run

[tensorrt-cicd]

PR_Github #52029 [ run ] triggered by Bot. Commit: 4bbea1b Link to invocation

[tensorrt-cicd]

PR_Github #52029 [ run ] completed with state SUCCESS. Commit: 4bbea1b
/LLM/main/L0_MergeRequest_PR pipeline #41367 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[taylor-yb-lee]

/bot run

[tensorrt-cicd]

PR_Github #52155 [ run ] triggered by Bot. Commit: 4bbea1b Link to invocation

[tensorrt-cicd]

PR_Github #52155 [ run ] completed with state SUCCESS. Commit: 4bbea1b
/LLM/main/L0_MergeRequest_PR pipeline #41477 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[taylor-yb-lee]

/bot run

[tensorrt-cicd]

PR_Github #52422 [ run ] triggered by Bot. Commit: 4bbea1b Link to invocation

[tensorrt-cicd]

PR_Github #52422 [ run ] completed with state FAILURE. Commit: 4bbea1b
/LLM/main/L0_MergeRequest_PR pipeline #41714 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[taylor-yb-lee]

/bot run

[tensorrt-cicd]

PR_Github #52526 [ run ] triggered by Bot. Commit: 4bbea1b Link to invocation

[tensorrt-cicd]

PR_Github #52526 [ run ] completed with state SUCCESS. Commit: 4bbea1b
/LLM/main/L0_MergeRequest_PR pipeline #41813 completed with status: 'SUCCESS'

CI Report

Link to invocation