[#14173][tests] move autodeploy accuracy tests to post merge and use model registry

[MrGeva]

Summary

This PR reduces pre-merge AutoDeploy GPU time by moving variant selection to CI test lists, while standardizing relevant AutoDeploy accuracy tests to use model registry configs with explicit overrides only where needed.

What changed

1. pre-merge cost reduction

• Keep one primary performant variant per model in pre_merge
• Move additional variants to post_merge

2. Registry adoption in test_llm_api_autodeploy.py

Updated selected tests to load config via model registry (yaml_extra) and then override as needed:

• TestLlama3_1_8B_Instruct_Eagle3
• TestNemotronH
• TestGLM4Flash
• TestQwen3NextInstruct
• TestMiniMaxM2

3. Preserved original test intent where registry defaults could change behavior

Added explicit pins to avoid unintentional behavior drift:

• TestNemotronH: world_size=1 (prevent registry world_size_2 from changing scope)
• TestGLM4Flash: explicit enable_chunked_prefill=False in the non-chunked path
• TestMiniMaxM2: explicit attn_backend="trtllm" to preserve original backend behavior

4. Removed redundant overrides now covered by registry

Dropped duplicated settings where registry already provides equivalent values, while keeping non-redundant behavioral overrides.

Removed as redundant (same as registry/default):

• skip_loading_weights=False in multiple registry-backed tests
• max_batch_size=128 in Eagle3 (already from dashboard default)
• duplicated GLM4Flash transform flags already present in glm-4.7-flash.yaml
• duplicated model_kwargs.torch_dtype=bfloat16 in MiniMax M2 (already in minimax_m2.yaml)

Kept intentional non-redundant overrides:

• TestNemotronH: world_size=1 (to preserve original test scope)
• TestGLM4Flash: explicit enable_chunked_prefill=False for the non-chunked variant
• TestMiniMaxM2: attn_backend="trtllm" (to preserve original backend behavior)
• model/test-specific overrides like compile_model.piecewise_enabled, sharding knobs, etc.

Summary by CodeRabbit

• Tests
  • Updated AutoDeploy test harnesses to utilize registry-derived configurations.
  • Reorganized test selections across multiple hardware configurations for improved coverage optimization.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

AutoDeploy accuracy tests are refactored to pull model-specific yaml_extra configurations from the AutoDeploy registry instead of hardcoding settings. GLM4Flash chunked prefill behavior is made explicit, and Nemotron-H world size is set explicitly. Test selections across multiple hardware configurations (Blackwell, H100, 4-GPU setups) are curated and reorganized to run specific backend and dtype combinations.

Changes

AutoDeploy Test Registry Integration and Curation

Layer / File(s)	Summary
Registry YAML integration for test defaults tests/integration/defs/accuracy/test_llm_api_autodeploy.py	TestLlama3_1_8B_Instruct_Eagle3, TestNemotronH, TestGLM4Flash, TestQwen3NextInstruct, and TestMiniMaxM2 now fetch yaml_extra from the AutoDeploy model registry and include it in their default kwargs instead of hardcoding batch/weight-loading settings.
Chunked prefill and world size refinements tests/integration/defs/accuracy/test_llm_api_autodeploy.py	TestGLM4Flash.get_default_kwargs explicitly forces enable_chunked_prefill=False when disabled and sets transforms.compile_model.piecewise_enabled=True when enabled, replacing unconditional transform merging. TestNemotronH.test_mtp explicitly sets world_size=1 in AutoDeploy invocation.
Test selection curation across hardware configurations tests/integration/test_lists/test-db/l0_b200.yml, l0_dgx_b200.yml, l0_dgx_h100.yml, l0_h100.yml	Pre-merge and post-merge AutoDeploy test selections are reorganized across Blackwell, 4-GPU Blackwell, 4-GPU H100, and single-GPU H100 configurations to focus runs on specific backends (trtllm, flashinfer) and quantization variants (fp8, nvfp4, bf16) for Eagle3 speculative decoding and Nemotron model accuracy/functional tests.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#13658: Both PRs update AutoDeploy test harness registry yaml_extra integration in test_llm_api_autodeploy.py and test-list configurations for l0_dgx_b200.yml, directly affecting how newly added TestNemotronUltraV3 test cases interact with the registry.
• NVIDIA/TensorRT-LLM#14243: Intersects on TestNemotronNanoV3::test_accuracy[bf16] parametrization; this PR adds it to certain AutoDeploy stages while the related PR skips bf16 on <80GB GPUs.
• NVIDIA/TensorRT-LLM#13630: This PR's registry yaml_extra integration directly supports Gemma E2B model-registry configs and test wiring added in the related PR.

Suggested reviewers

• galagam
• greg-kwasniewski1
• lucaslie

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title '[#14173][tests] move autodeploy accuracy tests to post merge and use model registry' clearly and concisely describes the main changes: moving test variants and adopting model registry configuration.
Description check	✅ Passed	The PR description provides a clear summary of changes, explains the rationale for reducing pre-merge GPU time, and documents specific code modifications with intentional behavioral overrides.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/integration/test_lists/test-db/l0_h100.yml (1)

507-507: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Reconsider pre-merge vs. post-merge assignment for GLM4Flash enable_chunked_prefill parameter.

The code's get_default_kwargs method includes an explicit comment: "Keep the original non-chunked variant behavior even when registry defaults enable chunked prefill." This indicates that enable_chunked_prefill=False (non-chunked) is the original/baseline behavior. However, the test-db currently assigns:

• Pre-merge: test_auto_dtype[trtllm-True] (chunked enabled)
• Post-merge: test_auto_dtype[trtllm-False] (non-chunked)

Since False is the original behavior that the code explicitly emphasizes preserving, it should likely be the primary pre-merge variant, not the secondary post-merge variant. Verify whether the current assignment aligns with the intended primary vs. secondary test strategy.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/test-db/l0_h100.yml` at line 507, The test-db
currently assigns the GLM4Flash enable_chunked_prefill variants such that the
chunked (True) case is pre-merge and the original non-chunked (False) case is
post-merge; update the assignment so the baseline behavior
(enable_chunked_prefill=False) is the primary pre-merge variant and
enable_chunked_prefill=True becomes the secondary post-merge variant; locate the
mapping that produces the test names test_auto_dtype[trtllm-True] and
test_auto_dtype[trtllm-False] and swap their pre-merge/post-merge roles (also
ensure this aligns with get_default_kwargs which documents preserving the
non-chunked default).

🧹 Nitpick comments (1)

tests/integration/test_lists/test-db/l0_h100.yml (1)

505-542: Verify QA test list coverage for reorganized AutoDeploy tests.

This PR reorganizes AutoDeploy accuracy tests between pre-merge and post-merge stages. Per coding guidelines, when integration tests under tests/integration/defs/ are materially altered, entries may be needed in tests/integration/test_lists/qa/.

Please confirm:

1. Are the AutoDeploy accuracy tests already covered in tests/integration/test_lists/qa/llm_function_core.txt or similar QA lists?
2. Do the QA lists need updates to ensure the primary variants (now in pre-merge) are captured for scheduled regression runs?
3. Should any of the post-merge-only variants be added to QA coverage?

As per coding guidelines, integration test changes should be evaluated for QA list inclusion to ensure scheduled regression coverage.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/test-db/l0_h100.yml` around lines 505 - 542, The
AutoDeploy accuracy tests were moved between pre-merge and post-merge in the
test list and you need to ensure QA regression lists include the primary
variants; search current QA lists (e.g., llm_function_core.txt) for the
AutoDeploy test IDs moved (examples: TestLlama3_1_8B::test_auto_dtype,
TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model,
TestNemotronV2::test_auto_dtype, TestGLM4Flash::test_auto_dtype,
TestNemotronNanoV3::test_accuracy) to confirm coverage, then add any missing
primary pre-merge variants to the appropriate QA list(s) so scheduled runs catch
them and consider adding high-risk post-merge-only variants if they lack QA
coverage; finally commit the QA list updates and run the integration list
grep/validation used in CI to verify the new entries are picked up.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tests/integration/test_lists/test-db/l0_h100.yml`:
- Line 507: The test-db currently assigns the GLM4Flash enable_chunked_prefill
variants such that the chunked (True) case is pre-merge and the original
non-chunked (False) case is post-merge; update the assignment so the baseline
behavior (enable_chunked_prefill=False) is the primary pre-merge variant and
enable_chunked_prefill=True becomes the secondary post-merge variant; locate the
mapping that produces the test names test_auto_dtype[trtllm-True] and
test_auto_dtype[trtllm-False] and swap their pre-merge/post-merge roles (also
ensure this aligns with get_default_kwargs which documents preserving the
non-chunked default).

---

Nitpick comments:
In `@tests/integration/test_lists/test-db/l0_h100.yml`:
- Around line 505-542: The AutoDeploy accuracy tests were moved between
pre-merge and post-merge in the test list and you need to ensure QA regression
lists include the primary variants; search current QA lists (e.g.,
llm_function_core.txt) for the AutoDeploy test IDs moved (examples:
TestLlama3_1_8B::test_auto_dtype,
TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model,
TestNemotronV2::test_auto_dtype, TestGLM4Flash::test_auto_dtype,
TestNemotronNanoV3::test_accuracy) to confirm coverage, then add any missing
primary pre-merge variants to the appropriate QA list(s) so scheduled runs catch
them and consider adding high-risk post-merge-only variants if they lack QA
coverage; finally commit the QA list updates and run the integration list
grep/validation used in CI to verify the new entries are picked up.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c75a516c-db73-4eec-b7fa-f1414a1ba0c6

📥 Commits

Reviewing files that changed from the base of the PR and between c71655e and 8ed8683.

📒 Files selected for processing (5)

• tests/integration/defs/accuracy/test_llm_api_autodeploy.py
• tests/integration/test_lists/test-db/l0_b200.yml
• tests/integration/test_lists/test-db/l0_dgx_b200.yml
• tests/integration/test_lists/test-db/l0_dgx_h100.yml
• tests/integration/test_lists/test-db/l0_h100.yml

[MrGeva]

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast

[tensorrt-cicd]

PR_Github #49409 [ run ] triggered by Bot. Commit: 7f377ea Link to invocation

[nvchenghaoz]

I think it is good to move some tests as it can reduce the overall pre_merge time. Possibly reduce the churn in the pre_merge..

One concern, should we force the AutoDeploy dev to run all AD related post-merge tests to make sure that all accuracy tests are covered in the PR? As the failures in the post-merge will be waived..

[tensorrt-cicd]

PR_Github #49409 [ run ] completed with state SUCCESS. Commit: 7f377ea
/LLM/main/L0_MergeRequest_PR pipeline #39057 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[MrGeva]

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast

[tensorrt-cicd]

PR_Github #49660 [ run ] triggered by Bot. Commit: 60f48ec Link to invocation

[tensorrt-cicd]

PR_Github #49660 [ run ] completed with state SUCCESS. Commit: 60f48ec
/LLM/main/L0_MergeRequest_PR pipeline #39272 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[MrGeva]

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast

[tensorrt-cicd]

PR_Github #50096 [ run ] triggered by Bot. Commit: e6590c5 Link to invocation

[tensorrt-cicd]

PR_Github #50096 [ run ] completed with state SUCCESS. Commit: e6590c5
/LLM/main/L0_MergeRequest_PR pipeline #39649 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[MrGeva]

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast

[tensorrt-cicd]

PR_Github #50111 [ run ] triggered by Bot. Commit: e6590c5 Link to invocation

[tensorrt-cicd]

PR_Github #50111 [ run ] completed with state SUCCESS. Commit: e6590c5
/LLM/main/L0_MergeRequest_PR pipeline #39664 completed with status: 'SUCCESS'

CI Report

Link to invocation