[None][test] Waive 7 failed cases for main in QA CI

[tensorrt-cicd]

Auto-generated Waive PR

Created by: TensorRT LLM CI Report (requested by qa@nvidia.com)
Target branch: main
Jenkins build: https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/LLM_FUNCTION_CLUSTER_TEST/1483/
Bug(s): 6059036, 6211193, 6223556, 6241842, 6241845, 6245651

Waive entries added

disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp1-TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6223556)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus[fp4_indexer_dsl_mtp2] SKIP (https://nvbugs/6241842)
accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ngram SKIP (https://nvbugs/6245651)
disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp4-TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6223556)
accuracy/test_llm_api_autodeploy.py::TestModelRegistryAccuracy::test_autodeploy_from_registry[google_gemma-3-1b-it-False] SKIP (https://nvbugs/6059036)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus[fp4_indexer_dsl_mtp3] SKIP (https://nvbugs/6241845)
test_e2e.py::test_openai_disagg_multi_nodes_completion_service_discovery[etcd] SKIP (https://nvbugs/6211193)

---

This PR was auto-generated by TensorRT LLM CI Report. Please review the waive entries before merging.

Summary by CodeRabbit

Tests

• Updated test waiver configuration with new entries documenting known issues affecting model inference across implementations: Gemma 3.1B Instruct and Llama 3.1 8B Instruct (dtype and inference operations), DeepSeek V32 (multi-GPU quantization), TinyLlama 1.1B (distributed generation routing), and OpenAI distributed service compatibility.

[tensorrt-cicd]

/bot run --skip-test

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request adds ten test waiver (SKIP) entries to tests/integration/test_lists/waives.txt for failing integration tests across multiple model configurations and deployment modes, with each entry annotated with NVBug references for tracking.

Changes

Test Waivers

Layer / File(s)	Summary
Model and configuration test waivers tests/integration/test_lists/waives.txt	Added SKIP entries for Gemma 3.1B Instruct (dtype and kv_cache tests), Llama 3.1 8B Instruct (ngram), registry accuracy, DeepSeek V32 NVFP4 multi-gpu, disaggregated overlap generation, and OpenAI disaggregated service discovery tests.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~5 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#14503: Both PRs add SKIP/waiver entries to the same test lists file for CI failures.
• NVIDIA/TensorRT-LLM#14163: Both PRs add SKIP waivers including overlapping Llama 3.1 8B Instruct test cases.
• NVIDIA/TensorRT-LLM#14086: Both PRs add SKIP waivers for DeepSeek NVFP4 multi-gpu and Llama 3.1 test cases.

Suggested reviewers

• crazydemo
• jieli-matrix
• StanleySun639
• LarryXFly

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The PR description deviates from the required template and is auto-generated, missing standard sections like Description and Test Coverage, though it provides sufficient context for the waive PR purpose.	Consider following the standard PR template structure with Description and Test Coverage sections, even for auto-generated waive PRs, to maintain consistency and clarity.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title '[None][test] Waive 7 failed cases for main in QA CI' accurately reflects the main change: adding test waivers to the waives.txt file.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

tests/integration/test_lists/waives.txt (1)

6-9: QA assessment: Test coverage impact and follow-up recommendations.

This PR waivers 10 integration test failures across critical functionality areas:

Affected test areas:

• Disaggregated serving: 5 tests (Gemma dtype/kv_cache, Llama ngram, TinyLlama overlap)
• NVFP4 quantization: 2 tests (DeepSeekV32 multi-GPU variants)
• Autodeploy/registry: 1 test (Gemma model deployment)
• Multi-node service discovery: 1 test (etcd backend)
• Overlap generation router: 1 test (TinyLlama context pipeline parallelism)

Recommendations:

1. Track bug resolution timeline: Monitor associated bugs (6059036, 6117811, 6211193, 6223556, 6241842, 6241845, 6245651) to ensure timely fixes
2. Verify coverage gaps: Confirm alternative test coverage exists for waived functionality, especially for:
   • Gemma 3.1B Instruct dtype handling and KV cache v2
   • DeepSeekV32 NVFP4 multi-GPU configurations
   • Disaggregated overlap generation routing
3. Monitor waive list growth: Consider establishing metrics for total waived tests and maximum waive duration

Coverage assessment: Coverage is insufficient for disaggregated serving (multiple Gemma/Llama waivers) and multi-GPU NVFP4 quantization. Suggest prioritizing bug fixes for 6117811 (affects 3 tests) and 6241842/6241845 (DeepSeek multi-GPU).

Based on learnings, all nvbug URL formats are consistent (short form).

Also applies to: 19-19, 37-38, 172-173, 305-305

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/waives.txt` around lines 6 - 9, The waives file
lists skipped integration tests but lacks per-entry rationale, tracking
metadata, and coverage-impact grouping; update the waives list entries for each
skipped test (e.g.,
accuracy/test_disaggregated_serving.py::TestGemma3_1BInstruct::test_auto_dtype,
::test_kv_cache_v2_nixl_python,
accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ngram,
DeepSeekV32 multi-GPU entries) to include the NV bug IDs, date added,
owner/responsible engineer, and brief impact note; also add a summary header
that tallies waived tests by category (disaggregated serving, NVFP4
quantization, autodeploy/registry, multi-node discovery, overlap router) and a
TODO to re-run/reevaluate after bug fixes so reviewers can track waiver growth
and duration.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Around line 6-8: PR description claims "Waive 7 failed cases" but the diff
adds 10 waivers including three Gemma entries referencing bug 6117811
(TestGemma3_1BInstruct::test_auto_dtype[False],
TestGemma3_1BInstruct::test_auto_dtype[True],
TestGemma3_1BInstruct::test_kv_cache_v2_nixl_python in
tests/integration/test_lists/waives.txt); either update the PR title/body to
list all 10 waived tests and mention bug 6117811 covers these three, or remove
those three lines if they were added by mistake, and before finalizing confirm
QA that bug 6117811 indeed applies to all three Gemma failures.

---

Nitpick comments:
In `@tests/integration/test_lists/waives.txt`:
- Around line 6-9: The waives file lists skipped integration tests but lacks
per-entry rationale, tracking metadata, and coverage-impact grouping; update the
waives list entries for each skipped test (e.g.,
accuracy/test_disaggregated_serving.py::TestGemma3_1BInstruct::test_auto_dtype,
::test_kv_cache_v2_nixl_python,
accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ngram,
DeepSeekV32 multi-GPU entries) to include the NV bug IDs, date added,
owner/responsible engineer, and brief impact note; also add a summary header
that tallies waived tests by category (disaggregated serving, NVFP4
quantization, autodeploy/registry, multi-node discovery, overlap router) and a
TODO to re-run/reevaluate after bug fixes so reviewers can track waiver growth
and duration.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fd498774-3b2f-4d99-9c14-41f1f8c112d8

📥 Commits

Reviewing files that changed from the base of the PR and between cde9963 and b012034.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

[xinhe-nv]

/bot run --skip-test

[tensorrt-cicd]

PR_Github #51328 [ run ] triggered by Bot. Commit: 7dcd6e5 Link to invocation

[tensorrt-cicd]

PR_Github #51328 [ run ] completed with state SUCCESS. Commit: 7dcd6e5
/LLM/main/L0_MergeRequest_PR pipeline #40742 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[tensorrt-cicd]

/bot reuse-pipeline

[tensorrt-cicd]

/bot reuse-pipeline

[xinhe-nv]

/bot reuse-pipeline

[tensorrt-cicd]

/bot reuse-pipeline

[tensorrt-cicd]

/bot reuse-pipeline

[tensorrt-cicd]

/bot reuse-pipeline

[xinhe-nv]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #51461 [ reuse-pipeline ] triggered by Bot. Commit: 953b1e9 Link to invocation

[tensorrt-cicd]

PR_Github #51461 [ reuse-pipeline ] completed with state SUCCESS. Commit: 953b1e9
Reusing PR_Github #51328 (Partly Tested) for commit 953b1e9

Link to invocation