Skip to content

[None][test] Remove 28 closed-bug waive entries for main#14545

Merged
xinhe-nv merged 21 commits into
NVIDIA:mainfrom
tensorrt-cicd:trtllm-ci-report/unwaive-20260526-032423
Jun 3, 2026
Merged

[None][test] Remove 28 closed-bug waive entries for main#14545
xinhe-nv merged 21 commits into
NVIDIA:mainfrom
tensorrt-cicd:trtllm-ci-report/unwaive-20260526-032423

Conversation

@tensorrt-cicd

@tensorrt-cicd tensorrt-cicd commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Auto-generated Un-waive PR

Created by: TensorRT LLM CI (requested by qa@nvidia.com)
Target branch: main
Jenkins build: https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/LLM_UPDATE_WAIVES/110/
Closed bug(s) un-waived: 5979673, 6050483, 6071081, 6109750, 6114140, 6114608, 6128419, 6162122, 6185196, 6211880

Waive entries removed

accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_dsa_host_cache_offload[host_cache_offload] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_dsa_host_cache_offload[host_cache_offload_mtp1] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_dsa_host_cache_offload[host_cache_offload_mtp3_no_adp] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_fp8_blockscale[baseline] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_fp8_blockscale[latency_default] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus[baseline] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus[baseline_mtp1] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[baseline] SKIP (https://nvbugs/6185196)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_python_scheduler[mtp_nextn=2-attention_dp=False-cuda_graph=False-overlap_scheduler=False-enable_chunked_prefill=True] SKIP (https://nvbugs/6071081)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=2-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_guided_decoding_4gpus[llguidance-mtp_nextn=0] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_guided_decoding_4gpus[llguidance-mtp_nextn=2] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_guided_decoding_4gpus[xgrammar-mtp_nextn=0] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_guided_decoding_4gpus[xgrammar-mtp_nextn=2] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[moe_backend=TRTLLM-mtp_nextn=0-fp8kv=False-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_eagle3_4gpus[v2_kv_cache-cutlass-two_model-no_overlap_scheduler] SKIP (https://nvbugs/6211880)
accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-fp8] SKIP (https://nvbugs/6109750)
accuracy/test_llm_api_pytorch.py::TestLlama3_3_70BInstruct::test_fp4_tp2pp2[torch_compile=False-enable_gemm_allreduce_fusion=False] SKIP (https://nvbugs/6162122)
accuracy/test_llm_api_pytorch.py::TestLlama3_3_70BInstruct::test_fp4_tp2pp2[torch_compile=False-enable_gemm_allreduce_fusion=True] SKIP (https://nvbugs/6162122)
disaggregated/test_disaggregated.py::test_disaggregated_ctxtp2_genpp2[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6114140)
full:RTX_6000D/accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_nvfp4[latency_moe_cutlass] SKIP (https://nvbugs/6128419)
test_e2e.py::test_multi_nodes_eval[DeepSeek-R1/DeepSeek-R1-0528-FP4-tp16-mmlu] SKIP (https://nvbugs/6114608)
test_e2e.py::test_multi_nodes_eval[Kimi-K2-Thinking-NVFP4-tp16-mmlu] SKIP (https://nvbugs/6114608)
test_e2e.py::test_multi_nodes_eval[Qwen3/saved_models_Qwen3-235B-A22B_nvfp4_hf-tp16-mmlu] SKIP (https://nvbugs/6114608)
test_e2e.py::test_multi_nodes_eval[nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1-tp16-mmlu] SKIP (https://nvbugs/6114608)
unittest/disaggregated/test_agent_multi_backends.py::test_run_with_different_env[1] SKIP (https://nvbugs/5979673)
visual_gen/test_visual_gen_benchmark.py::test_offline_benchmark SKIP (https://nvbugs/6050483)
visual_gen/test_visual_gen_benchmark.py::test_online_benchmark[openai-videos] SKIP (https://nvbugs/6050483)

This PR was auto-generated by TensorRT LLM CI. Please review before merging.

Summary by CodeRabbit

  • Chores
    • Updated test infrastructure by cleaning up the test waiver list, removing obsolete skip entries and reorganizing test requirements across multiple model configurations. These changes streamline test coverage requirements while maintaining essential testing standards.

Review Change Stack

Closed bug(s): 5979673, 6050483, 6071081, 6109750, 6114140, 6114608, 6128419, 6162122, 6185196, 6211880
Requested by: qa@nvidia.com
Jenkins build: https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/LLM_UPDATE_WAIVES/110/

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
@xinhe-nv
xinhe-nv marked this pull request as ready for review May 26, 2026 04:01
@xinhe-nv
xinhe-nv requested a review from StanleySun639 May 26, 2026 04:01
@xinhe-nv
xinhe-nv enabled auto-merge (squash) May 26, 2026 04:01
@xinhe-nv

Copy link
Copy Markdown
Collaborator

/bot run

@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 04c702df-7797-46f7-87d4-877ff8ab3f37

📥 Commits

Reviewing files that changed from the base of the PR and between 526d7ee and 86cbed5.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR removes 27 lines from the test waiver list by consolidating and removing skip entries across multiple model test sections. DeepSeek model waivers are reduced to core test_fp8_blockscale cases, while other model and utility test waivers are cleaned up by removing entries for GPTOSS, Llama, Qwen3, and disaggregated tests.

Changes

Test Waiver Consolidation

Layer / File(s) Summary
DeepSeek model waiver consolidation
tests/integration/test_lists/waives.txt
DeepSeekV32 waivers reduced to only test_fp8_blockscale for baseline_mtp1 and disable_skip_indexer. DeepSeekV3Lite waivers reorganized: removed bfloat16_python_scheduler and guided_decoding_4gpus entries, added cute_dsl_nvfp4_4gpus variants, and retained test_no_kv_cache_reuse for the specified configuration.
Other model and utility test waivers
tests/integration/test_lists/waives.txt
Removed waivers for GPTOSS test_w4_4gpus, Llama3_3_70BInstruct fp4 tp2pp2, Qwen3_235B_A22B latency_moe_cutlass, and disaggregated test_disaggregated_ctxtp2_genpp2. Consolidated test_e2e.py::test_multi_nodes_eval to retain only Qwen3 235B tp16 mmlu. Removed waivers for disaggregated test agent and visual generation benchmark tests.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#14515: Both PRs modify the same waiver list with overlapping DeepSeekV3Lite changes including bfloat16_python_scheduler and cute_dsl_nvfp4_4gpus entries.
  • NVIDIA/TensorRT-LLM#14523: Both PRs modify the same file specifically in the TestDeepSeekV3Lite waiver entries for test_fp8_block_scales_4gpus and test_nvfp4_4gpus parameterizations.
  • NVIDIA/TensorRT-LLM#14440: Both PRs modify tests/integration/test_lists/waives.txt with overlapping changes to the disaggregated test_disaggregated_ctxtp2_genpp2 waiver entry.

Suggested reviewers

  • crazydemo
  • LarryXFly
  • jieli-matrix
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: removing 28 closed-bug waive entries for the main branch.
Description check ✅ Passed The description provides comprehensive context including automated origin, target branch, Jenkins build link, closed bug IDs, and detailed list of removed waiver entries, meeting the information needs despite not following the optional template format.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #50269 [ run ] triggered by Bot. Commit: 86cbed5 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #50269 [ run ] completed with state SUCCESS. Commit: 86cbed5
/LLM/main/L0_MergeRequest_PR pipeline #39802 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@xinhe-nv

Copy link
Copy Markdown
Collaborator

/bot --help

@xinhe-nv

Copy link
Copy Markdown
Collaborator

/bot -h

@github-actions

Copy link
Copy Markdown

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@xinhe-nv

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #50423 [ run ] triggered by Bot. Commit: 55a980c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #50423 [ run ] completed with state SUCCESS. Commit: 55a980c
/LLM/main/L0_MergeRequest_PR pipeline #39947 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51470 [ run ] completed with state SUCCESS. Commit: a7e5624
/LLM/main/L0_MergeRequest_PR pipeline #40876 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xinhe-nv added 3 commits June 2, 2026 17:16
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
@xinhe-nv

xinhe-nv commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51585 [ run ] triggered by Bot. Commit: 6f9fb38 Link to invocation

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51585 [ run ] completed with state SUCCESS. Commit: 6f9fb38
/LLM/main/L0_MergeRequest_PR pipeline #40975 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@xinhe-nv

xinhe-nv commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51715 [ run ] triggered by Bot. Commit: ba866a7 Link to invocation

@xinhe-nv

xinhe-nv commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

/bot run --stage-list ""

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51745 [ run ] triggered by Bot. Commit: c92a615 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51715 [ run ] completed with state ABORTED. Commit: ba866a7

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51745 [ run ] completed with state SUCCESS. Commit: c92a615
/LLM/main/L0_MergeRequest_PR pipeline #41118 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@xinhe-nv

xinhe-nv commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

/bot reuse-pipeline

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51759 [ reuse-pipeline ] triggered by Bot. Commit: 81448ac Link to invocation

@xinhe-nv

xinhe-nv commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

/bot reuse-pipeline

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51759 [ reuse-pipeline ] completed with state SUCCESS. Commit: 81448ac
Reusing PR_Github #51745 (Partly Tested) for commit 81448ac

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51764 [ reuse-pipeline ] triggered by Bot. Commit: 03f636f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #51764 [ reuse-pipeline ] completed with state SUCCESS. Commit: 03f636f
Reusing PR_Github #51745 (Partly Tested) for commit 03f636f

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants