[TRTLLM-13022][test] remove deprecated models from tests by xinhe-nv · Pull Request #14660 · NVIDIA/TensorRT-LLM

xinhe-nv · 2026-05-28T02:38:45Z

Summary by CodeRabbit

Tests
- Removed test coverage for legacy model variants (Llama v2, Gemma 3, EXAONE) and LoRA-related functionality across multiple test suites.
- Updated test configurations to focus on newer model versions and simplified test matrices.
- Removed model-specific test fixtures, examples, and benchmark entries.
Chores
- Consolidated test lists and removed obsolete test cases from CI pipelines.
- Updated test data files and model path mappings.

Description

delete deprecated tests based on https://docs.google.com/spreadsheets/d/1GGrxEJ6qCmMXgpw_OhFM80N9lcEm3dqci_M-_lrt2RM/edit?gid=1239703108#gid=1239703108

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-05-28T02:43:32Z

📝 Walkthrough

Walkthrough

This PR systematically removes test coverage for deprecated models (Llama 2, Gemma 3 variants, EXAONE4, GLM4Flash) and LoRA features across the entire test suite. Changes include removing accuracy reference data, test classes and methods, example tests, unit test modules, end-to-end tests, fixtures, performance configurations, Triton tests, and updating all test selection lists in CI configurations.

Changes

Test suite deprecation cleanup

Layer / File(s)	Summary
Accuracy reference data removal `tests/integration/defs/accuracy/references/*.yaml`	Removed accuracy benchmark reference entries for Llama 2, Gemma 3 (1B and 27B), EXAONE4, and GLM4Flash across CNN-DailyMail, GSM8K, JSON mode, MMLU, and MMMU benchmark datasets.
Accuracy test class removals `tests/integration/defs/accuracy/test_cli_flow.py`, `test_disaggregated_serving.py`, `test_llm_api_autodeploy.py`, `test_llm_api_pytorch.py`, `test_llm_api_pytorch_encode.py`, `test_llm_api_pytorch_multimodal.py`	Deleted test classes `TestLlama7B`, `TestLlama2_7B`, `TestEXAONE4`, `TestGemma3_*`, `TestGLM4Flash`; replaced `TestGemma3_1BInstruct` with `TestGPTOSS` and `TestGLM4Flash` with `TestQwen3NextInstruct`; updated `TestModelRegistryAccuracy` parameter lists.
Example test file updates and removals `tests/integration/defs/examples/test_gemma.py`, `test_exaone.py`, `test_llama.py`	Deleted entire `test_gemma.py` and `test_exaone.py` modules; removed Llama v1/v2 test functions and reduced parametrization for Llama 3 tests; cleaned up imports.
LoRA and KV cache feature test removals `tests/integration/defs/kv_cache/test_kv_cache_v2_scheduler.py`, `tests/unittest/llmapi/.py`, `tests/unittest/llmapi/test_llm_multi_gpu.py`	Removed `TestKVCacheV2LoRA` class and multiple LoRA-related test functions; deleted LoRA helper utilities and prompt-adapter tests; updated imports to remove PEFT/LoRA dependencies.
Unit test file and method removals `tests/unittest/_torch/modeling/test_modeling_.py`, `tests/unittest/trt/model_api/test_model_.py`, `tests/unittest/llmapi/apps/*.py`, `tests/unittest/others/test_layer.py`, `test_llama_conversion.sh`	Deleted entire test modules (`test_modeling_exaone4.py`, `test_modeling_gemma3.py`, `test_model_level_api.py`, `test_model_quantization.py`, `_test_openai_lora.py`, `_test_trtllm_serve_lora.py`, `test_llama_conversion.sh`); removed individual test methods and helpers.
End-to-end test updates `tests/integration/defs/test_e2e.py`	Removed `test_llama_e2e`, `test_mistral_e2e`, `test_mistral_large_hidden_vocab_size`, `test_trtllm_serve_lora_example`, `test_openai_lora`; pruned Llama 3.1 70B/405B variants from quickstart test parameters.
Fixture removal and model path updates `tests/integration/defs/conftest.py`	Removed `gemma_example_root`, `gemma_model_root`, `llama_tokenizer_model_root` fixtures; removed Llama 2 and Llama 3.1 70B mappings from `llama_model_root` and `draft_target_model_roots`; removed LoRA adapter mappings.
Performance test configuration updates `tests/integration/defs/perf/_model_paths.py`, `allowed_configs.py`, `pytorch_model_config.py`, `sampler_options_config.py`, `test_perf.py`	Updated model path mappings to replace Llama 3.1 variants with Llama 3.3 and v4; removed `llama_7b` from allowed configs; updated tokenizer fallback path; removed pattern entries and sampler option mappings.
Triton server test updates `tests/integration/defs/triton_server/build_engines.py`, `build_model.sh`, `test_triton.py`, `test_triton_llm.py`, `test_triton_rcca.py`	Replaced Llama v2 engine helpers with Llama v3 variants; removed `test_llama_v2_*_ifb` functions; removed Llama model mapping from build script; deleted RCCA bug test case.
LLM API end-to-end test updates `tests/integration/defs/llmapi/test_llm_e2e.py`	Removed `test_llmapi_load_engine_from_build_command_with_lora` and `test_llmapi_load_ckpt_from_convert_command`; removed Llama 7B from parametrization list.
Test list configuration updates across all GPU targets `tests/integration/test_lists/qa/.txt`, `test-db/.yml`, `waives.txt`	Updated QA and test-db YAML/TXT configurations for A10, A30, A100, H100, H200, L40S, GH200, B200, and DGX platforms to remove entries for deleted test classes and models; updated Spark test lists; updated Triton integration tests; removed waivers for deleted tests.
Test data and benchmark mappings `tests/integration/defs/.test_durations`, `tests/test_common/llm_data.py`, `tests/microbenchmarks/build_time_benchmark.py`	Updated benchmark duration mapping to remove deleted test entries; removed GLM and Gemma model entries from HuggingFace ID mappings; updated model loader test parameter generation; pruned microbenchmark model list.

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#13504: Both PRs modify tests/integration/test_lists/waives.txt by changing which QA integration test cases are marked SKIP.
NVIDIA/TensorRT-LLM#13790: Both PRs update tests/integration/test_lists/waives.txt to adjust QA CI SKIP coverage.
NVIDIA/TensorRT-LLM#14352: Both PRs modify AutoDeploy accuracy coverage in tests/integration/defs/accuracy/test_llm_api_autodeploy.py.

Suggested reviewers

tijyojwad
moraxu
brb-nv
nvchenghaoz

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

coderabbitai

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/integration/test_lists/test-db/l0_h100.yml (1)
51-56: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Comment scope is inconsistent with the listed model tests.

Line 51 says H100 key models are llama/mixtral/gpt-oss, but this block also includes modeling_gemma3 and test_modeling_nemotron_h. Please update the comment (or the list) so scope reviewers aren’t misled.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/test-db/l0_h100.yml` around lines 51 - 56, The
comment above the H100 block currently states "Only key models in H100:
llama/mixtral/gpt-oss" but the test list also includes modeling_gemma3 and
test_modeling_nemotron_h; update either the comment or the list so they match:
either remove modeling_gemma3 and test_modeling_nemotron_h from the block
(leaving modeling_llama, modeling_mixtral, modeling_gpt_oss) or expand the
comment to include gemma3 and nemotron_h (reference the entries modeling_gemma3
and unittest/_torch/modeling/test_modeling_nemotron_h::test_nemotron_h_sanity)
to make the scope accurate.

🧹 Nitpick comments (2)

tests/integration/defs/perf/test_perf.py (1)

1069-1073: ⚡ Quick win

Guard fallback tokenizer path with an explicit check.

If this fallback path is missing in a given environment, failure currently happens later in command execution. A direct check here gives a clearer error.

♻️ Proposed change

         else:
             tokenizer_dir = os.path.join(llm_models_root(), "llama-models-v2",
                                          "TinyLlama-1.1B-Chat-v1.0")
+            if not os.path.exists(tokenizer_dir):
+                raise FileNotFoundError(
+                    f"Fallback tokenizer path does not exist: {tokenizer_dir}"
+                )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/perf/test_perf.py` around lines 1069 - 1073, Add an
explicit existence check for the computed tokenizer_dir (the fallback path built
with llm_models_root() and "TinyLlama-1.1B-Chat-v1.0") immediately after it is
assigned; if os.path.exists(tokenizer_dir) is False, raise a clear exception or
call process exit with a descriptive error message indicating the missing
tokenizer directory so failures fail fast and clearly rather than later during
command execution.

tests/integration/defs/conftest.py (1)

1234-1279: ⚡ Quick win

Fail fast on unsupported LoRA adapter names.

This fixture silently drops unknown adapter IDs and can return an empty/partial path list, which makes downstream failures harder to diagnose—especially after removing deprecated mappings.

♻️ Proposed change

 def llm_lora_model_root(request):
@@
-    for item in model_list:
+    unsupported_items = []
+    for item in model_list:
         if item == "Japanese-Alpaca-LoRA-7b-v0":
@@
         elif item == "gpt-oss-20b-lora-adapter_NIM_r8":
             model_root_list.append(
                 os.path.join(models_root, "gpt_oss",
                              "gpt-oss-20b-lora-adapter_NIM_r8"))
+        else:
+            unsupported_items.append(item)
+
+    if unsupported_items:
+        raise ValueError(
+            f"Unsupported LoRA adapter(s): {unsupported_items}. "
+            "Please update tests/integration/defs/conftest.py mappings."
+        )
 
     return ",".join(model_root_list)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/conftest.py` around lines 1234 - 1279, The loop over
model_list (building model_root_list and returning ",".join(model_root_list))
currently ignores unknown adapter IDs; change it to fail fast by detecting any
items not matched by the existing if/elif branches and raising a clear exception
(e.g., ValueError) listing the unsupported adapter names; implement this by
collecting unmatched_items while iterating (or compute set(model_list) -
set(supported_names)) and raising with the offending IDs so callers get an
immediate, actionable error instead of a silent partial/empty result.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/test_e2e.py`:
- Line 31: Remove the duplicate import of convert_weights from the absolute
module and keep only the relative import style used elsewhere in this file:
consolidate imports so convert_weights is imported from .common (remove the
import from defs.common) to prevent symbol shadowing and ambiguous resolution;
update the import line(s) in tests/integration/defs/test_e2e.py to reference
.common for convert_weights and ensure any other imported names
(get_mmlu_accuracy, venv_check_call) remain correctly imported from their
intended module.

In `@tests/unittest/trt/model/test_llama.py`:
- Around line 439-453: get_loader_test_cases currently builds test_cases using
product([], ...), which yields an empty list and causes
parameterized.expand(get_loader_test_cases, ...) to raise ValueError; fix it by
ensuring get_loader_test_cases returns at least one valid loader pair (e.g., add
a non-deprecated HF/meta pair to the iterable instead of the empty list) or add
a guard so the decorator never receives an empty iterable (for example, change
the first argument to product([( "hf", "meta" )], ...) or wrap the parameter
provider so it returns a default single-case list when empty); update references
to get_loader_test_cases and the test_loaders parameterization accordingly.

---

Outside diff comments:
In `@tests/integration/test_lists/test-db/l0_h100.yml`:
- Around line 51-56: The comment above the H100 block currently states "Only key
models in H100: llama/mixtral/gpt-oss" but the test list also includes
modeling_gemma3 and test_modeling_nemotron_h; update either the comment or the
list so they match: either remove modeling_gemma3 and test_modeling_nemotron_h
from the block (leaving modeling_llama, modeling_mixtral, modeling_gpt_oss) or
expand the comment to include gemma3 and nemotron_h (reference the entries
modeling_gemma3 and
unittest/_torch/modeling/test_modeling_nemotron_h::test_nemotron_h_sanity) to
make the scope accurate.

---

Nitpick comments:
In `@tests/integration/defs/conftest.py`:
- Around line 1234-1279: The loop over model_list (building model_root_list and
returning ",".join(model_root_list)) currently ignores unknown adapter IDs;
change it to fail fast by detecting any items not matched by the existing
if/elif branches and raising a clear exception (e.g., ValueError) listing the
unsupported adapter names; implement this by collecting unmatched_items while
iterating (or compute set(model_list) - set(supported_names)) and raising with
the offending IDs so callers get an immediate, actionable error instead of a
silent partial/empty result.

In `@tests/integration/defs/perf/test_perf.py`:
- Around line 1069-1073: Add an explicit existence check for the computed
tokenizer_dir (the fallback path built with llm_models_root() and
"TinyLlama-1.1B-Chat-v1.0") immediately after it is assigned; if
os.path.exists(tokenizer_dir) is False, raise a clear exception or call process
exit with a descriptive error message indicating the missing tokenizer directory
so failures fail fast and clearly rather than later during command execution.

🪄 Autofix (Beta)

❌ Autofix failed (check again to retry)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7074a434-7487-4541-89fd-671b790023d0

📥 Commits

Reviewing files that changed from the base of the PR and between 757f1e7 and d11225e.

📒 Files selected for processing (67)

tests/integration/defs/.test_durations
tests/integration/defs/accuracy/references/cnn_dailymail.yaml
tests/integration/defs/accuracy/references/gsm8k.yaml
tests/integration/defs/accuracy/references/json_mode_eval.yaml
tests/integration/defs/accuracy/references/mmlu.yaml
tests/integration/defs/accuracy/references/mmmu.yaml
tests/integration/defs/accuracy/test_cli_flow.py
tests/integration/defs/accuracy/test_disaggregated_serving.py
tests/integration/defs/accuracy/test_llm_api_autodeploy.py
tests/integration/defs/accuracy/test_llm_api_pytorch.py
tests/integration/defs/accuracy/test_llm_api_pytorch_encode.py
tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py
tests/integration/defs/conftest.py
tests/integration/defs/examples/test_exaone.py
tests/integration/defs/examples/test_gemma.py
tests/integration/defs/examples/test_llama.py
tests/integration/defs/kv_cache/test_kv_cache_v2_scheduler.py
tests/integration/defs/llmapi/test_llm_e2e.py
tests/integration/defs/perf/_model_paths.py
tests/integration/defs/perf/allowed_configs.py
tests/integration/defs/perf/pytorch_model_config.py
tests/integration/defs/perf/sampler_options_config.py
tests/integration/defs/perf/test_perf.py
tests/integration/defs/test_e2e.py
tests/integration/defs/triton_server/build_engines.py
tests/integration/defs/triton_server/build_model.sh
tests/integration/defs/triton_server/test_triton.py
tests/integration/defs/triton_server/test_triton_llm.py
tests/integration/defs/triton_server/test_triton_rcca.py
tests/integration/test_lists/qa/llm_function_core.txt
tests/integration/test_lists/qa/llm_function_l20.txt
tests/integration/test_lists/qa/llm_function_rtx6k.txt
tests/integration/test_lists/qa/llm_spark_func.yml
tests/integration/test_lists/qa/llm_spark_perf.yml
tests/integration/test_lists/qa/llm_triton_integration.txt
tests/integration/test_lists/test-db/l0_a10.yml
tests/integration/test_lists/test-db/l0_a100.yml
tests/integration/test_lists/test-db/l0_a30.yml
tests/integration/test_lists/test-db/l0_b200.yml
tests/integration/test_lists/test-db/l0_dgx_b300.yml
tests/integration/test_lists/test-db/l0_dgx_h100.yml
tests/integration/test_lists/test-db/l0_dgx_h200.yml
tests/integration/test_lists/test-db/l0_gb203.yml
tests/integration/test_lists/test-db/l0_gh200.yml
tests/integration/test_lists/test-db/l0_h100.yml
tests/integration/test_lists/test-db/l0_l40s.yml
tests/integration/test_lists/waives.txt
tests/microbenchmarks/build_time_benchmark.py
tests/test_common/llm_data.py
tests/unittest/_torch/modeling/test_modeling_exaone4.py
tests/unittest/_torch/modeling/test_modeling_gemma3.py
tests/unittest/llmapi/apps/_test_openai_consistent_chat.py
tests/unittest/llmapi/apps/_test_openai_lora.py
tests/unittest/llmapi/apps/_test_trtllm_serve_lora.py
tests/unittest/llmapi/lora_test_utils.py
tests/unittest/llmapi/test_executor.py
tests/unittest/llmapi/test_grpc.py
tests/unittest/llmapi/test_llm.py
tests/unittest/llmapi/test_llm_multi_gpu.py
tests/unittest/llmapi/test_llm_multi_gpu_pytorch.py
tests/unittest/llmapi/test_llm_pytorch.py
tests/unittest/others/test_layer.py
tests/unittest/others/test_llama_conversion.sh
tests/unittest/trt/model/test_llama.py
tests/unittest/trt/model_api/test_model_api_multi_gpu.py
tests/unittest/trt/model_api/test_model_level_api.py
tests/unittest/trt/model_api/test_model_quantization.py

💤 Files with no reviewable changes (52)

tests/unittest/others/test_llama_conversion.sh
tests/integration/test_lists/qa/llm_function_l20.txt
tests/integration/defs/accuracy/references/json_mode_eval.yaml
tests/integration/test_lists/test-db/l0_gh200.yml
tests/unittest/llmapi/apps/_test_trtllm_serve_lora.py
tests/integration/defs/examples/test_exaone.py
tests/integration/defs/perf/sampler_options_config.py
tests/integration/test_lists/test-db/l0_dgx_b300.yml
tests/integration/test_lists/test-db/l0_gb203.yml
tests/integration/test_lists/test-db/l0_a10.yml
tests/integration/defs/accuracy/test_llm_api_pytorch_encode.py
tests/integration/test_lists/test-db/l0_dgx_h100.yml
tests/unittest/trt/model_api/test_model_quantization.py
tests/integration/test_lists/qa/llm_triton_integration.txt
tests/integration/defs/examples/test_gemma.py
tests/integration/defs/accuracy/references/gsm8k.yaml
tests/integration/test_lists/test-db/l0_dgx_h200.yml
tests/integration/defs/accuracy/references/mmmu.yaml
tests/microbenchmarks/build_time_benchmark.py
tests/integration/test_lists/test-db/l0_l40s.yml
tests/unittest/trt/model_api/test_model_level_api.py
tests/unittest/_torch/modeling/test_modeling_gemma3.py
tests/integration/defs/perf/pytorch_model_config.py
tests/integration/test_lists/test-db/l0_b200.yml
tests/integration/test_lists/qa/llm_function_rtx6k.txt
tests/unittest/llmapi/apps/_test_openai_lora.py
tests/unittest/_torch/modeling/test_modeling_exaone4.py
tests/integration/test_lists/qa/llm_spark_perf.yml
tests/integration/test_lists/test-db/l0_a100.yml
tests/integration/defs/llmapi/test_llm_e2e.py
tests/integration/defs/accuracy/references/cnn_dailymail.yaml
tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py
tests/integration/defs/accuracy/references/mmlu.yaml
tests/integration/defs/triton_server/test_triton.py
tests/integration/test_lists/qa/llm_spark_func.yml
tests/unittest/others/test_layer.py
tests/integration/defs/.test_durations
tests/integration/defs/perf/allowed_configs.py
tests/integration/defs/triton_server/build_model.sh
tests/integration/defs/accuracy/test_llm_api_autodeploy.py
tests/integration/defs/triton_server/build_engines.py
tests/integration/defs/perf/_model_paths.py
tests/integration/test_lists/waives.txt
tests/unittest/llmapi/test_llm_multi_gpu_pytorch.py
tests/integration/defs/accuracy/test_cli_flow.py
tests/integration/defs/accuracy/test_llm_api_pytorch.py
tests/integration/defs/triton_server/test_triton_llm.py
tests/integration/test_lists/qa/llm_function_core.txt
tests/test_common/llm_data.py
tests/integration/defs/kv_cache/test_kv_cache_v2_scheduler.py
tests/integration/defs/accuracy/test_disaggregated_serving.py
tests/integration/test_lists/test-db/l0_a30.yml

coderabbitai · 2026-05-28T09:16:00Z

Note

Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it.

An unexpected error occurred while generating fixes: Not Found - https://docs.github.com/rest/git/refs#get-a-reference

xinhe-nv · 2026-05-28T09:22:44Z

/bot run

tensorrt-cicd · 2026-05-28T09:28:44Z

PR_Github #50749 [ run ] triggered by Bot. Commit: 548dc9f Link to invocation

tensorrt-cicd · 2026-05-28T11:39:04Z

PR_Github #50749 [ run ] completed with state FAILURE. Commit: 548dc9f
/LLM/main/L0_MergeRequest_PR pipeline #40228 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

2ez4bz · 2026-05-29T04:02:35Z

Thanks for doing this!

xinhe-nv · 2026-06-02T07:54:41Z

/bot run --skip-test

tensorrt-cicd · 2026-06-02T08:01:16Z

PR_Github #51553 [ run ] triggered by Bot. Commit: a19a75a Link to invocation

xinhe-nv · 2026-06-02T09:23:53Z

/bot run --skip-test

tensorrt-cicd · 2026-06-02T09:26:35Z

PR_Github #51553 [ run ] completed with state FAILURE. Commit: a19a75a
/LLM/main/L0_MergeRequest_PR pipeline #40948 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-06-02T09:30:20Z

PR_Github #51586 [ run ] triggered by Bot. Commit: a8bbdf9 Link to invocation

tensorrt-cicd · 2026-06-02T09:36:45Z

PR_Github #51586 [ run ] completed with state FAILURE. Commit: a8bbdf9

Link to invocation

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-06-03T01:10:41Z

/bot run --skip-test

tensorrt-cicd · 2026-06-03T01:17:02Z

PR_Github #51710 [ run ] triggered by Bot. Commit: 3476271 Link to invocation

tensorrt-cicd · 2026-06-03T02:25:59Z

PR_Github #51710 [ run ] completed with state SUCCESS. Commit: 3476271
/LLM/main/L0_MergeRequest_PR pipeline #41087 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

A missing comma at the boundary of two merged duration blocks made .test_durations invalid JSON, causing pytest_split to abort collection with an INTERNALERROR during --collect-only. Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>

xinhe-nv · 2026-06-03T02:38:28Z

/bot run --skip-test

tensorrt-cicd · 2026-06-03T02:44:51Z

PR_Github #51734 [ run ] triggered by Bot. Commit: b89efee Link to invocation

tensorrt-cicd · 2026-06-03T05:50:53Z

PR_Github #51734 [ run ] completed with state SUCCESS. Commit: b89efee
/LLM/main/L0_MergeRequest_PR pipeline #41109 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

xinhe-nv · 2026-06-03T06:02:46Z

/bot reuse-pipeline

tensorrt-cicd · 2026-06-03T06:09:14Z

PR_Github #51770 [ reuse-pipeline ] triggered by Bot. Commit: 4c28726 Link to invocation

tensorrt-cicd · 2026-06-03T06:21:49Z

PR_Github #51770 [ reuse-pipeline ] completed with state SUCCESS. Commit: 4c28726
Reusing PR_Github #51734 (Partly Tested) for commit 4c28726

Link to invocation

xinhe-nv · 2026-06-03T06:27:08Z

/bot reuse-pipeline

tensorrt-cicd · 2026-06-03T06:32:42Z

PR_Github #51775 [ reuse-pipeline ] triggered by Bot. Commit: 4c28726 Link to invocation

tensorrt-cicd · 2026-06-03T06:43:40Z

PR_Github #51775 [ reuse-pipeline ] completed with state SUCCESS. Commit: 4c28726
Reusing PR_Github #51734 (Partly Tested) for commit 4c28726

Link to invocation

xinhe-nv requested review from a team as code owners May 28, 2026 02:38

xinhe-nv requested review from brb-nv, moraxu, nvchenghaoz and tijyojwad May 28, 2026 02:38

github-actions Bot assigned xinhe-nv May 28, 2026

coderabbitai Bot reviewed May 28, 2026

View reviewed changes

Comment thread tests/integration/defs/test_e2e.py

Comment thread tests/unittest/trt/model/test_llama.py

QiJune reviewed May 28, 2026

View reviewed changes

Comment thread tests/integration/defs/kv_cache/test_kv_cache_v2_scheduler.py

xinhe-nv force-pushed the clean-models branch 2 times, most recently from 88792e9 to 9829d94 Compare May 28, 2026 08:50

xinhe-nv changed the title ~~[TRTLLM-13022][test] remove deprecated models from tests.~~ [TRTLLM-13022][test] remove deprecated models from tests May 28, 2026

xinhe-nv force-pushed the clean-models branch from 9829d94 to 548dc9f Compare May 28, 2026 09:14

xinhe-nv enabled auto-merge (squash) May 28, 2026 09:22

nvchenghaoz reviewed May 28, 2026

View reviewed changes

Comment thread tests/integration/defs/accuracy/references/mmlu.yaml

QiJune reviewed May 29, 2026

View reviewed changes

jieli-matrix approved these changes May 29, 2026

View reviewed changes

2ez4bz approved these changes May 29, 2026

View reviewed changes

2ez4bz reviewed May 29, 2026

View reviewed changes

Comment thread tests/unittest/_torch/modeling/test_modeling_exaone4.py

Merge branch 'main' into clean-models

a8bbdf9

xinhe-nv added 2 commits June 2, 2026 17:39

Merge branch 'main' into clean-models

4665365

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

Merge branch 'main' into clean-models

3476271

xinhe-nv added 3 commits June 3, 2026 10:36

Merge branch 'main' into clean-models

79efc6f

Merge branch 'main' into clean-models

b89efee

Merge branch 'main' into clean-models

3685f64

Merge branch 'main' into clean-models

4c28726

xinhe-nv merged commit 514afc8 into NVIDIA:main Jun 3, 2026
7 checks passed

xinhe-nv deleted the clean-models branch June 3, 2026 06:44

coderabbitai Bot mentioned this pull request Jun 4, 2026

[None][test] Update DGX spark CI/QA tests #14978

Open

1 task

coderabbitai Bot mentioned this pull request Jul 22, 2026

[TRTLLM-14473][chore] Remove legacy TensorRT backend tests examples and CI plumbing #16610

Merged

1 task

Uh oh!

Conversation

xinhe-nv commented May 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Possibly related PRs

Suggested reviewers

Uh oh!

coderabbitai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xinhe-nv commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

2ez4bz commented May 29, 2026

Uh oh!

Uh oh!

xinhe-nv commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

xinhe-nv commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

xinhe-nv commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

xinhe-nv commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

xinhe-nv commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

xinhe-nv commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

Uh oh!

xinhe-nv commented May 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 28, 2026 •

edited

Loading

coderabbitai Bot left a comment •

edited

Loading

coderabbitai Bot commented May 28, 2026 •

edited

Loading