[None][infra] Improve unit test CI coverage by yuxianq · Pull Request #15368 · NVIDIA/TensorRT-LLM

yuxianq · 2026-06-15T07:44:32Z

Summary by CodeRabbit

Release Notes

Tests
- Updated test coverage across multiple GPU configurations (A10, A100, A30, B200, H100, DGX H100, sanity checks)
- Adjusted test selection for PyTorch, TensorRT, and AutoDeploy test suites
Chores
- Removed deprecated profiling decorator utilities
- Updated code linting and pre-commit configurations

Description

Improve CI coverage for unit tests by mapping uncovered or relocated unit test directories into existing L0 stages, moving several tests under stage-covered directories, and making CBTS path matching robust to directory test-db entries with a trailing slash.

This also normalizes the unittest/kv_cache_manager_v2_tests test-db and memory-parallelism entries so CBTS and runtime test-list data use the same canonical id.

Test Coverage

Commit pre-commit hooks passed.
pre-commit run --files jenkins/scripts/cbts/blocks.py tests/integration/test_lists/test-db/l0_a10.yml tests/integration/test_lists/test-db/l0_b200.yml tests/integration/test_lists/test-db/l0_h100.yml tests/integration/defs/agg_unit_mem_df.csv
Verified CBTS lookup maps unittest/kv_cache_manager_v2_tests/test_branch_reuse.py through the A10, B200, and H100 blocks.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

yuxianq · 2026-06-15T07:44:46Z

/bot run

coderabbitai · 2026-06-15T07:48:48Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ac4d8d7e-e3f2-4d0d-a9d6-e669f3ec148a

📥 Commits

Reviewing files that changed from the base of the PR and between 26ea499 and a851455.

⛔ Files ignored due to path filters (1)

tests/integration/defs/agg_unit_mem_df.csv is excluded by !**/*.csv

📒 Files selected for processing (20)

.pre-commit-config.yaml
jenkins/scripts/cbts/blocks.py
legacy-files.txt
pyproject.toml
ruff-legacy-baseline.json
ruff-legacy.toml
tests/integration/test_lists/test-db/l0_a10.yml
tests/integration/test_lists/test-db/l0_a100.yml
tests/integration/test_lists/test-db/l0_a30.yml
tests/integration/test_lists/test-db/l0_b200.yml
tests/integration/test_lists/test-db/l0_dgx_h100.yml
tests/integration/test_lists/test-db/l0_h100.yml
tests/integration/test_lists/test-db/l0_sanity_check.yml
tests/unittest/_torch/attention/test_attention_op_sync.py
tests/unittest/_torch/attention/test_triton_prefill.py
tests/unittest/_torch/executor/test_iter_stats_populate.py
tests/unittest/_torch/executor/test_model_loader_gms.py
tests/unittest/_torch/executor/test_model_loader_mx.py
tests/unittest/_torch/executor/test_multimodal_embedding_lengths.py
tests/unittest/trt/model_api/profile_utils.py

💤 Files with no reviewable changes (7)

tests/unittest/trt/model_api/profile_utils.py
legacy-files.txt
ruff-legacy-baseline.json
.pre-commit-config.yaml
pyproject.toml
ruff-legacy.toml
tests/integration/test_lists/test-db/l0_sanity_check.yml

📝 Walkthrough

Walkthrough

Removes tests/unittest/trt/model_api/profile_utils.py and its profile decorator, cleans it from all linting/pre-commit registrations, and adds test_nemotron_nas.py to pre-commit file lists. Introduces _normalize_target_path in blocks.py to strip trailing slashes from test target paths. Updates integration test-selection YAMLs across six GPU configs to add new test directories, remove stale entries, and fix trailing-slash inconsistencies.

Changes

Test infrastructure and linting config updates

Layer / File(s)	Summary
profile_utils.py removal and linting config cleanup `tests/unittest/trt/model_api/profile_utils.py`, `legacy-files.txt`, `pyproject.toml`, `ruff-legacy-baseline.json`, `ruff-legacy.toml`, `.pre-commit-config.yaml`	Removes all content from `profile_utils.py` (deleting the `profile` decorator), removes its entry from every linting and pre-commit registration file, and adds `tests/unittest/trt/model/test_nemotron_nas.py` to both pre-commit regex lists.
blocks.py trailing-slash path normalization `jenkins/scripts/cbts/blocks.py`	Adds `_normalize_target_path` to trim trailing slashes from directory-style target paths, then applies it in `_entry_target`, `_target_in_filter_subtree`, `_path_lookup_anchor`, and `YAMLIndex` canonical key generation.
Integration test-selection YAML updates `tests/integration/test_lists/test-db/l0_a10.yml`, `tests/integration/test_lists/test-db/l0_a100.yml`, `tests/integration/test_lists/test-db/l0_a30.yml`, `tests/integration/test_lists/test-db/l0_b200.yml`, `tests/integration/test_lists/test-db/l0_dgx_h100.yml`, `tests/integration/test_lists/test-db/l0_h100.yml`, `tests/integration/test_lists/test-db/l0_sanity_check.yml`	Adds new test directories (`unittest/trt/python_plugin`, `unittest/trt/model/redrafter`, `unittest/auto_deploy/singlegpu/mlir`, `unittest/auto_deploy/multigpu/compile`, `unittest/_torch/auto_deploy`, `unittest/_torch/distributed`, `unittest/scaffolding`, `unittest/tools`), removes stale entries, broadens a checkpoint-loader include, and normalizes trailing slashes on `unittest/kv_cache_manager_v2_tests` paths across multiple GPU configs.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main objective as improving CI coverage for unit tests, which directly aligns with the primary changes across multiple test-db YAML files and CBTS infrastructure improvements.
Description check	✅ Passed	The description comprehensively explains the changes (test directory mapping, CBTS path robustness, normalization), includes test coverage validation details, and follows the repository template with all major sections completed.
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

⚔️ Resolve merge conflicts

Resolve merge conflict in branch infra/unit-test-ci-coverage

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

yuxianq · 2026-06-15T07:48:57Z

/bot run

tensorrt-cicd · 2026-06-15T07:56:02Z

PR_Github #54258 [ run ] triggered by Bot. Commit: c843f4d Link to invocation

tensorrt-cicd · 2026-06-15T08:29:22Z

PR_Github #54258 [ run ] completed with state FAILURE. Commit: c843f4d
/LLM/main/L0_MergeRequest_PR pipeline #43332 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-06-16T02:57:23Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T03:02:52Z

PR_Github #54438 [ run ] triggered by Bot. Commit: 42a6f27 Link to invocation

tensorrt-cicd · 2026-06-16T10:14:32Z

PR_Github #54438 [ run ] completed with state SUCCESS. Commit: 42a6f27
/LLM/main/L0_MergeRequest_PR pipeline #43502 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-06-17T11:13:03Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-17T11:19:31Z

PR_Github #54841 [ run ] triggered by Bot. Commit: d1b85f3 Link to invocation

tensorrt-cicd · 2026-06-17T17:53:10Z

PR_Github #54841 [ run ] completed with state SUCCESS. Commit: d1b85f3
/LLM/main/L0_MergeRequest_PR pipeline #43853 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

yuxianq · 2026-06-18T02:11:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-18T02:17:25Z

PR_Github #54897 [ run ] triggered by Bot. Commit: e525195 Link to invocation

tensorrt-cicd · 2026-06-18T09:09:36Z

PR_Github #54897 [ run ] completed with state SUCCESS. Commit: e525195
/LLM/main/L0_MergeRequest_PR pipeline #43903 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-06-18T10:59:48Z

/bot run --disable-fail-fast

yuxianq · 2026-06-18T11:38:57Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-18T11:44:21Z

PR_Github #54928 [ run ] triggered by Bot. Commit: a5a2759 Link to invocation

yuxianq · 2026-07-01T04:39:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-01T04:45:30Z

PR_Github #56831 [ run ] triggered by Bot. Commit: 0802c34 Link to invocation

tensorrt-cicd · 2026-07-01T12:17:34Z

PR_Github #56831 [ run ] completed with state SUCCESS. Commit: 0802c34
/LLM/main/L0_MergeRequest_PR pipeline #45645 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-07-01T13:23:51Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-01T13:29:42Z

PR_Github #56934 [ run ] triggered by Bot. Commit: 0802c34 Link to invocation

tensorrt-cicd · 2026-07-01T17:43:31Z

PR_Github #56934 [ run ] completed with state SUCCESS. Commit: 0802c34
/LLM/main/L0_MergeRequest_PR pipeline #45739 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-07-02T06:36:17Z

/bot run --disable-fail-fast

yuxianq · 2026-07-02T06:45:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-02T06:52:19Z

PR_Github #57151 [ run ] triggered by Bot. Commit: 760357f Link to invocation

tensorrt-cicd · 2026-07-02T14:36:11Z

PR_Github #57151 [ run ] completed with state SUCCESS. Commit: 760357f
/LLM/main/L0_MergeRequest_PR pipeline #45931 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

yuxianq · 2026-07-03T03:02:19Z

/bot skip --comment "This PR adds new tests to CI, but the failed tests in CI are unrelated to this PR, skip CI"

tensorrt-cicd · 2026-07-03T03:07:59Z

PR_Github #57327 [ skip ] triggered by Bot. Commit: f63b2e3 Link to invocation

tensorrt-cicd · 2026-07-03T03:14:25Z

PR_Github #57327 [ skip ] completed with state SUCCESS. Commit: f63b2e3
Skipping testing for commit f63b2e3

Link to invocation

@staticmethod

…k fake PR NVIDIA#15368 promoted _is_stats_dummy_request from a closure to a @staticmethod on PyExecutor. Production callsites still invoke it as self._is_stats_dummy_request(req), which resolves correctly on a real PyExecutor instance via the descriptor protocol. The unit tests in tests/unittest/pyexecutor/test_iter_stats_populate.py call PyExecutor._update_iter_stats unbound against a MagicMock fake self. On a bare MagicMock, self._is_stats_dummy_request auto-resolves to a child MagicMock whose call returns yet another (truthy) MagicMock, so every request is filtered as a dummy and the KV-token/count counters (num_ctx_kv_tokens, num_gen_kv_tokens, num_paused_kv_tokens, and under attention-DP also num_context_requests/num_gen_requests/num_paused_requests) collapse to zero — failing test_dummy_filtering_on_kv_token_fields and test_attention_dp_dummy_filtering_on_count_fields. Bind the real PyExecutor._is_stats_dummy_request static method onto the fake in _build_fake_self so the production filter dispatches to the real predicate (which reads req.is_dummy, a real @Property on _StubRequest). Test-only change; no production code is modified. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>

@staticmethod

…k fake PR NVIDIA#15368 promoted _is_stats_dummy_request from a closure to a @staticmethod on PyExecutor. Production callsites still invoke it as self._is_stats_dummy_request(req), which resolves correctly on a real PyExecutor instance via the descriptor protocol. The unit tests in tests/unittest/pyexecutor/test_iter_stats_populate.py call PyExecutor._update_iter_stats unbound against a MagicMock fake self. On a bare MagicMock, self._is_stats_dummy_request auto-resolves to a child MagicMock whose call returns yet another (truthy) MagicMock, so every request is filtered as a dummy and the KV-token/count counters (num_ctx_kv_tokens, num_gen_kv_tokens, num_paused_kv_tokens, and under attention-DP also num_context_requests/num_gen_requests/num_paused_requests) collapse to zero — failing test_dummy_filtering_on_kv_token_fields and test_attention_dp_dummy_filtering_on_count_fields. Bind the real PyExecutor._is_stats_dummy_request static method onto the fake in _build_fake_self so the production filter dispatches to the real predicate (which reads req.is_dummy, a real @Property on _StubRequest). Test-only change; no production code is modified. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

yuxianq requested a review from a team as a code owner June 15, 2026 07:44

yuxianq requested a review from a team June 15, 2026 07:44

yuxianq requested a review from a team as a code owner June 15, 2026 07:44

yuxianq requested review from a team, mzweilz and yiqingy0 June 15, 2026 07:44

github-actions Bot assigned yuxianq Jun 15, 2026

yuxianq force-pushed the infra/unit-test-ci-coverage branch from a851455 to c843f4d Compare June 15, 2026 07:48

yuxianq requested a review from QiJune June 15, 2026 08:21

yuxianq force-pushed the infra/unit-test-ci-coverage branch from c843f4d to 42a6f27 Compare June 16, 2026 02:49

yuxianq mentioned this pull request Jun 17, 2026

[TRTLLM-13490][feat] Support cross-attention with FlashInfer TRTLLM-Gen kernels on Blackwell #15429

Merged

1 task

yuxianq force-pushed the infra/unit-test-ci-coverage branch from fcb958f to d1b85f3 Compare June 17, 2026 11:03

yuxianq force-pushed the infra/unit-test-ci-coverage branch from 084d6fa to a5a2759 Compare June 18, 2026 11:38

yuxianq force-pushed the infra/unit-test-ci-coverage branch from 8e6de36 to 0802c34 Compare July 1, 2026 04:39

yuxianq force-pushed the infra/unit-test-ci-coverage branch from 6d18661 to 760357f Compare July 2, 2026 06:45

yuxianq added 6 commits July 3, 2026 02:48

[None][infra] Improve unit test CI coverage

78c5743

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][infra] waive newly covered unit tests

a30189e

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][infra] use pytest skips for unit test waives

8db7dd5

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][infra] skip new unit coverage failures

7d6afce

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][infra] move scaffolding unit tests to H100

8fbe430

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][test] Skip flaky MCP worker echo test

6f367d2

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

yuxianq force-pushed the infra/unit-test-ci-coverage branch from 760357f to f63b2e3 Compare July 3, 2026 02:48

yuxianq added 2 commits July 3, 2026 02:48

[None][test] Skip flaky MCP worker multi-call test

5c2c8e5

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][test] Skip flaky MCP same-tool test

f63b2e3

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

yuxianq merged commit fb0d68b into NVIDIA:main Jul 3, 2026
7 checks passed

YihuiLu512 mentioned this pull request Jul 3, 2026

[https://nvbugs/6337231][fix] Unskip and fix iter-stats unit tests' fake-self predicate #15927

Merged

1 task

chienchunhung mentioned this pull request Jul 6, 2026

[https://nvbugs/6337235][test] Remove stale MX/GMS model loader skips #15987

Merged

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[None][infra] Improve unit test CI coverage (NVIDIA#15368)

f55f718

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Uh oh!

Conversation

yuxianq commented Jun 15, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

yuxianq commented Jun 15, 2026

Uh oh!

coderabbitai Bot commented Jun 15, 2026

Walkthrough

Changes

Estimated code review effort

Uh oh!

yuxianq commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

yuxianq commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

yuxianq commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

yuxianq commented Jun 18, 2026

Uh oh!

tensorrt-cicd commented Jun 18, 2026

Uh oh!

tensorrt-cicd commented Jun 18, 2026

Uh oh!

yuxianq commented Jun 18, 2026

Uh oh!

yuxianq commented Jun 18, 2026

Uh oh!

tensorrt-cicd commented Jun 18, 2026

Uh oh!

yuxianq commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

yuxianq commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

yuxianq commented Jul 2, 2026

Uh oh!

yuxianq commented Jul 2, 2026

Uh oh!

tensorrt-cicd commented Jul 2, 2026

Uh oh!

tensorrt-cicd commented Jul 2, 2026

Uh oh!

yuxianq commented Jul 3, 2026

Uh oh!

tensorrt-cicd commented Jul 3, 2026

Uh oh!

tensorrt-cicd commented Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

yuxianq commented Jun 15, 2026 •

edited by coderabbitai Bot

Loading