[TRTLLM-12893][infra] Parallelize post stages: Rerun Report, Test Coverage, and AI Failure Analysis by ZhanruiSunCh · Pull Request #14528 · NVIDIA/TensorRT-LLM

ZhanruiSunCh · 2026-05-25T09:29:39Z

Summary by CodeRabbit

Refactor
- Restructured build pipeline test result processing workflow for improved efficiency.
- Implemented parallel execution of test coverage reporting, test rerun analysis, and failure diagnostics for faster feedback cycles.
- Pre-installed shared dependencies to optimize build performance.
- Added build information artifact for enhanced build diagnostics and troubleshooting.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

ZhanruiSunCh · 2026-05-25T09:29:54Z

/bot run

coderabbitai · 2026-05-25T09:32:55Z

📝 Walkthrough

Walkthrough

The pipeline refactors test result collection by updating collectTestResults to accept globalVars, reorganizing test handling to parallelize report generation, conditionally include AI failure analysis on build failures, and update the caller to pass the new parameter through post.always.

Changes

Test Result Collection and Processing Refactoring

Layer / File(s)	Summary
collectTestResults function signature and globalVars parameter `jenkins/L0_MergeRequest.groovy`	`collectTestResults` signature updated to accept `globalVars` as a third parameter.
Test result handling and parallel task orchestration `jenkins/L0_MergeRequest.groovy`	Test result processing is refactored to pre-install dependencies, upload build info, parallelize rerun-report and test-coverage tasks, and conditionally execute AI failure analysis when the build has failed.
Pipeline post.always integration with updated collectTestResults `jenkins/L0_MergeRequest.groovy`	The pipeline's `post.always` block is updated to call `collectTestResults(this, testFilter, globalVars)` with the new parameter signature.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

tburt-nv
mzweilz
yuanjingx87

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description body is empty—only the template structure remains without any content in the Description or Test Coverage sections.	Add a clear description explaining the issue, solution, and test coverage. At minimum, describe why parallelization is needed and what tests validate the changes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the JIRA ticket, type, and main change: parallelizing post stages (Rerun Report, Test Coverage, AI Failure Analysis) in the pipeline.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

🧹 Nitpick comments (1)

jenkins/L0_MergeRequest.groovy (1)

1257-1259: 💤 Low value

Consider logging the exception for debuggability.

The empty catch block silently swallows all exceptions. While the best-effort approach is appropriate, adding minimal logging would help diagnose issues when they occur.

Suggested improvement

                     } catch (Exception e) {
-                        // Analysis is best-effort; do not fail the pipeline
+                        // Analysis is best-effort; do not fail the pipeline
+                        echo "AI Failure Analysis skipped due to error: ${e.toString()}"
                     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@jenkins/L0_MergeRequest.groovy` around lines 1257 - 1259, The empty catch
(Exception e) block silently swallows failures from the analysis try/catch;
update that catch to log the exception (message and stacktrace) for
debuggability. Inside the catch referencing the existing catch variable e, call
the pipeline logger if available (e.g., logger.error(...) or
currentBuild/steps.echo) or fallback to println, and include
e.toString()/e.getMessage() plus the stacktrace (e.printStackTrace() or logger
with throwable) so the analysis failure is visible without failing the pipeline.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@jenkins/L0_MergeRequest.groovy`:
- Around line 1257-1259: The empty catch (Exception e) block silently swallows
failures from the analysis try/catch; update that catch to log the exception
(message and stacktrace) for debuggability. Inside the catch referencing the
existing catch variable e, call the pipeline logger if available (e.g.,
logger.error(...) or currentBuild/steps.echo) or fallback to println, and
include e.toString()/e.getMessage() plus the stacktrace (e.printStackTrace() or
logger with throwable) so the analysis failure is visible without failing the
pipeline.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e856f02e-cf40-4698-aea7-a11a1aa0c116

📥 Commits

Reviewing files that changed from the base of the PR and between 2e3a75c and 31f3ab0.

📒 Files selected for processing (1)

jenkins/L0_MergeRequest.groovy

tensorrt-cicd · 2026-05-25T09:36:06Z

PR_Github #50198 [ run ] triggered by Bot. Commit: 31f3ab0 Link to invocation

ZhanruiSunCh · 2026-05-25T09:38:05Z

/bot run

tensorrt-cicd · 2026-05-25T09:44:25Z

PR_Github #50201 [ run ] triggered by Bot. Commit: 9559d3f Link to invocation

tensorrt-cicd · 2026-05-25T09:47:54Z

PR_Github #50198 [ run ] completed with state ABORTED. Commit: 31f3ab0

Link to invocation

tensorrt-cicd · 2026-05-25T14:00:51Z

PR_Github #50201 [ run ] completed with state SUCCESS. Commit: 9559d3f
/LLM/main/L0_MergeRequest_PR pipeline #39740 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

…erage, and AI Failure Analysis Restructure the post block in L0_MergeRequest to run Rerun Report, Test Coverage, and AI Failure Analysis in parallel after Collect Test Result and Upload Build Info complete serially. This reduces post stage wall time by ~3.5min on failed builds with coverage data. - Refactor collectTestResults to run parallel tasks inside the pod - Move Upload Build Info into collectTestResults (runs 2nd, serial) - Move AI Failure Analysis from failure block into always block, conditionally triggered via currentBuild.currentResult - Remove container("alpine") wrapper (already in alpine pod) - Pre-install shared dependencies (py3-pip) in serial phase to avoid apk concurrent install conflicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>

Add 10-min timeout to each parallel branch (Rerun Report, Test Coverage, AI Failure Analysis) with outer try-catch to ensure timeout or failure in any branch does not affect the pipeline result. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>

ZhanruiSunCh · 2026-06-03T08:30:27Z

/bot run

tensorrt-cicd · 2026-06-03T08:36:38Z

PR_Github #51805 [ run ] triggered by Bot. Commit: fb4427e Link to invocation

tensorrt-cicd · 2026-06-03T09:31:42Z

PR_Github #51805 [ run ] completed with state SUCCESS. Commit: fb4427e
/LLM/main/L0_MergeRequest_PR pipeline #41168 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

ZhanruiSunCh · 2026-06-04T02:42:18Z

/bot run

tensorrt-cicd · 2026-06-04T02:47:47Z

PR_Github #51944 [ run ] triggered by Bot. Commit: fb4427e Link to invocation

tensorrt-cicd · 2026-06-04T05:34:29Z

PR_Github #51944 [ run ] completed with state FAILURE. Commit: fb4427e
/LLM/main/L0_MergeRequest_PR pipeline #41295 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

ZhanruiSunCh · 2026-06-05T10:28:27Z

/bot skip --comment "Failed not related this PR change."

tensorrt-cicd · 2026-06-05T10:34:52Z

PR_Github #52331 [ skip ] triggered by Bot. Commit: fb4427e Link to invocation

tensorrt-cicd · 2026-06-05T10:40:50Z

PR_Github #52331 [ skip ] completed with state SUCCESS. Commit: fb4427e
Skipping testing for commit fb4427e

Link to invocation

…erage, and AI Failure Analysis (NVIDIA#14528) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: NVFB <186336021+NVFB@users.noreply.github.com>

…erage, and AI Failure Analysis (NVIDIA#14528) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ZhanruiSunCh requested a review from a team May 25, 2026 09:29

ZhanruiSunCh requested a review from a team as a code owner May 25, 2026 09:29

ZhanruiSunCh requested review from tburt-nv and yuanjingx87 May 25, 2026 09:29

github-actions Bot assigned ZhanruiSunCh May 25, 2026

coderabbitai Bot reviewed May 25, 2026

View reviewed changes

ZhanruiSunCh force-pushed the user/zhanruis/parallelize_post_stages branch from 31f3ab0 to 9559d3f Compare May 25, 2026 09:37

ZhanruiSunCh requested review from EmmaQiaoCh, chzblych, niukuo and yiqingy0 May 28, 2026 09:36

tburt-nv approved these changes Jun 1, 2026

View reviewed changes

Comment thread jenkins/L0_MergeRequest.groovy

ZhanruiSunCh force-pushed the user/zhanruis/parallelize_post_stages branch from 9559d3f to 5478a96 Compare June 3, 2026 08:09

ZhanruiSunCh and others added 2 commits June 3, 2026 01:22

ZhanruiSunCh force-pushed the user/zhanruis/parallelize_post_stages branch from 5478a96 to fb4427e Compare June 3, 2026 08:28

ZhanruiSunCh enabled auto-merge (squash) June 3, 2026 08:30

ZhanruiSunCh merged commit 6387eac into NVIDIA:main Jun 5, 2026
8 checks passed

Uh oh!

Conversation

ZhanruiSunCh commented May 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

ZhanruiSunCh commented May 25, 2026

Uh oh!

coderabbitai Bot commented May 25, 2026

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

ZhanruiSunCh commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

Uh oh!

ZhanruiSunCh commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

ZhanruiSunCh commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

ZhanruiSunCh commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ZhanruiSunCh commented May 25, 2026 •

edited by coderabbitai Bot

Loading