[None][infra] use default split when CBTS test-db download fails

[crazydemo]

Summary by CodeRabbit

• New Features
  • Improved test sharding behavior for smaller filtered test sets, helping reduce unnecessary split groups and avoid wasted scheduling.
  • Added smarter fallback handling so jobs keep the expected shard layout when narrowed test data isn’t available.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR refactors CBTS test-sharding split-collapse into two phases: a caller-side early-skip (_cbtsShouldSkipSplitGroup) before SLURM allocation, and an executor-side collapse (_cbtsEffectiveSplits) conditioned on whether renderTestDB confirms the narrowed CBTS test-db was actually loaded. renderTestDB now returns a {path, usedCbtsOverride} map, and a new CBTS_SPLIT_COLLAPSE_THRESHOLD field (default 20) controls the threshold. README documentation is updated to match.

Changes

CBTS Two-Phase Split-Collapse Refactor

Layer / File(s)	Summary
renderTestDB return type and CBTS override tracking jenkins/L0_Test.groovy	renderTestDB now tracks usedCbtsOverride to record whether the narrowed CBTS test-db directory was valid and loaded, adds a CBTS_SPLIT_COLLAPSE_THRESHOLD @Field configurable via env (default 20), and returns {path: testList, usedCbtsOverride: boolean} instead of a raw path string. CI-budget estimate labels are also branched on usedCbtsOverride.
New CBTS Layer 2.5 helpers replacing _cbtsMaybeCollapseSplits jenkins/L0_Test.groovy	Removes _cbtsMaybeCollapseSplits and introduces three replacements: _cbtsNarrowedBelowThreshold checks the narrowed count against threshold; _cbtsShouldSkipSplitGroup provides caller-side early-return for splitId > 1 groups before allocation; _cbtsEffectiveSplits collapses to splits=1, splitId=1 on the executor only when usedCbtsOverride is true, preserving default splits otherwise to avoid collapse-then-fallback timeouts. runLLMTestlistOnSlurm is updated to call _cbtsShouldSkipSplitGroup.
Call-site integration in SLURM and platform runners jenkins/L0_Test.groovy	runLLMTestlistWithSbatch, runLLMTestlistOnPlatformImpl, and runLLMTestlistOnPlatform unpack the new renderTestDB map result and pass renderResult.usedCbtsOverride into _cbtsEffectiveSplits, replacing all former _cbtsMaybeCollapseSplits call sites.
README two-phase collapse and fallback documentation jenkins/scripts/cbts/README.md	Rewrites the "Split-collapse heuristic (Layer 2.5)" section to describe both phases, updates the summary table to reflect the renderTestDB-gated rule, and expands the "Fallback paths" entry to specify that a failed cbts_test_db tarball reverts the surviving stage to default splits (running only ~1/N of the source list).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

• niukuo
• yuanjingx87
• dpitman-nvda

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR body is still just the template with placeholders and no actual description or test coverage details.	Fill in the Description and Test Coverage sections with the issue, solution, and relevant tests; keep checklist items as applicable.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main change and clearly describes the CBTS fallback behavior fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

jenkins/L0_Test.groovy (1)

2834-2837: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Do not swallow pipeline interruptions during CBTS artifact download.

The generic catch (Exception e) also catches abort/timeout interruption paths used elsewhere in this file as hard stops. Re-throw InterruptedException before falling back to source.

Proposed fix

-            } catch (Exception e) {
+            } catch (InterruptedException e) {
+                throw e
+            } catch (Exception e) {
                 echo "CBTS Layer 3: artifact download failed " +
                      "(${e.class.simpleName}: ${e.message}); falling back to source test-db"
             }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@jenkins/L0_Test.groovy` around lines 2834 - 2837, The CBTS artifact download
fallback in the surrounding try/catch is swallowing pipeline interruption
signals because the generic catch on the artifact download path also captures
hard-stop cases; update the exception handling in the CBTS Layer 3 download
block to detect InterruptedException and re-throw it immediately before any
fallback logic, while keeping the existing source test-db fallback only for
non-interruption failures. Use the existing artifact download and fallback code
in L0_Test.groovy as the target, and preserve the current logging for ordinary
exceptions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@jenkins/L0_Test.groovy`:
- Around line 2828-2832: The current download guard in the Jenkins flow only
checks the override directory, so a stale cbts_test_db/ can prevent re-download
even when the expected context YAML is missing. Update the logic around the
dirExists check and the subsequent artifact extraction in this block to verify
the presence of ${testContext}.yml inside the extracted context first; if it is
missing, remove the partial directory and re-download/re-extract the artifact
instead of falling back to source.
- Around line 1842-1844: The CBTS split collapse threshold is being initialized
too early from env, but it should stay at the default value at declaration time.
Keep CBTS_SPLIT_COLLAPSE_THRESHOLD as a plain `@Field` default of 20 in
L0_Test.groovy, and move the env.CBTS_SPLIT_COLLAPSE_THRESHOLD override logic
into the Setup Environment step so the value is read after startup.

---

Outside diff comments:
In `@jenkins/L0_Test.groovy`:
- Around line 2834-2837: The CBTS artifact download fallback in the surrounding
try/catch is swallowing pipeline interruption signals because the generic catch
on the artifact download path also captures hard-stop cases; update the
exception handling in the CBTS Layer 3 download block to detect
InterruptedException and re-throw it immediately before any fallback logic,
while keeping the existing source test-db fallback only for non-interruption
failures. Use the existing artifact download and fallback code in L0_Test.groovy
as the target, and preserve the current logging for ordinary exceptions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4c35d53d-b836-4af6-ad5e-776220f792a6

📥 Commits

Reviewing files that changed from the base of the PR and between 71613f9 and 4c1c73a.

📒 Files selected for processing (2)

• jenkins/L0_Test.groovy
• jenkins/scripts/cbts/README.md

[crazydemo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #55507 [ run ] triggered by Bot. Commit: a10f3be Link to invocation

[tensorrt-cicd]

PR_Github #55507 [ run ] completed with state FAILURE. Commit: a10f3be
/LLM/main/L0_MergeRequest_PR pipeline #44431 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[crazydemo]

/bot skip --comment "the failed stages is not related to this PR"

[tensorrt-cicd]

PR_Github #55661 [ skip ] triggered by Bot. Commit: a10f3be Link to invocation

[tensorrt-cicd]

PR_Github #55661 [ skip ] completed with state SUCCESS. Commit: a10f3be
Skipping testing for commit a10f3be

Link to invocation