[None][fix] Cherry-pick CBTS-fanout #15592+#15614 onto 2806 HEAD to bisect-confirm 2810 perf-sanity aborts - #15724
Conversation
📝 WalkthroughWalkthrough
ChangesCBTS Shard Collapse and Artifact Download
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/L0_Test.groovy`:
- Around line 1706-1727: _cbtsMaybeCollapseSplits is collapsing shards based
only on CBTS metadata, but renderTestDB may still fall back to the full source
test-db if the narrowed artifact/override is unusable. Update the split-collapse
logic to rely on a success/usable signal from renderTestDB (or another
confirmed-load flag) before skipping extra shards; keep the default splits in
_cbtsMaybeCollapseSplits unless the narrowed DB is actually available. Reference
the CBTS_RESULT lookup and the renderTestDB flow when wiring this check so the
shard count only shrinks after the narrowed test-db is confirmed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 62de761b-5f0e-47fc-b886-83a01703d54b
📒 Files selected for processing (1)
jenkins/L0_Test.groovy
| // CBTS Layer 2.5: a narrowed stage runs K = min(narrowed count, default splits) shards. Returns [skip, splits, splitId]. | ||
| def _cbtsMaybeCollapseSplits(stageName, splitId, splits) { | ||
| def cbts = testFilter[(CBTS_RESULT)] | ||
| def counts = cbts?.affected_stage_test_counts | ||
| if (!counts) { | ||
| // No CBTS or tarball not uploaded (renderTestDB then runs full source) -> keep default splits. | ||
| if (cbts == null || !cbts.cbts_test_db_artifact_path) { | ||
| return [skip: false, splits: splits, splitId: splitId] | ||
| } | ||
| def count = counts[stageName] | ||
| if (count == null || count >= 20) { | ||
| def counts = cbts.affected_stage_test_counts | ||
| def count = counts ? counts[stageName] : null | ||
| if (count == null) { | ||
| return [skip: false, splits: splits, splitId: splitId] | ||
| } | ||
| if (splitId > 1) { | ||
| echo "CBTS [${cbts.scope}]: ${stageName} narrowed to ${count} (< 20), skipping group ${splitId}/${splits}" | ||
| return [skip: true, splits: splits, splitId: splitId] | ||
| int k = Math.max(1, Math.min(count as int, splits as int)) | ||
| if (k >= splits) { | ||
| return [skip: false, splits: splits, splitId: splitId] | ||
| } | ||
| if (splits > 1) { | ||
| echo "CBTS [${cbts.scope}]: ${stageName} narrowed to ${count} (< 20), collapsing splits=${splits} → 1" | ||
| if (splitId > k) { | ||
| echo "CBTS [${cbts.scope}]: ${stageName} narrowed to ${count} tests → ${k}/${splits} shards, skipping group ${splitId}" | ||
| return [skip: true, splits: splits, splitId: splitId] | ||
| } | ||
| return [skip: false, splits: 1, splitId: 1] | ||
| return [skip: false, splits: k, splitId: splitId] | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify how affected_stage_test_counts and cbts_test_db_artifact_path are produced,
# and whether they are guaranteed consistent with the narrowed test-db presence.
fd -e groovy | xargs rg -nP -C3 'affected_stage_test_counts|cbts_test_db_artifact_path|test_db_dir_override|getCbtsResult'Repository: NVIDIA/TensorRT-LLM
Length of output: 8222
Do not reduce shard count until the narrowed test-db is actually used _cbtsMaybeCollapseSplits skips extra shards based on CBTS metadata, but renderTestDB can still fall back to the full source test-db when the artifact download/extraction fails or the override YAML is missing/empty. That can silently drop tests; keep the default split count unless the narrowed DB is confirmed usable, or propagate a success signal back before collapsing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@jenkins/L0_Test.groovy` around lines 1706 - 1727, _cbtsMaybeCollapseSplits is
collapsing shards based only on CBTS metadata, but renderTestDB may still fall
back to the full source test-db if the narrowed artifact/override is unusable.
Update the split-collapse logic to rely on a success/usable signal from
renderTestDB (or another confirmed-load flag) before skipping extra shards; keep
the default splits in _cbtsMaybeCollapseSplits unless the narrowed DB is
actually available. Reference the CBTS_RESULT lookup and the renderTestDB flow
when wiring this check so the shard count only shrinks after the narrowed
test-db is confirmed.
392ff0d to
a4c64ee
Compare
chenfeiz0326
changed the title
|
/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE4-GPU16-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-2,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-3,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-4,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-6,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-7,GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-8,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-7,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-7,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-8,GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1" |
|
/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-6,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-8,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-2" |
a4c64ee to
4ab9bc5
Compare
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
|
/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-6,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-8,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-2" |
1 participant
This PR is a bisect-confirmation build only — do not merge.
Context
In post-merge
L0_Test-SBSA-Multi-GPUbuild 2810 (commitc25c23f7), 26 GB200 perf-sanity stages aborted with "Submit Test Result" and "Clean Up Slurm Resource" SKIPPED. The abort rate stepped up between builds 2806 (a51931ad) and 2809 (86bb1f46). The hypothesis is that the CBTS-fanout commit pair7cc568c4ee(PR #15592) +86bb1f4612(PR #15614) is the culprit: it changed CBTS Layer 2.5 from "collapse narrowed stage to 1 shard" to "k = min(count, splits) shards" then to duration-basedk, multiplying simultaneous SSH-probe load on OCI HSG head nodes insideuploadResults's freshwithSlurmSshCredentialssession.What this branch does
Branch base:
a51931ad2f(2806 HEAD).Cherry-picks (in merge order):
7cc568c4ee—[None][infra] use default split when CBTS test-db download fails (#15592)86bb1f4612—[None][infra] take test durations into account to determine cbts splits num (#15614)The branch is therefore 2806 state + the two suspected CBTS-fanout commits and nothing else from 2807/2808/2809. This isolates the pair's effect.
Expected verification result
Trigger the 2810 aborted stage list on this PR. If the CBTS-fanout pair is the culprit, the abort symptom should appear at 2809-level rate. If aborts stay at 2806-level, the regression involves another commit in the range.
Paired with companion PR #15723 (revert the same two commits on 2809 HEAD) for the bisect-confirmation in the opposite direction.