[None][test] Promote disagg perf sanity tests to pre-merge for functional verification by chenfeiz0326 · Pull Request #16351 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-07-14T03:21:19Z

Summary

Adds pre-merge functional coverage for disaggregated perf-sanity so that functional regressions on the disagg path are caught before merge instead of only at post-merge. Today all disagg gen_only / e2e stages are post-merge-only.

Selection rule for the 4 picks: every pre-merge test is one that is already actively running in L0 CI post-merge on origin/main. No QA-lane borrowing, no uncommenting of previously-disabled lines, no new config yamls, no first-time-in-L0 combinations. Each pick physically moves from a post-merge block to a pre-merge block of the same test-db yml, and the post-merge testCount is decremented so nothing runs twice.

Coverage matrix (4 tests, all at 8k1k)

#	Model	ISL	Concurrency	Gen Par	Mode	MTP	GPU stage
1	gpt-oss-120b-fp4	8k1k	large (con1024)	tp4	e2e	✗	GB200 8 GPUs / 2 nodes (ctx1_gpu1 gen1_gpu4)
2	kimi-k25-thinking-fp4	8k1k	small (con4)	tep8	gen_only	✓	GB300 12 GPUs / 3 nodes (ctx1_gpu4 gen1_gpu8)
3	glm-5-fp4	8k1k	large (con1024)	dep8	e2e	✓	GB300 12 GPUs / 3 nodes (ctx1_gpu2 gen1_gpu8)
4	deepseek-r1-fp4	8k1k	large (con1536)	dep8	gen_only	✓	B200 x86 16 GPUs / 2 nodes (ctx1_gpu4 gen1_gpu8)

3 GPU families (B200 / GB200 / GB300); both e2e and gen_only modes; large + small concurrency; dep / tep / tp gen-worker parallelism; MTP0 / MTP1 / MTP3.

Not covered at pre-merge (by design) — EPLB-only configs (smallest is 24 GPUs / 6 nodes, over budget); non-NIXL transports (already exercised by H200 hopper stages); DSV32 (per chzblych's review: deprioritized); PP variants (all PP disagg configs are DSR1 128k8k on topologies outside the picked slots); 32k4k / 128k8k ISL/OSL (only exist for DSV32 / DSR1 on topologies outside the picked slots — moving to them would need new Jenkins stages).

FUNCTIONAL-ONLY gate

Per review feedback (discussion), pre-merge PerfSanity stages default to failing CI on perf regression, which is too noisy for the disagg path where we only want to catch functional regressions. New convention:

Stage names containing FUNCTIONAL-ONLY run the full perf harness (numbers still upload to OpenSearch, dashboards still update) but do not fail CI on perf regression — identical to post-merge behavior. Only functional failures (build errors, crashes, empty output) fail the stage. All pre-merge stages added by this PR use the FUNCTIONAL-ONLY marker.

Implementation: test_perf_sanity.py reads os.environ["stageName"]; when it contains FUNCTIONAL-ONLY it passes fail_on_regression=False to process_and_upload_test_results(). No changes to perf_regression_utils.py — the shared pipeline stays test-agnostic.

Full list of pre-merge tests + stage names

Every stage runs one test (testCount = 1).

#	Test	Stage
1	`disagg_upload-e2e-gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL`	`GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-1`
2	`disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con4_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL`	`GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-1`
3	`disagg_upload-e2e-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL`	`GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-1`
4	`disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL`	`DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-1`

Files changed

Test-db (tests/integration/test_lists/test-db/l0_*_multi_nodes_perf_sanity_*.yml) — 4 files gain a pre_merge block; the same test line is removed from the file's post_merge block so nothing double-runs. system_gpu_count, wildcards.gpu, and TIMEOUT values preserved.

File	pre_merge count	post_merge (main → this PR)
`l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml`	1 (gpt-oss 8k1k con1024 e2e tp4)	7 → 6
`l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.yml`	1 (kimi 8k1k con4 tep8 mtp3 gen_only)	5 → 4
`l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu2_gen1_node2_gpu8.yml`	1 (glm-5 8k1k con1024 e2e dep8)	5 → 4
`l0_b200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu8.yml`	1 (DSR1 8k1k con1536 gen_only dep8)	6 → 5

jenkins/L0_Test.groovy — new pre-merge buildStageConfigs entries added (one FUNCTIONAL-ONLY stage per new pre-merge yml block); matching post-merge testCounts decremented so tests are not double-executed.

tests/integration/defs/perf/test_perf_sanity.py — reads stageName env var; passes fail_on_regression=False to the regression pipeline when the name contains FUNCTIONAL-ONLY. Otherwise unchanged.

tests/integration/defs/perf/README_test_perf_sanity.md — new "Pre-merge vs Post-merge Perf-Regression Gating" section documenting the FUNCTIONAL-ONLY stage-name convention.

tests/integration/test_lists/waives.txt — no changes.

Test plan

test_perf_sanity.py syntax check
CI pipeline picks up all 4 new pre-merge disagg FUNCTIONAL-ONLY stages (B200 + GB200 + 2 × GB300)
FUNCTIONAL-ONLY stages report SUCCESS on perf regression, FAIL on functional error
Post-merge stages still run with reduced counts (no duplicate execution of the moved tests)
No pytest collection errors from any of the edited test-db files
Every pre-merge test was already active in L0 CI post-merge on origin/main (no first-time-in-L0 combinations)
Each pre-merge stage has testCount = 1 in groovy matching the pre_merge count in its yml

Generated with Claude Code

coderabbitai · 2026-07-14T03:25:38Z

📝 Walkthrough

Walkthrough

Perf sanity pipeline definitions and YAML test matrices redistribute selected B200, GB200, and GB300 disaggregated tests between pre-merge and post-merge stages, while adjusting generated test split counts.

Changes

Perf sanity stage split

Layer / File(s)	Summary
Jenkins stage definitions `jenkins/L0_Test.groovy`	Adds B200 and GB200 pre-merge disaggregated stages, reduces related post-merge test counts, and renames a GB300 stage from post-merge to pre-merge.
Test matrix stage alignment `tests/integration/test_lists/test-db/l0__multi_nodes_perf_sanity_.yml`	Moves selected B200 and GB200 cases into pre-merge blocks, removes them from post-merge lists, and changes the GB300 matrix stage to pre-merge.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: promoting disaggregated perf-sanity tests to pre-merge for functional verification.
Description check	✅ Passed	The description covers summary, test coverage, file changes, gating behavior, and checklist items, matching the template well.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

chenfeiz0326 · 2026-07-14T05:43:40Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-2,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU2-GEN1-NODE1-GPU4-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-1"

tensorrt-cicd · 2026-07-14T05:49:27Z

PR_Github #59118 [ run ] triggered by Bot. Commit: 3f1b5f3 Link to invocation

tensorrt-cicd · 2026-07-14T07:13:21Z

PR_Github #59118 [ run ] completed with state SUCCESS. Commit: 3f1b5f3
/LLM/main/L0_MergeRequest_PR pipeline #47629 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-07-14T08:36:17Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-4,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-1"

tensorrt-cicd · 2026-07-14T08:42:11Z

PR_Github #59157 [ run ] triggered by Bot. Commit: a0c0cdd Link to invocation

chenfeiz0326 · 2026-07-14T08:56:10Z

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-4,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-FUNCTIONAL-ONLY-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-1"

tensorrt-cicd · 2026-07-14T09:01:36Z

PR_Github #59162 [ run ] triggered by Bot. Commit: a0c0cdd Link to invocation

tensorrt-cicd · 2026-07-14T09:06:52Z

PR_Github #59157 [ run ] completed with state ABORTED. Commit: a0c0cdd

Link to invocation

Shixiaowei02 · 2026-07-14T09:38:36Z

Thanks for the careful updates! I have no concerns about the test methods. Given the limited GPU resources available, it may be worth discussing what we can do to further reduce resource consumption. For example, could we consider using smaller models where appropriate, or reusing the same runs for both correctness validation and perf-sanity in the pre-merge workflow to avoid duplicate resource usage?

tensorrt-cicd · 2026-07-14T10:14:03Z

PR_Github #59162 [ run ] completed with state SUCCESS. Commit: a0c0cdd
/LLM/main/L0_MergeRequest_PR pipeline #47669 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-07-15T02:44:07Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-15T02:49:47Z

PR_Github #59338 [ run ] triggered by Bot. Commit: a0c0cdd Link to invocation

fredricz-20070104

Approved.

fredricz-20070104 · 2026-07-15T05:49:09Z

Execution stage & CI-gating summary

For the record, here's how the two groups of disagg cases behave after this change:

Category	Stage	Perf regression	Functional failure (build error / crash / empty output)
FUNCTIONAL-ONLY	pre-merge	warn only, does not fail CI (explicit `fail_on_regression=False`)	fails CI, blocks merge
Regular disagg (kept in post)	post-merge	warn only, does not fail CI (auto-detect)	fails, but already merged — recorded only

Note that perf numbers for both groups are collected/uploaded only and never gate CI; the real differences are when they run (pre vs. post) and whether a functional failure blocks the merge.

⚠️ Cost note

This promotes 13 disagg cases (~132 GPU-hours) into the pre-merge pipeline, so that cost is paid on every PR run before merge. Please keep an eye on the added pre-merge GPU-hour budget — it's non-trivial. (The 23 cases left in post-merge add ~256 GPU-hours, but only after merge, so they don't gate PRs.)

tensorrt-cicd · 2026-07-15T06:07:06Z

PR_Github #59338 [ run ] completed with state FAILURE. Commit: a0c0cdd
/LLM/main/L0_MergeRequest_PR pipeline #47818 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mzweilz

Since all 13 cases are multi-GPU tests, which can only be triggered in pre-merge pipelines when the gatekeeper allows, I think moving them to pre-merge shouldn't have a significant impact on overall pre-merge stability or efficiency. And it seems that the FUNCTIONAL-ONLY behavior also provides a good balance between functional coverage and avoiding noisy performance gating. Approving.

Adds pre-merge coverage for disaggregated perf-sanity so that functional regressions on the disagg path are caught before merge rather than post-merge. The 8 picks span the axes we currently regress on (GPU family, model, parallelism, run mode, ISL, concurrency) while capping resource cost at 3 nodes / 12 GPUs per test: * GB200 8/2 gpt-oss-120b-fp4 1k1k con2048 tp1/dep2 gen_only+e2e * GB200 8/2 deepseek-r1-fp4 1k1k con3072 dep4/dep4 gen_only * GB200 8/2 kimi-k25-thinking-fp4 1k1k con4 dep4/tep4 gen_only * GB200 12/3 deepseek-v32-fp4 1k1k con1 dep4/tep8 gen_only * GB200 12/3 deepseek-v32-fp4 32k4k con1 dep4/tep8 gen_only (long-context) * GB300 8/2 glm-5-fp4 1k1k con1 dep2/tep4 gen_only (GB300 smoke) * B200 16/2 deepseek-r1-fp4 1k1k con2048 dep4/dep8 gen_only (x86 smoke) Each test is moved from its existing post_merge block in the corresponding l0_*_multi_nodes_perf_sanity_*.yml test-db file into a new pre_merge block; the post_merge counterparts and their Post-Merge testCount values in jenkins/L0_Test.groovy are decremented accordingly. New pre-merge Jenkins stages are added under x86SlurmTestConfigs (B200) and multiNodesSBSAConfigs (GB200/GB300). Note: gpt-oss 1k1k con2048 gen_only, deepseek-r1 1k1k con3072 gen_only, and gb300 glm-5 1k1k con1 gen_only are currently SKIPped by waives.txt (nvbugs 6324123 / 6422339 / 6418834). Those stages will run as no-ops until the underlying bugs close, at which point pre-merge coverage lights up automatically. Signed-off-by: chenfeiz0326 <chenfeiz@nvidia.com>

… stages Addresses review feedback: pre-merge PerfSanity stages default to failing CI on perf regression, which is too noisy for the newly-added pre-merge disagg coverage whose goal is functional-failure detection, not perf gating. New convention: any stage name containing "FUNCTIONAL-ONLY" runs the full perf harness (numbers still upload to OpenSearch, dashboards still update) but does not fail CI on perf regression -- identical to post-merge behavior. Only functional failures (build errors, crashes, empty output) fail the stage. Changes: * test_perf_sanity.py: pass fail_on_regression=False to the pipeline when os.environ["stageName"] contains "FUNCTIONAL-ONLY". No changes to perf_regression_utils.py so the generic pipeline stays test-agnostic. * L0_Test.groovy: rename the 5 pre-merge disagg PerfSanity stages added in the prior commit to include "-FUNCTIONAL-ONLY-" between "PerfSanity" and the CTX/GEN specifier. * README_test_perf_sanity.md: document the new stage-name flag under the Pre-merge vs Post-merge gating section. Signed-off-by: chenfeiz0326 <chenfeiz@nvidia.com>

…isagg Complements the initial 8 disagg pre-merge picks by ensuring every model has both a TEP-gen and a DEP-gen variant in pre-merge, so functional regressions in either parallelism path are caught before merge. Adds 5 tests (all under FUNCTIONAL-ONLY -- perf regressions still do not fail CI): * GB200 gpt-oss-120b-fp4 1k1k con64 tp1/tp4 gen_only (TEP) * GB200 deepseek-r1-fp4 1k1k con1 dep4/tep8 gen_only (TEP) * GB200 deepseek-v32-fp4 1k1k con2048 dep4/dep4 gen_only (DEP) * GB200 kimi-k25-thinking-fp4 1k1k con4096 dep4/dep8 gen_only (DEP) * GB300 glm-5-fp4 1k1k con4096 dep2/dep8 gen_only (DEP) Coverage per model after this commit: gpt-oss-120b-fp4 : DEP2 (gen_only+e2e) + TEP4 (gen_only) deepseek-r1-fp4 : DEP4 + DEP8 + TEP8 deepseek-v32-fp4 : DEP4 + TEP8 (1k1k) + TEP8 (32k4k) kimi-k25-thinking-fp4 : DEP8 + TEP4 glm-5-fp4 : DEP8 + TEP4 Also unwaives two tests that were only waived for perf regression noise on the shared 6422339 / 6426890 bugs (FUNCTIONAL-ONLY handles perf gating; if they hit a real functional failure the pre-merge stage will catch it): * gb200 deepseek-v32-fp4 1k1k con2048 dep4/dep4 gen_only (6426890) * gb300 glm-5-fp4 1k1k con4096 dep2/dep8 gen_only (6422339) Two new pre-merge stages added under multiNodesSBSAConfigs; three existing pre-merge stages grow their testCount; matching Post-Merge testCounts decrement so the 5 tests are not double-executed. Signed-off-by: chenfeiz0326 <chenfeiz@nvidia.com>

Moves three tests back to post_merge that were only picked up because they had existing NVBug waives against them; keeping them in pre-merge would just consume the FUNCTIONAL-ONLY escape hatch every run for suites known to fail. Leaves waives.txt as-is so their post-merge runs remain SKIPped until the underlying bugs close: * gb200 gpt-oss-120b-fp4 1k1k con2048 dep2 gen_only (nvbugs/6324123) * gb200 deepseek-r1-fp4 1k1k con3072 dep4 gen_only (nvbugs/6422339) * gb200 deepseek-v32-fp4 1k1k con1 tep8 gen_only (nvbugs unset -- was previously added by prior commit; drop from pre-merge until we're confident) Adjusts: * gpu1_gen1_node1_gpu2 pre-merge testCount 2 -> 1, post-merge 2 -> 3 * gpu4_gen1_node1_gpu4 pre-merge testCount 3 -> 2, post-merge 2 -> 3 * gpu4_gen1_node2_gpu8 pre-merge testCount 4 -> 3, post-merge 4 -> 5 Pre-merge disagg coverage after this commit: 10 tests, still spanning DEP + TEP for every model except gpt-oss (DEP dropped, TEP kept) and DSR1 (DEP4 dropped -- DEP8 on B200 x86 still covers DSR1 DEP). Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Per PR review feedback: DeepSeek V3.2 is no longer a release-priority model, so V3.2-specific regressions caught in post-merge do not block a release. Removes both V3.2 tests from the pre-merge disagg set: * gb200 deepseek-v32-fp4 1k1k con2048 dep4 gen_only * gb200 deepseek-v32-fp4 32k4k con1 tep8 gen_only testCount adjustments: * gpu4_gen1_node1_gpu4 pre-merge 2 -> 1, post-merge 3 -> 4 * gpu4_gen1_node2_gpu8 pre-merge 3 -> 2, post-merge 5 -> 6 Pre-merge disagg coverage after this commit: 8 tests, still spanning DEP + TEP for the remaining 4 models (gpt-oss, DSR1, K25, GLM-5). DSV32 coverage retained via existing post-merge stages. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

…o e2e At large concurrency, e2e exercises the full ctx/gen worker interaction and KV-cache transfer path -- more likely to catch a functional regression than gen_only which only stresses the decode side. Swaps: * gb200 kimi-k25-thinking-fp4 1k1k con4096 dep8: gen_only -> e2e * gb300 glm-5-fp4 1k1k con4096 dep8: gen_only -> e2e Both e2e counterparts already exist as post-merge tests, so this is a 1:1 stage-move (pre_merge/post_merge counts unchanged in both files; no jenkins/L0_Test.groovy testCount edits needed). B200 DSR1 con2048 (test NVIDIA#8) intentionally left as gen_only -- its e2e counterpart is commented out in the test-db and has never run; not enabling as a pre-merge blocker. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Reduces the pre-merge disagg PerfSanity set from 8 tests to 4 by moving these back to post-merge: * NVIDIA#2 gpt-oss-120b-fp4 TEP4 gen_only 1k1k con64 (GB200) * NVIDIA#4 deepseek-r1-fp4 TEP8 gen_only 1k1k con1 (GB200) * NVIDIA#5 kimi-k25-thinking-fp4 DEP8 e2e 1k1k con4096 (GB200) * NVIDIA#6 glm-5-fp4 TEP4 gen_only 1k1k con1 (GB300) Remaining 4 pre-merge FUNCTIONAL-ONLY stages: * gpt-oss-120b-fp4 DEP2 e2e 1k1k con2048 (GB200, 2n/8g) * kimi-k25-thinking-fp4 TEP4 gen_only 1k1k con4 (GB200, 2n/8g) * glm-5-fp4 DEP8 e2e 1k1k con4096 (GB300, 3n/12g) * deepseek-r1-fp4 DEP8 gen_only 1k1k con2048 (B200, 2n/16g) Corresponding Jenkins adjustments: 2 GB200 pre-merge stages deleted, 2 GB200 Post-Merge testCounts bumped, 1 GB300 pre-merge stage renamed back to Post-Merge. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-17T04:24:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T04:30:09Z

PR_Github #59873 [ run ] triggered by Bot. Commit: 4d5fca9 Link to invocation

Per review: 1k1k coverage is not important; use longer-ISL variants even though pre-merge disagg stages are functional-only. Only 8k1k variants of the four (model, parallelism, run-mode) combos have config yamls today (no 32k4k / 128k8k for these exact shapes). Swaps applied where such a yaml exists; kimi-k25-thinking TEP4 keeps its 1k1k con4 pick because no 8k1k tep4 yaml exists. Coverage matrix (4 tests, unchanged shape — just longer ISL/OSL): | Model | Coverage | |-----------------------|---------------------------------------------| | gpt-oss-120b-fp4 | DEP2 e2e (GB200, 8k1k con512) | | kimi-k25-thinking-fp4 | TEP4 gen_only (GB200, 1k1k con4, unchanged) | | glm-5-fp4 | DEP8 e2e (GB300, 8k1k con1024) | | deepseek-r1-fp4 | DEP8 gen_only (B200 x86, 8k1k con1536) | For each swapped combo the previous 1k1k pre-merge line is restored to the post_merge block so overall coverage is preserved. Every axis the reviewer flagged — small/large concurrency, dep/tep/pp, gen_only vs e2e, MTP presence — is preserved per combo. Files changed: - 3 test-db yamls: pre-merge line swapped 1k1k → 8k1k; the 1k1k line moved back into the same file's post_merge block. - 1 test-db yaml (kimi tep4): unchanged (no 8k1k tep4 config). - jenkins/L0_Test.groovy: gpt-oss post-merge testCount 3 → 4 to match the yaml now that 1k1k e2e con2048 was re-added there. Other three post-merge testCounts unchanged (swap-in-place). Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Follow-up: user relaxed the constraint — only 4 shape axes matter (small/large concurrency, dep/tep/pp, gen_only/e2e, MTP y/n), NOT GPU count / GPU family / model. That lets the kimi slot move to 8k1k after all: the kimi 8k1k con4 tep8 mtp3 gen_only config exists; it just needs 12 GPUs (3 nodes) instead of 8. Changes: - l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4.yml: drop the pre_merge block (only entry was kimi 1k1k con4 tep4); move that line into the post_merge block (restores pre-PR state). - l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.yml: add a new pre_merge block with kimi 8k1k con4 tep8 mtp3 gen_only; delete the commented-out post_merge copy of the same test (was commented so its removal doesn't change what runs anywhere). - jenkins/L0_Test.groovy: swap the kimi pre-merge FUNCTIONAL-ONLY stage from GB200-8_GPUs-2_Nodes-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4 to GB200-12_GPUs-3_Nodes-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8; bump the 8-GPU kimi post-merge testCount 4 → 5 to match the yml. Pre-merge shape matrix now: all 4 slots at 8k1k. dep(3) / tep(1); e2e(2) / gen_only(2); large-con(3) / small-con(1); MTP✓(3) / MTP✗(1). Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

tensorrt-cicd · 2026-07-17T07:53:48Z

PR_Github #59873 [ run ] completed with state SUCCESS. Commit: 4d5fca9
/LLM/main/L0_MergeRequest_PR pipeline #48276 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Follow-up: user requires each pre-merge pick to already be actively running in L0 CI post-merge on origin/main — i.e. no QA-lane borrowing, no uncommenting of previously-disabled lines, no new config yamls, no first-time-in-L0 e2e/gen_only combinations. Prior revision's two picks that failed that bar: * gpt-oss 8k1k con512 e2e dep2 — active in QA lane only; the e2e mode of this 8k1k dep2 config was not in L0 anywhere. * kimi 8k1k con4 tep8 mtp3 gen_only GB200 — was commented out in the GB200 12-GPU 3-node post_merge block since April. Replaced with: * gpt-oss 8k1k con1024 tp4 e2e MTP0 — was active in L0 post-merge on GB200 ctx1_node1_gpu1_gen1_node1_gpu4 (8 GPUs, 2 nodes). Gen-worker parallelism shifts dep→tp for this slot. * kimi 8k1k con4 tep8 mtp3 gen_only — was active in L0 post-merge on GB300 ctx1_node1_gpu4_gen1_node2_gpu8 (12 GPUs, 3 nodes). GPU family shifts GB200→GB300 for this slot. File changes: - l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu2.yml: reverted to origin/main state (no pre_merge block). - l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml: new pre_merge block with gpt-oss 8k1k con1024 e2e; test line removed from post_merge to avoid double-run. - l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.yml: reverted to origin/main state (kimi 8k1k stays commented out). - l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.yml: new pre_merge block with kimi 8k1k con4 tep8 mtp3 gen_only; same test removed from post_merge. Jenkins: - Dropped stages: FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2 (GB200) and FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8 (GB200 kimi variant). - New stages: FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4 (GB200 gpt-oss) and FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8 (GB300 kimi variant). - Post-merge decrement: GB200-8_GPUs-2_Nodes CTX1-NODE1-GPU1-GEN1- NODE1-GPU4 7→6; GB300-12_GPUs-3_Nodes CTX1-NODE1-GPU4-GEN1-NODE2- GPU8 5→4. Others revert to origin/main testCounts. Shape matrix (all 4 pre-merge slots now at 8k1k): NVIDIA#1 DSR1 B200 8k1k con1536 dep e2e=NO gen_only MTP✓ NVIDIA#2 gpt-oss GB200 8k1k con1024 tp e2e=YES MTP✗ NVIDIA#3 glm-5 GB300 8k1k con1024 dep e2e=YES MTP✓ NVIDIA#4 kimi GB300 8k1k con4 tep e2e=NO gen_only MTP✓ Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-17T07:59:56Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T08:06:25Z

PR_Github #59932 [ run ] triggered by Bot. Commit: 2dfe33f Link to invocation

tensorrt-cicd · 2026-07-17T11:02:31Z

PR_Github #59932 [ run ] completed with state SUCCESS. Commit: 2dfe33f
/LLM/main/L0_MergeRequest_PR pipeline #48334 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-17T14:26:11Z

/bot skip --comment "Move post-merge test cases to pre-merge, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-07-17T14:32:27Z

PR_Github #59991 [ skip ] triggered by Bot. Commit: 2dfe33f Link to invocation

tensorrt-cicd · 2026-07-17T14:38:47Z

PR_Github #59991 [ skip ] completed with state SUCCESS. Commit: 2dfe33f
Skipping testing for commit 2dfe33f

Link to invocation

chenfeiz0326 requested review from a team as code owners July 14, 2026 03:21

chenfeiz0326 requested review from dpitman-nvda and yuanjingx87 July 14, 2026 03:21

longlee0622 reviewed Jul 14, 2026

View reviewed changes

Comment thread jenkins/L0_Test.groovy Outdated

github-actions Bot assigned chenfeiz0326 Jul 14, 2026

chenfeiz0326 requested a review from Shixiaowei02 July 14, 2026 03:28

chenfeiz0326 requested review from a team as code owners July 14, 2026 04:53

chenfeiz0326 changed the title ~~[None][test] Promote 8 disagg perf sanity tests to pre-merge~~ [None][test] Promote disagg perf sanity tests to pre-merge Jul 14, 2026

chenfeiz0326 changed the title ~~[None][test] Promote disagg perf sanity tests to pre-merge~~ [None][test] Promote disagg perf sanity tests to pre-merge for functional verification Jul 14, 2026

Shixiaowei02 approved these changes Jul 14, 2026

View reviewed changes

chenfeiz0326 enabled auto-merge (squash) July 14, 2026 14:32

chenfeiz0326 requested review from ZhanruiSunCh, chzblych and mzweilz July 14, 2026 14:33

fredricz-20070104 approved these changes Jul 15, 2026

View reviewed changes

mzweilz approved these changes Jul 15, 2026

View reviewed changes

chzblych reviewed Jul 15, 2026

View reviewed changes

Comment thread jenkins/L0_Test.groovy

chenfeiz0326 requested review from chzblych and longlee0622 July 16, 2026 07:42

dpitman-nvda approved these changes Jul 16, 2026

View reviewed changes

chenfeiz0326 added 7 commits July 16, 2026 19:51

chenfeiz0326 force-pushed the chenfeiz/premerge-disagg-perf-sanity branch from 93db0ed to 4d5fca9 Compare July 17, 2026 02:57

chzblych approved these changes Jul 17, 2026

View reviewed changes

chenfeiz0326 added 2 commits July 17, 2026 00:14

chenfeiz0326 merged commit 3eefb00 into NVIDIA:main Jul 17, 2026
10 checks passed

Uh oh!

Conversation

chenfeiz0326 commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Coverage matrix (4 tests, all at 8k1k)

FUNCTIONAL-ONLY gate

Full list of pre-merge tests + stage names

Files changed

Test plan

Uh oh!

coderabbitai Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Uh oh!

chenfeiz0326 commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

chenfeiz0326 commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

chenfeiz0326 commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

Shixiaowei02 commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

chenfeiz0326 commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

fredricz-20070104 commented Jul 15, 2026

Execution stage & CI-gating summary

⚠️ Cost note

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

mzweilz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chenfeiz0326 commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

chenfeiz0326 commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

chenfeiz0326 commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

chenfeiz0326 commented Jul 14, 2026 •

edited

Loading

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading