Skip to content

[None][test] CI bisect nvbugs/6280721: probe efb71c7 / pm-2755 (Cherry-pick kv_cache_manager_v2 fixes to main #14725) - #15242

Closed
tensorrt-cicd wants to merge 2 commits into
chenfeiz/bisect-6280721-parent-basefrom
chenfeiz/bisect-6280721-parent-head
Closed

[None][test] CI bisect nvbugs/6280721: probe efb71c7 / pm-2755 (Cherry-pick kv_cache_manager_v2 fixes to main #14725)#15242
tensorrt-cicd wants to merge 2 commits into
chenfeiz/bisect-6280721-parent-basefrom
chenfeiz/bisect-6280721-parent-head

Conversation

@tensorrt-cicd

@tensorrt-cicd tensorrt-cicd commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Pre-merge CI bisect probe for nvbugs/6280721.

This PR isolates the parent (baseline) commit 910826b [TRTLLMINF-69][infra] Migrate A100X-FMHA-Post-Merge-1 and A100X-Triton-Post-Merge-[1,2] to SLURM (#14921) — the commit immediately before the suspected culprit 8e5d9e2 — by basing the PR on its parent a8c4007. The PR diff is exactly that one commit (plus an empty sign-off commit; tree is identical to 910826b).

Background

Companion to the culprit probe. 910826b is the commit right before the MTP one-model merge (#12353). If the test passes here and fails on the culprit PR, the regression is pinned to 8e5d9e2.

Expected result: this test passes here.

Test under observation

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_1k1k_con2048_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL]

Waiver note

This case is not present in tests/integration/test_lists/waives.txt at this commit, so no waiver removal is needed — the test runs unwaived.

Test Coverage

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_1k1k_con2048_ctx1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL]

@lowsfer @lancelly @Tabrizian
[None][fix] Cherry-pick kv_cache_manager_v2 fixes to main (#14725)
efb71c7 
Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>
Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd force-pushed the chenfeiz/bisect-6280721-parent-head branch from 8c6cba0 to db24ec0 Compare June 11, 2026 08:21
@chenfeiz0326

chenfeiz0326 commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4"

@tensorrt-cicd

tensorrt-cicd commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

PR_Github #53532 [ run ] triggered by Bot. Commit: db24ec0 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

PR_Github #53532 [ run ] completed with state FAILURE. Commit: db24ec0
/LLM/main/L0_MergeRequest_PR pipeline #42685 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tensorrt-cicd tensorrt-cicd mentioned this pull request Jun 11, 2026
Closed
@tensorrt-cicd
tensorrt-cicd force-pushed the chenfeiz/bisect-6280721-parent-head branch from db24ec0 to efb71c7 Compare June 15, 2026 06:57
@tensorrt-cicd tensorrt-cicd reopened this Jun 15, 2026
@tensorrt-cicd tensorrt-cicd changed the title [None][test] CI bisect nvbugs/6280721: parent 910826b (baseline before MTP one-model merge) [None][test] CI bisect nvbugs/6280721: probe efb71c7 / pm-2755 (Cherry-pick kv_cache_manager_v2 fixes to main #14725) Jun 15, 2026
@chenfeiz0326

chenfeiz0326 commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2"

@tensorrt-cicd

tensorrt-cicd commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

PR_Github #54248 [ run ] triggered by Bot. Commit: efb71c7 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

PR_Github #54248 [ run ] completed with state SUCCESS. Commit: efb71c7
/LLM/main/L0_MergeRequest_PR pipeline #43322 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chenfeiz0326

chenfeiz0326 commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2"

@tensorrt-cicd

tensorrt-cicd commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

PR_Github #54308 [ run ] triggered by Bot. Commit: f39c446 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

PR_Github #54308 [ run ] completed with state FAILURE. Commit: f39c446
/LLM/main/L0_MergeRequest_PR pipeline #43379 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chzblych chzblych closed this Jun 16, 2026
@chzblych
chzblych deleted the chenfeiz/bisect-6280721-parent-head branch June 16, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@tensorrt-cicd tensorrt-cicd

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tensorrt-cicd @chenfeiz0326 @chzblych @lowsfer