[None][fix] Return 3-tuple from check_gen_transfer_status fast path by qiaoxj07 · Pull Request #14162 · NVIDIA/TensorRT-LLM

qiaoxj07 · 2026-05-15T01:28:10Z

Summary

Fix tuple-arity mismatch in KvCacheTransceiverV2.check_gen_transfer_status: the pure-CTX early-out returned [], [] while the normal path and the sole caller (py_executor._check_disagg_gen_cache_transfer_status) expect a 3-tuple (completed, failed, cancelled_reqs).
Both return paths now return [], [], [] / (completed, failed, cancelled_reqs) consistently.

Repro

On a fresh disagg run (DSv4-Pro, ctx1_gen1_dep16, GB300), the GEN side's executor died with:

[RANK 0] Error in event loop: not enough values to unpack (expected 3, got 2)
  File ".../py_executor.py", line 3874, in _check_disagg_gen_cache_transfer_status
    _, _, cancelled_reqs = result
ValueError: not enough values to unpack (expected 3, got 2)

Triggered when _ever_had_recv_session is still False on the first scheduling iteration after warm-up (no recv session opened yet on this rank). The 2-tuple was introduced in #14042 (Skip transceiver tp_allgather when no sessions ever opened).

Test plan

Re-run the failing benchmark bm_deepseek-v4-pro-sol-1024-1024-ratio08-20260514-GB300/ctx1_gen1_dep16_concurrency16_eplb0_mtp0 and confirm no ValueError is raised.
Existing disagg unit tests under tests/unittest/disaggregated/ pass.

The pure-CTX early-out in `KvCacheTransceiverV2.check_gen_transfer_status` returned a 2-tuple `[], []` while the normal path (and the sole caller in `py_executor._check_disagg_gen_cache_transfer_status`) expects a 3-tuple `(completed, failed, cancelled_reqs)`. On a GEN rank whose `_ever_had_recv_session` is still False (e.g. the very first scheduling iteration after warm-up, before any session has been opened), the early-return triggered: ValueError: not enough values to unpack (expected 3, got 2) File py_executor.py:3874, in _check_disagg_gen_cache_transfer_status _, _, cancelled_reqs = result which killed the executor thread and hung the disagg-gen workers. Fix by returning `[], [], []` so both return paths agree on arity. Signed-off-by: Xianjie Qiao <xqiao@nvidia.com> Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

lfr-0531 · 2026-05-15T01:30:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-15T01:36:31Z

PR_Github #48475 [ run ] triggered by Bot. Commit: 73e1942 Link to invocation

tensorrt-cicd · 2026-05-15T06:40:36Z

PR_Github #48475 [ run ] completed with state SUCCESS. Commit: 73e1942
/LLM/main/L0_MergeRequest_PR pipeline #38270 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

qiaoxj07 · 2026-05-15T08:16:37Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-15T08:23:26Z

PR_Github #48563 [ run ] triggered by Bot. Commit: 73e1942 Link to invocation

tensorrt-cicd · 2026-05-15T11:01:01Z

PR_Github #48563 [ run ] completed with state SUCCESS. Commit: 73e1942
/LLM/main/L0_MergeRequest_PR pipeline #38352 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-18T09:54:37Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-18T10:01:31Z

PR_Github #48891 [ run ] triggered by Bot. Commit: 73e1942 Link to invocation

tensorrt-cicd · 2026-05-18T10:30:45Z

PR_Github #48891 [ run ] completed with state FAILURE. Commit: 73e1942
/LLM/main/L0_MergeRequest_PR pipeline #38641 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-18T13:31:23Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-18T13:38:08Z

PR_Github #48917 [ run ] triggered by Bot. Commit: 73e1942 Link to invocation

tensorrt-cicd · 2026-05-18T14:14:54Z

PR_Github #48917 [ run ] completed with state FAILURE. Commit: 73e1942
/LLM/main/L0_MergeRequest_PR pipeline #38666 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-18T15:53:10Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-18T16:01:40Z

PR_Github #48929 [ run ] triggered by Bot. Commit: 73e1942 Link to invocation

tensorrt-cicd · 2026-05-18T16:47:48Z

PR_Github #48929 [ run ] completed with state FAILURE. Commit: 73e1942
/LLM/main/L0_MergeRequest_PR pipeline #38676 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-05-19T01:28:02Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-19T01:34:45Z

PR_Github #49024 [ run ] triggered by Bot. Commit: 73e1942 Link to invocation

tensorrt-cicd · 2026-05-19T02:58:29Z

PR_Github #49024 [ run ] completed with state SUCCESS. Commit: 73e1942
/LLM/main/L0_MergeRequest_PR pipeline #38762 completed with status: 'SUCCESS'

CI Report

Link to invocation

…VIDIA#14162) Signed-off-by: Xianjie Qiao <xqiao@nvidia.com> Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

qiaoxj07 requested a review from a team as a code owner May 15, 2026 01:28

qiaoxj07 requested review from dongxuy04 and leslie-fang25 and removed request for a team May 15, 2026 01:28

github-actions Bot assigned qiaoxj07 May 15, 2026

lfr-0531 requested review from Shixiaowei02 and removed request for dongxuy04 and leslie-fang25 May 15, 2026 01:29

Shixiaowei02 approved these changes May 15, 2026

View reviewed changes

lfr-0531 added the deepseek-v4 label May 19, 2026

qiaoxj07 merged commit ddd96e6 into NVIDIA:feat/deepseek_v4 May 19, 2026
9 of 11 checks passed

Uh oh!

Conversation

qiaoxj07 commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Repro

Test plan

Uh oh!

lfr-0531 commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

qiaoxj07 commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

lfr-0531 commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

lfr-0531 commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

lfr-0531 commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

lfr-0531 commented May 19, 2026

Uh oh!

tensorrt-cicd commented May 19, 2026

Uh oh!

tensorrt-cicd commented May 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

qiaoxj07 commented May 15, 2026 •

edited

Loading