[https://nvbugs/6422334][fix] In PyExecutor._prepare_draft_requests, gate the py_last_draft_tokens snapshot… by trtllm-agent · Pull Request #16121 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-08T13:06:17Z

Summary

Root cause: PyExecutor._prepare_draft_requests snapshotted py_last_draft_tokens = py_draft_tokens for DISAGG_GENERATION_INIT requests every iteration, leaving a stale dummy [0]*max_total_draft_tokens in py_last_draft_tokens by the time the request transitioned to GENERATION_IN_PROGRESS
Fix: In PyExecutor._prepare_draft_requests, gate the py_last_draft_tokens snapshot on req.state == GENERATION_IN_PROGRESS so DISAGG_GENERATION_INIT dummies never leak; also remove the two H20 waivers this makes obsolete
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6422334

Summary by CodeRabbit

Bug Fixes
- Improved request handling so initial draft-token state is no longer carried into the first real generation step, reducing the risk of incorrect output during early decoding.
Tests
- Removed obsolete waiver entries related to guided decoding coverage.

coderabbitai · 2026-07-08T13:09:21Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0c63492f-84b9-4d19-8e70-c40e75785ff9

📥 Commits

Reviewing files that changed from the base of the PR and between c92ac90 and fb88d38.

📒 Files selected for processing (2)

tensorrt_llm/_torch/pyexecutor/py_executor.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

Modifies draft-token snapshot logic in PyExecutor to only capture py_last_draft_tokens when a request is in GENERATION_IN_PROGRESS state, excluding DISAGG_GENERATION_INIT. Also removes two H20 test waiver entries for guided decoding tests that were previously skipped.

Changes

Draft token snapshot fix and waiver update

Layer / File(s)	Summary
Restrict draft token snapshot to GENERATION_IN_PROGRESS `tensorrt_llm/_torch/pyexecutor/py_executor.py`	`_prepare_draft_requests` now conditionally snapshots `py_draft_tokens` into `py_last_draft_tokens` only for `GENERATION_IN_PROGRESS` requests, skipping `DISAGG_GENERATION_INIT`.
Remove stale H20 guided decoding waivers `tests/integration/test_lists/waives.txt`	Removes two SKIP waiver entries for `TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3` with `eagle3_one_model=False` under `llguidance` and `xgrammar` on H20.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#16103: Both PRs modify tests/integration/test_lists/waives.txt to adjust waived/skipped integration tests.

Suggested reviewers: Wanli-Jiang, achartier

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR has Summary, Test plan, and Links, but it omits the required Description, Test Coverage, and PR Checklist sections.	Add the missing Description and Test Coverage sections, and include the PR Checklist items using the template headings.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the bug fix and identifies the affected function with the required ticket/type format.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

zhaoyangwang-nvidia · 2026-07-08T13:30:13Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-08T13:37:01Z

PR_Github #58236 [ run ] triggered by Bot. Commit: fb88d38 Link to invocation

zhaoyangwang-nvidia · 2026-07-09T06:47:31Z

/bot run

tensorrt-cicd · 2026-07-09T06:54:07Z

PR_Github #58405 [ run ] triggered by Bot. Commit: 9f5ea3f Link to invocation

tensorrt-cicd · 2026-07-09T06:54:18Z

PR_Github #58236 [ run ] completed with state ABORTED. Commit: fb88d38
/LLM/main/L0_MergeRequest_PR pipeline #46877 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-07-09T13:40:05Z

PR_Github #58405 [ run ] completed with state SUCCESS. Commit: 9f5ea3f
/LLM/main/L0_MergeRequest_PR pipeline #47024 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

zhaoyangwang-nvidia · 2026-07-10T07:47:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-10T07:53:56Z

PR_Github #58639 [ run ] triggered by Bot. Commit: e3ba765 Link to invocation

tensorrt-cicd · 2026-07-10T18:59:19Z

PR_Github #58639 [ run ] completed with state SUCCESS. Commit: e3ba765
/LLM/main/L0_MergeRequest_PR pipeline #47226 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

zhaoyangwang-nvidia · 2026-07-11T07:20:50Z

/bot run

tensorrt-cicd · 2026-07-11T07:26:13Z

PR_Github #58754 [ run ] triggered by Bot. Commit: e3ba765 Link to invocation

tensorrt-cicd · 2026-07-11T12:20:32Z

PR_Github #58754 [ run ] completed with state FAILURE. Commit: e3ba765
/LLM/main/L0_MergeRequest_PR pipeline #47336 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ENERATION_INIT Two-model Eagle3 disagg with disable_overlap_scheduler=True crashes with File "tensorrt_llm/_torch/speculative/model_drafter.py", line 240 assert num_draft_tokens == 0 AssertionError when guided decoding is enabled. The AssertionError kills the drafter thread on the generation server; every subsequent request fails with "Event loop terminated with error", so pytest's HTTP client can no longer connect and the test times out at 3600 s. Root cause: while a request waits in DISAGG_GENERATION_INIT for KV cache transfer, PyExecutor._prepare_draft_requests runs each iteration and unconditionally snapshotted py_last_draft_tokens = py_draft_tokens for both GENERATION_IN_PROGRESS and DISAGG_GENERATION_INIT. Because it also refills py_draft_tokens with [0]*max_total_draft_tokens right after, py_last_draft_tokens accumulates a dummy [0, 0, 0]. When the request finally transitions to GENERATION_IN_PROGRESS, ModelDrafter enters the first-time-context branch and asserts num_draft_tokens == 0, but the stale snapshot makes _initialize_draft_tokens return 3 -> assert fires. Only snapshot py_last_draft_tokens for requests that have actually been through the drafter (GENERATION_IN_PROGRESS). Also remove the two H20 waivers that this fix makes obsolete. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

zhaoyangwang-nvidia · 2026-07-12T06:25:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-12T06:31:10Z

PR_Github #58787 [ run ] triggered by Bot. Commit: e7b25f0 Link to invocation

tensorrt-cicd · 2026-07-12T06:31:10Z

PR_Github #58787 [ run ] completed with state DISABLED
Pipeline is freezed and top-1 instance is under maintenance.

Link to invocation

zhaoyangwang-nvidia · 2026-07-12T08:59:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-12T09:05:01Z

PR_Github #58803 [ run ] triggered by Bot. Commit: e7b25f0 Link to invocation

tensorrt-cicd · 2026-07-12T09:05:02Z

PR_Github #58803 [ run ] completed with state DISABLED
Pipeline is freezed and top-1 instance is under maintenance.

Link to invocation

chzblych · 2026-07-12T17:27:51Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-12T17:52:06Z

PR_Github #58835 [ run ] triggered by Bot. Commit: e7b25f0 Link to invocation

tensorrt-cicd · 2026-07-13T01:19:44Z

PR_Github #58835 [ run ] completed with state SUCCESS. Commit: e7b25f0
/LLM/main/L0_MergeRequest_PR pipeline #47388 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-13T06:32:10Z

/bot run

tensorrt-cicd · 2026-07-13T06:37:26Z

PR_Github #58915 [ run ] triggered by Bot. Commit: e7b25f0 Link to invocation

tensorrt-cicd · 2026-07-13T07:43:56Z

PR_Github #58915 [ run ] completed with state SUCCESS. Commit: e7b25f0
/LLM/main/L0_MergeRequest_PR pipeline #47452 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

zhaoyangwang-nvidia · 2026-07-13T15:05:03Z

/bot run

tensorrt-cicd · 2026-07-13T15:11:07Z

PR_Github #58987 [ run ] triggered by Bot. Commit: e7b25f0 Link to invocation

tensorrt-cicd · 2026-07-13T16:29:02Z

PR_Github #58987 [ run ] completed with state SUCCESS. Commit: e7b25f0
/LLM/main/L0_MergeRequest_PR pipeline #47517 completed with status: 'SUCCESS'

CI Report

Link to invocation

trtllm-agent requested a review from a team as a code owner July 8, 2026 13:06

trtllm-agent requested a review from achartier July 8, 2026 13:06

trtllm-agent assigned ziyixiong-nv Jul 8, 2026

github-actions Bot assigned trtllm-agent Jul 8, 2026

DomBrown approved these changes Jul 8, 2026

View reviewed changes

zhaoyangwang-nvidia force-pushed the repair-bot-bug6422334 branch from fb88d38 to 9f5ea3f Compare July 9, 2026 06:47

zhaoyangwang-nvidia force-pushed the repair-bot-bug6422334 branch from 9f5ea3f to e3ba765 Compare July 10, 2026 07:47

zhaoyangwang-nvidia force-pushed the repair-bot-bug6422334 branch from e3ba765 to e7b25f0 Compare July 12, 2026 06:25

crazydemo approved these changes Jul 14, 2026

View reviewed changes

zhaoyangwang-nvidia approved these changes Jul 14, 2026

View reviewed changes

zhaoyangwang-nvidia merged commit 890e108 into NVIDIA:main Jul 14, 2026
8 checks passed

coderabbitai Bot mentioned this pull request Jul 25, 2026

Add output-derived knowledge graph and indexed routes NaHyeongyu/promty#33

Merged

Uh oh!

Conversation

trtllm-agent commented Jul 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 8, 2026

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

zhaoyangwang-nvidia commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

zhaoyangwang-nvidia commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

zhaoyangwang-nvidia commented Jul 10, 2026

Uh oh!

tensorrt-cicd commented Jul 10, 2026

Uh oh!

tensorrt-cicd commented Jul 10, 2026

Uh oh!

zhaoyangwang-nvidia commented Jul 11, 2026

Uh oh!

tensorrt-cicd commented Jul 11, 2026

Uh oh!

tensorrt-cicd commented Jul 11, 2026

Uh oh!

zhaoyangwang-nvidia commented Jul 12, 2026

Uh oh!

tensorrt-cicd commented Jul 12, 2026

Uh oh!

tensorrt-cicd commented Jul 12, 2026

Uh oh!

zhaoyangwang-nvidia commented Jul 12, 2026

Uh oh!

tensorrt-cicd commented Jul 12, 2026

Uh oh!

tensorrt-cicd commented Jul 12, 2026

Uh oh!

chzblych commented Jul 12, 2026

Uh oh!

tensorrt-cicd commented Jul 12, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

xxi-nv commented Jul 13, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

zhaoyangwang-nvidia commented Jul 13, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

trtllm-agent commented Jul 8, 2026 •

edited by coderabbitai Bot

Loading