[TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats by xwang233 · Pull Request #16571 · NVIDIA/TensorRT-LLM

xwang233 · 2026-07-19T00:35:14Z

Description

The per-iteration speculative-decoding stats loop in PyExecutor._append_iter_stats (tensorrt_llm/_torch/pyexecutor/py_executor.py) aggregates acceptance rate (AR) and acceptance length (AL) over scheduled_batch.generation_requests without filtering out dummy requests.

On a disaggregated-generation worker running under attention-DP (and likewise for cuda-graph padding), an idle rank is padded with a generation dummy request. That dummy carries synthetic py_draft_tokens = [1] * k (the is_gen branch of resource_manager.add_dummy_requests) and is assigned a spec-sampler accepted count, so it enters the AR/AL aggregate as a mostly-rejected drafted request. The result is a depressed, misleading reported acceptance rate/length — the more idle-rank dummies an iteration accumulates, the lower the reported AR/AL, even though real model behavior is unchanged.

The fix mirrors the filter that already exists in the parallel speculation-gate loop in the same file (if draft_len <= 0 or request.is_dummy: continue): skip any request whose is_dummy is set before it contributes to the specdec metric.

This is a metrics-only change. It touches only the stats-aggregation loop — not scheduling, forward, sampling, or resource management — so model outputs and throughput are unaffected; only the reported specDecodingStats are corrected.

Test Coverage

The fix mirrors an existing, tested filter (request.is_dummy) already applied in the speculation-gate loop of the same method, so no new code path or public API is introduced.
Empirically validated on GLM-5.2 MTP disaggregated serving on GB200 (1p1d, attention-DP, cuda-graph ON): at c128 SLO20 the reported k=1 AL recovered from 1.813 to 1.873 and k=4 from 2.908 to 3.536, with zero throughput change (SSE p25 tok/s within noise), confirming the change corrects the metric without altering execution. The corrected disagg number (0.873 AR) matches the aggregated-serving ceiling on the same traffic (0.879), confirming the previously-reported disagg "AL penalty" was substantially this accounting artifact.

PR Checklist

PR description clearly explains what and why.
PR follows TRT-LLM coding guidelines.
No API changes; no new dependencies; no CODEOWNERS/doc/architecture changes needed.

…e stats The per-iteration speculative-decoding stats loop in _append_iter_stats counted attention-DP (and cuda-graph padding) dummy generation requests as real drafted requests. On a disagg-GEN worker under attention-DP, an idle rank is padded with a *generation* dummy that carries synthetic py_draft_tokens=[1]*k (resource_manager.add_dummy_requests is_gen branch) and is assigned a spec-sampler accepted count, so it entered the AR/AL aggregate as a mostly-rejected drafted request and depressed the reported acceptance rate/length. The parallel speculation-gate loop already filters is_dummy; mirror that here. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

coderabbitai · 2026-07-19T00:38:36Z

📝 Walkthrough

Walkthrough

Speculative decoding per-iteration statistics now exclude dummy generation requests from draft-token and accepted-token aggregation.

Changes

Speculative decoding metrics

Layer / File(s)	Summary
Filter dummy requests from aggregation `tensorrt_llm/_torch/pyexecutor/py_executor.py`	`_update_iter_stats` skips requests with `is_dummy` before aggregating speculative decoding token counts.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: chienchunhung

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title follows the required ticket/type format and accurately summarizes the dummy-request metrics fix.
Description check	✅ Passed	The description includes the required sections and clearly explains the issue, fix, and test coverage.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

xwang233 · 2026-07-19T03:16:30Z

/bot run

tensorrt-cicd · 2026-07-19T03:21:51Z

PR_Github #60139 [ run ] triggered by Bot. Commit: 223df8c Link to invocation

tensorrt-cicd · 2026-07-19T05:33:51Z

PR_Github #60139 [ run ] completed with state FAILURE. Commit: 223df8c
/LLM/main/L0_MergeRequest_PR pipeline #48519 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-19T15:42:55Z

/bot run --reuse-test

tensorrt-cicd · 2026-07-19T15:49:25Z

PR_Github #60181 [ run ] triggered by Bot. Commit: 223df8c Link to invocation

tensorrt-cicd · 2026-07-19T18:17:52Z

PR_Github #60181 [ run ] completed with state FAILURE. Commit: 223df8c
/LLM/main/L0_MergeRequest_PR pipeline #48554 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung

Thanks for the fix!

Added a comment: we should add an unit test to guard the code change.

…xclude-adp-dummy

…ltering Address review feedback on the specdec AR/AL dummy-request fix: - Add focused unit tests in test_iter_stats_populate.py covering the speculative-decoding aggregation branch of _update_iter_stats. The existing dummy-request tests leave stats.specdec_stats as None, so that branch was previously uncovered. New tests allocate specdec_stats and verify that attention-DP / CUDA-graph dummy generation requests are excluded from draft/accepted token counts and acceptance length. - Collapse the verbose filter comment to a concise one-liner per reviewer suggestion. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

xwang233 · 2026-07-20T22:39:55Z

/bot run

tensorrt-cicd · 2026-07-20T22:45:41Z

PR_Github #60480 [ run ] triggered by Bot. Commit: 1b97470 Link to invocation

tensorrt-cicd · 2026-07-20T23:25:23Z

PR_Github #60480 [ run ] completed with state FAILURE. Commit: 1b97470
/LLM/main/L0_MergeRequest_PR pipeline #48811 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-20T23:34:14Z

/bot run

tensorrt-cicd · 2026-07-20T23:40:37Z

PR_Github #60496 [ run ] triggered by Bot. Commit: 1b97470 Link to invocation

tensorrt-cicd · 2026-07-21T01:12:42Z

PR_Github #60496 [ run ] completed with state SUCCESS. Commit: 1b97470
/LLM/main/L0_MergeRequest_PR pipeline #48825 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-21T04:26:57Z

/bot run

xwang233 · 2026-07-22T18:29:38Z

/bot run

tensorrt-cicd · 2026-07-22T18:36:30Z

PR_Github #61059 [ run ] triggered by Bot. Commit: 0df54dc Link to invocation

tensorrt-cicd · 2026-07-22T19:58:46Z

PR_Github #61059 [ run ] completed with state FAILURE. Commit: 0df54dc
/LLM/main/L0_MergeRequest_PR pipeline #49314 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-23T03:27:28Z

/bot run

tensorrt-cicd · 2026-07-23T03:34:06Z

PR_Github #61182 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-23T03:59:44Z

PR_Github #61182 [ run ] completed with state FAILURE. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49431 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-23T05:07:33Z

/bot run

tensorrt-cicd · 2026-07-23T05:12:57Z

PR_Github #61193 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-23T06:54:01Z

PR_Github #61193 [ run ] completed with state SUCCESS. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49440 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-23T15:53:28Z

/bot run --reuse-test

tensorrt-cicd · 2026-07-23T16:01:35Z

PR_Github #61324 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-23T16:25:16Z

PR_Github #61324 [ run ] completed with state FAILURE. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49554 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-23T16:44:04Z

/bot run --reuse-test

tensorrt-cicd · 2026-07-23T16:50:03Z

PR_Github #61337 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-23T17:18:54Z

PR_Github #61337 [ run ] completed with state FAILURE. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49567 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-23T18:12:08Z

/bot run --reuse-test --disable-fail-fast

tensorrt-cicd · 2026-07-23T18:18:16Z

PR_Github #61359 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-23T21:22:13Z

PR_Github #61359 [ run ] completed with state FAILURE. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49588 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-24T00:39:23Z

/bot run --reuse-test --disable-fail-fast

tensorrt-cicd · 2026-07-24T00:45:30Z

PR_Github #61440 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-24T05:52:13Z

PR_Github #61440 [ run ] completed with state SUCCESS. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49667 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 · 2026-07-24T18:04:14Z

/bot run --reuse-test --disable-fail-fast

tensorrt-cicd · 2026-07-24T18:10:01Z

PR_Github #61618 [ run ] triggered by Bot. Commit: 461262a Link to invocation

tensorrt-cicd · 2026-07-24T21:32:18Z

PR_Github #61618 [ run ] completed with state SUCCESS. Commit: 461262a
/LLM/main/L0_MergeRequest_PR pipeline #49825 completed with status: 'SUCCESS'

CI Report

Link to invocation

…ative-decode acceptance stats (NVIDIA#16571) Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (54 commits) [NVIDIA#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 (NVIDIA#12705) [None][test] Adjust timeout cases in QA perf test (NVIDIA#16894) [https://nvbugs/6157892][fix] Mistral format refactor (NVIDIA#15123) [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe (NVIDIA#16830) [https://nvbugs/6501376][fix] Test-only fix — drop the `if hidden_size % 2 != 0: with pytest.raises(...)`… (NVIDIA#16844) [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support (NVIDIA#16355) [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert `bfloat16` under… (NVIDIA#16203) [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft (NVIDIA#16831) [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test (NVIDIA#16878) [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests (NVIDIA#16838) [None][infra] Waive 21 failed cases for main in post-merge 2862 (NVIDIA#16882) [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host (NVIDIA#16791) [None][perf] Optimize Blackwell fused MHC half-MMA kernel (NVIDIA#16799) [None][infra] Auto-update test durations from OpenSearch (last 7 days) [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path (NVIDIA#16789) [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks (NVIDIA#16774) [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer (NVIDIA#16505) [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue (NVIDIA#16775) [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats (NVIDIA#16571) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

…nnahz/dep-1082-shared-mnnvl-moe-lifecycle * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (142 commits) [NVIDIA#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 (NVIDIA#12705) [None][test] Adjust timeout cases in QA perf test (NVIDIA#16894) [https://nvbugs/6157892][fix] Mistral format refactor (NVIDIA#15123) [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe (NVIDIA#16830) [https://nvbugs/6501376][fix] Test-only fix — drop the `if hidden_size % 2 != 0: with pytest.raises(...)`… (NVIDIA#16844) [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support (NVIDIA#16355) [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert `bfloat16` under… (NVIDIA#16203) [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft (NVIDIA#16831) [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test (NVIDIA#16878) [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests (NVIDIA#16838) [None][infra] Waive 21 failed cases for main in post-merge 2862 (NVIDIA#16882) [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host (NVIDIA#16791) [None][perf] Optimize Blackwell fused MHC half-MMA kernel (NVIDIA#16799) [None][infra] Auto-update test durations from OpenSearch (last 7 days) [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path (NVIDIA#16789) [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks (NVIDIA#16774) [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer (NVIDIA#16505) [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue (NVIDIA#16775) [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats (NVIDIA#16571) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

xwang233 requested review from a team as code owners July 19, 2026 00:35

xwang233 requested review from Tabrizian, chienchunhung, lori-ren and mikeiovine July 19, 2026 00:35

github-actions Bot assigned xwang233 Jul 19, 2026

xwang233 changed the title ~~[None][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats~~ [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats Jul 19, 2026

chienchunhung approved these changes Jul 19, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py

mikeiovine removed the request for review from Tabrizian July 20, 2026 15:44

Tabrizian reviewed Jul 20, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Outdated

xwang233 added 2 commits July 20, 2026 14:54

Merge remote-tracking branch 'origin/main' into fix/specdec-metrics-e…

862b26a

…xclude-adp-dummy

xwang233 requested a review from Tabrizian July 20, 2026 23:35

Tabrizian approved these changes Jul 20, 2026

View reviewed changes

Merge branch 'main' into fix/specdec-metrics-exclude-adp-dummy

461262a

xwang233 merged commit 4b7d719 into NVIDIA:main Jul 24, 2026
7 checks passed

ishovkun mentioned this pull request Jul 28, 2026

[None][feat] Support the DMD2-distilled Cosmos3-Super-Text2Image-4Step checkpoint #16563

Open

1 task

Uh oh!

Conversation

xwang233 commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Uh oh!

coderabbitai Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

xwang233 commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

xwang233 commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

chienchunhung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xwang233 commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

xwang233 commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

xwang233 commented Jul 21, 2026

Uh oh!

xwang233 commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

xwang233 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

xwang233 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

xwang233 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

xwang233 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

xwang233 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

xwang233 commented Jul 19, 2026 •

edited

Loading

coderabbitai Bot commented Jul 19, 2026 •

edited

Loading