[None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor by lancelly · Pull Request #16422 · NVIDIA/TensorRT-LLM

lancelly · 2026-07-15T10:48:48Z

Description

_handle_disagg_cache_errors_synced (a tp_allgather error vote at the top of every executor iteration when disagg + attention DP are enabled) and _can_queue (a post-schedule batch-size tp_allgather under attention DP) are blocking host-side collectives on the executor loop's critical path.

Unlike the sibling disagg status checks (_check_disagg_gen_transfer_status, _check_disagg_ctx_schedulable_status, _check_disagg_*_cache_transfer_status, _pad_attention_dp_dummy_request, _fetch_new_requests), they carry no NVTX annotation, so in nsys traces they show up as anonymous gaps — one at the top of each iteration and one between scheduling and forward — and are easy to misread as scheduler overhead or idle host time.

This PR adds @nvtx_range to both, matching the existing style in the file. No functional change.

Summary by CodeRabbit

Performance Monitoring
- Added runtime tracing markers to improve visibility into executor activity.
- No user-facing behavior or functionality has changed.

…xecutor _handle_disagg_cache_errors_synced (tp_allgather error vote at the top of every executor iteration) and _can_queue (post-schedule batch-size tp_allgather under attention DP) are blocking host-side collectives, but unlike the sibling disagg status checks they carry no NVTX range, so they show up as anonymous gaps in nsys traces. Annotate both for profiling visibility. No functional change. Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>

lancelly · 2026-07-15T10:52:20Z

/bot run --disable-fail-fast

coderabbitai · 2026-07-15T10:54:36Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b89ab467-5019-4d4b-a5b9-fccf4c8b5d9e

📥 Commits

Reviewing files that changed from the base of the PR and between e54fe55 and 51011c8.

📒 Files selected for processing (1)

tensorrt_llm/_torch/pyexecutor/py_executor.py

📝 Walkthrough

Walkthrough

Adds NVTX range decorators to _can_queue and _handle_disagg_cache_errors_synced in PyExecutor; method logic and control flow remain unchanged.

Changes

PyExecutor instrumentation

Layer / File(s)	Summary
Add NVTX ranges `tensorrt_llm/_torch/pyexecutor/py_executor.py`	Adds NVTX decorators to `_can_queue` and `_handle_disagg_cache_errors_synced` without changing their implementations.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: chienchunhung, 2ez4bz, zheyuf

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the change, but it omits the required Test Coverage and PR Checklist sections from the template.	Add the missing Test Coverage and PR Checklist sections, and include any relevant tests or note that none were run.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific and matches the NVTX instrumentation change in PyExecutor.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-15T11:04:55Z

PR_Github #59441 [ run ] triggered by Bot. Commit: 51011c8 Link to invocation

tensorrt-cicd · 2026-07-15T15:59:00Z

PR_Github #59441 [ run ] completed with state SUCCESS. Commit: 51011c8
/LLM/main/L0_MergeRequest_PR pipeline #47909 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-17T06:08:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-17T06:14:05Z

PR_Github #59899 [ run ] triggered by Bot. Commit: 51011c8 Link to invocation

tensorrt-cicd · 2026-07-17T18:47:03Z

PR_Github #59899 [ run ] completed with state FAILURE. Commit: 51011c8
/LLM/main/L0_MergeRequest_PR pipeline #48301 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-19T08:38:09Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-19T08:43:26Z

PR_Github #60152 [ run ] triggered by Bot. Commit: 51011c8 Link to invocation

tensorrt-cicd · 2026-07-19T21:11:31Z

PR_Github #60152 [ run ] completed with state SUCCESS. Commit: 51011c8
/LLM/main/L0_MergeRequest_PR pipeline #48530 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-20T09:11:56Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-20T09:17:15Z

PR_Github #60336 [ run ] triggered by Bot. Commit: 4f1abf5 Link to invocation

tensorrt-cicd · 2026-07-20T12:17:35Z

PR_Github #60336 [ run ] completed with state FAILURE. Commit: 4f1abf5
/LLM/main/L0_MergeRequest_PR pipeline #48681 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-21T02:28:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-21T02:34:17Z

PR_Github #60537 [ run ] triggered by Bot. Commit: 4f1abf5 Link to invocation

tensorrt-cicd · 2026-07-21T05:18:29Z

PR_Github #60537 [ run ] completed with state FAILURE. Commit: 4f1abf5
/LLM/main/L0_MergeRequest_PR pipeline #48856 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-21T05:44:09Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-21T05:50:48Z

PR_Github #60596 [ run ] triggered by Bot. Commit: 4f1abf5 Link to invocation

tensorrt-cicd · 2026-07-21T07:03:18Z

PR_Github #60596 [ run ] completed with state SUCCESS. Commit: 4f1abf5
/LLM/main/L0_MergeRequest_PR pipeline #48907 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-21T07:29:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-21T07:35:48Z

PR_Github #60633 [ run ] triggered by Bot. Commit: 4f1abf5 Link to invocation

tensorrt-cicd · 2026-07-21T08:50:23Z

PR_Github #60633 [ run ] completed with state SUCCESS. Commit: 4f1abf5
/LLM/main/L0_MergeRequest_PR pipeline #48940 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-21T09:36:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-21T09:42:43Z

PR_Github #60662 [ run ] triggered by Bot. Commit: 74fdce8 Link to invocation

tensorrt-cicd · 2026-07-21T13:17:23Z

PR_Github #60662 [ run ] completed with state FAILURE. Commit: 74fdce8
/LLM/main/L0_MergeRequest_PR pipeline #48960 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-22T09:02:09Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-22T09:08:35Z

PR_Github #60941 [ run ] triggered by Bot. Commit: 74fdce8 Link to invocation

tensorrt-cicd · 2026-07-22T11:08:37Z

PR_Github #60941 [ run ] completed with state SUCCESS. Commit: 74fdce8
/LLM/main/L0_MergeRequest_PR pipeline #49205 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-22T11:12:19Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-22T11:19:18Z

PR_Github #60973 [ run ] triggered by Bot. Commit: 74fdce8 Link to invocation

tensorrt-cicd · 2026-07-22T15:07:28Z

PR_Github #60973 [ run ] completed with state FAILURE. Commit: 74fdce8
/LLM/main/L0_MergeRequest_PR pipeline #49235 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-22T15:10:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-22T15:15:48Z

PR_Github #61003 [ run ] triggered by Bot. Commit: 74fdce8 Link to invocation

tensorrt-cicd · 2026-07-22T17:11:35Z

PR_Github #61003 [ run ] completed with state FAILURE. Commit: 74fdce8
/LLM/main/L0_MergeRequest_PR pipeline #49261 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-23T06:16:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T06:22:58Z

PR_Github #61206 [ run ] triggered by Bot. Commit: 74fdce8 Link to invocation

tensorrt-cicd · 2026-07-23T07:29:50Z

PR_Github #61206 [ run ] completed with state SUCCESS. Commit: 74fdce8
/LLM/main/L0_MergeRequest_PR pipeline #49451 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-23T07:41:27Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T08:05:55Z

PR_Github #61242 [ run ] triggered by Bot. Commit: 74fdce8 Link to invocation

tensorrt-cicd · 2026-07-23T09:25:34Z

PR_Github #61242 [ run ] completed with state SUCCESS. Commit: 74fdce8
/LLM/main/L0_MergeRequest_PR pipeline #49483 completed with status: 'SUCCESS'

CI Report

Link to invocation

…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (83 commits) [None][feat] Bind SourceIdentity to checkpoint artifacts (NVIDIA#16159) [None][infra] Waive 4 failed cases for main in pre-merge 49550 (NVIDIA#16798) [None][fix] Make FlashInfer sampling op wrappers opaque to Dynamo (NVIDIA#16732) [None][feat] top-k: route decode to CuTe DSL GVR top-k in e2e (NVIDIA#16420) [None][feat] Default GLM-5 to the Python KV-cache transceiver (NVIDIA#16524) [None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor (NVIDIA#16422) [https://nvbugs/6426850][test] Unwaive Qwen3.5 397B NVFP4 ADP4 TRTLLM test (NVIDIA#16348) [https://nvbugs/6445456][fix] Restore inplace ops for functionalization v2 (NVIDIA#16410) [None][infra] Waive 1 failed cases for main in pre-merge 49229 (NVIDIA#16786) [None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints (NVIDIA#16433) [None][feat] ADP conversation router: configurable least-queued placement for new conversations (NVIDIA#16294) [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16780) [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16781) [TRTLLMINF-188][infra] Require approval for PerfSanity wildcard runs (NVIDIA#16777) [TRTLLM-13969][feat] Support MiniMax M3 for Disaggregated Serving (NVIDIA#16017) [NVIDIA#11932][fix] Filter CUTLASS MoE GEMM tile configs by device shared memory on SM121 (NVIDIA#12704) [None][chore] Remove attention backend test waivers (NVIDIA#16723) [TRTLLM-14540][perf] Skip fp32 state round-trip in FlashInfer GDN pre… (NVIDIA#16703) [TRTLLM-13694][feat] Add IBDB recipe provenance and refresh configs (NVIDIA#16254) [None][infra] Preview/bump/main (NVIDIA#16758) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

…xecutor (NVIDIA#16422) Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>

github-actions Bot assigned lancelly Jul 15, 2026

lancelly marked this pull request as ready for review July 15, 2026 10:51

lancelly requested review from a team as code owners July 15, 2026 10:51

lancelly requested review from Tabrizian, YihuiLu512, chuangz0 and lori-ren July 15, 2026 10:51

Merge branch 'main' into laliao/nvtx-adp-disagg-sync-ranges

4f1abf5

mikeiovine removed the request for review from Tabrizian July 20, 2026 15:44

Merge branch 'main' into laliao/nvtx-adp-disagg-sync-ranges

74fdce8

Tabrizian approved these changes Jul 23, 2026

View reviewed changes

lancelly merged commit 83733c4 into NVIDIA:main Jul 23, 2026
7 checks passed

yuanjingx87 pushed a commit to yuanjingx87/TensorRT-LLM that referenced this pull request Jul 26, 2026

[None][chore] Add NVTX ranges to per-iteration ADP sync points in PyE…

d76b0b1

…xecutor (NVIDIA#16422) Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>

Uh oh!

Conversation

lancelly commented Jul 15, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Summary by CodeRabbit

Uh oh!

lancelly commented Jul 15, 2026

Uh oh!

coderabbitai Bot commented Jul 15, 2026

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

lancelly commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

lancelly commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

tensorrt-cicd commented Jul 19, 2026

Uh oh!

lancelly commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

lancelly commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

lancelly commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

lancelly commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

lancelly commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

lancelly commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

lancelly commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

lancelly commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

lancelly commented Jul 23, 2026

Uh oh!

lancelly commented Jul 15, 2026 •

edited by coderabbitai Bot

Loading