[None][perf] executor: memcpy int32 token buffer into tle::Request ctor by hyukn · Pull Request #15211 · NVIDIA/TensorRT-LLM

hyukn · 2026-06-10T08:25:45Z

Description

RpcWorker.submit → BaseWorker._enqueue_request builds a tle::Request from prompt_token_ids on a GIL-held thread. With a Python list[int], nanobind casts it to std::vector<int32> element-by-element (one PyLong read per token) — O(ISL). In the decode phase (short forward steps, host-bound iteration) this stalls the PyExecutor loop on the GIL.

This PR makes the token ids cross the proxy→worker boundary as an int32 buffer and memcpys them into the C++ vector, and avoids rebuilding the Python list[int] on the decode path entirely — while keeping prompt_token_ids a plain list[int] for every Python consumer that does read it.

What it does (3 parts)

request.cpp — tle::Request ctor gains a buffer fast-path (toVecTokens): a 1-D contiguous int32 ndarray is memcpy'd into VecTokens; list[int] still works via the default cast (back-compatible).
request.py — GenerationRequest.__getstate__/__setstate__ encode token-id lists as int32 bytes on the wire (no per-token PyLong frame). On decode they do not eagerly rebuild the list: the int32 ndarray is stashed in a private _prompt_token_ids_i32 side-channel for the memcpy ctor, and prompt_token_ids is exposed as a property that materializes the list[int] lazily (and caches it) only if a consumer actually reads it. Plain decode never reads it → the O(ISL) .tolist() never runs.
base_worker.py — _enqueue_request hands _prompt_token_ids_i32 to the memcpy ctor when present (no prompt adapter); otherwise the list path.

prompt_token_ids always returns list[int], so list-assuming consumers (prompt-logprobs concat, tracing truthiness, star-attention, the List[int] result property) are unaffected.

Why a lazy property (not ndarray-everywhere, not `getattr`)

Not ndarray-everywhere: making prompt_token_ids itself an ndarray breaks list[int] consumers (tokens[1:] + first silently broadcasts; if <ndarray> raises). The property keeps the List[int] contract and confines the buffer to the C++ ctor.
Lazy: the decode hot path submits via the int32 buffer and never touches the list, so eagerly rebuilding it in __setstate__ would re-add an O(ISL) .tolist() per request for nothing. Materializing on first read avoids it.
Property, not a class-level __getattr__: a __getattr__ is invoked by the interpreter on every missing-attribute access on the object (hasattr/getattr(default) probes, copy/pickle dunder lookups, duck-typing), each paying a Python frame + an AttributeError raise instead of the C-level miss. A property is scoped to this one attribute → identical lazy behavior, zero blast radius. (microbench, ISL=40k: one attribute miss 0.06 µs with a property vs 0.51 µs with __getattr__.)

Measured (nsys, ctx-only, rank0 RPC worker, base vs fix)

NVTX range	base	fix	Δ
`rpc_worker_submit` (`_enqueue_request`)	555 µs	157 µs	−72%
`rpc_token_prep`	127 µs	~0 (buffer; no list copy)	−127
`rpc_request_ctor`	271 µs	117 µs (token cast → memcpy)	−154
`rpc_engine_enqueue`	18 µs	17 µs	~0

Confirmed in the decode-exposed regime (disagg GEN worker, c2880, all 8 ranks): RpcWorker.submit ≈ 146 µs (rank0) and GPU idle between decode iters drops 14.9% → 11.1%, with no regression vs the equivalent eager/__getattr__ variants.

Test Coverage

Serialization round-trip (__getstate__/__setstate__) validated: prompt_token_ids returns the original list[int] (lazily), _prompt_token_ids_i32 matches, re-pickle of an un-read request round-trips, empty-list and query_token_ids paths covered, and a missing attribute raises a plain AttributeError (no __getattr__).
tests/unittest/bindings/test_executor_bindings.py: add a case constructing Request(input_token_ids=<int32 ndarray>) and asserting round-trip equivalence with list[int] + fallback (empty / non-int32 / non-contiguous). (TODO in this PR.)

Notes

C++ change requires a rebuild; the ABI base is feat/deepseek_v4 (= f04 + [None][perf] reduce rank-0 GIL contention in disaggregated generation #15133 + [None][perf] disagg: serialize Request input_token_ids as int32 bytes #15134).
Self-contained — no dependency on a separate serialization PR.

🤖 Generated with Claude Code

hyukn · 2026-06-10T09:06:09Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-10T09:13:14Z

PR_Github #53284 [ run ] triggered by Bot. Commit: 8fe0a80 Link to invocation

tensorrt-cicd · 2026-06-10T12:22:29Z

PR_Github #53284 [ run ] completed with state SUCCESS. Commit: 8fe0a80
/LLM/main/L0_MergeRequest_PR pipeline #42474 completed with status: 'SUCCESS'

CI Report

Link to invocation

hyukn · 2026-06-11T14:59:03Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-11T15:07:47Z

PR_Github #53595 [ run ] triggered by Bot. Commit: 28979a1 Link to invocation

tensorrt-cicd · 2026-06-11T18:08:44Z

PR_Github #53595 [ run ] completed with state SUCCESS. Commit: 28979a1
/LLM/main/L0_MergeRequest_PR pipeline #42741 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-11T23:54:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-12T00:01:19Z

PR_Github #53718 [ run ] triggered by Bot. Commit: 28979a1 Link to invocation

tensorrt-cicd · 2026-06-12T02:02:17Z

PR_Github #53718 [ run ] completed with state SUCCESS. Commit: 28979a1
/LLM/main/L0_MergeRequest_PR pipeline #42847 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-12T02:45:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-12T02:51:49Z

PR_Github #53762 [ run ] triggered by Bot. Commit: 28979a1 Link to invocation

tensorrt-cicd · 2026-06-12T04:32:28Z

PR_Github #53762 [ run ] completed with state FAILURE. Commit: 28979a1
/LLM/main/L0_MergeRequest_PR pipeline #42882 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

On the hot proxy->worker RPC-submit path, constructing tle::Request from a list[int] of prompt token-ids costs O(ISL): pickling emits one PyLong frame per token on the wire, and the nanobind ctor does an element-wise cast into VecTokens -- all on the GIL-held submit thread. - request.cpp: add `toVecTokens`, which memcpy's a 1-D contiguous int32 ndarray straight into VecTokens (list[int] falls back to the element-wise cast, so the binding stays backward compatible). - request.py: GenerationRequest.__getstate__/__setstate__ encode token-ids as int32 bytes for the wire (no per-token PyLong frame). On decode the int32 ndarray is stashed in `_prompt_token_ids_i32` for the ctor memcpy and the list is NOT eagerly rebuilt; the `prompt_token_ids` list is materialized lazily-and-cached via a property only if a consumer actually reads it. Implemented as a property (scoped to this one attribute) rather than a class-level __getattr__, which would fire on every missing-attribute access (hasattr/getattr-default probes, pickle/ copy dunder lookups) -- same lazy behavior, zero blast radius on other attributes. - base_worker._enqueue_request: hand the int32 buffer to the ctor when present; prompt_token_ids stays a list[int] for all other consumers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

hyukn · 2026-06-14T17:09:10Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-14T17:15:02Z

PR_Github #54141 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-14T18:30:28Z

PR_Github #54141 [ run ] completed with state SUCCESS. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43224 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-15T01:27:59Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-15T01:33:41Z

PR_Github #54172 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-15T04:33:57Z

PR_Github #54172 [ run ] completed with state SUCCESS. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43255 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-15T11:01:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-15T11:08:11Z

PR_Github #54290 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-15T12:51:25Z

PR_Github #54290 [ run ] completed with state SUCCESS. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43361 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-16T01:44:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T01:51:15Z

PR_Github #54405 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-16T04:01:27Z

PR_Github #54405 [ run ] completed with state FAILURE. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43469 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-16T04:30:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T04:37:12Z

PR_Github #54459 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-16T06:18:57Z

PR_Github #54459 [ run ] completed with state FAILURE. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43523 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-16T07:42:13Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T07:47:57Z

PR_Github #54534 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-16T11:22:25Z

PR_Github #54534 [ run ] completed with state FAILURE. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43588 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-06-16T12:21:22Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T12:28:32Z

PR_Github #54583 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-16T15:13:15Z

PR_Github #54583 [ run ] completed with state SUCCESS. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43626 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-16T23:36:48Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T23:43:34Z

PR_Github #54688 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-17T01:30:31Z

PR_Github #54688 [ run ] completed with state FAILURE. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43718 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

hyukn · 2026-06-17T01:51:30Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-17T01:57:22Z

PR_Github #54702 [ run ] triggered by Bot. Commit: 57aaadf Link to invocation

tensorrt-cicd · 2026-06-17T04:23:47Z

PR_Github #54702 [ run ] completed with state SUCCESS. Commit: 57aaadf
/LLM/main/L0_MergeRequest_PR pipeline #43732 completed with status: 'SUCCESS'

CI Report

Link to invocation

…or (NVIDIA#15211) Source-Commit: c4cd713 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

…event loop Activates the Phase 1 lazy int32 submit path end to end. The GEN OpenAIServer now keeps the base64-decoded prompt_token_ids as an int32 ndarray (drops the O(ISL) `.tolist()`), and the metadata-only placeholder uses `np.full` (a single C-level fill) instead of `[pad] * N`. The ndarray flows through unchanged: * inputs.prompt_inputs: new 1-D ndarray branch -> TokensPrompt (no copy). * LLM._preprocess: the multimodal fast-path guard now tests `prompt_token_ids is not None` instead of truthiness (a numpy array has an ambiguous truth value); the text-only path already extracts by key. * executor.generate_async accepts np.integer[0] (Phase 1) and GenerationRequest stashes the int32 ndarray as the lazy `_prompt_token_ids_i32` buffer, memcpy'd into the C++ Request ctor. Net effect: on the single GEN asyncio event loop -- which serializes every concurrent request and dominates gen_preprocessing/TTFT at high concurrency -- the prompt token array is never materialized into a Python list. The list is built lazily only if a consumer reads the values (prompt logprobs); the plain disagg-gen forward path never does. Unlike the length-only placeholder this carries the REAL tokens, so it is correct for all sampling features and composes with the base64 relay (NVIDIA#15698) and the C++ ctor memcpy (NVIDIA#15211): CTX base64 -> orchestrator string relay -> GEN int32 ndarray (lazy) -> C++ memcpy, no Python list on the path. `request.prompt_token_ids` (ChatCompletionRequest, OpenAIBaseModel: no validate_assignment) keeps the ndarray on assignment. Needs a GPU smoke test to confirm end-to-end TTFT/throughput; CPU unit test covers ndarray preservation through prompt_inputs + the lazy GenerationRequest path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

…tadata-only + int32 lazy) Reduces the GEN OpenAIServer's per-request front-end cost -- the ISL-sized prompt token array on the single asyncio event loop that dominates gen_preprocessing at high concurrency -- via two opt-in, correctness-guarded paths. 1. gen_tokids_metadata_only: the orchestrator sends the GEN worker only prompt_len; the GEN worker builds a length-only placeholder and generation uses the KV transferred from the context worker (the forward path never dereferences prompt token VALUES, and seq_len/position come from prompt_len). GUARDRAIL: this fast path is taken only when the context worker reported a prompt_len AND the request does not read prompt token VALUES -- sampling penalties (frequency / presence / repetition) or prompt echo -- which the placeholder would silently corrupt; otherwise it falls back to relaying the real tokens. Deployment constraints: text-only, non-harmony, and GEN KV block reuse OFF (block-reuse keys blocks by token-value hash; a placeholder would collide across prompts and reuse the wrong KV). 2. int32-ndarray lazy submit path (correct for ALL requests). The GEN server keeps the base64-decoded prompt_token_ids as an int32 ndarray (drops the O(ISL) .tolist()); prompt_inputs gains a 1-D ndarray branch; LLM._preprocess guards prompt_token_ids with `is not None` (an ndarray truth value is ambiguous); executor.generate_async accepts np.integer; and GenerationRequest stashes the ndarray as the lazy `_prompt_token_ids_i32` buffer, memcpy'd into the C++ Request ctor. The Python list is built lazily only if a consumer reads the values (prompt logprobs); the plain disagg-gen path never does. Unlike the placeholder this carries the REAL tokens, so it is correct under all sampling features and composes with the base64 relay (NVIDIA#15698) and the ctor memcpy (NVIDIA#15211): CTX base64 -> orchestrator string relay -> GEN int32 ndarray (lazy) -> C++ memcpy, with no Python list built anywhere on the path. Both opt-in (default off). Note: the int32 path does not remove the ISL request body from the wire (only metadata-only does) and neither addresses the SSE output-streaming load on the same event loop; the dominant "wait" (queuing) at very high concurrency needs metadata-only body removal for all requests (off-loop real-token delivery) and/or multiple front-end processes. Tested: CPU unit tests (int32 laziness + real-value read + pickle round-trip; prompt_inputs ndarray preservation; the metadata-only guardrail predicate). Needs a GPU disagg smoke test for end-to-end TTFT/throughput/accuracy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

github-actions Bot assigned hyukn Jun 10, 2026

hyukn force-pushed the perf/request-ctor-int32-buffer branch 3 times, most recently from 1684a55 to 28979a1 Compare June 11, 2026 08:31

hyukn force-pushed the perf/request-ctor-int32-buffer branch from 28979a1 to a7ebe37 Compare June 13, 2026 12:08

hyukn force-pushed the perf/request-ctor-int32-buffer branch from a3699f2 to 57aaadf Compare June 14, 2026 15:58

hyukn marked this pull request as ready for review June 15, 2026 03:07

hyukn requested a review from a team as a code owner June 15, 2026 03:07

hyukn requested review from schetlur-nv and removed request for a team June 15, 2026 03:07

hyukn added the deepseek-v4 label Jun 15, 2026

hyukn requested a review from lfr-0531 June 17, 2026 04:27

lfr-0531 merged commit c4cd713 into NVIDIA:feat/deepseek_v4 Jun 17, 2026
9 checks passed

lfr-0531 approved these changes Jun 17, 2026

View reviewed changes

jiaganc added a commit to jiaganc/TensorRT-LLM that referenced this pull request Jun 26, 2026

[None][perf] executor: memcpy int32 token buffer into tle::Request ct…

d23b687

…or (NVIDIA#15211) Source-Commit: c4cd713 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

jiaganc added a commit to jiaganc/TensorRT-LLM that referenced this pull request Jun 26, 2026

[None][perf] executor: memcpy int32 token buffer into tle::Request ct…

19f3389

…or (NVIDIA#15211) Source-Commit: c4cd713 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

jiaganc added a commit to jiaganc/TensorRT-LLM that referenced this pull request Jun 29, 2026

[None][perf] executor: memcpy int32 token buffer into tle::Request ct…

4bf4703

…or (NVIDIA#15211) Source-Commit: c4cd713 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

hyukn mentioned this pull request Jul 16, 2026

[None][perf] disagg: keep GEN prompt_token_ids off the event loop (metadata-only + int32 lazy) #16475

Closed

2 tasks

Uh oh!

Conversation

hyukn commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What it does (3 parts)

Why a lazy property (not ndarray-everywhere, not __getattr__)

Measured (nsys, ctx-only, rank0 RPC worker, base vs fix)

Test Coverage

Notes

Uh oh!

hyukn commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

hyukn commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

hyukn commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

hyukn commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

hyukn commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

tensorrt-cicd commented Jun 14, 2026

Uh oh!

hyukn commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

hyukn commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

hyukn commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hyukn commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hyukn commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

lfr-0531 commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hyukn commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

hyukn commented Jun 10, 2026 •

edited

Loading

Why a lazy property (not ndarray-everywhere, not `getattr`)