[None][perf] disagg: serialize Request input_token_ids as int32 bytes - #15134
Merged
Tabrizian merged 1 commit intoJun 10, 2026
Conversation
Member
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #52934 [ run ] triggered by Bot. Commit: |
lfr-0531
approved these changes
Jun 9, 2026
Collaborator
|
PR_Github #52934 [ run ] completed with state
|
Collaborator
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #53013 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #53013 [ run ] completed with state
|
Member
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #53170 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #53170 [ run ] completed with state
|
Member
Author
|
/bot skip --comment "the change has been validated ci failure is un related" |
Collaborator
|
PR_Github #53179 [ skip ] triggered by Bot. Commit: |
The nanobind Request __getstate__ returned input_token_ids as a Python list[int] (and __setstate__ cast it back to a vector), so every Request pickle/unpickle paid a PyLong per token. On the disaggregated generation path the Request is pickled on hot, GIL-held paths -- the per-iteration request broadcast across DP ranks and the RpcWorker submit IPC -- so for long (ISL-sized) prompts this dominated host-side time and starved the executor loop. Serialize input_token_ids as a raw little-endian int32 byte buffer in __getstate__/__setstate__ (vector<->bytes is a memcpy, no per-element Python objects). The pickle format is symmetric and only consumed within a single build (broadcast + IPC), so there is no cross-version concern. Measured (DSv4-Pro disagg, GB300, ISL ~40k): - Request pickle+unpickle: 1.21 ms -> 0.06 ms (~19x) - broadcast_requests (GEN executor loop): 54.6 ms -> 6.6 ms avg (~8x) Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>
Tabrizian
force-pushed
the
user/imant/dsv4-request-input-token-ids-bytes
branch
from
June 10, 2026 00:27
ec5c712 to
3017c57
Compare
Collaborator
|
PR_Github #53179 [ skip ] completed with state |
jiaganc
added a commit
to jiaganc/TensorRT-LLM
that referenced
this pull request
Jun 26, 2026
…NVIDIA#15134) Source-Commit: ec3da87a9b688016f8f76cc43be9cf30c92c868e Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
jiaganc
added a commit
to jiaganc/TensorRT-LLM
that referenced
this pull request
Jun 26, 2026
…NVIDIA#15134) Source-Commit: ec3da87 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
lancelly
added a commit
to lancelly/TensorRT-LLM
that referenced
this pull request
Jun 29, 2026
…t_token_ids
In disaggregated serving the orchestrator is a single asyncio event loop that,
per request, handles the context server's prompt_token_ids (~40k ints for long
agentic prompts) three times on that one loop: json.loads of the ctx response,
pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize
onto the generation request. At high concurrency this is the binding host
bottleneck and sits on the TTFT critical path.
Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes
(default off, like gen_strip_message_history): the orchestrator instructs context
workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context
request) to serialize prompt_token_ids into the context response as a
base64-encoded int32 buffer (one JSON string). The orchestrator then relays that
string verbatim onto the generation request without materializing the int list on
its event loop; the generation worker decodes it back. The encode runs on the N
context workers instead of the one orchestrator loop. Off => byte identical to
baseline.
Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent):
- c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%.
- c2600 (orchestrator-bound): +74% mean completions; raises the
sustainable-concurrency-before-collapse ceiling.
Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134
(executor IPC int32 bytes). Safe for text-only, non-harmony deployments.
Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
lancelly
added a commit
to lancelly/TensorRT-LLM
that referenced
this pull request
Jun 29, 2026
…t_token_ids
In disaggregated serving the orchestrator is a single asyncio event loop that,
per request, handles the context server's prompt_token_ids (~40k ints for long
agentic prompts) three times on that one loop: json.loads of the ctx response,
pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize
onto the generation request. At high concurrency this is the binding host
bottleneck and sits on the TTFT critical path.
Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes
(default off, like gen_strip_message_history): the orchestrator instructs context
workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context
request) to serialize prompt_token_ids into the context response as a
base64-encoded int32 buffer (one JSON string). The orchestrator then relays that
string verbatim onto the generation request without materializing the int list on
its event loop; the generation worker decodes it back. The encode runs on the N
context workers instead of the one orchestrator loop. Off => byte identical to
baseline.
Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent):
- c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%.
- c2600 (orchestrator-bound): +74% mean completions; raises the
sustainable-concurrency-before-collapse ceiling.
Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134
(executor IPC int32 bytes). Safe for text-only, non-harmony deployments.
Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
lancelly
added a commit
to lancelly/TensorRT-LLM
that referenced
this pull request
Jun 29, 2026
…t_token_ids
In disaggregated serving the orchestrator is a single asyncio event loop that,
per request, handles the context server's prompt_token_ids (~40k ints for long
agentic prompts) three times on that one loop: json.loads of the ctx response,
pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize
onto the generation request. At high concurrency this is the binding host
bottleneck and sits on the TTFT critical path.
Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes
(default off, like gen_strip_message_history): the orchestrator instructs context
workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context
request) to serialize prompt_token_ids into the context response as a
base64-encoded int32 buffer (one JSON string). The orchestrator then relays that
string verbatim onto the generation request without materializing the int list on
its event loop; the generation worker decodes it back. The encode runs on the N
context workers instead of the one orchestrator loop. Off => byte identical to
baseline.
Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent):
- c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%.
- c2600 (orchestrator-bound): +74% mean completions; raises the
sustainable-concurrency-before-collapse ceiling.
Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134
(executor IPC int32 bytes). Safe for text-only, non-harmony deployments.
Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
jiaganc
added a commit
to jiaganc/TensorRT-LLM
that referenced
this pull request
Jun 29, 2026
…NVIDIA#15134) Source-Commit: ec3da87 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nanobind Request getstate returned input_token_ids as a Python list[int] (and setstate cast it back to a vector), so every Request pickle/unpickle paid a PyLong per token. On the disaggregated generation path the Request is pickled on hot, GIL-held paths -- the per-iteration request broadcast across DP ranks and the RpcWorker submit IPC -- so for long (ISL-sized) prompts this dominated host-side time and starved the executor loop.
Serialize input_token_ids as a raw little-endian int32 byte buffer in getstate/setstate (vector<->bytes is a memcpy, no per-element Python objects). The pickle format is symmetric and only consumed within a single build (broadcast + IPC), so there is no cross-version concern.
Measured (DSv4-Pro disagg, GB300, ISL ~40k):
@coderabbitai summary
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.