[None][perf] disagg: opt-in ctx-side base64 int32 transport for prompt_token_ids by lancelly · Pull Request #15698 · NVIDIA/TensorRT-LLM

lancelly · 2026-06-28T23:26:26Z

What

In disaggregated serving the orchestrator (trtllm-serve disaggregated) is a single asyncio event loop. Per request it handles the context server's prompt_token_ids (~40k ints for long agentic prompts) three times on that one loop:

json.loads of the ctx response body (openai_client._post_with_retry),
pydantic-validate into ChatCompletionResponse.prompt_token_ids (openai_client response_type(**resp_json)),
model_dump_json re-serialize onto the generation request (openai_client).

At high concurrency this is the binding host bottleneck (the orchestrator loop saturates), and it sits on the TTFT critical path.

This PR adds an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes (default off): the orchestrator instructs context workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context request) to serialize prompt_token_ids into the context response as a base64-encoded int32 buffer (one JSON string); the orchestrator then relays that string verbatim onto the generation request without ever materializing the int list (no np.asarray, no int-array (de)serialization); the generation worker decodes it back. The encode is thereby spread across the N context workers instead of concentrated on the one orchestrator loop. Off ⇒ byte-identical to baseline.

New fields: prompt_token_ids_b64: Optional[str] on ChatCompletionResponse and ChatCompletionRequest; return_prompt_token_ids_b64: bool on DisaggregatedParams (orchestrator→ctx instruction); gen_tokids_ctxbytes: bool on DisaggServerConfig.

Why (measured: DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent dataset)

c2048 (stable, GPU-bound): TTFT p50 −57% (2.95s→1.28s), p95 −41% (consistent across reps); throughput flat (GPU-bound there, so the orchestrator relief shows purely as lower first-token latency on the critical path).
c2600 (orchestrator-bound, multi-rep): +74% mean completions; raises the sustainable-concurrency-before-collapse ceiling (baseline collapses ~2/3 runs near the cliff; the fast path stays healthy).

A counter-experiment that pins the mechanism: encoding the same base64 on the orchestrator loop instead (np.asarray there) regresses ~6× — it adds work to the binding loop. This PR does the opposite (encode on the ctx workers, orchestrator only relays a string)

…t_token_ids In disaggregated serving the orchestrator is a single asyncio event loop that, per request, handles the context server's prompt_token_ids (~40k ints for long agentic prompts) three times on that one loop: json.loads of the ctx response, pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize onto the generation request. At high concurrency this is the binding host bottleneck and sits on the TTFT critical path. Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes (default off, like gen_strip_message_history): the orchestrator instructs context workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context request) to serialize prompt_token_ids into the context response as a base64-encoded int32 buffer (one JSON string). The orchestrator then relays that string verbatim onto the generation request without materializing the int list on its event loop; the generation worker decodes it back. The encode runs on the N context workers instead of the one orchestrator loop. Off => byte identical to baseline. Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent): - c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%. - c2600 (orchestrator-bound): +74% mean completions; raises the sustainable-concurrency-before-collapse ceiling. Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134 (executor IPC int32 bytes). Safe for text-only, non-harmony deployments. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Tabrizian · 2026-06-29T22:19:53Z

/bot run --disable-fail-fast

Tabrizian · 2026-06-29T22:23:30Z

+    # Ask context workers to return prompt_token_ids as a base64 int32 buffer so
+    # the orchestrator relays a string instead of materializing the token-id list
+    # on its event loop. Text-only, non-harmony deployments (see _get_ctx_request).
+    gen_tokids_ctxbytes: bool = False


Should we always do this, instead of having a separate parameter?

If AgentPerf needs this change now, we can merge the current PR first. As for whether to enable it by default or remove this parameter, I need to run a few more benchmarks. What do you think?

Tabrizian · 2026-06-29T22:23:45Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-29T22:26:57Z

PR_Github #56473 [ run ] triggered by Bot. Commit: 7c13fc1 Link to invocation

tensorrt-cicd · 2026-06-29T22:30:36Z

PR_Github #56475 [ run ] triggered by Bot. Commit: 7c13fc1 Link to invocation

tensorrt-cicd · 2026-06-29T22:34:21Z

PR_Github #56473 [ run ] completed with state ABORTED. Commit: 7c13fc1

Link to invocation

lancelly · 2026-06-30T01:03:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-30T01:13:02Z

PR_Github #56475 [ run ] completed with state FAILURE. Commit: 7c13fc1
/LLM/main/L0_MergeRequest_PR pipeline #45316 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-06-30T01:16:31Z

/bot run --disable-fail-fast

lancelly · 2026-06-30T01:38:04Z

/bot run --disable-fail-fast

lancelly · 2026-06-30T14:04:10Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-30T14:11:13Z

PR_Github #56615 [ run ] triggered by Bot. Commit: 7c13fc1 Link to invocation

tensorrt-cicd · 2026-06-30T15:17:33Z

PR_Github #56615 [ run ] completed with state SUCCESS. Commit: 7c13fc1
/LLM/main/L0_MergeRequest_PR pipeline #45442 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

pcastonguay · 2026-06-30T17:27:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-30T17:34:27Z

PR_Github #56677 [ run ] triggered by Bot. Commit: 7c13fc1 Link to invocation

tensorrt-cicd · 2026-06-30T18:58:18Z

PR_Github #56677 [ run ] completed with state SUCCESS. Commit: 7c13fc1
/LLM/main/L0_MergeRequest_PR pipeline #45500 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-06-30T23:30:16Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-30T23:36:22Z

PR_Github #56757 [ run ] triggered by Bot. Commit: 31bf984 Link to invocation

tensorrt-cicd · 2026-07-01T02:32:28Z

PR_Github #56757 [ run ] completed with state SUCCESS. Commit: 31bf984
/LLM/main/L0_MergeRequest_PR pipeline #45577 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-07-01T06:03:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-01T06:09:56Z

PR_Github #56856 [ run ] triggered by Bot. Commit: 8823bb5 Link to invocation

tensorrt-cicd · 2026-07-01T09:15:34Z

PR_Github #56856 [ run ] completed with state SUCCESS. Commit: 8823bb5
/LLM/main/L0_MergeRequest_PR pipeline #45668 completed with status: 'SUCCESS'

CI Report

Link to invocation

…t_token_ids (NVIDIA#15698) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com> Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>

…event loop Activates the Phase 1 lazy int32 submit path end to end. The GEN OpenAIServer now keeps the base64-decoded prompt_token_ids as an int32 ndarray (drops the O(ISL) `.tolist()`), and the metadata-only placeholder uses `np.full` (a single C-level fill) instead of `[pad] * N`. The ndarray flows through unchanged: * inputs.prompt_inputs: new 1-D ndarray branch -> TokensPrompt (no copy). * LLM._preprocess: the multimodal fast-path guard now tests `prompt_token_ids is not None` instead of truthiness (a numpy array has an ambiguous truth value); the text-only path already extracts by key. * executor.generate_async accepts np.integer[0] (Phase 1) and GenerationRequest stashes the int32 ndarray as the lazy `_prompt_token_ids_i32` buffer, memcpy'd into the C++ Request ctor. Net effect: on the single GEN asyncio event loop -- which serializes every concurrent request and dominates gen_preprocessing/TTFT at high concurrency -- the prompt token array is never materialized into a Python list. The list is built lazily only if a consumer reads the values (prompt logprobs); the plain disagg-gen forward path never does. Unlike the length-only placeholder this carries the REAL tokens, so it is correct for all sampling features and composes with the base64 relay (NVIDIA#15698) and the C++ ctor memcpy (NVIDIA#15211): CTX base64 -> orchestrator string relay -> GEN int32 ndarray (lazy) -> C++ memcpy, no Python list on the path. `request.prompt_token_ids` (ChatCompletionRequest, OpenAIBaseModel: no validate_assignment) keeps the ndarray on assignment. Needs a GPU smoke test to confirm end-to-end TTFT/throughput; CPU unit test covers ndarray preservation through prompt_inputs + the lazy GenerationRequest path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

…tadata-only + int32 lazy) Reduces the GEN OpenAIServer's per-request front-end cost -- the ISL-sized prompt token array on the single asyncio event loop that dominates gen_preprocessing at high concurrency -- via two opt-in, correctness-guarded paths. 1. gen_tokids_metadata_only: the orchestrator sends the GEN worker only prompt_len; the GEN worker builds a length-only placeholder and generation uses the KV transferred from the context worker (the forward path never dereferences prompt token VALUES, and seq_len/position come from prompt_len). GUARDRAIL: this fast path is taken only when the context worker reported a prompt_len AND the request does not read prompt token VALUES -- sampling penalties (frequency / presence / repetition) or prompt echo -- which the placeholder would silently corrupt; otherwise it falls back to relaying the real tokens. Deployment constraints: text-only, non-harmony, and GEN KV block reuse OFF (block-reuse keys blocks by token-value hash; a placeholder would collide across prompts and reuse the wrong KV). 2. int32-ndarray lazy submit path (correct for ALL requests). The GEN server keeps the base64-decoded prompt_token_ids as an int32 ndarray (drops the O(ISL) .tolist()); prompt_inputs gains a 1-D ndarray branch; LLM._preprocess guards prompt_token_ids with `is not None` (an ndarray truth value is ambiguous); executor.generate_async accepts np.integer; and GenerationRequest stashes the ndarray as the lazy `_prompt_token_ids_i32` buffer, memcpy'd into the C++ Request ctor. The Python list is built lazily only if a consumer reads the values (prompt logprobs); the plain disagg-gen path never does. Unlike the placeholder this carries the REAL tokens, so it is correct under all sampling features and composes with the base64 relay (NVIDIA#15698) and the ctor memcpy (NVIDIA#15211): CTX base64 -> orchestrator string relay -> GEN int32 ndarray (lazy) -> C++ memcpy, with no Python list built anywhere on the path. Both opt-in (default off). Note: the int32 path does not remove the ISL request body from the wire (only metadata-only does) and neither addresses the SSE output-streaming load on the same event loop; the dominant "wait" (queuing) at very high concurrency needs metadata-only body removal for all requests (off-loop real-token delivery) and/or multiple front-end processes. Tested: CPU unit tests (int32 laziness + real-value read + pickle round-trip; prompt_inputs ndarray preservation; the metadata-only guardrail predicate). Needs a GPU disagg smoke test for end-to-end TTFT/throughput/accuracy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

github-actions Bot assigned lancelly Jun 28, 2026

lancelly force-pushed the feat/disagg-ctx-tokids-b64 branch 2 times, most recently from 0b9a730 to 8983ba7 Compare June 29, 2026 04:20

lancelly force-pushed the feat/disagg-ctx-tokids-b64 branch from 8983ba7 to 7c13fc1 Compare June 29, 2026 04:28

Tabrizian marked this pull request as ready for review June 29, 2026 22:22

Tabrizian requested a review from a team as a code owner June 29, 2026 22:22

Tabrizian requested review from JunyiXu-nv and removed request for a team June 29, 2026 22:22

Tabrizian reviewed Jun 29, 2026

View reviewed changes

Merge branch 'feat/deepseek_v4' into feat/disagg-ctx-tokids-b64

31bf984

Merge branch 'feat/deepseek_v4' into feat/disagg-ctx-tokids-b64

8823bb5

Tabrizian merged commit dfb0609 into NVIDIA:feat/deepseek_v4 Jul 1, 2026
6 checks passed

hyukn mentioned this pull request Jul 16, 2026

[None][perf] disagg: keep GEN prompt_token_ids off the event loop (metadata-only + int32 lazy) #16475

Closed

2 tasks

Uh oh!

Conversation

lancelly commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why (measured: DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent dataset)

Uh oh!

Tabrizian commented Jun 29, 2026

Uh oh!

Tabrizian Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

lancelly Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Tabrizian commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

lancelly commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

lancelly commented Jun 30, 2026

Uh oh!

lancelly commented Jun 30, 2026

Uh oh!

lancelly commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

pcastonguay commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

lancelly commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jun 30, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

lfr-0531 commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lancelly commented Jun 28, 2026 •

edited

Loading