[None][perf] disagg: keep GEN prompt_token_ids off the event loop (metadata-only + int32 lazy) - #16475
Closed
hyukn wants to merge 1 commit into
Closed
[None][perf] disagg: keep GEN prompt_token_ids off the event loop (metadata-only + int32 lazy)#16475hyukn wants to merge 1 commit into
hyukn wants to merge 1 commit into
Conversation
…tadata-only + int32 lazy) Reduces the GEN OpenAIServer's per-request front-end cost -- the ISL-sized prompt token array on the single asyncio event loop that dominates gen_preprocessing at high concurrency -- via two opt-in, correctness-guarded paths. 1. gen_tokids_metadata_only: the orchestrator sends the GEN worker only prompt_len; the GEN worker builds a length-only placeholder and generation uses the KV transferred from the context worker (the forward path never dereferences prompt token VALUES, and seq_len/position come from prompt_len). GUARDRAIL: this fast path is taken only when the context worker reported a prompt_len AND the request does not read prompt token VALUES -- sampling penalties (frequency / presence / repetition) or prompt echo -- which the placeholder would silently corrupt; otherwise it falls back to relaying the real tokens. Deployment constraints: text-only, non-harmony, and GEN KV block reuse OFF (block-reuse keys blocks by token-value hash; a placeholder would collide across prompts and reuse the wrong KV). 2. int32-ndarray lazy submit path (correct for ALL requests). The GEN server keeps the base64-decoded prompt_token_ids as an int32 ndarray (drops the O(ISL) .tolist()); prompt_inputs gains a 1-D ndarray branch; LLM._preprocess guards prompt_token_ids with `is not None` (an ndarray truth value is ambiguous); executor.generate_async accepts np.integer; and GenerationRequest stashes the ndarray as the lazy `_prompt_token_ids_i32` buffer, memcpy'd into the C++ Request ctor. The Python list is built lazily only if a consumer reads the values (prompt logprobs); the plain disagg-gen path never does. Unlike the placeholder this carries the REAL tokens, so it is correct under all sampling features and composes with the base64 relay (NVIDIA#15698) and the ctor memcpy (NVIDIA#15211): CTX base64 -> orchestrator string relay -> GEN int32 ndarray (lazy) -> C++ memcpy, with no Python list built anywhere on the path. Both opt-in (default off). Note: the int32 path does not remove the ISL request body from the wire (only metadata-only does) and neither addresses the SSE output-streaming load on the same event loop; the dominant "wait" (queuing) at very high concurrency needs metadata-only body removal for all requests (off-loop real-token delivery) and/or multiple front-end processes. Tested: CPU unit tests (int32 laziness + real-value read + pickle round-trip; prompt_inputs ndarray preservation; the metadata-only guardrail predicate). Needs a GPU disagg smoke test for end-to-end TTFT/throughput/accuracy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> (cherry picked from commit 6da6e5b)
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #59636 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #59636 [ run ] completed with state
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The GEN OpenAIServer front-end is a single asyncio event loop; at high concurrency the ∝ISL prompt-token array per request dominates gen_preprocessing (the queuing "wait" before the engine admits a request). This PR cuts that via two opt-in, correctness-guarded paths (both default off, so no behavior change unless enabled):
gen_tokids_metadata_only— the orchestrator sends the GEN worker onlyprompt_len; GEN builds a length-only placeholder and generation uses the KV transferred from the context worker. The forward path never dereferences prompt token VALUES (seq_len/position come fromprompt_len; the decode boundary token arrives separately viadisaggregated_params.first_gen_tokens). Guardrail: taken only when CTX reported aprompt_lenAND the request does not read prompt token values (frequency/presence/repetition penalty, echo) — otherwise it falls back to relaying real tokens. Deployment constraints: text-only, non-harmony, GEN KV block reuse OFF.prompt_token_idsas an int32 ndarray (drops the O(ISL).tolist()on the GIL-held submit path); the Python list is materialized lazily only if a consumer reads the values. Composes with the base64 relay ([None][perf] disagg: opt-in ctx-side base64 int32 transport for prompt_token_ids #15698) and the ctor memcpy ([None][perf] executor: memcpy int32 token buffer into tle::Request ctor #15211).Perf evidence (DSv4-Pro disagg, c3120, 60 GPU, GB300; isolated metadata_only ON vs OFF, same build, N=1)
Correctness
first_gen_tokens) and GEN continues from that real token using the transferred KV.Tests
prompt_inputsndarray preservation; the metadata-only guardrail predicate. (14 passed)Follow-ups before ready-for-review
no_repeat_ngram_size/prompt_logprobs/ the/v1/completionspath are covered or N/A.🤖 Generated with Claude Code