[None][perf] executor: avoid deepcopy of prompt_token_ids on enqueue#14889
Merged
lancelly merged 1 commit intoJun 3, 2026
Merged
Conversation
`_enqueue_request` deep-copied `request.prompt_token_ids` on every request. The value is a flat list of token ids (immutable ints), so a shallow `list(...)` gives the same isolation far more cheaply -- nothing downstream mutates the list in place (the prompt-adapter path rebuilds a new list, and `tllm.Request` copies `input_token_ids` into its own C++ buffer). For long prompts this matters: `copy.deepcopy` of a large id list runs in Python holding the GIL on the enqueue path, stalling the executor loop. On a DeepSeek-V4 disaggregated run (~40k input tokens, c160) replacing it with `list(...)` collapsed the generation-worker per-iter tail (host_step_time p99 ~127ms -> ~63ms, p95 ~78ms -> ~50ms; p50 unchanged) and improved per-request output speed (SSE p25 +~13%) and throughput (+~9%). Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #51772 [ run ] triggered by Bot. Commit: |
yuxianq
approved these changes
Jun 3, 2026
juney-nvidia
approved these changes
Jun 3, 2026
Collaborator
|
PR_Github #51772 [ run ] completed with state |
4 tasks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
_enqueue_requestdeep-copiedrequest.prompt_token_idson every request. It's a flat list of token ids (immutable ints), so a shallowlist(...)gives the same isolation far more cheaply — nothing downstream mutates the list in place (the prompt-adapter path rebuilds a new list;tllm.Requestcopiesinput_token_idsinto its own C++ buffer). Also drops the now-unusedimport copy.