[None][perf] avoid full input_token_ids copy in ADP router token count - #15362
[None][perf] avoid full input_token_ids copy in ADP router token count#15362lancelly wants to merge 1 commit into
Conversation
The ADP router only needs the input length to load-balance new requests, but it read ExecutorRequest.input_token_ids and called len() on it. That C++ getter returns VecTokens by value, so every access materializes a fresh ~ISL-sized Python list (one PyObject per token) under the GIL -- redundantly on every rank (routing is symmetric) for every new request. At high concurrency with long context (e.g. DeepSeek-V4 disagg generation, avg ISL ~39k) this is a measurable rank-0 host cost. Add an O(1) Request::getNumInputTokens() that reads mInputTokenIds.size() directly -- no VecTokens copy and no Python list -- and bind it as the num_input_tokens property. Route the Default/KVCacheAware token-count reads through a _num_input_tokens() helper backed by that accessor. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #54222 [ run ] triggered by Bot. Commit: |
|
PR_Github #54222 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54255 [ run ] triggered by Bot. Commit: |
|
PR_Github #54255 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54287 [ run ] triggered by Bot. Commit: |
|
PR_Github #54287 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54422 [ run ] triggered by Bot. Commit: |
|
PR_Github #54422 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54536 [ run ] triggered by Bot. Commit: |
|
PR_Github #54536 [ run ] completed with state
|
|
We no longer use the ds v4 side branch. Closing this PR. Please prepare PR against main. |
3 participants
Summary
The ADP request router only needs the input length to load-balance new
requests across attention-DP ranks, but it read
ExecutorRequest.input_token_idsand called
len()on it.Request::getInputTokenIds()returnsVecTokensby value, so each access materializes a fresh ~ISL-sized Python list (one
PyObject per token) under the GIL — redundantly on every rank (routing is
symmetric), for every new request.
On DeepSeek-V4 disaggregated generation at high concurrency (avg ISL ~39k), the
per-iteration
_fetch_new_requestshost path is a known rank-0 GIL hotspot.Change
Request::getNumInputTokens()that readsmInputTokenIds.size()directly — no
VecTokenscopy and no Python list — and bind it as thenum_input_tokensnanobind property.DefaultADPRouter/KVCacheAwareADPRoutertoken-count readsthrough a small
_num_input_tokens()helper backed by that accessor.Test
TestNumInputTokenscovers the helper (None request, prefers the accessorwithout touching
input_token_ids)._MockRequestwhosenum_input_tokensmirrorslen(input_token_ids).Notes
(no Python fallback).
feat/deepseek_v4_bench.