[None][perf] avoid full input_token_ids copy in ADP router token count - #15199
Conversation
21d8e77 to
9b5ca6f
Compare
The ADP router only needs the input length to load-balance new requests, but it read ExecutorRequest.input_token_ids and called len() on it. That C++ getter returns VecTokens by value, so every access materializes a fresh ~ISL-sized Python list (one PyObject per token) under the GIL -- redundantly on every rank (routing is symmetric) for every new request. At high concurrency with long context (e.g. DeepSeek-V4 disagg generation, avg ISL ~39k) this is a measurable rank-0 host cost. Add an O(1) Request::getNumInputTokens() that reads mInputTokenIds.size() directly -- no VecTokens copy and no Python list -- and bind it as the num_input_tokens property. Route the Default/KVCacheAware token-count reads through a _num_input_tokens() helper backed by that accessor. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
9b5ca6f to
cca1ca0
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #54220 [ run ] triggered by Bot. Commit: |
|
PR_Github #54220 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54242 [ run ] triggered by Bot. Commit: |
|
PR_Github #54242 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
1 similar comment
|
/bot run --disable-fail-fast |
|
PR_Github #54278 [ run ] triggered by Bot. Commit: |
|
PR_Github #54279 [ run ] triggered by Bot. Commit: |
|
PR_Github #54278 [ run ] completed with state |
|
PR_Github #54279 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54424 [ run ] triggered by Bot. Commit: |
|
PR_Github #54424 [ run ] completed with state |
NVIDIA#15199) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
NVIDIA#15199) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
NVIDIA#15199) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>
NVIDIA#15199) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Signed-off-by: Liao Lanyu <108499334+lancelly@users.noreply.github.com>
2 participants
Summary
Follow-on to #15133. The ADP request router only needs the input length to
load-balance new requests across attention-DP ranks, but it read
ExecutorRequest.input_token_idsand calledlen()on it.Request::getInputTokenIds()returnsVecTokensby value, so each accessmaterializes a fresh ~ISL-sized Python list (one PyObject per token) — under the
GIL, redundantly on every rank (routing is symmetric), for every new
request. #15133 reduced the access from 2→1 per request; this removes the
copy entirely by counting the tokens without materializing the list.
Change
num_input_tokensaccessor to theRequestnanobind binding(returns
getInputTokenIds().size()— no Python list is built).DefaultADPRouter/KVCacheAwareADPRoutertoken-count readsthrough a
_num_input_tokens()helper. It falls back tolen(input_token_ids)when the accessor is absent (a Python-only deploy whoseC++ bindings predate it, or test mocks), so routing results are unchanged.
Why it matters
On DeepSeek-V4-Pro disaggregated generation @ 2880 concurrency (avg ISL ~39k),
the per-iteration
_fetch_new_requestshost path is a known rank-0 GILhotspot. Each new request previously triggered a full ~39k-element Python list
build on all 8 ranks just to obtain its length.
Test
Adds
TestNumInputTokens: prefers the accessor (and assertsinput_token_idsis not materialized), falls back to
len()when absent, ignores a non-intaccessor, and handles a
Nonerequest.Notes
fallback keeps behavior/CI unchanged without it.
Request::getNumInputTokens()on the C++ sidewould avoid even the (cheap) internal
VecTokenscopy that.size()incurs.