Skip to content

[NVBUG-6448152][test] isolate double disagg admission interaction - #17086

Closed
chienchunhung wants to merge 12 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-b6-capacity-exact
Closed

[NVBUG-6448152][test] isolate double disagg admission interaction#17086
chienchunhung wants to merge 12 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-b6-capacity-exact

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

TEST ONLY — DO NOT MERGE

This diagnostic isolates whether the C++/native transceiver regression introduced by the admission and bounded-polling change is caused by serial enforcement at two admission layers.

Adjacent matched evidence already localizes the regression to that commit as a unit: the exact parent completed 512/512 at 1515.84 output tok/s, while the exact child completed 512/512 at 799.73 output tok/s. Sanitized adjacent evidence

A prior Python admission-bypass diagnostic completed at 800.49 output tok/s, but its divergence marker never fired: the native capacity scheduler had already reduced every eligible batch before the Python gate. That run therefore changed no admission decision.

Treatment

On the exact frozen child baseline, this draft:

  • excludes DISAGG_GENERATION_TRANS_IN_PROGRESS requests from the native GuaranteedNoEvictScheduler logical request-slot count;
  • preserves their physical KV and PEFT reservations;
  • separately records treatment-only disaggregated-init and ready-generation capacity outputs;
  • shadow-runs the unchanged Python transfer-admission controller exactly once on every eligible post-warmup call;
  • bypasses that Python gate only when both diagnostic switches are enabled, returning the original ordered candidate list;
  • leaves warmup behavior and PP allocation reconciliation unchanged.

Only six treatment files differ from the frozen baseline. Runtime, image, model, workload, selector, topology, physical buffers, scheduler policy, async consensus, cancellation mode, request count, concurrency, and round count remain frozen.

Required causal evidence

The run is valid only with:

  • an in-window C++ event=exercised marker and shutdown summary on each relevant GEN scheduling rank;
  • for the init path, the immediately following Python event=python_gate_exercised marker plus its shutdown summary;
  • 512/512 requests, zero failures, active mode-2 consensus with cancellation disabled on CTX ranks 0–3, clean shutdown, and a valid throughput record.

Interpretation:

  • init-path marker + Python divergence, with no ready-generation marker, and at least 1402 output tok/s: the double-admission interaction is dominant;
  • ready-generation marker alone with recovery: native logical slot accounting is directly dominant, subject to actual executor scheduling evidence;
  • both paths exercised: the combined bundle recovered and further isolation is required;
  • near 800 with the required markers: this admission bundle is not sufficient;
  • missing markers, transfer or memory failure, timeout, protocol/request failure, unclean shutdown, or invalid metric censors the result.

The C++ capacity-output histogram is not a device-forward batch histogram, and Python “treatment admitted” means passed the gate, not completed transfer.

Reproducibility and review

  • exact frozen baseline: dd4a7ac2992f2b02dce381ce4e630c4ba4199436;
  • signed treatment commit: 8c1be138b193c333f170d2179ac9dba426c7b91c;
  • treatment patch ID: cc75c9866a5d67b0aebd46a669bf90765cdde423;
  • treatment diff SHA-256: b598bfbdddbe87f407282ae5605f500a9ef4158e2b1baa4f9dc69885cd4f5f1e;
  • final head: 7fdad6ad098c8efb843f753ff9887cc7bfa77aea;
  • final head has exact current main as its second parent and an identical first-parent treatment tree.

Configured hooks, Python 3.13 compile, test-list AST validation, diff checks, DCO, and independent C++/Python/no-confound reviews passed. Focused Python pytest cannot collect locally because transformers is absent; no local C++ build tree is available.

Because the final merge is ancestry-only, audit the causal change only as the frozen baseline to the signed treatment commit. This draft must be closed unmerged after terminal evidence, and the branch must be preserved.

chienchunhung and others added 12 commits July 29, 2026 12:36
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 6fc7f33)
…sensus factor

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

Superseded before any CI trigger.

This draft combines two factors: native logical-slot accounting and the Python transport-admission bypass. The next diagnostic will keep the PR #15356 transport-budget gate enabled at its normal 1× setting and vary only whether DISAGG_GENERATION_TRANS_IN_PROGRESS consumes maxNumRequests, while retaining physical KV, future-KV, and PEFT reservations.

Closing unmerged; the diagnostic branch remains frozen and preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants