Skip to content

[NVBUG-6448152][test] TEST ONLY: bypass disagg transfer admission - #17072

Closed
chienchunhung wants to merge 11 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-admission-bypass
Closed

[NVBUG-6448152][test] TEST ONLY: bypass disagg transfer admission#17072
chienchunhung wants to merge 11 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-admission-bypass

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

TEST ONLY — do not merge

This diagnostic isolates the reachable PyExecutor transfer-admission gate within the adjacent native admission/bounded-polling change.

The immediate parent checkpoint completed at 1,515.84 output tok/s. The adjacent native admission-change checkpoint completed at 799.73 output tok/s. Both used the same current-runtime harness, protected workload, and asynchronous-consensus factor.

Treatment

  • Preserve the exact native b6eacd1f baseline construction, runtime overlay, image selection, harness, selector, topology, NIXL configuration, consensus mode, and cancellation setting.
  • Enable TRTLLM_DIAGNOSTIC_DISAGG_ADMISSION_BYPASS=1 only in the exact GB300 workload YAML.
  • On PyExecutor scheduling ranks, preserve the scheduler's original ordered disaggregated-generation candidates and return wait_for_progress=false instead of applying the transfer-admission selection.
  • Preserve those allocations on non-root PP ranks as the immediate parent did; do not change the physical transfer-buffer size or compute-activation limit.
  • Emit one ID-free DIAGNOSTIC OBSERVED marker per executor only when a pure comparison shows the default gate would have deferred requests. The marker reports candidate/admitted/deferred counts and transfer-block totals, never request IDs.

Local verification

  • Changed-Python compile: pass.
  • Complete changed-file pre-commit, DCO, YAML, test-list, Ruff, formatting, and diff checks: pass.
  • Focused isolated admission-path smoke: pass.
  • Independent P0-P3/no-confound review: clean.

Required experiment evidence

The sole target is GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, selector disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL.

A valid result requires all 512 requests to succeed, the bypass marker to prove the default gate would bind on every relevant GEN scheduling rank, consensus active on CTX PP ranks 0-3 with cancellation disabled, clean CTX/GEN/application shutdown, and a valid official output-token metric.

  • At least 1,402 output tok/s implicates the PyExecutor admission gate as the dominant regression source for this workload.
  • Near 800 output tok/s with exercised markers rules the gate down as dominant and promotes the GEN caller-cadence discriminator.
  • Missing markers, build/import failure, timeout, cancellation, protocol mismatch, failed requests, or an invalid metric censor the experiment.

chienchunhung and others added 11 commits July 29, 2026 12:36
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 6fc7f33)
…sensus factor

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

This draft was detected as conflicting with the latest main before any CI trigger was posted. No test run was requested and no experiment was duplicated.

Closing it unmerged while preserving the exact diagnostic branch at 235d69128ccd62b6e7cc98c67645a2ff1231a673. A replacement draft will add only a reviewed, tree-preserving ancestry merge of the current main; the diagnostic source tree and treatment patch will remain identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants