[NVBUG-6448152][test] TEST ONLY: bypass disagg transfer admission - #17073
[NVBUG-6448152][test] TEST ONLY: bypass disagg transfer admission#17073chienchunhung wants to merge 12 commits into
Conversation
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Yihan Wang <yihwang@nvidia.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> (cherry picked from commit 6fc7f33)
…sensus factor Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> (cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> (cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1" |
|
PR_Github #62816 [ run ] triggered by Bot. Commit: |
|
PR_Github #62816 [ run ] completed with state
|
Terminal TEST ONLY evidence: admission-bypass diagnosticThe exact frozen workload at head
This is only +0.095% versus the matched exact-admission result of 799.73 output tok/s. The reported CI failure is the expected performance-threshold result, not a request, protocol, or shutdown failure. However, the required Therefore the throughput record is valid, but the causal discriminator is censored: this run does not prove that admission was bypassed in a consequential case and does not rule admission control in or out. The unchanged head should not be rerun. Closing this TEST ONLY draft unmerged and preserving its branch. |
TEST ONLY — do not merge
This replacement diagnostic isolates the reachable PyExecutor transfer-admission gate within the adjacent native admission/bounded-polling change. The superseded conflicting draft was closed without any CI trigger; its branch remains frozen.
The immediate parent checkpoint completed at 1,515.84 output tok/s. The adjacent native admission-change checkpoint completed at 799.73 output tok/s. Both used the same current-runtime harness, protected workload, and asynchronous-consensus factor.
Exact construction
235d69128ccd62b6e7cc98c67645a2ff1231a673.a8b4c17046daa51ba6ba740f831cddee9697685a, a signed ancestry-only merge whose first parent is the diagnostic commit and whose second parent is currentmainatb8a4af190dad7429886e3c90fb11dc6e47ad85f2.e1426af69a95bf80ae77c0b2e03927b2794f0742; the merge has no first-parent file diff.1f6f1e2677c83bef4f26c0e2f2a8c6c0aebbcc46.Treatment
b6eacd1fbaseline construction, runtime overlay, image selection, harness, selector, topology, NIXL configuration, consensus mode, and cancellation setting.TRTLLM_DIAGNOSTIC_DISAGG_ADMISSION_BYPASS=1only in the exact GB300 workload YAML.wait_for_progress=falseinstead of applying the transfer-admission selection.DIAGNOSTIC OBSERVEDmarker per executor only when a pure comparison shows the default gate would have deferred requests. The marker reports candidate/admitted/deferred counts and transfer-block totals, never request IDs.Verification
Required experiment evidence
The sole target is
GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, selectordisagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL.A valid result requires all 512 requests to succeed, the bypass marker to prove the default gate would bind on every relevant GEN scheduling rank, consensus active on CTX PP ranks 0-3 with cancellation disabled, clean CTX/GEN/application shutdown, and a valid official output-token metric.