Skip to content

[NVBUG-6448152][test] TEST ONLY: bypass disagg transfer admission - #17073

Closed
chienchunhung wants to merge 12 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-admission-bypass-current-main
Closed

[NVBUG-6448152][test] TEST ONLY: bypass disagg transfer admission#17073
chienchunhung wants to merge 12 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-admission-bypass-current-main

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

TEST ONLY — do not merge

This replacement diagnostic isolates the reachable PyExecutor transfer-admission gate within the adjacent native admission/bounded-polling change. The superseded conflicting draft was closed without any CI trigger; its branch remains frozen.

The immediate parent checkpoint completed at 1,515.84 output tok/s. The adjacent native admission-change checkpoint completed at 799.73 output tok/s. Both used the same current-runtime harness, protected workload, and asynchronous-consensus factor.

Exact construction

  • Diagnostic commit: 235d69128ccd62b6e7cc98c67645a2ff1231a673.
  • Head: a8b4c17046daa51ba6ba740f831cddee9697685a, a signed ancestry-only merge whose first parent is the diagnostic commit and whose second parent is current main at b8a4af190dad7429886e3c90fb11dc6e47ad85f2.
  • The head and first-parent trees are byte-identical: e1426af69a95bf80ae77c0b2e03927b2794f0742; the merge has no first-parent file diff.
  • The treatment stable patch ID remains 1f6f1e2677c83bef4f26c0e2f2a8c6c0aebbcc46.

Treatment

  • Preserve the exact native b6eacd1f baseline construction, runtime overlay, image selection, harness, selector, topology, NIXL configuration, consensus mode, and cancellation setting.
  • Enable TRTLLM_DIAGNOSTIC_DISAGG_ADMISSION_BYPASS=1 only in the exact GB300 workload YAML.
  • On PyExecutor scheduling ranks, preserve the scheduler's original ordered disaggregated-generation candidates and return wait_for_progress=false instead of applying the transfer-admission selection.
  • Preserve those allocations on non-root PP ranks as the immediate parent did; do not change the physical transfer-buffer size or compute-activation limit.
  • Emit one ID-free DIAGNOSTIC OBSERVED marker per executor only when a pure comparison shows the default gate would have deferred requests. The marker reports candidate/admitted/deferred counts and transfer-block totals, never request IDs.

Verification

  • Changed-Python compile: pass.
  • Complete changed-file pre-commit, DCO, YAML, test-list, Ruff, formatting, and diff checks: pass.
  • Focused isolated admission-path smoke: pass.
  • Independent treatment and ancestry-merge P0-P3/no-confound reviews: clean.

Required experiment evidence

The sole target is GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, selector disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL.

A valid result requires all 512 requests to succeed, the bypass marker to prove the default gate would bind on every relevant GEN scheduling rank, consensus active on CTX PP ranks 0-3 with cancellation disabled, clean CTX/GEN/application shutdown, and a valid official output-token metric.

  • At least 1,402 output tok/s implicates the PyExecutor admission gate as the dominant regression source for this workload.
  • Near 800 output tok/s with exercised markers rules the gate down as dominant and promotes the GEN caller-cadence discriminator.
  • Missing markers, build/import failure, timeout, cancellation, protocol mismatch, failed requests, or an invalid metric censor the experiment.

chienchunhung and others added 12 commits July 29, 2026 12:36
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 6fc7f33)
…sensus factor

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62816 [ run ] triggered by Bot. Commit: a8b4c17 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62816 [ run ] completed with state FAILURE. Commit: a8b4c17
/LLM/main/L0_MergeRequest_PR pipeline #50944 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Copy link
Copy Markdown
Collaborator Author

Terminal TEST ONLY evidence: admission-bypass diagnostic

The exact frozen workload at head a8b4c17046daa51ba6ba740f831cddee9697685a completed successfully:

  • 512/512 requests succeeded; 0 failed
  • b_is_valid=true
  • Output throughput: 800.49 tok/s
  • Total throughput: 13,608.34 tok/s
  • CTX, GEN, and router each recorded exactly 512 HTTP 200 responses and clean application shutdown
  • Protocol mode 2 was active with cancellation disabled on CTX ranks 0–3 in both benchmark rounds

This is only +0.095% versus the matched exact-admission result of 799.73 output tok/s. The reported CI failure is the expected performance-threshold result, not a request, protocol, or shutdown failure.

However, the required DIAGNOSTIC OBSERVED marker appeared zero times across all retained logs. That marker was the reachability proof that the default admission gate would have deferred a candidate. Its absence means the bypass treatment was not proven to exercise the decision path in this workload.

Therefore the throughput record is valid, but the causal discriminator is censored: this run does not prove that admission was bypassed in a consequential case and does not rule admission control in or out. The unchanged head should not be rerun. Closing this TEST ONLY draft unmerged and preserving its branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants