[NVBUG-6448152][test] TEST ONLY: 4x admission-window discriminator

[chienchunhung]

TEST ONLY — do not merge

This is a single-factor performance discriminator for the regression introduced by the admission and bounded-polling change.

The adjacent source comparison is now categorical: the native parent completed at 1515.84 output tok/s, while the exact admission commit completed at 799.73 output tok/s in the frozen admission-commit diagnostic. The earlier 2× logical admission-window diagnostic remained slow at 799.32 output tok/s, so this draft does not repeat that treatment.

Frozen baseline

• Parent diagnostic head: dd4a7ac2992f2b02dce381ce4e630c4ba4199436
• Parent tree: 899bdb3449199c821d157c9690f7d9aae9551e66
• Native source under test: b6eacd1f (the admission/bounded-polling commit)
• Current treatment head: 4571e90b1ec4c4effd673ceab99e7479158cd87f
• Current tree: 270608b5b53c623fb318fa24bf3a31ec16219d77

Runtime overlay, current CI harness and image inputs, exact workload, C++ consensus factor, cancellation-off setting, topology, selector, physical NIXL buffers, and active-transfer accounting are unchanged from the valid 1× baseline.

Only treatment

The exact GB300 workload sets TRTLLM_DIAGNOSTIC_DISAGG_ADMISSION_BUDGET_MULTIPLIER=4. This multiplies only the PyExecutor logical transfer-admission budget. The CTX and GEN max_tokens_in_buffer values remain 131104; no physical NIXL buffer is enlarged.

For this workload the base logical window is 2,049 blocks and the 4× effective window is 8,196 blocks. One full request uses 2,048 blocks, so four full transfers use 8,192 blocks.

The diagnostic marker is deliberately stronger than a configuration check: it latches only above the 3× comparison window (6,147 blocks). Therefore the expected used=8192 marker proves that the fourth logical window was exercised and necessarily exceeds the previously tested 2× effective window of 4,098 blocks. It cannot be satisfied by the oversized-idle-head exception or active transfers alone.

Required valid evidence

• Exact frozen GB300 stage and selector only
• Three nodes, twelve tasks, four GPUs per node, and no home-directory container mount
• 512/512 requests with zero failures
• C++ consensus active on all four CTX PP ranks with cancellation disabled
• Clean CTX, GEN, and disaggregated-service shutdown
• Valid official output-token metric
• Configuration marker: base 2,049, effective 8,196, multiplier 4
• Observation marker: used 8,192, comparison window 6,147

Build/import failure, timeout, cancellation, protocol mismatch, any failed request, invalid metric, or failure to prove capacity beyond the previous 2× window censors the result.

Interpretation

• At least 1402 output tok/s: the numerical admission-window magnitude is the dominant regression source.
• Near 800 output tok/s with the 4× marker proven: admission-window magnitude is ruled down for this exact workload; bounded polling or another scheduler-timing subchange in the same commit becomes the next discriminator.
• Partial recovery: admission pressure contributes but is not the complete cause.

Validation

• Python 3.13 compile passed for both changed Python files.
• Complete changed-file pre-commit suite and DCO check passed.
• YAML invariants passed: multiplier 4, two rounds, concurrency 256, physical buffers 131104/131104.
• Independent final review found no P0–P3 issue or A/B confound.
• Focused local pytest collection is unavailable only because the local environment lacks transformers; no test was claimed as executed.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62627 [ run ] triggered by Bot. Commit: 4571e90 Link to invocation

[chienchunhung]

Terminal evidence — valid workload, admission-magnitude discriminator censored

The exact frozen GB300 workload completed successfully at head 4571e90b1ec4c4effd673ceab99e7479158cd87f:

• 512/512 requests succeeded; 0 failed
• 800.58 output tok/s
• 0.10 req/s
• 13,609.79 total tok/s
• 6.41 user tok/s
• Official performance record: b_is_valid=true
• Asynchronous context-transfer consensus version 1 was active on CTX ranks 0–3, with in-flight cancellation disabled on every rank
• CTX, GEN, and disaggregated services reached clean application shutdown; no fatal or protocol errors were found

The metric remains effectively identical to the adjacent 1× admission baseline: +0.85 output tok/s, or +0.106%. It is also only 52.81% of the 1,515.84 output tok/s native-parent result.

However, this run does not satisfy the discriminator's capacity-exercise requirement. Runtime reported a base window of 4,097 transfer blocks and an effective 4× window of 16,388 blocks on all CTX and GEN ranks, correcting the pre-run 2,049/8,196 estimate. The required DIAGNOSTIC OBSERVED marker was absent from the complete CTX and GEN logs. That marker latches only when active plus newly admitted work exceeds the 3× comparison window of 12,291 blocks, excluding the oversized-idle-head case.

Therefore:

• The workload and throughput measurement are valid.
• Configuring the multiplier to 4 without observed occupancy above 3× did not recover throughput.
• The intended claim that the fourth logical admission window was actually exercised is censored.
• This result must not be used to claim that 4× admission-window magnitude is ruled down.

The recorded pytest failure was solely the expected performance gate against the historical fast baseline. A post-result retry classifier also matched transient startup Connection refused health probes, but no second workload ran and the successful archived result remained intact.

Closing this TEST ONLY draft unmerged; the branch is preserved.

[tensorrt-cicd]

PR_Github #62627 [ run ] completed with state FAILURE. Commit: 4571e90
/LLM/main/L0_MergeRequest_PR pipeline #50767 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation