[NVBUG-6448152][test] TEST ONLY: legacy GEN selection and blocking discriminator

[chienchunhung]

TEST ONLY — do not review or merge

This is the next one-function discriminator for the C++ throughput regression localized to the admission and bounded-polling change.

The adjacent native source evidence is categorical:

• the pre-admission native discriminator completed at 1515.84 output tok/s;
• the exact admission-commit discriminator completed at 799.73 output tok/s;
• the 4x admission-window discriminator completed at 800.58 output tok/s, but its intended greater-than-3x occupancy marker did not appear, so it did not prove the larger window was exercised and was censored for magnitude attribution.

Frozen baseline

• Exact parent: dd4a7ac2992f2b02dce381ce4e630c4ba4199436
• Exact parent tree: 899bdb3449199c821d157c9690f7d9aae9551e66
• Native source under test: b6eacd1f725d0fbeb888e09ff2585e5ab23b0856
• Treatment commit: 71e40682331ef098853e9c0224ad60ae6900e8dc
• Exact diagnostic head: 41ff39b455aab9ffa97cf218661bdf1c772eb215
• Exact diagnostic tree: 1b3b314924174fbe741c6614c6dabb95ea603b24
• Combined treatment stable patch ID: 9cd001cb170b6d8ad975d7f58ee34f9f546136c2
• Exact treatment diff SHA-256: 9d597690ef588bc1565aaa9a0a277fa1d07db51e44c4976c6e4138f26379ddd6

The final commit is a signed CI-only no-tree merge with current main commit c0eac26d4a4d874bf23829dd68415dc3fdabb7be as its second parent. Its tree is byte-identical to the treatment commit, so it changes ancestry and mergeability only.

Runtime image, NIXL/runtime requirements, Jenkins and Slurm harnesses, exact workload and selector, physical buffers, admission controller and call sites, active-transfer accounting, context-side polling, asynchronous consensus, encoded protocol mode 2, cancellation-disabled setting, topology, and request count are unchanged from the valid exact-admission baseline.

Only treatment

The only changed tracked file relative to the frozen baseline is cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp.

Inside CacheTransceiver::checkGenTransferStatus, this draft restores the pre-admission generation-side selection and completion bundle:

1. one-shot readiness scanning instead of the admission commit's finite wait-slice/deadline loop;
2. frequency-first and then local insertion-order fallback selection to satisfy atLeastRequestNum;
3. unconditional future.get() for selected requests instead of bounded wait_for with keep-queued behavior.

Admission control itself, its checkGenTransferStatus(1) call cadence, accounting, transfer traffic, compute, context-side bounded polling, and consensus remain at the frozen admission baseline.

This is intentionally described as the legacy GEN selection plus blocking-wait bundle. A recovery cannot distinguish selection order from blocking backpressure within this function.

Execution marker

A one-shot, process-local, ID-free INFO marker fires only when an atLeastRequestNum > 0 call has selected a request whose future is still non-ready immediately before the restored future.get():

DIAGNOSTIC OBSERVED: legacy generation transfer polling selected a non-ready request immediately before future.get()

The marker uses a relaxed atomic only for duplicate suppression. It adds one zero-time readiness query until first observation and at most one INFO line per process. It proves entry into the treated non-ready path, not measurable blocking duration.

Required valid evidence

• Only stage GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1
• Exact selector disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL
• Three nodes, twelve tasks, four GPUs per node, and --no-container-mount-home
• 512/512 requests with zero failed requests
• Asynchronous context-transfer consensus on all four context PP ranks, protocol mode 2, cancellation disabled
• The execution marker on every relevant generation worker/rank during benchmark traffic
• Clean context, generation, and disaggregated-service shutdown
• A valid official output-token metric

Build/import failure, timeout, deadlock, cancellation, protocol mismatch, failed request, missing execution marker, unclean shutdown, or invalid metric censors the result.

Interpretation

• At least 1402 output tok/s: the legacy GEN selection/blocking bundle is a dominant factor under admission-control call cadence.
• Near 800 output tok/s with the execution marker proven: this bundle is ruled down as dominant for this exact workload; context-side bounded polling or another subchange becomes the next discriminator.
• Partial recovery: quantify the contribution without assigning it to selection versus blocking.

This is a hybrid treatment: admission-control scheduling and call cadence remain active while the generation-side function is restored. It cannot establish that admission control is universally irrelevant.

Pre-run validation

• One DCO-signed treatment commit on the exact frozen baseline plus one signed no-tree current-main ancestry merge.
• The final ancestry merge has exact parents 71e40682331ef098853e9c0224ad60ae6900e8dc and c0eac26d4a4d874bf23829dd68415dc3fdabb7be; its tree exactly matches its first parent.
• git diff --check, clang-format, codespell, and all applicable changed-file hooks passed.
• Repository test-list duplicate and AST validation passed under Python 3.13; the default local hook interpreter is Python 3.9 and cannot parse the repository's str | None annotation.
• Focused local pytest collection was unavailable only because the local environment lacks transformers; no test execution is claimed.
• Two independent reviews found no P0–P3 issue or material A/B confound.
• GitHub reports the exact diagnostic head mergeable against current main.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62689 [ run ] triggered by Bot. Commit: 41ff39b Link to invocation

[tensorrt-cicd]

PR_Github #62689 [ run ] completed with state FAILURE. Commit: 41ff39b
/LLM/main/L0_MergeRequest_PR pipeline #50829 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Terminal result for the legacy GEN transfer-polling discriminator at the exact diagnostic head:

• The exact GB300 disaggregated selector completed 512/512 requests with 0 failures in 5,251.87 seconds.
• The official record is valid (b_is_valid=true): 798.63 output tok/s, 0.10 req/s, and 13,576.71 total tok/s.
• Asynchronous context-transfer consensus version 1 (encoded protocol mode 2) was enabled consistently on CTX PP ranks 0–3, with in-flight cancellation disabled on every rank.
• CTX, GEN, and the disaggregated frontend each reported application shutdown complete; no fatal or protocol errors were present. Residual Slurm-step termination happened only during harness cleanup after the benchmark and metric completed.
• The overall CI failure is solely the expected performance gate: 798.63 versus the 1,550.25 output-tok/s reference (−48.48%).
• The one-shot DIAGNOSTIC OBSERVED marker was absent from the complete archived logs. During this workload, the restored legacy path therefore never selected a non-ready transfer future immediately before future.get().

Interpretation: restoring the legacy GEN selection and blocking bundle did not recover throughput: 798.63 is effectively unchanged from the matched exact-admission result of 799.73 output tok/s. This rules that bundle down as the dominant cause for this exact workload; the absent marker means this run does not measure the cost of the blocking branch when that condition is actually exercised.

Closing this TEST ONLY diagnostic unmerged; the branch remains preserved.