[NVBUG-6448152][test] isolate GNE transfer slot accounting

[chienchunhung]

TEST ONLY — DO NOT MERGE

This draft isolates one semantic change from PR #15356 on the exact frozen slow-child baseline. The adjacent matched evidence is 1515.84 output tok/s at the immediate parent and 799.73 output tok/s at the native child; see the published baseline record.

Question

Does restoring the pre-PR logical request-slot treatment recover throughput while retaining the PR #15356 transport-budget gate at its normal 1× setting?

Single treatment

For GUARANTEED_NO_EVICT only:

• DISAGG_GENERATION_TRANS_IN_PROGRESS no longer increments the logical maxNumRequests count;
• prompt allocation, remaining-to-completion KV reservation, cross-KV reservation, and PEFT reservation remain unchanged;
• transfer-only requests are never compute-scheduled;
• STATIC_BATCH and MAX_UTILIZATION remain unchanged.

The Python transport-budget admission behavior remains enabled and unchanged:

• max_tokens_in_buffer: 131104 on CTX and GEN;
• multiplier 1 and bypass off;
• the normal selector, deferred-allocation reverts, and wait decision still execute.

Observer-only Python counters report whether that unchanged gate admits or defers the candidates exposed by the native treatment. They do not change its return value or side effects.

Runtime, image, model, physical buffers, compute cap, async consensus, harness, workload, selector, and three-node/twelve-GPU topology remain frozen.

Required causal evidence

The native scheduler emits ID-free startup, exact-decision-divergence, and shutdown-summary records. Divergence is separated into:

• path=ready_generation: a ready request is returned only because transfer-only requests no longer consume logical slots;
• path=disagg_init: an init request passes the unchanged KV/PEFT checks and is returned only under the treatment.

The observer reports the unchanged 4097-block transport budget and admitted/deferred totals. For this 128K workload, an active transfer is approximately 4096 blocks and a new 128K candidate is another 4096 blocks. Therefore an init-only native divergence may be immediately masked by the retained 1× transport gate; that outcome is an interaction result, not evidence against native slot accounting.

A valid run additionally requires 512/512 successful requests, zero failures, b_is_valid=true, async consensus mode 2 with cancellation disabled on CTX ranks 0–3, and clean CTX/GEN/disaggregated shutdown.

Interpretation:

• recovery to at least 1402 output tok/s with an exercised path implicates logical slot accounting as dominant;
• near 800 with a treatment-only ready request reaching scheduling rules the single native factor down as sufficient;
• native init divergence plus transport deferral shows that the unchanged second gate masks the first-gate relaxation;
• missing divergence evidence, failed requests, timeout, cancellation, protocol error, unclean shutdown, or invalid metric censors the causal conclusion.

Frozen identities and checks

• exact baseline: dd4a7ac2992f2b02dce381ce4e630c4ba4199436
• signed treatment: 6eaf334e827120f0d951c64631bbc17a70c18540
• final tree-preserving head: fc86d178d2f8562cec440ae5cc4a60d3a301b82f
• tree: 6227dfcf14a808628cbb47e2b566ef8cb64ea217
• stable treatment patch ID: 84bfc903f46d49103cd4a39082de6ac48582ffb8
• treatment diff SHA-256: 0a1ea3c204785690fa763322a3b0ae574423132dfda391c23c5155ad03be888e

Full configured changed-file hooks, formatting, YAML validation, test-list AST validation under Python 3.13, diff checks, and DCO passed. Python unit-test collection is locally blocked only by the absent transformers dependency; the C++ unit binary is not available in the local checkout.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #62911 [ run ] triggered by Bot. Commit: fc86d17 Link to invocation

[tensorrt-cicd]

PR_Github #62911 [ run ] completed with state FAILURE. Commit: fc86d17
/LLM/main/L0_MergeRequest_PR pipeline #51035 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Terminal native-slot-only diagnostic

The single frozen targeted run completed on exact head fc86d17 with the requested three-node/twelve-GPU topology.

• Workload evidence is valid: 512/512 requests succeeded, zero requests failed, b_is_valid=true, output throughput was 797.62 tok/s, and total token throughput was 13,559.59 tok/s.
• This is -0.26% versus the exact slow-child baseline of 799.73 tok/s from the adjacent parent/child record: there was no throughput recovery.
• CTX ranks 0–3 agreed on asynchronous consensus packed mode 2 (protocol version 1, cancellation disabled). CTX, GEN, and disaggregated frontends each completed all 512 HTTP requests and logged application-shutdown completion.
• The treatment configuration was active on GEN scheduling ranks 0–7, but the targeted native path was not exercised: no NVBUG6448152_CAPACITY event=exercised record appeared during the run. The only emitted native lifecycle summaries reported exercised_iterations=0 and transfer_active=0; the measured lifecycle did not emit the required final native summary.
• The unchanged multiplier-1 Python transport gate did execute on every GEN rank and admitted every observed candidate: deferred_total=0, limited_calls=0, and wait_calls=0. It therefore did not mask a treatment candidate in this execution.

The CI headline is a performance-regression assertion against the fast reference after the valid benchmark record was produced; it is not a request, protocol, crash, or infrastructure failure.

Conclusion: no recovery was observed, but this run cannot implicate or rule out PR #15356's transfer-in-progress logical-slot accounting because that treatment never changed a scheduling decision in the measured window. It does show that the retained transport gate's decision effect did not mask the treatment here. A follow-up must first prove transfer-active old-versus-treatment divergence before its throughput can answer the native-slot question.

Closing this TEST ONLY draft unmerged and preserving the diagnostic branch.