[NVBUG-6448152][perf] TEST ONLY; DO NOT REVIEW: disable PP CuTe BF16 auto selection

[chienchunhung]

TEST ONLY; DO NOT REVIEW

This is a one-factor performance diagnostic for the GB300 PP4 disaggregated context workload.

Hypothesis

A later-main change automatically enables persistent CuTe DSL BF16 BMM and GEMM kernels for SM100/SM103 when pipeline parallelism is greater than one. The historical high-throughput tree left both kernels disabled, while the current low-throughput tree enables both on every context PP rank. The bounded phase trace localizes the first inflated interval to PP forward/device-loop overlap, upstream of admission, NIXL transfer, and asynchronous outcome agreement.

Diagnostic change

• Suppress only the automatic SM100/SM103 PP CuTe BF16 selection when TRTLLM_NVBUG_6448152_DISABLE_PP_CUTE_BF16_AUTO=1.
• Preserve explicit user-selected CuTe BMM/GEMM settings.
• Emit one bounded NVBUG6448152_CUTE event=effective_config marker.
• Enable the diagnostic only in the exact GB300 perf-sanity YAML.
• Keep the workload, image, scheduler, admission, transfer, timeouts, and asynchronous coordinator unchanged.

Verification

• DCO-signed commit and clean focused diff.
• Added focused coverage for normal auto-enable, diagnostic suppression, explicit settings, and PP1.
• Pre-commit hooks passed where applicable.
• Python compile/AST checks and a focused validator harness passed.
• Local pytest could not run in this checkout because its Python environment lacks pytest and torch.
• Exact GB300 result is pending.

Interpretation

• A valid 512/512 result at or above 1402 output tok/s would localize the later-main regression to the automatic CuTe BF16 PP path.
• A valid result near 800 output tok/s would rule down this factor and move the next discriminator to the runtime/container stack.

Do not review or merge this diagnostic.

[chienchunhung]

/bot run --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60863 [ run ] triggered by Bot. Commit: 26746fa Link to invocation

[tensorrt-cicd]

PR_Github #60863 [ run ] completed with state FAILURE. Commit: 26746fa
/LLM/main/L0_MergeRequest_PR pipeline #49135 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation