[NVBUG-6448152][test] Recover token throughput for DS PP4 tests (DO NOT REVIEW YET)

[chienchunhung]

Purpose

TEST ONLY; DO NOT REVIEW.

Narrow the source interval for the later C++ GB300 context-pipeline forward/device-loop throughput regression while holding the current user-space runtime and exact workload fixed.

The official asynchronous-consensus implementation fixes the earlier blocking pipeline rendezvous. It recovers the historical frozen tree to 1548.84 output tokens/s, but its current review head measures 794.61 output tokens/s. This diagnostic asks whether the separate later slowdown is already present at an earlier native source checkpoint.

Exact construction

• Native source checkpoint: 52ae70934f4357d4aa8e0b3c6ed87f83d53d3149, tree 3638dd83cc9c4ef9b7e79eb30eef98957a633622.
• Compatibility-only commit: ec93e4860ab32273a3003e1b8838b1f0f9daf86a.
  • It reproduces only the two-file DSpark collection compatibility change: rename DSparkWorker.forward to _forward_impl and add the corresponding test stub.
  • Diff: 4 insertions, 1 deletion; stable patch ID 79aed56a23240a65ec02e9f95888712bdd8b5af1.
  • It is required only so this native checkpoint can collect under the current test interface; it is outside the target workload path.
• Production-factor commit: 68acb46c159989eb47bca824fffbd1f222085ba3.
  • The complete eleven-file asynchronous-consensus factor is applied byte-for-byte.
  • Stable patch ID: 549e0085a06212cde0c47ab47072ad6acdcd1f58, identical to the official implementation.
• Exact head: c7d38a45e30665639b450d9ff8b5428551b31024, tree 1dd9e358f1b641a42091b2db2cfea17aec6b84bd.
• Current main is included only as the second parent of the signed CI-only merge commit. The merge changes ancestry, not the diagnostic tree.

Fixed runtime and workload

The candidate preserves these verified blobs:

• jenkins/current_image_tags.properties: 504fd8f234a8d10076e82f000d35e41507566593.
• tests/scripts/perf-sanity/disaggregated/gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml: 795a6192bf410b35d67535a1b85ce44e42398905.

Target selector:

disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL

Target stage:

GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

No runtime image, workload, scheduler setting, transport setting, timeout, traffic shape, or production consensus behavior is intentionally changed.

Interpretation

This checkpoint is eleven first-parent commits earlier than the verified-slow native checkpoint 3ab71753fe71d8ea4ae73427d1632fe2f7ecf201.

• A valid result near the historical approximately 1550 output tokens/s localizes the slowdown to those following eleven commits.
• A valid result near the current approximately 800 output tokens/s excludes those eleven later commits and places the slowdown at or before 52ae7093.
• Partial recovery will be quantified before choosing another split.
• Any build/import incompatibility, timeout, or failed request is censored and cannot support a throughput conclusion.

Acceptance

Interpret the run only after verifying:

• the exact head and tree above;
• the exact selector, three nodes, twelve tasks, four GPUs per node, and --no-container-mount-home;
• asynchronous coordinator activation on all four context pipeline ranks;
• 512/512 successful requests with zero failures;
• clean protocol and service shutdown; and
• a valid official output-token throughput metric.

The draft will be closed after terminal evidence is captured. It is not intended for review or merge.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #61356 [ run ] triggered by Bot. Commit: c7d38a4 Link to invocation

[tensorrt-cicd]

PR_Github #61356 [ run ] completed with state FAILURE. Commit: c7d38a4
/LLM/main/L0_MergeRequest_PR pipeline #49585 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #61372 [ run ] triggered by Bot. Commit: c7d38a4 Link to invocation

[tensorrt-cicd]

PR_Github #61372 [ run ] completed with state FAILURE. Commit: c7d38a4
/LLM/main/L0_MergeRequest_PR pipeline #49600 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation