[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: earlier C++ source discriminator

[chienchunhung]

Purpose

TEST ONLY; DO NOT REVIEW OR MERGE.

This diagnostic narrows the separate later-source C++ GB300 context-pipeline forward/device-loop throughput regression while holding the current user-space runtime and exact workload fixed.

The merged asynchronous-consensus implementation fixes the earlier blocking pipeline rendezvous. On the matched historical product tree it measured 1548.84 output tokens/s versus 1557.83 for its control. The most recent closed source discriminator completed 512/512 at 795.22 output tokens/s, placing the separate slowdown at or before native source 52ae70934f4357d4aa8e0b3c6ed87f83d53d3149.

Exact construction

• Native source checkpoint: 274043a75a53441b98a057321c75ae9215dbf44b, tree 6d51ad3d7bea310ff3db79917ddee35be0e7ccc2.
• This checkpoint is exactly 50 first-parent commits earlier than the verified-slow native checkpoint 52ae7093.
• Context-only scaffolding commits: 1e9400f65b8c81999f98b16e5ef77819a717a692 and 5ba029d0bcd4f1a1cba41f4711139a7e8d1ff5a5.
  • They temporarily move one existing CMake source-list entry, then restore the original order after applying the factor.
  • Their net tree effect is zero. The final CMake delta is only the coordinator source plus the factor's copyright-year update.
• Production-factor commit: 228bc57299bab8910d2f03fb0ba9527e21947a9e.
  • Standard stable patch ID: 549e0085a06212cde0c47ab47072ad6acdcd1f58, exactly matching the merged implementation.
  • A range-diff against the official factor is identical.
• Exact head: d0bfb69a84356be6a24f215c0f5a76cca74ca8f7, tree d650cc86d5cd44937d7545dc03c9a65677ec743e.
• Current main 75b39d4368204267e70fc3daadaf1820d1fc99ed is included only as the second parent of the signed CI-only merge commit. The merge changes ancestry, not the diagnostic tree.
• No DSpark compatibility patch is present or required at this checkpoint.

The combined native-to-head diff changes exactly the eleven asynchronous-consensus files. It is byte-identical to a direct application of the factor.

Fixed runtime and workload

• jenkins/current_image_tags.properties: 504fd8f234a8d10076e82f000d35e41507566593.
• GB300 DeepSeek workload YAML: 795a6192bf410b35d67535a1b85ce44e42398905.
• Exact selector is present in this native test-list database with TIMEOUT (180).

Target selector:

disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL

Target stage:

GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

No runtime image, workload, scheduler setting, transport setting, traffic shape, timeout, or production consensus behavior is intentionally changed.

Interpretation

• A valid result near the historical approximately 1550 output tokens/s localizes the slowdown to the following 50 commits.
• A valid result near the current approximately 800 output tokens/s moves the source bound earlier than 274043a7.
• Partial recovery will be quantified before choosing another split.
• Any build/import incompatibility, timeout, or failed request is censored and cannot support a throughput conclusion.

Acceptance

Interpret the run only after verifying:

• the exact head and tree above;
• collection of exactly the target selector;
• three nodes, twelve tasks, four GPUs per node, and --no-container-mount-home;
• asynchronous coordinator activation on all four context pipeline ranks;
• 512/512 successful requests with zero failures;
• clean protocol and service shutdown; and
• a valid official output-token throughput metric.

This draft will be closed promptly after terminal evidence is captured.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #61549 [ run ] triggered by Bot. Commit: d0bfb69 Link to invocation

[tensorrt-cicd]

PR_Github #61549 [ run ] completed with state FAILURE. Commit: d0bfb69
/LLM/main/L0_MergeRequest_PR pipeline #49762 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Terminal TEST ONLY result for the single-stage run:

• Exact head d0bfb69a84356be6a24f215c0f5a76cca74ca8f7, native checkpoint 274043a75a53441b98a057321c75ae9215dbf44b, current image and exact workload.
• The benchmark completed 512/512 requests with 0 failed requests.
• Asynchronous context-transfer consensus v1 activated on CTX pipeline ranks 0 through 3. Both service wrappers completed application shutdown; no fatal, protocol, segmentation, or double-free errors were found.
• Official output-token throughput: 800.18 tok/s; request throughput: 0.10 req/s; total-token throughput: 13,603.11 tok/s.
• The CI failure is solely the expected performance-regression gate against its 1550.2489 tok/s baseline, not a request or protocol failure.

This is +0.70% versus the later current-source 794.61 tok/s result, +0.62% versus the preceding 795.22 tok/s discriminator, and 51.66% of the historical 1548.84 tok/s result. There is no meaningful recovery, so the slowdown is already present at or before native checkpoint 274043a7; the next compatible source discriminator must move earlier.

Closing this TEST ONLY draft. The branch is preserved.