Skip to content

[NVBUG-6448152][test] TEST ONLY bisect native 9ff00c86 - #16903

Closed
chienchunhung wants to merge 7 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-native-midpoint-9ff00c86
Closed

[NVBUG-6448152][test] TEST ONLY bisect native 9ff00c86#16903
chienchunhung wants to merge 7 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-native-midpoint-9ff00c86

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

TEST ONLY

Historical-source/current-runtime discriminator for a separate C++ throughput regression. This draft is diagnostic only and is not intended for merge.

  • Native source checkpoint: 9ff00c869a, with exact native tree dd93f3cecf225b261951d7d315560dfa01a4081c and parent b155a516154198d77c65309680dfa1a761465a23.
  • This is the balanced first-parent split: 100 commits after pre-admission anchor 85665f5fd331d0154a78172954846d843085e83f, 99 commits after the admission and bounded-polling change, and 100 commits before the known-slow native checkpoint.
  • Runtime-only overlay: NIXL 1.3.1 and the same five CI image tags from build 202607151440-16194.
  • CI-only compatibility holds the Jenkins harness byte-identical to the known-slow diagnostic. A compile-only upstream Marlin empty-stub repair is also carried; the selected CUTEDSL workload does not execute Marlin.
  • The reviewed pre-cancellation asynchronous-consensus factor runs protocol mode 2 with cancellation fixed off.
  • CMake context normalization and restoration have zero net ordering effect.
  • No DSpark, workload YAML, test-list, selector, timeout, scheduler, or product adaptation is included.
  • The target remains the exact GB300 3-node/12-GPU disaggregated performance selector. A result at or above 1402 output tokens/s places the regression in the following 100 commits; a result near 800 moves the slow bound to this checkpoint. Any build/import failure, timeout, protocol mismatch, cancellation, or failed request is non-interpretable.

The merged asynchronous-consensus change is not under re-review here; this draft only localizes the separate later throughput regression.

chienchunhung and others added 7 commits July 27, 2026 10:01
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 6fc7f33)
…sensus factor

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61958 [ run ] triggered by Bot. Commit: 2de8762 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61958 [ run ] completed with state FAILURE. Commit: 2de8762
/LLM/main/L0_MergeRequest_PR pipeline #50151 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Copy link
Copy Markdown
Collaborator Author

Terminal TEST ONLY result

The exact native checkpoint 9ff00c869a36055fcf2ebf4de4950a0471a3039f completed the sole requested GB300 discriminator:

  • 512/512 requests succeeded; 0 failed
  • 6,005.89 s selector duration
  • 795.48 output tok/s
  • 0.10 requests/s
  • 13,523.19 total tok/s
  • protocol mode 2 with cancellation disabled on all four CTX PP ranks
  • clean CTX, GEN, and disaggregated-service shutdown

The red result is solely the expected performance gate. This is +0.11% versus the later-tree 794.61 result and 51.36% of the historical repaired 1,548.84 result, so there is no recovery. The slowdown is present at or before native 9ff00c86; the following 100 first-parent commits through f4742f68 are excluded.

No rerun is needed. Closing this TEST ONLY draft unmerged while preserving the branch for reproducibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants