Skip to content

[NVBUG-6448152][test] TEST ONLY bisect native f4742f68 - #16879

Closed
chienchunhung wants to merge 5 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-native-midpoint-f4742f68
Closed

[NVBUG-6448152][test] TEST ONLY bisect native f4742f68#16879
chienchunhung wants to merge 5 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-native-midpoint-f4742f68

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

TEST ONLY

Historical-source/current-runtime discriminator for a separate C++ throughput regression. This draft is diagnostic only and is not intended for merge.

  • Native source checkpoint: f4742f6868, with exact native tree ebd8c9a8fdd239a2d40381e505985fb5b3445dc5.
  • The checkpoint is an exact first-parent midpoint: 199 commits after the admission and bounded-polling change and 199 commits before the known-slow native checkpoint 544199a47b91b65d295322c6bb1f74dadbcf6a79.
  • Runtime-only overlay: NIXL 1.3.1 and the current five CI image tags from build 202607151440-16194.
  • The reviewed pre-cancellation asynchronous-consensus factor runs protocol mode 2 with cancellation fixed off.
  • CMake context normalization and restoration have zero net semantic effect.
  • No DSpark, workload YAML, selector, timeout, or product compatibility changes are included.
  • The test target remains the exact GB300 3-node/12-GPU disaggregated performance selector. A result at or above 1402 output tokens/s places the regression in the later half; a result near 800 moves the slow bound to this checkpoint. Any build/import failure, timeout, protocol mismatch, cancellation, or failed request is non-interpretable.

The merged asynchronous-consensus change is not under re-review here; this draft only localizes the separate later throughput regression.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 6fc7f33)
…sensus factor

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61747 [ run ] triggered by Bot. Commit: 9cf779e Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61747 [ run ] completed with state FAILURE. Commit: 9cf779e
/LLM/main/L0_MergeRequest_PR pipeline #49949 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

Terminal evidence for the native f4742f68 checkpoint at the exact TEST ONLY diagnostic head:

  • The sole selected GB300 workload completed 512/512 requests with 0 failed requests in 5249.71 seconds.
  • Official metrics: 798.96 output tokens/s, 0.10 requests/s, and 13582.30 total tokens/s.
  • Context-transfer consensus protocol mode 2 (version 1 with cancellation disabled) was active on all four CTX PP ranks, ranks 0 through 3.
  • CTX, GEN, and disaggregated applications shut down cleanly. No fatal, protocol-mismatch, double-free, or uncaught-exception evidence was found.
  • The red result is solely the expected performance-regression gate after the valid workload completed. A post-run classifier noticed startup readiness noise, but only one workload was submitted and no retry ran; this evidence is not censored and should not be rerun.

The result is +0.55% versus the 794.61 current-source point and 51.58% of the historical 1548.84 point: no recovery. Therefore the following 199 first-parent commits through the later slow checkpoint are excluded as the origin, and the slow source bound moves to at or before this native checkpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants