Skip to content

[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW current source with historical runtime - #16750

Closed
chienchunhung wants to merge 3 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-current-source-historical-runtime
Closed

[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW current source with historical runtime#16750
chienchunhung wants to merge 3 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-current-source-historical-runtime

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

Purpose

TEST ONLY. DO NOT REVIEW OR MERGE.

This reverse crossover isolates the runtime/toolchain contribution after the historical-source/current-runtime diagnostic could not reach its requested performance stage because the prerequisite build was incompatible. It retains the exact reviewed current C++ source at commit a2fca780 and replaces only its five CI image assignments with the historical image set used by the known-fast validation tree.

Factor isolation

  • Current source parent: commit a2fca780.
  • The five current image assignments from build 202607151440-16194 are replaced with the exact five assignments from historical source commit 5e2498b, build 202606051544-14972.
  • Product code, benchmark YAML, selector, workload, coordinator behavior, traffic, admission, and timeout settings are unchanged from the current reviewed source.

This is a one-factor reverse crossover: current source plus historical runtime/toolchain. It is not intended for review or merge.

Why the reverse crossover is required

The forward crossover used historical source with current runtime. Its exact-head helper entered pipeline 49303 but failed in the x86 prerequisite during DeepEP device linking against current NVSHMEM symbols. Fail-fast then aborted the SBSA build before any requested test child, Slurm allocation, selector execution, or performance metric. That result is build-incompatible and therefore inconclusive for throughput.

Validation

  • The factor commit has exactly one parent, the reviewed current-source commit.
  • The committed delta from that parent is exactly one file and five line substitutions.
  • The resulting image-tag file is byte-identical to the historical comparison file.
  • Shell parsing and all five required key/value checks passed.
  • The image-renaming utility parsed all five entries in dry-run mode under Python 3.13.
  • Repository pre-commit and DCO checks passed.

Targeted validation is limited to the exact GB300 PP4 disaggregated performance-sanity workload:

disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL

Throughput from this diagnostic will be interpreted only if every submitted request succeeds, the generation-first configuration is confirmed, coordinator mode is active on all four context pipeline ranks, shutdown is clean, and the official output-token metric is valid.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61061 [ run ] triggered by Bot. Commit: f44f23f Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61061 [ run ] completed with state FAILURE. Commit: f44f23f
/LLM/main/L0_MergeRequest_PR pipeline #49317 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants