Skip to content

[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: compatible current-runtime midpoint - #16765

Closed
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-current-runtime-midpoint-compatible
Closed

[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: compatible current-runtime midpoint#16765
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-current-runtime-midpoint-compatible

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

Caution

TEST ONLY; DO NOT REVIEW OR MERGE. This draft exists only to run one bounded source-localization experiment.

Purpose

This arm applies the C++ context-transfer async-consensus factor from the official implementation draft to native main checkpoint 454fed35cc2950a771dea1140620e3b427347697 while preserving the current runtime image and exact GB300 workload.

The earlier midpoint was too old to benchmark with the current image: pytest stopped during import because DSparkWorker still overrode SpecWorkerBase.forward. Native commit fa54a19df526fb133791d55eb718a0c757c9c751 is the first compatible checkpoint because it completes the forward to _forward_impl migration.

This checkpoint is the first-parent median of the remaining compatible interval:

  • 17 commits after the compatibility boundary
  • 18 commits before current product base 6df513420e9024dd09064492a953dda579eb4247

The lower boundary is only a compatibility boundary; it is not a measured-good throughput point under the current runtime. This is therefore compatibility-constrained localization, not a conventional known-good/known-bad binary bisect.

Factor control

  • Exact native parent: 454fed35cc2950a771dea1140620e3b427347697
  • Async factor: byte-identical to 6df513420e9024dd09064492a953dda579eb4247..a2fca7808c94f90ca396fb047fd3d0887e293706
  • Stable patch ID: 549e0085a06212cde0c47ab47072ad6acdcd1f58
  • Factor tree: d428de675a98b7d88e06dbb355059308cf56f198
  • Current image-tag blob unchanged: 504fd8f234a8d10076e82f000d35e41507566593
  • Exact GB300 workload YAML blob unchanged: 14aa2d0a4b24c912e8f0ebed751974fbe1ade9f6
  • No workload, selector, timeout, admission, transport, consensus, or logging adaptation

Targeted verification

Local static verification:

  • full pre-commit and commit-message hooks passed
  • independent diff review found no definite P0-P2 issue
  • all 11 factor endpoint blobs and modified-file preimages match the official factor exactly

CI target:

  • Stage: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1
  • Selector: disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL
  • Fail-fast disabled from launch so unrelated sibling failures cannot censor the target stage

The run is interpretable only if the exact selector completes all requests and emits a valid official output-token metric. Any request failure, timeout, or pre-benchmark failure censors throughput.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61119 [ run ] triggered by Bot. Commit: fdb52f9 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #61119 [ run ] completed with state FAILURE. Commit: fdb52f9
/LLM/main/L0_MergeRequest_PR pipeline #49373 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants