Skip to content

[https://nvbugs/6418912][fix] Prevent BufferIndexHolder test deadlock - #15994

Closed
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:user/chienchunhung/nvbug-6418912-buffer-holder-test
Closed

[https://nvbugs/6418912][fix] Prevent BufferIndexHolder test deadlock#15994
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:user/chienchunhung/nvbug-6418912-buffer-holder-test

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Configure BufferIndexHolderLifecycleTest with two send and receive slots so move-assignment coverage can hold both the source and destination indices simultaneously.
  • Assert the two-slot precondition before acquiring indices, turning any future fixture regression into an immediate failure instead of a one-hour timeout.

Root cause

MoveAssignReleasesPriorThenTransfers acquired two buffer indices before constructing either RAII holder. Both pools default to one slot, so the second acquisition waited indefinitely on the pool condition variable and the move assignment was never exercised.

This was reported by NVBUG 6418912 in A30-CPP-Post-Merge-2; both the send and receive variants timed out.

Why this escaped earlier

PR #14768 added the test, but its recorded pre-merge runs did not execute the A30 C++ post-merge stage. The final PR head used /bot skip after separate performance verification in #14916. The fixture also depended implicitly on CI-provided concurrency environment values, so it could appear healthy anywhere those values provided at least two slots.

This change makes the required capacity explicit and removes that environment dependence.

Validation

Earlier standalone pipelines #46503, #46532, and #46562 did not contain all companion fixes, so their later KV-cache failures do not invalidate this BufferIndexHolder fix.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "A30-CPP-Post-Merge-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #57806 [ run ] triggered by Bot. Commit: 2478323 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #57806 [ run ] completed with state SUCCESS. Commit: 2478323
/LLM/main/L0_MergeRequest_PR pipeline #46503 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "A30-CPP-Post-Merge-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #57836 [ run ] triggered by Bot. Commit: 2478323 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #57836 [ run ] completed with state SUCCESS. Commit: 2478323
/LLM/main/L0_MergeRequest_PR pipeline #46532 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #57865 [ run ] triggered by Bot. Commit: 2478323 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #57865 [ run ] completed with state FAILURE. Commit: 2478323
/LLM/main/L0_MergeRequest_PR pipeline #46562 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-9"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58042 [ run ] triggered by Bot. Commit: 2478323 Link to invocation

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

Closing this PR as it is subsumed by #16019

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58042 [ run ] completed with state ABORTED. Commit: 2478323

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants