[https://nvbugs/6410963][fix] Make cache transceiver transport reporting deterministic

[chienchunhung]

@coderabbitai summary

Description

test_single_node_transfer currently sets UCX_PROTO_INFO=used and requires selected_transport to be non-empty.

used emits diagnostics during UCX worker teardown, so the output depends on the active MPI/UCX runtime and object lifecycle. After the PyTorch 26.04 image transition, the KV transfer still completes successfully, but report.py observes no transport and the test fails.

This has caused the exact test to fail in all 46 observed DGX_H100-PyTorch-4 runs, including unrelated PRs and main.

Changes

• Use UCX_PROTO_INFO=y in test_single_node_transfer so protocol-selection tables are emitted deterministically.
• Keep the existing transfer-status and non-empty transport assertions.
• Include captured ctx/gen logs when transport detection fails.
• Bound TERM/SIGKILL cleanup and retain process-group IDs so failed launchers cannot leave MPI ranks behind.
• Add parser coverage ensuring transport detection uses a CUDA KV-data path rather than incidental MPI control traffic or echoed UCX_TLS values, including weighted UCX 1.21 multi-lane rows.
• Leave the normal cache-transceiver sweep default unchanged to avoid increasing production harness log volume.

Root Cause

The test and mandatory transport assertion were introduced by #14933. The PyTorch 26.04 runtime transition from #12643 exposed the teardown-dependent diagnostic behavior; it did not cause the KV transfer itself to fail.

Test Coverage

• tests/unittest/disaggregated/test_cache_transceiver_harness_report.py
• tests/unittest/disaggregated/test_cache_transceiver_harness.py
• Targeted CI: /bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-4"

Related PRs

• [TRTLLM-13168][feat] test best ucx env for cache transceiver #14933
• [TRTLLM-11715][infra] Upgrade dependencies for dlfw 26.04 stack #12643

GitHub Bot Help

To see a list of available CI bot commands, comment /bot help.

[chienchunhung]

/bot run --disable-fail-fast

[coderabbitai]

📝 Walkthrough

Walkthrough

This change updates a disaggregated cache transceiver test harness to manage two mpirun subprocesses via file-backed logging and explicit process-group lifecycle handling, replacing communicate()-based timeouts with deadline-based waits and SIGTERM/SIGKILL escalation. It also updates the UCX_PROTO_INFO environment value and related failure-message text.

Changes

Cache Transceiver Harness Test

Layer / File(s)	Summary
Timeout/termination helpers and constants tests/unittest/disaggregated/test_cache_transceiver_harness.py	Adds signal/time imports, log-tail size and timeout constants, and helper functions for log truncation, multi-process deadline waiting, and process-group termination.
Subprocess execution flow and log-file redirection tests/unittest/disaggregated/test_cache_transceiver_harness.py	Changes UCX_PROTO_INFO from "used" to "y", redirects mpirun output to per-role log files instead of pipes, launches processes as new sessions, waits via shared deadline, and terminates process groups on timeout/failure.
Assertion message updates using log tails tests/unittest/disaggregated/test_cache_transceiver_harness.py	Updates missing gen status file and aggregated-results assertion messages to include tailed ctx/gen logs with revised UCX protocol-table wording.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Test
  participant CtxProcess
  participant GenProcess

  Test->>CtxProcess: launch mpirun (new session, log file)
  Test->>GenProcess: launch mpirun (new session, log file)
  Test->>Test: wait_for_processes until deadline
  alt timeout occurs
    Test->>CtxProcess: terminate process group (SIGTERM then SIGKILL)
    Test->>GenProcess: terminate process group (SIGTERM then SIGKILL)
  end
  Test->>Test: read tailed logs from files
  Test->>Test: assert results / build failure message with log tails

Loading

Related Issues: None found in provided context.

Related PRs: None found in provided context.

Suggested labels: tests, disaggregated-serving

Suggested reviewers: None found in provided context.

🐰 A rabbit taps the process tree,
Signals fly with SIGTERM glee,
Logs now tucked in tidy files,
Timeouts tamed for country miles,
Hop along, the tests run free.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title matches the main change and follows the required ticket/type pattern.
Description check	✅ Passed	The description covers the issue, fix, root cause, test coverage, and related PRs, so it is mostly complete.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unittest/disaggregated/test_cache_transceiver_harness.py`:
- Around line 102-108: The cleanup path in test_cache_transceiver_harness’s
process shutdown logic leaves proc.wait() unbounded after the SIGKILL loop,
which can still hang if a child is stuck during teardown. Update the wait in the
processes loop to use a timeout, following the bounded pattern already used by
terminate() in disagg_test_utils, and keep the existing
killpg/ProcessLookupError handling intact so the helper cannot block
indefinitely.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 48d6c89b-cfe3-4139-828c-6f58602511dc

📥 Commits

Reviewing files that changed from the base of the PR and between 08d505f and 7350a92.

📒 Files selected for processing (1)

• tests/unittest/disaggregated/test_cache_transceiver_harness.py

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57270 [ run ] triggered by Bot. Commit: 79c053d Link to invocation

[chuangz0]

LGTM

[tensorrt-cicd]

PR_Github #57270 [ run ] completed with state SUCCESS. Commit: 79c053d
/LLM/main/L0_MergeRequest_PR pipeline #46032 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57574 [ run ] triggered by Bot. Commit: 79c053d Link to invocation

[tensorrt-cicd]

PR_Github #57574 [ run ] completed with state SUCCESS. Commit: 79c053d
/LLM/main/L0_MergeRequest_PR pipeline #46302 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_H100-PyTorch-4"

[tensorrt-cicd]

PR_Github #57795 [ run ] triggered by Bot. Commit: 723dd63 Link to invocation

[tensorrt-cicd]

PR_Github #57795 [ run ] completed with state SUCCESS. Commit: 723dd63
/LLM/main/L0_MergeRequest_PR pipeline #46492 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57833 [ run ] triggered by Bot. Commit: 723dd63 Link to invocation

[tensorrt-cicd]

PR_Github #57833 [ run ] completed with state ABORTED. Commit: 723dd63

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57864 [ run ] triggered by Bot. Commit: 723dd63 Link to invocation

[tensorrt-cicd]

PR_Github #57864 [ run ] completed with state FAILURE. Commit: 723dd63
/LLM/main/L0_MergeRequest_PR pipeline #46560 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #58040 [ run ] triggered by Bot. Commit: 723dd63 Link to invocation

[tensorrt-cicd]

PR_Github #58040 [ run ] completed with state SUCCESS. Commit: 723dd63
/LLM/main/L0_MergeRequest_PR pipeline #46711 completed with status: 'SUCCESS'

CI Report

Link to invocation