Skip to content

[https://nvbugs/6094100][fix] add ucx tls env in disagg related tests - #14626

Merged
chuangz0 merged 2 commits into
NVIDIA:mainfrom
chuangz0:test_disagg_worker_add_env
May 28, 2026
Merged

[https://nvbugs/6094100][fix] add ucx tls env in disagg related tests#14626
chuangz0 merged 2 commits into
NVIDIA:mainfrom
chuangz0:test_disagg_worker_add_env

Conversation

@chuangz0

@chuangz0 chuangz0 commented May 27, 2026

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Tests
    • Enhanced test configuration to properly handle GPU and architecture-specific settings
    • Adjusted test skip lists to refine test coverage across different system configurations

Review Change Stack

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@chuangz0
chuangz0 marked this pull request as ready for review May 27, 2026 09:19
@chuangz0
chuangz0 requested a review from a team as a code owner May 27, 2026 09:19
chuangz0 added 2 commits May 27, 2026 17:19
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
@chuangz0
chuangz0 force-pushed the test_disagg_worker_add_env branch from aa85131 to 70b730b Compare May 27, 2026 09:19
@chuangz0

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@coderabbitai

coderabbitai Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

This PR adds dynamic UCX transport configuration to disaggregated integration tests by introducing a get_ucx_tls() helper function across three test modules that computes transport settings from GPU SM version and host architecture, then applies the computed value to environment variables for context/generation servers and worker processes. Test waivers are updated to reflect tests now passing with the correct configuration.

Changes

UCX Transport Dynamic Configuration

Layer / File(s) Summary
etcd server UCX transport helper
tests/integration/defs/disaggregated/test_disaggregated_etcd.py
Adds platform and get_sm_version imports and introduces get_ucx_tls() helper; applies the computed UCX transport setting to both CONTEXT and GENERATION trtllm-serve server environments.
Single GPU test UCX transport helper
tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py
Adds platform and get_sm_version imports and introduces get_ucx_tls() helper; replaces hardcoded ^ib,gdr_copy with the dynamic helper across six test functions that launch MPIPoolExecutor.
Worker processes UCX transport helper
tests/integration/defs/disaggregated/test_workers.py
Adds platform and get_sm_version imports and introduces get_ucx_tls() helper with special ARM/SM 103 handling; updates background_workers() context manager to copy the base environment and apply the computed UCX_TLS value.
Test waivers cleanup
tests/integration/test_lists/waives.txt
Removes previously-waived TinyLlama-1.1B-Chat-v1.0 disaggregated and worker test entries; retains waivers only for a smaller targeted set of tests now with the UCX transport fix.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#14460: Adds dynamic get_ucx_tls() helper propagation to disaggregated integration test worker and server processes.
  • NVIDIA/TensorRT-LLM#14440: Removes and adjusts disaggregated test waivers that were blocking due to missing UCX transport environment configuration.

Suggested reviewers

  • jieli-matrix
  • xinhe-nv
  • fredricz-20070104
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description contains only the template with placeholders and no actual implementation details, explanations, or test coverage information filled in. Fill in the Description section explaining the issue and solution, and the Test Coverage section listing relevant tests that safeguard these changes.
Docstring Coverage ⚠️ Warning Docstring coverage is 61.54% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly indicates a fix to add UCX TLS environment variables in disaggregated-related tests, which aligns with the file changes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #50518 [ run ] triggered by Bot. Commit: 70b730b Link to invocation

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
tests/integration/defs/disaggregated/test_workers.py (1)

1-1: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add the required NVIDIA copyright/SPDX header to this modified file.

The file is modified but does not include the mandated header.

Suggested fix
+ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ # SPDX-License-Identifier: Apache-2.0
+
  import asyncio

As per coding guidelines: "**/*.{cpp,cc,h,hpp,py,cu,cuh}: Include NVIDIA copyright header on ALL new files; update year on modified files."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_workers.py` at line 1, This
modified test file is missing the required NVIDIA copyright/SPDX header; add the
standard multi-line NVIDIA copyright header and SPDX license identifier at the
very top of the file (above the existing import asyncio), and update the year
range in the header to include the current modification year so the file
complies with the coding guidelines.
tests/integration/defs/disaggregated/test_disaggregated_etcd.py (1)

1-1: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Update the modified-file copyright year in the header.

This file was changed, but the header still ends at 2024.

Suggested fix
-# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

As per coding guidelines: "**/*.{cpp,cc,h,hpp,py,cu,cuh}: Include NVIDIA copyright header on ALL new files; update year on modified files."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_disaggregated_etcd.py` at line 1,
The SPDX header's copyright year range (the line containing
"SPDX-FileCopyrightText") was not updated after modifying the file; update that
header in this file (the SPDX-FileCopyrightText line) to reflect the current
year by changing the trailing year from 2024 to 2026 (e.g., "2022-2026") so the
file header correctly shows the updated modification year.
🧹 Nitpick comments (4)
tests/integration/defs/disaggregated/test_workers.py (1)

30-30: ⚡ Quick win

Add a return type annotation to get_ucx_tls.

Please annotate it as returning str.

Suggested fix
-def get_ucx_tls():
+def get_ucx_tls() -> str:

As per coding guidelines: "Always annotate Python functions with return type; use None if the function does not return anything."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_workers.py` at line 30, Annotate
the function signature for get_ucx_tls with an explicit return type of str
(i.e., change its definition to include -> str) so the function declaration
reads with a return type annotation; update the get_ucx_tls function declaration
accordingly and ensure any related docstrings or type hints remain consistent.
tests/integration/defs/disaggregated/test_disaggregated_etcd.py (1)

29-29: ⚡ Quick win

Add an explicit return type to the new helper.

Please annotate get_ucx_tls with -> str.

Suggested fix
-def get_ucx_tls():
+def get_ucx_tls() -> str:

As per coding guidelines: "Always annotate Python functions with return type; use None if the function does not return anything."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_disaggregated_etcd.py` at line 29,
Annotate the helper function get_ucx_tls with an explicit return type by
changing its signature to include "-> str" (i.e., declare get_ucx_tls returns a
str) so the function follows the project's typing guideline for return
annotations; update the function definition in
tests/integration/defs/disaggregated/test_disaggregated_etcd.py accordingly.
tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py (2)

287-289: QA list follow-up: no update needed for this cohort.

These changes adjust runtime env for existing disaggregated tests but do not add new pytest node IDs; QA list file updates are unnecessary here as long as selectors/names remain unchanged.

As per coding guidelines: "If the change adds or materially alters an integration test under tests/integration/defs/... call out whether an entry is needed under tests/integration/test_lists/qa/."

Also applies to: 436-438, 547-551, 638-640, 747-749, 853-855

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py` around
lines 287 - 289, This change only adjusts the runtime env passed into the
executor (the UCX_TLS via get_ucx_tls() and UCX_MM_ERROR_HANDLING) inside the
executor context in
tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py, so verify
that no test function or pytest node id was renamed/added and therefore no
update to tests/integration/test_lists/qa/* is required; if you did change any
test names/selectors or added new tests, update the corresponding QA list entry,
otherwise leave QA files untouched.

20-20: ⚡ Quick win

Annotate get_ucx_tls return type.

Please declare -> str on this helper.

Suggested fix
-def get_ucx_tls():
+def get_ucx_tls() -> str:

As per coding guidelines: "Always annotate Python functions with return type; use None if the function does not return anything."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py` at
line 20, Annotate the helper function get_ucx_tls with an explicit return type
by changing its signature to include "-> str"; locate the get_ucx_tls definition
and add the return type annotation so the function reads like "def get_ucx_tls()
-> str:" ensuring it satisfies the project's function return-type requirement.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py`:
- Line 1: This file is missing the mandated NVIDIA copyright/SPDX header: add
the project's standard NVIDIA header block at the very top of
test_disaggregated_single_gpu.py (before any imports such as asyncio), update
the copyright year range to include the current modification year, and include
the required SPDX identifier exactly as used across the repo so the header
matches other modified Python files.

---

Outside diff comments:
In `@tests/integration/defs/disaggregated/test_disaggregated_etcd.py`:
- Line 1: The SPDX header's copyright year range (the line containing
"SPDX-FileCopyrightText") was not updated after modifying the file; update that
header in this file (the SPDX-FileCopyrightText line) to reflect the current
year by changing the trailing year from 2024 to 2026 (e.g., "2022-2026") so the
file header correctly shows the updated modification year.

In `@tests/integration/defs/disaggregated/test_workers.py`:
- Line 1: This modified test file is missing the required NVIDIA copyright/SPDX
header; add the standard multi-line NVIDIA copyright header and SPDX license
identifier at the very top of the file (above the existing import asyncio), and
update the year range in the header to include the current modification year so
the file complies with the coding guidelines.

---

Nitpick comments:
In `@tests/integration/defs/disaggregated/test_disaggregated_etcd.py`:
- Line 29: Annotate the helper function get_ucx_tls with an explicit return type
by changing its signature to include "-> str" (i.e., declare get_ucx_tls returns
a str) so the function follows the project's typing guideline for return
annotations; update the function definition in
tests/integration/defs/disaggregated/test_disaggregated_etcd.py accordingly.

In `@tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py`:
- Around line 287-289: This change only adjusts the runtime env passed into the
executor (the UCX_TLS via get_ucx_tls() and UCX_MM_ERROR_HANDLING) inside the
executor context in
tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py, so verify
that no test function or pytest node id was renamed/added and therefore no
update to tests/integration/test_lists/qa/* is required; if you did change any
test names/selectors or added new tests, update the corresponding QA list entry,
otherwise leave QA files untouched.
- Line 20: Annotate the helper function get_ucx_tls with an explicit return type
by changing its signature to include "-> str"; locate the get_ucx_tls definition
and add the return type annotation so the function reads like "def get_ucx_tls()
-> str:" ensuring it satisfies the project's function return-type requirement.

In `@tests/integration/defs/disaggregated/test_workers.py`:
- Line 30: Annotate the function signature for get_ucx_tls with an explicit
return type of str (i.e., change its definition to include -> str) so the
function declaration reads with a return type annotation; update the get_ucx_tls
function declaration accordingly and ensure any related docstrings or type hints
remain consistent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e609413b-5da6-4367-998c-25c4c8270e42

📥 Commits

Reviewing files that changed from the base of the PR and between 8cd28ed and 70b730b.

📒 Files selected for processing (4)
  • tests/integration/defs/disaggregated/test_disaggregated_etcd.py
  • tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py
  • tests/integration/defs/disaggregated/test_workers.py
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #50518 [ run ] completed with state SUCCESS. Commit: 70b730b
/LLM/main/L0_MergeRequest_PR pipeline #40026 completed with status: 'SUCCESS'

CI Report

Link to invocation

@chuangz0
chuangz0 enabled auto-merge (squash) May 28, 2026 01:31

@fredricz-20070104 fredricz-20070104 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No major issues. Approved

@chuangz0
chuangz0 merged commit c90fe19 into NVIDIA:main May 28, 2026
11 checks passed
bmarimuthu-nv pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull request May 28, 2026
…NVIDIA#14626)

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants