Skip to content

[https://nvbugs/6244695][fix] Revert Pass IPC HMAC key through file descriptor - #14782

Merged
longlee0622 merged 1 commit into
NVIDIA:mainfrom
chenfeiz0326:chenfeiz/revert-ipc-hmac-key-fd-50ca49f8c5
Jun 1, 2026
Merged

[https://nvbugs/6244695][fix] Revert Pass IPC HMAC key through file descriptor#14782
longlee0622 merged 1 commit into
NVIDIA:mainfrom
chenfeiz0326:chenfeiz/revert-ipc-hmac-key-fd-50ca49f8c5

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented May 30, 2026

Copy link
Copy Markdown
Collaborator

…descriptor (#14378)"

This reverts commit 50ca49f.

Summary by CodeRabbit

Release Notes

  • Refactor

    • Streamlined inter-process communication (IPC) authentication mechanism for internal proxy processes by simplifying key management and eliminating legacy file descriptor-based passing.
  • Tests

    • Removed outdated tests for deprecated authentication method.

Review Change Stack

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@chenfeiz0326
chenfeiz0326 requested a review from a team as a code owner May 30, 2026 11:16
@chenfeiz0326
chenfeiz0326 requested a review from zhenhuaw-me May 30, 2026 11:16
@chenfeiz0326 chenfeiz0326 changed the title Revert "[https://nvbugs/5972776][fix] Pass IPC HMAC key through file … [https://nvbugs/6244695][fix] Revert Pass IPC HMAC key through file descriptor May 30, 2026
@coderabbitai

coderabbitai Bot commented May 30, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Spawn-proxy IPC authentication migrates from file-descriptor-based passing with cached helpers to direct hex-encoded environment variable export. The core accessor function, launcher script generation, and MPI server integration all simplify by removing FD plumbing and cached state, and the test module covering the old FD mechanism is removed.

Changes

IPC HMAC Key Mechanism Refactor

Layer / File(s) Summary
IPC HMAC key contract and accessor
tensorrt_llm/executor/utils.py
Enum adds TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY environment variable, removes TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD. Function get_spawn_proxy_process_ipc_hmac_key_env() now reads hex-encoded key directly from env, asserts presence, and converts to bytes; removes FD-based reading and cached normalization helpers.
HMAC key generation in launch script
tensorrt_llm/llmapi/trtllm-llmapi-launch
Generates random 32-byte hex key via openssl rand and exports as TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY. Rank-0 task execution and MGMN leader invocation remove the run_with_ipc_hmac_key wrapper, execute directly, and preserve exit-code capture.
Disaggregated MPI leader HMAC key integration
tensorrt_llm/commands/serve.py
Imports LlmLauncherEnvs, directly sets TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY in child-process environment dict, removes FD assertions, and launches proxy via simplified subprocess.Popen without file-descriptor passing. Signal handler cleanup logic is removed from the finally block.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly Related PRs

  • NVIDIA/TensorRT-LLM#14378: Prior work introducing the FD-based TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD mechanism that this PR replaces with direct environment variable passing.
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The PR description lacks substantive explanation beyond the revert statement; required template sections (Description, Test Coverage) are empty with only placeholder comments shown. Add a clear explanation of why the previous IPC HMAC key file descriptor approach is being reverted and document any test coverage or validation that verifies the revert works correctly.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies this as a revert of a previous commit addressing IPC HMAC key handling via file descriptors, which matches the changeset modifications.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/executor/utils.py`:
- Line 26: The new constant TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY exposes the
IPC HMAC secret via process environment—restore the previous non-environment
mechanism: remove usage of TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY from any
subprocess env setup and instead pass the HMAC key over the narrowly-scoped
inherited-FD / dedicated IPC channel used by ZeroMqQueue (or via explicit file
descriptor/pipe passed to the child), update the code paths in ZeroMqQueue and
the launcher/spawn code that currently read/write this env var so they accept
the key from the inherited-FD/IPC handle, and delete the environment-constant
definition so the secret is never placed into os.environ or subprocess env.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 90405160-bfc2-4fc3-a9e1-5c178eafe732

📥 Commits

Reviewing files that changed from the base of the PR and between f20858c and 99d27ff.

📒 Files selected for processing (4)
  • tensorrt_llm/commands/serve.py
  • tensorrt_llm/executor/utils.py
  • tensorrt_llm/llmapi/trtllm-llmapi-launch
  • tests/unittest/executor/test_launcher_envs.py
💤 Files with no reviewable changes (1)
  • tests/unittest/executor/test_launcher_envs.py

TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR = "TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR"
TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD = (
"TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD")
TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY = "TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Keep the IPC HMAC key off the environment.

ZeroMqQueue uses this key to authenticate pickle traffic before pickle.loads() in tensorrt_llm/executor/ipc.py:33-103,354-389. Switching the contract back to TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY means the secret now propagates through process environments in both launcher paths instead of staying on a narrowly scoped channel like the previous inherited-FD mechanism. That weakens the only barrier guarding unauthorized IPC deserialization.

Also applies to: 39-43

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/executor/utils.py` at line 26, The new constant
TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY exposes the IPC HMAC secret via process
environment—restore the previous non-environment mechanism: remove usage of
TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY from any subprocess env setup and instead
pass the HMAC key over the narrowly-scoped inherited-FD / dedicated IPC channel
used by ZeroMqQueue (or via explicit file descriptor/pipe passed to the child),
update the code paths in ZeroMqQueue and the launcher/spawn code that currently
read/write this env var so they accept the key from the inherited-FD/IPC handle,
and delete the environment-constant definition so the secret is never placed
into os.environ or subprocess env.

@chenfeiz0326
chenfeiz0326 force-pushed the chenfeiz/revert-ipc-hmac-key-fd-50ca49f8c5 branch from 99d27ff to 588868b Compare May 30, 2026 11:22
@chenfeiz0326 chenfeiz0326 changed the title [https://nvbugs/6244695][fix] Revert Pass IPC HMAC key through file descriptor [https://nvbugs/6244695][fix] Revert Pass IPC HMAC key through file descriptor May 30, 2026
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --post-merge --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51187 [ run ] triggered by Bot. Commit: 588868b Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51187 [ run ] completed with state FAILURE. Commit: 588868b
/LLM/main/L0_MergeRequest_PR pipeline #40617 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…descriptor (NVIDIA#14378)"

This reverts commit 50ca49f.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@chenfeiz0326
chenfeiz0326 force-pushed the chenfeiz/revert-ipc-hmac-key-fd-50ca49f8c5 branch from 588868b to a003643 Compare May 31, 2026 03:04
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "*PerfSanity*"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51230 [ run ] triggered by Bot. Commit: a003643 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #51230 [ run ] completed with state FAILURE. Commit: a003643
/LLM/main/L0_MergeRequest_PR pipeline #40653 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@longlee0622
longlee0622 merged commit a422420 into NVIDIA:main Jun 1, 2026
6 of 7 checks passed
chenfeiz0326 added a commit to chenfeiz0326/TensorRT-LLM that referenced this pull request Jul 6, 2026
…escriptor (NVIDIA#15654)

Reverts NVIDIA#15654 (48fc753).

PR NVIDIA#15654 passes the IPC HMAC key from the parent process to the LLM-API
proxy child via a file descriptor. In the QA-pytest Slurm/MPI launch
path used by trtllm-bench and trtllm-serve, the fd handshake between
Rank0's forked subshell (spawning internal mpirun) and the
mgmn_leader_node RemoteMpiCommSessionServer deadlocks silently:

  - trtllm-bench (nvbugs/6388787): after "start MpiSession with N
    workers" the bench process emits no further stdout for ~30 min,
    workers stay alive at 0% GPU / 4 MiB VRAM, no exception, no MPI
    abort. Perf harness SIGKILLs at _STALL_TIMEOUT=1800s.

  - trtllm-serve (nvbugs/6405747): serve mode's /health endpoint never
    binds, and pytest fails with
    "Server http://localhost:<port>/health did not become ready within
    3600s" on both baseline and candidate wheels.

This is the third time the "Pass IPC HMAC key via fd" idea has broken
bench/serve (previously PR NVIDIA#14378 was reverted by PR NVIDIA#14782 for the
same class of failure: BlockingIOError in _read_spawn_proxy_process_
ipc_hmac_key_fd). Reverting until the fd inheritance across fork+exec
into the mpirun child is verified end-to-end for both bench and serve
launch paths.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
chenfeiz0326 added a commit to chenfeiz0326/TensorRT-LLM that referenced this pull request Jul 6, 2026
…escriptor (NVIDIA#15654)

Reverts NVIDIA#15654 (48fc753).

PR NVIDIA#15654 passes the IPC HMAC key from the parent process to the LLM-API
proxy child via a file descriptor. In the QA-pytest Slurm/MPI launch
path used by trtllm-bench and trtllm-serve, the fd handshake between
Rank0's forked subshell (spawning internal mpirun) and the
mgmn_leader_node RemoteMpiCommSessionServer deadlocks silently:

  - trtllm-bench (nvbugs/6388787): after "start MpiSession with N
    workers" the bench process emits no further stdout for ~30 min,
    workers stay alive at 0% GPU / 4 MiB VRAM, no exception, no MPI
    abort. Perf harness SIGKILLs at _STALL_TIMEOUT=1800s.

  - trtllm-serve (nvbugs/6405747): serve mode's /health endpoint never
    binds, and pytest fails with
    "Server http://localhost:<port>/health did not become ready within
    3600s" on both baseline and candidate wheels.

This is the third time the "Pass IPC HMAC key via fd" idea has broken
bench/serve (previously PR NVIDIA#14378 was reverted by PR NVIDIA#14782 for the
same class of failure: BlockingIOError in _read_spawn_proxy_process_
ipc_hmac_key_fd). Reverting until the fd inheritance across fork+exec
into the mpirun child is verified end-to-end for both bench and serve
launch paths.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
chenfeiz0326 added a commit to chenfeiz0326/TensorRT-LLM that referenced this pull request Jul 6, 2026
…escriptor (NVIDIA#15654)

Reverts NVIDIA#15654 (48fc753).

PR NVIDIA#15654 passes the IPC HMAC key from the parent process to the LLM-API
proxy child via a file descriptor. In the QA-pytest Slurm/MPI launch
path used by trtllm-bench and trtllm-serve, the fd handshake between
Rank0's forked subshell (spawning internal mpirun) and the
mgmn_leader_node RemoteMpiCommSessionServer deadlocks silently:

  - trtllm-bench (nvbugs/6388787): after "start MpiSession with N
    workers" the bench process emits no further stdout for ~30 min,
    workers stay alive at 0% GPU / 4 MiB VRAM, no exception, no MPI
    abort. Perf harness SIGKILLs at _STALL_TIMEOUT=1800s.

  - trtllm-serve (nvbugs/6405747): serve mode's /health endpoint never
    binds, and pytest fails with
    "Server http://localhost:<port>/health did not become ready within
    3600s" on both baseline and candidate wheels.

This is the third time the "Pass IPC HMAC key via fd" idea has broken
bench/serve (previously PR NVIDIA#14378 was reverted by PR NVIDIA#14782 for the
same class of failure: BlockingIOError in _read_spawn_proxy_process_
ipc_hmac_key_fd). Reverting until the fd inheritance across fork+exec
into the mpirun child is verified end-to-end for both bench and serve
launch paths.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants