[https://nvbugs/6435642][fix] detect killed MPI executor workers

[chienchunhung]

What changed

• Have executor workers report a stable process identity containing rank, PID, Linux process start time, hostname, and PID namespace.
• Monitor locally spawned MpiPoolSession workers with pidfds, with a /proc identity check as fallback.
• Treat detected worker death as fatal in both GenerationExecutorProxy.check_health() and the background error monitor, then initiate the existing nonblocking shutdown path.
• Protect registration against PID reuse and avoid monitoring workers from a different host or PID namespace.

Why

NVBug 6435642 tracks a failure mode where the mpi4py.futures.server process is killed by OOM/SIGKILL while its future remains pending and no error reaches the proxy error queue. The parent TRT-LLM/Dynamo worker therefore remains alive and continues reporting healthy even though inference capability is lost.

PR #12718 added the fatal-error propagation and shutdown foundation, but hard-killed MPI workers can bypass its future/error-queue checks. Active OS process-liveness monitoring closes that gap for the locally spawned MpiPoolSession topology.

Impact

When an engine-core MPI worker is killed, check_health() now returns False, records a fatal error containing the worker rank and PID, and allows embedding runtimes such as Dynamo to remove and recover the failed worker instead of routing requests that will hang.

Validation

• pytest -q tests/unittest/executor/test_fatal_error_health_check.py --confcutdir=tests/unittest/executor -k 'not TestOpenAIHealthEndpoint'
  • 72 passed, 1 skipped, 3 deselected
  • The real SIGKILL regression is Linux-only and was skipped on the macOS development host.
• Python bytecode compilation passed for all changed Python files.
• isort, YAPF, Ruff, Ruff formatting, and all applicable commit hooks passed.
• Two unrelated repository-wide test-list hooks were skipped because their cached Python interpreter cannot evaluate the existing str | None annotation in scripts/check_test_list.py.

NVBug: 6435642

Summary by CodeRabbit

• New Features

  • Improved executor health monitoring to detect failed local worker processes, including cases where communication status remains pending.
  • Automatically records fatal worker failures and begins shutdown when a dead worker is detected.
  • Added platform-aware process liveness detection for more reliable failure identification.

• Tests

  • Added coverage for process monitoring, worker registration, and fatal health-check behavior after unexpected worker termination.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-PyTorch-1, A10-PyTorch-2"

[tensorrt-cicd]

PR_Github #59029 [ run ] triggered by Bot. Commit: 67d5ab1 Link to invocation

[tensorrt-cicd]

PR_Github #59029 [ run ] completed with state SUCCESS. Commit: 67d5ab1
/LLM/main/L0_MergeRequest_PR pipeline #47555 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59050 [ run ] triggered by Bot. Commit: 67d5ab1 Link to invocation

[coderabbitai]

📝 Walkthrough

Walkthrough

The proxy now monitors local worker process liveness, workers report process identities during initialization, and fatal shutdown can occur when a process dies while its MPI future remains pending. Tests cover pidfd, procfs, identity filtering, and proxy health behavior.

Changes

Worker liveness monitoring

Layer / File(s)	Summary
Process identity and liveness monitor tensorrt_llm/executor/worker_process_monitor.py	Adds worker identity capture and pidfd/procfs-based detection of dead local processes, with cleanup support.
Worker identity handshake tensorrt_llm/executor/worker.py	Gathers per-rank process identities and includes them in the worker-ready payload.
Proxy liveness integration tensorrt_llm/executor/proxy.py	Registers worker identities, checks monitored workers during health and error monitoring, and closes the monitor during shutdown.
Liveness detection validation tests/unittest/executor/test_fatal_error_health_check.py	Tests process detection mechanisms, registration filtering, shutdown cleanup, and detection of killed workers with pending MPI futures.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Worker as MPI worker leader
  participant Proxy as GenerationExecutorProxy
  participant Monitor as WorkerProcessMonitor
  participant Process as Local worker process

  Worker->>Proxy: ready status with worker identities
  Proxy->>Monitor: register identities
  Monitor->>Process: open pidfd or read procfs state
  Process-->>Monitor: exit event or changed process state
  Monitor-->>Proxy: dead worker identity
  Proxy->>Proxy: record fatal error and call pre_shutdown()

Loading

Suggested reviewers: junyixu-nv, hhzhang16, sunnyqgg, tedzhouhk, boreas618

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR's main change and includes a valid NVBugs reference and fix type.
Description check	✅ Passed	The description explains the problem, solution, impact, and validation, covering the template's required intent.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/unittest/executor/test_fatal_error_health_check.py (1)

523-624: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Two coverage gaps in TestWorkerProcessMonitor.

Well-covered overall, but missing:

1. A test where os.pidfd_open raises a generic OSError (not ProcessLookupError) — should fall back to procfs registration (lines 122-123 in worker_process_monitor.py), currently untested.
2. A test for the zombie-state ("Z") death branch specifically (only start-time mismatch is exercised today, in both the pidfd post-open-validation and find_dead_worker procfs paths).

As per path instructions, "suggest concrete list file names and whether coverage is sufficient, insufficient, or needs follow-up outside the PR": add these two cases to tests/unittest/executor/test_fatal_error_health_check.py::TestWorkerProcessMonitor.

🧪 Suggested additional tests

def test_pidfd_open_generic_oserror_falls_back_to_procfs(self, identity):
    if identity.start_time is None:
        identity = identity._replace(start_time=100)
    monitor = WorkerProcessMonitor()
    with patch.object(_monitor_mod.os, "pidfd_open", side_effect=OSError("EMFILE"), create=True):
        monitor.register([identity])
    assert identity in monitor._procfs_identities


def test_zombie_state_is_detected(self, identity):
    if identity.start_time is None:
        identity = identity._replace(start_time=100)
    monitor = WorkerProcessMonitor()
    with (
        patch.object(_monitor_mod.os, "pidfd_open", None, create=True),
        patch.object(_monitor_mod, "_read_process_state", return_value=("Z", identity.start_time)),
    ):
        monitor.register([identity])
        assert monitor.find_dead_worker() == identity

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/executor/test_fatal_error_health_check.py` around lines 523 -
624, Add two cases to TestWorkerProcessMonitor: verify a generic OSError from
os.pidfd_open falls back to procfs registration and retains the identity in
_procfs_identities, and verify a procfs state of "Z" is reported by
find_dead_worker as the dead identity. Preserve the existing start-time setup
and mocking patterns used by the neighboring tests.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/unittest/executor/test_fatal_error_health_check.py`:
- Around line 523-624: Add two cases to TestWorkerProcessMonitor: verify a
generic OSError from os.pidfd_open falls back to procfs registration and retains
the identity in _procfs_identities, and verify a procfs state of "Z" is reported
by find_dead_worker as the dead identity. Preserve the existing start-time setup
and mocking patterns used by the neighboring tests.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 00a922fa-3189-4f1f-8116-5cbf65686d17

📥 Commits

Reviewing files that changed from the base of the PR and between 43dfc81 and 67d5ab1.

📒 Files selected for processing (4)

• tensorrt_llm/executor/proxy.py
• tensorrt_llm/executor/worker.py
• tensorrt_llm/executor/worker_process_monitor.py
• tests/unittest/executor/test_fatal_error_health_check.py

[JunyiXu-nv]

Overall LGTM. CC @QiJune for viz

[tensorrt-cicd]

PR_Github #59050 [ run ] completed with state SUCCESS. Commit: 67d5ab1
/LLM/main/L0_MergeRequest_PR pipeline #47575 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_B200-2_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-Ray-1, Build-SBSA"

[tensorrt-cicd]

PR_Github #59123 [ run ] triggered by Bot. Commit: 67d5ab1 Link to invocation

[tensorrt-cicd]

PR_Github #59123 [ run ] completed with state FAILURE. Commit: 67d5ab1
/LLM/main/L0_MergeRequest_PR pipeline #47633 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_B200-2_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-Ray-1"

[tensorrt-cicd]

PR_Github #59231 [ run ] triggered by Bot. Commit: 67d5ab1 Link to invocation

[tensorrt-cicd]

PR_Github #59231 [ run ] completed with state SUCCESS. Commit: 67d5ab1
/LLM/main/L0_MergeRequest_PR pipeline #47725 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59284 [ run ] triggered by Bot. Commit: 67d5ab1 Link to invocation

[tensorrt-cicd]

PR_Github #59284 [ run ] completed with state SUCCESS. Commit: 67d5ab1
/LLM/main/L0_MergeRequest_PR pipeline #47771 completed with status: 'SUCCESS'

CI Report

Link to invocation