[TRTLLM-13553][feat] Add WideEP rank health telemetry (1d.3)

[chienchunhung]

Summary

• add a passive EPGroupHealth metrics adapter for WideEP FT item 1d.3
• carry coherent per-rank health snapshots from the rank-0 PyExecutor through a dedicated internal RPC to trtllm-serve
• expose atomic Prometheus rank, active/failed-count, generation, and availability metrics
• cover producer replacement, pending registration, transient RPC loss, unsupported backends, and cancellation-safe shutdown

Why

EPGroupHealth is process-local, while Prometheus is served by the OpenAI server process. Existing multiprocess gauges also merge each metric independently, so they cannot guarantee that the rank vector, aggregate counts, generation, and validity bit come from one coherent health snapshot.

This change keeps telemetry observational: it reads the producer-owned tracker from the canonical ModelConfig.extra_attrs seam and never creates a separate always-healthy tracker or changes communication selection. On the current 1a.1-only base, the adapter remains dormant until a detection/communication producer registers that tracker. An explicit key: None reservation supports late attachment without polling ordinary unsupported backends.

Behavior

• exports trtllm_ep_rank_active{ep_rank=...}, trtllm_ep_active_ranks, trtllm_ep_failed_ranks, trtllm_ep_health_generation, and trtllm_ep_health_available
• emits no EP metric families before the first valid producer snapshot
• retains the last coherent snapshot during an outage and sets availability to 0
• uses a producer epoch so generation can safely restart after rank-0 replacement
• retries transport timeouts, ZeroMQ failures, and RPCCancelled; deterministic configuration/topology failures stop polling
• rejects multi-group PP / MoE-TP / MoE-cluster topologies until the serving endpoint can identify every EP group

Validation

• 52 focused EPGroupHealth / serialization tests
• 105 metrics collector tests with prometheus_client 0.23
• 105 metrics collector tests with prometheus_client 0.18
• exact-source harnesses for PyExecutor late attachment, RPC forwarding, pending registration, producer replacement, and cancellation-safe drain
• Ruff format/lint, YAPF, isort, autoflake, legacy lint baseline, py_compile, Python 3.10 grammar, YAML parsing, and git diff --check

Full Torch/GPU execution was not available locally; the new unit files are added to B200 CI routing.

Corrected telemetry contract (2026-06-30)

This PR is a passive observer of coordinator-committed data-plane membership:

• active, failed, and generation describe the committed EP mask, not raw physical liveness or suspicion;
• failedRanks and existing metric names remain unchanged for compatibility, while documentation and Prometheus HELP text clarify their meaning;
• telemetry never creates, enables, or mutates EPGroupHealth, and it cannot drive recovery;
• detected/suspected state and recovery-phase metrics remain separate follow-up work;
• the shared EPGroupHealth module contract is textually aligned with [TRTLLM-12557][feat] WideEP FT: add AlltoAll watchdog (1a.3 + 1a.4) #15524 so the two PRs compose without a needless conflict.

No payload key, metric name, public API, or control flow changed in this correction.

Correction commits/heads: 61bef56e, 94274a3f

Canonical plan: https://github.com/chienchunhung/TensorRT-LLM/blob/docs-and-plans/docs/design/wide-ep-fault-tolerance/pr-execution/08-implementation-plan.md

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56690 [ run ] triggered by Bot. Commit: eacdcd0 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56716 [ run ] triggered by Bot. Commit: 61bef56 Link to invocation

[tensorrt-cicd]

PR_Github #56690 [ run ] completed with state ABORTED. Commit: eacdcd0

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56724 [ run ] triggered by Bot. Commit: 94274a3 Link to invocation

[tensorrt-cicd]

PR_Github #56716 [ run ] completed with state ABORTED. Commit: 61bef56

Link to invocation

[tensorrt-cicd]

PR_Github #56724 [ run ] completed with state SUCCESS. Commit: 94274a3
/LLM/main/L0_MergeRequest_PR pipeline #45544 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-9,DGX_H100-PyTorch-1,DGX_B200-4_GPUs-PyTorch-3,DGX_H100-4_GPUs-PyTorch-DeepSeek-1,DGX_B200-8_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-2"

[tensorrt-cicd]

PR_Github #57010 [ run ] triggered by Bot. Commit: 94274a3 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57036 [ run ] triggered by Bot. Commit: 2ea53a2 Link to invocation

[tensorrt-cicd]

PR_Github #57010 [ run ] completed with state ABORTED. Commit: 94274a3

Link to invocation

[tensorrt-cicd]

PR_Github #57036 [ run ] completed with state FAILURE. Commit: 2ea53a2
/LLM/main/L0_MergeRequest_PR pipeline #45830 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-TensorRT-1,DGX_B200-4_GPUs-PyTorch-1,DGX_H100-4_GPUs-PyTorch-Others-1,H100_PCIe-AutoDeploy-1"

[tensorrt-cicd]

PR_Github #57567 [ run ] triggered by Bot. Commit: 2ea53a2 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57576 [ run ] triggered by Bot. Commit: 32e5210 Link to invocation

[tensorrt-cicd]

PR_Github #57567 [ run ] completed with state ABORTED. Commit: 2ea53a2

Link to invocation

[tensorrt-cicd]

PR_Github #57576 [ run ] completed with state SUCCESS. Commit: 32e5210
/LLM/main/L0_MergeRequest_PR pipeline #46304 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_B200-PyTorch-4,DGX_B200-4_GPUs-PyTorch-2,GH200-PackageSanityCheck-PY312-DLFW,GB200-4_GPUs-PyTorch-PerfSanity-2"

[tensorrt-cicd]

PR_Github #57851 [ run ] triggered by Bot. Commit: 32e5210 Link to invocation

[tensorrt-cicd]

PR_Github #57851 [ run ] completed with state SUCCESS. Commit: 32e5210
/LLM/main/L0_MergeRequest_PR pipeline #46547 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57901 [ run ] triggered by Bot. Commit: 32e5210 Link to invocation

[tensorrt-cicd]

PR_Github #57901 [ run ] completed with state SUCCESS. Commit: 32e5210
/LLM/main/L0_MergeRequest_PR pipeline #46595 completed with status: 'SUCCESS'

CI Report

Link to invocation