[TRTLLM-12561][feat] Add recoverable MoE A2A execution abort (1a.8)

[chienchunhung]

@coderabbitai summary

Description

Implements TRTLLM-12561 / Wide-EP fault-tolerance work item 1a.8.

• Adds a per-workspace, per-EP-rank execution-epoch control backed by mapped pinned host memory.
• Lets dispatch and combine kernels cooperatively observe host-requested aborts and bounded wait timeouts, report a sticky host-visible status, and return without a device trap.
• Wires the control through both MoeAlltoAll and NVLinkOneSided, with fail-closed validation for ownership, CUDA device, rank, lifetime, and stream use.
• Keeps execution invalidation separate from committed EP membership and rank-mask updates; admission, quiescence, reset, and any membership change remain coordinator-owned.
• Preserves the legacy native overloads for ABI compatibility while making missing control state fail closed before collective state is mutated.

Test Coverage

• Added native dispatch/combine coverage for host-requested aborts, native timeouts, stale epochs, queued dispatch-to-combine propagation, host-visible status, recovery, and a healthy collective after reset.
• Added Python lifecycle and validation coverage, including shared-wrapper reset, missing controls, workspace/rank/device ownership, and both Python frontends.
• Registered the native unit-test selectors in the GB200 test list.
• Passed the repository pre-commit hooks for all changed files, plus Python byte-compilation, YAML parsing, git diff --check, and clang-format --dry-run --Werror.
• Native CUDA build/GPU execution was not available locally and is delegated to CI.
• The full-tree validate-test-lists hook cannot complete in this sparse checkout because 1,371 unrelated test sources are absent; the newly added source and selectors are present and were not among its reported missing paths.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57273 [ run ] triggered by Bot. Commit: ff2c92c Link to invocation

[tensorrt-cicd]

PR_Github #57273 [ run ] completed with state FAILURE. Commit: ff2c92c
/LLM/main/L0_MergeRequest_PR pipeline #46034 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57278 [ run ] triggered by Bot. Commit: ff2c92c Link to invocation

[tensorrt-cicd]

PR_Github #57278 [ run ] completed with state SUCCESS. Commit: ff2c92c
/LLM/main/L0_MergeRequest_PR pipeline #46040 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "A10-PackageSanityCheck-PY310-UB2204, A100X-PackageSanityCheck-PY312-UB2404, GB200-4_GPUs-PyTorch-2, GB200-4_GPUs-PyTorch-5"

[tensorrt-cicd]

PR_Github #57570 [ run ] triggered by Bot. Commit: 6f46379 Link to invocation

[tensorrt-cicd]

PR_Github #57570 [ run ] completed with state SUCCESS. Commit: 6f46379
/LLM/main/L0_MergeRequest_PR pipeline #46298 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #57872 [ run ] triggered by Bot. Commit: a9c950c Link to invocation

[tensorrt-cicd]

PR_Github #57872 [ run ] completed with state FAILURE. Commit: a9c950c
/LLM/main/L0_MergeRequest_PR pipeline #46568 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation