Skip to content

(DO NOT SUBMIT) WideEP FT MVP prorotype - #14198

Draft
chienchunhung wants to merge 5 commits into
NVIDIA:mainfrom
chienchunhung:WideEP-FT/mvp-prototype
Draft

(DO NOT SUBMIT) WideEP FT MVP prorotype#14198
chienchunhung wants to merge 5 commits into
NVIDIA:mainfrom
chienchunhung:WideEP-FT/mvp-prototype

Conversation

@chienchunhung

@chienchunhung chienchunhung commented May 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Status: PAUSED (2026-05-19) — Draft, DO NOT SUBMIT

The prototype's primary mandate is empirically discharged. Six findings (F1–F5)
plus two open questions (OQ2, OQ4) closed; full write-up in the design doc:
docs/design/wide-ep-fault-tolerance/mvp-prototype-findings.md.

Headline result: single-rank kill → first new request completed at N-1 in 5.11s
(well under the 10s budget), identical at --np 4 and --np 8. Detection is
the entire budget; recovery ≈ watchdog_timeout + 100ms (linear).

Why this PR is not for merge

This branch carries throwaway scaffolding under prototypes/wide_ep_ft_mvp/
that is explicitly designed to be discarded once the production PRs land
(see mvp-prototype-plan.md §9). It also carries private cherry-picks of:

When either parent PR lands on main, a rebase here will drop the cherry-pick
as already-applied with no manual intervention.

Resumption triggers (4 remaining items)

Trigger Unblocks
PR #13404 (1a.2 kernel mask) lands Seam-stressing kill points + OQ1 (NCCL ordering)
PR 1a.4 lands Reproduce F3/F4/F5 under real MNNVL fabric memory
PR 1c.3 lands Swap stubs/mpi_ft_subcomm.py + tighten F2 against world_is_poisoned()
NVL72 access False-positive floor + 72-rank scale validation (Audit 1b)

See the "When to resume" section of mvp-prototype-findings.md for full
mechanical steps.

What this PR demonstrates (already useful for review)

  • The prototypes/wide_ep_ft_mvp/ directory structure is the reference
    implementation for how stub components compose.
  • Six per-event timeline JSONs in prototypes/wide_ep_ft_mvp/results/ are
    the regression baseline for the eventual PR 1d.4 fault-injection harness.
  • The four design findings (F1-F4) route to specific production PRs and
    should inform their reviews even before this branch is touched again.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@chienchunhung
[None][feat] WideEP FT: add EPGroupHealth thread-safe rank mask
5b42b89 
EPGroupHealth is the process-local, thread-safe primitive that tracks
which ranks in an EP group are alive vs. failed. It is the foundational
data structure for WideEP fault tolerance (PR 1a.1 of the implementation
plan), shared across the AlltoAll communication backends, the host-side
watchdog, the MoE load balancer, and PyExecutor health reporting.

Internally backed by an arbitrary-precision Python int bitmask; the
uint64[N] format expected by the NVLink AlltoAll kernels is exposed via
get_mask_words() (default 2 words covers the NVL72 72-rank case).
mark_failed / mark_active are idempotent and bump a monotonic generation
counter only on effective state change, so consumers can compare a
cached generation to detect mask changes without diffing the full mask.

Tests cover the single-threaded API surface plus three concurrent race
scenarios (distinct-rank fan-out across 128 threads, same-rank
idempotency under contention, mixed fail/reactivate pairs).

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Made-with: Cursor
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Made-with: Cursor
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Made-with: Cursor
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
[None][feat] WideEP FT: add MPI signal handler replacement (1d.0)
588472d 
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
[TRTLLM-12728][feat] WideEP FT: scaffold MVP end-to-end prototype
f2a5912 
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
chienchunhung added a commit to chienchunhung/TensorRT-LLM that referenced this pull request May 16, 2026
@chienchunhung
[None][docs] WideEP FT: link draft PR NVIDIA#14198 for the MVP protot…
14543bc 
…ype scaffold

Adds PR NVIDIA#14198 (DO NOT SUBMIT — preview only) to the in-flight PRs table in
the README and updates mvp-prototype-plan.md to reflect that the prototype
scaffolding is now shipped on branch WideEP-FT/mvp-prototype:

  * README.md In-flight PRs table: new row for NVIDIA#14198 (status flagged as
    "Draft (DO NOT SUBMIT — preview only)" to distinguish from the MVP
    component PRs).
  * README.md prototype paragraph: links NVIDIA#14198 alongside the cherry-picked
    NVIDIA#13302 and NVIDIA#14160; notes the discardable prototypes/wide_ep_ft_mvp/
    directory.
  * mvp-prototype-plan.md Status header: bumped to reflect scaffolding-shipped
    state.
  * mvp-prototype-plan.md §6 sequencing: new "Current status (2026-05-15)"
    paragraph documenting that both 1a.1 (NVIDIA#13302) and 1d.0 (NVIDIA#14160) are
    private cherry-picks on the prototype branch pending their merges to
    main, and that the kernel-side 1a.2 + 1a.3 work is deferred per
    prototypes/wide_ep_ft_mvp/kernel/README.md.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
[None][feat] WideEP FT MVP prototype: zero-collective shm flags + ate…
f6469d3 
…xit _exit (F1+F2)

See docs-and-plans:docs/design/wide-ep-fault-tolerance/mvp-prototype-findings.md
for the full rationale. Two prototype-level fixes:

  * stubs/shm_completion_flags.py (new): POSIX shared-mem completion-flag
    table to replace the initial MPI allgather-based flag view, which
    would block on the dead peer and prevent the watchdog from firing.
    (F1)
  * scripts/kill_and_survive_worker.py: importlib shim for EPGroupHealth
    (sidesteps the broken tensorrt_llm package init in this dev env);
    os._exit(0) after main loop to bypass mpi4py atexit's MPI_Finalize
    hang on poisoned-world shutdown (F2).
  * scripts/kill_and_survive_driver.py: --kill-at-iteration N flag for
    deterministic kill points (preferred over heartbeat-based); pass
    --run-id to worker for shm-file namespacing; cleanup shm dir
    after run.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
chienchunhung added a commit to chienchunhung/TensorRT-LLM that referenced this pull request May 19, 2026
@chienchunhung
[None][docs] WideEP FT: add MVP prototype findings doc (F1 + F2)
d28be2d 
New file mvp-prototype-findings.md collects the seam-contract issues and
integration-risk discoveries surfaced by running the throwaway scaffolding
at prototypes/wide_ep_ft_mvp/ on the WideEP-FT/mvp-prototype branch
(preview draft PR NVIDIA#14198). Each finding routes to the specific production
PR that should incorporate the fix.

Two findings so far:

  F1. Watchdog's completion-flag view must NOT require peer participation.
      Surfaced during scaffolding review when the initial implementation
      used MPI.COMM_WORLD.allgather as the completion-flag view -- which
      would block on the dead peer and prevent the watchdog from firing.
      Production PR 1a.4 must reject any design that requires peer
      participation; the MNNVL fabric-memory read pattern is the contract,
      not an optimization. Prototype substitutes POSIX shared memory.

  F2. Survivors hang in MPI_Finalize after a peer SIGKILL. Surfaced
      during first Level A smoke run on 8x B300 node. Even with 1d.0,
      --mca orte_enable_recovery 1, and the full FT cascade working,
      survivors silently hang on shutdown because MPI_Finalize is a
      collective and the dead victim can't participate. This is
      audit-1a Day 2 F4 manifesting at process-shutdown time rather
      than at the next in-loop collective. Production fix belongs in
      PR 1c.4 (model engine health-check hook), consulting a new
      MpiFtSubcomm.world_is_poisoned() check from PR 1c.3. Explicitly
      out of scope for 1d.0.

mvp-prototype-plan.md status header + exit criteria 7.4 updated to point
at the new findings doc; README.md prototype paragraph likewise linked.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
[None][feat] WideEP FT MVP prototype: latency sweep + parallel-detect…
c98b32a 
…ion fix (F3+F4+F5)

Three new findings empirically resolved by running the prototype end-to-end
on an 8x B300 node; full write-up in docs-and-plans:
docs/design/wide-ep-fault-tolerance/mvp-prototype-findings.md.

  F3. Driver fix: detection is parallel (every survivor's local watchdog
      fires within ~ms), not serial through the MPI broadcast. Driver's
      t_mark_failed_propagated measurement was counting only ranks that
      received a broadcast; updated to count "ranks whose local mark_failed
      succeeded via either watchdog or broadcast" so parallel-detection
      runs report propagation correctly. This reframes broadcast as
      consensus backup, not primary spreading.

  F4. Worker + driver: --watchdog-timeout-sec and --watchdog-poll-interval-sec
      flags plumbed through for the detection-latency sweep. Sweep at
      1/2/5/10s confirms recovery = timeout + ~100ms (linear). Production
      default of 5s fits the 10s budget with ~50% headroom; 10s blows the
      budget on detection alone. F4 closes OQ4 from the prototype plan.

  F5. Validated empirically: recovery time is identical at --np 4 and --np 8
      (7.168s in both cases, all six measured events within 1ms). Validates
      the §5 "detection-dominated and scale-independent" claim within the
      prototype's scope.

Per-event timeline JSONs committed under prototypes/wide_ep_ft_mvp/results/
as the regression baseline for PR 1d.4 per mvp-prototype-plan.md §4.4. Six
JSONs total (~120KB): one np4/np8 baseline + 4-point latency sweep.

Pending items (deferred): OQ1 (NCCL ordering — requires torch.distributed
worker, significant new work), seam-stressing kill points (blocked on
1a.2/1a.3), false-positive floor characterization (needs NVL72), multi-
failure ordering (out of MVP scope).

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung chienchunhung mentioned this pull request Jun 30, 2026
Draft
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@chienchunhung chienchunhung

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chienchunhung