[None][fix] Autotuner deadlock fixes (split from #14053)

[lancelly]

Summary

Split out the autotuner deadlock fixes from #14053 as an independent, model-agnostic distributed-correctness PR. These two changes were always-on (gated only by tp_size > 1) in #14053 and have no dependency on the DSv4 mem-opts in that PR.

Changes

1. Pre/post tp_barrier around _run_autotuner_warmup (pyexecutor/model_engine.py)

Symptom: when one rank's autotuner cache is already populated and _run_autotuner_warmup returns near-instantly (e.g. "Cache size after warmup is 0"), that rank races ahead into the next collective-issuing _general_warmup step while a peer is still inside its own autotuner. The two collectives then mismatch, and the slower rank's autotuner spins forever waiting for peers that have moved on — producing a silent hang with no error message.

Fix: bracket _run_autotuner_warmup with TP-group barriers so every rank enters and exits the autotuner together. Barriers only fire when tp_size > 1 (no overhead on single-GPU). Helix CP is excluded by the existing outer has_cp_helix() check.

2. Collective vote on cache hit in AutoTuner.choose_one (autotuner.py)

Symptom: with ADP + MoE EP, different ranks see different per-expert routing shapes → is_cache_hit (profiling_cache.search_cache) diverges across ranks. The slow path below ends in _maybe_sync_cache_data, which issues a TP/CP collective (allgather or broadcast). If some ranks early-return at the cache-hit shortcut while others enter the collective, the collective deadlocks.

Fix: insert an unconditional tp_cp_allgather(is_cache_hit) collective vote so every rank participates in the allgather regardless of local cache state, and only short-circuit when all ranks hit. The allgather must be unconditional (not gated on local is_cache_hit) — otherwise hit ranks call the allgather while miss ranks skip it, recreating the deadlock at the vote site.

Validation

Validated on DSv4-Pro on GB300 (TP=4 EP=4, ADP on, gpu_frac=0.5) — same workload as #14053:

• 5-job parallel pass rate: 4/5 (was ~0% before fixes, ~60% with only expandable_segments)
• bench throughput (per-job, when not hung): 50,427–50,681 tok/s
• gsm8k strict-match (1319 samples): 96.02–96.47%

The remaining 1/5 failure is a different rank-divergent issue inside _profile_runners (PARALLEL strategy distributes tactics across ranks; MoE forward calls on different tactics can desync on internal all-to-all collectives). Not addressed by this patch — tracked as a follow-up.

Scope

Both changes are always-on (gated only by tp_size > 1 or _is_distributed()). They are NOT gated on TRTLLM_DSV4_MEM_OPTS. Behavior for tp_size == 1 is unchanged.

The unconditional tp_cp_allgather adds one extra collective per AutoTuner.choose_one invocation in tuning mode — autotuner warmup only, not steady-state inference, so the overhead ceiling is hundreds-of-tactics × microseconds, paid once per engine init.

Why split this out

In #14053 these autotuner fixes were bundled with three DSv4-specific memory optimizations gated on a new env switch. The autotuner deadlock fixes are correctness bugs that benefit any distributed PyTorch backend run, so they can land independently and earlier than the env-gated mem-opts. This also reduces the review surface for both PRs.

## Related

- Parent PR: #14053 (DSv4 mem-opts master switch — env-gated memory optimizations remain there).
- Unrelated but adjacent: NVIDIA/TensorRT-LLM#14126 ("Fail loudly on asymmetric warmup batch under attention-DP"), which already merged into `feat/deepseek_v4` and covers a separate rank-divergent failure mode in `_create_warmup_request`.