[TRTLLM-14810][chore] Catch-up merge: main into feat/kimi_k3

[brnguyen2]

@coderabbitai summary

Description

Catch-up merge of main into feat/kimi_k3 (TRTLLM-14810): merges main @ e34d3d4 (362 commits since the last sync point b602fa6) into the feature branch.

Draft status / how to use this branch: conflict resolution is complete and reviewed; build + unit-test validation has passed (details below), and the wider integration/accuracy qualification is still in progress. Teams blocked on the merge can base work on this branch now, accepting that qualification may still produce small fixups on it.

Note: this branch includes the commits of #17088 (TRTLLM-14703), which was open at merge time. #17088 should merge before or together with this PR.

Conflict resolution summary

13 files had textual conflicts. The notable resolutions:

• pyexecutor/mamba_cache_manager.py, pyexecutor/_util.py, pyexecutor/config_utils.py: Kimi K3 stays on the V1 compatibility cache managers (CppMambaHybridCacheManager / MixedMambaHybridCacheManager). Explicitly requesting use_kv_cache_manager_v2=True for K3 now raises with a pointer to TRTLLM-14769 (the V2 port, tracked separately); "auto" resolves to the V1 managers. The K3 branch of _create_kv_cache_manager was adapted to main's reworked MambaKVCacheParams / _get_mamba_cache_layer_masks API.
• speculative/sa_worker.py resolved to main's version (main's [None][fix] SA spec dec: promote accepted hybrid recurrent states in-worker #16759 upstreamed the same change; only the forward → _forward_impl interface rename remained, from [https://nvbugs/6442074][fix] Make one-model spec-dec attn-metadata save/restore exception-safe #16382).
• evaluate/lm_eval.py + its test resolved to main's version (main's [None][feat] Log running metric estimates during long lm-eval runs #16752 is the reviewed successor of the branch-side commit). Acceptance-rate (AR) logging is deferred as documented in that PR; AL logging is unchanged.
• fused_moe/fused_moe_trtllm_gen.py, create_moe.py, moe_scheduler.py: re-applied the branch's K3 additions on top of main's legacy-MoE-path removal ([TRTLLM-14609][chore] Remove legacy MoE path in TRTLLMGenFusedMoE #16862) and ENABLE_CONFIGURABLE_MOE escape-hatch removal ([TRTLLM-14609][chore] Remove ENABLE_CONFIGURABLE_MOE escape hatch and remaining MoE legacy relics #17009). The branch-side legacy forward_impl copy (dead code after [TRTLLM-14609][chore] Remove legacy MoE path in TRTLLMGenFusedMoE #16862) is dropped.
• attention_backend/fmha/flashinfer_trtllm_gen.py: kept the branch's cute-dsl MLA decode path; adopted main's flashinfer 0.6.15 multi-CTAS KV-counter-buffer mechanism on the trtllm-gen path.
• models/modeling_speculative.py: trivial (both sides landed the same optional hidden_size/vocab_size change, [None][fix] SpecDecOneEngineForCausalLM: accept optional hidden_size/vocab_size for composite configs #16762). The branch's DFlash extensions coexist with main's new modeling_dspark.py; deduplication is deferred to the spec-dec upstreaming PR.

All ~39 auto-merged files touched by both sides were hand-reviewed: branch-side deltas survived intact; requirements.txt and sa_worker.py intentionally resolve to main.

Test Coverage

• Build of the merged branch (RelWithDebInfo): passing. The one semantic collision the build caught — the MNNVL allreduce MoE-scale additions vs main's nvinfer1::DataType → tensorrt_llm::DataType migration — is fixed in a dedicated commit on this branch.
• KDA / SA / MoE unit suites on GB300: 115 passed, 3 skipped, 0 failed (tests/unittest/_torch/modeling/test_kda_mtp_decode_cute_parity.py, test_kimi_kda_fused_verify_parity.py, test_kimi_kda_verify_parity.py, test_kimi_kda_fp8_packed_prefill.py, tests/unittest/_torch/modules/kimi_kda/, tests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.py, tests/torch/speculative/test_suffix_automaton.py).
• In progress (will be reported on this PR before undrafting): 4-GPU SA-vs-baseline logits-parity integration run, 16-GPU smoke with and without KV block reuse, GSM8K accuracy sweep (baseline / reuse / SA), serving performance sweep vs a matched-commit baseline, and disaggregated-serving suites.

PR Checklist

• PR title and description follow the repo conventions
• Test coverage listed above

[brnguyen2]

/bot run

[tensorrt-cicd]

PR_Github #63130 [ run ] triggered by Bot. Commit: 78ae85b Link to invocation

[brnguyen2]

/bot run

[tensorrt-cicd]

PR_Github #63138 [ run ] triggered by Bot. Commit: ff30e9f Link to invocation

[tensorrt-cicd]

PR_Github/17143-78ae85b #63130 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

[tensorrt-cicd]

PR_Github #63138 [ run ] completed with state FAILURE. Commit: ff30e9f
/LLM/main/L0_MergeRequest_PR pipeline #51226 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[brnguyen2]

/bot run

[tensorrt-cicd]

PR_Github #63192 [ run ] triggered by Bot. Commit: 78e0c47 Link to invocation

[brnguyen2]

Validation summary

Accuracy — GSM8K with the three serving recipes (baseline decode, KV-block-reuse, suffix-automaton speculative decoding): 96.74 / 96.89 / 96.66 vs a 97.01 pre-merge reference (all within run-to-run noise). The reuse recipe requires the periodic_snapshot_interval setting restored in this PR (the merge renamed the option and its default changed to disabled).

Speculative-decoding parity — SA-vs-baseline logits-parity integration run on a truncated model: post-merge statistics are identical to the pre-merge branch tip (52/52 prompts diverge only at benign near-ties, 11 non-tie divergences, zero drift; same numbers on both trees).

Serving performance — 17-point 8K/1K benchmark sweep across three recipes, merge vs pre-merge branch tip, single run per point plus repeat runs on outliers:

• Low/mid concurrency: parity within ±2%.
• Attention-DP EP16 at extreme saturation (concurrency 1024): reproducible ~15% output-throughput deficit vs the pre-merge tip (and −4–5% at concurrency 128–256). Tracked in TRTLLM-14904; low/mid-concurrency operating points are unaffected.

Disaggregated smoke — context/generation split with MTP on the V2 transceiver path: server healthy, benchmark completes.

Known issue (introduced upstream, not K3-specific code) — speculative decoding with a self-spawned MPI session and fraction-based KV-cache sizing hangs during the KV-cache-size-estimation phase (the estimation executor's warmup never completes; the pre-merge tip passes the identical run). This branch carries a temporary workaround that skips the estimation phase whenever a speculative config is set (cache sized analytically instead; non-speculative runs keep normal estimation) — to be removed when the underlying hang is fixed. Production launch paths (trtllm-llmapi-launch, trtllm-serve) are unaffected. Tracked in TRTLLM-14903.

[brnguyen2]

Closing this PR without merging via the web UI: the repository is configured for squash merges, and squashing a true merge commit would linearize main's history into feat/kimi_k3 and break every future main → feat/kimi_k3 sync. The validated branch is instead direct-pushed (fast-forward) to feat/kimi_k3, preserving the merge topology. Final content = the merge plus one squashed follow-up commit (post-merge formatting, the reuse snapshot-interval restore, and the TRTLLM-14903 workaround documented above).

Rollback reference: the feat/kimi_k3 tip prior to this merge is ec52c6418b801b2c444f8ee2bedc35b914351725. Anyone needing the pre-merge behavior (e.g. to A/B the TRTLLM-14904 performance regression or bisect) should use that commit; it remains an ancestor of the pushed branch, so the push is a pure fast-forward with no history rewritten.

[tensorrt-cicd]

PR_Github #63192 [ run ] completed with state FAILURE. Commit: 78e0c47
/LLM/main/L0_MergeRequest_PR pipeline #51276 completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation