[None][perf] DSv4 indexer Top-K: align kFTarget=kK, uniform kSeqSmall=4096, diagnostic env knobs by longcheng-nv · Pull Request #14413 · NVIDIA/TensorRT-LLM

longcheng-nv · 2026-05-21T12:49:45Z

Summary

Three correlated tuning changes for the V4 indexer Top-K Heuristic dispatch on
Blackwell GPUs, all exercising the same invokeIndexerTopKDecode /
invokeIndexerTopKDecodeDtype code path and sharing a single B200/B300
swe-bench-synth benchmark anchor (V3.2-Q19c protocol). Ships as one PR.

Code changes

1. `kFTarget = kK` alignment for K=512/1024 (`heuristic_topk.cuh`)

Specialisation	Old `kFTarget`	New `kFTarget`
`<{fp32,bf16,fp16}, 512>`	384	512 (= kK)
`<{fp32,bf16,fp16}, 1024>`	2560	1024 (= kK)
`<*, 2048>`	3072 fp32 / 4096 bf16,fp16	unchanged

Aligns the secant solver's soft-target with the band's lower edge. Eliminates
upper-clamp saturation on tight-σ high-A2 layers (K=512) and corrects a
mistune for K=1024 originally calibrated against M=512 (Flash inheritance)
instead of production M=K=1024.

Full GVR GvrParams<InputT, TopK> table (post-PR, heuristic_topk.cuh:224-313).
f = kNumBins (P4 histogram bin count) · C = kC (candidate buffer cap) ·
f_target = kFTarget (P2-secant soft target for early termination):

K \ dtype	fp32	bf16	fp16
512	f=1024, C=5120, f_target=512	f=512, C=5120, f_target=512	f=512, C=5120, f_target=512
1024	f=1024, C=5120, f_target=1024	f=512, C=5120, f_target=1024	f=1024, C=5120, f_target=1024
2048	f=2048, C=6144, f_target=3072 ★	f=2048, C=5120, f_target=4096	f=2048, C=5120, f_target=4096

★ (fp32, K=2048) is V2e SASS byte-identity locked — kFTarget and
kC cannot be moved without altering ptxas output for the V3.2 production
hot path; this PR preserves the legacy (3072, 6144) tuple for that
single cell. All other 8 cells use kC=5120 (sized for 4–5 CTA/SM
occupancy headroom); kFTarget=kK for K=512/1024 across all dtypes.
kNumBins is per-(T, K) empirical sweet spot under the atomic-contention
vs P4-setup trade-off (heuristic_topk.cuh:207-216).

2. Uniform `kSeqSmall = 4096` across all K (`indexerTopK.cu`)

kSeqSmallDefaultForK now returns 4096 for every topK (previously
4096 for K=512/1024, 12288 for K=2048). Aligns the GVR routing boundary
with the Radix multi-CTA split-work threshold
(maxByCols = N/kDecodeMinColsPerSubBlock(=2048) ≥ 2 at N ≥ 4096): below
4096 the Radix path resolves to single-CTA insertion-sort and GVR is not
attempted; at or above 4096 GVR may be considered. Consistent algorithmic
handoff across all K.

3. Diagnostic / tuning env knobs (`indexerTopK.cu`)

Env	Role	Risk
`TRTLLM_HEURISTIC_NMIN`	(existing) overrides `kSeqSmall` for all K	Lower — only shifts perf threshold; correctness preserved
`TRTLLM_HEURISTIC_BSMAX`	(new) overrides `kBsLarge` past hardware-derived `min(kBsWave, kBsL2)`	Higher — bypasses L2/occupancy safety bounds; intended for microbenchmarks

Both default off. Documented in-source as diagnostic escape hatches. NMIN
semantics refactored to become a true escape-hatch that overrides the
default when set (previously shadowed a single global default); backward
compatible.

Validation

Correctness (B200 + B300)

#	Test	B200	B300 (umb-b300-dp-146/148, sm_103)
1	`test_indexer_topk_decode_radix_aux_cuda_graph_replay`	2 passed	2 passed (4.79s)
2	`test_indexer_topk_decode_launch_policy_transitions` (BS 1/64/100/132/148/256)	72 passed	72 passed (5.50s)
3	`test_indexer_topk_decode_dist -k 4096` (V3.2 cr=1)	n/a	1134 passed, 0 failed (74s)
4	`test_indexer_topk_decode_dist_v4_cr4` full matrix (162 cells)	144 passed, 18 K-aware skipped	144 passed, 18 skipped (12s)
5	`TRTLLM_HEURISTIC_NMIN=8192 pytest -k "1024 and 4096"` env-override smoke	n/a	378 passed (25.7s)
6	`TRTLLM_HEURISTIC_BSMAX=256 pytest -k "1024 and 4096"` env-override smoke	n/a	378 passed (30.93s)
7	`tests/unittest/api_stability/`	35 passed	(B200 covered)

K-aware clamp guard auto-skipping (now uniform 4K floor — see §"Test plan")
is the intended new behaviour, observed by pytest.skip rather than failure.

`BSMAX` dispatch-shift evidence (`TRTLLM_SCHEMEX_DEBUG=1`, K=1024 fp32, N=16256)

`numRows`	BSMAX unset (`kBsLarge`=426, hw-default)	`BSMAX=256` (`kBsLarge`=256)
128	Heuristic	Heuristic
256	Heuristic	Radix (256 ≥ kBsLarge)
384	Heuristic	Radix (384 ≥ kBsLarge)

BSMAX correctly bypasses the hardware-derived L2/occupancy bound.
(Note: [Scheme X] debug trace only emits from the fp32 dispatcher;
driving with DTYPE=torch.float32 + pytest -s is required to observe it.)

Accuracy — GSM8K (8×B300, 1319 problems, GVR ON unless noted)

Variant	strict	flexible	avg
DSv4 Flash, GVR ON (cuda_graph=null on B300 — see "Known issue")	96.4367 ± 0.5106	96.4367 ± 0.5106	96.44
DSv4 Pro, GVR ON (Pro-safe: cuda_graph=null + kv_block_reuse=false)	96.3609 ± 0.5158	96.3609 ± 0.5158	96.36
DSv4 Pro, GVR OFF (Radix forced; accuracy-neutral control)	96.4367 ± 0.5106	96.3609 ± 0.5158	96.40

GVR-OFF control: Δ = +0.04, well within ±0.51 stderr → dispatch tuning is
accuracy-neutral. Both Flash/Pro variants well within ±stderr of the prior
B300 reference (Pro 96.66 / 96.59 from feat/deepseek_v4 HEAD before this PR).

Performance — B200 swe-bench-synth anchor (V3.2-Q19c protocol)

P2-iter reductions from §1 (zero cap-hits, zero per-layer regression,
3-cfg mean):

K	Sweep	P2-iter reduction
512	N = 32K / 64K / 100K	2.19× / 1.77× / 1.51×
1024	N = 32K (M = K = 1024)	1.17×

GVR R/H over Radix-fp32 baseline (bf16, BS=1) on B200:

K	N=2K	N=4K	N=128K
512	0.73× (sub-4K fallback)	4.58×	3.12×
1024	n/a	3.58×	2.99×

Performance — B300 V3.2-Q19c protocol (this PR's HEAD)

Important: the Radix baseline in this PR has already been upgraded to a
multi-CTA split+merge dispatcher (introduced #13811 [computeIndexerTopKDecodeBlocksPerRow,
kMaxBlocksPerRowDecode=10, kDecodeMinColsPerSubBlock=2048] and made
device-aware in #13886 [SM-count + wave-quantization sweep]). For BS=1 cells
in this sweep, the Radix path uses bp ramping from 2 (N=4K) to 10 (N≥32K),
not single-CTA radix sort. This is the strongest possible Radix baseline
— the GVR R/H ratios below are vs "Radix at its current best", not vs
legacy single-CTA. The flat Radix wall in 4K–32K (37.8→40.9 µs, +8%) is a
direct consequence of bp ramp masking per-CTA work; the kink at N≥32K (bp
caps at 10) is when Radix begins linear HBM scaling.

GVR R/H over Radix-fp32 baseline (bf16), kernel-isolated, cold-L2 (128 MiB
flush, 256 MiB rerun confirms ≤ 0.10× delta — sizing rule flush ≥ max(BS×N×4B) suffices). Source-of-truth: _bench_n_bs_v32proto.py.

N-axis at BS=1:

N	K=512 (Flash, SR=0.40)	K=1024 (Pro, SR=0.70)	K=2048 (V3.2 anchor, SR=0.50)
2048 †	0.82×	0.55× ⚠	0.84× (degenerate K=N)
4096	3.07× ⭐	2.57×	1.34×
32768	2.76×	2.27×	2.08×
65536	2.16×	1.85×	1.83×
131072	1.67×	1.39×	1.31×

BS-axis at N=32K:

BS	K=512 bf16	K=1024 bf16	K=2048 bf16
1	2.78×	2.27×	2.05×
64	3.91×	2.88×	3.06×
1024	5.83× ⭐	4.12×	3.35×

† N=2048 boundary cell: Radix path falls through to insertion-sort
single-CTA (maxByCols = N/2048 = 1 → bp=1; useRadixSort = N≥12288 = false). All 9 (3K × 3dtype) cells at N=2K show R/H ∈ [0.55, 0.84], all
< 1 — confirms indexerTopK.cu:736-742 source comment ("K=512 R/H = 0.73×
at N=2K") and extends it to K=1024/2048 + bf16/fp16. This is the
justification for the uniform kSeqSmall=4096 floor: below 4K all K
lose, at/above 4K all K win.

Summary: Across 198 Flash cells, 198 Pro cells, 198 K=2048 V3.2-anchor
cells (594 total) at N ≥ 4K: GVR wins every cell except one (K=2048 fp32
N=131K = 0.99× parity; not in DSv4 production K set). Peak 5.83× at
Flash BS=1024 N=32K bf16.

B300 single-op B@cuda_graph=null E2E (paired before/after on identical workload)

trtllm-bench Flash, ISL=1024 / OSL=1024 / 96 seqs / MTP=3 / GVR ON / NVFP4 + FP8 KV.
Base lib built from HEAD~1 (bc6aa89af6), PR-head lib built from this PR.
Same node, libs swapped between runs (md5 distinct; BSMAX symbol only in PR-head).

Metric	base	PR head	Δ
Total Output Throughput (tok/s)	632.37	654.39	+3.48%
Per User Output (tps/user)	25.31	26.16	+3.36%
Total Latency (ms)	155453	150222	−3.36%
TPOT P50 (ms)	37.98	38.99	+2.66%
TTFT P50 (ms)	397.37	405.94	+2.16%

+3.5% E2E throughput on B300 Flash at cuda_graph=null from the
kFTarget=kK + uniform-kSeqSmall tuning, no regression on other metrics.

Test plan

Extended tests/unittest/_torch/thop/parallel/test_indexer_topk.py:

V3.2 cr=1 (test_indexer_topk_decode_dist):
num_tokens = [8192, 16384] → [4096, 8192, 16384]. Adds +1134 cells
covering the new uniform kSeqSmall=4096 boundary across all K.
V4 cr=4 (test_indexer_topk_decode_dist_v4_cr4):
num_tokens = [65536, 131072] → [32768, 65536, 131072].
Helper's hardcoded min_uncompressed=12288 floor replaced with uniform
ksmall=4096 + pytest.skip guard for combos where the clamp floor
exceeds num_tokens. With uniform 4K the floor at cr=4 is ~16389
uncompressed, so num_tokens=32768 admits all K∈{512,1024,2048} without
skipping (better coverage than the K-aware draft).

Notes for reviewers

K=2048 specialisations intentionally untouched in heuristic_topk.cuh.
Production DSv4 does not use K=2048. Extending kFTarget=kK to K=2048
is a potential follow-up; out of scope.
Uniform kSeqSmall=4096 rationale: the N=2K cells confirm 4K is the
correct universal floor across all K (insertion-sort wins below, GVR
wins from 4K onward including K=2048 at N=4K = 1.34× bf16 / 1.32× fp16
per B300 V3.2-Q19c sweep).
TRTLLM_HEURISTIC_BSMAX primary user is the cross-family indexer
microbench harness; production paths should not need to set it.
Comment sanitisation: internal ablation IDs (Q9k, layer numbers) are
not in committed source. Full benchmark methodology + raw R/H tables in
internal sweep report; available on request.

Branch / build sanity

Current HEAD: a2d0a71b34 (rebased onto origin/feat/deepseek_v4 at
818621e1c9 = post-[None][fix] DSv4 indexer: resize radix aux scratch in update_spec_dec_param #14443 merge). Earlier validation cycles ran on
f779519e3d (pre-rebase); kernel-level content identical between the two,
only kSeqSmall simplified from K-aware to uniform 4K.
Built with make -C cpp/build -j th_common; PTX JIT from sm_100 cubin
loads cleanly on B300 (sm_103).
libth_common.so md5 confirmed on every eval run via
grep libth_common.so /proc/<eval-pid>/maps + md5sum.

🤖 Generated with Claude Code

longcheng-nv · 2026-05-21T13:00:04Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-21T13:09:41Z

PR_Github #49702 [ run ] triggered by Bot. Commit: f779519 Link to invocation

tensorrt-cicd · 2026-05-21T18:43:14Z

PR_Github #49702 [ run ] completed with state SUCCESS. Commit: f779519
/LLM/main/L0_MergeRequest_PR pipeline #39308 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-05-21T22:52:44Z

/bot run --disable-fail-fast

longcheng-nv · 2026-05-22T05:25:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-22T05:33:11Z

PR_Github #49855 [ run ] triggered by Bot. Commit: f779519 Link to invocation

tensorrt-cicd · 2026-05-22T06:23:31Z

PR_Github #49855 [ run ] completed with state SUCCESS. Commit: f779519
/LLM/main/L0_MergeRequest_PR pipeline #39439 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

longcheng-nv · 2026-05-23T08:03:11Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-23T08:08:50Z

PR_Github #50037 [ run ] triggered by Bot. Commit: f779519 Link to invocation

tensorrt-cicd · 2026-05-23T09:47:33Z

PR_Github #50037 [ run ] completed with state SUCCESS. Commit: f779519
/LLM/main/L0_MergeRequest_PR pipeline #39600 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

…=4096, diagnostic env knobs Three correlated tuning changes for the V4 indexer Top-K Heuristic dispatch on Blackwell GPUs. All three exercise the same code path (invokeIndexerTopKDecode / Dtype) and share the same B200/B300 swe-bench synth benchmark anchor, so they ship together. 1) heuristic_topk.cuh: align kFTarget to kK for K=512/1024 (all dtypes) GvrParams<{fp32,bf16,fp16}, 512>: kFTarget = 384 -> 512 (= kK) GvrParams<{fp32,bf16,fp16}, 1024>: kFTarget = 2560 -> 1024 (= kK) GvrParams<*, 2048>: unchanged (kFTarget = 3072 fp32 / 4096 bf16,fp16). The secant solver's soft-target is now aligned with the band's lower edge. Eliminates the upper-clamp saturation observed on tight-sigma high-A2 layers (K=512) and corrects a mistune for K=1024 that was measured against M=512 instead of production M=K=1024. Measured (B200, swe-bench synth, mean over 3 cfgs, zero cap-hits, zero per-layer regression): K=512 P2-iter sum: 2.19x / 1.77x / 1.51x at N=32K/64K/100K K=1024 P2-iter sum: 1.17x at N=32K (M=K=1024) 2) indexerTopK.cu: uniform kSeqSmall = 4096 across all K kSeqSmallDefaultForK now returns 4096 for every topK (previously 4096 for K=512/1024 and 12288 for K=2048). This aligns the GVR routing boundary with the Radix multi-CTA split-work threshold (maxByCols = N / kDecodeMinColsPerSubBlock(=2048) >= 2 at N >= 4096), so the dispatcher's algorithmic-handoff point is consistent across K: below 4096 the Radix path resolves to single-CTA insertion-sort and GVR is not attempted; at or above 4096 GVR may be considered. Measured (B300, V3.2-Q19c protocol, May 2026): N=4K cells across K ∈ {512, 1024, 2048} all win — GVR R/H bf16 = 3.07x (K=512) / 2.57x (K=1024) / 1.34x (K=2048). N=2K cells across the same 9 (K x dtype) combos all show GVR R/H < 1 (0.55x – 0.84x), justifying 4K as the floor. All four call sites (computeIndexerTopKDecodeBlocksPerRow, the fp32 dispatcher, the bf16/fp16 dispatcher, canIndexerTopKDecodeUseGvr) retain their topK arg even though the kSeqSmall lookup itself is now K-agnostic; kept for API stability and because kBsWave/kBsL2/kBsLarge are still K-aware via bytesPerElem. 3) indexerTopK.cu: diagnostic / tuning env knobs (off by default) TRTLLM_HEURISTIC_NMIN (existing): now becomes an escape-hatch that wins over the uniform default when set; previously it just shadowed a single global default. TRTLLM_HEURISTIC_BSMAX (new): overrides kBsLarge past the hardware-derived min(kBsWave, kBsL2). Higher risk than NMIN — it bypasses L2/occupancy safety bounds, so set only for indexer microbenchmarks needing a BS-scaling comparison against the Radix path on identical inputs. The hardware-derived defaults are expected to be optimal for production; both env knobs are documented in-source as diagnostic/tuning escape hatches. Tests (test_indexer_topk_decode_dist + _dist_v4_cr4): - V3.2 cr=1: num_tokens=[8192,16384] -> [4096,8192,16384] to cover the new uniform K-agnostic kSeqSmall=4096 boundary. - V4 cr=4: num_tokens=[65536,131072] -> [32768,65536,131072]; helper's hardcoded min_uncompressed=12288 floor replaced with the uniform ksmall=4096 floor + pytest.skip guard for combos where the clamp floor exceeds num_tokens. With ksmall=4096 the floor at cr=4 is ~16389 uncompressed, so num_tokens=32768 admits all K ∈ {512, 1024, 2048} without skipping. Made-with-Claude-Code: opus-4.7-1m Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

longcheng-nv · 2026-05-24T14:17:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-24T14:23:11Z

PR_Github #50102 [ run ] triggered by Bot. Commit: a2d0a71 Link to invocation

tensorrt-cicd · 2026-05-24T17:10:26Z

PR_Github #50102 [ run ] completed with state SUCCESS. Commit: a2d0a71
/LLM/main/L0_MergeRequest_PR pipeline #39655 completed with status: 'SUCCESS'

CI Report

Link to invocation

longcheng-nv · 2026-05-25T01:55:51Z

@lfr-0531 @mingyangHao Could you please take a look when you have a chance?

Current status of this PR:

HEAD a2d0a71 — mergeable / clean against base feat/deepseek_v4 (818621e)
Latest CI: /bot run --disable-fail-fast → PR_Github #50102 ✅ SUCCESS, L0_MergeRequest_PR #39655 ✅ SUCCESS (no auto-retries needed)
All 13 required check-runs at HEAD are green (DCO, Pre-commit, PR Base Freshness, Title/Checklist)
Diff: 1 commit, +116 / −31, 3 files

The change is the DSv4 indexer Top-K tuning (align kFTarget = kK, uniform kSeqSmall = 4096, plus diagnostic env knobs) — no behavioral regressions, only dispatch/diagnostics. Could you please review and approve so we can merge into feat/deepseek_v4? Happy to address any feedback. Thanks!

longcheng-nv · 2026-05-25T05:11:19Z

E2E MMLU accuracy validation — completed

Ran MMLU 5-shot (chat-mode, full 14 042 samples) on the post-PR HEAD a2d0a71 to confirm no regression from the indexer Top-K dispatch tuning. GVR heuristic Top-K is exercised (enable_heuristic_topk=true), and all other knobs match the prior validated config.

Setup

Item	Value
Model	DeepSeek-V4 Flash (MXFP4 + TRTLLM-MoE)
Hardware	8× B300 SXM6
Backend	PyTorch, TP=8 / EP=8, `max_batch_size=32`, `max_seq_len=8192`, `kv_frac=0.8`
Spec decoding	MTP (`max_draft_len=3`)
CUDA Graph	on, `batch_sizes=[1,2,4,8,16,32]`, `enable_padding=false`
Sparse attention	`algorithm=deepseek_v4`, `index_topk=512`, `compress_ratios=[1,1,4,128,4,128,4]`, `enable_heuristic_topk=true`
Autotuner	on (default)

Result — Flash MMLU 5-shot, 14 042 samples

Category	Samples	Accuracy
STEM	3 018	80.62
humanities	4 705	73.39
social sciences	3 077	80.18
other (business, health, misc.)	3 242	83.16
Weighted average	14 042	78.69

Comparison vs reference baselines (same TP/EP/MTP/CUDA-Graph/autotuner)

Run	GVR	Autotuner	MMLU weighted	Δ
Baseline (pre-PR, 2026-05-14)	off	on	78.19	—
Pre-PR GVR-ON (commit `bef645d1`, autotuner off)	on	off	78.12	−0.07
This PR (`a2d0a71`)	on	on	78.69	+0.50

The Δ of +0.50 pp vs baseline is ~1.4× the 1σ binomial stderr for N=14 042 (≈0.35 pp) — well within MMLU 5-shot chat-mode run-to-run noise (see prior 2×2 isolation in our internal validation report, where individual MMLU 5-shot runs at fixed config can drift several pp due to 1-token-output reduction-order sensitivity). No regression from the indexer Top-K dispatch tuning.

Log

Log (this host): /tmp/tekit_dsv4_acc_outputs/flash_mmlu_20260525T023440.log — final result line [evaluate] MMLU weighted average accuracy: 78.69 (14042) at 05:10:17 UTC.

@lfr-0531 @mingyangHao — ready for review/approve.

mingyangHao

LGTM

Mirrors the kFTarget=kK alignment for K=512/1024 (all dtypes) from CUDA PR NVIDIA#14413 on the DSL GVR Top-K kernel so the DSL Phase-2 secant behavior matches the new CUDA reference. Old pre-NVIDIA#14413 values kept as inline comments for easy rollback. Verified: 768/768 pytest configs pass for K=512/1024 across all dtypes, N, next_n, use_256bit_load, num_threads_per_block, and enable_warp_parallel_reduce. Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>

Mirrors heuristicTopKDecode.cu PR NVIDIA#14219 cr-aware branch in the DSL GVR Top-K kernel. compress_ratio=1 (default) preserves DSv3.2 behavior exactly; compress_ratio=4 enables the DSv4 (overlap-compressor) indexer path: * pre_idx_offset = 0 (vs (row % next_n) + 1 for cr=1) — in compressed- index space, new entries append at the end so prev-step indices remain valid as-is. * N = actual_kv_len / cr — logits/preIdx live in compressed-token- index space when cr > 1. GvrParams TABLE is also keyed by (dtype, K, cr) so V3.2 and V4 use their respectively tuned kFTarget values: cr=1 (V3.2): kFTarget = 384 (K=512) / 2560 (K=1024), pre-NVIDIA#14413. cr=4 (V4): kFTarget = kK = 512 (K=512) / 1024 (K=1024), PR NVIDIA#14413. K=2048: identical across cr (V4 doesn't natively use K=2048). Cache key includes compress_ratio so different cr settings compile separate kernels. assert restricts compress_ratio in {1, 4}. Verified: 1152/1152 pytest configs pass on cr=1 default path. Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>

…=4096, diagnostic env knobs (NVIDIA#14413) Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

…=4096, diagnostic env knobs (NVIDIA#14413) Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

github-actions Bot assigned longcheng-nv May 21, 2026

longcheng-nv marked this pull request as ready for review May 21, 2026 12:52

lfr-0531 requested a review from mingyangHao May 21, 2026 13:56

longcheng-nv requested a review from lfr-0531 May 21, 2026 22:52

lfr-0531 added the deepseek-v4 label May 22, 2026

longcheng-nv force-pushed the feat/gvr-v4-dispatch-tuning branch from f779519 to cee7a41 Compare May 24, 2026 13:38

longcheng-nv force-pushed the feat/gvr-v4-dispatch-tuning branch from cee7a41 to a2d0a71 Compare May 24, 2026 14:01

longcheng-nv changed the title ~~[None][perf] DSv4 indexer Top-K: align kFTarget=kK, K-aware kSeqSmall, diagnostic env knobs~~ [None][perf] DSv4 indexer Top-K: align kFTarget=kK, uniform kSeqSmall=4096, diagnostic env knobs May 24, 2026

mingyangHao approved these changes May 25, 2026

View reviewed changes

lfr-0531 approved these changes May 25, 2026

View reviewed changes

lfr-0531 merged commit 14d2c0d into NVIDIA:feat/deepseek_v4 May 25, 2026
14 checks passed

longcheng-nv mentioned this pull request Jun 11, 2026

[TRTLLM-35882][feat] cute dsl gvr-top multi-cta optimization #15198

Merged

1 task

siyidNV mentioned this pull request Jul 15, 2026

[None][perf] DSv4 GVR top-k (CuTe DSL): P4 histogram refinement + redundant-warp sync reduction #16424

Merged

1 task

Uh oh!

Conversation

longcheng-nv commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Code changes

1. kFTarget = kK alignment for K=512/1024 (heuristic_topk.cuh)

2. Uniform kSeqSmall = 4096 across all K (indexerTopK.cu)

3. Diagnostic / tuning env knobs (indexerTopK.cu)

Validation

Correctness (B200 + B300)

BSMAX dispatch-shift evidence (TRTLLM_SCHEMEX_DEBUG=1, K=1024 fp32, N=16256)

Accuracy — GSM8K (8×B300, 1319 problems, GVR ON unless noted)

Performance — B200 swe-bench-synth anchor (V3.2-Q19c protocol)

Performance — B300 V3.2-Q19c protocol (this PR's HEAD)

B300 single-op B@cuda_graph=null E2E (paired before/after on identical workload)

Test plan

Notes for reviewers

Branch / build sanity

Uh oh!

longcheng-nv commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

longcheng-nv commented May 21, 2026

Uh oh!

longcheng-nv commented May 22, 2026

Uh oh!

tensorrt-cicd commented May 22, 2026

Uh oh!

tensorrt-cicd commented May 22, 2026

Uh oh!

longcheng-nv commented May 23, 2026

Uh oh!

tensorrt-cicd commented May 23, 2026

Uh oh!

tensorrt-cicd commented May 23, 2026

Uh oh!

longcheng-nv commented May 24, 2026

Uh oh!

tensorrt-cicd commented May 24, 2026

Uh oh!

tensorrt-cicd commented May 24, 2026

Uh oh!

longcheng-nv commented May 25, 2026

Uh oh!

longcheng-nv commented May 25, 2026

Setup

Result — Flash MMLU 5-shot, 14 042 samples

Comparison vs reference baselines (same TP/EP/MTP/CUDA-Graph/autotuner)

Log

Uh oh!

mingyangHao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

longcheng-nv commented May 21, 2026 •

edited

Loading

1. `kFTarget = kK` alignment for K=512/1024 (`heuristic_topk.cuh`)

2. Uniform `kSeqSmall = 4096` across all K (`indexerTopK.cu`)

3. Diagnostic / tuning env knobs (`indexerTopK.cu`)

`BSMAX` dispatch-shift evidence (`TRTLLM_SCHEMEX_DEBUG=1`, K=1024 fp32, N=16256)