Bump onnx from 1.12.0 to 1.13.0#1
Closed
dependabot[bot] wants to merge 1 commit into
Closed
Conversation
Bumps [onnx](https://github.com/onnx/onnx) from 1.12.0 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](onnx/onnx@v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
added
the
dependencies
Author
|
OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting If you change your mind, just re-open this PR and I'll resolve any conflicts on it. |
4 tasks
nv-guomingz
pushed a commit
that referenced
this pull request
Jan 24, 2025
* Add README * Add unified converter (#1) * init v3 lite feat * fix moe topk method * fix noaux_tc logic * fix deepseek v3 normal rope * refactor * wo conversion ok debugging build * add quantize for attn.dense * add unified converter support * testing unified converter * add convert checkpoint and update docs --------- Co-authored-by: Zeyu Wang <zeyuw@nvidia.com> * update README * add FP8 notes * Update run.py result * Update V3 README * Update usages of FP8 to BF16 instruction * fix model name mapping (#2) * Update HF ckpt BF16 conversion. * fix config of deepseek kv cache * Remove source code * Deepseek V3 FP8 Support --------- Co-authored-by: jershi425 <83951930+jershi425@users.noreply.github.com> Co-authored-by: Zeyu Wang <zeyuw@nvidia.com> Co-authored-by: Hanyue He <hanyueh@nvidia.com> Co-authored-by: root <root@h20-2.cm.cluster>
4 tasks
4 tasks
1 task
zhengd-nv
pushed a commit
to zhengd-nv/TensorRT-LLM
that referenced
this pull request
Nov 21, 2025
…ug fix (NVIDIA#1) Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * feat: ut for mooncake Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: add mooncake backend for cacheTransceiverTest Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: remove link Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * feat: add missing build Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * feat: add missing link Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: transferAgentTest Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: transferAgentTest Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: transferAgentTest Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: transferAgentTest Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: add status->wait Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * feat: add mooncake option for cacheTransceiver Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: rollbakc add status->wait Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * test: add debug msg Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: set batchId to invalid after xfer complete Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: set batchId to invalid after xfer complete Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: set batchId to invalid after xfer complete Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: set batchId to invalid after xfer complete Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * fix: add status->wait Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * refactor: simplify err msg Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> * refactor: isCompleted always return true after batchID freed Signed-off-by: wjueyao <wyao123@terpmail.umd.edu> --------- Signed-off-by: wjueyao <wyao123@terpmail.umd.edu>
nv-lschneider
added a commit
to nv-lschneider/TensorRT-LLM
that referenced
this pull request
Dec 1, 2025
Signed-off-by: Ludwig Schneider <lschneider@nvidia.com>
5 tasks
5 tasks
karljang
added a commit
to karljang/TensorRT-LLM
that referenced
this pull request
Feb 19, 2026
- Revert utils.py docstring changes (NVIDIA#1) - Remove disable_deep_gemm flag, use module-level quant_config=None (NVIDIA#2) - Replace Flux2SwiGLU with shared swiglu from _torch/modules (NVIDIA#3) - Unify Flux2PosEmbed into FluxPosEmbed with parameterized axes_dim (NVIDIA#4) - Replace Flux2FeedForward with shared GatedMLP (NVIDIA#6) - Revert pipeline.py post_load_weights; add create_weights() in FLUX transformer __init__ to match WAN's __post_init__ pattern (NVIDIA#7) - Refactor _get_quant_algo_for_layer to encapsulate hasattr logic (NVIDIA#9) - Remove partial docstrings that just repeat class names (NVIDIA#10) - Simplify bias flags to direct booleans (NVIDIA#11) - Unify apply_rotary_emb: delete FLUX copy, FluxPosEmbed outputs [1,S,1,D] matching WAN format so shared function unchanged (NVIDIA#12) - Remove dynamic=True from maybe_compile (NVIDIA#13) - Add @torch.compiler.disable on FluxPosEmbed.forward - Tighten FLUX.2 PSNR test threshold from 12 to 20 dB Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
1 task
3 tasks
3 tasks
1 task
1 task
chienchunhung
referenced
this pull request
in chienchunhung/TensorRT-LLM
Apr 16, 2026
The ~21s MPI worker spawn overhead (Python imports, CUDA context, NCCL setup) runs concurrently with the server's model download. In S1, the 63.5s download fully hides this cost; in S2/S3, it becomes visible. Key updates: - "Reading the Results Tables": show separate S1/S2 timeline diagrams explaining the concurrency effect - Part 2 table: add "Worker spawn overhead" row (hidden/~21s) - S2>S1 explanation: split into real I/O difference + hidden parallelism - Insight #1: incorporate hidden parallelism finding - Insight #2: decompose ~25s "executor overhead" into ~21s worker spawn + ~4s other; identify two irreducible floors (spawn + warmup) Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
chienchunhung
referenced
this pull request
in chienchunhung/TensorRT-LLM
Apr 16, 2026
The S2>S1 result reflects the specific test environment (fast internal HF mirror at ~2 GB/s, 8 concurrent NFS readers, worst-case cold NFS). Public HF downloads would likely reverse the ordering. Updated both the Part 2 explanation and Analysis insight #1 to frame this as an observation about the benchmark environment rather than a general claim about NFS vs CDN performance. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
chienchunhung
referenced
this pull request
in chienchunhung/TensorRT-LLM
Apr 16, 2026
…ta flow - S2 (NFS cold) is the realistic production cold-start scenario; S1's faster result is an artifact of a fast internal HF mirror - Added "Weight Loading Data Flow (TP=8)" section explaining the cooperative prefetch → full load → shard pattern and why every rank must load all safetensors files (HF shards by layer groups, TP shards by tensor slicing) - Highlighted the 8x redundant CPU memory inefficiency that MX addresses by streaming only the relevant shard to each rank - Updated Analysis insight #1 and MX+GMS impact projection to reflect production-centric framing Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
chienchunhung
referenced
this pull request
in chienchunhung/TensorRT-LLM
Apr 16, 2026
The download in S1 inherently warms the OS page cache (Linux retains written pages via write-back caching), so workers always hit warm page cache after a download — regardless of tmpfs vs local disk. This explains why S1 prefetch (3.5s) ≈ S3 prefetch (3.4s). S2 is slower because no prior operation warms the cache — workers read directly from cold NFS (65-99s). Rewrote "Why S2 > S1" explanation, Analysis insight #1, Storage Tier Matrix, and Part 1 description to center on this page-cache-warming mechanism. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
2ez4bz
pushed a commit
to 2ez4bz/TensorRT-LLM
that referenced
this pull request
Apr 16, 2026
Three issues fixed: 1. past_key_value (singular) renamed to past_key_values (plural) for LlamaAttention and GPT2Attention in transformers 5.x. 2. use_cache parameter removed from attention forward calls in transformers 5.x (cache is always updated in-place). 3. rope_theta/rope_scaling config attributes replaced with unified rope_parameters dict (same fix as test_gpt_attention.py). Without fix NVIDIA#1-2, the DynamicCache was never populated because the kwarg was silently ignored, causing an empty tuple from dynamic_cache_to_legacy and an IndexError at line 1026. Signed-off-by: Jonas Li <6110159+longlee0622@users.noreply.github.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 23, 2026
Moves 107 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml
to l0_gb200_multi_gpus.yml, and deletes 4 tests that already exist on GB200
(dedup). Pure test-list YAML changes — no Jenkins groovy, no new YAML files.
Existing GB200-4_GPUs-PyTorch-{1,2,Post-Merge-1} stages absorb the moved
tests automatically.
Going-forward capacity reclaimed on B200: ~2,122 GPU-hrs/wk
(raw 30-day average: ~2,125 GPU-hrs/wk).
Scope breakdown:
* pre-merge pytorch: 28 tests appended to l0_gb200_multi_gpus.yml
* post-merge pytorch: 79 tests appended to l0_gb200_multi_gpus.yml
* dedup deletes: 4 tests removed from l0_dgx_b200.yml only
Out of scope for this PR:
* 1-GPU tests (156 tests) - PR NVIDIA#2, needs GB200-PyTorch-1 stage
* 4-GPU autodeploy tests (11) - PR NVIDIA#2, needs GB200-4_GPUs-AutoDeploy-1 stage
* Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW
Verified with scripts/test_to_stage_mapping.py that moved samples route to
GB200-4_GPUs-PyTorch-{1,2} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-1
(post-merge), not DGX_B200-PyTorch-*.
Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 24, 2026
Moves 107 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml
to l0_gb200_multi_gpus.yml, and deletes 4 tests that already exist on GB200
(dedup). Pure test-list YAML changes — no Jenkins groovy, no new YAML files.
Existing GB200-4_GPUs-PyTorch-{1,2,Post-Merge-1} stages absorb the moved
tests automatically.
Going-forward capacity reclaimed on B200: ~2,122 GPU-hrs/wk
(raw 30-day average: ~2,125 GPU-hrs/wk).
Scope breakdown:
* pre-merge pytorch: 28 tests appended to l0_gb200_multi_gpus.yml
* post-merge pytorch: 79 tests appended to l0_gb200_multi_gpus.yml
* dedup deletes: 4 tests removed from l0_dgx_b200.yml only
Out of scope for this PR:
* 1-GPU tests (156 tests) - PR NVIDIA#2, needs GB200-PyTorch-1 stage
* 4-GPU autodeploy tests (11) - PR NVIDIA#2, needs GB200-4_GPUs-AutoDeploy-1 stage
* Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW
Verified with scripts/test_to_stage_mapping.py that moved samples route to
GB200-4_GPUs-PyTorch-{1,2} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-1
(post-merge), not DGX_B200-PyTorch-*.
Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 27, 2026
Moves 108 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml
to l0_gb200_multi_gpus.yml, and deletes 4 tests that already exist on GB200
(dedup). Pure test-list YAML changes — no Jenkins groovy, no new YAML files.
Existing GB200-4_GPUs-PyTorch-{1,2,Post-Merge-1} stages absorb the moved
tests automatically.
Going-forward capacity reclaimed on B200: ~2,200 GPU-hrs/wk on B200.
Scope breakdown:
* pre-merge pytorch: 29 tests appended to l0_gb200_multi_gpus.yml
* post-merge pytorch: 79 tests appended to l0_gb200_multi_gpus.yml
* dedup deletes: 4 tests removed from l0_dgx_b200.yml only
Out of scope for this PR:
* 1-GPU tests (156 tests) - PR NVIDIA#2, needs GB200-PyTorch-1 stage
* 4-GPU autodeploy tests (11) - PR NVIDIA#2, needs GB200-4_GPUs-AutoDeploy-1 stage
* Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW
Verified offline that moved tests route to GB200-4_GPUs-PyTorch-{1,2}
(pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-1 (post-merge), not
DGX_B200-PyTorch-*. Ground-truth verification via post-merge pipeline
diff (per Tyler's recommendation): compare tests in
L0_PostMerge/<latest_job> vs L0_MergeRequest/<this_pr_job> via
OpenSearch to confirm no coverage loss.
Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 27, 2026
… 1 PR #1) Moves 108 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml to l0_gb200_multi_gpus.yml, deletes 4 dedup tests, and reshards two existing GB200 stages to keep wall-clock under the 4h cluster ceiling. Test moves (test-list YAML edits only): * pre-merge pytorch: 29 tests appended to l0_gb200_multi_gpus.yml * post-merge pytorch: 79 tests appended to l0_gb200_multi_gpus.yml * dedup deletes: 4 tests removed from l0_dgx_b200.yml only Sharding adjustments (jenkins/L0_Test.groovy): * GB200-4_GPUs-PyTorch-{1,2} 2-shard → 4-shard split (add -3, -4) * GB200-4_GPUs-PyTorch-Post-Merge-1 1-shard → 3-shard split (add -2, -3) Mirrors the established DGX_B200-4_GPUs-PyTorch-{1,2,3} + DGX_B200-4_GPUs-PyTorch-Post-Merge-{1,2} pattern, scaled for our larger added test load. Without resharding, projected wall-clocks hit ~4.75h pre-merge / ~7h post-merge, both over the 4h cluster ceiling. After resharding, projected wall-clock per shard: ~2.9h pre-merge / ~3.0h post-merge — within budget with margin. Going-forward capacity reclaimed on B200: ~2,200 GPU-hrs/wk. Out of scope for this PR (Wave 1 PR NVIDIA#2 will handle): * 1-GPU tests (156 tests) - needs new GB200-PyTorch-{1,Post-Merge-1,AutoDeploy-1} stages * 4-GPU autodeploy tests (11) - needs new GB200-4_GPUs-AutoDeploy-1 stage * Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW Verified with scripts/test_to_stage_mapping.py that moved samples route to GB200-4_GPUs-PyTorch-{1,2,3,4} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-{1,2,3} (post-merge), not DGX_B200-PyTorch-*. Ground-truth verification recipe (per Tyler's guidance): compare tests that ran in L0_PostMerge/<latest_job> vs L0_MergeRequest/<this_pr_job> via OpenSearch — every test removed from B200 stages should now appear in GB200 stages, no coverage loss. Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 28, 2026
… 1 PR #1) Moves 108 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml to l0_gb200_multi_gpus.yml (skipping 9 that already exist there as duplicates), deletes 4 hard-coded dedup tests, and reshards the affected GB200 stages to keep wall-clock under the 4h cluster ceiling. Test moves and dedup deletes (test-list YAML edits only): * pre-merge pytorch: 29 tests appended to l0_gb200_multi_gpus.yml * post-merge pytorch: 70 tests appended to l0_gb200_multi_gpus.yml (9 entries already existed in target — dedup, not added) * dedup deletes: 4 TestDeepSeekV3Lite::test_bfloat16_4gpus_python_scheduler entries removed from l0_dgx_b200.yml only (already in l0_gb200_multi_gpus.yml) Sharding adjustments (jenkins/L0_Test.groovy): * GB200-4_GPUs-PyTorch-{1,2} 2-shard → 4-shard split (add -3, -4) * GB200-4_GPUs-PyTorch-Post-Merge-1 1-shard → 3-shard split (add -2, -3) Mirrors the established DGX_B200-4_GPUs-PyTorch-{1,2,3} + DGX_B200-4_GPUs-PyTorch-Post-Merge-{1,2} pattern, scaled for our larger added test load. Without resharding, projected wall-clocks hit ~4.75h pre-merge / ~7h post-merge, both over the 4h cluster ceiling. After resharding, projected wall-clock per shard: ~2.9h pre-merge / ~3.0h post-merge — within budget with margin. Going-forward capacity reclaimed on B200: ~2,200 GPU-hrs/wk (slightly more, since 9 of the moved tests were previously running on BOTH B200 and GB200 post-merge — this consolidation removes the B200 copy). Out of scope for this PR (Wave 1 PR NVIDIA#2 will handle): * 1-GPU tests (156 tests) - needs new GB200-PyTorch-{1,Post-Merge-1,AutoDeploy-1} stages * 4-GPU autodeploy tests (11) - needs new GB200-4_GPUs-AutoDeploy-1 stage * Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW Verified with scripts/test_to_stage_mapping.py that moved samples route to GB200-4_GPUs-PyTorch-{1,2,3,4} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-{1,2,3} (post-merge), not DGX_B200-PyTorch-*. Verified l0_gb200_multi_gpus.yml has zero duplicate test entries (trt-test-db rejects YAMLs with same test/conditions twice). Ground-truth verification recipe (per Tyler's guidance): compare tests that ran in L0_PostMerge/<latest_job> vs L0_MergeRequest/<this_pr_job> via OpenSearch — every test removed from B200 stages should now appear in GB200 stages, no coverage loss. Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 28, 2026
…1 PR #1) Moves 95 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml to l0_gb200_multi_gpus.yml, and reshards the affected GB200 stages to keep wall-clock under the 4h cluster ceiling. Test moves (test-list YAML edits only): * pre-merge pytorch: 25 tests appended to l0_gb200_multi_gpus.yml * post-merge pytorch: 70 tests appended to l0_gb200_multi_gpus.yml 13 tests intentionally kept on BOTH B200 and GB200 (conservative cross-platform coverage; pre-existing duplication, ~90 GPU-hrs/wk): * 4 pre-merge: TestDeepSeekV3Lite::test_bfloat16_4gpus_python_scheduler[*] * 9 post-merge: TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-FLASHINFER-False], TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-...-TRTLLM-False], 7 × TestDeepSeekV3Lite::test_nvfp4_4gpus[CUTLASS|TRTLLM-mtp_nextn=*-...] These tests were originally added to both YAMLs (commits NVIDIA#11884, NVIDIA#7073/NVIDIA#7074, NVIDIA#11844, NVIDIA#13366) and have no platform-specific decorators. We could dedup, but keeping the duplication is safer if there's any latent host-CPU or NVLink topology-specific behavior. Sharding adjustments (jenkins/L0_Test.groovy): * GB200-4_GPUs-PyTorch-{1,2} 2-shard → 4-shard split (add -3, -4) * GB200-4_GPUs-PyTorch-Post-Merge-1 1-shard → 3-shard split (add -2, -3) Mirrors DGX_B200-4_GPUs-PyTorch-{1,2,3} + DGX_B200-4_GPUs-PyTorch-Post-Merge-{1,2}. Without resharding, projected wall-clocks hit ~4.75h pre-merge / ~7h post-merge, both over the 4h cluster ceiling. After resharding, projected wall-clock per shard: ~2.9h pre-merge / ~3.0h post-merge — within budget with margin. Going-forward capacity reclaimed on B200: ~2,030 GPU-hrs/wk (from the 95 moves; the 13 kept-duplicate tests continue running on B200). Out of scope for this PR (Wave 1 PR NVIDIA#2 will handle): * 1-GPU tests (156 tests) - needs new GB200-PyTorch-{1,Post-Merge-1,AutoDeploy-1} stages * 4-GPU autodeploy tests (11) - needs new GB200-4_GPUs-AutoDeploy-1 stage * Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW Verified with scripts/test_to_stage_mapping.py that moved samples route to GB200-4_GPUs-PyTorch-{1,2,3,4} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-{1,2,3} (post-merge), not DGX_B200-PyTorch-*. Verified l0_gb200_multi_gpus.yml has zero duplicate test entries (trt-test-db rejects same-test-same-conditions duplicates). Ground-truth verification recipe (per Tyler's guidance): compare tests run in L0_PostMerge/<latest_job> vs L0_MergeRequest/<this_pr_job> via OpenSearch. Every B200-removed test should appear in GB200 stages; the 13 intentionally kept tests will appear on both. Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 28, 2026
…1 PR #1) Moves 95 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml to l0_gb200_multi_gpus.yml, and reshards the affected GB200 stages to keep wall-clock under the 4h cluster ceiling. Test moves (test-list YAML edits only): * pre-merge pytorch: 25 tests appended to l0_gb200_multi_gpus.yml * post-merge pytorch: 70 tests appended to l0_gb200_multi_gpus.yml 13 tests intentionally kept on BOTH B200 and GB200 (conservative cross-platform coverage; pre-existing duplication, ~90 GPU-hrs/wk): * 4 pre-merge: TestDeepSeekV3Lite::test_bfloat16_4gpus_python_scheduler[*] * 9 post-merge: TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-FLASHINFER-False], TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-...-TRTLLM-False], 7 × TestDeepSeekV3Lite::test_nvfp4_4gpus[CUTLASS|TRTLLM-mtp_nextn=*-...] These tests were originally added to both YAMLs (commits NVIDIA#11884, NVIDIA#7073/NVIDIA#7074, NVIDIA#11844, NVIDIA#13366) and have no platform-specific decorators. We could dedup, but keeping the duplication is safer if there's any latent host-CPU or NVLink topology-specific behavior. Sharding adjustments (jenkins/L0_Test.groovy): * GB200-4_GPUs-PyTorch-{1,2} 2-shard → 4-shard split (add -3, -4) * GB200-4_GPUs-PyTorch-Post-Merge-1 1-shard → 3-shard split (add -2, -3) Mirrors DGX_B200-4_GPUs-PyTorch-{1,2,3} + DGX_B200-4_GPUs-PyTorch-Post-Merge-{1,2}. Without resharding, projected wall-clocks hit ~4.75h pre-merge / ~7h post-merge, both over the 4h cluster ceiling. After resharding, projected wall-clock per shard: ~2.9h pre-merge / ~3.0h post-merge — within budget with margin. Going-forward capacity reclaimed on B200: ~2,030 GPU-hrs/wk (from the 95 moves; the 13 kept-duplicate tests continue running on B200). Out of scope for this PR (Wave 1 PR NVIDIA#2 will handle): * 1-GPU tests (156 tests) - needs new GB200-PyTorch-{1,Post-Merge-1,AutoDeploy-1} stages * 4-GPU autodeploy tests (11) - needs new GB200-4_GPUs-AutoDeploy-1 stage * Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW Verified with scripts/test_to_stage_mapping.py that moved samples route to GB200-4_GPUs-PyTorch-{1,2,3,4} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-{1,2,3} (post-merge), not DGX_B200-PyTorch-*. Verified l0_gb200_multi_gpus.yml has zero duplicate test entries (trt-test-db rejects same-test-same-conditions duplicates). Ground-truth verification recipe (per Tyler's guidance): compare tests run in L0_PostMerge/<latest_job> vs L0_MergeRequest/<this_pr_job> via OpenSearch. Every B200-removed test should appear in GB200 stages; the 13 intentionally kept tests will appear on both. Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
zheyuf
referenced
this pull request
in zheyuf/TensorRT-LLM
Apr 30, 2026
…1 PR #1) Moves 95 pre-merge + post-merge pytorch 4-GPU tests from l0_dgx_b200.yml to l0_gb200_multi_gpus.yml, and reshards the affected GB200 stages to keep wall-clock under the 4h cluster ceiling. Test moves (test-list YAML edits only): * pre-merge pytorch: 25 tests appended to l0_gb200_multi_gpus.yml * post-merge pytorch: 70 tests appended to l0_gb200_multi_gpus.yml 13 tests intentionally kept on BOTH B200 and GB200 (conservative cross-platform coverage; pre-existing duplication, ~90 GPU-hrs/wk): * 4 pre-merge: TestDeepSeekV3Lite::test_bfloat16_4gpus_python_scheduler[*] * 9 post-merge: TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-FLASHINFER-False], TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-...-TRTLLM-False], 7 × TestDeepSeekV3Lite::test_nvfp4_4gpus[CUTLASS|TRTLLM-mtp_nextn=*-...] These tests were originally added to both YAMLs (commits NVIDIA#11884, NVIDIA#7073/NVIDIA#7074, NVIDIA#11844, NVIDIA#13366) and have no platform-specific decorators. We could dedup, but keeping the duplication is safer if there's any latent host-CPU or NVLink topology-specific behavior. Sharding adjustments (jenkins/L0_Test.groovy): * GB200-4_GPUs-PyTorch-{1,2} 2-shard → 4-shard split (add -3, -4) * GB200-4_GPUs-PyTorch-Post-Merge-1 1-shard → 3-shard split (add -2, -3) Mirrors DGX_B200-4_GPUs-PyTorch-{1,2,3} + DGX_B200-4_GPUs-PyTorch-Post-Merge-{1,2}. Without resharding, projected wall-clocks hit ~4.75h pre-merge / ~7h post-merge, both over the 4h cluster ceiling. After resharding, projected wall-clock per shard: ~2.9h pre-merge / ~3.0h post-merge — within budget with margin. Going-forward capacity reclaimed on B200: ~2,030 GPU-hrs/wk (from the 95 moves; the 13 kept-duplicate tests continue running on B200). Out of scope for this PR (Wave 1 PR NVIDIA#2 will handle): * 1-GPU tests (156 tests) - needs new GB200-PyTorch-{1,Post-Merge-1,AutoDeploy-1} stages * 4-GPU autodeploy tests (11) - needs new GB200-4_GPUs-AutoDeploy-1 stage * Perf (41), legacy TRT/Triton (14), 8-GPU, Ray, fabric-dep, aarch64-LOW Verified with scripts/test_to_stage_mapping.py that moved samples route to GB200-4_GPUs-PyTorch-{1,2,3,4} (pre-merge) and GB200-4_GPUs-PyTorch-Post-Merge-{1,2,3} (post-merge), not DGX_B200-PyTorch-*. Verified l0_gb200_multi_gpus.yml has zero duplicate test entries (trt-test-db rejects same-test-same-conditions duplicates). Ground-truth verification recipe (per Tyler's guidance): compare tests run in L0_PostMerge/<latest_job> vs L0_MergeRequest/<this_pr_job> via OpenSearch. Every B200-removed test should appear in GB200 stages; the 13 intentionally kept tests will appear on both. Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
1 task
chenfeiz0326
pushed a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
May 6, 2026
Three issues fixed: 1. past_key_value (singular) renamed to past_key_values (plural) for LlamaAttention and GPT2Attention in transformers 5.x. 2. use_cache parameter removed from attention forward calls in transformers 5.x (cache is always updated in-place). 3. rope_theta/rope_scaling config attributes replaced with unified rope_parameters dict (same fix as test_gpt_attention.py). Without fix NVIDIA#1-2, the DynamicCache was never populated because the kwarg was silently ignored, causing an empty tuple from dynamic_cache_to_legacy and an IndexError at line 1026. Signed-off-by: Jonas Li <6110159+longlee0622@users.noreply.github.com>
2 tasks
luyiyun1021
added a commit
to luyiyun1021/TensorRT-LLM
that referenced
this pull request
May 12, 2026
…change) Addresses post-review feedback on the v2a Ulysses wrapper. Pure refactoring; no functional changes. 1. PE pad moved into prepare_text_cache (done once per generate() instead of per forward step). 2. _pad_pe takes seq_dim explicitly (was fragile auto-detect by shape match); raises on out-of-range seq_dim. 3. Forward audio pad guard reduced from 4 conjuncts to 2. 4. Removed defensive ``if model_config is not None else True`` fallback. 5. PE pad pre-assign + overwrite block removed from forward (subsumed by NVIDIA#1). 6. configure_audio_ulysses two branches flattened to a single formula. 7. v2a fallback restructured to flat if/elif/else (was nested). Plus a stale comment fix: ``Ulysses wrap (a2v in LTX-2)`` → ``(v2a in LTX-2)``. Diff: ~44 insertions, ~57 deletions in transformer_ltx2.py only. Signed-off-by: Yiyun Lu <55233584+luyiyun1021@users.noreply.github.com>
farazkh80
added a commit
to farazkh80/TensorRT-LLM
that referenced
this pull request
May 19, 2026
Eleven trtllm-bench / trtllm-serve configs under .claude_docs/tokenspeed-kimik25/ scoping the TokenSpeed MLA evaluation for TRTLLM-12510. Each adjacent pair of configs changes exactly one experimental dial (TP, concurrency, MTP, AttentionDP, ISL, dataset, serving mode, host KV cache) so the leadership-sign-off A/B is a clean orthogonal sweep rather than eleven unrelated runs. understanding.md indexes the grid with a per-row purpose summary and an accumulating risk register (NVIDIA#1 - NVIDIA#12) cross-referenced from each config's header. Configs cover: - 1k context kernel-isolation (TP4/TP8, conc=1/16, MTP=1/3, AttnDP on/off) - 8k long-context floor (MTP=1 and MTP=3) - 60k blog-regime infra-fit + multi-turn dataset variant - 60k disagg (P=4 / D=4) for decode-only TS-win isolation - 60k + CPU KV offload to close the JIRA-flagged BS=16/60k OOM gap Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
luyiyun1021
added a commit
to luyiyun1021/TensorRT-LLM
that referenced
this pull request
May 22, 2026
…change) Addresses post-review feedback on the v2a Ulysses wrapper. Pure refactoring; no functional changes. 1. PE pad moved into prepare_text_cache (done once per generate() instead of per forward step). 2. _pad_pe takes seq_dim explicitly (was fragile auto-detect by shape match); raises on out-of-range seq_dim. 3. Forward audio pad guard reduced from 4 conjuncts to 2. 4. Removed defensive ``if model_config is not None else True`` fallback. 5. PE pad pre-assign + overwrite block removed from forward (subsumed by NVIDIA#1). 6. configure_audio_ulysses two branches flattened to a single formula. 7. v2a fallback restructured to flat if/elif/else (was nested). Plus a stale comment fix: ``Ulysses wrap (a2v in LTX-2)`` → ``(v2a in LTX-2)``. Diff: ~44 insertions, ~57 deletions in transformer_ltx2.py only. Signed-off-by: Yiyun Lu <55233584+luyiyun1021@users.noreply.github.com>
luyiyun1021
added a commit
to luyiyun1021/TensorRT-LLM
that referenced
this pull request
May 23, 2026
…change) Addresses post-review feedback on the v2a Ulysses wrapper. Pure refactoring; no functional changes. 1. PE pad moved into prepare_text_cache (done once per generate() instead of per forward step). 2. _pad_pe takes seq_dim explicitly (was fragile auto-detect by shape match); raises on out-of-range seq_dim. 3. Forward audio pad guard reduced from 4 conjuncts to 2. 4. Removed defensive ``if model_config is not None else True`` fallback. 5. PE pad pre-assign + overwrite block removed from forward (subsumed by NVIDIA#1). 6. configure_audio_ulysses two branches flattened to a single formula. 7. v2a fallback restructured to flat if/elif/else (was nested). Plus a stale comment fix: ``Ulysses wrap (a2v in LTX-2)`` → ``(v2a in LTX-2)``. Diff: ~44 insertions, ~57 deletions in transformer_ltx2.py only. Signed-off-by: Yiyun Lu <55233584+luyiyun1021@users.noreply.github.com>
luyiyun1021
added a commit
to luyiyun1021/TensorRT-LLM
that referenced
this pull request
May 28, 2026
…change) Addresses post-review feedback on the v2a Ulysses wrapper. Pure refactoring; no functional changes. 1. PE pad moved into prepare_text_cache (done once per generate() instead of per forward step). 2. _pad_pe takes seq_dim explicitly (was fragile auto-detect by shape match); raises on out-of-range seq_dim. 3. Forward audio pad guard reduced from 4 conjuncts to 2. 4. Removed defensive ``if model_config is not None else True`` fallback. 5. PE pad pre-assign + overwrite block removed from forward (subsumed by NVIDIA#1). 6. configure_audio_ulysses two branches flattened to a single formula. 7. v2a fallback restructured to flat if/elif/else (was nested). Plus a stale comment fix: ``Ulysses wrap (a2v in LTX-2)`` → ``(v2a in LTX-2)``. Diff: ~44 insertions, ~57 deletions in transformer_ltx2.py only. Signed-off-by: Yiyun Lu <55233584+luyiyun1021@users.noreply.github.com>
luyiyun1021
added a commit
to luyiyun1021/TensorRT-LLM
that referenced
this pull request
May 29, 2026
…change) Addresses post-review feedback on the v2a Ulysses wrapper. Pure refactoring; no functional changes. 1. PE pad moved into prepare_text_cache (done once per generate() instead of per forward step). 2. _pad_pe takes seq_dim explicitly (was fragile auto-detect by shape match); raises on out-of-range seq_dim. 3. Forward audio pad guard reduced from 4 conjuncts to 2. 4. Removed defensive ``if model_config is not None else True`` fallback. 5. PE pad pre-assign + overwrite block removed from forward (subsumed by NVIDIA#1). 6. configure_audio_ulysses two branches flattened to a single formula. 7. v2a fallback restructured to flat if/elif/else (was nested). Plus a stale comment fix: ``Ulysses wrap (a2v in LTX-2)`` → ``(v2a in LTX-2)``. Diff: ~44 insertions, ~57 deletions in transformer_ltx2.py only. Signed-off-by: Yiyun Lu <55233584+luyiyun1021@users.noreply.github.com>
brb-nv
referenced
this pull request
in brb-nv/TensorRT-LLM
May 29, 2026
…rve-up
The first version of the plan was written before the cherry-picks were
exercised. Two recipes turned out to be wrong in practice; this commit
brings the doc back in line with what was actually committed on
`user/brb/moe-lora-{helpers,sidecar,feature}`.
* MR #1.2: drop *both* doc edits, not just the dev-note. The
`lora_layout.json` subsection in `lora.md` cannot land without the
parent "Routed-Expert MoE LoRA" section, which is added by
`4407871ce4` in MR #1.3. Both doc edits are re-added in MR #1.3.
* MR #1.3, conflict 1: prefer HEAD's wording on the
`test_moe_lora_slot_indexed_cuda_graph_replay_matches_eager` skip
reason. The `3c9060e2dd` wording references the `moe_device_lora_env`
fixture, which is owned by the device-path commits and is therefore
not present on MR #1.3.
* MR #1.3, conflict 2: drop `test_moe_lora_device_path_matches_host_path`
from the cherry-picked tail; it reads `TLLM_MOE_LORA_USE_DEVICE_PATH`,
also a device-path artifact. Keep `_LORA_REFERENCE_SCALE` onward.
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
brb-nv
referenced
this pull request
in brb-nv/TensorRT-LLM
May 29, 2026
Folds the original MR #1.2 (sidecar) into MR #1.1 (helpers) and renames the original MR #1.3 (feature) to MR #1.2. The combined helpers MR is on `user/brb/moe-lora-helpers` with two commits (helpers + sidecar); the remote `user/brb/moe-lora-sidecar` branch has been deleted. Rationale: helpers and sidecar are both Python-only pre-feature plumbing reviewed by overlapping reviewers. Two near-trivial Python MRs is more PR overhead than warranted; one MR with two clearly named commits gives the same review granularity at half the PR-process cost. The feature branch (`user/brb/moe-lora-feature`) is unaffected -- its base commit `53bf87e87c` (helpers) is still part of the consolidated helpers branch, so the feature MR's diff vs `helpers` and vs `main` both stay correct. Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
1 task
1 task
1 task
SpencerGarnets
added a commit
to ai-blaise/TensorRT-LLM
that referenced
this pull request
Jun 4, 2026
…ernels.cu Production C++ surface for the BDR dense MLA latent KV (directive NVIDIA#1, zero round-trip). Adds: * KvCacheDataType::KVARN to the enum (unfusedAttentionKernels.h). * dequantCopyKVarN<DstType,ELTS>: per-decode-step dequant-on-read = unpack ELTS INT4 codes from the 4-byte vec slot + per-token (scale,zp) -> ROTATED-frame fp16. NO inverse Hadamard, NO per-channel gather (the matching H is folded into k_b_proj_trans, Q-correction). Drop-in alongside dequantCopy. * bdRotate128InWarp<T,ELTS>: warp-cooperative order-128 block-diagonal FWHT for the quantize-on-write path (intra-lane butterfly + __shfl_xor_sync cross-lane). Both compile clean (nvcc -arch=sm_100, isolated harness RC=0). Modeled on the validated kvarn_inkernel/bdr_inkernel_bench.cu kernels (read 1.45x fp8, cos 0.99298). Write-branch wiring + dispatch follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 task
KleinBlueC
pushed a commit
to KleinBlueC/TensorRT-LLM
that referenced
this pull request
Jul 7, 2026
…E. All 8 acceptance criteria hold at runtime against...
QA verdict: APPROVE (weighted_score 8.6)
Change vs main: 7 files changed, 1376 insertions(+), 8 deletions(-)
Build iterations run: 13
Code-quality verification: clean after 2 round(s)
- round 1 (REJECT):
ROUND MIS-SCOPED — patch not actually reviewed. This round's /simplify, checklist, and /code-review executed with cwd = the agent-flow framework repo (branch kleinc/code_quality_improvement_v1) and reviewed the code-quality-stage feature itself, NOT this task's PR patch. The ChatGLM3 code is on branch agent-team/chatglm3-6b-bringup-v1 in the TensorRT-LLM snapshot repo (a different directory), so `/simplify`/`/code-review` (which diff the cwd's HEAD) never touched it. Root cause: the Quality agent runs in the framework repo's cwd, not the snapshot repo; another round will misfire identically unless the Quality agent runs with the TensorRT-LLM task branch as its working directory.
I re-reviewed the ACTUAL patch (git diff main in TensorRT-LLM). Real, unaddressed findings:
- TESTS (duplication): `_patch_chatglm3_hf_tied_compat()` duplicated in all 4 new test files; `_find_cuda_graph_runner()` and `_assert_cuda_graph_hard_path()` duplicated across test_chatglm3_gsm8k.py and test_chatglm3_replay.py; CHATGLM3_CKPT const, skip decorators, and RuntimeCfg class repeated. Consolidate to a shared util/conftest (keep all coverage).
- modeling_chatglm.py (simplification): redundant post-normalization reads of attention_bias/mlp_bias/head_dim via `getattr(...) or ...` after normalize_chatglm_config already set them; normalize_chatglm_config returns config but both callers discard it; a few verbose comments.
- ALTITUDE (larger, behavior-sensitive; flagged not applied): custom imperative load_weights vs framework BaseWeightMapper; ChatGLM-specific normalization special-cased in shared config_utils.py; partial_rotary_factor/rope_fusion hardcoded.
No fixes applied: the round did not validly review this patch, and edits to modeling/test code cannot be validated against the GSM8K accuracy criterion in this environment (risking a criterion regression). Recommend fixing the Quality agent's working directory (must be the snapshot repo/task branch) before the next round, then applying the test-dedup and post-normalization simplifications above.
- round 2 (REJECT):
Round 2 finally assessed the ACTUAL task patch (git diff main...agent-team/chatglm3-6b-bringup-v1 in the TensorRT-LLM repo), unlike round 1 whose skill steps mis-scoped to the agent-flow framework repo (cwd). Note: the automated stage still mis-scopes — /simplify and /code-review diff the cwd (agent-flow), not this task's snapshot repo — so I targeted the correct repo manually.
/code-review (correctness): CLEAN — returned []. No correctness bugs survive verification. The high-risk items are validated: partial-RoPE, QKV/GQA split sizes (32 heads / 2 KV / head_dim 128), and transformer.* weight-name mapping are exercised by the GSM8K-parity acceptance gate QA approved; the fused-residual path is completed by the final self.norm(hidden_states, residual); embedding matches the Qwen sibling and load_weights handles attention-DP (tp_size = 1 if enable_attention_dp else mapping.tp_size). Speculative "other ChatGLM variant" concerns (_set_missing fill-if-None, head_dim None if kv_channels absent, missing dense/output biases) are not reachable for chatglm3-6b and are guarded by strict _take/consumed accounting.
/simplify (round 2): applied one behavior-preserving cleanup (dropped the unused `return config` from normalize_chatglm_config, annotation -> None); it was subsequently reverted in the working tree, so the patch is unchanged.
OUTSTANDING (why REJECT): the checklist's test item is not satisfied — heavy test-helper duplication across the 4 new test files (`_patch_chatglm3_hf_tied_compat` in all 4; `_find_cuda_graph_runner`/`_assert_cuda_graph_hard_path`/`RuntimeCfg` across gsm8k+replay; repeated CHATGLM3_CKPT/skip decorators), ~100+ duplicated lines to consolidate into a shared util; plus minor post-normalization config-read redundancy in modeling_chatglm.py. NOT applied here because tensorrt_llm is not importable in this environment and there is no GSM8K/test rig, so a test-file move cannot be validated for collection/pass (risking the acceptance criterion), and the working-tree edit I did make was reverted.
RECOMMENDATION: this cannot be resolved by another automated round as currently wired — fix the Quality agent's working directory so /simplify and /code-review target the snapshot repo/task branch, then apply the test-helper de-duplication with the test rig available (or the author applies it directly). Correctness and framework-conformance are clean; the residual is a non-behavioral maintainability cleanup.
- round 1 (REJECT):
Code-quality round 1 for the chatglm3-6b bringup patch (modeling_chatglm.py, config_utils.py, _torch/models/__init__.py + 4 new test files).
/simplify applied (safe, behavior-preserving, verified against repo convention):
- modeling_chatglm.py: consume the normalized config fields instead of re-deriving booleans inline — `bias=config.attention_bias`, `dense_bias=config.mlp_bias`, GatedMLP `bias=config.mlp_bias`, and `head_dim=config.head_dim` at both sites (attention + load_weights). Confirmed no base infra reads these fields and that the model already hard-depends on normalize_chatglm_config having run; matches Llama/Phi3/Nemotron/Parakeet.
- Both integration tests: removed the unreachable BFS-over-object-graph fallback in `_find_cuda_graph_runner`, keeping the deterministic `_executor.engine.model_engine.cuda_graph_runner` chain (verified in source; the runner is at a fixed location in single-process mode and in a subprocess otherwise, where BFS also fails).
Codex checklist pass: the working tree carried minor docstring/comment trims in the ChatGLM3 test files (removed helper docstrings on `_patch_chatglm3_hf_tied_compat`, `_pick_metric_key`, `greedy_generate`, and an inline comment); no behavioral change.
/code-review (high) found 10 items. Fixed this turn: (NVIDIA#4) `_pick_metric_key` in test_chatglm3_gsm8k.py now raises loudly instead of silently gating on an arbitrary non-exact_match metric — strict hardening, no change to any real gsm8k run.
OUTSTANDING (why not auto-fixed):
- CORRECTNESS, latent/out-of-target: (NVIDIA#1) load_weights only handles chatglm3-6b's bias layout — it unconditionally fetches the QKV bias and never loads dense/MLP biases, so ChatGLM2/3 variants with add_qkv_bias=False or add_bias_linear=True would KeyError or trip the strict unconsumed-weight ValueError. (NVIDIA#2) normalize_chatglm_config never maps ChatGLM `rope_ratio` to `rope_theta` (get_hf_rope_theta reads config.rope_theta), so chatglm3-6b-32k/128k silently get theta=10000 and degrade at long context. Both are no-ops for base chatglm3-6b (add_qkv_bias=True/add_bias_linear=False/rope_ratio=1) so QA passed; fixing them adds untestable variant paths (no GPU/checkpoint here) and is scope creep on a base bringup — hand to the coder to fix+test or narrow the module's "ChatGLM2/3" claim.
- TEST ROBUSTNESS: (NVIDIA#3) the CUDA-graph hard-path tests require TLLM_WORKER_USE_SINGLE_PROCESS=1 but only name it in an assert string and never set it; without it, ENABLED-cfg tests hard-fail on `assert runner is not None` and BASELINE silently skips its checks. Not auto-fixed because gating/skipping would weaken the author's intended hard-path assertion — needs a CI-env or fixture decision.
- ALTITUDE/REUSE (larger refactors, unsafe without test execution): (NVIDIA#5) hand-rolled load_weights + consumed-set + ignorable-suffix lists reimplement BaseWeightMapper/register_mapper (cf. Glm4WeightLoader in modeling_glm.py); (NVIDIA#9) normalize special-cased in shared load_pretrained_config + duplicated in __init__ vs a ChatGLMConfigLoader; (NVIDIA#6) `_patch_chatglm3_hf_tied_compat` (4 copies) + RuntimeCfg/_build_llm/_find_cuda_graph_runner/_assert_cuda_graph_hard_path (2 copies) duplicated across two separate, GPU-gated test trees; (NVIDIA#10) _GreedyHFLM._model_generate hand-rolls HF greedy decode.
- EFFICIENCY: (NVIDIA#7) test_chatglm3_source_activation_replay reloads the 6B HF model + rebuilds TRT model for both parametrized scenarios though only the trailing cuda_graph block differs (use the _HFRef cache pattern).
- SCOPE CREEP: (NVIDIA#8) the diff reflows unrelated bart/minimaxm3/qwen imports (+ a mistral import in config_utils.py); left as-is because formatter ownership is ambiguous (line 33 is 89 chars vs isort/yapf's 80) and reverting risks fighting the pre-commit hook.
Rejecting: a substantive edit was made this turn and several actionable findings remain (test env-var robustness is the most important; the two latent correctness gaps and the altitude/reuse refactors should be triaged by the coder).
- round 2 (APPROVE):
Code-quality round 2 for the chatglm3-6b bringup patch.
/simplify (round 2) applied 3 trivial, behavior-neutral cosmetic cleanups in modeling_chatglm.py: inlined the single-use `head_dim` local in ChatGLMAttention (`head_dim=config.head_dim` passed directly to super()), inlined the single-use `head_dim` temp in normalize_chatglm_config, and dropped the noise `: set` annotation on `consumed`. Reuse/efficiency/altitude agents found nothing new safely-actionable and confirmed the two "small fold" hypotheses don't hold (base `skip_modules` matches model module names, not the checkpoint keys `_IGNORABLE_*` filters; the second normalize_chatglm_config call is load-bearing for the direct-ModelConfig test path) — the only remaining altitude/reuse items are large refactors.
Codex checklist pass (round 2): no net changes — independent verification confirmed the only diffs since round 1 were the 4 quality edits above/below, all behavior-neutral.
/code-review (high) re-surfaced the same stable finding set (the substantive code was unchanged; no coder commits landed between rounds). I acted on the most actionable one this turn:
- FIXED (NVIDIA#3, test robustness): the CUDA-graph hard-path tests required TLLM_WORKER_USE_SINGLE_PROCESS=1 (their ENABLED config hard-asserts the in-process runner) but never set it, so they would spuriously fail/silently-skip when run with a checkpoint outside a harness that exports it. Added an autouse `monkeypatch.setenv` fixture to both integration files (test_chatglm3_gsm8k.py, test_chatglm3_replay.py). Verified the env is read at runtime (utils.py:380, os.environ.get inside a function) so the fixture takes effect before _build_llm; the fix aligns each test with its own documented requirement and cannot regress the already-validated path.
- Previously fixed (round 1): _pick_metric_key now fails loudly instead of gating on an arbitrary metric; consumed bias/head_dim inline derivation replaced with normalized fields; BFS runner-finder fallback removed.
RESIDUAL (documented follow-ups — not safely resolvable by the automated quality loop):
- Out-of-target latent correctness: (NVIDIA#1) load_weights only handles the chatglm3-6b/chatglm2-6b bias layout (add_qkv_bias=True/add_bias_linear=False) — it unconditionally fetches the QKV bias and never loads dense/MLP biases, so ChatGLM variants with add_qkv_bias=False (KeyError) or add_bias_linear=True (unloaded bias + unconsumed-weight ValueError) fail; (NVIDIA#2) rope_ratio is never mapped to rope_theta, so long-context chatglm3-6b-32k/128k (rope_ratio>1) silently use theta=10000. Both are no-ops for the bringup target (base chatglm3-6b/chatglm2-6b, rope_ratio=1); fixing needs variant checkpoints to test and expands scope beyond the bringup — for the coder/a follow-up.
- Larger altitude/reuse/efficiency refactors needing test-execution access: (NVIDIA#4/NVIDIA#7) adopt BaseWeightMapper/register_mapper for load_weights + a ChatGLMConfigLoader/config-subclass to remove the shared-runtime normalize special-case and its second call; (NVIDIA#5) consolidate the 4x/2x-duplicated test helpers across the two GPU-gated test trees; (NVIDIA#6) cache the 6B HF/TRT model in the parametrized attention test; (NVIDIA#8) the _GreedyHFLM hand-rolled reference decode.
APPROVE rationale: base chatglm3-6b is fully correct (QA-approved, acceptance criteria met); two thorough rounds applied all safe in-scope polish culminating in the NVIDIA#3 test-robustness fix; the quality loop has converged (round 2 reproduced round 1's findings with no coder changes); and every remaining finding is either out-of-target latent correctness for untested ChatGLM variants or a larger refactor requiring test-execution access — none of which another automated /simplify→/code-review round can safely resolve. Residuals are surfaced above for human/coder triage.
QA report:
DECISION: APPROVE. All 8 acceptance criteria hold at runtime against the current committed code (HEAD 2323424, iter11). Reference is the checkpoint's own trust_remote_code modeling_chatglm.py — an independent HF reference sharing zero code with the TRT-LLM path.
PER-CRITERION (runtime evidence):
1. Bootstrap chain — PASS. slurm/core_gpu_tests.4147161.log: "Built target build_wheel_targets", "Successfully installed tensorrt_llm-1.3.0rc21 transformers-5.5.4" (pip install -e .[devel]), "Claude Code successfully installed!", "claude-agent-sdk 0.2.93"; all inside srun pyxis container with --container-mounts=<repo>:<repo> --container-workdir=<repo> under set -eo pipefail (tests ran only because bootstrap rc=0).
2. config_registration_and_weight_accounting — PASS (TIER1 rc=0). Real ckpt loads unedited, arch resolves to ChatGLMForCausalLM, TRTLLM backend + KVCacheManagerV2, strict state-dict accounting (raises on missing/unexpected/shape-mismatch; only rotary_pos_emb.inv_freq tolerated).
3. source_activation_replay — PASS both graph modes. layer0 cosine=1.000000 mean_abs=0.00000, layer27 cosine=0.999999; decode graph-vs-eager max_abs=0.0 cosine=1.0. Companion partial_rope_boundary_and_theta PASS (dims [0:64] rotate, [64:128] pass-through, theta=10000^(-i/32)).
4. source_logit_replay — PASS both configs. trt_tok==hf_tok (5231/30910/4802), cosine~0.99999; enabled num_captured_graphs=8, baseline 0.
5. generation_parity — PASS both configs. 5 prompts x 32 steps, token_mismatches=0, min per-step cosine~0.99998.
6. llm_api_smoke — PASS both configs (nonempty deterministic; V2 + TRTLLM asserted; enabled enabled=True graphs=8, baseline graphs=0).
7. gsm8k accuracy_canary — PASS (slurm/gsm8k_gate.4145862.log): HF=40.00, baseline=40.00 gap 0.00, enabled=40.00 gap 0.00; enabled num_captured_graphs=32.
8. gsm8k full_trtllm_eval — PASS: HF=52.77, baseline=52.99 gap 0.23, enabled=53.07 gap 0.30 (both < 2.0 tol), enabled captured 32 CUDA graphs; RESULT: PASS. Absolute ~53% matches published ChatGLM3-6B GSM8K, ruling out a shared-bug artifact.
PROVENANCE: Core log 4147161 started 05:55, after the 05:33 HEAD commit -> definitively iter11 (criteria 1-6). git diff iter10->iter11 (2d58cc7..2323424) touched ONLY the two test files (additive CUDA-graph hard-path assertions); tensorrt_llm/ is byte-identical. GSM8K log 4145862 prints the iter11-only [cuda_graph_hard_path] output -> it ran the current test file against the unchanged runtime code (live bind-mount imported iter11 mid-bootstrap). Criteria 7-8 are therefore verified against current code.
INDEPENDENT RERUN THIS TURN: I resubmitted both jobs. Core (4149172) matched the prior post-commit run and was cancelled as redundant. My GSM8K resubmit (4149173) FAILED in ~1.5 min inside build_wheel.py (FileNotFoundError: inherit_graph_6.md5 in copy_resolving_symlink) — a concurrent-build race from my own simultaneous submission of two bootstrap-heavy jobs on one live repo mount, NOT a ChatGLM defect. Accepted evidence is the sequential post-commit runs + full code/provenance review.
RED-TEAM (all addressed): independent HF reference (no shared helpers); partial GPT-J RoPE covered analytically + behaviorally (layer0 exact); CUDA-graph silent fallback ruled out by in-process CUDAGraphRunner introspection (runner.enabled + >=1 torch.cuda.CUDAGraph; baseline 0); KV-cache/mask/scale drift ruled out by 32-step parity (0 mismatches) + GSM8K absolute-score match; transformers 5.x bridges (all_tied_weights_keys={}, max_length, num_hidden_layers) are loader-compat only and do not alter ChatGLM forward semantics.
EVALUATION SCORES:
- Functionality (2.0): 9 — every criterion passes with exact argmax parity, cosine >0.9999, GSM8K within 0.3 pts of HF on both configs.
- Code Quality (1.0): 8 — clean, well-documented 373-line model reusing existing modules; narrow config hook; strict weight accounting; minor incidental import reformatting in __init__.py; a pre-existing (non-diff) ruff E501 at __init__.py:130.
- Performance (1.5): 8 — parity-first bring-up (no perf gate in task.yaml); production TRTLLM backend + CUDA graph + overlap scheduler + fused QKV/gate-up + KVCacheManagerV2 all working.
- Completeness (1.0): 9 — complete, buildable, no placeholders, full state-dict accounting, both runtime configs.
- Technical Sophistication (1.0): 9 — correct unfused partial GPT-J RoPE with TRTLLM backend + CUDA-graph capture, compact MQA KV, config/transformers bridges.
Weighted = (9x2.0 + 8x1.0 + 8x1.5 + 9x1.0 + 9x1.0)/6.5 = 56/6.5 = 8.6.
STRENGTHS: genuine independent HF reference; real CUDA-graph hard-path proof (not just config flags); strict weight accounting; large GSM8K margin; minimal composable design (no new schemas). WEAKNESSES: multi-GPU/perf/chunked-prefill deferred (out of scope per task.yaml); harness must run the two jobs sequentially to avoid the concurrent-build race; pre-existing E501 worth a one-line fix. RECOMMENDATION: APPROVE — no code defect identified; task.yaml completion_criteria (GSM8K within 2 pts with and without CUDA graph + overlap scheduler) is met on both configs.</summary>
</invoke>
Merged
7 tasks
1 task
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jul 17, 2026
Follow-up: user requires each pre-merge pick to already be actively
running in L0 CI post-merge on origin/main — i.e. no QA-lane
borrowing, no uncommenting of previously-disabled lines, no new
config yamls, no first-time-in-L0 e2e/gen_only combinations.
Prior revision's two picks that failed that bar:
* gpt-oss 8k1k con512 e2e dep2 — active in QA lane only; the
e2e mode of this 8k1k dep2 config was not in L0 anywhere.
* kimi 8k1k con4 tep8 mtp3 gen_only GB200 — was commented out
in the GB200 12-GPU 3-node post_merge block since April.
Replaced with:
* gpt-oss 8k1k con1024 tp4 e2e MTP0 — was active in L0 post-merge
on GB200 ctx1_node1_gpu1_gen1_node1_gpu4 (8 GPUs, 2 nodes).
Gen-worker parallelism shifts dep→tp for this slot.
* kimi 8k1k con4 tep8 mtp3 gen_only — was active in L0 post-merge
on GB300 ctx1_node1_gpu4_gen1_node2_gpu8 (12 GPUs, 3 nodes).
GPU family shifts GB200→GB300 for this slot.
File changes:
- l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu2.yml:
reverted to origin/main state (no pre_merge block).
- l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml:
new pre_merge block with gpt-oss 8k1k con1024 e2e; test line
removed from post_merge to avoid double-run.
- l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.yml:
reverted to origin/main state (kimi 8k1k stays commented out).
- l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.yml:
new pre_merge block with kimi 8k1k con4 tep8 mtp3 gen_only; same
test removed from post_merge.
Jenkins:
- Dropped stages: FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2
(GB200) and FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8
(GB200 kimi variant).
- New stages: FUNCTIONAL-ONLY-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4
(GB200 gpt-oss) and FUNCTIONAL-ONLY-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8
(GB300 kimi variant).
- Post-merge decrement: GB200-8_GPUs-2_Nodes CTX1-NODE1-GPU1-GEN1-
NODE1-GPU4 7→6; GB300-12_GPUs-3_Nodes CTX1-NODE1-GPU4-GEN1-NODE2-
GPU8 5→4. Others revert to origin/main testCounts.
Shape matrix (all 4 pre-merge slots now at 8k1k):
NVIDIA#1 DSR1 B200 8k1k con1536 dep e2e=NO gen_only MTP✓
NVIDIA#2 gpt-oss GB200 8k1k con1024 tp e2e=YES MTP✗
NVIDIA#3 glm-5 GB300 8k1k con1024 dep e2e=YES MTP✓
NVIDIA#4 kimi GB300 8k1k con4 tep e2e=NO gen_only MTP✓
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps onnx from 1.12.0 to 1.13.0.
Release notes
Sourced from onnx's releases.
... (truncated)
Changelog
Sourced from onnx's changelog.
... (truncated)
Commits
1ba7856Mark final RC (#4696)7adb421misc fixes for issues found in ort integration (#4681) (#4695)a913015Mark release as rc1 (#4674)3fd41d2Bump version (#4666)bad0697Add LpPool-18 - addceil_modeanddilationsattributes (#4534)7a1fae4make primary ops function step 2 (#4512)8fb26edFixed some typos in python.rst (#4668)9955f35Fix typo in python.rst (#4667)cd6e5dbConsider GraphInferenceContext in inference functions: InferenceContext (#4632)466edb7Add Python 3.11 support (#4490)You can trigger a rebase of this PR by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)You can disable automated security fix PRs for this repo from the Security Alerts page.