[None][fix] Stabilize perf-sanity tests by chenfeiz0326 · Pull Request #15440 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-06-17T03:15:39Z

Summary

Stabilize the perf-sanity tests waived in #15373. Three independent test fixes plus a CI-side fix that brings local/submit.py in line with the pipeline's submit.py. Also drops four upstream waivers that the fixes here repair.

What changed

Group 3 — `KimiK25ForConditionalGeneration` not in `moe_model_arch_list`

On Kimi K2.5 MoE configs with EPLB enabled (e.g. gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3), maybe_create_moe_load_balancer skipped MoeLoadBalancerConfig.setup() because KimiK25ForConditionalGeneration was not registered. The wrapped DeepseekV3 backbone then raised Cannot calculate num_local_slots on every rank during model load. Add the arch to moe_model_arch_list.

Group 4 — `glm5_fp4_tep8_mtp3_8k1k` `RequestError`

The aggregated server config had max_num_tokens: 256 while ISL is 8192, which raised RequestError: The sum of prompt length (8192.0), query length (0) should not exceed max_num_tokens (256). Bump to 8192 to match every other 8k1k aggregated config in the repo.

Group 6 — Long-context 128k8k > 5400s server timeout

The in-test wait_for_server_config / wait_for_benchmark_ready paths use slurm_config.get("timeout", DEFAULT_TIMEOUT). DEFAULT_TIMEOUT=5400 is too short for 128k context fills, so gb300_deepseek-r1-fp4_128k8k_con256 hung. Bump DEFAULT_TIMEOUT to 10800 (3h) and re-balance the pytest TIMEOUT markers across the perf-sanity test-db YAMLs:

128k8k entries: TIMEOUT (120) → TIMEOUT (180) to match the new 3h server budget
All other entries: TIMEOUT (120) → TIMEOUT (90) to tighten the outer cap on tests that don't need 2h

CI infra — GB300 `UCX_TLS` in `local/submit.py`

local/submit.py was missing the GB300 UCX_TLS path that jenkins/scripts/perf/submit.py already has. Without it, AWS GB300 disagg tests crashed with Failed to create NIXL backend: UCX before any work started. Mirror the CI logic (jenkins/scripts/perf/submit.py:543-548):

is_gb300 → export UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp
is_b200 → export UCX_TLS=^ib (unchanged)
default → unset UCX_TLS UCX_NET_DEVICES (was: unset UCX_TLS only)

Drop waivers that this PR repairs

Remove four entries from tests/integration/test_lists/waives.txt:

aggr_upload-glm5_fp4_2_nodes_grace_blackwell-glm5_fp4_tep8_mtp3_8k1k (was nvbugs/6324131)
disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL (was nvbugs/6323074)
disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL (was nvbugs/6323074)
disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL (was nvbugs/6323889)

Hardware verification

Test	Cluster	Slurm	Result
`aggr_upload-glm5_fp4_2_nodes_grace_blackwell-glm5_fp4_tep8_mtp3_8k1k`	OCI GB200	3320259	PASS — `output_token_throughput=238.94 tok/s`
`disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL`	OCI GB200 (9 nodes)	3320245	`Cannot calculate num_local_slots` is GONE; gen server reaches EPLB initial assignments and starts model load (regression unblocked)
`disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL`	AWS GB300	560881	PASS — `output_token_throughput=820.77 tok/s` (with `DEFAULT_TIMEOUT=10800` + GB300 `UCX_TLS`)

Test plan

CI perf-sanity runs over all touched test-db lists pass
No new perf regressions on the rebalanced TIMEOUT markers
AWS GB300 disagg_upload-e2e-gb300_*128k8k* benchmarks complete

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Added support for KimiK25ForConditionalGeneration model architecture with expert-parallelism enabled.
Tests
- Updated performance test configurations and timeout adjustments across multiple test scenarios for improved test execution reliability.
- Adjusted server configuration parameters to optimize performance testing.
Chores
- Enhanced GPU compatibility detection to support GB300 hardware configurations in disaggregated mode operations.

coderabbitai · 2026-06-17T03:20:20Z

📝 Walkthrough

Walkthrough

Adds GB300 GPU detection and a dedicated UCX_TLS export path in submit.py for disaggregated mode, registers KimiK25ForConditionalGeneration in the MoE load balancer allowlist, doubles DEFAULT_TIMEOUT to 10800 seconds, fixes glm5 max_num_tokens, rebalances per-test TIMEOUT values across 20+ GB200/GB300 YAML configs, and removes four stale SKIP waivers.

Changes

GB300 UCX Support and KimiK25 MoE Enablement

Layer / File(s)	Summary
GB300 UCX/TLS disagg command generation `jenkins/scripts/perf/local/submit.py`	Detects GB300 via `is_gb300` flag; adds a dedicated branch exporting `UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp` for GB300, and modifies the fallback to unset both `UCX_TLS` and `UCX_NET_DEVICES`.
KimiK25 added to MoE load balancer allowlist `tensorrt_llm/_torch/modules/fused_moe/moe_load_balancer.py`	Appends `'KimiK25ForConditionalGeneration'` to `moe_model_arch_list` so `maybe_create_moe_load_balancer()` can instantiate an `MoeLoadBalancer` for this architecture.

Perf Test Configuration Tuning

Layer / File(s)	Summary
DEFAULT_TIMEOUT increase and glm5 max_num_tokens fix `tests/integration/defs/perf/test_perf_sanity.py`, `tests/scripts/perf-sanity/aggregated/glm5_fp4_2_nodes_grace_blackwell.yaml`	`DEFAULT_TIMEOUT` raised from 5400 to 10800 seconds; `max_num_tokens` for `glm5_fp4_tep8_mtp3_8k1k` corrected from 256 to 8192.
GB200 multi-GPU and multi-node TIMEOUT rebalancing `tests/integration/test_lists/test-db/l0_gb200_multi_gpus_perf_sanity.yml`, `tests/integration/test_lists/test-db/l0_gb200_multi_nodes_perf_sanity_*.yml`	Most disagg/ctx_only GB200 cases reduced from 120 to 90 seconds; large-scale deepseek-r1 128k8k cases in higher-node configs increased from 120 to 180 seconds.
GB300 multi-GPU and multi-node TIMEOUT rebalancing `tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml`, `tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_*.yml`	Kimi-K25, GLM-5, and most DeepSeek cases reduced from 120 to 90 seconds; DeepSeek 128k8k and larger-scale gen_only/e2e cases increased from 120 to 180 seconds.
SKIP waiver cleanup `tests/integration/test_lists/waives.txt`	Removes four stale SKIP entries: `glm5_fp4_2_nodes_grace_blackwell` aggregated variant, `gb300_deepseek-r1` con256 disagg variant, and `gb200_kimi-k25-thinking` dep32/eplb416 gen_only variant; remaining entries are reordered.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14960: Modifies the same GB300 multi-GPU perf sanity YAML test list and related scheduling configs that this PR also updates.
NVIDIA/TensorRT-LLM#15250: Both modify tests/integration/test_lists/waives.txt to update perf/test_perf_sanity.py::test_e2e SKIP waivers for disaggregated GB200/GB300 upload configurations.
NVIDIA/TensorRT-LLM#15269: Both remove/update perf/test_perf_sanity.py skip waivers for GB300/disagg variants in the same waiver list file.

Suggested reviewers

schetlur-nv
tburt-nv
byshiue
chang-l

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title '[None][fix] Stabilize perf-sanity tests' clearly and concisely describes the main objective of the changeset: stabilizing perf-sanity tests.
Description check	✅ Passed	The PR description comprehensively covers all changes, including detailed explanations of four independent fixes, hardware verification results, and test plans.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

chenfeiz0326 · 2026-06-17T03:40:05Z

/bot run --disable-fail-fast --stage-list "GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-10,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4"

chenfeiz0326 · 2026-06-17T04:43:21Z

/bot run --disable-fail-fast --stage-list "GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-10,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4"

tensorrt-cicd · 2026-06-17T04:48:49Z

PR_Github #54754 [ run ] triggered by Bot. Commit: e092cf7 Link to invocation

tensorrt-cicd · 2026-06-17T07:16:07Z

PR_Github #54754 [ run ] completed with state FAILURE. Commit: e092cf7
/LLM/main/L0_MergeRequest_PR pipeline #43773 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-06-17T07:42:05Z

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-17T07:47:41Z

PR_Github #54799 [ run ] triggered by Bot. Commit: e092cf7 Link to invocation

tensorrt-cicd · 2026-06-17T09:43:23Z

PR_Github #54799 [ run ] completed with state FAILURE. Commit: e092cf7
/LLM/main/L0_MergeRequest_PR pipeline #43816 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-06-23T01:35:12Z

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-23T01:42:01Z

PR_Github #55115 [ run ] triggered by Bot. Commit: e6793c0 Link to invocation

tensorrt-cicd · 2026-06-23T03:53:03Z

PR_Github #55115 [ run ] completed with state FAILURE. Commit: e6793c0
/LLM/main/L0_MergeRequest_PR pipeline #44100 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

fredricz-20070104

Approved

chenfeiz0326 · 2026-06-23T07:29:09Z

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-06-23T07:35:35Z

PR_Github #55199 [ run ] triggered by Bot. Commit: 8334860 Link to invocation

tensorrt-cicd · 2026-06-23T10:18:47Z

PR_Github #55199 [ run ] completed with state FAILURE. Commit: 8334860
/LLM/main/L0_MergeRequest_PR pipeline #44161 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

…num_tokens, perf-sanity timeout Group 3 (KimiK25ForConditionalGeneration not in moe_model_arch_list): Add 'KimiK25ForConditionalGeneration' to moe_model_arch_list so maybe_create_moe_load_balancer calls MoeLoadBalancerConfig.setup() for Kimi K2.5 MoE+EPLB configs. Without it, the wrapped DeepseekV3 backbone raised 'Cannot calculate num_local_slots' on every rank during model load (gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3 failures, OCI verify Slurm 3320245). Group 4 (glm5 8k1k tep8_mtp3 RequestError): max_num_tokens=256 < ISL=8192 in glm5_fp4_tep8_mtp3_8k1k config; bump to 8192 to match every other 8k1k aggregated config in the repo. OCI verify (Slurm 3320259) shows output_token_throughput=238.94 tok/s. Group 6 (long-context 128k8k > 5400s): DEFAULT_TIMEOUT 5400 -> 10800 in test_perf_sanity.py so the in-test wait_for_server_config / wait_for_benchmark_ready give the engine enough budget for 128k context fills. Re-balance pytest TIMEOUT markers across the perf-sanity test-db YAMLs: - 128k8k entries: TIMEOUT (120) -> TIMEOUT (180), matches the new 3h server budget; - all other entries: TIMEOUT (120) -> TIMEOUT (90), tightening the outer cap on tests that don't need 2h. AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256 (Slurm 560881) reached output_token_throughput=820.77 tok/s. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

…ve fixed cases local/submit.py was missing the GB300 UCX_TLS path that the CI submit.py already has. Without it, AWS GB300 disagg tests crashed with 'Failed to create NIXL backend: UCX' at startup. Mirror the CI logic (jenkins/scripts/perf/submit.py:543-548): is_gb300 -> export UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp is_b200 -> export UCX_TLS=^ib (unchanged) default -> unset UCX_TLS UCX_NET_DEVICES (was: only UCX_TLS) AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256 (Slurm 560881) reached output_token_throughput=820.77 tok/s after the fix; before it died at 5min with the NIXL backend error. Drop waivers for the four cases that the fixes in this PR repair: - aggr_upload-glm5_fp4_2_nodes_grace_blackwell-glm5_fp4_tep8_mtp3_8k1k (was nvbugs/6324131; fixed by max_num_tokens 256->8192) - disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3 (was nvbugs/6323074; fixed by adding KimiK25 to moe_model_arch_list) - disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3 (was nvbugs/6323074; same fix) - disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_pp4_dep8_mtp1 (was nvbugs/6323889; fixed by DEFAULT_TIMEOUT 5400->10800 + UCX_TLS) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Pre-merge L0 SBSA-Multi-GPU run NVIDIA#2155 timed out at 1.5h on test_e2e[disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL]. Bump TIMEOUT marker from 90 → 120 minutes so the test gets the full 2h. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Post-Merge-2 reproduction with sm dropped from GB300 UCX_TLS still fails: GEN workers crash with SIGBUS (signal 7, "Non-existant physical address") at ~80% weight loading on multiple ranks. The fault is in libc memcpy on high-VA mmap pages — i.e. SHM/mmap-backed-page exhaustion in the loader path itself, before UCX/NIXL is exercised. Dropping sm from UCX_TLS does not address it. Restore sm in the GB300 UCX_TLS list (back to the original `cuda_copy,cuda_ipc,sm,self,tcp`) and re-waive the still-failing case under the existing nvbugs/6323074: perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL] Logs: results-GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2.tar.gz (PR 44161). Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-06-24T02:51:35Z

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-06-24T02:57:41Z

PR_Github #55382 [ skip ] triggered by Bot. Commit: 4525daa Link to invocation

tensorrt-cicd · 2026-06-24T03:03:43Z

PR_Github #55382 [ skip ] completed with state SUCCESS. Commit: 4525daa
Skipping testing for commit 4525daa

Link to invocation

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 requested a review from a team as a code owner June 17, 2026 03:15

chenfeiz0326 requested a review from a team June 17, 2026 03:15

chenfeiz0326 requested review from a team as code owners June 17, 2026 03:15

chenfeiz0326 requested review from QiJune, ZhanruiSunCh and mlefeb01 June 17, 2026 03:15

github-actions Bot assigned chenfeiz0326 Jun 17, 2026

chenfeiz0326 changed the title ~~[None][fix] Stabilize perf-sanity tests broken by #15373~~ [None][fix] Stabilize perf-sanity tests Jun 17, 2026

chenfeiz0326 force-pushed the fix-pr15373-clean branch from 7d6b52c to e092cf7 Compare June 17, 2026 04:40

chenfeiz0326 requested review from litaotju, mzweilz and ruodil June 23, 2026 02:09

mzweilz approved these changes Jun 23, 2026

View reviewed changes

chenfeiz0326 requested review from a team and fredricz-20070104 June 23, 2026 02:37

ruodil approved these changes Jun 23, 2026

View reviewed changes

chenfeiz0326 requested a review from xxi-nv June 23, 2026 02:47

xxi-nv approved these changes Jun 23, 2026

View reviewed changes

chenfeiz0326 force-pushed the fix-pr15373-clean branch from e6793c0 to e332b54 Compare June 23, 2026 03:14

fredricz-20070104 approved these changes Jun 23, 2026

View reviewed changes

Shixiaowei02 approved these changes Jun 23, 2026

View reviewed changes

chenfeiz0326 force-pushed the fix-pr15373-clean branch from 8334860 to e576dc4 Compare June 24, 2026 02:42

chenfeiz0326 added 4 commits June 24, 2026 02:48

chenfeiz0326 force-pushed the fix-pr15373-clean branch from e576dc4 to 4525daa Compare June 24, 2026 02:48

chenfeiz0326 enabled auto-merge (squash) June 24, 2026 02:51

chenfeiz0326 merged commit cdd6230 into NVIDIA:main Jun 24, 2026
8 checks passed

xinhe-nv pushed a commit to tensorrt-cicd/TensorRT-LLM that referenced this pull request Jun 24, 2026

[None][fix] Stabilize perf-sanity tests (NVIDIA#15440)

9a87c8a

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[None][fix] Stabilize perf-sanity tests (NVIDIA#15440)

46b4c17

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

coderabbitai Bot mentioned this pull request Jul 17, 2026

[None][test] Remove all 1k1k perf-sanity cases from CI #16535

Merged

4 tasks

Uh oh!

Conversation

chenfeiz0326 commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Group 3 — KimiK25ForConditionalGeneration not in moe_model_arch_list

Group 4 — glm5_fp4_tep8_mtp3_8k1k RequestError

Group 6 — Long-context 128k8k > 5400s server timeout

CI infra — GB300 UCX_TLS in local/submit.py

Drop waivers that this PR repairs

Hardware verification

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

chenfeiz0326 commented Jun 17, 2026

Uh oh!

chenfeiz0326 commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

chenfeiz0326 commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

chenfeiz0326 commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 23, 2026

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 23, 2026

Uh oh!

chenfeiz0326 commented Jun 24, 2026

Uh oh!

tensorrt-cicd commented Jun 24, 2026

Uh oh!

tensorrt-cicd commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

chenfeiz0326 commented Jun 17, 2026 •

edited

Loading

Group 3 — `KimiK25ForConditionalGeneration` not in `moe_model_arch_list`

Group 4 — `glm5_fp4_tep8_mtp3_8k1k` `RequestError`

CI infra — GB300 `UCX_TLS` in `local/submit.py`

coderabbitai Bot commented Jun 17, 2026 •

edited

Loading