[None][infra] Waive 21 failed cases for main in post-merge 2780 - #15373
Merged
mzweilz merged 2 commits intoJun 16, 2026
Conversation
Bug(s): 6221055, 6323074, 6323889, 6324123, 6324131 Requested by: @mzweilz Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughUpdates ChangesTest Waiver List Updates
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Collaborator
|
/bot run --stage-list "A10-Build_Docs" |
mzweilz
approved these changes
Jun 15, 2026
Collaborator
|
PR_Github #54282 [ run ] triggered by Bot. Commit: |
Signed-off-by: Abby Wei <18545893+mzweilz@users.noreply.github.com>
Collaborator
|
/bot run --stage-list "A10-Build_Docs" |
Collaborator
|
PR_Github #54286 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #54282 [ run ] completed with state |
Collaborator
|
PR_Github #54286 [ run ] completed with state |
Collaborator
|
/bot skip --comment "Tests passed for waive PR created via CI report" |
mzweilz
enabled auto-merge (squash)
June 16, 2026 02:13
Collaborator
|
PR_Github #54418 [ skip ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #54418 [ skip ] completed with state |
This was referenced Jun 16, 2026
3 tasks
2 tasks
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jun 17, 2026
…num_tokens, perf-sanity timeout
Group 3 (KimiK25ForConditionalGeneration not in moe_model_arch_list):
Add 'KimiK25ForConditionalGeneration' to moe_model_arch_list so
maybe_create_moe_load_balancer calls MoeLoadBalancerConfig.setup() for
Kimi K2.5 MoE+EPLB configs. Without it, the wrapped DeepseekV3 backbone
raised 'Cannot calculate num_local_slots' on every rank during model
load (gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
failures, OCI verify Slurm 3320245).
Group 4 (glm5 8k1k tep8_mtp3 RequestError):
max_num_tokens=256 < ISL=8192 in glm5_fp4_tep8_mtp3_8k1k config; bump
to 8192 to match every other 8k1k aggregated config in the repo. OCI
verify (Slurm 3320259) shows output_token_throughput=238.94 tok/s.
Group 6 (long-context 128k8k > 5400s):
DEFAULT_TIMEOUT 5400 -> 10800 in test_perf_sanity.py so the in-test
wait_for_server_config / wait_for_benchmark_ready give the engine
enough budget for 128k context fills. Re-balance pytest TIMEOUT
markers across the perf-sanity test-db YAMLs:
- 128k8k entries: TIMEOUT (120) -> TIMEOUT (180), matches the new 3h
server budget;
- all other entries: TIMEOUT (120) -> TIMEOUT (90), tightening the
outer cap on tests that don't need 2h.
AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256
(Slurm 560881) reached output_token_throughput=820.77 tok/s.
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jun 17, 2026
…ve fixed cases
local/submit.py was missing the GB300 UCX_TLS path that the CI submit.py
already has. Without it, AWS GB300 disagg tests crashed with
'Failed to create NIXL backend: UCX' at startup. Mirror the CI logic
(jenkins/scripts/perf/submit.py:543-548):
is_gb300 -> export UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp
is_b200 -> export UCX_TLS=^ib (unchanged)
default -> unset UCX_TLS UCX_NET_DEVICES (was: only UCX_TLS)
AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256
(Slurm 560881) reached output_token_throughput=820.77 tok/s after the
fix; before it died at 5min with the NIXL backend error.
Drop waivers for the four cases that the fixes in this PR repair:
- aggr_upload-glm5_fp4_2_nodes_grace_blackwell-glm5_fp4_tep8_mtp3_8k1k
(was nvbugs/6324131; fixed by max_num_tokens 256->8192)
- disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
(was nvbugs/6323074; fixed by adding KimiK25 to moe_model_arch_list)
- disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
(was nvbugs/6323074; same fix)
- disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_pp4_dep8_mtp1
(was nvbugs/6323889; fixed by DEFAULT_TIMEOUT 5400->10800 + UCX_TLS)
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
xinhe-nv
pushed a commit
to tensorrt-cicd/TensorRT-LLM
that referenced
this pull request
Jun 23, 2026
…IA#15373) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Signed-off-by: Abby Wei <18545893+mzweilz@users.noreply.github.com> Co-authored-by: Abby Wei <18545893+mzweilz@users.noreply.github.com> Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jun 23, 2026
…num_tokens, perf-sanity timeout
Group 3 (KimiK25ForConditionalGeneration not in moe_model_arch_list):
Add 'KimiK25ForConditionalGeneration' to moe_model_arch_list so
maybe_create_moe_load_balancer calls MoeLoadBalancerConfig.setup() for
Kimi K2.5 MoE+EPLB configs. Without it, the wrapped DeepseekV3 backbone
raised 'Cannot calculate num_local_slots' on every rank during model
load (gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
failures, OCI verify Slurm 3320245).
Group 4 (glm5 8k1k tep8_mtp3 RequestError):
max_num_tokens=256 < ISL=8192 in glm5_fp4_tep8_mtp3_8k1k config; bump
to 8192 to match every other 8k1k aggregated config in the repo. OCI
verify (Slurm 3320259) shows output_token_throughput=238.94 tok/s.
Group 6 (long-context 128k8k > 5400s):
DEFAULT_TIMEOUT 5400 -> 10800 in test_perf_sanity.py so the in-test
wait_for_server_config / wait_for_benchmark_ready give the engine
enough budget for 128k context fills. Re-balance pytest TIMEOUT
markers across the perf-sanity test-db YAMLs:
- 128k8k entries: TIMEOUT (120) -> TIMEOUT (180), matches the new 3h
server budget;
- all other entries: TIMEOUT (120) -> TIMEOUT (90), tightening the
outer cap on tests that don't need 2h.
AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256
(Slurm 560881) reached output_token_throughput=820.77 tok/s.
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jun 23, 2026
…ve fixed cases
local/submit.py was missing the GB300 UCX_TLS path that the CI submit.py
already has. Without it, AWS GB300 disagg tests crashed with
'Failed to create NIXL backend: UCX' at startup. Mirror the CI logic
(jenkins/scripts/perf/submit.py:543-548):
is_gb300 -> export UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp
is_b200 -> export UCX_TLS=^ib (unchanged)
default -> unset UCX_TLS UCX_NET_DEVICES (was: only UCX_TLS)
AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256
(Slurm 560881) reached output_token_throughput=820.77 tok/s after the
fix; before it died at 5min with the NIXL backend error.
Drop waivers for the four cases that the fixes in this PR repair:
- aggr_upload-glm5_fp4_2_nodes_grace_blackwell-glm5_fp4_tep8_mtp3_8k1k
(was nvbugs/6324131; fixed by max_num_tokens 256->8192)
- disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
(was nvbugs/6323074; fixed by adding KimiK25 to moe_model_arch_list)
- disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
(was nvbugs/6323074; same fix)
- disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_pp4_dep8_mtp1
(was nvbugs/6323889; fixed by DEFAULT_TIMEOUT 5400->10800 + UCX_TLS)
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jun 24, 2026
…num_tokens, perf-sanity timeout
Group 3 (KimiK25ForConditionalGeneration not in moe_model_arch_list):
Add 'KimiK25ForConditionalGeneration' to moe_model_arch_list so
maybe_create_moe_load_balancer calls MoeLoadBalancerConfig.setup() for
Kimi K2.5 MoE+EPLB configs. Without it, the wrapped DeepseekV3 backbone
raised 'Cannot calculate num_local_slots' on every rank during model
load (gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
failures, OCI verify Slurm 3320245).
Group 4 (glm5 8k1k tep8_mtp3 RequestError):
max_num_tokens=256 < ISL=8192 in glm5_fp4_tep8_mtp3_8k1k config; bump
to 8192 to match every other 8k1k aggregated config in the repo. OCI
verify (Slurm 3320259) shows output_token_throughput=238.94 tok/s.
Group 6 (long-context 128k8k > 5400s):
DEFAULT_TIMEOUT 5400 -> 10800 in test_perf_sanity.py so the in-test
wait_for_server_config / wait_for_benchmark_ready give the engine
enough budget for 128k context fills. Re-balance pytest TIMEOUT
markers across the perf-sanity test-db YAMLs:
- 128k8k entries: TIMEOUT (120) -> TIMEOUT (180), matches the new 3h
server budget;
- all other entries: TIMEOUT (120) -> TIMEOUT (90), tightening the
outer cap on tests that don't need 2h.
AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256
(Slurm 560881) reached output_token_throughput=820.77 tok/s.
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Jun 24, 2026
…ve fixed cases
local/submit.py was missing the GB300 UCX_TLS path that the CI submit.py
already has. Without it, AWS GB300 disagg tests crashed with
'Failed to create NIXL backend: UCX' at startup. Mirror the CI logic
(jenkins/scripts/perf/submit.py:543-548):
is_gb300 -> export UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp
is_b200 -> export UCX_TLS=^ib (unchanged)
default -> unset UCX_TLS UCX_NET_DEVICES (was: only UCX_TLS)
AWS GB300 verify of e2e-gb300_deepseek-r1-fp4_128k8k_con256
(Slurm 560881) reached output_token_throughput=820.77 tok/s after the
fix; before it died at 5min with the NIXL backend error.
Drop waivers for the four cases that the fixes in this PR repair:
- aggr_upload-glm5_fp4_2_nodes_grace_blackwell-glm5_fp4_tep8_mtp3_8k1k
(was nvbugs/6324131; fixed by max_num_tokens 256->8192)
- disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
(was nvbugs/6323074; fixed by adding KimiK25 to moe_model_arch_list)
- disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_dep32_eplb416_mtp3
(was nvbugs/6323074; same fix)
- disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_pp4_dep8_mtp1
(was nvbugs/6323889; fixed by DEFAULT_TIMEOUT 5400->10800 + UCX_TLS)
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
xinhe-nv
pushed a commit
to tensorrt-cicd/TensorRT-LLM
that referenced
this pull request
Jun 24, 2026
…IA#15373) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Signed-off-by: Abby Wei <18545893+mzweilz@users.noreply.github.com> Co-authored-by: Abby Wei <18545893+mzweilz@users.noreply.github.com> Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-generated Waive PR
Created by: TensorRT LLM CI Report (requested by @mzweilz)
Target branch:
mainBug(s): 6221055, 6323074, 6323889, 6324123, 6324131
Waive entries added
This PR was auto-generated by TensorRT LLM CI Report. Please review the waive entries before merging.
Summary by CodeRabbit
Note: This release includes internal test infrastructure updates with no user-facing changes.