[None][test] Un-waive K2.5 Thinking FP4 disagg-NIXL e2e/gen_only tests - #15443
Conversation
📝 WalkthroughWalkthroughRemoves three SKIP waive entries from Changeskimi-k25-thinking waive removal
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Validated 5/5 PASS on ToT main wheel for the GB300 e2e and GB200
gen_only variants on Lyris with the configured perf-sanity threshold:
- disagg_upload-e2e-gb300 K2.5 8k1k con4096 dep16 NIXL:
119242 - 120884 tok/s, 1.38% spread (5/5)
- disagg_upload-gen_only-gb200 K2.5 8k1k con4096 dep16 NIXL:
92186 - 94509 tok/s, 2.50% spread (5/5)
The GB200 e2e variant (NVBug 6179661) shares the same model and
config family and is unblocked by the same ToT fixes, so it is
unwaived together with the other two.
NVBugs:
- https://nvbugs/6280649 (gb300 e2e)
- https://nvbugs/6221022 (gb200 gen_only)
- https://nvbugs/6179661 (gb200 e2e)
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
49c177b to
ce1e98c
Compare
|
/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4" |
|
PR_Github #54756 [ run ] triggered by Bot. Commit: |
|
PR_Github #54756 [ run ] completed with state
|
…6 NIXL The newly un-waived disagg_upload-e2e/gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL test still timed out in pre-merge with the default kv_transfer_timeout_ms. Raise it to 600000 ms to match the deepseek-r1/v32 NIXL perf-sanity configs. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
|
/bot run --disable-fail-fast --stage-list "GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2" |
|
PR_Github #54859 [ run ] triggered by Bot. Commit: |
|
PR_Github #54859 [ run ] completed with state |
|
/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline" |
|
PR_Github #55127 [ skip ] triggered by Bot. Commit: |
|
PR_Github #55127 [ skip ] completed with state |
NVIDIA#15443) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>
NVIDIA#15443) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Summary
Removes 3 SKIP entries from
tests/integration/test_lists/waives.txtfor the K2.5 Thinking FP4 disagg-NIXL family — the underlying issues are resolved on current ToT main.Tests removed from waives.txt
disagg_upload-e2e-gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXLdisagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXLdisagg_upload-e2e-gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXLBoth validated cases were run as 5 separate Slurm jobs (per-run output dir) on Lyris against a freshly built ToT main wheel:
gb300, dep4 ctx + dep16 gengb200, dep4 ctx + dep16 gen, gen_only modeSpread is well under the 5% CV threshold used by perf-sanity.
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit