Skip to content

[https://nvbugs/6335726][fix] In test_qwen_moe_routed_expert_multi_lora_varying_ranks, drop ranks… - #15488

Open
tensorrt-cicd wants to merge 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6335726
Open

[https://nvbugs/6335726][fix] In test_qwen_moe_routed_expert_multi_lora_varying_ranks, drop ranks…#15488
tensorrt-cicd wants to merge 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6335726

Conversation

@tensorrt-cicd

@tensorrt-cicd tensorrt-cicd commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Root cause: Three full Qwen1.5-MoE-A2.7B-Chat builds + 5 routed-expert LoRA adapter sets per parametrize item exceed the unittest harness's 2400 s pytest-timeout (the test-list TIMEOUT (90) does not propagate into the inner pytest), causing pytest-timeout to fire mid-test.
  • Fix: In test_qwen_moe_routed_expert_multi_lora_varying_ranks, drop ranks [8,16,32,16,64] → [8,32,64] (still exercises slot-indexed per-slot rank dispatch min/mid/max + rank-0 skip row) and max_tokens 20 → 5 (single divergent-token assert holds), then remove the stale waivers.txt entry.
  • Automated fix generated by repair-bot

Test plan

  • Verify fix on the same GPU type as the original failure
  • Check for regressions in related tests

Links

Summary by CodeRabbit

  • Tests
    • Re-enabled a previously skipped multi-adapter LoRA test for execution.
    • Optimized test parameters to reduce resource consumption, including simplified adapter configurations and reduced generation limits.

@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The test test_qwen_moe_routed_expert_multi_lora_varying_ranks is modified to use three routed-expert LoRA adapters with ranks [8, 32, 64] instead of five, and max_tokens is reduced from 20 to 5. Its corresponding CI skip waiver entry is removed from waives.txt.

Changes

MoE LoRA varying-ranks test fix and re-enablement

Layer / File(s) Summary
Reduce adapter ranks and max_tokens
tests/unittest/llmapi/test_llm_pytorch.py
Adapter rank list changed from [8, 16, 32, 16, 64] (5 adapters) to [8, 32, 64] (3 adapters) with updated comments, and SamplingParams.max_tokens reduced from 20 to 5.
Remove CI skip waiver
tests/integration/test_lists/waives.txt
Deletes the full:H100_PCIe waiver for test_qwen_moe_routed_expert_multi_lora_varying_ranks, allowing it to run in CI.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#15271: Adds/validates test_qwen_moe_routed_expert_multi_lora_varying_ranks in the same test file that this PR modifies.
  • NVIDIA/TensorRT-LLM#15439: Adds the waiver for test_qwen_moe_routed_expert_multi_lora_varying_ranks that this PR removes, creating a direct conflict at the waiver level.
  • NVIDIA/TensorRT-LLM#15411: Also modifies tests/integration/test_lists/waives.txt to change CI skip entries for other test cases.

Suggested reviewers

  • 2ez4bz
  • venkywonka
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Title check ❓ Inconclusive The title is partially related to the changeset but appears truncated/incomplete, making it unclear what the actual change entails. Complete the title to clearly convey the main change (e.g., 'Reduce test workload in test_qwen_moe_routed_expert_multi_lora_varying_ranks to fix timeout').
✅ Passed checks (4 passed)
Check name Status Explanation
Description check ✅ Passed The description provides comprehensive technical details about root cause, fix, test plan, and validation, but does not follow the repository's PR template structure.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/unittest/llmapi/test_llm_pytorch.py (1)

1164-1164: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update stale adapter count in docstring.

The docstring mentions "Five dummy adapters" but the implementation now uses three adapters with ranks [8, 32, 64] (changed at line 1198).

📝 Proposed fix
-    Five dummy adapters of varying rank target the routed experts (moe_h_to_4h,
+    Three dummy adapters of varying rank target the routed experts (moe_h_to_4h,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/llmapi/test_llm_pytorch.py` at line 1164, The docstring
starting at line 1164 mentions "Five dummy adapters" but the actual
implementation uses three adapters with ranks [8, 32, 64] as per the changes
made around line 1198. Update the docstring to accurately reflect that three
adapters are being used and consider mentioning their ranks to provide clearer
documentation of what the test is doing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tests/unittest/llmapi/test_llm_pytorch.py`:
- Line 1164: The docstring starting at line 1164 mentions "Five dummy adapters"
but the actual implementation uses three adapters with ranks [8, 32, 64] as per
the changes made around line 1198. Update the docstring to accurately reflect
that three adapters are being used and consider mentioning their ranks to
provide clearer documentation of what the test is doing.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 41a537ca-5d53-4dde-b24f-7847870865be

📥 Commits

Reviewing files that changed from the base of the PR and between 4a8b7af and f64942b.

📒 Files selected for processing (2)
  • tests/integration/test_lists/waives.txt
  • tests/unittest/llmapi/test_llm_pytorch.py
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

@tensorrt-cicd
tensorrt-cicd force-pushed the repair-bot-bug6335726 branch 2 times, most recently from 07a5eff to 0603cb2 Compare June 25, 2026 10:57
… pytest-timeout

The H100_PCIe-PyTorch-Ray-1 stage runs the unittest harness with
--timeout=2400 (40 min) for the inner pytest, while the test list's
TIMEOUT (90) annotation is consumed by the integration harness and not
propagated. test_qwen_moe_routed_expert_multi_lora_varying_ranks
parametrizes 3 routed-expert MoE LoRA execution paths (host_path,
device_path_eager, device_path_cudagraph), and each item rebuilds
Qwen1.5-MoE-A2.7B-Chat (~14B) plus 5 dummy LoRA adapters and runs
generate() twice; the combined cost exceeds 2400 s and pytest-timeout
fires inside _write_routed_expert_lora_adapter.

Reduce per-item compute by:
- Dropping max_tokens 20 -> 5. The only generate-side assertion is
  out_tokens[i+1] != base_tokens, which holds at the first divergent
  token under greedy temperature=0.0 with nonzero LoRA-B weights.
- Dropping the duplicated rank-16 adapter and one extra: ranks
  [8, 16, 32, 16, 64] -> [8, 32, 64]. This still exercises the
  slot-indexed device-path's per-slot rank dispatch (min, mid, max)
  with a no-LoRA row mixed into the same decode batch, and still
  builds adapters across every routed expert and MoE layer.

Total wall-clock per item drops from >2400 s timeout to ~120 s
(verified: 3 passed in 348 s for all three modes). Also remove the
now-stale waiver entry so the test runs again in CI.

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd force-pushed the repair-bot-bug6335726 branch from 0603cb2 to 199145a Compare June 28, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant