[None][fix] Align perf launcher with pytest shard - #17134
Conversation
chienchunhung
changed the title
|
/bot run --disable-fail-fast |
|
PR_Github #63099 [ run ] triggered by Bot. Commit: |
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
05880f6 to
7a2e0ec
Compare
|
/bot run --disable-fail-fast |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughThe performance launcher now selects tests with pytest-split duration balancing. It validates split settings, requires one selected test, and reuses that test line for name and output-directory derivation. Tests cover selection, mismatches, and missing duration data. ChangesPytest-split test selection
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Launcher
participant select_test_case_line
participant PytestCommand
participant DurationData
participant TestList
Launcher->>select_test_case_line: provide pytest command and split group
select_test_case_line->>PytestCommand: parse split configuration
select_test_case_line->>DurationData: load test durations
select_test_case_line->>TestList: read candidate test lines
select_test_case_line-->>Launcher: return selected test line
Launcher->>parse_test_case_name: pass selected test line
parse_test_case_name-->>Launcher: return test-case name
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
jenkins/scripts/perf/submit.py (2)
77-86: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider sharing the
pytestCommandline lookup.
_pytest_command_tokensandget_pytest_commands(Line 536) each locate theexport pytestCommand=line with their ownnext(...)expression. Extract one helper that returns the line. Both call sites then stay in sync if the prefix format changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/perf/submit.py` around lines 77 - 86, Extract the shared pytestCommand export-line lookup from _pytest_command_tokens and get_pytest_commands into a helper that returns the matching line or an empty value. Update both functions to use this helper, preserving their existing parsing and fallback behavior.
211-228: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider removing the second selection path.
mainalways passesselected_line, so thesplit_groupbranch at Lines 221-226 is unused in the CI flow. Two selection paths for one concept can diverge after a later change. Makeselected_linerequired, or have this fallback callselect_test_case_lineinstead of re-implementing positional selection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/perf/submit.py` around lines 211 - 228, The parse_test_case_name selection logic duplicates positional test selection even though main always supplies selected_line. Make selected_line required and remove the test_list_path, split_group, and fallback branch, or delegate fallback selection to select_test_case_line instead of indexing lines directly; preserve the existing selected-line parsing behavior.tests/unittest/scripts/test_perf_submit.py (1)
113-145: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest coverage summary
Changed test functions:
test_ci_submit_selects_same_least_duration_shard_as_pytest_split(added) — verifies least-duration group selection and the--durations-pathfallback tollm_src.test_ci_submit_rejects_split_group_disagreement(added) — verifies the--groupconsistency check.Test list registration: these are unit tests under
tests/unittest/scripts/, not integration tests, so no entry undertests/integration/test_lists/test-db/ortests/integration/test_lists/qa/is required. Confirm thattests/unittest/scripts/is already collected by an existing L0 unit-test stage.Verdict: sufficient for the primary fix, with these gaps worth closing:
- Equal durations. With
--splits 4and four tests, every group receives one test, so the tie-break order atsubmit.pyLine 155 is not exercised. Add a case with two identical durations.len(selected) != 1. Add a case where one group receives two tests, and assert the "requires exactly one test"ValueError.- Missing
--splits/--group. Add a case that asserts the positional fallback still returnslines[split_group - 1].As per path instructions: "Always produce a test coverage summary, even if no issues are found."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/scripts/test_perf_submit.py` around lines 113 - 145, Extend tests around select_test_case_line and test_ci_submit_selects_same_least_duration_shard_as_pytest_split to cover equal-duration tie-breaking, asserting the expected deterministic group selection. Add a case where one group contains multiple tests and assert the exact-one-test ValueError. Add a case without --splits or --group and verify positional fallback returns lines[split_group - 1]. Confirm the existing L0 unit-test collection includes tests/unittest/scripts/.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/scripts/perf/submit.py`:
- Around line 107-131: Update _load_pytest_split_durations so a missing
durations file is not silently converted to an empty dictionary: emit a clear
warning containing durations_path, or propagate an error when split_group > 0.
Preserve normal loading and validation for existing files, and ensure callers
can identify that duration-based selection is unavailable.
- Around line 134-165: Pin pytest-split to exactly version 0.10.0 in
requirements-dev.txt, preserving the existing dependency entry while adding the
explicit version constraint so CI uses the algorithm matched by
_select_least_duration_group.
---
Nitpick comments:
In `@jenkins/scripts/perf/submit.py`:
- Around line 77-86: Extract the shared pytestCommand export-line lookup from
_pytest_command_tokens and get_pytest_commands into a helper that returns the
matching line or an empty value. Update both functions to use this helper,
preserving their existing parsing and fallback behavior.
- Around line 211-228: The parse_test_case_name selection logic duplicates
positional test selection even though main always supplies selected_line. Make
selected_line required and remove the test_list_path, split_group, and fallback
branch, or delegate fallback selection to select_test_case_line instead of
indexing lines directly; preserve the existing selected-line parsing behavior.
In `@tests/unittest/scripts/test_perf_submit.py`:
- Around line 113-145: Extend tests around select_test_case_line and
test_ci_submit_selects_same_least_duration_shard_as_pytest_split to cover
equal-duration tie-breaking, asserting the expected deterministic group
selection. Add a case where one group contains multiple tests and assert the
exact-one-test ValueError. Add a case without --splits or --group and verify
positional fallback returns lines[split_group - 1]. Confirm the existing L0
unit-test collection includes tests/unittest/scripts/.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 272c70dc-8d52-4c8a-aac5-c830373da971
📒 Files selected for processing (2)
jenkins/scripts/perf/submit.pytests/unittest/scripts/test_perf_submit.py
|
PR_Github #63099 [ run ] completed with state
|
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #63248 [ run ] triggered by Bot. Commit: |
|
PR_Github #63248 [ run ] completed with state |
2 participants
Summary
Root cause
The launcher interpreted
--split-group Nas the Nth line in the test list. Pytest instead applies pytest-split'sleast_durationalgorithm using.test_durations, so a shard can select a different test from the raw Nth line.That allowed the launcher to configure servers and the output directory for one test while pytest executed another test. During the NVBUG#6481034 investigation, this caused the targeted Kimi shard to write and collect artifacts under a different test identity.
Fix
Parse the pytest-split options from the generated
pytestCommand, load the same duration data, and mirror pytest-split 0.10.0'sLeastDurationAlgorithm. The launcher now validates that its group matches pytest's group and requires one selected test for each multi-node perf shard.This PR contains no transfer tracing, runtime product changes, KV-cache fraction changes, timeout changes, test configuration changes, or waiver changes.
Validation
pre-commit run --files jenkins/scripts/perf/submit.py tests/unittest/scripts/test_perf_submit.pypython -m py_compile jenkins/scripts/perf/submit.py tests/unittest/scripts/test_perf_submit.pypython -m pytest -q --confcutdir=tests/unittest/scripts tests/unittest/scripts/test_perf_submit.py— 12 passedRelated investigation: NVBUG#6481034 and #16918.
Dev Engineer Review
submit.pynow mirrors pytest-split 0.10.0 duration-balanced shard selection.selected_line=None.pytest-splitis pinned to version0.10.0.QA Engineer Review
test_ci_submit_selects_same_least_duration_shard_as_pytest_split.test_ci_submit_rejects_split_group_disagreement.test_ci_submit_rejects_missing_pytest_split_durations.test-db/orqa/files.