Skip to content

Allow --minimum-expected-tests 0 (#7457)#9724

Merged
Evangelink merged 2 commits into
dev/amauryleve/curly-couscousfrom
dev/amauryleve/allow-min-expected-tests-zero
Jul 8, 2026
Merged

Allow --minimum-expected-tests 0 (#7457)#9724
Evangelink merged 2 commits into
dev/amauryleve/curly-couscousfrom
dev/amauryleve/allow-min-expected-tests-zero

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Summary

Follow-up to #9709, part of #7457 (item #2). Allows the literal value --minimum-expected-tests 0.

Previously validation rejected 0 (value <= 0). It now rejects only negative and non-integer values. Combined with the zero-tests verdict governance in #9709, an explicit --minimum-expected-tests 0 accepts an empty run and returns exit code 0 — the same outcome the reporter asked for in #7457 (""--minimum-expected-tests 0 should have the same outcome as --ignore-exit-code 8""). It gives users an explicit, opt-in ""zero tests is acceptable"" escape hatch per test host.

Why this is separate from #9709

Note: allowing a user-typed --minimum-expected-tests 0 is deliberately different from having the SDK auto-inject --min 0 into every module — the latter was rejected because older Microsoft.Testing.Platform versions reject --min 0 and the SDK cannot know a module's MTP version at argument-build time. Whoever types --min 0 explicitly owns that assumption.

Stacked on #9709

This PR targets the #9709 branch, because --minimum-expected-tests 0 + zero tests only resolves to success once #9709's verdict-governance logic is in place (otherwise the ZeroTests(8) verdict is computed first). Review/merge #9709 first; the diff here is only the validation change.

Change

  • PlatformCommandLineProvider.IsMinimumExpectedTestsOptionValidAsync: reject only value < 0.
  • Validation message: ""single non-zero positive integer"" → ""single non-negative integer"" (+ regenerated 13 PlatformResources.*.xlf via /t:UpdateXlf).

Tests

  • PlatformCommandLineProviderTests: 0 / 1 / 10 now valid; negatives / non-integers still invalid.
  • TestApplicationResultTests: --minimum-expected-tests 0 + zero tests → success.
  • ExecutionTests (acceptance): --filter-uid matching nothing + --minimum-expected-tests 0 → exit code 0; negative-value validation-message expectation updated.

Unit tests pass locally (110/110 across TestApplicationResultTests + PlatformCommandLineProviderTests). Acceptance tests require -pack and run in CI.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

Evangelink and others added 2 commits July 8, 2026 10:58
Validation now rejects only negative (or non-integer) values, so an explicit
'--minimum-expected-tests 0' is accepted. Combined with the zero-tests verdict
governance, a zero minimum accepts an empty run (exit code 0) -- the same
outcome as '--ignore-exit-code 8' -- giving users a per-module 'zero tests is
acceptable' opt-in.

Stacked on the verdict-governance change (#9709), which is what makes an
explicit '--minimum-expected-tests 0' with zero tests resolve to success
instead of the ZeroTests (exit code 8) verdict.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@Evangelink
Evangelink marked this pull request as ready for review July 8, 2026 11:54
@Evangelink
Evangelink merged commit addaa4c into dev/amauryleve/curly-couscous Jul 8, 2026
24 of 26 checks passed
@Evangelink
Evangelink deleted the dev/amauryleve/allow-min-expected-tests-zero branch July 8, 2026 11:54

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

✅ 22/22 dimensions clean — no findings.

Summary: Clean, well-scoped change. The validation relaxation (value <= 0value < 0) is correct — when --minimum-expected-tests 0 is provided, the option's presence routes GetProcessExitCode() into the minimum-expected branch where _totalRanTests < 0 is always false, yielding ExitCode.Success and bypassing the zero-tests-policy check entirely. This is the correct design for an explicit "zero tests is acceptable" opt-in.

Tests thoroughly cover boundaries (negative, non-integer, zero, positive), both at the unit and acceptance integration level. Localization is properly done via .resx with XLF targets correctly marked needs-review-translation. Changelog updated. No public API surface change. No threading, security, or performance concerns.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🧪 Test quality grade — PR #9724

5 test methods graded across 3 files, all earning A. The PR cleanly extends unit and acceptance coverage for the new --minimum-expected-tests 0 behaviour at every layer (unit, integration, and acceptance). Tests are well-named, well-commented, and assertive throughout.

GradeTestNotes
A (90–100) mod ExecutionTests.
Exec_
WhenMinimumExpectedTestsIsNegative_
ResultIsNotOk
Two assertions (exit code + output text) with a helpful inline comment; update aligns the expected string to the new validation wording.
A (90–100) new ExecutionTests.
Exec_
WhenMinimumExpectedTestsIsZeroAndNoTestsRun_
ResultIsOk
Well-named and commented; exit-code assertion is the complete observable verification for this acceptance scenario.
A (90–100) mod PlatformCommandLineProviderTests.
IsInvalid_
When_
MinimumExpectedTests_
Is_
Negative_
Or_
Not_
An_
Integer
Expanded DataRows (negative, float, empty, non-integer); dual assertions on validity flag and error message.
A (90–100) new PlatformCommandLineProviderTests.
IsValid_
When_
MinimumExpectedTests_
Is_
NonNegative_
Integer
Complements the invalid test; the IsValid check is the complete assertion for the happy path — no error message to also verify.
A (90–100) new TestApplicationResultTests.
GetProcessExitCodeAsync_
If_
MinimumExpectedTests_
Zero_
And_
No_
Tests_
Ran_
Returns_
Success
Well-commented with issue reference; exit-code equality check is exactly right for the zero-minimum scenario.

This advisory comment was generated automatically. Grades are heuristic
and informational — they do not block merging. Re-run with
/grade-tests.

🤖 Automated content by GitHub Copilot. Generated by the Grade Tests on PR (on open / sync) workflow. · 92.2 AIC · ⌖ 13.6 AIC · ⊞ 9.5K · [◷]( · )

Evangelink added a commit that referenced this pull request Jul 8, 2026
Reverts the ##9724 change that allowed the literal value
'--minimum-expected-tests 0'. Keeping '0' invalid restores an unambiguous
meaning for GetMinimumExpectedTests() == 0 ('option not set'), which avoids the
terminal-reporter verdict divergence tracked in ##9744: with '--min 0' the exit
code was 0 (success) while the terminal still rendered a failed 'Zero tests ran'
verdict, because the shared verdict helpers could not distinguish 'unset' from
'explicit 0' without a cross-repo change to the vendored TerminalTestReporterOptions.

Issue ##7457 item #2 ('--min 0 == --ignore-exit-code 8') remains served by the
existing '--ignore-exit-code 8'. The load-bearing behavior for the
'dotnet test --test-modules' orchestrator -- an explicit '--minimum-expected-tests N'
governs the zero-tests verdict (exit code 9, not 8) -- is unaffected and stays.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Evangelink added a commit that referenced this pull request Jul 8, 2026
Reverts the #9724 change that allowed the literal value
'--minimum-expected-tests 0'. Keeping '0' invalid restores an unambiguous
meaning for GetMinimumExpectedTests() == 0 ('option not set'), which avoids the
terminal-reporter verdict divergence tracked in #9744: with '--min 0' the exit
code was 0 (success) while the terminal still rendered a failed 'Zero tests ran'
verdict, because the shared verdict helpers could not distinguish 'unset' from
'explicit 0' without a cross-repo change to the vendored TerminalTestReporterOptions.

Issue #7457 item #2 ('--min 0 == --ignore-exit-code 8') remains served by the
existing '--ignore-exit-code 8'. The load-bearing behavior for the
'dotnet test --test-modules' orchestrator -- an explicit '--minimum-expected-tests N'
governs the zero-tests verdict (exit code 9, not 8) -- is unaffected and stays.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant