Skip to content

Terminal verdict shows "Zero tests ran" for --minimum-expected-tests 0 while process exits 0 #9744

Description

@Evangelink

Summary

Follow-up from PR #9709 review (thread).

After #9709, an explicit --minimum-expected-tests 0 with a zero-test run correctly resolves to ExitCode.Success (0) in TestApplicationResult.GetProcessExitCode(). However, the terminal reporter verdict diverges: it still classifies the run as failed and prints "Zero tests ran".

Root cause

PlatformCommandLineProvider.GetMinimumExpectedTests() returns 0 for both "option not set" and "explicitly set to 0". The shared verdict helpers (TestRunSummaryHelper.IsRunFailed / GetVerdictText) receive only that int, so they cannot tell the two apart:

bool noTestsWereFound = totalTests == 0;            // true
bool ranZeroTests = noTestsWereFound || ...;         // true
return anyTestFailed || notEnoughTests || ranZeroTests || wasCancelled; // → true (failed)

and GetVerdictText falls through to the totalTests == 0 → "Zero tests ran" branch, because the min-violation branch (totalTests < minimumExpectedTests) is false for 0 < 0.

Net effect: the process exits 0 but the user sees a failure-styled "Zero tests ran" verdict in the terminal — the exit-code and terminal-verdict paths are inconsistent.

Impact

  • Primarily cosmetic. The exit code (the contract the dotnet test --test-modules orchestrator relies on) is correct.
  • For the main orchestrator use case, per-module terminal output isn't prominent, so this is low severity.
  • A user who types --minimum-expected-tests 0 directly against a single test host will see a confusing "Zero tests ran" failure verdict despite a success exit code.

Proposed fix

Propagate the "explicit minimum" intent to the terminal verdict path so it mirrors the exit-code logic (an explicit minimum governs and supersedes the zero-tests verdict). Options considered:

  • Make GetMinimumExpectedTests return int? (null = not set), or
  • Add a companion bool isMinimumExpectedTestsExplicit.

Note: TerminalTestReporterOptions.MinimumExpectedTests is vendored to dotnet/sdk (it appears in eng/vendored-files.json), so changing its shape needs a coordinated vendored-source update and drift-detector pass. That cross-repo coordination is why this was split out of #9709 rather than bundled in.

Acceptance

  • --minimum-expected-tests 0 + zero tests → terminal verdict is not rendered as a failed "Zero tests ran" run (consistent with exit code 0).
  • Behavior when --minimum-expected-tests is unset is unchanged (empty run still shows "Zero tests ran").

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions