Skip to content

Apply dotnet test zero-tests verdict at the whole-run level - #55362

Merged
Evangelink merged 6 commits into
mainfrom
dev/amauryleve/fix-zero-tests-ci
Jul 20, 2026
Merged

Apply dotnet test zero-tests verdict at the whole-run level#55362
Evangelink merged 6 commits into
mainfrom
dev/amauryleve/fix-zero-tests-ci

Conversation

@Evangelink

Copy link
Copy Markdown
Member

A test run should not fail just because one module matched no tests when other modules executed successfully. The current per-module exit code aggregation also leaves the terminal summary inconsistent: the command exits successfully, but reports the run as failed.

This change:

  • normalizes per-module ZeroTests results before aggregate exit-code calculation
  • decides the zero-tests verdict once from the whole-run test count
  • treats an empty module as successful in terminal summary accounting while retaining its Exit code: 8 diagnostic
  • preserves explicit minimum-test policy failures
  • adds end-to-end and focused reporter coverage for an empty module alongside a passing module

This implements the SDK side of microsoft/testfx#7457.

Evangelink and others added 5 commits July 20, 2026 09:23
The whole-run zero-tests normalization (microsoft/testfx#7457) means an empty
module's exit code 8 (ZeroTests) is normalized to success before aggregation,
so it no longer collides with the failing module's exit code 2 to produce
GenericFailure (1). The run's verdict is now AtLeastOneTestFailed (2). Update
the expectation and rename the test accordingly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e0b86e41-a49a-4a68-bea8-3cf5df5694d8
…dule E2E test

- Skip the whole-run zero-tests verdict when invoked in help mode so the help command keeps returning exit code 0 (Success) instead of ZeroTests.

- Add an E2E test (plus MultiTestProjectSolutionWithZeroTestsAndPassingTests asset) covering a run where one module matches no tests (exit code 8) but another passes, asserting the overall exit code is Success.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8abef3da-3035-4f48-85ab-485d5f704fc2
Treat an empty module as successful when computing the terminal run summary, matching the whole-run exit-code aggregation. Add focused coverage for an empty module alongside a passing module.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 182327bb-5826-4966-a2cc-1b92338b706f
Copilot AI review requested due to automatic review settings July 20, 2026 08:13
@Evangelink
Evangelink requested a review from a team as a code owner July 20, 2026 08:13
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts dotnet test (Microsoft.Testing.Platform path) so that a single module reporting ZeroTests (exit code 8) does not fail an otherwise successful multi-module run, and so the “zero tests ran” verdict is decided once at the whole-run level based on the total executed test count.

Changes:

  • Normalize per-module ZeroTests results before aggregate exit-code calculation, and decide whole-run ZeroTests only from the overall TotalTests count.
  • Update terminal reporting and exit-code behavior for mixed runs (empty module + passing/failing module).
  • Add new test assets and tests to cover the mixed “zero tests + passing” scenario.
Show a summary per file
File Description
src/Cli/dotnet/Commands/Test/MTP/TestApplicationActionQueue.cs Normalizes per-module ZeroTests before aggregating exit codes.
src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs Applies whole-run ZeroTests verdict based on aggregate TotalTests == 0.
src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs Treats ZeroTests as success for per-assembly success tracking.
test/dotnet.Tests/CommandTests/Test/TerminalTestReporterTests.cs Adds focused reporter test for “zero tests + passing” summary behavior.
test/dotnet.Tests/CommandTests/Test/GivenDotnetTestBuildsAndRunsTests.cs Updates/extends end-to-end tests for mixed-module exit code + summary behavior.
test/TestAssets/TestProjects/MultiTestProjectSolutionWithZeroTestsAndPassingTests/MultiTestProjectSolutionWithZeroTestsAndPassingTests.sln New multi-project solution test asset for “zero tests + passing tests”.
test/TestAssets/TestProjects/MultiTestProjectSolutionWithZeroTestsAndPassingTests/global.json Configures test asset to use Microsoft.Testing.Platform runner.
test/TestAssets/TestProjects/MultiTestProjectSolutionWithZeroTestsAndPassingTests/TestProject/TestProject.csproj New empty-module MTP test project (intentionally produces zero tests).
test/TestAssets/TestProjects/MultiTestProjectSolutionWithZeroTestsAndPassingTests/TestProject/Program.cs Implements an MTP adapter that publishes no tests (zero-tests module).
test/TestAssets/TestProjects/MultiTestProjectSolutionWithZeroTestsAndPassingTests/AnotherTestProject/AnotherTestProject.csproj New passing-module MTP test project.
test/TestAssets/TestProjects/MultiTestProjectSolutionWithZeroTestsAndPassingTests/AnotherTestProject/Program.cs Implements an MTP adapter that publishes a passing test node.

Copilot's findings

  • Files reviewed: 11/11 changed files
  • Comments generated: 2

Comment thread src/Cli/dotnet/Commands/Test/MTP/TestApplicationActionQueue.cs
Comment thread src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs
Preserve dispose failures after normalizing zero-test module results and retain the per-assembly zero-tests label in terminal output.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 182327bb-5826-4966-a2cc-1b92338b706f
@Evangelink
Evangelink enabled auto-merge July 20, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants