Skip to content

Commit ac098f7

Browse files
Youssef1313radicaldanmoseleymitchdennydavidfowl
authored
Migrate from VSTest to Microsoft.Testing.Platform (#8498)
* Migrate from VSTest to Microsoft.Testing.Platform * Address TODO * Cleanup stuff around runsettings * Add comment * Progress * Remove locale * Always show live output * Restore test session timeout * Fix timeout * Fix typo * Fix duplicate commnand-line options * Fix Linux * Fix for Linux * Fix * Fix * Filter failing * Fix * ignore exit code 8 * Fix extra dot * Ignore exit code * Fix duplicate --ignore-exit-code * Fix playground tests on helix * Move to RepoTesting.props * Fix results dir * Fix * Progress * Fix duplicate command-line option * Missing report-trx * Run executable * Address review comments * Revert "Run executable" This reverts commit d10e168. * dotnet dll * Introduce version selector for Aspire templates (#8625) * Introduce version selecto to aspire new * Update src/Aspire.Cli/Commands/NewCommand.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/Aspire.Cli/Commands/NewCommand.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: David Fowler <davidfowl@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Error handling for GetCapabilitiesAsync connection issues (#8614) * Error out when GetCapabiltiesAsync is missing or unexpected exception occurs. * Update src/Aspire.Cli/DotNetCliRunner.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Obsolete AddAzureContainerAppsInfrastructure (#8639) * Obsolete AddAzureContainerAppsInfrastructure This method is no longer meant to be used. Instead developers should be calling AddAzureContainerAppEnvironment. Covert the tests using this API to the new API * Fix volume output naming issue We weren't discriminating between volumes and bindmounts, which caused a cache collision. * Switch the prefix at the beginning * Add error handling for package updates in workflow (#8390) * Filter out tracing health endpoints in service defaults templates (#8643) * Filter out tracing health endpoints Fixes #8580 * Fix typo * Show neutral icon for container exiting with status code 0 (#8477) * Show neutral icon for container exiting with status code 0 * Show info/stop icons for all exited resources with status code 0 * Add finished test for custom resource, remove exit code expectation * update tests * [tests] Correctly mark Aspire.Dashboard.Tests.Integration.Playwright.AppBarTests with RequiresPlaywright (#8647) * [tests] Skip generating Directory.Packages.Versions.props for Test (#8621) .. Utility projects. * Fix running playground tests on windows --------- Co-authored-by: Ankit Jain <radical@gmail.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Mitch Denny <midenn@microsoft.com> Co-authored-by: David Fowler <davidfowl@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com> Co-authored-by: Jose Perez Rodriguez <joperezr@microsoft.com> Co-authored-by: Damian Edwards <damian@damianedwards.com> Co-authored-by: Adam Ratzman <adam@adamratzman.com>
1 parent cff75b6 commit ac098f7

28 files changed

+181
-289
lines changed

‎.github/workflows/run-tests.yml‎

Lines changed: 20 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@ on:
1313
# relative the repo root
1414
required: false
1515
type: string
16-
testSessionTimeoutMs:
16+
testSessionTimeout:
1717
required: false
1818
type: string
19-
default: "900000"
19+
default: "15m"
2020
testHangTimeout:
2121
required: false
2222
type: string
@@ -220,18 +220,14 @@ jobs:
220220
TEST_LOG_PATH: ${{ github.workspace }}/artifacts/log/test-logs
221221
TestsRunningOutsideOfRepo: true
222222
run: >
223-
dotnet test -s .runsettings -v:n ${{ env.TEST_ASSEMBLY_NAME }}.dll
224-
-l "console;verbosity=normal"
225-
-l "trx;LogFilePrefix=${{ inputs.testShortName }}"
226-
-l "GitHubActions;summary.includePassedTests=true;summary.includeSkippedTests=true"
227-
--blame
228-
--blame-hang-timeout ${{ inputs.testHangTimeout }}
229-
--blame-crash
223+
dotnet ${{ env.TEST_ASSEMBLY_NAME }}.dll
224+
--report-trx --report-trx-filename "${{ inputs.testShortName }}.trx"
225+
--hangdump --hangdump-timeout ${{ inputs.testHangTimeout }}
226+
--crashdump
230227
--results-directory ${{ github.workspace }}/testresults
228+
--filter-not-trait "category=failing"
229+
--timeout ${{ inputs.testSessionTimeout }}
231230
${{ inputs.extraTestArgs }}
232-
--
233-
RunConfiguration.CollectSourceInformation=true
234-
RunConfiguration.TestSessionTimeout=${{ inputs.testSessionTimeoutMs }}
235231
236232
- name: Run tests
237233
if: ${{ ! inputs.requiresNugets }}
@@ -240,24 +236,24 @@ jobs:
240236
CI: false
241237
DCP_DIAGNOSTICS_LOG_LEVEL: debug
242238
DCP_DIAGNOSTICS_LOG_FOLDER: ${{ github.workspace }}/testresults/dcp
239+
# During restore and build, we use -ci, which causes NUGET_PACKAGES to point to a local cache (Arcade behavior).
240+
# In this step, we are not using Arcade, but want to make sure that MSBuild is able to evaluate correctly.
241+
# So, we manually set NUGET_PACKAGES
242+
NUGET_PACKAGES: ${{ github.workspace }}/.packages
243243
run: >
244244
${{ env.DOTNET_SCRIPT }} test ${{ env.TEST_PROJECT_PATH }}
245245
/p:ContinuousIntegrationBuild=true
246-
-s eng/testing/.runsettings
247-
-l "console;verbosity=normal"
248-
-l "trx;LogFilePrefix=${{ inputs.testShortName }}"
249-
-l "GitHubActions;summary.includePassedTests=true;summary.includeSkippedTests=true"
250-
"--blame"
251-
--blame-hang-timeout ${{ inputs.testHangTimeout }}
252-
--blame-crash
253-
--results-directory testresults
246+
/p:TrxFileNamePrefix="${{ inputs.testShortName }}"
247+
-bl:${{ github.workspace }}/testresults/test.binlog
254248
--no-restore
255249
--no-build
256-
-bl:${{ github.workspace }}/testresults/test.binlog
257-
${{ inputs.extraTestArgs }}
258250
--
259-
RunConfiguration.CollectSourceInformation=true
260-
RunConfiguration.TestSessionTimeout=${{ inputs.testSessionTimeoutMs }}
251+
--report-trx
252+
--hangdump --hangdump-timeout ${{ inputs.testHangTimeout }}
253+
--crashdump
254+
--results-directory ${{ github.workspace }}/testresults
255+
--timeout ${{ inputs.testSessionTimeout }}
256+
${{ inputs.extraTestArgs }}
261257
262258
# Save the result of the previous steps - success or failure
263259
# in the form of a file result-success/result-failure -{name}.rst

‎.github/workflows/tests.yml‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ jobs:
7777
with:
7878
testShortName: ${{ matrix.shortname }}
7979
os: "ubuntu-latest"
80-
extraTestArgs: "--filter \"(quarantined!=true)\""
80+
extraTestArgs: "--filter-not-trait \"quarantined=true\""
8181

8282
integrations_test_win:
8383
uses: ./.github/workflows/run-tests.yml
@@ -90,7 +90,7 @@ jobs:
9090
with:
9191
testShortName: ${{ matrix.shortname }}
9292
os: "windows-latest"
93-
extraTestArgs: "--filter \"(quarantined!=true)\""
93+
extraTestArgs: "--filter-not-trait \"quarantined=true\""
9494

9595
templates_test_lin:
9696
name: Templates Linux
@@ -103,10 +103,9 @@ jobs:
103103
testShortName: ${{ matrix.shortname }}
104104
os: "ubuntu-latest"
105105
testProjectPath: tests/Aspire.Templates.Tests/Aspire.Templates.Tests.csproj
106-
testSessionTimeoutMs: 1200000
106+
testSessionTimeout: 20m
107107
testHangTimeout: 12m
108-
# append '.' to the name so only the test class with exactly that name is run
109-
extraTestArgs: "--filter \"(quarantined!=true)&(FullyQualifiedName~Aspire.Templates.Tests.${{ matrix.shortname }}.)\""
108+
extraTestArgs: "--filter-not-trait quarantined=true --filter-class Aspire.Templates.Tests.${{ matrix.shortname }}"
110109
requiresNugets: true
111110
requiresTestSdk: true
112111

@@ -121,10 +120,9 @@ jobs:
121120
testShortName: ${{ matrix.shortname }}
122121
os: "windows-latest"
123122
testProjectPath: tests/Aspire.Templates.Tests/Aspire.Templates.Tests.csproj
124-
testSessionTimeoutMs: 1200000
123+
testSessionTimeout: 20m
125124
testHangTimeout: 12m
126-
# append '.' to the name so only the test class with exactly that name is run
127-
extraTestArgs: "--filter \"(quarantined!=true)&(FullyQualifiedName~Aspire.Templates.Tests.${{ matrix.shortname }}.)\""
125+
extraTestArgs: "--filter-not-trait quarantined=true --filter-class Aspire.Templates.Tests.${{ matrix.shortname }}"
128126
requiresNugets: true
129127
requiresTestSdk: true
130128

‎eng/.runsettings‎

Lines changed: 0 additions & 33 deletions
This file was deleted.

‎eng/Testing.props‎

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -9,23 +9,18 @@
99
<RunOnAzdoCILinux>true</RunOnAzdoCILinux>
1010
<RunOnAzdoHelixWindows>true</RunOnAzdoHelixWindows>
1111
<RunOnAzdoHelixLinux>true</RunOnAzdoHelixLinux>
12-
</PropertyGroup>
13-
14-
<PropertyGroup Condition="'$(UseVSTestRunner)' != 'true'">
15-
<_QuarantinedTestRunAdditionalArgs>-trait "quarantined=true"</_QuarantinedTestRunAdditionalArgs>
16-
<_NonQuarantinedTestRunAdditionalArgs>-notrait "quarantined=true"</_NonQuarantinedTestRunAdditionalArgs>
17-
</PropertyGroup>
1812

19-
<PropertyGroup Condition="'$(UseVSTestRunner)' == 'true'">
20-
<_QuarantinedTestRunAdditionalArgs>--filter "quarantined=true"</_QuarantinedTestRunAdditionalArgs>
21-
<_NonQuarantinedTestRunAdditionalArgs>--filter "quarantined!=true"</_NonQuarantinedTestRunAdditionalArgs>
13+
<_QuarantinedTestRunAdditionalArgs>--filter-trait "quarantined=true"</_QuarantinedTestRunAdditionalArgs>
14+
<_NonQuarantinedTestRunAdditionalArgs>--filter-not-trait "quarantined=true"</_NonQuarantinedTestRunAdditionalArgs>
2215
</PropertyGroup>
2316

2417
<PropertyGroup>
2518
<BlameHangTimeout Condition="'$(BlameHangTimeout)' == ''">10m</BlameHangTimeout>
26-
<_BlameArgs>--blame --blame-hang-timeout $(BlameHangTimeout) --blame-crash</_BlameArgs>
19+
<_BlameArgs>--hangdump --hangdump-timeout $(BlameHangTimeout) --crashdump</_BlameArgs>
20+
21+
<TestRunnerAdditionalArguments>$(TestRunnerAdditionalArguments) --filter-not-trait &quot;category=failing&quot;</TestRunnerAdditionalArguments>
2722

28-
<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' == ''">$(TestRunnerAdditionalArguments) $(_NonQuarantinedTestRunAdditionalArgs) $(TestRunnerAdditionalArguments) $(_BlameArgs)</TestRunnerAdditionalArguments>
29-
<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' == 'true'">$(TestRunnerAdditionalArguments) $(_QuarantinedTestRunAdditionalArgs) $(TestRunnerAdditionalArguments) $(_BlameArgs)</TestRunnerAdditionalArguments>
23+
<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' != 'true'">$(TestRunnerAdditionalArguments) $(_NonQuarantinedTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
24+
<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' == 'true'">$(TestRunnerAdditionalArguments) $(_QuarantinedTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
3025
</PropertyGroup>
3126
</Project>

‎eng/Versions.props‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,10 @@
1212
<DotNetRuntimePreviousVersionForTesting>8.0.13</DotNetRuntimePreviousVersionForTesting>
1313
<!-- dotnet 8.0 versions for running tests - used for templates tests -->
1414
<DotNetSdkPreviousVersionForTesting>8.0.406</DotNetSdkPreviousVersionForTesting>
15-
<UseVSTestRunner>true</UseVSTestRunner>
1615
<XunitV3Version>2.0.0</XunitV3Version>
1716
<XUnitAnalyzersVersion>1.20.0</XUnitAnalyzersVersion>
1817
<XunitRunnerVisualStudioVersion>3.0.2</XunitRunnerVisualStudioVersion>
18+
<MicrosoftTestingPlatformVersion>1.6.3</MicrosoftTestingPlatformVersion>
1919
<!-- Enable to remove prerelease label. -->
2020
<StabilizePackageVersion Condition="'$(StabilizePackageVersion)' == ''">false</StabilizePackageVersion>
2121
<DotNetFinalVersionKind Condition="'$(StabilizePackageVersion)' == 'true'">release</DotNetFinalVersionKind>

‎eng/Xunit3/Microsoft.Testing.Platform.targets‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. -->
2+
<Project>
3+
4+
<Target Name="RunTests"
5+
Outputs="%(TestToRun.ResultsStdOutPath)"
6+
Condition="'@(TestToRun)' != ''">
7+
<Telemetry EventName="NETCORE_ENGINEERING_TELEMETRY" EventData="Category=Test" />
8+
9+
<PropertyGroup>
10+
<_TestResultDirectory>$([System.IO.Path]::GetDirectoryName('%(TestToRun.ResultsTrxPath)'))</_TestResultDirectory>
11+
<_TestResultTrxFileName>$([System.IO.Path]::GetFileName('%(TestToRun.ResultsTrxPath)'))</_TestResultTrxFileName>
12+
<_TestResultXmlFileName>$([System.IO.Path]::GetFileName('%(TestToRun.ResultsXmlPath)'))</_TestResultXmlFileName>
13+
<_TestResultHtmlFileName>$([System.IO.Path]::GetFileName('%(TestToRun.ResultsHtmlPath)'))</_TestResultHtmlFileName>
14+
</PropertyGroup>
15+
16+
<PropertyGroup>
17+
<_TestEnvironment>%(TestToRun.EnvironmentDisplay)</_TestEnvironment>
18+
<_TestAssembly>%(TestToRun.Identity)</_TestAssembly>
19+
<_TestAssembly Condition="'$(OS)'=='Windows_NT'">$([System.IO.Path]::ChangeExtension($(_TestAssembly), '.exe'))</_TestAssembly>
20+
<_TestAssembly Condition="'$(OS)'!='Windows_NT'">$(_TestAssembly.TrimEnd('.dll'))</_TestAssembly>
21+
<_TestRuntime>%(TestToRun.TestRuntime)</_TestRuntime>
22+
<_TestTimeout>%(TestToRun.TestTimeout)</_TestTimeout>
23+
<_TestRunnerAdditionalArguments>%(TestToRun.TestRunnerAdditionalArguments)</_TestRunnerAdditionalArguments>
24+
25+
<_TestRunner>$(_TestAssembly)</_TestRunner>
26+
<_TestRunnerArgs>$(_TestRunnerAdditionalArguments) --results-directory "$(_TestResultDirectory)" --report-xunit --report-xunit-filename "$(_TestResultXmlFileName)" --report-xunit-html --report-xunit-html-filename "$(_TestResultHtmlFileName)" --report-trx --report-trx-filename "$(_TestResultTrxFileName)"</_TestRunnerArgs>
27+
</PropertyGroup>
28+
29+
<PropertyGroup Condition="'$(_TestRuntime)' == 'Core'">
30+
<_TestRunnerArgs>$(_TestRunnerArgs) --auto-reporters off</_TestRunnerArgs>
31+
</PropertyGroup>
32+
33+
<PropertyGroup>
34+
<_TestRunnerCommand>"$(_TestRunner)" $(_TestRunnerArgs)</_TestRunnerCommand>
35+
36+
<!--
37+
Redirect std output of the runner.
38+
Note that xUnit outputs failure info to both STDOUT (stack trace, message) and STDERR (failed test name)
39+
-->
40+
<_TestRunnerCommand Condition="'$(TestCaptureOutput)' != 'false'">$(_TestRunnerCommand) > "%(TestToRun.ResultsStdOutPath)" 2>&amp;1</_TestRunnerCommand>
41+
</PropertyGroup>
42+
43+
<ItemGroup>
44+
<_OutputFiles Include="%(TestToRun.ResultsXmlPath)" />
45+
<_OutputFiles Include="%(TestToRun.ResultsHtmlPath)" />
46+
<_OutputFiles Include="%(TestToRun.ResultsStdOutPath)" />
47+
</ItemGroup>
48+
49+
<MakeDir Directories="@(_OutputFiles->'%(RootDir)%(Directory)')"/>
50+
<Delete Files="@(_OutputFiles)" />
51+
52+
<Message Text="Running tests: $(_TestAssembly) [$(_TestEnvironment)]" Importance="high"/>
53+
<Exec Command='$(_TestRunnerCommand)'
54+
LogStandardErrorAsError="false"
55+
WorkingDirectory="$(_TargetDir)"
56+
IgnoreExitCode="true"
57+
Timeout="$(_TestTimeout)"
58+
EnvironmentVariables="DOTNET_ROOT=$(DotNetRoot);DOTNET_ROOT_X86=$(DotNetRoot)x86"
59+
ContinueOnError="WarnAndContinue">
60+
<Output TaskParameter="ExitCode" PropertyName="_TestErrorCode" />
61+
</Exec>
62+
63+
<!--
64+
Add command line to the log.
65+
-->
66+
<WriteLinesToFile File="%(TestToRun.ResultsStdOutPath)"
67+
Overwrite="false"
68+
Lines=";=== COMMAND LINE ===;$(_TestRunnerCommand)"
69+
Condition="'$(TestCaptureOutput)' != 'false'" />
70+
71+
<!--
72+
Report test status.
73+
-->
74+
<Message Text="Tests succeeded: $(_TestAssembly) [$(_TestEnvironment)]" Condition="'$(_TestErrorCode)' == '0'" Importance="high" />
75+
76+
<PropertyGroup>
77+
<_ResultsFileToDisplay>%(TestToRun.ResultsHtmlPath)</_ResultsFileToDisplay>
78+
<_ResultsFileToDisplay Condition="!Exists('$(_ResultsFileToDisplay)')">%(TestToRun.ResultsStdOutPath)</_ResultsFileToDisplay>
79+
</PropertyGroup>
80+
81+
<!--
82+
Ideally we would set ContinueOnError="ErrorAndContinue" so that when a test fails in multi-targeted test project
83+
we'll still run tests for all target frameworks. ErrorAndContinue doesn't work well on Linux though: https://github.com/Microsoft/msbuild/issues/3961.
84+
-->
85+
<Error Text="Tests failed: $(_ResultsFileToDisplay) [$(_TestEnvironment)]" Condition="'$(_TestErrorCode)' != '0' and '$(_ErrorOnTestFailure)' != 'false'" File="XUnit" />
86+
87+
<ItemGroup>
88+
<FileWrites Include="@(_OutputFiles)"/>
89+
</ItemGroup>
90+
</Target>
91+
92+
</Project>

‎eng/Xunit3/Xunit3.targets‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,23 @@
44
<PackageVersion Include="xunit.v3.core" Version="$(XunitV3Version)" />
55
<PackageVersion Include="xunit.analyzers" Version="$(XunitAnalyzersVersion)" />
66
<PackageVersion Include="xunit.v3.assert" Version="$(XunitV3Version)" />
7-
<PackageVersion Include="xunit.runner.visualstudio" Version="$(XunitRunnerVisualStudioVersion)" />
87
<PackageVersion Include="xunit.v3.runner.console" Version="$(XunitV3Version)" />
8+
<PackageVersion Include="Microsoft.Testing.Platform" Version="$(MicrosoftTestingPlatformVersion)" />
9+
<PackageVersion Include="Microsoft.Testing.Platform.MSBuild" Version="$(MicrosoftTestingPlatformVersion)" />
10+
<PackageVersion Include="Microsoft.Testing.Extensions.HangDump" Version="$(MicrosoftTestingPlatformVersion)" />
11+
<PackageVersion Include="Microsoft.Testing.Extensions.CrashDump" Version="$(MicrosoftTestingPlatformVersion)" />
12+
<PackageVersion Include="Microsoft.Testing.Extensions.TrxReport" Version="$(MicrosoftTestingPlatformVersion)" />
913

1014
<PackageReference Include="xunit.v3" />
1115
<PackageReference Include="xunit.v3.core" />
1216
<PackageReference Include="xunit.analyzers" />
1317
<PackageReference Include="xunit.v3.assert" />
14-
<PackageReference Include="xunit.runner.visualstudio" />
18+
<PackageReference Include="Microsoft.Testing.Platform" />
19+
<PackageReference Include="Microsoft.Testing.Platform.MSBuild" />
20+
<PackageReference Include="Microsoft.Testing.Extensions.HangDump" />
21+
<PackageReference Include="Microsoft.Testing.Extensions.CrashDump" />
22+
<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" />
1523
</ItemGroup>
1624

17-
<Import Project="..\tools\VSTest.targets" Sdk="Microsoft.DotNet.Arcade.Sdk" />
18-
</Project>
25+
<Import Project="Microsoft.Testing.Platform.targets" />
26+
</Project>

‎eng/testing/.runsettings‎

Lines changed: 0 additions & 35 deletions
This file was deleted.

‎tests/Aspire.Components.Common.Tests/Aspire.Components.Common.Tests.csproj‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
<PropertyGroup>
44
<TargetFramework>$(DefaultTargetFramework)</TargetFramework>
5+
<!-- https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-exit-codes -->
6+
<!-- Exit code 8 is "zero tests ran" -->
7+
<!-- Currently, none of the tests in this project run in CI. All are ignored -->
8+
<TestingPlatformCommandLineArguments>$(TestingPlatformCommandLineArguments) --ignore-exit-code 8</TestingPlatformCommandLineArguments>
59
</PropertyGroup>
610

711
<ItemGroup>

‎tests/Aspire.EndToEnd.Tests/.runsettings‎

Lines changed: 0 additions & 26 deletions
This file was deleted.

0 commit comments

Comments
 (0)