Add `dotnet test` integration tests for all Apple platforms by jonathanpeppers · Pull Request #25320 · dotnet/macios

jonathanpeppers · 2026-05-04T21:58:14Z

This PR adds end-to-end dotnet test integration tests for all Apple platforms (iOS, tvOS, Mac Catalyst, macOS) using the MTP (Microsoft Testing Platform) test templates.

Changes

tests/dotnet/UnitTests/DotNetTestTest.cs: New parameterized test that creates a project from dotnet new <template>, replaces the generated tests with a single passing test, and runs dotnet test.
tests/common/DotNet.cs: Added test verb support to DotNet.Execute.

The test reads the MLAUNCH_PATH environment variable to override the mlaunch binary for mobile platforms (iOS/tvOS), enabling testing with the MTP-enabled mlaunch build.

Screenshot

Create DotNetTestTest that runs 'dotnet test' on projects created from the iostest, tvostest, maccatalysttest, and macostest templates. This verifies the MTP (Microsoft Testing Platform) pipeline works end-to-end, including the --server/--dotnet-test-pipe argument passthrough via RunArguments. Also add 'test' as a supported verb in DotNet.Execute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

macOS and Mac Catalyst don't use mlaunch (they use 'open' via Desktop.targets), so MTP support requires a different approach. Comment out those test cases for now. Also skip the /v:diag and /consoleloggerparameters workaround args when the verb is 'test' to prevent them leaking through to the MTP test runner via RunArguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The NUnit runner process sets MSBuildSDKsPath and MSBUILD_EXE_PATH which interfere with the child dotnet process SDK resolution. Every DotNet.Execute call clears these; we must do the same. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rolfbjarne · 2026-05-18T15:16:29Z

+			env ["MSBuildSDKsPath"] = null;
+			env ["MSBUILD_EXE_PATH"] = null;
+			var testArgs = new List<string> { "test", proj };
+			var testResult = Execution.RunAsync (DotNet.Executable, testArgs, env, Console.Out, workingDirectory: outputDir, timeout: TimeSpan.FromMinutes (10)).Result;


It's failing with "Running the ComputeRunArguments target to discover run commands failed for this project. Fix the errors and warnings and run again.", so maybe a binlog would be able to reveal what's going on?

The latest has a .binlog now, but it's partial (because something aborted), and I can't seem to get the error message out of it.

I think the problem is something to do with having no Simulator running at all -- are these tests setup for having a simulator? Or should we test in a different place?

Error message is:

Running the ComputeRunArguments target to discover run commands failed for this project. Fix the errors and warnings and run again.

I tried it locally, and this is what happens:

Exit code: 1 Standard output: Using Xcode 26.4.1 found in /Applications/Xcode_26.4.1.app/Contents/Developer Running in dotnet test mode... xcrun simctl list --json --json-output /var/folders/wh/sk44ynps73x6w9_ktl99dr7c0000gp/T/tmp12ZLGx.tmp Target simulator: iPad Air 11-inch (M4) (2F869A26-03F4-4965-A010-FD33C6C2D994) (2F869A26-03F4-4965-A010-FD33C6C2D994) App path: bin/Debug/net11.0-ios/iossimulator-arm64/iostest.app/ Bundle identifier: com.companyname.iostest dotnet test mode (pipe: /tmp/7f82dab60eaf49cd8ee2f1768f441d70) [RemoteTestAdapter] Installing app: bin/Debug/net11.0-ios/iossimulator-arm64/iostest.app/ xcrun simctl install 2F869A26-03F4-4965-A010-FD33C6C2D994 bin/Debug/net11.0-ios/iossimulator-arm64/iostest.app/ Error output: An error was encountered processing the command (domain=com.apple.CoreSimulator.SimError, code=405): Unable to lookup in current state: Shutdown error HE1045: Failed to execute 'simctl': 'simctl install 2F869A26-03F4-4965-A010-FD33C6C2D994 bin/Debug/net11.0-ios/iossimulator-arm64/iostest.app/' returned the exit code 149. error HE1046: Additional output: xcrun simctl install 2F869A26-03F4-4965-A010-FD33C6C2D994 bin/Debug/net11.0-ios/iossimulator-arm64/iostest.app/ error HE1046: Additional output: An error was encountered processing the command (domain=com.apple.CoreSimulator.SimError, code=405): error HE1046: Additional output: Unable to lookup in current state: Shutdown Test run summary: Zero tests ran error: 1 total: 0 failed: 0 succeeded: 0 skipped: 0 duration: 4s 175ms Test run completed with non-success exit code: 1 (see: https://aka.ms/testingplatform/exitcodes)

The simulator has to be launched before you can install apps into it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-mobiletools-engineering-service2 · 2026-05-18T15:56:10Z

✅ [CI Build #3885b03] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 3885b03cf886c807bf89f818d5dff9d0855f2184 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-18T15:56:14Z

✅ [PR Build #3885b03] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 3885b03cf886c807bf89f818d5dff9d0855f2184 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-18T16:00:10Z

✅ [CI Build #3885b03] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 3885b03cf886c807bf89f818d5dff9d0855f2184 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-18T16:13:16Z

✅ API diff for current PR / commit

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

✅ API diff vs stable

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 3885b03cf886c807bf89f818d5dff9d0855f2184 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-18T16:20:54Z

✅ [CI Build #3885b03] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 3885b03cf886c807bf89f818d5dff9d0855f2184 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-18T18:55:07Z

🔥 [CI Build #3885b03] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

1 tests crashed, 28 tests failed, 128 tests passed.

Failures

❌ dotnettests tests (iOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(iOS,"iostest"): 'dotnet test' failed with exit code 1.
  Binlog: /Users/builder/azdo/_work/3/s/macios/tests/dotnet/UnitTests/bin/Debug/net11.0/t...

Html Report (VSDrops) Download

❌ dotnettests tests (tvOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(TVOS,"tvostest"): 'dotnet test' failed with exit code 1.
  Binlog: /Users/builder/azdo/_work/4/s/macios/tests/dotnet/UnitTests/bin/Debug/net11.0/t...

Html Report (VSDrops) Download

❌ introspection tests

6 tests failed, 3 tests passed.

Failed tests

introspection/iOS - simulator/Debug: LaunchTimedOut
introspection/iOS - simulator/CoreCLR: LaunchTimedOut
introspection/iOS - simulator/Debug (ARM64): LaunchTimedOut
introspection/tvOS - simulator/Debug: LaunchTimedOut
introspection/tvOS - simulator/CoreCLR: LaunchTimedOut
introspection/tvOS - simulator/Debug (ARM64): LaunchTimedOut

Html Report (VSDrops) Download

❌ monotouch tests (iOS)

19 tests failed, 0 tests passed.

Failed tests

monotouch-test/iOS - simulator/Debug: LaunchTimedOut
monotouch-test/iOS - simulator/Debug (CoreCLR): LaunchTimedOut
monotouch-test/iOS - simulator/Release (CoreCLR, ARM64): LaunchTimedOut
monotouch-test/iOS - simulator/Release (CoreCLR, x64): LaunchTimedOut
monotouch-test/iOS - simulator/Release (CoreCLR, Universal): LaunchTimedOut
monotouch-test/iOS - simulator/Debug (LinkSdk): LaunchTimedOut
monotouch-test/iOS - simulator/Debug (static registrar): LaunchTimedOut
monotouch-test/iOS - simulator/Release (all optimizations): LaunchTimedOut
monotouch-test/iOS - simulator/Debug (ARM64): LaunchTimedOut
monotouch-test/iOS - simulator/Release (NativeAOT, ARM64): LaunchTimedOut
monotouch-test/iOS - simulator/Release (trimmable static registrar, NativeAOT, ARM64): LaunchTimedOut
monotouch-test/iOS - simulator/Debug (managed static registrar): LaunchTimedOut
monotouch-test/iOS - simulator/Debug (trimmable static registrar): LaunchTimedOut
monotouch-test/iOS - simulator/Release (managed static registrar, all optimizations): LaunchTimedOut
monotouch-test/iOS - simulator/Release (trimmable static registrar, all optimizations): LaunchTimedOut
monotouch-test/iOS - simulator/Release (NativeAOT, x64): LaunchTimedOut
monotouch-test/iOS - simulator/Release (trimmable static registrar, NativeAOT, x64): LaunchTimedOut
monotouch-test/iOS - simulator/Debug (interpreter): LaunchTimedOut
monotouch-test/iOS - simulator/Release (interpreter): LaunchTimedOut

Html Report (VSDrops) Download

❌ monotouch tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - monotouch_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ Tests on macOS Tahoe (26) tests

1 tests failed, 4 tests passed.

Failed tests

monotouch-test: Failed (exit code 2)
- TestNSurlSessionHandlerCookieContainerSetCookie : Cookies received from server.

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ linker: All 44 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 18 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 19 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 3885b03cf886c807bf89f818d5dff9d0855f2184 [PR build]

jonathanpeppers changed the base branch from main to net11.0 May 4, 2026 21:58

This comment has been minimized.

Sign in to view

rolfbjarne mentioned this pull request May 11, 2026

[CI Postmortem] Flaky: TestNSUrlSessionEphemeralHandlerCookies on iOS/MacCatalyst #25384

Closed

jonathanpeppers and others added 2 commits May 12, 2026 12:33

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch 2 times, most recently from 32ce330 to 5c37575 Compare May 12, 2026 17:34

Bump Microsoft.Tools.Mlaunch to 1.1.115

4c9418b

Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch from 5c37575 to 4c9418b Compare May 12, 2026 17:35

This comment has been minimized.

Sign in to view

This was referenced May 18, 2026

[CI Postmortem] Widespread LaunchTimedOut failures on iOS/tvOS simulator tests #25299

Open

[CI Postmortem] Flaky: NSPasteboardTests.DetectMetadataTests_SomewhatStronglyTyped on macOS #25446

Closed

rolfbjarne reviewed May 18, 2026

View reviewed changes

Add /bl: to dotnet test for CI diagnostics

3885b03

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This comment has been minimized.

Sign in to view

Conversation

jonathanpeppers commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Screenshot

Related

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rolfbjarne May 18, 2026

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers May 18, 2026

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers May 18, 2026

Choose a reason for hiding this comment

Uh oh!

rolfbjarne May 19, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 18, 2026

✅ [CI Build #3885b03] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 18, 2026

✅ [PR Build #3885b03] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 18, 2026

✅ [CI Build #3885b03] Build passed (Detect API changes) ✅

Uh oh!

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 18, 2026

✅ API diff for current PR / commit

✅ API diff vs stable

ℹ️ Generator diff

Uh oh!

vs-mobiletools-engineering-service2 commented May 18, 2026

✅ [CI Build #3885b03] Build passed (Build macOS tests) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 18, 2026

🔥 [CI Build #3885b03] Test results 🔥

Test results

jonathanpeppers commented May 4, 2026 •

edited

Loading