Enable NativeAOT library tests on Apple mobile platforms#125437
Enable NativeAOT library tests on Apple mobile platforms#125437kotlarmilos wants to merge 22 commits intodotnet:mainfrom
Conversation
Replace NativeAOT runtime test legs with library test legs on iOS/tvOS devices, iOS/tvOS simulators, and Mac Catalyst. The runtime tests (nativeaot/SmokeTests) provide minimal value on mobile since the compiler doesn't meaningfully distinguish between iOS and macOS. Library tests exercise OS-specific APIs (crypto, networking, etc.) that are the real quality gate. Changes: - ioslike: remove NativeAOT runtime test leg, update existing library test leg to use dynamic smoke test arg - ioslikesimulator: replace NativeAOT runtime test leg with library test leg, fix isiOSLikeSimulatorOnlyBuild parameter - maccatalyst: replace both NativeAOT runtime test legs (regular + AppSandbox) with library test legs, fix coreclrContainsChange variable (was incorrectly using monoContainsChange) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the Apple “extra platforms” CI legs to validate NativeAOT on iOS/tvOS devices, iOS/tvOS simulators, and Mac Catalyst by running libraries tests (via Helix) instead of the minimal NativeAOT runtime SmokeTests, improving OS-specific coverage (crypto/networking/globalization) on mobile.
Changes:
- Switch NativeAOT Apple-mobile legs from runtime SmokeTests submission to
eng/pipelines/libraries/helix.ymlsubmission of libraries tests. - Use
eng/pipelines/libraries/helix-queues-setup.ymland passNeedsToBuildAppsOnHelix=truefor the new legs. - Update naming/args to use
$(_runSmokeTestsOnlyArg)and fix path-evaluation variable naming in affected legs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml | Replaces NativeAOT runtime SmokeTests legs with libraries test legs (including App Sandbox variant) using libraries Helix templates/queues. |
| eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml | Replaces NativeAOT runtime SmokeTests on iOS/tvOS simulators with libraries tests using libraries Helix templates/queues. |
| eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml | Adjusts NativeAOT iOS/tvOS device leg to use $(_runSmokeTestsOnlyArg) for libraries tests and removes the old NativeAOT runtime SmokeTests device leg. |
You can also share your feedback on Copilot code review. Take the survey.
| runtimeFlavor: coreclr | ||
| isExtraPlatformsBuild: ${{ parameters.isExtraPlatformsBuild }} | ||
| isiOSLikeOnlyBuild: ${{ parameters.isiOSLikeOnlyBuild }} | ||
| isiOSLikeSimulatorOnlyBuild: ${{ parameters.isiOSLikeSimulatorOnlyBuild }} | ||
| platforms: |
There was a problem hiding this comment.
isiOSLikeOnlyBuild appears to have been renamed to isiOSLikeSimulatorOnlyBuild for this template, but this file still references parameters.isiOSLikeOnlyBuild in other platform-matrix invocations (e.g., the Mono/CoreCLR runtime-test legs). Since isiOSLikeOnlyBuild is no longer declared in this file's parameters: block, those references will either fail template evaluation or silently pass an empty value. Update the remaining ${{ parameters.isiOSLikeOnlyBuild }} usages to ${{ parameters.isiOSLikeSimulatorOnlyBuild }} (or remove the parameter entirely if it’s unused by the downstream templates).
…60312.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26162.1
…60312.3 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26162.3
…60314.3 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26164.3
…60317.2 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26167.2
…60318.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26168.1
…bb6-53258cb4f260' into enable-nativeaot-libs-tests-mobile
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
You can also share your feedback on Copilot code review. Take the survey.
Library test DLLs have no Main() method, causing ILC to throw 'No entrypoint module' when --splitinit is active. Add AppleTestRunner.dll (which is an EXE with Main) as an additional ILC input to provide the entrypoint, and add published DLLs as ILC references for non-framework dependency resolution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run runtime-ioslike,runtime-ioslikesimulator,runtime-maccatalyst |
|
Azure Pipelines successfully started running 3 pipeline(s). |
When IncludesTestRunner is true with NativeAOT, the AppleTestRunner P/Invokes mono_ios_append_output and mono_ios_set_summary which are only defined in main-console.m. The UseConsoleUITemplate condition previously excluded all NativeAOT builds, causing main-simple.m to be used instead, resulting in undefined symbol linker errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| <IlcCompileInput Include="$(OriginalPublishDir)AppleTestRunner.dll" /> | ||
| <IlcReference Include="$(OriginalPublishDir)*.dll" | ||
| Exclude="$(OriginalPublishDir)AppleTestRunner.dll;$(OriginalPublishDir)$(AssemblyName).dll" /> |
There was a problem hiding this comment.
_SetupNativeAOTLibraryTestInputs hard-codes AppleTestRunner.dll for IlcCompileInput/IlcReference include/exclude. Since MainLibraryFileName is configurable (it’s only defaulted to AppleTestRunner.dll when empty), this target should use $(MainLibraryFileName) (and exclude that) to avoid breaking builds that override the runner assembly name.
| <IlcCompileInput Include="$(OriginalPublishDir)AppleTestRunner.dll" /> | |
| <IlcReference Include="$(OriginalPublishDir)*.dll" | |
| Exclude="$(OriginalPublishDir)AppleTestRunner.dll;$(OriginalPublishDir)$(AssemblyName).dll" /> | |
| <IlcCompileInput Include="$(OriginalPublishDir)$(MainLibraryFileName)" /> | |
| <IlcReference Include="$(OriginalPublishDir)*.dll" | |
| Exclude="$(OriginalPublishDir)$(MainLibraryFileName);$(OriginalPublishDir)$(AssemblyName).dll" /> |
In NativeAOT mode, test assemblies are statically linked into the native
binary and no DLL files exist on disk. The file-based discovery
(Directory.GetFiles("*.Tests.dll")) finds nothing, causing the runner to
exit with 'Test libs were not found' before tests can execute.
Add a NativeAOT-aware discovery path that uses
AppDomain.CurrentDomain.GetAssemblies() to find test assemblies matching
*.Tests when RuntimeFeature.IsDynamicCodeSupported is false. Update
GetTestAssemblies() to return the already-loaded Assembly objects directly
instead of attempting Assembly.LoadFrom() on non-existent files.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…obile ILC trims types not reachable from the entrypoint. Library test types are only discovered via xunit reflection, so they were trimmed away, causing AppDomain.CurrentDomain.GetAssemblies() to not return the test assembly. Add TrimmerRootAssembly items for the test assembly, test runner, and xunit framework assemblies in _SetupNativeAOTLibraryTestInputs so ILC preserves them (--root: arguments). Add diagnostic logging to AppleTestRunner to dump all loaded assemblies when NativeAOT test discovery runs, making future issues easier to debug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| if (!RuntimeFeature.IsDynamicCodeSupported) | ||
| { | ||
| // NativeAOT: test assemblies are statically linked into the native binary, | ||
| // so there are no DLL files on disk. Discover them from loaded assemblies. | ||
| Console.WriteLine("NativeAOT mode: discovering test assemblies from loaded assemblies."); | ||
| Assembly[] allAssemblies = AppDomain.CurrentDomain.GetAssemblies(); | ||
| Console.WriteLine($"Loaded assemblies ({allAssemblies.Length}):"); | ||
| foreach (Assembly a in allAssemblies) | ||
| { | ||
| Console.WriteLine($" {a.GetName().Name}"); | ||
| } | ||
|
|
||
| s_testAssemblies = allAssemblies | ||
| .Where(a => a.GetName().Name?.EndsWith(".Tests") == true) | ||
| .ToList(); | ||
| s_testLibs = s_testAssemblies.Select(a => a.GetName().Name!).ToList(); | ||
| } |
There was a problem hiding this comment.
Using RuntimeFeature.IsDynamicCodeSupported to detect NativeAOT is unreliable on Apple platforms (it can be false for non-NativeAOT AOT/interpreter scenarios too). This can cause the runner to skip the on-disk *.Tests.dll discovery and then fail to find/load tests. Consider detecting NativeAOT by first checking for any *.Tests.dll files on disk and only falling back to assembly-based loading when none exist (or attempt Assembly.Load for discovered/arg-specified assemblies).
| Console.WriteLine($"Loaded assemblies ({allAssemblies.Length}):"); | ||
| foreach (Assembly a in allAssemblies) | ||
| { | ||
| Console.WriteLine($" {a.GetName().Name}"); | ||
| } |
There was a problem hiding this comment.
The runner prints the full list of loaded assemblies unconditionally in the new NativeAOT path. This can create very large logs in CI/device runs and make failures harder to triage. Consider gating this behind --verbose (or a dedicated flag) and keep the default output to a short summary.
| Console.WriteLine($"Loaded assemblies ({allAssemblies.Length}):"); | |
| foreach (Assembly a in allAssemblies) | |
| { | |
| Console.WriteLine($" {a.GetName().Name}"); | |
| } | |
| if (verbose) | |
| { | |
| Console.WriteLine($"Loaded assemblies ({allAssemblies.Length}):"); | |
| foreach (Assembly a in allAssemblies) | |
| { | |
| Console.WriteLine($" {a.GetName().Name}"); | |
| } | |
| } | |
| else | |
| { | |
| Console.WriteLine($"Discovered {allAssemblies.Length} loaded assemblies."); | |
| } |
…le mobile ILC trims types not reachable from the entrypoint. Library test types are only discovered via xunit reflection, so they were trimmed away, causing AppDomain.CurrentDomain.GetAssemblies() to not return the test assembly. Add TrimmerRootAssembly items for the test assembly, test runner, and xunit framework assemblies in _SetupNativeAOTLibraryTestInputs so ILC preserves them (--root: arguments). Resolve IgnoredTraitsFilePath against AppContext.BaseDirectory instead of using a relative path, since Environment.CurrentDirectory is / on iOS under NativeAOT while the xunit-excludes.txt file lives in the app bundle. Add diagnostic logging to AppleTestRunner to dump all loaded assemblies when NativeAOT test discovery runs, making future issues easier to debug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Xcode.cs blanket-excludes .txt files from NativeAOT app bundles, so xunit-excludes.txt is not present at runtime. Return empty string from IgnoredTraitsFilePath when the file does not exist to skip trait filtering instead of crashing. Also resolve the path against AppContext.BaseDirectory since Environment.CurrentDirectory is / on iOS under NativeAOT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| if (!RuntimeFeature.IsDynamicCodeSupported) | ||
| { | ||
| // NativeAOT: test assemblies are statically linked into the native binary, | ||
| // so there are no DLL files on disk. Discover them from loaded assemblies. | ||
| Console.WriteLine("NativeAOT mode: discovering test assemblies from loaded assemblies."); | ||
| Assembly[] allAssemblies = AppDomain.CurrentDomain.GetAssemblies(); | ||
| Console.WriteLine($"Loaded assemblies ({allAssemblies.Length}):"); | ||
| foreach (Assembly a in allAssemblies) | ||
| { | ||
| Console.WriteLine($" {a.GetName().Name}"); | ||
| } | ||
|
|
||
| s_testAssemblies = allAssemblies | ||
| .Where(a => a.GetName().Name?.EndsWith(".Tests") == true) | ||
| .ToList(); | ||
| s_testLibs = s_testAssemblies.Select(a => a.GetName().Name!).ToList(); | ||
| } | ||
| else | ||
| { | ||
| // Look for *.Tests.dll files if target test suites are not set via "testlib:" arguments | ||
| s_testLibs = Directory.GetFiles(Environment.CurrentDirectory, "*.Tests.dll").ToList(); | ||
| } |
There was a problem hiding this comment.
RuntimeFeature.IsDynamicCodeSupported is not a reliable way to detect “NativeAOT mode” here (it will also be false for other AOT configurations). In those cases this branch skips the on-disk *.Tests.dll discovery and may fail to find any tests because the assemblies are not yet loaded. Consider attempting file-based discovery first (as before) and only falling back to reflection-based discovery when no *.Tests.dll files are present; for the fallback, prefer loading candidate assemblies (e.g., from referenced assemblies) rather than relying on AppDomain.CurrentDomain.GetAssemblies() alone.
| // NativeAOT excludes .txt files from the app bundle (Xcode.cs predefinedExcludes), | ||
| // so xunit-excludes.txt may not be present. Return empty string to skip trait filtering. | ||
| protected override string IgnoredTraitsFilePath | ||
| { | ||
| get | ||
| { | ||
| string path = Path.Combine(AppContext.BaseDirectory, "xunit-excludes.txt"); | ||
| return File.Exists(path) ? path : string.Empty; |
There was a problem hiding this comment.
Returning string.Empty here effectively disables xUnit trait exclusion when xunit-excludes.txt is missing. Given the file is intentionally generated during test bundling, skipping exclusions may cause known-bad traits/tests to run under NativeAOT. Consider addressing this by ensuring xunit-excludes.txt is included in the NativeAOT app bundle (or passing the excludes via args/env) rather than bypassing trait filtering.
| // NativeAOT excludes .txt files from the app bundle (Xcode.cs predefinedExcludes), | |
| // so xunit-excludes.txt may not be present. Return empty string to skip trait filtering. | |
| protected override string IgnoredTraitsFilePath | |
| { | |
| get | |
| { | |
| string path = Path.Combine(AppContext.BaseDirectory, "xunit-excludes.txt"); | |
| return File.Exists(path) ? path : string.Empty; | |
| // NativeAOT may exclude .txt files from the app bundle (Xcode.cs predefinedExcludes), | |
| // which can cause xunit-excludes.txt to be missing. Use an environment variable override | |
| // if provided, otherwise fall back to the bundle path. If no excludes file is found, | |
| // return the expected bundle path and log a warning so trait filtering is not silently | |
| // disabled and bundling/configuration issues are visible. | |
| protected override string IgnoredTraitsFilePath | |
| { | |
| get | |
| { | |
| const string excludesFileName = "xunit-excludes.txt"; | |
| const string excludesEnvVarName = "XUNIT_EXCLUDES_FILE"; | |
| string? envPath = Environment.GetEnvironmentVariable(excludesEnvVarName); | |
| if (envPath is not null && envPath.Length > 0 && File.Exists(envPath)) | |
| { | |
| return envPath; | |
| } | |
| string bundlePath = Path.Combine(AppContext.BaseDirectory, excludesFileName); | |
| if (File.Exists(bundlePath)) | |
| { | |
| return bundlePath; | |
| } | |
| Console.Error.WriteLine($"[AppleTestRunner] Warning: xUnit excludes file '{bundlePath}' was not found. " + | |
| $"Set the '{excludesEnvVarName}' environment variable to a valid file path or " + | |
| "ensure the file is included in the app bundle to enable trait filtering."); | |
| return bundlePath; |
| testGroup: innerloop | ||
| nameSuffix: AllSubsets_NativeAOT_RuntimeTests | ||
| nameSuffix: AllSubsets_NativeAOT | ||
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true | ||
| timeoutInMinutes: 180 |
There was a problem hiding this comment.
$(_runSmokeTestsOnlyArg) expands to /p:RunSmokeTestsOnly=$(isRunSmokeTestsOnly). For the dedicated runtime-ioslikesimulator / runtime-extra-platforms pipelines, isRunSmokeTestsOnly is false (per eng/pipelines/common/variables.yml), so this will run non-smoke library tests. If the intent is to keep this leg smoke-only (as the previous NativeAOT runtime leg effectively was), set RunSmokeTestsOnly=true explicitly or adjust timeouts/naming accordingly.
| nameSuffix: AllSubsets_NativeAOT_RuntimeTests | ||
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs -c $(_BuildConfig) | ||
| nameSuffix: AllSubsets_NativeAOT | ||
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:DevTeamProvisioning=adhoc /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true |
There was a problem hiding this comment.
$(_runSmokeTestsOnlyArg) expands to /p:RunSmokeTestsOnly=$(isRunSmokeTestsOnly). In the dedicated runtime-maccatalyst / runtime-extra-platforms pipelines, isRunSmokeTestsOnly is false, so this leg will run non-smoke library tests (potentially much larger than the previous NativeAOT runtime smoke leg). If the intent is smoke-only coverage, set RunSmokeTestsOnly=true explicitly (and similarly for the AppSandbox variant) or adjust timeout/expectations.
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:DevTeamProvisioning=adhoc /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true | |
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=true /p:DevTeamProvisioning=adhoc /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true |
| nameSuffix: AllSubsets_NativeAOT_Smoke | ||
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=true /p:DevTeamProvisioning=- /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true | ||
| nameSuffix: AllSubsets_NativeAOT | ||
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:DevTeamProvisioning=- /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true |
There was a problem hiding this comment.
This switches from an explicit RunSmokeTestsOnly=true (previously) to $(_runSmokeTestsOnlyArg), which evaluates to RunSmokeTestsOnly=false in the runtime-ioslike / runtime-extra-platforms pipelines. That materially increases test scope and may exceed the current timeout or intended coverage for this leg. If the intent is still smoke-only NativeAOT validation, set RunSmokeTestsOnly=true explicitly (or update timeout/nameSuffix to reflect running full innerloop).
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:DevTeamProvisioning=- /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true | |
| buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=true /p:DevTeamProvisioning=- /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true |
The previous version (26168.1) predates the ReflectionBasedXunitTestRunner added in dotnet/xharness#1554. That runner uses reflection-based discovery (NativeAOT-compatible) instead of XunitFrontController which tries to load assemblies from disk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Description
Replace NativeAOT runtime test legs with library test legs on iOS/tvOS devices, iOS/tvOS simulators, and Mac Catalyst. Library tests validates OS-specific APIs (crypto, networking, globalization).
Depends on dotnet/xharness#1554
Fixes #81075