[tests] Upload expected app size files as artifacts on test failure

.github/skills/update-expected-app-size/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: update-expected-app-size
+description: >-
+  Download updated expected app size files from Azure DevOps CI artifacts for
+  the current branch. Use when app size tests fail in CI and the user wants to
+  update the expected files locally. Trigger on "update app size", "download
+  expected files", "fix app size test", or "update expected app size files".
+---
+
+# Update Expected App Size Files
+
+Download updated expected app size files from Azure DevOps artifacts for the current branch's PR build, and apply them to the repository.
+
+## When to Use
+
+- App size tests failed in CI and the user wants to update the expected files
+- User asks to "update app size", "download expected files", or "fix app size test failures"
+- User wants to pull the updated expected files from a recent CI build
+
+## Background
+
+The app size tests (`tests/dotnet/UnitTests/AppSizeTest.cs`) compare the built app's size and preserved APIs against expected files stored in `tests/dotnet/UnitTests/expected/`. When the test detects a difference and `WRITE_KNOWN_FAILURES` is not set, it writes the updated expected file to `$(Build.ArtifactStagingDirectory)/updated-expected-sizes/`, and a pipeline step publishes this directory as a build artifact.
+
+The artifact name follows the pattern `updated-expected-sizes-{testPrefix}-{attempt}` (e.g., `updated-expected-sizes-dotnettests_ios-1`). Inside the artifact, files are named:
+- `{Platform}-{Runtime}-size.txt` — e.g., `iOS-MonoVM-size.txt`
+- `{Platform}-{Runtime}-preservedapis.txt` — e.g., `iOS-MonoVM-preservedapis.txt`
+
+The expected files on disk are at:
+- `tests/dotnet/UnitTests/expected/{Platform}-{Runtime}-size.txt`
+- `tests/dotnet/UnitTests/expected/{Platform}-{Runtime}-preservedapis.txt`
+
+## Workflow
+
+### 1. Determine the current branch and PR
+
+```bash
+BRANCH=$(git branch --show-current)
+# Find the PR number for this branch
+gh pr list --head "$BRANCH" --repo dotnet/macios --json number,url --jq '.[0]'
+```
+
+If no PR is found, inform the user that this skill requires a PR to exist for the current branch (so that CI has run).
+
+### 2. Find the Azure DevOps build
+
+The CI builds for PRs in dotnet/macios run in the `devdiv` Azure DevOps organization, project `DevDiv`.
+
+Use the GitHub PR checks to find the Azure DevOps build URL:
+
+```bash
+gh pr checks <PR_NUMBER> --repo dotnet/macios
+```
+
+Look for a check that links to Azure DevOps. The build URL will look like:
+```
+https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=XXXXXXX
+```
+
+Extract the `buildId` from the URL.
+
+### 3. Download the artifacts
+
+Use the Azure DevOps REST API to list and download artifacts:
+
+```bash
+# List artifacts for the build
+TOKEN=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)
+curl -s "https://devdiv.visualstudio.com/DevDiv/_apis/build/builds/{buildId}/artifacts?api-version=7.0" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+Look for artifacts whose names contain `updated-expected-sizes` (e.g., `updated-expected-sizes-dotnettests_ios-1`). Get the artifact's `downloadUrl` and download it:
+
+```bash
+# Get the download URL for a specific artifact
+ARTIFACT_INFO=$(curl -s "https://devdiv.visualstudio.com/DevDiv/_apis/build/builds/{buildId}/artifacts?artifactName={artifactName}&api-version=7.0" \
+  -H "Authorization: Bearer $TOKEN")
+DOWNLOAD_URL=$(echo "$ARTIFACT_INFO" | python3 -c "import sys,json; print(json.load(sys.stdin)['resource']['downloadUrl'])")
+
+# Download the artifact zip
+curl -sL "$DOWNLOAD_URL" -H "Authorization: Bearer $TOKEN" -o artifact.zip
+```
+
+If `az` is not available or not authenticated, direct the user to download manually from the Azure DevOps build artifacts page.
+
+### 4. Place the files
+
+Extract the downloaded artifact zip and place the files in the expected directory:
+
+```bash
+EXPECTED_DIR="tests/dotnet/UnitTests/expected"
+unzip -o artifact.zip -d /tmp/updated-sizes/
+cp /tmp/updated-sizes/*/*.txt "$EXPECTED_DIR/"
+```
+
+The files inside the zip already have the correct names (e.g., `iOS-MonoVM-size.txt`) and can be copied directly.
+
+### 5. Verify and commit
+
+After placing the files:
+1. Run `git diff` to show what changed
+2. Ask the user if the changes look correct
+3. If confirmed, commit the changes:
+   ```bash
+   git add tests/dotnet/UnitTests/expected/
+   git commit -m "[tests] Update expected app size files"
+   ```
+
+## Fallback: Manual Download
+
+If automated download fails (auth issues, etc.), provide the user with:
+1. The Azure DevOps build URL
+2. Instructions to navigate to the build → Summary → Artifacts section
+3. Look for individual artifacts whose names match the patterns above
+4. Download each file and place it as `tests/dotnet/UnitTests/expected/{artifactName}.txt`
+
+## Fallback: Run Locally
+
+If the user can build locally, they can update the expected files directly:
+
+```bash
+WRITE_KNOWN_FAILURES=1 tests-dotnet AppSizeTest
+```
+
+This runs the tests, updates the expected files in place, and marks the tests as passed.
+

tests/dotnet/UnitTests/AppSizeTest.cs

@@ -114,7 +114,8 @@ void Run (ApplePlatform platform, string runtimeIdentifiers, string configuratio
 			DotNet.AssertBuild (project_path, properties);
 
 			// FORCE_UPDATE_KNOWN_FAILURES will update the known failures files even if the test doesn't actually fail
-			// WRITE_KNOWN_FAILURES will only update the known failures files if the test fails
+			// WRITE_KNOWN_FAILURES will only update the known failures files if the test fails (and mark the test as passed)
+			// If neither is set, the updated expected file is uploaded as an Azure DevOps artifact.
 
 			var forceUpdate = !string.IsNullOrEmpty (Environment.GetEnvironmentVariable ("FORCE_UPDATE_KNOWN_FAILURES"));
 			var update = forceUpdate || !string.IsNullOrEmpty (Environment.GetEnvironmentVariable ("WRITE_KNOWN_FAILURES"));
@@ -167,39 +168,49 @@ static void AssertAppSize (ApplePlatform platform, string name, string appPath,
 				msg = $"App size changed significantly ({FormatBytes (appSizeDifference, true)} different > tolerance of +-{FormatBytes (toleranceInBytes)}). Expected app size: {FormatBytes (expectedAppBundleSize)}, actual app size: {FormatBytes (appBundleSize)}.";
 			}
 
-			var updated = false;
-			if (forceUpdate || (update && !withinTolerance)) {
-				Directory.CreateDirectory (expectedDirectory);
-				File.WriteAllText (expectedSizeReportPath, report.ToString ());
-				msg += " Check the modified files for more information.";
-				updated = true;
-			} else if (!withinTolerance) {
-				msg += " Set the environment variable WRITE_KNOWN_FAILURES=1, run the test again, and verify the modified files for more information.";
-			}
-
 			Console.WriteLine ($"    {msg}");
 
+			// Compare individual files in the app bundle
 			var expectedLines = expectedSizeReport.SplitLines ().Skip (2).Where (v => v.IndexOf (':') >= 0).ToDictionary (v => v [..v.IndexOf (':')], v => v [(v.IndexOf (':') + 1)..]);
 			var actualLines = report.ToString ().SplitLines ().Skip (2).Where (v => v.IndexOf (':') >= 0).ToDictionary (v => v [..v.IndexOf (':')], v => v [(v.IndexOf (':') + 1)..]);
 			var allKeys = expectedLines.Keys.Union (actualLines.Keys).OrderBy (v => v);
+			var filesAdded = new List<string> ();
+			var filesRemoved = new List<string> ();
 			foreach (var key in allKeys) {
 				if (!expectedLines.TryGetValue (key, out var expectedLine)) {
 					Console.WriteLine ($"        File '{key}' was added to app bundle: {actualLines [key]}");
-					if (!updated)
-						Assert.Fail ($"The file '{key}' was added to the app bundle.");
+					filesAdded.Add (key);
 				} else if (!actualLines.TryGetValue (key, out var actualLine)) {
 					Console.WriteLine ($"        File '{key}' was removed from app bundle: {expectedLine}");
-					if (!updated)
-						Assert.Fail ($"The file '{key}' was removed from the app bundle.");
+					filesRemoved.Add (key);
 				} else if (expectedLine != actualLine) {
 					Console.WriteLine ($"        File '{key}' changed in app bundle:");
 					Console.WriteLine ($"            -{expectedLine}");
 					Console.WriteLine ($"            +{actualLine}");
 				}
 			}
 
-			if (!updated && !withinTolerance)
-				Assert.Fail (msg);
+			// Determine if there are any meaningful differences
+			var hasFileDifferences = filesAdded.Count > 0 || filesRemoved.Count > 0;
+			var hasSizeDifference = !withinTolerance;
+			var hasDifferences = hasFileDifferences || hasSizeDifference;
+
+			if (forceUpdate || (update && hasDifferences)) {
+				Directory.CreateDirectory (expectedDirectory);
+				File.WriteAllText (expectedSizeReportPath, report.ToString ());
+				Console.WriteLine ($"    Updated expected file: {expectedSizeReportPath}");
+			} else if (hasDifferences) {
+				UploadUpdatedExpectedFile (expectedSizeReportPath, report.ToString ());
+				if (hasFileDifferences) {
+					var details = new List<string> ();
+					foreach (var key in filesAdded)
+						details.Add ($"added: '{key}'");
+					foreach (var key in filesRemoved)
+						details.Add ($"removed: '{key}'");
+					Assert.Fail ($"The app bundle's file list changed ({string.Join (", ", details)}). The updated expected file is available as a build artifact (set WRITE_KNOWN_FAILURES=1 to update locally).");
+				}
+				Assert.Fail ($"{msg} The updated expected file is available as a build artifact (set WRITE_KNOWN_FAILURES=1 to update locally).");
+			}
 		}
 
 		// Create a file with all the APIs that survived the trimmer; this can be useful to determine what is not trimmed away.
@@ -238,9 +249,29 @@ void AssertAssemblyReport (ApplePlatform platform, string name, string appPath,
 			}
 
 			if (!update) {
-				Assert.That (addedAPIs, Is.Empty, "No added APIs (set the environment variable WRITE_KNOWN_FAILURES=1 and run the test again to update the expected set of APIs)");
-				Assert.That (removedAPIs, Is.Empty, "No removed APIs (set the environment variable WRITE_KNOWN_FAILURES=1 and run the test again to update the expected set of APIs)");
+				if (addedAPIs.Count > 0 || removedAPIs.Count > 0) {
+					UploadUpdatedExpectedFile (expectedFile, string.Join ('\n', preservedAPIs) + "\n");
+					var updateMsg = " The updated expected file is available as a build artifact (set WRITE_KNOWN_FAILURES=1 to update locally).";
+					Assert.That (addedAPIs, Is.Empty, "No added APIs." + updateMsg);
+					Assert.That (removedAPIs, Is.Empty, "No removed APIs." + updateMsg);
+				}
+			}
+		}
+
+		static void UploadUpdatedExpectedFile (string expectedFilePath, string content)
+		{
+			var fileName = Path.GetFileName (expectedFilePath);
+			var artifactStagingDir = Environment.GetEnvironmentVariable ("BUILD_ARTIFACTSTAGINGDIRECTORY");
+			string outputDir;
+			if (!string.IsNullOrEmpty (artifactStagingDir)) {
+				outputDir = Path.Combine (artifactStagingDir, "updated-expected-sizes");
+			} else {
+				outputDir = Path.Combine (Cache.CreateTemporaryDirectory ("AppSizeTest"), "updated-expected-sizes");
 			}
+			Directory.CreateDirectory (outputDir);
+			var outputFile = Path.Combine (outputDir, fileName);
+			File.WriteAllText (outputFile, content);
+			Console.WriteLine ($"    Updated expected file written to: {outputFile}");
 		}
 
 		static string FormatBytes (long bytes, bool alwaysShowSign = false)

tests/dotnet/UnitTests/expected/MacCatalyst-MonoVM-size.txt

@@ -1,10 +1,10 @@
-AppBundleSize: 16,325,774 bytes (15,943.1 KB = 15.6 MB)
+AppBundleSize: 16,321,420 bytes (15,938.9 KB = 15.6 MB)
 # The following list of files and their sizes is just informational / for review, and isn't used in the test:
 Contents/_CodeSignature/CodeResources: 4,134 bytes (4.0 KB = 0.0 MB)
-Contents/Info.plist: 1,119 bytes (1.1 KB = 0.0 MB)
-Contents/MacOS/SizeTestApp: 13,814,640 bytes (13,490.9 KB = 13.2 MB)
+Contents/Info.plist: 1,085 bytes (1.1 KB = 0.0 MB)
+Contents/MacOS/SizeTestApp: 13,810,240 bytes (13,486.6 KB = 13.2 MB)
 Contents/MonoBundle/aot-instances.aotdata.arm64: 1,045,032 bytes (1,020.5 KB = 1.0 MB)
-Contents/MonoBundle/Microsoft.MacCatalyst.aotdata.arm64: 36,288 bytes (35.4 KB = 0.0 MB)
+Contents/MonoBundle/Microsoft.MacCatalyst.aotdata.arm64: 36,368 bytes (35.5 KB = 0.0 MB)
 Contents/MonoBundle/Microsoft.MacCatalyst.dll: 50,688 bytes (49.5 KB = 0.0 MB)
 Contents/MonoBundle/runtimeconfig.bin: 1,481 bytes (1.4 KB = 0.0 MB)
 Contents/MonoBundle/SizeTestApp.aotdata.arm64: 1,552 bytes (1.5 KB = 0.0 MB)

tests/dotnet/UnitTests/expected/MacOSX-CoreCLR-Interpreter-TrimmableStatic-size.txt

@@ -1,12 +1,12 @@
-AppBundleSize: 260,131,912 bytes (254,035.1 KB = 248.1 MB)
+AppBundleSize: 260,129,862 bytes (254,033.1 KB = 248.1 MB)
 # The following list of files and their sizes is just informational / for review, and isn't used in the test:
 Contents/_CodeSignature/CodeResources: 67,868 bytes (66.3 KB = 0.1 MB)
-Contents/Info.plist: 750 bytes (0.7 KB = 0.0 MB)
-Contents/MacOS/SizeTestApp: 7,431,792 bytes (7,257.6 KB = 7.1 MB)
+Contents/Info.plist: 716 bytes (0.7 KB = 0.0 MB)
+Contents/MacOS/SizeTestApp: 7,430,800 bytes (7,256.6 KB = 7.1 MB)
 Contents/MonoBundle/.xamarin/osx-arm64/_Microsoft.macOS.TypeMap.dll: 4,847,616 bytes (4,734.0 KB = 4.6 MB)
 Contents/MonoBundle/.xamarin/osx-arm64/_SizeTestApp.TypeMap.dll: 3,072 bytes (3.0 KB = 0.0 MB)
 Contents/MonoBundle/.xamarin/osx-arm64/Microsoft.CSharp.dll: 893,192 bytes (872.3 KB = 0.9 MB)
-Contents/MonoBundle/.xamarin/osx-arm64/Microsoft.macOS.dll: 38,524,416 bytes (37,621.5 KB = 36.7 MB)
+Contents/MonoBundle/.xamarin/osx-arm64/Microsoft.macOS.dll: 38,523,904 bytes (37,621.0 KB = 36.7 MB)
 Contents/MonoBundle/.xamarin/osx-arm64/Microsoft.VisualBasic.Core.dll: 1,335,096 bytes (1,303.8 KB = 1.3 MB)
 Contents/MonoBundle/.xamarin/osx-arm64/Microsoft.VisualBasic.dll: 17,680 bytes (17.3 KB = 0.0 MB)
 Contents/MonoBundle/.xamarin/osx-arm64/Microsoft.Win32.Primitives.dll: 16,144 bytes (15.8 KB = 0.0 MB)
@@ -182,7 +182,7 @@ Contents/MonoBundle/.xamarin/osx-arm64/WindowsBase.dll: 16,688 bytes (16.3 KB =
 Contents/MonoBundle/.xamarin/osx-x64/_Microsoft.macOS.TypeMap.dll: 4,847,616 bytes (4,734.0 KB = 4.6 MB)
 Contents/MonoBundle/.xamarin/osx-x64/_SizeTestApp.TypeMap.dll: 3,072 bytes (3.0 KB = 0.0 MB)
 Contents/MonoBundle/.xamarin/osx-x64/Microsoft.CSharp.dll: 796,432 bytes (777.8 KB = 0.8 MB)
-Contents/MonoBundle/.xamarin/osx-x64/Microsoft.macOS.dll: 38,524,416 bytes (37,621.5 KB = 36.7 MB)
+Contents/MonoBundle/.xamarin/osx-x64/Microsoft.macOS.dll: 38,523,904 bytes (37,621.0 KB = 36.7 MB)
 Contents/MonoBundle/.xamarin/osx-x64/Microsoft.VisualBasic.Core.dll: 1,166,648 bytes (1,139.3 KB = 1.1 MB)
 Contents/MonoBundle/.xamarin/osx-x64/Microsoft.VisualBasic.dll: 17,680 bytes (17.3 KB = 0.0 MB)
 Contents/MonoBundle/.xamarin/osx-x64/Microsoft.Win32.Primitives.dll: 16,176 bytes (15.8 KB = 0.0 MB)

tests/dotnet/UnitTests/expected/TVOS-MonoVM-interpreter-size.txt

@@ -1,12 +1,12 @@
-AppBundleSize: 3,616,438 bytes (3,531.7 KB = 3.4 MB)
+AppBundleSize: 3,616,596 bytes (3,531.8 KB = 3.4 MB)
 # The following list of files and their sizes is just informational / for review, and isn't used in the test:
 _CodeSignature/CodeResources: 3,999 bytes (3.9 KB = 0.0 MB)
 archived-expanded-entitlements.xcent: 384 bytes (0.4 KB = 0.0 MB)
-Info.plist: 1,138 bytes (1.1 KB = 0.0 MB)
+Info.plist: 1,104 bytes (1.1 KB = 0.0 MB)
 Microsoft.tvOS.dll: 154,624 bytes (151.0 KB = 0.1 MB)
 PkgInfo: 8 bytes (0.0 KB = 0.0 MB)
 runtimeconfig.bin: 1,405 bytes (1.4 KB = 0.0 MB)
-SizeTestApp: 2,404,416 bytes (2,348.1 KB = 2.3 MB)
+SizeTestApp: 2,404,608 bytes (2,348.2 KB = 2.3 MB)
 SizeTestApp.dll: 7,680 bytes (7.5 KB = 0.0 MB)
 System.Private.CoreLib.aotdata.arm64: 41,312 bytes (40.3 KB = 0.0 MB)
 System.Private.CoreLib.dll: 988,160 bytes (965.0 KB = 0.9 MB)

tests/dotnet/UnitTests/expected/TVOS-NativeAOT-TrimmableStatic-size.txt

@@ -1,8 +1,8 @@
-AppBundleSize: 7,922,040 bytes (7,736.4 KB = 7.6 MB)
+AppBundleSize: 7,922,198 bytes (7,736.5 KB = 7.6 MB)
 # The following list of files and their sizes is just informational / for review, and isn't used in the test:
 _CodeSignature/CodeResources: 2,589 bytes (2.5 KB = 0.0 MB)
 archived-expanded-entitlements.xcent: 384 bytes (0.4 KB = 0.0 MB)
-Info.plist: 1,138 bytes (1.1 KB = 0.0 MB)
+Info.plist: 1,104 bytes (1.1 KB = 0.0 MB)
 PkgInfo: 8 bytes (0.0 KB = 0.0 MB)
 runtimeconfig.bin: 1,889 bytes (1.8 KB = 0.0 MB)
-SizeTestApp: 7,916,032 bytes (7,730.5 KB = 7.5 MB)
+SizeTestApp: 7,916,224 bytes (7,730.7 KB = 7.5 MB)

tests/dotnet/UnitTests/expected/iOS-MonoVM-interpreter-size.txt

@@ -1,12 +1,12 @@
-AppBundleSize: 3,616,747 bytes (3,532.0 KB = 3.4 MB)
+AppBundleSize: 3,603,385 bytes (3,518.9 KB = 3.4 MB)
 # The following list of files and their sizes is just informational / for review, and isn't used in the test:
 _CodeSignature/CodeResources: 3,997 bytes (3.9 KB = 0.0 MB)
 archived-expanded-entitlements.xcent: 384 bytes (0.4 KB = 0.0 MB)
-Info.plist: 1,161 bytes (1.1 KB = 0.0 MB)
+Info.plist: 1,127 bytes (1.1 KB = 0.0 MB)
 Microsoft.iOS.dll: 154,624 bytes (151.0 KB = 0.1 MB)
 PkgInfo: 8 bytes (0.0 KB = 0.0 MB)
 runtimeconfig.bin: 1,405 bytes (1.4 KB = 0.0 MB)
-SizeTestApp: 2,404,704 bytes (2,348.3 KB = 2.3 MB)
+SizeTestApp: 2,391,376 bytes (2,335.3 KB = 2.3 MB)
 SizeTestApp.dll: 7,680 bytes (7.5 KB = 0.0 MB)
 System.Private.CoreLib.aotdata.arm64: 41,312 bytes (40.3 KB = 0.0 MB)
 System.Private.CoreLib.dll: 988,160 bytes (965.0 KB = 0.9 MB)

tests/dotnet/UnitTests/expected/iOS-NativeAOT-size.txt

@@ -1,8 +1,8 @@
-AppBundleSize: 2,800,030 bytes (2,734.4 KB = 2.7 MB)
+AppBundleSize: 2,784,380 bytes (2,719.1 KB = 2.7 MB)
 # The following list of files and their sizes is just informational / for review, and isn't used in the test:
 _CodeSignature/CodeResources: 2,589 bytes (2.5 KB = 0.0 MB)
 archived-expanded-entitlements.xcent: 384 bytes (0.4 KB = 0.0 MB)
-Info.plist: 1,161 bytes (1.1 KB = 0.0 MB)
+Info.plist: 1,127 bytes (1.1 KB = 0.0 MB)
 PkgInfo: 8 bytes (0.0 KB = 0.0 MB)
 runtimeconfig.bin: 1,808 bytes (1.8 KB = 0.0 MB)
-SizeTestApp: 2,794,080 bytes (2,728.6 KB = 2.7 MB)
+SizeTestApp: 2,778,464 bytes (2,713.3 KB = 2.6 MB)

tools/devops/automation/templates/tests/run-tests.yml

@@ -172,6 +172,15 @@ steps:
   continueOnError: true
   condition: succeededOrFailed()
 
+# Upload updated expected app size files if the app size tests produced any.
+- task: PublishPipelineArtifact@1
+  displayName: 'Publish Artifact: Updated expected app size files'
+  inputs:
+    targetPath: '$(Build.ArtifactStagingDirectory)/updated-expected-sizes'
+    artifactName: '${{ parameters.uploadPrefix }}updated-expected-sizes-${{ parameters.testPrefix }}-$(System.JobAttempt)'
+  continueOnError: true
+  condition: succeededOrFailed()
+
 - pwsh: |
     $summaryName = "TestSummary-${{ parameters.testPrefix }}.md"
     $summaryPath = "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY/$(BUILD_REPOSITORY_TITLE)/tests/TestSummary.md"