[devops] Make test steps fail if there are any test failures. by rolfbjarne · Pull Request #15764 · dotnet/macios

rolfbjarne · 2022-08-25T08:09:21Z

This makes it possible to re-run tests when they fail (since Azure DevOps only
allows re-running failed jobs).

It shouldn't affect any release pipelines anymore, because the release
pipeline only depends on the job that builds the packages now.

This also involved some CI changes, to be able to figure out the last test results when a test step is executed multiple times. Also, the GitHub comment will now state the run attempt (if >1) for each test (example)

This makes it possible to re-run tests when they fail (since Azure DevOps only allows re-running failed jobs). It shouldn't affect any release pipelines anymore, because the release pipeline only depends on the job that builds the packages now.

…y work.

…lures

vs-mobiletools-engineering-service2 · 2022-09-02T21:01:14Z

🔥 [CI Build] Test results 🔥

Test results

❌ Tests failed on VSTS: simulator tests

0 tests crashed, 41 tests failed, 182 tests passed.

Failures

❌ dotnettests tests [attempt 2]

1 tests failed, 0 tests passed.

Details

DotNet tests: BuildFailure

Html Report (VSDrops) Download

❌ linker tests [attempt 2]

38 tests failed, 27 tests passed.

Details

link sdk/Mac [dotnet]/Debug [dotnet]: BuildFailure
link sdk/Mac [dotnet]/Release [dotnet]: BuildFailure
link sdk/Mac Catalyst [dotnet]/Debug [dotnet]: BuildFailure
link sdk/Mac Catalyst [dotnet]/Release [dotnet]: BuildFailure
link sdk/iOS Unified 64-bits - simulator/Debug [dotnet]: BuildFailure
link sdk/iOS Unified 64-bits - simulator/Release [dotnet]: BuildFailure
link sdk/iOS Unified 64-bits - simulator/Debug: BuildFailure
link sdk/iOS Unified 64-bits - simulator/Release: BuildFailure
link sdk/tvOS - simulator/Debug [dotnet]: BuildFailure
link sdk/tvOS - simulator/Release [dotnet]: BuildFailure
link sdk/tvOS - simulator/Debug: BuildFailure
link sdk/tvOS - simulator/Release: BuildFailure
link sdk/watchOS 32-bits - simulator/Debug: BuildFailure
link sdk/watchOS 32-bits - simulator/Release: BuildFailure
link all/Mac [dotnet]/Debug [dotnet]: BuildFailure
link all/Mac [dotnet]/Release [dotnet]: BuildFailure
link all/Mac Catalyst [dotnet]/Debug [dotnet]: BuildFailure
link all/Mac Catalyst [dotnet]/Release [dotnet]: BuildFailure
link all/Mac Modern/Debug: BuildFailure
link all/Mac Modern/Release: BuildFailure
link all/iOS Unified 64-bits - simulator/Debug [dotnet]: BuildFailure
link all/iOS Unified 64-bits - simulator/Release [dotnet]: BuildFailure
link all/iOS Unified 64-bits - simulator/Debug: BuildFailure
link all/iOS Unified 64-bits - simulator/Release: BuildFailure
link all/tvOS - simulator/Debug [dotnet]: BuildFailure
link all/tvOS - simulator/Release [dotnet]: BuildFailure
link all/tvOS - simulator/Debug: BuildFailure
link all/tvOS - simulator/Release: BuildFailure
link all/watchOS 32-bits - simulator/Debug: BuildFailure
link all/watchOS 32-bits - simulator/Release: BuildFailure
trimmode link/Mac [dotnet]/Debug [dotnet]: BuildFailure
trimmode link/Mac [dotnet]/Release [dotnet]: BuildFailure
trimmode link/Mac Catalyst [dotnet]/Debug [dotnet]: BuildFailure
trimmode link/Mac Catalyst [dotnet]/Release [dotnet]: BuildFailure
trimmode link/iOS Unified 64-bits - simulator/Debug [dotnet]: BuildFailure
trimmode link/iOS Unified 64-bits - simulator/Release [dotnet]: BuildFailure
trimmode link/tvOS - simulator/Debug [dotnet]: BuildFailure
trimmode link/tvOS - simulator/Release [dotnet]: BuildFailure

Html Report (VSDrops) Download

❌ mmp tests [attempt 2]

1 tests failed, 1 tests passed.

Details

mmptest/macOS/Debug: BuildFailure

Html Report (VSDrops) Download

❌ mtouch tests [attempt 2]

1 tests failed, 0 tests passed.

Details

MTouch tests/NUnit: BuildFailure

Html Report (VSDrops) Download

Successes

✅ bcl: All 69 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 7 tests passed. Html Report (VSDrops) Download
✅ framework: All 8 tests passed. Html Report (VSDrops) Download
✅ generator: All 2 tests passed. Html Report (VSDrops) Download
✅ interdependent_binding_projects: All 7 tests passed. Html Report (VSDrops) Download
✅ install_source: All 1 tests passed. Html Report (VSDrops) Download
✅ introspection: All 8 tests passed. Html Report (VSDrops) Download
✅ mac_binding_project: All 1 tests passed. Html Report (VSDrops) Download
✅ mononative: All 12 tests passed. Html Report (VSDrops) Download
✅ monotouch: All 23 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ xammac: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 8 tests passed. Html Report (VSDrops) Download
✅ xtro: All 2 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: [PR build]

This reverts commit 554109d.

vs-mobiletools-engineering-service2 · 2022-09-05T16:51:25Z

📚 [PR Build] Artifacts 📚

Artifacts were not provided.

Pipeline on Agent XAMBOT-1104.Monterey'
Hash: aa07d2b087a9148e589919b57463ffc6a098a8c8 [PR build]

vs-mobiletools-engineering-service2 · 2022-09-05T17:06:10Z

💻 [PR Build] Tests on macOS Mac Catalina (10.15) passed 💻

✅ All tests on macOS Mac Catalina (10.15) passed.

Pipeline on Agent
Hash: aa07d2b087a9148e589919b57463ffc6a098a8c8 [PR build]

vs-mobiletools-engineering-service2 · 2022-09-05T17:07:09Z

✅ API diff for current PR / commit

Legacy Xamarin (No breaking changes)

~~iOS~~ (no change detected)
~~tvOS~~ (no change detected)
~~watchOS~~ (no change detected)
~~macOS~~ (no change detected)

NET (empty diffs)

~~iOS~~: (empty diff detected)
~~tvOS~~: (empty diff detected)
~~MacCatalyst~~: (empty diff detected)
~~macOS~~: (empty diff detected)

✅ API diff vs stable

Legacy Xamarin (No breaking changes)

iOS: vsdrops gist (No breaking changes)
tvOS: vsdrops gist (No breaking changes)
watchOS: vsdrops gist (No breaking changes)
macOS: vsdrops gist (No breaking changes)

.NET (No breaking changes)

iOS: vsdrops gist (No breaking changes)
tvOS: vsdrops gist (No breaking changes)
MacCatalyst: vsdrops gist (No breaking changes)
macOS: vsdrops gist (No breaking changes)
Microsoft.iOS vs Microsoft.MacCatalyst: vsdrops gist

Legacy Xamarin (stable) vs .NET

iOS: vsdrops gist
tvOS: vsdrops gist
macOS: vsdrops gist

✅ Generator diff

Generator diff is empty

Pipeline on Agent
Hash: aa07d2b087a9148e589919b57463ffc6a098a8c8 [PR build]

vs-mobiletools-engineering-service2 · 2022-09-05T17:25:51Z

❌ [PR Build] Tests on macOS M1 - Mac Big Sur (11.5) failed ❌

Failed tests are:

introspection
xammac_tests
monotouch-test

Pipeline on Agent
Hash: aa07d2b087a9148e589919b57463ffc6a098a8c8 [PR build]

vs-mobiletools-engineering-service2 · 2022-09-05T19:52:34Z

🚀 [CI Build] Test results 🚀

Test results

✅ All tests passed on VSTS: simulator tests.

🎉 All 222 tests passed 🎉

Tests counts

✅ bcl: All 69 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests: All 1 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 7 tests passed. Html Report (VSDrops) Download
✅ framework: All 8 tests passed. Html Report (VSDrops) Download
✅ generator: All 2 tests passed. Html Report (VSDrops) Download
✅ interdependent_binding_projects: All 7 tests passed. Html Report (VSDrops) Download
✅ install_source: All 1 tests passed. Html Report (VSDrops) Download
✅ introspection: All 8 tests passed. Html Report (VSDrops) Download
✅ linker: All 65 tests passed. Html Report (VSDrops) Download
✅ mmp: All 2 tests passed. Html Report (VSDrops) Download
✅ mononative: All 12 tests passed. Html Report (VSDrops) Download
✅ monotouch: All 23 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ mtouch: All 1 tests passed. Html Report (VSDrops) Download
✅ xammac: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 8 tests passed. Html Report (VSDrops) Download
✅ xtro: All 2 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: [PR build]

mandel-macaque · 2022-08-29T13:38:26Z

  inputs:
    targetPath: 'xamarin-macios/tests/TestSummary.md'
-    artifactName: TestSummary-${{ parameters.testPrefix }}
+    artifactName: TestSummary-${{ parameters.testPrefix }}-$(System.JobAttempt)


This is the right way, but I wonder if the code that publish the result in the comment does not need to be updated here: https://github.com/xamarin/xamarin-macios/blob/main/tools/devops/automation/templates/tests/publish-html.yml#L61

That code was updated in this PR.

rolfbjarne added the not-notes-worthy Ignore for release notes label Aug 25, 2022

This comment has been minimized.

Sign in to view

Make macOS errors actual errors.

46e8da7

rolfbjarne changed the title ~~[devops] Make steps that run tests fail if there are any test failures.~~ [devops] Make test steps fail if there are any test failures. Aug 26, 2022

rolfbjarne added 3 commits August 29, 2022 11:37

[devops] Sprinkle 'System.JobAttempt' to make re-running jobs actuall…

0748276

…y work.

[tests] Speed this up by making numerous tests fail early.

554109d

Merge remote-tracking branch 'origin/main' into test-failures-are-fai…

506b4f4

…lures

This comment has been minimized.

Sign in to view

I got a bit too eager...

6198772

This comment has been minimized.

Sign in to view

[devops] Powershell magic.

458a06e

This comment has been minimized.

Sign in to view

Show attempt # in the test summary.

b2200fd

This comment has been minimized.

Sign in to view

Don't make attempt numbers render like github links.

dbe2486

This comment has been minimized.

Sign in to view

Revert "[tests] Speed this up by making numerous tests fail early."

1481848

This reverts commit 554109d.

This comment has been minimized.

Sign in to view

[devops] More failures to not ignore

aa07d2b

rolfbjarne marked this pull request as ready for review September 5, 2022 19:58

rolfbjarne requested a review from mandel-macaque September 6, 2022 14:29

mandel-macaque approved these changes Sep 6, 2022

View reviewed changes

rolfbjarne merged commit 494e4f3 into dotnet:main Sep 7, 2022

rolfbjarne deleted the test-failures-are-failures branch September 7, 2022 08:38

Uh oh!

Conversation

rolfbjarne commented Aug 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Sep 2, 2022

🔥 [CI Build] Test results 🔥

Test results

Failures

❌ dotnettests tests [attempt 2]

❌ linker tests [attempt 2]

❌ mmp tests [attempt 2]

❌ mtouch tests [attempt 2]

Successes

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Sep 5, 2022

📚 [PR Build] Artifacts 📚

Uh oh!

vs-mobiletools-engineering-service2 commented Sep 5, 2022

💻 [PR Build] Tests on macOS Mac Catalina (10.15) passed 💻

Uh oh!

vs-mobiletools-engineering-service2 commented Sep 5, 2022

✅ API diff for current PR / commit

✅ API diff vs stable

✅ Generator diff

Uh oh!

vs-mobiletools-engineering-service2 commented Sep 5, 2022

❌ [PR Build] Tests on macOS M1 - Mac Big Sur (11.5) failed ❌

Uh oh!

vs-mobiletools-engineering-service2 commented Sep 5, 2022

🚀 [CI Build] Test results 🚀

Test results

Tests counts

Uh oh!

mandel-macaque Aug 29, 2022

Choose a reason for hiding this comment

Uh oh!

rolfbjarne Sep 7, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

rolfbjarne commented Aug 25, 2022 •

edited

Loading