[Tests] Try different urls before failing the test. by mandel-macaque · Pull Request #6237 · dotnet/macios

mandel-macaque · 2019-06-10T20:45:31Z

We do have a test that fails when there are issues accessing the remote
resource. If an error is raised, rather than setting the tests as an
error, set them to inconclusive with the message of the error.

Related issue: https://github.com/xamarin/maccore/issues/1725

We do have a test that fails when there are issues accessing the remote resource. If an error is raised, rather than setting the tests as an error, set them to inconclusive with the message of the error. Related issue: https://github.com/xamarin/maccore/issues/1725

spouliot · 2019-06-10T20:52:04Z

+					Assert.Inconclusive ($"Could not access url error is '{error.Description}'");
+				} else {
+					Assert.That ((x != null) && (x.Length > 0));
+				}


the test was to detect an error...
with that change I don't think you can trigger the 2nd assert (unless an empty file was present)
so basically the test can't fail (so it's not worth much)

@spouliot we can check the domain of the error and decide what to do?

Well.. thx Cocoa, we would get the not so useful "NSCocoaErrorDomain Code=256", which means an unknown error happened.

yeah, a reachability (to host) error would be fine to ignore, but other errors would not be

I have used several company urls, lets hope trying several will work, else we can do an increasing timeout retry.

monojenkins · 2019-06-10T21:53:53Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
🔥 Failed to compare API and create generator diff 🔥
    ** Error: Working directory isn't clean:
    HEAD detached at 1c56879
    Untracked files:
    (use "git add ..." to include in what will be committed)

    tests/bcl-test/BCL tests group 1.csproj
    tests/bcl-test/BCL tests group 2.csproj
    tests/bcl-test/BCL tests group 3.csproj
    tests/bcl-test/BCL tests group 4.csproj
    tests/bcl-test/BCL tests group 6.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-modern.csproj
    tests/bcl-test/generated/
    tests/bcl-test/mscorlib-mac-full.csproj
    tests/bcl-test/mscorlib-mac-modern.csproj
    tests/bcl-test/mscorlib.csproj

    nothing added to commit but untracked files present (use "git add" to track)
    Search for Comparing API & creating generator diff in the log to view the complete log.
✅ Test run succeeded

spouliot · 2019-06-11T01:57:48Z

for reference

	[INCONCLUSIVE] NSDataTest.Https : Could not access url error is 'Error Domain=NSCocoaErrorDomain Code=256 "The file “robots.txt” couldn’t be opened." UserInfo={NSURL=https://www.microsoft.com/robots.txt}'
		  at MonoTouchFixtures.Foundation.NSDataTest.Https () [0x00021] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/monotouch-test/Foundation/NSDataTest.cs:162 
		  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
		  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/src/Xamarin.iOS/mcs/class/corlib/System.Reflection/RuntimeMethodInfo.cs:391

https://jenkins.mono-project.com/job/xamarin-macios-pr-builder/10214/Test_20Report/

rolfbjarne

Two ideas to improve this test:

Ignore only when doing CI (like 7d67e05), this way we'll eventually see failures when running into them locally.
Have a list of URLs to try, and pass the test if any of them work.

spouliot · 2019-06-11T15:54:00Z

it's unclear why it fails so it's possible the complete list will fail (even more since it's all MS hosts) but if it works then it's much better than the current test :)

monojenkins · 2019-06-11T16:54:44Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
🔥 Failed to compare API and create generator diff 🔥
    ** Error: Working directory isn't clean:
    HEAD detached at 79f7c48
    Untracked files:
    (use "git add ..." to include in what will be committed)

    tests/bcl-test/BCL tests group 1.csproj
    tests/bcl-test/BCL tests group 2.csproj
    tests/bcl-test/BCL tests group 3.csproj
    tests/bcl-test/BCL tests group 4.csproj
    tests/bcl-test/BCL tests group 6.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-modern.csproj
    tests/bcl-test/generated/
    tests/bcl-test/mscorlib-mac-full.csproj
    tests/bcl-test/mscorlib-mac-modern.csproj
    tests/bcl-test/mscorlib.csproj

    nothing added to commit but untracked files present (use "git add" to track)
    Search for Comparing API & creating generator diff in the log to view the complete log.
🔥 Test run failed 🔥

Test results

4 tests failed, 90 tests passed.

Failed tests

apitest/Mac Classic/NonSystem: BuildFailure
dont link/Mac Classic/All: BuildFailure
introspection/Mac Classic/Modern: BuildFailure
MSBuild tests/iOS: Failed (Execution failed with exit code 1)

mandel-macaque · 2019-06-11T16:59:43Z

Issues are related to the msbuild problems we have, all other tests, specially the NSDataTests passed.

Known issues are:

mandel-macaque · 2019-06-11T18:03:00Z

@monojenkins backport to d16-2

mandel-macaque · 2019-06-11T18:03:12Z

@monojenkins backport to d16-1

VincentDondain · 2019-06-12T16:15:37Z

@monojenkins backport to xcode11

spouliot suggested changes Jun 10, 2019

View reviewed changes

rolfbjarne reviewed Jun 11, 2019

View reviewed changes

Comment thread tests/monotouch-test/Foundation/NSDataTest.cs Outdated

Use a list of urls to try several times.

54bb886

spouliot approved these changes Jun 11, 2019

View reviewed changes

mandel-macaque changed the title ~~[Tests] If there are network issues do not set the tests as a failure.~~ [Tests] Try different urls before failing the test. Jun 11, 2019

mandel-macaque merged commit 46c16eb into dotnet:master Jun 11, 2019

mandel-macaque deleted the nsdatatest-account-network-issues branch June 11, 2019 17:04

This was referenced Jun 11, 2019

[d16-2] [Tests] Try different urls before failing the test. #6265

Merged

[d16-1] [Tests] Try different urls before failing the test. #6266

Merged

monojenkins mentioned this pull request Jun 12, 2019

[xcode11] [Tests] Try different urls before failing the test. #6276

Closed

Uh oh!

Conversation

mandel-macaque commented Jun 10, 2019

Uh oh!

spouliot Jun 10, 2019

Choose a reason for hiding this comment

Uh oh!

mandel-macaque Jun 10, 2019

Choose a reason for hiding this comment

Uh oh!

mandel-macaque Jun 10, 2019

Choose a reason for hiding this comment

Uh oh!

spouliot Jun 10, 2019

Choose a reason for hiding this comment

Uh oh!

mandel-macaque Jun 11, 2019

Choose a reason for hiding this comment

Uh oh!

monojenkins commented Jun 10, 2019

Uh oh!

spouliot commented Jun 11, 2019

Uh oh!

rolfbjarne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spouliot commented Jun 11, 2019

Uh oh!

monojenkins commented Jun 11, 2019

Test results

Failed tests

Uh oh!

mandel-macaque commented Jun 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mandel-macaque commented Jun 11, 2019

Uh oh!

mandel-macaque commented Jun 11, 2019

Uh oh!

VincentDondain commented Jun 12, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mandel-macaque commented Jun 11, 2019 •

edited

Loading