Skip to content

[tests][monotouch] Generalize some tests to be independent of iOS version. Fixes #51801 - #1598

Merged
spouliot merged 1 commit into
dotnet:xcode8.3from
spouliot:bug51801
Feb 1, 2017
Merged

[tests][monotouch] Generalize some tests to be independent of iOS version. Fixes #51801#1598
spouliot merged 1 commit into
dotnet:xcode8.3from
spouliot:bug51801

Conversation

@spouliot

Copy link
Copy Markdown
Contributor

The exact values are not what we need to test for and varies with
different OS versions - making tests fails for no good reason (i.e.
they are not canary used to detect changes)

reference:
https://bugzilla.xamarin.com/show_bug.cgi?id=51801

…sion. Fixes #51801

The exact values are not what we need to test for and varies with
different OS versions - making tests fails for no good reason (i.e.
they are not canary used to detect changes)

reference:
https://bugzilla.xamarin.com/show_bug.cgi?id=51801
@monojenkins

Copy link
Copy Markdown
Contributor

Build failure

@VincentDondain

Copy link
Copy Markdown
Contributor

they are not canary used to detect changes

@spouliot Canary used? Can you explain? I don't see how it applies here, in the context of the monotouch-tests.

Here's the definition of "canary testing" I found:

In software testing, a canary is a push of programming code changes to a small group of end users who are unaware that they are receiving new code. Because the canary is only distributed to a small number of users, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.

For the curious who, like me, didn't know where "canary testing" came from:

The word canary was selected to describe the code push to a subset of users because canaries were once used in coal mining to alert miners when toxic gases reached dangerous levels.

@spouliot

spouliot commented Jan 31, 2017

Copy link
Copy Markdown
Contributor Author

@VincentDondain

Canary Tests are minimal tests to quickly and automatically verify that the everything you depend on is ready. You run Canary tests before other time-consuming tests, and before wasting time investigating in your code when the other tests are red. If the canary test fails, you know you have to fix something on the environments first.

In this case the value we checked was not a canary and had no real value (beyond confirming the call worked).

note: your definition match the "canary deployment" (not test)

@spouliot

Copy link
Copy Markdown
Contributor Author

monotouch-test crashes are tracked in https://bugzilla.xamarin.com/show_bug.cgi?id=52162 but hides the result of the current PR :(

@spouliot

Copy link
Copy Markdown
Contributor Author

build

@VincentDondain

Copy link
Copy Markdown
Contributor

@spouliot ah yes makes sense, I guess mine was "canary deployment".

Too many canaries! 🦆

@monojenkins

Copy link
Copy Markdown
Contributor

Build failure

@spouliot

Copy link
Copy Markdown
Contributor Author

build

@monojenkins

Copy link
Copy Markdown
Contributor

Build failure

@spouliot

spouliot commented Feb 1, 2017

Copy link
Copy Markdown
Contributor Author

Unrelated failures

@spouliot
spouliot merged commit fd964fe into dotnet:xcode8.3 Feb 1, 2017
@spouliot
spouliot deleted the bug51801 branch February 1, 2017 00:46
spouliot added a commit to spouliot/xamarin-macios that referenced this pull request Mar 28, 2017
…sion. Fixes #51801 (dotnet#1598)

The exact values are not what we need to test for and varies with
different OS versions - making tests fails for no good reason (i.e.
they are not canary used to detect changes)

reference:
https://bugzilla.xamarin.com/show_bug.cgi?id=51801
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants