Firebase Android tests flapping

[friedbunny]

In the last week or so, Firebase tests have been flapping:

https://www.bitrise.io/build/74f7be21c7dedd19 — two firebase test failures
https://www.bitrise.io/build/a1ae533de44ce8bd — one firebase test failure
https://www.bitrise.io/build/fa9e4efdc1c76881 — took 45 minutes to timeout when downloading Java
https://www.bitrise.io/build/e3f62dfb41987c22 — firebase networking timeout
https://www.bitrise.io/build/73e363e750edb3d1 — five firebase test failures
https://www.bitrise.io/build/91f588750e7ed5af — device farm timed out

Even after digging through the (very long) logcat, it’s not obvious to me why these are failing. These builds all passed after restarting them. Some are likely because of Bitrise networking instability, but others could represent bugs in our code or tests — it’s hard to say.

Let’s investigate these failures and see if there’s anything we can do to improve reliability.

/cc @tobrun @zugaldia

[tobrun]

Thank you for listing these up @friedbunny! I will keep 👀 on it.
From my perspective, the test were running pretty reliable for the last 2-3 weeks (after we fixed some native crashes), I was actually surprised by the quality of service that Firebase offers. Now after last week I'm a bit in doubt. Let's see what this week brings!

[tobrun]

Looked at all the builds from yesterday and was able to locate one flaky build vs ~ 40 builds:

• https://www.bitrise.io/build/6ddb8a6fddd52c5b

I will continue to monitor the job queu in the following days

[jfirebaugh]

https://www.bitrise.io/build/c9056b65300cf01e

[jfirebaugh]

https://www.bitrise.io/build/c2257b07c175bdfe

[tobrun]

We need to determine if the source of the behaviour is:

• a bug in the testing framework (espresso)
• instability in the firebase infrastructure

Short term actions:

• @tobrun tries reproducing flaky tests locally, if reproducible, try optimising/improving stability
• @zugaldia contacts Firebase support

What can we do if we can't resolve this issue? If the failing tests are perceived as more hinderance as actually beneficial we should look into downscaling the amount of tests we run. IMO the most important thing these tests need to flag is the regression that core crashes on startup. Having one test that validates this instead of having 150+ tests currently could make the test more reliable

eg. If you would take 20 PR's. you would find one with a flaky test. This roughly means that we have one flaky test for 3000 different test cases. If we would execute 1 test case for each PR we would be able to test 3000 PR's successfully without the tests failing because of flakyness.

[zugaldia]

@zugaldia contacts Firebase support

Just did.

[tobrun]

Been running some test runs locally and was able to occasionally reproduce it. Going to see if I can make the test harness run more robust. I'm also thinking that changes to the OnMapReady callback could have resulted in the test being less reliable.

[tobrun]

Found another failing build, this time it couldn't download a dependency.

• https://www.bitrise.io/build/437949c0099fc872

[friedbunny]

A whopping 30 failed tests: https://www.bitrise.io/build/006abf10f975dbbf

[tobrun]

I light of the above reports and to unblock other contributors: I'm going to downscale our +150 tests to just one. This should still catch the most important regressions that show when a MapView is rendered on screen. I'm going to look into #6366 to run scheduled builds with the full set of tests on a daily basis and run manual tests on the firebase gui to resolve our issues.