Update exit code reporting

[bootstraponline]

No description provided.

[codecov-io]

Codecov Report

Merging #536 into master will increase coverage by 0.29%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #536      +/-   ##
============================================
+ Coverage     79.29%   79.59%   +0.29%     
- Complexity      557      561       +4     
============================================
  Files            77       77              
  Lines          2106     2102       -4     
  Branches        279      276       -3     
============================================
+ Hits           1670     1673       +3     
+ Misses          267      265       -2     
+ Partials        169      164       -5

[gtroshin]

Will it change the flaky-test-attempts logic somehow? The exit code now is 0 even if there is some matrix failed.

[bootstraponline]

FYI: The matrix roll-up change has been released.

Hmm. FTL said the matrix status should be correct now. They pinged me yesterday.

[bootstraponline]

@gtroshin I tested flank master and I'm getting the correct exit code (1) when the tests fail every time when using flaky-test-attempts. If a test passes at least once then the matrix status will be successful. If there are any issues, let me know. I think everything is working as expected now that FTL has fixed the bug on their side.

[gtroshin]

@bootstraponline Ok, thank you! Then I will update to 5.0.2 :)

[gtroshin]

@bootstraponline little bit late response.

For the one android project, we use v5.0.1 and it works as expected. When there is at list one Re-attempt passed it exit with 0 status code.

But I just upgraded to the v6.2.2 for our iOS project - the result is it exit with 1 status code in case there are any failures in some matrices.

[bootstraponline]

It sounds like I should restore the hack of using all the XML files to calculate the exit code.

Firebase's retry is defective by design.

I'll work on this soon.

[bootstraponline]

I reverted the commit here:
#572

[bootstraponline]

I updated #552

The fix has been released in flank v6.2.3