Issue #428: Smart Flank - print how accurate test times are

[Macarse]

Opening this for feedback.

Example output:

Shard times: 10s, 20s, 30s
3 tests / 3 shards

Actual shard times: 9s (-11%), 21s (5%), 30s (0%)

[codecov-io]

Codecov Report

Merging #436 into master will decrease coverage by 0.21%.
The diff coverage is 56.66%.

@@             Coverage Diff              @@
##             master     #436      +/-   ##
============================================
- Coverage      78.5%   78.28%   -0.22%     
- Complexity      646      652       +6     
============================================
  Files            73       73              
  Lines          1847     1870      +23     
  Branches        274      277       +3     
============================================
+ Hits           1450     1464      +14     
- Misses          221      228       +7     
- Partials        176      178       +2

[cryptoman256]

The percentage calc is the wrong denominator. you have 10 expected, 9 actual which is an error of 1. 11% is 1/9. Should be 1/10, 10%.

[cryptoman256]

if you do it this way, it would be (finalTime * 100 / expectedTime) - 100
Ex. expected 10 sec, actual 9 sec
(9 * 100.0 / 10) - 100 = -10% -----> You were 10% off and it was to the downside
(11 * 100.0 / 10) - 100 = +10% -----> You were 10% off and it was to the upside

Or a more readable error calc imo would be:
(finalTime - expectedTime) / expectedTime * 100
(9-10)/10 * 100 = -10%
(11-10)/10 * 100 = +10%

[inktomi]

I would recommend printing the time diff instead of a percent diff as well. That seems more actionable as a user - a percentage is not so great as it'd require some math to figure out how long each took compared to the expected time.

[bootstraponline]

Discussed using human readable time reports:

1h 2m 3s

I agree with the other feedback as well.

[Macarse]

The output will look something like this:

Smart Flank cache hit: 100% (172 / 172)
Shard times: 46s, 46s, 46s, 46s, 46s, 46s, 47s, 47s, 47s, 47s, 48s, 48s, 48s, 48s, 48s

... 

Actual shard times:
  Shard 0: Expected: 46s, Actual: 46s, Diff: 0s
  Shard 1: Expected: 46s, Actual: 47s, Diff: 1s
  Shard 2: Expected: 46s, Actual: 45s, Diff: -1s
  Shard 3: Expected: 46s, Actual: 46s, Diff: 0s
  Shard 4: Expected: 46s, Actual: 46s, Diff: 0s
  Shard 5: Expected: 46s, Actual: 44s, Diff: -2s
  Shard 6: Expected: 47s, Actual: 48s, Diff: 2s
  Shard 7: Expected: 47s, Actual: 46s, Diff: -1s
  Shard 8: Expected: 47s, Actual: 50s, Diff: 3s
  Shard 9: Expected: 47s, Actual: 47s, Diff: -1s
  Shard 10: Expected: 48s, Actual: 48s, Diff: 1s
  Shard 11: Expected: 48s, Actual: 50s, Diff: 3s
  Shard 12: Expected: 48s, Actual: 48s, Diff: 0s
  Shard 13: Expected: 48s, Actual: 46s, Diff: -2s
  Shard 14: Expected: 48s, Actual: 47s, Diff: -1s

[cryptoman256]

The output will look something like this:

Smart Flank cache hit: 100% (172 / 172)
Shard times: 46s, 46s, 46s, 46s, 46s, 46s, 47s, 47s, 47s, 47s, 48s, 48s, 48s, 48s, 48s

... 

Actual shard times:
  Shard 0: Expected: 46s, Actual: 46s, Diff: 0s
  Shard 1: Expected: 46s, Actual: 47s, Diff: 1s
  Shard 2: Expected: 46s, Actual: 45s, Diff: -1s
  Shard 3: Expected: 46s, Actual: 46s, Diff: 0s
  Shard 4: Expected: 46s, Actual: 46s, Diff: 0s
  Shard 5: Expected: 46s, Actual: 44s, Diff: -2s
  Shard 6: Expected: 47s, Actual: 48s, Diff: 2s
  Shard 7: Expected: 47s, Actual: 46s, Diff: -1s
  Shard 8: Expected: 47s, Actual: 50s, Diff: 3s
  Shard 9: Expected: 47s, Actual: 47s, Diff: -1s
  Shard 10: Expected: 48s, Actual: 48s, Diff: 1s
  Shard 11: Expected: 48s, Actual: 50s, Diff: 3s
  Shard 12: Expected: 48s, Actual: 48s, Diff: 0s
  Shard 13: Expected: 48s, Actual: 46s, Diff: -2s
  Shard 14: Expected: 48s, Actual: 47s, Diff: -1s

I like this, fine with me!