RandomNumberGeneratorTests.GetInt32_MaskRangeCorrect failing sporadically in CI

[ericstj]

https://dev.azure.com/dnceng/public/_build/results?buildId=216796&view=ms.vss-test-web.build-test-results-tab&runId=5324506&resultId=383947&paneView=debug

System.Security.Cryptography.RNG.Tests.RandomNumberGeneratorTests.GetInt32_MaskRangeCorrect(fromInclusive: 16777214, toExclusive: 16777217)

Occurred number of times within threshold. Actual: 0.07166666666666671\nExpected: True\nActual: False

Stack trace
   at System.Security.Cryptography.RNG.Tests.RandomNumberGeneratorTests.VerifyDistribution(ReadOnlySpan`1 numbers, Double expected) in /_/src/System.Security.Cryptography.Algorithms/tests/RandomNumberGeneratorTests.netcoreapp.cs:line 279
   at System.Security.Cryptography.RNG.Tests.RandomNumberGeneratorTests.GetInt32_MaskRangeCorrect(Int32 fromInclusive, Int32 toExclusive) in /_/src/System.Security.Cryptography.Algorithms/tests/RandomNumberGeneratorTests.netcoreapp.cs:line 257

I've seen this failure on a few PRs but haven't yet seen a bug. In this case it was ARM64.

[bartonjs]

@vcsjones Looking at the numbers, I think that means that instead of 333 of one of the values there were 357; which is (per the error message) just barely out of conformance.

For RandomNumberGenerator.Fill, the test does 2048 bytes, then checks that the bits are within 7% of 50/50; allowing a maximum off-perfect of 573 (the 574th skewed bit fails the test).

I think the int tests are failing because of the law of medium numbers... that is, there aren't enough of them to hit the law of large numbers 😄. When the range is really wide, the tolerance band is more forgiving... e.g. for a range of 1000 the tolerance band allows E(x) = 0.001 to range up to 0.071. When the range is 3, E(x) = 0.333 becomes [0.263, 0.403]. Proportionally speaking, the 1000 range was a "71x tolerance" and the 3 range was a "1.42x tolerance". Yeah, this is just rambling nonsense saying "I'm not sure we're testing the same thing for each of the theories".

Realistically, perhaps just doubling the sample size will make the problem go away (hopefully making the data set "large" enough that statistics become our friend again). Also, maybe we should be doing something with variance (which should be range*range / 12, if I've given my stats-brain enough coffee).

[vcsjones]

@bartonjs hm. I see. I thought I stressed all of them enough to make sure sporadic failures weren't going to happen when I looked at the last PR. I'll revisit.

I'm currently traveling on vacation where the most sophisticated computer I have with me is an iPhone - I won't be able to look at it until early next week.