Randomization of benchmark's Arguments

[MineCake147E]

Often in benchmarks, the same value is given to the same argument of the same function over and over again in succession, but for some functions, this is almost never the case in practice.
Randomizing the order of argument values given to the same function in repeated calls can be useful in determining the performance impact of the order in which they are given.
Therefore, it would be nice to have an Attribute that specifies a list of argument values to be given in a ArgumentsSourceAttribute-like fashion, and randomly gives values to arguments from that list at each iteration.

[timcassell]

This looks like a problem that could be solved by my idea in #1782. You could randomize the values in the creator method.

Thoughts @adamsitnik?

[adamsitnik]

Often in benchmarks, the same value is given to the same argument of the same function over and over again in succession

We do that on purpose, in order to get repeatable results.

In #1587 I've introduced the concept of memory randomization to have buffers of different alignment. You could use that to achieve the desired effect by moving the logic that generates the random values to the [GlobalSetup] method which then is invoked once per iteration (not per process lifetime).

Pseudocode:

[MemoryRandomization]
public class Sample
{
    private double _x;
    
    [GlobalSetup]
    public void Setup() => _x = Random.Shared.NextDouble();
    
    [Benchmark]
    public double RandomPow() => Math.Pow(_x, _x);
}

Please keep in mind that BDN has a heuristic that decides when to stop and if benchmark interations perform different amount of work, it won't be happy and will take more time to produce the results.

[timcassell]

@adamsitnik I don't think that does what @MineCake147E wants. That would still use the same value at every invocation, and just change every iteration. The results would be meaningless. @MineCake147E seems to want a different value at every invocation, but for repeatable results, the same values on every iteration. The only way I see that working is either special logic in BDN to do that, or my idea that can also do it (is also special logic, but more general for other purposes).

[AndreyAkinshin]

I don't like the idea of exposing such a feature out of the box. Here are some reasonings:

1. In BenchmarkDotNet, we want to make the measurements as repeatable as possible, not the opposite. Such a randomization feature could be dangerous since incorrect usage of such API may easily produce misleading results.
2. If we want to get more "real-life average" performance metrics, it's better to measure the performance distribution of the target method on a set of fixed arguments and aggregate the results manually.
3. Various randomization strategies can be easily implemented on top of the existing API.

[timcassell]

2. If we want to get more "real-life average" performance metrics, it's better to measure the performance distribution of the target method on a set of fixed arguments and aggregate the results manually.

Of course, you could do something like this:

public class HalfCast
{
    // public field to prevent dead code elimination
    public Half halfField;
    
    // 4 operations per invoke averages the cost of the 4 operations
    [Benchmark(OperationsPerInvoke = 4)]
    [Arguments(6.097555E-05f, 12345.0f, 65520.0f, float.NaN)]
    public void SingleToHalf(float arg1, float arg2, float arg3, float arg4)
    {
        halfField = (Half) arg1;
        halfField = (Half) arg2;
        halfField = (Half) arg3;
        halfField = (Half) arg4;
    }
}

[adamsitnik]

I don't like the idea of exposing such a feature out of the box

I agree.