Some questions about benchmark stablility

[hai-x]

Hi, we are using Codspeed for benchmark in the Webpack repo, specifically in the BenchmarkTestCases.benchmark.mjs file:

We've noticed that some benchmark results are not stable enough. These benchmarks involve tasks that use fs.writeFile and followed by triggering one compilation.

The benchmark reports show inconsistent results across PRs, with random improvements or regressions.

Could file I/Os (system calls) be affecting the stability of these benchmark results? And do you have any suggestions for improving benchmark stalblility in this context?

[adriencaccia]

Hey, when using the default CPU instrument on CodSpeed, file I/Os can definitely affect the stability of the benchmarks, you can find some docs about it here: https://codspeed.io/docs/instruments/cpu/overview#system-calls

I have the following suggestions:

• If possible, remove the fs.writeFile part of some of the benchmark, to only measure CPU bound code
• You can ignore flaky benchmarks, to remove them from the reports but still have their measurements and history
• As explained in the docs on system calls, we recommend using the Walltime instrument instead for benchmarks with I/Os. We are soon going to release support for it in Bump node, bump tinybench, and add support of walltime runs for tinybench #48

[hai-x]

Hey, when using the default CPU instrument on CodSpeed, file I/Os can definitely affect the stability of the benchmarks, you can find some docs about it here: codspeed.io/docs/instruments/cpu/overview#system-calls

I have the following suggestions:

• If possible, remove the fs.writeFile part of some of the benchmark, to only measure CPU bound code
• You can ignore flaky benchmarks, to remove them from the reports but still have their measurements and history
• As explained in the docs on system calls, we recommend using the Walltime instrument instead for benchmarks with I/Os. We are soon going to release support for it in Bump node, bump tinybench, and add support of walltime runs for tinybench #48

Thank you for the comprehensive explanation. ❤️