test: imageinout_test: add benchmark of read and write speed vs tile size

[lgritz]

For various tile sizes (and scanline), benchmark how long it takes to read and write a 4k x 2k image.

[lgritz]

@lji-ilm

[lgritz]

Any comments? I will merge over the weekend if nobody objects.

[lji-ilm]

Hi Larry,

thanks for doing this

I am on parental leave in Feb, the first few weeks were busy.

This looks great to me, as it is merged now does the CI run it? or the bench things are all manually, locally run?

[lgritz]

The CI runs it to ensure it's correct and contributes to the code coverage, dynamic analysis, sanitizers.

But if you want accurate benchmarks, you should run it by hand ($ ./build/bin/imageinout_test) because the CI tries to be efficient by reducing the number of times the benchmarks are run, but that reduces timing accuracy.

[lji-ilm]

Ah, thank you for the explanation. So the CI ensures that the benchmark is not broken and runnable but does not actually collect benchmark data.
I was thinking about the following:

1. since the CI is running so many times, although each individual run would be spotty and not that accurate, maybe some 🆓 free 🆓 analytics regarding larger trend can still be pulled out by collecting these "free" benchmark data?

It doesn't look like collecting is built in other than the benchmark results may appear on the CI logs, but that might be something worthwhile?

2. I have been thinking about having a "Performance CI". The initial intention is to build it on the openEXR repo, but after some progress I thought it is better to make it on the OIIO repo with something like this to enable horizontal (exr v.s. tiff, for example) in addition to vertical (exr 3.4 vs 3.0) analytics. This would be running periodically instead of triggered by OIIO commits, since dependent library update would change results.

I essentially already have this on my own computer already but I'd like to edit my ad-hoc code to make use of the benchmark class which looks fantastic. I also have a dataset (all open source) that is a bit better than the hard-coded procedural data. However, any thoughts about hosting it somewhere in cloud and connect it to a, say, Grafana?

My thinking about hosting it in a CI-like environment is that those environments does not represent real-world production environment, which are typically beefy workstations. Some, if not most, performance traits should still be represented proportionally (if it's slower then it's slower everywhere) but I'm hesitate to say that loud until the data really shows it. Until then it might need to stay on a box in my basement...

[lgritz]

I do run the benchmark and report the times in the CI logs (only for some CI job variants). It's not completely useless, in fact it seems more stable than I would have expected, but it should be taken with a huge grain of salt because (a) in CI, it runs the tests for a reduced number of iterations, to save CI time, and (b) we have no control over the underlying hardware of the GHA runner, so from run to run it may be a different machine class.

If we really want to track performance over time or accurately know if a PR affects perf, I think we would need to test on a fully controlled, known machine with stable hardware and nothing else using machine resources at the same time.

The existing benchmarks are useful for a sanity check, or in cases where the same test is benchmarking two ways of doing things, like reading two different tile sizes. On that one run and one machine, I do think it can tell you if 16x16 tiles are faster or slower than 64x64 tiles. But it can't tell you accurately if it's faster or slower than it was last month, nor if it's faster or slower on Windows vs Linux, since there's no way to ensure that separate CI jobs are running on the same hardware configuration.

[lji-ilm]

Correct. Results can be compared from the same session, but not across different CI sessions.

Well, I'll continue to look into that fully controlled known machine with stable hardware in my basement then...