Allow to benchmark multi-machine clusters

[danielmitterdorfer]

Currently, it is only possible to benchmark on a single machine (except for the special case where we run with --pipeline=benchmark-only which (a) puts the burden of provisioning on the user and (b) does not gather system metrics (like CPU usage or index size). Rally should be able to run benchmarks also on clusters across multiple machines.

In this ticket we want to collect the high-level ideas around making this possible. Work should be done in smaller, more focused tickets. At least these areas need to change:

• Provisioning (consider also Evaluate provisioning of benchmark machines with Ansible #42 and Allow setting cluster config #117)
• Gathering metrics (we need to gather them on the machines that are being benchmarked)
• Reporting: We currently report system level metrics only once (e.g. CPU usage) and need to reconsider that (although we will prepare for that case already in Gather benchmark environment data #54)

Note: Distribution of the load generator is handled separately in #257.

[siennathesane]

What's the status of this issue?

[danielmitterdorfer]

It's in the early concept stage. We use Github's milestone feature and it's planned for 0.5.0 (no release date).

The current "workaround" is to use the pipeline "benchmark-only" as I've mentioned in the description of this ticket provided you are able to apply enough load with a single Rally instance. I'd expect that you can apply a bit more load when #108 is ready on which I am currently working on.

[siennathesane]

Gotcha, thanks!