Distribute load-test driver

[danielmitterdorfer]

At the moment, Rally can apply load only from one machine. This is fine for smaller clusters but if we want to benchmark larger ones, we want to apply load from more than one machine. Hence, we need to distribute the load test driver.

Design

Rally's internal design uses the actor approach and is already suited for distribution (we already did it with ES provisioning). At the moment we have one global Driver actor that runs on the coordinating node (this is the node where the user invokes esrally on the command line) and one LoadGenerator actor per client.

Rally will attempt to balance clients across all machines that are provided by the user, i.e. if the track defines that we need to run 8 clients and the user provides 4 machines, we will run two clients on each machine (clients will be assigned round-robin to machines). This may lead to hot-spots but we should start simple.

Upon startup, the coordinating Driver will launch one TrackPreparationActor per host. This actor will take care of tasks that need to be done once per machine. These tasks are mainly on track management side (e.g. checking out the right version of a track or downloading benchmark data). In the first step we will not attempt to optimize downloading of benchmark data; to be precise: every machine will download all benchmark data for the track that is run.

After the track is prepared, these actors are shutdown and load generation is coordinated by the Driver (as is). Clients get assigned to machines in a round-robin fashion.

[danielmitterdorfer]

Reminder: We should keep #261 in mind, because this will not work with a queue-based actor system (which we use when the actor system cannot determine a valid socket address).