Run test sets in different Travis jobs

[hugovk]

Experiment

This wasn't as successful as hoped, but there was some improvement.

Hypothesis

Running tests in separate jobs will speed up the Travis CI builds.

Results

Initially, build slowed down. Probably due to the overhead of installing dependencies and building Pillow at the start. If that bit can be sped up (possibly via caching) then this approach may help.

Changing Python order sped up build a bit (total waiting time), but uses more CPU time.

Details

Using an environment variable, we can run different test sets in different build jobs.

Reference:
http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines

Currently we have two main tests sets: self-test, and the main tests.

Using an environment variable, Travis will run a different job for each value.

env:
  - TEST_CMD="coverage run --append --include=PIL/* selftest.py"
  - TEST_CMD="python Tests/run.py --coverage"

Then instead of running both in script:

  - coverage run --append --include=PIL/* selftest.py
  - python Tests/run.py --coverage

just run one:

  - $TEST_CMD

This changes build times from:

26:32 https://travis-ci.org/hugovk/Pillow/builds/22906842
24:25 https://travis-ci.org/hugovk/Pillow/builds/23269205

to:

36:52 https://travis-ci.org/hugovk/Pillow/builds/23268580
33:26 https://travis-ci.org/hugovk/Pillow/builds/23269260

Hmm. That may be partly because the pypy build is 2-3x slower and run last. We're just sitting around waiting for it to finish on the last job-runner. If we run pypy first, we can make better use of the parallel job-runners:

39:48 https://travis-ci.org/hugovk/Pillow/builds/23269381
35:48 https://travis-ci.org/hugovk/Pillow/builds/23269381

Still bad. Most of this is probably from the dependency install and Pillow build overhead at the start of each job.

But hang on! These times are the total CPU time used across all jobs. The actual time from start to finish is much less.

Here's the actual elapsed start-to-end times:

Before: ~12m (about the time of the slowest pypy job)
With env: ~13m
pypy at end: ~10m (about the time of the slowest pypp job)

It uses more CPU time, but us humans get the build results in a bit less time.

Conclusion

Should we include this change? Perhaps it'd be better to until #540 has been finalised and test again, as the dependency/build overhead will remain.

At the very least, pypy should come first as that's the slowest and needs a head start. I've made another PR for that (#633).

[wiredfool]

Closing for now, I don't think it was a successful experiment.