CI Review

[wiredfool]

With the news that travis has been acquired, I've seen a bunch of discussion about CI platforms, leading to some questions:

• Is the current CI system covering what we need it to do now?
• Is it working with our long term goals?

One of the goals that I had originally with Appveyor was to be able to produce our windows binaries off of a CI system, rather than bothering @cgohlke for them every single time, as he's a single point of failure for the entire windows python ecosystem. I'm not sure where we are on the windows testing WRT external dependency completeness or python version coverage. Azure pipelines has now introduced free build time for open source projects, and I believe that the python core is now on that for their packages. I believe that they also do linux builds, but not macos. It may be worth investigating if their environment is any easier to get all the dependencies to build.

[hugovk]

It's definitely worth looking into. Building Windows binaries would definitely be great.

More info from GitHub: https://github.blog/2018-09-10-azure-pipelines-now-available-in-github-marketplace/

I've seen a few other projects move over to Azure Pipelines, but haven't looked into it much. It can do Linux, Windows and macOS, and CPython, pip and NumPy are testing those with Azure.

I think our current CI system is doing quite a good job, it'd be nice for all three platforms to be tested in one place, and especially if Azure is faster (with 10 parallel builds).

Right now:

• Appveyor has just 1 parallel worker and takes ~1h15m for 12 jobs
• Travis CI has 5 parallel workers and takes ~26m for 24 jobs

From a NumPy discussion:

The "free for open source" tier is 10 concurrent jobs that can run up to 60
minutes each (although the website says 30 minutes), and no limit on
minutes per month. Like the other CI services, it uses YAML files to
define the build and test process. I found the documentation of the YAML
pipeline spec to be large, but still missing a number of small details and
explanations about how features worked together. That said, the build
performance is significantly better than Travis CI and AppVeyor for all 3
platforms, and the convenience of all major platforms going through one
service is very nice. Despite not liking the UI quite as much as Travis
CI, I think Numba will likely switch completely over to Azure assuming the
free offering continues to be this good.

What are our long term goals?

• Run tests time, with good coverage of supported platforms, Python versions and depedencies
• Build binaries

[hugovk]

I had a play with Azure Pipelines yesterday, adding it to another pure Python (with no C) project. Some notes.

• You define the config in /azure-pipelines.yml, and it will be found automatically. Or you can define it in any other YML filename and location, and manually add it via the dashboard.

• A pipeline is made up of a number of jobs (like Travis build jobs). A job can run on Linux, macOS, Windows or Docker:

  • Ubuntu 16.04
  • macOS 10.13
  • Visual Studio 2017 on Windows Server 2016
  • Windows Server 1803 - for running Windows containers
  • Visual Studio 2015 on Windows Server 2012R2
  • https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azdevops&tabs=yaml

• Here's a single pipeline config:
  https://github.com/hugovk/pypistats/blob/72bd55c8fcc9c659910c73ba4e8c593659560c67/azure-pipelines.yml

• It has one job for linting, and jobs for each OS, and a final publish job that's conditional on all the others succeeding (this doesn't actually publish yet, just validates it can build sdist):
  https://dev.azure.com/hugovk/hugovk/_build/results?buildId=26

• It uses templates to remove duplication. In this example, .azure-pipelines/jobs/test.yml defines identical test steps for each OS.

• Each job has a Python version matrix, and is made up of steps. Steps are a bit like the install:, script: bits in .travis.yml.
  https://github.com/hugovk/pypistats/blob/72bd55c8fcc9c659910c73ba4e8c593659560c67/.azure-pipelines/jobs/test.yml

• A task is some pre-defined Azure recipe, like UsePythonVersion@0 and PublishTestResults@2

• script or bash (and there's many more, like for PowerShell) is where you put your own commands. It's nice that you can split up your commands into chunks like this, and the Azure UI splits up the logs by these chunks.

• You can have a single pipeline added, like above, or you can add several, like pip. I presume they added each of manually via the dashboard, as there's no top-level /azure-pipelines.yml.
  https://github.com/pypa/pip/tree/e5f4bbb7ddff87f48f2b5815513e4211ccdde83a/.azure-pipelines

• A benefit of multiple pipelines is having each OS show up as a separate build. For example:

• Templates allow each top-level linux.yml, macos.yml and windows.yml re-use the same config for jobs and also steps (pip uses the same config for Linux and macOS, a bit different config for Windows).

• There's no caching yet (eg. for pip installs)

As the serial one-worker builds of Appveyor are so slow, the biggest benefit for us would be to get Windows builds onto Azure.

There's nothing to stop us having more than one CI running, especially whilst trying things out. Pip are still using Travis, Appveyor and Azure, with the first two marked as "required" for PRs.

[hugovk]

See #3670 for a first step to get the plumbing installed.

[hugovk]

#3670 is merged to add config to the repo.

I've created an org and project at https://dev.azure.com/python-pillow/pillow/

@python-pillow/pillow-team I'll add you as admins, please let me know your Azure (https://azure.microsoft.com) sign-in address or display name (either here or privately via https://github.com/orgs/python-pillow/teams/pillow-team or email).

[aclark4life]

Weee thanks @hugovk. Looks like I'm aclark4life@hotmail.com.

[hugovk]

@aclark4life Added!

Please could you add a new CI "pipeline"? Something like this:

1. At https://dev.azure.com/python-pillow/pillow/_build click "New pipeline"
2. Select GitHub (do OAuth)
3. Select python-pillow/Pillow
4. Select "Existing Azure Pipelines YAML file"
5. Install the Azure Pipelines GitHub App to this repository (above of the YAML)
6. After installed, select the project at Azure
7. Click "Run"
8. That should save it and trigger the first build

[aclark4life]

Thanks, looks like it's doing something.

[hugovk]

Thanks, it's working!

I rebased PR #3661 to test automated builds, and that's working too:

https://dev.azure.com/python-pillow/pillow/_build/results?buildId=2

[hugovk]

In the time it took for 12 Azure jobs to finish (5m 43s), 6 Travis jobs finished (full build 21m 29s), and the first AppVeyor job was still going (full build ~63m).

TODO:

• Get Windows builds onto Azure, remove AppVeyor
• Get coverage working from Docker builds on AP, remove Docker from Travis
• Get non-Docker builds onto Azure, remove Travis (or, because Travis is more familiar and contributors are more likely to enable it, reduce matrix to most important jobs)

[aclark4life]

Do we have a way to skip Azure builds per commit?