ci: Shift to a prebuilt Docker image

[nathanchance]

I think it is time that this is brought to a pull request for discussion.

This moves our setup to having our Docker environment be prebuilt and
setup automatically daily with a Travis cron.

The benefits of this include:

1. Removing the potentially messy dependency dance that we are currently
   doing with our QEMU PPA.

2. Having a consistent environment that can not only be used by Travis
   but by other developers looking to get involved with the project.

3. Faster CI. The current image clocks in at around 450MB, which is the
   only thing that needs to be downloaded because everything else is
   already installed and setup.

4. If for some reason we wanted to switch environments (such as to a
   smaller distro like Alpine Linux), we could built more things from
   source because we would have a full 50 minutes with a Travis job.

Currently, the Dockerfile and other files are on my GitHub, they are
fairly easy to understand and modify.

https://github.com/nathanchance/cbl-dockerimage

I can create a ClangBuiltLinux organization on Docker Hub and move the
image there and the repo to this organization if people so desire. I am
also open to having that reviewed as well, all comments are welcome!

Closes #34.

[nickdesaulniers]

Cool, this sounds like a good idea to me, and even at least one kernel dev found this useful!

The benefits of this include:
Removing the potentially messy dependency dance that we are currently
doing with our QEMU PPA.

So not having a custom PPA we need to fetch from, just our custom build in our repo? Seems good, in case the PPA accidentally disappears.

Having a consistent environment that can not only be used by Travis
but by other developers looking to get involved with the project.

Yep, external devs already found this useful.

Faster CI. The current image clocks in at around 450MB, which is the
only thing that needs to be downloaded because everything else is
already installed and setup.

Probably don't even need a full blown debian distro, could roll our own. Definitely out of scope for V0. Do you have any measurements what the image size is AFTER this change, for comparison?

If for some reason we wanted to switch environments (such as to a
smaller distro like Alpine Linux), we could built more things from
source because we would have a full 50 minutes with a Travis job.

I don't follow; we can't download build artifacts from Travis, so the time it takes to build the docker image in travis doesn't matter. Or do you mean having more time to run continuous-integration on travis?

Currently, the Dockerfile and other files are on my GitHub, they are
fairly easy to understand and modify.
https://github.com/nathanchance/cbl-dockerimage

Would you mind moving these to our organization? I trust everyone in the org with the keys, but I would like to prevent departures from github from affecting our infra.

I can create a ClangBuiltLinux organization on Docker Hub and move the
image there and the repo to this organization if people so desire. I am
also open to having that reviewed as well, all comments are welcome!

With the above change in repo, I'm cool with you setting up the docker org, and trusting you with the responsibility of releasing updates to the docker image. Maybe you could share the pw privately with me and Joel should we ever need it?

TODO: will have more comments inline in the PR

[shenki]

When using Docker I shy away from the idea of downloading binary images (we do this by relying on a distro, but there is a level of assurance that the binaries from a distro have been generated safely). Have we given consideration to this?

Another downside is it breaks people running on on-x86 systems. This detail is less of a blocker as we already have this problem with the nightly clang images being x86-only.

[nickdesaulniers]

When using Docker I shy away from the idea of downloading binary images (we do this by relying on a distro, but there is a level of assurance that the binaries from a distro have been generated safely). Have we given consideration to this?

That's a fair point.

Another downside is it breaks people running on on-x86 systems. This detail is less of a blocker as we already have this problem with the nightly clang images being x86-only.

I think the current cl makes it so that you can still invoke driver on a non-x86 host. Is the issue that the docker image is not garunteed to work across architectures? If so, could we make an additional ppc docker image that is NOT used on travis (since their infra is only x86_64 to my knowledge), but for folks prefering/using docker+ppc?

[nathanchance]

So not having a custom PPA we need to fetch from, just our custom build in our repo? Seems good, in case the PPA accidentally disappears.

Yes, the QEMU binaries are built and installed during the Dockerfile build and the image will be updated daily so there will never be dependency issues (because as noted in b7ee15e, the QEMU binaries are built on Ubuntu).

Probably don't even need a full blown debian distro, could roll our own. Definitely out of scope for V0. Do you have any measurements what the image size is AFTER this change, for comparison?

I don't but I will take a look tonight/tomorrow. I have noticed that the builds on average with a 100% ccache hit rate seem to be about a minute faster with this change.

I don't follow; we can't download build artifacts from Travis, so the time it takes to build the docker image in travis doesn't matter. Or do you mean having more time to run continuous-integration on travis?

I made the comment more thinking down the line if we ever need to build something that is out of date with Debian, we wouldn't be cutting into the build time of this repo because the Dockerfile is in another one so we have a full 50 minutes to do whatever we need to there.

Would you mind moving these to our organization? I trust everyone in the org with the keys, but I would like to prevent departures from github from affecting our infra.

Absolutely, just wanted to get your approval for it first :)

Done: https://github.com/ClangBuiltLinux/dockerimage

With the above change in repo, I'm cool with you setting up the docker org, and trusting you with the responsibility of releasing updates to the docker image. Maybe you could share the pw privately with me and Joel should we ever need it?

https://hub.docker.com/r/clangbuiltlinux/

I can add you guys as owners of the organization and you should be able to manage it too.

When using Docker I shy away from the idea of downloading binary images (we do this by relying on a distro, but there is a level of assurance that the binaries from a distro have been generated safely). Have we given consideration to this?

Hmmm not sure I understand exactly. What binary images are we downloading?

I think the current cl makes it so that you can still invoke driver on a non-x86 host.

Well I guess the driver.sh change could break other environments (not architecture specific but worth noting) but that line has always been tailored for our setup. If it needs to be something different, we should have it be an environment variable with a sane default.

Is the issue that the docker image is not garunteed to work across architectures? If so, could we make an additional ppc docker image that is NOT used on travis (since their infra is only x86_64 to my knowledge), but for folks prefering/using docker+ppc?

Yeah it's not (I just tested on my Raspberry Pi) so I think having various architecture images wouldn't be a bad idea (link to look into later). I guess QEMU would need to be cross compiled in those cases.

[nickdesaulniers]

https://hub.docker.com/u/ndesaulniers/ is my account.

[nathanchance]

https://hub.docker.com/u/ndesaulniers/ is my account.

Done.

[nickdesaulniers]

LGTM, let's make sure to resolve any issues @shenki foresees, or at least file some TODO's in the dockerimage issue tracker.

[shenki]

When using Docker I shy away from the idea of downloading binary images (we do this by relying on a distro, but there is a level of assurance that the binaries from a distro have been generated safely). Have we given consideration to this?

That's a fair point.

Another downside is it breaks people running on on-x86 systems. This detail is less of a blocker as we already have this problem with the nightly clang images being x86-only.

I think the current cl makes it so that you can still invoke driver on a non-x86 host. Is the issue that the docker image is not garunteed to work across architectures?

Right, the Docker image will not work on ppc64 as it is now a binary that is built and published for x86 only (where as before it was dynamically generated).

If so, could we make an additional ppc docker image that is NOT used on travis (since their infra is only x86_64 to my knowledge), but for folks prefering/using docker+ppc?

That would help.

(The ppc64le support for Travis is in soft release. eg: https://github.com/grooverdan/hello-world/blob/master/.travis.yml#L5)

re: qemu, I would prefer we help Debian update their packaging instead of working around it by building locally. Local builds work around the problem for this project, but it does not help the wider ecosystem. I have engaged some Debian developers to see if what can be done to help.

For example, I did this for the skiboot package and the Debian maintainer has now published a new version:

https://metadata.ftp-master.debian.org/changelogs/main/s/skiboot/skiboot_6.1-3_changelog

[nathanchance]

Right, the Docker image will not work on ppc64 as it is now a binary that is built and published for x86 only (where as before it was dynamically generated).

Fair point, I hadn't considered that. However, a user wanting to take advantage of it could still just run docker build . and get an identical image for their architecture, which is almost identical to how it is now (docker run -ti debian + debian-setup.sh). I feel like not having non-x86 binaries out of the gate shouldn't be a merge blocker.

re: qemu, I would prefer we help Debian update their packaging instead of working around it by building locally. Local builds work around the problem for this project, but it does not help the wider ecosystem. I have engaged some Debian developers to see if what can be done to help.

I mean those two things don't have to be mutually exclusive. We can work on getting their package updated and build locally in the meantime. Hopefully, they will update the package after we make it clear that we need an updated version. If you wanted to absolutely minimize our delta with Debian, we could install qemu-system-arm and qemu-system-x86 from the official repos then just build qemu-system-ppc from source.

[nickdesaulniers]

I feel like not having non-x86 binaries out of the gate shouldn't be a merge blocker.

I agree. I've filed ClangBuiltLinux/dockerimage#1.

I mean those two things don't have to be mutually exclusive. We can work on getting their package updated and build locally in the meantime.

Porque no los dos? 🌮 😋

I would prefer we help Debian update their packaging instead of working around it by building locally.

How do we contact the current maintainer?

[nathanchance]

If you wanted to absolutely minimize our delta with Debian, we could install qemu-system-arm and qemu-system-x86 from the official repos then just build qemu-system-ppc from source.

I've given this some thought and I think this is probably a reasonable way to ensure that we stay close in line with Debian: ClangBuiltLinux/dockerimage@0706b50

Additionally...

• I've adopted the --no-install-recommends flag from debian-setup: Install clang-8, lld-8 and llvm-8 with --no-install-rec… #67: ClangBuiltLinux/dockerimage@b47138f

• Only installed GCC for building QEMU then uninstalling it (to close verify we can build WITHOUT GCC in the env #66, requires removing command -v gcc from driver.sh, which I will push here after the image is updated): ClangBuiltLinux/dockerimage@530b675

For example, I did this for the skiboot package and the Debian maintainer has now published a new version:

https://metadata.ftp-master.debian.org/changelogs/main/s/skiboot/skiboot_6.1-3_changelog

I've now switched to this: ClangBuiltLinux/dockerimage@1fa7a8f

Assuming there are no major issues, I will pull request these tonight and push the changes needed in this repo after that (can be viewed here: https://github.com/nathanchance/continuous-integration/commits/prebuilt-dockerimage-after-review)

[nickdesaulniers]

@shenki do you have reservations that have not been satisfied with this PR or filed issues?

[shenki]

Apologies for the radio silence. I've been off on another project.

I don't think we had any issues with the using the PPA as we were. The advantage of sticking with packaged software is when the maintainer updates Debian/Ubuntu, we would see the new package included in the image without doing any work.

I preferred the previous setup where an image was built at runtime, as the idea of downloading a preconstructed image worries me. However given we are only using this for throw away builds to spot build errors, I think the risk is acceptable.

I appreciate the hard work that has gone in to this change, and if consensus to move in that direction then we should merge it. Thanks @nathanchance!

[nathanchance]

@tpimh's suggestion to move to Ubuntu Bionic/Cosmic in #34 would allow us to continue using the PPA and potentially make creating multi-arch containers easier but that's something that can be discussed in the dockerimage repo (current version of that change) because that's where it would be merged. I'm happy with this change as is if everyone else is!