Explore creating a Docker container specifically for building the kernel with Clang

[nathanchance]

Michal Simek mentioned that we would download a container to test building the kernel with Clang: https://lore.kernel.org/lkml/9d52cd84-440f-9a40-d0ec-1397abb24c98@xilinx.com/

Reading over the Dockerfile documentation and Olof Johansson's kernelbuilder, I think it would be relatively easy to just take our current debian-setup.sh and package it up into a Docker image on the hub that we refresh daily (with the nightly Clang builds) so that people have an easy way to build the kernel with Clang without messing with their setup. We could also use it for CI, which Stefan had suggested on the introduction commit (15a2288).

I'll take look at this after work today, this is just to elicit comments/feedback before I dive in.

[nathanchance]

Seems relatively easy: ClangBuiltLinux/dockerimage@a7b98c6

We don't necessarily have to replace our current setup with debian-setup.sh (this could just be something additional) but if we wanted to, it would look something like this: nathanchance@807ddc5

We would probably want the image in a ClangBuiltLinux org on Docker's hub but other than that, works great: https://travis-ci.com/nathanchance/continuous-integration/builds/90663742

[nathanchance]

Looks like pushing the images to Docker's hub is super easy via Travis as well:

https://docs.travis-ci.com/user/docker/#pushing-a-docker-image-to-a-registry

ClangBuiltLinux/dockerimage@0ebe3a9

https://travis-ci.com/nathanchance/cbl-dockerimage/builds/90724882

[nickdesaulniers]

I'm guess that will significantly cut down on the time we spend installing dependencies, which I suspect are a majority of our test times with a hot cache? For example, a build with near 100% cache hits takes about 1min to compile the kernel, and 30s to boot test. But the total test time is around 4-5min. That's 2.5-3.5min to install dependencies!

[nathanchance]

Building QEMU is extremely simple and doesn't take a ton of time. We would avoid the hassle of keeping a PPA up to date and making sure dependencies are all good (and we can revert back to Debian stable so that nothing unexpectedly blows up).

Dockefile change: ClangBuiltLinux/dockerimage@e8352c1

Dockefile Travis build: https://travis-ci.com/nathanchance/cbl-dockerimage/jobs/157439564

Successful Travis build from this repo: https://travis-ci.com/nathanchance/continuous-integration/builds/90806728 (commit: nathanchance@903c412)

I'm guess that will significantly cut down on the time we spend installing dependencies, which I suspect are a majority of our test times with a hot cache? For example, a build with near 100% cache hits takes about 1min to compile the kernel, and 30s to boot test. But the total test time is around 4-5min. That's 2.5-3.5min to install dependencies!

I am not sure that it will save a ton of time because downloading/unpacking the image will take a bit given it's grown quite a lot as there are a lot of dependencies to the few packages that we actually need (88.09s, 99.97s, 87.78s in the build above, we don't currently time debian-setup.sh so I can't empirically compare but that doesn't seem unreasonable) BUT it gives us the flexibility to build things from source reliably (like QEMU) without sacrificing time in our regular builds.

Takes about 9-10 minutes to build a Docker image (which would happen daily one time in its own Travis build) and about 1-2 minutes to download and set it up. Beats downloading, installing, and building any dependencies that we need every single build.

The other benefit is having a unified environment that we can use to get others easily involved. It's easy to tell people to grab Docker (which has easy setup instructions), run docker run --rm -ti <name_of_repo>, clone torvalds/linux, then run make CC=clang-8 ..., regardless of distro or experience.

[nickdesaulniers]

BUT it gives us the flexibility to build things from source reliably (like QEMU)

I'm not sure we need/want to be building QEMU from source, or even most recent builds (in my experience, the top of tree was not green/didn't build). I'd rather conditionally fetch a newer QEMU from @shenki if ARCH=ppc64le. And once QEMU 3 gets packaged, we can drop the custom PPA.

The other benefit is having a unified environment that we can use to get others easily involved.

...if they use docker. It does look pretty simple, though. And I don't think local workflows with driver.sh would change, even if you're not using docker, right?

And I guess that would make it easier for users of non-debian-based distros to get top of tree clang, right? Like they can run the docker image from say Arch Linux, and have a working clang-8 nightly build within the docker image?

If we can:

1. make it easier for users of non-debian-based distros to test with.
2. not break existing local workflow.
3. not trash build time.
   then it would be hard to say no. Number 2 is not a hard requirement, and 3 has some flexibility.

So docker lets you publish docker images to a repo that anyone can fetch from? Would we be able to have one for the github group? Guess I should read up on 🐋

[nathanchance]

I'm not sure we need/want to be building QEMU from source, or even most recent builds (in my experience, the top of tree was not green/didn't build).

It would be better to build from the 3.0.0 tarball instead of from git master, fair point. I'll change that.

I'd rather conditionally fetch a newer QEMU from @shenki if ARCH=ppc64le.

His PPA already provides QEMU 3.0 for arm, arm64, and x86_64, which have been shown to work. The benefit of building from source and installing every time is moving back to Debian stable (not that unstable is crazy unstable but I've known a few people who have had nasty breakage on it) because we don't need to build QEMU on Ubuntu and make it fit Debian. It also makes sure that its dependencies never come back to bite us. I'm not necessarily thinking ahead to the next few weeks but the next few months, where new versions and such could cause problems. It would also probably fix #38 because we run make install instead of packaging up a deb file (EDIT: although we'd still would want to get that fixed in Debian upstream):

root@b38b715ef70c:/# ls /usr/local/share/qemu/skiboot.lid
/usr/local/share/qemu/skiboot.lid

EDIT #2: Upstream skiboot works fine it seems according to Travis

https://travis-ci.com/nathanchance/continuous-integration/jobs/157571857

nathanchance@3edb8ed

And once QEMU 3 gets packaged, we can drop the custom PPA.

If QEMU 3 gets packaged given that it sounds like QEMU is fairly neglected on Debian.

...if they use docker. It does look pretty simple, though. And I don't think local workflows with driver.sh would change, even if you're not using docker, right?

Correct, driver.sh hasn't changed at all (which is the super neat thing, imo).

And I guess that would make it easier for users of non-debian-based distros to get top of tree clang, right? Like they can run the docker image from say Arch Linux, and have a working clang-8 nightly build within the docker image?

Yes and even Debian/Ubuntu users will probably like it because they aren't installing a bunch of packages onto their system that they might not want permanently.

If we can:

1. make it easier for users of non-debian-based distros to test with.
2. not break existing local workflow.
3. not trash build time.
   then it would be hard to say no. Number 2 is not a hard requirement, and 3 has some flexibility.

All three of those are true.

So docker lets you publish docker images to a repo that anyone can fetch from? Would we be able to have one for the github group? Guess I should read up on 🐋

Yes, we could make a ClangBuiltLinux organization on there. For right now, it's on my user and you can run it with docker run --rm -ti nathanchance/cbl.

[nathanchance]

With these two Dockerfile changes, I've slimmed about 400MB off of the compressed image and it now takes a little over a minute to download the image (57.72s, 68.21s, 67.2s, 68s):

ClangBuiltLinux/dockerimage@7eaaf32

ClangBuiltLinux/dockerimage@906c897

https://travis-ci.com/nathanchance/continuous-integration/builds/90876837

[nathanchance]

Hmmm I just noticed that Stretch nightly binaries haven't been updated for two weeks so we wouldn't be able to make the switch back until that is fixed :/

https://llvm-jenkins.debian.net/view/Debian%20Stretch/job/llvm-toolchain-stretch-binaries/

I guess this should be reported to the LLVM bug tracker according to apt.llvm.org?

[dileks]

On Mon, Nov 12, 2018 at 5:59 AM Nathan Chancellor ***@***.***> wrote: Hmmm I just noticed that Stretch nightly binaries haven't been updated for two weeks so we wouldn't be able to make the switch back until that is fixed :/ https://llvm-jenkins.debian.net/view/Debian%20Stretch/job/llvm-toolchain-stretch-binaries/ I guess this should be reported to the LLVM bug tracker according to apt.llvm.org?

[ CC Sylvestre Debian llvm-toolchain maintainer ] Looks like Debian/experimental has newer clang-8 packages. root@iniza:~# LC_ALL=C apt-cache policy clang-8 clang-8: Installed: (none) Candidate: 1:8~svn346586-1~exp1 Version table: 1:8~svn346586-1~exp1 99 99 https://incoming.debian.org/debian-buildd buildd-experimental/main amd64 Packages 99 http://deb.debian.org/debian experimental/main amd64 Packages 1:8~svn345496-1~exp1+0~20181029105533.852~1.gbpf10f36 99 99 http://apt.llvm.org/stretch llvm-toolchain-stretch/main amd64 Packages 1:8~svn343154-1 99 99 http://deb.debian.org/debian unstable/main amd64 Packages Sylvestre was very helpful in answering llvm-toolchain debian-specific questions. - Sedat -

…

[nathanchance]

Thanks. My goal is to move us back to Debian stable to make sure that the environment never becomes an issue with our CI setup but that can't happen if we aren't getting nightly updates to LLVM/Clang.