Prebuilt and dynamically linked binaries released somewhere as artifacts

[thesayyn]

You are probably sick of seeing this coming up over and over again and I apologize for bringing this up yet again. I am just curious about whether the proposed approach below would work for everyone.

Before I begin, I want to express that I am well aware of the conversation that went on #1478 #755 #932 #670

Use case

Recently, I started working on https://github.com/thesayyn/rules_container which is a bazel rule for building container images. It utilizes skopeo to fetch OCI and DockerV2 images and lay them out in the bazel-out tree as OCI layout. Then we use umoci to manipulate the OCI layout (adding cross-platform layers etc.) that resides in the bazel-out tree that we fetched previously. All is well and works perfectly.

Problem

By the nature of bazel, everything is hermetic and reproducible. but what does this mean? well, it means that bazel has to know everything that goes into the pot (aka: targets) to determine whether a compilation unit is affected. this includes tools that are part of the compilation; which is skopeo in this case. hope all is well to this point because things are gonna complicated.

there are a few ways to make skopeo part of the compilation;

1. We could rely on users to get skopeo despite being non-deterministic and impractical way.
   This is the only way forward we can go currently as skopeo is only distributed as container images and proprietary packages (deb etc).
   This method has a few drawbacks. 1) it is non-hermetic as two distinct people can have different versions of skopeo installed on their machine thus leading to different outputs and hashes. breaks the determinism and this is big. 2) users have to fetch skopeo for every building environment.

2. Fetching the skopeo repository under bazel and building it via prefetched go toolchain.
   This is the most feasible way in terms of determinism and hermeticity as we will be fetching a pinned version of the repository and building it via a deterministic go toolchain. everything might sound all right when looked at from the maintainers of skopeo maintainers, unfortunately, this way is also sub-optimal as it has a great overhead of fetching the toolchain and building the source each time. also, we will have our own version of BUILD files (similar to Makefiles) and they can break at any given upgrade.

3. Fetching prebuilt binaries from Github and comparing checksums
   This is the most favorable and optimal way to fetch external tools within the bazel ecosystem. Basically, we tell the bazel where to look for platform-specific binaries and their checksums. then bazel fetches them for us and checks if the binary was tempered with (addresses the concern Publish binary releases to GitHub #1478 (comment)) as we had the checksums declared beforehand. No overhead of fetching go toolchain and building the source. no risk of being disconnected from the upstream repository.

Intention

No matter what we do, we will have to invest in skopeo so it is a matter of whether we do it in a disconnected manner which wouldn’t work well enough or we do it by investing right here in the upstream repository.

Proposal

I am no linux expert here, and I only have a little experience with the ld so please correct me if I am wrong or missing any points. statically linked binaries have come up a lot of times but were rejected due to its limitation so I'm not gonna propose that again. I have no intent on creating a maintenance burden to maintainers of this project.

I believe most of skopeo's functionality is in go with a few exceptions like libdevmapper, libgpgme. it is unfortunate that we can't use the rest of the functionality due to a few dynamically linked libs.

My proposal is to publish dynamically linked and let the users be responsible for bringing in the shared libraries. Would be even better if we can make them optional and warn the users when they try to use a feature that depends on these shared libraries. perhaps with ldconfig on Linux or something similar (gcc option --as-needed). this would prevent problems such as #1336. I am not sure if this is practical/feasible or not as the source is written in go not c. perhaps LDFLAGS would help here.

I am available to get the necessary work done if this sounds feasible. I will do my best to introduce the smallest patch possible. I have no intention of putting an extra maintenance burden onto maintainers.

My two cents

One of the goals of skopeo is to build images without needing a container runtime daemon present, which we share with rules_container as well. we do not ever want to rely on a container runtime to build images so we can't really use skopeo images distributed on quay.io. On the other hand, images are not really useful if you are not using them as your CI executor image or k8s pod image.

[thesayyn]

Side note: if anything brokes at any given time, rules_container users will be the first ones to discover it and report it to rules_container. so fixing this would be solely our job and skopeo maintainers will not have to deal with it all the time.

[mtrmac]

Thanks for your report.

By the nature of bazel, everything is hermetic and reproducible. but what does this mean? well, it means that bazel has to know everything that goes into the pot (aka: targets) to determine whether a compilation unit is affected.

Does this include a glibc version? The shell version? If so, they presumably come from some distribution, and are “hermetic and reproducible” using some acceptable mechanism. Why can’t Skopeo come from the same place and use the same mechanism?

If it doesn’t include the shell version, I’m not sure that Skopeo needs to be fixed more than the shell (admittedly, most shells have much more history and trust to them, but still…).

Fetching prebuilt binaries from Github and comparing checksums
…
My proposal is to publish dynamically linked and let the users be responsible for bringing in the shared libraries.

If this is acceptable, is wget https://download.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/Packages/s/skopeo-1.4.1-1.fc35.x86_64.rpm | rpm2cpio | … acceptable?

Prebuilt binaries already exist, and are well-maintained with the intent to be compatible with specific distributions — just not on GitHub.

Especially if the base system is supposed to be 100% reproducible, then it’s known what distribution it is built on — so this is an easier problem than the “I need a magic static binary that works on every distribution” wish that comes up from time to time.

Or is the base OS somehow out of scope of reproducibility, but Skopeo a part of the required-reproducible tool set?

My proposal is to publish dynamically linked and let the users be responsible for bringing in the shared libraries. Would be even better if we can make them optional and warn the users when they try to use a feature that depends on these shared libraries.

It’s technically feasible for the rarely used dependencies like device-mapper, via dlopen(3) and the like (although pretty ugly code and likely to break without anyone noticing on ABI breaks, i.e. it would require a comprehensive automated ABI test) — but it isn’t practical for glibc (NSS modules) for user and host name lookups, because they happen on pretty much every single run. Glibc does not change sonames, but it does introduce new symbol versions fairly frequently — OTOH I haven’t checked how frequently (or if at all) that affects Skopeo binaries.

[thesayyn]

It’s technically feasible for the rarely used dependencies like device-mapper, via dlopen(3) and the like (although pretty ugly code and likely to break without anyone noticing on ABI breaks, i.e. it would require a comprehensive automated ABI test) — but it isn’t practical for glibc (NSS modules) for user and host name lookups, because they happen on pretty much every single run. Glibc does not change sonames, but it does introduce new symbol versions fairly frequently — OTOH I haven’t checked how frequently (or if at all) that affects Skopeo binaries.

Bazel is pretty much capable of bringing in the libc hermetically via cc_toolchain as far as I can tell. Even if we get mismatching libc version, bringing in the correct libc is on the table.

Does this include a glibc version? The shell version? If so, they presumably come from some distribution, and are “hermetic and reproducible” using some acceptable mechanism. Why can’t Skopeo come from the same place and use the same mechanism?

bazel is capable of bringing in the libc in a hermetic way. though not sure about shell.

Especially if the base system is supposed to be 100% reproducible, then it’s known what distribution it is built on — so this is an easier problem than the “I need a magic static binary that works on every distribution” wish that comes up from time to time.

bazel is still working on its cross platform reproducibility but it provides strong guarantees around darwin & linux and the gcc toolchain.

If this is acceptable, is wget https://download.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/Packages/s/skopeo-1.4.1-1.fc35.x86_64.rpm | rpm2cpio | … acceptable?

I believe something could be done to go with this approach but 1) we would be downloading unnecessary bits; not that it is so important i felt like mentioning it 2) i believe it would be still lacking the support for darwin which is an important platform for us atm.

Or is the base OS somehow out of scope of reproducibility, but Skopeo a part of the required-reproducible tool set?
Base os rarely effects your builds because of sandboxing feature of bazel. Unless you do something crazy to break out of the sandbox.