Jenkinsfile: test building Debian 11 "bullseye" by thaJeztah · Pull Request #213 · docker/containerd-packaging

thaJeztah · 2021-01-12T13:00:15Z

Depends on:

deb: make dh-systemd dependency optional as it's deprecated #212 deb: make dh-systemd dependency optional as it's deprecated
deb: keep '/var/lib/apt/lists/' to allow building for Debian unstable #219 deb: keep '/var/lib/apt/lists/' to allow building for Debian unstable

thaJeztah · 2021-02-19T14:06:12Z

Hmm.. interesting failure;


#24 144.2 dpkg-deb: building package 'containerd.io' in '../containerd.io_0.20210219.014044~e58be59-1_amd64.deb'.
#24 144.2 dpkg-deb: building package 'containerd.io-dbgsym' in '../containerd.io-dbgsym_0.20210219.014044~e58be59-1_amd64.deb'.
#24 184.7  dpkg-genbuildinfo
#24 185.1  dpkg-genchanges  >../containerd.io_0.20210219.014044~e58be59-1_amd64.changes
#24 185.1 dpkg-genchanges: warning: the current version (0.20210219.014044~e58be59-1) is earlier than the previous one (1.4.3-1)
#24 185.4 dpkg-genchanges: info: including full source code in upload
#24 185.4  dpkg-source --after-build .
#24 185.5 dpkg-buildpackage: info: full upload; Debian-native package (full source is included)
#24 185.6 renamed '../containerd.io-dbgsym_0.20210219.014044~e58be59-1_amd64.deb' -> '/build/debian/n/a/amd64/containerd.io-dbgsym_0.20210219.014044~e58be59-1_amd64.deb'
#24 185.6 renamed '../containerd.io_0.20210219.014044~e58be59-1_amd64.deb' -> '/build/debian/n/a/amd64/containerd.io_0.20210219.014044~e58be59-1_amd64.deb'
#24 185.6 tar (child): /archive/debian-n/a-amd64.tar.gz: Cannot open: No such file or directory
#24 185.6 debian/n/a/amd64/
#24 185.6 debian/n/a/amd64/containerd.io_0.20210219.014044~e58be59-1_amd64.deb
#24 185.6 tar (child): Error is not recoverable: exiting now
#24 185.6 tar: /archive/debian-n/a-amd64.tar.gz: Wrote only 4096 of 10240 bytes
#24 185.6 tar: Child returned status 2
#24 185.6 tar: Error is not recoverable: exiting now
#24 ERROR: executor failed running [/bin/sh -c /root/build-deb]: exit code: 2

thaJeztah · 2021-02-19T14:19:20Z

Looks like the failure is somewhere in this region;

renamed '../containerd.io-dbgsym_0.20210219.014044~e58be59-1_amd64.deb' -> '/build/debian/n/a/amd64/containerd.io-dbgsym_0.20210219.014044~e58be59-1_amd64.deb'

It's renaming it to /build/debian/n/a/amd64... (notice the n/a in there, which looks like some info is "not available" (n/a)

Some other warnings;

probably expected (as we're creating a "pseudo version", and that's before the latest entry in "changelog"

#24 210.1  dpkg-genchanges  >../containerd.io_0.20210219.014044~e58be59-1_arm64.changes
#24 210.2 dpkg-genchanges: warning: the current version (0.20210219.014044~e58be59-1) is earlier than the previous one (1.4.3-1)

Wondering if the "may not have a revision" is related

#24 1.534  dpkg-source -b .
#24 1.742 dpkg-source: warning: native package version may not have a revision

This warning is harmless, but perhaps something we should fix to reduce noise (use containerd.io instead of containerd as working directory)

#24 1.742 dpkg-source: warning: source directory 'containerd' is not <sourcepackage>-<upstreamversion> 'containerd.io-0.20210219.014044~e58be59'
#24 1.742 dpkg-source: info: using source format '1.0'
#24 1.742 dpkg-source: info: building containerd.io in containerd.io_0.20210219.014044~e58be59-1.tar.gz
#24 1.757 dpkg-source: info: building containerd.io in containerd.io_0.20210219.014044~e58be59-1.dsc

thaJeztah · 2021-02-19T14:37:33Z

Looking if the problem is in this area;

containerd-packaging/scripts/build-deb

Lines 51 to 61 in 1da2018

    
           DIST_ID="$(. /etc/os-release; echo "${ID}")" 
        
           DIST_VERSION=$(lsb_release -sc) 
        
           ARCH=$(dpkg --print-architecture) 
        
           DEST_DIR="/build/${DIST_ID}/${DIST_VERSION}/${ARCH}/" 
        
           mkdir -p "${DEST_DIR}" 
        
           mv -v ../*.deb "${DEST_DIR}" 
        
           # Only create an archive if env variable is specified 
        
           if [ -n "${CREATE_ARCHIVE}" ]; then 
        
           	tar -C /build -cvzf "/archive/${DIST_ID}-${DIST_VERSION}-${ARCH}.tar.gz" "${DIST_ID}/${DIST_VERSION}/${ARCH}" 
        
           fi

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ apt-get update -q && apt-get install -q -y lsb-release
$ lsb_release -sc
bullseye

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing
Codename:	bullseye

thaJeztah · 2021-02-19T14:54:23Z

Ok, confirmed that build passes without CREATE_ARCHIVE=1, and fails if that's set, so the problem is related to that code

thaJeztah · 2021-02-19T16:08:11Z

So I tried running lsb_release in a debian:bullseye container, and it worked. Then tried running it in the build-env stage of the Dockerfile, and there I was able to reproduce. So I did some digging;

reproduction steps

$ docker run --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye sh -c 'apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && lsb_release -a && lsb_release -sc'

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing
Codename:	bullseye
bullseye

$ docker run --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye sh -c 'apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && apt-get clean && lsb_release -a && lsb_release -sc'

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing
Codename:	bullseye
bullseye

$ docker run --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye sh -c 'apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && rm -rf /var/cache/apt && lsb_release -a && lsb_release -sc'

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing
Codename:	bullseye
bullseye

$ docker run --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye sh -c 'apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && rm -rf /var/lib/apt/lists/* && lsb_release -a && lsb_release -sc'

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing/unstable
Codename:	n/a
n/a

Interesting! Looks like removing /var/lib/apt/lists/* causes this:

OK?	`lsb_release -sc`	`apt-get clean`	`rm -rf /var/cache/apt`	`rm -rf /var/lib/apt/lists/*`
✅	`bullseye`	-	-	-
✅	`bullseye`	✅	-	-
✅	`bullseye`	-	✅	-
❌	`n/a`	-	-	✅

thaJeztah · 2021-02-19T16:27:00Z

Further testing to see what file is needed for this to work;

$ docker run -it --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye

$ apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && apt-get clean && rm -rf /var/cache/apt
$ lsb_release -sc
bullseye

$ ls -l /var/lib/apt/lists/
total 17204
drwxr-xr-x 2 _apt root     4096 Feb 19 16:09 auxfiles
-rw-r--r-- 1 root root    40083 Feb 19 14:10 deb.debian.org_debian_dists_bullseye-updates_InRelease
-rw-r--r-- 1 root root   123326 Feb 19 14:10 deb.debian.org_debian_dists_bullseye_InRelease
-rw-r--r-- 1 root root 17380409 Feb 19 13:56 deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages.lz4
-rw-r----- 1 root root        0 Feb 19 16:09 lock
drwx------ 2 _apt root     4096 Feb 19 16:10 partial
-rw-r--r-- 1 root root    44114 Feb 19 15:41 security.debian.org_debian-security_dists_bullseye-security_InRelease

$ rm -rf /var/lib/apt/lists/auxfiles /var/lib/apt/lists/partial /var/lib/apt/lists/lock
$ lsb_release -sc
bullseye

$ rm /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_InRelease
$ lsb_release -sc
bullseye

$ rm /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_InRelease
$ lsb_release -sc
bullseye

So none of the above are needed; remaining two files:

$ ls -l /var/lib/apt/lists/
total 17112
-rw-r--r-- 1 root root   123326 Feb 19 14:10 deb.debian.org_debian_dists_bullseye_InRelease
-rw-r--r-- 1 root root 17380409 Feb 19 13:56 deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages.lz4

And removing either of these causes lsb_release -sc to fail (print n/a);

$ rm /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_InRelease
$ lsb_release -sc
n/a

(in a separate try)

$ rm /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages.lz4
$ lsb_release -sc
n/a

Steps to reproduce

$ docker run -it --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye

$ apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && apt-get clean && rm -rf /var/cache/apt
$ lsb_release -sc
bullseye

$ rm /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_InRelease
$ lsb_release -sc
n/a

$ docker run -it --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye

$ apt-get update -qq && apt-get install -qq -y --no-install-recommends lsb-release > /dev/null && apt-get clean && rm -rf /var/cache/apt
$ lsb_release -sc
bullseye

$ rm /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages.lz4
$ lsb_release -sc
n/a

thaJeztah · 2021-02-19T16:39:31Z

So yes, the deb.debian.org_debian_dists_bullseye_InRelease file does contain "a" codename. Not sure why it needs both that one and the .lz4 file (for verification)?

$ docker run --rm -e DEBIAN_FRONTEND=noninteractive debian:bullseye sh -c 'apt-get update -qq && cat /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_InRelease | head -n 15'

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Origin: Debian
Label: Debian
Suite: testing
Codename: bullseye
Changelogs: https://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog
Date: Fri, 19 Feb 2021 14:09:29 UTC
Valid-Until: Fri, 26 Feb 2021 14:09:29 UTC
Acquire-By-Hash: yes
No-Support-for-Architecture-all: Packages
Architectures: all amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
Components: main contrib non-free
Description: Debian x.y Testing distribution - Not Released

thaJeztah · 2021-02-19T16:45:24Z

possibly related discussion; https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845651
but that's about /etc/os-release having incorrect information. Still it's surprising that it doesn't "persist" the codename beyond the InRelease file 🤔

thaJeztah · 2021-02-19T17:33:03Z

Right, so following that thread, this is indeed "by design";

Debian "unstable" releases use apt caching information to get the codename see discussion on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845651:

That's all to say that this bug is (to my belief) actually expected behaviour;
and fixing it through forcing the codename to be interpreted as "stretch" when
apt-cache information is unavailable would be wrong. When /etc/debian_version
contains "potato/sid", the codename is either potato xor sid, and only apt-
cache can discriminate a testing host from a sid host. Therefore, in such a
situation, the correct answer is actually "I can't tell", aka "n/a".

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the debian_bullseye branch from f84a1f7 to 8d48596 Compare January 25, 2021 17:43

thaJeztah mentioned this pull request Feb 19, 2021

deb: add support for building debian 11 "Bullseye" docker/docker-ce-packaging#521

Merged

4 tasks

thaJeztah force-pushed the debian_bullseye branch from 8d48596 to 3b8aa3f Compare February 19, 2021 13:57

thaJeztah marked this pull request as ready for review February 19, 2021 13:58

thaJeztah mentioned this pull request Feb 19, 2021

deb: keep '/var/lib/apt/lists/' to allow building for Debian unstable #219

Merged

thaJeztah force-pushed the debian_bullseye branch from 3b8aa3f to 9056823 Compare February 19, 2021 17:38

Jenkinsfile: test building Debian 11 "bullseye"

928df20

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the debian_bullseye branch from 9056823 to 928df20 Compare February 22, 2021 18:38

tiborvass approved these changes Feb 23, 2021

View reviewed changes

tiborvass merged commit ef81e81 into docker:master Feb 23, 2021

thaJeztah deleted the debian_bullseye branch February 23, 2021 18:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jenkinsfile: test building Debian 11 "bullseye"#213

Jenkinsfile: test building Debian 11 "bullseye"#213
tiborvass merged 1 commit into
docker:masterfrom
thaJeztah:debian_bullseye

thaJeztah commented Jan 12, 2021 •

edited

Loading

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021 •

edited

Loading

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021 •

edited

Loading

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

thaJeztah commented Jan 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Feb 19, 2021

Steps to reproduce

Uh oh!

thaJeztah commented Feb 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

thaJeztah commented Feb 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thaJeztah commented Jan 12, 2021 •

edited

Loading

thaJeztah commented Feb 19, 2021 •

edited

Loading

thaJeztah commented Feb 19, 2021 •

edited

Loading