Skip to content

[master] Makefile: add "verify" target to test install of packages#697

Merged
thaJeztah merged 32 commits into
docker:masterfrom
thaJeztah:integrate_verify
May 19, 2022
Merged

[master] Makefile: add "verify" target to test install of packages#697
thaJeztah merged 32 commits into
docker:masterfrom
thaJeztah:integrate_verify

Conversation

@thaJeztah
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah commented May 19, 2022

This integrates one of the verify scripts from our internal pacakging repository

This verify step is primarily intended to verify that dependencies are defined
correctly, and available on the given distro.

Migrated from commit a712afb008b56c7572e87b2bca81f1c50022ad2b in our internal
repo (from the ce-nightly branch, which corresponds with master in the
docker-ce-packaging repository); https://github.com/docker/release-packaging/commit/a712afb008b56c7572e87b2bca81f1c50022ad2b

Strategy taken to preserve history:

# install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
brew install git-filter-repo

cd ~/projects

# create a temporary clone of docker
git clone https://github.com/docker/release-packaging.git release_packaging_verify
cd release_packaging_verify
git checkout ce-nightly

# remove all code, except for verify and install-containerd-helpers
git filter-repo  --path verify --path install-containerd-helpers --force

# go to the target github.com/docker/docker-ce-packaging repository
cd ~/projects/docker-ce-packaging

# create a branch to work with
git checkout -b integrate_verify

# add the temporary repository as an upstream and make sure it's up-to-date
git remote add release_packaging_verify ~/projects/release_packaging_verify
git fetch release_packaging_verify

# merge the upstream code
git merge --allow-unrelated-histories --signoff -S release_packaging_verify/ce-nightly

zelahi and others added 30 commits October 21, 2019 13:49
@zelahi
[TAR-1162][ce-nightly] Backport changes for adding verify script (doc…
6dfaf9c 
…ker#435)

* * Verification scripts for rpm
* Verification scripts for deb
* Integrated verify step in release-packaging Jenkinsfile

Signed-off-by: Zuhayr Elahi <elahi.zuhayr@gmail.com>

* Merge pull request docker#439 from seemethere/remove_f31

[ce-19.03] ci: Remove fedora 31 arm64

* Revert "Merge pull request docker#439 from seemethere/remove_f31"

This reverts commit 700e7a6b75938d96ab587b889077ec6a3dec9963, reversing
changes made to caa1930f1f60c52afccd1ff06882c6e7c2200440.

* [ce-19.03] ci: Re-add saveS3 (docker#437)

[ce-19.03] ci: Re-add saveS3
@thaJeztah
Fix verify to not depend on download.docker.com
5c12ba8 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3880335d06a5960bc5754d4dc9c5c796ae06885b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
verify: use --force-depends to be more future proof
8dac84e 
Use the --force-depends in case other dependencies are
added in future, which would make dpkg -i fail.

Also bring back --no-install-recommmends to make install
slightly faster, and document the steps we're doing in
this script:

    Install the locally built packages using 'dpkg' because installing with
    'apt-get' would attempt to install dependency packages (such as the CLI)
    from download.docker.com instead of the locally built CLI package. Given
    that 'dpkg -i' does not install any dependency (but will fail if depen-
    dencies are missing), we use the '--ignore-depends' option to ignore
    packages we know to be missing at this stage, and '--force-depends' to
    only warn about any other missing dependency.

    Afterwards, we run 'apt-get install' with the '--fix-broken' option to
    trigger installation of the dependencies, which should succceed succesfully.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7da772e08ba12a09471650dc5b6d311c39470668)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Merge pull request docker#475 from thaJeztah/ce_nightly_backport_fix_…
0ac2fa9 
…verify

[ce-nightly backport] Fix verify to not depend on download.docker.com
@thaJeztah
install-containerd-helpers: use tabs for indentation
72fe7e1 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
verify: use tabs for indentation
849e2db 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
fix some linting warnings
5338978 
Move the `ARCH` variable to the containerd script, as that's where it's
used

use the `pkg_config_manager` variable, which is not strictly nescessary
(main use of the variable looks to be for `install_rpm_containerd`),
but using the variable prevents linting warnings.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
verify: install ca-certs, and use "--no-install-recommends"
68e8179 
ca-certs was not installed (or outdated) in the raspbian
images, so explicitly install it.

Also adding "--no-install-recommends" flag, to force explicitly
including packages we need, instead of the implicitly
being installed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
install-containerd-helpers: make sure ca-certificates are up-to-date
dfd278c 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
install-containerd-helpers: fix hard-coded repository for RPMs
24b2107 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Remove "components" subdirectory
6d241dc 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@silvin-lubecki
Merge pull request docker#500 from thaJeztah/remove_docker_ce_repo
c4accf8 
[ce-nightly] build from upstream repositories instead of docker-ce mono-repo
@thaJeztah
Verify: use "download-stage.docker.com" as default for deps
acd8612 
This allows running the verify step in situations where containerd.io
has not yet been pushed to the production package repo.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@silvin-lubecki
Merge pull request docker#526 from thaJeztah/ce_nightly_verify_from_s…
943d793 
…tage

[ce-nightly] Verify: use "download-stage.docker.com" as default for deps
@thaJeztah
Remove "docker-ce" prefix/sub-directory
a306441 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@tiborvass
Merge pull request docker#535 from thaJeztah/ce_nightly_remove_stretch
7d6303c 
[ce-nightly] Fix for upstream changes, and remove EOL distros
Always use test channel when installing containerd
c839f4e 
Signed-off-by: Tibor Vass <tibor@docker.com>
@tiborvass
Merge pull request docker#553 from tiborvass/use-test-channel
855149b 
[ce-nightly] Always use test channel when installing containerd
deb/rpm: reuse bundles targets from docker-ce-packaging
9d9ebb0 
This is to avoid duplicate codepaths.
Note that this completes the move of the tar.gz logic from
release-packaging to docker-ce-packaging only for the deb/rpm cases.

Static Linux, Windows and Mac were not touched.

Also I still think this whole bundles tar.gz thing is useless,
but this is what release-repo is currently consuming.

This also makes rpms work, as the path structure changed in docker-ce-packaging
to include distro.

Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah
Merge pull request docker#569 from tiborvass/bundles_from_packaging
6a476d0 
deb/rpm: reuse bundles targets from docker-ce-packaging
@thaJeztah
verify: run --version on installed binaries
70e213c 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
verify: temporarily disable docker scan --version
52cf182 
It tries to make a daemon connection, which fails in this case;

        docker scan --accept-license --version
        Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        Makefile:193: recipe for target 'verify' failed

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
verify: fix verify to account for scan-plugin on download.docker.com
960961e 
Verify was failing, because we did a test-build to staging:

        docker-ce is already the newest version (5:0.0.0-20210401100358-59fd6f0-0~ubuntu-xenial).
        docker-ce-rootless-extras is already the newest version (5:0.0.0-20210401100358-59fd6f0-0~ubuntu-xenial).
        docker-ce-cli is already the newest version (5:0.0.0-20210401100358-59fd6f0-0~ubuntu-xenial).
        The following additional packages will be installed:
          containerd.io iptables libnfnetlink0 libxtables11
        The following NEW packages will be installed:
          containerd.io iptables libnfnetlink0 libxtables11
        The following packages will be DOWNGRADED:
          docker-scan-plugin

What's happening is that;

- first, we `dpkg-install` docker-ce, docker-ce-cli, and docker-scan-plugin
  from the packages we just built
- then, we `apt-get install` to verify that dependencies (`containerd`)
  are installed

However, we have `docker-scan-plugin` 0.7 on staging, which it then tries to
install, but sees it's an older build, so a _downgrade_

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@tiborvass
Merge pull request docker#563 from thaJeztah/nightly_test_scan_plugin
f042386 
[ce-nightly] Updates to build and verify docker-scan-plugin
@thaJeztah
verify: don't attempt to upgrade packages
60abadb 
If download-stage.docker.com already contains a previous build
of docker-ce/docker-ce-cli, apt attempts to install that version,
only to discover that it's an older build (and considered a "downgrade")

This patch (hopefully) fixes that case, by not attempting to upgrade.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@tiborvass
Merge pull request docker#577 from thaJeztah/no_upgrade_verify
97eb421 
[ce-nightly] verify: don't attempt to upgrade packages
@thaJeztah
verify: remove old hack, and update comments to describe the process
fa57144 
- now that we added the '--no-upgrades' option in 60abadb,
  we no longer need the hack from 960961e
- add more comments, describing each step of the verify script, because
  it may not be obvious why we do some steps.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Merge pull request docker#579 from thaJeztah/verify_update_comments_a…
8245227 
…nd_strategy

[ce-nightly] verify: remove old hack, and update comments to describe the process
@thaJeztah
verify: fix hard-coded distro name for "sid" (Debian unstable)
5f157eb 
Debian "sid" is not an actual distro version. Sid represents the "unstable"
channel ("next stable in-progress"). Currently, it is equivalent to "bullseye",
but once "bullseye" is stable, it becomes "bookworm", "trixy", etc (see the
list of Debian releases at https://wiki.debian.org/DebianReleases).

We should fix this hard-coded override, and instead try to get this name from
information in /var/lib/apt/lists/.

Also see docker/containerd-packaging#219 for details.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
verify: fail if we try to use debian "sid" as distro name
12187a5 
We should never build packages for "sid", which is a codename for "unstable",
and instead use the codename that it's currently referring to (codename of
next stable release).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah added 2 commits May 19, 2022 16:07
@thaJeztah
Integrate release-packaging verify scripts
a18ae7e 
This integrates the verify scripts from our internal release-packaging
repository (ce-nightly branch), from commit a712afb008b56c7572e87b2bca81f1c50022ad2b

docker/release-packaging@a712afb

Strategy taken:

    # install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
    brew install git-filter-repo

    cd ~/projects

    # create a temporary clone of docker
    git clone https://github.com/docker/release-packaging.git release_packaging_verify
    cd release_packaging_verify
    git checkout ce-nightly

    # remove all code, except for verify and install-containerd-helpers
    git filter-repo  --path verify --path install-containerd-helpers --force

    # go to the target github.com/docker/docker-ce-packaging repository
    cd ~/projects/docker-ce-packaging

    # create a branch to work with
    git checkout -b integrate_verify

    # add the temporary repository as an upstream and make sure it's up-to-date
    git remote add release_packaging_verify ~/projects/release_packaging_verify
    git fetch release_packaging_verify

    # merge the upstream code
    git merge --allow-unrelated-histories --signoff -S release_packaging_verify/ce-nightly

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Makefile: add "verify" target to test install of packages
6978154 
This verify step is primarily intended to verify that dependencies
are defined correctly, and available on the given distro.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah changed the title Makefile: add "verify" target to test install of packages [master] Makefile: add "verify" target to test install of packages May 19, 2022
@thaJeztah
Copy link
Copy Markdown
Member Author

thaJeztah commented May 19, 2022

Ah, cool, so this is failing on ubuntu 18.04 "bionic", which is expected, and the reason for this PR; this shows the regression introduced in #686 (comment)

@thaJeztah thaJeztah marked this pull request as ready for review May 19, 2022 14:50
@thaJeztah
Copy link
Copy Markdown
Member Author

thaJeztah commented May 19, 2022

So, the intent is to have this merged (even though it's failing currently), so that we can verify fixes for the issue mentioned (such as #696)

@thaJeztah
Copy link
Copy Markdown
Member Author

thaJeztah commented May 19, 2022

I'll bring this one in; then we can continue debugging the problem 👍

@thaJeztah thaJeztah merged commit 9e18d43 into docker:master May 19, 2022
@thaJeztah thaJeztah deleted the integrate_verify branch May 19, 2022 14:57
@thaJeztah thaJeztah mentioned this pull request May 19, 2022
Closed
@thaJeztah
Copy link
Copy Markdown
Member Author

thaJeztah commented May 19, 2022

I guess we should also add this to the GitHub actions (which we're test-driving for x86 / amd64 builds)

@zhangyoufu zhangyoufu mentioned this pull request Sep 20, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rumpl rumpl rumpl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@thaJeztah @rumpl @zelahi @silvin-lubecki @tiborvass