Allow to override the repository name and config location

[williamdes]

I did see on
That the image was built as docker and not phpmyadmin because of:

bashbrew/scripts/github-actions/generate.sh

Line 4 in b4880ac

image="${GITHUB_REPOSITORY##*/}" # "python", "golang", etc

So I did a small tweak to change this (I was thinking that that would make my config.sh work but it is not read)

  generate-jobs:
    name: Generate Jobs
    runs-on: ubuntu-latest
    outputs:
      strategy: ${{ steps.generate-jobs.outputs.strategy }}
    steps:
      - uses: actions/checkout@v3
      - id: generate-jobs
        name: Generate Jobs
        # See: https://github.com/docker-library/bashbrew/blob/b4880ac5f1bf6b9340536f1c5325afa9733bd619/scripts/github-actions/generate.sh#L4
        run: |
          git clone --depth 1 https://github.com/docker-library/bashbrew.git -b master ~/bashbrew
          export GITHUB_REPOSITORY="phpmyadmin"
          strategy="$(~/bashbrew/scripts/github-actions/generate.sh)"
          jq . <<<"$strategy" # sanity check / debugging aid
          echo "::set-output name=strategy::$strategy"

[williamdes]

I am trying to migrate our test suite into one that the Docker official repository actually runs to prevent publishing deeply broken images like it was the case recently.
But because the script was not designed for this I can not add a config.sh in my repository to make the new test run.

Test is missing in: https://github.com/williamdes/docker/runs/6832928117?check_suite_focus=true#step:7:6
Source: https://github.com/williamdes/docker/blob/master/config.sh

[tianon]

Heh, yeah - this isn't very well-documented, sorry!

It's a little bit of a pathological case (because I maintain so many different images in one repository), but you can see one example where I use this script with a custom test config in https://github.com/tianon/dockerfiles/blob/b7d4f92f6e50dd17572dd9dfd6f82060e0f856c8/.github/workflows/generate.sh

I guess it wouldn't be too unusual to have this automatically pick up test-config.sh in the target repository and avoid that really hacky bit at the end where I inject --config. 🤔

@yosifkit thoughts?

[williamdes]

I guess it wouldn't be too unusual to have this automatically pick up test-config.sh in the target repository and avoid that really hacky bit at the end where I inject --config.

That could be a great easy solution to allow test-config.sh in repositories for users to be able to test their suites until they are part of the file upstream (that I found looking quite empty for the amount of official docker images, is there not more images with tests ?)

Also why not including this in the stackbrew format ?
I would need to have different tests per image variant
Maybe we can take some inspiration from Debian packaging format: https://people.debian.org/~eriberto/README.package-tests.html

And have a global field to list test names and an override to add one per variant (or override the global setting)

[yosifkit]

have this automatically pick up test-config.sh in the target repository

This seems reasonable. I assume that would be in addition to the default config (like your dockerfiles repo) and not test-config.sh only.

looking quite empty for the amount of official docker images, is there not more images with tests ?

Most of the image-specific tests were just smoke tests added for images that we maintain directly to ensure we didn't publish something obviously broken. The general tests are just to help us catch common image problems. It is not required to have any new tests for a new image.

Also why not including this in the stackbrew format ?

I don't think we want to complicate the format with things unrelated to building. The tests are only run for PRs, not on the built image before pushing to Docker Hub.

The tests aren't meant to be comprehensive; any complex set of tests probably ought to remain part of an upstream's repo in testing the images before they become an update PR to docker-library/official-images.

[tianon]

Agreed -- also, the nature of the way all of these tests have to work are integration tests, which means they're inherently flaky. 😬

[williamdes]

I don't think we want to complicate the format with things unrelated to building. The tests are only run for PRs, not on the built image before pushing to Docker Hub.

Okay, that's enough for us
I agree they could be flakky

The tests aren't meant to be comprehensive; any complex set of tests probably ought to remain part of an upstream's repo in testing the images before they become an update PR to docker-library/official-images.

Okay, then just having a way to add tests manually will be perfect
It would avoid having custom test suites to test images

This seems reasonable. I assume that would be in addition to the default config (like your dockerfiles repo) and not test-config.sh only.

I am not sure to have understood the Dockerfiles part but that's nice we agree to have a test config

[yosifkit]

This seems reasonable. I assume that would be in addition to the default config (like your dockerfiles repo) and not test-config.sh only.

I am not sure to have understood the Dockerfiles part but that's nice we agree to have a test config

Apologies for being unclear; the "your" was meant to mean "@tianon's".

And speaking of @tianon, he already has a PR open for this: #47 🎉. Let us know what you think. ❤️

[roshkhatri]

RESOLVED - seems like I did not make them executables.

Hi, Thanks for implementing this, I was trying to build a fork of valkey-container and wanted to run the same tests for valkey-containers as redis.

I must not be understanding it correctly but it keeps giving me this error https://github.com/roshkhatri/valkey-container/actions/runs/10103405467/job/27940741234 for

error: /home/runner/work/valkey-container/valkey-container/.test/tests/valkey-basics/run.sh missing, not executable or is a directory

Can anyone help me with this please? Branch I am working on