Skip to content

feat: multi-arch operator, bundle, and catalog image builds - #3055

Merged
openshift-merge-bot[bot] merged 12 commits into
redhat-developer:mainfrom
polasudo:feat/multi-arch-operator-build
Jul 29, 2026
Merged

feat: multi-arch operator, bundle, and catalog image builds#3055
openshift-merge-bot[bot] merged 12 commits into
redhat-developer:mainfrom
polasudo:feat/multi-arch-operator-build

Conversation

@polasudo

@polasudo polasudo commented Jun 23, 2026

Copy link
Copy Markdown
Member

Summary

  • Replace the single-runner amd64-only CI build with a matrix strategy that builds on both ubuntu-24.04 (amd64) and ubuntu-24.04-arm (arm64)
  • Add a merge job that creates multi-arch manifest lists using docker buildx imagetools create
  • All three images (operator, operator-bundle, operator-catalog) are now published as multi-arch manifest lists

What changes

File: .github/workflows/next-container-build.yaml

The single next-build job is replaced with two jobs:

  1. build — matrix over [ubuntu-24.04, ubuntu-24.04-arm], each runner builds all 3 images for its native architecture using existing Makefile targets (image-build, bundle-build, catalog-build with PLATFORM override), then pushes with per-arch tag suffixes (e.g., operator:next-amd64)

  2. merge — downloads build markers, creates multi-arch manifest lists via docker buildx imagetools create, tags final images (e.g., operator:next), and optionally cleans up per-arch tags via Quay API

Why

The community hub image (quay.io/rhdh-community/rhdh) already ships as a multi-arch manifest list (amd64 + arm64). The operator stack is the remaining gap blocking ARM64 RHDH deployments via the operator.

No Makefile or Dockerfile changes are needed — the operator Dockerfile already uses TARGETOS/TARGETARCH build args and bundle images use FROM scratch.

Pattern

Follows the same proven multi-arch build pattern from the hub repo's next-build-image.yaml workflow (matrix runners → per-arch push → manifest merge).

Test plan

  • Both matrix runners (amd64, arm64) complete successfully
  • docker buildx imagetools inspect quay.io/rhdh-community/operator:next shows both architectures
  • podman pull --platform linux/arm64 quay.io/rhdh-community/operator:next succeeds
  • Same for operator-bundle and operator-catalog images
  • Per-arch tags are cleaned up after manifest creation

🤖 Generated with Claude Code

Related

Comment thread .github/workflows/next-container-build.yaml Fixed
@Fortune-Ndlovu

Copy link
Copy Markdown
Member

/agentic_review

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Jun 26, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 18 rules
✅ Cross-repo context
  Explored: repo: redhat-developer/rhdh (sha: 5992bddd)
  Not relevant to this PR: redhat-developer/rhdh-plugins

Grey Divider


Action required

1. Branch detection uses HEAD ✓ Resolved 🐞 Bug ≡ Correctness
Description
LATEST_NEXT is computed using git rev-parse --abbrev-ref HEAD, but the checkout step does not set
ref: and commonly results in a detached HEAD; this can resolve to HEAD and incorrectly pick
latest even on main. That mis-tags published images/manifests (next vs latest) for the main
branch.
Code

.github/workflows/next-container-build.yaml[R93-98]

+          latestNext="next"
+          # for main branch, use next tags; for 1.y branches, use :latest tags
+          if [[ $(git rev-parse --abbrev-ref HEAD) != "main" ]]; then
+            latestNext="latest"
+          fi
+          echo "LATEST_NEXT=$latestNext" >> $GITHUB_ENV
Relevance

⭐⭐⭐ High

Team accepted pinning checkout ref/SHA to avoid ambiguous HEAD in workflows (PR #2293, #3076).

PR-#2293
PR-#3076

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow derives the branch name from git, while the checkout step doesn’t pin a branch ref;
elsewhere in the repo, workflows use github.ref_name to avoid detached-HEAD ambiguity.

.github/workflows/next-container-build.yaml[34-38]
.github/workflows/next-container-build.yaml[93-99]
.github/workflows/update-rpm-lockfile.yaml[77-82]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Branch/tag selection uses `git rev-parse --abbrev-ref HEAD`, which is unreliable under `actions/checkout` because the repo may be in detached HEAD state. This can miscompute `LATEST_NEXT`.

## Issue Context
Other workflows in this repo already use `${{ github.ref_name }}` to determine the branch.

## Fix Focus Areas
- .github/workflows/next-container-build.yaml[34-38]
- .github/workflows/next-container-build.yaml[93-98]
- .github/workflows/next-container-build.yaml[191-195]

## Implementation notes
Replace the `git rev-parse --abbrev-ref HEAD` checks with `${GITHUB_REF_NAME}` (or `${{ github.ref_name }}`) and compare against `main`. Apply the same fix in both the build and merge "Prepare" steps.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Wrong SHA manifest sources ✓ Resolved 🐞 Bug ≡ Correctness
Description
The merge job creates the ${BASE_VERSION}-${SHORT_SHA} manifest list from
${BASE_VERSION}-amd64/arm64 instead of the SHA-scoped per-arch tags, so the commit-specific tag
points at the wrong image content. This also makes the pushed per-arch SHA tags unused (and cleanup
deletes tags that were never referenced).
Code

.github/workflows/next-container-build.yaml[R228-232]

+            # Create manifest list for version-sha tag
+            docker buildx imagetools create \
+              -t ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-${SHORT_SHA} \
+              ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-amd64 \
+              ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-arm64
Relevance

⭐⭐ Medium

No historical evidence found about correctness of manifest tag inputs (SHA-scoped vs version-scoped)
in workflows.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The build job pushes SHA-scoped per-arch tags, but the merge job doesn’t use them when building the
${BASE_VERSION}-${SHORT_SHA} manifest list.

.github/workflows/next-container-build.yaml[130-139]
.github/workflows/next-container-build.yaml[228-233]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The merge job tags `${BASE_VERSION}-${SHORT_SHA}` but incorrectly uses `${BASE_VERSION}-{amd64,arm64}` as sources. This makes the SHA tag not represent the commit build.

## Issue Context
The build job already pushes per-arch SHA tags `${BASE_VERSION}-${SHORT_SHA}-${arch}`.

## Fix Focus Areas
- .github/workflows/next-container-build.yaml[228-233]

## Implementation notes
Update the `docker buildx imagetools create` invocation for the `${BASE_VERSION}-${SHORT_SHA}` tag to reference:
- `${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-${SHORT_SHA}-amd64`
- `${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-${SHORT_SHA}-arm64`
(instead of `${BASE_VERSION}-amd64/arm64`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Missing set -u strictness ✓ Resolved 📘 Rule violation ☼ Reliability
Description
New GitHub Actions run scripts use set -ex (or no strict mode) but omit set -u and do not
validate required environment variables, which can mask unset-variable failures. The scripts also
use unquoted variable expansions (e.g., >> $GITHUB_ENV, ${REGISTRY_WITH_ORG}/${image}...),
increasing brittleness and ShellCheck-style compliance risk.
Code

.github/workflows/next-container-build.yaml[R185-242]

+        run: |
+          SHORT_SHA=$(git rev-parse --short HEAD)
+          echo "SHORT_SHA=$SHORT_SHA" >> $GITHUB_ENV
+          BASE_VERSION=$(grep -E "^VERSION \?=" Makefile | sed -r -e "s/.+= //")
+          echo "BASE_VERSION=$BASE_VERSION" >> $GITHUB_ENV
+
+          latestNext="next"
+          if [[ $(git rev-parse --abbrev-ref HEAD) != "main" ]]; then
+            latestNext="latest"
+          fi
+          echo "LATEST_NEXT=$latestNext" >> $GITHUB_ENV
+
+          # Check which architectures actually built
+          echo "Build markers found:"
+          ls -la /tmp/build-markers/ || echo "No markers found"
+
+      - name: Login to registry (${{ env.REGISTRY }})
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ vars.QUAY_USERNAME }}
+          password: ${{ secrets.QUAY_TOKEN }}
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
+
+      - name: Create manifest lists and push
+        run: |
+          export REGISTRY_WITH_ORG=${{ env.REGISTRY }}/${{ vars.REGISTRY_ORG }}
+          export OPERATOR_IMAGE_NAME=${{ vars.OPERATOR_IMAGE_NAME }}
+          OPERATOR_IMAGE_NAME=${OPERATOR_IMAGE_NAME:-operator}
+
+          set -ex
+
+          for image in ${OPERATOR_IMAGE_NAME} ${OPERATOR_IMAGE_NAME}-bundle ${OPERATOR_IMAGE_NAME}-catalog; do
+            echo "=== Creating manifest list for ${image} ==="
+
+            # Create manifest list for version tag
+            docker buildx imagetools create \
+              -t ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION} \
+              ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-amd64 \
+              ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-arm64
+
+            # Create manifest list for version-sha tag
+            docker buildx imagetools create \
+              -t ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-${SHORT_SHA} \
+              ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-amd64 \
+              ${REGISTRY_WITH_ORG}/${image}:${BASE_VERSION}-arm64
+
+            # Create manifest list for latestNext tag (next or latest)
+            docker buildx imagetools create \
+              -t ${REGISTRY_WITH_ORG}/${image}:${LATEST_NEXT} \
+              ${REGISTRY_WITH_ORG}/${image}:${LATEST_NEXT}-amd64 \
+              ${REGISTRY_WITH_ORG}/${image}:${LATEST_NEXT}-arm64
+
+            echo "=== Inspecting manifest for ${image}:${LATEST_NEXT} ==="
+            docker buildx imagetools inspect ${REGISTRY_WITH_ORG}/${image}:${LATEST_NEXT}
+          done
Relevance

⭐⭐ Medium

Evidence on shell hardening is mixed; repo has accepted some quoting, but no set -u enforcement
history.

PR-#1585
PR-#2293

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 5 requires hardened shell scripting (strict mode incl. -u, quoting, and required
env validation). The new workflow adds multiple run: | blocks that (a) write to GITHUB_ENV using
an unquoted redirection target and (b) use set -ex (not set -euo pipefail) and unquoted
expansions in loops/commands, without any : "${VAR:?}" validations.

.github/workflows/next-container-build.yaml[76-99]
.github/workflows/next-container-build.yaml[125-139]
.github/workflows/next-container-build.yaml[213-242]
Best Practice: Repository guidelines

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow `run` scripts are not hardened: they omit `set -euo pipefail` (notably `-u`), do not validate required env vars, and contain unquoted variable expansions.

## Issue Context
PR Compliance ID 5 requires strict modes, quoting, and env validation for shell scripts embedded in workflows.

## Fix Focus Areas
- .github/workflows/next-container-build.yaml[71-99]
- .github/workflows/next-container-build.yaml[114-140]
- .github/workflows/next-container-build.yaml[184-242]
- .github/workflows/next-container-build.yaml[247-264]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View more (1)
4. Merge runs without builds ✓ Resolved 🐞 Bug ☼ Reliability
Description
All build/push and marker upload steps are gated by any_changed == 'true', but the merge job
runs unconditionally and will attempt to download markers and create manifests even when nothing was
built. This can fail the workflow (no artifacts/tags) on pushes that only change ignored files
(e.g., docs/tests).
Code

.github/workflows/next-container-build.yaml[R161-183]

+  merge:
+    name: Create multi-arch manifests
+    runs-on: ubuntu-latest
+    needs:
+      - build
+    permissions:
+      contents: read
+      packages: write
+    env:
+      HAS_QUAY_OAUTH: ${{ secrets.QUAY_OAUTH_TOKEN != '' }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 0
+
+      - name: Download build markers
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          path: /tmp/build-markers
+          pattern: build-marker-*
+          merge-multiple: true
+
Relevance

⭐⭐ Medium

No direct precedent for gating merge/manifest job when builds skipped; only general use of
conditional guards (PR #1136).

PR-#1136

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow explicitly skips marker upload when any_changed is false, but still executes merge
and its download/manifest steps without any equivalent guard.

.github/workflows/next-container-build.yaml[39-62]
.github/workflows/next-container-build.yaml[71-160]
.github/workflows/next-container-build.yaml[161-183]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`merge` always runs after the matrix job, but the matrix build/push steps are skipped when `changed-files` reports `any_changed != 'true'`. In that case, `merge` still tries to download build markers and create manifests from tags that were never pushed.

## Issue Context
The build job already computes `steps.changed-files.outputs.any_changed`, but that value is not available to `merge` and the build marker artifacts are only uploaded when changes exist.

## Fix Focus Areas
- .github/workflows/next-container-build.yaml[39-62]
- .github/workflows/next-container-build.yaml[146-160]
- .github/workflows/next-container-build.yaml[161-183]

## Implementation notes
Recommended approach:
1. Add a new `changes` job that runs `tj-actions/changed-files` once and exposes `outputs.any_changed`.
2. Add job-level `if: needs.changes.outputs.any_changed == 'true'` to both `build` and `merge`.
3. Update `merge.needs` to include `changes`.

Alternative (less clean): re-run `changed-files` in `merge` and gate the manifest/cleanup steps on its result.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

Comment thread .github/workflows/next-container-build.yaml
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
@polasudo
polasudo force-pushed the feat/multi-arch-operator-build branch from 367d356 to 38ac280 Compare June 29, 2026 09:52
@polasudo
polasudo marked this pull request as ready for review July 1, 2026 08:22
@polasudo
polasudo requested a review from a team as a code owner July 1, 2026 08:22
@polasudo
polasudo force-pushed the feat/multi-arch-operator-build branch from 09fe0c7 to 256a876 Compare July 1, 2026 09:17
@Fortune-Ndlovu

Copy link
Copy Markdown
Member

/review

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Jul 17, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 9a96327)

Warning

/review is deprecated. Use /agentic_review instead (removal date not yet scheduled).

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Architecture

The bundle is built only on amd64 and published as a single-platform image. Architecture-independent contents do not make its OCI platform metadata multi-arch, so ARM64 pulls or cluster scheduling may reject it. Publish a manifest containing both platforms or create the bundle image with architecture-neutral metadata.

# The bundle image is FROM scratch and contains only YAML manifests/metadata -- no
# compiled binaries -- so its content is identical regardless of build architecture
# (confirmed against the currently-published quay.io/rhdh-community/operator-bundle:next:
# ~40KB, empty base image annotations). It's built and pushed exactly once here, rather
# than per-arch in the build matrix, so the two matrix legs' catalog-build step can both
# safely pull this same already-published, stable tag instead of racing each other to
# push/pull a shared tag themselves (opm index add always pulls the bundle from the
# registry, never the local image -- see the Makefile's own comment on bundle-push).
bundle:
Wrong Registry

Cleanup always calls quay.io even though image publication uses configurable REGISTRY. A non-Quay registry configuration could leave temporary tags behind while potentially deleting matching tags from an unrelated Quay repository. Restrict cleanup to Quay or derive and validate the API endpoint from the configured registry.

for image in "${OPERATOR_IMAGE_NAME}" "${OPERATOR_IMAGE_NAME}-catalog"; do
  for arch in amd64 arm64; do
    for tag in "${BASE_VERSION}-${arch}" "${BASE_VERSION}-${SHORT_SHA}-${arch}" "${LATEST_NEXT}-${arch}"; do
      echo "Deleting per-arch tag: ${image}:${tag}"
      status=$(curl -s -o /dev/null -w '%{http_code}' -X DELETE \
        -H "Authorization: Bearer ${QUAY_OAUTH_TOKEN}" \
        "https://quay.io/api/v1/repository/${NAMESPACE}/${image}/tag/${tag}" \
        || true)
Silent Cleanup Failure

Cleanup accepts every HTTP response and always succeeds, including authentication failures and server errors. This conflicts with the stated cleanup requirement and can silently accumulate per-architecture tags; validate expected success or not-found status codes and fail on other responses.

status=$(curl -s -o /dev/null -w '%{http_code}' -X DELETE \
  -H "Authorization: Bearer ${QUAY_OAUTH_TOKEN}" \
  "https://quay.io/api/v1/repository/${NAMESPACE}/${image}/tag/${tag}" \
  || true)
echo "  -> HTTP ${status} for ${image}:${tag}"
📄 References
  1. No matching references available

@Fortune-Ndlovu Fortune-Ndlovu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for pushing this PR the design is sound and well-documented, I've added a few nitpicks. PTAL thanks

Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
@polasudo
polasudo force-pushed the feat/multi-arch-operator-build branch from b111709 to 2bb081f Compare July 17, 2026 14:48

@Fortune-Ndlovu Fortune-Ndlovu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new workflow_dispatch trigger (line 11) will be silently skipped when the latest commit didn't change relevant files, because tj-actions/changed-files still gates everything. Manual triggers typically mean 'build regardless.' Add || github.event_name == 'workflow_dispatch' to the if: on this line, line135, and line 267 (with parentheses on 267 to preserve the && precedence).

Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml Outdated
@polasudo
polasudo requested a review from Fortune-Ndlovu July 21, 2026 07:30
@rm3l

rm3l commented Jul 21, 2026

Copy link
Copy Markdown
Member

/agentic_review

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Jul 21, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 18 rules
✅ Cross-repo context
  Explored: repo: redhat-developer/rhdh (sha: 5992bddd)
  Not relevant to this PR: redhat-developer/rhdh-plugins

Grey Divider


Action required

1. Bundle pull may break arm 🐞 Bug ≡ Correctness
Description
The arm64 matrix leg runs catalog-build with BUNDLE_IMGS pointing at a bundle image tag
built/pushed only once on an amd64 runner, and opm render may fail on arm64 if it enforces
platform matching for the bundle image. If that happens, the arm64 build leg fails and the merge
job is gated on needs.build.result == 'success', preventing multi-arch manifests from being
published.
Code

.github/workflows/next-container-build.yaml[R204-219]

+          # Point catalog-build's bundle reference at the already-published, stable bundle
+          # tag from the dedicated `bundle` job instead of rebuilding/pushing it again here.
+          export BUNDLE_IMGS="${REGISTRY_WITH_ORG}/${OPERATOR_IMAGE_NAME}-bundle:${VERSION}"

-          export VERSION=${{ env.BASE_VERSION }}
-          export REGISTRY_WITH_ORG=${{ env.REGISTRY }}/${{ env.REGISTRY_ORG }}
-          export OPERATOR_IMAGE_NAME=${OPERATOR_IMAGE_NAME:-operator}
-          export IMAGE_TAG_BASE=${REGISTRY_WITH_ORG}/${OPERATOR_IMAGE_NAME}
+          : "${PLATFORM:?PLATFORM must be set}"
+          : "${SHORT_SHA:?SHORT_SHA must be set}"
+          : "${LATEST_NEXT:?LATEST_NEXT must be set}"
+          : "${OPERATOR_IMAGE_NAME:?OPERATOR_IMAGE_NAME must be set}"

-          set -ex
+          # Build the operator image, then the catalog image. catalog-build's only other
+          # prerequisite (besides the bundle-push we're skipping) is `opm`, which downloads
+          # the opm CLI if needed -- that still runs normally. `-o bundle-push` tells make to
+          # treat that phony prerequisite as already satisfied, so `opm index add` runs
+          # directly against BUNDLE_IMGS above (a safe concurrent *read* of an already-stable
+          # tag) instead of each matrix leg re-pushing its own bundle to a shared tag first.
+          CONTAINER_TOOL="${CONTAINER_TOOL}" VERSION="${VERSION}" PLATFORM="${PLATFORM}" BUNDLE_IMGS="${BUNDLE_IMGS}" make -o bundle-push image-build catalog-build
Relevance

⭐⭐ Medium

Team cares about multi-arch correctness (PR1882), but no prior evidence about opm/bundle cross-arch
pulls failing.

PR-#1882
PR-#1136

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The build job explicitly points catalog-build at a registry-pulled bundle tag, while the bundle is
built once on amd64 with an amd64 default platform; opm render is invoked without a platform
override and therefore may require an arm64-compatible manifest when running on arm64.

.github/workflows/next-container-build.yaml[81-130]
.github/workflows/next-container-build.yaml[195-231]
Makefile[246-257]
Makefile[298-350]
hack/opm-render.sh[1-14]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow builds/pushes the operator-bundle only once (on amd64) and then both amd64 and arm64 `catalog-build` steps pull/render that bundle from the registry. If `opm render` (via containers/image) selects manifests by the host platform, the arm64 job can fail to pull/render an amd64-only bundle image even though its filesystem contents are architecture-independent.

## Issue Context
- `catalog-build` uses `hack/opm-render.sh`, which calls `opm render $BUNDLE_IMGS` without any platform override.
- `bundle-build` uses `--platform $(PLATFORM)` and the default `PLATFORM` is `linux/amd64`.

## Fix Focus Areas
- .github/workflows/next-container-build.yaml[73-135]
- .github/workflows/next-container-build.yaml[195-231]
- Makefile[246-257]
- Makefile[298-350]
- hack/opm-render.sh[1-14]

## Suggested fix approach
1. Build and push bundle images for both amd64 and arm64 (e.g., in the existing matrix, tag `...-bundle:${VERSION}-amd64` and `...-bundle:${VERSION}-arm64`).
2. In `merge`, create/push a multi-arch manifest list for the bundle tags as well (like operator/catalog), and point `BUNDLE_IMGS` at the final multi-arch bundle tag (`...-bundle:${VERSION}`), not a per-arch one.
3. Optionally, keep the single-build optimization only if you can reliably publish a multi-arch manifest list for the bundle (e.g., via copying/annotating) so arm64 pulls always succeed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Cleanup never fails job 🐞 Bug ☼ Reliability
Description
The Quay per-arch tag deletion loop swallows curl failures (|| true) and never validates the
returned HTTP status, so the workflow can succeed even when tags were not deleted (e.g.,
auth/rate-limit/5xx). This creates a false-green for the optional cleanup behavior and can leave tag
clutter behind.
Code

.github/workflows/next-container-build.yaml[R317-323]

+              for tag in "${BASE_VERSION}-${arch}" "${BASE_VERSION}-${SHORT_SHA}-${arch}" "${LATEST_NEXT}-${arch}"; do
+                echo "Deleting per-arch tag: ${image}:${tag}"
+                status=$(curl -s -o /dev/null -w '%{http_code}' -X DELETE \
+                  -H "Authorization: Bearer ${QUAY_OAUTH_TOKEN}" \
+                  "https://quay.io/api/v1/repository/${NAMESPACE}/${image}/tag/${tag}" \
+                  || true)
+                echo "  -> HTTP ${status} for ${image}:${tag}"
Relevance

⭐⭐⭐ High

They’ve accepted failing/handling unexpected HTTP statuses to avoid false-green workflows (PR2828)
and similar reliability hardening (PR3169).

PR-#2828
PR-#3169

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The step captures an HTTP code but continues regardless of curl execution failures and regardless of
whether the HTTP code indicates success, so cleanup problems are not surfaced as job failures.

.github/workflows/next-container-build.yaml[304-326]
PR-#2828

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The cleanup step prints HTTP status codes from Quay tag deletions but never checks them, and it also suppresses curl transport failures. This makes cleanup silently unreliable.

## Issue Context
Cleanup is conditional (`HAS_QUAY_OAUTH`) and runs after manifests are created, so it primarily impacts registry hygiene and operational signal rather than artifact correctness.

## Fix Focus Areas
- .github/workflows/next-container-build.yaml[304-330]

## Suggested fix approach
- Remove `|| true` from the curl call (or keep it only to capture a distinct status like `000`), and explicitly handle acceptable statuses:
 - Treat `200/204` as success.
 - Treat `404` as already-deleted (optional, but often reasonable).
 - Fail the step on `401/403/429/5xx` (or at least emit `::warning::` and set a failure flag, then `exit 1` at the end if any unexpected statuses occurred).
- Consider printing a GitHub Actions annotation summarizing any failures to improve visibility.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

@polasudo
polasudo force-pushed the feat/multi-arch-operator-build branch from a61c9dc to 9a96327 Compare July 28, 2026 08:08
@Fortune-Ndlovu

Copy link
Copy Markdown
Member

/review

@rhdh-qodo-merge

Copy link
Copy Markdown

Persistent review updated to latest commit 9a96327

Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml
Comment thread .github/workflows/next-container-build.yaml Outdated
Comment thread .github/workflows/next-container-build.yaml
polasudo and others added 12 commits July 29, 2026 11:03
Replace the single-runner amd64-only build with a matrix strategy
that builds on both ubuntu-24.04 (amd64) and ubuntu-24.04-arm (arm64),
then merges per-arch images into multi-arch manifest lists.

This follows the same proven pattern used by the hub repo's
next-build-image.yaml workflow. The operator Dockerfile already
supports TARGETOS/TARGETARCH and bundle images use FROM scratch,
so no other changes are needed.

All three images (operator, operator-bundle, operator-catalog) are
now built and published as multi-arch manifest lists.

Assisted-by: Claude Code
GitHub Actions doesn't allow direct secrets references in step `if`
conditions. Use an env var intermediary instead.

Assisted-by: Claude Code
The --annotation flag is not supported by docker buildx imagetools
create on the ubuntu-latest runner. Expiry labels are already set
during the per-arch image build via the Makefile LABEL variable.

Assisted-by: Claude Code
- Use ${{ github.ref_name }} instead of detached-HEAD-unsafe git command
  for branch detection; avoids mistagging images (next vs latest)
- Add strict shell mode (set -euo pipefail) and env var validation
  across all run: blocks; quote all variable expansions
- Fix SHA-scoped manifest list to use ${BASE_VERSION}-${SHORT_SHA}-{amd64,arm64}
  instead of ${BASE_VERSION}-{amd64,arm64} (was using wrong image content)
- Add separate `changes` job to gate both `build` and `merge` on
  any_changed output; prevents merge from running when no files changed
- Remove unquoted redirections and improve shell robustness

All 4 Qodo findings resolved.

Assisted-by: Claude Code
Add if: needs.changes.outputs.any_changed == 'true' to individual
steps in build job (Prepare, Setup Go, Login, Build, Upload marker)
to explicitly skip when no relevant changes detected. This makes the
gating more granular and avoids unnecessary compute.

Add clarifying comment above Prepare step.

Assisted-by: Claude Code
GitHub Actions doesn't support bash-style default syntax (:-).
Use || 'operator' instead for fallback values.

Assisted-by: Claude Code
- Move QUAY_OAUTH_TOKEN from inline ${{ secrets }} expansion in the
  curl command to the step's env: block, resolving SonarCloud warning
  about secrets expanded directly in run steps
- Add always() && needs.build.result == 'success' to merge job's if
  condition so it won't attempt manifest creation when build failed

Assisted-by: Claude Code
operator-bundle is FROM scratch (pure YAML manifests/metadata, no
compiled binaries), so its content is identical regardless of build
architecture. The build matrix was rebuilding and pushing it on both
the amd64 and arm64 legs to the same shared tag, which catalog-build's
opm index add then pulled back non-deterministically (opm always pulls
the bundle from the registry, never the local image).

Build and push the bundle exactly once in a new, non-matrixed job.
Both matrix legs' catalog-build now read that already-published,
stable tag instead of racing to write it -- via `make -o bundle-push`,
which skips catalog-build's phony bundle-push prerequisite. Verified
locally end-to-end: real bundle build (FROM scratch, no RUN steps),
real opm index add against a local registry, real arm64 catalog build.

Assisted-by: Claude Code
Centralize image metadata in the changes job, drop unused skopeo and
build markers, remove redundant per-step if guards, pass image name via
env blocks, and log Quay cleanup HTTP status codes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Manual runs should always build even when changed-files reports no
matching paths, matching Fortune's review suggestions.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Fortune Ndlovu <ndlovufortune97@gmail.com>
Remove unused PLATFORM_PAIR, move github.ref_name into an env block
to avoid Actions expression injection in run scripts, and warn on
unexpected Quay tag-deletion HTTP status codes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@polasudo
polasudo force-pushed the feat/multi-arch-operator-build branch from aa2f840 to 90c5644 Compare July 29, 2026 09:04
@sonarqubecloud

Copy link
Copy Markdown

@polasudo
polasudo requested a review from Fortune-Ndlovu July 29, 2026 09:05
@polasudo

Copy link
Copy Markdown
Member Author

Addressed the outstanding review comments and rebased onto main:

  1. Removed unused PLATFORM_PAIR
  2. Clarified SonarCloud vars.* note — those values were already in env: blocks; also moved github.ref_name into env: in the changes job
  3. Added Quay cleanup HTTP status warnings (200/204/404 OK, others emit ::warning::)

@Fortune-Ndlovu PTAL — ready for re-review. Thanks!

@Fortune-Ndlovu Fortune-Ndlovu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@openshift-ci openshift-ci Bot added the lgtm label Jul 29, 2026
@Fortune-Ndlovu

Copy link
Copy Markdown
Member

/lgtm

@openshift-merge-bot
openshift-merge-bot Bot merged commit 27f8511 into redhat-developer:main Jul 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants