Skip to content

ci(release): parallelize image builds + add buildx cache + fix e2e#374

Merged
huang195 merged 2 commits into
rossoctl:mainfrom
huang195:ci/parallel-builds-and-e2e-fix
May 23, 2026
Merged

ci(release): parallelize image builds + add buildx cache + fix e2e#374
huang195 merged 2 commits into
rossoctl:mainfrom
huang195:ci/parallel-builds-and-e2e-fix

Conversation

@huang195

Copy link
Copy Markdown
Member

What

Three changes to .github/workflows/release.yml, all addressing pre-existing CI issues.

1. Parallelize the two image builds (matrix)

build-and-push was a single job building kagenti-operator and agentcard-signer sequentially. With QEMU-emulated linux/arm64 builds, each took ~20 minutes. Total wall time: ~42 minutes per run.

Refactored into a matrix of two jobs running in parallel on separate runners. Wall time drops to ~22 minutes (the longest of the two parallel builds).

2. Add buildx GitHub Actions cache

Each matrix instance now has cache-from: type=gha / cache-to: type=gha,mode=max with a per-image scope. Warm builds (where Go module cache + earlier layers can be reused) typically drop to ~5–10 minutes per image. Per-image scoping prevents cache collisions between the two matrix variants.

3. Fix e2e-helm-install image-tag override (the actual bug)

The chart's values.yaml ships with:

controllerManager:
  container:
    image:
      tag: __PLACEHOLDER__

The placeholder is substituted only in the tag-push code path:

- name: Package and push Helm chart
  if: github.ref_type == 'tag'
  run: |
    yq -i '.controllerManager.container.image.tag = strenv(chartVersion)' values.yaml
    helm package . ...

The mutation lives only in that job's working directory. On main pushes, the e2e-helm-install job re-checks-out the repo (pristine __PLACEHOLDER__), runs helm install kagenti-operator ./charts/kagenti-operator with no --set, and the rendered Deployment tries to pull ghcr.io/.../kagenti-operator:__PLACEHOLDER__ErrImagePull → 120s rollout timeout.

Every recent main-push run of this workflow has been failing for this reason (PRs #366, #369, #370, #373). This means the e2e step has been providing zero coverage — it can't catch real regressions when it always fails for the same trivial reason.

Fix: pass --set controllerManager.container.image.tag=latest. The build-and-push job tags :latest on default-branch pushes, so the e2e installs whatever was just built.

Bonus changes

  • Split release-only steps into a release job (gated if: github.ref_type == 'tag'). The original sequence-of-steps shape doesn't survive matrix-ization since the helm-chart packaging and gh release create shouldn't fan out per image. The release job runs once, after both matrix jobs complete.

  • Idempotent Create GitHub Release: treat Release.tag_name already exists (HTTP 422) as a benign no-op. This handles the case where a maintainer ran gh release create out of band before pushing the tag (e.g., for custom release notes).

Test plan

This PR's branch is on a feature branch in the fork; pushing the PR will trigger the modified workflow on a main push (the merge commit). Expected outcomes:

  • build-and-push matrix produces 2 parallel jobs (kagenti-operator, agentcard-signer)
  • Each finishes in 5–22 minutes (cold cache the first time, warmer thereafter)
  • e2e-helm-install runs and the Deployment becomes Ready (because :latest resolves)
  • No release job (since this is a main push, not a tag push)

Once merged, the next maintainer-cut tag will exercise the release job (helm chart push + gh release create).

Sequencing

This PR is independent of any other in-flight work. Can land at any time.

Assisted-By: Claude Code

Three changes to release.yml, addressing distinct issues:

1. **Parallel matrix builds**. The build-and-push job built kagenti-
   operator and agentcard-signer sequentially in one runner — with
   QEMU-emulated arm64 builds taking ~20 minutes each, total wall
   time was ~42 minutes. Refactor into a matrix of two jobs (one
   per image) running on separate runners. Wall time drops to ~22
   minutes (max of the two parallel jobs) with no other change.

2. **buildx GitHub Actions cache**. Add cache-from / cache-to with
   per-image scope so subsequent runs benefit from layer reuse.
   Warm builds drop further (~5–10 minutes typical). Per-image
   scope prevents cache key collisions across the matrix.

3. **e2e-helm-install image-tag override**. The chart ships with
   `tag: __PLACEHOLDER__`, which is substituted only in the tag-push
   release path. On main pushes the e2e job re-checks-out the repo
   (pristine values.yaml), runs `helm install` with no override, and
   the rendered Deployment tries to pull `:__PLACEHOLDER__` —
   ErrImagePull, deployment never Ready, 120s rollout timeout. Pass
   `--set controllerManager.container.image.tag=latest` so the e2e
   actually exercises the just-built image.

Bonus: split the helm-chart-packaging + GitHub-Release steps into a
separate `release` job (gated `if: github.ref_type == 'tag'`) since
the original sequence-of-steps shape doesn't survive matrix-ization.
Also harden the `gh release create` step against `Release.tag_name
already exists` (occurs when a maintainer pre-creates the release
out of band before pushing the tag).

The e2e-helm-install bug is pre-existing (every recent main push has
been failing on the rollout timeout). The build slowness is
pre-existing too. Bundling the fixes lets one workflow review cover
them together.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Address review feedback on PR rossoctl#374. The previous idempotency guard
grepped stderr from `gh release create` for "already_exists" /
"Release.tag_name already exists" — fragile to gh CLI message
changes and locale. Replace with a `gh release view` precheck:
short-circuit if the release already exists, otherwise create it.

Same outcome, no string matching.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>

@pdettori pdettori left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-structured CI refactoring. Three solid improvements:

  1. Parallel matrix builds — correct fail-fast: false, per-image cache scoping prevents collisions, needs: build-and-push gates the release job properly.
  2. buildx GHA cache — per-image scope is the right approach for matrix builds sharing the same Actions cache namespace.
  3. E2e fix--set controllerManager.container.image.tag=latest correctly resolves the __PLACEHOLDER__ issue on main-push runs.

The gh release view precheck is cleaner than stderr-grepping — good response to review feedback.

Areas reviewed: CI/GitHub Actions, Shell scripting, Security (action pinning)
Commits: 2 commits, all signed-off ✓
CI status: All 15 checks passing ✓

@huang195
huang195 merged commit 7b56efd into rossoctl:main May 23, 2026
15 checks passed
@github-project-automation github-project-automation Bot moved this from New /:ToDo to Done in Kagenti Issue Prioritization May 23, 2026
@huang195
huang195 deleted the ci/parallel-builds-and-e2e-fix branch May 23, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants