ci(release): parallelize image builds + add buildx cache + fix e2e#374
Merged
Merged
Conversation
Three changes to release.yml, addressing distinct issues: 1. **Parallel matrix builds**. The build-and-push job built kagenti- operator and agentcard-signer sequentially in one runner — with QEMU-emulated arm64 builds taking ~20 minutes each, total wall time was ~42 minutes. Refactor into a matrix of two jobs (one per image) running on separate runners. Wall time drops to ~22 minutes (max of the two parallel jobs) with no other change. 2. **buildx GitHub Actions cache**. Add cache-from / cache-to with per-image scope so subsequent runs benefit from layer reuse. Warm builds drop further (~5–10 minutes typical). Per-image scope prevents cache key collisions across the matrix. 3. **e2e-helm-install image-tag override**. The chart ships with `tag: __PLACEHOLDER__`, which is substituted only in the tag-push release path. On main pushes the e2e job re-checks-out the repo (pristine values.yaml), runs `helm install` with no override, and the rendered Deployment tries to pull `:__PLACEHOLDER__` — ErrImagePull, deployment never Ready, 120s rollout timeout. Pass `--set controllerManager.container.image.tag=latest` so the e2e actually exercises the just-built image. Bonus: split the helm-chart-packaging + GitHub-Release steps into a separate `release` job (gated `if: github.ref_type == 'tag'`) since the original sequence-of-steps shape doesn't survive matrix-ization. Also harden the `gh release create` step against `Release.tag_name already exists` (occurs when a maintainer pre-creates the release out of band before pushing the tag). The e2e-helm-install bug is pre-existing (every recent main push has been failing on the rollout timeout). The build slowness is pre-existing too. Bundling the fixes lets one workflow review cover them together. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
Address review feedback on PR rossoctl#374. The previous idempotency guard grepped stderr from `gh release create` for "already_exists" / "Release.tag_name already exists" — fragile to gh CLI message changes and locale. Replace with a `gh release view` precheck: short-circuit if the release already exists, otherwise create it. Same outcome, no string matching. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
pdettori
approved these changes
May 23, 2026
pdettori
left a comment
Member
There was a problem hiding this comment.
Clean, well-structured CI refactoring. Three solid improvements:
- Parallel matrix builds — correct
fail-fast: false, per-image cache scoping prevents collisions,needs: build-and-pushgates the release job properly. - buildx GHA cache — per-image
scopeis the right approach for matrix builds sharing the same Actions cache namespace. - E2e fix —
--set controllerManager.container.image.tag=latestcorrectly resolves the__PLACEHOLDER__issue on main-push runs.
The gh release view precheck is cleaner than stderr-grepping — good response to review feedback.
Areas reviewed: CI/GitHub Actions, Shell scripting, Security (action pinning)
Commits: 2 commits, all signed-off ✓
CI status: All 15 checks passing ✓
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three changes to
.github/workflows/release.yml, all addressing pre-existing CI issues.1. Parallelize the two image builds (matrix)
build-and-pushwas a single job buildingkagenti-operatorandagentcard-signersequentially. With QEMU-emulatedlinux/arm64builds, each took ~20 minutes. Total wall time: ~42 minutes per run.Refactored into a matrix of two jobs running in parallel on separate runners. Wall time drops to ~22 minutes (the longest of the two parallel builds).
2. Add buildx GitHub Actions cache
Each matrix instance now has
cache-from: type=gha/cache-to: type=gha,mode=maxwith a per-imagescope. Warm builds (where Go module cache + earlier layers can be reused) typically drop to ~5–10 minutes per image. Per-image scoping prevents cache collisions between the two matrix variants.3. Fix
e2e-helm-installimage-tag override (the actual bug)The chart's
values.yamlships with:The placeholder is substituted only in the tag-push code path:
The mutation lives only in that job's working directory. On main pushes, the
e2e-helm-installjob re-checks-out the repo (pristine__PLACEHOLDER__), runshelm install kagenti-operator ./charts/kagenti-operatorwith no--set, and the rendered Deployment tries to pullghcr.io/.../kagenti-operator:__PLACEHOLDER__—ErrImagePull→ 120s rollout timeout.Every recent main-push run of this workflow has been failing for this reason (PRs #366, #369, #370, #373). This means the e2e step has been providing zero coverage — it can't catch real regressions when it always fails for the same trivial reason.
Fix: pass
--set controllerManager.container.image.tag=latest. The build-and-push job tags:lateston default-branch pushes, so the e2e installs whatever was just built.Bonus changes
Split release-only steps into a
releasejob (gatedif: github.ref_type == 'tag'). The original sequence-of-steps shape doesn't survive matrix-ization since the helm-chart packaging andgh release createshouldn't fan out per image. Thereleasejob runs once, after both matrix jobs complete.Idempotent
Create GitHub Release: treatRelease.tag_name already exists(HTTP 422) as a benign no-op. This handles the case where a maintainer rangh release createout of band before pushing the tag (e.g., for custom release notes).Test plan
This PR's branch is on a feature branch in the fork; pushing the PR will trigger the modified workflow on a
mainpush (the merge commit). Expected outcomes:build-and-pushmatrix produces 2 parallel jobs (kagenti-operator, agentcard-signer)e2e-helm-installruns and the Deployment becomes Ready (because:latestresolves)releasejob (since this is a main push, not a tag push)Once merged, the next maintainer-cut tag will exercise the
releasejob (helm chart push +gh release create).Sequencing
This PR is independent of any other in-flight work. Can land at any time.
Assisted-By: Claude Code