Prod changeset sync — Stage 1: infra prep (bucket + dedicated IRSA role) - #1061
Merged
Conversation
Stage 1 (infrastructure prep) for enabling registry changeset sync in production. Additive only — does not touch the running prod app. - Provision the cer-registry-changesets-prod S3 bucket (reuses the generic bucket module: versioning, AES256, public-access-block). - Add a dedicated least-privilege prod IRSA role (ce-registry-eks-prod-application-irsa-role) so prod is isolated from the shared staging/prod role. Scoped to the buckets the prod app actually uses, verified against its S3 code paths and DB config: cer-envelope-downloads, cer-envelope-graphs-prod-us-east-1, ocn-exports (ce_registry OCN export is enabled in prod), and cer-registry-changesets-prod. Deferred to the app cutover (Stage 2): repoint the prod service account to the new role + remove credreg-prod from the shared role trust, configmap env vars, image bump, migration, and backlog handling.
arielr-lt
marked this pull request as ready for review
August 11, 2026 14:29
chuang-CE
approved these changes
Aug 11, 2026
arielr-lt
added a commit
that referenced
this pull request
Aug 11, 2026
… runbook) (#1062) **Stage 2 (app cutover) for registry changeset sync in production.** Depends on Stage 1 (#1061, infra — already applied). **Draft — do not merge/apply until the deploy window.** ## Changes (`k8s-manifests-prod/`) - **app-configmap**: add `REGISTRY_CHANGESET_SYNC_*` + `AWS_REGION`. **`REGISTRY_CHANGESET_SYNC_ENDPOINT` intentionally omitted** — changesets generate + store in S3 but aren't delivered, so the Publisher blob can be backfilled first; endpoint is set afterward. - **app-service-account**: repoint to the dedicated least-privilege `ce-registry-eks-prod-application-irsa-role` (verified: db-dump jobs use a separate SA, so nothing loses bucket access). - **app/worker/db-migrate**: pin immutable image `2026.07.30.0154` (was floating `:master`/`:production`). Tag verified in ECR. - **RUNBOOK**: full ordered cutover (marker reset → apply SA+configmap → Deploy workflow + migration → verify → set endpoint after backfill) + rollback notes. ## Key deploy facts - Marker reset needed for **`ce_registry` only** (stale row ~89K versions behind); other communities self-initialize on first publish. Reset runs first (image-independent) to pre-empt the ~180 MB backlog ZIP under the publish-lock. - The **Deploy image** workflow runs the migration **after** the rollout, so new pods (which don't use the Argo columns) are up before `RemoveArgoFields` drops them. - Configmap + SA are **not** applied by the Deploy workflow — applied separately per the runbook. ## Not in this PR - Setting `REGISTRY_CHANGESET_SYNC_ENDPOINT` (post-backfill, via configmap workflow). - Publisher-side blob backfill (Cary) + accumulated-changeset apply. --------- Co-authored-by: Ariel Rolfo <arielr-lt+username@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 (infrastructure prep) for enabling registry changeset sync in production. Additive only; does not touch the running prod app.
Status: APPLIED (2026-08-11)
Applied to prod via a targeted apply of only these resources — verified 8 added, 0 changed, 0 destroyed. Live-verified: bucket is private (public-access-block) + versioned + AES256; role trusts only
credreg-prod:main-app-service-account. The pre-existing Phase-2 drift below was deliberately left untouched. This PR merges to bringmasterin sync with what's applied.What this adds
cer-registry-changesets-prod(versioning + AES256 + public-access-block).ce-registry-eks-prod-application-irsa-role, trusting onlycredreg-prod:main-app-service-account— isolates prod from the shared staging/prod role.cer-envelope-downloads,cer-envelope-graphs-prod-us-east-1,ocn-exports(prodce_registryOCN export is enabled),cer-registry-changesets-prod.Deferred to Stage 2 (app cutover) — intentionally not here
REGISTRY_CHANGESET_SYNC_*vars (endpoint initially unset), image bump to2026.07.30.0154, DB migration, and sync-marker reset to avoid the historical backlog.Note on the CI plan
A full plan here also shows the pre-existing Phase-2 drift (db-dumps policy,
ng_sandbox_largemin_size, rds-sandbox SG) — those are not from this PR and were not applied. They will be reconciled separately.