fix(ci): fix operator deployments on K8s — inherit rendering, rollout wait, and orchestrator config [release-1.9] by gustavolira · Pull Request #4423 · redhat-developer/rhdh

gustavolira · 2026-03-18T14:39:04Z

Summary

Fixes multiple CI issues affecting K8s operator deployments (AKS, EKS, GKE, OSD-GCP) on the release-1.9 branch.

1. Render Helm template escapes in operator ConfigMap

Value files use {{ "{{" }}inherit{{ "}}" }} which Helm renders automatically, but operator deployments use these files directly via yq without Helm rendering. Added sed post-processing in config::create_dynamic_plugins_config to convert the escapes to {{inherit}}.

Related: #4414 (merged to main)

2. Wait for operator rollout to stabilize

Added wait_for_operator_rollout() to AKS, EKS, and GKE operator deployment scripts. The RHDH operator may update the Deployment spec multiple times after the Backstage CR is applied, causing race conditions with subsequent steps (e.g., ingress patching, spot instance configuration).

3. Orchestrator config in diff-values files

Moved orchestrator-related plugin configuration to the appropriate diff-values files and disabled orchestrator plugins on K8s RBAC deployments where they are not needed.

Changed files

.ci/pipelines/utils.sh — {{inherit}} sed fix + wait_for_operator_rollout() function
.ci/pipelines/cluster/{aks,eks,gke}/*-operator-deployment.sh — call wait_for_operator_rollout
.ci/pipelines/value_files/diff-values_showcase*.yaml — orchestrator config adjustments

Test plan

Verify AKS operator nightly passes
Verify EKS operator nightly passes
Verify GKE operator nightly passes
Verify OSD-GCP nightly passes
Verify {{inherit}} is correctly rendered in operator dynamic-plugins ConfigMap

🤖 Generated with Claude Code

gustavolira · 2026-03-18T14:41:03Z

/test ?

gustavolira · 2026-03-18T14:41:31Z

/test e2e-gke-operator-nightly

github-actions · 2026-03-18T15:21:07Z

Image was built and published successfully. It is available at:

github-actions · 2026-03-18T17:06:10Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-03-18T17:07:07Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-03-18T19:24:20Z

/test ?

gustavolira · 2026-03-18T19:24:40Z

/test e2e-ocp-operator-nightly

gustavolira · 2026-03-18T23:40:55Z

The operator wasn’t running Backstage, it failed during the dynamic plugins container installation, so Backstage never started. Now Backstage is up and the tests are running as expected.

The operator creates orchestrator resources asynchronously after the Backstage CR is applied. The cleanup now polls for up to 60s for the sonataflow jobs to appear before attempting deletion, instead of running immediately when the resources may not exist yet. Addresses review feedback: redhat-developer#4423 (comment) [skip-build] Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions · 2026-03-19T13:31:14Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

zdrapela · 2026-03-19T14:13:24Z

/test e2e-eks-operator-nightly
/test e2e-aks-helm-nightly

github-actions · 2026-03-19T17:24:37Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-03-19T17:42:04Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-03-19T17:50:20Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-03-19T17:59:52Z

/test e2e-eks-operator-nightly
/test e2e-aks-helm-nightly

github-actions · 2026-03-19T19:50:52Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-03-19T20:02:34Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-03-19T20:18:24Z

/test e2e-eks-operator-nightly
/test e2e-aks-helm-nightly

gustavolira · 2026-03-19T21:56:14Z

@zdrapela now the issues were resolved!

github-actions · 2026-03-19T23:27:02Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-03-19T23:46:27Z

/test e2e-eks-operator-nightly
/test e2e-aks-helm-nightly

github-actions · 2026-03-20T12:46:01Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-03-23T23:35:17Z

/test e2e-ocp-helm

The operator creates orchestrator resources asynchronously after the Backstage CR is applied. The cleanup now polls for up to 60s for the sonataflow jobs to appear before attempting deletion, instead of running immediately when the resources may not exist yet. Addresses review feedback: redhat-developer#4423 (comment) [skip-build] Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions · 2026-03-23T23:40:35Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-03-24T12:21:02Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

zdrapela

Shouldn't wait_for_operator_rollout be used in ocp-operator.sh after deploy_rhdh_operator too? Or is this K8s specific?

…deploy Address review comment from zdrapela on PR redhat-developer#4423: ocp-operator.sh was missing wait_for_operator_rollout calls after deploy_rhdh_operator, inconsistent with AKS/EKS/GKE scripts. Added the wait in all 5 call sites across initiate_operator_deployments, initiate_operator_deployments_osd_gcp, and run_operator_runtime_config_change_tests to prevent the same race condition on OCP deployments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-03-24T18:49:24Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-03-24T18:53:37Z

Shouldn't wait_for_operator_rollout be used in ocp-operator.sh after deploy_rhdh_operator too? Or is this K8s specific?

Thanks, you are right. I just fixed it

openshift-ci · 2026-03-24T19:34:14Z

@gustavolira: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gke-operator-nightly	`2fc0eee`	link	false	`/test e2e-gke-operator-nightly`
ci/prow/e2e-ocp-operator-nightly	`c451bc3`	link	false	`/test e2e-ocp-operator-nightly`
ci/prow/e2e-eks-operator-nightly	`e0ac2e9`	link	false	`/test e2e-eks-operator-nightly`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

github-actions · 2026-03-24T20:18:31Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

…figMap The value files use Helm escaping for {{inherit}} tags: {{ "{{" }}inherit{{ "}}" }}. Helm renders this automatically, but operator deployments use create_dynamic_plugins_config() which extracts the YAML directly without Helm rendering. The install-dynamic-plugins.py init container then fails because it receives the raw Helm escape syntax instead of {{inherit}}. Add sed substitutions to strip the Helm escaping when generating the dynamic plugins ConfigMap for operator deployments. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The RHDH operator's internal Helm chart creates orchestrator/SonataFlow infrastructure (database jobs, platform resources) even on non-OpenShift environments where it's not supported. This causes CreateContainerConfigError on GKE/AKS/EKS because required secrets and operators are not available. Add cleanup_orchestrator_resources() that removes SonataFlow jobs and resources after operator deployment on K8s. Called in all GKE, AKS, and EKS operator deployment scripts for both showcase and showcase-rbac. [skip-build] Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The operator creates orchestrator resources asynchronously after the Backstage CR is applied. The cleanup now polls for up to 60s for the sonataflow jobs to appear before attempting deletion, instead of running immediately when the resources may not exist yet. Addresses review feedback: redhat-developer#4423 (comment) [skip-build] Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The RBAC diff files for AKS/EKS/GKE had `plugins: []` which left orchestrator plugins from the base file enabled after merge. This caused init containers to download unnecessary OCI plugins and the operator to create sonataflow-database jobs that fail (missing PostgreSQL secrets on K8s). Add explicit orchestrator plugin disable entries to all K8s RBAC diff files and fix cleanup_orchestrator_resources to match job names by prefix instead of exact field-selector (jobs have random suffixes). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…unction Fix prettier code style in utils.sh and add explicit return statement at the end of cleanup_orchestrator_resources (Sonar S7682). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…oyments The cleanup function deletes resources managed by the RHDH operator, triggering reconciliation that creates a second ReplicaSet (double rollout). On EKS this causes PostgreSQL EBS Multi-Attach errors because the volume can't be reattached to a different node during the rollout. Since orchestrator plugins are now disabled in all K8s RBAC diff files, the failing sonataflow-database jobs are harmless background noise that don't block Backstage from starting. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add --set orchestrator.enabled=false to all K8s helm upgrade commands (AKS, EKS, GKE) to prevent the chart from creating sonataflow-database jobs and other orchestrator infrastructure that is not supported on K8s. The orchestrator: null in diff files gets removed by yq merge's del(.. | select(. == null)), so the chart falls back to its default (enabled: true). The --set flag takes highest precedence in Helm. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The RHDH operator updates the Deployment spec multiple times after the Backstage CR is applied, creating multiple ReplicaSets (double rollout). Two competing pods cause 503 errors as they conflict on the database. Add wait_for_operator_rollout() that waits for the deployment to exist and for kubectl rollout status to complete before proceeding, ensuring only one ReplicaSet is active. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…d code Move `orchestrator.enabled: false` from `--set` flags in helm commands to the diff-values YAML files for all K8s platforms (EKS, AKS, GKE, OSD-GCP, upgrade). This follows the existing pattern and makes the configuration more readable and declarative. Also removes the unused `cleanup_orchestrator_resources()` function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…deploy Address review comment from zdrapela on PR redhat-developer#4423: ocp-operator.sh was missing wait_for_operator_rollout calls after deploy_rhdh_operator, inconsistent with AKS/EKS/GKE scripts. Added the wait in all 5 call sites across initiate_operator_deployments, initiate_operator_deployments_osd_gcp, and run_operator_runtime_config_change_tests to prevent the same race condition on OCP deployments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-03-25T14:24:46Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

sonarqubecloud · 2026-03-25T14:26:16Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

gustavolira · 2026-03-25T18:30:29Z

/test e2e-aks-helm-nightly

openshift-ci Bot requested review from kadel and rostalan March 18, 2026 14:39

gustavolira force-pushed the fix/ci-operator-inherit-template branch from e84fb70 to c451bc3 Compare March 18, 2026 17:06

zdrapela reviewed Mar 19, 2026

View reviewed changes

Comment thread .ci/pipelines/cluster/aks/aks-operator-deployment.sh Outdated

zdrapela reviewed Mar 20, 2026

View reviewed changes

Comment thread .ci/pipelines/cluster/aks/aks-helm-deployment.sh Outdated

zdrapela reviewed Mar 20, 2026

View reviewed changes

Comment thread .ci/pipelines/utils.sh Outdated

zdrapela suggested changes Mar 20, 2026

View reviewed changes

openshift-ci Bot assigned zdrapela Mar 20, 2026

gustavolira force-pushed the fix/ci-operator-inherit-template branch from f955465 to 851cb28 Compare March 23, 2026 23:40

openshift-ci Bot removed the lgtm label Mar 23, 2026

gustavolira changed the title ~~fix(ci): render Helm template escapes in operator dynamic plugins ConfigMap~~ fix(ci): fix operator deployments on K8s — inherit rendering, rollout wait, and orchestrator config [release-1.9] Mar 23, 2026

zdrapela reviewed Mar 24, 2026

View reviewed changes

gustavolira and others added 10 commits March 25, 2026 11:22

fix(ci): fix prettier formatting and add explicit return to cleanup f…

e4a5daa

…unction Fix prettier code style in utils.sh and add explicit return statement at the end of cleanup_orchestrator_resources (Sonar S7682). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

gustavolira force-pushed the fix/ci-operator-inherit-template branch from 7268891 to 4cd1983 Compare March 25, 2026 14:24

zdrapela approved these changes Mar 26, 2026

View reviewed changes

openshift-ci Bot added the lgtm label Mar 26, 2026

openshift-merge-bot Bot merged commit 5cc4db2 into redhat-developer:release-1.9 Mar 26, 2026
16 checks passed

Uh oh!

Conversation

gustavolira commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. Render Helm template escapes in operator ConfigMap

2. Wait for operator rollout to stabilize

3. Orchestrator config in diff-values files

Changed files

Test plan

Uh oh!

gustavolira commented Mar 18, 2026

Uh oh!

gustavolira commented Mar 18, 2026

Uh oh!

github-actions Bot commented Mar 18, 2026

Uh oh!

github-actions Bot commented Mar 18, 2026

Uh oh!

github-actions Bot commented Mar 18, 2026

Uh oh!

gustavolira commented Mar 18, 2026

Uh oh!

gustavolira commented Mar 18, 2026

Uh oh!

gustavolira commented Mar 18, 2026

Uh oh!

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

zdrapela commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

gustavolira commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

gustavolira commented Mar 19, 2026

Uh oh!

gustavolira commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

gustavolira commented Mar 19, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

gustavolira commented Mar 23, 2026

Uh oh!

github-actions Bot commented Mar 23, 2026

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

zdrapela left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

gustavolira commented Mar 24, 2026

Uh oh!

openshift-ci Bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

Uh oh!

sonarqubecloud Bot commented Mar 25, 2026

Quality Gate passed

Uh oh!

gustavolira commented Mar 25, 2026

gustavolira commented Mar 18, 2026 •

edited

Loading

openshift-ci Bot commented Mar 24, 2026 •

edited

Loading